Reflections on quitting my job.
Programming lessons I’ve learnt over the last 7 years.
Last month I quit my job. There were several reasons for my decision but it mainly boiled down to me turning 30 this year, having an existential crisis questioning wtf I’m doing with my life and ultimately, realising I wasn’t having fun programming professionally anymore. I concluded that I needed some time away to fall back “in love” with programming, as a hobby with no financial strings attached to it. Obviously, I recognise this is an incredibly privileged thing to do, not working is something that many people can only dream of.
So here I am, in my PJs after waking up at 12pm on a Tuesday, reflecting on my time working as a “professional” programmer, a career I’ve largely fallen into by accident.
Backstory
I was never one of those kids that got into programming at 10 yrs old. When I was 10 I was at the cage in the estate next to my nan’s house shouting ”top bins!” as I did my best Bend it Like Beckham impression (I’m still bitter I could never get the spiky hair going). Throughout my school life my main guiding heuristic was fun, I only ever did things I enjoyed. So when the time came to pick a degree I chose psychology because the syllabus looked cool and I’ve always found humans pretty interesting. Its funny, as I’ve gotten more experience as a developer one of the things that stand out to me the most is that there are a lot more human problems than people realise in any given software project. Problems that cant simply be coded away, so understanding humans has been surprisingly useful.
I’ve always been an avid reader of blogs and online publications. While at uni I has subscriptions to Scientific American, Wired, Arch Daily and Dezeen (what can I say, I’m a cultured guy). Toward the end of my time at uni there was a lot of buzz around CRISPR and self-driving cars. It was around then that I heard of neural networks for the first time. I think the fact that I understood the underlying analogy of neural nets was really important in helping me get over the subconscious mental barrier that I couldn’t be involved in technology in any other way than as a consumer.
Anyways, as I got toward the end of my degree I could no longer ignore the ”What are you going to do with psychology?” question. After sitting on the question for some time I (naively) figured the three most important areas in the working world I could have an impact on were: humans, finance and technology. I knew I didn’t want to become a therapist because I had already been unofficially doing that for my friends and family for a while and peoples problems are tiring af. I’d always disliked finance (and still do tbh) so the only option left was technology.
I remember the day I wrote my fist line of code, it was 2nd of Jan 2016. I was 2 months away from 23. I started the day full of optimism and tried the python course on CodeAcademy (it was still free back then) and literally everything went over my head. I couldn’t grasp any of it. I remember feeling like an idiot for even thinking I could do this at 23. The failure confirmed the stereotypes I’d seen on TV, that if you didn’t start writing code at 10, then this wasn’t for you. Just as I was about to give up I decided to try CodeAcademy’s HTLM&CSS course. To my surprise, I managed to whizz through it and firmly understood the material. I genuinely think if I hadn’t done that course I might have become an investment banker or something 🤢.
Fast forward two years, an internship at AKQA, a shitty SWE job at a shitty startup, and then I got the offer to join the team I have just left. The best description of our role in the company is probably what AWS is to Amazon, the team manages the company’s internal infrastructure and data centre. Everything from procuring racks and servers; designing and implementing the data centres network topology; configuring servers and all the services that run on top (all the way up to JS for the interface used by other teams); configuring and running a bare metal Kubernetes (k8s) cluster and probably a lot more stuff that currently escapes my mind. It is a real diverse set of things to work on.
After exactly 5 yrs (to the day) with the team, I decided it was time to pack it all in for a bit. However, while the experiences are still fresh in my mind and I have some breathing space I want to go over the things I’ve learned and discovered in my journey from programmer training wheels to today, where I would describe myself as whatever the programming equivalent of puberty is.
Have an opinion
After my first year in the team, I remember asking one of our senior engineers for technical feedback. I asked him what I could do to improve. He told me a bunch of stuff but the one that stuck was, ”have an opinion”. Over the years I’ve thought about this a lot and I’ve come to the conclusion that having an opinion on a technical decision forces you to be able to justify that decision and to justify it you need a deeper understanding of that technology or choice.
I’ve always described SWEs as 15-year-old girls fighting over their favourite boy bands, jumping from one band to the next when something new and hot comes along. Having an opinion grounded in sound technical understanding can help calm that mental storm of feeling like you always need to keep up because you can either disagree with something based on evidence or can find connections between things based on your deeper understanding.
I mean, he could have literally meant ”fuck everyone, just bully your opinion onto people” with no deeper meaning but this is how I’ve interpreted it. I guess you could say its just my opinion, innit.
Debugging is a science
Fast forward a few years and I was having some problems getting through tickets. Everything felt slow and laboured. I was struggling to get the last 10% of tickets working and over the line. I wasn’t really sure what to do and I guess others in the team picked up on it because our tech lead offered to take me for a coffee and some programmer therapy.
He asked me, ”whats your approach to writing software?”. At the time I remember thinking ”What do you mean bruv? The same as you obviously, I use a text editor.”. I was a little confused by the question because I had never really thought about it. After some discussion we highlighted that my approach to writing code was akin to an artist throwing paint at a blank canvas, I tend to write what feels good. I also came to the realisation I didn’t really know how to debug code efficiently. I was fine debugging my own code but I would struggle to debug systems and components I was unfamiliar with. At the time a lot of my work was combining systems/components rather than writing fresh code myself, I was slightly out of my depth.
Debugging is a separate skill from writing code. It requires a different way of working and without deliberate practice, like anything, you won’t improve. Our tech lead told me, ”Writing code is creative, its like art. Debugging is like science, you need to formulate a small hypothesis and test it, then keep doing that till you’ve fixed the problem”. That was incredibly eye-opening.
Since that conversation, I’ve come to the conclusion that there are three main pillars of being a good software engineer: writing code, debugging code and reviewing code. We spend a lot of time focusing on and actively trying to improve the code-writing part but not so much on the other two. While I’m not a pair programming fanboy there are definitely times where its well worth sitting down and watching someone more experienced than you solve a problem, then comparing it to how you would do it and using that data to figure out how to improve. I would say that pair debugging and pair code review are just as important. You can speed run your personal development just by seeing what kind of questions the other person asks while reviewing and how they break down their debugging into testable chunks.
Pragmatic programming
Unlike my actual politics, I would describe my philosophy to software changes as small C conservative. I feel there needs to be a strong reason to make a drastic change to working code. Fundamentally, I believe giving your customers high-quality, working code trumps everything else. Customers really don’t care about how your team is collectively jerking off to memory-safe, perfectly functional code with no side effects. The end user doesn’t care if your code is readable, DRY, maintainable or spaghetti code. They care that the product works as intended and does so reliably.
As SWEs, I think we sometimes forget that. I would describe the team I just left’s approach to software delivery as pragmatic, working code trumps perfect code. It doesn’t matter if your code isn’t abstracted into a perfectly generic DSL, does that refactor need to happen? One of the things about gaining experience is you learn when to break the rules. You learn when its fine to copy and paste working code instead of trying to abstract it away, which then not only risks delaying that new feature or proof of concept but also risks breaking systems that we already know works well.
I believe software gets complicated at the edges, at the points where systems interact with each other. Ensuring that the core of the system is as simple as possible to write, reason about and manage is the most important goal of any software project. I find saying no more often than saying yes makes reaching this target easier for everyone.
You’re still a code parent even after you’ve merged and deployed
Like everyone else, when the pandemic started we had a bunch of problems (other than the pandemic itself) transitioning from an in-office team to a remote team. During the first lockdown, we found ourselves in the office regularly doing maintenance and fixing fires. We’d always had monitoring but after this experience, we wanted to ramp up the observability of our systems.
To hark back to my earlier point about human problems, throwing new tools at the problem is the easy part, the cultural change required for the tools to be useful is what’s actually hard. So we’ve just added tracing, that’s cool, now what?
You need to make sure your SWEs understand that they still have responsibilities even after their PR has been merged. They need to feel a sense of ownership over the entire system and lifecycle of a program/feature/change. Jenkins is broken? CI failed? We need to tweak the resources used by a service in our k8s cluster? It shouldn’t matter if a SWE mainly works on the frontend or backend or infrastructure, everyone should be able to jump in and fix the issue. It shouldn’t be any single persons responsibility, it should be our responsibility.
Its this cultural change that necessitates the tooling. When SWEs realise its a pain to be consistently retroactive to fires instead of proactive to prevent the fire in the first place, they start thinking about the best ways to approach the problem. The solution and/or tooling is not important as long as it helps the team achieve their targets, its the collective desire of the team to ensure quality thats important.
Git — you’re doing it wrong (probably)
Although I said I’m a small C conservative, there are very few technology hills I’m willing to die on. After a convincing conversation, I’m happy to change my mind on most things, except git. Git is a hill I’m willing to die on.
Every software team in the world has its own culture and ways of doing things, which is all good but when I see teams using git as a glorified `ctrl + s` it really breaks my heart.
This should be its own blog post (one I’m planning on writing) but these are some of my general issues with how I see teams use git. I apologise in advance for any hurt feelings but here goes…
If you squash on merge, you’re doing it wrong. If you’ve never used git blame, you’re doing it wrong. If your repo’s history is useless, you’re doing it wrong. If you didn’t add a typo fix or white line change as a separate commit, you’re doing it wrong. If you’re commit message (when raising a PR) only consists of the words ”fixup” or a ticket number or a PR link, you’re doing it wrong. If you have a policy of one commit per PR, you’re doing it wrong. If you’re scared of a merge conflict, you’re doing it wrong.
In the time I’ve been working I’ve seen Jira projects change names, I’ve seen repos move from stash (bitbucket predecessor) to GitHub.com and then from GitHub.com to GitHub Enterprise. The only constant has been the `main` branch. As a tech company, as long as you’re in business the `main` branch will exist. Knowing that, it should be the single source of truth for your code base.
In my eyes, `main` has two main functions: firstly, it captures a moment in time. It shouldn’t redirect you to a ticket or PR, there is no guarantee either of those will exist. It should be self-encompassing. You shouldn’t try preserve history because I dont care about the journey or what discussions were had, all I care about is the final copy that got merged into `main`.
Secondly, its a forensic tool. You whip it out when something goes wrong and you need to go into NCIS mode. You should be able to meaningfully bisect `main` (read: without getting rid of whole features), you should be able to use `git blame` and be able to check the history of a line to see its accompanying changes. I dont wanna check the history of some line of code and see a whole heap of changes that have absolutely nothing to do with it because someone decided to fix a typo as part of some other work.
Your history should be clean and easily readable. Each commit should contain one logical change and ideally pass CI. If you dont know what that means its something that comes with experience but in a nutshell if you change some bit of code adding/removing/editing docs, comments and tests relevant to that change should all be a part of that commit. Nothing else, no typo fixes, no refactors, renaming another function etc, all of that should be separate commits. Your commit message should have a short title explaining the change and the body should go into as much detail as necessary into why the change took place, I can see the what from looking at the diff.
Ok, I’ll stop there before someone spontaneously combusts in flames of rage.
Hardware still matters
We live in a world of abstractions. Whether its the cloud or the frameworks we use every day, we can sometimes forget that underneath it all everything is still running on physical computers somewhere.
Over the last five years, I’ve changed hard drives and SSDs; I added network cards to server chassis; I’ve diagnosed kernel panics on servers without BMC or IPMI interfaces by literally pushing in a trolley with a monitor and keyboard; I’ve installed device drivers for an array of weird devices; I’ve sat in hardware procure meetings; I’ve configured network switches; I’ve written kickstart files and configured `PXE` boots using MAC addresses; I’ve plugged ethernet cables in and out more times than I can count.
I may never again work in a team where I have a mental map of all the hardware we own or a team where `ssh` + `dmseg/journalctl` is the starting point of my debugging but the education I’ve had working on those systems will last me a lifetime. I’ve had a crash course in electrical engineering and computer science that I could have never gotten from any university in the world. I firmly believe its easier to go up the stack than it is to go down.
In the outside world, there is a generation of people who only think in EC2s and AWS lambda’s and I dont blame them for that, its the world we live in. I just don’t believe that `ssh`-ing into a machine and digging around the file system using `vi` will ever not be a useful skill to have.
Being dumb is fun
As I alluded to at the start of this post I’ve been playing both team and individual sports my whole life. One of the unique properties of sport is that you are viscerally aware of where you sit on the sucks -> excellent spectrum. And if you are not sure someone will definitely let you know, whether its by running rings around you or literally shouting, “Oi you’re fucking shit, go home” across the field (true story, it’s happened to me). Inevitably, even if you’re a candidate with a realistic chance of going pro, at some point you’re going to suck.
There are two ways you can deal with this: 1) actually go home and either leave the team or worse, leave the sport altogether, 2) suck it up and train harder so you can be the one telling people they’re shit. I’ve found that everyone who plays sports long enough is pretty competitive and doesn’t take criticism to heart in a way that people who haven’t sometimes do.
What’s this got to do with software engineering? I feel like there is a weird parallel, where there is a certain class of engineer, usually good at what they do and/or in positions of power, who are more than happy to tell you you’re shyte. They’re the type who don’t talk to you, they talk at you. The type who dont ever ask you questions in a conversation outside of “What, you really dont know what X is??”. The type who don’t explain things in an audience-appropriate manner, you either get it on their level or you dont.
As a normie, when I first met these types of engineers I didn’t know how to talk or deal with them. I’ve had engineers publically ask me, “Did they not ask you what your favourite Linux distro is before they offered you the job?” in a condescending tone when they discovered I didn’t know much about Linux. I’ve had engineers give me 101 comments on a PR starting with “What the fuck is this…” and ending with me coming in one morning to find the whole thing rewritten.
Over time I learned that while there are a bunch of them blowing hot smoke out their arses, there are also some who actually know what they’re talking about. The competitor in me always wants to improve so occasionally stroking their egos and making them feel like the smartest person in the room can lead to a lot of valuable and obscure bits of information that might have taken you months or years to discover by yourself. And in all honesty, it’s not that hard to do. It usually just involves you asking them a question and shutting the fuck up because as I said, they dont need more than themselves to carry a conversation.
I’ve also learnt that in any given team there are a lot more me’s than there are them’s. People who are confused but are too intimidated to ask questions or dont want to look stupid. As a result, I always pitch myself as the dumbest person in the room and ask all the stupid questions I can think of. Its beneficial for not only me but for others in the team too. Not everyone is comfortable playing this role or looking stupid in front of their peers but growing up being berated on a football pitch by your mate’s dad gives you some pretty thick skin.
Fin
So that’s mostly what I can think of for now. It’ll be fun to come back in 7 yrs’ time to see what changed and what stayed the same in terms of my perspective on software engineering (if I’m still doing it), until then and in the wise words of Bill & Ted, “Be excellent to each other!”.
