Post-mortem

After going viral, there is one comment I'd like to address. One guy pointed out how comfortable this work seemed, how high the pay was, and said I am out-of-touch and privileged. To that I say, yes, obviously. I moved to the first-world to have first-world problems. Mission accomplished.

There is a flavor of despair I've become accustomed to, so deeply ingrained in the hearts of myself and my colleagues that it has settled into a hopeless passivity. It's the despair that comes from knowing that we spend most of our time producing nothing of value.

It is something we admit to quietly, sometimes laughingly, because we know it could be much worse. For the most part, we all work in the data space at large organisations, either in the public sector or adjacent to it. It is an incredibly comfortable living. The pressure is non-existent, we clock out at 5PM, and there are certainly fates far worse than that. But still, we despair.

I remember my first job, fresh out of university, where I was paid just enough to say that I was earning six figures, which is a heady experience for a young man from a developing country. At the time, I felt like quite the hotshot, and had a title to back that up - 'data scientist'.

What did such a position entail? I spent about two years in that role and produced absolutely nothing of value to society, and more strangely, nothing of value to the organisation I was in. I largely produced PowerBI dashboards from awful data sources that I knew no one was looking at - the same people who sent the emails tagged 'URGENT!' and demanded a product ASAP were seemingly unaware that we could view whether anyone actually looked at the work we produced. Sometimes we would have huge errors in the work due to the horrific undocumented database we drew from, and no one would ever notice.

But I wasn't the only person on that team - we had three staff, so we were spending something in the range of $400,000 in payroll and producing no revenue. We frequently went through a rollercoaster of emotions, where we'd strategise and think about how to finally do something interesting and worthwhile, get rebuffed because people actually just wanted their spreadsheets delivered faster, and... despair.

This began to wear on me eventually, but I put it down to the first organisation being unusually bad. Eventually, I found a job that should have come with considerably more autonomy at a non-profit. And it was exactly the same. I found myself in a dreary gray office, where many of the staff rather transparently had an intense dislike for the people they were supposed to be helping. The institution produced a fair amount of academic research, but the vast majority of struck me as borderline fraudulent. We had some contracts with industry to produce technology in the space, but these mostly seemed to consist of the organisation fumbling the projects and massaging the truth (read: lying) until they could collect more funding.

I had one more attempt at doing something where I'd try to produce something - anything - working in a unit that assisted emergency services. This time I was a lot more optimistic. The people were extremely intelligent and clearly cared about the mission... but I think anyone that's worked for anything government-adjacent will understand that we never delivered anything. I joined towards the tail-end of a failed cloud migration, was immediately pulled off a project that might do something meaningful to connect spreadsheets to databases, and didn't last long.

At this stage I was three-for-three on jobs that did absolutely nothing, and more importantly, I had formed some deep relationships. The funny thing about everyone secretly despairing is that they're usually not too far from forming deep connections the moment you let your own mask slip. The absolutely fucked up thing is that everyone I've met in this space seems to have totally given up on doing anything meaningful at work. The goal is to get paid, not stress out, have a happy office where everyone can collect their strange handout, and not think too deeply about how unfulfilling is it to produce nothing for forty hours a week. As far as lives go, in some ways it's more admirable than the blatant veneration of money. At least human happiness is a major concern.

I've heard from more than one person, and heard myself saying, that the first thing I would do if I was in charge would be to fire myself, as the immediate savings would be the greatest outcome anyone had ever produced in the team.

Still, I was perplexed. Of course, I was familiar with cynicism around work. It's hardly a novel concept, it permeates the culture, and it was further popularized by the late David Graeber's writing on Bullshit Jobs. Also, as a rough rule, cynicism makes you seem clever (and to spare other young people learning this the hard way, it makes you obnoxious to be around when expressed too much), so it was certainly something I had started to affect, but never internalized. How on earth could we have what seemed to be an entire industry of people who all knew their jobs were pointless?

Talking about data is signalling

What I hadn't really grasped was the degree to which some organisations have grown so absurdly fat that they could afford to dispose of millions of dollars on employees who did nothing other than reaffirm a vague commitment to being data driven. The same people that love technology affirming language seem to largely be the same people who will insist that everything be delivered in a spreadsheet format. They also can't code - the degree to which my teams sucked was basically directly correlated to how good my manager's people skills were and whether they had programming experience. In any case, it's fine to talk about modernization but also demand everything be done the lazy way, because talking about the love of technology signals that everyone involved is modern and willing to endorse any viewpoint beneficial to the organisation, but getting the spreadsheet right now means that you can look good for quick delivery. Why not have your cake and eat it too?

Terrible data makes doing anything good impossible

Most organisations seem to have created their data stored as compacted Hell-bricks jammed into a teetering edifice which threatens to crush anyone who so much as breathes near it. It is frequently such a Herculean task to even put together a dataset that represents a clear view of what the organisation is doing that you just can't. That is, the thing you have been hired to do is not possible. But that would mean that you, your colleagues, and almost all those managers should be fired, so we just don't talk about it aloud.

Vision is hard

Most organisations don't have people with the right incentives and vision to push for real change, and only getting part of the formula right seems to lead to no outcome at all. I've seen a machine learning project take several years to come to fruition, but there was no plan to do anything with the model's output, so we have a model and no plan to intervene based on its output. But that doesn't stop management from discussing it like it's a huge success, as per the idea of data talk really being signalling (the secret is not measuring any outcomes).

Friction with internal policy

There are usually at least a few tasks required for the organisation to function, such as producing some sort of report for the government, that technically does need to happen, so we can't simply lay everyone off. Frequently it's a task that one competent developer could probably automate forever, were it not for that developer needing a small army to handle the logistics of navigating the bureaucracy of getting one change request passed. My first team asked for an EC2 instance with just enough compute to trigger a bash script using Airflow, and we were quoted $40,000. This is the some section of the organisation's way of saying "fuck off". Getting these requests pushed through requires enough staff that the person asking for them is considered important.

So one person could do the job, but the organisation forces us to hire more people (and listen, I'm sympathetic - the team telling us to fuck off were themselves in some sort of perpetual gridlock), so that the manager of those people is important enough to enable the one person to do the job. And, you know, it's government reporting. Even when I did do one of these necessary tasks, it wasn't exactly what I imagined doing with my finite time on this earth, at least when it was my only measurable output.

Free money attracts conmen

Piles of money + unclear outcomes = every grifter under the sun begins to migrate to your organisation. It is very hard to keep them all out, and they naturally begin to let other grifters in because they all run interference for each other. Sure, they might betray each other constantly, but they won't challenge the social fiction that some sort of meaningful work is happening.

Conclusions right now

I've punched this all out over one evening, and I'm still figuring things out myself, but here's what I've got so far.

I've realized that applying to jobs expecting this to get better is probably a terrible play. There are almost certainly organisations that don't run this way, but it's not like producing something that isn't literally worthless is sufficient to enjoy spending most of your life doing it. For some people, it's taking care of their families, or they're trapped in the job because they have debt. For others, they really have found something they deeply enjoy. But for many people still on their way to some level of actualization at work, we're going through the process of gradually shedding each layer of dissatisfaction, only to turn up new ones. There's presumably a layer when you're done with this, but I frankly don't feel like I have enough time in my life to hop through these jobs until I find the right fit when I see people in their 40s that haven't managed it yet.

I've spoken to consultants who feel successful, but long for meaning. And public sector workers who deeply believe in the mission, but long for actual success. And yes, a few people who are actually deeply happy with their work (but none of them are in I.T thus far).

The two people who were most happy? The first was a friend who had worked for the government, and gradually become incredibly disillusioned with their labour after producing nothing for years but continuing to get paid. They eventually quit their incredibly cushy job and sold books for a few months at a big retail chain - and they said it was incredibly therapeutic. They finally felt they were doing something worthwhile, as simple as it was. That said, they did also say the experience was great only while romanticizing it a bit, and that it probably would have felt very different if they felt trapped in it as a career for years. They're still figuring things out (they've moved from the "this is worthless" layer of the despair lasagna to the "this has worth, but not enough to spends years on" layer), but they're certainly doing better.

The second was my barber. He's an immigrant here, and started a very successful business. Unlike the young tech entrepreneurs I typically meet (who glorify money and the wealthy in a way I find grotesque), he's a very quiet man who has incredibly happy employees and remembers every customer by name. On an impulse, I asked him why he started his own business when he could easily have earned a good living just working for an existing company.

He said that he hated the feeling of churning work out on an assembly line ($25 per haircut, you'd better get four customers done an hour!), and wanted to replicate the experience he was used to in his home country. He doubled the price and the amount of time he could spend with each customer, and held to the conviction that people would be willing to pay for quality and time, even if it wasn't cheap or expedient. He has three branches up now, and his employees talk about him like he's a father to them. He has carved out a little part of the city where people are taken care of, feel safe, and deliver something they care about.

Right now, that's what feels right to me - those of us who are despairing, we're chasing quality and meaning, and we can't do it while we're taking orders from people with the wrong vision, the wrong incentives, at dysfunctional organisations, and with data that makes our tasks fundamentally impossible in the first place. Quality takes time, and right now, it definitely feels like there isn't much of a place for that in the workplace.