A note from the future: I no longer believe the corporate facade is complicated. It is complicated in that there are many factors at play and many of the actors in the system believe in the beautiful vision, but the truth is that it's all an elaborate justification for getting money and status without producing value. If you're already what a friend of mine terms "post-cynical", feel free to skip this one.
This Monday, I wrote about most data work I've witnessed seems fundamentally worthless. A lot of my time over the past few months has been dedicated to figuring out what that perspective means for my career. It was also very quickly suggested that I need professional help, which was very funny. Anyway, given the amount of traction that post got, it's clear that it resonated, though I am not closed to the possibility that we all need therapy. It has made me like I'm what happens when you put the Mouth of Sauron in an office job.
Something that has always worked out for me is noticing when I'm confused and seeking clarity, and I have been utterly perplexed as to how some people seemed fine with the grind. I have seen staff go so far above and beyond, hammering out Python to crunch spreadsheets to drag-and-drop PowerBI visuals to tick off Jira tickets to please stakeholders who didn't read the damn things. And at the same time, I've seen plenty of managers who managed to turn up for their team every day, attending the endless, thankless meetings where their peers spouted off absolute bullshit.
It isn't always a mystery. Some of them just had kids, the pay is comfortable, and there are things in life that are more important than your personal happiness at all moments, so they make do. But for the others, what gives? Did they just vaguely believe that they were going to live forever and this was a good way to spend their blessed hours on this green earth?
So, instead of just guessing at what's really going on, I asked some of them to mentor me. These are my thoughts on a three hour conversation with the most insightful (or at least, most willing to share that insight with me) people manager I've met thus far.
Does our super senior leadership know that things are fucked up?
I've had the fortune (and sometimes, incredible misfortune) of being exposed to a fair number of people in the C-suite, and they're always a mystery to me. Here's a pro-tip for young programmers - most people in the business don't give two shits about code quality, developer practices, or any of that jazz unless you convince them that it is affecting something that makes them look bad. I mention this because, well, it's common for us to feel misaligned.
Executives and directors were this strange class of person who seemed to be constantly engaged with the most obnoxious part of work (arguing with people, being obviously lied to, and obviously lying), and did none of the parts that seemed particularly ensouled to me (programming, some forms of manual labour, teaching, whatever). And despite this, many of them turn up smiling-faced, day-after-day, and proceed as if their spirits aren't slowly being compacted.
A few possibilities occurred to me:
-
The flicker of humanity that drives the rest of us to despair has long since been extinguished in them, leaving dead-eyed automatons that exist only to repeat that the project is currently Red, but if we just demonstrate a level of competence that we clearly don't possess or we wouldn't be here in the first place, we could be back to Green soon
-
They realized that they need to turn up and play the game as aggressively as possible to ensure that their children had the best possible opportunities in life vis-à-vis whatever money can buy for them
-
They're absolute sociopaths, either from a young age, or after being exposed to sycophants for dozens of years
-
They're deeply incompetent, and the organisation is enough of a sham that somehow this is totally okay
-
Their direct reports have successfully woven a web of deceit around them so compelling that they actually think everything is okay
-
They're perfectly aware that everything is terrible and want to do better, but management at that level is engaged in some horrific game-theoretic dilemma where anyone that defects by not lying about their success is immediately eaten alive
The main one here that didn't feel right to me was the dead-eyed automaton theory. I don't think people can actually become that dead inside from mere office drudgery, but I didn't know for sure. The others were all certainly possible, but a person suppressing all traces of humanity on a deep level seemed a bridge too far. But then again, whenever I've heard a lawyer describing what it takes to be a partner at a major legal firm, I also think that humans can't possibly put those hours in, yet here we are.
And guess what! I finally got some clarity! Someone way up there in an organization I'm familiar with sat down and answered every question I had with brutal honesty. Here's everything that came up in the discussion over three hours, in an increasingly dark and spooky office.
Are they dead-eyed automata?
To my relief, I've so far failed to uncover anyone forced into this lamentable position thus far. Each person that I thought might be like this, at least where I've worked, has in fact been a human with real interests and dreams. That probably sounds like a stupid thing to confirm, but I have done it nonetheless.
Are they grifting because people rely on them?
We didn't touch on this one - but I know enough people who tolerate their horrific non-management jobs that a few of them must be in this position. Even if they're not terribly unhappy, it's obvious that economic realities can easily force people to do things that they'd otherwise prefer not to do.
Are they sickos?
We didn't cover this one either, but I can tell you that I've certainly met a few people who would likely qualify. I worked, very briefly, for a CEO that was born incredibly wealthy, and spent all his time playing his subordinates off against each other in sick games. He'd call my manager and just scream that sales had to be doubled by the end of the week... or else. I don't know what the or else actually would have been, but it certainly sounded concerning at the time.
I've also met people who will simply assert they are correct and then fucking bulldoze anyone that gets in their way mercilessly. I fortunately haven't worked with such a person, but I can confirm that I've met some that are very successful. In my experience, it isn't a strategy so much as a horrific personality defect - they are like this in all settings. They do this to their friends, family, waiters, leveraging overwhelming, reality-rewriting conviction that they are correct, you are wrong, and no, they never said the thing that they clearly said.
In my experience, they are usually terrible at their jobs in addition to obviously being incredibly abusive. Being good at their jobs would require learning, which would require changing what they're doing, which would mean admitting that they're wrong, and these people will sooner lay you into the barren earth than admit to being wrong, unless they're trying to self-signal that they must be good people because they admitted they were wrong that one time.
Unfortunately, this seems to largely work for them. No idea how. We'll figure that out later. My only advice is to get as far away as physically possible if you feel that you're near one of these people.
Are they deeply incompetent?
Yes, some people are. I don't know how they end up in such senior positions in the cases where they weren't just born into it or one of the people that simply bulldoze everyone. It's mind boggling. It's partially explained by things like the incentives in government, but sometimes they're so astoundingly bad that I still feel that I'm missing something. Mark me down as 'still kinda confused' on this one.
Do they live in a direct report constructed fantasy land?
Ah, now we get to the real spicy meatball.
Some of them certainly do - but the degree to which that is true varies intensely.
Take it on faith that no one has the time or expertise to personally investigate what's going on in the organisation at both enough breadth and depth to formulate a clear picture of what's happening themselves. I don't think that's a difficult proposition. For example, I can evaluate a codebase pretty well, but I must concede that I don't understand a thing about our finances beyond the crudest of details.
Secondly, I have been told by someone in the C-suite that your main job at a company is frequently managing what information reaches your manager. Probably obvious from observing behaviour, but helpful to have articulated. This is probably why the same people who reported having fulfilling data jobs were quite close to areas that make money - you can certainly fudge numbers a bit, but if you're operating at a huge loss then people are probably going to find out eventually, so the bullshit opportunities are substantially smaller. I assume. Whenever I've been near the money, it has been in a startup with a fundamentally unviable business model (i.e, earning money was never an option), so we're in the realm of pure theorycraft, which is always a dangerous thing.
So here we have an executive acknowledging that it's well-known that your main job in management is to package information into inoffensive (and, if you're doing poorly, as most places are, misleading) tidbits to go into a PowerPoint presentation somewhere upstream. My mistake here was thinking that this could only be done for the benefit of whoever is doing the lying. It's the hip and cynical take that makes you feel wise. What is actually happening is that this massaged version of reality can be of instrumental value further up the chain, and with the right people in charge, it can be part of the convoluted process of actually making things better.
Let's say you have some data project that was poorly scoped from the very beginning. I've been reading Machine Learning Engineering by Andriy Burkov (cool book, by the way!), and it opens with all the ways that you can technically create a functioning machine learning project that accomplishes no business objectives. So let's imagine we've done one of those.
Now if you aren't measuring outputs in any meaningful way, then you can report that this project's capabilities have been made available to thousands of staff, it's revolutionizing the industry, your partner thinks you've become more attractive upon deployment, and your first project that wasn't a dashboard no one reads is a resounding success.
Anyone smart will know this isn't true. All the engineers know this isn't true. We are incredibly demoralized as we watch the managers take a victory lap, pat themselves on the back, thank us for the terrible work we know we've done. What I was gratified to hear is that some executives actually know the whole enterprise is a total hack fraud, but reporting the success allows them to secure resourcing to do better next time, for everyone's sake.
That is, yes, we're constructing a fantasy land together, but it really can be together, not just one manager throwing everyone's dreams under the bus. We can't have meetings where we discuss this as an explicit strategy, but with a wink and a nudge, when you have the right people who actually care about each other, we're all grifting together for a better future. We have to abuse the terrible output tracking collectively to get the resources we need to actually create output.
Are they in a gigantic game theory problem?
We'll get back to the previous point in a second, because it's the main thing worth discussing. But yes, senior management is in a dilemma. If they're honest, someone else will eat their funding. You have two options: play the game or don't play the game (i.e, lose by default).
What does this mean?
Firstly, my immediate take is that it becomes very hard to evaluate what kind of senior leadership you're dealing with from the ground. The problem is that, since they are all playing the game and can't announce that they are doing so in front of everyone, they can only tell you how they really feel in private. If almost all the realities above result in leadership saying the same things, it's really hard to tell what's happening. It's like a murder suspect pleading Not Guilty. That's exactly what a murderer would say!
Secondly, it matters massively who you're working with, as always. If I'm your manager and you send me some nicely massaged information, it's basically up to me to decide what to do with it. There will be a layer of management that will receive this beautiful lie, and they will then have the option of betraying everything you hold dear, or they will try to protect everyone's jobs and create a happy organisation. In fact, I now realize that if everyone is aligned - that is, everyone is on the same side with the same goal and we know this is the case - then the fabric we're constructing is an exercise in cooperation, not malice. We are weaving a wondrous multi-hued fabric of a hypothetical better future, passed from hand to hand, until the CEO finally kills our last fucking spreadsheet. I think I understand now why people talk so much about the value of vision in leadership. It's obvious at places that produce effectively, and seems almost offensive at places that don't perform well, but it is a strategy at the latter place to become the former place. I need to know what helpful lies to weave.
However, I have no idea what type of person is generally attracted to the executive level, because see point one, so it's hard to stay motivated without that insider intel on how the managers are actually aligned. You're liable to spend years working on something like this, then someone up the chain turns the fabric into a cool suit, nails a job interview with it, and guess what! You just wasted years of your precious life so some millionaire can be more of a millionaire. Reading books like Robert Jackall's Moral Mazes (which I should finish, whereupon I will probably find out it has discussed all of this) leads me to believe that there is some intense selective pressure for pretty awful people in many industries.
Thirdly, you must play the game, or move into an industry (or startup, or launch your own thing) where you have real exposure to consequences and reality. Nassim Taleb has prescribed this as the antidote to bullshit in his book, Skin in the Game. If you try to avoid playing the game where too many actors don't have skin in the game, or if the consequences are too far removed from the present moment, avoiding the game means you're default dead.
Fourthly, some of the work I thought was worthless (admittedly, a lot of it was still worthless, and I feel bad for people doing Business As Usual tasks under such a regime) was actually helping good people fight the good fight, just in an ethically ambiguous way. It's unclear to me about whether I'm happy to trade years off a relatively short life to nudge this ship ever-so-slightly in the right direction. It's certainly less discouraging to think that my focus on good engineering is applying a teeny, tiny vector that moves the entire enterprise in a better direction, rather than being literally completely worthless in principle. The real issue is that, given the right colleagues, it's worth lots but isn't the right strategy.
Fifthly, man, I've thought about this a lot more than the average person (I think) and I still have no idea what the fuck is going on or what I should do next. Someone also just reminded me about how much time was wasted talking about Agile as a solution to our delivery problems (it was neither better nor worse), so there's probably some further questions about whether we've got people good enough to implement whatever the vision of the day might be. Sign me up for some of that therapy.