The most talented data engineer I know, and my first manager, is a man that loves cooking. I mean, you might think you love cooking, but this guy loves cooking. The most excited text message I received this year was from him, proudly sharing the goddamn analytics on the heat distribution of chicken in his oven with his newly imported thermometer. We're talking about a culinary madman, capable of anything. The first book he gave me was not remotely related to any technical stack we ran, but was instead a 2,438 page manifesto on cooking. My current relationship is built on the back of a meal he advised me to prepare on the third date.
Every day, we would step out of the office to get some coffee, and on at least three separate occasions we ended up talking about the difficulty of running a top-tier kitchen. And, most shockingly, on all of those occasions, he professed an admiration for the complexity required for McDonald's to run.
At first, this startled me - McDonald's! I enjoy myself a mediocre burger as much as the next guy fresh out of university, but the man that graphs the cooking of his chicken is impressed by the place that hungover college students go for their nutrition? Well, it turns out that even if you don't hold the quality of the output in particularly high regard, the optimizations and discipline required to ensure that two different acne-ridden teens with wildly varied education levels, separated by oceans, are capable of producing the same burger are non-trivial to say the least.
The man in question catches up on this blog every few weeks, and will be horrified to discover that when you tell a young, impressionable engineer anything, they're going to think about it for three years and then write their own manifesto.
A Brief Glimpse Into The C-Suite Mindset
Many years ago, trying to figure out what the goddamn hell is going on in the tech industry, I ended up reading The Phoenix Project. This isn't the main point of bringing the book up, but let's take a brief detour to appreciate how bizarre some parts of it are. Here's the blurb:
Bill is an IT manager at Parts Unlimited. It's Tuesday morning and on his drive into the office, Bill gets a call from the CEO.
The company's new IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over budget and very late. The CEO wants Bill to report directly to him and fix the mess in ninety days or else Bill's entire department will be outsourced.
With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.
That's right, there's a whole genre of corporate fanfiction out there. Was it useful to read? Yes. Does it miss some of the real barriers to organizations improving? Yes, which I should talk about in another article. Was it cringe-inducing at points? Hell yes.
“I’d use that information to drive our production schedule, so we can manage our supply and demand curves. We’d keep the right products on the right store shelves and keep them stocked. Our revenue per customer would go through the roof. Our average order sizes would go up. We’d finally increase our market share and start beating the competition again.”
As she’s telling us this, she looks animated and excited.
Oh baby, average order size would go up? I couldn't be more animated and excited. It should not surprise anyone that it required a CTO to unironically write the prose above. It's almost surprising that the next page doesn't talk about how the newfound synergies between the infrastructure team and the developers have made the protagonist positively turgid. But I digress.
The general thrust of the book describes how I.T operations work a lot like a factory floor, and that this has implications for how the flow of work through the organization should be thought about and managed. In fact, the entire book consists of a strange man appearing periodically and giving the protagonist cryptic hints on how to be DevOps-pilled, like if Gandalf knew the only way to defeat Sauron was to help Bilbo fulfill all of his organizational commitments in the third quarter.
Some interesting ideas there, but then I realized that there are a few books that express this kind of sentiment. Quite a few. For one thing, the author has another very popular book out called The Unicorn Project. Then there's Investments Unlimited. And they're all inspired by another book called The Goal, which is almost the same plot but instead involves a character changing the operations on an actual factory floor, with slightly less cringe-worthy dialogue.
Written in a fast-paced thriller style, The Goal is the gripping novel which is transforming management thinking throughout the Western world.
Ooh, a fast-paced story about improving factory throughput? Fuck yes, I'm getting shivers just thinking about it. These all sound like novels that aliens wrote because they've only been able to decrypt the traffic from LinkedIn.
And then I read High Output Management which really was good without having as many caveats around the tepid style. It's straightforward and gives good advice for the effective operation of the soulless corporate machine, and it does not aspire to be anything else. And even that starts with an example of how works flows through a restaurant, complete with a description of how putting eggs on to boil at the wrong time is going to result in cold toast by the time it reaches the customer. A very large component to all this thinking is about economies of scale and the flow of work - these themes come up over and over again.
McData
Many months later, and I was walking around the floor of a vendor conference and something began to strike me in the offerings. On this particular day, I happened to be wearing a white shirt with a nice coat, and I suspect many of the salespeople mistook me for management, because they started to pitch me aggressively on their products.
As a rough rule, I knew the products were not particularly good. I've been burned too many times by seeing organizations purchase shitty enterprise software which promises to fix all of the organization's woes. Just buy Salesforce, or Workday, or Informatica, or whatever! Your staff said $CURRENT_SYSTEM is bad, so buy $NEW_SYSTEM. I'd heard it all before, usually in the context of deprecating $NEW_SYSTEM for $NEW_NEW_SYSTEM because $NEW_SYSTEM fucking suuuuucks. I currently believe that there's only one way to buy yourself out of technical issues and bad data models, and that's buying a really talented engineer whose sole focus is fixing the problem.
But what interested me is that they did not spend any time selling me on the technical details of the product. The pitches were entirely focused on something I wouldn't have noticed prior to reading all those books - they were focused on the reproducible delivery of work that was good enough to accomplish business objectives, without having to rely on all those pesky technicians to be good enough to do the job correctly.
For example, one pitch in particular was for a product which promised to remove the need for me to write SQL in exchange for being able to set up all my dependencies from a drag-and-drop editor, with the sales pitch consisting of "You can get rid of thousands of lines of all that SQL you hate!" - no I can't, fucko, because your application is still connecting to Postgres so it's just writing the SQL for me with another layer of licensed abstraction on top of it. Why would I pay to have more abstractions designed for you to sell software to multiple clients, you blue-suited dementor? Eight times out of ten, I want to pay you to remove them from my codebase.
Well, you and I understand that. But I suspect that what an almost savvy executive hears is a little bit closer to this:
You know that slow, painful SQL that you keep hearing is causing problems? We've got a drag-and-drop interface, so that you can now hire replaceable cogs to do the dragging-and-dropping at a consistent rate. And there's a consistent interface, and a bunch of specialists in this tool on the market, so you can easily ensure the smooth, consistent delivery of data through the organization! Consistent!
Or in another context:
If you buy a BurgerMaster 5000, you can just drop any teenager behind this thing with thirty minutes of training, and they're going to be able to turn out a burger that's good enough to sell! It's going to be the same burger every time, wherever you ship it!
They might not be thinking in exactly these terms, because I genuinely have good reason to believe that the median executive has trouble reading anything that isn't printed in a font so large that it feels like the writer is screaming affirmations at them, but I think this is the gist of the sales pitch. It is the promise that you can do away with the pesky implementation details of hiring correctly, or addressing all the bad decisions your predecessors made, or even telling your stakeholders that you've over-promised. Nah, you just buy the increased throughput widget, and now all the story points will be done on time.
Heck, even the way Agile is typically implemented at your average (read: dysfunctional and ineffective) workplace reflects this mindset. You count up how many units (story points) of parts (deliverables) that your machines (engineers) can produce, and then you let 'em rip, only to return next week and wonder why the machines only ever hit 50% of their target.
It's Fucking Raw
And here's where it all falls apart. All of that sounds great - economies of scale, reproducibility, robustness, blah, blah, blah. McDonald's almost never fails to produce fries of approximately the correct quality, within five minutes, every single time I've asked for them, and some of the absolute sons of bitches that I've worked for have clearly never shipped something that delivered value in their whole careers. They just buy a license for bad software, say that they've successfully implemented it since no one can really check, then leave before the ambient hatred radiating off the people forced to work with the system reaches a level that they can't tolerate.
Let me repeat that again more bluntly - we haven't figured out how to commodify a lot of this work. Despite my adoration and vested financial interested in computing being something companies need to pay me for, it would actually be really nice if we could do this successfully. I would adore it if the doctors and nurses in my life didn't constantly lament the stream of indignities that their single-neuron administrators heap upon them with each new proprietary system, even if it forced me to find a new career. Some of these things are just important, but we can't do it yet. There might be the appearance that many of these things can be neatly commodified, but the majority of these seem to simply be companies that shouldn't exist selling a product that doesn't work to other companies that have oblivious decision-makers.
(And sometimes, an evil decision-maker that has realized you can just man the phone lines from the third world for pennies because it technically means you're legally compliant, and you don't care about customers that are cancelling anyway. Whenever I need to talk to support staff, I try call the sales line first and see if they'll fast-track me, because I will get a local with the authority to do things in 30 seconds.)
Rich Hickey has a delightful talk titled Hammock Driven Development, which I highly recommend. On the topic of innovation, Rich Hickey is the creator (or at least, the most prominent creator) of Clojure, so he probably has more innovation points than the lower half of the world's managers combined. My man is out here advocating of a workflow that consists of feeding your subconscious mind research for four hours, then meditating on it for another two, then sleeping and praying that the Gods of Design simply bless you with an answer in the morning - a strange workflow that I experienced when I was writing my Master's thesis, where I solved problems that feel well beyond my ability now, or even in old stories I've heard about the legendary Poincaré.
In fact, all programmers seem to have experienced the strange phenomena where they walk away and the answer comes to them.
There's plenty of work that consists of simply churning out widgets faster, and I'm happy to see that work disappear (so long as we find a way for people to continue living healthily without it), but it must be acknowledged that many of the things we value in society come from an ill-defined, more vital place, and there is an intersection of that spark with the realities of production.
You can pay people to churn out bad self-help all day, but none of those are going to be worth a damn without allowing the human element to flourish. But you still need a factory which can print the things, and as clinical as a mass bookbinding operations sounds, I really believe that you're only going to get a beautiful binding when that factory is run by people who have the connection to the work necessary to exercise taste.
Maybe you can refine the delivery of a beautiful meal at a restaurant, but the meal itself is born from a person's creativity and aesthetic sense, trying to weigh itself against the realities of financial constraints. You can streamline it, and you can deliver the same meal every day, but I've seen the meticulous care of a professional chef running a restaurant in my home country, and successful commodification still requires intense caring.
You can try all you want to reduce the delivery of features to a Jira board and mumbling about Agile, but you're not going to get anything worthwhile out of people if you can't connect with them. You think an engineer good enough to have opinions on the relative elegance of cryptic symbols in an IDE can work without an element of soul, because it would be convenient if he was more like a widget-producing machine? Hah.
Heck, I know someone that designs some of the chips that go into iPhones, and you'd think that would be commodified to hell and back, but at the end of the day it's actually my uncle staying up to 3 AM, calling me from the U.S because he'd like to keep in touch despite the workload - and if that guy gets sick, they're probably actually going to have to delay the next release. There's crazy infrastructure to allow them to test and mass produce those chips, but individual people make that machine run.
There are harsh realities to grapple with, and society runs on commodification, but anyone that thinks that you can run on pure commodification, without any understanding of their specific craft, or the human complexities, needs, and frailties of the people around them, who think that you can just buy more enterprise licenses and that giving someone a salary is enough reason for them to subjugate the entirety of themselves as they turn up to work every day...
Well, you're wrong, and you can fucking bite me.