There are a few things that inspire deep revulsion in me. The episode of Star Trek: Voyager I just watched was pretty gross. I definitely saw some pretty bad roadkill on my daily walks during the Covid lockdowns. But nothing - nothing - comes close to the white hot fury that radiates from me the moment someone suggests plugging a spreadsheet into some sort of automated process.
A sane engineer would come to you, palms upturned, arms wide, and preach the virtues of Excel in a conciliatory tone. "It really empowers non-technical staff", they would say. "The democratization of data is an important element to the modern workplace, and it is an incredible testament to Excel's flexibility that it has enabled so many people to leverage the power of computing."
But that is a mere sane engineer, the first to die in the revolution, and when you have seen enough of what happens when you yield to this blatant cowardice, you move beyond mere sanity, into righteous logic so cold that it burns like fire. I tell you now, if you suggest that we put that spreadsheet anywhere near any process, no force on this planet will save you from me.
But Why Not?
Listen.
I'm sure it seems like a good idea right now. I'm sure it does. It always does. "They're already used to filling out the spreadsheets, and we can make sure of the information without changing any processes." I am deeply sympathetic, and will feel bad as I haymaker you through ten cycles of reincarnation. I don't want to do this, but you've forced my hand.
I've seen a multi-million dollar analytics platform that is dynamically constructed from Excel spreadsheets. Do you think that was the intention when the damn thing was being assembled? Do you think any sane person would ever want to do that? No, of course not. One script got written that could parse a spreadsheet, and next thing you know, you're running a greedy search algorithm every sprint, and it never quite makes sense to spend the time removing them. Now you've got fucking twenty, and I am hunting you on a deserted island. Was it worth it? Was it worth it, you absolute son of a bitch?
You don't have to live like this. I've gotten through my whole career without understanding how to use VLOOKUP. It's possible, I swear. Just put the spreadsheet down and we can work with databases. Do you remember databases, and possibly happiness?
But I Wrote This Great VBA Macro That -
Fuck you. Well, actually it's great that you've shown that initiative and learned that skill, and it shows that you have the ability to do this in a better way. I will support anything you decide to do as long as it isn't in that spreadsheet. I'll do anything. Anything. And if you don't take me up on that offer, then I'll see you hang.
Do you know how absolutely out of control VBA can get? I worked at a non-profit once that produced reports for the federal government by processing millions of medical records in one gigantic Access script. It took two senior engineers months to even partially unravel the mess, and it was well-written. Do you know how it happened? They got one thing done, and then they executed a greedy search algorithm and kept adding one more thing to the script. But it stops here, you've left me no choice and if I see a single line of VBA, you and I will forever be at war.
What About SharePoint -
Don't you even goddamn talk to me about SharePoint lists. Don't you dare. The goddamn access key juggling when you want API access, the endless discussions with ITS because no one knows how the damn thing even works. Did you know that the names in the API response can't be changed after you've made the sheet, even if on the webpage you've renamed them? I will kill ten sharks with my bare hands if it means that I can draw even a drop of your blood. If you make me manually map unnamed_column_1 to 50 to fields in a database again, then you have initiated violence against me, and the horror that ensues will be self-defense.
Do you know why all these processes are run through SharePoint sites? Someone stored one thing, greedy search algorithm, you get it.
But It's Really Only Temporary
How fucking dumb do you think I am? It's always temporary. It's always just this one time.
Do you know how many times I've written a script that was really only run once? Never. It has never happened across my entire career. Every single thing I've ever written has been fucking welded into the soul of every organisation I've ever worked at. They'll never get rid of any of it. My sins will echo for eternity, and I will be damned if I let anyone else get away with what I have. I will send you straight to hell and meet you there.
Do you really, sincerely, think that you're only going to use it just this once? Well, I'm sorry, all the goodwill in the universe around this particular topic has been used up, and instead of taking it out on me, I will hand you a longsword and we can hunt down those responsible together.
Even if you think that you're going to replace this before starting any new work, you know who doesn't care? The stakeholder that's going to, as if psychic, flood our inbox with tickets tagged URGENT every time we talk about decommissioning the spreadsheets. Microsoft pays them to keep Excel in circulation. They've struck a bargain with the Devil and live an extra hour every time a person highlights cells to look for errors. Okay, fine, I don't know why they want to stop us, but they do.
Then we're going to look at our tasks, see the stakeholder screaming at us, and we're going to execute a greedy search algorithm, keeping the spreadsheet around forever. It's like goddamn gravity.
I Don't Have Any Other Tools
Deeply sympathetic. Still have to kill you. I'm so sorry. I really am. I don't do this out of hatred, but out of duty. Someone has to stop you, and I'm sorry that it's me.
I once worked with a database, where the entire daily data extraction process for business consumption was a batch script that fired off a spreadsheet, that spreadsheet contained a list of other spreadsheets and some VBA, which looped through the list of other spreadsheets, copied them to new folders to be used by staff, and those copied files then executed SQL against a database to have yesterday's data in them.
Think about how clever someone would have to be to construct a process when given only those tools. Think about how someone that clever could never have meant for it to stay this way - they had the best of intentions. Such a person is far smarter than I, and it is with deep regret that I wring the life from their mortal coil. In the words of Rich Burlew, "Intentions matter only in absence of results. And then, only for the blaming."
All My Colleagues Use Spreadsheets
I will come for them too. I have a friend in Southeast Asia who has exactly one job - he is paid a grotesque amount of money to automate Google Sheets for an e-commerce company. Do you think that was the original plan? Everyone at the company is incapable of working with anything else, and you know what, that isn't an excuse because you can learn. And if you cannot learn from Google, then you will learn when I yell, when I point, when I beat you with my stick.
This Is Gatekeeping
Yes. I have put this gate here and I am keeping it for a reason. The last time I stopped keeping this gate, I wrote a Python script that took a spreadsheet as an input, ran regular expressions in it against a database, spat out ten more spreadsheets, then used those to power a generic PowerBI template which had to have all ten spreadsheet targets repointed by hand. This became one of the most requested services my team performed for several months.
Someone should have fucking stopped me, and I must pay for my crimes by keeping this gate forever. I swear to you, there is nothing good past this gate. Turn away. Turn away. I beg you, turn away. Beyond this lies naught but trying to work out why all the numbers are wrong, only to realize that Excel thought those IDs were integers and dropped all the leading zeroes. This death is a kindness.
This Is Unreasonable
Do I seem like a reasonable person?
What Should I Do Instead?
Anything. I am begging you on my hands and knees, anything. Write a SQLite database on your local hard drive. Do some garbage in Python. Encode the data in binary using a series of pebbles on your front lawn. If necessary, I will personally call your manager and explain the problem. I will actually do this. It's easy, I swear. They're all definitely easier than being defenestrated, which is the only alternative I am offering.
Please. Listen, you can email me. I'll help. I don't want to hurt anyone again. ludicity.hackernews@gmail.com. There.
You've got an offer of professional help. Do you know why I'm being so generous with my time?
It's so that I can fucking dropkick the animals that don't take it with a clear conscience.