I did not understand tradeoffs until the database started dying

My first real job was writing C++ that read call detail records and wrote them into an Oracle database. I had never seen either one before I was handed the task.

A call detail record is what a telephone exchange produces every time something happens on the network: who called whom, when it started, how long it lasted, which number gets billed, which routes the call took through the exchange, whether it connected at all. One line per event. A telco produces them by the million, continuously, and somebody has to turn that pile into rows that a billing system can charge against. That somebody was me, and I was very sure I knew what I was doing.

The first half-successful run took the machine down with it. Pentium II, 128 MB of RAM, and a program that had cheerfully read the whole input before deciding what to do with any of it. There was nothing subtle about the failure. I had written the shape of the code I could picture in my head, which was the file arriving as one thing, and the machine informed me that the file was not one thing.

That was the closest I had ever come to thinking about streaming, years before I heard the word used that way. I knew there were better memory practices. I went and asked people how this is actually handled, and the answers were correct and did not help much, because the gap was not knowledge, it was that I could not yet hold the ambiguity: at what point does a thing become too big to hold, and how do you know before it happens rather than after.

Here is what I find interesting about that memory now. Not once did I ask whether the program should have been written in a different language. The other language available to me at the time was Java, I had no idea what I would even do with it, so I learned it. That was the whole reasoning. I was making decisions with real consequences and I had no framework at all for what I was trading away, because I did not yet know that I was trading anything.

Why put logic in the code when the database has triggers

Not long after, I was deploying an Oracle installation across a government network. Three hundred plus nodes. And I was happily pushing more and more business logic into PL/SQL, because why would you write it in the application when the database can do it for you. Triggers fire whether or not the caller remembered to call anything. Constraints hold no matter which client connects. It is genuinely closer to the data, it is genuinely faster, and at the time it was genuinely the right call for how quickly we needed to move.

The invoice for that arrives later, and it arrives in pieces small enough that no single one of them makes you reconsider. Logic in triggers is logic that does not appear in any call stack. It does not show up in code review, it is hard to test in isolation, it deploys by a different mechanism than everything around it, and it is invisible to the person debugging the application six years later who is looking for the code that changed that column and cannot find it, because there is no such code.

None of that felt like a cost. It felt like leverage. That is the part worth naming: bad tradeoffs almost never feel like tradeoffs. They feel like getting something for free.

Dot, tab, enter

Then I learned C# and .NET, quickly, because that was what paid for lunch, and I was writing ASP.NET MVC applications and being called senior by people who had watched me do it. The thing that impressed me most in that whole stack was the ORM.

I want to be honest about how good it felt, because the rest of this post does not land otherwise. You describe your objects. You get a database. You navigate a relationship by typing a dot, and the data is there. Dot, tab, enter, dot, tab, enter, done, ship it. The distance between the thought and the working feature collapsed to almost nothing, and everybody around me got faster at the same time.

Price? What price. I did not ask what SQL any of it produced. It did not occur to me that there was a question there. The abstraction was good enough that the layer underneath stopped existing for me, and for a few years nothing at all punished me for that.

Vertical scaling has a ceiling and nobody tells you where it is

It took me the better part of twenty years to see the full bill, and the reason it takes that long is that all three of those decisions only become expensive when something succeeds.

A monolith with a monolithic data model is the fastest thing in the world to build. Every table can join to every other table, so every new feature is cheap. Every report is a query away. The ORM makes each of those queries appear without you writing it. The database holds a chunk of the business rules, so the rules are always enforced. This is a machine for converting developer time into shipped features at an exceptional rate, and while the load is modest it has essentially no downside.

Then the load stops being modest. And you discover what vertical scaling actually means, which is not “we buy a bigger machine” but “we buy a bigger machine, and there is a last machine”. You can climb that ladder for years. It works every time, right up until the rung you are standing on is the top one, and the thing about the top rung is that you usually find it during an incident rather than during planning.

What arrives before the ceiling is worse than the ceiling. You get a database server that starts dying and nobody can say what is killing it. Not because the team is weak, but because there are hundreds of code paths that produce SQL nobody wrote, plus logic living in the database itself, all landing on one shared data model where every workload can touch every table. The expensive query is in there somewhere. So is the one that got slightly worse after a schema change last month. So is the report somebody scheduled hourly. Load is aggregate, and an aggregate has no author. You cannot page the person responsible, because responsibility was distributed across a decade of individually reasonable decisions.

And when you decide the answer is to split the data model, you find out that splitting a data model is far harder than it looked. Every join you casually wrote across two areas of the business is now a boundary you have to design, and each one has to be dismantled in production, under load, without stopping the delivery of everything else, while the business keeps growing, which is the very thing making it urgent. You are racing your own success. Modularisation under that pressure is a completely different exercise from modularisation on a whiteboard.

I would not trade that experience for anything, and I mean that plainly. Reading about coupling teaches you the word. Watching a business you care about get throttled by decisions you personally made teaches you what the word costs. That is where tradeoffs stopped being a topic for me and became a reflex.

Design for 1, then 1,000, then 100,000

The old advice about designing for one user, then a thousand, then a hundred thousand, then whatever comes next, is not there because someone liked round numbers. It exists because each of those steps is a different architecture, and because the only reliable way to know which one you need is to actually be at that number.

The failure mode people talk about is building for a hundred thousand while you have eleven. That is real, and it is expensive, and it kills companies that die of complexity before they die of load.

The failure mode I lived is the mirror image, and I think it is the more common one: the shape you built for the first thousand quietly becomes permanent, because nothing forces a rewrite while the numbers are still going up and to the right. There is never a convenient quarter to split a data model. So the design that was correct at one scale gets carried, unexamined, into a scale where it is wrong, and by the time it is obviously wrong it is also load-bearing.

Designing for the step you are on is not about restraint for its own sake. It is about being able to see the step you are on, which means knowing what you traded to get here and what that will cost at the next number.

Go as minimal as you can

If I compress all of it into one sentence, that is the sentence.

Not minimal as in primitive. Minimal as in: whatever you can leave out, leave out. Fewer components, fewer clever layers, fewer places where behaviour can hide. It is the same argument every time, and it is not really an aesthetic argument, it is an operational one:

Less to remember. Less to fail. Less to monitor. Less to be woken up about at two in the morning, and less to reconstruct at that hour while trying to work out which of the fourteen things in the path is the one that broke.

Every abstraction that saves you time now is a thing you will have to understand later, under worse conditions, with less patience, usually in front of an audience. Sometimes that trade is clearly worth it. An ORM in a small service with a load you can predict is fine. A trigger enforcing an invariant that must never be violated is fine. A monolith at the start is not just fine, it is usually correct. The point is not that any of these are wrong. The point is to know you made the trade, write down what you expect it to cost, and know which signal tells you the bill is coming due.

The version of me on the Pentium II did not know he was trading anything. That is the actual failure, and it is not a failure of skill. He shipped the thing. He just could not have told you what it would cost, because he did not know there was a question to ask.

The question is always: what am I giving up, and when does it come back. Ask it early enough and you get to choose. Ask it late enough and the database asks it for you, at two in the morning, in production.