← All writing

AI Is the Ultimate Leaky Abstraction

A generated answer is an abstraction over reasoning you never see. It works until it leaks, and when it leaks it hands you a bill for exactly the understanding it let you skip.

An abstraction is a promise that you won’t have to look underneath. Generated answers make that promise too, right up until the moment they break it, and the moment they break it they hand you back a bill for exactly the understanding they let you skip.

I left a promissory note at the end of the last post. I’d spent the whole thing arguing that every abstraction is really a contract (here are the inputs, here’s the behavior you’ll get back) and that it loses nothing only when the part it hides produces no effect outside that contract, that almost nothing real clears that bar because the hidden part nearly always leaks something, some side effect, some more or less than the behavior you were promised, and then I pointed at the one abstraction eating the industry right now and said it deserved more than a paragraph. This is the paragraph, grown up. If you want the machinery underneath the claim, that post is where it lives; here I just want to follow it to the end of the line.

You don’t need to write code to know this feeling; you just need to have leaned on something that mostly works. Think about the auto mode on a camera. It’s a lovely abstraction over a genuinely hard problem, the three-way balance of aperture, shutter speed, and ISO that decides how a photograph comes out. Point it at a sunny street and it nails the shot every time, and you get to never learn what those three dials actually do. Then you try to catch your kid blowing out birthday candles in a dim room, or a friend backlit against a bright window, and the camera’s little exposure model guesses wrong the way it was always eventually going to. The candles smear into blur, or your friend goes to a black silhouette in front of a perfectly exposed window. Nothing is broken. The camera did exactly what it was built to do. But the only way to rescue the shot is to drop into manual and actually understand the exposure triangle the auto mode spent years letting you ignore, and if you never built that understanding, the moment is just gone. The abstraction was free right up until the instant it wasn’t, and the price it charged was the exact knowledge it had been saving you from. Same story in the driveway, by the way: the car starts every morning and you never think about the battery, right up until the cold snap when it doesn’t, and the thing that let you not think about it is suddenly the only thing worth knowing.

This is the whole pattern, and it’s everywhere once you can see it. Computing is just where somebody finally bothered to write it down. Start where the idea started. In 2002 Joel Spolsky wrote down a “law” every working engineer already knew in their bones but had never quite named. (It’s a “law” the way Murphy’s is, an observation that keeps being right, not the kind physics or a statute can hold you to.) He called it the Law of Leaky Abstractions, and it has one clause: all non-trivial abstractions, to some degree, leak. TCP promises you a reliable stream of bytes and hides the snarl of dropped packets and retransmissions and three-way handshakes underneath. The promise holds almost all the time. Then a backhoe somewhere severs a fiber, a router upstream starts flapping, and the reliable stream stops being reliable in a way no amount of staring at your own code will explain. You open a packet capture, watch the retransmissions stack up, and there you are, debugging a layer you were assured you’d never have to see. The abstraction saved you time while it worked. It never saved you the obligation to understand what it was standing on.

That last bit is the part people forget, and it’s the heart of Spolsky’s “law” rather than a footnote to it. An abstraction doesn’t erase the complexity beneath it. It hides the complexity and changes when you pay for it. While the illusion holds you’re fast, productive, blissfully ignorant of the planner choosing your query path or the allocator deciding where your objects live. The bill comes due at the leak, and the currency the leak demands is comprehension: a working mental model of the substrate, assembled under pressure, usually at the worst possible hour. (This is the same bill I called debt a couple of posts back. Same loan, different collector.) Spolsky’s quiet warning was that as our tools climbed higher, the people maintaining the stack had to know more, not less, because every new floor was one more height the leak could fall from.1

The interface over reasoning you never see

Now set a generative model on top of that stack, and notice it’s a different kind of layer entirely. When you ask a model to write a function, summarize a contract, or propose an architecture, what comes back is an abstraction over a process of reasoning the model never shows you. The prompt goes in. Something happens inside a distribution of hundreds of billions of parameters that nobody can hold in their head. Fluent prose comes out, or code that compiles, or a confident three-paragraph summary that reads like the work of someone who understood the material. The surface has the texture of competence. It reaches for the right nouns, the right libraries, the right register. And here’s the move worth sitting with: the output is an interface, and like every interface it’s a promise that you don’t need to know what happened underneath in order to act on it.

For most of computing history the promise leaked too, of course. If code matched intent perfectly we’d have far fewer bugs, and nobody would have had to invent return-to-libc to climb around a stack canary, or chain gadgets to chip away at ASLR. Intent and reality drift apart in odd ways, and a lot of people make a living in that gap. What made those leaks survivable was inspectability: the substrate underneath was always there to climb down into. This is the difference that matters, and it’s easy to miss because the word “abstraction” papers right over it. When a traditional abstraction leaks, you can descend. The compiler’s output disassembles. The query planner explains its choices with one command. The network stack sniffs packet by packet. The substrate under every classical abstraction is hard, but it’s readable, deterministic, and there: someone forced to can climb down through the layers and reconstruct, step by step, exactly why the system did what it did. Spolsky’s developer, stranded by a leaky TCP connection, had somewhere to go. The leak hurt, but it was legible, and legibility is the thing that makes a leak survivable.

A generated answer withdraws that bargain without telling you. The model’s interior isn’t intricate-but-deterministic engineering; it’s a probabilistic latent space, and the path from any one parameter to any one token can’t be traced by the person leaning on it. There are no intermediate stages to inspect, because the reasoning was never broken into stages. There’s no log of the deliberation, because there was no deliberation in the auditable sense, just a sample drawn from a distribution. So when this abstraction leaks you’re not in Spolsky’s situation at all. You can’t descend into the model the way he descended into TCP. The leak surfaces inside an artifact you didn’t write, produced by reasoning you can’t replay, and the only road back to solid ground is to reconstruct, after the fact, the mental model the model itself never had. The time you saved during generation gets repaid, with interest, during diagnosis. And the interest compounds hardest in exactly the places the stakes are highest, because those are the places the leak is quietest.

The leak you don’t see coming

Let me make this concrete with the kind of failure that doesn’t announce itself. Say you ask a model for a small piece of concurrent code: a counter shared across worker threads, bumped as jobs finish, so a dashboard can show progress. The model hands back something clean. It reads the current value, adds one, writes it back. It’s commented. It compiles. You run it with a handful of jobs and the count is right every time. You write a unit test with two workers and it passes. The demo works. By every check you’re in the habit of running, it looks like understood code.

It’s ninety-five percent correct, and the missing five percent is a race condition.2 Read-modify-write on a shared counter isn’t atomic. Two threads can read the same value, both add one, both write back the same result, and now you’ve counted one job instead of two. Under the load of your test, two workers and a few jobs, the threads almost never collide at the exact wrong instant, so the bug is invisible. Under production load, hundreds of workers hammering the counter thousands of times a second, the collision stops being rare and becomes constant, and your progress dashboard drifts quietly below the truth. Nothing crashes. No exception fires. No test goes red. The system is locally correct and globally wrong, and it stays that way until somebody notices the numbers don’t add up and goes hunting for a cause the generated code gives them no help in finding.

This is the leak in its purest form. The abstraction didn’t fail loudly the way a hallucinated API or a syntax error fails (both of which at least slam into reality right away and leave you the gift of an error message). It failed silently, in the gap between appearance and behavior, under a condition the prompt never specified and the dev environment never exercised. Recovering from it needs the exact thing the generation let you skip: a working feel for memory ordering, atomicity, and why time isn’t as simple as the syntax makes it look. If you fought your way to that understanding once, debugging a deadlock at two in the morning, you’ve got it, and you’ll smell the bug before you finish reading the function. If you’ve only ever accepted generated concurrency code that worked in the demo, the model has handed you a fault whose root cause lives a full layer below anything you were ever asked to learn. The same fluency that produced the bug is no help at all in finding it.

The confident summary leaks the same way, in a register that travels further. Ask a model to compress a long technical document, a vendor contract, a post-incident review, and you’ll get back something smooth, well-organized, authoritative. Almost everything in it is right. But summarization is compression, and compression is a string of decisions about what’s safe to throw away, and the model makes those decisions by statistical salience, not by knowing which clause is load-bearing. The one caveat that mattered, the carve-out that voids the warranty under exactly your conditions, the single sentence in the post-mortem that breaks the tidy narrative, is precisely the low-frequency, high-consequence detail a fluency-optimized system is most likely to smooth away. The summary doesn’t look incomplete. Incompleteness has no texture. It reads as whole, you act on it as whole, and the gap only shows up when the dropped caveat comes back wearing the costume of a surprise.

When the abstraction is knowledge

George Box gave statisticians a line they’ve repeated for fifty years: all models are wrong, but some are useful.3 A model is a deliberate simplification, valuable exactly because it leaves things out, and dangerous exactly when its users forget what got left out. A generated answer is a model in this sense, a compressed, lossy stand-in for a reasoning process and a body of fact, and it’s useful for the same reason it’s wrong. It discards. McLuhan4 would push it one step further. The medium is the message, he said, meaning the form of a communication system reshapes the people inside it more than any particular content it carries. The form of the generated answer is the answer-with-its-derivation-removed, and a civilization that meets its knowledge in that form gets slowly trained to treat the surface as the substance, to take the conclusion and lose the habit of asking what it stands on.

Which is where the leak stops being a software problem. When the abstraction is a piece of code, the leak is a bug, and a bug is recoverable: expensive, embarrassing, bounded, fixable by someone who descends and does the work. When the abstraction is knowledge itself, the leak is epistemic, and there’s no packet capture for that. A student who takes the worked answer and never builds the model that produces it. An analyst who ships the generated synthesis without the background to feel where it’s thin. A profession that accepts fluent output across a thousand small calls and slowly stops growing the depth that would catch the one call that was wrong. The failure isn’t that any single answer is false. It’s that the capacity to tell has quietly migrated out of the human and into a system that can’t be inspected. Not incompetence. Not automation. The slow withdrawal of the bargain every abstraction before this one honored: that if you were ever truly forced to, you could descend.

The defense isn’t to refuse the tool, which would be both futile and a little dishonest given how much it actually buys you. Used well, the payoff is huge. The defense is to keep paying down the understanding before the leak calls it in all at once. Treat the generated answer as a draft over reasoning you stay responsible for reconstructing. Stay fluent enough in the layer beneath to climb down when the surface lies. Keep the world the smooth output hides somewhere in view. Spolsky’s “law” never promised a stack that doesn’t leak. It promised the leaks would come, and that on the day they did, the only thing between you and the substrate would be how much of it you still understood. That was true of TCP. It’s more true, not less, of the most fluent abstraction we’ve ever built, because it’s the first one you can’t climb back down. (There’s a related ache here, the way the tool gets you to eighty percent and the last twenty was always where the engineer actually lived. That one’s its own post, soon.)

If you’ve watched a generated answer pass every check you had and fail anyway, in production, under load, in the one condition nobody typed into the prompt, I’d love to hear it. Those are the stories worth collecting.

Sources & Further Reading

Footnotes

  1. This is also why we test, and test, and test. Once you’ve stacked abstractions on abstractions, it’s hard, and past a certain depth basically impossible, to prove the contracts hold under every circumstance, so we stop pretending we can. We test the paths we actually intend to use, pin the behavior down there, and quietly accept that everything outside that is uncharted. As the saying goes, if it isn’t tested, it’s probably broken. Testing is mostly how we choose which leaks we’re willing to discover on purpose, in daylight, instead of at 3am. ↩

  2. A fair question is hiding inside that word, correct. Correct by what standard? The counter code is correct on my laptop with two threads and wrong in production with two hundred, and nothing about the code changed in between. What changed was the context it had to survive in. Obi-Wan got there ahead of the rest of us: “you’re going to find that many of the truths we cling to depend greatly on our own point of view.” That isn’t relativism, it’s the last post in a bathrobe: a claim is true relative to the context it was formed in, and the context is the point of view. A generated answer carries the claim and drops the point of view, which is the exact part that decided whether the claim was ever true. ↩

  3. The aphorism gets quoted to death, but the work behind it is worth the shelf space. If you ever want the full George Box, read Box and Draper’s Empirical Model-Building and Response Surfaces (Wiley, 1987), a classic on fitting models to data without fooling yourself, and the home of the longer line: “Essentially, all models are wrong, but some are useful.” Everyone who builds models, statistical or otherwise, should read it once. ↩

  4. Marshall McLuhan, Understanding Media: The Extensions of Man (McGraw-Hill, 1964). “The Medium Is the Message” is the title of the opening chapter, and the argument that a medium’s form, not its content, does the deeper work on the people who use it is the spine of the whole book. ↩

Share this post