Software Failure Wears the Clothes of Success

Software introduced something genuinely new into the world: the thing that runs and is wrong. A program can execute without a single error, render beautifully, return a number, and be false in a way nothing on your screen will indicate. That option was not available with a soldering iron.

Why Physical Systems Are Blunt Reviewers

In electronics, a circuit works, or nothing happens, or something gets warm and you learn a permanent lesson about polarity. The physical world cannot produce a convincing imitation of what you asked for. It has no idea what you were going for, is not impressed by your intentions, and takes no account of how long you spent. It just answers. For most of engineering history, that blunt reviewer was the only one anyone had.

Hardware is not honest all the way down. A board can pass on the bench and fail at temperature. A meter calibrated wrong reads confidently wrong all day. Intermittent faults are real. But the narrower claim survives: physical failure mostly announces itself as failure. Software failure learned to show up wearing the clothes of success.

AI as the Extreme Case

A language model is optimized to produce output that looks like what was asked for. That is a literal description of the training objective. The technology is therefore aimed with unusual precision at producing plausible-looking results, and only secondarily at producing correct ones.

Veracode tested more than a hundred models across eighty coding tasks and found security flaws in 45% of the generated code, while the security pass rate stayed flat through mid-2025 even as the models got functionally better at writing code that works. The models improved at looking right without improving at being right on the dimension that matters most for production safety.

The Perception Gap

The subtler danger is what plausible output does to the person using it. METR ran a randomized trial with experienced open-source developers working on real tasks in repositories they already knew well. They finished 19% slower with AI assistance than without it, but estimated afterward that they had been about 20% faster. The people inside the loop could not feel which direction the effect was pointing, and these were not beginners.

When the output always looks like success, the human ability to detect failure degrades. This is not a tooling problem that better models will solve on their own. It is a structural feature of any system optimized to produce output that matches the shape of what was requested.