A simulator will run a billion miles overnight for the cost of the compute. It is the best value in autonomy development and it is indispensable. It is also structurally incapable of the one thing the industry most needs from it.
Simulation earns its place, and the reason is arithmetic. Work published by RAND in 2016 calculated what it would take to demonstrate autonomous safety by driving alone: 275 million failure-free miles to show a failure rate no worse than human, 8.8 billion to estimate the rate within twenty percent, and 11 billion to show a twenty percent improvement with statistical confidence. Their conclusion was that developers and third-party testers "cannot drive their way to safety."
They are right, and their recommended answer is simulation. This essay accepts the statistical argument entirely and disputes a different one.
A simulator renders what was specified
Every scenario in a simulator exists because a person wrote it, or because a generative model produced a variation on something a person wrote. The environment, the actors, the physics and the behavioral models all encode somebody's understanding of how the world works. The field's own survey literature concedes the shape of the problem: an infinite number of scenarios can theoretically occur in real traffic, and scenario-based testing works by reducing that infinity to a finite, human-authored set.
The international safety standard for this has a name for the residual. Its framework sorts scenarios by whether they are known and whether they are hazardous, and reserves a quadrant for the unknown and hazardous, the situations nobody has thought of. The entire process is defined as shrinking that box. A simulator can only exercise the boxes that have already been filled.
This means a simulator can test coverage against known unknowns with enormous efficiency. It cannot produce an unknown unknown, because producing one would require it to contain information nobody put into it.
You cannot be surprised by a world you built. At most you can be surprised by an implication of a rule you wrote.
That second thing is real and valuable. Emergent behavior from simple rules genuinely does reveal problems. But it is a different category from the road throwing something at you that nobody in the industry had considered, which is where the serious incidents come from.
The specific failures
Four things simulation systematically under-represents, in rough order of how much they matter.
Human irrationality. Behavioral models in simulators are built around plausible agents. But real road users reverse down slip roads, cross against traffic while looking at their phone, stop dead in live lanes, and wave through a vehicle that does not have the right of way. The tail of human behavior is far stranger than any model of human behavior.
Compound conditions. Simulators test factors and combinations of factors that somebody enumerated. However reality combines low sun, a wet surface, a temporary road layout, a poorly sited sign and a distracted driver simultaneously and without warning. Compound events are rarer than single factors and reliably more severe.
The physical world's untidiness. Faded markings, a bollard that has been hit and never replaced, contradictory temporary signage, a pothole full of water that reads as solid. Simulated environments are built; real ones have decayed for thirty years under budget pressure.
The unknown outcome. In simulation you know what happened because you defined it. In the world, the value of an event is often that somebody survived it in a way that tells you something, and you only know that because you have the outcome.
The circularity risk
There is a compounding version of this problem that concerns us more than any single limit.
Train a perception model on real data. Use that model to inform the behavioral models in a simulator. Generate training data in the simulator. Train the next perception model partly on that. Repeat.
Each step is defensible. The system as a whole drifts steadily toward internal consistency and away from the world, and nothing in the loop is measuring the gap. It becomes excellent at the world as the pipeline understands it and no better at the world as it is, and the error is invisible from inside, because every validation is also run inside.
The only correction is a periodic collision with reality: held-out real events, with real outcomes, that the pipeline did not generate.
Measuring the drift should be routine rather than clever. Run each generation against the same held-out set and track whether the divergence between simulated performance and measured performance is widening or not. It is unglamorous work and it is rarely published, which is itself informative.
What the pairing looks like
The useful conclusion is not "simulate less." It is that simulation and “measured reality” do different jobs and are both needed; and the second is currently under-supplied.
Simulation for coverage. Exhaustive testing of the situations you know about, at a volume road testing cannot approach.
Measured reality for discovery. Finding out what is out there that nobody put in the scenario library, and then feeding it back so simulation can cover it too.
Measured reality for validation. Held-out real events with known outcomes, as the check on whether the whole apparatus still corresponds to anything.
Reality is the source of new scenarios. Simulation is the multiplier on them. Run the multiplier without the source and the result converges on a world of your own design.
The correction this essay needs
"Simulation never discovers" is too strong, and a knowledgeable reader will catch it. Formal scenario generation has produced test cases that transferred from simulation to a real test track and exposed genuine defects nobody had catalogued. A published result, and a good one. That looks like discovery, and in a narrow sense it is.
The defensible version is sharper. Simulation finds unknown points inside a specified space. Only reality supplies unknown dimensions.
A search can find a parameter combination nobody tried. It cannot invent an axis nobody encoded. Jaywalking was a missing dimension. A body beneath a chassis was a missing dimension. A towed vehicle at the wrong orientation was a missing dimension. None of those is a point a simulation sampler would have reached, however cleverly it searched.
The same paper that demonstrates the transfer also demonstrates the gap. Small rendering details in simulation can be inconsistent with what the real world produces, and this has been known to throw off perception algorithms in particular, so that an AV system performs very differently in real life than it did in simulation.
It is worth being specific about why, because it is not a rendering-quality problem that better graphics will eventually solve. A simulator renders an image. A camera produces an artifact. Rolling shutter skews a fast-moving object. Lens flare erases a region of the frame at exactly the sun angle where a pedestrian is hardest to see. Motion blur, sensor noise at low light, compression, a smear of road grime on the housing. Perception models are sensitive to all of it, and almost none of it is what a renderer is trying to reproduce.
A photorealistic frame and a real frame can look identical to a person and remain measurably different to the model consuming them. Which means the gap is not between a simulator and reality. It is between a simulator and a sensor.
Where the industry agrees with us
The leading autonomy program published this in late 2025:
There is simply no substitute for this volume of real-world fully autonomous experience. No amount of simulation, manually driven data collection, or operations with a test driver can replicate the spectrum of situations and reactions the Driver encounters when it's fully in charge.
Its own safety methodology paper is more precise still: any simulation requires validation, so the company relies on road driving to validate its simulator. The novelty budget is imported from reality. That is not our claim about a competitor. It is their claim about themselves.
The question we would put to a program
Not "how many miles have you simulated." That number is a function of budget.
The question is this: when did a scenario last enter your library because the world produced it and nobody on your team had thought of it? If the answer is ‘a while ago’, then the library has stopped growing in the direction that matters most.
A related industry argument is running in public about sensing, and it is worth mentioning here only to set aside. Whether a system should rely on cameras alone or fuse them with lidar, radar and other sensors is a real technical question, and it is being argued by people whose positions correlate closely with what they have already spent. We will take that one on separately. It is a question about how you perceive the world. This essay is about whether you have seen enough of it.
What would change our mind
A documented case of a simulator surfacing a genuinely novel dimension: a failure mode whose axis was absent from the scenario schema, discovered in simulation and later confirmed on the road. That would change how we think about the role of simulation.
What we are least sure about
Where the line sits between a point and a dimension, in the space of all possible scenarios. If a scenario language is expressive enough, almost anything can be described as a point inside it, and the distinction starts to dissolve. We think it holds, because in the jaywalking case the schema genuinely had no way to express a pedestrian crossing away from a crosswalk. But it is not a line we would want to defend in every case.