Many companies building AI for the physical world evaluate their performance on their own data and using their own evaluation metrics. In any other field with this much at stake, that arrangement has a name and a regulator.
Consider how it works elsewhere. Accounts are audited by a firm that did not prepare them. Drugs are trialed under protocols set outside the company that will sell them. Vehicle crashworthiness is rated by bodies with no commercial interest in the result. Building work is signed off by an inspector who does not work for the builder.
None of these systems is perfect. All of them exist because we learned, expensively, that self-assessment under commercial pressure produces predictable answers.
Take medicine. Prospective declaration of outcomes became the norm around 2000. Before that, large trials run by a US health institute showed significant benefit in 57% of cases. After that, it was 8%. Nothing about the underlying science changed. What changed was who wrote down the outcome measure, and when.
The authors are careful. They say pre-registration "may have contributed" to the trend, and it is an observational finding across fifty-five trials. We are careful with it too. But a fifty-point swing is not a rounding error, and it is the most direct evidence available that defining the test in advance and in public changes the result.
Now consider a company deploying an autonomous system, a driver-monitoring product, or a collision-prediction model. Standards exist, and operators of autonomous vehicles do have to publish a safety case that regulators and the public can dissect. But those safety cases are each articulated differently, the company still selects its own test data, defines what counts as a detection, a miss and a false positive, and chooses the baseline to compare against. In most cases nobody outside the company can reproduce the result.
How the numbers get made
The manipulation, where it happens, is rarely fraudulent. It is definitional, and it is often invisible even to the people doing it.
Choose the test set. Every dataset has a distribution. Evaluate on data drawn from the same distribution you trained on and the result can be excellent, but uninformative.
Define the categories. What counts as an edge case is a choice. Specify your operational design domain narrowly enough and your performance improves without a line of code changing.
Pick the comparator. Comparing against a weak baseline, or against a system optimized for something else, produces a large multiple that means very little.
Blend unlike tests. The most common one, and the most damaging: reporting an independent institution's test of one product alongside an internal benchmark of another, in a single sentence, so the credibility of the first transfers to the second.
The clearest demonstration of the failure mode is probably the Dieselgate scandal: real-world nitrogen-oxide emissions from certain diesel vehicles were found to exceed the standard by fifteen to thirty-five times, from vehicles that had passed laboratory certification. The divergence was only discoverable because researchers tested off the known test conditions. The mechanism was literal evaluation-gaming: the software detected that it was being tested and behaved differently.
That last one is worth being specific about, because we have to be careful of it ourselves. Independent controlled testing by the Virginia Tech Transportation Institute measured our alert performance. Separately, we measured the performance of our BADAS collision anticipation model across different types of events ourselves. These are two different things, measured by two different parties, about two different products. Writing "independently ranked first across every benchmark" would merge them into a claim neither one supports, and it is the sort of sentence that gets written by accident in a hurry.
The failure mode is not lying. It is a true sentence and a true sentence placed next to each other so that a third, untrue thing is implied.
What independent verification would need
Five things have to be true before anyone can effectively evaluate the performance of an automotive safety system.
Held by someone with no stake. This should go without saying: the evaluating party must not sell a competing system and must not be paid on the outcome.
Data the system has not seen. Performance must be evaluated on held-out real-world events, drawn from a distribution the developer did not choose, with the outcome known.
Definitions fixed in advance and published. What counts as a detection, a miss, a false positive and an edge case must be written down before the run and readable afterwards.
Reproducible. Another party running the same protocol should get the same answer, within a stated tolerance. Modern systems are stochastic, so identical is the wrong bar; a published variance is the right one. What should never vary is the direction of the finding. If two competent parties running the same protocol disagree about whether a system improved, it was a demonstration rather than a measurement.
Refreshed, not just held out. A validation dataset stops being held out the moment people start optimizing against it. Nobody has to cheat for this to happen. Teams iterate, they compare, they keep what scores better, and after enough cycles the dataset has quietly leaked into the design of the thing it was meant to judge. This is the documented history of long-lived benchmarks in machine learning, and the regimes that hold up are the ones whose evaluation data is rotated on a schedule rather than fixed forever.
Which means a benchmark has an expiry date, and any credible verification regime has to publish when its evaluation set was last refreshed. A number measured against a five-year-old fixed set is a statement about that set rather than about the world, and it is worth very little.
Our position, and its obvious problem
We are not a neutral commentator here. We hold a large real-world record, we do not sell autonomous driving systems and we do not write insurance: this puts us in an unusually good position to supply held-out evaluation datasets to the industry. That is a commercial opportunity for us and you should read this essay knowing that.
The honest version of our position is therefore narrower than "we should be the referee." It is this: the data used to evaluate a system should come from somewhere other than the party being evaluated, and the definitions should be public. We are one possible source of the first. We are not, and should not be, the authority on the second.
A standards body, a public benchmark with governance, or a regulator would all be better than any vendor, including us. In the absence of any of those, published definitions and held-out datasets are the minimum.
A correction we owe the reader
It would be easy, and wrong, to say that autonomy developers never submit to peer review. They do. The leading program has published crash-rate comparisons in peer-reviewed traffic-safety journals, and independent academics have published methodological critiques of them in the same venues. That is a functioning scholarly conversation and it is more than most of this industry manages.
The narrower point survives, and it is the one that matters: those papers are typically authored by the party being evaluated, and the exposure data behind them is proprietary, so a reviewer usually cannot recompute the result independently.
Peer review checks reasoning, and in some venues it checks more. Computer Vision and AI conferences increasingly require that data and benchmarks be published so a reviewer can verify a result independently. Where that requirement is absent, nobody re-runs the numbers. When a manufacturer retroactively revised its own published safety statistics and the downward adjustments fell disproportionately on the comparison category that flattered its own product, the only reason anyone knows is that an outside academic went and checked.
Regulators find what self-reporting does not. One investigation into an autonomous program’s incident surfaced nine additional similar incidents after it began. Another operator was fined $1.5 million by the National Highway Traffic Safety Administration (NHTSA) under a consent order for submitting incomplete crash reports, which initially omitted details that its autonomous vehicle had dragged a pedestrian after a collision.
How other industries actually solved it
Two mechanisms are worth copying and neither requires secrecy.
Anonymous procurement. Vehicle safety testing bodies buy their test cars from dealerships as an ordinary customer would. No vehicle comes from the manufacturer. The protocol is fully published; what the manufacturer cannot control is which exact car gets tested. That is the physical-world equivalent of a held-out set the developer cannot access, and it defeats the standard objection that independent evaluation means secret evaluation.
Publication contingent on registration. Medical journal editors could not compel anyone to pre-register a trial, so they made publication conditional on it. The AI equivalent is available today and needs no regulator at all: conferences and journals can increasingly require that, where possible, results be verified against a held-out dataset using a published evaluation method. That is a change to reviewing guidelines, nothing more.
What would change our mind
A credible independent institution emerging: a standards body, a consortium, an insurer-funded test house. If one exists, our role is to supply held-out data to it and stop arguing. We are making this case because the institution is absent, not because we want the job.
What we are least sure about
Whether independence survives contact with money. Audit independence is imperfect.
Independence regimes decay. The honest claim is that they decay more slowly than self-assessment does, not that they solve the problem.
The cost of not fixing this
Self-graded safety claims work until the first serious public failure of a system that had excellent published numbers. At that point the reasonable inference is not "that company overstated" but "none of these numbers mean anything," and every honest actor in the category pays for it.
Which is the practical argument for fixing it now, while it is a discipline the industry chooses rather than one imposed on it by regulatory bodies after an incident.