How do you "cheat" on a quantum benchmark (and how do you avoid it)?

Status as of: August 2026. A quantum benchmark can mislead without anyone lying. The number that gets published is the output of a long chain of design decisions — which instances, how much tuning, which statistic, which metric, which shots count, which rival, which platform — and each of those decisions can quietly move the verdict before a single qubit fires. This post is a taxonomy of the seven moves that do it, each with a documented public episode and its countermeasure. One thing before the list: "cheat" here names a property of experiment design, not of intent. None of the episodes below is an accusation of bad faith — every one is a measurement corrected in the open, a rule proposed in public, or a disclosure that the headline outran. That is exactly why the taxonomy matters: these failure modes do not need a villain.

Why can an honest team still publish a misleading benchmark?

Because the genre is older than quantum computing. In 1991, David Bailey wrote "Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers," followed by a 1992 companion cataloguing real cases — the point of both was that misleading performance claims in supercomputing were mostly legal moves: true statements arranged so the reader draws a false conclusion. In November 2024, Catherine McGeoch — who spent years running benchmarks inside a quantum vendor — wrote the quantum sequel, "How NOT to Fool the Masses When Giving Performance Results for Quantum Computers" (arXiv:2411.08860). Her four rules compress the whole problem: don't claim superior performance without runtimes; don't report optimized results without the tuning time it took; don't claim speed against solvers on imaginary platforms; no cherry-picking without justification. Every trick below is a way of breaking one of those rules while every individual sentence in the paper stays true.

The weak classical baseline — the single most common sin — has its own post, and we won't re-litigate it here. This post is about the rest of the protocol: everything that can bend a benchmark even when the baseline is strong.

What are the seven tricks — and where is each one documented?

1. Cherry-picking instances. Run on many problem instances; report the ones where the quantum side looked good. The canonical demonstration that this changes verdicts is Rønnow et al., "Defining and detecting quantum speedup" (Science 345, 420, 2014): on random spin-glass instances, analyzing subsets of the data could suggest speedup where the entire data set showed none. McGeoch's rule four ("no cherry-picking without justification") exists because the move is still common. Countermeasure: a pre-registered instance suite that is fixed before results are known — this is exactly what QOBLIB now provides for optimization (10 problem classes, 1,200+ instances, published baselines; arXiv:2504.03832, Nature Computational Science 2026), and why IBM's 2026 benchmarking playbook argues advantage claims must run against a community-fixed set rather than instances of the claimant's choosing.

2. Uneven budgets. Spend weeks tuning the quantum pipeline; run the classical rival out of the box — or vice versa. This is the tuning-side twin of the weak-baseline problem: McGeoch's rule two ("don't report optimized results without mentioning the tuning time") targets it directly, and one of her anonymized case studies is a gate-model heuristic whose reported superiority rested on parameter tuning with the optimization time undisclosed. Countermeasure: declare the tuning budget for both sides, in the same units, in the paper.

3. Best-of-N versus the median. Run the experiment many times; publish the best run. Why this misleads is not just intuition — it is measured: time-to-solution distributions for annealing-style solvers have power-law heavy tails (Steiger, Rønnow & Troyer, arXiv:1504.07991, 2015), so the best run, the mean, and the median can tell three different stories about the same machine. A single reported number silently chooses one. Countermeasure: publish the full distribution with seeds, so anyone can recompute any statistic — radical reproducibility is the infrastructure for this.

4. Choosing the metric that favors you. Time-to-solution, time-to-target, quality-at-fixed-time, success probability, energy — different metrics crown different winners on the same data. The instructive episode here is open and honest: in 2015, D-Wave scientists (King et al., arXiv:1508.05087 — McGeoch among the authors) openly proposed a new metric, time-to-target, arguing that ground-state metrics conflate algorithm quality with analog noise. The proposal was transparent about its rationale; that is precisely what makes it instructive — it shows, from inside the field, that the choice of metric is a degree of freedom that moves outcomes. McGeoch's rule one (no performance claims without runtimes) is the guard on the silent version of the same move. The metric wars between vendors have their own post. Countermeasure: name the metric before the run, and report the ranking under the standard alternatives too.

5. Silent post-selection. Discard "bad" shots or runs without saying so. The physics literature shows why this is powerful: in noisy Gaussian boson sampling, post-selecting on a subset of outcomes measurably improves the quality of the sampled distribution ("part is better than whole," arXiv:2205.02586, 2022). Declared, post-selection is a legitimate technique with a cost that can be accounted. Undeclared, the same operation converts a noisy experiment into a clean-looking one with no visible trace. Countermeasure: report every discard and the rule that produced it, with raw counts published.

6. The wrong classical rival. Compare against a classical algorithm of the wrong class — an exact solver when heuristics dominate, or a generic method when a specialized one exists. The canonical public episode is December 2015: a widely reported "100,000,000×" speedup held against simulated annealing, on instances native to the hardware's own connectivity graph; against quantum Monte Carlo the advantage was constant-factor, and a purpose-built classical algorithm (Selby's) outperformed the hardware on the same problems — all of which was laid out publicly within days (Aaronson, Dec 9, 2015). The technical paper disclosed its comparisons; the headline number traveled without them. Countermeasure: the strongest known classical rival of the right class, on the same instance — the weak-baseline post has the full checklist.

7. Simulation presented as if it were hardware. Report numbers from noiseless simulation — or extrapolations to machines that do not exist — in a way that lets readers believe hardware did it. McGeoch's rule three calls these "imaginary platforms," and one of her case studies extrapolated performance to problem sizes no existing device can hold. The honest version of the same practice is simply the label: our own sealed series V-0012 runs its quantum side in noiseless simulation, says so in the verdict, and notes that the simulation favors the quantum side — which makes the classical solver's 20-for-20 record against it more informative, not less. Countermeasure: the platform label travels with the number, everywhere the number goes.

Where in the pipeline does each trick enter?

The mechanism behind all seven tricks is the same and worth naming: decisions drift toward whichever option flatters the result — unless they are pinned before the result exists. Nobody has to intend this. If the instances, budgets, metric, and statistic are still open questions when the data arrives, the flattering configuration is simply the one that survives drafting.

What does the taxonomy look like as a table?

How do you read a benchmark, protocol edition?

The weak-baseline post gives five questions about the classical rival. These seven are about everything else in the protocol:

1. Were the instances fixed — publicly — before any results existed?
2. What did each side spend on tuning, and is that number in the paper?
3. Is the headline a median over declared runs, or a best run over an undeclared count?
4. Was the metric named before the run? Would the ranking survive the standard alternatives?
5. How many shots or runs were discarded, under what rule, and are raw counts available?
6. Is the rival the strongest known classical method of the right class?
7. Does the platform label (hardware, noisy simulation, noiseless simulation, extrapolation) travel with the number?

A benchmark that answers all seven in the affirmative can still be wrong — but it can no longer be quietly wrong.

What do we do about this ourselves?

Rosetta Q's answer is the seal protocol: for each sealed run in our series (V-0012 and successors), the instance, the seeds, the compute budget, and the metric are declared before the run, and the result is recorded whichever way it lands — which, in our small declared classes of portfolio optimization, has been a classical win 20 out of 20 times, with the quantum side simulated noiselessly and labeled as such. That is tricks 1 through 7 answered by construction, at small declared scope. It is not a universal verdict about anything — it is what the countermeasure column of the table looks like when you actually run it.

What we know / What we don't know

What we know: every trick above has a documented public episode or a measured demonstration, each linked in the table; the field's own practitioners have codified the rules (Bailey 1991/1992 for HPC; McGeoch 2024 for quantum); pre-registered suites now exist for optimization (QOBLIB); and the measured end-to-end scoreboard for useful problems remains at zero, which is the backdrop that makes protocol quality decisive.

What we don't know: how prevalent each trick is — nobody audits the quantum benchmarking literature systematically, so the episodes here are the documented cases, not a measured rate; whether pre-registration will actually be adopted beyond optimization (QOBLIB is young and adoption is voluntary); whether any mechanism will ever enforce discard-reporting on hardware runs, where re-execution is costly by the shot; and whether our own seven-question checklist covers moves not yet documented — taxonomies of tricks are always one trick behind.

Nothing in this post accuses any vendor, lab, or author of bad faith. Every episode cited is a measurement corrected, a rule proposed, or a disclosure completed — in the open, by the field itself. That is the system working; the taxonomy exists so it can work earlier.

Sources

• McGeoch, How NOT to Fool the Masses When Giving Performance Results for Quantum Computers (arXiv:2411.08860, 2024)
• Bailey, Misleading Performance Reporting in the Supercomputing Field (1992)
• Rønnow et al., Defining and detecting quantum speedup (Science 345, 420, 2014)
• Steiger, Rønnow & Troyer, Heavy tails in the distribution of time to solution (arXiv:1504.07991, 2015)
• King et al., Benchmarking a quantum annealing processor with the time-to-target metric (arXiv:1508.05087, 2015)
• Aaronson, Google, D-Wave, and the case of the factor-10⁸ speedup for WHAT? (Dec 9, 2015)
• Post-selection in noisy Gaussian boson sampling: part is better than whole (arXiv:2205.02586, 2022)
• QOBLIB — The Intractable Decathlon (arXiv:2504.03832; Nature Computational Science, 2026)
• IBM Research, A new playbook for quantum optimization benchmarking (Jun 24, 2026)

Rosetta Q publishes verdicts with reproducible raw data. This is educational content, not a product claim.