What does "radical reproducibility" mean in benchmarking?
Status as of: August 2026. This is a methodology post. It defines one term, then checks the term against the public record.
What exactly is "radical reproducibility"?
A benchmark result is radically reproducible when the complete re-run unit is published together: the raw data, the full code, the random seeds, the exact problem instance, and the compute budget. Anyone — a reviewer, a competitor, a stranger with a laptop — can re-execute the run and obtain the same number, without emailing the authors and without trusting them. The operating slogan is: don't believe us — run the notebook.
Each missing component is a specific escape hatch. No raw data: only the summary claim exists. No code: every re-implementation differs and the gap gets blamed on the re-implementer. No seeds: a stochastic solver gets quiet retries. No exact instance: the rebuttal is "you ran a similar problem". No compute budget: the classical rival can be starved of time and tuned down — the failure mode we documented in why a weak classical baseline ruins a quantum benchmark. A benchmark missing any of the five is not auditable; it is a press release with error bars.
Why isn't peer review enough?
Because peer review reviews the text, not the experiment. Reviewers judge whether the methods sound plausible; almost none re-run the computation. The journals themselves are explicit about what they require — and the requirement is weaker than the term "reproducible" suggests.
The one large audit that checked policy against practice is Stodden, Seiler & Ma (PNAS, 2018): they took 204 computational papers published in Science in 2011–2012 — after its policy took effect — and simply asked for the artifacts. 36% of the 180 contactable author groups provided at least some material. 26% of the sampled findings could actually be reproduced (95% CI: 20–32%).
| What | Status | Date | Source |
|---|---|---|---|
| Science: "all data necessary… must be available to any reader… all computer codes involved… must also be available" | Journal policy, condition of publication | Effective Feb 11, 2011 | Stodden et al., PNAS (2018), quoting the policy |
| Nature Portfolio: data availability statement in every research paper | Mandatory | Since Sep 2016 | Nature Portfolio reporting standards |
| Nature Portfolio: releasing custom code so readers can repeat results | "Best practice" (statement mandatory; release is not) | Accessed Aug 2026 | Nature Portfolio reporting standards |
| Authors actually providing artifacts on request | 36% of contacted (65 of 180) | Measured 2018 | Stodden et al., PNAS (2018) |
| Findings actually reproduced | 26% (95% CI 20–32%) | Measured 2018 | Stodden et al., PNAS (2018) |
Read the gap: the strongest data policy in mainstream science produced artifacts-in-hand for roughly a third of requests, and reproduced results for a quarter of papers. Policy is not the mechanism. The mechanism is publishing the re-run unit up front, so nobody has to ask. That is what the word "radical" adds.
What does the track record show in quantum computing?
Here is the historical thesis of this post, stated plainly: every famous reversal of a quantum computing claim was executed by someone re-running public artifacts. Not by re-reading the paper — by re-running the target. Where the claim, instance and protocol were fully specified, the classical side could show up with a stronger rival; where they were not, the claim would have been unfalsifiable.
| Claim | Public artifact | Who re-ran it | Outcome | Source |
|---|---|---|---|---|
| Exponential QML speedup for recommendation systems (2016) | Algorithm + data-access model, fully specified | Tang (2018), classical algorithm under matched access | Exponential gap collapsed to polynomial | arXiv:1807.04271 |
| IBM 127-qubit kicked Ising, "utility beyond brute force" (2023) | Circuits + observables fully specified in the paper | Tindall et al. (2024), tensor networks on modest classical hardware | Matched the quantum results | PRX Quantum 5, 010308 |
| D-Wave Advantage2 spin-glass dynamics "beyond classical" (Mar 2025) | Lattice geometries + simulation protocol, published | Tindall et al. (Science, Jul 2026) — early calculations on a personal laptop with open-source ITensor | Matched the covered regimes; in declared dispute — D-Wave's May 2026 rebuttal argues the hardest instances and highest-order observables remain out of classical reach | Science 392, 868 (2026) · D-Wave rebuttal |
Two honesty notes. First, none of this is an accusation: each original claim was serious work, published with enough specification that others could test it — that is science functioning, not fraud exposed. The corrections are measurements corrected, exactly the pattern we mapped in dequantization and in weak baselines. Second, note the symmetry: the re-runs themselves were reproducible — Tindall's replica runs on an open-source library anyone can install, which is why the correction stuck instead of becoming a he-said-she-said. The D-Wave case shows the standard working in both directions at once: the artifacts were public enough to attack, and the dispute is now about which regimes the replica covers — an argument you can only have when both sides published something re-runnable.
Who is building reproducibility infrastructure?
The direction of travel is shared instance banks with published baselines. QOBLIB (Nature Computational Science, 2026; 32 authors across 18 institutions including IBM) publishes ten optimization problem classes with exact instances, classical baseline solutions from named solvers, and validation routines — designed, in the authors' words, "to ensure comparability of the used methods, reproducibility of the respective results, and trackability of algorithmic and hardware improvements over time". That is the re-run unit, institutionalized: anyone claiming quantum progress on those classes can be checked against the same instances and the same baselines. It is the structural answer to the incomparability problem we documented for vendor metrics, where every vendor scores on a scale it defined itself.
How does Rosetta apply this?
Our sealed-run protocol is the re-run unit applied to our own numbers first. Every run in our sealed series (the V-0012 portfolio-optimization series and the RQ quantum-walk series) seals the exact instance, the random seeds, the compute budget and the result together, per run — the same five components as the checklist above. The scope is deliberately small and we state it: a handful of optimization and quantum-walk classes, in which the classical baseline is unbeaten in every sealed run to date. This is a worked example of the standard, not a universal verdict and not a product claim.
What we know / what we don't know
What we know. The major journals' data policies are real and dated (Science 2011, Nature 2016), but the only large audit found 36% artifact provision and 26% reproduction. The three best-known reversals of quantum computing claims (2018, 2024, 2026) all ran on public artifacts. QOBLIB now publishes instances, baselines and validation code for ten optimization classes.
What we don't know. There is no public measurement of what fraction of quantum computing claims publish re-runnable artifacts — the 26% figure is from general computational papers in Science 2011–2012, not quantum-specific, and nobody has run the equivalent audit on quantum benchmarks. On real QPUs, bit-exact re-runs are impossible in principle — calibration drifts between sessions and machines get decommissioned — and re-running the quantum side costs real money per shot (what a run costs), so "radical" is often fully achievable only on the classical or simulated side; what statistical standard should replace bit-exactness on hardware is not settled. The D-Wave/Flatiron dispute is not resolved: reproducibility executed the check, not the final verdict on every regime. And whether QOBLIB becomes the field's shared standard is too early to call.
Rosetta Q publishes verdicts with reproducible raw data. This is educational content, not a product claim.