What is a blind challenge and why is it the hardest judge of a model?

TL;DR (status as of October 15, 2026): a blind challenge is a test where an external judge holds back the answers until all predictions are submitted. It is the hardest judge a model can face: the scorer is not the author, the rules are fixed in advance, and the answer key does not exist in public, so there is nothing to overfit. Protein structure prediction has had this institution since 1994 — CASP — and it is where AlphaFold2 earned its credit, scoring a median GDT of 92.4 at CASP14 in 2020 (deepmind.google). On October 14, 2026, OpenBind launched the first blind challenge for protein–ligand complexes: 400 unseen compounds against the Zika virus NS2B-NS3 protease, final submissions due December 16, 2026 at 23:59 UTC (blopig.com). Quantum computing has no CASP. Rosetta Quantum maintains a ledger of quantum claims precisely because that judge is missing — and this post states plainly where Rosetta's own sealed predictions sit on the same ladder: the lowest rung.

What makes a judge «blind»?

A judge is blind when it scores predictions against answers that did not exist in public while the predictions were being made. That single property kills the two failure modes that corrupt most model evaluations: training on the test set (the answer leaked into the model) and selective reporting (the author publishes the wins and shelves the losses). Rosetta Quantum uses a four-rung ladder to classify how hard a judge is, and the rungs are not interchangeable.

On rung 1 sits the self-sealed claim: the author freezes rules, data and a hash before the outcome is known, then scores themself later. Rosetta's own PREDICTION seals — the Q-Day bet of October 9 and the two allosteric-site commitments of October 13 — live here. The freeze is auditable; the scorer is still the house. On rung 2, the pre-registered benchmark: instances are fixed and published before entrants run, like QOBLIB's 1,264 optimization instances (our explainer, arXiv:2504.03832) — but entrants score themselves, on their own schedule. On rung 3, the external auditor: an outside team with access and no stake, like DARPA's Quantum Benchmarking Initiative, which moved 6 companies to its final stage on October 7, 2026 (our explainer, darpa.mil) — though its verification results are not public. And on rung 4, the blind challenge: external judge, preset rules, withheld answers, public ranking. Every rung above 1 removes a degree of freedom from the person who wants to believe their own model.

What has 30 years of CASP proven?

CASP — the Critical Assessment of protein Structure Prediction — has run every two years since 1994 (predictioncenter.org). Its mechanism is simple and brutal: experimentalists deposit structures that are about to be solved but are not yet public; modelers predict them; assessors score the predictions once the real structures are released. The answers exist, but the predictors cannot reach them.

The scale of the current round shows what a mature blind institution looks like. CASP16 drew around 110 prediction centers from 22 countries working on 103 selected targets, and its pharmaceutical ligand track alone scored 233 pose targets across 34 groups, plus 140 affinity targets across 28 groups (CASP16 introduction, predictioncenter.org). The field's most famous result was decided in exactly this arena: AlphaFold2's median GDT of 92.4 at CASP14 in 2020 was a score assigned by assessors who held answers the model had never seen (deepmind.google; Jumper et al., Nature 596, 2021). That is why the result convinced structural biologists in a way no self-reported benchmark ever had. The credit was issued by the judge, not claimed by the author.

The same CASP16 also shows what blind judging reveals when a field is not solved: pose prediction and affinity ranking were assessed as separate categories precisely because getting the geometry right and ranking binding strength are different problems — and the second remains much harder, as the OpenBind benchmark below measures independently.

What exactly will OpenBind's first blind challenge measure?

OpenBind is a UK-based consortium — Diamond Light Source with the universities of Oxford, Washington and Columbia, EMBL-EBI and OMSF — launched with up to £8M from the UK Department for Science, Innovation and Technology (announced at London Tech Week, June 2025; htworld.co.uk), with additional funding from the Gates Foundation announced August 18, 2026 (openbind.uk). Its stated goal: more than 500,000 experimentally validated protein–ligand structures and measurements over five years (htworld.co.uk) — against a public record that its own first paper describes as scarce. That first release, in May 2026, delivered 925 crystallographic binding events from 699 compounds on an enteroviral 2A protease, with affinity measured for 601 compounds (bioRxiv 10.64898/2026.08.27.747600).

The blind challenge announced on October 14, 2026 is the consortium's move from dataset to judge (blopig.com, openbind.uk). The specs, per the organizers: predict protein–ligand complexes for 400 compounds — fragments to lead-like molecules — against the Zika virus NS2B-NS3 protease; an interim submission on November 11, 2026; final submissions due December 16, 2026 at 23:59 UTC; run in partnership with OpenADMET and the ASAP Discovery Consortium. The pass bar is stricter than the field's habit: a pose counts as correct at RMSD ≤ 1.5 Å — tighter than the common 2 Å convention — and must also clear lDDT-PLI > 0.8 and PoseBusters validity checks (blopig.com). The compounds' experimental structures will be withheld until after the close; that withholding is the entire point.

One number from OpenBind's own benchmark explains why this judge is needed. When the consortium evaluated current methods on its first dataset, a molecular-weight baseline achieved the highest Spearman correlation for affinity ranking — a trivial heuristic beat the structure-based models at ordering binders (bioRxiv 10.64898/2026.08.27.747600). A field where molecular weight outranks the models is a field that needs a blind scoreboard, not more self-reported wins.

Why does a quantum verification ledger care about a drug-discovery challenge?

Because quantum computing is missing this institution entirely, and the gap is measurable. As of October 15, 2026, there is no public blind challenge for quantum-advantage claims: no external body holds back problem instances, collects quantum and classical solutions under preset rules, and publishes a ranking. What exists instead maps onto the lower rungs of the ladder: vendor self-reported metrics (Quantum Volume vs Algorithmic Qubits), one pre-registered benchmark suite (QOBLIB, rung 2), and one external auditor — DARPA's QBI, rung 3 — whose IV&V results are explicitly not public (our explainer, darpa.mil). The verdict signal travels without the data. That missing fourth rung is the vacuum Rosetta's ledger exists to document: until quantum has a CASP, every advantage claim is scored by someone with a stake in the outcome.

The judge ladder, with one dated example per rung:

Where do Rosetta's own seals sit on this ladder?

On the lowest rung, and the ledger prints it. Rosetta's two PREDICTION seals freeze their scoring rules and sha256 hashes before the outcomes are knowable, and both commit to scoring only against public registries — the Q-Day bet against measured hardware milestones (#73), the WEE1 and PPP2R1A site commitments against the public structural record with a HIT/MISS/UNSCORABLE rule and a 2029-12-31 deadline (#79). That is a real constraint: the rules cannot quietly move after the fact, which is the failure mode that made 9 of 15 retrospective commitments lose significance under a matched null in our own earlier work. But a frozen rule does not change who reads the scoreboard. The scorer is still the house, and a reader should weight these seals accordingly — exactly as this ladder says: above an unsealed claim, below a pre-registered benchmark, far below a blind challenge. Validating against data that did not exist is the direction of travel; an external judge is the destination.

What we don't know

• How OpenBind's results will read. One challenge, one target. Zika NS2B-NS3 is a single protease; whatever wins in December will be evidence about this pocket, not a verdict on the field. We do not know how many teams will enter, and the sources we accessed do not state when or in what form results will be published.
• Whether the 1.5 Å bar changes behavior. A stricter cutoff plus PoseBusters checks is designed to filter physically implausible poses; whether leading cofolding models clear it on unseen ligands will only be known after the close (blopig.com states the thresholds, not the outcome).
• Whether quantum gets a rung-4 judge. DARPA's QBI is the closest existing structure and its results are private by design (darpa.mil). We know of no announced public blind challenge for quantum-advantage claims as of this writing.
• How our own seals will score. RQ-PRED-ALLO-001 is unscorable until the public structural record moves or the 2029-12-31 deadline passes; the Q-Day seal reviews semiannually. Both could end in MISS — that is what makes them predictions.

Sources

• Imrie, F. — «The First OpenBind Blind Challenge», OPIG / BLOPIG blog (October 2026) — blopig.com
• OpenBind — official site and news: challenge launch, Gates Foundation funding, release timeline (accessed 2026-10-15) — openbind.uk
• OpenBind consortium — «The first OpenBind release: an open experimental structure–affinity dataset and benchmark», bioRxiv 10.64898/2026.08.27.747600 (2026) — biorxiv.org
• Kryshtafovych, A. — «CASP16 introduction», predictioncenter.org, Day-1 presentation (2025) — predictioncenter.org
• Jumper, J. et al. — «Highly accurate protein structure prediction with AlphaFold», Nature 596, 583–589 (2021) — nature.com
• DeepMind — «AlphaFold: a solution to a 50-year-old grand challenge in biology» — median GDT 92.4 at CASP14 (2020) — deepmind.google
• Health Tech World — «UK launches £8m project to build largest drug–protein interaction dataset» (June 2025) — htworld.co.uk
• Rosetta Quantum — related ledger entries: #79 sealed hypotheses · #73 Q-Day · #66 prospective validation · #43 QOBLIB · #80 DARPA QBI · #23 the missing ledger · #65 matched nulls

Rosetta Quantum publishes verdicts with raw, reproducible data. This is educational content, not a product claim. Quantum, independently measured.