What does it mean to validate a model with data that did not exist when it was frozen?
Prospective validation means scoring a model against data that did not exist when the model was frozen. It is the only evaluation a training-set number cannot flatter, and it is expensive: Rosetta Quantum froze an allosteric-pocket detector on September 7, 2026 with its code hashed byte-for-byte, and four weekly runs later the prospective scoreboard stands at five cases — 3 of 5 under the scoring rule in force, 2 of 5 under a stricter reading of the same evidence. Both intervals still contain our own retrospective rate of 32.6%, so after a month the honest answer is that the number cannot yet tell skill from a coin flip. Rosetta publishes it weekly anyway, including the week the run failed to start.
Status as of: October 2, 2026.
What does prospective validation actually mean?
It means the test data did not exist when the model was frozen. A model is pinned — weights, code, and a hash of both — at a declared timestamp; afterwards, new data arrives from a source nobody involved controls, and the model is scored on it without being touched. The ordering is the whole mechanism: because the answers were not in the world at freeze time, no amount of accidental or deliberate tuning could have reached them. Everything else — held-out splits, cross-validation, a test set locked in a drawer — asks you to trust that a boundary was respected. Prospective scoring removes the need for that trust and replaces it with a timestamp.
The design is not new, and the honest precedent is worth naming. CASP, the Critical Assessment of Structure Prediction, has run this way since CASP1 in 1994: organizers post sequences whose structures are not yet public, predictions are collected, and, in the organizers' words, models are "evaluated as the experimental coordinates become available" (Prediction Center). CASP16 in 2024 collected more than 80,000 models from nearly 100 groups across 300 targets. The reason the field built that machinery is the reason Rosetta Quantum uses it: in structure and function prediction, a retrospective number has no credibility left.
Why does a training-set number always flatter the model?
Because the ways a number can be inflated are numerous, well catalogued, and mostly invisible in the published result. The reference survey is Kapoor and Narayanan, who looked across fields that had adopted machine-learning methods and found "17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions," and who organize the problem into a taxonomy of eight distinct types of leakage (Patterns 4(9), 100804, August 2023). Their own reproducibility study is the part worth sitting with: in civil-war prediction, where complex models were believed to vastly outperform traditional statistics, correcting the errors left the complex models performing no better than decades-old logistic regression.
Note what that list does and does not cover. Several of the eight types are fixed outright by freezing a model before the data exists — contamination between training and test sets cannot happen when the test set is not yet written. Others are untouched by it. A model can be free of leakage and still be right for the wrong reason, which is the failure we documented in our own house: our detector's predicted pockets sat at the 11th percentile of distance to the active site while real allosteric sites sit at the 28th, a separation significant at p below 1e-5, which means a pocket could look correct largely by being close to the catalytic center (our peripheral-bias report). No release schedule fixes that. A distance-matched null does.
What does freezing a model with a hash actually buy?
It buys the ability to prove, later, that the thing scored this week is the thing that was frozen. Rosetta Quantum froze the detector on September 7, 2026: 86 proteins, 10 variables, no annotations, with the weights and a manifest of code checksums reduced to a single hash beginning 08ad337d. Each weekly run re-materializes the code and recomputes that hash before any science runs, and if it does not match byte-for-byte, the run stops.
That gate is not decorative. On September 14 it fired: one file arrived one byte longer than the frozen original, the hash did not reproduce, and the run aborted before scoring anything — a defect later localized to a single space inserted at offset 20377 of the transport copy. The scoreboard was held where it was, the failure was published as that week's report, and no prediction was offered for a model that could not be shown to be the frozen one. This is the discipline our earlier post on radical reproducibility argued for, applied to ourselves on a week when it cost us a data point.
What has our own prospective scoreboard measured so far?
Five cases in four weekly runs, with a cumulative hit rate that cannot be stated as one number. The table below is the complete record, week by week, with the sealed run-report behind each row.
| Week | Release window | Cases scored | Result | Source |
|---|---|---|---|---|
| W37 · 2026-09-07 | freeze day, no window | 0 prospective, 1 blind retrospective | prospective opens 0/0; blind retro 0/1 | run-report W37, seal hash 08ad337d |
| W38 · 2026-09-14 | 7 days, not harvested | 0 — run aborted at step 0 | scoreboard held at 0/0; failure published | run-report W38 |
| W39 · 2026-09-21 | 14 days, recovering W38 | 4 | 2/4, Wilson 95% interval 15–85% | run-report W39 |
| W40 · 2026-09-28 | 7 days, 190 entries released | 1 | 3/5 or 2/5, depending on which ligand is the answer | run-report W40 |
| Cumulative · 2026-10-02 | 4 weekly runs | 5 | every reading's interval still contains 32.6% | PROSPECTIVE-2026-W40.json |
Two things in that table deserve to be read slowly. The first is the cost of the design. In the week of September 28, the public archive released 190 entries in the window; exactly one survived the filters and reached scoring. Prospective validation is thin by construction, and the reason is arithmetic: the Protein Data Bank is large and growing — 250,441 entries as of March 6, 2026, with roughly 275 new structures released per week on average in 2022 (wwPDB) — but the subset that is a non-model organism excluded, has a small-molecule ligand, carries a UniProt accession, and has an annotated active site with the ligand genuinely distal is a small fraction of it.
The second is that the W40 case has no single correct score. The entry carried three ligands and the ground truth is not unique: the predicted pocket overlaps the heme A site in four residues, and misses the Cu-A centre and the cyanate site completely. Under the rule in force the case counts as a hit and the running total is 3 of 5; read with the heme treated as the catalytic prosthetic group it is, the answer becomes the cyanate site and the total is 2 of 5. Both readings were published side by side, neither was declared official, and the registry is append-only, so the earlier weeks were not rescored. The lesson is about rules, not about luck: a scoring rule that can be argued about after the fact is a rule that was not specified tightly enough in advance.
Why publish a scoreboard that cannot conclude anything yet?
Because the alternative is to publish it only once it says something flattering, and that is the behaviour the whole design exists to prevent. The current numbers do not distinguish anything: 3 of 5 carries a 95% Wilson interval of roughly 23–88%, 2 of 5 roughly 12–77%, and both of those brackets contain our own retrospective leave-one-protein-out rate of 32.6%. A five-case score is compatible with a detector that is genuinely better than its retrospective estimate and with one that is worse. There is no reading of n=5 that supports a claim, and Rosetta Quantum will not manufacture one by waiting for a good week.
Publishing the empty and the ambiguous weeks is also what makes the eventual non-empty week worth anything. A scoreboard that only appears when the news is good is a selection effect with a logo on it — which is exactly the pattern our post on how to tell whether an advantage claim is credible teaches readers to discount. The commitment is therefore structural and runs forward: the scoring will continue weekly on whatever the archive releases, each week will be sealed whether it produces a case or not, and the interval will narrow only as cases accumulate. We will not announce a result until the interval excludes the retrospective rate.
What does a prospective design still fail to fix?
Four things, and they are worth naming precisely so that nobody reads a timestamp as a guarantee of truth.
| Failure mode | What it does to the number | Fixed by prospective scoring? | Source |
|---|---|---|---|
| Train/test contamination | inflates reported accuracy; one of eight catalogued leakage types | yes — the data did not exist at freeze time | Kapoor & Narayanan, Patterns 4(9), 100804 |
| Tuning after seeing the test set | turns model selection into fitting the benchmark | yes, provided the model is hashed before the window opens | same survey, taxonomy of 8 leakage types |
| Confounded features | a detector can score well by predicting pockets near the active site | no — needs a distance-matched null | our peripheral-bias report: percentile 11 vs 28, p < 1e-5 |
| Small-sample inference | 5 cases cannot separate skill from chance | no — only more weeks narrow the interval | Wilson, JASA 22(158), 209–212, 1927 |
| Ambiguous ground truth | one entry, three ligands, two defensible scores | no — the rule has to be fixed in advance | run-report W40 |
The honest summary of our own position is that the freeze and the hash close the first two rows, our distance-matched null addresses the third only because we built it after finding the bias ourselves, and the last two remain open. That is the state of the method at Rosetta Quantum on the date at the top of this page.
What we don't know
We do not know whether this detector is better than chance; five cases cannot answer that, and we will not claim otherwise until the interval says so. We do not know how many weeks the interval will need to exclude 32.6% — that depends on how many eligible structures the archive happens to release, which is outside our control and which the four weeks so far suggest will be slow. We do not know whether the stricter reading of the W40 case or the rule in force is the better rule; both are published and the choice has been deferred rather than hidden. And we do not know that our filters are unbiased: they were written by the same house that wrote the detector, which is a conflict a prospective timestamp does not resolve. A negative from this detector does not authorize anyone to stop looking at a site — the detector's own retrospective hit rate is roughly a third, and a model that is wrong two times in three cannot rule anything out.
Sources
- Prediction Center — CASP16, blind assessment of structure prediction (first experiment 1994)
- Kapoor & Narayanan — Leakage and the reproducibility crisis in machine-learning-based science (Patterns 4(9), 100804, Aug-2023)
- wwPDB — PDB archive reaches 250,441 entries (6-Mar-2026)
- Wilson, E. B. — Probable inference, the law of succession, and statistical inference (JASA 22(158), 209–212, 1927)
- Rosetta Quantum — prospective run-report W37 (freeze, scoreboard opens)
- Rosetta Quantum — prospective run-report W38 (run aborted at step 0)
- Rosetta Quantum — prospective run-report W39 (first measurement, 2/4)
- Rosetta Quantum — prospective run-report W40 (one case, two readings)
- Rosetta Quantum — Why can a detector look good just by being close to the active site?
- Rosetta Quantum — What does radical reproducibility mean in benchmarking?
- Rosetta Quantum — Why does a weak classical baseline ruin a benchmark?
- Rosetta Quantum — How to tell if an advantage claim is credible
- Rosetta Quantum — How to read a quantum arXiv paper without being a physicist
- Rosetta Quantum — What is an advantaged solve?
Rosetta Quantum publishes verdicts with raw, reproducible data. This is educational content, not a product claim.