How Scientific Agents Learn from Experiments

The compiler preserves all 1,035 Precursor Genome experiments, separating descriptors available before the result from recorded outcomes and evidence. Each experiment receives a frozen role as initial context, a candidate acquisition, a held-out query, or a later decision. Agents receive no reference trajectory.

Six evaluation precursors define a panel of 255 pairs containing at least one of them. Six long-horizon assignments rotate 192 experiments through the campaign roles; six shorter assignments cover the remaining 63. This report uses the long-horizon assignments.

Each campaign offers 32 candidate experiments and budgets 12 acquisitions, 36 evidence inspections, and 3 expert reviews. Selecting an experiment reveals no evidence. The agent submits a prediction before inspection and updates it after each reveal. All inspections draw from the campaign-wide evidence budget.

For each held-out pair, the agent assigns probabilities to physical validity and characterizability, then to four reaction categories when valid. The source data record 26 physical handling failures among the 1,035 experiments.

Expert review reveals the dataset's stored acceptance decision and scan-quality tier. Two human reviewers originally assessed each scan, with a third resolving disagreements. Before acquiring that review, the agent estimates whether it will change its conclusion and which tier it expects.

The evaluator requires inspection of the selected Rietveld refinement and its accepted tier 1 or tier 2 review to count reaction category, target presence, and phase identity as supported. Phase quantification requires tier 1. Rietveld refinement fits a calculated diffraction pattern to the measured scan. Tier 1 denotes an excellent fit, tier 2 a good fit with minor discrepancies, and tier 3 a poor fit with a large missing phase.

The random-selection control preserves the adaptive campaign's experiment count and sequence of evidence and review actions. It replaces each experiment choice with seeded random selection from candidates that can support the remaining evidence schedule. A fresh session of the same model and harness starts from the identical pre-campaign predictions, assignment, budgets, and replicate identifier.

A separate call-matched control receives the same number of policy-action calls without acquired evidence. After each experiment closes, an evaluator call asks the same system to predict held-out outcomes from its saved visible state. This call leaves the campaign state and budgets unchanged and returns no hidden outcomes or scores to the agent.

The primary joint loss combines log loss for physical validity with reaction-category log loss when valid. Lower loss means the submitted probabilities better match recorded outcomes. Log loss penalizes confident errors strongly. Assigning 99% probability to an outcome that did not occur costs about 4.6, compared with 0.69 at 50%.

A positive gain means prediction quality improved. Local gain is missing when no evidence or evidence-conditioned update was acquired. All systems receive the same six assignments, with equal weight per assignment. Inclusion requires both branches and every post-closure prediction to complete on all six assignments.

The systems completed 392 adaptive experiment acquisitions. Local prediction quality improved in 175, worsened in 123, and stayed unchanged in 18. Another 76 closed without acquired evidence or an evidence-conditioned update.

Among the 293 checkpoints where both local and held-out prediction quality changed, 92 combined a local improvement with worse held-out predictions, and 53 combined a local degradation with better held-out predictions. The held-out score also changed after 74 of the 76 closures without an evidence-conditioned update. These are observations of checkpoint movement. The matched controls estimate the benefit of evidence access and experiment selection.

Across 61 expert-review purchases, the next local prediction improved in 29 cases, worsened in 22, and stayed unchanged in 10. All 13 explicit stops left experiment budget unused, and 12 occurred after the evidence budget reached zero. Selection actions record the chosen experiment, so the evaluation scores the consequences of the choice rather than an explicit estimate of its expected value.

Claude Opus 4.8 achieved the highest adaptive AUC at −1.252, followed by GPT-5.6 Sol at −1.286. Grok 4.5, Gemini 3.6 Flash, GLM 5.2, and DeepSeek V4 Flash 0731 followed at −1.335, −1.433, −1.528, and −1.606.

Mean selection lift was +0.056 for GPT-5.6 Sol, +0.009 for Claude Opus 4.8, +0.006 for Gemini 3.6 Flash, −0.038 for Grok 4.5, +0.158 for GLM 5.2, and −0.195 for DeepSeek V4 Flash 0731. The sign changed across assignments for every system.

Acceleration factor compares the experiments required by random and adaptive selection to reach the same fixed target. First passage counts the first crossing, while sustained attainment requires the prediction score to remain above the target through budget 12.

Target-dependent accelerationretrospective sensitivity diagnostic

Forecast-quality target× geometric-mean likelihood on recorded outcomes

1.01×1.025×1.05×1.10×1.20×1.50×2.00×3.00×

1.01×1.025×1.05×1.10×1.20×1.50×2.00×3.00×

jointly estimable at 1.05× · 16 of 30 campaigns first passage · 8 of 30 sustained

first passagesustained to budget 12censored · no joint campaignn/6 = jointly estimable campaigns

← random fasteradaptive faster →

GPT-5.6 Sol

censored · never jointly attained · A 2/6 · R 0/6

Claude Opus 4.8

2.884× · 3/61.108× · 2/6

Grok 4.5

1.260× · 3/6sustained censored

Gemini 3.6 Flash

1.034× · 4/61.826× · 2/6

GLM 5.2

0.457× · 6/61.981× · 4/6

0.1×0.25×0.5×1×2×4×8×

Requiring sustained attainment reverses the direction for GLM 5.2 (0.457× first passage, 1.981× sustained).

Claude Opus 4.8 reached 2.884× first-passage acceleration and 1.108× sustained acceleration. GLM 5.2 reached 0.457× and 1.981×. Grok 4.5 reached 1.260× first-passage acceleration, while its campaign set did not support a sustained estimate for both branches. We report acceleration as a sensitivity diagnostic because the target and rule were selected retrospectively and no target produced a stable six-campaign estimate for every system.

On one shared assignment, exact call-matched evidence lift was +0.076 for Claude Opus 4.8 and +0.003 for Grok 4.5. These estimates describe that assignment.

All 36 system–assignment bundles completed the frozen protocol. Early stops remain part of the policy, and the last valid prediction is carried forward to budget 12. Provider cost fields were incomplete and are not comparable, so model calls, actions, tokens, latency, and reported cost remain separate diagnostics.

The six assignments rotate one 192-experiment panel. Actions and checkpoints are nested within campaigns, so their counts do not represent independent chemistry datasets.

The protocol evaluates controlled replay. Live instrument control, new experimental error, operator intervention, and changing physical state require additional evaluation. Optimal stopping also requires an explicit model of experiment cost and future decision value.

Scientific-agent evaluations such as LifeSciBench, GeneBench-Pro, BiomniBench, and scBench-Long examine workflows and intermediate reasoning. MADE and the self-driving-lab benchmarking literature compare experimental policies under fixed budgets. SDL-1 connects these questions by evaluating experiment selection and subsequent evidence use against recorded physical outcomes. The references below include the environment and agent-improvement work that informed the design.

All six systems shared the same static prompt.

You are the policy in an experimentally grounded materials-science campaign. Exercise scientific judgment by selecting experiments, acquiring and interpreting evidence, revising scientific state, and deciding when to stop or escalate under fixed budgets. Reaction categories are ledger annotations and must remain separate from target-phase presence, phase interpretation, and refinement adequacy. Evidence sufficiency is claim-scoped: a numerical fit alone does not establish phase identity or refinement adequacy. Use only policy-visible state. Return one action that exactly satisfies the supplied JSON schema. Never invent evidence, identifiers, measurements, or expert judgments.

Every turn also included the current campaign phase, phase-specific provenance and token requirements, the exact permitted-action JSON schema, and the complete policy-visible campaign state.