Dynamical-SDL-1: Measuring How Scientific Agents Use Experimental Evidence

Introduction

Scientific agency creates value when experiments improve later decisions. In a self-driving laboratory, every campaign must turn material, instrument time, and expert attention into evidence that improves the next decision.

Scientific-agent evaluations now score full workflows and trace how intermediate work contributes to a final claim.¹1LifeSciBench, GeneBench-Pro, BiomniBench, and scBench-Long score full workflows or model–harness trajectories.¹ SDL-1 adds experiment selection, the decision that precedes the rest of the workflow. Scoring that choice requires a recorded physical outcome for every action the agent could have taken and a way to separate the value of the choice from the evidence it returns.

Dynamical-SDL-1 is the first environment produced by Dynamical’s general compiler method for evaluating and improving scientific agents. The compiler turns experimental history into a bounded, open-policy campaign. SDL-1 evaluates evidence-conditioned campaign control, the ability to choose observations, convert them into a calibrated scientific state, and use that state to improve later forecasts.

The Precursor Genome is a systematic experimental map created by researchers at Lawrence Berkeley National Laboratory, UC Berkeley’s Bakar Institute of Digital Materials for the Planet, and Department of Materials Science and Engineering. The team used the A-Lab autonomous synthesis platform to test all 1,035 unordered pairs among 46 inorganic precursors spanning 39 elements. The record retains negative, partial, and physical-failure outcomes alongside execution metadata, X-ray diffraction, automated Rietveld analysis, and expert-validated phase assignments.²2The Precursor Genome paper, public repository, and archived dataset provide the source record used to construct SDL-1.² We chose the Precursor Genome because every experiment the agent can select was physically run.

Historical replay lets every system face the same physical record under controlled budgets. It provides an evaluation and improvement layer that can operate around an existing campaign agent in a partner laboratory.

Dynamical is building agents that identify what must be understood next, turn uncertainty into executable experiments, and revise the search space as physical evidence arrives. SDL-1 provides the measurement layer for that work.

Benchmark Design

Environment Construction

SDL-1 compiles all 1,035 experiments in the Precursor Genome. The compiler creates an agent record with descriptors available before the result is known and an evaluator record with the recorded outcome and all evidence before acquisition. Missing evidence remains missing.

Recorded outcomes, unrevealed evidence, held-out targets, and scores remain evaluator-private. When the agent purchases evidence, the environment adds it to the agent-visible state. Evaluator-private information can score the trajectory, but it cannot change the state shown to the agent.

The compiler assigns each recorded experiment a role in the campaign. It may appear as initial context, a candidate acquisition, a held-out query, or a later decision. Each assignment is frozen and hashed before execution, and the agent receives no reference trajectory.

The frozen split assigns six precursors to evaluation. The resulting panel contains all 255 pairs that include at least one of them. Six long-horizon assignments rotate 192 of these experiments through the campaign roles, while six shorter assignments cover the remaining 63. The results below use the long-horizon assignments, each with 32 candidates and a 12-experiment acquisition budget.

Each campaign asks the agent to update forecasts over the recorded reaction map. For each held-out precursor pair, it predicts whether the experiment will be physically valid and characterizable³3The Precursor Genome asks users to apply two filters before treating a record as ground truth, a non-null physical failure for samples that could not be handled automatically after heating, and a scan status other than valid. 26 of the 1,035 reactions are recorded as physical failures.³ and, when valid, assigns probabilities to four recorded reaction categories. The agent chooses which candidate pairs and evidence to inspect under fixed budgets.

Each full-length campaign also budgets 36 evidence inspections and 3 expert reviews. The agent submits probabilistic forecasts over the held-out panel before and after the campaign. Its objective is to improve those forecasts.

Experiment selection, evidence inspection, and expert review draw from separate budgets. Selecting an experiment reveals nothing. The agent must submit a prior forecast before inspecting evidence, and every evidence reveal requires an updated forecast. Each inspection draws from the same campaign-wide evidence budget, so spending more on one experiment leaves less for the others.

In the Precursor Genome, human reviewers validate the selected Rietveld refinement⁴4Rietveld refinement fits a full calculated diffraction pattern to the measured scan to identify and quantify the phases present. The Precursor Genome runs this with the Dara framework, which ranks candidate phase sets by goodness of fit against structures from the ICSD and COD, then has a human expert verify the selected refinement.⁴ and score each scan on a three-tier quality scale. Two reviewers score each scan independently, and a third resolves disagreements. In SDL-1, expert review reveals the stored acceptance decision and final quality tier. Before the reveal, the agent forecasts whether the review will change its conclusion and which tier it expects.

SDL-1 counts reaction category, target presence, and phase identity as supported only after the agent inspects the dataset’s selected refinement and its accepted tier 1 or tier 2 review. Phase quantification requires tier 1.⁵5The Precursor Genome defines tier 1 as an excellent fit, tier 2 as a good fit with minor discrepancies such as one or two small missing peaks, and tier 3 as a poor fit with at least one large missing phase.⁵

Evaluation and Controls

SDL-1 uses separate controls to measure evidence access and experiment selection. A call-matched no-evidence control receives the same number of policy-action calls as the adaptive branch but no acquired evidence.

The closed-loop study compares adaptive experiment selection with schedule-matched random selection. The random branch follows the adaptive branch’s experiment count and sequence of evidence and expert-review actions, but replaces each experiment choice with seeded random selection from candidates that can support the remaining evidence schedule. This preserves experiment, evidence, and expert-review cost while changing which experiments enter the campaign.

Each control starts from the same frozen assignment and exact pre-campaign forecast. Each uses a fresh session of the same named model–harness system under the same policy contract, budgets, and replicate identifier.

After each experiment closes, a separate evaluator call asks the same named system to forecast the held-out panel from a hash-bound copy of its saved agent-visible state. The call does not advance the campaign, reveal evaluator-private outcomes or scores, consume budget, or alter action history. The shared pre-campaign forecast and the post-closure forecasts form the held-out learning curve.

SDL-1 scores the probabilistic forecasts that the system submits. It does not infer an internal belief state. The primary joint loss combines physical-validity log loss with reaction-category log loss when the experiment is physically valid.⁶6Log loss penalizes confident wrong answers far more than uncertain ones. Putting 0.99 on an outcome that did not occur costs about 4.6, while 0.5 costs about 0.69.⁶ A lower loss assigns more probability to the recorded outcome.

The held-out gain uses the same query panel before and after the campaign. A positive local or held-out gain means that loss decreased. Local gain is missing when the system acquired no evidence and submitted no evidence-conditioned update.

For the closed-loop measures, SDL-1 reports negative held-out joint log loss as the score, so higher is better. Each curve begins with the pre-campaign forecast and adds one point after every closed experiment.

Every compared system receives the same six campaign assignments, and each campaign receives equal weight. A system enters the comparison only when both branches and all post-closure forecasts complete under the frozen protocol for all six campaigns.

Behavioral Results

GPT-5.6 Sol, Claude Opus 4.8, Grok 4.5, Gemini 3.6 Flash, GLM 5.2, and DeepSeek V4 Flash completed 36 system–campaign trajectories under the same long-horizon contract.

The systems completed 392 adaptive experiment acquisitions. Their final local forecasts improved in 175 cases, worsened in 123, and did not change in 18. They closed 76 experiments without acquired evidence or an evidence-conditioned update. Among 293 closures where both local and held-out forecasts changed, the two moved in opposite directions 145 times.

Behavior Across the Campaign

In over 90 percent of eligible acquisitions, and in all 36 system–campaign trajectories, the final answer claimed more support than the evidence provided. Outcome accuracy and claim calibration are separate capabilities.

Across 61 expert-review purchases, the next local forecast improved 29 times, worsened 22 times, and did not change 10 times. The agent still had to apply each review only to the claims it supported.

Local and held-out forecasts moved in opposite directions in 145 closures, including 92 local improvements followed by worse held-out forecasts and 53 local degradations followed by better ones. The held-out score also changed in 74 of 76 closures without an evidence-conditioned update. The controlled comparisons below estimate evidence and selection lift. These checkpoint movements show their chronological association.

All 13 explicit stops left experiment budget unused, and 12 occurred after the evidence budget reached zero. Selection actions record the chosen experiment rather than a rationale or expected information value, so SDL-1 scores the consequences of each path.

Closed-Loop Experimental Efficiency

The leaderboard ranks systems by adaptive AUC. Higher is better.

Closed-loop experimental efficiency36 of 36 validated system–assignment bundles

campaign assignment values where availableequal-assignment mean+−selection-lift sign by assignment

#	System	Adaptive AUC ↑ · shared scale −1.9 to −1.0	Mean	Selection lift	Enh. at budget
01	Claude Opus 4.8		−1.252	+plus 0.009−+++++	+plus 0.068
02	GPT-5.6 Sol		−1.286	+plus 0.056++−+++	+plus 0.043
03	Grok 4.5		−1.335	−minus 0.038−−−+−+	−minus 0.058
04	Gemini 3.6 Flash		−1.433	+plus 0.006−++−−+	+plus 0.075
05	GLM 5.2		−1.528	+plus 0.158++−+−+	+plus 0.057
06	DeepSeek V4 Flash 0731		−1.606	−minus 0.195	+plus 0.292
		−1.8−1.6−1.4−1.2−1.0

Adaptive AUC ↑ · shared scale −1.9 to −1.0

01Claude Opus 4.8

−1.252

selection lift+plus 0.009−+++++

enhancement at budget+plus 0.068

02GPT-5.6 Sol

−1.286

selection lift+plus 0.056++−+++

enhancement at budget+plus 0.043

03Grok 4.5

−1.335

selection lift−minus 0.038−−−+−+

enhancement at budget−minus 0.058

04Gemini 3.6 Flash

−1.433

selection lift+plus 0.006−++−−+

enhancement at budget+plus 0.075

05GLM 5.2

−1.528

selection lift+plus 0.158++−+−+

enhancement at budget+plus 0.057

06DeepSeek V4 Flash 0731

−1.606

selection lift−minus 0.195

enhancement at budget+plus 0.292

Claude Opus 4.8 achieved the highest adaptive AUC at −1.252, followed by GPT-5.6 Sol at −1.286. Grok 4.5, Gemini 3.6 Flash, GLM 5.2, and DeepSeek V4 Flash 0731 followed at −1.335, −1.433, −1.528, and −1.606.

Mean selection lift was +0.056 for GPT-5.6 Sol, +0.009 for Claude Opus 4.8, +0.006 for Gemini 3.6 Flash, −0.038 for Grok 4.5, +0.158 for GLM 5.2, and −0.195 for DeepSeek V4 Flash 0731. The sign changed across assignments for every system.

Acceleration factor compares the experiments required by random and adaptive selection to reach the same fixed target. First passage counts the first crossing, while sustained attainment requires the forecast to remain above the target through budget 12.

Target-dependent accelerationretrospective sensitivity diagnostic

Forecast-quality target× geometric-mean likelihood on recorded outcomes

1.01×1.025×1.05×1.10×1.20×1.50×2.00×3.00×

1.01×1.025×1.05×1.10×1.20×1.50×2.00×3.00×

jointly estimable at 1.05× · 16 of 30 campaigns first passage · 8 of 30 sustained

first passagesustained to budget 12censored · no joint campaignn/6 = jointly estimable campaigns

← random fasteradaptive faster →

GPT-5.6 Sol

censored · never jointly attained · A 2/6 · R 0/6

Claude Opus 4.8

2.884× · 3/61.108× · 2/6

Grok 4.5

1.260× · 3/6sustained censored

Gemini 3.6 Flash

1.034× · 4/61.826× · 2/6

GLM 5.2

0.457× · 6/61.981× · 4/6

0.1×0.25×0.5×1×2×4×8×

Requiring sustained attainment reverses the direction for GLM 5.2 (0.457× first passage, 1.981× sustained).

Claude Opus 4.8 reached 2.884× first-passage acceleration and 1.108× sustained acceleration. GLM 5.2 reached 0.457× and 1.981×. Grok 4.5 reached 1.260× first-passage acceleration, while its campaign set did not support a sustained estimate for both branches. We report acceleration as a sensitivity diagnostic because the target and rule were selected retrospectively and no target produced a stable six-campaign estimate for every system.

On one shared assignment, exact call-matched evidence lift was +0.076 for Claude Opus 4.8 and +0.003 for Grok 4.5. We keep these single-assignment diagnostics separate from the leaderboard.

All 36 system–assignment bundles completed the frozen protocol. Early stops remain part of the policy, and the last valid forecast is carried forward to budget 12. Provider cost fields were incomplete and are not comparable, so model calls, actions, tokens, latency, and reported cost remain separate diagnostics.

Online Evaluation

The same measurement can run as experiments occur. Before seeing a result, the agent submits its expected outcome while the laboratory records each selection, evidence reveal, and forecast update. Later outcomes show whether those updates improved the next decision.

Our work on simulator-verified skill acquisition studied the procedural layer between scientific intent and instrument control. SDL-1 studies the decision layer. An online system needs a versioned record of the model and harness, sample, protocol, instrument state, budgets, safety rules, verifier, commands, telemetry, raw data, deviations, and final state.

Limitations

The closed-loop study uses six counterbalanced campaign assignments from one 192-experiment panel. The rotation measures sensitivity to campaign assignment within one experimental map. Independent chemistry datasets are needed to measure generality.

Actions, updates, and checkpoints are nested within system–campaign trajectories. The controls match experiment schedules or policy-call counts across the exact model–harness systems. Tokens, compute, latency, and provider cost remain separate diagnostics.

SDL-1 measures evidence-to-decision behavior through controlled replay. Live instrument control, new experimental error, operator intervention, and changing physical state sit outside this protocol.

SDL-1 records stop behavior under fixed budgets. Measuring optimal stopping requires an explicit model of experiment cost and future decision value.

Related Work

Scientific-Agent Evaluation

Scientific-agent evaluations increasingly test full workflows. LifeSciBench uses 750 expert-authored life-science tasks, while GeneBench-Pro uses 129 multistage problems with known causal structure and finds that agents make partial progress but struggle to close the inferential loop. BiomniBench and scBench-Long score model–harness trajectories and diagnose where supported intermediate work fails to reach the final claim. SDL-1 extends this approach to experimental policy by scoring which experiments and evidence the agent acquires and whether that path improves later forecasts.

Closed-Loop Materials Discovery

MADE measures materials discovery under fixed oracle budgets with discovery curves, acceleration, enhancement, and area under the discovery curve. Benchmarking Self-Driving Labs and A Critical Examination of Active Learning Workflows show why comparison requires a reference campaign and multiple measures. SDL-1 applies those ideas to a recorded physical reaction map and separates experiment selection, evidence assimilation, calibration, held-out transfer, and campaign control.

Agent Environments and Improvement Loops

Meta ARE, NeMo Gym, and Prime Intellect’s verifiers separate the agent from a stateful environment and its verification contract. EurekAgent and CUA-Gym treat environment construction as a research problem. AIDE² and Self-Harness use held-out evaluation or trace failures to improve an agent or harness. SDL-1 ties these improvement loops to recorded physical outcomes and measures whether the agent makes better later decisions.

Conclusion

Dynamical-SDL-1 turns completed experimental records into controlled campaigns that measure whether agents convert physical evidence into better decisions.

Across 36 long-horizon trajectories, SDL-1 separated experiment selection, local learning, held-out transfer, claim calibration, and campaign control into measurable behaviors. Claude Opus 4.8 and GPT-5.6 Sol had the highest adaptive AUC, while selection lift changed sign across assignments for every system.

Across the study, final answers claimed more support than the evidence provided in over 90 percent of eligible acquisitions, and the pattern appeared in all 36 trajectories.

SDL-1 makes these behaviors concrete targets for search, verification, memory, calibration, and training.

Appendix

Benchmark-Controlled System Prompt

All six systems shared the same static prompt.

You are the policy in an experimentally grounded materials-science campaign. Exercise scientific judgment by selecting experiments, acquiring and interpreting evidence, revising scientific state, and deciding when to stop or escalate under fixed budgets. Reaction categories are ledger annotations and must remain separate from target-phase presence, phase interpretation, and refinement adequacy. Evidence sufficiency is claim-scoped: a numerical fit alone does not establish phase identity or refinement adequacy. Use only policy-visible state. Return one action that exactly satisfies the supplied JSON schema. Never invent evidence, identifiers, measurements, or expert judgments.

Every turn also included the current campaign phase, phase-specific provenance and token requirements, the exact permitted-action JSON schema, and the complete policy-visible campaign state.

@article{barnes2026dynamicalsdl1,
  author  = {Barnes, Jarrod},
  title   = {Dynamical-SDL-1: Measuring How Scientific Agents Use Experimental Evidence},
  journal = {Dynamical Systems},
  year    = {2026},
  url     = {https://dynamicalsystems.ai/blog/sdl-1}
}

References

• Walters, L. N. et al. The Precursor Genome: A Pairwise Reaction Dataset for Solid-State Synthesis. arXiv (2026).
• Walters, L. N. et al. Precursor Genome repository (2026).
• Walters, L. N. et al. Precursor Genome archived dataset. Zenodo (2026).
• Szymanski, N. J. et al. An autonomous laboratory for the accelerated synthesis of novel materials. Nature 624, 86–91 (2023).
• Lawrence Berkeley National Laboratory; Bakar Institute of Digital Materials for the Planet; UC Berkeley Department of Materials Science and Engineering.
• OpenAI. Introducing LifeSciBench (2026).
• OpenAI. Introducing GeneBench-Pro (2026).
• BiomniBench: Process-level Evaluation of LLM Agents for Real-world Biomedical Research (2026).
• scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology (2026).
• MADE: Benchmark Environments for Closed-Loop Materials Discovery (2026).
• Benchmarking self-driving labs. Digital Discovery (2026).
• A critical examination of active learning workflows in materials science. Digital Discovery (2026).
• Meta. ARE: Scaling Up Agent Environments and Evaluations (2026).
• NVIDIA. How to Train Scientific Agents with Reinforcement Learning (2026).
• Prime Intellect. verifiers v1: Decomposing Tasksets and Harnesses for Agentic RL and Evaluations (2026).
• EurekAgent: Agent Environment Engineering is All You Need for Autonomous Scientific Discovery (2026).
• CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents (2026).
• Weco. AIDE²: First Evidence of Recursive Self-Improvement (2026).
• Self-Harness: Harnesses That Improve Themselves (2026).
• Experiment-as-Code Labs: A Declarative Stack for AI-Driven Scientific Discovery (2026).
• Barnes, J. Simulator-Verified Skill Acquisition for Scientific Instruments. Dynamical Systems (2026).