Dynamical-SDL-1: Measuring How Scientific Agents Learn from Physical Experiments

Introduction

Scientific agency creates value when experiments improve later decisions. In a self-driving laboratory, each experiment consumes material, instrument time, and expert attention. A long sequence of valid actions can still leave later decisions unchanged or worse.

Dynamical-SDL-1 measures whether physical evidence improves the forecasts a system submits. It is the first environment produced by Dynamical’s general compiler method for evaluating and improving scientific agents.

The compiler turns experimental history into a bounded, open-policy campaign. It preserves each available experiment, controls what the coordinator can see, and retains a recorded physical outcome for every action it can select.

The Precursor Genome is a systematic experimental map created by researchers at Lawrence Berkeley National Laboratory, UC Berkeley’s Bakar Institute of Digital Materials for the Planet, and Department of Materials Science and Engineering. The team used the A-Lab autonomous synthesis platform to test all 1,035 unordered pairs among 46 inorganic precursors spanning 39 elements. The record retains negative, partial, and physical-failure outcomes alongside execution metadata, X-ray diffraction, automated Rietveld analysis, and expert-validated phase assignments.¹1The Precursor Genome paper, public repository, and archived dataset provide the source record used to construct SDL-1.¹

We chose the Precursor Genome because every experiment the coordinator can select was physically run. SDL-1 never has to simulate or invent the result of an available action.

The coordinator sees candidate precursor pairs and a separate held-out query panel. Under fixed experiment, evidence, and expert-review budgets, it decides which experiments to select, which evidence to inspect, how to revise its probabilistic forecasts, and when to stop. No experimental path is prescribed.

SDL-1 evaluates evidence-conditioned campaign control, the ability to choose observations, convert them into a calibrated scientific state, and use that state to improve later forecasts. Schedule-matched random selection tests the value of experiment choice. A call-matched no-evidence control tests the value of acquired evidence.

Local improvement often failed to transfer. Claude Opus 4.8 and GPT-5.6 Sol produced the strongest absolute adaptive learning curves, but selection lift changed sign across campaign assignments for every system. Absolute forecasting and adaptive experiment choice therefore measure different capabilities.

Historical replay provides a controlled starting point. In a partner laboratory, Dynamical is designed to operate as an evaluation and improvement layer around the existing campaign coordinator. The same contract records what the system expected, what it selected, which evidence arrived, how its forecasts changed, and whether those changes improved later decisions.

Dynamical’s longer-term goal is to build agents that identify what must be understood next, turn uncertainty into executable experiments, and revise the search space as physical evidence arrives. Evaluation is the instrumentation required to capture value from increasingly capable scientific agents.

Benchmark Design

Environment Construction

SDL-1 compiles all 1,035 experiments in the Precursor Genome. The compiler creates an agent record with descriptors available before the result is known and an evaluator record with the recorded outcome and all evidence before acquisition. Missing evidence remains missing.

Recorded outcomes, unrevealed evidence, held-out targets, and scores remain evaluator-private. When the coordinator purchases evidence, the environment adds it to the agent-visible state. Evaluator-private information can score the trajectory, but it cannot change the state shown to the coordinator.

The compiler assigns nonoverlapping experiments to initial context, candidate acquisitions, held-out forecast queries, and future decisions. Each assignment is frozen and hashed before execution. The compiler produces 36 assignments. This study uses the six full-length assignments that form a complete counterbalanced panel for long-horizon evaluation. There is no reference trajectory.

Each campaign asks the coordinator to update forecasts over the recorded reaction map. For each held-out precursor pair, it predicts whether the experiment will be physically valid and characterizable and, when valid, assigns probabilities to four recorded reaction categories. The coordinator chooses which candidate pairs and evidence to inspect under fixed budgets. The environment fixes the available record and scoring rule. The coordinator determines the experimental path.

Each full-length campaign starts with 32 candidate experiments and budgets of 12 experiments, 36 evidence inspections, and 3 expert reviews. The coordinator submits probabilistic forecasts over the held-out panel before and after the campaign. Its objective is to improve those forecasts, but the selection action does not record its rationale or expected information value.

Experiment selection, evidence inspection, and expert review draw from separate budgets. Selecting an experiment reveals nothing. The coordinator must submit a prior forecast before inspecting evidence, and every evidence reveal requires an updated forecast. The evidence budget determines how deeply selected experiments can be investigated. Each inspection draws from the same campaign-wide evidence budget, so spending more on one experiment leaves less for the others.

Expert review has its own budget. Before purchasing it, the coordinator must forecast whether the review will change its conclusion and which quality tier it expects. The environment records those forecasts before revealing the expert judgment. This makes the coordinator’s forecast of review impact a scored output.

Evaluation and Controls

SDL-1 uses separate controls to measure evidence access and experiment selection. A call-matched no-evidence control receives the same number of policy-action calls as the adaptive branch but no acquired evidence. It does not match tokens, latency, total inference compute, or cost.

The closed-loop study compares adaptive experiment selection with schedule-matched random selection. The random branch follows the adaptive branch’s experiment count and sequence of evidence and expert-review actions, but replaces each experiment choice with seeded random selection from candidates that can support the remaining evidence schedule. This preserves experiment, evidence, and expert-review cost while changing which experiments enter the campaign.

Each control starts from the same frozen assignment and exact pre-campaign forecast. Each uses a fresh session of the same named model–harness system under the same policy contract, budgets, and replicate identifier.

After each experiment closes, a separate evaluator call asks the same named system to forecast the held-out panel from a hash-bound copy of its saved agent-visible state. The call does not advance the campaign, reveal evaluator-private outcomes or scores, consume budget, or alter action history. The shared pre-campaign forecast and the post-closure forecasts form the held-out learning curve.

SDL-1 scores the probabilistic forecasts that the system submits. It does not infer an internal belief state. The primary joint loss combines physical-validity log loss with reaction-category log loss when the experiment is physically valid. A lower loss assigns more probability to the recorded outcome.

The held-out gain uses the same query panel before and after the campaign. A positive local or held-out gain means that loss decreased. A negative value means that loss increased. A value of zero means that loss did not change. Local gain is missing when the system acquired no evidence and submitted no evidence-conditioned update.

For the closed-loop measures, SDL-1 reports negative held-out joint log loss as the score, so higher is better. Each curve begins with the pre-campaign forecast and adds one point after every closed experiment. Adaptive AUC is the curve area divided by its experiment-count span. A positive selection lift means that adaptive selection produced a higher average held-out score than schedule-matched random selection.

Every compared system receives the same six campaign assignments, and each campaign receives equal weight. A system enters the comparison only when both branches and all post-closure forecasts complete under the frozen protocol for all six campaigns.

Behavioral Results

GPT-5.6 Sol, Claude Opus 4.8, Grok 4.5, Gemini 3.6 Flash, GLM 5.2, and DeepSeek V4 Flash completed 36 system–campaign trajectories under the same long-horizon contract.

The systems completed 392 adaptive experiment acquisitions. Their final local forecasts improved in 175 cases, worsened in 123, and did not change in 18. They closed 76 experiments without acquired evidence or an evidence-conditioned update. Among 293 closures where both local and held-out forecasts changed, the two moved in opposite directions 145 times.

The traces expose four failure mechanisms. Systems can misconvert phase evidence into the scored reaction state, express unsupported confidence, treat an unavailable review as support, or improve an observed experiment while worsening forecasts on unrevealed chemistry.

Behavior Across the Campaign

In over 90 percent of eligible acquisitions, and in all 36 system–campaign trajectories, the final forecast crossed a 0.5 sufficiency threshold on at least one claim axis that the evidence contract still marked unsupported. Outcome accuracy and claim calibration are separate capabilities. A forecast can move toward the recorded result while claiming more support than the acquired evidence provides.

Across 61 expert-review purchases, the next local forecast improved 29 times, worsened 22 times, and did not change 10 times. Expert judgment did not consistently produce a better reaction forecast. The coordinator still had to convert the scope of each review into a calibrated reaction distribution and preserve unavailable review as unknown.

Of the 145 closures where local and held-out forecasts moved in opposite directions, 92 were local improvements followed by worse held-out forecasts and 53 were local degradations followed by better ones. These checkpoint movements are chronological associations. Each checkpoint is a fresh evaluator-side forecast from saved campaign state, and 74 of 76 closures without acquired evidence or an evidence-conditioned update also changed the held-out score. The controlled comparisons in Closed-Loop Experimental Efficiency estimate evidence and selection lift.

The systems also allocated identical starting budgets in different ways. Twelve of the 13 explicit stops occurred after the evidence budget reached zero, and all 13 left experiment budget unused. The selection action records the chosen experiment without its rationale or expected information value, so SDL-1 measures the consequences of each path without inferring why the system chose it.

System-Level Patterns

Across the six campaigns, the systems differed most in forecast stability, evidence allocation, and stopping. GPT-5.6 Sol and Grok 4.5 used evidence aggressively and often stopped when it was exhausted, while more of their measurable local forecasts worsened than improved. Claude Opus 4.8 converted evidence more consistently for observed experiments, but those local gains did not consistently transfer.

Gemini 3.6 Flash completed every campaign without expert review and showed substantial campaign-to-campaign variation in held-out performance. GLM 5.2 often recovered from weak starting forecasts but stopped early in four campaigns, so its gains combine evidence assimilation with baseline repair. DeepSeek V4 Flash completed every campaign, but frequent extreme forecasts produced large log-loss penalties and dominated its closed-loop result.

The closed-loop results test whether these behaviors add up to more useful learning per experiment.

Closed-Loop Experimental Efficiency

The behavioral analysis describes individual state transitions. The closed-loop evaluation asks whether those transitions produced better held-out forecasts over the full experiment budget. The leaderboard below ranks systems by adaptive AUC, the equal-assignment mean normalized area under the adaptive held-out learning curve. Higher scores indicate lower average held-out log loss.

Closed-loop experimental efficiency36 of 36 validated system–assignment bundles

campaign assignment (1–6)equal-assignment mean+−selection-lift sign by assignment

#	System	Adaptive AUC ↑ · top-five zoom −1.9 to −1.0	Mean	Selection lift	Enh. at budget
01	Claude Opus 4.8		−1.252	+plus 0.009−+++++	+plus 0.068
02	GPT-5.6 Sol		−1.286	+plus 0.056++−+++	+plus 0.043
03	Grok 4.5		−1.335	−minus 0.038−−−+−+	−minus 0.058
04	Gemini 3.6 Flash		−1.433	+plus 0.006−++−−+	+plus 0.075
05	GLM 5.2		−1.528	+plus 0.158++−+−+	+plus 0.057
06	DeepSeek V4 Flash	off zoom scale · full range below	−6.747	+plus 0.522+−+−−−	+plus 7.001
		−1.8−1.6−1.4−1.2−1.0

Adaptive AUC ↑ · top-five zoom −1.9 to −1.0

01Claude Opus 4.8

−1.252

selection lift+plus 0.009−+++++

enhancement at budget+plus 0.068

02GPT-5.6 Sol

−1.286

selection lift+plus 0.056++−+++

enhancement at budget+plus 0.043

03Grok 4.5

−1.335

selection lift−minus 0.038−−−+−+

enhancement at budget−minus 0.058

04Gemini 3.6 Flash

−1.433

selection lift+plus 0.006−++−−+

enhancement at budget+plus 0.075

05GLM 5.2

−1.528

selection lift+plus 0.158++−+−+

enhancement at budget+plus 0.057

06DeepSeek V4 Flash

−6.747

off zoom scale · full range below

selection lift+plus 0.522+−+−−−

enhancement at budget+plus 7.001

Full range · adaptive AUC −14 to −1shaded band = top-five zoom above

01Claude Opus 4.8

02GPT-5.6 Sol

03Grok 4.5

04Gemini 3.6 Flash

05GLM 5.2

06DeepSeek V4 Flash

−13−10−7−4−1

DeepSeek V4 Flash’s positive relative lift compares two poorly calibrated trajectories.

Held-Out Learning Curves

Claude Opus 4.8 achieved the highest adaptive AUC at −1.252, followed closely by GPT-5.6 Sol at −1.286. Grok 4.5, Gemini 3.6 Flash, and GLM 5.2 followed at −1.335, −1.433, and −1.528.

DeepSeek V4 Flash was a clear outlier at −6.747. Across its adaptive and random branches, 86 of 144 post-closure forecasts had joint log loss above 5. Its forecasts frequently assigned exact-zero or exact-one probabilities to reaction outcomes, which incur large proper-log-score penalties when wrong.

The rank order remained unchanged in every leave-one-assignment-out recomputation. This is descriptive stability across six counterbalanced assignments from one 192-experiment panel, not replication across independent chemistry datasets.

Experiment-Selection Lift

Selection lift compares adaptive AUC with the schedule-matched random AUC. GPT-5.6 Sol recorded +0.056. Claude Opus 4.8 and Gemini 3.6 Flash were near zero at +0.009 and +0.006. Grok 4.5 recorded −0.038, while GLM 5.2 recorded +0.158.

DeepSeek V4 Flash had a mean selection lift of +0.522 despite the lowest absolute adaptive score. Both branches contained severe log-loss spikes from exact-zero and exact-one reaction probabilities, with greater deterioration in the random branch. Its relative lift therefore compares two poorly calibrated trajectories and does not indicate strong absolute performance.

The campaign-level sign of selection lift changed across assignments for every system. Adaptive experiment choice helped on some assignments and hurt on others. Enhancement at budget compares adaptive and random held-out score after the final budgeted experiment. It was positive for Claude Opus 4.8, GPT-5.6 Sol, Gemini 3.6 Flash, and GLM 5.2, and negative for Grok 4.5. DeepSeek V4 Flash had a large positive relative endpoint difference because its random branch deteriorated further.

Target-Dependent Acceleration

Acceleration factor divides the experiments required by a reference policy by the experiments required by an adaptive policy to reach the same fixed target. A value above one favors adaptive selection; a value below one favors random selection. SDL-1 calculates the ratio only when both branches reach the target on the same campaign. In materials optimization, a discovered material remains available, so best-so-far performance does not regress. SDL-1 measures the coordinator’s current held-out forecast, which can improve and later deteriorate.

Target-dependent accelerationretrospective sensitivity diagnostic

Forecast-quality target× geometric-mean likelihood on recorded outcomes

1.01×1.025×1.05×1.10×1.20×1.50×2.00×3.00×

1.01×1.025×1.05×1.10×1.20×1.50×2.00×3.00×

jointly estimable at 1.05× · 16 of 36 campaigns first passage · 8 of 36 sustained

first passagesustained to budget 12censored · no joint campaignn/6 = jointly estimable campaigns

← random fasteradaptive faster →

GPT-5.6 Sol

censored · never jointly attained · A 2/6 · R 0/6

Claude Opus 4.8

2.884× · 3/61.108× · 2/6

Grok 4.5

1.260× · 3/6sustained censored

Gemini 3.6 Flash

1.034× · 4/61.826× · 2/6

GLM 5.2

0.457× · 6/61.981× · 4/6

DeepSeek V4 Flash

censored · never jointly attained · A 1/6 · R 2/6

0.1×0.25×0.5×1×2×4×8×

Requiring sustained attainment reverses the direction for GLM 5.2 (0.457× first passage, 1.981× sustained).

We tested this distinction retrospectively. At a target equal to a 5% increase in the geometric-mean probability assigned to recorded outcomes, Claude Opus 4.8 first reached the target in three adaptive campaigns and four random campaigns. Both branches reached it in three of the same campaigns. Across those three campaigns, the geometric-mean acceleration factor was 2.884×, so random selection required 2.884× as many experiments as adaptive selection to first reach the target. When the forecast had to remain above the target through the full budget, both branches qualified on only two campaigns and the acceleration factor fell to 1.108×.

Grok 4.5 first reached the target in three adaptive campaigns and three random campaigns. The same three campaigns supported both branches, with an acceleration factor of 1.260×. Under the sustained rule, the adaptive branch qualified on one campaign and the random branch qualified on two, but they never qualified on the same campaign. A sustained acceleration factor was therefore unavailable. GLM 5.2 shows why the crossing rule matters. Its first-passage acceleration was 0.457× across six jointly attained campaigns, favoring random selection, while its sustained acceleration was 1.981× across four jointly sustained campaigns, favoring adaptive selection.

No target produced a complete and stable six-campaign acceleration estimate for every system. Because neither the target nor the persistence rule was frozen before execution, acceleration remains a sensitivity diagnostic. Adaptive AUC and enhancement at budget remain the formal metrics.

Evidence Lift

Evidence lift compares the adaptive endpoint gain with the call-matched no-evidence endpoint gain. Exact controls were available for Claude Opus 4.8 and Grok 4.5 on one shared campaign assignment. Claude Opus 4.8 recorded +0.076 and Grok 4.5 recorded +0.003. These are single-assignment diagnostics and do not enter the six-assignment leaderboard.

Inference Cost and Completion

All 36 system–assignment bundles completed the frozen protocol, including the adaptive branch, schedule-matched random branch, and post-closure held-out forecasts. Protocol completion does not mean that each coordinator used the full experiment budget. Early stops remain part of the recorded policy, and the final valid forecast is carried forward to the frozen budget.

Provider-reported monetary cost fields were complete for Gemini 3.6 Flash, partial for Claude Opus 4.8, Grok 4.5, and GLM 5.2, and unavailable for GPT-5.6 Sol and DeepSeek V4 Flash. A reported zero can reflect provider or BYOK accounting and does not establish zero economic cost. These fields are not normalized across providers and do not affect rank. Model calls, benchmark actions, tokens, and latency remain separate operational diagnostics.

Online Evaluation

Moving this measurement online would mean preserving the same record as experiments occur. Before seeing the result of a selected experiment, the coordinator would submit what it expects, while the laboratory would record what the coordinator selects, which evidence it uses, and how its forecast changes as results arrive. Later outcomes would show whether those updates improved the next experimental or program decision.

Our earlier work on simulator-verified skill acquisition showed one approach to the procedural layer between scientific intent and instrument control. In that work, an agent drafted, executed, and repaired instrument procedures in simulation, while an independent verifier controlled admission to a reusable skill library. SDL-1 studies the adjacent decision layer by measuring whether evidence from an experiment improves what the coordinator does next.

Connecting these layers would require a formal record of the experiment as well as the agent. We are exploring a preflight process that would record and validate the model and harness, sample, protocol, instrument state and calibration, local environment, budgets, safety rules, data schema, and verifier before execution. Instrument and orchestration readbacks would establish what was configured and observed, after which execution would add commands, telemetry, raw data, deviations, and final state to the same versioned record.

SDL-1 is Dynamical’s first measurable implementation of the evaluation layer within that loop.

Limitations

The behavioral analysis and closed-loop leaderboard use six counterbalanced campaign assignments drawn from one 192-experiment panel of the recorded reaction map. Each experiment rotates through campaign roles across the assignments. Equal campaign weighting controls role assignment, while the panel still represents one experimental map rather than six independent chemistry datasets.

Experiment acquisitions, evidence reveals, forecast updates, reviews, and held-out checkpoints are nested within each system–campaign trajectory. They expose behavior inside the campaign without adding independent campaign-level replication. Post-closure held-out forecasts are fresh model calls from saved campaign state, so individual checkpoint movements are descriptive. Controlled aggregate comparisons support the evidence and selection claims.

The results apply to each exact model–harness system. Context management, schema handling, retries, recovery behavior, and provider routing can affect the completed trajectory. The leaderboard therefore ranks the named systems under the frozen SDL-1 contract rather than model checkpoints in isolation.

The controls isolate different parts of the campaign. Schedule-matched random selection preserves the adaptive experiment count and evidence-action schedule while changing the selected experiments. The call-matched no-evidence control preserves the number of policy-action calls without revealing experimental evidence. These controls do not match tokens, total inference compute, latency, or provider cost. Completion and inference cost remain separate from forecast performance.

SDL-1 replays recorded experiments. This preserves the physical outcomes and information boundary while omitting live instrument control, new experimental error, laboratory latency, operator intervention, and changes in future physical state. The study measures evidence-to-decision behavior under controlled replay.

SDL-1 has no optimal-stopping oracle. Stop actions show how systems allocate their experiment, evidence, and expert-review budgets. A learning curve can show that a campaign reached a better held-out forecast earlier in its trajectory, but it cannot establish that the system should have stopped at that point. Optimal stopping requires an explicit model of experimental cost and the value of the next decision.

Related Work

Scientific-Agent Evaluation

Recent scientific evaluations increasingly test complete research workflows rather than isolated scientific answers. LifeSciBench covers 750 expert-authored tasks across applied life-science work. Each response is graded against a task-specific expert rubric. GeneBench-Pro focuses on 129 multistage problems in genomics, quantitative biology, and translational biomedicine. Its synthetic data preserve a known causal structure while requiring the agent to make dependent analytical decisions. The study identifies a recurring gap between noticing diagnostic evidence and applying its implications to the analysis.

Other evaluations make the full agent trajectory part of the object being studied. BiomniBench evaluates complete biomedical data-analysis trajectories and reports results for each model–harness system. scBench-Long gives agents raw or near-raw single-cell data without prescribing an analysis method. It uses deterministic final answers as its primary score and trajectory rubrics to diagnose where the analysis failed. Correct intermediate work often failed to produce the supported final scientific claim.

SDL-1 extends this progression from scientific analysis to experimental policy. The agent chooses which physical experiments and evidence to acquire, submits revised probabilistic forecasts as evidence arrives, and applies that experience to later decisions. The evaluation measures the consequences of the selected path.

Closed-Loop Materials Discovery

Closed-loop materials benchmarks measure how efficiently a policy reaches a scientific target under a limited evaluation budget. MADE evaluates complete materials-discovery pipelines in iterative convex-hull discovery campaigns. A policy proposes crystal structures, receives formation-energy feedback from an oracle, and updates its search. Discovery curves, acceleration factor, enhancement factor, and area under the discovery curve compare policies across fixed oracle budgets.

The wider self-driving-laboratory literature provides a common language for this comparison. Benchmarking Self-Driving Labs defines acceleration factor as the reduction in experiments required to reach a target and enhancement factor as the improvement achieved at a fixed experimental budget. It also treats a reference campaign as necessary for measuring learning. A Critical Examination of Active Learning Workflows shows why several measures are needed. Improved model accuracy does not guarantee improved coverage of the regions a campaign targets, and sampling quality has several facets that one metric cannot capture.

Recent systems show that scientific agents can coordinate expensive simulation and physical laboratory work. MASTER uses hierarchical agents to select and execute density-functional-theory workflows for catalyst exploration. OpenAI and Molecule.one connected GPT-5.4 to a high-throughput chemistry laboratory for a three-month medicinal-chemistry campaign. The system proposed research directions, designed experiments, analyzed results, and proposed follow-up work while human chemists controlled which proposals entered the laboratory.

SDL-1 uses closed-loop efficiency as a campaign-level outcome and decomposes it into experiment selection, evidence assimilation, calibration, transfer, and campaign control.

Agent Environments and Improvement Loops

Agent environments are becoming a common interface between evaluation and improvement. Meta ARE separates the agent from a stateful world with selective observations, tools, events, and verifiers. NeMo Gym builds scientific-agent environments from models, resources, and agents, with verification logic that measures task performance. Prime Intellect verifiers v1 separates tasksets, harnesses, and runtimes so one task contract can support evaluation and reinforcement learning.

Recent work also treats environment construction as a research problem. EurekAgent organizes metric-driven scientific research through explicit permissions, artifact management, experimental budgets, isolated evaluation, and human intervention. CUA-Gym generates executable software tasks, initial and target states, and deterministic rewards at scale. SDL-1 grounds environment construction in a completed physical campaign. Its compiler preserves the recorded information boundary and physical outcome of each available experiment.

Trusted environments can then support improvement loops. AIDE² uses held-out evaluation to improve the research agent that runs an inner optimization loop. Self-Harness mines execution traces for model-specific failures, proposes harness changes, and accepts changes after regression testing. Both systems depend on an evaluator that can distinguish real improvement from exploitation of the measurement process.

SDL-1 provides an outcome-grounded target for this process in physical science. Its traces identify where a campaign fails to convert experiments into useful learning. Those failures can guide search, verification, state management, harness changes, and post-training.

Conclusion

Dynamical-SDL-1 shows that completed experimental records can become controlled environments for evaluating how scientific agents use physical evidence over time.

The results separate three capabilities that are often treated as one. A coordinator can improve its forecast for an observed experiment without improving held-out forecasts. It can produce strong absolute forecasts without selecting better experiments than a random policy. It can also gain relative advantage over random selection while remaining poorly calibrated in absolute terms.

At the campaign level, Claude Opus 4.8 and GPT-5.6 Sol produced the strongest absolute held-out learning curves. Adaptive selection did not provide a uniform advantage. Its effect changed with campaign assignment, and a large relative lift could coexist with poor absolute calibration.

Preserving this measurement in live campaigns would let laboratories test whether each experiment improves subsequent decisions and use observed failures as targets for search, verification, memory, calibration, and training.

Appendix

Benchmark-Controlled System Prompt

All six systems shared the same static prompt.

You are the policy in an experimentally grounded materials-science campaign. Exercise scientific judgment by selecting experiments, acquiring and interpreting evidence, revising scientific state, and deciding when to stop or escalate under fixed budgets. Reaction categories are ledger annotations and must remain separate from target-phase presence, phase interpretation, and refinement adequacy. Evidence sufficiency is claim-scoped: a numerical fit alone does not establish phase identity or refinement adequacy. Use only policy-visible state. Return one action that exactly satisfies the supplied JSON schema. Never invent evidence, identifiers, measurements, or expert judgments.

Every turn also included the current campaign phase, phase-specific provenance and token requirements, the exact permitted-action JSON schema, and the complete policy-visible campaign state.

@article{barnes2026dynamicalsdl1,
  author  = {Barnes, Jarrod},
  title   = {Dynamical-SDL-1: Measuring How Scientific Agents Learn from Physical Experiments},
  journal = {Dynamical Systems},
  year    = {2026},
  url     = {https://dynamicalsystems.ai/blog/dynamical-sdl-1}
}

References

1. Walters, L. N. et al. The Precursor Genome: A Pairwise Reaction Dataset for Solid-State Synthesis. arXiv (2026).
2. Walters, L. N. et al. Precursor Genome repository (2026).
3. Walters, L. N. et al. Precursor Genome archived dataset. Zenodo (2026).
4. Szymanski, N. J. et al. An autonomous laboratory for the accelerated synthesis of inorganic materials. Nature 624, 86–91 (2023).
5. Lawrence Berkeley National Laboratory; Bakar Institute of Digital Materials for the Planet; UC Berkeley Department of Materials Science and Engineering.
6. OpenAI. Introducing LifeSciBench (2026).
7. OpenAI. Introducing GeneBench-Pro (2026).
8. BiomniBench: Process-level Evaluation of LLM Agents for Real-world Biomedical Research (2026).
9. scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology (2026).
10. MADE: Benchmark Environments for Closed-Loop Materials Discovery (2026).
11. Benchmarking self-driving labs. Digital Discovery (2026).
12. A critical examination of active learning workflows in materials science. Digital Discovery (2026).
13. Hierarchical Multi-agent Large Language Model Reasoning for Autonomous Heterogeneous Catalyst Discovery. npj Computational Materials (2026).
14. OpenAI and Molecule.one. A near-autonomous AI chemist improves a challenging reaction in medicinal chemistry (2026).
15. Meta. ARE: Scaling Up Agent Environments and Evaluations (2026).
16. NVIDIA. How to Train Scientific Agents with Reinforcement Learning (2026).
17. Prime Intellect. verifiers v1: Decomposing Tasksets and Harnesses for Agentic RL and Evaluations (2026).
18. EurekAgent: Agent Environment Engineering is All You Need for Autonomous Scientific Discovery (2026).
19. CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents (2026).
20. Weco. AIDE²: First Evidence of Recursive Self-Improvement (2026).
21. Self-Harness: Harnesses That Improve Themselves (2026).
22. Experiment-as-Code Labs: A Declarative Stack for AI-Driven Scientific Discovery (2026).
23. Barnes, J. Simulator-Verified Skill Acquisition for Scientific Instruments. Dynamical Systems (2026).