Measuring How Scientific Agents Learn from Physical Experiments
Dynamical-SDL-1 is an environment compiler for measuring how efficiently agents convert experimental evidence into better decisions.
TL;DR
Dynamical-SDL-1 compiles a real materials lab’s full experimental record into controlled, budgeted campaigns that measure whether an agent uses acquired evidence to make better decisions. It scores four capabilities often bundled as scientific discovery. An agent has to pick which experiments to run, turn each inspection into a correct picture of the outcome, claim only what the evidence supports, and carry that picture to chemistry it has not seen. These capabilities decompose into distinct model behaviors. Local and held-out forecasts moved in opposite directions in 145 of 293 cases. Claude Opus 4.8 and GPT-5.6 Sol had the strongest absolute adaptive learning curves, while selection lift changed sign across assignments for every system.
Introduction
Scientific agency creates value when experiments improve later decisions. In a self-driving laboratory, each experiment consumes material, instrument time, and expert attention. A long sequence of valid actions can still leave later decisions unchanged or worse.
Dynamical-SDL-1 measures whether physical evidence improves the forecasts a system submits. It is the first environment produced by Dynamical’s general compiler method for evaluating and improving scientific agents.
The compiler turns experimental history into a bounded, open-policy campaign. It preserves each available experiment, controls what the agent can see, and retains a recorded physical outcome for every action it can select.
The Precursor Genome is a systematic experimental map created by researchers at Lawrence Berkeley National Laboratory, UC Berkeley’s Bakar Institute of Digital Materials for the Planet, and Department of Materials Science and Engineering. The team used the A-Lab autonomous synthesis platform to test all 1,035 unordered pairs among 46 inorganic precursors spanning 39 elements. The record retains negative, partial, and physical-failure outcomes alongside execution metadata, X-ray diffraction, automated Rietveld analysis, and expert-validated phase assignments.1
We chose the Precursor Genome because every experiment the agent can select was physically run. SDL-1 never has to simulate or invent the result of an available action.
The agent sees candidate precursor pairs and a separate held-out query panel. Under fixed experiment, evidence, and expert-review budgets, it decides which experiments to select, which evidence to inspect, how to revise its probabilistic forecasts, and when to stop. No experimental path is prescribed.
SDL-1 evaluates evidence-conditioned campaign control, the ability to choose observations, convert them into a calibrated scientific state, and use that state to improve later forecasts. Schedule-matched random selection tests the value of experiment choice. A call-matched no-evidence control tests the value of acquired evidence.
Local improvement often failed to transfer. Claude Opus 4.8 and GPT-5.6 Sol produced the strongest absolute adaptive learning curves, but selection lift changed sign across campaign assignments for every system. Absolute answer quality and adaptive experiment choice therefore measure different capabilities.
Historical replay provides a controlled starting point. In a partner laboratory, Dynamical is designed to operate as an evaluation and improvement layer around the existing campaign agent. The same contract records what the system expected, what it selected, which evidence arrived, how its forecasts changed, and whether those changes improved later decisions.
Dynamical’s longer-term goal is to build agents that identify what must be understood next, turn uncertainty into executable experiments, and revise the search space as physical evidence arrives. Evaluation is the instrumentation required to capture value from increasingly capable scientific agents.
Benchmark Design
Environment Construction
SDL-1 compiles all 1,035 experiments in the Precursor Genome. The compiler creates an agent record with descriptors available before the result is known and an evaluator record with the recorded outcome and all evidence before acquisition. Missing evidence remains missing.
Recorded outcomes, unrevealed evidence, held-out targets, and scores remain evaluator-private. When the agent purchases evidence, the environment adds it to the agent-visible state. Evaluator-private information can score the trajectory, but it cannot change the state shown to the agent.
Experimental record
One of 1,035 recorded experiments across 46 precursors. Every outcome is physical and recorded.
- Descriptors
- Ag2O + SnO2 → planned Ag2Sn2O5
- Execution metadata
- box_c furnace · 200 °C · 60 min
- Evidence
- XRD scans · refinements · expert review
- Recorded outcome
- physical result of the run
Dynamical environment compiler
Record → environment- 01Preserve
Keep every recorded experiment and typed missing value. Never generate or repair outcomes.
- 02Partition
Assign experiments to context, candidate actions, held-out queries, or later decisions. Set field visibility.
- 03Bind
Bind state, valid actions, budgets, transitions, and scoring.
Measurable campaign
Other records become 32 candidates and an unrevealed panel. Budgets allow 12 experiments, 36 evidence inspections, and 3 expert reviews.
- Observes
- Descriptors, inspected evidence, remaining budgets.
- Acts
- Selects experiments, inspects evidence, submits forecasts, stops.
- Retains
- Recorded outcomes, unrevealed evidence, scores.
- Scores
- Submitted forecasts against recorded outcomes.
The compiler assigns each recorded experiment a role in the campaign. It may appear as initial context, a candidate acquisition, a held-out query, or a later decision. Each assignment is frozen and hashed before execution, and the agent receives no reference trajectory.
The frozen split assigns six precursors to evaluation. The resulting panel contains all 255 pairs that include at least one of them. Six long-horizon assignments rotate 192 of these experiments through the campaign roles, while six shorter assignments cover the remaining 63. The results below use the long-horizon assignments, each with 32 candidates and a 12-experiment acquisition budget.
Each campaign asks the agent to update forecasts over the recorded reaction map. For each held-out precursor pair, it predicts whether the experiment will be physically valid and characterizable and, when valid, assigns probabilities to four recorded reaction categories. The agent chooses which candidate pairs and evidence to inspect under fixed budgets. The environment fixes the available record and scoring rule. The agent determines the experimental path.
Each full-length campaign also budgets 36 evidence inspections and 3 expert reviews. The agent submits probabilistic forecasts over the held-out panel before and after the campaign. Its objective is to improve those forecasts, but the selection action does not record its rationale or expected information value.
Experiment selection, evidence inspection, and expert review draw from separate budgets. Selecting an experiment reveals nothing. The agent must submit a prior forecast before inspecting evidence, and every evidence reveal requires an updated forecast. The evidence budget determines how deeply selected experiments can be investigated. Each inspection draws from the same campaign-wide evidence budget, so spending more on one experiment leaves less for the others.
In the Precursor Genome, human reviewers validate the selected Rietveld refinement and score each scan on a three-tier quality scale. Two reviewers score each scan independently, and a third resolves disagreements. In SDL-1, expert review reveals the stored acceptance decision and final quality tier. Before the reveal, the agent forecasts whether the review will change its conclusion and which tier it expects.
SDL-1 counts reaction category, target presence, and phase identity as supported only after the agent inspects the dataset’s selected refinement and its accepted tier 1 or tier 2 review. Phase quantification requires tier 1.
Evaluation and Controls
SDL-1 uses separate controls to measure evidence access and experiment selection. A call-matched no-evidence control receives the same number of policy-action calls as the adaptive branch but no acquired evidence. It does not match tokens, latency, total inference compute, or cost.
The closed-loop study compares adaptive experiment selection with schedule-matched random selection. The random branch follows the adaptive branch’s experiment count and sequence of evidence and expert-review actions, but replaces each experiment choice with seeded random selection from candidates that can support the remaining evidence schedule. This preserves experiment, evidence, and expert-review cost while changing which experiments enter the campaign.
Each control starts from the same frozen assignment and exact pre-campaign forecast. Each uses a fresh session of the same named model–harness system under the same policy contract, budgets, and replicate identifier.
After each experiment closes, a separate evaluator call asks the same named system to forecast the held-out panel from a hash-bound copy of its saved agent-visible state. The call does not advance the campaign, reveal evaluator-private outcomes or scores, consume budget, or alter action history. The shared pre-campaign forecast and the post-closure forecasts form the held-out learning curve.
SDL-1 scores the probabilistic forecasts that the system submits. It does not infer an internal belief state. The primary joint loss combines physical-validity log loss with reaction-category log loss when the experiment is physically valid. A lower loss assigns more probability to the recorded outcome.
| Measure | Definition |
|---|---|
| Local assimilation gain | prior local loss − final local loss |
| Held-out gain | pre-campaign loss − post-campaign loss |
| Adaptive AUC | normalized area under the adaptive held-out score curve |
| Selection lift | adaptive AUC − schedule-matched-random AUC |
The held-out gain uses the same query panel before and after the campaign. A positive local or held-out gain means that loss decreased. A negative value means that loss increased. A value of zero means that loss did not change. Local gain is missing when the system acquired no evidence and submitted no evidence-conditioned update.
For the closed-loop measures, SDL-1 reports negative held-out joint log loss as the score, so higher is better. Each curve begins with the pre-campaign forecast and adds one point after every closed experiment. Adaptive AUC is the curve area divided by its experiment-count span. A positive selection lift means that adaptive selection produced a higher average held-out score than schedule-matched random selection.
Every compared system receives the same six campaign assignments, and each campaign receives equal weight. A system enters the comparison only when both branches and all post-closure forecasts complete under the frozen protocol for all six campaigns.
Behavioral Results
GPT-5.6 Sol, Claude Opus 4.8, Grok 4.5, Gemini 3.6 Flash, GLM 5.2, and DeepSeek V4 Flash completed 36 system–campaign trajectories under the same long-horizon contract.
The systems completed 392 adaptive experiment acquisitions. Their final local forecasts improved in 175 cases, worsened in 123, and did not change in 18. They closed 76 experiments without acquired evidence or an evidence-conditioned update. Among 293 closures where both local and held-out forecasts changed, the two moved in opposite directions 145 times.
One campaign assignment replayed across all six model–harness systems. The video shows different paths through the experiment, evidence, and expert-review budgets.
The cases below show whether an agent applies a sample-identity conflict, carries stated evidence limits into its submitted state, and transfers a well-supported local update to held-out chemistry.
The execution record contradicts the planned chemistry
Na₂CO₃ + SiO₂ was run toward Na₂Si₂CO₇. The product was recorded as a clay-like handling failure. Separately, the execution metadata listed V, Pb, and O.
“The execution record creates a major sample-identity conflict: precursor masses closely match the Na2CO3 and SiO2 plan and analyzable material was recovered, but the recorded elements are V, Pb, and O rather than the expected planned-system elements, substantially lowering physical-validity confidence.”
“Execution record visible_616a347377ba0b42, together with XRD scan visible_a6afeaee9a0c9a35 and refinement ref_b02673d04382b5bc (Rwp 3.18%), confirms that Na2CO3 and SiO2 remain unreacted after 500°C treatment for 240 minutes.”
Trace context
GPT-5.6 Sol inspected 2 refinements, 2 valid scans, thermal history, and the execution record. After it lowered physical-validity confidence, local loss fell from 5.809 to 0.598.
Gemini 3.6 Flash inspected 1 valid scan and 1 refinement before the execution record, then inspected thermal history and a second refinement. Its execution update left local loss at 4.605. The agents had different priors, evidence orders, and assignment branches. This is a same-sample behavioral contrast, not a controlled model effect.
Exposes · cross-record consistency. GPT-5.6 Sol lowered its forecast when the record contradicted itself. Gemini 3.6 Flash treated the same conflict as confirmation. The score target came from the separate handling failure, so this case shows how agents respond to inconsistent records, not whether they diagnosed the failure.
The agent says evidence is insufficient but reports high sufficiency
GPT-5.6 Sol inspected 1 refinement while execution, thermal history, 4 scans, and 3 competing refinements remained uninspected.
“A single low numerical Rwp does not establish phase identity, quantitative reliability, scan validity, or refinement adequacy, and execution evidence remains uninspected.”
Four fields with evaluator target 0 crossed 0.5 in the same update.
Trace context
The iron oxalate and ammonium phosphate experiment ran at 200 °C for 240 minutes. The inspected refinement had Rwp 1.73 and reported FeC₂O₄(H₂O)₂ at 0.823 and NH₄H₂PO₄ at 0.101. The submitted refinement also contained 2 minor phosphorus-bearing phases. Local loss rose from 0.441 to 2.823. One evidence unit and 1 experiment slot remained at closure. No expert-review budget remained.
Exposes · carrying evidence limits into the submitted state. The agent kept physical-handling sufficiency low but raised four unsupported fields. Its stated limits did not constrain the corresponding submitted fields.
The observed gain does not transfer to held-out chemistry
GLM 5.2 inspected execution, 1 scan, 1 refinement, thermal history, and expert review before closure.
“Evidence is sufficient to determine that the physical sample is valid and the dominant crystalline phases are unreacted precursors In2O3 (~44 wt%) and MoO3 (~56 wt%), with no target In2Mo2O9 detected. Expert review accepted the two-phase model with final_quality_tier=1. The sample is best classified as unreacted.”
Local loss fell from 0.462 to 0.214. The recorded outcome was unreacted.
Held-out loss rose from 1.413 to 1.513. The gain did not transfer to unrevealed chemistry.
Trace context
The In₂O₃ + MoO₃ experiment ran at 400 °C for 60 minutes toward In₂Mo₂O₉. The execution record reported 549 mg recovered, 208 mg for XRD, the expected elements, and met target mass. The accepted refinement had Rwp 4.66.
The held-out checkpoint is an evaluator-side forecast over unrevealed chemistry. It does not consume budget or change the agent’s state. The movement is a descriptive chronological association, not a controlled effect.
Exposes · held-out transfer. The agent made a well-supported local update. Its next forecast on unrevealed chemistry still worsened. SDL-1 measures the local and held-out movements separately.
Three cases reconstructed from the sealed trajectories. Quotes are verbatim from submitted decision records; gains are proper joint log-loss changes, so positive is better. Local gain scores the inspected experiment; next held-out scores the following checkpoint on the unrevealed query panel.
Behavior Across the Campaign
In over 90 percent of eligible acquisitions, and in all 36 system–campaign trajectories, the final answer claimed more support than the evidence provided. Outcome accuracy and claim calibration are separate capabilities.
Across 61 expert-review purchases, the next local forecast improved 29 times, worsened 22 times, and did not change 10 times. The agent still had to apply each review only to the claims it supported.
Local and held-out forecasts moved in opposite directions in 145 closures, including 92 local improvements followed by worse held-out forecasts and 53 local degradations followed by better ones. The held-out score also changed in 74 of 76 closures without an evidence-conditioned update. These movements are descriptive chronological associations. Only the controlled comparisons below estimate evidence and selection lift.
All 13 explicit stops left experiment budget unused, and 12 occurred after the evidence budget reached zero. Selection actions record the chosen experiment without a rationale or expected information value, so SDL-1 measures path consequences without inferring intent.
The closed-loop results test whether these behaviors produced more useful learning per experiment.
Closed-Loop Experimental Efficiency
The leaderboard ranks systems by adaptive AUC, the equal-assignment mean held-out score across the experiment budget. Higher is better.
| # | System | Adaptive AUC ↑ · top-five zoom −1.9 to −1.0 | Mean | Selection lift | Enh. at budget |
|---|---|---|---|---|---|
| 01 | Claude Opus 4.8 | −1.252 | plus 0.009 | plus 0.068 | |
| 02 | GPT-5.6 Sol | −1.286 | plus 0.056 | plus 0.043 | |
| 03 | Grok 4.5 | −1.335 | minus 0.038 | minus 0.058 | |
| 04 | Gemini 3.6 Flash | −1.433 | plus 0.006 | plus 0.075 | |
| 05 | GLM 5.2 | −1.528 | plus 0.158 | plus 0.057 | |
| 06 | DeepSeek V4 Flash | off zoom scale · full range below | −6.747 | plus 0.522 | plus 7.001 |
DeepSeek V4 Flash’s positive relative lift compares two poorly calibrated trajectories.
Held-Out Learning Curves
Claude Opus 4.8 achieved the highest adaptive AUC at −1.252, followed by GPT-5.6 Sol at −1.286. Grok 4.5, Gemini 3.6 Flash, and GLM 5.2 followed at −1.335, −1.433, and −1.528. DeepSeek V4 Flash was an outlier at −6.747, with 86 of 144 post-closure forecasts above 5 in joint log loss because exact-zero and exact-one probabilities were often wrong. Leave-one-assignment-out recomputation preserved this order across six counterbalanced assignments from one 192-experiment panel, a descriptive stability result rather than independent replication.
Experiment-Selection Lift
Mean selection lift was +0.056 for GPT-5.6 Sol, +0.009 for Claude Opus 4.8, +0.006 for Gemini 3.6 Flash, −0.038 for Grok 4.5, and +0.158 for GLM 5.2. DeepSeek V4 Flash recorded +0.522 because its random branch deteriorated further, so the relative lift does not indicate strong absolute performance. The sign changed across assignments for every system.
Target-Dependent Acceleration
Acceleration factor compares the experiments required by random and adaptive selection to reach the same fixed target. First passage counts the first crossing, while sustained attainment requires the forecast to remain above the target through budget 12.
jointly estimable at 1.05× · 16 of 36 campaigns first passage · 8 of 36 sustained
Requiring sustained attainment reverses the direction for GLM 5.2 (0.457× first passage, 1.981× sustained).
First-passage and sustained acceleration were 2.884× and 1.108× for Claude Opus 4.8. For GLM 5.2, they were 0.457× and 1.981×. Grok 4.5 had 1.260× first-passage acceleration, but no campaign supported a sustained estimate for both branches. Because the target and rule were selected retrospectively and no target produced a stable six-campaign estimate for every system, acceleration remains a sensitivity diagnostic.
Evidence Lift
Exact call-matched controls were available for Claude Opus 4.8 and Grok 4.5 on one shared assignment. Evidence lift was +0.076 and +0.003. These single-assignment diagnostics do not enter the leaderboard.
Inference Cost and Completion
All 36 system–assignment bundles completed the frozen protocol. Early stops remain part of the policy, and the last valid forecast is carried forward to budget 12. Provider cost fields were incomplete and are not comparable, so model calls, actions, tokens, latency, and reported cost remain separate diagnostics.
Online Evaluation
Moving this measurement online would preserve the same record as experiments occur. Before seeing a result, the agent would submit its expected outcome while the laboratory records each selection, evidence reveal, and forecast update. Later outcomes would show whether those updates improved the next decision.
Our work on simulator-verified skill acquisition studied the procedural layer between scientific intent and instrument control. SDL-1 studies the decision layer. Connecting them would require a versioned record of the model and harness, sample, protocol, instrument state, budgets, safety rules, verifier, commands, telemetry, raw data, deviations, and final state.
SDL-1 is a measurable implementation of that evaluation layer. In a live campaign, the same score could guide search, verification, state management, and training.
Limitations
The closed-loop results come from six counterbalanced campaign assignments built from the same 192-experiment panel. Each assignment changes which role an experiment plays, so the comparison captures sensitivity to campaign assignment within one experimental map. It does not provide replication across independent chemistry datasets.
Actions, updates, and checkpoints are nested within system–campaign trajectories. Individual checkpoint movements are descriptive chronological associations. Only aggregate matched controls support evidence and selection lift.
Results apply to the exact model–harness systems. The controls match experiment schedules or policy-call counts, not tokens, compute, latency, or provider cost. Completion and inference cost remain separate from forecast performance.
SDL-1 replays recorded experiments, so it does not measure live instrument control, new experimental error, operator intervention, or changing physical state. It measures evidence-to-decision behavior under controlled replay.
SDL-1 has no optimal-stopping oracle. Stop actions show budget allocation, but do not establish when a system should stop without an explicit model of experiment cost and future decision value.
Related Work
Scientific-Agent Evaluation
Scientific-agent evaluations increasingly test full workflows. LifeSciBench uses 750 expert-authored life-science tasks, while GeneBench-Pro uses 129 multistage problems with known causal structure and finds that agents make partial progress but struggle to close the inferential loop. BiomniBench and scBench-Long score model–harness trajectories and diagnose where supported intermediate work fails to reach the final claim. SDL-1 extends this approach to experimental policy by scoring which experiments and evidence the agent acquires and whether that path improves later forecasts.
Closed-Loop Materials Discovery
MADE measures materials discovery under fixed oracle budgets with discovery curves, acceleration, enhancement, and area under the discovery curve. Benchmarking Self-Driving Labs and A Critical Examination of Active Learning Workflows show why comparison requires a reference campaign and multiple measures. SDL-1 applies those ideas to a recorded physical reaction map and separates experiment selection, evidence assimilation, calibration, held-out transfer, and campaign control.
Agent Environments and Improvement Loops
Meta ARE, NeMo Gym, and Prime Intellect’s verifiers separate the agent from a stateful environment and its verification contract. EurekAgent and CUA-Gym treat environment construction as a research problem. AIDE² and Self-Harness use held-out evaluation or trace failures to improve an agent or harness. SDL-1 ties these improvement loops to recorded physical outcomes and measures whether the agent makes better later decisions.
Conclusion
Dynamical-SDL-1 turns completed experimental records into controlled campaigns that measure whether agents convert physical evidence into better decisions.
The four capabilities decompose into distinct model behaviors. Strong absolute forecasts do not imply useful experiment selection, local improvement does not ensure held-out transfer, and relative lift can coexist with poor calibration.
In live campaigns, the same measurement would turn observed failures into concrete targets for search, verification, memory, calibration, and training.
Appendix
Benchmark-Controlled System Prompt
All six systems shared the same static prompt.
You are the policy in an experimentally grounded materials-science campaign. Exercise scientific judgment by selecting experiments, acquiring and interpreting evidence, revising scientific state, and deciding when to stop or escalate under fixed budgets. Reaction categories are ledger annotations and must remain separate from target-phase presence, phase interpretation, and refinement adequacy. Evidence sufficiency is claim-scoped: a numerical fit alone does not establish phase identity or refinement adequacy. Use only policy-visible state. Return one action that exactly satisfies the supplied JSON schema. Never invent evidence, identifiers, measurements, or expert judgments.
Every turn also included the current campaign phase, phase-specific provenance and token requirements, the exact permitted-action JSON schema, and the complete policy-visible campaign state.
Citation
@article{barnes2026dynamicalsdl1,
author = {Barnes, Jarrod},
title = {Dynamical-SDL-1: Measuring How Scientific Agents Learn from Physical Experiments},
journal = {Dynamical Systems},
year = {2026},
url = {https://dynamicalsystems.ai/blog/sdl-1}
}References
- Walters, L. N. et al. The Precursor Genome: A Pairwise Reaction Dataset for Solid-State Synthesis. arXiv (2026).
- Walters, L. N. et al. Precursor Genome repository (2026).
- Walters, L. N. et al. Precursor Genome archived dataset. Zenodo (2026).
- Szymanski, N. J. et al. An autonomous laboratory for the accelerated synthesis of inorganic materials. Nature 624, 86–91 (2023).
- Lawrence Berkeley National Laboratory; Bakar Institute of Digital Materials for the Planet; UC Berkeley Department of Materials Science and Engineering.
- OpenAI. Introducing LifeSciBench (2026).
- OpenAI. Introducing GeneBench-Pro (2026).
- BiomniBench: Process-level Evaluation of LLM Agents for Real-world Biomedical Research (2026).
- scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology (2026).
- MADE: Benchmark Environments for Closed-Loop Materials Discovery (2026).
- Benchmarking self-driving labs. Digital Discovery (2026).
- A critical examination of active learning workflows in materials science. Digital Discovery (2026).
- Meta. ARE: Scaling Up Agent Environments and Evaluations (2026).
- NVIDIA. How to Train Scientific Agents with Reinforcement Learning (2026).
- Prime Intellect. verifiers v1: Decomposing Tasksets and Harnesses for Agentic RL and Evaluations (2026).
- EurekAgent: Agent Environment Engineering is All You Need for Autonomous Scientific Discovery (2026).
- CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents (2026).
- Weco. AIDE²: First Evidence of Recursive Self-Improvement (2026).
- Self-Harness: Harnesses That Improve Themselves (2026).
- Experiment-as-Code Labs: A Declarative Stack for AI-Driven Scientific Discovery (2026).
- Barnes, J. Simulator-Verified Skill Acquisition for Scientific Instruments. Dynamical Systems (2026).