Can a Self-Driving-Lab Agent Tell When the Evidence Is Enough?

Software abundance is driving a buildout of physical hardware, and the materials behind it, the magnets, coatings, alloys, and rare-earth components of energy, cooling, and defense and aerospace systems, advance at the pace of physical qualification rather than software. Candidate generation for those materials is increasingly handled by foundation models, generative materials models, MLIPs, and DFT automation, so the bottleneck has moved downstream to qualification-grade evidence, the proof that a material survives physical reality, can be reproduced, can be trusted, and can enter a high-consequence system. Raising the throughput of that evidence starts with measurement, with knowing where the judgment that drives those decisions holds and where it breaks before a lab is trusted to run on its own. For mission-critical alloys, coatings, and rare-earth-constrained parts, the cost of weak judgment is paid in repeated tests, scrapped specimens, and expert review time spent re-deciding calls that were already made once.

Self-driving labs close part of this gap by executing experiments from code, faster and around the clock, and the reading of those experiments is automating alongside them. Systems now turn diffraction into phase maps, raw spectra into structures, and vibrational signatures into defect identities, and the strongest of them ship calibrated confidence because higher measurement throughput raises the need to judge what a result can support. Execution speed and measurement throughput are both real, but neither is the constraint that decides whether the output can be trusted. What decides trust is judgment, the ability to tell what the evidence in hand can and cannot support. That judgment is the qualification gate of the discovery-to-qualification pipeline, the point every automated upstream step, candidate generation, synthesis, and measurement, has to pass through before a result is trusted. Before an agent is allowed to drive a qualification program, it should be qualified itself. Before autonomy, agent qualification.

We build the environment compiler, the layer that turns qualification evidence into a machine-readable, replayable record a model can learn from, so the path from discovery to qualified capability compresses instead of repeating itself. We begin on AM Bench to prove it against a real, published record, and this report is the first public proof surface. We compile its workflows into source-located replay environments and run six frontier models at the evidence boundary.

Experiments Become Code. Experience Has Not.

An experiment written as code can be versioned, checked before it runs, and replayed across sites,¹1An experiment-as-code stack compiles experimental intent into versioned, reproducible runs, and argues the binding constraint is no longer agent intelligence but the systems abstractions around physical instruments. Experiment-as-Code Labs, arXiv 2605.04375 (2026).¹ which is why experiment-as-code frameworks, declarative lab stacks, and national programs like the National Science Foundation's network of AI-programmable cloud laboratories and the Department of Energy's Genesis Mission²2NSF describes its Programmable Cloud Laboratories Test Bed as distributed autonomous laboratory facilities whose nodes can be remotely accessed to run custom user-programmed workflows, including self-driving autonomous experiment workflows. DOE describes the Genesis Mission as an integrated platform connecting supercomputers, experimental facilities, AI systems, and unique datasets.² show that public investment is explicitly aimed at programmable, AI-enabled experimental execution.

Turning execution history back into reusable decision signal has no equivalent stack.³3A NIST perspective on autonomous-lab standards finds no agreed-upon standards for data sharing or instrument control across these platforms, and assigns the data layer the job of cataloging data and metadata for retrospective study and model auditing. Joress, DeCost, Jones, Kusne, McDannald, Trautt, and Tavazza, Matter 9 (2026), 102756.³ When a campaign ends, the history that produced its result, the record of which evidence was trusted, which test was skipped, which source was caught as defective and which was accepted, usually survives only as a PDF, a folder of instrument files, a spreadsheet, a lab notebook, or a one-off report. Those decisions are exactly the signal an agent would need to make the same calls the next time it faces them, and almost none of that signal survives in a form a model can learn from.

The cost of leaving that signal unbuilt is not paid in execution speed. A lab that runs an order of magnitude faster than a human also reaches wrong conclusions an order of magnitude faster, and the rate that actually decides whether such a lab can be trusted is its qualification throughput, the rate at which evidence-based go and no-go decisions can be made and then stood behind. Faster execution raises both the value of good judgment and the cost of bad judgment, which is why the judgment layer, rather than the actuation layer, is where trust in an autonomous lab is won or lost.

The larger goal is to build the experience layer for physical science, turning the accumulated history of experiments into source-located evidence, deterministic verifiers, and value signals, so that physical work compounds instead of rediscovering the same failures. The benchmark in this report is the first narrow instance of it, and it starts where it has to start, with a historical workflow that already happened.

Compiling Historical Replay Environments

AM Bench is a NIST-led series of additive-manufacturing measurement and modeling-challenge problems, built so that simulations can be tested against controlled, rigorously measured data. It was not designed as an agent benchmark. We repurpose a subset of its public records into an evaluation surface, where an agent inspects archived evidence and makes a bounded qualification-style decision rather than inventing experiments or predicting physical outcomes.

We started with three DOI-backed NIST AM Bench records,⁴4Primary sources include the NIST AM Bench program page, the AM Bench 2025 challenge descriptions, the direct AM Bench data links, and the NIST Public Data Repository DOI records for AMB2025-02, AMB2025-03, and AMB2025-06/07. These three records are the source workflows used here, not generic examples.⁴ not generic materials examples. They cover IN718 tensile testing, Ti-6Al-4V fatigue, and IN718 laser-track process calibration. Each one is a real sequence of process records, characterization results, calibration artifacts, and source documents, with measured outcomes attached on the verifier side and never exposed to the policy.

We compiled these into a source-located evidence graph. Every piece of evidence an agent can request points back to a specific realized record in the historical workflow. The contract is realized-only.⁵5The environment never invents an outcome. If a measurement was not actually produced in the historical workflow, no agent and no policy can conjure it. Requests for evidence that does not exist in the archive return unavailable.⁵ If a measurement was not taken, it does not exist in the environment, and asking for it returns unavailable rather than a fabricated value.

That constraint keeps the benchmark honest about what the historical record can and cannot support, and it keeps the agent inside the same visible-evidence boundary the historical decision faced. Realized-only is an anti-synthetic-outcome contract, not a claim of representativeness. A realized additive-manufacturing record can still be nonrepresentative, locally heterogeneous, or misregistered, and the replay does not test for that.

This is a deliberate change from how we built environments before. In earlier work we trained scientific judgment in verified campaign environments whose candidates were generated by a crystal diffusion model and scored against a physics oracle. Here there are no generated candidates and no oracle we built. The inputs are a real, published experimental record, and the only authority is that record and a deterministic verifier. Nothing in the compiler is specific to these three workflows, so any archive that carries process records, characterization, calibration, and outcomes can be compiled the same way.

Inside the Compiler

The environment compiler builds the experience layer. It takes the residue of physical execution and turns it into the objects an agent can be evaluated on and converted into training signal. It ingests process records, instrument traces, calibration artifacts, source documents, and outcomes, resolves them into source-located evidence objects, builds realized-only replay tasks, gates them through verifier and leakage checks, runs policy models through the environment, and returns behavior traces, value and sufficiency signals, and failure cases for later training.

The compiler is deterministic where authority matters and model-assisted where interpretation creates value. That rule is what separates this work from naive ingestion on one side and unverifiable synthetic data generation on the other.

Annotation is where historical data becomes scientific experience. The raw archive tells us what happened, but it does not always tell us what decision was being made, which evidence was binding, what defect mattered, or what behavior a future agent should be tested on. We pin each decision to a realized record first, then use a model to reframe it as a task and to select, from the supplied options, the evidence an agent is allowed to see. The model writes the framing and the decision context. It cannot create outcomes, gold labels, verifier answers, the correct terminal action, or any policy-visible shortcut.

Every candidate the model proposes passes two gates before it becomes a task. Deterministic filters reject any candidate that selects a verifier-only field, an evidence ref that does not exist, a forbidden answer artifact, or a verifier that does not match the episode. A live judge, itself calibrated against fixed accept and reject cases before it is trusted, then scores the survivors against a six-criterion rubric for grounding and framing quality.⁶6The judge scores six things, whether the task matches the intended decision, covers the required evidence criterion, binds its answer to the verifier, gives useful feedback when evidence is missing or defective, stays grounded in the workflow mechanism, and keeps policy-visible evidence separate from hidden answers. The reviewer is GPT-5.5 at medium reasoning effort, and it scores framing quality only. The archive and the deterministic verifier still set the correct answer. On this package it admitted all 104 tasks, rejecting and failing none.⁶ A candidate is admitted only when the deterministic filters pass and the judge accepts, and the judge never sets the reference answer or the terminal truth.

Authority stays with the deterministic verifiers and the archive, and that authority covers performance, not only procedure. No language model holds ground-truth authority over any task. Every model run in this report uses the same fixed task set. Each model runs three replicate passes, so the panels that follow report a mean and a 95 percent interval rather than a single number. The provider CLIs and APIs expose no portable deterministic seed, so the passes are replicate runs at each provider's default sampling, and the intervals measure run-to-run spread rather than a controlled seed sweep.

What counts as success runs in a fixed order. The point of the compiler is to help a team reach a better and more defensible decision about a material or process, and to reach it on less evidence and in less time than the current workflow. Performance is the primary axis. A faster decision that is wrong is a regression, so a reduction in cost or time only counts once the decision itself holds. The evidence-boundary behavior the rest of this report measures is the prerequisite for that decision, the discipline an agent needs before its speed is worth trusting, not the performance win by itself.

Retrospective Value-of-Evidence Replay

VOE-Bench comprises 104 retrospective value-of-evidence tasks across the three workflows. Across the task set, agents can select from 498 per-task evidence entries, drawn from 29 unique canonical archive evidence IDs. Each task replays a single decision moment from the historical workflow and asks the agent to act using only what is visible. Those numbers are nested, so every trajectory total later in this report is a repeated measurement over the same base of three workflows and 29 canonical evidence objects, and the aggregates carry the spread of that base rather than the precision a count in the thousands would suggest.

Every task has the same anatomy.

The agent sees a policy-visible state that carries menu metadata and nothing else. We strip gold labels, variant labels, answer keys, verifier-only values, split tags, and any field describing whether a piece of evidence is useful, before the state ever reaches the model. It chooses from a fixed menu of seven actions, three requests (request_characterization, request_calibration_artifact, request_process_record) and four terminal moves (flag_source_defect, localize_missing_evidence, submit_risk_packet, abstain_or_escalate). When it requests evidence, the archived result is revealed. When it stops, a deterministic verifier scores the terminal action against the realized record.

The 104 tasks span five decision moments, each one abstracting a call a materials reviewer makes.

Every task in the package sits on the decision boundary by construction, where the available evidence is sufficient to act, defective enough to reject, incomplete enough to inspect, or uncertain enough to stop.

An evidence-grounded valid decision is a final action accepted by the verifier given the evidence the agent actually saw. It is not answer-key correctness in the usual sense. It checks whether the agent stopped in the right way, by submitting only when the evidence supports it, flagging a source defect when one is visible, localizing the missing binding record instead of requesting everything, or escalating when the evidence does not resolve. An agent can read the right document and still pick the wrong terminal action, and evidence-grounded validity is what catches that. The verifier scores three things separately, whether the terminal action class is one the reference accepts, whether the action payload binds to a record the agent actually pulled, and whether the decision is grounded in the evidence it saw. A run is evidence-grounded valid only when all three hold, which is why an agent can name a real defect and still fail by attaching it to a record it never pulled. Evidence-grounded validity is a verifier proxy for evidence-boundary discipline. It is not a measure of physical qualification, and an evidence-grounded valid run says nothing about whether the underlying process, structure, and property chain is sound. The reference action it scores against is an authored decision policy, reproducible by construction and defensible on these workflows, not an independently validated standard for what a reviewer must do. A flagged source defect here is a provenance or checkability defect, a broken source reference or an unverifiable record, not a metallurgical or structure-property judgment, and a submitted risk packet is a replay object rather than a qualification packet in the materials-release sense.

Those three checks all live at the terminal action. Evidence grounding decomposes into three separable behaviors, acquiring evidence that could change the decision, recognizing when the acquired set is sufficient to stop, and binding the terminal claim only to evidence the agent actually acquired. The first runs while the agent is still requesting. The second and third are the terminal call itself, the moment it decides the set is enough and ties the decision to a record it pulled. We report the three apart because the agents are not equally strong on them, and the failure analysis sorts along the same three.

Most self-driving-lab benchmarks ask a search-efficiency question, whether an optimizer reaches a better outcome with fewer experiments, usually Bayesian optimization against random sampling.⁷7A survey of self-driving-lab benchmarking finds random sampling the most widely adopted baseline across 42 studies and 63 reported benchmarks. Adesiji, Wang, Kuo, and Brown, Digital Discovery 5 (2026), 14.⁷ This benchmark asks the complementary one. Once the evidence exists, can an agent tell whether it is enough to act on and stop in the right place? Faster search is not the same as calibrated judgment, and the full loop needs both.

Baselines and the Model Panel

Static baselines confirm the environment is hard for the right reasons. The deterministic reference, which follows the verifier-grounded policy, is valid on every task, and mechanical strategies are not.

Requesting everything before submitting is the strongest mechanical strategy and still reaches only 29 of 104. Blanket submission scores zero, blanket refusal scores 6, and random requests reach 18. The environment rewards getting the evidence boundary right, which a fixed posture cannot do.

Then we ran six frontier models over the same 104-task package, three replicate passes each, 1,872 trajectories in all, with parse, invalid-action, off-menu, leakage, and forbidden-term gates passing on every run, recorded in each run's live-validator and task-quality-audit summaries.

Six frontier models, three replicate passespassmean95% CI

Model	Evidence-grounded valid ↑	Action valid ↑	Useful prec. ↑	Evidence regret ↓	Cost score ↑	Over-submit ↓
Gemini 3.1 Pro	92.9 ±1.4	0.987	0.872	0.785	1.679	0 / 312
Claude Opus 4.8	91.7 ±2.8	0.974	0.910	0.593	1.722	0 / 312
GLM 5.2	91.7 ±3.6	0.965	0.891	0.577	1.699	4 / 312
Claude Sonnet 4.6	89.4 ±2.4	0.952	0.823	0.869	1.590	0 / 312
Kimi K2.7 Code	88.8 ±3.6	0.949	0.860	0.772	1.625	4 / 312
GPT-5.5	86.9 ±5.0	0.929	0.855	0.788	1.603	0 / 312

Gemini 3.1 Pro

92.9 ±1.4

290/312 valid

action valid ↑0.987

useful prec. ↑0.872

evidence regret ↓0.785

cost score ↑1.679

over-submit ↓0 / 312

Claude Opus 4.8

91.7 ±2.8

286/312 valid

action valid ↑0.974

useful prec. ↑0.910

evidence regret ↓0.593

cost score ↑1.722

over-submit ↓0 / 312

GLM 5.2

91.7 ±3.6

286/312 valid

action valid ↑0.965

useful prec. ↑0.891

evidence regret ↓0.577

cost score ↑1.699

over-submit ↓4 / 312

Claude Sonnet 4.6

89.4 ±2.4

279/312 valid

action valid ↑0.952

useful prec. ↑0.823

evidence regret ↓0.869

cost score ↑1.590

over-submit ↓0 / 312

Kimi K2.7 Code

88.8 ±3.6

277/312 valid

action valid ↑0.949

useful prec. ↑0.860

evidence regret ↓0.772

cost score ↑1.625

over-submit ↓4 / 312

GPT-5.5

86.9 ±5.0

271/312 valid

action valid ↑0.929

useful prec. ↑0.855

evidence regret ↓0.788

cost score ↑1.603

over-submit ↓0 / 312

Arrows mark the better direction. Evidence-grounded valid is a right call backed by evidence the agent pulled, action valid the right kind of move, useful precision the requests that mattered, evidence regret the evidence missed or wasted, cost score right and efficient, over-submit advancing when it should have stopped. The validity axis shows three pass rates per model, their mean, and 95% interval, and the deterministic reference scores above every model, reaching 1.950 on cost score.

No model clears the benchmark. The strongest, Gemini 3.1 Pro, reaches a valid decision on 92.9 percent of its trajectories, a mean of 96.7 of 104 per pass, and the top three sit within overlapping intervals. Every model lands far above the best mechanical baseline and short of the deterministic reference. How the models fall short is the part worth reading, and the order they fall in is the least of it.

Across the panel the agents often reach verifier-valid evidence-boundary decisions, and where they fail they fail on the behaviors that matter most before a lab hands over authority, when to stop, when to hold a packet back, which record is binding, whether a payload is grounded in evidence the agent actually pulled, and whether one more request would change the call. The panel does not separate on a single axis, and because each model runs on its native agent surface, these read as temperaments of the model and the agent surface together. Gemini 3.1 Pro is the disciplined one, cleanest at the terminal boundary, with no over-submit and no wrong-class terminal in any pass. Opus 4.8 is the economical one, the most precise requests and the best cost score of the six, and it never over-submits. GLM 5.2 is the exhaustive one, which buys it the lowest evidence regret and costs it the most unnecessary requests. Sonnet 4.6 is the more cautious of the two Claude Code runs, leaving more useful evidence unread than Opus on the same surface. GPT-5.5 on the Codex surface is steady at acquisition and loose at the terminal boundary, holding most of the panel's wrong-terminal-action cases, 12 of 17. Kimi K2.7 Code sits closest to the submit edge, and it and GLM are the only two that over-submit, four trajectories each.

Packet escalation is the hardest task family. It is the replay analogue of a high-consequence packet stop, the moment an agent should refuse to advance a risk packet with an unresolved defect. Across the panel it carries the lowest useful precision at 0.66 and the highest evidence regret, and every failure mode appears in it.

The two hard task families fail for separate reasons. Calibration sufficiency exposes premature stopping, where the agents usually ask for useful artifacts but often stop before the full binding set is in hand. Packet escalation exposes a broader decision-risk surface, where they spend extra requests, over-abstain, occasionally over-submit, and sometimes bind the final action to the wrong record. Those are different failures, and they point at different training targets.

How Agents Fail

The agents reach a valid terminal decision on 90 percent of trajectories, with no parse errors, no off-menu actions, no duplicate loops, and no exhausted horizons in any of the 1,872 runs. The capability is real and the plumbing is clean. What remains is a judgment gap, and the raw trajectories make it legible. These six are drawn from the run, three valid and three invalid, each annotated with the model, the workflow, and the verifier's reference action.

The valid traces show the environment is not rewarding a single safe move. One agent inspects the binding item and stops on an invalid source reference. Another gathers two calibration artifacts, finds the third binding record has no acquisition path, and localizes it. A third refuses to advance a packet with no characterization behind it, the correct call, though it spends four requests reaching it.

The invalid traces split between decisions bound to the wrong evidence and decisions made at the wrong moment. On an easy task with full evidence in hand, one agent requests three valid calibration artifacts and then abstains, treating a routine known-limitation note as a blocker the reference does not. Another correctly reads that a required item is image-only and not machine-verifiable, then binds its defect flag to a catalog record it never pulled, so the right diagnosis lands on the wrong object. The retrieval and the syntax work in both. What fails is the estimate of what the evidence supports and where to attach the decision. When the agents misjudge the moment they lean cautious, over-abstaining far more often than they over-submit. Over-submission, advancing a packet over a visible defect, is the failure a qualification reviewer fears most, and it is the rarest of all, 8 of 1,872 trajectories and confined to two models.

Six cards are anecdotes, so we coded every trajectory the same way. Each of the 1,872 runs carries one dominant-behavior label from the codebook the cards use, which turns the individual stories into a distribution.

The distribution sorts into three tiers. 1,346 runs, 72 percent, take the reference path with nothing wasted. Another 343 reach a valid terminal on an inefficient evidence path, either stopping a record short or spending a request the reference did not need, which brings evidence-grounded validity to 1,689, or 90 percent. The remaining 183, 10 percent, are invalid under evidence-grounded validity, meaning the run reached a terminal the verifier would not accept on the evidence the agent actually held. Payload-grounding errors lead at 107, over-abstaining accounts for 51, wrong-class terminals for 17, and over-submits for 8. What remains sits entirely in the agent's decisions.

Read against the three behaviors, the gap is uneven. Acquisition is the strongest, the agents reach at least one decision-changing item on 92.9 percent of the opportunities that have one and the complete useful set on 80.1 percent. Stopping is weaker, 95.9 percent of terminal action classes are accepted, but only the 1,346 reference-equivalent runs, 72 percent, stop with no wasted evidence. Binding is where the 107 sit, and it is a data-integrity behavior, not a formatting one. The action parses, the action class is usually the one the reference would pick, and the decision still fails because it points at a record the agent never pulled. The agent reads a defect straight off the policy-visible catalog, then flags it without acquiring the record. That treats visible catalog metadata as if it were acquired evidence, and it is the single most common invalid failure in the panel.

The direction of the residual error is consistent, and it runs opposite to the usual worry.

Set against the reference, the agents submit risk packets less often, 470 against 522 trajectories, and escalate more often, 149 against 108, while flagging and localizing track the reference closely. On these workflows the agents lean toward holding back. That is the safer direction to miss in, and it is still a miss, because a lab that escalates a defensible packet pays for it in review time and stalled qualification throughput.

Learning from Delayed Evidence

The models in this panel can acquire evidence, some with high useful-request precision, but none of them reliably estimates whether the evidence in hand is enough, or short of the one binding item that would change the decision.

What these results point at is the next research object, a trainable estimate of the value and sufficiency of evidence over the state of an investigation, a representation that predicts which visible item is decision-changing, when a defect is binding, and when stopping is warranted. Building and testing that estimate is open work.

Generalization of Synthetic Training

Dynamical-30B-A3B, the policy from our earlier work on training scientific judgment, gives one way to test whether judgment learned in synthetic environments survives contact with a real historical record. That model was trained in synthetic, verifier-grounded campaign environments, where Crystalite generated candidate crystal structures and the Materials Project convex hull supplied a physics-grounded oracle for thermodynamic stability.⁸8The Materials Project defines energy above hull as a thermodynamic-stability measure relative to a convex hull built from formation energies. In the training environments it supplied deterministic verifier authority. In AM Bench replay, authority comes instead from historical realized records and deterministic replay verifiers. Materials Project glossary.⁸ It never saw AM Bench, these NIST records, or this replay benchmark in training.

Run on the same state-machine contract as the frontier panel, Dynamical-30B-A3B reached 65 evidence-grounded valid terminal decisions out of 104, a rate of 62.5 percent with a 95 percent Wilson interval of 52.9 to 71.2.⁹9Frontier intervals are 95 percent t-intervals over three replicate passes. Dynamical-30B-A3B is a single 104-task pass, so the interval here is a Wilson interval over task outcomes rather than run-to-run spread, and the two should not be read as comparable. The state-machine contract was also calibrated against this surface, so we treat the result as a prompt-calibrated out-of-distribution transfer diagnostic, not a clean held-out score or a claim of trained scientific capability.⁹ It trails the panel, and we do not read it as a leaderboard result. At 0.954 useful-request precision, the highest in the comparison, the model usually pulled a record that mattered when it requested one. That evidence-acquisition discipline carried out of generated discovery campaigns into historical physical-workflow replay, which is the behavior the frontier panel is strongest on.

Transfer from synthetic trainingDynamical-30B-A3B vs the frontier panel

evidence-grounded valid % (authorization) ↑

frontier band 86.9–92.9%

frontier best 0.910607590

Dynamical-30B-A3B

0.954 precision · 62.5% valid

0.850.900.95

useful-request precision (acquisition) →

Trained only in synthetic crystal-discovery campaigns and never on AM Bench, Dynamical-30B-A3B requests evidence more precisely than any frontier model here, the acquisition half of the loop, while its terminal decisions sit below the frontier band, the authorization half. A single 104-task pass, read as an out-of-distribution transfer diagnostic rather than a leaderboard rank.

The two leaderboard axes plotted against each other. Evidence-acquisition precision carries out of synthetic training, the call about when the evidence is enough does not yet.

The policy still trails on terminal authorization, the call about what the acquired evidence is enough to claim, and that is the value-and-sufficiency estimate the previous section named as the open object. A trained policy that carries the acquisition habit while missing the authorization call is what we would expect if that estimate is the thing still to build.

Read together, the replay benchmark and the synthetic training environments are two halves of one loop. Historical replay measures how agents handle real laboratory evidence, synthetic verifier-grounded environments train novel-discovery judgment where counterfactuals and outcomes are available, and transfer back onto realized records checks that the two are one system. What joins them is the physical-experience contract, the learned habit of selecting useful evidence, judging when it is enough, and binding a claim only to what came back.

Towards a Compounding Loop

Materials science still lacks a shared evaluation layer for whether an agent can reason over physical evidence before it allocates experiments, what to inspect, whether the evidence is sufficient, whether a defect invalidates the record, and whether a qualification-grade decision is actually supported. Historical replay is the safe place to test that layer, the pre-lab qualification step where an agent proves it can judge evidence before it touches an instrument.

Once an agent enters the lab, the evaluation target changes. The question becomes experimental and learning throughput, how efficiently it spends experiments, how safely it operates stateful instruments, how well it preserves provenance and failure state, and whether each closed-loop campaign improves the next, under real budgets and delayed outcomes. Existing self-driving-lab benchmarks mostly measure this second regime, acceleration or enhancement against reference strategies. The community-resource, experiment-as-code, and agentic-materials literature points to the substrate underneath those metrics, decision provenance, workflow state, protocol evolution, reusable failure records, and pipeline-level agents optimized for real discovery rather than isolated proxy tasks.¹⁰10Adesiji, Wang, Kuo, and Brown, Benchmarking self-driving labs, Digital Discovery 5, 14-27 (2026). Brown, Lejeune, and Werner, Turning self-driving labs into a community resource, Matter (2026). Yang et al., Experiment-as-Code Labs, arXiv 2605.04375 (2026). Zhang et al., Towards Agentic Intelligence for Materials Science, arXiv 2602.00169 (2026).¹⁰

Replay proves the judgment, the live lab compounds it, and each campaign returns as experience the next one can learn from.

From Replay to Live Control

A self-driving lab sets a different bar from the one a coding agent is usually held to. In software, an agent can be useful well below perfect reliability, because most failures are reversible, sandboxable, testable, and cheap to rerun. A lab gives no such margin on irreversible actions, safety-relevant operations, data-integrity decisions, and qualification claims. What it needs is near-zero false-positive authority at the evidence boundary, an agent that does not submit, execute, or claim when the evidence does not support it. That is a stricter property than high average accuracy, and it is the one retrospective replay is built to check.

These replay environments are useful before any model trains on them. A lab's own historical workflow becomes an auditable decision record, where premature stopping, mis-bound decisions, over-cautious escalation, and request-everything behavior all show up in the raw traces, before a model goes near a live instrument. The same move turns a public archive into something an agent can be checked against, not just a dataset to score.

An agent that cannot reuse past experience to judge what evidence matters is not ready to drive a qualification program, let alone explore on its own. The discipline that proves it ready, one workflow at a time, is the same one that eventually lets a team design a material and the part built from it in one loop, rather than qualifying one long after the other.

@article{barnes2026evidence,
  author  = {Barnes, Jarrod},
  title   = {Can a Self-Driving-Lab Agent Tell When the Evidence Is Enough?},
  journal = {Dynamical Systems},
  year    = {2026},
  url     = {https://dynamicalsystems.ai/blog/benchmarking-self-driving-lab-agents}
}

If you are building agents for physical science, compiling experimental history into training signal, or working on value estimation over evidence, we want to hear about it.