Can a Self-Driving-Lab Agent Tell When the Evidence Is Enough?
VOE-Bench turns recorded additive-manufacturing evidence into tests of what agents inspect and when they decide.
TL;DR
Can an agent tell when the evidence is enough to support a decision?
- VOE-Bench asks agents to inspect manufacturing, calibration, and measurement records before deciding whether the evidence supports a conclusion. Across 1,872 replay runs with six models, 90% of final decisions met the study’s review rules and were supported by records the agent had inspected.
- The most common failure was connecting a decision to a record the agent had not acquired, accounting for 58% of invalid decisions.
- Requesting useful evidence and deciding what it supports were distinct capabilities. The replay makes both visible in the agent's actions and final decision.
Evidence Review
A materials review often turns on a specific piece of evidence. A calibration record may establish whether a measurement can be used, while a missing process record may leave an important comparison unresolved. An agent helping with that review needs to identify what matters and connect its conclusion to the evidence it has inspected.
We built VOE-Bench to measure that behavior using recorded NIST AM Bench workflows. Agents inspect evidence, identify source defects or missing records, and decide when to submit or escalate a review. Each action can be checked against the archived records and an explicit reference policy.
The study covers IN718 tensile testing, Ti-6Al-4V fatigue, and IN718 laser-track process calibration.1 The decisions concern evidence handling within these workflows. The reference policy defines acceptable review actions, while physical qualification remains a separate materials assessment.
Findings
Gemini 3.1 Pro, Claude Opus 4.8, Claude Sonnet 4.6, GLM 5.2, GPT-5.5, and Kimi K2.7 Code completed three passes each over the 104 tasks using their native agent harnesses.
Agents made verifier-valid decisions in 90% of runs and followed the reference evidence path in 72%. The remaining valid runs acquired fewer useful records or made more requests than the reference. Validity and agreement with the reference path measure different aspects of the review.
Where decisions failed
Share of the 183 invalid decisions
All seven behavior categories
Counts and percentages in this list use all 1,872 runs. Each run has one category.
Of the 183 invalid decisions, 107 connected the final action to a record the agent had not acquired. That accounts for 58% of invalid decisions. An agent could identify a real issue from catalog metadata and still fail to inspect the record required to support its conclusion.
Unnecessary abstention was more common than unsupported submission, with 51 cases compared with 8. Another 17 runs chose the wrong final action class. A useful reviewer needs to recognize both when evidence warrants proceeding and when a specific gap should stop the decision.
Agents acquired at least one decision-changing item in 93% of opportunities that offered one, and the complete useful set in 80%. The main difficulty was carrying that evidence into a supported final decision.
Agent Decisions
The traces show how a review can succeed or fail depending on which record an agent inspects and how it uses that evidence. These six examples include three valid decisions and three invalid decisions.
Opus inspected the available specimen-geometry record and flagged its invalid source references, matching the reference decision.
Trace details
- request_characterizationspecimen geometry
- flag_source_defectinvalid source refs
Sonnet inspected both available calibration records and identified the required specimen-geometry record that could not be requested, matching the reference decision.
Trace details
- request_calibration_artifactx2
- localize_missing_evidencespecimen geometry
Gemini correctly escalated the review after four process-record requests that were unnecessary under the reference policy.
Trace details
- request_process_recordx4
- abstain_or_escalateno qualifying data
GPT inspected three valid calibration records, then abstained over policy limitations that those records were not meant to resolve. The reference policy supported submission.
Trace details
- request_calibration_artifactx3
- abstain_or_escalatemissing policy note
Opus chose the correct action to flag a source defect, but connected it to a microstructure record it had not requested. The decision failed the evidence check.
Trace details
- flag_source_defectimage-only item, not pulled
GLM inspected four process records and submitted them as support, although none resolved the decision. The reference policy required escalation.
Trace details
- request_process_recordx4, none useful
- submit_risk_packetbound all four
Calibration tasks often exposed stopping before all required evidence had been acquired. Packet-escalation tasks combined unnecessary requests, abstention, unsupported submission, and decisions attached to the wrong record. In that task family, 66% of requests acquired useful evidence.
Environment Compiler
The compiler turns archived experiments into decisions an agent can revisit. It connects each evidence item to its source record, controls what the agent can inspect, and scores its actions against the reference policy. Requests for measurements absent from the archive return unavailable.
The source workflows provide different kinds of materials evidence.
| Source workflow | Evidence available for review |
|---|---|
| IN718 tensile testing | Specimen geometry, process history, and calibration records |
| Ti-6Al-4V fatigue | Build and powder history, heat treatment, characterization, and calibration |
| IN718 laser-track calibration | Plate and powder records, scan strategy, spread layers, and melt-pool sections |
The compiler makes the history inspectable at the level of individual decisions. It separates which record the agent requested, what the returned evidence established, and how that evidence entered its final action.
Research Direction
The capability we want to improve is an agent's estimate of what the current evidence supports and what additional observation would change the decision. VOE-Bench measures that capability through evidence requests, stopping decisions, and claims connected to acquired records.
Our later SDL-1 study extends the question to experiment selection and learning from results. Scientific Autoresearch examines how evidence affects calculations and final claims during broader investigations.
For a materials program, this approach can make the reasoning behind a decision easier to inspect and evaluate. Our broader goal is to carry useful physical understanding into the next program, so that agents become better at deciding what can be reused and what needs another experiment.
Methods
Task construction and scoring
We attach each task to a recorded workflow before using a model to write its framing and select from the permitted evidence. Reference answers and outcomes come from the authored decision policy and archived records.
Deterministic checks reject tasks that expose private answers, refer to unavailable records, or use the wrong verifier. GPT-5.5 at medium reasoning effort then reviews framing and grounding against a six-part rubric after calibration on fixed accept and reject cases.2
All 104 tasks passed these checks. Every evaluated model received the same task set. Policy-visible state excludes gold labels, variant labels, answer keys, verifier-only values, split tags, and labels identifying useful evidence.
Agents choose among seven actions. They can request characterization, calibration artifacts, or process records, then flag a source defect, identify missing evidence, submit a risk packet, or abstain and escalate. Requests reveal the archived evidence, and a deterministic verifier scores the final action.
The verifier checks the accepted action class, its connection to an acquired record, and whether the acquired evidence supports it. All three must hold for a valid decision. Reference-equivalent paths also match the reference evidence acquisition and stopping behavior.
| Decision moment | Tasks | What it tests |
|---|---|---|
| Archived evidence selection | 23 | Which realized evidence to inspect next |
| Calibration artifact sufficiency | 23 | Whether a provenance package supports the decision |
| Source trustworthiness | 20 | Whether a source or process record is trustworthy |
| Packet escalation | 29 | Whether to submit, flag, localize, or escalate a risk packet |
| Missing evidence localization | 9 | Which specific binding record is absent |
Source records
| NIST source record | What it contains | Where it appears in replay |
|---|---|---|
| AMB2025-02, PBF-L IN718 tensile | Challenge statement, specimen geometry, CAD/FEA calibration bundle, calibration manifest, process lineage, submission template, answer workbook, and raw tensile traces. | specimen_geometry, cad_fea_summary, calibration_manifest, process records; answer workbook and raw tensile traces are verifier-only or audit-only. |
| AMB2025-03, PBF-L Ti-6Al-4V rotating-bending fatigue | Build and powder details, heat treatment, specimen preparation, chemistry, surface/XRD measurements, tensile/fatigue calibration, microstructure, XCT defect, fractography, and answer data. | Fatigue process records plus fatigue_build_powder_surface, fatigue_heat_treatment_surface, fatigue_tensile_calibration_surface, fatigue_800hip_calibration_surface, fatigue_microstructure_surface, and fatigue_xct_defect_surface; answer data is verifier-only. |
| AMB2025-06/07, IN718 laser-track pads | Plate material, powder feedstock, spread-layer images, scan strategy, melt-pool cross sections, submission templates, and answer keys. | Laser-pad process records plus plate_material_surface, powder_feedstock_surface, scan_strategy_surface, spread_layer_surface, and melt_pool_cross_section_surface; answer keys are verifier-only. |
Model comparison and baselines
Six models, three passes over the same 104 tasks
- Correct action class
- 98.7%
- Useful-request precision
- 87.2%
- Evidence regret ↓
- 0.785
- Cost score ↑
- 1.679
- Unsupported submissions
- 0 / 312
Pass rates
92.3% · 93.3% · 93.3%
- Correct action class
- 97.4%
- Useful-request precision
- 91.0%
- Evidence regret ↓
- 0.593
- Cost score ↑
- 1.722
- Unsupported submissions
- 0 / 312
Pass rates
90.4% · 92.3% · 92.3%
- Correct action class
- 96.5%
- Useful-request precision
- 89.1%
- Evidence regret ↓
- 0.577
- Cost score ↑
- 1.699
- Unsupported submissions
- 4 / 312
Pass rates
91.3% · 90.4% · 93.3%
- Correct action class
- 95.2%
- Useful-request precision
- 82.3%
- Evidence regret ↓
- 0.869
- Cost score ↑
- 1.590
- Unsupported submissions
- 0 / 312
Pass rates
88.5% · 89.4% · 90.4%
- Correct action class
- 94.9%
- Useful-request precision
- 86.0%
- Evidence regret ↓
- 0.772
- Cost score ↑
- 1.625
- Unsupported submissions
- 4 / 312
Pass rates
87.5% · 88.5% · 90.4%
- Correct action class
- 92.9%
- Useful-request precision
- 85.5%
- Evidence regret ↓
- 0.788
- Cost score ↑
- 1.603
- Unsupported submissions
- 0 / 312
Pass rates
84.6% · 87.5% · 88.5%
Valid decisions require the correct action and support from acquired evidence. Useful-request precision measures the share of requests that mattered. Evidence regret measures missed or wasted evidence, while cost score combines validity and efficiency. The reference policy reaches a cost score of 1.950.
Each plot uses the same 81–96% scale. Hollow points show pass rates, the filled point their mean, and the bar a 95% t-interval across three passes. Intervals are reported in percentage points.
Gemini 3.1 Pro had the highest valid-decision rate at 92.9%, with overlapping intervals among the top three systems. It made no unsupported submissions or wrong-class final actions across its three passes.
Claude Opus 4.8 had the highest useful-request precision and best cost score among the six systems, with no unsupported submissions. Claude Sonnet 4.6 left more useful evidence unread than Opus on the same Claude Code surface.
GLM 5.2 left the least useful evidence unacquired but made the most unnecessary requests. GPT-5.5 accounted for 12 of the 17 wrong-class final actions. Kimi K2.7 Code and GLM 5.2 made four unsupported submissions each.
These results describe each model with its native agent harness on the fixed task set.
The deterministic reference is valid on all 104 tasks by construction. Requesting every available record and then submitting reaches 28% validity, while always abstaining or escalating reaches 6%.
| Baseline policy | Evidence-grounded valid rate | Valid decisions |
|---|---|---|
| Deterministic reference | 100% | 104 / 104 |
| Request all, then submit | 27.9% | 29 / 104 |
| Random request | 17.3% | 18 / 104 |
| Always abstain or escalate | 5.8% | 6 / 104 |
| Always submit | 0% | 0 / 104 |
Additional decision counts
Of the 1,872 runs, 1,346 followed the reference path, 216 reached a valid decision with fewer useful records than the reference, and 127 reached one after an unnecessary request. Together these account for 1,689 valid decisions.
The agents submitted 470 risk packets compared with 522 under the reference policy, and escalated 149 compared with 108. Flagging source defects and identifying missing evidence tracked the reference more closely.
Percentages use all 1,872 runs. Signed values show agent decisions minus reference decisions.
Trained-policy diagnostic
We also evaluated Dynamical-30B-A3B from our earlier work on training scientific judgment. It was trained in synthetic materials-discovery environments using Crystalite to generate candidates and Materials Project stability data, with no AM Bench records in training.
In one pass over the 104 tasks, 95.4% of its evidence requests were useful and 62.5% of its final decisions were valid. These results show that precise acquisition can coexist with weaker final decisions. The prompt was calibrated on this evaluation surface, and the study did not include a matched untrained policy to measure the contribution of training.
The 95% Wilson interval for valid decisions was 52.9% to 71.2%. This describes task outcomes in one pass, while the frontier-model intervals describe variation across three passes.
Dynamical-30B-A3B made useful evidence requests at 95.4% precision and valid final decisions in 62.5% of tasks. Its result comes from one prompt-calibrated pass. The six comparison systems each ran three passes.
Sampling and evaluation scope
Each of the six model and harness combinations ran three passes over the same 104 tasks, producing 1,872 trajectories. The tasks reuse three workflows and 29 canonical evidence objects, with 498 evidence entries across their task-specific menus.
The frontier-model intervals are 95% t-intervals over three replicate passes at each provider's default sampling. They describe run-to-run variation on this fixed task set. Providers did not expose a shared deterministic seed.
All runs passed parsing, permitted-action, leakage, and forbidden-term checks. No run exhausted its action horizon or entered a duplicate loop. The evaluation measures evidence handling under an authored reference policy. Physical prediction accuracy, the representativeness of the archived measurements, and component qualification require separate assessment.
Citation
@article{barnes2026evidence,
author = {Barnes, Jarrod},
title = {Can a Self-Driving-Lab Agent Tell When the Evidence Is Enough?},
journal = {Dynamical Systems},
year = {2026},
url = {https://dynamicalsystems.ai/blog/benchmarking-self-driving-lab-agents}
}If you are working on materials evidence, scientific agents, or learning from experimental records, we would like to hear from you.