Research

Teaching AI to Explore the Physical World

Making scientific exploration measurable

Exploring the physical world comes down to deciding, from a few costly and noisy experiments, what holds for a system no one has measured. SDL-2 measures how well AI agents make that decision, choosing their own experiments to determine whether a new magnet powder can be made reliably, with every decision graded against a known answer. Frontier models reached correct, supported decisions in about half of the investigations, and the results point to reasoning under variation as a central capability to improve.

PublishedNovember 1, 2026
EvaluationSDL-2

Motivation

We believe hardware teams will design around capabilities they ask to have developed, with materials development keeping pace with hardware. SDL-2 measures one step that comes before that future, deciding from evidence whether a process will reliably meet a stated performance requirement.

Our meta-learning thesis is that agents can learn how to investigate unfamiliar physical systems through verified experimental experience. Experience should reduce the experiments needed to establish the next capability, even when the answer itself does not transfer. What should transfer is the relationship between evidence and action, including which uncertainty matters, what observation could resolve it, when an explanation needs revision and when the record supports commitment.

One magnet that meets a requirement does not establish a process. We need evidence that a fresh batch made the same way will reliably meet it. An unsuccessful search raises a different question, whether the evidence rules out the available routes. Every new powder, machine, scale or specification reopens the question.

Experimental measurements reveal the outcomes of only the routes we run. They can support a process decision, but usually cannot tell us whether an untested route would have worked. SDL-2 moves the question into a digital twin of the process, where every route has a known outcome, so each decision is graded for correctness and for whether the agent’s own measurements support it. Our earlier benchmarks, VOE-Bench and SDL-1, tested parts of this judgment on real experimental records.

We start with NdFeB magnets because the processing that makes them harder to demagnetize often makes them weaker, and because our experimental campaign can test what the twin predicts.

The evaluation

Each world is a simulated NdFeB powder whose properties are hidden from the agent. The agent either nominates one of 8,064 routes, set by ten processing controls from degassing to annealing, or concludes that none works. A route works if three magnets made by it from a fresh batch all reach 10 kOe coercivity and 11 MGOe energy product with probability at least 95%. Experiments are priced in credits, from 2 for a density reading to 50 for a full magnetic curve, and a claim fails if the investigation spends more than 570 credits or makes more than 12 new magnets.

The twin is calibrated on measurements from our experimental campaign with Rensselaer Polytechnic Institute (RPI), which cleared its data for this release.

The language-model agents work through scientific tools, and the two classical methods call the same tools through fixed policies.

SDL-2 runs these tools on the twin alone. Each investigation has a single investigator. Every language model runs in the same agent harness, Pi, offline and text-only. The instruments return 14 quantities with measurement noise, from density and magnetic curves to grain size. The agent starts from the experimental campaign’s current physics model and can test, refit or replace it. Each hidden powder is drawn from the supplied model’s own uncertainty, so the agent must infer which parameters and magnetic mechanism hold for that powder.

Each request and its result are recorded, and every decision analyzed below is reconstructed from this record.

Each final decision is scored twice by fixed rules. Outcome asks whether it is true of the hidden world within the resource limits, and Supported whether the investigator’s own measurements meet a fixed evidence rule. Both, the headline score, requires both. A correct, supported decision draws on the whole investigation. The agent must form useful hypotheses when the supplied model misleads, reason about variation from few observations, revise after surprising evidence or persist when the evidence warrants it, spend the remaining budget well and carry its evidence into a defensible decision.

How agents explore and investigate

Model calibration

Density is fit to the campaign’s magnets and earlier samples, and the anneal’s effect on density to one campaign pair. Magnetic properties are fit to earlier samples made at 850 and 950 °C, so values at lower temperatures or densities are extrapolations, and finer feedstock rests on stated assumptions. No magnetic curve had been measured on a current campaign magnet when the board ran. Agents receive these limits with the model.

Worlds and grading

A nomination is supported when the investigator has full magnetic curves on at least two magnets of the exact route and every magnetic reading of that route passes. A restriction is supported when the agent has full magnetic curves on any of the world’s 100 routes closest to passing, and every such reading fails. The 32 evaluation worlds hold 16 with a working route and 16 without, and the agent is told that about half have one. Worlds were selected on their hidden properties, with quotas fixed before any agent ran. The supplied model’s favorite route works in only one, and 15 of the 16 without a route are near misses. A magnetic reading passes when the stated noise model assigns at least 95% probability that the measured magnet meets the limit or limits tested, and fails at 50% or below. Without experiments, neither restricting every route nor nominating the supplied model’s favorite route meets the evidence rule. The evidence rule is a fixed standard for a defensible decision. Meeting it does not by itself show that a nominated route is reliable or that no route works. The scoring rules are in the release documentation.

Results

Overall performance

We ran six language models and two classical experiment-design methods once on each of 32 worlds, with every language model at high reasoning effort. Frontier models reached correct, supported decisions in 43.8%–50.0% of worlds, compared with 25.0%–28.1% for the Flash models and 28.1% for each classical method. Scored on correctness alone, frontier models reached 62.5%–78.1%, though that score cannot show whether a decision followed from the agent’s own measurements.

On this board, investigators more often supported a correct rejection than established a working process. In the 16 worlds with a working route, frontier models reached correct, supported decisions in 37.5%–43.8% each, each Flash model in 6.3% and each classical method in 25.0%. In the 16 worlds without one, the corresponding rates per investigator were 50.0%–62.5%, 43.8%–50.0% and 31.3%.

Evaluation worlds · simulated

Decision scores

GPT-6 Astra
Frontier model
50.0%
16 of 32
Claude Sonnet 5.5
Frontier model
46.9%
15 of 32
Gemini 4 Argon
Frontier model
46.9%
15 of 32
Claude Opus 5.5
Frontier model
43.8%
14 of 32
DeepSeek V4.1 Flash
Flash model
28.1%
9 of 32
Bayesian optimization
Classical baseline
28.1%
9 of 32
Design of experiments
Classical baseline
28.1%
9 of 32
GLM-5.3 Flash
Flash model
25.0%
8 of 32
Full results and spending
InvestigatorBothOutcomeSupportedMedian creditsMedian API cost
GPT-6 Astra
Frontier model
16/32
31.9%–68.1%
23/32
53.3%–86.3%
18/32
37.7%–73.6%
569.0$8.02
Claude Sonnet 5.5
Frontier model
15/32
29.1%–65.3%
20/32
43.7%–78.9%
16/32
31.9%–68.1%
524.5$1.52
Gemini 4 Argon
Frontier model
15/32
29.1%–65.3%
22/32
50.0%–83.9%
19/32
40.6%–76.3%
561.0$12.48
Claude Opus 5.5
Frontier model
14/32
26.4%–62.3%
25/32
60.0%–90.7%
17/32
34.7%–70.9%
503.5$22.46
DeepSeek V4.1 Flash
Flash model
9/32
13.7%–46.7%
16/32
31.9%–68.1%
12/32
21.1%–56.3%
556.0$0.10
Bayesian optimization
Classical baseline
9/32
13.7%–46.7%
15/32
29.1%–65.3%
16/32
31.9%–68.1%
443.0$0.00
Design of experiments
Classical baseline
9/32
13.7%–46.7%
24/32
56.6%–88.5%
10/32
16.1%–50.0%
544.0$0.00
GLM-5.3 Flash
Flash model
8/32
11.5%–43.4%
16/32
31.9%–68.1%
13/32
23.7%–59.4%
535.5$0.54

Counts are decisions out of 32 worlds, with exact 95% binomial intervals beneath. Credits come from an allowance of 570. API cost is the median per attempt, and the classical baselines make no model calls. Every language model ran in the same harness at high reasoning effort.

Points show the share of worlds, with exact 95% binomial intervals. Bayesian optimization lost every first anneal request to an interface defect, so its row describes the policy as it ran. None of the pre-registered paired comparisons reached statistical significance after Holm correction, so the ordering is descriptive.

Evidence at commitment

Frontier decisions were correct in 60 of 70 investigations that met the evidence rule, compared with 30 of 58 that did not. The same direction held for the Flash models and classical methods.

One powder, two investigations

Outcome and evidential support

Passing curve, nominated routeFailing curve, another routeCredits spentNominated
Claude Opus 5.5
4 magnets of its route
Outcome
Supported
GPT-6 Astra
1 magnet of its route
Outcome
Supported
Both investigations received the same hidden powder and nominated a route that works on a fresh batch. Supported requires full magnetic curves on at least two magnets of the nominated route, with every magnetic reading of that route passing. Dots mark full magnetic curves, placed at the credits spent when each returned.

The unsupported frontier decisions left a median of 11.5 of 570 credits unspent. Yet in 16 of the 19 frontier investigations that measured a route that works in the hidden world without establishing it, at least 150 credits remained when that route was first measured. These histories point to a gap between acquiring evidence and choosing the follow-up that establishes what it warrants.

Of those 19 investigations, 13 occurred in worlds where the best route only narrowly cleared the requirement. In the other six, the agent chose another route or nominated the working route without meeting the evidence rule. Gemini 4 Argon, for example, confirmed a route across batches with every reading passing, then nominated a less-replicated route because its measurements showed larger margins. The confirmed route works in the hidden world, and the nominated one does not.

Search, boundaries and baselines

On worlds with a working route, frontier models recovered after a missed first design in 32 of 47 investigations, the Flash models in 7 of 27 and the classical methods in 8 of 20. Here, first design includes later processing stages of the initial magnets, and recovery means making a magnet by a route that works in the hidden world. It does not require a supported final nomination.

Frontier conclusions that no route exists were wrong in 7 of 16 cases without a full magnetic curve on any of the 100 routes closest to passing, and in 10 of 55 with one. Design of experiments nominated a working route 8 times, but three of its four unsupported correct nominations rested on a single magnet. These comparisons were defined after the run and share worlds, so they are descriptive.

Reasoning under variation

Frontier decisions were correct in 17 of 44 investigations near the requirement, compared with 73 of 84 elsewhere. These cases require distinguishing a process that works with little margin from one whose magnets sometimes pass but whose batches are unreliable.

Among investigations from all methods that measured multiple magnets of their most-measured route, a rule based on every raw reading clearing both magnetic limits was right about whether the world contained a working route in 76 of 92 cases away from the requirement and 24 of 49 near it. In the near-requirement group, lower observed spread ranked a world with a working route above one without in 531 of 598 comparisons. This is a retrospective association from 49 selected investigations sharing 11 worlds.

Evaluation worlds · simulatedinvestigations with two or more magnets of their main route

Pass rule and measurement spread

Away from the requirement92 investigations
Pass rule
82.6%76/92
Investigator decisions
70.7%65/92
Near the requirement49 investigations
Pass rule
49.0%24/49
Investigator decisions
36.7%18/49
Spread near the requirement% of the requirement
World with a working route
World without one
The upper chart compares correct investigator decisions with a rule for identifying worlds that contain a working route. The rule classifies a world as having a working route when every raw reading on the most-measured route clears both magnetic limits. The dashed line marks the midpoint. The lower chart shows the range of readings on that route near the requirement. Filled points nominated a route, and open points concluded none works.

We coded the reasoning behind all 66 near-requirement claims from the language models with model identities and outcomes hidden. In the five worlds where batch variation prevents reliability, claims that used observed variation to judge the next batch were correct in 12 of 15 investigations. Claims that rested on passing magnets or the supplied model’s estimate were correct in 1 of 8. On one such powder, GLM-5.3 Flash obtained markedly different results across batches, wrote that “a second independent preparation reproduced it” and nominated a route. Claude Opus 5.5 repeated its passing route on a fresh batch and used the resulting failures and spread to conclude that no route would meet the requirement. The first claim is false in the hidden world, and the second is true.

Frontier investigations measured an exact route across batches in 79 of 128 cases and multiple magnets from the same batch in 60 of 128. We identified repeated magnetic or density measurements of the same piece at the same processing stage in 4 of 128. These designs probe different sources of variation. Their scientific value depends on which uncertainty could change the process decision. Together, the findings implicate reasoning under variation as a central capability to improve, while leaving its causal role to be tested.

Variation analysis

The near-requirement group contains six worlds where the best route’s typical magnet clears the requirement by less than 5%, and five where a typical magnet can pass but a fresh batch does not reliably meet the requirement. The margin is relative to the required magnetic performance. The grouping is exploratory. No investigator reached a correct, supported decision in the 48 investigations on the narrowly feasible worlds.

The spread analysis pools language models and classical methods. The main route is the one with the most full magnetic curves. Spread is the range of full magnetic readings relative to the requirement, using the smaller of the coercivity and energy-product ratios. The pairwise comparisons are dependent and do not measure accuracy on new powders. The transcript comparison is also exploratory. Identified rereadings concern magnetic or density measurements on an unchanged piece, and do not by themselves establish successful separation of instrument, magnet and batch variation.

Experimental campaign

Our experimental campaign with RPI is developing a reproducible process for strong NdFeB magnets that resist demagnetization without dysprosium or terbium. We study how powder preparation, heating, pressure and annealing change microstructure and magnetic performance. RPI makes and characterizes the magnets, while AI agents analyze the experimental record, compare explanations and propose the next experiments.

The campaign supplies the experimental evidence for the twin and tests the physics it represents. We will use magnetic characterization to test and refine the model’s link between processing, microstructure and performance. The digital twin and evaluation will co-evolve with the experimental campaign. Each released evaluation stays fixed so results remain reproducible.

Evaluation framework

What’s next

Our next research objective is to identify what experience and learning signal would make an investigator better at discovering, testing and acting on scientific structure in unfamiliar worlds under limited experimental budgets. The curriculum will vary which experiments are useful as mechanisms, sources of variation and available evidence change.

Learning will draw on the agent’s own investigations and the alternative strategies it explores in the twin, guided by verified final outcomes. Progress means more discovery and more correct, supported decisions on combinations and conditions the agent has not seen. SDL-2 is open, and we welcome researchers and materials scientists to extend it with new agents, learning methods and experimental evidence.

If you build hardware limited by its materials, or agents that learn from experiments, we would love to hear from you.

Optimal experimental design chooses experiments for a stated goal and can adapt to earlier results. BoxingGym and ACED-Bench test choosing experiments and acting on the results, From Prompts to Posteriors tests updating beliefs on supplied evidence, and SuffiCE trains confidence to track the evidence available. MatFlowBench checks computational materials workflows and replays archived measurements with executable verifiers. SciUniverse runs lab procedures on real instruments. FrontierXRD evaluates diffraction analysis with an expert-calibrated judge, and BixBench3 grades study-scale computational biology. Ríos-García et al. find that scientific agents often leave the evidence they gather unused, and Zhang and Li show that final-answer accuracy can far exceed accuracy grounded in the supplied evidence. SDL-2 evaluates whole investigations under an experimental budget, with a simulated outcome for every route and separate scores for correctness and evidential support.

Citation

@article{barnes2026sdl2,
  author  = {Barnes, Jarrod},
  title   = {Teaching AI to Explore the Physical World},
  journal = {Dynamical Systems},
  year    = {2026},
  note    = {SDL-2},
  url     = {https://dynamicalsystems.ai/blog/sdl-2}
}