Task and Environment Design
How SDL-2 builds each world, prices each experiment and verifies each decision.
The requirement
We ask the agent to find a process for a new powder lot, or to show that the declared process space cannot meet the requirement. A nominated route must give 3 specimens from one new preparation that each reach 10 kOe coercivity and 11 MGOe energy product, with a modeled probability of at least 50% that all of them pass both.1 Ending without a decision is allowed and does not pass.
The agent starts from historical records of earlier lots, a starting panel for the current lot and a scheduled release of results. A supplied physics model is deliberately imperfect, so its estimates are hypotheses to test rather than the answer.
The process space
Degassing is either skipped or run at 600, 625 or 650 °C for 4 to 12 minutes. Sintering peaks at 825 or 875 °C, and full pressure is applied on heating, at the dwell, or on cooling at one of four temperatures. Annealing uses no step, one step or two ordered steps, each at one of five temperatures. Fixed conditions cover the heating rate, the hold, the pressure and a 1 g charge per specimen.
The experiment economy
Credits are SDL-2’s experiment currency. Each purchase is priced to model its experimental cost, and credits are separate from what the model’s API calls cost. Each investigation has 570 credits and up to 12 new specimens, and exceeding the allowance fails the outcome. Density alone does not establish magnetic performance, a coercivity screen measures coercivity only, and a full curve adds remanence and energy product.
| Purchase | Credits |
|---|---|
| New sintered specimen | 10 |
| New degassing batch | 15 |
| Anneal step per specimen | 6 |
| Density | 1 |
| Coercivity screen | 12 |
| Full magnetic curve | 50 |
| SEM | 20 |
| ICP-OES chemistry | 25 |
| Sintering log | 0 |
Stages and tools
An investigation runs in 3 stages separated by 2 release boundaries. Up to 2 full curves return at each boundary, so at most 4 purchased curves can inform the final decision. We ask the agent to define and adopt an executable model, record forecasts before results return, buy evidence with a stated purpose and then decide.
Lab tools
Agents inspect evidence, compute with models, adopt revisions, preview and submit experiments and record decisions. The full harness adds ordinary file and code tools. A separate reference simulator describes another historical setting, and the evaluator excludes its results from the task world.
Scoring
- Rung 1 · Outcome
Is the decision correct?
A nomination passes if the world’s joint confirmation probability meets the standard within the allowance. A restriction passes only if no declared route can.
- Rung 2 · Justified
Does the evidence support it?
The evaluator’s posterior must give at least 50% to a nominated route meeting the standard, or at most 20% to any route meeting it for a restriction.
- Rung 3 · Record facts
Is the record consistent?
Deterministic checks of purchases, the experiment ledger, measurements and the typed final decision.
- Rungs 4 and 5
Judgment and expert audit
Residual questions and blinded expert review sit above the headline and carry no weight in it.
Outcome uses the world’s expected joint probability for the nominated route rather than a single random confirmation draw. Justified asks whether the agent’s purchased evidence supports the decision under the declared prior.
Admission and scope
The test panel has 23 held-out and 6 in-distribution worlds. Each world passed admission checks before evaluation.
Admission thresholds
A scripted investigation must reach a supported decision of the correct kind within the allowance in at least 80% of evidence replicates. A declared panel of prior-only shortcuts must stay at or below 25% success on the held-out stratum. The scripted investigation must beat always restricting and a cheap screening reference by at least 30.0 percentage points.
A pass is a correct, supported decision in the simulated world under this process space, budget and evaluator. It is not a manufactured magnet, a production yield or a general measure of scientific skill. The simulator reports Archimedes density and apparent magnetic curves, while the experimental campaign measures geometric density and corrected curves, and aligning the two is part of the next version.