# Deployment assessment

The candidate merits a private pilot: it improves agreement with the original benchmark reference by 44/376 cases (+11.70 percentage points). This experiment does not establish readiness for a public release. No server was restarted or deployed as part of the benchmark.

The main effect is supported by the 133 cases whose final request actually contained the simulation brief: 79/133 (59.40%) for the control versus 117/133 (87.97%) for the candidate, with 42 corrections and four regressions. These are the same cases and the same first-stage choices. Coverage is selective; these percentages must not be extrapolated to the other cases or to full battles.

Those delivered briefs cover 22,440 successfully resolved branches, with zero failed or omitted branches within their declared budgets. This does not make the opponent hypotheses exhaustive.

The other 243 cases received no simulated evidence. Their net gain of six answers is associated with different fallback explanation text, not a demonstrated simulation benefit. Overall, there were 55 corrections and 11 regressions; all 376 cases produced valid decisions and no API calls failed.

## Why coverage is low

- 220 cases could not build legal wiki-based opponent completions. Nine affected species have a wiki card but no structured build templates: Toxicroak, Slowbro, Machamp, Vaporeon, Jolteon, Greninja, Rotom, Garganacl and Golurk. Gengar and Pelipper have templates, but some observed evidence rejects all available completions. A species article is not the same as a usable set library.
- 23 cases successfully ran simulations but could not deliver their complete brief within the configured size limit. Replaying the formatter without API calls confirmed byte-budget rejections in all 23. Twenty-two exceed the V2 40,000-byte limit; one exceeds the underlying V1 formatter limit. No evidence was truncated to force a successful comparison.
- Only 133/376 (35.37%) final decisions received simulation evidence. The raw number of simulated cases or branches would overstate useful coverage if it included rejected briefs.

See `wiki-diagnostics.json` and `brief-diagnostics.json` for the checks.

## Regressions worth investigating

All four regressions where the model received simulations are mid-turn Baton Pass replacements:

| Cases | Original accepted replacement | Candidate replacement |
|---|---|---|
| bulk-15-switch-ghost-entry-b / d | Indeedee | Pincurchin |
| bulk-15-switch-water-entry-a / c | Pelipper | Torkoal |

The first pair changes terrain and the second changes Weather Ball's type. Jev assigned only 53–55% to these choices, with confidence 0.05–0.11. Investigate the reconstruction of pending actions and the conditional opponent hypotheses before interpreting these as ordinary move-selection failures. The original reference may rely on hidden information; none of these observations justify giving the simulator private pending opponent commands.

## Cost and timing

The paired experiment made 1,654 unique API calls, consuming 7,263,620 input and 160,944 output tokens. Identical retrieval and shortlist calls were reused within each pair; neither arm received the other's final decision. These token counts are provider-reported usage, not a dollar-cost estimate.

With shared calls counted in each arm's logical budget, the control used 1,128 calls and the candidate 1,278 calls. Median summed API latency per scenario was 0.629 s versus 0.749 s; p95 was 0.970 s versus 1.205 s. These are API timings under three concurrent workers, excluding engine work, and most candidate cases fell back without simulation. They are not full-turn latency estimates for the playable demo.

## Recommended next step

Keep the saved baseline available and use the candidate only for private review for now. Before a wider release, add a legal completion strategy for species without templates, deliver large briefs without losing evidence, and audit the mid-turn replacement cases. Retest those changes on this fixed corpus and a separate closed-information set whose expected actions are justified by observable evidence.

## Reproduction and validation

Run from `decision-lab`:

```powershell
node --import tsx scripts/benchmark-hidden-sets.ts --mode prepare --out runs/NEW_RUN
node --import tsx scripts/benchmark-hidden-sets.ts --mode dry-run --out runs/NEW_RUN
node --env-file-if-exists=.env --import tsx scripts/benchmark-hidden-sets.ts --mode run --out runs/NEW_RUN --live
node --import tsx scripts/evaluate-hidden-sets.ts --run runs/NEW_RUN --out experiments/NEW_REPORT
```

Preparation freezes inputs, code, configuration and wiki hashes before inference. Changed sources require a new run. All model artifacts are checked before the separate evaluation process reads reference answers. The protocol explicitly excludes the optional demo team-brief and memory calls from both arms, while retaining public battle history in the log.

Type checking passed. The test suite passed 694 tests, with one optional server integration test skipped. The new regression test checks all 376 projected native menus and preserves public faint evidence without revealing opponent sheets. Both arms use the same small observer correction; the saved baseline tag remains unchanged.

The reference scores come from an open-information development corpus projected to closed information. They are agreement measures, not win rates or proof of optimal play with hidden sets. The historical 371/376 open-information result is a different condition and is not the control in this experiment.