# Hidden-set simulation: paired benchmark

Deployment interpretation, diagnostic findings and reproduction instructions: [ANALYSIS.md](ANALYSIS.md).

One pass over all 376 frozen development scenarios using jev-1.13.0. Both arms receive the same closed-information log, freshly retrieved V4 context and identical Jev shortlist. The candidate additionally infers wiki sets and simulates them. No deployment was performed.

| Arm | Agreement with original reference | Percentage |
|---|---:|---:|
| Saved behavior: V4 + top4, no hidden-set simulation | 251/376 | 66.76% |
| Candidate: inferred sets + simulation | 295/376 | 78.46% |

55 corrections; 11 regressions. Accepted action in top four: 374/376.

## Simulation coverage

Evidence reached the final decision in 133/376 cases. Within that subset: baseline 79, candidate 117; 42 corrections and 4 regressions.

The remaining 243 cases use the production fallback. Different unavailability explanations may change their final choices even without simulated evidence. Their changes must not be credited to simulation.

- 127: No legal wiki-based completion for observed participants: Toxicroak.
- 23: Conditional simulation preparation failed at brief; no private engine error text or incomplete evidence is sent to Jev.
- 23: No legal wiki-based completion for observed participants: Slowbro.
- 20: No legal wiki-based completion for observed participants: Toxicroak, Machamp.
- 10: No legal wiki-based completion for observed participants: Machamp.
- 10: No legal wiki-based completion for observed participants: Vaporeon.
- 8: No legal wiki-based completion for observed participants: Jolteon.
- 8: No legal wiki-based completion for observed participants: Greninja.
- 4: No legal wiki-based completion for observed participants: Rotom.
- 4: No legal wiki-based completion for observed participants: Garganacl.
- 2: No legal wiki-based completion for observed participants: Golurk.
- 1: No legal wiki-based completion for observed participants: Gengar.
- 1: No legal wiki-based completion for observed participants: Toxicroak, Pelipper.
- 1: No legal wiki-based completion for observed participants: Pelipper.
- 1: No legal wiki-based completion for observed participants: Greninja, Vaporeon.

## Cost and latency

1654 unique API attempts; 0 failed attempts; 7263620 input and 160944 output tokens. Identical retrieval/shortlist requests are shared within each pair, with no repeated sampling. Per-arm logical costs count those shared calls in both arms. Timing uses three concurrent workers and excludes no-simulation cache replay from claims about speed. See results.json for API latency and per-case measurements.

## Interpretation limits

The original reference answers were created with open team sheets. Only those sheet lines were removed; public move/ability/item reveals, requests and native actions remain unchanged. Some reference answers can depend on information now hidden. These scores measure agreement with that original oracle, not verified optimality under partial information or battle win rate. The old open-information score of 371/376 is not a comparable control. This is a known development corpus, not a held-out test.

The benchmark omits the demo's extra team brief and memory calls in both arms; public history remains in the log. The shared observer was corrected to retain explicitly public bench faints even when earlier entry events are absent in synthetic logs. Unknown sets and selected participants are never filled from the original sheets. All inference uses the production pipeline, with no answer keys available. Labels were opened only after validating all saved model artifacts.