All configurations / harness
cited abstract supplied
- Configuration
- The claim's cited abstract is handed to the model before it answers. No search, no tools; the evidence document itself.
- Source
- gym.py scifact · contestants V_ev_*
- Recorded runs
- 300 scored outputs · $2.71 measured model spend · runs on 2026-08-31
- Access
- open harness · standard tools
FIG. 01
By dimension
Scores /10 · Paired p-values are exploratory and unadjusted8.40Verify
Compared with baselinen=300 · verdict accuracy · evidence in hand · Δ +3.63 vs Opus 4.8 · 123W 14L 163T on 300 paired · p<.001 (unadjusted) · $0.0090/task · 2026-08-31
ran at a 512-token cap; 179 naked responses hit it; history row, not comparable to the 4096-cap matrix; naked 0.477 at $0.0125 per task; the harness is cheaper because the naked model burns tokens hedging Test: exact McNemar on the tasks both cells answered.
Run provenance
Report path: outputs/scifact/results_opus48_n300_2026-07-13.json
Task pack: scifact-dev-300
Run: matrix-opus-4.8
—Ground
Not yet tested.
—Find
Not yet tested.
—Extract
Not yet tested.
—Analyze
Not yet tested.
—Compute
Not yet tested.
—Synthesize
Not yet tested.
—Review
Not yet tested.
—Hypothesize
Not yet tested.