All configurations / harness
cited abstract supplied
- Configuration
- The claim's cited abstract is handed to the model before it answers. No search, no tools; the evidence document itself.
- Source
- gym.py scifact · contestants V_ev_*
- Recorded runs
- 300 scored outputs · $5.01 measured model spend · runs on 2026-08-31
- Access
- open harness · standard tools
FIG. 01
By dimension
Scores /10 · Paired p-values are exploratory and unadjustedvoidVerify
Serving failuren=300 · verdict accuracy · evidence in hand · Δ +0.47 vs Fable 5 · 30W 16L 254T on 300 paired · p=.054 (unadjusted) · $0.0167/task · 2026-08-31
two of every three calls returned zero output tokens; a serving failure, not a score; naked 0.237 at $0.0169 per task; the harness is cheaper because the naked model burns tokens hedging Test: exact McNemar on the tasks both cells answered.
Run provenance
Report path: outputs/scifact/results_matrix_fable5.json
Task pack: scifact-dev-300
Run: matrix-fable-5
—Ground
Not yet tested.
—Find
Not yet tested.
—Extract
Not yet tested.
—Analyze
Not yet tested.
—Compute
Not yet tested.
—Synthesize
Not yet tested.
—Review
Not yet tested.
—Hypothesize
Not yet tested.