Dimensions

The tasks, datasets and scoring methods behind each column. 5 dimensions have recorded scores; 4 are still in development.

FIG. 01

Scientific tasks

score = metric ×10 · untested = — · not testable = N/A

Verify

“Is this claim supported by this abstract? Claim plus sentence-numbered evidence. Return SUPPORT, CONTRADICT or NEI and the sentence IDs.”

Metric: Verdict accuracy
Limitations: rationale overlap reported separately, never multiplied in; review found incomplete and disputed annotations
Dataset: SciFact dev, 300 claims, 5,183-abstract corpus
Licence: CC-BY-NC: score on it, do not redistribute

25 configurations tested

Ground

“Which paper established this? Give the DOI.”

Metric: Accuracy, the right DOI
Limitations: paired with grounding (the DOI resolves) and hallucination rate; two splits, canonical and long tail, shown side by side
Dataset: 12 canonical + 14 niche 2024-25 claims, DOIs API-confirmed
Licence: hand-authored

2 configurations tested

Find

“Pick the five sources this claim cites. Rerank a frozen BM25 candidate pool to exactly five IDs.”

Metric: recall@5
Limitations: paired with the candidate-pool oracle (the ceiling) and free BM25 (the floor); unreachable pools stay in the denominator
Dataset: SciFact claims with cited-source sets over the 5,183-abstract corpus
Licence: CC-BY-NC

5 configurations tested

Extract

“Turn this abstract into typed entities and directed relations.”

Metric: mean of entity F1 and relation F1
Limitations: malformed-output rate reported separately so format compliance cannot pass as accuracy
Dataset: SciERC test abstracts, published entity and relation annotations
Licence: research use

10 configurations tested

Analyze

“Run this analysis on this dataset and return the numbers. Code executes in a Seatbelt sandbox with NumPy, SciPy and pandas; no network, no host files.”

Metric: Typed numeric checks on the executed artifact, original and hidden perturbed inputs
Limitations: the story is never scored, only the artifact
Dataset: Palmer penguins, NIST StRD (Norris, Mavro, AtmWtAg), one synthetic fixture
Licence: public domain / CC0

3 configurations tested

Compute

“Write the code that reproduces this result.”

Metric: Run it, check the number
Limitations: held-out check to punish overfitting the visible test
Dataset: wrap SciCode or CORE-Bench; only an arithmetic stand-in exists today
Licence: Apache

Not yet tested

Synthesize

“Answer this research question with citations.”

Metric: Citation F1
Limitations: a calibrated judge ships only with its kappa against a 100-400 item human meta-eval
Dataset: wrap ScholarQABench
Licence: Apache

Not yet tested

Review

“Find the weakness in this paper.”

Metric: Judge against expert-written weaknesses
Limitations: A/B and B/A averaged; no model grades itself
Dataset: AAAR-1.0 PaperWeakness; NLPeer
Licence: mixed

Not yet tested

Hypothesize

“Propose the next experiment.”

Metric: Arena only: Bradley-Terry over expert votes
Limitations: the best LLM judge reaches 65% alignment with experts on scientific-literature tasks, so a judge score here would be a guess
Dataset: HypoBench, IdeaBench as seeds; no votes exist yet
Licence: mixed

Expert evaluation planned

Scores use each dimension’s own metric: 8.4 in Verify means 84% verdict accuracy; 3.3 in Extract means F1 0.33. There is no combined score.

FIG. 02

Excluded pilot runs

Retained for reference

Run	Why it is not a dimension	Where
sci_qa	20 multiple-choice science questions; every model scores 100%. Saturated; not a dimension.	outputs/sci_qa/results.json
compute (stand-in)	24 arithmetic MCQ via code execution; a placeholder for SciCode. Every model scores 100%. Not a Compute score.	outputs/compute/results.json
litsearch	LitSearch open-web finding, 3 queries run. Too small to show.	outputs/litsearch/results.json
content pilot	12 questions × 6 arms over five books and papers with model-drafted reference answers; a development pilot, not gold.	outputs/content-pilot/