Extract entities and relations
Turn an abstract into named concepts and the relationships between them.
Gain depends on output handling
Mean F1 rose from 0.240 to 0.334 with the local procedure. Most observed improvement came from avoiding malformed outputs.
Skill tested: Local procedure: local-scierc-procedure. “AI alone” uses the task instructions and required tools; the other setups add a checklist or skill guidance. Everything else is held the same.
How this was scored, and what it does not establish
F1 balances missed items and incorrect additions on a scale from 0 to 1. We average entity F1 and relation F1; malformed output receives zero. It is not a percentage of papers extracted correctly.
- The skill condition is a local SciERC procedure, not an upstream scientific-skill package.
- Eight AI abstracts; published annotations have not been independently reviewed.
- Format failures receive zero. A shared schema-enforcement control is needed before attributing the difference to extraction accuracy.
- No PDF parsing, tables, or scientific results extraction were tested.
SKILL.md supplied; no explicit reference-file reads recorded.
Extraction: a check of output formatting. Removing only wrappers and extra outer fields from the saved answers changes mean F1 to 0.400 for AI alone, 0.341 with a checklist, and 0.391 with the skill. The apparent skill gain disappears. Entity and relation content is unchanged. This is a post-hoc sensitivity check, chosen after inspecting failures; it is not a new experiment or evidence that the skill is worse. The original scores stay in the table. Diagnostic data ↗
Examples and actual answers
Open an example to see all three attempts.
Preserving eye contact in video callsSciERCAI aloneInvalid formatWith checklist0.349With skill0.287
Identify stereo-matching and view-generation methods, and map how they support eye contact and reduce visual artifacts.
Source dataset ↗ · Source cluster: scierc:ICCV_2003_151_abs
AI alone
The submitted structure failed the required schema and received zero.
Entity F1: 0.000 · Relation F1: 0.000
Output excerpt
{
"text": {
"entities": [
{
"text": "novel view generation",
"type": "Task"
},
{
"text": "one-to-one teleconferencing applications",
"type": "Task"
},
{
"text": "video streams",
"type": "Material"
},
{
"text": "two cameras",
"type": "Material"
},
{
"text": "computer monitor",
"type": "Material"
}
],
"relations": [
{
"head": "dynamic-programming, stereo algorithm",
"tail": "novel-view generation",
"type": "USED-FOR"
},
{
"head": "three-plane graph",
"tail": "occlusion labeling",
"type": "USED-FOR"
},
{
"head": "three-plane graph",
"tail": "dense-stereo dynamic-programming",
"type": "HYPONYM-OF"
},
{
"head": "geometric derivation",
"tail": "novel-view synthesis",
"type": "USED-FOR"
},
{
"head": "minimum-cost surface",
"tail": "novel-view synthesis",
"type": "USED-FOR"
}
]
}
}With checklist
Entity F1 0.441; relation F1 0.258.
Entity F1: 0.441 · Relation F1: 0.258
Output excerpt
{
"entities": [
{
"text": "algorithm",
"type": "Method"
},
{
"text": "novel view generation",
"type": "Task"
},
{
"text": "one-to-one teleconferencing applications",
"type": "Task"
},
{
"text": "video streams",
"type": "Material"
},
{
"text": "two cameras",
"type": "Material"
}
],
"relations": [
{
"head": "algorithm",
"tail": "novel view generation",
"type": "USED-FOR"
},
{
"head": "novel view generation",
"tail": "one-to-one teleconferencing applications",
"type": "USED-FOR"
},
{
"head": "virtual camera",
"tail": "eye contact",
"type": "USED-FOR"
},
{
"head": "dynamic-programming, stereo algorithm",
"tail": "novel-view generation",
"type": "USED-FOR"
},
{
"head": "three-plane graph",
"tail": "dense-stereo dynamic-programming",
"type": "USED-FOR"
}
]
}With skill
Entity F1 0.386; relation F1 0.188.
Entity F1: 0.386 · Relation F1: 0.188
Output excerpt
{
"entities": [
{
"text": "novel view generation",
"type": "Task"
},
{
"text": "one-to-one teleconferencing applications",
"type": "Task"
},
{
"text": "video streams",
"type": "Material"
},
{
"text": "two cameras",
"type": "Material"
},
{
"text": "computer monitor",
"type": "Material"
}
],
"relations": [
{
"head": "dynamic-programming, stereo algorithm",
"tail": "novel view generation",
"type": "USED-FOR"
},
{
"head": "three-plane graph",
"tail": "dense-stereo dynamic-programming",
"type": "USED-FOR"
},
{
"head": "three-plane graph",
"tail": "occlusion labeling",
"type": "USED-FOR"
},
{
"head": "geometric derivation",
"tail": "novel-view synthesis",
"type": "USED-FOR"
},
{
"head": "novel-view synthesis",
"tail": "minimum-cost surface",
"type": "USED-FOR"
}
]
}Recorded case extract-scierc-ICCV_2003_151_abs · Run four-track-canary-v3. Output excerpts are bounded; the full submitted answer was scored.
Choosing Thai classifier wordsSciERCAI alone0.500With checklist0.500With skill0.399
Identify corpus-based methods, noun–classifier associations, and linguistic constraints, then map how they support classifier selection.
Source dataset ↗ · Source cluster: scierc:C94-1091
AI alone
Entity F1 0.600; relation F1 0.400.
Entity F1: 0.600 · Relation F1: 0.400
Output excerpt
{
"entities": [
{
"text": "algorithm",
"type": "Method"
},
{
"text": "classifier word",
"type": "OtherScientificTerm"
},
{
"text": "noun",
"type": "OtherScientificTerm"
},
{
"text": "Thai language",
"type": "Material"
},
{
"text": "classifier",
"type": "OtherScientificTerm"
}
],
"relations": [
{
"head": "algorithm",
"tail": "classifier selection",
"type": "USED-FOR"
},
{
"head": "default rule",
"tail": "classifier selection",
"type": "USED-FOR"
},
{
"head": "rule-based approach",
"tail": "classifier selection",
"type": "USED-FOR"
},
{
"head": "corpus-based method",
"tail": "classifier assignment",
"type": "USED-FOR"
},
{
"head": "corpus-based method",
"tail": "semantic construction of noun phrase",
"type": "USED-FOR"
}
]
}With checklist
Entity F1 0.571; relation F1 0.429.
Entity F1: 0.571 · Relation F1: 0.429
Output excerpt
{
"entities": [
{
"text": "algorithm for selecting an appropriate classifier word for a noun",
"type": "Task"
},
{
"text": "classifier word",
"type": "OtherScientificTerm"
},
{
"text": "noun",
"type": "OtherScientificTerm"
},
{
"text": "Thai language",
"type": "Material"
},
{
"text": "classifier",
"type": "OtherScientificTerm"
}
],
"relations": [
{
"head": "corpus-based method",
"tail": "Noun Classifier Associations ( NCA )",
"type": "USED-FOR"
},
{
"head": "Noun Classifier Associations ( NCA )",
"tail": "classifier assignment",
"type": "USED-FOR"
},
{
"head": "Noun Classifier Associations ( NCA )",
"tail": "semantic construction of noun phrase",
"type": "USED-FOR"
},
{
"head": "large corpus",
"tail": "Noun Classifier Associations ( NCA )",
"type": "USED-FOR"
},
{
"head": "default rule",
"tail": "classifier",
"type": "USED-FOR"
}
]
}With skill
Entity F1 0.483; relation F1 0.316.
Entity F1: 0.483 · Relation F1: 0.316
Output excerpt
{
"entities": [
{
"text": "algorithm",
"type": "Method"
},
{
"text": "classifier word",
"type": "OtherScientificTerm"
},
{
"text": "noun",
"type": "OtherScientificTerm"
},
{
"text": "Thai language",
"type": "Material"
},
{
"text": "classifier",
"type": "OtherScientificTerm"
}
],
"relations": [
{
"head": "algorithm",
"tail": "classifier word",
"type": "USED-FOR"
},
{
"head": "default rule",
"tail": "classifier",
"type": "USED-FOR"
},
{
"head": "unit classifier",
"tail": "classifier",
"type": "HYPONYM-OF"
},
{
"head": "corpus-based method",
"tail": "classifier assignment",
"type": "USED-FOR"
},
{
"head": "corpus-based method",
"tail": "semantic construction of noun phrase",
"type": "USED-FOR"
}
]
}Recorded case extract-scierc-C94-1091 · Run four-track-canary-v3. Output excerpts are bounded; the full submitted answer was scored.
Comparing sentence planners for dialogueSciERCAI alone0.374With checklistInvalid formatWith skill0.469
Extract the sentence-planning methods and human-evaluation procedure, including which systems are compared.
Source dataset ↗ · Source cluster: scierc:P01-1056
AI alone
Entity F1 0.462; relation F1 0.286.
Entity F1: 0.462 · Relation F1: 0.286
Output excerpt
{
"entities": [
{
"text": "natural language generator",
"type": "Method"
},
{
"text": "trainable components",
"type": "Method"
},
{
"text": "hand-crafted template-based",
"type": "Method"
},
{
"text": "rule-based approaches",
"type": "Method"
},
{
"text": "trainable sentence planner",
"type": "Method"
}
],
"relations": [
{
"head": "trainable sentence planner",
"tail": "spoken dialogue system",
"type": "USED-FOR"
},
{
"head": "hand-crafted template-based",
"tail": "rule-based approaches",
"type": "CONJUNCTION"
},
{
"head": "trainable sentence planner",
"tail": "hand-crafted template-based generation component",
"type": "COMPARE"
},
{
"head": "trainable sentence planner",
"tail": "rule-based sentence planners",
"type": "COMPARE"
},
{
"head": "trainable sentence planner",
"tail": "baseline sentence planners",
"type": "COMPARE"
}
]
}With checklist
The submitted structure failed the required schema and received zero.
Entity F1: 0.000 · Relation F1: 0.000
Output excerpt
{
"args": {
"entities": [
{
"text": "modules of a natural language generator",
"type": "OtherScientificTerm"
},
{
"text": "trainable components",
"type": "Method"
},
{
"text": "hand-crafted template-based",
"type": "Method"
},
{
"text": "rule-based approaches",
"type": "Method"
},
{
"text": "trainable sentence planner",
"type": "Method"
}
],
"relations": [
{
"head": "trainable sentence planner",
"tail": "spoken dialogue system",
"type": "EVALUATE-FOR"
},
{
"head": "human judgments",
"tail": "trainable sentence planner",
"type": "EVALUATE-FOR"
},
{
"head": "trainable sentence planner",
"tail": "hand-crafted template-based generation component",
"type": "COMPARE"
},
{
"head": "trainable sentence planner",
"tail": "rule-based sentence planners",
"type": "COMPARE"
},
{
"head": "trainable sentence planner",
"tail": "baseline sentence planners",
"type": "COMPARE"
}
]
}
}With skill
Entity F1 0.483; relation F1 0.455.
Entity F1: 0.483 · Relation F1: 0.455
Output excerpt
{
"entities": [
{
"text": "natural language generator",
"type": "Method"
},
{
"text": "trainable components",
"type": "Method"
},
{
"text": "template-based",
"type": "Method"
},
{
"text": "rule-based approaches",
"type": "Method"
},
{
"text": "trainable sentence planner",
"type": "Method"
}
],
"relations": [
{
"head": "trainable sentence planner",
"tail": "spoken dialogue system",
"type": "USED-FOR"
},
{
"head": "human judgments",
"tail": "trainable sentence planner",
"type": "EVALUATE-FOR"
},
{
"head": "trainable sentence planner",
"tail": "rule-based systems",
"type": "COMPARE"
},
{
"head": "trainable sentence planner",
"tail": "baselines",
"type": "COMPARE"
},
{
"head": "trainable sentence planner",
"tail": "hand-crafted system",
"type": "COMPARE"
}
]
}Recorded case extract-scierc-P01-1056 · Run four-track-canary-v3. Output excerpts are bounded; the full submitted answer was scored.
Reconstructing 3D shapes from stereo imagesSciERCAI aloneInvalid formatWith checklist0.292With skillInvalid format
Identify mesh-based reconstruction methods and geometric constraints, and connect them to shape recovery.
Source dataset ↗ · Source cluster: scierc:ICCV_2001_47_abs
AI alone
The submitted structure failed the required schema and received zero.
Entity F1: 0.000 · Relation F1: 0.000
Output excerpt
{
"text": {
"entities": [
{
"text": "a priori geometric constraints",
"type": "OtherScientificTerm"
},
{
"text": "3 -- D stereo reconstruction scheme",
"type": "Method"
},
{
"text": "image information",
"type": "OtherScientificTerm"
},
{
"text": "3 -- D shape",
"type": "OtherScientificTerm"
},
{
"text": "iterative deformation",
"type": "Method"
}
],
"relations": [
{
"head": "a priori geometric constraints",
"tail": "3 -- D stereo reconstruction scheme",
"type": "USED-FOR"
},
{
"head": "3 -- D stereo reconstruction scheme",
"tail": "3 -- D shape",
"type": "USED-FOR"
},
{
"head": "iterative deformation",
"tail": "objective function",
"type": "USED-FOR"
},
{
"head": "3 -- D surface mesh",
"tail": "objective function",
"type": "USED-FOR"
},
{
"head": "anisotropic meshing",
"tail": "non-quadratic approach to regularization",
"type": "CONJUNCTION"
}
]
}
}With checklist
Entity F1 0.500; relation F1 0.083.
Entity F1: 0.500 · Relation F1: 0.083
Output excerpt
{
"entities": [
{
"text": "geometric constraints",
"type": "OtherScientificTerm"
},
{
"text": "3 -- D stereo reconstruction scheme",
"type": "Method"
},
{
"text": "image information",
"type": "OtherScientificTerm"
},
{
"text": "3 -- D shape",
"type": "OtherScientificTerm"
},
{
"text": "iterative deformation",
"type": "Method"
}
],
"relations": [
{
"head": "geometric constraints",
"tail": "3 -- D stereo reconstruction scheme",
"type": "USED-FOR"
},
{
"head": "iterative deformation",
"tail": "objective function",
"type": "USED-FOR"
},
{
"head": "anisotropic meshing",
"tail": "non-quadratic approach to regularization",
"type": "CONJUNCTION"
},
{
"head": "anisotropic meshing",
"tail": "reconstruction results",
"type": "USED-FOR"
},
{
"head": "non-quadratic approach to regularization",
"tail": "reconstruction results",
"type": "USED-FOR"
}
]
}With skill
The submitted structure failed the required schema and received zero.
Entity F1: 0.000 · Relation F1: 0.000
Output excerpt
{
"parameters": {
"entities": [
{
"text": "a priori geometric constraints",
"type": "OtherScientificTerm"
},
{
"text": "3 -- D stereo reconstruction scheme",
"type": "Method"
},
{
"text": "image information",
"type": "OtherScientificTerm"
},
{
"text": "3 -- D shape",
"type": "OtherScientificTerm"
},
{
"text": "iterative deformation",
"type": "Method"
}
],
"relations": [
{
"head": "a priori geometric constraints",
"tail": "3 -- D stereo reconstruction scheme",
"type": "USED-FOR"
},
{
"head": "anisotropic meshing",
"tail": "non-quadratic approach",
"type": "CONJUNCTION"
},
{
"head": "anisotropic meshing",
"tail": "triangulations",
"type": "USED-FOR"
},
{
"head": "non-quadratic approach",
"tail": "regularization",
"type": "USED-FOR"
},
{
"head": "Structural or numerical constraints",
"tail": "reconstruction process",
"type": "USED-FOR"
}
]
}
}Recorded case extract-scierc-ICCV_2001_47_abs · Run four-track-canary-v3. Output excerpts are bounded; the full submitted answer was scored.
Fitting data with many outliersSciERCAI alone0.474With checklist0.549With skill0.474
Extract MINPRAN, its sampling assumptions, comparison method, and evaluation data, and map their roles in robust fitting.
Source dataset ↗ · Source cluster: scierc:CVPR_1994_10_abs
AI alone
Entity F1 0.519; relation F1 0.429.
Entity F1: 0.519 · Relation F1: 0.429
Output excerpt
{
"entities": [
{
"text": "MINPRAN",
"type": "Method"
},
{
"text": "robust operator",
"type": "Method"
},
{
"text": "outliers",
"type": "OtherScientificTerm"
},
{
"text": "error bound",
"type": "OtherScientificTerm"
},
{
"text": "good data",
"type": "Material"
}
],
"relations": [
{
"head": "MINPRAN",
"type": "HYPONYM-OF",
"tail": "robust operator"
},
{
"head": "MINPRAN",
"type": "COMPARE",
"tail": "least median of squares"
},
{
"head": "random sampling",
"type": "USED-FOR",
"tail": "MINPRAN"
},
{
"head": "MINPRAN",
"type": "EVALUATE-FOR",
"tail": "synthetic data"
},
{
"head": "MINPRAN",
"type": "USED-FOR",
"tail": "range and intensity data"
}
]
}With checklist
Entity F1 0.483; relation F1 0.615.
Entity F1: 0.483 · Relation F1: 0.615
Output excerpt
{
"entities": [
{
"text": "MINPRAN",
"type": "Method"
},
{
"text": "robust operator",
"type": "Method"
},
{
"text": "data sets",
"type": "Material"
},
{
"text": "outliers",
"type": "OtherScientificTerm"
},
{
"text": "error bound",
"type": "OtherScientificTerm"
}
],
"relations": [
{
"head": "MINPRAN",
"type": "HYPONYM-OF",
"tail": "robust operator"
},
{
"head": "random sampling",
"type": "USED-FOR",
"tail": "MINPRAN"
},
{
"head": "MINPRAN",
"type": "COMPARE",
"tail": "least median of squares"
},
{
"head": "synthetic data",
"type": "EVALUATE-FOR",
"tail": "MINPRAN"
},
{
"head": "MINPRAN",
"type": "USED-FOR",
"tail": "complex range and intensity data"
}
]
}With skill
Entity F1 0.414; relation F1 0.533.
Entity F1: 0.414 · Relation F1: 0.533
Output excerpt
{
"entities": [
{
"text": "MINPRAN",
"type": "Method"
},
{
"text": "robust operator",
"type": "Method"
},
{
"text": "data sets",
"type": "Material"
},
{
"text": "outliers",
"type": "OtherScientificTerm"
},
{
"text": "error bound",
"type": "OtherScientificTerm"
}
],
"relations": [
{
"head": "MINPRAN",
"type": "HYPONYM-OF",
"tail": "robust operator"
},
{
"head": "random sampling",
"type": "USED-FOR",
"tail": "MINPRAN"
},
{
"head": "error bound",
"type": "FEATURE-OF",
"tail": "good data"
},
{
"head": "dynamic range",
"type": "PART-OF",
"tail": "sensor"
},
{
"head": "MINPRAN",
"type": "COMPARE",
"tail": "least median of squares"
}
]
}Recorded case extract-scierc-CVPR_1994_10_abs · Run new-sources-and-analysis-v3. Output excerpts are bounded; the full submitted answer was scored.
Recognizing named entities in JapaneseSciERCAI alone0.316With checklist0.410With skill0.367
Identify dictionaries, rules, and entity categories, and map how Amorph uses them to recognize names, numbers, and time expressions.
Source dataset ↗ · Source cluster: scierc:X96-1059
AI alone
Entity F1 0.311; relation F1 0.320.
Entity F1: 0.311 · Relation F1: 0.320
Output excerpt
{
"entities": [
{
"text": "Recognition of proper nouns in Japanese text",
"type": "Task"
},
{
"text": "morphological analysis in Japanese text processing",
"type": "Task"
},
{
"text": "Japanese information extraction",
"type": "Task"
},
{
"text": "Multi-lingual Evaluation Task",
"type": "Task"
},
{
"text": "MET",
"type": "Task"
}
],
"relations": [
{
"head": "morphological analyzer",
"type": "USED-FOR",
"tail": "recognition and classification of proper names , numerical and temporal expressions"
},
{
"head": "Amorph",
"type": "USED-FOR",
"tail": "NE items"
},
{
"head": "dictionary lookup",
"type": "CONJUNCTION",
"tail": "rule application"
},
{
"head": "dictionaries",
"type": "USED-FOR",
"tail": "Japanese character strings"
},
{
"head": "rules",
"type": "USED-FOR",
"tail": "segmented strings"
}
]
}With checklist
Entity F1 0.533; relation F1 0.286.
Entity F1: 0.533 · Relation F1: 0.286
Output excerpt
{
"entities": [
{
"text": "Recognition of proper nouns",
"type": "Task"
},
{
"text": "morphological analysis",
"type": "Task"
},
{
"text": "Japanese text processing",
"type": "Task"
},
{
"text": "Japanese information extraction",
"type": "Task"
},
{
"text": "Multi-lingual Evaluation Task ( MET )",
"type": "Task"
}
],
"relations": [
{
"head": "Recognition of proper nouns",
"type": "PART-OF",
"tail": "morphological analysis"
},
{
"head": "morphological analysis",
"type": "PART-OF",
"tail": "Japanese text processing"
},
{
"head": "morphological analyzer",
"type": "USED-FOR",
"tail": "recognition and classification of proper names"
},
{
"head": "morphological analyzer",
"type": "USED-FOR",
"tail": "Named Entity ( NE ) items"
},
{
"head": "Amorph",
"type": "USED-FOR",
"tail": "NE items"
}
]
}With skill
Entity F1 0.458; relation F1 0.276.
Entity F1: 0.458 · Relation F1: 0.276
Output excerpt
{
"entities": [
{
"text": "proper nouns",
"type": "OtherScientificTerm"
},
{
"text": "Japanese text",
"type": "Material"
},
{
"text": "morphological analysis",
"type": "Task"
},
{
"text": "Japanese text processing",
"type": "Task"
},
{
"text": "Japanese information extraction",
"type": "Task"
}
],
"relations": [
{
"head": "morphological analysis",
"type": "PART-OF",
"tail": "Japanese text processing"
},
{
"head": "proper nouns",
"type": "HYPONYM-OF",
"tail": "morphological analysis"
},
{
"head": "Amorph",
"type": "USED-FOR",
"tail": "NE items"
},
{
"head": "morphological analyzer",
"type": "USED-FOR",
"tail": "Named Entity (NE) items"
},
{
"head": "dictionary lookup",
"type": "CONJUNCTION",
"tail": "rule application"
}
]
}Recorded case extract-scierc-X96-1059 · Run new-sources-and-analysis-v3. Output excerpts are bounded; the full submitted answer was scored.
Parsing grammars with functional programsSciERCAI alone0.255With checklist0.344With skill0.262
Extract parser types, grammar classes, and memoization methods, and map their relationships to parsing behavior and computational cost.
Source dataset ↗ · Source cluster: scierc:E91-1012
AI alone
Entity F1 0.414; relation F1 0.095.
Entity F1: 0.414 · Relation F1: 0.095
Output excerpt
{
"entities": [
{
"text": "LR-parsers",
"type": "Method"
},
{
"text": "correctness proof",
"type": "OtherScientificTerm"
},
{
"text": "recursive descent parser",
"type": "Method"
},
{
"text": "non-LR grammars",
"type": "Material"
},
{
"text": "time-complexity",
"type": "Metric"
}
],
"relations": [
{
"head": "correctness proof",
"type": "USED-FOR",
"tail": "LR-parsers"
},
{
"head": "recursive descent parser",
"type": "HYPONYM-OF",
"tail": "LR-parsers"
},
{
"head": "memo-functions",
"type": "USED-FOR",
"tail": "LR-parsers"
},
{
"head": "time-complexity",
"type": "EVALUATE-FOR",
"tail": "non-LR grammars"
},
{
"head": "memo-functions",
"type": "USED-FOR",
"tail": "parse forest"
}
]
}With checklist
Entity F1 0.467; relation F1 0.222.
Entity F1: 0.467 · Relation F1: 0.222
Output excerpt
{
"entities": [
{
"text": "LR-parsers",
"type": "Method"
},
{
"text": "correctness proof",
"type": "OtherScientificTerm"
},
{
"text": "recursive descent parser",
"type": "Method"
},
{
"text": "non-LR grammars",
"type": "Material"
},
{
"text": "time-complexity",
"type": "Metric"
}
],
"relations": [
{
"head": "LR-parsers",
"type": "USED-FOR",
"tail": "non-LR grammars"
},
{
"head": "LR-parsers",
"type": "HYPONYM-OF",
"tail": "recursive descent parser"
},
{
"head": "time-complexity",
"type": "EVALUATE-FOR",
"tail": "non-LR grammars"
},
{
"head": "memo-functions",
"type": "USED-FOR",
"tail": "parse forest"
},
{
"head": "recursive ascent parsers",
"type": "COMPARE",
"tail": "LR ( 0 ) grammars"
}
]
}With skill
Entity F1 0.414; relation F1 0.111.
Entity F1: 0.414 · Relation F1: 0.111
Output excerpt
{
"entities": [
{
"text": "LR-parsers",
"type": "Method"
},
{
"text": "correctness proof",
"type": "OtherScientificTerm"
},
{
"text": "recursive descent parser",
"type": "Method"
},
{
"text": "non-LR grammars",
"type": "Material"
},
{
"text": "time-complexity",
"type": "Metric"
}
],
"relations": [
{
"head": "LR-parsers",
"type": "COMPARE",
"tail": "recursive descent parser"
},
{
"head": "time-complexity",
"type": "FEATURE-OF",
"tail": "non-LR grammars"
},
{
"head": "memo-functions",
"type": "USED-FOR",
"tail": "parse forest"
},
{
"head": "recursive ascent parsers",
"type": "COMPARE",
"tail": "LR-parsers"
},
{
"head": "regular expressions",
"type": "FEATURE-OF",
"tail": "Extended CF grammars"
}
]
}Recorded case extract-scierc-E91-1012 · Run new-sources-and-analysis-v3. Output excerpts are bounded; the full submitted answer was scored.
Combining systems to resolve word meaningsSciERCAI aloneInvalid formatWith checklistFailedWith skill0.417
Identify unsupervised word-sense systems, voting and arbiter strategies, and evaluation datasets, then map how the ensemble components work together.
Source dataset ↗ · Source cluster: scierc:P06-1013
AI alone
The submitted structure failed the required schema and received zero.
Entity F1: 0.000 · Relation F1: 0.000
Output excerpt
{
"parameter_name": "\n",
"entities": [
{
"text": "Combination methods",
"type": "Method"
},
{
"text": "system performance",
"type": "Metric"
},
{
"text": "system combination",
"type": "Method"
},
{
"text": "unsupervised WSD",
"type": "Task"
},
{
"text": "voting - and arbiter-based combination strategies",
"type": "Method"
}
],
"relations": [
{
"head": "system combination",
"type": "USED-FOR",
"tail": "unsupervised WSD"
},
{
"head": "voting - and arbiter-based combination strategies",
"type": "USED-FOR",
"tail": "unsupervised WSD systems"
},
{
"head": "predominant senses",
"type": "USED-FOR",
"tail": "Combination methods"
},
{
"head": "raw text",
"type": "USED-FOR",
"tail": "predominant senses"
},
{
"head": "SemCor",
"type": "CONJUNCTION",
"tail": "Senseval-3 data sets"
}
]
}With checklist
The response used multiple final tool calls instead of one answer.
With skill
Entity F1 0.667; relation F1 0.167.
Entity F1: 0.667 · Relation F1: 0.167
Output excerpt
{
"entities": [
{
"text": "Combination methods",
"type": "Method"
},
{
"text": "system performance",
"type": "Metric"
},
{
"text": "unsupervised WSD",
"type": "Task"
},
{
"text": "voting - and arbiter-based combination strategies",
"type": "Method"
},
{
"text": "unsupervised WSD systems",
"type": "Method"
}
],
"relations": [
{
"head": "Combination methods",
"type": "USED-FOR",
"tail": "system performance"
},
{
"head": "voting - and arbiter-based combination strategies",
"type": "USED-FOR",
"tail": "unsupervised WSD systems"
},
{
"head": "predominant senses",
"type": "USED-FOR",
"tail": "voting - and arbiter-based combination strategies"
},
{
"head": "predominant senses",
"type": "HYPONYM-OF",
"tail": "raw text"
},
{
"head": "SemCor",
"type": "CONJUNCTION",
"tail": "Senseval-3 data sets"
}
]
}Recorded case extract-scierc-P06-1013 · Run new-sources-and-analysis-v3. Output excerpts are bounded; the full submitted answer was scored.