The setup
Vector RAG
530 documents (one per race, one per season's standings) embedded with MiniLM. The 8 closest to the question go to the model in one call.
Graph agent
A knowledge graph of drivers, teams, races, circuits and seasons. The model plans a chain of
lookups through 9 typed tools, such as teammates(driver, season).
Hybrid
The graph agent plus a vector_search tool for anything the graph tools don't cover.
All three use the same model (). Every gold answer is computed by code from the same data the systems read, so scoring is exact, with no human labels and no LLM judge.
Accuracy by question type
Bars show the share answered correctly; whiskers are 95% confidence intervals.
Explore the questions
The graph
734 nodes and 12,246 edges. Result edges carry position, points, grid slot and status; each team stint is a DROVE_FOR edge keyed by season.
Method and limits
- Data: Formula 1 results from the open Jolpica F1 API (successor to Ergast), seasons 2000–2025, used for non-commercial research.
- Questions: 10 templates, 12 questions each, sampled with a fixed seed. 1-hop questions need one fact; 2- and 3-hop questions chain facts; aggregation questions count across many records.
- The catch, and the held-out set: the graph tools were designed alongside those 10 templates, so some questions map onto one tool call. To test the tool design itself, 5 more question types (8 each) were written after the tools: distinct winners in a season, pole-to-win conversions, most wins at a circuit, podiums without a win, and the runner-up's team. No single tool answers them; the agent averaged 6 tool calls per question, up from 2.3.
- Scoring: names in answers are resolved to entities (so "Hamilton" and "Lewis Hamilton" match; ambiguous names like "Schumacher" don't count), numbers must match exactly.
- Limits: one model, one run per question, 12 questions per template, so differences of a few questions within a type are noise; the confidence intervals show how much. A perfect score on 40 held-out questions is still an upper bound for this one dataset: real questions are messier. The vector baseline uses one reasonable document design; other chunkings would score differently.