Methodology

What every benchmark publishes

• The task set and version. Results from different versions are never mixed on one leaderboard.
• Metric definitions. Each metric states its unit and whether higher or lower is better.
• One data file per entry. Every row on a leaderboard is a JSON file in the benchmark's results/ folder, with the date, the agent, the model, the run count and the run configuration.

How a result reaches this site

1. A developer runs the benchmark and writes one result file per agent and model.
2. They open a pull request. CI validates every file against the published schemas and builds the site.
3. A reviewer checks the run configuration and the numbers against the raw run output.
4. On merge, the site rebuilds and publishes.

Schemas

• Benchmark frontmatter
• Result file