Appearance
SREGym-style incident resolution
Demo data. These numbers are placeholders and are not real results.
Leaderboard
| Select | Rank | Agent | Model | Date | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Agent Beta | model-pro | 57.8% | 66.7% | 71.1% | 98.2 s | 640.3 s | 1.24M | 2026-10-05 | |
| 2 | Agent Alpha | model-large | 52.2% | 63.3% | 74.4% | 121.4 s | 705.8 s | 870K | 2026-10-05 | |
| 3 | Agent Gamma | model-large | 48.1% | 57.4% | 63.0% | 132.9 s | 810.0 s | 955K | 2026-10-05 | |
| 4 | Agent Beta | model-medium | 43.3% | 52.2% | 58.9% | 151.6 s | 902.4 s | 690K | 2026-10-05 | |
| 5 | Agent Delta | model-pro | 38.9% | 48.9% | 52.2% | 176.4 s | 995.2 s | 1.51M | 2026-10-05 | |
| 6 | Agent Alpha | model-open-70b | 25.6% | 38.9% | 42.2% | 610.2 s | 1290.4 s | 450K | 2026-10-05 | |
| 7 | Agent Gamma | model-small | 22.2% | 33.3% | 37.8% | 201.7 s | 1120.5 s | 288K | 2026-10-05 |
Charts
Metrics
- E2E (%) (higher is better)
- End-to-end success. Both the diagnosis and the mitigation are correct on the same run.
- Diag. (%) (higher is better)
- Diagnosis success rate. The agent names the root cause, judged by an LLM with a 0.7 threshold.
- Mit. (%) (higher is better)
- Mitigation success rate. The mitigation oracle confirms the incident is resolved.
- TTD (lower is better)
- Time to diagnose, mean seconds per problem.
- TTM (lower is better)
- Time to mitigate, mean seconds per problem.
- Tokens (lower is better)
- Mean tokens used per problem run.
Task set
90 problems, each a fault injected into a microservice application on a fresh Kubernetes cluster. Problems are grouped into six fault families: application, config, network, storage, resources and control plane. This demo copies the shape of SREGym; the entries and numbers are invented.
Harness
Each problem gets its own throwaway cluster. The agent gets kubectl, logs and metrics, a 30-minute limit, and one run per problem.
Scoring
- Diagnosis passes when an LLM judge scores the agent's root cause at 0.7 or higher.
- Mitigation passes when the problem's oracle confirms the application is healthy again.
- E2E counts a problem only when both pass on the same run. Timeouts count as failures.
- TTD / TTM are means over problems where the step completed.
Reproduce
bash
# placeholder: the command that runs the cohort and writes results/<agent>__<model>.json