# SREGym-style incident resolution

> Demo benchmark in the shape of SREGym. Agents diagnose and mitigate injected Kubernetes faults; all numbers are placeholders.

Status: demo · Version: demo-90 · Published: 2026-11-10 · Ranked by: E2E (%)

These are placeholder numbers, not real results.

Page: https://benchmarks.komodor.com/benchmarks/sregym-demo/ · Data: https://benchmarks.komodor.com/benchmarks/sregym-demo.json

## Leaderboard

| Rank | Agent | Model | E2E (%) | Diag. (%) | Mit. (%) | TTD | TTM | Tokens |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | Agent Beta | model-pro | 57.8% | 66.7% | 71.1% | 98.2 s | 640.3 s | 1.24M |
| 2 | Agent Alpha | model-large | 52.2% | 63.3% | 74.4% | 121.4 s | 705.8 s | 870K |
| 3 | Agent Gamma | model-large | 48.1% | 57.4% | 63.0% | 132.9 s | 810.0 s | 955K |
| 4 | Agent Beta | model-medium | 43.3% | 52.2% | 58.9% | 151.6 s | 902.4 s | 690K |
| 5 | Agent Delta | model-pro | 38.9% | 48.9% | 52.2% | 176.4 s | 995.2 s | 1.51M |
| 6 | Agent Alpha | model-open-70b | 25.6% | 38.9% | 42.2% | 610.2 s | 1290.4 s | 450K |
| 7 | Agent Gamma | model-small | 22.2% | 33.3% | 37.8% | 201.7 s | 1120.5 s | 288K |

## Metrics

- **E2E (%)** (higher is better): End-to-end success. Both the diagnosis and the mitigation are correct on the same run.
- **Diag. (%)** (higher is better): Diagnosis success rate. The agent names the root cause, judged by an LLM with a 0.7 threshold.
- **Mit. (%)** (higher is better): Mitigation success rate. The mitigation oracle confirms the incident is resolved.
- **TTD** (lower is better): Time to diagnose, mean seconds per problem.
- **TTM** (lower is better): Time to mitigate, mean seconds per problem.
- **Tokens** (lower is better): Mean tokens used per problem run.

## Task set

90 problems, each a fault injected into a microservice application on a fresh Kubernetes cluster. Problems are grouped into six fault families: application, config, network, storage, resources and control plane. This demo copies the shape of [SREGym](https://github.com/SREGym/SREGym); the entries and numbers are invented.

## Harness

Each problem gets its own throwaway cluster. The agent gets `kubectl`, logs and metrics, a 30-minute limit, and one run per problem.

## Scoring

- **Diagnosis** passes when an LLM judge scores the agent's root cause at 0.7 or higher.
- **Mitigation** passes when the problem's oracle confirms the application is healthy again.
- **E2E** counts a problem only when both pass on the same run. Timeouts count as failures.
- **TTD / TTM** are means over problems where the step completed.

## Reproduce

```bash
# placeholder: the command that runs the cohort and writes results/<agent>__<model>.json
```