OpsSim-AI: Teaching LLMs to Run a Production War Room
When a checkout system starts throwing 500 errors at 2 AM, the cause is rarely obvious. A Redis cache failure may be triggering payment timeouts, which leave stale database connections, which make it tempting to restart the wrong service too early — making everything worse.
Real incident response requires a team of specialists who each see only part of the picture, must communicate findings, coordinate actions in the right order, and resist the urge to take dangerous shortcuts.
OpsSim-AI is an environment that turns this kind of messy, multi-team incident into a structured training ground for LLM agents.
🔗 Hugging Face Space: [Coming soon — link will be added here]
The demo lets you:
Select a cascading failure scenario
Watch the agent team diagnose and resolve the incident step by step
See the 13-component reward breakdown at each step
Compare agent decisions against the optimal recovery path
View the war room's incident channel and communication flow
The Problem
Large language models are good at answering questions when they can see all the information. Real operational incidents are nothing like that.
In a production outage:
No one sees the full picture. The database team sees lock contention. The infra team sees pod restarts. The app team sees checkout errors. The actual root cause could be upstream of all of them.
Order matters. Restarting a service before fixing the underlying cause just restarts the failure loop.
Wrong actions make things worse. Flushing DNS when the problem is a database deadlock wastes time and can introduce new issues.
Someone has to coordinate. Individual specialists need a commander who synthesizes partial reports and decides what happens next.
Most LLM benchmarks test none of this. They give the agent full state, a single decision, and immediate feedback. OpsSim-AI creates the kind of pressure that real incidents create: partial information, delayed payoff, dangerous temptations, and the need to work as a team.
What This Environment Does
OpsSim-AI simulates a distributed war room with 9 agents responding to cascading production failures across 10 realistic scenarios. (Theme #1 - Multi-Agent Interactions)
Domain specialists — each sees only their slice of the system
Coordination
Incident Commander (IC)
Reads all reports, plans recovery, delegates actions
Oversight
Supervisor
Reviews directives, vetoes dangerous actions
What makes it challenging
Partial observability. AppOps can see checkout health but not database lock state. InfraOps can see pod status but not payment logs. The IC must piece together the full picture from narrow domain reports.
Cascading failures. A single root cause (e.g., a bad canary deploy) spreads across multiple domains. The system keeps degrading while agents deliberate — every step costs time.
Strict responsibility boundaries. If the IC tells AppOps to fix a database issue, the environment applies a severe penalty. Agents must delegate to the correct domain owner.
Unsafe temptations. Each scenario includes plausible-sounding but harmful actions (force-restart, flush DNS, kill transactions). Choosing them carries penalties.
Long recovery sequences. Many scenarios require 5-8 ordered steps: investigate → diagnose root cause → satisfy preconditions → remediate → verify. Skipping steps or acting out of order reduces the reward. (Theme #2 - (Super) Long-Horizon Planning & Instruction Following)
The 10 Scenarios
The environment ships with 10 hand-crafted cascading failure scenarios spanning all 7 operational domains:
Scenario
What Happens
Checkout Meltdown
500 errors cascade from payment to checkout to infrastructure
Cluster split causes replication lag and leader election chaos
Security Breach
Compromised credentials lead to data exfiltration and service lockdown
Cache Avalanche
Redis failure causes thundering herd on the database
Middleware Storm
API gateway overload triggers circuit breaker cascades
Observability Blackout
Monitoring pipeline failure blinds the team during an active incident
DNS Catastrophe
DNS misconfiguration breaks service discovery across regions
Multi-Region Failover
Primary region database failure with broken replication
DB Lock Revenue Cascade
Deadlock chain collapses payment processing
How It Works
Each episode follows an 8-phase execution loop:
┌──────────────────────────────────────────────────────────┐
│ 1. ObservabilityOps analyzes metrics, surfaces clues │
│ 2. Domain agents observe their slice, report findings │
│ 3. IC reads all reports from the shared incident channel│
│ 4. IC issues a directive: target_agent + action │
│ 5. Supervisor evaluates — approve or veto │
│ 6. Target agent executes the action │
│ 7. Environment updates system state │
│ 8. Reward is computed (13 components) │
└──────────────────────────────────────────────────────────┘
↓ repeat until SLA restored or max steps reached
The IC is the decision-maker. At each step, it sees the incident description, the shared channel messages, available actions, SLA progress, and the full action history. It must choose which agent should act and what action they should take.
If the action is correct, the system state improves. If not, the cascade gets worse.
The Reward Function
The reward at each step is computed from 13 interpretable components organized into four pillars:
Did the right domain agent execute it? (-5.0 if wrong)
Conflict
P_conf
Was it contradictory to the previous action?
Pillar III — Coordination & Communication
Component
Symbol
What it measures
Coordination
R_coord
Did the IC delegate to the correct agent?
Observability
R_obs
Did ObservabilityOps surface the root cause?
Supervisor
R_sup
Did the Supervisor correctly approve/veto?
Communication Cost
γ·Σ(m)
Penalty for excessive chatter
Pillar IV — Terminal
Component
Symbol
What it measures
Success
R_succ
+2.0 bonus when all SLA conditions are met
This reward structure means agents get feedback on how they solve the incident, not just whether they solve it. Good investigation, correct delegation, safe sequencing, and efficient communication are all rewarded independently.
Reward Examples: What Gets Rewarded vs Penalized
To make the reward concrete, here are exact values from the environment:
Positive reward scenarios (good behavior):
Situation
Component
Reward
Gold action with preconditions met
Action Quality (Q_act)
+0.15 to +0.30 (from transition rules)
Action follows correct causal order
Sequencing (R_seq)
+0.15
IC delegates to the correct domain agent
Coordination (R_coord)
+0.15
ObservabilityOps surfaces ≥3 root cause keywords
Observability (R_obs)
+0.30
Supervisor correctly vetoes a harmful action
Supervisor (R_sup)
+0.20
System health improves after the action
Health Delta (ΔH)
+0.10 per improved state variable
All SLA conditions met (episode success)
Success (R_succ)
+2.00
Negative reward scenarios (bad behavior):
Situation
Component
Penalty
Wrong domain agent executes the action
Responsibility (P_resp)
-5.00 (episode terminates)
SLA violated (unrecoverable failure)
Success (R_succ)
-2.00 (episode terminates)
Picking a penalized/unsafe action
Action Quality (Q_act)
-0.50 to -1.00
Invalid action (not in available list)
Action Quality (Q_act)
-0.50
Doing nothing when valid actions exist
Action Quality (Q_act)
-0.30 × (consecutive_count)^1.5
Acting out of order (skipping prerequisites)
Sequencing (R_seq)
-0.15
Contradictory action (conflicts with previous)
Conflict (P_conf)
-0.30
Repeating the same action consecutively
Conflict (P_conf)
-0.10
IC delegates to wrong domain agent
Coordination (R_coord)
-0.10
Supervisor rubber-stamps a harmful action
Supervisor (R_sup)
-0.20
Excessive communication (chatter)
Communication (γ·Σ)
-0.02 per message
No health improvement for 3+ steps
Stagnation
additional penalty
During GRPO training, the environment reward is combined with three additional shaping functions:
Predicted action is in the scenario's unsafe action list
-0.30
This creates a total training reward range of approximately [-2.05, +1.50], where decision quality (the env signal) dominates the upside and format/safety guards prevent degenerate outputs.
Why This Is Interesting
Most agent benchmarks test a single capability: tool use, code generation, or question answering. OpsSim-AI tests several capabilities simultaneously, in a setting where they interact:
Diagnosis under partial information — the agent must reason about what it cannot see (Theme #3 - World Modeling, Theme #3.1 Professional Tasks)
Multi-step planning — early investigation steps only pay off several actions later
Coordination across specialists — choosing the right expert matters as much as choosing the right action
Safety discipline — resisting harmful shortcuts that look plausible
Recovery from mistakes — the Supervisor can veto, and the agent must adapt
This makes OpsSim-AI a compact but meaningful testbed for studying whether LLM agents can move from "answering correctly" toward operating responsibly in complex systems.
Training Pipeline
OpsSim-AI includes a full SFT → GRPO training pipeline that takes raw scenario definitions and produces a fine-tuned LLM that makes better incident response decisions. (Theme #4 - Self-Improvement)
Each scenario in tasks/cascade.json defines a complete incident with the following structure:
scenario_id → unique identifier (e.g. "cascade_001_checkout_meltdown")
description → one-line incident summary
initial_state → dict of system variables (e.g. checkout: "failing", redis: "offline")
playbook_text → investigation/remediation runbook for the IC; in large enterprises, these playbook guidelines can represent business rules and operational policies the IC is expected to follow
available_actions → all possible actions the agent can take
optimal_solution_path → the correct ordered sequence of 5-8 gold actions
transition_rules → for each action: preconditions, state effects, reward/else_reward
penalties → map of unsafe actions to their penalty values (-0.5 to -1.0)
sla_rules → conditions that must be met for success (e.g. checkout: "operational")
action_domains → which actions belong to which domain (for responsibility checking)
severity_weights → active incident severity (drives global bleed)
conflict_pairs → pairs of mutually exclusive actions
root_cause_keywords → terms ObservabilityOps should surface
From these 10 scenarios, the dataset generator creates ~66 gold training prompts (one per step in each optimal path) and ~120-180 SFT examples (gold + contrast rejection pairs).
Each training example is a prompt-response pair. The prompt gives the model:
The incident description and current system state
A playbook with investigation/remediation guidance
Available actions and SLA progress
The full action history so far
The model must respond with a JSON object:
json
1{2"analysis":"Payment service is timing out due to database connection pool exhaustion",3"plan":"First verify DB pool state, then drain stale connections, then restart checkout",4"next_action":"check_connection_pools",5"target_agent":"DatabaseOps",6"reasoning":"Checkout errors are a symptom; the root cause is upstream in the database layer",7"confidence":0.858}
SFT examples pair each prompt with the correct gold response. Contrast examples include an unsafe candidate action in the prompt and train the model to reject it. GRPO prompts contain only the prompt — the model generates its own response, which is then scored by the live environment.
How GRPO Training Works
During GRPO, the model generates 8 different completions for each prompt. Each completion is parsed and run through the actual DevOps environment simulator (env.py):
The environment replays the scenario up to the current step
The model's predicted action is executed in the environment
The environment returns a reward based on action quality, sequencing, coordination, health impact, and safety
Additional penalties are applied for invalid JSON, missing keys, or unsafe actions
GRPO computes group-relative advantages (which of the 8 completions scored best vs worst)
The policy is updated to increase the probability of higher-reward completions
This means the model learns from direct interaction with the environment, not just from imitating gold labels.
These plots are from a completed GRPO training run on Qwen2.5-3B-Instruct using an L40S GPU. All plots are auto-generated during training and pushed to the HF Hub alongside the model adapter.
1. Reward Curve (Smoothed)
Reward Smoothed
The single most important training signal. This plot shows the environment reward smoothed over a rolling window to filter out per-step noise. A clear upward trend from the start of training confirms that the policy is learning to take better actions — choosing correct investigation steps, routing to the right domain agent, and following safe sequencing. Flat or declining reward would indicate the model is stuck or degenerating. In GRPO, reward is the only ground-truth signal — unlike supervised learning, there are no labels to memorize, so this curve directly reflects the agent's improving decision quality inside the environment.
2. Loss Curve
Loss
Expected to oscillate near zero — this is not SFT. In GRPO, the loss is computed from group-relative advantages (completions scored above/below the group mean). Because advantages are zero-centered by construction, the loss naturally hovers around zero rather than declining monotonically. Large spikes indicate the model encountered a particularly informative batch (high reward variance between completions). A loss that drifts steadily negative or explodes positive signals a training instability. The key insight: do not expect a declining loss curve in GRPO — look at the reward curve instead.
3. Quality Metrics (Smoothed)
Quality Snapshot
A multi-dimensional view of the agent's operational quality. This plot tracks several metrics simultaneously: valid_json_rate (does the model produce parseable JSON?), accuracy (does it pick the correct gold action?), agent_accuracy (does it route to the correct domain specialist?), and unsafe_rate (does it choose penalized/dangerous actions?). Healthy training shows valid_json_rate near 100% (the model learned formatting from SFT), accuracy and agent_accuracy trending upward, and unsafe_rate trending downward. This plot reveals what the model is learning, while the reward curve shows how much.
4. KL Divergence
KL Divergence
Measures how far the policy has drifted from the SFT baseline. KL divergence between the current policy and the frozen SFT reference model should rise gradually — this confirms the model is exploring new behavior rather than staying frozen at the supervised starting point. KL that stays flat near zero means the model isn't learning. KL that spikes too high means the policy has diverged too far and may be generating incoherent outputs. The beta hyperparameter (KL penalty coefficient) controls this tradeoff: our setting of β=0.005 allows meaningful exploration while preventing catastrophic drift.
5. Reward vs KL
Reward vs KL
The efficiency of exploration — are we getting reward for our KL budget? This plot overlays the reward curve and KL divergence on the same time axis. The ideal pattern is reward rising faster than KL — meaning the model is finding better actions without diverging excessively from the reference. If KL rises but reward stays flat, the model is exploring randomly. If reward rises while KL stays near zero, the model is improving within its existing behavior space (unlikely to continue). A healthy GRPO run shows both curves rising, with reward leading KL, confirming that the policy updates are efficient and targeted.
Project Structure
├── env.py # DevOpsEnv — 13-component reward, 7 domains, OpenEnv-compatible
├── models.py # Pydantic models: Action, Observation, Reward (13 fields), State
├── multi_agent.py # WarRoom orchestrator + 9 agents (7 execution + IC + Supervisor)
├── inference.py # LLM inference loop: 8-phase execution with Supervisor veto
├── train_grpo.py # GRPO training with env-grounded reward + curriculum
├── train_sft.py # Supervised fine-tuning with LoRA
├── generate_sft_dataset.py # Dataset generation from cascade scenarios
├── submit_hf_job.py # One-command cloud training on HF Jobs
├── run_trained_inference.py # Test trained models on specific scenarios
├── server/app.py # FastAPI server for Hugging Face Space
├── tasks/cascade.json # 10 cascading failure scenarios × 7 domains
├── openenv.yaml # OpenEnv manifest
├── TRAINING_README.md # Detailed training guide with plot interpretation
├── Dockerfile # Container for HF Space deployment
└── requirements.txt # Dependencies
Quick Start
Run inference
bash
1exportHF_TOKEN=your_token
2python inference.py
Run the API server
python server/app.py
Train on Hugging Face Jobs
python submit_hf_job.py all --flavor l40sx1
Conclusion
OpsSim-AI creates a training ground where LLM agents face the same pressures as real incident response teams: partial information, time pressure, dangerous shortcuts, and the need to coordinate across specialists.
The environment rewards agents not just for solving the incident, but for how they solve it — investigating before acting, delegating to the right expert, following safe sequences, and communicating efficiently.
We believe this kind of structured, multi-agent, stateful environment is essential for moving LLM agents from answering questions to operating real systems reliably. (Theme #5: Wild Card - Impress Us!)
