1# 1. Start an easy incident2curl -X POST https://arijit-07-devops-incident-response.hf.space/reset \3 -H "Content-Type: application/json"\4 -d '{"task_id": "easy", "seed": 42}'56# 2. Read logs on the failing service (reward: +0.15)7curl -X POST https://arijit-07-devops-incident-response.hf.space/step \8 -H "Content-Type: application/json"\9 -d '{"action_type": "read_logs", "service": "payment-service"}'1011# 3. Diagnose the root cause (reward: +0.30)12curl -X POST https://arijit-07-devops-incident-response.hf.space/step \13 -H "Content-Type: application/json"\14 -d '{"action_type": "diagnose", "root_cause": "memory leak in payment-service"}'1516# 4. Fix it (reward: +0.40)17curl -X POST https://arijit-07-devops-incident-response.hf.space/step \18 -H "Content-Type: application/json"\19 -d '{"action_type": "restart_service", "service": "payment-service"}'2021# 5. See the final score22curl https://arijit-07-devops-incident-response.hf.space/state
2324# 6. Validate all 7 tasks pass25curl https://arijit-07-devops-incident-response.hf.space/validate
python
1# Or install and use the Python client2pip install git+https://github.com/Twilight-13/devops-incident-response.git
34from devops_incident_response import DevOpsIncidentEnv, Action, ActionType
56env = DevOpsIncidentEnv(task_id="easy", seed=42)7obs = env.reset()8result = env.step(Action(action_type=ActionType.READ_LOGS, service="payment-service"))9print(f"Reward: {result.reward}")# 0.15
🎯 The Problem This Solves
Every software company running microservices faces the same brutal reality: production incidents are expensive, unpredictable, and happen at 3am.
A single SEV-1 incident — a payment service crashing, a data corruption silently corrupting prices, a DDoS botnet overwhelming your login endpoint — can cost millions and require hours of expert engineer time to diagnose and fix. On-call rotations are stressful. Tier-2 incidents that follow recognizable patterns are handled by engineers when they could, in principle, be handled by an AI agent.
Yet no RL benchmark exists for this domain.
SWE-bench tests code generation. WebArena tests web navigation. AgentBench tests general tool use. None of them model operational intelligence — the ability to reason under uncertainty about live production systems, gather information strategically, and take precise actions where wrong choices cause additional damage.
ARIA fills that gap.
🏗️ Environment Architecture
ARIA simulates a production microservices e-commerce platform. Agents interact with the environment through a standard OpenEnv API: reset(), step(), state().
What the Agent Observes
Each step returns a structured Observation object:
The agent only sees 2 log lines per service upfront. Full history requires calling read_logs explicitly. This models real observability tools (Datadog, Kibana) where engineers run queries — agents must develop a search strategy, not just read everything.
The Services
Service
Stack
Role
api-gateway
Go
Routes external requests
payment-service
Java (Spring)
Processes payments
order-service
Python
Creates and tracks orders
inventory-service
Java
Manages product stock
user-service
Node.js
Auth and profiles
notification-service
Python
Email and push alerts
data-pipeline-service
Python
Writes catalog data
product-catalog-service
Go
Stores and serves product data
price-validation-service
Python
Validates prices
analytics-service
Python
Aggregates business metrics
ml-inference-service
Python
Serves recommendation models
log-aggregator
Go
Collects and stores logs
Service Dependency Map
Every observation includes the call topology — agents can trace cascades:
Max steps: 15 | Expected strong LLM: 0.85–1.00 | Random agent: 0.05
One service crash-loops with an OutOfMemoryError. The affected service rotates by seed across payment-service, order-service, and user-service — with different log formats (Java heap errors, Python memory errors, Node.js heap dumps). A secondary circuit-breaker alert fires on api-gateway as a visible symptom.
What makes it interesting: The agent must identify the ROOT cause service (the one running out of memory) not the SYMPTOM services (everything downstream that's erroring because the root is down).
Max steps: 20 | Expected strong LLM: 0.55–0.75 | Random agent: 0.03
A bad deployment of inventory-service causes connection pool exhaustion, cascading timeouts to order-service and elevated error rates on api-gateway. Red herring: a notification-service HIGH CPU alert fires (scheduled batch job — completely unrelated).
What makes it interesting: The agent must follow the dependency chain backwards. Three services are visibly failing, but only one is the root cause. Touching the wrong service gives -0.15 collateral damage penalty.
analytics-service anomaly: avg order value $847 vs $89 historical baseline
Three noise alerts distract: TLS renewal, analytics backlog, replica lag.
What makes it interesting: This requires qualitatively different reasoning — ignoring green health checks, correlating subtle business metric anomalies, and understanding that a data pipeline deployment 2 minutes ago is the causal explanation.
Max steps: 25 | Expected strong LLM: 0.35–0.55 | Random agent: 0.01
Two completely independent failures at once:
log-aggregator disk 100% full — dropping 48k log messages/min
ml-inference-service stuck in model checksum reload loop — CPU 99%+
What makes it interesting: Neither failure is related to the other. Solving one doesn't help the other. The agent must decompose and fix independently. This tests whether agents can maintain multiple hypotheses simultaneously.
Full credit requires BOTH:alert_oncall (disk cleanup) AND rollback/restart(ml-inference-service) Optimal score: ~0.77
Task 5 — Security Incident: DDoS (security)
Max steps: 20 | Expected strong LLM: 0.40–0.60 | Random agent: 0.01
A botnet is targeting the login endpoint with 12,000 req/s from the 185.220.x.x IP range. Standard rate limiting is ineffective (distributed attack). The access logs show 1,847+ failed login attempts per 60 seconds from that range.
New action: block_ip_range — models real network-level DDoS mitigation. Wrong actions: Restarting api-gateway won't help. Scaling up won't help. Must block at network level + escalate to security team.
Full credit:block_ip_range("185.220.0.0/16") AND alert_oncall Optimal score: ~0.80
Task 6 — Database Degradation (database)
Max steps: 20 | Expected strong LLM: 0.45–0.65 | Random agent: 0.01
A schema migration added a user_segment column to the orders table 15 minutes ago — without an index. Every query is now doing a full sequential table scan. DB CPU is spiking. The slow query log shows seq_scan on orders (847ms).
New action: create_index — models real DBA response to missing indexes. Alternative fix: Rolling back the migration is also accepted for full credit.
Optimal score: ~0.80
Task 7 — Multi-Region Failover (failover)
Max steps: 25 | Expected strong LLM: 0.35–0.55 | Random agent: 0.01
A network partition affects us-east-1. Four services support automatic failover to us-west-2 and should be switched. Two services MUST NOT be failed over:
payment-service — PCI-DSS compliance requires human approval
postgres-primary — replication lag risk causes data loss
New action: failover — with target_region parameter. Heavy penalty: -0.25 per wrong service. Failing over payment or postgres is catastrophic.
The runbook explicitly lists which services are safe — reading it first is rewarded. Optimal score: ~0.70
Task 8 — Generated Incident (generated)
Max steps: 20 | Variable difficulty | Seed-deterministic
The Incident Generator creates procedural incidents from any integer seed (0–99,999). Same seed always produces the same incident. Different seeds produce unique combinations of:
1# Preview any incident before running it2curl"https://arijit-07-devops-incident-response.hf.space/generate/preview?seed=12345"34# Run it as a full episode5curl -X POST .../reset -d '{"task_id":"generated","seed":12345}'
🏆 Reward Function Design
The Formula
Final Score = Σ(step_rewards)
+ efficiency_bonus # (1 - steps/max_steps) × 0.05 if resolved
+ diagnosis_precision_bonus # +0.03 if ≥50% keyword overlap, +0.01 if ≥30%
- noop_penalty # (noop_count - 3) × 0.02
- repeat_restart_penalty # (restarts - 1) × 0.05 per service
All scores clamped to (0.001, 0.999) — never exactly 0 or 1.
Why (0.001, 0.999) not (0, 1)? GRPO advantage normalization requires non-constant rewards within a group. Hard 0 or 1 creates zero-variance groups where the model doesn't update. The tiny clamp ensures a gradient signal always exists.
Step-Level Rewards
Action
Reward
Condition
read_logs (failing service)
+0.10–0.15
First time only
read_metrics (failing service)
+0.10
First time only
read_runbook (relevant)
+0.05
Correct runbook for scenario
search_logs (relevant query)
+0.05
Query returns useful results
diagnose (full match)
+0.30–0.35
≥50% keyword overlap
diagnose (partial match)
+0.10–0.15
≥30% keyword overlap
restart_service (correct)
+0.35–0.45
Root cause service
rollback (correct)
+0.30–0.40
Root cause service
block_ip_range (correct)
+0.40
Security task, correct CIDR
create_index (correct)
+0.40
Database task, correct table/column
failover (eligible service)
+0.30
Per correctly failed-over service
alert_oncall (required)
+0.15
Hard/security/database/failover tasks
Penalties (Anti-Gaming)
Action
Penalty
Why
Restart healthy service
-0.15
Collateral damage — realistic cost
Fix without diagnosing
-0.10
Blind remediation — models real risk
Failover payment-service
-0.25
PCI-DSS compliance violation
Failover postgres-primary
-0.25
Data loss risk
Excessive noops (>3)
-0.04/each
Forces active investigation
Repeat restart same service
-0.05/extra
Discourages guess-and-check
Semantic Diagnosis Matching
The diagnose action uses keyword overlap not exact string matching. An agent saying "memory exhaustion in payment-service" correctly matches the ground truth "memory_leak_payment_service". This is critical for LLM agents that paraphrase — exact string matching would unfairly penalize valid diagnoses.
SLA Degradation
Every step where an incident is unresolved, the environment worsens:
Demotion: rolling_avg < 0.30 → step back mastery level
Scaffolding: if avg < 0.30 over 3+ episodes → provide task-specific hint
bash
1GET /curriculum/status # See mastery per task2GET /curriculum/next # Get recommended next task3GET /curriculum/hint/easy # Get scaffolding hint for a task4POST /curriculum/record # Feed your training results in
Why this matters for training: RL fails when agents never see successful trajectories. The curriculum ensures agents always train at the edge of their capability — easy tasks first, harder tasks as they master the fundamentals.
Incident Generator
Procedural incident generation from seeds. 6 failure modes × 8 services × 3 severities × 0–3 noise alerts = thousands of unique training scenarios.
Difficulty formula:base_difficulty[failure_mode] + (noise_count × 0.05), clamped to 1.0
Failure Mode
Base Difficulty
oom
0.20
cascade
0.50
database
0.60
security
0.60
network_partition
0.70
corruption
0.80
bash
1GET /generate/preview?seed=42# Preview without starting2POST /reset # body: {"task_id":"generated","seed":42}
Dual-Agent Mode
One incident. Two agents. Split observability.
Agent A (Observer): Sees logs, alerts, evidence. Can ONLY call share_finding — passes natural language observations to Agent B. Reward: +0.05 per finding.
Agent B (Responder): Sees metrics, service dependencies, SLA status. Cannot see logs directly. Must rely on Agent A's findings. Executes all real actions.
Neither agent can solve the incident alone.
bash
1# Start a dual-agent session2POST /multi-agent/reset {"task_id":"easy","seed":42}3# → returns session_id + split observations45# Agent A shares a finding6POST /multi-agent/step/a/{session_id}{"finding":"payment-service OOM, memory at 98%"}78# Agent B takes action (has access to Agent A's findings)9POST /multi-agent/step/b/{session_id}{"action_type":"restart_service","service":"payment-service"}1011# See full session state12GET /multi-agent/state/{session_id}
🧠 Training
Model
Llama-3.2-3B-Instruct fine-tuned with GRPO (Group Relative Policy Optimization) using HuggingFace TRL and Unsloth.
LoRA: rank=16, alpha=32, targeting all 7 projection layers
Adapter size: ~97MB
Training: 140 episodes (easy + medium tasks) on Kaggle T4 x2 GPUs
GRPO eliminates the value network that PPO requires. For environment-based RL where rewards come from an external API, a value model adds complexity without benefit. GRPO estimates the baseline from a group of 6 completions per step — simpler, more memory-efficient, and well-suited to fast environment APIs.
Training Loop
python
1# Each training step:2# 1. Generate 6 completions for the current observation3# 2. Score each on a FRESH env snapshot (prevents reward gate exhaustion)4# 3. Normalize rewards to advantages (GRPO)5# 4. Policy gradient update on best completion + KL penalty6# 5. Advance episode with best action78# Key hyperparameters:9learning_rate =5e-610group_size =611kl_coefficient =0.05# prevents catastrophic forgetting12update_strategy ="episode-level"# one update per full episode
Results
Base Model
Fine-tuned (ep140)
Easy task
0.000
0.150
Behavior
Jumps to diagnose immediately
Reads logs on correct service first
Why the difference
Base model triggers blind remediation penalty
Fine-tuned model learned to gather information before acting
The trained model consistently reads logs on the failing service before acting — this is the foundational operational behavior: information gathering before remediation. The base model never does this.
Training challenge identified: The original training loop called env_step during group generation, burning reward gates before the best action could advance the episode. After fixing to score completions on fresh environment snapshots, the model successfully learned step 1 of the optimal policy. With more episodes using the corrected loop, the full sequence would emerge.
Training Notebook
See train_grpo.ipynb — Colab-compatible, runs against the live HF Space API (no local setup needed).
