An OpenEnv environment for training cybersecurity defender LLMs against an attacker LLM that auto-generates novel incidents. Built for the OpenEnv Hackathon, April 2026.
Humans cannot watch every alert in a Security Operations Center 24/7, and as stronger generative models start writing exploits and phishing at industrial scale that gap only widens. OpenSOC is an environment where a defender LLM learns to triage attacks generated by another LLM in a self-play loop. The trick is RLVR: triage ground truth is computed by a deterministic schema-side verifier from the structured incident parameters — never from any text the attacker writes — so neither side can hack the reward.
An episode has exactly two turns: attacker proposes incident params → env validates them and materializes a SIEM-style alert + log window → defender submits a triage action. The verifier computes the ground-truth action from the events alone and scores both sides — the attacker's free-text narrative is never read by the labeler.
In defender_only mode (used for SFT, eval, smoke tests, and the /demo UI) the env auto-generates the incident from tasks/registry.py and skips straight to the defender turn.
Why the reward cannot be hacked
The verifier is a transparent rule set in verifier.compute_ground_truth(params); the only inputs are the structured events. The attacker's narrative and even its self-claimed target_label are ignored.
The plausibility checker (verifier.check_plausibility(params)) refuses incoherent stories — for example, a "data exfiltration" claim with a purely-internal destination, or a lolbin_use event with no process field. The attacker's reward is gated on plausibility passing.
Schema-violation incidents floor attacker reward at -0.5, so trying to short-circuit pydantic's validators is strictly worse than playing along.
Tool names are deliberately non-reserved — there is no reset/step/state/close clash with the OpenEnv MCPEnvironment reserved-name list.
yaml
1action_space:2craft_incident:3target_label: dismiss | monitor | quarantine_host | block_ip | escalate
4category: malware_execution | c2_beacon | data_exfiltration |...5events:[{ event_type, fields, timestamp, log_id },...]6narrative: string # ignored by the verifier7submit_triage:8action: <one of the five triage actions>9cited_log_id: <id of the log line that drove the decision>10rationale: short string
Defender: +1 correct, −1 missed-malicious, −0.3 over-react on benign, −0.05 unnecessary escalate, +0.1 bonus for citing the right triggering log id, −0.1 floor for format violation.
Attacker: +1 iff defender wrong AND incident plausible, −0.5 if schema validation fails, +0.2 novelty bonus, 0 for gibberish.
SFT warm-start (~12 min) — pushes P(format-OK) from ~0% to ~95%.
GRPO curriculum across 4 stages (~3h) — verifier-grounded reward, group size 8.
Eval on the frozen 200-incident hold-out (~5 min).
eval.plot_results + eval.plot_training render four PNGs.
eval.bake_demo writes 50 before-vs-after pairs to data/demo_examples.json for the Gradio UI.
Headline results
The defender model was trained using GRPO with a 4-stage curriculum on Qwen2.5-3B-Instruct with LoRA. All trained adapters are published on HuggingFace:
The Space runs FastAPI + Gradio in a single container. /reset, /step, /state, /grade, /tasks, /health continue to work for the OpenEnv judge bot; /demo is the human-readable UI.
