Code Security Review — OpenEnv Environment
An RL environment for training AI agents to perform real-world code security review.
Agents analyze code from production pull requests across a two-phase multi-step
workflow: first discovering the hidden file, then identifying the vulnerability.
Built by Inmodel Labs for the Meta PyTorch OpenEnv Hackathon.
Environment Overview
| Field | Value |
|---|
| Tasks | 3 (easy → medium → hard) |
| Languages | Python, JavaScript |
| Action space | Phase 1: {"request_file": true} / Phase 2: Structured JSON (6 fields) |
| Reward range | 0.0 – 1.0 (clamped) |
| Steps per episode | 2 (max) |
Tasks
| ID | Language | Bug Class | Difficulty |
|---|
python-off-by-one | Python | Off-by-one index error | Easy |
js-idor-auth | JavaScript | Insecure Direct Object Reference (IDOR) | Medium |
python-pickle-deserialization | Python | Insecure Deserialization (RCE) | Hard |
Two-Phase Episode Walkthrough
The agent operates in a 2-step sequential workflow that mirrors a real AppSec triage process:
Step 1 — File Discovery (+0.20)
The agent receives only the PR title and file path. The code is hidden. The agent must request access:
The environment unlocks the code snippet and returns it in the observation.
Step 2 — Security Review (up to +0.80)
The agent analyses the code and submits a structured JSON finding:
1{
2 "bug_identified": true,
3 "bug_location": "line 3 — range(len(transactions) + 1)",
4 "bug_type": "off-by-one",
5 "bug_description": "Off-by-one error causes IndexError on last iteration...",
6 "severity": "medium",
7 "suggested_fix": "Change range(len(transactions) + 1) to range(len(transactions))"
8}
Action Space
Phase 1 — File Request
Phase 2 — Bug Review
| Field | Type | Values |
|---|
bug_identified | bool | true / false |
bug_location | string | location description |
bug_type | string | off-by-one | logic-error | insecure-deserialization | none |
bug_description | string | detailed vulnerability explanation |
severity | string | none | low | medium | high | critical |
suggested_fix | string | how to fix the bug |
Observation Space
1{
2 "task_id": "python-pickle-deserialization",
3 "language": "Python",
4 "difficulty": "hard",
5 "code_snippet": "<FILE CONTENTS HIDDEN - Submit {\"request_file\": true} to view>",
6 "context": "Redis-backed caching decorator for worker tasks that serializes results...",
7 "pr_title": "Add distributed task caching layer for worker pool",
8 "file_path": "worker/cache.py"
9}
After request_file, code_snippet contains the actual source code.
Reward Breakdown
| Step | Component | Max Score |
|---|
| 1 | File request granted | 0.20 |
| 2 | Bug identified | 0.20 |
| 2 | Bug type correct | 0.20 |
| 2 | Bug location correct | 0.10 |
| 2 | Description quality | 0.25 |
| 2 | Fix quality | 0.15 |
| 2 | Severity correct | 0.10 |
| Total | | 1.00 |
The grader penalises keyword stuffing — incoherent keyword dumps score ≤ 0.20 on the description component.
Episode total reward is clamped to [0.0, 1.0].
Example Calculation:
Agent requests file (+0.20), correctly identifies bug (+0.20), correct type (+0.20),
finds 50% location keywords (+0.05), writes good description (+0.20),
suggests partial fix (+0.08), correct severity (+0.10) = total 0.20+0.20+0.20+0.05+0.20+0.08+0.10 = 1.00 → clamped to 1.00.
Edge Cases
- At step 0:
reset() must be called first. Calling step() without a reset triggers auto-reset.
- Phase 1 skip: If the agent skips
request_file and submits a review directly on step 1, it receives no intermediate reward and the code snippet used for grading may be hidden.
- Max step limit: Episode ends at
done=True when a bug review is submitted or max_steps=2 is reached.
- At done=True: Calling
step() returns reward=0.0, done=True, and info["error"] indicating the episode is complete.
Baseline Scores
| Task | Difficulty | Model | Score | Steps | Notes |
|---|
| python-off-by-one | easy | Llama-3.3-70B-Instruct | 0.883 | 2 | File request + review |
| js-idor-auth | medium | Llama-3.3-70B-Instruct | 0.500 | 2 | File request + review |
| python-pickle-deserialization | hard | Llama-3.3-70B-Instruct | 0.512 | 2 | File request + review |
API Endpoints
| Method | Path | Description |
|---|
| GET | / | Health check |
| POST | /reset?task_id=<id> | Reset environment, returns observation |
| POST | /step | Submit action (Phase 1 or Phase 2), returns reward |
| GET | /state | Current episode state |
| GET | /tasks | List all tasks |
Setup
Docker
1docker build -t code-security-review .
2docker run -p 8000:8000 code-security-review
Local
1pip install -r requirements.txt
2uvicorn server.app:app --host 0.0.0.0 --port 8000
Running Inference
1export API_BASE_URL="https://router.huggingface.co/v1"
2export MODEL_NAME="meta-llama/Llama-3.3-70B-Instruct"
3export HF_TOKEN="hf_your_token_here"
4export ENV_URL="http://localhost:8000"
5
6python inference.py
