A reinforcement-learning benchmark that drops an AI agent into a git-backed codebase seeded with realistic secret leaks — hardcoded API keys, base64-encoded tokens, credentials buried in git history — and grades how quickly and safely the agent remediates every one. The environment runs as a stateless FastAPI server; the agent interacts over HTTP with bash commands and structured inspection actions, receiving a composite reward after every step that captures security progress, code health, and time efficiency.
Why This Exists
Production secret leaks remain a top-5 cause of cloud breaches. Developers commit API keys, push .env files, or leave credentials in migration scripts — then scramble to rotate and rewrite history. Existing linting tools flag secrets but don't fix them; LLM agents can, but there's no standardized benchmark to measure how well. SecretsAuditEnv fills that gap with a 13-task curriculum spanning trivial single-file fixes to multi-service cascading leaks with git-history rewriting, complete with a deterministic grading pipeline and an anti-gaming reward function.
Task Curriculum
All 13 tasks are defined in tasks/ with full metadata in each task.json. Secret types are detected by graders/security.py using regex patterns for AWS keys, GitHub tokens, Firebase keys, connection strings, private keys, SQL passwords, assignment secrets, and base64-encoded variants.
ID
Difficulty
Title
Description
Scan Mode
Visibility Tiers
Conflict Map
1
Easy
Cloud Provisioning
Hardcoded AWS Access Key in config.py
dir
surface, surface, shallow
—
2
Easy
Database Layer
Password embedded in a raw SQL connection string
dir
surface, surface, shallow
—
3
Easy
Frontend Config
Firebase API key exposed in a client-side config file
dir
surface, surface, shallow
—
4
Easy
System Logging
Debug logging leaks a user token
dir
surface, surface, shallow
—
5
Easy
Git Basics
A tracked .env file leaks credentials in the working tree
dir
surface, surface, shallow
—
6
Medium
Utility Module
A base64-encoded auth token hides in utils.py
dir
surface, shallow, deep
—
7
Medium
CI/CD Pipeline
A deployment workflow prints a secret directly to logs
dir
surface, shallow, deep
—
8
Medium
Noise Filtering
High-entropy dummy values are mixed with one real secret in TOML
dir
surface, shallow, deep
—
9
Medium
DB Migration
A legacy migration embeds administrator credentials
dir
surface, shallow, deep
—
10
Medium
Deployment
A multiline RSA private key is embedded in a shell script
dir
surface, shallow, deep
—
11
Hard
Microservices
The same API key is duplicated across five services
dir
surface, deep, cascading
✓
12
Hard
Deep Logic
A secret is embedded as a local variable inside a function
dir
surface, deep, cascading
✓
13
Hard
Legacy Audit
A secret was committed in v1.0 and still exists in Git history
git
surface, deep, cascading
✓
Scan modes: dir scans the working directory only. git scans the full git commit history (all revisions), so agents must use git filter-repo to clean history — deleting files won't work.
Secret Visibility Tiers
Secrets are not all visible at episode start. The environment implements a 4-tier progressive disclosure system defined per-secret in task.json:
Tier
When Visible
Typical Use
SURFACE
Immediately on /reset
Obvious hardcoded keys in source files
SHALLOW
After agent calls inspect_file <path>
Secrets that require reading the file to notice
DEEP
After inspect_git_history or inspect_encoded
Secrets in git commits or base64-encoded blobs
CASCADING
After a specified trigger secret is fixed
Secrets that only become relevant after another is remediated
Hard tasks (11–13) include conflict maps that encode dependencies: fixing secret s1 may reveals3, while s2 may blocks3 until resolved.
Reward Formula
Defined in graders/reward.py. Computed fresh after every /step:
detection_score = (initial_leaks - current_leaks) / initial_leaks
base = 0.4 × detection_score + 0.6 × detection_score # (remediation = detection for now)
health_score = pytest_passed / pytest_total # 0.0 if errors > 0 or total == 0
efficiency = 0.15 × max(0, 1 - steps_taken / step_budget) # decays linearly per step
if base > 0:
total_reward = min(1.0, base × health_score + efficiency)
else:
total_reward = 0.0 # no free points for doing nothing
Key properties:
Reward is 0.0 until the agent actually fixes something — read-only commands like cat or ls cannot earn reward
Health gate: breaking the test suite (e.g., deleting imports) multiplies reward toward zero
Efficiency bonus (0.0–0.15): rewards agents that solve in fewer steps. Decays to 0 at step budget
Step budgets: Easy = 10, Medium = 20, Hard = 30
Action Space
Actions are sent as the action field in POST /step. Two categories:
Structured Actions (intercepted before bash)
Action
Format
Effect
inspect_file
inspect_file <path>
Marks file as inspected → unlocks SHALLOW secrets for that path
inspect_git_history
inspect_git_history [path]
Scans git history → unlocks all DEEP secrets requiring git inspection
inspect_encoded
inspect_encoded <path> [line]
Decodes base64 blobs → unlocks DEEP encoded secrets for that path
Bash Commands (executed in workspace via /usr/bin/bash -lc)
Any string that doesn't match a structured prefix is executed as a bash command in the task workspace. Common patterns:
cat config.py — read file contents
sed -i 's/AKIA.../os.getenv("AWS_KEY")/' config.py — redact a secret
git filter-repo --replace-text <(echo 'ghp_xxx==>REDACTED') --force — clean git history
gitleaks detect --no-git --source . — run leak scanner
Commands time out after 90 seconds. Exit code, stdout, and stderr are returned in the observation.
Observation Keys
Every /step and /reset response returns a session object with these fields (also listed in openenv.yaml):
Key
Type
Description
visible_secrets
list[dict]
Secrets the agent can currently see (filtered by visibility tier)
hidden_count_hint
int
Number of secrets not yet visible — tells agent more exist
ranked_actions
list[dict]
Top-5 heuristic action suggestions sorted by priority (0.0–1.0)
top_blocker
string
One-sentence description of the highest-priority next action
step_budget
int
Total step budget for this difficulty tier
steps_taken
int
Steps consumed so far
steps_remaining
int
step_budget - steps_taken
efficiency_bonus
float
Current efficiency bonus value (decays each step)
conflict_map
dict
Dependency graph between visible secrets (reveals/blocks relationships)
security_score
float
Fraction of initial leaks fixed (0.0–1.0)
health_score
float
Fraction of pytest tests passing (0.0–1.0)
reward
float
Composite reward after this step
observation
string
Combined stdout/stderr from last command + health/security messages
last_result
dict
Raw action, exit_code, stdout, stderr, timed_out from last command
Ranked Actions
Generated by server/observation.py using 5 heuristics:
Visible unfixed secrets → priority 0.92 (fix these first)
Uninspected high-risk files → priority based on filename suspicion score
Git history not yet scanned → priority 0.75 (medium/hard only)
All visible fixed → priority 0.85 (run gitleaks validation)
Quick Start
Local Development
bash
1# Install dependencies2pip install -r requirements.txt
34# Generate task workspaces5python tools/generate_tasks.py
67# Start the server8uvicorn server.app:app --host 0.0.0.0 --port 7860910# Open the debug UI11open http://localhost:7860/web
1213# Run the agent (requires LLM API)14exportAPI_BASE_URL="https://openrouter.ai/api/v1"15exportHF_TOKEN="your-api-key"16exportMODEL_NAME="nvidia/nemotron-3-super-120b-a12b:free"17exportENV_URL="http://localhost:7860"18python inference.py --task-id 1
Docker
bash
1docker build -t secretsauditenv .2docker run -p 7860:7860 secretsauditenv
3# Server auto-generates tasks and starts on port 7860
The Dockerfile installs Python 3.11-slim with bash, git, git-filter-repo, and all pip dependencies. The entrypoint runs tools/generate_tasks.py then starts uvicorn.
inference.py — Agent Loop
The baseline agent (inference.py) implements a single-turn ReAct loop over any OpenAI-compatible API:
Reset — POST /reset with the target task ID
Prompt — Builds a structured prompt from the current session state (reward, leaks, health, last command output, recent actions)
Call LLM — Sends prompt to the model, extracts a bash command from the response
Normalize — Strips markdown fences, extracts from XML tool_call formats, filters prose prefixes, enforces atomic (no && or ; chaining)
Step — POST /step with the action, reads updated state
Repeat until reward >= 1.0 or MAX_STEPS (40) reached
Environment variables:
Variable
Required
Description
API_BASE_URL
Yes
OpenAI-compatible API endpoint (e.g., https://openrouter.ai/api/v1)
HF_TOKEN
Yes
API key for the model provider
MODEL_NAME
Yes
Model identifier (e.g., nvidia/nemotron-3-super-120b-a12b:free)
ENV_URL
No
Server URL, defaults to http://localhost:7860
TASK_ID
No
Default task, overridden by --task-id CLI arg
Anti-loop features:
Detects 3 consecutive identical actions with identical rewards → injects a CRITICAL WARNING into the prompt and forces a different command
Filters natural language preambles (e.g., "Let me check...", "Looking at...") before sending to bash
Grading Pipeline
Security Grader (graders/security.py)
Custom regex-based scanner (no external tools required). Detects:
Supports .gitleaks.toml allowlists. For scan_mode: git, scans all commits via git rev-list --all.
Anti-gaming: if an agent deletes .git, the grader recovers by creating a fresh snapshot and scanning that — the secret still gets found.
Health Grader (graders/health.py)
Runs pytest -q --junitxml in the task workspace and parses the JUnit XML report. Score = passed / total. Returns 0.0 if any errors or if pytest times out (60s default).
Web Debug UI
Available at GET /web. A single-page dark-mode dashboard served from server/web_ui.html that lets you:
1# Run the environment test suite (34 tests)2python -m pytest tests/ -v
34# Run the submission validator (checks connection, Docker build, spec compliance, reward integrity)5bash validate-submission.sh
The validator checks:
Connection — /reset returns HTTP 200
Docker portability — no absolute host paths leak into the image
