This repository contains a LoRA adapter trained for the Canary Release Manager OpenEnv environment.
Model overview
Base model:unsloth/Qwen2.5-1.5B-Instruct
Training pipeline: SFT warm-start + GRPO
Hardware: Tesla T4
Environment calls: 1021
Task
The model is trained to make rollout decisions in a canary deployment environment.
It predicts:
rollout action
structured state assessment
SLO breach flag
short grounded reasoning
Supported actions in the environment include:
increase_5
increase_10
increase_25
hold
rollback
The environment is hosted separately as an OpenEnv-compatible service.
Training summary
The training run completed successfully with:
SFT warm-start completed
GRPO completed for 40 steps
100% parse success during RL
no reward pipeline errors
Heldout evaluation
Task
Pre-train
Post-train
Delta
easy
0.2646
0.5350
+0.2704
medium
0.2937
0.5043
+0.2106
hard
0.3755
0.6186
+0.2431
expert
0.4084
0.6375
+0.2291
recovery
0.8391
0.8391
+0.0000
silent
0.2828
0.4871
+0.2044
Heldout performance comparison
Heldout Comparison
Reward curve
Reward History
Before vs After Demo Examples
Example 1 — hard / seed 903
Base model action: hold
Base reward: 0.40
Trained model action: increase_25
Trained reward: 0.56
Observation: healthy canary metrics with no meaningful drift.
The base model stayed conservative and held.
The trained adapter promoted traffic and received a higher environment reward.
Example 2 — expert / seed 905
Base model action: hold
Base reward: 0.40
Trained model action: increase_25
Trained reward: 0.49
Observation: healthy metrics with slightly more drift than the easier case.
The trained adapter again promoted instead of over-holding and achieved a higher score.
Reward design
The environment reward combines multiple components, including breach detection, rollout timing and safety, promotion behavior, and reasoning quality. The setup is designed to reward correct rollout behavior while discouraging unsafe or reward-hacked shortcuts.
Key observations
Heldout performance improved across easy, medium, hard, expert, and silent tasks.
Recovery performance stayed strong and unchanged.
Parse success during RL was 100%, meaning the model reliably produced the required structured format during training.
On heldout demo examples, the trained adapter outperformed the base model by choosing more promotive actions instead of defaulting to hold.
Training metrics
Train runtime: 282.03 s
Train steps per second: 0.142
Final GRPO train loss: 0.00114
Notes / limitations
The learned policy is still narrow and often prefers aggressive promotion actions such as increase_25.
Reward saturation suggests the policy improved, but not with especially rich behavior diversity.
The before-vs-after demo improvements are real but modest.
This is a hackathon-scale training run, not a large-scale research benchmark.
The final results shown in this README come from the Hugging Face Jobs run. The linked Colab notebook is provided as the minimal reproducible training pipeline using OpenEnv + Unsloth/TRL.
