This repository contains a LoRA adapter trained for the Canary Release Manager OpenEnv environment using Qwen2.5-7B-Instruct with SFT warm-start + GRPO.
Model overview
Base model:unsloth/Qwen2.5-7B-Instruct
Training pipeline: SFT warm-start + GRPO
Hardware: NVIDIA A100-SXM4-80GB
Environment calls: 2109
GRPO runtime: 460.03 s
Final GRPO train loss: 0.00210
Task
The model is trained to make rollout decisions in a canary deployment environment from telemetry signals.
It predicts:
rollout action
structured state assessment
breach / warning interpretation
short grounded reasoning
Supported rollout actions in the environment include:
increase_5
increase_10
increase_25
hold
rollback
The environment is hosted separately as an OpenEnv-compatible service.
Training summary
This run completed successfully with:
SFT enabled
GRPO enabled
parse_ok telemetry of 100% during RL
reward_error_rate of 0.0
diagnostic verdict: PASS
Evaluation summary
Before training
Task
Score
easy
0.0500
medium
0.0500
hard
0.7298
expert
0.2907
recovery
0.4605
silent
0.4931
Average
0.3457
After SFT
Task
Score
easy
0.6456
medium
0.5667
hard
0.6081
expert
0.7358
recovery
0.5954
silent
0.5113
Average
0.6105
Final model (after GRPO)
Task
Score
easy
0.6800
medium
0.7064
hard
0.6358
expert
0.6461
recovery
0.7282
silent
0.5353
Average
0.6553
Generalization / stress
Generalization average: 0.6553
Stress average: 0.6544
Heldout / generalization comparison
Before vs After Scores
Reward curve
Reward Curve
Per-task reward during GRPO
Per Task Reward
GRPO loss curve
Loss Curve
Action distribution during RL
Action Distribution
Key observations
The final model improved the overall average score from 0.3457 before training to 0.6553 after GRPO.
The RL stage improved over the post-SFT checkpoint, increasing the average from 0.6105 to 0.6553.
Heldout and generalization averages are identical in this run, suggesting stable behavior across the chosen evaluation seeds.
The learned policy uses multiple rollout actions during RL telemetry, including increase_25, increase_10, rollback, and hold.
Reward and telemetry
The final run reported:
parse_ok_rate: 1.0
reward_error_rate: 0.0
diagnostic verdict: PASS
Action counts during RL telemetry:
increase_25: 155
increase_10: 118
rollback: 28
hold: 19
Training metrics
Global GRPO steps: 80
Reward calls: 80
Train runtime: 460.03 s
Train steps per second: 0.174
Training loss: 0.00210
Evidence included in this repository
This repository includes:
reward curve
per-task reward plot
GRPO loss curve
before/after/generalization comparison
action distribution plot
expert trace
structured JSON results
Trace note
A saved expert trace (seed=950) achieved an episode score of 0.4975. The trace shows some sensible warning/hold behavior, but it also still includes parse failures in several steps. This means the model is meaningfully improved over the raw baseline, but it is not yet a perfectly reliable controller.
Notes / limitations
The final average score (0.6553) is a solid improvement, but it is still below the hand-coded shallow and cautious reference policies on most tasks.
The policy still leans toward promotive actions, especially increase_25 and increase_10.
Some trace-time generations still fail strict JSON parsing even though RL telemetry reported perfect parser success.
Silent-task performance remains weaker than easy, medium, and recovery settings.
This is a hackathon-scale training run, not a production deployment or a large-scale research benchmark.
