LoRA adapter for Qwen-2.5-1.5B-Instruct, post-trained with GRPO on the Eco-Logistics OpenEnv environment — a 3-warehouse supply chain task that forces the agent to balance profit against carbon emissions under non-stationary demand shocks.
Submitted to the OpenEnv Hackathon — India 2026 (theme: World Modeling for Professional Tasks).
Headline result
On the held-out hardest task (net_zero_profit), 3-run averaged across 30 episodes:
25.6× improvement in profit-per-carbon ratio vs the base model
Grader 0.273 ± 0.019 (vs base Qwen 0.259)
The trained policy generalizes from the medium training task (inventory_balanced) to the harder evaluation task (net_zero_profit) with stable variance across re-runs.
Visual comparison
Held-out grader scores
Profit/carbon ratio improvement
Cross-task evaluation
We evaluate the same trained model on two tasks. Both use held-out seeds 500–509.
Task 1 — inventory_balanced (training-distribution, single 10-episode run)
We trust the net_zero_profit numbers more — 3-run averaging gives σ=0.019 on grader, and inventory_balanced showed substantial single-run variance (re-runs gave profit/carbon of 4421/695 and 4767/217). The heuristic still wins on grader (0.292), but our learned policy closes ~67% of the gap between base Qwen and the heuristic without any env-specific rules.
The model expects a ChatML prompt with the system message describing the 3-warehouse task and the initial observation. It outputs a JSON array of 10 actions (one per episode step). Full prompt template and rollout loop are in the training notebook.
Training details
Method: GRPO via TRL + Unsloth, LoRA r=16, 4-bit quantization
Hardware: Single T4 GPU (Colab)
Steps: 30 GRPO steps, learning rate 2e-6
Dataset: 50 unique initial-state prompts across all 3 tasks, seeds 0–49
Held-out eval: seeds 500–509 (never seen during training)
Design choice — Upfront Trajectory Planning: the model emits the entire 10-step plan as one JSON array per inference. Cuts HTTP round-trips by 10×, makes GRPO-over-HTTP tractable on a T4.
Anti-reward-hacking patch (critical detail)
The first training run collapsed at step 15: the model discovered that outputting invalid JSON triggered a SAFE_FALLBACK_ACTIONS plan that shipped almost nothing, which meant near-zero carbon penalty. The −5.0 format penalty was smaller in magnitude than the carbon savings, so the model learned to output garbage on purpose. Fix: change the format penalty to −1000.0, making invalid output strictly worse than any reachable valid reward. Full debugging story in the writeup.
A documented negative result: SFT-then-GRPO
We attempted to fix our weak valid-action rate (~20%) with an SFT warmup phase. SFT did fix valid-rate (20% → 80%), but subsequent GRPO collapsed in 3 different reward configurations. Diagnosis: SFT made the policy too uniform, so GRPO had no variance among the N=4 sampled completions to learn from. Detailed in the writeup. We submit the no-SFT model as our headline; the SFT experiment is documented as honest evidence of what we tried.
Limitations
Valid-action rate of 20% on held-out seeds. The biggest weakness — model produces parseable JSON only 1 in 5 times; the rest fall through to a heuristic fallback.
Heuristic still beats us on grader (0.292 vs 0.273). Not a SOTA claim.
Upfront planning constraint — agent commits to all 10 steps at t=0 without intermediate replanning.
Profit didn't generalize from inventory_balanced (+27%) to net_zero_profit (essentially flat). The carbon-efficiency story is what travels across tasks.
