agentica-org/

DeepCoder-1.5B-Preview

DeepCoder Overview

Data

• Taco-Verified
• PrimeIntellect SYNTHETIC-1
• LiveCodeBench v5 (5/1/23-7/31/24)

Training Recipe

GRPO+

• Offline Difficulty Filtering: DAPO employs online dynamic sampling, discarding both entirely correct and entirely incorrect samples on the fly. While this helps maintain a more stable effective batch size, it introduces significant runtime overhead due to rejection sampling. Instead, we perform offline difficulty filtering on a subset of coding problems to ensure the training dataset remains within a suitable difficulty range.
• No Entropy Loss: We observed that including an entropy loss term often led to instability, with entropy growing exponentially and ultimately collapsing training. To mitigate this, we eliminate the entropy loss entirely.
• No KL Loss: Eliminating KL loss prevents the LLM from staying within trust region of the original SFT model. This removal also obviates the need to compute log probabilities for the reference policy, thereby accelerating training.
• Overlong Filtering (from DAPO): To preserve long-context reasoning, we mask the loss for truncated sequences. This technique enables DeepCoder to generalize to 64K-context inference despite being trained with a 32K context.
• Clip High (from DAPO): By increasing the upper bound in GRPO/PPO’s surrogate loss, we encourage more exploration and more stable entropy.

Iterative Context Lengthening

Evaluation

Serving DeepCoder

• vLLM
• Hugging Face Text Generation Inference (TGI)
• SGLang
• TensorRT-LLM

License

Acknowledgement

• Our training experiments are powered by our heavily modified fork of Verl, an open-source post-training library.
• Notably, we train 1.5B with verl pipeline, an extension of the original verl.
• Our model is trained on top of DeepSeek-R1-Distill-Qwen-1.5B.
• Our work is done as part of Berkeley Sky Computing Lab and Berkeley AI Research.

Citation