CodeFu-7B-v0.1 is a 7B parameter model trained using Reinforcement Learning for competitive programming tasks. Built on the DeepSeek-R1-Distill-Qwen-7B base model, CodeFu is capable of algorithmic reasoning to solve complex problems and generate efficient C++ solutions.
Specicially, CodeFu-7B-v0.1 achieves 13.7% Pass@1 on the USACO benchmark, outperforming models >4x larger.
Trained solely on problem statements—without access to any ground-truth solutions—CodeFu achieved >10x performance improvement over its base model, demonstrating the effectiveness of our RL approach..
To assess CodeFu's genuine problem-solving abilities, we used USACO benchmark, which consists of 307 high-quality problems from the past USA Computing Olympiad contests.
⚡ Outperforms 32B base model (13.7% vs 11.7% Pass@1)
📈 >10x improvement over 7B base model (13.7% vs 1%)
For systematic and robust evaluation, we used standardized code extraction logic across all model responses. This process identifies solution code by parsing either <code></code> tags or ```cpp code blocks, always selecting the final code block to ensure we capture each model's ultimate solution after any intermediate reasoning steps. GPT-3.5/4 scores are copied from the USACO benchmark as baselines
All extracted code solutions are executed with strict time limit enforcement - any code exceeding the problem's specified time limit is marked as incorrect, ensuring realistic competitive programming conditions.
All open-weight models were tested using vLLM v0.6.3 with identical sampling parameters: a temperature of 0.8 and a top_p of 0.95. Claude-3.7-Sonnet was evaluated at a temperature of 1.0. We set the maximum output length (max_tokens) to 28,672 for all models to ensure sufficient length for reasoning and code solutions.
Result analysis
We provide access to the complete CodeFu-7B-v0.1 evaluation results on the USACO benchmark as a CSV file containing fields such as problem_name, prompt, response, response_length, solution_code, status, and score. Notably, the status field breakdown is as follows:
Success: 42 cases
Failure (code runs but incorrect or timed out): 37 cases
Fail to compile: 8 cases
No code: 220 cases
Analysis of the response length distribution shows that successful solutions typically have concise responses around 5,000 tokens, while unsuccessful attempts often reach the maximum token limit. While some correct solutions do exceed 20,000 tokens, the vast majority of long responses correspond to the "No code" category, where the model engages in extensive reasoning that eventually degenerates into repetitive patterns or incoherent text without producing executable code. Future work is needed to improve training objectives that better distinguish between useful deliberation and unproductive verbosity.
Usage
python
1# CodeFu works with vLLM for inference2# pip install vllm==0.6.334from vllm import LLM, SamplingParams
56model_name ="aws-prototyping/codefu-7b-v0.1"78# Initialize vLLM9llm = LLM(model=model_name, trust_remote_code=True)10sampling_params = SamplingParams(11 temperature=0.8,12 top_p=0.95,13 max_tokens=28672,14)1516# The `Hay Bales` problem in USA Computing Olympiad benchmark17prompt ="""In your role as an algorithmic problem-solver, write a C++ solution for this problem. Put your thought process in <think> tags and your solution in <code> tags.
1819Problem:
20Problem 1: Hay Bales [Brian Dean, 2011]
2122The cows are at it again! Farmer John has carefully arranged N (1 <= N <=
2310,000) piles of hay bales, each of the same height. When he isn't
24looking, however, the cows move some of the hay bales between piles, so
25their heights are no longer necessarily the same. Given the new heights of
26all the piles, please help Farmer John determine the minimum number of hay
27bales he needs to move in order to restore all the piles to their original,
28equal heights.
2930PROBLEM NAME: haybales
3132INPUT FORMAT:
3334* Line 1: The number of piles, N (1 <= N <= 10,000).
3536* Lines 2..1+N: Each line contains the number of hay bales in a single
37 pile (an integer in the range 1...10,000).
3839SAMPLE INPUT:
40414
422
4310
447
451
4647INPUT DETAILS:
4849There are 4 piles, of heights 2, 10, 7, and 1.
5051OUTPUT FORMAT:
5253* Line 1: An integer giving the minimum number of hay bales that need
54 to be moved to restore the piles to having equal heights.
5556SAMPLE OUTPUT:
57587
5960OUTPUT DETAILS:
6162By moving 7 hay bales (3 from pile 2 to pile 1, 2 from pile 2 to pile 4, 2
63from pile 3 to pile 4), we can make all piles have height 5.
64"""6566# Generate solution67outputs = llm.generate([prompt], sampling_params)68solution = outputs[0].outputs[0].text
69print(solution)7071# Alternative: OpenAI-compatible API server72# Start vLLM server first:73# python -m vllm.entrypoints.openai.api_server --model aws-prototyping/codefu-7b-v0.1 --port 80007475from openai import OpenAI
7677client = OpenAI(78 api_key="EMPTY",79 base_url="http://localhost:8000/v1"80)8182response = client.completions.create(83 model="aws-prototyping/codefu-7b-v0.1",84 prompt=prompt,85 temperature=0.8,86 top_p=0.95,87 max_tokens=28672,88)8990solution = response.choices[0].text
91print(solution)
We can examine CodeFu's generated solution for this problem, which has been verified as correct.
Prompt Format
CodeFu works best with structured prompts that request both reasoning and code:
[Role] Please solve this programming problem in C++. Show your thinking process in <think> tags and provide your solution in <code> tags.
[Problem Description]
Replace [Role] with phrases like:
"As a competitive programming expert"
"Working as an experienced competitive programmer"
"As a master of algorithms and data structures"
CodeFu Training Pipeline
train_arch
Figure 1 - CodeFu training pipeline
We trained CodeFu using the veRL library. Ray orchestrates the distributed execution and synchronization of vLLM rollout, reward evaluation (code compilation and execution), FSDP model parallelism, and Ulysses sequence parallelism. We set the degree of sequence parallelism to 4 for long-form reasoning and code generations.
We extended the TinyZero code repository by using Ray to manage and distribute reward function calculation. This enables parallel C++ code compilation and evaluation across the same cluster to address the compute-intensive and latency-bound nature of code execution. The entire pipeline is executed as a SageMaker training job running on ml.p4de.24xlarge instances. The training pipeline consists of the following steps as shown in Figure 1:
Rollout: Coding problem prompts are fed into the vLLM inference engine for rolling out potential solutions
Response Generation: vLLM generates multiple responses (reasoning + code) for each prompt
Code Execution: Code solutions are extracted from responses, and are compiled and executed by distributed workers (compilers and runtime) managed by Ray
Reward Calculation: Execution outcomes are used to calculate rewards (i.e. testcase pass ratios) and advantages are computed using group-relative baselines
Policy Update: The Actor uses advantages and token probabilities to compute the PPO loss, which is then used to update CodeFu's parameters through gradient descent
Iteration: The process repeats with batches of prompt-response-reward cycles, with Ray managing the distributed sampling, execution, and training synchronization across the pipeline
CodeFu Training Approach
CodeFu employs a 2-stage curriculum learning approach:
Stage
Data
Max resp token
Batch size
Mini batch size
# of Rollouts
Reward
Focus
# of nodes
1
easy problems
28K
8
8
5
Exp smooth w. public scores
Basic algorithmic reasoning
1
2
hard problems
20K
256
32
8
Linear w/o public scores
Quality and Robustness
4
Reward
Stage 1 (Exponential smooth with public scores) - Uses exponential smoothing and both public/private test cases for clearer learning signals on easier problems.
Stage 2 (Linear without public scores) - Shifts to linear rewards using only private test cases to encourage robust problem-solving on harder problems.
Here is the pseudocode for the reward calculation across both training stages:
Training data is sourced from CodeForces problems within the DeepMind CodeContest dataset, chosen for their reliable CF rating system. Easy problems (CF rating 800-1000) are used in Stage 1 for basic algorithmic reasoning, while relatively Hard problems (CF rating 1100-2200) are used in Stages 2 for intermediate to advanced challenges. Both the Easy and Hard datasets were trained for approximately 2 epochs.
Training Stability
We encountered a response length collapse issue when training CodeFu on the hard problems dataset - the mean response length would plunge significantly after some period of learning, causing a catastrophic drop in training rewards as shown in Figure 2.
Mean Response Length
Figure 2 - Mean response length and reward collapsed with a small batch size
Despite attempts to mitigate this through increased batch sizes (up to 416) as shown in Figure 3, increased rollout samples (up to 32), plain PPO implementation with a co-trained critic model, and KL coefficient adjustments, the collapse persisted or reappeared after periods of stability.
Mean Reward
Figure 3 - Mean response length and reward plummet despite the much larger batch size
We eventually resolved this issue by using a true off-policy PPO learning configuration. This is achieved by reducing the mini-batch size to at least 4× smaller than the global batch size, resulting in multiple clipped "mini-" updates as per the original PPO algorithm. This approach has since stabilized response length and prevented reward collapse (Figure 4), allowing us to pass multiple epochs on the dataset.
Mean Reward
Figure 4 - Mean response length and reward do not collapse during true off-policy learning
A detailed paper is in preparation that will describe our training stability solutions and review related work on policy optimization for reasoning models, including recent methods like DAPO, OPO, Dr.GRPO, and GSPO.
Citation
CodeFu is developed by the AWS WWSO Prototyping Team. If you find CodeFu helpful, feel free to give us a cite.
