Guided Reasoning Window Optimization (GRWO) is a training strategy for improving reasoning models by optimizing local reasoning windows instead of full long-form solution traces.
The core idea is simple:
Given a problem and a partial reasoning prefix, train the model to prefer a better next reasoning window over a weaker next reasoning window.
Instead of forcing a model to generate and train on a full 5k–10k token solution, GRWO focuses on the next N tokens of reasoning, usually around a meaningful branch point.
This makes reasoning training cheaper, more targeted, and more aligned with how models actually fail.
1. Motivation
Long reasoning traces are expensive.
A hard math problem may require thousands of tokens of exploration, correction, theorem selection, calculation, and verification. Training directly on full traces has several problems:
It is costly to generate.
It is costly to train.
It may overwrite the model's natural reasoning style.
It gives loss on many tokens that are not the real failure point.
It can teach verbosity instead of better reasoning decisions.
Most reasoning failures happen at local branch points:
The model chooses the wrong theorem.
The model treats an inconclusive test as proof.
The model chooses bad coordinates.
The model forgets a Jacobian.
The model keeps exploring instead of finishing.
The model gives a confident but unsupported conclusion.
GRWO targets these local branch points directly.
2. One-Line Thesis
Train reasoning models by correcting local self-generated reasoning windows instead of supervising full solution traces.
3. Core Definition
A GRWO sample contains:
problem + partial reasoning prefix
Then the model generates two possible continuations:
text
1rejected = unguided continuation
2chosen = guided or corrected continuation
Only the next local reasoning window is optimized.
text
1prompt = problem + partial reasoning prefix
2chosen = better next reasoning window
3rejected = weaker next reasoning window
The key constraint:
The keypoints or guided answer are used only to create the chosen continuation. They are not exposed in the student model's prompt.
4. Why Windowed Reasoning?
Instead of this:
Question -> full 10k-token reasoning trace -> answer
GRWO trains this:
Question + reasoning prefix -> next 512–1500 reasoning tokens
This changes the objective from:
Solve the entire problem from scratch.
to:
Given this current reasoning state, choose the better next reasoning move.
That is the behavior weak reasoning models need most.
5. Name
Recommended method name:
Guided Reasoning Window Optimization (GRWO)
Variants:
text
1GRWO-DPO = DPO over local reasoning windows
2GRWO-SFT = SFT on corrected local reasoning windows
3GRWO-GRPO = RL/GRPO over local reasoning windows
Primary direction:
GRWO-DPO: preference training over local self-generated reasoning continuations.
6. Dataset Structure
6.1 DPO Format
json
1{2"prompt":"Problem statement...\n<think>\nPartial model-generated reasoning prefix...",3"chosen":"Guided/correct next reasoning window...",4"rejected":"Unguided/weaker next reasoning window..."5}
The prompt contains the problem and the partial reasoning prefix.
The chosen and rejected completions should continue from the exact same prefix.
6.2 SFT Format
json
1{2"prompt":"Problem statement...\n<think>\nPartial model-generated reasoning prefix...",3"completion":"Correct next reasoning window..."4}
SFT is optional, but useful when the model lacks a reasoning move entirely.
7. Core Training Modes
7.1 GRWO-DPO
GRWO-DPO is the preferred main method when the base model already has decent reasoning ability and follows the desired format.
It teaches:
This reasoning direction is better than that one.
This theorem choice is better than that theorem choice.
This continuation is more grounded than the weaker continuation.
This recovery path is better than wandering.
This confidence is justified; that confidence is not.
GRWO-DPO preserves the model's natural language style better than pure SFT because it does not force exact imitation.
7.2 GRWO-SFT
GRWO-SFT is useful when the model does not know how to produce a required reasoning move.
It teaches:
How to perform a missing transformation.
How to recover from a known trap.
How to continue after an inconclusive test.
How to set up a correct local calculation.
SFT should be used carefully because it can overwrite the model's native style if the teacher traces are too polished or unnatural.
7.3 GRWO-GRPO
GRWO-GRPO is a later-stage method.
It can be used when DPO/SFT are not enough and you want online sampling with a reward function.
Possible reward components:
text
1+ correct final answer, if available
2+ correct theorem/test selection
3+ valid transformation
4+ keypoint coverage
5+ forward progress
6+ calibrated confidence
7- invalid conclusion
8- theorem misuse
9- wandering
10- fake final answer
11- malformed protocol tags
8. Recommended First Pipeline
Start with GRWO-DPO-first if the model already follows the format and can produce plausible reasoning.
text
11. Select a hard problem.
22. Let the model generate a partial reasoning prefix.
33. Stop around a meaningful branch point.
44. Generate an unguided continuation for the next N tokens.
55. Generate a guided/corrected continuation using keypoints, verifier, or teacher.
66. Build a DPO pair:
7 prompt = problem + partial reasoning prefix
8 chosen = guided continuation
9 rejected = unguided continuation
107. Train with DPO on only this local continuation window.
Optional:
8. Add a small SFT bucket for cases where the model cannot produce the desired move at all.
The model does not automatically learn to stop at 1500 tokens just because generation is capped there. The cap is an external data-generation/training budget.
However, avoid making accepted samples look like naturally finished answers if they are actually truncated.
For process-window training, it is acceptable for the chosen continuation not to finish the whole problem. The objective is local reasoning continuation, not full solution completion.
10. Loss Scope
The loss should only apply to the local continuation window.
text
1problem tokens: no loss
2partial reasoning prefix: no loss
3next N local window tokens: loss active
4future tokens after window: not included / no loss
For SFT, this means:
text
1labels for prompt/prefix tokens = -100
2labels for continuation tokens = token IDs
For DPO, the row should be structured so that:
text
1prompt = prefix
2chosen = continuation window only
3rejected = continuation window only
Do not include future tokens after the supervised window, because that leaks information.
11. Good Chosen vs Rejected Design
The chosen and rejected continuations should be as similar as possible except for reasoning quality.
Good pair design:
text
1same prompt
2same prefix
3similar length
4similar format
5similar style
6similar confidence level visually
7only reasoning direction differs
Bad pair design:
text
1chosen = polished teacher solution
2rejected = messy model rambling
That would teach surface style instead of reasoning direction.
Better pair:
text
1chosen = model-like continuation, corrected at the branch
2rejected = model-like continuation, wrong branch
12. Example Reasoning Trap
Problem type: series convergence.
Partial reasoning prefix:
text
1The absolute series behaves like the harmonic series, so it does not converge absolutely.
2Now I need to determine whether the original alternating series converges conditionally.
3The alternating series test is awkward because monotonicity is not obvious...
Rejected continuation:
Since the alternating series test fails, the series diverges.
Chosen continuation:
text
1But failure of the alternating series test would only be inconclusive, not proof of divergence.
2I should try another method. A useful approach is to rewrite the term as an alternating harmonic component plus an absolutely convergent correction...
This pair teaches the model:
A failed test is inconclusive, not proof of divergence.
That is a local reasoning correction.
13. Reasoning Categories to Target
Good GRWO windows should target common reasoning branch errors.
Series
Ratio test equals 1 means inconclusive.
Root test equals 1 means inconclusive.
Alternating series test failure is inconclusive.
Absolute convergence must be checked separately.
Conditional convergence requires convergence without absolute convergence.
nth-term test only proves divergence when term limit is nonzero.
Use SFT only when the model cannot produce the desired reasoning move at all.
15. Unguided Correct Samples
Do not automatically reject all unguided continuations.
If the unguided continuation is correct, either:
text
1keep it as a positive SFT sample
2use it as chosen against a weaker continuation
3or skip it
Do not train the model to believe all of its natural reasoning is bad.
The rule:
text
1unguided correct -> positive or skip
2unguided wrong -> rejected against guided correction
3unguided wandering -> rejected against progress continuation
16. Confidence Calibration
DPO can teach confidence in language even when both samples are technically correct.
Example rejected:
This probably converges because it looks alternating.
Example chosen:
text
1The absolute series behaves like the harmonic series, so it is not absolutely convergent.
2The remaining signed series can be rewritten as an alternating harmonic term plus an absolutely convergent correction, so it converges conditionally.
Both may reach the same answer, but the chosen one teaches:
Better justification.
More grounded confidence.
Cleaner theorem use.
Stronger conclusion control.
The goal is not confidence alone.
The goal is:
calibrated confidence
The model should be decisive when the logic supports it and cautious when a test is inconclusive.
17. Training Configuration Starting Point
For GRWO-DPO on a small high-quality dataset:
yaml
1learning_rate: 5e-6 to 1e-52beta: 0.03 to 0.05
3num_train_epochs: 1 to 2
4prompts_per_step: 5 to 10
5dpo_epochs_per_step: 3 to 4
6max_length: 3072 to 4096
7max_prompt_length: 1024 to 1536
8max_completion_length: 1536 to 3072
9per_device_train_batch_size:110gradient_accumulation_steps: 4 to 8
11save_steps:2512logging_steps:1
For GRWO-SFT, if used:
yaml
1learning_rate: 5e-5 to 1e-42num_train_epochs: 1 to 2
3max_length: 3072 to 4096
DPO should usually be softer than SFT because it can over-steer quickly.
18. Evaluation
Evaluate GRWO by behavior, not just loss.
Test whether the model:
Avoids known reasoning traps.
Recovers from inconclusive tests.
Chooses better next steps.
Keeps protocol format.
Does not hallucinate unsupported conclusions.
Does not become too short or robotic.
Maintains mathematical accuracy.
Knows when to continue and when to finish.
Useful benchmark prompts:
text
11. Series trap where AST failure is inconclusive.
22. Ratio/root test equals 1 trap.
33. Double integral with coordinate-bound trap.
44. Spherical-coordinate Jacobian trap.
55. Problem where early answer is tempting but wrong.
66. Problem where reasoning should finish instead of wander.
19. Expected Benefits
GRWO should reduce training/data cost because it avoids full-trace generation.
1Experiment A: GRWO-DPO only
2Experiment B: small GRWO-SFT + GRWO-DPO
Compare:
base/SFT model vs GRWO-DPO vs GRWO-SFT+DPO
Main question:
Does the model choose better reasoning branches under the same token budget?
23. Current Working Definition
GRWO is a process-preference training method where a reasoning model is optimized on local continuation windows from its own partial traces. A guided continuation is preferred over an unguided continuation under the same prefix, with loss applied only to the local window. The goal is to improve reasoning trajectory decisions while preserving the model's native reasoning style and reducing full-trace training cost.
24. Research Notes
GRWO is promising because it combines three useful ideas:
Self-generated prefixes preserve the model's natural reasoning distribution.
