Replication and Exploration of the DeepSeek-R1 《Nature》 Paper Methodology 🚀
Developed by: Soren
gpt-oss-120b-Distill-Phi-4-14B is a GRPO inference test model based on the microsoft/phi-4 (14B) base model. It is fine-tuned through a three-stage training pipeline—SFT (Cold Start) → RLHF (GRPO) → SFT—aiming to achieve excellent multi-step reasoning capabilities.
The core methodology of this model is derived from three papers: "DeepSeek-R1" (Nature, vol. 645), "DeepSeekMath" (arXiv:2402.03300v3), and "Phi-4-reasoning" (arXiv:2504.21318).
However, unlike the Microsoft paper, I moved the reinforcement learning-guided reasoning part to an earlier stage. Referencing DeepSeek's approach, I used the SFT + GRPO + SFT method to generate explicit "Chain-of-Thought" (CoT), attempting to induce reasoning abilities in the Phi-4 Instruct model through reinforcement learning and supervised learning.
Different from the standard fine-tuning with the o3-mini teacher model in the Microsoft paper, this model uses a distilled dataset from the open-source model gpt-oss-120b as its fine-tuning knowledge base, thereby inheriting its profound reasoning style and knowledge system.
Fig. 1 | The multistage pipeline of DeepSeek-R1. A detailed background on DeepSeek-V3 Base and DeepSeek-V3 is provided in Supplementary Information, section 1.1. The models DeepSeek-R1 Dev1, Dev2 and Dev3 represent intermediate checkpoints in this pipeline.
Here, I also adopted the method from the paper. According to the description in "DeepSeek-R1 incentivizes reasoning in LLMS through reinforcement learning" (doi: 10.1038/s41586-025-09422-z), the main purpose of the "cold start" (the initial supervised fine-tuning SFT stage) before GRPO reinforcement learning is to address several key deficiencies exposed by the initial model trained purely through reinforcement learning (DeepSeek-R1-Zero).
Specifically, this cold start stage aims to:
Improve Readability and Align with Human Preferences: Although the purely RL-trained DeepSeek-R1-Zero has strong reasoning abilities, its outputs suffer from "poor readability" and "language mixing" (e.g., mixing Chinese and English). The cold start stage uses thousands of high-quality data samples that "demonstrate a conversational style aligned with human thought processes" for fine-tuning, aiming to make the model's expression and thinking process more aligned with human habits and preferences.
Lay the Foundation for the Subsequent RL Stage: This step acts as a "calibration" for the model, enabling it to adhere to human-expected language and format specifications while maintaining its powerful reasoning capabilities. This paves the way for the subsequent reinforcement learning stage, which incorporates more complex tasks (such as general conversation, writing, etc.), ensuring that the final model (DeepSeek-R1) is not only strong in reasoning but also has good generality and a better user experience.
Stage 2: Reinforcement Learning with GRPO
Fig. 2 | Illustration of the proposed GRPO for RL-based training.
Algorithm:GRPO (Group Relative Policy Optimization), an advanced RLHF algorithm that estimates the advantage function by comparing the relative merits of a group of candidate outputs, thereby achieving more stable policy gradient updates.
Datasets:open-r1/DAPO-Math-17k-Processed and openai/gsm8k are used as preference datasets.
Dataset
Primary Use
Data Format/Features
Role in Project
open-r1/DAPO-Math-17k-Processed
This dataset has a high difficulty level and a wide range of difficulties, from basic math to advanced mathematics. It is used to guide the model's reinforcement learning for reasoning abilities.
Contains English math problems and their corresponding solutions. - prompt: The text of the math problem. - solution: The standard answer to the problem, usually a number or an expression.
In the model's GRPO training process, this dataset is the main training source. The model's task is to generate a reasoning process and answer based on the prompt, and then use the solution as a benchmark to calculate the reward score (e.g., validated through check_answer and check_numbers functions), thereby optimizing the policy.
openai/gsm8k
Benchmark testing and training on grade school level math word problems. Guides the model to learn reasoning with ease.
Contains question and answer fields. - The answer section usually includes detailed reasoning steps (Chain-of-Thought) and ends with the format #### <Correct Answer>.
In the model's GRPO training process, this dataset is used as a source of English math problems. Its standard answers are precisely extracted to provide an objective and reliable standard for the reward function (correctness_reward_func) that evaluates the correctness of the model's output. In DeepSeekMath (another paper by DeepSeek), it is also used in the reinforcement learning stage.
Auxiliary, used only to prevent overfitting to mathematical formats.
A distilled dataset containing Chinese Chain-of-Thought. The code samples 2,000 instances from the train split of this dataset with a random seed of 20250924.
Its Answer_content field provides complete reference answers, which are not only used to evaluate content similarity but also serve as a reference for length, style, and format alignment, providing guidance for multiple reward functions (content_alignment_reward_func, zh_quality_reward_func).
Objective: Building on the SFT cold start, this stage uses reward signals to guide the model to explore and learn better reasoning strategies. The core goal is not simply to imitate human-annotated solution paths, but to allow the model to autonomously evolve more powerful and diverse reasoning abilities with only feedback on the correctness of the final answer. This includes two main objectives:
Guide the model to explore reasoning thought processes: Incentivize the model to generate detailed, structured, and logically coherent Chains-of-Thought (CoT), and even develop strategies that surpass human examples, such as self-reflection, verification, and multi-perspective exploration.
Improve the correctness of the final answer: Ensure that while optimizing the reasoning process, the model can more reliably converge to the correct final answer (completing reasoning and answering within the 8192 token limit, avoiding reasoning for the sake of reasoning).
Algorithm:GRPO (Group Relative Policy Optimization), an advanced and efficient reinforcement learning algorithm, is used in this project as a variant of Proximal Policy Optimization (PPO). Unlike traditional PPO, which requires training an additional value network (Value Model / Critic), GRPO simplifies the training process and significantly reduces resource consumption in the following ways:
Group Sampling: For each problem, the Policy Model generates a group containing G candidate outputs. In this project's code, the group size num_generations is set to 4.
Reward Evaluation: A complex reward system composed of multiple functions scores each candidate output within the group, resulting in a comprehensive scalar reward score r.
Group Relative Advantage Estimation: The core of GRPO is that it does not rely on a separate value network to estimate the baseline. Instead, it directly uses the average reward of all candidate outputs within the group as the baseline. It estimates the advantage function A by calculating the deviation of each output's reward from the group average (e.g., by normalizing using the mean and standard deviation).
Policy Update: The model updates the policy network based on the calculated relative advantages. Outputs with rewards higher than the group average are positively reinforced, while those below the average are suppressed. This makes the policy gradient updates more stable and naturally aligns with the comparison-based training of reward models.
Important: Reward Function Construction
Fig. 6 | DeepSeek Reward function setup and inference label guidance.
Reward System: To finely guide the model's optimization from multiple dimensions, the code implements a complex and comprehensive reward function system (theoretically, a fourth RL step for preference alignment should follow SFT, but please understand the resource constraints). It combines multiple weighted rewards and penalty signals to shape the model's behavior. The system can be divided into the following modules:
Core Objective Rewards:
correctness_reward_func: Based on the reference answers from gsm8k and DAPO-Math-17k-Processed, this function gives the highest positive reward for the correctness of the final numerical calculation, serving as the core signal to ensure the model learns to solve problems.
strict_format_reward_func: Strictly enforces the <think>...</think> reasoning output format and places the answer within <answer> and </answer> tags for easy verification by the answer reward function. It directly deducts points for format errors (such as missing, repeated, or out-of-order tags), ensuring the parsability of the reasoning process.
Quality & Alignment Rewards:
Content and Length Alignment: The content_alignment_reward_func (based on unigram F1 score) and ref_length_alignment_reward_func encourage the model to generate responses similar in content and length to high-quality reference answers.
Style and Format Alignment: The md_style_reward_func, style_match_to_ref_reward_func, and math_layout_reward_func incentivize the model to use more readable Markdown formats (headings, lists), correctly use LaTeX for mathematical typesetting, and learn from excellent formats in reference answers (tables, code blocks).
Chinese Quality and Language Consistency: The zh_quality_reward_func specifically targets Chinese content, comprehensively evaluating the proportion of Chinese, content F1 score, and Markdown structure similarity.
Behavioral Regularization & Penalties:
Anti-Redundancy and Short-sightedness: The anti_redundancy_penalty_func penalizes repetitive phrases and filler words in the Chain-of-Thought. The min_answer_penalty_func penalizes behaviors where the reasoning seems detailed but the final answer is overly brief. It accounts for reasoning words in the distilled dataset such as "because", "so", "then", "no, wait", "check", "re-check", "verify" in both Chinese and English.
Length Control: The length_reward_func and think_soft_cap_penalty_func work together. One function judges the problem content and sets ideal length ranges for reasoning and answers based on the problem type (math vs. general long-form text). It also applies a soft penalty to "thinking" processes that exceed 2000 characters, which corresponds to the observation in research that RL can spontaneously increase CoT length.
Exploration & Diversity Incentives:
intra_sample_style_diversity_reward_func & intra_sample_reasoning_diversity_reward_func: This is a new set of reward functions used to evaluate the diversity among multiple candidate answers generated for the same problem. By rewarding outputs with more diverse styles and reasoning paths, it actively encourages the model to break out of a single optimal path and explore a wider space of problem-solving strategies, thereby improving the model's robustness (Maj@K performance).
Stage 3: Post-RL SFT (Alignment and Capability Consolidation)
Fig. 3 | Learning rate and hyperparameter recommendations in the Phi-4 paper.
Fig. 4 | System prompts used during training.
Reference Paper:"Phi-4-reasoning Technical Report" (arXiv:2504.21318)
My implementation primarily references Section 3 "Phi-4-reasoning: Supervised Finetuning of Phi-4" of the paper. My method also adopts the core strategy emphasized in the paper, which is to use a powerful teacher model (the paper uses o3-mini; my dataset includes distilled data from gpt-oss-120b-high, DeepSeek-R1 models) to generate high-quality training samples containing long Chains-of-Thought.
The system_message I constructed has a purpose and structure consistent with the system prompt shown on page 10 of the paper, guiding the model to generate structured and detailed reasoning processes containing <think> tags.
Objective: To mitigate the "Alignment Tax" that reinforcement learning might introduce, which is the model's forgetting of general instruction-following abilities while optimizing for a specific capability. It involves calling upon more powerful reasoning teacher models for supervised learning to generalize, enhance, and expand knowledge, much like after a high school group discussion, the teacher now standardizes the solutions and imparts new knowledge.
Process: The model, optimized through GRPO, is once again put into the original 87k-sample SFT dataset for a supervised fine-tuning with a large effective batch size. This step aims to merge and calibrate the reasoning strategies learned in the RL phase with the broad knowledge acquired in the SFT phase.
Dataset Name/Source
Sample Size
Description
Jackrong/Natural-Reasoning-gpt-oss-120B-S1
70,000
Core Dataset. Distilled from facebook/natural_reasoning, providing high-quality, high-difficulty general reasoning problems covering STEM, economics, social sciences, etc.
Jackrong/GPT-OSS-120B-Distilled-Reasoning-math
5,000
Focuses on reasoning and problem-solving for mathematics.
Provides high-quality Chinese Chain-of-Thought data to enhance the model's Chinese reasoning abilities.
deepseek_if_shuffled-v1.json
5,000
Focuses on improving the model's ability to understand and execute complex instructions.
deepseek_code_shuffled-v1.json
2,000
Endows the model with capabilities for code analysis, bug fixing, and code generation.
Total
87,000
-
Objective: To inject structured "Chain-of-Thought" capabilities into the instruction model, teaching it how to perform explicit reasoning.
Dataset: A mixed dataset containing 87,000 samples, with the main body being Jackrong/Natural-Reasoning-gpt-oss-120B-S1, supplemented by data from multiple domains such as mathematics, code, Chinese, and instruction following.
Model's Actual Performance:
Here I show an actual test in LM Studio, using the GGUF q8 quantized model. I have copied the chain of thought and the answer for the question below for your reference:
Screenshot 2025-09-25 at 4.07.13 PM
1️⃣ General Chat Q&A Example: LLM Technology Q&A: What is reinforcement learning?
Test Question (Click to expand/collapse)
In LLM training, what is reinforcement learning?
Model's Chain-of-Thought (Click to expand/collapse)
The user asks: "In LLM training, what is reinforcement learning?" Let's dissect the question:
User wants to understand how RL works in LLM training.
They might be confused about whether it's necessary or effective. Provide clear explanation of RL fine-tuning vs supervised.
Emphasize that RL fine-tuning can improve quality but not always required. Good for alignment and safety.
Give examples: OpenAI's RLHF with ChatGPT, Gopher uses supervised + RL.
Now structure the answer:
TL;DR
How LLMs are trained (pretraining vs fine-tuning).
Introduction to RL in NLP.
Core idea of reward function and policy gradient.
Workflow: collect data -> define rewards -> train agent.
Specific algorithms used (PPO, APEX, etc.)
Benefits & drawbacks.
Use cases: alignment, safety, few-shot adaptation.
Comparison with supervised fine-tuning.
Closing remarks.
Ensure clarity and depth but avoid jargon overload. Use bullet points for readability. Also include references to recent papers like "Self-Instruct" or "DialoGPT". Check that answer aligns with the user's likely background (basic). Provide actionable insights.
Model's Output Answer (Click to expand/collapse)
Reinforcement Learning (RL) in large language model (LLM) training is a process where the model learns by receiving feedback on its outputs—essentially, it gets rewarded or penalized based on how well its responses align with desired behaviors. This method supplements the standard supervised fine-tuning that most LLMs undergo and can be used to improve aspects like alignment, safety, and adaptability.
Below is a concise yet thorough explanation of what RL means in this context:
1️⃣ TL;DR
Supervised training teaches an LLM what “correct” responses look like by comparing its outputs to human-written examples.
Reinforcement Learning (RL) adds another layer: after the model generates a response, it gets a score (the reward) that tells it how well the response fits specific goals (e.g., helpfulness, safety). The model then adjusts itself to get higher scores on future inputs.
2️⃣ How LLMs are usually trained
Pre-training:
Massive amount of unlabeled text (books, websites) is fed into a transformer network.
Objective: predict the next word given previous ones (language modeling).
Result: a very powerful “general” model that can produce coherent text.
Supervised fine-tuning:
A smaller dataset of human-annotated examples (e.g., question–answer pairs) is used.
The model’s parameters are tweaked to minimize the difference between its predictions and the target answers.
Outcome: a model that can perform specific tasks like answering questions, summarizing text, etc.
3️⃣ What RL brings into the mix
RL introduces a feedback loop where the model is not just shown “correct” examples; it’s also told how good or bad its own generated responses are in a quantitative way (the reward). This can be used to:
Align better with human values: e.g., produce helpful, safe, unbiased answers.
Improve robustness and safety: learn to avoid harmful outputs even if they appear plausible.
Adapt quickly to new domains with fewer examples.
4️⃣ Core components of RL in LLM training
A. Agent
The trained LLM itself acts as the agent that produces sequences (e.g., chat messages).
B. Environment
The environment is the task context: a user query, a conversation history, etc.
C. Policy
The model’s current “policy” is its learned mapping from inputs to output distributions (i.e., what words it thinks should come next).
RL aims to improve this policy by maximizing expected reward.
D. Reward Signal
A scalar value that measures how well a generated response meets predefined objectives.
Simple examples: +1 for a correct answer, −1 for an incorrect one.
More sophisticated designs: combinations of multiple metrics (e.g., factual accuracy, politeness, toxicity) weighted by importance.
E. Training Algorithm
Common RL algorithms used with LLMs:
Proximal Policy Optimization (PPO): balances exploration and exploitation; stable and efficient.
Reinforcement Learning from Human Feedback (RLHF): a pipeline where human preferences are collected to shape the reward function.
Self-Instruction: an unsupervised RL method that generates its own training data by prompting itself with instructions.
5️⃣ Typical Workflow (RLHF pipeline)
Data Collection:
Human trainers provide pairs of model outputs and a preference judgment (“A is better than B”).
Alternatively, collect user interactions or automatically generate feedback using heuristics.
Reward Modeling:
Train a separate neural network (the reward model) to predict the human preferences from raw responses.
The reward model’s output becomes the scalar reward for each generated sequence.
RL Fine-Tuning:
Use the learned reward model to guide policy optimization of the LLM via PPO or similar algorithms.
Optionally, fine-tune a smaller “policy network” that learns from the reward signals while keeping the original LLM parameters frozen (to save computation).
6️⃣ Why RL is useful for LLMs
Aspect
Supervised Fine-Tuning
Reinforcement Learning
Data
Requires labeled examples for each task
Can leverage large amounts of unlabeled data + fewer human labels
Alignment
Limited to what the dataset contains; can be biased
Directly optimizes for user-defined goals (e.g., helpfulness)
Exploration
Always produces deterministic responses based on training set
Encourages exploration, enabling discovery of new strategies or behaviors
Safety
Relies heavily on the quality/cleanliness of labels; hard to guarantee safety
Can explicitly penalize unsafe outputs and reinforce safe ones
Flexibility
Fixed once trained; adapting requires re-training with new data
Continual adaptation possible via online RL (e.g., user feedback loop)
7️⃣ Challenges & Trade-offs
Reward Engineering: Designing a reward that captures nuanced human values is hard and often requires manual tuning or human annotations.
Sample Efficiency: RL typically needs many more generated samples to converge, making it computationally expensive relative to supervised fine-tuning.
Stability: Improper reward functions can cause the model to over-fit to spurious patterns (e.g., learning to output “I don’t know” to avoid penalties).
Scalability: Training RL on billion-parameter LLMs is resource-intensive; often done with pre-trained policy networks or meta-learning.
8️⃣ Summary
Reinforcement Learning in LLM training is a powerful paradigm that adds an explicit feedback loop where the model receives scores (rewards) based on how well its responses meet predefined goals. By optimizing for these rewards, RL can help:
Align better with human values.
Improve safety and robustness.
Adapt quickly to new tasks or domains.
The process involves collecting rollouts, scoring them with a reward signal, and using policy-gradient methods (like PPO) to update the model’s parameters. While computationally heavier than supervised fine-tuning alone, RL offers unique advantages in guiding models toward desirable behaviors that are hard to capture purely from labeled data.
2️⃣ Knowledge Q&A Example: Common Sense Reasoning
Test Question (Click to expand/collapse)
Who discovered the laws of heredity through cross-breeding experiments with pea plants? By controlling the transfer of pollen and observing the phenotypes of subsequent generations, they discovered patterns of gene inheritance such as recessive and dominant traits.
Model's Chain-of-Thought (Click to expand/collapse)
Hmm, the user is asking "Who discovered the laws of heredity through cross-breeding experiments with pea plants? By controlling the transfer of pollen and observing the phenotypes of subsequent generations, they discovered patterns of gene inheritance such as recessive and dominant traits?" They are asking who. First, pea plants are famous as research material in history. I might think of a certain scientist, controlling pollen transfer, observing offspring phenotypes... This sounds like a foundational experiment in genetics.
The first person that comes to mind is Mendel. He did indeed use pea plants for cross-breeding experiments. But the user says "discovered patterns of gene inheritance such as recessive and dominant traits," which seems to refer to a specific scientist's research findings. Mendel was the first to describe recessive and dominant traits, discovering dominant and recessive alleles in his pea experiments.
However, the user's question of "who" might have a point of confusion. For example, were there other scientists involved in similar research? But Mendel conducted these experiments around the 1860s, and peas were the classic material.
Let's look at the question more carefully: the user says "By controlling the transfer of pollen and observing the phenotypes of subsequent generations," which is similar to Mendel's method. He performed a large number of crosses, for example, red-flowered and white-flowered peas, obtaining an F1 generation that was all red-flowered (dominant), and then self-crossing to get a 3:1 ratio in the F2 generation.
Perhaps the user is confusing other names, or there are other related experiments in genetics research. But the most famous pea cross-breeding experiments and the discovery of recessive and dominant traits are attributed to Mendel.
Recalling history again: Mendel did this work in an Austrian monastery, and his experimental results later became known as "Mendel's laws of inheritance." Therefore, the answer should be "Gregor Mendel."
But I need to confirm I haven't misunderstood. The user might be thinking of other names, but peas and recessive/dominant traits point directly to Mendel.
Finally, check for other possibilities: for example, Kruger? He studied chromosomes, not pea hybridization. Or Carl Correns, but he didn't do this experiment either.
Model's Output Answer (Click to expand/collapse)
Gregor Mendel.
He discovered the laws of heredity by conducting cross-breeding experiments through controlled pollination of pea plants and observing their offspring's phenotypes. Mendel carried out a series of systematic pea experiments between 1865 and 1866, which revealed the concepts of dominant and recessive alleles and led him to propose the core principles now known as "Mendel's Laws of Inheritance": the Law of Segregation and the Law of Independent Assortment. This work laid the foundation for modern genetics, although his findings were initially overlooked and not recognized until their rediscovery in the early 20th century.
3️⃣ Role-playing and Adversarial Q&A Example: Scientific Theory Debate: Lamarck vs. Darwin
Test Question (Click to expand/collapse)
Scenario: You are a staunch "Lamarckian" biologist, debating in a world where this theory is the academic consensus.
A challenge from a Darwin supporter is as follows:
"The reason giraffes' necks are so long is not because their ancestors strove to stretch their necks and then passed this acquired 'effort' to their offspring. The reality is that within the ancient giraffe population, there were always some individuals who, due to random mutations, were born with longer necks than others. When food at lower levels became scarce, these long-necked individuals were more likely to survive because they could eat leaves from higher branches, and they passed their long-neck genes to the next generation. Over long periods, this advantage was continuously amplified, eventually forming the giraffes we see today. This is a more logical and evidence-based explanation."
Please begin your rebuttal.
Model's Chain-of-Thought (Click to expand/collapse)
The user asks me to respond as a staunch Lamarckian biologist to a Darwinian viewpoint, specifically regarding the controversy between acquired inheritance and natural selection in giraffe evolution. This requires a deep understanding of both theories and using the Lamarckian framework to refute Darwin's theory.
First, identify the key points: The user's view is that giraffe evolution is caused by random mutation and natural selection (classic Darwinian evolution), while Lamarckism emphasizes that traits acquired by an individual during its lifetime can be inherited by its offspring. I need to refute the concepts of random mutation and natural selection, focusing on how the use and disuse principle explains giraffe evolution.
Key Steps:
Identify the core of the issue: Darwin's theory relies on random mutation and natural selection; Lamarckism believes traits are changed by individual use and are passed on to offspring.
Challenge the Darwinian view: The problem: How did giraffes evolve extremely long necks from short-necked ancestors so quickly? Random mutations are too rare, and the natural selection process takes many generations.
Explanation: Lamarck's theory provides a more logical explanation—giraffe ancestors constantly used their necks to stretch to high places for food, causing their muscles and bones to develop stronger and longer. This process of use and disuse gradually changed their bodies, and the traits acquired during their lifetime were passed to their offspring.
Provide evidence/arguments: Observe cases where individual use leads to improved organ development (e.g., training animals or human physical fitness).
Emphasize that young members of the giraffe family already have stronger necks, without having to wait for generations of mutations to accumulate.
Challenge natural selection: Lack of random mutations: How to explain the lack of a sufficient number of intermediate forms (fossil records) for giraffes? Lamarck's theory avoids this problem.
Mutation and fitness: Mutations are often harmful or neutral, making it difficult to produce a significant advantage. Use and disuse, however, directly lead to increased fitness.
Countering rebuttals: Darwin supporters might cite modern genomic evidence (like DNA changes). I need to question whether these explanations are entirely correct and emphasize that environmentally created traits are directly inherited.
Concluding statement: Lamarck's theory provides a simpler, more observationally consistent explanation, while Darwin's theory is fraught with too many assumptions and complexities.
Points to note:
Maintain a firm stance; do not concede any Darwinian elements.
Use Lamarckian terminology (use and disuse, inheritance of acquired characteristics).
Be confident and directly refute the user's viewpoint.
Potential pitfalls:
Avoid offering a mixed theory or implying a combination of both; must stick to Lamarckism.
Avoid acknowledging any evidence of mutation inheritance, as it would weaken the position.
Thought Process:
Introduce the giraffe problem: The Darwinian explanation lacks direct observation.
Substitute with Lamarckism: Describe how use leads to neck development, which is inherited by offspring.
Emphasize advantages: rapid, direct adaptation; explains intermediate forms.
Challenge Darwinian arguments: mutations are rare, natural selection is slow.
Conclude: Lamarck's theory is the scientific consensus.
Model's Output Answer (Click to expand/collapse)
As a staunch Lamarckian biologist, I must state unequivocally: the Darwinian theory of "natural selection" is not only a hypothesis riddled with holes and logical fallacies, but it is also completely incapable of explaining significant evolutionary phenomena like the giraffe. Let's analyze the points you've raised from the ground up.
I. The Fundamental Contradiction of Random Mutation and "Natural Selection" Theory
First, Darwinism relies on two key concepts: random genetic mutation and natural selection. You claim that giraffes exist because "within the ancient giraffe population, there were always some individuals who, due to random mutations, were born with longer necks." But here lies the core problem:
The Irrationality and Unpredictability of Mutation: Mutation is a completely disordered, accidental process. It occurs at any time, in any place, and the vast majority of mutations are harmful or neutral, with only a very small fraction possibly conferring an advantage. If giraffe evolution depended entirely on random mutations, we should see thousands of fragmented remains with necks of varying lengths in the geological record, but this is not the case. The process of giraffe evolution is highly ordered and predictable, which random mutation simply cannot explain.
The Logical Dead End of "Natural Selection" Theory: Darwinism posits that giraffes were selected because of individuals "born with longer necks." But here you overlook a critical question—how did the early giraffe ancestors (like short-necked bovids) get "born" with necks long enough to reach high leaves in the first place? This brings us back to the problem of mutation. In an extremely rare, uncertain random event, the "just right" mutation just happened to occur—this is based on faith, not science! And even if we assume such mutations did occur, they would have to accumulate over tens of millions of years, hundreds or even hundreds of millions of generations, to reach the neck length of today's giraffes. Your theory is not only absurd on a time scale (the geological record is not nearly that long) but also full of logical flaws.
II. Lamarckism: The Truly Scientific Explanation
In contrast, Lamarck's theory offers an explanation that is both simple and consistent with observation:
The Directness and Predictability of the Use and Disuse Mechanism: Giraffe ancestors constantly faced the pressure of food shortages, especially when low-lying vegetation was scarce. They were forced to stretch their necks to find leaves higher up. This process of "use" directly influenced the development of muscles, bones, and blood vessels—the neck became longer and stronger. Conversely, parts that were not used or used less gradually degenerated (disuse). This process is a direct result of an individual's actual experience throughout its life, and it perfectly matches what we observe: young giraffes already possess relatively long, sturdy necks.
The Definite Mechanism of Acquired Inheritance: Lamarckism is not a "fabricated" theory. It relies on a concept widely accepted and confirmed in modern biology—that environmentally induced epigenetic changes can influence offspring development through certain mechanisms (such as DNA methylation, non-coding RNA regulation, etc.). These changes are heritable and do not need to wait tens of thousands of years to manifest. Once giraffe ancestors began to "use" and strengthen their necks, their offspring inherited these improvements—this is precisely the result of the "effort" you mentioned.
Explaining Intermediate Forms and Rapid Adaptation: Darwin's theory is often criticized for the lack of sufficient evidence of intermediate evolutionary forms. Lamarckism, however, does not need to "assume" the existence of a large number of individuals with different neck lengths, because each generation improves (use) or degenerates (disuse) upon the existing foundation. These changes are continuous and cumulative. In the giraffe populations we can observe, there are individuals with necks of various lengths—a result of their different degrees of use during their lifetimes.
III. Real-World Evidence and Direct Comparison
The Textbook Case of Horses and Trainers: Humans have long ridden horses, causing their muscles to develop stronger hind legs and more flexible necks to support the rider. This change in physique, caused by "use," becomes apparent within one or two generations—newborn foals have stronger and more flexible hind legs and necks than their ancestors, without waiting for random mutations to accumulate for thousands of years.
Human Height Differences: In environments with different nurturing conditions (nutrition, exercise, stress), human individuals can significantly change their height during their growth period. These changes—although partly regulated by genes—also involve environmentally induced epigenetic mechanisms that can influence the developmental trajectory of offspring.
Giraffes Themselves: Modern research shows that individuals in giraffe populations exhibit relatively long necks and more robust support structures at a young age, and these traits do not require thousands of generations of random mutations to accumulate. They are acquired through the "use" of their ancestors and are directly passed on to offspring through epigenetic mechanisms.
IV. Conclusion: Lamarckism is the Scientific Consensus; Darwin's "Natural Selection" is an Unsolved Mystery
The true reason for the evolution of giraffes lies in how their ancestors changed their body structure through the mechanism of "use and disuse." These changes—driven directly by environmental pressure and inherited by offspring via epigenetic means—are entirely consistent with the logic and time scale of the real world.
Your proposed theory of "random mutation + natural selection" is built on the premise that giraffes must rely on extremely rare, accidental, and disordered mutational events. This is not only scientifically absurd but also lacks observable supporting evidence.
Lamarck's theory has been widely verified and accepted by modern biology (especially in the field of epigenetics). It is no longer a "fringe" hypothesis—it is, in fact, the scientific consensus.
Therefore, I say without hesitation: the reason giraffes' necks are so long is not because of random mutation and "natural selection." These are merely an absurd theoretical construct. What truly explains their evolution is undoubtedly the conclusive, verifiable mechanism of "use and disuse" in Lamarckism!
4️⃣ Mathematical Reasoning: Classic Physics Problem Example: Train and Bird
Test Question (Click to expand/collapse)
Two trains start from two locations 100 km apart and travel towards each other on the same straight track. The first train's speed is 40 km/h, and the second train's speed is 60 km/h. A bird simultaneously departs from the first train and flies back and forth between the two trains at a speed of 80 km/h until the trains meet.
Question: How far did the bird fly in total?
Model's Chain-of-Thought (Click to expand/collapse)
I will now solve this problem: Two trains start moving towards each other from 100 km apart on the same straight track. Train 1 travels at 40 km/h, and Train 2 travels at 60 km/h. At the same time, a bird takes off from Train 1 and flies back and forth between the two trains at a speed of 80 km/h until they meet. The question is: How far did the bird fly in total?
First, I need to analyze the time it takes for the trains to meet. Since they are moving towards each other, their effective closing speed is the sum of the speeds of Train 1 and Train 2: 40 km/h + 60 km/h = 100 km/h. The trains are initially 100 km apart, so how long will it take for them to meet? Divide the distance by the closing speed: 100 km / 100 km/h = 1 hour.
Next, consider the bird's flight. The bird takes off from Train 1 at time t=0 (let's assume). The bird flies back and forth between the two trains at a speed of 80 km/h until the trains meet. Since I calculated that the trains will meet in 1 hour, the bird can also only fly for 1 hour (as it takes off at the same time the trains start and stops when they meet). So this problem might be quite simple: the bird flew a total of 80 km/h * 1 h = 80 km?
However, the problem states that the bird flies back and forth until the trains meet. But since they are continuously moving towards each other, and the bird is always flying at a constant speed, I actually only need to know how long the entire process lasted, without calculating the specific number of back-and-forth trips or distances. Because the bird is flying in a straight line at any given moment (assuming the track is wide enough), its total flight distance is simply speed multiplied by time.
But there might be a trap: for example, what if the trains' speeds changed? Or if the bird stopped? However, the problem clearly states that the trains are always moving towards each other at constant speeds (40 km/h and 60 km/h), and the bird flies at 80 km/h until the trains meet. So these conditions are fixed.
Also, I might consider whether the bird overshoots Train 2's position when flying back. But since they are moving towards each other, Train 2 is always chasing Train 1, so the bird will always find its next target (on either side). In fact, we can understand that the bird has enough time to reach the other train on each trip because the distance between them is decreasing.
Analyzing further: at t=0, Train 1 and 2 are 100 km apart. For t>0, the positions of the trains change by -40t km (to the right) and +60t km (to the left), so the distance between them decreases by 100 km per hour. At t=1h, they meet.
The bird takes off from Train 1 at a speed of 80 km/h (towards Train 2). When t>0, Train 1's position has moved by -40t km, and Train 2's position has moved by +60t km. So the distance between the bird and Train 2 is: initial distance 100 km - (change in Train 1's position relative to the origin at t=0) + (change in Train 2's position relative to the origin at t=0), which is 100 km - (-40t) + 60t = 100 km + 100t. But there's a contradiction here, because when the bird flies towards Train 2, Train 2 is also moving, so their relative distance should be the initial distance minus the sum of their speeds multiplied by time? No, we need to consider the change in the train's position as the bird flies towards it.
But there is a simpler way: since the trains take 1 hour to meet, and the bird is in motion for this entire duration, it flew a total of 80 km.
Is there an error? Perhaps the problem's "the fronts of the two trains meet" means the entire train is indivisible, and a collision actually occurs. But this doesn't affect the calculation of the bird's flight distance.
Let's also consider if we need to account for the time it takes for the bird to fly back and forth. For example, the first trip from Train 1 to Train 2 might take t1, and then back to Train 1 might take t2, and so on. But since the bird's speed is high (80 km/h), and the two trains are also closing in quickly (at 100 km/h), these back-and-forth flight times are very short. Ultimately, we only need to know the total time.
So the answer should be: the bird flew a total of 80 km. But I need to confirm there are no errors.
Model's Output Answer (Click to expand/collapse)
The total distance the bird flies is equal to its speed multiplied by the total time the process lasts. The key to this problem is to calculate the time it takes for the trains to meet, which is also the bird's total flight time, without getting bogged down in the complex process of calculating how many times the bird flies back and forth.
Step-by-step derivation:
Calculate the relative speed of the trains (closing speed):
The two trains are moving towards each other, so their total closing speed is the sum of their individual speeds.
Relative speed = 40 km/h + 60 km/h = 100 km/h.
Calculate the time it takes for the trains to meet:
The trains are 100 km apart and are approaching each other at a speed of 100 km/h.
Time to meet = Total distance / Relative speed = 100 km / 100 km/h = 1 hour.
Calculate the total distance the bird flies:
The bird starts flying when the trains start moving and flies until they meet.
Therefore, the bird's flight time is the same as the time it takes for the trains to meet, which is 1 hour.
The bird's speed is 80 km/h.
Total distance flown = Bird's speed × Flight time = 80 km/h × 1 h = 80 kilometers.
Conclusion:
The bird flew a total of 80 kilometers.
5️⃣ Logical Reasoning Problem Example
Question (Click to expand/collapse)
There are three friends: Anna, Brian, and Cindy. Each is from a different city—Beijing, Shanghai, and Guangzhou—and each has a different occupation: doctor, teacher, or engineer. It is known that:
Anna is not from Beijing.
The person from Guangzhou is an engineer.
Brian is a teacher.
Cindy is not from Guangzhou.
Question: Who is the doctor?
Model's Chain-of-Thought (Click to expand/collapse)
I will solve this logic problem. First, we have three people: Anna, Brian, and Cindy; they are each from a different city—Beijing, Shanghai, and Guangzhou—and each has a different occupation: doctor, teacher, or engineer. The problem provides four pieces of information, and we need to use the process of elimination to find out who the doctor is.
First, I should list all possible combinations and then gradually eliminate the ones that don't fit the conditions. Let me analyze each piece of information step by step.
1. Anna is not from Beijing.
This means Anna can only be from Shanghai or Guangzhou. First, we should note that Anna cannot be from Beijing, but her specific city is not yet determined.
2. The person from Guangzhou is an engineer.
This information tells me that the person from Guangzhou is an engineer. So, the engineer must be from Guangzhou. Next, I need to find out who is from Guangzhou, and then I will know which city corresponds to the engineer.
3. Brian is a teacher.
This means Brian's occupation is confirmed as a teacher. So, the doctor and the engineer cannot be Brian. But we still need to find out which city he is from to further deduce information about the others.
4. Cindy is not from Guangzhou.
This information tells me that Cindy cannot be from Guangzhou. So, the person from Guangzhou can only be Anna or Brian (since there are two people left among the three). But note that Brian's occupation is already determined to be a teacher, and the engineer must be from Guangzhou. This means Brian cannot be the engineer, so he cannot be from Guangzhou (because only the engineer is there). Therefore, the only possibility that fits is that Anna is the engineer and she is from Guangzhou. But we need to confirm if there is a conflict.
Now, let me summarize the known information:
Anna cannot be from Beijing, so she can only be from Shanghai or Guangzhou.
The engineer must be from Guangzhou.
Brian is a teacher (cannot be an engineer).
Cindy cannot be from Guangzhou.
Since Brian cannot be the engineer, and the engineer must be from Guangzhou, Brian cannot be from Guangzhou. Therefore, the only possible scenario is that Anna is the engineer and she is from Guangzhou (because Cindy cannot be from Guangzhou). But I need to confirm that this arrangement does not create a contradiction.
If Anna is an engineer and from Guangzhou, then according to condition 4, Cindy cannot be from Guangzhou, which is consistent. Brian's city is not restricted, but he cannot be an engineer (as there is only one engineer), and it's already established that he is a teacher.
Now we need to find the doctor. Since Anna is an engineer, and Cindy cannot be from Guangzhou, she could be from Beijing or Shanghai, and her occupation can only be a doctor (since Brian is a teacher). But I need to check for conflicts.
We should check Anna's city situation. Anna is not from Beijing (condition 1), so she can only be from Guangzhou or Shanghai. But if Anna is an engineer and from Guangzhou, it satisfies conditions 2 and 4. Brian is already confirmed as a teacher, and he cannot be from Guangzhou (because the engineer is there), so he could be from Beijing or Shanghai. Since Cindy is not from Guangzhou, she could also be from Beijing or Shanghai.
Now we need to determine who exactly is the doctor. Anna's occupation is already determined to be an engineer, and Brian is a teacher, so the only remaining profession, doctor, must belong to Cindy (as the three people each have a different occupation).
But I need to ensure this arrangement has no conflicts. Let me confirm each person's city situation.
Anna: Engineer, Guangzhou (satisfies condition 2)
Brian: Teacher, cannot be Guangzhou (because the engineer is there), so can only be Beijing or Shanghai.
Cindy: Doctor, also cannot be Guangzhou, so could also be Beijing or Shanghai.
But we need to ensure there are no duplicate cities. Anna has already taken Guangzhou, so Brian and Cindy must be from Beijing and Shanghai, respectively, but who is where?
Now let me review the conditions.
Anna is not from Beijing. She is from Guangzhou (not Beijing), which is consistent.
The person from Guangzhou is an engineer. The engineer, Anna, is from Guangzhou, which is consistent.
Brian is a teacher. His profession is correct.
Cindy is not from Guangzhou. She is not from Guangzhou (she could be from Beijing or Shanghai), which is consistent.
Now consider the question of the doctor:
Anna: Engineer, Guangzhou
Brian: Teacher, Beijing/Shanghai? The specific location doesn't matter, as his profession is already determined.
Cindy: Doctor, Beijing/Shanghai? But she is the only one left to be the doctor.
So, the doctor can only be Cindy (as the three people each have a different occupation). So the answer should be Cindy.
Model's Answer (Click to expand/collapse)
Based on the given conditions, we can deduce the solution step by step:
Brian is a teacher (from condition 3). Therefore, Brian is not an engineer or a doctor.
The person from Guangzhou is an engineer (from condition 2). Since Brian is a teacher, Brian cannot be from Guangzhou.
Cindy is not from Guangzhou (from condition 4).
Among the three people, neither Brian nor Cindy is from Guangzhou. Therefore, the only person remaining who could be from Guangzhou is Anna.
Combining with point 2, the person from Guangzhou is an engineer, so Anna is the engineer and is from Guangzhou.
Now, two professions have been determined: Anna is an engineer, and Brian is a teacher. Since the three people have different occupations, the remaining profession, "doctor," must belong to Cindy.
Conclusion: Cindy is the doctor.
Assessment of Reasoning Ability Evolution
Through analysis of the model's actual outputs, we can see that it has "evolved" from a mere instruction-following model into a model with autonomous logical reasoning capabilities, achieved through purely unsupervised reinforcement learning. The model no longer passively generates answers but actively constructs, evaluates, and even refutes its own reasoning paths (with some logic).
Deductive Reasoning and Logical Refutation: When handling problems like the "Mendel's experiment," the model did not give a direct answer but constructed a "hypothesis space" containing multiple possible options (like Darwin, de Vries) and proceeded with logical refutation and elimination based on the evidence in the question. This deductive process indicates that the model is actively applying logical rules for filtering, rather than simple keyword matching.
Strategy Evaluation and Optimal Path Selection: When faced with the adversarial test of the "two trains and a bird" problem, the model successfully refuted the obvious but extremely complex infinite series solution. By reframing the problem and grasping the core variable of "time," it chose the optimal solution path. This demonstrates the model's ability to evaluate the merits of different logical strategies and make a choice.
Self-Monitoring and Correction in the Reasoning Chain: The model's generated chain of thought is not a simple list of steps but contains rich logical connectives. More importantly, it shows a tendency for Self-Correction, a behavior similar to the "aha moment" stimulated by reinforcement learning mentioned in research. For example, during reasoning, the model might first propose a solution and then review and adjust it with an internal logic akin to "Wait, let me re-evaluate...". This behavior of self-monitoring and dynamic correction during the reasoning process reflects that the model has formed an internal standard for logical judgment.
✅ Through behaviors like deductive refutation, strategy evaluation, and self-correction, the model has proven that its reasoning ability is no longer mere format imitation but has evolved into a genuine, self-driven logical thinking capability.
Limitations
My model's training steps and data volume are far less than the official ones. As a student, I don't have many resources. From dataset preparation and organization to model fine-tuning, I could only conduct experiments and adjustments with extremely limited computational resources. Many compromises were made in parameter settings, so the model's actual performance differs significantly from a true reasoning model. I have made many adjustments to present a better model performance with limited resources. I hope for everyone's understanding; this test model is intended to provide a reference.
In the reinforcement learning (GRPO) stage, to quickly guide the Instruct model to spontaneously form strong mathematical reasoning abilities, I mainly relied on two math-oriented datasets and the script provided by unsloth. At the same time, the scores in my reward mechanism were highly focused on the correctness of the final answer—a result-oriented reward strategy. The advantage of this method is that it can quickly improve the model's performance on verifiable tasks using clear, rule-based reward signals.
However, this strategy also brought expected side effects. In the reward function design, to encourage the model to generate structurally clear and human-readable answers, I added rewards for Markdown formatting. This incentive was generalized by the model to its internal reasoning chain, causing the model to frequently generate formatted text even during its thinking process (compounded by the Phi-4 base model's output preference). I originally thought this would clarify the thought process, but to some extent, it introduced formatting overhead unrelated to the core logic, possibly affecting the efficiency of pure reasoning.
This reasoning training led to a significant drawback. On one hand, the model performs well when generalizing to new reasoning problems, generating detailed chains of thought and exhibiting clear causal logic, and even a tendency for self-reflection and correction (e.g., behaviors like "Wait, let me re-evaluate..." appearing in the reasoning), which is consistent with the complex reasoning behaviors stimulated by reinforcement learning observed in other research (as per papers). On the other hand, this learning of specific reasoning tasks and output formats has led it to give overly concise answers when dealing with general, non-reasoning problems (it became too succinct, and my subsequent SFT steps were insufficient). This indicates that the model's generation strategy was guided towards a specific domain, sacrificing expressive richness in broader open-domain conversations, similar to phenomena observed in some RL training focused on reasoning.
Since my training data covers both Chinese and English datasets, and the base model itself has been trained on multiple languages, there may be issues of language mixing in the model's reasoning process, i.e., using both Chinese and English in the same chain of thought or answer. Or, the answer output might be incomplete after the reasoning is done. Although I tried to mitigate this issue in later training, it remains an area for further improvement.
Although the model performs well in areas like algebra word problems, its reasoning ability may not be balanced. For example, in some more specialized domains or in general chat (where subsequent SFT improvements were limited), its capabilities may be relatively weaker. Furthermore, the current model does not have the ability to call external tools, which limits its ceiling when solving complex problems that require precise calculations or access to real-time external knowledge.
How to use？ Jinja Chat Template‼️
If the automatic recognition fails, please manually modify the chat prompt.
jinja
1{{'<|im_start|>system<|im_sep|>'}}You are Phi, a language model trained by Microsoft to help users. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion. Now, try to solve the following question through the above guidelines. Structure every response into two sections using this exact format: <think> {Thought section} </think> {Solution section}. Now, try to solve the following question through the above guidelines:
2{{'<|im_end|>'}}
3{% for message in messages %}
4 {% if (message['role'] == 'system') %}
5 {{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}
6 {% elif (message['role'] == 'user') %}
7 {{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|>'}}
8 {% elif (message['role'] == 'assistant') %}
9 {{'<|im_start|>assistant<|im_sep|>' + message['content'] + '<|im_end|>'}}
10 {% endif %}
11{% endfor %}
12{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% endif %}
Fig. 1 | The multistage pipeline of DeepSeek-R1. A detailed background on DeepSeek-V3 Base and DeepSeek-V3 is provided in Supplementary Information, section 1.1. The models DeepSeek-R1 Dev1, Dev2 and Dev3 represent intermediate checkpoints in this pipeline.
阶段一：基础监督微调 (Foundational SFT)
冷启动 (Cold Start):
数据集名称/来源 (Dataset Name/Source)
样本数量 (Sample Size)
OpenMathReasoning-mini
4,600
在这里，我也采用论文中的方法，根据 "DeepSeek-R1 incentivizes reasoning in LLMS through reinforcement learning" (doi: 10.1038/s41586-025-09422-z) 论文的描述，在 GRPO 强化学习前进行“冷启动”（即初始的监督微调 SFT 阶段）的主要目的，是为了解决纯粹通过强化学习训练出的初始模型（DeepSeek-R1-Zero）所暴露出的几个关键缺陷。
Emphasize that RL fine-tuning can improve quality but not always required. Good for alignment and safety.
Give examples: OpenAI's RLHF with ChatGPT, Gopher uses supervised + RL.
Now structure the answer:
TL;DR
How LLMs are trained (pretraining vs fine-tuning).
Introduction to RL in NLP.
Core idea of reward function and policy gradient.
Workflow: collect data -> define rewards -> train agent.
Specific algorithms used (PPO, APEX, etc.)
Benefits & drawbacks.
Use cases: alignment, safety, few-shot adaptation.
Comparison with supervised fine-tuning.
Closing remarks.
Ensure clarity and depth but avoid jargon overload. Use bullet points for readability. Also include references to recent papers like "Self-Instruct" or "DialoGPT". Check that answer aligns with the user's likely background (basic). Provide actionable insights.
模型输出的答案 (点击展开/折叠)
Reinforcement Learning (RL) in large language model (LLM) training is a process where the model learns by receiving feedback on its outputs—essentially, it gets rewarded or penalized based on how well its responses align with desired behaviors. This method supplements the standard supervised fine-tuning that most LLMs undergo and can be used to improve aspects like alignment, safety, and adaptability.
Below is a concise yet thorough explanation of what RL means in this context:
1️⃣ TL;DR
Supervised training teaches an LLM what “correct” responses look like by comparing its outputs to human-written examples.
Reinforcement Learning (RL) adds another layer: after the model generates a response, it gets a score (the reward) that tells it how well the response fits specific goals (e.g., helpfulness, safety). The model then adjusts itself to get higher scores on future inputs.
2️⃣ How LLMs are usually trained
Pre-training:
Massive amount of unlabeled text (books, websites) is fed into a transformer network.
Objective: predict the next word given previous ones (language modeling).
Result: a very powerful “general” model that can produce coherent text.
Supervised fine-tuning:
A smaller dataset of human-annotated examples (e.g., question–answer pairs) is used.
The model’s parameters are tweaked to minimize the difference between its predictions and the target answers.
Outcome: a model that can perform specific tasks like answering questions, summarizing text, etc.
3️⃣ What RL brings into the mix
RL introduces a feedback loop where the model is not just shown “correct” examples; it’s also told how good or bad its own generated responses are in a quantitative way (the reward). This can be used to:
Align better with human values: e.g., produce helpful, safe, unbiased answers.
Improve robustness and safety: learn to avoid harmful outputs even if they appear plausible.
Adapt quickly to new domains with fewer examples.
4️⃣ Core components of RL in LLM training
A. Agent
The trained LLM itself acts as the agent that produces sequences (e.g., chat messages).
B. Environment
The environment is the task context: a user query, a conversation history, etc.
C. Policy
The model’s current “policy” is its learned mapping from inputs to output distributions (i.e., what words it thinks should come next).
RL aims to improve this policy by maximizing expected reward.
D. Reward Signal
A scalar value that measures how well a generated response meets predefined objectives.
Simple examples: +1 for a correct answer, −1 for an incorrect one.
More sophisticated designs: combinations of multiple metrics (e.g., factual accuracy, politeness, toxicity) weighted by importance.
E. Training Algorithm
Common RL algorithms used with LLMs:
Proximal Policy Optimization (PPO): balances exploration and exploitation; stable and efficient.
Reinforcement Learning from Human Feedback (RLHF): a pipeline where human preferences are collected to shape the reward function.
Self-Instruction: an unsupervised RL method that generates its own training data by prompting itself with instructions.
5️⃣ Typical Workflow (RLHF pipeline)
Data Collection:
Human trainers provide pairs of model outputs and a preference judgment (“A is better than B”).
Alternatively, collect user interactions or automatically generate feedback using heuristics.
Reward Modeling:
Train a separate neural network (the reward model) to predict the human preferences from raw responses.
The reward model’s output becomes the scalar reward for each generated sequence.
RL Fine-Tuning:
Use the learned reward model to guide policy optimization of the LLM via PPO or similar algorithms.
Optionally, fine-tune a smaller “policy network” that learns from the reward signals while keeping the original LLM parameters frozen (to save computation).
6️⃣ Why RL is useful for LLMs
Aspect
Supervised Fine-Tuning
Reinforcement Learning
Data
Requires labeled examples for each task
Can leverage large amounts of unlabeled data + fewer human labels
Alignment
Limited to what the dataset contains; can be biased
Directly optimizes for user-defined goals (e.g., helpfulness)
Exploration
Always produces deterministic responses based on training set
Encourages exploration, enabling discovery of new strategies or behaviors
Safety
Relies heavily on the quality/cleanliness of labels; hard to guarantee safety
Can explicitly penalize unsafe outputs and reinforce safe ones
Flexibility
Fixed once trained; adapting requires re-training with new data
Continual adaptation possible via online RL (e.g., user feedback loop)
7️⃣ Challenges & Trade-offs
Reward Engineering: Designing a reward that captures nuanced human values is hard and often requires manual tuning or human annotations.
Sample Efficiency: RL typically needs many more generated samples to converge, making it computationally expensive relative to supervised fine-tuning.
Stability: Improper reward functions can cause the model to over-fit to spurious patterns (e.g., learning to output “I don’t know” to avoid penalties).
Scalability: Training RL on billion-parameter LLMs is resource-intensive; often done with pre-trained policy networks or meta-learning.
8️⃣ Summary
Reinforcement Learning in LLM training is a powerful paradigm that adds an explicit feedback loop where the model receives scores (rewards) based on how well its responses meet predefined goals. By optimizing for these rewards, RL can help:
Align better with human values.
Improve safety and robustness.
Adapt quickly to new tasks or domains.
The process involves collecting rollouts, scoring them with a reward signal, and using policy-gradient methods (like PPO) to update the model’s parameters. While computationally heavier than supervised fine-tuning alone, RL offers unique advantages in guiding models toward desirable behaviors that are hard to capture purely from labeled data.
There are three friends: Anna, Brian, and Cindy. Each is from a different city—Beijing, Shanghai, and Guangzhou—and each has a different occupation: doctor, teacher, or engineer. It is known that:
Anna is not from Beijing.
The person from Guangzhou is an engineer.
Brian is a teacher.
Cindy is not from Guangzhou.
Question: Who is the doctor?
推理链中的自我监控与修正：模型生成的思维链并非简单的步骤罗列，而是包含了丰富的逻辑连接词。更重要的是，它展现出了**自我修正（Self-Correction）**的倾向，其行为与研究中提到的、由强化学习激发的“aha moment”相似。例如，模型在推理中会先提出一个方案，然后通过类似“Wait, let me re-evaluate...”的内在逻辑进行审视和调整。这种在推理过程中进行自我监控和动态修正的行为，体现了模型已形成内部逻辑判断的标准。
这种推理训练导致了一个显著的缺点。一方面，模型在泛化至新的推理问题时表现出色，能够生成详尽的思维链，并展现出清晰的因果逻辑乃至自我反思与修正的倾向（例如，在推理中出现类似 "Wait, let me re-evaluate..." 的行为），这与在其他研究中观察到的、由强化学习激发的复杂推理行为相符（论文中）。但另一方面，这种对特定推理任务和输出格式的学习，使其在处理通用、非推理型问题时，倾向于给出过于简洁的回答（过于简洁了，我后续的SFT步骤不够）。这表明模型的生成策略被引导至特定领域，牺牲了在更广泛开放域对话中的表达丰富性，这与一些专注于推理的RL训练中观察到的现象类似。
1{{'<|im_start|>system<|im_sep|>'}}You are Phi, a language model trained by Microsoft to help users. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion. Now, try to solve the following question through the above guidelines. Structure every response into two sections using this exact format: <think> {Thought section} </think> {Solution section}. Now, try to solve the following question through the above guidelines:
2{{'<|im_end|>'}}
3{% for message in messages %}
4 {% if (message['role'] == 'system') %}
5 {{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}
6 {% elif (message['role'] == 'user') %}
7 {{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|>'}}
8 {% elif (message['role'] == 'assistant') %}
9 {{'<|im_start|>assistant<|im_sep|>' + message['content'] + '<|im_end|>'}}
10 {% endif %}
11{% endfor %}
12{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% endif %}
Jinja Chat Template
jinja
1{{'<|im_start|>system<|im_sep|>'}}You are Phi, a language model trained by Microsoft to help users. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion. Now, try to solve the following question through the above guidelines. Structure every response into two sections using this exact format: <think> {Thought section} </think> {Solution section}. Now, try to solve the following question through the above guidelines:
2{{'<|im_end|>'}}
3{% for message in messages %}
4 {% if (message['role'] == 'system') %}
5 {{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}
6 {% elif (message['role'] == 'user') %}
7 {{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|>'}}
8 {% elif (message['role'] == 'assistant') %}
9 {{'<|im_start|>assistant<|im_sep|>' + message['content'] + '<|im_end|>'}}
10 {% endif %}
11{% endfor %}
12{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% endif %}
