Soren-Logos-3B is a deeply evolved version built upon the Soren-Oracle-Chat-3B model, created to address specific challenges. The core motivation of this project is to improve the relatively weak mathematical and logical capabilities of the base Llama-3.2-3B model.
To achieve this, I introduced Group Relative Policy Optimization (GRPO), an advanced reinforcement learning technique. Building on the capabilities of the Oracle model, a series of carefully designed reward functions were used to guide the model in generating a transparent "engine of thought." The core of this engine is to teach the model to think step-by-step logically and to externalize its complete thought process using a specific format—the <think>...</think> tags—before arriving at a conclusion. (The reward for this was kept small to prevent it from becoming a model that always explicitly shows its chain of thought). The goal is to create a 3B-level model that not only answers but also presents its reasoning clearly and methodically.
Core Upgrade: GRPO and the Engine of Thought
✨ The core upgrade of Soren-Logos focuses on improving the base model's capabilities and correctness (Process) on complex problems, as well as enhancing the quality of its responses.
It's no longer just about getting the right answer but pursuing a verifiable and traceable source for that answer through GRPO reinforcement learning.
Guiding Logical Chain Generation: Utilizes reward functions to incentivize the model to first generate a step-by-step, coherent chain of logical reasoning when faced with a problem, rather than directly outputting the answer. (This reward is intentionally small, so the model doesn't behave like a pure reasoning model).
Externalizing the Thought Process: We specifically guided the model to learn to use the <think>...</think> block to enclose its complete thinking and reasoning steps. This makes the model's "mind" no longer a black box, allowing users to clearly see how it analyzes a problem and reaches a conclusion.
Multi-dimensional Reward Function Optimization
In addition to the core logical reasoning training, this GRPO optimization includes a series of reward functions targeting response quality, achieving multi-dimensional alignment and improvement:
Dialogue Alignment: Rewards more natural and human-preferred conversational styles, making communication smoother.
Structured Output Tasks: For tasks like code generation and text formatting, it rewards outputs that strictly adhere to format requirements, enhancing the model's practicality in these scenarios.
Penalizing Redundancy and Repetition: Penalizes verbose, repetitive, and meaningless answers, guiding the model to produce more concise and information-dense content.
Training Method
🔥The training of Soren-Logos-3B is divided into two core phases:
Phase 1: Building "Oracle" Foundational Capabilities
The model first inherits all the training achievements of Soren-Oracle-Chat-3B, completing a foundational fine-tuning on a high-quality mixed dataset of 86,448 samples to establish improvements in response depth, professionalism, and formatting.
Building on the first phase, the model entered a targeted reinforcement learning stage:
Core Technology: Adopted Group Relative Policy Optimization (GRPO), an efficient RLHF method.
Core Dataset: Used the openai/gsm8k dataset, focusing on training the model's step-by-step logical reasoning abilities through its math word problems.
Composite Reward Functions: Designed and implemented multi-dimensional reward functions, including:
Logical Format Reward: Strongly guides the model to generate the <think>...</think> structure.
Answer Accuracy Reward: Scores the correctness of the final answer.
Dialogue Alignment Reward: Encourages the generation of more fluent, human-preferred responses.
Conciseness Penalty: Reduces the weight of redundant and repetitive content.
Below is a comparison of responses from the fine-tuned model and the base model on several questions, intuitively demonstrating the model's improvements in formatting, logic, and professionalism.
Model Capability Comparison
Question
Soren-Logos-3B (After Fine-tuning)
Official Llama-3.2-3B-Instruct
1. Can machines think?
Answer: Machines do not possess the ability to "think" or have "thoughts."
🧠 Why can't machines "think"?: - 1️⃣ Data-Driven: Models rely on training data, not subjective consciousness. - 2️⃣ Lack of Self-Awareness: Cannot generate subjective experiences or self-awareness.
Commentary: Uses paragraphs, lists, and emojis for a clear structure and greater depth.
Answer: Currently, machines do not possess true thinking ability. However, Machine Learning and Deep Learning can already mimic human thinking in some aspects, making predictions and decisions, but they lack critical thinking and creativity.
Commentary: The answer is relatively plain and lacks a hierarchical structure.
2. Age Calculation Problem
Setup: Let's say Xiaoming is x years old, and his father is 4x years old. In 5 years: Xiaoming will be x + 5, and his father will be 4x + 5. Set up the equation: 4x + 5 = 3(x + 5) → Solving for x gives x = 10.
Answer: ✅ Xiaoming is currently 10 years old, and his father is 40 years old.
Commentary: The reasoning steps are clear, and the result is correct.
Explanation: The original answer attempted to set up an equation but incorrectly incorporated the "in 5 years" condition into the initial equation, leading to a final calculation error. For example, it produced incorrect forms like d = 4m + 5.
Commentary: Logical error, incorrect answer.
3. The history of Apple Inc.
Answer: 🍎 Apple Inc. is a globally renowned innovation company. Early Years (1976–1980): Founding and First Products: Introduction to its establishment and initial products; Entry into Education: Early strategies for the education market. Notable Milestones: Lists major time points and products.
Commentary: Uses subheadings and bullet points for a clearer and more organized structure.
Answer: Apple Inc. has a rich history. Early Years (1976–1980): Introduction to the company's founding and first products; The Macintosh Era (1980–1996): Basic entries like the Macintosh release in 1984.
Commentary: Presented as a simple list with more generalized information.
4. Explain what a "neural network" is.
Answer: Introduction to Neural Networks and Core Concepts. Definition and Historical Background: Introduces the origin and development. Core Components: Neurons and Network Structure: Neurons, Layers, Connection Weights, etc. Working Principle and Architecture Types: Feedforward Networks, Convolutional Neural Networks, Recurrent Neural Networks, etc., along with application scenarios.
Commentary: A comprehensive and in-depth answer covering history, components, principles, architecture, and applications with a high degree of structure.
Answer: A neural network is a machine learning model that mimics the brain's neural system. Basic Components: Composed of multiple "neurons" and a hierarchical structure. Working Principle: Data input, weights and activation functions, etc.
Commentary: The explanation is relatively basic and general.
Actual Output Showcase:
Intended Use and Limitations
Intended Use
This model is designed for various tasks that require clear, reliable, and transparent reasoning processes, with improvements in several capabilities:
General-purpose conversation in Chinese and English, instruction following, and logical reasoning.
Educational, learning, and tutoring scenarios that require showing problem-solving steps.
Debugging and verifying the reliability of AI outputs.
Content creation tasks that require high-quality structured output.
Limitations
Potential Inaccuracies: Like all language models, it may produce inaccurate or biased content. Always verify critical information.
Knowledge Cutoff: The model's knowledge base ends in December 2023.
Reasoning Capability Limits: Although its reasoning ability has been significantly enhanced, as a 3B-level model, it may still make mistakes when handling extremely complex reasoning problems.
Acknowledgments
The creation of Soren-Logos-3B would not have been possible without the many outstanding prior works and the collective wisdom of the open-source community. We express our sincerest gratitude to all contributors who provided the foundation, tools, and inspiration for this project.
Core Contributors
Base LLM: We used meta/Llama-3.2-3B-Instruct, developed by Meta, as the starting point for our model. Its excellent architecture and powerful foundational capabilities were key to the project's success.
Fine-tuning Framework: We used the Unsloth library for efficient training and optimization.
I encourage users of this model to also acknowledge and cite the original contributors of the base models and datasets mentioned above. The open-source community thrives on sharing, and we hope Soren-Logos-3B can also be a part of this force.
⚠️ Disclaimer
Please note that the core objective of this fine-tuning was to optimize and enhance the model's capabilities in specific areas, but it was not a complete retraining from scratch.
The final performance, knowledge boundary, and capability limits of this model are strictly constrained by the inherent framework of its base model. Fine-tuning can improve performance in certain aspects but cannot overcome the fundamental limitations of the base model itself.
Therefore, the improvements brought by fine-tuning are incremental and do not represent a qualitative leap. I advise users to independently cross-verify all critical information and to evaluate the model's output with caution.
