Thought-Ranked Llama 3.2 3B

Model Description

Training Process

1. Initial Generation: For each training sample, the model generates multiple thought chains by prefixing different thought tokens: <thought>{char}</thought> for each character in [a-zA-Z0-9]. Each thought chain is allowed up to 128 tokens.
2. Answer Generation: Following each thought chain, the model generates a complete answer with up to 2048 tokens.
3. Ranking & Selection: An external LLM ranking system evaluates the quality of answers without seeing the thought processes, creating a ranking of the most effective thought patterns.
4. Final Training: The model is then trained on the highest-ranked thought-answer pairs, learning to generate the most effective thought patterns autonomously.

Key Features

• Thought Chain Generation: The model has learned to generate explicit thought processes before providing answers
• Greedy Sampling: Uses greedy sampling for both thought generation and final answers
• Length Parameters:
  • Thought chains: Up to 128 tokens
  • Final answers: Up to 2048 tokens

Model Architecture

• Base model: Llama 3.2 3B (Base)
• Architecture: Transformer-based language model
• Parameters: ~3.2 billion
• Training Strategy: Supervised Fine-Tuning (SFT) with thought-chain ranking

Intended Use

• Problem-solving
• Mathematical reasoning
• Logical deduction
• Step-by-step explanations
• Complex decision making

Out-of-Scope Uses

• Direct deployment without safety measures
• Applications requiring guaranteed accuracy
• Critical decision-making without human oversight
• Tasks requiring capabilities beyond the base Llama 3.2 3B model

Training Details

Training Data

• Sample questions paired with multiple thought variations
• Thought chains generated using systematic character prefixes
• Rankings derived from LLM evaluation of answer quality

Training Procedure

1. Thought Generation Phase
   • Generated 62 variations of thoughts per sample (a-z, A-Z, 0-9)
   • Sampled with temperature=0.0
   • Maximum thought length: 128 tokens
2. Answer Generation Phase
   • Generated completions following each thought chain
   • Maximum answer length: 2048 tokens
   • Sampled with temperature=0.0
3. Ranking Phase
   • External LLM evaluated answer quality
   • Ranking performed without access to thought chains
   • Selected highest-performing thought-answer pairs
4. Final Training Phase
   • Fine-tuned on best-performing thought-answer combinations
   • 4 complete rounds of training

Usage

Limitations

• Limited to the capabilities of the base Llama 3.2 3B model
• May generate thought chains that are not always optimal
• Performance depends on the quality of the LLM ranking system used during training
• Training process may not capture all possible effective thought patterns
• Limited by the context window of the base model

Ethical Considerations

• The model inherits biases from the base Llama 3.2 3B model
• Generated thought chains should be reviewed for accuracy and appropriateness
• The model's reasoning process should not be relied upon for critical decisions without human verification
• Users should implement appropriate content filtering and safety measures

Citation