Model Card for GRPO Enhanced SmolLM-135M-Instruct
This model extends the capabilities of the SmolLM-135M-Instruct model with specific enhancements for task-oriented language generation, emphasizing concise and formatted text generation using generative reinforcement learning.
Model Details
Model Description
This model, developed using the Transformers library, is fine-tuned from the HuggingFaceTB/SmolLM-135M-Instruct base model on the mlabonne/smoltldr dataset. It incorporates enhancements via a reward-based training system to improve the specificity and formatting of generated text, ideal for applications requiring structured output.
- Model type: GRPOTrainer with TRL
- Language(s) (NLP): English
- License: MIT
- Finetuned from model:
HuggingFaceTB/SmolLM-135M-Instruct
Model Sources
Uses
Direct Use
This model is ready to generate structured text outputs directly for applications such as automated reasoning or instructional content generation without the need for additional fine-tuning.
Bias, Risks, and Limitations
The model may inherently carry biases from the training data and the base model it was fine-tuned from. It might not handle out-of-distribution inputs well or generate less accurate outputs in languages other than English.
Recommendations
Continual monitoring and updating of the model with diverse datasets can help mitigate biases. Users should also validate model outputs before use in critical applications.
How to Get Started with the Model
To start using this model, integrate it within your Transformers-based pipeline, specifying its model ID on the HuggingFace Hub. Ensure your environment supports bf16 for optimal performance.
Training Details
Training Data
The model was trained on the mlabonne/smoltldr dataset, which consists of structured prompts and completions designed for text generation tasks.
Training Procedure
Preprocessing
Data preprocessing involved tokenization using the tokenizer from the base model, with text split into prompts and completions.
Training Hyperparameters
- Training regime: bf16 mixed precision
- Total training time: Approximately 494 seconds
- Steps per second: 0.101
- Batch size: 1 (with gradient accumulation over 2 steps)
- Training loss: 5.720701068639755e-05
Evaluation
Testing Data, Factors & Metrics
Evaluated on a held-out portion of the mlabonne/smoltldr dataset.
Results
The final model achieved a very low average training loss of 5.720701068639755e-05, suggesting highly effective learning under the configured reward functions.
Environmental Impact
Training was conducted on GPU hardware compatible with bf16 precision, optimizing compute efficiency and reducing energy consumption.
Technical Specifications
Model Architecture and Objective
The model architecture is based on a causal language model with enhancements for token-level rewards to improve response quality in task-specific generations.
Compute Infrastructure
Training was performed on cloud-based GPUs with bf16 support to ensure fast computation and reduced environmental impact.
More Information
For more detailed documentation and usage examples, visit the repository link provided above.
