Model Card: DoRA Tuned Lite Oute 1 300M Instruct
Model Overview
• Base Model: OuteAI/Lite-Oute-1-300M-Instruct
• Fine Tuning Strategy: DoRA (Weight Decomposed Low Rank Adaptation)
• Architecture Modifications:
o Extends the LoRA approach by incorporating a learnable diagonal scaling vector to modulate the normalized weight updates.
o The model uses a custom LinearWithDoRA module which integrates both the low rank update (via the previously defined LoRA layer) and a diagonal multiplier (m parameter) that adjusts the magnitude of the combined weights.
o Target layers remain those within the key and value projection modules (k_proj and v_proj).
Intended Use
• Primary Application: Sentiment classification of tweets with an emphasis on improved output stability by incorporating the DoRA mechanism.
• Task Suitability: Ideally suited for scenarios where a slight increase in model capacity—by learning additional scaling factors—improves classification accuracy without incurring full model fine tuning costs.
Training Data & Process
• Dataset: cardiffnlp/tweet_eval (sentiment subset)
• Task: Tweet sentiment classification with labels: negative, neutral, and positive.
• Training Modifications:
o The base model’s parameters are frozen except for the learnable LoRA/DoRA components.
o A custom training loop (using PyTorch DataLoader and gradient accumulation) is employed alongside gradient clipping for stability.
• Hyperparameters:
o Fine tuning is conducted over a minimal number of epochs with specific learning rates and accumulation settings optimized for the DoRA method.
Evaluation & Performance
• Metrics: The model is benchmarked using Macro F1 score on the test split.
• Observations: After applying DoRA, a noticeable improvement is observed in sentiment classification performance, meeting a target threshold (e.g., F1 > 0.5) in initial experiments.
• Usage Recommendation: This model is recommended for use in production environments where performance gain justifies a slightly more complex adaptation mechanism than standard LoRA.
Limitations & Caveats
• The additional scaling in DoRA introduces extra hyperparameters that require validation tuning for different datasets or tasks.
• Domain specificity remains: While optimized for tweet sentiment classification, performance on other text types may vary.
• Users should ensure careful monitoring of model behavior during inference.
