Lily-1.1-10M
Model Description
Lily-1.1-10M is a continued pretraining of the SENAGI/Lily-1.0-10M base language model.
Unlike instruction tuning or supervised fine-tuning, this model was trained by continuing the self-supervised language modeling objective on additional text data. The architecture remains unchanged, while the model's weights have been updated through further pretraining to improve language understanding and generation.
- Base model: SENAGI/Lily-1.0-10M
- Model type: Causal Language Model (CLM)
- Training method: Continued Pretraining
- Parameters: ~10M
Training Objective
The model was trained using the standard causal language modeling (next-token prediction) objective. This is not an instruction-tuned or supervised fine-tuned model.
Intended Uses
Lily-1.1-10M is intended for:
- Language modeling research
- Text generation
- Continued pretraining experiments
- Educational purposes
- Small-scale NLP projects
Limitations
- Not instruction-tuned.
- May not reliably follow chat or instruction prompts.
- Can generate incorrect or biased information.
- Performance is limited by its small parameter count (~10M).
Training Procedure
Training Type
Continued Pretraining (Domain-Adaptive / Continual Pretraining)
Training Hyperparameters
- Learning rate:
5e-4
- Train batch size:
8
- Gradient accumulation steps:
4
- Effective batch size:
32
- Optimizer:
AdamW (Torch Fused)
- LR scheduler:
Linear
- Warmup steps:
50
- Training steps:
1525
- Mixed precision:
Native AMP
- Seed:
42
Framework Versions
- Transformers 5.13.1
- PyTorch 2.11.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2
Citation
If you use this model in your work, please cite both this repository and the original Lily-1.0 model.
