Tiny Knight-1.1b-v0.1 is a specialized language model crafted for generating knight and medieval-themed content. This iteration is built upon the foundations of TinyLlama-1.1B-Chat-v1.0, tailored to operate within environments constrained by computing resources.
Performance
While this model excels in creating knight-themed narratives, its specialization, however, limits its effectiveness in broader language tasks, especially those requiring detailed knowledge outside the medieval theme.
Direct Use
Tiny Knight-1.1b-v0.1 is particularly suited for generating content within medieval, knightly, or fantasy settings, ideal for storytelling, educational content, and thematic exploration. It is not recommended for general-purpose tasks or technical domains.
Context Setting and Interaction Guidelines
Given its specialized nature, Tiny Knight-1.1b-v0.1 benefits significantly from detailed context-setting. Providing a rich thematic backdrop in prompts enhances the model's performance, guiding it to generate more accurate and immersive content.
Training Data
Incorporates a dataset focused on knightly tales, medieval history, and literature, derived from the foundational TinyLlama-1.1B model.
Custom Stopping Strings
Custom stopping strings were used to refine output quality:
"},"
"User:"
"You:"
"\nUser"
"\nUser:"
"me:"
"user"
"\n"
Training Hyperparameters and Fine-Tuning Details
Base Model Name: TinyLlama-1.1B-Chat-v1.0
Base Model Class: LlamaForCausalLM
Projections: gate, down, up, q, k, v, o
LoRA Rank: 16
LoRA Alpha: 32
True Batch Size: 32
Gradient Accumulation Steps: 1
Epochs: 0.18
Learning Rate: 3e-4
LR Scheduler: Linear
Step: 75
Loss: 1.87
Limitations
While adept at producing themed content, Tiny Knight-1.1b-v0.1's applicability is limited outside its specialized domain of knights and medieval themes.
Summary
Tiny Knight-1.1b-v0.1 represents a significant advancement in thematic language models, offering a specialized tool for exploring the medieval era. Its emphasis on context for optimal performance and the use of custom stopping strings make it a sophisticated asset for generating historically rich content.
Acknowledgments
Special thanks to the TinyLlama-1.1B team, whose pioneering work laid the groundwork for the creation of Tiny Knight-1.1b-v0.1.
