LLM Course HW1 Transformer
Model Description
This model is a Transformer-based causal language model developed as part of an educational assignment. It employs a custom Byte-Level BPE tokenizer and incorporates modern techniques such as:
- ALiBi for positional embeddings
- Grouped-Query Attention (GQA)
- SwiGLU in the feed-forward layers
Additional optional components include Rotary Positional Embedding and Multi-Head Latent Attention.
Model Architectures
Multiple configurations are available:
- Nano: 3 layers, 4 heads, 96 hidden dimensions
- Mini: 6 layers, 6 heads, 384 hidden dimensions
- Small: 12 layers, 12 heads, 768 hidden dimensions
Training Details
- Dataset: Trained on IgorVolochay/russian_jokes
- Sequence Length: 128 tokens per sequence
- Tokenizer: Custom Byte-Level BPE tokenizer with a vocabulary size of 1024
- Objective: Causal language modeling with cross-entropy loss
Intended Use
This model is intended for educational and experimental purposes in the realm of language modeling. It is suitable for generating informal and creative text (e.g., jokes) and can serve as a baseline for research on efficient Transformer architectures.
Limitations
- Trained on a niche dataset (Russian jokes), which may limit its generalizability.
- Its small-scale architecture might not perform well on large or diverse text corpora.
How to Use
- Tokenization: Utilize the provided Byte-Level BPE tokenizer for encoding and decoding text.
- Generation: Leverage the built-in generation function with adjustable parameters (max tokens, temperature, top-k sampling).
- Integration: Load the model using Hugging Face’s
from_pretrained method.
