tags: [text-generation, causal-lm, rotary-positional-embedding, gradio-app]
language: en
license: apache-2.0
metrics: [perplexity]
Model Card — fast_stupidity
One-line summary:
A 34-layer causal Transformer language model with rotary embeddings, trained from scratch with a BERT tokenizer and served via Gradio for interactive text generation.
Model details
- Architecture: Decoder-only Transformer (GPT-style)
- Layers: 34
- Embedding dimension: 1280
- Attention heads: 20 (64 dim/head)
- FFN hidden dim: 5120 (4 × embedding dim)
- Context length: 1216 tokens
- Tokenizer: BERT tokenizer
- Parameters: 748,089,280
- Positional encoding: Rotary embeddings (RoPE)
- Activation: SwiLU/SiLU
- Norm: LayerNorm
- Checkpoint format: PyTorch
final_checkpoint.pt
Intended uses
- General text generation (English, tokenizer-dependent)
- Interactive prompting with sampling controls (temperature, top-k) via Gradio UI
Limitations & Risks
- May hallucinate facts or generate incoherent outputs
- Limited context length (1216 tokens)
Training data & procedure
- Tokenizer: local BERT tokenizer (vocab size 30,522)
- Objective: Causal language modeling
- Training framework: PyTorch + custom FSDP/AMP setup
- Checkpoints: Final checkpoint (
final_checkpoint.pt) available
