GPT-2 Medium Somali — Text Generation Only

Suggested model ID: FatihJimale/gpt2-medium-somali

🔑 Key facts

• Architecture: GPT‑2 Medium (12 layers × 1024 hidden size × 16 heads, ~345M params)
• Objective: causal language modeling (next‑token prediction)
• Context length: 1024 tokens (set to your actual value if different)
• Tokenizer: GPT‑2 BPE (fast)
• Framework: 🤗 Transformers
• Precision: FP16/BF16 compatible at inference

✅ Intended use

• Somali text generation (stories, headlines, news‑style sentences, prompts)
• Assistant‑style completions in Somali

⚠️ Limitations

• May generate inaccurate, offensive, or biased content.
• Not suitable for factual QA without verification.
• Avoid safety‑critical usage.

🚀 Quick start (Transformers)

Inference tips

• If repetition appears, increase repetition_penalty to 1.1–1.2 or lower temperature (0.7–0.9).
• For more focused generations, reduce max_new_tokens and set top_p around 0.9.
• Deterministic output: do_sample=False, tune top_k=None, temperature=1.0.

🔧 Model details

• Training steps: ≈14,850 (completed at ~epoch 2.00)
• Epochs: 2
• Effective batch size: 64
• Learning rate & schedule: final logged LR ≈ 8.998e-10
• Optimizer: AdamW (β1=0.9, β2=0.999)
• Weight decay: 0.01
• Mixed precision: bf16
• Hardware: AWS ml.g5.24xlarge — 4× NVIDIA A10 (24 GB each), 96 vCPU, 384 GiB RAM; data-parallel across 4 GPUs
• Context length: 1024 tokens
• Tokenizer: GPT‑2 BPE (fast) (no custom Somali tokenizer in this version)
• Train date: 2025‑09‑25
• Runtime: evaluation runtime ≈ 1652.22 s (~27.5 min); overall training wall‑clock ≈ 1.337 days (≈ 32 h 05 m 17 s)

Note: Dataset specifics and cleaning steps are intentionally not disclosed here, per the author's request. This card focuses on model size, parameters, and usage.

📊 Evaluation (please populate)

• Train loss (last logged): 1.8449 @ step 14850 (~epoch 2.00)
• Eval/validation loss: 1.78604
• Perplexity (valid/test): 5.9658 (final recorded value @ 2025‑09‑25 09:06:42)
• Eval runtime: 1652.22 s, 72.272 samples/s, 9.035 steps/s
• Human eval notes: TBD (fluency, coherence)
• Train loss: 1.8449
• Eval/validation loss: 1.78604
• Perplexity (valid/test): 5.9658
• Human eval notes: TBD (fluency, coherence)

📁 Repo layout

config.json
pytorch_model.bin  (or model.safetensors)
merges.txt
vocab.json
tokenizer.json
tokenizer_config.json
special_tokens_map.json (if any)
README.md (this file)

Tusaale (Isticmaal Somali)

📣 Citation

🔐 Safety