🌟 SmolLM-135M SQuAD v1.1 Generative Question Answering Model
Fine-tuned SmolLM-135M-Instruct on SQuAD v1.1 for English generative question answering.
This model takes a context paragraph and a question and generates a natural-language answer based on the text.
Unlike extractive QA models (e.g., BERT), this model does not select spans—it produces free-form answers using a causal language modeling head.
It is intended as a compact, educational example of fine-tuning a small (~135M parameter) generative LM for QA on Google Colab and deploying it on Hugging Face.
🔎 Use Cases
- Educational demos of generative question answering
- Small QA systems over short English paragraphs
- Teaching / learning how to:
- convert extractive data to generative format
- fine-tune LMs with
Trainer
- publish models + Spaces on Hugging Face
⚠️ Not intended for production-critical use (medical/legal/financial advice, etc.).
🧠 Model Details
- Base model:
HuggingFaceTB/SmolLM-135M-Instruct
- Architecture: Decoder-only Transformer with causal LM head
- Parameters: ~135M
- Task: Generative Question Answering
- Language: English
- Author:
omarbayoumi2
- Training platform: Google Colab (free GPU)
The model generates answers instead of choosing spans directly from the context.
📚 Training Data
- Dataset: SQuAD v1.1 (
rajpurkar/squad)
- Domain: Wikipedia articles (encyclopedic text)
Each example was converted into a generative prompt:
context: paragraphquestion: question stringanswers:
Only answer tokens contribute to the training loss (prompt tokens masked with -100).
⚙️ Training Configuration
Fine-tuning was performed with the Hugging Face Trainer API.
- Objective: Causal LM
- Epochs: 2
- Learning rate: 2e-4
- Batch size: 8 (train), 8 (eval)
- Max sequence length: 256 tokens
- Weight decay: 0.01
- Padding strategy: Max length
- Mixed precision: FP16 (when GPU supports it)
Loss is computed over answer tokens only.
📊 Evaluation
Evaluation was performed on a subset of the SQuAD v1.1 validation set using extractive metrics:
- Exact Match (EM): strict text match
- F1 score: token-level overlap
Because outputs are generative rather than span-based, metrics may underestimate correctness.
| Metric | Score (approx) |
|---|
| Exact Match | ~1% |
| F1 | ~9% |
This model is optimized for educational usage, not leaderboard performance.
🚀 How to Use
1. With Transformers pipeline
1from transformers import pipeline
2
3qa = pipeline(
4 "text-generation",
5 model="omarbayoumi2/smollm-135m-genqa",
6)
7
8context = "BERT is a language representation model developed by researchers at Google."
9question = "Who developed BERT?"
10
11prompt = f"Context: {context}\n\nQuestion: {question}\n\nAnswer:"
12
13result = qa(prompt, max_new_tokens=64, do_sample=False)[0]["generated_text"]
14print(result)
