Views
No views yet
1Training Configuration:
2├─ Epochs: 3
3├─ Batch size: 32 per device (effective: 128 with gradient accumulation)
4├─ Learning rate: 5e-5 (cosine with 500 warmup steps)
5├─ Block size: 512 tokens
6├─ Weight decay: 0.01
7├─ Gradient clipping: 1.0
8└─ Optimizer: AdamW| Metric | Value |
|---|---|
| Validation Loss | 3.206 |
| Training Loss | 3.379 |
| Perplexity | ~25 |
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3# Load model
4tokenizer = AutoTokenizer.from_pretrained("bekalebendong/pendo-distilgpt2-wikitext")
5model = AutoModelForCausalLM.from_pretrained("bekalebendong/pendo-distilgpt2-wikitext")
6
7# Generate text
8prompt = "The history of"
9inputs = tokenizer(prompt, return_tensors="pt")
10outputs = model.generate(
11 **inputs,
12 max_new_tokens=50,
13 do_sample=True,
14 top_k=50,
15 top_p=0.95,
16 temperature=0.8
17)
18
19print(tokenizer.decode(outputs[0], skip_special_tokens=True))1from transformers import pipeline
2
3# Create prediction pipeline
4predictor = pipeline('text-generation', model="bekalebendong/pendo-distilgpt2-wikitext")
5
6# Get next word predictions
7text = "Machine learning is"
8predictions = predictor(
9 text,
10 max_new_tokens=1,
11 num_return_sequences=5,
12 return_full_text=False
13)
14
15for pred in predictions:
16 print(pred['generated_text'])1@misc{pendo-distilgpt2-wikitext-103,
2 author = {Dimitri Bekale},
3 title = {Pendo DistilGPT2 - Fine-tuned on WikiText-103},
4 year = {2025},
5 publisher = {HuggingFace},
6 howpublished = {\url{https://huggingface.co/bekalebendong/pendo-distilgpt2-wikitext}}
7}