Views
No views yet
| Parameter | Value |
|---|---|
| Learning Rate | 6e-4 |
| Batch Size (effective) | 32 |
| Warmup Steps | 100 |
| Weight Decay | 0.1 |
| Adam Betas | (0.9, 0.95) |
| Gradient Clipping | 1.0 |
| Learning Rate Scheduler | Cosine |
| Precision | FP16 |
| Metric | Value |
|---|---|
| Final Training Loss | 6.2279 |
| Validation Loss | 5.5396 |
| Perplexity | 254.57 |
1from transformers import GPT2LMHeadModel, GPT2TokenizerFast
2
3# Load model and tokenizer
4model = GPT2LMHeadModel.from_pretrained('aa-nadim/nadim-recreate-gpt-2-124M')
5tokenizer = GPT2TokenizerFast.from_pretrained('aa-nadim/nadim-recreate-gpt-2-124M')
6
7# Set pad token
8tokenizer.pad_token = tokenizer.eos_token
9
10# Generate text
11prompt = 'The future of artificial intelligence'
12inputs = tokenizer(prompt, return_tensors='pt')
13outputs = model.generate(
14 inputs.input_ids,
15 max_new_tokens=100,
16 temperature=0.8,
17 do_sample=True,
18 top_k=50,
19 top_p=0.95
20)
21print(tokenizer.decode(outputs[0], skip_special_tokens=True))1from transformers import pipeline
2
3generator = pipeline('text-generation', model='aa-nadim/nadim-recreate-gpt-2-124M')
4output = generator('Once upon a time', max_length=100)
5print(output[0]['generated_text'])1@misc{nadim-gpt2-124m-2024,
2 author = {Nadim},
3 title = {Recreate GPT-2 124M from Scratch},
4 year = 2026,
5 publisher = {Hugging Face},
6 howpublished = {\url{https://huggingface.co/aa-nadim/nadim-recreate-gpt-2-124M}}
7}