You can deploy this model and start earning money today!

CodeDevX/
5m-text-generator

Name: 5m-text-generator
Rating: 2.5 (17 reviews)

Views

No views yet

5M Text Generator

A small 5M parameter text generation model trained on educational/scientific text data.

Model Details

Parameters: 4,983,808 (4.98M)
Architecture: Transformer Decoder with GQA, RoPE, SwiGLU
Training: 1000 steps on 5000 text samples
Vocabulary: 8000 tokens (BPE)

Usage

python

from transformers import PreTrainedTokenizerFast
import torch
import sys
sys.path.insert(0, ".")  # if needed

from model import TextDecoder

# Load model
model = TextDecoder.from_pretrained("CodeDevX/5m-text-generator")
tokenizer = PreTrainedTokenizerFast.from_pretrained("CodeDevX/5m-text-generator")

# Generate text
input_ids = tokenizer("The quick brown fox", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, temperature=0.8)
print(tokenizer.decode(output[0]))

Architecture

Hidden size: 256
Layers: 6
Attention heads: 8 (KV heads: 2)
Intermediate size: 768
Max sequence length: 512

Training Results

Step	Loss
100	12.68
200	2.17
500	0.15
1000	0.13

License

MIT