You can deploy this model and start earning money today!

tetsydorenko/
my-transformer-lm

Name: my-transformer-lm
Availability: InStock

pytorch

transformerlm

Views

No views yet

📘 My Transformer Language Model

This is a small Transformer-based language model trained from scratch on the Wikitext-2 dataset. It was developed as part of an educational project to understand transformer architectures and language modeling. The model learns to predict the next token in a sequence using an autoregressive approach.

📦 Model Details

Architecture: Decoder-only Transformer (GPT-like)
Layers: 6
Attention Heads: 8
Hidden Size (Embedding Dimension): 512
Block Size (Context Window): 128 tokens
Dropout: 0.1
Vocabulary Size: 50257 (from GPT-2 tokenizer)
Total Parameters: ~32M Estimated for 6 layers of self-attention and feedforward networks using 512-dimensional embeddings and 8 heads.

🔤 Tokenizer

Type: AutoTokenizer from Transformers
Pretrained Model: gpt2
Padding Token: set to eos_token
Tokenization Method: Byte-level BPE (as used in GPT-2)

📊 Training Configuration

Dataset: WikiText-2-raw-v1
Epochs: 5
Batch Size: 32
Optimizer: AdamW
Learning Rate: 1e-4
Scheduler: Cosine Annealing
Loss Function: CrossEntropyLoss (ignoring padding token)

Trained using Google Colab with GPU acceleration.

🧪 Evaluation

The model was evaluated on the validation split of Wikitext-2. Validation loss and training loss are tracked via Weights & Biases.

📎 Files Included

pytorch_model.bin: Trained model weights.
config.json: Training configuration.
model.py: Script with the model's description.
tokenizer_config.json, special_tokens_map.json, vocab.json, merges.txt, tokenizer.json: Tokenizer.
README.md: This model card.
(to be added): Script to generate text using the model (generate.py - optional, to be added).

💡 How to Use

Here’s an example of how to load and use the model for generation:

python

import torch
from model import TransformerLM  # model class
from transformers import AutoTokenizer

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# Load model (adjust path if using from HF Hub or locally)
model = TransformerLM(
    vocab_size=tokenizer.vocab_size,
    embed_dim=512,
    n_heads=8,
    n_layers=6,
    block_size=128,
    dropout=0.1
)
model.load_state_dict(torch.load("path_to_model.pth"))
model.eval()

# Generate text
prompt = "our future"
output = generate(model, tokenizer, prompt)
print(output)

Note: The generate function is custom, not from Transformers API.

🔒 Limitations

Not suitable for production or safety-critical applications.
Trained on a small subset of text data.
Not aligned or filtered for offensive content.

🧠 Intended Use This model is for educational and research purposes. It is not fine-tuned for production tasks such as conversation, summarization, or question answering.

✍️ Author Developed by: [Tetiana Sydorenko]

Project GitHub: (to be added)

🤝 License MIT License

tetsydorenko/my-transformer-lm