You can deploy this model and start earning money today!

Drshyamsundaramindia/
shakespeare-transformer-lm

Name: shakespeare-transformer-lm
Availability: InStock

Views

No views yet

Shakespeare Character-Level Transformer Language Model

This is a character-level Transformer Language Model (TransformerLM) trained from scratch on the complete works of William Shakespeare. It demonstrates the fundamental principles of Transformer architecture, including multi-head self-attention and feed-forward networks, applied to sequence generation at the character level.

Model Architecture

Type: Character-level Transformer Language Model (TransformerLM)
Context Length (block_size): 256 characters
Embedding Dimension (n_embd): 384
Number of Attention Heads (n_head): 6
Number of Transformer Blocks (n_layer): 6
Dropout: 0.2
Vocabulary Size: 39 (based on lowercase characters from Shakespeare's text)

Training

The model is trained using the Adam optimizer with a learning rate of 0.001 on the "tinyshakespeare/input.txt" dataset.

Usage (Example)

To load and use this model, you would typically:

Load the model's state dictionary.
Recreate the vocabulary mapping (itos/stoi).
Use the generate method to produce new text.

python

# Example of how to load and use the model (pseudo-code)
# import torch
# from your_model_definition import TransformerLM

# # Assuming vocab_size and itos/stoi are available from training setup
# # model = TransformerLM(vocab_size)
# # model.load_state_dict(torch.load("pytorch_model.bin"))
# # model.eval()

# # context = torch.zeros((1, 1), dtype=torch.long, device='cpu') # Starting token
# # generated_indices = model.generate(context, max_new_tokens=100)
# # generated_text = "".join([itos[idx.item()] for idx in generated_indices[0]])
# # print(generated_text)

This model serves as an educational example to understand the mechanics of Transformer-based language models.

Drshyamsundaramindia/shakespeare-transformer-lm

Shakespeare Character-Level Transformer Language Model

Model Architecture

Training

Usage (Example)

Drshyamsundaramindia/
shakespeare-transformer-lm