You can deploy this model and start earning money today!

bwalshe/
bigram-model

Name: bigram-model
Rating: 2.5 (6 reviews)

Views

No views yet

bigram-model

This model is intended as a demo example of creating a custom Hugging Face model from scratch, and the code for this is availabel here

It is based on the bigram model demonstrated by Andrej Karpathy in his Nano-GPT lecture series.

It achieves the following results on the evaluation set:

This is a very basic bigram bassd model which uses a single embedding layer to predict the next token

This model is inteded to demonstrate the use of a custom model with the Trainer and Pipeline APIs

It is not intended for actual use as a causal language model.

More information needed

The following hyperparameters were used during training:

learning_rate: 0.002
train_batch_size: 16
eval_batch_size: 16
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 5

Training Loss	Epoch	Step	Validation Loss
7.1744	1.0	786	6.7975
6.4152	2.0	1572	6.4194
6.1543	3.0	2358	6.1970
5.8633	4.0	3144	6.0804
5.7901	5.0	3930	6.0437