griffin-v0.01-c3t-8layer-simplewiki-silu – AI Model by pszemraj | AlphaNeural AI

You can deploy this model and start earning money today!

pszemraj/
griffin-v0.01-c3t-8layer-simplewiki-silu

like

0

recurrent_gemma

text-generation

generated_from_trainer

pszemraj/simple_wikipedia_LM

autotrain_compatible

endpoints_compatible

Views

No views yet

griffin-v0.01-c3t-8layer-simplewiki-silu

griffin/recurrent_gemma arch
claude3 tokenizer (as an HF gpt2 tokenizer)

Model description

pretrain experiment on the pszemraj/simple_wikipedia_LM dataset.

It achieves the following results on the evaluation set:

Loss: 4.0476
Accuracy: 0.4224

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0003
train_batch_size: 4
eval_batch_size: 4
seed: 80085
gradient_accumulation_steps: 32
total_train_batch_size: 128
optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
lr_scheduler_type: constant_with_warmup
lr_scheduler_warmup_ratio: 0.05
num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
13.3276	0.2548	100	12.0402	0.0131
8.9207	0.5095	200	8.0312	0.0360
7.2681	0.7643	300	6.4775	0.0506
6.3187	1.0190	400	5.6227	0.0434
5.5695	1.2738	500	4.7796	0.3635
5.2926	1.5285	600	4.3923	0.3952
4.878	1.7833	700	4.1877	0.4085

Framework versions

Transformers 4.40.1
Pytorch 2.2.0+cu121
Datasets 2.19.0
Tokenizers 0.19.1