You can deploy this model and start earning money today!

lsmille/
lora_evo_ta_all_layers_2

Name: lora_evo_ta_all_layers_2
Availability: InStock

Views

No views yet

lora_evo_ta_all_layers_2

This model is a fine-tuned version of togethercomputer/evo-1-8k-base on the None dataset. It achieves the following results on the evaluation set:

lora_alpha = 32

lora_dropout = 0.05

lora_r = 16

epochs = 9 <---------------

learning rate = 3e-4

warmup_steps=0.5

gradient_accumulation_steps = 8

train_batch = 1

eval_batch = 1

More information needed

More information needed

The following hyperparameters were used during training:

Training Loss	Epoch	Step	Validation Loss
3.0681	0.9925	33	2.9815
2.9165	1.9850	66	2.9530
2.8091	2.9774	99	2.9446
2.6361	4.0	133	2.9406
2.6312	4.9925	166	2.9409
2.57	5.9850	199	2.9978
2.5215	6.9774	232	3.0450
2.4107	8.0	266	3.0763
2.4272	8.9323	297	3.1660