You can deploy this model and start earning money today!

meng-lab/
AdaDecode-CodeLlama-34B-Instruct-GSM8K

Views

No views yet

Visualize in Weights & Biases

CodeLlama-34b-Instruct-sft-5e-3-epoch-100-gsm8k

This model is a fine-tuned version of meta-llama/CodeLlama-34b-Instruct-hf on the meng-lab/CodeLlama-34B-Instruct-gsm8k dataset. It achieves the following results on the evaluation set:

Loss: 4.0230
Loss Layer 6 Head: 1.2898
Loss Layer 12 Head: 1.0049
Loss Layer 18 Head: 0.9093
Loss Layer 24 Head: 0.4408
Loss Layer 30 Head: 0.2683
Loss Layer 36 Head: 0.1391
Loss Layer 42 Head: 0.0639

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.005
train_batch_size: 1
eval_batch_size: 2
seed: 42
distributed_type: multi-GPU
num_devices: 8
gradient_accumulation_steps: 16
total_train_batch_size: 128
total_eval_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 100

Training results

Training Loss	Epoch	Step	Validation Loss	Loss Layer 6 Head	Loss Layer 12 Head	Loss Layer 18 Head	Loss Layer 24 Head	Loss Layer 30 Head	Loss Layer 36 Head	Loss Layer 42 Head
2.6241	25.8065	200	4.3768	1.3707	1.0927	0.9492	0.4907	0.2888	0.1534	0.0899
1.6189	51.6129	400	4.0476	1.3067	0.9916	0.9104	0.4445	0.2716	0.1405	0.0663
1.3737	77.4194	600	4.0230	1.2898	1.0049	0.9093	0.4408	0.2683	0.1391	0.0639

Framework versions

Transformers 4.43.2
Pytorch 2.1.2
Datasets 3.2.0
Tokenizers 0.19.1

meng-lab/AdaDecode-CodeLlama-34B-Instruct-GSM8K

CodeLlama-34b-Instruct-sft-5e-3-epoch-100-gsm8k

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

meng-lab/
AdaDecode-CodeLlama-34B-Instruct-GSM8K