This model is a fine-tuned version of unsloth/Llama-3.2-3B-Instruct on the lmsys_chat dataset. It achieves the following results on the evaluation set:

Loss: 1.1817

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
training_steps: 1000
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.256	0.0160	100	1.1817
1.236	0.0320	200	1.1817
1.2212	0.0480	300	1.1817
1.1804	0.0641	400	1.1817
1.2801	0.0801	500	1.1817
1.2232	0.0961	600	1.1817
1.2433	0.1121	700	1.1817
1.2231	0.1281	800	1.1817
1.2272	0.1441	900	1.1817
1.2843	0.1602	1000	1.1817

Framework versions

Transformers 4.45.0
Pytorch 2.4.0+cu124
Datasets 2.19.1
Tokenizers 0.20.0

RichardErkhov/win10_-_Llama-3.2-3B-Instruct-24-9-29-4bits

Original model description:

my DC sever

我正在計畫微調64K指令模型，請幫助我進行計畫

Ko-fi: https://ko-fi.com/ogodwin10

Llama-3.2-3B-Instruct-24-9-29

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

RichardErkhov/
win10_-_Llama-3.2-3B-Instruct-24-9-29-4bits