You can deploy this model and start earning money today!

RichardErkhov/
tanliboy_-_llama-3.2-3b-dpo-2-gguf

Name: tanliboy_-_llama-3.2-3b-dpo-2-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

llama-3.2-3b-dpo-2 - GGUF

Model creator: https://huggingface.co/tanliboy/
Original model: https://huggingface.co/tanliboy/llama-3.2-3b-dpo-2/

Name	Quant method	Size
llama-3.2-3b-dpo-2.Q2_K.gguf	Q2_K	1.27GB
llama-3.2-3b-dpo-2.IQ3_XS.gguf	IQ3_XS	1.38GB
llama-3.2-3b-dpo-2.IQ3_S.gguf	IQ3_S	1.44GB
llama-3.2-3b-dpo-2.Q3_K_S.gguf	Q3_K_S	1.44GB
llama-3.2-3b-dpo-2.IQ3_M.gguf	IQ3_M	1.49GB
llama-3.2-3b-dpo-2.Q3_K.gguf	Q3_K	1.57GB
llama-3.2-3b-dpo-2.Q3_K_M.gguf	Q3_K_M	1.57GB
llama-3.2-3b-dpo-2.Q3_K_L.gguf	Q3_K_L	1.69GB
llama-3.2-3b-dpo-2.IQ4_XS.gguf	IQ4_XS	1.71GB
llama-3.2-3b-dpo-2.Q4_0.gguf	Q4_0	1.79GB
llama-3.2-3b-dpo-2.IQ4_NL.gguf	IQ4_NL	1.79GB
llama-3.2-3b-dpo-2.Q4_K_S.gguf	Q4_K_S	1.8GB
llama-3.2-3b-dpo-2.Q4_K.gguf	Q4_K	1.88GB
llama-3.2-3b-dpo-2.Q4_K_M.gguf	Q4_K_M	1.88GB
llama-3.2-3b-dpo-2.Q4_1.gguf	Q4_1	1.95GB
llama-3.2-3b-dpo-2.Q5_0.gguf	Q5_0	2.11GB
llama-3.2-3b-dpo-2.Q5_K_S.gguf	Q5_K_S	2.11GB
llama-3.2-3b-dpo-2.Q5_K.gguf	Q5_K	2.16GB
llama-3.2-3b-dpo-2.Q5_K_M.gguf	Q5_K_M	2.16GB
llama-3.2-3b-dpo-2.Q5_1.gguf	Q5_1	2.28GB
llama-3.2-3b-dpo-2.Q6_K.gguf	Q6_K	2.46GB
llama-3.2-3b-dpo-2.Q8_0.gguf	Q8_0	3.19GB

Original model description:

library_name: transformers license: llama3.2 base_model: tanliboy/llama-3.2-3b-sft-2 tags:

alignment-handbook
trl
dpo
generated_from_trainer
trl
dpo
generated_from_trainer datasets:
HuggingFaceH4/orca_dpo_pairs
HuggingFaceH4/ultrafeedback_binarized model-index:
name: llama-3.2-3b-dpo-2 results: []

llama-3.2-3b-dpo-2

This model is a fine-tuned version of tanliboy/llama-3.2-3b-sft-2 on the HuggingFaceH4/orca_dpo_pairs and the HuggingFaceH4/ultrafeedback_binarized datasets. It achieves the following results on the evaluation set:

Loss: 0.5814
Rewards/chosen: 1.7432
Rewards/rejected: -4.1735
Rewards/accuracies: 0.7848
Rewards/margins: 5.9167
Logps/rejected: -388.2242
Logps/chosen: -338.5596
Logits/rejected: 0.2395
Logits/chosen: 0.1826

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
gradient_accumulation_steps: 4
total_train_batch_size: 128
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.03
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.7596	0.1741	100	0.7588	0.1349	-1.4398	0.6994	1.5747	-360.8871	-354.6434	0.6135	0.5482
0.6725	0.3483	200	0.6680	0.6247	-2.7323	0.7278	3.3569	-373.8118	-349.7451	0.5335	0.4718
0.6452	0.5224	300	0.6514	0.1770	-3.8036	0.75	3.9807	-384.5256	-354.2216	0.5477	0.4866
0.6259	0.6966	400	0.6328	0.9885	-3.5382	0.7722	4.5267	-381.8713	-346.1070	0.4531	0.3927
0.5709	0.8707	500	0.6219	0.9150	-4.0091	0.7816	4.9242	-386.5804	-346.8415	0.4148	0.3563
0.5835	1.0448	600	0.6094	1.5034	-3.6390	0.7722	5.1423	-382.8790	-340.9584	0.3504	0.2933
0.5571	1.2190	700	0.5992	1.5696	-3.7206	0.7690	5.2901	-383.6949	-340.2962	0.3217	0.2649
0.5532	1.3931	800	0.5954	1.7147	-3.7261	0.7785	5.4408	-383.7506	-338.8453	0.2961	0.2383
0.5168	1.5673	900	0.5930	1.9934	-3.3982	0.7753	5.3916	-380.4709	-336.0577	0.2838	0.2266
0.5232	1.7414	1000	0.5884	1.7308	-4.0024	0.7816	5.7332	-386.5127	-338.6839	0.2787	0.2220
0.5574	1.9155	1100	0.5849	1.8420	-3.9351	0.7911	5.7771	-385.8401	-337.5714	0.2706	0.2134
0.5077	2.0897	1200	0.5842	1.6188	-4.2472	0.7880	5.8659	-388.9607	-339.8043	0.2657	0.2083
0.4952	2.2638	1300	0.5837	1.9316	-3.8913	0.7816	5.8229	-385.4018	-336.6759	0.2694	0.2115
0.5236	2.4380	1400	0.5812	1.8289	-4.0636	0.7880	5.8925	-387.1253	-337.7025	0.2465	0.1895
0.5001	2.6121	1500	0.5814	1.7432	-4.1735	0.7848	5.9167	-388.2242	-338.5596	0.2395	0.1826
0.5246	2.7862	1600	0.5809	1.8622	-4.0120	0.7880	5.8742	-386.6093	-337.3701	0.2395	0.1825
0.5042	2.9604	1700	0.5808	1.8125	-4.0822	0.7880	5.8947	-387.3112	-337.8669	0.2355	0.1785

Framework versions

Transformers 4.44.2
Pytorch 2.4.0+cu121
Datasets 2.19.1
Tokenizers 0.19.1

RichardErkhov/tanliboy_-_llama-3.2-3b-dpo-2-gguf

Original model description:

llama-3.2-3b-dpo-2

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

RichardErkhov/
tanliboy_-_llama-3.2-3b-dpo-2-gguf