You can deploy this model and start earning money today!

ChenDRAG/
zephyr-NCA-preference

Name: zephyr-NCA-preference
Availability: InStock

text-generation-inference

endpoints_compatible

4-bit

bitsandbytes

Views

No views yet

zephyr-NCA-preference

This model is a fine-tuned version of HuggingFaceH4/mistral-7b-sft-beta on the openbmb/UltraFeedback dataset. It achieves the following results on the evaluation set:

Loss: 1.3030
Rewards/chosen: 0.0489
Rewards/rejected: -0.5399
Rewards/accuracies: 0.7820
Rewards/margins: 0.5888
Verify/constant 1: 1.0
Verify/constant 1len: 1000.0
Logps/rejected: -287.1594
Logps/chosen: -270.2584
Verify/bz: 1.0
Verify/gather Bz: 2.0
Regularization/forward Kl: 0.6109
Regularization/reverse Kl: 0.4631
Regularization/policy Data Loss: 1.8007
Regularization/reference Data Loss: 1.3337
Regularization/policy Ref Data Loss Gap: 0.4670
Mask/mask Ratio: 0.4809

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-06
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 2
gradient_accumulation_steps: 16
total_train_batch_size: 32
total_eval_batch_size: 2
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Verify/constant 1	Verify/constant 1len	Logps/rejected	Logps/chosen	Verify/bz	Verify/gather Bz	Regularization/forward Kl	Regularization/reverse Kl	Regularization/policy Data Loss	Regularization/reference Data Loss	Regularization/policy Ref Data Loss Gap	Mask/mask Ratio
1.3844	0.05	100	1.3839	0.0037	-0.0061	0.7075	0.0098	1.0	1000.0	-233.7844	-274.7838	1.0	2.0	0.0009	0.0009	1.3404	1.3337	0.0067	0.4809
1.3593	0.1	200	1.3605	-0.0445	-0.1811	0.7320	0.1366	1.0	1000.0	-251.2808	-279.5988	1.0	2.0	0.1063	0.0867	1.4942	1.3337	0.1604	0.4809
1.3432	0.15	300	1.3399	-0.0181	-0.2809	0.7695	0.2628	1.0	1000.0	-261.2633	-276.9577	1.0	2.0	0.2787	0.2104	1.5199	1.3337	0.1862	0.4809
1.3404	0.21	400	1.3251	0.0042	-0.3854	0.7720	0.3896	1.0	1000.0	-271.7116	-274.7323	1.0	2.0	0.5454	0.4274	1.5819	1.3337	0.2481	0.4809
1.3295	0.26	500	1.3173	0.0213	-0.4300	0.7770	0.4513	1.0	1000.0	-276.1767	-273.0250	1.0	2.0	0.5684	0.4290	1.6808	1.3337	0.3471	0.4809
1.3187	0.31	600	1.3122	0.0267	-0.4649	0.7790	0.4917	1.0	1000.0	-279.6683	-272.4786	1.0	2.0	0.5839	0.4556	1.7090	1.3337	0.3753	0.4809
1.3105	0.36	700	1.3106	0.0180	-0.5079	0.7685	0.5259	1.0	1000.0	-283.9655	-273.3516	1.0	2.0	0.5818	0.4701	1.8137	1.3337	0.4800	0.4809
1.3086	0.41	800	1.3094	0.0287	-0.5003	0.7820	0.5290	1.0	1000.0	-283.2076	-272.2820	1.0	2.0	0.5724	0.4410	1.7950	1.3337	0.4613	0.4809
1.3164	0.46	900	1.3071	0.0494	-0.4863	0.7865	0.5356	1.0	1000.0	-281.7993	-270.2156	1.0	2.0	0.5937	0.4471	1.6937	1.3337	0.3599	0.4809
1.3065	0.52	1000	1.3058	0.0442	-0.5122	0.7875	0.5564	1.0	1000.0	-284.3954	-270.7371	1.0	2.0	0.6214	0.4609	1.7262	1.3337	0.3925	0.4809
1.3274	0.57	1100	1.3097	0.0187	-0.5605	0.7765	0.5792	1.0	1000.0	-289.2202	-273.2801	1.0	2.0	0.6048	0.4467	1.9267	1.3337	0.5930	0.4809
1.3128	0.62	1200	1.3053	0.0391	-0.5393	0.7795	0.5784	1.0	1000.0	-287.1077	-271.2448	1.0	2.0	0.5974	0.4596	1.8496	1.3337	0.5159	0.4809
1.3018	0.67	1300	1.3043	0.0370	-0.5532	0.7765	0.5902	1.0	1000.0	-288.4903	-271.4501	1.0	2.0	0.6164	0.4737	1.8233	1.3337	0.4896	0.4809
1.3137	0.72	1400	1.3040	0.0532	-0.5183	0.7790	0.5715	1.0	1000.0	-285.0031	-269.8345	1.0	2.0	0.5985	0.4642	1.7409	1.3337	0.4072	0.4809
1.304	0.77	1500	1.3034	0.0489	-0.5344	0.7815	0.5833	1.0	1000.0	-286.6187	-270.2639	1.0	2.0	0.6056	0.4668	1.7960	1.3337	0.4623	0.4809
1.3194	0.83	1600	1.3033	0.0496	-0.5367	0.7770	0.5864	1.0	1000.0	-286.8489	-270.1884	1.0	2.0	0.6093	0.4660	1.7863	1.3337	0.4526	0.4809
1.3194	0.88	1700	1.3030	0.0498	-0.5367	0.7820	0.5865	1.0	1000.0	-286.8430	-270.1689	1.0	2.0	0.6106	0.4640	1.7905	1.3337	0.4568	0.4809
1.32	0.93	1800	1.3031	0.0475	-0.5425	0.7815	0.5901	1.0	1000.0	-287.4280	-270.3985	1.0	2.0	0.6118	0.4635	1.8042	1.3337	0.4705	0.4809
1.3119	0.98	1900	1.3030	0.0490	-0.5398	0.7810	0.5888	1.0	1000.0	-287.1560	-270.2523	1.0	2.0	0.6107	0.4630	1.8007	1.3337	0.4670	0.4809

Framework versions

Transformers 4.35.0
Pytorch 2.1.0
Datasets 2.14.6
Tokenizers 0.14.1

ChenDRAG/zephyr-NCA-preference

zephyr-NCA-preference

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

ChenDRAG/
zephyr-NCA-preference