Training and evaluation data

KI dataset

hf-causal-experimental (pretrained=BEE-spoke-data/NanoLlama-GQA-L10-A32_KV8-v13-KI,revision=main,trust_remote_code=True,dtype='float'), limit: None, provide_description: False, num_fewshot: 0, batch_size: 8

Task	Version	Metric	Value		Stderr
arc_easy	0	acc	0.4322	±	0.0102
		acc_norm	0.3960	±	0.0100
boolq	1	acc	0.6196	±	0.0085
lambada_openai	0	ppl	61.6595	±	2.4362
		acc	0.2779	±	0.0062
openbookqa	0	acc	0.1540	±	0.0162
		acc_norm	0.2840	±	0.0202
piqa	0	acc	0.6028	±	0.0114
		acc_norm	0.6028	±	0.0114
winogrande	0	acc	0.5193	±	0.0140

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.00025
train_batch_size: 8
eval_batch_size: 4
seed: 2280
gradient_accumulation_steps: 16
total_train_batch_size: 128
optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
lr_scheduler_type: inverse_sqrt
lr_scheduler_warmup_ratio: 0.05
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.5744	0.08	200	2.7138	0.4776
2.5387	0.16	400	2.6713	0.4836
2.4718	0.23	600	2.6462	0.4873
2.4681	0.31	800	2.6328	0.4892
2.5351	0.39	1000	2.6227	0.4908
2.5316	0.47	1200	2.6159	0.4914
2.527	0.54	1400	2.6103	0.4921
2.4838	0.62	1600	2.6058	0.4930
2.4483	0.7	1800	2.6024	0.4934
2.426	0.78	2000	2.5998	0.4937
2.4685	0.86	2200	2.5961	0.4944
2.4473	0.93	2400	2.5937	0.4948

Framework versions

Transformers 4.36.0.dev0
Pytorch 2.1.0
Datasets 2.15.0
Tokenizers 0.15.0

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	29.23
AI2 Reasoning Challenge (25-Shot)	23.81
HellaSwag (10-Shot)	29.39
MMLU (5-Shot)	25.37
TruthfulQA (0-shot)	44.77
Winogrande (5-shot)	51.14
GSM8k (5-shot)	0.91