See axolotl config
axolotl version: 0.4.1
1adapter: null
2base_model: Qwen/Qwen2-0.5B
3bf16: auto
4chat_template: chatml
5dataset_prepared_path: ./data/last_run_prepared
6datasets:
7- path: Magpie-Align/Magpie-Qwen2-Pro-300K-Filtered
8 type: sharegpt
9deepspeed: null
10early_stopping_patience: null
11eval_sample_packing: true
12evals_per_epoch: 4
13flash_attention: true
14fp16: null
15fsdp: null
16gradient_accumulation_steps: 4
17gradient_checkpointing: true
18gradient_checkpointing_kwargs:
19 use_reentrant: false
20group_by_length: false
21hf_use_auth_token: true
22hub_model_id: CoolSpring/Qwen2-0.5B-Abyme
23learning_rate: 2e-5
24load_in_4bit: false
25load_in_8bit: false
26local_rank: null
27logging_steps: 1
28lr_scheduler: cosine
29micro_batch_size: 4
30num_epochs: 1
31optimizer: adamw_torch
32output_dir: ./outputs/out
33pad_to_sequence_len: true
34resize_token_embeddings_to_32x: true
35resume_from_checkpoint: null
36sample_packing: true
37saves_per_epoch: 1
38sequence_len: 4096
39tf32: true
40tokens:
41- <|im_start|>
42- <|im_end|>
43train_on_inputs: false
44val_set_size: 0.05
45wandb_entity: null
46wandb_log_model: null
47wandb_name: Qwen2-0.5B-Abyme
48wandb_project: Qwen2-0.5B-Magpie-Qwen2-Pro-300K-Filtered
49wandb_watch: null
50warmup_steps: 100
51weight_decay: null
52xformers_attention: null
53
Qwen2-0.5B-Abyme
This model is a fine-tuned version of
Qwen/Qwen2-0.5B on the
Magpie-Align/Magpie-Qwen2-Pro-300K-Filtered dataset. It was created to explore the effects of training the smallest model in the Qwen2 series on data extracted from the largest model in the Qwen2 series (as of July 18th, 2024).
It achieves the following results on the evaluation set:
Model description
Qwen2-0.5B-Abyme is a 0.5 billion parameter language model fine-tuned on a dataset of conversation samples from the much larger 72 billion parameter Qwen2-72B model. The purpose of this experiment is to investigate whether a smaller model can effectively learn and reproduce the knowledge and capabilities of a significantly larger model through the fine-tuning process.
Intended uses & limitations
This model is intended for research purposes to study the knowledge transfer and distillation capabilities of language models. It may have practical applications in scenarios where the computational resources for running large language models are limited, and a smaller, fine-tuned model can provide comparable performance.
However, it is important to note that the model's capabilities and limitations are yet to be fully evaluated. Its performance may vary depending on the task and domain, and it may exhibit biases or limitations inherited from the original models.
Training and evaluation data
The model was fine-tuned on the
Magpie-Align/Magpie-Qwen2-Pro-300K-Filtered dataset, which contains 300,000 conversation samples from the Qwen2-72B model. 5% of this dataset was held out as the evaluation set for calculating the reported loss metric.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|
| 0.9947 | 0.0004 | 1 | 0.9683 |
| 0.8385 | 0.2501 | 597 | 0.8338 |
| 0.7636 | 0.5002 | 1194 | 0.8249 |
| 0.8124 | 0.7502 | 1791 | 0.8229 |
Framework versions
- Transformers 4.42.3
- Pytorch 2.3.1+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1
Detailed results can be found
here| Metric | Value |
|---|
| Avg. | 4.76 |
| IFEval (0-Shot) | 19.15 |
| BBH (3-Shot) | 2.28 |
| MATH Lvl 5 (4-Shot) | 1.51 |
| GPQA (0-shot) | 0.45 |
| MuSR (0-shot) | 1.48 |
| MMLU-PRO (5-shot) | 3.70 |
