Views
No views yet
0.13.0.dev01base_model: SaurabhJ20/deva1-pretrain
2tokenizer_type: AutoTokenizer
3model_type: AutoModelForCausalLM
4load_in_8bit: false
5load_in_4bit: false
6strict: false
7datasets:
8- path: pretraining_subset_3002.jsonl
9 type: completion
10- path: axolotl_correction_conversations_storeflow.json
11 type: input_output
12- path: axolotl_rag_conversations_storeflow.jsonl
13 type: input_output
14- path: factual_sft_completion/combined_all_0.jsonl
15 type: completion
16- path: factual_sft_completion/combined_all_1.jsonl
17 type: completion
18- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_200000.jsonl
19 type: completion
20- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_200000.jsonl
21 type: completion
22- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_400000.jsonl
23 type: completion
24- path: generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_800000.jsonl
25 type: completion
26- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_200000.jsonl
27 type: completion
28- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_200000.jsonl
29 type: completion
30dataset_prepared_path: last_finetune_prepared
31output_dir: ./finetune-model-output
32seed: 1337
33sequence_len: 5000
34sample_packing: true
35pad_to_sequence_len: false
36shuffle_merged_datasets: true
37gradient_accumulation_steps: 8
38micro_batch_size: 1
39eval_batch_size: 4
40num_epochs: 5
41optimizer: paged_adamw_8bit
42lr_scheduler: constant
43learning_rate: 2.0e-05
44noisy_embedding_alpha: 5
45weight_decay: 0
46train_on_inputs: false
47group_by_length: false
48bf16: true
49fp16: false
50tf32: false
51gradient_checkpointing: true
52logging_steps: 1
53xformers_attention: false
54flash_attention: true
55chat_template: chatml
56auto_resume_from_checkpoints: false
57warmup_ratio: 0.1
58evals_per_epoch: 1
59val_set_size: 0.04
60saves_per_epoch: 1
61eval_sample_packing: false
62save_total_limit: 2
63special_tokens:
64 pad_token: <unk>
65use_liger_kernel: true
66plugins:
67- axolotl.integrations.liger.LigerPlugin
68liger_rope: true
69liger_rms_norm: true
70liger_glu_activation: true
71liger_layer_norm: true
72liger_fused_linear_cross_entropy: true
73sequence_length: 10000
74wandb_project: test-project
75wandb_entity: ''
76wandb_watch: ''
77wandb_run_id: ''
78wandb_log_model: ''
79hub_model_id: SaurabhJ20/deva1-sft
80hub_strategy: end
81| Training Loss | Epoch | Step | Validation Loss | Mem Active(gib) | Mem Allocated(gib) | Mem Reserved(gib) |
|---|---|---|---|---|---|---|
| No log | 0 | 0 | 1.6836 | 19.63 | 19.63 | 26.05 |
| 0.9692 | 0.9872 | 58 | 0.9428 | 29.42 | 29.42 | 31.46 |
| 0.6315 | 1.9702 | 116 | 0.9980 | 29.42 | 29.42 | 31.46 |
| 0.2773 | 2.9532 | 174 | 1.1260 | 29.42 | 29.42 | 31.51 |
| 0.216 | 3.9362 | 232 | 1.2337 | 29.42 | 29.42 | 31.51 |
| 0.0997 | 4.9191 | 290 | 1.2796 | 29.42 | 29.42 | 31.53 |