Views
No views yet
| Name | Quant method | Size |
|---|---|---|
| Dante_9B.Q2_K.gguf | Q2_K | 3.54GB |
| Dante_9B.IQ3_XS.gguf | IQ3_XS | 3.86GB |
| Dante_9B.IQ3_S.gguf | IQ3_S | 4.04GB |
| Dante_9B.Q3_K_S.gguf | Q3_K_S | 4.04GB |
| Dante_9B.IQ3_M.gguf | IQ3_M | 4.19GB |
| Dante_9B.Q3_K.gguf | Q3_K | 4.43GB |
| Dante_9B.Q3_K_M.gguf | Q3_K_M | 4.43GB |
| Dante_9B.Q3_K_L.gguf | Q3_K_L | 4.78GB |
| Dante_9B.IQ4_XS.gguf | IQ4_XS | 4.86GB |
| Dante_9B.Q4_0.gguf | Q4_0 | 5.07GB |
| Dante_9B.IQ4_NL.gguf | IQ4_NL | 5.1GB |
| Dante_9B.Q4_K_S.gguf | Q4_K_S | 5.1GB |
| Dante_9B.Q4_K.gguf | Q4_K | 5.37GB |
| Dante_9B.Q4_K_M.gguf | Q4_K_M | 5.37GB |
| Dante_9B.Q4_1.gguf | Q4_1 | 5.55GB |
| Dante_9B.Q5_0.gguf | Q5_0 | 6.04GB |
| Dante_9B.Q5_K_S.gguf | Q5_K_S | 6.04GB |
| Dante_9B.Q5_K.gguf | Q5_K | 6.19GB |
| Dante_9B.Q5_K_M.gguf | Q5_K_M | 6.19GB |
| Dante_9B.Q5_1.gguf | Q5_1 | 6.52GB |
| Dante_9B.Q6_K.gguf | Q6_K | 7.07GB |
| Dante_9B.Q8_0.gguf | Q8_0 | 9.15GB |
0.4.11base_model: IntervitensInc/gemma-2-9b-chatml
2model_type: AutoModelForCausalLM
3tokenizer_type: AutoTokenizer
4
5load_in_8bit: false
6load_in_4bit: false
7strict: false
8
9datasets:
10 - path: FourOhFour/Instruct_Phase
11 type: sharegpt
12 conversation: chatml
13
14chat_template: chatml
15
16val_set_size: 0.0025
17output_dir: ./outputs/out
18
19adapter:
20lora_r:
21lora_alpha:
22lora_dropout:
23lora_target_linear:
24
25sequence_len: 8192
26sample_packing: true
27eval_sample_packing: false
28pad_to_sequence_len: true
29
30plugins:
31 - axolotl.integrations.liger.LigerPlugin
32liger_rope: true
33liger_rms_norm: false
34liger_swiglu: true
35liger_fused_linear_cross_entropy: false
36
37wandb_project: chatml9B
38wandb_entity:
39wandb_watch:
40wandb_name: chatml9B
41wandb_log_model:
42
43gradient_accumulation_steps: 32
44micro_batch_size: 1
45num_epochs: 2
46optimizer: adamw_bnb_8bit
47lr_scheduler: cosine
48learning_rate: 0.000008
49weight_decay: 0.05
50
51train_on_inputs: false
52group_by_length: false
53bf16: auto
54fp16:
55tf32: true
56
57gradient_checkpointing: true
58early_stopping_patience:
59resume_from_checkpoint:
60local_rank:
61logging_steps: 1
62xformers_attention:
63flash_attention: true
64
65warmup_ratio: 0.1
66evals_per_epoch: 4
67eval_table_size:
68eval_max_new_tokens: 128
69saves_per_epoch: 2
70
71debug:
72deepspeed: deepspeed_configs/zero3_bf16.json
73fsdp:
74fsdp_config:
75
76special_tokens:
77 pad_token: <pad>
78| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.0841 | 0.0048 | 1 | 1.0069 |
| 0.7835 | 0.2518 | 53 | 0.7502 |
| 0.7594 | 0.5036 | 106 | 0.7305 |
| 0.7348 | 0.7555 | 159 | 0.7231 |
| 0.6312 | 1.0055 | 212 | 0.7228 |
| 0.6077 | 1.2574 | 265 | 0.7362 |
| 0.607 | 1.5092 | 318 | 0.7328 |
| 0.5734 | 1.7611 | 371 | 0.7320 |