See axolotl config

axolotl version: 0.10.0.dev0

yaml

1base_model: RedHatAI/Sparse-Llama-3.1-8B-2of4 2 3load_in_8bit: false 4load_in_4bit: false 5strict: false 6 7datasets: 8 - path: trl-lib/tldr 9 type: 10 system_prompt: "Give a TL;DR of the following Reddit post." 11 field_system: system 12 field_instruction: prompt 13 field_output: completion 14 format: "<|user|>\n{instruction}\n<|assistant|>\n" 15 no_input_format: "<|user|>\n{instruction}\n<|assistant|>\n" 16 split: train 17 18dataset_prepared_path: last_run_prepared 19output_dir: Sparse-Llama-3.1-8B-2of4-tldr 20 21sequence_len: 4096 22sample_packing: true 23pad_to_sequence_len: true 24eval_sample_packing: true 25 26torch.compile: true 27gradient_accumulation_steps: 1 28micro_batch_size: 1 29num_epochs: 1 30optimizer: adamw_bnb_8bit 31lr_scheduler: cosine 32learning_rate: 1e-5 33max_grad_norm: 1.0 34 35gradient_checkpointing: true 36gradient_checkpointing_kwargs: 37 use_reentrant: false 38 39train_on_inputs: false 40bf16: auto 41fp16: 42tf32: false 43 44early_stopping_patience: 45resume_from_checkpoint: 46logging_steps: 1 47flash_attention: true 48 49warmup_ratio: 0.05 50eval_steps: 0.05 51val_set_size: 0.05 52save_strategy: "best" 53metric_for_best_model: "loss" 54 55debug: 56deepspeed: 57weight_decay: 0.0 58special_tokens: 59 pad_token: "<|end_of_text|>" 60 61seed: 0 62 63plugins: 64 - axolotl.integrations.liger.LigerPlugin 65 - axolotl.integrations.llm_compressor.LLMCompressorPlugin 66 67liger_rope: true 68liger_rms_norm: true 69liger_glu_activation: true 70liger_layer_norm: true 71liger_fused_linear_cross_entropy: true 72 73llmcompressor: 74 recipe: 75 finetuning_stage: 76 finetuning_modifiers: 77 ConstantPruningModifier: 78 targets: [ 79 're:.*q_proj.weight', 80 're:.*k_proj.weight', 81 're:.*v_proj.weight', 82 're:.*o_proj.weight', 83 're:.*gate_proj.weight', 84 're:.*up_proj.weight', 85 're:.*down_proj.weight', 86 ] 87 start: 0 88 save_compressed: true

Sparse-Llama-3.1-8B-2of4-tldr

• Loss: 1.8295

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 0
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 8
• total_eval_batch_size: 8
• optimizer: Use adamw_bnb_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 66
• num_epochs: 1.0

Training results

Framework versions

• Transformers 4.51.3
• Pytorch 2.7.0+cu126
• Datasets 3.5.1
• Tokenizers 0.21.1