See axolotl config

axolotl version: 0.6.0

yaml

1base_model: meta-llama/Meta-Llama-3.1-8B 2model_type: AutoModelForCausalLM 3tokenizer_type: AutoTokenizer 4load_in_8bit: false 5load_in_4bit: true 6strict: false 7datasets: 8- path: tatsu-lab/alpaca 9 type: alpaca 10 format: csv 11 prompt_template: '### Instruction: {instruction} 12 13 ### Input: {input} 14 15 ### Response: {output}' 16dataset_prepared_path: null 17val_set_size: 0.1 18output_dir: /root/outputs/fine_tuned_model 19adapter: qlora 20lora_model_dir: null 21sequence_len: 2048 22sample_packing: true 23eval_sample_packing: false 24pad_to_sequence_len: true 25lora_r: 16 26lora_alpha: 8 27lora_dropout: 0.05 28lora_target_modules: null 29lora_target_linear: true 30lora_fan_in_fan_out: null 31wandb_project: null 32wandb_entity: null 33wandb_watch: null 34wandb_name: null 35wandb_log_model: null 36gradient_accumulation_steps: 4 37micro_batch_size: 1 38num_epochs: 10 39max_steps: 10000000 40optimizer: paged_adamw_32bit 41lr_scheduler: cosine 42learning_rate: 0.0002 43train_on_inputs: false 44group_by_length: false 45bf16: auto 46fp16: null 47tf32: false 48gradient_checkpointing: true 49early_stopping_patience: 3 50save_strategy: steps 51save_steps: 20 52evaluation_strategy: steps 53eval_steps: 20 54load_best_model_at_end: true 55save_total_limit: 3 56metric_for_best_model: loss 57greater_is_better: false 58resume_from_checkpoint: null 59local_rank: null 60logging_steps: 1 61xformers_attention: null 62flash_attention: true 63warmup_steps: 10 64debug: null 65deepspeed: null 66weight_decay: 0.0 67fsdp: null 68fsdp_config: null 69special_tokens: 70 pad_token: <|end_of_text|> 71mlflow_tracking_uri: https://mlflow-dev.qpiai-pro.tech 72mlflow_experiment_name: llama-8B-medical-alpaca 73hf_mlflow_log_artifacts: 'true' 74local_files_only: true 75

root/outputs/fine_tuned_model

• Loss: 2.0808

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Use OptimizerNames.PAGED_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 5950

Training results

Framework versions

• PEFT 0.14.0
• Transformers 4.47.0
• Pytorch 2.3.1+cu121
• Datasets 3.1.0
• Tokenizers 0.21.0