See axolotl config

axolotl version: 0.7.0

yaml

1base_model: NousResearch/Meta-Llama-3.1-8B-Instruct 2model_type: LlamaForCausalLM 3tokenizer_type: AutoTokenizer 4hub_model_id: bkciccar/llama-3.1-8b-instruct-culture-lora 5 6load_in_8bit: false 7load_in_4bit: false 8strict: false 9 10datasets: 11 - path: json 12 data_files: culture_shuffle.jsonl 13 split: train 14 type: alpaca 15 16test_datasets: 17 - path: json 18 data_files: 19 - culturalbench-hard-preprocessed.jsonl 20 split: train 21 type: 22 system_prompt: Below is a question with a potential answer. Please respond with only 'true' or 'false'. 23 field_system: 24 field_instruction: prompt_question 25 field_input: prompt_option 26 field_output: answer 27 format: |- 28 ### Question: 29 {instruction} 30 31 ### Option: 32 {input} 33 34 ### Response (true or false): 35 36dataset_prepared_path: 37output_dir: /scratch/bkciccar/outputs/llama-3.1-8b-lora 38 39sequence_len: 4096 40sample_packing: true 41eval_sample_packing: true 42pad_to_sequence_len: true 43 44adapter: lora 45lora_model_dir: 46lora_r: 32 47lora_alpha: 16 48lora_dropout: 0.05 49lora_target_linear: true 50lora_fan_in_fan_out: false 51lora_modules_to_save: 52 - embed_tokens 53 - lm_head 54 55wandb_project: "CultureBank_FineTuning" 56wandb_entity: "bcicc" 57wandb_watch: 58wandb_name: "Lora_FineTuning_Run_02" 59wandb_log_model: 60 61gradient_accumulation_steps: 4 62micro_batch_size: 2 63eval_batch_size: 10 64num_epochs: 4 65optimizer: paged_adamw_8bit 66lr_scheduler: cosine 67learning_rate: 2e-5 68 69train_on_inputs: false 70group_by_length: false 71bf16: auto 72fp16: 73tf32: false 74 75gradient_checkpointing: true 76early_stopping_patience: 77resume_from_checkpoint: 78local_rank: 79logging_steps: 1 80xformers_attention: 81flash_attention: true 82s2_attention: 83 84warmup_steps: 10 85eval_table_size: 86eval_max_new_tokens: 5 87evals_per_epoch: 16 88saves_per_epoch: 4 89debug: 90deepspeed: 91weight_decay: 0.0 92fsdp: 93fsdp_config: 94special_tokens: 95 pad_token: <|end_of_text|> 96 97overrides_of_trainer_kwargs: 98 compute_metrics: "custom_metrics.compute_metrics" 99

llama-3.1-8b-instruct-culture-lora

• Loss: 2.5461

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 10
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4.0

Training results

Framework versions

• PEFT 0.14.0
• Transformers 4.48.3
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0