Llama-3 8B Self-Instruct: PEFT Edition

See axolotl config

axolotl version: 0.4.0

yaml

1base_model: llama3-8B 2model_type: LlamaForCausalLM 3tokenizer_type: AutoTokenizer 4 5load_in_8bit: false 6load_in_4bit: true 7strict: false 8 9datasets: 10 - path: llama-3-8b-self-align-data-generation-results/sanitized.jsonl 11 ds_type: json 12 type: 13 system_prompt: "You are an exceptionally intelligent coding assistant that consistently delivers accurate and reliable responses to user instructions." 14 field_system: system 15 field_instruction: instruction 16 field_output: response 17 format: "### Instruction:\n{instruction}\n\n### Response:\n" 18 no_input_format: "### Instruction:\n{instruction}\n\n### Response:\n" 19dataset_prepared_path: 20val_set_size: 0.05 21 22sequence_len: 2048 23sample_packing: true 24pad_to_sequence_len: true 25 26adapter: qlora 27save_safetensors: true 28lora_model_dir: 29lora_r: 64 30lora_alpha: 32 31lora_dropout: 0.05 32lora_target_linear: true 33lora_fan_in_fan_out: 34 35log_with: None 36wandb_project: llama-3-8b-self-align-axolotl 37wandb_entity: 38wandb_watch: 39wandb_name: qlora-prince-hps-promptfix 40 41output_dir: qlora_decrease_lr_promptfix 42wandb_log_model: 43 44gradient_accumulation_steps: 8 45micro_batch_size: 2 46num_epochs: 4 47optimizer: paged_adamw_32bit 48lr_scheduler: cosine 49learning_rate: 2e-5 50 51train_on_inputs: false 52group_by_length: false 53bf16: auto 54fp16: 55tf32: false 56 57gradient_checkpointing: true 58gradient_checkpointing_kwargs: 59 use_reentrant: false 60early_stopping_patience: 61resume_from_checkpoint: 62local_rank: 63logging_steps: 1 64xformers_attention: 65flash_attention: true 66 67warmup_steps: 100 68evals_per_epoch: 8 69eval_table_size: 70eval_max_new_tokens: 128 71saves_per_epoch: 2 72debug: 73deepspeed: 74weight_decay: 0.0 75fsdp: 76 - full_shard 77 - auto_wrap 78fsdp_config: 79 fsdp_limit_all_gathers: true 80 fsdp_sync_module_states: false 81 fsdp_offload_params: false 82 fsdp_use_orig_params: false 83 fsdp_cpu_ram_efficient_loading: false 84 fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP 85 fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer 86 fsdp_state_dict_type: FULL_STATE_DICT 87 fsdp_sharding_strategy: FULL_SHARD 88special_tokens: 89 eos_token: "<|im_end|>" 90 pad_token: "<|end_of_text|>" 91tokens: 92 - "<|im_start|>" 93 - "<|im_end|>" 94lora_modules_to_save: 95 - embed_tokens 96 - lm_head

qlora_decrease_lr_promptfix

• Loss: 0.4121

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 4

Training results

Framework versions

• PEFT 0.11.1
• Transformers 4.42.0.dev0
• Pytorch 2.3.0+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1