See axolotl config

axolotl version: 0.4.1

yaml

1base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T 2model_type: LlamaForCausalLM 3tokenizer_type: LlamaTokenizer 4 5load_in_8bit: true 6load_in_4bit: false 7strict: false 8 9datasets: 10 - path: mhenrichsen/alpaca_2k_test 11 type: alpaca 12dataset_prepared_path: 13val_set_size: 0.05 14output_dir: ./outputs/lora-out 15 16sequence_len: 4096 17sample_packing: true 18eval_sample_packing: false 19pad_to_sequence_len: true 20 21adapter: lora 22lora_model_dir: 23lora_r: 32 24lora_alpha: 16 25lora_dropout: 0.05 26lora_target_linear: true 27lora_fan_in_fan_out: 28 29wandb_project: 30wandb_entity: 31wandb_watch: 32wandb_name: 33wandb_log_model: 34 35gradient_accumulation_steps: 4 36micro_batch_size: 2 37num_epochs: 4 38optimizer: adamw_bnb_8bit 39lr_scheduler: cosine 40learning_rate: 0.0002 41 42train_on_inputs: false 43group_by_length: false 44bf16: auto 45fp16: 46tf32: false 47 48gradient_checkpointing: true 49early_stopping_patience: 50resume_from_checkpoint: 51local_rank: 52logging_steps: 1 53xformers_attention: 54flash_attention: true 55 56warmup_steps: 10 57evals_per_epoch: 4 58saves_per_epoch: 1 59debug: 60deepspeed: 61weight_decay: 0.0 62fsdp: 63fsdp_config: 64special_tokens: 65

outputs/lora-out

• Loss: 1.2115

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Framework versions

• PEFT 0.11.1
• Transformers 4.42.3
• Pytorch 2.1.2+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1