See axolotl config

axolotl version: 0.4.0

yaml

1adapter: lora 2base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T 3bf16: auto 4dataset_prepared_path: null 5datasets: 6- path: joseagmz/MedQnA_version3 7 type: context_qa.load_v2 8debug: null 9deepspeed: null 10early_stopping_patience: null 11evals_per_epoch: 4 12flash_attention: true 13fp16: null 14fsdp: null 15fsdp_config: null 16gradient_accumulation_steps: 4 17gradient_checkpointing: true 18group_by_length: false 19is_llama_derived_model: true 20learning_rate: 0.0002 21load_in_4bit: false 22load_in_8bit: true 23local_rank: null 24logging_steps: 1 25lora_alpha: 16 26lora_dropout: 0.05 27lora_fan_in_fan_out: null 28lora_model_dir: null 29lora_r: 32 30lora_target_linear: true 31lr_scheduler: cosine 32micro_batch_size: 2 33model_type: LlamaForCausalLM 34num_epochs: 4 35optimizer: adamw_bnb_8bit 36output_dir: ./lora_test 37pad_to_sequence_len: true 38resume_from_checkpoint: null 39sample_packing: true 40saves_per_epoch: 1 41sequence_len: 4096 42special_tokens: null 43strict: false 44tf32: false 45tokenizer_type: LlamaTokenizer 46train_on_inputs: false 47val_set_size: 0.05 48wandb_entity: null 49wandb_log_model: null 50wandb_name: null 51wandb_project: null 52wandb_watch: null 53warmup_steps: 10 54weight_decay: 0.0 55xformers_attention: null 56

lora_test

• Loss: 0.7337

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Framework versions

• PEFT 0.8.2
• Transformers 4.38.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.17.1
• Tokenizers 0.15.0