See axolotl config

axolotl version: 0.4.1

yaml

1adapter: qlora 2base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T 3bf16: false 4dataset_prepared_path: null 5datasets: 6- ds_tipe: json 7 path: pubmed_continual_pretraning_dataset.jsonl 8 type: completion 9debug: null 10deepspeed: null 11early_stopping_patience: null 12eval_sample_packing: false 13evals_per_epoch: 4 14flash_attention: false 15fp16: null 16fsdp: null 17fsdp_config: null 18gradient_accumulation_steps: 4 19gradient_checkpointing: true 20group_by_length: false 21learning_rate: 0.0002 22load_in_4bit: true 23load_in_8bit: false 24local_rank: null 25logging_steps: 1 26lora_alpha: 16 27lora_dropout: 0.05 28lora_fan_in_fan_out: null 29lora_model_dir: null 30lora_r: 32 31lora_target_linear: true 32lora_target_modules: null 33lr_scheduler: cosine 34micro_batch_size: 8 35model_type: LlamaForCausalLM 36num_epochs: 4 37optimizer: paged_adamw_32bit 38output_dir: ./outputs/qlora-out 39pad_to_sequence_len: false 40resume_from_checkpoint: null 41sample_packing: false 42saves_per_epoch: 1 43sequence_len: 4096 44special_tokens: null 45strict: false 46tf32: false 47tokenizer_type: LlamaTokenizer 48train_on_inputs: false 49val_set_size: 0.05 50wandb_entity: null 51wandb_log_model: null 52wandb_name: null 53wandb_project: null 54wandb_watch: null 55warmup_steps: 10 56weight_decay: 0.0 57xformers_attention: null 58

outputs/qlora-out

• Loss: 1.7613

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.1.2+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1