See axolotl config

axolotl version: 0.4.1

yaml

1adapter: lora 2base_model: Qwen/Qwen2-1.5B-Instruct 3batch_size: 8 4bf16: true 5chat_template: tokenizer_default_fallback_alpaca 6datasets: 7- data_files: 8 - 19637e66dc3ec99a_train_data.json 9 ds_type: json 10 format: custom 11 path: /workspace/input_data/19637e66dc3ec99a_train_data.json 12 type: 13 field_instruction: drugName 14 field_output: review 15 format: '{instruction}' 16 no_input_format: '{instruction}' 17 system_format: '{system}' 18 system_prompt: '' 19early_stopping_patience: 3 20eval_steps: 50 21flash_attention: true 22gpu_memory_limit: 80GiB 23gradient_checkpointing: true 24group_by_length: true 25hub_model_id: willtensora/0eda4152-e58c-4e24-b30e-71e456fb3b24 26hub_strategy: checkpoint 27learning_rate: 0.0002 28logging_steps: 10 29lora_alpha: 256 30lora_dropout: 0.1 31lora_r: 128 32lora_target_linear: true 33lr_scheduler: cosine 34micro_batch_size: 1 35model_type: AutoModelForCausalLM 36num_epochs: 100 37optimizer: adamw_bnb_8bit 38output_dir: miner_id_24 39pad_to_sequence_len: true 40resize_token_embeddings_to_32x: false 41sample_packing: false 42save_steps: 50 43sequence_len: 2048 44tokenizer_type: Qwen2TokenizerFast 45train_on_inputs: false 46trust_remote_code: true 47val_set_size: 0.1 48wandb_entity: '' 49wandb_mode: online 50wandb_project: Gradients-On-Demand 51wandb_run: your_name 52wandb_runid: default 53warmup_ratio: 0.05 54xformers_attention: true 55

0eda4152-e58c-4e24-b30e-71e456fb3b24

• Loss: 2.4073

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 8
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 15107
• num_epochs: 100

Training results

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1