See axolotl config

axolotl version: 0.4.1

yaml

1adapter: lora 2base_model: unsloth/gemma-2b-it 3batch_size: 8 4bf16: true 5chat_template: tokenizer_default_fallback_alpaca 6datasets: 7- data_files: 8 - 88cfea977fe74782_train_data.json 9 ds_type: json 10 format: custom 11 path: /workspace/input_data/88cfea977fe74782_train_data.json 12 type: 13 field_instruction: smiles 14 field_output: molt5 15 format: '{instruction}' 16 no_input_format: '{instruction}' 17 system_format: '{system}' 18 system_prompt: '' 19evals_per_epoch: 1 20flash_attention: true 21gpu_memory_limit: 80GiB 22gradient_checkpointing: true 23group_by_length: true 24hub_model_id: willtensora/f0d6caa9-89a9-4666-9a6d-c8cda2015281 25hub_strategy: checkpoint 26learning_rate: 0.0002 27logging_steps: 10 28lora_alpha: 256 29lora_dropout: 0.1 30lora_r: 128 31lora_target_linear: true 32lr_scheduler: cosine 33micro_batch_size: 1 34model_type: AutoModelForCausalLM 35num_epochs: 100 36optimizer: adamw_bnb_8bit 37output_dir: miner_id_24 38pad_to_sequence_len: true 39resize_token_embeddings_to_32x: false 40sample_packing: false 41saves_per_epoch: 2 42sequence_len: 2048 43tokenizer_type: GemmaTokenizerFast 44train_on_inputs: false 45trust_remote_code: true 46val_set_size: 0.1 47wandb_entity: '' 48wandb_mode: online 49wandb_project: Gradients-On-Demand 50wandb_run: your_name 51wandb_runid: default 52warmup_ratio: 0.05 53xformers_attention: true 54

f0d6caa9-89a9-4666-9a6d-c8cda2015281

• Loss: 1.9427

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 8
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 12
• num_epochs: 100

Training results

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1