See axolotl config

axolotl version: 0.4.0

yaml

1# use google/gemma-7b if you have access 2base_model: google/gemma-2b 3model_type: AutoModelForCausalLM 4tokenizer_type: AutoTokenizer 5 6load_in_8bit: false 7load_in_4bit: true 8strict: false 9 10# huggingface repo 11datasets: 12 - path: OdiaGenAIdata/culturax-odia 13 type: completion 14val_set_size: 0.1 15output_dir: ./gemma-odia-2b-pretrain 16hub_model_id: sam2ai/gemma_odia_2b 17 18adapter: qlora 19lora_r: 32 20lora_alpha: 16 21lora_dropout: 0.05 22lora_target_linear: true 23 24sequence_len: 4096 25sample_packing: true 26pad_to_sequence_len: true 27 28wandb_project: gemma-completion-2b-odia 29wandb_entity: 30wandb_watch: 31wandb_name: 32wandb_log_model: 33 34 35gradient_accumulation_steps: 3 36micro_batch_size: 2 37num_epochs: 10 38optimizer: adamw_bnb_8bit 39lr_scheduler: cosine 40learning_rate: 0.0002 41 42train_on_inputs: false 43group_by_length: false 44bf16: auto 45fp16: 46tf32: false 47 48gradient_checkpointing: true 49early_stopping_patience: 50resume_from_checkpoint: 51local_rank: 52logging_steps: 1 53xformers_attention: 54flash_attention: false 55 56warmup_ratio: 0.1 57evals_per_epoch: 4 58eval_table_size: 59eval_max_new_tokens: 128 60saves_per_epoch: 1 61debug: 62deepspeed: 63weight_decay: 0.0 64fsdp: 65fsdp_config: 66special_tokens: 67

gemma_odia_2b

• Loss: 13.3986

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 3
• total_train_batch_size: 48
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 87
• num_epochs: 10

Training results

Framework versions

• PEFT 0.9.0
• Transformers 4.40.0.dev0
• Pytorch 2.4.0.dev20240326+rocm6.0
• Datasets 2.18.0
• Tokenizers 0.15.0