See axolotl config

axolotl version: 0.9.2

yaml

1base_model: pretrained_models/Spark-TTS-0.5B/LLM 2# Automatically upload checkpoint and final model to HF 3hub_model_id: muhtasham/spark-llm-finetune-tj 4 5trust_remote_code: true 6 7strict: false 8 9datasets: 10 - path: data/output_prompt.jsonl 11 type: completion 12 13dataset_prepared_path: 14val_set_size: 0.05 15output_dir: ./outputs/out 16 17 18sequence_len: 4098 19sample_packing: true 20eval_sample_packing: true 21pad_to_sequence_len: true 22 23wandb_project: spark-tts 24wandb_entity: 25wandb_watch: 26wandb_name: 27wandb_log_model: 28 29gradient_accumulation_steps: 4 30micro_batch_size: 4 31num_epochs: 50 32optimizer: adamw_torch_fused 33lr_scheduler: cosine 34learning_rate: 0.0002 35 36train_on_inputs: false 37group_by_length: false 38bf16: auto 39fp16: 40tf32: true 41 42gradient_checkpointing: true 43gradient_checkpointing_kwargs: 44 use_reentrant: false 45early_stopping_patience: 46resume_from_checkpoint: 47local_rank: 48logging_steps: 50 49xformers_attention: 50flash_attention: true 51 52warmup_steps: 10 53evals_per_epoch: 1 54save_steps: 5000 55debug: 56deepspeed: 57weight_decay: 0.0 58

spark-llm-finetune-tj

• Loss: 5.2546

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 50.0

Training results

Framework versions

• Transformers 4.51.3
• Pytorch 2.7.1+cu126
• Datasets 3.5.1
• Tokenizers 0.21.1