See axolotl config

axolotl version: 0.8.0.dev0

yaml

1adapter: lora 2base_model: Qwen/QwQ-32B-Preview 3trust_remote_code: true 4bf16: true 5dataset_processes: 64 6datasets: 7- path: phxdev/creed 8 type: completion 9 field: text 10 trust_remote_code: false 11 streaming: true 12gradient_accumulation_steps: 1 13gradient_checkpointing: true 14gradient_checkpointing_kwargs: 15 use_reentrant: false 16learning_rate: 0.001 17lisa_layers_attribute: model.layers 18lisa_enabled: true 19lisa_layers_fraction: 0.25 20load_best_model_at_end: true 21load_in_4bit: false 22load_in_8bit: true 23lora_alpha: 128 24lora_dropout: 0.15 25lora_r: 64 26lora_target_modules: 27- q_proj 28- v_proj 29- k_proj 30- o_proj 31- gate_proj 32- down_proj 33- up_proj 34lora_fan_in_fan_out: false 35modules_to_save: 36- embed_tokens 37- lm_head 38loraplus_lr_embedding: 1.0e-06 39loraplus_lr_ratio: 16 40lr_scheduler: cosine_with_min_lr 41lr_scheduler_kwargs: 42 min_lr: 0.00001 43max_prompt_len: 1024 44mean_resizing_embeddings: false 45micro_batch_size: 1 46num_epochs: 3.0 47optimizer: adamw_torch 48# optim_args: 49# weight_decay: 0.05 50# betas: [0.9, 0.95] 51# eps: 1.0e-8 52output_dir: ./outputs/heisenberg-crystal 53pretrain_multipack_attn: true 54pretrain_multipack_buffer_size: 20000 55qlora_sharded_model_loading: false 56ray_num_workers: 1 57resources_per_worker: 58 GPU: 1 59resume_from_checkpoint: null 60sample_packing: false 61sample_packing_bin_size: 200 62sample_packing_group_size: 100000 63sample_packing_seq_len_multiplier: 1.0 64save_only_model: true 65save_safetensors: true 66save_strategy: steps 67save_steps: 100 68save_total_limit: 3 69eval_strategy: steps 70eval_steps: 100 71metric_for_best_model: loss 72greater_is_better: false 73sequence_len: 512 74shuffle_merged_datasets: true 75skip_prepare_dataset: false 76strict: false 77train_on_inputs: false 78neftune_noise_alpha: 5.0 79model_config: 80 rope_scaling: 81 type: linear 82 factor: 1.5 83dataloader_prefetch_factor: 4 84dataloader_num_workers: 8 85dataloader_pin_memory: true 86dataloader_persistent_workers: true 87max_grad_norm: 1.0 88adam_beta2_schedule: cosine 89torch_compile: true 90torch_compile_backend: inductor 91trl: 92 log_completions: true 93 ref_model_mixup_alpha: 0.9 94 ref_model_sync_steps: 64 95 sync_ref_model: false 96 use_vllm: false 97 vllm_device: auto 98 vllm_dtype: auto 99 vllm_gpu_memory_utilization: 0.9 100use_ray: false 101val_set_size: 0.05 102warmup_steps: 100 103warmup_ratio: 0.0 104weight_decay: 0.05 105flash_attention: true 106flash_attn_cross_entropy: true 107flash_attn_rms_norm: true 108flash_attn_fuse_qkv: false 109flash_attn_fuse_mlp: false 110ddp_backend: nccl 111ddp_broadcast_buffers: false 112ddp_find_unused_parameters: false 113tf32: true 114bf16_full_eval: false 115fp16: false 116# unfrozen_parameters: 117# - lm_head.* 118# - embed_tokens.* 119# - norm.* 120xformers_attention: false 121s2_attention: false 122sdp_attention: false 123pad_to_sequence_len: true 124peft_use_dora: false 125peft_lora_modules_to_save: null 126special_tokens: 127 pad_token: <|endoftext|> 128deepspeed: null 129fsdp: null 130fsdp_config: null 131# wandb_project: heisenberg-qwen 132# wandb_entity: null 133# wandb_name: blue-crystal-run 134# wandb_log_model: checkpoint 135hub_model_id: null 136hub_strategy: null 137report_to: [] 138logging_strategy: steps 139logging_steps: 10 140logging_first_step: true

outputs/heisenberg-crystal

• Loss: nan

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine_with_min_lr
• lr_scheduler_warmup_steps: 100
• num_epochs: 3.0

Training results

Framework versions

• PEFT 0.14.0
• Transformers 4.49.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0