See axolotl config

axolotl version: 0.10.0.dev0

yaml

1base_model: THUDM/GLM-4-32B-Base-0414 2model_type: AutoModelForCausalLM 3tokenizer_type: AutoTokenizer 4 5trust_remote_code: 6 7# wandb configuration 8wandb_project: 32b-glm4-dans-personality-engine 9wandb_watch: 10 11wandb_run_id: V1.3.0-1-4 # V{Version}-{Run Number}-{Attempt Number} 12wandb_log_model: 13 14# push checkpoints to hub 15hub_model_id: Dans-DiscountModels/32b-glm4-dans-personality-engine-v1.3.0-TestArticle-1 16# how to push checkpoints to hub 17# https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.hub_strategy 18hub_strategy: "every_save" 19# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets 20# Required to be true when used in combination with `push_dataset_to_hub` 21hf_use_auth_token: true 22 23# where to save the finished model to 24output_dir: ./32b-glm4-dans-personality-engine 25 26save_safetensors: true 27 28datasets: 29 - path: Dans-DiscountModels/pretokenization-test-4 30 ds_type: parquet 31 type: 32 33plugins: 34 - axolotl.integrations.liger.LigerPlugin 35 - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin 36liger_rope: false 37liger_rms_norm: true 38liger_glu_activation: true 39liger_fused_linear_cross_entropy: false 40cut_cross_entropy: true 41 42load_in_8bit: false 43load_in_4bit: false 44strict: false 45 46dataset_prepared_path: ./32b-glm4-dans-personality-engine-data 47val_set_size: 0.003 48 49sequence_len: 32768 50 51sample_packing: true 52eval_sample_packing: true 53 54pad_to_sequence_len: true 55 56gradient_checkpointing: unsloth 57 58gradient_accumulation_steps: 4 59micro_batch_size: 1 60 61num_epochs: 2 62 63optimizer: ademamix_8bit 64optim_args: "beta1=0.9,beta2=0.999,beta3=0.999,alpha=5" 65 66lr_scheduler: rex 67learning_rate: 0.000008 68cosine_min_lr_ratio: 69 70weight_decay: 0 71 72max_grad_norm: 0.001 73 74train_on_inputs: false 75group_by_length: false 76 77bf16: true 78fp16: false 79tf32: false 80 81early_stopping_patience: 82 83resume_from_checkpoint: 84auto_resume_from_checkpoints: false 85 86local_rank: 87logging_steps: 1 88xformers_attention: 89flash_attention: true 90 91warmup_ratio: 0.1 92 93evals_per_epoch: 24 94eval_table_size: 95eval_max_new_tokens: 96 97saves_per_epoch: 8 98save_total_limit: 1 99 100debug: false 101 102deepspeed: /alloc/pocketdoc/axolotl/deepspeed_configs/zero3_bf16.json 103 104fsdp: 105fsdp_config: 106 107special_tokens: 108

32b-glm4-dans-personality-engine-v1.3.0-TestArticle-1

• Loss: 1.6235

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 8e-06
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Use ademamix_8bit and the args are: beta1=0.9,beta2=0.999,beta3=0.999,alpha=5
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 332
• num_epochs: 2.0

Training results

Framework versions

• Transformers 4.51.3
• Pytorch 2.4.1+cu121
• Datasets 3.5.0
• Tokenizers 0.21.1