• Model creator: https://huggingface.co/Dans-DiscountModels/
• Original model: https://huggingface.co/Dans-DiscountModels/Dans-Instruct-Mix-8b-ChatML-V0.2.0/

Original model description:

• axolotl
• generated_from_trainer model-index:
• name: Dans-L3.1-Test results: []

---

See axolotl config

axolotl version: 0.4.1

yaml

1base_model: Dans-DiscountModels/Meta-Llama-3.1-8B-ChatML 2model_type: AutoModelForCausalLM 3tokenizer_type: AutoTokenizer 4 5trust_remote_code: 6 7# wandb configuration 8wandb_project: l3.1-8b-dans-instruct 9wandb_watch: 10wandb_run_id: attempt-03 11wandb_log_model: 12 13# push checkpoints to hub 14hub_model_id: anthracite-core/Dans-L3.1-Test 15# how to push checkpoints to hub 16# https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.hub_strategy 17hub_strategy: "all_checkpoints" 18# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets 19# Required to be true when used in combination with `push_dataset_to_hub` 20hf_use_auth_token: true 21 22# where to save the finished model to 23output_dir: ./l3.1-8b-dans-instruct 24 25# dataset settings (local or huggingface repo) 26datasets: 27 - path: PocketDoc/Dans-MemoryCore-CoreCurriculum-Small 28 type: dan-chat 29 - path: AquaV/Energetic-Materials-Sharegpt 30 type: dan-chat 31 - path: AquaV/Chemical-Biological-Safety-Applications-Sharegpt 32 type: dan-chat 33 - path: AquaV/US-Army-Survival-Sharegpt 34 type: dan-chat 35 - path: AquaV/Resistance-Sharegpt 36 type: dan-chat 37 - path: AquaV/Interrogation-Sharegpt 38 type: dan-chat 39 - path: AquaV/Multi-Environment-Operations-Sharegpt 40 type: dan-chat 41 - path: PocketDoc/Dans-Mathmaxx 42 type: dan-chat 43 - path: PJMixers/Math-Multiturn-1K-ShareGPT 44 type: dan-chat 45 - path: PocketDoc/Dans-Benchmaxx 46 type: dan-chat 47 - path: PocketDoc/Dans-Codemaxx-LeetCode 48 type: dan-chat 49 - path: PocketDoc/Dans-Codemaxx-CodeFeedback-Conversations 50 type: dan-chat 51 - path: PocketDoc/Dans-Codemaxx-CodeFeedback-SingleTurn 52 type: dan-chat 53 - path: PocketDoc/Dans-Taskmaxx 54 type: dan-chat 55 - path: PocketDoc/Dans-Taskmaxx-DataPrepper 56 type: dan-chat 57 - path: PocketDoc/Dans-Taskmaxx-ConcurrentQA-Reworked 58 type: dan-chat 59 - path: PocketDoc/Dans-Toolmaxx-Agent 60 type: dan-chat 61 - path: PocketDoc/Dans-Toolmaxx-ShellCommands 62 type: dan-chat 63 - path: PocketDoc/Dans-ASCIIMaxx-Wordart 64 type: dan-chat 65 - path: PocketDoc/Dans-Prosemaxx-Gutenberg 66 type: dan-chat 67 - path: PocketDoc/Dans-Prosemaxx-Cowriter-XS 68 type: dan-chat 69 - path: PocketDoc/Dans-Prosemaxx-Adventure 70 type: dan-chat 71 - path: PocketDoc/Dans-Prosemaxx-Opus-Writing 72 type: dan-chat 73 - path: PocketDoc/Dans-Assistantmaxx-Sharegpt 74 type: dan-chat 75 - path: PocketDoc/Dans-Assistantmaxx-OpenAssistant2 76 type: dan-chat 77 - path: PocketDoc/Dans-Assistantmaxx-Opus-instruct-1 78 type: dan-chat 79 - path: PocketDoc/Dans-Assistantmaxx-Opus-instruct-2 80 type: dan-chat 81 - path: PocketDoc/Dans-Assistantmaxx-Opus-instruct-3 82 type: dan-chat 83 - path: PocketDoc/Dans-Assistantmaxx-NoRobots 84 type: dan-chat 85 - path: PocketDoc/Dans-Personamaxx 86 type: dan-chat 87 - path: PocketDoc/DansTestYard 88 type: completion 89 90chat_template: chatml 91 92plugins: 93 - axolotl.integrations.liger.LigerPlugin 94liger_rope: true 95liger_rms_norm: true 96liger_swiglu: true 97liger_fused_linear_cross_entropy: true 98 99load_in_8bit: false 100load_in_4bit: false 101strict: false 102 103dataset_prepared_path: ./l3.1-8b-dans-instruct-data 104val_set_size: 0.0 105 106sequence_len: 8192 107 108sample_packing: true 109eval_sample_packing: false 110 111pad_to_sequence_len: true 112 113gradient_checkpointing: true 114gradient_checkpointing_kwargs: 115 use_reentrant: false 116 117gradient_accumulation_steps: 2 118micro_batch_size: 2 119 120num_epochs: 3 121 122optimizer: adamw_torch 123 124lr_scheduler: cosine 125learning_rate: 0.0000015 126cosine_min_lr_ratio: 127 128adam_beta1: 0.9 129adam_beta2: 0.95 130adam_epsilon: 0.00000001 131 132weight_decay: 0.01 133 134max_grad_norm: 20 135 136train_on_inputs: false 137group_by_length: true 138 139bf16: true 140fp16: false 141tf32: false 142 143early_stopping_patience: 144 145resume_from_checkpoint: 146auto_resume_from_checkpoints: 147 148local_rank: 149logging_steps: 1 150xformers_attention: 151flash_attention: true 152 153warmup_ratio: 0.1 154saves_per_epoch: 2 155 156debug: false 157 158deepspeed: deepspeed_configs/zero2.json 159 160fsdp: 161fsdp_config: 162 163special_tokens: 164 pad_token: <|finetune_right_pad_id|> 165 eos_token: <|im_end|>

Dans-L3.1-Test

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 1.5e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 230
• num_epochs: 3

Training results

Framework versions

• Transformers 4.45.0.dev0
• Pytorch 2.3.1+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1