See axolotl config

axolotl version: 0.3.0

yaml

1base_model: mistralai/Mistral-7B-v0.1 2model_type: MistralForCausalLM 3tokenizer_type: LlamaTokenizer 4is_mistral_derived_model: true 5 6load_in_8bit: true 7load_in_4bit: false 8strict: false 9 10datasets: 11 - path: datasets/norobots_150/norobots_150 12 type: completion 13 - path: datasets/separated/bloke-separate 14 type: completion 15 - path: datasets/separated/kcpp-separate 16 type: completion 17 - path: datasets/separated/kcpp-support-separate 18 type: completion 19 - path: datasets/separated/st-chat-separate 20 type: completion 21 - path: datasets/separated/exllama2_readme.txt 22 type: completion 23 - path: datasets/separated/koboldcpp_readme.txt 24 type: completion 25 - path: datasets/separated/llama_readme.txt 26 type: completion 27 - path: datasets/separated/ooba_readme.txt 28 type: completion 29 - path: datasets/separated/sillytavern_readme.txt 30 type: completion 31 - path: datasets/separated/sillytavern_simple_setup_guide.txt 32 type: completion 33 - path: datasets/transformer_article.txt 34 type: completion 35 - path: datasets/lmg_thread.txt 36 type: completion 37 38dataset_prepared_path: last_run_prepared 39val_set_size: 0.1 40output_dir: ./llmTechChat-lora 41 42adapter: lora 43lora_model_dir: 44 45chat_template: chatml 46 47sequence_len: 8192 48sample_packing: true 49pad_to_sequence_len: true 50 51lora_r: 128 52lora_alpha: 64 53lora_dropout: 0.20 54lora_target_linear: true 55lora_fan_in_fan_out: 56lora_target_modules: 57 - gate_proj 58 - down_proj 59 - up_proj 60 - q_proj 61 - v_proj 62 - k_proj 63 - o_proj 64 65wandb_project: llmTechChat 66 67gradient_accumulation_steps: 8 68micro_batch_size: 1 69num_epochs: 4 70optimizer: adamw_bnb_8bit 71lr_scheduler: cosine 72learning_rate: 0.0003 73 74train_on_inputs: false 75group_by_length: false 76bf16: true 77fp16: false 78tf32: false 79 80gradient_checkpointing: true 81early_stopping_patience: 82resume_from_checkpoint: 83local_rank: 84logging_steps: 1 85xformers_attention: 86flash_attention: true 87 88loss_watchdog_threshold: 5.0 89loss_watchdog_patience: 3 90 91warmup_steps: 10 92evals_per_epoch: 4 93eval_table_size: 94eval_table_max_new_tokens: 128 95saves_per_epoch: 1 96debug: 97deepspeed: 98weight_decay: 0.0 99fsdp: 100fsdp_config: 101special_tokens: 102 bos_token: "<s>" 103 eos_token: "</s>" 104 unk_token: "<unk>" 105 106save_safetensors: true 107

llmTechChat-lora

• Loss: 1.9365

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

• quant_method: bitsandbytes
• load_in_8bit: True
• load_in_4bit: False
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: fp4
• bnb_4bit_use_double_quant: False
• bnb_4bit_compute_dtype: float32

Training hyperparameters

• learning_rate: 0.0003
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Framework versions

• PEFT 0.7.0
• Transformers 4.37.0.dev0
• Pytorch 2.0.1+cu118
• Datasets 2.16.1
• Tokenizers 0.15.0