See axolotl config

axolotl version: 0.4.0

yaml

1base_model: TheBloke/SOLAR-10.7B-Instruct-v1.0-uncensored-GPTQ 2is_llama_derived_model: false 3gptq: true 4gptq_disable_exllama: true 5model_type: AutoModelForCausalLM 6tokenizer_type: LlamaTokenizer 7tokenizer_use_fast: true 8tokenizer_legacy: true 9load_in_8bit: false 10load_in_4bit: false 11strict: false 12push_dataset_to_hub: 13hf_use_auth_token: true 14datasets: 15 - path: datasets_cleansinng/datasets/helper_selector_1280_0305_v01.jsonl #Path to json dataset file in huggingface 16 #for type,conversation arguments read axolotl readme and pick what is suited for your project, I wanted a chatbot and put sharegpt and chatml 17 type: 18 system_prompt: "Instruction에 따라 적절하게 Input 데이터를 활용하여 Output 답변을 하세요. 너는 사용자 질문(Instruction)에 실시간으로 API 호출을 위한 Json 형식의 구조화된 결과를 생성하는 인공지능이야." 19 format: "[INST]### Instruction:\n{instruction}\n\n### Input:{input}\n\n[/INST]### Output: " 20 no_input_format: "[INST]### Instruction:\n{instruction}\n\n[/INST]### Output: " 21 field_instruction: Instruction 22 field_input: Input 23 field_output: Output 24dataset_prepared_path: 25val_set_size: 0.05 26adapter: lora 27lora_model_dir: 28sequence_len: 4096 29sample_packing: 30lora_r: 32 31lora_alpha: 32 32lora_dropout: 0.05 33lora_target_modules: 34 - k_proj 35 - o_proj 36 - q_proj 37 - v_proj 38lora_target_linear: 39lora_fan_in_fan_out: 40wandb_project: 41wandb_watch: 42wandb_name: 43wandb_log_model: 44output_dir: ./output_solor/exp_16 45gradient_accumulation_steps: 8 46micro_batch_size: 8 47num_epochs: 5 48optimizer: adamw_torch 49adam_beta2: 0.95 50adam_eps: 0.00001 51max_grad_norm: 1.0 52torchdistx_path: 53lr_scheduler: cosine 54lr_quadratic_warmup: true 55learning_rate: 0.0005 56train_on_inputs: false 57group_by_length: false 58bf16: false 59fp16: false 60float16: true 61tf32: true 62gradient_checkpointing: true 63early_stopping_patience: 64resume_from_checkpoint: 65local_rank: 66logging_steps: 1 67xformers_attention: 68flash_attention: 69sdp_attention: 70flash_optimum: 71warmup_steps: 100 72evals_per_epoch: 4 73saves_per_epoch: 1 74debug: 75deepspeed: deepspeed_configs/zero1.json 76weight_decay: 0.1 77special_tokens: 78 bos_token: "<s>" 79 eos_token: "</s>" 80 unk_token: "<unk>" 81

output_solor/exp_16

• Loss: 0.2015

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0005
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 5

Training results

Framework versions

• PEFT 0.9.1.dev0
• Transformers 4.38.0.dev0
• Pytorch 2.0.1+cu117
• Datasets 2.18.0
• Tokenizers 0.15.0