See axolotl config

axolotl version: 0.4.0

yaml

1base_model: /media/renfroe/llms/SmolLM-360M/ 2 3model_type: LlamaForCausalLM 4tokenizer_type: GPT2Tokenizer 5seed: 122887 6load_in_8bit: false 7load_in_4bit: false 8strict: false 9 10max_steps: 0 11resume_from_checkpoint: 12datasets: 13 - path: /home/renfroe/Desktop/sqa_tiny-llama_dataset/Dynamic_Optimization_Methods_with_Applications_sqa_answers_only.json 14 type: 15 field_instruction: question 16 field_output: answer 17 format: "<|im_start|>user\n{instruction}<|im_end|>\n<|im_start|>assistant\n" 18 no_input_format: "<|im_start|>user\n{instruction}<|im_end|>\n<|im_start|>assistant\n" 19 - path: /home/renfroe/Dev/tinyllama-models/dataset/open_hermes_top_tech.json 20 type: sharegpt 21 - path: /home/renfroe/Desktop/sqa_tiny-llama_dataset/hermes_prior_knowledge_question_expansion_with_answers.json 22 type: 23 field_instruction: question 24 field_output: answer 25 format: "<|im_start|>user\n{instruction}<|im_end|>\n<|im_start|>assistant\n" 26 no_input_format: "<|im_start|>user\n{instruction}<|im_end|>\n<|im_start|>assistant\n" 27 - path: /home/renfroe/Desktop/sqa_tiny-llama_dataset/hermes_prior_knowledge_question_expansion_with_answers.json 28 type: 29 field_instruction: question 30 field_output: answer 31 format: "<|im_start|>user\n{instruction}<|im_end|>\n<|im_start|>assistant\n" 32 no_input_format: "<|im_start|>user\n{instruction}<|im_end|>\n<|im_start|>assistant\n" 33 - path: /home/renfroe/Desktop/sqa_tiny-llama_dataset/or-farm_sharegpt.json 34 type: sharegpt 35 36 37 38dataset_prepared_path: 39val_set_size: 0.2 40output_dir: ./SmolLM-Ora 41auto_resume_from_checkpoints: false 42 43sequence_len: 2048 44sample_packing: true 45chat_template: chatml 46 47wandb_project: SmolLM-Ora 48wandb_entity: 49wandb_watch: 50wandb_name: 51wandb_log_model: 52 53gradient_accumulation_steps: 1 54micro_batch_size: 10 55num_epochs: 4 56optimizer: adamw_bnb_8bit 57lr_scheduler: linear 58weight_decay: 0.0000001 59learning_rate: 0.0001 60lr_scheduler_kwargs: 61 # num_cycles: 3 62 63train_on_inputs: false 64group_by_length: false 65bf16: auto 66fp16: 67tf32: false 68 69gradient_checkpointing: true 70early_stopping_patience: 71resume_from_checkpoint: 72local_rank: 73logging_steps: 1 74xformers_attention: 75flash_attention: true 76 77eval_sample_packing: False 78 79warmup_steps: 50 80evals_per_epoch: 4 81eval_table_size: 82saves_per_epoch: 4 83debug: 84deepspeed: 85 86fsdp: 87fsdp_config: 88special_tokens: 89 bos_token: "<|endoftext|>" 90 eos_token: "<|endoftext|>" 91 pad_token: "<|endoftext|>"

SmolLM-Ora

• Loss: 0.8298

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0001
• train_batch_size: 10
• eval_batch_size: 10
• seed: 122887
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 50
• num_epochs: 4

Training results

Framework versions

• Transformers 4.40.0.dev0
• Pytorch 2.0.1+cu117
• Datasets 2.18.0
• Tokenizers 0.15.0