See axolotl config

axolotl version: 0.4.0

yaml

1base_model: premai-io/prem-1B 2model_type: LlamaForCausalLM 3tokenizer_type: AutoTokenizer 4 5load_in_8bit: false 6load_in_4bit: false 7strict: false 8 9datasets: 10 - path: argilla/distilabel-capybara-dpo-7k-binarized 11 type: orpo.chat_template 12dataset_prepared_path: last_run_prepared 13val_set_size: 0.001 14output_dir: ./prem-1B-32k 15save_safetensors: true 16sequence_len: 8192 17sample_packing: false 18pad_to_sequence_len: false 19use_pose: true 20pose_max_context_len: 262144 21min_sample_len: 6144 22pose_num_chunks: 16 23curriculum_sampling: true 24 25overrides_of_model_config: 26 rope_theta: 500000.0 27 max_position_embeddings: 262144 28 29 # peft_use_dora: true 30adapter: lora 31peft_use_rslora: true 32lora_model_dir: 33lora_r: 1024 34lora_alpha: 1024 35lora_dropout: 0.1 36lora_target_modules: 37 - q_proj 38 - k_proj 39 - v_proj 40 - o_proj 41lora_modules_to_save: 42 - embed_tokens 43 - lm_head 44 45wandb_project: 46wandb_entity: 47wandb_watch: 48wandb_name: 49wandb_log_model: 50 51gradient_accumulation_steps: 8 52micro_batch_size: 1 53num_epochs: 20 54optimizer: adamw_bnb_8bit 55lr_scheduler: cosine 56learning_rate: 0.00001 57max_grad_norm: 1.0 58adam_beta2: 0.95 59 60train_on_inputs: false 61group_by_length: false 62bf16: true 63fp16: 64tf32: false 65 66gradient_checkpointing: true 67gradient_checkpointing_kwargs: 68 use_reentrant: true 69early_stopping_patience: 70resume_from_checkpoint: 71local_rank: 72logging_steps: 1 73xformers_attention: 74flash_attention: true 75sdp_attention: 76s2_attention: 77 78warmup_steps: 10 79evals_per_epoch: 8 80saves_per_epoch: 8 81debug: 82deepspeed: 83weight_decay: 0.0 84fsdp: 85fsdp_config: 86special_tokens: 87 pad_token: <|end_of_text|> 88 89 90

prem-1B-32k

• Loss: 1.0059

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 20

Training results

Framework versions

• PEFT 0.10.0
• Transformers 4.40.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.15.0
• Tokenizers 0.15.0