See axolotl config

axolotl version: 0.4.0

yaml

1base_model: codellama/CodeLlama-7b-hf 2base_model_config: codellama/CodeLlama-7b-hf 3model_type: LlamaForCausalLM 4tokenizer_type: LlamaTokenizer 5is_llama_derived_model: true 6hub_model_id: EvolCodeLlama-7b 7 8load_in_8bit: false 9load_in_4bit: true 10strict: false 11 12datasets: 13 - path: ptoro/Evol-Instruct-Python-1k-testing 14 type: alpaca 15dataset_prepared_path: last_run_prepared 16val_set_size: 0.02 17output_dir: ./qlora-out 18 19adapter: qlora 20lora_model_dir: 21 22sequence_len: 2048 23sample_packing: true 24 25lora_r: 32 26lora_alpha: 16 27lora_dropout: 0.05 28lora_target_modules: 29lora_target_linear: true 30lora_fan_in_fan_out: 31 32wandb_project: axolotl 33wandb_entity: 34wandb_watch: 35wandb_run_id: 36wandb_log_model: 37 38gradient_accumulation_steps: 4 39micro_batch_size: 2 40num_epochs: 3 41optimizer: paged_adamw_32bit 42lr_scheduler: cosine 43learning_rate: 0.0002 44 45train_on_inputs: false 46group_by_length: false 47bf16: true 48fp16: false 49tf32: false 50 51gradient_checkpointing: true 52early_stopping_patience: 53resume_from_checkpoint: 54local_rank: 55logging_steps: 1 56xformers_attention: 57flash_attention: true 58 59warmup_steps: 100 60eval_steps: 0.01 61save_strategy: epoch 62save_steps: 63debug: 64deepspeed: 65weight_decay: 0.0 66fsdp: 67fsdp_config: 68special_tokens: 69 bos_token: "<s>" 70 eos_token: "</s>" 71 unk_token: "<unk>"

EvolCodeLlama-7b

• Loss: 0.3828

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 3

Training results

Framework versions

• PEFT 0.7.2.dev0
• Transformers 4.37.0
• Pytorch 2.0.1+cu118
• Datasets 2.16.1
• Tokenizers 0.15.0