Koto Small 7B (Instruct-Tuned)

Usage

Chat template

<|im_start|>system
system prompt<|im_end|>
<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant

Samplers

Datasets

Acknowledgements

• Thank you very much to Delta-Vector/Mango for providing the compute used to train this model.
• Fizz for the pretrain.
• Pocketdoc/Anthracite for da cool datasets.
• Hensen chat.
• Thank you to the illustrator of WataNare for drawing the art used in the model card!
• Thanks to Curse for testing, ideas.
• Thanks to Toasty for some data, ideas.
• Thanks to everyone else in allura!

Call for Help

Technical Appendix

Training Notes

Same as before, It was trained over the course of 12 hours for over 2 epochs, on an 8xA100 DGX node, Using Ademamix and REX LR schedular, High grad-clipping was used for regularization with NO WEIGHTDECAY because it sucks.

WandB

Axolotl Config

yaml

1# ============================================================================= 2# Model + Saving 3# ============================================================================= 4base_model: allura-forge/Koto-Small-7b-rc1 5output_dir: ./koto-sft 6saves_per_epoch: 2 7deepcompile: true 8# ============================================================================= 9# DATASET CONFIGURATION 10# ============================================================================= 11datasets: 12 - path: /home/Ubuntu/Mango/pretok/test-koto-sft-7b-rc-1.parquet 13 ds_type: parquet 14 type: 15 16shuffle_merged_datasets: true 17dataset_prepared_path: ./dataset_prepared 18train_on_inputs: false 19 20# ============================================================================= 21# EVALUATION SETTINGS 22# ============================================================================= 23#evals_per_epoch: 4 24#eval_table_size: 25#eval_max_new_tokens: 128 26#eval_sample_packing: false 27val_set_size: 0.0 28 29# ============================================================================= 30# MEMORY OPTIMIZATION 31# ============================================================================= 32plugins: 33 - axolotl.integrations.liger.LigerPlugin 34 - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin 35liger_rope: true 36liger_rms_norm: true 37liger_layer_norm: true 38liger_glu_activation: true 39liger_fused_linear_cross_entropy: false 40cut_cross_entropy: true 41sample_packing: true 42pad_to_sequence_len: true 43gradient_checkpointing: true 44flash_attention: true 45 46# ============================================================================= 47# MULTI-GPU TRAINING 48# ============================================================================= 49deepspeed: ./deepspeed_configs/zero2.json 50 51# ============================================================================= 52# LOGGING & MONITORING 53# ============================================================================= 54wandb_project: Koto-Small 55wandb_entity: 56wandb_watch: 57wandb_name: sft 58wandb_log_model: 59logging_steps: 1 60debug: false 61 62# ============================================================================= 63# TRAINING PARAMETERS 64# ============================================================================= 65micro_batch_size: 6 66gradient_accumulation_steps: 2 67num_epochs: 2 68sequence_len: 16000 69optimizer: paged_ademamix_8bit 70lr_scheduler: rex 71learning_rate: 8e-6 72warmup_ratio: 0.1 73max_grad_norm: 0.0001 74weight_decay: 0.0 75 76 77# ============================================================================= 78# ADDITIONAL SETTINGS 79# ============================================================================= 80local_rank: 81group_by_length: false 82early_stopping_patience: 83save_safetensors: true 84bf16: auto 85special_tokens: