Suzume - a Japanese tree sparrow
Suzume
This Suzume 8B, a Japanese finetune of Llama 3.
Llama 3 has exhibited excellent performance on many English language benchmarks.
However, it also seemingly been finetuned on mostly English data, meaning that it will respond in English, even if prompted in Japanese.
We have fine-tuned Llama 3 on almost 3,000 Japanese conversations meaning that this model has the smarts of Llama 3 but has the added ability to chat in Japanese.
Please feel free to comment on this model and give us feedback in the Community tab!
How to use
You can use the GGUF using LM Studio
LM Studioで簡単に使えます！
こちら は使い方を説明します。
LM Studioで「lightblue/suzume-llama-3-8B-japanese-gguf」を検索して下さい。
Evaluation scores
We find that this is the best performing model in the 7/8B class of LLMs on a multitude of Japanese language benchmarks.
image/png Training data
We train on three sources of data to create this model
megagonlabs/instruction_ja - 669 conversations
A hand-edited dataset of nearly 700 conversations taken originally from translations of the kunishou/hh-rlhf-49k-ja dataset.
openchat/openchat_sharegpt4_dataset (Japanese conversations only) - 167 conversations
Conversations taken from humans talking to GPT-4
lightblue/tagengo-gpt4 (Japanese prompts only) (Link coming soon!) - 2,482 conversations
Almost 2,500 diverse Japanese prompts sampled from lmsys/lmsys-chat-1m and then used to prompt gpt-4-0125-preview
Training config
See axolotl config
axolotl version: 0.4.0
1 base_model : meta - llama/Meta - Llama - 3 - 8B - Instruct
2 model_type : LlamaForCausalLM
3 tokenizer_type : AutoTokenizer # PreTrainedTokenizerFast
4
5 load_in_8bit : false
6 load_in_4bit : false
7 strict : false
8
9 datasets :
10 - path : /workspace/llm_training/axolotl/llama3 - ja/openchat_megagon_lbgpt4_ja.json
11 ds_type : json # see other options below
12 type : sharegpt
13 conversation : llama - 3
14 dataset_prepared_path : /workspace/llm_training/axolotl/llama3 - ja/prepared_openchat_megagon_lbgpt4_ja
15 val_set_size : 0.01
16 output_dir : /workspace/llm_training/axolotl/llama3 - ja/output_openchat_megagon_lbgpt4_ja_8B_instruct
17
18 sequence_len : 8192
19 sample_packing : true
20 pad_to_sequence_len : true
21 eval_sample_packing : False
22
23 use_wandb : true
24 wandb_project : axolotl
25 wandb_entity : peterd
26 wandb_name : openchat_megagon_lbgpt4_ja_8B_instruct
27
28 gradient_accumulation_steps : 2
29 micro_batch_size : 2
30 num_epochs : 1
31 optimizer : paged_adamw_8bit
32 lr_scheduler : cosine
33 learning_rate : 1e-5
34
35 train_on_inputs : false
36 group_by_length : false
37 bf16 : auto
38 fp16 :
39 tf32 : false
40
41 gradient_checkpointing : true
42 gradient_checkpointing_kwargs :
43 use_reentrant : false
44 early_stopping_patience :
45 resume_from_checkpoint :
46 logging_steps : 1
47 xformers_attention :
48 flash_attention : true
49
50 warmup_steps : 10
51 evals_per_epoch : 5
52 eval_table_size :
53 saves_per_epoch : 1
54 debug :
55 deepspeed : /workspace/axolotl/deepspeed_configs/zero2.json
56 weight_decay : 0.0
57 special_tokens :
58 pad_token : < | end_of_text | >
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 1e-05
train_batch_size: 2
eval_batch_size: 2
seed: 42
distributed_type: multi-GPU
num_devices: 3
gradient_accumulation_steps: 2
total_train_batch_size: 12
total_eval_batch_size: 6
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 10
num_epochs: 1
Training results
Training Loss Epoch Step Validation Loss 1.303 0.08 1 1.2664 1.4231 0.23 3 1.2409 1.1007 0.46 6 1.0264 1.0635 0.69 9 1.0154 1.0221 0.92 12 0.9555
Framework versions
Transformers 4.40.0.dev0
Pytorch 2.2.1+cu121
Datasets 2.18.0
Tokenizers 0.15.0
How to cite
Please cite
this paper when referencing this model.
1 @article { devine2024tagengo,
2 title= { Tagengo: A Multilingual Chat Dataset } ,
3 author= { Devine, Peter } ,
4 journal= { arXiv preprint arXiv:2405.12612 } ,
5 year= { 2024 }
6 }
Developer
