shisa-base-7b-v1 takes Mistral 7B and adds an additional 8B tokens of primarily Japanese pre-training. Japanese tokens were sourced from MADLAD-400, using DSIR, along with 10% English tokens sampled from a mix of MADLAD-400 EN and various open datasources added in to prevent catastrophic forgetting.
We have extended the Mistral tokenizer to 120k tokens to improve Japanese efficiency. Our tokenizer achieves ~2.3 characters per token in JA, versus the base Mistral 7B tokenizer which is <1 character per token. Code for our implementation is available in our Shisa repo.
This base model was created for use with Shisa 7B, our JA/EN fine-tuned model, but we provide it for the community as we believe the combination of strong performance and efficient bilingual tokenizer could be useful.
Training took 2,400 A100-40 GPU hours on a single 16 x A100-40 machine with DeepSpeed ZeRO-3. (WandB logs)
Performance
This base model was able to attain class-leading Japanese performance in standardized benchmarks with significantly less additional pre-training than previously released models. We speculate this may be due to the use of a better-curated pre-training dataset, but ablations at even 2.5B additional JA tokens still showed very strong Japanese performance.
We used a slightly modified llm-jp-eval (our base model requires a bos_token to be prepended to the prompt; we tested other models with and without the modification and took the higher results for all models tested). Here we validate versus the original Mistral 7B base model as well as Japanese Stable LM Instruct Gamma 7B, which is a Mistral 7B base with an additional 100B tokens of JA/EN pre-training. We also include Japanese-StableLM-Base-Beta-70B, which is a Llama 2 70B that also has an additional 100B tokens of JA/EN pre-training as a reference:
Mistral llm-jp-eval Comparison
Here we also compare shisa-base-7b-v1 to other recently-released similar classed (7B parameter) Japanese-tuned models. ELYZA 7B fast model and Youri 7B are Llama 2 7B models with 18B and 40B of additional pre-training respectively, and CALM2-7B and llm-jp-13b are pretrained models with 1.3T and 300B JA/EN tokens of pre-training:
7B llm-jp-eval Performance
Tokenizer
As mentioned in the introduction, our tokenizer is an extended version of the Mistral 7B tokenizer, with a vocab size of 120073 and aligned to 120128 for better performance. The remaining unused tokens are assigned as zero-weighted <|extra_{idx}|> tokens.
We use the "Fast" tokenizer, which should be the default for AutoTokenizer, but if you have problems, make sure to check tokenizer.is_fast or to initialize with use_fast=True.
Japanese efficiency from sampling 50K items (~85M characters) from the JA subset of the CulturaX dataset:
We also test English efficiency using a sampling of 50K items (~177M characters) from the EN subset of the CulturaX dataset as a sanity check (and to see how other tokenizers fare):
With our extended tokenizer, we are able to achieve class-leading JA token efficiency without any losses in EN performance vs the base tokenizer. This bears out in our testing, and we often see >2X JA inference speedups with our tokenizer.
