shisa-ai/

030-swallow-8b-0.5-base-v2new-sft

So just one of the additional fun experiments we're doing on the path to Shisa V2.1.

This is an SFT of the tokyotech-llm/Llama-3.1-Swallow-8B-v0.5 base model.

This came out from a curiousity to see if a well-curated CPT can beat simply using strong Instruct models as base models in Japanese... tentatively yes (we haven't run on our full suite or analyzed English capabilities) and if our Shisa V2 datasets are better than the Swallow v0.5 Instruct approach (knowledge distill of Gemma 3 27B) - also, tentatively yes.

Anyway, throwing this model out there so we can cheekily reclaim SOTA as running the latest Shisa V2 SFT dataset (which we'll publish with official V2.1 releases) manages to edge out the (very strong, congrats!) tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.5 on our GPT-4.1 judged Shaberi results:

Note: the Swallow team has released this under the Llama 3.3 Community License and Gemma Terms of Use and as a derived work we inherit the model. Please carefully read and respect the licenses if they apply to you.

Compute sponsored by Hot Aisle and AMD.