This repository hosts all models from the Vibrato(
url) GitHub release.
Models that were compressed using zstd have already been decompressed, so they are ready for direct download and use.
For those seeking models under the BSD license, you can visit the following repository (
ryan-minato/vibrato-models-bsdonly).
1from huggingface_hub import hf_hub_download
2import vibrato
3
4# Load tokenizer from `.cache/hf`
5model_path = hf_hub_download("ryan-minato/vibrato-models", "<<model_name>>/system.dic")
6with open(model_path, "rb") as f:
7 tokenizer = vibrato.Vibrato(f.read())
8
9text = """\
10「四十二だと!」ルーンクォールが叫んだ。
11「七百五十万年かけて、それだけか?」
12「何度も徹底的に検算しました」コンピュータが応じた。
13「まちがいなくそれが答えです。率直なところ、みなさんのほうで究極の疑問が何であるかわかっていなかったところに問題があるのです」
14"""
15
16tokenizer.tokenize(text)
