Views
No views yet
transformers and ctranslate2 is installed.
In the interactive python shell:# Import the transformers module
import transformers
# Import the AutoTokenizer submodule (so you won't run into NameError: name 'AutoTokenizer' is not defined)
from transformers import AutoTokenizer
# Load tokenizer from existing files
tokenizer = AutoTokenizer.from_pretrained("<source_model_dir>")
# Save the tokenizer data as tokenizer.json
tokenizer.save_pretrained("<output_dir>", legacy_format=False)ct2-transformers-converter --model <model_dir> --copy_files tokenizer.json preprocessor_config.json --output_dir <output_dir> --quantization float32