Views
No views yet
tokenizer.json file, which was missing in the original release.IDEA-CCNL/Randeng-T5-784M-MultiTask-Chinese model is an excellent T5-based model for various Chinese NLP tasks. However, it was released with only a spiece.model file for its tokenizer, lacking the tokenizer.json file.transformers library can generally load the tokenizer from spiece.model, this absence caused issues for environments that strictly prefer or require tokenizer.json (e.g., certain versions or implementations of the Rust tokenizers library, or other frameworks that rely on this standardized format).tokenizer.json file.IDEA-CCNL/Randeng-T5-784M-MultiTask-Chinese model files:tokenizer.json: The primary change is the inclusion of the tokenizer.json file, generated from the original spiece.model using the Python transformers library's save_pretrained() method. This ensures broader compatibility and easier loading for various applications.pytorch_model.bin or model.safetensors) themselves have not been altered in any way. This repository provides the exact same powerful pre-trained model, just with an updated tokenizer serialization format.transformers library:1from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
2
3model_name = "your-username/Randeng-T5-784M-MultiTask-Chinese-with-Tokenizer-JSON" # Replace with your actual repository name
4
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
7
8text = "你好，这是一个测试。"
9inputs = tokenizer(text, return_tensors="pt")
10outputs = model.generate(**inputs)
11print(tokenizer.decode(outputs[0], skip_special_tokens=True))tokenizer.json):1use tokenizers::Tokenizer;
2use std::error::Error;
3
4#[tokio::main]
5async fn main() -> Result<(), Box<dyn Error>> {
6 let model_id = "your-username/Randeng-T5-784M-MultiTask-Chinese-with-Tokenizer-JSON"; // Replace with your actual repository name
7
8 // The Tokenizer::from_pretrained will now find and use tokenizer.json
9 let tokenizer = Tokenizer::from_pretrained(model_id, None).await?;
10
11 let text = "你好，这是一个中文文本。";
12 let encoding = tokenizer.encode(text, true).unwrap();
13
14 println!("Original text: {}", text);
15 println!("Tokens: {:?}", encoding.get_tokens());
16 println!("IDs: {:?}", encoding.get_ids());
17
18 let decoded_text = tokenizer.decode(encoding.get_ids(), true).unwrap();
19 println!("Decoded text: {}", decoded_text);
20
21 Ok(())
22}IDEA-CCNL/Randeng-T5-784M-MultiTask-Chinese model, its training, capabilities, and benchmarks, please refer to its official repository: IDEA-CCNL/Randeng-T5-784M-MultiTask-Chinese.