This is a 4-bit AWQ quantized version of
sbintuitions/sarashina2-7b.
Quantization was done through Activation-aware Weight Quantization (AWQ) with the Japanese Wikipedea dataset (range3/wikipedia-ja-20230101).
1from awq import AutoAWQForCausalLM
2from transformers import AutoTokenizer
3
4model_path = "ronantakizawa/sarashina2-7b-awq"
5
6# Load quantized model
7model = AutoAWQForCausalLM.from_quantized(
8 model_path,
9 fuse_layers=True,
10 device_map="auto"
11)
12tokenizer = AutoTokenizer.from_pretrained(model_path)
13
14# Generate text
15prompt = "おはようございます、今日の天気は"
16inputs = tokenizer(prompt, return_tensors="pt")
17inputs = {k: v.to(model.device) for k, v in inputs.items()}
18
19outputs = model.generate(
20 **inputs,
21 max_new_tokens=50,
22 do_sample=True,
23 temperature=0.7,
24 top_p=0.95,
25)
26
27print(tokenizer.decode(outputs[0], skip_special_tokens=True))
1@misc{sarashina2-7b-awq,
2 author = {Ronan Takizawa},
3 title = {Sarashina2-7B AWQ 4-bit Quantized},
4 year = {2025},
5 publisher = {Hugging Face},
6 howpublished = {\url{https://huggingface.co/ronantakizawa/sarashina2-7b-awq}}
7}
Please refer to the
original model card for the base model citation.
