from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("marcusmi4n/abeja-qwen2.5-7b-japanese-qnn", subfolder="quantized_simple")
tokenizer = AutoTokenizer.from_pretrained("marcusmi4n/abeja-qwen2.5-7b-japanese-qnn", subfolder="quantized_simple")

# Japanese text generation
inputs = tokenizer("こんにちは、私は", return_tensors="pt")
outputs = model.generate(**inputs, max_length=100, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

ONNX Inference

python

import onnxruntime as ort

# Load ONNX model
session = ort.InferenceSession("marcusmi4n/abeja-qwen2.5-7b-japanese-qnn/onnx/prefill/model.onnx")
# Run inference...

QNN Deployment

bash

# Deploy to Snapdragon device
adb push marcusmi4n/abeja-qwen2.5-7b-japanese-qnn/qnn_compiled/ /data/local/tmp/qnn_model/
# Use QNN runtime for inference

Performance

Quantization: 75% size reduction
Speed: 2-3x faster inference
Memory: ~4.5GB RAM usage
Tokens/sec: 8-15 tokens/sec on Snapdragon 8cx Gen 2+

Hardware Compatibility

✅ Snapdragon 8cx Gen 2+
✅ Snapdragon 8cx Gen 3
✅ Snapdragon 8 Gen 1+
✅ Windows on ARM devices
✅ Microsoft Surface Pro X
✅ Dell Latitude 7420

Files Structure

marcusmi4n/abeja-qwen2.5-7b-japanese-qnn/
├── quantized_simple/          # 4-bit quantized PyTorch model
│   ├── model.safetensors
│   ├── config.json
│   ├── tokenizer.json
│   └── model_info.json
├── onnx/                      # ONNX models
│   ├── prefill/model.onnx
│   └── token_gen/model.onnx
├── quantized_onnx/            # Quantized ONNX models
│   ├── prefill/model_quantized.onnx
│   └── token_gen/model_quantized.onnx
├── qnn_compiled/              # QNN compiled models
│   ├── prefill/
│   ├── token_gen/
│   └── deployment_info.json
└── README.md                  # This file

License

Apache 2.0 - Same as base ABEJA Qwen 2.5 model

Citation

bibtex

@misc{abeja-qwen25-qnn,
  title={ABEJA Qwen 2.5 7B Japanese - QNN Optimized},
  author={QNN Conversion Pipeline},
  year={2025},
  url={https://huggingface.co/marcusmi4n/abeja-qwen2.5-7b-japanese-qnn}
}

Base Model Citation

Please cite the original ABEJA Qwen 2.5 paper:

bibtex

@article{abeja-qwen2.5,
  title={ABEJA Qwen 2.5: Japanese Language Model},
  author={ABEJA Inc.},
  journal={arXiv preprint},
  year={2024}
}