| Model | GGML quantize method | HDD size | 1 token* |
|---|---|---|---|
| chatglm3-ggml-f16.bin | f16 | 12.5 GB | 189ms |
| * ms/token (CPU @ Platinum 8260) from reference |
pip install chatglm-cpp transformerswget https://huggingface.co/npc0/chatglm3-6b-fp16/resolve/main/chatglm3-ggml-f16.bin1import chatglm_cpp
2
3pipeline = chatglm_cpp.Pipeline("./chatglm3-ggml-f16.bin")
4pipeline.chat([chatglm_cpp.ChatMessage(role="user", content="你好")])
5# Output: ChatMessage(role="assistant", content="你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。", tool_calls=[])