Converted INT8/INT4 files for
fastllm with
baichuan-13b-chatbaichuan-13b-chat-int4.flm:
+-----------------------------------------------------------------------------+
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| N/A 53C P0 56W / 250W | 7083MiB / 23040MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
baichuan-13b-chat-int8.flm:
+-----------------------------------------------------------------------------+
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| N/A 51C P0 162W / 250W | 13151MiB / 23040MiB | 95% Default |
+-------------------------------+----------------------+----------------------+
1from fastllm_pytools import llm
2model = llm.model("baichuan-13b-chat-int4.flm")
3for response in model.stream_response("介绍一下南京"):
4 print(response, flush = True, end = "")
