1# Assuming the model is downloaded in /root/Kimi-K2-Instruct-INT4MIX
2ftllm run /root/Kimi-K2-Instruct-INT4MIX # chat
3ftllm server /root/Kimi-K2-Instruct-INT4MIX # api server (default model_name = /root/Kimi-K2-Instruct-INT4MIX, port = 8080)
optimize
single CPU
If you are using a single CPU, set the number of threads with the -t parameter (generally set to CPU core count - 2).
If the speed is extremely slow, it may be due to too many threads—consider reducing them.
for example:
ftllm server /root/Kimi-K2-Instruct-INT4MIX -t 12
multi cpu (multi numa node)
If using a multi-socket CPU machine, you need to enable CUDA + NUMA heterogeneous acceleration mode.
Set the number of threads using the environment variable FASTLLM_NUMA_THREADS (typically set to the number of cores per NUMA node - 2).
If performance is extremely slow, it may be due to excessive threads—consider reducing them.
for example:
export FASTLLM_NUMA_THREADS=12 && ftllm server /root/Kimi-K2-Instruct-INT4MIX --device cuda --moe_device numa -t 1
