I quantized the base model using compressed tensorers + big calibration dataset and re-added MTP weights from the original Qwen model as they were missing in heretic model.
Two token MTP prediction seems to work fine with decent acceptance rate. I have included compression script, vllm inference script and MTP copy script in experts. To get
VLLM working I had to disable cuda graph compilation and install conch-triton-kernels. After this I get about 15tps peak generation speed on my NVIDIA Thor dev kit. If you
get various exceptions, take a look at inference script and copy various tuning parameters.
