This repository represents an effort to graft and re-train an MTP layer for
DavidAU's Qwen3.6-40B model, inspired by
Piehsoft's untrained MTP grafts.
Currently only offering q8_0 quantizations, but this will eventually expand once MTP retraining is finalized.
Version 1 (complete)
A simple retraining attempt that used the following procedure:
- Ran the 40B model on 117 random prompts
- Record what it was thinking at every position by introspecting llama.cpp
- Specifically, recorded which 64 tokens it thought were most likely next and how confident it was in each.
- Note: Substituted
h_t with embedding(input_token) to move forward with the data that was immediately accessible.
- Use the recorded data to retrain the MTP layer: align what the MTP's predictions with reality
- This data did not include hidden states, but since the trained head is shown real-world verifier logprobs during training its bias toward the most common tokens is effectively re-calibrated.
Compared to Piehsoft's untrained MTP head, this methodology
- Significantly improved acceptances rates in structured text / coding scenarios
- Provided little to no impact on high-entropy scenarios such as creative writing / casual chat.
Ultimately proved that improving acceptance rates via retraining was possible. Further improvements seem to require:
- Larger corpus (dataset)
- Higher text entropy within corpus
- Most importantly, data collection of the verifier hidden states
Version 2 (in-progress)
EDIT: Training using hidden states ended up generating significantly worse results than the methodology used in v1. Alternative paths are being considered, but this may potentially end as a stalemate.
Training is in-progress, which builds on-top of the v1. This version likely won't be a final version, but rather will be used to help steer direction. The main changes will be:
Collect data using a modified version of llama.cpp that exposes data from the 40B model's hidden states (h_t)Use a dataset with high entropy: WildChat-1M
Not training on the full dataset (would take a long time), will use a subset of randomly selected samples.
Increase epochs from 3 to 5 (and tweak a few other training settings)
