The hybrid quant employs different quantization levels on a per layer basis to enable
both high performance and small file size at the same time. This quant is sized at ~Q4_K_M bpw. The quants employed are all K to avoid
slow CPU or older GPU processing of IQ quants. For this file the Q4_K_H layer quants are as follows:
The quant was optimized for strong reasoning performance across a curated set of test prompts.
Usage:
The unique feature of phi-4 is strong reasoning without having used RL methods in the instruct training. Instead it was pre trained on a highly curated
set of (apparently) very high quality data (i.e. strong textbooks) to help provide an inherent strong reasoning ability. In tests it does perform extremely well
on reasoning, exhibiting muc higher than typical "common sense", with none of the nauseous and inefficient overthinking endemic of most RL models.
The layer quants for this model were evaluated on a set of test/eval prompts using greedy sampling. The quant/model show extremely good performance on reasoning problems.
Its mostly useless for code prompts however, even though it doesn't score terribly bad on code evals it cannot reliable generate working code on even simple tasks.
The model can be speculated using Qwen2.5 0.5B Instruct if the inference engine can support dynamic vocab translation between
draft and target models. Approx gen performance using a downstream speculator with llama.cpp on a 4070:
