Since we have convrot int4 and int8, why not mix it up? Int4 kinda sucks, int8 is kinda slow.
But it's a piece of shit and it doesn't follow the profile.. plus it casts things to half and BF16 blindly. Bad job. Here's a fix.
https://huggingface.co/Lockout/krea2-comfy-int4-mixed/blob/main/star_model_converter_pro.pyIt's tested on krea2. But I'm sure someone will come up with a new and exciting way to make some int8/int4 mixed models with trial and error.
That's really all there is to it. Kinda int4 speeds at quality somewhere between int8 and int4. Less body horrors.
I will upload some quants I converted too.
bonus: Script to upcast int8 to BF16 in case they don't release the full model.
https://huggingface.co/Lockout/krea2-comfy-int4-mixed/blob/main/glm-dequant.py
Must be run from the comfy main folder.
2080ti with sage attention and compiled:
8/8 [00:13<00:00, 1.63s/it]
Prompt executed in 21.91 seconds
8/8 [00:07<00:00, 1.11it/s]
Prompt executed in 7.58 seconds
8/8 [00:07<00:00, 1.11it/s]
Prompt executed in 7.58 seconds
vs 9.5s for pure int8 and 6.5s for pure int4.
Slightly bigger model 9.6-10.3gb, less body horrors.
Kung fu the search continues. Tried some approaches to take metrics. You capture int8 activations first
and then you point metrics scripts at them. Either by mean error or RMSE scores.
python claude_scorer_v3.py
--model "/gonzalomoKrea2_v10.safetensors"
--activations-dir "/outputs/star_activations/capture1"
--out ./profiles/krea_real_activation_v3.json
--device cuda --mean-ceiling 0.01 --max-ceiling 0.15
python glm_score_rmse.py
--model "/gonzalomoKrea2_v10.safetensors"
--activations-dir "/outputs/star_activations/capture1"
--out ./profiles/krea_real_rmse4.json
--device cuda
--rmse-ceiling 0.4
Hard to find a metric that corresponds to visual quality.
