python ./quantize_nvfp4.py --model TheDrummer/Magidonia-24B-v4.2.0 --output ./Magidonia-24B-v4.2.0-NVFP4 --size 4096 --seed 42 --ultra_chat 3 --c4_en 2 --fiction_v8 2

Quantization Evals

Base Magidonia

|    Tasks     |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------------|------:|------|-----:|---------------|---|-----:|---|------|
|arc_challenge |      1|none  |     0|acc            |↑  |0.6237|±  |0.0142|
|              |       |none  |     0|acc_norm       |↑  |0.6604|±  |0.0138|
|hellaswag     |      1|none  |     0|acc            |↑  |0.6368|±  |0.0048|
|              |       |none  |     0|acc_norm       |↑  |0.8301|±  |0.0037|
|lambada_openai|      1|none  |     0|acc            |↑  |0.7023|±  |0.0064|
|              |       |none  |     0|perplexity     |↓  |3.2404|±  |0.0705|
|wikitext      |      2|none  |     0|bits_per_byte  |↓  |0.5641|±  |   N/A|
|              |       |none  |     0|byte_perplexity|↓  |1.4785|±  |   N/A|
|              |       |none  |     0|word_perplexity|↓  |8.0935|±  |   N/A|
|winogrande    |      1|none  |     0|acc            |↑  |0.7727|±  |0.0118|

NVFP4 Quant

|    Tasks     |Version|Filter|n-shot|    Metric     |   |Value |   |Stderr|
|--------------|------:|------|-----:|---------------|---|-----:|---|------|
|arc_challenge |      1|none  |     0|acc            |↑  |0.6101|±  |0.0143|
|              |       |none  |     0|acc_norm       |↑  |0.6263|±  |0.0141|
|hellaswag     |      1|none  |     0|acc            |↑  |0.6303|±  |0.0048|
|              |       |none  |     0|acc_norm       |↑  |0.8241|±  |0.0038|
|lambada_openai|      1|none  |     0|acc            |↑  |0.7002|±  |0.0064|
|              |       |none  |     0|perplexity     |↓  |3.3804|±  |0.0785|
|wikitext      |      2|none  |     0|bits_per_byte  |↓  |0.5919|±  |   N/A|
|              |       |none  |     0|byte_perplexity|↓  |1.5072|±  |   N/A|
|              |       |none  |     0|word_perplexity|↓  |8.9689|±  |   N/A|
|winogrande    |      1|none  |     0|acc            |↑  |0.7632|±  |0.0119|

Bias, Risks, and Limitations

This is already a creative fine-tune. It was quantized with that usecase in mind. Probably not gonna pass any leet-coder challenges with this one.

How To Use

bash
vllm serve ealexeev/The-Drummer-Magidonia-24B-v4.2.0-NVFP4 \
    --tensor-parallel-size 1 \      # 1 GPU
    --gpu-memory-utilization 0.8 \  # Else it will take it all for KV