Views
No views yet
google/gemma-4-12B-it quantized to MXFP4 (W4A16) using GPTQ with special techniques that preserve Korean and English performance simultaneously.| Property | Value |
|---|---|
| Base model | google/gemma-4-12B-it (BF16, ~24.4 GB) |
| Quantization format | MXFP4 W4A16 (compressed-tensors) |
| Disk size | 7.9 GB (68% reduction from BF16) |
| VRAM (model weights) | ~8 GB |
| Quantized scope | Decoder Linear layers (lm_head, embed_vision, embed_audio kept in BF16) |
| Calibration data | BCCard on-policy dataset (Korean/English split) |
The 12B checkpoint is thegemma4_unifiedarchitecture variant — its vision stack (embed_vision.*) differs from the 31B'svision_tower, and is kept at original precision.
1# Standard environment (CUDA 12.9+):
2vllm serve gemma-4-12B-it-MXFP4 \
3 --port 8000 \
4 --max-model-len 32768 \
5 --gpu-memory-utilization 0.90
6
7# Blackwell with CUDA < 12.9:
8VLLM_USE_FLASHINFER_SAMPLER=0 \
9vllm serve gemma-4-12B-it-MXFP4 \
10 --port 8000 \
11 --linear-backend marlin \
12 --max-model-len 32768 \
13 --gpu-memory-utilization 0.90| Setting | Purpose |
|---|---|
VLLM_USE_FLASHINFER_SAMPLER=0 | Disables FlashInfer sampler (uses native PyTorch) |
--linear-backend marlin | Uses pre-compiled MXFP4 GEMM kernels (no JIT) |
| Component | Backend | JIT Required |
|---|---|---|
| MXFP4 GEMM | Marlin (pre-compiled) | No |
| Attention | TRITON_ATTN | No (Triton JIT, not nvcc) |
| Sampling | Native PyTorch | No |
| Task | MXFP4 (this model) | BF16 baseline | Recovery |
|---|---|---|---|
| HellaSwag (English) | 83.5% | 81.0% | 103.1% |
| KoBEST-HellaSwag | 62.0% | 64.0% | 96.9% |
| KoBEST-COPA | 98.5% | 98.5% | 100.0% |
| KoBEST-BoolQ | 94.0% | 93.0% | 101.1% |
| KoBEST-SentiNeg | 99.0% | 99.5% | 99.5% |
| KoBEST-WiC | 82.0% | 79.0% | 103.8% |
| KMMLU (5 subjects) | 55.5% | 60.5% | 91.7% |
| BC-Finance QA (char-F1) | 0.350 | 0.344 | 101.7% |
| Average recovery | 99.7% | ||
| Korean Generation | 495 tok, rep 1.00, Hangul 58.8% | 492 tok, rep 1.00, Hangul 58.3% | — |
| English Generation | 430 tok, rep 0.99 | 432 tok, rep 0.99 | — |
| Model | Params | Disk | Score | Benchmark (setting/source) |
|---|---|---|---|---|
| A.X 4.0 (SKT) | 72B | ~144 GB | 78.3 | KMMLU, vendor |
| EXAONE 4.0 (LG) | 32B | ~64 GB | 75.2 | KMMLU, as reported in the HyperCLOVA X 32B Think report |
| GPT-4o (OpenAI) | — | — (API) | 72.5 | KMMLU, as reported by SKT A.X |
| HyperCLOVA X 32B Think (NAVER) | 32B | ~64 GB | 71.3 | KMMLU, vendor report |
| A.X 3.1 (SKT) | 34B | ~68 GB | 69.2 | KMMLU, vendor |
| A.X 3.1 Light (SKT) | 7B | ~14 GB | 61.7 | KMMLU, vendor |
| Gemma 4 12B IT (BF16 baseline) | 12B | ~24.4 GB | 60.5 | KMMLU 5-subject sample, 0-shot chat MCQ (this harness) |
| This model (Gemma 4 12B MXFP4) | 12B | 7.9 GB | 55.5 | KMMLU 5-subject sample, 0-shot chat MCQ (this harness) |
| Gemma 3 27B IT (Google) | 27B | ~54 GB | 54.0 | KMMLU-Redux, KMMLU-Redux/Pro paper |
| HyperCLOVA X SEED (NAVER) | 3B | ~6.5 GB | 48.5 | KMMLU, vendor |
| Gemma 3 12B IT (Google) | 12B | ~24 GB | 46.7 | KMMLU-Redux, KMMLU-Redux/Pro paper |
Disk sizes for third-party models are approximate BF16 weight footprints (params × 2 bytes); this repo's model and its BF16 baseline are measured on-disk values.
| Format | Disk Size | vs BF16 | Notes |
|---|---|---|---|
| BF16 (original) | ~24.4 GB | 100% | google/gemma-4-12B-it |
| FP8-dynamic | 13 GB | ~53% | community quantization with the same llm-compressor FP8_DYNAMIC recipe (no official RedHatAI 12B release) |
| This Model (MXFP4) | 7.9 GB | 32% | W4A16 GPTQ + MHR + PSA |
@misc{gemma4-12b-mxfp4-mhr-psa,
title={Gemma 4 12B IT MXFP4},
author={Taeyoung Lee},
year={2026},
url={https://huggingface.co/BCCard/MoAI-gemma-4-12B-it-mxfp4}
}