You can deploy this model and start earning money today!

alegsandyr/
gemma_3_12B_qkv512_ff1620_q4_g32_full_submission

Name: gemma_3_12B_qkv512_ff1620_q4_g32_full_submission
Availability: InStock

safetensors

gemma3_text

8-bit

Views

No views yet

WMT26 Model Compression Submission

This submission runs the Czech-to-German compressed Gemma 3 12B system.

Default Hugging Face repository for this self-contained submission:

alegsandyr/gemma_3_12B_qkv512_ff1620_q4_g32_full_submission

The best locally validated quality-first q4 checkpoint from the latest experiments is:

outputs/models/gemma_3_12B_qkv512_ff1620_q4_g32_gptq_calib1024_infprompt

It was quantized from:

alegsandyr/gemma_3_12B_qkv512_ff1620

This checkpoint uses all 1024 calibration examples with calibration_field=inference_prompt. The Hugging Face repository is intended to contain these weights at the repository root together with this submission runner.

The checkpoint uses a custom SVD-factorized format with factor quantization. The submission bundle includes the minimal loader needed to instantiate SVDLinear = up(down(x)), wrap quantized factors when quantization_config.json is present, and run Hugging Face generation.

Setup

bash setup.sh

The evaluator may also preinstall dependencies from requirements.txt.

Run

Official entrypoint:

bash

bash run.sh \
  --lang-pair ces-deu \
  --batch-size 8 \
  --input input.txt \
  --output output.txt

The script writes exactly one translation line for each input line. Logs and progress messages go to stderr.

The official --batch-size argument is accepted. To reduce OOM risk with long WMT inputs, the runner internally caps the effective micro-batch size to MAX_EFFECTIVE_BATCH_SIZE (default 1). Set MAX_EFFECTIVE_BATCH_SIZE=8 if the evaluation GPU has enough memory and full batching is desired.

Model Location

By default, run.sh first checks whether model weights are present next to run.sh. In a clone/snapshot of the self-contained Hugging Face repository, it loads the local repository root directly. If weights are not present next to the script, it falls back to:

alegsandyr/gemma_3_12B_qkv512_ff1620_q4_g32_full_submission

For local smoke tests before upload, set:

MODEL_DIR=outputs/models/gemma_3_12B_qkv512_ff1620_q4_g32_gptq_calib1024_infprompt bash run.sh ...

Smaller local experiments were also tested:

outputs/models/gemma_3_12B_qkv512_ff1620_q4_g64_gptq_calib1024_infprompt is about 4.7G, but smoke tests showed occasional English fragments.
outputs/models/gemma_3_12B_qkv512_ff1620_q4_g48_gptq_calib1024_infprompt is about 4.8G, but smoke tests showed word artifacts.
outputs/models/gemma_3_12B_qkv512_ff1620_q4_g128_gptq_calib128 is about 4.5G, but quality dropped more noticeably.

The model repository is expected to contain:

text

config.json
tokenizer files
model.safetensors or model.safetensors.index.json with shards
compression_config.json
quantization_config.json

Notes

Supported language pair: ces-deu / cs-de.
Default precision: bf16.
Default decoding: greedy, max_new_tokens=256.
Quantized SVD factors are expanded once at load time by default (DEQUANTIZE_SVD_ON_LOAD=1) to avoid repeated per-forward dequantization overhead.
On a single H100, the default path loads the model on CUDA when available. Set DEVICE_MAP=auto to use Accelerate dispatch.