You may notice the sidebar shows "Model size: 4B params" — this is not accurate and is a known limitation of HuggingFace's automatic parameter counter for EXL3 models.

This model has the full ~12B parameters of the original PLLuM-12B-chat-2512.

Why the discrepancy? EXL3 uses trellis-encoded weight packing, where each weight tensor is compressed into a .trellis sub-tensor with a different shape than the original weight matrix (e.g. a [14336, 5120] weight becomes a [320, 896, 64] int16 trellis tensor). HuggingFace's parameter counter sums raw tensor element counts from the safetensors header, which — for trellis-packed formats — does not correspond to the actual number of model parameters. This affects all EXL3 models on HuggingFace, not just this one.

Ground truth parameter count: 12B (verify via config.json: hidden_size=5120, num_hidden_layers=40, intermediate_size=14336, vocab_size=131072 — matching the original PLLuM-12B-chat-2512 architecture exactly).

📦 Quantization Specification

Parameter	Value
Format	EXL3 (ExLlamaV3)
BPW (decoder)	4.5
BPW (head)	6.0
Model Size	7.4 GB
SQNR (average)	~33 dB
Quantization Tool	exllamav3 v0.0.43
Quantization GPU	NVIDIA L4 24 GB (AWS g6.2xlarge)
Quantization Time	~75 min
Calibration	Default (250 rows × 2048 columns)

Quantization Strategy (auto)

Layer Group	BPW	Note
Embedding / Norm	16 bpw	Unquantized
Attention (layers 0–14, first)	6 bpw	Higher precision for early layers
Attention (layers 1–39)	5 bpw	Standard precision
MLP (layers 0–14)	5 bpw	Higher precision for early layers
MLP (layers 15–39)	4 bpw	Aggressive compression
lm_head	6 bpw	Higher precision for output

💾 VRAM Requirements

Component	Usage
Model Weights	~7.4 GB
Context (4096 tokens)	~1.5–2 GB
Total	~9–10 GB

✅ Comfortably fits in 12 GB VRAM with ~2 GB headroom for longer contexts.

Context Length	Estimated VRAM	Fits 12 GB?
2048 tokens	~8.5 GB	✅ Yes
4096 tokens	~9.5 GB	✅ Yes
8192 tokens	~11 GB	✅ Tight
16384 tokens	~14 GB	❌ Needs 16 GB

🏗️ Model Architecture

Parameter	Value
Architecture	`MistralForCausalLM`
Parameters	~12B
Layers	40
Embedding Dimension	5,120
Attention Heads	32
KV Heads (GQA)	8
Context Length	32,768 tokens
Vocab Size	128,259 tokens
Original Precision	BF16

🇵🇱 About PLLuM

PLLuM (Polish Large Language Model) is a family of LLMs specialized for the Polish language, developed by the PLLuM consortium and continued under HIVE AI — an alliance of research institutions delivering digital public services for Polish public administration.

Key Features

🏆 State-of-the-art in Polish NLP tasks and Polish public administration benchmarks
📚 Largest organic Polish instruction dataset — manually authored by human experts, not distilled
🛡️ Polish preference corpus — first of its kind, with demographically diverse annotators
🔬 Instruction-tuned via SFT + preference learning + advanced alignment
🌐 Bilingual — Polish primary + English for broader generalization

🚀 How to Use

With TabbyAPI

yaml

# config.yml
model_dir: /path/to/PLLuM-12B-chat-2512-EXL3-4.5bpw
max_seq_len: 4096

python start.py --model-dir PLLuM-12B-chat-2512-EXL3-4.5bpw

With ExLlamaV3 (Python)

python

from exllamav3 import ExLlamaV3Config, ExLlamaV3, ExLlamaV3Tokenizer
from exllamav3.generator import ExLlamaV3DynamicGenerator

model_dir = "PLLuM-12B-chat-2512-EXL3-4.5bpw"

config = ExLlamaV3Config(model_dir)
model = ExLlamaV3(config)
model.load()

tokenizer = ExLlamaV3Tokenizer(config)
generator = ExLlamaV3DynamicGenerator(model=model, tokenizer=tokenizer)

prompt = "[INST]Wyjaśnij czym jest uczenie maszynowe w 3 zdaniach.[/INST]"
output = generator.generate(prompt=prompt, max_new_tokens=512)
print(output)

💬 Chat Template

This model uses a Llama 3 / Mistral Instruct style template:

<s>[INST] <system prompt (optional)>

<user message> [/INST] <assistant response> </s>[INST] <next user message> [/INST]

The chat_template.jinja file is included in this repo and will be automatically detected by TabbyAPI and ExLlamaV3.

Recommended Sampling Parameters

Parameter	Value
Temperature	0.6
Top-p	0.9
Repeat penalty	1.1

💻 Hardware Requirements

Minimum (12 GB VRAM)

GPU: Any GPU with ≥12 GB VRAM (RTX 3060 12 GB, RTX 4070, L4, etc.)
RAM: 16 GB system RAM
Storage: 8 GB free
Software: ExLlamaV3 v0.0.43+ or TabbyAPI with exllamav3 backend

📊 Quality Assessment

Metric	Value	Interpretation
SQNR (average)	~33 dB	Well above the perceptible degradation threshold for 12B models
BPW	4.5	Sweet spot between quality and compression
Polish quality	✅ Preserved	Diacritics, grammar, and fluency intact
Compression ratio	3.3×	From ~24.5 GB (BF16) to 7.4 GB

🔧 Quantization Process

Download original BF16 model from CYFRAGOVPL/PLLuM-12B-chat-2512
Quantize using ExLlamaV3 convert.py with target BPW 4.5
Verify output quality (SQNR ~33 dB)

bash

# Quantization command
python convert.py \
    -i /path/to/PLLuM-12B-chat-2512 \
    -w /path/to/work_dir \
    -o /path/to/PLLuM-12B-chat-2512-EXL3-4.5bpw \
    -b 4.5

Environment

Component	Detail
Platform	AWS EC2 g6.2xlarge
GPU	NVIDIA L4 24 GB
ExLlamaV3	v0.0.43
PyTorch	2.7.0+cu128
CUDA	12.8
Source Precision	BF16 → EXL3

📝 Notes

Quantized with the default calibration dataset (250 rows, 2048 columns)
EXL3 format provides the fastest GPU inference among available quantization formats
Polish language quality preserved — SQNR ~33 dB is above the perceptible degradation threshold for 12B-class models
Minimum required version: ExLlamaV3 v0.0.43+ or TabbyAPI with exllamav3 backend

📜 License

This quantization is distributed under the Apache 2.0 License, inherited from the original model.

🔗 Links

Original model: CYFRAGOVPL/PLLuM-12B-chat-2512
PLLuM project: pllum.org.pl
PLLuM paper: arXiv:2511.03823
ExLlamaV3: turboderp-org/exllamav3
TabbyAPI: theroyallab/tabbyAPI
PLLuM-70B GGUF (also by Mati83moni): Mati83moni/Llama-PLLuM-70B-instruct-2512-GGUF
Quantized by: @Mati83moni

🙏 Credits

CYFRA.GOV.PL / HIVE AI — original PLLuM model training & research
turboderp & ExLlamaV3 contributors — EXL3 format and quantization tools
theroyallab — TabbyAPI inference server
Mati83moni — EXL3 quantization and community distribution

Setup	VRAM	Context	Speed
RTX 3060 12 GB	12 GB	4K tokens	~30-40 tok/s
RTX 4070 12 GB	12 GB	4K tokens	~40-55 tok/s
RTX 4090 24 GB	24 GB	16K+ tokens	~60-80 tok/s
L4 24 GB	24 GB	16K+ tokens	~40-60 tok/s

Mati83moni/PLLuM-12B-chat-2512-EXL3

🇵🇱 PLLuM-12B-chat-2512 — EXL3 4.5bpw

⚠️ Note on HuggingFace "Model size" Display