⚠️ Important note about quantization resources:

Quantization was performed on CPU using llama.cpp.

The reported ~3 GB refers to processing overhead, not model size.

Memory requirements depend on runtime, context length, and offload settings.

📦 Available Quantizations

Filename	Quant Method	Bits per Weight	File Size	RAM Required*	Quality
PLLuM-70B-instruct-Q4_K_M.gguf	Q4_K_M	4.83 BPW	39.6 GB	~42 GB	⭐⭐⭐⭐ Good — best size/quality balance
PLLuM-70B-instruct-Q5_K_M.gguf	Q5_K_M	5.66 BPW	46.5 GB	~49 GB	⭐⭐⭐⭐⭐ Very good — near-lossless quality

* RAM required = file size + ~2-3 GB overhead for context. For GPU offloading, VRAM required per layer ≈ file size ÷ 80 layers.

🏗️ Model Architecture

Parameter	Value
Architecture	`LlamaForCausalLM`
Parameters	~70.6B (71B size label)
Layers	80 (`blk.0` – `blk.79`)
Embedding Dimension	8,192
Feed-Forward Dimension	28,672
Attention Heads	64
KV Heads (GQA)	8
Context Length	131,072 tokens
RoPE θ	500,000
RMS Norm ε	1e-05
Vocab Size	128,259 tokens
Tokenizer	BPE (`gpt2` / `llama-bpe`)
Original Precision	BF16
Total Tensors	724

🇵🇱 About PLLuM

PLLuM (Polish Large Language Model) is a family of LLMs specialized for the Polish language, developed by the PLLuM consortium and continued under HIVE AI — an alliance of research institutions delivering digital public services for Polish public administration.

Key Features

🏆 State-of-the-art in Polish NLP tasks and Polish public administration benchmarks
📚 Largest organic Polish instruction dataset — manually authored by human experts, not distilled
🛡️ Polish preference corpus — first of its kind, with demographically diverse annotators
🔬 Instruction-tuned via SFT + preference learning + advanced alignment
🌐 Bilingual — Polish primary + English for broader generalization

💬 Chat Template

This model uses a Llama-style [INST] / [/INST] chat template:

<|begin_of_text|>[INST]<system prompt (optional)>

<user message>[/INST]<assistant response></s>[INST]<next user message>[/INST]

Jinja Template (embedded in GGUF)

jinja

{%- if messages[0]['role'] == 'system' %}
    {%- set system_message = messages[0]['content'] %}
    {%- set loop_messages = messages[1:] %}
{%- else %}
    {%- set loop_messages = messages %}
{%- endif %}
{{- bos_token }}
{%- for message in loop_messages %}
    {%- if message['role'] == 'user' %}
        {%- if loop.first and system_message is defined %}
            {{- '[INST]' + system_message + '\n\n' + message['content'] + '[/INST]' }}
        {%- else %}
            {{- '[INST]' + message['content'] + '[/INST]' }}
        {%- endif %}
    {%- elif message['role'] == 'assistant' %}
        {{- message['content'] + eos_token }}
    {%- endif %}
{%- endfor %}

Recommended Sampling Parameters

Parameter	Value
Temperature	0.6
Top-p	0.9
Repeat penalty	1.1

🚀 How to Use

With llama.cpp (CLI)

bash

# Download the Q4_K_M version (~40 GB)
huggingface-cli download Mati83moni/Llama-PLLuM-70B-instruct-2512-GGUF \
  PLLuM-70B-instruct-Q4_K_M.gguf --local-dir .

# Run with llama-cli
./llama-cli \
  -m PLLuM-70B-instruct-Q4_K_M.gguf \
  -p "[INST]Napisz krótki wiersz o Krakowie.[/INST]" \
  -n 512 --temp 0.6 --top-p 0.9 --repeat-penalty 1.1 \
  -ngl 999  # offload all layers to GPU

With llama-server (OpenAI-compatible API)

bash

./llama-server \
  -m PLLuM-70B-instruct-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 999 -c 4096

# Then use any OpenAI client:
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "pllum-70b",
    "messages": [
      {"role": "system", "content": "Jesteś pomocnym polskim asystentem AI."},
      {"role": "user", "content": "Wyjaśnij czym jest uczenie maszynowe w 3 zdaniach."}
    ],
    "temperature": 0.6,
    "top_p": 0.9
  }'

With Ollama

bash

# Create a Modelfile
cat > Modelfile <<EOF
FROM ./PLLuM-70B-instruct-Q4_K_M.gguf
TEMPLATE "[INST]{{ if .System }}{{ .System }}\n\n{{ end }}{{ .Prompt }}[/INST]"
PARAMETER temperature 0.6
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1
PARAMETER stop "</s>"
PARAMETER stop "[INST]"
EOF

# Create and run
ollama create pllum-70b -f Modelfile
ollama run pllum-70b "Jak działa komputer kwantowy?"

With Python (llama-cpp-python)

python

from llama_cpp import Llama

llm = Llama(
    model_path="PLLuM-70B-instruct-Q4_K_M.gguf",
    n_gpu_layers=-1,  # offload all layers
    n_ctx=4096,
)

output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "Jesteś ekspertem od polskiego prawa."},
        {"role": "user", "content": "Jakie są podstawowe prawa pracownika w Polsce?"}
    ],
    temperature=0.6,
    top_p=0.9,
)

print(output["choices"][0]["message"]["content"])

💻 Hardware Requirements

Minimum (Q4_K_M — CPU only)

RAM: 48+ GB
Storage: 40 GB free
Speed: ~2-5 tokens/sec on modern CPU (AVX2)

Recommended (Q4_K_M — GPU offload)

Setup	VRAM	Speed
1× A100 80GB	80 GB (full offload)	~30-40 tok/s
1× A6000 48GB	48 GB (full offload)	~20-30 tok/s
2× RTX 4090 48GB	2×24 GB (split)	~25-35 tok/s
1× RTX 4090 24GB	24 GB (partial, ~48 layers)	~10-15 tok/s
1× L4 24GB	24 GB (partial, ~48 layers)	~8-12 tok/s

For Q5_K_M

Add ~7 GB to all VRAM/RAM requirements above.

🔧 Quantization Details

Process

Download original model (30 safetensors shards, BF16)
Convert to GGUF F16 format using hf-to-gguf (output: 132 GB)
Quantize to Q4_K_M and Q5_K_M using llama-quantize

Environment

Component	Detail
Platform	AWS EC2 (CPU-based quantization, ~3 GB processing overhead)
llama.cpp build	`465b1f0`
GGUF version	V3 (latest)
Source precision	BF16 → F16 GGUF intermediate
Quantization time	~15 min per quant (Q4_K_M + Q5_K_M)

Compression Summary

Format	Size	BPW	Compression vs F16
Original (BF16)	141.1 GB	16.00	1.0×
F16 GGUF	132 GB	16.00	0.94×
Q4_K_M	39.6 GB	4.83	3.6×
Q5_K_M	46.5 GB	5.66	3.0×

Quantization Strategy (K-quant mixed)

Q4_K_M / Q5_K_M: Uses mixed precision — attention value projections (attn_v) and FFN down projections (ffn_down) quantized to Q6_K (higher precision) every 3 layers for better quality retention
Output layer (output.weight): Quantized to Q6_K for both variants
Norm weights: Kept in F32 (negligible size impact)

📊 Quality Comparison Guide

Quant	Use Case	Quality Loss vs F16
Q5_K_M	Production / research / highest quality	Negligible (~0.1-0.3 perplexity)
Q4_K_M	Daily use / fast inference / constrained VRAM	Minimal (~0.3-0.5 perplexity)

💡 Recommendation: Start with Q4_K_M for most use cases. Choose Q5_K_M if you have sufficient VRAM/RAM and need maximum quality for Polish NLP tasks.

📜 License

This model is distributed under the Llama 3.1 Community License, inherited from the base model.

🔗 Links

Original model: CYFRAGOVPL/Llama-PLLuM-70B-instruct-2512
PLLuM project: pllum.org.pl
PLLuM paper: arXiv:2511.03823
llama.cpp: github.com/ggerganov/llama.cpp
Quantized by: @Mati83moni

🙏 Credits

CYFRA.GOV.PL / HIVE AI — original PLLuM model training & research
Meta AI — Llama 3.1 base architecture
ggerganov & llama.cpp contributors — GGUF format and quantization tools
Mati83moni — GGUF conversion and quantization