This repository contains quantized GGUF versions of Meno-Lite-0.1. All variants were produced using an importance matrix computed on the train split of the ru_llm_calibration dataset, and are intended to be run with llama.cpp.

Available Formats

Quantization type	File size	Quality	Recommendation
Q8_0	~8.05 GB	Virtually identical to FP16	Best quality. Ideal for CPU inference when memory is not a constraint.
Q5_K_M	~5.41 GB	Minimal degradation	Recommended balance. Excellent speed and quality, fits most consumer GPUs.
Q4_K_M	~4.65 GB	Moderate degradation	"Golden standard". Best trade-off between size and quality.
IQ3_M	~3.54 GB	Noticeable degradation	Maximum memory savings. Quality drops visibly; suited for highly constrained devices.

Quality Evaluation

Quality was measured on the test split of the Ru LLM Calibration dataset using the llama-perplexity utility. The original FP16 model served as the reference.

Metric	Q8_0	Q5_K_M	Q4_K_M	IQ3_M
Mean PPL (Q) ↓	9.047	9.075	9.135	9.689
PPL correlation ↑	99.97%	99.87%	99.69%	98.64%
Mean KLD ↓	0.0020	0.0077	0.0174	0.0804
Same top p ↑	96.71%	94.36%	92.16%	84.58%

↑ – higher is better; ↓ – lower is better

How to interpret these metrics:

Mean PPL (Q): Lower is better. Shows the average perplexity of the quantized model.
PPL correlation: Closer to 100% indicates the quantized model behaves almost identically to FP16. Values above 99.5% are considered excellent.
Mean KLD: Measures the divergence between the output probability distributions. Lower is better; 0 means identical distributions.
Same top p: The percentage of tokens where the quantized model's top prediction matches the FP16 model. Higher is better – it reflects how often the model's first-choice token remains unchanged.

Usage

1. Install `llama.cpp`

Follow the official build instructions.

2. Run the model

bash

# CLI
./llama-cli -hf bond005/meno-lite-0.1-gguf -m meno-lite-0.1-Q4_K_M.gguf -p "Привет, как дела?"

# Server with WebUI (default http://127.0.0.1:8080)
./llama-server -hf bond005/meno-lite-0.1-gguf -m meno-lite-0.1-Q4_K_M.gguf --host 0.0.0.0 --port 8080

For more details on available parameters, see the llama.cpp documentation.

About Meno-Lite-0.1

Meno-Lite-0.1 is a 7B model based on Qwen2.5, fine-tuned for RAG, document QA, information extraction, and knowledge graph construction. Read more about its capabilities, training procedure, and limitations in the main model card.

License

All quantized variants inherit the license of the original model (Apache 2.0).

bond005/meno-lite-0.1-gguf

Meno-Lite-0.1 GGUF