Views
No views yet
| Type | Description | Quality loss |
|---|---|---|
| F16 | Original fp16 | None |
| Q8_0 | Minimum | |
| Q6_K | Very Small | |
| Q5_K_M | Small | |
| Q4_K_M | ~ | Moderate (recommended) |
| Q4_K_S | ~ | Moderate |
| Q3_K_M | ~ | Noticeable |
| IQ4_XS | ~ | Moderate (new format) |
| IQ3_M | ~ | Noticeable (new format) |
llama-cli -m model-q4_k_m.gguf -p "How to solve this problem?" -n 5121from transformers import AutoModelForCausalLM, AutoTokenizer
2model = AutoModelForCausalLM.from_pretrained("FGFGFGGDFGDFGSD/qwen3-0.6-Claude4.6-4.7-Gpt5.5-Gemini3.1pro-gguf")