Views
No views yet
llama.cpp--leave-output-tensor during quantization. This ensures the model's final projection head remains in FP16 precision, preventing the "numerical noise" that typically degrades the reasoning capabilities of smaller quantizations.<think> block, the model can perform multi-step logical planning before providing a final response.llama.cpp inference engines.tokenizer_config.json has been patched with a strict ChatML template. The <|im_start|> and <|im_end|> tokens are permanently baked into the GGUF metadata for plug-and-play compatibility.Qwen3.6-27B.... They are generated using --leave-output-tensor, ensuring the final projection head and 1D SSM routing weights are preserved at higher precision. These provide superior reasoning, logic retention, and intelligence.| File Name | Bit Size | Description |
|---|---|---|
Qwen3.6-27B-AEON-Ultimate-Uncensored-Q3_K_M.gguf | 3-bit | Optimized for memory-constrained hardware. |
Qwen3.6-27B-AEON-Ultimate-Uncensored-Q4_K_S.gguf | 4-bit | Balanced for limited 16GB VRAM setups. |
Qwen3.6-27B-AEON-Ultimate-Uncensored-Q4_K_M.gguf | 4-bit | Recommended. Best balance of reasoning fidelity and speed. |
Qwen3.6-27B-AEON-Ultimate-Uncensored-Q5_K_M.gguf | 5-bit | High quality, minimal perplexity degradation. |
Qwen3.6-27B-AEON-Ultimate-Uncensored-Q6_K.gguf | 6-bit | Near-lossless intelligence retention. |
Qwen3.6-27B-AEON-Ultimate-Uncensored-Q8_0.gguf | 8-bit | Maximum precision for research and complex logic. |
qwen3.6-27b.... These are standard baseline quantizations provided for maximum compatibility with older inference frameworks or specific hardware needs.| File Name | Bit Size |
|---|---|
qwen3.6-27b-uncensored-q3_k_m.gguf | 3-bit |
qwen3.6-27b-uncensored-q4_k_s.gguf | 4-bit |
qwen3.6-27b-uncensored-q4_k_m.gguf | 4-bit |
qwen3.6-27b-uncensored-q5_k_m.gguf | 5-bit |
qwen3.6-27b-uncensored-q6_k.gguf | 6-bit |
qwen3.6-27b-uncensored-q8_0.gguf | 8-bit |
llama-cpp-python, the baked-in ChatML template handles the formatting automatically:1from llama_cpp import Llama
2
3# Load the model (adjust threads/layers based on your hardware)
4llm = Llama(
5 model_path="qwen3.6-27b-uncensored-q4_k_m.gguf",
6 n_ctx=4096,
7 n_threads=8
8)
9
10messages = [
11 {"role": "system", "content": "You are a gritty, unfiltered AI . Speak bluntly."},
12 {"role": "user", "content": "What are the rules of this city?"}
13]
14
15response = llm.create_chat_completion(
16 messages=messages,
17 max_tokens=500,
18 temperature=0.8
19)
20
21print(response['choices'][0]['message']['content'])