I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.

In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the --tensor-type option in llama.cpp to manually "bump" important layers to higher precision. You can see the implementation here:
👉 Layer bumping with llama.cpp

While this does increase model file size, it significantly improves precision for a given quantization level.

I'd love your feedback—have you tried this? How does it perform for you?

Click here to get info on choosing the right GGUF model format

LongWriter-Zero ✍️ — Mastering Ultra-Long Text Generation via Reinforcement Learning

🤗 HF Dataset • 📃 Paper

🚀 LongWriter-Zero

LongWriter-Zero is a purely reinforcement learning (RL)-based large language model capable of generating coherent passages exceeding 10,000 tokens.

Built upon Qwen 2.5-32B-Base, the training process includes:

30 billion-token continual pretraining on long-form books and technical reports to enhance fundamental writing capabilities;
Application of Group Relative Policy Optimization (GRPO) with a composite reward function:
- Length Reward Model (RM) enforces the desired output length,
- Writing RM scores fluency, coherence, and helpfulness,
- Format RM ensures strict adherence to the <think>…</think><answer>…</answer> structure, and also detects repeated content to avoid redundancy;
A dedicated prompting strategy that encourages models to explicitly reflect before answering, thereby improving structural planning and fine-grained length control.

The resulting model, LongWriter-Zero-32B, matches or surpasses the performance of 100B-scale models in ultra-long-form generation.

📊 Benchmarks & Evaluation

LongWriter-Zero’s effectiveness is demonstrated on two fronts: WritingBench and Arena-Write for automatic scoring and a human-in-the-loop win-rate study for pairwise quality comparison.

📝 WritingBench & Arena-Write Results

WritingBench (scale 1–10) & Arena-write (Elo) performance of different LLMs .

🏆 Win-Rate Results

Donut charts showing win/tie/loss proportions against six baselines (left) and aggregated human evaluation (right).

Summary: LongWriter-Zero achieves the highest automatic WritingBench score among open models and secures dominant win-rates in pairwise GPT-4.1 evaluations, confirming its superior quality in ultra-long-form generation while maintaining efficiency.

⚡ Quick Start (HF generate)

python

import re
model_name = "THU-KEG/LongWriter-Zero-32B"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
def format_prompt_with_template(prompt):

    base_format_zn = r"用户与助手之间的对话。用户提供一个写作/通用任务，助手完成它。助手首先在脑海中深入思考写作/回答过程，然后向用户提供最终的书面作品。助手应进行全面而深入的规划，确保写作/通用任务的每个方面都详细且结构合理。如果写作要求存在任何不确定性或歧义，助手应反思，向自己提出澄清性问题，并探索多种写作方式，以确保最终作品达到最高质量标准。由于写作是一个既富有创造性又需要结构性的任务，助手应从多个角度进行分析，考虑连贯性、清晰度、风格、语气、受众和目的，等等因素。此外，助手还应对作品进行审查和优化，以增强其表达效果。写作思考过程和最终的书面作品分别用 <think> </think> 和 <answer> </answer> 标签包裹，如下所示：<think>详细的写作规划和结构设计，可能包括头脑风暴、大纲制定、风格选择、受众适配、反思以及质量检查等等。</think> <answer>经过充分优化和润色的最终书面作品。</answer> <|用户|>: {question} <|助手|>:"
    base_format_en = r"A conversation between the user and the assistant. The user provides a writing/general task, and the assistant completes it. The assistant first deeply thinks through the writing/answering process in their mind before providing the final written work to the user. The assistant should engage in comprehensive and in-depth planning to ensure that every aspect of the writing/general task is detailed and well-structured. If there is any uncertainty or ambiguity in the writing request, the assistant should reflect, ask themselves clarifying questions, and explore multiple writing approaches to ensure the final output meets the highest quality standards. Since writing is both a creative and structured task, the assistant should analyze it from multiple perspectives, considering coherence, clarity, style, tone, audience, purpose, etc.. Additionally, the assistant should review and refine the work to enhance its expressiveness. The writing thought process and the final written work should be enclosed within <think> </think> and <answer> </answer> tags, respectively, as shown below: <think>A comprehensive strategy for writing that encompasses detailed planning and structural design—including brainstorming, outlining, style selection, audience adaptation, self-reflection, quality assurance, etc..</think> <answer>The final written work after thorough optimization and refinement.</answer>  <|user|>: {question} <|assistant|>:"
    base_format = base_format_zn if re.search(r'[\u4e00-\u9fff]', prompt) else base_format_en
    formatted_prompt = base_format.format(question=prompt)
    return formatted_prompt

prompt = "Write a 500-word story."
messages = [
    {"role": "user", "content": format_prompt_with_template(prompt)}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=2048,
    temperature=0.6,
    do_sample=True,
    stop_strings=["<|user|>", "<|endoftext|>", "</answer>"],
    tokenizer=tokenizer
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(response)

Note: We use a slightly different tokenizer and chat template compared to the original Qwen2.5-32B-Instruct model.

⚡ Quick Start (SGlang)

The snippet below shows how to format prompts with LongWriter-Zero’s <think> … </think><answer> … </answer> protocol and call the model through an SGlang-powered endpoint supporting streaming responses.

python

import json, requests, re

def format_prompt_with_template(prompt):

    base_format_zn = r"用户与助手之间的对话。用户提供一个写作/通用任务，助手完成它。助手首先在脑海中深入思考写作/回答过程，然后向用户提供最终的书面作品。助手应进行全面而深入的规划，确保写作/通用任务的每个方面都详细且结构合理。如果写作要求存在任何不确定性或歧义，助手应反思，向自己提出澄清性问题，并探索多种写作方式，以确保最终作品达到最高质量标准。由于写作是一个既富有创造性又需要结构性的任务，助手应从多个角度进行分析，考虑连贯性、清晰度、风格、语气、受众和目的，等等因素。此外，助手还应对作品进行审查和优化，以增强其表达效果。写作思考过程和最终的书面作品分别用 <think> </think> 和 <answer> </answer> 标签包裹，如下所示：<think>详细的写作规划和结构设计，可能包括头脑风暴、大纲制定、风格选择、受众适配、反思以及质量检查等等。</think> <answer>经过充分优化和润色的最终书面作品。</answer> <|用户|>: {question} <|助手|>:"
    base_format_en = r"A conversation between the user and the assistant. The user provides a writing/general task, and the assistant completes it. The assistant first deeply thinks through the writing/answering process in their mind before providing the final written work to the user. The assistant should engage in comprehensive and in-depth planning to ensure that every aspect of the writing/general task is detailed and well-structured. If there is any uncertainty or ambiguity in the writing request, the assistant should reflect, ask themselves clarifying questions, and explore multiple writing approaches to ensure the final output meets the highest quality standards. Since writing is both a creative and structured task, the assistant should analyze it from multiple perspectives, considering coherence, clarity, style, tone, audience, purpose, etc.. Additionally, the assistant should review and refine the work to enhance its expressiveness. The writing thought process and the final written work should be enclosed within <think> </think> and <answer> </answer> tags, respectively, as shown below: <think>A comprehensive strategy for writing that encompasses detailed planning and structural design—including brainstorming, outlining, style selection, audience adaptation, self-reflection, quality assurance, etc..</think> <answer>The final written work after thorough optimization and refinement.</answer>  <|user|>: {question} <|assistant|>:"
    base_format = base_format_zn if re.search(r'[\u4e00-\u9fff]', prompt) else base_format_en
    formatted_prompt = base_format.format(question=prompt)
    return formatted_prompt



prompt = "XXXX"          # ← replace with your writing task
data = {
    "model": "LongWriter-Zero-32B",
    "prompt": format_prompt_with_template(prompt),
    "temperature": 0.6,
    "top_p": 0.95,
    "max_tokens": 15500,
    "stop": ["<|user|>", "<|endoftext|>", "</answer>"],
    "stream": True,
}

# SGlang Gateway (example)
response = requests.post(
    "http://XXXX:9999/v1/completions",  # ← replace with your IP
    json=data,
    headers={"Content-Type": "application/json"},
    timeout=1200,
    stream=True,
)

for chunk in response.iter_lines():
    if chunk and chunk.startswith(b"data:"):
        if chunk == b"data: [DONE]":
            break
        payload = json.loads(chunk[5:])
        print(payload["choices"][0]["text"], end="", flush=True)

📝 Citation

bibtex

@misc{wu2025longwriterzeromasteringultralongtext,
      title={LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning}, 
      author={Yuhao Wu and Yushi Bai and Zhiqiang Hu and Roy Ka-Wei Lee and Juanzi Li},
      year={2025},
      eprint={2506.18841},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.18841}, 
}

<!--End Original Model Card-->

---

# <span id="testllm" style="color: #7F7FFF;">🚀 If you find these models useful</span>

Help me test my **AI-Powered Quantum Network Monitor Assistant** with **quantum-ready security checks**:  

👉 [Quantum Network Monitor](https://readyforquantum.com/?assistant=open&utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme)  


The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : [Source Code Quantum Network Monitor](https://github.com/Mungert69). You will also find the code I use to quantize the models if you want to do it yourself [GGUFModelBuilder](https://github.com/Mungert69/GGUFModelBuilder)

💬 **How to test**:  
 Choose an **AI assistant type**:  
   - `TurboLLM` (GPT-4.1-mini)  
   - `HugLLM` (Hugginface Open-source models)  
   - `TestLLM` (Experimental CPU-only)  

### **What I’m Testing**  
I’m pushing the limits of **small open-source models for AI network monitoring**, specifically:  
- **Function calling** against live network services  
- **How small can a model go** while still handling:  
  - Automated **Nmap security scans**  
  - **Quantum-readiness checks**  
  - **Network Monitoring tasks**  

🟡 **TestLLM** – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):  
- ✅ **Zero-configuration setup**  
- ⏳ 30s load time (slow inference but **no API costs**) . No token limited as the cost is low.
- 🔧 **Help wanted!** If you’re into **edge-device AI**, let’s collaborate!  

### **Other Assistants**  
🟢 **TurboLLM** – Uses **gpt-4.1-mini** :
- **It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited. 
- **Create custom cmd processors to run .net code on Quantum Network Monitor Agents**
- **Real-time network diagnostics and monitoring**
- **Security Audits**
- **Penetration testing** (Nmap/Metasploit)  

🔵 **HugLLM** – Latest Open-source models:  
- 🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.

### 💡 **Example commands you could test**:  
1. `"Give me info on my websites SSL certificate"`  
2. `"Check if my server is using quantum safe encyption for communication"`  
3. `"Run a comprehensive security audit on my server"`
4. '"Create a cmd processor to .. (what ever you want)" Note you need to install a [Quantum Network Monitor Agent](https://readyforquantum.com/Download/?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme) to run the .net code on. This is a very flexible and powerful feature. Use with caution!

### Final Word

I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is [open source](https://github.com/Mungert69). Feel free to use whatever you find helpful.

If you appreciate the work, please consider [buying me a coffee](https://www.buymeacoffee.com/mahadeva) ☕. Your support helps cover service costs and allows me to raise token limits for everyone.

I'm also open to job opportunities or sponsorship.

Thank you! 😊

Mungert/LongWriter-Zero-32B-GGUF

LongWriter-Zero-32B GGUF Models

Model Generation Details

Quantization Beyond the IMatrix