QwennAI/

Qwen3.9-245B-A29B

# QwennAI / Qwen3.9-245B-A29B 🚀✨

## Introduction 🌟🔥

**Qwen3.9-245B-A29B** is an advanced open-weights Mixture-of-Experts (MoE) foundation model from the Qwen series. Built with hybrid thinking mechanisms and sparse routing, it features **245B total parameters** while activating only **29B parameters** per forward pass. This architecture enables frontier-grade reasoning, long-horizon tool execution, and code synthesis at high throughput.

### Highlights 💡

* **Sparse MoE Architecture:** 245B total parameters with 29B active per token across routed expert layers.
* **Hybrid Reasoning Control:** Native support for dual-mode generation (`thinking` with chain-of-thought vs. efficient `instruct` mode).
* **Ultra Long-Context Support:** 131,072 native context window, extendable up to 1,000,000 tokens with YaRN / Dual-Chunk Attention.
* **Agentic & Tool Calling:** Enhanced AST parsing for JSON-based function calling, multi-turn tool loops, and code interpreters.

---

## Model Overview 🛠️📐

* **Architecture:** Sparse Mixture-of-Experts (MoE) Transformer
* **Total Parameters:** 245B
* **Activated Parameters:** 29B
* **Number of Layers:** 96
* **Hidden Dimension:** 8,192
* **Attention Heads:** 64 for Q, 8 for KV (Grouped-Query Attention)
* **MoE Routing:** Top-4 expert routing with shared routing paths
* **Vocabulary Size:** 248,320 (Padded)
* **Context Length:** 131,072 tokens (Native) / 1M tokens (Extensible)

---

## Benchmark Results 📊🏆

| Benchmark | Setting | Metric | Qwen3.9-245B-A29B |
| --- | --- | --- | --- |
| **MMLU-Pro** | 5-shot | Accuracy | 83.2% |
| **MATH-500** | 0-shot | Accuracy (Thinking) | 95.1% |
| **AIME 2026** | Pass@1 | Accuracy | 89.4% |
| **LiveCodeBench** | 0-shot | Pass@1 | 71.3% |
| **GPQA Diamond** | 0-shot | Accuracy | 73.8% |
| **BFCL v3** | Multi-Turn | AST Match | 93.6% |

---

## Best Practices & Sampling Parameters ⚙️🎯

| Mode | Temperature | Top-P | Top-K | Presence Penalty | Max Reasoning Tokens |
| --- | --- | --- | --- | --- | --- |
| **Thinking Mode** | `1.0` | `0.95` | `20` | `0.0` | 262,144 |
| **Instruct Mode** | `0.7` | `0.80` | `20` | `1.5` | Disabled |

---

## Quickstart 🚀💻

### Transformers Inference

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "QwennAI/Qwen3.9-245B-A29B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": "Prove that the square root of 2 is irrational."}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True  # Toggle thinking process
)

inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=4096,
    temperature=1.0,
    top_p=0.95
)

response = tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
print(response)