You can deploy this model and start earning money today!

RTP-LLM/
Qwen3-Coder-30B-A3B-Instruct-RTPurbo

Name: Qwen3-Coder-30B-A3B-Instruct-RTPurbo
Availability: InStock

safetensors

qwen3_moe

custom_code

Views

No views yet

Qwen3-Coder-30B-A3B-Instruct-RTPurbo

Model Overview

Model Optimizations:
- Sliding Window Attention: 85%
- Full Attention: 15%
Version: 1.0

RTPurbo uses hybrid HeadWise Attention to compress the Qwen3Coder model. Specifically, it divides attention into two parts according to attention type:

Retrieval Heads: These heads perform Full Attention over the entire sequence (or a large chunk), allowing them to capture rich, long-range dependencies and act as a powerful information retrieval component.
non Retrieval Heads: These heads use Sink SWA Attention, processing tokens in a sliding-window or fixed-cache manner. They are highly efficient and ideal for handling very long sequences while maintaining local context.

The following code can be used for inference. HeadWise will be triggered in scenarios where SeqLen > 16,384.

python

from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig

model_name = "RTP-LLM/Qwen3-Coder-30B-A3B-Instruct-RTPurbo"

tokenizer = AutoTokenizer.from_pretrained(model_name)
config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config, 
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Write a quick sort algorithm."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=128
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

content = tokenizer.decode(output_ids, skip_special_tokens=True)

print("content:", content)

Evaluation

This model was evaluated in the lm_eval benchmark using Qwen3-Coder-30B-A3B-Instruct as evaluator.

Longbench	lcc	repo-p	samsum	trec	lsht	2wikim	hotpot	multi-en	multi-zh	musique	qasper	vcsum	qmsum	PR-en	PR-zh	Avg. (%)
Qwen3-Coder-30B-A3B
Full Attn	34.34	27.14	45.80	81.00	47.50	42.08	57.64	52.89	65.99	38.30	39.25	13.55	23.77	99.00	99.75	51.20
RTPurbo	35.96	35.21	46.49	81.00	49.00	47.39	55.44	52.93	65.23	35.58	39.78	13.80	23.68	99.00	99.75	52.02

Media Coverage

Our work has been featured by Minds in AI (机器之心). Please visit it for more details.