You can deploy this model and start earning money today!

keras/
qwen3_5_moe_35b_a3b

Name: qwen3_5_moe_35b_a3b
Rating: 2.5 (330 reviews)

keras-hub

text-generation

Views

No views yet

Model Overview

Model Summary

Qwen3.5 MoE is a highly efficient multimodal language model family developed by the Qwen Team (Alibaba Cloud). It extends the standard Qwen3.5 dense architecture by introducing a Sparse Mixture-of-Experts (MoE) design and a hybrid attention mechanism.

By utilizing a routed expert system along with a shared expert, the model maintains vast total capacity while keeping the activated parameter count relatively low for each token, dramatically improving inference efficiency. The architecture features hybrid attention layers that alternate between standard grouped-query attention and recurrent linear attention. Additionally, Qwen3.5 MoE models are natively multimodal, supporting interleaved image and video inputs.

Key Features:

Sparse Mixture-of-Experts (MoE) architecture with shared and routed experts
Hybrid attention (Grouped-Query Attention + Linear Attention)
Native multimodal support (Image and Video understanding)
High efficiency: 35B total parameters with only 3B activated per token

Model Details

Model Family: Qwen3.5 MoE
Architecture: Sparse Mixture-of-Experts (MoE) with Hybrid Attention
Developer: Qwen Team (Alibaba Cloud)
Source: Qwen Team Hugging Face
Task Type: Causal Language Modeling / Vision-Language Understanding
Multimodal: Yes (Text, Image, Video)

Architecture Details

Total Parameters: ~35 Billion
Activated Parameters: ~3 Billion
Expert Routing: Top-k routed experts + 1 shared expert
Attention Mechanism: Hybrid (Grouped-Query Attention and GatedDeltaNet Linear Attention)
Vision Encoder: Native multimodal vision backbone
Max Sequence Length: 32,768 (context window size varies by specific checkpoint)

More Details

[Qwen3.5 MoE Quickstart Notebook](coming soon..!)
Qwen3.5 MoE API Documentation
Qwen3.5 MoE Model Card
KerasHub Beginner Guide
KerasHub Model Publishing Guide

Installation

Keras and KerasHub can be installed with:

pip install -U -q keras-hub
pip install -U -q keras

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.

Preset Table

Preset	Architecture	Total Params	Active Params	Description
`qwen3_5_moe_35b_a3b_base`	Qwen3.5 MoE	~35B	~3B	Qwen3.5 MoE base model.
`qwen3_5_moe_35b_a3b`	Qwen3.5 MoE	~35B	~3B	Qwen3.5 MoE instruction-tuned chat model.

Example Usage

python

import keras
import keras_hub
import numpy as np

python

# Load pre-trained Qwen3 model
qwen3_lm = keras_hub.models.Qwen3_5MoeCausalLM.from_preset( "qwen3_5_moe_35b_a3b")

# Generate text from prompt
response = qwen3_lm.generate("I want to learn about", max_length=50)
print(response)

# Batch generation with multiple prompts
prompts = ["The future of AI is", "Machine learning helps us"]
responses = qwen3_lm.generate(prompts, max_length=30)
for prompt, response in zip(prompts, responses):
    print(f"Prompt: {prompt}")
    print(f"Response: {response}\n")

Custom Sampling Strategies

python

# Greedy sampling (default)
qwen3_lm.compile(sampler="greedy")
response = qwen3_lm.generate("Explain quantum computing", max_length=100)

# Top-k sampling
qwen3_lm.compile(sampler="top_k")
response = qwen3_lm.generate("Write a story about", max_length=80)

# Beam search
qwen3_lm.compile(sampler=keras_hub.samplers.BeamSampler(num_beams=4))
response = qwen3_lm.generate("The best way to learn programming is", max_length=60)

Fine-tuning with LoRA

python

# Enable LoRA for efficient fine-tuning
qwen3_lm.backbone.enable_lora(rank=8)

# Prepare training data
training_texts = [
    "The quick brown fox jumped over the lazy dog.",
    "Machine learning is a subset of artificial intelligence.",
    "Python is a popular programming language for data science.",
    "Deep learning models require large amounts of training data.",
    "Natural language processing helps computers understand human language."
]

# Compile for training
qwen3_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(1e-4),
    metrics=["accuracy"]
)

# Fine-tune the model
qwen3_lm.fit(x=training_texts, batch_size=2, epochs=3)

# Generate with fine-tuned model
response = qwen3_lm.generate("The importance of", max_length=50)
print(response)

Custom Backbone Configuration

python

# Create custom Qwen3_5 MoE backbone
backbone = keras_hub.models.Qwen3_5MoeBackbone(
    vocabulary_size=151936,
    num_layers=12,  # Smaller model for faster training
    num_query_heads=16,
    num_key_value_heads=8,
    head_dim=128,
    hidden_dim=1024,
    intermediate_dim=2048,
    layer_norm_epsilon=1e-6,
    dropout=0.1,
    dtype="float32"
)

# Create tokenizer first
tokenizer = keras_hub.models.Qwen3_5MoeTokenizer.from_preset("qwen3_5_moe_35b_a3b")

# Create preprocessor with tokenizer
preprocessor = keras_hub.models.Qwen3_5MoeCausalLMPreprocessor(
    tokenizer=tokenizer,
    sequence_length=512
)

# Create custom causal LM
custom_qwen3 = keras_hub.models.Qwen3_5MoeCausalLM(
    backbone=backbone,
    preprocessor=preprocessor
)

# Compile and train
custom_qwen3.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(1e-4)
)

# Training data
texts = ["Hello world", "How are you", "Machine learning"]
custom_qwen3.fit(x=texts, batch_size=2, epochs=1)

Example Usage with Hugging Face URI

python

import keras
import keras_hub
import numpy as np

python

# Load pre-trained Qwen3 model
qwen3_lm = keras_hub.models.Qwen3_5MoeCausalLM.from_preset( "hf://keras/qwen3_5_moe_35b_a3b")

# Generate text from prompt
response = qwen3_lm.generate("I want to learn about", max_length=50)
print(response)

# Batch generation with multiple prompts
prompts = ["The future of AI is", "Machine learning helps us"]
responses = qwen3_lm.generate(prompts, max_length=30)
for prompt, response in zip(prompts, responses):
    print(f"Prompt: {prompt}")
    print(f"Response: {response}\n")

Custom Sampling Strategies

python

# Greedy sampling (default)
qwen3_lm.compile(sampler="greedy")
response = qwen3_lm.generate("Explain quantum computing", max_length=100)

# Top-k sampling
qwen3_lm.compile(sampler="top_k")
response = qwen3_lm.generate("Write a story about", max_length=80)

# Beam search
qwen3_lm.compile(sampler=keras_hub.samplers.BeamSampler(num_beams=4))
response = qwen3_lm.generate("The best way to learn programming is", max_length=60)

Fine-tuning with LoRA

python

# Enable LoRA for efficient fine-tuning
qwen3_lm.backbone.enable_lora(rank=8)

# Prepare training data
training_texts = [
    "The quick brown fox jumped over the lazy dog.",
    "Machine learning is a subset of artificial intelligence.",
    "Python is a popular programming language for data science.",
    "Deep learning models require large amounts of training data.",
    "Natural language processing helps computers understand human language."
]

# Compile for training
qwen3_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(1e-4),
    metrics=["accuracy"]
)

# Fine-tune the model
qwen3_lm.fit(x=training_texts, batch_size=2, epochs=3)

# Generate with fine-tuned model
response = qwen3_lm.generate("The importance of", max_length=50)
print(response)

Custom Backbone Configuration

python

# Create custom Qwen3_5 MoE backbone
backbone = keras_hub.models.Qwen3_5MoeBackbone(
    vocabulary_size=151936,
    num_layers=12,  # Smaller model for faster training
    num_query_heads=16,
    num_key_value_heads=8,
    head_dim=128,
    hidden_dim=1024,
    intermediate_dim=2048,
    layer_norm_epsilon=1e-6,
    dropout=0.1,
    dtype="float32"
)

# Create tokenizer first
tokenizer = keras_hub.models.Qwen3_5MoeTokenizer.from_preset("hf://keras/qwen3_5_moe_35b_a3b")

# Create preprocessor with tokenizer
preprocessor = keras_hub.models.Qwen3_5MoeCausalLMPreprocessor(
    tokenizer=tokenizer,
    sequence_length=512
)

# Create custom causal LM
custom_qwen3 = keras_hub.models.Qwen3_5MoeCausalLM(
    backbone=backbone,
    preprocessor=preprocessor
)

# Compile and train
custom_qwen3.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(1e-4)
)

# Training data
texts = ["Hello world", "How are you", "Machine learning"]
custom_qwen3.fit(x=texts, batch_size=2, epochs=1)