You can deploy this model and start earning money today!

kerasformers/
qwen1.5-moe-a2.7b

Name: qwen1.5-moe-a2.7b
Availability: InStock

Qwen/Qwen1.5-MoE-A2.7B

finetune

other

Views

No views yet

See our collection for all Qwen2 versions.

Run Qwen2 with Keras 3: JAX, PyTorch, or TensorFlow

GitHub

zeromodels/qwen1.5-moe-a2.7b

Qwen1.5-MoE-A2.7B is Alibaba's first Qwen mixture-of-experts LLM: a top-k router over 60 fine-grained experts plus shared experts, reaching 7B-dense quality with only ~2.7B active parameters. Same Qwen2-MoE backbone (grouped-query attention with q/k/v bias, SwiGLU, RMSNorm, rotary positions).

For more details on the model, please see the upstream model card.

Pure-Keras 3 conversion of Qwen/Qwen1.5-MoE-A2.7B for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX. The mixture-of-experts banks are stored fused (the hub layout) and routed on every backend.

This is a base (pretrained) checkpoint, for completion or fine-tuning.

Quick start

python

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from zeromodels.models.qwen2_moe import Qwen2MoeTextGenerate, Qwen2MoeTokenizer

model = Qwen2MoeTextGenerate.from_weights("zeromodels/qwen1.5-moe-a2.7b")
tokenizer = Qwen2MoeTokenizer.from_weights("zeromodels/qwen1.5-moe-a2.7b")

inputs = tokenizer([
    {"role": "user", "content": "Explain rotary embeddings in one sentence."}
])
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0]))

Load any Qwen2 variant the same way with from_weights("zeromodels/<variant>"):

Variant	Hub	Type
`qwen2-0.5b`	`zeromodels/qwen2-0.5b`	base
`qwen2-0.5b-instruct`	`zeromodels/qwen2-0.5b-instruct`	instruct
`qwen2-1.5b`	`zeromodels/qwen2-1.5b`	base
`qwen2-1.5b-instruct`	`zeromodels/qwen2-1.5b-instruct`	instruct
`qwen2-7b`	`zeromodels/qwen2-7b`	base
`qwen2-7b-instruct`	`zeromodels/qwen2-7b-instruct`	instruct
`qwen2-72b`	`zeromodels/qwen2-72b`	base
`qwen2-72b-instruct`	`zeromodels/qwen2-72b-instruct`	instruct
`qwen2-57b-a14b`	`zeromodels/qwen2-57b-a14b`	MoE base
`qwen2-57b-a14b-instruct`	`zeromodels/qwen2-57b-a14b-instruct`	MoE instruct
`qwen1.5-moe-a2.7b`	`zeromodels/qwen1.5-moe-a2.7b`	MoE base
`qwen1.5-moe-a2.7b-chat`	`zeromodels/qwen1.5-moe-a2.7b-chat`	MoE chat

Tips

Set KERAS_BACKEND before importing Keras / zeromodels.
Prefer Qwen2MoeTokenizer.from_weights(...) so the chat template matches.
Larger checkpoints: try load_dtype="bfloat16" or quantization="int8".
Experts are stored fused; loading is weight-only bf16 (saves memory, not compute).
See Loading Weights and the Qwen2-MoE docs.
Community / upstream safetensors still work via the hf: prefix, e.g. Qwen2MoeTextGenerate.from_weights("hf:Qwen/Qwen1.5-MoE-A2.7B").

Special Thanks

A huge thank you to the Qwen team at Alibaba for creating and releasing these models.

License: Tongyi Qianwen (see the upstream license).

kerasformers/qwen1.5-moe-a2.7b