This is a Mixture-of-Experts (MoE) decoder built directly from the dense Qwen 3.5 base model (empero-ai/Qwythos-9B-v2) and 5 dense Qwen 3.5 fine-tuned experts, without relying on CobrIXKit.

The model features 33B total parameters, 13B active parameters (A13B) per token, and a native 1,048,576-token context window.

Architecture

Model Class: Qwen35MoEForCausalLM (decoder-only), model_type=qwen35_moe.
Context Window: Native support for up to 1,048,576 tokens (max_position_embeddings=1048576).
Layer Design: Every transformer layer replaces the dense mlp with a sparse MoE block:

text

input_layernorm
    -> linear_attn
    -> post_attention_layernorm
    -> gate / experts[0..12] / shared_expert
    -> residual

Routing: num_local_experts=5, num_experts_per_tok=2 (top-2 routing with softmax over the 5 experts).
Shared Expert: An always-active shared expert based on the base MLP, gated by sigmoid(x @ w), with num_shared_experts=1.
Precision: Router logits are computed in float32.
Initialization: Router initialization is random. Shared-expert gates are initialized to zero.

Layer Types

python

layer_types = [
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention'
]

Weights

Base Model

The base model contributes:

embed_tokens
linear_attn.*
Layer normalization parameters
norm
rotary_emb
lm_head

Experts

The 5 expert models contribute only:

gate_proj
up_proven
down_proj

for every transformer layer.

Integrity

No weight is modified, averaged, or interpolated during assembly.

The MoE model is constructed by preserving the original base-model components while routing the expert MLP projections through the sparse expert system.

Usage

The model directory contains self-contained:

configuration_qwen35_moe.py
modeling_qwen35_moe.py

These modules are loaded automatically through Hugging Face auto_map.

Therefore, no monkey patching is required.

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CobrIX/CobrIX-1.0-Coder-Flash-33B-A13B"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    trust_remote_code=True
)

inputs = tokenizer(
    "Hello, how are you?",
    return_tensors="pt"
)

out = model.generate(
    **inputs,
    max_new_tokens=64
)

print(
    tokenizer.decode(
        out[0],
        skip_special_tokens=True
    )
)

Building & Verification

To replicate the build or verify the model structure, use the provided scripts.

Build the MoE Model

bash

python convert.py \
    --base empero-ai/Qwythos-9B-v2 \
    --experts <e0> <e1> <e2> <e3> <e4> \
    --output Qwen35-CobrIX-MoE

Verify Architecture Integrity

bash

python verify.py \
    --model Qwen35-CobrIX-MoE

Run Generation Tests

bash

python test.py \
    --model Qwen35-CobrIX-MoE

Model Specifications

Specification	Value
Architecture	Qwen 3.5 MoE
Model Class	`Qwen35MoEForCausalLM`
Model Type	`qwen35_moe`
Total Parameters	~33B
Active Parameters	~13B (A13B)
Local Experts	5
Experts per Token	2
Shared Experts	1
Context Length	1,048,576 tokens
Attention Types	Linear + Full Attention
Router Precision	`float32`
Framework	PyTorch
Transformers	Hugging Face Transformers
License	MIT

CobrIX Coder and CobrIX Code Models — Early Access

CobrIX Coder is the AI ecosystem that runs and provides access to models such as CobrIX-1.0-Coder-Flash-MoE and CobrIX-1.0-Coder-Full-MoE.

🌐 Check out and test the AI: https://cobrix.vercel.app/coder

Visit the site to explore the AI and follow the development of CobrIX Code.

Interest List

If you want to know when the AI ecosystem and CobrIX Code will be available to the public, visit:

https://cobrix.vercel.app/coder

On the site, you will find a field to enter your email and join the early access waitlist.

Registered users will receive an email when the CobrIX AI ecosystem and CobrIX Code become available.

We currently have the funds to host CobrIX-1.0-Coder-Flash-33B-A13B on an RTX 6000 Ada Generation GPU (48GB VRAM), supporting between 50 and 100 concurrent users. Please help us by donating (links below); help us create AI that is usable for everyone—open-source AI forever—so we can evolve together.

🚀 Join the waitlist and get notified when CobrIX Code is released.

Donations for Infrastructure

The development of custom AI architectures such as CobrIX-1.0-Coder-Flash-MoE requires ongoing computing resources, research, experimentation, and maintenance.

If this project brings value to your work or company, you can support the development and infrastructure of the CobrIX ecosystem through the wallets below.

All donations help fund:

AI infrastructure
GPU computing - Nvidia RTX PRO 6000 Blackwell 96 gb vram
Open-source model development
New model training
Research and experimentation
Community support

Bitcoin (BTC)

bc1q8mu8fjak4y84qj4dlk8pu4d3zhknm92zra4r4m

Ethereum (ETH / ERC-20)

0x8D9187dEa0a77390ef668361cd5b236DE54af2BB

Solana (SOL)

GQR2jZnWuWP1c3dbuz4mC7ZnyveacBKy63q8qf9nj8bp

Thank you to all developers, researchers, enthusiasts, and partners supporting the evolution of open and accessible artificial intelligence.

CobrIX — Open AI infrastructure and custom model research.