Pre-converted weights for colibrì — a pure-C engine that runs huge MoE models on consumer hardware by keeping the dense stack resident in RAM and streaming routed experts from disk.
This repo is the first public Colibri-format checkpoint for Tencent Hy3. It lets you run Hy3's 295B-parameter MoE (21B active per token) on a machine with modest RAM and a fast local NVMe drive, instead of an 8×GPU datacenter setup.
⚠️ This is NOT a GGUF / AWQ / GPTQ / MLX / vLLM model. It uses colibrì's own safetensors container: for each quantized weight, name (packed int4/int8) + name.qs (F32 per-row scales), with packing math bit-identical to the engine's C kernels. It only works with the colibrì hy3 engine (see the Hy3 port in this conversion fork).
What we contribute
The upstream tencent/Hy3 weights ship for BF16 / FP8 inference stacks (vLLM, SGLang, Transformers). Community quants (GGUF, AWQ, etc.) target other runtimes and cannot be loaded by colibrì.
This release adds the missing disk-streaming Colibri container plus the runtime pieces needed to use it:
Contribution
Description
Colibri int4 container
107 out-*.safetensors shards (~142 GB) converted from tencent/Hy3-FP8 with convert_hy3.py — shard-at-a-time, resumable, never needs the full FP8 tree on disk at once
Bottom line: same model family as tencent/Hy3, but repackaged so a single NVMe + ~16–64 GB RAM machine can run it via expert streaming — the same idea as jlnsrk/GLM-5.2-colibri-int4, applied to Hy3.
About Hy3 (from the base model)
Hy3 is Tencent's 295B-parameter Mixture-of-Experts instruct model with 21B activated parameters per token, developed by the Tencent Hy Team. See the official model card for benchmarks, agent capabilities, and deployment recipes for GPU stacks.
Property
Value
Architecture
Mixture-of-Experts (MoE), hy_v3
Total parameters
295B
Activated parameters
21B
Layers (excl. MTP)
80 (layer 0 dense, 79 sparse MoE)
Attention
GQA — 64 heads, 8 KV heads, head dim 128
Hidden size
4096
Experts
192 routed + 1 shared, top-8 per token
Context length
256K
Vocabulary
120832
Usage
1. Get the engine (Hy3 port)
bash
1git clone https://github.com/JustVugg/colibri.git # or your Hy3 port fork with c/hy3.c2cd colibri/c
3./setup.sh # builds hy3 + self-test on hy3_tiny fixture
2. Download this repo
Store on a fast local disk (NVMe, native ext4). Avoid WSL /mnt/c (9p) or network mounts — random expert reads are disk-bound.
The converter downloads one FP8 shard at a time, dequants, requantizes to the Colibri layout, and deletes the source shard — the full FP8 checkpoint never has to exist on disk at once.
Hy3 is smaller per expert than GLM-5.2 (~9 MB vs ~19 MB int4), with 192 experts/layer and 79 MoE layers — cold decode still moves ~5–6 GB of expert data per token from disk. Throughput is disk-bound on a cold cache (often 0.1–0.5 tok/s class on consumer NVMe, highly setup-dependent). Warm LRU / pinned hot experts improve this substantially.
This is not a replacement for vLLM / SGLang deployment on 8× datacenter GPUs. It is a way to run the same model class on hardware you already own.
Status & limitations (early port)
Beta: the Hy3 hy3.c engine is a new port (validated on oracle fixtures; full-model chat is still being stabilized).
MTP: weights are included; native speculative decode in hy3.c is not implemented yet.
CUDA expert tier: planned (make hy3 CUDA=1); not production-ready for Hy3 yet.
KV disk persistence: not yet ported from GLM's MLA path (GQA layout differs).
Not loadable in transformers.AutoModelForCausalLM — use colibrì only.
