Custom fork required. This model uses a novel architecture (MLA + MoE with identity experts + N-gram embeddings) that is not supported by upstream llama.cpp. You must build from the longcat-flash-ngram branch of the linked fork.
About LongCat-Flash-Lite
LongCat-Flash-Lite is a 68.5B parameter Mixture-of-Experts language model from Meituan, with only 3–4.5B parameters activated per token. It combines three architectural innovations that make it unusually efficient:
N-gram embeddings augment the standard token embedding with context from neighboring tokens
Multi-head Latent Attention (MLA) compresses the KV cache for efficient long-context inference
Identity experts in the MoE layer allow tokens to bypass expert computation via learned residual paths
The model supports a 327,680 token context window.
Why a custom fork?
Two upstream llama.cpp PRs attempted to add this architecture:
PR #19167 (ngxson) — N-gram embedding support, blocked by base model not yet being supported
PR #19182 (ngxson) — LongCat-Flash base architecture, abandoned after maintainers deemed identity experts too complex
This fork implements the complete architecture in a single self-contained addition (903 lines across 15 files). The implementation was AI-generated using Claude Code, which means it cannot be submitted upstream per llama.cpp's AI usage policy. It will remain available as a standalone fork.
Available Quantizations
Quantization guidance: The sweet spot for this MoE architecture is Q4_K_M or Q5_K_M — best balance of quality, speed, and VRAM. Hallucination rate climbs monotonically as quantization increases: going above Q4 yields only marginal accuracy gains at steep speed/VRAM cost, while going below Q4 loses real knowledge with no quality benefit. Q3_K_L is usable but noticeably degraded. Lower quantizations (Q2 and below) are not provided as the model degenerates — accuracy halves, response times spike from looping, and hallucination rate exceeds 91%.
The server exposes an OpenAI-compatible API at http://localhost:8080/v1.
Inference Performance
Measured on NVIDIA GB10 (128 GB unified memory) with full GPU offload:
Quantization
Generation Speed
Q4_K_M
~57 tok/s
Architecture Details
LongCat-Flash-Lite uses a double-block layout: the original 14 transformer layers each contain two sub-blocks, mapped to 28 llama.cpp blocks. Key parameters:
Parameter
Value
Total parameters
68.5B
Activated parameters
3–4.5B
Vocabulary
131,072 tokens
Hidden dimension
3,072
Attention heads
32
KV heads (GQA)
1
Q LoRA rank
1,536
KV LoRA rank
512
Real experts
256
Identity experts
128
Active experts (top-k)
12
Shared experts
1
Expert FFN dimension
1,024
N-gram tables
12 (4 neighbor x 3 split)
Context window
327,680
RoPE
YaRN (factor=10, base=5M)
N-gram Embeddings
Instead of using only the current token's embedding, the model hashes neighboring tokens (4 neighbors, split into 3 groups) through 12 polynomial rolling hash tables. The final embedding is computed as:
This gives the model sub-word and local context awareness at the embedding level.
Multi-head Latent Attention (MLA)
MLA compresses keys and values through a low-rank bottleneck (KV LoRA rank 512), reducing the KV cache size while maintaining attention quality. LoRA scaling factors (sqrt(2) for Q, sqrt(6) for KV) are applied at runtime.
Identity Experts
Of the 384 total experts per MoE layer, 128 are "identity" experts that pass the input through unchanged. When the router selects an identity expert, the token's representation is carried forward via a residual connection without any computation. This allows the model to learn which tokens benefit from expert processing and which are better left alone.
Acknowledgments
ngxson for the initial llama.cpp PRs #19167 and #19182 that explored this architecture
