EXL3 mixed-precision quantization with 512 MoE Experts fully preserved (no pruning) and Attention at 8bpw for enhanced multi-language token quality.

Base model: deepreinforce-ai/Ornith-1.0-397B (BF16, 794GB)

Quantization Specification

Mixed Precision Bit Allocation (--hq)

Module	bpw	Role
Attention (linear + full)	8.0	Token logit generation — high precision for multi-language
Shared Expert	8.0	Shared knowledge routing
Expert (gate/up/down x 512)	3.5 (3.0/4.0 mixed)	MoE experts — bulk capacity
LM Head	16.0 (BF16)	Final token mapping — lossless
Embedding	16.0 (BF16)	Original
LayerNorm / Router	16.0 (BF16)	Original
Average	3.57
Total Size	~178 GB	60 shards

Model Scale

Property	Value
Total parameters	397B (active 35B)
Layers	60 (45 linear attention + 15 full attention)
Experts	512 (fully preserved, no pruning)
Active experts per token	10
Context window	262,144 (256K)
Vocabulary	248,320

Serving (TabbyAPI + exllamav3)

This model uses the EXL3 format (exllamav3 only). SGLang, vLLM, and Transformers cannot load it.

Requirements

GPU: 2x 96GB VRAM (or equivalent total)
Python: 3.12+
exllamav3: 0.0.43+
TabbyAPI: latest

Quick Start

bash

python main.py \
  --host 0.0.0.0 --port 28000 \
  --model-dir /path/to/models \
  --model-name Ornith-397B-EXL3-HQ-35bpw \
  --backend exllamav3 \
  --cache-mode 8,8 \
  --max-seq-len 262144 \
  --tensor-parallel true \
  --reasoning true \
  --reasoning-start-token "<think>" \
  --reasoning-end-token "</think>" \
  --tool-format qwen3_coder \
  --config config.yml

config.yml

yaml

model:
  model_dir: /path/to/models
  model_name: Ornith-397B-EXL3-HQ-35bpw
  backend: exllamav3
  max_seq_len: 262144
  cache_size: 262144
  cache_mode: 8,8
  gpu_split_auto: true
  tensor_parallel: true
  reasoning: true
  reasoning_start_token: "<think>"
  reasoning_end_token: "</think>"
  tool_format: qwen3_coder

sampling:
  default_temperature: 0.3
  default_top_p: 0.8
  default_top_k: 20

VRAM Usage (2x RTX PRO 6000 Blackwell 96GB)

GPU 0: ~90 GB / 96 GB
GPU 1: ~88 GB / 96 GB
Total: ~178 GB (model weights + KV cache + CUDA)

API Usage

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:28000/v1", api_key="YOUR_KEY")

response = client.chat.completions.create(
    model="Ornith-397B-EXL3-HQ-35bpw",
    messages=[{"role": "user", "content": "Write a Python REST API with Flask."}],
    temperature=0.3,
    max_tokens=2048,
)

msg = response.choices[0].message
print("Answer:", msg.content)
print("Reasoning:", getattr(msg, "reasoning_content", None))

Korean Language Patch (korean_fix.py)

Problem

Quantized MoE models using BPE tokenizers may produce U+FFFD (replacement character) for certain Korean words (실행, 결과, 선택, 아키텍처, etc.). This is caused by multi-byte Korean sub-tokens being structurally vulnerable to quantization noise — even at 8bpw attention precision.

Root cause: BPE byte-level sub-tokens for Korean syllables produce invalid UTF-8 sequences when the model predicts slightly wrong token IDs. This is a tokenizer-level issue, not a model quality issue.

Solution: Post-correction Module

korean_fix.py intercepts API responses and replaces U+FFFD patterns with context-appropriate Korean words.

Installation:

Copy korean_fix.py to your TabbyAPI directory:

cp korean_fix.py /path/to/tabbyAPI/

Patch TabbyAPI's exllamav3 backend to apply the fix. Add to backends/exllamav3/model.py:

python

# At the top of the file, after existing imports:
try:
    from korean_fix import fix_korean_ufffd
    def _fix_text(text):
        return fix_korean_ufffd(text) if text else text
except ImportError:
    def _fix_text(text):
        return text

# In the generation loop, before yielding chunks:
chunk = _fix_text(chunk)  # Apply Korean fix to each streaming chunk

# In handle_finish_chunk:
"full_text": _fix_text(full_text),  # Fix the complete response

Restart TabbyAPI.

How It Works

Step	Mechanism
1. Detect	Regex matches consecutive U+FFFD sequences (1-15 chars)
2. Context match	Preceding/following text matched against keyword rules
3. Partial reconstruct	Korean chars adjacent to U+FFFD used to reconstruct full word
4. Frequency fallback	If no context match, uses most likely word (never outputs placeholder)

Covered Words

실행 (execution)     결과 (result)       선택 (selection)    설계 (design)
적용 (application)   구현 (implementation) 확인 (check)       평가 (evaluation)
아키텍처 (architecture) 다운로드 (download)  페이지 (page)

Test Results (with korean_fix)

Test	U+FFFD (before fix)	U+FFFD (after fix)
Korean technical Q&A (7 prompts)	0.488%	0.000%
English technical Q&A (7 prompts)	0.000%	0.000%

Agentic Coding Benchmark

Hardware: 2x RTX PRO 6000 Blackwell (SM120, 96GB x2) Engine: TabbyAPI + exllamav3, tensor-parallel Date: 2026-07-06

Summary

Category	Pass Rate	Avg Response Time
Code Generation	1/3 (33%)	84s
Bug Fixing	3/3 (100%)	12s
Algorithm	1/3 (33%)	47s
Tool Calling	1/2 (50%)	3s
Multi-step (Refactor + Test)	1/1 (100%)	11s
SQL	1/1 (100%)	13s
Overall	8/13 (62%)	28s

Detailed Results

Code Generation

Task	Patterns	Compiles	Time
REST API (Flask)	Yes	Yes	85s
Binary Search Tree	Yes	Yes	28s
Async Web Scraper	Yes	Yes	110s

Bug Fixing

Task	Fixed	Time
Off-by-one (range fix)	Yes	5s
Mutable Default Argument	Yes	10s
Division by Zero (try/except)	Yes	12s

Algorithm

Task	Logic Correct	Compiles	Time
Two Sum (O(n))	Yes	Yes	11s
Merge Intervals	Yes	Yes	57s
LRU Cache	Yes	Yes	40s

Tool Calling

Task	Function Called	Args Correct	Time
Weather (Seoul)	Yes	No	3s
Calculate (125 x 48)	Yes	Yes	2s

Multi-step & SQL

Task	Refactor	Type Hints	Tests	Time
Refactor + pytest	Yes	Yes	Yes	11s
SQL (JOIN + GROUP BY + Subquery)	Yes	Yes	Yes	13s

Inference Speed

Metric	Value
Average response (with reasoning)	28s
Quick response (simple Q&A)	2-5s
Code generation (1024+ tokens)	30-110s
Throughput	~35-49 tok/s

Quantization Verification

Static Analysis

Check	Result
NaN / Inf in tensors	None
Attention bpw	8.01 (expected 8.0)
Expert bpw	3.50 (expected 3.5)
Shared Expert bpw	8.02 (expected 8.0)
LM Head bpw	16.0 (BF16)
Layer coverage	60/60
Tensor count	370,686
Tokenizer integrity (MD5 match with original)	Verified

Conversion Details

Property	Value
Framework	exllamav3 0.0.43
Calibration	250 rows x 2048 cols
Conversion method	GPU split (GPU0: L0-14, GPU1: L15-59)
select_hq_bits	5 (Attention = 3+5 = 8bpw)
MTP	Skipped (no MTP tensors in source)

Limitations

exllamav3 only: Cannot load with SGLang, vLLM, or Transformers. TabbyAPI required.
Inference speed: ~35-49 tok/s (slower than BF16 on high-end GPUs).
Text-only serving: Original is a VLM, but exllamav3 does not support vision inference. Vision encoder weights are not included.
Reasoning model: All responses include a <think>...</think> reasoning block. Final answer in content, reasoning in reasoning_content.
Korean U+FFFD: Without korean_fix.py patch, ~0.5% of Korean characters produce U+FFFD. Apply the patch described above for clean Korean output.

File Structure

Ornith-397B-EXL3-HQ-35bpw/
├── config.json
├── quantization_config.json
├── model.safetensors.index.json
├── model-00001-of-00060.safetensors
├── ...
├── model-00060-of-00060.safetensors
├── tokenizer.json
├── tokenizer_config.json
├── chat_template.jinja
├── generation_config.json
├── preprocessor_config.json
├── processor_config.json
├── video_preprocessor_config.json
├── vocab.json
└── README.md

Original Model

This is an EXL3 quantization of deepreinforce-ai/Ornith-1.0-397B.

Ornith-1.0-397B is a 397B MoE model (post-trained on Qwen 3.5) specialized for agentic coding, achieving SOTA on Terminal-Bench 2.1, SWE-Bench, and NL2Repo.

Benchmark	Ornith-397B (BF16)
Terminal-Bench 2.1 (Terminus-2)	77.5
SWE-bench Verified	82.4
SWE-bench Pro	62.2
NL2Repo	48.2
ClawEval Avg	77.1

Citation

bibtex

@misc{ornith_397b,
    title = {{Ornith-1.0-397B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

License

MIT (same as original model)

gigascake/Ornith-397B-EXL3-HQ-35bpw

Ornith-1.0-397B EXL3 --hq 3.5bpw (512 Experts, Mixed Precision)