Pure Rust ternary inference engine based on BitNet b1.58-2B-4T. No Python, no CUDA, no external ML frameworks. Single executable + model weights = portable AI that runs on any machine.
Zero-multiplication inference — ternary weights {-1, 0, +1} mean the inner GEMV loop uses only addition and subtraction, no floating-point multiply. Smart system awareness — detects RAM and CPU at startup and adjusts generation limits automatically.
License
This project is licensed under Apache 2.0. The base model BitNet b1.58-2B-4T is released by Microsoft under the MIT license.
What It Does
QORA-LLM-2B is a 2-billion parameter language model. It can:
Text generation — answer questions, write code, explain concepts
Chat mode — multi-turn conversation with LLaMA 3 chat template
Raw mode — direct text completion without chat formatting
Architecture
BitNet b1.58 uses a modified transformer with ternary quantized projections and SubLN normalization:
Component
Details
Parameters
2B total
Hidden dim
2560
Layers
30
Attention
GQA (20 query / 5 KV heads), head_dim=128
FFN
6912 intermediate, Squared ReLU activation
Vocabulary
128,256 tokens (LLaMA 3)
Context
4096 tokens
RoPE
rotate_half, theta=500,000
SubLN Pattern (4 norms per layer)
Unlike standard LLaMA (2 norms per layer), BitNet uses SubLN with extra normalization before output projections:
residual = x
x = input_layernorm(x) # RMSNorm [2560]
q, k, v = q/k/v_proj(x) # Ternary linear (add/sub only)
q, k = apply_rope(q, k)
attn = attention(q, k, v) # GQA: 20Q/5KV
attn = attn_sub_norm(attn) # SubLN RMSNorm [2560]
attn = o_proj(attn) # Ternary linear
x = residual + attn
residual = x
x = post_attention_layernorm(x) # RMSNorm [2560]
gate = relu2(gate_proj(x)) # Squared ReLU: max(0,x)^2
up = up_proj(x)
x = ffn_sub_norm(gate * up) # SubLN RMSNorm [6912]
x = down_proj(x)
x = residual + x
Ternary GEMV (No Multiplication)
Each weight is one of {-1, 0, +1}, packed 4 per byte (2 bits each). The inner loop:
A single scalar multiply by the layer scale factor happens only at the end. This makes BitNet inference fundamentally different from traditional float/quantized models.
AVX-512 SIMD Acceleration
On CPUs with AVX-512 support (Intel 11th gen+, AMD Zen 4+), QORA-LLM-2B automatically uses hand-written AVX-512 SIMD kernels for significant CPU speedup:
The ternary kernel processes 16 values per cycle using a 4-entry lookup table replicated across 16 AVX-512 lanes. Each byte contains 4 packed ternary values (2 bits each), decoded in 4 passes with shift+mask.
Detection is automatic at runtime — falls back to scalar code on non-AVX-512 CPUs with zero overhead.
Smart System Awareness
QORA-LLM-2B detects your system at startup and automatically adjusts generation limits:
1# Chat mode (default)2qor2b --prompt "Explain how ternary neural networks work"34# With token limit5qor2b --prompt "Write a haiku about Rust" --max-tokens 10067# Raw text completion (no chat template)8qor2b --prompt "Once upon a time" --raw
910# Greedy decoding (deterministic)11qor2b --prompt "What is 2+2?" --greedy
CLI Flags
Flag
Description
--prompt TEXT
Input prompt (default: "Hello, how are you?")
--max-tokens N
Max tokens to generate (default: auto based on RAM)
--raw
Raw text completion (skip chat template)
--greedy
Greedy decoding (temperature=0)
--load PATH
Custom model path (default: model.qor2b next to exe)
--convert DIR
Convert safetensors from DIR to .qor2b format
--save PATH
Output path for conversion (default: model.qor2b)
Sampling Defaults
Parameter
Value
temperature
0.7
top_k
40
top_p
0.95
repetition_penalty
1.1
presence_penalty
0.6
Converting from Safetensors
To convert the original bf16 weights yourself:
bash
1# Download the model from HuggingFace2# (requires: pip install huggingface_hub)3python -c "from huggingface_hub import snapshot_download; snapshot_download('microsoft/bitnet-b1.58-2B-4T-bf16', local_dir='bitnet-bf16')"45# Convert to .qor2b format6qor2b --convert bitnet-bf16 --save model.qor2b
Conversion takes ~2 minutes and compresses 4.8 GB bf16 safetensors to a 1.13 GB ternary binary.
Building from Source
cargo build --release
Dependencies
Language: Pure Rust (2024 edition)
rayon — Thread pool for parallel GEMV and attention
half — F16 support for embeddings
tokenizers — HuggingFace tokenizer (LLaMA 3)
safetensors — Model conversion from HuggingFace format
serde_json — Config parsing
No ML framework — all matrix ops are hand-written Rust
File Structure
src/
main.rs — CLI entry point, argument parsing, smart system
config.rs — BitNet model configuration
gemv.rs — Ternary GEMV kernel, forward pass, attention, RoPE
simd.rs — AVX-512 SIMD kernels (ternary GEMV, fused relu² MLP)
generate.rs — Text generation loop with sampling
tokenizer.rs — LLaMA 3 tokenizer and chat template
save.rs — Binary model format (.qor2b) save/load
convert.rs — Safetensors bf16 -> ternary .qor2b converter
system.rs — System resource detection and smart limits
lib.rs — Module exports
