Random Haar quaternions without calibration are approximately neutral vs plain affine 4-bit on this model. See the source repo for details on why and what to try next.

Algorithm

Per group of 64 weight elements:

Split norm ρ = ‖x‖₂ from direction x̄ = x/ρ
Reshape x̄ into 16 blocks of 4 coordinates
Sample 16 pairs of unit quaternions from a per-tensor seed (SHA-256 of tensor name → 32-bit seed → Haar-on-S³)
Apply T(v) = q_L · v · q̄_R per block
Group-wise affine 4-bit quantize the rotated coordinates

Storage

Per group of 64	Bytes
Quantized codes (uint8, 1 byte/code)	64
`scale` (FP32)	4
`bias` (FP32)	4
`ρ` (FP32)	4
Total	76 B / group → ~1 GB for 1.1 B params

Bit-packed nibble storage (2 codes per byte) would give a further ~2x reduction. Future work.

Loading

This model uses a custom format (model.safetensors with sub-keys <tensor>.quantized / .scales / .biases / .rho, plus a quadrotor.json sidecar). Load via the quadrotor package:

python

import json, torch
from pathlib import Path
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from quadrotor.quantize import QuadRotorConfig
from quadrotor.state_dict import decode_state_dict

p = Path(snapshot_download("majentik/TinyLlama-1.1B-Chat-v1.0-QuadRotor-Full-4bit"))
sidecar = json.loads((p / "quadrotor.json").read_text())
cfg = QuadRotorConfig(variant=sidecar["variant"], bits=sidecar["bits"], group_size=sidecar["group_size"])
state = load_file(str(p / "model.safetensors"))
restored = decode_state_dict(state, cfg, sidecar)

Source

Algorithm + tooling: github.com/ajentik/quadrotor
Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0

License

Apache 2.0 (inherited from base model).