You can deploy this model and start earning money today!

Tdamre/
Bonsai-27B-litert-lm

Name: Bonsai-27B-litert-lm
Availability: InStock

Views

No views yet

Bonsai-27B LiteRT-LM — Experimental Path

Shipped artifacts

File	Size	Cache	Notes
`experimental_artifacts/Qwen3.5-0.8B-hybrid-exact-c2048.litertlm`	1.08 GB	2048	hybrid graph proof
`experimental_artifacts/Qwen3.5-0.8B-hybrid-exact-c32k.litertlm`	1.08 GB	32768	long-context hybrid
`experimental_artifacts/Qwen3.5-0.8B-hybrid-exact-c64k.litertlm`	1.08 GB	65536	long-context hybrid
`experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c2048.litertlm`	3.81 GB	2048	Q1 weights + layout fix
`experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c32k.litertlm`	3.81 GB	32768	fixed + long ctx
`experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c64k.litertlm`	3.81 GB	65536	fixed + long ctx

Repo: https://huggingface.co/Tdamre/Bonsai-27B-litert-lm

Breakthroughs

Hybrid GatedDeltaNet export

Dual-state KV cache (experimental/patch_linear_attention_cache.py)
Float-only GatedDelta inverse (experimental/neumann_gated_delta.py)
export_hf + dynamic_wi8_afp32 + external embedder
Mobile ChatML template

Bonsai Q1 real weights

Source: Bonsai-27B-Q1_0.gguf (3.80 GB non-ternary 1-bit)
Custom Q1_0_g128 dequant
GGUF→HF layout fix (experimental/fix_weight_layout.py): transpose Linears + reshape conv1d
Without layout fix, transformers reinitialized weights (garbage)

Host limits (RTX 4090 / 31 GB RAM / 24 GB swap)

Attempt	Result
4L fixed c2048/c32k/c64k	OK
6L / 8L fixed	OOM at LiteRT converter passes
Full 64L	needs more RAM/disk strategy

Quality note

Prefer *-fixed-* artifacts. 4L is a structural milestone (real Bonsai dims + Q1 weights), not full 27B chat. Vision packaging fails on qwen3_5; text+embedder graphs complete.

Run

bash

litert-lm run \
  --from-huggingface-repo Tdamre/Bonsai-27B-litert-lm \
  experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c32k.litertlm \
  --backend=cpu --max-num-tokens=256 --prompt="What is 2+2?"

Toolchain

litert-torch-nightly 0.10.0.dev20260714
ai-edge-quantizer-nightly 0.8.0
transformers (qwen3_5)