In your terminal, or in your browser · Runs on the CPU · No cloud account · No telemetry
Foundation and model credit. Samosa Chat is built on
colibrì by JustVugg; its
expert-streaming design, SIMD kernels, and utility headers made this project
possible. The converted checkpoint comes from
Qwen/Qwen3.6-35B-A3B by the
Qwen team. This is an independent, unofficial Apache-2.0 text-only
conversion. It is not affiliated with or endorsed by either upstream team.
What Samosa adds: its own Qwen3.6 engine in C, the group-32 format and
its converter, the expert cache that fits 35B parameters into 16 GB, sealed
resumable conversations, a local server and app, and an atomic installer.
Details.
This repository hosts the model that powers Samosa Chat. These are the
group-32 model files used by Samosa Chat,
a free, open-source project that runs Qwen3.6-35B-A3B fully locally on a
16 GB machine. The complete source, documentation, and issue
tracker live on GitHub:
github.com/deepanwadhwa/samosa-chat
Samosa Chat packages the group-32 int4 model and a dependency-free C inference
engine. You use it from the terminal (samosa "your question"), which is the
normal way, or through a local browser app (samosa app), which is currently a
demo. Everything stays on your computer: no account, no telemetry, and no remote
request during inference. The server binds only to 127.0.0.1.
What Samosa adds
Qwen's checkpoint and the colibrì runtime are the starting point. Samosa adds:
A Qwen3.6 engine in C, covering the 30 Gated DeltaNet layers, the 10
gated attention layers, the shared/routed MoE path, the tokenizer, and the
chat template.
The group-32 format and its converter — a shard-by-shard conversion of
the original checkpoint into a manifest-based expert container. Group-32
gives every 32 weights their own int4 scale instead of one scale per whole
matrix row, which is what makes the output quality hold up.
A byte-budgeted expert cache with LRU eviction, per-layer floors,
reusable slabs, memory-pressure monitoring, and I/O telemetry — the piece
that lets a 35B model run in 16 GB by streaming experts from SSD.
Sealed, resumable conversations (QWSESS01): geometry-checked,
SHA-256-sealed, atomically written, and byte-exact on resume.
A dependency-free local server and browser app in C: JSON/SSE, a bounded
queue, cancellation, health telemetry, and clean shutdown.
An atomic installer that verifies every byte, compiles and smoke-tests an
inactive release, and switches only on success — rolling back otherwise.
The tests around all of it: quantized kernels, KV math, sessions, server
behaviour, stop conditions, the wrapper, installer rollback, and PATH setup.
macOS on Apple Silicon (arm64) runs natively via the one-line installer,
with at least 16 GB of RAM. Linux and Windows run via Docker, from the
repository. Every measurement below is from the machine named beside it:
Platform
How
Measured decode
Verified on
macOS, Apple Silicon
native installer
5–7 tok/s
one fanless 16 GB M3 MacBook Air
Windows, x86_64
Docker in WSL2
1.26 tok/s
one ASUS Zenbook, i7-1260P, 16 GB
Linux, x86_64
Docker
not yet measured
build + test suite green on Debian 12, Ubuntu 26.04
macOS is the fast path; x86 is ~4–5x slower, for a known and fixable reason.
The engine picks its vectorised kernels at compile time and the build passes no
-march, so on x86 the AVX2 kernels are compiled out and it falls back to a
scalar loop — measured 7.6x slower on identical hardware. The Zenbook above
has AVX2 and gets none of it. The fix is runtime CPU dispatch; it is tracked
in the repository as G10/H2.
Output is identical across platforms: the same prompt and seed returns the same
tokens on macOS/NEON, arm64 Linux, and x86_64 Linux, at the same ~3.84 GB
footprint. Only speed differs.
“CPU-only” describes the inference backend: NEON/SDOT on Apple Silicon, the
scalar or AVX2 path on x86, optional OpenMP — not Metal, and not CUDA. It does
not mean an arbitrary 16 GB laptop runs this well. It needs an NVMe SSD
(expert weights stream from disk on every token; a host bind mount instead of a
named Docker volume costs ~6x, and a hard drive is unusable) and ≥6 GB given
to the Docker VM on Linux/Windows. M1/M2 and other machines still need
independent release validation.
Install
macOS (Apple Silicon)
curl -fsSL https://huggingface.co/deepanwa/Samosa-Chat-Qwen3.6-35B-A3B-group32/resolve/main/install.sh | sh
Then open a new terminal and ask it something:
samosa "explain how DNS works"
Allow roughly 30 GB free. The installer downloads into an inactive versioned
release, verifies every byte size and SHA-256 digest, compiles the C engine,
and smoke-tests it before atomically switching the live release. A corrupt or
interrupted upgrade leaves the prior release untouched. No administrator
rights are required.
Linux and Windows
Use Docker, from the GitHub repository — the installer above is macOS-only:
Then open http://127.0.0.1:8642. samosa pull fetches these model files into
the volume and verifies each one by size and SHA-256; an interrupted download
resumes if you re-run it. On Windows this runs inside WSL2 — the full
step-by-step, including the Docker VM memory requirement, is in the
repository README.
Expect ~1.3 tok/s on x86 today rather than the Mac's 5–7 — see
Supported hardware for why.
Everything installs under ~/.samosa — the samosa command at
~/.samosa/bin/samosa, the active release at ~/.samosa/current, and your
conversations at ~/.samosa/chats. Nothing is installed system-wide. The
installer adds ~/.samosa/bin to PATH via one line in your shell rc file,
which only affects terminals opened afterwards — hence the new terminal
above. If samosa is still not found, either
export PATH="$HOME/.samosa/bin:$PATH" in the current shell, or run it
directly as ~/.samosa/bin/samosa "how are you". To uninstall, delete
~/.samosa and remove that line.
Two ways to use it
The terminal is the normal way to use Samosa.
sh
1samosa "explain how a hash table handles collisions"
2samosa --continue "and which strategy does Python use?"
3samosa --think "solve this logic puzzle"
4samosa --think-code "build a responsive settings page"
5samosa --fast "summarize this design"
6samosa --max-tokens 2048 "write a long explanation"
7samosa doctor
The browser app is a demo at this point. It works, and it is the nicest way
to watch answers stream and see the model's reasoning, but it exists to show the
engine off rather than as a polished interface. samosa app starts one resident
local model process and opens it; samosa serve --stop stops it. It provides:
token-by-token answer streaming;
a separate collapsible thinking view;
direct, general-thinking, and precise-code profiles;
stop/cancel, seed, and output-ceiling controls;
browser-local conversation display history plus sealed server snapshots;
tokens/s, RSS, and thinking-closure telemetry;
responsive light/dark presentation with no framework, CDN, analytics, or
external script.
The complete UI and logo total 181,552 bytes in this release.
Local API
The resident server exposes local JSON/SSE chat completions at
POST /v1/chat/completions, plus health, model discovery, cancellation, and
clean shutdown endpoints. Requests are serialized through a bounded queue;
the model is never mutated by concurrent generations.
Generation stops when Qwen emits its end-of-turn token. The outer ceiling is
8,192 new tokens. General reasoning defaults to a 1,024-token internal budget
and precise code to 2,048. On budget exhaustion Samosa appends Qwen's trained
early-stop transition before </think> instead of injecting a bare control
token. Natural closure, budget transition, repetition stop, cancellation, and
length stop remain distinct in telemetry.
Saved history, the newly tokenized turn, and the requested completion ceiling
must fit a 24,576-token total context cap. Samosa rejects an oversized request
before queueing or allocating KV state. One conversation state is resident at
a time; other conversations remain as sealed snapshots on disk.
Stopping a generation is safe for the saved conversation: a cancelled turn is
persisted only up to its last complete sentence, so an interrupted answer does
not poison later turns. If it produced no complete sentence, the previous
snapshot is kept and session_saved reports false.
Model layout
Qwen describes the upstream language model as 35B parameters total with 3B
activated per token, 40 layers, 256 routed experts, and 8 routed plus 1 shared
expert active per MoE layer. Samosa keeps dense weights resident and streams
only selected routed experts from SSD.
file
bytes
purpose
experts.bin
20,942,159,872
group-32 symmetric-q4 routed experts
resident.safetensors
3,015,056,192
resident dense row-q8 weights
tokenizer_qwen36.json
28,142,621
248,320-token vocabulary
app.html
~32 KB
complete local chat UI
samosa-chat.png
~149 KB
transparent app mascot
engine/
~300 KB
complete C source
release-manifest.tsv
—
SHA-256, size, and install path for every file
The previous whole-row q4 artifact had materially higher measured weight
reconstruction error. This release uses group size 32. The format is custom to
Samosa and is not a GGUF or Transformers-loadable checkpoint.
Measured behavior
All measurements below are from the 16 GB M3 MacBook Air reference machine.
They are workload-specific observations, not guarantees for every Mac.
workload
threads
measured result
group-32 direct control
2
7.27 tok/s
live 132-token app turn
2
7.11 tok/s
933-token general-thinking control
2
4.85 tok/s
selective-precision 5,000-token WebDev control
4
6.47 tok/s
A bounded release-path check served the real UI and logo, streamed exactly
Samosa app works locally., stopped naturally on Qwen's end-of-turn token,
saved the conversation snapshot, decoded at 5.13 tok/s, and peaked at 3.28 GB
RSS. The full bounded test suite covers grouped quantization, sessions, server
behavior, special-token stopping, cancellation, the wrapper, and corrupt
atomic-upgrade rollback.
For the resident app, macOS physical footprint measured 2.51 GiB after model
load and 4.07 GiB after a real two-turn continuation; the independent
footprint tool reported 2,566 MiB and 4,170 MiB. A live Activity Monitor
comparison also matched the value displayed by Samosa. GQA KV grows by about
40 KiB per context token, so the 24,576-token cap bounds that variable
component to about 960 MiB. The allocator may retain its high-water mark, but
chat history cannot grow KV memory beyond the cap.
Long thinking is expensive: one 933-token group-32 control reread 376.77 GB of
expert data. More output tokens can be useful, but they increase time, power
draw, and heat even when RAM stays bounded. SSD reads do not consume drive
endurance (TBW is write-only); the genuine cost is speed and energy.
Privacy and storage
Normal inference does not contact Hugging Face or any other remote service.
The installer contacts Hugging Face only to download public release files.
The listener is hard-bound to IPv4 loopback.
Model files are read-only during inference.
A continuing conversation atomically replaces a roughly 63–70 MB sealed
session snapshot at turn end.
Display transcripts are currently stored in the browser's local storage.
Known limitations
Only Apple Silicon macOS is supported; only one M3 16 GB machine has
completed the full test program so far.
Metal acceleration is not implemented. The CPU path is partly constrained
by repeated SSD expert reads; a future Metal backend must be measured
end-to-end rather than assumed faster.
Group-32 improves the original quantization error, but broad benchmark and
upstream-parity coverage remains incomplete.
Deleting a conversation in the UI removes its browser transcript but does
not yet remove the sealed server snapshot.
In-RAM conversation slots, document chat, and web access remain roadmap
work.
Text only: the Qwen vision tower, image/audio/video input, and tool calling
are not included.
SSD speed is the primary performance bottleneck. Routed experts are streamed
from storage on every token; NVMe bandwidth directly determines tok/s. Measured:
2.3+ GB/s (native NVMe) gives 5-7 tok/s; ~0.5 GB/s (a Docker host bind mount
through virtiofs) gives ~0.9 tok/s.
SSD reads do not consume drive endurance. Flash endurance is rated in TBW
(Terabytes Written, JEDEC JESD218) and DWPD (Drive Writes Per Day); program/
erase cycles wear NAND, reads do not. Read disturb is real but its thresholds
are orders of magnitude beyond this workload. Stated from the endurance rating
definition, not from a SMART measurement on the reference machine — Apple
Silicon's internal NVMe does not expose endurance counters. The genuine costs of
streaming are speed, power, and heat.
