INT4-packed SafeTensors weights for Needle v1 — 26M parameters, encoder–decoder — ready to load into the needle-rs pure-Rust + WebAssembly runtime.
Looking for Needle v2?
needle-rs runs both generations. v2 needs no conversion and is not hosted
here — point the runtime straight at upstream's container:
Cactus-Compute/needle2 →
needle2.cact. This repository exists only because v1 shipped as
Flax/Pickle, which no Rust runtime can load.
Both run in the live demo — switch generation in the browser.
This is a format conversion. The model itself — its architecture, training procedure, dataset, and original weights — is the work of Cactus Compute (Henry Ndubuaku et al., 2026), released under MIT. Original repository: Cactus-Compute/needle.
If you build with these weights, you are building on Cactus's Needle. Please credit them in any publication, blog post, product, or downstream model that incorporates this work. Citation template below.
needle-rs >= 0.2.0 reads upstream's .cact container directly — one runtime,
both generations:
bash
1hf download Cactus-Compute/needle2 needle2.cact --local-dir weights/
23needle-rs --json weights/needle2.cact "What's the weather in Paris?"\4'[{"name":"get_weather","parameters":{"type":"object","properties":{"location":{"type":"string"}}}}]'5# → [{"name":"get_weather","arguments":{"location":"Paris"}}]
One runtime, both generations — NeedleEngine / NeedleWasm for v1 here, and
V2Engine / NeedleV2Wasm for v2. v2 is the newer, larger (45M) decoder-only
model and is where upstream development continues; prefer it for new work, and
use these v1 weights when you specifically want the smaller model.
Files
File
Size
Description
needle.safetensors
22.3 MB
INT4-packed attention/FFN weights + BF16 norms
vocab.txt
122 kB
8,192 SentencePiece pieces (TSV: piece\tscore)
config.json
320 B
Geometry, for tooling that expects a config file
banner.svg
5.3 kB
Repo banner
Model summary
Architecture
Encoder–decoder transformer (SAN)
Parameters
26M
Hidden size
512
Encoder / decoder layers
12 / 8
Attention heads (Q / KV)
8 / 4 (GQA, repeat=2)
Vocabulary
8,192 (SentencePiece BPE)
Max encoder length
1,024 tokens
Quantization
INT4 group-wise (group_size=32) for attention + FFN; BF16 for norms/embeddings
The SafeTensors file uses a custom I4 dtype for quantized kernels:
Group-wise INT4 with group_size=32, per-group scale = max|w| / 7, packed as nibbles (low nibble = even row, high nibble = odd row, per output column).
Non-kernel parameters (RMSNorm γ, gate vectors, embeddings) stored in BF16.
Model config ships as a separate config.json (the SafeTensors header carries no __metadata__ block). needle-rs does not read it — the engine derives geometry from tensor shapes — but it is there for tooling that expects one.
This format is consumed directly by needle-rs. It is not compatible with transformers, safetensors-rust direct loading without the needle-rs engine, or other generic SafeTensors consumers, because the I4 dtype is non-standard.
How to use
The intended runtime is needle-rs. The same weights work across all its deployment targets — native CLI, Rust API, C FFI, and browser/Node.js via WebAssembly.
12# Single inference3./needle-rs weights/needle.safetensors weights/vocab.txt \4"What's the weather in Paris?"\5'[{"name":"get_weather","parameters":{"type":"object","properties":{"location":{"type":"string"}}}}]'6# → [{"name":"get_weather","arguments":{"location":"Paris"}}]
Rust
rust
1useneedle_infer::NeedleEngine;23let engine =NeedleEngine::load(4"weights/needle.safetensors",5"weights/vocab.txt",6)?;7let result = engine.run(query, tools_json);8println!("{}", result.text);
Browser (WebAssembly)
js
1importinit,{NeedleWasm}from"needle-rs";23awaitinit();45constHF="https://huggingface.co/Abdalrahman/needle-rs-safetensors/resolve/main";67const[weights, vocab]=awaitPromise.all([8fetch(`${HF}/needle.safetensors`).then(r=> r.arrayBuffer()).then(b=>newUint8Array(b)),9fetch(`${HF}/vocab.txt`).then(r=> r.text()),10]);1112const engine =NeedleWasm.load(weights, vocab);13const result = engine.run("Book a flight from London to JFK tomorrow", toolsJson);14// → [{"name":"book_flight","arguments":{"origin":"London","destination":"JFK","date":"tomorrow"}}]
Live demo:needle-rs.pages.dev — the demo loads exactly these files from this repository.
Python
pip install needle-rs
python
1from needle_rs import NeedleEngine
23engine = NeedleEngine.load("weights/needle.safetensors","weights/vocab.txt")45# Single call6result = engine.run(7"Book a flight from London to JFK tomorrow",8'[{"name":"book_flight","parameters":{"type":"object","properties":{"origin":{"type":"string"},"destination":{"type":"string"},"date":{"type":"string"}}}}]',9)10# → [{"name":"book_flight","arguments":{"origin":"London","destination":"JFK","date":"tomorrow"}}]1112# Streaming (callback fires per token)13result = engine.run_stream(query, tools_json,lambda token_id, piece:print(piece, end="", flush=True))1415# Batch16results = engine.run_batch([("query1", tools1),("query2", tools2)])1718# Semantic tool retrieval. NOTE: the weights in THIS repository carry no19# contrastive head (339 tensors, none of them a projection head), so this20# returns [] and encode_contrastive() returns None. The API is here for21# checkpoints that do have the head.22ranked = engine.retrieve_tools(23"What's the weather in Paris?",24["Get current weather for a location","Book a flight","Send an email"],25 top_k=2,26)27# → [] with these weights
Multi-tool routing example
Needle is trained to pick the right tool from a list, not just fill a single tool's parameters:
An empty array — [] — is a deliberate abstention, not a failure to parse: the
model declined to route. v1 abstains readily as the catalogue grows or the
phrasing drifts, which is a real limitation of the 26M model rather than a bug in
the runtime. See Limitations.
Intended use
Client-side intent routing in web applications — decide which API endpoint to call before issuing the network request, with no server-side LLM.
Edge function dispatch — Cloudflare Workers, Vercel Edge, Deno Deploy, anywhere with a WASM engine and ≤30 MB of available memory.
On-device function calling in privacy-sensitive contexts (healthcare, legal, personal data) where sending user queries to a hosted LLM is unacceptable.
Embedded agents on hardware with enough RAM for the weights (≈30 MB working set including activations).
Tool retrieval — needle-rs exposes encode_contrastive() / retrieve_tools() for ranking a large tool catalogue before passing the top-K to the generator. This needs a checkpoint with a contrastive head; the weights in this repository do not have one, so both return empty on these files.
Limitations
Tool calling only. Needle is trained for the single task of mapping a query plus tool definitions to a JSON call. It is not a chat model and will not produce meaningful free-form text.
Single-shot. No multi-turn dialogue, no chain-of-thought, no tool-use feedback loop. Each call is independent.
English-trained. Multilingual behavior is not evaluated by upstream and is not guaranteed.
Greedy decoding only on the v1 path in needle-rs — stochasticity is undesirable for routing, so no sampling is exposed. (--temperature and --seed exist, but apply to v2 only.)
Encoder length ≤ 1,024 tokens. Long tool catalogues must be truncated or pre-filtered before being passed in.
Routing degrades as the catalogue grows. Measured on these weights: with
three tools, clean queries route correctly; with four, several queries that a
human would find unambiguous return [] instead, and one produced a malformed
call with a repeated argument key. Keep the catalogue small, or use Needle v2,
which handled the same four-tool cases correctly.
No contrastive head in this checkpoint, so the retrieval API cannot be used
to do that pre-filtering with these weights.
Small-model failure modes apply. Ambiguous queries, tools with overlapping descriptions, or unusual parameter schemas can produce unexpected routings. The constrained decoder guarantees syntactic validity, not semantic correctness.
Out of scope
General-purpose text generation, summarization, translation, or chat.
Long-context reasoning (>1,024 tokens of input).
Reasoning over tool outputs (the model produces calls, not results — your application executes the call and decides what to do with the response).
Production use in safety-critical domains without an evaluation suite covering the specific tool catalogue and query distribution.
Citation
If you publish or distribute work that uses these weights, please cite the upstream Needle paper/repository:
bibtex
1@misc{ndubuaku2026needle,
2 title = {Needle: A 26M-Parameter Tool-Calling Transformer},
3 author = {Ndubuaku, Henry and Mroz, Jakub and Mosoyan, Karen and Shemet, Roman
4 and Sandhu, Parkirat and Kumar, Satyajit and Cylich, Noah and Lee, Justin H.},
5 year = {2026},
6 url = {https://github.com/cactus-compute/needle}
7}
Optionally, cite the runtime if relevant to your work:
bibtex
1@misc{ibrahim2026needlers,
2 title = {needle-rs: Pure-Rust + WebAssembly Runtime for Needle},
3 author = {Ibrahim, Abdalrahman},
4 year = {2026},
5 url = {https://github.com/geekgineer/needle-rs}
6}
License
MIT — matching the upstream Needle release.
This repository performs only format conversion (Flax/Pickle → SafeTensors with INT4 packing) and quantization (BF16 → INT4 group-wise) of weights originally released by Cactus Compute under MIT. No retraining, fine-tuning, distillation, or modification of model behavior has been performed. All learned parameters originate from the upstream release.
Acknowledgments
The Needle model is the work of Henry Ndubuaku and the Cactus Compute team. Their decision to release the weights, training code, and dataset generation pipeline under MIT is what makes downstream runtimes like needle-rs possible. If this conversion is useful to you, please consider starring the upstream repository as well.
