This HuggingFace repository is a compatibility mirror, not a loadable model. There is no
.safetensors / .bin / .gguf here — this repo is just this README.
python
1# ❌ These will all FAIL — this repository contains no model artifact:2from transformers import AutoModelForCausalLM
3model = AutoModelForCausalLM.from_pretrained("unnowataru/granite-4.1-8b-tt-metal")45from vllm import LLM
6llm = LLM(model="unnowataru/granite-4.1-8b-tt-metal")78# Any HF Inference API / TGI / generic loader with this repo ID will also fail.
Use the upstream IBM repository instead for the actual weights:
python
1# ✅ Correct: weights come from IBM's Apache-2.0 repository2from transformers import AutoModelForCausalLM
3model = AutoModelForCausalLM.from_pretrained(4"ibm-granite/granite-4.1-8b",5 revision="1504002f650e656a0a3789d99574df12e3e94ed0"# pinned by this port6)
library_name, pipeline_tag, and base_model are intentionally omitted from this repo's
metadata because it contains no loadable model artifact and no derivative weights.
Community, non-affiliated port.
IBM, Tenstorrent, and Networld Corporation were not involved in the
development, validation, or publication of this project and do not endorse it.
Model: IBM Granite 4.1-8B instruction-tuned model (the no-suffix repository; the -base
repository denotes the base checkpoint).
Copyright: International Business Machines Corporation.
License: Apache-2.0.
Weights are not modified or redistributed by this repository. Download and use IBM's weights directly.
Supported hardware and software stack
All measurements used one pinned software/firmware stack but two device configurations:
N150 for single-device numerical checks and N300 for serving and performance measurements.
Cross-stack and independent cross-device reproduction have not been tested; other revisions or
configurations may differ.
Component
Version / pin
Tenstorrent card
Wormhole b0
Device configurations (measured)
N150 for single-device numerical validation; N300 for serving and performance
Firmware bundle
19.6.0
KMD
2.7.0
tt-metal
commit 805f43d (specific main point, not a release tag)
ibm-granite/granite-4.1-8b snapshot 1504002f650e656a0a3789d99574df12e3e94ed0 (pin your HF_REVISION to this SHA for exact reproduction)
Port revision
release tag v0.1.0 (see the GitHub repo's tags for the corresponding Git commit SHA)
T3K (8-device) has not been measured.
Abbreviated integration path
This is a condensed integration outline. It starts after entering the pinned container
(see the GitHub repo's docs/stack.md
and the full reproduction procedure for the clean-host startup). For a clean-host reproduction
from scratch, follow the pinned full procedure in the GitHub repository rather than this excerpt.
bash
1# Assumes you are inside the pinned container (image digest above), with tt-metal at commit 805f43d.2exportTT_METAL_HOME=/home/container_app_user/tt-metal
3test"$(git -C "$TT_METAL_HOME" rev-parse --short HEAD)"="805f43d"||{4echo"tt-metal at unexpected revision; abort";exit15}67# Fetch the port at a specific release tag8git clone --branch v0.1.0 --depth 1 https://github.com/unnowataru/granite-4.1-tt-metal
9cd granite-4.1-tt-metal
1011# Overlay the 6-file Granite diff onto the tt-metal tree12cp overlays/tt_transformers/tt/*.py "$TT_METAL_HOME/models/tt_transformers/tt/"1314# Register TTGraniteForCausalLM in the vLLM(tt) plugin (idempotent)15python scripts/serving/patch_tt_py_granite.py
1617# Start the OpenAI-compatible server, pinning the HF model snapshot18exportHF_MODEL=ibm-granite/granite-4.1-8b
19exportHF_REVISION=1504002f650e656a0a3789d99574df12e3e94ed0
20exportMESH_DEVICE=N300
21bash scripts/serving/run_granite_server.sh
Note: the patch_tt_py_granite.py script targets the vLLM(tt) plugin bundled in the pinned
container image. If your plugin path or revision differs, the patcher should refuse to apply —
verify the run header before proceeding.
The full server setup, verification harness, and integration tests are documented in the GitHub
repository.
Observed checks (scope-limited; on the stack above)
These are the specific checks performed on the pinned stack, not general capability claims.
One full 40-layer teacher-forced forward pass completed on N150 (single-prompt, next-token
logit comparison against HF CPU bfloat16 reference): PCC = 0.999410, top1 match, top5 overlap 4/5.
One manual generation spot-check on N150 (approximately 280 prompt tokens, 130 generated
tokens, greedy, EOS-terminated): the preserved output was judged semantically responsive and
free of gibberish. This is not a long-context or model-quality evaluation.
Serving on N300 via vLLM(tt): both /v1/completions and /v1/chat/completions respond;
the chat_template round-trip between the two endpoints is byte-exact on the fixed test message.
One scripted structured-tool-calling fixture on N300 passed the following checks: the
function name, JSON-valid arguments, the expected argument value (city=Tokyo), the final
answer's reflection of the returned tool data, and the streaming form's reconstruction of a
valid tool call. This uses --tool-call-parser hermes (parser name matters; see
docs/traps.md).
OpenAI Python SDK smoke tests passed for the following seven tested operations: /v1/models,
system+user, multi-turn context in a scripted 2-turn exchange, streaming, four concurrent
requests, stop parameter, greedy determinism across two identical calls.
Not yet validated (open gaps, honestly disclosed)
Multi-device teacher-forced numerical comparison on N300 / T3K. The single-device (N150)
numerical pass does not validate the tensor-parallel path — TP introduces
mesh_partition / all_gather / all_reduce collective operations that the harness did not
cover.
Multi-prompt statistical numerical coverage. Only one prompt has been used for the teacher-
forced comparison. A defensible statistical claim requires many more independent trials.
Autoregressive long-generation agreement. Autoregressive token-by-token agreement over
hundreds to thousands of generated tokens has not been measured. The teacher-forced next-token-
logit comparison does not establish long-horizon greedy-decoding agreement.
Chat/instruct decoding path in the numerical harness. The generation spot-check ran with
instruct=0 (base decoding); the chat/instruct decoding path is only exercised by the serving
smoke, not by the numerical harness.
max_num_seqs = 8 on N300. One attempt ended with a chip-lock error consistent with a
stale lock from a prior process, before model capacity could be tested. This run provides no
evidence either for or against N300 KV-cache capacity at msq=8; a clean-start measurement is
still required.
Any T3K measurement. Not attempted in this session.
Percentile latency (p50 / p95 / p99). Only means are reported below.
Sustained-load characteristics. Runs are on the order of minutes, not hours.
Method: Python requests + streaming SSE from a Windows client to the container's docker bridge
IP on port 8000 via SSH tunnel; one warm-up request (16 tokens) per configuration. Sample sizes
range from 1 (single-observation concurrency runs) to 5 (baseline single-stream) per data point —
each row lists its own n below. Values are arithmetic means of that row's sample; variance and
percentiles are not analyzed.
Single-stream
Config
TTFT (s)
TPOT (ms/tok)
tok/s
n
N300, msq=2, mml=4096
0.126
42.7
23.5
5
N300, msq=4, mml=8192
0.117
40.7
24.8
2
Concurrent (N300)
Config
Concurrency
Aggregate tok/s
n
Notes
msq=2, mml=4096
2
41.3
1
1.76× single-stream aggregation observed
msq=2, mml=4096
4 (queue)
44.4
1
Two streams served immediately (~150 ms TTFT), two queued ~4.5 s
msq=4, mml=8192
3 (Hermes-agent delegation fit)
46.0
1
Prefill (msq=2 mml=4096, greedy /v1/completions)
Prompt tokens
TTFT_avg (s)
TPOT_avg (ms/tok)
n
64
0.66
31.9
3
512
0.85
33.0
3
1024
0.85
33.2
3
2048
1.14
33.1
3
3800
1.91
34.0
3
Across these sample means, TPOT ranged from 31.9 to 34.0 ms/token while TTFT increased at longer
prompt lengths. With only 3 runs per point and no variance analysis, these observations are
descriptive; prompt-length independence of TPOT and a causal prefill attribution to the TTFT
trend are not established.
Detailed methodology, raw evidence logs, and additional measurements: see
docs/performance.md
and the raw logs under docs/evidence/.
Kernel-imposed constraint worth knowing
The upstream tt_transformers model_config enforces supported_batches = {1, 2, 4, 8, 16, 32}.
max_num_seqs = 3, 5–7, etc. fail at startup with
ValueError: Batch size N not supported. No fine-grained tuning between the powers of two.
Reproduction
Every measured number in this card is tied to the revisions listed in the Supported stack table
above; the linked GitHub release contains the corresponding harness scripts and raw logs. The
full reproduction procedure — clean-host startup, dev-container launch, overlay + registration
patch, weight download with HF_REVISION pinning, server startup, smoke — is documented step-by-
step in the GitHub repository. See
docs/validation.md
and docs/performance.md
for links to each raw log.
Known limitations (in addition to Not yet validated above)
Single stack. All numbers anchor to the pinned software/firmware stack. Drift on any
component is uncharacterized. Upstream tt-metal moves rapidly.
No SLA. This is a community port, single maintainer. Response times on issues are
best-effort.
No re-hosted weights. Users must obtain the weights separately and comply with their
applicable license; this repository neither redistributes the weights nor grants additional
rights in them.
License, attribution, and non-affiliation
This repository's content: Apache-2.0.
Model weights: IBM's, Apache-2.0, not redistributed here.
Overlays under overlays/tt_transformers/tt/: Derivative Works of upstream tt_transformers
at tt-metal commit 805f43d. Each file carries a SPDX-License-Identifier: Apache-2.0
header and a "Modified for Granite 4.1" notice per Apache-2.0 §4(b).
See the GitHub repository's
NOTICE for the full
attribution list.
Non-affiliation and trademarks: this project is not endorsed by, or officially affiliated
with, International Business Machines Corporation, Tenstorrent Inc., or Networld Corporation.
Names such as IBM, Granite, Tenstorrent, Wormhole, and tt-metal are trademarks or
product names of their respective owners. These names are used only to identify the upstream
components; no trademark license, endorsement, or affiliation is claimed.
Citation
If you use this port in academic or industry work, please cite both the upstream Granite model
and this repository:
