Mistral-Nemo-Instruct-2407-FP8
This repository contains
RedHatAI/Mistral-Nemo-Instruct-2407-FP8
together with a Furiosa Executable Bundle (FXB) for running it on
FuriosaAI RNGD with
Furiosa-LLM.
The same model also runs on other frameworks (such as vLLM, SGLang, and
Transformers); for usage with those, see the upstream
RedHatAI/Mistral-Nemo-Instruct-2407-FP8
model card.
Overview
Mistral NeMo Instruct 2407 is a 12B instruction-tuned, auto-regressive dense
transformer developed jointly by Mistral AI and NVIDIA for assistant-style text
generation. This is the FP8-quantized variant published by RedHatAI, derived
from
mistralai/Mistral-Nemo-Instruct-2407.
Its intended use is the same as the upstream
RedHatAI/Mistral-Nemo-Instruct-2407-FP8.
- Architecture: Mistral NeMo (dense)
- Input / Output: Text / Text
- Supported Inference Engine: Furiosa LLM
- Supported Hardware: FuriosaAI RNGD
Quantization
The linear (Transformer block) weights are quantized to FP8 (static), and
activations use dynamic FP8 quantization at runtime. Activations are quantized
per token at inference time rather than with offline calibration. The KV cache
stays in 16-bit precision.
Features
Mistral-Nemo-Instruct-2407-FP8 provides non-reasoning, assistant-style text
generation. Its chat template supports system, user, and assistant messages.
Parallelism Strategy
On RNGD, Mistral-Nemo-Instruct-2407-FP8 runs with a tensor-parallel size of
16 PEs, which maps to two RNGD cards (8 PEs per card).
Usage
To run this model with Furiosa-LLM, follow the example commands below after
installing Furiosa-LLM and its prerequisites.
Launch the server
The simplest way to serve the model is:
1# Launch the server, listening on port 8000 by default
2furiosa-llm serve furiosa-ai/Mistral-Nemo-Instruct-2407-FP8
When the server is ready, you will see:
1INFO: Started server process [27507]
2INFO: Waiting for application startup.
3INFO: Application startup complete.
4INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Basic Usage
The server exposes an OpenAI-compatible API. You can send a request with curl:
1curl http://localhost:8000/v1/chat/completions \
2 -H "Content-Type: application/json" \
3 -d '{
4 "model": "furiosa-ai/Mistral-Nemo-Instruct-2407-FP8",
5 "messages": [{"role": "user", "content": "What is the capital of France?"}]
6 }' \
7 | python -m json.tool
Advanced Usage
Mistral-Nemo-Instruct-2407-FP8 is a non-reasoning chat model and does not
require a reasoning parser or model-specific chat-template flags. For general
serving options, see the
Furiosa-LLM Server guide.
Learn more
- Tool Calling — supported parsers, tool-choice options, and examples
- Furiosa-LLM Server (
furiosa-llm serve) — full OpenAI-compatible API reference and serving options
- RedHatAI/Mistral-Nemo-Instruct-2407-FP8 — upstream model card
