AWQ INT4 checkpoint of LiquidAI/LFM2.5-8B-A1B, calibrated on agentic/chat-style prompts for vLLM serving. Uses compressed-tensors format — auto-detected by vLLM 0.24+.

Model Details

Property	Value
Base Model	LiquidAI/LFM2.5-8B-A1B
Total Parameters	8.3B
Active Parameters	1.5B (MoE)
Architecture	LFM2.5 MoE (24 layers: 18 LIV conv + 6 GQA)
Context Length	128K (native), benchmarked to 4096
Vocabulary	128,000

Quantization

Setting	Value
Method	AWQ W4A16 asymmetric (compressed-tensors)
Group size	128
Calibration	Agentic + chat prompts (tool-use, support dialogues)
Samples	256
Max seq len	512
Disk size	~5.6 GB
Runtime	vLLM 0.24+ (auto-detected)

Conv layers, MoE gates, dense FFN in layers 0–1, and lm_head kept in higher precision. Post-quant export renames linearized MoE expert keys for vLLM compatibility.

Benchmarks

Hardware: NVIDIA GeForce RTX 5060 Ti (16 GB), vLLM 0.24, CUDA 13.3, flash-attn

Speed (single request, 256 output tokens)

Metric	Value
TTFT (Time to First Token)	0.020s (20ms)
Median Decode TPS	194 tok/s
Time to 50 tokens	0.233s

Concurrency Scaling (128 output tokens)

Concurrency	TTFT	Median TPS	Aggregate TPS
1	0.019s	194	194
2	0.030s	157	313
4	0.041s	132	528
6	0.057s	108	646
8	0.069s	112	898
10	0.080s	96	960
12	0.089s	94	1,129
16	0.111s	95	1,511
20	0.130s	87	1,742

Peak aggregate throughput: 1,742 tok/s at concurrency 20, zero errors across all levels.

Context Length Scaling (single request, 128 output tokens)

Prompt Tokens	TTFT	Decode TPS
128	0.024s	193
512	0.050s	193
1,024	0.077s	193
2,048	0.129s	191
4,096	0.255s	190

Decode speed stays flat across context lengths — TTFT scales linearly with prompt size as expected.

Quality

LFM2.5-8B-A1B is a reasoning-only model (it always emits a <think> chain of thought — there is no non-thinking mode), so it was not run through this project's local chat_core lighteval suite: those numbers are collected in native mode across a mix of thinking and non-thinking models, and a reasoning-only model isn't apples-to-apples there, nor tractable at full sample count on a single 16 GB GPU.

For quality, refer to LiquidAI's published benchmarks for the base model (LiquidAI/LFM2.5-8B-A1B): IFEval 91.84, IFBench 56.47, Multi-IF 79.93, MATH500 88.76, AIME25 42.53, BFCLv3 64.79. This INT4 (W4A16) checkpoint targets vLLM serving throughput; expect small quantization deltas from those BF16 figures. Speed/concurrency for this checkpoint is benchmarked below.

vLLM Usage

bash

vllm serve LostGentoo/LFM2.5-8B-A1B-AWQ-INT4-agentic \
  --trust-remote-code \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.75

No --quantization flag needed — vLLM auto-detects compressed-tensors format.

OpenAI-compatible test

bash

curl http://127.0.0.1:8012/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"LFM2.5-8B-A1B-AWQ-INT4-agentic","messages":[{"role":"user","content":"ping"}],"max_tokens":32}'

Troubleshooting

CUDA/nvcc errors: Set CUDA_HOME=/opt/cuda (flashinfer requires CUDA toolkit)
ninja not found: Install ninja-build (flashinfer JIT dependency)
flashinfer JIT failures: Set VLLM_ATTENTION_BACKEND=FLASH_ATTN to bypass
Model config mismatch: This model uses compressed-tensors quant format — do NOT pass --quantization awq

License

Same as base model: LiquidAI/LFM2.5-8B-A1B License