The packaged text runtime uses the GPTQ INT4 AX650 build.
The compiled text profile uses prefill_len=256, kv_cache_len=8191, and a last_kv_cache_len ladder up to 6144 (P6K).
The packaged vision runtime uses a fixed-shape 448x448 MiniCPM-V-4.6 vision encoder.
The package supports text-only chat, single-image understanding, and video understanding through axllm.
The default runtime config keeps video content available for later text-only follow-up turns.
The package includes sample image and video assets for board-side validation.

Supported Platform

AX650 / NPU3

Performance

All measurements below were taken on AX650 / NPU3 with the packaged axllm serve runtime. TTFT stands for time to first token.

For the rows below, TTFT is measured end-to-end from request arrival to the first generated token. The image row uses the packaged assets/sample.png image. The video row uses the packaged assets/red-panda-openai.mp4 sample with video:/path/to/video.mp4:2. The long-context row uses a text-only retrieval prompt that fills the packaged P6K profile to 6260 input tokens and asks the model to recover an exact code placed near the beginning of the prompt.

Scenario	Input tokens	Prefill chunks	TTFT	Decode
Text-only smoke prompt	`23`	`1 x 256`	`313.71 ms`	`n/a (single-token reply)`
Image prompt	`88`	`1 x 256`	`307.94 ms`	`17.84 token/s`
Video prompt	`1207`	`5 x 256`	`1661.86 ms`	`18.11 token/s`
Long-context text retrieval	`6260`	`25 x 256`	`10331.60 ms`	`18.28 token/s`

The packaged runtime uses the following context layout:

prefill_len=256
kv_cache_len=8191
prefill_max_token_num=6400

In the validated long-context text retrieval run above, the model returned the expected exact code from the beginning of the prompt.

Multi-turn Long Dialogue Validation

The packaged text runtime was validated with an accumulated 6-turn conversation that filled the history close to the P6K profile and then repeatedly recalled facts introduced in the earliest turn.

Board-side results showed that:

the earliest facts remained correct across all six turns
short recall turns stayed around 506-508 ms TTFT
requests beyond the current context budget returned a clear context-length error instead of hanging or producing corrupted output

Video Follow-up Validation

The packaged axllm binary was also validated with a video-first conversation followed by text-only turns that continue discussing the same video content.

Board-side results:

text

TURN1: The video shows a red panda and another red panda playing with a rope.
TURN2: 2
TURN3: rope

During this validation, later text-only turns answered correctly without re-sending the video input.

Startup Runtime Footprint

Item	Value
`Flash total (.axmodel files + embedding bin)`	`1.80 GiB` (`1843.49 MiB`)
`Package flash total (excluding runtime-generated vision_cache/)`	`1.82 GiB` (`1861.63 MiB`)
`Runtime CMM increment during board-side startup`	`1.57 GiB` (`1611.54 MiB`)
`Runtime process RSS after startup`	`72.80 MiB` (`74544 KB`)

The runtime CMM increment above was measured on a shared AX650 board by comparing /proc/ax_proc/mem_cmm_info before and after service startup. On the validated board, CMM usage increased from 275020 KB to 1925236 KB.

Vision Encoder Latency

The packaged vision encoder is the same fixed-shape 448x448 AX650 encoder used by the current AX650 GPTQ package, so its standalone latency is unchanged.

Model	Resolution	Soft Tokens	Time (ms)
`minicpmv4_6_vision_448.axmodel`	`448x448`	`64`	`235.285 ms avg`

For the shipped vision encoder:

vision_width = 448
vision_height = 448
vision_patch_size = 14
patch grid = (448 / 14) x (448 / 14) = 32 x 32
raw patch tokens = 32 x 32 = 1024
packaged visual compression path = 16x
Soft Tokens = 1024 / 16 = 64

Soft Tokens is fixed by the packaged vision encoder in this repository and is not a runtime-configurable value.

Package Layout

text

.
├── README.md
├── .gitattributes
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── assets/
│   ├── openai_api_demo.png
│   ├── red-panda-openai.mp4
│   ├── sample.png
│   ├── sample_1.png
│   └── smoke_image.png
├── minicpmv4_6_vision_448.axmodel
├── qwen3_5_text_p256_l0_together.axmodel
├── ...
├── qwen3_5_text_p256_l23_together.axmodel
├── qwen3_5_text_post.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── config.json
├── post_config.json
└── minicpm_v46_tokenizer.txt

This package keeps the runtime files at the repository root so it can be served directly by axllm. Board-side multimodal runs may create a temporary vision_cache/ directory at the repository root. That directory is runtime-generated and should not be included in release uploads.

Sample Image

The package includes a sample image for board-side validation: assets/sample.png

Sample Video

The package includes a sample video for board-side validation:

assets/red-panda-openai.mp4

Download the Package

shell

mkdir -p AXERA-TECH/MiniCPM-V-4.6-GPTQ-INT4-C256-P6K-CTX8K
cd AXERA-TECH/MiniCPM-V-4.6-GPTQ-INT4-C256-P6K-CTX8K
hf download AXERA-TECH/MiniCPM-V-4.6-GPTQ-INT4-C256-P6K-CTX8K --local-dir .

Install `axllm`

Option 1: use the validated binary included in this repository:

chmod +x ./bin/axllm

Option 2: install axllm from the public repository:

shell

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

Option 3: install with a one-line command:

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

Run on the Board

The package root is already arranged for axllm, so no extra runtime path arguments are required.

For multimodal testing, you can use ./assets/sample.png or ./assets/red-panda-openai.mp4.

./bin/axllm run .

In interactive mode:

press Enter directly for text-only chat
input an image path for single-image chat
input video:/path/to/frames_dir or video:/path/to/video.mp4 for video chat

Serve with `axllm`

From the package root on the board:

./bin/axllm serve . --port 8000

Expected model id:

AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K

Health check:

bash

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models

Packaged context profile:

max_token_len = 8191
kv_cache_size = 512, kv_cache_num = 8191
prefill_token_num = 256
prefill_max_token_num = 6400

Request Examples

Text Request

bash

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K",
    "messages": [
      {"role": "user", "content": "1+1 equals what? Reply with digits only."}
    ],
    "max_tokens": 32,
    "temperature": 0
  }'

Image Request

axllm serve expects the image payload as a data URI. From the package root:

bash

python3 - <<'PY'
import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

img = Path("assets/sample.png").read_bytes()
payload = {
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Please briefly describe this image."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64," + base64.b64encode(img).decode()
                    },
                },
            ],
        }
    ],
    "max_tokens": 64,
    "temperature": 0,
}
req = Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(req, timeout=300) as resp:
    print(resp.read().decode())
PY

Example output:

json

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The image shows a colorful, cartoon-style red lobster or lobster-like character with a cheerful expression, raised claws, and a dynamic pose."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K",
  "object": "chat.completion"
}

Video Request

axllm serve accepts either a frames directory or a raw video file. To test the packaged sample video from the package root:

VIDEO_PATH="$(pwd)/assets/red-panda-openai.mp4"

Then use video:${VIDEO_PATH}:2 in the request payload:

bash

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "{
    \"model\": \"AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K\",
    \"messages\": [
      {
        \"role\": \"user\",
        \"content\": [
          {\"type\": \"image_url\", \"image_url\": {\"url\": \"video:${VIDEO_PATH}:2\"}},
          {\"type\": \"text\", \"text\": \"Describe this video briefly.\"}
        ]
      }
    ],
    \"max_tokens\": 128
  }"

Example output:

json

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The video shows two red pandas, one on a branch and the other on the ground, interacting with a rope."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K",
  "object": "chat.completion"
}

Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

Original Hugging Face model: openbmb/MiniCPM-V-4.6-GPTQ
AXERA conversion and deployment workflow: AXERA-TECH/MiniCPM-V-4.6.axera

Discussion

GitHub Issues
QQ group: 139953715

AXERA-TECH/MiniCPM-V-4.6-GPTQ-INT4-C256-P6K-CTX8K

MiniCPM-V-4.6-GPTQ INT4 C256 P6K CTX8K on AXERA NPU

Supported Platform

Performance

Multi-turn Long Dialogue Validation

Video Follow-up Validation

Startup Runtime Footprint

Vision Encoder Latency

Package Layout

Sample Image

Sample Video

Download the Package

Install axllm

Run on the Board

Serve with axllm

Request Examples

Text Request

Image Request

Video Request

Conversion References

Discussion

AXERA-TECH/
MiniCPM-V-4.6-GPTQ-INT4-C256-P6K-CTX8K

Install `axllm`

Serve with `axllm`