The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

How to use

安装 axllm

方式一：克隆仓库后执行安装脚本：

shell

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

方式二：一行命令安装（默认分支 axllm）：

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）：

如果没有编译环境，请到： https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm 下载 最新 CI 导出的可执行程序（axllm），然后：

shell

chmod +x axllm
sudo mv axllm /usr/bin/axllm

模型下载（Hugging Face）

shell

mkdir -p AXERA-TECH/Qwen3.5-0.8B-AX650-C256-P6K-CTX8K
cd AXERA-TECH/Qwen3.5-0.8B-AX650-C256-P6K-CTX8K
hf download AXERA-TECH/Qwen3.5-0.8B-AX650-C256-P6K-CTX8K --local-dir .

# structure of the downloaded files
tree -L 3
`-- AXERA-TECH
    `-- Qwen3.5-0.8B-AX650-C256-P6K-CTX8K
        |-- qwen3_5_vision.axmodel
        |-- README.md
        |-- config.json
        |-- image.png
        |-- model.embed_tokens.weight.bfloat16.bin
        |-- post_config.json
        |-- qwen3_5_tokenizer.txt
        |-- qwen3_5_text_p128_l0_together.axmodel
        ...
        |-- qwen3_5_text_p128_l23_together.axmodel
        |-- qwen3_5_text_post.axmodel
        `-- vision_cache

3 directories, 39 files

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

运行（CLI）

这个模型是基于英文数据进行量化的，推荐使用英文进行问答

图像理解

shell

root@ax650 ~/yongqiang/lhj/Qwen3_5.AXERA/ax-llm # axllm run Qwen3.5-0.8B-AX650-C256-P6K-CTX8K/
19:14:47.144 INF Init:218 | LLM init start
19:14:47.144 INF Init:226 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
tokenizer_type = 3
 96% | ##############################   |  26 /  27 [28.70s<29.80s, 0.91 count/s] init post axmodel ok,remain_cmm(5497 MB)
19:15:15.845 INF Init:368 | max_token_len : 2047
19:15:15.845 INF Init:371 | kv_cache_size : 512, kv_cache_num: 2047
19:15:15.845 INF Init:374 | prefill_token_num : 128
19:15:15.845 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
19:15:15.845 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
19:15:15.845 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
19:15:15.845 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
19:15:15.845 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
19:15:15.845 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 768
19:15:15.845 INF Init:379 | grp: 7, prefill_max_kv_cache_num : 896
19:15:15.845 INF Init:379 | grp: 8, prefill_max_kv_cache_num : 1024
19:15:15.845 INF Init:379 | grp: 9, prefill_max_kv_cache_num : 1152
19:15:15.845 INF Init:384 | prefill_max_token_num : 1152
19:15:15.845 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  27 /  27 [28.71s<28.71s, 0.94 count/s] embed_selector init ok
19:15:17.168 INF Init:643 | Qwen-VL token ids: vision_start=248053 image_pad=248056 video_pad=248057
19:15:17.168 INF Init:668 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=1024, out_dtype=fp32
19:15:17.168 WRN Init:677 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
19:15:17.173 INF load_config:282 | load config: 
19:15:17.173 INF load_config:282 | {
19:15:17.173 INF load_config:282 |     "enable_repetition_penalty": false,
19:15:17.173 INF load_config:282 |     "enable_temperature": false,
19:15:17.173 INF load_config:282 |     "enable_top_k_sampling": true,
19:15:17.173 INF load_config:282 |     "enable_top_p_sampling": false,
19:15:17.173 INF load_config:282 |     "penalty_window": 20,
19:15:17.173 INF load_config:282 |     "repetition_penalty": 1.2,
19:15:17.173 INF load_config:282 |     "temperature": 0.9,
19:15:17.173 INF load_config:282 |     "top_k": 10,
19:15:17.173 INF load_config:282 |     "top_p": 0.8
19:15:17.173 INF load_config:282 | }
19:15:17.173 INF Init:448 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input image path (empty = text-only). Use "video:<frames_dir>" for video.
----------------------------------------
prompt >> describe this image
image >> image.png
15:52:39.666 INF EncodeForContent:919 | vision cache hit (disk): image.png
15:52:39.666 INF EncodeForContent:928 | vision cache hit (mem): image.png
15:52:39.669 INF SetKVCache:747 | prefill_grpid:3 kv_cache_num:256 precompute_len:0 input_num_token:168
15:52:39.669 INF SetKVCache:749 | current prefill_max_token_num:1152
15:52:39.669 INF SetKVCache:750 | first run
15:52:39.718 INF Run:805 | input token num : 168, prefill_split_num : 2
15:52:39.718 INF Run:845 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
15:52:39.718 INF Run:868 | prefill indices shape: p=0 idx_elems=128 idx_rows=1 pos_rows=3
15:52:39.833 INF Run:845 | prefill chunk p=1 history_len=128 grpid=2 kv_cache_num=128 input_tokens=40
15:52:39.833 INF Run:868 | prefill indices shape: p=1 idx_elems=128 idx_rows=1 pos_rows=3
15:52:39.968 INF Run:1010 | ttft: 249.97 ms
<think>

</think>

This image captures three astronauts in space, set against a forest backdrop that resembles a bamboo grove. The lighting is stark and monochromatic, giving the scene a surreal or high-contrast aesthetic.

- **The Astronaut:** In the foreground sits a tall astronaut, viewed from the front. This individual has long blonde hair and is wearing an all-white, puffy space suit with a dark helmet. Their stance is upright, suggesting they are standing in front of the camera or in the scene itself.
- **Background:** Behind the astronaut, the scene transitions into a dense forest. To the immediate left, the foliage is out of focus, creating a sense of depth. The background is filled with tall, thin bamboo stalks and their feathery fronds, which dominate the upper half of the frame. The right half of the background is also mostly obscured by the dense foliage. The entire image is rendered in grayscale with some bright white highlights on the left edge of the frame and a dark, shadowy left edge of the astronaut's head.

Overall, the composition creates a sense of vastness and scale through the large amount of foliage. The bright white highlights are likely from the sunlight hitting the left edge of the frame, which contrasts with the dark shadow on the astronaut.

15:52:51.081 NTC Run:1132 | hit eos,avg 23.76 token/s
15:52:51.081 INF GetKVCache:721 | precompute_len:300, remaining:852

视频理解

shell

prompt >> 描述视频内容
media >> video:assets/football.mp4:1
11:01:19.677 INF extract_video_frames_ffmpeg:299 | Extracting raw video container to frames: assets/football.mp4 -> /tmp/axllm_video_frames/video_205262774277_0 fps=0.99900099900099892
11:01:35.462 INF collect_video_frame_paths:379 | Video fps sampling: path=assets/football.mp4 fps=1 duration=60.060s target_frames=60 selected=60
11:01:45.789 INF SetKVCache:2543 | decode_grpid:2 prefill_grpid:4 history_cap:0 total_cap:256 symbolic_cap:1 precompute_len:0 input_num_token:4344 prefer_symbolic_group:0
11:01:45.790 INF SetKVCache:2565 | current prefill_max_token_num:6400
11:01:45.887 INF SetKVCache:2581 | first run
11:01:45.929 INF Run:2738 | input token num : 4344, prefill_split_num : 17
11:01:45.929 INF Run:2818 | prefill chunk p=0 history_len=0 grpid=4 kv_cache_num=0 input_tokens=256
11:01:46.184 INF Run:2818 | prefill chunk p=1 history_len=256 grpid=6 kv_cache_num=512 input_tokens=256
11:01:46.471 INF Run:2818 | prefill chunk p=2 history_len=512 grpid=7 kv_cache_num=768 input_tokens=256
11:01:46.766 INF Run:2818 | prefill chunk p=3 history_len=768 grpid=8 kv_cache_num=1024 input_tokens=256
11:01:47.068 INF Run:2818 | prefill chunk p=4 history_len=1024 grpid=9 kv_cache_num=1280 input_tokens=256
11:01:47.375 INF Run:2818 | prefill chunk p=5 history_len=1280 grpid=10 kv_cache_num=1536 input_tokens=256
11:01:47.694 INF Run:2818 | prefill chunk p=6 history_len=1536 grpid=11 kv_cache_num=1792 input_tokens=256
11:01:48.027 INF Run:2818 | prefill chunk p=7 history_len=1792 grpid=12 kv_cache_num=2048 input_tokens=256
11:01:48.375 INF Run:2818 | prefill chunk p=8 history_len=2048 grpid=13 kv_cache_num=2304 input_tokens=256
11:01:48.721 INF Run:2818 | prefill chunk p=9 history_len=2304 grpid=14 kv_cache_num=2560 input_tokens=256
11:01:49.070 INF Run:2818 | prefill chunk p=10 history_len=2560 grpid=15 kv_cache_num=2816 input_tokens=256
11:01:49.423 INF Run:2818 | prefill chunk p=11 history_len=2816 grpid=16 kv_cache_num=3072 input_tokens=256
11:01:49.784 INF Run:2818 | prefill chunk p=12 history_len=3072 grpid=17 kv_cache_num=3328 input_tokens=256
11:01:50.142 INF Run:2818 | prefill chunk p=13 history_len=3328 grpid=18 kv_cache_num=3584 input_tokens=256
11:01:50.507 INF Run:2818 | prefill chunk p=14 history_len=3584 grpid=19 kv_cache_num=3840 input_tokens=256
11:01:50.882 INF Run:2818 | prefill chunk p=15 history_len=3840 grpid=20 kv_cache_num=4096 input_tokens=256
11:01:51.266 INF Run:2818 | prefill chunk p=16 history_len=4096 grpid=21 kv_cache_num=4352 input_tokens=248
11:01:51.691 INF Run:3045 | ttft: 5761.65 ms
<think>11:01:51.691 INF Run:3076 | VLM decode positions: rope_start=4212 dense_kv_start=4344


</think>

这是一个足球比赛的集锦片段，主要展示了巴塞罗那队（身穿蓝红条纹球衣）与塞维利亚队（身穿白色球衣）在2019年10月1日的一场比赛中。

- **比赛信息**：根据屏幕左上角的文字，这场比赛是巴塞罗那队主场对阵塞维利亚队，比分是6-1，比赛日期是2019年10月1日。
- **比赛场景**：
    - 比赛发生在巴塞罗那主场体育场。
    - 比赛进行到第10分钟，比分是6-1。
    - 画面中可以看到巴塞罗那队的球员在进攻，他们正在尝试射门。
    - 一个球员在射门，但被塞维利亚队的门将成功扑救。
    - 随后，巴塞罗那队的球员庆祝进球，他们拥抱在一起。
- **观众**：在比赛的背景中，可以看到观众席上坐满了观众，他们正在观看比赛。
- **解说**：在画面的右上角，有一个小窗口，显示了一位正在解说比赛的解说员。他穿着黄色的衣服，戴着黑色的帽子，正在用手势和语言进行解说。

总的来说，这是一个足球比赛的精彩集锦，展示了巴塞罗那队的一次进球和庆祝。

11:02:07.175 NTC Run:3445 | hit eos,decode avg 16.53 token/s
11:02:07.175 INF GetKVCache:2496 | precompute_len:4601, remaining:1799 (tracked)
prompt >> 视频中展示了几次进球
media >> 
11:02:15.358 WRN build_qwen_cached_text_turn_tokens:1624 | Qwen cached text turn full-template prefix mismatch: offset=4386 cached=4601 full=4616, fallback to suffix
11:02:15.358 INF build_qwen_cached_text_turn_tokens:1628 | cached token window near mismatch: [4382:15='0', 4383:96212='月', 4384:16='1', 4385:95971='日', *4386:127241='的一场', 4387:104955='比赛中', 4388:1710='。', 4389:271='
11:02:15.358 INF build_qwen_cached_text_turn_tokens:1628 | 
11:02:15.358 INF build_qwen_cached_text_turn_tokens:1628 | ', 4390:12='-']
11:02:15.358 INF build_qwen_cached_text_turn_tokens:1630 | full-template token window near mismatch: [4382:15='0', 4383:96212='月', 4384:16='1', 4385:95971='日', *4386:96424='的一', 4387:133765='场比赛中', 4388:1710='。', 4389:271='
11:02:15.358 INF build_qwen_cached_text_turn_tokens:1630 | 
11:02:15.358 INF build_qwen_cached_text_turn_tokens:1630 | ', 4390:12='-']
11:02:15.359 INF build_qwen_cached_text_turn_tokens:1647 | Qwen cached text turn uses suffix fallback: input_tokens=15 head=[248046='<|im_end|>', 198='
11:02:15.359 INF build_qwen_cached_text_turn_tokens:1647 | ', 248045='<|im_start|>', 846='user', 198='
11:02:15.359 INF build_qwen_cached_text_turn_tokens:1647 | ', 133866='视频中', 98845='展示', 102130='了几', ...]
11:02:15.359 INF Run:3507 | reuse cached KV for text-only turn: cached_tokens=4601 append_contents=1 input_tokens=15, skip vision Prepare
11:02:15.359 INF SetKVCache:2543 | decode_grpid:2 prefill_grpid:22 history_cap:4608 total_cap:4864 symbolic_cap:4608 precompute_len:4601 input_num_token:15 prefer_symbolic_group:0
11:02:15.359 INF SetKVCache:2565 | current prefill_max_token_num:1792
11:02:15.529 INF Run:3658 | VLM cached mRoPE positions: prefill_rope_start=4469 dense_kv_start=4601 input_tokens=15
11:02:15.541 INF Run:2738 | input token num : 15, prefill_split_num : 1
11:02:15.542 INF Run:2818 | prefill chunk p=0 history_len=4601 grpid=22 kv_cache_num=4608 input_tokens=15
11:02:15.970 INF Run:3045 | ttft: 428.46 ms
<think>11:02:15.970 INF Run:3076 | VLM decode positions: rope_start=4484 dense_kv_start=4616


</think>

根据视频内容，我们可以看到以下两次进球：

1.  **第一次进球**：
    - 在视频的前半部分，巴塞罗那队的球员（身穿蓝红条纹球衣）在进攻。
    - 一个球员在射门，但被塞维利亚队的门将成功扑救。
    - 随后，巴塞罗那队的球员庆祝进球，他们拥抱在一起。

2.  **第二次进球**：
    - 在视频的中间部分，巴塞罗那队的球员再次进行进攻。
    - 一个球员在射门，但被塞维利亚队的门将成功扑救。
    - 随后，巴塞罗那队的球员庆祝进球，他们拥抱在一起。

因此，视频中展示了**两次**进球。

11:02:24.906 NTC Run:3445 | hit eos,decode avg 16.67 token/s
11:02:24.907 INF GetKVCache:2496 | precompute_len:4766, remaining:1634 (tracked)
prompt >> 两次进球分别是几号球员进的
media >> 
11:03:01.365 WRN build_qwen_cached_text_turn_tokens:1624 | Qwen cached text turn full-template prefix mismatch: offset=4386 cached=4766 full=4783, fallback to suffix
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1628 | cached token window near mismatch: [4382:15='0', 4383:96212='月', 4384:16='1', 4385:95971='日', *4386:127241='的一场', 4387:104955='比赛中', 4388:1710='。', 4389:271='
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1628 | 
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1628 | ', 4390:12='-']
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1630 | full-template token window near mismatch: [4382:15='0', 4383:96212='月', 4384:16='1', 4385:95971='日', *4386:96424='的一', 4387:133765='场比赛中', 4388:1710='。', 4389:271='
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1630 | 
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1630 | ', 4390:12='-']
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1647 | Qwen cached text turn uses suffix fallback: input_tokens=17 head=[248046='<|im_end|>', 198='
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1647 | ', 248045='<|im_start|>', 846='user', 198='
11:03:01.365 INF build_qwen_cached_text_turn_tokens:1647 | ', 109039='两次', 111687='进球', 110060='分别是', ...]
11:03:01.365 INF Run:3507 | reuse cached KV for text-only turn: cached_tokens=4766 append_contents=1 input_tokens=17, skip vision Prepare
11:03:01.365 INF SetKVCache:2543 | decode_grpid:2 prefill_grpid:23 history_cap:4864 total_cap:5120 symbolic_cap:4864 precompute_len:4766 input_num_token:17 prefer_symbolic_group:0
11:03:01.365 INF SetKVCache:2565 | current prefill_max_token_num:1536
11:03:01.536 INF Run:3658 | VLM cached mRoPE positions: prefill_rope_start=4634 dense_kv_start=4766 input_tokens=17
11:03:01.544 INF Run:2738 | input token num : 17, prefill_split_num : 1
11:03:01.544 INF Run:2818 | prefill chunk p=0 history_len=4766 grpid=23 kv_cache_num=4864 input_tokens=17
11:03:01.980 INF Run:3045 | ttft: 436.19 ms
<think>11:03:01.980 INF Run:3076 | VLM decode positions: rope_start=4651 dense_kv_start=4783


</think>

根据视频中的庆祝画面，我们可以识别出进球的球员号码：

1.  **第一次进球**：
    - 庆祝时，球员背对着镜头，球衣背面清晰可见。
    - 号码显示为 **9**。
    - 球员名字是 **SUÁREZ**（苏亚雷斯）。

2.  **第二次进球**：
    - 庆祝时，球员背对着镜头，球衣背面也清晰可见。
    - 号码显示为 **10**。
    - 球员名字是 **10**（10号球员）。

所以，两次进球分别是 **9号球员** 和 **10号球员** 进的。

11:03:10.846 NTC Run:3445 | hit eos,decode avg 16.58 token/s
11:03:10.846 INF GetKVCache:2496 | precompute_len:4931, remaining:1469 (tracked)

启动服务（OpenAI 兼容）

shell

root@ax650:~# axllm serve AXERA-TECH/Qwen3.5-0.8B-AX650-C256-P6K-CTX8K
[I][                            Init][ 138]: LLM init start
tokenizer_type = 1
 96% | ███████████████████████████████   |  30 /  31 [4.63s<4.79s, 6.47 count/s] init post axmodel ok,remain_cmm(9563 MB)
[I][                            Init][ 199]: max_token_len : 2047
[I][                            Init][ 202]: kv_cache_size : 1024, kv_cache_num: 2047
[I][                            Init][ 205]: prefill_token_num : 128
[I][                            Init][ 209]: grp: 1, prefill_max_kv_cache_num : 1
[I][                            Init][ 209]: grp: 2, prefill_max_kv_cache_num : 128
[I][                            Init][ 209]: grp: 3, prefill_max_kv_cache_num : 256
[I][                            Init][ 209]: grp: 4, prefill_max_kv_cache_num : 384
[I][                            Init][ 209]: grp: 5, prefill_max_kv_cache_num : 512
[I][                            Init][ 209]: grp: 6, prefill_max_kv_cache_num : 640
[I][                            Init][ 209]: grp: 7, prefill_max_kv_cache_num : 768
[I][                            Init][ 209]: grp: 8, prefill_max_kv_cache_num : 896
[I][                            Init][ 209]: grp: 9, prefill_max_kv_cache_num : 1024
[I][                            Init][ 209]: grp: 10, prefill_max_kv_cache_num : 1152
[I][                            Init][ 214]: prefill_max_token_num : 1152
[I][                            Init][  27]: LLaMaEmbedSelector use mmap
100% | ████████████████████████████████ |  31 /  31 [4.64s<4.64s, 6.69 count/s] embed_selector init ok
[W][                            Init][ 457]: Qwen-VL vision size override: cfg=448x448 bytes=1204224, model_input_bytes=884736 -> 384x384 (square).
[I][                            Init][ 641]: Qwen-VL token ids: vision_start=151652 image_pad=151655 video_pad=151656
[I][                            Init][ 666]: VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
[I][                            Init][ 672]: VisionModule deepstack enabled: layers=3
[I][                     load_config][ 282]: load config:
{
    "enable_repetition_penalty": false,
    "enable_temperature": false,
    "enable_top_k_sampling": false,
    "enable_top_p_sampling": false,
    "penalty_window": 20,
    "repetition_penalty": 1.2,
    "temperature": 0.9,
    "top_k": 10,
    "top_p": 0.8
}

[I][                            Init][ 272]: LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/Qwen3.5-0.8B-AX650-C256-P6K-CTX8K'...
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/Qwen3.5-0.8B-AX650-C256-P6K-CTX8K

OpenAI 调用示例

python

from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/Qwen3.5-0.8B-AX650-C256-P6K-CTX8K"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
completion = client.chat.completions.create(
    model=MODEL,
    messages=messages,
)

print(completion.choices[0].message.content)

OpenAI 流式调用示例

python

from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/Qwen3.5-0.8B-AX650-C256-P6K-CTX8K"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
stream = client.chat.completions.create(
    model=MODEL,
    messages=messages,
    stream=True,
)

print("assistant:")
for ev in stream:
    delta = getattr(ev.choices[0], "delta", None)
    if delta and getattr(delta, "content", None):
        print(delta.content, end="", flush=True)
print("
")