Qwen3-0.6B with Q8_0 quantization, running on
Metal (Apple Silicon) and
CUDA (NVIDIA GPUs) via the
executorch-ggml backend.
With QKV + gate/up projection fusion: MUL_MAT reduced from 197 to 113 per decode step.
1 split, 113 MUL_MAT (fused QKV + gate/up), graph cache HIT.
1 pip install huggingface_hub
2 python -c "
3 from huggingface_hub import snapshot_download
4 snapshot_download('larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML',
5 local_dir='models/qwen3')
6 "
1 git clone https://github.com/larryliu0820/executorch-ggml
2 cd executorch-ggml
3 git submodule update --init --recursive
1 cmake -B build_native \
2 -DEXECUTORCH_GGML_BUILD_LLAMA_RUNNER = ON \
3 -DCMAKE_BUILD_TYPE = Release
4 cmake --build build_native --target benchmark_llm --parallel 16
1 cmake -B build_native \
2 -DEXECUTORCH_GGML_BUILD_LLAMA_RUNNER = ON \
3 -DGGML_CUDA = ON \
4 -DCMAKE_CUDA_ARCHITECTURES = 80 \
5 -DCMAKE_BUILD_TYPE = Release
6 cmake --build build_native --target benchmark_llm --parallel 16
1 # Benchmark: .pte (graph) + .gguf (weights)
2 ./build_native/benchmark/benchmark_llm \
3 models/qwen3/qwen3_q8_0.pte \
4 --gguf models/qwen3/Qwen3-0.6B-Q8_0.gguf \
5 --n-decode 128 --prompt-len 5
1 import torch
2 from executorch_ggml . gguf_module import GGUFModule
3
4 # Load graph from .pte, weights from .gguf
5 module = GGUFModule ( "models/qwen3/qwen3_q8_0.pte" ,
6 "models/qwen3/Qwen3-0.6B-Q8_0.gguf" )
7
8 # Print model info
9 module . print_info ( )
10
11 # Prefill
12 input_ids = torch . tensor ( [ [ 1 , 2 , 3 , 4 , 5 ] ] , dtype = torch . long )
13 cache_pos = torch . arange ( 5 , dtype = torch . long )
14 logits = module . forward ( input_ids , cache_pos )
15 next_token = logits [ 0 ] [ : , - 1 , : ] . argmax ( dim = - 1 ) . item ( )
16
17 # Decode loop
18 import time
19 tokens = [ next_token ]
20 t0 = time . time ( )
21 for i in range ( 127 ) :
22 tok_input = torch . tensor ( [ [ next_token ] ] , dtype = torch . long )
23 pos_input = torch . tensor ( [ 5 + i ] , dtype = torch . long )
24 logits = module . forward ( tok_input , pos_input )
25 next_token = logits [ 0 ] [ 0 , 0 , : ] . argmax ( dim = - 1 ) . item ( )
26 tokens . append ( next_token )
27 dt = time . time ( ) - t0
28 print ( f" { len ( tokens ) / dt : .1f } tok/s" )
1 # Per-call timing breakdown
2 GGML_PERF_LOG = 1 ./build_native/benchmark/benchmark_llm \
3 models/qwen3/qwen3_q8_0.pte \
4 --gguf models/qwen3/Qwen3-0.6B-Q8_0.gguf \
5 --n-decode 32
6
7 # Per-op timing (adds sync overhead — use for relative comparison only)
8 GGML_PROFILE = 1 ./build_native/benchmark/benchmark_llm \
9 models/qwen3/qwen3_q8_0.pte \
10 --gguf models/qwen3/Qwen3-0.6B-Q8_0.gguf \
11 --n-decode 5
1 cd third-party/llama.cpp
2 cmake -B build -DGGML_METAL = ON -DCMAKE_BUILD_TYPE = Release # or -DGGML_CUDA=ON
3 cmake --build build --target llama-bench --parallel 16
4 cd .. / ..
5
6 third-party/llama.cpp/build/bin/llama-bench \
7 -m models/qwen3/Qwen3-0.6B-Q8_0.gguf \
8 -ngl 99 -p 5 -n 128 -r 5
Export (one-time):
GGUF file ──> GGUFAnalyzer ──> model config + weight names
──> PyTorch model (no weights loaded)
──> torch.export + GGML partitioner
──> .pte with GGUF tensor names as data_keys
(213 KB, graph only)
Runtime:
.pte (graph) ──> ExecuTorch Program
.gguf (weights) ──> GGUFNamedDataMap (implements NamedDataMap)
──> Backend loads weights via get_data(key)
──> Same performance as embedded weights1 from executorch_ggml . export_gguf import export_gguf_to_pte , GGUFExportConfig
2
3 config = GGUFExportConfig (
4 max_seq_len = 128 ,
5 preserve_dynamic_shapes = True ,
6 enable_quantization = True ,
7 )
8 export_gguf_to_pte ( "Qwen3-0.6B-Q8_0.gguf" , "qwen3_q8_0.pte" , config )
