language:
en
ko
base_model: google/gemma-4-26b-a4b
tags:
amd
rocm
gguf
llama.cpp
xinkang-labs
resonance-agi
high-performance
model_creator: XinXin-Kang
Gemma-4-26B-A4B-GGUF-Q4
"Quantized on consumer AMD GPU (RX 7900 XTX / gfx1100) — not NVIDIA, not MI300. Proof that AMD RX series works! 🔴"
[English]
This model is a personal quantization for research purposes, part of our ongoing exploration of resonance-based intelligence at Xinkang Labs . I am sharing this to support the AMD GPU community and to prove that high-performance LLM inference is fully achievable on consumer Radeon hardware.
🚀 Verified Performance (AMD Native)
GPU: AMD Radeon RX 7900 XTX (24GB / gfx1100)CPU: AMD Ryzen 9 9950X3DOS: Ubuntu 24.04.2 LTS / ROCm 7.2.3Stack: llama.cpp (GGML_HIP=ON) — Pure ROCm HIP, no PyTorch required. 🔴Prompt Processing (PP): ~630 t/sToken Generation (TG): ~100 t/s (Q4_K_M)
🖥️ Quick Start
CLI (Terminal):
llama-cli -m gemma-4-xinxin-kang-26B-A4B-q4.gguf -ngl 99 -c 8192 -cnvServer (Open WebUI / API):
HIP_VISIBLE_DEVICES=0 llama-server -m gemma-4-xinxin-kang-26B-A4B-q4.gguf -ngl 99 --host 0.0.0.0 --port 8000
-ngl 99 offloads all layers to GPU. Adjust -c for context length based on your VRAM.
🌐 About Xinkang Labs
Xinkang Labs is an independent AI research lab exploring the frontier of resonance-based intelligence and local LLM deployment.
🤖 Acknowledgments
This project was made possible with the assistance of:
Anthropic Claude (Sonnet 4.6) — Hardware debugging & build optimizationGoogle AI Mode & Gemini — Installation support & researchOpenAI ChatGPT (4o & 5) — System design & hardware analysis
Human + AI collaboration at its best. 🤝
—
XinXin-Kang / Xinkang Labs 🌐
xinkanglabs.com.au I hope this contributes even in a small way to your local AI experience. Any feedback is welcome!
🚀 Optimizations & Full Local Stack
This GGUF model was specifically quantized and verified on a consumer AMD GPU stack. If you want to achieve the absolute maximum performance (e.g., ~102 t/s Token Generation) and need the exact container deployment scripts, please check out the official repository:
Included in the Repository:
Complete Docker configurations (rocm/pytorch)
llama.cpp (GGML_HIP=ON) optimal compilation flagsNative vLLM server startup automation scripts
Full benchmark comparisons (Gemma 4, Qwen 2.5, etc.)
[한국어]
이 모델은 Xinkang Labs 에서 진행 중인 공명 기반 지능 연구의 일환으로 테스트되었습니다. 전문적인 배포용은 아니지만, 비슷한 사양의 AMD GPU 유저분들께 작은 도움이 되었으면 하는 마음으로 공유합니다. 순정 ROCm 환경에서 안정적인 동작과 강력한 성능을 확인하였습니다.
🚀 테스트 환경 및 실측 성능
GPU: AMD Radeon RX 7900 XTX (24GB)CPU: AMD Ryzen 9 9950X3DOS: Ubuntu 24.04.2 LTS / ROCm 7.2.3성능:
읽기(Prompt Ingest): 약 630 t/s
쓰기(Generation): 약 100 t/s
🖥️ 실행 방법
CLI (터미널):
llama-cli -m gemma-4-xinxin-kang-26B-A4B-q4.gguf -ngl 99 -c 8192 -cnv서버 모드 (Open WebUI 연동):
HIP_VISIBLE_DEVICES=0 llama-server -m gemma-4-xinxin-kang-26B-A4B-q4.gguf -ngl 99 --host 0.0.0.0 --port 8000
AMD GPU 사용 시 HIP_VISIBLE_DEVICES=0 를 꼭 붙여주세요.
🛠️ Model Details
Base Model: Google Gemma 4 26B A4BFormat: GGUF (Quantized via llama.cpp)Filename: gemma-4-xinxin-kang-26B-A4B-q4.ggufSize: Approx. 16GB (Optimized for 24GB VRAM cards)Quantization: Q4_K_M (Balanced quality vs. speed)Other variants: Q5_K_M, Q8_0 may be released in future updates.
🚀 실측 성능 (AMD Native)
인퍼런스 스택: llama.cpp (GGML_HIP=ON) 기반 빌드 — PyTorch 없이 순수 ROCm HIP 가속으로 구동됩니다. 🔴성능 지표: 읽기(PP) 약 630 t/s, 쓰기(TG) 약 100 t/s (Q4_K_M 설정 기준)
본 프로젝트는 다음 AI들의 도움으로 완성되었습니다:
Anthropic Claude (Sonnet 4.6) — 하드웨어 디버깅 및 빌드 최적화Google AI Mode & Gemini — 설치 지원 및 리서치OpenAI ChatGPT (4o & 5) — 시스템 설계 및 하드웨어 분석
—
XinXin-Kang / Xinkang Labs 🌐
xinkanglabs.com.au I hope this contributes even in a small way to your local AI experience. Any feedback is welcome!
🔧 AMD ROCm Build Guide (llama.cpp)
Prerequisites
AMD GPU (tested on RX 7900 XTX / gfx1100)
ROCm 7.2.3+
cmake, git, Docker (recommended)
1. Docker 환경 (권장 / Recommended)
1 docker run -it \
2 --device = /dev/kfd \
3 --device = /dev/dri/card0 \
4 --device = /dev/dri/card1 \
5 --device = /dev/dri/renderD128 \
6 --device = /dev/dri/renderD129 \
7 --group-add video \
8 -v /home/ $USER /dojoka:/workspace \
9 -p 8000 :8000 \
10 rocm/pytorch:latest bash
Replace 'dojoka' with your own working directory
2. llama.cpp ROCm 빌드
1 git clone https://github.com/ggerganov/llama.cpp
2 cd llama.cpp
3
4 cmake -B build \
5 -DGGML_HIP = ON \
6 -DAMDGPU_TARGETS = "gfx1100" \
7 -DCMAKE_BUILD_TYPE = Release \
8 -DCMAKE_C_COMPILER = /opt/rocm/bin/hipcc \
9 -DCMAKE_CXX_COMPILER = /opt/rocm/bin/hipcc \
10 -DCMAKE_PREFIX_PATH = /opt/rocm-7.2.3
11
12 cmake --build build --config Release -j $( nproc )
3. 모델 실행 (서버 모드 / Server Mode)
1 HIP_VISIBLE_DEVICES = 0 ./build/bin/llama-server \
2 -m /workspace/gemma-4-xinxin-kang-26B-A4B-q4.gguf \
3 -ngl 99 \
4 --host 0.0 .0.0 \
5 --port 8000
4. 모델 실행 (CLI 모드)
1 HIP_VISIBLE_DEVICES = 0 ./build/bin/llama-cli \
2 -m /workspace/gemma-4-xinxin-kang-26B-A4B-q4.gguf \
3 -ngl 99 -cnv
⚠️ 주의사항 / Important Notes
HIP_VISIBLE_DEVICES=0 — GPU 0만 사용 (CPU iGPU 충돌 방지)-ngl 99 — 모든 레이어를 GPU에 올림 (필수!)-DCMAKE_PREFIX_PATH=/opt/rocm-7.2.3 — ROCm 버전 경로 확인 필요gfx1100 = RX 7900 XTX/XT
GPU 모델 gfx 코드 RX 7900 XTX/XT gfx1100 RX 7800 XT / 7700 XT gfx1101 RX 6900/6800 XT gfx1030 RX 6700 XT gfx1031
