Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
grok-1-W4A8KV8 – AI Model by amd | AlphaNeural AI
You can deploy this model and start earning money today!
amd
/
grok-1-W4A8KV8
like
0
grok-1
custom_code
lmzheng/grok-1
quantized
apache-2.0
fp8
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
Grok-1-W4A8KV8
Introduction
This model was created by applying
Quark
with calibration samples from Pile dataset.
Quantization Stragegy
Quantized Layers
: All linear layers excluding "lm_head", "*.gate"
Weight
: FP8 symmetric per-tensor, additionally, INT4 symmetric per-channel for MoE linear
Activation
: FP8 symmetric per-tensor
KV Cache
: FP8 symmetric per-tensor
INT4 Packing
Every eight
int4
values are packed into a single
int32
integeter following the sequence defined by
order_map = [0, 2, 4, 6, 1, 3, 5, 7]
.
Quick Start
Follow
Quantizing Sharded Grok-1 with Quark for SGLang
to produced the quantized model using Quark.
Deployment
Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the SGLang backend.
Evaluation
Evaluation scores
Benchmark
grok-1
grok-1-W4A8KV8(this model)
gsm8k
0.821
0.817
License
Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights reserved.
