The hybrid quant employs different quantization levels on a per layer basis to increase
flexibility of trading off performance vs file size. Less parameter bits are used at deep layers
and more bits at cortex layers to simultaneously optimize quantized size and model performance.
For this file the layer quants are as follows:
The layer quants were optimized for solid performance across a set of curated test prompts on the
RL variant of the model (Ring-mini-2.0) and reused with no further adjusments on this non RL version.
Comparison:
Quant
size
PPL
Comment
Q6_K
13.4e9
19.0
default embed and output
Q6_K_H
13.2e9
18.9
Q6_K embed Q6_K output
Usage:
This is a compact moe model which shows quite strong performance across a range of curated test prompts.
The unique feature of this model is its overall size is only 16G so with moe it will run very efficiently
with experts offloaded to CPU. Example offload configs:
12G VRAM (4070), 32k context. Offload layers 16-19 to CPU and rest on GPU, VRAM will be about full and
token gen will be about 120tps.
OT="-ot blk\.[1][6-9].*exps=CPU -ngl 99"
12G VRAM (4070), 128k context. Offload layers 10-19 or 9-19 to CPU and rest on GPU, token gen will be
around 80tps.
# Offload 10-19 to CPU This will rail out VRAM and leave no room for a browser to run
OT="-ot blk\.[1][0-9].*exps=CPU -ngl 99"
# Offload 9-19 to CPU this will leave some VRAM for a browser to run
OT="-ot blk\.9|1[0-9].*exps=CPU -ngl 99"
# Config YARN for 128k context
--rope-scaling yarn --yarn-orig-ctx 32768 --rope_scale 4.00000
Grouped experts:
The model can be run with or without grouped expert logic. It defaults with grouped experts logic on. To
turn if off launch with:
"Time is the canvas upon which the masterpiece of our existence is painted, and every moment, every choice, every experience is a stroke of color that defines the picture."
This statement reflects the profound interconnectedness of time, choice, and experience, suggesting that our lives are shaped by the cumulative impact of the moments we live and the decisions we make. It invites reflection on the beauty and complexity of the journey we call life.
