I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.
While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.
Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.
This is a Derivative AI Model of upstage/Solar-Open2-250B,
distributed under the Upstage Solar License. A copy of the license is
included in this repository as LICENSE.
이 모델은 upstage/Solar-Open2-250B 의
파생 AI 모델이며 Upstage Solar License 로 배포됩니다. 라이선스 사본은
이 저장소의 LICENSE 에 포함돼 있습니다.
Weight-only NVFP4A16 (W4A16) compressed-tensors quantization of
Baekpica/Solar-Open2-120B-A15B-REAM-148E-BF16-v2. Only the routed expert projections are
quantized; every path that all tokens traverse stays in BF16.
Because the router is not quantized, quantization cannot change routing
decisions. Because attention, the shared expert, the embedding and the LM head
are not quantized, no token passes through FP4 on its dense path — only the
8 of 148 routed experts selected per token.
gate/up global-scale sharing
The runtime loads gate_proj and up_proj fused into a single
gate_up_proj [148, 2560, 4096] tensor, so the pair must share one NVFP4 global
scale. llm-compressor's default fused pattern only matches .mlp.gate_proj and
misses Solar's .mlp.experts.N.gate_proj. A Solar-specific fused mapping was
registered so that all 7,104 pairs share
a global scale.
Tensor composition
dtype
Keys
Purpose
U8
21,312
packed FP4 E2M1 weights (2 values per byte)
F8_E4M3
21,312
per-group scales (group size 16)
F32
21,360
per-tensor global scales + router correction bias
BF16
891
unquantized paths kept at full precision
Quality
Measured with the same fixed prompts, same engine and same sampling as the
BF16 parent: vLLM v0.22.0-solar-open2, tensor parallel 4, temperature=0,
top_p=1.0, repetition_penalty=1.0, seed=42.
Metric
v1 (before healing)
v2 BF16
v2 NVFP4A16
distinct-1 ↑
0.3225
0.4633
0.4357
distinct-3 ↑
0.5954
0.827
0.8183
top 3-gram ratio ↓
0.2612
0.018
0.0164
distinct word ratio ↑
0.4441
0.621
0.5956
Repetition collapse / 16
5
0
0
Closed <|think:end|> / 16
0
8
9
Stopped on its own / 16
0
4
4
NVFP4A16 is effectively on par with the BF16 parent — repetition collapse stays
at 0 of 16 and the top 3-gram ratio is marginally lower. The full transcript is
in healing_comparison.md.
Load through a Solar Open 2-compatible runtime with the compressed-tensors
NVFP4 path. This release was verified on the Upstage vLLM Solar branch
v0.22.0-solar-open2 at tensor parallel 4. Tensor naming follows the same
unpacked per-expert convention as v1.
Limitations
W4A16 keeps activations at 16-bit. The goal is memory and bandwidth
reduction, not FP4 compute throughput.
FP4 is lossy. Even though quantization error is confined to the routed
experts, outputs are not bit-identical to the BF16 parent.
Upstage Solar License, identical to the BF16 parent. Redistribution and use
must satisfy that license, including its Solar naming and attribution
requirements.
라우터를 양자화하지 않았으므로 양자화가 라우팅 결정 자체를 바꾸지 못합니다.
attention·shared expert·embedding·LM head도 양자화하지 않았으므로 dense 경로에
FP4가 개입하지 않고, 토큰당 선택되는 148개 중 8개 routed expert에만 적용됩니다.
gate/up global scale 공유
런타임은 gate_proj 와 up_proj 를 gate_up_proj [148, 2560, 4096] 하나로
합쳐 로드하므로 두 텐서가 동일한 NVFP4 global scale을 가져야 합니다.
llm-compressor의 기본 fused 패턴은 .mlp.gate_proj 만 인식해 Solar의
.mlp.experts.N.gate_proj 를 놓치므로, Solar 전용 fused 매핑을 등록해
7,104개 페어가 모두 global scale을
공유하도록 했습니다.
텐서 구성
dtype
키 수
용도
U8
21,312
packed FP4 E2M1 가중치 (바이트당 2값)
F8_E4M3
21,312
그룹 스케일 (group size 16)
F32
21,360
텐서별 global scale + router correction bias
BF16
891
양자화하지 않은 경로
품질
BF16 부모와 동일한 고정 프롬프트·동일 엔진·동일 샘플링으로 측정했습니다.
vLLM v0.22.0-solar-open2, tensor parallel 4, temperature=0, top_p=1.0,
repetition_penalty=1.0, seed=42.
지표
v1 (힐링 전)
v2 BF16
v2 NVFP4A16
distinct-1 ↑
0.3225
0.4633
0.4357
distinct-3 ↑
0.5954
0.827
0.8183
top 3-gram ratio ↓
0.2612
0.018
0.0164
distinct word ratio ↑
0.4441
0.621
0.5956
Repetition collapse / 16
5
0
0
Closed <|think:end|> / 16
0
8
9
Stopped on its own / 16
0
4
4
NVFP4A16은 BF16 부모와 사실상 동등합니다 — 반복 붕괴가 16개 중 0개로 동일하고
최빈 3-gram 비율은 오히려 근소하게 낮습니다. 생성 결과 전문은
healing_comparison.md 에 있습니다.
Solar Open 2 호환 런타임에서 compressed-tensors NVFP4 경로로 로드합니다.
이 릴리스는 Upstage vLLM Solar 브랜치 v0.22.0-solar-open2, tensor parallel 4
에서 검증했습니다. 텐서 이름 규약은 v1과 같은 unpacked per-expert 형식입니다.
한계
W4A16이므로 활성값은 16비트입니다. 목적은 메모리·대역폭 절감이지 FP4 연산
처리량이 아닙니다.
FP4는 손실 압축입니다. 양자화 오차가 routed expert에 국한되더라도 출력이 BF16
부모와 비트 단위로 같지는 않습니다.
