Checkpoint size: 94.30 GB (87.82 GiB) of safetensors weights.
This project was built to make the 118B-parameter Laguna S 2.1 MoE model usable on 4× Tesla V100 32GB with tensor parallel size 4 and a checkpoint below 100 GB. The size ceiling, W4A16 g32 asymmetric quantization, FP16 weight storage, and the protected routed-expert allocation—Layers 1–4 and 46–47—were all selected for that deployment target.
The main checkpoint is the general form of the same work: one unscaled weight artifact configured for BF16 activations, intended to run directly on a BF16-capable SM80+ system with stock vLLM. The V100 runtime add-on reuses those exact weights; there is no second V100 checkpoint.
This is an independently produced selective AWQ quantization of poolside/Laguna-S-2.1. It is not an official Poolside release.
Critical dtype requirement for the main checkpoint
Run the unscaled main checkpoint with BF16 activations. Do not force FP16 (float16/half).
The original model can produce very large late-layer hidden states at BOS/EOS anchor positions. In our fixed calibration replay, the largest observed BF16 hidden value was approximately 950,272, above the FP16 finite limit of 65,504. Each of the 1,097 token-id-2 BOS/EOS anchor rows contained at least one component outside the FP16 range, while the largest observed non-anchor value was 17,152. Normalized outputs and logits remained finite in BF16.
This is an activation dynamic-range property, not an AWQ packing failure. A runtime that silently executes the main checkpoint with FP16 activations can produce non-finite logits at EOS positions.
V100 runtime add-on — primary deployment target
The included v100-1cat/ directory is a small, fail-closed runtime patch for the configuration that motivated this quantization:
4× Tesla V100 PCIe 32GB;
tensor parallel size 4;
1Cat-vLLM 1.2.2;
FP16 activations and FP16 KV cache;
TRITON_ATTN, chunked prefill, and prefix caching disabled.
It is deliberately version-pinned, not a cross-version plugin. It uses the exact main-model revision without copying or permanently rewriting the approximately 94 GB checkpoint.
Rc4 fixes two independent FP16 runtime problems. The pinned 1Cat/vLLM MoE runner's overflow-protection branch represents Laguna's combined routed/shared output at 1 / routed_scaling_factor and requires a matching decoder compensation. The pinned Laguna decoder omitted that multiply, so each MoE layer returned approximately 0.4x of the intended branch output for the configured factor 2.5. Rc4 restores the required FP16 x2.5 after every MoE layer. BF16 inference does not use this branch and is unaffected.
The second problem is the late-layer BOS/EOS activation overflow. For that range repair, the add-on applies a local power-of-two reparameterization to Layers 45–47, while the forward path multiplies hidden/residual by 1/64 once before Layer 45:
attention o_proj: × 1/64;
routed/shared expert up_proj: × 1/256;
routed/shared expert down_proj: × 4.
The net branch factor is 1/64. Layer 45 remains W4; its qweight and qzeros are unchanged, while the relevant floating AWQ scale carriers are transformed. The add-on rejects an unknown model revision, quantization layout, or tensor count.
FLASH_ATTN_V100 is deliberately rejected by the add-on. On this model, the tested Flash-V100 path became non-finite after chunked prefill crossed its observed boundary. Triton is somewhat slower on short prompts, but it is the accepted V100 path from 512 tokens through long context.
The same-weight loader-time implementation has reproduced the materialized reference:
Test
Loader-time add-on result
Loader transform versus materialized reference
1,545 transformed and 2,337 untouched tail tensors bitwise identical
Complete EOS-containing calibration blocks
256/256 finite
Generation immediately after internal EOS
11/11 finite
Explicit multi-EOS histories
4/4 finite
Chinese/English chat, Python code, raw Chinese completion
Passed; no prior corruption/repetition or replacement characters
Mean coarsened KLD versus corrected unscaled FP16 arm
6.0291049932e-05 nats over 8,212 positions
Token top-1 agreement versus corrected unscaled FP16 arm
99.9391%
Actual-token logprob MAE versus corrected unscaled FP16 arm
The non-zero KLD and 99.9391% top-1 figures compare rc4's Layer-45 /64 repair with an unscaled FP16 runtime that includes the same required MoE x2.5 correction. The diagnostic PPL changed from 21.9363 to 21.9160, but 1Cat's Laguna prompt-logprob token attachment is shifted by one position, so that value is not presented as a language-model perplexity result.
The 30-case smoke used thinking=off with a 512-token cap. It is a small gross-regression check, not a public benchmark or an exact numerical gate.
A standalone global /32 weight-folding experiment was also tested and rejected: against the same unscaled baseline it produced 0.0661285 nats mean coarsened KLD and 91.0010% token top-1 agreement because FP16 rounding accumulated through all 48 layers. The V100 release therefore uses the local runtime add-on.
Model summary
Laguna S 2.1 is a text-to-text, 118B-total-parameter, approximately 8B-active-per-token Mixture-of-Experts model for agentic coding and long-horizon work. It is not a vision or audio model. The base model has 48 decoder layers, 256 routed experts plus one shared expert, and a configured context window of 1,048,576 tokens. See the official base model card for architecture, chat formatting, intended use, responsible-use guidance, and base-model benchmark results.
W4A16 denotes packed 4-bit weights with 16-bit activations. The main path uses BF16 activations; the V100 add-on uses FP16 activations with the local range repair above.
A single FP16-stored weight artifact serves both the BF16 main path and the V100 add-on. The precision effect of storing protected tensors in FP16 is already included in the KLD, perplexity, and top-1 measurements reported below.
Some protected FP16 tensors participate in mathematically equivalent AWQ scale compensation. They remain FP16 and were not packed to W4, but they need not be byte-identical to the corresponding base-model tensors.
quantization_config.modules_to_not_convert has been checked against this exact FP16/W4 layout and the machine-readable tensor audit. Stock vLLM 0.24.0 also loaded the mixed layout successfully; this is no longer inferred only from the JSON declaration.
Why this allocation?
The protected prefix and tail were selected from actual packed-model comparisons under the sub-100-GB V100 budget.
FP16 routed-expert layers
Directory bytes
Final hidden rel-L2
Mean full-vocab KLD
PPL
Top-1 agreement
30-task smoke
1–3 and 45–47
94,311,697,448
0.006003
0.241694
24.3297
86.56%
28/30
1–4 and 45–47
97,746,514,057
0.005274
0.233194
23.7390
86.94%
28/30
1–4 and 46–47
94,311,700,368
0.005288
0.234471
23.7372
86.67%
29/30
1–5 and 46–47
97,746,516,977
0.005844
0.245507
23.6633
86.77%
28/30
1–6 and 47
97,746,519,891
0.010638
0.249403
24.0867
86.48%
29/30
1–7
97,746,522,803
0.060652
0.246453
24.4271
86.35%
29/30
The selected layout protects four early and two final routed-expert layers while remaining below the 100,000,000,000-byte checkpoint limit. The 30-task column is a small deterministic regression smoke test, not a general benchmark.
Main-checkpoint usage
The supported main path uses upstream vLLM's native Laguna implementation and does not depend on copied model remote code. Transformers inference is not currently a validated path.
The command below is the intended BF16 serving form. The Poolside parser flags remain separately qualified below because the stock-vLLM probe exercised the native model and dtype path, not the HTTP parser stack:
The parser and chat-template flags are copied from the base model's recommended vLLM setup. Reasoning and forced tool calls passed on the validated 1Cat V100 runtime; the same parser flags have not yet been re-run through the upstream-vLLM HTTP server.
The accepted stock-vLLM probe positively verified the loaded dtype rather than relying on the absence of a cast warning. On both TP workers, the 917 exposed parameter tensors consisted of 624 BF16, 47 FP32 router-bias, and 246 INT32 packed tensors, with no FP16 parameter. Hooks observed all 48 decoder layers with BF16 finite hidden/residual tensors and a finite BF16 final norm.
Offline packed-model comparison against the frozen BF16 base
Validated
Materialized 4× V100 tail-repair reference
Validated as development evidence
Final same-weight V100 add-on rc4, 1Cat-vLLM 1.2.2
Validated with corrected FP16 MoE semantics through EOS, 259,989 tokens, reasoning, tool calls, and fixed-corpus drift
Exact main checkpoint, upstream vLLM 0.24.0, BF16
Native load and positive 48-layer dtype/EOS forward validated on heterogeneous RTX 3090 + RTX 5060 Ti TP2 with CPU offload; not a performance test
Transformers/AutoAWQ inference
Not evaluated
SGLang
Not evaluated
TensorRT-LLM
Not evaluated
Standalone RTX 3090
Not evaluated; the validated compatibility probe used RTX 3090 + RTX 5060 Ti together
DFlash or MTP
Not evaluated / not included
The library_name: transformers metadata describes the Hugging Face configuration/tokenizer format and base architecture; it is not a claim that Transformers inference has been validated.
Calibration data was used for post-training quantization only; the base model was not retrained.
Held-out numerical comparison
The main numerical comparison used 16 disjoint 512-token held-out blocks and 8,176 scored next-token positions against the fixed raw-BF16 base revision. Decoder replay used BF16, final normalization accumulated in FP32, and the protected FP16 lm_head used the FP16 logit-compute path.
These measurements describe quantization drift on a fixed corpus. They do not claim to reproduce the base model's published software-engineering benchmark scores.
Checkpoint shard bytes counts only the 49 safetensors files. The shards and
model index remain frozen. The release process regenerates the repository-root
SHA256SUMS for the corrected rc4 add-on. The public file set and its
integrity metadata were verified against the frozen rc4 candidate before the
v1.0 tag was created. Additional clean-download runtime checks are tracked
separately and are not a prerequisite for reading or verifying this checkpoint.
Context length
The checkpoint inherits the base model's configured maximum context of 1,048,576 tokens, but that length has not been validated for this exact quantized release. Rc4 passed 4,095-, 15,352-, and 259,989-token needle tests and exposes 395,822 KV tokens at the validated 262,144-token server profile. The 259,989-token run recovered the exact needle with 1,384.602 s TTFT (187.77 prompt tok/s); peak sampled memory was 31,968 MiB used and 527 MiB free on each V100. One-million-token context remains not evaluated.
Intended use and limitations
This derivative is intended for local or hosted text generation and agentic coding workflows where the selective W4/FP16 trade-off is useful, particularly 4× V100 32GB TP4 deployments.
The main checkpoint requires BF16 activations; forcing FP16 can overflow at EOS positions.
The V100 path requires the version-pinned add-on and must not be assumed compatible with other runtimes.
Only routed experts in Layers 5–45 are W4, so this is larger than a uniform all-expert W4 checkpoint by design.
The stock-vLLM BF16 compatibility test used heterogeneous TP2 plus 36 GiB CPU offload per GPU at a 512-token profile. It validates native loading and dtype/range behavior, not production speed or long-context capacity on that machine.
Transformers/AutoAWQ, SGLang, TensorRT-LLM, standalone RTX 3090, DFlash, MTP, and 1M context are not evaluated.
Quantization can change generations even when aggregate numerical metrics remain close.
Follow the original model card's chat-template, thinking-preservation, intended-use, and responsible-use guidance.
License and attribution
The original model is poolside/Laguna-S-2.1, created by Poolside and released under OpenMDW-1.1. This repository retains the complete original LICENSE.md, copyright notices, and source notices required by that license.
Integrity file: repository-root SHA256SUMS; the public file set and integrity metadata were checked against the frozen rc4 candidate before v1.0 was created. Additional clean-download runtime checks remain tracked separately.
