privacy-filter-nemotron-v2 — ONNX quantizations for the browser

Files

Usage in a web app (transformers.js v4+)

• Load in a Web Worker, once (singleton). A ~1 GB session load freezes the UI thread otherwise. Wire progress_callback to a progress bar — the first load downloads the full variant; transformers.js caches it in the browser Cache API, so subsequent loads are local.
• Self-hosting the files instead of the Hub (optional — keeps your app independent of the Hub CDN): download this repo to your static server, set env.allowLocalModels = true (browser builds default it to false), env.allowRemoteModels = false, env.localModelPath = "/models/", and pass the directory name as the pipeline id. Your server must support HTTP Range requests — onnxruntime-web fetches large files in ranges.
• External data: model.onnx (fp32) and the big three (model_quantized, model_mixed48, model_fp16) each have a <name>.onnx_data sidecar; only model_q4.onnx is single-file. config.json's transformers.js_config.use_external_data_format maps each file to its shard count — keep it truthful if you regenerate the config, or transformers.js will fetch a nonexistent sidecar (or skip a needed one) and hang at 100%.
• WebGPU bundle for q8/mixed48: import ort from onnxruntime-web/jspi (its native WebGPU QMoE kernel) for those two on GPU — the default asyncify bundle loads them but runs the experts on CPU (~100× slower). q4 and all CPU paths use the default bundle. JSPI needs Chrome/Edge ≥ 137.
• Verify the device. A silent WebGPU→wasm fallback is easy to miss: after load, inspect the model's session execution providers and fail loudly if you required WebGPU.
• Character offsets: the transformers.js token-classification pipeline doesn't emit char offsets. For exact redaction spans, run tokenizer + model directly and reconstruct offsets by incremental prefix-decode (exact for byte-level BPE like o200k) — see src/worker.ts in the conversion project for a reference implementation.

How these were made (the interesting part)

• swiglu is interleaved (g0,u0,g1,u1,…) in the ORT MoE kernel (swiglu_fusion=1); the HF checkpoint stores concatenated halves → gate_up weights and biases are row-permuted during transplant.
• q/k scaling is folded into the weights: the modeling code multiplies both q and k by head_dim**-0.25 after projection; the ONNX graph expects that factor pre-folded into q/k weights and biases (verified via RMS ratios vs upstream — without it, span parity collapses to 54%).
• Router: softmax over top-k raw logits (the HF code's softmax/top_k × num_experts_per_tok cancels to exactly the ORT kernel's normalize_routing_weights=1), so raw router logits feed QMoE directly.
• QMoE quant convention (byte-exact vs upstream files; kernel is exact dequant + fp32 GEMM, matches a numpy reference to ~1e-7): half = 2^(bits-1); s = absmax/half; q = clamp(round(w/s)+half, 0, 2^bits-1), uint8, 4-bit packs low-nibble-first, no zero points.
• Rope cos/sin caches are byte-identical to upstream (YaRN, factor 32, mscale ≈ 1.3466, θ = 150000).

Accuracy

Benchmark: "filtered or not" (bench10)

In-browser measurements

• q4 stays the recommended browser default: its 4-bit GatherBlockQuantized embeddings only have a WebGPU kernel (no wasm CPU kernel), so q4 is GPU-only — but it is the smallest download (0.92 GB) with the broadest device support and the fastest inference.
• q8 / mixed48 keep fp32 embeddings, so they run on either WebGPU (JSPI) or CPU (wasm). On WebGPU they need the JSPI bundle + a JSPI-capable browser; on CPU they need no special bundle.
• fp16 requires the WebGPU shader-f16 feature; on GPUs without it (Turing and earlier) the session fails to create. There is no CPU benefit to fp16, so it is a WebGPU-only, modern-GPU-only variant.
• Warm (Cache API) reloads are ~7–10 s for any variant; the cold cost is the one-time hub download. Memory: ~1 GB GPU for q4; the big variants sit larger.

Positioning