Views
No views yet
deepseek-ai/DeepSeek-V4-Flash-0731
(43 layers, hidden size 4096, four-stream hyper-connection residual, fp8 MoE).
The Jacobian lens (Anthropic, Verbalizable Representations Form a Global
Workspace in Language Models) reads out what an internal residual is disposed
to make the model say:lens_l(h) = unembed( J_l @ h ), J_l = E[ ∂h_target / ∂h_l ]J_l is a
closed-form running mean of per-prompt Jacobians.jacobian-lens-deepseek-v4-flash-preview
— fitted with an identical recipe, so the pair supports recipe-controlled
comparisons of what the re-post-training changed.hc_mult = 4 parallel residual streams per position
(hyper-connections), so a square J_l ∈ R^{d×d} is not well-defined. These
lenses are the rectangular generalization:[4096, 16384] per layer, fp16 on disk.d_source = 16384): a block's output with the four streams
flattened, hc_mult * hidden_size.d_model = 4096): the output of model.hc_head, the module that
collapses the four streams back to one vector per position just before the
final norm and unembedding. The collapse operator is absorbed into J_l,
which keeps the target in the exact basis the unembedding expects. The square
single-stream case is a bit-identical special case of this construction.| file | prompts | layers | shape | note |
|---|---|---|---|---|
lens.pt | n=1000 | L19–39 (21 layers) | [4096, 16384] fp16 | default; paper-scale corpus |
lens_n100.pt | n=100 | L19–39 (21 layers) | [4096, 16384] fp16 | the lens most replication numbers were produced with |
lens_disjoint100.pt | n=100, corpus records 900–1000 | L19–39 (21 layers) | [4096, 16384] fp16 | noise floor: same checkpoint, disjoint corpus — the control every floor-referenced comparison uses |
lens_n100.pt was produced by
restricting a full 43-layer fit to the band; because per-prompt normalization
is applied per layer, this restriction is bit-identical to a band-restricted
fit (verified in float64). The n=1000 fit moved no experimental conclusion
relative to n=100 (every delta within the corpus-sampling noise floor measured
on a disjoint-prompt lens), consistent with the estimator being converged from
roughly n=60.Salesforce/wikitext, wikitext-103-raw-v1, train
split, streamed), records ≥ 600 characters, truncated to 128 tokens.
The n=1000 prompt set is a superset of the n=100 set (first 100 records);
lens_disjoint100.pt uses records 900–1000, disjoint from the n=100 set,
which is what makes it a valid sampling-noise floor.skip_first=16,
attention-sink positions) and the final position is dropped; the per-prompt
Jacobian is the mean over the remaining ~111 source positions, with
cotangents set at every valid target position at or after the source.normalize_per_prompt=True): each prompt's
J_l is scaled to unit Frobenius norm before averaging. Per-prompt Jacobian
gain is heavy-tailed (up to ~30× between prompts) and the model's final
RMSNorm makes a uniform rescaling of J_l invisible in the logits, so a
plain mean weights prompts by an unobservable quantity. This improved pass@1
by 73% and is what all released numbers use. A consequence: ||J_l||_F ≤ 1
by construction (measured 0.51–0.89 across the band) — the absolute scale of
J_l is not meaningful, only its direction.J) version of the jlens package —
the upstream square-only release cannot represent d_source != d_model.1from transformers import AutoModelForCausalLM, AutoTokenizer
2from jlens.hf import from_hf
3from jlens.lens import JacobianLens
4
5lens = JacobianLens.from_pretrained("xiangchensong/jacobian-lens-deepseek-v4-flash-0731")
6# or: JacobianLens.from_pretrained(..., filename="lens_n100.pt")
7
8name = "deepseek-ai/DeepSeek-V4-Flash-0731"
9tok = AutoTokenizer.from_pretrained(name)
10hf_model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")
11model = from_hf(hf_model, tok) # auto-detects the hyper-connection layout
12
13lens_logits, model_logits, input_ids = lens.apply(
14 model, "The capital of France is", layers=[30], positions=[-1]
15)
16print(lens_logits[30].topk(5).indices) # what L30 is disposed to sayJ_l is
an expectation of gradients of deepseek-ai/DeepSeek-V4-Flash-0731's forward
pass, i.e. a function of its weights. The base model is released under the MIT
License (Copyright (c) DeepSeek), and these matrices are released under the
same terms. No base-model weights are contained in or recoverable from these
matrices at fidelity; they are averaged input–output sensitivities.