Views
No views yet
Qwen/Qwen3.6-27B (d_model=5120), trained with dictionary_learning
so it loads directly.| Base model | Qwen/Qwen3.6-27B |
| Hook | model.model.layers[42] output (resid_post), [B,T,5120] |
| Dictionary size (F) | 131072 (25.6x expansion) |
| Active latents (k) | 64 (BatchTopK, learned threshold 1.6539) |
| Training tokens | 250,085,376 |
| Corpus | openbmb/Ultra-FineWeb (en) |
| Explained variance | 0.6925 |
| L0 | 64.0 |
| Dead features | 0.02% (22/131072) |
dictionary_learning BatchTopKTrainer), same aux-k dead-feature loss
(auxk_alpha=1/32), same warmup + learned-threshold schedule. Differences: this SAE is on the 27B
model (d=5120, layer 42), uses F=131072 / k=64, and was trained on ~250M tokens — about half the
~500M tokens Adam trains (his 8B batch_top_k is F=65536, k=80, ~500M tokens) on a 2x larger dictionary,
so treat this as a research checkpoint (a longer run would likely lower the dead-feature rate). Method is
otherwise identical.saes_Qwen_Qwen3.6-27B_batch_top_k/resid_post_layer_42/trainer_0/ae.pt — SAE state_dict.../trainer_0/config.json — trainer/buffer configmaxacts/acts_Qwen_Qwen3.6-27B_layer_42_trainer_0_context_length_32.pt — per-feature top-32 activating windows {"max_tokens":[F,N,32], "max_acts":[F,N,32]}data/examples.parquet — decoded top-activating text windows for every live feature (browse in the HF Data Studio)data/examples.parquet.| feature | peak act | top-activating window |
|---|---|---|
| 5307 | 3.14 | milk proteins; or differences between the structure of the chromosomes or DNA fragments. Any of these studies might suggest that the subpopulation represents a⟦ unique⟧ a |
| 12431 | 2.18 | information to us, you agree to its transfer to and processing in the United⟦ States⟧.↵10. Accessing and Correcting Your Personal Information↵If you |
| 18256 | 11.59 | . Another standard is to ask if an individual’s contribution is “original”.↵The ICMJE also recommends that all authors be accountable⟦ for⟧ all aspects of the |
| 18318 | 8.61 | optimal route selection methods in terms of representing human-like behavior.↵Showing items related by title,⟦ author⟧, creator and subject.↵Bang, H.; Agrawal |
| 19264 | 10.68 | . Carbon is also passed onto the animals that eat those plants.↵Radiocarbon dating—also known as carbon dating—is⟦ a⟧ technique used by In this process |
| 28636 | 24.49 | guidance for injections in the shoulder. Moderate-cert⟦ainty⟧ evidence indicates that ultrasound-guided injection in the treatment of shoulder pain probably provides litt |
| 32841 | 9.38 | The Mathematical Principles of Natural Philosophy (c1846), including BOOK III. RULE⟦S⟧ OF REASONING IN PHILOSOPHY↵- ↑ Leng |
| 33943 | 18.66 | build their independent state since the early 1970s. By and large, they have⟦ rejected⟧ Ethiopiyawinet or Ethiopian identity, which the |
| 36948 | 7.13 | editor), Animal Diversity Web.↵Christopher Bonadio (author), University of Michigan-Ann Arbor, Teresa Friedrich (editor), University⟦ of⟧ Michigan-Ann Arbor. |
| 39762 | 23.56 | often have several small teeth along the edge. Frequently the lower leaf surface is silvery, and the upper greenish.↵The small inconspicuous⟦ flowers⟧ |
| 46937 | 32.17 | jacket in good condition, store it in a cool, dry place. Avoid hanging it in direct sunlight or in a damp area, as this can cause⟦ mold⟧ or |
| 50502 | 19.67 | product can expose you to a chemical[s] such as [name or names of chemicals] that is [are] known⟦ to⟧ the State of California to cause cancer |
| 53083 | 3.51 | as a dental surgeon, is a doctor who specializes in the diagnosis, prevention, and treatment of diseases and conditions⟦ of⟧ the oral cavity.↵- Gingivitis |
| 55134 | 16.08 | Health and Nutrition Examination Surveys (NH⟦AN⟧ES) dataset and has found that arthritis among adults with prediabetes was 32 per cent. People suffering |
| 62478 | 16.87 | 97–300.↵Cho HS, Seo SH, Park SH, Park JH, Shin DS, Park IH. Minimal invasive surgery⟦ for⟧ unic |
| 63701 | 31.98 | Aortic St⟦en⟧osis Treatment trial↵- transthoracic echocardiography↵- Received December 20, 2005.↵ |
| 66161 | 25.9 | certain change on the x axis, often taught as "rise over run" in the classroom. The⟦ slope⟧ of this line turns out to be about 3km |
| 67024 | 25.12 | What Is a Slot?↵A slot is a position within a group, sequence or set. It can also refer⟦ to⟧ a specific position in a machine or system |
| 69816 | 15.46 | the slot 23 includes locking lug shoulders 28, such that the shoulders⟦ ⟧28 are selectively received within one of the intersecting slots 2 |
| 76478 | 17.14 | ’t just about style. You also need to be aware of the emotions colors can convey, and how to take advantage of those powerful effects.↵Let’s⟦ review⟧ |
| 78906 | 15.96 | declaration of independence from Great Britain. On June 7, 1776, Richard Henry⟦ Lee⟧ of Virginia introduced a motion in the Second Continental Congress to |
| 81064 | 18.91 | trucks, trailers, food transport, and United States↵- The Global Positioning System (GPS) is⟦ a⟧ radio navigation system that uses information from satellites to calculat |
| 92444 | 22.64 | as well as alter the corresponding neural activity. The major neural underpinnings of mindfulness-based pain reduction consisted of altered activity in the anterior c⟦ing |
| 99080 | 5.83 | appealing situation comedies, to the high-style, verbally elaborate tragedies of Seneca. Although Rome had a native⟦ tradition⟧ of performance, the Hellenization of Roman |
| 99362 | 4.57 | they are willing to accept. He argues that wages are based⟦ on⟧ the marginal productivity of the job, not of the worker. Workers who compete for jobs offered at |
| 102752 | 17.72 | !↵Now, good digestion wait on appetite,↵And health on both!↵- Lennox. May't please your highness sit.↵⟦[⟧The |
| 104797 | 27.97 | Realistic Nit picking Overly emotional Too involved Negative↵An Introduction to Myers Briggs Personality Type Indicator MBTI↵MBTI describes your preferred style⟦ of⟧ beha |
| 106214 | 16.54 | , namely sequences of three shapes presented in a fixed order. Given that the inter-stimulus interval was held constant within and between triplets, the only⟦ cues⟧ |
| 108789 | 8.17 | evolution, as state in his book, The Origins of Species claims,↵“If my theory be true, numberless intermediate varieties,⟦ linking⟧ most closely all of the |
| 110699 | 14.57 | beta carotene converts into a form of vitamin A during digestion called “retinal,” which is important in⟦ maintaining⟧ normal vision.↵Dark, leafy green |
| 116705 | 9.16 | th of the gravitational force. If the Earth stopped rotating (whether suddenly or not), this outward ‘force’⟦ would⟧ cease and we would actually be held more firmly |
| 116890 | 14.44 | The folklore associated with the bird relates to its extraordinary habit of laying its egg in the nest of another bird. The⟦ cuck⟧oo chick hatches before those of the |
| 118267 | 4.15 | mass vaporized upon impact, while hundreds of tons of fragments fell around the crater within a radius of several miles. The site is erroneously named Meteor⟦ Cr⟧ater |
| 119048 | 16.5 | learned the essential teachings of Buddhism from the Lotus⟦ S⟧utra and then expressed its essence as Nam-myoho-renge-kyo, which Nichiren asserted could help |
| 119262 | 9.0 | ,” Buckeridge told Agência FAP⟦ESP⟧. “This potential expansion of sugarcane wouldn’t work without integration between the agricultural and industrial segments, and this |
| 120374 | 11.46 | 30 users can follow your presentation↵- Learn more about this feature in our knowledge base article↵Do you really want to delete this prezi?↵⟦Neither⟧ |
| 126566 | 18.06 | BER: A seizure is defined as a paroxysmal event characterized by temporary involuntary changes⟦ in⟧ the patient caused by abnormal and excessive activity of a group of co |
| 126872 | 10.56 | they know to be true, that is, the parables that Allah has mentioned. This is guidance that Allah grants them; And He mis⟦leads⟧ thereby only the |
| 128837 | 10.98 | the authors hypothesize that augmenting neovascularization by increasing the number of circulating progen⟦itor⟧ cells will improve bony healing.↵Methods: Bilateral full |
| 129424 | 3.83 | to certain limitations, you have the right to (1) request to know more about the categories and specific pieces of personal information we collect,⟦ use⟧, and disclose |
1import torch
2p = torch.load("ae.pt")
3km = {"encoder.weight":"W_enc","decoder.weight":"W_dec","encoder.bias":"b_enc","bias":"b_dec","b_dec":"b_dec"}
4r = {km[k]: v for k, v in p.items() if k in km}
5W_enc, W_dec = r["W_enc"].T.contiguous(), r["W_dec"].T.contiguous() # [d,F], [F,d] (unit-norm rows)
6b_enc, b_dec = r["b_enc"], r["b_dec"]
7def encode(x): # x = RAW layer-42 residuals
8 pre = torch.relu((x - b_dec) @ W_enc + b_enc)
9 return pre * (pre > p["threshold"])