Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
qwen36-crest-cognitive-heads – AI Model by caiovicentino1 | AlphaNeural AI
You can deploy this model and start earning money today!
caiovicentino1
/
qwen36-crest-cognitive-heads
like
0
mechanistic-interpretability
cognitive-heads
crest
hybrid-moe
2512.24574
Qwen/Qwen3.6-35B-A3B
finetune
mit
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
Qwen3.6-35B-A3B Cognitive Heads — CREST Phase 1
Per-head logistic regression probes trained on Qwen3.6-35B-A3B Gated-Attention layers (L3, L7, L11, L15, L19, L23, L27, L31, L35, L39) × 16 Q heads = 160 total heads.
Replication of CREST (arXiv:2512.24574) methodology adapted for hybrid MoE+GDN+GA architecture.
Data
Source: Qwen3.6-35B-A3B MCR Stage B corpus (624 rollouts, 310 correct / 314 wrong)
Activation: pre-o_proj output at last prompt token
Probe: Logistic regression (C=0.5) with 5-fold CV
Task: binary correct vs wrong
Results
Top-20 cognitive heads identified (see
crest_phase1_results.json
)
Top-38% default (60/160 heads) marked as cognitive
Files
head_acts.npz
— raw per-head activations (711 × 10 × 16 × 256, fp16)
probe_coefs.npz
— compact probe coefficients + CV accuracies
crest_phase1_results.json
— full results including coefficients
Phase 2 (next)
Apply norm-preserving rotation at top-60 cognitive heads during inference, eval on GSM8K + MATH500, compare vs base model.
