You can deploy this model and start earning money today!

blanar/
bicompartmental-mlp-results

Views

No views yet

Bicompartmental MLP Gating: Cross-Source Modulation in Transformer Feed-Forward Layers

Implementation and experimental results for bicompartmental MLP gating in transformers, inspired by cortical pyramidal neurons.

Architecture

Standard SwiGLU: out = W_down · (SiLU(W_gate · x) ⊙ W_up · x) — gate and value share input source. Bicompartmental: out = W_down · (σ(W_apical · context + s) ⊙ SiLU(W_up · x)) — gate from external context (attention output), value from input.… See the full description on the dataset page: https://huggingface.co/datasets/blanar/bicompartmental-mlp-results.