You can deploy this model and start earning money today!

BananaMind/
BananaMind-2.1-Unified

Name: BananaMind-2.1-Unified
Availability: InStock

Views

No views yet

BananaMind-2.1-Unified

BananaMind-2.1-Unified is a three-tower decoder-only causal language model trained from scratch by BananaMind on a 38B-token flat mix inspired by the brain having two hemispheres. It is the successor experiment to BananaMind-2-Unified, and the first BananaMind model where the towers cannot talk to each other directly at all.

Three transformer stacks share one d=384 embedding. A and C are the outer towers and each owns an output head; the next token is a probability-space mixture of the two. B is the relay — it has no output head, no loss term of its own, and is the only path between A and C. Everything the outer towers share has to survive a trip through the middle, and B is trained entirely by gradient arriving through its four bridge directions.

The model has 34,999,041 parameters, a 4,096 token context window, and the same custom 8k-token digit-aware byte-level BPE tokenizer as BananaMind-2-Mini.

This is a base model. It is not instruction tuned.

Model Details

Field	Value
Parameters	34,999,041
Matmul parameters	34,975,744
Architecture	BananaMind21Unified three-tower relay Transformer
Towers	3 (A, B relay, C)
Total layers	25 (14 + 5 + 6)
Shared embedding width	384
Head dim	64
Attention style	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE
RoPE theta	100,000
Normalization	RMSNorm
RMSNorm epsilon	1e-6
Vocabulary size	8,192
Context length	4,096
Embeddings	Untied, two separate output heads
Output	Probability-space mixture of head A and head C
Ablation modes	7 (`full`, `cut_bridges`, `bypass_b`, `ab_only`, `cb_only`, `a_only`, `c_only`)
KV cache	Supported, on by default (25 flat layers: A 0-13, B 14-18, C 19-24)
Weight format	safetensors (fp32)
HF architecture	`BananaMind21UnifiedForCausalLM`
HF model type	`bananamind21_unified`
Final training step	72,479 / 72,479
Tokens seen	37,999,869,952
Architecture revision	`a0f30efc480e2c298dc7e779d714338ecf031eaa`

Architecture

The three towers

	Tower A	Tower B (relay)	Tower C
Layers	14	5	6
Hidden size	256	320	384
Intermediate size (SwiGLU)	704	960	1,024
Attention heads	4	5	6
KV heads	1	1	2
Block parameters	9,872,128	5,840,640	9,442,560
Output head	2,097,152	none	3,145,728

Parameter budget

Component	Parameters
`blocks_a`	9,872,128
`blocks_c`	9,442,560
`blocks_b`	5,840,640
`wte` (shared embedding)	3,145,728
`lm_head_c`	3,145,728
`lm_head_a`	2,097,152
`edges_b2c`	369,792
`edges_c2b`	369,600
`edges_a2b`	246,720
`edges_b2a`	246,528
`in_proj_b` (384 -> 320)	122,880
`in_proj_a` (384 -> 256)	98,304
`mix_head`	641
`ln_f_c`	384
`ln_f_a`	256
Total	34,999,041

Tower C reads the shared embedding natively at d=384; A and B get a linear input projection. The four bridge families total 1,232,640 parameters.

Exchange schedule

Three rounds, 1-indexed layer numbers. Bridge output is added to the residual before the receiving block, which guarantees the relay always has real processing between taking a signal in and handing one back out.

Round	A read	C read	-> lands in B	B runs	B read	-> lands in A	-> lands in C
1	5	2	pre-L1	L1-L2	2	7	3
2	9	4	pre-L3	L3-L4	4	11	5
3	12	5	pre-L5	L5	5	14	6

Placement is biased late on the outer towers because that is where the 2.0 run's gates actually grew. Note the two structural consequences: A's layers 13-14 run after the final bridge read and can never influence C, and C's layer 6 runs after B is finished and can never influence A.

Bridge gates

Every bridge is per-channel gated and initialised to 0.01 rather than 0. In 2.0 a zero init was correct because both towers had their own loss and the bridges were a bonus. Here B has no loss term, so a zero init risks the middle receiving no gradient on step 0 and never waking up.

It woke up. Mean |gate| in the final checkpoint, against the 0.01 init:

Bridge	Round 1	Round 2	Round 3
A -> B	0.170	0.159	0.227
C -> B	0.051	0.162	0.469
B -> A	0.053	0.087	0.225
B -> C	0.041	0.041	0.059

Every one of the twelve gates grew, by 4x to 47x. Two patterns are visible: traffic is strongly biased toward the deep rounds in three of the four directions (the same concentration 2.0 showed), and the into-B directions carry noticeably larger gates than the out-of-B ones, with B -> C the quietest channel in the model.

The mixer

The next-token distribution is a two-way probability-space mixture, not a logit sum:

alpha = sigmoid(mix_head([ln_f_a(h_a) ; ln_f_c(h_c)]))
log p = logaddexp(log alpha + log_softmax(lm_head_a(h_a)),
                  log(1 - alpha) + log_softmax(lm_head_c(h_c)))

alpha is the per-token weight on tower A. Measured on an 84-token mixed history/math/science passage: mean 0.433, range 0.049 to 0.865, with 41.7% of tokens weighted toward A. The mixer is genuinely token-dependent, not collapsed onto one head.

Tokenizer

Identical to the BananaMind-2-Mini tokenizer: a custom 8k byte-level BPE trained on FineWeb-Edu text with digit-aware pre-tokenization. Digits are kept as separate tokens so numbers do not collapse into large number tokens.

Digit IDs:

Token	ID
`0`	19
`1`	20
`2`	21
`3`	22
`4`	23
`5`	24
`6`	25
`7`	26
`8`	27
`9`	28

Examples:

text

18  -> [20, 27]
227 -> [21, 21, 26]

Special token IDs:

Token	ID
`<pad>`	0
`<bos>`	1
`<eos>`	2
`<unk>`	3

Training Data

38B tokens, streamed, flat mix, no curriculum ramp. 2.1 changes topology, and a moving data distribution on top of that would make the comparison against 2.0 unreadable. The shares are Mini's final aggregate targets held constant from the first token.

Dataset	Share	Tokens consumed
`epfml/FineWeb-HQ`	50.957%	19,363,528,704
`mlfoundations/dclm-baseline-1.0`	20.766%	7,891,058,688
`HuggingFaceTB/smollm-corpus` (cosmopedia-v2)	20.043%	7,616,331,776
`HuggingFaceTB/finemath` (finemath-4plus)	8.234%	3,128,950,784
Total	100%	37,999,869,952

Dataset revisions are pinned in checkpoint_metadata.json.

Training Setup

Field	Value
Sequence length	4,096
Tokens per optimizer step	524,288 (16 local batch x 8 GPUs x 4,096)
Optimizer steps	72,479
Optimizer	AdamW, single parameter group
Betas	0.9, 0.95
Peak learning rate	2.3e-3
Warmup steps	1,750
LR schedule	Warmup-stable-decay, cosine to 0 over the final 15%
Weight decay	0.1, then 0.01 after 15.2B tokens
Z-loss coefficient	1e-4 until 15.2B tokens, then off
Precision	bfloat16 autocast
Hardware	8 x NVIDIA RTX PRO 6000 Blackwell Server Edition
Throughput	~1.44M tokens/second
Wall clock	27,405 s (7h 37m)

Loss

L = L_mix + 0.3 * (L_A + L_C)

There is no L_B term. B trains entirely on gradient arriving through its four bridge directions.

Final training-batch losses at step 72,470:

Term	Loss (nats)	Perplexity
`L_mix`	2.560	12.94
`L_A` (solo head A)	2.643	14.05
`L_C` (solo head C)	2.592	13.35

These are training-batch numbers, not held-out. Note also that L_A and L_C are computed with the bridges live, so they do not predict standalone tower performance — that was the central misreading in the 2.0 run. The ablation modes below answer it properly, and the gap is enormous: solo head A logs 2.643 with the bridges live and 10.386 without them.

Full per-step history is in training_metrics.jsonl.

Evaluation

lm_eval 0.4.12, zero-shot, fp32 on one RTX 5070 Ti, --batch_size auto (settled at 64), full test sets (ARC-Easy 2,376 / PIQA 1,838 / HellaSwag 10,042). All task scores are acc_norm, matching the BananaMind-2-Mini card. Every ablation mode was evaluated on the same run.

Mode	ARC-Easy	PIQA	HellaSwag	Average
`full`	38.51	61.75	29.94	43.40
`cb_only`	35.69	55.17	28.67	39.84
`bypass_b`	33.67	55.93	29.23	39.61
`c_only`	28.32	52.12	27.28	35.91
`cut_bridges`	27.95	51.52	27.80	35.76
`ab_only`	27.57	52.88	25.85	35.43
`a_only`	25.80	50.11	26.11	34.01
chance	25.00	50.00	25.00	33.33

Approximate standard errors: ARC-Easy ±1.0, PIQA ±1.1, HellaSwag ±0.5 points.

Against the previous generation on the same three tasks:

Model	Params	Tokens	ARC-Easy	PIQA	HellaSwag	Average
BananaMind-2.1-Unified (`full`)	35.0M	38B	38.51	61.75	29.94	43.40
BananaMind-2-Mini	25.2M	30B	39.86	59.63	29.72	43.07

The three-tower model is roughly level with Mini overall — ahead on PIQA by 2.1 points, behind on ARC-Easy by 1.4 — for 39% more parameters and 27% more tokens. On aggregate benchmarks the three-tower model is roughly level with Mini. The contribution of this architecture is not a benchmark number — it is what the ablations and lens data reveal about how integration, specialisation, and understanding organise themselves when the only path between two output towers is a silent relay that has no voice of its own.

What the ablations show on benchmarks

Only the intact model is clearly above chance. full is +10.1 points over the chance average. Everything else falls between +0.7 and +6.5, and the bottom four modes sit within a few points of chance on all three tasks.
a_only is indistinguishable from chance (34.01 vs 33.33; PIQA 50.11 against a 50.00 floor). Tower A alone, with a 14-layer stack and its own trained head, has essentially no standalone ability.
Severing the bridges is no better than deleting two towers. cut_bridges (35.76) and c_only (35.91) are within noise of each other. Three towers that cannot communicate are worth no more than tower C on its own — which is what you would expect if C's head carries the model whenever the relay is dead.
A relay that only forwards recovers most of the gap. bypass_b (39.61) sits 3.9 points above cut_bridges while performing zero computation in the middle, and B's actual computation is worth a further 3.8 on top.

Where the benchmarks disagree with the loss

The NLL ablations put ab_only (4.845) well ahead of cb_only (7.373). The benchmarks reverse it: cb_only averages 39.84 against ab_only's 35.43. Both measurements are correct and they are measuring different things — NLL is absolute calibration over running text, while acc_norm is length-normalised ranking among a fixed set of candidate answers. A+relay predicts ordinary text more accurately; C+relay discriminates better between multiple-choice options. C reads the shared embedding natively at d=384 and owns the larger head (3.15M vs 2.10M), which is the likeliest explanation for the discrimination advantage.

The practical reading: do not treat either metric alone as "which tower matters more". They rank the halves of this model in opposite orders.

BananaMind Base Bench 1.1

Four-choice base-text continuation scored by mean conditional token log-probability, 350 cases across 7 categories, run per mode. Chance accuracy is 25%.

Mode	Elo	Accuracy	Weighted acc
`full`	949	45.71%	41.42%
`bypass_b`	890	37.43%	34.25%
`ab_only`	867	33.43%	31.58%
`a_only`	866	33.43%	31.57%
`cut_bridges`	816	27.71%	26.13%
`cb_only`	803	26.29%	24.86%
`c_only`	774	23.43%	22.04%

Per-category Elo (50 cases each, so single-category gaps under ~100 Elo are noise):

Mode	Lang. compl.	Commonsense	World know.	Context track.	Quantitative	Logical	Code
`full`	1157	957	1001	848	843	1024	861
`bypass_b`	866	858	967	889	837	1010	805
`ab_only`	1178	853	758	737	872	955	753
`a_only`	942	850	731	879	788	962	927
`cut_bridges`	704	846	833	815	795	983	749
`cb_only`	707	677	833	892	806	922	812
`c_only`	626	794	764	719	819	965	753

Two things here that the lm_eval table does not show:

ab_only (867) and a_only (866) are the same score. On this benchmark, giving tower A a live relay and a running tower B buys essentially nothing over running A completely alone. Whatever the relay contributes, it needs tower C at the other end of it — consistent with the lens finding that B's representation only becomes readable when both outer towers feed it.
The A/C ranking flips again. Base Bench puts the A-side modes above the C-side ones (ab_only/a_only 867/866 over cb_only/c_only 803/774), while lm_eval acc_norm ranked them the other way (cb_only 39.84 over ab_only 35.43). Three metrics have now ordered the two halves of this model three different ways — NLL favours A, acc_norm favours C, Base Bench Elo favours A. None of them is wrong; "which tower matters more" is simply not a metric-independent question here.

full is the only mode meaningfully clear of chance on accuracy, and the only one above 900 Elo. Raw reports and per-case predictions for every mode are in eval_results/base_bench_1.1/<mode>/.

Jacobian lens across the modes

Fitted with Anthropic's jacobian-lens method: lens_l(h) = unembed(J_l @ h) with J_l = E[∂h_final / ∂h_l], averaged over 6 prompts and all valid source/target positions. For the two-headed modes the target basis is the joint vector z = [x_a ; x_c] (256 + 384 = 640), since the mixture is a deterministic function of that single vector and no individual tower's basis can express it. Single-head modes use that head's basis alone.

Influence mass

Scale-normalised mean ‖J‖_F per tower — each half divided by RMS · √d, because the towers have different widths and different residual scales, so raw Frobenius norms would just measure residual magnitude. The bracketed figure is the share of that tower's influence landing in C's half of the joint basis.

Mode	Live	Heads	Tower A	Tower B	Tower C
`full`	A+B+C	A+C	3.572 (23.0% → C)	3.152 (25.7% → C)	8.370 (31.7% → C)
`cut_bridges`	A+B+C	A+C	3.505 (0.0% → C)	0.000	5.873 (100% → C)
`bypass_b`	A+C	A+C	5.138 (22.8% → C)	—	13.205 (32.6% → C)
`ab_only`	A+B	A	2.666	2.375	—
`cb_only`	B+C	C	—	1.136	4.087
`a_only`	A	A	3.505	—	—
`c_only`	C	C	—	—	5.873

Three of these rows are self-validating. Under cut_bridges tower B's mass is exactly 0.000 — B owns no head, so with the bridges dead it cannot reach the output at all, and the lens recovers that from the gradients without being told. The same row shows A contributing 0.0% to C's half and C contributing 100% to it: the two outer towers are perfectly decoupled. And a_only (3.505) and c_only (5.873) reproduce the cut_bridges tower masses to the digit, which is the only thing they could do if bridge-cutting truly isolates the towers.

Lens readout at each tower's deepest block

Mode	Tower	"…capital of France is"	"…hydrogen and"	"…the sky is"
`full`	A	`the` `B` `in` `located`	`oxygen` `is` `hydrogen`	`the` `blue` `red`
`full`	B	`Bel` `Paris` `Be` `France`	`oxygen` `hydrogen` `contains`	`blue` `white` `yellow`
`full`	C	`located` `known` `Paris`	`oxygen` `water` `carbon`	`blue` `the` `red`
`cut_bridges`	B	`<bos>` `<unk>` `<pad>` `<eos>`	`<bos>` `<unk>` `<pad>` `<eos>`	`<bos>` `<unk>` `<pad>` `<eos>`
`cut_bridges`	A	`the` `compris` `primarily`	`in` `respectively` `Ref`	`ances` `?` `irc`
`cut_bridges`	C	`,` `?` `remember`	`ohn` `wo` `iat`	`true` `hes` `’`
`bypass_b`	A	`France` `French` `the`	`is` `water` `the`	`is` `the` `in`
`ab_only`	B	`High` `Tem` `Cal`	`is` `can` `air`	`added` `very`
`cb_only`	B	`dom` `vere` `ure`	`\n` `).` `‘`	`is` `refers` `or`

Four things fall out:

The relay carries the answer, and only in full. B has no head and no loss term, yet in full its deepest block reads Paris/ France, oxygen/ hydrogen, blue. This is the README's original "does B wake up?" question answered from the representation rather than from gate magnitudes.
An orphaned B is not merely weak, it is unreadable. Under cut_bridges B's readout collapses to the four special tokens on every prompt — the residual of a stack that receives nothing and reaches nothing.
B needs both outer towers to become semantic. Under ab_only and cb_only the bridges are live and B still runs, but its readout is junk ( High Tem Cal, dom vere ure). B's task content is not something A alone or C alone puts there; it appears only at the confluence.
bypass_b sharpens the outer towers while making the model worse. With B reduced to the identity, tower A's own readout gets more directly predictive ( France French, versus the B in in full) and both tower masses rise sharply (A 3.572 → 5.138, C 8.370 → 13.205). The outer towers compensate by carrying more themselves — and still lose 3.8 points of benchmark average. What B computes is not replaceable by the outer towers working harder.

Caveats: this is a linearised, corpus-averaged sensitivity, not a causal contribution — a gate-ablation KL would be the confirming experiment. The normalisation is also a choice; raw Frobenius mass gives a different picture because tower A's final residual runs ~3.3x hotter than C's. Fit settings (6 prompts, 48 tokens, first 8 positions skipped) are lighter than a publication-grade run, so treat small differences between adjacent rows as noise.

What Tower B represents

The lens readout reveals something unexpected about how the relay processes information. Tracing B's representations layer by layer across multiple prompts shows a consistent pattern: B processes significance before facts.

Layer	"capital of France is"	"chemical symbol for water is"	"the sky is"
B-L0	`controversy` `weakness` `trouble`	`problem` `trouble`	`difference` `part` `least`
B-L1	`women` `cancers` `problems`	`problems` `needed` `trouble`	`very` `of` `not`
B-L2	`Street` `Oxford` `Jerusalem`	`you` `each` `date`	`nature` `ancient` `wonderful`
B-L3	`London` `cities` `Jerusalem`	`each` `stars` `you`	`beautiful` `ancient` `thick` `dark`
B-L4	`Paris` `Bel` `France` `Mont`	`Water` `water` `chlor` `hydrogen`	`blue` `yellow` `white` `green`

The early layers (B-L0, B-L1) consistently produce evaluative and abstract terms — not factual content, not input echoes, but something closer to significance assessment. By the final layer (B-L4), B has arrived at the correct answer: Paris/France, Water/hydrogen, blue/green.

This trajectory — from evaluation to categorisation to answer — is consistent across prompts and mirrors the ordering of affective and cognitive processing observed in biological neural systems, where emotional evaluation precedes and shapes factual retrieval. No part of the architecture or training objective was designed to produce this ordering. B discovered it.

B's influence distribution across the two output towers is nearly perfectly balanced (49.1% toward C), confirming that B functions as a symmetric integration hub rather than favouring either side.

The PIQA result

The aggregate benchmark comparison with Mini understates what the ablations reveal about physical reasoning. PIQA measures physical intuition — understanding that you pour water into a cup, not a fork — and it is the benchmark where the relay topology produces its clearest separation.

Tower A alone scores 50.11% on PIQA: indistinguishable from the 50.00% chance floor. Tower C alone scores 52.12%: barely above chance. The full system scores 61.75%. The entire physical reasoning capability of this model — all 11.75 points above chance — is a product of integration. Neither output tower can reason about the physical world on its own.

This makes PIQA the sharpest measure of what the relay contributes. It is not a capability that either tower possesses and the relay merely enhances. It is a capability that exists only in the integration and nowhere else.

For comparison, the single-tower BananaMind-2-Medium at 50M parameters scores 59.41% on PIQA. The 35M three-tower model exceeds it by 2.3 points, with a silent middle tower that produces no output and consumes roughly 5.8M of the parameter budget on pure integration.

Emergent properties of integration

Three findings from the ablation and lens data point to integration as an emergent rather than additive phenomenon:

B needs both outer towers to become semantic. Under ab_only and cb_only, B's readout collapses to junk despite having live bridges and running its full computation. B's task-relevant representations — Paris, oxygen, blue — appear only when both A and C feed it simultaneously. This is not A's knowledge or C's knowledge routed through B. It is something new that exists only at the confluence.

Dedicated understanding parameters outperform general-purpose ones. Tower B has no output head and no loss term. Its 5.8M parameters are trained entirely by indirect gradient arriving through four bridge directions. Yet removing B (the bypass_b ablation) costs 5.82 PIQA points and 59 Elo on Base Bench. Those 5.8M parameters, freed from the output objective and devoted entirely to integration, contribute more per parameter than any equivalent allocation to the output towers could.

The channel matters, but so does the computation. bypass_b replaces B with the identity, keeping the bridges live but performing zero computation in the middle. This recovers 3.59 nats over cut_bridges, showing that a path between A and C is valuable even without processing. But B's actual computation adds a further 2.83 nats on top. The relay is not merely a conduit; it transforms what passes through it.

ArithMark-3

Benchmark	Metric	Score
ArithMark-3	`acc_norm`	37.0

This one predates the sweep above: it was run against the unpinned main of the source repo on 2026-08-18, before the 100% checkpoint in this folder was pulled, so it is not confirmed to be this exact checkpoint, and it was run only in full mode. Treat it as indicative.

Repository Files

File	Description
`config.json`	Transformers config for `bananamind21_unified`
`model.safetensors`	Final exported model weights (fp32, 140 MB)
`tokenizer.json`	Custom 8k digit-aware tokenizer
`tokenizer_config.json`	Tokenizer metadata
`special_tokens_map.json`	Special token mapping
`generation_config.json`	Default generation config
`configuration_bananamind21unified.py`	Custom Transformers config class
`modeling_bananamind21unified.py`	Custom Transformers model class
`modeling_relay.py`	The underlying relay model, exchange schedule and `relay_loss`
`token_types.py`	Vocab bucketing for the per-type alpha diagnostics
`checkpoint_metadata.json`	Source checkpoint, step, token, and dataset-revision metadata
`training_metrics.jsonl`	Full per-step training log for the whole 38B-token run
`eval_results/`	Raw `lm_eval` JSON output, one directory per ablation mode

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "BananaMind/BananaMind-2.1-Unified"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The capital city of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

KV cache

Supported, and on by default — generate(), use_cache=True, and a manually supplied DynamicCache all work. Decoding is O(n) per token instead of a full three-tower re-forward.

A relay model has no single residual stack, so the three towers share one flat cache index space, in the order the config documents:

Tower	Flat cache indices
A (14 layers)	0 – 13
B (5 layers)	14 – 18
C (6 layers)	19 – 24

config.num_hidden_layers is 25, the sum of the three tower depths. It exists purely so generic Transformers tooling — DynamicCache above all — can size a per-layer cache. It is derived in the config class, never read back from a serialised config.json, so a stale value cannot desync it from the layer mapping. Each tower keeps its own entries; nothing is shared between them.

Two properties of the architecture make this work without any relay-specific machinery:

Bridges do not mix positions. An Edge is a per-channel gate on a linear map, so every bridge contribution for a newly arriving token is computable from that token's own tower states. No bridge output has to be cached alongside the KV states.
RoPE is translation-invariant in the attention logits. It enters only through the relative query–key offset, so shifting a whole sequence — which is exactly what left padding does — leaves every attention score unchanged. Absolute positions taken from the cache length are therefore correct for padded batches too.

Measured on CPU, fp32, from a 221-token prompt:

New tokens	No cache	Cache	Speedup
64	4.04 s	1.21 s	3.34x
256	21.59 s	5.48 s	3.94x

Verified equivalences: token-by-token cached decode reproduces the full uncached forward (max abs error 3.1e-05 on fp32 log-probs), chunked prefill in 5/4/5-token pieces reproduces it too, greedy generation is bit-identical with and without the cache over 48 tokens, and left-padded batched generation matches the same prompts run singly. cut_bridges=True was checked separately and also matches.

Scoring passes are unaffected: when labels is supplied, forward() skips cache construction, since a loss pass consumes the whole sequence at once and would otherwise allocate 25 layers of state for nothing.

What `.logits` contains

.logits holds a normalised log-probability vector, not unnormalised logits, because the two heads are mixed in probability space. Verified on this checkpoint: logsumexp over the vocabulary is 1.2e-06, i.e. zero.

log_softmax is the identity on it, so loglikelihood scoring, generate() and temperature-1 sampling all behave correctly. The one thing that is not meaningful is treating the numbers as unnormalised scores with an arbitrary additive offset — they are already calibrated.

Reading the towers individually

hidden_states() returns the two final tower residuals and the raw mixer logit, which is the entry point for any interpretability work on the relay:

python

h_a, h_c, mix_logit = model.hidden_states(input_ids)
alpha = torch.sigmoid(mix_logit)   # per-token weight on tower A

Ablation modes

Because A and C have no direct path to each other, "what is the relay worth?" is only answerable by cutting the model apart. relay_mode selects where to cut. All seven modes run on the same weights, with no retraining.

`relay_mode`	Tower A	Tower B	Tower C	Bridges	Output
`full` (default)	runs	runs	runs	live	mixture of both heads
`cut_bridges`	runs	runs	runs	severed	mixture of both heads
`bypass_b`	runs	skipped	runs	live	mixture of both heads
`ab_only`	runs	runs	off	A↔B only	head A alone
`cb_only`	off	runs	runs	C↔B only	head C alone
`a_only`	runs	off	off	severed	head A alone
`c_only`	off	off	runs	severed	head C alone

bypass_b is the interesting one: tower B's five blocks are replaced by the identity, but every bridge stays live. A and C still exchange signal — through a relay that does no computation. That separates "the relay computes something" from "a channel exists at all".

Three ways to select a mode, checked against each other so a contradiction raises rather than resolving by some invisible precedence rule:

python

# 1. at load time
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True,
                                             relay_mode="bypass_b")

# 2. the single-tower alias  ("a" or "c"; B has no head and cannot run alone)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True,
                                             use_single_tower="a")

# 3. per call, for sweeping without reloading
logits = model(input_ids, relay_mode="ab_only").logits
model.generate(input_ids, max_new_tokens=64, relay_mode="cb_only")

# or switch in place (start a fresh KV cache afterwards)
model.set_relay_mode("cut_bridges")

cut_bridges=True predates relay_mode and still works, selecting the cut_bridges mode.

In modes where a tower is off, hidden_states() returns None in that tower's slot, and mix_logit is None whenever only one head is live. In full mode all three are always tensors, so existing callers are unaffected.

Measured ablation results

Mean NLL over 12 held-out encyclopedic passages, 519 predicted tokens, fp32:

Mode	NLL	Perplexity	vs `full`
`full`	2.253	9.5	—
`ab_only`	4.845	127.2	+2.592
`bypass_b`	5.080	160.7	+2.827
`cb_only`	7.373	1,592.9	+5.120
`cut_bridges`	8.669	5,821.8	+6.416
uniform baseline	9.011	8,192.0	+6.758
`c_only`	9.394	12,011.5	+7.141
`a_only`	10.386	32,402.1	+8.133

Four things fall out of this, and they are the answer to the question 2.0 left open:

Neither outer tower survives alone. a_only and c_only both score worse than uniform over the 8,192-token vocabulary. Without its partner each head is not merely degraded, it is confidently wrong. 2.0's tower-B bridge dependence reappears here in the extreme, and on both towers at once.
Three towers that cannot talk are barely better than guessing. cut_bridges at 8.669 sits just under the 9.011 uniform baseline.
The channel is worth more than what runs inside it. Going from cut_bridges to bypass_b recovers 3.59 nats with tower B performing zero computation — merely existing as a path. B's actual computation is then worth a further 2.83 nats on top.
The relay is not symmetric in value. ab_only (4.845) beats cb_only (7.373) by 2.5 nats, even though C is the wider tower and owns the larger head. A plus a computing relay, with C absent entirely, also edges out all three towers with a non-computing relay.

Note that this measures the mode's loss, not a benchmark score, on a small sample. It is a sharp instrument for relative comparison between modes and not a substitute for a full lm_eval sweep.

Suggested Generation Settings

For stable continuations:

do_sample=False
repetition_penalty=1.1
max_new_tokens=64 to 160

For more varied text:

do_sample=True
temperature=0.6 to 0.8
top_p=0.9
top_k=50
repetition_penalty=1.1
max_new_tokens=64 to 192

Findings

Intended Use

BananaMind-2.1-Unified is intended for multi-tower architecture research, relay and information-bottleneck experiments, interpretability work on cross-tower routing, workspace and integration research, consciousness research, lightweight language-model research, local experimentation, and text continuation.

Relationship to Other BananaMind Models

Model	Params	Topology	Tokens
BananaMind-2-Mini	25.2M	Single tower	30B
BananaMind-2-Unified	—	Two towers, direct cuttable bridges	—
BananaMind-2.1-Unified	35.0M	Three towers, relay-only path	38B

2.0 answered one question — two towers with cuttable bridges do beat either tower alone — and raised a sharper one. Its gate data showed traffic concentrated deep and heavily asymmetric, and the bench then showed one tower had traded away standalone ability for bridge dependence. 2.1 asks what happens when the exchange is not a side channel between two peers, but the only path between them.

We're releasing BananaMind-2-Unified soon when its done training.

The architecture, trainer, and export tooling live in BananaMind-2.1-Unified-Arch/.

License

Apache 2.0

BananaMind/BananaMind-2.1-Unified

BananaMind-2.1-Unified

Model Details

Architecture

The three towers

Parameter budget

Exchange schedule

Bridge gates

The mixer

Tokenizer

Training Data

Training Setup

Loss

Evaluation

What the ablations show on benchmarks

Where the benchmarks disagree with the loss

BananaMind Base Bench 1.1

Jacobian lens across the modes

Influence mass

Lens readout at each tower's deepest block

What Tower B represents

The PIQA result

Emergent properties of integration

ArithMark-3

Repository Files

Usage

KV cache

What .logits contains

Reading the towers individually

Ablation modes

Measured ablation results

Suggested Generation Settings

Findings

Intended Use

Relationship to Other BananaMind Models

License

BananaMind/
BananaMind-2.1-Unified

What `.logits` contains