This wasn't meant to be seen by others, but it's too damn messy so there you go. I can't keep rewriting the same explanations over and over
to justify the empirical data here.
Almost within the dimensional spectrum of aligned label counts
image
The 512 anchor system seems perfectly suited for text + vision combinatory shared space, but the concern of
baking the two together means they can never be separated.
Even now the label system is essentially baked into the soup as a sinking signal to create output,
but the output isn't the geometric structure that needs to exist to apply an nth anchored patchwork to
a representational space as necessary just yet.
The outcome of the tests show the 80 BCE labels pretty much cover what they need on the embedding space
and the rest is just whatever. They may or may not snap to it.
I forced anchor hops through dropout, otherwise the system would just default to CLS anchor because
downstream funnels more conveniently that way. Path of least resistance.
Soup imperfect
Even with the high fidelity structural integrity, the structure still finds a way to learn differently than the intended outcome's shape.
I'll be debugging for the coming hours until I get the exact topology I need.
image
Invariantly no matter how many safeguards I instill, the objective somehow finds a way to skip steps.
Without the full anchored topology in the trained and prepared orthogonality, with the correct association, the correct differentiation, the
correct offset, the correct procrustes alignment, and the correct... y'know what it needs about a billion needles lined up.
This might be a multipass train now matter how I size it up.
Heavy Soup
2048 anchor soup going to take a few hours to cook but it'll be ready.
image
Bert just kinda works like this, but the VIT structure doesn't. The patches require training.
image
V3 experiments incoming
This is a highly experimental repo, expect rapid changes.
I've begun training a much thicker soup with a ton more anchors and the advanced constellation. That should help keep cohesion much cleaner.
Upgrades V3
Anchor dropout, which helps discourage anchor dependency.
image
Residual access to hypersphere params in the transformer path. Potentially produces more accurate downstream for no cost params.
Fused and expanded anchor constellation to support a wider formula for experimentation on the geolip-vit-tiny 256
GeoLIP ViT Base x3
Geometric vision system: 3-expert consensus soup + from-scratch ViT encoder.
Components
1. Base Tier Soup (teacher)
800K parameter geometric fusion of 3 pretrained vision experts on a 128-d hypersphere.
11M parameter ViT trained from Xavier initialization against the soup's consensus targets. No pretrained weights anywhere. Same architecture pattern as CaptionBERT.
All metrics still climbing at E20. Model needs 60-90 epochs to fully converge (matching CaptionBERT's text encoder trajectory).
Architecture
Training (soup as teacher):
3 expert features → Procrustes projectors → mean → L2-norm → 128-d consensus targets
Raw images → from-scratch ViT → 128-d embedding
Losses: InfoNCE + MSE + CV + BCE(through frozen soup) + Procrustes alignment
Geometric autograd: tangential=0.01, separation=1.0
Inference (standalone):
Raw image → ViT encoder → 128-d embedding (on hypersphere)
No experts needed. Geometry is baked in.
Key Findings
800K soup params beat 81.7M (34-expert soup at 0.732 mAP) and 75.6M (34-expert bank at 0.782 mAP)
Proper calibration (GPA + whitened Procrustes + measured CV target) is essential — without it, constellation collapses to 1/256 active anchors
From-scratch ViT learns the 3-expert consensus representation from raw pixels with the same convergence dynamics as CaptionBERT on text
Cross-model weight cosine is 0.000 but activation Procrustes is 0.999 — the models encode identical geometry through completely different weight configurations
Experiment 2.5 Update: COCO convergence is slow but steady.
BCE loss isn't the best catalyst for geometry but it does work to funnel through an aligned transformer.
I underestimated the complexity of associative cross-modal differences, but it is converging. Shared space is a very tricky
catalyst to teach as an associative connection. Routing is easy, distilling is not as easy with multimodal structures and multiple
adjacent representations used as loss learning targets.
If this fails to meet direct expectations, I'll form a proper hub and teach using the bertenstein method. Bertenstein works because it's
always expecting to hear from the experts and there is always one anchored expert in charge.
The expert/student distillation process requires skilled teachers with similar utility, which is different than simply funneling
information through a route and pooling it.
geolip-captionbert-8192 accepts this pooled funneled information and produces useful output due to the shared
expert informations having similar access utilities.
In either case, geolip-captionbert-8192 was trained from scratch and so is this model. They are not inheriting weights from any large-training,
they are inheriting the geometry and structure through distillation in order to represent complex structure that quite simply should not exist
in the smaller model by direct implicit learning.
geolip-vit-x3 must learn to predict the pixel data using the output of the experts as markers for loss, which means it can never get a full picture of anything outside of it's own tools.
This model is exceptionally small, absurdly small even by vit standards. This is because even at this size, this is too much. The model cannot
overfit if the model uses every tool at the expense, this model will train indefinitely unless a cascade overflow happens, a math continuity corruption
occurs, or the substructure collapses to a simpler shortcut-centric behavior that would require scrambling.
The anchors are strong enough and tuned to the experts, the external losses are tuned to teach the expert responses, the expert data is used
as loss methods of attenuation, and the structure conforms to those losses specifically because it's required to teach the model tobe
standalone and compliant without requiring the experts later.
I gave the model everything I could geometrically, and it must discover the way to connect them.
I'm teaching siglip, dinovit and clip-vit to communicate on the same manifold. They are essentially speaking three dialects of foreign offshoot
evolved thousand year later Roman.
The fact that this works at all is a testament to the hypersphere attenuation.
The xavier aligned and procrustes embedding array attached to a standard patch16 subset should suffice.
I'll be training this like CaptionBERT but with a twist, the soup expert is the alignment bank for this one, and I trained it first instead of later.
The alignment and R1 is nearly perfect, so it should be cohesive enough through the chain of conceptualization to coalesce through the implications.
Now it's another story, if the actual patches will learn based on the embedding and encoding spectrum, and how quickly I can make them learn.
The output this encoder produces is a 128 dimensional enriched representational lookup plane on a hypersphere.
This is more than enough information to house access to any data route that exists.
The dimensional spectrum of a 5d object is so expansive and so enriched, that the entire spectrum of this shape requires a specific
curation of the behavior. This is what most of the mechanisms are tasked with overall, pruning the effect of rigidity indifference preservation
on the hypersphere represented structure.
In other words, that 128 dimensions represents more information than I could express with words.
R@1= 97.1%, not quite but getting there. Experiment 2 was successful enough to push harder in this direction.
Anchor collapse says it doesn't need all those anchors. It started grabbing at more by the end, which means
the system was aligned and then started growing further on a constraint that I was unaware of.
This drift curve needs to be controlled. Direct anchored emergence while training is risky. The bank itself
survived so well because it was anchored post training, which gave added cohesion and complexity association
that I have yet to discover the runtime process to train. I will be analyzing the emergence to preserve the anchoring.
Total collapse. The three models did not conform and the patchwork did not learn. The objectives are not correct.
One anchor was defaulted to, none of the others utilized. The memory bank solves this problem through queue assessment with the INFONCE hub processing,
but this model is a different form of anchoring that did not work.
THE ENTIRE MODEL became the anchor, instead of the anchorpoints within the model. I'm thinking there wasn't enough scattering, so I'll try some additional tweaks.
Post
Active anchors: 1/256 (0.4%)
Every single image → anchor 65
Anchor entropy: 0.0000
Anchors within cos>0.5 per image: 1.0
Nearest anchor dist: 0.016 — next nearest: 0.665
Effective dim: 23.6/128
Top-20 SVs explain 99.2%
Self-sim off-diag: 0.969
Expert uniqueness: 0.0008–0.0011
There is only one active anchor, which is essentially CLS. The uniqueness collapsed. The distance is fine, the entropy is dead.
Shortcut bypass, additional nonlinearity must be made.
Assessment
Without the centered procrustes loss the same result happened. The collapse forms around one of the earlier anchors, around the outside middlepoint of
where all three models are simultaneously rotating around a point, which is not the direct center.
This point has noise, invalidity, incorrect association, and additional problems based on the attention mechanisms internally to the models queried.
Hypothesis based on research
The procrustes alignment must align centerwise, and it must be defined specifically to specifications.
Revision note (2026-07-27)
modeling_geolip_vit.py fixed: the original file did not match the released
checkpoint (transformer saved under encoder.layers.*, class declared
layers.*, plus an untrained geometric-injection path), so AutoModel
silently returned a mostly-random model. The fix loads with zero missing /
zero unexpected keys and reproduces the recorded COCO metrics (anchors
94/256, mAP ~0.43). Same fix applied to geolip-vit-large-x3.
