You can deploy this model and start earning money today!

Dhruv1000/
cet-vit-cifar100

Views

No views yet

CET-ViT: Causal Emergence Transformer for Vision

Model Description

CET-ViT introduces a learnable causal emergence operator (V-CEO) inside a hierarchical Vision Transformer. Instead of fixed pooling, V-CEO learns a soft assignment matrix S that groups patch tokens into macro visual regions guided by Effective Information (EI) — a measure of causal power.

Key innovation: The first end-to-end trainable causal emergence operator inside a ViT, using SVD-based dynamical reversibility as a differentiable EI proxy (Yuan et al., npj Complexity 2025).

Results on CIFAR-100 (training from scratch)

Model	Params	Val OA	Val F1-macro	Reversibility
CET-ViT (ours, with EI)	33M	73.32%	73.12%	0.9162
CET-ViT ablation (no EI)	33M	TBD	TBD	~0.35-0.45

Causal Emergence Probing

Mean reversibility: 0.9162 ± 0.012 (STRONG causal emergence)
Mean K selected: 8.4 / 32 (model self-organized scale)
EI trajectory: -0.48 (init) → -0.87 (epoch 300)

Architecture

Scale: base (embed_dim=192, depths=[2,6,6,2])
Parameters: 33M
V-CEO: Dynamic K selection via Gumbel-softmax + soft assignment S
Loss: CrossEntropy + λ·SVD-EI loss (cosine warmup epochs 50-200)
Training: 300 epochs, batch 128, Mixup α=0.8, AutoAugment, RandomErasing

Training Config

python

make_model_config(
    scale="base", img_size=32, patch_size=2, num_classes=100,
    drop_path_rate=0.3, drop_rate=0.1, attn_drop_rate=0.1,
    k_candidates=[4, 8, 16, 32],
)
# lambda_ei_end=0.05, warmup_epochs=50-200 cosine
# lr=5e-4, weight_decay=0.1, AdamW

Citation

bibtex

@article{cetvit2026,
  title={CET-ViT: Causal Emergence Transformer for Vision},
  author={Dhruv Jyoti Das},
  year={2026}
}

Dhruv1000/cet-vit-cifar100