Views
No views yet
facebook/tribev2.| Feature | Description |
|---|---|
| Streaming Inference | Sliding-window real-time predictions from live feature streams |
| ROI Attention Maps | Visualize which brain regions attend to which temporal moments |
| Modality Attribution | Per-vertex importance scores for text, audio, and video |
| Cross-Subject Adaptation | Ridge regression or nearest-neighbour adaptation for new subjects |
| Brain-Alignment Benchmark | Score how "brain-like" any AI model's representations are (RSA, CKA, Procrustes) |
| Cognitive Load Scorer | Predict cognitive demand of media from predicted brain activation patterns |
huggingface-cli login1git clone https://github.com/siddhant-rajhans/cortexlab.git
2cd cortexlab
3pip install -e ".[analysis]"1from cortexlab.inference.predictor import TribeModel
2
3model = TribeModel.from_pretrained("facebook/tribev2", device="auto")
4events = model.get_events_dataframe(video_path="clip.mp4")
5preds, segments = model.predict(events)1from cortexlab.analysis import BrainAlignmentBenchmark
2
3bench = BrainAlignmentBenchmark(brain_predictions, roi_indices=roi_indices)
4result = bench.score_model(clip_features, method="rsa")
5print(f"Alignment: {result.aggregate_score:.3f}")1from cortexlab.analysis import CognitiveLoadScorer
2
3scorer = CognitiveLoadScorer(roi_indices)
4result = scorer.score_predictions(predictions)
5print(f"Overall load: {result.overall_load:.2f}")| Component | VRAM | Notes |
|---|---|---|
| TRIBE v2 encoder | ~1 GB | Small (1.15M params) |
| LLaMA 3.2-3B (text) | ~8 GB | Features cached after first run |
| V-JEPA2 (video) | ~6 GB | Features cached after first run |
| Wav2Vec-BERT (audio) | ~3 GB | Features cached after first run |
src/cortexlab/
core/ Model architecture, attention extraction, subject adaptation
data/ Dataset loading, transforms, HCP ROI utilities
training/ PyTorch Lightning training pipeline
inference/ Predictor, streaming, modality attribution
analysis/ Brain-alignment benchmark, cognitive load scorer
viz/ Brain surface visualization (nilearn, pyvista)facebook/tribev2.1@article{dascoli2026tribe,
2 title={A foundation model of vision, audition, and language for in-silico neuroscience},
3 author={d'Ascoli, St{\'e}phane and others},
4 year={2026}
5}