VEDB and Reference SimCLR ResNet-18 — NSD Voxelwise Encoding Models
This repository contains subject-specific NSD voxelwise encoding-model fits and the variance-partitioning fits reported in the associated study from:
Diaz, D. M., & Henderson, M. M. (2026). Eccentricity-Constrained CNN Training Reveals Adaptive Information Coding Around the Visual Field. Proceedings of the 9th Conference on Cognitive Computational Neuroscience.
Voxelwise encoding models were fit to fMRI responses from the Natural Scenes Dataset (NSD) using features extracted from seven SimCLR ResNet-18 models evaluated in the associated study.
The models comprise four ResNet-18 encoders pretrained on the Visual Experience Dataset (VEDB) under different visual-field conditions:
Baseline
Fovea-Gaze
Periph
Periph-NF
and three non-egocentric reference models pretrained on conventional image datasets:
STL-10
ImageNet-100
ImageNet-1K
The ImageNet-100 and ImageNet-1K reference models were trained for the associated study using SimCLR implemented with the Lightly self-supervised learning framework. The STL-10 model was obtained from the pretrained ResNet-18 SimCLR release provided by Spijkervet/SimCLR.
The reference models provide comparison points for evaluating how representations learned from naturalistic, gaze-aligned egocentric visual experience correspond with human visual cortex relative to representations learned from conventional image datasets.
Each encoding-model folder contains voxelwise fits for NSD subjects S1–S8. The four VEDB folders contain fits derived from the Baseline, Fovea-Gaze, Periph, and Periph-NF models. The reference-models/ directory contains the corresponding fits derived from the STL-10, ImageNet-100, and ImageNet-1K reference models. The variance-partitioning/ directory contains the two model comparisons reported in Figure 4C–D of the paper, with one fit for each of the same eight NSD participants.
Overview of the VEDB preprocessing, SimCLR pretraining, downstream linear probes, and voxelwise encoding workflow.
NSD Data and Evaluation
Voxelwise encoding models were fit using fMRI responses from the Natural Scenes Dataset (NSD), which contains 7T whole-brain fMRI measurements collected while participants viewed large sets of natural-scene images. The present analysis used data from 8 NSD participants.
For each participant, the 1,000 images shared across all NSD participants were reserved as the final held-out evaluation set. The remaining approximately 9,000 participant-specific images were used for model fitting and regularization selection. Within this fitting set, a nested holdout partition was used to select the ridge penalty independently for each voxel.
The same intact 224 × 224 NSD images were presented to each of the seven pretrained ResNet-18 models. The Baseline, Fovea-Gaze, Periph, and Periph-NF transformations were applied only during VEDB SimCLR pretraining and were not reapplied to NSD stimuli during feature extraction. The same NSD feature-extraction procedure was used for the STL-10, ImageNet-100, and ImageNet-1K reference models.
Encoding performance was evaluated on the held-out 1,000-image set using voxelwise coefficient of determination (R²) and Pearson correlation (corr). These held-out metrics are stored directly in each released encoding-model fit.
Encoding-Model Procedure
The encoding models were fit separately for each NSD subject (S1–S8) and each of the seven pretrained visual models.
NSD stimuli were presented to the pretrained ResNet-18 models as intact 224 × 224 images. The Baseline, Fovea-Gaze, Periph, and Periph-NF transformations were used during SimCLR pretraining and were not reapplied to NSD images during encoding-model evaluation.
Before feature extraction, NSD images were rescaled to [0, 1] and normalized using ImageNet channel statistics:
Convolutional feature maps were reduced with adaptive average pooling and then flattened. PCA was applied separately to each layer, retaining the top 200 principal components. The six 200-component representations were then concatenated before voxelwise model fitting.
PCA was fit separately for each subject, model condition, and feature layer. In the analysis used for the study, PCA was applied to the full subject-specific feature matrix before the encoding-model train and holdout partitions were applied.
Feature Normalization and Ridge Fitting
For each subject, the feature matrix was divided into a training partition, a nested holdout partition for ridge-penalty selection, and a final held-out evaluation partition containing the 1,000 NSD images shared across participants.
Feature-wise means and standard deviations were estimated from the combined training and nested-holdout partitions. The same parameters were then used to z-score all three partitions. The final held-out evaluation set was excluded when estimating these normalization parameters.
A column of ones was appended to the feature matrix to provide an intercept term.
Voxel responses were modeled with L2-regularized linear regression (ridge regression). Twenty candidate ridge penalties were evaluated. The penalty that minimized prediction error on the nested holdout partition was selected independently for each voxel.
Performance on the final held-out set was quantified with voxelwise coefficient of determination (R²) and Pearson correlation (corr).
Saved Fit Contents
Each .npy file contains a Python dictionary with the fitted encoding model and associated metadata:
subject — NSD subject number
model — visual model identifier
features_file_list — feature files used for model fitting
best_lambda_inds — selected ridge-penalty index for each voxel
r2 — held-out voxelwise coefficient of determination
corr — held-out voxelwise correlation
voxel_mask — subject voxel mask
voxel_index — voxel indices
voxel_nc — voxel noise-ceiling estimates
brain_nii_shape — original volumetric brain-image shape
Because the intercept was added as a final column in the feature matrix, the final row of weights contains the fitted intercept term.
Internal Model Labels
The released .npy files retain the model identifiers used by the original NSD feature-extraction and encoding-model analysis pipeline. These identifiers are analysis-time labels and should not be interpreted as model names stored within the underlying SimCLR checkpoints.
Public model name
Encoding-analysis identifier
Baseline
resnet18-Baseline
Fovea-Gaze
resnet18-FoveaGaze
Periph
resnet18-PeriphNonTTM
Periph-NF
resnet18-PeriphTTM
STL-10
resnet18-pretrained-simclr
ImageNet-100
resnet18-simclr-imgnet100
ImageNet-1K
resnet18-simclr-imgnet1k
The ImageNet-100 and ImageNet-1K PyTorch Lightning checkpoints themselves store model parameters under backbone.* and projection_head.* namespaces and do not contain the dataset-specific analysis identifiers shown above. The STL-10 checkpoint obtained from Spijkervet/SimCLR uses its original encoder.* and projector.* parameter naming, which was remapped during feature extraction.
The internal identifier resnet18-pretrained-simclr corresponds to the STL-10 SimCLR ResNet-18 checkpoint obtained from the external Spijkervet/SimCLR release.
These analysis identifiers are preserved in the model field of the released encoding-model fits and, where applicable, in fields such as model1, model2, and variance-partitioning dictionary keys. They reflect the naming conventions used during model development and do not indicate additional model conditions.
The features_file_list, features_file_list1, and features_file_list2 fields also preserve the original analysis paths for provenance. These paths refer to the computing environment used for the original study and are not expected to resolve outside that environment.
Variance-Partitioning Fits
The repository also contains the subject-specific variance-partitioning fits reported in the associated paper:
Fovea-Gaze vs. Periph — Figure 4C
Periph vs. Periph-NF — Figure 4D
For each comparison and NSD participant, three voxelwise encoding models were fit using (1) features from model A alone, (2) features from model B alone, and (3) the concatenated feature spaces from both models. Unique variance was computed from held-out voxelwise R² using the equations reported in the paper:
$$
R^2_{\mathrm{unique},A}
R^2_{\mathrm{combined}}
R^2_{B\text{-only}}
$$
$$
R^2_{\mathrm{unique},B}
R^2_{\mathrm{combined}}
R^2_{A\text{-only}}
$$
Thus, the unique variance attributed to one model is the variance explained by the combined representation minus the variance explained by the other model alone.
Each variance-partitioning file contains a Python dictionary with:
subject
model1
model2
features_file_list1
features_file_list2
lambdas
voxel_mask
voxel_index
voxel_nc
brain_nii_shape
weights_varpart
r2_varpart
corr_varpart
best_lambda_inds_varpart
The weights_varpart, r2_varpart, corr_varpart, and best_lambda_inds_varpart fields are dictionaries with entries for model A alone, model B alone, and the combined feature space. Their exact keys retain the original internal model identifiers listed above.
Because the six retained layers contribute 200 PCA components each, a single-model variance-partition fit contains 1,200 model features plus one intercept, giving weights_varpart matrices with 1,201 rows for the model-only fits. The combined fit contains 2,400 model features plus one intercept, giving 2,401 rows.
The repository contains 56 subject-specific encoding-model fits across seven pretrained visual models: 32 fits from the four VEDB-pretrained models (4 models × 8 NSD participants) and 24 fits from the three non-egocentric reference models (3 models × 8 NSD participants).
The repository additionally contains 16 variance-partitioning fits corresponding to the two variance-partition analyses reported in the paper (2 model comparisons × 8 participants), for a total of 72 fitted .npy artifacts.
The released files contain fitted voxelwise weights, held-out prediction metrics, ridge-penalty information, and voxel metadata used in the study. The variance-partitioning artifacts additionally contain the model-A-only, model-B-only, and combined fits used to compute unique variance.
Reproducing voxel predictions for new images also requires the corresponding pretrained ResNet-18 checkpoint and the feature-extraction, spatial-pooling, PCA, concatenation, and normalization procedures used during fitting. These steps will be documented in the accompanying public code release.
SimCLR ResNet-18 — STL-10(external pretrained reference model; checkpoint provided by Spijkervet/SimCLR and not redistributed by this project)
Computational Resources
The computational experiments and analyses for this study were conducted primarily using Carnegie Mellon University Neuroscience Institute's MiND computing cluster.
Citation
If you use these encoding-model fits in academic work, please cite the associated study:
bibtex
1@inproceedings{diaz2026eccentricity,
2 author = {Diaz, Dylan M. and Henderson, Margaret M.},
3 title = {Eccentricity-Constrained CNN Training Reveals Adaptive Information Coding Around the Visual Field},
4 booktitle = {Proceedings of the 9th Conference on Cognitive Computational Neuroscience},
5 address = {New York, NY, USA},
6 year = {2026},
7 doi = {10.32470/0416gfsq}
8}
Proceedings: Conference on Cognitive Computational Neuroscience 2026 Preprint:arXiv:2607.19316
NSD Citation
Researchers using the Natural Scenes Dataset should cite:
Allen, E. J., et al. (2022). A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25, 116–126. https://doi.org/10.1038/s41593-021-00962-x
Researchers using the underlying Visual Experience Dataset should also cite:
Greene, M. R., et al. (2024). The Visual Experience Dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video. Journal of Vision, 24(11), 6.
See the VEDB dataset paper for the complete author list and citation information.
License
The encoding-model fits in this repository are released under the Apache License 2.0.
The VEDB dataset and other third-party resources used in the associated study remain subject to their respective licenses, access requirements, and terms of use. This repository does not redistribute the full VEDB dataset; a small number of example frames are included to illustrate the published visual-field manipulations.
