NEvo is a self-contained Hugging Face custom Diffusers pipeline for neural-response-guided visual stimulus synthesis. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.
It orchestrates three frozen models. The models below are placeholders / defaults and can be swapped for any compatible models (weights are not bundled — they are pulled from their own repos):
Each clip is from the top results of a NEvo search targeting one visual region — the model discovers, from scratch, stimuli that drive that region's known selectivity.
Off-the-shelf — no install. Load NEvo as a custom Diffusers pipeline; the package and its bundled data are fetched from the Hub automatically (you only need the usual dependencies below):
Or install the package (for cleaner from stimulus_synthesis import ... imports / development):
bash
1conda create -n nevo python=3.10 -y
2conda activate nevo
3pip install"git+https://huggingface.co/epfl-neuroai/NEvo"4# or from a local clone:5# git clone https://huggingface.co/epfl-neuroai/NEvo && pip install ./NEvo6# then: from stimulus_synthesis import NevoPipeline; pipe = NevoPipeline.from_pretrained("epfl-neuroai/NEvo")
Runtime dependencies (either way): torch, diffusers, transformers, huggingface_hub, numpy, pillow, av (pytest for tests). No nilearn / atlas downloads — ROI masks are shipped as small precomputed data files.
Quickstart
Target a brain region by name — NEvo resolves its voxels and searches for a video predicted to drive it:
python
1from diffusers import DiffusionPipeline
23# fetches the pipeline (and package) from the Hub; model weights are pulled on first use4pipe = DiffusionPipeline.from_pretrained(5"epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,6)78out = pipe(roi="FFA", progress=True)# omit seed (default) -> different result each run; pass seed=<int> to reproduce (the seed used is in out.metadata["seed"])9print(out.best_prompt, out.best_score)1011from stimulus_synthesis.media import save_video, video_to_t_c_h_w # importable once the pipeline has loaded12save_video(video_to_t_c_h_w(out.best.video),"best_stimulus.mp4", fps=12)# save the synthesized video (mp4 playback fps; default 12)13out.best.image.save("best_stimulus.png")# and the stage-1 best image (PIL)
This runs the two-stage search with the defaults — up to 400 image evaluations then 200 video evaluations (population 20), using the fast distilled-model defaults (1-step 512×512 SDXL-Turbo, 8-step 512×512 LTX). A run takes a few minutes and a good amount of GPU memory.
Faster run
For a quicker first result, shrink the search and the video:
python
1out = pipe(2 roi="FFA",3 progress=True,4 image_max_evals=80,# stage-1 (image) evaluation budget (default: 400)5 video_max_evals=40,# stage-2 (video) evaluation budget (default: 200)6 population_size=8,# GA population per generation (default: 20)7 image_batch_size=8,# images generated per batch (default: 16)8 video_batch_size=4,# videos generated per batch (default: 8)9 seed=0,# RNG seed, for reproducibility10 video_kwargs={# merged over the fast defaults (8 steps / 25 frames / 512²); override any key11"num_inference_steps":8,# denoising steps — the distilled LTX model needs only a few12"num_frames":25,# clip length; LTX requires 8*k + 1 frames13"height":256,"width":256,14},15)
Enhanced search space. Selecting a region (roi=...) restricts the prompt search to the categories relevant to that region — a smaller space that converges faster. Pass enforce_general_search_space=True to search the full general space instead.
Available ROI tokens (comma-separated tokens are unioned):
fsaverage5 only. The bundled ROI/searchlight masks are defined on the fsaverage5 cortical surface (20 484 vertices). Targeting a region by name therefore requires an encoder whose predict_fmri output lives in that same space — the default epfl-neuroai/vjepa2-encoder-basic. A custom encoder with a different output space can still be driven with explicit vector/indices targets, but not with the named-ROI helper.
Custom targets
Instead of a region name, pass raw voxel indices or a full target fMRI vector:
fast LTX settings (merged under call-time video_kwargs)
Each stage runs a genetic search with population population_size (default 20) until it hits its evaluation budget — image_max_evals (default 400) and video_max_evals (default 200) generate→score passes. Image and video generation use fast distilled defaults out of the box (default_image_kwargs / default_video_kwargs); anything you pass as image_kwargs / video_kwargs is merged over them, so you only override the keys you care about.
Robust scoring
By default each candidate is scored with a single clean encoder pass. An optional robust mode — the mean over 4 augmented draws (random crop 0.8, Gaussian σ=0.1) via RobustTransformScorer — reduces sensitivity to encoder artifacts; turn it on by setting "enabled": true in default_score_transform.
Cache configuration
Model weights and outputs cache location resolves in priority order:
NEvo_CACHE_DIR — set it in a repo-root .env file (see .env.example) or the environment.
Otherwise the system/user-default HuggingFace cache (HF_HOME, else ~/.cache/huggingface) is used and left untouched.
Only if no default is resolvable, a repo-local cache/ is used.
cache/ and .env are git-ignored.
Batch runners
Two ROI-driven, two-stage (image-search → video-search) runners are included:
run_regional_asset_pilot.py — same search but exports every candidate to a deterministically-encoded file (PNG/MP4), hashes it (sha256), and scores the decoded file — producing provenance-tracked, reproducible published assets with manifests.
Both take --rois, --seeds, --image-evals / --video-evals, --encoder-model, --out-dir, etc., and default to the config's encoder and a cache-relative output directory.
Reproducibility
The pipeline is deterministic for a fixed seed/config: the shipped ROI masks reproduce the original atlas masks bit-for-bit, and a fixed-seed run reproduces prior scores exactly. Encoder scores are a target-matching signal, not ground-truth reconstruction quality.
Intended use & limitations
Research use in visual neuroscience / brain-decoding. Outputs are predicted to drive a target region under a specific encoder — they are hypotheses to validate, not ground truth.
Optimizing hard against a single encoder can exploit encoder artifacts; inspect images and use held-out validation.
Requires a CUDA GPU with enough memory for the 13B video model; you must accept the license/access terms of the referenced upstream models.
Citation
If you use NEvo, please cite:
bibtex
1@article{tang2026nevo,
2 title={NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity},
3 author={Tang, Yingtian and Salehi, Sogand and Zhou, Ming and Zamir, Amir and Isik, Leyla and Schrimpf, Martin},
4 journal={arXiv preprint arXiv:2607.02317},
5 year={2026}
6}
Builds on BrainDiVE-style encoder-guided synthesis, vJEPA-2, SDXL-Turbo, and LTX-Video. ROI/searchlight definitions derive from an fsaverage-space group atlas (precomputed and bundled).
Intended Use, Out-of-Scope Use & Risks
NEvo is a research-only tool for computational visual neuroscience: it searches for stimuli predicted to drive a target brain region under a fixed image/video→fMRI encoder, and its outputs are unvalidated hypotheses, not scientific findings. It is not a medical device and must not be used for diagnosis, treatment, or to design, guide, or deliver any real brain stimulation (TMS, tDCS/tACS, DBS, intracranial, etc.) or in-vivo neuromodulation — it only optimizes a predicted fMRI response, not effects on a living brain, and any use involving human or animal subjects requires independent ethical review (e.g. IRB). Optimizing against a single encoder can exploit artifacts, and the generative models may produce biased or disturbing imagery; the model is provided as is, without warranty, for non-commercial use under CC BY-NC 4.0 (upstream models carry their own licenses).
