Skip to content
Preprint · CC-BY via bioRxiv

One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment

Sartzetaki, C., Roig, G., Snoek, C. G. M., Groen, I. I. A.

biorxiv · 2025

Abstract

What can we learn from comparing video models to human brains, arguably the most efficient and effective video processing systems in existence? Our work takes a step towards answering this question by performing the first large-scale benchmarking of deep video models on representational alignment to the human brain, using publicly available models and a recently released video brain imaging (fMRI) dataset. We disentangle four factors of variation in the models (temporal modeling, classification task, architecture, and training dataset) that affect alignment to the brain, which we measure by conducting Representational Similarity Analysis across multiple brain regions and model layers. We show that temporal modeling is key for alignment to brain regions involved in early visual processing, while a relevant classification task is key for alignment to higher-level regions. Moreover, we identify clear differences between the brain scoring patterns across layers of CNNs and Transformers, and reveal how training dataset biases transfer to alignment with functionally selective brain areas. Additionally, we uncover a negative correlation of computational complexity to brain alignment. Measuring a total of 99 neural networks and 10 human brains watching videos, we aim to forge a path that widens our understanding of temporal and semantic video representations in brains and machines, ideally leading towards more efficient video models and more mechanistic explanations of processing in the human brain.

◌ CITATION ONLY
Full text is not openly licensed for redistribution here. Read it at the source:

Read at source →

Provenance

Source
bioRxiv
DOI
10.1101/2024.12.05.626975
Canonical
link ↗
Fetched
2026-10-10 MST

Cite this

APA
C., S., G., R., M., S.C.G., &amp; A., G.I.I. (2025). One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment. <em>biorxiv</em>. https://doi.org/10.1101/2024.12.05.626975
Vancouver
C. S, G. R, M. SCG, A. GII. One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment. biorxiv. 2025. doi:10.1101/2024.12.05.626975.
BibTeX
@unpublished{sartzetaki2025OneHun, title = {One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment}, author = {Sartzetaki, C. and Roig, G. and Snoek, C. G. M. and Groen, I. I. A.}, journal = {biorxiv}, year = {2025}, doi = {10.1101/2024.12.05.626975}, }

Research neighborhood

References, citing works, and semantically nearest findings. Click a node to open it.

Related findings