Your head is there to move you around:
Goal-driven models of the primate dorsal pathway

Patrick J Mineault, Shahab Bakhtiari, Blake A Richards, Christopher C Pack

McGill University, MILA

NeurIPS 2021 (spotlight presentation, <3% selected)

Code | Preprint

Abstract

Neurons in the dorsal visual pathway of the mammalian brain are selective for motion stimuli, with the complexity of stimulus representations increasing along the hierarchy. This progression is similar to that of the ventral visual pathway, which is well characterized by artificial neural networks (ANNs) optimized for object recognition. In contrast, there are no image-computable models of the dorsal stream with comparable explanatory power. We hypothesized that the properties of dorsal stream neurons could be explained by a simple learning objective: the need for an organism to orient itself during self-motion. To test this hypothesis, we trained a 3D ResNet to predict an agent's self-motion parameters from visual stimuli in a simulated environment. We found that the responses in this network accounted well for the selectivity of neurons in a large database of single-neuron recordings from the dorsal visual stream of non-human primates. In contrast, ANNs trained on an action recognition dataset through supervised or self-supervised learning could not explain responses in the dorsal stream, despite also being trained on naturalistic videos with moving objects. These results demonstrate that an ecologically relevant cost function can account for dorsal stream properties in the primate brain.

Training sequences

Here are some example training sequences we used. They were generated in Microsoft Airsim.

Prefered stimuli

Our network, called DorsalNet, learns motion representations similar to V1, MT and MST of the visual cortex. By contrast, a control network trained on Kinetics400, r3d_18, does not learn motion-like representations.

Select a layer to visualize prefered stimuli for selected layers of these networks.

Hover over an animation to freeze the other ones.