Graduate students and postdocs are invited to join the speaker for lunch after the seminar. Space is limited – to sign up, email: pennmindcore@sas.upenn.edu

Leyla Isik
Department of Cognitive Science
Johns Hopkins University

Seeing social interactions

Humans see the world in rich social detail. We effortlessly recognize not only objects and people in our environment, but also social interactions between people. The ability to perceive and understand others’ interactions is critical to function in our social world, yet the underlying neural computations remain poorly understood. In this talk, I will first argue that social interaction perception should be studied with the same computational vision tools that are now widely applied to other areas of vision, like scene and object recognition. I will then present our recent research using a large-scale, naturalistic video dataset and condition-rich fMRI experiment, demonstrating that social interaction information is extracted hierarchically by the visual system along the recently proposed lateral visual pathway. In ongoing work with this same dataset, we find that, unlike static scene and object vision, current AI vision models do a poor job of matching human behavior and neural responses to dynamic, social scenes. To help close this gap, we developed a novel graph neural network model, SocialGNN, that instantiates insights from cognitive (neuro)science. SocialGNN reproduces human judgments of social interactions in both controlled and natural videos using only visual information, without any explicit model of agents’ minds or the physical world. Critically, the model’s relational, graph-based structure and processing are required for accurate social interaction recognition. Together, this research suggests that social interaction recognition is a core human ability that relies on structured visual representations, highlighting a path forward for better human-aligned AI.

A pizza lunch will be served. Please bring your own beverage.

We will also stream this seminar via Zoom.
For the link, please email: pennmindcore@sas.upenn.edu