Research Focus

We study how our goals, beliefs, and emotions influence our subjective interpretations of the physical and social world. A central question we explore is why people interpret and respond to identical situations in vastly different ways. The subjectivity in how we perceive and respond to the world is a key aspect of what makes us unique as individuals. After all, the study of the human mind would be much less interesting if we were all the same! Subjective interpretations can significantly impact our well-being and actions, explaining why some people thrive while others struggle in similar circumstances. Understanding these differences is critical in advancing our theories of the neural basis of human cognition in a way that accounts for individual variability.

In our research, we combine functional brain imaging, behavioral experiments and computational techniques to study how subjective interpretations are reflected in patterns of brain activity. There is also a recent interest in the lab in using artificial neural networks, particularly large language models and computer vision models, as cognitive models that allow us to probe the inner workings of the mind.

Below we highlight some of our major research threads:

I. Neuroscience of narrative understanding and memory

Narratives have a powerful hold over the human mind. People are more often convinced by a compelling story than by concrete facts. Furthermore, people use narratives to organize their thoughts and communicate ideas. Our earlier work explored the neuroscience of how narratives are remembered and communicated (Chen, Leong et al., 2017, Nat Neurosci; Zadbood et al., 2017, Cereb Cortex). For example, we found that viewing, listening to an audio description, and verbally recalling events from a movie elicited similar event-specific neural patterns that were common across participants. More recently, we have focused on studying how prior beliefs and affective states influence the interpretation and memory of narratives.

Effects of emotional arousal on brain dynamics and memory during narrative processing

The ability to remember our past in vivid detail is a significant aspect of human experience, allowing us to relive previous encounters and providing us with the store of memories that shape our identities. Emotional events are often remembered more vividly and exert a strong influence over our thoughts and actions. My research uses emotional narratives to investigate the neural processes underlying this phenomenon. In a recent study, we demonstrated that predictive models trained on whole-brain functional connectivity patterns reliably predicted moment-to-moment fluctuations in emotional arousal during movie-watching and generalized across multiple datasets, suggesting dynamic functional connectivity may encode a generalizable neural representation of emotional arousal during narrative perception (Ke et al., 2024, bioRxiv).

To further unpack this neural representation, a follow-up study applied graph theoretic approaches to the fMRI data and showed that heightened emotional arousal was associated with enhanced functional integration (i.e., increased connectivity and cohesion) across large-scale brain networks, suggesting that emotional arousal drives the brain into an efficient state conducive to inter-network communication. This network integration, in turn, predicted more accurate and detailed memory of the corresponding moments in the movie, consistent with the hypothesis that functional network integration facilitates the stronger encoding of emotionally arousing events (Park et al., in prep). Together, these studies build towards a theoretical framework that connects affective states, brain dynamics, and ongoing cognition.

Hostile attribution bias shapes subjective interpretations of ambiguous social narratives

A predisposition to interpret ambiguous social situations as intentionally hostile, often referred to as hostile attribution bias, predicts interpersonal conflict and aggression. We we used fNIRS to measure neural activity while individuals listened to nuanced real-world social situations (e.g., a professor forgetting to submit a letter of recommendation; Lyu, Su et al., 2024, J Neurosci). Neural activity was synchronized among individuals with similar levels of hostile attribution bias, suggesting shared interpretations of the scenarios. This effect was localized to the left ventromedial prefrontal cortex, a brain area implicated in decision-making, regulating emotions, and social evaluation, and was particularly prominent in scenarios where the character’s intentions were highly ambiguous. Applying machine learning techniques to the neural data, we were able to distinguish between individuals with high and low hostile attribution bias with 86% accuracy. Hostile attribution bias was stronger among individuals who scored lower on attributional complexity, a measure of one’s tendency to consider multifaceted causes when explaining behavior. Our findings reveal how subjective interpretations of social situations are reflected in the temporal dynamics of brain activity and highlight the potential of using fNIRS to develop non-intrusive and cost-effective neural markers of socio-cognitive biases. In ongoing work, 

Predicting whole-brain neural dynamics from prefrontal fNIRS during movie-watching

Relative to other neuroimaging methods like fMRI, fNIRS offers several advantages, including portability, lower cost, and fewer constraints on participant movement, making it especially suitable for studies involving naturalistic social interactions. One limitation of fNIRS, however, is that it is limited to measuring activity near the scalp and does not have access to deeper areas of the brain. We recently developed a method for predicting whole-brain activity from fNIRS (Gao et al., in prep). Using a shared movie stimulus align the fNIRS and fMRI data, we “reconstructed” whole-brain activity from prefrontal fNIRS data. The model generalized to new participants watching a different segment of the movie, even in areas of the brain not accessible by fNIRS. We further showed that the reconstructed brain activity retained information about the semantic content of the movie. This work introduces a promising approach for combining the flexibility of fNIRS with the spatial coverage of fMRI, opening new possibilities for studying the brain in naturalistic contexts.

II. Affective influences on visual perception and decision-making

Emotion and affect guide our attention, which in turn shapes our perception and decisions. However, attention is an internal and often unobservable process, making it challenging to study directly. Our work utilizes fMRI to measure attentional processes, allowing us to observe how affective states influence where and how individuals direct their attention and the consequences this has for decision-making. Additionally, we use computational models to decompose affective biases in cognition into separable cognitive processes, which can then be mapped onto the brain to identify the specific computational role performed by different brain regions.

Neurocomputational mechanisms underlying motivated visual perception

Individuals with different motivations often report seeing the same image differently, but it is often disputed whether this reflects a bias in what they see or what they report seeing. Can people with diverse goals and motives truly perceive the same visual stimuli differently? We reexamined this decades-old debate in a series of studies. Using fMRI, we show that motivation not only biased perceptual judgments, but also enhanced the neural representation of desirable visual stimuli, suggesting that participants were indeed perceiving them differently (Leong et al., 2019, Nat Hum Behav). Analyses using drift diffusion models decomposed participants’ motivational bias into response and perceptual components, with the former associated with activity in the ventral striatum and the latter associated with the enhancement of activity in the visual cortex. Follow-up work found that the perceptual bias tracked with heightened amygdala activity (Calabro, Lyu & Leong, 2023, Cereb Cortex), as well as trial-by-trial fluctuations in physiological arousal, as measured by pupil dilation (Leong et al., 2021, Psychol Sci). In a collaboration with Phil Corlett’s lab, we also found that individuals who score high on trait paranoia were more strongly biased by wanting to see an outcome, suggesting that motivational biases in information processing might play a role in paranoid thought patterns (Rossi-Goldthorpe et al., 2021, PLoS Comput Biol). Together, this line of work provides a neurocomputational account of how motivation biases perception and investigates the consequences this has for adaptive behavior.

Dynamic interaction between reinforcement learning and visual attention

Attention determines the information that enters our awareness, shaping our perceptions based on our focus. How does the brain select what to attend to and what to ignore? One hypothesis is that people learn to attend to aspects of the environment relevant for predicting rewards. To test this, I developed a novel measure of attention that combines eye-tracking, multivariate decoding of fMRI data and behavioral modeling (Leong, Radulescu et al., 2017, Neuron; Niv et al., 2015, J Neurosci). Using this measure, I found behavioral and neural evidence that participants learned to attend to task dimensions that predict reward via mechanisms of reinforcement learning. In turn, attention constrains learning and value signals in the brain to those dimensions. The ventromedial prefrontal cortex, a region associated with the encoding of subjective value, exhibited enhanced connectivity with frontoparietal control regions when participants sustained attention to the same task dimensions over multiple trials, highlighting a potential mechanism underlying value-driven modulation of attention. This work sheds light on how reinforcement learning mechanisms help individuals focus on reward-predictive aspects of their environment to guide adaptive decision-making. The results also suggest that individuals with different past experiences of rewards and punishments are likely to attend to different aspects of their environment, which can give rise to divergent perceptions and decisions.

Computational models of intuitive physical reasoning

When reasoning about the physical world around them, people can rapidly infer the properties of different objects and the relationships between them. However, this becomes more challenging as the relationships between objects grow more complex or ambiguous. People are prone to making false inferences about physical scenarios and are influenced by perceptual biases, as exemplified by real-world optical illusions. Using eye-tracking, we explore how visual attention and motivational biases influence people’s intuitive judgments about physical scenes. We employ convolutional neural networks (CNNs) to develop tools that allow us to better approximate, predict, and understand biases in human intuitive physical judgments.

III. Understanding divergent interpretations of political information  

People with different political beliefs often have starkly different reactions to the same political event. For example, protests in support of defunding the police are likely to anger conservatives but not liberals. These divergent responses to political events threaten to exacerbate political polarization and social tensions. Why do conservatives and liberals respond differently to the same event? One reason is that the same actions and words can have different “meanings” to the two groups. For instance, the phrase “defunding the police” carries different connotations for conservatives and liberals. How exactly these connotations differ, however, is not always evident. Our research examines why and how the same political information is interpreted differently by people with different political beliefs, using a combination of functional neuroimaging and natural language processing methods.

Divergent neural and semantic representations of political information

We used fMRI and semantic analyses of real-world political content to study the psychological and neural processes that drive partisan biases in how people interpret political messages (Leong et al., 2020, PNAS). Neural activity diverged between conservative and liberal participants while watching videos about immigration policy, with greater divergence during moments with moral-emotional and threat-related language. One explanation for these results is that conservatives and liberals differ on what they consider moral and threatening. To test this possibility, we used word embedding models to analyze two large datasets comprising ~300 million Reddit comments in partisan communities and ~7 million articles from partisan news outlets (Rim et al., 2023, arXiv). These models allow us to capture the meaning of words from conservative and liberal perspectives. We then probed the semantic associations of words related to seven political topics (e.g., abortion, immigration) along the dimensions of morality, threat, and valence. Across both Reddit communities and news outlets, we identified a systematic divergence in the moral associations of words between text sources with different partisan leanings, consistent with the idea that the moral meaning of words diverged between conservatives and liberals.

We recently collected new fMRI data from conservatives and liberals watching videos across multiple issues, specifically videos where issues are being debated. The goal is to combine fMRI with cutting-edge language models to better understand the semantic content that drives differences in responses to political messages. By doing so, we aim to build more sophisticated language models that can capture and predict the differences in interpretations between conservatives and liberals, ultimately contributing to a deeper understanding of the neural and cognitive mechanisms underlying political polarization.

Characterizing political language on social and news media

We developed a new computational approach using word embeddings trained on Reddit comments and news articles to measure affective alignment and polarization between conservatives and liberals (Rim et al., under review). Our analyses revealed that affective associations toward political issues (e.g., abortion, immigration) were more aligned across party lines than those toward partisan identity (e.g., Democrat, Republican). We found that Reddit communities exhibited affective polarization on both partisan identity and political issues, with a stronger effect for partisan identities, while news sources showed polarization around partisan identities but not political issues. These findings suggest that social media discussions are more polarized than news reporting, supporting theories that social media amplifies political divisions. This work our advances understanding of affective polarization by providing precise tools to measure identity-based and issue-based polarization, indicating that partisan identities are more divisive than specific political issues. This highlights the need for strategies that bridge identity divides and emphasize issue-based alignment to foster constructive political discourse.

IV. Neurocomputational mechanisms of social learning and decision-making

In addition to our work studying how motivation affects how people encode their physical environments, we have also studied how motivation biases how people encode their social environments. Collectively, these studies have identified the patterns of neural activity encoding motivationally relevant social information and how these representations modulate ongoing social learning and decision-making.

Biases in person perception and social decision-making

To build supportive social connections, individuals must identify partners likely to reciprocate their interest. A kind and intelligent acquaintance offers little social value if they reject our attempts to engage with them, while a caring friend provides limited support if their care is underestimated and they are subsequently avoided. Our research uses fMRI and computational models to investigate how people learn who to interact with and who to avoid. In a study involving students from two undergraduate dormitories, we found that multivariate patterns in the prefrontal cortex differentiated between passively viewing photos of socially valued individuals (i.e., trustworthy, empathetic, and socially supportive) in one’s real-world social network. This suggests that people spontaneously and accurately monitor social value, potentially to guide subsequent social interactions (Morelli, Leong et al., 2018, PNAS). However, when expectations of social value are misinformed, such as thinking someone is trustworthy when they are not, these beliefs can be resistant to change due to confirmation bias (Leong & Zaki, 2017, J Exp Psychol Gen).