Sensory-Grounded Affective Intelligence in Large Language Models

Supervisors: Dr Yunfei Long (https://www.seresearch.qmul.ac.uk/chcc/people/ylong/), Prof Massimo Poesio (https://www.seresearch.qmul.ac.uk/cmai/people/mpoesio/)

Project Description: How does a familiar voice evoke comfort, a crowded environment generate anxiety, or a sensory description communicate emotion without explicitly naming it? Human emotional understanding is closely connected to sensory perception, personal experience and social context. This PhD project will investigate how large language models (LLMs) can represent and reason about these connections, enabling AI systems to interpret affective meaning with greater sensitivity to how people perceive and experience the world.

Situated within the Cognitive Science Group at Queen Mary University of London and supported by Prof Massimo, the project will build on research in neuro-cognitive natural language processing, affective computing and culturally aware language models. It will bring together insights from embodied cognition and cognitive appraisal: sensory experiences acquire emotional significance through their relationship to an individual’s goals, expectations and prior experiences. The central research question is whether explicitly modelling this relationship can improve the accuracy, robustness and interpretability of LLM-based affective understanding.

The project will examine the distinction between recognising emotional expressions and understanding the sensory experiences that give rise to them. For example, the same sound may be reassuring to one person and distressing to another, while apparently positive language may convey discomfort when interpreted alongside vocal tone or contextual information. These cases motivate models that connect sensory evidence, contextual interpretation and emotional response, while recognising uncertainty and legitimate differences between individuals.

The research will focus firstly on language, visual scenes and everyday sounds, using existing multimodal models with targeted adaptation/tuning. Controlled experiments will investigate whether models use relevant sensory evidence, how their interpretations change when this evidence is altered, and whether they generalise across unfamiliar situations and participant groups. Expected outputs include evaluation resources, computational methods and an experimentally evaluated prototype, supporting future applications in accessible and emotionally responsive human–AI interaction.

This PhD research aims to develop and validate a sensory-grounded framework for affective language understanding with the following specific objectives:

Primary Objective: Develop and evaluate methods that enable LLMs to connect sensory evidence with contextual appraisal and emotional meaning, while accounting for individual variation and uncertainty.

Secondary Objectives:

1. Establish a sensory–affective evaluation framework: Develop tasks linking sensory descriptions, images and sounds with human emotional judgements, including implicit emotion and cases where different modalities provide conflicting evidence.
2. Develop sensory-grounded representations: Investigate methods for aligning language and sensory representations, testing whether explicit modelling of contextual appraisal improves emotion understanding beyond text-only and conventional multimodal baselines.
3. Model variation in affective interpretation: Examine how personal context and cultural background influence responses to the same sensory evidence, using participant-provided information and preserving meaningful disagreement in evaluation.
4. Investigate interpretability and robustness: Use controlled perturbations, counterfactual examples and representation analysis to assess which evidence drives model predictions, how models respond to missing or misleading cues, and whether their confidence reflects uncertainty.
5. Validate emotionally responsive interaction: Build and evaluate a prototype that uses sensory and conversational context to interpret emotional meaning and produce appropriate responses, assessing performance through computational experiments and ethically approved human studies.

Prerequisites:

• A Master’s degree at Distinction level or equivalent, or expected completion before starting the PhD, in computer science, artificial intelligence, cognitive science, computational linguistics or a closely related discipline.
• A strong interest in the relationship between language, sensory perception and human emotion.
• Solid programming skills including vibe coding skills, preferably in Python with Codex, and familiarity with machine learning including conventional machine learning and deep learning.
• An interest in building and evaluating working AI systems. Experience with NLP, multimodal learning or human-participant research would be beneficial but is not essential. Top conference publications would be welcome but is not essential.

Funding: The PhD studentship is funded by an EPSRC Doctoral Landscape Award and is open to applicants with Home and International fee status. The number of students with international fee status who can be recruited is capped under EPSRC terms and conditions, so competition for international places is particularly strong.

Awards are tenable for up to 3.5 years and cover tuition fees and a maintenance stipend at the applicable UKRI rate, including the London allowance. The indicative stipend for 2026/27 is £23,805 per annum for full-time study, with the 2027/28 rate to be confirmed.

How to apply:

Queen Mary University of London welcomes applications from candidates interested in developing the next generation of human-centred artificial intelligence.

Applicants are encouraged to discuss their research interests with the prospective supervisor before applying and should follow the instructions at:
https://www.qmul.ac.uk/eecs/phd/how-to-apply/.

The application should include:

• CV, maximum two pages.
• Cover letter, maximum 4,500 characters, explaining your suitability for the project and indicating your anticipated Home or International fee status.
• Statement of Research Interests, maximum 500 words, explaining your interest in the proposed research area and your relevant experience.
• Two references.
• Evidence of English language proficiency, where required.
• Academic transcripts, degree certificates and any other required supporting documents.

Fee status and studentship eligibility will be assessed by Queen Mary in accordance with the applicable requirements.

Application Deadline: 16:00 GMT, 27 January 2027, subject to confirmation for this studentship.

For specific enquiries, contact Dr Yunfei Long at qp241311@qmul.ac.uk.

For general enquiries, contact eecs-phd-enquiries@qmul.ac.uk, using the subject “EECS 2027 PhD scholarships enquiry”.