Affect Detection in Consumer Neuroscience: a Multimodal Machine Learning Model based on bioelectrical and biometric signals

Emotions are integral to human cognition and behaviour, shaping attention, decision-making, and social interaction. Within the marketing context, they act as antecedents, mediators and moderators of consumption-related processes, influencing the perception and evaluation of products, brands, and experiences. Traditional approaches to emotion measurement, such as self-reports and behavioural observations, are affected by well-known biases and direct measurements based on physiological signals have been proposed as a more reliable alternative. Consumer Neuroscience attempt to estimate marketing-related cognitive and emotional states from physiological signals, typically relying on large-scale correlational studies and reverse inference reasoning. However, this approach has been shown problematic, particularly when the physiological measures exhibit low specificity. Affect detection techniques, which apply machine learning models to features extracted from multiple modalities, represent a promising alternative. Nevertheless, such methods remain still under-represented. This work focuses on the design of a multimodal Affect Detection model based on bioelectrical and biometric signals. The thesis begins with a theoretical part briefly (albeit carefully) reviewing the main theories of emotion, the approaches of Consumer Neuroscience and Neuromarketing, the inference in psychophysiology and the principles of Machine Learning. The experimental section presents I DARE (IULM Dataset for Affective Response), a standardised multimodal dataset that outperforms existing ones in terms of dimension and minimum detectable effect size. It then describes the training, testing and evaluation of a subject-independent affect detection model, developed using Bayesian optimisation for model selection and hyperparameter tuning, and robust error estimation through cross-validation and concentration inequalities.