This thesis presents a speech emotion recognition model, integrating it into a digital therapeutic tool for enhancing emotion recognition therapy in individuals with autism spectrum disorder. Motivated by the need for interactive and software-based therapeutic tools in the field of ASD therapy, this research endeavours to bridge the gap between new technological innovation and therapeutic application.

The SER model uniquely combines a CNN spectral model and a linear regression prosodic model for accurate emotional arousal prediction from speech, validated using the IEMOCAP and MSP-IMPROV datasets. The mean ensembled model, merging spectral and prosodic features, showcased superior accuracies in key datasets: achieving a MAE of 0.52, PCC of 0.67, and CCC of 0.66 on the IEMOCAP Test Set, and a MAE of 0.44, PCC of 0.67, and CCC of 0.66 on the MSP-IMPROV Test Set, surpassing individual CNN and LR models. This underscores the ensemble’s robustness and its usefulness in ASD emotion recognition therapy.

The project further developed a user-interactive digital therapeutic system, incorporating Soul Machines for the digital avatar and Google Dialogflow for natural language processing, operating on a local server to adapt conversational dynamics based on emotion predictions from the SER model. A GUI was also created to display emotion estimates in real-time and to provide flexibility for integrating additional emotion estimation models. This work sets the stage for real-time testing and the incorporation of ASD-specific therapy modules as areas for future enhancement.

This research lays the groundwork for a comprehensive, technology-enabled therapeutic system for ASD, demonstrating the effectiveness of combining SER models with digital tools. It promises significant advancements in creating accessible, effective, and personalised therapeutic interventions, potentially enriching ASD individuals’ lives through cutting-edge technology.