A quality of experience evaluation of an immersive multimedia speech & language application

Abstract

Recent advances in technology have supported multimedia experiences to become more interactive and immersive. Traditional multimedia devices aimed to capture user attention through content rich visuals presented on two-dimensional displays. However, in recent times, emerging head mounted display (HMD) technologies such as Virtual reality (VR) and augmented reality (AR) HMDs aim to captivate users through the delivery of 360° visuals, immersive audio, and environments within which the user can interact. In this research, an immersive multimedia speech and language disorder assessment application was developed and novel user perceptual quality evaluations across three different platforms were undertaken. The speech and language disorder assessment application gave context to this research and directed it in terms of how to design an ecologically valid health application. The user perceptual quality evaluations were carried out on tablet, AR, and VR platforms under the auspices of the quality of experience (QoE) framework. In this context, the novel contribution of the PhD work presented in this thesis reflects efforts to design, develop, and understand user QoE of an immersive multimedia speech and language disorder assessment application. The research involved a comprehensive and rigorous comparison of three different platforms (AR, Tablet, and VR) by exploring the use of: explicit (subjective ratings); implicit (e.g. physiological, and psychophysiological); and objective measures of user performance and interaction. The comparison required a novel QoE assessment methodology and evaluation which facilitated not only comparison between the different platforms but also analysis of various captured modes of user responses (objective and implicit) for each platform. The findings from this first multiplatform QoE evaluation led this PhD work towards the need to understand physiological measures at a deeper level. More specifically, the next novel study reports the efforts undertaken to understand task-evoked physiological response within the immersive virtual speech and language disorder assessment application in VR. A correlation is discovered between implicit measures of Electrodermal Activity (EDA) and pupillary response. This finding is reported within the context of data derived from the monitoring of objective (interaction) metrics throughout the immersive experience. Finally, whilst a holistic approach to understanding user QoE is crucial, it also brings challenges with respect to the amount of data and different modes of explicit, implicit, and objective data captured. Processing this data through traditional techniques is very time consuming and challenging. In this context, the work places an emphasis on the automatic data processing and classification of user emotional states during an experiment as an insightful measure of QoE.