Multimodality: Exploring Sensibility and Sense-Making Beyond the Metaverse.

Immersive technologies are increasingly integrated into creative practice and various aspects of everyday life as augmented and virtual reality (AR/VR). These extended reality (XR) technologies are particularly relevant for researchers exploring human-computer relationships where users intend to interact with 3D objects and real-time multimedia in a spatial context. XR is being researched, developed, and deployed for communicating and building knowledge around what is being called the metaverse. The present trajectory for creating metaverse experiences strongly tends toward attaining heightened realism. Here we discuss the various factors of multimodality that challenge the idea that research in interaction for mixed realities should narrowly converge on or limit itself to developing applications intended to captivate users in these realist environments. We emphasize advancements in multimodality to further experimentation with human sensory modalities. To this end, we discuss the recent aesthetic theory proposed by Fuller and Weizman (2021), which is oriented to the attunement of the senses to further our capacities for sensing and sense-making. Building on Fuller and Weizman’s references to notions of memory, synaesthesia, kinaesthetics, and chronesthesia, we consider multimodality within mixed realities beyond realism to raise users’ sensibility and discernment toward both virtual and physical environments via new forms of experimentation with the senses.