When we read a book we immerse ourselves in imaginary spaces thanks to the details narrated, when we watch a film we become involved in the plot thanks to the audiovisual effects, when we listen to music we connect with the rhythms and enter into different states of mind and so on, each person lives a different and immersive experience regardless of the work with which he or she interacts. In these cases the artists (be it writer, producer or musician) materialise their ideas with details that allow us to empathise with the visual and/or sound element as the work unfolds, but what about us designers when we want to connect our users with a physical/virtual service? How do we manage to involve them in an immersive experience through what we design?
A few days ago I was watching Christopher Nolan’s film Interstellar, and when I finished it I asked myself why I can fully connect with a particular film, just by looking at a screen? My answer was not just one, but the agglomeration of several factors. It’s not just the story that strikes me the most, nor the personality of the characters that comes through in their dialogue, it’s the details that were thought out by the director and producer with the purpose of connecting the audience to their film. It is the soundtrack, it is the melody or song that plays at the right moment, it is the costumes that match the colour palette of the space, it is the smells and textures that I imagine when the camera takes certain approaches and, in conclusion, what these elements have in common is that they awaken my senses to associate what I am seeing and hearing with experiences I have already lived; that is when fiction no longer seems so distant from my reality and, on the contrary, I begin to feel part of the story.
If we translate the above to our life as UX/UI designers we could say that, just as a film director tries to include many details in his productions to involve the audience, we as designers must strive to create touch points in our services, with outstanding details compared to the conventional, and when I say outstanding I mean the fact of overflowing the screens and joining the virtual with the physical. In this way, the qualities of the digital world can be enjoyed and enhanced with our senses (which belong to the physical world), and all of this is possible to achieve through multi-experiences.
As in films, multiexperiences also tell a story, the story of making a virtual purchase, the story of getting to know a new product on the market, etc., but the fascinating thing is that, through this medium, one can access scenarios, situations and even non-existent dimensions. It sounds like fiction, but it is not. Multi-experiences, according to Paulette Saldaña, innovation leader at Stefanini Group, “is the way people perceive and live the digital world, through a wide offer of devices and touch points, such as virtual reality (VR), augmented reality (AR) and mixed reality (MR)” (Saldaña, 2019).
Entering the three realities
Let us understand VR, AR and MR as the special effects of a movie, because through them a set of elements or realities that do not exist are conveyed. The interesting thing about multiexperiences is that in order to interact with this physical/virtual universe, the intervention of the body is needed to activate functions, for example: touch, voice commands, facial gestures, body movements, among others; it is like experiencing an interactive movie first hand. However, to better understand each reality, what better way to do so than through its applicability in today’s markets?
Virtual Reality (VR)
If you have ever seen Google Cardboard Glasses, you can already associate the term and imagine what we mean when we talk about a VR experience. However, this does not mean that the glasses are the only means by which you can have one of these experiences, as VR seeks to simulate a space by creating a digital environment and for this, only a device that generates visual, auditory and kinesthetic stimuli is required.
In the case of Kellogg’s Company, they wanted to determine the ideal location and assortment of their products in stores and supermarkets. To do this, they partnered with Accenture Extender Reality (XR) and developed a virtual reality solution, using a helmet with integrated eye-tracking that would allow them to read the visual path consumers take on supermarket shelves (Accenture, 2019).
Augmented Reality (AR)
Unlike VR, AR does not aim to simulate a virtual space in order to immerse oneself in an experience, as AR involves virtual elements within real environments.
Nike has used augmented reality to help consumers find the right shoe size, depending on the desired model. To do this, users must use the Nike Fit feature in the Nike app and measure their feet with their smartphone camera to get guidance on their size (Nike, 2019).
Augmented Reality (AR)
Unlike VR, AR does not aim to simulate a virtual space in order to immerse oneself in an experience, as AR involves virtual elements within real environments.
Nike has used augmented reality to help consumers find the right shoe size, depending on the desired model. To do this, users must use the Nike Fit feature in the Nike app and measure their feet with their smartphone camera to get guidance on their size (Nike, 2019).
Mixed Reality (MR)
Combining VR and AR, mixed reality seeks to combine the real environment with virtually designed content. With this technology, the real physical world coexists with digital elements and unlike VR and AR, the virtual elements interact directly with the real environment.
In October 2020, Nintendo launched its new MR game, Mario Kart Live: Home, which allows players to race remote-controlled vehicles around their homes. Interacting with both virtual and real-world elements (Chiu,2021).
On trend
According to Chiu (2021) in the report “The Future 100. The trends and change to watch in 2021”, the union between the physical and virtual world becomes an attractive and enhanced version of the usual experiences, becoming a norm in the market (pp.40,41). In fact, the number of patents in AR and VR is growing strongly, with seed funding growing by 237% between 2014 and 2016 (not to mention the growth between 2017 and today).
The graph represents VR and AR growth in patents. Taken from A responsible future for immersive technologies, Accenture, 2019.
VR, AR and MRI is clearly not new, but the pandemic provided a push that drove markets to evolve their new experiences, products and services. In retail brands for example, multi-experiences are signalling new growth, as many markets are now breathing new life into e-commerce with sophisticated immersive spaces, which has been accelerating innovation and competition in retail.
One such example is Random Studio, which is currently creating multi-sensory experiences for retail brands. They propose that the encounter between users and brands should be immersive and adaptable to the shop space, transforming it into a dynamic landscape for customers to understand the creative vision of emerging collections. How does it work? A dynamic image is projected onto a screen and each time the person changes clothes, the image of the landscape changes according to the climate and environment that corresponds to the particular garment.
The above is just one of the few examples that give us to understand that nowadays multiexperiences are becoming part of people’s everyday lives, especially at a time when, thanks to the pandemic, people have a greater interest in consuming new interactions and creative content. However, there are other changes in people’s behaviours that technology is triggering and it is worth keeping in mind to continue to respond to the transformations in everyday life. The New York Times called 2020 “The year of Blur”, referring to a year in which monotony and isolation have warped the notion of time and in addition to that the parallel life between virtual and digital is difficult to cope with 24/7. According to David Byrne in his article “Eliminating The Human”: “The more time we spend in the virtual, the greater the risk that we will lose interest and the ability to be present in the real world” (Byrne, 2017).
In this sense, today more than ever it is worthwhile to continue rescuing the idea of preserving physical experiences where enjoyment is achieved through the connection between our senses and the material world. In fact, why should we continue to limit physical contact with the environment now, when it is the time when we most yearn for it? And this does not mean moving away from technology and crowd encounters, but incorporating responsible practices in immersive technologies to guarantee individuals trust and genuinely pleasurable experiences.
“To design deeply moving experiences is to weave dialogs between our bodily senses and the digital world behind the screen” (Marino A. 2020).
We need to create technologies that transmit more than just information, as screens have over time become synonymous with distance and privacy. To go no further, when we walk alone in the street or on public transport, our first reaction is to sit down and look at our mobile phones, and what we transmit to others is a sign of isolation and personal space. However, as Marino (2020) mentions in his article: How can screens be more tactile? examples such as Alexa, Siri and Google Assistant support the growing sense that meaningful interactions can happen through the interplay of our senses, and not just visually but through other means such as voice, hearing, touch and even taste.
At Random Studios they designed a series of experiments, creating digital content with textures and movements, in order to generate physical sensations. In the Future Spring project, a digital representation of a composition of flowers representing autumn was made. The three-dimensionality of the graphics invites the user to interact with the screen and the desire to touch what can be seen is satisfied by controlling the graphics with body gestures.
In conclusion, brands and big companies are undoubtedly seeing the potential of immersive technology and as a consequence, it is becoming a norm in innovation, but this is not the real competition in the technological market, as our real projection must be to create experiences without forgetting to respond to the desires of our nature as human beings. Creating immersive technological environments is not about replicating the physical world, as it is irreplaceable, but about creating complements to what we are naturally used to living in our daily lives, even before technology was so latent in our lives.
The future of the digital lies in the approximation with our existence: the material, and to achieve this it is necessary to draw on multiple senses in order to reconcile ourselves with the experiences of closeness and of being rooted in our bodies, emotions and our shared realities.
