Introduction

This text discusses the relationship between scripting, virtual space design, and interactive design, and the audience’s movements and expected position in room-scale Virtual Reality, based on the experience of designing, developing and exhibiting two VR interactive works. Through the analysis of these works, it looks at how different approaches allow or impede to anticipate the specific position of the audience at any given time. This anticipation is key to articulate the virtual elements in a way that responds to not just the gaze but also the overall movement of the person wearing the headset.

In Pieces VR is an artwork created in 2018 as a response to the imprisonment of the Catalan politicians and activists that led the 2017 attempt at independence. It documents the personal impact of political prison on those affected and their immediate family. The artwork premiered at London’s Gazelli Art House in October 2018. During the following year, it was exhibited at New Yorks’ Jump Into the Light VR cinema, Berlin’s House of Democracy and Human Rights, and the Catalan College of Music, in Barcelona.

The Smallest of Worlds was developed during the 2020-21 edition of the CPH:LAB, the development lab of CPH:DOX Copenhagen. The piece started as a response to the 2020 global lockdown, and became affected by it throughout the production, to the point that its exhibition form mutated and evolved from a VR piece into a transmedia project that included also a web version and a physical installation. It was part of the official selection of DOK:Lepizig’s DOK:Neuland 2020, INTER:ACTIVE 2021 (CPH:DOX), Les Ailleurs (Paris, 2021) and Munich’s VRHAM 2021. The Smallest of Worlds is coauthored by Uwe Brunner, Bettina Katja Lange and myself.

VR Experimental Documentary

I have labelled both artworks as experimental documentaries. They are non-fiction pieces that align with the field of i-docs in their approach to combining documentary and interactivity (Aston, Gaudenzi, and Rose, 2017), and also relates to immersive journalism (De la Peña et al., 2010; Uricchio et al., 2015) and the wider context in which interactive VR draws from many previous media experiences (Engberg and Bolter 2020; Murray 2020).

Additionally, both artworks share a large degree of artistic freedom in how these facts are presented to the audience. Although with two very different approaches, both works depart from traditional nonfiction and base their approach on a large degree of abstraction and spatial fragmentation, initial lack of contextualization, and a storytelling strategy based on micronarrative. These common features afford an experience in which the audience is left to piece together the broken-down elements, making connections between them and with their own cultural and social context and experience. The idea behind this approach, in which micronarrative is at the very centre, is that this can create a very different sense of engagement to that of a traditional documentary.

Both artworks are single user room-scale VR experiences. This means that the way to visualise the piece is through a VR headset in a designated area –of about 4 by 4 metres– where the audience can move in the physical and the corresponding virtual space. The result is a radical departure from experiencing a flat-screen piece, or even a 360 degree experience –that is, an immersive experience where the user is always at a fixed point in the virtual world, being able to look in any direction. In room-scale VR, one can go towards the virtual objects, look at them from different angles, walk through them, or kneel down to get close to a detail placed next to the floor. Additionally, the interactions can also be more (virtually) physical, such as pushing something, and may often imply having to physically get close to the virtual interactive element.

It is in this context that the scripting and design of such experiences becomes a sort of score to the audience’s movements. The spatial design of the virtual spaces, the timings in which everything appears or disappears, and the placement of each object and spatialized sound, are all part of what will affect how the users move, where they might be looking at any given moment, etc. In Pieces VR and The Smallest of Worlds are two examples that illustrate how different approaches to this can generate a very different experience from this point of view of anticipating the user’s movements. While none of the design approaches is more desirable than the other per se, the comparison and conclusions drawn from the analysis can be a very valid tool for future designers of room-scale VR experiences, when deciding how much they want to control, or anticipate, the body movements of their audience when experiencing the piece.

While I have discussed elsewhere these and other relevant aspects of these works (Soler-Adillon, 2020; 2022; Soler-Adillon, Brunner and Lange, 2023), the focus here will be on using them as examples to analyse how the audience’s movements, can be determined by design, thought the positioning of the visual, audio and interactive elements.

In Pieces VR

In Pieces VR is organised as a concentric narrative. This means that one of the nodes operates as a central hub from which all the other nodes are linked. In this case, this central node is also the entry point to the story, and there are four stories attached to it. Once each story has been experienced, the user is taken back to it to choose another one. Only after seeing all four, they will be taken to a final space with some information and credits, that operates as an off-landing for the VR experience.

Each of the four stories has a voice-over that drives the narrative, which varies from 45 seconds to two minutes in length. Overall, the experience takes about ten minutes to complete. With very few exceptions, everything shown in these spaces are 3-D hand-drawn elements that generally appear and disappear, or move as the narrative unfolds. Each story works independently and it is not explicitly connected to the rest. They narrate a train arrival to see an imprisoned father, a day-long trip for a 45-minute visit, a lonely walk in the prison yard, and the first family visit with a small child. The overall aesthetic is of very abstract and stripped down graphic elements sculpture-like figures, with spatialized sound. Animated and interactive elements are presented differently from one story to the next, in a way that greatly affects how the users move and, thus, making it possible to anticipate the probable position and gaze. This allows the introduction of elements placed in respect to this anticipated position. For example, falling at their back, going past them, etc.

In the first space, the positioning of a series of plinths –which are used in the artwork as navigation devices– placed in one corner allows very easily to anticipate where the user will be, and where they will be most likely facing, at the start of each of the four stories. From here on, in each of these the sequence of elements acts as a guide to create this sort of meta-performance of the person experiencing the piece moving along the physical space.

The train scene starts off with a seemingly infinite path on top of which the user is located. From one end, a group of figures approach the position of the user, leaving only a small space –the aisle– where the virtual figures won’t pass through them as the virtual drawing goes to the other side of the path. Interestingly enough, every single one of the observed users would move away from the path of the virtual figures and onto the aisle, so as not to (virtually) collide with them. Inevitably, users would be always looking at the train, guided by both the moving graphic elements and the spatialized sound of the voice-over that travels with it. So as the train disappears on one side of the path, it was a safe assumption that something appearing –and making noise– at the other side would do so at the back of the user.

The visit scene is the only one where the voice-over is triggered by the user. A small square on one side needs to be touched, and then it moves to the floor to mark the boundaries of a prison cell. As the story unfolds, the elements start to appear (always as hand-drawn sketches): a blanket on the floor, two chairs on an edge, then two parents and a child on the blanket, and finally the plinth once the voice-over ends. Here the sequencing of the appearing elements is the key to trigger the user’s movements. First, because they focus their attention. When the blanket is mentioned and appears on the floor, they will look down and tend to walk over it. However, once the three human figures appear, they will inevitably walk away so as not to step on anybody.

This is one of the very interesting things that was observed with this artwork. No matter how far from realism the elements were, the audience would feel very much inside the story and involved. Some even reported a certain degree of claustrophobia in that particular scene, despite the only visible boundary being a line on the floor on a black empty background (see Figure 1). Also, once the figures appeared, the parents sitting and the baby laying on the floor, no one wanted to be on top of them as it felt too intrusive.

Figure 1

The parents and child in the family visit scene. Image by Joan Soler-Adillon.

Finally, the other two scenes presented a sort of opposite layout that affected the position and gaze of the user. First, in the travel scene there is a central figure sitting in the middle of the space, with an infant on their arms. Here, the audience would step away to face them from the edge of the designated space, in an angle where they would be facing them. Some would move around a bit, but mostly would stay rather put. Here the only playful interactive element appeared. As some calendar leaves fell from the sky, only some of the users would realise that they could touch and move such leaves.

In a sort of opposite narrative design, the prison yard scene starts all in black. Then the spatialized sound indicates the audience where to face, and one sculpture appears at that particular edge. As the voice-over starts, sculptures appear and disappear forming one big circle, with the sound also moved around. Here, then, it was very easy to anticipate that the users will be standing more or less in the middle, and always facing the visible graphic elements. So it was possible to implement smaller, less important sounds that would go past their back, or run next to them, as to mimic the uneasiness of trying to have an intimate moment in a prison yard. In this one scene, it was once more quite easy to ensure that the plinth appearing at the end would do so at the back of the user, so they would have the chance to take their time before turning around to interact with it.

Overall, the design of In Pieces VR has a very strong component of anticipation and guidance of user movements. While there is no physical constraint to be at and look towards anywhere in particular within the designated room-scale area, it became very clear that the immersion in the virtual space generated a sort of self-imposed set of constraints, to the point that it was, in general terms, rather easy to anticipate the movement of the audience and, thus, designing accordingly when desired.

The Smallest of Worlds

Despite the similarities mentioned above, The Smallest of Worlds has many elements that differ from In Pieces VR. First of all, while it is also non-linear, it is organised as a branching narrative. There is a starting space, and from it the user chooses one in three options, that in turn will lead to three options, then two, etc. In total, there are seventeen nodes and each user will visit ten every time they do the experience, which lasts about twenty minutes in total.

The Smallest of Worlds is an artwork that is based on a large number of contributions from around the world. During the Covid19 pandemic lockdown, we collected videos scanning private spaces, and audio fragments documenting personal experiences during that time. The videos were converted to 3D objects using photogrammetry, and the audios were used as received. These are the building blocks of the work. In each of the nodes in the piece, point cloud objects are combined to create new virtual spaces from the real ones, and audio bits are placed on the scene to be activated by the users. Figure 2 shows one of these spaces, with its unfinished edges and point cloud aesthetics. The users will stay in each node between one and three minutes, just enough time to trigger the audios and unfold the hidden virtual objects when these are not shown from the start.

Figure 2

A view of one of the spaces of The Smallest of Worlds. Image by Uwe Brunner, Bettina Katja Lange and Joan Soler-Adillon.

In general, the movements of the users in The Smallest of Worlds were much more difficult to anticipate, even though a general positioning and hotspots for action were easy to identify. This artwork presents much richer virtual environments than In Pieces VR. The visual and interactive elements appear here simultaneously, rather than in a sequence. The only exceptions are some point clouds that are shown only after the user interacts with their corresponding trigger object. Their timings are not scripted at a specific moment. Similarly, the audios tend to be linked to an interactive object and are only triggered when it is touched, so the way in which the audio microstories are experienced is totally dependent on user actions.

This higher degree of user agency means less control or anticipation of the user’s movements, position and gaze at any given time. The invitation here is to wander around and explore the space from within. In general, in this piece the audience will tend to stay at the centre of the room-scale VR space and move to the sides to interact with objects such as the sound triggers. Only right after activating the navigation to a scene it is possible to anticipate which side they will be facing.

There is a relevant design constraint at play here. In order to avoid involuntary collisions with the interactive elements, they needed to be placed away from these anticipated positions. They were displaced towards the edges, thus generating these movements from within towards the outside and back. The complexity of the spaces of The Smallest of Worlds makes it somewhat unavoidable to fall into this scheme. However, this is precisely compensated by the visual richness of the environments, and the much richer spaces in terms of details to explore. So the more curious users will in fact escape these constraints in a freer way than in In Pieces VR since once you decide to explore in detail, going through the virtual objects becomes a feature, and you can end up finding easter eggs such as a scene inside an scaled-up human head, or a very crowded room where everything is small and about half a metre elevated from the floor, where laying down on your back can offer a very unique perspective of the space.

Conclusions

When designing immersive experiences, the storyteller is faced with the challenge of offering the viewer a frame-less mediated environment, where one has the freedom to look anywhere, yet at the same time have some degree of control or, at least, anticipation of where the user is placed and where they are looking at any given time. At the very least, relinquishing this control should be a design choice. In scripting and designing the virtual spaces, there is a balancing act about this knowledge of the users’ position and gaze, and thus the possibility of using it to the narrative’s advantage, or letting go in favour of user agency or, simply, an aesthetic choice to create a different type of environment.

The comparison of In Pieces VR and The Smallest of Worlds shows that some design and scripting decisions are particularly important to this effect. Not surprisingly, the placement of the graphic and interactive elements is a key issue, but there are many others that come into play here. First, simplicity vs. complexity. In a less crowded space, directing the user’s attention is much easier. There’s simply less to look at. However, this should not be done in a way that betrays the fully immersive virtual space. There should be some degree of exploration and movement available to the audience, which adds to the experience in comparison to what would mean to look away from the traditional frame and into somewhere else in the space that is simply a narratively empty expansion of it.

Finally, a second relevant decision is sequencing vs. simultaneity. If the graphic and interactive elements appear sequentially, it is relatively easy to anticipate where the audiences’ focus will be at any given moment. If they all (or many) appear at once, the attention is divided and, thus, it becomes much more difficult, if not impossible, to anticipate where it will be at any given point. This can also balance out with agency or lack of it, and with the prevalence of animation or interaction.

None of these strategies is more effective than the other in isolation. It depends on the type of experience that one intends to create, and what should be the prevailing tone, aesthetics, approach to user agency, etc. However, the knowledge of which design and scripting decisions lead to which effects and affordances in this respect is, as it has been argued here, a fundamental aspect in designing immersive narrative environments.

Competing Interests

The author has no competing interests to declare.

Author Information

Joan Soler-Adillon (PhD) is a Catalan artist and associate professor at Universitat Oberta de Catalunya (UOC), Barcelona. He has held previous academic positions at Universitat Pompeu Fabra (Barcelona) and Royal Holloway, University of London (UK). His research and practice revolve around digital interactive media and its manifestation in digital art –particularly interactive installation, interactive storytelling and Virtual Reality. From a full-body interactive game on an inflatable slide to VR-based experimental documentary, he has worked on a myriad of projects with a focus on behaviour design and interactivity, and on fostering audience collaboration and participation.