Introduction

In a typical visit of a heritage site, walking through the place, looking at and sensing the surroundings immerse the visitor in the historical experience. Audio guides, labels, leaflets, and tour guides provide additional information that connects the visitor to the place’s history, especially for sites that are quite empty of physical objects. Yet, the supporting material offered to visitors is often dissociated from the experiential demands of 21st century audiences (Lang, C., & Reeve, J., 2016). Furthermore, such methods of engagement have become increasingly unsatisfactory as tools for the educational goals of many heritage sites which have developed ‘more sophisticated understanding[s] of the complex relationships between culture, communication, learning and identity’ (E. Hooper-Greenhill, 2007: 1). In order to lessen the dissociation, increase experiential and situated knowledge building, and invite younger audiences to museums and heritage sites, several performative tools have been introduced over the last few years, such as interactive scavenger hunts, theatrical performances, dramatized walking tours, immersive storytelling, AR and VR technologies, and multimedia (Gheorghilas, et als, 2017). These tools allow visitors to not only witness historical narratives, but physically and socially interact with a localized past (Auslander, 2013).

Through collaborations with professional theatre makers, performers, digital technicians and designers, the heritage industry is increasingly using sensorial storytelling devices to enhance participatory experiences between audiences and heritage sites (Kidd, 2018). In immersive heritage performance, the public become a part of, rather than separate from, the story being told. At the same time, heritage organisations are exploring the use of immersive technologies such as Augmented and Virtual Reality (AR/VR) to create more detailed reconstructions of the past and attract younger audiences. What is common in these experiences is the use of storytelling to guide the visitor, a powerful tool museums and heritage sites use to construct nuanced historical details and create experiential frameworks that encourage internal dialogue, connections with the past, and provide complex and satisfying engagements for visitors (Bedford, 2001; Wong, 2015).

However, in heritage performance, immersive storytelling is often framed in a linear, didactic way, one that critics find stripped of its ability to evoke emotional and affective responses (Tivers, 2002). On the other hand, the use of digital technology in AR experiences can appear disjointed and may lack the logical journey found in more theatrical heritage experiences leaving users less connected to the historical subject matter. The aim of this paper is to introduce a performance methodology for the heritage industry that combines AR design with immersive theatre techniques focusing on sensorial and interactive storytelling interweaving digital technologies, the heritage audience, and the historical setting. By combining immersive theatrical storytelling devices with digital design technology, the sensorial engagement of the AR device is enhanced through an embodied storytelling design, a technique we describe as affectual dramaturgy. This technique was created to fuse the users’ interactions with and through the AR technology with the historical characters, the story, the journey, and the site of the experience. Affectual Dramaturgy has the power to enhance visitor experience by increasing historical knowledge through emotional engagement and contextual understanding, leading to a more personal and complex relationship to the heritage site (Perry et als, 2017; Tivers, 2002). The following introduces the concept of Affectual Dramaturgy as a lens for creating digitally mediated immersive performances for the heritage industry.

Immersive Performance

For both digital technologies and live performance genres, immersive experience building uses interactive methods to engage the public with the tangible and intangible heritage of a site. The term ‘immersive’ has been both broadly defined and contested in both the live and virtual performance genres, and yet continues to be used as a term for a type of audience experience found in the industry. The authors are not arguing that immersive theatre is unique in providing a sensorial experience by its audience; it is merely a tool to draw audiences’ awareness to the sensorial aspects of what occurs in every viewing experience within the dramaturgical process. The participatory nature of immersive experience, whether live or virtual, has proved popular to both attract and educate audiences through their emphasis on embodied and affectual engagements with heritage sites, historic artifacts, or archival material on display. Immersive experience encourages multi-sensorial modes of audience engagement while also created thematic relationships enhancing spectators’ imaginative response to the space, the subject, and the performers (Machon, 2013). Moreover, it is the sensorial experience of the event that particularly defines the immersive experience’s relationship between spectator and performance.

Immersive experience design has drawn interest from the heritage industry for its potential as a pedagogical tool because of its focus on participatory audience interaction to ‘animate’ the historical content, objects, and characters of the site (Maples, 2021). In Maples heritage performance at Oxnead Hall in Norfolk, audience members described how much easier it was to absorb information through the performance. They reflected that heritage sites ‘tend to have limited entertainment or engagement value, so what you learn is not so well retained,’ while the Paston heritage performance did not feel ‘touristy’ but rather allowed them to ‘indulge in a genuine sense of being transported through time to another era’ (SurveyMonkey, 2018). There is a reciprocity in sensorial engagement, allowing for an intimate relationship between the environment and the individual or group. In the context of the heritage industry, Auslander’s (2013) definition of the ‘intimate trafficking’ found in immersive heritage performance creates a connection between the ‘mind’ and the environment allowing for an imaginative, affectual relationship between audiences’ present and the heritage site’s past. By combining historical objects and archival documents with a lived sense of historical storytelling, immersive heritage performance can engage in audiences’ affectual experiences of history. Such techniques turn heritage sites into what Lucas describes as places of ‘symbolic action’, theatricalizing the space to give it a narrative relationship with its audience. Theatre makers are increasingly employed at museums and other heritage sites to ‘better immerse guests and effectively play out the theme or context of the spaces at hand’ (Lucas, 2016: p4). Interest in this phenomenon is seen in the growing industry of immersive technologies and participatory theatrical techniques found in the heritage industry to reach new audiences and create an animated museum of the 21st century (Cotter, 2015).

Maples’s recent immersive creations for the heritage industry have included site specific performances, immersive storytelling, dramatized tours, and scavenger hunts at museums and heritage sites across Ireland and the UK, combining historical dramatized storylines from archival sources with immersive theatre designed to enhance audiences’ affectual, imaginative, and embodied response to history. Dima has experimented with dramaturgy as a design methodology to create an AR-enabled experience through which audiences engage with the history of a heritage site and the many stories the building and its objects tell through that site. Through our collaboration, we aim to create hybrid performances in heritage spaces designed to let the audience members walk into history, and doing so, imagine the challenges of living through a time period, rather than looking back at it.

Immersive techniques are used to create ‘not simply a meta-theatrical work used in the heritage industry, but one which is meta-historical’ (Maples, 2021). By drawing the audience’s attention to the characters and their story, the authors desire to build an imagined, ‘lived’ historic environment for the audience and one that complicates our understanding of that environment and its relationship to the past. Affective dramaturgy’s focus on experiential audience response from within the storytelling structure, is designed to encourage a sensorial experience of history, provoking an empathic relationship to historic characters and events by members of the public through physical sensation.

Smart Glass AR Experiences

AR technology projects computer generated digital material such as 3D artefacts, video and images onto the real environment. In its inception experimental work used a pair of smart glasses to superimpose the digital material on the real setting, however, with the rapid advancement of mobile technology, mobile AR became an easy, affordable, and well adopted technology, especially in cultural heritage.

In the last years, the development of AR has seen the return to developing a new type of headsets that look like a pair of glasses. These use advanced optics technology to project 3D computer generated imagery on the lenses. The earliest example was the, now discontinued, Google Glass project, while at the moment there are a few headsets on the, rapidly growing, market (e.g. Microsoft’s Hololens 2 (2020) and Magic Leap One (2020)). AR headsets superimpose 3D computer generated imagery onto the real world, producing a similar result to mobile AR. As opposed to VR headsets, AR glasses do not create a totally virtual environment, allowing the user to have a clear view of the physical space. The difference of using an AR headset to using the mobile is the opportunity and potential for full body interaction, since the hands are not engaged (Vainstein, Tsvi & Joel, 2016). Looking through a headset while not losing the view of the physical environment and being able to have the hands free is a more immersive experience than using the mobile (Mason, 2016). In addition, the experience design can make use of a variety of sensorial input, such as eye tracking and voice, to enable further interaction mechanisms that may be more natural to visitors of all ages. The majority of research on AR headsets have so far focused on areas of science and ergonomics (Mason 2016; Mokatren 2018, Vainstein 2016; Kerr 2011) such as navigation to a target location, situational awareness, and augmented imagery visibility. At the same time, an increasing number of applications are being designed for different contexts, and indoor and outdoor cultural heritage sites have been a fruitful ground for research and experimentation with AR. The hardware interface is visibly lowering the barrier to accessing and communicating cultural heritage information (Litvak 2020) and scientific research helps with improving the ergonomics with each new model. This is particularly important for heritage site visits as they require the interface to be as transparent as possible so that the focus of the visit is put on the navigation through the site for learning and social interaction.

Although there is more than 20 years of research on AR in cultural heritage, the literature focuses mainly on technological factors and very little on the context in which it is used. A few projects have addressed the necessity to utilise the power of stories (e.g. Liestoel 2019; Spierling 2014), however, these too have approached storytelling as a linear, didactic experience with stories being object-centric or applying simple game mechanics in a way that the experience is primarily a game that includes a learning aspect. In the area of AR headsets, with its immense potential for engaging visitors, the few heritage applications rely on the novelty of the technology and do not employ any narrative. A few indicative apps that have been developed over the last four years include the TouristicAR system, an AR application that provided context-aware content for tourists at UNESCO World Heritage sites in Malaysia (Obeidy, Arshad & Huang 2018), (MR Museum in Kyoto 2018)’s Kennin-ji, a 10-minute AR experience that combines The Folding Screen of Fujin and Raijin with 3D graphics, and ‘HoloMuse’ that allows user interaction with archaeological artefacts from the Anonymous Museum’s collection (Pollalis et als. 2017). With the lack of attention to the dramatic potential of the headsets these works miss the opportunity to connect into the stories of the site, and touch on the intangible heritage and the physical surroundings and bring them together in a unifying narrative that creates an affectual experience. Aspects of the experience such as the use of multisensorial modalities, points that provide opportunities for interaction, playful encounters, and emotional and moral engagement have proven to motivate audiences and offer a meaningful dialogue that helps them learn and be entertained (Vagnone 2015). AR headsets provide a different experience and, as such, open up a plethora of different opportunities for connecting to the past.

As with every new interaction technology, along with scientific research it is necessary to research the design processes under which the interface and the context are tied harmoniously together to produce a meaningful experience (Dima forthcoming). Mason (2016) and Tom Dieck (2014; 2016) used early prototype headsets to study user interaction through AR headsets in a cultural context, mapping requirements and offering recommendations based on the use of prototype experiences in cultural places. Hammady (2019) explored consideration for the design of the visual interface with Microsoft Hololens 1, and Pollalis et al. (2017) studied gesture-based interactions with holographic artefacts. In the Sutton House Stories project Dima researched the application of dramaturgy as a design methodology to create an affective AR experience using Microsoft’s Hololens 1. Dima (forthcoming) suggested that looking at the design of these AR experiences through the lenses of dramaturgy offers a fresh perspective in creating experiences that are immersive not only because of the technology used but because of their design. The methodology focuses on orchestrating elements within and across three intertwined design pillars, that of narrative design, technological considerations, and learning objectives for the heritage site. Design pillars, a term used in games development, are a list of no more than three or four design goals that act as a compass for the development of the interactions and help keep a clear vision of the final work. Dima’s design approach offers a holistic view of the experience without focusing only on simple superimposition of the virtual material but considering how to work with this material in ways that make up an engaging encounter with the past. The desired outcome is that when visitors leave, they have not only looked at fragments of the place’s history but also walked through the memories of the place. (Figure 1).

Figure 1

Visitor looking at superimposed content through the Hololens AR glasses, Sutton House Stories, 2019.

This methodology was used to create Sutton House Stories, a 4 minute AR experience designed and developed in 2019 for Sutton House, a Tudor house in the London Borough of Hackney (UK). The narrative was developed based on the curatorial requirements of the site and a lot of the content was based in the house’s archives. The viewer walks around the Great Chamber, guided by three voices representing three key figures that lived in the house across four centuries. Professional voice actors were employed to act out the script in the form of a monologue as if they are talking to the viewer. While viewers listen to the voices, they see and hear superimposed content which may be related to what the narrator tells them or may not be. At certain points they also have the opportunity to interact although this is not made explicit to them. For example, there is a sequence where a pair of dance shoes perform a minuet (Figure 2).

Figure 2

Dancing the Minuet, Sutton House Stories, 2019.

Viewers are free to walk around the dancing shoes and watch their dance and if they feel like it try to mimic the steps. A few participants did so or thought of doing so during the evaluation. When asked how the experience felt, the majority of the participants used the words ‘the house came to life’. This sentence reveals that the viewers were not just engaged or immersed in what they were experiencing but that they viscerally felt closer to the house’s history and its past inhabitants. Dima found that this feeling is something that can be enhanced by carefully orchestrating the materials of the experience, which include but are not limited to the 3D computer generated superimposed material. Grabbing and manipulating a holographic object is impressive on its own but if this act is placed within a wider story that unfolds from the moment the visitors wear the AR headset it acquires a more meaningful stance. The third pillar, narrative, can be viewed as a source of stories attached to the site but also as a means to bring the visitor experience under a unifying narrative. In Sutton House Stories this narrative was revealed at the end with one of the characters asking the visitors to share their story so that it is not forgotten. In this way the experience concludes with a moment of reflection by means of direct dialogue with one of the characters that accompanies the visitor through their journey.

Affectual Dramaturgy

We have seen so far how immersive heritage performance and smart glass AR design have separately the potential to create an embodied and affective experience of a heritage site. Deriving from our individual practices and in light of the need for new perspectives in digital heritage design is the exploration of a new form of audience interaction with, and participation in, cultural heritage. How best can the two practices be combined and what opportunities does this create for research, practice, and audience development strategies? By making a few steps towards a new perspective for creating such experiences in heritage sites, we evolve Dima’s dramaturgical approach and look at what affective dramaturgy entails, and how narrative design and technological considerations change to accommodate and enhance the affective qualities.

Combining live immersive performance and enhanced digital experiences, such as the AR design, allows audience members to encounter digitized historical performers, sounds, and objects in and around them, just as they encounter the heritage site, fusing the real with the virtual. Furthermore, interactions with live performers may also enhance an ‘intimate trafficking’ between an audience member and the historical characters and events (Auslander, 2013). Staging intimate encounters between live actors and audience, (such as performers whispering in the ear of an audience member, delivering a letter to them, or seeing them move in the distance), may enhance sensorial and embodied sensations and reactions from members of the public. By combining the virtual with the live performance, the richness of detail available through the digital design may be further enforced through encounters with live performers.

Our hybrid methodology combines AR design with immersive theatre by designing the visitor experience through an embodied sense, or indeed, affectual notion, of dramaturgical storytelling. Ann Pais distinguishes affect from the ‘emotion’ which is a common terminology for theatrical performance. For Pais (2016), ‘affect’ is an embodied experience of ‘social atmospheres’, one that enhances ‘the sensitive charges or felt intensities carried by words, sensations, thoughts and emotions that circulate in social spaces (3)’. Our definition of immersive storytelling as ‘affectual dramaturgy’ embraces the idea that immersive experience creates an embodied and sensorial relationship to the function of storytelling in the heritage site. The story is not solely material, fact-based, or text driven, but a lived, and embodied, experience which relies on affect as a physical, sensorial, and imaginative, act for the participant.

Cross-disciplinary methodology of sensorial research in this project allows for a multivalent understanding of human nature and society through public reception of heritage performance. The enhancement of sensorial experience for audiences in both the heritage and theatre industry have included touch, taste, smell, and sound as important ways to both engage and challenge audiences in their experience of a site. By creating immersive heritage performances through a dramaturgical focus on affectual, embodied experiences which allow the visitor to experience a heritage site as a sensorial, embodied event, the entire relationship between the museum goer or heritage audience and the site becomes a lived experience of the past.

Design Pillars For Affectual Dramaturgy

With Affectual Dramaturgy we widen the technology and narrative aspects of Dima’s methodology, to include interaction methods and processes from immersive heritage performance that enriches the dramaturgical process with affectual and embodied qualities. We integrate these methods with consideration of how immersive technologies can enable and enhance these interactions. The design pillars of Affectual Dramaturgy are focused on the experience of the user through their journey, the story being told of the heritage site, and the integration of the live and virtual experience.

Participatory Journey

One of the important aspects of the hybrid experience is the guiding and occasionally leading of the visitor throughout the experience and in the time shortly before it begins, what is usually called ‘user onboarding’. More often than not AR experiences start when the visitor is on site, and if there is a mobile AR application perhaps the visitor has been asked to download it on their phone prior to the visit. Immersive performance, by contrast, often incorporates the introduction of the visitor to the experience as a key part of the dramaturgical structure, with a character, or characters, greeting the audience and explaining the rules of ‘the game’ through world building devices which incorporate all aspects of the experience into the logic of the story. The AR headset can be integrated into the story through the use of a live performer explaining the magical or time travelling world of the AR experience and the visitor’s role in it.

Before or during the entrance to the site is when the AR headset becomes merged into the story an actor can be used to introduce the visitor to the experience while also offering key information the visitor needs to know about the device and rules of the journey they will undertake. The narrative should answer questions such as: What is this device for? Why does the visitor need it? What role does it play in their journey? This leads to questions about the role of the visitor who either constantly wears it or uses it at specific moments. For example, in workshops conducted using Dimas’s dramaturgical framework, the role of the ‘time traveller – historian’ was the most common suggestion. In this role the device can be a time travel device worn at all times, or a special magic lens through which one is able to see things of the past. In any case, the visitors need to be accustomed to wearing and using the device, and actors can help greatly with the smooth integration of this part into the story. Visitors, who are now viewers, need to be allowed some time to get used to seeing through the glasses, and practice some of the gestures that will be needed during the experience. Normally this part is done ‘before the experience begins’, but in this case the actors can weave this part into the story.

Maples’ immersive theatre techniques celebrate the sights, sounds, and smells of heritage location by drawing audience members’ awareness to them within the storytelling itself, and the performers’ looks, gestures, and choreography (Figure 3).

Figure 3

Sarah Lawrie and Sarah Edwardson (actors), Oxnead Hall, 2018.

Maples uses not only performers, but objects and sounds, to guide visitors from site to site, allowing spectators to experience an environmental, as well as narrative-led, journey through time and space Storytelling may incorporate affectual dramaturgy devices by instructing the performers to enhance the environmental impact on the senses of the audience through the performers engaging in acts of listening and looking at the world around them. Through the performers’ attention to objects and the environment, the audience also begins to interact with the site in different ways. In Maples Paston heritage project in Norfolk audience members followed the performers across the vast grounds of a 17th century estate by following and interacting with actors and objects such as letters, funeral tokens, and flowers. Audience members reflected on how the participatory nature of the production enhanced their experience of ‘living history’ (SurveyMonkey, 2018). Members of the public mentioned how ‘[the play was] surreal as if we stepped back in time’, the past ‘came to life’, ‘history was brought to life’, or ‘I felt like Robert Paston was walking amongst us’, while other members of the audience spoke of the feeling of moving ‘in and out’ of the past as they followed, interacted with, and were guided by performers on their route through the heritage site (SurveyMonkey, 2018).

This technique creates a kind of performative mindfulness, bringing the site to the foreground of attention (Figure 4). Similarly, in Dima’s Sutton House Stories, environmental virtual materials were an important aspect of the experience. There was a virtual fire burning in the actual fireplace during the whole time with fire cracking sounds on the background of the monologues, and bird chirping when a virtual window opens on top of a real one to reveal the countryside as it used to be 400 years ago (Figure 5).

Figure 4

Ben Porter (actor), Oxnead Hall, 2018.

Figure 5

Virtual window opening to the view of the country side, Sutton House Stories, 2019.

According to Rickett, the use of affect in the heritage industry incorporates, ‘a wide range of social, sensory, and emotive registers’ into the meaning making constructed at the site’ (Fisher et als, 2015). Participatory performance practices invite the public to contribute to a multisensory experience of the site. They also allow for a level of improvisation and personalized experience which can enhance the public’s affectual, temporal, and embodied relationship to the heritage site.

Conclusion

We have outlined the first steps towards a methodology for creating immersive experiences in heritage sites that can bring their histories to life through affectual dramaturgy. We have introduced methods for engaging emotionally the viewer that derive from theatre practice and are enriched with opportunities offered by the technological devices. Under this new perspective, affective interactions are at the centre of the creation process while the acting and technology design support their creation.

This work is timely as AR hardware is rapidly developing so that AR smart glasses could be the next commodity immersive technology. With audiences growing more used to immersive technologies, a future where visitors will be entering a site with their own pair is not far away. In addition, AR allows a shared hybrid experience, which feels more natural than collaborative VR as parties are constantly in the real world and able to see each other there. The affectual dramaturgy perspective can help AR users through the change of their embodied sense when wearing the headset by providing tools to make the integration of the device into the storyworld seamless. Finally, our work emphasises the importance of cross and transdisciplinary collaboration. Digital interpretation is increasingly becoming the focus of heritage organisations and the heritage staff’s skill set is being updated extensively. Though there is not always the resources to combine live with the virtual AR experience in a work, we argue that the fusion of different disciplines, from technology-oriented experiences to theatrical embodied storytelling remains fundamental to creating digitally-mediated, affectual, interactive, historical narrative experiences designed to successfully immerse visitors in the stories and histories of the heritage site. Our proposed methodology is an example of different disciplines working together in a way that can open up opportunities for new skills and business models. For the heritage industry this can mean new methods of work to attract diverse publics and new roles that rise at the intersection of heritage education, immersive performance, and digital design. Equally, creative industries such as theatre and digital design can find a new outlet for their skillset that can offer resilience and innovation in current uncertain futures.

Competing Interests

The authors have no competing interests to declare.

Author Information

Dr Mariza Dima is an interaction designer and Lecturer in Games Design with a focus on Creative Technology for Games at Brunel University London. With a background in Human-Computer Interaction (HCI) and User Experience design, she has worked with physical, mobile, and haptic interfaces to serve interactions in socio-cultural contexts, prominently in performance art, cultural heritage and social innovation. She has led several projects in partnership with prestigious theatrical companies, museums, cultural organisations, SMEs, and visual arts institutions, and published widely on audience engagement, digital media and immersive technologies.

Dr Holly Maples is the Director of Impact and Post Graduate Research at East 15 School of Acting, University of Essex. She is a theatre practitioner, educator and scholar. Her performance practice particularly focuses on dramatized immersive and sensorial experience techniques in the heritage industry. From 2017–2020 Dr Maples acted as Drama lead on two distinct HLF and Arts Council funded heritage projects in Norfolk with the Paston Footprints project and the Norwich Castle Museum. In 2019–2021 she further created a heritage performance exploring the 18th century slave trade and abolitionist movement in collaboration with historian, Dr Inge Dornan.