The backdrop

Museums have long been providing novel forms of technologically mediated experiences which have not only extended their visitor reach, generating new revenue nationally and internationally, but also provided original ways to engage audiences with their collections. While museums nowadays routinely host augmented and immersive experiences, there have, however, not been many examples to date of scalable and modular mixed reality theatrical experiences in the sector, by which I mean experiences including live actors, sets, and a certain suspension of disbelief.

A wide range of technologies are used in the design of cultural experiences for education, exhibition enhancement, reconstruction and the creation of virtual museums (Bekele et al 2018). These include augmented, virtual and mixed reality. The latter term, which is especially relevant for this project, indicates an overlap between physical and virtual worlds produced through the use of a range of more or less immersive technologies. The term was first discussed by Paul Milgram and Furnio Kishino, who proposed the idea of a ‘mixed reality continuum’, which connects what they called ‘real’ and ‘virtual’ environments, including intermediate points spanning from ‘augmented reality’ to ‘augmented virtuality’ (1994).

Mixed Reality has been successfully used in several contexts spanning from training and medicine to entertainment. While in performance mixed reality and immersive storytelling have been used for some time, for example, by Brighton-based collective Blast Theory (Benford and Giannachi 2011) and London-based company Punchdrunk (Machon 2018), in the museum context, the use of mixed reality is still relatively new, though examples developed by Microsoft have been piloted in recent years, and include the experience of a Ford car at the Petersen Locomotive Museum (2018); encounters with old Japanese artworks at Kennin-Jim, the oldest Zen temple in Japan (2018); a meeting with an astronaut in Defying Gravity exhibition at the Smithsonian (2018); and The Met Unframed (2021), an immersive virtual art experience which was developed during the pandemic. Interestingly, the Defying Gravity exhibition allowed four people to share the experience at the same time, while the Met Unframed, powered by Verizon 5G Wideband, made it possible to literally superimpose over 50 artworks from across the Met’s collection onto the walls of users’ own homes.

It has been shown that heritage providers hope that immersive experiences will ‘(a) increase visibility and contribute to a culture of innovation; (b) appeal to new audiences (c) allow for more meaningful participation, (d) facilitate better engagement and (e) provide additional revenue’ (Kidd and McAvoy 2019). Thus immersive experiences of art and heritage could enhance museum visiting, facilitate social interaction, emotional engagement, offer embodied and spatial interaction, and promote novel forms of learning, for example about sites which were destroyed or can no longer be accessed. Altogether the use of these technologies has been shown to generate deep spaces that can be penetrated both inside and outside the museum in which visitors continuously reposition their own presence across different temporalities and spatial configurations (Giannachi 2021). It is worth noting, of course, that theatre is in itself an immersive form and that it has, throughout its history, often used a wide range of technologies to deepen and widen its audience’s sense of immersion and presence.

The mixed reality technology selected for this project was Magic Leap which uses a head-mounted virtual retina display that superimposes 3D computer-generated images over real-world objects. The display works by projecting a field of light into users’ eyes. When using Magic Leap, it is literally possible to see the physical world and, blended within it, the digital, hologram-like world created through Magic Leap. The digital world in this case does not look like a film, but rather like sets of 3D entities, making it possible for users to establish a strong sense of presence in both the physical and digital world. Because of the innovative nature of adopted technology, it was decided to involve audiences who would be attracted into the museums by building novel kinds of experiences which were a hybrid of a game, film, theatre, and mixed reality. This would be immersive (referring to the objective level of sensory fidelity in virtual reality) and enable a sense of presence (referring to a user’s psychological response) (Slater 2003), producing a highly effective learning environment that could help audiences to imagine life at another time and in another space. In this respect, it was thought that the National History Museum experience could bring the past to life while the Science Museum experience could immerse them in a futuristic scenario featuring a world where hazard robots could be created to resolve several crises.

Target audience

The target audience was identified by Factory 42 through a shared psychographic mindset rather than by a demographic segment. The name initially given to the target audience was The Open Gen, and their common characteristic was identified in their willingness to ‘give it a go, up for it’ mindset (Optimistic, Proactive, Engaging, Now). The closest museum segment to this group is formed by those defined as Entertainment Seekers, Engaged Community Drivers and Trend Awares, which are based on the Science Museum group audience segmentation and are based on the Morris Hargreaves Mcintyre (MHM) Culture Segments work and include a science engagement aspect (Factory 42 2019). As a broad set of characteristics, this audience likes to feel special, is engaged socially, culturally and possibly even politically, identifies value with shared and shareable moments, actively enjoys finding out about new products and experiences, especially in association to key influencers, and wants to define the personalised aspect of the experience.

To attract this audience, a franchise name was selected to comprise both experiences with the overarching aim being the creation of a new kind of entertainment that would be intriguing, fun and educational at the same time and address the fact that audiences nowadays tend to consume entertainment and even education in different ways. The name of the franchise, Dimension X, intended to invite the audience to step into ‘a new world’ where something ‘other worldly’ was happening. The consumer-facing materials intended to highlight the fact that audiences would become ‘dimensioneers’, finding their bearing within this new kind of experience by co-operating with family and friends as part of ‘an action’ that could be repeated across platforms and venues, either by progressing the narrative, like in a sequel, or by creating different versions of the experience, using distinctive content, for example by different museums. In this sense, the experience, aimed to be the pilot for a franchise that could then be expanded on and built on in subsequent years with other museums.

Early Research & Development

The original project packages included cross-team creative workshops, prototyping of concepts and testing with target audiences, followed by creative development, using themebooks whose different versions show the evolution of the project, to discuss ideas, software and physical design. The first objective was to develop, engineer and/procure the software, hardware and physical resources required to build the visitor experience needed for Beta testing with audiences. This in turn was meant to be followed by physical build and testing to integrate software, hardware and physical resources. Business models were meant to be developed throughout to maximise the commercial potential of the work.

The aim had been to conduct initial research in November 2019. This included obtaining ethics approval for audience research conducted by the Exeter research team (granted by the University of Exeter). Further tests with audiences were meant to be conducted in December and January 2020, with the plan to go live in the museums in the Spring of 2020, with an international rollout planned for 2021. Preliminary research conducted by Factory 42 before the start of the project showed that the new technology had inspired ‘wow’ movements; that at that point in time audiences had not quite experienced anything like Magic Leap before; that audiences were especially interested in seeing the overlay of physical and digital worlds; that the use of the technology would appeal to most ages, except very young children for whom Magic Leap is not suitable. Interestingly, films such a Jurassic Park and Minority Report were mentioned by audiences as possible points of reference.

The first project-wide workshops (see Figure 1) saw Almeida Theatre Director of Participation and project creative director Dani Parr involve the team in thinking about which kinds of robots or dinosaurs audiences might expect to see in Magic Leap; how groups would work together in mixed reality; how the chosen spaces within the museum would work in practice; what active learning could mean in this context; how the experiences could be accessible and facilitate a sense of exploration while also being semi-scaffolded and facilitate live orchestration on the day; how the story might be extended before and after the museum visit; and what the relationship between fact and fiction, imagination and museum learning might be. Discussions focussed on how the design of the experience would need to appeal to people from different age groups, cultures, and competences; work for families, friends, and individuals; have multi-sensory triggers; be accessible and scalable for touring; encourage repeat visits; have a wow factor and be scientifically accurate.

Figure 1

Initial workshop led by Dani Parr. Photo: Gabriella Giannachi.

These workshops found that the team felt it was important to design collective experiences, and that audiences should be able to discover something, for example that different species of dinosaurs lived at different points in time in history. From a story point of view, roleplay was discussed as a strategy for discovery, for example of a new fossil, or time travel to explore different kinds of dinosaurs, raising ethical concerns as to how to communicate to audiences what was real and what wasn’t. Among the wow moments identified were the creation of very large dinosaurs, the possibility of seeing the world through the eyes of a dinosaur, the use of historic insects and plants and the possibility of creating a multi-sensory environment.

Subsequently to these workshops, the research focused on how to build the mixed reality experiences technically, creatively, and spatially, and understand what places they would occupy within the two museums both physically and curatorially. For the latter, the two museum audience research teams investigated how visitors would arrive at the two experiences, and what they would be encountering before and after each of the experiences. To carry out this element of the research, the Natural History Museum conducted several walk-throughs, testing how participants, who were observed by team members, moved within the museum, and what they engaged with and documented (see Figure 2).

Figure 2

Walk-though captured what audiences documented. Photo: Gabriella Giannachi.

At the same time, the creative team led by the Almeida ran a number of workshops looking into how audiences would behave in narrow spaces while wearing headsets (see Figure 3); how they might relate to and collaborate with each other; how easily they would understand and respond to instructions; how long they would be able to concentrate within the experience; and how they would cope with being asked to take on different roles, acting as onlookers, participants, gamers, learners, teams, or even researchers at different points in time. Concurrently, a series of intensive retreats were held, using storyboarding to workshop immersive theatrical approaches with an actor and a limited number of users in the studio. These tests, focussing on immersion, engagement, emotion, and learning, run in parallel with audience research into collection engagement led by the museums, and produced user feedback that informed the initial prototyping work at Factory 42 which, crucially for the museums, aimed to emphasise the delivery of scientific engagement and ensure the accuracy of the science drawn from.

Figure 3

Performers and audiences testing a basic run-through with Magic Leap. Photo: Gabriella Giannachi.

Early prototype testing

A new iteration of the design of the pop up version of the experience was finalised in a theme book in late November-December 2019 and testing with around 100 people was completed at the Natural History Museum Jerwood Gallery providing crucial insights for subsequent versions, including the March 2020 launch at the Metrocentre, one of Europe’s largest malls based in Gateshead. The testing at the Jerwood Gallery, which was not for the main location-based entertainment (LBE) version but a pop up version that could tour while the LBE version remained located in the two museums, was done behind closed doors, but many members of the public became intrigued by the sign placed in front of the gallery and hovered outside trying to have a peak into the gallery and find out what the experience consisted of and when it would open to them (see Figure 4). The Jerwood Gallery experience took place within a black box type of set that was to be positioned inside the shopping centre experience. The first section of the set was built to host the on-boarding for each of the experiences and saw staff help audiences putting on the headsets and receive basic instructions about the experience. The set was then divided in two parts, one for each of the experiences. In the Dinosaur Experience, the core gameplay consisted of cleaning up a ghostly outbreak of dinosaurs by collecting their paleontological residue. In the Robots Experience, audiences built their custom robot which then appeared in front of them as a digital translucent blueprint model before being 3D printed and transformed into a solid photoreal machine. The robots were then used to navigate a city and either lift citizens in danger, repair collapsed buildings or spray water onto fires. In both cases scoreboards appeared at the end of the experience showing how the audience did in the experience.

Figure 4

Natural History Museum prototype testing board. Photo Gabriella Giannachi.

Feedback from testing at the Jerwood Gallery indicated that the version had a lot of potential, but more attention needed to be paid to onboarding and the orchestration of the experience as audiences appeared to get confused about the overarching aim, context, purpose, and progression of what they had been asked to engage with. The testing also showed the complexity of the Magic Leap integration with the set built, especially in relation to lighting and materials used which could severely affect the Magic Leap ability to deliver the mixed reality. Feedback suggested some audiences struggled to fit the equipment on by themselves; were confused as to whether the experience was competitive or collaborative and whether they would receive a reward for achieving their mission and obtaining the highest scores. Feedback also indicated that troubleshooting had not been factored into the narrative, that people could not understand the relevance of the set in the experience and were not clear about what learning had occurred. Having said this, many enjoyed the prototype. Generally, audiences rated the experience highly, with 89.8% rating it at 4-5 (on a scale of 1 to 5) and 83% would do the experience again. Testing thus revealed that the experience needed to be rescoped and that more attention needed to be paid to onboarding (by including, for example, something to ‘entertain’ participants who were waiting to be onboarded with some initial snippets of the experience to heighten anticipation). Moreover, testing showed that the Magic Leap lost its calibration when moving from one room to another. Therefore, the decision was made to reduce each of the experiences to two rooms and rework the narrative in relation to the digital and software assets while also scoping a lighter mobile version to be used in festivals or shopping centres.

The shopping centre experience testing

Factory 42’s original proposal to the Audiences of the Future programme envisaged that a version of the mixed reality experiences designed for the two museums should also be experienced in shopping centres. This proposal raised interesting questions during the research and development phase of the project as to the design of the experience in a non-museum space, both in terms of its content and structure, and in terms of its layout and learning outcomes. Detailed attention was paid to Intu’s research into their own audiences, including as to how e.com, computer vision and AI are changing shopping experiences, and this, alongside recommendations by the two museums audience research teams, prompted decisions about the length of the experience, its content, structure, format, and pricing.

Intu is the largest shopping network in the UK with over 22 m² ft of retail. The leisure space and the site put forward for the experience, the Intu Metrocentre near Newcastle, is one of the largest city centre shopping destinations in the UK with more than 370 shops, making it the second largest shopping centre in the UK. The shopping centre has an average dwell time of 1 hour and it is anticipated that the provision of an exciting mixed reality experience might prolong that while also offering a flavour of the longer and more complex London versions. Interestingly, while the setting up of museum experiences in a shopping centre, as well as in other non-museum spaces, such as airports, is a fairly common practice, especially in the USA and in Europe, they still rarely involve interactive and immersive technologies or performative events.

It was decided that for the Intu shopping centre, the two pop up versions of the Dinosaurs and Robots museum experiences, focussing on dinosaurs and robots respectively, would be built side by side, using the model tested at the Jerwood Gallery. This was largely to maximise marketing and visitor engagement within the confines brought on by this environment. Despite the fact this was designed separately from the other two museum experiences, and that it was meant to be a different version of it with not much narrative and no actors, it used some of the same digital assets and so it was considered crucial that it should align with both museums’ brands and that the content should be curated with a learning perspective in mind. This version, which could be described as a 10-minute game, showed that visitors craved a story. Moreover, while there had been some clear improvements and people really liked the T-Rex and the other dinosaurs, some people still struggled to find meaning in the overall experience, especially from a learning perspective, and it became clear that more work needed to go into the parallel development of the tech and the storyboarding so that the narrative could take better advantage of the technology. Hence it was found that if the shopping centre experience was redesigned, it should consist of a simpler version with a different narrative from the main LBE version.

Interestingly, as the Digital Director at the Science Museum Group, John Stack, pointed out, it had also become apparent that the management of the design of immersive experiences required a different approach from the one that had originally been envisioned. In fact, initially, this project, like other creative projects, had been planned in a linear way: narrative > concept design > technical design > build > commercial model > marketing > go live. But as all these components proved to be interrelated, they needed to be carried out in parallel. Not only, as different disciplines had been involved concurrently, more time than had originally planned needed to be devoted to considering differing practices and terminologies, but also the various creative dimensions of the project were moving at varying velocities and rhythms, in that not everything was simultaneously being progressed at the same speed. For example, at Factory 42 some aspects seemed to be developing rapidly for the immersive theatre team and slowly to the software development team. This finding is crucial in terms of the identification of an underpinning framework for the design of mixed reality theatrical experiences in that ways of prototyping and testing need to be adapted and so also scaled right across the development and production period.

Redesign due to COVID-19

The COVID-19 pandemic significantly affected this project, pausing the work for nearly a year during the lock-down periods. First, the enforced closures meant that the Intu experience was cut short, and the scheduled openings of the main experiences in the two museums had to be postponed. Revised scheduling in the museums during the pandemic meant that, upon reopening, the pre-planned exhibition spaces were no longer available. Due to a renewed exhibition programme and regulations regarding safe visiting, the museums were no longer able to commit to the new and still, at that stage, uncertain delivery dates of the project. Moreover, Magic Leap indicated that they would shift the development of the product to enterprise rather than the consumer, presenting additional challenges to the project and illustrating the problematics of developing content for immature technologies. However, new funding was made available which enabled Factory 42 to continue working on this project during the pandemic. This brought on two fundamental changes: the first led to Factory 42’s development of a mobile AR Route, resulting in the production of two augmented reality apps called ‘My Dino Mission AR’ and ‘My Robot Mission AR’ that launched in the Apple App Store and Google Play in 2020, and which this report will not cover. The second led to Factory 42’s production of a revised experience which opened for a short run in London at the Hoxton Docks in the Autumn of 2021.

It was also decided that the latter and final deliverable of the project was to be a single LBE called Lost Origin with a revised learning dimension that was not as closely tied to the Natural History Museum’s learning objectives. This single experience combined elements of Dinosaurs and Robots which were reimagined to match the chosen venue, Hoxton Docks, so that the story became about why dinosaur bones ended up in a warehouse. Assets for each of these experiences, including the volumetric audio and performance by Jodie Whittaker, dinosaur and fossil 3D models were built on, using aspects of the Robots Experience with an AI storyline as well as aspects of the Dinosaur experience, so that through a plot twist, audiences who thought they were involved in an experience resembling the former found to be immersed in a storyline more akin to the latter. When the venue was finally secured, the audience flow and size of sets in the rooms were refined, including several elements that were infused with environmental storytelling, so that props would respond to the audience and individual audience members would have a different experience according to how they engaged with the props on stage.

Lost Origin

The final part of the development was dedicated to a new world building and consequent drafting of the world story bible and narrative treatment. These, building on previous iterations, were also devised and led by the Almeida Theatre Director of Participation and project creative director Dani Parr and co-written with Hannah Wood. The wider experience design team comprised Parr and Wood, as well as Michelle Feuerlicht, Ross Phillips, Mike Golembewski and other members of the set design and technology/interaction design teams, including also the Magic Leap experience design team. In their vision, the audience was meant to work actively and as a team by engaging with the story, its characters, puzzles, narrative mechanics, and, at the end, the mixed reality environment. The experience was to be narrative-driven, and include live performance, environmental storytelling, narrative mechanics, and elements of improvisation and audience interaction, where actors respond to audience personality but also around the use of the Magic Leap, especially in the Magic Leap onboarding area where audience members are fitted with the Magic Leap kit. Elements of the plot would be discernible through projections, 3-D miniatures, lighting and sound, and the Magic Leap itself, to create a fully immersive theatrical experience bringing together visual effects with credits spanning The Hobbit, Matrix, Harry Potter and Hunger Games. In other words, the story was embedded in all elements of production, including the set, so that the digital and physical dimensions worked together to create one world.

Bespoke multiuser VR tools, built in Unity for the Oculus and Photon platforms, made it possible for the team to develop spatial understandings of the proposed design concepts, and explore the impact of the physical environment on the narrative pacing and development even before the physical set was constructed through a 3D model that was built in Unity to enable all teams to see/experience how rooms would work. This was vital for the writers who needed to adapt the script to fit the interactions and timings and for the set designers for the placement of key features like the foliage and miniatures. This was also key to the iterative design process, and the emphasis on innovation and technology embedded in storytelling. Through developing and employing bespoke process-specific tools for the collaborative spatial pre-visualisation of in-progress work, the team were able to proactively address and mitigate some of the challenges of designing a highly site-specific project at Hoxton Docks in East London.

The story was described initially as a ‘supernatural thriller set in an authentic, realist world of dark web criminal networks, deep learning automation and clandestine operations infused with eerie paranormal mystery’ (Factory 42 2019). Subsequently described as a ‘supernatural mystery set in a dark web marketplace where a clandestine operation evolves into a magical adventure’, the story started when, the day before the experience, audiences received an in-world email and video briefing telling them about a suspected criminal activity linked to a deliveries business called origin. The email confirmed their callup to a secret task force called Wing 7 and the security clearance for Operation Origin, described as ‘an undercover mission to investigate an illegal dark web marketplace’. Through the briefing, the audience was made familiar with the task, to collect incriminating evidence on the person behind the criminal business. The live experience started when the audience, upon arriving at the Hoxton Docks venue, was snuck into a field base where Marsha Ingham, a computer engineer and cybercrime expert, played by a live actor, and Logan Flynn, a paramilitary logistics and surveillance specialist, also played by a live actor, introduced evidence that Origin was a fake front for an unregulated dark web marketplace called Emporium. This included a hacked call suggesting that Emporium’s founder, a mysterious figure called Haggledance, was about to go to ground to escape the authorities. The mission was set to break into Origin and collect evidence on Emporium and Haggledance’s real identity before he escaped (see Figure 5). An almost imperceptible interference in the video in which the Haggledance suspect appeared to be incriminating himself disclosed a ghostly apparition, suggesting that there may have been more at stake in Origin than met the eye.

Figure 5

The briefing. Image Factory 42. Photo Seamus Ryan.

After the briefing, the audience was guided to a reception where they were asked to solve a puzzle to access the Origin warehouse. In the security tunnel connecting the reception to the warehouse, they had to solve a new puzzle to bypass an AI-driven security system, part aided by an unexplained presence. By solving another puzzle, they proceeded to the next room where they uncovered several valuable artefacts and scientific objects being sold by Emporium and learnt that Josh Knox, the Origin warehouse CEO, is behind the username Haggledance. Here they could discern the supernatural presence of Gerel Orlowska, a Victorian palaeontologist whose ghost had been raised by the arrival of rare specimen at the warehouse, which she believed to be the one she had discovered in the 1880s. Gerel, interpreted by Jodie Whittaker, whose performance for this role had been captured volumetrically, though she used her own voice, then haunted the goods in the room so as to persuade the audience to proceed to the vault where they could hack into a safe and find her journal. This prompted the audience to move into a new space where they learnt about Gerel’s life and found out that she had taught herself palaeontology from childhood and over the course of multiple digs before finding what she believed to be a new dinosaur specimen. Because her discovery was ignored, she became a restless spirit until she found the bones up for sale in the warehouse. This room was literally formed by the giant pages of the journal (see Figure 6), which meant that the audience was inside the journal, surrounded by interactive fires, energy, and butterflies, who later became an older Gerel seen digging, immersed in the dust of history. Technically, the room was built on a foundation of cutting-edge depth sensing cameras (Intel RealSense), in combination with AI-based human segmentation and pose-detection software (NuiTrack AI) which were extended via the creation of a range of bespoke gesture-detection heuristics, grounded in an understanding of ergonomics and user-centred interactive design. At the end of this section of the experience, the audience was able to strengthen the supernatural energy in the building and bring to life Late Cretaceous era dinosaur ghosts which Gerel believed gave the audience the opportunity to identify her dinosaur bones.

Figure 6

Gerel’s journal. Image Factory 42. Photo Seamus Ryan.

Equipped with Magic Leap headsets, which were said to be para-forensic scanners, the audience then entered a new space called deep time at the Origin distribution room. Here, the cretaceous period had cracked through revealing a room covered in plants and a swamp. The room was populated with a range of dinosaurs including a T-Rex and the new dinosaur Gerel had discovered (see Figure 7). Technically, the set design integrated code within Magic Leap so that the headset could trigger associated physical actions, such as physical objects moving/reacting to the digital assets. By scanning the dinosaurs, audiences thus strengthened the dinosaurs and were able to confirm that the hatched dinosaur was a new species, proving Gerel right.

Figure 7

The Origin warehouse. Marsha Ingram wearing the Magic Leap headset. Image Factory 42. Photo Seamus Ryan.

In the final room, the audience encountered Josh, played by a live actor. Here, the audience was caught in a clash of values between what Josh and Gerel wanted to do with the specimen, resulting in a conflict that raised the question of who has the right to own knowledge. The audience could then decide the ending, choosing whether Josh would be able to escape or not, witnessing the final moments of his freedom on CCTV and herewith also learning from the news that the dinosaur had been recognised as a new species and Gerel finally credited with its discovery.

Lost Origin proved to be a highly complex work, in which the interplay between the physical and digital in both the journal room and in deep time was very innovative and required co-ordination across all departments from set design, to show control, to technical, art development and live action. These where held together by Lost Origin’s executive producer Michelle Feuerlicht, a specialist in interactive narrative production and user experience. Lessons learnt throughout the development may be usedful for the design of future mixed reality theatrical experiences.

Towards the creation of a design framework for mixed reality theatrical experiences

One of the main findings was that lessons learnt throughout the development phase of the project could be useful for the design of future mixed reality theatrical experiences. The multi-disciplinary team who created Lost Origin came from different backgrounds and sectors, including theatre, films/gaming, and coding. The original intention of the project had been to adopt an iterative design practice that was based on the Almeida and Factory 42’s previous experience in developing theatrical and digital experiences and products. Traditionally the management of theatrical experiences follows a linear pattern (rehearsal to production). Subsequently, as theatre productions became more hybrid and used innovative technologies, more iterative models started to be introduced using steps that could be described as loosely based on design frameworks such as Bruce Archer’s 1963 development model which comprises observation (programming and data collection), evaluation (analysis, synthesis and development) and transmission (communication) (Archer 1965). Interestingly, Archer’s model was subsequently built on by Brian Lawson in 1990, who, still following a linear model, added analysis, synthesis, evaluation (The Design Council 2007 and Lawson 1997), recommending a process altogether closer to the iterative processes used nowadays in several production contexts including the original approach for Dinosaurs and Robots. However, Stuart Pugh put forward a design process of iteration, testing and evaluation which is both linear (market, specification, concept design, detail design, manufacture, sell) and circular (analysis, implementation, needs assessment, problem formulation, abstraction, and synthesis) (Pugh 1990). This process, largely due to the impact of COVID-19, was closer to the design approach finally used for Lost Origin, suggesting that for the creation of immersive experiences theatrical and design production models will tend to merge.

Lost Origin’s production process was led by Feuerlicht through a hybrid comprising aspects of agile development, which for her is key to success in any R&D project that aims for innovation. For her any work involving cutting edge technologies should approach the narrative and technical developments so that they allow the best possible use of the technology for the best possible audience-driven experience. To achieve this, she identified the ‘constraints’ pertaining to this project and then proceeded through an iterative development including prototyping and testing which, because of the Audiences of the Future brief, had the question of the audience experience and the identification of new types of audiences at heart. She also used a survey conducted by the team to identify the new audiences and remodelled concept, prototype and audience testing iterations on the basis of audience testing conducted throughout the concepting/development phases. The target audience spanned all age ranges, peaking between 35 and 65, including pioneers and early adopters. Testing at this stage revealed that the audience fund the concept multi-faceted, entertaining, and exciting.

The design approach finally used for Lost Origin builds on the Double Diamond model, which is formed by the four distinct phases Discover (market research, user research, managing information and design research), Define (project development, project management, project sign-off), Develop (multi-disciplinary working, visual management, development methods testing) and Deliver (final testing, approval, launch, evaluation and feedback) (The Design Council 2007), though it has a circular as well as linear development which allows for iterative testing and scaling, which, as Joris Weijdom has suggested, is crucial for the production of theatre in mixed reality (Weijdom 2017: 17). This approach enables designers to gain an understanding of their audience’s engagement throughout the experience. Thus, rather than designing and producing all content at the beginning, a selection of the content (and story) was produced as a rough-cut segment, which could then be experienced in a mock-up of the location to see if the interaction, subject matter and intended designs work together (Reid et al 2005: 7-8). Because of the constraints identified for Lost Origin, the Discover phase was limited while, in her words, the emphasis was on Define (test/iterate), Develop (test/iterate), and Deliver, though the Deliver phase did not in fact deliver a ‘traditional product’ but rather learnings, audience feedback and dissemination to the industry.

Several reports were written just before and during the Audience of the Future programme which influenced the writing of this report, and which had, like Lost Origin, the aim of providing toolkits and frameworks for the design of mixed reality experiences, whether in the theatre, museums, the games, or sport industries (Kidd and McAvoy 2019; Lessiter et al 2018; Ambasna-Jones et al 2018). For example, The UK Creative Immersive Landscape 2020: Business Models in Transition report (2020a) authored by Aki Järvinen published in 2020 included an Immersive Production Matrix dividing the field in distribution, technology and user experience. According to the author, the matrix can be used to look at any immersive production and break down its elements according to these categories. Interestingly, under distribution the matrix indicated several platforms which may be used concurrently or sequentially, also listing several technologies and related user experiences. Järvinen’s second report, The Immersive Audience Journey report, also published in 2020, divided the audience journey into five phases: awareness, consideration, decision, immersion, and loyalty (2020b). In analysing the audience’ journey through immersive experiences, Järvinen suggested that the audience should not be considered as a single cohort as traditional forms of marketing do not communicate the nature of the experience, pointing out also that creators need to invest in the customer journey to assess the audience’s willingness to engage and then re-engage with an experience, thus extending the impacts of the experience ‘beyond the immediate boundaries of the experience itself’ (Ibid.). In terms of design, the principal area of recommendation of his second report highlighted the importance of user testing with a wide range of target audiences not only from an audience experience point of view but also from a stakeholder responsibility point of view and not only at the end but, again, throughout the creation of the experience. The report also noted the value of collecting the long-term impact of a work, which could be useful in terms of building up an appetite for a subsequent version of the experience or new work by the same creators. The final recommendation had to do with the use of merchandising which plays a role in sustaining interest in an experience and creating a brand for its creators.

Both Järvinen’s reports found that audiences are increasingly active and often adopt a range of roles within an experience, something that confirms findings from previous studies such as Steve Benford and Gabriella Giannachi’s 2011 Performing Mixed Reality which showed that audiences can act as participants, onlookers, performers, re-enactors within the same piece and Catherine Allen and Dan Tucker’s 2018 Immersive Content Formats for Future Audiences, which noted that future audiences might be engaged as explorers (of another time or space), perspective shifters (entering another person’s body or experiencing a slice of someone else’s life even if only empathetically), gamers or seekers in a treasure hunt, interactors within data simulators, creators (through the use of immersive media toolkits such as Tilt Brush which lets you paint in 3D) etc. These studies are crucial in that they show that audiences are not only more active in these works, they also are asked to adopt different and even multiple roles, which makes testing even more important to ensure audiences are prepared to engage with all elements of the experience.

In Lost Origin, Feuerlicht introduced several kinds of testing aimed at identifying interest and understanding in the narrative and experience journey; identifying interest in the Magic Leap technology and the possibilities offered by immersion and presence; and testing the varying levels of interactivity and appetite to engage. As is often the case with designing in immersive contexts, for Feuerlicht the trick was to ensure ‘the audience wasn’t alienated by complex tasks, and that they retained full immersion’, maintaining a sense of presence as well as immersion throughout the work. This was especially complex in that audience members might have arrived with different expectations, with some, Feuerlicht noted, looking for an escape room and others wanting a more complex narrative-driven experience. Because it was decided to use the Magic Leap in one room only, so as to avoid problems with calibration, the interactivity was introduced gradually, building up throughout the experience.

The aesthetic forms (theatre and, broadly, intermedia, digital performance) that influenced the design of this project fall within the broad field that Richard Schechner famously defined as environmental theatre in his seminal 1968 text ‘6 Axioms for Environmental Theatre’ (Schechner 1968). Here, Schechner described environmental theatre in relation to a wide spectrum or continuum of theatrical events spanning from public events to traditional theatre, including intermedia and environmental theatre. For Schechner these were characterised by transactions, among performers, members of the audience and between performers and members of the audience, as well as transactions among production element and between these and the performers and the audience. For Schechner, in environmental theatre all space was used for both the audience and the performance. This space could be found or transformed with a focus that could be flexible and variable. He also noted that in environmental theatre all elements of production spoke their own language and pointed out that the text was not the fulcrum of the work, which disassociated it from drama, and other forms of narrative-based performance.

Building on Schechner’s axioms, as well as my previous research with Steve Benford on the design of mixed reality experiences, I propose a design framework using not six, but seven axioms for the design of mixed reality theatrical experiences which encompasses the forms described in Paul Milgram and Furnio Kishino’s continuum and so also includes elements of environmental theatre. By looking at Schechner’s axioms for environmental theatre and my axioms for mixed reality theatrical experiences next to each other it is possible to see how deeply theatre underpins the development of the immersive experience sector. But whereas Schechner’s axioms were in part dramaturgical and in part performance-oriented, the axioms I discuss here are oriented towards experience design, which in turn builds on the field of Human Computer Interaction (HCI) and can be viewed as an extension of interaction design, including experiences which, like Lost Origin, are grounded in a combination of a theatrical setting (however minimal), the physical world, and a digital world, and in which the hybridity of the environment becomes an intrinsic part of the experience. As the environment includes the people who are placed in it, be it performers, audiences, or passers-by, the interactions among those involved in the experience are a fundamental part of the experience design, which is why repeated testing with audiences is so important at all stages of the work.

Steve Benford and I have noted in Performing Mixed Reality how the audience’s experience arc, or, to use Alan Brown and Rebecca Ratzkin’s successful expression ‘arc of engagement’ (2011), is formed by the use of trajectories which often precede and follow the main experience and that may the audience’s presence (or co-presence) within the work. These define the difference between designed paths, experience paths and replay paths, each of which have their own undisputed value within an experience. In fact, documentation (and so replay) can play a significant role in this context not only to create a legacy of an experience, which could be used for preservation, but also to prompt people to revisit an experience to improve on their previous involvement in it or, in the case of very subjective experiences, simply re-encounter the story arc from a different perspective. It is therefore crucial for the design of mixed reality experience that this documentation or archival feature is built within the experience, so as to encourage repeat visiting or episode building and therefore augment the life of the work.

Below is a table with Schechner’s Environmental Theatre axioms (Schechner 1968) on the left and the Mixed Reality Design Framework for Theatrical Experiences on the right.

To conclude, I suggest that these axioms might be understood as the stepping-stones of a framework for mixed reality experiences. The first axiom focuses on audiences, but not on audiences per se, rather on what activities, creativity, and roles, in other words, what transactions, they are prepared to take on during the experience. Audiences do not exist in isolation – they are defined in relation to what they do – they are the performers, spectators, participants, interactors, gamers. Complexity here is key: the storyline should neither be too simple nor too complex. The use of space in mixed reality performance is hybrid, spanning both physical (or to use Schechner’s term, environmental) and digital elements which make it possible for audiences to act in a wide range of more or less hybrid spaces. Too much of a set, and the experience becomes more theatrical, too little, and the theatrical dimension may be lost. The performance event can be convergent and pervasive, which means that the audience’s involvement may start and end before and after the actual experience. Because of the exemplarity of the design of environments involving audience participation, the production of technological, physical and narrative components must occur concurrently, iteratively but also, thinking of our future, sustainably and relationally. Rather than be developed in a linear way, these experiences ought to be developed through the use of a Double Diamond equivalent model considering circular iteration (re-use, re-play). Complex mixed reality theatrical experiences are likely to involve live actors as well as virtual actors and digital dramaturgs and orchestrators who would provide guidance and continuity among different elements of production during the design phase (by testing) and by interacting with audiences during the mixed reality experience. Audiences may need to be guided through these changes in character during the experience. To facilitate audience interaction, narratives will most probably be semi-scaffolded, combining linear (plot based), immersive (gameful and task based), including documentation (replay) and improvisatory (free flow) elements on multiple platforms. Iterative testing with a range of audiences around the scale of the scaffolding will be key to the success of the final experience. Finally, the analysis of the audience’s presence and co-presence will a key factor for the interpretation of the audience experience for it is when the audience feels present (and not only when it feels immersed) that the most rewarding and memorable experiences occur.

These key design findings, combined with the finding that the various creative dimensions of a project can move at varying velocities and rhythms, which means that any underpinning framework for the design of mixed reality theatrical experiences needs to identify from the outset ways of prototyping and testing that can be adapted and scaled right across the development and production period, are most probably just the initial step-stones of a new dramaturgy for mixed reality theatrical experiences.

Competing Interests

The author was part of the research team that worked on the Audience of the Future project Dinosaurs and Robots, funded by UKRI.

Author Information

Gabriella Giannachi is Professor in Performance and New Media at the University of Exeter, UK. She has published a several books including: Virtual Theatres (2004); The Politics of New Media Theatre (2007); Performing Presence, co-authored with Nick Kaye (2011); Performing Mixed Reality, co-authored with Steve Benford (2011); Archaeologies of Presence, co-edited with Michael Shanks and Nick Kaye (2012); Archive Everything (2016 and, in Italian translation, 2021); Histories of Performance Documentation, co-edited with Jonah Westerman (2017); Moving Spaces: Enacting Dance, Performance and the Digital in the Museum, co-edited with Susanne Franco (2021); Documentation as Art, co-edited with Annet Dekker (2022) and Technologies of the Self-Portrait (2022 and in Italian translation, 2023). Her research has been funded by the RCUK, the NLHF, Nesta, UKRI, and the EU.