Introduction

Virtual worlds have, since the early 1990s, been explored by technologists, artists, researchers and niche communities interested in interdisciplinary working environments. There have also been efforts to try and create a discourse and shared vocabulary for these interdisciplinary projects. For example, VR expert Raph Koster, in the early 1990s stated that, ‘a virtual world is a spatially based depiction of a persistent virtual environment, which can be experienced by numerous participants at once, who are represented within the space by avatars’ (Koster 2004). Edward Castronova, expert in synthetic worlds, suggested that a ‘virtual world’ is a ‘crafted place(s) inside computers that are designed to accommodate large numbers of people’ (Castronova 2005), while Bell (2008) claims that the virtual world must consider the individual within that virtual world. The WhoLoDancE: Whole-Body Interaction Learning for Dance Education¹ project is exploring ways in which the virtual environment can augment and contribute to dance learning and expand choreographic techniques. An intention of working with Virtual Reality (VR) technologies is to create new kinds of immersive experiences whereby the dancer can learn independently with a virtual teacher, by being able to dance with a dance expert or even the learner’s own projected image. The dancer may find herself dancing closely with her own avatar, discovering more about the spatial or dynamic properties of her dance by receiving immediate 3-D feedback. Whilst VR is thus proving to be a valuable tool for dance learning, teaching and creating, questions arise about the accessibility and portability of the technology.

To integrate expensive and technically demanding equipment into artists’ working spaces is not always possible. The challenge for the project, and hence the discussion here, is to offer the choreographers and dance students tools that are meaningful, accessible, useful, intuitive and provide them with tools that can advance the field of dance whilst genuinely supporting the needs of dance artists, teachers and learners. Albeit nearly two decades ago, Sparacino et al. (1999) wanted to augment the expressive range of possibilities for performers and stretch the grammar of the traditional arts rather than suggest ways and contexts to replace the embodied performer with a virtual one. Within WhoLoDancE, a similar approach is being taken whereby the tools being developed are not intended to replace the live experience or teacher but rather to enhance and expand current practices and to push the boundaries of the dance art form. The virtual environment and digital tools will thus provide an expanded range of ways to create dance material, which in turn may build on knowledge in the broad area of VR whilst influencing how dance is taught and made, and may gradually change methods of working and collaborating. In short, what we are interested in is how these tools can be used to explore new ground and rethink current practices in the domain of dance.

VR has received a great deal of attention in the last two decades, with the most important information technology and communication (ITC) companies leading the technical evolution of head-mounted displays to offer experiences in Cross Reality (XR), which includes Virtual, Mixed or Augmented Reality. In 2014, Google started releasing the design of a cardboard-made VR visor, which could be used with a smartphone to enjoy a low-cost VR experience. In the same year, Facebook acquired the company Oculus VR. One year later, Microsoft presented the HoloLens device for Augmented/Mixed Reality. In 2016 the Mozilla VR team and Google Chrome team announced the version 1.0 of the WebVR Api, a set of tools to develop VR tools within web pages, and the following year Microsoft updated Windows 10 to support third-party Mixed-Reality devices within the Windows Mixed Reality platform. Apple introduced ARKit in iOS11 to natively support Augmented-Reality applications. Beside these ITC leaders, other companies produced their devices for XR, with the notable mention of the state-of -the-art VR visor, the HTC Vive. Despite the large interest from companies, a clear application to exploit the full potential of VR has not emerged yet. For this reason, researchers, developers and artists are still investigating and experimenting, assisted by the current high level of technological maturity of devices.

VR headsets have been used for watching 360 degree images and videos (see Figure 1), which are videos or images shot with a spherical fashion to allow spectators to choose their viewing angle, as if they were in the centre of the scene. This application provides a low degree of interaction, since it only applies to watching pre-recorded videos or live streams, with some marginal possibility of hypermediality for receiving information on what appears in the screen. An example is the Dutch Opera Ballet’s VR ballet.² The company teamed up with Samhoud Media and produced a VR performance that employed 360 degree stereoscopic cameras and quadraphonic audio. A higher degree of flexibility is provided by the combination of motion capture techniques, animation and computer graphics, that allow the user to digitally acquire the kinematics behind motion and instantaneously render it through a number of scenes and characters. For this reason, in our project we focus on the latter kind of application.

Figure 1

Dancer wearing Microsoft HoloLens Photo: WhoLoDancE.

VR devices can be roughly divided into two categories: desktop-based and mobile-based. The former typology requires a desktop computer to perform the computation of the graphics, and may need a setup for the room in order to track the headset position. The latter are stand-alone viewers usually supported by smartphone devices that are more portable and less expensive, but at the usual cost of lower computational power. Moro et al. (2017) conducted an investigation on the different degree of engagement and effectiveness in student learning when using desktop-based or mobile-based devices, and discussed that the latter are more prone to produce a sense of cybersickness in the viewer. Nevertheless, given the rapid evolution of VR devices, it is likely that the next generation of devices will be able to combine the advantages of both categories. For this reason, in this paper we discuss the elements and questions raised by the applications of VR in choreography and teaching scenarios, regardless of the current VR devices, whilst mentioning the level of computational power or generic specifications of devices where appropriate.

Background and Context to WhoLoDancE Project

In the mid-1990s when dance motiontracking systems were relatively new, performers learned how to play the motiontracking instrument (Dixon 2007). There have been several publications which look closely at the relationship between live performances and digital technologies.³ Since then techniques and the technology itself have developed, which has offered those dancers and artists interested in the relationship between dance and technology new ways of looking at and thinking about their practice. Bailey’s (2007) notion of collaborative research environments could be applied to help frame the WhoLoDancE project working methodology. From inception, WhoLoDancE set out to work with professional dancers interested in the intersections of dance and technology, dance learners, and technologists keen to deepen the discourse on mediated performances. While the development of technology is a key factor of the the project, thinking about how the technology functions is not the primary focus for this paper. More particularly, exploring how the technology enhances the learners’ experiences, the choreographer’s practice and offers new ways for choreographers to think about bodies in real space and also within a virtual space, is at the crux of this discussion, which is grounded in working directly across disciplinary boundaries. The paper also considers the implications for a choreographer when working and within what we are calling a virtual studio.

The collaborative relationships in the WhoLoDancE project has been one based on curiosity in each other’s disciplines and enquiry into what each discipline can offer the other. Identifying terminology used by each of the dance genres that are the focus in the project, which include Greek folk, ballet, contemporary and flamenco, and finding a shared language among the dance partners, was a fundamental component of the first phase of the project. This agreed language was developed from Laban’s Movement Principles and allowed the dance partners to conduct a questionnaire with professional dancers, teachers, practitioners and young dance learners who may or may not have an interest in dance and technology. The aim was to see if the project consortium’s agreed definitions and terminology resonated with practitioners and choreographers from each of the identified genres.

The design of the tools has thus been developed through a dialogic process, taking account of requirements provided by users, partners from the dance community, and the technical possibilities and limitations highlighted by the scientific partners. The tools have since been tested in several demonstration events in several European countries as well as the UK, in order to collect feedback for improving usability. Whilst the tools are expected to be released to the dance community for pedagogical, choreographic and artistic purposes by January 2019, what has emerged thus far, and hence forms the basis for this discussion, are the questions raised by the use of VR in choreography.

As noted earlier, WhoLoDancE is working with four dance genres – flamenco, Greek folk dance, contemporary and ballet. Dance partners who are professional practitioners have served as representatives of specific instances of each genre and were captured using sophisticated motion capture equipment and a large repository of movement has been created. For some, dancing in the motion capture studio was a new experience, finding out about how it felt to wear tight suits with optical markers, performing in a studio surrounded by cameras (see Figures 2, 3 and 4). The cameras captured the markers from several angles. Software processed the information on the location of markers to compute their positions in the 3D space, from which it was possible to obtain the 3D positions and orientations of the limbs of the dancers, such as head, spine, limbs, arms and legs. Motion capture work tends to accurately capture larger sections of the body but it was important in WhoLoDancE to focus on, and document, the smaller gestural motions that are particular to the genres, for example the intricate hand/finger movement in flamenco dance, as in Figure 5.

Figure 2

Greek folk dance motion capture session in Amsterdam, Holland. Copyright: 2016 WhoLoDancE photographer Giulio Bottini.

Figure 3

Contemporary dance motion capture session in Amsterdam, Holland. Copyright: 2016 WhoLoDancE photographer Giulio Bottini.

Figure 4

Ballet dance motion captured in Amsterdam, Holland. Copyright: 2016 WhoLoDancE photographer Giulio Bottini.

Figure 5

Flamenco dance motion captured in Amsterdam, Holland. Copyright: 2016 WhoLoDancE photographer Giulio Bottini.

The aforementioned round of motion-capture sessions generated a significant database and led to the development of the Movement Library, a web-service that users (choreographers, teachers, dancers, students, etc.) are able to navigate to find recordings of different movements across the four dance genres. The Movement Library allows users to annotate each recording with qualities, properties, or actions, through an easy-to-use interface; such annotations are then used to retrieve items in the library. Other tools created to easily navigate the Movement Library are the search by similarity and by movement sketching. The former allows users to choose an excerpt of a recording (a query) and quickly retrieve all the recordings in the Library that are similar to that query according to a user-defined criterion, for instance, similar path through the stage, or similar acceleration of one limb. The latter allows the user to record the query movement, hence sketching a movement, and retrieve the most similar ones in the Library, according to different criteria. While the latter requires users to have some kind of motion-capture device of their own, the former requires users to start with a movement before retrieving the others.

The Movement Library and the Sketching tools may be useful for providing dance students or choreographers with examples of movements. The set of movements also lend themselves to research scenarios as they can enable researchers to investigate similarities among steps across different genres. Through the Blending Engine, a creator tool for choreography, with strategies to combine different movements together both in time and in the virtual space, also allows the user to mix and match and build an avatar that can perform several combinations of ‘steps’ and movement phrases. With regards to the time dimension, the engine blends two movements in order to make the transition between them as smooth as possible. In terms of the space dimension, two or more recordings can be combined in the same moment starting from different limbs to create new steps with, for example, the leg movements of a Greek folk dance and the torso, arms, and head movement from flamenco. Using the Blending Engine, a choreographer may use search by similarity and by movement sketching to retrieve recordings from the Movement Library, and combine them to make a whole new performance. This ability to construct and deconstruct a sequence of steps and movement phrases affords the user an opportunity to choreograph and to work creatively with a body that exists within a virtual environment.

Alongside these tools that are immediately usable for the dance community, other issues have been investigated in WhoLoDancE that are speculative yet pertinent to this discussion. Through formal conversations⁴ between technologists and artists, we aimed at identifying those qualities that are more effective to describe a dance performance and then develop techniques to automatically extract those qualities from a dance recording. These strategies are either manually designed by a technician (rule-based) or developed using machine learning algorithms, which needs to process annotations collected in the Movement Library (data-driven).

Most of the tools in the WhoLoDancE project have been developed for traditional devices, such as computers or mobile phones, on flat screen, but with the intention to bring them eventually to an immersive environment. To turn now to our work specifically with VR technologies, we designed a framework for immersive visualisations of dance performances using low-end head mounted devices, and we developed a web-based tool as a prototype for visualising recordings from the Movement Library or performances created with the Blending Engine. During one of the recording sessions, we experimented with the use of Microsoft HoloLens to watch a recording session as a projection in the physical space (Mixed Reality). Doing this, artists had perspectives of their own body that they may not have experienced before, re-watching their own recorded movements through a different virtual body (an avatar), from every angle.

The artists discovered that the visualisations encouraged them to think differently about the space, the bodies (the avatar and the live body) in space and their creative practice in that space. Seeing different types of avatars in the virtual environment has the potential to prime the user and ask them to think differently about how they moved. When the dancer moved with one of the avatars used within the project, its ability to swell and shrink in response to the dancer was intriguing and allowed for a playful quality and creative exploration of the live and virtual environment (see Figures 6 and 7). In summary, the general feeling was that the visualisations are an integral part of the creative process. These experiences have been particularly important for opening up debate in the project about the body in VR environments and performance (see Figure 8).

Figure 6

Screengrab from WhoLoDancE film⁵ which shows Sarah Whatley wearing the Microsoft HoloLens and dancing flamenco with Rosamaria Cisneros.

Figure 7

Flamenco Dancer wearing HoloLens and dancing with her avatar. Copyright: 2016 WhoLoDancE photographer Giulio Bottini.

Figure 8

Dancers interacting with the Blending Engine tool. The avatar is performing ballet arms and flamenco legs. Photo: Amalia Markatzi.

Thinking of a VR Performance

VR has the ability to shape choreographic formations and patterns that have not been fully explored or even considered before, broadening the possibilities for choreographic investigation. For example, choreographer Laura Kriefman (2014), when discussing her company the Guerilla Dance Project,⁶ suggested that choreographic language is interrogated not for form or content sake but in response to the changing stimuli and physical liberties of the technology itself.⁷ Choreographers may have two main reasons to use VR technology. On the one hand, they may want to exploit immersive digital technologies in their creative process. Being able to watch and re-watch their created performance, at every angle, from any point of the space, would provide insight that is impossible to have when rehearsing and creating a performance in the traditional way. On the other hand, the product of this creation using VR tools may be the performance itself, designed to be watched using VR devices. In this sense, VR is not only a creative tool, but also an artistic medium.

When considering how the VR technology can assist choreographers in their creative process, from the inspiration of the idea, to the production and the final refinements, there are four key elements to be modelled: the virtual performer (or virtual dancer), the virtual environment, the virtual performance and the spectator. Figure 9 provides a representation of the modelling of these four elements and of their relationship as in the WhoLoDancE project.

Figure 9

A representation of the four basic elements in a VR performance. Created by technical partners from Politecnico di Milano.

The virtual dancer emulates the real dancer (in fact, it can be a 3D recording of a real performer) and it is composed of a body, the graphical representation of the dancer, namely the avatar. The virtual dancer acts in a virtual world, which may be a blank 3D space, a 3D model of a real space or any other virtual environment. Using a virtual environment, it is even possible to overcome the limitations of the physical space, which is usually static (it cannot change) or slowly dynamic (it needs a certain amount of time to evolve). An example of the dynamicity introduced by the virtual environment is the possibility of changing the perception of the size of a room from a few square meters to a vast space within milliseconds or to change some properties, such as the shape and the colours according to the performance. This dynamicity is one of the possibilities provided by the digital technology, and it is a part of the virtual performances. Performances that take place in physical locations are commonly enriched with elements such as music, projections or interactive elements. Nevertheless, their implementation may present some issues that virtual technologies can easily overcome, such as the variety of choice or the time required for their actuation. The performance, created by the virtual environment and the virtual dancer, embodies choreographic principles even if expanded through the virtualisation, and it is this that will be watched, or experienced, by a spectator. The spectator may also play an active role in realising the performance, requiring careful design to implement interaction with the VR experience.

Virtual dancer

Steve Dixon (2007) suggests ‘when the body is “transformed,” … into digital environments, it is not an actual transformation of the body, but of the pixilated composition of its recorded or computer generated image’ (2007, 212). As a result of the motion capture process, which produces a transformation of the body as Dixon suggests, we acquire a set of digitalized dancers, which ‘perform’ movements containing information about the different limb position and orientation at each instant, or frame. This is organized in a hierarchical structure that replicates the kinematics of body, as in the body skeleton. This information is then linked to a graphical representation, which is commonly defined as avatar or virtual dancer.

The easiest implementation of an avatar involves reproducing the skeleton structure of the motion-capture recording, where each joint is usually represented by a sphere and a line connects joints to create limbs. This visualisation, similar to a stick man, has been used in the first versions of the WhoLoDancE tools for their simplicity. In some scenarios we have increased the size of the head joint to make it more realistic, as shown in Figure 10. Consortium dance partners in the project found the stick-man avatar useful to reduce the dance to its primitive movements, deprived of the properties of the physical body or the facial expression. An evolution of the stick-man avatar in the project is the arrow-man, with arrows indicating the orientation of the main limbs, with the aim of highlighting the directionality movement principle, as shown in Figure 11. In later versions of our tools we employed two humanoid avatars called snowman and android, the former developed within the project and the latter from a repository of avatars that technologists within the project built and can access. The two avatars are shown in Figures 12 and 13, respectively. Beyond these four examples, there is room for experimenting further with the virtual dancers. Of the avatars that were shown to potential users (choreographers, dance learners, practitioners and artists) interested in the intersections between arts and technology, the feedback was positive suggesting that the avatars could prime the user to think and create in new ways.

Figure 10

A representation of a simple avatar representing the hierarchical skeleton acquired with motion capture (referenced in the text as stick man). Avatar provided by technical partners from Politecnico di Milano.

Figure 11

A representation of the arrow-man avatar, with arrows indicating the orientation of main limbs (head, chest, hips, hands). Avatar provided by technical partners from Motek.

Figure 12

A representation of a humanoid avatar (snowman) developed within the project. Avatar provided by technical partners from Politecnico di Milano.

Figure 13

A representation of an android avatar used for some tool within the project for its higher graphical accuracy. Avatar provided by technical partners from Motek.

Within commercial environments, choreographers have indeed access to a potentially infinite amount of avatars of all kinds, with different degrees of resemblance to a human body, of accuracy and of refinement (Freiknecht and Effelsberg 2017). As the tools in the project are designed to create welcoming environments, which facilitate a form of agency, choreographers, dancers and other users can decide how far away they want to be from the human form of a dancer.

While in the physical world, choreographers work with dancers to fulfil their idea of performance, whereas in VR they are free to tune virtual performers as they desire and test different avatars instantaneously. The avatars offer the user a variety of choices that are not there with the live body. The choreographer in the virtual environment has the ability to choose what types of dance genres or avatars they want to dance or create with, potentially expanding conventional ways of dance making. The avatar is familiar as it mirrors a dancing body but unfamiliar as it performs dance in its own unique way. This virtual personality that the avatar represents, presents the choreographer with new ideas to play with inside this virtual studio.

The freedom to access a virtual dancing body that may perform a dance move in different ways can enhance the choreographic imagination. Brockhoeft et al. (2017, 399) identify several researchers (e.g., Lee and Nevatia 2009; Peursum, Venkatesh, and West 2010; Caillette, Galata, and Howard 2008) who have built systems using commercial cameras that rely heavily on statistical methods and machine learning models to predict the location of a person’s limbs during body movement. Many commercial tools allow users to create human avatars and customise appearance, sizes, shapes, clothes and so on (Freiknecht and Effelsberg 2017). Within WhoLoDancE the user has a choice of the above mentioned avatars and we stayed away from the customization options, which Freiknecht and Effelsberg reference.

However, choreographers are experts in human movement so digital tools that offer the means to work with humanoid or geometric shapes able to inspire the inner semantic of intention or quality of movements are likely to be most appealing. In this regard, Tsampounaris et al. (2016) investigate the impact of different visualisations in dance motion capture. In their work, Tsampounaris et al. let the users choose different avatars, both anthropomorphic and abstract, to test how this could affect dancers’ style and movement qualities. Their work has informed the investigation of different avatars in the WhoLoDancE project, providing alternatives to the initial stick-man avatar. They implemented a tool, named Choreomorphy (El Raheb et al. 2018), aiming at collecting information on users’ preferences or thoughts on different avatars. While their work focuses on the real-time capture for dance education, choreography may also be able to take advantage of the variety of visualisations for dance.

The more accurate and complex an avatar, the higher the demand of computational and graphical power. Hence, choreographers who do not have access to high-end devices may have a limited choice of avatars. Nevertheless, even simple avatars, such as a stick-man representation, can reveal some important technical execution of the performance. Technical limitations may steer the choreographer to use the motions available to them to convey their ideas, possibly resulting in something more explicit or entirely different than they intended. All the four mentioned avatars have low requirements in order to improve accessibility of the tools created within WhoLoDancE, while the avatars created with Choreomorphy are more complex to render.

When there is the possibility to access high-end hardware, other considerations are worth mentioning. When an avatar aims at closely resembling a human figure, there is the risk to induce in the spectator a sentiment of revulsion and fear. Mori et al. (2012) studied this phenomenon in robotics by noting that when the robots’ resemblance to the human body grows, spectators’ sentiment of familiarity grows accordingly up to a certain point when it dramatically drops, and requires a higher amount of resemblance to grow again. For this reason, Mori et al refer to the phenomenon as ‘uncanny valley’ and assume that it may be caused by the resemblance of the robot to a dead body (see Figure 14). McDonnell and Breidt (2010) found evidence that virtual characters rendered with high photorealism are perceived by spectators as less trustworthy than those rendered with lower realism. Taking this into account, the four avatars we selected in the WhoLoDancE project are intentionally abstract and not photorealistic in an attempt to avoid the uncanny valley issue.

Figure 14

A graphical representation of the uncanny valley. Figure from (Mori, MacDorman and Kageki 2012).

Mori et al also note that movement affects and enhances perception, especially when motion looks unfamiliar (for example, in robotics or prosthetic limbs). When movements are acquired through motion capture, they are could be perceived as natural since they are performed by a dancer. With the Blending Engine tool, however, a choreographer can create any new movement from recordings, even some that go beyond the human possibilities, for example with the torso rotated 180 degrees with respect to the limb. Choreographers have, consequently, to pay attention, when preparing the choreography, to the possible distress they may create, while they could choose to use this distress as an emotional trigger for the audience.

The Spectator

We have discussed the design of the virtual space and of the virtual dancers who are going to use this space. In this section we provide an overview of the choices related to how the audience should enjoy, or be part of, the virtual space.

Commonly, dance performances are thought to have an audience watching or participating in them as collective experiences. VR devices are designed to be used by a single user, offering a first-person experience. Hence, in the virtual scenario, the performance could be thought through from an individualistic perspective. In this basic scenario, the user wears a VR headset and attends to the performance unaware of any other people in the audience. This may have the effect of increasing the spectator’s individual engagement with the artistic piece, while maintaining the ‘fourth wall’ between the spectator and the performance. One solution for breaking the wall is to assign a virtual entity to each spectator and make therefore the audience part of the virtual environment. We refer to this entity as the virtual spectator, i.e., an avatar that represents the spectator in whichever form is meaningful in the performance. The use of virtual audience may reduce the intimacy, since each spectator would be aware of the rest of the audience, but conversely increases the engagement of audience -as a collective entity- with the performance.

When a user wears a VR device, he/she is immersed in a virtual, alternative reality yet still living and existing within a real architectural space. As basic features, VR devices show a 3D view of the virtual environment and follow the user’s head rotation and orientation to provide the sense of immersiveness. Another choice of the choreography concerns the degree of freedom attributed to the user to navigate the virtual space. When thinking about their work, choreographers need to select whether to allow users to be in control only of the view given by the head’s rotation (3 Degrees of Freedom, DoF) or also allow them to move through the space, hence including the position (6 DoF). With the former, choreographers need to choose at any instant the position of the audiences’ point of view, similarly to movie directors when placing cameras. If, for example, they want the scene to be watched from above, they will place the audiences’ point of view accordingly. While this is the easiest choice to make if the choreographer wishes to determine the position of the audience, it limits the VR experience. With the latter, instead, spectators are free to move through the virtual space, hence choosing their favourite angle, movement or dancer on whom to focus. The modality of interaction with the space depends on the actual hardware employed. Being able to freely move through real space and be mapped in the virtual space would require a physical space of the same size as the virtual one, and VR devices able to track the users’ position. A cheaper solution is achieved by using external controllers, such as joypads, with buttons and pads to move, as shown in Figure 15.

Figure 15

Spectator watching a VR performance with a joypad that allows them to move around the virtual space. Photo: Amalia Markatzi.

It is important to note the issue of cybersickness when using VR devices and their relationship with movement. Cybersickness occurs due to the mismatch between users’ perceptions of the virtual reality and the real world. As discussed by Behr and colleagues, cybersickness raises ethical questions on conducting research on VR or, in our case, designing VR dance performances (Behr, et al. 2005). Behr et al suggest instructing participants on how to quit the experience in the occurrence of excessive distress. In the scenario of VR performances, cybersickness is related to the design of the interaction with the virtual environment. In this scenario, sudden changes or tilts of the points of view should be avoided, and instead, adhere to steady or slowly moving points of view. The cybersickness is also reduced when the spectators are allowed to interact in the virtual space and are, therefore, in full control of their movements. Nevertheless, the choices made by the choreographers in the design of their performance should aim at minimizing the risk of spectators’ cybersickness.⁸

The audience may also have an active/interactive role in the performance. We described how choreographers can link dance properties to avatars or the environment to enhance the performance, and that audiences may be part of the VR environment. Combining these two aspects, the choreographers may use the audience itself as a trigger in their performance. The choreographer may consider the positions of single spectators, or the audience as a collective entity distributed in the environment. Returning to the example of the performance in a forest, a choreographer may choose to set it at night, with the audience rendered as fireflies that light up the environment. In this way, the audience would be an active part of the performance, and it would also increase the complexity for the choreographer, who now may take the audiences’ possible behaviour into consideration.

VR and the Choreographic Body

The VR tool in WhoLoDancE allows the choreographer time to experiment with new material either from a pre-recorded source or to experiment in real-time using motion capture. Some of the tools in the project have been used by a small group of dancers and choreographers so far who have commented on the unexpected richness and fullness of the information they received when experimenting with the technology. When using the tool, users described being immersed in the environment and becoming aware of the peripheral objects. Sherman and Craig separate mental and physical immersion and describe the latter as ‘bodily entering into a medium’ (2003, 9). However, our work thus far points to how entering a VR environment is both mental and physical immersion. One can mentally engage with the surroundings but also physically respond to positioning and peripheral vision. Moreover, this is similar to the term ‘dilated reality’ coined by Isabelle Choiniere (2017) whereby our sensory perception is recalibrated by being immersed in a VR environment. It opens up space around us and within us.

There is a paradox that surfaces when VR and dance are combined as there remain inherent tensions between the physical and the virtual. Virtual worlds are indeed unreal as they are artificial, fictitious, imaginary, intangible, and sometimes invented. Yet virtual environments are real, as well, as they include the human, live body navigating through the space. The dancer has the ability to move through spatial representations and the physical, embodied reality is part of a faux representation of an environment. VR allows for the unreal and real to be blended together, which fosters a social interaction between the technology and dancing body (see Figure 16). The two can meet in this virtual space and the option of dancing with the avatar allows for the bodies to be suspended in time and space and to co-exist with the real world.

Figure 16

Student interacting with the Choreomorphy tool in Madrid, Spain. Photo: Amalia Markatzi.

Conclusion

Dance is an art form that will always be dependent on body to body transmission. However, there are now many examples of how the transmission of dance knowledge is mediated through technology (see, for example, Bleeker 2017). It is plausible that VR offers a different realm of possibility to create choreography and inform the body. Susan Broadhurst (2006) suggests that at the threshold of the physical and the virtual, tension exists and she proposes a ‘liminal space’ where there is ‘potential for a reconfiguration of creativity and experimentation’ (Broadhurst and Machon 2006, xix). This space is what is now being exploited by choreographers to bring new ideas in to the studio and to expand the making processes of performance.

The integration of VR and interactive technology in dance performances is leading to new insights and experiments with choreographic methods that may ultimately take dance into a new direction. While immersive VR devices are finally starting to spread, immersive VR is only one component of the recent interest in Cross-Reality (XR). Mixed-Reality and Augmented-Reality devices recognize the surrounding external objects and coherently juxtapose them with virtual 3D elements, in an immersive experience or on a 2D screen respectively. It may then be possible to watch a small virtual avatar dancing on our desk from any possible angle, smoothing the boundary between virtual and real entities.

The technical evolution of VR devices is fortunately matched by that of motion capture systems. Due to the requirement of high precision, in WhoLoDancE we acquired dance movements from numerous cameras using high-end Vicon and Qualisys systems. While this cost would be prohibitive for a large part of the dance community, lower-end devices are emerging. The first Microsoft Kinect, in 2010, was crucial to provide the research community with a low-cost device for motion capture through a system of cameras and depth sensors (Jais, Mahayuddin and Arshad 2015). Nowadays, it is possible to record or even real-time capture motion with affordable sensors and suits like the Notch sensors, the Rokoko Smartsuit Pro or the XSENS MVN Awinda (Marin, Blanco and Marin 2017). This is beginning to allow more dancers and choreographers to experiment with these technologies.

Mindful of the need to find ways for the WhoLoDancE tools to be more accessible, our aim is to extend the tools to VR and MR devices. Two of the tools that have more potential in VR are the Blending Engine and the Movement Library. The Blending Engine would allow choreographers to create and rehearse with an avatar in a virtual space, or in their studio using MR, similarly to composers working at their piano. The Movement Library is already a useful resource to access a large database of movements, but with VR it will become an important support for engaging with digital records of dance forms such as Greek folk or flamenco that are otherwise hard to find.

Our work in WhoLoDancE aims to show how dance can benefit from the spread of VR and MR devices. However, there is much work to do before a 3D immersive environment is commonplace for dance learners, teachers, performers and choreographers. But our work in the project is showing us that the dialogues that take place between technologists, dance artists, educators and scholars are very rich and lead to the development of tools that provide new insights to how we choreograph, perform and encounter dancer.

Funding Information

The work was supported by the Research and Innovation Action funded under the EU Horizon 2020 programme.

Competing Interests

The authors have no competing interests to declare.

Author Informations

Karen Wood is Dance Practitioner/Researcher/Educator. Her research has a practical perspective – the training principles of dancers, teaching dance techniques, somatic work – and a theoretical perspective considering embodiment and audiences experiences of dance. She has worked on artistic projects, supported by Arts Council England, collaborating with other art forms, such as fine art, lighting design and music. Orchid ID: 0000-0002-6120-9866, University Profile: https://pureportal.coventry.ac.uk/en/persons/karen-wood.

Rosemary Cisneros is a researcher, dancer, choreographer and curator who works closely with the RomArchive and many NGOs. She is involved in various EU-funded projects which aim to make education accessible to vulnerable groups and ethnic minorities, and part of cultural heritage projects that bring dance and digital technologies together. ORCID ID: https://orcid.org/0000-0002-8169-0642 University Profile: https://pureportal.coventry.ac.uk/en/persons/rosemary-cisneros.

Professor Sarah Whatley is Director of the Centre for Dance Research (C-DaRE) at Coventry University, UK. Her research interests extend to dance and new technologies, intangible cultural heritage, somatic dance practice and pedagogy, dance documentation, and inclusive dance practice; she has published widely on these themes. https://pureportal.coventry.ac.uk/en/persons/sarah-whatley.

Massimiliano Zanoni is a Postdoctoral Researcher in the Image and Sound Processing Group (ISPG), Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, Italy. He received the Ph.D. degree from Politecnico di Milano. His main research interests include music information retrieval, music emotion recognition, ontology-based information management for modeling musical instruments knowledge and feature-based analysis of musical instruments. http://home.deib.polimi.it/zanoni.

Michele Buccoli is a Postdoctoral Researcher in the Image and Sound Processing Group (ISPG), Department of Electronics, Information and Bioengineering, Politecnico di Milano, Italy. His research focuses on automatic semantic description of musical items. Since 2016, he has been investigating techniques for motion analysis and VR visualisations of dance performances. http://home.deib.polimi.it/buccoli.

Augusto Sarti is full professor at the Politecnico di Milano, Italy. He coordinates the activities of the Musical Acoustics Laboratory and the Sound and Music Computing Laboratory. He is an active member of the IEEE Technical Committee on Audio and Acoustics Signal Processing, and the chairman of the EURASIP Special Area Team on Acoustic, Speech and Music Signal Processing. http://home.deib.polimi.it/sarti.