The Evolving Oblique: an interactive sound and video installation

The Evolving Oblique is an interactive video and sound installation using multiple gauze screens and projections, which explores the way cinematic perception may incorporate spatial perception, the motor and sensory, vision and movement.

Centred upon the theme of transformative nature, the installation re-contextualizes notions of landscape and architectural form into topology through the embodiment or spatialisation of the moving image.

The condition of topology questions the relation between the viewing subject and perceived environment, transforming vision to a bodily engagement with a dispersed and unfolding spatial field. The movement of the viewer interactively modulates the imagery and sound. The environment becomes a result of the body timing its actions from breath to horizon.

The installation questions how to pass movement on from the virtual to the real, from imagery to structure, from spatial structure to the body and from the body to the virtual: within one conceptual continuum construction, perception and action are combined. This perceptual shift is achieved by creating an immersive environment situated around the concept of liminal and interstitial spatial conditions.

As mutable constructions subject to processes or forces, architecture may no longer speak of the representation of ideas, but rather the performance of the event, and as such it is inextricably linked to duration formed through densities.

Architecture becomes a migrant, expanding the notion of place, space becomes fluid, a hybrid of the physical and virtual domain, founded upon the actualisation of the virtual, rather than realisation of the possible. Once architecture has been redefined as information it can be linked to other environments, creating a new spatial extensibility. Through projection this media becomes substantial.

The integration of real and virtual landscapes

The thematic of the installation has arisen through an engagement of architectural forms and site or location, particularly through notions of situating the body within a landscape, and progression through spatial conditions.

The original inspiration of the project was a marginal territory of marshland. Constantly subject to dynamic tidal and wind forces this shifting landscape is perpetually in flux. An environment characterised by palimpsest, sedimentation and erosion.

In the Evolving Oblique, architectural forms and metaphors of landscape combine to form a notional topology. Operating between field and fragment the virtual imagery produces an environment, integrating the horizon into the curved lines of a vortex to destabilize the perceptual space. Rather than giving the territory or topology singular limits, which are characterised by traditional relationships between figure and ground, these limits are set into motion, by combining figure and ground.

The imagery is composed of animations from 3D models. Tectonically, these animated forms are both skin and structure. Open to variations of transparency and porosity, these dynamic membranes enact reflexive relationships creating forms which oscillate between exposure and enclosure. These mobile forms inflect, rotate, extrude, mutate. The dynamic 3D forms are mapped with video textures from real landscape. The imagery operates somewhere between the virtual and the real, allowing for multiple interpretations.

There is no final goal, end point or culmination, rather a territory that binds to the movement of the viewer. There is not a specific one to one correlation of interaction but a gradual build up or understanding of how the body may relate to form, by progression through the landscape.

Structure

The installation is constructed by 6 semi transparent hung screens and 3 projectors situated at each edge projecting towards the centre. A small video camera is mounted in the ceiling in the central interactive zone. The installation is intended for a single user within the interactive zone.

The triangular layout of the double screens is articulated to form an immersive environment. The notion of a disclosure of space blurs the boundaries between the projected image and structure. The edge of the imagery spills between screens effecting to bind the screens together. The projected surfaces lose their coherence or objectness . Through the juxtaposition of screen imagery notions of figure and ground, scale and orientation are in flux.

The semi- transparent multiple screens allow for an embodiment of the imagery. The use of monochrome induces the black background to disappear and the lighter/white forms appear to float on and between the screens. The screens are not the means to support but lift the imagery, morphing 3 or 4 dimensions of the perspectival screen image. The dialogue between the screens creates forces of withdrawal and disclosure of the projected image, oscillating the perception of the installation space, between an extension or three dimensionality of the imagery and marking surface tensions.

The installation is intended to be viewed from both within the central interactive zone and from the exterior, such that changes in viewpoint allows for different perceptions and compositions of the imagery. And thus the installation operates between object and environment.

Composition of the imagery

The imagery is composed from over 20 complex 3D models which have virtual cameras animated to form paths of movement transgressing through them.

The viewer's movement relates to the movement within the models. The animations consist of 15 second video clips relating to each camera movement, combined to form a sequence through cross fading between each clip. Such that for example a movement to the left will trigger a corresponding pan to the left. Space is invested with a haptic potential as the mobile body relates to points of view within the virtual models.

There are also a series of clips with less complex imagery composed of different forms of horizons and cloud formations which form vistas of exposed topology compared to the enclosed nature of the complex forms.

The complex models are combined in a random sequence, such that the viewer will explore an evolving landscape, and the dialogue between the screens will be constantly transformative. Through the viewers movement a unique spatial trajectory is enacted on the level of presentation through the real- time video processing.

The composition is such that at any time a one screen is designated as the major screen, with a complex model, and two minor screens have the same simpler horizon imagery or exposed vista.

The concept of the interaction and the technical implementation

The installation is an interactive environment that allows the participant to explore a virtual landscape by movement, a navigable body space. Therefore the movement inside the installation should lead to a choreography through this synthetic territory.

Thus the interaction through the intuitive interface allows for a range of movement and dialogue with the environment.

Rather than a process of appropriation of a topographical system, the installation is based on the experience of a dynamic terrain; each moment activates a new relation or condition between the body and imagery through the real-time video processing.

The configuration of the topology is such that allows a nomadic conception of space, such that space is a medium constantly in flux, whereby definition you are the only fixed point, walking not as traversing space but pushing away space beneath your feet. Space only becomes ones skin through walking, being in motion. The evolving topologies constructed are transient, ephemeral structures.

Rather than a one to one relation, action reaction, the interaction is designed such that the viewer gradually understands how their movement may relate to form through the virtual camera movements, not a direct control of the landscape.

By moving in different directions or moving with different speeds, the experience of the landscape varies. Rather than concentrating on positions within space, it is the direction of movement which is key. The same is true for the speed. It is not a result of changing positions in a specific period time but about moving in certain directions in a specific period of time. By movement people change the position of their viewpoint therefore the interaction cannot be directly based on positional data.

If the participant stands still the system slows down the movies and consecutively only blend stills, motivating the participant to move. This means he will not stay at that fixed point in the virtual landscape, but will only see a frozen landscape and the sound, akin to a pulsating texture.

Another motivation for movement and change of orientation is that the main screen is not set to one of the three screens but rather is reassigned randomly over time. There also the system tries to have an impact on the user, to interact with him as the screen environment oscillates.

Technical implementation

The installation is processed by four Macintosh computers. One computer is retrieving the information of the users movement inside the interactive zone (in the following referred as tracking computer) and three other computers are providing the video signals for the three projectors (referred as video players). One computer also generates the sound. The tracking computer has the role as control master and the video players are control slaves.

At the current state the communication between the computers is based on MIDI. The tracking computer sends all necessary information to the video players and the video player with the main movie/ complex animation sends back the information when one landscape is finished and a new one has to be assigned.

The video tracking

The interactivity network operates through Max/MSP-Jitter. The only sensor used is a video camera the signal of which is evaluated by a Max/Jitter-patch. As this installation was designed to adapt to the recipients and not to specific performers one of the main challenges was to implement the tracking in a way that automatically detects people and does not rely on an operator to define the recipient who is to be tracked.

A form of color tracking is used but with an infra-red camera filter and infra-red directional light. Therefore the lighting conditions due to artificial light sources (as the projectors) have no effect on the tracking itself. Other limitations due to camera tracking (for tracking the center of a recognized object) have no significant impact due to the interaction concept of the installation.

The direction of movement and speed to generate control values are utilized rather than the mathematical x-y-values as control inputs.

The video signal from the tracking camera is processed in different ways. The real-time video stream is subtracted from a memorized image of the empty environment. To subtract the images successfully, sharp edges have to be avoided because they lead to visual noise. The movement of the lightweight screens is also considered.

Due to the infra-red light and infra-red filter the contrast of the image from the tracking camera is very low. So the contrast and brightness are adjusted and the image is blurred for smoother edges and the image is subtracted. Any fine visual noise is filtered according to the pixel size. The resulting image is a good source for the tracking algorithm.

From that image, the tracking computer mainly retrieves three values - the direction of movement (left, right, forward/backward), the speed of movement (normal, stop, fast) and the coarse horizontal position (left, center, right). The speed and direction are results of observing the participants' movement over a certain amount of time. Also in this case movement information (based on x-y position changes) rather than simple x-y position calculations are used to evaluate these values. The interaction is designed such that state transitions are smooth, so the tracking does not attach to each and every movement.

The movement data is evaluated and weighted in order that the response to movement is not overly sensitive, to avoid a fragmentation of the environment.

Stand stills and movement must happen in several consecutive frames to change the status of the environment. As speed is a parameter that varies considerably (especially in a limited space where no constant speed can be expected) the speed up is triggered if the speed value crosses a threshold, stays at that state and finally slows down to normal speed again.

The most significant drawback inherited from color- tracking is that the system works best with only one person inside. Multiple persons inside will inevitably lead to a group-bang! With more than one person the interaction is no longer related to the movement of the recipients as individuals, but as a result of their group behavior. This does obviously not affect the functionality of the installation but adapts the concept of a solitude experience of the landscape.

Video streaming

The video players provide the imagery for projection. To keep the environment interactive the video sequence is composed of short movie clips. Each video player is always playing two clips, constantly cross-fading. At either side of the cross-fade the invisible movie gets exchanged, according to the movement of the user. For this process several databases are evaluated to determine the next movie.

According to the speed information sent by the tracking computer the video players gradually speed the movies up, or gradually slow them down and stop them. As higher speed means moving faster through the landscape the speed of the cross-fade and the playback rate of the movies are modulated. This means that the same information is provided in a shorter period of time as is the case if running through a real terrain.

At the end of one landscape, the video-player with the main focus sends back a notification to the tracking computer informing it that the focus can be re-assigned after the current movies. The tracking computer then determines a new focus randomly and sends information about the next landscape to the three computers, according to their status.

Audio

The imagery is supported by spatialized interactive sound. The soundscape is more underlying and contra-punctual towards the video scenery. It also has different themes which are subtle variations of each other. The audio uses four-channel audio-files as the sound source. This material is processed in different ways. The speed of the movement affects the speed of the audio playback. The horizontal position as well as the video cross-fade modulates the spatial distribution of the sound.

The sound and imagery in conjunction with the fluid screens allows the participant to experience the installation as a fragile rather than dominating environment.

The Evolving Oblique was shown as part of IAMAS Ogaki Biennale 2004, and produced at IAMAS, in Japan during a research visit by Joanna Walker. Joanna Walker produced the concept and animation, Steffen Bluemm collaborated on the interactive programming, Bill Haslett designed the interactive sound.

BIOGRAPHY

Originally trained as an architect, Joanna Walker is now working in the field of interactive video installation and practising as a media artist. After several years in architectural practice in San Francisco, and London, she completed a Masters in architecture and the Moving Image at the Digital Studio, Department of Architecture, University of Cambridge. She has exhibited photographic and design works in the U.K, U.S and Japan. Her recent installation Evolving Oblique was exhibited as part of IAMAS Ogaki Biennale 2004 in Japan. She has published and critiqued in the field of architectural design and new media, at the University of Greenwich and Cambridge. Currently Joanna is in the final year of a PhD (funded by the ARHB) at the Digital Studios in Cambridge and has recently completed a 6 month residency at IAMAS in Japan with Professor Christa Sommerer.