Artworks: from human to software

The artistic field (in the United States as well as in Europe) moved in parallel with this context, becoming one of the many areas where the human-machine relationship was explored. Experimentation laid in a structuralist view, moving from the axiom that ‘creative thinking’ resides in ‘the educated guess or the hunch [that] includes controlled randomness in otherwise orderly thinking’ (McCarthy et al., 2006 [1955]: 14). In the algorithmic works produced in those years, a dichotomy was discernible between those focused on ‘interaction and response, translation across sensory-kinetic modes’ and those ‘generative, exploring aesthetics of permutation’ (Whitelaw 1998). The first case involved the translation of an input into an output (e.g. sound into light), where the randomness came from human action rather than machine processing. Instead, the second case, more akin to the present discussion, involved data reorganised by algorithms according to probabilistic rules: in musical compositions such as Lejaren Hiller’s Illiac Suite (1957) for string quartet, in which the author encoded certain musical parameters (pitch, rhythm, and dynamics) and assigned rules based on, for example, 16th century counterpoint, to create melodic lines (Hiller 1959); in textual works such as Theo Lutz’s Stochastische Texte (1959), in which the software, on the basis of a set of given words, rearranged sequences of logical-grammatical functions according to a certain probability of occurrence (Bajohr 2020); of visual works such as Georg Nees’s Schotter (1968), in which the author placed a row of 12 squares at the top of the painting and had the algorithm generate the rotation and position in subsequent rows according to an increasing disorder (Harmon 2021).

In all these fixed works, the author selected the material to assemble a univocal form, with the generative outputs produced before the exhibition. The random elaboration was strictly controlled according to the so-called ‘generative aesthetics’:

Generative aesthetics… implies a combination of all operations, rules and theorems which can be used deliberately to produce aesthetic states (both distributions and configurations) when applied to a set of material elements. Hence generative aesthetics is analogous to generative grammar, in so far as it helps to formulate the principles of a grammatical schema–realizations of an aesthetic structure. (Bense 1971 [1965]: 57)

The creative process remained anchored in a combinatorial approach with known parameters or styles (Boden 2009: 24–25), whereby machine cognition was not yet taken into account. Therefore, the focus of the so-called generative artworks was not so much on the relationship between biological and artificial beings, but on the exploration of open form at a structural level. The analogy to intelligence arose when machines began to relate to humans in a continuous or contingent way – that is, as a consequence of agency. However, the first attempts were mainly based on deceiving users’ perception by profiting from the social meanings they might project onto actions, appearances and affects (Ekbia 2015; Hofstadter 1995; Natale 2021). Take, for example, the well-known case of ELIZA, a chatbot programmed by Joseph Weizenbaum in 1966 that simulated a Rogersian therapist with whom it was possible to communicate through a command line interface. ELIZA’s script reformulated what users wrote by defining the syntactic function of words and changing their order to generate questions (Weizenbaum 1966). People, unaware of chatting with a software program, reacted as if they were talking to a human (McCorduck 2004). The bot thus acquired a dramatic function as a fictitious simulation of a living being (Pizzo 2011). Although ELIZA made no artistic claim, the interaction with artificial performers would be particularly explored in the coming decades (see Blue Bloodshot Flowers and Prosthetic Head discussed later). Human-machine feedback progressively moved away from closed systems towards multifaceted relationships (Bateson 1972).

Another notable case was AARON, one of the first algorithms to be regarded as symbiotic with the human creative process. The software, based on a wide range of if-then rules, was conceived by painter Harold Cohen and continuously implemented from its birth in 1973 until Cohen died in 2016 (McCorduck 1991; Sundararajan 2021). AARON was able to compose increasingly refined pictures, from the definition of shapes to the use of colours since the 1990s. Cohen himself referred to AARON as his ‘other half’, also stating:

AARON exists; it generates objects that hold their own more than adequately, in human terms, in any gathering of similar, but human-produced, objects, and it does so with a stylistic consistency that reveals an identity as clearly as any human artist’s does… It constitutes an existence proof of the power of machines to do some of the things we had assumed required thought, and which we still suppose would require thought – and creativity, and self-awareness – of a human being. (Cohen 1995: 158)

Here the author speaks explicitly about software intelligence and does so by recontextualising it in light of the specific aesthetic meanings it is capable of producing. In contrast with the works of the 1950s and 1960s mentioned above, AARON ’implied an ongoing and contingent human-machine interaction. Therefore, it was also prone to the attribution of agency as manifested in a ‘reflexive choice… constituted within relationships as they unfold across space and time’ (Burkitt 2015: 15). Both AARON and ELIZA focused not so much on the specific result achieved by the machine, but on the machine itself and its ability to communicate by sharing perceivable outputs. Notably, the software was acquiring its prominence connected not only to statistical processing but to the relationships that processing could establish with humans and, in general, the ecosystem it related with. In this regard AARON and ELIZA emphasised two approaches that would become particularly important in the years to come, as the relationship between machine and human (i.e. author, audience and/or actor) occurred before the staging (as in AARON) or during the play (as in ELIZA).

Artworks: from software to embodiment

Despite the technical developments, live plays with AI remained sporadic and embryonic in those years. This presumably occurred because of the low performance of commonly used processors, the lack of functional graphical interfaces, and the ever-present difficulty in coding. Instead, authors explored audio-visual media, more accessible to non-experts and also related to the ongoing reflection on mass-media (Dixon 2007; Lehmann 2006). Since the late 1980s, object-oriented software has also been consistently developed (Castagna 1997: 40). These consist on programming units called ‘objects’ and were employed for assisted composition and choreography (e.g. Max/MSP, Isadora, TouchDesigner) or for creating virtual environments (e.g. Unity, Unreal Engine, Adobe Flash). Parallelly, hardware platforms for controlling analogue devices (e.g. Arduino, Raspberry Pi) emerged. The relationship with code was thus accompanied by predefined frameworks and graphical interfaces (Mancuso 2018; Manovich 2001), which often enabled real-time processing and expanded the possibilities of exchange between interacting participants (Birringer 2008).

AI technologies entered this context contingently and opened up new creative potentials according to extemporaneous interactivity inherent in digital processing (Monteverdi in press). Interactivity in regard to networked or neural systems can indeed be recognised back in the 1980s, albeit not as an applied model but as a theoretical reference. In his installation Very Nervous System (1983), for example, David Rokeby proposed the live translation of the participants’ movements into sound. The title suggested an analogy between the software and a ‘simplified fragment of human perception, judgment, and expression mechanisms’ (Rokeby 2019: 90) that was able to perceive movements and select an output from a range of possibilities. The reference to neural networks was even more explicit in Tod Machover’s The Brain Opera (1996), which implied a similarity to the principles described in Marvin Minsky’s aforementioned The Society of Mind (Orth 1997). In some sections of the work, material was used that had been provided beforehand or in real time by the audience. These inputs represented the mindless agents that were connected (and thus made ‘intelligent’, to use Minsky’s terminology) by the performance.

In addition to using AI as a conceptual reference – which would also persist in the years to come (Befera 2021; Otto 2019) – authors employed AI as an interactive computational tool engaging with human beings. This was the case in Susan Broadhurst’s Blue Bloodshot Flowers (2001), a performance based on the interaction between an actress and Jeremiah, an AI avatar head in computer graphics projected on a screen. The virtual character was able to see what was in front of it through cameras, recognise the background and foreground, estimate the size and velocity of the figures, and react accordingly through facial expressions – e.g. by showing boredom when nothing was happening (Bowden, Kaewtrakulpong & Lewin 2002). In the first part, the performer related to the avatar according to a predefined script; in the second one, the audience was allowed to interact with the virtual character. The performance thus emerged from the ‘enhancement and reconfiguration of an aesthetic creative potential which consists of the interaction and reaction with a physical body, not an abandonment of that body’ (Broadhurst 2002: 162). The virtual head expressed a specific embodiment by confronting the performer on stage. However, the system was not capable of learning, so the algorithm was not particularly creative and almost bound to predefined reactions.

Stelarc went a small step further in his Prosthetic Head (2002), which was similarly conceived as a responsive 3D head. The avatar resembled the artist and also implied verbal outputs. It reacted to questions written by the participants via a chat room, which in most cases was accessible at the installation venue. The character was provided with lip-syncing, speech synthesis and facial expressions performed in real time, so that the responses seemed realistic. It was also able to compose poems and songs extemporaneously. The software was based on ALICE, an Artificial Intelligence Markup Language (AIML) chatbot that extended and outperformed ELIZA stimulus-response architecture. Indeed, ALICE was able to recognise patterns in dialogues and improve its communication skills through supervised learning, where a botmaster ‘monitors the robot’s conversations and creates new AIML content to make the responses more appropriate, accurate, believable, “human,” or whatever the botmaster intends’ (Wallace 2008: 182). Unlike Jeremiah, the head showed the ability to learn from participants’ inputs, albeit this learning was mediated by a human being. Moreover, the fact that the features of the head, even the skin texture, recalled those of Stelarc reflected the author’s transhuman perspective, in which the body is an ‘inadequate evolutionary architecture that requires additional instrumentation to navigate unexpected temporal and spatial expansions of its operation’ (Denejkina & Stelarc 2015). The head can thus be seen as an extension of the author: in this, a strong similarity can be observed with AARON, which also learned from data manually provided by Cohen, although not having a body and not taking users’ inputs into account.

These two cases involved an interface that was not only text-based (as in ELIZA) and through which the audience could perceive the processing and even interact. Still, the outputs had strong anthropomorphic connotations: whether an extension or an alterity, AI was always based on expectations towards a human behaviour. How Long Does the Subject Linger on the Edge of the Volume… (2005) by Marc Downie, Paul Kaiser and Shelley Eshkar, choreographed by Trisha Brown, on the other hand, featured geometric figures projected onto a transparent scrim in the foreground superimposed on the dancers in the background. The play employed a motion capture system based on cameras and markers on performers’ bodies to make movements and positions detectable. Extemporaneous outputs, called ‘thinking images’ by Downie and Kaiser themselves, were generated during the performance: as the software had ‘its own structures and its own intentions, we set it free to figure things out on its own over the duration of the dance’ (Downie & Kaiser 2005). These figures were based on triangles that moved from right to left on the screen, changing their shape and direction to follow the dancers as they deviated from the given trajectory. The algorithm had a prior memory for the choreography, and the projected figures appeared to continuously improve as the play progressed (Downie 2005). Learning was thus based on domain-specific knowledge which, however, did not involve long-term data storage, but rather a memory that was limited to the piece unfolding.

The symbolic models on which the software in these works was programmed left little room for the virtual characters’ autonomy, as they encoded precisely defined instructions and parameters to be processed. One could say that, although a certain agency was observable, the algorithms did not result in proposing new content, but rather in being responsive to given inputs. The human-machine relationship thus turned out to be mostly fictional. After all, authors were still experimenting with the possibilities of AI at this time, when the large-scale use of intelligent algorithms, their actual ability to learn and the significant influences on the social fabric were still to come. This limited interaction was also accompanied by an anthropomorphic perspective expressed in human-like graphical and robotic entities that extended the anthropocentric analogy between human and artificial cognition. How Long… instead provided a more abstract manifestation of AI that somewhat overshadowed the reference to human beings. The performance thus moved in the direction of a newer aesthetic in which an increasing autonomy of machines could be observed, both in terms of action and representation. As we will see in the next section, the increasing abstraction of outputs brought to the fore not so much the manifestation of the software, but the dynamics by which this manifestation occurred, so that the algorithmic processing gradually emerged.

Artworks: from embodiment to relationship

In the artistic field, such issues might be challenged by aesthetic insights that question pre-existing imagery and explore new creative possibilities. Consider, for example, the interface. In social media, user engagement is categorised and targeted due to their quantified behaviour, and the interface is used for conveying marketing purposes and possibly for the devious pursuit of power (Crawford 2021; Natale 2021). In contrast, the artistic aim of the interface is not to control the audience, but rather to translate one content into another and to foster relationships and communication. Moreover, extemporaneity and interactivity often aim to overcome the technocentric perspective of Western culture (Hui 2021): by reinstating ‘the values of singular experiences and unrepeatable acts’ against ‘abundance and overproduction of goods’ (Berghaus 2005: 260) or by making tangible problems that are part of the algorithms’ employment (Otto 2019). Socio-political aspects might be taken into account implicitly – e.g. in the use of applications such as the aforementioned DALL-E and ChatGPT, pre-trained on unknown data, or software and libraries run by large companies, such as Google’s Colab and Keras – or explicitly – e.g. in the use of well-known platforms or human-machine relations as dramaturgical content.

Insofar as contemporary approaches view AI and enactment ‘as a process rather than as a datum’ (Pizzo 2021: 98), they have more prominently regarded software action beyond its concrete manifestation as a staged object. Authors began to ‘explore other ideas about what AI can do for art’ (Manovich 2022a: 65) and ‘to understand not only how [algorithms] work, but how we work with them’ (Dorsen 2012). The notion of intelligence has increasingly become a legacy of the past, where the focus on processing is regarding not just computation (as in the 1950s and 1960s), but also openness to the alterity and specificity of AI. Ultrachunk (2018), for example, is an improvisational duet between singer Jennifer Walshe and her AI-generated double from Memo Akten (Akten 2018). The musician provided audio-video recordings of improvisation sessions over a year, which were processed by the neural network Granma MagNet to play new content. During the performance, the resulting character was projected onto a screen and its voice outputs interacted with Walshe’s improvisation. AI resulted in an instrument that managed another instrument (the performer’s vocal cords and body) towards new digital outcomes. In contrast with commercial AI, the morphing effect of the images employed a database well known to the user (the singer herself), thus establishing an intrinsic relationship between physical/virtual corporeality and performing attitudes. The result differed from both the anthropomorphic avatars in Blue Bloodshot Flowers and Prosthetic Head: from the former because it represented the action of the singer herself through the algorithm management; from the latter because it involved no prior shaping but was generated entirely through AI learning.

As in Ultrachunk, the extensive data and program refinement required for training has meant that the relationship between author/performer and machine is increasingly symbiotic, especially when the database is built from scratch. The audience thus sees the result of a long-term collaboration which is somehow reflected in the extemporaneous performative result. This dynamic is very different from the one used in AARON, as learning is nowadays autonomous and potentially free from the author’s control. AI also mediates and defines interaction with other entities or realities, categorising the content coming from this otherness and transforming it into something else. Metabolo (2023) by Valerie Tameau in collaboration with Sineglossa involved the interaction between the dancer (Tameau herself) and a marine ecosystem in North Carolina (LaV 2023). The movements of the fauna were tracked via underwater webcams, transmitted to YouTube, stored as motion data and processed by the algorithm to produce sounds that modify an audio track. Tameau moved on these sounds in the guise of the marine deity MamiWata (coming from the tradition of equatorial West Africa from which she descends), also governing an abstract figure projected in the background through motion capture. In this way, she proposed a multi-species relationship anchored in a spiritual context and mediated by AI processing. As in How Long…, the dancer moved together with generative elements, but she did so not according to a predefined script, but to the ever-changing ecosystem. Hence, it was not the AI that followed the human: if anything, it was the human that followed the musical outputs generated by the AI, which in turn depends on the marine fauna on which the concept of the piece is focussed.

The scenic environment is also becoming more and more tailored to processing. As in Cat Royale (2023) by Blast Theory, in which a physical location was specifically designed to provide maximum comfort for three cats according to AI mediation (Blast Theory 2023). The animals stayed in the space for 12 days while their actions were observed and recorded. Parallelly, a robotic arm acted according to their measured degree of happiness by giving them food and making them play. The event left several interpretations open: the dominance of humans over animals, the self-reliance of cats in the absence of humans, or the comparison between the algorithmic monitoring of cats and that of humans in other contexts (such as social media). As Cat Royale was conceived as a performative experiment involving cats and robots, the stage was not structured as a neutral space where AI outputs appeared. Rather, the space itself was designed specifically to enable AI to perform, thus being essential for its processing and activation. In this regard, it recalled the envelope principle mentioned above, as animals and artificial entities were placed in an environment that both influenced and enabled their intermingled behaviour. At the same time, it refers to a constantly monitored space of which its inhabitants are unaware, recalling systems of artificial surveillance.

The ethical background implied in Cat Royale can also be observed in the other pieces mentioned in this section: implicitly in Akten’s Granma MagNet, which plays AI-processed images that are very common in commercial tools; explicitly in Metabolo, where pre-trained motion capture is used to suggest ecological and spiritual meanings. Insofar as extemporaneous interactions are involved, enactment implies less control by the author and a more valuable performative relationship between physical and artificial agents. This unpredictable background, though typical of improvisation and interactive systems, is increasingly related to AI processing, as depending on Big Data, predictive settings and statistical analysis. AI is thus conceived as a medium interconnected to the contemporary social fabric, as it can express the implications of data processing (e.g. predictive models developed in social media) on the one hand, and a more general sensitivity to the relationships within complex ecosystems involving other cultures, species and computational beings (e.g. African culture and marine fauna) on the other. The authors design the software not only to make data perceivable but also to allow the digital entity to develop its own behaviour, expressiveness and social function, reflecting the intertwining between aesthetics and techno-political implications of our time.