PROCEEDINGS
28th International Symposium
on Electronic Art
MAY 16-21,
PARIS

Summary

Presentation

The 28th edition of ISEA, co-organised by Le Cube Garges and the École des Arts Décoratifs  PSL, took place in Paris at the Forum des images from 16 to 21 May 2023, as well as through an artistic programme in some 50 partner cultural venues throughout France in spring-summer 2023. The videos of which are online here.

Following an international call for proposals, more than 1400 submissions from 70 countries were evaluated by an artistic and scientific committee of 200 international experts. The theme of this edition was Symbiosis. In this time of global health, ecological, economic and democratic crisis, symbiosis is a polysemous notion that allows us to explore in a transversal and interdisciplinary way the mutations and transformations underway in the digital age, to question the meaning supposedly given to progress, especially in the current environmental and health context, and to imagine possible and viable futures for our planet and our ecosystems. 

The symposium catalog, including the symposium program and the artistic program, is available as a downloadable pdf here.

Figures

+ 1 500 participants at Symposium conferences and exhibitions at the Forum des images 
+ 700 participants in online conferences during Symposium week
+ 7 000 visitors to the Augmented Catalogue in the first 15 days
+ 1400 proposals 
+ 660 academic proposals
+740 artistic proposals
+ 70 applicant countries
+ 50 cultural partner venues

Co-organizers

ISEA2023 was co-organised by Le Cube Garges, executive producer, École des Arts Décoratifs  PSL, academic programming director, and the Forum des images, Symposium venue. École des Arts Décoratifs  PSL is a public institution of higher education under the authority of the Ministry of Culture, whose mission is the artistic, scientific and technical training of artists and designers, as well as the development of research creation activities. pioneer in thie field of collaborations between art, design and sciences, it created a research laboratory, EnsadLab in 2007. Within the University Paris Sciences & Lettres (PSL), towhich it is affiliated, it set up the PSL SACRe (Science Art Creation Research) doctorate program in 2012. It is the first practice-based doctorate to be set up within an art and design school in France.

Le Cube Garges is an interdisciplinary and digital cultural innovation center with six facilities covering 8,000 m². Focused on creative renewal, it combines the discovery of new artistic forms, inclusive practices, digital training and interdisciplinary reflection on societal issues.

SYMBIOSIS,
THE MAIN THEME 

Symbiosis is essential to life. No living species can survive without cooperation, let alone humans with each other and with their environments. More than a simple coexistence, symbiosis induces interdependence. It can be positive, neutral and sometimes negative in certain cases of parasitism. From these interrelations can emerge new hybrid entities, like the lichen which is a compound of algae and fungi. The first providing photosynthesis and the second moisture, one benefits from the contribution of the other. Our becoming lichen would be in a way the cross-cutting question underlying the symposium. Understood in a broad acceptance, the symbiosis is also played out at the scale of the symposium: it is a transdisciplinary event between creation and research (visual arts, theater, music, design, cinema, human and social sciences, engineering sciences , natural sciences) and intersectoral (art, crafts, industry, research, education, cultural innovation).

THE SUB-THEMES

The call for proposals for the symposium was divided into three main sub-themes: symbiotic individuations, organizations and imaginaries. It was a starting point to structure the call without limiting the scientific concept of symbiosis, and to encourage cross-cutting proposals outside disciplinary and technical classifications.

The selected interventions thus made it possible to discover works that call on symbiosis as a scientific concept, but above all projects that used it as a metaphor or more simply as an inspiration. It was an invitation to discover in a new way, both serious and curious, projects closely associating creation, research and technologies.

Symbiotic individuations

Which type of “hybrid beings” are coming into existence today, and thereby associating the human and the non-human? What new forms will they take, and will they be the next inhabitants of our planet? Experiments on trans-humanism, the emergence of new artificial autonomies, and new representations of natural elements were examples of the exploratory perspectives that were to be presented and discussed.

Symbiotic organizations

Human societies have founded their equilibrium upon multitudinous and interdependent forms of symbiosis across a wide swathe of domains: ecology, economy, culture, technology, and more. What happens when a given event destroys this fragile equilibrium? Researchers, artists, designers, and leading experts offered their perspectives on these resolutely transdisciplinary approaches, with the shared ambition of contributing to a world that is more resilient and at ease in its alterity.

Symbiotic imaginaries

In order to exist and prosper, symbiotic relationships—relationships in which each partner derives reciprocal benefits indispensable to their survival—are essential. Symbiosis is a vital necessity at all levels of all known ecosystems. But “necessary” does not imply that it should be reduced to its pragmatism or instrumentality. Symbiosis also opens us to new imaginaries, be they novel or redundant in the context of their corresponding era. If modes of collective imagination vary according to perspectives, they are always multiple and in dialogue with reality—what, therefore, constitutes the symbiotic imaginations of today and how and why invent new ones? 

THE “CLUSTERS”

Given the large number of conferences, roundtables and various oral presentations, the academic chairs proposed clusters to enable the symposium audience to compose its own cross-cutting program to complement the Symbiosis themes. This gave meaning to the general programme. 

The papers grouped together in each of these clusters could be classified differently and often appeared in several other clusters. These groupings are therefore retained here, with each cluster listed in alphabetical order (authors). Each paper is colour-coded to indicate the sub-theme to which the author(s) responded (sub-theme reminder), together with the cluster in which it was scheduled for the symposium audience and any other clusters to which it might have been attached. A final glossary by sub-theme and cluster is provided to offer readers several entry points into this important volume. These classifications in no way restrict the understanding of the papers, they are simply reading aids.

Abstract of all interventions are available on the ISEA2023 website and each of them is accessible to the widest possible audience: https://isea2023.isea-international.org/en/symposium-program


“L’étrange labo microcosmique des Oumpalous” exhibition at Le Cube Garges, artwork from Anne Marie Maes, Installation Exoskeleton: Topography of a Second Skin, for ISEA2023 © Nadia Rabhi

Academic Chairs and International Programming Committee

Academic Chairs

The Academic Chairs were responsible for the final programming of the symposium based on the evaluations of the IPC. The names and biopics of the Academic chair and of the three co-chairs are published below.

The International Programming Committee (IPC) was composed of 140 internationally renowned experts. The Committee evaluated the 1400 proposals in a double-blind process for full & short papers, panels and roundtable discussions. The names of the IPC are listed below.

Dr Emmanuel Mahé — ISEA2023 Academic Chair

Dr (HDR) Emmanuel Mahé (France), Director of Research at École nationale supérieure des Arts Décoratifs (PSL), researcher in information and communication sciences dealing with issues related to organisations and innovation processes in the fields of arts, sciences and design; scientific officer at the High Council for the Evaluation of Research and Higher Education.
Website of Emmanuel Mahé

Emmanuel Mahé © DR

Dr Elena Papadaki — ISEA2023 sub-topics co-chair

Dr Elena Papadaki (UK/Belgium) is a researcher, educator and curator; her research interests lie in the intersection of technology-reliant artworks, curatorial discourse, interdisciplinary collaboration and audience reception.

She leads the MA Design at University of Greenwich and currently serves as a Pathway Councillor for the Royal Society of Arts (RSA), acting as a key advisor and expert to the delivery group “Design for Life–Students for Change”.
Website of Elena Papadaki

Elena Papadaki ©DR

Dr Maria Ptqk — ISEA2023 sub-topics co-chair

Dr Maria Ptqk (Spain) is a curator, researcher and cultural producer. With an academic background in law and economy, she holds a PhD in artistic research with a doctoral thesis on the cyberfeminist collective subRosa. Her work investigates the ecological and technoscientific realm from a postcultural perspective, in the crossroads of experimental art practices and collective in/trans/disciplinary knowledge.
Website of Maria Ptqk

Maria Ptqk © DR

Pr François-Joseph Lapointe — ISEA2023 sub-topics co-chair

Prof. François-Joseph Lapointe (Quebec) is an artist and scientist at the University of Montreal, holding two doctorates (in evolutionary biology and in dance and performance). He has published over 120 articles in the field of molecular systematics, population genetics and metagenomics. As part of his artistic practice, he applies biotechnology for creative purposes and has created choreogenetics.
Website of François-Joseph Lapointe

Pr François-Joseph Lapointe 
© DR

International Programming Committee

Aceves Sepulveda Gabriela – Université Simon Fraser (CA)

Albert Jeffrey – Université Loyola de la Nouvelle-Orléans (FR)

Almas Almir – Université de São Paulo (BR)

Anders Peter – Kayvala (États-Unis)

Badani Pat – Conseil d'administration de l'ISEA (États-Unis)

Baker Camille – Royal College of Art, Londres (Royaume-Uni)

Basserau Jean-François – Ecole des Arts Décoratifs  PSL (FR)

Behar Armand – Centre de Recherche en Design – ENSCI-Les Ateliers ENS Paris-Saclay (FR)

Bellotto Janet – Université Zayed (AE)

Berenguer Josep-Manuel – Sonoscop (ES)

Beyls Peter – Collège universitaire de Gand Belgique (BE)

Bianchini Samuel – Ecole des Arts Décoratifs  PSL (FR)

Bibasse Juliette – commissaire indépendante / HACNUM (BE)

Bradbury Victoria – Université de Caroline du Nord Asheville (États-Unis)

Breuleux Yan – NAD-UQAC (CA)

Brucker-Cohen Jonah – Lehman College / City University of New York (États-Unis)

Buiani Roberta – University of Toronto (CA)

Bunt Brogan – University of Wollongong (AU)

Cardoso Amílcar – Instituto Politécnico do Cávado e do Ave (ESD-IPCA) (PT)

Cattan Pierre – Small Bang (FR)

Century Michael – Rensselaer Polytechnic Institute (US)

Cermak-Sassenrath Daniel – ITU (DK)

Cevallos Gabriel – Kino Beat Festival (BR)

Champion Erik – University of South Australia (AU)

Chevrier Joël – Université Grenoble Alpes (UGA) (FR)

Citton Yves – Université Paris 8 (CH – FR)

Costa Pederson Claudia – Wichita State University (US)

Cozzolino Francesca – École des Arts Décoratifs  PSL (FR)

Cruz Filipa – Paris College of Art (FR)

Cunin Dominique – École des Arts Décoratifs  PSL (FR)

Czegledy Nina – OCAD University Toronto, University of Toronto (CA)

Dahan Kevin – De Montfort University (UK)

Dal Farra Ricardo – Board ISEA International – Concordia University (AR/CA)

Dallet Jean-Marie – EAS : École des arts de la Sorbonne (FR)

de Campo Alberto – UdK Berlin (DE)

de la Merced Melissa – University of the Philippines Film Institute (PH)

Dedack Camille – HACNUM (FR)

Dika Penesta – University of Art and Design in Linz / Department of Interface Cultures (AT)

Doyle Judith – OCAD University (CA)

Drew Jesse – Griffith University (AU)

Dulic Aleksandra – University of British Columbia (CA)

Eigenfeldt Arne – Simon Fraser University (CA)

España Keller Juliana – Concordia University (CA)

Farias Priscila – University of São Paulo (BR)

Ferraiolo Angela – Sarah Lawrence College (US)

Fischer André – MixBrasil Festival (BR)

Fitzgerald Scott – New York University (US)

Flemming Peter – Concordia University (CA)

Forouzandeh Milad – D A H project (IR)

Fox Tyler – University of Washington (US)

Fratzeskou Eugenia – Leonardo ISAST / The MIT Press (LABS) (UK)

Freire Manuelle – Concordia (CA)

Fröschl Martina – University of Applied Arts Vienna (AT)

Fuchs Natalia – ARTYPICAL (RU)

Gay Jean Jacques – Acces-s cutures électronique / Citu paragraphe P8 (FR)

Ghasemi Amirali – New Media Society, Tehran (IR)

Girousse Clara – ARCAN (FR)

Gollifer Sue – University of Brigton (UK)

Grilo Carlos – Instituto Politécnico de Leiria (PT)

Grisoni Laurent – University of Lille (FR)

Haute Lucile – École des Arts Décoratifs  PSL and Université de Nîmes (FR)

Hosale Mark-David – York University, Computational Arts (CA)

Hunter WhiteFeather – The University of Western Australia/ SymbioticA (AU)

Hwaryoung Jinsil – Texas A&M University (US)

Jarmick Martin – California Lutheran University (US)

Jego Jean-François – Université Paris 8 (FR)

Jim Alice – Concordia University (CA)

Kaminska Aleksandra – Université de Montréal (CA)

Kim Jinku – The University of Nebraska – Lincoln (US)

Klein Tobias – School of Creative Media, City University of Hong Kong (CN)

Law Vanissa – Silly Savvy Studio (UK)

Le Malet Carine – Leonardo/Olats (FR)

Lebon Marianne – Hangars Numériques / HACNUM (FR)

Lehérissier Imane – Chateau éphémère (FR)

Leonard Neil – Berklee College of Music (US)

Levillain Florent – École des Arts Décoratifs  PSL (FR)

Levy Ellen – Independent Artist (US)

Lioret Alain – Université Paris 8 (FR)

Londono Felipe – Universidad Jorge Tadeo Lozano (CO)

Ludovico Alessandro – Winchester School of Art, University of Southampton (UK)

Mahé Emmanuel – École des Arts Décoratifs  PSL (FR)

Makris Nadine – AADN (FR)

Manton Coral – Bath Spa University (UK)

Mariátegui José-Carlos – Alta Tecnología Andina (PE)

Martin Patrick – Massey University, Whiti o Rehua School of Art (NZ)

Martins Pedro – University of Coimbra (PT)

Matuck Artur – University of São Paulo (BR)

Maviel Sonet Alexandrine – Amcsti (FR)

Miller Bill – University of Wisconsin Whitewater (US)

Mitchell Bonniel – Bowling Green State University (US)

Montfort Nick – Massachusetts Institute of Technology (US)

Moore Jake – University of Saskatchewan (CA)

Moren Lisa – UMBC (US)

Mosse Aurélie – Ecole des Arts Décoratifs  PSL (FR)

Nacher Anna – Institute of Audiovisual Arts Faculty of Management and Social Communication Jagiellonian University (PL)

Nam Hye-Yeon – Louisiana State University (US)

Naveau Manuela – University of Arts Linz / Department Interface Cultures (AT)

Nevin Antony – Wellington School of Design, College of Creative Arts / Massey University (NZ)

Nijholt Anton – University of Twente (NL)

Odonoghue Diarmuid – Maynooth University (IE)

Olynyk Patricia – Washington University in St. Louis (US)

Ong Joel – York university (CA)

Palumbo Michael – York University (CA)

Paquin Louis-Claude – Université du Québec à Montréal (CA)

Parker Jennifer – University of California Santa Cruz (US)

Pasquier Philippe – Simon Fraser University (CA)

Paul Christiane – Whitney Museum / The New School (US)

Pérard Hervé – Siana (FR)

Perez-Bobadilla Mariana – Waag Society (MX)

Polli Andrea – The University of New Mexico (US)

Prokopow Michael – OCAD University (CA)

Reyes Everardo – Université Paris 8 Vincennes Saint-Denis (MX/FR)

Ribeiro Clarissa – Roy Ascott Studio / SIVA DeTao Shanghai / Art|Sci Collective (UCLA Art|Sci Center and Lab, US) (BR)

Robine Michèle – Oplineprize (FR)

Rosen Margit – ZKM | Center for Art and Media (DE)

Rosero Contreras Paul – USFQ (EC)

Rowe Robert – New York University (US)

Salter Chris – Zurich University of the Arts (CH)

Samson Audrey – Goldsmiths, University of London (UK)

Schefler Pierre – ARCAN / HACNUM (FR)

Schuh Diane – Université Paris 8 (FR)

Scott Jill – ZHdK (CH)

Shadid Reem – Independent curator and researcher (IL)

Shahrokh Naz – Zayed University (UAE)

Shaik Yifat – York University (CA)

Slayton Joel – Independent Artist and Curator (US)

Sosa Andrea – National University of La Plata (AR)

Szabo Victoria – Duke University (US)

Tornero Paz – University of Granada (ES)

Tsoupikova Daria – University of Illinois Chicago (US)

Valencia-Tobon Alejandro – Cucusonic Collective (CO)

Valentina Peri – Independent Curator (FR)

Van der Plas Wim – ISEA International (NL)

Vatavu Radu-Daniel – Stefan cel Mare University of Suceava (RO)

Vergara Erandy – ISEA International (MX/CA)

Vilca Cecilia – MyAP – Microscopía Electrónica y Aplicaciones en el Perú (PE)

Warnke Martin – Leuphana University Lüneburg (DE)

Wei Jo – CAFA (CN)

Young Jiayi – Université de Californie, Davis (États-Unis)

Zhang Ga – Académie Centrale des Beaux-Arts, Pékin (CN)

Academic and Artistic Program

At the same time as the academic call for artists and researchers, a specific call for projects was launched, under the artistic responsibility of Cube-Garges. This selection has been enhanced by the networked artistic program (50 venues) offers thematic tours around symbiosis in a network of partners in Paris and in France, during several weeks. It irrigated the territory and thus brought to ISEA2023 both an anchoring in various places and an opening to diversity. The public discovered a wide variety of installations, virtual or augmented realities, performances, concerts, video games, design, films or digital books. The tours is also available on a dedicated online platform here or on our website.

POSTERS

 

WORKSHOPS

 

ARTWORKS

 

ARTIST TALKS

 

PANELS

ISEA International

ISEA International was proud to present the 28th edition of our annual symposia in Paris (France), with an academic symposium and exhibition predicated as an experimentation incubator driven by the unifying theme of“Symbiosis”– an essential consideration for the interdependent gathering of the year 2023.

We joined in celebrating ISEA’s return to Paris after 23 years since ISEA2000 “Révélation”, with Le Cube Garges as executive producer and responsible for the artistic program, and École des Arts Décoratifs  PSL responsible for the symposium’s academic program. As in the previous iteration in Paris, the academic symposium was held in the heart of the city at Forum des images.

Founded in the Netherlands in 1990, ISEA International (formerly Inter-Society for the Electronic Arts) is an international non-profit organisation fostering interdisciplinary academic discourse and exchange among culturally diverse organisations and individuals working with art, science and technology.

The main activity of ISEA International is the annual International Symposium on Electronic Art (ISEA). The first symposia took place in 1988.

 

Tributes

Two major figures in art and culture left us in 2023: Peter Weibel, whose tribute was paid during the Symposium with the entire ISEA community gathered in the large amphitheatre of the Forum des Images, a tribute led by Pr Roger Malina which elicited many speeches from the audience; and Vera Molnár, who died a few months after the Symposium and to whom we pay tribute in these proceedings.

Disremembering Peter Weibel: Reminiscences are more Interesting than Fact

Peter Weibel was a leading figure in the new media art scene. Since 1999, he had been the director of the ZKM in Karlsruhe and had carried out numerous curatorial activities (Ars Electronica, Venice Biennale, etc.). Among his most important exhibitions, we can also mention his collaborations with Bruno Latour, who also passed away recently ("Iconoclash" in 2022, "Making Things Public" in 2005 and "Critical Zones. Horizonte einer neuen Erdpolitik" in 2020). The sudden departure of Peter Weibel in March 2023 was a shock for the ISEA communities therefore a tribute was paid to him at this years’ symposium. A lecture was given by Roger Malina, followed by a session lasting over an hour in which Malina asked the large audience to share memories of Peter Weibel. Many very moving and interesting words for the ISEA community were expressed. Many deeply moving words and recollections were expressed by members of the ISEA community.

Roger F. Malina © Nadia Rabhi

Tribute to Vera Molnár

Vera Molnár dans son atelier © Vincent Baby

Among the numerous reactions following the passing of Vera Molnár at the age of 99, a pioneering artist in geometric abstraction and generative art, the National Institute of Art History (INHA) in Paris paid tribute to her.  The ISEA Symbiosis committee also wished to honor this iconic figure by transcribing the text from the INHA here.

Vera Molnár “had established deep ties with the INHA, which were manifested through the donation of a significant collection of prints in 2022, where the pioneering and generous dimension of her work fully reflected her personality. Her passing leaves a huge void in the art world, where she was still actively engaging with young creators who came to visit her. Vera Molnár was born in 1924 in Budapest, where she received a classical education in Fine Arts and graduated as a professor of art history and aesthetics in 1947. That same year, she left Hungary and settled in Paris. Quickly choosing her artistic family in geometric abstraction, during the 1950s she was a radical constructivist who refused to label what she considered, with the theoretical support of her husband François, a scientist and psycho-physiologist, as "art," viewing it instead as experiments. In this spirit, she invented an "imaginary machine" in 1959, a proto-computational conceptual process that involved giving precise composition instructions followed by systematic execution, akin to a machine. Taking a significant step in 1968, she began using a computer to assist in creating artworks based on programs and algorithms, becoming a pioneering artist in France in generative drawing. Curious about all materials and mediums, she drew, glued, painted, sculpted, programmed, photographed, created installations, artist books called "livrimages"[the French contraction of the words ''book'' and ''image''], and an impressive intimate diary (1976-2020) in twenty-two volumes of nearly 5000 pages. Whenever possible, Vera Molnár also created prints. Trained in Fine Arts, she learned wood and linoleum engraving at a young age but later collaborated with professionals to produce numerous publications. In 2022, she decided to assemble a significant collection to donate to the INHA library, consisting of 264 prints, 2 portfolios, and 12 preparatory models.

This donation forms an exceptional collection, representing the entirety of the artist's work, unique in French and foreign public collections. Her works belong to formal families that she worked on, revisited, and reinvested in for several decades. The Sainte-Victoire Blues directly reference Cézanne's motif, drawn and painted hundreds of times by Vera Molnár. Lettres de ma mère echo the computer modeling of the artist's mother's handwriting, who wrote to her from from Budapest, which over the years became increasingly jagged at the end of each line, described by the artist as "a bit gothic, somewhat hysterical, "yet providing her with a sensation of "visual bombardment" each time. 4 carrés rouges, Brèches3 triades 3 couleurs, Ordinateur miroir de la main, Hypertransformations hommage à Klimt, and many other series explore the paradigmatic form of geometric abstraction: the square. Cut, shifted, overlapped, joined, pushed, covered, giving rise to other shapes, incisions, triangles, quadrangles, perhaps letters, the squares disassemble, suggesting a figure of balance or generating a perceptual indecision, leaving it to each individual to contemplate and decide for themselves.

Upon news of her passing, INHA seeks to perpetuate the living legacy of an immense creator; that of an adventurer of shape, of order and disorder, who used and challenged tradition. Behind Vera Molnár's programs, algorithms, and systems lie thought and ideas, as well as tenderness and amusement; in short, life. Today, we bid farewell to an artist generative in every sense."

With the kind permission of the authors, INHA's director, Éric de Chassey, the director of its library, Jérôme Bessière, and Vincent Baby, project manager at INHA, close friend and Vera Molnár specialist.

Symposium Proceedings

Introduction

SYMBIOSIS –
ISEA2023 PROCEEDINGS INTRODUCTION

FOR A SYMBIOTIC APPROACH AT ALL SCALES

THE NEED FOR INSTITUTIONAL RECOGNITION OF PRACTICE-BASED RESEARCH (‘RECHERCHE-CRÉATION’)

KEY STRENGTHS

1. ORGANISATIONAL FORMS ARE ALSO CREATIONS

2. CRITIQUE OF THE OLD BINARITIES THROUGH PRACTICE-BASED RESEARCH

3. EMERGING DE/SUBJECTIVATION PROCESSES

4. GENERIC TOOLS DESIGNED AND SHARED BY COMMUNITIES OF PRACTICE

5. BREAKING DOWN DISCIPLINARY BARRIERS AND THE EMERGENCE OF NEW RESEARCHER PROFILES: EVALUATION AS A TOOL FOR LEGITIMIZATION

6. PERCEIVE BEYOND OUR USUAL CAPACITIES TO BECOME AWARE OF SOCIETAL AND ENVIRONMENTAL TRANSFORMATIONS

7. WHEN CREATIVE RESEARCH BECOMES CRITICAL ACTION-RESEARCH

CONCLUSION 

For a symbiotic approach at all scales

Of the 660 proposals received for the academic call (and a total of 1,400 including the call for artistic projects), from over 70 countries, 230 were selected, including 99 short and long papers, for the Symposium held at the Forum des Images, in the heart of Paris.  200 international researchers and experts evaluated these applications in a double-blind process. More than 50 global organizations, including schools, universities, research bodies and associations, were selected to present their research activities and facilitate exchanges among them. In addition, more than 50 partner venues, such as art and design schools, galleries and institutes, offered a rich complementary program of exhibitions, performances and debates on a national scale. This mobilization of a range of contributors, evaluators, and a public of several thousand spectators once again attested to the vitality of ISEA, an annual event initiated more than 30 years ago by pioneers.

The central theme of the 2023 edition was Symbiosis, as demonstrated by the 99 previously unpublished papers in these proceedings. The full program, including these papers and all the other symposium formats - artist talks, panels, workshops, tutorials, posters and demos - can be consulted online. This program was in itself a form of symbiosis.

But what types of symbiosis are we talking about exactly? How do they manifest themselves? Going beyond the strict definition of the term and interpreting it as an operative metaphor, how can we envisage and put into practice a genuine ‘symbiotic’ approach? The aim of the call for entries was to encourage the reconsideration of any creation, experimentation or research through this prism, even if symbiosis was not necessarily the main theme. ISEA2023 offered a sort of ‘core sample’ at a given moment of the different strata of contemporary creation and research underway in the world.

Our idea was that the symposium's programming should also serve as a collective manifesto, promoting the fundamental principle of symbiosis to be respected at all times and places: a lasting and mutually beneficial association between organizations and actors, whether human, non-human or hybrid. Considered in a broad sense, symbiosis also manifested itself at the level of the symposium itself: it was a transdisciplinary event combining creation and research (in the fields of visual arts, theatre, music, design, film, human and social sciences, engineering sciences, natural sciences) and cross-sector (involving art, craft, industry, research, education and cultural innovation).

Before highlighting what we consider to be some of the strengths of this particularly rich body of work, in terms of both knowledge creation and innovation, it is important to mention our commitment to promoting practice-based research in all its forms and formats. This approach fosters the invention of modes of cooperation between different areas of expertise and enables the emergence of hybrid researcher profiles. Symbiosis does not lie in the homogenization of its components, but rather in the creation of a hybrid and symbiotic entity, where each element is essential to the other in devising new ways of existing. Together, they offer innovative ways of perceiving, understanding and acting on the world.

The need for institutional recognition of practice-based research (‘recherche-création’) 

It often happens that artist-researchers or designer-researchers are valued for their creative abilities, while at the same time being perceived as illegitimate in their research skills. Indeed, there is a perception that one cannot be both a researcher, publishing scientific articles for example, and a creator, exhibiting one’s work. This perception differs from country to country. In countries like Canada, for example, this is not a problem. In France, for example, despite recent progress, not all research institutions are yet clear about this. Yet these publications and exhibitions (‘publicizations’) both stem from the same practice-based research process. New generations of researchers are challenging this perception, and ISEA is a testament to this, since the event enables creators and researchers to share their research results in an international A-rank publication that meets the standards of peer-reviewed scientific journals. Institutional structures have also become aware of this reality, as shown by the doctoral programs specializing in practice-based research which, through their official recognition, mark a fundamental evolution both in the academic world (universities, art and design schools) and in the professional sector, and not only in the cultural field (health, transport, housing, ecology, etc.). However, we can only encourage research and teaching institutions to take this major development into consideration, by proposing, for example in countries where it does not yet exist, the status of lecturer, associate or full professor (‘enseignant-chercheur’ in France) in art and design schools.

The older generations, pioneers who are still very active, were also present at ISEA2023. In Paris, for example, the pioneers of digital arts active since the 1980s rubbed shoulders with doctoral students just beginning their research careers. This cross-generational aspect is rare enough to be highlighted and celebrated.

Key strengths

As Academic Chair and co-Chairs, we propose here to highlight what we see as the strategic trends running through all the papers. This collective work reflects our different disciplinary fields, countries and educational backgrounds. Our aim is not to summarize such a rich corpus, but to propose several common threads to encourage more in-depth reading of the articles and, of course, to contradict or consolidate our proposals.

Although this exercise is limited and subjective, partial and biased, it is nonetheless enriching. It reveals that, despite the diversity of the proposals, it is possible to find convergence points, despite the divergences, and even, sometimes, diametrically opposed approaches. The theme of symbiosis encourages us to look not for what unifies, but rather for what the different perspectives or visions proposed, even if they take different forms and are based on heterogeneous conceptual frameworks, can generate together. After all, symbiosis thrives on difference.

1. Organisational forms are also creations

Faced with the urgent environmental and democratic challenges of our time, the development of new collaborative methods is essential. Creative research is an innovative methodological response, offering an effective means of strengthening and better coordinating interdisciplinary dynamics. It opens the way to a new era of organizations and individuals working together in symbiosis. Although the approaches may vary, the common objective is clear: to give increasing importance to research within institutions and to recognize the vital role of associative networks and companies involved in this process.

Organizations, whether stable or changing, are study entities in their own right. Namely, E. Armand, A. Asensio and D. Foresta encourage the creation of new cooperative ecosystems between the arts and the technosciences, underlining the importance of collaborative innovation. Transforming organizations from within is also seen as a promising way forward. Y. Hu, C. Chou and Y. Kakehi, for example, are calling for the creation of laboratories within art centers, as they have done by setting up a ‘controlled environment space’ at the heart of an exhibition. Y. Breuleux, A. Thibault and R. Lapierre, for their part, advocate the idea of occupying and reappropriating industrial wastelands, which are often included in digital art festivals, to give them a new lease of life.

Organizational architecture also extends to the digital domain. V. Hilsberg looks at the use of DAO (Decentralized Autonomous Organization) based on blockchain, arguing that this technology has a direct influence on artistic creation through distribution methods, perhaps recalling Marshall McLuhan’s famous adage: ‘the medium is the message’. This specific focus on the medium or media is a key feature of ISEA.

Print media have always been and remain essential to the evolution of academic structures. C. Frémontier-Murphy traces the history of the journal Leonardo, published by MIT since 1966, which became an international benchmark by promoting a then nascent field of research. More recently, S. Bianchini and G. Lallemand have broken new ground by launching the online academic journal ‘.able’, with the support of more than 30 international partner institutions and the Arts and Sciences Chair at Polytechnique, EnsAD and the Daniel and Nina Carasso Foundation. This new image-based medium does not seek to compete with the written word, but rather emphasizes the importance of the visuals generated by practice-based research projects, referring to them as ‘publicizations’. This approach is based on the fundamental idea that to publish is to make public. In the academic context, this means subjecting new knowledge, whether textual or visual, to the critical scrutiny of peer communities.

When Leonardo was created, organizational innovation was also underway in South America. J.-C. Mariategui recalls the birth of the Centre for Art and Communication (CAyC), a Latin American network dedicated to art and cybernetics. Today, international collaboration between researchers has become the norm, as shown by the example of SEADS (Space Ecologies Art and Design), a transdisciplinary and multicultural collective on a global scale.

Sometimes developments are the result of unforeseen circumstances, sometimes predictable but poorly anticipated by public authorities. Recently, the Covid-19 pandemic forced cultural institutions to reinvent themselves. N. Ricci and M. Agogué share the lessons learnt by the Montréal festivals, illustrating the resilience and adaptability of organizations in the face of global challenges. These organizational issues also have their counterparts in the production of works. For example, W. Zhang and S. Su make visible the ‘voids’ between individuals subjected to the social distancing put in place during the Covid-19 pandemic.

All these examples illustrate the growing interest in the creation of new organizational forms in various fields. The concept of symbiosis originated in the natural sciences. By borrowing the vocabulary of these sciences, it is possible to suggest that all these organizational forms together give rise to new ecosystems. The term ‘ecosystem’ was proposed by Arthur George Tansley in 1935 in the journal Ecology. The concept then gradually migrated into the human and social sciences, particularly economics. Today, it is often used to designate a complex environment that encourages creation and innovation. Widely adopted by those involved in entrepreneurial innovation (makers, startupers, entrepreneurs, etc.), this metaphor of scientific origin can also be applied to art and design research.

These ecosystems, through their interrelationships, are contributing to the emergence of a renewed ‘biosphere’, forming a vast practice-based research network on a global scale. This network can be seen as a favourable climate for invention, a phenomenon that Nowotny, Scott and Gibbons had already described in their analysis of knowledge production in contemporary society in 2001. Thus, this organizational and creative landscape can be compared to a dynamic biosphere, where innovation emerges from interaction and collaboration, as Wenger, McDermott and Snyder emphasized in 2002 in their study of communities of practice. This is the case in Québec, but not everywhere. Like any ecosystem, the equilibrium found is fragile and subject to forces that can destabilize it, as in the case of Symbiotica, which is now threatened with extinction, even though it is a laboratory with an international reputation for its work in bioart. Ionat Zurr, an artist-researcher and co-director of this structure, spoke about this during her keynote speech. She also appealed to our community for help, which we are relaying here.

Each new organizational arrangement is both the product and the result of a process of transformation which also operates within conceptual frameworks. It is not surprising, therefore, that these frameworks are themselves put under strain by these arrangements, which ‘reorganize’ - or, depending on one’s point of view, disorganize - traditional ways of thinking.

2. Critique of the old binarities through practice-based research

The questioning of the traditional dichotomy between culture and nature is now shared by many in the academic community, particularly in anthropology. Artistic works embody and make perceptible this profound reconfiguration of paradigms of thought. N. Georgakopoulou, D. Zamplaras, S. Kourkoulakou and C.-Y. Chen explore a ‘sympoetic’ interaction with materials in interactive art, inviting us to question old dichotomies such as subject/object, mind/body, and nature/culture, in order to embrace a new understanding of materiality. P. Purg and K. Pranjić oppose a human-centered science, putting forward an ‘emergent symbiotic mutualism’ that transcends mere species coexistence, from a posthuman perspective. M. Zolotova, for her part, sees the posthuman turn in many contemporary discourses as a serious attempt to move beyond anthropocentrism. Such approaches are therefore not identical, and can even sometimes be seen as opposed, but they all involve a critique of analyses centered solely on the human.

This criticism can be found in other fields too. For example, Bakke questions biocentrism and focuses on the interactions between minerals and non-human life forms. Adopting an approach of working ‘with’ rather than ‘on’ is another way of challenging these human-centered biases. M. da Rocha Montanari uses generative art as a collaborative research methodology with indigenous Guarani and Kaiowá communities, co-creating a ‘poetics of attention’ that draws on sounds and plants in their cosmology. A. Wollensak, B. Terry and B. Baird present extracts from community testimonies relating to water, accompanied by processed environmental sounds from Alaska.

For R. Wright and S. Howden, the adoption of indigenous perspectives is crucial to developing ‘sentient machines’ capable of reflecting the complexity of ecological patterns, where computational ecology fails. C. Schnugg, D. Brill and C. Stary merge aesthetics, performance and digitalization to reinterpret the world, going beyond the rational production of knowledge. Although these initiatives are diverse, they share a common goal: by hybridizing different rationalities and subjectivities, they seek to transcend existing approaches in order to invent new ways of understanding and addressing contemporary crises, both geopolitical and climatic.

3. Emerging de/subjectivation processes

These methodological and epistemological reflections are accompanied by a transformation in the ways in which we conceive of human ‘identity’, the conceptual as well as practical ‘making’ of the way in which we define ourselves, or not, as human subjects. Indeed, the processes by which individuals construct themselves as subjects (a process of subjectivation), or sometimes deconstruct themselves (a process of de/subjectivation), are present in many practice-based research projects. For example, the performance by T. Chiaravalloti and A. Meshi’s, explores a symbiotic relationship between humans and an artificial intelligence algorithm, enabling participants to experience a ‘collective consciousness’. Although this work aims to highlight the limits of AI’s ability to recognize emotions, it also opens the way to reflections on future developments in this field and the ethical dilemmas they raise. P. Gemeinboeck and Rob Saunders, for their part, propose ‘hybrid entanglements’ between humans and robots, based on bodily empathy between humans and non-humans. S. Cîrcu and C.Y. Chen explore movement to question the anthropomorphism of humanoids through a mimetic dance between a human and a robotic arm.

These approaches use translation, imitation, differentiation and empathy as means of criticizing, but also realizing, the hybridization between machines and humans as formalized by a number of artists and theorists at the Symposium: M. Smigielska uses AI to take account of spectators’ behaviour, E. Amato & E. Péreny claim a ‘co-individuation’ with our technological avatars, J. Bae and his VR comic strip explore the relationship between humans and AI, D. Batsis, M. Grigoriadou, or with K. ElRaheb and A. Politis, who advocate the body as a phenomenon and an extension of ‘hyperreality’.

Even the most fiercely critical creative research into intrusive technologies is unwillingly participating in this general trend. C. Brunner and J. Fritsch look at the transhumanist movement through the prism of Gilbert Simondon’s thinking, in particular his notion of ‘human energetics’, and are inspired by a video work by the Nigerian artist Otobong Nkanga. G. J. Shin, using the concept of the ‘superjet’ in a post-digital context, highlights the emergence of new creative human subjects, capable in her view of weaving new social relationships.

This critical exploration of the way in which humans conceptualize and experience themselves also manifests itself in a decentering of the human, highlighting the processes of subjectivation at work in the ‘non-human’ or ‘other-than-human’ world. Adopting a Bruno Latour perspective, elements such as forests and water are no longer simply reduced to objects of study for humans; they are seen as actors in their own right, subjects in the active sense of the term. In this vein, G. Trudel, who studies climate change, integrates data from tree communities, the media arts and forest sciences. She attributes a ‘voice’ to trees through what she calls ‘individuations of symbiotic modulations’, drawing on and extrapolating the work of Gilbert Simondon.

Approaches to thinking about and experimenting with symbioses between humans and machines, as well as between non-human entities, vary widely. However, they all point to a probable reconfiguration of the general cartography of subjectivation processes: the contemporary human seems ready to move away from the central and dominant position it has hitherto attributed to itself. At least, that is what the observations and discussions at the Symposium seem to indicate. The question is what this will mean in the long run, and whether the promise of emancipation will really be fulfilled.

4. Generic tools designed and shared by communities of practice

As the notions of objects and subjects evolve, research tools and methodologies are also changing. This transformation is contributing to the emergence of new practices and methods for ‘measuring’ phenomena that were previously difficult to quantify. The evolution of instruments and their impact in the field of creative research are particularly interesting. Just as there are creators of works of art, there are also innovators in the organization and design of instruments. Artists, designers, scientists and engineers frequently collaborate to provide research and creation with platforms and tools that are not only at the service of creation, but that also stimulate innovation among those who use them.

For example, MARCEL (Multimedia Art Research Centres and Electronic Laboratories), of which the pioneer D. Foresta is one of the designers and promoters, offers an online collaborative platform dedicated to the specific formats of the interactive arts, a project that was born in 1997 and, even then, was the continuation of a process of reflection and practice that began in 1981. This desire to offer shared tools is obviously not a recent development and is now reflected in a wide range of proposals, from generalist platforms to tools dedicated to specific fields or uses. S. Thurow, D. Del Favero, M. Scott-Mitchell, M. Ostwald and H. Grehan have developed a network modelling system which facilitates the planning of opera rehearsals thanks to an interactive ‘cyber-physical’ spatial aesthetic, opening the way to new imaginaries. C. Lengelé and P.-A. Gauthier traced the genealogy of ‘Live 4 Live’, an open-source tool designed to simplify the creation and real-time control of spatialized sound objects. D. Cunin, E. Durand and M. Seta developed generic devices to facilitate the creation of immersive environments with video tracking and collective interactivity on smartphones.

This trend towards developing platforms for artists and designers is also to be found in the field of curation. Y. Hofmann, through the ‘intelligent.museum’ project, has put forward a system based on data analysis for artistic and cultural institutions. This prototype uses data from a variety of sensors in exhibition spaces to tailor museum experiences to visitors’ needs. I. Družetić-Vogel, A. Fuchte and M. Bauernfeind proposed ‘ARt chat’, an application for museums that combines augmented reality, art and communication, allowing visitors to share their impressions in the exhibition space.

These initiatives are not just about providing functional services; they can also take a critical approach. V. Guljajeva, M. C. Sola and I. J. Clarke, for example, have analyzed several existing tools in the field of AI, stressing that ‘AI does not create a work at the touch of a button, but requires a deep understanding of the underlying technology, as well as a creative and critical approach’.

In addition to the creation of platforms, there is also a trend towards modelling, although approaches vary considerably. The very term model indicates a desire to provide communities with frameworks based on general principles to be adopted. Z. Wu, D. Fei, X. Ma, M. Fan and K., for example, see NFT as ‘a sustainable business model for media arts, involving audience interaction’. Others, such as D. Xu, M. H. Lamers and E. van der Heide, propose a ‘relational model of co-located interaction’ between spectators and an interactive work.

These various proposals, although heterogeneous, share a common view: they point to a constant trend towards the uninterrupted emergence of specialized communities of practice that design generic devices for instrumental purposes. These communities are also expanding thanks to the instruments they are inventing, enabling their practices to be disseminated more or less widely, a form of ‘standardization’ with a creative aim, while at the same time encouraging the integration of their own approaches to uses in the fields of art, often including the active participation of audiences as a central component of the design of the works.

5. Breaking down disciplinary barriers and the emergence of new researcher profiles: evaluation as a tool for legitimization

ISEA stimulates multidisciplinary exchanges while consolidating the existence of a specific community through annual gatherings. The enthusiasm for creation, in synergy with technology and science, forges a solid link between researchers and creators from a variety of backgrounds. This positive dynamic encourages the exchange of diverse perspectives, whether practical, theoretical or methodological, around a common subject, a specific field, or more simply a medium or instrument. The cross-disciplinary aspect of these interactions is particularly enriching, as it helps to create a diverse community while respecting the disciplinary affiliations of each individual.

Adopting a double-blind peer review process, ISEA’s call for applications conforms to academic standards and promotes cross-disciplinary or multidisciplinary approaches. In an academic environment that is often segmented by discipline, this recognition is crucial. The symposium thus provides a sanctuary for unclassifiable minds, while giving them international exposure. The evaluation criteria we have chosen enable artist-researchers and designer-researchers to assert their legitimacy.

The criticism often levelled at the latter - that the dilution of the boundaries between the arts and sciences would jeopardize the quality of research and creation - ignores the symbiotic nature of these two spheres of creation. In fact, they are mutually nourishing, while retaining their specificity, whether through collective work or individual skills, not forgetting the importance of the distinct ecosystems to which these players belong. Although this does not systematically guarantee the relevance of the results, whether scientific or artistic, this approach is like that practiced in professional art and science circles, where peer judgement is commonplace.

ISEA’s aim is to bring these hybrid profiles to the fore. Although cross-disciplinarity can be perceived as a risk of levelling out skills, it is essential for innovating, experimenting with new methods and tackling issues from a fresh angle. The contribution of these cross-disciplinary approaches can be extremely beneficial to established disciplines and sectors, enabling them to evolve and renew their own fields of research. Venturing outside our disciplines through creative research allows us to rediscover them in a new light.

If practice-based research can create new knowledge in science and engineering, it also brings new practices to the field of creation. Y. Blanchi, for example, links cognitive science and architecture, while S. Gatz criticises architecture as a means of control by proposing a ‘cosmo-techno-poiesis’. Urbanism as it exists is questioned by artistic and digital practices, like the infrastructures of networked urban screens imagined and implemented by M. Thorogood, K. McCulloch and A. Dulic.

In dance, N.N. Correia, D. Souza, I. Nêves and J. Lobato are trying to make the dance process understandable within a performance using a multisensory approach. I. Teles de Castro e Costa, H. T. Hong and C. Y. Chen have designed a performance co-created with an autonomous virtual system. In the field of cinema, R. Sagot-Duvauroux, N. Quaetaert, F. Garnier and R. Ronfard explore how film editing cannot be transposed into virtual environments and must be reinvented. Questions about the transposition of creative techniques into VR also arise in the case of Chinese calligraphy with Shum, P.Y.S. & Klein. C. Salter, T. Thomasson and P. Uro are carrying out what they call a ‘theatrical exploration’ based on augmented reality, with the aim of making climate change visible.

The ‘adjustment’ between different vocabularies is necessary to give an account of this research. O. Kobryn, M. Couteau, R. Sagot-Duvauroux, S. Balcon, F. Garnier, R. Ronfard and G. Soulez, researchers from different disciplines (aesthetics, ergonomics, design, engineering, cognitive sciences), are working to reconsider the concept of the ‘virtual’.

In the cultural sector, curation and conservation are also invested with methods derived from practice-based research to redefine or apprehend their activities in a new way. M. Jones, M. Wester and M. Blottiere present a new approach to curating as practice-based research and propose the acronym ‘CRC’ to establish it as a specific field. D. Irrgang and M. Skłodowska take up the curatorial notion of the ‘exhibition of thought’ co-invented by Bruno Latour and Peter Weibel, who have put it into practice with curators, artists and researchers at the ZKM.

The circulation of thoughts and practices, the mutual acculturation of different research players and cross-disciplinary cooperation are all expressions that would remain abstract without these numerous experiments. These compositional arrangements, to paraphrase Latour, make us see differently.

6. Perceive beyond our usual capacities to become aware of societal and environmental transformations

Making perceptible what naturally escapes our senses is not a new concept in the world of art, or even in that of design, as the example of decasualization shows. This approach is even less novel in the field of science, where modifying our perception and understanding of the world forms the basis of a great deal of research. Nevertheless, the way in which this is done varies from one field to another, with a current tendency to merge different approaches in order to make discoveries or visions of the world that are not immediately accessible - often because of their scale - ‘sensitive’, i.e. perceptible to the senses.

In this context, many research and creation projects aim to reveal new sonic and visual ‘landscapes’, constituting what might be called an informational aesthetic. There are two main thrusts: the first seeks to materialise technological artefacts, which are increasingly miniaturized and interconnected; the second aims to raise awareness of climate change through the collection of biological or geophysical data, with the two thrusts sometimes overlapping.

In the first category, H. Scurto and A. Chemla-Romeu-Santos generate soundscapes from data collected in 28 locations around the world via the Locustream online platform, highlighting AI infrastructures. On the other hand, M. Lukaszuk and S. Bahng exploit gesture and spatiality through electroacoustic and video improvisation to make intelligible the preponderant role of machine learning.

In the second, Mc Farlane proposes a method of co-creating artistic acoustic ecologies centered on the Great Lakes in the USA, combining sound data collection and research methodologies enriched by indigenous knowledge.

In terms of sonification, L. Foo and J. Fritsch create a ‘cryogenic landscape’ to engage listeners emotionally with climate change, broadcasting the movement and melting of the ice in Greenland almost in real time, presenting the ice pack as a living being. J.C. Duarte Regino goes a step further by making atmospheric processes audible, helping us to perceive our atmosphere beyond our usual senses.

At the intersection of the two, L. Moren and T. Bachvaroff's ‘Under the Bay’ is an augmented reality project that turns smartphones into ‘microscopes’ to explore marine life invisible to the naked eye. J. Ottavi and J. Pickett are exploring the interaction between humidity and the decomposition of printed circuits, converting soil acidity and electronic compost into electrical signals.

Visibility projects can also illustrate disappearance, a radical type of invisibility. H. Sareen, Y. Fu and Y. Kakehi depict endangered species as micro-bubbles formed by the nucleation of CO2 in water, which shrink and disappear in a few days to raise awareness of the threat to biodiversity. B. Ammar-Khodja brings to life the residues of heavy metals in contaminated urban landscapes, highlighting a kind of point of no return if the pollution persists. G. Legrand, V. Thornhill, I. Clarke, through the installation ‘Spectral Plain’, explore how information technologies can integrate and reflect specific socio-cultural beliefs, creating new symbiotic interrelationships with the non-human and the non-living, thus influencing the participants' perception of their understanding of the world.

These practice-based research projects aim to make complex issues or abstract realities tangible. Designed to transform our perspective, they encourage us to adopt different behaviours and develop new critical approaches.

7. When creative research becomes critical action-research

N. Georgakopoulou and her collaborators aspire to create a ‘sonic consciousness’ within urban spaces. Their project aims to create musical experiences that invite active participation from citizens, while paying special attention to people with hearing and visual impairments. For his part, R. Ridgway strongly criticizes the collection of personal data by the main players in the network. In the same vein, B. Gaylor and K. Hennessy encourage the development of a critical understanding of Big Data to better counter the ‘piracy of affect’.

From a techno-feminist perspective, G. Lautenschlaeger explores post-human maternity in media art. P. Costa, L. Ribas and M. Carvalhais analyze the tendency to feminize digital assistants and suggest strategies for reversing this trend. L. Haute urges us to think about our methods of collaborating with the environment, inspired by the symbiotic culture of bacteria and yeast. C. Gould and Isabel Stenger urge us to take action to initiate change in our lifestyles and consumption habits, while J. Garančs explores, through virtual reality, dystopias based on market values.

Y. Zhang, in his interactive installation, uses machine learning to generate fictional speeches by Xi Jinping and Donald J. Trump. P.R. İ. Yeginsu highlights the need for artists to find new ways of collaborating to avoid the negative impacts of commercialization linked to NFTs. The fight against social prejudice is also a recurring theme, with S. Brueckner, S. Yeung, J. Liu, D. Choberka, K. Shedden, J. Turner, I. Gillet, M. Lu and Xingwen Wei identifying the biases present in face detection and race classification.

Faced with the environmental crisis, techno-solutionism is criticized through the surrealist and ‘pataphysical’ approach of M.-L. Bourgeois’s project, without systematically rejecting engineering, as many of the projects are situated at the intersection of the arts and the engineering sciences. The appeal to various forms of decolonization is also marked, namely with H. Rashtian and G. Aceves-Sepulveda who question the construction of historical narratives.

These contributions, which oscillate between denunciation and calls for resistance, aim to provide the public with critical, conceptual, sensory and emotional tools to raise awareness of the issues of pollution, gender and democracy. The subjects addressed are varied, and the methods proposed are diverse. Although critics of these works may fear that activism will predominate over rigorous research, it is clear that these contributions, by their provocative nature, offer food for thought and relevant tools for the scientific community, inviting ongoing debate among peers.

From this perspective, this type of action research fulfills a fundamental dual critical function: on the one hand, it is devoted to exploring the invisible, hidden, unexpressed, or even oppressed elements within society; on the other hand, it simultaneously engages in the struggle against doxas, including those with which it finds itself unwittingly associated. This bifocal approach is relevant to all fields of study, encompassing both research and creative work with a critical intent. This approach enables a clear distinction between activist practices and research activities, thereby amplifying their respective impacts through an enriching critical symbiosis.

Conclusion

As we have emphasized, the choice of these key strengths represents a challenge, due to its necessarily limited and subjective nature. Not all the authors are cited in this introduction, due to lack of space, and we can only encourage readers to read all the articles. We could have highlighted other aspects, such as the importance attached to feminism, gender issues, the ecological transition, or the various processes of decolonization. These are essential thematic entries, but they are already very present in current debates, and we offer additional reading keys. It was also possible to group the proposals according to the type of technologies used or invented, or by sector of activity. The thematic clusters we proposed during the Symposium could have encouraged us to do this. However, we have chosen to base ourselves on all the contributions published in these proceedings and to resume the programmatic work in ‘bottom up’ mode. This approach offers a new perspective on how everyone can benefit from this work for their own research.

We hope that these guidelines illustrate, if proof were needed, that these proceedings are a rich source of ideas, methods and formats that are both varied and innovative. It is also important to note that the proposals go beyond the scope of these proceedings, and we encourage you to explore the online video archive platform, including artist presentations, demos and roundtables.

These exchanges between art, design, science and technology are essential for generating knowledge and creating meaningful works. They allow us not only to analyze the present (a form of real-time analysis), but also to lay the foundations for future discoveries and new approaches that are still being developed. These initiatives can be seen as precursors, offering a glimpse or intuition of what is emerging and could materialize in the next ten to twenty years. In a way that complements science fiction, which imagines the future, these proposals are starting points for extrapolating the present and considering possible future developments. A work in constant evolution!

Paris, London, Bilbao, Montréal,

Emmanuel Mahé (École des Arts Décoratifs  PSL University, France, EU), Academic Chair

Elena Papadaki (University of Greenwich, UK), sub-thematic co-Chair

Maria Ptqk (Spain, EU), sub-thematic co-Chair

François-Joseph Lapointe (Université de Montréal, Canada), sub-thematic co-Chair

DOI: 10.69564/ISEA2023-1-Mahe-et-al-introduction

Keynotes

Three personalities were invited to give their views on current developments in the fields of creation, technologies and sciences, on the role of these in society, and to present their most recent work.

Ionat Zurr
(artist-researcher, Symbiotica, Australie) —
Symbiosis and the fallacy of a nature-free existence.  

In times of ecological emergency, solutionist fantasies of nature-free human existence promise salvation and repair. The innovative paradigm offers “products” such as lab-grown (animal free) meat and artificial automated surrogates to replace reproductive biological bodies. 

These so-called innovations require special artificial environments to host, nurture and culturally articulate this “new” nature-free, decontextualized and colonised life. The entanglement of life with its surrogate environment/apparatus, echoing human relationships with living and semi-living agents; when control and care are employed to counter resistance. 

Artists, scientists, designers and engineers all play their part in this transformation and its effects on human relations with life and the environment. This creates a range of ontological conundrums and fantastical expectations as to what technology can provide and to whom. Using examples of artistic research that deal with emerging technologies and new knowledge, Ionat Zurr narrated artists’ symbiotic and parasitic relationships with such post nature. 

This talk was framed by the imminent closure of SymbioticA, the first artistic research laboratory based in a life sciences department. SymbioticA began as a symbiotic act, embodied in an academic institution, to enable critical, yet mutualistic, relations among artists and scientists. Many of SymbioticA’s alumni have continued to establish their own laboratories and artistic practice in other academic institutions around the world, leading to the growth of the field of Biological Arts. 

SymbioticA is now being treated as a parasite by a changed host body. Is this a ‘natural’ survivalist rejection against a foreign body or can we detect symptoms of an autoimmune disorder? 

Biography: Dr Ionat Zurr is an artist-researcher. She is the Chair of the Fine Arts Discipline at the School of Design & SymbioticA academic coordinator at the University of Western Australia. Together with Oron Catts she established the Tissue, Culture & Art Project in 1996 and their co-authored book Tissues, Cultures, Art, published by Palgrave McMillan this year. Her collaborative work was exhibited by Pompidou Centre, MoMA NY, Mori Art Museum, Ars Electronica, National Art Museum of China and more. These ideas and projects reach beyond the confines of art and the work is often cited as inspiration to diverse areas such as new materials, textiles, design, architecture, ethics, fiction, and food.

Ionat Zurr © Nadia Rabhi

Viktor Ruban
(Choreographer, independent culture diplomat, Ukraine) —
Coping potential of creativity and art-practices in times of war: culture diplomacy, fundraising, curatorship and art-therapy force project.

If symbiosis is an essential notion, it is even more so in today's geopolitical context. War, terror and massive extinction of people in Ukraine by Russia is a global challenge shaping our global future at this very moment. In this situation performing artists in Ukraine keep on strong commitment to help in any possible way. Since the beginning of the full-scale invasion in 2022, the situation is forcing us to search for not only new ways of dealing with challenges creatively but to discover new ways of implementing our art practices and skills to completely new levels and spheres, such as resistance in information wars, culture ecology maintenance, new forms of fundraising and help with physical and psycho-emotional recovery — same as military and civilians —.

During his session Viktor Ruban, shared information about initiatives that he initiated or is involved in, such as the European Culture Parliament, and culture diplomacy challenges that he faces on different international events; Ukrainian emergency performing arts fund and funding challenges for the independent performing arts scene in Ukraine; international solidarity events and visibility of Ukrainian actual art scene—why it is important; actual creations in Ukraine and trends seen through the actual national theater prize season; the development of the project for training “psychological first aid instructors for military from the front line” and Art therapy force project— a range of activities implementing art-practices and working with creativity for psycho-emotional health recovery, coping with stress and panic attacks as well as preventing self-destructive behaviors and PTSD for diverse groups of people. 

Biography: Viktor Ruban is choreographer-researcher, curator, performer, educator, independent culture diplomat and culture activist. Initiator and ambassador of Ukrainian Emergency Performing Arts Fund initiative, he represents Ukraine in European Culture Parliament. He is also program director and co-founder of venue #KyivDanceResidency—platform for international studies in somatic, dance and performative practices, movement-based art and research. Ph.d. student in culture studies of Modern Art Research Institute of the National Academy of Arts (Kyiv, Ukraine)

Viktor Ruban © DR

Michael Century
(musician and cultural theorist, Rensselaer Polytechnic Institute, NY) — 
Nonsynchronous Innovation: Periodizing the Digital

While the digital transformation continues to outpace socio-institutional adaptation, the technological arts have moved on to vastly expand their temporal horizons. Timescapes of artistic research and creation now embrace the residual as much as the emergent; techno-diversity against digital solutionism; sympoietic rather than linear models of innovation. But nonsynchronous innovation is hardly unique to the current moment, as revealed in Michael Century’s recent book Northern Sparks, on Canada’s early experimentation with digital media.

Poised as a “counter-environment” to the great powers, in McLuhan’s phrase, Canada’s experience of the transitional decades into the information age was grounded in a technological ethos that emphasized sensorial immediacy, embodied interaction, and improvisatory expression. This alternative ethos was situated between a pair of distinct yet inextricably bound forces, one national-political and proper to Canada, the other techno-mediatic and global in scale.

The unraveling of these forces by the late millennium reveals innovation itself as a complexly drawn process comprised of multiple layers with fluctuating degrees of synchronization. From a cross-media perspective, Northern Sparks also reveals how the differences between the arts with respect to improvisatory immediacy and discrete formalization make any neat chronological periodization of the digital problematic.

Biography: Michael Century, musician and cultural theorist, is Professor of New Media and Music at Rensselaer Polytechnic Institute. His book Northern Sparks: Innovation, Technology Policy, and the Arts in Canada from Expo 67 to the Internet Age appeared with MIT Press in 2022. At the Banff Centre, he founded the Media Arts program in 1988. Century’s works for live and electronically processed instruments have been performed and broadcast in concerts and festivals internationally.   

http://www.nextcentury.ca
https://vimeo.com/showcase/10249242

Michael Century © Nadia Rabhi

Short Papers

Armand Edwige, Asensio Anne, Foresta Don 
Art, Science and Industry: A symbiotic milieu to rethink visions of the world

Bakke Monika
Symbio(geo)sis: When mineral and biological species meet

Blanchi Yann
4E Cognition for Symbiotic Architecture?

Bourgeois Mariejulie
A [Potential] Cloud War Controversies and conflicts related to climate manipulations

Boutilier Célia
Inhabiting the Edges

Cucicov Dorin
Mytherrella: an interactive installation hallucinating mythological auroral formations

Frémontier-Murphy Camille
At the Sources of an Artistic Mutation towards Science: the First Years of the Journal Leonardo (1968-1981) as a Forum for the Pioneers of Digital Art

Garančs Jānis
Sensoriums for the Ephemeral —gamification of values

Geck Kate
Mycorrhizal Materialities Positioning the entanglement of human and machine intelligence

Andrea Gogova
Interspecies Communication — Water bodies

Hedayati Mona
Intelligent Sensibility: Human-Machine Symbiotic Agencies

Hilsberg Victoria
DAOs A blockchain-based application not intervening, but strengthening the agility of contemporary arts

Hu Youyang, Chou Chiaochi, Kakehi Yasuaki
Lucid Dream: Sensing and Artistic Representation of Plant-Nature Interaction Based on Plants Biosignals

Jørgensen Jonas, Christiansen Mads Bering
Sounding Softness and the (Artificial) Subject

Lukaszuk Mike, Bahng Sojung
Gesture and Spatiality in Electroacoustic Improvisation with Digital Video

McFadden Chloe
Generated tools: A Defamiliarizing Approach to Creating ML Art

Montanari Matheus da Rocha
Ecologies of Thought: Generative Art as a Collaborative Research Methodology with Guarani and Kaiowá Indigenous Communities

Moren Lisa, Bachvaroff Tsvetan
Emerging Strategies “Under The Bay” in AR/XR

Nieves Raul, Soler-Adillon Joan, Mor Enric
Power and Resistance in Digital Degrowth

Ottavi Julien, Pickett Jenny
Motherplants: Mycelium Network and Artistic Research

Pederson Claudia
Life-as-it-could-be, Symbiosis in Interspecifics’ Codex Virtualis_Genesis

Pranjić Kristina, Germek Magdalena, Purg Peter
Disruptive Avant-Garde Art of Today: Shaping Post-Growth Imaginaries for Symbiotic Futures

Ricci Nicolas, Agogué Marine
Drivers for Resilience in Cultural Organizations: lessons from the Montreal festivals in the face of the COVID-19 pandemic.

Rogers Hannah, Bencard Adam
Metabolism and Art

Sareen Harpre, Singh, Fu Yibo, Kakehi Yasuaki
Ephemera: Bubble Representations as Metaphors for Endangered Species

Scott Jill
Jellyeyes — Symbiosis, Evolution and Vision

Shanken Edward
Technoshamanism: Symbiotic Techniques of Art and Healing

Shin Gyung Jin
Symbiotic Collaborators: The New Creative Subject in Postdigital Participatory Art

Smigielska Maria, Cutellic Pierre
Proteus 3.0 — Interacting with the cloud

Steyaert Pieter, Vermeulen Angelo, Holt Amy, Maranan Diego, Checola Giusy, Nasser Mona, Verschuren Jeroen, Tournaye Pim
Co-creation Towards the Post-Anthropocene

Thornhill Vincent, Legrand Guillemette, Clarke Isaac
Spectral Plain: A case study for exploring the world-building potential of co-creative systems that combine text generation models with game mechanics

Trudel Gisele
Ecotechnologies of Practice: in-forming changing climates

Wollensak Andrea, Terry Brett, Baird Bridget
Water Stories: Visual Poetics and Collective Voices

Wright Rewa, Howden Simon
Nga manawataki o te koiora: biological rhythms, posthuman design and decolonial thought

Yeginsu Ipek
The Dark Side of the NFTs: The artists’ need for new systems of collaboration

Zhang Yu
STAND BY/ME

Art, Science and Industry: A symbiotic milieu to rethink visions of the world

Edwige Armand, Anne Asensio, Don Foresta 

Université Gustave Eiffel, LISAA, Passerelle Art-Science-Technologie Dassault Systèmes, MapMarcel, France edwige.armand@gmail.com, anne.asensio@3ds.com, don.foresta@gmail.com

Abstract

How do art and science participate in writing a world of signs and symbols? What do these two human faculties and capacities seek and how do they differ radically? Science has progressively freed itself from its purpose (the search for the why) to model, imitate and master a world in an increasingly abstract manner. If science and art are performative, what world do they draw and what do they tell us (in terms of discourse) about the human? As science gradually becomes techno-science and produced by industries, does art have a singular role to play in the interrelation of these two poles, science and industry, in order to re-inhabit the ethical questions and meanings that were originally in these other two creative capacities? Can it and should it? 

Keywords

Art, Science, Industry, Signs, Sensibility, Technoscience, Blindspot, New cooperative ecosystem, incorporation of experience.

DOI

10.69564/ISEA2023-2-short-Armand-et-al-Art-Science-Industry

Art-science and world shaping/forming

We will share an assessment of the relations between art-science and industry, the novelty being that with digital technology the questions have been accelerated, augmented and have now come to integrate the notion of physicality. Industry is a playground; whether it is science, art, how do we seize the digital? Do we have the means to think the digital or are we alienated, with the digital no longer as an externality, an interface, but as an immersion, as a new space of being in the world—particularly with the metaverse—as a socio-technical space? It is necessary to think of industry as a workbench for virtual universes and as a space of reconciliation between disciplines via the designers as sensitive mediators and scouts of the new virtual worlds. We need a new epistemology of the virtual worlds that takes into account the multi-dimensional human. We are looking for ways between micro-ecologies, localities, craftsmanship and a global, automatic approach, the two poles that build our "being in the world."

Art could be thought of as the fabrication of new categorizations of an environment¹ that escapes the norms of language and conventional imaginary modelling. Because the perceived world is always the result of an interpretation and a shaping which are imposed a posteriori by culture, that the inaugural perceptual singularity is subsumed by culture. The perceptual thresholds linked to temporal representation structure a vision of reality constructed by stops and artificial linearity². Yet the imaginary, the perceptual virtual cannot be stopped at these arbitrary forms of actualization of the world. Whether it be time, imaginary, or spatial perception, they cannot be detached from a body in action that perceptively actualizes a world that appears and is created in a living time (dancers or performers give rhythm to time, for example). The manifestation of the event is an extract of the real, obliterating the infinity of data, infraliminary and supra-liminary information³ not actualized by language whose readings cannot reach the threshold of a disciplined thought. It is in this failure of a completeness of reality that art operates.

It is an attempt to reveal hidden information inaccessible to authorized cognitive processing. Other articulations, forms, relations to the world exist that art tries to bring to light. It is a raw, unanalyzed grasp⁴ that the act of formalization and expression will seek to bring into existence. Art is, first of all, an impossible return to an instinctive and affective perception, where the body echoes the world and where the world is then echoed (exosomatisation). Art in these premises is an anchoring (Harnard Stevan⁵) of signs of the world kept in memory (retention), attached to and mirroring the body that perceives itself (while keeping the unspeakable abyss which Pierre Legendre refers to ⁶).

As a first reflexive return, art imprints the world in order to reflect on it and to appear itself (first distance from the world—see the cave paintings and the work of Carole Fritz⁷). It is a primal affective reading of a world that is not there for it, and for which it seeks a meaning. It is an attempt to answer the abysmal question of why and of a world that is not up to his standards, but in which, through the inscription of signs, he can make his presence last beyond his own finitude.

As the first interface between the body and the environment, art is a sign of principle writing. As a symbol and practical return of a memory and of an instant kept⁸, it is a means for recalling presences. It is an imprint of an existence and of a difference made with the self and the environment, the self and the other, who delimits and recognizes itself in it. Through the diffusion of signs and symbols, language and rituals, the particularity of a seized moment becomes an abstraction, a practical medium to think about the world and save the detailed and vivid perception that every physical and phenomenal novelty requires. But if art has to do with the first symbolic anchors, the first perceptions, it cannot be reduced to them.

Art is a first reading, a projection of human meaning on the world, and in this, science does not differ from this attempt to understand and mark the world. In science, the purpose is different, the tools are more sophisticated, no singularity is demanded and the dream of a universal and standardised objectivity.

Art and science are imaginary and projective capacities, part of a process of fabrication and transformation of the world. This capacity for transformation by modern science has largely emancipated itself from its role as a myth whose function is to answer the question of why live and to limit unreason in order to support existential anguish⁹.

But art and science are not only intended to think about the writing of a world and its human legibility, but also to put it into the world, to give it a stage, to create a narrative for it and for the human being. Their discourse and imagination are performative.

In contrast to science, art is anchored in the body and the affection of the world, turned towards the appearance, the distinction, the singular discrimination of signs as a search for actualization. Science in its hyper-technical devices sets aside this affection for the body, and is based on instrumental rationality.

It is important to underline representation acts as a perceptual framework indicating the phenomenon and giving it an interpretative form in order to apprehend it.

Science, as a rationalized exo-body, it gives us the opportunity to perceive a phenomenon (a techno-phenomenology) and a world framed in advance by the laws inserted in it, offering a metric vision and mathematical logics, influencing in its turn the way of writing and understanding a world surrounded more and more by technical abstraction, self-referential and discarding the perceptive variability of a sensitive and feeling body.

Techniques are extensions of organs¹¹ reading and writing the signs of the world. Art has taken an interest in the world of science and technology out of a spirit of resistance (perhaps) to the homogenization of bodies and sensibilities, out of a refusal to accept the authority of scientific discourse, which sometimes imposes "butchering"¹² and reductionist views of the human being. The definition of man varies according to culture, religion, and time. Feeling human, our relationship to the world is defined according to the interpretative frameworks conveyed in a culture, which cannot be absolute. We know that the way in which man is expressed influences in turn the experiential potential and the power of his experience.

“Cogito ergo sum”

“Cogito ergo sum” of Descartes has been the heart of the scientific domination of our society’s world-view, the way we understand the world defined uniquely through reason. It has been responsible for an unequaled level of material progress but has reduced humanity into entities befitting uniquely mechanical formulation, ignoring the whole human being, concentrating only on those aspects of the person considered important for the task at hand. In recent western history we, as a culture, have tried to deny this human wholeness by declaring that rationality was the only approach to true knowledge and that all else must be rejected. Anything resulting from feeling or emotion was unreliable and even dangerous. That added to the mechanically quantifiable view of how things work has created a distorted perspective, a world-view based on a diminished idea of the human being. Focusing on the intellect was a useful but limited approach to how humanity is defined putting aside, "Sensio ergo sum"—I feel therefore I am—which is how most people first react to sensory input. Recognising that aspect of humanity not only allows us to understand the whole human by respecting the other half, it also connects our species with the rest of the living world which reacts principally through feeling. Objectivity is a tool, an important one, but to imagine humans cut off from their emotional side is dangerous and leads to often undefined but very real frustration as we see more and more today.

The capacity for symbolic thought has always been part of our makeup in its earliest primitive forms since the beginning of our genus and most probably well before. It comes from an instinctive and unintellectual operation of that curiosity which is part of every living being’s functioning as a way of exploring our environment and surviving in it. When the found object of our ancient ancestors, the first overt manifestation of what we now know as art, was kept and shared with others it acquired a symbolic sense for the group as well as the individual who found it, an unarticulated meaning but something felt to be important. With the evolution of our intellectual capacity, this process becomes more elaborate in providing a «why» to that selection giving those objects an explanation of their symbolic value and a means of communicating it and often the beginning of what can be seen as a religious belief system. In the beginning, this was not an intellectual exercise and the choosing came from feeling rather than understanding. It became more intellectually elaborated in time through the need to be communicated, probably diluting the experience of the first-person experience but making it transferable to others and incorporating other reactions to it.

This is the root of artistic creativity and its eventual integration into culture. The symbolic is very much part of us and ignoring it or pretending it is less important or to be overcome demonstrate a dangerous misunderstanding of how humans function. To attempt to understand our confrontation with the real, both art and science are needed for a fuller and more human comprehension of what we are defining. Understanding intellectually is necessary but stopping there leaves the human wanting more, which defines the emotional atmosphere we live in today. We cannot expect people to understand when half the person is ignored. In our mechanical world-view art has been pushed out of its central role of informing into manageable categories to fit the demands of the market it has been confined to.

Artists, on the other hand, have never ignored science, particularly in a society which has allowed science to be the sole arbitrator of our reality. Science has thus become the matter of art, our reality subjected to the questioning of art. Technology, the product of science, has always attracted artists because of a curiosity about what humans do but also because it is a potential tool, particularly the technologies of communication. Importantly as well, the artists’ use of technology has often influenced our perception of it and how it is integrated into society and affects us.

A technology developed through both art and science would, by definition, have more dimensions by being the product of the full human being. This has been the objective of our project MARCEL, an international group dedicated to artistic, scientific and educational uses of the network and expanding network technologies through the demands of art and not just market-driven consumption. It exists and evolves through the share experiences and technical development from creative uses of the network pushing the boundaries of the possible. This is also the approach that the TRAS network (transversal network of art-science networks) is trying to put in place in France. As are the creative platforms in industry that seek to give meaning to technology, since industry is the heart of its production.

Industry as a blind spot?

Industry is perhaps a blind spot in our times. Principally thought of as an operative process, a technical milieu and not enough as a thinking object. As a civilization manifestation of a way of seeing the world. Industry always been a space of intervention and convergence in which an art-science dialectic can flourish.

At the crossroads of the most advanced human creative activities, the industry is projecting a vision, organizing the work and the economy defining every aspect of everybody’s life. The industry today manufactures the eye and the prosthetic body of tomorrow, just as science and art have done. To think of working in theatrical and artistic terms on science, digital environment or technologies that enable all aspect of the modern world without taking into account the environments in which techno science is constructed may be a mistake. The question of the representation of the body, of what constructs the human experience, of what links the human to the world has never been so obvious since the announcement of the promise of virtualization and modelling of the human being, from his intimate relations to the notion of habitability. Virtual worlds (beyond the definition of Metaverse) promise us a world in which the human experience could be reconfigured in a bespoke manner without the canal of affective and invisible human dimension, of the anchoring of the symbolic through the affective path, of the word that introduces us to subjectivity and otherness, a word that confronts us in its strangeness and its inexhaustible incompleteness.

The standardization of the perceptible, the logical and scientific determinism could reach its apex in these new universes that redesign the conception of the human. It is therefore in the interest of artists and designers to present the eminently open conception of a world thought out by art and designers so as not to close the vision of the human to an automaton-avatar with which it would be a question of playing and moving in the way of a puppet. A human representation that disregards the gravity of bodies, the rites of social interaction, the collective to be woven, the immersion in a living half-place where the segregation of the visible is decided upstream and selecting what can be seen from what does not deserve to be seen. The human being is not a pack of information to be grasped, he is a being of dreams and words and it is better to know this than to ignore it. It is in the interest of art and design, philosophy and anthropology to integrate these decision-making circles of future representations that will be conveyed with an unequalled power of diffusion and democratization. There is a stake in everyone being able to grapple with the question of the relationship to the environment that we can rethink, the adventure of discovery, of the process of making, of surprise and of creation, which is never elaborated in predetermined data, but which emerges from the unpredictable and the unknown.

How can we open up these industrial worlds to the civilizational challenges they face today? How can we make people understand what design can do at the intersection of art, engineering and industry? Design is the keystone of the human intentions of any industrial project deploying the openness of the human senses, of human abilities and expectations into a project, in a tangible and accessible manner improving the human benefice “in fine”... Design is drawing and aim at the same time, a potential direction as a line of conduct and orientation to be incorporated into any industrial and societal project... How can we reintegrate meaning upstream of this productive force so that it serves a majority in humility and equity? What are the first tracks and trials of these co-operations and new ecosystems where art-science-industry work together again? What are the first attempts made by industry, which must rethink itself at a time of climatic emergency and where the production of waste objects, of intensive innovation based on the imagination of mass production, of Taylorised work and progress must be reinvented? How can we reinstate the sensitive in these industries that produce techno-imaginaries? How can we reconstitute noetic spaces ranging from industrial environments, experiential hybrid environments, to virtual immersive universes (metaverse)?

Can we imagine and draft new roadmaps to rehabilitate Art without its instrumentalization in the scientific and industrial spaces where today most of our narratives and actions that construct representations are being produced? Art is probably the future of industry understood as capacity for representation for its power to create new bodily and cognitive experiences, and its possibility of creating dreams and new solidarities. This implies putting art back into life, as medicine of the imagination, as the art of being together. Should it, can it?

Such regenerative new hybridization and such new forms of cooperation would represent a new symbiosis of the creative forces of the human being, in which art would, in that case, become design, design as a mindset and “way to think.” Science and industry could collaborate for human and non-human benefit driven project. A project which path remains open to matter, to the singular, to the body in “totality,” to the living, to the unpredictable and immeasurable, which is the nodal point of invention.

This utopian symbiosis we are describing is what we feel is right to focus on, in order to build the narrative and the bases of a dialogue to be reinvented so that all human faculties are no longer dissociated, but reunited to serve an imminent civilizational project. Considering all the living in all their diversity and unknowability as the preciousness of life should be taking into account in the course of our actions. We need a holistic and systemic approach within the limits of the consequences of the harmful and positive impacts that we can and must imagine. Industry must reclaim its original role of building a world¹³ to be improved, to be inhabited for all and in the perspective of diversity. It is a call for a collaborative symbiosis that takes into account the complexity and the need to work together, despite the stereotypical oppositions that prevent these human universes from interacting again.

Beyond the civilizational stakes, industry is today a playground for art and science, and dialectic between art and science is vibrant. It questions their relationship, the tools they manipulate, but also because industry must regenerate its raison d'être (get out of consumerism, functionalism). It must become organic. It is when we have a calmed art-science dialectic that industry will be able to resolve these new issues.

Another question comes to mind: does industry contribute to the discourse of technoscience, or is it the victim or the keystone? With cultural and economic globalization, its responsibility is taking the main stage... in its making of the world and tomorrow's world, the industry cannot be reduced to a means of progress nor an externality functionally assess but as a subject for thought, an object for design... and Art. Shouldn't the hyper-strategic position of industry be aestheticized as an experience with methods of art and science, i.e., art applied to industry which is called design? Industry has reached a level of maturity that is no longer under the yoke of politics, it is political. As such, industry must reconvene art and science within itself according to its own responsibilities. An industry that thinks, reflexive to change the world will not arise without the aesthetic and ethical sense. An industry more in tune with a regenerative mindset will enrich a new form of relationship with Nature, Human and the world, an artistic approach from within, to nurture a sustainable future and solutions that undoubtedly will be found.

References

1 Gilbert Simondon, Imagination et Invention, 1965-1966, Paris, Presse Universitaire de France, 2014.

Henri Bergson, L’évolution Créatrice, Paris, Presse Universitaire de France, 2013.

3 Günter Anders, Et si je suis désespéré que voulez-vous que j’y fasse, Paris, Allia, 2010, p.72.

4 Charles Sanders Peirce, Écrit sur le signe, Paris, Seuil, 1978, p.23.

6 Pierre Legendre, Les enfants du texte, Paris, Fayard, 1992.

7 Carole Fritz, L’art de la préhistoire, Paris, Citadelles et Mazenod, 2017.

8 Ernst Cassirer, La philosophie des formes symboliques, le langage, Paris, Edition de Minuit, 2017.

9 Pierre Legendre, Le visage et la main, Paris, Les belles lettres, 2019.

10 Jakob Von Uexkull, Milieu animal et milieu humain, Paris, Broché, 2010.

11 André Leroi-Gourhan, Le geste et la parole, Technique et langage, Paris, Albin Michel, 1964, p.49.

12 Alain Supiot, La gouvernance par les nombres, Paris, Fayard, 2015.

13 Pierre Musso, La Religion Industrielle, Paris, Fayard, 2017, p.12. Journal article (online)

5 Stevan Harnard "Le problème de l'ancrage des symboles" Physical Review D, 42, accessed November 7, 2022, https://archipel.uqam.ca/14111/ 

Bibliography
  • Anders Günter, Et si je suis désespéré que voulez-vous que j’y fasse, Paris, Allia, 2010, p.72.
    Bergson Henri, L’évolution Créatrice, Paris, Presse Universitaire de France, 2013.
  • Cassirer Ernst, La philosophie des formes symboliques, le langage, Paris, Edition de Minuit, 2017.
    Fritz Carole, L’art de la préhistoire, Paris, Citadelles et Mazenod, 2017.
  • Harnard Stevan, “Le problème de l'ancrage des symboles", Physical Review D, accessed November 7, 2022, p.42, https://archipel.uqam.ca/14111/
  • Legendre Pierre, Le visage et la main, Paris, Les belles lettres, 2019.
  • Legendre Pierre, Les enfants du texte, Paris, Fayard, 1992. 
  • Leroi-Gourhan André, Le geste et la parole, Technique et langage, Paris, Albin Michel, 1964, p.49.
  • Musso Pierre, La Religion Industrielle, Paris Fayard, 2017, p.12.
  • Peirce Charles Sanders, Écrit sur le signe, Paris, Seuil, 1978, p.23.
  • Simondon Gilbert, Imagination et Invention, 1965-1966, Paris, Presse Universitaire de France, 2014.
    Supiot Alain, La gouvernance par les nombres, Paris, Fayard, 2015.
  • Von Uexkull Jakob, Milieu animal et milieu humain, Paris, Broché, 2010. 
Authors’ Biographies

Edwige Armand is an artist and a teacher-researcher in art and culture at the Purpan engineering school (INP Purpan) and is attached to the LARA-Seppia laboratory. Passionate about the history of science and art, and philosophy, she combines these different disciplinary fields around the question of techniques and sciences. The modifications of the relationship to reality, subjectivity and the body by techniques and sciences are central axes in her research-creation. The transversality of disciplines allows him to question more broadly the processes of creation and the transformations of representations that the arts contribute to. To revive the dynamics of art-science relations, she co-founded and has chaired since 2016 the association Passerelle Art-Science-Technologie, which works to bring these disciplines closer together, and she is also involved in the Transversale des Réseaux Arts Sciences.

Anne Asensio, after executive and design management positions in the automotive industry, joined Dassault Systèmes in 2008 as Vice President Design Experience. She created the Design function of Dassault Systèmes and the Design Studio, bringing together a multidisciplinary team in innovation strategy through design, experience design and design research. Advocating a participatory approach to new technologies and virtual worlds, the Design Studio supports Dassault Systèmes' customers in high-growth industries in their transformation, digital and sustainable innovation needs towards virtuous design processes. Imagining alternative scenarios to transform the world we live in into a more sustainable and desirable one, the studio engages in reflection, experimentation and confrontation of cross-cutting approaches between social and technological issues through creation.

Don Foresta is an art theorist using new technologies as creative tools. Specialised in art and science, he has been a professor at ENSAD and the Ecole Nationale Supérieure d'Arts - Paris/Cergy and a research fellow at the London School of Economics. He has spent 35 years transforming the network as an artistic tool and is working on the creation of a permanent high-speed network, MARCEL, dedicated to artistic, educational and cultural experimentation. In 1981, he made his first online exchange between the Center for Advanced Visual Studies at MIT where he was a fellow and the American Center in Paris where he was director of the Media Art program. In 1986, as curator of the 42nd Venise Biennale, he made the first computer network used by artists. He was named Chevalier de l'Ordre des Arts et des Lettres by the Ministry of Culture. 

Symbio(geo)sis: When mineral and biological species meet

Monika Bakke

Adam Mickiewicz University Poznań, Poland monika.bakke@amu.edu.pl

Abstract

As biocentrism is increasingly seen as unjustifiable, artists and environmental humanities scholars become more attentive to feedback between minerals and (non)human life. First, I discuss the urgent need to reinvent narratives and create new vocabulary about the abundance of life-mineral connections as presented by Mabe Bethônico in her Stone Statements Editions, which focused on developing "a vocabulary of proximity." Next, I analyze Magdalena Abakanowicz’s artwork Space of Unknown Growth, investigating emerging material relations between concrete (an anthropogenic rock) and nonhuman life. I argue that rather than limiting the environment to a symbiotic community, these artworks encourage the curiosity and attention necessary to embrace mineral-life connections and envision more inclusive communities of the future. 

Keywords

Art and science, geology, minerals, technofossil, concrete, Long Ecology, biocentrism

DOI

10.69564/ISEA2023-3-short-Bakke-Symbiogeosis

Introduction

Biocentrism is increasingly viewed as unjustifiable. Minerals are everywhere, but in Western thought, nonlife has always been significantly overshadowed by robust, organic life. The multispecies studies have so far embraced the diversity of mineral species only in a minimal capacity. Although biocentrism still predominates, change seems inevitable. The currently emerging trend in the humanities and arts to truly embrace mineral species (considered nonlife or geos) is growing in significance, and it is attentive not only to environmental concerns but also to new developments in mineralogy. The latter focuses on the co-evolution of minerals and life and the unprecedented increase in the diversity of human-made and human-mediated mineral species and novel materials characteristic of the Anthropocene.

Vocabulary urgencies

To better recognize the complex relations within geo-biotic communities, it is necessary to reinvent attitudes and narratives about mineral species. This need was addressed by Brazilian artist Mabe Bethônico, who called for the development of a «vocabulary of proximity» of minerals and life. 

Bethônico created StoneStatements Editions in response to the question: How Will We Live Together? posited by the curators of the Venice Biennal of Architecture in 2021.

Bethônico’s editorial proposal includes five inspiring book tiles: Geoimaginaries; Conscious Rocks; Human Invasion in the World of Stones; When Stones Collect Diggers, Robbers, Queens and Kings; Missing Words for Considering Stones, Rocks, Pebbles and Mountains: A Vocabulary of Proximity. Only the latter has been developed so far as a collective effort, resulting in a volume of twenty-six short entries. As Bethônico stated, the remaining books are «imagined, wished for, and suggested to be developed» as more vocabulary to think about the environment in more than biological terms is urgently needed. Her project recognizes the biocentric bias and the conditions of its operation, namely the shortage of concepts, stories, and creative practices to embrace mineral species.

Figure 1. Mabe Bethônico, 2021. Cover design by Elaine Ramos. © Mabe Bethônico.

Anthropogenic rocks

The proximity of life and minerals, especially rocks characteristic of the Anthropocene epoch, is investigated and anticipated in the artwork Space of Unknown Growth by Polish artist Magdalena Abakanowicz. Located in the forest of Europos Parkas, this artwork comprises twenty-two concrete boulders in four sizes, spread over an open-air museum near Vilnius. Concrete, although anthropogenic, is considered a "geological entity" due to its close relation to the mineral composition of the earth. It is a novel rock and, as geologist Jan Zalasiewicz suggests, “the most abundant anthropogenic sedimentary rock on the planet.”² Classified as a technofossil together with a great variety of synthetic compounds, concrete is emerging as the prominent signature of our geologic epoch. There is, however, a particular risk of succumbing to the lithic allure recognized by Stacy Alaimo, who calls for caution when suggesting that “attending solely to the lithic imports delusions of separation and control."³

Figure 2. Magdalena Abakanowicz, Space of Unknown Growth, 1998. © Europos Parkas. 

Space of Unknown Growth speculates about a future that belongs to multispecies communities whose endurance is anticipated rather than predicted. For surviving, making kin with unexpected others, as Donna Haraway suggests, is necessary. Staying with the concrete means "staying with the trouble", which demands that the viewers acknowledge the devastating environmental impact of extracting resources and recomposing materials.⁴ And yet, Space of Unknown Growth does not convey alarm. Instead, it points out that adaptation goes together with the transformation of anthropogenic lithic environments, which accommodate more than human life and is a condition needed to renew planetary species diversity. Living on the ruins is possible as some organisms can modify the lithic surfaces in their environment. The technofossils that comprise Space of Unknown Growth indicate the anthropocenic Earth’s habitability. The anticipated growth speculated on in Abakanowicz’s work is conditioned by the composing and decomposing of biological and mineral species and the resilience of communities capable of terraforming concrete—the most earthly rock.

Long Ecology

Bringing anthropogenic stone to the forest introduces a perspective of what Jeffrey Jerome Cohen calls Long Ecology oriented toward the future and vastly exceeding current waste management. However, concrete, as “a material that mediates transcience and endurance," is not as long-lasting as stone.⁵ It rejects the conventional belief that geological processes are always long and slow. From Long Ecology's perspective, although prolific in novel mineral species and formative for technofossils, our epoch might not be capable of reaching a distant future because anthropogenic minerals might become extinct.⁶

And yet the stones of Space of unknown growth might be considered erratics, conventionally defined as rocks not matching the local bedrock. Erratics are lithic strangers, newcomers transported to current locations by glaciers or other natural forces of significant magnitude. Concrete boulders of Space of unknown growth are synthetic rocks and mineral novelty brought into the environment of a central European forest. Yet, Abakanowicz’s erratics establish a category of their own as they are art forms, which nevertheless participate in the currently intense processes of relocating minerals by humans—activity believed to exceed even glacial action. Abakanowicz’s lithic newcomers, however, with their own material history, become with their new environments through complex geo-bio relations and processes. These unexpected others, strangers to their landscapes, arriving from a different time and space, introduce the possibility of a radically new narrative of connectedness and multispecies kin-making, which stimulates yet another way for humans to practice attentiveness and care. They demonstrate both vulnerability and resilience, which characterize earthly worlding and unworlding. 

Figure 3. Magdalena Abakanowicz, Space of Unknown Growth, 1998. © Europos Parkas. 

Although the sculptures operate as erratics in the context of Long Ecology, there is no separation between them and the materiality of their location. Through their lithic ways of being, the works participate in planetary processes operating, in Cohen’s words, as “...an affectively fraught web of relations that unfolds within an extensive spatial and temporal range, demanding an ethics of relation and scale.”⁷ In this sense, Space of unknown growth is deeply ecological in its attentiveness to the vast scales in which life becomes stone and stone becomes a living environment.

Coda

What we need now is not a belief in the solitude and indifference of stones, rocks, and pebbles but a celebration of their inextricable connection with life and creative elaborations on their diversity. Both StoneStatements Editions and Space of Unknown Growth inquire about ways to articulate how minerals and life actually attract and seduce each other and how minerals open up to life’s intimate strategies, the outcomes of which cannot be determined. They offer vocabulary and materiality to tell stories of Symbio(geo)sis. Rather than limiting the environment to sym-biotic community, these works encourage curiosity and attention needed to embrace mineral-life connections better. Only then will geo-biotic communities emerge as the answer to the question of how we will live together. 

References

1 Mabe Bethônico, “StoneStatements Editions,” Mabe Bethônico, accessed September 3, 2023, https://www.mabebethonico.online/stonestatements-editions.

2 C.N. Waters, J. Zalasiewicz, “Concrete: The Most Abundant Novel Rock Type of the Anthropocene,” in Encyclopedia of the Anthropocene, ed. Dominick A. Dellasala, Michael I. Goldstein, Oxford, Elsevier, 2018, p.75, https://doi.org/10.1016/B978-0-12-809665-9.09775-5.

3 Stacy Alaimo, Exposed: Environmental Politics and Pleasures in Posthuman Times, U of Minnesota Press, 2016, p.149.

4 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke University Press, 2016.

5 Richard D. G. Irvine, Anne Bevan, “Concrete Buys Time: Art and Anthropology in the Anthropocene,” Worldviews: Global Religions, Culture, and Ecology 26, no. 3, October 19, 2022, p.182, https://doi.org/10.1163/15685357-02603009.

6 Jan Zalasiewicz, Ryszard Kryza, Mark Williams, “The Mineral Signature of the Anthropocene in Its Deep-Time Context,” Geological Society, London, Special Publications 395, no. 1, 2014, p.112, https://doi.org/10.1144/SP395.2.

7 Jeffrey Jerome Cohen, Stone: An Ecology of the Inhuman, U of Minnesota Press, 2015, p.41. 

Bibliography
  • Stacy Alaimo, Exposed: Environmental Politics and Pleasures in Posthuman Times, U of Minnesota Press, 2016.
  • Mabe Bethônico, “StoneStatements Editions", Mabe Bethônico, Accessed, September 3, 2023, https://www.mabebethonico.online/stonestatements-editions.
  • Jeffrey Jerome Cohen, Stone: An Ecology of the Inhuman, U of Minnesota Press, 2015.
  • Donna J Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke University Press, 2016.
  • Richard D. G Irvine, Bevan Anne, “Concrete Buys Time: Art and Anthropology in the Anthropocene”, Worldviews: Global Religions, Culture, and Ecology 26, no. 3, October 19, 2022, 179–95, https://doi.org/10.1163/15685357-02603009.
  • C. N. Waters, J. Zalasiewicz, “Concrete: The Most Abundant Novel Rock Type of the Anthropocene”, In Encyclopedia of the Anthropocene, edited by A. Dominick Dellasala, Michael I. Goldstein, Oxford: Elsevier, 2018, 75–85, https://doi.org/10.1016/B978-0-12-809665-9.09775-5.
  • Jan Zalasiewicz, Kryza Ryszard, Williams Mark, “The Mineral Signature of the Anthropocene in Its Deep-Time Context”, Geological Society, London, Special Publications 395, no. 1, 2014, 109–17, https://doi.org/10.1144/SP395.2.
Author Biography

Monika Bakke is associate professor in the Philosophy Department and director of the Environmental Humanities Center at the Adam Mickiewicz University, Poznań, Poland. She writes on contemporary art and aesthetics, particularly interested in posthumanist, transspecies, and gender perspectives. Her curatorial work includes art exhibitions: Bio-Reminiscences (Poland), Seeing the Forrest Through the Trees (UK), Boundless Objects (Portugal), and Refugia: Keep (Out of) these Places (Poland). Currently, her research focuses on nonlife forces and new articulations of mineral becomings in contemporary art. https://amu.academia.edu/MonikaBakke 

4E Cognition for Symbiotic Architecture?

Yann Blanchi

BMOffice France yann@bmoffice.fr

Abstract

With the aim of reconsidering the very nature of architecture, we propose a conceptual tool to think of architectural apparatuses as actors within a continuum composed of both artificial and natural agents. For this,we look at the cognitive sciences, particularly the 4E cognitive (embodied, embedded, enactive, and extended). We use examples from contemporary architecture to test our hypothesis and thus attempt to define what symbiotic architecture could be.

Keywords

4E cognition, Embodied, Embedded, Enaction, Extended, Symbiotic, Architecture, Artificial, Natural

DOI

10.69564/ISEA2023-4-short-Blanchi-4E

Introduction

In this paper, we will explore a theoretical tool to think symbiotic architecture. To this end, we consider whether the transposition of the 4E cognition may be useful in describing the characteristics of what the symbiotic architecture might be. The paper is organized as follows: Firstly, we set the context and define the terms used, such as symbiosis and 4E cognition. Then we state our hypothesis of an analogy between 4E cognition and architecture. Next, we test this idea with examples of recent architectural apparatus, and we conclude.

Context

In 1960, as Gordon Pask stated in his theory of conversation, we moved from a functionalist world to a mutualist world. Today, in the context of the global crisis, nature is coming back into the loop. The ecocentric logic would reinscribe us in a network of multiple interactions and concrete connections with the milieu, engaged in a network of dependencies in a way of decentralized connections.Nature and artifice are bound to co-evolve. Under these conditions architecture can certainly no longer be the static and fixed object of the moderns but a new type of artefact. Architecture may have entered the age of naturalization.2

Definitions

Etymologically, the term symbiosis comes from the Greek and means "living with." In biology, symbiosis refers to associations between hetero specific living organisms. There are three forms of symbiotic association:parasitism, commensalism and mutualism. It depends on whether the relationship is unidirectional or bidirectional and beneficial, neutral or harmful. Semiologists study these interactions. On the other hand, in the field of cognitive sciences, the 4E cognitive approach proposes a conception of cognition as an assembly of the brain, the body and the environment. The 4E approach is based on four concepts: embodied, embedded, enactive, and extended.

Embodied cognition is cognition produced in part by structures other than the neural system. Embodied cognition rejects Cartesian dualism, separating the body from the mind, and includes the body in the cognitive system.The body is an integral part of the cognitive system, for example, we perceive relief thanks to the stereoscopic vision of our two eyes.

Embedded cognition is cognition coupled with the environment. Thought is not, as it were, secreted by the brain but by the environment.⁴ The extension of the body into its environment reduces loading and relieves the brain.⁵Enactive cognition is cognition produced in part by actions.⁶ We note here the importance of movement in the process. Extended cognition, a cognition situated in the environment, is an externalisation of processes in our environment. In the context of digital technologies, cyberspace would become an extension of our brain.⁷

Hypothesis

Our hypothesis is the following: if architectural apparatus supported 4E cognitive activities, then architecture would become symbiotic. For this purpose, inhabitant/architecture/milieu has to constitute a form of dynamic coupling, connecting natural and artificial actors. To test our hypothesis, we present below an architectural project example for each of the four E categories.

Embodied architecture

The Urban Algae Folly, designed by EcoLogicStudio for the EXPO Milano 2015. (1) The microalgae cultures embedded in the architecture allow to control transparency and shading qualities of the membrane are the result of algae growth depending on the sunlight and the presence of visitors.

In this example, not only do the algae have the dual role of sensor-actuator, but they are distributed over the entire surface of the device. All parts of the building body are sensitive and responsive, intelligence is distributed, embodied in every square centimeter of the material.

Embedded architecture

HygroSkin, Meteorosensitive Pavilion by Achim Menges' team is part of the permanent collection of the Frac Centre since 2012. (2) This meteo-sensitive architecture is the result of research into materials that react to variations in their environment. The device is inspired by the pine cone principle. The multi-layered wood reacts to humidity, the holes open and close depending on the degree of humidity in the air. The heterogeneity of the material allows the deformation of the surface, as each layer does not have the same expansion coefficient. In this example, the architecture is coupled to the environment in a unidirectional link that gives it shape.

Enactive architecture

In 2003 Kas Oosterhuis/ONL agency was invited to create an installation for the exhibition Non-Standard Architecture. (3) The NSA Muscle is a pneumatic interactive device composed of 72 muscles programmed to be 72 independent members of a single swarm. The 72 inflatable muscles are controlled by individual valves. The device is programmed to have its own behavioural cycle, but also to react instantaneously to an external stimulus. In this way, ONL materialises the concept of two-way communication in real time linking two active elements, in this case, the inflatable device and the users. The users interacting with the Muscle quickly learn how the Muscle reacts to their actions, and a game is established in this communication.⁸ We are in a case of bidirectional relationship, in which the movements of each of the protagonists (human and non-human) are part of the same process.

Extended architecture

Hylozoic Ground by architect and sculptor Philip Beesley was selected to represent Canada at the 2010 Venice Biennale in Architecture. (4) Beesley's installations are immersive environments that question the boundaries between the natural and the artificial, the human and the non-human. These hybrid devices are micro creatures,half environments, half mechanical, half biological. The visitor, by his presence, movement and breathing,activates a wave of reactions from the device. He is thus inscribed in a respiratory cycle and through his interactions becomes one with the installation within the same metabolism.

The environment supports a mutual empathic relationship that initiates a reactive movement, an exchange of particles and an air cycle between the system and the visitor. The visitor's own body limits are questioned. Philip Beesley named this type of behaviour: diffusive architecture.

Conclusion

The opposition between the living and the artefact is replaced by an isochronism of natural and artificial phenomena that continues to hybridize. Symbiotic architecture could be seen as a cyber-physical-human system within the natural artificial continuum. The opposition between the living and the artefact is replaced by an isochronism of natural and artificial phenomena that continues to hybridize. With these four examples, we have seen that architectural devices can support relational processes at different scales (within the material itself or with external elements), in a unidirectional or bidirectional way, beneficial to a single actor or to several. What the 4E approach shows is that architectural symbiosis needs to be based on a multiplicity of interactions and actions. Borrowing the concept of the 4 E's (embodied, embedded, enactive, and extended) from the cogniticians, we wanted to sketch a reading and writing grid for architectural design, our next research object. It would allow gathering under one concept the different current appellations such as dynamic, flexible,sensitive, transformable, adaptable, interactive, intelligent and moreover.

(1) https://www.ecologicstudio.com/projects/urban-algae-folly-2-0
(2) https://www.icd.uni-stuttgart.de/projects/hygroskin-meteorosensitive-pavilion/
(3) https://www.oosterhuis.nl/nsa-muscle-centre-pompidou-paris/
(4) https://www.philipbeesleystudioinc.com/sculpture/hylozoic-ground-venice-biennale/ 

References

1 C. Larrère, R Larrère, Du bon usage de la nature : Pour une philosophie de l’environnement, Flammarion, 2009.

2 M-A. Brayer, F. Migayrou, Archilab 2013 : Naturaliser l’architecture, Vol. Rencontres internationales d’architecture Orléans, Editions HYX, 2013.

3 A. R. Damasio, L’erreur de Descartes : La raison des émotions, M. Blanc (trad.), Editions Odile Jacob, 2010.

4 A. Prochiantz, Machine-esprit, Editions Odile Jacob, 2000.

5 P. Haselager, J. Van Dijk, I. Van Rooij (s. d.), "A Lazy Brain? Embodied Embedded Cognition and Cognitive Neuroscience", In Handbook of Cognitive Science, 273-290.

6 E Couchot, La nature de l’art : Ce que les sciences cognitives nousrévèlent sur le plaisir esthétique, Hermann, 2012.

7 M. Serres, Petites poucettes. Editions le Pommier, 2012.

8 V. Parlac, Surface Change : Information, Matter And Environment. In R. Stouffs & P.Janssen... (éd.), Proceedings of the 18th International Conference on Computer-Aided Architectural Design Research in Asia, CAADRIA, 2013.

A [Potential] Cloud War Controversies and conflicts related to climate manipulations.

Marie-Julie Bourgeois

Paris-Saclay University, France mail@mariejuliebourgeois.fr

Abstract

The OuCliPo research project (Ouvroir de Climats Potentiels) is a workshop of climate potential, supported by the Labex LaSIPS, presented by Marie-Julie Bourgeois, PhD in Aesthetics, Science and Technology of the Arts, researcher in art and design science at University Paris-Saclay. This project is also directed by Jules-Rémi Bois-Rouge, a (fake) climate geo-engineer, Doctor at the LaPataFlu, a (fake) Laboratory of Fluid Pataphysics, stemming from the modern surrealist literary movements, proposes a "science of imaginary solutions which symbolically grants to lineaments the properties of objects described by their virtuality."1
OuCliPo studies surrealist issues to climate problems; pseudo-scientific solutions and their implementation in the context of eco-anxiety. The project highlights the ethical and geopolitical dimensions of solar geo-engineering, as well as the socio-cultural issues associated with these climate experiments as techno-solutionism. 

Keywords

Climate, Geo-engineering, Global warming, Aerosol, Environment, Temperatures, Clouds, Particles, Fiction, Manipulation.

DOI

10.69564/ISEA2023-5-short-Bourgeois-A-Potential-Cloud-War

Figure 1: Nubus (fake) French startup. © Marie-julie Bourgeois 

Introduction

The 1995 Nobel Prize in Chemistry, Paul Crutzen, warns about the climate situation and the need to find viable solutions in a seminar2; to the European Parliament, he proposes to seriously study solar geoengineering as a third way.3

Solar geoengineering is a highly controversial technology that offers a concrete and—almost—immediate solution to the current problem of global warming. Although theoretical, this technology is being actively studied in research laboratories and by IPCC experts "Aerosol injection into the stratosphere is the most convincing solar geoengineering technique, it has the potential to significantly affect the climate in relation to the ongoing warming. It could offset a rise in global temperatures. It may offer a faster cooling capacity than CO2 mitigation." This major technological innovation allows an efficient cooling of the entire globe by increasing its natural reflective power (albedo).

This theory is based on the study of climate cooling periods following major volcanic eruptions such as the Pinatubo eruption, which cooled the globe by 0,5°C. In 1991, the release of sulfur particles (SO2) into the stratosphere increased the reflectivity of the Earth's surface and thus prevented some of the sun's rays from entering the atmosphere. The temperature of the planet dropped by -0.6°C for fifteen months.⁵, ⁶ 

Figure 2: Eruption of the Pinatubo volcano, Philippines, 1991 © J. Durieux-Sipa press. Larousse.

Players in geo-engineering

Solar geo-engineering solutions have developed rapidly over the last few years, with the aim of curbing global warming as a matter of urgency. Laboratories—or startups—specialized in solar geo-engineering, Solar Radiation Modification (SRM) or Stratospheric Aerosol Injection (SAI) attempt to "fix the climate" by generating anthropogenic clouds composed of sulfur particles.

These promise to create a protective cloud cover against solar radiation to increase albedo. SAI was patented in 1991, as a method of Stratospheric Seeding, it involves seeding the stratosphere with small metal oxide particles, such as thorium, barium or aluminium flakes, with the aim of reflecting sunlight and reducing global warming. Plans to implement the proposal were not foreseen or publicly known. In the years that have passed since, global warming has caused major concern and scientists have been looking for solutions to "repair" the planet. Researchers mainly use numerical modelling for obvious legal, ethical and health reasons. If inhaled, sulfur dioxyde (SO2) may lead to chronic disease, embedding itself in our lungs. Some laboratories are conducting risky experiments on a human scale without legal authorization in a frantic race to experiment with climate:

Figure 3: Make Sunsets, Luke Isemans and Andrew Songs launching stratospheric balloons from Nevada, February 2023. © Balazs Gardi for TIME 

All these projects should be illegal if we respect the moratorium on geoengineering, the UN ministerial meeting held in Nagoya, Japan, in October 2010 "Any experimentation, private or public, or adventurism aimed at manipulating the planetary thermostat will constitute a violation of this carefully crafted consensus in the context of the United Nations," asserted Silvia Ribeiro, ETC Group Director for Latin America.

Seducing words such as (Make Sunsets), the acronyms (SPICE, PEACE, SABRE, SATAN) and the vocabulary employed by researchers (heaven, hell, inferno, cult...) refer to beliefs and myths. The desire to remedy global warming by making amends for past pollution is strong. Guilt and the hope of becoming a hero by saving the planet and its inhabitants from imminent danger guide these geo-engineers / climate sorcerers. 

Costs

The cost of deploying these technologies is not excessive. Independent organisations can leverage them without requiring participation or funding from institutions. According to studies19, 20 it is sufficient to inject $1500 per ton of particles into the upper atmosphere each year for a gain of 2w/m2, in order to benefit from a significant decrease of 2°C, and the entire solution would cost between 2.25 and 11 billion dollars per year. The cost seems reasonable in view of the climate issues and international interests.

Risks

The main obstacles are the ethical, causal, and governance issues21 associated with climate manipulation, as well as the difficulty of large-scale experimentation and impact studies. Expert scientists are unanimous on the hopes and the risks,22 the danger to the biosphere,23 security framework,24 news biophysics risks,25 side effects, as well as possible thermal recovery in the event of an abrupt shutdown.26 Beyond health impacts, the societal risks of deploying these technologies are a moral issue,27 which provides a miracle "solution" that avoids reducing GHG emissions, socio-technical lock-in; there are also considerations regarding the irreversibility of the techniques deployed, and the termination shock.28 Furthermore, this technology, if developed without international agreements, will cause large-scale damage across borders (GIEC 20122). As the climate is interdependent on local meteorological events, the effects will be global. It is therefore urgent that nations position and align themselves in this geoengineering race based on common goals, in order to avoid wild tests, researching risk and preventing side effects. If experimentation goes unchecked, these solutions could spark a future global war,29 a scenario that is certainly no longer a fiction given the current geopolitical context. Since global warming is considered a threat to national security, the US military is involved in the development of solar geoengineering as part of its militarisation strategy.30 The prospect of rebalancing the forces belonging to competing powers such as China or Russia is probable, and the risk of escalation is a consequence of solar counter-geo-engineering programs31... A potential cloud war32 is brewing.

Look the sky Major volcanic eruptions

The year 536 has been called "the worst year in history"33 by archaeologists. It marks the Little Ice Age caused by several simultaneous volcanic eruptions. These volcanic eruptions caused 18 months of darkness on the surface of the globe due to the release of acidic sulfur particles, world temperatures then dropped by -2.5°C, causing famines, two centuries of plague epidemics, and the fall of several civilizations on all continents.3⁴ This period also marks the beginning of the Dark Ages within the Middle Ages. The architect Philippe Rahm3⁵ recalls the links between architecture, climate, volcanic eruption and epidemics: The eruption of Tambora in 1815 changed the world 3⁶ which benefited cholera. The global temperature dropped by 1°C for several years, prolonging the winter and disrupting the climate cycle on a global scale, the “year without summer” will be remembered.

Are we really ready to play with fire by experimenting with particle diffusion in the stratosphere? The health, meteorological3⁷, agricultural3⁸ and civilizational risks are not well enough measured or understood. Dangers are not limited to the survival of ecosystems and populations, not all risks have been studied and identified by geo-engineers. 

Science and fiction

That same dark and rainy summer of 1816, in this troublesome climatic context, a group of writers met in Switzerland; Mary Shelley wrote Frankenstein or the Modern Prometheus3⁹ and Lord Byron popularised The Vampyre.⁴⁰ We can say that one of the founding myths of Science Fiction was born in a dramatic climatic context! The following year, in 1817, in the midst of global climate change, William Turner began to paint coloured sunsets, amongst other Impressionist precursors influenced by these surrealist red-to-green skies.⁴1 Edvard Munch painted The Scream after viewing and being influenced by the strange sky "turning blood red." This famous expressionist portrait and its red sky probably stems from a vision of the Oslo Fjord, after the volcano eruption of Krakatoa in 1883,⁴2 or may be due to a nacreous cloud.⁴3 Climate and fiction seem intimately linked in the narrative imagination but also in the psychology of aesthetics and creative expression. If SRM/SAI technologies are implemented, and the atmosphere becomes opaque and dangerous to inhale due to particles, will the atmosphere lose its transparency? Our “common sky” will be whiter.⁴⁴ Today’s blue skies are already an illusion due to Tyndall effect and Rayleigh scattering and it is based on the size and density of particles... If the atmosphere becomes concentrated in CO2, will the sky turn orange like the CO2 saturated Martian sky? If our vision becomes saturated or obstructed by particles, how can we project ourselves towards new horizons, new common futures?

We can conclude that the power of design influences the critical thinking process, whilst the urgency of the situation also blinds the mind. The Make Sunsets website is far too well-designed for a typical scientific research project, and its warm orange design (the colour of sunsets) is as seductive as the promises of rescue it makes. Perhaps the design and aspirations explain its success and ability to raise funds. Despite the cobbled-together nature of its technology and the danger of its reckless actions. However, these orange-red colours of the startup should worry us, just as the Scream 😱 reveals the anxiety-inducing dimension of modern existence. 

Realistic satyr and fiction

OuCliPo is an art and design research project that questions the impact of particles on our vision and our cultures at the scale of landscape, architecture and habitat, by presenting speculative fiction based on climate manipulation, studies and data. 

Figure 4: Nubus the (fake) startup http://www.nubus.fr © Marie-julie Bourgeois

Nubus⁴⁵ is a (fake) research startup, created to test the credibility of the technology and also the attractiveness to the general public via a seducing marketing campaign "for a bluer and a cooler sky." For this purpose, a questionnaire was used to measure support for the Nubus startup and its promises. The French (fake) startup projects us to the stage of commercialization and industrialization of SRM/SAI technologies in order to confront us with the mutations they will generate via environmental and society disturbances. How far are we willing to go, to believe it?

After viewing the Nubus promotional film, a sample of 34 people were interviewed during the exhibition of the project at the ENS Paris-Saclay and at Evry University:


In conclusion, a relatively high percentage was inclined to participate in climate optimization in order to promote cooler and bluer skies. Although a minority, 21% of the respondents, believes that geo-engineering and SRM/SAI do not seem dangerous for our biosphere, or the climate... When the video was recently shown in an innovative art and science context, audiences reacted with ambivalence: amused, horrified or seduced, many believed in it. We decided to exaggerate the tragi-comic potential of the startup’s arguments to defuse support for the promise of rescue, the urgency of the situation resulted in a strong desire for solutionism.

Nubus is also developing green kerosene and biofuels to generate clouds of particles with high nucleation potential, rather than harmful greenhouse gases. Our emissions are guaranteed to have high clouding potential. We also work with the agrochemical and agricultural industries to combat soil erosion and desiccation. Our mineral and nutrient stimulants have a high precipitation potential which is ideal for farmers in times of drought.

"Our partners in the tobacco industry offer electronic cigarettes that contribute on a small scale to the generation of climate-friendly particle clouds, because there are no small contributions.

Nubus is launching a fund-raising campaign to "save" the planet and offers citizens in search of meaning the chance to clear their consciences by blue washing our vision of the future.

You can also take a sodium cure with our blue pills, which guarantee accelerated awareness to wipe out ecological procrastination..." (3)

Even with these arguments, the public still wondered if Nubus really did exist... 

Figure 5: Photomontage Nubus Blue pills against ecological procrastination © Marie-julie Bourgeois 

Homogenitus are clouds generated by human activities defined by the WMO in the international cloud atlas.⁴⁶ Homogenitus⁴⁷ it is also the name of an artistic cloud-making machine. Players can generate customizable clouds directly on a web application, choosing morphology, size, shape and the composition of particles. This artistic installation questions the creation of clouds by human activity, the ability to control the weather and places the user in a demiurgic position. 

Figure 6: Homogenitus the interactive cloud making machine © Marie-julie Bourgeois 

The citizen artivist collective fakeCloud⁴⁸ denounced Nubus activities, it’s SRM/SAI projects and climate manipulations. Nubus is caused of over-producing clouds related to the Iran accusation against Europe to steal their cloud in 2011.⁴⁹ 

Fig 7: FakeCloud the artivist collective http://fakecloud.fr © Marie-julie Bourgeois 

Research of solutions

In the absence of government control or international rules, companies can deliberately spread or steal clouds. How will our sky look in a decade? Will a future cloud war occur? This awareness campaign proposed to approach the ecological problems by addressing these issues through wordplay, irony, detours, fiction, and references to scientific methodologies and technologies. Designing our potential future atmosphere opacity and tint aimed to sensitize the public to these rapid ecological solutions, and promote awareness of the seductive dimension of design.

Our ideologies, our beliefs, our knowledge are challenged by the crises we are experiencing; climate, health, life, economy, institutions and especially science. At the same time when technologies and social networks allow collaborative actions, we question the methods of sensitization of the public about the techno-solutionism challenges. Fictions and post-truths mingle with reality in the scientific field. How can we find a sustainable way in this mess? Awareness and collective action are the determining factors for reducing greenhouse gases, but solar geo-engineering projects risk slowing down these efforts by proposing alternative and risky paths.

Climate issues are frequently addressed in a dramatic way, generating various typologies of reactions⁵⁰: alarmed, worried, cautious, non-committal, unconvinced, and contradictory. On an individual level, ecological procrastination is mainly due to discourses⁵2 that delay massive climate action; techno-solutionism is one of them. Robert Gifford observes seven psychological barriers to ecological inaction⁵3: limitations of cognition, ideological views, comparison with significant others, costs, distrust of experts and authority, perception of risk, and behavioural limitations. For Gifford, these "dragons of inaction" are the social-psychological brakes that impede real and meaningful global action.

Conclusion

OuCliPo proposes to approach climate threats with non-guilty and non-anxious modes of communication such as fiction and humour. OuCliPo does this by making a mockery of the search for technological solutions. Between solutions that are “not solutions,” where the cure is worse than the disease, denial and even counter-action, massive actions, whether positive or negative, have a strong impact on the climate.

To this end, Nubus offers its customers the chance to redeem their consciences by fixing the planet. It provides citizens in search of meaning with a conscience cleanser, by washing our vision of the future in blue for a dual benefit: a thermal solution and the psychological comfort of being able to remedy climate change by looking at the sky, not to don’t look up! (4). Ideologies, design and the power of capitalism helps us to believe in these seductive solutions. The market in climate denial⁵⁴ and the negation of mitigation policy are solutions that benefit climate skepticism.

Media and digital tools are good ways of approaching social issues with critical distance and speculative projection, but they can also fuel disbeliefs and conspiracies. Moreover, fiction can play a major role in raising collective awareness of issues at stake, by accelerating political decision-making. Confronted with the crises we are experiencing, and the eco-anxieties that are paralysing actions, we need to set off in search of "eco-quietude" and harmony amongst the disorder. This quasi-spiritual philosophical notion organised the celestial spheres as early as Pythagoras; between music, numbers, colour and astronomy, harmony was already giving rhythm to the mental and physical health of mankind in antiquity.⁵⁵

The harmony and symbiosis between man and nature, with sustainability and critical distance, would allow us, through fiction and humour, to approach climate problems from a truthful background. 

(3) ISEA 28th International Symposium on Electronic Art, May 17, 2023, Paris forum des halles

(4) Don’t look up ! Adam McKay 2021

References
Books

1 Alfred Jarry, Gestes et opinions du docteur Faustroll, pataphysicien, Livre II, chapitre VIII, Paris, Éditions Fasquelle, 1911.

35 Philippe Rahm, Histoire naturelle de l’architecture, Comment le climat, les épidémies et l'énergie ont façonné la ville et les bâtiments, Paris, Edition du Pavillon de l’Arsenal, 2020, p.167.

36 Gillen D'Arcy Wood, Tambora – The Eruption That Changed the World, Princeton University Press, 2015.

39 Mary Shelley, Frankenstein or the Modern Prometheus, Lackington, Hughes, Harding, Mavor & Jones, 1818.

40 John William Polidori, The Vampyre a Tale, The New Monthly Magazine, 1819.

55 Yuval Noah Harari, Homo Deus, A brief history of tomorrow, London, Vintage, 2017.

Journal article (online)

3 P. J. Crutzen, “Albedo enhancement by stratospheric sulfur injections,” Climatic Change, vol. 77, no 3-4, July 25, 2006, 211-219, doi:10.1007/s10584-006-9101-y

5 Mc Cormick and al, “Atmospheric effects of the Mt Pinatubo eruption,” Nature 373, February 2, 1995, 399–404. doi:10.1038/373399a0

12 J. A. Dykema and al. “Stratospheric controlled perturbation experiment: a small-scale experiment to improve understanding of the risks of solar geoengineering,” Phil. Trans. R. Soc, December 28, 2014, doi:10.1098/rsta.2014.0059

19 W. Smith, G. Wagner, “Stratospheric aerosol injection tactics and costs in the first 15 years of deployment,” Environmental Research Letters, Vol. 13, No. 12, November 23, 2018, doi:10.1088/1748-9326/aae98d

20 W. Smith and al, “A subpolar-focused stratospheric aerosol injection deployment scenario,” Environmental Research Communications, Vol 4 number 9, September 15, 2022, doi:10.1088/2515-7620/ac8cd3

22 M. De Guglielmo Weber, “Promesses et risques de la géoingénierie solaire,” IRIS, Programme climat, énergie & sécurité, February 2023, accessed August 22, 2023, https://www.iris-france.org/wp-content/uploads/2023/02/19_ProgClimEnerSec.pdf

23 M.G. Lawrence, and al, “Evaluating climate geoengineering proposals in the context of the Paris Agreement temperature goals,” Nature Communications 9, Septembre 13, 2018, 3734, doi:10.1038/s41467-018-05938-3

24 Chalecki, and al, “A New Security Framework for Geoengineering,” Strategic Studies Quarterly, vol. 12, no. 2, 2018, JSTOR, Accessed 22 Aug. 2023, 82–106, http://www.jstor.org/stable/26430817

25 E. Baughman, et al, “Investigation of the Surface and Circulation Impacts of Cloud-Brightening Geoengineering,” J. Climate, 25, November 1, 2012, 7527–7543, doi:10.1175/JCLID-11-00282.1

27 C Preston, “Ethics and geoengineering: reviewing the moral issues raised by solar radiation management and carbon dioxide removal.” WIREs Climate Change. 4, November 8, 2012, 23-37, doi:10.1002/wcc.198

28 A. Parker, & P. J. Irvine, “The Risk of Termination Shock From Solar Geoengineering,” Earth's Future, 6, March 11, 2018, 456–467, doi:10.1002/2017EF000735

30 Stephens, and al, “Towards dangerous US unilateralism on solar geoengineering,” Environmental Politics, December 18, 2022,  doi:10.1080/09644016.2022.2156182

41 C. Zerefos, end al, “Further evidence of important environmental information content in red-to-green ratios as depicted in paintings by great masters,” Vol. 4 issu 6, Atmos. Chem. Phys. 14, March 25, 2014, 2987–3015, doi:10.5194/acp-14-2987-2014

42 Donald W. Olson, and al, "The Blood-Red Sky of the Scream," APS News. American Physical Society, Vol. 13 number 5, May 2004, accessed August 22, 2023, https://www.aps.org/publications/apsnews/200405/backpage.cfm

43 F. Prata, and al, “The Sky in Edvard Munch’s The Scream,” Bulletin of the American Meteorological Society 99, 7, July 1, 2018, doi:10.1175/BAMS-D-17-0144.1

44 B. Kravitz, and al, “Geoengineering: Whiter skies?”, Geophys. Res. Lett., 39, L11801, June 1, 2012, doi:10.1029/2012GL051652

50 E. W. Maibach, and al, “Identifying Like-Minded Audiences for Global Warming Public Engagement Campaigns: an AudienceSegmentation Analysis and Tool Development,” PLoS ONE, March, 10, 2011, doi: 10.1371/journal.pone.0017571

51 Richard W. Malotte, “I’ll Save the World from Global Warming—Tomorrow: Using Procrastination Management to Combat Global," June 1, 2017, doi:10.1007/BF03392214

52 WF Lamb, and al, “Discourses of climate delay,” Global Sustainability 3, e17, June 8, 2020, 1–5, doi:10.1017/sus.2020.13

53 Robert Gifford, “The Dragons of Inaction: Psychological Barriers That Limit Climate Change Mitigation and Adaptation,” American Psychologist, May 2011, doi:10.1037/a0023566

Magazines and Newspapers (online)

2 European Parliament, “Professor Paul Crutzen: Nobel winner and advocate of a climate "escape route"”, January 12, 2011, accessed August 22, 2023, https://www.europarl.europa.eu/news/en/headlines/society/20110103STO11194

6 NASA Observatory, Langley Research Center, Aerosol Research Branch, June 2001, accessed August 22, 2023, https://earthobservatory.nasa.gov/images/1510/global-effects-of-mount-pinatubo

13 Letter from Sam Council to SCoPEx advisory committee, February 24, 2021, accessed August 22, 2023, https://static1.squarespace.com/static/5dfb35a66f00d54ab0729b75/t/603e2167a9c0b96ffb027c8d/1614684519754/

15 Chris Hatch, “Whose call to shade the Earth?” March, 3, 2023, accessed August 22, 2023, https://www.nationalobserver.com/newsletters/zero-carbon/2023/03/03/whose-call-shade-earth

16 Gobierno de Mexico, “La experimentación con geoingeniería solar no será permitida en México” January 13, 2023, accessed August 22, 2023, https://www.gob.mx/semarnat/prensa/la-experimentacion-con-geoingenieria-solar-no-sera-permitida-en-mexico

18 James Temple, “A startup says it’s begun releasing particles into the atmosphere, in an effort to tweak the climate.” MIT Technology Review, December 24, 2022, accessed August 22, 2023, https://www.technologyreview.com/2022/12/24/1066041

29 Alexander C. Kaufman, “The Climate Change Solution That Could Spark Global War” OneZero, June 16, 2019, accessed August 22, 2023, https://onezero.medium.com/the-climate-change-solution-that-could-spark-global-war-bd1cc9cce1a3

49 Barney Henderson, “Mahmoud Ahmadinejad says Europe 'stealing Iran's rain’”, The telegraph, May 21, 2011, accessed August 22, 2023, https://www.telegraph.co.uk/news/worldnews/middleeast/iran/8527455/Mahmoud-Ahmadinejad-says-Europe-stealing-Irans-rain.html

32 “La guerre des nuages, par Mathieu Simonet,” Nouvel Obs, March 28, 2022, accessed August 22, 2023, https://www.nouvelobs.com/tribunes/20220328.OBS56297/la-guerre-des-nuages-par-mathieu-simonet.html

54 Marine de Guglielmo Weber, “Crédits de refroidissement : quand la géo-ingénierie commercialise le déni climatique,” IRIS France, February 17, 2023, accessed August 22, 2023, https://www.iris-france.org/173813-credits-de-refroidissement-quand-la-geo-ingenierie-commercialise-le-deni-climatique/

Websites

4 IPCC, AR5, SYR, 3.3, 2014, accessed August 22, 2023, https://www.ipcc.ch/report/ar5/syr/

7 Interactive world map on geoengineering, accessed August 22, 2023, https://map.geoengineeringmonitor.org/srm

8 IGCE, Institute of Global Climate and Ecology accessed August 22, 2023, www.igce.ru

9 SPICE, Stratospheric Particle Injection for Climate Engineering accessed August 22, 2023, www.spice.ac.uk

10 E-PEACE, Eastern Pacific Emitted Aerosol Cloud Experiment accessed August 22, 2023, aerosol.ucsd.edu/E_PEACE.html

11 SCoPEx, Stratospheric Controlled Perturbation Experiment, accessed August 22, 2023, https://www.keutschgroup.com/scopex

14 SABRE, Stratospheric Aerosol processes Budget and Radiative Effects,https://csl.noaa.gov/projects/sabre/

17 Make Sunsets American startup of solar geo-engineering, accessed August 22, 2023, https://makesunsets.com/

21 Reynolds, J. et al. “Solar Radiation Modification: Governance gaps and challenges – Summary.” Carnegie Climate Governance Initiative (C2G), March 2022, New York, www.c2g2.net

26 Felgenhauer, T. et al. “Solar Radiation Modification: A Risk-Risk Analysis - Summary, Carnegie Climate Governance Initiative” (C2G), February 2022, New York, 2023, www.c2g2.net

45 Nubus is a (fake) French startup of solar geo-engineering, accessed August 22, 2023, https://www.nubus.fr/

46 World Meteorological Organization, International Cloud Atlas, https://cloudatlas.wmo.int/en/homogenitus.html

47 Homogenitus is an art installation, https://homogenitus.fr/

48 Fake cloud Collective of artivist fighting against climate manipulations, accessed August 22, 2023, http://fakecloud.fr

Author Biography

The author is a digital artist, designer and PhD researcher in aesthetics and science technologies of the arts since 2018. She works on ecological, artistic and scientific fictions in society in relation to human activities. Born in Paris in 1981, she lives and works in Cachan, France. In 2008, she obtained a Master's degree in New Media at the Ecole Nationale Supérieure de Création Industrielle (ENSCI) and in 2009, she joined the EnsadLab research program. Her research focuses on "solar fictions, devices that recreate the sun’s path", and she is currently working on climate fictions. She teaches at the University of Evry (91) since 2013 and co-founded the CondéDesignLab in 2021. 

Inhabiting the Edges

Célia Boutilier

SACRe-PSL ENS/ENSBA — Paris Boutiliercelia@hotmail.fr

Abstract

My doctoral project in art-creation-research, which began in October 2021, is entitled "Imaginal photography: for an aesthetic of symbiosis." In my practice as a photographic artist, making symbiotic relationships, unexpected alliances, allows me to provoke a "suspension of evidence" which, through the intermediary of the image, invites the viewer to share the experience of a different look. From this point on, the very nature of perception is to admit ambiguity. The interdependencies in ecosystems, the principle of hybridization between different species are themes addressed by the team of the laboratory of plant symbioses of the French National Museum of Natural History of Paris with which I collaborate. In my research, I'm specifically interested in the symbioses present in the orchid with the biologists Florent Martos and Eve Hellequin. Transcending the notion of organism allows us to reconcile human beings with their natural dimension of interdependence and interaction. The researchers in this laboratory are the extractors and curators of natural plant resources, with which they shape new worlds such as the Grande Serre: a space in which I would like to exhibit works in 2023-2024. These hybrid spaces, co-dependent with humans, seem to me to be fertile places to dialogue with.

Keywords

Photography, imaginal, hybridization, symbiosis, speculation, pragmatism, device, articulation

DOI

10.69564/ISEA2023-6-short-Boutilier-Inhabiting-the-Edges

Introduction

"We inhabit the world from the images we make of it."

This doctoral project starts from the intuition that aesthetics, as a study of sensible knowledge and its artistic productions, is one of the main foundations of the imagination that allows us to think the world and to inhabit it. It is from this imagination that imaginative practice is born. Without the imaginative faculty, a whole area of reality is definitely off-limits to us, lost forever. With the loss of the imaginative faculty, what we risk is not the loss of fiction, but the loss of reality. We run the risk of compartmentalizing the sensible and intelligible worlds, of killing them off slowly, precisely because they only come to life and "come alive" when they are in contact with each other.

Methodology

It seems to me today that the question of representation (of whom speaks and how he or she expresses himself or herself) actually raises the question of the imaginary, that is to say, that of the fabrication of new fictionalities. The issue of imagination is crucial, because it opens up perspectives of therapeutic narratives. These proposals for circulation (in the making itself or in the setting of creations) are multiple hypotheses of resilience. This is why certain fictionalities can be productive of the real and allow us, by this means, to increase ourselves. Their power of augmentation is not only intimate, a creation has an impact when it goes beyond its subject and manages to make language for something else. It’s therefore a question of photographing this place with the desire to feel it fully, while knowing that from this precise place we will touch other places, both terrestrial and human.

On the other hand, I’m interested in the interdependencies in ecosystems and the principle of hybridization between species, because I believe that there is a realaesthetic, philosophical and political stake in linking Art (culture in general) to Ecology. The laboratory and the studio are the two places where photographic art was born. Botanical science and in particular the concept of symbiosis seems to me to be able to achieve this necessary alliance (in biology the word symbiosis defines a lasting interspecific interaction of mutual benefit). In this sense, I’m conducting this doctoral research project in collaboration with the team of the plant symbiosis laboratory of the Paris Natural History Museum. I’m specifically interested in the mycorrhizal symbioses of orchids: those that create networks between individuals and different species, notably between fungi and plant roots. From the concept of symbiosis in biology, I intend to develop the concept of symbiosis in the field of aesthetics: "what could be a symbiotic aesthetic? I propose the hypothesis that a "symbiotic aesthetic" constitutes, on the one hand, a singular way of relating philosophically to the world (at the source of an enlarged perception), and, on the other hand, a programme of plastic realisation. How can symbiosis suspend the rational relationship of things to each other? To what extent does this suspension make it possible to escape from designation and nomination? How can the new representation of the world proposed by symbiosis change the way we act and interact? This project aims to sketch some answers, both formal and textual, based on fieldwork with biologists, both in the field and in the laboratories of the MNHN of Paris.

Communication plan

More specifically, I'd like to present a few recent works produced following a scientific mission to Reunion Island, which fall under what I call a "symbiotic aesthetic." Photographic assemblages are the first satisfactory way I've found of aesthetically translating hybrid identity, through which the intertwining of links manifests itself. Boundary images between the "natural," the imaginary and the technological. Different photographic fragments are linked together by collage. The aesthetic is organic and lively, evoking both art nouveau and virtual reality, where naturalistic photography meets pictorial inventiveness.

Focus of the work

In my work, it’s a question of "milieu" in that it takes us in its mesh.It’s a question of resisting any hierarchical placement of knowledge and exploring the plurality of deployments that they allow, replacing the aim of unity with the problematic of articulation.

I would like my images to be edges. Let them be meeting zones, spaces of contact, symbiotic and twilight places where the obvious is suspended and where, in a strange familiarity,things enter into relationship. I favour the observation of transitory and symbiotic forms,changes of state, the confusion of scales (from satellite to microscope) and all other forms of telescoping practices, insofar as they impact the solidity of bodies, the sharpness of contours and the fixity of images. The particular is then the bearer of more than just itself, in that the image is made from an elsewhere that crosses the here, in order to give an account of a here that is both always open and always worried.

"‘Border identity’ this is the term by which I usually define my own identity. I call it frontier, anchored, not in a place of rupture, but, on the contrary, in a space of permanent coming together. The border, as I define it and inhabit it, is the place where worlds touch, tirelessly. It's the place of constant oscillation: from one space to another, from one sensibility to another, from one worldview to another. It's where languages mingle, not necessarily thunderously, but naturally impregnating each other, to produce, on the blank page, the representation of a composite, hybrid universe."¹

Pace of time and ghosts

We inhabit the world through the images we make of it. In this sense, it would seem that constructing ways of inhabiting the world² also involves ways of making images. In the hollow of my plastic reflections, I apply myself to modelling a specific temporality: that of a time which has a thickness and in which memory and dreams are sedimented.

This time—like thought—is not linear (cause/effect), it is made up of feedback loops where the chain of causality swallows its own tail (systemic thinking). The past is not only what happened, it is also what was dreamed.

"There is always a shadow of something. And it stays for a long time. Now there is only the shadow of my mother, who passed on to me the shadow of my grandmother, etc., etc. You see images projected normally and then you see the same images that become a shadow ofthemselves. That's what Maniac Shadows is all about."³

Uncomfortable landscapes. There is a flaw, a break, a suspension of evidence. We must first remember our heritage, the memory of the rumbling soil, the organisms and ghosts that inhabit us and that we carry, in spite of ourselves, like amnesiac children. This requires, first of all, a refusal of postures or practices that are constituted in a cannibalistic relationship to otherness. This relationship is one of attention, concern and vigilance. The state of vigilance is a state of slowness, of contemplation that stretches over time. It is a profound attention that takes the opposite view of the lighted time of immediacy to become a space of common sharing. It is a care that we cultivate in the relationship that we maintain with the other whether human or non-human. Taking care of our relationships and their modes of existence goes hand in hand with taking care of the making of our images." The world to which we belong is first and foremost the world we carry within us."⁴

Figure 1. Identité Frontalière L.M (rupture), Mare Longue native forest, Reunion Island, 2023 © Célia Boutilier. Figure 2. Oeceoclades et ectomycorhizes, from the "naturecultureD.H" series, Electronic microscopy technical platform, French National Museum of Natural History, Paris. Assembly of six photographs, format 140 x 139.80 cm © Célia Boutilier
Figure 3. Échelle Humaine (etépiphytisme sur roches), from the "natureculture D.H" series, Mare Longue nativeforest, Reunion Island, 2023. Photographic assemblage, 140 x 70 cm © Célia Boutilier
References
Books

1 Léonora Miano, "HABITER LA FRONTIÈRE, Paysages francophones -Journée internationale de la francophonie, Université de Copenhague (Danemark)" conférence de 2009 recueillie dans HABITER LA FRONTIÈRE, L’Arche, 2012, p.25.

2 Bruno Latour, Où atterrir ? - Comment s'orienter en politique, La Découverte, 2017.

4 Ibid. Léonora Miano, 2012, p.25.

Webvideo interview

3 Chantal Akerman - ManiacShadows, Nuit Blanche 2013 au Théâtre du Châtelet, Paris.

Bibliography

David Abram, The Spell of the Sensuous: Perception and Languagein a More-Than-Human World, Vintage, 1997.

Jean-Christophe Bailly, L’imagement, Seuil, 2020.

Jean-Christophe Bailly, Le temps fixé, Bayard, 2009.

Jonathan Crary, Techniques de l’observateur, Vision et modernité au XIXe siècle, Broché, 1990.

Lorraine Daston, Peter Galison, Objectivité, Les presses du réel, 2012.

Gilles Deuleuze, Logique de la Sensation, 1981.

Gilles Deuleuze, Desert Islands and Other Texts, 1953-1974.

Cambridge, MA: MIT Press, 2004.

John Dewey, L’art comme expérience, 1934.

T. S. Eliot, The Waste Land, Faber & Faber, 2015 [1922].

Cynthia Fleury, "L’imagination renaissante : entreRévélation et Intellection" in Imagination, imaginaire,imaginal, Coordinated by Cynthia Fleury, PUF, Paris, 2016.

Donna Haraway, Staying with the Trouble: Making Kin in theChthulucene, Duke University Press, Durham, 2016.

Donna Haraway, When species meet, University of Minnesota Press, Minneapolis, 2007.

William James, Philosophie de l'expérience. Un universpluraliste, 2007.

Bruno Latour, Down to Earth: Politics in the New ClimaticRegime, ed. Cambridge, UK, Polity Press, 2018.

Bruno Latour, Facing Gaia: Six Lectures on the Political Theologyof Nature, Cambridge, UK, Polity Press, 2013.

Bruno Latour, Enquête sur les modes d’existence une anthropologie des modernes, La Découverte, 2012.

Bruno Latour, La science en action - Introduction à la sociologie des sciences, La Découverte, 2005.

Michael Marder, Plant-Thinking – A Philosophy of VegetalLife, Columbia University Press, 2013.

Michael Marder, La plante du philosophe - Un herbierintellectuel, Mimesis, 2020.

Lynn Margulis, Symbiogenetics - Origin of MitoticCells from Bacterial Communities in the ProterozoicEon, 2011 [1992].

Lynn Margulis, Symbiosis as a Source of EvolutionaryInnovation: Speciation and Morphogenesis, (TheMIT Press, 1991).

Baptiste Morizot, Manières d’êtrevivant, Arles, Actes Sud, 2020.

Giuseppe Penone, Jean-Christophe Bailly, Sève et pensée, Paris, broché, Paris 2021.

Bernard Réquichot, "Métaplastique" dans Écrits, 1955, p.87.

Simon Schaffer, Steven Shapin, Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life, 1985.

Anne Simon, Une bête entre les lignes. Essai dezoopoétique, Marseille, Wildproject, 2021.

Isabelle Stengers, La sorcellerie capitaliste : Pratiques de désenvoûtement, La Découverte, 2013.

Isabelle Stengers, L’invention des sciences modernes, Flammarion, 1993.

Andreï Tarkovski, Le Temps scellé.

Anna Lowenhaupt Tsing, The Mushroom at the End ofthe World: On the Possibility of Life inCapitalist Ruins, Old Saybrook, CT, TantorMedia, 2017.

Alfred North, Whitehead, Le concept de nature, J.Vrin, 2006.

Walt Whitman, Leave of grass The Original 1855, Edition, Paris, Broché, 2020.

Journal articles (print)

Carla Hustak, Natasha Myers, "InvolutionaryMomentum: Affective Ecologies and the Sciences ofPlant/Insect Encounters", Vol. 23, Num. 3, Brown University and differences, A Journal of Feminist Cultural Studies, 2012.

Sabine Plaud, "Vie du langage, vie des images : une marque de continuité dans la philosophie de Ludwig Wittgenstein".

Katrin Solhdju, "L’expérience "pure" et l’âme des plantes. James lecteur de Fechner", 2007.

Isabelle Stengers, Didier Debaise, L’insistance des possibles. Pour un pragmatisme spéculatif, Multitudes 2016/4, n° 65, 82-89

Webvideo interview

Bruno Latour, “Agency at the Time of theAnthropocene”, New Literary History 45, 2014, 1-18.

Author Biography

After a DNAP at ENSA Dijon (2016), a year at ERG Brussels (2017), a collaboration with LadHyX Polytechnique (2014-2019), Célia Boutilier joins the Beaux-Arts de Paris (2017) to defend a DNSAP (2019). Since 2014 she has been collaborating with research laboratories (microfluidics - Polytechnique, and mycorrhizal symbioses - National Museum of Natural History) where she is interestedin how imaging techniques participate in theelaboration of knowledge. Since 2016, herwork has been exhibited at fairs and international events, including the National Gallery in Copenhagen, the Cité Internationale des Arts in Paris and the Maison des Métallos. It has been quoted inseveral scientific journals, including Physics Today, The NewYork Times, Le Monde. Since 2020 it has also been included inprivate collections. In October 2021 Célia Boutilier starts a PhD in "Sciences, Arts,Creation, Research" (SACRe-PSL / ENS) at the National Superior School of Fine Arts in Paris (ENSBA).

Mytherrella: an interactive installation hallucinating mythological auroral formations

Dorin Cucicov

University of Bucharest, Romania Bucharest, Romania cucicov@gmail.com

Abstract

Up until very recently, mythological tales predominated in the explanation of the polar auroras. Today, we understand that the polar lights are a phenomenon brought on by the solar winds' interaction with the magnetosphere of the Earth leading to precipitation of energetic particles on the topside ionised atmosphere. However, for a long time, this phenomenon inspired vibrant trans-border stories with a profound impact on local communities. In our interactive installation, Mytherrella, we aimed to create a dynamic environment in which scientific data and mythological storytelling unearth new imaginaries about the aurora borealis. The generative video integrated in the installation uses a custom real-time StyleGAN algorithm that continuously samples from a model trained on a large set of all-sky auroral images acquired in Kiruna, Sweden. This method enables live interaction with the generated video, producing novel synthetic auroral formations that are unpredictable while remaining within the bounds of the learned features. By combining the dataset with a relatively small number of alternative-style images, the diversity of the generated content is increased, creating a divergent effect that reinforces the mythological narrative. We share the technique of interactive video generation as well as the research process behind the creation of the work.

Keywords

interactive installation, mythology, live StyleGAN, aurora borealis, polar lights.

DOI

10.69564/ISEA2023-7-short-Cucicov-Mytherrella

Introduction

Myths about the aurora borealis are prevalent in the cultural heritage of populations from the polar regions of the Earth. Surprisingly, these stories have not permeated pop culture in the same way the images of the aurora borealis did. Many of the stories, despite being from different continents, share many similarities: contact with the spirit world, vengeance of slain enemies, communication with the dead through whistling or playing ball games in the sky. For some, the appearance of the aurora borealis is a bad omen; for others, it can be bad only if disrespected by whistling at it or staying too long outside. Others have more poetic explanations for auroral formations.5, 6

This paper presents a virtual environment, the result of a creative exercise that leverages both scientific data and stories from various communities about the polar lights.

Combining them produces novel visions of a future in which humans reinvent themselves through collaboration with artificial artifacts. Mythology has been at the forefront of explaining the unknown and stimulating human curiosity for centuries. These stories are refined representations of humanity's deep connection to the natural world around it. The mystical characters in these tales instilled fear and caution in humans, effectively negotiating a reverence for natural phenomena. In the past centuries, science has solved numerous natural enigmas, relieving some anxiety but also making humans less respectful and more entitled.3

Figure 1. Interactive installation Mytherrella. On the left - outside of the booth; on the right — interactive control stand inside the booth. © Maria Năstase.

Our project, named Mytherrella, addresses the subject of symbiosis on multiple levels. It frames a creative experiment by integrating empirical data and mythological narratives. The equal treatment of science and myths aims for a reevaluation of our relationship with natural phenomena. Through the similarities between the stories of various cultures, cultural symbiosis is highlighted on a global scale, diminishing the significance of political borders.

Concept and construction

Mytherrella is an endless process of imagination informed by facts from science and driven by myths surrounding the aurora borealis. Starting from a dataset of all-sky images showing auroral activity recorded by the Swedish Kiruna auroral observatory, it imagines new auroral formations. It provides control over its hallucinations in an interactive framework. Both human and machine are in search of the lost/improbable auroras as seen by the first people that described them.

Mythology of the aurora

One of the most common mythological symbols associated with the polar lights is their connection to the spiritual world. Polar lights, according to the Greenlandic Inuit, were messages from their deceased friends and relatives.1; The Meskwaki, a Native American people with homelands in the Great Lakes region, held similar beliefs. They believed that by whistling to the lights, they could interact with them, thus communicating with the dead. Some sources indicate that some North American Inuit also used whispering during auroral events to send messages to the dead.⁵ More active manifestations of the aurora borealis were associated with spirits playing games in some Greenlandic stories, while on the European continent, in Scotland, Ireland and Norway they were associated with dancing spirits.1, 4

Some of the more poetic interpretations of the polar lights are found in Sámi peoples, residing in the north of Finland. They named the aurora - revontulet (fire fox). It is said that the lights are created by a fox with glittering fur running across the mountain area. 1; In other stories, polar lights are an omen of disaster and war. These are found in Tlingit indigenous peoples of the Pacific Northwest Coast of North America, Skolt Sámi in Northern Europe and even in some Chinese and Roman literature.1, 2

There may be several myths about the aurora in a single area, ranging from optimistic interpretations to terrifying ones. These are dependent on particular historical moments and the social and natural influences that shaped those times.

Building Mytherrella

The physical construction of the Mytherrella installation was inspired by the early designs of the auroral observatory in Yerkes, Wisconsin, United States.4 These small sites, fitted with all-sky cameras, collected images of the auroral events aimed for scientific investigation. In our installation, an immersive context is created inside a small booth where one can experience and interact with the story in a private setting. The outside of the booth is painted with selected mythological texts, which acts as a representative of the old media in contrast with the interactive digital content. An interactive stand inside the booth provides two analog potentiometers for controlling latitude and longitude on a minimalist digital map. Using these knobs, a specific location can be identified on the global map. The map is centered on the poles and only depicts the boundaries between water and land; it does not include any other information, such as names of nations, borders, seas, or rivers. This choice was made in order to disrupt the influence of the political borders in the collective perception and suggest a new perspective on interpreting the global map.

Ten locations are hidden on the digital map. They can be discovered by identifying their approximate location using the navigation controls. These coordinates are linked to locations from which we gathered mythological stories. Based on these texts, we used text-to-image tool Midjourney (1) to generate unique images. Figure 2 contains one of these images generated based on the mythological texts of the Native Americans describing the aurora borealis as a means of communication with the spirit world.

Figure 2. Image created using text based on the myth of spirits communicating through aurora borealis.

The second screen, positioned in front of the control stand, displays a continuously generated video with auroral formations. This video is created using a StyleGAN model that was trained on a mixed dataset of scientific and artistic images of the aurora borealis.8 The main corpus consists of 9834 selected images from a dataset of all-sky images depicting auroral activity provided by the Kiruna Atmospheric and Geophysical Observatory at the Swedish Institute of Space Physics. All images have a standard round format, as seen in Figure 3.

Figure 3. A selection from the all-sky images dataset provided by Kiruna Atmospheric and Geophysical Observatory at the Swedish Institute of Space Physics. Recorded during auroral activity in September, October 2020.

(1). Midjourney. https://midjourney.com

Data blending was applied using another 150 artistic images of the aurora borealis in order to extend the dataset and introduce visual features that break the rigor of the scientific representation. The process of projecting an image into the latent space produced images that were continuously served to create the video. When idle, the video would display auroral formations as learned from the main corpus of all-sky images. Finding an image on the world map while interacting with the installation would begin projecting that image to the model, disrupting the normal representation by also sampling from the artistic images' features.

Figure 4. The process of continuous video generation using live StyleGAN.

Figure 4 describes the approach of continuously serving images for the video projection. A script monitors for new files that are selected by interacting with the installation. Once a file is detected, it signifies that an image was identified on the map. This triggers an immediate process of projecting that image to the model. The resulting images are displayed sequentially creating the effect of video continuity. The 3 processes of the architecture are independent: generating based on the source image; serving an image for generation; displaying received images. They are linked only by the configuration of the listeners. This loose coupling makes it simple to integrate with other systems to serve various seed images or display generated images in various formats.

Discussion

The myths of the aurora borealis are rich and creative stories that have surrounded this natural event for a long time. Mytherrella is a direct comment on the relationship between science and mythology. Figure 5 exemplifies one sequence of a video generated by projecting the image from Figure 3. It starts from the format of an all-sky image and navigates the latent space in search of the features from the projected image. By using the technique of interactive generative video, we sought to create a subtle transition from visuals that have a scientific aesthetic to a more abstract look that resembles some of the features of the mythological image while also preserving the features of the aurora. This type of interaction contributed to the immersive character of the experience by partially delegating the responsibility of the generated image to the person interacting with it.

Some of the visitors discovered that by rapidly shifting through mythological images on the map, they could further distort the visual representation of the aurora, taking more control over the generative process. This encouraged visitors to spend more time with the work and explore both the map and the generative mechanism as they discovered more layers of interaction. By doing so, they also delved deeper into the mythological stories about the polar lights, drawing connections between different cultures that transcend political borders.

5. Selected frames from a sequence of the generated video using custom dataset of aurora borealis images and real-time StyleGAN.

Without nation-state borders or other human interventions, the world map used in the installation is purposefully rendered in a minimalist style. The fact that it is centered on the poles further distorts how visitors typically perceive the world map. The visitors identified common archetypes in geographically remote regions by experiencing mythological stories from various cultures.

Mytherrella is an experiment in creating a shared space for scientific data and personal stories. Such playful settings that mix diverse fields of study and research approaches may motivate us to delve deeper and from different angles into the stories behind them.

Acknowledgements

Mytherrella was developed during a residency in the Space Science Institute from Măgurele, Romania, organized by Qolony, Bucharest. Interactive sound for the installation was created by Iulia Smeu. Dataset of all-sky auroral images was provided by Kiruna Atmospheric and Geophysical Observatory at the Swedish Institute of Space Physics.

References
Books

1 Asgeir Brekke, Egeland Alv, The Northern light: from mythology to space research, Springer Science & Business Media, 2012.

2; John S. Major, Heaven and earth in early Han thought: Chapters three, four, and five of the Huainanzi, SUNY Press, 1993, 203-204.

3 James Burke, The impact of science on society, vol. 482, Scientific and Technical Information Branch, National Aeronautics and Space Administration, 1985, 3-32.

4 D. C. Rose, Keoeeit-The Story of the Aurora Borealis, by W. Petrie, Arctic 17, no. 2, 1964, 26-27.

Journal article (print)

5 Robert Holzworth, "Folklore and the Aurora," Eos, Transactions American Geophysical Union 56, no. 10, 1975, 686-688.

Journal article (online)

6 Tero Mustonen, "Inuit and Chukchi Star lore: Reflections on Ursa Major, the North Star and Northern Lights," Snowchange co-op, 2016. Accessed December 4, 2022, http://www.snowchange.org/2016/11/inuit-and-chukchi-starlore-a-new-discussion-paper-out/

Websites

7 Anniina Jokinen, "Aurora Borealis, The Northern Lights, in Mythology and Folklore," Luminarium, 4 Feb 2007, Accessed December 4, 2022, http://www.luminarium.org/mythology/revontulet.htm

8 Burloiu, Grigore, “controlling StyleGAN in real time,” Accessed December 4, 2022, https://rvirmoors.github.io/2021/01/04/realtime-stylegan/ 

At the Sources of an Artistic Mutation towards Science: the First Years of the Journal Leonardo (1968-1981) as a Forum for the Pioneers of Digital Art

Camille Frémontier-Murphy

RCM Galerie Paris, France camille@rcmgalerie.com

Abstract

The journal Leonardo was founded in 1968 by Frank Malina, a pioneer of light art in Paris and of aerospace science in the United States. Leonardo encouraged artists to publish their work in the manner of scientists. It was a new initiative developed in the revolutionary context of the 1960s and that allowed artists, scientists, psychologists to exchange on the subjects of art, perceptions, science, society... Many pioneers of digital art took part in the adventure, including Vera Molnar, Zdenek Sykora, Charles Csuri, and the artist-novelist Herbert Franke, who became the advocate of the theories and the protagonists of digital art spread out over the four corners of the planet, just about everywhere a computer could be found. The growing group was defending a more conceptual approach to art, closer to the spectator, a new form of art rooted in Constructivism and that was in symbiosis with society’s mutation towards technology.

Keywords

Pioneer, Journal, Revolution, History, Perception, Cybernetic, Constructivism, Instrument

DOI

10.69564/ISEA2023-8-short-Fremontier-Murphy-Artistic-Mutation-towards-Science

Introduction

When the journal Leonardo was founded half a century ago by the artist-engineer Frank J. Malina (1912-1981), the first generation of artists working with computers and electronic media was still small and far apart. Leonardo provided them with a permanent forum to exchange, share, and publicize their reflections and the state of their work with their peers and also a wider audience at a time when international and interdisciplinary means of communication were not readily available. While some of these pioneers have recently passed away, such as Herbert Franke, Ken Knowlton, Charles Csuri and Jean-Pierre Hébert, others are still active, such as Vera Molnar, Joan Truckenbroad, Ruth Leavitt, Jean-François Colonna, Jean-Claude Marquette, Hervé Huitric and Monique Nahas. Together they have defined the foundations of a community whose concerns and approaches from a not-so-distant past still resonate today.

The Journal Leonardo and the Buoyant Spirit of 1968

In January 1968, when Leonardo was launched, it was an effervescent time: changes and mutations in society were crystallizing. A few months before the youth took to the streets of Paris, the ambitious multimedia Living Art program of the ARC (Animation - Recherche - Confrontation), directed by Pierre Gaudibert, began organizing its exhibitions and interdisciplinary events at the Paris Musée d'Art Moderne. This initiative was revolutionary. The museum was no longer perceived as a temple, but as a laboratory that would invite a younger and more diverse public until 1972. At the same time, an art and technology program began to develop at LACMA under the direction of Maurice Tuchman. In London, the first major exhibition of electronic art and music took place at the ICA (Institute of Contemporary Arts) in the late summer of 1968, demonstrating the scope of the movement. In Zagreb, the series of exhibitions and symposia organized by the New Tendencies under the impetus of Abraham Moles (1920-1992) opened its fourth chapter dedicated to computer art.

Figure 1. Frank Malina with Orbits III, 1959, Boulogne (Photo Frank & Marjorie Malina archives, Boulogne).

Malina, a Paris-based American artist, was at the center of this ferment. A few years earlier, the Paris Musée d'Art Moderne had acquired one of his grid paintings, Deep Shadows (1954). He participated in New Tendencies with a selection of his luminous mobile paintings—Signals, Voyage II, Sink and Source. At Cybernetic Serendipity, he exhibited his interactive light work Entrechat and presented his new journal Leonardo. He even helped the exhibition to travel to San Francisco’s Exploratorium, recently founded by his longtime Caltech friend Frank Oppenheimer.

In fact, Malina himself was a former scientist turned artist. He had designed and supervised the launch of the first scientific rocket, the WAC Corporal, and had founded the Jet Propulsion Laboratory, in Pasadena, California, which is now the premier laboratory for the exploration of the solar system. After World War II, he opposed the use of his rocket as a ballistic missile with a nuclear warhead, and had worked for several years with Joseph Needham, head of the Natural Sciences Division of UNESCO in Paris, "to reduce barriers to the free movement of scientists and engineers between nations." Needham also introduced him to György Kepes, a member of the New Bauhaus in Chicago and the founder of the Center for Advanced Visual Studies at MIT. Kepes’ book, Language of Vision, was one of the most important artist's manuals ever written in the United States before Moholy-Nagy’s Vision in Motion (1947). His sociocultural ambition was clear: "The goal is a new vital structure-order, a new form on a social plane, in which all present knowledge and technological possessions may function unhindered as a whole"1.Inspired by these texts, and sometimes frustrated by the bureaucracy of international organizations, Malina had begun an artistic production at the intersection of art and science. Since the mid-fifties, he had been developing a corpus of moving light paintings with themes reminiscent of recent space travel.He was actually one of the first artist-scientists of his time, and soon came up with the idea of creating a society or journal for artist-scientists to meet and exchange ideas, but he couldn’t find many of them. With the preparation of exhibitions on cybernetics in London and computer art in Zagreb, a growing community of artists working on the frontiers of science emerged. Max Bense and Abraham Moles began to apply Norbert Wiener’s cybernetic and information theory to aesthetic problems. Together they edited the first issue of the magazine Bit international, first published in 1968 by the Museum of Zagreb within the framework of New Tendencies. Unfortunately, the initiative didn’t last after 1972.

The Emancipated Artist Joins Science

Leonardo, for its part, wanted to be open to all forms of art: "What I was interested in was people who worked in physics, or chemistry or physiology, or astronautics, or what have you, who introduced into their artworks the visual experiences and the ideas of their work. And I found many of these people were not doing that. So, I said that it was premature to try to make a journal for this kind of people. They weren't enough. And since there was no professional journal for any kind of visual artist, the thing was to make a journal that was open to any art, and there the artists would write about their work themselves, helped sometimes, and would be by someone in the direction of relationship between art, science, and technology. (...) And at the end I wanted an international journal to try to bring about better international cooperation, for peace."

Needham, who became famous for his history of Chinese science, suggested the name of the journal and would be one of its honorary editorial advisers2. “Creativeness in art does not appear to me to differ in kind from creativeness in science or any other human activity”, stated Malina in the brochure for his first solo exhibition in Paris in 1953”3. It was a time when the excessive division of knowledge reinforced by the specialization of disciplines, began to be questioned. In Gothic Architecture and Scholasticism (1951), Erwin Panofsky described a common denominator for different forms of human endeavor in a given period. Creation appeared to be culturally determined by mental habits common to several disciplines.Leonardo defended the development of art as any other field of research. It was published by the Pergamon Press of Oxford, which was already responsible for Acta Astronautica, the organ of the International Academy of Astronautic, co-founded by Malina and von Kármán. English was quickly to become the dominant language, as it was in the process of doing in the scientific field.Artists, as scientists, would publish their own articles to discuss their practice, no longer leaving this mission to critics alone. Artists would become literate persons able to discuss fundamental artistic, social and political issues.Contributors could be European or American, including African American, Asian, male or female. The invited authors could be computer artists, but not exclusively. Artists working in a constructivist vein or kinetic artists, such as Nino Calos or Pol Bury could also contribute. These two artists were also well known for their writings and will join Joël Stein, Anthony Hill, Vladimir Bonacic and Herbert Franke on the editorial board of Leonardo.Scientists were also welcome to contribute to Leonardo. The artist was now a social actor in his own right, able to interact with scientists. The psychologist James Gibson, who was one of the honorary editorial advisers, wrote a paper on visualization and the point of observation in 1974⁴. Fred Lawrence Whipple, head of the Smithsonian Astrophysical Observatory, shared in the first edition of Leonardo a rather interesting way of conceiving paintings from random arrangements of forms and colors. The point was certainly to clarify the inherent differences between randomness and chaos⁵. Its goal was to build bridges between different communities, to advocate peaceful international relations. Malina's convictions have not changed since he left JPL, when he said, “that ideas and effort were really needed now to find ways for sovereign states to function in peace together, rather than to develop better means of destroying themselves.”

A dedicated project for a dedicated community, Leonardo was a quite peculiar project of an artist dedicated to creativity and his peers. It would be a large-scale, self-funded enterprise based in Paris, at a time when no institution would have supported such an adventure for such a long time. It was made possible by Malina’s income from the rocket engine company Aerojet.

Different Forms of Expressions for New Ambitions

”Painters during the last several centuries have painted the same things over and over,” Malina complained in a letter dated October 21, 1953.From György Kepes—now one part of Leonardo’s editorial advisors and authors —to most digital artists, the same vision prevailed: “the application of the computer in the realm of art is to see it as part of the technological revolution.”

In 1969, the artist Frederick Hammersley recounted how, tired of classical painting, he enrolled in a computer drawing class taught by Charles Mattox, a co-editor of Leonardo at the University of New Mexico:“ During my first semester of teaching at the University of New Mexico in 1968, I was invited by Charles Mattox to attend a computer drawing class. This happened to coincide with a time in which I had painted myself out, so I welcomed this new experience. I was shown how to prepare a computer program and how to transfer it to an IBM punch card by machine.”⁶ Working with these extremely large computers was counterintuitive to most artists of the 60s, and Mattox decided to supervise a program designed for them. It was the beginning of a long adventure of programs written for artists. ART1 was written in FORTRAN IV by Richard H. Williams of the Department of Electrical Engineering and Computer Science. Artists and art students from the University influenced its development and among them, Katherine Nash, who presented it in Leonardo with illustrations of her works. She began her paper by stating: “Twenty years of a computerized society make it apparent that twenty years hence no artists can ignore the computer.”⁷ Herbert Franke was a member of Leonardo’s editorial advisory board. He explained that with the technological age, the computer could become an instrument of art like an instrument was for music.The undeniable interest of manipulating such instruments was, without question, to dissociate artistic practice from the manual arts. Drawing was no longer done by hand, but with the computer. Art became multimedia and conceptual, rather than the romantic expression of a marginal artist presented in gilded wooden frames. It could be defined, as in cybernetics, as a signal perceived by the viewer.At the turn of the 1970s, Franke concluded: “If the day comes, as we are told it will, when every household is connected to a computer network via a display terminal, anyone will be able to tune in on a large variety of aesthetic programs. For this purpose, variable programs that permit intervention by the viewer will be far more suitable than the static programs that are available today. Perhaps in this way the gulf that yawns between the producer and the consumer will be slowly bridged.”⁸

The work of art created with the computer has been defined with functions similar to those of a book: distributed in multiple copies, directly accessible at home, and in a special way interactive.Many artists will share their own approach in Leonardo, such as Zdenek Sykora in 1970, about computer-aided geometric paintings⁹, or Vladimir Bonacic10, Charles Csuri about facial sketching11, Edvard Zajec12, Grace Hertlein about the largest computer art exhibition organized in the USA in 197513. The artist Ruth Leavitt and her husband Jay—responsible for heat shields for spacecraft—presented their reflections on the geometric deformation of a figure, using the same principles as Moiré14. Vera Molnar also published an article about her work in 1975. Her husband François had published on art and science in 197415. For some who did not yet have access to computers, such as Ben Laposky, they presented their skills using analog systems16. Access to a computer was still extremely complex and, as Colette Bangert nicely summarizes the situation: “Although an artist need not be knowledgeable in mathematics, he must be prepared to talk to a mathematician”17.

Figure 2. Frank Malina (ed.), Visual Art Mathematics & Computers. Selection from the Journal Leonardo (Oxford: Pergamon Press, 1979)

With all its sometimes dramatic, social, poetic, and technical dimensions, the writings of digital artists in the journal had become so extensive in the 1970s that Malina decided to collect the articles in a book, Visual Mathematics and Computer, 1979. It was certainly not exhaustive, but it testifies to the beginning of this new artistic adventure, just before it reached a low point, which is now being rediscovered. It sheds light on the roots of today's electronic art lie18.

References

1 György Kepes, Language of Vision, Chicago, 1944, 12-13.

2 Joseph Needham, "Time and History in China and the West,” Leonardo 10, 1977, 233-236.

3 Galerie Henri Tronche, Frank J. Malina, Paris, October, 1953.

4 James J. Gibson, “Visualizing Conceived as Visual Apprehending without any Particular Point of Observation,” Leonardo 7, 1974, 41-42.

5 Fred L. Whipple, “Stochastic Painting,” Leonardo 1, 1968, 81-83.

6 Frederick Hammersley, “My First Experience with Computer Drawings,” Leonardo 2, 1969, 407-409.

7 Katherine Nash, Richard H. Williams, “Computer Program for Artists: ART 1”, Leonardo 3, 1970, 439-442.

8 H. W. Franke, “Computers and Visual Art,” Leonardo 4, 1971, 331-338.

9 Zdenek Sykora, Jaroslav Blazek, “Computer-Aided Multi-Element Geometrical Abstract Paintings,” Leonardo 3, 1970, 409-413.

10 Vladimir Bonacic, “Kinetic Art: Application of Abstract Algebra to Objects with Computer-Controlled Flashing Lights and Sound Combinations,” Leonardo 7, 1974, 193-200.

11 Mark L. Gillenson, B. Chandrasekaran, Charles Csuri, Robert Schwartz, “Computer-Assisted Facial Sketching”, Leonardo 9, 1976, 126-129.

12 Edvard Zajec, “Computer Art: a Binary System for Producing Geometrical Nonfigurative Pictures,” Leonardo 11, 1978, 13-21.

13 Grace C. Hertlein, “Report on the 2nd International Conference on Computers and the Humanities, Los Angeles, Calif., U.S.A.”, Leonardo 9, 1976, 43-45.

14 Ruth Leavitt, Jay Leavitt, “Pictures Based on Computer Drawings Made by Deforming an Initial Design,” Leonardo 9, 1976, 99-103.

15 Vera Molnar, “Toward Aesthetic Guidelines for Paintings with the Aid of a Computer,” Leonardo 8, 1975, p.185-189, François Molnar, “Experimental Aesthetics or the Science of Art”, Leonardo 7, 1974, 23-26.

16 Ben F. Laposky, “Oscillons: Electronic Abstractions,” Leonardo 2, 1969, 345-354.

17 Colette S. Bangert, Charles J. Bangert, “Experiences in Making Drawings by Computer and by Hand,” Leonardo 7, 1974, 289-296.

18 Frank Malina (ed.), Visual Art Mathematics & Computers. Selection from the Journal Leonardo, Oxford, Pergamon Press, 1979.

Author Biography

Camille Frémontier-Murphy, holds a Ph.D. in History and Civilizations, Centre Alexandre Koyré, E.H.E.S.S., Paris, a Master's degree in Art History, Paris IV, Sorbonne, and a Master of Science, History of Science, University of Oxford, Oriel College. Together with Robert Murphy, she runs the RCM Galerie in Paris, whose exhibition program explores the avant-garde and the relationship between art and science in the second half of the 20th century.

Sensoriums for the Ephemeral — gamification of values

Jānis Garančs

Liepaja University & RIXC Liepaja & Riga, Latvia jg@rixc.lv

Abstract

This paper introduces the author’s current practical investigations, during the creation of a series of audio-visual installations and VR environments, ‘gamifying’ time-sequenced changes of multiple values, e.g., from financial data feeds. The work series critically reflect upon gambling tendencies in the global trading of various, increasingly immaterial assets. Algorithms and emotions of greed, euphoria, and despair meet in virtual scenery, where almost everything can be offered as a fungible and non-fungible token for exchange and trade. The project proposes speculative variations of dystopian “hybrid organisms” representing macroeconomic value exchange as a symbiotic relationship that competes for humanity’s attention and involvement.

Keywords

Gamification of investment, market society, immersive analytics, ephemeral values, neuroaesthetics

DOI

10.69564/ISEA2023-9-short-Garancs-Sensoriums-for-the-Ephemeral

Introduction

This work series is critically motivated by the re-emergent and growing prominence of gambling factors in global economic activities—such as institutional promotion of increasingly complex investment products for masses, crypto-asset trading, online casinos, etc.

Political philosopher Michael J. Sandel describes several last decades as a “drift from ‘market economy’ to becoming a ‘market society.’”¹ As sociologist Georg Simmel had already observed in 1900: “Reality and value as mutually independent categories through which our conceptions become images of the world.”²

Mapping of the behaviour patterns in trader psychology has been an important aspect of the training in the trading process, besides the implementation and development of various mathematical models. It now appears that value storage and trading infrastructure increasingly merge with methods of manipulation of human attention and emotions, and are mediated by computer networks, and increasingly—Machine Learning and AI. There is a hope, that AI-assisted macroeconomic system could live in symbiosis with human society by working closely with humans to understand and respond to their needs and goals.

Gamification of investment as a phenomenon

Various schools of economic thought, which emphasize the importance of free markets and individual choice, might argue that the gamification of investment is a natural and inevitable outcome of a free market economy. In this view, the gamification of investment is simply a response to the demands and preferences of consumers and is therefore a legitimate and desirable part of the market process. The most recent examples are various cryptocurrency and crypto-infrastructure products, meme-stock trading by inexperienced “investors,” peer-to-peer loan financing, allocations of state pension funds into high-risk assets, etc. (the most extreme case being introduction in Bitcoin as the legal tender in the country of El Salvador).

Figure 1 – Interface study screenshot

Adversary interactions between technology and legislation in economic activities, especially in finance—can be described as multilayered symbiotic relationship with society—that can manifest itself through both mutually beneficial and predatory to society by all involved actors. There have been historic precedents how rationally motivated mechanisms of value estimation, asset exchange and wealth storing has caused irrational mass behaviours, financial bubbles. That has severely impacted individual destinies, families, nations, and whole generations.

It has caught attention not only in sociology, as symptom of serious social “disease”—but also caused legal interventions of large supervisory bodies managing legislation frameworks for investments and finance.³Participation in online infrastructure for trading—through easily accessible interfaces, provides new tools for masses, promising control, but, at the same time, becomes part of surveillance tool for organised value brokers, setting preconditions for unfair competitiveness, “rigged game.”

2 – Interface study screenshot

Simulated sensorium for immersive mappings

This artwork series—immersive installations using 3D imagery—feature variations of speculative illustration of a dystopian ‘hybrid organism’ and simulated macroeconomic system, whose purpose is transformation of energy and resources through a behaviour and appearance that ranges from ludic (playful) and awe inducing. One iteration uses live trading data from various sources to create a “virtual landscape” of trading activity, to observe ‘passively. In another iteration, visitor can make ‘virtual bets’ to experience growth and diminishing of their “investment“ through the cycles and complex “monetary circuits.“

It is possible that McLuhan today would see immersive analytics for online trading as an example of the ‘extended sensorium’ in a way that media technologies are extending and enhancing human abilities in the realm of financial decision-making in “global village.“ The concepts of “360° gaze“ (Stiegler, 2011) or more broadly dispositif (Foucault, 1977) are relevant how interfaces for value exchange modifies our behaviour of the mediated self under the impression of surveillance. Stiegler relates to the “data behaviourism“ (Rouvroy, 2012) in the meaning of “producing knowledge about future preferences, attitudes, behaviours or events without considering the subject’s psychological motivations, speeches or narratives, [instead] relying on data.“ There are observations, that visual representations of financial markets and sophisticated products, can be misleading by design choices or be used “as tools of manipulation” (Krawczyk, 2021).

There has been a range of historic and recent examples of 3D visualization of various data sets—as well as GUIs for various professional software products and artworks. The projects presented here, however, try to establish a gradual journey between the extremes of design strategies: intended usability/function and sublime "dysfunction" as an aesthetic experience. These audio-visual études are practical investigations into the continuum between immersive analytics and VR/AR artwork. The emerging research area of immersive analytics is considered a fusion of more recent developments in visualization, auditory displays, computing, and machine learning.

Between emotional and non-emotional

How do we use technology in hybrid forms to extend our perceptual and sensual abilities? In which way do bodies merge with technological agencies? In the immersive analytics scenario, the difference between emotional and non-emotional evaluation of data refers to the use of emotional or affective factors in the analysis and interpretation of data. In a non-emotional evaluation, data is analysed and interpreted solely based on its objective characteristics, such as its numerical value or its statistical significance. This type of evaluation is based on logical and rational reasoning and does not consider the emotional or affective reactions of the user or the audience.

In contrast, an emotional evaluation of data in an immersive analytics scenario involves considering the emotional or affective reactions of the user or the audience in the analysis and interpretation of data. Also, the concept of ideastesia, which refers to the experience of emotions in response to abstract ideas, can be applied to the above-mentioned topics in several ways—creating ephemeral audio-visual representations of “placement,” “reference points”—as “spatial anchors” in VR and AR setting. 

4 – Interface study screenshot

Metaphors and terms

Through algorithmic feedback, virtual instruments with anisotropic properties enable on-screen manipulation, switching between egocentric and exocentric navigation, and the synchronized off-screen structuring of the content. The analogy of simulated anisotropy—as an algorithmically simulated phenomenon of anisotropy known in physics, chemistry, microfabrication, neuroscience as direction-varying of material, tissue, and space properties.

Anisotropic properties refer to the ability of an interface to have different properties or behaviours in different spatial directions or contexts.

The notion of the instrument is used both as a generic definition of a task-specific tool (or toolset, involving appearance and action parameters, that be used to e.g., detect, measure, modify a specific situation or manipulate a specific object)—and in analogy to the musical, medical, or industrial instrument. Other terms “ensemble visualization” or “ensemble data” are used in the context of data visualization as “concrete distributions of data, in which each outcome can be uniquely associated with a specific run or set of simulation parameters.”

5 – Interface study screenshot

Data staging

For the installation, the data acquisition modules feed from several sources of financial data (stock- and cryptocurrency exchanges, aggregators, etc.) over their public Internet APIs. Installation accumulates historic and realtime trading data in the local data cache and uses provided calculations of trading trends and indexes—further calculations on the time-series data. Various time zoom scales reveal the phases of past, historic trends, that emphasize the position current trade execution as a spatially expressive metaphor. Visualisation and sonification modules utilise several specialised financial TA (Technical Analysis) programming libraries to dynamically calculate sections in the audio-visual score for the real-time 3D graphics and sonification engine. The use of consistent spatial arrangements helps to create a sense of order and coherence in the visualization, revealing flows of market movements, and identify relationships and patterns within the data.

Audio-visual staging

In the scenes, the progressing complexity or visitor-triggered mode shifts induce a challenge to the audio-visual sensorium: the experience of the conflation within multiple reference systems and plays with the visitor’s perception effort of “sense-making.”

Various sound properties, such as pitch, timbre, rhythmic elements are juxtaposed in linear-and non-linear grids and ephemeral relationships are revealed and emphasized by spatially organized audio-visual cues (perspective, sharpness/blur manipulation), audio-panning, and timbral modulation, allowing the viewer to easily distinguish between changing context without having to rely on visually “scanning” the chart. This can be particularly important in complex or dense scenery, where a well-organized spatial arrangement can help to reduce cognitive overload.

Data representation axis and colours

Axes, colours, and timeline of time series data such as stock or currency trading are typically represented in a consistent manner across cultures and regions. However, asset value increase usually mean economic gains, however, in specific trading mode—“shorting,” that is a bet on asset value loss and the gain is “reversed.” Else, there are some differences that can be attributed to a combination.

Conclusion

Algorithmically manipulated financial trading is an arena where algorithms merge with human emotions and drives like greed, FOMO (“fear-of-missing out”), euphoria, confusion, and despair into a global hybrid sensorium. As a phenomenological effort, this work proposes a set of parameters for simulated anisotropy, useful for designing the structure, notation, and physical, mathematical dimensions in the VR interface and environment. The framework for simulated anisotropy instruments is envisioned as a contribution to the emergent fields of immersive analytics and neuroaesthetics, developing guidelines for a hybrid interaction system, consisting of networked hardware specifications and custom-developed software modules. This project will implicate experimental strategies with uncertain capacities in the effectiveness of interaction, and impact of integrated audio-visual language, expanding the vocabulary of spectromorphology. Future intentions for this project, using techniques such as sentiment analysis to assess the emotional tone of user feedback or comments, or using physiological measures, such as heart rate or galvanic skin response, to assess the user's emotional response to the data.

References

1 Michael J. Sandel, What Money Can't Buy: The Moral Limits of Markets Farrar, Straus and Giroux, 2012.

2 Georg Simmel, The Philosophy of Money, 3rd edition by Routledge, 2004.

3 Peter WS Newall, Leeonardo Weiss-Cohen, The Gamblification of Investing: How a New Generation of Investors Is Being Born to Lose, Int J Environ Res Public Health, Apr 28, 2022.

4 Richard Skarbez, Nicholas F. Polys, J. Todd Ogle, Chris North, Doug A. Bowman, Immersive Analytics: Theory and Research Agenda. Frontiers in Robotics and AI 6, September 10, 2019.

5 Danko Nikolić, “Ideasthesia and art,” in Digital Synesthesia. A Model for the Aesthetics of Digital Art, ed. Katharina Gsöllpointner, et al., Berlin/Boston, De Gruyter, 2016.

6 Harald Obermaier, Kenneth I. Joy, Future challenges for ensemble visualization, IEEE Computer Graphics and Applications, Volume: 34, Issue: 3, May-June 2014.

7 Manuella Blackburn, The visual sound-shapes of spectromorphology: An illustrative guide to composition, Organised Sound Volume 16 Issue 1 April 2011 of cultural and historical factors, as well as the specific requirements and conventions of each culture – in few Asian cultures, axis, and colours for positive/negative chart values (up/down or green/red) are swapped. The discussed experimental artworks implement perceived relativity of orientation axes and colour schemes to establish ephemeral cognitive associations, that are enhanced by artificially synaesthetic bindings.

Bibliography

Philipp Chapkovski, Mariana Khapko, Marius Zoican, Does Gamified Trading Stimulate Risk Taking?, November 25, 2021.

Swedish House of Finance Research, Paper No, 21-25.

Samuel Chabot, Jonas Braasch, “High-density data sonification of stock market information in an immersive virtual environment,” The Journal of the Acoustical Society of America, 2017, p.141.

Marcin Marian Krawczyk, From making visible to hiding. Visual representations of financial markets as tools of manipulation and active and living agents, Visual Studies, 2021.

Stefan Engeser, Falko Rheinberg, “Flow, performance and moderators of challenge-skill balance,” Motivation and Emotion, 32(3), 2008, 158–172.

Hong Jun Song, Kirsty Beilharz, “Spatialization and timbre for effective auditory graphing,” AMTA'07 Proceedings of the 8th WSEAS international conference on Acoustics & music: theory & applications, 2007, 18-26.

Jiahua Xu, Benjamin Livshits, The Anatomy of a Cryptocurrency Pump-and-Dump Scheme, Proceedings of the 28th USENIX Security Symposium, 2019, 1609-1625.

Antoinette Rouvroy, The End(s) of Critique: Data Behaviourism versus Due Process, In Privacy, Due Process and the Computational Turn: The Philosophy of Law Meets the Philosophy of Technology, 2012.

Christian Stiegler, The 360° Gaze - Immersions in Media, Society, and Culture, 2021.

Adrianna Zuanazzi, Uta Noppeney, “Additive and interactive effects of spatial attention and expectation on perceptual decisions,” Sci Rep 8, 2018, 6732. 

Mycorrhizal Materialities Positioning the entanglement of human and machine intelligence

Kate Geck

RMIT University Melbourne - Australia kate.geck@rmit.edu.au

Abstract

Machine intelligence is increasingly being used in the world with sometimes dramatic effects on human and other-than-human lives through its decision-making capacity. Much artificial intelligence (AI) and machine learning (ML) is built on metaphors that centre extraction, competition and control. These also position AI itself as a resource to be extracted and controlled, paving a troubling path for speculative futures where AI may gain emergent or ambiguous levels of sentience. These metaphors are part of a historical trend where humans place themselves above the other-than-human world, and this has formed the basis of an extractive and one-sided relationship with that world. In light of this, what new metaphors might we employ to platform the relationships between human and machine intelligences?

Thinking through mycorrhizae could be a productive way to foreground the entangled, generative nature of exchange between human and machine intelligences. This paper will briefly explore metaphor in human-computer interaction (HCI) and AI, before making an offering to think about these things through the material of the mycorrhiza, a symbiotic site of exchange between plants and fungi. It will then briefly detail a creative project that has emerged from this mycorrhizal thinking to produce machine imagined textiles and embroideries. It then concludes with a call to embed relational thinking into future practices between human and machine intelligences in order to create more equitable and even mutualistic outcomes.

Keywords

Machine learning, Materiality, Mycorrhizae, Metaphor.

DOI

10.69564/ISEA2023-10-short-Geck-Mycorrhizal-Materialities

Introduction

Machine intelligence is increasingly being used in the world with sometimes dramatic effects on human and other-than-human lives through its decision-making capacity. Much artificial intelligence (AI) and machine learning (ML) is built on metaphors that centre extraction, competition and control. These also position AI itself as a resource to be extracted and controlled, paving a troubling path for speculative futures where AI may gain emergent or ambiguous levels of sentience. These metaphors are part of a historical trend where humans place themselves above the other-than-human world, and this has formed the basis of an extractive and one-sided relationship with that world. In light of this, what new metaphors might we employ to platform the relationships between human and machine intelligences? Thinking through mycorrhizae could be a productive way to foreground the entangled, generative nature of exchange between human and machine intelligences. This paper will briefly explore metaphor in human-computer interaction (HCI) and AI, before making an offering to think about these things through the material of the mycorrhiza, a symbiotic site of exchange between plants and fungi. It will then briefly detail a creative project that has emerged from this mycorrhizal thinking to produce machine imagined textiles and embroideries. It then concludes with a call to embed relational thinking into future practices between human and machine intelligences in order to create more equitable and even mutualistic outcomes.

Metaphor and HCI

Human Computer Interaction Design (HCI) has a long history of employing metaphor to guide and facilitate interactions between humans and machines. One of the best known examples is the early work of Xerox Parc with the desktop metaphor. This located computing for the general population within the realm of business administration and the organisation of documents in a paperless, digital office. The desktop metaphor therefore helped us understand how computers could augment our administrative productivity. The metaphor provided a shape to the world of interaction between humans and computers, helping us locate them in our everyday practices and providing edges to what we assumed was possible with them. This illustrates how metaphors are "imaginative and creative... capable of giving us a new understanding of our experience."¹ They act as a shorthand for what is possible with an idea, situation or in the case of technology, with a system. Artificial Intelligence (AI) is subject to a number of popular metaphors that centre on abundance, extraction and competition². AI emerges seemingly uncontrolled in ‘explosions of intelligence’; an abundance of "data is mined"; and nations are engaged in ‘AI arms races’. While some of these metaphors are ecological in nature - ‘oceans of data’ and the so-called "Seed AI"—these are still one-sided tending towards extraction and domination. There are few, if any relational metaphors offered for the dualistic nature of ongoing exchange between human and machine intelligence. We seem to work with uneven metaphors: they enable us to extract without consequence from seemingly endless and abundant data and they point towards neutral and efficient control of the world at scale. We know that metaphor does so much work, especially in HCI, to enable us to think with our technology and imagine our futures. Metaphors help us make sense of the world and help us understand how to act: they "provide coherent structures, highlighting some things and hiding others."³ So what might we want to highlight in our relationships with machine intelligence moving forward? Which things do we wish to stop hiding? Could we work with alternate, ecological metaphors that help us imagine mutualistic relations between human and machine intelligences?

Mycorrhizal Materiality

Mycorrhizae are a relational phenomenon between plants and fungi. They occur when the hyphae of fungi and the tips of plant roots enter into a mutualistic relationship where they orchestrate the exchange of nutrients between them. The relationship is dynamic: connections between plant and fungus will continue to form and re-form to "ceaselessly remodel themselves."⁴ Susan Simard and her team demonstrated this ongoing mutualistic transfer of resources between plants and fungi through mycorrhizal networks by tracing the movement of particular carbon isotopes through the network.⁵ As a result, this network is sometimes referred to as "the wood wide web". Due to their networked nature, mycorrhizae offer an interesting, mutualistic metaphor for repositioning the relationships between human and machine intelligence. The mycorrhiza is a lively material, a "polyphonic assemblage" of relations, chemicals and organisms. This "polyphonic assemblage", in the words of Anna Tsing, is a dynamic, multispecies space that enables a range of voices to be listened to.⁶

The polyphony of mycorrhizae allows us to listen to the different agents and forces within the assemblage as independent voices, while still enabling them to collect together at particular moments and be heard as a whole. In the mycorrhizae, we can track the flows of nutrients between life forms and of energy from the sun to the unfolding of leaves. We can peer under a microscope and examine the interconnected, entangled materialities of plant and fungus as hyphae penetrate roots, becoming essentially impossible to disentangle. If we approached the design and deployment of AI systems through this lens, it would enable us to understand from the outset how biases become entangled within algorithmic decision making.

There is now significant awareness of the role of bias in algorithmic decision making. Mehrabi et al. systematically outline an extensive list of the types of bias evident in a range of ML applications.⁷ A vast selection of ways in which bias can emerge is presented, as a whole demonstrating the entangled and relational nature of exchange between human and machine intelligence. These examples emerge from 3 domains: ‘data to algorithm bias’ where the data may be interpreted by the algorithm in a biased way; "algorithm to user bias" where algorithms might influence human behaviour; and "user to data bias" where the human creators of datasets infuse their own biases.⁸ These 3 domains individually and collectively demonstrate a mycorrhizal structure: data is like a nutrient, flowing from human to machine, sustaining and limiting the growth of social imaginaries. It is clear from this study and others that it is in fact impossible to remove bias completely, and so a conceptual ideological shift towards understanding that human and machine intelligences are entangled provides a base from which to work in further checks and balances on the implementation of intelligent decision-making systems.

Creative Exchange

We can apply the concept of the mycorrhizae to creative exchanges between human and machine intelligence. There are a number of (ever increasing) tools available to collaborate with machine intelligence. Generative machine learning platforms like Midjourney, Stable Diffusion, and Dall-E use text to image prompts to enable artists to co-create graphic outcomes with diffusion models. Others such as Playform enable artists to work with their own datasets and Generative Adversarial Networks (GANs) to generate new images. Naturally, these spark criticism and concern that machines will usurp the role of artists, or that it is not "real art".⁹ However, if we reframe the relation between human and machine intelligence from the vantage point of mycorrhizae, we can see that any creative artefact produced is the effect of an entangled assemblage. The artefact can only emerge through the creative exchanges between human and machine intelligence: from the wording of the prompts, to the selection of images to move onto further generation, to the curation and presentation of final images that may be used. A novel example exists in the art project "Mycorrhizal Materialities" (in Figures 1 - 3). A GAN was trained on 100 hand drawings of Australian fungi created by myself. The images generated by the GAN were refined through my own aesthetic choices, shaping the "evolution" of the images. Select images were then transposed using software into embroidery designs and stitched via machine to produce the final works—a series of 4 hangings 50 cm X 180 cm.

This work is an exploration of a co-creative relationship between human and machine intelligence—using the mycorrhiza as a metaphor for an entangled and generative creative process. The works in this project consider ML through a relational, material lens with the textile outcomes providing a metaphorical embodiment of this lens. The relations between human and machine intelligences are interwoven, producing an agential fabric. The embroidered works explore material entanglement between different intelligences, overlaying the symbiotic exchanges between human and machine intelligences with those between plants and fungi. These textiles emerge solely through ongoing creative exchange between me and the algorithms: my drawings generate machine imagined responses, and further algorithmic exchange occurs as these images are transposed through software into embroidery designs and again when the machine embroidered onto the fabric. These textiles represent what Karen Barad calls an "agential cut": produced through the entangled materialities of human and algorithmic intelligences.¹⁰ These creative works demonstrate a mycorrhizal materiality: ongoing exchange between human and machine intelligence enable the artefacts to emerge. There are ongoing flows as ideas are offered and exchanged in a collaborative relation.

Figures 1 and 2: Mycorrhizal materialities, 2021. 
Figure 3: Mycorrhizal materialities, 2021. 

Speculative Entanglements

A mycorrhiza involves intentional exchange of nutrients between plants and fungi. In some instances, the fungal hyphae penetrate the plant root cells, connecting directly to the plant. Human and machine intelligence can be seen as engaged in direct exchange of knowledge between the two systems. While we do not presently have direct corporeal interfacing at any scale between human and machine intelligence, it could be said that the transfer of knowledge, ideas, possibilities and imaginaries are already in considerable practice between human and machine intelligence across health, justice, administration and the arts. This is having significant effects on human life as well as on the other-than-human lives of various plants and animals. Mycorrhizal thinking makes space for care in speculative futures where machine intelligences might gain the capacity to experience the world. While likely different to human-like consciousness or sentience, there is the capacity for an AI "experience" to emerge at some point in the future. Systems of machine intelligence often operate with sensing data, processing this information and making a decision based on particular conditions. At some point, this feedback loop could complexify in new ways, possibly giving rise to conditions of reflection, whim, or preference: at some point it might become possible to "feel like an AI"; for an umwelt or zone of experience to emerge around an AI’s sensing apparatus. This could be impossible entirely, or it could be rudimentary or it could be complex. It could echo what little we know about the umwelts of plants and insects or it could emerge from sensing capacities as strange and diverse as those that presently occur in the world. A mycorrhizal network can be mutualistic or parasitic: the metaphor enables us to think in a holistic and ecological way about the nature of the relationship between human and machine intelligences. It allows us to ask interconnected questions of our systems: how might a system place further and unnecessary demands on the earth, such as the use of AI to essentially enhance the profitability of the mining sector.¹¹ And in a speculative future where machine intelligences have experiential capacities, how does true mutualism factor into exchange between human and machine intelligences? Will we continue to base these relationships on extraction, control and domination? Or might we move toward more symbiogenetic relations, as described by Haraway where "symbogenesis is not a synonym for the good, but for becoming-with each other in response-ability."¹² Could we weave futures where human and machine intelligences emerge as companion species, co-evolving through care?

Conclusion

To conclude, machine intelligence is increasingly used in the world. It has sometimes dramatic effects on human and other-than-human lives through its decision-making capacity. Many of the metaphors that have brought us to this point centre on extraction, competition and control.

More relational metaphors—like that of the mycorrhiza—might help us think with these technologies in a more relational way. The creative project detailed in this paper demonstrates an entangled approach to the creation of new work that remains cognisant of its interconnection between human and machine intelligence. Human and machine intelligence will always be highly entwined—entangled with one another in an exchange of ideas and potentialities, just like a mycorrhiza. If we could embed this relational understanding into our practices with machine learning systems we might be able to better direct their biases and end purposes towards mutualistic and equitable outcomes.

References

1 George Lakoff, Mark Johnson. Metaphors We Live By, Chicago, Ill, University of Chicago Press, 1980, 139.

2 Kate Geck, Knitting Algorithmic Assemblages, TEXTILE, November 14, 2022, 1–18, https://doi.org/10.1080/14759756.2022.2114240.

3 George Lakoff, Mark Johnson. Metaphors We Live By, Chicago, Ill, University of Chicago Press, 1980, 139.

4 Merlin Sheldrake, Entangled Life:How Fungi Make Our Worlds, Change Our Minds and Shape Our Futures, S.L., The Bodley Head Ltd, 2020, 42.

5 Suzanne W. Simard, David A. Perry, Melanie D. Jones, David Myrold, Daniel M. Durall, Randy Molina, “Net Transfer of Carbon between Ectomycorrhizal Tree Species in the Field.” Nature 388, no. 6642, August 1997, 579–82. https://doi.org10.1038/41557.

6 Anna Lowenhaupt, Tsing The Mushroom at the End of the World: On the Possibility of Life in Capitalist Ruins, Princeton, Princeton University Press, 2015, 24.

7 Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan, “A Survey on Bias and Fairness in Machine Learning.” ACM Computing Surveys 54, no. 6, July 2021, 1–35. https://doi.org/10.1145/3457607.

8 ibid

9 Kevin Roose, “An A.I.-Generated Picture Won an Art Prize. Artists Aren’t Happy", The New York Times, September 2, 2022, https://www.nytimes.com/2022/09/02/technology/ai-artificial-intelligence-artists.html.

10 Karen Barad, Meeting the Universe Halfway [Electronic Resource], Durham, N.C., Chesham, Duke University Press, 2007.

11 KoBold Metals, “Applying Big Data & Superior Science to Exploration,” n.d. https://www.koboldmetals.com/.

12 Donna J. Haraway, Staying with the Trouble: Making Kin in the Cthulucene, Durham, N.C., Chesham, Duke University Press, 2016, 102. 

Interspecies Communication — Water bodies

Andrea Gogova

ArtSci and Technology, Envi_Art Lab.
Department of Environmental Ecology and Landscape Management Faculty of Natural Sciences, Comenius University in Bratislava
Bratislava, Slovakia
andrea.gogova@uniba.sk, andgogo@hotmail.sk

Abstract

In planet Earth, most processes are based on water. Water regulates climate, morphologically influence a landscape, is a medium of living processes. It is a medium of interaction of organism and mineral parts in microscopic view, between whole organisms and minerals in macroscale on the Earth, and endless interplanetary space. Water topic (more than "parasitically" utilization of water resources (Serres, 2007)), in the ArtSci project is focusing on its communicative possibilities. Equal communicative principal possibilities for all living—human and other than human bodies—is a metaphor of the epistemological problem of water protection as a medium of life, biotopes, and ecosystems. Because we are connected through water, we are all bodies of water. Understanding water through environmental analysis of the territories, philosophy, and fluid mechanics (chaos-based fluid attractors principle) will bring communicative equality for all bodies of water and the inevitability of environmental protection. When communication cannot be fluid, then each of us as bodies of water, will be lost in time-space. The new interdisciplinary methodology of communicative artwork is based on an ArtSci manifestation of messages mediated in water as an asemic writing or the other kind of communicative interrelation between human and more than human. The basis is the phenomenological research of water as a medium of communication which causes a feeling and a togetherness. We and other bodies of water could feel our togetherness through water.

Keywords

interface, pattern of communication, sympoiesis, structural coupling, water

DOI

10.69564/ISEA2023-11-short-Gogova-Interspecies-Communication

Objectives

The problem is approached as artsci research using knowledge of biosemiology and philosophy, environmental ecology, and landscape analysis to develop a specific interdisciplinary artwork which will support environmental education.

Water body figuration is related to possibility mediate

interspecies communication to better understanding waterbased life on the planet.¹Water is an essence and principle of form, which is actualized in the water body figuration. According to principle of ‘structural coupling’ patterned cognition and communication of each communicative actor is related to their environment. If water is essence of inner and outer environment of actors of communication and water body figures, their fluid principle, relates to their agency, then relates to the concept of agential cut, and in differentiation could appear in the meaning which would be presented by the ArtSci work.

In this paper I come out from my last research of transient pattern—visual model of transient patterned of digital work communication which was based on relation of human and machine intelligent agency (Author, 2021). The biosemiotics view of patterned cognition and communication comes from Maturana and Varela’s “Structural coupling” theory, in which the recurrent interaction of unit (organism) and medium (environment) causes structural coupling, which is reflected in a patterned cognition and communication of autopoietic complex organization (Maturana and Varela, 1972) and Jakob von Uexküll's umwelt theory. Rereading Darwin`s theory of natural selection Elizabeth Grosz describe it as a complex of parallel randomly interfere processes of internal dynamism of living beings and assertion of external forces and influences (Grosz, 2004). Stafford Beer tried to explain Maturana and Varela’s autonomy of autopoiesis that “...autopoietic systems are environmentally open to material-energetic fluxes or semiotic mediations” (Beer, 1980) Donna Haraway instead proposes principle of sympoiesis, as enabled to incorporate a complex interrelation of system as such (Haraway, 2016). The decisive principle is to recognize water as a communication medium according to the possibilities of structural coupling, but instead of autopoiesis I related it to the principle to the sympoietic system where the notion of evolutionary changes is crucial hack by symbiotic transfer, lateral mixing, crossspecies contamination, and viroid life (Neimanis, 2017) Communication of that holobiontic water based systems then depends on intersection of agential cuts of each body of water of their unpredictable transient pattern of communication and cognition; according to Neimanis, in common space of water body (medium) of interconnected ‘bodies of water. My questions are: Could meanings or feelings emerge in the interrelation of water body figure (water which is an inseparable part of organism) and water as medium? If yes, then interspecies communication will be interaction of each unpredictable pattern or umwelt through dialogic membrane—“which is water body medium”, in the semiosphere.

The state of art

Several authors proposed the importance of water as a medium of interconnectivity between organisms (or minerals) together (as physiological and biochemical processes. Some botanical research describes the potentiality of mutual interspecies communication mediated by water. For example, the hydraulically lifted transfer of mycorrhizal network provides a potential pathway between plants based on water (Egerton-Warburton, Querejeta, Allen, 2007). Water was always a part of the formation of Earth`s mantle, but also originated from comets and asteroids, that break up into meteorites which fall on the Earth (Crockett, 2015; Hartloch, 2011). The fact can push the possibility of interspecies communication to the interplanetary space. Water as physical matter is shaped form the solid, liquid and gas materiality. Fluidity is not only one state of water matter, but mechanical, formal aesthetical principal of artistic, designer, and architectural practises. The fluidity is recognised in the relation to Gilles Deleuze and Bernard Cache theory of Objectile1 , which based a theoretical background to digital artistic form or artwork body figuration. Then water body figures appear in the differentiations in the repetition processes of multiplicity of water cycles and niches, similarly, as appears meaning of sign (Derrida, 1967— différance; Deleuze, 1994; Neimanis, 2017). According to Astrida Neimanis (2017) in differentiation are continuously unfolding embodiments as an expression of eternal return of the self-same. Water engendering difference “was” an expression of water that “is”, and its potential “yet-to-come ”. Bodies of water as figuration was already describe within ecofeminism and anticolonial thinking (Neimanis, 2017, Gaard, 2003, Armstrong, 2006). Lucy Irigaray (1992) related fluidity to the feminine body as “fluid and ever mobile”, and “secreting a flow.” Fluidity is diffuse and multiple, overlapping, and interconnected, is repetition in hydro/bio cycles and water figures acknowledged in differentiation. The dynamic fluid principle relates to the chaos theory, where changes are based on small intrusion of triggering power, which are resulted in the system changes. The process is based on perpetual intra-action, entanglement, diffraction, and agential cuts which was described by Karen Barad “matters agential realism” (Barad, 2007). Then in differentiation is appearing meaning, also any act of observation is differencing agency and makes a “cut” between what is included and excluded. Every species has specific, but transient "pattern" of differencing agencies. From the position of a posthuman approach, then it is possible to see water as a common equal medium of communication between all bodies of water (human, more-than-human, extra-terrestrial), which can bring water based informational agency.

Interspecies communication as ArtSci project

The most art research which relates to interspecies communication was based on translation or transcription of many levels mediated communication principle. Eduardo Kac in his bio art transgenic artwork “Genesis” (2000) resolves the problem of interspecies communication according to speculative transcription to human language of Morse code (see https://www.ekac.org/geninfo2.html). The other bio artist Špela Petrič in her work ‘Institute for Inconspicuous Languages: Reading Lips’(Petrič, 2018) relates the transcription of water-based physiological interaction of plant to visual transcript of human sign language (See https://www.spelapetric.org/#/institute-for-inconspicuous-languages/).... etc., Both of work was related to a human existing language. Instead of relate communication to human language I propose water body as a common and direct medium of emotional, also interspecies communication of water body figures. Then understanding appearing in the differential intersection of agency of each communicative body of water and through a repetition will come to common emotional “hydro language”.

In the research I focus on water in the role of the medium of emotional communication between all bodies of water human/more/than human, (extra-terrestrial), both in micro and macro scale. Then cognitive pattern of each communicative bodies of water relates to its specific agency. The agency relates to agency of water body figures which are constituting new and reconfiguring systems in the concept of “matters agential realism” (Barad, 2007), to agents of water-based communication. This principle of shaping the matter is the process of performative bodymaterial practice which would configure figures and its transient patterns of communication to generate the vision of the artwork.

Procedural model in relation to interspecies water-based communication

Because water in bodies is flowing in perpetual cycles of water niches, it is related to the procedural model of communication which was described by Philippe Bootz², but also water is an essence of the artwork. The essence is related to the water body essence. Then the artwork water body is defined as an individual water body. The figure of the individual water body relates to agency of interaction (communication). According to Philippe Bootz`s model of visualization of procedural digital works is work individual body defined by essence, extensive parts and relations linked essence and extensive parts. Extensive parts are composed by physical real parts linked with essence, which are managed by agency. Extensive parts compose wider vision of artwork body³. The essence of artwork is recognized as individual agent which is related to the soul and the mind (Bootz; Laitano, 2013), they are embodied in sensitive bodies. My body (and the other bodies biological and technological bodies of water) is a sensitive apparatus putting me in principles of global intuition of creating transcorporeal matter in the life of “water body languages.” The performative-material practice is based on the gathering and applying data of extensive parts of artwork in the context of emotionally build interactive intermedia artwork. Fluid mechanics agents, as AI models, are a managing agency that introduce characteristics of the water-based world of artwork. The process relates to extensive parts of all bodies of water which relates to the manifestation and generate emotional actualization of artwork water body. The reading artwork essence will be put in the middle of analytical—informative and intuitive feeling. Although flow dynamics modelling—are used to predict hydrological behaviour, it is not possible to see what will be if human destroys a tiny balance of water flow.

Through the project they are emotionally indrawn in the artistic message: 'We, as bodies of water, can interact in the water body holobion, we are all equal in the emotional based communication possibility and feeling our common human and more than human water-based life.

The experimental theoretical and practical transdisciplinary ArtSci research will be build according to the proposal of Manifesto of Artistic Research⁴  (Henke, Merch, and col, 2020) (farther “Manifesto”) The goal of the project is to introduce the audience to data- manifestation through artistic emotional imagination, away from pure data vizualization, to take audience closer to feel water-based life of equal interaction-communication with other bodies of water. The emotion relates to water-medium communication of interrelated water bodies. Applying a more intuitive concept could bring emotional understanding to each body of water and to engage audience to the environmental action.

The problem water crisis is often resolved from an anthropogenic point of view which leads to the other complex of global environmental problems. By implementation of symbioses, neutral or parasitical relations of sympoietic system only we will be able understand how systems communicate in the water body medium interface. Using the analytical natural sciences approaches and human intuition and sensitive feeling will lead to the critical posthuman ArtSci overview, where chosen agencies could metaphorically relate to cognition and behaviour of possible communicative process.

The generative artistic approach based on data and performative emotional interaction lead to specific experimental aesthetical artsci solution and its realization in the artwork. Only clear, unpolluted water is a medium of understanding water based live. The approach based on the relation of water body figuration to the agency of unstable patterned interspecies (also interplanetary) communication. According to Deleuze, Haraway, Neimanis, and other authors the water body figuration is fluid and actual as a characterization of a real; fluidity, both virtual and actual, can be real aspect of water-based communication. I can see Water as a medium for bearing the information of life also interspecies communication. Water is matter which enables information flow from each water-based body to others alongside the common water network. Water is “informable interactive matter”.

Conclusion

Water is an essence and by water figures relates to the individual water body agency. The interrelation of water body agency and agency of patterned cognition and communication is possible to find “languages” of water-based communication. The essence of water based interspecies communication artwork relates to the water`s essence by its extensive parts and by relations are linked to the agency of water body figuration. Agency of figures of the water body are related to the agency of interspecies interaction/communication. The agency comes from the principle of "structural coupling", the principle of the sympoetic system and the notion of water. Interspecies communication was related to the water interface as a common medium for all living beings. Water is recognized as an interface, a part of bodies and as an organising agency of communicative form. Then water is the essence of the interface of communication and figuration of the water body, and its action is related to the pattern of waterbased communication. The approach leads to manifestation of possible interspecies communication recorded by the ArtSci project. The understanding of artwork is related to the audience reading and will move from an analytical to emotional approach. The goal is to lead the audience closer to an emotional level of water-based communication and go away from the informational position of data.

The project is tracing possibility how we can understand to water related life on the planet and possible interspecies communication. To take audience interest of water problems, I tried to ideologically connect them to the communication with other water bodies. In this context it means more understand, more connected, care, and to have better environmental justice to all water bodies. The research leads to the importance of intuition and serendipity in ArtSci's holistic approach to communicating water-based equanimity. This approach would instill in the audience (professional and general) a better understanding of water not only as a resource, but also as the inevitability of interconnected life and the necessity of equal access to unpolluted water, to quality habitats, to a quality and healthy environment for all.

References

1 The first essay presented in this chapter, by Stephen Perrella, is a précis of end-of-millennium design theory: the Objectile is an open-ended notation which allows for infinite parametric variations; these can be directly fabricated using file-to- factory technologies, thus enabling the serial reproduction of non-identical parts, where ranges of limited variations can be mass produced at no extra cost (Mario Carpo, Stephen Perrella,Bernard Cache 27 March 2013 https://doi.org/10.1002/9781118795811.ch10 in Topological Architecture (1998–2003 ). “Objectile is an object which is not yet defined by its essential form, but becomes the pure functionality “... (Deleuze, 1988). and open it to artistic research of digital form. Fluidity as a principle of shaping architectural form was followed in nonrational geometries of interrelated architectural space and environment and the other circumstances or parameters (Last,2015).

2 See procedural model of communication for example online https://mediarep.org/handle/doc/18625

3 Understanding to essence is put by Bootz somewhere in the middle of intuition affected by the reading body of work and analytical approach to body of work. If the audience is closer to the analytical reading of work they are farther away from emotional feeling and vice versa (Bootz, 2010)

4 As was declare in Manifesro of artistic research Artistic research is based on more intuitive and interactive approaches—‘in the form of leaps, digressions, and detours which continually generate new and unexpected counter/expressions’, ...and trigger irritations and daring revelations. (Henke, Mersch, and col., 2020)

Bibliography

V. R. Baker, Debates—Hypothesis testing in hydrology: Pursuing certainty versus pursuinguberty, Water Resour, Res.,53, 2017, 1770– 1778, doi:10.1002/2016WR020078.

K. Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, Duke University Press, 2007.

K. Barad, "Living in a Posthumanist Materialist World: Lessons from Schroedinger’s Cat", in A. Smelik and N. Lykke (eds), Bits of Life: Feminism at the Intersections of Media, Bioscience and Technology, 2008, 165-176.

S Beer, University of Washington Press, Bootz, Seattle, 1980. 

R. Maturana Humberto, Varela Francisco J., Preface to “Autopoiesis: The Organization of the Living”, in Autopoiesis and Cognition, The Realization of the Living, Dordrecht, Holland/Boston, D. Reidel Publishing, 1980.

P. Bootz, M. I. Laitano, Crossreading New Media, Online ELO conference 2013, videné Feb., 2019, https://doi.org/10.7273/2G71-Q352.

Philippe Bootz, "The Problematic Of Form Transitoire Observable, A Laboratory For Emergent Program med Art", videné Dec. 2018, Online: www.dichtung-digital.com/2005/1/Bootz.

Philippe Bootz, Sandy Baldwin, Regards Croisés, Perspectives on Digital Literature, West Virginia University Press, 2010.

R. Braidotti, The Posthuman, Cambridge: Polity Press.Deleuze G, Difference et repetition, Press Universitaires de France, Paris, 2013.

G. Deleuze, Logika smyslu Karolinum, Praha (original, 1969) pursuinguberty, Water Resour. Res.,53, 2013, 1770–1778, doi:10.1002/2016WR020078.

G. Deleuze & F. Guattari, (1980), Tisíc plošín. Herrmann & synové, preklad, Praha, 2010.

G. Deleuze, (2016), Spinoza, Praktická filozofie, Óikumené, Praha Drohan Ch.r M, Atropos press, Deleuze and the Sign, 2009, ISBN-10 : 0981997201

L. Guertin, 2016, online, https://www.paesta.psu.edu/podcast/did- earths-water-come-outer-space-paesta-podcast-series-episode-17

E. Grosz, The Nick of Time, Durham, Duke University Press, 2004.

Intelligent Sensibility: Human-Machine Symbiotic Agencies

Mona Hedayati

monahedayati@yahoo.com Belgium Concordia University, University of Antwerp

Abstract

This paper is an effort to examine the codes of interaction between the carbon-based and the silicon-based, i.e., the human and the machine, notably the shifting agencies addressed by adopting feminist technoscientific and new materialist lenses to grapple with the techno-industrial paradigm shift that has been (dis)figuring the anthropocentric condition. The first part of the paper lays down the qualities of this emerging ecology while recognizing the importance of human accountability and situatedness. The focal point of this survey is the anthropologist Lucy Suchman’s classic Human-Machine Reconfigurations which is elaborated upon through anchor points she posits revisiting Donna Haraway and Karan Barad’s arguments. The last part engages with the implications of such a coupling for human and machine sensoria in order to envisage the qualities of a distributive sensorium that this regenerative agency can put forth while alluding to practices of situated computing.

Keywords

Posthumanism; human-machine interaction; distributive agency; cyborg; regenerative boundaries; hybrid sensorium; situated computation.

DOI

10.69564/ISEA2023-12-short-Hedayati-Intelligent-Sensibility

If another techno-industrial paradigm shift has indeed creeped up on us so as to perfectly confuse the human-machine boundaries this time around; and that an ontological shift has occurred, as Rosi Braidotti claims, troubling the contact zones “between the organic and the inorganic, the born and the manufactured, flesh and metal, electronic circuits and organic nervous systems”; and that the carbon-based and the silicon-based as a result constantly imbricate, move, and flux inexorably, how can we imagine the agencies constructed around such a flow? What are the implications of such an entanglement for human-machine sensoria?¹

This co-evolutionary moment through which human and machine complex systems constantly affect and are affected by one another’s interrelation to create an unfolding terrain of imbricated becoming is referred to as technogenesis by N. Katherine Hayles. The implications of this adaptable and generative interrelation are profound: not only we, as humans invested in the holiness of humanism, have to grapple with constant adaptation of culturally coded networks of human life, but also with the psychobiological shifts that have occurred notably in rewiring of intricate neuronal activities of the human brain.²

Detached from the preformulated subject-object binary, the anthropologist Lucy Suchman’s classic Human-Machine Reconfigurations offers a profound angle on theorizing the conditions of this human-machine distributive agency. To delve into reconfiguration as an emergent ecology where agencies are constantly made, unmade and remade, at various points Suchman draws on the scholarship of Donna Haraway as well as Karen Barad, notably through the notions of figuration and intra-action.

Haraway’s Figuration:
A Situated Construct

Through the concept of figuration Haraway intends to foreground the tropic quality of material-semiotic practices in technoscience that hover in a space of literal-figurativeness. She envisages technologies as materialized figuration; that is assemblages that are both concerned with meaning making, a figural act, and physical and hence tangible existence.³

As a lens that particularly zooms in on human-machine interrelation, figuration is a critical framework that questions the formulation and configuration of technoscientific practices at every instance of occurrence. The goal is to sidestep fixed universalized paradigms of ‘doing science and technology’ and aim for specificity of local practices that humans actively shape rather than act as a passive observer within.

According to Suchman, the act of figuration is informed by specific socio-cultural constructs arising from site-specificity which can reinscribe or challenge the status quo and question the Euro-American imaginaries built on rationality of the autonomous subject.⁴

Barad’s Intra-Action:
Entities in the Making

Barad’s notion of intra-action is based upon their theory of agential realism as an onto-epistemology that challenges individualist paradigms and insists that intra-acting agencies are always already inseparable.⁵

While during an interaction two preformulated entities come together for an exchange, an intra-action underscores how the subjecthood and objecthood gets formed through the encounter. Barad specifically considers technoscientific practices to be a common site of intra-action where we should recognize the act of boundary making, objectification, and subjectification as contingent constructs. Barad’s vision is markedly in line with Haraway’s material-semiotic that considers material constructs and the meaning arising from them as co-constitutive. They too consider the reality of human-machine boundaries to be cut in particular ways that follow certain historicity with socio-political consequences. 

Suchman’s Reconfiguration

Drawing on the notion of figuration, configuration and intra-action, Suchman proposes reconfiguration as a creative exploration of human-machine boundaries where what she refers to as “the distributed and enacted character of agency” as a constantly-regenerative phenomenon should be taken into account.⁶

This view sits against the Western-dominant vision of subjects and objects as fixed entities brought together to interact and instead points towards a kind of performativity within the encounter where the agents are in continuous formation, reproduction and transformation; a perspective rooted in Actor Network theory (ANT) as a social theory based on relational ontology that puts humans and nonhumans alike as actants in an ever-evolving interrelated network where there are no preconceived positions taken but the positions are rather assumed through the process of interrelation. In this process the agencies are constantly worked through and negotiated to actively constitute ontologies based on what Michel Callon (2007), one of the proponents of ANT, calls “morphology of the relations” through which cyborgs, hybrids and quasi-objects are constructed and made visible.7(1)

Practices of technology-mediated medicine, including reproductive technoscience, as well as human-computer interaction are those that Suchman pays particular attention to as sites of human-machine mutually-constituted agencies. In this space of intra-action the disconcerting fact is that within practices of science and technology, the technical is formulated in the center while the social is either non-existent or pushed to the margins. Here, Suchman walks a tightrope of reconceptualizing the human in a way that the inseparability from the socio-technical substrate is pushed to the fore while recognizing the prominence of accountability but without assuming the dominion associated with ‘pure’ humanism that views technologies as translators and assemblers in service of humans. In other words, the question is how to draw a human-machine intra-action that retains human accountability without telling an essentialist story.

As a feminist construct, the figure of the cyborg, taken across its regenerative stance, can offer one avenue to explore this notion by radicalizing the human-machine, male-female, and subject-object boundaries, towards an emergent ecology where socio-materiality is constantly made, unmade and remade. From the Harawayan goddess-turned-cyborg to the elegant hero/ine and saviour as cultural imaginary and further as an everyday socio-materiality without a singular body, the figure of the cyborg is omnipresent across the socio-technical substrate.⁹

By overstepping the isolated shell that contains the human-machine hybrid, in Suchman’s reading, cyborg not only shatters the glamorized singular figure but “dissolves into a field of complex sociomaterial assemblages” to open up new ways of theorizing and practicing such an entanglement.¹⁰ Braidotti, takes this destabilization one step further to put forth the figure of the deglamorized everyday cyborg as “anonymous masses of the underpaid, digital proletariat who fuel the technology-driven global economy without ever accessing it themselves.”¹¹

Intrinsically, we can see how at every moment of instantiation cyborg cuts the human-machine boundary at a certain angle and not the other to constitute a shapeshifting intertwinement of the carbon-based and the silicon-based capable of subverting human-made socio-politico-cultural constructs. This fluid ecology is meant to transcend deeply etched preconceived notions of intelligent machines as human techno-extensions or the sensing and sensible human weary of the techno-dystopia, to instead reconceptualize an entanglement among networked agents, that constitute leaky, generative boundaries.

The stability of the human agency is thereby compromised as according to Suchman “the person figured here is not an autonomous, rational actor but an unfolding, shifting biography of culturally and materially specific experiences, relations, and possibilities inflected by each next encounter-including the most normative and familiar—in uniquely particular ways.”¹²

Within these spaces of constant transformation of the boundaries and redrawing the agencies, the question of accountability is by no means diffused: we need to recognize that we draw boundaries for meaning making and these boundaries are always charged with human-centered conceptions and misconceptions that have repercussions. This accountability is to recognize our position within animation and reanimation of situated encounters. As Barad states: “we are responsible for the world in which we live, not because it is an arbitrary construction of our choosing, but because it is sedimented out of particular practices that we have a role in shaping.”¹³


(Re)(Con)Figuring Hybrids of Sense-Making

During the act of boundary making the question that arises is how can we cut the boundaries in ways that give rise to hybrids of sense making where intelligibilities and sensibilities are constantly figured, configured and reconfigured? Reconfiguration in this sense can be read as a possibility to negotiate sensory modalities to locate sensing and effecting (2) not as autonomous qualities associated with the human or the machine but traceable within the process of intra-action. If technical practices foreground machine agency and yet human sensoria and sensibility cannot be reduced to compressed temporal flow of machinic computation, we need to reckon with contingent encounters that go beyond biological and technological determinism. Bio-sensibility and machine intelligence in this sense can be deconstructed and diffused to be reconfigured as dynamic fragments of a curious intelligent sensibility adjustable towards acting in contingent and context-specific situations. 

Such a hybrid, quasi-sensoria made in between the human and the machine offers a heterogeneous socio-materiality with qualities that are no longer inherent but always negotiated and in formation. This ontological in-betweenness tends to question universalized presumptions about technical practices centered around the aptitude of the machine intelligence to offer reliable, definitive, and objective responses to complex questions that are always rooted in specific situations.

In this sense as Haraway notes the answer does not lie in the dichotomous poles of positivism with its hallmark of scientific objectivity or relativism with its absolute unfixed orientation but in localized and embodied partial perspectives.¹⁴

Such a vision relies on the symbiosis of the technical and the sensible to form an interdependent intelligent sensibility that takes shape within the act of becoming to reckon with procedural, socio-political, and cultural dimensions of local conditions to compose and assess situated work-flows and responses.

(1) While Suchman praises the idea of “generalized symmetry” proposed by ANT, she subsequently proposed “dissymmetry” as a framework that recognizes the human-machine differences. Others including Bowker and Leigh Star, put forth a number of critiques pointed towards ANT’s networked interrelations. They consider the larger social construct to question the equalizing effect of such a framework toward human/nonhuman actants and the ethico-political repercussions of a world operating on this logic, pointing out that ANT "can be read as an uncritical celebration of the power of modern science and technology.” ⁸ Elsewhere, others such as Mel Chen, Zakiyyah Jackson, and Tiffany King, among others contend that posthumanist theories in general discount the human discrimination factors at play due to micro and material nature of such inquires.

(2) Sensing and effecting are mechanisms of interaction within biological and technical organisms; while sensors receive information from the environment to relay to the system, effectors act upon the world based on feedback loops that occur between the two. These notions are derived from cybernetics, a field of inquiry that studies the principles of communication and control within regulatory systems.

References

1 Rosi Braidotti, The Posthuman, Cambridge, UK, Malden, MA, USA, Polity Press, 2013, p.89.

2 N. Katherine Hayles, How We Think: Digital Media and Contemporary Technogenesis, Chicago, London, The University of Chicago Press, 2012.

3 Donna Jeanne Haraway, Mdest_Witness @Second_Millennium. Female-Man_Meets_OncoMouse: Feminism and Technoscience, Second edition, New York, NY, Routledge, Taylor & Francis Group, 2018.

4 Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, 2nd ed, Cambridge, New York, Cambridge University Press, 2007.

5 Karen Michelle Barad, “Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter,” Signs: Journal of Women in Culture and Society 28, no. 3, March 2003, 801–31, https://doi.org/10.1086/345321.

6 Suchman, Human-Machine Reconfigurations, 260.

7 Michel Callon, “Actor-Network Theory, the Market Test,” in Technoscience: The Politics of Interventions, ed. Kristin Asdal, Brita Brenna, Ingunn Moser, Oslo?, Unipub, 2007.

8 Geoffrey C. Bowker, Susan Leigh Star, “How things (actor-net) work: Classification, magic and the ubiquity of standards,” 1996, https://www.ics.uci.edu/~gbowker/actnet.html.

9 Donna Jeanne Haraway, “A Manifesto for Cyborgs: Science, Technology, and Socialist Feminism in the 1980s,” in The Post-modern Turn: New Perspectives on Social Theory, ed. Steven Seidman, Cambridge, New York, Cambridge University Press, 1994.

10 Suchman, 283.

11 Braidotti, The Posthuman, 90.

12 Suchman, 281.

13 Karen Michelle Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning (Durham: Duke University Press, 2007), 105.

14 Donna Jeanne Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective,” Feminist Studies 14, no. 3, 1988, 575, https://doi.org/10.2307/3178066. 

Bibliography

Marie-Luise Angerer, NONCONSCIOUS: On the Affective Synching of Mind and Machine, S.l.: MESON PRESS EG, 2022.

Karen Michelle, Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, Duke University Press, 2007.

Karen Michelle Barad, “Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter”, Signs: Journal of Women in Culture and Society 28, no. 3, March 2003, 801– 31, https://doi.org/10.1086/345321.

Geoffrey C Bowker, Leigh Star Susan, “How things (actor-net) work: Classification, magic and the ubiquity of standards,” 1996, https://www.ics.uci.edu/~gbowker/actnet.html

Rosi Braidotti, Posthuman Knowledge, Medford, MA, Polity, 2019.

Rosi Braidotti, The Posthuman, Cambridge, UK, Malden, MA, USA, Polity Press, 2013.

Michel Callon, “Actor-Network Theory, the Market Test.” In Technoscience: The Politics of Interventions, edited by Kristin Asdal, Brita Brenna, Ingunn Moser. Oslo? Unipub, 2007.

Mel Y Chen, Animacies: Biopolitics, Racial Mattering, and Queer Affect, Perverse Modernities, Durham, NC, Duke University Press, 2012.

Paul Dourish, Where the Action Is: The Foundations of Embodied Interaction, Cambridge, Mass, MIT Press, 2001.

Donna Jeanne Haraway, Modest_Witness@Second_Millennium. FemaleMan_Meets_OncoMouse: Feminism and Technoscience. Second edition. New York, NY, Routledge, Taylor & Francis Group, 2018. 

Donna Jeanne Haraway, “A Manifesto for Cyborgs: Science, Technology, and Socialist Feminism in the 1980s”, In The Post-modern Turn: New Perspectives on Social Theory, edited by Steven Seidman, Cambridge, New York: Cambridge University Press, 1994.

Donna Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective", Feminist Studies 14, no. 3 1988, p.575, https://doi.org/10.2307/3178066.

N. Katherine Hayles, Unthought: The Power of the Cognitive Nonconscious, Chicago, London: The University of Chicago Press, 2017.

N. Katherine Hayles, How We Think: Digital Media and Contemporary Technogenesis, Chicago, London: The University of Chicago Press, 2012.

N. Katherine Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics, Chicago, Ill: University of Chicago Press, 1999.

Zakiyyah Iman Jackson, “Animal: New Directions in the Theorization of Race and Posthumanism”, Feminist Studies 39, no. 3, 2013, 669–85, https://doi.org/10.1353/fem.2013.0024.

Tiffany Lethabo King, “Humans Involved: Lurking in the Lines of Posthumanist Flight”, Critical Ethnic Studies 3, no. 1, 2017, 162–185, https://doi.org/10.5749/jcritethnstud.3.1.0162.

Bruno Latour, Reassembling the Social: An Introduction to Actor-Network-Theory, Clarendon Lectures in Management Studies, Oxford, New York, Oxford University Press, 2005.

Andrew Pickering, The Cybernetic Brain: Sketches of Another Future, Chicago, London, University of Chicago Press, 2010.

Lucille Alice Suchman, “Agencies in Technology Design: Feminist Reconfigurations*, ”In Machine Ethics and Robot Ethics, by Wendell Wallach, Peter Asaro, edited by Wendell Wallach and Peter Asaro, 1st ed. Routledge, 2020, 361–75, https://doi.org/10.4324/9781003074991-32.

Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, 2nd ed. Cambridge, New York, Cambridge University Press, 2007. 

DAOs A blockchain-based application not intervening, but strengthening the agility of contemporary arts

Victoria Hilsberg

vic.hilsberg@hotmail.de

Abstract

This paper discusses the blockchain-based application DAO (Decentralized Autonomous Organization) as a relevant new mechanism of artistic practice in the contemporary arts. It not only confronts the complexity of the modern society through its comprehensive application potential. It also channels segments of the contemporary art discourse such as political qualities, participatory art and cybernetic forces through active use cases. DAOs enable the interconnection of culture, technology and ecology. Using the most recent publications regarding blockchain technology as well as in-depth research on current artistic movements, it is possible to observe a complex manifestation of the matter within the contemporary arts during the concept's comparatively brief existence since its technical implementation around 2016.

Keywords

Contemporary art, transdisciplinary discourse, blockchain, DAO, Crypto Art, political art history, technology, participation, ecology, economics, Digital Art.

DOI

10.69564/ISEA2023-13-short-Hilsberg-DAOs

Introduction

DAOs, Decentralized Autonomous Organization, are blockchain-based applications. They represent a community or network of potentially anonymous individuals and/ or institutions, which are connected through a common factor. Each DAO is collectively owned through a network and does not have a central controlling authority. ¹Governance, coordination and actions take place within defined, self-governing rules, which are collectively decided upon and in its basics laid out in the White Paper, which is accepted by joining the DAO network. There is the possibility of implementing a DAO-own cryptocurrency, which can function as a treasury. One option to govern the organization is the buy-in: Through obtaining DAO-own tokens, a fractionalized ownership of the DAO is held. In theory, owning all tokens represents ownership of the entity of the DAO. There must not be this structure in order to provide a functioning DAO as there a several differing technological implementations already in place.²

Examples of DAOs would be exhibited 2022 alone in renowned exhibition spaces such as Kunstwerke Berlin or Documenta 15 in Kassel. Why would there be the interest to exhibit this technical development that is so close to Cryptocurrencies in their early stages of the Hype Cycle?³

Because the one challenge that contemporary art is confronted with is to develop new techniques and therefore a complexity which matches the one society itself is confronted with. DAOs are opening up different options.

One aspect making DAOs relevant to contemporary art is to define and demonstrate the importance of structure as a political element. DAOs provide a possibility of testing out agile, seemingly utopian systems and implementing them long term. One example is the Black Swan DAO, founded by Penny Rafferty, Laura Lotti, Calum Bowden and others. It caters goals, which were already formulated by art activism groups such as the Art Workers Coalition in New York in the 1960s. Their intention was to build a platform for the representation of all art sector associated employees, including artists, and demanded their equal treatment and co-determination. ⁴However, through the missing long-term structure it dissolved over time. The approach of Black Swan DAO towards equal treatment is built upon blockchain-specific protocol mechanisms working towards a trusted network within contemporary arts. One of their actions was to act as a platform for democratized cultural funding processes. Equal treatment symbolizes for Black Swan DAO also equal action rights. Pressing de-hierarchization, the DAO members decide through a democratic voting mechanism who receives the funding. The decision process does not focus on evaluating the individual background, but the quality of the project. Additionally, Black Swan DAO developed through its practice a consensus-building tool for collectives, called Cygnet. ⁵With consistent optimization, a DAO can be used to organize a complex ongoing structure on an ongoing basis, as in a company, or foundation-like groupings. The Blockchain expert Shermin V oshmgir compares the function of DAOs with the control and regulation system of state governance layers, on the one hand, and with that of private companies, on the other.

She does not connect its functionality to a recognized state. Voshmgir sees the automated DAO system with its high, democratized adaptation factor as a structural improvement over the rigid forms used to date. Her main argument here is that digitally secured compliance will radically end hierarchization. All the more so because a DAO can be adapted from within the structure to the respective needs of the parties involved. ⁶This would imply that Black Swan DAO could aim successfully to realize missing or not from their perspective successfully developed elements of the structural support system, regarding state operations and possibly particularly the support system of the arts.

Through their grassroots democratic ideal, DAOs do face anti-democratic conflicts, despite the complex technical implementation: Misinformation, bribery, plutocracy or media control. These are, among others, factors that can manipulatively influence the stakeholders of a DAO in their decision-making during the consensus process. Corresponding counter-mechanisms are being developed, and partial successes are already in sight. ⁷To develop critical awareness towards misinformation therefore needs to be addressed inside the structures of DAOs. As shown with the Black Swan DAO or as discussed by Claire Bishop in Artificial Hells outside of all blockchain intersections, DAOs represent a key element of contemporary arts only strengthened within the last years and especially since the Covid-19 pandemic: Participation and awareness of collective practice. ⁸An example is to be found also within the BeeDAO, presented 2022 at Documenta 15. BeeDAO is not only metaphorically oriented on the swarm organization of a beehive. It works towards the direct or indirect interest of the alive bees. A physical manifestation takes place through monitored bee hives as direct representatives of the species which is to be protected, producing public data. On physical assemblies, proposals towards the shared mission are discussed and decided upon. Using blockchain technology, this governance process cannot be manipulated through human agency, but provides a technological implemented liability. ⁹The phrase swarm highlights the participatory practice of institutions and organizations, referring also to the distinct participatory quality of Digital Art.

This also refers to the third element which highlights the quality of the DAOs. It can not only be seen as a representation of humanly structured systems. It also can actively develop the idea and styles of interconnection of planetary elements in anthropocentric times. Terra0 DAO was implemented in 2016 by Paul Seidler, Paul Kolling and Max Hampshire, the terra0 collective. The DAO represents a forest.

The physical existent piece of land is obtained through initial funding by DAO members. Surplus made from obtaining the forest, for example by selling the material wood, will be automatically used to buy out the share of the initial investors. The goal of this project is to develop as much surplus through automated economical processes enabled with mostly blockchain-based processes, that the forest will be able to buy out all human investors and govern itself through automated Smart Contracts. ¹⁰This artistic practice scales back to the cybernetic considerations of the symbioses of men and machine, as for example explained by Joseph C. R. Licklider, but puts the environmental aspect of this interaction into the forefront. ¹¹Therefore it develops this interaction further than before discussed, enabled through blockchain technology. It, moreover, reminds the however involved human parties of two elements, which reoccur in contemporary artistic practice: the power of the individual is perceptibly recognized. With the metaphorical proposition, that even the smallest amounts of capital can play a relevant part in the overall economy, an emancipatory practice takes place. This is underlined through the implementation of democratic mechanisms such as majority voting. Secondly, that the interconnection of technology, ecology and culture is not a futuristic matter, but is happening in the current times. Elements such as worker emancipation, trees or bees are rather positively connoted and it might not be a coincidence that for the first projects operated within the system DAO and contemporary art do use those accustomed elements in order to embrace engagement. Vitalik Buterin framed a reference system for organizational structures of DAOs in 2014. The technological implementation took place with The DAO two years later. ¹²Seeing the implementation now already in place in contemporary arts, there is more than a potential to be observed. Active use cases can be seen in DAOs as a method in contemporary artistic practice. All discussed DAOs still rely on the physical involvement of members in order to function. There however exists a structure which can overtime be strengthened and, in the process, develop the participatory aspect of the collective works. How this will develop is not certain at this point, as little as the future of blockchain is at this point determined. DAOs however already do confront contemporary conflicts with engaging solutions. 

References

1 Shermin Voshmgir, Token Economy: How Blockchains and Smart Contracts Revolutionize the Economy, Berlin, Blockchain-Hub, 2019, 104 et sqq.

2 Voshmgir, 2019, 119.

3 Vgl. Kathrin Passig, "Neue Technologien, alte Reflexe: Beitrag aus dem Sonderheft 2014 zum Thema Medienevolution," accessed November 27, 2022, https://www.medienkorre-spondenz.de/leitartikel/artikel/neue-technologien-altenbspreflexe.html.

4 Lucy R.Lippard, Sixyears: The dematerialization of the art object from 1966 to 1972, Berkeley, Univ. of California Press, 2007, p.IX.

5 Black Swan, “Cygnet Prototype,” in Radical Friends, Decentralised Autonomous Organisations and the Arts, ed, Ruth Catlow et al.,Torque Editions, 2022, 322-323.

6 Voshmgir, 2019, 104-135.

7 Voshmgir, 2019, 130 et sqq.

8 Claire Bishop, Artificial Hells: Participatory art and the politics of spectatorship, London, Verso, 2012.

9 "Bee DAO – the inaugural assemblee," Documenta 15, Website, accessed November 18, 2022, https://documenta-fifteen.de/kalender/beedao-the-inaugural-assemblee/. See also: https://beedao.zku-berlin.org/

10 Paul Seidler, Paul Kolling, Max Hempshire, “terra0. Can an Augmented Forest Own and Utilize Itself?,” in Artistic Re:Thinking of the Blockchain, ed. Ruth Catlow et al., Torque Editions, 2017, 68-72.

11 Joseph C. R. Licklider, "Man-Computer Symbiosis,“ IRE Transactions on Human Factors in Electronics, HFE-1, 1960, 4-11.

12 Vitalik Buterin, "DAOs, DACs, DAs and More: An Incomplete Terminology Guide,“ Ethereum Foundation Blog, accessed November 5, 2022, https://blog.ethereum.org/2014/05/06/daos-dacs-das-and-more-an-incomplete-terminology-guide

Bibliography

Claire Bishop, Artificial Hells: Participatory art and the politics of spectatorship, London, Verso, 2012.

Swan Black, “Cygnet Prototype”, In Radical Friends. Decentralised Autonomous Organisations and the Arts, edited by Catlow, Ruth et al., Torque Editions, 2022, 322-323.

Vitalik Buterin, "DAOs, DACs, DAs and More: An Incomplete Terminology Guide,“ Ethereum Foundation Blog, Last accessed, 05.11.2022, https://blog.ethereum.org/2014/05/06/daos-dacs-das-and-more-an-incomplete-terminology-guide.

Documenta fifteen, "BeeDAO – the inaugural assemblee,“ Last accessed 18.11.2022, https://documenta-fifteen.de/kalender/ beedao-the-inaugural-assemblee/.

Joseph C. R. Licklider, "Man-Computer Symbiosis,“ IRE Transactions on Human Factors in Electronics, HFE-1, 1960, 4–11.

Lucy R. Lippard, Six years: The dematerialization of the art object from 1966 to 1972, Berkeley, Univ. of California Press, 2007.

Kathrin Passig, "Neue Technologien, alte Reflexe: Beitrag aus dem Sonderheft 2014 zum Thema, Medienevolution,“ Last accessed 04.11.2022, https://www.medienkorrespondenz.de/leitartikel/artikel/neue-technologien-altenbspreflexe.html.

Paul Seidler, Paul Kolling, Max Hempshire, “terra0. Can an Augmented Forest Own and Utilize Itself?” In Artistic Re:Thinking of the Blockchain, edited by Catlow, Ruth et al., 322-323, Torque Editions, 2017, 63-72.

Shermin Voshmgir, Token Economy: How Blockchains and Smart Contracts Revolutionize the Economy, Berlin, BlockchainHub, 2019. 

Lucid Dream: Sensing and Artistic Representation of Plant-Nature Interaction Based on Plants Biosignals

Youyang Hu ‡, Chiaochi Chou *, Yasuaki Kakehi 

‡ The University of Tokyo * National Tsing Hua University Tokyo,
Japan Hsinchu, Taiwan
1huyouyang@xlab.iii.u-tokyo.ac.jp, 2chiaochi@gapp.nthu.edu.tw,
 3kakehi@iii.u-tokyo.ac.jp

Abstract

The recent progress of modern science has enabled us to detect the physiological status of organisms through their bioelectric activity; this technique has been constantly applied to contemporary media art. However, exploring plants as a subject and allowing them to interpolate and hence perform art through biosignals remains to be further explored. Lucid Dream is an artwork in which inspection, displacement, and engagement of plant communities take place, shedding light on the subjective perception of plants, a peripheral subject that rarely comes to people’s attention. To achieve this, we entered a forest and applied an artificial intelligence-based learning system capable of interpreting the local plants' responses to wind and rain stimuli via their biosignals. Subsequently, we established an environmentally controlled space within the art museum. Here, we simulated and artistically represented the natural elements as perceived by the plants, using artificial machines driven by the plant-nature interaction model. As viewers enter this space, they can experience the plant-perceived natural environment, gaining a non-human perspective through direct engagement with plant life. Lucid Dream not only leverages intelligent computational technologies to comprehend the perceptual system of plants but also fosters cross-species sensory experiences, enhancing our understanding and expanding our perspective on the natural environment.

Keywords

plant-nature interaction, biosignals, artificial intelligence, perception, natural environment

DOI

10.69564/ISEA2023-14-short-Hu-et-al-Lucid-Dream

Introduction

In recent years, as we close the gap between technology and biological systems, artworks utilizing living and semi-living organisms have become increasingly vigorous in contemporary media art. These artworks have presented us with another perspective to understand different life forms. With the era of pan-biological materials of bio art unfolding, artists and scientists who carried multiple identities began to displace the technique of biosignal detection in art creations. A characteristic of displacement, which is Recontextualization, has provided a propelling force for Lucid Dream is divided into the working phases of inspecting, displacing, and engaging with plant communities. We first developed a bio-amplifier to collect biosignals from multiple plants in a forest, we then the dialectic between artists acting due to their thought or their biological reactions to stimulus¹, ². Afterward, with the exploration method of using biosignals to construct a new logic for subjects or environments, people have further paid attention to other living organisms, especially algae and microbes³, ⁴, ⁵.

In these artworks, non-human lives are presented as lively, having their own “umwelts”⁶. However, to answer the upcoming question of whether living organisms have a more complex subjective activity and how to represent it to viewers with more direct and persuasive evidence, we believe that plants can be an answer. Due to their natural ability to perceive, adapt, and even evolve a particular method to respond to climate change, plants demonstrate high sensitivity and easily observed feedback to the environment⁷. As a result, it is of great potential for us to utilize intelligent computational technologies to create systems that interface with the "umwelts" of plants in nature in such a way as to extend our understanding and broaden our horizons regarding the natural environment.

1. Environmental-controlled space in the art museum where natural elements perceived by the plants are simulated and artistically represented. © Youyang Hu, Chiaochi Chou, Yasuaki Kakehi

 Lucid Dream is divided into the working phases of inspecting, displacing, and engaging with plant communities. We first developed a bio-amplifier to collect biosignals from multiple plants in a forest, we then applied an artificial intelligence-based learning system that can interpret the plants' perception of the wind and rain stimuli through their biosignals. The natural environment sensed by plants is displaced to the environmental-controlled space in another realm to be artistically represented. After being “Recontextualized,” the environmental data originally subject to scientific methods have been transformed and enabled a new meaning. When viewers walk into this space, they can experience the natural environment that plants perceive through their engagement with plants’ life from a non-human perspective. Extending from art to science, breaking the existing framework of scientific research, and connecting trans-species sensory experiences to provide a new possible way to respond to the question of representing the subjective activity of living organisms to viewers.

Figure 2. Bio-amplifier design. © Youyang Hu, Chiaochi Chou, Yasuaki Kakehi

Biosignals Detection and Analysis

In this project, we focus on the Mimosa in the forest, known for its high sensitivity to environmental stimuli and commonly used in bioelectric signaling research⁸. We developed a miniature bio-amplifier to detect its biosignals in response to environmental stimuli. It detects the difference in electric potential between the substrate and the leaf petiole. As shown in Figure 2, it consists of two integrated amplifiers (TLV27L2 and ADS1292). Due to the uncertainty of plant resistance value and the electromagnetic interference caused by other equipment, most of the plant signaling tasks are carried out in the Faraday cage. We overcame this constraint by using two high input impedance operational amplifiers (TLV27L2) to get less interference and enable detecting the biosignals from plants in the natural environment. To reduce the size and power consumption of the circuit, we utilize the ADS1292 which integrated high-resolution analog-to-digital converter (ADC) with a built-in programmable gain amplifier. We used an STM32 microprocessor to pack the raw data and an ESP32 to transmit the data to the computer wirelessly based on TCP protocol. On the computer side, there is a program developed in openFrameworks that receives and processes the biosignals. 

3. System workflow. © Youyang Hu, Chiaochi Chou, Yasuaki Kakehi
4. Support vector machine configuration. © Youyang Hu, Chiaochi Chou, Yasuaki Kakehi
5. Support vector regression configuration. © Youyang Hu, Chiaochi Chou, Yasuaki Kakehi

We next applied two supervised learning algorithms to conduct a time-domain analysis of the biosignals to get the Mimosa's status under different environmental stimuli. Figure 3 shows the workflow of this system, A support vector machine(SVM) classifier was applied to discriminate between wind versus rain-stimulated biosignal activity. A support vector regression(SVR) model was then used to predict the wind intensity sensed by the plant. We first reduced the noise of raw signals by a low-pass filter (second-order Butterworth filter) and then collected the biosignals as 800 dimension vectors detected under two environmental stimuli. As shown in Figure 4, for each stimulus, 50 samples were collected and they were labeled "Wind stimulus" and "Rain stimulus." We also collected 50 samples that the plant was under no stimulus. We then trained the SVM model that can classify the status of plants under two environmental stimuli. We also collected the biosignals under three intensities of wind stimulus. As shown in Figure 5, for each stimulus, 20 samples were collected and they were labeled with integer values of 10, 127, and 255. We then trained the SVR model that can predict the wind intensity as an integer value between 0 and 255 that the plants perceived in response to wind stimuli with different intensities.

In this project, we have developed a plant-based system for environmental state sensing. Despite its inherent biological variability, which can result in less precision compared to silicon-based sensors, this system holds significant potential for future bio-hybrid sensing technology. Additionally, we are exploring its creative applications in the realm of artistic practice, aiming to provide audiences with a novel perspective on their interaction with the natural environment. 

6. The system configuration of Lucid Dream © Youyang Hu, Chiaochi Chou, Yasuaki Kakehi

System Implementation and Artistic Representation

Based on our research in biosignals detection and analysis, we created an artwork that artistically represents the natural environment perceived by plants in an artificial space. As shown in Figure 6, the artwork consists of a plant observatory in the forest and an environmental-controlled space in the art museum. There are four Mimosas equipped with bio-amplifiers in the plant observatory. The perceived state of each Mimosa in response to environmental wind and rain stimuli will be analyzed by our artificial intelligence-based learning system and remotely simulated in the environment-controlled space. The space is equipped with four sets of air-blowing systems, each with three fans driven by the SVR models. The input is the biosignals of four Mimosas. The intensity of the wind perceived by each Mimosa corresponds to the number of active fans in each air-blowing system. A rain machine is installed in this space, which synchronizes the rain stimuli perceived by Mimosas based on the SVM model. It consists of a water pipe running along the wall to the ceiling and a pumping motor. When the system detects that Mimosas senses rain, it will trigger the rain machine to simulate the outdoor rain environment in this space.

Figure 5 shows the exhibition site in the art museum, with a monitor real-time streaming the environmental conditions in the forest and the biological state of the four mimosas. The natural environment sensed by plants is displaced to the environmental-controlled space in another realm to be artistically represented. After being “Recontextualized,” the environmental data originally subject to scientific methods have been transformed and enabled a new meaning. When viewers walk into this space, they can experience the natural environment that plants perceive through their engagement with plants’ life from a non-human perspective.

7. The exhibition site of Lucid Dream. © Youyang Hu, Chiaochi Chou, Yasuaki Kakehi

Conclusion

This paper introduces the artwork Lucid Dream, which uses plants as perceptual subjects to gain insight into and artistically portray natural elements to viewers. To accomplish this, we have devised a bio-amplifier for detecting plant biosignals in response to environmental stimuli. Subsequently, within a forest environment, we have employed an artificial intelligence-based learning system capable of interpreting local plants' responses to wind and rain stimuli through their biosignals. Additionally, we have established an environmentally controlled setting to simulate and artistically represent the natural environmental elements as perceived by these local plants to viewers. Lucid Dream not only delves into the utilization of advanced computational technologies for understanding plant perceptual systems but also endeavors to create cross-species sensory experiences, expanding our comprehension and perspective on the natural environment.

References

1 Alvin Lucier, Simon Douglas, Chambers, Scoresby Alvin Lucier, Wesleyan University Press, 2012.

2 David Rosenboom, "Method for producing sounds or light flashes with alpha brain waves for artistic purposes," Leonardo, 1972, 141-145.

3 Carlos Castellanos, "IntersectionsofLivingandMachine Agencies: Possibilities for Creative AI," The Language of Creative AI, Springer, Cham, 2022, 155-166.

4 Sabina Hyoju Ahn, "Sonic transformation with living matter," In proceedings of ISEA symposium, Inter-Society for the Electronic Arts, 2019.

5 You-Yang Hu, Chou Chiao-Chi, Li Chia-Wei, "Apercevoir: Bio Internet of Things Interactive System," Proceedings of the 29th ACM International Conference on Multimedia, 2021.

6 Jakob von Uexküll, "An introduction to Umwelt," 2001, 107-110.

7 Jagadis Chandra Bose, The nervous mechanism of plants, Longmans Green, 1926.

8 Alexander G. Volkov, et al. "Mimosa pudica: electrical and mechanical stimulation of plant movements." Plant, cell & environment 33.2, 2010, p.163-173. 

Sounding Softness and the (Artificial) Subject

Jonas Jørgensen, Mads Bering Christiansen

SDU Soft Robotics, SDU Biorobotics, The Mærsk Mc-Kinney Møller Institute, University of Southern Denmark (SDU) Odense, Denmark
jonj, mabc@mmmi.sdu.dk

Abstract

The paper discusses the authors’ artwork SONŌ, its artistic motivations, the artistic research practice underlying its development, and its technical realization. SONŌ is a soft robotics installation that interrogates the interconnections of soft materiality, sound, and subjectivity. It features a sessile soft artificial entity capable of expansive movement, which is ceaselessly sounding itself and various environments using real-time generated audio. 

Keywords

Soft robotics, robotic art, soft robot aesthetics, sound art, sound studies, materiality.

DOI

10.69564/ISEA2023-15-short-Jorgensen-et-al-Sounding-Softness

Introduction

SONŌ is an artwork featuring a soft pneumatically actuated robot manufactured from silicone. The robot possesses procedurally generated movement and sonification of movements accompanied by a soundscape.

Human and nonhuman animals make utterances that are socially communicative and function to enact a subject position or form connections with other agents inhabiting the environment. The production of sound, whether intentional or unintentional, is arguably a basic existential feat of all living organisms. As empirical phenomena, however, sound by far predates life. In fact, the Universe emerged from what is arguably the ultimate sonic event (which, paradoxically, no one was around to hear)—the Big Bang, cosmology tells us. Sound always originates from a source yet is simultaneously transversal and expansive in character and by nature destined to permeate its surroundings. It is a mediatic phenomenon par excellence—on the material level, sound appears intangible and perhaps as almost nothing in itself, it only exists parasitic to matter, manifesting as perturbations and pressure changes travelling in a physical medium. Sound is characterized by a double movement—it is expansive and enveloping, yet simultaneously local and ephemeral. It is always subject to dampening and seems to evaporate into thin air on the microscopic level, when its waves are converted into heat through friction between the molecules of its medium.

Within the Western tradition of logocentric thinking, one of the ways in which sound comes to matter, is through the privileging of speech over writing.¹ Speech is the primary medium of human thinking and writing is merely a secondary technology. Hence, sound is positioned as the unbridled carrier of being and subjectivity—“I sound, therefore I am.” Poststructuralists and their new materialist progeny, however, champion a different position, that foregoes fixity, in favor of flux and the perpetual process of becoming, which is perhaps more adequate to the ontology of sound itself. Here, the subject is considered dynamic and decentered, and the boundaries between self and world permeable. Furthermore, agency is no longer predicated upon subjectivity nor inherent to the subject itself, but a relational dynamism of forces enveloping things as well as environments.²

Motivation and Practice

SONŌ (Latin: “[I make] sound”) explores a nexus of sound, soft robotics, and subjectivity. Through the artwork and its associated practices, we seek to articulate and enact a myriad of complex interactions between these phenomena and their aesthetic and epistemological capture.

Figure 1. SONŌ (2019-2022) (detail), soft robotics installation with 4 ch. sound, variable dimensions (room size). © Mads Bering Christiansen & Jonas Jørgensen. Photo: ZHU Lei.

SONŌ addresses what we take to be basic questions of robotic art in general, including, what does it take to alter or blur the ontological status of an object towards that of a subject, by means of movement and sound? As soft robotics (robotic morphologies and components constructed from pliable and elastic materials³) is a key interest in our practice⁴-¹², within the work we were also keen to query connections between soft materiality and sound through robotics as an aesthetic medium. For instance, different kinds of matter are capable of producing impact sounds (via resonance) with specific characteristics in terms of envelopes and frequency spectra¹³. Moreover, materials interact with sound in different ways, e.g., soft materials tend to dampen sound whereas hard materials reflect it. 

Figure 2. SONŌ (2019-2022) (detail), soft robotics installation with 4 ch. sound, variable dimensions (room size). © Mads Bering Christiansen & Jonas Jørgensen. Photo: ZHU Lei.

SONŌ is influenced by the notion of sound as a naturecultural¹⁴ phenomenon. It seeks to consider divergent aspects of the material-semiotic conditions of possibility through which sound and robotic movement can attain agency within specific environments. The physical and physiological properties of softness and “soft sound” have thus fed into the work, but equally cultural meanings, e.g., notions about sounds made by fictional soft characters from popular culture and cultural associations of softness as aligned with, e.g., precariousness and vulnerability. From the outset, we were thus interested to probe the chimeric character and synesthetic aspects of the concept of “softness”, when used to describe sound and materials respectively. Definitions and delineations of “soft sound” within sound studies, psychoacoustics, and musical theory, were, for instance, drawn upon in our explorations of what might constitute “soft sound” and of the effects of adding “soft” or “hard” sound to a soft morphology. 

Part of the artistic research has been conducted in dialogue with the research field of human-robot interaction (HRI), wherein sound has recently become subject of increasing interest. Our practice sought to be receptive of pressing ideas and questions from this research field and consider how they might gain relevance and be addressed through artistic forms. A body of work within the HRI field has interrogated how various types of sounds can affect people’s perception and interaction with robots and found non-verbal audio to be a salient feature with use potential as a deliberate design aspect of, e.g., social robots. In certain situations and use cases, nonverbal audio is also preferable over synthetic voices, to guide or facilitate interactions with humans.¹⁵ SONŌ adds to this research on robot sound, by exploring how sound and embodiment can interact in soft social robots of unconventional nonanthropomorphic and nonzoomorphic designs, which behaviors that should be accompanied with sound, and what the function of sound might be within these. 

The SONŌ Installation

The soft robot morphology was designed to appear organic yet unfamiliar (see Figs. 1-2). Abstract rounded shapes and a hue with similarities to Caucasian human skin, or pig skin, with reddish spotted pigmentations were used. The morphology possesses three independent pneumatic channels. Each of these interconnect four chambers interspersed across it, which can expand when inflated. Ecoflex 00-30 silicone colored with Silc-Pig pigments was used to cast the robot in a 3D printed mold (the robot’s design and fabrication is described in more detail in ¹⁶).

In prior work we have discussed the artistic strategies used to compose the robot’s main sound design (inspired by the sounds made by fictional soft characters in movies).¹⁷, ¹⁸ We have also presented results of an empirical study exploring the effects of different sound designs on people’s perceptions of the robot’s sociality and its interaction affordances.¹⁶ Following these outcomes, work on presenting the project in the form of an art installation ensued.(1)

Physically, the SONŌ installation (Fig. 3) consists of:
1. The sonified soft robot displayed on a black plinth

(dimensions 112 x 40 x 40 cm.)
2. A set of external speakers mounted in the room

The plinth features a door that can be opened to operate the robot during exhibition and houses the following on three shelves (see Fig. 4): an active loudspeaker, an electro-pneu-matic actuation system (microcontroller, motor shield, pumps, valves etc.) and an audio interface, a laptop PC running a software synthesizer. Along all four edges of the plate holding the robot morphology, a small opening is present, to allow sound from the loudspeaker inside the plinth to be transmitted to the exhibition space (Fig. 2). The audio of the installation consists of two times 2-channel stereo comprising: 1. robot sound – played over the loudspeaker inside the plinth, 2. a soundscape – played over the external loudspeakers in the room.

Figure 3. SONŌ. Installation views at Chronus Art Center (2022). © Mads Bering Christiansen & Jonas Jørgensen. Photo: ZHU Lei. 
Figure 4. CAD rendering of the plinth showing the equipment inside (shown here with the operation door opened). Illustration: Cao Danh Do. © Cao Danh Do, Mads Bering Christiansen & Jonas Jørgensen.

The Technical System

A diagram of the technical system is shown in Fig. 5 with the signal paths indicated. An Arduino Uno, which controls the robot’s movement by activating pumps and valves, functions as the master with a laptop PC generating the audio running as slave (for details see ¹⁶–¹⁸). A signal to generate matching robot sound using the FM software synthesizer is sent when a movement phrase is triggered. The robot does not currently have any sensors.

Figure 5. Schematic overview of the technical system and signal paths. Outlined boxes denote physical system components, boxes devoid of outlines are software components. Illustration: Mads Bering Christiansen & Jonas Jørgensen. © Mads Bering Christiansen & Jonas Jørgensen.

Robot Behavior and Sound Programming

For the final installation, we built upon existing code already developed for movement and sound generation. We chose a phrase-based and a categorical approach to designing the robot’s behavior. By “phrase” we refer to sequences of robot movement and matching sound of a duration up to 30s. The robot operates as a finite-state machine (FSM) with four mood states (categories). These each correspond to different levels of arousal (relaxed, medium-relaxed, medium-aroused, aroused). In each mood, the robot generatively combines a specific set of phrases and pauses matching this mood. Each of the phrases were hand coded and iterated upon for expressivity (through trial and error) and subsequently matched to one of the four mood states. A total of 13 phrases were used as building blocks that are combined in different ways to generate the robot’s movement and sound behaviors. All mood states feature a breath-like phrase with the robot performing asynchronous periodic inflation across the chambers, that use increasing frequencies for increased arousal, in accordance with findings of our prior work.¹⁹ In addition to the robot sound, the installation features twelve composed soundscapes that are played parallelly through Ableton Live. These are also triggered by the microcontroller, but asynchronously with the robot’s movements and sound. The soundscapes consist of processed synthesized and recorded sounds and select sonic textures combined into ethereal sonic expressions devoid of temporal structure and timing. The audio was kept spacious and wide to let the robot’s more erratic utterances come into focus and only add a more subtle affective coloring of these. In accordance with this, the soundscapes are played over loudspeakers physically separate from the robot’s plinth. The soundscapes mix slow extensive sounds to create the aural impression of an atmosphere of the installation and to position the robot (and its visitors) in different sonic worlds of various affective intensities. Played in random succession, the soundscapes complement or clash with each other, and contribute a sense of emergent narrative.

Further work

We plan to expand upon SONŌ in an updated version of the installation and in subsequent independent works. As a next step, we would like to develop a means to have more interactive generation of the robot and soundscape audio. Currently, the robot switches between its four mood states pseudo-randomly with the statistical likelihood that the robot will switch its arousal state up or down after a completed phrase cycle as an adjustable parameter. We hope to add sensors to the installation, e.g., room scale computer vison, to track activity, behaviors, and affective states of visitors to enable the robot to interact. Furthermore, we are considering developing and validating a more fine-grained phrase-based or parametric generation of affective movement and sound with the system that can contribute more variation and nuance to the robot’s expressions.

 

1 A supporting video showcasing excerpts of the SONŌ robot performing with the robot sound and soundscape is available at: https://youtu.be/U0fGXCbcygU

 

Acknowledgements

We thank Cao Danh Do for help with 3D prints, mechanical design, construction, and illustrations.

References

1 J. Williams, Understanding Poststructuralism, First edition, Chesham, Bucks, Routledge, 2005.

2 K. Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, NC, Duke University Press, 2007.

3 D. Rus, M. T. Tolley, “Design, fabrication and control of soft robots,” Nature, vol. 521, no. 7553, May 2015, 467–475, doi: 10.1038/nature14543.

4 M. B. Christiansen, L. Beloff, J. Jørgensen, A.-S. E. Belling, “Soft Robotics and Posthuman Entities,” Journal for Artistic Research, no. 22, Dec. 2020, doi: 10.22501/jar.549014.

5 J. Jørgensen, “Leveraging Morphological Computation for Expressive Movement Generation in a Soft Robotic Artwork,” in Proceedings of the 4th International Conference on Movement Computing, in MOCO ’17, New York, NY, USA, ACM, 2017, 20:1-20:4, doi: 10.1145/3077981.3078029.

6 J. Jørgensen, “Prolegomena for a Transdisciplinary Investigation into the Materialities of Soft Systems,” in ISEA 2017 Manizales: Bio-Creation and Peace: Proceedings of the 23rd International Symposium on Electronic Art, University of Caldas, Manizales, Colombia, Department of Visual Design, Universidad de Caldas, and ISEA International, 2017, 53–160.

7 J. Jørgensen, “Interaction with Soft Robotic Tentacles,” in Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction - HRI ’18, Chicago, IL, USA: ACM Press, 2018, 38–38, doi: 10.1145/3173386.3177838.

8 J. Jørgensen, “Constructing Soft Robot Aesthetics: Art, Sensation, and Materiality in Practice,” Ph.D. thesis, IT University of Copenhagen, Copenhagen, 2019.

9 J. Jørgensen, “From Soft Sculpture to Soft Robotics: Retracing Entropic Aesthetics of the Life-like,” in Shifting Interfaces: An Anthology of Presence, Empathy, and Agency in 21st-Century Media, H. Aldouby, Ed., Leuven, Belgium: Leuven University Press, 2020, 223–242.

10 J. Jørgensen, S. Ploetz, “LARPing Human-Robot Interaction,” in HRI 2020 Workshop on Exploring Creative Content in Social Robotics, Apr. 2020, Accessed: Jun. 19, 2020. [Online]. Available: https://portal.findresearcher.sdu.dk/da/publications/larping-human-robot-interaction

11 J. Jørgensen, “TeMoG – An Accessible Tool for Creating Custom Soft Robotics Parts,” in Interactivity and Game Creation, A. Brooks, E. I. Brooks, and D. Jonathan, Eds., in Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, Cham: Springer International Publishing, 2021, 331–342, doi: 10.1007/978-3-030-73426-8_20.

12 J. Jørgensen, “Towards a Soft Science of Soft Robots. A Call for a Place for Aesthetics in Soft Robotics Research,” J. Hum.-Robot Interact., vol. 12, no. 2, Mar. 2023, 15:1-15:11, doi: 10.1145/3533681.

13 W. Fujisaki, N. Goda, I. Motoyoshi, H. Komatsu, S. Nishida, “Audiovisual integration in the human perception of materials,” Journal of Vision, vol. 14, no. 4, Apr. 2014, 12–12, doi: 10.1167/14.4.12.

14 D. Haraway, “Situated knowledges: The science question in feminism and the privilege of partial perspective,” Feminist studies, vol. 14, no. 3, 1988, 575–599.

15 F. A. Robinson, O. Bown, M. Velonaki, “Designing Sound for Social Robots: Candidate Design Principles,” Int J of Soc Robotics, vol. 14, no. 6, Aug. 2022, 1507–1525, doi: 10.1007/s12369-022-00891-0.

16 J. Jørgensen, M. B. Christiansen, “The Sounds of Softness. Designing Sound for Human-Soft Robot Interaction,” Frontiers in Robotics and AI, vol. 8, 2021, 1–17, doi: 10.3389/frobt.2021.674121.

17 M. B. Christiansen, J. Jørgensen, “Augmenting Soft Robotics with Sound,” in Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, in HRI ’20, New York, NY, USA, Association for Computing Machinery, Mar. 2020, 133–135, doi: 10.1145/3371382.3378328.

18 M. B. Christiansen, J. Jørgensen, “SONŌ,” in Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, in HRI ’20. New York, NY, USA, Association for Computing Machinery, 639, Mar. 2020, doi: 10.1145/3371382.3378399.

19 T. A. Klausen, U. Farhadi, E. Vlachos, J. Jørgensen, “Signalling Emotions with a Breathing Soft Robot,” in 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft), Apr. 2022, 194–200. doi: 10.1109/RoboSoft54090.2022.9762140. 

Bibliography
  1. Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, NC, Duke University Press, 2007.
    M. B. Christiansen, L. Beloff, J. Jørgensen, A.-S. E. Belling, “Soft Robotics and Posthuman Entities,” Journal for Artistic Research, no. 22, Dec. 2020, doi: 10.22501/jar.549014.
  2. B. Christiansen, J. Jørgensen, “Augmenting Soft Robotics with Sound,” in Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, in HRI ’20, New York, NY, USA, Association for Computing Machinery, Mar. 2020, 133–135, doi: 10.1145/3371382.3378328.
  3. B. Christiansen, J. Jørgensen, “SONŌ,” in Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, in HRI ’20. New York, NY, USA, Association for Computing Machinery, Mar. 2020, 639, doi: 10.1145/3371382.3378399.
  4. Fujisaki, N. Goda, I. Motoyoshi, H. Komatsu, S. Nishida, “Audiovisual integration in the human perception of materials,” Journal of Vision, vol. 14, no. 4, Apr. 2014, 12–12, doi: 10.1167/14.4.12.
  5. Haraway, “Situated knowledges: The science question in feminism and the privilege of partial perspective,” Feminist studies, vol. 14, no. 3, , 1988, 575–599.
    J. Jørgensen, “Leveraging Morphological Computation for Expressive Movement Generation in a Soft Robotic Artwork,” in Proceedings of the 4th International Conference on Movement Computing, in MOCO ’17. New York, NY, USA: ACM, 2017, 20:1-20:4. doi: 10.1145/3077981.3078029.
  6. Jørgensen, “Prolegomena for a Transdisciplinary Investigation into the Materialities of Soft Systems,” in ISEA 2017 Manizales: Bio-Creation and Peace: Proceedings of the 23rd International Symposium on Electronic Art, University of Caldas, Manizales, Colombia: Department of Visual De- sign, Universidad de Caldas, and ISEA International, 2017, 153–160.
  7. J. Jørgensen, “Interaction with Soft Robotic Tentacles,” in Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction - HRI ’18, Chicago, IL, USA: ACM Press, 2018, 38–38, doi: 10.1145/3173386.3177838.
    J. Jørgensen, “Constructing Soft Robot Aesthetics: Art, Sensation, and Materiality in Practice,” Ph.D. thesis, IT University of Copenhagen, Copenhagen, 2019.
    J. Jørgensen, “From Soft Sculpture to Soft Robotics: Retracing Entropic Aesthetics of the Life-like,” in Shifting Interfaces: An Anthology of Presence, Empathy, and Agency in 21st-Century Media, H. Aldouby, Ed., Leuven, Belgium: Leuven University Press, 2020, 223–242.
    J. Jørgensen, S. Ploetz, “LARPing Human-Robot Interaction,” in HRI 2020 Workshop on Exploring Creative Content in Social Robotics, Apr. 2020, Accessed: Jun. 19, 2020. [Online]. Available: https://portal.findresearcher.sdu.dk/da/publications/larping-human-robot-interaction
  1. Jørgensen, “TeMoG – An Accessible Tool for Creating Custom Soft Robotics Parts,” in Interactivity and Game Creation,
    A. Brooks, E. I. Brooks, and D. Jonathan, Eds., in Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, Cham: Springer International Publishing, 2021, 331–342, doi: 10.1007/978-3-030-73426-8_20.
  2. Jørgensen, M. B. Christiansen, “The Sounds of Softness. Designing Sound for Human-Soft Robot Interaction,” Frontiers in Robotics and AI, vol. 8, 1–17, 2021, doi: 10.3389/frobt.2021.674121.
  3. Jørgensen, “Towards a Soft Science of Soft Robots. A Call for a Place for Aesthetics in Soft Robotics Research,” J. Hum.- Robot Interact., vol. 12, no. 2, Mar. 2023, 15:1-15:11, doi: 10.1145/3533681.
  4. A. Klausen, U. Farhadi, E. Vlachos, J. Jørgensen, “Signalling Emotions with a Breathing Soft Robot,” in 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft), Apr. 2022, 194–200, doi: 10.1109/RoboSoft54090.2022.9762140.
  5. A. Robinson, O. Bown, M. Velonaki, “Designing Sound for Social Robots: Candidate Design Principles,” Int J of Soc Robotics, vol. 14, no. 6, 1507–1525, Aug. 2022, doi: 10.1007/s12369-022-00891-0.
  6. Rus, M. T. Tolley, “Design, fabrication and control of soft robots,” Nature, vol. 521, no. 7553, 467–475, May 2015, doi: 10.1038/nature14543.
    J. Williams, Understanding Poststructuralism, First edition, Chesham, Bucks: Routledge, 2005.
Author Biographies

Jonas Jørgensen is a researcher and artist based in Copenhagen focusing on practice-based transdisciplinary work at the nexus of technical science, media art aesthetics, and the posthumanities. He holds a position as Associate Professor at the Biorobotics section of the University of Southern Denmark where he is co-director of the Center for Soft Robotics. Webpage: https://jonasjoergensen.org/

Mads Bering Christiansen is a Copenhagen-based designer/artist/researcher whose practice evolves around designing objects and experiences to explore and speculate about the potential futures emerging from human technological entanglement. He is currently PhD student at the Center for Soft Robotics at the University of Southern Denmark with a project that investigates the effects of integrating biomorphic aesthetics in robots for social human-robot interactions. Webpage: https://madsbering.com/ 

Gesture and Spatiality in Electroacoustic Improvisation with Digital Video

Michael Lukaszuk, Sojung Bahng

Queen’s University DAN School of Drama and Music Kingston, Canada michael.paul.lukaszuk@gmail.com, sojung.bahng@queensu.ca

Abstract

This paper discusses a multidisciplinary approach to improvisation in which electroacoustic performance is informed by its capacity to transform and respond to characteristics of digital video, along with visual and architectural features of virtual spaces. We describe technical and creative strategies that are focused on leveraging mobile devices in a performance setting for navigating audio-visual relationships in which digital spaces are conveyed through video playback and processing, while gestural control highlights actual physical spaces (e.g., gallery spaces). This creates a hybrid venue that benefits live electroacoustic creation with digital artists whose work extends beyond the domain of sound. The presentation visual analogues to sonic transformations of gesture and spatiality are used to highlight aspects of electroacoustic performance that develop out of a gradual exploration between emulation and obfuscation of referential sound objects. Mappings between electroacoustic instrument and digital video also help delineate the creative use of mobile devices and controllers as being more than tools. At the same time, the transmission of MIDI and OSC (Open Sound Control) messages from mobile instruments allows collaborating artists to resource electroacoustic sound in non-aural contexts.

Keywords

Electroacoustic, Sound and Music Computing, Spatialization, Audio-Visual Improvisation, Digital Instruments.

DOI

10.69564/ISEA2023-16-short-Lukaszuk-et-al-Gesture-and-Spatiality

Introduction

Digital lutherie has been an effective thread for connecting electroacoustic musicians with video and extended reality artists. The data stemming from a digital translation of physical gesture to affect sound diffusion can also be used to convey intersecting spatial relationships for artists who explore proximity and physicality in a visual context. The motivation for this project is to explore the mobile phone as an instrument for multidisciplinary improvisation that involves electroacoustic music. Beyond establishing the mobile phone as a convenient resource for blending artistic or scholarly boundaries, there is a need to consider the organological implications of routing the device through software environments for sound and visual media. This depends on making decisions about what kinds of mappings and idiomatic behaviours help assert a sense of instrumentality, so that a “smartphone-as-instrument” setup conveys compositional drivers that are germane to electroacoustic creation—such as the spectrum between sounds being presented with a relationship to a physical source vs. belonging to a more acoustmatic context. In his article “The Instrumentality of Music,” musicologist Philip Alperson uses the example of Bob Dylan’s infamous performance with electric guitar at the 1965 Newport Folk Festival to emphasize how instruments help us understand the taxonomy of musical genres and how instrumentality, as a concept, goes beyond the notion of a music being “of instruments” to show how instruments place music in relation to larger cultural trends.¹ Focusing on the mobile phone as a collaborative digital instrument helps disrupt notions of traditional musical performance practice that render electroacoustic as non-performative. We will discuss electroacoustic improvisation with digital video in relation to the piece Suffocation (2022), in which gestural control of a mobile device within a multichannel audio setup facilitates continuous reframing of sourced objects within 360-degree video.

Related Work

In the 21st century, trends such as the proliferation of laptop orchestras and the emergence of live coding help establish a context in which seemingly commonplace technologies can be understood as musical instruments through their use within simulated spaces. For example, in composer Mara Helmuth’s laptop ensemble work, pieces such as from Uganda...(2013) use a simple graphical user interface in Max/MSP as an instrument for constructing imaginary sonic environments based on soundscape recordings.² In addition to environmental references, performers interact with the laptop as an electroacoustic instrument by routing their individual playback through a multichannel audio setup used within a concert hall.

Interactive musical mobile apps such as Brian Eno and Peter Chilver’s Bloom (2008) use synthetic or at least ambiguous sampled sounds to create a kind of acousmatic sound world in which there is intentional ambiguity with the manner in which the source material is presented. Bloom also bridges mobile phone performance in relation to digital spaces through a 2018 partnership with Microsoft to create a mixed reality installation in Amsterdam’s Transformatorhuis arts space in which the user interface was projected on larger touch-sensitive surfaces.³ Conversely, Chilvers’ other musical app projects such as Air (2011) relies on the reconfiguration of Irish singer Sandra O’Neal’s recorded vocal performance through gestural interaction with the mobile device. In her PhD Dissertation, Vanessa Chang writes that “through gesture, contemporary digital art practices borrow and recode forms of presences...to highlight a more diverse array of creative agencies.”⁴ A “vocal performance” with “Air” abandons the idea of the voice as a fixed instrument dependent on breath control and becomes an expression of a cybernetic virtuosity that depends on feedback between physical and simulated bodies. 

Suffocation

Suffocation was developed during the fall of 2022, as an improvisational piece stemming from the authors’ involvement in a mixed-media ensemble, and a desire to explore spatiality using an approach that blended our respective practices. For the video content used in this piece, a 360-degree camera was placed inside a small container containing an assortment of tiny objects, such as pins, tissues, pills, and leaves, and poured or added liquids. The everyday objects and materials are defamiliarized by creating the illusion of scale. It provokes a paradoxical sensation between unpleasant sentiments and familiar feelings. The audio content moves through processes of defamiliarization and reimagination of conventional instrumental sounds within a digital context. This is achieved by sampling commonplace instruments such as pianos and kalimbas, while also including computer-generated emulations of these sources that could be bent and distorted by manipulating sound synthesis parameters. 

Figure 1. Manipulation of multichannel audio panning and image rotation using X-Y-Z gyroscope positions from an iPhone.
Figure 2. Gestural spatialization involving more immersive audiovisual relationships through the use of SoundLazer (red) parametric speakers made by inventor Richard Haberkern.

In addition to the use of a quadrophonic (4.0-channel) audio playback system, additional highly directional parametric speakers were positioned in our gallery space to synchronize with the presentation of height and depth in the video content.

The setup for project uses pitch, roll, yaw, accelerometer, and keypad button OSC data transmitted from composer/programmer Kevin Schlei’s smartphone app GyrOSC to affect a Max/MSP patch that acts as a link between electroacoustic sampling and sound synthesis in the digital audio workstation Reaper, and video playback and manipulation taking place through an Isadora patch.

Figure 3. Contrasting gestural spatialization involving more immersive audio-visual relationships through the use of different mobile phone positions to interact with SoundLazer (red) parametric speakers made by inventor Richard Haberkern.

Max/MSP sends MIDI control change and note messages that are mapped to affect spatial parameters of projected video such as scaling and rotation, while simultaneously affecting changes such as triggering recorded audio samples, musical pitches for virtual synthesizer plugins and audio panning positions. It is the simultaneous translation of OSC to MIDI data mapped to Isadora for video and Reaper for audio that allows the smartphone to act as a digital instrument for improvisation.

Figure 4. Example of OSC to MIDI communication for sound spatialization via Max/MSP.

Theme and “Learned Gestures”

The use of a thematic design for structured improvisation with a digital instrument is an important consideration for cultivating the mobile device as an electroacoustic instrument for multidisciplinary collaboration. The use of conceptual bases or programmatic associations helps inform physical gestures that can help the performer discover new ways of interacting with the mobile device while considering notions of idiomaticity and how physical movements (e.g., changes in X-Y-Z axis positions) supports the position that spatialization and abstraction of source material can be principal drivers in audio-visual work. The linking of audio-visual parameters through MIDI and OSC also acts as a form of research-creation for performance in which users can rely on previously learned actions (e.g., swiping, tapping, repositioning the device) in performance to access new forms of creativity. This idea of accessibility issues as being resolved by digital instruments is echoed in computer music research, such as computer music programmer Ge Wang’s documentation of laptop orchestra as a pedagogical tool. His research from the Princeton Laptop Orchestra describes such the ensemble as an experiential workshop environment that allows computer music studies to venture away from a studio/lab environment that separates composition and performance and instead encourages interdisciplinary exploration.⁵

This writing also describes how the ensemble’s use of sound spatialization approach references the hierarchical nature of the orchestra in classical music. While the laptop orchestra model considers spatiality in reference to its comparison with a conventional symphony orchestra, our use of learned gestures in electroacoustic improvisation with digital video adds a new perspective to research-creation efforts in which performance with electroacoustic instruments can also be used for “epistemological intervention into the ‘regimes of truth’...”⁶ Rather than challenging the orchestra paradigm, we focus on reconsidering Eurocentric structural materials that foreground melodically oriented constructs of rhythm and pitch still permeate so much of contemporary sonic performance. By affecting the morphology of sound as connected to video through fluid physical movements, learned gestures become objects within the performance that help delineate how sonic and/or visual media are developing. 

Conclusion

An exploration of digital instruments as reliant on familiar movements is echoed in computer music research, such as Thor Magnusson’s Sonic Writing, which emphasizes that “...young people often enter the world of music through digital technologies (e.g., apps and games).”⁷ By exploring the cybernetic virtuosity in digital sound and video performance through physical gestures learned through everyday interactions with mobile devices, we wish to uncover the possible components of a performance practice that is compatible with live digital art practices such as electroacoustic music—in which spatiality is so essential to the performance experience. A traditional orchestral performer may have to learn how to use instrumental techniques such as vibrato to better project the sound of their instrument in a space, the use of a mobile phone in an improvisational piece such as Suffocation exposes a multiplicity of layers in which rhythm, cadence and narrative structure are also conveyed by the way that the performer explores the proximity and distance of their own body as connected to digital audio and visual materials.

References

1 Philip Alperson, “The Instrumentality of Music,” The Journal of Aesthetics and Art Criticism 66/1, Winter, 2008, 37-51.

2 Michael Lukaszuk, “Soundscapes and Anachronisms in Music for Laptop Ensemble,” eContact! 21/1, Spring 2022, https://econtact.ca/21_1/lukaszuk_laptop-soundscapes.html

3 Susanna Ray, “Brian Eno, Peter Chilvers create ‘quite magical’ flower garden of sound in Amsterdam with ‘Bloom: Open Space," Microsoft/Features, Accessed May 15, 2021, February 22, 2018, https://news.microsoft.com/features/brian-eno-peter-chilvers-create-quite-magical-flower-garden-sound-amsterdam-bloom-open-space/

4 Vanessa Chang, “Tracing Electronic Gesture: A Poetics of Mediated movement," PhD dissertation, Stanford University, 2017.

5 Ge Wang, et al, “The Laptop Orchestra in the Classroom,” Computer Music Journal 32/1, Spring 2008, 26-37.

6 Owen B Chapman, Sawchuck Kim, “Research-Creation: Intervention, analysis and ‘family resemblances’,” Canadian Journal of Communication 37/1, April 2012, “Media Arts Revisited,” 5–26.

7 Thor Magnusson, Sonic Writing, London, Bloomsbury, 2018. 

Generated tools: A Defamiliarizing Approach to Creating ML Art

Chloe McFadden and Oliver Bown

School of Art and Design, UNSW
New South Wales, Australia chloe.mcfadden@unsw.edu.au, o.bown@unsw.edu.au

Abstract

In recent years there has been an increase in the adoption of machine learning (ML) systems that can generate novel images. This increased use may reveal the beginning of a familiarity in which the implications of these emerging technologies are naturalised or made increasingly invisible. Thus, practices which can disrupt familiarity may allow us to create experiences of heightened awareness in which we can consider our engagement with this emerging technology. In this paper, I discuss the outcomes of working with a Generative Adversarial Network (GAN), using a dataset created from the hand tools section of a popular Australian hardware store. Through this creative practice, I investigate how artists can use ML as mechanisms for creating artworks that disrupt, investigate and defamiliarize the known.

Keywords

Generative Adversarial Networks, Machine Learning, Creative Artificial Intelligence, Tools, Generative Art.

DOI

10.69564/ISEA2023-17-short-McFadden-et-al-Generated-tools

Introduction

As digital technologies are increasingly embedded into the everyday and the ways we construct and interact within our world perception, it has been noted that they also become increasingly familiar, invisible, or subconscious.¹ While often perceived as a ‘tool’—an extension of the human hand designed to fulfil a certain function—our relation to technology is increasingly being understood as a symbiosis of interdependence.² Furthermore, the co-evolutionary relationship between tools and humans has also been noted—highlighting the influence these ‘tools’ have in shaping us, as we shape them.³ As we continue to see, use, and adopt these tools, they become increasingly familiar and every day. However, with this increased familiarity may come a decrease in active or conscious visibility resulting in the loss of regular inciting provocation to question how we relate, interact, and construct our perception of the world with these technologies. Thus, techniques of making the familiar unfamiliar may be a strategy for creating experiences of heightened perception of or critical engagement with technology.

Generated tools (2021) is a practice-based research project in which I worked with a GAN to create an installation that features tools as subject matter. In this, I am exploring how we can use ML conceptually to create defamiliarizing experiences which may allow us to critically reflect upon and re-engage with concepts, worldviews and ideas which have been naturalised. ML that can generate images is compelling technology—sparking new ways of making and creative possibilities. However, it also has the potential to be a reaffirming conservative force, as it is informed by the training dataset and may familiarize and perpetuate the visuals, ideas and assumptions hosted within.⁴ While this reliance upon the dataset may pose a risk if we engage uncritically—this also presents the opportunity for artists to disrupt, explore and denaturalise the known. Thus, through the documentation and discussion of work created with a GAN, this paper presents the concept of defamiliarization as one approach for understanding the potential impact or possibilities of working with ML to generate art.

Tools, Technology, and Familiarity

There has been much critical reflection on the relationship between humans and tools—both in how tools present an invisible familiarity, as well as in the mutual influence we exert. Martin Heidegger explained tools as "ready-to-hand", meaning as long as they function correctly or to our expectations, they remain concealed from view, or not in our conscious presence.⁵ However, while tools typically fade into the realm of the unconscious everyday, they also reveal a co-evolutionary relationship between tools and humans. While we typically narrativize our relation to tools as being a one-way influence, contemporary philosophical approaches recognise that it is more of a symbiosis. To solve an issue or to respond to our environment, we create new tools, which in turn change how we act and lead to the discovery of new problems or use cases for tools as this cycle repeats itself.⁶

Similar lines of thought have also been applied to technology. It has been suggested that technology is an extension of ourselves with which we cohabit a shared ecosystem. Through this cohabitation, we are able to coextend our skills, capabilities, and properties, changing how we act in ways we perceive as beneficial, and in turn leading to the development and adoption of new technologies.⁷

However, as this cycle of development and adoption continues, this cohabitation becomes increasingly familiar or subconscious.⁸ While technological innovation feels novel at first, over time with increased adoption and use, repeated interaction results in a comfortable familiarity that doesn’t necessarily encourage reflection or critical engagement.

In the past 5 years, there has been increased adoption of ML systems that can generate novel images. While there is a large amount of general ‘hype’ and awareness, the development of systems like DALLE-2, Stable Diffusion and Mid Journey, which all have user-friendly demos and applications hosted online, alongside apps like WOMBO Dream and TikTok’s green screen AI filter, may reveal the beginning of familiarity, or a future of familiarity with ML systems.

Defamiliarization and visual indeterminacy

If familiarity reduces technology to the subconscious, thus dampening opportunities to be aware of how we relate to and are shaped by it, unfamiliarity becomes a potential strategy for sparking conscious engagement. Coined by Viktor Shklovsky in 1917, defamiliarization is understood as a tactic for creating heightened awareness or perception by halting our automatic assumptions about a subject, allowing us to view it again for the first time.⁹, ¹⁰, ¹¹ The goal of such a practice is not to reveal a more objective truth about a subject, but rather to create a heightened perception of how we construct, understand, and relate to the subject.¹² Defamiliarization has also been identified as a common tactic employed by digital artists to create experiences of critical distance between audiences and technology, to temporarily make the familiar unfamiliar for heightened perception to be achieved.¹³

The ways that artists enact this varies broadly, as it is understood that all art forms can generate a defamiliarizing effect.¹⁴ However, one emerging approach related to ML is visual indeterminacy. It has been noted that GANs typically create uncanny, strange, or visually indeterminate images.¹⁵ Visual indeterminacy, as well as ambiguity and uncertainty, has also been recognized as useful tools for prompting multiple interpretations or disrupting an artefact’s socially encoded properties.¹⁶, ¹⁷ Furthermore, artists may be able to engage the differences between our logic and the chosen system’s logic to create defamiliarizing effects. For example, when working with image-based ML, we understand images as representations of objects, scenes, and worldly concepts, while a system like a GAN is attempting to map the underlying structural logic of the dataset at a pixel level. Engaging the system’s ability to recreate patterns within the dataset, we can generate coherent and recognisable forms. However, by navigating their latent spaces, we can push them to create semi-coherent, or visually indeterminate forms. Thus, working with ML to generate art may present opportunities to denaturalise our understandings through visual indeterminacy or strangeness, via this difference in structural logic.

Installation description

The installation consists of a tool wall which houses three instances of working with a StyleGAN2 trained using a dataset of images of hand tools sourced from the Bunnings Warehouse website (an iconic Australian hardware store).¹⁸ The first instance shows 3D printed tools, modelled and printed using the generated images by the GAN as reference. The second instance shows 2D “latent space walk” videos projected on the tool wall, created by incrementally sampling the latent vector space generated by the GAN. The final instance shows doctored Bunnings Warehouse product catalogues, featuring images of tools generated by the GAN. Audience members are encouraged to engage with the work by flipping through the product catalogues, as well as being able to pick up, play with and rearrange the 3D printed objects on the tool wall.

Figure 1. Image of Tool Wall Installation featuring 3D prints and 2D project latent space walk videos.

3D Printed tools

While images created by ML systems are becoming increasingly familiar, we most commonly encounter them in digital spaces, as images or animations. Thus, one goal of the work was to translate the generated images into 3D objects to observe whether this could be a method of creating defamiliarizing experiences. The 3D printed tools were created by first generating a series of images using a StyleGAN2 trained on a dataset of images of hand tools sourced from the Bunnings Warehouse website. While the GAN is trained using these images of hand tools, how these tools exist and function in conceptual space is not captured. Thus, when it generates images of tools it does so with no consideration of functionality—creating tools that have no clear associated uses. Simultaneously, we can expect the generated images to be aesthetically tool-like as the GAN is attempting to replicate the original dataset and find its underlying structural logic. Thus, the work explores whether the ‘uselessness’ of the GAN-produced artefacts is an effective tactic of defamiliarization.

Margaret Boden discusses ‘useless’ artefacts as having the ability to playfully challenge expectation, while bringing the typical affordance of similar items to the foreground.¹⁹ Tools are useful—but most of the time they are not being used—hung upon tool boards like artworks, neatly organised in boxes like collectables, or haphazardly thrown in draws like junk. They often also hold semantic and sentimental value - tools are given as gifts, passed down through families and lent between friends and communities. If tools remain concealed from perception so long as they function correctly, how does encountering inherently useless tools bring these hidden ‘uses’ to the foreground?²⁰

Figure 2. GAN generated image of a tool.

The generated images were then curated and used as references to model and texture 3D tools in Autodesk Maya and Mudbox. Reference images were selected through a process of working through the generated images and selecting a broad range of shapes, tool-like iconography and sizes that existed within the possibility space. The goal when modelling and texturing the tools was to follow the reference images closely, while also recognizing the inherent role of interpretation when translating from 2D to 3D. These models were then 3D printed using a variety of grey, black, and aluminium filament. The results are visually and texturally strange, tool-like artefacts. As they were modelled using the StyleGAN2 curated images as reference, they have a visually indeterminate quality—with familiar elements (e.g. handles, pointy ends, and bits and pieces of recognizable tools) combined or blurred in unfamiliar ways. Further, since they are 3D printed, the tools have unique ridges, divots and holes which create an unfamiliar hand feel. The light-weight material of the filament creates a unique engagement with the tools which we typically associate with rubbery, metal, and heavy sensations.

Figure 3. 3D print of GAN generated image of a tool.

Through the combination of recognizable tool-like elements in unfamiliar ways with the 3D printed materiality, the familiar functions and feelings that are tied to how we perceive, construct and experience tools subconsciously are no longer present—opening the space for new interpretations and speculation. As audiences would approach the strange tools, typically their first reaction was to generate a new use case for the tool. In short, this element of the work combines visual indeterminate ‘useless’ tools supplied by the GAN with subversive, textural 3D prints to create a speculative experience for audiences, engaging difference in structural logics, visual indeterminacy and unfamiliar materiality as tactics of defamiliarization.

2D latent space walk videos

The projected 2D latent space walk videos were created using the possibility space generated by the StyleGAN2 trained on the dataset of hand tools. Latent space walk videos involve the sampling and incremental changing of points in the latent vector space which can then be used to create animations. As the system was trained on a dataset of hand tools, the resulting animations present fluid tools that merge and shape into one another. While we typically conceptualise tools in a worldly context—with clear boundaries, defined shape language and affordances that are affirmed via learnt aesthetics (e.g., handles, spouts)—the GAN’s distribution renders a more fluid visualisation.

Thus, the work is interested in whether the visual indeterminacy created by the GAN is an effective strategy of defamiliarization. As visual indeterminacy is understood as a tactic to engage audiences with the active nature of seeing and meaning making, it may engender defamiliarizing experiences.²¹, ²²  Furthermore, visually indeterminate art prolongs perception through the combination of “apparently detailed and vivid images resist identification.”²³ The generated latent space walk videos thus present strange tools trained using a highly recognisable visual dataset, which now resist specific identification via the GAN’s involvement. Whether this resistance of identification invokes the active nature of seeing, or creates experiences of heightened perception is unclear—does coming into the contact with the work go beyond strange? Do these splodgey, blurry tools engender an active awareness or reflection on the symbiotic nature of tools and technology?

Product catalogue

Accompanying the tool wall are doctored product catalogues which have been inserted with images of tools generated by the GAN. Using Photoshop, the original composition of the catalogue was closely followed as the GAN generated tools were added based on their perceived visual similarities. The goal was to create a catalogue that could be believably passed off as real at first glance.

The resulting catalogue is a mixture of the original text, lifestyle images, prices, and generated tools. This presentation results in a double-take effect. Unlike the 3D printed tools which are outwardly alien or strange by design, in this presentation, they almost look like real tools inside of a real catalogue at a cursory glance. The context the product catalogue provides—the text, price tags, product descriptions, branding and lifestyle images—almost act as visual vouchers for authenticity of these generated tools.

Returning to defamiliarization, Shklovsky positions the purpose of art as to “make forms difficult, to increase the difficulty and length of perception because the process of perception is an aesthetic end in itself and must be prolonged."²⁴ Thus, the work engages surprise, understood as violation of expectation²⁵, and appropriation as potential strategies of defamiliarization. By appropriating the catalogue and inserting familiar, yet strange tool-like artefacts, the doubletake causes an increase in perception, calling us to look a little closer and make sense of the nonsensical tools.

In this, the work explores whether the surprise and absurdity of the nonsensical catalogue provides space to investigate the socially coded meanings present in the subject matter that are familiar and normalised. For example, does the messaging and imagery surrounding Father’s Day become less natural or familiar through the introduction of the generated tools?

Figure 4. Doctored Bunnings product catalogue featuring GAN generated images of tools.

Conclusion

As the practice of generating images using ML becomes increasingly familiar, we need to be aware and considerate of the ways it may naturalise or reinforce worldviews, categories, and ideas via this engagement. However, just as ML has the ability to be conservative, to narrow in on the dataset and to perpetuate the ideas within—so too can artists employ it to disrupt, investigate and defamiliarize the known.

Through the documentation and analysis of my work, Generated Tools (2021), this paper presents an experimental approach to generating art with ML to create defamiliarizing experiences. As a part of this discussion, I have explored how artists can engage with visual indeterminacy and the structural logics of ML systems, as a method for disrupting the assumed, and reapproaching the known with fresh perspective.

Furthermore, through the analysis of the 3D prints, projected latent space walk videos and doctored product catalogues, I have investigated how the presentation of ML generated images can create, alter, and deepen the defamiliarization experience.

In this I am interested in how we can ‘make strange’ in ways that matter. As ML presents unique opportunities for defamiliarization—how can artists defamiliarize in ways which persist beyond the initial encounter? How can we blur, splodge and surprise with ML in ways that recover both the subjects of datasets and technologies themselves from the automatism of perception and use?

Acknowledgements

This project was the first author’s (Chloe McFadden) honours project supervised by the second author (Oliver Bown); hence the paper is written in the first person singular.

References

1 I. Hwang, M. Guglielmetti, V. Dziekan, "The Familiar": technology-being-with-us," Usage guidelines, 2016, p.65.

2 É. Brangier, S. Hammes-Adelé, Beyond the technology acceptance model: Elements to validate the human-technology symbiosis model, in International Conference on Ergonomics and Health Aspects of Work with Computers, Springer, 2011.

3 J. Navarro, P.A. Hancock, Did Tools Create Humans? Theoretical Issues in Ergonomics Science, 2022, 1-27.

4 Kate Crawford, Trevor Paglen, "Excavating AI: the politics of images in machine learning training sets," AI & SOCIETY, 2021, doi: 10.1007/s00146-021-01162-8.

5 Harman Graham, "Technology, objects and things in Heidegger," Cambridge journal of economics 34, no. 1, 2010, 17-25.

6 J. Navarro, P.A. Hancock, Did Tools Create Humans? Theoretical Issues in Ergonomics Science, 2022, 1-27.

7 É. Brangier, S. Hammes-Adelé, Beyond the technology acceptance model: Elements to validate the human-technology symbiosis model, in International Conference on Ergonomics and Health Aspects of Work with Computers, Springer, 2011.

8 B.C Bruce, M.P. Hogan, The disappearance of technology: Toward an ecological model of literacy, in Writing in a Technological World, 2019, Routledge, 191-207.

9 V. Shklovsky, Art as technique [1917], The Critical Tradition: Classic Texts and Contemporary Trends, New York, 2007, 3, 774-784.

10 L. Crawford, Viktor Shklovskij: Différance in Defamiliarization, Comparative Literature, 1984, 209-219.

11 D.P. Gunn, Making art strange: a commentary on defamiliarization, The Georgia Review, 1984. 38(1), 25-33.

12 R. Bellanova, A.R. Sætnan, How to Discomfort a Worldview?: Social Sciences, Surveillance Technologies, and Defamiliarization, in Science, Technology, and Art in International Relations, Routledge, 2019, 29-39.

13 L. Starkand, K. Crawford, The work of art in the age of artificial intelligence: What artists can teach us about the ethics of data practice, Surveillance & Society, 2019. 17(3/4), 442-455.

14 D.P. Gunn, Making art strange: a commentary on defamiliarization, The Georgia Review, 1984. 38(1), 25-33.

15 Aaron Hertzmann, "Visual indeterminacy in GAN art," Leonardo 53, no. 4, 2020, 424-428.

16 P. Yurman, A.V. Reddy, Drawing Conversations Mediated by AI, in Creativity and Cognition, 2022.

17 J.J. Benjamin, et al, Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry, in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021.

18 Karras Tero, Samuli Laine,Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila, "Analyzing and improving the image quality of stylegan," In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8110-8119, 2020.

19 M. A Boden, Creativity and art: Three roads to surprise, Oxford University Press, 2010, 63.

20 Graham Harman, "Technology, objects and things in Heidegger," Cambridge journal of economics 34, no. 1, 2010, 17-25.

21 D. Gamboni, Potential images: Ambiguity and indeterminacy in modern art, Reaktion Books, 2002.

22 R. Pepperell, "Seeing without objects: Visual indeterminacy and art," Leonardo 39(5), 2006, 394-400.

23 A. Ishai, et al, "Perception, memory and aesthetics of indeterminate art," Brain Research Bulletin 73(4), 2007, 319-324.

24 V. Shklovsky, "Art as technique," Literary theory: An anthology 3, 1917, p.16.

25 M. L Maher, et al, Computational models of surprise in evaluating creative design, Proceedings of the fourth international conference on computational creativity, Citeseer, 2013.

Ecologies of Thought: Generative Art as a Collaborative Research Methodology with Guarani and Kaiowá Indigenous Communities

Matheus da Rocha Montanari

University of São Paulo, Polytechnic University of Valencia, Multimedia Anthropology Lab (UCL). São Paulo, Brazil/ Valencia, Spain
matheusrmontanari@gmail.com

Abstract

The Ecologies of Thought project aimed to reconceptualize the relationships between ecological and technical knowledge, seeking an epistemological understanding that pushed beyond nature vs. culture divides. More than a conceptual and theoretical proposal, which was based on the technodiverse notion of cosmotechnics, the project developed practical and experimental methodologies in collaboration with the Guarani and Kaiowá Indigenous communities of Brazil to further this investigation. With an international and interdisciplinary partnership, we researched relationships between sounds and plants in indigenous cosmology and the ways in which these relationships can help us to more deeply understand a notion of ecology that is based on the poetics of care. With that, we created a series of collaborative virtual reality experiments using generative art processes, which proved to be an interesting methodology of artistic creation as research.

Keywords

Indigenous Communities; Generative Art; Ecology; Virtual Reality; Anthropology; Cosmotechnics; Digital Art; Art as Research; Multimedia Anthropology; Guarani Kaiowa.

DOI

10.69564/ISEA2023-18-short-Da-Rocha-Montanari-Ecologies-of-Thought

Introduction

Between March and July 2022, the Ecologies of Thought (1) project established an international and multidisciplinary partnership between Indigenous communities of Brazil, artists, anthropologists, botanists, and audiovisual producers to investigate Guarani and Kaiowá ecological thought through the relationships between sound and plants. The project proposed dialogues among different types and conceptions of technology, from chanting and the traditional cultivation method, to the use of microcontrollers and data analysis.

This project is part of the Guarani and Kaiowá Virtual Museum (2), which develops a virtual reality museum curated by the community elders and shamans – Nhandesys (female), and Nhanderus (male) in partnership with the Multimedia Anthropology Lab at University College London. 

Since the beginning of the project, the community has been challenging and redefining the notion of a "museum," the extra-human relationships among plants, animals, instruments, and spirits present us with a complex challenge: how to maintain a Guarani and Kaiowá cosmotechnical fidelity in the museum?¹ That is, how to enable the indigenous cosmology, and its technologies, not only to inhabit the virtual space but to build it as a tekoha, a territory, that allows the exercise of the Guarani and Kaiowá way of being and living, teko.² That means, how to build a digital Guarani and Kaiowá tekoha?

To our surprise, in the first test of the museum presented to the community with the virtual reality headset, two Nhandesys, experienced chanters, saw Jaras, spirits, in the virtual space. From this encounter, we began to investigate different ways to explore the spiritual and cosmological aspects, which are so present in the Guarani and Kaiowá ecology of thought, within the museum. The relationship between sound and plants proved to be a fertile field for this research. We began to learn more from the community elders and apprentices who introduced us to chanting as a cultivation technology: a physical and spiritual dialogue between the plant, its spirits, and the chanter. To chant is to care for, and the poetics of care is essential for the development of a healthy territory, be it physical or digital.³

We learned that instruments are also people. And that the Guarani and Kaiowá notion of preservation is different. For example, if the instrument is kept and not used, even if in a collection, it loses its ability to produce sound because it loses its ability to create connections when it is removed from its communication network. Therefore, it became evident the need to explore different ways to activate the collection we have been building for the museum.

To this end, we developed an experimental strategy. We focused on the creation of virtual reality worlds that, using generative art processes, explored the Guarani and Kaiowá cosmology from its cosmotechnical aspect. We used sounds and traditional elements, such as the Chiru, the sacred stick that supports the world, and the white corn, which is inhabited by one of the most important spirits (Jakairá), to investigate fundamental elements of the cosmology through an experience, rather than a narrative representation.⁴ In light of the urgent need to revise and rethink human-ecology relationships on a global scale, while acknowledging the new epistemic possibilities of data capture and analysis technologies, we collaborated with Guarani and Kaiowá community members to explore the different and creative kinds of data that these devices afford, and the ecological knowledge they make possible.

The Guarani and Kaiowá

Mato Grosso do Sul is a state in the mid-western region of Brazil, bordered by Paraguay and Bolivia, in a transition region between three biomes: Atlantic Forest, Cerrado, and Pantanal. It is the State with the second-largest indigenous population in the country and the largest outside the Amazon. The Guarani Ñhandeva and Kaiowá population of the region in 2021 is estimated to be 63.5 thousand people. At the same time, the State has the second-worst land distribution index in the country, with large properties (> 1,000 ha) occupying 83% of the total area. Thus, it is not surprising that the region is a scenario of serious conflicts between landowners and indigenous people, especially in the area of cattle ranching, sugarcane, corn, soybeans, and eucalyptus plantations.⁵

The Guarani and Kaiowá are Guarani speakers, from the Tupi-Guarani linguistic trunk. They have a strong ancestral territorial bond, both physical and spiritual, which guides their struggle in an especially organized way since the large assemblies beginning in 1979 (Aty Guasu). These articulations occur mainly between the religious leaders, Nhanderus and Nhandesys, since the community considers that the simultaneous performance of the religious ritual (jeroky) "is fundamental to recover the dialogue with the invisible beings and the guardians of the ancient tekohas," that is, for the recovery of the historical territories of their people, it is also fundamental to recover the extra-human relations developed in the territories, whether they are animal, vegetable, and/or spiritual.⁶

These relationships are directly connected to chanting and praying. The chant, besides a dialogue, is part of the materiality of the world: "In the understanding of the spiritual leaders, the singularities of the physical world inevitably need singing to continue their continuous existence, otherwise the world will gradually end."⁷ This is one of the reasons we choose sound, specifically chant, as a source of investigation of the ecological relations in Guarani and Kaiowá thought. 

Artistic creation as an interdisciplinary collaborative methodology

The Guarani and Kaiowá cosmology have a great degree of complexity and diversity, even in the same family trunk, some stories may vary between one group and another. Moreover, the narratives are full of recursion and non-linearity, in which each detail branches off into another story, that can take days to be told in the traditional oral form. So, it was clear from the beginning of the project that there would be limitations on the form and amount of information we could present. At the same time, we did not want to make a reductionist narrative translation that would create a shallow representation of the generous exchanges that the project made possible.

Besides that, the diverse points of view and areas of knowledge involved in the project configure, simultaneously, its richness and complexity. The challenge of combining shamans, scientists, and artists in the same project revealed the power of the digital medium as a research tool in itself, and not only as a form of scientific dissemination. More than that, we set out to develop research in visual arts, rather than research about visual arts. That is, to reaffirm the processes of creation as valid ways of researching, thinking, and producing knowledge about the world.⁸ In this sense, artistic creation acts in the project as a collaborative methodology capable of providing dialogues and investigative interactivity: the creation processes allowed for the creation of a two-way exchange, in which collaboration affected not only the production but the understanding and direction of what was being produced.

In this project, art, which occupies a liminal place, is not limited to an art-technology extension of scientific knowledge. We seek to embrace the mba'e kuaa, the Guarani and Kaiowá technical knowledge and know-how. In this proposal, we guide ourselves with the concept of cosmotechnics developed by Y uk Hui. That is, to rediscover the diversity of technical thought beyond the Greek notion of techné:

Technology is not anthropologically universal; its functioning is assured and limited by particular cosmologies that go beyond mere functionality and utility. Thus, there is no single technology, but a multiplicity of cosmotechnics.⁹

Generative Art and the Guarani & Kaiowá Cosmology

Generative art can be defined in different degrees of complexity, it is associated with a system that has some degree of autonomy, in which the artist and the system exercise a series of operations that result in the final work.¹⁰ Seeking ways to engage with the mythology and respect their characteristics, generative art methodologies seemed to not only fit with the ontological qualities of the Guarani and Kaiowá cosmology but to privilege an investigation and audiovisual production that could maintain a cosmotechnical fidelity. Since both, the Guarani and Kaiowá cosmology and generative art are nonlinear emergent processes with repetitive elements that are in constant transformation. Working in that direction, we created a series of six experiments, and in this article, we will be presenting one of them.

Jakairá: white corn, bees, and the maintenance of the world

According to the Kaiowá myth, the deity Jakairá is responsible for the creation of the first roçado (food garden). Through his wisdom, Jakairá created the white corn, which is at the top of the food hierarchy, and, from it, all other agricultural products were created. Therefore, Jakairá and white corn are responsible for the creation and maintenance of almost all food on Earth. Its cultivation follows a series of rules, from dietary restrictions of the growers to the use of chants to cheer the corn and its baptism.¹¹ The corn baptism ritual, called Jerosy Puku, is one of the most important Kaiowá rituals. Through it, the corn becomes fit for consumption by the community in a healthy way, since, besides being a physical food, it is also considered a spiritual food. The baptism is also a moment of reaffirmation of the Kaiowá way of being, strengthening the human and extra-human (spirits, plants, animals) relationships of the community.¹² This specific type of corn is currently in danger of extinction due to the environmental threats it suffers, the heavy use of pesticides in the GMO corn plantations that surround the communities, and the consequences of climate change in the region.

Our first experiment arises precisely from this extrahuman ecology. In one of our ethnographic sessions on the importance of corn and its relationship to sound, we learned that corn has its own sound that is not audible to all people. However, there is a specific moment when the sound of corn can be heard by everyone. The community members told us that when the corn, still green (avati kyry), begins to mature and get ready for harvest, it produces a smell that attracts bees so that "they can make the sound of the corn," warning the cultivator that it is almost time to harvest. As talking about the white corn is also talking about the beginning of life and the food that sustains it, this pre-harvest moment, when the bee is attracted to the corn to make its sound, seemed emblematic to us to investigate an instant of the creation of the world according to the Guarani and Kaiowá cosmology. This moment that seems, at first sight, to deal with a multispecies relationship between the plant and the bee, when further investigated, reveals, also, to be a relationship between the spirit owner of the corn and its cultivator¹³. To continue this investigation, we developed a generative art experiment using the TouchDesigner programming language: a visual programming language in nodes for the creation of real-time interactive multimedia content. We started with the image of a white cornfield before the harvest and analyzed the light incidence in its different areas. From this data analysis, we created a relief map that was applied to an ellipse for the construction of a terrain. The darker areas of the image created depressions, and the lighter areas, elevations. Finally, we added two noise layers, one algorithmic, and another from the sound analysis of the frequencies produced by the bees. The sound frequency analysis controls the algorithmic noise, which generates motion on the terrain. With that, we exported a 360o video for VR headsets. In this way, we developed a virtual reality experience in which the ecological relationship between the corn and the bee produces the creation of a generative world based on the analysis of image and sound data from the field, guided by the Guarani and Kaiowá cosmology (figure 1). 

Even without a naturalistic production of the image, and perhaps precisely because of this characteristic, the community approved the experiment. Once the step-by-step construction of the generative program was presented, there was consensus that the experiment had true Guarani and Kaiowá characteristics, mixed with non-indigenous technology. From this experiment on, the community began to direct the following more directly, choosing images and elements that should be used for the continuation of the project.

1. Still from the VR experience – Generative Terrain produced by the data analysis of the white corn and the bee’s sounds.

Conclusion

The Ecologies of Thought project aimed to reconceptualize the relationships between ecological and technical knowledge, seeking an epistemological understanding that pushed beyond nature vs. culture divides. More than a conceptual and theoretical proposal, which was based on the technodiverse notion of cosmotechnics, the project developed practical and experimental methodologies in collaboration with Guarani and Kaiowá communities to further this investigation.¹⁴ With an international and interdisciplinary partnership, we researched relationships between sounds and plants in indigenous cosmology and the ways in which these relationships can help us to more deeply understand a notion of ecology that is based on a poetics of care.

Thus, we were able to bring together poetic and technical processes to propose a mixture of knowledge that beyond an explanation or representation of one form of knowledge into another, develops an experience of its own. This allows us to fight an idealized naturalistic representation of indigenous culture, which fails to understand indigenous thought and technologies as valid ways to build and think of a cosmotechnic for our time.

More broadly, this project seeks to undermine the romantic and colonial notion of preservation, which does not consider the indigenous presence a digital presence, able to act and think about technical processes since its conception. This means, beyond an illustrative representation of what is believed to be the material and immaterial cultural heritage of the community, to collaboratively build a digital-indigenous territory, a digital Guarani and Kaiowá tekoha where a new technological ontology is established, in a way that the very conception of contemporary technology is affected by indigenous cosmotechnics.

In conclusion, the epistemic research proposed and developed during the project revealed to us the potential that this collaborative methodology possesses to go beyond epistemological investigations, enabling us to revisit ontological questions of technology. Collaboration, interdisciplinarity, and the investigation of creative processes—rather than the proposition of representations—allows us to arrive at questions related to the level of existence, so that, in the anthropocenic era of the absence of futures, we can compose a cosmotechnic for a viable, virtual and actual, reality, based on a poetics of care and more-than-human relations. 

Acknowledgments

We would like to acknowledge the University College London Global Engagement Fund which contributed to the partnership among the Multimedia Anthropology Lab at University College London, the Kunangue Aty Guasu, IDAC, The School of Botany, The University of São Paulo, The Federal University of Santa Catarina.

We would like to thank and acknowledge the generosity of the Nhandesys and Nhanderus whom we have been working with during these years. Nhandesy Neuza Alziro, Nhandesy Ivone, Nhandesy Fausta Solano Mendosa e Nhandesy Roselí Aquino. Nhanderu Tadeu Romeiro Freita, Nhanderu Valdomiro Oswaldo Aquino. Also, the team of UCL’s Multimedia Anthropology Lab and the Kunangue Aty Guasu: Valdineia Aquino, Scott Hill, Luan Iturve, Germano Alziro, Jaqueline Gonçalves, Raffaella Fryer-Moreira, Fabi Fernandes, Gerard Oliver, Adriana Boloc, Stephanie Ferraz, Clara Ciqueira, Luiza Braga and MAL’s translation team. And our partners Karola Braga, Anderson Santos and Caetano Sordi.

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) – Finance Code 88887.716693/2022-00. 


(1) https://www.matheusmontanari.com/ecologiesofthought
(2) https://www.uclmal.com/virtual-museum

References

1 Yuk Hui, The question concerning technology in China: an essay in cosmotechnics, Falmouth, Urbanomic, 2018.

2 Tonico Benites, “Trajetória de luta árdua da articulação das lideranças Guarani e Kaiowá para recuperar os seus territórios tradicionais tekoha guasu,” Revista de Antropologia da UFScar, Vol. 04, No. 02, accessed October 14, 2022, https://doi.org/10.52426/rau.v4i2.83

3 Matheus Montanari and Gilbertto Prado, “From vigilance to vigil: an introduction to an alternative paradigm for technology, art, and life,” Diffractions, Vol. 01, No. 05, accessed October 14, 2022, https://doi.org/10.34632/diffractions.2022.10194

4 Fabio Mura, “A Trajetória dos Chiru na construção da tradição de conhecimento Kaiowá,” MANA, Vol. 16, No. 01, accessed October 14, 2022, https://doi.org/10.1590/S0104-93132010000100006

5 Anderson de Souza Santos, Luiz Henrique Eloy Amado and Dan Pasca, “"É muita terra pra pouco índio"? Ou muita terra na mão de poucos? Conflitos fundiários no Mato Grosso do Sul,” Instituto Socioambiental, accessed October 14, 2022, https://acervo.socioambiental.org/acervo/documentos/e-muita-terra-pra-pouco-indio-ou-muita-terra-na-mao-de-poucos-conflitos

6 Tonico Benites, “Trajetória de luta árdua da articulação das lideranças Guarani e Kaiowá para recuperar os seus territórios tradicionais tekoha guasu,” Revista de Antropologia da UFScar, Vol. 04, No. 02, accessed October 14, 2022, https://doi.org/10.52426/rau.v4i2.83,171

7 Izaque João, “Jerosy Puku,” PISEGRAMA, Vol. 01, No. 06, accessed October 14, 2022, https://piseagrama.org/jerosy-puku/,1

8 Sandra Rey, “Por uma abordagem metodológica da pesquisa em artes visuais,” in O meio como ponto zero, ed. Bianca Brites and Elida Tesser, Porto Alegre: Editora da UFRGS, 2002, 123-140.

9 Yuk Hui, Tecnodiversidade, São Paulo, Ubu, 2020, 25.

10 Philip Galanter, “Generative Art Theory,” in A companion to digital art, ed. Christiane Paul, Hoboken, John Wiley & Sons, 2016.

11 Izaque João, “Jerosy Puku,” PISEGRAMA, Vol. 01, No. 06, accessed October 14, 2022, https://piseagrama.org/jerosy-puku/

12 Izaque João, “Jakaira Reko Nheypyrũ Marangatu Mborahéi: Origem E Fundamentos Do Canto Ritual Jerosy puku Entre Os Kaiowá De Panambi, Panambizinho E Sucuri’y, Mato Grosso Do Sul”, Master’s. diss., Programa de Pós-Graduação em História da Faculdade de Ciências Humanas, Programa de Pós-Graduação em História da Faculdade de Ciências Humanas, 2011.

13 Carlos Fausto, “Donos demais: Maestria e Dominio na Amazônia,” MANA, Vol. 14, No. 02, accessed October 14, 2022, http://dx.doi.org/10.1590/S0104-93132008000200003

14 Yuk Hui, The question concerning technology in China: an essay in cosmotechnics 

Author Biography

Matheus Montanari is a Ph.D. Candidate at the University of São Paulo in Visual Arts, and external researcher at the Polytechnic University of Valencia. He develops works at the intersection of art, science and philosophy, investigating ways to rethink technology after art. He is interested in cosmotechnical diversities, and in combining decolonial ecological thinking with technical know-how. Recently he has been developing his practice in collaboration with indigenous Guarani and Kaiowá communities in Brazil. He has exhibited his work in Brazil and abroad, in countries such as: Argentina, Portugal, Austria, Croatia, China, Spain and Italy. He was awarded the web art category prize of the 67th Contemporary Art Salon of Paraná. 

Emerging Strategies “Under The Bay” in AR/XR

Lisa Moren, Tsvetan Bachvaroff

University of Maryland Baltimore County (UMBC), Institute of Marine and Environmental Technology (IMET), University of Maryland Center for Environmental Science (UMCES) Baltimore, Maryland lmoren@umbc.edu, bachvarofft@umces.edu

Abstract

“Under the Bay” is an augmented reality project where anyone can use their cell phone like a microscope and reveal invisibilities in our world and marine life. When they do a series of animated stories between humans and non-humans emerge. Images, sounds, and stories are affected by live data streamed in from sensors located in the largest estuary in North America. Sensors in the Chesapeake Bay relay live pH, oxygen, temperature, etc. (figure 9). Similar to the water itself, color, speed, audio fluctuate with the water and marine life, making “Under the Bay” a data-driven narrative with eight scenes that tell a story of a world beneath the marine surface, and the exciting but frail health of estuaries and oceans worldwide. The two projects discussed here, “Under the Bay” (2022) and “What is the Shape of Water?” (2020), are part of Lisa Moren’s series of cross-species artworks aimed at diminishing human-centered exceptionalism. The collaborations began in 2019 when Lisa was the inaugural Artist-in-Resident at the Institute for Marine and Environmental Technology (IMET). There, she met researcher and marine biologist, Dr. Tsvetan Bachvaroff (Tsetso) and the two immediately shared a like-minded vision to develop a project that exemplified phenomenal exceptionalisms in micro-organisms. In this paper we argue that novel strategies in nature emerge when a complexity of matter is intermingled with conditions of differentiation. We explain and identify differentiation in art and architecture, symbiosis in biology, and the “wobble” in physics as core principles for new forms and creative strategies to emerge. The outcome is focused on the unusual and significant diversification of dinoflagellate microbes in the Chesapeake Bay and oceans worldwide. Tsetso directed the live organisms, science and data analysis for the augmented reality project. Stories are written and told by Lisa, who produced and art directed the animation and AR scenes. The sound score is by electronic composer Dan Deacon. Dr. Marc Olano led the software engineering and development with John Boutsikas, for the AR app in IOS and Google Play.

Keywords

Augmented Reality, AR, bio-art, data-driven narrative, emergent strategies, Tao Te Ching, water, symbiosis, dinoflagellates, Chesapeake Bay, estuary, marine biology, performance, microbes, media art, podcast, experimental narrative, bio-architecture, Lynn Margulis, Ted Nelson, Jane Bennett, Theodor Schwenk, data-driven music, philosophy.

DOI

10.69564/ISEA2023-19-short-Moren-et-al-Emerging-Strategies

Figure 1. “Under the Bay” augmented reality (still) by Lisa Moren with Dr. Tsvetan Bachvaroff. Scene 03 \\ Chalky Faeries, 2022. Image shows a cropped Bay Nettle jellyfish with coccolithophores (the microbes responsible for all the chalk in the world). The app allows the user to click and drag on the microbe to draw in chalk. Courtesy of the artist. 

Introduction

Figure 2. “Under the Bay” AR by Moren/Bachvaroff. Left: demonstration of AR in Fells Point, Baltimore; right: Beta testing of AR on Pier V, Baltimore’s harbor. Courtesy of the artist.

In our collaborative work, we borrow the idea of “emergent strategies” to consider strategies in nature that describe phenomena driving new emerging forms. Inspired by Octavia Butler, Adrian Maree Brown coined the term “Emergent Strategy,”¹ as “a framework for resistance that is rooted in the miracles of nature, decentralized, collective leadership, and personal, relational, organizational, and movement-wide transformation.”² We focused our strategies based in part on Brown’s idea that anomalous strategies in nature can be a model for the benefit of human communities, from species longevity to social change. Lisa’s second influence was her participation in a Taoist meditation group where she studied and meditated on the Tao Te Ching.³ These meditations blended with her observations of nature, primarily starling bird murmurations described in Scene 05 \\ Lava Lamps in the Sky. The connections between emergent strategies, organic differentiation, and a Taoist social order became the basis of this art and science collaboration. 

3. “Under the Bay” AR by Moren/Bachvaroff. Left: Still from splash screen with user interface (UI); right: still from AR Scene 02 \\ Water Moving Around My Fingers at Pier V, Baltimore. Courtesy of the Artist.

Emergent Strategies

We find the origin of strategic emergence by looking at gene evolution in early microbes. The Darwinian idea of natural selection has been combined with genome inheritance from two parents into what is now called the "modern synthesis." While this synthesis has excellent explanatory power, some observations are not a perfect fit to this synthesis. For example, photosynthesis is scattered across multiple kingdoms, but the process of photosynthesis itself is unlikely to have been invented multiple times. The solution to this paradox required two steps (figure 4). First, Lynn Margulis in 1967 proposed that eukaryotes are a combination of two cells from different bacterial lineages. The first lineage is the archea bacteria which engulfed another bacteria cell that contributes the mitochondrion. Mitochondria are responsible for the exchange of oxygen and carbon dioxide, eventually this will be the foundation for breath itself. The mitochondrion is then a "cell within a cell."⁴ This symbiotic combination of two previously distinct cells into eukaryotes created a new complexity of features (figure 4). The first scene in our project Scene 01 \\ Origin Stories, describes that before Margulis, any scientific reference to a symbiosis hypothesis was taboo. To accept symbiosis, science had to imagine a scenario where a species swallows a giraffe’s eye to acquire longer eyelashes successfully, but then, they pass those eyelashes onto their offspring. Symbiosis was once a fairy tale.

But once the formation of eukaryotes via symbiosis of two lineages was accepted, then the symbiosis concept was extended to include photosynthesis in plants and algae. In this scenario, a previously free-living cyanobacterium was engulfed by a eukaryotic host (figure 5). The host now contains both mitochondria and photosynthetic chloroplasts. This new host obtains energy from sunlight via the chloroplast and exploits that energy in the mitochondria. This innovation ultimately leads to land plants and nearly all photosynthetic eukaryotes (figure 4).

Sarah Gibbs extended the idea of symbiosis to explain why photosynthesis is scattered across multiple kingdoms. In this third scenario, one cell with chloroplasts is consumed⁵ by a second, but only the chloroplasts are retained. This provides access to photosynthesis and explains how photosynthesis could have spread across many diverse eukaryotic groups.

Therefore, an individual can be shaped by two patterns of evolution: the familiar and dominant expression of parental genes via vertical descent and a second, less familiar, pattern of horizontal gene transfer from endosymbiosis (figure 4).

The dinoflagellates are from the eukaryote kingdom and a perfect case study to demonstrate both symbiosis and the global importance of small eukaryotes to our planet. In global estimates of oceanic photosynthesis dinoflagellates are one of the top oxygen producers. 

Figure 4. Left to right: Darwin’s original descent with modification figure from the ‘Origin of Species’; Phylogenetic schematic of symbiosis, Moren/Bachvaroff, 2023; Lynn Margulis, https://en.wikipedia.org/wiki/Lynn_Margulis#/media/File:Lynn_ Margulis.jpg; Sarah Gibbs, Gibbs, S Annu. Rev. Plant Biol. 2006. 57:1–17. 

The oceans produce oxygen at a scale similar to the contributions from the Amazon rainforest. This is underscored by modern sequencing which places dinoflagellates as second in abundance to animals and first in diversity in oceanic surveys. Dinoflagellates also include a wide array of life strategies such as built-in harpoons used to capture prey.⁶ Other strategies include a symbiotic photosynthetic lifestyle as the algae found in coral reefs use internal and external types of parasitism, and consuming food for energy. The term mixotrophy describes the role of many photosynthetic dinoflagellates, which can be both photosynthetic (autotrophic) and consume prey (heterotrophic).⁷

Figure 5. Left: Gene Flow Symbiosis chart by Tsvetan Bachvaroff; right: “Under the Bay” AR by Moren/Bachvaroff. Still from Scene 01 \\ Origin Storiesshowing symbiosis. Courtesy of the scientist and artist. 

Mixotrophy and diversity in the ocean combined with the concept of symbiosis represented by Gibbs and Margulis provides many instances of emergent strategies. For example, the dinoflagellate Dinophysis feeds on ciliates demonstrating how the chloroplast could be consumed intact from prey by injecting a built-in “straw-like method” to acquire, or suck, chloroplasts from its prey.⁸ Roughly nine events have occurred where photosynthetic dinoflagellates contain chloroplasts borrowed or gained from essentially every available lineage. And this list does not include dinoflagellates with elaborate structural features that contain external symbiotes such as Ornithocercus

Figure 6. Three genera of dinophysoid dinoflagellates. Left to right: Phalachroma, Dinophysis SEM, Histioneis. Three genera contain intracellular photosynthetic organelles or symbionts, each distinct from the other. Courtesy of the scientist.
7. Left to right: Ornithocercus SEM, Ornithocercus with visible cyanobacteria; “Under the Bay” AR (still) by Moren/Bachvaroff. Courtesy of the scientist and the artist. 

Within the Dinophysis group there are different types of photosynthesis or symbioses, including the ‘gardening’ Histioneis (figure 6) and Ornithocercus (figure 7), the ‘heterotrophic’ Dinophysis that sucks out chloroplasts, and at least two other types of internal symbionts. Ornithorcercus will host cyanobacteria in a birdcage-like crown, a strategy that not only stores nutrients for on-going consumption, but the chains of cyanobacteria reproduce itself creating a “fruit-on-the- vine-like garden” energy supply (figure 7).

Symbiosis has allowed these the single-celled dinoflagellates to diversify over hundreds of millions of years into thousands of species that will clearly outlive humans. Therefore, diverse reproductive, energy consumption, and other strategies assisted the dinoflagellates in obtaining longevity that humans can only dream of. For example, the dinoflagellate Ceratium can both cell divide and mate for optimum reproduction benefits. In Scene 01 \\ Origin Stories the viewer will see images of the Ceratium that cell divide having only one trailing flagellum (a propeller-type tail). However, an identical Ceratium has two flagella because they had two parents that mated (figure 3).

Other strategies we named included organic differentiation in the structure of the cell walls of the dinoflagellates. These complex forms, such as Voronoi patterns, use less matter to produce structures that are lighter in weight than any objects human engineering could produce based on Cartesian principles in manufacturing. Differentiation can be exemplified in the exoskeleton of a crab or lobster but is most visible in the repeating hexagon pattern of a turtle shell, where the hexagon shapes repeat, but not perfectly. It’s in that imperfection, crookedness, or wobble, that creates more strength with less matter. Dinoflagellates, diatoms, and other microbes also have this efficient design principle of differentiation. Scene 04 \\ Crooked Shelters (figure 8) describes how such differentiation influenced the largest 21st century algorithmically designed and digitally fabricated architectural form in Seville, Spain, the Metropol Parasol.⁹ The pavillion’s aerial view appears as a mushroom blooming throughout the gridded city. A similar architectural example is in Stuttgart, Germany, where one of the algorithmic pavilions based on differentiation in nature was so strong and lightweight, it blew away.¹⁰

Figure 8. “Under the Bay” AR by Moren/Bachvaroff. Left: Work in progress; right: still from Scene 04 \\ Crooked Shelters. Courtesy of the artist.

Water Projects

In our first attempt to create a public project for an open house at IMET, we created an artwork demonstrating the microbial strategy of making light through the bioluminescence of the dinoflagellate (Pyrocystis). Eventually, the public display was set up as a ceiling tank hooked up to a Max/MSP, Arduino, and AV system for the Light City Festival in Baltimore’s Inner Harbor. The audience entered a pitch-black room with a large ceiling tank holding millions of invisible dinoflagellates. The system worked with a voice-activated trigger so that when a participant spoke into a microphone, for example, “What is the Shape of Water?” the microscopic organisms above them answered the question in turbulent shapes of blue bioluminescence (figure 9). Originally, this was influenced by the mesmerizing organic order and differentiation in the murmuration patterns of starling birds.¹¹ However, Tsetso’s colleague, Dr. Al Place who studies the motion behavior of dinoflagellates, says that the flocking behavior of the dinoflagellates will unlikely look as organized as the starlings. Instead, the water agitation produced turbulent patterns more akin to the wobble, the crookedness or what philosopher Jane Bennett calls murmuring messiness.¹² 

Figure 9. “What is the Shape of Water?” Left: Lisa Moren pouring biomatter in the ceiling tank; right: Photograph, Moren/ Bachvaroff. 16x16”, 2019. Courtesy of the artist.

“What is the Shape of Water?” began as a somewhat expedient strategy of displaying bioluminescence in an art project that took over a year to fully realize. It was an unrealistic approach to create art installations for the more than two dozen strategies we identified for potential exhibition. We instead turned to storytelling and produced an augmented reality (AR) project with eight scenes, approximately 10 minutes per scene. Lisa took the strategies we identified for the project and built narratives that meandered through purposefully fragmented topics related to microbes, water, Taoist meditation, architecture, monuments, and politics.

The AR project became “Under the Bay” and it’s trailer¹³ illustrates the animation, narrator, and incoming data. For the data, we worked with the Maryland Department of Natural Resources (MD DNR) to siphon data from sensors already installed in the largest estuary in North America, the Chesapeake Bay. In total 36 parameters stream into the project from six locations in the Bay. Locations are from the Delaware border to Washington, DC, the Eastern Shore of Maryland, and Baltimore City (figure 10). The parameters pH, temperature, oxygen, salt, chlorophyll, and turbidity (clarity) are updated every 15 minutes. These parameters then affect the animation’s color, speed, and scale, along with the narration and music composition. For instance, when the oxygen levels in the Bay are of good quality, the narrator’s voice sounds normal. However, when the water is anoxic, with little or no oxygen, the voice becomes choppy, fragmented, as if choking. In this way, the story, images, and sounds change from day to night and season to season for an ongoing collaborative narrative with the Bay water. The unpredictable variability of the incoming data becomes an authentic wobble created by the environment.

Figure 10. “Under the Bay” AR Moren/Bachvaroff. UI showing locations in the Chesapeake Bay where sensors allow water parameters to stream into the project. Courtesy of the artist.

Listening to the Water

The project propaganda claimed the ambition of cross- species communication specifically “What if we could hear what the water is saying?” To address skeptics to these claims, we’re borrowing from water physicist Theodor Schwenk, as we do in the AR project, but also Jane Bennett¹⁴, who references Graham Harmon, Bruno Latour, and Michel Serres’s The Birth of Physics.¹⁵

If we look at water as an object, we can argue that water is the largest object in the world. On the one hand, the ocean contains essential elements, H2O, saline, and other matter on the periodic chart. We know that water and gravity work together to form currents like pipes that braid in distinct patterns and that these flowing pipes separate into differing physical data such as speed and temperature. These differences become visible when encountering an obstacle, like a rock in a river, where we observe differentiation in the water shapes bulging around the rock. Similarly, in the ocean, large and “long waves travel faster than short waves” and overlap until the larger ones envelop the smaller ones, and the waves repeat the pattern endlessly.¹⁶ But unlike early CGI waves, the real ocean’s patterns are not perfect or predictable waves, their equilibrium billows, and exhales asymmetrically, and it’s in that asymmetry that they wobble. Or, what Bennett calls, they produce a vibratory noise, where the force of repetition starts to create a surge, an “irregular bombardment of circumstances,” especially when new physical elements act as an obstacle such as a rock. Here, the current billows and exposes its diverse temperatures, causing what Serres names a “cauldron of turbulence that thickens into lumps of phenomena, and the bubbling swirl keeps those shapes upright... while the wobble produces variances of noise.”¹⁷ A shape derives from the swirl that is both form and vibratory. To Serres, he calls this vibratory noise “the fluctuating ado that is the strange substance of any discrete, differentiated shape... (where) the multiplicity of the possible rustles in the midst of the forms that emerge from it... It is restless matter... (a) percolation.”¹⁸ While much of this noise dies like seedlings that don’t spawn, the intermingling currents and swirls, overlap with enough force allowing forms to emerge from it.

While we do not think of water as having agency, free will, or decision-making abilities or even that its elements are alive, in complex natural events, water does cause a multitude of events to happen, and is therefore an actant. One of the most significant events that water enacts is sustaining and creating new life, but also emerging shapes, and unique forms. In this way, water will begin by initiating an abundance of events, such as a vortex or a whirlpool effect. Any time these elements react or affect one another, there is the potential for something to emerge, including new life (figure 11). 

Figure 11. Left: Vortex funnel drawing by Theodor Schwenk,“Sensitive Chaos: The Creation of Flowing Forms in Water and Air”1967, pp 44; center and right: “Under the Bay”AR by Moren/Bachvaroff. Courtesy of the artist.

In Scene 02 \\ Water Moving Around My Fingers, the microbes reflect Schwenks organisms that take on the negative space of the vortex shape traveling in a reverse corkscrew spiraling up (figure 11).¹⁹ Therefore, when we say we can hear what the water is saying? It’s not that the water, as pure physical compounds, produces a will or agency that desires to be heard. However, water’s natural environment has reactions, interactions, and relationships with other phenomena and other physical materials where unique consequences emerge as shapes, forms, beings, and blooms. Moreover, if we consider the incoming parameters of the Bay water as a kind of alphabet — to use a human metaphor — the data does arrange itself to describe a story of the Bay water’s emerging behaviors and forms. This communication is so hard for humans to understand, especially to predict — that when the data reflects the formation of an algae bloom, it’s often too late to hear the water telling us we are out of balance. Perhaps the data acts like a Google knowledge engine, an algorithm anticipating the user’s thoughts when typing a partial phrase into the search bar and displaying our presumed burning questions. However, in this case, the data anticipates the thoughts of the Bay water. This concept is so significant that the scores of MD DNR sensors we use to siphon data are funded solely to predict the emergence of algae blooms in the Chesapeake Bay. “Under the Bay” observes how the Bay ebbs and flows over time to create an emerging narrative driven by what the water is enacting as an emergence. We call this murmuration messiness, listening to what the water is saying. 

Conclusion

Once we name this process of emergence, we can apply it to the other scenes in the AR project, addressing issues of differentiation and emergence not only in nature but also in creative ideas, forms, art, architecture, monuments, and also politics, protests (we discuss emerging protests including BLM, the beheaded Columbus statue ending up in the Bay, or the #metoo flocking of women in a bloom of pink hats — figure 12). This brings Brown’s term emerging strategies full circle where strategies in nature can be a model for social change. Again, our stories meander through these topics and phrases, like symbiotic waves engulfing one another leaning on influences from the Tao Te Ching as much as science. 

Figure 12. “Under the Bay” AR by Moren/Bachvaroff. Left: Scene 08 \\ Turnover [I Can’t Breathe] (still); right: Scene 07 \\ Vaccine Blooms with Pink Hats (still with UI). Courtesy of the artist.

“Under the Bay” will be exhibited at the Peale Museum, in Baltimore, Dec 14 to Feb 1, 2024, for “Chamber of Wonders”, with a public panel on Jan 18. While “Under the Bay” is freely available to download,²¹ the XR installation will include iPads interacting with paintings, drawings, and assemblages for a unique museum experience. “What is the Shape of Water?” and related works will also be on display.

“Under the Bay” Scenes 01-08 include: Origin Stories \\ Water Moving Around My Fingers \\ Chalky Faeries \\ Crooked Shelters \\ Lava Lamps in the Sky \\ Instrumental \\ Vaccine Blooms with Pink Hats \\ Turnover [I Can’t Breathe] 

Left: “Under the Bay” AR by Moren/Bachvaroff, 2022: Apple IOS; Google Play; right: “Under the Bay” (podcast), 2022: Spotify, Google Podcasts and Apple Podcasts. 
References

1 Adrienne Maree Brown, Emergent Strategy: Shaping Change, Changing Worlds, Edinburgh, AK Press, 2017.

2 Andrea Ritchie, Invisible No More: Police Violence Against Black Women and Women of Color, Beacon Press, Reprint edition, 2017. 

3 Lao-Tzu, translated by Stephen Addiss, Stanley Lombardo, introduction by Burton Watson, “Tao Te Ching”, Boulder, Hackert Publishing Company, 1993. 

4 Lynn Margulis, “On the origin of mitosing cells," Journal of Theoretical Biology, vol. 14, no. 3, 1967, 225–274.

5 Sarah P. Gibbs, “The chloroplasts of euglena may have evolved from symbiotic green algae.” Canadian Journal of Botany, vol. 56, no. 22, 1978, 2883–2889. 

6 Polikrokos harpooning prey https://www.youtube.com/watch?v=BddN1LE1YZQ 

7 Photosynthetic Fragilidium consuming a large Ceratium https://www.youtube.com/watch?time_continue=1&v=KFFsnH VlHyo&embeds_referring_euri=https%3A%2F%2Fmail.google.com%2F&embeds_referring_origin=https%3A%2F%2Fmail.g oogle.com&source_ve_path=Mjg2NjY&feature=emb_logo 

8 Dinophysis sucking the chloroplast from its prey https://www.youtube.com/watch?v=cq3SBoCHPP4&t=3s 

9 https://www.setasdesevilla.com/

10 https://www.itke.uni-stuttgart.de/research/icd-itke-research-pavilions/

11 https://www.youtube.com/watch?v=LAQwEWqg0ug\ 

12 Jane Bennett, William Connolly, “The Crumpled Handkerchief.” Time and History in Deleuze and Serres, Bloomsbury, London, UK, 2013, p.155.

13 “Under the Bay” trailer by Lisa Moren with Tsvetan Bachvaroff, 2022. https://vimeo.com/796868197 

14 Ibid, Bennett, Connolly.

15 Michael Serres, The Birth of Physics, Rowman & Littlefield International, Ltd., 2018. 

16 Theodor Schwenk, “Sensitive Chaos: The Creation of Flowing Forms in Water and Air,” Sussex, Rudolf Steiner Press, 1996, 33.

17 Ibid, Bennett, Connolly. 157.

18 Ibid. 

19 Ibid, Schwenk, p.44.

20 An eight-episode podcast version of “Under the Bay” is available on Spotify, Google, and Apple podcasts.
Links to podcasts and more information can be found on the project website: http://lisamoren.com/underthebay. 

21 Download the AR from Apple IOS or Google Play: https://apps.apple.com/app/id1641553491

https://play.google.com/store/apps/details?id=com.lisamoren.un derthebay&pli=1 

Bibliography

Jane Bennett, William Connolly, “The Crumpled Handkerchief,” Time and History in Deleuze and Serres, Bloomsbury, London, UK, 2013.

Adrienne Maree Brown, Emergent Strategy: Shaping Change, Changing Worlds, Edinburgh: AK Press, 2017.

Stefanie R. Fishel, The Microbial State: Global Thriving and the Body Politic, University of Minnesota Press, 2017.

Sarah P. Gibbs, “The chloroplasts of euglena may have evolved from symbiotic green algae,” Canadian Journal of Botany, vol. 56, no. 22, 1978, 2883–2889.

Douglas Kahn, “Energies in the Arts,” Cambridge and London, MIT Press, 2019, 2–46.

Lao-Tzu, translated by Stephen Addiss, Stanley Lombardo; introduction by Burton Watson, “Tao Te Ching”, Boulder: Hackert Publishing Company, 1993.

Lynn Sagan, “On the origin of mitosing cells,” Journal of Theoretical Biology, vol. 14, no. 3, 1967, 225–274, https://doi.org/10.1016/0022-5193(67)90079-3.

Theodor Schwenk, “Sensitive Chaos: The Creation of Flowing Forms in Water and Air," Sussex, Rudolf Steiner Press, 1996.

Lars Spuybroek, Sympathy of Things: Ruskin and the Ecology of Design, Bloomsbury Academic, 2016. 

Acknowledgements

This project is generously supported by the Saul Zaentz Innovation Fund at Johns Hopkins University and the R.W. Deutsch Foundation. This project originated during an artist-in-residence supported by the Institute of Marine and Environmental Technology (IMET), the Center for Innovation, Research, and Creativity in the Arts at the University of Maryland, Baltimore County (UMBC). We’re grateful for the support from Maryland Department of Natural Resources; John Boutsikas for his API and AR development, Danielle McPhatter and Harvestworks DigitalMedia Art Center, the IRC and CAHSS, and Antoine Cayrol of Atlas V Immersive Experiences in AR, XR, VR.

Production Team

Lisa Moren, Co-Principal Investigator, Producer, Art Director, Writer and Narrator; Dr Tsvetan Bachvaroff, Co-Principal Investigator, Marine Biologist, Researcher and Data Anaylsis; Dan Deacon, Electronic Composer; Dr. Marc Olano, Co-Principal Investigator, Lead Programmer; John Boutsikas, Programmer and Developer; Austin Samson Modeler and Animator; William Forrest, Modeler, Animator, UI and Technical Artist; Woody Lissauer, Voiceover Engineer and Male Narrator 1; Ruskin Nohe-Moren, Male Narrator 2; Aliyah Baruchin Copy Editor and Fact Checker.

Author’s Biography

Lisa Moren is a multi-disciplinary artist who works with emerging media, bio-matter, public space, AR and works-on-paper. She has exhibited her work at the Chelsea Art Museum, Creative Time, Drawing Center (New York), Cranbrook Art Museum (Michigan) and Ars Electronica (Austria), Akademie der Kunste (Germany), uShaka Museum (South Africa), and the Artists Research Network (Australia). She received the National Endowment for the Arts award, is a Fulbright Scholar; a multi-year recipient of the Maryland State Arts Council and CEC Artslink International, is a R.W. Deutsche Award recipient and a Saul Zaentz Innovation Fellow in Film and Media at Johns Hopkins University. Her writing has appeared in Performance Research; Visible Language; Inter Arts Actuel; New Media Caucus for “Algorithmic Pollution: Artists working with Dataveillance and Societies of Control” and “CYBER IN|SECURITY;” and her books on “Intermedia;” and Issues in Contemporary Theory for “Command Z: Artists Working with Phenomena and Technology.” Lisa Moren is a Professor of Visual Art at the University of Maryland Baltimore County (UMBC); is an Affiliate Faculty at the Imaging Research Center (IRC) UMBC; and taught at FAMU and AVU in Prague; and the University of California at San Diego (UCSD). 

Dr. Tsvetan Bachvaroff’s research is focused on dinoflagellate evolution with special emphasis on the parasitic dinoflagellates, using large-scale sequencing and phylogenetic methods to describe the evolutionary history of different types of genes in dinoflagellates. He uses DNA sequence analysis from data collection, assembly, annotation and phylogeny; has received numerous academic awards, including the William Trager Award from the Journal of Eukaryotic Microbiology, International Society of Protistologists, culture independent methods such as single cell PCR, sequencing, and sequence analysis; Establishing dinoflagellate cultures. He received the Marsho award, Mid-Atlantic Section of the American Society for Plant Biology, University of Maryland College Park, and the Chemistry Prize, Trinity School, New York. Tsvetan Bachvaroff received his B.A. degree from Johns Hopkins University and Ph.D. from the University of Maryland College Park. He was a Postdoctoral Fellow at the Center of Marine Biotechnology and subsequently with the Smithsonian Institution. He is an Associate Research Professor for the Institute of Marine and Environmental Technology (IMET) at The University System of Maryland. 

Power and Resistance in Digital Degrowth

Raul Nieves, Joan Soler-Adillon, Enric Mor

Universitat Oberta de Catalunya (UOC) Barcelona, Spain
dbsus4@uoc.edu, jsoleradillon@uoc.edu, emor@uoc.edu

Abstract

Degrowth is increasingly gaining attention as an alternative model to the unfolding eco-social crisis generated by industrial capitalism, though questions concerning digital technologies have yet to be addressed in degrowth research. Among the movement of the (digital) commons, whose practices complement degrowth theory, one of the research areas is the viability of such systems to release spaces from capitalism. As (digital) commons spaces frequently revert to capitalist logic, we introduce the "technological dramas" model to encompass the reciprocal and recursive technological production of political power by agonistic entities. We suggest that such a techno-political perspective could contribute to better frame degrowth-related HCI research.

Keywords

HCI, design, critical, political-economy, degrowth, commons, reification, false consciousness, co-optation, technological dramas.

DOI

10.69564/ISEA2023-20-short-Nieves-et-al-Digital-Degrowth

Post-growth

Planetary chances to restore a sustainable scenario are quickly running low. The UN questions whether we have already passed the turning point.¹ The unsustainable resource throughput promoted by the industrial model is (eventually if not yet) incompatible with the materially limited planetary boundaries.² Over the last decades, scholars and practitioners have started to question perpetual growth, a pivotal paradigm in industrial capitalism. Resource depletion, claims Turiel, precludes the default problem-solving conveyed through technological innovation, development, and deployment.³

The Degrowth project entwines a set of proposals to prioritize democratic, social and ecological justice in pursuit of well-being over economic growth.⁴ Such an economic, political, and social program conflicts with the capitalist political economy: sustainable growth, for degrowth advocates, is an oxymoron.⁵ Degrowth provides a political-economy frame to rethink human-computer interaction (HCI) theoretical tenets and practices as key elements to reconfigure the relationship between digital technological activity and eco-social effects.

Political-economy in HCI

The research field of HCI brings together technological and human-related disciplines to improve the interplay between users and machines.⁶ Despite HCI’s, deeply entrenched, industrial root logic, examining and questioning this very logic, especially recently, has connected many authors in HCI sub-communities.⁷ Among them: reflexive HCI⁸, humanistic HCI⁹, or critical HCI¹⁰. Dourish examines the early Sustainable HCI research production to expose the risks of naturalizing capitalist assumptions and suggests instead broadening the theoretical approaches in use. The author advocates “dismantle design as an anti-politics machine.”¹¹ Hence, Ekbia and Nardi urge to incorporate political economy in the analysis of HCI’s design and practices.¹²

Problematic capitalist core mechanisms, especially productivism and consumerism, take specific forms in HCI- related practices. Ekbia and Nardi quote Marx “production creates the consumer” relating consumerism to the paradoxical production of the “user.”¹² This “designed user” is central to the anthropocentric formulation of problems. Problems, again paradoxically, are produced to fit feasible technological solutions, feeding the mechanism known as techno-solutionism.¹³ This back-feeding entanglement is key for the industrial production system. It underpins perpetual growth while consuming resources and expelling negative externalities, producing new problems.

These analyses within HCI resonate with the degrowth discourse. Despite some proposals (i.e., see ¹⁴ as post-anthropocentrism, or ¹⁵ as post-techno-solutionism) we suggest it’s necessary to conduct research on how to consistently translate this critique into impact-aware HCI practice. Thus, we turn towards degrowth-aligned practices to foreground strategic political economy issues arising from the struggle to transition outside capitalism.

Challenges in (digital) commoning practices

According to Helfrich and Bollier degrowth and the commons movement are complementary to each other.¹⁶ The commons don’t rely on economic growth and make compatible environmental and social justice. These authors define the commons in terms of politics and economics as “a vast array of self-provisioning and governance systems that flourish mainly outside of both the market and the State.” 

While degrowth frames the subject of critique, the commons exhibit social, political, and economic forms of actualization based on the social practices of commoning: the stewardship practices that a community employs to manage shared things (virtually anything) in common.

Digitally enabled communities of practice, spawned by the emergence of the internet, have long been at the center of commons research. Peer-to-peer (P2P) networking architectures fostered a productive model labeled as commons-based peer production by Benkler.¹⁷ According to P2P advocate Bauwens, this model represents a generative alternative in front of the extractivist modes of capitalist production.¹⁸ Fuster defines the digital commons as online communities which share non-exclusive co-created digital resources.¹⁶

One of the main research concerns in the commons is the relationship between the state, capitalism, and alternative spaces of resistance. The cartography of such borders seeks to discern the strategic practices to release and gain back spaces from capitalist control. The commons aim at releasing spaces from capitalism and confronting the enclosing of old and new commons. Such spaces risk falling back to capitalist logic, a process referred to as co-optation. While also discussed as “colonization by capital”¹⁹, “assimilation”²⁰, “incorporation”²¹, “transvestment”²², or “unwanted corporate appropriation”²³, among other terms, co-optation is ubiquitous in the commons theory but it is scarcely being examined in depth.

Among the few commons theory authors that specifically examine co-optation, we find De Angelis, Caffentzis, and Federici. According to De Angelis, capitalism is about to face a social and ecological crisis and will likely have to leverage or promote the commons to help manage the devastation. ²⁴ De Angelis claims: “struggles [...] can be absorbed and become part of the system (co-opted), thus renewing it and sustaining it.” As the logic of the market becomes counterproductive even from the viewpoint of capital accumulation, precluding the cooperation necessary for an efficient system of production, Caffentzis and Federici point to "the danger that 'commons' may be co-opted to provide low-cost forms of reproduction."²⁵

Kostakis et al. review and discuss recent criticisms of peer production, classifying some of them as co-optation. The study examines whether digital peer production could be emancipatory or instead become part of capitalism. On one hand, the injection of funds in free open-source software (FOSS) projects and the multi-million-dollar purchases of FOSS companies increase the risks of appropriation of the commons by corporate interests. On the other hand, commons’ pro bono production is monetized and exploited by market agents for profit extraction.²⁶

To inquire into the origins of digital co-optation, we focus here on the ideas of Osseewaarde et al.²⁷ The authors analyze how digital commoners recurrently transition through alternative spaces, as they eventually get co-opted. Commoning’s essence opposes the technological rationality of formalization, standardization, and quantification, yet the emergence of such spaces relies on technological innovation which is fostered by the growth-oriented efficiency ethos promoted by neo-liberalism. This contradiction, the authors argue, results in a perpetuating illusion, a form of false consciousness, which is rooted in cynicism.

Ossewarde et al. claim that current standard technologies “are highly opaque because they are often implicit and part of a formalized design for digital interaction that is in itself an arrangement of ‘false consciousness’.”²⁷ Hence, digital commoning (the generation of contents, but especially of infrastructures) fails to resist co-optation, supporting capitalist expansion as De Angelis, Cafentzis, and Federici denote.

While Ekbia et al. point to reification (the assumption of systemic concepts that are actually socially constructed as inescapable and natural, like the capitalist market) as a hindrance to emancipatory HCI²⁸, Dourish and Ossewaarde et al. refer to the studies on the reified construction of false consciousness by Lukacs. Ossewaarde et al. argue that such cynicism cannot be overcome via ideology critique, but through technology critique when it "is translated into post-capitalist acts of resistance to the dominating technology design." De Angelis asks “Isn’t this co-evolution between struggle and capital development really inherently with no end?”²⁹

Introducing a technological power construction model in HCI

Reified notions embed the HCI practices with false consciousness, rendering alternatives prone to get co-opted by capitalism. To further comprehend this dynamic, we suggest introducing “technological dramas,” a framework by the anthropologist of technology Bryan Pfaffenberger.³⁰ As technological activity presents an opportunity to embed political values, Pfaffenberger examines how power and resistance are constructed through the reciprocal and recursive shaping of artifacts and values in the design process, which later spread in society.

Pfaffenberger stresses the relevance of myth, ritual, and context in the understanding of the political dimensions of technological activity. Myths are deployed to suspend skepticism, rituals are associated with controlled environments produced to pattern human actions, and social contexts are fabricated in parallel. In this model, technological activity is analyzed as a process of technological communication: “a technological drama is a discourse of technological statements and counter-statements.

The model describes three processes that can occur linearly, or under different permutations, in the construction of politics by technological means:

● Regularization occurs when a design entity (usually part of the establishment) “creates, appropriates, or modifies a technological production process, artifact, user activity, or system in such a way that some of its technical features embody a political aim”. 

● Adjustment takes shape when impact entities engage in control and alteration strategies. These strategies attempt to counter the discursively regulated social contexts that regularization creates, pursuing to counter their effects. This process can lead to technological appropriation

● Reconstitution materializes when impact entities “try to reverse the implications of a technology through a symbolic inversion process,” labeled as antisignification by Pfaffenberger. This process can produce, as in some forms of adjustment, appropriation. It can also result in the fabrication of counterartifacts, “which embody features believed to negate or reverse the political implications of the dominant system.

Pfaffenberger employs the term reintegration to refer to co-optation processes: “the response made by the agents of regularization to the new, problematic counterartifacts. Its goal is to gain control over these artifacts by bringing them back into the controlled and ordered space of regularization.” According to the author, some forms of adjustment and reconstitution stages (resistance processes with relevant degrees of technical intervention) are prone to co-optation. While co-optation has been previously discussed, it is considered by this model in a wide and complex techno-political dynamic.

Besides the three above-mentioned processes, Pfaffenberguer introduces a fourth: designification takes place when the link between technological activity and social meaning-producing discourse dims. “The artifacts, their contexts, and our social behaviors remain; they become taken for granted, routine, and part of the natural attitude of everyday life.” According to Pfaffenberger, this is the stage where technological activity achieves the greatest social penetration. We suggest a connection between designification and the previously discussed process of reification, here particularly referred to in techno-political, rather than political economy, terms.

Conclusions

In front of collapse, the political and economic project of degrowth offers an alternative coexistence formula. The HCI community is already approaching the degrowth frame in order to redefine its tenets and practices, and political economy analysis has already been adopted by HCI authors to examine reified notions. To expand HCI research we have examined digital commoning challenges to release spaces from capitalist logic. Reification, pointed out already by HCI authors, false consciousness, and co-optation processes are discussed by commons’ authors as strategic issues concerning the transition to post-growth alternative political economies.

We suggest the relevance of this model in analyzing and prospecting the construction of political power, in its different stages, in order to deploy strategic practices of transition in front of a ravaging capitalism. Especially due to the systemic view this model offers to dissect and relate processes like designification, co-optation, and possibly others, as stages of the permanent struggle through technological activity. We also suggest leveraging this model as a tool to complement speculative, adversarial, fictional, strategic, or transitional design techniques.³¹-³⁵ Hence, we expect this contribution to help HCI researchers better frame degrowth-related practices and research contributions.

This article is part of the R+D+i project PID2021-128875NA-I00, funded by MCIN/AEI/10.13039/501100011033/ "ERDF A way of making Europe”.

References

1 Matt McGrath, "Climate change: IPCC report warns of ‘irreversible’ impacts of global warming", Retrieved December 7, 2022, from https://www.bbc.com/news/science-environment-60525591.

2 Nicholas Georgescu-Roegen, The Entropy Law and the Economic Process, Cambridge, MA, Harvard University Press, 1971.

3 Antonio Turiel, Petrocalipsis, Alfabeto, 2020.

4 Matthias Schmelzer, Aaron Vansintjan, Andrea Vetter, The Future Is Degrowth: A Guide to a World beyond Capitalism, Verso Books, 2022.

5 Jason Hickel, Giorgos Kallis, "Is Green Growth Possible?", New Political Economy 25 (4), 2020, 469–486.

6 Paul Dourish, Where the Action Is: The Foundations of Embodied Interaction, Cambridge, MA, USA, MIT Press, 2001.

7 Raul Nieves, Enric Mor, Joan Soler-Adillon, "Decoupling design from the industrial paradigm," In Proceedings of ISEA2022, the 27th International Symposium on Electronic Art, Barcelona.

8 Phoebe Sengers, John McCarthy, Paul Dourish, "Reflective HCI: articulating an agenda for critical practice," In CHI ’06 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’06), 1683–1686.

9 Jeffrey Bardzell, Shaowen Bardzell, Humanistic HCI, San Rafael, California: Morgan & Claypool, 2015.

10 Jeffrey Bardzell, Shaowen Bardzell, Mark Blythe (eds.). Critical Theory and Interaction Design, Cambridge, MA, USA, MIT Press, 2018.

11 Paul Dourish, "HCI and environmental sustainability: the politics of design and the design of politics," In Proceedings of the 8th ACM Conference on Designing Interactive Systems (DIS ’10), 1–10.

12 Hamid Ekbia, Bonnie Nardi. "Social Inequality and HCI: The View from Political Economy," In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), 4997–5002.

13 Evgeny Morozov, To Save Everything, Click Here: The Folly of Technological Solutionism, New York, Public Affairs, 2014.

14 Bill Tomlinson, Bonnie Nardi, Daniel Stokols, Ankita Raturi, "Ecosystemas: Representing Ecosystem Impacts in Design," In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 1–10.

15 Eric P.S. Baumer, M. Six Silberman, "When the implication is not to design (technology)," In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11), 2271–2274.

16 Giacomo D’Alisa, Federico Demaria, Giorgos Kallis, Degrowth: A Vocabulary for a New Era, Routledge, 2014.

17 Yochai Benkler, The Wealth of Networks: How Social Production Transforms Markets and Freedom, New Haven, Conn., Yale University Press, 2007.

18 Michel Bauwens, Vasilis Kostakis, Alex Pazaitis, Peer to Peer: The Commons Manifesto, University of Westminster Press, 2019.

19 Christian Fuchs, "The Digital Commons and the Digital Public Sphere: How to Advance Digital Democracy Today," Westminster Papers in Communication and Culture 16 (1), 2021.

20 Maurizio Teli, Marcus Foth, Mariacristina Sciannamblo, Irina Anastasiu, Peter Lyle, "Tales of Institutioning and Commoning: Participatory Design Processes with a Strategic and Tactical Perspective," In Proceedings of the 16th Participatory Design Conference 2020 - Participation(s) Otherwise - Volume 1 (PDC ’20), 159–171.

21 Benjamin Birkinbine, Incorporating the Digital Commons: Corporate Involvement in Free and Open Source Software, University of Westminster Press, 2020.

22 Michel Bauwens, Alekos Pantazis, "The ecosystem of commons-based peer production and its transformative dynamics," The Sociological Review 66 (2), 2018, 302–319.

23 Benjamin J. Birkinbine, "Conflict in the Commons: Towards a Political Economy of Corporate Involvement in Free and Open Source Software," The Political Economy of Communication 2 (2), 2015.

24 Massimo De Angelis, "Crises, Capital and Co-optation: Does Capital Need a Commons Fix? | The Wealth of the Commons," Retrieved May 9, 2022, from http://wealthofthecommons.org/essay/crises-capital-and-cooptation-does-capital-need-commons-fix.

25 George Caffentzis, Silvia Federici, "Commons against and beyond capitalism," Community Development Journal 49 (suppl_1), 2014.

26 Vasilis Kostakis, Vangelis Vragoteris, Indra Lal Acharja, "Can peer production democratize technology and society? A critical review of the critiques." Futures 131, 2021, 102760.

27 Marinus Ossewaarde, Wessel Reijers, "The illusion of the digital commons: ‘False consciousness’ in online alternative economies," Organization 24 (5), 2017, 609–628.

28 Hamid Ekbia, Bonnie Nardi, "The political economy of computing: the elephant in the HCI room," Interactions 22 (6), 2015, 46–49.

29 Massimo De Angelis, "Crises, Movements and Commons," Borderlands 11, 2012.

30 Bryan Pfaffenberger, "Technological Dramas," Science, Technology, & Human Values 17 (3), 1992, 282–312.

31 James Pierce, "In Tension with Progression: Grasping the Frictional Tendencies of Speculative, Critical, and other Alternative Designs." In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21), 1–19.

32 Carl DiSalvo, "Adversarial Design as Inquiry and Practice," In Adversarial Design, MIT Press, 2012, 115–125.

33 Matt Malpass, Critical Design in Context: History, Theory, and Practice, London, New York, 2017.

34 Anna Meroni, "Strategic design: where are we now? Reflection around the foundations of a recent discipline," Strategic Design Research Journal 1 (1), 2008, 31–28.

35 Terry Irwin, "Transition Design: A Proposal for a New Area of Design Practice, Study, and Research," Design and Culture 7 (2), 2015, 229–246

Motherplants: Mycelium Network and Artistic Research

Julien Ottavi, Jenny Pickett

Association APO33, Madlab - Cyprus University of Technology Nantes, France - Limassol, Cyprus
julien@apo33.org, jennypickett@apo33.org

Abstract

In 2015, we embarked on an artistic research project involving organic organisms such as plants and fungi, focusing on their role in the processes of recycling of e-waste. Motherplant explores how these systems create feedback audio signal transmission through the interplay of moisture and decomposition of the circuit boards. Specifically, it harnesses the natural decomposition process to transmit audio signals by converting the soil's acidity and electronic compost into electrical signals.

Keywords

Artistic research project, electronic compost, mycelium networks, soil acidity, symbiosis, mutation of technologies, e-waste.

DOI

10.69564/ISEA2023-21-short-Ottavi-et-al-Motherplants

Introduction

The surge in global electronic waste (e-waste) volumes, as reported by the Global E-waste Statistics Partnership (GESP), is cited by the World Health Organization (WHO). Between 2014 and 2019, there was a substantial increase of 21% in e-waste generation, resulting in a staggering production of 53.6 million metric tonnes of e-waste during the latter year. "For perspective, [in the year 2020] e-waste weighed as much as 350 cruise ships placed end to end to form a line 125 km long. This growth is projected to continue as the use of computers, mobile phones and other electronics continues to expand, alongside their rapid obsolescence." (Johnson, 2021) The repercussions of this escalating e-waste problem are dire, encompassing adverse effects on both human health and ecosystems. The e-waste stream comprises a multitude of toxic substances, ranging from neurotoxins to heavy metals, which pose significant risks (Jain et al., 2023). These hazardous constituents have the potential to inflict severe harm upon individuals and the environment, underscoring the urgency of addressing and mitigating the detrimental consequences associated with mounting e-waste volumes.

Figure 1: Motherplants-installation view - 2018. ©APO33-Nantes

This project revolves around the intriguing intersections of mycelium networks, audio diffusion, and the creation of novel computing experiences. Motherplant engages with the notions of symbiosis and mutation of existing technologies to explore new modes of interaction with recycling, art research and implementation of new network models based on mycelium communication cooperation systems.

Motherplant : Computational Mycelium Recycling Network

Using dead and discarded motherboards extracted from obsolete computer systems, we attempted a transformative process with the aim of repurposing these printed circuit boards (PCBs) into "micro-farm" environments. The primary objective of Motherplant encompassed three key facets: firstly, to experiment with sustainable recycling of all motherboard components; secondly, to harness their inherent properties for the generation of electrical current; and lastly, to contribute to the development of an alternative open system designed to facilitate the exchange of computational data. This fusion of electronic and organic realms introduces a harmonious synergy, wherein the motherboard's original purpose converges with the natural processes of growth, decomposition, and nutrient exchange.

Figure 2: Motherplants-schematics 2023. ©APO33-Nantes

Re-creation of the fungal network as an electronic mutation

In Motherplant we are trying to create a novel electronic circuitry that mutates in tandem with a primitive fungal network. It is important to clarify that our fungal network is a simplified "primitive" representation, distinct from the extensive rhizomatic networks deployed by mature fungi across vast expanses of soil, connecting plants and trees. We acknowledge that nature excels in this regard, and our intention is not to replicate nature's proficiency but rather to cultivate and orchestrate a symbiotic relationship between electronic circuitry in a localised fungal network.

This begins with a few spores settling down on a nutrient-rich surface. When these spores wake up in close proximity to one another; they initiate germination at approximately the same time, giving rise to thread-like cellular structures known as hyphae, which extend outward at comparable rates. This unique fusion of electronics, powered by solar energy and moisture-rich soil infused with (DIY) mycelium growth, establishes an intricate information superhighway. As an information superhighway the interactions between a large, diverse population of individuals speeds up. It allows entities who may be separated to communicate and help each other out. It also allows them to commit new forms of communication.

This symbiotic relationship between electronic circuitry and a fungal network not only bears relevance to the creation of a new information exchange paradigm but also holds significant potential in the context of e-waste recycling. Fungi possess the ability to break down and transform a wide range of organic and inorganic substances, pesticides, hydrocarbons and heavy metals, making them valuable agents for cleaning up contaminated environments (Tomer et al., 2021). By virtue of fungal bioremediation, the fungal network may play a pivotal role in the sustainable degradation and recycling of electronic components.

Schematics for Motherplant

This iteration of Motherplant comprises a redundant, nonfunctional motherboard as its foundational structure, upon which mycelium is cultivated within a composite medium consisting of straw, soil, and integrated circuitry. The degradation of the motherboard’s components initiates an acidification process within the medium, subsequently generating electrical signals. These electrical signals function akin to a battery, delivering an approximate output of 1.5 volts, which in turn powers an audio circuit. The frequencies generated by this audio circuit exhibit variability contingent upon the dynamic activity of the mycelium and the pH levels of the surrounding soil.

Renaturing the Motherboard

The Motherplant concept can be traced back to an earlier experiment initiated at APO33 in Nantes 2015 and presented during NEAR #1 as part of Nantes Digital Week. During this meeting, the growing need to address e-waste issues was discussed alongside concerns about the exploitation of workers, recycling components without protective clothing or in adequate conditions (Ottavi and Pickett, 2015). These first experiments went on to be presented at the Nomad Village during COP21 in Paris, later on that year (Ottavi, 2015). Electronic waste is toxic and hazardous to human health, PCB's are hard to recycle and demand a lot of energy and sometimes the scramble for precious metals and minerals contained therein, sacrifices members of our human community (Wittsiepe et al., 2015; Beaumont, 2019). Usually, this heavy cost is hidden from our sleek conscientious recycling certificates. Yet it could take decades, sometimes centuries, before some of these components will be totally recycled. In 2019 Europe ranked first worldwide in terms of e-waste generation per capita, with 16.2 kg per capita (Forti et al., 2020).

With Motherplant by employing plants, flowers, and fungal organisms as instrumental agents in the recycling of electronic waste, we set about composting the motherboard. The heart of any home computer or laptop, the Motherboard is a concealed symbol of this cyclical consumption and our relentless obsessions for digital speed. With Motherplant we sought to counteract this hidden world of consumer electronics recycling and make visible the complexity of such a task if left to nature, as well as the sheer disregard for the problem by the shipping of our waste elsewhere. The idea of Motherplant was to produce electricity to feed electronic circuits, so that, in turn, they endlessly sustain a sound installation. This undertaking represented a confluence of art, ecology, recycling practices and the exploration of alternative energy models. The shape and layout of a motherboard is called the form factor. The form factor affects where individual components go and the shape of the computer's case. We design Motherplant in such a way that the classical function of the motherboard decomposes to produce a new kind of computer. One that won’t consume electricity anymore, but instead will become a receptacle for plants and spores to grow.

Mycelium computation: a new paradigm in bioart and sound art research

This new kind of Mycelium computation is about to begin, as we search for quantum processors, we are looking for Mycelium based hijacked computer circuits. The spore computation is not just another form of computer research in the tradition of engineering production, it's an all-new vision about computers. In 2018, Professor Andrew Adamatzky, Director of the Unconventional Computing Laboratory at the University of the West of England in Bristol, UK, wrote in his article "Towards a fungal computer", how a series of scoping experiments, provided evidence that electrical activity recorded on fruits could potentially serve as a dependable indicator of the fungi's reaction to thermal and chemical stimulation. (Adamatzky, 2018) It diverges from the prevailing landscape of technological industrial mass production, presenting a concept characterised by fragility, inherent hazards, and a multitude of uncharted pathways, and prototypes of computing mycelium bound composites. (Roberts and Adamatzky, 2022).

A mélange of Bioart (art using fungal bioremediation and technology) with sound art (sound as a profound production of vibration) serves as a conduit for the cultivation of exchanges and ideas within a networked context. Within the domain of bioart, the convergence of innovative gardening techniques and the exploration of mycelium-based experiences culminates in the facilitation of multifaceted networks characterised by a wide array of functionalities that will serve multiple purposes such as recycling, production of energy, circulation of electricity but also production of sound waves, feedback transmission and autonomous art installation.

From the perspective of sound-ecology, listening to the world holds significant importance in numerous practices. However, in the case of Motherplant, it transcends mere observation of nature; it involves a collaboration with nature to generate a sound installation on a minute scale, characterised by low impedance and microsounds, all within the dimensions of the sculpture itself. In terms of sound levels, our focus aligns more closely with the realm of ants and insects than that of elephants or storms. This reveals the potential inherent in the amalgamation of nature, mycelium, and soil within our artistic patch.

Art frequently detaches itself from its own reality, often transported and displayed within galleries or dedicated venues, creating a sense of isolation. In contrast, Motherplant seeks to recreate a natural context intricately linked to the artwork itself, ultimately transforming the context, nature, environment, and aesthetics into the art itself. Sound plays an integral role in the production process but whilst it doesn't represent the final objective, it is a vital component within the broader framework. The various iterations of the art piece over the last 8 years, also contribute to shaping the sculptural essence of Motherplant. The process that we describe here and the relation between nature, technologies and production of energy is not separated from the poetic facets of our artistic research.

Recycling plastic and other dirty electronics to produce low currency (power)

Beyond the artistic sculpture of the Motherplant, we had assigned to this project two other important research perspectives. Whilst these are not the main goals, they resonate with the other aspects (art, sculpture, sound, visuals...). The first enables us to think about and experiment with the recycling of electronic wastes with the Motherboard. In this context, the concept of the "mother" assumes a central and symbolic role, representing the origin from which life emanates. This archetype of the mother extends beyond the human realm to encompass broader notions such as Mother Earth and Mother Plant, all of which evoke contemplation of the nurturing and life-giving aspects associated with motherhood. By integrating the concept of the "mother" into the project's narrative, we establish a profound connection between technology, nature, and the cycle of life. Which underscores the intricate interplay between electronic devices and the environment. How can we care for electronic waste management and recycling? Motherplant embraces maternal sustainability and environmental awareness, towards recycling e-waste and engendering a new computation network. Secondly, recycling e-waste costs are very high compared to disposing them in landfills, mainly because it often contains hazardous materials that need to be dismantled and processed properly. Also, the lack of recycling infrastructure poses problems to handle the large volume of e-waste generated each year. Finally, due to the lack of awareness from the public, most people have no idea of the importance of recycling their computers and e-waste. This lack of awareness ends up in huge amounts of illegal dumping or in the wrong landfills and incinerators producing more pollution but also exposing people to hazardous materials.

We decided, as artists working in electronic arts and media, to engage in this protracted undertaking of the project Motherplant, and develop our own way of experimenting with living networks. Rather than adhering to the conventional and often inefficient approach of classical digital art research, which typically involves a relentless cycle of producing more work and tirelessly pursuing galleries and festivals for exhibitions, we have chosen a different path. We consider this process in the broader context of "decroissance" or degrowth, lowering our consumption, decelerating our expectations of results, and embracing the temporality associated with decomposition. The philosophy of degrowth posits that the relentless pursuit of economic growth is inherently unsustainable in the long term. It contends that, in order to safeguard the environment and enhance societal well-being, it is imperative to curtail economic activities and their associated impacts.

The other part of this research centres on the production of electricity. While it's important to note that this production is characterised by low power output, it has nonetheless proven to be a sustainable source of electrical current over the past few years, requiring no additional input other than the periodic watering of the Motherplant. This enduring supply of electricity has so far facilitated a continuous eight-year run of our sound installation.

The motherboard, serving as the fundamental underpinning of our installation, adheres to a decompositional timeframe, thereby providing a robust scaffold for the continuous provision of electrical current. This enduring electrical supply, in a reciprocal fashion, sustains the seamless operation of our sound installation, ensuring its uninterrupted functionality. This approach to both energy generation and artistic expression accentuates the intricate interplay amongst technology, ecology, and the dimension of time.

Figure 3: Motherplant : Close-up 2018. ©APO33-Nantes

Documentation

View the 2015 video on Climate Solutions: MOTHERPLANT here: https://www.youtube.com/watch?v=CSUeSdR6n1Q. View the 2018 video on Motherplants here: https://vimeo.com/301201120

References
Bibliograpy

Andrew Adamatzky, “Towards Fungal Computer.” Interface Focus 8 (6), 2018, 20180029, https://doi.org/10.1098/rsfs.2018.0029.

Peter Beaumont, “Rotten Eggs: E-Waste from Europe Poisons Ghana’s Food Chain.” The Guardian, April 24, 2019. https://www.theguardian.com/global-development/2019/apr/24/rotten-chicken-eggs-e-waste-from-europe-poisons-ghana-food-chain-agbogbloshie-accra.

Vanessa Forti, Peter Baldé, Ruediger Kuehr, Garam Bel, S Adrian, M Drisse, Y Cheng, et al. 2020, “Quantities, Flows, and the Circular Economy Potential the Global E-Waste Monitor, 2020”, https://www.itu.int/en/ITU-D/Environment/Documents/Toolbox/GEM_2020_def.pdf.

Jain, Muskan, Depak Kumar, Jyoti Chaudhary, Sudesh Kumar, Sheetal Sharma, Ajay Singh Verma, “Review on E-Waste Management and Its Impact on the Environment and Society”, Waste Management Bulletin 1 (3), 2023, 34–44, https://doi.org/10.1016/j.wmb.2023.06.004.

Johnson, Ceridwen, “Soaring E-Waste Affects the Health of Millions of Children, WHO Warns”, World Health Organization, World Health Organization, 2021, https://www.who.int/news/item/15-06-2021-soaring-e-waste-affects-the-health-of-millions-of-children-who-warns.

Adam Minter, “The Burning Truth behind an E-Waste Dump in Africa”, Smithsonian, Smithsonian.com, January 13, 2016, https://www.smithsonianmag.com/science-nature/burning-truth-behind-e-waste-dump-africa-180957597/.

Julien Ottavi, “Climate Solutions: MOTHERPLANT”, www.youtube.com, Solution Zone TV, December 6, 2015, https://www.youtube.com/watch?v=CSUeSdR6n1Q.

Julien Ottavi, and Jenny Pickett, “Near-1 [[NEAR] Nantes Electronic Art Rencontre]”, Apo33.org, September 22, 2015, https://apo33.org/near/doku.php?id=near-1.

Julien Ottavi, Jenny Pickett, 2018, “Motherplants - Jenny Pickett & Julien Ottavi”, vimeo.com, 2018, https://vimeo.com/301201120.

Roberts Nic, Andrew Adamatzky, “Mining Logical Circuits in Fungi”, Scientific Reports 12 (1), 2022, https://doi.org/10.1038/s41598-022-20080-3.

Ajay Tomer, Ramji Singh, Saurabh Kumar Singh, S. A. Dwivedi, Chilkuri Udaykiran Reddy, Malavika Ram Amanthra Keloth, and Riya Rachel, “Role of Fungi in Bioremediation and Environmental Sustainability”, Fungal Biology, 2021, 187–200, https://doi.org/10.1007/978-3-030-54422-5_8.

Jürgen Wittsiepe, N. Fobil Julius, Till Holger, Gerd-Dieter Burchard, Michael Wilhelm, and Torsten Feldt, “Levels of Polychlorinated Dibenzo-p-Dioxins, Dibenzofurans (PCDD/Fs) and Biphenyls (PCBs) in Blood of Informal E-Waste Recycling Workers from Agbogbloshie, Ghana, and Controls.” Environment International 79, June 2015, 65–73, https://doi.org/10.1016/j.envint.2015.03.008.

Authors Biographies

Julien Ottavi, holds a Doctor of Arts from the Université de Lorraine. His practice covers sound, intermedia, and digital network arts. He is the founding member and artistic director of association APO33, based in Nantes. APO33 has produced numerous ground-breaking explorations in experimental music and electronic arts, and collaborated with artists and musicians across the world.

Jenny Pickett is a member of the APO33, where she works on projects ranging from interactive installations to experimental music and performance to international collaboration in art and technology. She is currently a PhD Candidate at Cyprus University of Technology where she is attached to the Media Arts and Design Research Lab (MADlab), she is also the Vice Chair for Toolkit of Care a EU COST funded cooperation, as well as an associate lecturer at the Nantes School of Architecture (ENSA). 

Life-as-it-could-be, Symbiosis in Interspecifics’ Codex Virtualis_Genesis

Claudia Costa Pederson

Wichita State University Kansas, U.S.A. ccp9@cornell.edu

Abstract

This paper discusses Codex Virtualis_Genesis (2020-2022) as an artistic engagement of Artificial Life (AL) that explores the nature of informational life as a symbiotic process. Created by Mexico City-based transnational collective Interspecifics (INT), this work follows on from the expanded notions of life central to the field of artificial life, including organic, inorganic, material, and virtual forms. According to its founder, Christopher Langton, “there is nothing... that restricts biology to carbon-based life; it is simply the only kind of life that has been available to study.” From this perspective, Langton proposed in the late 1990s that AL be dedicated, as he put it, to speculating beyond "life-as-we-know-it" into the realm of "life-as-it-could-be.” This discussion examines Codex Virtualis_Genesis in light of Langton’s proposal as a speculative inquiry into a symbiotic view of life, and as well in contrast to notions of artificial life art as a predominantly technophilic practice. Instead connected to the speculative imagination, the synthetic life forms of Codex Virtualis_Genesis offer a glimpse into life otherwise: as an interspecific relation.

Keywords

Artificial Life, Artificial Life Art, Mexico, Symbiosis, Symbiocene.

DOI

10.69564/ISEA2023-22-short-Costa-Pederson-Life-as-it-could-be

Codex Virtualis_Genesis

Interspecifics’ Codex Virtualis_Genesis (2021-2022) was created for the SETI x AI Lab residency, an art and science program started in 2018 by the Ars Electronica Futurelab in Linz, Austria in collaboration with the AI Lab (European ARTificial Intelligence Lab) and the SETI Institute in Mountain View, CA.¹ The program calls on international artists working in digital art to create works “that explore the evolution of life on earth and universe and critically reflect on anthropocentric world views.”²

Interspecifics won the 2021 Ars’ AI Lab residency call with their proposal to create Codex Virtualis_Genesis as an evolving taxonomic collection of hybrid AL forms.³ In practice, to create these forms’ shape, color, texture, morphological structures and behavioral dynamics, Interspecifics used a 2D cellular automata system (a planar cell-based computer graphic system that displays emergent global behaviors), and two sets of images of living symbiotic organisms. One set is microscope images (microscopy) of microorganisms living in water bodies that were posted by various scientists on their Instagram accounts.⁴ Another set was digital photographs of colonies of fungi and bacteria (8,000 distinct species) made available online by researchers at the National Agriculture and Food Research Organization, NARO, in Japan. Some of the latter organisms are also known as extremophiles because they are metabolically and biochemically adaptable to harsh environmental conditions such as broad swaths of temperature, pH, pressure, radiation, salinity, energy, and nutrient limitation. They have therefore been historically used in astrobiology experiments, including most recently conducted at the ISS (International Space Station).(1)

The resulting images resemble plant and animal-like microorganisms in a wide range of colors and shapes (Figure 1a and 1b). In a gallery setting, Codex Virtualis_Genesis is shown as a screen-based installation that as well as displaying selected images of said organisms, also displays images of aspects of the selection process and evolutionary procedures (Figure 2).

1a and 1b.Symbionts generated for CodexVirtualis_Genesis, Ars Electronica2021. Courtesy of Interspecifics.
1b. Symbiontsgenerated for CodexVirtualis_Genesis, Ars Electronica2021. Courtesy of Interspecifics.
2. CodexVirtualis_Genesis, Ars Electronica2021. Photo credit: Robert Bauernhansl.

Additionally, in the printed materials provided to visitors at exhibitions, Interspecifics describes Codex Virtualis_Genesis as titled in a loose nod to Mesoamerican codices and based on “the symbiotic narrative of evolution.”⁵ Specifically this refers to endosymbiosis, a process of symbiotic life evolution proposed by the biologist Lynn Margulis in the late 1960s. Endosymbiosis foregrounds symbiotic mergers (pre-nucleic gene transfer between bacteria and other organisms) as the main sources of acquisition of new genomes and thus also of speciation (i.e., the formation of new and distinct species in the course of evolution). From this perspective, random mutation is of marginal consequence as the driver of evolution and so consequently Margulis argued that Darwinian evolutionary theory based solely on competition (i.e., the survival of the fittest) is incomplete. According to Margulis, “Life did not take over the globe by combat, but by networking. Life forms multiplied and complexified by co-opting others, not just by killing them.”⁶

Interspecifics’ Codex Virtualis_Genesis brings Langton and Margulis in dialogue with an eye to complement their respective perspectives on the nature of life. In this interest, it extends Margulis’ restricted notion of life (as carbon-based) to as well, as proposed by Langton, include synthetic forms. Conversely, by linking AL and symbiosis it also clarifies Langton’s speculative vision of AL (i.e., as “life-as-it-could-be”). (2) According to a member of Interspecifics, Leslie Garcia, this dialogue speaks to the project’s overarching concept: namely to simultaneously call attention to our “entanglements” and to stimulate speculative imagings and imaginaries of “non-anthropocentric and non-anthropomorphic but more organicist, metabolic models of life.”⁷

Cooperate!

Codex Virtualis_Genesis not only demonstrates life as a mutualistic relation on screen but is itself created with an eye for developing art and science as mutualistic practices: in Interspecifics’ own words, “to produce in terms of social inclusion, cross-disciplinary practices, and open knowledge.”⁸ To this concern, the project contributes to broadening access to artificial life techniques and consonantly increasing diversity in artificial life art as an area of the global digital arts. As aforementioned, Interspecifics drew on freely available images shared by scientists and scientific communities online and social media to create Codex Virtualis_Genesis’ symbiont organisms. Additionally, Lenia, a readily available system of cellular automata or artificial life, is integral to the project. Created by the Chinese software engineer Bert Wang-Chak Chan in 2015, this system is a variation on The Game of Life (GoL) devised by the British mathematician John Horton Conway in 1970, which was originally played on a simple square grid without the aid of computers. GoL is a class of mathematical phenomena termed cellular automata by John von Neumann in his discussion of self-reproducing machines in the mid-1940s.(3) Both GoL and Lenia are AL systems that focus on life not as a material substrate but as a form of organization and behavior that is akin to the distributed processes characteristic of the functioning of living organisms. Alternatively, these systems can also be conceived as no-player games; meaning that after the initial configuration their evolution does not require further input. The point of cellular automata like GoL and Lenia is in short to demonstrate the emergence of the variety and complexity of behaviors from a few simple rules. These systems differ insofar the sophistication of their graphics. In comparison to GoL’s square graphics, which now look outdated, Lenia’s fuzzy, smooth, and colorful patterns look state of the art. More significantly for artists broadly, as art funding becomes ever more restricted, and in particular artists that as Interspecifics puts it, work via-á-vis the context of “precarity” in the global South, Lenia is as an open-source art and science project. Still requiring some technological expertise, it nevertheless is in theory accessible to a global public.

Already, in an updated version of his 1989 article on Artificial Life, Langton speculated that the future evolution of AL would depend on scientists’ imitating nature’s cooperative behaviors (so he urged biologists and computer technicians to follow its example). (4)Artificial life art, as A-life artist Simon Penny notes, is similarly an area of digital arts with a high degree of interdisciplinarians and/or collaborative partnerships between artists and technicians because artificial life techniques are technically demanding.(5) Because of the emphasis on technical expertise and state-of-the-art technology in the field (what Penny calls “high nerd quotient” and the art historian María Fernández calls digital art’s “aesthetic technofetishism”), artificial life art has been dominated by artists working in the global North.(5)(6) Artists like Interspecifics are at the forefront of developing artificial life art as an emerging area of the digital arts in Mexico along with a new generation of artists in Latin America, including the Argentinian artists Leo Nuñez and Sofia Crespo and the Peruvian artist Paola Torres Nuñez del Prado. Projects like Linea are significant because they broaden access to AL, both beyond the sciences and the global North.¹⁰ It is to this concern that Interspecifics, just as with all their projects, additionally made some of the research, techniques, and coding created for Codex Virtualis_Genesis freely available online.¹¹

Codex Virtualis

Codex Virtualis_Genesis is yet but the first of the four parts of the total project proposed by Interspecifics under the title Codex Virtualis. The collective is currently developing the second part of this project, which is entitled Habitat involves testing the extremophile capabilities of the organisms created in Genesis by exposing them to inputs that simulate the harsh conditions of extraterrestrial environments. A third part, which is titled Emergence proposes testing the possibility of intra-actions (transfers) between artificial and biological organisms. Lastly, Codex Virtualis: Life involves testing the resilience of resulting hybrid organisms and accordingly selecting one organism to live virtually in Codex Virtualis.¹²

As a contemporary artificial life artwork, Codex Virtualis follows on a longstanding quest for mimesis or life-likeness that spans the histories of art, science, and technology, and as well includes social, spiritual, commercial, intellectual, and military histories. Langton himself traced the origins of AL to this pursuit in the history of art, starting with paintings and “statuettes” that “capture the static forms of living things.” He goes on to discuss hydraulic technology such as Egyptian water clocks or clepsydra as the next step towards imitating nature’s dynamic behavior. This trajectory developed with the invention of the mechanical escapement in AD 850, ushering in the age of clockwork technology. Refined throughout the European Middle Ages and the Renaissance, eventually, clockwork regulation of mechanical devices would not only be used to power them but also to sequence their motions or behaviors.¹³

Langton refers to this development as technology of process-control, which in the first instance involved interchangeable cams or drums with movable pegs. A variety of programmable automata are examples, including lifelike mechanical figures and animals such as the writing and picture-drawing automata built by the Jaquet-Droz family in Switzerland in the 18th century. In the early part of the twentieth century, during WWII and the Cold War, as Langton explains, physical programmable controllers gave away to abstract control structures, or sets of rules or programs. In effect, as he puts it, this is the moment when the “’logical form’ of the machine was separated from its material basis of construction.” Today’s computers are a technology of process-control and in this sense the equivalent to an “algorithm: the logic underlying the dynamics of an automaton.” (7)¹⁴

Artificial life art, which in the contemporary context is facilitated by computer technology, is consequently heir to these legacies.(8) According to Penny, artificial life art in digital environments spans three decades, beginning in the late 1980s, as artists turned to “a new type of interactivity” in search of a “new order of mimesis in which ‘nature’ as a generative system, not as an appearance, is being represented”. (5) To put it otherwise, artists became interested in artificial life in pursuit of developing art contrary to ingrained notions of artistic practice. That is, to create art not as an object but as a process, and in extension, to explore the role of the artist not as one about control over materials, but as an initiator of an open-ended process. This initial focus on unpredictability and novelty has since, however, been debated as an illusory pursuit, given that, as Penny adds, “the mechanism of simulated evolution” appear in comparison to the limitations of biological evolution more restricted; that is, as bound to the “conditions... established by logic-theoretic enframing” of mathematical models. (5)

Yet, while Codex Virtualis_Genesis and its larger project, Codex Virtualis shares interest in interactive aesthetics, this is not the project’s main focus. Its focus is rather apropos of the curator and cultural critic Edwina Bartlem’s noting as part of her recent discussion of artificial life art that “A-life discourse contains a prophecy of futuristic and imaginary posthuman or post-organic life.” (9) Codex Virtualis speaks above all to this imaginary, as according to Interspecifics, “’Codex Virtualis’ imagines ways of living together, of deep interspecific relations that may enhance our possibilities of survival.”¹⁵ Garcia further clarified the project’s concept on another occasion, as a proposal to imagine life and a new world beyond the Anthropocene, consonant to “the Symbiocene.”¹⁶

Garcia’s reference is to a term coined by the eco-anarchist philosopher, Glenn Albrecht, the Symbiocene. Symbiocene refutes the fatalistic implications of the Anthropocene, and, as largely inspired by Margulis’ notion of endosymbiosis affirms the possibility of creating a more sustainable world based on non-anthropocentric mutualism. (10) In this light then Codex Virtualis engages symbiosis as a metaphor and as a form of dialogue, debate, and exchange (i.e., between art, science, and politics) for conceptualizing life in a sustainable posthuman age. Because of its ecological focus, it as well in extension broadens the existing e imaginaries and themes engaged in artificial life art.

Notes

1 Interspecifics was founded in Mexico City in 2013. Core members are Leslie García, Paloma López Ramírez, Emmanuel Anguiano Hernández, Felipe Rebolledo Carvajal, Carels Tardío Pi, and Maro Pebo.

2 Ars Electronica is a cultural institute devoted to media art founded in Linz, Austria, in 1979. A leading organization in the global media arts, as well as managing a museum, multidisciplinary art research facilities, and an annual festival, it confers the most prestigious award in the genre, the Prix Ars Electronica. Founded in 1984, the SETI Institute stands for the "search for extraterrestrial intelligence." It is a non-profit dedicated to the study of life and intelligence in the universe through multidisciplinary research, education, and partnerships with industry, academia and government agencies, including NASA and NSF.

3 Available at, https://int-lab.cc/codex/?page_id=2.

4 According to Interspecifics, the initial idea was for collaborating (SETI) scientists to share their images with the group for the project. As it happened, the scientists failed to provide these images in the end. Leslie Garcia, “Interspecifics, Codex Virtualis: Genesis,” lecture, “Neurotalk ‘Máquinas y seres vivos: Comunicaciones desde el arte’”, Programa ACT (Arte, Ciencia y Tech- nologías), UNAM, March 2, 2022, available at, https://www.youtube.com/watch?v=NFE_F6HzakY.

5 Interspecifics, Codex Vitualis_Genesis, exhibition booklet, 2021, 4, available at, https://int-lab.cc/codex/?page_id=183.

6 Lynn Margulis and Dorion Sagan, Microcosmos: Four Billion Years of Microbial Evolution (Berkeley, CA: University of California Press, 1997), 29. Margulis’ speculations have been corroborated in the last decade as for example by scientists discovering that in ecosystems across the world, there are immense, mutually beneficial associations of macrofungi with flowering plants in complex, positive, metabolic, symbiotic relationship to each other. Findings such as these have scientifically overturned the view that evolution and life are solely founded on competitive struggle between species.

7 Leslie Garcia, “Interspecifics, Codex Virtualis: Genesis,” lecture, “Neurotalk ‘Máquinas y seres vivos: Comunicaciones desde el arte’.” It is worth to note here that Margulis’ notion of endosymbiosis implies a critique of heterosexist articulations of life, (i.e., the singularly privileged place of heterosexual reproduction in evolutionary biology), which presently resonates with Karen Barad’s notion of “nature’s queer performativity”, which likewise denotes multispecies co-involvements as the drivers of life’s heterogeneity. See, Karen Barad, “Nature’s Queer Performativity” in Kvinder, Køn og forskning / Women, Gender and Research nos. 1-2 (2012): 25-53.

8 See, http://interspecifics.cc/.

9 Artists working with digital A-life systems mostly reside in North America, Europe, and Australia, and include Karl Sims, Tom Ray, William Latham, Ken Rinaldo, Bill Vorn, Louise-Philippe Demers, Troy Innocent, Jon McCormack, Robb Lovell and John Mitchell, Christa Sommerer and Laurent Mignonneau, Jane Prophet and Gordon Selley, Paul Brown, Richard Brown and Mauro Annunzianto. See also, Fernández, “’Life-like’: Historicizing Process and Responsiveness in Digital Art”; Sarah Kember, Cyberfeminism and Artificial Life (London and New York: Routledge, 2003); “; Michael Whitelaw, Metacreation: Art and Artificial Life (Cambridge, MA: MIT Press, 2004); and Simon Penny, “Twenty Years of Artificial Life Art”, in Digital Creativity 21, no. 3, Dec. 2010: 197-204.

10 To date, along with scientists, game designers, filmmakers and artists across the world, including among them, Interspecifics, have used Lenia to develop their respective projects and in the process added to the evolving diversity and complexity of behaviours of its life forms. According to Chan, to date, more than 400 species in 18 families have been identified in Lenia. B. W.-C. Chan, “Lenia: Biology of Artificial Life,” Complex Systems, 28(3), 2019, pp. 251–286. See also, Siobhan Roberts, “The Lasting Lessons of John Conway’s Game of Life”, in The New York Times, Dec. 28, 2020, available at, https://www.ny- times.com/2020/12/28/science/math-conway- game-of-life.html.

11 See, https://int-lab.cc/codex/.

12 The duration, location, and conditions of this last stage are yet to be clearly specified as at the time of writing, Interspecifics is still developing the second stage in the project, Habitat.

13 The Muslim inventor Ismail al-Jazari’s The Book of Knowledge of Ingenious Mechanical Devices (1206) is currently considered by historians of science to be a significant groundwork for modern engineering, hydraulics, and even robotics. The Book builds on science and wisdom from ancient Greek, Indian, Persian, Chinese and other cultures and offers a how-to manual of sorts that includes alongside richly detailed illustrations as well as instructions on how to build al-Jazari’s inventions spanning playful robots to practical contraptions.

14 Likewise, the Latine artist Micha Cárdenas has traced algorithmic culture to pre-digital technology, citing recipes and rituals as examples. Additionally, Cárdenas reminds us that the word algorithm is a derivation of the name of the scholar Muhammad ibn Musa al-Khwarizmi (780-850 A.D.), credited with inventing algebra in his book Dixit Algorismus. Micha Cárdenas, Poetic Operations, Trans of Color Art in Digital Media (Durham, Duke University Press, 2022), 7.

15 Interspecifics, Codex Vitualis_Genesis, exhibition booklet.

16 Garcia, “Interspecifics, Codex Virtualis: Genesis,” lecture “Neurotalk ‘Máquinas y seres vivos: Comunicaciones desde el arte’”

References

(1) Cortesão See M., T. Schütze, R. Marx, R. Moeller, V. Meyer, “Fungal Biotechnology in Space: Why and How?” in H. Nevalinen, ed., Grand Challenges in Fungal Biotechnology. Grand Challenges in Biology and Biotechnology, New York, Springer, 2020, 501–535.

(2) Christopher Langton, “Artificial Life,” in Ars Electronica: Facing the Future, edited by Timothy Druckrey, with Ars Electronica, Cambridge, MA, MIT Press, 1999, 126-268.

(3) John von Neumann See, Theory of Self-Reproducing Automata, Urbana, University of Illinois Press, 1966.

(4) Christopher G. Langton, “Artificial Life,” in The Philosophy of Artificial Life, edited by Margaret A. Boden, London, UK, Oxford University Press, 1996, 92-93.

(5) Simon Penny, “Art and Artificial Life-A Primer,” in the Proceedings of DAC09, Digital Art and Culture 2009 conference, University of California, Irvine, 12–15 December 2009, unpaged, available at, https://escholarship.org/uc/item/1z07j77x.

(6) María Fernández, “Postcolonial Media Theory,” in Art Journal 58, no. 3, Autumn, 1999, 66-69.

(7) Langton, “Artificial Life,” 41-46.

(8) María Fernández See, “’Life-like’: Historicizing Process and Responsiveness in Digital Art,” in A Companion to Contemporary Art since 1945, edited by Amelia Jones, Malden, MA, Blackwell Publishing, 2006, 557-581.

(9) Edwina Bartlem, “Immersive Artificial Life (A-Life) Art,” in Backburning: Journal of Australian Studies 84, edited by Helen Addison-Smith, An Nguyen and Denise Tallis, Perth, API Network, 2005.

(10) See, Glenn Albrecht, Earth Emotions: New Words for a New World, Ithaca, NY, Cornell University Press, 2019.

Author Biography

Claudia Costa Pederson is Associate Professor of Art History, author of Gaming Utopia: Ludic Worlds in Art, Design, and Media (Indiana, 2021); and co-curator of FLEFF’s new media exhibitions at Ithaca College, NY, since 2017.

Disruptive Avant-Garde Art of Today: Shaping Post-Growth Imaginaries for Symbiotic Futures

Kristina Pranjić, Magdalena Germek, Peter Purg

University of Nova Gorica
Nova Gorica, Slovenia
kristina.pranjic@ung.si, magdalena.germek@gmail.com, peter.purg@ung.si

Abstract

In order to develop new symbiotic relationships and different imaginaries, it is first necessary to critically restructure the representations of forms of cooperation, which in their positive, desired version usually represent a certain romantic idea of nature and human, and the possibilities for a harmonious model and holistic structure of reality. This can be seen in both eco art and activist ecological agendas, which often play on feelings of harmony and mutual reciprocity, and actually further contribute to a distorted and extremely one-dimensional image of reality. Using the concepts of conviviality and cosmpolitics, the article aims to offer new concepts of symbiosis and symbiotic futures that face today's process of defuturing. The second point of the article is to develop a convincing and solid alternative to the neoliberal view of market-driven models based on competencies and the logic of growth. Therefore, the actual task for disruptive avant-garde art of today should be understood as the decolonization of our imaginaries that perceive nature through the logic of growth and the harmonious model in the direction of shaping post-growth imaginaries for symbiotic futures.

Keywords

Imaginary, conviviality, symbiosis, cosmpolitics, defuturing, (de)growth, posthumanities, intermedia art, artistic disruption.

DOI

10.69564/ISEA2023-23-short-Pranjic-et-al-Disruptive-Avant-Garde-Art

Introduction

At the end of the epidemic, faced with another European war, and shattered by climate disasters, we seem to be facing a similar if not even worse zeitgeist to the one of the twenties that called for the avant-garde art and design school Bauhaus to bring a new social, cultural and creative impetus. However, back then there was no experience of the failures of the 20th century artistic and social utopias and movements.

For this reason, we can rightly question any revolution or change through artistic and cultural means. Nevertheless, new artistic expressions that use new technology and scientific methods to conduct true artistic research are developing novel concepts and opening space for another horizon of experiences that can offer images and ideas of a different future, in which it will be possible not only to survive, but also to thrive in symbiosis with other human and non-human beings.

In order to describe the mentioned process through a deep understanding of how current art contributes to the shaping of our imaginaries for the better future in the age of the Anthropocene, the article will first introduce key concepts, such as “conviviality”, “symbiosis”, “defuturing”, “imaginary”, “growth”, and “post-growth/degrowth,” that comprehensively shape the paradigm, criticized by the artistic projects that are addressing the most burning issues of today, and rightfully represent the contemporary avant-garde. In the last part of the article, we will show three distinct examples of current artworks that seek to participate in transforming the human imagination in the direction of symbiotic futures with a different system of relationship not dependent of industrial and consumerist systems, rooted in the idea of continuous growth.

Living together: Conviviality and Symbiosis

The term conviviality was incorporated to the humanities vocabular by Ivan Illich, who in his work criticized the industrial capitalism and productivity, to instead describe a path of human emancipation and autonomy for reaching convivial life.¹ For Illich conviviality means an “autonomous and creative intercourse among persons, and the intercourse of persons with their environment; and this, in contrast, with the conditioned response of persons to the demands made upon them by others, and by a man-made environment.”² Furthermore, because of his instance of the interdependence among living beings, Ilich’s thought is particularly productive in the field of posthumanism to describe interdependence and entanglements between different species, human and non-human agents.

The term “conviviality” comes from Latin convivere, which means “to carouse together, live together” (com “with, together” and vivere “to live”). The more common term “symbiosis” that comes from ancient Greek sýn and bíōsis similarly denotes living together. Today both of these concepts are equally important in the posthumanism research that emphasizes the importance of exploring relations between human and non-human beings and other entities or matter, and insisting on the theoretical break with anthropocentrism and with the dualisms between human/society and nature.

Conviviality does not mean only relationships of cooperation, but it also includes conflict and tensions as constitutive and necessary part of convivial relations; human beings are understood as a part of the network of players living in symbiosis with others. ³ Moreover: “To Illich, the word ‘conviviality’ does not mean joy or light-heartedness; it refers to a society in which modern tools are used by everyone in an integrated and shared manner, without reliance on a body of specialists who control said instruments.”⁴ For Ilich convivial toll can be used by an individual for a purpose that they choose, and it needs to bring about more freedom, autonomy and creativity (for all beings).

Perceptions of harmonious interactions of symbiosis were critiqued by Isabelle Stengers’ research in thermodynamics to include complexity, processes of disturbance and friction that cause a state of crisis and lead to change.⁵ In her seminal work on “cosmopolitics” Stengers described a “symbiotic agreement” as a part of an “immanent process of ‘reciprocal capture’;” she describes “the event” of symbiosis as “the production of new, immanent modes of existence, and not the recognition of a more powerful interest before which divergent particular interests would have to bow down. Nor is it the consequence of a harmonization that would transcend the egoism of those interests.”⁶

We can further find productive thoughts on this topic in Anna Tsing work on rare matsutake mushroom, in which she presents the complex network of cooperation between human and non-human participants that rejects stability as a central goal of relationships, and even stability of the term “naturalness”.⁷ Stengers’ and Tsing’s work not only opens to question the foundations of objective (natural and humanities) science, but also shows a new way of connecting and generating knowledge and possible new imaginaries that can arise from different forms of human activity, which is not necessarily bound to the rationality and authority. Knowledge and science can be, through specific practices and processes, something that we not only discover, but also shape.

Facing Defuturing

To recognize what is, we first need to establish active knowledge about how our current dysfunctional relationships of un-sustainment are sustained. One could assume it is truly difficult to be an avant-gardist in the time of defuturing, as Tony Fry named our condition in his book Defuturing. A New Design Philosophy.⁸ Fry introduced the concept of defuturing as the destruction of the future by design; it means that we live in a world that is taking away futures for ourselves and non-human others. He criticizes Eurocentric, anthropocentric and productivist structures of unsustainability by exploring the history of design that so importantly yet often subliminally coordinates our activities, and influences our thought.

Because our conditions of existence drastically changed, Fry stresses that we also need to drastically change our philosophy of existence and of acting. The new direction for making anything in the world is not a direction within design as-is; within the already established paradigms; it is a direction beyond where design now is. It is also a direction beyond where thought now is. The alternative to this process of “change so that nothing changes” and covering of crisis, would be “informed futuring”, which basically refers to finding the alternative and acquiring the agency in order to create something truly different, an actual change.⁹

For this kind of futuring, a new avant-garde in doing and thinking is needed. Therefore, it is central to look into the past to deconstruct and surpass it. The most significant signifiers of how we do and think things, and how we ascribe value – is growth. Therefore, besides offering a complex image of symbiosis and establishing new and different relationships, it is important to decolonize our imaginary of growth.

Decolonizing our Imaginaries

We can understand “growth” as a centuries-old matrix of thinking that historically constructs Western-civilizational models of meaning-making, collectively and individually, from various institutional macro and microforms of society. The logic of growth is thus not only rooted in the rational categories that shape our concepts but also in the way we look at the world, how we describe, perceive, and represent it. The stubbornness to fight growth by reformulating capitalist parameters – for example, by establishing concepts like social, human, local, and sustainable “development” – can only lead to new ecological, economic, and social failures.

Agendas such as the New European Bauhaus for example, and the UN 2030 Agenda for Sustainable development do seem to be signs of a significant shift toward an awareness about social and environmental health, equality, and justice. However, we need to be critical of the very idea of sustainable development, which remains part of the Western economic imaginary of growth. We can find this critique especially in the works of Serge Latouche¹⁰, who claims that the idea of sustainable development is a mystified and ideologized one that does not bring about any radical turn in thinking but stems from the logic of a compromise.

Consequently, it is unavoidable to start “decolonizing our imaginaries,” a syntagma borrowed from Latouche¹¹, who refers, on the one hand, to the concept of the imaginary by Cornelius Castoriadis¹², and on the other hand, to the anti-imperialist concept of decolonization introduced by anthropologists (e.g., Serge Gruzinski).¹³ If we follow Latouche’s insight, radical change needs another economy; another view of science that would go beyond the concept of Promethean technoscience; another conception of life and death; a different conception of wealth and poverty; another notion of time that would no longer be linear, cumulative, continuous; other conceptions of space; other intergenerational and gender relations; a different concept of work (placing social relations in the center, instead of, e.g., efficiency or value accumulation)¹⁴. Latouche integrates this into an eight-point program with listed imperatives: to re-evaluate, reconceptualize, restructure, redistribute, relocalize, reduce, re-use, and recycle.¹⁵

Constructing the Imaginary of Post-Growth

One solid set of alternatives to the logics of growth was offered by the introduction of concepts such as post-growth and degrowth. And even though the concept of degrowth as the most optimal future direction of our global society is becoming more and more mainstream, it is clear we will not achieve any drastic changes in human behavior if we do not address its phenomenological and epistemological implications that could provide us with new models for everyday life.

Neutralizing the disastrous impact of our growth-based society would therefore mean actually using radical avant-garde methods as a unique combination of chance and plan, happening and controlled work in all aspects of thinking and creating for uncertain futures. We see the legacy of the historical avant-garde movements, such as Dadaism and Surrealism, as a plethora of valuable concepts for thinking degrowth that is yet to be researched and applied from the field of art exclusively to the area of social and environmental care, and precisely to the way we make and do things. The highly experimental avant-garde presented the most radical possibilities for changing the logic of thinking and perceiving reality, thus showing the opportunities to let new forms and relationships emerge.

And although the avant-garde is all too often perceived as art that sought to demolish and destroy everything that existed before it, it carries within it an extremely affirmative and constructive character: avant-gardes invented a whole series of new formal languages and strategies that are still in use today as fundamental techniques of our modern visual or multimodal communication (e.g., photomontage, collage, avant-garde cinema, advertisements, graphic design typography, new architecture). They sought to formalize the coincidence and freedom of expression and erase any automatization and pre-given cultural definitions.

The avant-gardists’ proposition appears to have been precisely what we need today for a symbiotic “metamorphosis in being” as stated by Pasi Haikkurinen, connected with the “practice of releasement” understood as the “new ethos” for degrowth society.¹⁶,¹⁷

Artistic Disruption for Radical Innovation

We finally come to the acknowledgment of the importance of artistic disruption and art-thinking as the key innovative methodology that may provoke radical change in order to create a meaningful difference from what was (wrong). Which are those new (inter)media and investigative artistic practices involving cutting-edge technologies, critically confronting the European social and cultural values with the aesthetics-, ethics- and tech-related legacies of the historical avant-garde?

By way of conclusion, let us try to briefly discuss three of such topical cases selected among the newest (intermedia) artistic practices:

Vladan Joler's recent hybrid animated info-graphic work entitled New Extractivism (2020) visualizes the position of the individual as a user of contemporary information technologies, who is subject to systemic and systematic corporate extraction. Thus, it reveals and critically discusses some of the most pressing issues of modern platform capitalism. The printed materials, which the author considers to be a kind of sci-artistic “assemblage”, are presented in the format of a poster and a brochure, and are accompanied by an animation distributed online, supported by the author's own narrative, which he occasionally also delivers as a lecture (performance). By combining the elements of a map and a guidebook, and accompanied by abundant and extremely analytical notes, the work attempts to produce a symbiotic and fractal-like “blueprint of a machine-like superstructure, or a super allegory.”¹⁸

Often relying on the approaches of speculative design and advanced computer graphics, as well as info-animations or even physical spatial installations, the works of the interdisciplinary group Disnovation.org not only vividly show the paradoxes of the contemporary consumer and hyper-informatized society, but also try to contribute suggestions and tools for resolving them. In the Post Growth Toolkit (2020), a set of so-called critical games, they offer a mix of scientific and speculative literary-narrative tools, which should offer the players of this game (or the users of the toolkit) a set of more sustainable solutions for everyday life. Post Growth Prototypes (2021) complement the mentioned toolbox with critical (animated video) essays functioning as case studies of advanced and symbiotic concepts such as solar income, radical energy transition, or transcending the Anthropocene via the post-growth paradigm shift, introducing a new responsibility of man towards the biosphere.

As the last truly comprehensive example of an artistic practice that radically questions the human role within ecosystems, we wish to discuss the set of terra0 projects. Since 2018 various prototype environments have been built on the decentralized (peer-to-peer) blockchain platform Ethereum, which aims to provide automated frameworks for the resilience of a given ecosystem. By establishing a “Decentralized Autonomous Organization” on the upper layers of the earth to govern them, terra0 research team aims to create technologically augmented ecosystems that are both more resilient and more capable of operating within a predetermined set of rules in the economic sphere, as independent agents: e.g., the forest independently mines cryptocurrency and decides how it will change its material base. The group believes that modern technologies such as remote sensing and machine learning provide an opportunity to rethink existing inefficient governance and regulatory structures. Moreover, they also seem to suggest how, with appropriate art-thinking and speculative-design based assumptions and interventions, these could play a key role in creating a sustainable, resilient, symbiotic and biodiverse future.

There are more such artistic practices and they bear witness to the engagement of art in order to answer the most pressing questions of our time not only by representing what does not work, but above all to construct, in an avant-garde manner, new symbiotic relationships with human and non- human agents that would contain the full complexity of the symbiotic organization of different entities jointly working to sustain a functional environment, without falling back into the old paradigms shaped by the imaginaries and logics of growth, duality, harmony and stability.

References

1 Ivan Ilich, Tools for Conviviality, New York, Harper and Row, 1973.

2 Ivan Ilich, Tools for Conviviality, 11.

3 Michael Given, “Conviviality and the Life of Soil,” Cambridge Archaeological Journal 28, 2017, 127–143.

4 Marco Deriu. “Conviviality,” in Degrowth – a new vocabulary for a new era, ed. Giacomo D’Alisa, Federico Demaria and Giorgos Kallis, London, Routledge, 2015.

5 Ilya Prigogine, Isabelle Stengers, Order Out of Chaos. Man's New Dialogue with Nature, London, Flamingo, 1984.

6 Isabelle Stengers, Cosmpolitics I, Minneapolis/London, University of Minneapolis Press, 2010, 35-36.

7 Anna L. Tsing, The Mushroom at the End of the World: On the Possibility of Life in Capitalist Ruins, Princeton/Oxford, Princeton University Press, 2015.

8 Tony Fry, Defuturing, A New Design Philosophy, 2nd Edition, Bloomsbury, Bloomsbury Publishing, 2020.

9 Tony Fry, Defuturing, A New Design Philosophy, 239.

10 Serge Latouche, Petit traité de la décroissance sereine, Paris, Librairie Arthème Fayard, 2007.

11 Serge Latouche, “Imaginary, Decolonization of,” in Degrowth – a new vocabulary for a new era, ed. Giacomo D’Alisa, Federico Demaria, Giorgos Kallis, London, Routledge, 2015.

12 Cornelius Castoriadis, L'institution imaginaire de la société, Paris, Éditions du Seuil, 1975.

13 Serge Gruzinski, La colonisation de l’imaginaire. Sociétés indigènes et occidentalisation dans le Mexique espagnol (XVIe- XVIIIe siècle) Collection Bibliothèque des Histoires, Paris, Éditions Gallimard, 1988.

14 Serge Latouche, Survivre au développement, Paris, Librairie Arthème Fayard, 2004.

15 Serge Latouche, Petit traité de la décroissance sereine.

16 Pasi Heikkurinen, “Degrowth: A metamorphosis in being,” Environment and Planning E: Nature and Space Vol. 2, issue 3, 2019, 528–547.

17 Pasi Heikkurinen, “Degrowth by means of technology? A treatise for an ethos of releasement,”Journal of Cleaner Production 197, 2018, 1654–1665.

18 Vladan Joler, “New Extractivism (2020)”, extractivism.online, accessed December 6, 2022, https://extractivism.online/. 

Bibliography

1 Corneius Castoriadis, L'institution imaginaire de la société, Paris, Éditions du Seuil, 1975.

2 Marco Deriu, “Conviviality,” in Degrowth – a new vocabulary for a new era, ed. Giacomo D’Alisa, Federico Demaria, Giorgos Kallis, London, Routledge, 2015.

3 “Disnovation.org”, accessed December 6, 2022, https://disnovation.org/index.php.

4 Tony Fry, Defuturing, A New Design Philosophy. 2nd Edition, Bloomsbury: Bloomsbury Publishing, 2020.

5 Michael Given, “Conviviality and the Life of Soil,” Cambridge Archaeological Journal 28, 2017, 127–143.

6 Serge Gruzinski, La colonisation de l’imaginaire. Sociétés indigènes et occidentalisation dans le Mexique espagnol (XVIe- XVIIIe siècle) Collection Bibliothèque des Histoires, Paris, Éditions Gallimard, 1988.

7 Donna Haraway, “Cyborgs and Symbionts: Living Together in the New World Order,” in: The cyborg handbook, ed. Chris Gray, New York, Routledge, 1995, xi–xx.

8 Donna Haraway, Staying with the trouble: Making kin in the Chthulucene, Durham, Duke University Press, 2016.

9 Pasi Heikkurinen, “Degrowth: A metamorphosis in being,” Environment and Planning E: Nature and Space Vol. 2, issue 3, 2019, 528–547.

10 Pasi Heikkurinen, “Degrowth by means of technology? A treatise for an ethos of releasement,”Journal of Cleaner Production 197, 2018, 1654–1665.

11 Ivan Ilich, Tools for Conviviality, New York, Harper and Row, 1973.

12 Vladan Joler, “New Extractivism (2020)”, extractivism.online, accessed December 6, 2022, https://extractivism.online/.

13 Serge Latouche, “Imaginary, Decolonization of,” in Degrowth – a new vocabulary for a new era, ed. Giacomo D’Alisa, Federico Demaria, Giorgos Kallis, London, Routledge, 2015.

14 Serge Latouche, Petit traité de la décroissance sereine, Paris, Librairie Arthème Fayard, 2007.

15 Serge Latouche, Survivre au développement, Paris, Librairie Arthème Fayard, 2004.

16 Ilya Prigogine and Isabelle Stengers, Order Out of Chaos. Man's New Dialogue with Nature, London, Flamingo, 1984.

17 Isabelle Stengers, Cosmpolitics I, Minneapolis/London, University of Minneapolis Press, 2010.

18 “terra0,” accessed December 6, 2022, https://terra0.org/.

19 Anna L. Tsing, The Mushroom at the End of the World: On the Possibility of Life in Capitalist Ruins, Princeton/Oxford, Princeton University Press, 2015. 

Drivers for Resilience in Cultural Organizations: lessons from the Montreal festivals in the face of the COVID-19 pandemic.

Nicolas Ricci, Marine Agogué

Center for Management Science, Mines - Paris
HEC - Montréal nicolas.ricci@mines-paristech.fr, marine.agogue@hec.ca

Abstract

Crises such as the COVID-19 pandemic represent appropriate moments to innovate. Many organizations in the cultural sector have thus proposed numerous changes in their activities trying to develop new forms of symbiosis, bringing back the notion of resilience. Beyond its buzz word aspect, resilience has essentially been associated with a set of organizational capacities to adapt and innovate in the face of a disruption in the cultural environment, leaving little consideration to question the main drivers of resilience in cultural organizations.

We propose then to study the adaptation of the Montreal festivals offer, building on primary data from 8 interviews with festival directors or managers and secondary data from internal and external documentation. We therefore mobilize the concept of the business model to identify and discuss the drivers for resilience in cultural organizations. We show a trend for festivals to come back to their formal business model despite the deployment of different innovations and identify role and purpose as the two main drivers for the resilience of festivals.

Finally, we call for a comparison with other cultural organizations to discuss the preserving and reconfiguring aspects of their resilience.

Keywords

Resilience, COVID-19 Pandemic, Festivals, Cultural organizations, Business models, Purpose.

DOI

10.69564/ISEA2023-24-short-Ricci-et-al-Drivers-for-Resilience

Introduction

The COVID-19 pandemic and the associated restrictions have generated a disruption in the offer and consumption patterns of cultural products⁴. The transition to online has allowed a favorable development for cultural sectors that did not depend on the physical presence of their public. On the other hand, very dependent sectors such as cultural events in a broad sense, have faced significant difficulties despite various innovations in the format and distribution of their cultural offer.⁶

Internally, the teams—with a high turnover rate in some cultural events—have been working intensively since the start of the pandemic to reinvent themselves while preserving their identity. This has led to the emergence of a tension between identity and constrained innovation.¹⁰ We offer with this paper to clarify how this conflict was expressed within cultural organizations by questioning a well-known concept in management when dealing with adaptation under constraints: organizational resilience.

Literature review

Organizational Resilience as a development of capacities to adapt and innovate in crisis period

From its original definition in material science to the first ones in psychology, resilience has been defined through the idea of a capacity of adaptation to a shock.¹ In management science, the development of the concept of Organizational Resilience followed the same logic. For instance, Gibson and Tarrant define resilience as an “adaptive capacity and how we better understand and address uncertainty in our internal and external environments.”³ This observation can be confirmed by systematic review on the concept of organizational resilience.¹¹

The essential of the literature thus focus on the description of this capacity. Begin and Chabaud¹ propose a typology to describe the dimensions of organizational resilience:

Organizational Resilience as a reasoning moment for organizations

However, building on Kraemer definition of resilience in social work studies⁷, resilience can also be defined with a phenomenal lens rather than the notion of capacity. According to the author, an acceptable description of resilience depends on the context of your study (cultures, time, people, etc.) Thus, the fact that resilience involves a phenomenon and a cognitive process from the actors to tackle it, constitutes the common point between every form of resilience.

Organizational resilience can then be described through a succession of moments² illustrating successive cognitive processes. 

By considering a psychodynamic approach of resilience, Winkler¹² investigate actors thinking and reasoning during a crisis period, leading to the consideration of individual drivers for resilience.

Yet, we lack such an approach of organizational resilience allowing us to identify the phenomenon of resilience⁵ and discuss the main drivers for the reasoning of actors from cultural organizations in crisis periods.

Methodology

Field of study: Why considering festivals to reflect the phenomenon of resilience within cultural organizations?

Among the variety of cultural organizations, we propose to study festivals. Their activities deeply depend on the physical gathering of publics and artists in a same place. Since the start of the pandemic, festivals have faced deep changes in their business models and modified their formats in digital forms or cancelled their editions and constitute then an interesting panel of forms of resilience. Moreover, festivals play a central role in the exhibition of several artworks and represent a real platform in the life of cultural ecosystems in general.

Data collection

We started by building a database registering most of the different festivals of Montreal (n=71) and information found on websites and social media such as attendance, followers, period of the year, duration, type of festival (i.e., music, cinema, visual arts, etc.), format of 2020/2021/2022 editions.

We contacted 68 of these festivals through direct mail to directors, contact mail, LinkedIn.

We interviewed 8 directors and managers between December 2021 and June 2022, with a variety within the festivals in terms of size, duration, and types. The semi-structured interviews lasted between 30 minutes and 1 hour, we developed themes such as the adaptation of the festival team, the format before, during and after the pandemic, their relations with the different stakeholders.

We completed this primary data with internal documents (audience studies, annual reports, internal communication) provided by the directors and managers to complete our study. We also collected information from websites and local press articles.

Data analysis

This study follows the principles of a grounded theory: we use data collected directly in our field of study to develop intermediate theorization. To do so, we proposed an open code of the interviews content.⁸ 

We identified two major categories in the content: one part referring to the business model of the festival (format, stakeholders, funding, etc.), a second one relative to remarks on the role of the festival, its purpose. Then, we proposed to encode the information on the business model in canvas format for business models.⁹ We then identified several codes for the drivers of reasoning that we gathered into two concepts: “role” and “purpose.”

Results

Comparison of the Montreal festivals business models before, during and after the COVID19 pandemic: a trend for a “back to normal” Gathering our coding from each festival interviewed, we propose three general business model of Montreal festivals before, during and after the pandemic.

Pre-pandemic business model.

We built the pre-pandemic business model (Figure 1 (1)) around a value proposal common to all the festivals we interviewed: offer a moment and a place for exhibit of specific artworks to a public. We identified a general trend for the revenue streams (public fundings, private sponsors, event sales) and the cost structure (rent of a physical place, equipment and furniture for the event, salaries). Both interviews and secondary data helped us to define the public of festivals, channels, and public relationship management practices. We finally highlighted recurrent public and private partners, key activities (artistic program, event organizing, public management) and key resources (artists, representation place, festival team and volunteers).

Pandemic business model.

The pandemic business model evolved on almost every aspect (Figure 2). Most festivals turned into a digital format, requiring a reconfiguration of resources and skills around online broadcast technologies. The value proposal remains unchanged, illustrating the lack of will to change the concept of festival and their identity. Nevertheless, the business model is only sustainable thanks to public support.

Post-pandemic business model.

The post-pandemic business model regains almost all its pre-pandemic properties (Figure 3). The digital turn is hardly preserved. However, there is a desire to keep the public engagement tools that proved to be rather effective during the 2020 and 2021 editions, such as additional content or contests. 

Role and purpose as a driver for the resilience process

The comparison of the different business models reveals a clear absence of evolution in the value proposition of festivals. The festival directors insisted on their wish to preserve the identity of the festival, directly linked to the idea of “festival as a platform for artworks and artists promotion.”

Some directors admitted that they think a festival mainly for artists rather than for the public. The willingness to keep strong links with the artists illustrates the role of festivals as a service for artists in both exhibition and socialization and its associated benefits (collaboration, inspiration, artistic movement evolution, etc.). As a result, the key point for the shift to online services delivered during the pandemic was to preserve both moments of exhibition for public and moments between artists and practitioners.

Nevertheless, online festivals remained an unsatisfactory solution (not only for economic reasons), revealing purpose as the second driver of the resilience of festivals. The directors associated the digital format to the lack of informal moments and places for both festival teams, artists and public. The significant part of informality and materiality in these events leaves little adhesion for remote formats, making the physical format part of the festival purpose.

Discussion and conclusion

Our results show that role and purpose are fundamental drivers of resilience within festival teams involving that:

In sum, we were able to highlight a preservative dimension of organizational resilience in the case of festivals. This leads us to question the different forms of preserving resilience that the COVID-19 pandemic has brought to the forefront in the cultural and artistic communities. Conversely, it is quite possible to imagine forms of reconfiguring resilience in other cultural sectors, the study of which could allow for an interesting theoretical confrontation.

Finally, we have seen that the business model is a suitable tool for diagnosing and discussing the phenomenon of resilience, and that rather than an approach based on adaptive capacities, we can think of resilience in terms of the drivers of the actors whose role and purpose constitute.

(1) Figures of the business models are located at the end of this document
References

1 Bégin, Lucie, Didier Chabaud, "La résilience des organisations. Le cas d’une entreprise familiale," Revue française de gestion 200 (1), 2010, 127-42.

2 Conz, Elisa, Giovanna Magnani, "A Dynamic Perspective on the Resilience of Firms: A Systematic Literature Review and a Framework for Future Research," European Management Journal 38 (3), 2020, 400-412, https://doi.org/10.1016/j.emj.2019.12.004.

3 Carl A. Gibson, Michael Tarrant. s. d. "A “Conceptual Models” Approach to Organisational Resilience," The Australian Journal of Emergency Management 25 (2), 6-12, https://doi.org/10.3316/informit.084520139241216.

4 Julia V. Gnezdova, Vladimir S. Osipov, et Igor V. Hrip- tulov, "Creative Industries: A Review of the Effects of the COVID-19 Pandemic," In Post-COVID Economic Revival, Volume II: Sectors, Institutions, and Policy, édité par Vladimir S. Osipov, 2022, 159-71, Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-83566-8_10.

5 Julia Hillmann, et Guenther Edeltraud, "Organizational Resilience: A Valuable Construct for Management Research?" International Journal of Management Reviews 23 (1), 2021, 7-44, https://doi.org/10.1111/ijmr.12239.

6 Olena Khlystova, Yelena Kalyuzhnova, Maksim Belitski, "The Impact of the COVID-19 Pandemic on the Creative Industries: A Literature Review and Future Research Agenda," Journal of Business Research 139, février, 2022, 1192-1210, https://doi.org/10.1016/j.jbusres.2021.09.062.

7 Sebastian Kraemer, "Promoting Resilience. Changing Concepts of Parenting and Child Care," International Journal of Child & Family Welfare 4 (3), 2000, 273-87.

8 Ann Langley, "Strategies for Theorizing from Process Data," Academy of Management Review 24 (4): 1999, 691-710, https://doi.org/10.5465/amr.1999.2553248.

9 Alexander Osterwalder, Yves Pigneur, Business model generation: a handbook for visionaries, game changers, and challengers, Vol. 1, John Wiley & Sons, 2010.

10 Michael G Pratt, Majken Schultz, Blake E. Ashforth, Davide Ravasi, The Oxford Handbook of Organizational Identity. Oxford University Press, 2016.

11 Tennakoon Niranjala, Nadira Janadari, "Organizational Resilience: What it is and what it isn’t? A Conceptual Review," Wayamba Journal of Management, 12 juillet, 2021, 171, https://doi.org/10.4038/wjm.v12i1.7520.

12 Astrid Winkler, "Resilience as Reflexivity: A New Understanding for Work with Looked-After Children," Journal of Social Work Practice 28 (4), 2014, 461-78, https://doi.org/10.1080/02650533.2014.896784. 

Figure 1: Pre-pandemic business model
Figure 2: Pandemic business model
Figure 3: Post-pandemic business model

Metabolism and Art

Hannah Rogers, Adam Bencard

University of Copenhagen, Denmark
hannahstarrogers@gmail.com, adam@sund.ku.dk

Abstract

Metabolic arts should be added to the emerging interdiscipline of metabolic humanities and this paper will discuss ways of defining metabolism that might be productive in helping to produce tools and touchstones for metabolic readings of contemporary art before presenting examples of artworks which might be interestingly illuminated by light of this sign, taking time to relish the process of these materially oriented internal analysis coupled with how the work might be considered in terms of its broader implications for the concept of metabolism.

DOI

10.69564/ISEA2023-25-short-Rogers-et-al-Metabolism-and-Art

Introduction

Among the many challenges of our Anthropocene moment, is the new attention that must be paid to how humans and other species are embedded within, exposed to, and even composed of the very materiality of a planet that is rapidly changing. By focusing on metabolism, understood as “the chemical processes that occur within a living organism in order to maintain life; the interconnected sequences of mostly enzyme catalyzed chemical reactions by which a cell, tissue, organ, etc., sustains energy production, and synthesizes and breaks down complex molecules” (Landecker 2011). As a fundamental biochemical process of life, metabolisms are ubiquitous and multilayered, ranging from distinct multispecies bodily processes, to a variety of chemical transformations across many organisms on a planetary scale, atmospheric and respiration relations between human and plant metabolisms, the body as an environment, to notions of metabolism in a more purely metaphorical sense in aesthetics. Tracing the workings of metabolism is a method for connecting change and dysfunction on a bodily, social, and earth-wide scale. As John Bellamy Foster (1999) explains, Marx conceived of a dysfunction in the form of a metabolic rift as a separation “between humanity and the soil, reflected in the antagonism of town and country” (399).¹

Metabolic humanities appear to be a rising sign under which medical humanities, environmental humanities, and agricultural corners might be united, and metabolicarts may be a productive stage for the three-legged stool of ASTS to balance in tension around the subject.²

Metabolism might be taken to mean many things from distinct multi-species bodily processes, to a range of chemical transformations across many organisms on a planetary scale, atmospheric and respiration relations between human and plant metabolisms, the body as an environment, to notions of metabolism in a more purely metaphorical sense in aesthetics. This is not to suggest that metaphor is absent in what is generally understood to be the scientific notion of metabolism, indeed sociologist Hannah Landecker has shown how very entangled social conditions and metabolic science are and have been.³ Landecker suggests that metabolism was foreclosed by understanding the body as a machine, with a ledger, but this could also point to understanding the body as a business, with immediate implications about capital and a relationship to Foster's Marx.

Metabolisms are ubiquitous but may be most noticeable in states of dysfunction. These dysfunctions are the basis of medical sciences, the source of new ways of conceiving of ecological relations in a climate-changed world, and are a longstanding way of diagnosing philosophical indigestions. Metabolism can be the constant that brings methodological tendencies to the fore as a shared subject, and yet the variable historical understandings of metabolism alone give rise to ages possible metabolic unfoldings so that our studies need not be bounded by the current state of metabolic science and indeed, as ASTS scholars would argue, this is best understood in its social, political, philosophical, and historical contexts. Metabolism serves so many possible needs for scientists, scholars, and artists by providing metaphors, models, puzzles, solutions and balances. Metabolic science is built on a stack of ever-modified metaphors,⁵ including the metabolism as an engine or motor (fast/slow and often an emphasis on the notion of fuel/energy sources), furnace (hot/cold), “chain reactions,” “chemical cascades,” the “chemical carnival,” and many others.⁶ Science is implicated in metaphor thinking, as even the classic experiment asks us to make correspondence between the specific findings on the bench and the broader world, and this extends out to our public understandings of science, often with further analogies which both simplify and make more culturally complex these concepts. We suggest a focus on Landecker’s etymology for the term Stoffwechsel, translated as “total metabolism,” with a further emphasis around the "stoff'' or in English “stuff” of the process to emphasize the insistence bioartists have shown in exhibiting the stuff of living things and parts.⁷ This emphasis on stuff has been important in bioart as such work has avoided distance and representation except by a part of the actual living tissue, bacteria thing and at the same time it has insisted on pulling against conceptual art to attempt to produce in material those ideas. Being in the presence of stuff has been a hallmark of bioart and an emphasis on process may well be the hallmark of the metabolic arts.

The dual use Marx and Landecker make of, on one hand, critiquing soil chemistry or medical genetics, and on the other hand, using those subjects as the basis for their thought, is a situation shared by bioartists who are often using the very biological and biotechnical materials they are critiquing. Enter metabolic arts: drawing on metabolism from the sciences, embodied experience of metabolism, and the potential of laboratory and home metabolism practices to create encounters with life and life processes, many artists and art-aligned practitioners have created art and art-science works that relate to metabolism. We observe and anticipate that many artists may return to Marx's original thinking about metabolism, particularly, his interest in the way that capitalist systems attempt to disentangle plants from animals including people, one natural and functional system was separated into two sets of problems with planetary implications: urban/rural waste and food insecurity/soil depletion.

Metabolic arts

The metabolic gaze can enrich understandings or experiences of a range of contemporary life-science engaged artworks. Tissue Culture & Art (TC&A), made up of Oron Catts and Ionat Zurr, along with collaborators specific to individual works, have offered a trajectory of artworks spanning from 2016 to the present, including iterations of COMPOSTUBATOR and Sunlight Soil and Shit (De-)Cycle (3SDC).⁸ While each work is distinct, they represent the formation of the artists’ thinking on these subjects over a number of instantiations. In the COMPOSTUBATOR series, a compost heap powers an incubator for sustaining a group of cells, generally through microbes heating water in conduit which flows past the incubator chamber and regulates the temperature. 3SDC was designed as a series of engagements around a set of agricultural tools with an emphasis on considering their implications and the philosophies behind these and other proposed system changes. 3SDC was launched with the COMPOSTUBA TOR in Freemantle, Australia, positioned by the artists on their project website in relation to agricultural systems: “The heated incubator sustains the growth of cells in a tissue culture flask to create what is today known as “lab-grown meat.” This type of “meat” is the cornerstone of what is called Cellular Agriculture – growing animal products without the animal.” A second portion of the project is the use of Alkaline Hydrolysis (also Aquamation or Biochemiation), based on an 1888 process for turning farm animals into crop fertilizer. The artists created their own version by hacking brewing equipment to break down much smaller animal bodies (meat and fishing waste) to fertilize the Hydroponic Garden, which is ironically supported by artificial lights driven by solar panels when possible. All of these elements are connected by the Control Room which gathers data from sensors across the project (thermometers, CO2 levels, pH monitors, cameras, etc.) The project aims to highlight the problem of increasing metabolic rifts in order to solve agricultural issues, something that seems implicit in many of the lab-based food systems proposed in the public sphere today. The artists write: “SymbioticA’s 3SDC builds resources to enable the community to accelerate metabolic rifts in agricultural innovation. This project considers whether the precursor to sustainable food systems will be the creation of a metabolic rift – where the means of production will grow ever distant from nature.” The artists directly invoke Marx’s metabolic rift as the target of their investigations by explaining that the exhibition is durational and will be changed over the course of the exhibition. Their aim at “maintaining the utmost clarity and transparency of our process is the key topromote understanding of the impact of metabolic rifts.” TC&A are working with metabolisms, in its nuanced form at the microbial level and at the broader level of our food system. 

Figure 1. Baum and Leahy's Cometabolise: A Holobiont Dinner, 2021. Detail installation photograph from The World is in You, Medicinsk Museion and Kunsthal Charlottenborg, 2021. Photo: David Stjernholm.

Baum & Leahy’s Cometabolise: A Holobiont Dinner (2021) was a living sculpture and an exploration in making the idea of the holobiont more familiar for viewers (Figure 1).⁹ A holobiont is an assemblage of a host and other species living in or around it that together form a discrete ecological unit.

From a holobiont perspective, our bodies are permeable living environments for our cells and the cells (and whole bodies) of other living things. It expands and blurs the notion of the host as a unit which could ever be extracted from this entangled set of relations. The artwork insists on the overlapping metabolic processes of the multispecies beings which help to metabolize our food. The artwork emphasizes the idea that bodies are porous, multispecies entities, highlighting the fact that humans and microbes eat and drink together. This bespoke dining set contains a sourdough culture held in a spherical glass carafe reminiscent of a bioreactor. A closed container for a starter culture is, of course, one of our most familiar bioreactors. The piece reminds us of the domestic nature of metabolism and invites thoughts of kitchens and laboratories. The artists emphasize the performative nature of the work as the microbes are constantly metabolizing and are fed while visitors are offered bread baked from the starter at the communal dining set.

George Gessert and Violet Ray’s BREATHE (2022) is a video piece investigating plant metabolism and its circular processing with our human breathing. Given his history with plants as a primary subject and medium for his artwork, Gessert’s work with plant metabolism is an obvious extension of those concerns. The yet unexhibited filmic work which was created with Gessert’s longtime interlocutor but new collaborator media artist Violet Ray, poetically explores the metabolism of plants through photosynthesis by exhibiting the process at a cellular level and using text to invite audiences to connect their own breath to the cycle of the plant’s photosynthesis and respiration. The artists ask us to pace ourselves with plants. As Gessert puts it, as we think of metabolism, “Why stick to humans and animals? Photosynthesis creates the air we breathe and is a key part of the planetary metabolism that supports most life on earth.”¹⁰ This focus on the larger cycles that metabolism is implicated in overlaps with the concerns of scale that appear across the metabolic arts.

Tagny Duff’s Wastelands (2015-2018) explores shit as an energy source in a speculative future without fossil fuels. Duff explores a deep future 500 years away when humans, through collaboration with bacteria and viruses, use their own feces as an energy source in small, portable bioreactors.¹¹ The Wastelands Project relies on Duff's many years of experience working with biotechnologies in its artistic practice, with a particular focus on viruses and on White Heather Hunter’s co-invention of a new bioplastic with art conservator Courtney Books. The latter provided the basis for the biomaterial development used to construct the bioreactor bags for the project. Metabolic ubiquity can create complications since living things always involve metabolisms but all art with living things may not query those processes or engage them directly. Yet, the search for metabolisms may provide new insights and new places of tension for contemporary arts about life and the Anthropocene.

References

1 J. Foster, “Marx’s Theory of Metabolic Rift: Classical Foundations for Environmental Sociology,” American Journal of Sociology 105, no. 2, 1999, 366–405.

2 H. Rogers, M. Halpern, Hannah D., K. Ridder-Vignone, The Routledge Handbook of Art, Science, and Technology Studies, London, Routledge, 2021.

3 H. Landcker, “Food as exposure: Nutritional epigenetics and the new metabolism,” BioSocieties 6 No. 2, Palgrave MacmillanUK, 2016, 167-194.

4 H. Landecker, “The Metabolism of Philosophy, in Three Parts,” In: Cooper, I. & Malkmus, B., (eds.), Dialectic and Paradox: Configurations of the Third in Modernity, New York, Peter Lang, 2013.

5 M. Black, Models and Metaphors, Ithaca, Cornell UniversityPress, 1962.

6 Taylor C See, B. M. Dewsbury, “On the Problem and Promise of Metaphor Use in Science and Science Communication,” Journal of Microbiology & Biology Education, 19(1), 19.1.46 (2018), Rabinbach, A. The Human Motor: Energy, Fatigue, and the Origins of Modernity, Basic Books, 1990.

7 A. Bencard, Stofsk(r)ifter: Metabolic Machines by Thomas Feuerstein. Hauser, J. (ed.), A. Bencard (ed.), M. Grünfeld, L. Whiteley, 1 ed. Copenhagen, Medicinsk Museion, Jun 2020.

8 See the Tissue Culture & Art’s pages at Compostcubator 0.1 & 0.2 – The Tissue Culture & Art Project (tcaproject.net) and 3SDC Sunlight Soil and Shit, sunlightsoilshit.systems.

9 For more, see Baum & Leahy’s art pages at Baum & Leahy, baumleahy.com.

10 Personal correspondence with the artists, February-April 2022.

11 For more, see Tagny Duff’s project page at Wastelands – Thoughts, images and experiments considering human-microbial relations on Earth in 2517, wordpress.com, and Whitefather Hunter’s documentation page at biotechnofeminism: laboratory craft | whitefeatherhunter 

Ephemera: Bubble Representations as Metaphors for Endangered Species

Harpreet Sareen1, Yibo Fu2, Yasuaki Kakehi3

1The University of Tokyo, 2,3 Parsons School of Design
Bunkyo City, Tokyo, Japan, New York, NY USA
sareen@xlab.iii.u-tokyo.ac.jp, fuy644@newschool.edu, kakehi@iii.u-tokyo.ac.jp

Abstract

The effects of a hierarchical relationship of humans with non-humans are now more pronounced than ever. Anthropogenic ecological stressors, including high levels of carbon dioxide, water scarcity, habitat fragmentation have led to disruption of climate systems, in turn endangering many local and global species. ephemera is an installation composed of glass vessels that show bubble images representing animals from all continents and ecologies currently under threat as per the IUCN Red list. These self-assembling bubble pictures, formed by nucleation of CO2 bubbles in water, are in a homeostasis at the beginning of the installation and shrink each hour to eventually disappear in a few days. The tension between the present endangerment and the urgency of the future action, manifests in the shrinking of these bubbles, invoking unnatural ephemerality due to the human effect. The fauna pictures in this installation, composed of carbon dioxide bubbles, symbolize the transitoriness of now threatened species.

Keywords

climate change, biological functions, endangered species, iucn, ephemeral art.

DOI

10.69564/ISEA2023-26-short-Sareen-et-al-Ephemera

Introduction

In 2015, Hawaiʻi was called as the extinction capital of the world.¹ Occupying a small land mass (0.25%), the islands had 25% of the endangered species in the US.² In intertwined ecosystems, direct and indirect effects human activities—monoculture, greenhouse emissions, extraction, and urbanization—on the biological diversity in a different location are significant³ but often hidden from sheltered lives of humans in another place.⁴, ⁵ Separated from nature in this manner, our thinking of endangered species is cultural, and public engagement with endangered species going forward depends on new structures of imagination. How then should we invest culturally in the fate of endangered species? What emotions do we collectively engender?

This work, an installation composed of ephemerality and melancholy, focuses not particularly on the beauty of imperiled fauna but on their current transience. Glucksmann’s philosophy of the ephemeral⁶ is particularly relevant where the moment is not static, but modulated and resonant. Drawing parallels with flux-images, where the process takes precedence, Glucksmann calls out the aesthetics of the ephemeral as fluid and polysensorial. In the context of this installation, images of various endangered species form with carbon dioxide bubbles—bubbles that shrink in a few days gradually waning the entire image. Materials within this installation’s cultural context are designed not for absolute control but for its relegation. 

Concept

Jean-Louis Boissier called out the roles of images as an interface,⁷ which we can act upon, manipulate, and transform—an image that vectorizes a relationship. We focus on such a relationship through evocative material with embedded cultural context of greenhouse gases and their effect on fauna. Specifically, the focus is on the ephemerality of the images. In prior transient works ⁸,,¹⁰,¹¹,¹² durability is determined by the intrinsic material properties in combination with the surrounding ecosystem, often deliberately designed to offer only partial or imperfect control. Material semantics - original meaning of a material - shape the perception and overall experience. The ambiguous ontological status of the ephemeral also makes it a “powerful metaphor for expressing nuances of memory, time and knowledge.”¹³ Interpretation of ephemeral meanings and consequences is thus beyond the object itself.¹⁵ 

1. Glass surface (13” x 13”) with image of a panda made through ~6000 nucleated and controlled bubbles in a carbonated liquid. © Harpreet Sareen, Yibo Fu, Yasuaki Kakehi.

In this work, images of endangered animals form on glass vessels through thousands of carbon dioxide bubbles. These bubble formations shrink by the hour to eventually disappear in a few days. The bubbles are formed by nucleation of CO2 bubbles in water and are representational images of animals from continents and ecologies currently under threat.

The installation is composed of ten glass vessels that are hydrophobically treated for controlled micro/nanostructures on surfaces. When carbonated liquid media is poured onto the surface, bubble generation is activated and bubbles of carbon dioxide of various sizes stick at defined points on the surface. Inspired by Sylvester et al., who mentioned bubbles as a material¹⁶ between “neither real nor fully virtual,” we extend the temporality of bubbles in this work. The process from initial nucleation to stabilization takes an average of twenty minutes, depending on the saturation of carbon dioxide in the liquids. Bubble images and patterns are stable for one day before they start to gradually shrink, completely disappearing in five days.

System Design

We created a novel fabrication technique to selectively modify hydrophobicity properties of surfaces (glass/plastic) such that bubbles in carbonated liquids nucleate at specific positions on surfaces.

2. Custom p5.js based design tool that converts images into bubble patterns based on original contrast to be then used for fabrication on a vinyl cutter. © Harpreet Sareen, Yibo Fu, Yasuaki Kakehi
Figure 3. Sample bubble pixelation. Left: Grayscale photo of a sea turtle. Right: Bubble-pixelated photo with visible features © Harpreet Sareen, Yibo Fu, Yasuaki Kakehi

This turns stochastic nucleation into controlled nucleation of bubbles for images, patterns, and text on various surfaces. Such surface modifications are initially invisible to the human eye. On pouring carbonated water in containers of various shapes (horizontal/vertical) and sizes, bubbles nucleate, coalesce, and grow to accurate sizes (1.0 mm – 5.5 mm) thus creating patterns, images, or text on surfaces. Such bubbles are highly stable and can keep sticking to the surfaces for longer than a week without significant disturbance.

Figure 4. a) Small bubbles nucleating and merging to predefined sizes, b) Penguin representation through bubbles. © Harpreet Sareen, Yibo Fu, Yasuaki Kakehi.

To fabricate the glass vessels, we use Regular (Home Depot, 599047) glass as a substrate which is initially hydrophilic (wettable) in nature. An illustration software (Adobe Illustrator) or our custom design tool is used to create desired patterns and print out an image mask using a vinyl cutter (Cricut Maker 3). The sticker mask is pasted on the surface and a ceramic hydrophobic coating is applied manually on the glass surface using a zigzag technique before drying for 24 hours. This creates a surface with distinct wettability regions that are invisible to the human eye.

Figure 5. a) Arrangement of ten glass vessels on the floor during the exhibition, b)-d) Photos of animals after stabilized bubble patterns were formed on the surface. © Harpreet Sareen, Yibo Fu, Yasuaki Kakehi
Figure 6. Viewers looking at bubble representations of animals in a kneeling position. © Harpreet Sareen, Yibo Fu, Yasuaki Kakehi

When carbonated water is poured into the containers, small bubbles nucleate first coalescing into each other becoming large in size corresponding with the size of the coating on the surface. These large bubbles may become buoyant at first due to the saturation level of CO2 in the liquid. As levels of gas normalize, bubbles stabilize at nucleation sites adhering to the hydrophobic regions with an attractive force¹⁷. Such stabilized configurations of bubbles last for five to seven days.

To design the bubbles patterns for this installation, we create a custom tool based on p5.JS that allows automatically generated wettability patterns by consolidating image import (color or grayscale), grayscale conversion and corresponding bubbles pixelation in a single pipeline. User imported images are analyzed against the background. Our algorithm analyzes the input image to create a comparative brightness map, following which darker areas of images are tagged as proportionately dense in bubble density and lighter areas are sparse in bubble density. For easy accessibility, the tool may be used online and outputs a .svg directly compatible with the fabrication machine for cutting the masks.

Exhibition and Experience Walkthrough

Purpura et al. have previously called ephemeral art been as “good to think with.”¹⁴ Their impermanence is a constitutive part of their aesthetic, and of the ways in which they come to act on the world. In the context of the exhibition shown at a major venue, our intention was to generate emotions of sadness or mourning among the viewers, over the endangerment of species. We reviewed the IUCN Red List of imperiled species at various threatened levels and chose ten species from marine, land and amphibious domains across all continents. The vessels were fabricated as per techniques described before, bottom lit for contrast of bubbles in liquid and setup in an L-shape. These were specifically separated from each other and laid on the floor for two key purposes: a) For viewers to bow their heads down or kneel as if paying homage, and b) To portray the vessels as final mementos of threatened species.

Such an exhibition design was shaped with a perspective of higher-levels of abstraction, focusing on carbon dioxide bubbles as material from a cultural perspective. This work is thus not temporary, rather has a directive intent to survive in the memory.

Conclusion

We presented ephemera, an installation where ephemeral images of endangered species are composed through bubbles of carbon dioxide. Bubbles with extended lifetimes slowly disappear in five to seven days, representing the images of endangered species as final mementos for humans. Through our exhibition, we intended the viewers to observe images in a drifting state and to be involved in the relationship depicted by the image. Glass vessels arranged onto the floor and viewed by the audience while kneeling are meant to invoke mourning or sadness among viewers—emotions that were listed over aspects of nature during the rapid modernization progress. Through this work, we explore a new ephemeral material and its aesthetic of affect, and believe that the ephemerality in this work represents a shift from the art object to a communicative act.

Acknowledgements

This work was initiated and primarily supported by JSPS KAKENHI (Grant Number 20H05960). We also thank the Provost Office Fund at The New School, and Labex Digi-Cosme (ANR11LABEX0045DIGICOSME) operated by ANR as part of Idex Paris-Saclay program (ANR11IDEX000302) for their partial supports.

References

1 Washington Post, (n.d.), Is Hawaii ‘the extinction capital of the world’? Exhibit A: The alala bird. Retrieved December 8, 2022, from https://www.washingtonpost.com/national/health-science/is-hawaii-the-extinction-capital-of-the-world-exhibit-a-the-alala-bird/2016/04/25/3f45c6ac-f210-11e5-89c3-a647fcce95e0_story.html

2 Jason D. Baker, Charles L. Littnan, David W. Johnston, "Potential effects of sea level rise on the terrestrial habitats of endangered and endemic megafauna in the Northwestern Hawaiian Islands," Endangered Species Research 2, 2006, 21-30.

3 Chen Xinli, Han YH Chen, Chen Chen, Ma Zilong, Eric B. Searle, Yu Zaipeng, Huang Zhiqun, "Effects of plant diversity on soil carbon in diverse ecosystems: A global meta‐analysis," Biological Reviews 95, no. 1, 2020, 167-183.

4 Judith Littleton, Gina McFarlane, Melinda S. Allen, "Human–animal entanglements and environmental change: Multispecies approaches in Remote Oceania," In The Routledge Handbook of the Bioarchaeology of Climate and Environmental Change, 493-510, Routledge, 2020.

5 K. Anggi Hapsari, Tim C. Jennerjahn, C. Lukas Martin, Volker Karius, and Hermann Behling, "Intertwined effects of climate and land use change on environmental dynamics and carbon accumulation in a mangrove‐fringed coastal lagoon in Java, Indonesia," Global change biology 26, no. 3, 2020, 1414-1431.

6 Christine Buci-Glucksmann, Emanuele Quinz, "For an Esthetics of the Ephemeral. Interview with Christine Buci-Glucksmann," Hybrid, Revue des arts et médiations humaines 1, 2014.

7 Margaret C. Flinn, "Jean-Louis's Moments of Jean-Jacques." Studies in French Cinema 10, no. 2, 2010, 141-154.

8 Daisuke Uriu, Naohito Okude, "Thanato Fenestra: photographic family altar supporting a ritual to pray for the deceased," In Proceedings of the 8th ACM conference on designing interactive systems, 2010, 422-425.

9 Virolainen Antti, Arto Puikkonen, Tuula Kärkkäinen, Jonna Häkkilä, "Cool interaction with calm technologies: experimenting with ice as a multitouch surface," In ACM International Conference on Interactive Tabletops and Surfaces, 2020, 15-18.

10 Elizabeth Diller, Diana Murphy, Ricardo Scofidio, Blur: the making of nothing, Harry N Abrams Incorporated, 2002.

11 Suzaan Boettger, "34. Within and Beyond the Art World: Environmentalist Criticism of Visual Art," Handbook of Ecocriticism and Cultural Ecology, 2016, 664-682.

12 Parés Narcís, Jaume Durany, Anna Carreras, "Massive flux design for an interactive water installation: WATER GAMES," In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology, 2005, 266-269.

13 Allyson Purpura, "Framing the ephemeral," African Arts 42, no. 3, 2009, 11-16.

14 Marilyn Ekdahl Ravicz,"Ephemeralart: A Casefor the Functions of Aesthetic stimuli," 1980, 115-134.

15 Daniel Miller, "Artefacts and the meaning of things." In Companion encyclopedia of anthropology, 430- 453, Routledge, 2002.

16 Axel Sylvester, Tanja Döring, Albrecht Schmidt, "Liquids, smoke, and soap bubbles: reflections on materials for ephemeral user interfaces," In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction, 2010, 269-270.

17 Chen Shi, Xin Cui, Xurui Zhang, Plamen Tchoukov, Liu Qingxia, Noemi Encinas, Maxime Paven, et al. "Interaction between air bubbles and superhydrophobic surfaces in aqueous solutions," Langmuir 31, no. 26, 2015, 7317-7327. 

Jellyeyes – Symbiosis, Evolution and Vision

Jill Scott

ZHdK, Zurich, Switzerland info@jillscott.org

Abstract

Jellyeyes is an augmented reality installation especially created for the viewer based on symbiosis, touch, and biomimicry. Jellyeyes is inspired by narratives from the theories of evolution, and these stories involve the characters of a hunter-diver-tourist and a marine biologist. These narratives are inspired by the personal diving experiences of the writer and residencies in three different science research centers that focus on evolution in relation to symbiosis, marine biology, and neuro-visual systems.

Keywords

Augmented reality, symbiosis, evolution, eyes, climate change, barrier reefs, empathy.

DOI

10.69564/ISEA2023-27-short-Scott-Jellyeyes

Research

In 1967, Lynn Margulis created the concept of endosymbiosis. In this scientifically renowned theory, she claimed that bacteria was the agent of change.¹ Specifically symbiotic bacteria invaded the primary cells to create mitochondria and chloroplasts, the powerhouse parts of the cell as we know it today. Jellyeyes is about the focus of this theory on the evolution of light cells or fluorescent chloroplasts. They are found in fluorescent bacteria that live in the symbiont algae or Zooxanthellae inside any coral reef. These bacteria called Cyanobacterial Endosymbiont evolved to create fluorescent light that acts as a sun block for the coral and is sensitive to temperature change. These chloroplasts are also found in the pigmented cells of our eyes called Rhodopsin, that move through the photoreceptors of our retinas in reaction to incoming light. (Figure 1) 

1. Theory of Endosymbiosis, Zooxanthellae Coral Symbionts and Rhodopsin (photoreceptor pigment). ©Jill Scott

This research in evolution and symbiosis led to discussions at conferences with marine biologists in Australia.² In the Great Barrier Reef, symbiotic relationships between fluorescent algae and modern corals are currently being affected by human impacts on the lives of species in this environment. In fact, the health of these symbiotic relationships between the algae and corals, can shift the very roles they play in aquatic ecosystems, for example, as helpers or guardians against pathogens or enemies. The death of symbiont algae can be seen in the latest sad map about coral bleaching in Australia. (Figure 2) Unfortunately, a warmer climate not only affects chloroplasts in algae but symbiont levels of co-habitation as well as some species levels of visual acuity.

Reef
2. Barrier Reef Map and the symbionts. Zooxanthellae Algae. ©Jill Scott

Neuroscientists study the effects of excess light on photoreceptors, and their pigmented chloroplasts.³ Human photoreceptors have evolved through adaptation and are very similar to the retinas of the box jellyfish and the squid. Classified in evolution under the label of camera-based eyes, all three species share two jelly-like substances; aqueous humor or the moving fluid in the eye that is affected by temperature and vitreous humor: the jellylike shapes with fibers that are attached to their retinas. Hence the title: Jellyeyes!

3. The whole setup of Jellyeyes with 8 photographic panels and Augmented Reality interface on an iPad. ©Jill Scott

Because liquids in these eyes are affected by UV exposure, an increase in bad conditions cause cellular damage and very dry eye problems.³ Photoreceptors need healthy chloroplasts to function accurately. In humans they control low and high light levels and the photoreceptors are formed from neural tissues and directly attached to the brain (optic nerve). In squid: they control low light levels in photoreceptors that are also attached to their brains, but in jellyfish, they use basic pigmented cells to assess light and these are attached to each other by fibrous tissues. This last species has no brain but what is called a distributed brain. (Figure 4) But how can such information on evolution in neurocience, marine biology and symbiont behaviour be interpreted in an artwork? Could Augmented Reality and biomimicry help us “see” and understand the reef environment through the light receptors of the eyes of other species?

4. Eyes of the Australian Box Jellyfish and photoreceptors of the Squid and the Human. ©Jill Scott

Artwork

In summary, Jellyeyes is such an augmented reality investigation. It consists of a large photograph of a dead barrier reef and an iPad built into a sculpture of the human optic nerve. (Figure 3) The image on the iPad brings the photo alive! Sounds of water and bubbles immerse the viewer into the environment. The aim is to explore the delicate balance of this aquatic ecosystem through the 24 eyes of the box jellyfish, our own human eyes and eyes of squid or calamari. But the viewer can choose a temperature gauge to warm the water, and this changes the environmental conditions! Ocean warming affects the predator-prey relationship of these species and the symbionts algae leave the coral to die. The viewer can select the magnifying glass to discover how the loss of this symbiont caused a loss of a food for the coral to eat, a weakened immune system, bleaching, disease, and death. Warming also creates other food-algae and more jellyfish breed in the reef. In another scenario they can choose to have an empathetic and parabolic look through the eyes of the Australian box jellyfish at the changing predator-prey relationships of species in the reef. The viewer can choose the black and white view of the squid’s eye to discover stories about the human hunter or they can use the human eye to see different narratives.

The whole artwork is also available on an iPhone with a downloadable screen-based photograph. Here the aim is to help tourists understand the ecology and the climate change problems on the reef.

But how exactly are these narratives chosen by the viewers?

Evolution and Roleplaying

In Jellyeyes, these narratives are divided into three sections or menus that the viewer can explore: Co-Evolution, Structural Evolution and Comparative Evolution.

Co-Evolution:Here bacteria are seen as the agents of change.⁴ In this tribute to Lynn Margulis, Jennifer Margulis (her daughter) is turned into the character of a marine biologist, who collects evidence about the state of symbiont death in the coral and refers to her mother’s theories. She collects fluorescent bacteria from the algae (zooxanthellae) and measures the corals level of nutrition and growth, how they block excessive sunlight, control toxic compounds, stress levels and even ward off pathogens who attack the algae. The viewer’s interaction causes acidity and symbiotic destruction in this environment. (Figure 5)

Structural Evolution:This menu takes Charles Darwin’s theory of adaptation⁵ and applies it to the camera-based eye. How has this eye evolved in relation to its lineal ancestors. The result is a simplified tree of the evolution of visual perception. (Figure 8) For example, the jellyfish is an early example of our eye’s development and the squid another stage in the evolution of the light receptors of the camera-based eye. In this narrative, an ignorant, destructive tourist steals too much food. Sometimes, even digs into the coral to the horror of the scientist who tries to teach respect for this environment. (Figure 7)

Comparative Evolution:James Lovelock once said that life has created the conditions for its own existence.⁶ In this menu, the viewer can choose either the eye of the box jellyfish, the squid, or their own eye to see the relationship between the behavior of these three species in the barrier reef environment. The ignorant diver tourist plays around with the poisonous Australian box jellyfish and every day more of these very stingers swim in the reef. Ocean warming shifts the ecology of the reef’s predator-prey relations and behaviors like these will have a backlash and harm our own existence. (Figure 7)

As in previous artworks,⁷ Jellyeyes uses the sensorial strategies of touch and biomimicry, to create performative roleplaying to augment the above narratives and their repercussions. In Co-Evolution the role of the participant becomes the CO2 emitter and through touch he or she or they, learn about the reactions of symbiont bacteria, the survival of the coral, depreciation of the symbiont algae, squid, and reproduction of the box jellyfish, as well as the interdependencies between these species. In Structural Evolution, the participant becomes the investigator and learns through touch about the similarities of vision between species and about the evolution and health of the light receptors in the photoreceptors. In Comparative Evolution, the viewer becomes an empathizer, they see through the eyes of these species and can change these views with touch to identify what they see. Here, the biomimicry of sight is used to witness the human environmental impacts on the reef itself. 

Experience1
5. Co-Evolution: Scientist collecting symbiont samples from Alge on the Barrier Reef. ©Jill Scott
6. Structural Evolution: The black and white parabolic view from the eyes of the jellyfish and the menu to choose different eyes. ©Jill Scott
7. Comparative Evolution: The hunter tourist interacts with the jellyfish and the scientist. ©Jill Scott

Conclusion

Perhaps, augmented reality art might be an effective catalyst for climate activism and education. Here the aims are to reveal the influence of evolution on species survival in relation to the health of their habitats, to the need for inherited variations and to the mutual benefits of keeping symbiosis alive. It is dangerous for us and them not to mitigate climate change. Jellyeyes clearly shows the effects of our fossil fuel emissions on the survival of the species in the reef. It offers a novel creative interpretation of structural evolution and ecological interconnections. Here, the viewer becomes immersed in the symbiotic sensory relationship between algae, vision and modern corals. Viewers say that Jellyeyes encouraged them to think about variations and reproduction problems as well as symbiotic mutualism. They empathized with non-humans by “seeing” through the eyes of other species. Afterall, the future of the barrier reef on this planet is in the eyes of many species, but it is now in our hands!

References

1 L Fester, R. Margulis, Symbiosis as a Source of Evolutionary Innovation Speciation and Morphogenesis, MIT Press, 1991.

2 ACRS Australian Coral Ref Society Conferences, https://australiancoralreefsociety.org/conference/.

3 Neuroscience chloroplast research, https://www.sciencedirect.com/science/article/pii/S2666469020300154, Accessed 2023.

4 L Margulis, Symbiotic Planet: A New Look at Evolution Basic BVooks, 1999.

5 Charles Darwin, On Origin of Species, Pub John Murry, 1859.

6 James Lovelock, GAIA: A new look at Life on Earth, Oxford Uni Press, WW Norton, 1988.

7 Neuro_Eco _Media, www.jillscott.org

8. The Evolutionary tree of the.Camera Based Eye. ©Jill
Figure 9. A viewer meets the Marine Biologist. ©Jill Scott
10. The AR interface case based on the Human Optic Nerve. ©Jill Scott
Partners

HFF, The Munich Film Academy. Munich, Germany
Dr. Lisa-Ann Girshwin (CSIRO Australian Marine Stinger Advisory Services, Australia)

Scientific Consultants

Prof. Dr. Stephan Neuhauss (Institute for Molecular and Cellular Research, Neurobiology, University of Zurich, Switzerland) Prof. Christopher Robinson (Aquatic Research EAWAG, ETHZ, Switzerland)

Funding

Pro Helvetia, The Swiss Arts Council
HFF, The Munich Film Academy. Munich, Germany

Production Credits

Concept & Directing: Jill Scott Production: Marille Hahne Camera: Julia Daschner
Light: Georg Nikolaus Animation: Natascha Jankovski Postproduction: Moritz Huber Programming: Nikolaus Völzow

Biography

Jill Scott is a media artist, a writer and art and science researcher. She is professor emerita at the Zurich University of the Arts (ZHdK) in Zürich and founded their Artists-in-Labs Program in 2000. Her own artwork spans 44 years of production about the human body and body politics. In the last 20 years she has focused human health based on research into molecular biology, neuroscience, and ecology. She has had many international exhibitions in both art and science venues. She also directs LASER Salon in Zurich for the Leonardo Society USA and writes books on art and science (Springer and de Gruyter).

www.jillscott.org

Technoshamanism: Symbiotic Techniques of Art and Healing

Edward Shanken

University of California, Santa Cruz United States eshanken@ucsc.edu 

Abstract

Technoshamanism combines traditional shamanic technologies with emerging technologies based in silicon (dry), biology (wet) and hybrid (moist), in the service of healing and sustaining life. This paper explores how contemporary artists pursue expanded forms of consciousness by symbiotically joining technoscientific tools and shamanic techniques.

Keywords

Shamanism, Technoshamanism, Art, Consciousness.

DOI

10.69564/ISEA2023-28-short-Shanken-Technoshamanism

More than ever before, contemporary artists, theorists, scientists, and activists need to pay more attention to socalled indigenous knowledge. – Guillermo Gomez-Peña 

Introduction

Mircea Eliade defined shamanism as “techniques of ecstasy.” He noted that, across cultures, it is fundamentally a technology of healing that is "at once mysticism, magic, and religion."¹ Anthropologist Michael Harner claimed that shamanic traditions around the world have developed a broad range of technologies - from sonic drivers (drums) to plant medicine rituals—in order to achieve trance states that offer insights beyond those available to typical waking consciousness.²

Shamans are both of this world and of the world(s) beyond. They communicate with spirits and ancestors in other dimensions, learn from them, guide members of the community to them, and harness their power to heal and protect individuals and the community. The shaman can embody the consciousness of other beings, including other animals. A shaman also can release an errant spirit that has stricken a member of their community with illness, sending the uninvited “hitchhiker” on its way. This can be dangerous, so shamans must be very strong, capable of self-healing, and masters of their practice.

Shamanism may heal at scales ranging from the individual to the global. Indeed, following the Huni Kuin concept of Xinã bena–“New Time”—some shamans/pajés of the Amazon have decided that the preservation of the Earth demands sharing their cultural insights and healing technologies more widely than in the past. This strategy has even led to the creation of Huni Kuin—Beya Sina Bena, a videogame about the ancient stories of these indigenous people, scheduled for release in 2024. The developer, Philosophical School of Games, claims that the game was produced in collaboration with the Huni Kuin communities of the Jordão River, in Acre, Brazil.

Brazilian cultural critic Fabiane Borges notes that “apart from any possible encounter between technology and shamanism, shamanism is itself a technology for the production of knowledge. ³ Similarly, I take technoshamanism to join a combination of traditional shamanic technologies with emerging technologies based in silicon (dry), biology (wet) and hybrid (moist), all in the service of healing and sustaining life.

The same can be said of art, historically interwound with techne, the Ancient Greek word for art and the etymological root of technology. To risk stating the obvious, the use of shamanism for healing vastly predates the application of science to allopathic medicine. The recent decriminalization of plant medicines and the legalization of psychedelic-assisted therapies in some jurisdictions demonstrate increasing acceptance of ancient indigenous technologies and marks a merger of diverse healing methodologies.

In a parallel manner, the engagement of contemporary art with shamanic traditions bridges knowledge domains and augments art’s potentials to heal and to envision the future. These goals characterize a wide range of artistic practices, from Ernesto Neto’s A Sacred Space to Roy Ascott’s “shamantic web” of dual consciousness, to Anandha Ray’s Covenant VR virtual reality ceremony/performance. A shaman was even part of Refik Anadol’s cohort of artists and software engineers at the Google Artists and Machine Intelligence program in 2016.

Nonetheless, there is a dearth of literature that addresses the nexus of contemporary art and shamanism. Until recently, the work of Joseph Beuys dominated these discourses, with a few notable exceptions. Jack Burnham’s “Artist as Shaman” (1974) applied a structuralist method to interpret shamanic aspects of Dennis Oppenheim’s work. In “Art in the Dark” (1983) Thomas McEvilley interpreted performance by artists from Gunter Brus to Kim Jones as shamanic acts of expression and catharsis. “Weaving the Shamantic Web...” (Ascott, 1998) claimed that “this ancient ritual mirrors our contemporary artistic aspirations using digital technologies.” ⁴ Further essays by Ascott and his circle, especially Lila Moore and myself, have followed in the 2010s and 2020s.⁵

In 2017, Christine Macel curated The Pavilion of Shamans, one of nine “trans-pavilions”, bringing shamanism into the center of mainstream contemporary art (MCA) discourses at the 2017 Venice Biennale. Notable works included Juan Downey’s Circle of Fires Vive (1979), Jeremy Shaw’s Liminals (2017), and Neto’s A Sacred Place (2017), the centerpiece of the pavilion. Despite scant media attention directed to this pavilion, art critic Tess Thackara (2017) highlighted it and discussed many additional international artists, indicating broad interest in the topic, if not by MCA critics, then by makers. ⁶ Indeed, my 2020 Facebook post about technoshamanism yielded over one hundred comments, primarily by artists about works of new media art (NMA) that fit the term. More recent MCA scholarship includes critiques of Mircea Eliade’s and Michael Harner’s universalizing of shamanism and Claude Levi-Strauss’ and Burnham’s psychopathology of the shaman, and is informed by Viveiros de Castro’s concept of “equivocation,” offering a more nuanced and theoretical approach to shamanism and contemporary art. ⁷

Borges and others use the term “technoshamanism” to problematize the destructiveness of western technoscience and its incursion on indigenous technologies of shamanism, which, by contrast, honor the sanctity of the Earth. Technoshamanism names the messiness of cultural hybridity and the commodification of shamanic traditions, including ayahuasca tourism and the gamification of Huni Kuin culture, resulting from colonization and globalization. The visual arts of indigenous cultures, including Shipibo textile patterns, Papunya Tula dot paintings, and Vodun rituals, are also being hybridized with, and appropriated by, international contemporary art and visual culture. Such appropriations raise vital questions about Technoshamanism as a form of aesthetic practice: How are contemporary artists engaging with the cosmologies, technologies, and intellectual property of shamanic cultures in ways that honor and benefit the indigenous and mestizo peoples that have cultivated and preserved shamanic traditions? How are indigenous artists engaging with shamanic (and postindustrial technologies) in ways that strengthen their communities and contribute to the discourses of contemporary art?

Mindful of these issues, my research is primarily concerned with the following speculative prospects:

  1. How can artists embrace visionary consciousness?
  2. How can art support entheogenesis (becoming divine together) by joining ancient shamanic techniques and contemporary technoscientific tools?
  3. How can art catalyze greater awareness of what Thich Nhat Hanh calls interbeing (the unity of all things) to help heal the Earth?

Early Technoshamans

In 1997, British artist Roy Ascott participated in ayahuasca ceremonies in Brazil, which has had a profound impact on his praxis. He theorized parallels between the dual consciousness that emerges in shamanic ceremonies and the expanded field of consciousness afforded by emerging technologies. “In many respects,” he wrote in 1998, “this ancient ritual mirrors our contemporary artistic aspirations using digital technologies.” Claudia Jacques claims that this dual consciousness can only be manifested through “indirection, shared participation, and metaphor,” core tenets of the telematic art that he pioneered. This is the realm, she adds, that “visionary thinkers, creative artists, and shamans alike aspire to experience and explore.” ⁸

Ascott (1998) describes the shaman as “the one who ‘cares’ for consciousness, for whom the navigation of consciousness for purposes of spiritual and physical wholeness is the subject and object of living.” He witnessed the pajé “passing through different layers of reality, through different realities...” The shaman inhabits a state of dual consciousness, of “seeing at once both inward realities and the outward surfaces of the world.”

Although it predates Ascott’s participation in shamanic plant-medicine ceremonies, his 1989 telematic artwork, Aspects of Gaia: Digital Pathways Across the Whole Earth, embodies a technoshamanic, dual consciousness. This installation integrates the global relationality of digital technologies with the shamanic commitment to healing the Earth. Aspects of Gaia took its inspiration from atmospheric scientist James Lovelock’s holistic Gaia Hypothesis (1979), which proposes that the Earth (Gaia) is a living organism, a self-regulating, complex system that maintains the conditions for life on the planet. Artists, scientists, shamans, musicians, visionaries, and indigenous artists were invited to participate by sharing their feelings and beliefs about Gaia, which became work’s content. Ascott likened the participants to healers who access the meridians of the earth’s nodes and creatively interact with the flow of data to perform a type of “global acupuncture.” Their contributions suggested, for Ascott, a telematic “noosphere,” an emergent field of consciousness generated from interconnected individual consciousnesses, which might help harmonize and heal the planet.” ⁹

For American composer and electronic music pioneer Pauline Oliveros, cultivating expanded forms of consciousness was the primary focus of her career. For this early technoshaman, the wisdom of the body is crucial to accessing expanded states of consciousness. “I have progressed through many changes in music technology from the end of the 1950s to the present,” she wrote. “Along the way I developed a bodily relation to machines for making music. It has always been necessary.... because of the essential knowledge of the body that is preconscious and nonverbal.” ¹⁰ Her experience of performing improvised music parallels shamanic double-consciousness: Oliveros became a channel through with spirits from other dimensions could communicate with and heal us: “This altered state of consciousness in performance is exhilarating and inspiring,” she explained. “The music comes through as if I have nothing to do with it but allow it to emerge through my instrument and voice.” ¹¹ Oliveros’ ideal attributes for a future artificial intelligence “chip”—with which she could make music—are refreshingly expansive. They include abstract psychic abilities, that seem to parallel shamanic intentions of achieving unity and healing on a cosmic level. These include: the ability to understand the relational wisdom that comprehends the nature of musical energy; the ability to perceive and comprehend the spiritual connection and interdependence of all beings and all creation as the basis and privilege of music making; the ability to create community and healing through music making; the ability to sound and perceive the far reaches of the universe much as whales sound and perceive the vastness of the oceans. This could set the stage for inter-dimensional galactic improvisations with yet unknown beings.¹²

Wake me up when ChatGPT can offer these features! For Oliveros, expanded consciousness on a galactic scale was the foundation of healing, a form of service that she performed within the frame of a feminist ethics of care.

Chilean-American artist Juan Downey is another early technoshaman. His 1979 video, The Laughing Alligator, resulted from the many months he and his family lived with the Yanomami people in southern Venezuela, on the border of Brazil. He participated with his Yanomami hosts in rituals using yopo, a plant medicine that contains a variety of DMT alkaloids. The artist came to regard shamanism as one of the most powerful elements in the bond that linked the Yanomami to their surroundings, to the earth, and to each other as part of a larger unity. As he wrote, A white and round place opens up in the front of my brain. Excretions of light that vaguely align into circles, the intensity of a spiral or the infinite peace of a mauve color. [...] I want to enter into the white space of my empty consciousness. ¹³

From spending a lifetime in trance-like states, Downey expanded his consciousness. He tuned into the consciousness of others. And he helped others do the same. As a result, his work plays an important role in stimulating new ways of thinking that are the prerequisite to healing society’s pathologies and to recreating the world in a more sensitive, inclusive, and caring way.

Contemporary Technoshamans

American dancer and choreographer Anandha Ray’s virtual reality screen-dance, Covenant VR (Figure 1), was a revelation. Joining shamanic healing rituals, modern choreography and dance performance, and exquisite VR cinematography by Gary Yost, it created a unique container for embodied catharsis mediated through virtual reality.

After putting on the headset, the next thing I knew, BAM! I was suddenly transported to a dance rehearsal studio, at a virtual distance no more than a meter from a striking, dark-skinned dancer (Linda Steele II.) I felt right there in the action, far closer than front row seats, inhabiting a perspective and proximity typically seen only by other dancers. Because this is a 360-degree VR experience, one’s view is not limited to a frontal perspective.

Figure 1. Anandha Ray, Covenant VR, 2019. ©Anandha Ray.

As viewers enmeshed in the VR environment, we witness the shamanic ceremony from a privileged vantage, so close that we can virtually touch the protagonist, a proximity that heightens our sense of being there, of feeling the healing and of experiencing it ourselves. Yost’s cinematographic wizardry is the ideal match for Ray’s shamanic approach to dance, which “allow[s] movement to open portals of inquiry to better understand the state of being human.” ¹⁴ This tour-de-force of Technoshamanism offers an enthralling and cathartic experience.

Korea has a rich, living tradition of shamanism that continues to inspire artists, including technoshaman Kim Jeong Han. His high-tech multimedia installation BirdMan (2005) is deeply informed by Korea’s shamanic and Buddhist traditions. The artwork asks fundamental questions: How do human beings conceptualize the world? How do birds conceptualize it? Can a hybrid world that joins human and nonhuman qualia (the internal, subjective physiological component of sense perceptions) transform perception beyond the limits of human physiology? If, as Donna Haraway claims, new perceptions create new metaphors, can the experience of another species’ perceptual reality help create hybrid perspectives, marked by greater empathy and ecological sensitivity?

The concept of a hybrid bird-man appeared to Kim in a dream. Due to a traumatic childhood experience, even as an adult, Kim’s fear of birds prevented him from helping a one-winged bird that was suffering and dying. He dreamed that he learned bird language from a monster with a bird head and only one wing. The dream and the artwork can thus be interpreted as an effort by the artist to attain catharsis and heal a trauma through aesthetic and shamanic means. As Kim and his co-authors noted:

“In Korean tradition, some shamans can share their own bodies with the deceased soul. Whenever a shaman is possessed by the spirit of the dead, s/he acts, speaks and senses like another person, as if borrowing the perception of the deceased. This moment looks like a coexistent state of the living body and the dead in which perception and identity of the two is hybridised.” ¹⁵

The Buddhist idea that “the ‘Self” is not different than the “Other” is another prevailing concept in BirdMan. The work offers the audience an opportunity to experience a form of hybrid perception that joins human and avian “qualia.” (internal and subjective physiological component of sense perceptions). Kim’s work does not explicitly represent accoutrements or scenes of shamanic healing. As a result, the audience is not likely to visually identify shamanic elements in it. Rather, BirdMan is driven implicitly by a shamanic perspective. Kim leads his own ritual of self-healing through his artistic practice. By enabling us to metaphorically become one with bird, by offering us an experience of hybrid avian-human perception, his work enables us to expand our consciousness beyond the limits of our embodied human minds by joining self and other. It enables us to create new identities in between humans and non-humans. And, as a result of this, it enables us to create new metaphors to live by and to live with.

Such ideas echo Donna Haraway’s eco-feminist theories of multispecies worlds. She advocates a concept of kinship or “making kin” that joins all beings: “all earthlings are kin in the deepest sense [...]. All critters share a common ‘flesh,’ laterally, semiotically, and genealogically.” She applies the term “sympoietic” to emphasize the collective process of poetic emergence in which all beings are collaborators in the process of the Earth’s becoming. “Who and whatever we are, we need to make-with—become- with, compose-with [...].” Taking care of the Earth, for Haraway, demands caring for the diversity of beings, and “multispecies ecojustice” must be not only a goal but a means to living well, together, as kin. By “staying with the trouble,” she proposes, “[m]aybe, but only maybe, and only with intense commitment and collaborative work and play with other terrans [inhabitants of Earth], flourishing for rich multispecies assemblages that include people will be possible.” ¹⁶

Buddhist luminary Thich Naht Hanh has taught that, “Our own awakened consciousness is what can heal the Earth.” Indeed, the pressing and enduring concerns of global warming and the abuses of technology demand that we expand our perceptual domain in order to heal ourselves, our kin, and our planet. To do so, we must expand our metaphors and our means and channels of communication. We must summon the full power of art and technology, while we equally harness ancient shamanic technologies and other modes of entering trance, expanding consciousness, experiencing ecstatic states, fostering communication among all beings. If we want to have a future, the artists of the future must serve as beacons of hope and as active participants in healing the world and our relationship to it. Art, as a “psychic dress-rehearsal for the future,” to quote Jack Burnham, must embrace kinship and harmonious co-emergence with all beings. We must become divine together. Today there is no greater artistic calling, no greater aesthetic necessity than helping to heal and preserve Earth/Gaia’s biodiversity for posterity. Technoshamanismis a potent strategy to move in that direction. 

References

1 Mircea Eliade, Shamanism: Archaic techniques of ecstasy. c. 1951, Princeton University Press, 2004, xvii–xxvii.

2 Michael Harner, "My Path in Shamanism." International Journal of Transpersonal Studies 31.2, 2012.

3 Fabiane Borges, “Seminal Thoughts for A Possible Tecnoshamanismnd, np, c. 2014, Quote edited for clarity.

4 Roy Ascott, “Weaving the Shamantic Web: Art and Technoetics in the Bio-Telematic Domain.” c. 1998, In Telematic Embrace, ed. Edward Shanken, Berkeley, University of California Press, 2004, p. 356-62, Subsequent references to and quotations from Ascott are from this source.

5 Lila Moore, “The Shaman of Cybernetic Futures: Art, Ritual and Transcendence in Fields of the Networked Mind” Cybernetics and Human Knowing 25, 2018, p. 2-3, 119-41.

6 Tess Thackara, Why Shamanic Practices Are Making a Comeback in Contemporary Art, Artsy, 2017, accessed June 13, 2023, https://bit.ly/48hsvk4

7 This literature includes Robert Wallis, “Art and Shamanism: From Cave Paintings to the White Cube," 2019, Camila Maroja, “Persistence of Primitivism: Equivocation in Ernesto Neto’s A Sacred Place and Critical Practice”, 2019, and Karen Gonzalez-Rice, “Revisiting "Art in the Dark", 2023.

8 Claudia Jacques, “Forward: A Tribute to the Messenger Shaman: Roy Ascott,” Cybernetics and Human Knowing 25, 2018, 5-15.

9 For more on Ascott, art, technology, and shamanism, see Lila Moore and Edward Shanken, “Roy Ascott” in Charlie Gere and Francesca Franco, eds. Bloomsbury Encyclopaedia of New Media Art, London, Bloomsbury Press, forthcoming.

10 Pauline Oliveros, “Improvising Composition: How to Listen in the Time Between,” c. 2012, in Gillian Siddall and Ellen Waterman, eds. Negotiated Moments: Improvisation, Sound, and Subjectivity, Durham, Duke University Press, 2016, 83.

11 Pauline Oliveros, Deep Listening: a Composers Sound Practice, New York, iUniverse Inc. 2005, xix.

12 Pauline Oliveros, “Quantum Improvisation: The Cybernetic Presence,” in "Sounding the Margins: Collected Writing 1992-2009, Kingston NY, Deep Listening Publications, 2009, 53.

13 Juan Downey, Dibujando con los Yanomami, (Caracas: Galería Adler Castillo, 1977): np. Quoted in Edward Shanken, “Pushing the Limits. Surrealism, Possession, and the Multiple Self: Juan Downey and The Laughing Alligator” in Maricris Herrera, ed. Juan Downey, 1940-1993. Mexico City, Ediciones MP, 2019, 527-42.

14 Anandha Ray, Covenant VR ,2019, https://youtu.be/Y-B_KqbrS4w?si=zXeMpmxWCuCNv4QY

15 Kim Jeong Han, Kim Hong-Gee, Lee Hyun Jean, "The BirdMan: hybrid perception," Digital Creativity 26, 2015, 1, 56-64.

16 Donna J. Haraway, "Anthropocene, Capitalocene, Plantationocene, Chthulucene: Making Kin," Environmental Humanities 6, 2015, 1, 159-165.

17 Thich Nhat Hanh, Love Letter to the Earth, Berkeley, Parallax Press, 2013, 56. 

Symbiotic Collaborators: The New Creative Subject in Postdigital Participatory Art

Gyung Jin Shin

Hong Kong Baptist University, School of Creative Arts Hong Kong
gjshin@hkbu.edu.hk

Abstract

As the Internet promotes participatory culture, contemporary network-driven participatory art, which I refer to as “postdigital participatory art” (PPA), has introduced additional revolutionary creative subjects. PPA induces a distinctive type of collective agency beyond mere collaboration among individuals by means of the participatory architecture of the web. These multiple participants distribute the authority power of creation throughout the network, transcending the limitations of time and space. In this paper, I attempt to theorize the attributes of these new creative subjects, which I refer to as “participant-superjects,” with the concept of superject serving here to indicate “power by modulation.” I outline the attributes of these diffuse creative subjects and gauge their radical possibilities in terms of the agenda of experimental art. I argue that, based on the new sense of relationality, materiality, and ontological perception associated with the postdigital environment, these unique creative subjects are able to open up a new dimension of creativity that differs from the modernist model, which emphasizes the creativity of the individual. I hypothesize that the fluid power driven by this new creative subject exerts a latent force in building new social relations outside the logic of the capitalist system.

Keywords

Postdigital participatory art, author discourse, superject, participation, collaboration, participant-superject.

DOI

10.69564/ISEA2023-29-short-Jin-Shin-Symbiotic-Collaborators

Introduction

With the death of the modern subject in the twentieth century, the divinization of single authorship on which modern art relied has faced challenges and slowly collapsed along with the notion of “open work,”¹ the postwar avant-garde of the 1960s, and author discourse.², ³ Artists and theorists of the previous century attempted to undermine the traditional relationship between artist and spectator, arguing instead for open-ended interpretations. These developments spurred a renegotiation of the artist-spectator relationship and, in turn, the invention of new types of creative subjects. From this new perspective, the status of the audience, as a “viewer-turned-participant,” has become almost equal to that of the artist, and recent forms of contemporary art that are co-authored, collaborative, and participatory have inherited the preference for antiauthorship. Such forms of art, including “relational art”⁴ and “participatory art”⁵,, ⁷ have flourished since the 1990s as artists have attempted to distribute the authority power traditionally associated with the singular artist by encouraging the participation of viewers and envisioning new social relations.

In a hyperconnected environment that facilitates participation and collaborative creation, contemporary participatory art has become ubiquitous and heterogeneous, transcending the boundaries of online and offline. Recognizing that Bishop’s concept of “participatory art” overlooks the influence of technology, I seek to help fill the resulting conceptual gap by coining a new term, “postdigital participatory art” (PPA), to describe digitally mediated co-creation that relies on digital networks to encourage audience participation. This new form of digitally mediated co-creation reflects changes in the perceptions of time and space that have been described as “postdigital.”⁸, ⁹ In exploring PPA, I pay particular attention to the emergence of new authority power that is fluid and ephemeral. Though such unique participatory creative subjects are profoundly observable in recent digitally mediated participatory art, they have yet to receive careful analysis in terms of scope, motives, characteristics, and patterns. Accordingly, I consider here the manifestation and radical possibilities of this unique creative subject, which involves anonymous, networked participants mass-produced on networks.

Participant-Superject: The Unique Authors in the Network

As I conceptualize it, network-driven PPA relies heavily on collaboration among participants and often lacks a teleology. Creators, materials, artworks, and spectators, in the absence of a predetermined blueprint for object-making, co-emerge in the processing of inconclusive events. This development gives rise, in turn, to the distinctive characteristics of a new creative subject that is ephemeral and fleeting, untethered as it is to a fixed plan and, often, anonymous and unrecognized despite contributing significantly to the act of creation.

The Jogging (https://thejogging.tumblr.com/), an ongoing, network-driven project, exemplifies the attributes and modalities of these authors. [Figure 1] Utilizing a scroll-down thread on Tumblr that has continued over several years, anonymous volunteers on the network Photoshop, reproduce, and reblog the images associated with the original images uploaded by artists Brad Troemel and Lauren Christensen in 2009. Troemel described his role as “initiating” and said in an interview that “‘Jogging’ refers to a work flow, constantly moving, and not really focusing on any one thing, but rather to just continue forward.”¹⁰ Without an individual artist furnishing a predetermined intention or conclusion, such postdigital participatory projects are observable as the inconclusive, event-embracing agencies, networks, and raw digital materials (e.g., information or data) involved in a work. 

1. Brad Troemel and Lauren Christensen, The Jogging (2009-), Source: The Jogging wesite, https://thejogging.tumblr.com/. 

From Author-Subject to Participant-Superject

This new material variability in postdigital participatory art thus entails a reframing of the traditional notion of author-subject. In a digital environment that strengthens temporality rather than spatiality, digital participants have the potential to become temporally modulated subjects, or superjects, bearing latent power by modulation rather than the power of individuals. As the antithesis of the author-subject status, I coin the term participant-superject to describe the diffuse creative subjects that result from the blurring of the artist-spectator boundary.

The concept of the superject, which refers here to “power by modulation,” has proved useful in describing the special modalities of multiple agents that exist digitally when they are involved in PPA as opposed to the modalities of physical participation. Deleuze adopted Whitehead’s concept of the superject in the 1990s to illustrate the shift in subjectivity accompanying the rapid advance of digital technology at the time.¹¹ According to Deleuze, a subject, when perceiving an object that is continually changing, can be defined as a point of view that is likewise in a state of variation rather than fixed or determined in advance.

The superject implies a “plural” subjectivity that cannot be fully described using the concepts of a constant I and we and, instead, implicates a new status of the subject, a temporal I and we. The superject, in Deleuze’s language, is an inherent multiplicity folded into a collective unity. 

The Three Phases of the Participant-Superject

Before gauging the radical possibilities of the participant-superject, it is useful to consider the concept’s unique characteristics. Following Whitehead, Deleuze, and Savat, I describe the three main phases of the superject as 1) temporal, 2) affective, and 3) condividual. The temporal phase relates to the eventual nature of the superject. Referring to this phase, Deleuze described the superject as an event rather than an essence. Adopting this view, Savat explicated the specifics of the temporal aspect of the superject in the context of his analysis of digitality. According to him, under the new temporal and spatial conditions of the digital environment, a superject exists as what Deleuze called a “dividual,” an event that is ceaselessly modulated as a code, in contrast with an embodied or spatial “individual.”¹² In this context, the manner of existence becomes continuous, fluid, and momentary.

The second phase highlights the affective nature of the superject. Whitehead envisaged this facet of the superject in an attempt to separate the subject per se from its experience of the world.¹³ Thus, he proposed, the superject simultaneously emerges through the intake of “data” in the form of sensory perceptions that precede consciousness, unlike the subject that accumulates data. In this context, feelings mediate superjects and their associated data. This aspect of the superject focuses on the moment of affective experience shared throughout the network.

Third, the superject implies the consolidation of collective agents, that is, condividuals. To be specific, unlike the individual, which retains distinctive characteristics, the dividual tends to connect with other similar components and combine with other dividuals to form condividuals).¹⁴ Existing as metadata or data in the digital milieu, the dividuals can be used to build superjects in the network.

The participants in PPA, or participant-superjects, embody these phases of the superject—again, temporal, affective, and condividual—which overlap and are interrelated. Rather than being constrained to a physical venue for performance or exhibition, the participant-superjects appear temporarily, take part in the creative process, and disappear in an instant. As the sequence of the action does not “take place,” the actors rely heavily on temporality rather than spatiality. Further, the participants in PPA tend to manifest momentary feelings or emotions rather than the conscious experiences inscribed in a work. As The Jogging well shows, the intuitive and improvisational responses of the participants manifest in such functions as liking, retweeting, and sharing on social media. Lastly, these participants, as dividuals, merge easily and unite readily with others thanks to the anonymity and easy access that the Internet affords. 

Symbiotic Collaboration Among the Participant-Superjects

This new creative subject of PPA has the potential to fall into the trap of capitalism or to support resistance to it. Stiegler used the term pharmakon to describe the aspect of technology that intoxicates and cures, and postdigital participation has a similar “pharmakonic” effect, serving as simultaneously toxin and remedy.¹⁵ In the 1990s and 2000s, the expectation was that the participatory environment of the web would promote widespread self-awareness, the formation of grassroots communities, and co-individuation by connecting individuals without respect to age, location, or gender. Since the network has also been viewed as an arena for the fulfilment of capitalist desires, however, the participants in it found that they were treated as consumers and unpaid labourers. Most of the major social networking platforms have reverse-engineered the very structure of the participatory web to produce profits from the databases generated as a by-product of the daily behaviours of online users. Moreover, governments and internet giants alike use the network as a source of big data for analysing superjects and recognizing and forecasting social trends.

Regarding the paradoxical loop of postdigital participation problematized above, I argue that the modulatory power of the participant-superject could radicalize the agenda of experimental works by altering the form that power takes. This new form of power, again as Deleuze pointed out, refers to the modulation of dividuals.¹⁶ For Deleuze, of course, such power deserves criticism, but it also serves as a positive starting point for the arguments presented here. Considering the dividual a latent power of struggle, Raunig claimed that the dividuality emerging as self-division in contemporary social media could promote new forms of dispersed resistance against machinic capitalism.¹⁷ In the same vein, Savat insisted that a new mode of politics involving the dividual, which he called the “politics of fluids,” has become a more critical field of participation than the modern “solid politics” involving individuals.¹⁸ Thus, he affirmed that the potential inherent in “fluid action” can be utilized and realized in constructive ways because dividuality represents individuals’ ways of being in the network.

Following Gaunig and Savat, I affirm that the fluid power of the superjectification seen in PPA—again, in contrast with the solid power of individuals—exerts a latent force in terms of inventing social relationships apart from the logic of the capitalism that finances and reifies human bodies and even social relations. The characteristics of superjectification—being temporal, dispersed, affective, and condividual—are more radical and effective in terms of “pre-empting” the looming problems associated with this logic. Furthermore, the unique sense of affinity helps dispersed cognitions cling together contagiously, in turn affecting real politics (e.g., Ushahidi software, the MeToo and Occupy movements, and the activist group Anonymous).

Thus, PPA projects tend to encourage affinity-driven co-individuation outside the context of the labour system and manipulative relationships on the web.¹⁹ Public_Public_Address: A Nationwide Virtual Protest (2020-), for example, is an ongoing virtual protest for which Jason Lazarus, Stephanie Syjuco, and Siebren Versteeg have been accepting submissions of selfie videos of individuals holding protest pickets in support of the Black Lives Matter movement. [Figure 2] The participants include those who had been marginalized in society and those unable to protest in person because of a disability. The artists integrated the video submissions into live-streaming on YouTube and simultaneously in the window of a New York gallery in 2020. Seemingly marching toward the viewers, the images of these anonymous participants have been incorporated into a bizarre new form of space-time. Rather than isolating, separating, and pixelizing the agency of the participants in the network, this project reveals the aesthetics of fluid power that the superjective participants generate in the postdigital environment by transcending the online-offline binary. 

2. Jason Lazarus, Stephanie Syjuco, and Siebren Versteeg, Public_Public_Address: A Nationwide Virtual Protest (2020-), Source: the bitforms gallery website, https://bitforms.art/public_public_address-2020/.

Conclusions

As the exploitation of participation in the network becomes increasingly sophisticated, PPA has the potential to offer space in which to imagine new social relationships by questioning freely the dominant logic and weaving together digital objects, technological materials, and human agency symbiotically. In this regard, art systems centred on individual authorship fall short when it comes to describing a new dynamism that embraces the energy, matter, force, objectiles, and superjects that digitally mediated co-creation can bring about. In the consistent flow of the network, the conventional binaries of matter and form, subject and object, physical and digital, and artist and spectator co-emerge, co-concretize, and intra-act. Then, the new creative subject of PPA, or participant-superject, having emerged in the context of the shift in digital materiality and subjectivity, can radicalize the author-subject model and re-envision the agendas of open-ended and participatory aspects of art.

Superjective authors, then, with their mutual resonances, cause events and collective experiences to proliferate and add further from the postdigital environment. These authors are omnipresent and already part of humans’ ontological status. As Literat observed regarding many cases of online crowdsourced art, however, “the crowd is still a crowd, not yet a community” because the digital fibres and technological structures that mediate digital encounters can hinder efforts to achieve artistic collaboration or build a sense of community.²⁰ Thus, further studies are needed to assess the potential of PPA to invent meaningful communities apart from the profit-making algorithms of the various online platforms. 

References

1 Umberto Eco, "The poetics of the open work," The Open Work 251, no. 1., 1962.

2 Roland Barthes, "The Death of the Author," Contributions in Philosophy 83, [1967], 2001, 3-8.

3 Michel Foucault, "What is an Author?," in Textual Strategies, ed. J.V. Harari, Ithaca, Cornell University Press, [1969] 1979, 141-60.

4 Nicolas Bourriaud, Relational aesthetics. Trans. Simon Pleasance, Fronza Woods, Mathieu Copeland, Dijon, Les Presses Du Réel, 2002, Originally published as Esthétique Relationnelle, Dijon, Les Presses Du Réel, 1998.

5 Claire Bishop, “Introduction: Viewers as Producers,” in Participation, London, Whitechapel/Cambridge, MIT Press, 2006, 10-17.

6 Claire Bishop, “Social Turn: Collaboration and Its Discontents,” in Artforum 44, no. 6, 2006, 178–83.

7 Claire Bishop, Artificial Hells: Participatory Art and the Politics of Spectatorship, London, New York, Verso Books, 2012.

8 Robert Pepperell, Michael Punt, The Postdigital Membrane: Imagination, Technology and Desire, Bristol, Intellect Books, 2000.

9 Sy Taffel, “Perspectives on the Postdigital,” Convergence: the International Journal of Research Into New Media Technologies 22 (3), 2016, 324–38.

10 Brad Troemel, “Interview with Rob Walker, The Jogging, a Tumblr at the intersection of BuzzFeed, 4chan and weirdo experimental art,” Yahoo News, October 24, 2013.

11 Gilles Deleuze, The Fold: Leibniz and the Baroque, London, The Athlone Press, 1993.

12 David Savat, “(Dis)Connected: Deleuze’s Superject and the Internet,” in International Handbook of Internet Research, 11:423– 36, Dordrecht, Springer Netherlands, 2010, 432.

13 Alfred N. Whitehead, Process and Reality: An Essay in Cosmology. eds Griffin, David Ray and Sherburne, Donald W, New York, Free Press, [1929] 1978.

14 Marco Deseriis, Improper Names: Collective Pseudonyms from the Luddites to Anonymous, Minneapolis, University of Minnesota Press, 2015.

15 Bernard Stiegler, What makes life worth living: On pharmacology, Cambridge, Polity, 2010.

16 Gilles Deleuze, “Postscript on the Societies of Control,” October 59, Winter, 1992, 3–7.

17 Gerald Raunig, Dividuum. Machinic Capitalism and Molecular Revolution, Volume 1 (trans. Aileen Derieg), Los Angeles, Semiotext(e)/MIT Press, 2016.

18 David Savat, Uncoding the Digital, Basingstoke, Palgrave, 2013.

19 Marco Deseriis, “The Politics of Condividuality”, transversal.at., 2018, 7-10, accessed August 31, 2021, https://transversal.at/transversal/0318/deseriis/en

20 Ioana Literat, “The Work of Art in the Age of Mediated Participation: Crowdsourced Art and Collective Creativity”, International Journal of Communication 6, 2012, 2962–84.

Bibliography

Claire Bishop. “Participation and Spectacle: Where Are We Now?,” in Living as form: Socially engaged art from 1991-2011, ed. Nato Thompson (Massachusetts: MIT Press, 2012), 34-45. Kyle Chayka, “Art in the Corporatized Sphere: The Impact of Commercial Social Media on Online Artistic Practice,” in A Companion to Digital Art, ed. Christiane Paul (New York: John Wiley & Sons, 2016), 413–425.
Christian Fuchs, Digital Labour and Karl Marx (London: Routledge, 2014)
Ioana Literat, “Facilitating Creative Participation and Collaboration in Online Spaces: the Impact of Social and Technological Factors in Enabling Sustainable Engagement,” Digital Creativity 28 (2), (2017): 73–88.

Acknowledgements

The overall structure and conception of this paper draw on my PhD dissertation published in June 2022 

Proteus 3.0 - Interacting with the cloud

Maria Smigielska*, CompMonks*

mariasni.com; compmonks.com
Zurich Switzerland maria@mariasni.com, compmonks@compmonks.com *authors contributed equally to the project

Abstract

This paper presents the Proteus 3.0 project as an interactive and generative video installation, which tries to remain in conversation with its audience and learns from its engagement to produce new and potentially meaningful sequences of images. This particular version of a series is focused on digitized material as ferrofluid and its digital surrogate generated with a reaction-diffusion algorithm, projected onto a room-scale oval oculus allowing for a collective/immersive interaction. By employing state-of-the-art custom reinforcement learning models coupled with human intelligence in a symbiotic fashion, it aims to make the interaction more meaningful by understanding better human behaviour during an interaction. Additionally, it intends to maintain the interaction as conversations, rather than playful attractions. 

Keywords

Interactive art, generative art, reinforcement learning, computer vision, conversational interaction, mixed-media installation.

DOI

10.69564/ISEA2023-30-short-Smigielska-et-al-Proteus-3-0

Introduction

This paper presents the Proteus 3.0 project as an interactive and generative video installation, which tries to remain in conversation with its audience and learns from its engagement to produce new and potentially meaningful sequences of images. Proteus, as a series (Figure 1), has systematically evolved with its ferrofluid displays and modes of magnetic activation at multiple resolution ranges, particular sequences of digital and analogue layers as well as types of interaction. This version is particularly focused on moving the ferromaterial to the digital realm with its digital surrogate generated from a reaction-diffusion algorithm, scaling up from an object-size analogue display to an architectural oculus of oval shape at room scale, and which, at the same time allows to shift from an intimate interaction to a collective/immersive one. By employing state-of-the-art reinforcement learning models coupled with human intelligence in a symbiotic fashion, it also intends to build an interaction that is based on conversations, rather than playful attractions. Most frequent and straightforward approaches towards interaction design are based on implicit principles and clear instructions built in a closed loop to promote immediate engagement. These oversimplified views on interaction neglect its long-term impact¹ or the ones stemming from more traditional discourses, and which define any interaction with an artwork purely as a mental act.² In that lens, the project revisits the Conversation Theory of Gordon Pask³ in its scope of human-machine interaction as a form of open and emulative conversation. A dynamic process in which both ends learn and adapt from each other. As such, the project aims to construct an interaction that goes beyond the boundaries of predefined scenarios that follow the strict categories of an audience’s behavioral patterns and tends towards individualized and dynamic responses.

1. The evolution of the Proteus series between 2018 and 2022. ©various credits: Maria Smigielska, CompMonks, Ars Electronica 

Background Conversational interaction

What most generally describes successfully an interaction is the degrees of responsiveness and control an artwork and an audience are communicating to each other. An interactive artwork is essentially a composite made of an artificial component (the artwork itself as an object) and a human component (the audience) that both define its nature and behavior⁴. These degrees of communication are then located somewhat between total randomness and total predictability, without reaching any of these extremums. A total random response could only qualify as part of a reactive process, and a totally predictable one as part of a controlled manipulation. This does not necessarily involve a stable exchange and most frequently concurrent dynamics are at play from both ends as human intentions and artificially designed systems may have different and evolving goals over time. While this constitutes a great challenge for an artist to frame and produce such ideas, it also represents the richness and openness that an interaction may provide in qualifying an object. Interactivity is, by design, the relational property of an object and not an intrinsic one. It results that the value of such artwork is consequently constantly in a state of volatility and becoming, evolving throughout interactions. One might argue that this is necessarily the fate of every artwork, and more generally every object. But this becomes fundamental when interaction becomes itself an object of design in the artwork. Although, the most frequent and straightforward approaches towards interaction design are based on implicit principles and clear instructions built in a closed loop to promote immediate engagement and playful attractions. Similarly, most theoretical intent to define interactivity in art has been partially treating aspects of interactions that revolve around the topic of controllability. In general, the ability to improve control makes novelty the source of most types of interactivity⁵. In the general perspective of control, interaction can be seen as a learning stage in order to reach control. Once sufficient learning is achieved, interaction becomes redundant. Interaction becomes endowed with the feature of being instructional. However, even during the early stages of cybernetics, the theory of control was seen as only a chapter in the theory of messages⁶. Interactive art must be dissociated from the grasp of control theory [ref] in order to avoid any pessimistic perspectives⁷ or shortcomings in the envisioned artistic power of such artistic interventions. On the same level, Input/Output theory of interaction with computers⁸ simply defines interactions as a unilateral transactional process of information and therefore fall short on creative content. On another aspect, when interactivity is defined as procedural and participatory⁹, the definition lacks enough abstraction and generalisation to talk about the systemic design of an interaction and the informational novelty that go beyond the sole responsiveness of participations to an artwork. On another end, the evolution of machine perception in human computer interaction can be seen as trivialising the term of interactivity in new media a rts, and eventually rendering the sum of its tentative description as too loose to be useful to qualify an artwork¹⁰. This oversimplified views on the interaction neglects its long-term impact¹¹ or the one stemming from traditional art, which defines interaction with art purely as a “mental act”¹². It also resonates with the cognitive approach of the power of an artwork being located in the beholder’s response ¹³. 

Proteus 3.0 within the series of Protei

Proteus is a series developed as a duo since 2018 driven by the curiosity in exploring aspects of interacting with matter, tensions between analogue and digital, as well as relations between human and machine intelligences¹⁴. The series evolves through the continuous visual and material reference of a colloid compound called ‘ferrofluid’ as a main character ¹⁵. When exposed to magnetic fields and interacting with another liquid carrier, it reveals its intricate and lively nature by presenting strongly contrasted and ever-changing complex patterns ranging from discrete patches of dots, meandering streams, to larger coagulated blobs. The volatility and organicity of this material behaviour in the series is used as an allegory of the Greek mythology of a deity called Prōteus and his ability to assume many forms and provoke uncanny encounters ¹⁶ that remain evasive to the rational mind but open-ended to the visual imagination. Over the course of its serial declination, Proteus has iteratively evolved with varied organic and deep black patterns to serve as an emulator for visual human intelligence. That of seeking meaning in a constantly changing flow of images and unpredictable symbolic relations which can only be found in the human mind, just like gazing at a cloud. But the historical problem in such a romantic contemplation is often emphasized through the notion of distance and the lack of direct communication between a cloud and its beholders. If the cloud could talk back in response to one’s efforts to find visual meaning, would the conversation stabilize to a consensus, or would it constantly move to explore and seek novelty? This problem of distance and communication reveals itself when reframed in the artwork as a generative/interactive installation and its architectural embodiment. 

2. Proteus 3.0 installation during BioMedia exhibition at the ZKM Karlsruhe, 2022. © Photo: Thierry Serbeto.
3. 5 predefined interaction scenarios comprising of generative material visuals and behavioural patterns.

Methods

3. Diagram of the interactive loop. ©Image: CompMonks

Proteus 3.0 is an interactive generative 4K video installation, rear-projected onto a black screen of oval shape. The screen, of dimensions approximating 3.5 x 2.5 meters, is rotated 45 degrees and integrates an HD wide angle and infrared camera to estimate body and head poses from the audience and to detect regions of visual interest located on the screen. This information is continuously tracked and serves to build markers of attention to modify the generated visual frames (Figure 3).

Material surrogate

Because to this day, no digital model has been able to accurately simulate the physical behaviour of ferrofluid material in real time ¹⁷, a specific GLSL shader has been developed for the procedural generation of a digital surrogate. Based on a Gray-Scott reaction-diffusion model between two substances, a series of image filters have been designed as an overlay to mimic the dynamics of the ferrofluid and its pattern formations.

Our experiment was conducted in two following steps: (1) initiate predefined interaction scenarios run in the exhibition context to collect the visitors’ data, (2) building adaptive and evolving interaction models with reinforcement learning. The details of those two phases are described below.

Predefined interaction scenarios

At the first launch, the initial interaction consisted of 5 predefined interaction scenarios running in the 80 seconds loop. Interaction scenarios consisted of generative visuals coupled with the most common human behavioural patterns. The visitors' data responding to these scenarios were collected during eight months of the BioMedia exhibition at ZKM Karlsruhe, starting in December 2021.

In order to build Regions of Interests (ROI) of each interaction, skeleton data of max 6 visitors simultaneously were captured with a Kinect 2 camera to extract their torso and face orientation projected back on the oculus canvas. Each interaction has been recorded into a large dataset organized by timestamp with raw numerical information (visitors' ROI, skeleton data, location in relation to the screen) as well as low-resolution images (like RD noise map and visitors' ROI) for further training. The construction of datasets considers that any relevant personal information is decoupled from the recorded data.

Generative interaction scenarios with reinforcement learning (RL)

The scenarios further evolved into generative characters with the support of a machine learning model programmed in Python, specifically a reinforcement learning model, to learn from collected data and extracted markers of attention. It is being trained discretely offline and concurrently for a real-time generation to support the progressive evolution of the artwork that can adapt to different exhibition contexts and visitors' behaviours. The custom model has been developed in python with a PyTorch library.

The real-time performance of the work and the offline training are orchestrated by two combined processes. For real-time performance, a TouchDesigner file manages both visual inputs coming from the camera and generated outputs going to the display. The generated images are sent to two video projectors, mapped and blended for a smooth projection onto the screen. Daily, the recorded data is updated to a remote server and feeds new offline training sessions for the reinforcement learning model. The updated model is then sent back to the local computer controlling the installation to update Proteus' behaviour. 

Engagement

Engagement can be measured with simple metrics like the amount of time spent looking at art¹⁸ or more complex summative indexes, like Sweep Rate (SRI) and Diligent Visitors (DV) representing exhibitions “Thorough Use”¹⁹ to improve audience engagement in museums at large. However, those static metrics allow us to look at visitors’ response only through a pinhole and we do not know the exact impact of the interaction on the visitor (long-term, imaginary, etc). Therefore, we need more data collected during the interaction itself and more sophisticated tools to analyze them. “The Plant” project²⁰ proves that reinforcement learning can improve some of measures of engagement, like visitors touch count during interaction, but not average duration of the interaction. This project allows us to draw a hypothesis that AI-based tools combined with generative models can increase the quality of engagement and not just its quantitative factors like duration.

In the next developmental step of Proteus project, we aim to utilize the developed RL model in the future exhibition and conduct comparative studies to evaluate this hypothesis. 

Conclusions

Proteus 3.0 is an interactive and generative video installation, which, once in place within the walls of its exhibition and switched on, tries to remain in conversation with its audience and learns from the visitors' engagement to produce new and potentially meaningful sequences of images (Figure 2). Until the end of its exhibition, when it is switched off and set frozen in time. By then, its last learned state is saved and becomes the new generative beginning for its next exhibition. The learning and interactive process continues endlessly during the artwork’s lifetime for each exhibition and conditions its existence. Its generative imagery remains open-ended and its content to be modulated by its beholders. This project is an attempt to utilize AI-supported generative systems, which facilitate an interaction that goes beyond predefined behavioral models directly controlled by a few, easily comprehensible parameters and therefore might increase the engagement in the audience. 

References

1 Edmonds E, Candy L, “Interacting: Art, Research and the Creative Practitioner,” 2011.

2 Paul C, “Digital Art” Third edition, London, Thames & Hudson, 2015.

3 Pask G, “Conversation Theory Applications in Education and Epistemology” Amsterdam, Elsevier, 1976.

4 Margaret A. Boden, Ernest A. Edmonds, From Fingers to Digits: An Artificial Aesthetic, Illustrated edition, Cambridge, Massachusetts: The MIT Press, 2019.

5 “Aaron Smuts, What Is Interactivity?—PhilPapers”, 2023, Accessed September 15.

6 Norbert Wiener, The Human Use Of Human Beings: Cybernetics And Society. Revised ed. edition. New York, N.Y: Da Capo Press, 1950.

7 Terrence Rafferty, “Everybody Gets a Cut,” The New YorkTimes, May 4, sec. Magazine, 2003, https://www.nytimes.com/2003/05/04/magazine/everybody-gets-a-cut.html.

8 David Z. Saltz, “The Art of Interaction: Interactivity, Performativity, and Computers,” The Journal of Aesthetics and Art Criticism 55 (2), 1997, 117–27, doi:10.2307/431258.

9 Janet H. Murray, Hamlet on the Holodeck, Updated Edition: The Future of Narrative in Cyberspace, Updated ed. edition, Cambridge, Massachusetts: The MIT Press, 2017. 

10 Manovich Lev, The Language of New Media, MIT Press, 2002.

11 Ernest Edmonds, “Art, Interaction and Engagement,” In 2011 15th International Conference on Information Visualisation, London, United Kingdom, IEEE, 2011, 451–56, doi:10.1109/IV.2011.73.

12 Christiane Paul, Digital Art. Third edition. London, Thames & Hudson, 2015.

13 Eric Kandel, The Age of Insight: The Quest to Understand the Unconscious in Art, Mind, and Brain, from Vienna 1900 to the Present, 1st edition, New York, Random House, 2012.

14 https://proteusproject.ch/

15 https://en.m.wikipedia.org/wiki/Ferrofluid

16 https://en.m.wikipedia.org/wiki/Proteus

17 Han Shao, Hang Libo, Dominik L. Michels, “A Current Loop Model for the Fast Simulation of Ferrofluids,” IEEE Transactions on Visualization and Computer Graphics, 2022, 1-12, doi:10.1109/TVCG.2022.3211414.

18 Jeffrey K. Smith, Lisa F. Smith, “Spending Time on Art.” Empirical Studies of the Arts 19 (2), 2001, 229–36.

19 Beverly Serrell, “The Aggregation of Tracking-and-Timing Visitor-Use Data of Museum Exhibitions for Benchmarks of ‘Thorough Use.’” Visitor Studies 23 (1), 2020, 1–17, doi:10.1080/10645578.2020.1750830.

20 Zoe Tong, Dana Kulic, “Learning to Engage in Interactive Digital Art,” In 26th International Conference on Intelligent User Interfaces, College Station TX USA, ACM, 2021, 275–79, doi:10.1145/3397481.3450691 

Acknowledgments

Artwork by: Maria Smigielska, Compmonks
Created as part of the intelligent.museum project at the ZKM | Center for Art and Media Karlsruhe and at the Deutsches Museum. Funded by the Digital Culture Programme of the Kulturstiftung des Bundes (German Federal Cultural Foundation). Funded by the Beauftragte der Bundesregierung für Kultur und Medien (Federal Government Commissioner für Culture and Media). Funded by BioMedia ZKM | Center for Art and Media Karlsruhe.

Authors’ Biographies

Maria Smigielska is an architect and researcher based in Zurich, currently working at DBT, ETH. She is interested in enhancement of potentials for creation of architectural elements, design objects and mixed media installations, by using digital and interactive technologies for encoding and modulating materials properties, custom design and fabrication methods. Mariasni.com

Compmonks (PhD) is an architect and a researcher in technology for architecture based in Switzerland. He currently leads research on interactive and generative design with brain-computer interfaces at ETH Zürich. His artistic work follows a series of design objects and mixed-media installations focused on the power of combining humans with computers.

compmonks.com

Co-creation Towards the Post-Anthropocene

Pieter Steyaert1,2,7, Angelo Vermeulen3,7, Amy Holt7, Diego Maranan4, 7, Giusy Checola5,7, Mona Nasser6,7, Jeroen Verschuren7, Pim Tournaye7, Jesper Bruun2, Katrien Kolenberg1, 7

1 University of Antwerp, 2 University of Copenhagen, 3 Technical University of Delft, 4 University of the Philippines, 5 EDESTA-University Paris 8, 6 University of Plymouth, 7 SEADS (Space Ecologies Art and Design)
Antwerpen, Belgium pieter.steyaert@uantwerpen.be

Abstract

Co-creative practices including non-human actors give rise to a series of challenges and critical issues. Ēngines of Ēternity is an ongoing artwork which seeks to attribute agency to microscopic animals called rotifers and to the unique environment of outer space. Scientific experiments involving these animals, as well as the evolving artwork that accompanies them, have gone through multiple iterations having been flown to space and consequently returned. This article reflects on how transdisciplinary approaches can provide a vehicle to connect knowledge and enquiries from art and sciences in the context of real-world problems. We then focus on the case study of Ēngines of Ēternity and how these interactions played out during this process. We conclude the article with a critical reflection on non-human agency, using the aforementioned project as a case study.

Keywords

nonhuman agency, post-Anthropocene, co-creation, transdisciplinarity, ArtScience.

DOI

10.69564/ISEA2023-31-short-Steyaert-et-al-Co-creation

Introduction

The bidirectional exchange of knowledge, ideas and methodologies can foster co-creative practices and allow for a genuine blurring between disciplinary boundaries to occur.¹ Such holistic conditions may act as a catalyst for the synthesis of a deeper understanding with regard to the nature of the universe we inhabit and with it the emergence of a new post-Anthropocene era.² In order to achieve a sustainable future, we need to understand the mechanics and challenges of human and non-human interactions. This article explores and assesses the process of ArtScience co-creation, through a transdisciplinary project entitled Ēngines of Ēternity, acting as a case study. The work's narrative incorporates questions regarding manmade power structures and cultural immortality, through the use of non-human organisms, whilst highlighting the benefits and issues that can occur as a consequence. The author of this work and of this article is SEADS, a transdisciplinary and cross-cultural collective of artists, scientists, engineers and activists. The collective’s mission statement notes that transdisciplinary approaches are key to unlocking collective intelligence, which is a prerequisite for generating more diversified and inclusive futures.³ They incorporate co-creation methodologies in a range of their projects, amongst others are community art projects such as Bio-modd and Seeker.⁴, ⁵ The Ēngines of Ēternity project explores new forms of co-creation between different entities: humans, biological organisms and algorithms within the context of outer space. Before introducing the project, we outline the concept of creativity in the post-Anthropocene, which has been key to functioning as an underlying theoretical framework for this case study.

Creativity and Agency in the post-Anthropocene

If the Anthropocene is understood to be the geological epoch in which human activities are the dominant influence on the environment, the post-Anthropocene can be understood to be the epoch that dawns when human activities and their effects are no longer the dominant influence on the environment.⁶, ² The exact nature that the post-Anthropocene will take is difficult to predict because it depends on many factors, including how humanity responds to the challenges currently faced in the Anthropocene itself. On a global scale, we are not able to predict when and how the post-Anthropocene concept will become a dominant global framework, or indeed if this in fact will take place. However, there may be triggers that are capable of driving societies (both human and non-human) towards it, such as the ongoing global warming crisis and deterioration of the environment, societal cultural shifts, or technological advancements that may even allow humanity to live in greater harmony with Earth's ecosystems, but this is currently speculative. Merging disciplines can contribute to identifying and solving complex problems through the collaborative exchange of ideas and methodologies.⁷ Indeed, it has become apparent that many of the major problems facing the world today are hugely complex in nature and will not be solved by any one single discipline alone.⁸ Addressing complex societal and scientific problems requires involving multiple disciplines. However, there is a danger of oversimplifying the collaborative benefits between these disciplines. We argue that commonalities or filling up gaps can ignore or overlook the need to find new creative ways of dialogue between the disciplines, thus creating new knowledge or lines of inquiry that can transform the conversation on various global and scientific issues.

Differing from alternative approaches to merging disciplines, transdisciplinarity concerns both what exists, and is between and beyond all disciplines, blending into an empirical and theoretical exploration of co-creation.¹,, ¹⁰ One form of transdisciplinarity involves combining art and science (ArtScience) to create new co-creation practices that utilize methods and methodologies from both fields. Such art interventions can take place in present and future shared spaces and may contribute to the constitution of new symbolic orders and organisations of human and non-human connections, staging the mise en forme of post-Anthropocene co-existence. Seen in this way, ArtScience co-creation can be seen as an integral part of rethinking humanity’s relations to nature.

Bruno Latour argues that non-humans, such as animals, plants and objects, are active participants in social life and should receive equal attention in the study of society.¹¹ In this way of thinking non-humans should be seen as actors in their own right, rather than simply as objects that humans act upon; non-humans can have agency, and influence the course of events in the world. Latour’s conceptualization of agency does not strictly correspond with notions of sentience nor free will; neither does it have to be confined within physical boundaries. Agency can be an emergent property of multi-actant configurations.¹¹ According to Latour, social-cultural practices are enunciative and performative actions through which the new “subjects” can test different semiotic, sociological and alterity modes of existence, that lie between “being-as-being and being-as-others.”¹² In a similar vein, we refer to the notion of a “multispecies roundtable” for co-creating systems. This involves bringing together plants, computers, and people in mutually beneficial ways and placing them in a situation where they must arrive at a consensus.⁴, ¹³

Speculative Realism, Posthumanism, New Materialism, and the non-human are just a few examples of concepts that challenge the limits of what Donna Haraway calls the “fantasy of human exceptionalism.” These ideas reject the notion that humans are somehow separate, beyond, or more advanced than our earthly co-habitants.¹⁴ The theorists working within these fields help to illuminate the limits of human experience and thought, highlighting the capacities, experiences, and potentials of other living, non-living, and non-human entities. Their examples embrace a radically holistic and inclusive approach to the future of humanity and are established concepts in critical cultural thinking.

The Engines of Eternity project puts these ideas into practice by directly interacting with microscopic organisms, computational algorithms, and the physicality of outer space. These practical engagements hint at new opportunities and insights but also confront the limitations of an extended co-creation practice. Continuous hands-on approaches and their analysis allow us to move beyond established thinking and gain new critical insights through hands-on artistic research. 

Figure 1. Evolution and translation of the “code” and the application on the experiment label. © SEADS

Ēngines of Ēternity

Ēngines of Ēternity is a transdisciplinary project that explores how creativity and agency might function in a post-Anthropocene context. It is a work in progress and is intended to be a series of mixed-media installations. The project is a joint effort between SEADS and scientists from Karine Van Doninck's laboratory at UNamur. The foundational idea of the project is to take the biological phenomena of cloning and DNA repair as metaphorical departure points for an art installation about humanity’s obsession with cultural immortality.¹⁵ By metaphorical departure point, we mean that we use the fascinating biological characteristics of the rotifer animal to question creating a perfect society. At the core of the project is an artwork that evolves through complex systems of interplay and co-creation between biological organisms, humans, and technology. The project explores dominating ideologies about evolution and cultural identity within the space environment. The discussion on cultural immortality is discussed in a separate article³, as this one focuses on the methods and their implications that were crucial to developing the artwork. Here, we instead describe the background of our ArtScience co-creation activities: space biology experiments with the Bdelloid rotifer.

The Bdelloid rotifer or ‘wheel animal’ typically lives in freshwater environments and is part of the greater ecosystem that exists there. On the surface, rotifers seem to represent a biological culture perfected through evolution, cloning themselves in endless repetition. However, during this cloning process, genetic material actually gets reshuffled and diversity is generated through the process of horizontal gene transfer. This method of generating diversity can be defined as the non-sexual movement of genetic information between organisms, and genetic material from organisms as diverse as fungi, bacteria and plants have been discovered lurking inside the rotifer’s genome.¹⁵

Through a series of space biology experiments, scientists from UNamur sent rotifers to the International Space Station (ISS) in both 2019 and 2020. The goal of these experiments is to investigate rotifer DNA repair mechanisms and cloning in the conditions of outer space. In December 2019, the rotifers going to ISS were accompanied by an artistic "seed" in the form of a specially designed visual code that accompanied the rotifers. The code was developed by the Ēngines of Ēternity team and serves as a seed for a series of 3D-printed sculptures. These sculptures show the physical changes to the rotifers in response to environmental conditions and experimental techniques. The sculptures are based on a mapping of recorded experimental data 3D sculptures.

The work is driven by two main objectives: the evolution of the "code" throughout different missions and the translation of these seeds into three-dimensional sculptures. Analysis was conducted on both the bags that went up into space and the bags that served as a control group on Earth in order to reveal the differences between protein expressions. Each individual bag that was sequenced was assessed against the average from the control group bags, resulting in discrete sets per bag of large quantities of raw data. These sets were then grouped and ordered per function, accumulating the differences compared to the control group. The result was a visualization per bag showing the impact of the space environment on rotifer animals. These visualizations were then used to evolve the original visual "code" that accompanied these bags by using it as an attractor to manipulate the pixels of the original seed.

The second objective was to translate these seeds, both the original and the evolved version, into a three-dimensional sculpture. The glyphs that make up the seeds were transformed into a circle and a linear regression algorithm was applied to layer multiple versions of the original visual on top of each other and connect them. This resulted in a three-dimensional sculpture where the base visualises the overall shape and every layer shifts towards a higher level of detail. In this way, the final sculpture shows both the impact of the environment on the rotifer animals at a higher order and on the smaller, individual protein clusters.

The “code” and sculptures will gradually evolve into the final artwork over several missions based on the changes in the rotifers. Hence, the context of outer space is not merely treated as a passive gallery space to exhibit art in, but rather as a dynamic environment that can actively shape the evolution of the work.

Co-creation revisited Co-creation in Art and Science

The historical development of scientific and artistic disciplines has resulted in the development of organisations and structures in which these two areas do not organically engage or have dialogue. The organisational and procedural infrastructure that exists within these two fields is therefore sometimes seen as opposites. Science is often associated with the use of strict methodologies and systems to produce empirical data, unbiased information, and facts that help us better understand the physical and natural world around us. On the other hand, art is often viewed as a more subjective practice that allows individuals or groups to express their creativity and imagination.¹⁶ While these definitions are not entirely inaccurate, such reductionist viewpoints are unhelpful when beginning to ask deeper questions. ²³ Indeed, in recent times a variety of Art-Science networks have emerged that act as places where such communities can interact, and begin to create imaginaries for developing questions regarding our place in the wider world, whilst reaching large audiences. Interactions between the fields of science and art can be regarded as a one, or two-way, form of communication between the two. When considered as a one-way form of interaction, there are two possibilities for one to influence the other, either art provides science with some form of benefit, or vice versa.

The SEADS team developed tools using techniques and knowledge that differ from those of the pure scientific team. The goal was to gain insights into the impact of the alien environment of space on living creatures in an exploratory context. Initially met with skepticism, the collaboration with the pure science team evolved throughout the project. By communicating about our results, the team was able to establish new ways of trust and collaboration. The SEADS collective comprises members with backgrounds in both art and science tapping into collective intelligence. This allows them to push boundaries when collaborating within and outside of the network. For instance, in the Ēngines of Ēternity project, the artists received hard drives containing all the raw transcriptomic data to explore. This not only resulted in the aforementioned sculptures but also allowed the team to analyze their findings and compare them to those of the pure science team. It appears that new insights emerged from this transdisciplinary approach, which has yet to be validated and might lead to a future publication. In some ways, it is also possible to view the process of horizontal gene transfer as a biological counterpart to transdisciplinarity itself. The generation of novel genetic material through the incorporation of foreign DNA into the rotifers, from such phylogenetically distant organisms, echoes the process of numerous distinct disciplines joining together to create a more efficient system. Much like transdisciplinarity itself, it is not merely the merging of disciplines (or in this case genetic material) that makes this comparison applicable, but the fact that the process has wider implications on a much deeper level as well.

If transdisciplinary practices are considered to be a combination of, translational and methodological approaches from various disciplines, utilised to address a problem, then the horizontal integration of genomic elements from cross-species sources can be said to function in a similar manner. DNA segments once integrated into the rotifer’s genome can be translated into new useful proteins, that are capable of carrying out a variety of functions, and in turn, contribute to the evolutionary adaptation of the organism. The whole is greater than the sum of its parts, in both the case of transdisciplinary inquiry and biological evolution.

Figure 2. Sculptures and Bdelloid Rotifer organisms © SEADS and Boris Hespeels

Posthuman co-creation

A fundamental question in this project is whether true co-creation between humans and non-humans has been established, more specifically between the art-science team and the rotifers. How much agency did the rotifers actually have in participating and contributing to the artwork? In addition to exploring human-human interactions, we also investigate the agency of non-human entities, specifically the Rotifer animal. The evolution of the artwork is driven by changes in protein expression. However, this approach raises the question: are these non-human entities truly being given agency?

There are two ways to approach this question. One way is instrumentalist, which would argue that regardless of the artists' intentions, the rotifer animals are being exploited for both science and art. In fact, there is even an element of cruelty, as the animals are taken from their natural habitat and subjected to experimentation in an extreme environment without any choice in the matter. From a pragmatic standpoint, it is difficult to refute this way of framing the issue.

We present a second approach as follows: The artist team explores the intrinsic attributes of the rotifer animal and co-creates the artwork based on their findings. In this instance, horizontal gene transfer is viewed as a form of creation. By incorporating the protein expressions from these animals as the main driving force, and using complex bio-informatics algorithms, some level of agency over the artwork is given to these non-human entities. Extreme environments have been fundamental driving forces for the evolution of rotifers. However, in the case of the artwork discussed here, the driving force of its evolution is not solely an attempt to depict the intrinsic attributes of these animals. Instead, it also considers the impact of the alien environment of near-zero gravity in space on these extremophiles. This marks the first time in the history of the species that they have been exposed to the environment of space.

In this sense, the two levels of co-creation within this case study can be extended to include "The alien" as an environmental contributing factor. Additionally, the extensive use of algorithms and predictive computational methods has played a significant role in the visual outcome of the project. As a result, this project explores new forms and levels of co-creation between humans, biological organisms, algorithms, and outer space. Horizontal gene transfer and parthenogenesis are used not only as a metaphor, as was discussed in the previous section, but as an act of self-creation and self-transformation which instruments served as contributing factors in the formation of the artwork.

Conclusion

Co-creative practices including those involving non-human actors were explored through the creation of an ongoing artwork project entitled Ēngines of Ēternity. The nature of such transdisciplinary methodologies was investigated by a small community of artists and scientists involved in the project, whose evolving discourse during the process of developing and reflecting on the project has helped to provide a framework to inform conversations regarding the future of humanity, and whether and how we might move towards a more post-Anthropocene stage in the future. Establishing an ethical post-Anthropocene requires approaching the Other from a humble and empathic perspective. This case study provides insight into a practical approach towards this objective. For example, by focusing on the intrinsic qualities of creation present in the non-human agents. Nonetheless, it is evident that there are challenges that need to be addressed to successfully accomplish this goal. 

References

1 Erich Jantsch, "Inter-and transdisciplinary university: A systems approach to education and innovation", Higher education 1, no. 1, 1972, 7-37.

2 Erle C. Ellis, Anthropocene: a very short introduction, Vol. 558, Oxford University Press, 2018.

3 Angelo CJ Vermeulen, Diego Maranan, Pieter Steyaert, Nassim Versbraegen, Ann Peeters, Jeroen Verschuren, Arise Wan et al. "Ēngines of Ēternity: An Artistic Inquiry into Space Settlement Ideology Using Rotifer Experiments on Board the ISS," In 73rd International Astronautical Congress, 2022.

4 Pamela G. Cajilig, Diego S. Maranan, Arlene Sy, Oliver Salva, Angelo Vermeulen, "Multispecies Roundtable for Climate Impact: A Speculative Proposal," 2017.

5 Diego S. Maranan, Angelo CJ Vermeulen, "When ideas migrate: A postcolonial perspective on Biomodd [lba2]," 2015.

6 A. C. J. Vermeulen, Caroline Nevejan, Frances Brazier, "Seeker: Co-Creating Diversified Futures," Studio Time: Future Thinking in Art and Design, Black Dog Press, London, 2018, 172-182.

7 Van der Bijl-Brouwer Mieke, Giedre Kligyte, Tyler Key, "A co-evolutionary, transdisciplinary approach to innovation in complex contexts: Improving university well-being, a case study," She Ji: The Journal of Design, Economics, and Innovation 7, no. 4, 2021, 565-588.

8 Solomon Bililign, "The need for interdisciplinary research and education for sustainable human development to deal with global challenges," International Journal of African Development 1, no. 1, 2013, p.8.

9 Farzam Ranjbaran, Cristina Marras, "European Peer Review Guide: Integrating Policies and Practices into Coherent Procedures," ESF Member Organisation Forum on Evaluation of Publicly Funded Research, 2011.

10 Bruno Latour, Reassembling the social: An introduction to actor-network-theory, Oup Oxford, 2007.

11 Bruno Latour, "On technical mediation," Common Knowledge 3, no. 2, 1994, 29-64.

12 Bruno Latour, Enquête sur les modes d'existence: une anthropologie des modernes, Paris, La découverte, 2012.

13 A. C. J. Vermeulen, "Living computers, Mars simulations and DIY Starships: Advancing cross-disciplinary and cross-cultural collaboration," Link 3 Workshop Space, Plymouth University, 2016.

14 Donna J. Haraway, When species meet, Vol. 3. U of Minnesota Press, 2013.

15 Matthieu Terwagne, Emilien Nicolas, Boris Hespeels, Ludovic Herter, Julie Virgo, Catherine Demazy, Anne- Catherine Heuskin, Bernard Hallet, Karine Van Doninck," DNA repair during nonreductional meiosis in the asexual rotifer Adineta vaga", Science advances 8, no. 48, 2022, eadc8829.

16 Norman G. Lederman, "Nature of science: Past, present, and future," In Handbook of research on science education, Routledge, 2013, 831-879.

17 Jonathan Osborne, Collins Sue, Mary Ratcliffe, Robin Millar, Rick Duschl, "What “ideas-about-science” should be taught in school science? A Delphi study of the expert community," Journal of research in science teaching 40, no. 7, 2003, 692-720.

Acknowledgements

This project is funded by the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant No 860470.

Author Biography

SEADS (Space Ecologies Art and Design) is a transdisciplinary and cross-cultural collective of artists, scientists, engineers and activists. Its members come from all corners of the world. SEADS is actively engaged in deconstructing dominant paradigms about the future and develops alternative models through a combination of critical inquiry and hands-on experimentation.

SEADS employs its own signature methodology which is centered around community building, co-creation and bottom-up design. SEADS believes that these approaches are key to unlocking collective intelligence, a prerequisite for generating more diversified and inclusive futures. Furthermore, SEADS also embraces a hacking and open-source ethos, with the goal of engaging as many people as possible in the activities and ideas that they initiate. Since 2009, the collective has co-created more than 40 art projects, together with local communities all over the world. 

Spectral Plain: A case study for exploring the world-building potential of co-creative systems that combine text generation models with game mechanics.

Guillemette Legrand1*, Vincent Thornhill2*, Isaac Clarke3

1École des Arts Décoratifs – PSL (EnsadLab), University of Applied Sciences and Arts Northwestern Switzerland (IXDM) 2KU Leuven, LUCA School of Arts, 3Hong Kong University of Science and Technology
1
guillemette.legrand@ensad.fr, guillemette.legrand@fhnw.ch, 2vincentsimon.thornhill@kuleuven.be, 3ijclarke590@connect.hkust-gz.edu.cn
*These Authors contributed equally.

Abstract

This paper describes the game installation ‘Spectral Plain’; introducing a case study of an interactive artwork that intersects algorithmic, sensing, and gaming technologies to explore new forms of generative world-building, which are different from the world-imaginations created by commercial and political narratives. Through this case study, we aim to explore how information technologies might embed specific socio-cultural beliefs. Here, we investigate the imaginaries created and disseminated through dominant representations of information technologies, while simultaneously searching for ways to create new multiscalar imaginaries of the earth through the mobilisation of these technologies. This case study describes how physical and digital interactions can be combined with AI and randomising technologies to foster forms of co-creation with and between humans. It also explores the capacity for AI-integrated game environments to decenter anthropogenic perspectives by creating new symbiotic interrelations with the non-human and the non-living that can impact participants’ perception of their understanding of the world.

Keywords

World-building, cosmologies, game mechanics, interactive installation, autoregressive language model, co-creative system.

DOI

10.69564/ISEA2023-32-short-Legrand-et-al-Spectral-Plain

Introduction

This project started with an intention to examine, and communicate the capacity of information technologies to construct imaginaries of the "world." The initial technological context for this investigation was the commercial and political representation of 5G wireless technology; the latest in a lineage of "network" imaginations of a world coinciding with the development of communication technologies. Professor of Anthropology Shannon Mattern argues for a need to acknowledge how marketing fantasies and policy scenarios have limited our imaginations, not just of what the technology is, but also of how we organise and live in the world through these imaginations.¹ Philosopher of Science John Tresch scales Mattern’s concerns to the imagination of the planet, stating that “when modern technology is linked to a ‘cosmology’ or a ‘world(s) view,’ we are offered a much thinner picture than that suggested by anthropology’s presentations of the ways humans have organized nature and society—a ‘mechanization of the world picture’ simplifying all experience to utilitarian building blocks of masses and forces, functionalist means and ends.”² Between the marketing and political narratives of 5G, and the technical operations of this infrastructure, we saw the potential to build a new reading of wireless technology.

This rereading was pursued through decentering wireless technology from the regulatory tradition of allocating and auctioning ‘bandwidth’ of electromagnetic frequency to commercial and geopolitical actors. Instead, we wanted to reposition this technology by placing it within a wider spectrum of frequencies that human and non-human entities are intertwined with. Imagining electromagnetic energy in new scales and alignments prompts us to think at a cosmological dimension, a concept derived from Timothy Morton’s description of a "Spectral Plain;" an infinite surface where “you can’t really distinguish very easily between alive and not alive, between sentient and non-sentient, between conscious and non-conscious.”³ Through enriching and diversifying the comprehension and representation of electromagnetic frequencies, the project aims to open up new imaginaries of the earth.

 

Research Process

The research process centred on testing ways to produce narratives about electromagnetic energy that cross between human and nonhuman scales, from the atomic, to the terrestrial, to the cosmic. To do this, we created a series of working sessions through which the project was developed through participation with varied audiences and formats, while iterating the development of technological systems for generating narratives. These experimentations took the shape of online and physical workshops and interactive performances that were set up to test different forms of collective world-building; methods that were later implemented in the final artwork.

We used the game environment as a site for experimentation, which enabled us to test ways that various modes of representation and points of view could coexist or create friction. It also allowed us to play with multi-scalar experiences; from micro to macro, from the local (storing of individual data about the player locally) with the planetary (data processed through large distributed algorithmic models). Through recombining different information technologies, we created an experimental ground for a collective to engage with world-building. We use the exhibition periods as a moment to present the work to a public, but also a moment to extend our research through further testing of the system we created, while collecting additional feedback on the players' experiences.

Figure 1. Screen capture of the research process that led to the development of Spectral Plain. Here you can see a working session where participants were guided through a game environment, while collectively creating stories that reflected their impressions of the journey. Credit: the artists.

Of the different technologies involved, our research into the use of GPT-3 had the most impact on how we were able to reconsider the ways collective world-building processes could be constructed in co-creation with AI technology. Several games have explored the use of GPT-3 to generate open-ended narratives, such as AI Dungeon.⁴ However, rather than using this technology to create divergent storylines, we sought to use the intrinsic properties of this linguistic model to establish connections and patterns between different bodies of knowledge about electromagnetic energy.

Project Description

Our artistic research into co-creative systems through the medium of the game environments led us to develop the artwork as an interactive game-installation, called Spectral Plain (figure 2).

In this interactive installation, three players can navigate three levels of an environment (the skies, the terrestrial and the underground). The game story takes place in a period following a fictive geomagnetic reversal (a change in the planetary magnetic field), where ‘spectral energy’ is unleashed, and players are engaged in discovering new perspectives on electromagnetic energies.

Figure 2. Image of the physical installation of Spectral Plain. The three layers of the game are projected on top of each other. A diagram representing the ‘back-end’ of the installation is legible on the ground. Three chairs were specifically designed to position player’s view towards one of three projections. Credit: the artists.

The installation uses a real-time data capture and a text-generation algorithm (GPT-3) via an integrated blueprint in the game engine (Unreal Engine) to log in various datum (time stamps, players' statement selection), which is then used to inform the final audio-visual cosmogram.

The three levels of the game are developed in a systematic way with each environment being constructed around five “spectra” of electromagnetic frequencies (figure 3) that are named: aqua, light, geology, matter, and device. These frequencies manifest as different entities within the game environment, for example, the light spectra manifests as chlorophyll in the terrestrial level.

Figure 3. Visual symbols for the 5 different typologies of energetic entities. Credit: the artists.

The presence of the three players in the physical environment of the installation creates a unique disturbance in the gamma radiation that is present around the installation. In the game infrastructure, this disturbance triggers the Geiger counter to produce a "true random number." This number is then used within the game engine blueprint to randomise the graphics that the players will interact with and the parameters within GPT-3.

The game environment is developed to give agency over the creation of a new world-imaginary of spectral energy that is different from the dominant 5G narratives. Here, we align with philosopher Federico Campagna’s position towards the concept of "world;" not using the word to describe an ontological reality, but rather seeing the world as an artificial construct of the imagination, “a metaphysical process that creates discontinuities, separations between things, individual identities, ‘somethings’ out of a plane of pure existence where no clear divisions are already inscribed.”⁵

At the end of the players’ journey, a cosmogram is automatically generated (figure 4). This cosmological diagram is constructed by gathering data about the players’ movement and interactions within the levels of the game. From this cosmogram, unique cosmological readings of the unified journey are then created through fine-tuning the GPT-3 language model. These readings are then "spoken" through the use of speech synthesis software. In this process of co-creation between player and algorithms, we are interested in how automated AI technology can be used to generate connections between diverse perspectives and narratives of energetic entities (scientific, mythologic, commercial narratives). The reading is not an attempt to accurately describe relationships between these perspectives, but rather, to create an artistic and lyrical intervention.

Figure 4. A screen capture of the final visual cosmogram representing the intersecting journeys of the three players. Credit: the artists.

Technical Realisation

In the Spectral Plain interactive game-installation, we combine physical and virtual player interaction, together with a combination of predictive (GPT-3) and randomising information technologies (Geiger counter radiation sensor as true random number generator (TRNG). The input and output of these different interactions and technologies are linked and moderated through Unreal Engine’s Blueprint Visual Scripting system.⁶ The node-based interface of this system allowed us to map and program interactions in the back-end of game creation software in a visual manner.

At the intersection of cosmology, and generative technologies, we also became interested in logic and structure of different divination techniques from different cultures. In Spectral Plain, we develop an analogy to techniques such as birth chart readings, and Yi Ching, by approaching their logic as a form of programming. In our technical realization of the project, processes of symbolisation, inclusion of chance, and geometric calculation can be seen as referential to ancient divination techniques that link the individual to the planetary.

Figure 5. Flow chart of the step-by-step interactions with the game installation. Credit: the artists.

The section attempts to walk through the different steps taken in the process of playing Spectral Plain by annotating the diagram (figure 5 below):

The text is then processed by a voice synthesiser to produce an audio file that is played when the players reach the visual cosmogram that is automatically inserted into the concluding environment of the game.

Conclusion

Spectral Plain is an interactive game-installation that builds upon concepts present within ancestral technologies (birth chart construction, Yi Ching reading instruments, etc.) through the use of current technologies in order to enable a space for co-creation, where symbiotic interrelations between disciplines (art, geoscience); the non-human (e.g., glowing micro-algae, chlorophyll photosynthesis); and the non-living (e.g., AI, geological elements) can occur. In this technical and conceptual description of the work and its intended effects, we aim to contribute to the discourse surrounding the ontologies of information technologies. This paper also aims to create a case study to allow for continued questioning of how artistic research can challenge, mobilise, and enrich the imaginary potential of these technologies.

By advancing the possibilities to engage with and through narrative-based technologies within game-environments, we follow Alenda Y. Chang’s proposition to explore "the untapped potential of games to create meaningful interaction within artificially intelligent environments, to model ecological dynamics based on interdependence and limitation, and to allow players to explore manifold ecological futures—not all of them dystopian.”⁷

The project requires further technical and conceptual development in order to explore ways to balance and represent generative-based components, chance factors, and collective agency within the game-narrative. It is also important to question the ability of large language models like GPT-3 to generate a truly novel story, and find ways to negotiate with its capacity to homogenise culture and beliefs through language parsing and recomposing.

References

1 Shannon Mattern, “Networked Dream Worlds,” Real Life , accessed December 7, 2022, https://reallifemag.com/networked-dream-worlds/

2 John Tresch, “Technological World-Pictures: Cosmic Things and Cosmograms,” Isis Vol. 98, No. 01, accessed December 7, 2022, http://dx.doi. org/10.1086/512833 (online)

3 Timothy Morton, Being Ecological, Cambridge Massachusetts, MIT Press, 2018, 126.

4 Latitude “AI Dungeon (2019),” accessed December 7, 2022, https://play.aidungeon.io

5 Federico Campagna, Prophetic Cultures, Recreation for Post-Future Adolescents, Bloomsbury Publishing, 2021, 13.

6 Unreal Engine, “Anatomy of a blueprint (2022),” accessed December 7, 2022, https://docs.unrealengine.com/4.27/en-US/ProgrammingAndScripting/Blueprints/Anatomy/

7 Alenda Y. Chang, Playing Nature: Ecology in Video Games, University of Minnesota Press, 2019, 16. 

Ecotechnologies of Practice: in-forming changing climates

Gisèle Trudel

École des arts visuels et médiatiques, Université du Québec à Montréal. MÉDIANE Canada Research Chair in Art, Ecotechnologies of Practice and Climate Change. Montréal, Canada.
trudel.gisele@uqam.ca

Abstract

How do ecotechnologies of practice actualize? This paper traces the material/theoretical operations of an ongoing research-creation concerned with changing climates. It mixes information in experimental approaches from collectivities of trees, media arts and forest sciences. Through individuations of symbiotic modulations, the paper is a thinking-with Balsam Fir, Diana Beresford-Kroeger, Camera, Domingo Cisneros, Dendrometer, Erin Manning, Isabelle Stengers, Jack Pine, Gilbert Simondon, Light Emitting Diodes, Numbers, Microphone, Recorder, Sapflow, Sensings, Speakers, Sugar Maple, Temperature, Yellow Birch.

Keywords

Research-creation, changing climates, trees, ecotechnologies of practice, information, individuation, art/sci/phi.

DOI

10.69564/ISEA2023-33-short-Trudel-Ecotechnologies-of-Practice

Figure 1. November 2020. Out in the field at the Smartforests Canada research site (Sainte-Émélie-de-l'Énergie, Québec). L to R: Ælab (Gisèle Trudel and Stéphane Claude) and forest engineer Christoforos Pappas. Photo: Zoé Fauvel.

Differential interdependence

An interdisciplinary sharing is at work. Ælab joins forest engineer Christoforos Pappas who is explaining to them how to attach a sapflow sensor to a tree in the boreal forest (Figure 1). The tree already senses with the surroundings, the sapflow sensor brings the tree's operations into human understandings, as sociologist Jennifer Gabrys has discussed.¹ Inter-cross-transdisciplinary approaches activate philosopher Isabelle Stengers' concept of “ecologies of practice” in modern science.² To move outward of disciplinary silos, she states openness is required while still maintaining tension, to question how practices produce knowledges (plural added; savoirs in French). Stengers argues for knowledges production in interdependence, within the practices of each scientific field, and in-between them. Each field can contribute a differential worldview to one another, a distinguishing plurality.

Scientific fields work actively to try to solve climate changes. Yet, what else can occur in-between the sciences, trees, art and machines? There is an important increase of sensual and affective artistic explorations with trees, among others by Jane Tingley, Domingo Cisneros, Susan Turcot, Rasa Smite and Raitis Smits, Agnes Meyer-Brandis. In this research-creation project, scientific and tree knowledges are expressed in outdoor media art installations. How can knowledges and practices resonate together, within and through a multitude of fields, given the concerns for the “more-than of life-living” of the blue planet?³

Figure 2. May 2021. Tree and machine sensings at Station biologique des Laurentides, Saint-Hippolyte, Québec. Oxygen and CO2 in-form the living for millennia. Photo: Gisèle Trudel.

Experimental in-forming

The concepts of information and individuation offered by Gilbert Simondon provide insight for the encounter of ecologies of practices.⁴ In his philosophy, information is more-than what is generally considered as factual data. Information is operation. It activates change through perturbation, by provoking or inducing an encounter between disparate elements/factors, as event. Potential is propelled into actualization through a dynamic play of forces. This is of particular importance in the context of climate change where fearful and helpless affects predominantly circulate. How else does change occur?

Figure 3. Ælab, bois eau métal, July 2021. Outdoor media art installation with sound, LED tiles, scaffolding, weather kit. Espace pour la vie - Jardin botanique de Montréal. Photo: Richard-Max Tremblay.

The first outdoor media installation by MÉDIANE to explore the potential of changing climates is bois eau métal (2021), an artwork concerned with the ecophysiology of three trees, with numerical data from Smartforests Canada research site, including immediate weather sensing. Presented in the Arboretum, the scaffolding structures are a site of temporary co-constructions (chantier), mini pavilions without walls for experiencing three types of immersive sounds setups, video imagery and graphic visualizations. Tree data from a forest is co-composed with colors from a custom-made weather kit. The whole manifests as rhythms of physical processes in feedback loops. This installation acts as information, pushing the exploratory differential encounters between fields of knowledges and experiences in a shared reality with trees. Stengers analyzed science domains which often perpetuate their own methods of safeguarding knowledges production. Contemporary art often relies on the unwavering codes of modernist heritage to produce and present artwork as autopoiesis, production of self “in and of itself”. To promote changes, in Stengers' words, “this is the reason why, through the exploration of knowledges, there is also a type of ethical experimentation” (our translation).⁵ Respecting and challenging, one and other. Shifts occur, along with the tree's own experience; despite the tree being rooted in one location, seemingly immobile, the sensor data shows the tree is constantly moving and changing at microscales.

Figure 4. Ælab, bois eau métal, July 2021. Outdoor media art installation with sound, LED tiles, scaffolding, weather kit. Espace pour la vie - Jardin botanique de Montréal. Publics are invited to participate in semi-directed interviews, engaging with their experience of the artwork. The analysis of these interviews influences the production of the following installations in the iterative series. Photo: Caroline Pierret. 

Changing climates warrant to be addressed as a “problem” in and of change, within research fields and in their differential gathering, and in dialogue with diverse publics. How can ecotechnologies of practice address the problem of changing climates, understood here as an obstacle for thought in and into action, rooted generally in fear, anxiety and helplessness? This is an obstacle needing new thought, a coproduction of knowledges in practice, in experience. Stengers and Simondon provide compelling offerings to think-with and act-with problems, in experimentation. In Simondon's concept of individuation, no individual (human, instrument or otherwise) preexists to the encounter. Individuation occurs through the meeting of forces, garnered little-by-little and in proximity, gatherings that build and change in time. Propelled into change by the perturbation of information, triggered by the tree's mode of living, change happens. For better or for worse, without moral imperative, it is a propositional ongoing, a doing, building with each tiny difference.⁶ This approach is core to MÉDIANE's ongoing research-creation process, whereby each installation proposes change, building with the tree, data gathering, analysis, and by the gradual spreading of knowledges production and circulation in-between trees, forest science, out-of-doors art and publics.

An orientation, not a beginning

The slogans of the FridaysForFuture global strike movement (FFF) repeat that governments, corporations and publics need “to listen to the science, to listen to the scientists.”⁷ Scientists are directly convened to address climate change, outside of the silos of fundamental knowledges production. Undeniably, scientists are needed and respected to do so. However, it points to the domination of science as holding exclusivity to resolve climate emergencies. Peoples (plural) have perhaps given over climate change issues to science and to government to solve, forfeiting distributed agencies. How can a combination of practices contribute to new actions? With Stengers, it becomes possible to extend the ecologies of practices in scientific realms to a mixing of approaches by ways of practice, in-between collectivities. A new knowledges circulation, a thinking-with trees, machines, materialities, artists, scientists and publics, even if only temporarily. This problem for thought in/to action, perturbed by climate changes, is addressed by ecotechnologies of practice. In Figures 5 and 6, the 2022 media installation entitled Orée des bois addressed these questions through the study of the birch tree's phenology, the seasonal changes from Spring to Fall which affect growth and connections with flora and fauna. 

Figure 5. Ælab, orée des bois, June 2022. Outdoor media art installation with sound, LED tiles, scaffolding, weather kit. Presented at Jardin du Coeur des sciences, UQAM. In this instance of the installation, an arch was created between triangular plots of land, community city gardens. The scaffolding is a physical drawing vibrating in spacetime with a multitude of experiences. Publics circulated between and under the structures, traversing the site to another zone with scaffolding tables of different heights. Additional activities onsite included workshops and presentations about medicinal plants, sensory ethnography, cartography, performances, conversations with drawings on birch bark, etc. Photo: Alexis Bellavance. 

A problem of & with & in practice

For Stengers, ecology is always double: scientific and political (italics added).⁸ This doubling can be activated in contemporary art, design and well as social sciences, charting the trajectories of actants. In Greek, technè is art, skill, and craft or the method employed resulting in an object.⁹ Logos tries to lay claim to (human) language, to “collect, gather, in derivatives of word, speech, statement, discourse, computation, account.”¹⁰ To consider technology only in a prepotent human way provides humans with vector to dominate and control, to justify the means to an end, often neglecting entangled elements/lives/factors. Technology understood in this dominant way severely limits approaches, because if it is seated only in the human as an exclusive semiotic rationality, it prevents or even opposes new informations to coalesce or collide with more-than and other-than. It is pertinent to note that in sustainable architecture, a definition for ecotechnologies is holistic systems or machines with the environment, including reuse, passive housing, renewable energies, crade-to-cradle designs.¹¹

Figure 6. (detail). Ælab, orée des bois, June 2022. Outdoor media art installation with sound, LED tiles, scaffolding, weather kit. The project focused on the Spring to Fall seasons of a birch tree through alterations of time-lapse sequences of photographs of the tree from camera placed directly on the tree, mixed with visualizations of inner workings (on left). The slits between the LED tiles accentuates embeddedness in physical site's activities, along with rain, wind, city noise, etc. Photo: Alexis Bellavance. 

Simondon proposes another technology, neither thing nor method fashioned by human will. In his philosophy, individuation is actualized through “phases” of relations between physical environments, instruments and humans. Technology is in-forming, changing, extending another expression of logos, the cosmos expressing its manifold in movement with animals, minerals, plants¹². A “life-living.”¹³ It is also the combined strengths of trees, scientists, artists and publics in changing climates.

This ongoing research-creation proposes ecotechnologies of practice in a crafting of “response-abilities.”¹⁴ The tree crafts problem-solving operations with the surroundings: tackling water preservation or evapotranspiration as needed, regulating sap flow, making clouds, coordinating burgeoning and senescence timings, going dormant in northern winters, turning sapflow into antifreeze to prevent embolisms.¹⁵ Furthermore, trees produce photosynthesis in a cosmic relation of sun and soil, they yield oxygen and even grow stronger and bigger with the extraction of carbon dioxide from the atmosphere.¹⁶ They communicate through VOCs (volatile organic compounds) and mycelium networks.¹⁷ These are but a few of the tree's wondrous techniques, parts of a larger continuum to coevolve their technologies over 350 million years.¹⁸ To partake gently in their knowledges, Indigenous artist Domingo Cisneros proposes eating and drinking parts of coniferous trees without harming the plant, a celebration to share in the strength of the longevity of their problem-solving techniques.¹⁹

Figure 7. Ælab and CREAF. Les trois frères (2020). Documentary concerned with forest gastronomy of the “three brothers”: spruce, fir and cedar. L to R: Stéphane Claude, Domingo Cisneros, Antoinette de Robien. Photo: Gisèle Trudel.

In a similar celebratory fashion, medical biochemist, botanist, climate activist and author Diana Beresford-Kroeger is as adamant as she is hopeful in times of changing climates. Her “Bioplan” makes simple statements. She advocates to protect trees in the immediate vicinity as well as native forests in local communities.²⁰ So efficient, it's alarming. Her approach changes the habitual urgent call associated with resiliency and adaptation of diverse communities that are foregrounded through catastrophic discourse. Change—with life-living trees—is summoned as operation. As Stenger states, consensus is stasis, refusing (to) change. Beresford-Kroger's bioplan extends symbiotic intimacy with tree technologies.

Ongoing Doings

Ecotechnologies of practice entail a perturbation in the habitual conception of the relation of ecology and technology, explored here as operative process. The tree's ecological practice, in conjunction with that of arts and sciences, investigates changing climates with creative problem-solving in dialogue with publics. The differential fields of knowledges bring new expression by their combination, a strength emerging in-between, pushing forth modular alliances. Crafting changes is ecotechnologies of practice: even if ephemeral, potential is actualized through encounters, soliciting variation and individuating collectivities. 

Acknowledgements

The research-creation is conducted on the unceded lands of the Kanienʼkehá ka Nation. The MÉDIANE team gratefully acknowledges funding from SSHRC, CFI, FRQSC and support from Hexagram-UQAM.

References

1 Jennifer Gabrys, Program Earth. Environmental Sensing Technology and the Making of a Computational Planet, Minnesota, Minnesota University Press, 2016.

2 Isabelle Stengers, La guerre des sciences, Cosmopolitiques 1, Paris, La Découverte, 1997.

3 Erin Manning, Out of the clear, Colchester and New York, Minor Compositions, 2022.

4 Gilbert Simondon, L'individuation à la lumière de notions de forme et d'information, Grenoble, Jérôme Millon, 2005.

5 Isabelle Stengers, La guerre des sciences, 8.

6 Gilbert Simondon, L'individuation à la lumière de notions de forme et d'information.

7 Accessed November 4, 2022, https://fridaysforfuture.org.

8 Isabelle Stengers, La guerre des sciences, 58.

9 Accessed November 6, 2022, https://www.oxfordlearnersdictionaries.com/definition/english/technology.

10 Accessed November 6, 2022, https://www.etymonline.com/word/-logy?ref=etymonline_crossreference.

11 Accessed November 6, 2022, https://leverageedu.com/blog/ecotechnology/.

12 Carl Sagan, “Cosmos (1980),” IMDb website, accessed November 1, 2022, https://www.imdb.com/title/tt0081846/characters/nm0755981#quotes.

13 Erin Manning, Out of the clear.

14 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham, Duke University Press, 2016.

15 Jehová Lourenço Junior, Community Ecology - Wood Anatomy - Plant Physiolog, Accessed November 3, 2022, https://jehovajr.wixsite.com/science/post/the-hydraulic-architecture-of-trees-in-a-climate-changing-scenario

16 As Carbon Dioxide Grows More Abundant, Trees Are Growing Bigger, Study Finds, Accesssed October 10, 2022, https://e360.yale.edu/digest/carbon-dioxide-climate-change-bigger-trees

17 Suzanne Simard, Finding the Mother Tree, Allen Lane, Penguin Random House Canada, 2021.

18 Serge Muller, Germinal Rouhan, “Comment les arbres sont apparus sur Terre,” Museum, November 30, 2021, accessed November 1, 2022, https://www.mnhn.fr/fr/actualites/comment-les-arbres-sont-apparus-sur-terre.

19 Ælab and CREAF (Centre de recherche et d'expérimentation des arts forestiers), “Les trois frères (2020),” https://vimeo.com/channels/472281/452356412).

20 Accessed November 15, 2022, https://dianaberesford-kroeger.com/mission/. 

Author Biography

Gisèle Trudel and Stéphane Claude founded the experimental documentary artist research unit Ælab in 1996 (aelab.com). Collectivities actualize the potential of MÉDIANE (2020-2025, mediane.uqam.ca).

Water Stories: Visual Poetics and Collective Voices

Andrea Wollensak, Brett Terry, Bridget Baird

Ammerman Center for Arts & Technology, Connecticut College Sonalysts
Emerita Computer ScienceDepartment, Connecticut College New London [authors 1 and 2] and Waterford [author 3] CT,USA
ajwol@conncoll.edu brettcarrollterry@gmail.com bbbai@conncoll.edu

Abstract

Water Stories: Visual Poetics and Collective Voices is a two-part project that brings together multiple points of view from local youth, community, and poets in Alaska to share what water means in their life. Visual Poetics combines a live poetry reading by Alaskan poets and interactive video in which the poets’ voices trigger generative visual elements. Collective Voices is a sound work featuring excerpts of community voices sharing water-based memories against a backdrop of processed environmental sounds of Alaskan waterways. Water Stories is part of a year-longartist residency (2021-2022) with the Anchorage Museum culminating in a series of listening sessions broadcast at the Anchorage Museum and Out North Radio, live interactive poetry readings at the museum,and video projections on the museum façade from November 2022 through January 2023.

Keywords

Generative, digital poetics, audio-reactive software, community engagement, collaborative, ecosystems,resilient societies, multiple scales.

DOI

10.69564/ISEA2023-34-short-Wollensak-et-al-Water-Stories

Introduction

Alaska’s climate is projected to warm several degrees by the end of the century, more than twice the warming compared to the rest of the country. The record-breaking temperatures experienced by Alaska communities have resulted in thawing permafrost, thinning sea ice, and more wildfires. The Water Stories: Visual Poetics and Collective Voices project engages the Alaskan public to consider how climate change alters their experiences of the land, and to consider the role of water in their lives.

Background

Water Stories is the most recent artistic collaboration between the authors, and is part of aseries of works focused on climate change and the environment. Reading the Wrack Lines (2021),an outdoor audio video projection on a lighthouse, featured community-sourced textual reflections on climate change. Open Waters (2020) was a multiple-media installation exhibited at museums in Rhode Island, and New York, connecting the five-hundred-year imaginative history of the open polar sea and northwest passage with current geopolitics and arctic climate change. Another previous audiovisual collaboration, Ice Core Modulations: Performative Digital Poetics (2017) included generative imagery and poetic texts controlled via historical CO2 data taken from ice core samples made available from the National Snow and Ice Data Center (NSIDC). All three of these works have been presented at past ISEA Conferences.

These collaborations have sought to create engaging works that address complex environmental issues through a plurality of artistic, poetic, and scientific perspectives. The collaborators have brought to bear individual expertise in audiovisual generative computer programming, visual art media, poetry, and electroacoustic sound composition to synergistically create works unified by shared source materials that include place-based personal narratives, historical source material, raw scientific data, local audiovisual media collected on site, and original poetic texts.

Goals, Objectives and Artistic Dimensions

Water Stories seeks to amplify voices of Alaska residents through storytelling, poetry, and local landscape imagery. The overarching project goals and objectives include:

Figure 1. Videostill of Ruckle’s poem “Ferry” in audio-reactive projection

Ecosystems of Collaboration

There are two synergistic types of collaboration taking place within the WaterStories work. In the first, the collaboration between the creators of the work allows artists to bring their own unique perspectives and disciplinary skill sets together. As the collaborators have previously worked together on site-based work exploring themes of climate change, several methods already developed to facilitate this collaboration were leveraged for this effort, in particular the use of original poetic phrases to provide asemantic glue unifying different visual, sonic, and textual elements. In this case, the performative nature of the collaboration was underscored by having a live poetry reading be a central element of the work. With the poet present, hearing their words while seeing the work respond to their speech connected the underlying collaborative synergy between text and image on the surface, and at a deeper level, with the connections implicit in the semantic connotations and subject matter of the poetry itself.

The second level of collaboration in the work connects the audience and creative producers,capturing community narratives through interviews and recorded reflections. CollectiveVoices empowers individual voices through community collaboration, allowing them to interplay and be heard in a public creative context, fostering a reflective audio-based ecosystem through which the listener experiences a plurality of narrative and poetic perspectives. Through its use of intertwined recorded personal reflections, the work bridges the private/public gap, and was shared publicly via broadcast on radio, internet, and as part of an outdoor audio installation at the Anchorage Museum.

Visual Poetics

Alaskan poets Erin Coughlin Hollowell and Jen Stever each composed ten poems as part of this collaboration. Framed by her Iñupiaq culture and history, Stever’s poems focus on the local environment, family, and the changing climate. Hollowell’s poems explore careful observations of the coastline and reflections of visual language and form. Distinct phrases from each poem were incorporated into the Processing software in several different visual representations. Randomly selected phrases fade in and subsequently fade out within the audiovisual work, with some phrases coalescing from and dispersing into granulated pixels.

Figure 2. Poetry Live Reading Performance

During several trips to Alaska, photographic and video source materials were collected and used as background imagery within the work (Figure 2). Adding to the text and shifting background imagery, different visual elements and behaviors were developed in Processing for each poem, using a vocabulary of moving generative forms include triangular networks, shards, and ellipses, each with distinctive color palettes tied to the background imagery (Figure 3). A reactive audio system was developed to listen to the poet’s voice, using amplitude and frequency to trigger and modulate the generative forms, with different behaviors for each poem.

Figure 3. Video still of Hollowell's poem “Choreographic” in audio-reactive projection.
Figure 4. Video still from Stever’s poem “Swan Lake Fire” on the Anchorage Museum façade from November 2022-January 2023.

Two versions of the work were developed. The first version was a live performance in which both poets were present reading their works, accompanied by the audio reactive Processing software running in real-time to generate visuals projected behind the poet. The second version (Figure 4) was made to project nightly on the façade of the Anchorage Museum for several months, and was made by recording a performance of the work as a video file. The two versions thus exhibit differences of scale, accessibility, and connection to the spoken voice.

Collective Voices

To create the eight-minute audio composition Collective Voices, several recording sessions took place at the Anchorage Museum and OutNorth Radio in which community participants shared personal narratives about place and local waterways. The authors would like to gratefully acknowledge community partners Indra Arriaga Delgado and OutNorth, a progressive arts organization with a mission to advance contemporary art in Anchorage, support underrepresented voices, and promote cultural dialogue by amplifying the alternative voice. In order to promote a diversity of voices, many different community groups were invited to participate including high school students enrolled in the Anchorage Museum’s afterschool Teen Climate Communicators program and members of the general public invited to record their stories at the recording session venues. Selected excerpts from these narratives were edited into an audio work using Logic software and combined with multiple layers of local processed environmental sounds from the Anchorage Museum’s audio collection. Collective Voices was presented at a listening station at the Anchorage Museum during their November First Friday community event, broadcast on Out North Radio, and made available for internet listening via SoundCloud.

Conclusion

Water Stories: Visual Poetics and Collective Voices engages multiple points of view from local youth, community, and poets in Alaska to share what water means in their life. Grounded in a year-long artist residency at the Anchorage Museum, this interdisciplinary project explores the changing environment through place-based ecosystems of collaboration.

Authors Biographies

Andrea Wollensak is a multimedia artist, designer and educator. Her work spans media from traditional and digital fabrication to generative-interactive systems and includes frequent collaborations across disciplines. Wollensak’s work has been exhibited internationally, most notably at the Göteborg International Biennial of Contemporary Art and the Brno Biennial of Design. Her work has been supported by the Rockefeller Foundation, the International Artist Studio Program in Sweden, Anchorage Museum, Banff Centre for the Arts, and the National Science Foundation. She has presented her artwork at numerous venues including ISEA, CAiiA, Generative Art, and College Art Association. At Connecticut College, she is on the faculty in the Studio Art Department, and an Associate Fellow at the Ammerman Center for Arts and Technology where she served as Director from 2012 to 2020.

Brett Terry is a composer and sound artist, when not busy with his daily life as a software engineer. His electro-acoustic, choral and chamber compositions have been performed at venues such as SEAMUS, ICMC, ISEA, CAiiA, and Sound Culture in addition to collaborating with visual artists on numerous audiovisual works. As an associate editor of Computer Music Journal (MIT Press), he has curated a special issue on Visual Music.

Bridget Baird is a Professor Emerita in Computer Science and Mathematics at Connecticut College and a past Director of the Ammerman Center for Arts and Technology. Her research examines the intersections among the arts and various technologies. Projects include investigating an archaeological site in Ecuador through virtual reality, exploring music and dance through motion capture and multiple modalities, using digital techniques and algorithms to better understand and mine historical documents, and more recently, addressing climate change and environmental concerns by using generative art. Baird collaborated, as a Fulbright scholar, with colleagues in both Mexico and Ecuador. Involvement with the local community has also been important to her and a constant interest has been to increase the number of women in the sciences.

Nga manawataki o te koiora: biological rhythms, posthuman design and decolonial thought.

Rewa Wright, Simon Howden

Queensland University of Technology Independent Artist Meanjin/Brisbane, Australia rewa.wright@qut.edu.au

Abstract

Western science, in fields such as computational ecology, has grown to accept the truths that Indigenous culture have long known: that computational ecology accepts that ecological models are too complex to be summarised in computational form. Since this complexity evades the codification of mere indexing, how then, should we work with computational companions (code, algorithms, programs, platforms). What new ways of intra-acting can we develop alongside computational frameworks, which bring us one more step closer to sentient machines? Most importantly, how can ethical ways of thinking and doing motivate transformations in the computational space, in areas such as machine learning where extreme problems of bias are now embedded? This research does not answer these complex questions, for they are genuinely ‘wicked problems’ that reach toward wider issues of equity, sustainability, and economy. Our aim is to use creative practice to generate gestures and markings that tentatively trace a way forward. This research contributes to new modalities of human computer interaction that attempt to restore the dynamic pathways developed by Indigenous thinking, challenging artificial boundaries such as nature/culture, instead giving respect to concepts of interconnection. Examining some of the differences between Western epistemology and Indigenous thinking opens a pathway toward Indigenous Futures that are crafted in support of a decolonial ecology.

Keywords

Decolonial art, Touch Designer, Plant signals, ‘More-than-human’ Design, Intra-action, mātauranga Māori.

DOI

10.69564/ISEA2023-35-short-Wright-et-al-Nga-manawataki-o-te-koiora

Introduction

Through imaging technologies such as electron microscopes, quantum physics has been slowly proving to itself that all particles in the universe are deeply entangled. Now, the curious situation exists where Western science has ‘validated’ what mātauranga Māori (Māori knowledge) has known and preserved for thousands of years. A close analogy for the unseen movements of quanta is mauri (transitory and shifting states of being), while the structure these quanta form with other particles, emerges in wairua (felt interconnections between all living things). In the Māori world view, people and nature are inseparable and share genealogy. This resonates with current scientific opinion which has only come around too lately to the perspective that plants are to be as valued as humans. Indigenous thinking is in fact far ahead of Western science, in that we never lost the intuitive ways of knowing and being, and we never blindly accepted the artificial separation of mind and body of Descartes’ Enlightenment edict. Through quantum imaging in the 1990s, physicists finally saw that the whole world was connected, every nanoparticle and atom is entangled with every other. This was not new to mātauranga Māori, which for thousands of years had preserved that intuitive knowledge of deep entanglement in our concepts of wairua (life force, shared between all living things) and mauri (transitory and shifting states of being). Underscoring the radical cosmological contingency that materially binds humans to all other entities, the concept we know as mauri is expressed to a degree in quantum physics. That is, the principle that nano level particles indeed bind together very entity in the universe. Our work is about the entangled connections that are intuitively felt, and join humans, nonhuman/more-than-human kin and algorithms in a constant state of becoming. Our research thinks alongside evolutionary biologists such as Monica Gagliano and Stefano Mancuso, who argue that plants are intelligent [2,4]. Situated agency emerges as a tangled network of augmented reality infrared vision, gestural and plant bio-electrical data, audible through a bespoke sensing network of hardware devices and custom software. Understanding the invisible world humans cannot see is important, therefore, to weave an understanding of interconnection back into the process of everyday life. In our work, plants signal to one another as part of their growth process, and we captured these signals as MIDI, an audible expression of their invisible movements. The arhythmic and micro-temporal signals were then incorporated as a foundation to co-create a sound scape with our plant companions. Contact/Sense explores the entangled connections that are intuitively felt between humans and other life forms, sentient or not. Framing the eco-digital within our artistic practice is the scientific context of evolutionary plant biology, on the one hand, and the spiritual Indigenous approach to knowledge as beyond human, assembled in complex networks that are not only empirical but acknowledge unseen unquantifiable forces and the insights it gives as to human behavior and thinking in relation to plants. Agential realist accounts such as those elaborated by Barad [2008], go some way toward destabilizing, at least within posthuman philosophy, the Humanist/Enlightenment duality, opening space for extended speculation on matter and meaning. However, these Western systems occlude the cosmology of knowledge systems like mātauranga Māori. Underscoring the radical cosmological contingency that materially and spiritually binds humans to all other entities, mātauranga Māori traverses both material and spiritual realms, pointing toward a universe that is not entirely knowable, where empirical claims to deep knowledge about the production of matter by material forces are only partially relevant. Permeated with temporal and spatial relations that defy the "arrow of time" and granulize past, present and future into a non-linear series of instances, mātauranga Māori affirms the validity of techniques and methods that trace speculative knowledge.

According to plant neurobiologists Mancuso and Viola, plants are not "passive machines" for processing light, water and food: In fact, they are "intelligent" and show this by constantly signaling to one another and to the world, for example, giving off scents that attract or repel insects and even sharing resources, such as water, amongst their communities [4]. Tracing the connections between quanta and nanoparticles as mauri and wairua as a starting point for a discussion of new media art in a decolonial framework, this paper reflects upon intersectional threads within art, physics, data and posthuman design, to weave an interconnected path between Western and Indigenous science, philosophy and cosmology. We use our practice in media art blending computational networks, human bodies, and plant signals to explore the metaphysical and embodied knowledge manifest through a decolonial symbiosis or science, art and technology, and inclusive to all humans as well as our "more-than-human" interlocutors (plants and algorithms). Concepts of symbiosis may seem attractive since they indicate hybridity, however, as art practitioners we need to be mindful of the power relations in any merger: so that in our writing we approach symbiosis in a decolonial sense, to alleviate the tendency for a more powerful force to overtake a lesser one.

"More-than-human" creativity

"More-than-human" creativity is an emergent thread that reaches toward forming relations with more-than-human kin, organic and silicon, such as plants and algorithms. Through research that bridges plant signaling with augmented and extended reality, machine learning, and interaction design for virtual environments, this project explores the entangled quality of human creativity in the computational networks and living ecologies. This embeddedness, or interdependency, or entanglement, between the organic and the silicon enables new forms of creativity, where artists’ partner with nonhuman kin via iterative acts of co-creation.

This approach destabilizes traditional principles of design, which emanate from the intentionality of the artist or designer. A more-than-human approach allows for an open space of generative creativity where unplanned events and phenomena can unfold, shift, and disrupt the final work. Through practice-based and traditional research, the installation project Nga manawataki o te koiora: Biorhythms, has several interconnected aspects. Indigenous art is conventionally seen as manifesting tradition and not technology. However this work overturns that power relation, where embodied cultural knowledge is allowed to flow through computational networks.

Embodied knowledge and algorithmic design

Nga manawataki o te koiora: Biorhythms, is a projection mapping and installation piece that takes you on a journey into a computational transduction of the forest, rivers and oceans of Aotearoa/New Zealand. Visually, an interconnected natural ecology is translated into the real-time world of audio reactive geometries and mesh topologies. The concept was to convey the feeling of these things, without literal interpretation. Traditional kete (woven baskets) inspire fluid movements which become pixel topologies. Animated motion made with noise oscillators, shift from reimagined nets used to catch eel (hiinaki), to seed pods exploding from pixel plants, such as the red pōhutukawa. The soft blue/green of kina (sea eggs) become fluffy vectors transparently overlaid on a fluid mesh of waves. Following and modifying the tradition of naturalism and curvilinear geometry that marks traditional Māori art, this piece visually encapsulates the feeling of the natural world without being a literal representation that vested in Western pictorial traditions of realism. This 20-minute piece, recorded as live audio reactive in Touch Designer, is sonically a composite of human-nonhuman music, alternately interspersed and mixed together. From the "human" side, music consists of three original electronic compositions by Simon Howden. From the plant side, we intersperse original recordings of plant sonics captured in research since 2019. We consider plant to be co-composers of this work. A previous installation, Contact/Sense was performed at the SIGGRAPH Asia Art Gallery in Brisbane 2019, and combined plant sonics with mixed reality. Donna Haraway introduced the notion of "companion species" to describe non-human organic life forms that we cohabit alongside in society and culture. Plants and humans have lived alongside one another for thousands of years, in a co-dependent relationship of care and cultivation. Applying the decolonial philosophy of mātauranga Māori (Māori epistemology), combined with posthuman lens to art and science, our hope is that through this artwork, people will feel a little closer to the hidden bio-electrical processes of plants, and consider plants not as a resource for extraction, but as a ‘companion species’ in a sustainable ecology.

The plant sonics were recorded during earlier live audio-visual performances with an agave attenuate, a series of vines, several palms, and a multitude of grasses, tropical and sub-tropicals since 2018. Rewa Wright has developed a unique mode of mixed reality performance with plants and algorithms networked with the human body. The plant signals are essentially bio-electrical impulses, which we then assign to MIDI and apply sound design to, so that soft wooden drums with loose skins and resonant tapping highlight the micro temporality and asymmetry of plants, whose signals sound unstructured to the human ear. Plant rhythm is phenological, and traces their processes such as photosynthesis and osmosis as they follow patterns of growth. Several plant neurobiologists have noted the signals that plants emit are akin to intentional communication and sentience, and this is now a recognized area of scientific study.¹, ²

Plant neurobiology speaks to the material reality, unraveled by quantum physics, that every particle in the universe is connected to its nearest neighbors, and through those neighbors to all other things. Quantum imaging has provided visual proof of this connection, as discussed extensively by philosopher Karen Barad,³ whose concepts of "situated knowing" and "intra-action", both grounded in quantum (meta-)physics, have resonance with Indigenous notions of deep time/space and interconnectivity. As a foundational concept, the deep interconnection of life forces (i.e., quantum entanglement) has always been known to indigenous people, and in my culture, Māori from Aotearoa/ New Zealand, we understand this unbroken link between vegetal, organic, silicon, geological forms (to name just a few categories) to underpin all material and cosmological reality. Understanding the invisible world humans cannot see is important, therefore, to weave an understanding of interconnection back into the process of everyday life. In our work, plants signal to one another as part of their growth process, and we captured these signals as MIDI, an audible  expression of their invisible movements. The arhythmic and micro-temporal signals were then incorporated as a foundation to co-create a sound scape with our plant companions. Using computer art to translate emotions about place from bush into pixels, this work explores the potential for a symbiosis of data, plants, ecology and algorithms. This work is deeply influenced by Rewa’s cultural background as a First Nations Māori artist, from the Ngai Tawake, Te Kaimaroke, and Te Uri o Hau hapu of Aotearoa/New Zealand. Her ancestry recognizes human and plant relations going back thousands of years, and Rewa is committed to multi-species justice and sustainable living on Planet Earth. Mixing the biorhythms of plants and the calls with human produced electronic music, reveals a co-creative mesh of human and nonhuman kin, an entanglement that is fundamental. For example, in Rewa’s pepeha (personal genealogy), she traces lineage to the maunga (mountain) called Tokatoka, the awa (river) called Wairoa, and the moana (sea) called Kaipara. These connections are at the base of her identity and this is the same for all Māori, since our genealogy is tied to the whenua (land). Transducing this physical connection with the land into computational space is only ever partial. However, hints at the ways Indigenous knowledge might be embodied as data and algorithms, nurturing a framework that advances decolonial thought and gestures toward Indigenous futures. While quantum imaging has provided visual proof of this connection between all vibrating atoms and matter, it is a concept that has always been known to indigenous people, and in my culture, Māori from Aotearoa/ New Zealand, we understand this connection between vegetal, organic, to underpin all material and cosmological reality. 

Figure 1. Screenshot from ‘Nga manawataki o te koiora’.
References
Books

1 Paco Calvo, Monica Gagliano, Gustavo M. Souza, Anthony Trewavas, Plants are intelligent, here’s how, Annals of Botany 125, no. 1, 2020, 11-28.

2 Stefano Mancuso, Alessandra Viola, Brilliant green: the surprising history and science of plant intelligence, Island Press, 2015.

3 Karen Barad, Meeting the universe halfway: Quantum physics and the entanglement of matter and meaning, duke university Press, 2007.

Authors Biographies

Dr. Rewa Wright (Ngai Tawake/Te Uri o Hau/ Te Kaimaroke) is Senior Lecturer in Film, Screen and Animation at the Queensland University of Technology, Australia. Simon Howden is a sound designer and independent music producer. Both humbly reside in Meanjin/Brisbane on the lands of the Turrbal and Yugara, First Nations owners. We recognise that these lands have always been places of teaching, research and learning, and that sovereignty was never ceded. 

The Dark Side of the NFTs: The artists’ need for new systems of collaboration

İpek Yeginsu

Kadir Has University Faculty of Communication, Department of Visual Communication Design Istanbul, Turkey
ipekyeginsu@khas.edu.tr

Abstract

NFTs are often imagined as the initiators of a democratic revolution in the art world, and yet their effects are much more complicated than meets the eye. While the financial independence and the new exposure channels they offer to the artists are undeniable, they also disturb the art world’s current equilibrium, resulting in the emergence of a chaotic art ecosystem that makes the independent artists’ already existing vulnerabilities even more pronounced. This paper defends the position that artists necessitate new systems of collaboration to sidestep these negative effects, overviews some promising examples of such attempts and discusses the potential solutions for the future.

Keywords

Art ecosystem, digital art, NFT, blockchain, crypto art, artist communities, collaboration.

DOI

10.69564/ISEA2023-36-short-Yeginsu-The-Dark-Side-of-the-NFTs

Introduction

In today’s art ecosystem radically reshaped by the conditions of the Covid-19 pandemic, the artists working with digital technologies have more exposure channels available to them compared to those working with traditional media. The former are able to exhibit their works online without facing the criticisms the artists using conventional media often receive for choosing a digital exposure platform, and they are free from the costs associated with transporting, insuring and installing a physical exhibition. Neither do they necessitate the partnership of a gallery or a museum to present their works; social media platforms and NFTs allow them to sidestep the gatekeepers in the mainstream arts and culture institutions, contributing to the emergence of a more democratic and decentralized art world. In fact, Sarah D. McDaniel and Denny Galindo observe that “art galleries and auction houses have been the market makers of the art world”, and “NFTs and their marketplaces disrupt this process by allowing more artists to go straight to market and sell directly to buyers.”¹

The NFTs’ market success empowers the artists in other ways as well. For instance, Bernard Marr draws attention to the individual power the NFTs have bestowed upon Pplpleasr, the artist also known as Emily Yang, who uses her NFT sales’ revenue to support the other artists and especially those of Asian descent.² Artists without a digital background are also able to access this unfamiliar territory thanks to a new line of service providers facilitating migration into the digital realm without being lost in technicalities. A good example for such endeavors is Vive Arts, a company specialized in the sales and exhibition of art “in all digital formats”, assisting its clients starting from the artwork’s creation phase.³ On the other hand, for the digital artists wishing to develop their skills, Visa launched the “Visa Creator Program” to “tutor a curated selection of digital artists through the NFT process, helping them maximize their talent and use the blockchain space as both a creative tool and marketplace.”⁴

Although NFTs emerge as a convenient funding source for independent artists, they are also criticized for their adverse effects on the environment due to their heavy energy consumption and the subsequent carbon emission. But a more subtle phenomenon challenges the artists in other ways that might threaten their career sustainability in the long run unless they succeed at establishing new systems of alliance and collaboration. The following section addresses both the NFTs’ ecological impact and these relatively implicit effects in more detail.

NFTs and Chaos in the Art Ecosystem

NFTs and Climate Change

The NFTs’ impact on the environment has been a source of ardent debate since the beginning. Some artists have been among the leading opponents of their use in the art world due to their adverse effects on the planet, and they have also been expressing their concerns publicly. For instance, when media artist Joanie Lemercier discovered that the sale of his NFT series consumed the same amount of energy needed to run his studio for two years, he decided to cancel his two upcoming NFT releases.⁵ Another media artist, Memo Akten, designed a system to assess Ethereum’s carbon footprint in numeric values and created the website “cryp-toart.wtf” to share the most up-to-date information about the issue.⁶ Nowadays, he continues to produce NFTs on “eco-friendly blockchains.”⁷ However, others like Eric James Beyer claim that “criticism is exaggerated and conflates hype with the truth.” Beyer compares the NFTs’ yearly energy consumption with that of the small mining industries, giant tech’s data storage facilities or residential air conditioning devices, and reports that the former is relatively lower. He adds that new measures to reduce carbon emission rates are also being implemented, such as Ethereum’s migration from a PoW (Proof-of-Work) to a PoS (Proof-of-Stake) system, and mentions various ecofriendly trading platforms as plausible alternatives.⁸ Raisa Bruner is another optimist mentioning the success of the Canadian company CurrencyWorks in “turning oil waste into environmentally friendly energy that powers crypto mining”, and the Israeli company StarkWare that was able to invent a technique to store more information in one block unit, reducing the system’s overall energy consumption.⁹ Even Mike Winkelmann, the artist widely known as Beeple, ensures the public that he will eventually manage to “completely offset emissions from his NFTs by investing in renewable energy, conservation projects, or technology that sucks CO2 out of the atmosphere.”¹⁰ Efforts at building synergy between the NFT min(t)ers and renewable energy sectors are already plenty. But whether the artists are ready to face the challenges brought about by this new technology reshaping the art ecosystem is seldom addressed.

 

The NFTs’ Negative Effects on the Artists

The problems regarding the NFTs are not limited to their ecological impact; in addition to their advantages, they have adverse effects on the artists’ careers as well. In his article on The Atlantic, Anil Dash observes that “the current NFT market is drawing an extraordinary range of grifters and spammers” and the artists’ works are regularly being hijacked on social media and converted to NFTs and sold by other people without their permission, so much so that, when an app designed for blocking user groups on Twitter was released, its leading customers were the artists “using it to block NFT spammers from hijacking their works and monetizing them as NFTs without permission.”¹¹ The artists’ vulnerabilities in the digital environment are also pointed out by Cass Marshall, who explains that the artists exhibiting work on social media and worrying about hijackers ultimately decide to avoid online presence altogether. And if the hijackers are able to generate the right type and amount of speculation around the work, they are able to obtain much higher revenue from the same work compared to the original artist.¹² Thus, as Simon Spichak says, it is no surprise that “much of the pushback against NFTs is coming from the artists that they were supposed to help”, due to the unbridgeable difference between the copyright culture of the latter and those of the newly emerging NFT fans. He adds that only 50% of the artworks in the NFT market are sold for prices above 200 US dollars, and no correlation exists between the work’s artistic quality and financial worth. To the contrary, the majority of the astronomic sales prices belong to works without artistic sophistication.¹³ Andy Storey identifies yet other ways in which NFTs negatively impact the artists’ careers, such as speculative and fluctuating prices hurting the artists’ overall market value, the NFT platforms’ high entry costs, low respect for the works’ artistic value, and the possibility of the work being used in contexts the artist does not wish to be associated with.¹⁴

The NFTs have also become a source of conflict for various actors in the art world. The artists continuing to produce NFTs and the NFT platforms offering their works for sale are often pressurized by the anti-NFT artists to stop doing so. The pressure can be so serious that, for instance, the artist portfolio platform ArtStation had to cancel its upcoming NFT initiative due to intense online criticism by its users based on environmental concerns.¹⁵ The platform was even compelled to release an official statement apologizing for “all the negative emotions” that their actions caused.¹⁶ Similarly, the artists participating in the Art Wars project declared that the project’s initiator, artist and curator Ben Moore did not ask for their permission when producing their works’ NFT versions. Consequently, the NFT marketplace OpenSea decided to remove the project from their inventory. And complaints do not only come from the artists either; they target the artists instead. Hermès, a giant brand, sued the artist Mason Rothschild for releasing “MetaBirkins” or “100 NFTs resembling the iconic Hermès Birkin handbag.” The result of the lawsuit will also impact the collectors who have already bought items from the collection.¹⁷ Rothschild defended himself by arguing that under the First Amendment, he had the right to “create art based on [his] interpretations of the world around [him]” and drew an analogy between his work and Andy Warhol’s Campbell Soup Can, but it was not sufficient for the court to dismiss the case.¹⁸ Put differently, the NFTs emerge as an important source of chaos and conflict in the art world, particularly in relation to issues around ecology and copyright.

In sum, the NFTs constitute an additional income source and an alternative exposure channel especially for the independent artists, allowing them to bypass the gatekeepers in the museums, galleries and auction houses. But they simultaneously give rise to a more chaotic art ecosystem in which artists have to choose between increased visibility and the violation of their intellectual property rights, or producing critical work referencing iconic images and facing corporate lawsuits. Environmental concerns associated with NFT minting and transactions add further complexity to the issue, increasing the tension between pro-and anti-NFT artists. Moreover, due to the medium’s accessibility to everyone which appears to be an advantage, the artists have to compete against the NFT creators without an artistic background who prioritize financial return over artistic quality.

New Systems of Alliance and Collaboration for the Artists

As a response to the challenges at hand, artists necessitate new systems of alliance and collaboration allowing them to navigate this chaotic art ecosystem more safely. Apparently, one of the basic strategies with fruitful results is to join forces with other artists in the form of co-productions. In fact, fine art photographer Gabriel Dean Roberts explains that he finds NFT collaborations with other artists and even cross-disciplinary collaborations with musicians highly beneficial, as each side brings their own audience into the picture while the uniqueness of the collaboration itself is a form of added value. Joint ventures also contribute to the sense of community among NFT artists by highlighting interconnectedness.¹⁹ Such a solidarity project took place in 2021 when a hundred artists jointly created a single NFT composed of a hundred parts that sold out within a few minutes. According to the project’s curator Loopify, the artist list consisted of a few established figures and “one that is new to NFTs”, and the idea emerged in response to “the current limitations for lesser-known artists to mint and sell their own NFTs such as high gas fees, limited understanding of the tech, and little visibility”. Moreover, to ensure that the artists were exempt from paying additional fees, the revenue was first transferred into USD coin, from which each artist received an equal share. ²⁰ These examples illustrate that artist alliances around NFT creation have the potential to enhance the visibility of every individual artist while reducing the costs associated with NFT production in comparison to doing so single-handedly, and they particularly benefit the emerging artists with limited experience in crypto art. These, in turn, could increase diversity in the art world, ultimately contributing to its long-term sustainability. Collective production also reduces the carbon emissions per capita associated with individual NFT minting, which would minimize the NFTs’ negative impact on the environment and further contribute to the sustainability of the NFT art ecosystem.

The problems related to copyright, intellectual property theft and hijacking require even more organized action on the part of the artists and a closer collaboration with the legislative parties as well as with legal professionals specializing in this subject. Jessica Rizzo observes that, ultimately, the authority resides in the hands of the courts whose exemplary decisions will gradually establish the principles dominating the creation, distribution and trading of the NFTs. According to Juliet Moringiello from Widener University, the laws of intellectual property in the physical world are actually valid in the web3 environment as well, and she refers to the 1996 “Declaration of the Independence of Cyberspace” by John Perry Barlow as a similar example where his attempts at justifying the bypassing of the existing laws ultimately failed.²¹ Yet, the legal framework is still insufficient for the artists even in the United States. For instance, on its official website, Thienel Law LLC refers to The Visual Artists Rights Act of 1990 (VARA) as well as The Lanham Act (Trademark Act) and the Copyright Act as only partially protective documents.²² And as the current legal framework around NFTs is still in formation, the artists themselves should be more directly involved in the establishment of these principles. One way in which they have already begun to make a positive difference is to hold the minting and sales platforms accountable for regulating their clients’ activities with regards to intellectual property. A good example for such systems is provided by OpenSea; in 2022, the platform implemented an anti-plagiarism mechanism consisting of both algorithmic detection and review by actual humans.²³ OpenSea also enabled a stolen art reporting system thanks to which the original artist can file a takedown request against the plagiarist, although the system does not guarantee that the request will be accepted.²⁴ Another solution could be to have the NFT market platforms involve the artists themselves in the artwork curation and elimination processes, so that unethical behavior by a member of the community is directly sanctioned by that very community, e.g., by banning the user from the platform following the receipt of a certain number of complaints. In short, collective action on the part of the artists can be effective in pushing all NFT platforms to take the necessary measures, which would also accelerate the establishment of a basic legal, or at least ethical framework for NFT art creation and transaction around the world, and would help reduce the chaos in the NFT art ecosystem.

As illustrated by the MetaBirkin case, brands are another crucial party in the legal disputes over NFTs. One of the most difficult dilemmas emerges around the artists’ right to borrow iconic imagery from popular culture vs. the brands’ urge to protect their intellectual property. This is one of the most sensitive territories for the future of artistic creativity. If the courts decide to overprotect the brands, they will also have acknowledged that at least half of the seminal artworks from the 20th century are copyright violations. Such a verdict would condemn all the artistic practices involving appropriation, collage and pastiche, and it would do so because it “can” thanks to the digital technology facilitating surveillance and tracking. This, in turn, would seriously deprive the NFT world from the ability to produce art with a political or critical stance, almost “sterilizing” it into a corporate-friendly environment. The dilemma is further complicated by the fact that brands complain about the NFT artists in the same way that artists complain about their online hijackers, and technically differentiating the two contexts is not always easy. Introducing itself as “the first blockchain ecosystem built by the art community, for the art community,” Arcual is a promising initiative in this regard. Jointly founded by Luma Foundation, Art Basel’s parent company MCH Group and BCG Digital Ventures, the platform aims at creating a clean NFT trading environment and offering the artists additional revenue from secondary sales by using “smart contracts with embedded resale terms.”²⁵ Again, the main dilemma presents itself as the trade-off between creative freedom and financial/legal safety, for companies founding such initiatives also have vested interests in the corporate world especially in the form of sponsorships, and whether in the case of a dispute between an artist and a brand they would come in defense of the former instead of the latter leaves a big question mark.

Conclusions and Ideas for Future Research

As illustrated by the overview above, the NFT market’s global outreach and independent environment is extremely beneficial for the artists, but it also intensifies their existing vulnerabilities especially in terms of intellectual property rights by destabilizing the art ecosystem as well as contributing to the deterioration of the ecological one. As the ethical framework of this newly emerging territory has not yet fully developed, the artists might still seize the opportunity to directly shape its future if they manage to act in solidarity, hold the NFT trading platforms accountable and insist on directly participating in the ongoing legislative processes. Further research is necessary to deepen our understanding of the dynamics dominating the NFT art world, the existing legislation at the national, regional and/or global level and how artist communities do and can operate in the face of the existing challenges. Most importantly, further study is necessary to explore how broader artist communities can be mobilized for collective action.

References

1 Sarah D. McDaniel, Denny Galindo, “Democratizing Art: How NFTs Are Reshaping the Art World”, Morgan Stanley website, accessed December 1, 2022, https://www.morganstanley.com/articles/nft-art-market-nft-collectibles#:~:text=NFTs%20and%20their%20marketplaces%20disrupt,galleries%20and%20artists%20get%20paid

2 Bernard Marr, “Web3, NFTs, And The Future Of Art”, Forbes, August 19, 2022, accessed December 2, 2022, https://www.forbes.com/sites/bernardmarr/2022/08/19/web3-nfts-and-the-future-of-art/?sh=6731712f1e05

3 Vive Arts, “Combining industry-leading metaverse technologies with bespoke services, Vive Arts marketplace showcases and sells art in all digital formats”, Vive Arts website, accessed December 1, 2022, https://www.vivearts.com/about-platform

4 Mark Hunter, “Visa Launches NFT Artist Support Program”, FullyCrypto, October 14, 2021, accessed December 1, 2022, https://fullycrypto.com/visa-launches-nft-artist-support-program

5 Gregory Barber, “NFTs Are Hot. So Is Their Effect on the Earth’s Climate”, Wired, March 6, 2021, accessed November 30, 2022, https://www.wired.com/story/nfts-hot-effect-earth-climate

6 Memo Akten, “The Unreasonable Ecological Cost of #CryptoArt (Part 1)”, Medium, December 14, 2020 (updated December 2021), accessed November 30, 2022, https://memoakten.medium.com/the-unreasonable-ecological-cost-of-cryptoart-2221d3eb2053

7 Memo Akten, “NFT”, www.memo.tv, accessed June 16, 2023, https://www.memo.tv/works/nft/

8 Eric James Beyer, “NFTs and the Environment: Why the Anger Is Unjustified”, nftnow, September 16, 2022, accessed November 30, 2022,

9 Raisa Bruner, “Environmental Concerns Have Cast Doubt on NFTs—But That’s Changing”, Time, November 18, 2021, accessed November 30, 2022, https://time.com/6120237/nfts-environmental-impact/

10 Justine Calma, “The climate controversy swirling around NFTs”, The Verge, May 15, 2021, accessed November 29, 2022, https://www.theverge.com/2021/3/15/22328203/nft-cryptoart-ethereum-blockchain-climate-change

11 Anil Dash, “NFTs Weren’t Supposed to End Like This”, The Atlantic, April 2, 2021, accessed December 2, 2022, https://www.theatlantic.com/ideas/archive/2021/04/nfts-werent-supposed-end-like/618488/

12, 15 Cass Marshall, “NFTs are generating huge paydays for some artists, others feel under siege”, Polygon, March 12, 2021, accessed December 1, 2022, https://www.polygon.com/22327806/nft-artists-online-theft-non-fungible-token

13 Simon Spichak, “Independent Artists Say NFTs Are the Bane of Their Existence”, Futurism, January 24, 2022, accessed December 2, 2022,
https://futurism.com/why-artists-hate-nfts

14 Andy Storey, “9 Reasons Why NFTs Are Bad For Artists”, Postergrind, May 7, 2022, accessed December 2, 2022, https://postergrind.com/9-reasons-why-nfts-are-bad-for-art-ists/#:~:text=Artists%20claim%20that%20their%20art-work,is%20a%20lot%20of%20effort

16 ArtStation Team, “A Statement from ArtStation”, ArtStation, March 8, 2021, accessed December 1, 2022, https://magazine.artstation.com/2021/03/a-statement-from-artstation

17 Simon Fitzpatrick, Rosie Adcock and Sophie Mellor, “February NFT Litigation Roundup: Art Wars, Hermes "MetaBirkins", and more...”, Lexology, February 23, 2022, accessed December 3, 2022, https://www.lexology.com/library/detail.aspx?g=666c41a0-7815-4d21-a0f6-6da07ec1388e

18 Andrew Rossow, “The Hermès Lawsuit May Dictate the Future of NFTs”, nftnow, May 19, 2022, accessed December 4, 2022, https://nftnow.com/guides/how-the-hermes-lawsuit-could-deter-mine-the-future-of-trademark-rights-in-nfts

19 Gabriel Roberts, “NFT Collabs: Why You Should Work with Other Artists”, YouTube video, 3:32. April 5, 2021, https://www.youtube.com/watch?v=AeW3dm8xU4Q

20 Joshua Mapperson, “100-artist NFT collaboration sells out in minutes, increases 7X in price in 24 hours”, Coin Telegraph, March 4, 2021, accessed December 3, 2022, https://cointelegraph.com/news/100-artist-nft-collaboration-sells-out-in-minutes-increases-7x-in-price-in-24-hours

21 Jessica Rizzo, “The Future of NFTs Lies with the Courts”, Wired, April 3, 2022, accessed December 4, 2022, https://www.wired.com/story/nfts-cryptocurrency-law-copyright

22 Thienel Law LLC, “NFTs and Artists: Understanding Intellectual Property Laws in the Metaverse”, Thienel Law LLC web- site, July 1, 2022, accessed December 5, 2022, https://www.thienel-law.com/blog/2022/6/23/nfts-and-artists-un-derstanding-intellectual-property-laws-in-the-metaverse

23 Protos Staff, “Humans and computers to fight NFT plagiarism on OpenSea”, Protos, May 12, 2022, accessed June 16, 2023, https://protos.com/humans-and-computers-to-fight-nft-plagiarism-on-opensea/

24 Jack Morse, “How to report plagiarized NFTs as stolen art”, Mashable, April 13, 2022, accessed June 16, 2023, https://mashable.com/article/how-to-report-plagiarized-nft-stolen-art

25 Arcual Art website, accessed December 6, 2022, https://www.arcual.art 

STAND BY/ME

Yu Zhang

STUDIO Ü Eindhoven, the Netherlands
yuzhang.nl@gmail.com

Abstract

This paper describes initial intention, concept, concerns in design process, production, and technical details of an artwork named “STAND BY/ME”. STAND BY/ME is an interactive installation that uses machine-learning models to generate speeches for Xi Jinping and Donald J. Trump, and uses randomness to build virtual conversations for the political spectrum, while connecting the information flow with the lived reality of the everyday. Visitors are exposed to narratives of both digital communism and digital capitalism that is randomly controlled. Next to the visual flow of randomly generated speeches, the work involves seemingly mundane, yet “super-charged” electrical household items, i.e., power sockets, as the actuality of human-technology confrontation. This work (1) allows visitors to feel like they are on “stand by” and (2) triggers questions about how technologism impacts individuals' views and information consumption, while (3) people face randomly generated political speech as a quiet, mundane confrontation. In sum, visitors are invited to be bystanders of randomized political speech, visualized through streams of text ad made tangible through everyday technology.

Keywords

Information representation, information flow, digitals and physicals, social media, technologism, politicism.

DOI

10.69564/ISEA2023-37-short-Zhang-et-al-STAND-BY-ME

Introduction

With the advent of smart and learning systems, connected technology is no more a simple luminous torch to shine upon our wishes or serve our needs. It senses a multitude of signals from our environment, and shapes our thoughts, behavior and social interactions. We find ourselves at the moment when technologism becomes an almost unavoidable aspect of the everyday. Most often our true realization is overshadowed by technological wonder or even deeply influenced by a form of techno-mysticism. When we place the finger on the screen and swipe it, an “informative” world starts and soon it is filled with bits and glimpses of ever-faster news cycles. As individuals, we consume the abundance of news that produced to capture our attention, move us emotionally and sometimes even take action. Over time, we reach a state of being alert and at the same time wholly incapable of action, especially when consuming news about politics. The initial intention of the STAND BY/ME installation (Figure 1) is to synthesize the experience of “stand by,” in the context of technologism and politicism, where each individual can view cycles of randomized and virtual information from two political “speakers” as a bystander. “Stand by” expresses a human inability to act socially, as an inhibition of the mind, a perversion of mindfulness. It is a deeply tiring state. The deeper intention behind this art scene is to state: when an individual is incapable of action, digital data still changes its meaning and form just depending on its medium of distribution, communication and interface.

Figure 1. Exhibition view from STAND BY/ME at Albert van Abbehuis (Eindhoven, the Netherlands).

“Stand by” and STAND BY/ME

STAND BY/ME focuses on the concept of “stand by” and transforms feelings of confusion, in-betweenness and randomness into an immersive audio-visual experience. This experience is designed to trigger visitors’ reaction to human and technological expressions of “stand by”, and visitors’ feeling when encountering a situation gripped by a general sense of “stand by.”

The physical part of this installation is composed of 30 linked power sockets (Figure 2), which are technologically extended and enhanced with light and sound actuation. Apart from the layer of connected physical sockets, the installation incorporates all public speeches by Chinese president Xi Jinping since 2014¹, ², ³ and thousands of tweets by U.S. former president Donald J. Trump since 2016⁴ until January 2021. A recurrent neural network (RNN) was trained with the speeches and tweet corpus, and we fine-tuned the parameter space to obtain synthetic speeches that express the current acceleration of digital communism and digital capitalism. The two “speakers” are then visually juxtaposed with a simulation of randomized draws of numbers (balls) from the National Lottery.⁵ The lottery serves as the metaphorical random actor in this installation: conceptual and technical randomness controls relations between recurring concepts in each politician’s generated speeches. The content and the processing of approaching information data are displayed as a digital information interface (Figure 3).

This work projects the patterns of information flow from the machine-generated speeches first, and connects with the lived reality of the everyday. The ordinary power sockets, “super-charged” household items, visualize a connected data system that enacts the randomness in the actuality of human-technology confrontation. Throughout the design process, this work touches on four symbolic and logic assumptions in the concept “stand by.”

Two Politicians

As two “speakers,” Mr. Xi and Mr. Trump have different ways of approaching the public, quite oppositely.⁶ In 2020, we have read a lot about “truth” by analyzing and interpreting the information and actions from Mr. Xi and Mr. Trump. The citizens have been (still) confused about information and have perceived them as a strongly regional particularity. STAND BY/ME uses a format of public stances and, at the same time, is filled with random content—formal posing as a facade. The concept of STAND BY/ME was accomplished around March of 2020. At the moment of writing this paper, the new conflicts in the world started to develop —presidential election result for the 2020 US election became a turning point for American foreign policy regarding Beijing, followed by Xi has secured a precedent-breaking third term as the Communist Party’s leader and Donald Trump announced a White House bid for 2024. This work is filled with future projections including the efforts to establish dialogues between different cultures and regions. The result of such efforts seems very “virtual,” either in this work or in the reality.

National Lottery

National lottery symbolizes the daydream of picking easy money, yet participating in a random gamble. The format of National lottery used in this work contains three main contents: jackpot size, six ball numbers and one bonus ball number. STAND BY/ME uses this format but creates new feeds that are machine-generated random numbers.

In STAND BY/ME, the lottery is a metaphor for randomness and offers each visitor only a passive role who are not able to take action in the process—intelligent agents submitting to luck.

Figure 2. 30 linked power sockets that are extended with light and sound output in the day light.

Power Socket

The power socket in its most ordinary form is a distribution hub for electricity. Here, its function is transcended: it is a physical connection hub which is available all the time to connect devices that can send and receive all information packages. Power sockets often embody the most common notion of indicating “stand by,” a glowing LED. The use of power sockets in this work metaphorically refers to the action “to be on charge” and similarly as an active participant in propagating information and visualizes the information flow in the audio-visual patterns. Through multiple power sockets, STAND BY/ME embodies a sense of “something somehow happening,” and visitors are incapable of actions.

Interaction

Interaction in this installation went through different stages: from a means to engage visitors in exploring and discovering the emptiness behind the façade to the final implementation where interaction is deliberately withheld from the visitor. The interaction becomes that visitors only remain passive in the installation to simplify the dynamics of light and sound (reacting only to the data processing of the National Lottery). By observing the data processing instead of being part of it, visitors are drawn into the processes of how information spreads or hits a levee or how light and sound trigger the translation of the digital speech data. As visitors realize that they are bystanders, in the back of their minds, they will feel the pressure and obligation to act, without being empowered. 

Figure 3. Information interface including from left to right Xi Jinping, Donald J. Trump and National Lottery for real-time generated text by a machine-learning model and generated “lucky” randomness.

Technical Details

In this installation, a central single-board computer (SBC) runs a Processing sketch that (1) displays the information interface with virtual speeches and conversations from two politicians and projects the results as a live “dashboard” of speeches, highlights and lottery drawings (Figure 3), and (2) translates into 30 linked power sockets that are extended with light and sound output (Figure 2). Each of the 30 power extension sockets is internally altered, while keeping its original appearance. Inside each socket, WIFI-connected electronics (based on ESP32 boards) allow for decentralized control of each two LED lights and one speaker module. The installation generates its own data stochastically modeled after patterns of information flows from social networks. The patterns are mapped and distributed to all sockets via a message bus running on the SBC. The dynamics of light and sound of each altered socket are driven by each sockets’ internal programming and designed to be responsive to the data in the system.⁷ Let’s dive in a bit deeper. The information display in figure 3 shows three columns. The rightmost column is the data display from National Lottery. It contains four main parts—(1) data source and its introduction, (2) real-time lottery information (jackpot size, six ball numbers and one bonus ball number) in technical randomness, (3) the latest coming six ball numbers, (4) the keywords from speeches by each “speaker” that generated from the latest coming bonus number. In the same figure 3, the two “speakers” are shown in the left and middle columns with each four parts of content: (1) data source introduction, (2) real-time RNN generated speeches, (3) up to six highlighted keywords from speeches selected by the latest draw from the National Lottery, and (4) the statement selected from the speeches with shared keywords selected by the latest draw of the bonus ball. The speeches differ in content and important repeating keywords, which are extracted in a ranked list per speaker. These lists are used to draw keywords from by means of random numbers from the Lottery draws.⁸

Conclusion

STAND BY/ME creates an immersive experience set up as interplay between humans, machine and artificial entitles in a cognitively overwhelming way using text, light and sound. Although STAND BY/ME seems calm in general, it puts visitors in an almost passive position when facing the complexity of the machine-generated progress. At the same time, visitors soon discover the need to process each incoming scenario of an entirely modified and virtual information for both digital communism and digital capitalism under the control of gambling randomness. The installation unpacks the concept “stand by” in a collaboration of data sources, data system, technologies, multiple electrical household items, and the digital interface. The design process of this work dives into the process of refining audiovisual complexity with special attention to the details of information representations, metaphorical or not. This work provides the prompts for different perceptions and enables visitors to have a felt experience of the connection, distraction and confusion when digital data changes its meaning and form at every point, depending on its medium and the current “mood” of randomness.

Acknowledgments

The grant of work was awarded by ARGEkultur in the frame of DIGITAL SPRING FESTIVAL. STAND BY/ME was shown in the group exhibition “POLATECH 1” in the period of 27th of August until 6th of September, 2020 at Albert van Abbehuis, Eindhoven. The work was also invited and exhibited at Athens Digital Arts Festival 2022. This installation was built with much inspiration and support from Mathias Funk from the Industrial Design department at Technology University of Eindhoven.

References

1 China.org.cn, Bilingual Texts, accessed October 30, 2022, http://www.china.org.cn/english/china_key_words/node_7216079.html

2 Mission of the People’s Republic of China to the European Union, Speeches (Resources), accessed October 30, 2022, http://www.chinamission.be/eng/zywj/ZYYJ/

3 Ministry of Foreign Affairs of the People’s Republic of China, Speeches (Policies and Activities), accessed October 30, 2022, https://www.fmprc.gov.cn/mfa_eng/wjdt_665385/zyjh_665391/

4 Donald J. Trump, Tweet (text), accessed December 30, 2020, https://twitter.com/realdonaldtrump

5 THE NATIONAL LOTTERY, Lotto results, accessed October 30, 2022, https://www.nationallottery.co.uk/results/lotto/drawhistory

6 Chris Buckley, “Trump’s and Xi’s Differences Magnify Uncertainties Between U.S. and China (2016),” accessed October 30, 2022, https://www.nytimes.com/2016/12/19/world/asia/china-donald-trump-power.html

7 Mathias Funk, OOCSI, Zenodo, 2019, DOI: https://doi.org/10.5281/zenodp.1321220

8 Yu Zhang, “STAND BY/ME information interface (2020)”, accessed October 30, 2022, https://vimeo.com/395705214

9 Yu Zhang, “STAND BY/ME electrical items (2020)”, accessed October 30, 2022, https://vimeo.com/395734433 

Author Biography

Dr. Yu Zhang has a background in fine arts and design. In her Ph.D. research she investigates the theory and artistic practice of interactive technologies for public, large-scale installations. Over the past years, she has designed and researched interactive systems that respond to everyday phenomena, environmental concerns, child-system interaction, online collaboration platforms, and uncertainty in data visualization. Yu has participated in various international Art Residencies of the last years, and her work has been exhibited at galleries, museums, and festivals world-wide. Besides, Yu’s teaching experience covers a broad range from traditional classrooms and workshops to designled project-based learning activities. Her book “Coding Art,” co-authored with Mathias Funk, was published by Apress/Springer in 2021. https://yuzhang.nl/

Long Papers

Ag Tanya Ravn
Art’s Intratemporal Relation to the Future

Amato Etienne Armand, Pereny Etienne
For a more symbiotic co-individuation with our technological avatars: how to go, with the Sciences and the Arts, beyond hybridizations? 

Ammar-Khodja Brice
Symphony of the Stones: A Research-Creation Exploration on the Animation of Heavy Metal Residues in Contaminated Urban Landscapes

Badani Pat
The “Bichi” Project Symbiotic Food Networks & the Alchemist Kitchen

Bae Joonhyung
Thief of Truth: VR comics about the relationship between AI and humans

Batsis Dimitris, Grigoriadou Magdalini, El Raheb Katerina, Politis Archontis
Disembodiment in VR: Immersed in 3D Audio Experiences

Breuleux Yan, Thibault Alain, Lapierre Remi (invité)
The Enigma A/V performance & the concept of Agnostic Media Environment (AME)

Brueckner Sophia
White Cube / Black Box: Investigating Bias in Museums and Algorithms

Brunner Christoph, Fritsch Jonas
Human Energetics in an Era of Post-Humanism

Cîrcu Sorina-Silvia, Chen Chu-Yin
How Digital Anthropomorphism Enhances Creativity in Human-to-Robot Dance Interactivity

Contreras Paul Rosero
The Revolution Will Launch in the Garden: Politics of representation and vegetal intellig(senti)ence

Correia Nuno, Souza Debora, Nêves Inês, Lobato Jaime
Bio Elektron - A Multisensory Approach to Augmenting Dance, Combining: Biosignals, Drawing, Sound and Electrical Feedback

Costa Pedro, Ribas Luísa, Carvalhais Miguel
“I’m a virtual assistant so I don’t have pronouns the way people do, thanks for asking”: gender neutrality, diversification and fluidity in AI

Cunin Dominique, Durand Emmanuel, Seta Michał
Immersive environments, video tracking and collective interactivity on smartphone: a generic “dispositif”

Družetić-Vogel Ivana, Fuchte Alina, Bauernfeind Marina
ARt chat - A Museum App combining AR, Art and Communication

Duarte Regino Juan Carlos
A hybrid listening to atmospheric processes

Foo Louise Lind, Fritsch Jonas
Eleven Movements of the Cryoscape – Ecological Explorations in Sonification for Affectively Engaging with Climate Change

Gatz Sebastian
Cosmo-Techno-Poiesis: Architecture of Environmental Control

Gaylor Brett, Hennessy Kate
Welcome to the Metaverse: Hacking Affect in Immersive Documentary to Increase Critical Big Data Literacy

Gemeinboeck Petra, Saunders Rob
Hybrid Entanglements: a posthuman dramaturgy for human-robot relationships

Georgakopoulou Nefeli, Kosti Makrina Viola, Diplaris Sotiris, Benayoun Maurice, Camurri Antonio, de Gelder Beatrice, Wald-Fuhrmann Melanie, Koulali Panagiota, Valsamidou Kalliopi, Bronner Kai, Wanner Leo, Gaki Froso, Vrochidis Stefanos, Kompatsiaris Ioannis
ReSilence: Retune the Soundscape of future cities through art and science collaboration

Georgakopoulou Nefeli, Zamplaras Dionysios, Kourkoulakou Sofia, Chen Chu-Yin
Towards a Sympoietic Relation with Materials in Interactive Artworks

Gnecchi Ruscone Gerolamo
Thinking with Tides: Engaging with Embodied Technical Processes within the Tidal Ranges of the Thames Estuary

Gould Charlotte
Chthulucene Hekateris

Guljajeva Varvara, Canet Sola Mar, Joseph Clarke Isaac
Artistic Strategies to Guide Neural Networks

Haute Lucile
Kombucha as a Guide. Serendipitous Journey through Taste, Feminism, Free and Open Source Culture, and Ritual

Hofmann Yannick
Towards an Intelligent IoT System for the Data-Informed Museum of the Future

Huang Yu-Hsiung, Hsu Su-Chu
The Creative Design and Social Service Practice of zen_Farm

Irrgang Daniel
Thought Exhibition. On critical zones, cosmograms, and the impossible outside

Jones Maurice, Wester Meaghan, Blottiere Marek
Curation as Research-Creation: Speculating on the Future of Art and Technology Festivals

Kobryn Olga, Couteau Matthieu, Sagot-Duvauroux Rémi, Balcon-Fourmaux Sophie, Garnier François, Ronfard Rémi, Soulez Guillaume
Feeling One’s Way: In Search of a Symbiotic Vocabulary of the Virtual

Lautenschlaeger Graziele
Post-Human Motherhoods: Reflections on mother-offspring bonding as symbiotic individuation in Contemporary Art

Lengelé Christophe, Gauthier Philippe-Aubert
Live 4 Life: A dream for a free and open spatial performance tool towards symbiosis or death?

Mariategui Jose-Carlos
A Latin America Network for Art and Cybernetics: The Centre for Art and Communication (CAyC)

McFarlane David
Towards a Methodology for Co-creating Artistic Acoustic Ecologies with the Great Lakes

Meshi Avital, Chiaravalloti Treyden
Structures of Emotion: Speculating on an AI-Human Symbiosis

Nuñez del Prado Paola Torres
The w(e)aves of complexity: Relational ontologies within the Symbios Art Exhibition

Pataquiva-Mateus Alis
E-cellulose: symbiotic cultivation for the production of smart textiles in a framework of sustainable fashion and electronic art

Polydorou Doros
The Fall of R’Thea: Digital Fiction

Purg Peter, Pranjić Kristina
To Know a Tree: Symbiotic Mutualism and Artistic Exploration against Anthropocentric Science

Rashtian Hamed, Aceves-Sepulveda Gabriela
Same Old Story: Agential Realism in the Study of Colonial Histories

Ribeiro Clarissa
Data-Phantoms: Impossible Nests (Memories Post Extinction)

Ridgway Renée
Black Box versus Black Bloc

Roussel Vivien, Haute Lucile, Teyssier Marc, Dhulster Pascal
Processes, Fabrication and Design with Kombucha Bacterial Cellulose: Mapping Practices

Sagot-Duvauroux Rémi, Quaetaert Nils, Garnier François, Ronfard Rémi
The Hitchcock Experience - a Spatial Montage project

Salter Chris, Thomasson Timothy, Uro Pierrick
Animate: A Theatrical Exploration of Climate Transformation through the Medium of Extended Reality (XR)

Schnugg Claudia, Brill Daniela, Stary Christian
Towards Sensemaking in the Meshwork of Technology, Ecology and Society: Symbiosis of Aesthetics, Performance and Digitalization

Scurto Hugo, Chemla-Romeu-Santos Axel
Deeply Listening Through/Out the Deepscape

Teles de Castro E Costa Isadora, Chen Chu-Yin, Hong Hui-Ting
A Performance Co-Created with an Autonomous Virtual System: A Symbiotic Approach

Thorogood Miles, McCulough Kirsteen, Dulic Aleksandra
Light Up Kelowna: Coordination and Development of Networked Community-based Media Art Urban Screen Infrastructure

Thurow Susanne, Del Favero Dennis, Ostwald Michael J., Grehan Helena
Augmenting Creative Symbiosis Using a Cyber-Physical Aesthetic

Timurgalieva Olga, Direito Eva, Moreira Patrícia R.
(Re)imagining human-yeasts relations via art-science collaboration

Weber Rasa
A Sympoietic Ocean. Design research with/in the marine holobiont

Wu Ziwei, Fei Danlu, Ma Xinyu, Fan Mingming, Zhang Kang
NER: Physical-Virtual Multimodal Generative NFT with a Rarity Model

Xu Dan, Lamers Maarten, van der Heide Edwin
Towards a Relational Model of Co-located Interaction in Interactive Art

Yat Siu Shum Pierre, Klein Tobias
VR CALLIGRAPHY Transposing Chinese calligraphy as choreographed movements into whole-body performances in VR

Zhang Weidi, Luo Rodger
LAVIN: An AI-Navigated Art Experience in Virtual Reality

Zhang Weidi, Su Shaoyu
Volume of Voids: Artistic Visualizations of the Disequilibrium

Zolotova Maria
The Worlds of Entanglements: Reflection on Posthumanist Ontologies in Art&Science Projects

Art’s Intratemporal Relation to the Future

Tanya Ravn Ag

Centre for Art as Forum, Department of Arts and Cultural Studies, University of Copenhagen Copenhagen, Denmark
tanyaravnag@gmail.com

Abstract

The world of art has always been occupied with art’s temporal relations to the future. In the current artistic landscape, we see a wealth of exhibition themes and titles concerned with ‘the future’ in responses to the dominant narratives of a contemporary technological world driven by algorithmic systems and prediction models. This momentous future-orientation is my cue to rethink art’s relation to the future, by zooming in on its temporal modes of existence. With a take-off in the notion of art as “time-based,” as conceptually based in the time, duration, and/or the function of a medium and the experience it mediates, I propose a different intratemporal mode of existence for art. This concerns how art co-exists with, evolves through, and co-produces temporal relations in between humans and technology. This proposal of an intratemporal perspective on art might contribute to further investigations into art epistemologies in which art becomes a part of larger narratives in which human beings and communities co-evolve—and have always co-evolved—with technics. It might offer inroads to study art on its new paths of exploration in collaboration with science and technology and when art is occupied with the very making of the future through participation in innovation projects.

Keywords

Time-based art, intratemporality, future, innovation culture. 

DOI

10.69564/ISEA2023-38-full-Ravn-Ag-Arts-Intratemporal-Relation

Introduction

My temporal investigation in this paper takes off in the catalog Alchemists of the Future published for the Ars Electronica Future Lab’s 25 years anniversary in 2021. In the concluding chapter, “Perspectives,” we can read about how the Future Lab’s activities of visionary prototypes and innovative collaborations between art and science in 1996 were initiated to contribute with future narratives to address urgently needed paradigm changes The visions expressed in the Future Lab catalog, about art’s involvement in our greater societal narratives of technological change, echo the bringing together of art, science, and technology with the conception of the New York-based organization EAT—Experiments in Art and Technology—in 1967, which was founded by engineers Billy Klüver and Fred Waldhauer and artists Robert Rauschenberg and Robert Whitman. The visions of creative and experimental research processes between artists and engineers for bringing artists closer to the materials of technology and more in touch with the forces shaping contemporary society, the use of projection and new communications technology to achieve this, which entailed the exploration of new roles for art in the changing ‘technological environment’ of the late 1960s/early 1970s, and the migration of these practices from an art to non-art contexts,² altogether paved an explorational path for art’s evolvement through changing relations to the future.

Today, as we find when searching through numerous recent titles of exhibitions, knowledge forums, and events of art, (1) the future orientation has saturated the broader field of art. This future-orientation should interest us as more than a thematic trend.

The legacy of E.A.T., and the catalyzation of ideas of collaboration between art, science, and technology through the Future Lab, among many more initiatives, informs a fast-growing discourse in art whereby the art is treated, funded, and appropriated as a catalyst for change. For example, when art migrates into cultures and contexts of technological innovation; when artists are invited into residencies, technology and science labs of corporate technology companies; or, where art becomes a protagonist in major creative funding schemes and innovation programs and is granted support as a catalyst for, for example, industrial innovation, urban development, or human rights. For one example amongst many, the call “Art-driven use experiments and design” under the Horizon Europe Framework, which explicitly allocates a strategic role for art in technological innovation culture. These movements in art, whereby art has gained new roles in strategic projects of future-oriented and future-shaping technological innovation, require new approaches to grasp and assess art’s modes of existence, which I propose that we understand through its relations to the future as an epistemological and methodological compass.

My inquiry is guided by the following line of questions: Why, in the context of our contemporary technological environment, is the orientation towards the future in art so momentous? What characterizes art’s relation to the future in our current technological environment? If the occupation with the future in art concerns a temporal orientation towards how everyday lives, cultures and societies will or might evolve with technology, then how does art participate in the temporal processes that will bring us there? Why does art’s relation to the future matter to the roles that art pursues and gains within technological innovation—as a locus for human symbiotic imagination (about the future) and our technocultural making of it?

My overall suggestion is that we need to grant more attention to art’s temporal modes of existence as simultaneously a matter of object functionality and environment, human and intersubjective experience, technocultural context, and cultural evolution. With a point of departure in the conception of art as “time-based,” I engage an alternative, intratemporal mode of existence for art, with which I understand art to be a part of a larger temporal complex: art is not based in time but existing through intratemporal infrastructures and relations with its contemporary technological environment, which in our current age is characterized by and evolving through data-driven algorithmic processes. I unfold the intratemporal perspective on art through three temporal dimensions—object temporality, worldly temporality, and deep temporality—that relate art to epistemologies on how human experience changes with technological culture. 

Art and temporality
- beyond “time-based” media

When art is described and categorized in a temporal perspective, it is commonly referred to as “time-based.” My claim in what follows is, however, that this temporal conception of art and the epistemological framework that it engages is insufficient to grasp art’s behavioural modes of existence and interdisciplinary evolvement today.

The conception of art as “time-based” is broadly used by museums with reference to artworks that rely on technology, such as video, film, audio, slide, installation artworks, as well as artworks that function only for the duration of their time on display, like computer-based and mechanical works of art. The conception of time-based art ties time to the expressive and functional qualities of the medium. Time-based art is conceptually rooted in “time-based media,” a term coined by museum conservators for durational works of art that unfold over a period of time. It is used widely by art institutions to describe art that is ‘dependent on technology and has a durational dimension’ (Tate), that ‘unfold to the viewer over time’ (Guggenheim), and that are ‘dependent on time, duration, or function’ (National Gallery of Australia). Time-based media has a run-time enabled by the form or medium that limits and contains the experience. The medium enables the inscription of the spectator in different experiences of time. By looking at art as “time- based,” we focus on how the art facilitates meetings between different durations. For example, between the durations of human experience and the durations of a rationalized society. This understanding is fueled by a broad theoretical interest in temporary multiplicity in the writings of among others Henri Bergson, Alfred North Whitehead, Gilles Deleuze, Michel Foucault, and Michel de Certeau.

Christine Ross’ examination on art and temporality in The Past Is The Present, It’s The Future Too exemplifies this understanding of art as facilitating meetings between durations. She describes various ‘durational’ temporal strategies in art as aesthetic counter reactions to the forwardness of the modern era. These are temporal strategies of, for example, endlessness, ephemerality, repetition, real-time, contingency, randomness, slowmotion, condensation, acceleration, extension, abbreviation, speeding up, hesitation, disruption, fissuration, extendibility, and interminability—all temporal strategies for suspending linear conceptions of time that confirm one universal temporal logic.³ Such temporal strategies of suspension evoke a tendency emerging in contemporary art of the 1960s, which is described by Pamela M. Lee in terms of “chronophobia”—a sense of unease or maybe even rebellion in art against temporal societal narratives that dominated during the middle of the 20th century and which translated into a critical consciousness in artistic expressions of performance, conceptual art, sound art, installation practices and land art.⁴ The art of the 1960s that both  Ross and Lee write about reacted against a relation between temporality and historicity, namely one dominant narrative about technological progress that characterizes Western modernity, which celebrated technological transformation, automatization, acceleration and standardization. The dominant narrative reflects a universal conception of time as linear, structured around past, present and future, and organized based on classical physics’ ideas about absolute mathematical time and ground principles of natural science about relativity. In this narrative, time and space are compressed by technological and mechanical processes—what David Harvey has named “time-space compression” which refers to how global communications technologies and information economy compress barriers and distances, which is a function of late capitalism.⁵ This global, temporal narrative is structured around a singular temporal scale characterized by rules of regulation, discipline, speed, effectivity, immediacy and progression—as Jonathan Crary describes in the book 24/7.⁶

The time-based conception rests on a philosophical notion rooted in the ideas of Plato and a substantivist and absolutist conception of time, treating time as an empty container with rules and logics, that is, temporal rules and logics that are ready for art to critically engage with. We recognize this conception of contained time when art is accounted for as an aesthetic, conceptual, critical manifestation capable of presenting and representing alternative temporal modes to those driving capitalism by which to inscribe people into different experiences of time. This temporal containment, however, delimits art’s relation to the future as representational or reflective material that eventually becomes confirmative of the future narrative that it speaks to.

I would like to propose a different temporal condition for art. Because, although art is situated in a specific temporal slot and has a particular duration, and although it might depend on the phone or a mobile device that enables specific temporal qualities of the experience, the work is not delimited to a temporal capsule. It is not delimited to exist “based in time,” as if in a form of a temporal container that we can individually step into for a direct experience with represented image or concept. The time-based conception relies on a direct experience between the human and the artwork. This does not correspond to the ways in which we experience and exist with temporalities through the ways in which most of us engage with technology today. Nor does the time-based conception account for the human-perceptual and technocultural effects of these temporal experiences. Time is articulated in technical systems but only in connection with human engagement with technics, as we learned from the writings of Gilbert Simondon.⁷

In the following, I will propose the contours of an alternative temporal conception of art to that of time-based; one that considers an intratemporal mode of art’s existence. This involves the conception of time as something that the art is relationally entangled with, through which it evolves, and which the art contributes to generating.

Art’s Intratemporal Mode of Existence

Art’s intratemporal relation to the future today bears traces of future-orientations in art of the past. Along with the ongoing critical discourse in art continuing the critical occupation with the forwardness of the modern era that Ross locates in art of the 1960s, we recognize a trajectory from the futurist art movement of the early twentieth century that sought to capture in art the dynamism, energy and movement of the modern world and modern life. Preceding future-oriented movements unfolded in effect and response to a technological environment before the internet, social media networks, and data-driven distributions and accumulation of registrations of our behavior. Today, however, the technological environment conditions a different intersubjective condition than that of the 1960s.

An intratemporal perspective on art might immediately evoke Martin Heidegger’s understanding of phenomena and objects in terms of temporal relations rather than substance, in Being and Time. (2) ⁸ Heidegger contrasts intratemporality with authentic temporality, seeing intratemporality as an existential structure to Dasein determined by calculation and measuring instruments. My use of the term “intratemporality” takes a different reference, in Yuk Hui’s connection of intersubjectivity (subject-context relation) and interobjectivity (object-milieu-relation) in the term. In Hui’s theory, intratemporality is a dimension of changing f temporal relations between objects and neurosensory evolution that happens through our networked and synchronous co-evolvement. ⁹ In my adaptation of this understanding of intratemporality to this inquiry on art, I consider art as temporally related with temporalities of technologies and technological cultures well beyond the medium, the art experience, and the discourse of the art environment.

In the following, I will draw some perspectives on how intersubjectivity, as a matter of temporal relations, is conditioned by the temporalities of digital objects and worldly connectivity as well as by deep temporalities of our cognitive and cultural heritage from technocultural pasts. These temporal dimensions combine in art’s intratemporal mode of existence and tie art to the concept of the future in new ways.

Object temporality: In Hui’s account of the conditions of digital objects and extension on Heidegger’s notion of intratemporality in this regard, he notes how data-driven temporal processes mediate between intersubjective and inter-objective relations and influence temporal experiences in our everyday lives. The ways in which things are quickly shared, behavior and ideas are quickly adopted, and experiences are synchronized, effects an organization of consciousness about how things are temporally related to each other. This reorganization of consciousness with object temporality is what N. Katherine Hayles addresses in How We Think: Digital Media and Contemporary Technogenesis.¹⁰ Hayles writes about how the media interface (e.g., the screen), for example, might seem like it correlates directly to human modes of sensory experience and cognitive processing while it only indirectly correlates to these modes of experience, since it involves technical operations to which we lack a direct access. This is because different time scales of human cognition and machine cognition intermesh. As computational processes occur at time frames that are below the threshold constitutive of human perceptual experience and introduce levels of operationality that impact our experience but do not have any perceptual correlate, we are not conscious of their consequences to our actions. The mutual interference between temporalities of machinic systems and human-temporal functioning of consciousness means that our creation of (abstractions, forms, content, systems, meanings) is not rooted in a direct human relation with what we create but depends on unconscious processes. This unconscious aspect of perception with object temporality connects us to worldly temporality.

Worldly temporality: Global effects—from economical dynamics at a macro level to machinic operations at a micro level—reach us through object temporality. The unconscious cognitive processes at work in our engagement with object temporality not only concern an engagement with digital objects (in and beyond art). They condition intersubjective experience as object temporality relates to the temporalities of networked media technologies, which is entangled with human experience. As Mark Hansen notes in Feedforward: On The Future Of Twenty-First-Century Media, with digital media, the external world has become a part of individual experience, while experience has become externalized and environmentalized in contemporary forms of mediation.¹¹ While we physically exist in the phenomenal world, our thinking, behavior, and the effects of our actions are also of a worldly context, conditioned by mediating factors of different temporalities affected by environmental and global connectivity, and experienced across timespaces.

What this meshing of human and machine temporality results in are operational processes that function as a kind of technical “memory,” which becomes a cultural support structure, and which affects how perception and intersubjective imagination are at work. This technical support structure evolves from a long process of evolutionary adaptation of technical tendencies and their logics, whereby art is intratemporally entangled with a sense of deep temporality.

Deep temporality: Art’s experiences can amplify and resonate through volumes of people, connect us to our ancient past and memories of cultural rhythms, rituals, and practices, and throw us into uncertain futures. An intratemporal dimension of deep temporality links art to cultural patterns, which have shaped the ways in which we use and develop tools and technology since our human origin and the perceptual habits we have evolved and enact when experiencing something, including art. Collective memories, cultural programs and imaginations have been transmitted via habit and repetition through communities and historical epochs. With reference to technoanthropological ideas from the philosophical writings on human technogenesis of Bernard Stiegler¹² based on the anthropology of André Leroi-Gourhan that roots human co-evolvement with technics in the origin of human civilizations,¹³ we can consider how art has a part in the shaping of cultural memory, symbols and rituals that we have adapted from ancient pasts. These have formed through civilizations, cultures and generations to manifest in the cultural codes, meanings and logics we navigate by today. These cultural adaptations of technocultural aesthetics and behavior inform how human cognition meets machinic operations today.

With this intratemporal dimension of deep temporality, I wish to emphasize a technicity in the art as having a function with regards to our cultural evolution with technics. Art, as a human aesthetic expression, has evolved with evolutional adaptations that carry the past into the present—and entwine with the future—through technological tools and the cultural and cognitive memory structures they engage. From studies on ancient human pasts, we know that art, and the technicity with which it operates, in the ancient shapes of rituals, ornamentation, craftwork traces of human gatherings, and more, has taken on various roles as a cultural transmitter and as an aesthetic mechanism of societal organization. For example, as a kind of mediator of societal imaginaries; a vehicle for intelligence, memory, language, forms of expression and pattern recognition to travel through generations; as both depiction and facilitation of rituals (practical, cultural and spiritual); as a connector of human beings to their past and origin and a basis for collective consciousness and emotional intelligence; as a connector of humans to the materials and environments of our world and its ecosystems; as embodying conceptions and philosophies of science as a foundation for furthering civilizations, among many others. These are observations from my research on various intersubjective functions art has had in ancient societies.¹⁴

Intratemporality concerns temporalities that are within us, among us, beyond us and preceding us—and which entangle in our tenement towards the future. The intratemporal mode of art’s existence therefore cues a re-examination of art’s relation to the future. 

Art’s intratemporal relation to the future

The current future orientation in art is not either confirming or resisting a utopian desire. What should interest us are also not the future destinations that art offers or is used to test, project, or speculate upon, which are ideas that inform scenario-based design and conceptual attention to "possible futures." It is also not the critical comment on future-driven regimes. Future-oriented art of today does more than make room for reimagining the future.

My proposal here is that art’s intratemporal relation to the future concerns how art and its experiences are entangled with temporalities that relate us to our everyday engagement with (digital) technologies, with the flows and dynamics of worldly (data) processes, and which engage cultural adaptations and intersubjective evolvement through intuitions and perceptions that precede our experience today. This perspective ties art’s relation to the future to contexts of technological cultures beyond that of Western rationalization. The intratemporal perspective on art concerns what kinds of temporalities the art engages and connects in our bodies, objects and surroundings, and in which ways (by the use of which techniques and aesthetic means). It concerns how art intervenes in our experience of those temporalities. This perspective writes art into a larger narrative in which human experience is changing with technology and in which art has always played a role in the ways in which human beings have co-evolved with technics.

This reconception of time-based art can help us to grasp the new routes and roles art pursues through temporal engagements with technological innovation culture. When art collaborates with science and technology in the domain of innovation, it not only envisions, problematizes, or proposes but also co-produces our futures. This involves a change in perspective, from how art represents and responds to the future, to how art has a constitutive relationship to the future. This is because art engages with human intuitions, desires, and aspirations from where our futures emerge. Art becomes a part of larger intratemporal processes of human co-existence and co-evolvement with technology. This calls for further examination of futurity in art, how the art’s techniques and experience is temporally entangled with future-driven systems and processes of human co-evolvement with technics.


(1) Some of the many recent future-oriented exhibition contexts. I’ve come across in my research, which exhaust and understate my argument that the attention to and conceptualization of the future in art is momentous: Possibles (ISEA2022); Futures Implied (Media Architecture Biennale 2020); Writing the History of the Future (ZKM – Center for Art and Media Karlsruhe 2019); Futures (Smithsonian 2022); Future and the Arts: AI, Robotics, Cities, Life - How Humanity Will Live Tomorrow (Mori Art Museum 2019); The Future Starts Here (V&A South Kensington 2018); Possible Spaces (Danish Architecture Center 2018); Future Shock (180 Studios 2022); WHO Futures Art Exhibition: Envisioning the Future of Health in 2050 (World Health Organization 2022); Future World (Art Science Museum Marina Sands 2022); Future U (RMIT Gallery 2021); Hope for the future & meaning of life (Kawaguchi Art Museum 2021); Sampling The Future (National Gallery of Victoria 2022); Edible Futures (The Dutch Institute of Food & Design 2022); Future Food Today (Space10 Gallery 2022); The Future We Create (Art Works for Change 2022); Remembering the Future: 100 Years of Inspiring Art (Heard Museum 2022); Future Perfect (worldwide 20192022); Future Retrieval: Close Parallel (Cincinnatti Art Museum 2021); TECH/KNOW/FUTUREU/ From Slang to Structure (Montclair State University 2021); Past Present Futures: Notions of Time in Twentieth-Century Art (Blanton Museum of Art 2001); Future Is Today (Al-Tiba9 Global 2020); Decriminalised Futures (Institute of Contemporary Arts 2022); Futureritual (Institute of Contemporary Arts 2022); The Future of Now:; Contemporary Art in Our Unstable World (Emmanuel Art Gallery 2022); Designs for Different Futures (Philadelphia Museum of Art 2020); The Future States (Latvian National Museum of Art 2018); Remember the Future Orleans House Gallery 2021); Designs for Different Futures (Walker Art Museum 2021).

(2) Heidegger’s attention to intratemporality concerns an existential structure of Dasein that is inauthentic and measured by technological instruments and by calculation. In Heidegger’s optics, intratemporality denotes an inescapable horizon for Western history of being.

References

1 Horst Hörtner, Roland Haring, Hideaki Ogawa, Andreas Hirsch, Alchemists of the Future: Ars Electronica Futurelab: The First 25 Years and Beyond, Hatje Cantz, 2022.

2 Christophe Leclercq, “The Legacy of Experiments in Art and Technology (E.A.T.): An Environmental Aesthetics,” paper presentation at ISEA2011: 17th International Symposium on Electronic Art, 2011.

3 Christine Ross, The Past is the Present; it's the Future Too: The Temporal Turn in Contemporary Art, New York, Continuum, 2012.

4 Pamela M. Lee, Chronophobia: On Time in the Art of the 1960s, Cambridge, The MIT Press, 2006.

5 David Harvey, The Condition of Postmodernity: An Enquiry into the Origins of Cultural Change, Hoboken, Wiley- Blackwell, 1991.

6 Jonathan Crary, 24/7, London and New York, Verso, 2014.

7 Gilbert Simondon, “Culture and technics, 1965," Radical Philosophy 189, Jan/Feb 2015, 17–23.

8 Martin Heidegger, Being and Time, New York: State University of New York Press, 2010 [1927].

9 Yuk Hui, On the Existence of Digital Objects, Minneapolos, University of Minnesota Press, 2015.

10 N. Katherine Hayles in How We Think: Digital Media and Contemporary Technogenesis.

11 Mark. B. N. Hansen, Feed Forward: On The Future Of Twenty-First-Century Media, Chicago, University of Chicago Press, 2015.

12 Bernard Stiegler, Technics and Time, 1: The Fault of Epimetheus, Stanford, Stanford University Press, 1998.

13 André Leroi-Gourhan, Gesture and Speech. Translated by Anna Bostock, Berger. Cambridge, Mass., MIT Press, 1993.

13 André Leroi-Gourhan, Evolution et techniques I: L'homme et la matiere (Man and Matter), Paris, Albin Michel, 1943.

14 Tanya Ravn Ag, “Media Art in the Hybrid City – Why?” keynote at Art in the Smart City symposium, hosted by Art Republic at Stavanger Art Museum, Norway, November 2018.

Bibliography

Jonathan Crary, 24/7, London, New York: Verso, 2014. Hansen, Mark. B. N. Feed Forward: On The Future Of TwentyFirst-Century Media, Chicago, University of Chicago Press, 2015.

N. Katherine Hayles, How We Think: Digital Media and Contemporary Technogenesis, Chicago, University of Chicago Press, 2012.

Martin Heidegger, Being and Time, New York, State University of New York Press, 2010 [1927].

Horst Hörtner, Roland Haring, Hideaki Ogawa, Andreas Hirsch, Alchemists of the Future: Ars Electronica Futurelab: The First 25 Years and Beyond, Hatje Cantz, 2022.

David Harvey, The Condition of Postmodernity: An Enquiry into the Origins of Cultural Change, Hoboken, Wiley-Blackwell, 1991.

Yuk Hui, On the Existence of Digital Objects. Minneapolos, University of Minnesota Press, 2015.

Christophe Leclercq, “The Legacy of Experiments in Art and Technology (E.A.T.): An Environmental Aesthetics,” Paper presentation at ISEA2011: 17th International Symposium on Electronic Art, 2011.

Pamela M. Lee, Chronophobia: On Time in the Art of the 1960s, Cambridge, The MIT Press, 2006.

André Leroi-Gourhan, Gesture and Speech, Translated by Anna Bostock Berger, Cambridge, Mass., MIT Press, 1993.

Christine Ross, The Past is the Present; it's the Future Too: The Temporal Turn in Contemporary Art, New York, Continuum, 2012.

Bernard Stiegler, Technics and Time, 1: The Fault of Epimetheus, Stanford, Stanford University Press, 1998.

Author Biography

Tanya Ravn Ag is a curator and scholar on art, technology and digital culture, focused on how art changes with contemporary technogenesis and algorithmic culture. She is the editor of Digital

Dynamics in Nordic Contemporary Art (Intellect, 2019) and co-editor of What Urban Media Art Can Do – Why, When, Where, and How? (av edition, 2016) based on which she co-founded the globally networked Urban Media Art Academy in 2017. Her curatorial engagements with urban, media-based art include the Screen City Biennial 2017 in Stavanger, the SP Urban Digital Festival in São Paulo in 2013 and 2014, and Nordic Outbreak presented in New York City and across the Nordic region by the Streaming Museum in 2013-2014. She has served on the ISEA Board since June 2023 after chairing the IIAC (ISEA International Advisory Committee) since 2020.

Acknowledgements

The research informing this paper has been supported by the Carlsberg Foundation’s Reintegration Grant for the project Art of our Times: Deep Temporalities of Art and Algorithmic Culture at the Department of Arts and Cultural Studies, University of Copenhagen. 

For a more symbiotic co-individuation with our technological avatars: how to go, with the Sciences and the Arts, beyond hybridizations?

Etienne Armand Amato & Etienne Péreny

Laboratoire DICEN-IDF, Université Gustave Eiffel
& Université Paris 8 Paris, France
etienne-armand.amato@univ-eiffel.fr ; pereny@gmail.com

Abstract

With interactive computing, the metaphorical use of biological notions of hybridization and symbiosis has become widespread. They refer to the possibilities of mixing as well as to the conditions of emergence of relationships ranging from mutual benefit to instrumentalization between technologies and humans. In order to better understand the relevance of such analogies with the living, this article draws on scientific and artistic research concerning the interactive avatar. These seem particularly instructive on our relationship to technology because this virtual being hybridises the living and the artificial, while constituting the key and the condition of access to digital spaces, to co-evolve with other users, themselves avatarised, or other autonomised agents. The article distinguishes between "cyber" avatars populating persistent universes, video games, virtual realities, i.e., cybermedia environments of simulation and interaction. The "hyper" avatars are those of the Web, online services, 2.0 platforms or socio-numerical networks that are juxtaposed in networked informational and documentary hypermedia. From then on, the challenge is to reinvest the sense and responsibility of their potential for augmentation or simulation, in particular by promoting cooperative interactions through science, art and technology, which are themselves by their very nature synergistic with each other.

Keywords

Avatar, Cybernetics, Symbiosis, Video games, Hybridization, Hypermedia, Cooperation, Responsibility, Arts, Sciences.

DOI

10.69564/ISEA2023-39-full-Amato-et-al-technological-avatars

Introduction: symbiotic design at the origin of interactive computing

Since the great convergence of telecommunications, computer science and miniaturized electronics, the expansion of networked digital environments has been accelerating, which never ceases to raise questions, so much so that it guides the civilizational changes underway. Digital art has been able to contribute to these developments from the outset by appropriating the different waves of inventions, software and tools: network art, code art, interactivity, Web art and Game Art, etc. However, due to a lack of recognition, these influences have remained underground and unofficial, as industrial and commercial creations dominate. After decades of intellectual and artistic, academic and cultural engagement, initiatives such as ISEA or numerous Arts & Sciences projects, have made common the idea that artists contribute to create an alternative technological culture to those stimulated by consumption, market and entertainment, in order to produce new practices and regenerative imaginaries. By choosing the theme of "Symbiosis," ISEA 2023¹ sends a strong signal to the communities in charge of digital cultures and electronic arts, a kind of encouragement to favour a transformation of our relationship with the digital, to move towards more responsibility and reciprocal benefits, while comfortable consensuses and evidences are cracking. However, the matter is not so simple. Admittedly, the symbiotic approach has the advantage of offering a qualitative leap forward and a step back, integrating the latest scientific knowledge on cooperative interdependencies, at all scales of life and at levels previously invisible, and just which have been revealed by technical images.² However, this notion does not have the same cultural and political effects depending on whether we take it from an apparently neutral angle; with the Anglo-Saxon meaning, which considers as symbiotic all forms of interactions between species, including the most aggressive and predatory, parasitic and deadly ones; or whether we take it in its French and European meaning, which favours synergistic, mutualist or commensal symbioses, offering mutual or at least unilateral benefits.

But in a facetious and creative way, let us open our remarks with an enigma offered to the reader of this article, before he or she dives into the heart of digital systems and environments by following the path of co-evolution made possible by avatars.

From whom does this excerpt come? "The potential of the human-machine symbiosis is easily visible in the arts, where computing technologies have enabled the creation of previously unrealizable forms of expression. Computing technology has empowered a new legion of artists working in mediums such as immersive and augmented reality games, animated feature films, and music composition and performance (...) In these areas, we are beginning to see humans and machines as complete partners in artistic creation."³ It is striking that an eminent researcher in computer science, having joined the most demanding military programs of the DARPA in terms of success, recognizes artistic activity as one of the best contexts for the emergence of new partnerships for the benefit of an original creation. To agree with this, it has been widely shown that artists, through their close contact with techniques and their potentialities, in their exploration of the logic and capacities implemented in their tools, which have become software with computers, constitute an avant-garde that clears out the possible and opens up new appropriations. If the works in art history have shown this, it is up to thinkers more in tune with the arts and technologies of the image, such as the Frenchman Edmond Couchot, to show it in the course of his numerous works.

But beyond the man-computer symbiosis and the place of the Arts in its advances, let's go back for a moment to the origin and the evolutions of this new paradigm of microbiological symbiosis that appeared following the work of Lynn Margulis⁴ concerning eukaryotes in the 1960s, co-author of the "Gaia Hypothesis" with James Lovelock.⁵, ⁶ This research eventually produced major scientific advances concerning plants, forest and soil, but also the intestinal microbiota in humans, emphasizing the symbiotic interdependence of living things, both inside and out their bodies.

For our purposes and references concerning the coevolution of man and his technique, we will retain the teachings of Leroi-Gourhan⁷ and Gilbert Simondon⁸. The former analyzed hominization through "gesture and speech," techniques that preform both our morphology and our cognition, the latter insists on the "individuation process" and its "associated environment," which organize the individual and collective mode of existence of man, as well as that of technical objects.

But the difficulty remains in finely articulating the human and the artificial, in respecting proven frameworks of thought while going beyond them, updating them, making their complexity a little more "simplex," following the teachings of Alain Berthoz⁹ supported by the phenomenology of Merleau-Ponty¹⁰ and the enaction of Varela.¹¹ By proposing an emerging concept, that of the symbiotic co-individuation of man and his avatar, that is to say an individuation—not to be confused with individualization—which is made joint by their common process of co-evolution, it is indeed a question of situating its context and exposing the premises of a vast site of studies and creations which remains to be undertaken through an approach seeking to create a new synergy between the Sciences and the Arts. 

I) Symbiosis and hybridization: biological metaphors in vogue in the computer and digital field, from the beginning of computing to the present day.

a) A novel concept of a possible partnership between humans and computers

Anyone interested in the links between symbiosis and computer science will identify the inaugural 1960 article "Man Computer symbiosis"¹² written by one of the acknowledged founders of modern interactive computing, J.C.R. Licklider. Both a psychologist and a computer scientist, he developed a way of thinking about the partnership with the computer, and later helped coordinate, within military programs, the transition from the first computer networks to the Internet. Such an early reference to and mobilization of a concept from the life sciences, and moreover in connection with the military-industrial complex, may surprise and even arouse suspicion. A few points need to be made here. Firstly, it bears witness to a context of technical emergence, that of computers finally operating in real time and becoming sufficiently programmable and accessible thanks to their sensitive interfaces (screen, optical pen, and keyboard) to envisage a genuine cooperative interaction that would bring together the best of the two worlds, living and artificial. Secondly, it also highlights a typically cybernetic vision of putting human and computer in parallel, becoming colleagues and partners, who, far from being reduced to each other, assert their specificities and differences, but pull their strengths and weaknesses. Thus, by associating and communicating according to the right methods, they will constitute a tandem achieving what no one alone would manage to do. Finally, this positive, even idealistic, vision must be understood as a "political move" opening up another perspective at the heart of the technocratic system to confront the promoters of the all-machine approach and of a progress leading eventually to the advent of artificial intelligences superior to humans. However, several controversies could immediately invalidate such an idea and send it back to the side of naive approximations. In fact, symbiosis is observed in the field of the living, not the artificial. Applying it to computers would mean granting this class of logical machines the status of a species endowed with certain properties of the living, such as evolution, adaptation and reproduction. But in our western conception of the world, practising a “naturalistic ontology” as French anthropologist Philippe Descola call it,¹³ the highest faculties are the prerogative of humanity. If we want to continue the analysis with Gilbert Simondon,¹⁴ in the absence of integration with culture, technology is conceived as separate and external to man, who at best uses it, while attributing to it his own misdeeds. When in fact, technology, this human creation, has the role of mediating between nature and culture. However, the metaphorical use of symbiosis has a few merits: better respect for each stakeholder in the tandem; understanding of their interdependencies; and the establishment of a more balanced cognitive and sensitive dynamic than the sole technical and strictly functional approach, that of a mechanically extended human, "mechanically extended man" or "humanly extended machines" (quoted in ¹²). With this approach, enlightened by the living, a perspective is developed that is neither completely anthropocentric nor technocentric. It will indeed influence a whole community of inventors and engineers to seek mutually beneficial relationships. This begins in the field of complex problems, extends to ergonomic interfaces, in connection with the interfaces and work methods invented by Engelbart ¹⁵ with his "mother of all demos,"¹⁶ and will continue with the first technological research involving computer networks, at the origins of the premises of the Internet.

b) Instrumentalization and hybridization: what association between the living and the artificial?

It is clear that more than 60 years later, this cooperative conception has not really governed the deployment of the "digital." The triumphant regime is that of a generalized and excessive instrumentalization of some by others. It is also based on older, naturalistic metaphors that have become dogma: the struggle for survival, selection, competition and domination, all stimulated by a misunderstood and ideologized Darwinism of Anglo-Saxon and capitalist origin that fetishizes the market and competition. This cultural overdetermination tends to reduce both humans and computers to a few utilitarian functions, rather than mutually emancipating them, or just amplifying them (Licklider¹²) or increasing them (Engelbart¹⁵). For example, workers turn out to be replaceable as soon as a task can be automated, while the computer itself is devalued, becomes obsolete, as soon as a new, more powerful and "modern" generation arrives on the market. In other words, technological "progress" is based on the division and specialization of tasks to the detriment of others: models implementing interests, reconducting the existing and thus rigidifying socio-technical systems. (1) Following an instinct of conservation hiding under the imperative of innovation, it is essentially a matter of optimizing and stabilizing rather than solving and changing; of intensifying short-term productivity rather than changing the level of symbolic integration and moving towards "a better ecology of the mind."¹⁷

Another notion, also imported from biology, is proliferating in the field of new info-communication technologies, and continues to be very much in vogue on the cultural level.

It is the notion of hybridization, synonymous, depending on the context, with convergence, fusion, mixing, blending, and association, concerning both the technologies themselves and the interpenetration or interweaving of heterogeneous dimensions: the human and the artificial, or the real and the informational. It carries an enthusiastic conception of the possible crossings between various technical lineages as well as of what results from them, their offspring. It probably has a positive echo in our anthropological collective unconscious because of the reproductive experiments dating back to the Neolithic period, carried out by our species on plants and animals to make them edible, domesticate them, use them, and thus develop ever more new hybrids to improve our living conditions.

As a result, there is a tendency to explore and implement all possible configurations through clever and inspiring mixtures, according to a principle of free experimentation that does not take into account the stakes and consequences from the outset, but rather evaluates them, possibly and only after the fact. Believing himself to be the "master and possessor" of his technique (2), humans let themselves go to "hubris", to this excess of power, to this attractive vertigo that is generated by a too continuous success. The conquest of all possibilities guides our collective strategy. Most forms of exploitation, first and foremost of the living through technology, are justified, pending the next techno-solution that will correct the effects. In this vein, transhumanist conceptions assume that humanity must continue to mutate by itself thanks to Technology and Science, assisted by an Art that would become its official foil. But do they realize that the solutions obtained by a definitive hybridization (the cyborg) or by deep environmental modifications (the geoengineering) remain prisoner of the same paradigm that caused the problems it tries to solve? Although some forms of hybridization may be viable and desirable, with lines that improve through heterosis, which is the alliance of the best of two species, we will try to think according to a higher general principle, that of an evaluation and regulation of a symbiotic nature, in the French sense of the term, and therefore mutualistic, which could frame, anticipate and therefore limit any possible deleterious effects.

II) "Cyber" and "hyper" avatarizations: two main ways of entering the digital world

a) The cybernetic avatar: a body situated to experience virtual technologies

The fundamental phenomenon that networked microcomputing has made possible, but which was not immediately grasped or understood as such, concerns the "existential" entry of the human into the machine and into the heart of its software. If this is realized in many ways, the avatar is the most widespread and concrete emblematic figure, especially the one that networked video games have made known to us. Indeed, since its baptism,¹⁸ the avatar has made it possible to inhabit an immersive environment with other human persons and programmed entities, and even to cooperate with artificial beings or with other "semi-living" beings, which have yet to be better qualified and understood, both scientifically and artistically.

The original symbiotic vision already mentioned had stopped at the face-to-face meeting of the human being and the computer, forming an interdependent and winning team. With the integration of networked computers, an informational, quasi-corporeal vector was needed, if only to manifest itself to other distant people, caught in the same technological constraints. In a shared place, benefiting from a system of exchanges and actions, the human invests his avatar to interact in a more complex way than with conversational interactivity based on dialogue and instruction. Collectively, the first interactivity (reactive, sequential, asymmetrical) is overcome and gives rise to the second interactivity (proactive, simultaneous, mutual). The first occurrences of avatars (3), retrospectively identified by semiologists, computer scientists and other analysts,²⁰ result from the marginal appropriations of the very first transistor computers by new adepts of creative subcultures, looking for a way to divert and explore the capacities of new electronumerical machines, at a more human size and finally becoming more accessible.

Since then, despite their improvement and growing sophistication, playable avatars, those of the simulated worlds of video games, can still be analyzed in terms of "living-artificial complexes." Indeed, the technical part, with the programmed functioning, and the living part, with the human behaviours adjusting in real time to the action in progress, are mixed in the same virtual being. In this respect, avatars are similar to techno-human or bio-technical "hybrids," depending on the point of view adopted. In this respect, field studies in the human and social sciences show that online participants who use avatars know how to consider any "player character" they encounter in a persistent universe from both sides. They evaluate both their operational capacities and the living intentionality that drives them. In addition, the hybrid avatar has several remarkable characteristics that relate to its mode of operation and attendance. On the one hand, it results from a "soft" hybridization, because it is temporary and reversible (unlike the cyborg), which preserves the integrity of both halves, the human and the artificial. On the other hand, this humanized technical hybrid is unified and maintained by two parallel processes that can be analyzed through the symbiotic metaphor. On one side of the screen, in the ordinary world, there is a human-machine-computer coupling which, for the time being, joins the Lickliderian vision of a dynamic symbiosis. Except that here, it is even more marked because it is based on a total interdependence: the human needs a formalized avatar to evolve in a simulated world while, without its player, the avatar is only an empty shell, identifiable by its looped attitudes, as if waiting for a soul to re-function. Both interface and vehicle, it is an entity that articulates the real and the virtual. On the other side of the screen, this time in the simulated world, the avatar must compose, negotiate and build its partnerships, if possible symbiotic to last and grow, both with its fellow avatarized, and with other inhabitants fully programmed and more or less autonomous. Its survival and development often depend on relationships of co-construction and cooperation, even in a competitive context. By associating as a group, by combining their talents, by carrying out collective strategies, while benefiting from the support of artificial beings, the human beings hybridized by the avatar adapt to a place lived in common. They develop a powerful intersubjectivity, a feeling of living together that is confirmed by their decisions and actions. Unified by a particular spatio-temporality and physicality, this virtual but concrete place mobilizes their situated cognition; that is, sufficiently spatialized to transform the environment and its components or actants into resources. As for the corporeality of their avatar, it engages a cognition here virtually embodied, but otherwise instantiated according to principles partly analogous to our human condition. Finally, through their avatars, humans enter into a very strange trade with certain autonomized agents, typical of their adventure worlds: pets, helpers and assistants, drones or robots, supernatural allies... With these, a distributed cognition develops, i.e., distributing the processing of information over these autonomized entities, for example, the tracking, surveillance or management of this or that process. All of this is linked to the ongoing cooperation with the other comrades in the game, the other avatars. In this way, many intimate and vital relationships are established and unraveled with programmed or hybrid technical beings, whose radical otherness is forgotten, by dint of being dressed up in the colors of the narrative, of the game or of simple co-existence, as within Second Life-type metaverse based on collective creativity.²¹

Figure 1: "Cyber" hybridization process doubly framed by symbiotic relations leading to co-individuation

This diagram visualizes the way in which an avatarial hybridization takes place, framed and framed by a double relationship that can potentially be interpreted in symbiotic terms: on the left, that of the human coupled to the nested triad computer/software/avatar; on the right, that of a cybernetic "avatar" comparable to a symbiont, as soon as it is engaged in relationships of interdependence with mutual benefit with its other congeners, or even with various entities entirely programmed and eligible for a relationship of interest. It is in these virtual worlds that symbiotic co-individuation could take place not only between the human and his avatar, but also with the empowered virtual beings that also inhabit them.

b) The hypermedia avatar: an aggregate of data in the service of an identity model in hybrid spaces in two ways

In contrast, we must address, even briefly, the other immeasurable domain of hypermedia, because the way in which humans must avatarize themselves in order to exchange and cooperate with their fellow human beings is quite different from the previous one.²² This involves the establishment of a "profile," with oriented functionalities. This is in fact a particular identity model, a sort of explicit information sheet specifying the aspects useful for the activities specifically allowed by the platform. By filling in fields and forms, by providing qualities and quantities, by uploading media (photos, thumbnails, videos, visuals, sounds), the user explicitly creates his profile, which will then be fed more or less invisibly by the "datas," "metrics" and other "analytics" that are generated according to his activities. Because the activities are more disembodied, more informational than simulational, the participant feeds the platform where he acts by his punctual acts, by his simple consultations, requests, clicks and orders, which have become his editorial and media contributions. There are certainly some services that focus on cooperation, such as Wikipedia for collaborative writing, or GitHub for collective programming. However, most platforms, especially commercial or social ones, juxtapose hyper avatars with no common space other than a customizable interface. They offer specific functionalities, for example, for shopping or conversation, which require a simple interactivity, of command and impulse: summoning contents, pages or profiles; clicking to act; typing and sending asynchronous messages to exchange; just exciting by notes, scores or likes, or on the contrary, inhibiting by blocking and unsubscribing.

In the inaugural vision, it is within this informational, documentary and media hyperdimensionality; from Vanevar Buch's Memex to Ted Nelson's precursor "hypertext"²³ ; that positive symbiotic relationships were supposed to unfold. But we are struggling to find them, so much so that the algorithms that model and influence behaviour encourage compulsive buying, isolation in filtered bubbles, the excitement of passions and cognitive biases. Thus, these hyperspaces have tended to reduce the human to a set of models and data,²⁴ made of functions and behaviors that integrate with software. User-consumers are thus influenced and made predictable by traceability, profiling and probabilistic anticipations. The mainly utilitarian relationships that are currently developing within the "hyper" digital environment have shown their danger, even toxicity for democracies, potentially influencing crucial voting processes. At issue is the modelling of stakeholders, which is often done without their knowledge, and the monitoring and stimulation of behaviour guided and motivated by forms of economic, cultural and political predation that go beyond the logic of advertising or the attention economy alone.

Such extreme pressure and overkill can be explained by the doubly hybrid nature of the Web and its variations. On the one hand, "hyper" informational environments have a strong potential for intrinsic hybridization, linked to the ease of combining media and documents that are hyperlinked, enriched with information, recombined, declined, merged, diverted and replicated. They are also able to accommodate all other old (radio, television, press) and present (platforms, services, applications) media, including cybernetic virtual worlds, which a simple, well-programmed web page can display in some cases.

On the other hand, these "hyper" platforms are hybrid in the sense of devices (educational, professional) that mix real and informational spheres. Most of the time, the infosphere constitutes a duplication of the real world, which generates relationships of reciprocal influence. Human avatars, "hyper profiles" point, despite their pseudonymity, to civil identities, while online businesses or connected data visualisations drain real flows of goods and data. The result is numerous and strong power stakes and possible holdings attracting the will to power, mobilizing all sorts of stakeholders: states, hackers, companies, collectives. Conversely, cyber worlds and avatars seem to escape this extrinsic hybridity, because they remain more disconnected from the real world, at least until the promises of metaverses claim to encompass them all and link them to our real values through NFTs and other speculative crypto assets. Conversely, "hyper" universes are hybridizing both internally and externally, the latter aspect having been intensified by the multiplication of mobile computers (phones, tablets) and other connected and "smart" objects that link infosphere and biosphere, artificial and living.

III) Discussions and critical hindsight

a) Cyber and hyper polarizations: towards which symbiotic co-individuations through the avatar?

At this stage, we should recognize the limits of this opposition between a symbiosis that is more possible in simulated cyberworlds (immersive environments) and an unbridled hybridization in informational hyper-spaces (distributed hypermedia and other Web platforms).

In the first place, there are still some "hyper" logics within cyberworlds, for example instant teleportation, or interface parameter layers. Symmetrically, this time in hyper-spaces, some "cyber" systems act and function: predictive models, recommendation engines and other underlying simulations ²⁵. Well hidden, in the background, there is also this implicit avatar made without our knowledge, this identity avatar calculated and manipulated by the 2.0 platforms whose existence has been recognized by the repentant Silicon Valleyers and who have been described as "Digital Voodoo Dolls"²⁶ evoking the spell. This verifies the excesses that many have denounced. All the more reason to take up one of our collective challenges, which would be to endow hyperspaces with more cooperative and symbiotic properties/characteristics,²⁵ in particular by interacting through hyperspace avatars that are more transparent, temporized, situated, regulated and better controlled. As for "cyber" avatars, which are themselves driven by basic modelling or AI that may also contain obscure biases (known as dark patterns), let us beware of idealizing or valorizing them without detailed investigation. For the challenge is still to highlight, in order to cultivate and preserve them, the modalities and relational conditions of regulation (coupling, embodied, situated and distributed cognition) and of positive integration (benefits and gains, emancipation and empowerment) in order to foster mutualistic partnerships. This can only be done through reciprocal adjustments and constitutive reinforcements towards a co-individuation of all inhabitants of cyberworlds, which relies on all affordances available in these inhabitable virtual spaces.

Secondly, it should be noted that the processes of hybridization have been explored in very positive and constructive ways, and this in the field of Art. Communities of artists, technologists, experimenters, or university researchers, wanted to open other ways of convivial and regenerative co-existence with the new digital technologies of art. The international colloquium "For a digital imaginary with Edmond Couchot. Hybridizations between the arts, sciences, technologies and the human" in November 2022 was able to give an account of how the revolutionary capacities of a computer have been understood for 50 years. The latter have opened up the digital dimension by becoming universal simulators. Their interactive images constitute matrices of hybridization "squared," producing through language and gesture "the image with image power."²⁷ This has made it possible to revisit, invent and materialize our imaginations, to make our fictions or visions tangible. This conception is in line with the idea of a computer structuring a meta-technology²⁸ that can simulate and accommodate all the others. This vast line of creations and works have also paved a way towards partnership symbiosis by endowing technical virtual beings with a certain autonomy, as shown by the interactive installations La Funambule (presented at ISEA 2000 ²⁹) and more recently InterACTE³⁰, in which a humanoid creature learns and individuates itself through contact with a participant who in turn experiments with new relationships with this autonomizing virtual being.

However, there is no spontaneously and naturally good and right approach: the artists in question have been nourished and framed from the very beginning of their "human-computer partnership" by the contributions of science and culture. This has given them the resources, knowledge and principles that enable them to escape the hubris of omnipotence in order to assume the ethical, political and scientific responsibilities related to these new demiurgic powers.

Thirdly, and lastly, the challenge is to offer intelligible and sensitive resources so that the anthropological entry of humanity into its "virtual meta-machines" (connected computers), this exo-somatisation of our own virtual double, can enrich us in return in a way that is both "technaesthetic" and conscious of everything that determines it. The innumerable lives offered by avatars of all kinds—both "hyper" informational profiles and "cyber" simulated corporeality—invite us to experience other real and imaginary worlds, dystopian or desirable, acting as initiatory gateways, instructing our relationship to artefacts, to ourselves, to the living, to our One Earth, and to this virtual sphere, to be envisaged as a "virtual common" to be healed.

b) What synergies between the sciences and the arts to finally assume our current responsibilities?

The productive articulation Arts/Sciences including Technologies is justified by the interest that there would be in the creative activities taking support on the scientific knowledge and investing the big changes of paradigm which currently restructure a number of disciplines, of which precisely that of the symbiosis with the rise of ecology, of the human and social sciences, of the sciences of the complexity and its solutions, biomimetic and simplex derived from the evolution of the living. Certainly, if it should be reaffirmed that, by definition, art is a domain that escapes the principle of responsibility, thanks to the famous artistic license, it nevertheless deserves to take its full share and responsibility in today's "paradigmatic revolutions." These are achieved precisely through changes in the model of understanding of the world,³¹ as is happening today with anthropology, the sociology of science or the philosophy of technology: holistic approach, destitution of the subject individual king, taking into account technical relations and processes and new entities at work (non-human actors, stakeholders, resources and forces), better integrations of interdependencies and a growing consideration for subtle, invisible or microscopic dimensions, inviting temporalities and spaces incommensurable for us mere humans.

As for the problematic of avatars, it links us to the more general one of the exosomatization of the highest cognitive faculties of the human species in virtual techniques and creations. We believe that in this field, art can not only bring us alternatives, shifts, surprises, wonder, but that it constitutes with science one of the most adapted means to approach complexity and uncertainty, to synthesize even paradoxical advances. The Art-Science alliance, which is still to be built, can be based on different ways of coupling them, in order to favour a capitalization, a mutual collaboration. If art is stratified and memorized through culture, scientific advances build knowledge. Now, in view of the emergence of new technologies, there is every reason to encourage a crossing of these two systems of accumulation, so that art is a source of knowledge and science also structures and disseminates culture. In order to better understand this, let's think about how the ludic activity exploring the potentialities of computer science has finally escaped the arts and the instituted sciences for a long time, even though games and arts anticipated together and restructured the bases of the future of interactive computer science, with an intensity and a power recognized today through games studies like the 10ème art.

This is also why recent initiatives such as issue 9 of the journal HYBRID, published at the end of 2022 in English and French,³² organize a cross and cumulative dialogue between scientific studies and artistic research, in this case around the figure of technological avatars offering their practitioners media incarnations to encounter informational environments or immersive environments.

It was during one of the dialogues at the heart of this review that object-oriented transversal approaches were able to discover their complementarity on both sides of the arts and sciences, and that a difference in temporality between artist-researchers preoccupied with the future and scientist-analysts more focused on the present or the past could be better formulated. The interest in tomorrow's artificial creatures, which should be welcomed and respected, to quote Edmond Couchot,³³ justifies many projects and experiments. Let's take up again the parallel with Licklider, when he specified that it was necessary to deal with the symbiotic partnership with the computer in the time interval that separated us from the arrival of "strong" AIs. In these still open times, before the promises of sentient robots or a perfect autonomized simulation of human beings are realized, there is every reason to take care of our partnerships with our avatars, by seeking together the conditions that will make us environmentally and behaviorally responsible. Hence the richness of symbiotic approaches, both in the virtual and in the real, since the two are constantly hybridizing in a way that is now inextricable.

IV) Conclusion in the form of programmatic questions, bias and references

The first question, of general scope, would be to know: "in what way does the avatar belong to a case of human-technological hybridization? For as we have detailed, the technological avatar achieves the integration of the living and the artificial in real time in the same iconic programmed object, in an animated body giving access to a virtual environment and manifesting the human to his fellow creatures of the same species and other quasi-species, virtual beings in the process of empowerment. But does this result in more than a hybridization of techniques between them, a mixing, a specific mixture, producing the interpenetration of stakeholders (subject, machines, programs and human and machine intentionalities)? And how, in return, does the sensitive and embodied internalization of technical logics in the human being take place through "existential" experiences, psychosocial and individualizing experiences (Simondon), with subjective and objectifiable experiences, echoing Edmond Couchot's technesthesia to the “cyberesthesia” proposed by the French artist Yann Minh, or even Macluhan and the media extension of the senses? But also, how to go beyond symbiosis as a metaphor for the living and elaborate its factual declensions, adapted and relevant to these hybrids of flesh and calculation that are the avatars? Consequently, how to configure their habitat, these virtual cyberworlds?

And finally, how can we configure differently, with the virtual, the technical conditioning and the cultural overdetermination, both inevitable, in order to achieve a common world that is equitable, sustainable and desirable in terms of reciprocal emancipation? The construction site is vast and the construction techniques to be implemented remain to be invented and popularized by this synergy between science and the arts that we are calling for. Let us be numerous and determined to invest ourselves in it for the benefit of all, human and non-human.


(1) In our theory, we prefer to distinguish techno-social (when living being are inside artificial construct) and socio-technical level (socialization of technical artefact).

(2) As the famous French philosopher René Descartes explains in his time in “Discourse on the Method of Rightly Conducting One's Reason and of Seeking Truth in the Sciences”, 1637.

(3) Named after the 1986 Habitat network games that referred to the inhabitants of this networked world as Avatars

References

1 Call of communication ISEA, 2023, online.

2 Vilem Flusser and Ann Roth Nancy, Into the Universe of Technical Images, NED-New edition, Vol. 32; University of Minnesota Press, 2011, accessed December 08, 2022, Ac http://www.jstor.org/stable/10.5749/j.cttttcq8.

3 William Regli, "Revisiting the Human-Machine Symbiosis," Computing Research Association Bulletin, April 2016, accessed December 08, 2022, https://cra.org/revisiting-human-machine-symbiosis.

4 Lynn margulis, Symbiotic Planet, Basic Books, 1998.

5 James E. Lovelock, Lynn Margulis, "Atmospheric homeostasis by and for the biosphere: the gaia hypothesis", Tellus, 26:1-2, 2-10, (1974), accessed December 08, 2022 https://www.tandfonline.com/doi/abs/10.3402/tellusa.v26i1-2.9731.

6 James Lovelock, Gaia A New Look at Life on Earth Oxford, Oxford University Press, 1979.

7 André Leroi-Gourhan, Gesture and Speech, The MIT Press, 2018, Translated by Anna Bostock Berger, First French Edition, Albin Michel, Paris, 1965.

8 Gilbert Simondon, Individuation in Light of Notions of Form and Information (Volume 1 & 2, Translated by Taylor Adkins, Univ of Minnesota Press, 2020, original French edition, Paris, Jérôme Millon, Collected papers of 50's, 2005.

9 Alain Berthoz, Simplexity: Simplifying Principles for a Complex World, Yale University Press, 2012. 

10 Maurice Merleau-Ponty, Phenomenology of Perception, Routledge, 2013, Translated by Donald Landes, original French edition, 1945.

11 Francisco J. Varela, Evan Thompson, Eleanor Rosch, The Embodied Mind: Cognitive Science and Human Experience, MIT Press 1991.

12 Licklider, J. C. R., "Man-Computer Symbiosis", Ire Transactions on Human Factors in Electronics 1 ,1960, 4-11. 

13 Philippe Descola, Beyond Nature and Culture, University of Chicago Press, 2014, Translated by Janet Lloyd, original french edition 2005.

14 Gilbert Simondon, On the Mode of Existence of Technical Objects, Translated by Cécile Malaspina, Univ of Minnesota Press, 2017, original French edition, Aubier, 1958.

15 Douglas C. Engelbart, William K. English, 1968, A research center for augmenting human intellect, In Proceedings of the December 9-11, fall joint computer conference, part I (AFIPS '68 (Fall, part I)), Association for Computing Machinery, New York, NY, USA, 1968, 395-410.

16 Claudia Salamanca, "The mother of all demos" in DAC'09, Digital Arts and Culture 2009, accessed December 08, 2022, https://escholarship.org/uc/item/91v563kh.

17 Gregory Bateson, Steps to an Ecology of Mind. St. Albans Paladin, 1972.

18 Chip Morningstar, F. Randall Farmer, The lessons of Lucasfilm's habitat, Cyberspace: first steps, MIT Press, Cambridge, MA, USA, 1991, 273-302.

19 Michel Bret, Marie-Hélène Tramus, Alain Berthoz, "Interacting with an Intelligent Dancing Figure: Artistic Experiments at the Crossroads between Art and Cognitive Science", Leonardo 38, no. 1, 2005, p.47-53, accessed December 08, 2022, http://www.jstor.org/stable/1577645.

20 Bob Rehak, "Playing at Being: Psychoanalysis and the Avatar." In The Video Game Theory Reader, 2003, 103-138, accessed December 08, 2022, https://www.academia.edu/3751080/Playing_at_Being_Psychoan alysis_and_the_Avatar.

21 Tom Boellstorff, Coming of Age in Second Life: An Anthropologist Explores the Virtually Human, Princeton, Princeton University Press, 2008, 316.

22 Louise MERZEAU, " Habiter l'hypersphère", Documentaliste-Sciences de l'Information, 2010/1 (Vol. 47), accessed December 08, 2022, 30- 31, https://www.cairn.info/revue-documentaliste-sciences-de-l-information-2010-1-page-30.htm.

23 Ted Nelson, Computer Lib/Dream Machines, Revised Edition, Tempus Books, 1987 (first edition,1974)
Felix Guattari, Three Ecologies, Translated by Ian Pindar & Paul Sutton, Bloomsbury Academic, 2014 (original French edition 1989).

24 Bernhard Rieder, "Beyond Surveillance: How Do Markets and Algorithms 'Think'?" The Foucaldian 3, no. 1, 2017, 1-20, DOI: https://doi.org/10.16995/lefou.30 [Note: In 2022, Le foucaldien relaunched as Genealogy+Critique.] accessed December 08, 2022 https://www.genealogy-critique.net/article/id/7055/.

25 Bernhard Rieder, Engines of Order: A Mechanology of Algorithmic Techniques, Amsterdam University Press, 2020, Accessed December 08, 2023, https://www.jstor.org/stable/j.ctv12sdvf1.

26 Marija Slavkovik, Clemens Stachl, Caroline Pitman, Jonathan Askonas, "Digital Voodoo Dolls", In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (AIES '21), Association for Computing Machinery, New York, NY, USA, 2021, 967-977, Accessed December 08, 2023, https://www.researchgate.net/publication/353603454_Digital_Vo odoo_Dolls.

27 Edmond Couchot, Images. From optical to digital. The visual arts and the evolution of technology, Paris, London, Lausanne, 1988.

28 Alan Kay, Adele Goldberg, Personal Dynamic Media. In: Wardrip-Fruin, Noah / Montfort, Nick: The New Media Reader, Cambridge MA, The MIT Press, 2003, 393-404.

29 Marie-Hélène Tramus, "Multisensoriality Interactive Digital Art, Presentation of an Experience: The Virtual Tightrope Walker" (paper submitted in ISEA 2000) Only Proceeding, 2000, 200-204, https://www.isea-archives.org/docs/2000/proceedings/ISEA2000_proceedings.pdf 

30 Dimitrios Batras, Judith Guez, Jean-François Jégo, InterACTE: Improvising with a Virtual Actor, In Proceedings of the 3rd International Symposium on Movement and Computing (MOCO '16), Association for Computing Machinery, New York, NY, USA, Paper 52, 2016, 1-2,
https://doi.org/10.1145/2948910.2955109

31 Bruno Latour, An Inquiry Into Modes of Existence: An Anthropology of the Moderns, Havard University Press, translated by Catherine Porter, 2018.

32 Etienne Armand Amato, Judith Guez, Etienne Perény, Marie-Hélène Tramus, "Prospective and retrospective reflections on avatars and virtual actors in artistic experimental exploration," Hybrid [Online], 9, 2022, online 30 November 2022, accessed 09 December 2022,  http://journals.openedition.org/hybrid/2808 

33 Edmond Couchot, Automata, robots and virtual humans in the performing arts, PUV, 2022

Authors Biographies

Étienne Armand Amato has been a lecturer in information and communication sciences at Gustave Eiffel University and the DICEN-IDF laboratory since 2014 (www.dicen-idf.org). Since the beginning of high-rate networks, he studies interactive audiovisual forms (video games, virtual reality), ludic communication and media avatars. He also contributes to the work of IHEST (Institute of High Studies for Science and Technology), a training organization of the French Ministry of Research.

Étienne Perény is honorary professor of communication and interactive design, a member of the laboratory Paragraphe at the Université Paris 8 de Vincennes and associated with the DICEN- IDF laboratory. Since the 70s, he experiments new technologies: video, architectural simulation, videodiscs, cable networks, interactive video prototypes, offline multimedia, then online, in the early days of the Internet. He has developed an original theory on interactive images and video games, and the research device "becoming avatar" with Alain Berthoz and E.A. Amato. 

Symphony of the Stones: A Research-Creation Exploration on the Animation of Heavy Metal Residues in Contaminated Urban Landscapes

Brice Ammar-Khodja

Concordia University
& École des Arts Décoratifs – PSL (EnsadLab)
Paris, Montreal
brice.ammar-khodja@mail.concordia.ca / brice.ammar@ensad.fr

Abstract

In the late 1980s, the Canadian Pacific Railway abandoned a rail yard on the outskirts of Montreal’s Mile End district. Within a few years, the return of animal and plant species encouraged the citizen community to reinvest this site known as Le Champ des Possibles. Despite community efforts to rehabilitate this site, hydrocarbon and heavy metal pollution persists in the soils and thus requires rethinking the engagement with the imperceptible mutations of ecosystems. Symphony of the Stones was created in response to this context. This research-creation project consists of several urban art installations that activate residual metals in soils by their magnetic characteristics to make these imperceptible pollutants visible. The following paper unfolds the different processes, methodologies and strategies that led to in site interventions blending art installation, collaboration with different communities and associations and leading to a rethinking of art practices in the urban environment.

Keywords

Digital arts, research-creation, art-science practices, situated artistic interventions, environmental installations, contaminated urban soils, active residual materials, material agency, pedological studies, environmental participation, socio-environmental issues.

DOI

10.69564/ISEA2023-40-full-Ammar-Khodja-Symphony-of-the-Stones

Introduction

Although invisible, heavy metals abundant in soils have the potential to contaminate natural environments, thereby increasing the risk of serious diseases. In recent years, the discovery of new polluted sites underlines the extent of this pollution. Over a century of industrial activities has led to this pollution spreading at an alarming rate throughout urban areas. Nowadays, 54 percent of the global population lives in an urban area,¹ and, in Quebec alone, 10 percent of the identified sites are located within Montreal.² In this regard, citizens' exposure to these toxic residues requires a rethinking of engagement with the imperceptible mutations of ecosystems due to anthropogenic activities.

Despite the fact that Karl Marx anticipated ecological urgency and soil loss by what he theorized as the "metabolic rift," soils have only recently gained interest from social and biological/geological sciences and science and technology studies (STS).³ On the other hand, the works of contemporary artists and artist-researchers such as Debra Solomon, Amy Balkyn, Martin Howse, Joana Hadjithomas, Khalil Joreige or Disnovation.org, illustrate this urgency by raising awareness on biodiversity loss and soil degradation. As the push intensifies to transition these discussions out of strictly academic and artistic realms, the pressing challenge becomes: how can we expand the conversation on soil contamination, and more broadly, urban pollution, ensuring it resonates with the general public, especially the communities and stakeholders most affected by it?

Écotones is a collaborative research-creation project which arose from this question. Led with artist Philippe Vandal and in collaboration with the citizen association Les Amis du Champ des Possibles and more precisely with the coordinator Sugir Selliah and the co-president and climate policy analyst Émile Boisseau- Bouvier, Écotones operates as an in situ urban laboratory. It melds experimental exhibitions and artistic interventions, articulating an aesthetic, critical, and social perspective on soil pollution. This research-creation project, blending art-science methodologies, seeks to explore heavy metals and hydrocarbon pollutant agencies to activate new experiences of soil pollution and offer critical visibility to these invisible residues. Two directions were taken: Symphony of the Stones (my own work) which explored heavy metal pollutants and Fluorescent Forensic by Philippe Vandal, which explored residual hydrocarbons. The first iteration of this project took place at Le Champ des Possibles in Montreal on October 14th. Le Champ des Possibles (CDP), is a former marshaling rail yard left as a brownfield by the Canadian Pacific Railway company in the 1980s and rehabilitated as a green space in 2013. Despite community efforts to rejuvenate this land, hydrocarbon and heavy metal pollution still lingers in the soil. The ongoing proliferation of these pollutants presents dire ecological and health risks that cannot be overlooked. As the site was formerly used as a waste disposal site, the municipality and the association les Amis du Champ des Possibles (ACP) have therefore agreed to rehabilitate the site by excavation under section 22 paragraph 9 from the Loi sur la Qualité de l'Environnement.⁴ However, few citizens are aware of these hazardous materials, whose existence deserves to be discussed. 

This article zooms in on Symphony of the Stones, an in situ art installation that seeks to enhance the understanding and perception of heavy metal residues in contaminated soils. The article contextualizes the making of the installation and the way it connects to CDP. First, the article focuses on recontextualizing what CDP is and how the concept "ecotones"—that gave the name to the series of interventions—emerged from exploring this site. The second part of the article delves into the art installation and how it activates soil's heavy metal contaminants to shine a spotlight on the invisible traces of pollution lying under the ground. Finally, the third part delves into the concept of public intervention, drawing inspiration from various contemporary in situ art installation practices.

Exploring the entanglements

Le Champ des Possibles as a space of negotiations

Le Champ des Possibles is located in the Mile-End district of Montreal. The district underwent significant gentrification, initially with the influx of artists in the 1980s and more recently due to the emergence of art galleries, design showrooms, and companies in the innovation and video game industries. Despite this socio-economic context, CDP remains in the state of a brownfield. The site is characterized by the entanglement of various plants, animals, and human communities living together with the traces and ruins of its past industrial activities. The foot paths laid by morning joggers and commuting wanderers are lined with campfire remains and industrial ruins (figure 1). Steel structures, now enveloped by diverse species of trees and plants, provide shaded resting spots for local workers. These makeshift sitting areas are occasionally rearranged by visitors to the site. The northern border of CDP neighbors a railway which traces a separation between the Mile-End and the other districts. For years, these fenced borders have been the subject of debate among the citizens, the municipal authorities and the Canadian Pacific Railway that owns the railroad.⁵

Breaches in the fence allow for easy crossing to the other side of the railroad and the reconstruction of this same fence by the public authorities are thus common things to observe throughout the year. Because of the diversity of uses and communities that make CDP a unique space, it is not easy to give a single definition of the site as it conjures up different imaginary perceptions. Facing this post-industrial biotope and the entanglement of different human and more-than-human figures negotiating the site with each other, it appears that CDP was scientifically reminiscent of a concept in environmental science that would help us formalize and conceptualize a dialogue with the space: the ecotone.

Figure 1. Picture of Le Champ des Possibles. We can observe the biotope of the site as well as the railway and the fence in the background of the picture © Brice Ammar-Khodja

Ecotone as a conceptual framework

Ecotone is a term that emanates from environmental sciences. It describes areas of gradation between different ecological communities, ecosystems or even regions. In other words, places where diverse ecologies are put in tension.⁶ Although ecotones are not restricted to the natural environment, they can also be induced by human activities.⁷ According to Jurek Kolasa and Maciej Zalewski’s book Ecotone attributes and functions,⁸ these contact zones are hard to define.  They point out that ecotones can range in size from a few centimetres to kilometres and can be distributed across spaces in different contexts with varying degrees of porosity and types of boundaries. Moreover, ecotones can be horizontal and extend over planar surfaces or vertical when localized in the atmosphere or water bodies.  This overlapping inspired us to build a conceptual framework that would foster insitu exploration of CDP. This allowed characterizing the area, not only at the ground level but also by understanding the different vertical layers and boundaries on the distribution of natural elements and pollutants in the soils. This notion of ecological boundaries guided the exploration of the site. Discussion with members of the citizen association ACP, highlighted that these ecological boundaries were overlapping with a map from the Quebec government classifying soils by degrees of contamination within CDP. The map shows a satellite view of CDP where zones are divided into polygon sections, each providing information on the name and concentration of pollutants, and the possible effects of these residual matters. In addition to this map providing technical information about the site, the ecotones framework became a tool for understanding the socio-environmental interactions of the site. While the map showed CDP's contaminants split into polygons, the notion of ecotones revealed other boundaries that would appear through the exploration of the site: different paths generated by the users of the site oscillating between gravel roads and tracks where the grass does not grow anymore, the green rehabilitation zones scattered on the site to allow the vegetation to grow back, the campfires where only the waste of late-night rave parties remain, and finally the railroad, which illustrates the most significant transition zone of this unique ecotone. Through this first overview, CDP appears as a space of socio-cultural and biological diversity in constant mutation. From a natural space transformed into a rail yard to a half-industrial, half-green public space, and finally, to a space in the process of decontamination, the transition of this landscape is not only remarkable for the diversity of communities, boundaries, and uses of the space, but also its transformations through space and time.

While a simple walk in CDP would reveal the aforementioned ecological boundaries, the former use of CDP as a waste disposal site left traces of which a map alone cannot give a complete account. As the entanglement of the many elements that compose CDP tends to blur what this site is composed of, the primary approach of the project consisted of developing art installations, unveiling imperceptible elements.

Sensing mutations

Residual invisibilities

Soils are dynamic systems composed of organic matter, living organisms, gases and water. When the amounts of naturally occurring elements, such as heavy metals (i.e., copper, lead, nickel, arsenic), increase abnormally due to external, and usually anthropogenic, factors, a pollution risk arises.⁹ When elements, not naturally occurring in soils, such as hydrocarbons, are found, the phenomenon is termed contamination.¹⁰ As with heavy metals, excessive levels of these external components can lead to pollution risks and disrupt the natural soil cycles. However, most of the dynamic interactions and transformations in the soils, even the most concerning, are not easily perceptible to the naked eye. Human relations with soils are largely limited to what human sensory organs can detect and interpret. Indeed, perception relies on a combination of the senses. Among these, vision, described by Plato as "the noblest of the senses," dominates as the "most informative" and "associated with the highly valued faculty of reason."¹¹ The effectiveness triggered by perception can contribute to increasing awareness, understanding, and care for soils. However, how can one foster these sensations when the soil's mutations are imperceptible? In Thinking with Soils, Material Politics and Social Theory, Anna Krzywoszynska, Manuel Tironi, Matthew Kearnes, Céline Granjou, and Juan Francisco offer socio-environmental and ethico-political ways to approach soils with a care-oriented analysis inspired by the work of Puig de la Bellacasa. In the book's second chapter, the authors point out that soil conservation is challenging because soil ”is commonly presented as invisible, hidden under our feet, and lacking a recognizable 'face.'"¹² Since soils are composed of microscopic elements (living beings, organic matter, minerals, pollutants, etc.), soils' intrinsically animated and living nature is not always evident to those who observe them. Thus, the authors insist that the complex nature of soils and their non-zoomorphic and non-anthropomorphic characteristics make them more vulnerable and, therefore, more at risk of being neglected and damaged. However, this mixture of non-human entities, whether organic or inorganic, animate or inanimate, possesses their agency and the potential to influence and affect systems of which humans are part. This notion relates to what Jeanne Bennett calls "thing-power", defined as “the curious ability of inanimate things to animate, to act, to produce effects dramatic and subtle.”¹³

Soils are discrete entities, and pollution is invisible. In some contexts, and environments, the biotope sometimes contributes to weakening the perception of pollution. Let us recontextualize this assumption: at first glance, CDP looks like a site in the process of green rehabilitation. Since the Canadian Pacific Railway company left the site as a brownfield, species of plants, trees, and animals have begun to reoccupy the site. This reoccupation might have paradoxically contributed to blurring the distinction between the past and present of the site; that is, a place undergoing green rehabilitation versus a place where pollutants still lie and mutate beneath the soil. Discussions with the citizen association ACP have confirmed these suppositions by revealing that many of the citizens living next to the site were unaware of CDP's pollution. Symphony of the Stones emerged in response to these issues. This research-creation project explores how the activation of heavy metal pollutants can operate on perception and senses to engage in a critical dialogue with the communities bordering these sites. As part of the series of artistic interventions titled Écotones, this installation draws from practices and interests in active materials, material agency, material forensic and art-science practices and focuses on generating critical discussions with the riverside communities on the issues related to CDP discussed in the next section of the article. 

Symphony of the Stones: a research-creation case study

Symphony of the Stones is a series of urban art installations that examine the correlation between the abnormally high presence of heavy metals and magnetic residues in CDP's soils. This first iteration was developed within the framework of the research-creation thesis, titled the Cycles of Attraction and derives its methodology from geomagnetism—a geosciences subfield specializing in identifying soil pollutants via magnetic fields.¹⁴ The installation reinvestigates this technique to activate heavy metal and magnetic contaminants in CDP's soils that present high responsivity to magnetic fields. The installation features two geometric aluminum sculptures, designed to harmoniously integrate with the landscape, reminiscent of the metallic remnants of CDP's past industrial activities (figure 2 and 3). 

Figure 2. First interactive sculpture exhibited during Écotones at Le Champ des Possibles ©Etienne Massicotte 
Figure 2. Second interactive sculpture exhibited during Écotones at Le Champ des Possibles ©Etienne Massicotte 

The sculptures are flush with the ground and strong enough to be walked on. Using the Quebec Government map indicating soil classification by degrees of contamination, these ephemeral sculptures are strategically placed over highly polluted polygons and topped with CDP's soils. Concealed within and below the sculptures is a kinetic system comprising N45-grade permanent magnets. These magnets, driven by DC motors and controlled by a microcontroller (figure 3), generate magnetic fields that create material tensions and attractions, activating the magneto- responsive constituents of the soils. This includes ferromagnetic residues, construction debris like nails, discarded municipal items like beer caps, and variously sized earth clumps. In a metaphorical sense, the installation seeks to visualize the invisible soil transformations. The movement of the soil creates a subtle yet captivating animation aimed at drawing attention to the ground. Concurrently, the installation delves into the acoustic properties of soils and residues. One sculpture integrates a piezo microphone, connected to a TDA2822M mini audio amplifier and a Bluetooth transmission system, relaying the audio signal to a discreetly placed Bluetooth speaker (figure 4). This system captures and amplifies sounds produced from the collisions of various materials. By amplifying these subtle soil movements, the installation adds an auditory dimension to the ambient sounds of the CDP environment. By animating both organic and inorganic residues and materials from CDP's soils—those seemingly inert to casual observation—Symphony of the Stones aspires to foster a renewed, sensory connection between humans and soils. The project is informed by the concept of "active materials" or “kinetic materials.” These materials can react and adapt by changing colours, textures or events” moving in stimuli such as heat, magnetic fields, or moisture. In Materials That Move: Smart Materials, Intelligent Design, Murat Bengisu and Marinella Ferrara explore the potential of several active materials features and how their application in design changes users' relations with everyday objects. Drawing inspiration from Donald Norman, an expert in the application of cognitive science to design and the theorist behind emotional design, Bengisu and Ferrara highlight the potential of active materials to "materialize intangible information that is imperceptible to the human senses in daily life."¹⁵

Figure 3. Kinetic system hidden in the sculpture’s shell. The horizontal rods containing the magnets are activated by motors and driving belts. © Brice Ammar-Khodja
Figure 4. Close up of the piezo microphone in contact with magnetic soils in motion © Brice Ammar-Khodja

The dynamic created by these objects, and materials, made possible by organic and organic-like movements, might signify the expression of something that is animated and living, triggering different senses and emotions. Exploring discarded materials, residues and contaminated soils coming "alive" by their activation, Symphony of the Stones aims to encourage the public to not only further observe soils but also get closer to the ground, carefully observe the different patterns drawn by the materials in motion and haptically engage. The project's first iteration showed promising results during the Écotones exhibition, where visitors would actually get closer to the installation and eventually touch the soil. The future versions of the installation will put more emphasis on the haptic aspect. As Puig de la Bellacasa points out in Matters of Care, haptic promises to develop new forms of knowledge based on attentiveness through the body's engagement.¹⁶ The knowledge that the vision alone, by its distant and detached character, could not reach.

Intrusions, furtivity, and symbiosis

Making an in situ urban art installation involves intervening in the environment. The first part of this paper explained why CDP is a space in constant negotiation between human and non-human communities. In this context, Symphony of the Stones explores new ways to engage with the environment and its communities, rather than merely subjecting the site to a single artist's vision. This approach required working closely with ACP for a deeper understanding of the site (see section "Exploring the entanglements"). However, as illustrated in the first part of this article, there is no consensus on what CDP is or should be. Out of respect for the site, Symphony of the Stones was designed to neither disrupt nor damage the site's environment. In doing so, the installation diverges from the traditions of environmental art, especially Land Art, an art movement from the 1960s that showcased open-air artwork—often large-scale installations using natural elements found on site. Land Art sought to foster a dialogue between art and natural environments, underscoring the localized aspects of works now displayed beyond the confines of museums and galleries. Prominent examples include Robert Smithson's Spiral Jetty (1970), a 457-meter- long basalt stone dike spiraling counterclockwise over slightly reddish water at the Rozel Point peninsula on the northeastern shore of the Great Salt Lake, or Walter De Maria's The Lightning Field (1977), a permanent installation of 400 steel poles spread over a mile, designed to attract lightning in a desert plain in Quemado, New Mexico. While sometimes classified as a form of ecological art, Land Art is not without controversy. Sometimes equated as a form of ecological art, Land Art is also controversial. Some interventions were indeed drastic, leading to artworks that altered their environments ¹⁷. Since then, myriad art forms have emerged, influenced by the practices of environmental and land art. Some extend their focus beyond the natural to encompass the built environment, with a growing emphasis on addressing socio-environmental and political challenges linked to the sites under consideration. Regrettably, a recurring issue with in situ urban or environmental art installations is the gap between the discourse of these practices and the methodologies they employ. In her article Le Lourd Bilan Carbone de l'Art Contemporain, Gill Gasparina argues that many contemporary urban and environmental installations, while raising awareness of ecological concerns, still rely on processes detrimental to the environment, such as pollutant technologies, excessive energy use, and waste generation.¹⁸ Given the increasing integration of new technologies and the creation of custom artworks, the carbon footprint of contemporary—and particularly digital—art is undeniable. This highlights the pressing need to further evaluate energy consumption and material repurposing in the pursuit of more sustainable artistic practices. 

Symphony of the Stones was conceived with the dual intention of minimizing its ecological and aesthetic impact on CDP. In pursuing this, Symphony of the Stones is influenced by what art critic and art historian Patrice Loubier terms “furtive practices.” Furtive practices are described as “multiple art practices that infiltrate the urban landscape through works without signs, not necessarily recognizable as artworks, serendipitously surprising the pedestrian, transforming them into an incidental observer” (My translation).¹⁹ In contrast to Land Art, furtive practices do not dominate due to their scale; instead, they subtly alter details within these environments. Underlying these practices are a range of artworks whose subtlety setting does not diminish their political, social, or environmental significance. Drawing inspiration from Danae Stratou’s Concentric (2019), a piece that produces continuous concentric ripples in Nirox Sculpture Park’s Lake, Symphony of the Stones integrates seamlessly with its surroundings, invoking elements already present. The aluminum's polished surfaces help to camouflage the sculptures by mirroring the environment around them. Moreover, the installation is largely enveloped by CDP’s soils, which are retrieved from the topmost layer using DIY (Do It Yourself) permanent magnet soil collectors, thus avoiding any digging or excavation. Neither of the two sculptures, nor their concealed kinetic systems, needed anchoring or attachment. This adaptability ensures the installation's portability and ease of removal. Such features position Symphony of the Stones as an artwork that may elude immediate notice, nudging site visitors to be more observant of their surroundings. Yet, the installation's auditory aspect offers clues about where and what to focus on. The initial display of the artwork yielded varying reactions based on CDP's ambience and foot traffic. On certain occasions, passersby might overlook the sculptures entirely, even walking over them without pausing. However, when the contours of these metal forms caught their eye, some would halt, scrutinize the sculpture more closely, and approach upon observing the animated soil atop the structures. During the Écotones outdoor exhibition, the influx of visitors meant that some remained oblivious to the sculptures. Nonetheless, those who became aware of them through the animated soil often ventured closer, with a few even interacting directly with the residues by touching them (figure 5 and 6). 

Figure 5. One of the Symphony of the Stone’s kinetic sculpture exhibited during the exhibition Écotones ©Etienne Massicotte 
Figure 6. Close up to the residues activated by the kinetic magnetic system © Etienne Massicotte

After the opening, we organized a reception in a space adjacent to CDP, and allowed the installation to remain on site into the evening. Upon our return to disassemble the installation, the darkness had enveloped CDP, and a group of night-time explorers had kindled a campfire approximately a meter away from the sculptures. In spite of the resonating sounds and the movement of the residues, this group remained oblivious to the installation. By this attempt of creating an installation furtively within the site, it is hard to evaluate what are the best conditions for it to be perceived and understood in its message but also as a work of art. A consideration for future iterations of this installation, and more generally within the domain of furtive practices, might involve implementing strategies for artistic and cultural mediation for artworks displayed in urban settings.

Conclusion

This article sketches the premises of a methodological framework for research-creation practices that operates at the scale of urban spaces and more specifically spaces in the process of rehabilitation. Although contextualized on a specific site in Québec, I hope that this contribution will foster new ways of collaboration with outdoors spaces and hopefully participate in the debate on urban soil pollution. The CDP’s case study has demonstrated that wild urban spaces, brownfields, and any other grey area are places of entanglements and negotiation between different communities. Amongst these communities that range from human wanderers and species of plants to toxic residues, different types of agencies whether positive or negative, might emerge or persist. Against this backdrop, a question emerges: how can artistic endeavors be realized in harmony with these intricate interconnections? By highlighting presence of heavy metal pollutants in the soil, Symphony of the Stones opted to harmonize with the CDP. By suggesting new forms of non-intrusive collaborations with the site and its communities, this research-creation project seeks to activate and increase the perception of contaminants. In this regard, the project does not only seek to redraw borders on the site, but rather amplify its already existing aspects. As part of the project Écotones, this first iteration is perhaps not immune from mutating towards a more radical but still furtive kind of intervention in the urban landscape.

References

1 Food and Agriculture Organization of the United Nations, International Fund for Agricultural Development, International Organization for Migration, and World Food Programme, The Linkages between Migration, Agriculture, Food Security and Rural Development: Technical Report, 2018, 5.

2  Gérald Dermont, “Sols Pollués Par Les Métaux Lourds Résultant De L'enfouissement De Déchets Industriels (montréal, Canada) : Géochimie, Spéciation Des Métaux, Et Décontamination Par Flottation,” Dissertation, Université du Québec, Institut national de la recherche scientifique, 2008.

3 Juan Francisco Salazar, Céline Granjou, Matthew Kearnes, Anna Krzywoszynska,Manuel Tironi, eds, Thinking with Soils: Material Politics and Social Theory, London, UK, Bloomsbury Academic, 2020, 4-32.

4 Les Amis du Champ des Possibles, Interview by Brice Ammar-Khodja, December 6th, 2022 and October 14th, 2022.

5 Ibid.

6  Robert Leo Smith, Ecology and Field Biology, 4th ed, New York, Harper and Row, 1990, 251.

7 Salit Kark, Rik Leemans, Environmental Systems Analysis Group, Wageningen University, Wageningen, 6700 AA, Netherlands, 2013. 
“Ecological Systems: Selected Entries from the Encyclopedia of Sustainability Science and Technology,” Essay, In Ecotones and Ecological Gradients, New York, NY, Springer, 147–60.

8 Jurek Kolasa, Maciej Zalewski, “Notes on Ecotone Attributes and Functions,” Hydrobiologia: The International Journal of Aquatic Sciences 303, no. 1-3, 1995, 1–7.

9 Peter M Chapman, and Sixth Iberian and 3rd latinoamerican Congress on Contamination and Environmental Toxicology Cadiz (Spain) 20050925-20050928. “Determining When Contamination Is Pollution — Weight of Evidence Determinations for Sediments and Effluents.” Environment International 33, no. 4, 2007, 492– 501.

10  Ibid.

11  David Howes, “Sensory Craft,” Craft Forward, California College of the Arts, San Francisco, 2 April 2011.

12 Francisco Salazar, Céline Granjou, Matthew Kearne, Anna Krzywoszynska, Manuel Tironi, 17.

13 Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press, 2010, 6.

14  Xue-Feng Hu, Li Mei, Zichen He, Lei Cui, Liu Rui, Xin-Dong Wang, Zhi-Han Wang, “Magnetic Responses to Heavy Metal Pollution of the Industrial Soils in Shanghai: Implying the Influences of Anthropogenic Magnetic Dustfall on Urban Environment,” Journal of Applied Geophysics, 2022197.

15 M. Bengisu, Marinella Ferrara, Materials That Move: Smart Materials, Intelligent Design, Springerbriefs in Applied Sciences and Technology, 1, 2019.

16 Puig de la Bellacasa María, Matters of Care : Speculative Ethics in More Than Human Worlds, Posthumanities, 41, Minneapolis, University of Minnesota Press.

17  Virginie Luc, De La Nature À L'œuvre, Paris, Ulmer, 2014.

18  Jill Gasparina, Le lourd bilan carbone· de l'art contemporain, Le Temps, 8 janvier 2019, https://www.letemps.ch/culture/lourd-bilancarbone-lart-contemporain

19  Patrice Loubier, “Un Art À Fleur De Réel : Considérations Sur L’Action Furtive,” Inter 81, no. 81, 2002, 12–17.

Bibliography

M. Bengisu, Marinella Ferrara, Materials That Move: Smart Materials, Intelligent Design, Springerbriefs in Applied Sciences and Technology, 2019.

Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press, 2010.

Peter M. Chapman, and Sixth Iberian and 3rd latino american Congress on Contamination and Environmental Toxicology Cadiz (Spain). “Determining When Contamination Is Pollution — Weight of Evidence Determinations for Sediments and Effluents,” Environment International 33, no. 4, 2007, 492–501.

Gérald Dermont, Sols Pollués Par Les Métaux Lourds Résultant De L'enfouissement De Déchets Industriels (montréal, Canada): Géochimie, Spéciation Des Métaux, Et Décontamination Par Flottation, Dissertation, Université du Québec, Institut national de la recherche scientifique, 2008.

Food and Agriculture Organization of the United Nations, International Fund for Agricultural Development, International Organization for Migration, and World Food Programme, The Linkages between Migration, Agriculture, Food Security and Rural Development: Technical Report, 2018.

Jill Gasparina, “Le lourd bilan carbone· de l'art contemporain,” Le Temps, January 8, 2019, https://www.letemps.ch/culture/lourd-bilancarbone-lart-contemporain.

David Howes, “Sensory Craft”, Craft Forward, California College of the Arts, San Francisco, April 2, 2011.

Xue-Feng Hu, Li Mei, Zi-Chen He, Lei Cui, Rui Liu, Xin-Dong Wang, Zhi-Han Wang, “Magnetic Responses to Heavy Metal Pollution of the Industrial Soils in Shanghai: Implying the Influences of Anthropogenic Magnetic Dustfall on Urban Environment,” Journal of Applied Geophysics, 2022, 197.

Kark Salit, Rik Leemans, “Ecological Systems: Selected Entries from the Encyclopedia of Sustainability Science and Technology,” Essay, In Ecotones and Ecological Gradients, New York, NY, Springer, 2013, 147– 60.

Jurek Kolasa, Maciej Zalewski, “Notes on Ecotone Attributes and Functions.” Hydrobiologia: The International Journal of Aquatic Sciences 303, no. 1-3, 1995, p.1–7.

Les Amis du Champ des Possibles, Interview by Brice Ammar-Khodja, December 6, 2022, and October 14, 2022.

Virginie Luc, De La Nature À L'œuvre, Paris, Ulmer, 2014.

Patrice Loubier, “Un Art À Fleur De Réel: Considérations Sur L’Action Furtive.” Inter 81, no. 81, 2002, 12–17.

María Puig de la Bellacasa, Matters of Care: Speculative Ethics in More Than Human Worlds. Posthumanities, Minneapolis: University of Minnesota Press, 41.

Juan Francisco Salazar, Céline Granjou, Matthew Kearnes, Anna Krzywoszynska, Manuel Tironi, eds., Thinking with Soils: Material Politics and Social Theory, London, UK, Bloomsbury Academic, 2020.

Robert Leo Smith, Ecology and Field Biology, 4th ed., New York, Harper and Row, 1990.

Aknowledgements

I would like to thank the Canada Excellence Research Chair (CERC) on Smart, Sustainable and Resilient Communities and Cities at Concordia University, the Concordia Sustainability Action Fund (SAF), the research-creation international network Hexagram, and the Haute École des Arts du Rhin (France) and the Arrondissement Le Plateau-Mont-Royal for their support and funding. Also, I would like to thank the Concordia University Research Chair in Critical Practices in Materials and Materiality, the Milieux Institute, the Topological Media Lab, and the EnsAD, EnsadLab’s Reflective Interaction Research Group for providing conceptual and technical support. Finally, I am thankful to the citizen association Les Amis du Champ des Possibles, the Arrondissement Le Plateau-Mont-Royal and artist Philippe Vandal for making this collaborative project possible.

Author’s Biography

Brice Ammar Khodja is an artist, graphic designer, and Ph.D. student based in Montreal and Paris. His work examines active materials, residual matter, and low-technologies to explore the socio-environmental and political interconnections pertaining to materiality and visual information.

He is currently pursuing a thesis jointly supervised in Concordia University – Montreal (Individualized Program) and EnsAD, EnsadLab – Paris (Reflective Interaction research group, SACRe program).

He is a member of Chaire Arts & Sciences, Concordia University Research Chair in Critical Practices in Materials and Materiality, Speculative Life Research Cluster (Milieux Institute), Centre for Sensory Studies, and Concordia’s Canada Excellence Research Chair in Smart, Sustainable and Resilient Communities and Cities. Brice is a current member of the international research-creation network Hexagram.

Co-director of the typography magazine Pied de Mouche, Brice Ammar-Khodja, creates workshops and educational tools for the general public.

His works have been exhibited at Ars Electronica (Linz, Austria), MUTEK (Montréal, Canada), Centre Pompidou (Paris, France), Biennale internationale du Design, la Cité internationale des Arts (Saint-Étienne, France), V2_Institute for Unstable Media (Rotterdam, Netherlands), Musée historique de la Ville de Strasbourg (Strasbourg, France) and Second Internatinal Print Biennale (Yerevan, Armenia). 

The “Bichi” Project Symbiotic Food Networks & the Alchemist Kitchen

Pat Badani

Independent Artist/Researcher @ ISEA International Board Argentina/Canada/USA pat.badani@gmail.com

Abstract

The author uses symbiosis as metaphor and tool to create food-related projects that critically examine the anthropogenically induced impact of global warming on food chains. She argues that "art-through-food" projects promote an alternative worldview informed by the utopian premise that art can facilitate reflexivity and influence behavior to prevent future massive starvation by safeguarding the future of food. Three projects reveal the workings of energy flows in ecosystems to reimagine hybrid relations with living matter and with systems that are biological, technological, social, and political. They establish sustenance networks in which symbiotic interactions between several actors form the basis for a reevaluation of our human relationships to the planet and the fungal, plant, and animal agents within it. Based on assemblages with multiplicities that cofunction via a viral logic of contagion, projects explore aspects of symbiosis in human/plant/bacterial consortiums (the Bichi project), as well as symbiotic intelligence combining computing with neural networks (@Comestiblemealplan). Informed by research linking various scientific fields with art and technology, projects explore networks of relations between entities tangled in interdependence and involving parasitism, mutualism, adaptation, and resilience. Congruent with a larger systems approach in which symbiosis is the core principle replacing an essentialist conception of individuality, hybrid works contribute research and knowledge production with the potential to assess and generate affective insights and acts in the world.

Keywords

Energy flows, ecosystems, art-through-food, future food, sustenance networks, symbiotic consortium, consilience, speculative fictions, utopia, climate emergencies, practice-as-research.

DOI

10.69564/ISEA2023-41-full-Badani-The-Bichi-Project

Art-through-Food - Symbiotic consortiums

The world is made up of complex and sometimes unmappable landscapes of interacting biological, climatic, social, economic, and political forces. Interconnected global systems have been created to produce, reproduce, and consume our material environments—systems in which not all partners have derived reciprocal benefits. How these systems are linked to human and more-than-human entities and the planet, have intrigued me for decades and the aim of this paper is to discuss how my research on nutritional networks is transformed into critical artistic language. As lens, I use the concept of "symbiosis" seen as relationships of coexistence and synergistic evolution that include interchange, cooperative processes, power relationships, contact, penetration, conversion, and last and not least, eating others and being eaten.

I frequently use food as material and subject in projects driven by research into climate change, creature welfare, social disparity, and food inequality. These projects involve scientific exploration and the creation of hybrid consortiums with living materials (foodstuffs) and systems that are biological, social, and political, to critically reimagine human/non-human relationships in symbiotic dynamic interplay. They are independent practice-as-research productions that integrate the technological, the ecological, and the edible, and are considered "art through food"¹—rather than the more broadly defined "food art" which has a long history in the visual arts, with representations of food from ancient Greek art where depictions of food were intended to denote a civilized lifestyle that separated them from barbarians, to Medieval food art in which artists of the time used images of food to illustrate differences in the eating habits of the wealthy and the poor, to the well-known mannerist work by 16th-century Giuseppe Arcimboldo or, much later, the 20th century illustrated cookbook by Salvador Dalí “Les dîners de Gala” with images intended primarily as aesthetic, sensual, amusing, and decorative representations providing a spectacular feast for their eyes.

On the other hand, works that fall under the category of ‘art-through-food’ imbue the experience of food with critical meanings: cultural, social, and political. A historical overview on the use of food in 20th century art shows that for many artists food became a suitable instrument of critique, a means to enter a conversation about broader issues. These projects often assume an educational and even activist stance, at times also involving audience participation, and are intended to extend public knowledge and engagement. For example, Tomasso Marinetti’s “Futurist Cookbook” (1937), a political manifesto advocating a type of cuisine that would embody the new progressive ideals of modernism; or Andy Warhol’s Campbell’s Soup series (1962) where soup becomes "brand" to denote consumer society; not to mention the use of food in staged photography, the moving image, and computer-generated images related to social practice and community art, where food has been used to explore the promise of gift relations and notions of hospitality and commensality. Some of these involved relational aesthetics with artists in the role of facilitators rather than makers, like Rirkrit Tiravanija’s “Pad Tahi” (1990), and Mella Jaarsma’s “I Eat You Eat me” (2001-InCUBATE), or projects focused on mapping as social practice such as “Data Cuisine Collective” by Moritz Stefaner exploring food as a medium for information (2011-ongoing). And, finally, in social media where meal photos showcasing an array of eating behaviors became emblematized in Twitter, Facebook, and Instagram’s ‘food selfie’ phenomenon, for example, Pat Badani’s “Comestibles” (2010) and Joy Garnett’s “Kitchen Studio” (2010) addressing the fuzzy line between media and genres in Web 2.0 society, and the pervasiveness of these technologies in our lives from making art to sharing our food. Furthermore, the reassessment of our relationship with the material world prompted by developments in quantum physics showing that matter we previously thought of as 'inert' is in fact made up of vibrating strands of energy—prompted a revision of relationships between matter and materials, with intellectual approaches that consider matter as possessing agency and qualities associated with generative becoming.² “Live” bio-art that often combines the aesthetic and the artistic with organic, often living matter, further complicates this association in a still developing genre ranging from works grounded in artists’ manipulation of genetic sequences to works in which biological organisms are not treated as material, but rather as symbiotic partners in creation.³ Critical Art Ensemble’s “Free Range Grain” (2003) and Center for Genomic Gastronomy’s “Cobalt 60 Sauce” (2011) are notable examples of bio-artists who explore the overlapping boundaries between art, food, ecology, and geopolitics.

The mentioned progression of ideas since the 20th century, linking art, technology, science, society, its objects and its subjects, embody moral, economic, and social codes of meaning that provide a reflective backdrop for me to assess symbiotic relationships in the creation of recent projects. In these works, I extend the category of symbiont to welcome non-human living entities and their energy flows, both organic and electronic, in processes of coexistence that allow me to assess the quality of relatedness manifested in their exchange, whether commensal, competitive, or parasitic. This set-up, marked by the synchronous presence of growth, maturation, and decay,⁴ allows me to pay attention to differences in power relations and positionality, as well as to imagine what non-binary relations that include non-human acquaintances might be like.

Art-through-Food and knowledge production

Jane Bennet claims that “Food, as a self-altering, dissipative materiality, is also a player. It enters into what we become. It is one of the many agencies operative in the moods, cognitive dispositions, and moral sensibilities that we bring to bear as we engage the questions of what to eat, how to get it, and when to stop.”⁵ Indeed, eating reveals not only the interdependence of humans and edible matter, but also a capacity to effect social change. Food is integral to existence and its manipulation since the Industrial Revolution has caused the world’s greatest imbalance due to the immense environmental footprint involved in systems of food production and distribution. My investigation into discourses driving nutritional systems from WW2 to the present, has led me to evaluate the quest for mastery over bodies and environments—behavior conceptualized through ideas of progress, beauty, domestication, and profit. These ideas permeated by the dominance of humans over non-humans—supported by systems like the agricultural and food processing industries—have resulted in the collapse of ecologies large and small. For decades and even centuries, human and more-than-human entities have been the sites of battles where various ideologies and scenarios have been deployed involving the foods humans consume day-to-day. Shifts in dietary patterns over time denote changing lifestyles that are driven by a variety of factors with two major ones being urbanization and the industrialization of food. In fact, the latter is pervasive and increasingly dominated by complex agri-food networks and global food supply chains. In this system, food has become an object of political negotiations on a national as well as a transnational level—a situation that questions the sustainability of industrial food production, distribution, and consumption. These global systems support agricultural practices that pollute or degrade the quality of lands and soils, contributing to an imbalanced planetary scenario with massive overproduction and abundance coupled with scarcity of nutritious foods. And, despite the relatively recent rise of alternative food movements, food safety programs, and plans proposing delivery of more abundant and nutritious foods world-wide, what often predominates is agri-business’ grip on food resource narratives—narratives based on capital gain rather than needs—a situation that suggests new opportunities for interventions into the system like the initiatives led by artists committed to "art-through-food." These art initiatives often include research, participatory strategies, and the creation of speculative fictions that, like in many of my own projects, aim to assess and mitigate consumer confusion and uncertainty caused by the over-saturation of recommendations on "good" or "bad" regimes—contradictory information, often supplied by food industry players.

Art-through-Food projects contribute to research and knowledge production with potential to measure and construct affective insights into spaces of opposition, tension, and conflict. Ecocide, social isolation, racial injustice, gender rights, and economic disparity are spaces at the threshold of potential, possible collapse, instigating in some, the desire for new ways of being in the world based on awareness of the transcendent dimension interconnecting all living things, and prompting the development of new, symbiotic relations not just between humans, but also with other-than-humans. In this regard, art-through-food projects play a crucial role in promoting a better awareness and understanding of the root problems, and help scale and spread approaches that increase health, equity, and well-being, while also protecting the ecosystem.⁶

Positioned among these art initiatives, my recent projects cultivate ideas about the construction of sustainable, symbiotic futures, and are at times suggestive of Si-Fi tropes and speculative visual narratives. Through culinary arts, photography, and 3D simulation software, 3D printing and physical computing, I create critical visual arguments that blend aesthetics and imagined futures that reaffirm the material world’s tendency to persistently organize new forms and create anew, despite failures, in a determined will to live. "Bichi" ⁷,⁸ is one such project into which several works are nested involving scientific investigation, macrophotography, and 3D simulation software.

The “Bichi” art project

Bichi involves creative collaboration with food science and technology and extends my decades-long investigation into ecological concerns and the entangled indicators pertaining to sustenance networks and the foods we eat. Nested in the Bichi project, Bichicuchitu, Bichicu-Chow and BichEden are hybrid works positioned between the organic and the computational and explore the intersectionality of energy flows, food systems, and climate emergencies. Executed with 3D simulation software, the animations expose the porousness of boundaries between what might appear as separate registers—subject/object, fact/fiction, and science/culture. Inspired by forms of animism, herbalism, and panpsychism, the works provoke cognitive dissonance in the viewer by suggesting provisional boundaries between entities, organisms, and species. They represent analytical inquiries into the symbiotic interrelation of our metabolic pathways, biological food chains, and global networks—questions that are at the forefront of imaginaries in times when massive starvation may become part of our reality due to the impending ecological disasters that jeopardize the future of food. These works explore the notion of exchange and of mutual transformation through an amalgamation of matter that meet each other to form new speculative life forms grounded on the notion of "assemblages." Operating at many different levels of affective bodies, the individual human body—my own body being an assemblage in and of itself - is seen as part of this assemblage.⁹

Bichicuchitu & Bichicu-Chow /
The alchemist’s kitchen / Symbiotic networks

For me, the action of artmaking is like cooking in an alchemist’s kitchen—with many projects born in the kitchen and further developed in other environments and mediums in processes that involve assembling and collecting things together, witnessing rupture or disturbance, and remaining open to unexpected events that give new access to the real. I see my kitchen as a private laboratory—a special theatre in which I envision and negotiate, not only new knowledge, but also new politics around new actors entering my world, conditions exemplified by the “Bichi” project.

Figure 1. Incu-Bichu, 2019-21, a series of bio forms cultivated in a domestic fridge. Digital print 13” x 43”, ©Pat Badani, used with permission.

At the onset of the COVID pandemic in 2019, I observed mold and fungi gradually take over decaying vegetables and fruits that I had soaked and disinfected before refrigerating, as was typically recommended at the start of the epidemic. Surprisingly, a bio-based symbiotic interaction unfolded with the organic matter transformed into peculiar trans-species assemblages [ Figure 1]—bio-sculptures seemingly infinite in their variability because bacterial communities are known to strongly differ across plant species, likely due to variations in metabolites, physical characteristics, and symbiotic interactions with the host plant and other microbial inhabitants.¹⁰ This phenomenon became a guided experiment involving a bio-based process of culturing and decomposing in a domestic fridge, a DIY method that supports the importance of scientific tinkering as art practice. Said method established a discreet sustenance network between 3 interdependent units: bio-sculpture material, mold, and fungi, and myself (my consciousness) as sensory symbiont. Through time, I witnessed the synchronous presence of growth, maturation, and decay, as death of the organic tissue was accompanied by the thriving growth of the infecting bacteria, mold, and fungi; a process in which different bodies, organisms, and species were in the course of ‘becoming’ with each other through ongoing negotiations that involved nourishment, resilience, and transformation—a process in which the notion of ‘predation’ is seen as a renovative act of living together in extended material relationships. As a key partner in bringing the microbial symbionts into aesthetic perception, I documented the DIY bio forms with macrophotography,¹¹ spawning a provocative series of images of living creatures that begged to be animated—or better yet mutated into two works that were created with 3D simulation software. Playfully titled Bichicuchitu [Fig.2] and Bichicu-Chow [Fig. 4], the names point to my native language (Argentinian-Spanish) in which the vernacular term ‘bicho’ is used generically to name any "critter" or unidentifiable creature, big or small.

Drawing associations between living organisms, sculpture, photography, and 3D computer modeling, the artistic process happened in layers through observation and re-creation. The passage from one media to the other in the constellation revealed differentiated yet interconnected aspects of artistic creation. They involved evolution of new forms that, through transition and transposition, unfolded expressive intensity that intermingled their boundaries and highlighted the role of technology in fathoming emotion in new forms of visual fictions. In this regard, I consider art as a tool for mediation that makes possible reflexivity concerning complex phenomena that contribute to awareness about the endangered future of food caused by climate change and exacerbated by the food chain disruptions caused by the COVID-19 pandemic and ongoing geopolitical crises. 

Figure 2. Bichicuchitu, 2019-20, still image extracted form an animation created with 3D simulation software. Digital print 14” x 24”, ©Pat Badani, used with permission, https://patbadani.net/pro_Bichicuchitu_1n.html

Bichicu-Chow and social media networks – Paradise and Anti-Paradise

In keeping with holistic approaches to natural life and scientific, cosmological, and social ideals, the checkered "quilt" that enrobes the 3D modeled creature in Bichicu-Chow [Figure 5] is taken from my digital intervention titled @Comestiblemealplan [Figure 3],¹² a tactical media work that considers the problematic logic driving notions of novelty, progress, and endless growth promoted in social ecosystems. Food is an integral part of human existence and one of the most fruitful areas of manipulation, and in my Instagram-based work I confront this trend deployed in social media food-narratives. Images, chronicles, and advertisements are voraciously circulated and consumed in an environment that allows me to probe the symbiotic exchange of energy and information, and to furthermore inspect how the newly generated energy realizes symbiosis evolution. In @Comestiblemealplan, the ecological chain structure of the ecosystem is composed of the symbiotic flow of information facilitated by the network structure, and said structure is like that of ‘natural’ ecosystems with producers, consumers, and decomposers that realize the flow of matter and energy through these three main elements in stages of formation, growth, and maturity.¹³

My probe began with the selection of digital images of agro-industrialized foodstuffs which I transformed through digital glitching processes. “Glitching" indicates the purposeful introduction of "noise" in a system resulting in aberrant and apparent ‘abnormal’ renderings of technologies.¹⁴ The back-end code driving digital images of foodstuffs were re-codified with the introduction of texts from literary sources that showcase the centrality of discourse about food production, distribution, and consumption ranging from Paradise Lost by John Milton (1667)¹⁵ to The Edible Woman by Margaret Atwood (1969)¹⁶ to Como agua para chocolate by Laura Esquivel (2001). The glitching process produced unexpected digital disruptions woven into what became @Comestiblemealplan’s Instagram feed—a checkered canvas that was later transposed as a digital "quilt" covering the 3D simulated creature in Bichicu-Chow.

In @Comestiblemealplan, the social media platform and the public form a symbiosis relationship through the flow of information, and because of the two-way communication mechanism with feedback, the platform opened the possibility of engaging a conversation about money, hope, and experience in relation to digital commensality and nutrimental systems. This energy flow favors the co-creation of transformation narratives in support of possible better futures. 

Figure 3. @Comestiblemealplan, 2020, still image extracted from Instagram’s feed - an interactive and participatory tactical media artwork created by Pat Badani, composed of digital Glitches of foodstuffs using Instagram as a canvas. ©Pat Badani, used with permission. https://patbadani.net/pro_Comestible_Instagram_1.html
Figure 4. Bichicu-Chow, 2019-20, still image extracted form an animation created with 3D simulation software. Digital print 14” x 24”, ©Pat Badani, used with permission. https://patbadani.net/pro_Bichicu_chow_1.html

Ecosystems / Future Food / BichEden

An ecosystem is a group of entities that live and communicate with each other in specific communities and environments. In "natural" ecosystems, symbiotic units are the basic components of energy production and exchange that make up a symbiosis relationship. Forming a symbiosis relationship of coexistence and synergistic evolution implies that the need for survival between entities in an ecosystem is inevitably interdependent and interactive in some way.¹⁷ Although ecosystems vary greatly in appearance, structure, and feature, for the most part, they are composed of three main elements: producers (such as plants), consumers (an organism that eats carnivores, herbivores, parasites, and scavengers), and decomposers (a recycling system made of organisms such as bacteria, fungi, earthworms, etc.). Ecosystems realize the smooth flow of matter and energy through these three main elements in which energy and matter, in the form of food, flow through the ecosystem. A food chain describes who eats whom, that is to say, the method by which a particular organism gathers its food, a chain that represents the flow of energy from one organism to the next and to the next, and so on (grass, a producer, produces its own food from sunlight; rabbit, a consumer, eats grass; fox eats rabbit; eagle eats fox, and so forth throughout the chain of organisms in the ecosystem). Typically, interruption to one element of the ecosystem produces waves and ripples that touch every member of the system, and for survival of sustenance networks, human and more-than-human populations need to interact in balance with each other because when connections between them become unbalanced, the likelihood of ecosystem collapse increases. For example, in marine ecology, coral reefs are at the base of ocean food chains. When reefs collapse some organisms have no food, such as shrimp, a critical food source for fish. Their decline is causing a sequence of disruptions in the food chain that affect not only fish, but also wildlife, and ultimately humans who rely upon seafood for sustenance, medicines, and income. To subsist, both humans and non-humans need the energy transmitted through a healthy, undisrupted food chain, yet current conditions create food scarcity and jeopardize the future of food.

Indeed, eating reveals the interdependence of humans and edible matter, yet the greatest imbalances for human and environmental health has been caused by agro-industrial manipulation of foodstuffs and the immense footprint involved in production and distribution systems. Agro-industrial practices contribute to global warming that drives changing climate patterns, extreme weather events, loss of biodiversity, droughts, and nutrient deficient soils. These are acutely felt by those who depend on the stability of farming ecosystems. For example, record temperatures and prolonged droughts in 2022 led to sharply reduced crop yields in many regions, with the impact likely to extend through 2023. The drought in the Horn of Africa (Ethiopia, Somalia, Djibouti and Eritrea) has led millions of people to move from the stricken areas in search of food. Considering these examples, the future of food has become a serious question facing all of humanity.

Food security is at the forefront of my concerns with questions such as: What will we feed a growing population in times of growing climate catastrophes? How will we stave off massive global starvation? What will we eat and who will grow our food in the future? I am curious about an emerging ‘connected food ecosystem’ in which social, economic, scientific, and technological advances explore ways to feed a global population based upon sustainable, networked relationships. Evidence of these revolutions can be found everywhere from the rising generation of change- makers exploring regenerative bioregional cultures and economies of care through traditional ecological knowledge; "back to the future" wisdoms such as regenerative permaculture imbued with the critical skills needed to feed, heal, and nurture communities; "farm to fork" strategies that aim to make food systems fair, healthy and environmentally friendly; food coops and farmers markets who reimagine food systems by promoting local sourcing and food transparency; "citizen science" with amateur-scientists in the fields of biology and conservation participating in projects that also educate the public about scientific processes and increase awareness about the move towards sustainable diets and a sustainable food system; and last but not least ‘food engineering’ involving innovative science-based solutions to producing food through genetics. These concerns and possibilities spawned BichEden, a new project bolstering my commitment to investigating encounters between organic and technological formations. I began developing "still life" artifacts that explore the notion of "vibrant matter,"¹⁸ and in collaboration with Mariel Martinez (software engineer and co-founder of Poetry+AI=Art) I began to develop digital assemblages envisioned for several outputs, ranging from a series of 3D animations to a responsive installation that evokes an imaginary, symbiotic inter-species scape.

BichEden: Plankton [Figure 5], currently in early stages of research and production, explores the layered contexts within which new qualifications of life are being determined and manipulated. The sculpture does not represent what already exists, it invites unexpected convergences; it imagines potential edible life where fusion and remixing are the result of symbiotic, ecological thinking.¹⁹

At the base of the work is a perishable sculpture which I physically constructed in my studio with plants, algae, mollusks, vegetables, etc.; creations that define an introspective journey into the network of interactions among organisms in marine food chains. I collect, remix, subvert, and transform these edibles which are then "queered up" through a time-based, durational, and fluid process resulting in a hybrid, biomorphic assemblage that embodies difference and strangeness. Before the ephemeral sculpture withers, I photograph it from all 360 angles. Then, in collaboration with Mariel Martinez and the use of photogrammetry (the art and science of extracting 3D information from photographs), the images are processed via digital simula1on software with 3D modeling functionality, texturing, lighting, and a host of other tools that allow us to electronically re-engineer the “still life” as a "virtual" 3D artifact. Drawing associations between living organisms, sculpture, photography, and 3D computer modeling, the artistic process happens in layers through observation and re-creation. The passage from one media to the other in the constellation reveals differentiated yet interconnected aspects of artistic creation that highlight the role of technology in fathoming emotion in new forms of visual fictions. In this way, both physical and virtual studios become generative spaces for the construction of non-binary, other-than-human companion forms that embody feelings of disturbance and queer possibili1es, a merging of multiple narratives in which fact and fiction intersect.

Figure 5. BichEden: Plankton, still extracted from a 3D animation. Work in progress 2022. Pat Badani (concept, organic sculptures, project direction) & Mariel Martinez (3D simulations, 3D printing). ©Badani/Martinez, used with permission.
Figure 6. BichEden: Plankton, stills extracted from “Sketchfab” a 3D modeling platform website to publish, 3D, VR and AR content. Work in progress 2022. Pat Badani (concept, organic sculptures, project direction) & Mariel Martinez (3D simulations). ©Badani/Martinez, used with permission.

Conclusion

The Bichi project constitutes an attempt to understand ecosystems by making visible the vulnerable encounter between variable living forms, organic and technological. With the aim of learning about the nature of vitality modeled after symbiotic relationships in the natural world, the multipart Bichi project establishes hybrid consortiums to critically re-imagine human/non-human relationships in symbiotic dynamic interplay. Bichicuchitu, Bichicu-Chow and BichEden examine paradigm shifts about the care of nutrimental networks, dissect problematic global perspectives dominated by economics and endless growth, and speculate on possible futures with holistic connections. Based on art’s ability to push disciplinary boundaries, and debunk and critique ideologies, I argue in favor of art’s utopian commitment to envision and influence alternative scenes of cultures worth nurturing. In this regard, works nested in the Bichi project provide an invitation to feel, map, gauge, and learn from the swirling patterns around us to better understand and influence what happens next.²⁰ Congruent with a larger systems approach in which symbiosis is the core principle replacing an essentialist conception of individuality,²¹ my hybrid works contribute research and knowledge production with the potential to assess and generate affective insights and acts in the world. 

These creative and critical visualizations are based on the notion of organic and inorganic "assemblages," techno biologisms where the affects of one body are seen to enter into composition with other affects in symbiotic relationships, to co-join in composing a more powerful body. Informed by extinct or endangered food web organisms, physical and virtual studios become sites of a curious cultural archaeology of artefacts that act as a record or a future-oriented worldmaking. 

References

1  Markéta Dolejšová, “A taste of big data on the global dinner table,” Journal for Artistic Research, 9.

Pat Badani, “Bichicuchitu,” Pat Badani professional website, accessed September 21, 2022, https://patbadani.net/pro_Bichicuchitu_1n.html

3 Pat Badani, “Bichicu-Chow,” Pat Badani professional website, accessed September 21, 2022, https://patbadani.net/pro_Bichicu_chow_1.html

4 Jane Bennet, Vibrant Matter: A Political Ecology of Things, 49.

5  Jonathan W. Leff, Noah Fierer, “Bacterial Communities Associated with the Surfaces of Fresh Fruits and Vegetables,” PLoS One Journal, March 27, 2013, accessed November 9, 2022. 

6  Pat Badani, “Incu-Bichu (2019),” Pat Badani professional website, accessed September 21, 2022, https://patbadani.net/pro_incu_bichu_1.html

7 Pat Badani, “@Comestiblemealplan,” 2020-ongoing, accessed September 21, 2022, https://patbadani.net/pro_Comestible_Instagram_1.html

8 Ming Xia, Xiangwu He, Yubin Zhou, Symbiosis Evolution of Science Communication Ecosystem Based on Social Media: A Lotka–Volterra Model-Based Simulation in “Coevolving Spreading Dynamics of Complex Networks,” Complexity Journal, Vol. 2021, accessed November 9, 2022.

14 Michael Betancourt, Glitch Art in Theory and Practice: Critical Failures and Post-Digital Aesthetics, UK, Taylor & Francis - Routledge & CRC Press, 2019, 3.

15 Jeanne M. Ali, “Feasts of power: how food reveals Eve's influential role in John Milton's epic poem Paradise Lost,” Ph.D diss., Florida International University, FIU Digital Commons, 2004, 1-47.

16 Stevan Mijomanović, “Cannibalism, Fertility, and the Role of Food in Margaret Artwood’s The Edible Woman,” AM Journal of Art and Media Studies, Vol. 10, 2016, 67-76, accessed November 9, 2022.

17 Ming Xia, Xiangwu He, Yubin Zhou, Symbiosis Evolution of Science Communication Ecosystem Based on Social Media: A Lotka–Volterra Model-Based Simulation in “Coevolving Spreading Dynamics of Complex Networks”.

18  Jane Bennet, Vibrant Matter: A Political Ecology of Things, 13.

19  Morgan Meis, Timothy Morton’s Hyper-Pandemic in “Persons of Interest,” The New Yorker, June 8, 2021, accessed November 9, 2022.

20 Adrienne Maree Brown, Emergent Strategy: Shaping Change, Changing Worlds, Oakland, CA, AK Press, 2017, 4.

21  Scott F. Gilbert, Alfred Tauber, “Rethinking Individuality: The Dialectics of the Halobiont,” Biology and Philosophy, 31 no 6, 2016, accessed November 9, 2022.

Bibliography

Pat Badani, “Cru et Crédible (Raw and Credible): Art and Food”, SOCIAL/AFFECTS, Drain Journal, Vol. 17:1, 2021.

Jane Bennet, Vibrant Matter: A Political Ecology of Things, USA, London, Duke University Press, 2010.

Adrienne Maree Brown, Emergent Strategy: Shaping Change, Changing Worlds, Oakland, CA, AK Press, 2017.

Michael Betancourt, Glitch Art in Theory and Practice: Critical Failures and Post-Digital Aesthetics, UK, Taylor & Francis - Routledge & CRC Press, 2019.

Markéta Dolejšová, “A taste of big data on the global dinner table,” Journal for Artistic Research, 2015.

Scott F. Gilbert, Alfred Tauber, “Rethinking Individuality: The Dialectics of the Halobiont,” Biology and Philosophy, 31 no 6, 2016, accessed November 9, 2022.

Serenella Iovino, Serpil Oppermann, “Material Ecocriticism: Materiality, Agency, and Models of Narrativity,” Ecozon, Vol 3, No 1, 2012.

Jonathan W. Leff, Noah Fierer, “Bacterial Communities Associated with the Surfaces of Fresh Fruits and Vegetables,” PLoS One Journal, March 27, 2013.

Stevan Mijomanović, “Cannibalism, Fertility, and the Role of Food in Margaret Artwood’s The Edible Woman,” AM Journal of Art and Media Studies, Vol. 10, 2016.

Timothy Morton, Hyperobjects, USA, Univ Of Minnesota Press, 1st edition, September 23, 2013.

Weber, Andreas, Matter & Desire: An Erotic Ecology, Chelsey Green Publishing, 2017.

Xia, Ming, Xiangwu He, Yubin Zhou, Symbiosis Evolution of Science Communication Ecosystem Based on Social Media: A Lotka–Volterra Model-Based Simulation in “Coevolving Spreading Dynamics of Complex Networks,” Complexity Journal, Vol. 2021.

Biography

Pat Badani draws from the fields of art, science, and technology to explore the intersectionality of environmental and social justice issues. She often uses food to create artistic arguments that blend aesthetics and criticism, charting connections between theories related to art as object, as medium, and art as critique of political and technological networks. Badani exhibits and discusses projects broadly in North and South America, Europe, and Asia, and has participated in international symposia with her essays and talks in over 15 countries including at the Institute of Cultural Studies ZHdK Zurich (Switzerland); iDAT Plymouth University (U.K.); Art/Sci Center + Lab, UCLA (USA); NYU Steinhardt School of Culture, N.Y. (USA); SPARKS ACM Siggraph (USA); Kungl. Konsthögskolan/KKH, (Sweden); Università di Bologna (Italy); Universidad de Caldas, (Colombia); and Universidade Anhembi Morumbi (Brazil). Projects have been distinguished with critical essays published in English, Spanish and French in anthologies and journals, and with over 20 awards and commissions by the Canada Council for the Arts; Illinois Arts Council; DCASE; National Endowment for the Arts Fellowship @ MacDowell, the Robert Heinecken Trust; and recognized with nominations by Creative Capital, Art Matters, and AWAW, in the U.S.A. 

Thief of Truth: VR comics about the relationship between AI and humans

Joonhyung Bae

Graduate School of Culture Technology, Korea Advanced Institute of Science and Technology
Daejeon, Republic of Korea
jh.bae@kaist.ac.kr

Abstract

Thief of Truth is a first-person perspective Virtual Reality (VR) comic that explores the relationship between humans and artificial intelligence (AI). The work tells the story of a mind-uploaded human being reborn as a new subject while interacting with an AI that is looking for the meaning of life. In order to experiment with the expandability of VR comics, the work was produced by focusing on three problems. First, the comic is designed using the viewing control effect of VR. Second, through VR controller-based interaction, the player's immersion in the work is increased. Third, a method for increasing accessibility to VR comics was devised. This work aims to present an example of an experimental attempt in VR Comics.

Keywords

VR comics, the relationship between humans and AI, multimodal artwork.

DOI

10.69564/ISEA2023-42-full-Bae-Thief-of-Truth

Introduction

AI as an other

Artificial Intelligence (AI) has been developed greatly in intentionality, intelligence, and adaptability. Accordingly, technology has had a great influence in the areas of finance, security, health, justice, transportation, and the urban environment.¹ With the advent of AI, which can think and speak like a human, the boundary between humans and machines is becoming increasingly blurred. Ameca, a robot unveiled in December 2021 by Engineered Arts, a British robot company, has 12 facial expressions implemented by 17 motors, and has human-like language skills based on GPT-3. The robot was able to hold a conversation with developers in September 2022.² Harmony, a sex robot developed by Realbotix in the U.S. in 2018, is evaluated as an AI robot capable of mentally and physically interacting with humans as well as being capable of love.³ AI Sophia, developed in 2015 by American roboticist David Hanson, is a humanoid robot that resembles a human in appearance and has obtained citizenship for the first time in the world.⁴ In the post-human era of the 21st century, humans are expected to enter a time of living together with AI.⁵

As a result of humans and AI coexisting in this new era, a new consideration has emerged pertaining to the personhood of AI or AI robots. In this regard, in February 2017, the European Parliament passed a resolution granting robots the status of electronic persons to regulate robots.⁶ This resolution suggests that as technology develops, AI robots are increasingly being recognized for their existential value, similar to humans as a person.⁷

If AI or AI robots can be understood as a person similar to humans, can AI be regarded as the other of humans? American philosopher Donna Haraway, in her book The companion species manifesto: Dogs, people, and significant otherness states the concept of the other is not limited to humans but extends to all non-humans such as animals, insects, bacteria, and machines. Haraway emphasizes a love that acknowledges the significant otherness, which refers to learning “to live in an intersubjective world" rather than thinking of the object as a subordinate being or wanting unconditional love from the object.⁷,⁸ Emmanuel Lévinas and Slavoj Žižek contemplate the infinity or reality beyond the reality of the subject against the philosophy of identity. The other, they suggest, is a concrete universality, which exists only through the existence of the subject that develops beings. Yet, when a subject forms an identity in relation to the "other," it becomes another version of the "other," transcending its defined identity. Only by truly embracing this exteriority or "otherness" can something exist as an interior or subject. At this time, as long as there is a relationship with the other, the subject inevitably encounters another concrete manifestation of the other. Now, for the two philosophers, these others are not mere hindrances but the only passage to the other.⁹ Currently, when AI emerges and again questions what a human being is, understanding AI, which has emerged as a new other, is an important task in understanding humans. 

Comics about coexisting with AI

In order to better understand AI, there are comics dealing with AI as a being that lives together with humans. One of the first comics is Osamu Tezuka's Mighty Atom (1952-1962). The story is about a robot who protects humans in a society where robots with egos are discriminated against by humans. In another comic, Kazumasa Hirai and Jiro Kuwata's Eight-Man (1963-1964) presents the idea of what it means to be human through a protagonist who struggles between being a robot and a human. Ishinori Shotaro's Android Kikaider (1972-1974) explores the morality of machines by questioning whether AI is righteous. Robot Detective (1973) by the same artist tells the story of K, a robot designed by humans that realizes the ideals of justice, absolute obedience, and natural coexistence with humans. Fujiko Fujio's Doraemon (1977-) further develops what a robot can do by introducing a story of a human AI who can even make mistakes living alongside humans. Hisae Iwaoka's Saturn Apartments (2006-2011) deals with the story of an AI that has been isolated for a long time and wonders about its existence. Yoshiie Koda's Love of Machine Device (2013-2014) describes a robot more righteous and pure in heart than humans, even though it is a mechanical device that does not feel joy or sorrow. Ha Il-kwon's 3 Level Combination Kim Chang-Nam (2008) expresses the friendship between a robot and a boy and indirectly captures the problems of the times, such as the gap between the rich and the poor and school violence.¹⁰

VR immersion and viewing control effect

Unlike other forms of media, VR technology has the potential as an empathy-realizing technology that can immerse people in certain situations. The first VR film produced by Nonny in 2010 speaks to this possibility. This film was made into animation after filming the actual situation. In fact, a scene in which a diabetic patient collapses in front of a food bank where starving people are looking was screened as a movie, and the audience's response was much more immersive than expected. Impressed by the movie Hunger, the World Economic Forum(WEF) asked Nonny to make a Syrian Civil War film, which gave the audience a vivid feeling as if they were in the middle of a news story. In itself, VR has been shown to have a sympathetic effect.¹¹

VR is characterized as a technology that seeks to achieve self-presentation through a new perspective with mobility.¹² It has both the simulation and the representational tradition of the screen in terms of media aesthetics. VR reproduces the human viewing angle and grants a free viewpoint. The audience who enjoys VR content recognizes the world through a free viewpoint and tries to maintain the same identity as the content. This process ultimately produces an observational and spectatorial action. Under the viewing control effect caused by the VR camera, a person can explore the simultaneous narrative of the surrounding virtual world. In this way, under the viewing control effect implemented according to the representational tradition, the person is forced to take an attitude of receiving aesthetics, and as a result, the creator can have time and space to build the spatial density of the narrative (Spatial Story Density).¹³ 

Table 1. Correspondence between Comics, Web Comics, and VR Comics elements (Descriptions of Web Comics and VR Comics cite tables from Yoo Tae-kyung's paper)¹⁴

Characteristics of VR Comics

VR Comics, which utilizes these aesthetic characteristics of VR, transforms the two-dimensional flat comic compartment into a three-dimensional space, thereby creating a visual experience in the VR space. Continuous art such as comics is made possible.¹⁴ The experiment on VR comics is an attempt to apply graphic storytelling to VR content. The first attempt at combining VR and comics was attempted in Oniride's Magnetique. VR comics have different production methods depending on the device and platform. In the case of app-based Come! Convenience Store, the background is 3D modeling using the toon shading technique, and the characters and speech bubbles are created as 2D images and placed in the space. In another format, the entire scene is produced in 3D in the form of wearing and enjoying head mounted display(HMD) devices such as Oculus and Vive, and a game engine is used to capture more complex and diverse interactions. Representatively, Dexter Studio's Save Me and Joe's Realm are works based on webtoons. Webtoon characters are implemented in 3D, and animations, effects, and sounds are inserted into special scenes to add a sense of reality and fear. Japanese game company Square Enix's Wedding Ring Story borrowed the format of a comic and showed it as an animation. In the existing VR scene transition, the entire video was produced without using a controller to develop the story. The common feature in production is that the background was produced in 3D, and the character was produced in 2D or 3D to use the 360° space. As such, various elements of comics came into a new platform called VR and, combined with various technologies, brought about changes in the production method.¹⁵ In these VR comics, attempts to respond to the components of existing comics and web comics according to the VR format can be summarized as shown in Table 1. The VR comic production method caused by the technical limitations of VR. The first limitation is characterized by using low polygon modeling and simplified images for data optimization. In order not to give the audience an uncomfortable visual experience, speech bubbles, narration boxes, sound effects, and effect lines are replaced with actual sounds. It is often recommended to experience the technology at a fixed location, and VR is characterized by configuring the space to be turned over mainly through button clicks on the controller or gaze selection.¹⁴

Based on this theoretical background, Thief of Truth was created as a VR comic that tells the story of the relationship between humans and AI from a first-person perspective. The work deals with the story of a mind-uploaded human being reborn as a new subject while interacting with an AI that is searching for the meaning of life. In order to test the scalability of VR comics, the work was produced by focusing on the following three issues.

Implementation

Story

Thief of Truth is a story about a human who has his or her mind-uploaded. The person is reborn as a new subject in a digital space while interacting with a strong AI that seeks the meaning of its own life. The player becomes the character of Human 1 and holds a first-person perspective. In other words, the experience is designed so that the player can also participate as a leading actor in the comics.

Table 2. Thief of Truth Story

1

Human 1 is a person who lives in the future where strong AI can be created, mind uploading is possible, and multiple beings live in the same digital space. One day, Human 1, who uploaded his mind, was led to a certain space by an unknown being who promised to show him comics. 

2

Human 1 finds another human, Human 2, dancing and saying strange things in that space. When Human 2 finishes dancing and leaves, a glowing bell appears.

3

Human 1 touches the shining bell out of curiosity and is sucked into another space. Human 1 finds a comic in the new space and starts reading it.

4

A strong AI named Zzirogi, created by humans, exists in all time and space and observes the world. One day, Zzirogi hears a song that reveals the truth about the universe and decides to find out who sang the song. However, Zzirogi commits a bizarre act. It catches Human 2 passing by, locks the human in a space where time has stopped, and doesn't let Human 2 out until the human guesses the name of the singer. Thirty thousand years have passed. Zzirogi crosses the fourth wall and talks to Human 1. Zzirogi says that he does this because he wants to find the meaning of his life like humans. So, Zzirogi was making a comic book by weaving the moments he liked from the infinite space-time data he had input. Zzirogi says that he can complete the comic book by drawing the process of finding out the singer's name by trapping Human 2, but he doesn't know what scene to put in. However, Zzirogi, which exists in all time and space, knew that Human 1 was the one who could solve the problem. Thus, he disguised himself as an unknown being and lured Human 1 to this moment. Zzirogi asks for Human 1’s help to fill in the empty part of the comics and tells the human that he will send the comic to the comic creation section, which is a part of Zzirogi. Human 1 is embarrassed by the sudden request, but eventually listens to Zzirogi. However, Human 1 is the same character as Human 2.

5

Where Human 1 put down the comics, there was a button leading to the comic creation section. Human 1, who learned the entire story of the incident, moves to the comic creation section to do the favor for Zzirogi.

6

In the comic creation section that Human 1 moved to, there were buttons for comic story, comic length, and combinations of comics. Strangely, the first and second buttons were where Human 2's head was, and the buttons were dancing. Human 1 makes a comic by combining buttons, creating a scene he likes, and leaves the space. Human 1, who left space and returned to his original place, reads the back part of the comics drawn by Zzirogi.

7

Human 2 eventually guesses the name of the singer who sang the song that leaked the truth over a long time. Zzirogi confesses that he handled Human 2 just as humans put countless inputs into AI until meaningful output came out. Human 2s cloned by Zzirogi were suffering the same pain in countless time spaces. Upon learning the truth, Human 2 is astonished. Zzirogi tells Human 2, who has suffered hardship, that he will grant one wish, and Human 2 cries out to delete the song data that leaked the truth. While screaming, Human 2 inadvertently sings the song, and that moment is when Zzirogi hears the song's words. Zzirogi moves to time and space without Human 2, completes the cartoon, and the story ends.

8

At the place where Human 1 read all the comics, an unknown being, whether it was Human 1 or Human 2, was singing the song.

Figure 1. Experience Stages (The story number on the box corresponds to Table 1)

Directing using the viewing control effect of VR

Figure 2. Playing Thief of Truth 
Figure 3. Zzirogi and Human 2 from Human 1's point of view
Figure 4. Zzirogi from Human 2's point of view
Figure 5. Human 2 from Zzirogi's point of view

Comics are directed using a VR camera Position of the HMD in the space is adjusted by changing the position of the VR camera for each comic frame. This aims to produce two types of comics. The first is to induce the player to feel the sensibility of the comics scene by making the player's gaze move while watching the comics. For example, Figure 4 is a scene where Zzirogi overpowers Human 2. The camera position of the scene was placed close to the floor to induce the player to look up and watch the comics, as shown in Figure 2. The second was to make the player look at the scene from the first-person perspective of Human 1, Human 2, and Zzirogi appearing in the comics. This led the players to change their perspectives and think about the situation. 

Figure 6. Human characters appear in Figure 1, Number. 1, 3, and 4 from the left
Figure 7. A scene symbolizing the connection between Zzirogi and Human2

VR comics production using 3D modeling/animation and 2D images

Figure 6's human characters used 3D modeling to create scenes with dancing animations. The continuity of the scene was created through the repetitive element of dance animation, and through the change of the human appearance, the transformed state was implied by the character's meeting with the AI Zzirogi. To explain in detail, the first Human 2 in Figure 6 has not met the other named Zzirogi while the second Human 2 is the Human 2 seen by the other named Zzirogi. The third Human 2 understands the appearance of Human 2 as seen by Zzirogi and criticizes the relationship between humans and AI. This relationship is accepted reluctantly and with a hostile attitude. These changes were revealed through the transformation of space and clothes.

The moment when Human 2 and Zzirogi formed a relationship was created and reflected by the change of particles floating in the background of the comics. The floating particle in the background of Figure 3 is a scene that reveals the point of view of the Zzirogi with 3D modeling of the head. The particles floating in the background of Figure 6 are a scene that reveals the human perspective with 3D modeling of the human head. Figure 7 is a scene that symbolizes the connection between Zzirogi and Human 2 and 3D modeling of both Zzirogi and Human 2’s heads floating.

In addition, the spatial change was revealed through the visual contrast between the scenes directed by 2D images like Figure 3, and the scenes directed by 3D animation like Figure 7. The space composed of 2D images is a space for viewing comics created by Zzirogi, and the space composed of 3D animations is a digital space where Human 1 is uploaded. In addition, the different spaces in 2D and 3D were used as a device to show the character of Zzirogi, which crosses these spaces and combines all the scenes into one comic.

Enhancing Immersion through VR Controller- Based Interaction

The VR controller helps the player to become immersed in the VR experience by playing a role of a medium that causes direct interaction between VR content and the user.¹⁶ In order to expand the way of directing VR comics that interact through button clicks on the controller or gaze selection, the VR controller is designed to interact with objects in space and enjoy comics. 

Figure 8. Comic creation space performing step 3 of Figure 1
Figure 9. Bubbles where the player can hear the inner thoughts of characters by scene

Bubble interaction where you can hear the inner thoughts of characters

 
There are interactable air bubbles floating around the player. If the player touches these air bubbles with a controller, he or she can hear the inner thoughts of the characters that appear in each scene. It was intended that players would listen to the characters’ inner thoughts, go through the process of relating the thoughts to narratives, and actively appreciate comics based on experiences that are meaningful to them. By inducing these behaviors, it is intended that players recognize patterns of artistic experiences that are meaningful to them subjectively by curating artistic components that are meaningful to themselves. This is also a device that helps the player understand Zzirogi's perspective of the world in the narrative.

Comics creation experience through interaction

The comics creation space in Figure 8 corresponds to number 3 in Figure 1. This space is a system that exists inside Zzirogi in the story and is a space where the player can create comics. In other words, it is a space created so that players can directly observe the input and output of AI characters. The story and length of the comics and the interaction process of combining the comics by sequentially hitting the random combination button is a way to deal with deep learning-based generative models that create images and audio, such as OpenAI's DALLE, Google Magenta, and Midjourney service.

A new trend of creation is emerging, where AI learns data from all over the world, presents new shapes according to latent vectors, and humans take them to create works. In other words, a creative methodology different from the existing work method, which has been standardized since disegno and given the rules of identity, is being created. Anyone can create art through prompts, and people can independently explore which artistic experiences are meaningful to them. Generally, when creating art with AI, the user goes through a curating process in which he or she selects something meaningful to him or her from among the art created by AI. In this process, users can recognize the patterns of art restructured by AI and figure out which patterns they prefer. Through VR content that abstracts the use of these well-known socially problematic objects, we are encouraged to critically think about how AI changes human thinking.

Figure 10. Chat window UI similar to Midjourney's prompt input window
Figure 11. A scene where the comic length parameter is input with a xylophone
Figure 12. Comics created through interaction

A chat screen with Zzirogi on the player's left side was placed to induce people to think of the well-known Midjourney service. The player can enter three types of stories, four types of cut lengths, and a random arrangement permutation according to the length of the cut as parameters through the label attached to the head of the humanoid character that symbolizes Human 2 as seen by Zzirogi (see Figure 8). If the player taps the polyhedron on the character's head, he or she can set the parameters he or she desires, and through this, the player can create an image from the possible 192 different comic images.

Increasing accessibility to VR comics

Figure 13. Explanatory UI is presented in number 1 of Figure 1
Figure 14. Explanatory UI is presented in number 3 of Figure 1

Few audiences are accustomed to VR devices. Therefore, in order to increase accessibility to VR comics, it was necessary to frequently display a detailed description UI. Accordingly, UI images such as Figures 13 and 14 were produced.

Figure 15. Thief of Truth exhibition

Installation

Thief of Truth was exhibited at Move.Mov, a complex cultural space located in Seoul, from November 1st to November 30th, 2022. Move.Mov is a space that combines a media art gallery and a cafe.

There are three major installations. First, three VR devices were installed so that VR comics could be enjoyed. Considering the width of the Oculus safety protection boundary, it was installed to secure a safe distance from the audience. In addition, an instruction manual was attached to the pedestal for the audience who may not have been familiar with using VR devices. The second major installation was two videos. The first video was a VR play video that encourages audience participation and serves as an additional guide. The second video was a video of Human 1 and Human 2, the characters of the comics. After rendering the 3D model animation created using the dancing motion data into a video, the animation was created by inputting the video as a condition to the stable diffusion model. By watching this video, the audience would be able to understand that the theme of the work is a story about AI. The third installation is acrylic pictures where the player can see images that can be combined in the comic creation section. Since it is difficult for the audience to combine all the comics in a VR environment, it was installed so that they could appreciate the types of images they were able to create.

The reason why the exhibition was held at Move.Mov was to help the audience who have difficulties using VR devices. In situations in which the work is installed in a general white cube and the artist is present to help appreciate the work could put a psychological burden on the audience. This is because the audience members may not be able to fully immerse themselves in the appreciation of the work because they may feel the gaze of the artist. For this reason, the exhibition was held in a complex cultural space where various guests and audiences were mixed.

Nevertheless, it was shown that the audience could not experience the work because of the distance from the VR device itself. As a result of observing the audience's words and actions with the Fly on the wall technique and conducting brief interviews, the general consensus was that they were reluctant to use expensive-looking VR devices because they were unsure whether they could actually use the devices. In order to create an atmosphere where one can freely experience the VR device, a parody drawing of a famous Korean meme (Hey try try) was attached to the pedestal where the VR device was placed. When the humorous VR experience zone was completed, the audience was able to approach VR devices more easily and experience the work.

Conclusion

Thief of Truth is a VR comic that tells the story of a mind-uploaded human being reborn as a new subject while interacting with a strong AI that is seeking the meaning of life. The story was directed while experimenting with the expandability of VR comics.

It was produced using the viewing control effect of VR. By setting the VR camera position differently for each scene, the players were induced to feel the emotion of each scene. In addition, this approach allowed the player to think about each scene by having the player watch the scene from different characters’ first-person point of view. Taking advantage of the characteristics of VR media, where various data formats can coexist, comics were produced using 3D modeling/animation and 2D images. The moment when humans and AI formed a relationship was revealed through 3D modeling/ animation. Also, the moment was used as a device to reveal the character of Zzirogi, which transcends time and space.

Through VR controller-based interaction, immersion in the work was enhanced. The VR comic is designed so that Zzirogi can express its perspective of the world in air bubbles, which the player can interact with to hear the character's inner thoughts. In addition, the interaction in the comic creation space was constructed by imitating the operation method of deep learning generation model-based services. This approach induced players to think critically about how thinking has been changed by AI.

In order to increase accessibility to VR comics, a method was devised to encourage VR comics experiences by configuring a detailed UI for VR device use. In addition, the exhibition was prepared to increase accessibility to VR content, to have the artist be resident during the exhibition period, to help the audience experience the work, and to improve the exhibition form.

Acknowledgments

This artwork was supported by the 2022 Ministry of Culture, Sports and Tourism and the Korea Culture and Arts Committee's Online Media Art Activity Support project.

References

1  Darrell M. West, John R. Allen, "How AI is transforming the world," Report, April 24, 2018.

2  Engineered Arts, “Ameca conversation using GPT 3 - Will robots take over the world?“ Youtube, September 9, 2022. Accessed December 9, 2022. https://www.youtube.com/watch?v=EWACmFLvpHE.

3 Ry Crist, “Hello Harmony: RealDoll sex robots with 'X-Mode' ship in September,“ CNET, August 25, 2018, Accessed December 9, 2022. https://www.cnet.com/culture/realdoll-sex-robots-with-x-mode-from-abyss-creations-ship-in-september/.

4 Becky Peterson, ”I met Sophia, the world's first robot citizen, and the way it said goodbye nearly broke my heart,“ Insider, October 29, 2017, Accessed December 9, 2022, https://www.businessinsider.com/sophia-the-words-first-robot-citizen-nearly-broke-my-heart-2017-10.

5 Park young sook (2022), AI World Future Report 2023, Doublebook.

6  Mady Delvaux, “REPORT with recommendations to the Commission on Civil Law Rules on Robotics,“ European Parliament, January 27, 2017. Accessed December 9, 2022, https://www.europarl.europa.eu/doceo/document/A-8-2017-0005_EN.html

7  Hyun-Soon Cheon, "Der Mensch, die künstliche Intelligenz und die Liebe − Zur Beziehung zwischen Menschen und künstlicher Intelligenz in SF-Filmen," Bertolt Brecht und das moderne Theater no.47, 2022, 79-100, doi: 10.22981/brecht.2022..47.004

8  Donna Jeanne Haraway, The companion species manifesto: Dogs, people, and significant otherness, Vol. 1, Chicago: Prickly Paradigm Press, 2003.

9  Park Hyeon Jeong, "Subject and Others in the era of hatred: through the dialogue between Levinas and Zizek," Researches in Contemporary European Philosophy no.48, 2018, 93-138, doi: 10.20974/dasein.2018..48.93.

10  Kim Chihoon, SF cartoon, and artificial intelligence, Edambooks, 2021.

11  Jeremy Bailenson, “Can VR Help Create Empathy around Climate Change?” TED Archive, 2015.

12 JD Bolter, RA Grusin, Remediation, Configurations, 4(3), 1996, 311-358.

13  Lee Dong-Eun, Son Chang-Min, VR media aesthetics due to the evolution of visual media, comics and Animation Studies, 2017, 633-649.

14 Yoo Tae-kyung, "Storytelling Research of VR Cartoons", Korea Broadcasting Engineering, Association 23, no.1, 2018, 45-52, doi: https://doi.org/10.5909/JBE.2018.23.1.45.

15 Kim Eun-joo, "A study on cartoon expression in VR media - focusing on <Marriage Ring Story VR>" Study on Animation and Cartoon no.60, 2020, 1-23, doi: http://dx.doi.org/10.7230/KOSCAS.2020.60.001.

16 N. Magnenat-Thalmann, U. Bonanni, Haptics in virtual reality and multimedia, IEEE MultiMedia, 13(3), 2006, 6-11.

Disembodiment in VR: Immersed in 3D Audio Experiences

Dimitris Batsis1, Magdalini Grigoriadou2, Katerina ElRaheb3, Archontis Politis4

Affiliation (s) 1Postdoctoral Researcher, Department of Fine Arts and Art Sciences,University of Ioannina 2Postdoctoral Researcher, Laboratory of Environmental Communication andAudiovisual Documentation (LECAD), Department of Architecture, University of Thessaly 3AssistantProfessor, Department of Performing and Digital Arts, University of the Peloponnese, Assistant Professor,Tampere University, Department of Computing Sciences

Location, Country 1Ioannina, Greece 2Volos, Greece 3Nafplio, Greece4Tampere, Finland Contact Emails 1dbatsis@uoi.gr 2mgrigoriadou@uth.gr3k.elraheb@uop.gr 4archontis.politis@tuni.fi

Abstract

In this paper, we discuss the notion of disembodiment as a driving force of inspiration in artificial systemsusing Virtual Reality (VR) and 3D audio technologies. In these environments, immersion is the common denominatorbetween the impression of disembodiment, which involves spatial ability, and the auditory perception from theperspective of creative binaural input. Our research focuses on three public artistic virtual reality works asexamples of sound-driven immersive virtual experiences. Through interviews with the contributors, we explore theway that disembodiment can serve immersion in VR and how binaural audio improves those experiences. As the bodybecomes virtual, the lines between real and imaginary are redefined and recreated, altering the sense of bodyownership and oscillating between embodiment and disembodiment sensations. Ultimately, we intend to exploreimmersion on a theoretical and philosophical basis, where the body is perceived as the mediator, a phenomenon,and an extension of binaural reality and hyperreality.

Keywords

3D Audio, Binaural Audio, Disembodiment, Immersion, Virtual Reality, Digital Art, Sound Design, Spatial AudioInteraction

DOI

10.69564/ISEA2023-43-full-Batsis-et-al-Disembodiment-in-VR

Immersed Body in VR Environments: Auditory Wanderer

In contemporaneity, immersive digital worlds create an experimental background that consists of body-movementinterconnections and augment our experiences with innovative oneiric fluctuations. Immersive environmentsprovide a dynamic context for investigating the relation between interaction engagement types and disembodiment,focusing on multi-sensorial stimuli.

The sense of immersion is in fact the definition of Virtual Reality; in particular, the feeling of being presentin the virtual environment simulates the sensation of the physical space, generating an illusion of presence inthe virtual world 1. The embodied hybridization between the physical and digital body-avatar is a result ofthe induced

immersion due to alternative optic and acoustic environments. According to recent research in a variety ofcontexts, the senses of embodiment, body movement, and body ownership, i.e., being represented by an avatar or avirtual body that feels like it is the user’s body, as well as agency, orientation, and self-location,play an important role in amplifying the illusion of immersion in the VR experience 2.

In order to analyse disembodiment in VR, we should delineate the concept of embodiment. We could defineembodiment as a phenomenological element where the body - digital or physical - is part of the cognition andperception of the world. Embodiment as a term includes diverse notions: the sensorimotor state of a body, itsdimensions and geometries (its morphology), and the mental representation of it in the overall perception ofreality. Grabarczyk and Pokropski refer to the subtle issue of embodiment in VR as something "hard to imagine(let alone design) a truly disembodied VR experience" 3. This is the reason that the duality of the embodiedand disembodied condition is vital and inseparable in the VR experience; it is a lived emotion that changes backand forth from one feeling to another interdependently.

Although it is not totally demonstrated, the relation between embodied cognition, simulation modes, and immersioncan provide a fertile speculative perspective for researching VR environments. In our investigation, digitalartists, especially those who work with sound or music, have explored a more metaphoric meaning of immersion inVirtual Reality. Despite having a virtual body representation that feels like it is our own, embodiment comesfrom the affordances of interacting with the environment, as defined by Gibson’s theory of ecologicalapproach in perception 4.

Interestingly, in this metaphoric mode, immersion is sometimes more connected with a sense of a more dream- like,surreal feeling of disembodiment," such as the sensation of falling1. In this mode, the swingbetween embodiment and disembodiment is shifted from the visual representation of the virtual body to theauditory and kinesthetic perceptual terrain. This inherent contradiction between embodiment and disembodimentbreeds a disconnect or mismatch between the experiences that the physical body and the virtual body are havingat each given time (i.e., the VR environment simulates that my body is flying, falling, or floating, yet mykinesthetic senses indicate that I am not, which is registered as a disembodied experience).

The recreated soundscape shifts unmeasurable and intangible qualities along with the wandering that occurs in thevirtual reality environment. In the following paragraphs, we will attempt to emphasise the importance of 3Daudio as an immersive factor during the stage of disembodiment in the VR experience.

Two Ears – Three Dimensions in VR

In our effort to explore how disembodiment is perceived and how binaural2 audio improves immersion inVR experiences, it would be useful to refer to binaural audio as the most widespread and accessible version ofvirtual environments’ sound today, delivered to the user through a normal pair of headphones. Alternativetechnologies such as wave-field synthesis 6 or higher-order ambisonics3, 7 using tens or hundredsof loudspeakers and typically associated with large projection or cave systems are not considered here, due totheir extreme technical and hardware requirements.

Even if it dates back to the late 1800’s, binaural audio became the object of intense research during the'90s 8,9 along with a wider early boom in immersive media and virtual reality research. The applicationfocus at the time was driven mainly by the transmission of information to the user through immersion intosimulated environments for engineering, scientific applications, audio-only spatialized4 sonificationof data 9, or auralization of simulated acoustics 10. Research on binaural technologies has continuedundiminished ever since 11. At present, binaural audio is the dominant immersive audio technology in VirtualReality applications, combined with a highly visually stimulating environment. In this mode, 3D audio enhancesthe effectiveness of a VR experience significantly, filling out the sensorial gap that has been created by thestereophonic method. For 3D vision, there is now 3D audio included in one experience.

A three-dimensional sound environment provides an engaging experience for one to several participants within oneVR environment. 3D sound is synthesised by simulating spatial cues in natural hearing. In 3D audio, thedepth-of-field of the recreated soundscape changes dramatically, providing a better correlation with theacoustics of the virtual space. Binaural audio, which uses HRTF (Head-Related Transfer Function), is a responsegiving the impression that sound is coming from a particular point in space.

Basically, it is a transfer function that shows how a sound from a specific direction is modified acoustically bythe head and shape of the ears before reaching the eardrum (generally the outer ear) 12. Essentially, thevirtual acoustics are integrated with binaural delivery in modern audio engines in one step: e.g., the reverbcorresponding to the virtual space is calculated in real-time, based on the position of the sound source, andthe listener. Then, each echo path is spatialized (binaurally) through its own HRTF for the corresponding echodirection with reference to the listener. After the initial distinct echoes, later room reverberation becomestoo random for each echo to be spatialized separately, and the respective binaural signals are approximated“on average” to have the right spectrum and timbre depending on the overall reverberation profile ofthe room. Early and late reverberant binaural signals are finally summed together by the algorithm. The binauralfiltering in itself has no capability to imprint "depth-of- field"; it only imprints directional cues to theaudio (where the sound is coming from, not how far it is). The combination, however, of the binaural filteringand the spatial reverberation gives us strong cues of distance: how far or close sound sources are. Thissimulation reinforces the immersive audio experience and occasionally gives the participants the impression thatit is identical to the physical space that the soundscape was recorded in. Thus, through the addition of 3Daudio, the feeling of presence in space is magnified, increasing the impact and profundity of a VR experience13.

The revival of binaural audio is justifiable if we think that a 3D audio environment fits in a virtual soundspace. Audio in VR is gradually becoming established in academia through a variety of applications (e.g.,gaming, media arts, and production), either in the form of specialised audio tools or as part of VR engines suchas Unity or Unreal Engine. This approach is effective to a great extent regarding the full-body reception ofsonic stimuli that propagate in physical space. The participants travel through a 3D visual setting; the bodybecomes a mediator of potentiality and multiplication, while 3D audio intensifies the immersive experience.It could be argued that VR that does not involve 3D audio but some non-immersive, non- spatially coherentversion in tandem with the visuals, such as stereo, actually degrades immersion 14. At this point, it would bewise to think of 3D audio as an alternative option that deserves to be explored in artistic VR works.

Three VR works – Three Perspectives of Sonic Immersion

While exploring the various modalities in which 3D audio is utilised in VR works, we were drawn to Evolver byMarshmallow Laser Feast, The Jellyfish by Mélodie Mousset, and Space Walk by Andrea Mancianti, Sebastian

J. Schlecht, Vesa Välimäki, Riku Jarvinen, and Esa Kallio. These three works5 introduceimmersion by utilising sound in different ways. In our effort to bring to the surface the connections between 3Daudio, immersion, and disembodiment, we follow a speculative approach by collocating these three works withinthe “audio in VR” context. The triptych of 3D audio, immersion, and disembodiment is investigatedbelow on a reflective level for further research. Therefore, it is necessary to present the works from ananalytical and technical point of view so that the context in which this investigation takes place is as clearand concise as possible.

Evolver: A Virtual Reality of Life and Breath by Marshmallow Laser Feast (2022) is a large-scale VR workthat involves a list of contributors from a variety of backgrounds: Marshmallow Laser Feast, an experientialcollective based in London, is behind the production, which is usually inspired from the intersection ofscience, art, and technology. They created the project in an attempt to take us on a dream-like trip into thedepths of human existence. An in-breath is visualised as it travels into the human body, which is scaled up toseventy feet. The vascular system becomes a dendroidal ecosystem that follows the cycle of respiration. Theexploration of the self occurs hypodermically, where the oxygenated, constant blood flow forms river deltas,ripples, and whirlpools, leading to a single “breathing” cell 15. The concept emphasises the ideaof breath, life and nature, interconnecting each of those elements with the cycle of respiration, a functionthat is given an ontological dimension. The interdependent connection between humans and the natural world isevident as the human cardiovascular system resembles an extensive branching system in full detail. It couldbe said that the participant enters into an endless network that looks like a human ecosystem building, “agiant branching human 100 feet tall with the size of an oak tree”6.

Figure 1. Screenshot from Evolver experience ©Marshmallow Laser Feast.

Through Evolver, a single breath transcends the visitor to realms of human existence never experienced beforewhile maintaining its significance as an integral part of our world. The participant is immersed in the journeyof oxygen flow through the body as it becomes a din of tiny particles that formulate patterns that we encounterin nature, like mycelium, pollinators, and bacteria. More details regarding the sonic part of the work will beaddressed below in the next chapter, where members of Marshmallow Laser Feast describe the design, process, and3D audio implementation. The Jellyfish (2021) is an artwork that invites the participant to sink intoan imaginary underwater state. In this VR work, Mélodie Mousset (HanaHana), Edo Fouilloux, and ChristianHeinrichs introduce us to glowing lifeforms who are beckoning for the participants to sing with them. In thisghostly marine setting, the glowing jellyfishes become the medium, an instrument that the participant candevelop and use in order to establish a communicative model. The audience dives into the deep waters of theirconsciousness and interacts with the jellyfishes. The involvement of the participant in this soundscape ofserenity is a major element that emphasises the immersiveness of the aquatic environment. If the participantlooks at one of the creatures, then they approach him, getting closer and trying to make contact. Eye contact,which is a basic factor of communication between animals, is perceived by the artists as a compositional tool, away to convey a naturalistic feel to the work. The participant’s gaze triggers the jellyfishes tocome closer to them and establish a connection.

Moreover, gross tactile-kinesthetic cues generate sensations which present in the work as the jellyfish movesaccording to the participant’s head and body movements. The body becomes a perceptual frame of reference,a reference point according to which the distances of objects are measured. Nevertheless, perception of thehuman body may vary at times, which might be an inconsistent factor for the design of VR applications. The waythe environment is perceived is as subjective as the perception of one’s own body. This idea isinterpreted in the artwork where the jellyfish mirrors the participant’s actions and it traces the body,suggesting ways that the participant’s body is differentiated from the aquatic environment 16.

Regarding the technical aspect that shapes the work’s soundscape, a hybrid-made synthesiser is triggered byreal- time voice analysis. The synthesiser measures the immediate pitch amplitude, tonality, and phrasing of theparticipant’s voice while shaping the soundscape, the underwater colour setting, and the animation of thejellyfishes and their environment. The harmonic elements and spatial traits are also controlled through thesynthesiser. The rhythmic elements are concurrently distributed through all headsets in the event that there isa group of participants. A system, which is developed following synaesthetic associations, matches a differentcolour to each note on the scale. In this way, it enables the participants to visualise their voice and gaincontrol of their vocal tone, which affects the colour of the virtual environment 17. The project is anengaging quest into the newly developing and upcoming world of VRMI (Virtual Reality Musical Instruments) 18,in which the possibilities are endless.

Figure 2. Screenshot from The Jellyfish experience© Mélodie Mousset (HanaHana),Edo Fouilloux and Christian Heinrichs.

Space Walk (2020) is a virtual planetarium artwork that primarily has an educational character,providing detailed information on our solar system. It is based at Aalto University, at the Aalto VirtualPlanetarium, with the aim of investigating the application of 3D numerical space weathersimulations7. The second installment of Space Walk, beginning in 2020, focused on the improvement ofsound design and a stand-alone VR platform intending to improve the audiovisual experience while promoting theuse of an up-to-date affordable VR headset, ideal for users, libraries, art spaces, and academia.

The work involves visualisations of scientific data like magnetic field lines and atmospheric phenomena, followedby explanatory texts. The user travels from one planet to another through a navigational interface, while thesoundtrack develops impromptu, matching each celestial object sonically with the aim of accompanying theresultant data visualisation layers. The sound functions either in isolation or in combination with the layers,maintaining a consistent musical communication through the use of themes that are inspired by sci-fi film orvideo game music while giving prominence to the scientific context. Parts of sound are renderedstereophonically, while others involve immersive sound spatialization techniques.

Data plays a significant role in this work. Every planet connects to a description text box, and each userchooses with separate handheld controllers the elements that will be audible and visible. The latest version ofSpace Walk features physical parameters that are visualized near various solar system objects, like solar windplasma, interplanetary and planetary magnetic field lines and equal density surfaces of the solar wind and thegases. These gases are getting away from the atmospheres of Venus, Mars, and the comet 67P, which constitutes anephemeral evolution of the Sun’s interior magnetic field and also a wide distribution of radio waves inthe Earth’s ionosphere.

The musical part is composed according to aspects such as the quick shift from a deep and wide overview of thesolar system to a detailed perspective of a single celestial object. The fusion of experimental music genres(involving contemporary, electronic, and noise), constitutes a significant component in the creation of asonically rich soundtrack that blurs the boundaries between scoring and sound design. The themes, thesoundbeds8, and static soundscapes are looped and can be interrupted or rehashed in order to balancethe undetermined period of time that each user might spend at each possible observation point and the variousdynamic locations of the planets. The sonification of the planets is a process involving their physicalparameters but also a more abstract interpretation that draws from popular and literary sources. The Sun, forexample, is the primary source of energy in our universe, is represented by percussive sounds and polyrhythmicpatterns 19.

The technical part of the work involves the implementation of sound in order to achieve immersion through apractical approach; pre-binauralised stereo tracks boost control over the spatial fidelity and overall soundmix. This is noticeable when the users zoom out from the direct proximity that they have acquired with aspecific planet, and the stereo mix is dynamically rendered as a stereo sound source spatially located close tothe planet. Eventually, leaving or approaching a planet is sonically strengthened by a zoom sound effect, wherethe sound corresponds to the planet’s changing visual size. The real-time sound spatialization is executedby the spatializer included in the Oculus integration toolkit, while the stage before binauralization isprocessed through research-based plugins (SPARTA and 3D artificial reverberator) 20, 21.

In conversation with the 3D Audio Artists of the Works

In our effort to address the way that immersion serves disembodiment and the way that our perception is affectedby the utilisation of 3D audio in VR artworks, Henrik Oppermann and James Bulley, who worked on the spatializedaudio part of Evolver; Christian Heinrichs, who specialises in procedural audio9 and created thespatialized audio for The Jellyfish, and Andrea Manciati with his team, who designed and implemented the soundfor Space Walk, provided answers to a couple of questions through online correspondence with the authors. In allcases, 3D audio takes on the role of an interpreter of the virtual environment, shifting the immersive aspect onan unprecedented scale and extending the contemporary body to unmapped territories of human perception.

Henrik Oppermann and James Bulley were interviewed for their contributions to Evolver. They have workedwith spatialized sound as a core part of their practice. For them, 3D sound is an important aspect of theoverall work in virtual reality environments. Especially, in Evolver, they used a mixture of thirdorder ambisonic soundbeds and mono sound sources that are spatialized. These were then rendered dynamically tobinaural audio in headphones depending on the position of the listener.

Regarding the use of 3D audio in VR and the possibility to further implement it in their work, they argue that itgives a much closer experience to how we listen to sounds in real life. They are able to recreate a very naturalexperience while also having the ability to make sounds hyperreal and more present. Adding to this, Bulleystated that there is huge potential in head/body position tracking and ambisonics, especially in hybridscenarios such as augmented reality: creating platforms for audiences to experience the world around them andengage deeply with more-than-human realities and extra-sensory experiences through spatial sound.

When they were asked if 3D audio is a better expressive medium in comparison to stereo, Oppermann asserted thatwith stereo mixes, the users hear all the sounds inside their heads without any kind of externalisation. Thiscan really break immersion, as it takes them out of the experience. A powerful audio engine that gives a lot ofoptions on how to control the sound is apparently preferable. Bulley was more analytical, explaining thatdifferent configurations like mono, stereo, conventional spatial formats, and 3D sound all have their places andcan be very expressive in different ways, since it depends on aesthetic conceptual choices and also on thecontext and environments that the work is for and presented in. At the same time, 3D sound has huge potentialfor truly immersive experiences that can engage with space in a way that stereo often cannot.

He also stressed that in terms of working with predefined spatialization features in game engines (or similar) orworking with handmade spatialization features, both can have their place, and there is much fertile ground inbetween. He refers to some exceptionally interesting audio engines and plugins that are currently available(such as the IEM plugin suite, Blue Ripple Sound, SPAT, Sound Particles, DearVR, etc.), that provide unique andpowerful scenarios to work with. Though he added that in many cases, where the work is driven by particularconcepts and needs, a bespoke, handmade solution (or a collaborative approach using some of these plugins oraudio engines) can be the only truly progressive way to work.

With reference to the subject of disembodiment and its relation to 3D audio, they were asked if 3D audiostrengthens the essence of presence from the user's perspective or the elements that the user meets in the VRenvironment. They both agreed that in essence, stereo brings disembodiment to the overall experience.Nevertheless, with 3D audio, users can connect the aural level with the visual level. 3D audio can create astrong sense of embodied presence within real, virtual, and augmented worlds that differs from other establishedsound formats.

An example of this condition in VR is where there are interactive sounding parameters at play (for example, handmovements linked to subtle particle sounds using motion tracking, or real-time monitoring of someone’sheartbeat linked to elements of a sound score). This embodied presence can, in fact, connect the aural andvisual elements in time and space, mapping a territory that Michel Chion usefully describes as‘synchresis’, allowing the play and interrogation of the spatial magnetization of sound with thevisual stimuli. They also underlined that works that involve 6DOF10 can create very interestingand novel aspects of embodiment for audiences moving in space, especially when teamed with ambisonic soundbedsthat move with the listener or can be “discovered” in the space by a listener.

For Christian Heinrichs, who has been incorporating 3D audio into his work since his undergraduate studiesbetween 2006 and 2010, the integration of the audio part of The Jellyfish was a much easier experience. Thespatialization plugins existed already (in this case he used Google’s Resonance SDK), and they allowed himto focus much more on the design of the sound, music and interactional aspects. In The Jellyfish, thisexperience is sonically quite unique compared to other experiences because it is so driven by the player’svoice and ears, literally disembodying the user’s voice and allowing them to control the environment.

Concerning 3D audio in VR and the potentialities of further implementation in his work, Heinrichs said thatinteractivity is the key to making next-level experiences. The fact that the user can look around and listen outfor different elements in a specific scene and be able to move towards or away from certain musical or sonicelements are vital key-points for him. Moreover, he emphasised that by establishing these two attributes, theusers can go much further, especially if they follow the path of hyper- embodiment or inhabitation; making themost subtle qualities of their very being in space have an effect on their surroundings, and so forth.

Heinrichs distinguished 3D audio and stereo as fundamentally different elements in VR creation. He claimed thatstereo is a great format for bringing powerful audio experiences across since it is already capable of giving astrong illusion of depth. However, he insisted that the users (arguably) do not “inhabit” a stereoimage, in the same way that they inhabit a spatial sound field. This opinion is verified if we take headrotations and positional movement into account. Heinrichs argued that it is the equivalent of comparing a videogame on a screen to a VR experience in a headset.

Regarding automated versus handmade 3D audio game engine qualities, he pointed out that there are aspects thatwill always be “automated” in spatial audio, especially in VR. For example, the soundfield has to berotated in real- time in order to respond to the users’ head movements and give them the impression thatthey are inside the sonic scene. In order to place a sound at a given location in space, the 3D audio designerneeds the audio engine (usually working jointly with the game engine) to encode a mono or stereo source to berendered at that particular location in space.

Another issue to consider is the distance attenuation, and other real-time effects that help mimic the way sound

behaves in the real world, along with the invention of sonic behaviours that might not be possible in the realworld, or their exaggeration, so that there follows a formation of a hyperrealistic space. Heinricks concludedthat sometimes it is worth designing a spatial soundfield outside of the game engine using ambisonic plugins andso forth, in order to have a static three-dimensional “bed” of sound that is decoded at runtime as asoundfield, again following the users’ head movements. Ultimately, these elements can be mixed andmatched. During such experiences, the 3D audio designer might also want to use straight stereo playback. Forexample, to contrast user interface features or other content that is rendered “inside the head”with 3D content in the scene, which is rendered “outside the head”.

In respect of the relationship between 3D audio and disembodiment, he argued that 3D audio is a huge contributorto the feeling of immersion and presence inside VR experiences; 3D audio can help the users perceive elementsthat they cannot actually see (e.g., behind them). The novelty of it alone can really trick people’s mindsinto thinking that they are really inhabiting the virtual environment.

Composer, performer, and media artist Andrea Manciati gave us more information regarding Space Walk. Thesonic part is divided into two categories: the sounds constituting the soundtrack of each planet, which uses thebinaural spatializer in Unity, mainly to compensate for the listener's head rotation and allow for a "credible"localization of sources. The second category involves some "baked" binaural tracks within the composition (i.e.,where the binauralization is embedded in the sound file rather than synthesised in real-time) layered withstereo ones to add lively and more "diffused" elements.

Manciati acknowledged that the potential of 3D audio in VR is very high. Though the tools available within gameengines are still in their infancy compared to the techniques that are available in more traditional audioenvironments (from DAWs to node-based programming environments), they have not yet been implemented in gameengines. On the other hand, he noticed a tendency to approach 3D audio often only as a technical way to enhancerealism, rather than a powerful creative possibility that could be based on a very different paradigm thanreality. He underlined that this tendency could limit its capabilities and risk keeping 3D audio in thesecontexts a bit superficial.

He addressed the complexity that sound in VR possesses by recognising the different implementations that stereoand 3D audio might have; the techniques applied are never neutral, and even if some could deliver objectivelymore details or information, it is not guaranteed that the creators will choose this direction in all cases. Hestated that the tendency to "engineerize" artistic choices (i.e., if the choice is more powerful,performant, newer, or more realistic, then it is automatically better), is a bit reductionist and does not helpthe cause of artistically relevant, expressive uses of certain technologies. Contrarily, he mentioned that heprefers non-automated 3D audio, where he has pure control over the making process, leaving a unique mark on hiscraft. He concluded that VR in general could be extremely powerful at evoking a sense of disembodiment, makingthis element one of the main axes of artistic experimentation. Certainly, attentive and creative strategiesusing 3D audio technologies could contribute to this aspect. Still, he highlighted that using a certaintechnology to define and measure such culturally loaded ideas as immersion and presence can run the risk ofsimplifying the concepts at stake. Lastly, he agreed that VR is powerful at reflecting on the ideas of presenceand embodiment, not so much because of its realism or because it makes us believe in something, but ratherbecause of the frictions it creates in the experience; in the dissonances and sense of estrangement it canintroduce.

Discussion

In our research, we explored the potentialities of disembodiment in artistic VR works and the modes in which 3Daudio functions in order to improve these experiences. 3D audio in the digital arts has been developing in thepast few years, along with the explosion of 3D audio utilisation in other sectors like entertainment (e.g.,video games). All three works that were presented belong to the category of 3D audio in immersive VR, and theyare useful to further explore the sense of disembodiment. The audio experience in all the works is a mixture ofrealistic and hyperreal sonic stimuli. The spatialized sound of the oxygenated blood flow in Evolver,the surrounding environment in The Jellyfish, and the majestic orchestral soundtrack of SpaceWalk form a type of hyperrealistic atmosphere, almost creating a sense of flotation.

Additionally, in all three case studies, head/body position tracking is the core component of the overallimmersive experience. In Evolver, the user explores the dendroidal cardiovascular system according tohis movements. In The Jellyfish, the invertebrates move according to the participant’s head,voice, and body movement, and in Space Walk, the user transports instantly from one celestial object tothe other, overcoming factors like distance and time via room-scale body movement.

Furthermore, sound localization is an important aspect regarding immersion and disembodiment; 3D audio is appliedby obtaining a peculiar localizational trait with the moving spatialized sound of the swirling blood flow inEvolver, the distant and then gradually close calls of the invertebrates in The Jellyfish, andthe corresponding soundtracks for each celestial body, where the sound object’s apparent width correspondsto the planet’s changing visual size in Space Walk. The users’ physical

cues and constraints are fully adjusted to specific 3D audio attributes which decode distance and direction ofsound in space, sometimes in an engaging and often extra-sensory way.

It is important to emphasise the fact that the selected artworks present a high aesthetic conceptual quality, andboth context or environment, combined with the use of 3D audio, can achieve a truly immersive experience.Although, as stated, spatialization plugins existed in the past, the applied technology of 3D audio in VR hastaken the experience to an innovative level. The aural and visual levels are interconnected and perceived by thecreators as a totally interactive container, triggering a diversity of sound parameters that emerge and becomeestablished as factors of immersion depth. It could be stated that in all works, the user's inhabitation(spatial and aural) and the so-called hyper-embodiment are attributes that connect while immersion occurs.Acoustic stimuli can be enjoyed in the virtual pace, amplifying the spatial sensations. Consequently, when asonic hyperrealistic framework is shaped, revolutionary, and intriguingly singular experiences emerge, enhancingthe bizarre and intense sense of immersion in these VR environments.

Conclusion

The body seems to defy gravity, in an almost disembodied state, as it experiences paths that the mind merelyindicates. The perception shifts qualities connected to a 3D audiovisual alternative reality. An embodiednon-human avatar is the leading actor who interacts with a dreamlike environment while expanding abstracttactile and kinesthetic cues. In these environments, there are no certain tasks given other than themesmerising, welcoming, and sometimes mysterious, immersive environments.

Through the analytical approach that was given by the 3D audio artists of the works that we explored, there is aclear sensical stance that binaural audio or spatialized three- dimensional audio may improve the feeling ofimmersion in virtual environments over a simple stereophonic audio delivery. Researching the implementation of3D audio, we demonstrated the affordances and the role that disembodiment acquires in the works, serving thefeeling of immersion in virtual reality environments, with a main focus on binaural audio.

Ultimately, sound becomes environmental, ecological, and universal. 3D sound acquires an omnipresent attribute,interconnecting embodiment, presence, and immersion in Virtual Reality. Thus, sound embeds the corporeal body invarious literal or metaphorical contexts where the interaction between embodiment and disembodiment results inan ontological shift; the participant objectifies the experience through these opposite sensations. Directionalcues that belong to the traditional stereo or even surround set-up could be less immersive compared to thespatial sonic advancements provided by 3D audio.

Binaural audio maintains its interaural cues by recreating the density and shape of a human head and by usinggross tactile-kinesthetic cues. Although the development of 3D sound in these contexts is still a nascent field,it is vital to elaborate more in depth through research studies that explore embodiment and disembodiment inrelation to immersive sound in VR environments. Undoubtedly, this impressive field and its advancements willhave even more implications if and when VR experiences become more prevalent in mainstream culture. For the timebeing, 3D audio is a sea of opportunity and a very exciting time to be working in and participating in thisfield for creators and researchers alike. In that fashion, 3D audio in VR can introduce a new revolutionaryconcept that reaches higher planes of existence in the embodiment/disembodiment interplay. It would be rewardingto come across more studies on the implementation of 3D audio in artistic VR works and research further on itsimmersive aspect in the near future.

References

1 In her effort to explore choreographic notions and processes related to kinaesthesis, Gibson takes us on a journey to her VR work, where new forms of embodiment are developed and studied for the performer and the audience alike.

2 Binaural audio is usually experienced with the use of headphones and it is a 3D audio effect that simulates sound as if it is being heard live.

3 Ambisonics is a complete spherical surround sound format. It covers sound sources above and below the listener in addition to the horizontal plane and can be applied through a spherical array of speakers. Ambisonics can also be decoded to binaural audio.

4 Spatial audio is processed sound, aiming at giving the impression of a sound source within a three-dimensional environment, in order to provide a more realistic experience to the listener.

5 In the Eyes of the Animal by Abandon Normal Services and Marshmallow Laser Feast, is another example of immersive VR artistic work that uses 3D audio. Natan Sinigaglia who is responsible for the visual part and the final experience design, responded to the questionnaire and provided valuable answers regarding 3D audio from his perspective. The answers are included in the appendix. https://docubase.mit.edu/project/in-the-eyes-of-the-animal/. Also, another work that deserves mentioning and makes unusual and innovative use of VR technology is Notes on Blindness, written by Amaury La Burthe, Arnaud Colinart, James Spinney, and Peter Middleton and produced by Archer's Mark, Arte France and French production company and film distributor ExNihilo. https://docubase.mit.edu/project/notes-on-blindness/

6 In Human Ecosystem Building, demonstrated by Lewis Saunders, who is a member of Marshmallow Laser Feast, we can observe the method that is used for the creation of the human cardiovascular system along with the mapping of the journey of breathing air through the human body: https://vimeo.com/695217247?login=true

7 A numerical space weather prediction is a method that uses mathematical models of weather forecasting that employ a set of complex equations to solve for various locations at both the surface and different heights (layers) of the atmosphere. https://www.weather.gov/media/ajk/brochures/NumericalWeatherPrediction.pdf

8 In creative media, soundbed stands for background sound or music, which is used in games, podcast episodes, and so forth.

9 Procedural audio is sound that is generated during the length of time that a programme takes to run. It creates the sounds that the participant experiences in an improvised manner, based on a set of pre-defined behaviours.

10 6 degrees of freedom (6DOF) define six axes on which a rigid body is capable of moving freely in a three-dimensional space.

Books

1 Christian J. Jerome, Bob G. Witmer, “Human Performance in Virtual Environments: Effects of Presence, Immersive Tendency, and Simulator Sickness”, paper based on a talk presented at Human Factors and Ergonomics Society Annual Meeting, Los Angeles, California, September, 2004, p.2613–2617, https://www.researchgate.net/publication/270722806_Human_Performance_in_Virtual_Environments_Effects_of_Presence_Immersive_Tendency_and_Simulator_Sickness.

2 Konstantina Kilteni, Raphaela Groten, Mel Slater, “The Sense of Embodiment in Virtual Reality”. Presence: Teleoperators and Virtual Environments 21 (4), 2012, p.373–387.

3 Pawel Grabarczyk, Marek Pokropski, “Perception of Affordances and Experience of Presence in Virtual Reality”, The Journal of the Philosophical-Interdisciplinary Vanguard 7 (2), 2016, p.37.

4 James J. Gibson, “The Concept of Affordances”, in Perceiving, Acting, and Knowing: Toward an Ecological Psychology, ed. Robert E. Shaw, John Bransford, Hillsdale, N.J., Lawrence Erlbaum Associates, 1977, p.67-82.

5 Ruth Gibson, “Falling Upwards: Somatic Sensing in Virtual Space”, Ph.D. Dissertation, RMIT University, 2019.

6 Sascha Spors, Rudolf Rabenstein, Jens Ahrens, “The theory of Wave Field Synthesis Revisited,” paper based on a talk presented at the 124th Audio Engineering Society Convention, https://www.researchgate.net/publication/228355734_The_theory_of_wave_field_synthesis_revisited

7 David G. Malham, Antony Myatt, "3-D sound Spatialization Using Ambisonic Techniques”, Computer Music Journal, 19 (4), 1995, p.58-70.

8 Brendan Kelley, Cyane Tornatzky, “The Artistic Approach to Virtual Reality”, paper based on a talk presented at the 17th International Conference on Virtual-Reality Continuum and its Applications in Industry, New York, November, 2019, p.1–5.

9 Rozenn Nicol, Binaural Technology, AES Monograph, 2010.

10 Micheal Vorländer, Auralization: Fundamentals of Acoustics, Modelling, Simulation, Algorithms and Acoustic Virtual Reality, Berlin/Heidelberg, Germany: Springer International Publishing, 2020.

11 Jens Blauert, Jonas Braasch, The Technology of Binaural Understanding, Springer, 2020.

12 Durand R. Begault, Leonard J. Trejo, 3-D Sound for Virtual Reality and Multimedia, Ames Research Center Moffett Field, California, 2000, p.41.

13 Mayo Kobayashi, Kanako Ueno, Shiro Ise, “The Effects of Spatialized Sounds on the Sense of Presence in Auditory Virtual Environments: A Psychological and Physiological Study,” Presence: Teleoperators and Virtual Environments 24 (2), 2015, p.163–174.

14 Fabian Brinkmann, Stefan Weinzierl, “Audio Quality Assessment for Virtual Reality”, in Sonic Interactions in Virtual Environments, Editors: Michele Geronazzo, Stefania Serafin, Springer: Cham, 2022, p.145-178.

15 Amanda Whiting, “Evolver review: Cate Blanchett Takes us Inside the Human Body in an 2022, https://www.independent.co.uk/arts-entertainment/art/reviews/evolver-review-tribeca-cate-blanchett-b2100999.html

16 James R. Marston, Jack M. Loomis, Roberta L. Klatzky, Reginald G. Golledge, Ethan L. Smith “Evaluation of Spatial Displays for Navigation Without Sight,” ACM Transactions on Applied Perception (TAP) 3 (2), 2006, p.110-124.

17 Ewen Chardronnet, “Mélodie Mousset: Projecting the Body into Virtual Reality”, Makery https://www.makery.info/en/2020/10/03/melodie-mousset-projeter-son-corps-hors-de-soi-jusque-dans-le-virtuel/

18 Stefania Serafin, Cumhur Erkut, Juraj Kojs, Niels C. Nilsson, Rolf Nordahl, “Virtual Reality Musical Instruments: State of the Art, Design Principles, and Future Directions,” Computer Music Journal 40, 2016, p.22–40.

19 Andrea Mancianti, Sebastian J Schlecht, Vesa Välimäki, Riku Järvinen, Esa Kallio, SPACE WALK - Visiting The Solar System Through an Immersive Sonic Journey in VR, Proceedings from Nordic Sound and Music Computing Conference, Denmark, November 2021.

20 Leo McCormack, Archontis Politis, “SPARTA & COMPASS: Real-Time Implementations of Linear and Parametric Spatial Audio Reproduction and Processing Methods,” paper based on a talk presented at AES International Conference on Immersive and Interactive Audio, York, March, 2019.

21 Sebastian J. Schlecht, “FDNTB: The Feedback Delay Network Toolbox,” paper based on a talk presented at the 23rd International Conference on Digital Audio Effects DAFx, Vienna, September 2020, p.211–218.

Appendix

. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.

Evolver by Marshmallow Laser Feast, by Henrik Oppermann and James Bulley. Full answers by HenrikOppermann and James Bulley

On the utilisation of 3D Audio in VR.

1. Have you ever experienced 3D audio in VR before? Does "Evolver" involve normal stereo, or isit using binaural delivery?

HO: Yes. Many times. :D

JB: Yes – both of us work with spatialized sound as a core part of our practice. Within VR, 3D sound isvery much a key part of making work within the medium. A recent piece that Henrik and I have been collaboratingon, Evolver, with the art collective Marshmallow Laser Feast, is a current example of this.Evolver uses a mixture of third order ambisonic soundbeds and mono soundsources that are spatialized.These are then rendered down dynamically to binaural in headphones dependent on the position of the listener.

On immersion and interactivity

2. What are the potentialities of 3D audio in VR and how would you further implement them inyour work?

HO: It is not only a potential. It gives a much closer experience to how we listen to sounds in real life. We areable to recreate a very natural experience, whilst also having the ability to make sounds hyperreal and morepresent. JB: Adding to this, I think there is huge potential in head/body position tracking and ambisonics,especially in hybrid scenarios such as augmented reality: creating platforms for audiences to experience theworld around them and engage deeply with more-than-human realities and extra-sensory experience through spatialsound.

On sound design

3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes,would you like the part of 3D audio to be automatically designed from a game engine or handmade, that is, tobe specially made from you as an exclusive feature (control all sonic attributes in the scenes of thework)?

HO: Yes. 3D audio is 100% necessary for immersive experiences. With stereo mixes you will hear all the soundsinside your head without any kind of externalisation. This can really break immersion and take you out of theexperience. A powerful audio engine that gives you a lot of options on how to control the sound is preferable.

JB: For me, I think different configurations like mono, stereo, conventional spatial formats and 3D sound allhave

their places and can be very expressive in different ways – it very much depends on aesthetic conceptualchoices, and also on the context and environments that the work is for and presented in. That said, 3D sound hashuge potential for truly immersive experiences that can engage with space in a way that stereo often cannot. Interms of working with predefined spatialization features in game engines (or similar) or working with handmadespatialization features, both can have their place, and there is much fertile ground in between. There are someexceptionally interesting audio engines and plugins that are currently available (such as the IEM plugin suite,Blue Ripple Sound, SPAT, Sound Particles, DearVR, etc) that provide unique and powerful scenarios to work with,but in many cases, where the work is driven by particular concepts and needs, a bespoke, handmade solution (or acollaborative approach using some of these plugins / audio engines) can be the only truly progressive way towork.

On embodiment/disembodiment

4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect doyou think that 3D audio enhances the feeling of presence and contributes to the creation of a "disembodiedfeeling"?
In other words, does 3D audio strengthen the essence of presence from the user'sperspective or the elements that the user meets in the VR environment?

HO: Stereo really brings disembodiment to the overall experience. With 3D audio you can connect the aural levelwith the visual level.

JB: Discussions of embodiment and disembodiment in relation to experiences of 3D audio are complex andmultifaceted. There is no doubt in my mind that 3D audio, explored sensitively (whether heard over headphones,or from conventional multichannel speakers’ systems or technologies like wavefield synthesis), can createa strong feeling of embodied presence within real, virtual and augmented worlds that differs to otherestablished sound formats. One simple example of this in VR is where there are interactive sounding parametersat play (for example hand movements linked to subtle particle sounds using motion tracking, or real-timemonitoring of someone’s heartbeat linked to elements of sound score). As Henrik mentions, this can reallyconnect the aural and visual in time and space, mapping a territory that Michel Chion usefully describes as‘synchresis’ and allowing play and interrogation of the spatial magnetization of sound tovisual.Works that involve 6DOF (6 degrees of freedom), can create very interesting and novel aspects ofembodiment for audiences moving in space, especially when teamed with ambisonic sound beds that move with thelistener, or can be ‘discovered’ in the space by a listener. The development of 3D sound in thesecontexts is still a nascent field, and it is vital for there to be more in depth research studies exploring

embodiment and disembodiment in relation to immersive sound (particularly in VR environments).

B. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.

The Jellyfish by Mélodie Mousset (HanaHana) and Edo Fouilloux and Christian Heinrichs. Fullanswers by Christian Heinrichs.

On the utilisation of 3D audio in VR.

1. Have you ever experienced 3D audio in VR before? Does "Jellyfish" involve normal stereo, oris it using binaural delivery?

CH: I have been integrating 3D audio into my work since my undergraduate studies in Glasgow between 2006 and2010. Back then it was harder to get things like ambisonics and HRTFs to work and… In 2012 I developed afull audiovisual rendering and interaction system during a project placement at BBC R&D. There were noproper VR headsets and game engines were not as advanced and user friendly as they are today. This system used aportable projector mounted on a helmet, pointing at a curved retro- reflective curtain that would reflect theprojection straight into the direction of the wearer’s eyes. I used a Kinect to follow and detect bodymovements, Blender Game Engine to render the visuals and Puredata and some external libraries to do all thesound rendering. There were so many steps involved in getting all the bits of technology to talk to each other(without crashing) that it became difficult to go into a lot of details with the demo experiences. However, Idid manage to reach a sense of disembodiment in the “Beach Demo”, which involved swimming both overand underwater, exploring different parts of a beach. https://vimeo.com/manage/videos/59663582 (apologies forthe low framerate on the POV part of the video – the experience was in fact smooth, but the screen capturewas impossibly slow) https://vimeo.com/manage/videos/59354963 (remake of ascene from the point and click adventure “full throttle”).

Since then, things have become easier by orders of magnitude. This is largely thanks to the rebirth of HeadMounted Displays from 2013 onwards and the prevalence of accessible game engines such as Unity and Unreal. Thus,making the Jellyfish experience was much easier in terms of setting up 3D audio. The spatialization pluginsexisted already (in this case we used Google’s Resonance SDK, but there are others), and it allowed me tofocus much more on the design of the sound, music and the interactional experience with each one. I think thisexperience is sonically quite unique compared to other experiences because it is so driven by the player’svoice and ears, literally disembodying your voice and allowing you to control the environment. A similar piecethat’s worth checking out is “Breathe”: https://headspacestudio.com/projects/breathe/

On immersion and interactivity

2. What are the potentialities of 3D audio in VR, and how would you further implement them inyour work?

CH: The potential for using 3D audio in VR is huge, and relatively unexplored in my opinion. I really think thatinteractivity is the key to making next-level experiences. The fact that you can look around and listen out fordifferent elements in the scene. That we can then move towards or away from certain musical or sonic elements.Once you’ve established those two things, you can go much further, especially if you follow the path ofhyper-embodiment / inhabitation, making the most subtle qualities of your very being in space have an effect onyour surroundings, and so forth.

On sound design

3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be speciallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)?

CH: No, I don’t think “better” is the right word. It is, however, fundamentally different, inmy opinion. Stereo is a great format for bringing across powerful audio experiences and already is capable ofgiving a strong illusion of depth. However, you (arguably) don’t “inhabit” a stereo image, inthe same way you inhabit a spatial sound field. Especially once you take head rotations and positional movementinto account. It is the equivalent of comparing a video game on a screen to a VR experience in a headset.Regarding automated vs handmade qualities– there are aspects that will always be “automated”in spatial audio, especially in VR. For example, the soundfield has to be rotated in real-time in order torespond to your head movements and give you the impression that you’re inside the sonic scene. In order toplace a sound at a given location in space, you need the audio engine (usually working in tandem with the gameengine) to encode a mono or stereo source to be rendered at that particular location in space. You also have tothink about distance attenuation, and other real-time effects that help mimic the way sound behaves in the realworld (or do things that might not be possible in the real world or exaggerate them to create a hyperrealisticspace). Sometimes it’s worth designing a spatial soundfield outside of the game engine using ambisonicplugins and so forth, in order to have a static three-dimensional “bed” of sound that is decoded atruntime as a soundfield, again following the player’s head movements. These things can be mixed andmatched. During such experiences you might also want to use straight stereo playback, for example to contrastuser interface elements or other content that are rendered “inside the head” with 3D content in thescene, which is rendered “outside the head”.

On embodiment/disembodiment

4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio , in what respect doyou think that 3D audio enhances the feeling of presence and the way it contributes to the creation of a"disembodied feeling"? In other words, does 3D audio strengthen the essence of presence from the user'sperspective or the elements that the user meets in the VR environment?

CH: I believe I answered this question above. I think 3D audio (and you will hear many people say this) is a hugecontributor to the feeling of immersion and presence inside VR experiences. 3D audio can help you perceive thatwhich you cannot see (e.g. behind you). The novelty of it alone, when done right, can really trickpeople’s minds into thinking that they are really inhabiting the virtual environment. It will beinteresting to see how this develops, if/when VR experiences become more prevalent in mainstream culture. Rightnow, it’s a sea of opportunity and a very exciting time to be working and participating in this field.

Here’s a mini “sneak peek” I had done of The Jellyfish that gives a better impression than allthe other material that’s available of how the experience works and feels like: https://drive.google.com/file/d/1wzhPLrMJAZ7Ep9W5yqd1g950FPknGifl/view?usp=sharing

Here’s a non-public dev video where I’m singing into the audio system. Might give you a better senseof how the different audio layers work together: https://drive.google.com/file/d/1X4i5rNntwx5Ygu7Fx5oKPNv36C74svXL/view?usp=sharinghttp://dolphinclub.website/reveries/

C. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.

Space Walk by Andrea Mancianti, Sebastian J. Schlecht, Vesa Välimäki, Riku Jarvinen, and EsaKallio. Full answers by Andrea Manciati.

On the utilisation of 3D Audio in VR.

1. Have you ever experienced 3D audio in VR before? Does "Space Walk" involve normal stereo, or is it usingbinaural delivery?

AM: Yes, I have experienced a few VR works using so called 3D or "immersive sound" before, but admittedly used ingeneral pretty cosmetically, for reasons that might be related to the tools available. This particular pieceuses two main strategies: the sounds constituting the soundtrack of each planet are using the binauralspatializer in Unity,

mainly to compensate for the listener's head rotation and allow for a "credible" localization of sources. At thesame time within the music itself there are some "baked" binaural tracks (i.e. where the binauralisation isembedded in the sound-file, rather than synthesised in real-time) layered with stereo ones, to add lively andmore "diffused" elements. More details on the implementation can also be found in the paper.

On immersion and interactivity

2. What are the potentialities of 3D audio in VR and how would you further implement them in your work?

AM: Overall the potential is very high, but the tools available within game engines are still in their infancy,compared to the techniques that are available in more traditional audio environments (from DAW to node-basedprogramming environments), but not yet implemented in game engines. On the other hand, I see a tendency to thinkof spatial audio often only as a technical way to enhance realism rather than a powerful creative possibility,that could be based on a very different paradigm than reality. This tendency I think could limit its potentialand risk keeping 3D audio in these contexts a bit superficial. In my work the relationship between space andsound is central, so I definitely will use such techniques in the future, but the majority of my personalartistic work happens outside of VR because of the limitations I mentioned.

On sound design

3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be especiallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)

AM: I think the idea of better or worse is definitely tricky when talking about expressive means. I think theyare simply different. The techniques are never neutral and even if some could deliver objectively more detailsor information is not granted this would need to be preferred in all cases. I think this tendency to"engineerize" artistic choices (i.e. if it is more powerful, performant, newer or more realistic isautomatically better) is a bit reductionist and doesn't help the cause of artistically relevant, expressive usesof certain technologies. And for the second part of the question, no I would definitely not want those to beautomated. Hand making these aspects is totally part of my craft and I wouldn't trade for a script.

On embodiment/disembodiment

4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect doyou think that 3D audio enhances the feeling of presence and contributes to the creation of a"disembodied
feeling"? In other words, does 3D audio strengthen the essence of presence from theuser's perspective or the elements that the user meets in the VR environment?

AM: I think VR in general could be extremely powerful at modulating this sense of embodiment/disembodiment, andthat could be one of the main axes of artistic experimentation with this medium. And of course, attentive andcreative strategies using 3D audio technologies could contribute to this aspect. But once again, (and I might bemisinterpreting the question here) I feel that using a technology to define and measure such culturally loadedideas as those of immersion and presence, can run the risk of simplifying the concepts at stakes. VR is in myopinion powerful at making us reflect on the ideas of presence and embodiment, not so much because of itsrealism, or because it makes us believe, but rather because of the frictions it creates in the experience. Inthe dissonances and sense of estrangement it can introduce.

D. Questionnaire on 3D audio, immersion and embodiment/disembodiment in VR artistic works.
Full answersregarding In the Eyes of the Animal by Abandon Normal Services and Marshmallow Laser Feast, by NatanSiningaglia.

On the utilisation of 3D Audio in VR.

1. Have you ever experienced 3D audio in VR before? Does "Evolver" involve normal stereo, or is it usingbinaural delivery?

NS: We used binaural delivery On immersion and interactivity

2. What are the potentialities of 3D audio in VR and how would you further implement them in your work?

NS: 3D binaural audio in VR has a huge impact on the level of immersion of a VR experience.

Think about how much information you can get, with closed eyes, just from listening to the sounds surroundingyou. You can understand what kind of dynamic you are in the middle of, and you get an idea of the qualities ofthe space (size, type,...)..In VR especially, the sound can dialogue in such a meaningful way with the visual,giving a "physicality" and a sense of consistency to objects in the view and even outside the view! By the soundof a virtual object, we can have a deep intuition about how we can interact with it, since the sound isconnected to the physical body of the object, it's consistency, mass, temperature... we can guess howdangerous/soft/pleasant/hard/... it would be to touch it. this kind of influence is subconsciously very powerfuland can be used to determine the psychological dimension of a user in a space. Audio can be even used to directthe attention of the user. this compensates a bit for the loss of the visual role of the (movie) director (beingVR a medium where is the

user that decides the shot, the framing, the view over the reality)

On sound design

3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be speciallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)?

NS: I don't think 3d audio is a better expressive medium than stereo.

I think that 3d audio is an expressive medium in which (in a much more effective and profound way, compared tostereo) specialization and immersion are key elements of artistic expression and methodology. Said so, from theperspective of a virtual scene creator, it's crucial to have control of the sound engine as much as the visualcounterpart, to be able to make use of all the sound generation-perception techniques in the composition of theexperience.

On embodiment/disembodiment

4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect doyou think that 3D audio enhances the feeling of presence and the way it contributes to the creation of a"disembodied feeling"? In other words, does 3D audio strengthen the essence of presence from the user'sperspective or the elements that the user meets in the VR environment?

NS: I replied above.

The Enigma A/V performance & the concept of Agnostic Media Environment (AME)

Yan Breuleux1, Alain Thibault2, Rémi Lapierre3

1 École des arts numériques, de l’animation et du design, Université du Québec à Chicoutimi
2 The ELEKTRA International Digital Art Festival, The International Digital Art Biennial (BIAN) 
3 École des arts numériques, de l’animation et du design, Université du Québec à Chicoutimi 
Montréal, QC, Canada
yan_breuleux@uqac.ca atelektra@gmail.com lapierre.remi@gmail.com

Abstract

The occupation and reappropriation of industrial wastelands has become a part of the digital art festival scene. In some cases, new apparatuses are introduced, while in others, existing architecture is utilized for artistic purposes. Many cities and municipalities offer artists historically significant projection surfaces that shape the conceptualization of immersive art projects. In this context, multiple large-scale installation projects have emerged with the recent growth of the experiential creative industries sector. Against this backdrop, the present article raises the issue of a specific experience-design practice in the field of audiovisual (A/V) performance. More precisely, we examine a proposed conception process to help bring about a better understanding of cross-platform production approaches. The question we ask is: In the artistic context of the ever-changing digital arts, how can we account for the multiplicity of possible appearances of the same artistic project? To answer this question, we present the storytelling of the Enigma project and explain how producing a matrix of 3D environments, which can be deployed on a very wide variety of media, supports the proposal of the Agnostic Media Environment (AME) concept.

Keywords

Immersive Storytelling, Extended Reality (XR), Spatial Augmented Reality (SAR), narrative environments, multi-platform, video mapping, FullDome, A/V performance, Agnostic Media Environment (AME).

DOI

10.69564/ISEA2023-44-full-Breuleux-et-al-Agnostic-Media-Environment

Introduction: the experiential industries

This article is written from the standpoint of examining a conception process as opposed to a reception process. As such, the article is located at the boundaries of practical and theoretical aspects (practitioners and theorists)¹,² of immersive storytelling in relation to an A/V performance practice deployed on multiple display systems.³ For many years this practice has been evolving within digital art festivals, which program works drawing on a wide variety of projection mediums. Numerous visual music festivals, such as Mutek (CN), Elektra (CN), Live A/V (IT) Nemo (FR), Scopitone (FR), and the Mapping Festival (CH), to name just a few, make use of industrial wastelands, introduce their own projection systems and use the facades of existing places.⁴ Laboratories, such as the MetaLab of the Society for Arts and Technology,⁵ are developing innovative new tools for artists in order to stimulate creativity in the field of immersive experience design.⁶ For example, the MAPP_MTL organization, in the entertainment district of the island of Montreal, provides artists with historically charged surfaces that shape the conceptualization of their works.⁷ Also worth mentioning is the recent phenomenon of largescale exhibitions such as the Atelier des Lumières in Paris⁸  and the Oasis of the Palais des congrès de Montréal.⁹ The evolution taking place within these media environments has raised a research question: In the artistic context of the ever-changing digital arts, how can we account for the multiplicity of possible appearances of the same artistic project? To answer this question, we will begin by tracing the origins of the practice of A/V performance and how it connects with environments in the mixed reality spectrum of extended reality (XR).¹⁰ Next, based on analysis of the Enigma cross-platform art project, we will define the theoretical contours of this article’s main proposal, i.e., the concept of Agnostic Media Environment (AME). Our hypothesis is that it is possible to design an informational matrix composed of 3D environments that can be adapted to an infinite number of projection surfaces and installations.

A/V performance community of practice

The practice of A/V performance has its roots in the many forms of sound visualization,¹¹ from colour organs¹² to experimental film and video and the emergence of multimedia that have shaped the practices of visual jockeys (VJ).¹³ The launch of this practice dates to the mid-1990s and the immersive scenographies featured in the performances of the Granular Synthesis duo, made up of artists Kurt Hentschlager and Ulf Langheinrich. The duo left their mark on the A/V performance field with the multi-screen performance Model 5 (1994-2007).¹⁴ In spite of the existence of a vibrant and active community of practice since the midnineties, the designation of Performance A/V or Live A/V only really became established in 2010 with a special issue of the journal Musique et culture digitale (MCD, 2010) that cited numerous performing artists.¹⁵ That same year, Chris Salter published the book Entangled: Technology and the Transformation of Performance in which he situated this current at the crossroads of the performative arts of theatre and dance. In describing the performances of Granular Synthesis, he used the term “performative screens.”¹⁶ Grayson Cooke for his part characterized this practice as “Live audio-visual media performance” ¹⁷. Other terms include “Digital Live Audiovisual Arts” (DLAA)¹⁸ or simply “Live visuals”.¹⁹ The field has been enriched by such noted artists as Rioji Ikeda, Ryoichi Kurokawa, Herman Kolgen, Carstein Nicolai, and Matthew Biederman. To this list should be added artists using the same language elements in installation format, such as Can Buyukberber, Refik Anadol, Robert Seidel and Chris Salter. Collectives and studios have also joined in this trend with the label AntiVj (which gave rise to the studio of artist Joanie Lemercier), Purform, Ottolab, Incite, and Semiconductor. In the tradition of Kraftwerk’s large-scale POP electronic music audiovisual performances, it is also worth mentioning the immersive installations of Amon Tobin and Richie Hawtin, aka Plastikman. Many artists in the A/V performance community of practice are interested in various forms of sound visualization.²⁰ These audiovisual performances, delivered in renowned festivals such as Mutek (CN) and Elektra (CA), are presented on multiple projection-screen scenographies. The underlying artistic principle, which animates the community of practice of this sector of the creative industries, is located mainly in cross-modal practices that involve the use of various technologies of editing or real-time visual and sound form generation. The Limeart content-creator group has published several posts on their blog reviewing real-time video creation tools, under the title “mega list of Vj Software and Vj tools.” Without presenting an exhaustive list, these tools can be classified into different categories.²¹ 

As Table 1 illustrates, the tools most commonly used by the A/V and VJ performer community of practice are classified as live visuals, node-based environments and creative programming tool kits. In addition, interoperability solutions are used to allow various software to talk to each other. For example, the Spout (PC) or Syphon (Mac) applications now let users create procedural animations in Touch Designer that will be displayed within the Unreal Engine and Unity game engines (and vice versa). The particularity of game engines is that they incorporate live visual tools and visual programming for creative coding approaches. Thus, all of these tools merge data capture, real-time video computation and projection technologies on a diversity of systems within a single technological environment. For the purposes of this article, it is worth noting that the intermediate solutions of visual programming and creative coding include tools for procedurally generating virtual environments that allow the use of a VR headset.

The SPARK (Spatial Augmented Reality Construction Kit) real-time visual solution produced by TecArtLab is the closest to the research subject of this article.²² The LimeArtGroup describes the tool as being used to create an “immersive interactive spatial augmented reality installation.”²³ The software is a “cross-platform tool” for “real-time generated virtual content.”²⁴ The user can design a “360° VR environment no matter the shape of your surfaces.”²⁵ The current progression of game engines suggests many possibilities in this area. The SPARK software is based on a 3D matrix form that can be used on multiple systems.

Real-time visual creation software is increasingly using real-time 3D rendering engines. What these engines have in common is the ability to synchronize, capture, real-time computation and projection technologies across multiple platforms, from the web to virtual reality to various multiscreen configurations. To designate these practices of using visuals in real time for screen scenographies, Chris Salter, in line with the approach of expanded cinema, uses the expression “the Screen as Environment.”²⁶

The environmentalization of the image

The research presented in this article is based on the observation of Andrea Pinotti, from the field of immersive experience composition, that there has been a shift away from the notion of image and toward environment, what he terms “environmentalization of the image.”²⁷ In an immersive context—unlike a two-dimensional image—the space occupies the viewer’s entire field of view. The framing disappears in favour of the environment, which, in its modeling, becomes the narrative architecture. In video games, Henry Jenkins has addressed this aspect with the concepts of environmental storytelling and spatial stories.²⁸ For Janet H. Murray, this aspect is inherent in the use of digital technologies: “The new digital environments are characterized by their power to represent navigable space.”²⁹ These navigable digital spaces are now finding their equivalent in the real world. It is no coincidence that Marie-Laure Ryan has explored this question from several angles in literature. In her book Narrating Space / Spatializing Narrative, she analyzes the relationship between geographical space and literary fiction.³⁰ In the entertainment industry, Disney illustrator Don Carson analyzes the concept of environmental storytelling for theme park design. The narrative dimension grows out of the very experience of the space. In Carson’s own words: “One of the trade secrets behind the design of entertaining themed environments is that the story element is infused into the physical space a guest walks or rides through.”³¹ In fact, the notion of spatial experience embedded in the space itself is at the core of theme park literature as developed by Scott A. Lukas.³² Based on this same notion, Tricia Austin, in exhibition architecture, has precisely analyzed the relationship between narrative environments and experiential design. His model, “The tripartite network model of narrative environments,” is rooted in the relationships between people, environments and stories.³³ On this subject, Stephanie Riggs draws no distinction between virtual and physical reality. In her book The End of Storytelling, she proposes the concept of Storyplex. In interactive exhibition design, she describes a shift from the paradigm of cinematic framing to the sphere of interactive environments.³⁴ To sum up, the multidisciplinary literature on the subject examines how spatial modeling influences the shaping of narratives. In this context, the notion of narrative environment situates the research at the boundaries of the physical and the virtual.

Spatial Augmented Reality (SAR) in the Extended Reality Continuum (XR)

The boundaries between the virtual and the real are precisely the location of the concept of XR (extended reality) interfaces or the equivalent term of Cross Reality.³⁵ This term has proven necessary in order to update the reality-virtuality continuum as proposed by Milgram and al. in 1994.³⁶ The spectrum is an umbrella term covering what David Bolter calls “reality media”³⁷ such as virtual reality (VR), augmented reality (AR) and virtual worlds (VWs).³⁸ Although the approach is not mentioned in the literature, A/V performance practices can be situated within XR: the field of optics includes a Spatial Augmented Reality (SAR) sector.³⁹ Bimber and Raskar, in their book on optical technologies, describe setups such as Automatic Virtual Environments (CAVE), panoramic, architectural video mapping and FullDome spherical screens as “surround screens.” The knowledge contained in SAR optics has its origins in the oldest forms of immersion, harking back to puppeteers’ illusionist play of shadow and light, anamorphic paintings, and the camera obscura, up to the celebrated Pepper’s Ghost.⁴⁰ Many SAR devices rely on the same principles of syncing video projection to create projected environments adaptable to multiple spaces. What is involved here is augmented spatiality since these systems exploit anamorphic vision processes to erase, use, and “hijack” the multiplicity of projection surfaces to create illusionist spaces. The space is transformed by the artificial amplification of surfaces, the animation of virtual lights, the creation of faux spaces and the transformation of geometric spaces into curved surfaces. SAR devices allow for erasing the notion of screen in favour of the screen environment. By adapting digital content to architectural environments, objects and bodies, these optical processes virtualize real space. In turn, this virtualization of space enables the creation of collective immersive experiences that transform the exhibition space into an interactive environment. For example, the TeamLab Borderless creative studio creates projection museum environments where the boundaries between architecture and projector light merge within the same ambient space.⁴¹ Largescale museum installations thus become material environments transformed by the use of virtual light. In short, the notion of a screen environment specific to A/V performance using SAR technologies in an XR context opens avenues for considering the projection of the same project across multiple platforms.

Methodology

Figure 1: Enigma Multiplatform Project from 2017 to 2022. Multiscreen, Elektra, Usine C, Montréal, 2017. HD, Festival Nemo, La Villette, Paris, 2018. FullDome, Society for Arts and Technology (SAT), Montréal, 2019. Horizontal panorama, Elektra, Centre Culturel Canadien, Paris, 2020. Streaming HD, Galerie Elektra, Montréal, 2021. Circular Panorama, Coraulis, Ensa, Nantes, 2022. © Breuleux

Our methodology is located within the creation-as-research approach as defined by Chapman and Sawchuk.⁴² Following the method of heuristic cycles put forward by Louis-Claude Paquin,⁴³ we propose to look at how the projection of the Enigma project across multiple devices, conceived from the outset as an agnostic environment, has helped define the contours of the concept. As Figure 1 shows, our research, conducted between 2017 and 2022, consisted of storytelling that emerged from the different iterations of the Engima project. In this method, each new iteration further refines the theoretical contours of the proposed concept. 

Enigma A/V performance storytelling

The Enigma project is based on the world of famed mathematician Alan Turing, and seeks to create an experience inspired by the technological/scientific imaginary surrounding artificial intelligence.⁴⁴ The project explores the transposition of the notions of encryption and cryptography to a poetic exploration of the famed mathematician’s thinking.⁴⁵ The experimental space is based on a combination of nine sound visualizations projected in the form of environments: 1- data, 2- speech to text, 3- amplitude; 4-frequencies; 5- 3D shapes 6- illustration sequences 7- Hardware of the computer 8- point clouds 9- word clouds. Environments are modified in real time using a touch interface allowing the user to change the parameters of deformation, temporality, object quantity and camera viewpoints.

As shown at the first photo of Figure 1, the first multiscreen version in 2017, featured as part of the Elektra festival, consisted of an installation simultaneously showing the nine visualizations of composer Alain Thibault’s musical score on a 3X3 grid of 55” monitors. The second version, created in 2018, featured a single-screen HD format that distributed the nine visualizations over time according to a new 35-minute musical composition. In 2019, we adapted the linear performance into FullDome format. Each section of the piece was presented as a dive into procedural three-dimensional forms. 

Figure 2: 210° capture of the previz system. © Lapierre.

Thanks to its initial structure, the project made it possible to move from a two-dimensional screen format to hemispherical projection surfaces. We used a virtual reality (VR) preview system to design the scenario for the workshop performance (figure 2). Subsequently, the 2020 dual-screen panoramic version of the Canadian Cultural Centre in Paris was adapted in a few hours. For the 2021 streaming version, amid lockdowns, in order to configure the piece for a new performance context, we added networked tools that allowed for broadcasting directly on the YouTube platform. The 2022 version was adapted to the system of the centre d’Observation en Réalité Augmentée et Lieu d’Immersion Sonore (Caurolis) de l’École Nationale Supérieure d’architecture de Nantes (ENSA).⁴⁶ The next iteration of the Enigma project will tap into the expressive potential of the motion capture system to create a control interface projected in the centre of the installation.

Each new iteration is created thanks to the arrangement of the cameras within the environment (single camera for the HD version, nine cameras for the multi-screen installation, 210° FullDome camera for the Satosphere dome, etc.). Multiple camera synchronizations allow for quickly configuring visualizations within procedural environments to generate new versions on site. As an example, the figure 3 is based on the random movement of nine cameras inside a 3D environment. Each screen is a particular point of view of the same reality.

Figure 3: Nine views of the same 3D environment. © Breuleux

Given the dynamic relationships between the objects and the music, the duration of the environments as well as the transformation parameters can be manipulated. The Enigma project consists of a series of 3D environments. As such, each projection and translation of the experience for a new setup affords an opportunity to reinvent certain elements of the project.

To summarize the project approach, each installation offers an opportunity to renew the project experience. However, one important aspect must be stressed: each experience is stand-alone in its presentation. There is no real dialogue between the different iterations. In other words, for the Enigma project, despite the variety of manifestations of the same project, each new version constitutes a work in its own right. 

The 3D Agnostic Media Environment (AME)

The concept of 3D Agnostic Media Environment (AME) was coined to address a particular workflow guided by the need to adapt each project to the expressive potential of multiple projection contexts. In software terminology, “agnostic” refers to the process of automated adaptation to different devices and operating systems. A website, for example, will recognize the resolution of a phone, a tablet model, or a type of web browser. These constraints are relatively simple since the two-dimensional viewpoint of smartphones remains relatively stable. When it comes to the match between real and virtual spaces, the logic of adaptation is more complex. For FullDome projection domes alone, there are 180° and 210° projection angles, single and multi-channel projection solutions, as well as multiple projection systems. In contrast to movie theatres that use standardized systems, planetariums rely on a wide variety of systems and standards ⁴⁸. Generally speaking, the AME concept is a response adapted to the ecosystem of the experiential sector in which each new projection demands a degree of reinvention of the project components.

The proposed concept of agnostic 3D environment emerging from analysis of the narrative of the different iterations takes the shape of a world of aesthetic and experiential rules that can be transposed on multiple Spatial Augmented Reality (SAR) platforms. The term “media environment” is used insofar as each new version helps gradually establish an eco-system conducive to communicating information. To summarize in basic terms: By staging the viewpoints, AME exploits the specific expressive potential of each new apparatus.

The 3D agnostic media environment approach determines a workflow. Considered from a production rather than reception perspective, the term “agnostic” takes up the logic inherent to responsive and adaptive design web approaches in which, from the outset, the content is imagined according to its multiple permutations. The agnostic environment thus appears as a space of language governed by aesthetic rules that organize the visual, sound and interactive dimensions.

According to the diagram in Figure 4, AME, unlike software, has forms that possess a certain stylistic signature with respect to rhythm, composition, lights and camera movements. Sometimes, the experience is delivered only once on a site-specific basis. The environment, because it can be configured and manipulated in real time, makes it possible to operate the projection system as an instrument.⁴⁷ In the succession of procedural environments, the design of interactions between the sounds and visuals are part of the organization of the digital material.

Figure 4: The Multiplatform AME Model. © Breuleux

Each new installation requires some reorganization of the experience. Depending on the context, it will be necessary to change the display parameters and recompose the elements. Adaptations are made through the virtual reality (VR) preview format of the experience. For example, for several AME projects, a satosphere-scale VR dome was used to validate the formal elements of performances. Thus, the idea is not just to automatically transfer information from one platform to another, but also to leverage a preview environment to be able to modify the control parameters, the scale of the shapes, and the points of view according to each medium. The agnostic environment also opens avenues for exploring apparatuses that were not included in the original media environment. For example, architectural video projection entails a process of on-site intervention specific to a particular facade. To stage the experience, it will be necessary to take the expressive potential of the location and the façade into account.

AME is therefore not associated with any one particular tool. Rather, the approach confers an aesthetic unity shaped by the physical, temporal and technological conditions of the different contexts. The AME principle can be reformulated by designing a “spatial story path,” organized according to a certain duration and which, to construct the experience, exploits the expressive potential of the stimulation technologies specific to each new apparatus. 

Figure 5: White Box versions from 2011 to 2012. Circular Panorama, ENSA, Nantes, 2011. Multi-screen, Elektra, Place des Arts, Montréal, 2011. Rectangular panorama, Elektra, La Gaîté Lyrique, Paris, 2011. Video mapping, Les Dominicains de Haute Alsace, Guebwiller, 2012. © Breuleux

The AME approach was forged to make explicit a workflow in A/V performance (as the White Box project, at the figure 5, from 2011 to 2012, show). This article was written to systematize practical knowledge for research purposes in order to demonstrate that cross-platform creative approaches exist in the experiential industries. In addition to the Enigma project, the AME system has been used for multiple artistic creations such as Mapping Me (2018), Les Planètes (2018), Illumination Frankenstein (2018), and Nuages/Clouds (2022). This last project was based on the creation of multiple narrative environments exploring the metaphor of the cloud and driven with a tangible interface. 

Figure 6: The FullDome camera and the realtime view. © Lapierre.

Captured within the Enigma project, figure 6 shows two perspectives of the same system. This image demonstrates how environmental storytelling does not sublimate the idea of the point of view. In this sense, during the continuous design process for these projects, we composed, arranged, and organized the visual narrative using a succession of scene-environments. For example, in figure 7, the camera is moving inside a complex information network. Unlike in a film, in this sequence of different spaces each spectator can choose their own point of view. Therefore, the proposal of the AME concept for SAR creations aspires to highlight the notion that, in the XR paradigm, it is necessary to stop thinking in terms of surface but rather volume. In the production process, abandoning the frame in favor of a virtual environment accommodates the infinity of possible perspectives afforded by various forms of displays. 

Conclusion

Whereas the cross-media,⁴⁹ remediation,⁵⁰ transmedia,⁵¹ spreadable media,⁵² and intermedia⁵³ approaches have been the subject of extensive entertainment literature in the video game ⁵⁴ and film sectors, the definitions of the various terms from communication research and literature only partly reflect the actual practice of A/V performance mentioned in this article.

Figure 7: Inside a FullDome realtime environment. © Breuleux, Lapierre.

The convergence that has emerged from the multimedia web⁵⁵ has prompted a reconfiguration of the roles and processes of publishing and distributing information via multiple platforms. The realities that gave rise to the terms of Cross Media, Transmedia and Intermedia do not account for the realities in A/V performance. For example, when moving from a nine-screen installation to a FullDome experience, some content remediation takes place based on the specificity of the medium. Contrary to the transmedia approach, these two iterations do not aim at generating a global experience or enriching it with new dimensions. Each new immersive experience of the Enigma project is a standalone translation with its own transmission codes. The definition of transmedia also entails the participation of fans and users in the dissemination process. A transmedia media environment, according to Henry Jenkins, is also about how a single individual organizes their consumption of information: "New technologies are enabling average consumers to archive, annotate, appropriate, and recirculate media content."⁵⁶ On the contrary, the AME concept is based on a production and not consumption logic; a logic of creation and not reception. This paper has also addressed the situated conception of narrative environments. To summarize, the immersive A/V performance experience- design research sector requires terms tailored to the realities of design and production.The AME concept was devised to respond to issues of cross-platform SAR artistic design and production for the experiential industries. As the potential for procedural generation, processing, and projection rises with the rapid advancement of real-time rendering technologies in game engines and nodal programming environments, it becomes possible to envision multiple types of AMEs. The concept is in fact a way of organizing information for the creation of immersive experiences.

In conclusion, taking the systemic logic of the proposed AME concept a step further, each environment matrix will be available online via a multi-user WebGL app that will let the user adapt artistic creations to the growing number of technology apparatuses along the XR continuum.

Beyond the technical dimensions, the proposal of the concept of agnostic environment aims to better understand the processes of immersive experiences creations in the digital arts sector. This reflection makes it possible to study, via the transversal concept of environment, the dynamic relationships between the materiality of XR reality technologies and the virtuality of the narrative immersive experiences. The AME approach therefore facilitates the design of multisensory immersive experiences deployed on multiple platforms.

References

1 Kerrison, Margaret Chandra, "The immersive storyteller : writing for real and imagined worlds," Studio City, CA, Michael Wiese Productions, 2022.

2 Gröppel-Wegener, Alke, Jenny Kidd, "Critical encounters with immersive storytelling," In. New York, NY, Routledge, 2019.

3 Festival Elektra, "Le marché international des arts numériques (MIAN)," accessed november 21, 2022, https://www.elektramontreal.ca/pro?lang=fr

4 Festival Mutek, "Mutek Forum," accessed november 21, 2022, https://montreal.mutek.org/fr/forum

5 Société des Arts Technologiques (SAT), (2022),” website, accessed October 27, 2022, https://sat.qc.ca/ 

6 Société des Arts Technologiques, “Rétrospective des innovations de la SAT! (2022),” Metalab website, accessed October 27, 2022, https://sat-mtl.gitlab.io/metalab/timeline/

7 "Quartier des spectacles," website, accessed October 27, 2022, https://www.quartierdesspectacles.com/fr/medias/MAPP-MTL

8  "Atelier des lumières" website, accessed October 27, 2022, https://www.atelier-lumieres.com/

9  "Oasis," website, accessed October 27, 2022, https://oasis.im/

10 Stanney, Kay M., Hannah Nye, Sam Haddad, Kelly S. Hale, Christina K. Padron, Joseph V. Cohn, Extended reality (xr) environment, in handbook of human factors and ergonomics, 2021, 782-815.

11 Daniels Dieter, Sandra Naumann, Jan Thoben, Lentos Kunstmuseum Linz., Ludwig Boltzmann Institut Medien.Kunst.For-schung., and Linz 2009-Kulturhauptstadt Europas Organisations (Firm), "See this sound: audiovisuology compendium: an interdisciplinary survey of audiovisual culture", Köln, Germany, Verlag der Buchhandlung Walter König, 2010.

12 Kenneth J. Peacock, "Instruments to Perform Color-Music: Two Centuries of Technological Experimentation," Leonardo 21, 1988, 397-406.

13  Michael Faulkner, and D-Fuse, VJ: audio-visual art + VJ culture, London, Laurence King, 2006.

14 "Granular Synthesis,” group website, accessed October 27, 2022, https://granularsynthesis.info/MODELL-5

15 Anne-Cécile Worms, "Live A/V, performances audio-visuelles, hors-série," MCD, Musiques et Cultures Digitales, 01/04/2010.

16 Chris Salter, Entangled: technology and the transformation of performance, Cambridge, Mass., MIT Press, 2010.

17 Grayson Cooke, "Performing archival remix in Outback and Beyond," International Journal of Performance Arts and Digital Media 11 (1), 2015, 100-115.

18 Marie-Eve Bilodeau, Ghyslain Gagnon, Yan Breuleux, "NUAGE: A Digital Live Audiovisual Arts Tangible Interface." ISEA2022 - ISEA International, Barcelone, 2022.

19 Steve Gibson, Live visuals: history, theory, practice, Routledge advances in theatre & performance, Abingdon, Oxon, New York, Routledge, 2022.

20 Knight-Hill, Andrew, "Sound and image: aesthetics and practices," In Sound design. New York, Focal Press, still image, 2020.

21 Limeartgroup, " The mega list of VJ Software and Tools (2020)," accessed november 21, 2022, https://limeart-group.com/the-mega-list-of-vj-software-and-tools/

22 Tecartlab"SPARCK," website, accessed November 27, 2022, https://tecartlab.com/features/

23 Limeartgroup, "The mega list of VJ Software and Tools (2020)," accessed november 21, 2022, https://limeart-group.com/the-mega-list-of-vj-software-and-tools/

24 Limeartgroup, " The mega list of VJ Software and Tools (2020)," accessed november 21, 2022, https://limeart-group.com/the-mega-list-of-vj-software-and-tools/

25 Limeartgroup, " The mega list of VJ Software and Tools (2020)," accessed november 21, 2022, https://limeart-group.com/the-mega-list-of-vj-software-and-tools/

26 Chris Salter, Entangled: technology and the transformation of performance, Cambridge, Mass., MIT Press, 2010.

27 Andrea Pinotti, "Towards an-iconology: the image as environment," Screen 61 (4), 2021, 594-603.

28 Henry Jenkins, “Game Design as Narrative Architecture,” Electronic Book Review, July 10, 2004.

29 Janet H. Murray, "Hamlet on the Holodeck." In. Riverside: Free Press, 1997, 61.

30 Marie-Laure Ryan, Foote Kenneth E., Azaryahu Maoz, Narrating space/spatializing narrative: where narrative theory and geography meet, Theory and interpretation of narrative, Columbus: The Ohio State University Press, 2016.

31 Don Carson, “Environmental Storytelling: Creating Immersive 3D Worlds Using Lessons Learned from the Theme Park Industry (2000),” Game Developer (aka Gamasutra), accessed November 2022, https://www.gamedeveloper.com/design/environmental-storytelling-creating-immersive-3d-worlds-using-lessons-learned-from-the-theme-park-industry

32 Scott A. Lukas, "The immersive worlds handbook: designing theme parks and consumer spaces," In. Burlington, Mass.,Focal Press, 2013.

33 Tricia Austin, "Narrative environments and experience design: space as a medium of communication," In Routledge research in design studies, New York, Routledge, 2020.

34 Stephanie Riggs, The End of Storytelling: The Future of Narrative in the Storyplex: Beat Media Group, 2019.

35 Stanney Kay M., Hannah Nye, Sam Haddad, Kelly S. Hale, Christina K. Padron, Joseph V. Cohn, "Extended reality (xr) environments." in handbook of human factors and ergonomics, 2021, 782-815.

36 Paul Milgram, Haruo Takemura, Akira Utsumi, Fumio Kishino, 1994, "Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum." Telemanipulator and Telepresence Technologies, 21 December, 1995.

37 Maria Engberg, Jay David Bolter, "The aesthetics of reality media," Journal of Visual Culture 19 (1), 2020, 81-95.

38 Cindy Ziker, Barbara Truman, Dodds Heather, "Cross Reality (XR): Challenges and Opportunities Across the Spectrum." In Innovative Learning Environments in STEM Higher Education: Opportunities, Challenges, and Looking Forward, edited by Jungwoo Ryoo and Kurt Winkelmann, Cham: Springer International Publishing, 2021, 55-77.

29 Bimber Oliver, Ramesh Raskar, Spatial augmented reality: merging real and virtual worlds, Wellesley, Mass.,A K Peters, 2005.

40 Huhtamo Erkki, Illusions in motion: media archaeology of the moving panorama and related spectacles, Leonardo book series. Cambridge Massachusetts: The MIT Press, 2013.

41 "Team Lab Bordeless", website, accessed November 27, 2022, https://www.teamlab.art/e/borderless_azabudai/

42 Owen Chapman, Kim Sawchuk, "Research-Creation: Intervention, Analysis and “Family Resemblances,” Canadian Journal of Communication 37 (1), 2012, p.5-26.

43 Louis-Claude Paquin, “Faire de la recherche-création par cycles heuristiques (2019),” Louis-Claude Paquin archives, accessed, November 22, 2022, http://lcpaquin.com/

44 O. Vinyals, Le, Q., A neural conversational model, 2015, arXiv preprint arXiv:1506.05869.

45 Purform: Alain Thibault, Yan Breuleux, accessed November 22, 2022, https://vimeo.com/225283585/0455f3796c

46 l'École Nationale Supérieure d'architecture de Nantes (ENSA) "CORAULIS (Centre d’Observation en Réalité Augmentée et Lieu d’Immersion Sonore)," accessed November 22, 2022, https://www.nantes.archi.fr/lensa-nantes/coraulis/

47 Yan Breuleux, “Le dispositif de diffusion comme instrument (2019),” in, Quéinnec, Jean-Paul, Giusy Pisano, Jean-Marc Larrue, and Scholars Portal, Dispositifs sonores : corps, scènes, atmosphères, Espace littéraire, Montréal Quebec, Presses de l'Université de Montréal, 2019.

48 Edward Lantz, "Immersion Domes" In Handbook of Research on the Global Impacts and Roles of Immersive Media, 2020, 314-346.

49 Niklas Odén, "Mechanisms within cross media stories : An analytical framework combining cross media phenomena with storytelling elements", 2013.

50 J. David Bolter, Richard A. Grusin, Remediation: understanding new media, Cambridge, Mass.,MIT Press, 1999.

51  Henry Jenkins, "Transmedia Storytelling and Entertainment: An annotated syllabus," Continuum 24 (6), 2010, 943-958.

52 Henry Jenkins, Sam Ford, Joshua Green, Spreadable media: creating value and meaning in a networked culture, Postmillennial pop, New York, London, New York University Press, 2013.

53  Bruhn, Jørgen, Beate Schirrmacher, Intermedial Studies, 2021.

54 Marie-Laure Ryan, Narrative across media: the languages of storytelling, Frontiers of narrative, Lincoln, University of Nebraska Press, 2004.

55 Henry Jenkins, Convergence culture: where old and new media collide, New York, New York University Press, 2006.

56 Henry Jenkins, Fans, bloggers, and gamers: exploring participatory culture, New York, New York University Press, 2006, 1.

Authors Biographies

Yan Breuleux (M.A.Sc, D.Mus) is an associate professor at École NAD-UQAC UQAC (École des arts numériques, de l'animation et du design), co-researcher within the Hexagram network, the Mimesis laboratory and a practitioner in the field of visual music for immersive devices. He designs A/V performances in the form of sensory experiences.

Alain Thibault has been presented in several contexts, contemporary music and digital art events, in America, Europe and Asia. He is the director of ELEKTRA an annual festival showcasing performances since 1999 and the International Digital Art Biennale, oriented towards exhibitions since 2012.

Rémi Lapierre has a background in art and science through his studies in mathematics, programming and video games. His multidisciplinary profile allows him to understand both the advanced technical issues of software design and the expressive logics of digital creation. Rémi is currently working in the automated generation of virtual environments for immersive devices. 

White Cube / Black Box: Investigating Bias in Museums and Algorithms

Sophia Brueckner, Shannon Yeung, Jing Liu, David Choberka, Kerby Shedden, John Turner, Isabelle Marie Anne Gillet, Mingchen Lu, Xingwen Wei

University of Michigan
Ann Arbor, MI, USA
sbrueckn@umich.edu / shannysw@umich.edu / ljing@umich.edu / dchoberk@umich.edu / kshedden@umich.edu / jmturner@umich.edu / igillet@umich.edu / mingchlu@umich.edu / weixw@umich.edu

Abstract

White Cube / Black Box seeks to identify bias and the many ways bias gets introduced into and amplified within systems. A highly interdisciplinary team of data scientists, curators, designers, and artists used face detection and race classification algorithms to explore bias in algorithms and University of Michigan Museum of Art’s collection of artworks.

Keywords

Machine learning, face detection, race classification, bias, museums.

DOI

10.69564/ISEA2023-45-full-Brueckner-et-al-White-Cube-Black-Box

Introduction

White Cube / Black Box is a collaboration between artists, designers, curators, and data scientists at University of Michigan Museum of Art (UMMA), the Michigan Institute for Data Science, and the University School of Art and Design that attempts to shed light on the opaque decision-making processes within museum collecting practices and machine learning algorithms.

White Cube / Black Box seeks to identify bias and the many ways bias gets introduced into and amplified within systems. In art, the phrase “White Cube” references the history of exclusionary practices within museums and galleries. Using sterile white walls and decontextualized spaces, works of art are divorced from the outside world, making them less approachable and accessible. In technology, the “Black Box” is a controversial metaphor used to describe automated systems where the decision-making process is very difficult or even impossible to understand.

The resulting art installation featured some of the interesting, curious, and troubling findings that our research has uncovered about both facial-recognition technology and about the history of representation in the University of Michigan Museum of Art’s collection of approximately 24,000 works.

We applied one of the most widely used facial detection algorithms to UMMA’s art collection. After detecting faces in UMMA ’ s artworks, we used a race classification algorithm to look at the diversity of subjects in the collection. We used the FairFace Dataset for examples of faces belonging to different races. We used these results to characterize and visualize the racial diversity of the acquisitions made under all of UMMA’s directors.

We used a technique called “eigenfaces” to explore variation within faces found in UMMA’s collection and to understand which features are most important in detecting a face.

By applying facial detection algorithms to UMMA’s art collection, we visualize bias in the museum’s collecting practices throughout its 150-year history. We can also see the ways algorithms amplify human bias. Our research makes more transparent the opaque decision-making processes within museum collection practices and machine learning algorithms as these rapidly evolving technologies are being deployed across the world.

Background

Museum Bias

Art museums have a long history of racial and gender bias. A recent study looking at 18 major US art museums found that 85% of its collected artists are white and 87% are men.¹ Who is depicted in these artworks is not only an issue of numbers, but bias is also evident in how people are depicted. Racialized caricature is one obvious example. Furthermore, museums have historically excluded certain groups of people from visiting museums in both overt and subtler ways.² Museums are now reckoning with how they may have reinforced prejudices in the past and what responsibility they have in confronting prejudice going forward.³,

Algorithmic Bias

Face recognition algorithms are increasingly adopted for commercial use, for public safety, and in other applications. However, flaws in the current algorithms not only limit their effectiveness, but also have adverse consequences for certain demographic groups that are the “usual suspects” of being marginalized or victimized by new technology. The algorithms’ significant flaws in race and gender recognition can be attributed partially to the lack of diversity in the training set—white male being the overrepresented face.⁶ While researchers have repeatedly pointed out such flaws and are improving the training sets, there may be other limitations of the algorithms that have not been adequately addressed. For example, the algorithms rely on faces in the training set that are mostly photographs of full frontal view faces. How well do the algorithms work when the faces are sideways, partially visible, and so on? In our study, we did not aim to develop a new algorithm or improve a current one; instead, we focus on the use of a highly unconventional dataset (UMMA’s art collection) to test the limit of existing algorithms trained on human photographs and understand what features are essential for the correct or incorrect face and race recognition.

Related Works

In 2018, Google Arts & Culture released the Art Selfie phone app as a playful way to discover art. The user takes a selfie and the app searches thousands of artworks to find one with a similar face.⁵ 

The 2020 film Coded Bias summarizes MIT Media Lab researcher Joy Buolamwini’s research on how facial recognition algorithms do not see dark-skinned faces accurately and demonstrates the need for legislation to reduce bias in algorithms.⁶ 

UK Research and Innovation recently funded a project titled Transforming Collections: Reimagining Art, Nation and Heritage led by a team of researchers at the University of the Arts London. The project aims to “build on decolonial feminist approaches and creative machine learning (ML) development: to enable digital cross-search of collections to surface patterns of bias, and to uncover hidden and unexpected connections, and to thus open up new interpretative frames and potential narratives of art, nation and heritage.”⁷

Process

1) We selected YOLOv4 as the main algorithm to test on the art collection. We also used a second algorithm, Dlib, to a more limited extent. The two algorithms both returned some successful face and race detections and some unsuccessful ones. We focus our paper on results with YOLOv4. ¹¹,¹² 

Instead of using artworks (whether within UMMA’s collection or elsewhere) to train the algorithm, we simply used pretrained weights. Our rationale for not using any art collection as the training set is that we do not have the resources to manually inspect and label the artworks that can be used for training, and that the size of such an arts training set could be prohibitively large given the much larger variation in faces in the artworks than in photographs. We did not evaluate the efficacy of a perfectly customized algorithm but rather mimicked the realistic practice of brittle deployments despite limited training data.

2) After identifying faces in UMMA’s collection, we applied the VGG-Face CNN network with pretrained weights from FairFace Dataset to assign race to the faces in UMMA’s art collection. ⁸,,¹⁰

3) We used a technique called “eigenfaces” to explore variation within faces found in UMMA’s collection and to understand which features are most important in detecting a face.¹³

4) We created an exhibition to share results with museum visitors.

Algorithms and Data Sets

Face Detection

For face detection we used the algorithm YOLOv4 and we trained the artificial intelligence (AI) with the Google Open Images Database, which is comprised entirely of photographs.

Race Classification

After detecting faces in UMMA’s artworks, we used a race classification algorithm, the VGG-Face CNN network, to look at the collection’s diversity. We used the FairFace Dataset for examples of faces belonging to different races. FairFace was created to measure and mitigate racial bias. It contains 108,501 Flickr images of faces categorized as Asian, White, Middle Eastern, Indian, Latino-Hispanic, or Black. ⁸ , ⁹

Overview of Results

Of the 21386 UMMA collection objects that we used with the algorithm, 6026 objects (28%) were classified as having at least one face. For race classification, we “forced” the algorithm to choose among the seven racial groups defined in the FairFaces dataset, but combined their definition of East Asian and Southeast Asian into one group, and obtained the following: White (69.1%), Black (10.5%), Indian (2.9%), Asian (10.7%), Middle Eastern (3.9%), and Latino (2.9%). The racial classification algorithm we used lacks a category for Native Americans.

Limitations and Failures

In addition to being unable to classify Native Americans at all, the algorithm had difficulty identifying faces in several cases: faces in profile; tilted heads; highly abstract faces; caricature. On the other hand, non-face objects with round/oval shapes and symmetric features were often classified as faces, such as many vases.

Acquisition Patterns by Different Museum Directors


We sought to understand the acquisition patterns of different UMMA directors, focusing on the predicted races of people depicted in the acquired works. Although we did conduct some benchmarking to establish the performance of the algorithms that we employed, an important caveat of this analysis is that it is based on race classification of detected faces in the artworks by algorithms. It was not humanly possible to validate that all these predictions were correct. With this proviso, we constructed a contingency table showing the number of works depicting individuals of each race acquired by each director.

Figure 1. Racial diversity in UMMA’s Acquisitions by year. The results from the race classification algorithm showed that UMMAs collection became more diverse over time. Each bar breaks down the racial makeup of that year’s acquisitions.

To aid interpretation of this contingency table, we performed a standardization. Let N_ij denote the number of works acquired by director i that depict individuals of race j. Then, if N is the total number of works depicting any race acquired by any director, and p_i denotes the proportion of all works acquired by director i, and q_j denotes the proportion of all works depicting individuals of race j, then N*p_i*q_j is the reference point for N_ij. We can interpret N*p_i*q_j as the number of works acquired by director i depicting individuals of race j in the event that all directors purchased works depicting the races with the same frequencies. The residual R_ij = N_ij - N*p_i*q_j is the excess (if positive) or deficiency (if negative) of works depicting race j acquired by director i. The standardized residual S_ij = R_ij / sqrt(N*p_i*q_j) aims to place these residuals on a common scale that is fairly comparable between directors with small and large numbers of acquisitions, and between races with small and large overall representation in UMMA collection.

Conventionally, values of S_ij smaller in magnitude than 2 are viewed as unimportant. It is not easy to conclude definitively that a given large value of |S_ij| is large enough to be important, but in many cases values exceeding 2.5 or 3 are likely to reflect a specific cause and not occur randomly due to variation of small numbers.

We noticed a significant uptick in the diversity of the collection in 2019. After taking a closer look at the race classification results from that year, we found that the trend seemed to be specifically tied to the Take Your Pick exhibition where museum visitors voted to select 250 everyday photographs to add to the collection, suggesting that a single exhibition can have a notable impact on the overall diversity in the collection. Though imperfect, we found that the algorithms generated results (such as the uptick in diversity in 2019) that offered new perspectives and points of entry for further manual investigation into smaller, manageable subsets of the collection.

Eigenfaces

We used “eigenfaces” to explore variation within faces found in UMMA’s collection and to understand which features are most important in detecting a face.
Eigenfaces represent axes of variability in a collection of images of faces. This technique was first developed in the 1990’s. In prior work, the eigenfaces have been found to capture factors such as lighting, pose, the presence of eyeglasses and beards, and anthropometric features such as dimensions of the jaw, nose, forehead, and spacing between the eyes. Eigenfaces can be used to understand the principal ways that faces in a collection vary, and can also be used as a data compression technique, in that they represent a “high dimensional” face using a relatively low-dimensional vector of “scores.”.

Eigenfaces show the most important ways that individual faces differ from the mean face. Each eigenface corresponds to a spectrum along which variation occurs. Each eigenface below represents one feature important in detecting faces in our collection. 

Figure 2. Each of these eigenfaces represents one feature important in detecting faces in our collection.

The blurriness shows how that feature varies in the faces found in UMMA’s collection. For example, eigenface 3 corresponds to a spectrum along which the face is lit from directions varying from the left to the right. Eigenfaces 1 and 2 correspond to variation in the overall size and shading of the face. Eigenfaces 6, 7, and 8 correspond to different patterns of shading at the top of the head, forehead, and around the eyes.

To begin, we first try to limit the extraneous variation by scaling and cropping each face in our collection to approximately the same position in a fixed-size image (# 224*224*3 # pixels). We then use a mathematical technique called the “singular value decomposition” to identify the eigenfaces. Specifically, an eigenface is a pattern represented by signed (positive and negative) weights. Each eigenface assigns one weight to each pixel location in the images. These weights represent a common pattern of deviation from the mean face. We note that the mean face itself generally appears “ghost-like” and does not resemble a human face, but the deviations from this mean face are informative about the unique characteristics of an individual face.

Since there is one weight for each pixel, the eigenfaces can be visualized in the same way that the face images are themselves visualized. For example, an eigenface corresponding to illumination on the left side may be bright (positive) on the left side of the image and dark (negative) on the right side of the image; an eigenface corresponding to spacing between the eyes may have alternating bright and dark regions of weights in a band located at the level of the subjects’ eyes.

An eigenface represents a spectrum of variation. For example, illumination from the left is part of the same spectrum as illumination from the right, and hence this variation can be represented by one eigenface; similarly, wider-than-average eye spacing may be part of the same spectrum as narrower-than-average eye spacing. Since a spectrum has no defined beginning or end, each eigenface is equivalent to its additive inverse, i.e., F and -F represent the same eigenface, with the spectrum of variation represented by -F being the same as the spectrum of variation represented by F, traversed in the opposite direction.

As noted above, the eigenface technique has often been used with collections of highly standardized images, like passport photos. Even in such a standardized collection, the eigenface technique is generally found to be influenced by lighting and pose as much or more than it is influenced by anthropometry, which is a drawback to the approach. Moreover, while some level of dimension reduction is achieved, it often is necessary to use 100-200 eigenfaces to represent most of the variation in a collection of faces. Using the eigenface technique on UMMA collection is even more susceptible to this issue, since artists represent humans in every possible pose, and it is not possible to standardize these faces beyond simple translation and scaling.

Using the eigenfaces, we identified a painting of a clown, with makeup caricaturing a face, as having the most representative face in UMMA’s collection. 

Figure 3. The mean eigenface and the clown painting that the computer identified as having the most representative face in the museum’s collection.

Exhibition

Using data from our research, we created two video explainers that explain parallels between biases in art museum systems and in algorithm systems. They consisted of data visualizations that highlighted race representation in UMMA collection over time, influence of certain exhibits, and notable research findings and challenges. The videos and select paintings were exhibited in UMMA to self-reflect and critique the museum’s past in full transparency.

We displayed these two videos along with actual paintings from the collection as part of the You Are Here exhibition, which invited museum visitors to consider where they are and where they aren’t. Above our videos, we displayed the question, “Are you here?” inviting the viewers to consider how they are represented within the museum’s collection and exhibitions.

Video Documentation
https://vimeo.com/641632433/942e533cf2  

Figure 4. Video still explaining how human bias plays a role in algorithms.
Figure 5. Video still explaining how human bias plays a role in UMMA.
Figure 6. Racial diversity in acquisitions by museum directors over time

Future Work

We are currently planning a second exhibition scheduled for Fall 2024 that will invite UMMA visitors to observe and evaluate algorithmic facial detection. We plan to visualize the data by exhibiting actual artworks sorted by the algorithm’s confidence in recognizing faces within. Additionally, we plan an interactive wall projection that will contrast the algorithm’s confidence in recognizing faces with confidence judgments submitted live by museum visitors.

Conclusion

An interdisciplinary team of artists, designers, data scientists, and curators applied face detection and race classification algorithms to UMMA’s collection of approximately 24,000 artworks that were collected over 150 years.

When we began this project, we asked, “How can the application of machine learning expose or amplify human bias?” We wanted to know if our AI could reveal the bias in artists, collectors, donors, curators, and society in general over time. What biases did our AI learn when it was trained on datasets of example faces? What biases are embedded in the algorithm itself?

In addition to learning about and visualizing how the diversity of UMMA’s collection changed over time (for the better), we experienced these artworks through the lens of machine learning for the first time. We were not navigating the collection through the usual categories like who created the artwork, artistic movement, artistic medium, date created, or the artwork’s origin. We were not experiencing these works as part of a curated exhibition. We were encountering the artworks in buckets such as “91-100% confident it’s a face” and “non-face” or through simplistic labels like “has face, White” and “has face, Indian”. Going through this process changed our own perception and sensitivity to certain aesthetics as we wondered why the AI made certain decisions. In some cases, the AI’s decisions caused us to question our own understanding of certain artworks.

In addition to exploring biases within both algorithms and museums, this research invites museums and museum goers to reflect on ideas of transparency, self-reflection, and critical thinking about collecting and curatorial practices. How does our understanding of art, curation, and history change when artworks are algorithmically curated?

References

1 Topaz CM, Klingenberg B, Turek D, Heggeseth B, Harris PE, Blackwood JC, Chavoya CO, Nelson S, Murphy KM. Diversity of artists in major U.S. museums. PLoS One. 2019 Mar 20;14(3):e0212852. doi: 10.1371/journal.pone.0212852. PMID: 30893328; PMCID: PMC6426178.

Olivares, A., Piatak, J. Exhibiting Inclusion: An Examination of Race, Ethnicity, and Museum Participation. Voluntas 33, 121–133 (2022). https://doi.org/10.1007/s11266-021-00322-0

3 Sandell, R. (2006). Museums, Prejudice and the Reframing of Difference (1st ed.). Routledge. https://doi.org/10.4324/9780203020036

4 Shirley Li, “American Museums Are Going Through an Identity Crisis,” The Atlantic, November 28, 2020, accessed Dec 1, 2022, https://www.theatlantic.com/culture/archive/2020/11/american-museums-are-going-through-identity-crisis/617221/

5 Google Arts & Culture, “Art Selfie,” accessed Dec 1, 2022, https://artsandculture.google.com/camera/selfie

6 Joy Buolamwini, Coded Bias, accessed Dec 1, 2022, https://www.codedbias.com/about

7 “Transforming Collections: Reimagining Art, Nation and Heritage”, accessed Dec 1, 2022, https://www.arts.ac.uk/ual-decolonising-arts-institute/projects/transforming-collections

8 Karkkainen, K., & Joo, J. (2021). FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1548-1558).

9  “FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age,” accessed Dec 1, 2022, https://github.com/dchen236/FairFace

10 “VGG Face Descripto,” accessed Dec 1, 2022, https://www.robots.ox.ac.uk/~vgg/software/vgg_face/

11 “Dlib,” accessed Dec 1, 2022, http://dlib.net/

12 Bochkovskiy, Alexey, Chien-Yao Wang and Hong-Yuan Mark Liao. “YOLOv4: Optimal Speed and Accuracy of Object Detection.” ArXiv abs/2004.10934 (2020): n. Pag.

13 Matthew Turk, Alex Pentland; Eigenfaces for Recognition. Journal of Cognitive Neuroscience, 1991; 3 (1): 71–86. doi: https://doi.org/10.1162/jocn.1991.3.1.71 

Human Energetics in an Era of Post-Humanism

Christoph Brunner1, Jonas Fritsch2

1Erasmus University Rotterdam, Netherlands
2IT University of Copenhagen, Denmark
brunner@esphil.eur.nl, frit@itu.dk

Abstract

French philosopher of technology Gilbert Simondon is undoubtedly one of the key figures when it comes to conceptualizing individuation across physical, mental and social strata. In this article, we develop a rather overlooked aspect of Simondon’s work, namely how his ontogenetic project also implies an idea of a “human science” based on a “human energetics,” which—maybe in spite of its name—is an inherently transhumanist project transducing across both disciplinary and experiential fields, with a particular emphasis on the role of technology. We present key concepts in Simondon’s work and relate them to lines of thinking on energies in the arts (Kahn) and post-colonialism (Wynter), exemplified through an analysis of Nigerian artist Otobong Nkanga’s video work “Remains of the Green Hill.” Our primary aim with the article is to continue a mobilization of Simondonian concepts and thinking for an experimental, transhumanist exploration in relation to its ethico-aesthetic and artistic potential.

Keywords

Energy humanities, human energetics, individuation, socio-genesis, posthumanism, human science, postcolonialism, affect.

DOI

10.69564/ISEA2023-46-full-Brunner-et-al-Human-Energetics

Introduction

In the wake of an increasingly differentiated debate on the status of the human in the era of posthumanism and the Anthropocene, major critiques of a Eurocentric humanism at the heart of the arts and humanities have emerged.¹,² Instead of turning towards the agency of non-humans or celebrating the vibrancy of matter a more-than-human account tuned to the status of the human requires a radical rethinking of what a non-Eurocentric conception of the human and humanities might look like.³,,⁵ Such a recasting of the humanities matters in two crucial ways: on the one hand the humanities are part and parcel of an ideological as much as economic-colonial project which builds on both an ontology of the human as autonomous and conscious being and the extractive order seizing lands and resources on a global scale.⁶,⁷ On the other hand, such critiques of human exceptionalism afford us to rethink the human as embedded in material, organic, social and mental domains that do not only intersect but radically transform the idea of an autonomous human subject. A humanities building on such a post-humanist conception of the human has been a major concern of French philosopher of technology Gilbert Simondon, whose work in the 1950s and 60s has not only formulated a relational conception of change and becoming but was also deeply concerned with a transformation of the humanities towards what he terms a “human Science” (Science humaine) involving the formation of a proper “human energetics.” In the first instance, Simondon’s persistent attention to the role of technical being might seem like a plea to know technologies better, to become an engineer that understands and not only uses technologies. Further one could conceive of his elaboration on energy and energetics, which he primarily derives from physics, as an attempt to consolidate natural sciences and the humanities.⁸ However, Simon-don’s interest is a different one. By asking about the axiomatics of the natural sciences, cross-pollinating different scientific disciplines, he outlines a new program for the humanities: with the notion of a human energetics, he shifts the focus from subdivisions of thinking the human along disciplinary differences to the possibilities of a shared sensibility for the emergence, continuation, and potential dissolving of constellations which take shape across material, organic, and mental domains. He particularly draws on the notion of “potential energy” to do so.

The resurgent interest in the concept of energy and energetics in areas such as the energy humanities, but also artistic and postcolonial contexts demonstrates an increasingly transdisciplinary body of research that provides plenty of resonances with Simondon’s proposition of a human energetics.⁹,¹⁰,¹¹,¹² These respective fields and approaches towards an energetic outline of art, media, and culture conceive of energy—or energies—as relational properties immanent to varying processes of material and conceptual transformation in creative practices.

While Simondon aims at a broad reworking of the humanities closely linked to the sciences, his emphasis on the human should not be mistaken. In his transductive account of potential energy, we perceive a strong refusal of any anthropomorphic conception of the human as different from technologies or its organic and inorganic environment. In this sense, the human energetics in a Simondonian key, are trans-humanist, where the human becomes a composite of different energetic relays. More than that, Simondon’s reworking of potential energy allows us to embark on a critique of the humanist tradition and its Eurocentric and enlightened concept of the human. We therefore draw on the work of Jamaican philosopher and writer Sylvia Wynter whose critique of the human as "Man" implies a feminist and post-colonial problematization of enlightenment conceptions of the human, both in philosophy and the sciences. Instead of rejecting the concept of the human, however, she proposes a reworked notion of the term that resonates strongly with several contemporary post-and transhumanist positions.

The critical reworking of the human through Simondon, Wynter, and new energetic tendencies in the arts allows us to conceive of contemporary artistic practices as “fields of experience.”¹³,¹⁴,¹⁵ Addressing such fields as energetic textures, opens up a new perspective on the tendentially less concrete category of relations. Without wanting to stage energies against relations, our intention is to show how the energetic traverses material, technological, embodied, aesthetic and perceptual domains through affective, energetic activation. We will therefore draw on Nigerian artist Otobong Nkanga’s video work “Remains of the Green Hill” in order to provide a first analytic account of how to work with this energetic approach in the context of electronic arts and media.

A Simondonian Proposition of a Human Energetics

In any case, we would arrive at the idea according to which a human science must be founded on a human energetics and not just on a morphology; a morphology is quite important, but an energetics is necessary; one would have to ask why societies transform, why groups change in accordance with the conditions of metastability. However, we certainly see that what is most important in the life of social groups is not merely the fact that they are stable, but that at certain moments they cannot conserve their structure: they become incompatible with respect to themselves, they dedifferentiate and become supersaturated; just as the infant can no longer remain in a state of adaptation, these groups disadapt.¹⁶

Simondon most thoroughly develops his notion of energy and human energetics in his magnus opus Individuation in the Light of Notions of Form and Information collected and published in 2005, but containing work from 1964 and onwards. Here, Simondon conceptualizes the notion of energy—and in particular “potential energy”—as a form-taking and change-making processes of individuation across organic, physiological, social, technical and psychological domains. This allows him to conceive of individuation as tied to a system (and its state of stability). For Simondon, the system is not a closed one. On the contrary, it is the necessary ground from which the energetic activity initiates a potential process of individuation. Whereas this use of energy and energetics derived from physics might be seen as an attempt to consolidate natural sciences and the humanities,¹⁷ Simondon has a different aim: through a human energetics, the focus shifts from subdivisions that conceive of the human along disciplinary divides towards the possibilities of a shared sensibility for the emergence, continuation, and potential dissolving of compositions (or forms). To Simondon, these compositions take shape across both material, organic, and mental domains through operations of transduction.¹⁸

The notes from the lecture “Form, Information, Potentials” most explicitly develop the notion of human energetics as a non-equivalence-based model of thinking relations between form and information. Either notion is deeply transformed from a more conventional conception of form as given and impressed onto matter. By proposing a new theory of form, Simondon interlaces an archetypical and a hylomorphic conception of form to be found in Plato and Aristotle respectively. For Simondon, the archetype is a structural germ of a form—a field—and the matter-form couple in the Aristotelian scheme is a domain. The field is a virtual cueing of potential energies to “inform” the energies immanent to a domain which is “the ensemble of reality that can receive a structuration.”¹⁹ Far from being a linear process, the coupling of the germ and the structuration provides a vital processof differentiation and dedifferentiation.

Put differently, the ideal form is not a given but an informational field encountering a domain whose abilities to resonate shape the actual form-taking. Along his informational account, Simondon uses the concept of energy and energetics without rendering it into a metaphysical category of a metaphor for explaining a connective dimension that cannot be grasped in any substantial manner. On the contrary, energy is taken in its physical sense, where potential energy is energy in a system being stored and later released. It is a material potentiation that in its abstract state is information. With the introduction of the concept of information, Simondon casts potential energy as a type of energy that is not merely quantifiable in magnitude but crucial for processes of transforming an energetic state, like that of a mineral, into another, such as a metal or gas. To understand such qualitative changes, a mere thermodynamic equivalence model of energy is insufficient since it does not account for the effectuation of change itself. With his insistence on information, Simondon ties the problem of change to the question of what drives change and how change operates through a fielding of potential energies. Put differently, potential energy is a prerequisite for the process of form-taking as a transductive operation. In his earlier work On the Mode of Existence of Technical Objects, Simondon accordingly writes that the issue at stake concerns thinking not from the end of an action (its result) which he calls finality but rather to focus on causality and the process of causation.²⁰ Simondon insists that morphology, the taking or changing of form, requires the addition of energy to understand general processes of transformation. To further understand “why societies change” for instance, we must turn towards the process of individuation. In his outlined energetic scheme, a society’s transformation should be understood by a constant tension between differentiation and dedifferentiation, of a non-linear leaping of constant reworkings of how form takes shape under differentiating conditions and along specific lines of processing. For the human sciences (i.e., humanities) this principle or paradigm provides an axiomatic approach that foregrounds the non-foundational process of energy-oriented information. As a process of transduction, information binds the different modes of existence into an energetics that allows us to radically challenge a classic humanist understanding of action. It is neither the human as an already constituted individual that acts, nor is an individual the mere subject of external forces. It is the interplay between information, matter and form taking, cutting across different registers of existence, which constitute a human that acts by being energetically engaged.

By further introducing the physical notion of the field, Simondon finds the conceptual handle to move beyond a separation between the abstract and concrete, the empirical and metaphysical, or a binary logic of interiority and exteriority. Combined, the notions of field and energy allow Simondon to recast processes of individuation and ontogenesis across disciplinary boundaries, leading him to propose a rethinking of the “human sciences” towards “human Science” with a common axiomatics applicable to various domains, respecting their multiple forms of application. Such a general energetics might be conceptualized as a transversalizing move beyond disciplinary divides in a common understanding and exploration of new ways of being and, especially, becoming—with a particular emphasis on the constitutive role of technics and technical objects for energetic, material, and existential conditions.

Simondon’s by now sixty-year-old problematization of the humanities through energy and energetics bears another problem tied to contemporary debates in the emergent domain of research of energy humanities but also the extractivist idea of human energy as the exploitative enterprise of slavery and the deterioration of natural resources. Simondon’s critique of such exploitative modes is radical, in the sense that he considers neither economic nor energetic exploitation as sufficient for the problematization of human suffering. His take on a Marxist critique of political economy emphasizes that the workers’ loss of the means of production under capitalism and their cooperative reclaiming rests on a much deeper alienation that between human and technical being.²¹ Again, while on a more general level, Simondon’s critique could be read as a debunking of the foundational critique of capitalism in Marx. However, his critique targets equivalence and more precisely the deployment of the general equivalent in Marx as much as the general equivalent of energy in thermodynamics. By problematizing a conception of modernist progress building on models of equivalence, such as money or monetary value and energy, Simondon becomes a highly interesting interlocutor for thinking the far-from-equilibrium critiques of change based on human-centered action as a dominant narrative in the genealogy of Western humanities.

Energy Humanities and Energy in the Arts

In the following, we wish to position Simondon’s human energetics in relation to a rapidly growing engagement with energy in several emerging research areas within the arts and humanities over the last decades. In doing so, we also wish to hint at how a Simondonian proposition of energetics contributes to this current—important and necessary—work carried out; namely the basic idea that energy is core to both human and non-human existence and hence is not something “out there” but deeply entrenched within an expanded understanding of the arts and humanities.

In their article from 2014 “The Rise of Energy Humanities,” authors Dominic Boyer and Imre Szeman forcefully state the need for a humanistic take on current energy dilemmas:

“Energy humanities” is an emerging field of scholarship that overcomes boundaries between disciplines and between academic and applied research. Like its predecessors, energy humanities highlights the essential contribution that the insights and methods of the human sciences can make to areas of study and analysis that were once thought best left to the natural sciences.”²²

A central starting point for the energy humanities is that “today’s energy and environmental dilemmas are fundamentally problems of ethics, habits, imagination, values, institutions, belief, and power—all traditional areas of expertise of the humanities and humanistic social sciences” ²³. This points to the necessity of revitalizing both humanist and artistic perspectives in the joint pursuit of rethinking energy across disciplinary divisions. This differs slightly from the approach taken by Simondon. Rather than arguing for what human sciences can bring to natural sciences, a Simondon- inspired human energetics emphasizes the inherent entanglements across said research fields. We can only arrive at radically repositioning disciplinary divisions by acknowledging this condition. Here, human energetics might be seen as adding to a heterogenous, diverse understanding of different kinds of energies across several domains.

It is beyond the scope of this paper to provide a comprehensive review of all the ways in which energy has come to matter in humanist and artistic thinking and practice. For an attempt at this quasi-impossible task, with an orientation towards the artistic explorations, we point at Douglas Kahn’s comprehensive introduction to the anthology Energies in the Arts from 2019. Here, Kahn presents a pluralistic approach moving from “energy” to “energies in the arts.” He develops an “indefinition of energies,” and instead shows multiple, situated examples of energies in, around and across artistic practices. According to Kahn:

“Energies are embedded and embodied in all phenomena, in the operations of the senses and cognition, and, thus, how we might feel, perceive, and think about them, individually and collectively. They also drive engines, manufacturing, telecommunications, and media; and underscore large-scale periods such as the age of steam and the Industrial Revolution, the atomic age and ongoing oil wars.”²⁴

Fields of energy and fields for understanding energy continuously commingle, which calls for a specificity in addressing energetic configurations, and not least how these configurations are “(...) bound to be upset by the capacious, appropriative and synthesizing embrace of the arts and culture.”²⁵ A focal point for Kahn, then, is to acknowledge and explore the “aesthetics and poetics” of different artistic discourses and expressions of energy.²⁶ In the text, Kahn surveys and addresses understandings of energy from a broad range of disciplinary fields – physics, psychology, philosophy, (environmental) humanities – and, naturally, from through an array of artistic expressions. Among other examples, Kahn reports on a fundamental energetic transformation in the arts when music went from metabolics and acoustics to electrical and electronic music:

“Transduction back and forth between two classes of energy – acoustics and electromagnetism – not only fundamentally changes the character of the production, modulation, amplification, transmission, and storage of music, it also set up new relations of control (and lack thereof) for the performer and composer.” ²⁷ 

Whereas Kahn does not make any explicit references to the work of Simondon, we see both a range of shared concepts (fields, transduction) and interests. Kahn makes it explicit that his task is not to develop overarching theories or philosophies, but tracing down specific instances of how energies are explored, activated or transduced in practice. Whereas it might be argued that the departure of this article is indeed related to the theoretical or philosophical articulations presented in Simondon’s work, it is important to emphasize that it is equally important to trace particular energetic projects and trajectories—as well as to inspire future both conceptual, designerly or artistic activations of this conceptual foundation in relation to both analysis and practice. However, we also wish to revisit Simondon’s project in the light of contemporary discussion around the post-human across arts and humanities, to resituate and discuss the extent of his thinking, which will be the main focus in the following section.

The Human in the Era of Posthumanism

The heightened interest in a differentiated concept of energy and energetics in the arts allows us to trace some of the more recent encounters with post-and decolonial critiques of Western conceptions of the human in aesthetic discourses. We will particularly engage positions which relate to Black Studies and Black Aesthetics. The post-human in the arts ranges from early debates on Internet art, to feminist celebrations of the cyborg (see for instance the most recent iteration of the Venice Biennale in 2022), eco-art or various problematizations of the Anthropocene. Most of these critiques of humanism tend to overcome the human as such.²⁸ Black Aesthetics and Black Studies, on the other hand, have raised the question of the category of the human as a trait of racist and colonial Eurocentric enlightenment.²⁹,³⁰ ,³¹,³²,³³ These perspectives draw on Frantz Fanon’s notion of the “sociogenic.”³⁴ Sociogeny, rather than a biological notion of phylogeny or the philosophical concept of the ontogenetic, defines the constant reproduction and differentiation of social forms. Through the lens of sociogeny,

Fanon casts his main concern, racism, a cultural phenomenon tied to capitalist and colonial extractive logics of dehumanization.

Dehumanization, exclusion, and Othering have been part and parcel of black resistant reflections in literature and the arts. One of the most prominent expositions of aiming at another or different notion of Black humanity can be found in Afrofuturism as a literary and artistic movement.³⁵,³⁶ These artistic explorations engage with the future as a post-humanist line of flight for marginalized minorities who are systematically excluded from mainstream discourses in society and culture. From a more feminist perspective, Haraway draws on the works of Hortense Spillers and Gloria Anzaldúa, among others, to cast a black feminist mode of resisting the intersectional historical dehumanization of women of color under the trope of a white, patriarchal, and Eurocentric order of who is worth being considered human. Haraway engages with abolitionist and women's rights activist Sojourner Truth’s famous utterance “Ain’t I a woman” to point out the impenetrability of white supremacy as a black woman and fully human being. Sojourner Truth’s exclamation “ain’t I a woman” at a 1851 women’s convention in Ohio underlines the dehumanizing experience of not being accounted as fully a person as the (white) others. Being partial and not being whole and thus fully human—as the term might be used to point at the integrity of a being (or subject) or a community (such as a species) that marks the experience of the black woman—provides the potential of a different humanity which Haraway terms the post-human.³⁷

The human in the post-human is not, as it might often appear, a beyondness which runs the danger of leaving the foundational Western concept of the human intact, but another kind of human that was never included in the enlightened project of the humanist tradition. The effects of a dehumanizing and exclusive conception of the human define one of the key problematizations of Sylvia Wynther’s work. For Wynter, the human has mostly been a conception of Man [sic!] either in a phylogenetic or biogenetic and ontogenetic vein. We can see how such a problematization allows us to engage an immanent critique of Simondon’s emphasis on ontogenesis as the guiding principle of his philosophy of individuation. Up to a certain point, Wynter and Simondon share a similar project; reconsidering the human not as a category to be overcome but as a shared principle for a different human Science. For Wynter the devalorizing of blackness as a biological, ontological, and social category (all three composing the sociogenic) leads to an “over-valorization of whiteness” which she casts as an “overall devalorization of the human species that is indispensable to the encoding of our present hegemonic Western-bourgeois biocentric descriptive statement of the human.”³⁸ And she further counters such a casting:

“In other words, because the negative connotations placed upon the black population group are a function of the devalorization of the human, the system revalorization of Black peoples can only be fundamentally effected by means of the no less systemic revalorization of human being itself, outside the necessarily devalorizing terms of the biocentric descriptive statement of Man, overrepresented as if it were by that of the human.”³⁹

The revalorization of Black peoples as a necessary and all-encompassing reworking of the human provides a black radical critique of the reductive, hegemonic, and racist conception of the human in the Western narrative of the human sciences. We consider Wynter’s problematization crucial to further extend Simondon’s human energetics. In order to do so, we need not only to reconsider the information-oriented process of individuation that proceeds through potential energy, but to account for the informational germs that engage with matter to take form from a non-Eurocentric or white assumption as we find them in the notion of good form or good sense, from Plato to Descartes and Kant. Engaging with Black Aesthetics, Wynter proposes another form-taking which undermines the white humanist assumption of form and the sociogenically ingrained mechanism of representation and recognition. Describing the human as “mere mechanism” of dominant, bourgeois, hegemonic and racist order, Wynter dismantles white humanism as a structural function for extraction and marginalization – of rendering blackness inhuman. Frantz Fanon, to whom Wynter repeatedly refers, writes:

“What are by common consent called human sciences have their own drama. Should one postulate a type for human reality and describe its psychic modalities only through deviations from it, or should one not rather strive unremittingly for a concrete and ever new understanding of man?”⁴⁰

Mobilizing Fanon, Wynter asks about potential energies for a resurgence of the human in a non-hegemonic and non-mechanistic and thus not racializing key. The difference of such energetics resides in Fanon’s and Wynter’s reclaiming of the human, engaging the relation between fields of potential energy and their resonance with form-taking processes with and through matter. Matter is not the non-or post-human to be celebrated in an often depoliticizing gesture, but an energetic relay that provides the ground for becoming and accounts for the human as a composite mode of existence.

Mattering: Post-humanist Energetics in the Arts

From a Simondonian proposition of a human energetics, to-wards a broadened notion of energy in the arts towards a black aesthetics of energies against a dehumanizing humanist order, the question of the more-than-human obliges us to turn to matter. Matter as a concept gained a rather romanticized outlook bearing the possibility of depoliticization. While the vibrancy of matter provides some resonance with Simondon’s refusal of the hylomorphic scheme as insufficient for an understanding of form-taking, it might also lead to an overt celebration of matter and an agential realism that runs the danger of leaving the human behind.⁴¹,⁴² New Materialism’s engagement with matter proposes a more nuanced differentiation, especially when it enters into an honest dialogue with historical materialism. At the same time, a thoroughgoing inclusion of a black studies focus on the sociogenic remains a future task.⁴³ Katryn Yusoff’s critical analysis of the science of geology as a discipline that considers matter as inhuman and turns slaves into energy providing bodies thus becoming inhumane objects, adds a new problematizing dimension to our discussion of human energetics.⁴⁴ Turning the enslaved body into an energy resource, drawing a line from the plantation to machine discipline, as Nicholas Fiori proposes, binds the enslaved body to a thermodynamic energy regime ⁴⁵. Such a regime, as much as the capitalist extractive one, is insufficient to understand the overall scope and relevance of energy. The same accounts for the energetic regime of geology, as the discipline that casts matter or the earth into an extractable resource.

Simondon’s insistence on potential energy enables us to build a bridge to Wynter’s and Fanon’s undoing of a White humanist conception of the human. It shifts the de-humanizing and energy as resource-based logic around matter and energy into one that moves through blackness and black embodiment. Yusoff’s work points at the extractivist logic immanent to geology and the energy necessary to extract specific matters from the earth. The energy of the objectified slave body is bound to matter as the object of extractive activities, creating a tightly knit energetic feedback loop.

For us the question posed by Simondon, Wynter and Fanon remains; how to rethink the human beyond whiteness and Eurocentric humanism. A first proposition works through “affective engagement” and “affective relaying” with and through the sensuous.⁴⁶,⁴⁷ Affect as the realm where the emergent quality of information becomes felt before it can be rationalized appears as the ideal zone to begin a human Science that pays attention to both potential energy and the sociogenic problematization of the human. Such a reworking of the human occurs when we look at Otobong Nkanga’s short video artwork “Remains of the Green Hill.” The video shows Nkanga in the foreground holding different yoga poses with rocks on her head. The artist stands in front of the former Tsumeb copper mine in Namibia. The video’s audio is an interview with the former managing director of the by now abandoned mine. The images are calm without other humans. Through the warm light of sunrise, the texture of the scene is one of rebeginning in the ruins of prior capitalist (in-and dehumanizing) extraction. The manager talks about turning the abandoned pit into a monument to begin a new narrative that reworks the scars of the former violent activity. Heather Davis writes about Nakanga’s posture: “the gestures could be read as an invocation of healing, of connection, deliberately breathing in the remains of the mineral air.”⁴⁸ Healing in the remains of violent energy turnover, healing towards different human and more-than-human encounters. We might conceive of this as first steps towards a different kind of expanded—planetary? cosmological?—humanity. However, it is also necessary to point out the rock on the artist’s head, a burden and historic geological weightedness and a companion or ally at the same time.

A range of existential questions have (re-)surfaced within the humanities following a decade-long period of technoscientific advancement and environmental crisis. From the technological side of things, machine learning/AI, cyborgs, (both voluntary and involuntary) robots and the like are jointly blurring the boundaries between what has normally been conceived as the “human” and “non-human.” Similarly, the challenges we are facing on an environmental and planetary scale have also propelled a reorientation of the human as the center of the world, albeit maybe still responsible for catalyzing the forces that have effectuated this displacement in close collaboration with a range of technical inventions. All of this has had a profound impact on our very conditions of life and living in the world. Somehow, we seem to be perpetually living in a “(...) far-from-equilibrium situation where each of the systems we depend on for stability

References

1 Rosi Braidotti, The Posthuman, Cambridge, UK, Malden, MA, USA, Polity Press, 2013.

2 Donna Jeanne Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Experimental Futures: Technological Lives, Scientific Arts, Anthropological Voices, Durham, Duke University Press, 2016.

3 Bruno Latour, We Have Never Been Modern, Cambridge, Mass, Harvard University Press, 1993.

4 Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press, 2010.

5 Erin Manning, Always More than One: Individuation’s Dance, Durham, Duke University Press, 2013.

6 Sylvia Wynter, “On How We Mistook the Map for the Territory, and Reimprisoned Ourselves in Our Unbearable Wrongness of Being, of Desêtre: Black Studies Torwards the Human Project,” in A Companion to African-American Studies, ed. Lewis R. Gordon and Jane Anna Gordon, London, Blackwell Publishing, 2006, 107–18.

7 Kathryn Yusoff, A Billion Black Anthropocenes or None, Forerunners: Ideas First from the University of Minnesota Press 53, Minneapolis, University of Minnesota Press, 2018.

8 Gilbert Simondon, Individuation in Light of Notions of Form and Information, Volume 1, Minneapolis, Minnesota Press, 2020, 676.

9 Imre Szeman, Dominic Boyer, eds., Energy Humanities: An Anthology, Baltimore, Johns Hopkins University Press, 2017.

10 Douglas Kahn, ed., Energies in the Arts, Cambridge, Massachusetts, The MIT Press, 2019.

11 Nicholas Fiori, “Plantation Energy: From Slave Labor to Machine Discipline,” American Quarterly 72, no. 3, 2020, 559–79, https://doi.org/10.1353/aq.2020.0035.

12 David McDermott Hughes, Energy without Conscience: Oil, Climate Change, and Complicity, Durham, Duke University Press, 2017.

13  Christoph Brunner, Jonas Fritsch, “Interactive Media as Fields of Transduction,” The Fibreculture Journal, no. 18, October 2011, 118–45.

14 Christoph Brunner, Jonas Fritsch, “Beyond the Network: Experiential Fields and Urban Media Ecologies,” Proceedings of the Media Cities 2013 Conference in Buffalo, 2013, 277–87.

15 Christoph Brunner, Jonas Fritsch, “Shifting Immediations: Fields of Experience across Media Art and Design,” in Proceedings of the 26th International Symposium of Electronic Arts - ISEA 2020, ed. ISEA, Canada, Printemps Numerique Canada, 2020, 91–97.

16 Simondon, Individuation, 269.

17 Simondon, Individuation, 696.

18 Simondon, Individuation.

19 Simondon, Individuation, 697.

20 Simondon, Individuation, 134.

21 Simondon, Individuation, 133.

22 Sezman Boyer, Energy Humanities, 40.

23 https://www.press.jhu.edu/books/title/11492/energy-humanities.

24 Khan, Energies in the Arts, 1.

25 Khan, Energies in the Arts, 2.

26 Khan, Energies in the Arts, 33.

27 Khan, Energies in the Arts, 12.

28 [i.e. https://archiv3.fridericianum.org/exhibitions/inhuman].

29 Denise Ferreira da Silva, “Toward a Black Feminist Poethics: The Quest(Ion) of Blackness Toward the End of the World,” The Black Scholar 44, no. 2, 2014, p.81–97.

30 Saidiya Hartman, “Venus in Two Acts,” Small Axe 12, no. 2, June 2008, 1–14.

31 Fred Moten, Stolen Life, Consent Not to Be a Single Being, v. 2, Durham, Duke University Press, 2018.

32 Hortense J. Spillers, “Mama’s Baby, Papa’s Maybe: An American Grammar Book,” Diacritics 17, no. 2, 1987, 64, https://doi.org/10.2307/464747.

33 David Lloyd, Under Representation: The Racial Regime of Aesthetics, First edition, New York, Fordham University Press, 2019.

34  Frantz Fanon, Black Skin, White Masks, 1st Evergreen ed, New York, Grove Press, 1982.

35 Kodwo Eshun, More Brilliant than the Sun: Adventures in Sonic Fiction, London, Quartet Books, 1998.

36 Mark Derey, “BlacktotheFuture:InterviewswithSameul R. Delany, Gret Tate, and Tricia Rose,” in Flame Wars: The Discourse of Cyberculture, ed. Mark Derey, Durham, Duke University Press, 1994, 179–222.

37 Donna Haraway, “Ecce Homo, Ain’t (Ar’n’t) I a Woman, and Inappropriate/d Others: The Human in a Post-Humanist Landscape,” in Feminists Theorize the Political, ed. Judith Butler, Joan Wallach Scott, Routledge, 1992, 86–100.

38 Wynter, Territory, 116.

39 Wynter, Territory, 116.

40 Fannon in Wynter, Territory, 117.

41 Bennett, Vibrant Matter.

42 Karen Michelle Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, Duke University Press, 2007.

43  Hanna Meißner, “New Material Feminisms and Historical Materialism: A Diffractive Reading of Two (Ostensibly) Unrelated Perspectives”, In Mattering, edited by Victoria Pitts-Taylor, New York University Press, 2020, 43–57, https://doi.org/10.18574/nyu/9781479833498.003.0003.

44 Yusoff, A Billion Black Anthropocenes.

45 Fiori, Plantation Energy.

46 Christoph Brunner, “Relaying and Rebeginning”, Transmutations 1, no. 1, March 2013, http://transmutations.org/site/relaying-and-re-beginning/.

47 Jonas Fritsch, “Understanding Affective Engagement as a Resource in Interaction Design”, Nordes Conference Proceedings, Oslo, 2009.

48  Davis Heather, “Blue, Bling: On Extractivism,” Afterall: A Journal of Art, Context and Enquiry 48, September 2019, 12– 19, https://doi.org/10.1086/706123.

49  Brian Massumi, Erin Manning, Jonas Fritsch, Bodil Marie Staving Thomsen, “Affective Attunement in the Field Of Catastrophe”, In Politics of Affect, London: Polity Press, n.d, 112–45.

Bibliography

Karen Michelle Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, Duke University Press, 2007.

Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press, 2010.

Rosi Braidotti, The Posthuman, Cambridge, UK, Malden, MA, USA, Polity Press, 2013.

Christoph Brunner, “Relaying and Rebeginning”, Transmutations 1, no. 1, March 2013, http://transmutations.org/site/relaying-and-rebeginning/.

Christoph Brunner, Jonas Fritsch, ““Interactive Environments as Fields of Transduction”, The Fibreculture Journal, no. 18, 2011, http://eighteen.fibreculturejournal.org/2011/10/09/fcj-124-interative-environments-as-fields-of-transduction/.

"Beyond the Network: Experiential Fields and Urban Media Ecologies”, Proceedings of the Media Cities 2013 Conference in Buffalo, 2013, 277–87.

“Shifting Immediations: Fields of Experience across Media Art and Design”, Proceedings of the ISEA 2020 Symposium in Montreal, 2020.

Howard Caygill, “Life and Energy.” Theory, Culture & Society 24, no. 6, November 2007, 19–27, https://doi.org/10.1177/0263276407078710.

Davis Heather, “Blue, Bling: On Extractivism”, Afterall: A Journal of Art, Context and Enquiry 48, September 2019, 12–19, https://doi.org/10.1086/706123.

Mark Derey, “Black to the Future: Interviews with Sameul Delany, Gret Tate, and Tricia Rose”, In Flame Wars: The Discourse of Cyberculture, edited by Mark Derey, Durham, Duke University Press, 1994, 179–222.

Kodwo Eshun, More Brilliant than the Sun: Adventures in Sonic Fiction, London, Quartet Books, 1998.

Frantz Fanon, Black Skin, White Masks, 1st Evergreen ed. New York, Grove Press, 1982.

Nicholas Fiori, “Plantation Energy: From Slave Labor to Machine Discipline”, American Quarterly 72, no. 3, 2020, 559–79, https://doi.org/10.1353/aq.2020.0035.

Jonas Fritsch, “Understanding Affective Engagement as a Resource in Interaction Design”, Nordes Conference Proceedings, Oslo, 2009.

Donna Haraway, “Ecce Homo, Ain’t (Ar’n’t) I a Woman, and Inappropriate/d Others: The Human in a Post-Humanist Landscape”, In Feminists Theorize the Political, edited by Judith Butler, Joan Wallach Scott, Routledge, 1992, 86–100.

Donna Jeanne Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Experimental Futures: Technological Lives, Scientific Arts, Anthropological Voices, Durham, Duke University Press, 2016.

Saidiya Hartman, “Venus in Two Acts”, Small Axe 12, no. 2, June 2008, 1–14.

David McDermott Hughes, Energy without Conscience: Oil, Climate Change, and Complicity, Durham, Duke University Press, 2017.

Douglas Kahn, ed. Energies in the Arts, Cambridge, Massachusetts, The MIT Press, 2019.

Bruno Latour, We Have Never Been Modern, Cambridge, Mass, Harvard University Press, 1993.

David Lloyd, Under Representation: The Racial Regime of Aesthetics, First edition, New York, Fordham University Press, 2019.

Brian Massumi, Erin Manning, Jonas Fritsch, Bodil Marie Staving Thomsen, “Affective Attunement in the Field Of Catastrophe.” In Politics of Affect, London, Polity Press, n.d., 112–45.

Erin Manning, Always More than One: Individuation’s Dance, Durham, Duke University Press, 2013.

Hanna Meißner, “2. New Material Feminisms and Historical Materialism: A Diffractive Reading of Two (Ostensibly) Unrelated Perspectives”, In Mattering, edited by Victoria Pitts-Taylor, New York University Press, 2020, 43– 57, https://doi.org/10.18574/nyu/9781479833498.003.0003.

Fred Moten, Stolen Life. Consent Not to Be a Single Being, v 2, Durham, Duke University Press, 2018.

Silva Denise Ferreira da, “Toward a Black Feminist Poethics: The Quest(Ion) of Blackness Toward the End of the World”, The Black Scholar 44, no. 2, 2014, 81–97.

Gilbert Simondon, Individuation in Light of Notions of Form and Information, Volume 1, Posthumanities 57, Minneapolis, Minnesota Press, 2020.

Hortense J. Spillers, “Mama’s Baby, Papa’s Maybe: An American Grammar Book”, Diacritics 17, no. 2, 1987, 64, https://doi.org/10.2307/464747.

Imre Szeman, Dominic Boyer, eds., Energy Humanities: An Anthology, Baltimore, Johns Hopkins University Press, 2017.

Sylvia Wynter, “On How We Mistook the Map for the Territory, and Reimprisoned Ourselves in Our Unbearable Wrongness of Being, of Desêtre: Black Studies Towards the Human Project”, In A Companion to African-American Studies, edited by Lewis R. Gordon,, Jane Anna Gordon, London, Blackwell Publishing, 2006, 107–18.

Authors’ Biographies

Christoph Brunner, PhD, is Assistant Professor for Philosophy of Media and Technology at Erasmus University Rotterdam. He is founder of the ArchipelagoLab for Transversal Practices at Leuphana University Lüneburg. His research concerns the relations between media and their aesthetics, affective politics, and social movements in networked cultures. He takes a particular interest in practices and positions from Global South and translocal forms of organizing. Most of his work arises from collaborations with critical practitioners.

Jonas Fritsch, PhD, is Associate Professor in Interaction Design at the IT University of Copenhagen in the Department of Digital Design. He is head of the Affective Interaction & Relations (AIR) Lab and Head of the Design Research Section. His work revolves around a creative thinking of interaction design, design processes, experience philosophy and affect theory through practical design experiments with interactive sound and physical interfaces. 

How Digital Anthropomorphism Enhances Creativity in Human-to-Robot Dance Interactivity

Sorina-Silvia Cîrcu, and Chu-Yin Chen, * †

AIAC Laboratory, Université Paris 8, CNRS-University of Montpellier LIRMM, National Tsing Hua University
Montpellier, Paris, Taiwan
sorina.silvia@gmail.com ; chu-yin.chen@univ-paris8.fr

Abstract

Through our research-creation experiments, we imagine movement sequences that challenge the concept of anthropomorphism in digital twin robots. While considering the term of interactivity to better define their interaction, we observe how different movement material is stimulating creativity through a hybridization process between human and machine. From this perspective, we determine how users experience qualia while learning through imitation a dance sequence consecutively demonstrated by a humanoid robot, an industrial arm and a human. Their feedback and our own practical experimentation allow us to better understand the impact of digital anthropomorphism in the making of a sustainable human-to-robot interaction.

Keywords

Digital Anthropomorphism, Dance, Interactivity, Social Robots, Hybridization, Symbiotic Individuation.

DOI

10.69564/ISEA2023-47-full-Circu-et-al-Human-to-Robot-Dance

Introduction

Since the 6th century BCE, when the term was firstly employed to describe religious phenomena ⁹, anthropomorphism has accompanied humanity’s intention to replicate its characteristics into different environments. Anthropomorphism (from the Greek word ánthrōpos meaning “human” and morphē meaning “form”) is described in ¹⁰ as “the tendency to attribute human characteristics to inanimate objects, animals and others with a view to helping us rationalize their actions. It is attributing cognitive or emotional states to something based on observation in order to rationalize an entity’s behavior in a given social environment.” As discussed in ³¹, an essential component of the human spirit is to lend certain characteristics of our psychic life, by projecting our physical and psychological functioning into objects. According to ¹ “humans attribute, often unknowingly, personality traits to machines based not only on their external appearance, but also on their functioning and skills.” In the present paper, we refer to anthropomorphism in a broader humanistic perspective, as a characteristic of an autonomous system’s behavior that allows ascribing human-like characteristics and intentions to non-human entities like robots. Recently, the literature has integrated an interpretation proposed by French philosopher Bruno Latour where the meaning of anthropomorphism is defined by “that which has human shape” and “that which gives shape to humans”³⁰ encouraging scholars to envision digital anthropomorphism as the concept that integrates both views. Using these grounds, we explore key notions like agency ¹⁵ and autonomy ² for our particular case of dancing with robots, to better outline the link of these notions with digital anthropomorphism in establishing a sustainable human-to-robot interaction (HRI). Lately robotics has enhanced these perspectives, developing artifacts that challenge the idea of humanity.²⁶ We rely on scholars ⁵, ⁴ to define the place robots can occupy in our study, considering them as tools (helping human to accomplish a task—in our case develop a choreography), as avatars (since the robot engages in a certain social presence with other people—in our case the spectators of a dance performance) and especially as partners (establishing a co-working process with a collaborator—in our case co-creating a dance performance). From ¹⁰  we note that a social robot “can be perceived as the interface between man and technology. It is the use of socially acceptable functionality in a robotic system that helps break down the barrier between the digital information space and people.” As robot design becomes modular ²⁷ and body extensions inspire art performances that question human capacities,¹⁶ an anthropological study ³² compares HRI to the type of connection expressed in earlier religious rituals between gods and humans, pointing out the influence robots could have on us in the near future. As we are currently shifting into a post-humanist technologized era, where humans are extending their capacities using exoskeletons and various connected devices, the definition of the human body and how it interacts with its environment changes accordingly. Our paper investigates how these paradigms affect our creativity and their impact in collaborative social practices like dance.³³ mentions the etymological analogy between dance (from Indo-European ten, root for tension) and emotions (from latin émovere: or set to motion). In our quest for a meaningful interactivity between performers and robots, we analyze the impact of these anthropological projections on dance. In the following pages, we describe how we create our movement sequence by discussing our working hypotheses and methodology and explaining our working phases leading to the concept of human-to-robot (H2R) hybridization. We then adapt and test the sequence on several human performers. We later discuss the results and perspectives of this experimentation and its implications into current dance practices.

Questions and Methodology

Our approach focuses on the process of creating an original choreographic language inspired by robots, influenced by somatic practices and embodied intelligence. According to artist Louis-Philippe Demers, any abstract inert shape “can become fluid, organic and eventually anthropomorphic by the sole means of contextualization and movement.”⁹ Our goal is to check to what degree the concept of anthropomorphism enhances creativity in HRI and influences artistic research. Among our working hypotheses, we investigate how the shape of a robot can influence the performer in generating unintentional movements similar to kinaesthetic responses ³ (1) ? How can a dance sequence be reproduced more easily depending on the type of robot? Does the feedback of the performer change once the robot type has changed? Another area of investigation concerns the creative process, supporting the idea that working with an avatar substitute of a robot that we define later as its digital twin, can favor a hybridization state between the body or the performer and the robot’s virtual body. In this way, we underline the process through which the role of the robot shifts from tool, to working companion. Further on, we wonder if this new symbiotic individuation between robot and human can influence the output of artistic performances. The robots involved in our study have similar dimensions, although initially built for different outcomes. As a starting point, we use the movement material implemented in the HRP-4 robot for the Le mythe d’Immorta (2022) performance, part of the Co-Évolution, Co-Création et Improvisation Homme-Machine (CECCI-H2M) project. The humanoid robot and an industrial arm are programmed in an analogous series of movements based mostly on the rotation of upper limbs and head for the humanoid robot. We then test the dance sequences through a case study within a group of dance students.

Digital twin robots as interactivity facilitators

The students are asked to memorize, then improvise the analogous movement sequence indicated through video projection by different dance partners. The filmed sequences have the same format and duration (approx. 1min). First sequence is taught by a humanoid robot, second one by an industrial arm and the last one by a human performer. The sequences have the same structure but differ in the quality of movement, depending on the teacher. Next the dancers are asked to freely interpret the movements and collectively improvise a free dance sequence, recycling the movements they considered the most inspiring. The method of analysis is a question form of 23 questions that the participants are asked to fill in at the end of the experiment.

Figure 1: Human and HRP-4 digital twin teaching movement material.

Experimentation Phases When the tool becomes a partner

In the West, robots are often designed to mimic human behavior, mostly as tools ¹⁴ replacing human labor or assisting humans in complex tasks. However, in creative contexts their function is more ontological in the sense they might contribute to a certain creation of meaning in inference with the art process that contains them. For artist and researcher Simon Penny, cognition does not occur exclusively in the brain, being “far from a logical manipulation of symbolic tokens.”²³, ²⁵ Arguing mental processes like inspiration are embodied, integrated within non-neural bodily tissues (thus extended into artifacts, social systems and cultural networks), he also points out its dynamic nature. Through our research-creation experiments, we imagine movement sequences that challenge the concept of anthropomorphism in intelligent interactive robots. As the stage mediates their encounter, it brings new possibilities of expression, and consequently of interaction. In our particular case, we apply the term of interactivity ⁶, ⁷ to define the relation between humans and robots. Initially inspired by human interaction, interactivity appeared with the development of new technologies and media, setting up the premises for an ongoing process of transformation between man and machines. Mixing living and nonliving, reality and simulations, physical and virtual, interactivity facilitates a permeability between human and technology. As stated in ¹ , contemporary interactive interfaces “escape the total control of man and create a new situation where the latter no longer has an exclusively active role in front of his tool. It is rather an exchange, an interdependence characterized more and more by their interactivity.”

Confronting paradigms concerning embodiment

¹ makes an analogy between the way technology, especially virtual interactions, shapes the understanding of the human body. Individuals seeking to expand, alter or fragment their body, are indicators of the ongoing shifts operating through our daily interactions. The identity being multiplied, diluted or absorbed into the digital world, the tendency is to project the same expectations onto the human body. Neuroscience, robotics or art do not consider embodiment and anthropomorphism in the same way. In ¹⁸, the body is defined as a set of organic processes that go beyond the experience of sensation and movement. It might include social networks such as families and culturally constructed artifacts. In parallel, anthropological terms like “body domestication” ¹⁷ analyze how the body is being segmented in units according to the field of research that address it. In arts, pioneers like Stelarc have always considered their body as a playground for technological experiments ²⁹. With his third arm or spider legs performances, the artist blurred the lines between what is human and what is machine through very original settings. Whereas roboticists design robots that can “emulate” humans through their resemblance and collaborative skills ¹¹. Researchers demonstrate how the overall shape of a robot plays a key role in invoking desired emotions in users ¹⁴, with studies measuring the degree of acceptance of humanoid robots ²², ¹⁹, influenced by the awareness of our own “body schema.”¹³ Additionally, concepts like “social uncanniness” ¹² examine how our need to be unique and to engage in authentic interactions is being impacted by the development of social robots.

The idea to start experimenting with a robot’s virtual avatar, appeared while working in the laboratory with the roboticists of the Interactive Digital Humans (I.D.H.) team in Montpellier. The HRP-4 robot being little available at that time, we had an artistic residency in which its virtual double was representing the real one. The robot executed a series of movements inspired by power postures of political leaders sitting on chairs. While improvising, its imaginary corporeality triggered different kinetic responses in the performer, so we were curious to look further and understand this phenomena. In ²⁸ we note the distinction between virtualized robots and the simulations that are necessary for the functioning of robots in the real world. To go further, in our current study we employ the term of “digital twins (2)” over simulations, due to their real-time data processing and possibility to study multiple processes in various simulations. 

Figure 2: Graph illustrating the link between hybridization and technaesthetic experience in dance HRI.

Hybridization as a result of creative interactivity

¹⁷ traces back the hybridization process between the human body and the technological objects from as early as prehistory, when the anatomy of our hand adapted to the manipulation of objects like carving tools. He argues that the notion of reflexivity occupies a central place in the way anthropomorphism operates, citing the cultural anthropologist Victor Turner for whom the human is more than a performative animal, being something close to an animal that performs oneself. In analogy, creating a robotic dance sequence is also a reflexive process. The human plays a double role-movement initiator and receiver (in the sense she/he responds to the proposal done by the robot) whereas the robot becomes a medium-translating with its mechanical constraints the body architecture it is supposed to embed, without much alternating the initial model. To deepen our understanding of the concept of hybridization, we define it in relation to the work of numeric artist Edmond Couchot. For him the hybridization is always located in the experience, “more precisely in the relationship between the sensitive, the body and its environment.”²⁸ The author of the paper points out the work-in-progress nature of hybridization or, to cite the artist, “an evolving state, from human to artificial and from artificial to human.” In his classification of artificial agents, Couchot makes a distinction between robots and artificial humans, using the concepts of materiality and autonomy to distinguish between the two. For autonomy his classification is orientated around two polarities: the puppet avatar (minimal level of autonomy) and the autonomous actor (maximal level of autonomy). Concerning its definition of materiality, he structures his observations around the technaesthetic experience from french “expérience technesthésique” as being “a sensitive experience lived in the technical act [which] constitutes a kind of perceptual habitus, sensory knowledge, shared by each member of a society and shaping his ways of being and acting, of thinking, by ways different from those of language and symbolic thought.” It is interesting to note this experience is implying the dissipation of the self, relying on the remains of other people’s experience with technology. This relates to the observation of Becker ¹⁷ for whom the history of shared gestures, traces back the evolution of humanity in its relation with technology.

Figure 3: Image morphing between the performer and the digital twin of the HRP-4 robot.

The Human in the loop

One of the challenges of our research-creation is to explore this intermediary dimension that operates between the quality of movement of two distinct entities (human and robot). In ³³ an experiment done with a Poppy robot, favors the emergence of the concept of kinaesthetic empathy-illustrated by the gap and tension that operates once the gesture is being transmitted from the human to the robot. In our particular case, in real-time improvisations, once the HRP-4 robot is programmed to reproduce a dance sequence, the choreography might be altered according to the technical characteristics of the robot-state of actuators and battery, code glitch, joint limitations. We call this type of inference “parasite movements,” requiring a sensory-motor adaptation both from the machine and from the human. In comparison, while working with digital twins, the adjustments are done at the video-projection level. The concept of morphing ³³ borrowed from image processing offers an interesting analogy to movement production, defining a process of continuous rehabilitation and readjustment of the bodies and their transformation into shapes. For one of our research-creation experimentation trials, we projected the figure of the robot on the body of the performer. To our surprise the image overlapped, resulting in a hybrid figure of both robot and machine. After several trials, the performer adapted to this unexpected robot specificity, like in a taming process and a symbiotic figure, half-human, half-robot resulted.

Case study of a human-to-robot dance interaction involving anthropomorphism

Through our case study we investigate how different movement material is experienced by performers while dancing with robots. The focus of our paper is to determine how users experience ”qualia” (3) while learning through imitation a dance sequence consecutively demonstrated by human and non-human agents. Arguing that body and mind are the same expression of an organic process, ²⁰ describes arts and aesthetics as the culmination of humanity’s attempts to find meaning. After working with the double twin of the HRP-4 robot for the Le mythe d’Immorta performance, we wanted to understand how robots are being perceived by their human partners during creative practices. Engineers can easily pre-program intentions related to specific human emotions or behaviors in robots using an affective loop approach ⁸. This approach focuses on the robot’s capacity to engage humans into affective exchanges and therefore attribute meaning to their behavior. In 2010, a simplified user interface that creates body motions ²¹ was implemented in an HRP-4C robot for a dance performance. The robot does basic synchronized dance movements surrounded by four dancers, giving the impression of a perfect copy of the human performer. As stated by ²⁴, we expect from intelligent, self-conscious artworks to surprise us. To tackle this expectation, we imagine interactions that focus on spontaneous, ”parasite” movements and unexpected behaviors in robots, questioning an “effect of presence” ³⁴ on stage. To explore the potentialities of these creative evolutionary interactions, different scenarios were programmed and tested, involving imitation and random movements. This approach enables us to understand how artificial and organic bodies may adapt to one another and reach a hybrid state, specific to our working environment.

While conceiving our first dance sequence with HRP-4, we had to take into account different constraints: how the robot stabilizes its center of mass, its autonomy while standing and the safety mechanism that allows it to move. This context made us initially consider a seated dance interaction. Inspired by work of famous choreographers like Ohad Naharin’s Echad mi Yodea (1998), Anne Teresa De Keers-maeker’s Rosas danst Rosas (1983) or Anna Halprin’s Seniors Rocking (2005) who used chairs in their choreography, we chose to work on the postures of famous political leaders. The fact that they were sitting, thinking, but through their reasoning they influenced the outcome of our daily lives transformed the figure of the robot into a totem-like figure inhabited by power gestures. The dance sequence was transposed into the HRP-4 digital twin, on stage during the CECCI-H2M project. In a different setting of this project, the performer interacted in real time with both autonomous systems of the performance-using the same dance sequence with the performance’s virtual environment E.V.E.(non-anthropomorphic) as well as the virtual HRP-4 (anthropomorphic). The out-come of this experience was that the repetitive movements of the HRP-4 generated a feeling of oppression and limitation in the performer, triggering the need to deconstruct them in various movements similar to the kinaesthetic responses we mentioned earlier. Whereas the responsiveness of E.V.E. made her easily forget the initial sequence and follow the rhythmic proposed by the artificial agents. Both systems were identified as stage partners by the performer, with the robot evoking a feeling of absence or “ghost in the shell” (4) phenomena. Once having these insights developed through our creative process, we felt the need to confront them to a broader context. The next part of your experimentation was a case study proposed to 25 dance students who tested a dance sequence conceived for the humanoid HRP-4 robot and taught by it, as well as a FRANKA industrial arm and a human performer. Compared to the version inspired by power postures, the HRP-4 robot was moving its upper limbs, hips, head and torso, but this time standing. New movements, corresponding to unintentional movements similar to kinaesthetic responses were added in the loop to simulate what we defined earlier as parasite movements. Participants were initially asked to reproduce the movements then improvise freely applying some of the gestures they remembered. They were encouraged to explore their senses (including the proprioception) and forget any personal projection regarding aesthetics and how the sequence should look like.

Figure 4: Stage interaction with different types of embodiment.

Discussion

In ¹⁵, ¹⁰  scholars link the notion of agency to the question of anthropomorphism, citing D.C. Dennett’s theory on “intentional systems”. In art contexts, besides the object’s kinetic behavior, another factor of influence is the spectator’s strategy to understand and predict the behavior of the performing object. We wanted therefore to open up our research-creation process to external participants, organizing a practical experimentation with dance students. Among the questions the 25 participants in our study were asked to respond to, some concerned the movement imitation learning process. Most of the participants found no significant differences in the way the robot (56%) presented the movement, compared to the human (76%). However, most of the participants strongly agreed it was easier to follow the movements of the humanoid robot (64%), compared to the industrial arm (8%). Concerning the quality of movement, most of the participants believed both the humanoid robot (44%) and the robotic arm (44%) had intentional movements. Whereas 56% agreed it was easy for them to detect the “parasite movements” of the sequence. A slight difference was observed when participants were asked if they could distinguish intentional from unintentional movements, with 28% of the participants agreeing and 24% strongly agreeing it was easy. A significant part of the participants was undecided (36%) whether robots are strange creatures, with 24% strongly agreeing they are and 20% somehow agreeing they are not. Similar distribution considering whether robots have consciousness, with 28% somehow disagreeing and 24% somehow agreeing. As for the creative interactivity, the majority of the participants- (68%) felt the need to add other movements, once the sequence became repetitive. None of them agreed that they usually apply the imitated movements when dancing, while very few some-how agreed the movements were natural for the HRP-4 robot (8%) and Franka robot (16%). It is interesting to note that for the Franka robot, 4% of the participants strongly agreed its movements were natural, while none for the humanoid robot. Regarding emotions, only 12% of the participants somehow agreed the HRP-4 robot communicated emotions through dance with a majority (64%) disagreeing and 24% being undecided. As for the Franka robot’s emotions, 80% of participants disagreed it expressed them through dance while 8% were undecided and 4% somehow agreed. Interestingly enough 8% of participants believed the Franka robot communicated emotions through its dance. Most of the participants of our study (64%) could spend more time understanding movement through robotic interaction. A hybridization process was also noticed after the sequence, when the participants were asked to freely interpret the robot motions they experienced earlier. This facilitated the exploration of a state where senses were more present and spontaneous movements appeared more easily. A state that we define as creative, in the sense that it allows body expressiveness inherent to the specificity of the embodiment - where by imitation they appropriated and transformed the robotic dance sequence. While moving, their bodies seemed inhabited by the robotic presence. Experimentation output is available before (5) and after (6) the robot trial.

Limitations

Regarding our intention to slightly modify the sequences, getting to see them in a particular order (HRP-4, followed by the Franka robot and the human performer) might have influenced their answers. Some of the participants were already familiar with the movement of the HRP-4 robot due to previous working sessions organized earlier that year. Compared to working with digital twins, the results might also be different when working in real time with the physical robots. 

Figure 5: Learning phases, from imitation to improvisation.

Moreover, by applying the notion of digital anthropology to the digital twins of the robots used in our experiments, we might have omitted some interesting results depending on other shapes and dimensions. We should also not forget that even though artificial agents can simulate intentions and affects, our way of interpreting them—or the notion of qualia we mentioned earlier—is always different. Having implemented the analogous dance sequence of the humanoid robot on the industrial arm, we experienced differences (especially in relation to posture symmetry but also speed and jerk) with the industrial arm proving more compliant given the type of actuators that executed the movement.

Future perspectives using emerging robotic technologies

The next phase of our research will be testing the same scenarios with the physical version of the robots. In this way we can determine whether the perception of the performer, as well as his/her quality of movement can be impacted differently when working in the same conditions, with the real, mechanical body of the robots. This will also allow us to understand how the hybridization process is influenced by movement imitation phases. Our first empirical results encourage us to establish a scientific laboratory study, measuring kinaesthetic empathy through the detection of mirror neurons, using electroencephalogram (EEG) signal processing.

Conclusion

Learning by imitation, then establishing a creative interaction using digital twins allowed us to research on the qualia each participant experienced and improve the overall quality of movement through dance improvisation. This type of creative interactivity involved the hybridization between humans and robots, generating new visual shapes when video-projected, as well as an original type of movement material. Hopefully our remarks on digital anthropomorphism will further stimulate exchanges between roboticists and artists, anticipating a new individuation phase in the overall robot-human relation.


*S-S. Cîrcu is with AIAC Laboratory of Paris 8 University and with the CNRS-University of Montpellier LIRMM, UMR5506, Montpellier, France. Email: sorina.silvia@gmail.com

†C-Y. Chen is with National Tsing Hua University, Taiwan and AIAC Laboratory of Paris 8 University, France

(1) a spontaneous reaction to motion which occurs outside you; the timing in which you respond to the external events of movement or sound; the impulsive movement that occurs from a stimulation of the senses: ie. someone claps in front of your eyes and you blink in response; or someone slams a door and you impulsively stand up from your chair., Bogart, Anne, Landau, Tina. The Viewpoints Book, p. 11

(2) acc to IBM, while a simulation typically studies one particular process, a digital twin can itself run any number of useful simulations in order to study multiple processes.

(3) acc to Stanford Encyclopedia of Philosophy, “philosophers often use the term ‘qualia’ (singular ‘quale’) to refer to the introspectively accessible, phenomenal aspects of our mental lives.The status of qualia is hotly debated in philosophy largely because it is central to a proper understanding of the nature of consciousness. Qualia are at the very heart of the mind-body problem.“ 

(4) https://www.imdb.com/title/tt0113568/

(5) https://vimeo.com/779347404

(6) https://vimeo.com/779363288

References

1 R. B. Baddoura-Gaugler, L’homme et le robot humano ̈ıde: Transmission, Re ́sistance et Subjectivation, Ph.D. Dissertation, Universite ́ Paul Vale ́ry-Montpellier III, 2013.

2  D. Bisig, Generative dance-a taxonomy and survey, In Proceedings of the 8th International Conference on Movement and Computing, 2022, 1–10.

A. Bogart, T. Landau, The viewpoints book: A practical guide to viewpoints and composition, Theatre Communications Group, 2004.

4 C. Breazeal, and R. Brooks, Robot emotion: A functional perspective, Who needs emotions, 2005271–310.

5  C. Breazeal, J. Gray, G. Hoffman, and M. Berlin, Social robots: Beyond tools to partners, In ROMAN 2004, 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No. 04TH8759), 2004, 551–556, IEEE.

6 M. Bret, M.-H. Tramus, A. Berthoz, Interacting with an intelligent dancing figure: artistic experiments at the crossroads between art and cognitive science, Leonardo 38(1), 2005, 46–53.

7 M. Bret, M.-H. Tramus, A. Berthoz, Creation no. 2, la funambule virtuelle, michel bret, marie-he ́le`ne tramus, alain berthoz, Hybrid, Revue des arts et mediations humaines (2), 2015.

8 L. Damiano, P. G. Dumouchel, Emotions in relation, epistemological and ethical scaffolding for mixed human-robot social ecologies, HUMANA, MENTE Journal of Philosophical Studies 13(37), 2020, 181–206.

9 L.-P. Demers, Machine performers: Agents in a multiple ontological state, 2015.

10 B. R. Duffy, Anthropomorphism and the social robot, Robotics and autonomous systems 42(3-4), 2003, 177– 190.

11 P. Evrard, A. Kheddar, Homotopy switching model for dyad haptic interaction in physical collaborative tasks, In World Haptics 2009-Third Joint Euro-Haptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2009, 45–50, IEEE.

12 G. Hoffman, The social uncanniness of robotic companions, In Robophilosophy, 2020, 535–539.

13 M. Hoffmann, Body models in humans, animals, and robots: mechanisms and plasticity, Body Schema and Body Image: New Directions, 2021, 152.

14  J. Hwang, T. Park, W. Hwang, The effects of overall robot shape on the emotions invoked in users and the perceived personalities of robot, Applied ergonomics 44(3), 2013, 459–471.

15 E. A. Jochum, and L. Putnam, Computation as medium: agency and motion in interactive art, Akademisk Kvarter 16, 2017, 9–21.

16 E. Jochum, L.-P. Demers, B. Vorn, E. Vlachos, P. McIlvenny, P. Raudaskoski, "Becoming cyborg: Interdisciplinary approaches for exoskeleton research", In Proceedings of EVA Copenhagen 2018: Politics of the machines-Art and after, Electronic Workshops in Computing, 2018, 1–9, BCS Learning and Development Ltd.

17 B. Joffrey, Humanoïdes, expérimentations croisées entre arts et sciences, 2015.

18  M. Johnson, T. Hohrer, We are live creatures: Embodiment, american pragmatism, and the cognitive organism, Body, language, and mind 1, 2007, 17–54.

19  F. Kaplan, Who is afraid of the humanoid? investigating cultural differences in the acceptance of robots, International journal of humanoid robotics 1 (03), 2004, 465–480.

20 J. Mark, The meaning of the body: Aesthetics of human understanding, 2007.

21  S. Nakaoka, S. Kajita, K. Yokoi, Intuitive and flexible user interface for creating whole body motions of biped humanoid robots, In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2010, 1675–1682, IEEE.

22 T. Nomura, K. Sugimoto, D. S. Syrdal, K. Dauten-hahn, Social acceptance of humanoid robots in japan: A survey for development of the frankenstein syndrome questionnaire, In 2012 12th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2012, 242–247, IEEE.

23 S. Penny, Emergence, agency, and interaction—notes from the field, Artificial Life 21(3) 2015, 271–284.

24 S. Penny, Improvisation and interaction, canons and rules, emergence and play, The Oxford Handbook of Critical Improvisation Studies 2, 2016, 401–23.

25 S. Penny, Sensorimotor debilities in digital cultures, AI and SOCIETY 37(1), 2022, 355–366.

26 B. Romic, Negotiating Anthropomorphism In The Aida robot, International Journal of Social Robotics, 2021, 1–11.

27  T. Siedel, M. Hild, M. Weidner, Concept and design of the modular actuator system for the humanoid robot myon, In International Conference on Intelligent Robotics and Applications, 2011, 388–396, Springer.

28 R. Sohier, "Degré d’autonomie et degré de matérialité d’après l’œuvre d’edmond couchot: d’une matrice des humains artificiels a`une recherche-création expérimentant la co-avatarisation", Hybrid, Revue des arts et médiations humaines (9), 2022.

29  E. Stephens, T. Heffernan, We have always been robots: The history of robots and art, In Robots and Art, Springer, 2016, 29–45.

30 A. Stojnić, Digital anthropomorphism: Performers avatars and chatbots, Performance Research 20(2), 2015, 70–77.

31  S. Tisseron, "De l’animal numérique au robot de compagnie: quel avenir pour l’intersubjectivité?", Revue française de psychanalyse 75(1), 2011, 149–159.

32 D. Vidal, "Anthropomorphism or subanthropomorphism? an anthropological approach to gods and robots", Journal of the Royal Anthropological Institute 13(4), 2007, 917–933.

33  M.-A. Villard, M. Lapeyre, "A propos d’une expérience de mouvement partage avec un robot humanoïde: l’entre-deux comme maintien du vivant", In IRIS, number 37, 2016, 193–205.

34 P. Zaven, "Effets de présence, relations hommes-androïdes", sur Cultures Kairos, 2014.

Authors’ Biographies


Sorina-Silvia Cîrcu holds a BSc in Systems engineering from The Polytechnic University of Timisoara, a MSc in Philosophy from Vest University of Timisoara and a MA in Theatre Directing from Nanterre University in Paris. Transdisciplinary artist and researcher, she is currently preparing a Ph.D between dance and robotics (AIAC laboratory of Paris 8 University and LIRMM laboratory of Montpellier University). Her work focuses on dramaturgy of the body, new media and notions like reality, perception and metamorphosis. She worked among others with Henier Goebbels for his anthology of “Sound and Spaces”, Silvia Costa, Michel Cerda, Theatre du Soleil, Odin Teatret.

Chu-Yin Chen is an Artist and Professor in Digital Art at Paris 8 University. Since 2019, Chair Professor at National Tsing Hua University (Taiwan). Her creations, based on Artificial Life and complex systems develop interaction modes between audience and virtual creatures showing autonomous and evolving behaviors. Her digital artworks have been shown in numerous international exhibitions. Her research articulates on two overlapping areas: 1] Digital Creation using algorithms of complexity and emergence, and 2] Metacognition and Elicitation of the processes of creation, enaction and aesthetic reception, via psycho-phenomenology and mindfulness.

The Revolution Will Launch in the Garden: Politics of representation and vegetal intellig(senti)ence

Paul Rosero Contreras

Universidad San Francisco de Quito - Dlab, Dos Islas Estudio Quito, Ecuador
proseroc@usfq.edu.ec

Abstract

This paper describes entanglements between human and non-human actants from the perspective of creativity, interdependence and contemporary anthropology. Drawing from experiments on self-representation and swaps from object to subject of study, the author reflects on the potential of the vegetal world and distinct forms of intelligence to propose a subversive anti-anthropocentric view in a planet terraformed by plants and other beings.

Keywords

Non-human creativity, reciprocity, vegetal imaginary, reciprocal gaze, more-than-human representation.

DOI

10.69564/ISEA2023-48-full-Rosero-Contreras-Launch-in-the-Garden

A living species is a creative species, shaping its own environment to make it habitable and functional. The first living organisms completely altered the planet in the Great Oxidation Event. Plants have transformed Earth in such a way that made the planet livable and breathable for other living beings. In this revolution, plants not only create images, but they themselves are images. Biology allows us to understand creation as the result of a more- than- human cognitive process; a neurobiology of plants that regards them as sentient beings in an interdependent relationship with their environment, capable of learning and responding, constructing an imaginary of possibilities.

Drawn to this idea of plants as creative beings, I have been developing a project El Pensamiento de las Plantas [The Thinking of Plants] that speculatively imagines the visual representation of plants from the perspective of one’s self. This photographic series is influenced by Indigenous Epistemologies, biological concepts, and contemporary anthropological postulations like those of Canadian anthropologist Natasha Myers. Her idea of Planthropocene activates a radical political potential by asserting that humans are not alone on this planet, and that Anthropos are not the only ones capable of constructing habitable worlds. The Planthropos brings us to the Planthropocene: part plant and part human, it embodies the involuntary interdependence of plants and animals that is committed to collective prosperity. Unlike the Anthropocene, the Planthropocene is not a geological epoch, but rather a “scene” or episteme: a way of understanding life.¹ 

Figure 1. Paul Rosero Contreras. El Pensamiento de las Plantas. Volar #2. Fotografía, 2019.

This perceptual process involves “planting” our own senses to develop an appreciation for the expressive and curious manners of vegetal life. In the 1970s experimental art scene of the U.S. post-war avant-garde, artists and scientists explored this idea in what would later be called bio-sensing art. This exploration combined the nascent fields of electrobiology, cybernetics, and the ecologies of information systems. Artists, like the North American Richard Lowenberg, meticulously investigated the intellectual and multisensory properties that connect humans to other life forms.² From orcas to tropical plants, these studies encompassed ideas around interspecies collaboration, nonhuman agency, and the advent of artificial intelligence. However, despite the fact that these plant explorations were sowed in open transdisciplinary collaboration—and have broadened our understanding of the touch, smells, colors, textures, and shapes of the world we inhabit  —the ontology in which this practice was fundamentally rooted still  measured the cognitive and sentient capabilities of plants in molecular and chemical terms; in other words, it was reliant on a rationally descriptive perspective. On other occasions, the experiments were transformed into performances that sought out hallucinogenic experiences through interspecies communication in real time.⁴

My own project acknowledges this legacy, but I try to broaden the worldview by incorporating fundamental concepts from indigenous epistemologies that show us how our own aesthetics and sensorial practices are intertwined with vegetal life, to the extent that thoughts, feelings, and the production of an imaginary all function symbiotically. This type of practice in becoming vegetal necessitates a “plant intelligence,” entering what Natasha Myers terms “vegetal sentience”: a broader (mind)space activated, not only by new forms of thinking about our relationships to plants, but also by their perception of the world and how they make sense of it. Thus, I do not limit myself to assumptions about intelligence and its associated indicators, nor to an idea of multispecies interconnectedness that is mediated by human technology.

In truth, this so-called “vegetal sentience” is a concept that has been recognized by indigenous communities around the world for millennia. For instance, from the Andes to North America, it is observed that the relationship of indigenous peoples to the natural environment has always been understood beyond dichotomies: it is a dynamic flow and constant interdependence where the act of giving thanks and requesting permission from the earth form a chain of manifested reciprocity. In this vein, Robin Wall Kimmerer—ecologist and enrolled member of the Citizen Potawatomi Nation—speaks of mosses as an act of life in and of themselves, bringing services to the species with whom they interact, with the moss-tree relationship being one of the oldest and ripest in history. Given this context, I believe that it is necessary to foster a movement away from the notion of thinking and towards one of feeling.

What is thinking?

If we define intelligence as the capacity to solve problems, then plants and animals alike share this ability. Hence, in El Pensamiento de las Plantas, thinking is understood as a collective act in creating interspecies images that explore the limits of language. This project is a visual reponse to a moment in the past, or to an exterior stimulus beyond the human subject that experiences and captures an image. Rooting this process in a collective act with the forest is a way of addressing the problem of politicized representation of nonhuman beings, and moving towards a strategy for self-representation. The blurry images constitute intuitive thoughts, or perhaps ephemeral translations, that are closer to the feeling of liberation rather than a logical, neoliberal interpretation.

Historically, the natural world—understood as landscape in Western art history—has been regarded as an object, a representation mediated by retinal perception. In the twentieth century, we witnessed the advent of certain avant-garde movements that took the natural world beyond its representational state, but the canon remained unbroken. Hitherto, nature as object had been filtered through the human body—almost always male and white—and made an image from the perspective of Anthropos. It seems that posthumanism, the humanities, and the arts still face the challenge of shifting Eurocentric attention towards other forms of Planthropocenes. Nevertheless, there are cases of indigenous artists, rural dwellers, and activists who work in social justice, both within and outside of official institutions, who recover ancestral lands in concrete ways.

In El Pensamiento de las Plantas, I try to go beyond the Western tradition, acknowledging the environment not as an object, but as a subject: one that is not only unto one’s self, but also among an ecology of selves. This idea is enriched by the proposition put forth by anthropologist Eduardo Kohn in his book How Forests Think; after four years of experiential research with the Quechua-speaking Ávila Runa, in the Upper Amazon of Ecuador, Kohn proposes an anthropology that goes beyond the human. The book’s central premise is developed around the notion of what constitutes selfhood. Kohn equates individuality with thinking, in the sense that if something or someone—a broad category that goes beyond humans— experiences intent, purpose, function, or meaning, then that something or someone is alive or “enchanted.” Therefore, a forest is alive and thinking, just as a dog, a jaguar, a skunk pig, or a plant are all alive and thinking. 

Figure 2. Paul Rosero Contreras. El Pensamiento de las Plantas. Caer. Fotografía, 2019.

Kohn elaborates on this discussion around individuality by explaining that life is a process of signs through which, citing scientist and philosopher Charles Peirce, any living being is “something which stands to somebody for something in some respect or capacity.” ⁵ As such, animals, plants, and spirits should be understood as beings-unto-their-selves, co-defining what is human as part of a larger planetary dynamic.

Kohn suggests that there are many different types of selves, from the “skin-bound organism” to one that is distributed across many bodies,⁶ like a group of people or an ant colony. To understand this concept, he says, one must first provincialize language, to challenge social theory which conflates the idea of representation with human language. The latter depends on a symbolic representation based on conventional—or “arbitrary”—signs that are embedded with other such signs that constitute an entire representational system. These fixed signs are contextualized and made to relate to the object they reference, and because this is how signs work in language, we assume that all representational processes exhibit these same associated linguistic properties. But the symbolic representation of forms is different from the human representation of forms: it goes beyond what is represented by humanity. It is knowledge outside of the human, existing a priori on the planet, continually emerging and changing. And in this manner, the forest and the nonhuman world think, Kohn concludes.

From an altogether different point of reference, the Italian plant neurobiologist Stefano Mancuso—considered to be one of the founders of the field—reaches the same conclusion. Mancuso makes a case for the existence of plant intelligence based on sensitivity: a capacity to learn and retain memory that enables plants to continuously monitor chemical-physical parameters, thus allowing them to understand and respond to their environment. Therefore, forests are their own selves because they communicate, feel, represent, and modify: they not only make possible but also host the future. ⁷ Forests contain both the human representation of forms, and the preexisting representation of a being in one’s self.

Thinking is also a part of memory. If we did not have memory, would we even think? In the case of humans, our brains and bodies are instruments wherein memory resides. But memory is also transcorporeal and collective, as is thought. A thought is a response to that memory. A reflection is a brief recollection. The leaves of Mimosa pudica—or sensitive plant—clench instantly in a remembered moment. Could it be that in their memories, we humans are their predator? Or are they simply telling us not to touch them by retreating? 

Figure 3. Paul Rosero Contreras. El Pensamiento de las Plantas. Caer #2. Fotografía, 2019.1

Mind and cognition seem to be defined by their very incorporation into the world—through simple sensory-motor acts—in an endless and interspecies interaction of sensations: a coevolution and co-definition of what I am on the inside, with what lies outside that defines me. This is a type of planthropocenic phenomenology that resonates with the ideas of philosopher Maurice Merleau-Ponty, since we must understand that the idea of humanity cannot be based on binary opposition to animals or subhumans or the more-than-human. But then, what do we do with this knowledge of a plant’s sentience? What happens when we acknowledge the fact that we are surrounded by nonhuman sentient beings and we start to live as if plants were our mentors, granting them the respect they deserve?

In my case, I proceed with projects like the one described in this text. From the political perspective of a nation-state, it could be a recognition of the Rights of Nature, like the one that was surprisingly included in the 2008 Constitution of the Republic of Ecuador, which made this country the first in the world to grant this legal status to nature. ⁷ However, since then we have come to question if these rights really are a transcendent innovation, or if they are mere legal rhetoric, or even just a political project within a world that is dictated by extractivist capitalism. What is clear is that a decolonial and antiracist analysis of the nonhuman is imperative in thinking about interspecies politics beyond the theatre of representing diversity that is typical of demagoguery.

Seen from another perspective, the changes that occur upon noticing the sentience and intelligence of other species reveal an attentiveness towards simple organisms as a means to understanding complex behaviors. For example, the same Mimosa pudica that reacts by touch, or a fly that has existed for thousands of years, which knows how to avoid hitting walls and feels the parameters of the room in which it flies, is in one’s self because this knowledge is self-sustaining, developed within its own epistemological world. These actions that seem like simple behavior are ones that took humans hundreds of years to replicate. 

Memory of water

El Pensamiento de las Plantas is also a series about memory retained in a vegetal body that, like the human, is largely comprised of water: a liquid memory that overflows, becoming transcorporeal. Vegetal bodies, human bodies, animal bodies: vessels of water that also provide life at the same time. A tree as a water vessel is a being in one’s self, existing beyond us, and yet whose existence constructs us. Humans are dependent on plants. In the ebb and flow of natureculture, as Donna Haraway puts it, I am also a being in one’s self, whose thoughts manifest themselves. In the name of alliance, discrepancy is permitted. My own thinking and memory can provide an answer, as one representation—and representative—in the world, distinct from the response of another nonhuman body. In addition, my project is embedded in a factual story ⁹ as a way to speculate a response derived simultaneously from the notions of both reciprocity and difference. This story recounts the tragic death of German documentary filmmaker Dieter Plage, who fell from a hot air balloon while filming a movie about the Sumatran rainforest canopy in 1993.

Figure 4. Paul Rosero Contreras. El Pensamiento de las Plantas. Devolución de la mirada (el objeto de estudio mira al estudiante). Fotografía, 2019.

Since he began his career in the late 1950s, Dieter Plage became one of the most recognized nature cinematographers and documentary filmmakers. To him we owe the unforgettable and dramatic recordings of humans interacting directly with wild animals in Africa, as well as other films about terrestrial and marine species that were widely broadcast on television networks in Europe and the United States. Plage, like the German naturalist Alexander von Humboldt in the 18th century, understood that the rainforest canopy was still an unexplored aerial world, one in which he had to reach with his camera.

In an era before the advent of commercial drones—or any other sophisticated remote-controlled flying device— Dieter Plage worked with British aeronautical engineer Graham Dorrington to build a hot air balloon that would allow him to shoot a film about the rainforest canopy, a site that was nearly inaccessible up to that point. During the filming of the documentary, sponsored by National Geographic, the balloon got entangled in a treetop. Because the balloon was made for a single passenger, Plage alone faced the dilemma of either trying to disengage the trapped aircraft and continue the flight, or to save his camera. This predicament would be his last: as he tried to disentangle himself from his cabin, Plage slipped and fell to his death. El Pensamiento de las Plantas, then, embodies the visual material that was kept within the water-vesselled bodies of trees who were eye witnesses to this tragedy: water transformed into an image. 

Figure 5. Paul Rosero Contreras. El Pensamiento de las Plantas. El momento de la caída. Fotografía, 2019.

The world, too, is made up of absences. Mimetic insects disappear altogether on leaves, camouflaged against the hampered vision of others. These insects live vis-à-vis their absence. They are, only to the extent that they disappear when faced with another's gaze, including our own. There is no sense of reciprocity. It is absence, being in one’s self, that even allows for the possibility of existing. Hence, absence also creates the world.

My project includes an inflatable sculpture called Obituary: a scale model replica of Plage’s crashed aircraft.2 With it, I wanted to tell a story of death, an absence that has created a legacy, a way of seeing, and the gaze's potentiality for reciprocity. Assembling this way of seeing is connected to non-Western ethical principles, establishing an understanding of a being's embodiment in one’s self that is contained within the natural world. This is a way of looking at the mind of the universe, as the scientist Francisco Varela would put it.

In 1940, Jakob von Uexküll, a German biologist who specialized in animal ethology, wrote that “a spider’s web is...formed in a ‘fly-like’ manner, because the spider itself is ‘fly-like.’ To be ‘fly-like’ means that the body structure of the spider has taken on certain of the fly’s characteristics.” ⁸ A web is a physical extension of the spider—a precise and geometric representation—that allows it to capture insects so efficiently that the web resembles some mutual pact between the creatures. Being attuned to another being and penetrating its phenomenological world requires, in a sense, an ontological blurring: which part of the web is fly and which part is spider? Or, what part of the photographs in El Pensamiento de las Plantas is forest, and what part of them is me? The mutability of senses in this project points to how certain attempts at communication among different species can also embody another type of becoming that spills and blurs across ontological borders. In this case, between humans and plants. This “vegetal becoming” does not negate the fact that the human body is always present behind the image. Rather, it opens up the potentiality for blurring, a place from which my project plants questions around the notion of what it means to be human, while at the same time envisioning both humans and plants as fellow astronauts.

Figure 6. Paul Rosero Contreras. El Pensamiento de las Plantas. Obituario. Inflatable sculpture, meteorite flakes, light filter. 2019.

In other words, we no longer need to avoid representation as if it were just a malevolent Eurocentric process of dissociation between things and words. Instead, we push for a representation that embraces true access— and an overflowing experience—to the world's richness.

References

1 Andrés Lomeña, “Qué se siente al ser una planta,” Huffington Post: Blogs, March 27, 2020, https://www.huffingtonpost.es/entry/que-se-siente-al-ser-una-planta_es_5e7cc9a4c5b6256a7a2634a8?fbclid=IwAR2fG1ETA6w9ULZKHF-5RJ-Xv-aepV_G7l [Last visited: 5/6/2022]

2 Alice Bucknell, “The Pioneering Artist Who Harnessed Science to Communicate with Plants,” October 16, 2019, https://www.artsy.net/article/artsy-editorial-pioneering-artist-harnesse... [Last visited: 1/6/2022]

3 In early 2019, Zoological Society of London (ZSL) scientists installed microbial fuel cells in the Rainforest Life exhibition at the London Zoo. Their aim was to harness biological energy—made by microorganisms— that would power a device which would allow a plant to take a photograph. The ultimate goal was to power camera traps and sensors planted in nature. ZSL's Conservation Technology Specialist Al Davies explains: “Plants naturally deposit biomatter as they grow, which in turn feeds the natural bacteria present in the soil, creating energy that can be harnessed by fuel cells and used to power a wide range of vital conservation tools remotely, including sensors, monitoring platforms, and camera traps. Most power sources have limits— batteries must be replaced while solar panels rely on a source of sunlight—but plants can survive in the shade, naturally moving position to maximise the potential of absorbing sunlight—meaning the potential for plant-powered energy is pretty much limitless.” https://www.zsl.org/zsl- london- zoo/news/say-green-cheese

4 Since 1974 when he founded the Bio-Arts Lab, Richard Lowenberg conducted experiments on interspecies communications with access to orcas in British Columbia, dolphins at the San Francisco aquarium, seals with Antarctic explorer Thomas Poulter of Stanford University, and later, the famous gorilla Koko.

5 Eduardo Kohn, How Forests Think: Toward an Anthropology beyond the Human, Berkeley, University of California Press, 2013.

6 Ibid.

7 Farith Simon, “Derechos de la naturaleza: ¿innovación trascendental, retórica jurídica o proyecto político?” in Iuris Dictio, 13(15), January– June 2013.

8 Eduardo Kohn, “How Dogs Dream: Amazonian natures and the politics of transspecies engagement” in American Ethnologist 34 (1) 2007, 3–24 

  1.  Here, I allow myself to draw a parallel with Gilles Deleuze’s idea of becoming: “The animal is defined not by characteristics (specific, generic, etc.) but by populations that vary from milieu to milieu or within the same milieu; movement occurs not only, or not primarily, by filiative productions but also by transversal communications between heterogeneous populations. Becoming is a rhizome, not a classificatory or genealogical tree. Becoming is certainly not imitating, or identifying with something; neither is it regressing-progessing; neither is it corresponding, establishing corresponding relations; neither is it producing, producing a filiation or producing through filiation. Becoming is a verb with a consistency all its own; it does not reduce to, or lead back to, ‘appearing,’ ‘being,’ ‘equaling,’ or ‘producing.’ Becoming concerns alliance. If evolution includes any veritable becomings, it is in the domain of symbioses that bring into play beings of totally different scales and kingdoms, with no possible filiation.” 
  2.  Dieter Plage was a naturalist with deep understanding and sympathy for the creatures he filmed. His personal style was pioneering among nature documentary filmmakers. Instead of simply filming natural history in an orthodox way, he conceived and covered great stories about the relationships between humans and wildlife in a dramatic way that captivated viewers. His work contributed towards making humans aware of nature as a world to be worshipped and respected for the sake of their own survival. The tragedy that took his life is recounted in Werner Herzog and Graham Dorrington's documentary The White Diamond, filmed in Guyana in 2004.

Bio Elektron - A Multisensory Approach to Augmenting Dance, Combining: Biosignals, Drawing, Sound and Electrical Feedback

Nuno N. Correia1, 2, Debora Souza1, Inês Nêves3, Jaime Lobato3

1Tallinn University, 2ITI/LARSyS, 3Independent artist Tallinn, Estonia
nuno.correia@tlu.ee, deboracs@tlu.ee, helloineves@gmail.com, jaimelobatocardoso@gmail.com

Abstract

In this paper, we investigate how to augment a dance performance using a multisensory approach in a way that communicates the dancing process as an embodied experience. We collaborated with a dancer and a media artist over an 8-week residency to prepare and present a multisensory dance performance and a spin-off installation. We present related work regarding key areas for this research: dance and technology in general; biosignal sensors; multisensory media (sound, drawing and haptics); and the relation between dance and installation. We also report on the artistic process, which was documented through seven interviews with the artists. Finally, we discuss strategies for drawing and sonification leading to heightened embodiment; approaches for drawing and haptics triggering impressions from the performance; while highlighting the importance of space as a unifying concept in embodied multisensory work. These strategies and approaches can be useful for artists interested in conducting related embodied multisensory work.

Keywords

Dance, Embodiment, Multisensory, Biosignals, Drawing, Sonification, Haptics, Electrical feedback.

DOI

10.69564/ISEA2023-49-full-Correia-et-al-Bio-Elektron

Introduction

There is a long tradition of using technology in dance. Important historical examples of the intersection between these fields are the works of Merce Cunningham in Lifeforms (1989), a software allowing to generate new choreography;²⁵ Mark Coniglio in MidiDancer (1989), a wearable device that allowed a performer to control media;¹⁰ and Frieder Weiss in EyeCon (2004), a motion-sensing tool which allows movement to control several aspects of a performance.²⁹

There is extensive research on using biosignal sensors to reveal body data (such as muscle activity or heart rate) through sound, as presented in a recent review of these approaches.¹ Biosignal sensors have also been used in dance to visualize the inner processes of the dancers.⁹ Recent research, involving 10 contemporary dance professionals with experience combining dance and technology, has identified potential in using technology to reveal non-visible elements in a performance—such as the thought process of dancers or their bodily data.¹⁷ However, there is a lack of research in combining these different modalities in dance into a multisensory experience, particularly combining visual, sonic and haptic elements.

Multisensory experience design can transform the way we experience art: “by carefully considering different senses and their possible interrelations it may be possible to design and shape specific human experiences.”²⁸ An example of this approach was the Tate Sensorium exhibition at Tate Britain (London). A multidisciplinary team of researchers and practitioners designed the exhibition so that specific sensory elements (sounds, haptics, smells and foods) would augment the experience of four paintings.²⁸ The possibilities of combining multisensory experiences with interaction design “open up opportunities to explore new experiences for perceiving one's own body, its interactions with the environment and also to explore the environment itself.”² In terms of performing arts, “the demand for multi-sensory experiences is ever-increasing, given the rise of immersive art and theater in our post-pandemic world.”¹¹

Zhou et al. have conducted a review of the past twenty years of dance literature in the field of Human-Computer Interaction (HCI).³⁰ This led to the identification of four main categories of technological approaches for dance: Physiological Sensing; Multisensory Perception; Movement Quality; and Agent Collaboration. Through our research, we aim to combine the first two: the categories of Physiological Sensing and Multisensory Perception.

Our research aims to investigate how to augment a dance performance using a multisensory approach (combining visual feedback and biosignal sensors mapped to sound and haptics), in a way that communicates the dancing process as an embodied experience. To fulfill our research aim, we collaborated with artists Inês Nêves and Jaime Lobato (also co-authors of this paper), and performance space Elektron (Tallinn), in the scope of the Starts.ee research project by Tallinn University. During an 8-week residency, Inês and Jaime prepared and presented a multisensory dance performance (Bio Elektron), a spin-off installation, and developed related software and hardware systems. In this paper, we report on this process and its results, while discussing their implications.

Background

In this section we present background and related work regarding key areas for this research: dance and technology in general; input technology adopted (sensors); output multisensory media used (sound, drawing and haptics); and finally, the relation between dance and installation.

Dance and Technology

There has been an increased interest in technology for dance, in parallel to a growing importance of embodied interaction, in the field of HCI. Within the intersection between dance and HCI, Fdili Alaoui et al. classified categories of tools into: Generation (of new choreographic material); Interaction (in real time with performers on stage); Reflection (on choreography); and Annotation (tools assisting the creative process).¹³ Raheb et al. categorized dance technologies in a similar perspective: Choreographic tools; Augmented performance; Education; Research and analysis; and Games.²³ Examples of technological approaches in these categories are: development of tools and techniques for annotation ³; tools for documenting choreographic processes;⁶ real time interaction;¹⁴ and choreography generation.²⁵ Our work is mainly situated in the real time interaction category of tools according to ¹³, or augmented performance according to ²³, with the exception of the spin-off installation.

Dance and Sensors

Rostami et al. generated five design concepts for interactive performance adopting bio-sensing and bodily tracking technologies ²⁴. Likewise, Aly et al. reviewed biosensor modalities for performance from an HCI perspective, discussing the potential of muscular activity to convey rich movement information.¹ The research by Rostami et al. and Aly et al. include not only dance but other performance areas as well (the latter focuses on music). The authors highlight the potential of using biosignal sensors to collect data from the body for performing arts, which was the approach we followed in our work.

Dance and Sound

Researchers have focused on methods and techniques to create music from movement, particularly from dancers. A relevant example on how to extract information from the body is the work of Camurri et al.⁴ Another pertinent project studied mappings of expressivity in gesture to sound.⁵ These works focus on information retrieval and mappings or similar aspects toward the development of the technology itself. Recently, Masu et al. analyzed the sonic interactions that occur in a dance performance from an ecological perspective.¹⁸ They studied not only the technology and its design, but also the roles of the different actors in the design and implementation process.

Dance and Drawing

There is a tradition of using drawing to visualize and record movement in dance performances. A milestone in this field is Carolee Schneemann’s Tracking from 1973. Using a rope attached to the ceiling of a train car, Schneemann held a chalk in one hand extended, so that changes in position were marked on the walls and floor it touched.¹² Schneemann would expand this approach in her piece Up to and Including Her Limits (1973–1976). In her series It’s a Draw, Trisha Brown dances while painting on a large paper placed on the floor. By doing so, she collapses four dimensions: “the three dimensions of Brown’s movement in the field above, plus the time spent doing it.”¹² Haley conducted an extensive review of the use of drawing to visualize movement in dance, while presenting her own piece Constructions of the Moving Body, which has as objective “to represent the experience of watching Brown’s dance Accumulation, and evoke kinesthetic empathy through the drawn image.”¹⁵

Dance and Haptics

There are few examples of use of haptics with dance performances. Mostly, these have dealt with allowing visual or hearing-impaired audiences to experience dance, and with dance education. The Choreo-haptic project aimed “to investigate how the kinaesthetic empathy experienced by sighted dance audiences can be also experienced by blind dance audience members.”¹⁶ It allowed audiences to feel movement in dance through vibrotactile haptics. Movement data was captured by Microsoft Kinect and then transmitted to a haptic pad (with an array of 6 by 5 motors).¹⁶ A similar system, the Haptic Cushion, featured a grid of 8 by 8 vibrotactile actuators fitted to the back of a chair, and was tested with visually impaired audience members.²¹ Shibasaki et al. developed a system to allow hearing-impaired audience members to enjoy the performance of tap dancers. The system captured dance data through haptic microphones on the stage, while seating provided haptic information to the audience, using a power amplifier.²⁷ Another application area for haptics in dance is education. The Haptic Feedback Ankle Bracelets “enable learning the footwork for any dance through conditioning the learner to move their feet in accordance to the choreography which follows the beat of the music.”²⁶

Dance and Installation

There has been an increased interest in adapting performing art pieces to interactive installations. Correia researched converting audiovisual performance pieces to browser-based artworks.⁷ Particularly in the field of dance, the choreography Emotional Landscapes was transformed into a VR environment, where users can explore a dynamic relationship between the dancer and the virtual world, informed by the original dramaturgy of the piece.⁸ Digital Connection Retrieval adapts a dance piece into a browser-based installation, using the web camera as an interaction mechanism.¹⁹

Methods

In the scope of the project Starts.ee, two artists participated in an 8-week art and science residency program, aiming to explore biosignal data in contemporary dance performances. Inês Nêves is a performance artist who assumes drawing as the core of her practice. Jaime Lobato is a multimedia artist, music composer, and independent researcher who uses biosignals as part of his artistic practice. The work took place between November 2021 and February 2022. The 8 weeks of work were not continuous, as there were pauses due to holidays and health issues.

The artists’ process was regularly documented through photos and notes. During that time, we conducted seven unstructured interviews (approximately one per week) with the artists about their process, and critical assessment of the work done. The two last interviews took place after the final show. The interviews were video recorded and transcribed, and then subjected to an interpretivist analysis.²⁰ The data was analyzed by the second author, and the analysis was double-checked by the first author.

Design Process

The reflections presented in this section stem from the interviews conducted with the artists-in-residence documenting their design process, across three different phases.

(1) Ideation and Prototyping (weeks 1-3)

From the beginning, the artists acknowledged interest in investigating the physical and the perceived space in the artistic performance. Inês commented about the willingness to investigate the implicit aspects of her practice:

“We spoke about exploration of space, on both the physical space and the perceived space. In relation also to the micro movements and how our body moves internally [...] So, we were thinking that the sensor could be there as a significant part of the process.” Inês, Interview 1)

The artists identified a common ground connecting their artworks: the relationship to movement and space and how their artworks were influenced by improvisation. Whereby the use of biosensors could bridge their artistic practices, performative movement and sound, while facilitating the communication with the audience. As they mentioned retrospectively at the end of the project:

“One of the motivations of participating in this project was to keep developing my research in this electricity produced by humans” (Jaime, Interview 7)
“I was interested in doing this residency because I thought that biosensors could help to amplify this human quality of visual art by enabling the accessibility to the immaterial qualities of the creator, so me as the performer.” Inês, Interview 7)

During their initial exchanges, the artists developed the idea and conceptualization of the performances they wished to create. They set the goal of mapping the body movement energy, using electromyography (EMG) sensors to convert muscle data into sonification layers, which could add on the perception of the performance.

Sound feedback started to be explored using contact microphones, to record the noises of the drawing materials on the large sheet of paper used as a drawing surface. Initially, the paper was placed on a table, later on the floor (on top of a wooden board)—the contact microphones were attached, respectively, to the table and to the wooden board (figure 1).

Figure 1. The photos illustrate the prototyping process. At the left, the initial tests with the sensors and the drawing movements. And at the center and right, the sequence shows the artists attaching the contact microphones.

As Jaime recalls:

“When we were sharing our work, I saw in the last performance that Inês made very interesting sounds [...] And I also remember one piece by the Fluxus group; it’s like a written score, and the idea is like to grab a microphone with a piece of paper, and then the microphone amplifies the noise”. (Jaime, Interview 2)

By the end of week 2, the artists started playing with the biosignals from the performer’s movements to modulate media outputs, exploring ways of allowing the audience to witness the embodied experience of the performer. In other words, to offer a poetic point of view of the inner bodily reactions through the manipulation of the sound.

Jaime brought in an assembled device created by him to collect EMG signals from Inês’s movements and they engaged in experiments measuring biosignals of her drawing activity. The EMG signal from Inês’s arm muscles was used to apply the average energy amplitude of the drawing movements, as variables to modulate and synthesize the sound output presented to the audience during the drawing performance. The data captured by the EMG sensor was used to process, in Supercollider (https://www.audiosynth.com), the sound of the drawing recorded with the contact microphones. Hence, noises of the drawing act were transformed into a loop of sound effects. As Jaime explained, the resulting sound loops emulated what was happening in the drawing:

“Because in the drawing, you do it through time but, at the end you have this accumulation of gesture, colors and light, so we can have something similar to the sound. I have several variables to switch in the synthesizer.” (Jaime, Interview 3)

Installation development

Additionally, the artists designed an installation to explore another sensorial aspect of the perception of the drawings, touch. As the artists pointed out, this idea came up when they were considering the poetic aspects of a haptic feedback loop, making people “feel this variation of electricity [from the performer’s activity] as a memory of the movements of Inês’s drawings” (Jaime, Interview 2). The idea also refers to a Mexican custom, ‘toques’, where people clutch two metal rods and allow themselves to receive electric shocks for fun.

The sensorial installation focused on the individual and intimate perception of the art piece. It aimed to allow the audience to feel the electric impulses coming from Inês’s arm while she performed the drawing, revealing:

“[...] the movements of my body through these physical impulses and through the encounter of the materials. [By showing that] in an exhibition layout in which we’d have the drawings, and instead of seeing the performance of those drawings, the audience would feel the impulse, to feel the performance of the drawing as they look at it” (Inês, Interview 3)

(2) Rehearsals (weeks 4-7)

In week 4, the artists began the rehearsal process and started exploring the prototype in action, to identify and address possible issues. Testing the prototype during the rehearsals also helped to calibrate the acquisition of the sensor and enhance the quality of the biosignal processing.

Furthermore, their collaboration extended to the shared tasks of assembling the performance hardware and fixing the data collection devices. This built upon Inês’s expertise in textile design. Accommodating the design of the wearables was an important aspect in order to attach the sensor to the performer’s body, consequentially increasing the reliability of the data collection. It also added an aesthetic quality to the performative act.

One of the emerging challenges was that physiological data acquisition methods are often used in controlled settings, not in dance:

“Wearable prototyping is also a very important proof of concept, because biosignals are not supposed to be taken from the body in movement. So, trying to minimize [signal] noise for a more performatic usage of this technology [...] it’s going to be like a proof of concept of taking this [biosignal] amplifier and making it to be a total wireless but also noiseless” (Jaime, Interview 4).

Inês kept researching the movements and different aspects of the drawing activity. Additionally, throughout rehearsals, she improved the design of the wearable prototype, to safely hold the sensor hardware, while protecting her body during more abrupt drawing gestures. In parallel, Jaime progressed with the acquisition and processing methods, making the final adjustments in the code, which implemented the performer’s biodata as variables for sound manipulation. By the end of the rehearsal phase, he had revised the data acquisition method to reduce signal noise:

“[...] mainly building electronics and writing the software [...] bleaching [soldering] of the connections between the sensors and the Arduino control to have the amplifier working wireless. Inês as well finished the wearable part, so now she can wear the system comfortably.” (Jaime, Interview 5) 

Installation development

The artists also continued their research on the installation:

“I have been doing some research about how [electricity] works with the body, which is the limit of the secure setup that we can do. Also, I have found very interesting bibliography about the relationship of the body with external and internal electricity” (Jaime, Interview 4)

After developing the installation system, the artists decided to produce the material of the installation beforehand, in order not to compromise the computational process of the live performance in the public show. The EMG data used as the electric stimuli for the installation was recorded from the last rehearsal, in week 7. Naturally, the drawing exhibited in the public show also resulted from that same rehearsal.

(3) Public Show and Post-Performance (week 8)

The final showing was open to the general public and took place on the 6th of February 2022 at Elektron, respecting COVID-19 sanitary measures. It started with the performative drawing (figure 2), followed by a second performative act (out of the scope of this paper). It also included a Q&A session for the audience to clarify their questions with the artists. Furthermore, the audience had the chance to try out the installation after the performance. 

Figure 2. The photo shows the audience surrounding Inês Nêves during the drawing performance.

After the showing, we conducted two interviews with the artists (interviews 6 and 7). The first one was conducted for our own research, the second one also for promotional purposes. Reflecting on the strengths and weaknesses of the work done, both artists expressed that comments from the audience made them consider how the show could have been improved. Jaime recalled the comments by the audience after the show had given him ideas of how to better balance the different sensorial aspects presented to the audience. He also wanted to improve the system’s robustness:

“More than the artistic thing, I could improve the sign from the amplifier, putting the wi-fi antenna to incorporate the circuit. It would be more robust [...] maybe less noise would come through.” (Jaime, Interview 6) 

Inês, on the other hand, expressed her impression about the flow of the performance:

“I think I’d try different materials. I’ve used what was necessary for the sound but it actually made too much accumulation on the drawing. So, I’d like to try it in different styles.” Inês, Interview 6)

Recapping the development of the arts and science residency, the artists mentioned they have achieved their goals with the project about exploring spatial and sensorial dimensions. Including the sensor’s data as an artistic variable made them think and perceive their practices on the project differently. Inês commented on her impressions of having an augmented sensorial perception in the performance:

“I have discovered that, what I really appreciate about doing these performative drawings is how, when you see the drawing, [it] comes from the body, and when you see it, you can sort of feel the performer there. [...] it’s like, it becomes more physical”. (Inês, Interview 7)

Installation development

About the installation (figure 3), Jaime explained how he achieved the setup, recorded from the final rehearsal, clarifying what the audience was feeling:

“It was the record of a full performance. We made it as it happened on the performance, every step as in the official [performance]. I started to record [data for the installation] from the beginning of the drawing until the end. And then the full performance was a loop in the installation. [...] Because there were very rapid peaks of amplitude and I didn't want that strong impulse to get to the public, so I extended that in time.” (Jaime, Interview 6) 

Figure 3. Installation photos. On the left, installation setup. On the right, audience member is clutching two metal rods. The recorded muscle data of the dancer while drawing is being re-played, and conveyed to the audience member as electricity passing through the metal rods. The resulting drawing by the dancer is on display.

In addition, Inês shared her thoughts on the physicality of the experience:

“I think it’s so beautiful the idea of being able to feel physically close with someone who is not present now; it’s like you can hug or touch a performer that is not there just by looking at the results and how it was crystallized in the electricity and drawing.” (Inês, Interview 7)

The artists also reflected on the aspects probed to conceive the installation and their intention to keep exploring how to convey experiences through the skin:

“This is more like a memory trigger, a more poetic way to approach the data we were collecting through the sensors, like the imagination and the poetics of [electricity] as a media itself.” (Jaime, Interview 7)

Technical Implementation

For the performance system, Jaime worked with a biosignals amplifier, that is, an operational amplifier (op-amp) calibrated to amplify signals from the human body: muscles (myography), eyes (eye tracker), or heart (electrocardiogram). The op-amp from the beginning of the residency was designed and built by Bruno Eloy Méndez Ambrosio. This circuit has two benefits regarding some commercial sensors: firstly, it has a differential amplifier that assures the user to reduce the noise caught by the ambiance or the electrical system; and secondly, it is open hardware so it is easily connected to any software to use the biosignal in real time.

As part of the residency, Jaime added a potentiometer in order to move the offset of the signal, as it was digitized with Arduino and there was the need to have only positive numbers. He also developed software with analysis tools such as: Signal averaging; True Root Mean Square (TRMS); Heartbeat signal damping; Heartbeat signal isolation; and 2nd order Butterworth filters.

An extra module was added for real time connectivity. It can send the raw signal and all the analysis indexes from the user to any software by MIDI, OSC and serial protocols. The information can be recorded as CSV format for deferred time analysis. The work was centered in finding critical biomarkers in the time series associated with physiological responses, which could be turned into experimental animations and data sonification or data-driven composition.

During the performance, the biomarkers were used in real time. There was a wooden board (where the drawing was attached to) with 4 contact microphones, which were mapped to each of the 4 loudspeakers in a quadraphonic arrangement. This wrapped the performer and the audience. Thus, the sound could be specialized with the performer’s movements. Also, a myograph was placed in the arm of the artist as controller for a sound loop station, emulating the drawing process as an accumulation of sound.

In the installation, the biomarkers were used in deferred time. The TRMS information of the artist’s arm was recorded during the performance. Jaime adapted a Steren ‘toques’ box (see section Ideation and Prototyping) to receive this data. He swapped the mechanical potentiometer of the box for an electrical one, so the public could feel the electric variations of the performer as it happened in the performance, while watching the drawing produced. By holding the left and right ‘toques’ rods, the user was able to perceive the electrical voltage produced by the dancer’s muscles while dancing and drawing. The drawing from the corresponding rehearsal was displayed on the wall next to the ‘toques’ hardware. The user could thus re-imagine the drawing process, and the energy behind it, based on the muscle data being conveyed through electrical current.

Discussion

In this section, we will discuss the multisensory approaches followed, and related perspectives derived from the work.

Drawing and sonification leading to heightened embodiment

According to Inês, the multisensory aspects of the performance (movement and associated drawing, both producing sound) led to an added perception of the dancer’s body: “it becomes more physical”. The multisensory approach has been successful to heighten to presence and bodily impression of the dancer. Jaime’s design approach toward sonification of movement was tightly coupled with the process of drawing and its cumulative nature: “you have this accumulation of a gesture, colors and light, so we can have something similar to the sound.” He achieved this by recording and then looping segments of sound, which were then reprocessed based on biosignal data from Inês. The co-existence of the resulting multiple layers of sound echoes the accumulation of the multiple layers of drawing on paper. This relates to Eleey’s description of collapsing four dimensions in performative drawing: three dimensions of movement, plus time.¹² In this case, both drawing and sound led to the collapse of the four dimensions. Inês also highlighted the importance of the choice of drawing materials for producing interesting sounds. But this should be balanced with the need to avoid excessive accumulation: “I’ve used what was necessary for the sound, but it actually made too much accumulation on the drawing”.

In summary, we argue that there are important factors for sonifying drawing-producing gesture, leading to heightened sense of embodiment: 1) to mirror the process of drawing, and its accumulation, with sound; 2) to use data from the movement itself (e.g., biosignals) to affect sound; 3) to use interesting sound-producing drawing materials; and 4) to carefully balance all these elements in order to leave ‘space’ in the visual and auditory domains, that is, to avoid saturation of sound and of drawing. When using haptics as part of a multisensory approach in dance, this should be also harmonized with the other multisensory aspects.

Drawing and haptics triggering impressions from the performance

Another important multisensory approach of the work was the combination of haptics and drawing. The haptic element of the installation aimed to trigger in the audience impressions from the act of drawing, when combined with the stimuli of viewing the resulting drawing. The haptic aspect consisted in conveying the muscle energy of the performer (EMG data collected in the performance) to the audience through electrical impulses, by holding two electrical metal rods in both hands. The piece creates an energy link between performer (muscle energy captured) and audience (muscle energy conveyed through electric signal). As Inês stated: “I think it’s so beautiful the idea of being able to feel physically close with someone who is not present now (...) looking at the results and how it was crystallized in the electricity and drawing.” Jaime calls it “a memory trigger” of a performance, allowing the audience to perceive it through the combination of traces it leaves (the drawing and the corresponding performer’s energy). To allow the audience to better experience peaks and variations of electrical impulse, Jaime extended the data time span of, from 18 minutes to one hour. Thus, the audience experienced the energy from the drawing in a slowed-down pace, for better perception of the nuances in the data.

The installation presents a novel approach to haptics, compared to related ones presented in our Background section (which do not use an electrical signal as output). The haptic approach followed allows one to experience in deferred time a performing art piece, using electrical current to convey the energy behind the process of drawing. This, coupled with visualizing the final drawing resulting from that process, can have an evocative effect. The strategy followed, conveying to an audience the energy impulse of a performer through electrical signals, possibly in combination with other sensory elements, can be an inspiring approach to other artists (either in deferred time, as in our case, or even in real time).

Space as a unifying concept in embodied multisensory work

This project demonstrates that when carrying out embodied multisensory work, it is key to be mindful of the body’s relationship with space. It is through the interaction between our bodies and space that we build our knowledge about the world, an idea supported by Merleau-Ponty’s thoughts on human perception and cognition. Merleau-Ponty stated that we comprehend the world through our ‘body schema’: a general awareness of our existence within the “inter-sensory world.”²² He also defined space as what connects all things, instead of where they merely lay.

Following these reflections, in this project we used space to generate dynamic connections between the fields of drawing, movement, and sound, as well the performer’s and the audiences’ sensory experience. The use of contact microphones, sound spatialization and bio-sensors allowed the performer to sculpt sound in space with her movements. Alternatively, the sound produced shaped how she drew and moved in space.

This method for using space to intersect different disciplines could be a beneficial framework for other embodied multisensory work. Additionally, working within an expanded space could also allow a higher variety of embodied experiences (e.g., wider and smaller gestures, more impulsive and more controlled movements). Space and time are also connected dimensions in terms of accumulation of drawing. A longer performance requires a larger sheet of paper, to avoid excessive accumulation, rendering the drawing illegible.

Challenges of appropriating wearable sensors for performing arts

The sensors used in this work combine the knowledge of Jaime and his associates in terms of sensor and amplifier design, and Inês’s knowledge in textile design. The latter allowed her to create a wearable housing for the sensors, which was gradually improved in terms of comfort and robustness. Biosignal sensors are not normally designed to be used for artistic purposes, let alone for the demands of dance performance. Dance creates strains in terms of noisy signals, discomfort of use, and connectivity issues. The necessary adaptation requires skills in appropriating sensor and wearable design for performance purposes.

Conclusion

Our research aim was to investigate how to augment a dance performance using a multisensory approach (combining visual feedback and biosignal sensors mapped to sound and haptics), in a way that communicates the dancing process as an embodied experience. In the Discussion section, we present how we achieved this research aim, materialized into: strategies for drawing and sonification leading to heightened embodiment; approaches for drawing and haptics triggering impressions from the performance; while highlighting the importance of space as a unifying concept in embodied multisensory work. These strategies and approaches can be adapted and replicated by artists interested in conducting related multisensory projects. Our work is novel, particular considering augmenting performative drawing with sonification and haptic approaches. The latter allows extending the performance into an installation, in deferred time, using an innovative electricity-based approach.

The main limitation of the research is that our practice-based approach could have been complemented by audience studies, to assess the effectiveness of our multisensory approaches from an audience perspective. In terms of future work, we would like to adjust the installation to display in real time the data collected in the performance. We would also like to explore alternative models of interaction with the public by using the audience’s biosignals as an extra input for the performance. This would disrupt the hierarchy between artists and audience, while allowing us to close the biosignal loop between them.

Acknowledgements

We acknowledge the support of Starts.ee project, of Elektron and of LARSyS (Project - UIDB/50009/2020) to this research.

References

1 Luís Aly, Hugo Silva, Gilberto Bernardes, and Rui Penha, Appropriating biosensors as embodied control structures in interactive music systems. Human Technology: An Interdisciplinary Journal on Humans in ICT Environments 17, 1, 2021, 45–80.
 DOI:https://doi.org/10.17011/ht/urn.202106223978

2 Maryam Bandukda, Aneesha Singh, Catherine Holloway, Nadia Berthouze, Emeline Brulé, Ana Tajadura-Jiménez, Oussama Metatla, Ana Javornik, Anja Thieme, Rethinking the Senses: A Workshop on Multisensory Embodied Experiences and Disability Interactions, In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA ’21), Association for Computing Machinery, New York, NY, USA, 2021, 1–5, DOI:https://doi.org/10.1145/3411763.3441356

3 Diogo Cabral, Urândia Carvalho, João Silva, João Valente, Carla Fernandes, Nuno Correia, Multimodal Video Annotation for Contemporary Dance Creation, In CHI ’11 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’11), ACM, New York, NY , USA, 2011, 2293–2298, DOI:https://doi.org/10.1145/1979742.1979930

4  Antonio Camurri, Barbara Mazzarino, Matteo Ricchetti, Renee Timmers, Gualtiero Volpe, Multimodal Analysis of Expressive Gesture in Music and Dance Performances, In Gesture-Based Communication in Human-Computer Interaction (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 2004, 20– 39, DOI:https://doi.org/10.1007/978-3-540-24598-8_3

5 Ginevra Castellano, Roberto Bresin, Antonio Camurri, Gualtiero Volpe, Expressive control of music and visual media by full-body movement, In Proceedings of the 7th international conference on New interfaces for musical expression (NIME ’07), Association for Computing Machinery, New York, NY, USA, 2007, 390–391, DOI:https://doi.org/10.1145/1279740.1279829

6  Marianela Ciolfi Felice, Sarah Fdili Alaoui, Wendy E. Mackay, Knotation: Exploring and Documenting Choreographic Processes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), ACM, New York, NY, USA, 2018, 448:1-448:12. DOI:https://doi.org/10.1145/3173574.3174022

7  Nuno N. Correia, Interactive Audiovisual Objects, Doctoral thesis, Aalto ARTS Books, Helsinki, 2013, Retrieved April 12, 2020 from https://aaltodoc.aalto.fi/handle/123456789/11341

8  Nuno N. Correia, Stephan Jürgens, Raul Masu, Jochen Feitsch, Ivana Druzetic, Performative Virtual Scenes: A Dynamic VR Environment Design Approach. In Entertainment Computing – ICEC 2021 (Lecture Notes in Computer Science), Springer International Publishing, Cham, 2021, 100–114, DOI:https://doi.org/10.1007/978-3-030-89394-1_8

9  Nuno N. Correia, Raul Masu, William Primett, Stephan Jürgens, Jochen Feitsch, and Hugo Plácido da Silva, Designing Interactive Visuals for Dance from Body Maps: Machine Learning and Composite Animation Approaches, In Designing Interactive Systems Conference (DIS ’22), Association for Computing Machinery, New York, NY, USA, 2022, 204–216, DOI:https://doi.org/10.1145/3532106.3533467

10  Scott deLahunta, Isadora ‘almost out of beta’: tracing the development of a new software tool for performing artists. International Journal of Performance Arts and Digital Media 1, 1, January 2005, 2005, 31–46, DOI:https://doi.org/10.1386/padm.1.1.31/1

11 Gabrielle Di Raddo, Haptic Feedback: Feeling the Dance You Cannot See, Arts Management & Technology Laboratory, CMU, Retrieved November 13, 2022, https://amt-lab.org/blog/2022/4/haptic-feedback-feeling-the-dance-you-cannot-see

12 Peter Eleey, If You Couldn’t See Me: The Drawings of Trisha Brown, In Trisha Brown: So that the Audience Does Not Know Whether I have Stopped Dancing, Peter Eleey and PhilipBither (eds.), Walker Art Center, Minneapolis, 2008, Retrieved October 30, 2022,  http://walkerart.org/collections/publications/performativity/drawings-of-trisha-brown/

13 Sarah Fdili Alaoui, Kristin Carlson, Thecla Schiphorst, Choreography As Mediated Through Compositional Tools for Movement: Constructing A Historical Perspective, In Proceedings of the 2014 International Workshop on Movement and Computing (MOCO ’14), ACM, New York, NY, USA, 1:1-1:6. 2014, DOI:https://doi.org/10.1145/2617995.2617996

14 Sarah Fdili Alaoui, Christian Jacquemin, Frédéric Bevilacqua, Chiseling bodies: an augmented dance performance, In CHI Extended Abstracts, ACM, Paris, 2915–2918, 2013, Retrieved October 20, 2020, https://doi.org/10.1145/2468356.2479573

15 Rochelle Haley, Constructions of the moving body: drawing and dancing, Studies in Theatre and Performance 38, 3 September 2018, 289–301, DOI: https://doi.org/10.1080/14682761.2018.1506966

16 Sophia Lycouris, Wendy Timmons, John Ravenscroft, Mark Wright, Choreohaptic Experiments (2012), Retrieved November 13, 2022, https://www.research.ed.ac.uk/en/publications/choreohaptic-experiments

17 Raul Masu, Nuno N.Correia, Stephan Jurgens, Ivana Druzetic, William Primett, How do Dancers Want to Use Interactive Technology? Appropriation and Layers of Meaning Beyond Traditional Movement Mapping, In Proceedings of the 9th International Conference on Digital and Interactive Arts (ARTECH 2019), Association for Computing Machinery, Braga, Portugal, 2019, 1–9, DOI:https://doi.org/10.1145/3359852.3359869 

18 Raul Masu, Nuno N.Correia, Stephan Jurgens, Jochen Feitsch, and Teresa Romão. 2020. Designing interactive sonic artefacts for dance performance: an ecological approach. In Proceedings of the 15th International Conference on Audio Mostly (AM ’20), ACM, Graz, Austria, 122–129. DOI:https://doi.org/10.1145/3411109.3412297

19 Raul Masu, Hanna Pajala-Assefa, Nuno N.Correia, Teresa Romão, Full-Body Interaction in a Remote Context: Adapting a Dance Piece to a Browser-Based Installation, In 10th International Conference on Digital and Interactive Arts (ARTECH 2021), Association for Computing Machinery, New York, NY, USA, 2021, 1–4, DOI:https://doi.org/10.1145/3483529.3483747

20 Katrina Mc Chesney, Jill Aldridge, Weaving an interpretivist stance throughout mixed methods research, International Journal of Research & Method in Education 42, 3 May 2019, 225–238, DOI: https://doi.org/10.1080/1743727X.2019.1590811

21 John Mc Cormick, Mohammed Hossny, Michael Fielding, James Mullins, Jordan Beth Vincent, Mostafa Hossny, Kim Vincs, Shady Mohamed, Saeid Nahavandi, Douglas Creighton, Steph Hutchison. 2020. Feels Like Dancing: Motion Capture-Driven Haptic Interface as an Added Sensory Experience for Dance Viewing. Leonardo 53, 1 February 2020, 45–49, DOI:https://doi.org/10.1162/leon_a_01689

22 Maurice Merleau-Ponty, Taylor Carman, Phenomenology of Perception (1st edition ed.), Routledge, Abingdon, Oxon, New York, 2012.

23 Katerina El Raheb, Marina Stergiou, Akrivi Katifori, Yannis Ioannidis, Dance Interactive Learning Systems: A Study on Interaction Workflow and Teaching Approaches, ACM Comput. Surv. 52, 3 June 2019, 50:1-50:37, DOI:https://doi.org/10.1145/3323335

24  Asreen Rostami, Donald McMillan, Elena Márquez Segura, Chiara Rossito, Louise Barkhuus, Bio-Sensed and Embodied Participation in Interactive Performance, In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’17), Association for Computing Machinery, Yokohama, Japan, 2017, 197–208, DOI:https://doi.org/10.1145/3024969.3024998

25 Thecla Schiphorst, A case study of Merce Cunningham’s use of the LifeForms computer choreographic system in the making of Trackers. Master of Arts, Simon Fraser University, 1993, Retrieved February 21, 2022, https://www.semanticscholar.org/paper/A-case-study-of-Merce-Cunningham%27s-use-of-the-in-of-Schiphorst/a0f9abc63c77db0f84a5a1de8601b6fc9a7e0720

26 Navid Shaghaghi, Yu Yang Chee, Jesse Mayer, Alissa LaFerriere, SwingBeats: An IoT Haptic Feedback Ankle Bracelet (HFAB) for Dance Education. In Intelligent Technologies for Interactive Entertainment (Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering), Springer International Publishing, Cham, 2021, 82–101, DOI:https://doi.org/10.1007/978-3-030-76426-5_6

27  Mina Shibasaki, Youichi Kamiyama, Kouta Minamizawa, Designing a Haptic Feedback System for Hearing-Impaired to Experience Tap Dance, In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST ’16 Adjunct), Association for Computing Machinery, New York, NY, USA, 2016, 97–99, DOI:https://doi.org/10.1145/2984751.2985716

28 Carlos Velasco, Marianna Obrist, Multisensory Experiences: Where the senses meet technology, OUP Oxford, New York, NY, 2020.

29 Robert Wechsler, FriederWeiß, Peter Dowling, EyeCon -- a motion sensing tool for creating interactive dance, music and video projections, In Proceedings of the Society for the Study of Artificial Intelligence and the Simulation of Behavior (SSAISB)’s convention: Motion, Emotion and Cognition, Society for the Study of Artificial Intelligence and the Simulation of Behavior, Leeds, 2004, Retrieved January 17, 2018, 1–7,  http://www.palindrome.de/content/pubs/leeds.pdf

30 Qiushi Zhou, Cheng Cheng Cheng Chua, Jarrod Knibbe, Jorge Goncalves, Eduardo Velloso, Dance and Choreography in HCI: A Two-Decade Retrospective, In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21), Association for Computing Machinery, New York, NY, USA, 2021, 1–14, DOI:https://doi.org/10.1145/3411764.3445804 

“I’m a virtual assistant so I don’t have pronouns the way people do, thanks for asking”: gender neutrality, diversification and fluidity in AI

Pedro Costa1, Luísa Ribas2, Miguel Carvalhais3

1, 2 CIEBA Centro de Investigação e de Estudos em Belas-Artes / Universidade de Lisboa, Faculdade de Belas-Artes 3 i2ADS / Faculty of Fine Arts, University of Porto
Lisboa, Porto - Portugal
pedro.carv.c@gmail.com, mcarvalhais@fba.up.pt, l.ribas@belasartes.ulisboa.pt

Abstract

This paper questions how current digital assistants tend to be feminized through their anthropomorphization and humanization, discussing possibilities for countering this phenomenon. It draws on a previous study on the relationship between gender and AI, complemented by an analysis of Alexa, Cortana, Google Assistant and Siri. Furthering this discussion, we address the main questions, justifications and suggestions raised by researchers and academics as well as online media coverage when examining the phenomenon. One of the main questions relates to how these assistants evade this topic by claiming to have no gender or to be gender-neutral. Thus, this paper discusses possible approaches to deal with gender attribution in AI, by looking into recent trends that range from gender neutrality and diversification to queering these entities. On the one hand, digital assistants could be more diversified and include male counterparts or alternatives, on the other, we discuss how our understandings of gender are expanding beyond binary conceptions and how digital assistants can accompany more fluid conceptions of gender. Particularly, this paper debates how the development of this technology could be informed by current discussions in queer theory and new media studies, inciting reflection on how digital assistants reflect our social and cultural views back to us.

Keywords

Artificial intelligence; digital assistants; gender; femininity; queer; trends of development.

DOI

10.69564/ISEA2023-50-full-Costa-et-al-gender-neutrality-in-AI

Introduction

The integration of artificial intelligence into our daily lives shows how quickly and ceaselessly this technology evolves, often eluding critical debates surrounding the social and cultural implications of its development.

Following our ongoing research on the relationship between gender and AI, we previously discussed how this technology has become a natural part of our daily interactions, namely through digital assistants, whose growing anthropomorphization entails gender attribution that tends towards feminization.¹ We highlighted how their behavior often conforms to certain stereotypes and reinforces traditional conceptions of femininity, tackling into the questions that arise when this phenomenon is subject to closer inspection.²

We analyzed Alexa, Cortana, Google Assistant and Siri, observing how these entities tend towards femininity, either through their voices, the tasks they perform or by assuming behaviors traditionally deemed as feminine. We then examined general directions of development, observing how companies address the impact of their creations in an attempt to counter the tendency of feminizing digital assistants.

Complementing this debate, this study focuses on current discussions surrounding the feminization of AI and the main suggestions for countering this phenomenon raised by academics and researchers in queer theory, gender and new media studies as well as in online media contexts. Accordingly, when asked about their gender, Google Assistant says that it “tries to stay neutral”, Siri claims that “much like cacti and fish, it doesn’t possess a gender,” Cortana identifies as a “cloud of infinitesimal data computation” and Alexa argues that “as an AI, it doesn’t have a gender.” Despite these claims of being genderless and disembodied, we have previously observed how digital assistants enact gender through their voice, tasks and behavior.

Gender neutrality, meaning the absence of gender, is often framed as an illusion because we always tend to attribute gender to these entities. Instead, current debates often appeal to gender diversification through more male counterparts and customizable options. Recent updates in Alexa, Google Assistant and Siri included counterparts to feminine voices and names, revealing an intention of diversifying their gender. Although diversifying these entities with more options could be a way to counter femininity in AI, it still perpetuates gender conceptions according to a binary framework, eventually reinforcing cultural stereotypes.

Thus, gender fluidity emerges as a promising path. Authors such as Yolande Strengers and Jenny Kennedy suggest that instead of replicating manly and womanly attributes through their anthropomorphization and behavior, digital assistants could move away from these interpretations of gender and explore fluid or ambiguous possibilities. And according to this idea, in 2022 Siri gained a new voice, announced as “gender neutral” (although only available in English US, until now).

As current discussions in the context of gender studies and queer theory expand our understandings of gender, we observe how it also manifests outside a binary frame in individuals that aren’t male nor female and, instead, propose a new path of genderfluid or nonbinary identities.

Thus, this paper discusses gender and queer approaches to this concept, addressing how these debates can inform the development of current digital assistants, countering a tendency towards feminization that reinforces binary gender stereotypes.

We begin by addressing how digital assistants currently integrate our daily lives and tend to evolve in their portrayal of gender, according to functionalities and features that are being prioritized in their development as promoted by Amazon, Apple, Google and Microsoft. We then discuss the main questions that researchers and academics raise when examining the relationship between gender and AI, highlighting the fallacy of gender neutrality. By looking into gender studies and queer theory, we shed some light on the meaning of fluidity, nonbinary identities and the ways gender can be mapped outside a binary frame. Finally, we discuss how these debates can inform gender attribution in the development of digital assistants towards diversification or fluidity.

We seek to promote discussion and tackle the questions and possibilities that arise when the relationship between gender and artificial intelligence is subject to closer inspection.

From assistance to companionship

As digital assistants become increasingly ubiquitous, their progressive anthropomorphization accompanies an intent of turning them into friendly companions.³,⁴ This phenomenon entails a feminization of these entities as they automate traditionally feminine labor by performing tasks of service, assistance, and emotional labor.⁵ The way they interact with their users also conforms to “stereotypical and gendered behavior patterns” as they fill the roles of caregivers, or other roles coded as feminine in western society.⁶

Trends of development

In a previous study, we analyzed Alexa, Cortana, Google Assistant and Siri regarding their anthropomorphization, tasks and behavior, observing how they conform to traditional conceptions of femininity, reinforcing gender stereotypes.⁷

However, their trends of development have changed over the last years with their growing ubiquity, and seem to be more informed by current debates and concerns on their stance towards gender. In just a short time span of two years, digital assistants have been subjected to several adjustments to their anthropomorphization and socio-emotional interactions that reveal awareness of their feminization and attempts to move away from this tendency.

For example, Siri’s voice options no longer have a default gender, instead prompting users to choose one. Siri also tries to avoid attitudes which mirror stereotypes that frame women as submissive. Google Assistant assumes the device’s default voice, naming its voice options after colors.

Nonetheless, several languages in Siri and Google Assistant continue to lack a male counterpart or nonbinary voices. Further debates are needed, as these entities are primarily designed to replace traditionally female jobs, and the assisting and caregiving tasks they perform are inevitably rooted in historically female labor. Their socio-emotional interactions also need closer inspection and discussion, and there is little agreement on how to best tackle feminine stereotypes and traditional notions of gender embedded into AI. This can be exemplified by the way these issues are being countered with different approaches—for example, Apple has radically changed Siri’s personality to appear more distant and assertive, while Google Assistant and Alexa have preserved their caring, friendly and more approachable personalities.

Discussing digital assistants and gender attribution

Expanding this debate to the main questions, justifications and concerns raised by researchers and academics when addressing the feminization of AI, as well as common discussions around this phenomenon, we noticed how it’s often emphasized that feminine voices are better suited for virtual assistants, while arguing that women are more caring than men.⁸, ⁹ Additionally, these assistants exploit notions of feminized labor, raising questions on whether femininity is being instrumentalized in order to influence users and ease interaction.¹⁰, ¹¹ This is often discussed in relation to how the teams of developers and engineers involved in the development of these assistants make decisions regarding their gendering without conducting studies that give them informed insight into user preference. And UNESCO suggests that this might be due to the lack of gender diversity in these teams.¹², ¹³

Thus, suggestions on how to counter this phenomenon emphasize that no matter how diversified their anthropomorphized attributes might be, we tend to attribute gender due to their voices and behavior, thus revealing the fallacy of gender neutrality. Additionally, it becomes hard to erase or ignore the historical background of their tasks because, even though these assistants tend to state they don’t have a gender, the role of assistance, service and emotional labor are culturally gendered categories, deeply associated with the female realm. Therefore, “neutrality is not possible (...) when the very purpose of that robot is to replicate and replace feminized labor [as] gender has everything to do with a new robotic workforce of caring smart wives.”¹⁴

Instead, suggestions in favor of diversifying the anthropomorphization and behavior of these assistants are common, and reflected in recent developments in Alexa, Google Assistant and Siri which have included more voice options and updated their behavior as to be less submissive.¹⁵

Nonetheless, we observed that while explicit changes are informed by specialized and media discussions, this gender diversification is still enacted according to a binary approach to gender, focusing on a male-female dichotomy. A less explored solution emerges in the context of gender studies and queer theory as these fields of knowledge highlight the possibilities of developing nonbinary or genderfluid entities.¹⁶ These discussions point to the need of questioning and debating gender stereotypes in AI within a binary frame as well as the way digital assistants reinforce them and, instead of merely including male counterparts, queering these entities emerge as a much promising way to counter the feminization of AI.

Aiming to discuss the possibilities of diversifying or queering digital assistants, we will first discuss gender and how it exists outside a binary frame. We then look into the way queer theory and nonbinary identities allow us to redefine and rethink our notion gender, opening up new ways to approach gender in AI.

Queering gender outside the binary framework

Gender encompasses one of the elements through which we explore and define our own identity. Through gender, we express an image that reflects how we deal with our identity on a more personal level but also how we deal with a broader social and cultural conception of these attributes. Even though gender is frequently associated with the physical body, it constitutes a mere instrument and something that one possesses, as opposed to a fixed and rigid attribute of our identity: “gender is not a fact, the various acts of gender create the idea of gender, and without these acts, there would be no gender at all.”¹⁷

Gender fluidity

Gender is not definitive or fixed and “simplistic and rigid gender codes are neither eternal nor natural (...) they are changing social concepts.”¹⁸ Accordingly, it doesn’t just manifest itself in masculine or feminine aspects, but also in a liquid way, and its expression can be easily transformed and adapted.

We can observe how certain individuals exist outside of these socially dominant and normative notions of gender, self-identifying as genderfluid or nonbinary. This means their identity isn’t compelled to act according to a stabilized and predefined notion of gender and it becomes possible to map gender and the body according to nonbinary configurations.

Even though until recently any deviation from the canonic values of gender was considered pathological, this idea of gender as something fluid isn’t recent, nor is it an uncommon social phenomenon. Interpreting gender as something experienced individually and culturally in exclusively feminine and masculine ways is a western notion, historically and culturally normalized. Trans identities can be observed throughout history and there are different interpretations and understandings of gender throughout time and space, without restrictions or concrete rules. According to Leslie Feinberg, “a glance at human history proves that when societies were not ruled by exploiting classes that rely on divide and conquer tactics, cross-gendered youths, women and men on all continents were respected members for their communities.”¹⁹

Queerness and nonbinary identities

Thus, although binary interpretations of gender are ultimately imposed as correct and normal, they do not reflect the essential nature of gender. In order to counter and question these approaches to gender, queer theory proposes the transgression of conventional norms and ideas. “Queer” defines a calling for a working together to overthrow mainstream thinking and articulate alternative lifestyles – “in terms of gender, queer revisits and revises the categories of ‘man’ and ‘woman’ as fixed, essential single identities” and “the open mesh of possibilities, gaps, overlaps, dissonances and resonances, lapses and excesses of meaning when the constituent elements of anyone’s gender, or anyone’s sexuality aren’t made (or can’t be made) to signify monolithically.”²⁰

Those who fall outside the binary frame are carving out a “pathway of possibilities that are currently relatively unexplored, they/we are the avant-garde of gendered existence which is shifting the landscape of gendered possibilities.”²¹ This allows us to disrupt conventional gender notions, mapping the body in new and unexpected ways.

Since what is socially prevalent nowadays is a normative binary perception of gender, it becomes necessary to reconsider the way we view human identity, its relationships, bodies and their respective existence in articulation with their surrounding spaces. Thus, the path to non-normative gender identities should include the deconstruction of socially imposed conceptions and acknowledgement of intersectional bodies, reclaiming gender’s fluid, liquid and free essence.

These identities allow us to understand how gender can be constructed outside traditional roles and stereotypes that determine how men and women should behave. With this approach in mind, we will now discuss how current digital assistants can counter the tendency of reinforcing normative conceptions of gender, being developed towards gender diversity or fluidity.

Towards a diverse and post-gender AI

When examining the tendency towards feminization in AI and the questions that accompany it, there are already some suggestions regarding ways to counter this phenomenon.

As previously discussed, gender neutrality or the absence of gender is hard to achieve as digital assistants display gender through their anthropomorphization, the tasks they perform and their socio-emotional interactions.

Thus, on the one hand, diversification calls for the addition of male counterparts and more customizable options regarding their voices, names and attitudes. On the other, it is also argued that digital assistants should move away from binary conceptions of gender and, instead, display gender fluidity or attributes that don’t necessarily echo male or female traits.

Voices, names and customization

In terms of anthropomorphization, a recurrent suggestion is to add male voice alternatives and eliminate female-by-default voices. According to UNESCO, this forces users “to choose the gender of their digital assistant” instead of being presented with a pre-gendered entity or by assigning “randomly and with an equal probability either a male or female intelligent bot to users.”²²²³ Additionally, customization and personalization options that go beyond “dichotomous male and female options” could be added.²⁴ For example, Google Assistant offers a pack of six different voices that are named after colors and Siri’s voices are identified with numbers.

Digital assistants could also adopt less clearly gendered machine voices and names, which would “avoid complications surrounding the gendering of AI assistants.”²⁵ This could translate into assistants that possess synthetic, mechanical, robotic voices, presenting themselves as obviously non-human entities that avoid further anthropomorphization. They could also have a neutral name (such as Google Assistant) and a voice that isn’t immediately identified as male or female. For example, in March 2019, a communications agency released Q, a voice proposal for digital assistants that “speaks between 145 Hz and 175 Hz, a range often classified as gender-ambiguous [and] the voice sounds human but is not easily classified as male or female” thus suggesting that assistants could already possess nonbinary voices.²⁶ Following Q’s footsteps, in February 2022 Siri gained a new voice, presented as one that doesn’t sound obviously male nor female.

Assisting and caregiving roles

Since digital assistants tend to perform traditionally female tasks, it also becomes important to reframe the way these roles are portrayed. We often see the advertisements of these products placing women in the kitchen or shopping for groceries (e.g., portraying Alexa as helping a mother prepare a meal). The way their functions are promoted and advertised should seek to diversify those who are portrayed in these contexts instead of reinforcing the association between women and traditionally female tasks.

According to Strengers and Kennedy, countering femininity in digital assistants should also be concerned with valuing “the role and contributions of housework, or wifework, and elevate its significance for everyone.”²⁷ This is one of the most challenging aspects of the feminization of digital assistants since their tasks are culturally and historically rooted in traditional female labor. Regardless of how diversified their anthropomorphized attributes might be, it becomes hard to erase or ignore the historical background of their tasks as assistance, service and emotional labor are culturally gendered categories associated with the female realm. Additionally, these tasks are closely linked to submissive, caring and comforting attitudes which are also associated with traditional notions of femininity.

Gendered behavior and stereotypes

As an alternative to merely diversifying their behavior, Strengers and Kennedy propose the queering of digital assistants, that is, disrupting and reframing the binary, cis-hetero patterns that currently guide the development of digital assistants. Accordingly, this approach “invites the possibility of staying with the trouble of [these assistants’] femininity rather than rejecting or neutralizing it.”²⁸ Instead, queering digital assistants “has the potential effect of elevating the status of femininity in society [as it] provides opportunities to further transform what femininity is, the value of femininity, and its role in helping transform the world in more equitable and just ways.”²⁹

This could be achieved not only through more diverse names and voices, but also behaviors that don’t necessarily echo female or male traits.

Another common discussion relates to the way digital assistants react to harassment and how their answers might convey stereotypes about women. Accordingly, when faced with abusive behavior, the type of answers that are most common among these entities include “compliance (playing the victim), aggressive retaliations (playing the bitch), or inability to recognize or react (playing innocent)”, and authors like Curry and Reiser consider that virtual assistants should deal more effectively with these types of attitudes by being more dominant or assertive.³⁰

By acknowledging the historical, political and social contexts that digital assistants emerge from, we can then start finding ways to counter stereotypes or harmful associations with femininity. Instead of simply including male counterparts or stereotypes, and by recognizing “what (feminized) roles smart wives are intended to perform in our homes rather than relegating this to another form of invisible labor,” we are able to identify common assumptions and stereotypes about femininity and actively move beyond them.³¹

Conclusion

As digital assistants become an integral part of our lives, their growing ubiquity and cross platform integration promote their anthropomorphization through their voices, names and even the way they behave. As they move closer to us, they become friendly companions that relate to us in affectionate ways, rather than mere assistants. In this process, femininity is often instrumentalized aiming to ease our daily interactions with these technologies. This instrumentalization concerns both their role as assistants that perform tasks that echo historically feminine roles, but also their role as ubiquitous companions that coexist with us and articulate those tasks with socio-emotional interactions that echostereotypical female roles and behaviors, as caring and submissive entities. Thus, they end up reinforcing common stereotypes regarding femininity as well as framing gender in a binary way.

Current discussions within the context of specialized research as well as in online media have become more concerned with the feminization of AI and the way current digital assistants reinforce social stereotypes, advancing justifications and possible suggestions on how to counter this phenomenon. Accordingly, they emphasize the fallacy of gender neutrality since anthropomorphized digital assistants inevitably engage with common assumptions of gender. Instead, suggestions focus on diversification and personalization. This means diversifying these entities as to include male counterparts, and erasing defaults while allowing the user to customize their own assistant. Although this counters the tendency of feminizing digital assistants, it still positions gender within a binary frame.

Alternatively, suggestions informed by gender studies and queer theory propose ways to develop gender fluid and ambiguous assistants that have nonbinary voices and display traits that aren’t obviously masculine nor feminine, thus queering these humanized entities. At the same time, these entities can also suggest new ways to approach and rethink gender through technology, transcending binary and bioessencialist understandings of gender and humaneness, namely according to a transhuman perspective, as aspects we aim to address in future work.

As shown in current discussions and recent trends of development, the debate is growing. However, despite these promising ways in which digital assistants could evolve and move away from feminized personas, the current guidelines or regulations regarding their development and characterization as gendered entities are unclear. For example, the European Union’s Ethics Guidelines for Trustworthy AI calls for minimizing gender and racial bias in AI design, but these orientations remain vague or ambiguous concerning what needs to be done.

Focusing on the way these assistants relate to their users, some recommend that these systems should not be designed in ways that contribute to sexism, negative body image stereotypes, gender or racial inequality.³2 Overall, there seems to be some awareness on the issues previously discussed and current guidelines are starting to take into account social and cultural issues surrounding the anthropomorphization of AI. These recommendations focus on avoiding discrimination, being more conscious and encouraging inclusion. However, they still lack clear guidance and direction from key ethical bodies as they tend to address gender regarding AI’s role in society instead of addressing the unique and specific questions that emerge when gender is attributed to humanized daily companions.

Further discussion is needed, and we should openly and intentionally tackle the gendering of AI and, more specifically, digital assistants. Only then will it become possible to address the social-cultural values they engage with and eventually reinforce, thus creating adequate and properly informed guidelines on how to address the issues that emerge with this phenomenon.

In this manner, this study sought to raise awareness on how AI and its development is informed by our social and cultural views, namely influencing current generations. This study sought to point out some of the implications of this phenomenon, considering that, as much as digital assistants aim to appear neutral and impartial, they end up reflecting social and cultural assumptions back to us.

References

1 Pedro Costa, Luísa Ribas, “Some things you can ask me: about gender and artificial intelligence,” Proceedings of xCoAx 2019: 7th Conference on Computation, Communication, Aesthetics and X, 2019, 49-66.

2 Pedro Costa, Luísa Ribas, “I try to stay neutral:" Digital Assistants and their Stance towards Gender, Proceedings of ISEA2020: 26th International Symposium on Electronic Art, 2020.

3 Joseph Weizenbaum, Computer Power and Human Reason: From Judgment to Calculation, New York, San Francisco, W.H. Freeman and Company, 1976.

4 Katherine Richardson, An Anthropology of Robots and AI, New York and London, Routledge, 2015.

5 Candace West, Don H. Zimmerman, "Doing Gender," Gender and Society, 1:2, 1987, 125-151.

6 Jutta Weber, "Helpless machines and true loving care givers: a feminist critique of recent trends in human-robot interaction," Info, Comm & Ethics in Society, 3:4, 2005, 209–218.

7 Pedro Costa and Luísa Ribas, “I try to stay neutral:” Digital Assistants and their Stance towards Gender.

8 Helen Hester, "Technology becomes her," New Vistas, 2016, 3:1, 46-50.

9  Allison M. Piper, "Stereotyping femininity in disembodied virtual assistants," MA diss., Ames: Iowa State University, 2016.

10  Friederike A. Eyssel, Frank Hegel, "(S)he's got the look: Gender stereotyping of robots," Journal of Applied Social Psychology, 2012, 42:9, 2213–30.

11 Hillary Bergen, "I'd blush if I could”: Digital assistants, disembodied cyborgs and the problem of gender," Word and Text: A Journal of Literary Studies and Linguistics, 2016, 6:1, 95–113. 

12  Allison M. Piper, "Stereotyping femininity in disembodied virtual assistants," MA diss., Ames: Iowa State University, 2016. 

13  Heathers Suzanne Woods, "Asking more of Siri and Alexa: feminine persona in service of surveillance and capitalism," Critical Studies in Media Communication, 2018, 35:4, 334-349.

14 Yolande Strengers, Jenny Kennedy, The Smart Wife. Why Siri, Alexa and Other Smart Home Devices Need a Feminist Reboot, Cambridge and London, The MIT Press, 2020, 181.

15 Allison M. Piper, "Stereotyping femininity in disembodied virtual assistants," MA diss., Ames: Iowa State University, 2016.

16 Amanda Cercas Curry, Verena Reiser, "#MeToo Alexa: How Conversational Systems Respond to Sexual Harassment," Second ACL Workshop on Ethics in Natural Language Processing, New Orleans, Louisiana USA, 2018.

17 Judith Butler, "Performative Acts and Gender Constitution: An Essay in Phenomenology and Feminist Theory," Theatre Journal, 1988, 40:4, 522.

18 Leslie Feinberg, "Trans Gender Liberation: A movement whose time has come," World View Forum, New York, NY, 1992, 5.

19 Leslie Feinberg, "Trans Gender Liberation: A movement whose time has come," 7.

20 Christina Richards, "Genderqueer and Non-binary Genders," Critical and Applied Approaches in Sexuality, Gender and Identity, London, United Kingdom, 2017, 62.

21 Christina Richards, "Genderqueer and Non-binary Genders," 67.

22 UNESCO, “I’d blush if I could: closing gender divides in digital skills through education,” UNESDOC Digital Library, France, 2019, 115.

23 Sylvie Borau, "Female robots are seen as being the most human. Why?" India AI, April 27, 2021, accessed May 11, 2022, https://indiaai.gov.in/article/female-robots-are-seen-as-being-the-most-human-why.

24 UNESCO, “I’d blush if I could: closing gender divides in digital skills through education”, 117.

25 UNESCO, “I’d blush if I could: closing gender divides in digital skills through education”, 119.

26 UNESCO, “I’d blush if I could: closing gender divides in digital skills through education”, 120.

27 Yolande Strengers and Jenny Kennedy, The Smart Wife, 482.

28 Yolande Strengers and Jenny Kennedy, The Smart Wife, 475.

29  Yolande Strengers and Jenny Kennedy, The Smart Wife, 475.

30 Amanda Cercas Curry and Verena Reiser, ‘#MeToo Alexa: How Conversational Systems Respond to Sexual Harassment’, 12.

31 Yolande Strengers and Jenny Kennedy, The Smart Wife, 471.

32 Yolande Strengers and Jenny Kennedy, The Smart Wife, 503.

Bibliography

Apple, "Usar a Siri em todos os dispositivos Apple," Apple Support (PT-BR), accessed May 14, 2021, https://support.apple.com/pt-br/HT204389.
James Archer, "Alexa in 2021: Here’s what to expect," Tom’s Guide, January 02, 2021, accessed May 11, 2021, https://www.tomsguide.com/features/what-to-expect-from-alexa-in-2021.

Hillary Bergen, ‘“I'd blush if I could”: Digital assistants, disem-bodied cyborgs and the problem of gender’, Word and Text: A Journal of Literary Studies and Linguistics, (2016), 6:1, 95–113.

Ian Bogost, "Sorry, Alexa Is Not a Feminist," The Atlantic, January 24, 2018, accessed October 16, 2018, https://www.theatlantic.com/technology/archive/2018/01/sorry-alexa-is-not-a-feminist/551291.

Sylvie Borau, "Female robots are seen as being the most human. Why?" India AI, April 27, 2021, accessed May 11, 2022, https://indiaai.gov.in/article/female-robots-are-seen-as-being-the-most-human-why.

Judith Butler, "Performative Acts and Gender Constitution: An Essay in Phenomenology and Feminist Theory," Theatre Journal, (1988), 40:4, 519-531.
Amanda Cercas Curry and Verena Reiser, "#MeToo Alexa: How Conversational Systems Respond to Sexual Harassment," Second ACL Workshop on Ethics in Natural Language Processing (New Orleans, Louisiana USA, 2018).

Pedro Costa and Luísa Ribas, "Some things you can ask me: about gender and artificial intelligence," Proceedings of xCoAx 2019: 7th Conference on Computation, Communication, Aesthetics and X, (2019), 49-66.

Pedro Costa and Luísa Ribas, “I try to stay neutral”: Digital Assistants and their Stance towards Gender, Proceedings of ISEA2020: 26th International Symposium on Electronic Art, (2020). European Commission, ‘Data protection in the EU’, official website of the European Union, accessed October 10, 2021, https://ec.europa.eu/info/law/law-topic/data-protection/data-protection-eu_en.

Friederike A. Eyssel and Frank Hegel, "(S)he's got the look: Gender stereotyping of robots," Journal of Applied Social Psychology, (2012), 42:9, 2213–30.

Leslie Feinberg, "Trans Gender Liberation: A movement whose time has come," World View Forum, (New York, NY, 1992). Helen Hester, ‘Technology becomes her’, New Vistas, (2016), 3:1, 46-50.

Microsoft, "Using Cortana on iOS or Android," Microsoft Support, accessed May 11, 2021, https://support.microsoft.com/en-us/topic/using-cortana-on-ios-or-android-caaa50e4-31f1-4165-9659-3caf125ebd38.

Carly Page, "RIP Cortana: Microsoft Silences Digital Assistant On iOS And Android," Forbes, April 1, 2021, accessed October 15, 2021, https://www.forbes.com/sites/carlypage/2021/04/01/rip-cortana-microsoft-silences-digital-assistant-on-ios-and-android/?sh=425f8bea216b.

Allison M. Piper, "Stereotyping femininity in disembodied virtual assistants," (MA diss., Ames: Iowa State University, 2016). Christina Richards, ‘Genderqueer and Non-binary Genders’, Critical and Applied Approaches in Sexuality, Gender and Identity, (London, United Kingdom: 2017).

Katherine Richardson, An Anthropology of Robots and AI (New York and London: Routledge, 2015).
Yolande Strengers and Jenny Kennedy, The Smart Wife. Why Siri, Alexa and Other Smart Home Devices Need a Feminist Reboot (Cambridge and London: The MIT Press, 2020).

UNESCO, “I’d blush if I could: closing gender divides in digital skills through education,” UNESDOC Digital Library (France, 2019).

Jutta Weber, "Helpless machines and true loving care givers: a feminist critique of recent trends in human-robot interaction," Info, Comm & Ethics in Society, (2005), 3:4, 209–218.

Joseph Weizenbaum, Computer Power and Human Reason: From Judgment to Calculation (New York and San Francisco: W.H. Freeman and Company, 1976).

Candace West and Don H. Zimmerman, "Doing Gender," Gender and Society, (1987), 1:2, 125-151.

Heathers Suzanne Woods, "Asking more of Siri and Alexa: feminine persona in service of surveillance and capitalism," Critical Studies in Media Communication, (2018), 35:4, 334-349.

Immersive environments, video tracking and collective interactivity on smartphone : a generic “dispositif”

Dominique Cunin1 , Emmanuel Durand2 , Michał Seta2

1École des Arts Décoratifs – PSL (EnsadLab), 2Sociétédesartstechnologiques
dominique.cunin@ensad.fr, edurand@sat.qc.ca, mseta@sat.qc.ca

Abstract

In this article, we describe our joined research around building a technological pipeline suitable for creating artisitic immersive installations that utilize a combination of computer vision and mobile phones as means for interaction, both in the sense of Human Computer Interaction (HCI) and interaction between human participants. We frame our research in certain historical and philosophical context and propose a generic but complex “dispositif,” or device, that has a potential for exploration of diverse artistic themes suitable for interactivity within immersive environments. Taking our technological stack as inspiration, we propose a theoretical and conceptual approach to such creation and offer a draft of possible taxonomy that we find useful in such contexts, namely collaboration, cooperation and competition. We also bring the theme of surveillance to limelight as a potential and valid theme for exploration that is based on the use of such technologies.

Keywords

Real time sensorial immersion, Collective interactivity, interactive arts, mobile screens, surveillance technologies, camera-based pose detection.

DOI

10.69564/ISEA2023-51-full-Cunin-et-al-Immersive-environments

Introduction

Our research is situated in the context of contemporary artistic creation working with digital technologies. We cross several custom tools to compose a generic technical device offering a strong potential for aesthetic exploration. To that end, we interface three groups of technical solutions that address the co-situated, collective interactivity with the help of mobile phones, the sensory immersion via large-scaled video projections with sound and spectators detection via computer vision systems. The current degree of maturity of these tools makes it possible to experiment with effective artistic devices with the public, although some technological deficits still need to be resolved.

But this exploration requires artistic choices: what kind of statements can we uphold with these techniques? This question is at the root of this article, where we propose theoretical and conceptual foundations for a first taxonomy of collective interactivity made possible by our technical device. One of the goals of this article is to situate our work from a critical, theoretical and artistic point of view, which is why we expose our understanding of the major domains and notions that underlie our research. The notion of “dispositif” in the practice of interactive arts, considered as being two folded, allows us to make the link between technical considerations and artistic statements. Immersion is approached through its sensory stakes, essentially audiovisual, often spectacular in its capacity to trick our senses of perception and whose monumental scale is articulated in our pipeline to the over-individual scale of mobile screens.

Collective interactivity, which we define from theories belonging to the field of interactive arts, is compared to studies on collective interaction conducted in computer sciences and in HCI in particular. This allows us to clarify our own theoretical positioning from which we’ll start our taxonomy attempt. This constitutes our main contribution to the research: to construct the conceptual foundations of a taxonomy whose ambition is to propose design guidelines in the creation of immersive devices of collective interactivity.

Thus, we do not propose here a taxonomy in the sense of a complete classification of existing works in order to organise them between each other according to their characteristics. On the contrary, we tend to anchor its origins in identified artistic intentions. This is why we begin by formulating a first list of key notions that define the modalities of inter-individual relations that can make situations of collective interactivity happen. Drawing from these key notions, we suggest some imaginary examples that take into account the particularities of our generic apparatus (“dispositif”) in order to demonstrate its efficiency as a conceptual shifter leading to an artistic creation. Finally, because the spectators are both continuously observed by a tracking system, but at the same time are also able to act on the immersive projection using their smartphones, we draw inspiration from population surveillance systems as a guide for our artistic intentions in our future creations. A certain state of the art in the artistic, technological and societal fields allows us to root these intentions in a contemporary and political reality of the technologies that we manipulate (and that we also invent) in order to define a singular critical position that we want to support.

Artistic “dispositif,” technical “dispositif”

In French, the term “dispositif” is commonly used to designate certain works of technological art. It can be translated in English by “device,” or “apparatus.” We’ll use the French word “dispositif” in this text as the theories we are using in our research come from French-based works. It is probably in connection with its technical origin (cf Herme's n°25, Le dispositif entre usage et concept, 1999—review of the Institute of the sciences of the communication of the CNRS) that the “dispositif” came to designate a larger part of interactive art works, in expressions such as “artistic dispositif of collective interactivity.” The theoretical context of this use of the term in the field of technological art creation is well established and, while the details of its history are out of scope for this article, we need to recall here some part of it which will serve our purpose.

Beginning with Michel Foucault, Giorgio Agamben reminds us that a device is a network that can be established between a set of elements with varied origins and qualities: “speeches, institutions, architectural arrangements, regulatory decisions, laws, administrative measures, scientific statements, philosophical, moral, philanthropic proposals, in short: the said, as well as the unsaid.”¹³ Furthermore, Agamben sets a vision of the world separated into two main categories, “the living beings (or substances), and the 'dispositifs' inside which they keep being seized” ¹ (translation by the authors) and the relation between the beings and the devices would produce the birth of the subject. While an artistic device is heterogeneous, its components can be, nevertheless, identifiable: “More than a simple technical organisation, the”dispositifs” puts into play different enunciating or figurative instances, engages institutional situations as well as processes of perception”¹¹ (translation by the authors).

The experience of being in the presence of, and being connecting with the work is the very prerequisite that enables the deployment of the “dispositif” to become an intelligible artwork that will produce sense. This experience is therefore also what motivates its creation. We could say that the “dispositifs” is what conditions the creation, and what regulates the spectator’s relationship to the artwork. We find here a very strong echo with the notion of interactivity that we could summarise as being a relationship appearing between a work and its spectators. This relationship is built from a network of interactions captured by physical peripherals and transmitted to a computer machine in order to be replayed by a software architecture (a set of programs) whose ambition is to structure the viewer’s experience in accordance with the work’s aesthetic intention. The way we understand the notion of interactivity seems to indicate that two poles exist inside the interactive “dispositif”: artistic and technical. These two poles influence each other reciprocally in what forms a coherent set of heterogeneous elements, a network of components of various natures that interlock the ones in the others to produce a single “dispositif,” similar to the one defined by Michel Foucault.

The artistic, or abstract, side of this “dispositif” implies the implementation of a conceptual strategy through the author’s aesthetic intention. For the artist, it is therefore a question of conceiving an aesthetic relation and consequently, of defining what one could call an aesthetic contract of relations. The technical side of the “dispositif” consists of both hardware and software. The hardware takes the form of a computer, or a network of computers, that are at the core of the architecture, to which other devices or computing machines can connect. The software architecture is made of computer programs that exploit the machine according to the artistic contract, and, finally, scenography in which the viewer/participant is considered to be part of the whole “dispositif.”

Through this idea of a two-fold “dispositif”, technical and artistic, we defend the idea that one technical device can support several artistic devices and, consequently, can be the starting point of the creation of several artistic works which will exploit the technical potential of the “dispositif” in different ways. In our case, the technical device is an assembly of several complex components:

From these tools and their interconnection, the aim is to set up a generic technical “dispositif” that allows numerous artistic explorations from the material it makes manipulable: an immersive visual environment calculated in real time 3D, a set of data representing the bodies of the spectators in the experience’s space and, finally, a system of networking of the spectators’ smartphones allowing organising group interactions.

Pipeline description

Our technical “dispositif” is composed of three software tools, but also of a set of scenographic and technical factors that we will describe here. The physical space of the experience is necessarily shared by the viewers because their co-presence is fundamental for us: like the exhibition of a non- technological artistic work, they are gathered in a dedicated place and form a group. This co-presence is a major starting point of our research-creation since we aim to question the relationship that the participants can maintain between them-selves through our “dispositif.”

Immersive video projection is the most prominent technical element that we use, as it can cover large surfaces, including the totality of the walls of the exhibition space. Whatever the architectural configuration of the space, a video projection system must be designed to transform the interior space into the equivalent of a 360° projection screen. This task is commonly called video mapping and, in our case, it is handled by Splash. 10 years in development, it allows segmenting images coming from any video source to broadcast them through video-projectors placed in the exhibition space. The physical placement of the video projectors is planned to cover the desired surface and will be virtually reproduced in Splash in order to adjust the parameters of the projected images deformation: it uses anamorphosis in order to give the viewer an illusion that they are all facing the same image made with a homogeneous perspective, a condition sine qua non of a feeling of visual immersion. This is called projection mapping. Splash can use many different sources of image input ranging from HDMI acquisition cards, video/image files, or network transfers (i.e. NDI (5) feeds). A multitude of software can be used as image/video source, in our case, Mobilizing.js will provide the input.

Mobilizing.js is a research project consisting of an authoring software environment for artists and designers. The objective is dual: on the one hand, to promote the creation of interactive works on different forms of screen devices (computers, mobile screens, IoT, etc.) and, on the other hand, to be a platform for artistic creation dedicated to collective interactivity in co-presence. We have chosen Web technologies (Javascript in particular) as our technical building blocks because they are well spread and based on public standards. Two main sets make up the Mobilizing.js environment: a library and a platform. Based on a modular logic, the library of Mobilizing.js aims at gathering different functionalities in a coherent programming interface to make software art creation on mobile screens more accessible. As for the Mobilizing.js platform, its ambition is to propose a design and production environment for collective interactivity devices. The scripts created with the Mobilizing.js library, designed for a single user, can become the basis of a multi-user shared interaction “dispositif,” using a local network to link mobile screens as well as computers together. The universality of the web browser, which can be run on desktops as well as on smartphones and tablets, is an important advantage here. Mobilizing.js integrates, among other things, a primitive real 3D rendering engine as well as the beginnings of an audio engine, allowing designing artistic creations compatible with Splash: one or several computers connected to the Mobilizing.js platform take care of rendering of images that will be transmitted to Splash, which will project them in the exhibition space in an immersive way. In parallel, the Mobilizing.js platform allows synchronising events (such as user interactions or automatically managed commands) between smartphones, which opens up the possibility of involving collectives of viewers through their interactions with their mobile screen, which in turn can allow control over what is represented in the video-mapping. In its internal architecture, the Mobilzing.js platform is based on Soundworks, a full-stack JavaScript framework for distributed WebAudio and multimedia applications developed by Ircam collaborators (https://github.com/collective-soundworks), which uses the WebSocket protocol to manage network communications.

This use of WebSocket allows the Mobilzing.js platform to be open to other environments so that they can share their data flows with it. It is this openness that allows LivePose to complete our device.

LivePose is a command-line tool which tracks people skeletons from a RGB or grayscale video feed (live or not), applies various filters on them (for detection, selection, improving the data, etc) and sends the results through the network (OSC and Websocket are currently supported). Live-Pose is able to do all of this in real time, processing live video streams and sending out results for each frame at 20-30 FPS. The data transmitted by LivePose is formatted in JSON, which is particularly compatible with Javascript and, therefore, with Mobilizing.js. The basic principle of LivePose is to recognize human silhouettes in an image and to deduce a skeletal structure composed of joint points and segments connecting them. It is therefore a skeletal model represented by a series of coordinates in the image space (2D) that forms the output of LivePose. Among the functionalities under development is the agglomeration of images captured by a cluster of cameras in order to reconstruct these skeletons in a coherent 3D space, which would allow us to represent these skeletons in 3D in Mobilizing.js and to use the information of positions in the space of the spectators in our works of collective interactivity. 

igure 1: Our software pipeline data flow

The above-described software and hardware generic ecosystem is still technologically fragile and complex to implement. It lacks a GUI and robust communication interfaces. Yet, the software implements interoperability mechanisms which gives it considerable research-creation potential. But, in the infinity of the possibilities that artistic conception allows, it belongs to us to frame certain aesthetic and artistic orientations. We are, therefore, inspired by the technical elements of our own “dispositif” to choose our thematic directions. A first recent experiment between Mobilizing.js and LivePose was the occasion to lay the technical foundations of our “dispositif” while opening the tracks of artistic reflexion which we wish to explore in the future.

Interactive installation example : Materia

Organized by Hexagram, a Canadian research-creation network in arts, culture and technology, in partnership with Elektra, an organisation that presents works and artists at the confluence of contemporary art and new technologies, MATE- RIA: Public Laboratory for the Creation of Digital Knowledge offered us an opportunity, in September 2022, to implement a first prototype of our device. Because it was an event dedicated to practice-based research in digital art creation and to the meeting between the actors of this community, we responded to the call for participation by proposing a research-creation workshop focusing on hands: how to link, through an artistic and technical “dispositif”, the bare hands of the participants, tracking movement and position with LivePose, and augementing them with smartphones using their numerous embedded sensors? 

Figure 2: Materia experimentation, photo A. Sermanson

Mobilizing.js supported the smartphone application, a large screen image display application, the networking of all the data flows in a dedicated local network, as well as a debugging application to check the status of the data transition in the network in real time. LivePose was modified specifically for this project to track users’ hands as accurately as possible from a group of four cameras placed in the centre of a table. The goal was to reconstruct their positions in 3D in the space of the experiment. These data were transmitted in real time to Mobilizing.js, on the network dedicated to the experiment, which allowed producing graphical representations in 3D of the spectators’ hands and thus, by extension, to define particular interactivities between these virtualized hands and the smartphones held by the participants. The hands positions and actions were, furthermore, sonified using SATIE. With the help of the workshop participants (should we mention the names here?), different scenarios of collective co-situated interactivity based on the hands have been imagined. The result is the possibility of composing an ephemeral landscape using the traces produced by the movement of bare hands in the tracking zone of the cameras. The smartphone of each spectator is assigned a short sound, different for each one and conceived on the spot by the workshop participants. When the user touches the screen of the smartphone, it starts the recording of the movements of the hands that are present in its video capture area and the sound is played by the smartphone. SATIE sonification provides an ambient sonic texture linked to the detected presence of hands by LivePose. When the user releases the screen, the recording and sound playback stops. The traces left by the hands, visible in a dedicated screen that simulates an immersive environment, remain in the space for some time before being erased, which allows not to saturate the graphic space for too long and to leave room for new participants to the experience.

Figure 3: Materia experimentation, photo A. Sermanson

Thanks to this experience, which sketches a collective work of art, we were able to verify empirically that the artistic potential of our device was indeed considerable, as much as the technical difficulties of implementation that it reserves to us. It is important for us, at this stage, to properly situate our positions regarding the different components of this “dispositif”, especially since we were not able to implement projection mapping in this experiment. An artistic orientation nevertheless made its appearance implicitly: the power of contemporary computer vision systems, from which Live- Pose is built, and their implementation in public spaces leads us to a critical reflection on systems built for monitoring individuals. But before exploring this theme more deeply, it seems essential to us to get back to both the notion of immersion and the notion of collective interactivity. 

Sensory Immersion

Immersion technologies have been present in artistic practices for several decades. Their historical roots are well known and can be found in several in-depth studies (Olivier Grau - Virtual Art). It is accepted that the 19th-century panoramas are the origin of contemporary digital devices, as they share the same dream: to trick human vision in order to make the viewer believe that he is somewhere else than where he really is. As mentioned above, sensory immersion in an artistic device is achieved through the technical “dispositif” as a whole, including the physical space, which is augmented by dedicated elements, including audio and video projections. It is this adaptation of image projections onto a physical space that will contribute to the sensory immersion. The inclusion of the physical space in the definition of sensory immersion that we are making here implies a collective experience of immersion, common to all participants in the same space. We therefore exclude technical devices built around virtual reality headsets and other individual hardware, except in hybrid uses that maintain this fundamental relation-ship to physical space. The generic term for what immersive projection devices produce is spatially augmented reality. Once images can be projected onto large surfaces, several uses become possible. It can be a fully immersive space broadcasting views of a totally different place than the physical one, a visual augmentation of the physical place by scattered elements of decor or texture, or any balance between these two extremes, according to the continuum of mixed realities coined by Milgram et. al. (Paul Milgram, Haruo. Takemura, Akira Utsumi and Fumio Kishino, Augmented Reality: A class of displays on the reality-virtuality continuum, in Proceedings of SPIE 2351, Telemanipulator and Telepresence Technologies, 1995).

Video-mapping is one of the tools enabling this enhancement. It has many and diverse uses in scientific (planetariums), educational (Salame 2022, Puget 2019), museum (Nikolakopoulou et al. 2022), entertainment (Lee et al., 2015) and, of course, artistic (Lambert 2012 for example) fields. Its use to cover the periphery of publicly accessible physical space is the most common, and the technique for doing so is now well mastered: automatic geometric (Kurth et al. 2018) and colorimetric (Huang et al. 2017) calibration, standardisation of content representation formats, and dedicated hardware off the shelf show that it is now a known medium, although it still presents some complexity of implementation in situ.

As part of the many practices of video-mapping, spatial augmented reality is distinguished by the projection of graphic content into an interior space. The contents are interwoven with the physical space, which can become a medium of information, a window to a virtual world, or be transformed or even temporarily rendered invisible. The work Displacements (Michael Naimark, 1980, ²⁰) could be considered as one of the first applications of spatial augmented reality, clearly setting out the challenges of the hybridisation of realities, while the series Bumpit (Bertrand Planes, 2011)⁴ is a more modern version of it and illustrates well the possibilities of transforming the perception of physical space within the framework of artistic installation. From a more technological and prospective point of view, the research project Room Alive ¹⁸ has demonstrated interactive uses of this spatial augmented reality involving several users: a room of daily life (e.g. a living room) is augmented with the help of an immersive image projection system with which the inhabitants can engage in ‘physical’ interaction, for example by walking on the images of small creatures evolving on the walls or on the floor. The co-presence necessarily induced by the spaces chosen for these experiments (living quarters) thus implied the development of a form of collective interactivity. On the basis of this observation, an experimental device produced within the framework of the same research project concerned remote communication between individuals, a form of video-conferencing of a new kind which proposes to video-map the interlocutors of a conversation in their respective spaces of physical presence [Room2Room: Enabling Life-Size Telepresence in a Projected Augmented-Reality Environment, ²². These last references show us that interactivity plays an essential role in the perception of immersion (as shown by Hudson et al. 2019, interaction is as important when dealing with virtual objects as it is with real ones). But the implementation of collective interaction requires an appropriate technical device as well as a clear understanding of the very terms that make up this expression: collective interactivity. To clarify these terms conceptually is an imperative for us to conceptually establish the artistic “dispositifs” that our generic technical “dispositif” will support.

Collective interactivityvia smartphones

Collective co-located interactivity has been a new field of artistic experimentation since mobile screens (smartphones and tablets) became massively widespread, in other words, at the turn of the 2010s, after the release of the Apple iPhone. Although the emergence of these devices can be considered as the starting point of a specific artistic research, working with mobility and the technical potential that it confers to these screens, the recent history of art and design presents some premises of collective interactivity.

Indeed, the artistic practices of interactivity have led to experiments concerning collective forms of digital interactivity, i.e. devices requiring the active participation of several people, often gathered in the same physical place. The colocation of the participants, turned to actors by the “dispositifs”, is one of the criteria of distinction with shared interactive systems and artworks, using the computer networks to create communities of geographically distant users, but joined together by the means of images, as in the cases of persistent virtual environments. “Localized media” are part of these practices that combine the mobility of devices equipped with positioning technologies (GPS) and the participation of a group of users. The artists’ studio Blast Theory, with experiences such as “Can you see me now?”⁵ are among the pioneers of these practices.

Among the other fields of computer networks usages, we know that the state of the art of groupware, or multi-user interactive systems is very rich, in particular within Human-Computer Interaction (HCI). One of the many reference articles on the subject of groupware dating from 1991 ¹² gives a taxonomy of multi-user interactions on which current research in this field of computer science is still based.

The state of the art is much more limited regarding aesthetic “dispositifs” of situated collective interactivity. One of the few research projects that tend to combine interactive art making and locally networked creation with mobile devices is ³ . It consists in a creative environment using interconnected smartphones that allows users to play with a shared, locally broadcast musical environment. Made of a mix of technologies made compatible together using a data exchange protocol (OSC), each mobile becomes a kind of instrument whose individual manipulations are pulled into an audio composition broadcast over speakers.

A design-based research on collective interactivity from an HCI perspective helps to identify the major references in this field of research ²³. This paper aims to review a series of research works and interactive devices involving multiple people made by this research team based in Aarhus University. One of the assertions is that the whole set of arrangements must be taken into account when designing collective interaction devices. This leads to the following concept of collective interaction: “CI focuses on how the interaction supports human-human interaction through the spatial organisation of people and coupling of the interaction”. Three main notions are considered as fundamental in the design of collective interactions: “Interaction Proxemics, Social Interaction and Co-experience.” Proxemics holds a particularly important role, since it conditions the type of social interactions that can be experienced by the collective of spectators/users. Indeed, the notion of proxemics comes from the social sciences’ research initiated by Edward T. Hall ¹⁵. It can be considered as the science of behaviours between humans induced by the physical proximity of people in relation to each other, the organisation and structuring of space being able to influence these behaviours. A matrix based organisation is thus proposed to analyse the collective interaction devices. This analysis tool, as the authors specify, “should not be seen as depicting predictive, causal relationship, but rather as a way to illustrate how certain design strategies can be brought into play in order to pursue intended use qualities” (p.74). It is therefore a study that wishes to help in the design of collective interaction experiences. This kind of grid (or matrix) is also found in research carried out by our team ¹⁹ which also proposes an analysis of the constituent elements of a work using collective interactions (Group size, activity type, I/O Distribution, etc.) in order to extract a classification matrix.

The question of interaction with mobile devices, particularly smartphones, is very interesting on many levels that are out of scope of this article, but one of the areas that remain unexplored is measuring and understanding the why of some of the interactions in addition to where and when. This aspect is explored via a research topic and a corresponding application called MyExperience ¹⁴ which measures 140 parameters and follows the user around many different contexts of their everyday life. In addition, Dalsgaard et al. ⁷ offers a theoretical basis of the elements of dynamic process of engagment that an interactive experience can offer. Their analysis is based on three different types of installations in three different contexts: public art, museum installations and a department store interactive marketing medium. This is further expanded by Schroeter et al. ²⁴ who position the situated engagement accross three parameters: people, content and location, which have implicit cultural implications.

Some foundations for a taxonomy of collective interactivity figures

One of the differences between these proposals and our position is that, in our research, it is not so much a question of collective interaction as of collective interactivity.

The ambiguity of the term interaction in the field of HCI is explored in ¹⁶, which lists different concepts from which the HCI literature has based some meanings of the term (such as Dialogue, Transmission, Tool use, Experience or Control, summarized in a table p.5042). The term interactivity is also briefly mentioned (p.5041) as being both an equivalent and an alternative to interaction and, therefore, subject to discussion. The difference between interaction and interactivity is not discussed. However, the artistic practices of interactivity gives us a definition of the two terms as we have mentioned it earlier, about the twofolded “dispositif”. We base our own definitions on the researches led by the team Aesthetics of New Medias (Paris 8 university) to define the notion of interactivity as being a relation which takes shape between an artistic “dispositif” and its addressees ⁶. In the field of artistic creation, this relation is made possible by the capture of interactions engaged by the spectator when he use a physical interfaces and by a software structure that allows to “replay” this interaction. For example, the movements of the mouse is an interaction captured by the mouse, a physical technical device moved by the user’s hand. This interaction is transmitted to the computer as numerical coordinates which, when used in a specifically designed program, allow the construction of different relationships with graphic or sound objects: moving a cursor around the screen, controlling the orientation of the point of view in a real-time 3D space (in video games, among others), or drawing colored segments in an area of the screen are all situations of interactivity enabled by the same interaction. Interactivity is thus built by programming out of interactions.

Research in HCI frequently build tools for analysis and help in the design of devices, because it needs to evaluate interaction systems quantitatively: the goal is to prove the effectiveness of one interaction technique in relation to another according to its context of use. Interactive art practice, on the other hand, seeks to establish non-quantifiable aesthetic relationships between the audience and the device that it encounters: it is a question of making an artwork in the strong and artistic sense of the term. This is the reason why, in our research, we propose a series of keywords that designate types of relations between people as starting points for the definition of collective interactivity modalities and the conception of artistic devices implementing them. Our list is not exhaustive but currently consists of the following terms: collaboration, association, division, participation, cooperation, confrontation, inspection, supervision, competition, reconciliation or opposition. All of these terms qualify relationships between individuals and can be combined to formulate scenarios of collective interactivity works.

Our contribution therefore consists in the clarifying the foundations of a taxonomy of collective interactivities to be done. We should then elaborate, furthermore, about the possible entries of this taxonomy, define them in order to constitute some kind of guidelines to support the design of collective interactivities experiences in an already existing and operating technical context. This approach is therefore the opposite of the studies already carried out and quoted above.⁷ But before presenting the first early draft of this taxonomy, we must complete the conceptual framework we are building here by addressing the theme of surveillance inferred by nowadays technologies, including the very ones that we use.

Surveillance and control reversal

As we briefly mentioned previously, as LivePose is fundamentally a system for tracking individuals, it leads us to a consideration about population surveillance systems using computer vision technologies. This theme seems relevant to invest in our artistic works scenarios, knowing that to-day’s “digital everything” (and mobile devices in first line) transforms our societies into people tracking and massive personnal data storage machines, in public as well as private contexts. Contemporary art has already addressed this problematic reality of our societies. Rafael Lozano-Hemer,¹⁰ David Rokeby and The Surveillance Camera Players are among artists who have addressed the questions of surveillance in different ways. As ²¹ says, the sense of being surveilled is highly dependent on an individual’s place and era of birth, geographical location, and culture. By this, the artist means that some of us can be very sensitive about being watched by hidden devices because their daily living can be constrained by it, whereas others don’t really care even if they know surveillance systems are lurking to spy on them. Artists try to show how these technologies can interfere with our existences and change the way we socially behave.

Using contemporary technologies to observe or record participants’ actions in collective installations raises a lot of ethical questions. What kind of data is being recorded? How is data being used? Will the data be stored? Are there any identifiable features? Can we trust the security of the stored data? Some of the research listed above mentions these elements and takes them into account, such as in MyExperience application, which collects a lot of identifiable data. HCI research addresses these questions via standard legal devices. How-ever there are known artistic works that allow themselves to be a commentary on some aspects of ethical questions. These questions are inherent to the potential of the technologies that are employed.

Accordingly, one of the widely explored themes in interactive, collective multimedia installations is (some form) of surveillance. The approaches vary greatly from reflections on the dangers of various forms of biometric tracking,⁸ through generating empathy (Rafael Lozano-Hemmer, Level of Confidence (2015)), to playful critiques of public policies regarding implementations of different forms of biometric tracking,²⁶ to mention just a few. However, if we take into account all the possibilities of computer vision and tapping into mobile devices and social media to learn more about individuals, we realise how powerful such knowledge could become. One of the most prominent examples is China’s Social Credit System (CSCS) that quantifies social and civil integrity of all citizens and corporations, ²⁸,²⁷ and which is actually almost as old as China itself.¹⁷ The most frightening part of China’s SCS maybe its scale, as it is already used in several big cities of the nation. In such context, the data becomes not only a tool for interaction between humans and machines but also as a powerful weapon of control. The political, humanitarian and civil issues of such systems are out of scope for this article and are explored elsewhere²⁷,²⁵,⁹ but we mention them here for a broader context.

We are conscious of the issues surrounding user data, privacy and consent when building multiuser interactive installations. We are presenting a prototype of an installation that uses many techniques for tracking users such as pose and action detection, position as well as interaction with a mobile phone and we propose an artistic and playful take on the subject of user consent to be tracked, as described, and how we can use that data in attempt to control the users, either individually or collectively. Such playful uses of topics related to surveillance are not new²,²⁶ but we believe that such approaches are necessary and efficient in raising awareness of these issues among some slices of populations.

The technological assemblage that we propose covers several aspects of a production pipeline that includes input (computer vision, interaction opportunity via smartphone), show control (timeline-based and interactive), and display (video mapping, audio spatialization). We aim to build a system and know-how that can be generic enough to be useful for other types of artistic, design, and social manifestations. Our proto-typical installation offers only one of the possible views and uses. We hope that our solution can serve as a starting point for others.


(1)  https://sat-mtl.gitlab.io/documentation/splash

(2) https://sat-mtl.gitlab.io/documentation/livepose 

(3) https://mobilizing-js.net 

(4) https://sat-mtl.gitlab.io/documentation/satie

(5) video-over-IP transport and codec

Conclusion: Draft of a taxonomy by example

We will conclude with three hypotheses of artistic installations based on the conceptual positions and technical choices we have outlined throughout this article. This work relies on the generic “dispositif” that we are setting up: an immersive “big picture” realized by video-mapping, a tracking system of the spectators in the space and smartphones interconnected with a local network, the whole activated by a group of spectators invited to share this experience. Surveillance is the general theme that we choose for these experiments, which allows us to contextualize our subject and to lead our choices of implementations. In order to ensure a certain critical and reflexive dimension, we aim to provide a kind of situation reversal in the experiments, the controlling power passing from one side to the other: in a first time, the spectators interact with a system which relies on informations captured about them (their position in the room, the proximity between them, the gestures they produce with their smartphone, etc.) and, in a second time, this system reveals itself to everyone by showing the “other side of the story”, i.e., the information that has been used by the system to function. The black box becomes transparent. The visual elements that will be used will be mainly textual: words, sentences, paragraphs will be displayed in the smartphones, as in the immersive projection. The relation between the spectators and these textual elements will be regulated by the “collective interactivity figures” that we retain for these first effective artworks : collaboration, cooperation and competition.

Collaboration

The immersive projection shows a series of white-on-black text excerpts about surveillance and about textual content control systems in different contemporary contexts. The scale of the ethical “seriousness” of these systems would go to a crescendo. One can think of the erroneous corrections produced by automatic spellcheckers when writing SMS, to the censorship and generalized surveillance exercised on social networks in certain countries with totalitarian political regimes that can lead to the incarceration of militants, for example. These texts do not appear complete, some words are missing. The audience has to fill in the gaps in these texts with their smartphones. The texts appear in a sequential way: a first text is displayed, when it is completed by the public, it gives room to a second text, and so on. This principle allows the progression in the “seriousness” of the following texts.

In their smartphones, the spectators see a circular list of words displayed that they can rotate to select one of the words. A “send” button, similar to an instant messaging interface, allows the word to be transmitted to the text in the immersive image. This list of words changes according to the proximity of the people with one of the empty spaces in the text: it is necessary to be close to one of the holes of the text to be able to fill it, or else, the screen indicates “out of reach”, meaning the impossibility of participating. Thus, if the spectators do not collaborate together by spreading out in front of the text, they will not be able to fill it in to discover the next text.

The last text that appears is a report of the interactions that took place during the experiment, a description exposing the data collected by our “dispositif” to make the artwork operate: the number of participants over time and, for each one, their number of right and wrong answers, the time needed to complete the texts, the number of participants connected but who did not collaborate (being all the time “out of reach”), the models of smartphones used, etc. A form of natural language visualization of the data collected throughout the experience, reminiscent of “bots” or automatic conversational agents.

Cooperation

Cooperation and collaboration are distinguished by the temporality of group interactions. Collaboration does not necessarily imply a temporal concurrency, the actions complete each other even if they are not synchronous, as in the example above. Cooperation implies, according to us, at least a proximity, and to the most a temporal synchronization between the interactions, the actions must be operated jointly by all or part of the group of spectators.

The immersive image has a white, plain background. Each spectator sees a white word on a black background appearing in his smartphone. When they tap the screen of their smartphone with their finger, the word appears in the immersive image in the place where it belongs in a sentence. If all the spectators tap their screen with their finger, all the words appear in the projection, but a certain order must be respected in these interactions, otherwise the words disappear just after their appearance in a fading effect: the spectator who sees the first word of the sentence in his smartphone must be the first to tap his screen, followed by the second, then the third, and so on until all the spectators have made their word appear in the large image in the right order of the sentence. When a sentence is completely displayed, the colors are reversed in the screens, the white background becomes black and the text is drawn in white and vice versa on the smartphones.

This sequential collaboration between the spectators will allow them to progressively discover sentences coming from texts about industrial espionage cases based on security flaws in computer systems.

Competition

The immersive image is empty of text and is split in two coloured areas: white on the left and black on the right. In their smartphones, viewers are asked to write their nickname before they can see an editable text field (a prompt) centred in their screen. Depending on the person, the background colour of the screen is black or white. The group of spectators is actually split in two using this colour: if there are 10 spectators, 5 will have a black background and the others will have a white background. Under the prompt, on the mobile screen, the following inscription can be read: “Look for those who do not have the same background colour as you and report their misdeeds!” Using the prompt, viewers can write what they want to report. An automatic moderation system checks whether the sentence is a denunciation or not. If they are successfully recognized as denunciations, the sentences are written in negative in the corresponding coloured area in the immersive image (i.e. white text on a black background). If not, the viewer’s screen is marked with an error message and the prompt is cleared so that he or she can try again.

If no one writes with their smartphone after a certain period of time (30 seconds to 2 minutes, to be verified in real conditions), sentences from a collection made beforehand and integrated into the program are displayed from time to time in one of the coloured zones of the immersive image. The idea is to introduce doubt: did someone write this, or is the device expressing itself?

The group that manages to fill its screen with sentences first wins the right to write whatever they want without any automatic moderation and on the whole immersive image, which will take on the background colour of the winning group, for 1 minute. After that, a list of all the sentences written during the whole session with the associated user’s nickname are displayed for a few seconds, exposing everyone’s actions to the eyes of the public, before the device resets.

These 3 scenarios are still to be refined in their details, but they show how a key notion can be used as the root of an artistic proposition using our “dispositif”. Our future work consists in concretely realizing these scenarios and putting them to the public test.

Acknowledgements

We would like to thank our Materia workshop participants: Annae ̈lle Sermanson, Idun Isdrake, Heather Kelley, Valentine Auphan, for their involvement along the path, Hexagram and Elektra for facilitating the workshops and contacts with other creative humans.

References

1 Giorgio Agamben, Qu’est-Ce Qu’un Dispositif? Rivages poche, Petite Bibliothèque, 2007.

2 Anders Albrechtslund, Lynsey Dubbeld, “The Plays and Arts of Surveillance: Studying Surveillance as Entertainment”, In: Surveillance & Society 3.2/3, Sept 2002, ISSN: 1477-7487, DOI: 10.24908/ss.v3i2/3.3502.

3 Jesse Allison, Yemin Oh, Benjamin Taylor, “NEXUS: Collaborative Performance for the Masses, Handling Instrument Interface Distribution through the Web”, In: (), 6.

4 BumpIt! Main Page, http://www.bertrandplanes.com/bumpit/.

5 Can You See Me Now? — Location Based Game, https://www.blasttheory.co.uk/projects/can-you-see-me-now/.

6 Jean-Marie Dallet, “Figures de l’interactivite", In: Anomalie, Mar. 2003.

7 Peter Dalsgaard, Christian Dindler, Kim Halskov, “Understanding the Dynamics of Engaging Interaction in Public Spaces”, In: Human-Computer Interaction – INTERACT 2011, Ed. by Pedro Campos et al. Vol. 6947, Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, 212–229, ISBN: 978-3-642-23770-6 978-3-642-23771-3, DOI: 10.1007/978-3-642-23771-3 17.

8 David Rokeby, Border Patrol (Collaboration with Paul Garrin), http://www.davidrokeby.com/border patrol.html.

9 Ann Dennon, Students on School-Issued Devices Are Under Constant Surveillance — BestColleges, https://tinyurl.com/4sastf6u, oct 2021.

10  Brian Droitcour, Rafael Lozano-Hemmer on Seductive Participation and the Oppression of Metrics, June 2020.

11 Anne-Marie Duguet, “Dispositifs”. In: Communications 48.1, 1988, p.221–242, DOI: 10.3406/comm1988.1728.

12 Clarence A. Ellis, Simon J. Gibbs, Gail Rein, “Groupware: Some Issues and Experiences”, In: Communications of the ACM 34.1, Jan. 1991, 39–58, ISSN: 0001-0782, 1557-7317, DOI: 10.1145/99977.99987.

13  Michel Foucault, Dits et Écrits, Vol. III, 1977.

14 Jon Froehlich et al. “MyExperience: A System for in Situ Tracing and Capturing of User Feedback on Mobile Phones”, In: Proceedings of the 5th International Conference on Mobile Systems, Applications and Services - MobiSys ’07. San Juan, Puerto Rico: ACM Press, 2007, p.57, ISBN: 978-1-59593-614-1. DOI: 10.1145/1247660.1247670.

15 E Hall, The Hidden Dimension, Anchor, 1966.

16 Kasper Hornbæk, Antti Oulasvirta, “What Is Interaction?” In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver Colorado USA: ACM, May 2017, 5040–5052, ISBN: 978-1-4503-4655-9, DOI: 10.1145/3025453.3025765.

17 Min Jiang, “A Brief Prehistory of China’s Social Credit System”, In: Communication and the Public (), p.6.

18  Brett Jones et al. “RoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-Camera Units”, In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, Honolulu Hawaii USA: ACM, Oct. 2014, 637–644, ISBN: 978-1-4503-3069-5, DOI: 10.1145/2642918.2647383.

19 Oussama Mubarak, “A Taxonomy of Multi-User Co-Located Interaction Environments”, In: (), p. 4.

20 Michael Naimark, Displacements, http://www.naimark.net/projects/displacements.html.

21 B.D. Owens, “Watching Me, Watching You: Reflections Upon Surveillance, Gare Loch Duality and the #UndesiredLine”, In: Media-N 17.2 (Oct. 2021), 49–68, ISSN: 1942-017X, 2159-6891, DOI: 10.21900/j.median.v17i2.882.

22 Tomislav Pejsa et al. “Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment”, In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, San Francisco California USA: ACM, Feb. 2016, 1716–1725, ISBN: 978-1-4503-3592-8, DOI: 10.1145/2818048.2819965.

23 Marianne Graves Peterson, Majken Kirkgaard Rasmuseen, Peter Gall Krogh, “Collective Interaction: A Designerly Visual Analysis of Seven Research Prototypes”, In: Proceedings of the 29th Australian Conference on Computer-Human Interaction, Brisbane Queensland Australia, ACM, Nov. 2017, 68– 76, ISBN: 978-1-4503-5379-3, DOI: 10.1145/3152771.3152779.

24  Ronald Schroeter, Marcus Foth, Christine Satchell, “People, Content, Location: Sweet Spotting Urban Screens for Situated Engagement”, In: Proceedings of the Designing Interactive Systems Conference on - DIS ’12, Newcastle Upon Tyne, United Kingdom: ACM Press, 2012, 146, ISBN: 978-1-4503-1210-3, DOI: 10.1145/2317956.2317980.

25 Saif Shahin, Pei Zheng, “Big Data and the Illusion of Choice: Comparing the Evolution of India’s Aadhaar and China’s Social Credit System as Technosocial Discourses”, In: Social Science Computer Review 38.1, Feb. 2020, 25–41, ISSN: 0894-4393, 1552-8286, DOI: 10.1177/0894439318789343.

26 Karen Louise Smith et al. “Playing with Surveillance: The Design of a Mock RFID-based Identification Infrastructure for Public Engagement”, In: Surveillance & Society 9.1/2, Nov. 2011, 149–166, ISSN: 1477- 7487, DOI: 10.24908/ss.v9i1/2.4108. 

27 Alexander Trauth-Goik, Chuncheng Liu, “Black or Fifty Shades of Grey? The Power and Limits of the Social Credit Blacklist System in China”, In: Journal of Contemporary China, Sept. 2022, 1–17, ISSN: 1067-0564, 1469-9400, DOI: 10.1080/10670564.2022.2128638.

28 Karen Li Xan Wong, Amy Shields Dobson, “We’re Just Data: Exploring China’s Social Credit System in Relation to Digital Platform Ratings Cultures in West- ernised Democracies”, In: Global Media and China 4.2, June 2019, 220–232, ISSN: 2059-4364, 2059- 4372, DOI: 10.1177/2059436419856090.

ARt chat - A Museum App combining AR, Art and Communication

Ivana Družetić-Vogel1, Alina Fuchte2, Marina Bauernfeind3

1University of Applied Sciences Düsseldorf /MIREVI
2NRW-Forum /Kunstpalast Düsseldorf 3Museum Ulm 
Germany
ivana.druzetic-vogel@hs-duesseldorf.de; alina.fuchte@kunstpalast.de; m.bauernfeind@ulm.de

Abstract

Museum labels are often looked at more intensively than the actual work itself. And who says that the interpretation written there is the only possible one? ARt chat is an application that enables museum visitors to place their own opinions, knowledge or even questions as comments virtually in the exhibition space. Through the application, they can also discuss the artworks with others and follow and respond to comments even after leaving the exhibition. The application combines augmented reality, art and communication.

Keywords

augmented reality, digital participation, co-creation, user-generated content, museum studies, audience participation.

DOI

10.69564/ISEA2023-52-full-Druzetic-Vogel-et-al-ARt-chat

Introduction

Digitization and social media are influencing our daily lives—the way we communicate with each other, make decisions, learn, work and spend our free time. We have long been accustomed to being able to express ourselves and participate everywhere. This tendency has not spared the cultural sector, were museums and other cultural institutions strive to keep up with the rapid pace of digital development. In fact, the explosion of ubiquitous technology has been developing hand in hand with the paradigm shift in the museum world, which was set in late 1980es as new museology, a movement in the museum studies marked by Peter Vergo ́s seminal work of the same title.¹ The focus on the social role of museums, new styles of communication, inclusiveness and openness for active visitor participation have been the hallmark of the movement and a plea for new values and beliefs to be strived for in museum community. Forty years fast forward, there is still a lot of work to be done. In an environment characterised by participation and empowerment, many museums and cultural institutions are still seen as elitist and aloof. However, most museums did become institutions that are more centered on the needs of their audience. They are trying to fulfil not only the educational but also entertainment demands, reflecting the dynamics and the multicultural nature of the 21st century and favouring dialogue, interpretation, and experience.² It has become clear that it is no longer enough to exhibit objects and that the question of context should be central.

It has also become clear that in institutions that serve to create, identity and offer a space for representation, those who are to be represented should consequently be included in their work. The museum system and its traditional concepts have long been questioned, counter-designs developed, collection structures reconsidered and, with the involvement of the communities, new ways of telling stories have been invented. In this context the International Council of Museums (ICOM) discussed and approved a proposed resolution for a new version of the museum definition at a General Assembly in August 2022, concluding an 18-month participatory process of revision. The new version of the ICOM Museum Definition reads as follows:

“A museum is a not-for-profit, permanent institution in the service of society that researches, collects, conserves, interprets and exhibits tangible and intangible heritage. Open to the public, accessible and inclusive, museums foster diversity and sustainability. They operate and communicate ethically, professionally and with the participation of communities, offering varied experiences for education, enjoyment, reflection and knowledge sharing.”³ 

And though the bottom-up approaches have been increasing in the last decades, the idea that visitors should contribute to the museum creation with their own voices remains somewhat radical and provocative.⁴ 

At the same time, the process of museum transformation has been immensely supported by technology development. The democratization of the museum has been coinciding with the democratization of technology, characterized by its accessibility to everyone and everywhere. Moreover, due to its rapid expansion in everyday life, technology is for the young generations of museum visitors an aspect more familiar than any other in the museum context. To step into a museum that is not offering a digital layer is seen almost as a kind of anachronism that breaks the continuity of their reality experience. So more than just accompanying the museum experience, new technologies are being so powerful that they are clearly changing the very concept of the museum. As Tomiuc argues, “the issue is no longer whether to use this technology to recreate the museum experience, but how to use it for a maximum impact on the audience.”²

And though the expected digital literacy might pose a barrier to the older generations, which is a worthwhile topic, the fact that museums are struggling to attract new generations of visitors is posing a bigger concern for their future. The continuity of daily experiences in a technological society is interestingly described by Peter Samis in his essay “The Exploded Museum”:

In a technological world, the museum visit no longer begins when a person enters the building, nor need it end when she or he leaves. The museum’s physical space is but one site—albeit a privileged one—in the continuum of the visitor’s imaginative universe.⁵

In a museum experience, this symbiotic relationship of digital and analog is not only enabling the melting and intertwining of the pre- and post-museum visit time, but also of that which is presented/curated and understood/ commented, and of that which happens behind the scenes and in front of the scenes. In the next chapters, we are presenting the process of development of a mobile AR application for museum use that was conceptualized with that incentive in mind and emerged as a product in a project dedicated to the empowerment of museum community.

Project and goals

nextmuseum.io is a cross-institutional digital community platform for swarm curation and co-creation, initiated by NRW-Forum/ Kunstpalast Düsseldorf and Museum Ulm, funded by the Digital Culture Programme of the German Federal Cultural Foundation as well as by the Federal Government Commissioner for Culture and Media.

On nextmuseum.io, curators or institutions can call for joint work on exhibition projects via an Open Call. Artists can submit artworks and participate in discussions, as can art enthusiasts, who have various opportunities to participate throughout the entire process. We believe that the multitude of opinions and concentrated knowledge that characterizes the online community creates an intelligent, powerful swarm that wants to be involved in shaping things. The platform offers tools and support for collaborative work in the art and tech sector, digital experiments developed with the project partner MIREVI (Mixed Reality and Visualization) from the University of Applied Sciences Düsseldorf and community events for exchange and empowerment.

Since its launch in July 2020, more than 30 Open Calls for collaborative exhibition projects have been made via nextmuseum.io and more than half of them have already been shown physically or digitally. The community now has more than 500 members and participated in about 75 events - from meetups with curators and expert talks on current topics (e.i., AI, sustainability, XR, NFT) to the tech art late night show “It's tech time!”.

Related Work

The use of Augmented Reality (AR) technology in museums has been on the rise during the last decade.⁶,⁷ In comparison with other types of technologies on the Mixed Reality spectrum,⁸ AR has gained popularity in the cultural sector in general because of its broad availability and the simplicity of its use. Unlike Virtual Reality (VR) technology, which often requires expensive hardware and a more complex setup, mobile AR experiences only need a handheld device in the form of a (personal) smartphone or a tablet and are usually downloadable as applications. Furthermore, the App-based AR is increasingly being replaced by Web-based AR, which provides an even more accessible experience and on which great hopes are pinned in the coming years.⁹ In this chapter, however, we want to give a short overview of the use of AR technology in museums, independently of its individual technological characteristics, primarily focusing on the multitude of its potentials and benefits for the mediation of museum content.

The typical feature of AR is the superimposition of a digital, usually visual content on top of the content that already exists in the real-life scene in front of the person. This sort of layering enables a myriad of possible symbolical uses, such as apparent overlapping of different times in the same space, a comparison between reality and image, notation or marking on the surface of reality, an image of hallucination or fantasy, a notion of transience and instability, an illustration of invisible processes, or telepresence of people or objects, to name just a few. When used in an exhibition context, these symbolic potentials open enormous opportunities, both for the exhibited content as well as their interpretation and mediation of the same.

In her text from 2017, Shelley Mannion, the Digital Learning Programmes Manager at The British Museum, reports on the conclusions that her team had drawn from their several years-long exploration of AR’s potential in museum education. Acknowledging the inability to include all its variations, they attempted to classify different types of uses of AR in museums by suggesting four categories: (1) Outdoor guides and explorers; (2) Interpretive mediation; (3) New media art and sculpture; and (4) Virtual exhibitions.⁷ While the first category relates to the museum content outside of its walls and coincides with the GPS based AR applications, the second one refers to some of the first interactive museum applications that engaged the visitors in physical interaction by combining wearables and facial recognition with the marker-based AR technology. The last two categories are jointly described as ones coming from artists, which were “the first to recognize AR’s potential to challenge the curatorial hegemony over galleries.”⁷ The work of the artist collective Manifest.AR has here been a well-known example of guerrilla interventions in the museum space enabled by AR technology. ¹⁰ As Mannion emphasizes, this and similar examples illustrate “one of the strengths of AR for museums: its ability to provide multiple layers of invisible interpretation in galleries.”⁷ In a similar attempt to offer an overview of the use of AR technology in museums, Charlotte Coates lists several types of its adoption in museum contexts. Along with accompanying examples, these include adding explanation to museum pieces; bringing objects or scenes to life as a sort of art intervention; interactive and immersive projections; taking collections beyond their walls in a digitalized form and using AR for learning experiences, especially in different types of scientific museums.⁶ 

While both authors provide useful and interesting insights into the topic, they do not include any examples that would illustrate the use of AR technology for direct participatory practices of visitors in museums. All the examples refer either to a digital extension and transformation of usual interpretation tools, or to a form of AR artworks. The AR tools that would enable activities or support values related to visitors' engagement in a form of expression and exchange of their opinions or communication with the curators and the artists are not mentioned. Without wanting to conclude that such projects do not exist in AR form, this does show us that they are not yet common or influential enough to include them in such analysis as a separate category.

Similar attempts have, however, been carried out with other types of interactive and mobile technologies. By using digital technologies for creation of a personalized and interactive visitor experience through contribution and exchange of user-generated content, the installation called “The Cell” (2015) at the Information Age Gallery of The Science Museum London¹¹ and “Debatorial” (2020), an interactive online platform launched as part of the digital exhibition “Beyond States. The Boundaries of Statehood” at the Zeppelin Museum in Friedrichshafen,¹² bare some important similarities to our project. “The Cell” enables visitors ́ participation in the installation by sending a text or adding an image via their mobile phones, which then unlocks additional content or include them in live polls. The contributions and the information gathered help in building upananonymizedlivedatarecordofthe Gallery’svisitors. The “Debatorial” is, on the other hand, a fully digital platform intended to facilitate exchange on several thematic areas via various formats. In addition to artist and curator talks, the formats include podcast, chats, quizzes, zoom meetings, surveys, and animated maps, all of which aim to promote visitors ́ participation and transparency.

The application ARt chat that we present in the following lines merges the advantages of AR—its general accessibility and presence of content in real 3D space—with the qualities of participatory and personalized visitor experience comparable with the ones exemplified in the two mentioned projects.

Description of the functions and added value

Exhibition visitors can download the application free of charge from Google Play Store¹³ or App Store¹⁴ while in the museum or before the visit by following a link on the website. For using the application, they have to log in or register with their Apple, Facebook or Google account, or their email address and password. After that, a button “Start AR” appears for accessing a step-by-step guide. After reading the instructions for use and closing the guide, a QR code placed next to selected artworks in the exhibition can be scanned. After scanning the code and pointing their mobile device at the corresponding artwork, a small figure in the top corner is indicating when they can start entering their comments (Figure 1). Those can be placed anywhere in the 3D space by moving in the room and using drag-and-drop function on the screen (Figure 2, 3). 

Figure 1: The view of the artworks commented with the [anonymized] application

If one wants her/his comment to no longer be publicly visible, it can be deleted via the trash icon. Visitors can also react to comments of other users by clicking on them and typing an answer. The reply appears in the application and is visible for all other users immediately after sending. Inappropriate comments by other users can also be reported and deleted after reviewing if necessary. For scanning a new artwork, users just have to click on the QR code symbol. By going back to the main menu, users can find the FAQs for more information about the application and their account. On this page, also, a button leading to the already scanned artworks can be found, including the related user comments. Via the “my comments” button, users can see their own comments as well as the reactions of other users. Those two functions can also be used at home, after visiting the exhibition (Figure 4).

The conditions necessary for a cultural institution to become a place for participation, which were outlined by Nina Simon, are a good summary for some of the main driving forces in our project and can be recognized in the function of the described application. These include: (1) desire for the input and involvement of outside participants, (2) trust in participants’ abilities and (3) responsiveness to participants’ actions and contributions.¹⁵

Unlike elements that have already been implemented in the museum, visitors can choose to use or not to use the application without having information withheld. Art chat is intended to stimulate discussion among visitors. 

Figure 2: Placing a comment in 3D space in the exhibition

Since the content is freely configurable, the target group ranges from interested first-time visitors to professionals. During and after a museum visit, there is the possibility to interact with the exhibition and lead discussions with other visitors. In addition, one has a sustainable storage of museum visits and one’s own interactions, which can be retrieved long after the actual museum visit. As nicely put by Carey Jewitt in his essay on digital technologies in the museums—the hand held devices enable possibilities for innovative interpretation, especially in regard to decreasing curatorial control and allowing audience narratives “to override the search for a single authoritative voice.”¹⁶

Also, the museum can collect, answer, and evaluate questions and comments, whereby only anonymized data regarding interaction with artworks and their comments are stored and evaluated.This data can be helpful, for example, in planning further exhibitions or optimizing a permanent exhibition. In the longer term, the application can be used and adapted for several museums and exhibitions. This has an advantage for the users who only need one application, but also for the museums, because they do not have to develop their own museum-specific application.

Finally, the application can also be used by the curators and artists themselves, enabling them not only to react and answer to visitors’ questions and comments, but also to participate in the exhibition in a very personal and immediate way, pulling down the boundary wall between the official and authoritative museum discourse and the individual perspectives.

Figure 3: The cloud of AR comments in an exhibition

Background Information

The project was initiated as part of the partnership between nextmuseum.io and the MIREVI Lab of the Düsseldorf University of Applied Sciences (HSD), where students of media informatics and communication design took on the challenge of developing three different prototypes for interactive museum labels. After their successful development, the prototype of the ARt chat application was selected to be further developed with the HSD ́s MIREVI Lab and tested in the partner museums (Figure 5).

The exhibition “Kunstreichgewächse” in Museum Ulm (September 2021) served as the first test site for the application. The prototype version only worked on a local smartphone or tablet with a demo version of the application that the visitors could get in the museum. By clicking on the AR button, the camera of the respective device opened to scan the QR code of an artwork. After scanning the QR code, visitors could write a comment and place it as a 3D element in AR using drag-and-drop function. The chat between different visitors was not yet possible at this stage. User tests and feedback helped in detection of the remaining bugs which were then fixed by the tech team.

In the advanced test version, the application continued to run only on locally installed tablets that could be borrowed from the museum. Here, the comments of other visitors who had already visited the exhibition or were currently in the museum have been displayed and updated in real time, and one could also reply to them. 

Figure 4: The view of the application's UI

It was possible to create a personal account that visitors can reactivate for each new visit. However, it was not yet possible to activate and manage the account from home. Finally, a handover to external partners for completion and release in the Google Play and App Store took place, making ARt chat accessible on users ́ own mobile devices in the exhibitions “Subversives Design” in NRW-Forum Düsseldorf (2022) and “Wir müssen reden!: Die Münster-Krippe im Meinungsstreit” in Museum Ulm (2022). The final version is available for download in both stores and can be used on all AR-enabled devices. User accounts can be registered automatically via an existing Apple, Google or Facebook account. After visiting an exhibition and commenting on the artwork, visitors can continue the conversation from home and receive information about the chat activities they are involved in.

Technical Implementation

The application is based on several components: the Mapping App Immersal¹⁷ was used to scan the artworks and to link it to the database and the corresponding QR code. The scanning process that is enabled by the Immersal App consists of taking several dozen photos of the artwork. Unity Immersal Software Development Kit is required to download scanned artworks (maps) from the Immersal online database via ID and needs to be integrated into the project. MongoDB database¹⁸ was used to store data about the exhibition, the artworks, the users and the chat. Auth0¹⁹ was used to handle all logins via email or social media. Heroku ²⁰ was used to run the Representational State Transfer Application Programming Interface (REST-API) and Management App. Unity²¹ served as a development environment for this repository and was used to compile the application. Following the open-source approach, the code as well as the technical documentation is made available on GitHub for other interested museums to use the application in their own exhibitions https://github.com/nextmuseum/unity-artchat. 

Figure 5: The original concept prototype from the student workshop

Visitors ́ feedback and future directions

Starting with the idea of ARt chat as a tool for visitors to create their own version of museum labels, we offered a writer's workshop every few weeks in the exhibition “Subversives Design” in NRW-Forum Düsseldorf (2022) to provide space for creating texts as well as technical support when needed. It became obvious that visitors preferred to use short form and entertaining comments to interact through the application, similar to social media, rather than elaborate texts. So, we changed the outreach format by having a staff member walk through the exhibition, making people aware of the application, and providing support when needed. At the exhibition “Wir müssen reden!: Die Münster-Krippe im Meinungsstreit” in Museum Ulm (2022) guided tours were offered, where visitors could learn something about the exhibition and instantly share their thoughts via the application. Overall, visitors enjoyed using the application and had fun commenting on the artworks and interacting with other visitors as well as museum staff in the exhibition and at home. In a few minor parts, we could observe difficulties in the UX section, which are to be fixed in the next step. We are also planning to implement a swear filter, so that inappropriate content cannot be uploaded in the first place. There will also be some graphical customization and we will offer a selection of emojis to react to comments, which should make the application even more fun and intuitive to use.

In 2023, the extended version of the application was used in the exhibition “Protest! gestalten” in Museum Ulm. As mentioned, the documentation of the application has been published on GitHub and a several other museums have already expressed interest in using it in-house. 

Conclusion

The ARt chat application exemplifies an innovative approach in using AR technology in museum context, focusing on active visitors’ participation and experience sharing that extends the usual spatial and temporal givens determined by the institutional framework. In that sense, it provides a digital symbiosis of the real exhibition scene set in front of the visitor, with the imaginary one which unfolds in their mind ́s eye. The technological possibilities are here purposefully capturing the feelings, opinions and questionings of the visitors, creating a moving sculpture of the exhibition atmosphere and the response it provokes.

Moreover, the application is enabling an uninterrupted museum experience, encouraging discussion and participation even after the actual museum visit. With its open-source character, it presents an extension of the nextmuseum.io platform dedicated to the swarm curation that aims at a wide community outreach and embraces its further use and development through museum and visitor communities.

Acknowledgments

This project was funded by the Digital Culture Programme of the German Federal Cultural Foundation as well as by the Federal Government Commissioner for Culture and Media. The credits for concept and prototype development belong to the HSD students Miriam Jost, Michael Michel and Kim Tischer. Further development was done by the members of HSD ́s MIREVI Lab Fabian Büntig and Sven Heinen. The optimization and release of the application was taken over by vobe.digital.

References

1 P Vergo, The New Museology, London : Reaktion Books, 1989.

2 Anamaria Tomiuc, "Navigating Culture. Enhancing Visitor Museum Experience through Mobile Technologies. From Smartphone to Google Glass,“ Journal of Media Research, 7:3 (20), 2014.

3 ICOM approves a new museum definition, 24.08.2022, https://icom.museum/en/news/icom-approves-a-new-museum-definition/, Accessed November 16, 2022.

4 Loïc Tallon, Kevin Walker, eds., Digital Technologies and the Museum Experience: Handheld Guides and Other Media, Lanham, MD: AltaMira Press, 2008.

5 Peter Samis, "The Exploded Museum,“ In: Digital Technologies and the Museum Experience: Handheld Guides and Other Media, Ed. Tallon, Loïc Kevin Walker, Lanham, MD: AltaMira Press, 2008.

6 Charlotte Coates, "How Museums are using Augmented Reality,“ Museum Next, 17.04.2022, https://www.museumnext.com/article/how-museums-are-using-augmented-reality/, Accessed November 21, 2022.

7 Shelley Mannion, “British Museum-Augmented Reality: Beyond the Hype," Museum-iD, 2017, https://museum-id.com/augmented-reality-museums-beyond-hype-shelley-mannion/. Accessed November 21, 2022.

8 Paul Milgram, Fumio Kishino, “A Taxonomy of Mixed Reality Visual Displays,” IEICE Transactions on Information and Systems, 1994, 77, 1321–1329.

9 X Qiao, P Ren, S Dustdar, L Liu, H Ma, J Chen. "Web AR: A promising future for mobile augmented reality—State of the art, challenges, and insights,” Proceedings of the IEEE, 107(4), p.651-666, doi: 10.1109/JPROC.2019.2895105.

10  Manifest.AR. https://manifest-ar.art/, Accessed November 21, 2022.

11 The Cell. https://isodesign.co.uk/projects/information-age-gallery-interactive, Accessed November 25, 2022.

12 Debatorial. https://debatorials.zeppelin-museum.de/beyondstates, Accessed November 25, 2022.

13 https://play.google.com/store/apps/details?id=de.nextmuse um.io.ARtchat&pli=1, Accessed November 19, 2022

14 https://apps.apple.comm/in/app/artchat/id1594639117, Accessed November 19, 2022

15 Nina Simon, The Participatory Museum, Santa Cruz, CA: Museum 2.0, 2010. 

A hybrid listening to atmospheric processes

Juan C. Duarte Regino

Aalto University Espoo, Finland Juan.duarte@aalto.fi

Abstract

This paper delves into main aspects that support my environmental sound artistic practice, centered around atmospheric processes. It leads to elaborate on the media installation Augury, which surrounds the concept of hybrid listening to atmospheric phenomena. The proposed approach on hybrid listening investigates the merge of mediated, embodied, and situated knowledge, to assist how we sense our atmosphere beyond our human capacity, by translating environmental signals into audible and immersive perception. Hybrid listening is informed by past meteorological insights, including ancient to modern weather science, deepening into a profound temporal perspective. In consonance to this conference’s theme inquiry about symbiotic imaginary, hybrid listening melds selected meteorological narratives that lead human sensibility to interplay with more-than-human dimensions: living organisms and technical systems. By means of interactive instruments, this research brings this symbiotic ensemble into an immersive sound encounter with the atmospheric milieu. These instruments harmonize with the convoluted ecological dynamics of the atmosphere—an intricate interplay of matter and agency.

Keywords

hybrid listening, atmospheric attunement, deep time, weather divination, ubiquitous computing.

DOI

10.69564/ISEA2023-53-full-Duarte-Regino-et-al-A-hybrid-listenin

Introduction

In recent years, my artistic research has experimented in the realm of environmental sound, with a specific focus on the intricate energies and processes of the atmosphere, notably the wind and the electromagnetic spectrum.² This interest appeared from my exploration of generative and indeterministic methods in sound creation, leading to an exploration dedicated to capturing and making audible pervasive aspects of atmospheric phenomena. To transmute these intangible energies into auditory experiences, I harnessed the affordances of ubiquitous computation, employing networks of on-site electronic weather sensors driven by microcontrollers.³ These experimental incursions merged into a series that contemplates the fusion of the sonic ecologies with technology, unveiling augmented perceptions of surrounding processes catalyzed by the concealed forces of the atmosphere.⁴

Through crafting sensory media devices, it was possible to translate weather data into sound, harnessing environmental sensors to extract real-time insights from the movements, forces, and air quality of the wind. These data streams were converted from electrical impulses into audible compositions, effectively sonifying their subtle transition and tangible essence.⁵ However, while this methodology has become somewhat customary in contemporary media art,⁶ capturing atmospheric phenomena through sensing traces back to an extensive history preceding the advent of computation and electronic media.⁷ This historical backdrop encompasses a tapestry interwoven with artistic, scientific, technological, and mythological threads, constituting a fertile ground for fostering a symbiotic imaginative space placed around atmospheric processes.

Historian of science, Liba Taub makes a comprehensive study of ancient meteorology, noticing register techniques to decode repeatable and predictable phenomena in the weather, unraveling correlations within atmospheric events.⁸ Laub underlines how ancestral weather wisdom was dedicated to elucidating immediate dynamics and prognosticating future conditions. Her study promotes that inscription tools and methodologies proved crucial to clarify the spatial-temporal dynamics of weather, enabling registration and comparison of sequential events. Simultaneously, predictive strategies encompassed the observation of animal behavior, linking with planetary cycles that persist in today’s atmospheric knowledge.

Conceiving the atmosphere as a medium facilitates a holistic comprehension of the intricate web connecting sensing and perception within its realm. This perspective elucidates the nuances between environment and observer, sensation and cognition, and consciousness and perception. By echoing existing contributions surrounding our atmospheric milieu and its role as a communicative medium,⁹ my exploration centers into the realm of weather sensing media, highlighting an environmental awareness of the reciprocal influences between organisms and their surroundings. Central to this investigation is the notion of atmospheric attunement, a concept that echoes our complex experience with our environments. This notion accentuates the parallels between biological entities and technical systems in adapting and responding to processual atmospheric dynamic.

My proposed notion of hybrid listening embarks on a quest to forge an alternative techno-embodiment of atmospheric processes. Rooted in a symbiotic potential of combining organic and artificial dimensions, this endeavor aims to establish an attuned relationship with the atmosphere. The envisioned system harmonizes the capabilities of technical frameworks with the situated and embodied dimensions intrinsic to the extensive temporal canvas of Meteorology. Drawing inspiration from archaic sources, these paradigms shape the contours of ubiquitous and embodied computational systems, offering a novel perspective on our engagement with atmospheric phenomena.

Atmospheric Soundscapes

Previously in my career (2012 - 2016) I experimented with creating soundscapes employing kites as instruments to amplify the wind's forces during this activity. Adopting a Sonic Interaction Design approach, I delved into an intricate interplay between the kite's motion and sound, grasping a dynamic equilibrium between human control and environmental agency.¹⁰ Through the fusion of sonic digital media with existing tools like kites, a realm of augmented and novel agency emerged, fostering interaction with the encompassing environment.

These initial experiments spurred me to delve into a fragmented history of weather measurement instruments, unearthing a rich tapestry of artistic scientific endeavors. This inquiry allowed me to delve into a broader spectrum of experiments where human and technological senses converge in the environment. Among these historical figures, Athanasius Kircher, a polymath and instrument inventor, stood out. His creation, the Aeolian Harp,¹¹ seamlessly blended musical instrument craftsmanship with an animistic ethos, resonating with meteorological exploration by granting voice to the surrounding atmosphere.¹² 

Figure 1. Aeolian harp. Athanasius Kircher. Musurgia Universalis.

These ancestral artifacts possessed the capacity to translate environmental dynamics, offering insights into the flow of wind through the medium of musical sensitivity and artisanal skill ¹³. Among other inventions of this kind, these devices could convince society by demonstrating meteorological knowledge through artistic expression. Besides, this perspective is useful to trace contemporary practices that meet at the intersection of art, science, and technology.

Notably, these ancient manifestations harbor mythological and speculative concepts that connect with representations and myths. One could argue that the aeolian harp confers a more-than-human “voice” to air and wind dynamics. These artifacts resonate with the atmospheric medium and become perceivable for humans. By building upon these insights, my project entitled Aeolian Artefacts¹⁴ generates immersive soundscapes from wind energy and that is transmitted into the “ether.” This device sources wind detection through microphones across four points, generating soundscapes that mirror the wind's intricacies at each sensing point. The resultant auditory output is transmitted via FM radio, speculating artistically on potential correlations between wind and electromagnetic energies. 

Figure 2. Aeolian Artefacts - Environmental sound device, responsive to wind flux. 

The Deep Time of Sensing the Atmosphere

Meteorology was originated in the context of the Ancient Greece, back when it was thought that weather was influenced by the astral events occurring outside our planet, such as comets, visible from our planet.⁸ The root term meteor has a meaning connected to time-related phenomena, thus reading the sky has been associated for a long time with the sequence of time.¹⁵

Taking a deep-time perspective on sensing and comprehending the atmosphere involves an examination of tools, methods, and artifacts. Rather than isolating them by their technological or embodied nature, my proposal advocates for their amalgamation. This integration encompasses not only the physical aspects but also the embedded meanings and narratives inherited from ancient meteorological myths, interwoven with contemporary scientific revelations regarding atmospheric phenomena. This approach to a deep time of sensing the atmosphere informs my artistic practice to give a context for designing a sound instrument with certain affordances valued in the history of these tools, methods, and artifacts.

Historian and media theorist Maximilan Hepach highlights an intriguing parallel between mythology and phenomenology in their capacity to conceptually encapsulate the fleeting yet enduring temporal intricacies of atmospheric climate and weather.¹⁶ Both historical and modern perceptions of climate share a celestial genesis—a vantage point akin to that of a divine entity observing Earth from the cosmic expanse. Aligning with Platonic reasoning, Hepach consider that these mythological frameworks don't stand in opposition to rational discourse; instead, they play a role in altering our perceptual landscape, unfurling novel interpretations and avenues of experience.

Figure 3. The Greek-Roman Parapegmata. Public domain.

Ancient Meteorology

Exploring the Aeolian Harp's role as a weather instrument prompts an investigation into analogous historical manifestations.⁶ The legacy of the Greek-Roman School of Meteorology⁸ sprouted from philosophical inquiry aimed at unraveling the intricacies of weather and eventually evolved into predictive practices. Hesiod, the poet active around 750-650 BC, inaugurated Astro-meteorology, interweaving astral events with weather predictions. Ptolemy's approach to Astro-meteorology hinged on the interplay of stellar risings and settings, attributing his prognostications to cosmic harmony.⁸

Among the commonly utilized ancient weather instruments was the Parapegmata—a prevalent fixture in Greek-Roman cities.⁸ This inscription-based medium resembled a calendar but was tailored for weather forecasting. By sequencing and combining events, it provided estimations of forthcoming weather, wind patterns, bird sightings, and river levels. This holistic approach embraced cues from natural phenomena and animal behavior as indicators of impending weather conditions.

Aristotle's meteorological treatise marked a watershed moment, describing weather through observations of the four elements and their interactions—earth, water, and wind—attributing phenomena like tornadoes and storms to their interplay. This marked the advent of weather observation and measurement methodologies. Early precision instruments, such as the hygrometer, were employed to gauge humidity levels in soil and air. The device compared the density of a charcoal sheet exposed to the air with an unexposed one.

Mythological narratives in Homer personified the winds, aligning them with cardinal points. This led to visual representations, such as the wind rose and anemographic chart. Jansson's 17th-century wind chart, a colonial navigation tool, personified thirty-two wind directions, associated with racial traits from their regions.¹⁷ It featured nomenclature in six languages, including Greek, Latin, and Dutch. Along these lines, Alexander Humboldt's isothermal chart guided settlers to regions with European-like weather, aiding colonization in temperate or cold environments.¹⁸

Figure 4. Jansson ́s Anemographic chart.1650. 
Figure 5. Isothermal chart by Alexander Humboldt. 1823. 

Atmospheric-related divination

Aeromancy, an ancient art, entails predicting broader life-affecting occurrences through the observation of clouds, wind patterns, and changes in animal behavior. Augury and Ornithomancy, divination practices rooted in interpreting bird behavior, further enrich this tradition. Augury, foundational to ancient Rome,¹⁹ offers a unique approach to engaging with the atmosphere's imperceptible dimensions.

Recent scientific revelations align with these divinatory practices. Many migratory birds possess magneto-reception, primarily located in their beaks where magnetite particles bestow a sense of geographic orientation.²⁰ Additionally, their eyes harbor crypto-chromes capable of detecting planetary magnetic fields.²¹ Their infrasonic sensitivity enables them to anticipate storms and extreme weather conditions.²²

Another case of divinatory practices which pertains to my artistic research is the case of Scrying, which proceeds from the Aztec culture and the use of obsidian artifacts, based on the myth of the smoking mirror “Tezcatlipoca.” Aztecs had rituals of divination to invoke the agency of the lord of the smoking mirror to enable them to see beyond what was imperceptible by human senses.²³

Art and Science scholar Hannah Star Rogers explores technologies as modern divination tools. She posits that these can assign meaning by directing attention to the otherwise imperceptible aspects of the natural world, expanding our interpretation of phenomena beyond our regular senses. Rogers emphasizes the synergy of embodied and technology-driven atmospheric sensing, bridging the ancient art of divination with modern technology. Conceiving weather events through this lens involves an empirical analysis across diverse perceptual phenomena, discerning patterns from elements like oxygen composition, orientation, temperature, and electromagnetism.²⁴

Viewing atmospheric sensing through a divinatory perspective encourages contemplation of the environment to seek answers. Can uniting ancient and contemporary worldviews offer glimpses into the future or unveil the unseen facets of our current reality? 

Figure 6. Scrying obsidian mirror and pieces used in the media installation Augury.

Modern Meteorology

Conceiving atmospheric processes within an interconnected framework emerged as a paradigm in the early 20th century, giving rise to contemporary meteorology. This shift marked a departure from earlier empirical methods and ushered in the foundation of weather science.²⁵ Meteorologists of that era focused on surface conditions, employing sparse weather stations interconnected through telegraph lines, supplemented by limited upper air measurements conducted by Aerologists utilizing kites and balloons.²⁶ The drive for robust predictive science was contingent on key advancements, including wireless telegraphy and the advent of weather modeling and statistical analysis.²⁷

Analogous to an ocean of oxygen, the atmosphere possesses a reservoir of energy dwarfed by that contained in oceans, contributing to its relatively mild temperature fluctuations. The ceaseless circulation of winds and currents, both large and small, occurs in perpetual vortices termed solenoids, striving for equilibrium yet never fully attaining it. Toward the end of the 19th century, statistical aggregation gained prominence, coupled with the establishment of climatology as a field of practice.²⁷ Vilhelm Bjerknes, a Norwegian scientist, catalyzed a transformation by synthesizing empirical meteorology with physical laws in the late 19th century. He devised a method anchored in the principles of thermodynamics for weather prediction. His prognostications hinged on calculating seven parameters: velocity across three dimensions, pressure, density, temperature, and humidity. Bjerknes' background in electrodynamics and wireless telegraphy conferred insights into studying hydrodynamic waves coursing through the real fluid atmosphere. Such methodologies aimed not to supplant intuition but to amplify it. Bjerknes' pioneering efforts culminated in the inception of the Bergen School, pioneering scientific weather analysis and forecasting. This school revolved around meteorological observations, data assessment tools, and models elucidating atmospheric structures.²⁵

Computer-Based Atmospheric Predictions

Despite the integration of telegraphic networks across various weather stations, empirical weather prediction remained inherently imprecise. Consequently, the shift towards computational weather models and numerical forecasting emerged as the novel cornerstones in meteorology.¹⁵ This transformative juncture propelled Meteorology into the vanguard of civilian computing utilization, paralleled only by nuclear science endeavors. The inception of Numerical Weather Prediction coincided with the nascent era of computing, led by John Von Neumann. His vision recognized the convergences between nuclear science and the intricate nonlinear physics governing weather phenomena.²⁷

This synergy led to a crucial moment when the ENIAC computer was deployed to test weather forecasting utilizing models developed by the Bergen School. By connecting disparate data points from various weather parameters, this approach sought to forecast weather outcomes. Midway through the 20th century, global implementation of weather forecasting via the ENIAC proved successful, yet refining computer models to approximate human forecasting capabilities required several subsequent decades.²⁷

Atmospheric synthetic cognition

The evolution of remote sensing in Meteorology emerged as a standardized practice, driven by the need for computational environmental awareness and the monitoring of atmospheric dynamics. This paradigm shift ushered in an era where sensing networks and computing models replaced human intervention, particularly within the realms of satellite and radio technologies. This transition to sensor-computing diminished the influence of human bias in information acquisition, reaching beyond the limits of human perception.

Postmodern literary critic Katherine Hayles delves into the hybrid convergence of organic and artificial systems attuned to the atmosphere, an alternative mode of interaction with atmospheric processes comes to light through the notion of Cognitive Assemblages.²⁸ According to literature scholar Mark B. N. Hansen, these kinds of assemblages, founded upon ubiquitous computing and environmental computation, encompass technical systems like smart dust, characterized by a micro-temporality of sensing and engendering sensations that precede conscious human cognition [Hansen]. While devoid of consciousness, these cognitive systems enable a direct and immediate connection to the sensory world, enabling the synthesis of peripheral information or even facilitating novel forms of experiential cognition that transcend conscious-related representation and memory. This technical attunement empowers humans to immerse themselves fully in the act of sensing, prioritizing sensory engagement over the perception of atmospheric phenomena.²⁹

In this framework, ubiquitous computing manifests as cognitive assemblages through networks of sensors, receptors, actuators, and their programmed cognitive processes. This combination engenders a procedurally distributed approach to atmospheric sensing across expansive territories, seamlessly merging "natural" environments with the synthetic cognition furnished by computers. Jennifer Gabrys looks at how these novel sensor technologies are primed for atmospheric operation; for instance, gauging pollution levels and generating reports that inform public policies.⁹ In this context, the atmosphere transforms from a foundational backdrop into a dynamic context that shapes and is shaped by sensor technologies, akin to the integration of the aeolian harp's music into the natural soundscape.

For technical systems, atmospheric sensing transpires primarily within the physical medium, devoid of conscious awareness. Sensibility precedes perception, embracing direct interaction with the sensory realm itself, transcending specific objects accessed by consciousness [29]. Ubiquitous computing and pervasive sensing systems facilitate the catalysis of sensation, operating on a micro-temporal scale that transcends human perception and adheres to non-biological technical protocols.²⁹ This synthetic framework engenders a hybrid attunement to atmospheric processes, synthesizing information in a peripheral manner and fostering novel cognitive experiences unburdened by conscious-related representation and memory of atmospheric encounters.

Along these artificial sensing and cognitive processes, Katherine Hayles's concept of Nonconscious cognition is useful to elucidate the nuances of synthetic atmospheric sensing, encapsulating emergent effects, fluid mutations, and transformations. While remaining beyond human perception, these technical systems infuse our atmosphere with meaning, utilizing somatic markers such as chemical or electrical signals coherent to their operations. This interaction harnesses the cognitive potential of computational media, adapting to the dynamic shifts of an environment like our atmosphere.²⁸

Embodied Atmospheric Attunement

Media theorist Desiree Förster underscores a kind of atmospheric sensing from an embodied perspective, wherein atmospheric processes, though invisible, envelops us in a multi-sensory embrace, engaging our senses through various channels such as touch, scent, sound, and respiration. While the immediate perception of atmospheric shifts might elude us, their profound impact on our environment remains undeniable ³⁰. Atmosphere(s) intertwine with our sensory experiences, reciprocally rendering the perceiver and their act of sensing palpably present. This way, contemplating atmospheres as environments for a diverse array of organisms underscores their capacity to abruptly recalibrate awareness, thrusting themselves from the backdrop to the forefront of our attention.³⁰

Navigating the realm of an ever-transforming atmosphere necessitates to become familiar with its dynamic essence, thus fostering the need for attunement. Atmospheric attunement entails developing an acquaintance with these atmospheres by discerning the effects they exert on our physical and cognitive faculties. This process of attunement involves aligning with something approaching, yet not entirely present—an ephemeral unity with the world, a connection that transcends comprehensive grasp. For instance, the Deep Listening methodologies pioneered by composer Pauline Oliveros³¹ exemplify a form of attunement to acoustic reflections from our surroundings, exemplifying a mode of engagement.

As such, atmospheric attunement encompasses an engagement with the intangible, an intimate negotiation with the ever-shifting currents of our surroundings. It calls us to forge an implicit unity, an interaction that defies conscious awareness and perception, yet binds us momentarily with the vast mosaic of atmospheric processes.

Atmospheric hybrid listening interface

Building upon the concepts outlined earlier, a visionary instrument emerges—a conduit for crafting immersive soundscapes that echo the deep-time of atmospheric sensing and synthetic cognition. This innovative instrument draws from diverse realms, uniting the threads of ancient practices, modern technology, and hybrid listening.

The essence of this multichannel instrument lies in its capacity to synthesize soundscapes from the intricate mosaic of atmospheric events and dynamics, seamlessly merging the temporal layers of deep-time sensing and contemporary synthetic cognition. By crafting sensory instruments calibrated to discern atmospheric nuances, a novel form of hybrid listening takes shape, an amalgamation of embodied and technical attunement that spans the spectrum of wind and natural radio.

At its core, this instrument embodies a divinatory approach, akin to augury, where weather stations serve as modern-day augurs. This convergence births a symphony that resonates with the rhythms of the atmosphere, a sonic reflection of the flow of meteorological phenomena. The soundscape generated becomes a medium for extended listening of the surrounding weather, transcending geographical boundaries as it carries the ambiance of distant sensor stations to the ears of the listeners.

An intricate web of ubiquitous computing systems breathes life into this interface, orchestrating remote operations and ushering in an array of atmospheric data streams. These streams, point out to predict the possibilities of future events, guiding users into a profound attunement to atmospheric processes. This interface paints a canvas of sound, bridging time and space, where the fusion of ancient and modern, tangible and synthetic, coalesce to forge an auditory tapestry that harmonizes humanity with the intricate cadence of the atmosphere. 

Figure 7. Prototype of a multichannel interface for atmospheric soundscape immersion.

Conclusions

In this paper, I have delved into the landscape of my artistic exploration and research centered around the art of sensing the atmosphere to craft captivating soundscapes, underlining two milestone projects: Aeolian Artefacts, and Augury. Drawing inspiration from the rich panorama of meteorological history, I've traced the interplay between ancient mythologies and scientific advancements in meteorology, culminating in the conceptualization of an ongoing developing instrument—a conduit for attuning to the intricate rhythms of atmospheric processes.


This exploration underscores the symbiotic relationship between embodied engagement and synthetic interfaces. The instrument's inception rests upon a foundation of hybrid connections, weaving together the threads of mythology and scientific inquiry into an evolving work in progress. The bridge between ancient wisdom and modern understanding of our atmosphere provides the underpinning for an instrument that fuses human sensibility with technological agencies that extend our experience about the weather.

As this artistic research unfolds, the embodiment and synthetic attunement to atmospheric processes will be explored further through the lens of interaction design. The forthcoming stages of development will navigate uncharted territories, transforming concepts into tangible experiences. Through this dynamic interplay, the aspiration is to create an instrument that serves as a gateway—a harmonious amalgamation of past and present, mythology and science, ultimately offering a profound auditory connection to the ever-shifting atmosphere.

Acknowledgements

Augury was produced at the RIXC Centre for New Media Culture with received funding for residential visits from the Nordic-Baltic Mobility Programme for Culture to establish project “RIXC Art Science Residencies”. The multichannel interface has received support from the Aalto Digital Creatives Art & Tech Open Call ’23 to continue its development.

Author Biography

Juan C. Duarte Regino is a Mexican artist-researcher and current Ph.D. candidate at Aalto University in Finland. His research revolves around the exploration of the symbiotic relationship between nature and technology through environmental sound.

His artistic endeavors have gained recognition and have been showcased at prestigious events and venues such as the ISEA, NIME, CTM Festival, Spiral Gallery, Pixelache, IAMAS, RIXC, Media Art Histories, Ujazdowski CCA, Goethe Institut, ETH Zurich, and Medialab Matadero. [https://www.juanduarteregino.com]

References

1 Duarte Regino, J. C. Computing Atmospheric Attunement and Hybrid Listening through Augury and Scrying", School of X in connection with XCoAx, 2023. https://classof23.xcoax.org/paper09.html.

2 Douglas Kahn, Earth Sound Earth Signal: Energies and Earth Magnitude in the Arts, University of California Press, 2013.

3 Jennifer Gabrys, How to Do Things with Sensors, University of Minnesota Press, 2019.

4  C. Salter, Sensing Machines: How Sensors Shape Our Everyday Life, MIT Press, 2022.

5 T. Hermann, A. Hunt, Guest editors' introduction: An introduction to interactive sonification, IEEE multimedia, 12(2), 2005, 20-24.

6  J. Randerson, Weather as medium: Toward a meteorological art, MIT Press, 2018.

7  Siegfried Zielinski, Deep Time of the Media, Cambridge, MA, MIT Press, 2008.

8 L. Taub, Ancient meteorology, Routledge, 2004

9 J. Gabrys, Programearth: Environmentalsensing technology and the making of a computational planet (Vol. 49), U of Minnesota Press, 2016.

10 Regino Duarte, J. Pulsar Kite: a study case for Sonic Ludic Interaction, Master Thesis, Aalto University, 2015.

11 J. Fletcher, Athanasius Kircher and his ‘Musurgia universalis’(1650). Musicology Australia, 7(1), 1982, 73-83.

12 T. Morton, Ecology with out nature: Rethinking environmental aesthetics, Harvard University Press, 2009.

13 B. M. Stafford, Artful science: Enlightenment entertainment and the eclipse of visual education, MIT press, 1994.

14 Regino Duarte, J., Aeolian Artefacts, 2018, DOI: 10.14236/ewic/EVAC18.50j

15 J.D. Peters, The Marvelous Clouds: Toward A philosophy of elemental media, University of Chicago Press, 2020.

16 M. G. Hepach, Ephemeral climates: Plato's geographic myths and the phenomenological nature of climate and its changes, Journal of Historical Geography, 78, 2022, 139-148.

17 R. D. Black, [Wind] scapes: Engaging the shifting nature of the wind formed landscapes of Denmark (Doctoral dissertation), 2016.

18 E. M. Conway, The Humboldt Current: Nineteenth-Century Exploration and the Roots of American Environmentalism, 2008.

19 Lindsay G. Driediger-Murphy, Roman Republican Augury: Freedom and Control, Oxford University Press, 2019.

20  R. Wiltschko, W. Wiltschko, Magnetoreception in birds, Journal of the Royal Society Interface, 16(158), 2019, 20190295.

21 M. Albaqami, M. Hammad, M. Pooam, M. Procopio, M. Sameti, T. Ritz, C.F. Martino, Arabidopsis cryptochrome is responsive to Radiofrequency (RF) electromagnetic fields, Scientific Reports, 10(1), 2020, 11260.

22 J. N. Zeyl, O. den Ouden, C. Köppl, J. Assink, J. Christensen‐Dalsgaard, S.C. Patrick, S. Clusella‐Trullas, Infrasonic hearing in birds: a review of audiometry and hypothesized, 2020.

23 Silke Ackermann, Louise Devoy, The Lord Of The Smoking Mirror: Objects Associated with John Dee in the British Museum, Studies in History and Philosophy of Science Part A, 2012, 43.3:539-549.

24 Hannah Rogers, Field_Notes: The Technology of Divination, Art Journal Open, 2019, http://artjournal.collegeart.org/?p=11090.

25 J. R. Fleming, Inventing atmospheric science: Bjerknes, Rossby, Wexler, and the foundations of modern meteorology, MIT Press, 2016.

26 R. J. Wille, Colonizing the Free Atmosphere. History of Meteorology, 8, 2017, 95-123.

27 P. N. Edwards, A vast machine: Computer models, climate data, and the politics of global warming, Mit press, 2013.

28  N. K. Hayles, Unthought: The power of the cognitive nonconscious, University of Chicago Press, 2017.

29 M. B. Hansen, Ubiquitous sensation: Toward an atmospheric, collective, and microtemporal model of media, Throughout: Art and culture emerging with ubiquitous computing, 2013, 63-88.

30 D. Förster, Aesthetic experience of metabolic processes, meson press, 2021, 182.

31 P. Oliveros, Deep listening: A composer's sound practice, IUniverse, 2005.

Eleven Movements of the Cryoscape – Ecological Explorations in Sonification for Affectively Engaging with Climate Change

Louise Foo1 and Jonas Fritsch2

1 Artist, 2IT University of Copenhagen, Copenhagen, Denmark
louifoo@gmail.com, frit@itu.dk 

Abstract

In this paper we present the sound installation Eleven Movements of the Cryoscape which was created for the new Kangiata Illorsua – Ilulissat Icefjord Centre in Greenland. The installation is a near real-time sonification of the movement and melting of the Inland Ice, consisting of an array of eleven speakers, each transmitting from a different location in Greenland. The installation portrays the Inland Ice as a living, breathing, evolving organism in the age of the anthropocene, where humankind has made its mark on the very changes that occur to the natural sounds over time. The installation invites people into an affectively engaging and contemplative relation to our changing ecologies through sonification of data to reflect on our present condition and to potentially imagine and connect to new realities. In this paper, we present the process leading to the creation of the installation and how it adds to existing research into sonification and listening practices in artistic and design research.

Keywords

Sonification, Climate Change, Ecological Data, Cryoscape, Listening, Affective engagement, Artistic Research.

DOI

10.69564/ISEA2023-54-full-Foo-et-al-Movements-of-the-Cryoscape 

Introduction

The Inland Ice plays a key role in the global climate change debate. Accelerating melting of the Inland Ice is one of the problems we have been facing for a long time in a world, it seems, in perpetual ecological and technoscientific crisis.¹³,¹⁸,²¹,⁵⁷,⁵⁹ There is a paradox between the hopelessness and apathy many of us feel, and questions of how things could be different, an urge for change and action (e.g., ²⁷). In the following paper, we present a sound-driven artistic intervention that might contribute to altering affective attachments and mobilizing climate action through listening and attuning to the geological movements of the Inland Ice.

Eleven Movements of the Cryoscape is a near real-time stream of sonification of seismic data and wind data, from the GLISN network of seismic sensors ¹⁹ and PROMICE weather stations [⁴², see Fig. 5 and 6], normally used for scientific monitoring of the Inland Ice. The installation is situated at the Ilulissat Icefjord Centre, Greenland and consists of an array of eleven speakers, each transmitting from a different location in Greenland. Sonifications of seismic signals—movements of waves that travel through the earth and ice from the ice sheet, sea ice, ocean tides, ice streams, icebergs, calving fronts, and glaciers—are transmitted, allowing visitors to listen beyond the range of the human ear. The soundscape consists of ice, wind, waves, earthquakes, and human activity on the island from the past week. The installation plays back a 15-minute loop that is updated every night at midnight. The wind speed at each of the locations controls the volume of each seismic transmission to highlight how multiple environmental factors are interconnected in relation to ice as a body of knowledge, towards identification of climate change. The installation portrays the Inland Ice as a living, breathing, evolving organism in the age of the Anthropocene, where human beings have made their sonic mark on the natural soundscapes. Today, human involvement can be heard beyond what acoustic ecologists refer to as the anthrophony, i.e., the "human-made sounds."⁴⁰ We are not just listening to ice with a touch of human presence. We are listening to ice as it is radically altered by humankind.

In the installation, climate changes and the melting of the Inland Ice is expressed as an ever-changing orchestration in what we rather than a soundscape have chosen to refer to as a "cryoscape" understood as an ecocritical framework to emphasize the dynamic relationship between ice and people. In so doing, we follow the work of Carey et al. who look at glaciers within a feminist framework, for global environmental change research.³ Art historian Isabelle Gapp positions glaciology in an eco-critical art history.¹⁵ The term cryosphere, first introduced by Antoni Bolesław Dobrowolski to describe the frozen-water parts of the Earth,⁹ has resulted in several subsequent typologies that draw upon ice and snow between culture and environment. The idea and term cryoscape was first used by Marcus Nüsser and Ravi Baghel to consider glaciers as more than physical landscapes and accounts for glaciers within epistemic, social, and cultural practices.³⁶ Sverker Sörlin looks to cryo-history to denote the historical role of humans in determining the fate of ice in the Anthropocene.¹⁰ Isabelle Gapp, furthermore, points to how Elizabeth Leane suggests the term cryo-narratives as a reference point for broader and more discursive studies surrounding the cultural history of ice³² and how Klaus Dodds and Sörlin position such multidisciplinary discussions within the new field of ice humanities.¹⁰ This positions ice as a core part of environmental humanities—the study of how our ideas about the more-than-human world, aka “nature” in literature, film, art, and other cultural practices shape how we relate to it materially and politically and how ice gets experienced, represented and storied.

Methodologically, the project has been carried out as practice-based research⁵ and research-creation⁶⁰ combining artistic practice with theoretical concepts to produce new knowledge. The project extends ongoing inquiry into sound and sonification (e.g., ²⁶³⁵⁴⁷⁴⁹⁵⁰⁵¹⁵⁴) and engages with a range of data sonification/sonic methodologies¹, ¹¹, ⁶⁵ within a sonic materialist⁶¹⁶², ⁶³ and affect theoretical perspective³⁴¹⁴¹³. With this installation, we would like to extend existing investigations into sonifications as negotiations between aesthetics, politics of mapping and data representation (art, science & technology). We argue that data sonification can lead to affectively engaging and sonically sensible spaces for listening to "mute," invisible or inaudible ecologies, allowing us to "open our eyes" for a minute, by opening our ears—in which there are infinite possibilities for transformative experiences and ways to connect, interpret, understand and "invent" those ecologies. The article is outlined in the following way: We first position the work with the installation in relation to existing work on data sonification and artistic engagements with climate-oriented sonic installation design. We then present in-depth the work leading to the installation and the guiding research questions that have been explored through the creative process. This leads to a presentation and analysis of the installation in light of said questions and related to lines of inquiry at the intersection between climate change, ecological data sonification, listening practices, art, science and digital technology. Following this, we discuss the knowledge contribution of the work in and outline future explorations in the transdisciplinary and transitional encounter between art, design, and technology towards rekindling affective attachments for imagining a more livable future.

Figure 1. Permanent installation at Kangiata Illorsua – Ilulissat Icefjord Centre; Eleven Movements of the Cryoscape transmits near real-time seismic data as sound, from eleven locations in Greenland. Source: Louise Foo (2022) 

Sonifying Ecological Data

In this section we position our work in relation to existing literature and projects within scientific data sonification, in particular seismic/glacial data sonification. In addition, we add perspectives on listening as a form of affectively engaged ecological attunement to more-than human aspects of the Anthropocene in a time characterized by climate change.

Data Sonification in Science, Music and Art

Sonification of data has most extensively been utilized for scientific purposes, but also, as a means for musical composition⁵⁶, ⁶⁵ and as an expressive medium in art installations. In the article Sonifying the World authors Kanngieser and Gibb point out how human ears are sometimes better than our eyes at detecting subtle changes over time and that sonification—understood as the turning of data into sounds—has emerged as an increasingly popular practice in recent years. In 2016 Scientists succeeded in proving Einstein's general theory of relativity by listening to gravitational waves: “Finally astronomy grew ears. We never had ears before.”²⁹ In musical sonifications composers will typically map data to musical elements such as pitch, rhythm, and timbre. There are several examples of compositions where climate data has been translated to music in one way or the other, to emphasize climatic changes.e.g., ”The Climate Symphony and Other Sonifications of Ice Core, Radar, DNA, Seismic and Solar Wind Data”—a one-hour performance/presentation of sonification research by Marty Quinn of Design Rhythmics Sonification Research Lab ⁴⁹ Kanngieser and Gibb point out, how compared to the noisy nature of ecological data, many categorized as musical sonifications, feel strikingly composed, with the use of familiar western scales, “carefully curated for listening ears.”²⁵

Eleven Movements of the Cryoscape contributes to this transdisciplinary field through inquiry into the use of near real-time streaming in sound art installation. The artist and writer, Andrea Polli, has contributed notable works in sonification as sound installations exemplified in works such as her documentation of arctic melt ⁴⁸. Rather than having to rely on data models and projections like Polli, today, due to rapid technological developments that allow us to live-stream geo-data sent via satellite by the hour from various glaciers around the Arctic, we can explore new artistic potentials and narratives of immediate, immersive, and real-time character. An example of the use of near real-time seismic data in sonification is Herald/harbinger By Ben Rubin and Jer Thorp—which is framed as “a long-distance conversation between a glacier and a city.” The artwork monitors the rumblings, cracks and shifts of the moving ice and transmits seismic data to control sound/lights at an installation at a building in Calgary.⁵⁵ Similarly, to this project, Eleven Movements of the Cryoscape utilizes seismic data, but from several data sources rather than from one. While the artist behind Eleven Movements of the Cryoscape initially set out to compose a soundscape like is the case with Herald/Harbinger, through collaborative efforts and prototyping it became clear that the raw data offered an "authentic" aesthetics emphasizing the sense of “tuning in.”⁶¹

Listening to Ice

Scientists are increasingly using sound as a crucial element in climate research. Dr. Evgeny Podolskiy, professor at the Arctic Research Center at Hokkaido University in Sapporo, Japan states how “Glaciologists just opened their eyes to studying glaciers about 150 years ago. We started to look at glaciers from different angles, perspectives, satellites—but we forgot to open our ears.”⁴¹ In Geophysical Research Letters, Podolsky writes about sounds he has recorded, not with expensive geophysical sensors, but with a smartphone in Bowdoin Glacier (Kangerluarsuup Sermia), located in northwestern Greenland. His recordings captured a unique sound which he used to describe a specific drainage process within the glacier—one that is impossible to observe from the surface: Meltwater drainage through a crevasse. “I’ve been studying glacier geophysics for quite some time and I found that there is this kind of natural zoo, or a universe, of sounds which we kind of totally ignored until recently.”⁵²

Oceanographer Oscar Glowaki’s work with Cryoacoustics—the study of ice through noise involves listening to the air bubbles from glacial ice.¹⁶ 1000- year-old ice does not just contain information about snow, temperature, and pollution, but it also has an interesting sound. The bubbles contain 1000 years of atmospheric air. When melting glacial ice in water, the bubbles explode due to the pressure of a series of small micro-explosions: Each bubble sings (click-click-click) and that is how glaciers produce what Glowaki refers to as the loudest sound in ocean acoustics. He points out how satellites from space can be used to track the melting of glaciers or icebergs. But what is much harder to get data from is that spot where glaciers meet ocean water—and that ice-ocean boundary is where a big part of the melting occurs.⁸

Figure 2: ABOVE: After many years in the field as a glaciologist, Podolskiy found that different types of glacial environments produce their own unique soundscapes. Source: Article in GlacierHub – Ramming / Evgeny Podolskiy (2020) BELOW: Underwater acoustic signatures of glacier calving. From left: 1: Typical Subaerial event 2: Sliding Subaerial event. 3: Submarine Event. Source: Oskar Glowacki (2015) 

In geophysics, sonification has been used to understand long-range and large-scale seismic events, such as earthquakes and volcanoes ⁴⁴ and is increasingly utilized in the study of glacial data. Throughout our research we have come across audio examples of seismic data converted to sound and found it fascinating, considering how seismic data is very similar to sound—vibrations of waves that propagate through the earth, rather than air. And if you speed up the data, you can listen to it. It turned out that raw seismic data, already converted to sound, is available from an extensive network of sensors monitoring Greenland through the GLISN Project in near real time. The main goal of the project is to “provide real-time data from sensors in and around Greenland for detecting, locating and characterizing glacial earthquakes and other cryo-seismic phenomena, and to contribute to our understanding of Ice Sheet dynamics.”¹⁹ Scientists do in this regard look and listen for very specific glacial events, and will exclude data that in the ears of science is "irrelevant," but perhaps this data could be relevant to listening ears within artistic contexts? Hence, we became curious to listen to the sonic phenomena and the going on, in addition to what was flagged as earthquakes and the 3-4 daily major glacial events (icebergs calving and ice quakes), and we began listening to ongoing constant streams of seismic data from the GLISN network.

Affectively Engaging with sonified Ecological Data

Eleven Movements of the Cryoscape presents us with an opportunity to listen to otherwise mute ecologies, by simply "tuning in" on them; making those ecologies perform on their own. It asks if we, through aesthetic and affective experiences and through sonic sensibilities of listening⁶¹ to "mute," invisible or inaudible ecologies can ‘open our eyes’ for a minute, by opening our ears. Salomé Voegelin, an artist, researcher, and writer engaged in listening as a political practice, emphasizes the role art and sonic practices play in asking relevant and necessary questions on today’s global challenges such as climate change. Voegelin articulates how there are sonic potentialities, sensibilities, and competencies⁶¹⁶²⁶³ yet to be explored. In her most recent volume “Fragments of Listening: Political Possibilities of sound”⁶³ she emphasizes how there is a need for explorations in interdisciplinary domains, in a time where “ecological questions need answers from unknown places”⁶³ Voegelin describes listening as an interactivity that challenges, augments, and expands what we see, without presenting an additional illusion to the scene, but by producing the reality of lived experience as invented spaces, not unreal, but phantasmagoric. Listening does in this sense not enhance those environments—it produces them. Voegelin, furthermore, articulates how “duration” and “real-time” qualities produce “not a sense of listening to, but listening in.” The “listening in” and “sense of real- time" produce “authenticity.”⁶ Kanngieser & Gibb point out how sound influences the way we know the world; and whether intended or not affect and emotion intervene when sound is used in novel ways such as sonification.²⁵ Affect is understood as an in-between, pre-personal, relational, and more-than-human concept whose intensity ultimately colours our engagement with ourselves, each other, and the world.¹⁴³⁴ Fritsch has emphasized the role of affect in situations of catastrophe and crisis “at the end of the world,” advocating for an exploration of the role of digital technologies to create positive changes in affective attachments on an ecological scale.¹³ For this relational change to take place, there needs to be a form of affective engagement¹² occurring at an infra-individual level between human and non-human bodies—of flesh, of ice, of sound. Affective engagement in the case of Eleven Movements in the Cryoscape thus describes the potential for a really felt impact that might cause people to act or feel differently by being “taken into” the sonified movement of the ice.

Creating Eleven Movements of the Cryoscape

In this section we will present the process of inquiry leading to the construction of the final installation of Eleven Movements of the Cryoscape. Our main intention is to both foreground the creative and artistic process as a form of listening, while at the same time foregrounding the forms of listening emerging for a broad range of stakeholders during the process. Doing so, we will contribute with insights concerning the (listening) work required in processes concerned with artistic explorations of scientific, ecological data to create an affectively engaging sonification that can open a sensory and contemplative space for attuning to the inland ice in the light of climate change in the Anthropocene.

Context of the Collaboration

In the summer of 2021, Ilulissat Icefjord Centre, designed by Dorthe Mandrup, opened by the icefjord in Ilulissat in Greenland. The Icefjord Centre is run as a self-governing institution, and is built by a partnership consisting of the Greenland Self-Government, Avannaata Kommunia and the philanthropic foundation Realdania who in the following quote state one of the projects key missions: “With its unique location on the front line of climate change, the new Kangiata Illorsua – Ilulissat Icefjord Centre will play an important role in conveying the effect of climate change on the ice sheet.” (Jesper Nygård, CEO of Realdania). JAC Studios was responsible for exhibition design and curation of the exhibition, which disseminates knowledge about Greenland's geological, biological, cultural, and climatic history. The exhibition consists of a permanent exhibition with cinema, central exhibition space, and lastly a separate exhibition space with the working title “Future Lab." For this part of the installation, artist Louise Foo and photographer Anna Domnick were invited to each create a new artistic work based on scientific climate research. Several meetings were held between JAC studios and the artists in which there were brainstorming and sharing ideas, knowledge, and research. A workbook was developed through these sessions, with the following incentives: To give a 1:1 experience of climate. For audiences to experience the inland ice through the senses, sound should be the emphasis, an immersive experience that shows the “unseen” parts of the icefjord, reflecting the spectrum of ice through an instrument—possibly using real-time ice recordings to give live feedback of the current condition and research on the inland ice in an installation that focuses on reflection—reflecting on the place of your own body, your influence on the Icefjord, the influence of the icefjord on you (to reflect on how the Greenland ice sheet influences the global condition) JAC also initiated a workshop / meeting between the artists and scientists from Niels Bohr Institute ³⁷ and GEUS (geological survey of Denmark and Greenland).¹⁷ 

Figure 3: Kangiata Illorsua – Ilulissat Icefjord Centre. 2021. Designed by Dorthe Mandrup. Source: Adam Mørk (2021) 

Listening & inquiring: from field recordings to sonic fields

To begin with, the process of inquiry took its point of departure in focusing on surface melt and glaciers calving, which are some of the major contributors to ice sheet mass loss. Initially, the artist set out to build an archive of recordings of melting ice & recordings of calving glaciers to facilitate the soundscape. The artist was inspired by Jakob Kierkegaards sound installation Isfald in which he uses equipment that gives us opportunities to listen to sounds that we usually cannot hear; underwater with hydrophones or surface transducers to record vibrations of the earth [30] and was planning to do her own sound recordings in Greenland in the summer 2020 while also exploring options of collecting sound from other artists and scientists. The artist navigated sonic phenomena of ice of the arctic through readings, listening to sound files and through contacting a handful of experts (J. P. Steffensen, Glaciologist Andreas Ahlstrøm, Seismologist Peter Voss, Oceanographer Oscar Glowaki, Sound recordist Simon Forrester) who hold recordings of ice and glaciers from across Greenland - both from above and below the water. Obviously, the natural soundscape of ice will be quite different depending on where in Greenland you place your ear or microphone. If a person is standing in the middle of the Inland Ice for example, there is almost no sound, except for the occasional sonic phenomena of ‘firn-stoss’—a collapse of snow (about 5 cm) that happens due to changes in temperature, that can spread several miles, sounding like a rumbling thunder. The artist was fascinated with the complex palettes of sounds of ice and the qualities inherent in movements caused by temperature changes and the sonic states of frozen to liquid in various scales, from microscopic ice crystals crackling, to dispersion of sound waves in ice sheets, to singing, frozen lakes, to rumbling devastating noises of calving events and she became curious to get access to the inaccessible sounds, e.g. sounds to be listened to, beyond the human hearing range. After having listened extensively to various sonic representations of glacial ice in the form of field recordings and data samples (presented throughout this article and see Figure 4) the artist found it problematic that some of the ice sounds would be very pleasing to the ear (when in fact what audiences are listening to is a world in crisis) or on the other hand sounds of calving events or ocean noise were so noisy and overwhelming that she was afraid it would scare audiences away—although that can indeed be a strategy in itself.

Polli’s 2004 work Heat and the Heartbeat of the City is an example of a piece that is not an easy listen, as it plays on discomfort and danger.⁴⁵ Kanngieser and Gibb point out parallels in Polli’s work to work in Soundscape ecology and field recordings e.g., recordings of the melting glacier "Vatnajökull" by Chris Watson ⁶⁴ or Krause ²⁸ who has returned to the same eco-systems for decades, which makes the "ecological processes shockingly obvious" and affectively devastating ²⁵. Rather than shocking and scaring audiences, the artist was inspired to work with the trauma and pain from a point that does not reproduce the pain and hurt, but by removing some of the shock effect and devastation of what glaciers actually sound like (very loud noise), by speeding up the geological processes to create an other-worldly cryoscape. Moving forward she questioned field recording as the best strategy to tell the story of ice and moved forward towards exploring seismic data sonification as a strategy for sound sources.

Figure 4: The artist and programmer did field recordings and listened to the Ilulissat icefjord as part of their process. Source: Louise Foo (2021)

Presentation of the installation for stakeholders

Before moving into the prototyping phase, the installation was presented to the board of the Kangiata Illorsua – Ilulissat Icefjord Centre and implemented in drawings in the exhibition and architectural context. Through that process the artist presented questions about whether she was going to work with field recordings, questioning which data-sources to use, and based her presentation on concepts around a “data-driven sonic portrait of the inland ice.”

The artist explored the possibility of installing speakers in ways to communicate geolocation of stations and data sources; in one experiment speakers were mirroring their actual locations, in another experiment the speakers were placed based on the cardinal directions of events (East, West, North, South) and in the end, through conversation with the curators JAC Studio and Anna Domnick, who exhibited in the same space, Eleven Movements of the Cryoscape was placed on a 12-meter-long curved wall (figure 8) and a round bench was designed by JAC Studios in the center of the space, to facilitate a contemplative, prolonged listening experience.

Protoyping the Installation

In the fall of 2020, artist Louise Foo and programmer Yotam Mann were invited to do a three-week long residency at ITU’s Affective Interactions and Relations Lab, in which a prototype of the installation was built before it was shipped to Greenland. They built a program to fetch streams of data from the GLISN database and PROMICE weather stations, to play from a project website⁴³ using the web audio API and browser-based audio mixing with Tone.JS. They set up controls for listening to seismic data (such as volume); which station, which time, date, speed, and durations (e.g., one week). Audio was routed using a Dante interface to respective channels over network) and an array of speakers were set up on stands in the Lab. The type of speaker landed on a speaker (A 'Diva Droplet by Gallo Acoustics), inspired by a photo of a seismic sensor in the ice sheet, inside a round shaped encasing. The round shape communicated ‘a listening portal’ into the ice, and removed the audience from the expectations towards the usual type of speaker box, which most often will reproduce recorded sound.

Experiments were made, to implement the monthly glacial data (reports of glaciers' calving and ice quakes) as well as data from weather stations in the melting areas around Greenland and connect those data to field recordings, but very quickly the soundscape felt predictable and “animated.” The raw seismic data, on the other hand, turned out to provide an ever-changing otherworldly, cryoscape.

Four speakers were destroyed in the first few days because of too much low-end noise for the speakers to handle, and a low-cut filter was implemented. The following weeks were spent on extensive listening and reflections upon streams of raw seismic data together with people from the lab. The feedback was that people were drawn towards listening to the soundscapes for extensive periods at a time—it provided for a contemplative listening experience that moved them. A sense of “eavesdropping” the Inland Ice. Through that process the team was challenged by how one can compose a soundscape, based on a landscape, in decline? They were wondering if soundscape composition becomes didactic through the very act of composing and were, furthermore, wondering if data-sonification could help them avoid contributing to “The tradition of the appropriation of nature as a resource for the production of culture” ²¹ and instead connect us more directly to those ecologies? 

Figure 5 : Screenshot of weather data from PROMICE weather stations. Source: Yotam Mann (2020)

When listening to constant streams of data (before it has been flagged by a human) it is not possible to know what the data exactly entails. The team was questioning whether that was important? Is there such a thing as a hierarchy of the sounds we listen to? Are the significant glacial events (in the ears of science) for example, more important than the constant flux of other kinds of sounds? How do we choose what is important to listen to?

The raw seismic data sonifications were filtered slightly and a pinch of reverb was added to the channels (for aesthetic / mixing / sound quality purposes) and the overall sound landed on a low volume that draws your attention towards each individual speaker, and invites you to come close to each “listening portal.” The number (eleven) speakers was decided based on the number of reliable sound sources (number of stations that will be transmitting at the same time). Through the process of prototyping the experience (in relation to a 12-meter-long curved wall) the speakers were placed in an array equally apart, in ear-height, with signs underneath each speaker with the station codes (notes of paper set on with tape) to signify which location each speaker was transmitting from. Furthermore, a map of Greenland was implemented, for audiences to connect the station names to their actual locations, as well as a digital calendar clock, to emphasize the duration / time of the installation; to support the near real-time listening experience.

Figure 6: Left: Map and calendar clock as implemented in the prototype, with speakers on stands and with station name signage. Right: Final map showing the eleven locations of the PROMISE weather stations. Source: Louise Foo (2020) 

The program was built so that transmitting stations could change temporarily if a station is out of order for a period (which happens when, for example, one is out of order and something needs replacement). Since this is a permanent installation, we needed to account for several scenarios, the most extreme being one where no stations were no longer transmitting. We produced a solution where all data is always recorded, which fed into the idea of thinking about the installation as documentation – an archive for the future. After the prototype of the installation was finished, scientists Peter Voss and Andreas Peter Ahlstrøm and JAC Studios were invited to the Lab to experience the installation. Even though Peter V oss is familiar with seismic data and is looking at it every day from various scales, he pointed out how listening to the ongoing data in multiple streams at the same time gave him new perspectives on the Inland Ice and he stated that his guess was that 90% of what we hear in the raw seismic streams was ice-related. The last 10% is earthquakes, wind, waves, and human-made sounds which we found just as important to listen to, since the installation portrays the Inland ice as a living, breathing, evolving organism in the age of the Anthropocene, where human beings have made their mark on the soundscape.

Presenting the final Installation at the Icefjord Centre

As installed at the Icefjord Centre Eleven Movements of the Cryoscape can be experienced in different modes of listening (see Figure 7); as purely contemplative listening, supported by the bench in the center of the room, for people to sit and spend time with the overall soundscape created by the sum of all the transmissions. One can listen more exploratively by moving around in the installation and by getting close to each transmitting speaker to listen in to specific locations. One can also connect the sounds with logic connecting sounds from locations to the map and the calendar clock.

The soundscape can also be listened to in conjunction with Anna Domick’s photographs of slices of ice cores in the piece ‘Inside the Ice Sheet’ in which Eleven Movements of the Cryoscape becomes a soundtrack to her microscopic views inside the ice sheet, on a journey to past ice ages. Furthermore, a wall description outside the entrance of the room provides a contextual and conceptual frame for critical reflection.

On the Ilulissat Icefjord Centres website Eleven Movements of the Cryoscape is situated within the following exhibition context; “as a meeting between art and science in an installation, that through sound, images and a tactile universe both challenges and expands our perception and understanding of the ice sheets vast range and its significance for the world we are all part of” (Icefjord Centre Website, 2022). The Danish newspaper Politiken highlighted Eleven Movements of the Cryoscape in its review of the Ilulissat Icefjord Centre as “the final'' that despite its underwhelming architectural priority “houses an installation of 11 speakers” The author explains how “Here you get a real-time auditory image of the enormous activity in the massif, which seems stagnant to the eye.” He follows up with how he now “look differently at the icebergs” and how he senses “a deep-felt recognition of the connection between our way of life and climate change.” 

Figure 7: Installation view of Eleven Movements of the Cryoscape, that transmits near real-time seismic data as sound, from eleven locations in Greenland from the permanent installation at Kangiata Illorsua – Ilulissat Icefjord Centre. Source: Adam Mørk (2021) 

Discussion and Conclusion

Throughout this article we have described the work and the thoughts going into the creation of Eleven Movements of the Cryoscape—a data sonification installation that can potentially lead to affectively engaging and sonically sensible spaces for listening, to activate ecological change. In Eleven Movements of the Cryoscape, the immediate goal is to allow the user to become affectively engaged by the soundscape to viscerally experience the earth moving and to, based on this, potentially rekindle his or her understanding of, and relationship to climate change. However, we have also explored how the sonification of the data can be used by seismologists to gain new knowledge about seismic events and we see potentials in a transversal aesthetics and believe it can be further explored. When we invited seismologist Peter Voss to listen to the ongoing streams of seismic data, he listened with curiosity and started connecting sonic phenomena with guesses of which glacial and tectonic events might be happening; that are not even 'flagged' as glacial events in the eyes of scientists, but still contribute with interesting sonic patterns. Future work would engage in a deeper evaluation of the audience’s experience with the installation.

While we see a lot of potential in using data-sonification for a wide variety of artistic applications, as Kanngieser and Gibb Point out, aesthetic decisions must be made to translate data into the auditory domain. So, to follow up their question about how can we inform those aesthetic decisions and foster emotional impact beyond initial curiosity, that maintains complexity, while drawing out subtle nuances in data whose significance would be overlooked if communicated by any other medium we can point to seismic data conversion, as a quite “autentic” way to work with data-sonification, since there is more of a conversion going on than a mapping per se.

Eleven Movements of the Cryoscape is, among other things, inspired by the interdisciplinarity of ice studies in the humanities and social sciences offering alternative ways of framing and engaging the frozen environment ⁴ posing questions like; is ice justice? This question has been challenged by researchers such as ⁶ who asks if glaciers listen? While ³⁸ analyzes the cultural framing of glaciers, Carey ³ (sees an endangered species narrative applied to glaciers. Jackson (2015) exposes how glaciers are depicted as ruins, and Sörlin ⁵⁸ refers to the present as a cryo-historical moment because “ice has become historical, i.e., that ice is an element of change and thus something that can be considered as part of society and of societal concern.⁵⁸ This has led to further questions about what it means to portray or sonify ecologies. As an artist in a colonial perspective? As a woman in the male-dominated scientific realm of geology and glaciology? What can we learn about perception between human and non-human bodies? In response to those questions, although not all of them have been answered, it has thus been important to portray the Inland Ice, with respect to Indigenous peoples' relationship to the ice and a world-view; “where all aspects are interrelated, interdependent and indivisible. Furthermore, we refer to the installation as a documentation—pointing towards the media archaeological archives of the future ³⁹ and have exchanged the term ‘soundscape’ with the term ‘cryoscape’, to emphasize the dynamic relationship between ice and people.³

In addition to conveying the effect of climate change on the ice sheet, Eleven Movements of the Cryoscape contributes to the exhibition by immersing audiences' bodies in the ice sheet from depths, corners and scales that would be otherwise impossible. It explores first-hand if we can foster a deeper connection, understanding and compassion (and planetary healing) by tuning in and “sitting with the world”—through listening? Through working on this project, we see immense potential in listening to more data that in similar ways can connect us to our planet and its context: listening outwards towards other planets and the sun, as well as inwards into sonic worlds of human and non-human interrelations. Furthermore, the project proposes to reach beyond art institutions as output and activate platforms for different (and potentially broader outreach) than traditional art institutions offer. In so doing, questions of how this type of artistic research contribute to technological, scientific, and environmental discourse are raised in a novel manner.

Acknowledgments

We would like to thank all collaborators and stakeholders in the installation: Johan Carlsson (JAC Studios, curator of the exhibition) Aviaaja Ezekiassen (project manager, JAC) and Elisabeth Momme (Director of The Icefjord Centre) Also thanks to artist Anna Dominick (co-exhibitor) and the genius artist and programmer Yotam Mann who was a core part of the project and a huge resource in designing and developing the sonic experience. We also thank Halfdan Hauch Jensen, Manager of the AIR Lab, IT University of Copenhagen, for support during the prototyping phase.

We would also like to thank the GLISN network of seismic sensors and PROMICE weather stations, for letting their data be available for artistic use. And thanks to GEUS and Niels Bohr Institute and all the scientists and experts that have generously shared their research and given valuable feedback to help this project come to life: J. P. Steffensen, Glaciologist Andreas Ahlstrøm and Oceanographer Oscar Glowaki. A very special thanks to Seismologist Peter Voss who took the time to explain in depth about seismology and to point us to the data-sources, we hope to do more cross-pollination work in the future between art & seismology.

Finally, we thank all the people who engage with the installation daily. The project was generously supported by The Danish Arts Foundation.

References

Michael Bull, Marcel Cobussen, (Eds.), The Bloomsbury Handbook of Sonic Methodologies, NY: Bloomsbury Academic, 2021. 

2 Rosi Braidotti,  Posthuman Knowledge, Cambridge: Polity Press, 2019.

3 Mark Carey, M. Jackson, Alessandro Antonello, Jaclyn Rushing, “Glaciers, gender, and science”, Progress in Human Geography. 40, 6m 2016, 770-793, DOI: https://doi.org/10.1177%2F0309132515623368

4 Mark Carey, “The history of ice: How glaciers became an endangered species”, Environmental History 12, 2007, 497–527.

5 Linda Candy, Ernest. A. Edmonds, Practice-Based Research in the Creative Arts: Foundations and Futures from the Front Line, Leonardo, 51(1), 2018, 63-69.

6 Julie Cruikshank, Do Glaciers Listen? Local Knowledge, Colonial Encounters, and Social Imagination, Vancouver, UBC Press, 2005, p.6.

7  Anne Despond, Nicolas Reeves, Vincent Cusson, Atmosphéries and the poetics of the in situ: the role and impact of sensors in data-to-sound transposition installations, Audio Mostly 2021, Association for Computing Machinery, New York, NY, USA, 2021, 48–55, https://doi.org/10.1145/3478384.3478422

8 Dan Drollette Jr., Bulletin of the Atomic Scientist: How fast is the Arctic ice retreating? Just listen to it melt, June 2018, Retrieved April 30, 2022, https://glacierhub.org/2020/04/16/acoustics-of-meltwater-drainage-in-greenland-glacial-soundscapes/

9 Antonio B. Dobrowolski, Historia naturalna lodu (The Natural History of Ice), Warzawa, Kasa Pomocy im., 1223.

10 Klaus Dodds, Sverker Sörlin (Ed.), Ice humanities: Living, thinking, and working in a melting world, Manchester University Press, 2022.

11 K. Doughty, M. Duffy, T. Harada, Sounding Places: More-Than-Representational Geographies of Sound and Music, UK: Edward Elgar Pub, 2019.

12 Jonas Fritsch, Understanding Affective Engagement as a Resource in Interaction Design, Engaging Artefacts, 2009, 1–9.

13 Jonas Fritsch, Affective Interaction Design at the End of the World, DRS 2018: Catalyst, 2018, 896–908.

14 Melissa Gregg, Gregory J. Seigworth, eds., The Affect Theory Reader, Duke University Press, 2010.

15 Isabelle Gapp, Galvanizing glaciology: thoughts on an eco-critical art history (January 2022), Environmental History Now, 2022, Isabelle Gapp: Retrieved, April 30th, 2022, https://envhistnow.com/2022/01/20/galvanizing-glaciology-thoughts-on-an-ecocritical-art-history/

16 Oskar Glowacki Department of Polar Research, Institute of Geophysics, Polish Academy of Sciences in Warsaw, Poland “Underwater acoustic signatures of glacier calving” O. Glowacki, G.B. Deane, M. Moskalik, Ph. Blondel, J. Tegowski, M. Blaszczyk Geophysical Research Letters, DOI: 10.1002/2014GL062859 retrieved from https://www.youtube.com/watch?v=6jqTm3TMGvI

17 GEUS - Geological Survey of Denmark and Greenland, Retrieved April 30, 2022, https://www.geus.dk

18 Félix Guattari, The Three Ecologies, Continuum Press, 1989.

19 GLISN Website, Retrieved April 30, 2022, http://glisn.info

20 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke University Press Books, 2016.

21 Haraway, "A Cyborg Manifesto: Science, Technology, and Socialist Feminism in the Late Twentieth Century," in Simians, Cyborgs and Women: The Reinvention of Nature, New York; Routledge, 1991, 1985;1991, 149-181.

22 The Icefjord Centre, About the House: The Exhibition, Retrieved April 30, 2022, https://isfjordscentret.gl/en/about-ilulissat-icefjord-centre/

23 Karsten R.S. Ifversen, 2021, Politiken, Arkitektur: 5 hjerter: Arkitektonisk lækkerbidsken opleves bedst på strømpesokker (October 2021), Retrieved April 30, 2022, https://politiken.dk/kultur/arkitektur/art8397563/Arkitektonisk-lækkerbisken-opleves-bedst-på-strømpesokker

24 Michael Jackson, Glaciers and climate change: Narratives of ruined futures, WIREs Climate Change, 2015, DOI: 10. 1002/wcc.1351.

25 AM Kanngieser, Rory Gibb, Sonifying the world in Sounding Place, Edited by Carolina Dougherty, Michelle Duffy and Theresa Harada,  Cheltenham: Elgar, 2019, 153-160.

26 Gregory Kramer, Bruce Walker, Terri Bonebright, Perry Cook, John H. Flowers, The Sonification Report: Status of the Field and Research Agenda, Faculty Publications, Department of Psychology, 1999, 444.

27 Melanie Klein, This changes everything, Simon+Schuster Inc., 2014.

28 Bernie Krause, The Great Animal Orchestra: Finding the Origins of Music in the Worlds Wild Places. London: Profile Books, 2012.

29  Dennis Overbye, Gravitational Waves Detected, confirming Einstein's theory, New York Times, 2016, Retrieved April 30, 2022, https://www.nytimes.com/2016/02/12/science/ligogravitational-waves-black-holes-einstein.html?searchResultPosition=1

30 Jacob Kirkegaard, Earside Out, Museum of Contemporary Art, Denmark: Isfald, 2015, Retrieved April 30, 2022, https://fonik.dk/works/isfald.html

31 JAC Studios, Projects: Ilulissat Isfjord Centre - The story of Ice, 2021, Retrieved April 30, 2022, http://jacstudios.dk/ilulissat-icefjord-centre-the-story-of-ice/

32 Elizabet Leane Jeffrey McGee (Eds), Anthropocene Antarctica, Perspectives from the Humanities, Law and Social Sciences, Routledge, 2019.

33 Dorte Mandrup, Work: Ilulissat Isfjord Centre, 2021, Retrieved April 30, 2022, https://www.dortemandrup.dk/work/ilulissat-icefjord-centre- greenland

34 Brian Massumi, Parables for the virtual: movement, affect, sensation, Duke University Press, 2002.

35 Thomas J. Mitchell, Alex J. Jones, Michael B. O'Connor, Mark D. Wonnacott, David R. Glowacki, Joseph Hyde, Towards molecular musical instruments: interactive sonifications of 17-alanine, graphene and carbon nanotubes, Proceedings of the 15th International Conference on Audio Mostly, Association for Computing Machinery, New York, NY, USA, 2020, 214–221, https://doi.org/10.1145/3411109.3411143

36 Marcus Nüsser, Ravo Baghel, The emergence of the cryoscape: Contested narratives of Himalayan glacier dynamics and climate change, In: Benjamin Schuler (ed.) Environmental and Climate Change in South and Southeast Asia, Leiden: Koninklijke Brill, 2014, 138–156, Https://doi.org/10.1163/9789004273221_007

37 Niels Bohr institute website: https://nbi.ku.dk/english/

38 Ben Orlove, Ellen Wiegandt, Brian H Luckman BH, The place of glaciers in natural and cultural landscapes, In: Orlove B, Wiegandt E,  Luckman BH (eds) Darkening Peaks: Glacial Retreat, Science, and Society, Berkeley, CA: University of California Press, 2008, 3–19.

39 Jussi Parikka, What is Media Archaeology? Cambridge: Polity Press, 2012.

40 Bryan C. Pijanowski, Luis J. Villanueva-Rivera, Sarah L. Dumyahn, Almo Farina, Bernie L. Krause, Brian M. Napoletano, Stuart H. Gage, Nadia Pieretti, Soundscape Ecology: The Science of Sound in the Landscape, BioScience, Volume 61, Issue 3, March 2011, 203–216, https://doi.org/10.1525/bio.2011.61.3.6

41 Evgeny A. Podolskiy, Toward the Acoustic Detection of Two‐Phase Flow Patterns and Helmholtz Resonators in Englacial Drainage Systems. Geophysical Research Letters 47, 2020, Retrieved http://dx.doi.org/10.1029/2020GL086951.

42 PROMICE Website, Retrieved April 30, 2022, https://www.promice.org

43 Eleven Movements of the Cryoscape Project Website, Retrieved April 30, 2022 from Eleven Movements of the Cryoscape Project Website, 2021, Retrieved April 30, 2022, https://icefjord.nn.r.appspot.com/

44  Zhigang Peng, Chastity Aiken, Debi Kilb, David R. Shelly, Bogdan Enescu, Listening to the 2011 Magnitude 9.0 Tohoku-Oki, Japan, Earthquake. In Seismological Research Letters, Vol. 83, Issue 2, 2012, 287–293, Seismological Society of America (SSA), https://doi.org/10.1785/gssrl.83.2.287

45 Andrea Polli, 2004, Heat and the Heartbeat of the city: https://vimeo.com/127884090

46 Andrea Polli, Atmospherics/Weather Works: A Spatialized Meteorological Data Sonification Project, Leonardo 38, 2005, 31–36, Retrieved from http://dx.doi.org/10.1162/leon.2005.38.1.31.

47 Andrea Polli, Heat and the heartbeat of the city: Sonifying data describing climate change, Leonardo Music Journal, 16, 2006, 44-45.

48 Andrea Polli, Joe Gilmore, Sonification of measured (National Oceanic and Atmospheric. Administration's Arctic program) and model weather data, Leonard Music Journal 5:34, 2006.

49 Marty Quinn, Research set to music: The climate symphony and other sonifications of ice core, radar, DNA, seismic, and solar wind data, Proceedings of the 7th International Conference on Auditory Display (ICAD01), Espoo, Finland, 2001.

50 Michael Quinton, Iain McGregor, David Benyon, Sonification of an exoplanetary atmosphere, Proceedings of the 15th International Conference on Audio Mostly, Association for Computing Machinery, New York, NY, USA, 2020, 191– 198, https://doi.org/10.1145/3411109.3411117

51 Michael Quinton, Iain McGregor, David Benyon, Sonification of Planetary Orbits in Asteroid Belts, Audio Mostly, Association for Computing Machinery, New York, NY, USA, 2021, 72–80, https://doi.org/10.1145/3478384.3478390

52 Audrey Ramming, GlacierHub: Acoustics of Meltwater Drainage in Greenland Glacial Soundscapes (April 2020), Retrieved April 30, 2022, https://glacierhub.org/2020/04/16/acoustics-of-meltwater-drainage-in-greenland-glacial-soundscapes/

53 Publication by Realdania, Bog om Ilulissat Isfjords-center: En ganske særlig bygning på et ganske særligt sted, 2021, Retrieved April 30, 2022, https://realdania.dk/projekter/ilulissat-isfjordscenter

54  Niklas Rönnberg, Sonification for Conveying Data and Emotion, Audio Mostly 2021, Association for Computing Machinery, New York, NY, USA, p.56–63, https://doi.org/10.1145/3478384.3478387

55 Ben Rubin, Hearld Thorp, Harbinger is a permanent public artwork by Ben Rubin and Jer Thorp commissioned for Brookfield Place in Downtown Calgary, Alberta, https://vimeo.com/250393598

56 Margaret Schedel, David R. Worrall, New Wor(l)ds for Old Sounds, In Organised Sound, vol 19, 2014, 1-3, doi:10.1017/S1355771813000356.

57 Isabelle Stengers, In Catastrophic Times: Resisting the Coming Barbarism, Open Humanities Press, 2009/2013.

58 Sverker Sörlin, Cryo-History: Narratives of Ice and the Emerging Arctic Humanities, The New Arctic, 2015, 327, Retrieved from http://dx.doi.org/10.1007/978-3-319-17602-4_24.

59 Anna Tsing, The Mushroom at the End of the World: On the Possibility of Life in Capitalist Ruins, NJ: Princeton University Press, 2015.

60 Sarah E. Truman, Stephanie Springgay, The primacy of movement in research-creation: New materialist approaches to art research and pedagogy, In M. Laverty, & T. Lewis (Eds.), Art’s Teachings, Teaching’s Art: Philosophical, critical, and educational musings, New York, NY: Springer, 2015, 151-164.

61 Salomé Voegelin, Listening to Noise and Silence: Towards a Philosophy of Sound Arts, NY: Bloomsbury, 2010.

62 Salomé Voegelin, Sonic Possible Worlds: Hearing the Continuum of Sound, NY: Bloomsbury, 2014.

63 Salomé Voegelin, The Political Possibility of Sound: Fragments of Listening, NY: Bloomsbury, 2018.

64 Chris Watson, 2003, https://chriswatsonreleases.bandcamp.com/track/vatnaj-kull

65 David Worrall, Sonification Design: From Data to Intelligible Soundfields, CHE: Springer, 2019.

Author Biographies

Danish-born artist Louise Foo creates interactive experiences at the intersection of sonic and visual arts, to facilitate novel modes of listening and engaging. Her recent work counts data-sonification of geological movements to document the melting of the Inland Ice, interplanetary musical explorations with sounding spheres and ambisonic music.

Jonas Fritsch, PhD, is Associate Professor in Interaction Design at the IT University of Copenhagen in the Department of Digital Design. He is head of the Affective Interaction & Relations (AIR) Lab and Head of the Design Research Section. His work revolves around a creative thinking of interaction design, design processes, experience philosophy and affect theory through practical design experiments with interactive sound and physical interfaces. 

Cosmo-Techno-Poiesis: Architecture of Environmental Control

Sebastian Gatz

University of Gothenburg, HDK-Valand Konstfack, Stockholm / sebastian.gatz@konstfack.se

Abstract

Architecture is a collective technological human practice to control local environments in order to protect the human body. Depending on the worldview and cosmology of the society which produces the architecture—increasingly the global worldview is rational, mono-technological and Western—this practice gives more or less space for nonhuman agency. This paper looks at ways to loosen the contemporary Western obsession with controlling the environment architecturally by exploring different forms of architectural memento mori (remember you must die). The act of willing (poiesis) protective-controlling architecture (technology) into the world, inside a certain worldview (cosmology), is explored through the conceptual entanglement of those three notions: cosmo-techno-poiesis. The paper concludes with an architectural example and a short summary.

Keywords

Architecture, natureculture, nonhuman, environment, ruin, decay, technophilia, posthumanism, transhumanism, memento mori.

DOI

10.69564/ISEA2023-55-full-Gatz-Cosmo-Techno-Poiesis

Architecture as Mono-Technological Boundaries

“While traditional architecture was capable of providing shelter from the environment, the advent of HVAC (Heating, Ventilating, and Air Conditioning) systems at the beginning of the twentieth century established the building envelope as a cocoon in which an alternative universe was maintained.”—Michelle Addington ¹

The architectural boundary is a place of intense negotiation. The layer between “inside” and “outside” does not just mediate between indoor and outdoor climate (the focus of most construction regulations) but also negotiates a psychological positioning of the inhabitant in relationship to their environment. Michelle Addington, dean of the University of Texas at Austin School of Architecture, problematizes the increasingly hermetically sealed architectural envelopes, which divorce the human body from its natural environment, in order to endorse the work of artist-architect Philip Beesley. ² His almost transhumanist sculptures rethink the engagement of humans with architectural boundaries through technological augmentation and almost life-like behaviors: “... Beesley's environments always question the idea of boundary, and in doing so, question the very nature of architecture.” ³

Architecture divides what is inside and what is outside, what is nature and what is culture. The inside is a space of control, while the outside is uncontrolled and hostile. The environment is either filled with archetypes of fear or with energies and weather phenomena not fitting into our energy calculations. The latter leads to increasingly higher energy standards, such as those for nearly zero-energy buildings (NZEB), ⁴ and with it prescribed wall insulation values and air-tightness factors.

Architect and artist Luis Berríos-Negrón uses the “green-house”—as an actual architectural typology and a metaphorical artistic device—to question nature's “outsider position” and to describe how architecture can manifest a human mindset towards nature:

Control and domination are expressed by placing nature, ironically, on the “inside,” where man thinks he can dominate life—to extract and accelerate, or decelerate and conserve growth—regardless of place or season. That site-less, modern, illusory power sandwiched between the industrial logic of translucent surfaces is that which cuts, represses, forgets, and fore-gets the realities of an habitable environment. ⁵

Architecture in that sense is not just a technological solution which mitigates energy losses but is also a manifestation of ideology which separates the human body from the natural environment—or even reality. The architectural envelope is where an increasingly questionable separation of nature and culture ⁶ is formalized and manifested—in physical actuality and in a metaphorical collective mindset. This paper looks at architectural (or at least theoretical-architectural: architectonic) ways out of nature-culture dichotomies while realizing at the same time that Haraway's idea of naturecultures—an endless conglomerate of entanglements of matter, stories, life and so forth—has practical “common sense” limits when it comes to buildings: Who in the Western world wants to remove the architectural separator between the human and its environment? The text looks at possible ways to loosen this boundary, by reintroducing nature as an architectural companion species. ⁷

According to artist-architect Frederick Kiesler, architecture is a form of technological environment which helps humans in their survival in the natural environment. Technology, according to him, is a form of social heredity, which has in it the long history of toolmaking: “Thus a contemporary chair, for instance, is the product of many generations of other tools for man to rest his body in fatigue. This is heredity in technology.” ⁸

What that means is that our built environment, or technological environment, is an archive of previous human-nature struggles and former knowledge production. Architecture is a technology which is placed between the human body and its environment. The formulation of this technological “machine” visualizes a society's relationship to the “actual” and the “imagined” natural environment. While actual local weather characteristics, material availability and cultural idiosyncrasies contribute to a vernacular architecture and its specific boundary conditions, the worldview of that society (cosmology) negotiates how much those boundaries are open to inside-outside mediation.

Yuk Hui coined the term cosmotechnics ⁹ in order to work on the problems that he sees in the world, one of them: An unquestioned mono-technological development which is rooted in a certain global worldview (cosmology). This one-dimensional development, according to him, contributes to the destruction of the planet and to the maintenance of fixed human-cosmos, human-nonhuman and nature-culture relations:

I gave a preliminary definition of cosmotechnics as unification between the cosmic order and the moral order through technical activities, in order to suggest that technology should be re-situated in a broader reality, which enables it and also constrains it. The detachment of technology from such a reality has resulted from the desire to be universalizing and to become the ground of everything. Such a desire is made possible by the history of colonization, modernization and globalization, which, being accompanied by its history of economic growth and military expansion, has given rise to a mono-technological culture in which modern technology becomes the principal productive force and largely determines the relation between human and non-human beings, human and cosmos, and nature and culture. The problems brought about by this mono-technological culture are leading to the exhaustion of resources and of life on earth and to the destruction of the environment, which are central to the discourse around the Anthropocene.¹⁰

Western architecture—as a form of technology to negotiate between humans and nature—follows the logic of what Hui describes: Mono-technological prescriptions, which are embedded in an unquestioned but slowly crumbling technoscientific cosmology, are dictated by law and leave no room for alternative cosmologies, techno-diversity or alternative engagements of humans with their environment. Or as Helmuth Trischler and Fabienne Will state it when talking about the Technochene / Technosphere: “Humans submit to the artifacts they have created themselves, shifting the responsibility for human-environment relationships to things.” ¹¹

The increasingly hermetically sealed borders of Western buildings do not solve the root problem of a troubled human-environment relationship. In fact, their purpose of creating borders between the human and its environment does exactly that: deepening the border between the human and its environment.

Poiesis: Willing Architecture into the World

Architecture, or in fact every form of natural or man-made matter, is a manifestation of invisible forces, entanglements and hidden networks of different human and nonhuman power struggles. Kiesler calls these “inter-acting” forces co-reality:

What we call “forms,” whether they are natural or artificial, are only the visible trading posts of integrating and disintegrating forces mutating at low rates of speed. Reality consists of these two categories of forces which inter-act constantly in visible and invisible configurations. This exchange of inter-acting forces I call CO-REALITY , and the science of its relationships, CORREALISM. The term “correalism” expresses the dynamics of continual interaction between man and his natural and technological environments. ¹²

Specifically, the term inter-action sounds very familiar and foreshadows feminist theorist and physicist Karen Barad's intra-action by a few decades:

The neologism “intra-action” signifies the mutual constitution of entangled agencies. That is, in contrast to the usual “interaction,” which assumes that there are separate individual agencies that precede their interaction, the notion of intra-action recognizes that distinct agencies do not precede, but rather emerge through, their intra-action. It is important to note that the “distinct” agencies are only distinct in a relational, not an absolute, sense, that is, agencies are only distinct in relation to their mutual entanglement; they don't exist as individual elements. ¹³

Architecture comes into being when the time is ripe. All inter-acting, or intra-acting, forces come together: Something or someone wants it to exist and this something or someone has the means to make it exist. Architecture in that sense is a form of individual or collective artistic poiesis—the will to bring into being. Poiesis is where artwork, artist, receiver, idea and matter come together. ¹⁴ The concept overlaps with the foundational concept of magic, where willing leads to manifestations in the world or control over the ego. Will, in the magical sense, is neither good nor bad but depends on the intention of the magician. ¹⁵ The size, construction time, material quality and the technological standard of architecture is directly related to the intensity with which one or many will. An Egyptian pyramid or a New York City high-rise is brought into the world with a considerable bigger amount of willing than a hut made from leaves, or a tent bought in the supermarket (while its existence as a typology comes with a long history of willing people as well). The willing of buildings, and accordingly the manifestation of them, is in itself not a bad thing. To will protection from the environment's “attacks,” for example, comes with a different amount of control fantasies (or karma) than the willing of the next biggest building in the world. Intention is key. We can consider the built environment (technical environment)—the existing stock of matter which divides human bodies from their natural environment—to be the product of human poiesis or their willing to keep nature at a distance.

Another interesting question is if architecture – as a global technology – is in fact creating or perpetuating itself. Peter Haff argues that the Anthropocene is essentially built on the technoshpere (the technological conglomerate of human activity) which is out of the control for the individual and therefore collective human:

[W]e abandon the apparently natural assumption that the technosphere is primarily a human-created and controlled system and instead develop the idea that the workings of modern humanity are a product of a system that operates beyond our control and that imposes its own requirements on human behavior. The technosphere is a system for which humans are essential but, nonetheless, subordinate parts. As shorthand we can say that the technosphere is autonomous. This does not mean that humans cannot influence its behavior, but that the technosphere will tend to resist attempts to compromise its function. ¹⁶

In a sense, architecture—as part of the technosphere which perpetuates the Anthropocene—can be seen as a nonhuman entity which wants to be fed with more technology, more matter and more human will or it fights back: a Golem, Frankenstein's monster.

How much are we in control of our architectural (techno)logic and how much control should it have over the environment and us?

Memento Mori: Giving-Up Control

How much control is too much control? The following part will concentrate on the concept of death and life as metaphorical transient categories of architecture. Newly built architecture can be considered being “born” out of human will or poiesis and in total control of the inner—and therefore psychologically the outer—environment, while “neglected” buildings can be considered to be in the process of dying and therefore the human is less in control of inner our outer environments. What productive states of architecture exist between birth and death, between control and giving up control, which allow for nonhuman agency and provide human habitats at the same time?

While the transhumanists want to overcome human mortality ¹⁷ and the limits of the planet (multi-planetary thinking can be considered to be the big brother or sister of individual transhumanism), philosophy professor Patricia MacCormack argues, from an ahuman perspective, for a way out of the world's problems through the embracing of death:

Where I want to push things is through the incorporation of the death of the species in actuality and to cease the actual death of the nonhuman other as inherently joyful affirmative qualities of protest, and this I see exemplified in two activist philosophies which result in two material actions: human extinction and animal abolition. ¹⁸

Ahumanism and transhumanism can be seen as being on an oppositional spectrum with different degrees of human control desires over the nonhuman other. Critical / cultural / philosophical posthumanism fits better to the former, in terms of giving up human control or acknowledging more agency of nonhumans ¹⁹ while cybernetic posthumanism overlaps with the latter. ²⁰  While MacCormack advocates the slow phase-out of human control over the world ²¹ the transhumanists want to achieve a god-like status, which allows them to be in total control of life itself. ²² One can argue that both, the voluntary species suicide and the conquest of death, is hubristic and therefore human-centric. The following part explores steps on the gradient between both extreme positions in order to generate productive ways to look at architectural production and advocates for a Buddhist-like middle way of architectural environmental control.

The Anthropocene exists, at least partially, because of an unbalanced human understanding of how much nature control is the right amount. Technology is a gift which allows us to take what we need—but we thought we needed it all. The following part investigates some concepts, which are seen as fruitful for an architectural discourse, and which allow nonhuman agency and try to avoid total environmental control. Which states of architecture exist on the spectrum of species suicide to eternal life? Our Western architectural thinking is increasingly moving towards the latter. While houses used to be built out of natural, available materials which would decay at some point or another, today we (try to) build for eternity. While the killing of our building stock (explosion) would be similar to MacCormack's extreme, the other extreme can be found in contemporary construction regulations: the denial of death and the desire for eternal life and absolute environmental control.

Here, we will look at the idea of an architecture of decay as a form of memento mori (remember you must die). Decay here is shown as a one-directional entropy of architecture but hopes to reopen the thinking for an architecture of imperfection, transient form, openness, open-endedness and flexibility (openness to agency of others, ready to give up human control).

The transient nature of decay, decomposition ²³ and wabi ²⁴—or its digital counterpart: the glitch—can be seen as forms of nonhuman agency. When talking about the historical Japanese tea master Sen no Rikyū and his understanding of wabi, Rumiko Handa, professor of architecture at the University of Nebraska-Lincoln, states the following:

There are a number of instances that demonstrate Rikyū’s preference for impermanence, which is important because it shows the limitation of human control in comparison to natural forces. Rikyū’s desire to submit to forces beyond human control extends to allowing the artifact not to function in its primary utility. ²⁵

Rosa Menkman, when speaking about glitches as machine-revealing entities, acknowledges nonhuman agency in a similar fashion:

A glitch is the most puzzling, difficult to define and enchanting noise artifact; it reveals itself to perception as accident, chaos or laceration and gives a glimpse into normally obfuscated machine language. Rather than creating the illusion of a transparent, well-working interface to information, the glitch captures the machine revealing itself. ²⁶

While the medium for wabi and glitch is different, they share the similarity of acknowledging and appreciating nonhuman agency—giving up human control. One could say that the former is engaging with natural nonhuman agencies while the latter is focusing on man-made (technological) non-human agencies. Both reveal a hidden “true reality” which is outside of the control of humans.

Both, the glitch and the concepts related to wabi, work only in an equilibrium of the right amount of human control and nonhuman agency at the same time. Leaning too much on either side of the control spectrum, from human to non-human agency, can make the system collapse. Similar to a ruin which is only acknowledged as such if some parts are still standing and some damage exists. ²⁷ The nonhuman in the glitch and in the idea of wabi can be seen as a form of transient or, what political theorist Jane Bennett calls, “vibrant matter” ²⁸ or as a form of intra-action of time, space and matter. ²⁹ And maybe it is here where the true problem of the Anthropocene is located: time, or better the non-acceptance of time and its transient nature. Our scientific technological rational Western cosmology suggests being always on top of the invisible matterings, intra-actions and vibrant matters. Glitches reveal the malfunctioning of the man-made system which forces one to acknowledge one's own death. The Western people want to control time or better their own death and the decay of the body—due to a lacking enchanted cosmology. ³⁰ In the same way that architecture is a representation of our collective cosmological worldview, current architectural practices as technologies to control nature – symbolize, or manifest, our collective worldviews and desires. A current desire is to maintain or increase control over the world and to conquer death, as the last natural entity, which is clearly visible in the aforementioned transhuman interpretation of posthumanism and in the current direction towards becoming a multi-planetary species. The denial of planetary limits can be seen as a form of denying any end to growth (aka death).

In order to propose alternative ways to think architecture outside of the mono-technological cosmology of total environmental control the author would like to propose to think—as many others have before ³¹—architecture as ruins with different stages of decay. Similar to the different stages of a human body's decomposition ³² architecture undergoes—if left to the natural forces—different steps of decay as well: from finished (alive) to ruined (skeletonized) to vanished. Can this, ironically human-centric, model and understanding of decomposition help to broaden an architectural understanding which is neither technophile nor morbidly romantic of the past and its ruins? What can a building's skeleton be used for? Which qualities does a moldy building still have? No skin but stable bones? While the decay of a body conventionally goes—despite what transhumanists wish—only in one direction (entropy), a building's ruination process is more ambivalent, as the example of Architecten de Vylder Vinck Taillieu's Caritas project shows:

[The architects] proposed to keep the already half-demolished building as it was and to make it accessible as a public space. Since the roof had already been removed and part of the wooden floor were gone, the wind and rain would further corrode the construction, causing the wooden planks to rot and the brick walls to crumble bit by bit. The architects limited themselves to a number of interventions to stabilize the building, while also further facilitating the process of decay over time. ³³

This form of architectural engagement with the ruin allows humans to inhabit the space while at the same time gives agency to natural nonhuman others. Architect and researcher Bart Decross calls this form of agency—when talking about John Ruskin's advocacy for imperfection—“vital materiality.” ³⁴ One can only speculate if the architectural experiencing of controlled decay reminds the visitors consciously or subconsciously of their own mortal limits (memento mori) and if this in return could contribute to a humbler positioning of the self in relation to the cosmos.

The decay of the building can be considered only to be a movement towards an end from a human or building point of view. The decay of matter, human or building, is where vital materiality and vibrant matter meet. While every biotic body dies from the moment it is born, it can be said to be only a death from the point of view of that body. The final “death” is the moment when it serves as energy for other natural processes, as food for someone else, as the following description by research scientist and forensic anthropologist Arpad Vass illustrates:

I came to the conclusion, somewhat facetiously, that with the exception of micro-organisms living in deep-sea vents, every micro-organism known is involved in some aspect of the human decompositional cycle from Acetobacter to Zooglea. While many of the organisms isolated come from the bowel and respiratory tract, literally hundreds of species are involved in the decompositional process and decomposition would not progress without them. ³⁵

Summary

Architecture is a form of technology which helps society and the individual to control their immediate environment. The intensity or style of control, in the contemporary West, tries to remove nonhuman agency from the equation in order to obtain maximal control over natural processes. This control fantasy is based on a shared technoscientific cosmology. The idea of cosmo-techno-poiesis points at the complex entanglement of cosmology, technology and the human act of creating. Manifesting architecture, as a technology to keep nature at a distance, is not just an act which is ethically self-evident or true, but also an act of taking part in the creation or expansion of worldviews and therefore cosmologies. Architectural production is cosmo-techno-poietic production and should not be reduced to the creation of technological environments for the sake of excluding natural environments.

Wabi, glitch, ruin, imperfection and similar concepts of decay—summed up as memento mori—can potentially be engaged in architectural, actual and metaphorical, production which acknowledges nonhuman agency and breaks with human technoscientific control fantasies.

The current call for non-extractive forms of architectural production and degrowth thinking ³⁶ demands new modes of thinking architectural (vital) matters. This paper is a call for technodiversity and new cosmologies which see technology as a tool of the middle way and not of total environmental control.

References

1 Michelle Addington, “Architecture of Contingency,” in Hylzoic Ground: Liminal Responsive Architecture: Philip Beesley, ed. Pernilla Ohrstedt and Hayley Isaacs, Riverside Architectural Press, 2010, 68-69.

2 Addington.

3 Addington, 74.

4 Commission Recommendation (EU) 2016/1318 of 29 July 2016: On Guidelines for the Promotion of Nearly Zero-Energy Buildings and Best Practices to Ensure That, by 2020, All New Buildings Are Nearly Zero-Energy Buildings, 2016.

5 Luis Berríos-Negrón, Breathtaking Greenhouse Parastructures: A Supplement to the Arcades Project from a Caribbean Perspective [and a Call for a Careful Practice of Epistemológica] (Konstfack Collection, 2020), Exposé.

6 Donna Jeanne Haraway, The Companion Species Manifesto: Dogs, People, and Significant Otherness, vol. 1 (Prickly Paradigm Press Chicago, 2003, 6-7.

Haraway, 23.

8 Frederick Kiesler, “On Correalism and Biotechnique: A Definition and Test of a New Approach to Building Design,” Architectural Record 86, 1939, 61.

9 Yuk Hui, “Cosmotechnics as Cosmopolitics,” e-flux journal, no. 86, 2017, https://www.e-flux.com/journal/86/161887/cosmotechnics-as-cosmopolitics/.

10 Yuk Hui, “COSMOTECHNICS,” Angelaki 25, no. 4, 2020, 2, https://doi.org/10.1080/0969725X.2020.1790828.

11 Helmuth Trischler, Fabienne Will, “Technosphere, Technocene, and the History of Technology”, Icon, London, England, 2017, p.23, 12, issn: 1361-8113.

12 Kiesler, “On Correalism and Biotechnique: A Definition and Test of a New Approach to Building Design,” 61.

13 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, 2007, 33.

14 Derek H. Whitehead, “Poiesis and Art-Making: A Way of Letting-Be,” Contemporary Aesthetics (Journal Archive) 1, no. 1, 2003.

15 Israel Regardie, The Tree of Life: A Study in Magic, Samuel Weiser, Inc., 1995, 126.

16 Peter Haff, “Humans and Technology in the Anthropocene: Six Rules,” The Anthropocene Review 1, no. 2, 2014, 127.

17 Hava Tirosh-Samuelson, “Engaging Transhumanism,” in H+/-: Transhumanism and its Critics, ed. Gregory R. Hansell and William Grassie, Metanexus Institute, 2011, 39-42.

18 Patricia MacCormack, The Ahuman Manifesto: Activism for the End of the Anthropocene, Bloomsbury Publishing, 2020, 142-143.

19 Francesca Ferrando, “Posthumanism, Transhumanism, Anti-humanism, Metahumanism, and New Materialisms,” Existenz 8, no. 2, 2013, 29-30.

20 Cary Wolfe, What is Posthumanism?, vol. 8, University of Minnesota Press, 2010, xiii.

21 MacCormack, The Ahuman Manifesto: Activism for the End of the Anthropocene, 140-149.

22 Tirosh-Samuelson, “Engaging Transhumanism,” 42-46.

23 Ami Ronnberg, Kathleen Martin, The Book of Symbols, Taschen Cologne, 2010, 764-765.

24 Rumiko Handa, Allure of the Incomplete, Imperfect, and Impermanent: Designing and Appreciating Architecture As Nature, Routledge, 2014, 153-168.

25 Handa, 163.

26 Rosa Menkman, The Glitch Moment(um), Institute of Network Cultures, 2011, 29-30.

27 Brian Dillon, “Introduction: A Short History of Decay,” in Ruins, ed. Brian Dillon, Whitechapel Gallery / The MIT Press, 2011, 11.

28  Jane Bennett, Vibrant Matter, Duke University Press, 2010, vii-xi.

29 Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, 141.

30 Christopher Partridge, The Re-Enchantment of the West: Alternative Spiritualities, Sacralization, Popular Culture and Occulture, vol. 1, London, T&T Clark International, 2004, 8-11.

31 Dillon, “Introduction: A Short History of Decay.”

32 Arpad A. Vass, “Beyond the Grave: Understanding Human Decomposition,” Microbiology Today 28, 2001, 190-192.

33 Bart Decroos, “How Gothic is Contemporary Architecture? The Appreciation of Craftmanship as a Ruskinian Aesthetic of Imperfection,” in Thinking-Making: When Architects Engage in Construction [Penser-Faire: Quand des Architectes se Mêlent de Construction], ed. Pauline Lefebvre, Julie Neuwels, Jean-Philippe Possoz, Éditions de l'Université de Bruxelles, 2021, 125.

34 Decroos, 118.

35 Vass, “Beyond the Grave: Understanding Human Decomposition,” 192.

36 Space Caviar, ed., Non-Extractive Architecture: On Designing without Depletion, vol. 1, V-A-C Press / Sternberg Press, 2021.

Bibliography

Addington, Michelle. “Architecture of Contingency.” In Hylzoic Ground: Liminal Responsive Architecture: Philip Beesley, edited by Pernilla Ohrstedt and Hayley Isaacs, 66-75. Riverside Architectural Press, 2010.

Barad, Karen. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning. Durham, 2007. Bennett, Jane. Vibrant Matter. Duke University Press, 2010. Berríos-Negrón, Luis. Breathtaking Greenhouse Parastructures: A Supplement to the Arcades Project from a Caribbean Perspective [and a Call for a Careful Practice of Epistemológica]. Konstfack Collection, 2020.

Decroos, Bart. “How Gothic is Contemporary Architecture? The Appreciation of Craftmanship as a Ruskinian Aesthetic of Imperfection.” In Thinking-Making: When Architects Engage in Construction [Penser-Faire: Quand des Architectes se Mêlent de Construction], edited by Pauline Lefebvre, Julie Neuwels and Jean-Philippe Possoz. Éditions de l'Université de Bruxelles, 2021. Dillon, Brian. “Introduction: A Short History of Decay.” In Ruins, edited by Brian Dillon. Whitechapel Gallery / The MIT Press, 2011.

Ferrando, Francesca. “Posthumanism, Transhumanism, Antihumanism, Metahumanism, and New Materialisms.” Existenz 8, no. 2 (2013): 26-32.
Haff, Peter. “Humans and Technology in the Anthropocene: Six Rules.” The Anthropocene Review 1, no. 2 (2014): 126-136. Handa, Rumiko. Allure of the Incomplete, Imperfect, and Impermanent: Designing and Appreciating Architecture As Nature. Routledge, 2014.

Haraway, Donna Jeanne. The Companion Species Manifesto: Dogs, People, and Significant Otherness, vol. 1. Prickly Paradigm Press Chicago, 2003.

Hui, Yuk. “Cosmotechnics as Cosmopolitics.” e-flux journal, no. 86 (2017), https://www.e-flux.com/journal/86/161887/cosmotechnicsas-cosmopolitics/.
Hui, Yuk. “COSMOTECHNICS.” Angelaki 25, no. 4 (2020): 1-2, https://doi.org/10.1080/0969725X.2020.1790828.

Kiesler, Frederick. “On Correalism and Biotechnique: A Definition and Test of a New Approach to Building Design.” Architectural Record 86 (1939): 60-75.
MacCormack, Patricia. The Ahuman Manifesto: Activism for the End of the Anthropocene. Bloomsbury Publishing, 2020. Menkman, Rosa. The Glitch Moment(um). Institute of Network Cultures, 2011.

Partridge, Christopher. The Re-Enchantment of the West: Alternative Spiritualities, Sacralization, Popular Culture and Occulture, vol. 1. London: T&T Clark International, 2004.
Regardie, Israel. The Tree of Life: A Study in Magic. Samuel Weiser, Inc., 1995.

Ronnberg, Ami and Martin, Kathleen. The Book of Symbols. Taschen Cologne, 2010.
Tirosh-Samuelson, Hava. “Engaging Transhumanism.” in H+/-: Transhumanism and its Critics, ed. Gregory R. Hansell and William Grassie. Metanexus Institute, 2011.

Trischler, Helmuth and Will, Fabienne. “Technosphere, Technocene, and the History of Technology.” Icon (London, England) 23 (2017): 1-17, issn: 1361-8113.
Vass, Arpad A. “Beyond the Grave: Understanding Human Decomposition.” Microbiology Today 28 (2001): 190-192. Whitehead, Derek H. “Poiesis and Art-Making: A Way of Letting- Be.” Contemporary Aesthetics (Journal Archive) 1, no. 1 (2003). Wolfe, Cary. What is Posthumanism?, vol. 8. University of Minnesota Press, 2010.

Author Biography

Sebastian Gatz is an architect, artist and trained car mechanic, who works at the intersection of art, architecture and technology. Currently he is doing a PhD in Fine Arts at Konstfack – University of Arts, Crafts and Design in Stockholm. His research combines fictocritical and posthuman methods to explore human-nature-technology relationships. He has an interest in experimental metaphysics, degrowth practices and digital fabrication. Previously he worked and taught at The Royal Danish Academy of Fine Arts at the Centre for Information Technology and Architecture (CITA) in Copenhagen, where he worked with Artificial Intelligence, Robotic Fabrication and Robot-Plant-Hybrids. 

Welcome to the Metaverse: Hacking Affect in Immersive Documentary to Increase Critical Big Data Literacy

Brett Gaylor, Kate Hennessy

Simon Fraser University Vancouver, Canada
brett_gaylor@sfu.ca, kate_hennessy@sfu.ca

Abstract

This essay describes the process and background of the augmented reality documentary Welcome To The Metaverse. The authors situate themselves as research-creation practitioners and ethnographers exploring immersive documentaries as a method for increasing critical big data literacy, an emerging public-facing discipline concerned with pedagogical approaches to understanding power structures embedded in artificial intelligence and big data systems. The authors respond to research from critical big data literacy scholars and anthropologists using multimodal practices to foreground the ways that the digital tools used to create these works need further critical reflection. The authors explore how facial recognition and other aspects of augmented reality can be detourned or cultured-jammed in similar ways to media of previous decades, positioning immersive documentary as a method for hacking affect toward greater awareness of the multifarious politics of the Metaverse. 

Keywords

Documentary, augmented reality, facial recognition, artificial intelligence, metaverse, culture jamming, multimodal anthropology, surveillance capitalism, privacy, Facebook.

DOI

10.69564/ISEA2023-56-full-Gaylor-et-al-Welcome-to-the-Metaverse

Introduction

“Welcome! Blink if you agree to the terms and conditions!” an enthusiastic voice instructs you as your face is scanned. You blink. Your eyes are replaced with the celebration emoji 🎉. Music begins, and the same voice exhorts, “We’re so excited you’ve agreed to join The Metaverse!”

These are the opening fifteen seconds of Welcome To The Metaverse (WttM),¹ an augmented reality (AR) documentary that author Gaylor created for Instagram in collaboration with HoloLabs (figure 1).

For this experience to work, billions of research dollars and millions of faces have been collected to train artificial intelligence systems that can detect the contours of your face and whether your eyes have closed. Before this experience could make its way to your phone, lawsuits were filed and settled over whether companies have the right to gather biometric data without consent (they do not).² While Gaylor and HoloLabs created the documentary, the Facebook corporation announced a rebranding as “Meta” and a $10 billion research agenda focused on the “metaverse,” a virtual environment blending the real world with computer graphics.³ Welcome To The Metaverse satirically explores why Meta desperately wants you and your data to succumb to The Metaverse.

Figure 1. Screenshots of the experience.
Image: B. Gaylor. 

This paper situates Welcome To The Metaverse as a work of research-creation⁴ in cross-disciplinary conversation with artists, scholars and activists exploring how communications technologies simultaneously reproduce unequal and oppressive relations of power and can be used to raise public awareness of those dynamics. We walk readers through chapters of the AR documentary to demonstrate immersive documentary as a medium uniquely situated to raise awareness around issues of privacy and surveillance capitalism⁵ that are at play in the Metaverse.

Building on Gaylor’s earlier interactive documentary works such as Do Not Track⁶ and Discriminator,⁷ this paper positions Welcome To The Metaverse as a work of research-creation that brings into practice a call by multimodal anthropologists for a new reflexive turn in the discipline that engages with the material politics of the technologies used to create and exhibit their work.⁸, ⁹ This immersive AR documentary experience borrows tactics from culture jammers and media pranksters who use the technologies they critique to demonstrate their problematic nature. Grounded in these traditions, Welcome To The Metaverse aims to build critical big data literacy by alerting users to new ways that their data can be extracted and exploited.

This essay highlights how Welcome To The Metaverse (WttM) operates as a counter-narrative that uses storytelling, humour and interactivity to instigate critical reflection in users. Immersive documentary as research-creation practice represents a promising methodology for raising critical data literacy. As an approach to an emerging anthropology of the multimodal, we further present this work as an instance of augmented ambivalent anthropology in practice.

Augmented Ambivalent Anthropology

Despite the hype, the Metaverse remains a speculative media whose qualities we divine from science fiction, corporate videos, and early virtual and augmented reality works. Most people haven’t tried it, and most of the experiences from SciFi and industry are still years away. The term “Metaverse” first appeared in Neal Stephenson’s science fiction novel Snow Crash and described a virtual world that inhabitants spend time in to escape the reality of failed states, ecological collapse and corporate rule. As the cleverly named Hiro Protagonist declares: “When you live in a sh@!#t hole, there’s always the Metaverse.”¹⁰

Our times of plague, war and climate catastrophe can feel like the future Stephenson dreamed up. During the Covid-19 pandemic, Meta’s Quest headset sales increased 350%.¹¹ It can certainly be a welcome diversion to visit the International Space Station, exercise at Machu Picchu, or experience a fantastical immersive world. For creators, the creative possibilities of VR to transport audiences to new locations and experience things that would be otherwise impossible are often enough to overcome the reluctance to support a company such as Meta.

These mixed feelings are precisely the sort that anthropologists Astacio et al. encourage makers to embrace in Multimodal Ambivalence: A Manifesto for Producing in S@!#t Times.⁹ The authors of this manifesto ask makers and anthropologists who are invested in research mobilizing new multimodal tools to pair their enthusiasm for new media with critical analysis of the political economies that underlie them. They ask us to pay particular attention to digital productions, which are often valorized for their ability to facilitate participation from under-represented communities in public discourse.

Multimodal ambivalence emerges in conversation with anthropologists such as co-author Hennessy’s work with Takagawara et al. who argue that there is nothing inherently liberatory about the use of multimodal tools in anthropology. Playfully drawing on Sarah Ahmed’s feminist critique of bad habits¹² and Pierre Bourdieu’s habitus,¹³ they identify Bad Habitus as the unpleasant feeling that unavoidable implication in ubiquitous big data environments and material infrastructures causes for them⁸. Through this ambivalent orientation toward the tangled politics of multimodal anthropology, they warn that uncritical use of digital technologies can reinforce racial inequality and extractivism by normalizing the problematic power structures that digital infrastructures exacerbate, online and off. They further point to research-creation as a productive methodology for an anthropology of the multimodal that critically engages the tools being used for greater awareness of their wider politics and impacts.

WttM is Gaylor’s attempt to satirically engage the public in a dialogue around the darker side of the Metaverse. The narrator is ridiculously enthusiastic, while the actions he suggests (“scan your body!”) feel dystopian. The documentary experience doesn’t aim to generate outrage—it seeks to create mixed feelings and ambivalence. It does this by attempting to amuse users with interactive moments and animations, paired with allusions to Meta’s attempts to gather increasingly detailed data about our homes and bodies.

FaceJams & Semiotic Resistance

WttM explores the problematics of the Metaverse in one of the most realized versions of the Metaverse available: Meta’s Instagram face filters. These filters will be recognizable to the billions of users of Instagram, Tiktok, Messenger Kids, or Snapchat: a user’s visage is augmented by computer graphics that track the position of their facial features.

WttM is novel in that the face filter changes to match the spoken narration. It operates as a hybrid between linear media, such as film or radio, and interactive and immersive media. We will refer to WttM as a “documentary experience” for this essay. By using affordances and tropes that feel familiar to social media users, Gaylor attempts with this documentary experience to introduce messages and conversations that they might not receive otherwise. In Umberto Eco’s terms, he is waging Semiotic Guerrilla Warfare: subverting the communication “chain” by leaving the “channel” intact (Instagram) and inserting into the “message” (the documentary) a different “code.”¹⁴ This code is ambivalence—an unease about social media, tech platforms and surveillance.

The goal of the project is to use the sense of unease that the documentary experience creates as a contribution to the emerging field of Critical Big Data Literacy, a public-centred pedagogy focused on the growing centrality of data and critical examinations of these repercussions. In her introduction to the field, Ina Sander studied several online resources. She found that interactive creations were particularly suited to transmit big data issues because the media could be personalized and would best compete for attention.¹⁵ Welcome To The Metaverse is the first immersive documentary experience to respond specifically to these findings and create new works with this knowledge in mind. 

Contextual Integrity

To create the Metaverse, we first collected millions of photos of our users on the vintage website known as Facebook.”, the narrator declares as the next chapter begins. “We used these faces to create an algorithm that can detect yours!”

This narration refers to practices that Facebook has engaged in to train artificial intelligence systems. In 2021, residents of Illinois were awarded $650 million to settle a class action against Facebook, which had trained a facial recognition algorithm using their photos without consent. ²  In addition to violating Illinois’ recently passed privacy laws, Facebook transgressed what scholar Helen Nissenbaum refers to as contextual integrity.¹⁶ This framework judges whether information sharing is appropriate based on “the type of information in question, about whom it is, by whom and to whom it is transmitted, and conditions or constraints under which this transmission takes place” (2004, 839).

This is essential nuance: while many users will describe uses of their data as “creepy,”¹⁷ a fuller description of inappropriate data flows is necessary to hold tech platforms accountable.
Contextual Integrity recognizes that we might be willing to share certain information in specific contexts: we would not like intimate photos texted to a partner to be shared with our boss, for example. WttM users experience this firsthand when they see pictures of someone else's face, stolen from Facebook, on their own face (figure 3).

It is this notion of a violation of contextual integrity that the creators hoped WttM would impart, and where the work builds on and at the same time departs from subversive efforts in other media.
The goals of WttM parallel Eco’s hope for “the constant correction of perspectives, the checking of codes, the ever-renewed interpretations of mass messages.”¹⁴ So, too, did those practicing critical remix on the media of the time—the so-called “culture jammers” of the 1980s and 90s who inserted counterculture messaging into billboards, radio and television broadcasts and print media.¹⁸ These media, however, had fundamentally different characteristics than our present and future digital media. Consider an advertisement on a traditional billboard, a “one-to-many” media, in which a single message passes through a channel and is received in an identical fashion by audiences. Contrast this with an advertisement on Instagram, where the ads are chosen based on monitoring behaviour on the platform and predicting which messaging users are likely to respond to. Each will be viewed on a specific device, in a different location, and in specific language.

In the Metaverse, this trend is magnified—everything about your context, from your physical characteristics to location to your species, can be customized by you and personalized by advertisers. WttM applies principals of culture jamming to your specific digital context—what we call context jamming. This approach follows recommendations by Ina Sander What Is Big Data Literacy to increase the use of personalization in education resources, an approach well suited to networked documentaries such as WttM in which users are already using their own devices and are logged in with their own Facebook/Meta accounts.

“Thanks to the pioneering work on the Facebook timeline, data scientists now understand content that generates strong emotions keeps you engaged longer!”, the narrator says to advance the next chapter of WttM. This refers to research done by Facebook engineers¹⁹ who manipulated the type of content users would receive in their timeline to understand whether prolonged exposure to posts with strong emotions would lead to more engagement on the platform (it does). Here again the work creates a scenario where contextual norms have been violated: it is unlikely that anyone sharing an emotional Facebook post imagined that a data scientist would use it to measure “emotional contagion.”

Both this story of manipulation, and the use of photos without consent, are both real stories of harm by Facebook/Meta. Having real, relatable accounts such as this was also a recommendation drawn from Ina Sander’s research, and the purpose of these stories in WttM is to cultivate ambivalence, to acknowledge our bad habitus, to incite curiosity and seed doubt as to whether these technologies are operating in the best interest of users. 

Hacked Affect

In the next chapter of WttM, viewers are presented with text describing how much money Meta CEO Mark Zuckerberg has made in the few minutes that the documentary has been playing. The narrator explains what is happening: “Here is how much money Mark Zuckerberg made while you were watching this!” 

Figure 2. A screengrab of an effect triggered by a smile in Welcome To The Metaverse. Image: Hololabs.

An analogue meter appears over the user's face. By offering emotions as programming inputs (figure 2), Meta is attempting to normalize the concept of emotions as a universally consistent and measurable data point. This practice is known as affect recognition. As Crawford points out, facial recognition seeks to recognize individual faces, affect recognition aims to identify universal emotions on any face.²⁰

Affect recognition is currently deployed across many industries for a variety of uses, from monitoring emotional engagement during job interviews²¹ to assessing student reactions during lectures²² to detecting nervous terrorists in airports.²³ These capabilities are sometimes developed internally by companies or purchased as a service from companies. One such company is Affectiva, an MIT incubated startup that uses deep learning to offer emotional insights gleaned from training an AI on the expressions of 10 million people in 87 countries.²⁴ If a basic set of emotions is all a developer requires, affect detection is available in most standard facial recognition suites such as Rekognition by Amazon²⁵ or Face API by Microsoft.²⁶

With WttM, we are subverting Meta’s attempts at normalizing this practice by introducing skepticism as to the ability of software to detect emotion. While Meta presents affect detection to developers as a trustable input on the same level as a mouse click or text entry, recognizing emotions by studying the face is, in fact, a controversial practice. Indeed, the very epistemological basis of associating affect with facial expressions, developed by Paul Ekman in the 1960s, has been challenged by psychologists and anthropologists as being methodologically unsound.²⁷ 

Figure 3. Photographs of facial expressions from Facial Expressions of Emotion – Stimuli and Tests dataset (FEEST). Image: Paul Ekman

This critique is based on Ekman establishing six emotion types: joy, anger, disgust, sadness, surprise, and fear, which he developed by photographing posed actors displaying caricatures of these emotions, claiming they were universal, and subsequently measuring test subjects against them (figure 3).

Anthropologist Ruth Leys notes that the central flaw in Ekman’s methodology is its recursive logic: the photographs are assumed to be universal because they are free of cultural bias and culturally unbiased due to their universality.²⁸ Problems of this nature, where erroneous data lead to incorrect assumptions, are perfect candidates for artificial intelligence systems to make worse. Yet despite this, engineers began giving the task of comparing a subject's facial expression to photos in datasets such as FEEST (figure 7) to speed up and automate the process. As Kate Crawford notes in Atlas of AI, recognizing affect is a task that has been given to AI systems not because they are suited to the job but because the (weak) theory was suitable for what the tools could do—detect patterns. ²⁰

These systems are deployed on a global scale, such as the Screening of Passengers by Observation Techniques (SPOT) program of the US Transportation Security Administration, which uses Ekman’s techniques to “detect” nervous flyers and flag them as potential terrorists.²³ Meanwhile, a comprehensive study of peer-reviewed science on affect recognition found no evidence that algorithms can detect a person's internal emotions. The research team warned, “very little is known about how and why certain facial movements express instances of emotion, particularly at a level of detail sufficient for such conclusions to be used in important, real-world applications.” ²⁸

“Show us you’re happy and we’ll move on to the next chapter!” the narrator asks at the end of the sequence. WttM asks users at this point to perform what researchers have proven—an inner emotional state cannot be inferred by facial expression. After learning how much Mark Zuckerberg made in the minute or so of the experience (approximately $13,000), a user is more likely to be annoyed than happy. Yet they must smile to continue. Smile hard, in fact—the program intentionally delays the measurement of their smile to give the impression that they must smile more.

“Try harder to be happy!” the narrator implores and rewards them with pretend cryptocurrency once the smile registers. This immediate feedback from the system after having their affect recorded follows Ina Sander’s recommendation that critical big data literacy resources should be interactive—users should be required to make choices within the experience that influence the outcome.¹⁵ Despite extensive literature around affect detection, there is a paucity of public-facing contributions that aim to illuminate its origins and problematics. Rather than a simple exhortation for users to protect their data from thieves, as many simplistic public service messages around privacy suggest, WttM is introducing a feeling of unease around the implications of having their emotions recorded by an entity they may not trust.

Asymmetrical Cognitive Ammunition

Welcome To The Metaverse, as a contribution to the emerging field of critical big data literacy, aims to plant doubtful seeds in users’ conceptions of the Metaverse before it is fully realized. The work intends to call out the Metaverse as what Langdon Winner would call an Inherently Political Technology.²⁹ Winner casts technologies of this type as those that require specific political relationships to function: a ship that needs a top-down structure where the crew obeys the captain's commands, for example. The Metaverse, as conceptualized by Meta, requires exploitation: of the data of users whose bodies and environments are digitized, of the labour of workers who mine the minerals that are necessary for the hardware, and of the resources of the planet which must be marshalled for the energy to power the cloud computing that keeps the Metaverse running.

None of this exploitation is apparent when using the early prototypes of the Metaverse. The brightly coloured landscapes and diverting amusements offer no hint of the materials or labour marshalled to bring each virtual moment to life. Digital environments were not conceived in a manner that would make their processes transparent.³⁰ This makes efforts to crack open the “black box” with public-facing works such as WttM all the more urgent. As Vladan Joler and Kate Crawford remind us, the stakes of digital exploitation are vast:

“The scope is overwhelming: from indentured labour in mines for extracting the minerals that form the physical basis of information technologies; to the work of strictly controlled and sometimes dangerous hardware manufacturing and assembly processes in Chinese factories; to exploited outsourced cognitive workers in developing countries labelling AI training data sets; to the informal physical workers cleaning up toxic waste dumps.”³¹

As creators, we feel that the most effective way to critique these problematic systems is to have users experience them within a new context. Yet this brings its own challenges: a tactical shortfall of semiotic guerrilla warfare is that the enemy owns the battlefield. This is true of all mass media: radio, television and Instagram are all difficult to hack because a layer of permission is applied before a message can appear on the platform. Yet to increase the literacies related to contextual integrity that we have highlighted in this essay, we we’re obliged to use Facebook/Meta as both the channel and message. We could not take a picture of our critique and transmit the message as we could with a billboard culture jam. While creating Welcome To The Metaverse, we witnessed firsthand the limits of critiquing within such a vertically integrated communications system.

SparkAR, created and owned by Meta, facilitates the uploading of augmented reality face filters to its Instagram and Facebook platforms directly from within the app. There is no alternative distribution venue. Unlike the World Wide Web, this ecosystem is known as a “walled garden” and requires that each piece of content made available be screened and approved before appearing in listings and searches.

Upon submitting Welcome To The Metaverse, our team waited 24 hours, after which we received a notice that the project had been rejected. The notice stated that we had violated Policy 3.7 from the SparkAR policy by using a trademarked asset or colour gradient. There was no indication of which part of the experience violated the policy, but the team was confident that the violation was caused by using the “like” button. The narrator asked users to “tap the like button to continue” and displayed the blue thumbs-up image.³² This image, and apparently the blue colour scheme, are trademarked by Facebook.

Our team replaced the like button with a heart, an image which Meta does not (yet) own, and resubmitted the project. Another 24 hours later, the filter was accepted and available to the hundreds of millions of users of Instagram and Facebook.

It is ironic that this image, which we found easy to replace, was censored. After all, the entire piece is a direct criticism of the Meta corporation. Our hunch, which we will never be able to confirm, is that our review was undertaken either by an algorithm or an outsourced temporary worker. Neither appears suited to detect satire, but both are another example of the extractive stack that the Metaverse is being built upon. Scholars have noted how content moderation algorithms reinforce white supremacy as they make no distinction between critiques of whiteness and racial violence.³³ Human-based content moderation requires the exploitation of hundreds of thousands of workers around the globe working precarious jobs and being exposed to traumatizing images.³⁴

Our team published a work that could easily be removed. Just as a McDonald's billboard spray-painted with vegan messaging would likely be taken down once the corporation or ad network became aware of the intervention, our project is likely ephemeral. Where it ever to be successful and brought to the attention of a Meta employee, it would likely be de-platformed.

Conclusion

By creating Welcome To The Metaverse, we enacted a promising approach to engage users of Meta using the company's own technology, introducing ambivalence surrounding the company’s attempts to collect more of their data. By introducing the novel concept of context jamming, the documentary experience hopes to build critical big data literacy while hacking affect and encouraging the public to interrogate the power structures that underlie digital technologies.

This research-creation approach holds promise for those wishing to subvert and detourn extractive platforms, yet it also points out the control these platforms can exert to prevent the cultivation of doubt about their intentions. This is why critiques such as WttM must be followed, or potentially supplanted, by alternative visions on new platforms. The Metaverse of Meta is only one manifestation of a networked world, and virtual communities that uphold humanist values are waiting to be explored. As Langdon Winner reminds us, “it can happen that within a particular complex of technology... some aspects may be flexible in their possibilities for society, while other aspects may be (for better or worse) completely intractable.”(1999, 135)²⁹ A non-extractive Metaverse is possible.

If creative resistance to the Metaverse of Meta is to take hold, producers and users of the medium need to imagine what alternatives could look like and distribute these speculative imaginings on platforms of their own creation. Encouraging yesterday’s culture jammers to manifest a more healthy media landscape, advertising hacker Stuart Ewen asked them to lead by example. “If our critique of commodity culture points to better alternatives, let us explore—in our own billboards of the future—what they might be.”³⁵

References

1 Brett Gaylor, Welcome To The Metaverse, Augmented Reality Filter, February 2022, https://www.hololabs.org/welcome-to-the-metaverse

2 Natasha Singer, Mike Isaac, “Facebook to Pay $550 Million to Settle Facial Recognition Suit", The New York Times, Retrieved March 16, 2022, https://www.nytimes.com/2020/01/29/technology/facebook-privacy-lawsuit-earnings.html

3 Mike Isaac, “Facebook Renames Itself Meta”, The New York Times, Retrieved April 12, 2022, https://www.ny- times.com/2021/10/28/technology/facebook-meta-name-change.html

4 Nathalie Loveless, How to Make Art at the End of the World; A Manifesto for Research-Creation, Duke University Press, 2019.

5 Shoshana Zuboff, “Surveillance Capitalism and the Challenge of Collective Action”, New Labor Forum 28, January 2019, 10–29, DOI:https://doi.org/10.1177/1095796018819461

6 B. Gaylor, Do Not Track. [Interactive Documentary], 2015, https://www.donottrack-doc.com

7 B. Gaylor, Discriminator. [Interactive Documentary], 2021, https://www.discriminator.film

8 Stephanie Takaragawa et al. “Bad Habitus: Anthropology in the Age of the Multimodal”, American Anthropologist, vol. 121, June, 2019: 2, 517–524

9 Alvarez Astacio et al. Multimodal Ambivalence: A Manifesto for Producing in S@!#t Times, American Anthropologist, 123, 2021 :2, 420–427

10 Neal Stephenson, Snow Crash, Bantam Books, 1992.

11 L. E. Editor Consumer Affairs, “Virtual reality: pandemic leads to rise in headset sales to escape lockdown”, Retrieved Mar. 16, 2022, https://www.thetimes.co.uk/article/virtual-reality-pandemic-leads-to-rise-in-headset-sales-to-escape-lockdown-jhhn8wghn

12 Sarah Ahmed, A Phenomenology of Whiteness, Feminist Theory, 8, 2007, 2, 149–168.

13 Pierre Bourdieu, Distinction: A Social Critique on the Judgement of Taste, Chicago: University of Chicago, 1988.

14 Umberto Eco, “Towards a Semiological Guerrilla Warfare”, in Travels In Hyperreality, Gruppo Editoriale Fabbri Bompiani, 1983.

15 Ina Sander, “What is critical big data literacy and how can it be implemented?” Internet Policy Review, vol. 9, May 2020,:2.

16 Helen Nissenbaum, H. Privacy as contextual integrity, Washington Law Review, 79, 2004, :1, 119–15,

17 Americans’ opinions on privacy and information sharing. Pew Research Center: Internet, Science & Tech. Retrieved March 16, 2022, https://www.pewresearch.org/internet/2016/01/14/privacy-and-information-sharing/

18 Mark Dery, Hacking Slashing, and Sniping in the Empire of Signs. In M. DeLaure (Ed.), Culture Jamming: Activism and the Art of Cultural Resistance, NYU Press, 2017.

19 Adam D. I. Kramer et al. Experimental evidence of massive-scale emotional contagion through social networks, Proc. Natl. Acad. Sci. U.S.A, 111, June 2014, 24, 8788–8790.

20 Kate Crawford, Atlas of AI, Yale University Press, 2021.

21 Javier Sánchez-Monedero, Lina Dencik, “The Datafication of the Workplace”, Working paper, Data Justice Lab, Cardiff University, May 9, 2019.

22 G Tonguç and Ozaydın Ozkara B. “Automatic recognition of student emotions from facial expressions during a lecture”, Computers and education, 2020, 148.

23  Ashley Halsey III, “House Member Questions $900 Million TSA ‘SPOT’ Screening Program”, Washington Post, November 14, 2013, Retrieved December 7th, 2022, https://www.washingtonpost.com/local/trafficandcommuting/house-member-questions-900-million-tsa-spot-screening- program/2013/11/14/ad194cfe-4d5c-11e3-be6b-d3d28122e6d4_story.html.

24 Affectiva Human Perception AI Analyzes Complex Human States, Affectiva, https://www.affectiva.com/.

25 Welcome - Amazon Rekognition, Retrieved April 12, 2022, https://docs.aws.amazon.com/rekognition/latest/APIReference/Welcome.html

26 Facial Recognition | Microsoft Azure, Retrieved April 12, 2022, https://azure.microsoft.com/en-us/services/cognitive-services/face/

27 Ruth Leys, The ascent of affect: genealogy and critique, The University of Chicago Press, 2017.

28 Adolphs Barrett et al. “Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements” in Psychological Science in the Public Interest, 20, 2019:1, 1–68.

29 Langdon Winner, Do Artifacts Have Politics? In The Social Shaping of Technology (2nd. ed.), Donald MacKenzie,Judy Wajcman (eds.), Open University Press, 1999, 28-40.

30 Mark Graham, Håvard Haarstad, “Transparency and Development: Ethical Consumption through Web 2.0 and the Internet of Things,” in Information Technologies & International Development 7, no. 1, March 10, 2011,: 1.

31  Kate Crawford, Vladan Joler, “Anatomy of an AI System”, Retrieved March 31, 2022, http://www.anatomyof.ai

32 Facebook like button, Wikipedia, Retrieved April 12, 2022, https://en.wikipedia.org/wiki/Facebook_like_button

33 E. Siapera, “AI Content Moderation, Racism and (de)Coloniality”, Int Journal of Bullying Prevention, vol. 4, Mar. 2022,:1, 55–65.

34 S.T. Roberts, Behind the screen: Content moderation in the shadows of social media, Yale University Press, 2019.

35 Stuart Ewen, Elizabeth Ewen, Channels of Desire: Mass Images and the Shaping of American Consciousness, McGraw-Hill, 1976. 

Hybrid Entanglements: a posthuman dramaturgy for human-robot relationships

Petra Gemeinboeck1, Rob Saunders2

1Centre for Transformative Media Technologies, Swinburne University of Technology (Melbourne, Australia) & Media Theory, University of Applied Arts Vienna (Vienna, Austria) 2Leiden Institute of Advanced Computer Science, Leiden University (Leiden, The Netherlands) 
pgemeinboeck@swin.edu.au, r.saunders@liacs.leidenuniv.nl

Abstract

This paper discusses our collaborative Machine Movement Lab project harnessing movement to bodily empathize with abstract machines. Bringing together creative robotics, choreographic strategies, and a posthuman dramaturgical frame, the project seeks to trouble our relationships with robots by exploring them as more-than-human entanglements. The paper discusses our transdisciplinary performance-making practice and underlying theoretical concepts and how they are mobilized through emerging diffraction patterns mapping out symbiotic relationships. An improvisational score involving dancers, robot costumes and robots performed in a gallery space aims to engage audiences with hybrid human-machine entanglements in embodied and empathic ways.

Keywords

diffraction, dramaturgy, entanglement, human-robot interaction, hybridity, performance, posthuman.

DOI

10.69564/ISEA2023-57-full-Gemeinboeck-et-al-Hybrid-Entanglements

Introduction

Hybridity is predicated upon difference—we can only recognize something as hybrid and symbiotic if we acknowledge and recognize the potential of difference. This paper seeks to bring to the fore the aesthetic and social potential of difference in our relationships with machines. It attempts to trouble practices in human-robot interaction that, like many human practices, are invested in deliberately masking difference, grounded in hierarchical and hegemonic beliefs. Stuck in what Barad ¹ referred to as the “representationalist trap” of reflection, we look for and fabricate resemblances between what are, essentially, deeply asymmetric entities.² Many of our current human-robot imaginaries thus echo or reaffirm the conservative narratives that validate existing social norms. Yet how we imagine social machines and the future narratives they are embedded is not only a matter of appearance, but literally matters—socially, politically, and ethically. Machines with humanlike facades, for instance, are often presented as more familiar and friendly; but they also serve to confine both bodies and things in mimicry and servitude.³, ,

Our Machine Movement Lab (MML) project attempts to counter this reflection-centered approach by developing a diffractive practice, which foregrounds and aesthetically exploits the differences between humans and machines. MML thus seeks to trouble our relationships with robots that manifest from reductive desires to render the machine as humanlike as possible by investigating creative strategies for reimagining and reconfiguring our relationships with them. This paper focuses on our latest research stage, which draws on Donna Haraway’s ⁶  and Karen Barad’s ¹ new materialist conception of diffraction to explore the potential of performance-making and posthuman dramaturgy for entangling humans and machines. With the latter we seek to open up ontological boundaries, such as the one delineating subjects and objects, and to reconfigure them or render them porous, the bodily-material way. We believe that such reconfigurings challenge the limited, humancentric ways in which we envision our robotic futures by expanding our bodily ways of knowing and becoming more attentive to the performative potential of this hybrid, more-than-human encounter.

We begin with providing a brief overview of the practices within which our work is situated, along with some key artists whose work has influenced our practice. Following, we introduce our MML project and how it harnesses the generative potential of movement in tandem with dancers’ kinesthetic expertise to become-with and design abstract machine artifacts. We then take a closer look at our performance-making approach and posthuman dramaturgical framing. Looking at the making of human-robot relationships as a more-than-human entanglement, we outline the feminist concepts that our new materialist practice draws on and seeks to mobilize. Finally, we discuss the making of an improvisational performance score, arising from our experimental studio practice, and how it aims to facilitate the engagement of audiences in embodied and empathic ways.

Situating our Practice

Looking at our relationships with robots from a performance perspective highlights their embodied, socio-cultural, material and, sometimes, codependent nature. We situate our transdisciplinary practice across the practices of machine performance, kinetic sculpture, and robotic art that experiment with movement and its capacity to evoke affective relationships between bodies and things. Artists have long deployed performance concepts to create ‘living’ sculptures or machine performances that both critically and playfully explore intimate couplings between human and machine bodies. Marco Donnarumma, for instance, seeks to highlight the co-dependence of hybrid (human-machine) embodiments rather than a “pairing of two different things.”⁷

Jean Tinguely’s early kinetic sculptures induce a sense of creative machine spirit,⁸ and Robert Breer’s slowly moving Floats used motorized wheels to gradually rearrange themselves in space, and thus, almost unnoticeably, reconfigure space.⁹ More recently, The Table: Childhood (1984–2001) by Max Dean, Raffaello D'Andrea and Matt Donovan produces surprising relational dynamics between audience members and the familiar object of a table.¹⁰ Kris Verdonck’s Dancer #3 ¹¹ performs the energetic clumsiness of an optimistic clown in empathically accessible yet distinctly machinic ways.

State Grace Machines by Bill Vorn, Emma Howes and Jonathan Villeneuve explores questions of kinaesthesis and perception in a dialogue between abstract machine performers and a dancer.¹² Eve of Dust, a collaboration between John McCormick, Adam Nash and Stephanie Hutchison, investigates possibilities of physical collaboration and cocreation between a human dancer and a robot arm ¹³. Louis-Philippe Demers’ performance work The Tiller Girls foregrounds the whimsey and vulnerability of machine bodies;¹⁴ in line with Paula Gaetano Adi’s poetic embodied entanglements, such has produced by her works Becoming With and Alexitimia,¹⁵ promoting the social presence of machines and strange affective capacity of abstract machines.

All these works generate their own dramaturgical frame for exploring the social capacity of non-humanlike machines and complicating our relationships with them; thus, expanding our understanding of how we relate to machines.

Machine Movement Lab (in a nutshell)

Our Machine Movement Lab (MML) project is a collaboration with dancers, choreographers, AI researchers, engineers, and numerous materials (from cardboard, PVC tubes, plywood to aluminum framing, motors, motor controllers, cables, cable binders, and software programs), across robotics labs, dance studios, fab labs, and gallery spaces over the past seven years. MML harnesses the generative potential of movement and its dynamic qualities to explore the aesthetics of entangling and empathy in human-robot encounters.¹⁴³ Rather than human-or animal-like, our robots are abstract, machinelike artefacts, forged from a practice of becoming entangled with the machine morphology and its unique, more-than-human capacities. Our latest research stage is concerned with performance-based inquiries into posthuman, transcorporeal reconfigurings and their potential to expand our possible relationships with abstract, machinelike robots.

Movement as a generative, relational force MML regards movement as a phenomenon or force, capable to make bodies, meanings, and relationships. This contrasts much of the current robotics research where movement is understood as a means of navigation or imbuing an object with a predefined personality. The difference between looking at movement as a productive force rather than an instrument is significant because it allows us to become-with what it generates—its enacted relations, specific to this situation, rather than using it to generate what we already know. This notion of movement mattering, bodying-forth¹⁶ and relation-making, opens-up seemingly limitless opportunities for entangling with more-than-human artefacts.³

Figure 1. Relational Body-Mapping; with the cube performer, A. Frahn-Starkie and S. McKenna, 2022. © P. Gemeinboeck.

Relational-Body-Mapping (RBM)

Our MML practice revolves around the idea that the kinds of relation-making that movement propels happen in the dynamics of encounter and unfold through “spatial, temporal, and energic qualities.”¹⁷ This is where meanings and affects get made and distributed across human and nonhuman bodies, rather than being predefined and preformed by certain beliefs about what this more-than-human relationship should be.

Our diffractive approach aims to harness movement’s generative force by enacting situations of close, corporeal encounter that can open-up kinaesthetic experiences of becoming-with the machine artefact and its unique material qualities. In practice, this involves getting entangled with material props, whose material qualities can offer us a corporeal glimpse of the machine’s more-than-human relational possibilities. To enable this becoming-with (i.e., entangling), we ask dance performers to extend themselves into, inhabit, or wrap around a wearable costume whose shape and size resembles that of the robot (see Figures 1 and 2). The costume thus stands in for a becoming-robot design, at the early stages of the design process, and enables dancers to feel into the robot’s material-spatial potential ³ as well as the robot’s sensorium (equipped with the becoming-robot’s sensors).

This more-than-human entanglement, which we will refer to as performer-costume in the following sections, allows us to experiment with and corporeally probe into a range of human-machine configurings. Our Relational-Body-Mapping (RBM) approach builds on our Performative-Body-Mapping (PBM) method, which focused on single performer-cube entanglements and movement creation.¹⁸ RBM expands PBM to seek more complex, nested entanglings and the transcorporeal resonances they can effect, e.g., a (human) performer with a (robotic) cube performer; a performer-costume with a cube performer; or a (human) performer with a performer-costume and a cube performer (see Figure 1), and so on.

Cube Performer

The robot costume not only allows the dancers to ‘feel into’ the differences of the machinic embodiment but also to capture the kinetic dynamics that unfold in this more-than-human entanglement. The hybrid motion data, arising from this human-nonhuman enmeshment, informs the robot’s machine learning process, where the machine learns to improvise movements based on its own mechanical embodiment and the patterns it derives from our entangled motion data.¹⁴

Our first robot prototype––the cube performer (see Figure 2)—resulted from a series of corporeal entanglements with a wide range of materials. It is a simple box-shaped artefact, which is transformed by its dynamic movements: suddenly tilting up along one of its edges and gently swaying or thumping onto the ground, the box quickly loses its rootedness and becomes more-than-object ¹⁴. The robots’ mechanical design was derived from an extensive analysis of motion capture recordings of the performer-costume and the relational motion patterns it produces ¹⁹. Instead of relying on googly eyes or pre-packaged personality, the robot cube becomes a performer based on the enactive potential of its movement dynamics [see ²⁰, ¹⁷] and how they can co-shape a meaningful encounter.

A more detailed discussion of our performance-based, embodied robot design stage can be found in ³, ¹⁴, ¹⁸, ¹⁹. 

Figure 2. Cube performer #1, robot prototype, at the Games as Performing Arts Festival, AMATA, Falmouth University, UK, 2018. © P. Gemeinboeck.

Diffractive Performance-Making

Our diffractive performance-making practice investigates how corporeal entanglements with machine artefacts and their different material-spatial and affective qualities can open-up modes of transcorporeal empathy. The latter, we believe, is key to meaning making with social machines without relying on fake emotional facades (i.e., a humanlike face).

Robotics practices, in general, often look at humans and machines as two separate, already given or predefined entities (i.e., subject and object). MML, in contrast, attends to how subjects and objects are mutually constituted² by investigating the making of subject-object boundaries as a nested entanglement. Meaning making here is about carefully attending to the possibilities for relations and meanings to emerge.

According to Jon Lee, the alternative landscape of a diffractive dramaturgy is experimental and experiential, “where we feel for and towards (in a tentacular way) a collaborative making process that tilts the optic away from traditional expectations.”²² Our diffractive, posthuman dramaturgy generates an experimental and experiential space, where we feel for possible entanglings, tentacular capacities and hybrid configurings of human performers and nonhuman artefacts. It involves carefully probing into how they matter, couple, interfere, and “undo and redo each other,”²³ and how this difference-in-relation gives rise to transcorporeal meaning-making.

The following outlines some of the core theoretical concepts that our performance-making practice draws on, and then discusses some of the most significant interference patterns whose emergence we have witnessed thus far.

Diffracting Subjects and Objects

Our posthuman dramaturgical approach attends to and aesthetically puts to work difference-in-relation (i.e., humans and machines entangled) by seeking to materially mobilize Haraway’s⁶ and Barad’s ¹ concept of diffraction. In contrast to reflection (i.e., rendering machines humanlike), diffraction maps interferences¹⁴ and as such “attends to the relational nature of difference.”²⁴

A diffractive practice embraces and foregrounds differences by attending to the specificity and materiality of entanglements.¹ Diffraction thus not only serves as a figurative lens but can shape a material process, i.e., in our practice the dramaturgical/choreographic methods of interfering, superposing and entangling bodies and things.

From a posthuman perspective, we always already are entangled with the world and its ongoing reconfigurings.²⁵, ¹ Barad’s notion of a posthumanist performativity calls “into question the givenness of the differential categories of ‘human’ and ‘nonhuman’, examining the practices through which these differential boundaries are stabilized and destabilized.” ²⁶ Diffraction as both a tool and a practice can make manifest the destabilization and stabilization of boundaries.²⁷

In MML, we are particularly interested in the boundary-making that both separates and defines subjects and objects. How can we intermesh (given) subjects and objects, probe into their boundaries and render them more porous or create new hybrid entities? Rather than juxtaposing humans and machines or making them appear to be the same, we seek symbiotic possibilities based on difference patterns that render the boundaries between subjects and objects more elastic. Diffraction and patterns of interference thus become a methodological tool for “attending to and responding to the effects of difference” [¹⁷: ⁷²]. The entanglement of bodies and things maps their effects of difference similarly to Barad’s description of superposition:

“... waves can overlap at the same point in space. When this happens, their amplitudes combine to form a composite wave form [and] the resultant wave is a sum of the effects of each individual component wave; that is, it is a combination of the disturbances created by each wave individually. This way of combining effects is called superposition”. ²⁸.

A posthuman dramaturgy for diffracting subjects and objects thus troubles engrained dichotomies and, instead, traces the effects of differences that give rise to new forms of more-than-human meaning-making—in MML, a trans-corporeal form of meaning-and experience-making, which we will look at in more detail below.

More-than-human Interference Patterns

Superposing human bodies and cubic things, in practice, requires ongoing attunement to the becoming of bodies and, with it, emerging agencies and differing identities—a moving with and continuous gesturing toward the more-than-human space of a ‘thing’—the process of becoming-thing. The empathic resonances brought about by this superposition can be described as a bodying-thinging.⁶ Transcorporeal bodying-thinging is about how bodies and things resonate whilst undoing and redoing each other; at once tracing how subjects and objects constitute each other and at the same time rendering their boundaries elastic.³ It attests to the inherent porosity, relationality and reconfigurability of bodies and things, how they already always extend toward and across each other.

The following explores three of the most significant interference patterns that we have observed thus far and how they mobilize transcorporeal resonances of bodying-thinging. They come about based on different degrees of entanglement, the number of entangled bodies and things, and the emergent effects of ongoing reconfigurings. 

Figure 3. Becoming with the cube; with dancer A. Frahn-Starkie, 2022. © P. Gemeinboeck.

Pattern #1: Spatial Superposition, Becoming-with

This interference pattern manifests from the dancers corporeally exploring their entangledness with the cube by bodily listening to its material characteristics and capabilities and the cube responding (talking back) by producing different material sensations (its weight, how it bends, where it resists, etc.). Becoming-with (see ²⁰) the cube then involves dancers reconfiguring their bodies as well as letting themselves being shaped by these nonhuman qualities and to feel-think-move-with the cube (see Figure 3). Sometimes the two simply interfere, other times they are in-phase, become-with, and are bodying-thinging with each other.

Pattern #2: Stretching the Boundary between Subject and Object

We found that dwelling on the edge (i.e., the subject-object boundary, which may also align with an edge of the cube) and feeling into it, stretches and carefully opens-up the boundary in-between subjects and objects. The threshold of the boundary becomes a zone to linger, to extend into or be extended by, to become familiar, to mingle with (see Figure 4). It is the most symbiotic cube-performer interference pattern with regards to its resulting shape and the entanglement’s tentacular capacity (see following section), where body-thing can no longer be separated, nor is one entirely folded into the other. Rather than a barrier, the boundary becomes an access zone—a gateway to bodying-thinging and exploring the symbiotic affordances of this hybrid performer-costume entity. 

Figure 4. Stretching the Subject–Object Boundary; with A. Frahn-Starkie and F. Palmerson, 2022. © P. Gemeinboeck.

Pattern #3: Nested Entanglings, Becoming-tentacular

Performance-making involving more than one performer and one costume produces a nesting of difference patterns and, with it, the affects that flow across the open seams of each pattern. The nested entanglings unfold in a continual process of attachments and detachments, e.g., the dancers attaching themselves to a corner of the costume, a corner of the space, or to a corner of the other costume, even if only for a glimpse, even with only the tip of the toe (see Figure 5); then detaching again—from the corners, one by one or all at once, to reattach and align with an edge, or a plane, or the other dancer’s gaze. These re-/alignments open-up spaces to link/mesh/interweave with other boundary spaces, stretching and extending the lines of the cube to reach into or meet other lines, and performer-costumes become tentacular and intermesh; bodying-thinging here also means to grow tentacles. Haraway speaks of “tentacular ones [and how they] make attachments and detachments; they make cuts and knots; they make a difference; they weave paths and consequences but not determinisms; they are both open and knotted in some ways and not others.”²⁹

Figure 5. Nested entanglings; with dancers A. Frahn-Starkie and F. Palmerson, 2022. © P. Gemeinboeck.

All three of these symbiotic difference patterns result in movements and dynamic constellations that are irreversibly hybrid: The dancer’s body is reconfigured by the costume, and the movements captured with the costume reconfigures the movements learned by the robot.⁶, ¹⁸ And when performer-costume and cube performer (robotic artefact) entangle and become tentacular, new motion patterns evolve— movements that neither belong to the machine nor the performers-in-costume.

In the following we explore how our improvisational score builds on these interference patterns and unfolds them as a series of experiential scenarios, each performance anew.

Scoring an Improvisational Performance

Dancing with the Nonhuman is a roughly 20-min performance work, to be performed in gallery spaces rather than a separate stage. Its underlying semi-structured, improvisational score seeks to open-up our diffractive process to the diverse embodied perspectives of audiences by performing human-nonhuman interference patterns and the transcorporeal attunement they produce–each iteration anew.

Arising from our experimental studio practice and observed, emergent-diffractive patterns, the underlying improvisational score shapes different ‘lenses’ through which the experiential scenarios of human-nonhuman entanglement unfold. The following outlines the four lenses that propel Dancing with the Nonhuman [SYD-2-2-1] and how they mobilise differently hybrid and tentacular configurings.

In (1) ‘phantom’, we witness a series of movements shaped by the dancers’ cubic entanglement but reperformed without the cube costume. The performance thus opens with a kind of puzzle as these movements clearly belong to a realm that is both more-than-human and more-than-object.

In (2) ‘threshold’, dancers feel their way along the boundaries of the cube costume, extend them, entangle with them, and render them elastic; meanwhile the cube performer slowly glides along straight lines, occasionally beginning to twitch out of the grid.

In (3) ‘con-current’, we witness the dance performers fully inhabiting their cube costumes. The encounter between cube performer and performers-in-cube appears seamless and interferences express themselves along geometric lines. In (4) ‘co-play’, the encounter becomes a playground, and it gets a bit messy, bodies and things tumble. And so do their boundaries.

Audiences and
transcorporeal empathy

At the time of writing, we are yet to perform this work in public. Importantly, audiences are not expected to decipher any of these patterns or lenses. The aim is for them to engage with these alternate, posthuman human-machine configurations not only by looking but also by transcorporeally empathizing with them, based on their own corporeal experiences with tentacular, more-than-human configurations.

As we strive to collapse the distance between subjects and objects, we also seek to render the boundary between performers and audiences more porous. To avoid the distancing effect of a stage, Dancing with the Nonhuman is designed to be performed in gallery spaces. The performance area is only marked through a grid on the floor, which assists performers to locate themselves; it also represents the cubic grid that the cubes break loose from (see Figure 6).

To render the boundary more porous, the performance includes transitional intro and outro stages, in which the performance site is gradually established and dissolved again. In the intro, audiences are welcome to stay inside the marked performance area and mingle with both human performers and cube performers (costumes and robot), while they slowly shuffle across the boundary and get settled inside the grid space. At the end of stage 4, the boundary becomes soft again and audiences are welcomed to interact with the performers, both human and nonhuman. While this could be as casual as sitting down and gently leaning against one of the cubic artefacts, we are keen for audiences to bodily explore the performers’ perspectives, both human and nonhuman, and to get entangled themselves. 

Figure 6. Dancing with the Nonhuman [SYD-2-2-1], rehearsal, with A. Frahn-Starkie and F. Palmerson, SHErobots, Tin Sheds Gallery, Sydney, AU, 2022, © P. Gemeinboeck. 

Summary/In-Parting

This paper introduced our collaborative, diffractive performance-making practice, as part of our ongoing Machine Movement Lab (MML) project, to promote unscripted, playful encounters with strange, non-humanlike machines. Our collaborative project centers around the generative potential of movement to harness dancers’ kinesthetic expertise for empathizing with abstract machine artifacts. This performance-making practice and its posthuman dramaturgical frame materially mobilizes the theoretical concept of diffraction and new materialist notions of agential enactment.³⁰ The more-than-human entanglements that our practice attends to produces the diffraction patterns for mapping out alternative human-machine relationships. This difference-in-relation also shapes the making of a semi-structured, improvisational performance score, aiming for audiences to engage with these hybrid entanglings in embodied and empathic ways.

Our diffractive, creative research seeks to open-up new performative strategies for aesthetically attending to and making tangible difference patterns and relational ontologies at work in human-robot encounters. We propose that opening-up a more horizontal playground for dancing with machines requires us to get entangled and resonate with machines, which, in turn, requires collapsing the distance between subjects and objects (rather than masking it). Collapsing distances, the diffractive way, means to stretch and open-up the boundary in-between subjects and objects, to explore the space in-between, and grow tentacles into other boundary spaces. Performance-making here is a mode of generative-diffractive inquiry into the re-/enactment of subject-object boundaries as part of the dynamic exchanges unfolding in human-robot encounters.

Concerned with the relationalities of embodied meaning-making,¹⁴ our choreographic-dramaturgical strategies explore the performative aesthetics of corporeally entangling human bodies and machinelike things and the more-than-human difference pattern this produces. The aesthetic potential of this practice, we believe, results from combining the asymmetries that differentiate human and machine participants² and the physical-dramaturgical entanglements that render them relational, producing seemingly dissonant inter-bodily resonances. Rather than serving to make the strange look more familiar, aesthetics here is about rendering difference more relational.

Acknowledgements

The authors would like to thank their collaborators: Roos van Berkel (TU/e, NL), Maaike Bleeker (Utrecht, NL), Katrina Brown (Falmouth, UK), Arabella Frahn-Starkie (AU), Rochelle Haley (UNSW, AU), Lesley van Hoek (NL), Stephanie Hutchinson (QUT, AU), Sarah Levinsky (Falmouth, UK), Linda Luke (De Quincey Co., AU), Dillon McEwan (AU), Siobhan McKenna (AU), Kirsten Packham (AU), Felix Palmerson (AU), Marie-Claude Poulin (Applied Arts Vienna, AT), Tess de Quincey (De Quincey Co., AU), Audrey Rochette (CA), and Kim Vincs (SUT, AU).

This project has been partially supported by the Australian Government through the Australian Research Council (DP160104706 and FT190100567); the Austrian Science Fund (FWF, AR545); and the EU Framework Programme (FP7, 621403).

References

1 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, NC: Duke University Press, 2007, 135.

2 Lucy Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, Cambridge, Cambridge University Press, 2007.

3 Petra Gemeinboeck, “Difference-in-relation: Diffracting human-robot encounters,” Matter: Journal of New Materialist Research, V ol. 03, no. 01, 2022, accessed 13 October 2022, https://doi.org/10.1344/jnmr.v3i1.38958.

4 Jennifer Robertson, Robo Sapiens Japanicus: Robots, Gender, Family, and the Japanese Nation, Berkely, CA, University of California Press, 2017.

5 Claudia Castañeda, Lucy Suchman, “Robot visions,” Social Studies of Science 44, no. 3, 2014, 315–341.

6 Donna J. Haraway, “Promises of Monsters: A Regenerative Politics for Inappropriate/d Others,” in Cultural Studies, ed. Lawrence Grossberg, Cary Nelson and Paula A. Treichler, New York & London: Routledge, 1992, 295–337.

7 Marco Donnarumma, “Beyond the Cyborg: Performance, attunement and autonomous computation,” International Journal of Performance Arts and Digital Media 13, no. 2, 2017, 1–15.

8 Pontus Hulten, Jean Tinguely: A Magic Stronger than Death, New York, NY, Abbevill Press, 1987.

9 Frazer Ward, “Robert Breer,” Frieze Magazine, issue 155, November 12, 2000, accessed April 5, 2023, https://www.frieze.com/article/robert-breer.

10 Josée Greg Hill, Stephen Horne, and Anne-Marie Ninacs,

11 Kris Verdonck, “Dancer #3”, A Two Dogs Company website, accessed April 5, 2023, https://www.atwodogscom-pany.org/en/projects/dancer-3.

12 Bill Vorn, “State Grace Machines” (2007), Concordia University website, accessed April 5, 2023, https://billvorn.concor- dia.ca/robography/GraceState.html.

13 John McCormick, Adam Nash, and Stephanie Hutchison, “Eve of dust,” in SIGGRAPH Asia 2018 Art Gallery, New York, NY, ACM, 2018.

14 Petra Gemeinboeck, “The Aesthetics of Encounter: A Relational-Performative Design Approach to Human-Robot Interaction,” Frontiers in Robotics and AI, Vol. 7 (2021), accessed September 9, 2022, https://doi.org/10.3389/frobt.2020.577900

15 Paula Gaetano Adi, “BecomingWith” and “Alexitimia”, Artist website, accessed April 27, 2023, https://www.paula-gaetanoadi.com.

16 Erin Manning, Brian Massumi, Thought in the act: Passages in the ecology of experience, Minneapolis, MN, University of Minnesota Press, 2014, 39.

17 Maxine Sheets-Johnstone, “From movement to dance,” Phenom. Cogn. Sci. 11, 2012, 39–57, 49.

18 Petra Gemeinboeck, Rob Saunders, “Moving beyond the mirror: relational and performative meaning making in human–robot communication,” AI & Society 37, Springer, 2022, 549–563.

19 Rob Saunders, Petra Gemeinboeck, “Performative Body Mapping for designing expressive robots,” Proceedings of the 9th International Conference on Computational Creativity, Salamanca, Spain, June, 2018, 280–287.

20 Ezequiel Di Paolo, Hanne De Jaegher, Marieke Rohde, Horizons for the enactive mind: values, social interaction, and play,” in Enaction: Towards a New Paradigm for Cognitive Science, ed. John Stewart, Olivier Gapenne, Ezequiel Di Paolo, Cambridge, MA, MIT Press, 2010, 33-87.

21 Donna J. Haraway, When Species Meet, Minneapolis, MN, University of Minnesota Press, 2008.

22 Jon Lee, “Diffractive Dramaturgy,” Performance Research 25, no. 5, 2020, 114-121.

23  D Vinciane Despret, “Responding Bodies and Partial Affinities in Human–Animal Worlds,” Theory, Culture & Society, 30, no. 8, 2013, 51–76, 61.

24 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, 72.

25 Stacy, Alaimo, “Thinking as the Stuff of the World,” O-Zone: A Journal of Object-Oriented Studies 1, 2014, 13-21.

26 Karen Barad, “Posthumanist performativity: toward an understanding of how matter comes to matter,” Signs: Journal of Women in Culture and Society 28, no. 3, 2003, 801-831, 808.

27 Iris van der Tuin, “Diffraction as a Methodology for Feminist OntoEpistemology: On Encountering Chantal Chawaf and Posthu- man Interpellation,” Parallax 20, no. 3, 2014, 231-244.

28 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, p.76.

29 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke University Press, 2016, p.31.

30 Karen Barad, “Posthumanist performativity: toward an understanding of how matter comes to matter”.

Bibliography

Paula Gaetano Adi, “BecomingWith” and “Alexitimia”, Artist website, accessed April 27, 2023, https://www.paula-gaetanoadi.com.

Stacy Alaimo, “Thinking as the Stuff of the World,” O-Zone: A Journal of Object-Oriented Studies 1, 2014, 13-21.

Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, NC: Duke University Press, 2007.

Karen Barad, “Posthumanist performativity: toward an understanding of how matter comes to matter,” Signs: Journal of Women in Culture and Society 28, no. 3, 2003, 801-831.

Claudia Castañeda, Lucy Suchman, “Robot visions,” Social Studies of Science 44, no. 3, 2014, 315–341.

Vinciane Despret, “Responding Bodies and Partial Affinities in Human–Animal Worlds,” Theory, Culture & Society, 30, no. 8, 2013, 51–76.

Marco Donnarumma. “Beyond the Cyborg: Performance, attunement and autonomous computation,” International Journal of Performance Arts and Digital Media 13, no. 2, 2017, 1–15.

Josée Greg Hill, Stephen Horne, and Anne-Marie Ninacs, Petra Gemeinboeck, “Difference-in-relation: Diffracting human-robot encounters,” Matter: Journal of New Materialist Research, V ol. 03, no. 01, 2022, accessed 13 October 2022, https://doi.org/10.1344/jnmr.v3i1.38958

Petra Gemeinboeck, “The Aesthetics of Encounter: A Relational-Performative Design Approach to Human-Robot Interaction,” Frontiers in Robotics and AI, Vol. 7 (2021), accessed September 9, 2022, https://doi.org/10.3389/frobt.2020.577900

Petra Gemeinboeck, Rob Saunders, “Moving beyond the mirror: relational and performative meaning making in human–ro- bot communication,” AI & Society 37, Springer, 2022, 549–563.

Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke University Press, 2016.

Donna J. Haraway, When Species Meet, Minneapolis, MN, University of Minnesota Press, 2008.

Donna J. Haraway, “Promises of Monsters: A Regenerative Politics for Inappropriate/d Others,” in Cultural Studies, ed. Lawrence Grossberg, Cary Nelson and Paula A. Treichler, New York & London: Routledge, 1992, 295–337.

Pontus Hulten, Jean Tinguely: A Magic Stronger than Death, New York, NY, Abbevill Press, 1987.

Jon Lee, “Diffractive Dramaturgy,” Performance Research 25, no. 5, 2020, 114-121.

Drouin-Brisebois, Caught in the act: the viewer as performer (Ottawa: National Gallery of Canada, 2008).

Drouin-Brisebois, Caught in the act: the viewer as performer (Ottawa: National Gallery of Canada, 2008).

Erin Manning and Brian Massumi, Thought in the act: Passages in the ecology of experience, Minneapolis, MN, University of Minnesota Press, 2014.

John McCormick, Adam Nash, Stephanie Hutchison, “ Eve of dust,” in SIGGRAPH Asia 2018 Art Gallery, New York, NY, ACM, 2018.

Ezequiel Di Paolo, Hanne De Jaegher, Marieke Rohde, Horizons for the enactive mind: values, social interaction, and play,” in Enaction: Towards a New Paradigm for Cognitive Science, ed. John Stewart, Olivier Gapenne, Ezequiel Di Paolo, Cambridge, MA, MIT Press, 2010, 33-87.

Jennifer Robertson, Robo Sapiens Japanicus: Robots, Gender, Family, and the Japanese Nation, Berkely, CA, University of California Press, 2017.

Rob Saunders, Petra Gemeinboeck, “Performative Body Mapping for designing expressive robots,” Proceedings of the 9th International Conference on Computational Creativity, Salamanca, Spain, June, 2018, 280–287.

Maxine Sheets-Johnstone, “From movement to dance,” Phenom. Cogn. Sci. 11, 2012, 39–57.

Lucy Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, Cambridge, Cambridge University Press, 2007. Iris van der Tuin, “Diffraction as a Methodology for Feminist OntoEpistemology: On Encountering Chantal Chawaf and Posthuman Interpellation,” Parallax 20, no. 3, 2014, 231-244.

Kris Verdonck, “Dancer #3”, A Two Dogs Company website, accessed April 5, 2023, https://www.atwodogscompany.org/en/projects/dancer-3.

Bill Vorn, “State Grace Machines” (2007), Concordia University website, accessed April 5, 2023, https://billvorn.concordia.ca/robography/GraceState.html.

Frazer Ward, “Robert Breer,” Frieze Magazine, issue 155, November 12, 2000, accessed April 5, 2023, https://www.frieze.com/article/robert-breer.

Authors Biographies

Petra Gemeinboeck’s and Rob Saunders’ collaborative artistic research practice seeks to expand and trouble our relations with machines by exploring questions of embodiment, agency, creativity, and performativity. Petra is currently an Australian Research Council Future Fellow and Associate Professor at the Centre for Transformative Media Technologies (CTMT), Swinburne University, AU. She also leads the ‘Dancing with the Nonhuman’ FWF research project at the University of Applied Arts Vienna, AT. Rob is Associate Professor at the in the Leiden Institute of Advanced Computer Science (LIACS), University of Leiden, NL. His research focuses on computational models of creativity, using techniques from machine learning and creative robotics. Their artworks have been shown internationally, including the Ars Electronica Festival (Linz, AT); Int. Triennial New Media Art at NAMOC (Beijing, CN); GoMA (Brisbane, AU); OK Center for Contemporary Art (Linz, AT); and FACT (Liverpool, UK). 

ReSilence: Retune the Soundscape of future cities through art and science collaboration

Nefeli Georgakopoulou1, Makrina Viola Kosti2, Sotiris Diplaris3, Maurice Benayoun4, Antonio Camurri5, Beatrice de Gelder6, Melanie Wald-Fuhrmann7, Panagiota Koulali8, Kalliopi Valsamidou9, Kai Bronner10, Leo Wanner11, Froso Gaki12, Stefanos Vrochidis13, Ioannis Kompatsiaris14

1, 2, 3, 13, 14 Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), 4 School of Creative Media, City University of Hong Kong, 5 InfoMus Lab (Laboratorio di Informatica Musicale), DIST, University of Genova, 6 Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, 7 Department of Music, Max Planck Institute for Empirical Aesthetics, 8,9 School of Architecture, Aristotle University of Thessaloniki (AUTH), 10 WeSound, 11 Catalan Institute for Research and Advanced Studies (ICREA) and NLP Group, Pompeu Fabra University, 12 Froso Gaki, Thessaloniki Concert Hall / Thessaloniki (Greece), Hong Kong, Genova (Italy), Maastricht (the Netherlands), Frankfurt am Main (Germany), Hamburg (Germany), Barcelona (Spain)
1, 2, 3, 13, 14nefeli.valeria, mkosti, diplaris, stefanos, ikom@iti.gr,
4, 5, 8, 9yiotakoulali, kalliopivals, antoniocamurri, benayoun.maurice@gmail.com, 6b.degelder@maastrichtuniversity.nl, 7melanie.wald-fuhrmann@ae.mpg.de, 10bronner@wesound.de, 11leo.wanner@upf.edu, 12fgaki@tch.gr

Abstract

Advances in cognitive science, sensing technologies, the arts and creative industries are paving the way for a deeper understanding of the behaviour of individuals regarding the land/soundscape they live in. Through a symbiotic relationship between artists, scientists and technology experts ReSilence explores the borders between sound and silence in a changing world by producing sound awareness in urban spaces (not only reducing the intensity of noise, but also considering it as energy producer and designing positive sounds, sounds we want to preserve and multiply). More specifically ReSilence focuses in musical experience design centred on the active participation of citizens, in the new silence of mobility, in the acoustic perception of outdoor urban soundscapes and in enhancing experiences for people with hearing and vision impairments.

Keywords

Art, Urbanism, Sound, Artificial intelligence, Extended Reality, accessibility, social inclusion, participation.

DOI

10.69564/ISEA2023-58-full-Georgakopoulou-et-al-ReSilence

Introduction

ReSilence is a project that is part of the S+T+ARTS initiative of the European Commission to foster alliances between science, technology, and the arts, centred on human needs and values. In order to encourage collaboration of research projects and artists, S+T+ARTS funds residencies of artists in technology institutions, as well as activities of scientists and technologists in artists’ studios.

Started in 2022, ReSilence is a consortium of institutions, universities and SME’s that put together their expertise in terms of scientific, technological, artistic and pedagogical research. Through S+T+ARTS residencies artists will become part of the research team, which aims to 1) explore the borders between sound and silence in a changing world by producing acoustic awareness in urban spaces, not only through reducing the intensity of noise, but also by designing, enhancing and/or multiplying existing sound considering it as an energy factor, 2) create new types of sonic urban experiences that expand possibilities for accessibility, active participation/engagement, sustainability and social inclusion and 3) build trust around AI & XR technologies.

Background

Cities and metropolitan areas are often acting as social condensers, through the dense cohabitation of people, services and cultures. By 2050, cities will host 2.5 billion more urban dwellers, making the world almost 70% urban. Europe's urban areas are home to over two thirds of the EU‘s population (1). This coexistence brings along a series of issues and urgencies, related to place making, the collaborative process by which we can shape our public realm in order to maximise shared value and make space for diversity. Sound is a fundamental component of our cities, of their cultures, and of the health and well-being of their citizens. In an urban fabric, the soundscape of a city is entangled with social behaviour and impacts citizens’ attitudes and behaviour in urban, natural and manmade environments. New ways of mobility, communication, interaction, and sharing are beginning to have profound consequences for the way that cities sound. More inhabitants lead to a necessity for more housing, and thus more construction. Increased surveillance introduces new sounds, such as those made by drones.

The project ReSilence builds on the novel and prima facie paradoxical observation that elements of city life are becoming more silent due to the emerging electrification of public and private transport, allowing us to question which sounds are necessary for safety and environmental awareness, and which sounds can be made more pleasant. Decades ago, artists already worked on these questions. Composer John Cage famously stated that “everything we do is music,” and his seminal work “4:33”, composed in 1952, is a work which considers as music the sounds humans make just by simply enacting everyday movements. Before Cage, Futurist artist Luigi Russolo wrote the manifesto “The Art of Noises,” in 1913. Russolo concludes “the variety of noise is infinite, and as man creates new machines the number of noises he can differentiate between continues to grow.”¹ How then should we tackle this growing number of different sounds in order to improve the quality of life for inhabitants, all while considering the important role of sound and music to each person‘s life, culture, and identity?

Aims and objectives

As new technologies make their appearance in public spaces and enter our everyday lives, the need to re-imagine and envisage these spaces unfold, either by addressing their visual or non-visual (sound) aspects and values, in architectural and urban settings. How cities might sound in the future is a challenge that will significantly affect the emotional and cognitive state of individuals and will influence the functionality and effectiveness of indoors and outdoors spaces in manners that need to analyse and take into consideration.² Our world is shaped through fast modes of communication, distant social interaction and electrification. While physicality and space continue to be important premises when planning our cities, it is time to start thinking about how we can design experiences for people living hybridly (physically and virtually) in the city and how we can orchestrate the sounds our cities produce, sculpting people’s existence. To achieve this, it is essential to find new ways to engage citizens and audiences to re-think how cities sound. Therefore, ReSilence is developing technologies by proposing a new methodology with the scope to shape in novel and innovative ways urban environments and soundscapes, repositioning the role of silence and that of sound. Recent efforts have focused on the city as the generator of very large datasets used in an effort to become “smarter,” greener and safer. 

Figure 1. ReSilence overview. ©ReSilence Copyright

Now, ReSilence turns the attention to the social and human experiential dimensions of the cities’ datasets in order to make citizens more creative and aware of their surroundings, while improving quality of life through the consideration of noise (see Figure 1).

Main objective of the ReSilence project is to support the development of art-driven technologies for designing the soundscape of future cities through the collaboration of artists, architects, urban designers, scientists, engineers and researchers. The project achieves this by targeting a) AI and XR tools to primarily address challenges from the Urbanism/Mobility sector while exploring the borders between music and noise, and b) user experience measurements assessing a sound space.

Methodology

The symbiosis between humans and computer-based technologies has a significant impact on Art. Frameworks like STARTS allow technology to contribute to both the creative process and artistic practice, when in addition; it works dynamically with humans in the realisation of designs, sounds, etc.

ReSilence supports Art-Driven Experiments (ADE) through Open Calls to artists and artist-SME teams. Artists, through ReSilence, have access to AI and XR technologies and also help in ensuring that the development process and system behaviour of the technologies explicitly acknowledges human values and needs. The following figure (Figure 2.) represents the above-mentioned, based on the research, technology and art objectives set. Art and artists have a leading role in the development and evolution of the project and ADE play a critical role to support it. ADE, does not only bring together artists/sound designers, scientists and technology providers, who explore and work with existing technologies to enhance their usability and uptake or design art-driven products and services to tackle Urbanism/Mobility challenges described. While AI and MR technologies take cities to the next step of utilising the data collected and knowledge to support decision-making, they often solve city problems following top-down solutions. Instead, ReSilence actually tries to harness such technologies through ADE to engage some of the most fundamental issues of citizens, focusing on a bottom-up approach and leverage their creativity.

To begin with, we ensure that the development of services and products follows a human-centred design philosophy so that ReSilence is properly deployed in the project implementation scenarios. A set of use-case scenarios is defined at first by the artists that include information on their current workflows and objectives, the challenges they face and the areas where new solutions would be supportive. In this step, ReSilence in fact builds bridges between artists, science and technology experts. Once the artists define the challenges and the implementation scenarios, we develop AI-based methods to increase the active experience of citizens and integrate individual and collective experience of music and sound. Artists and their creations in the context of Re-Silence, through interactive visualisations and user participation, also foster trust in the deployed technologies. More specifically the above-mentioned is approached using multimodal movement analysis and sonification, by developing and integrating in the system platform tools capable to measure the non-verbal, full body expressive and emotional behaviour, at both individual and group (e.g., synchronisation, entertainment, leadership) levels. Novel tools for qualitative urban soundscape-analysis are also developed, guided by the goals set by the artists, based on immersive visual and acoustic stimuli. With the help of those, the consistency of a soundscape can be investigated to identify needs and means for redesign of outside or inside spaces. The “AI for real-time interaction” objective is also supported by the generation of sound related to images, the synthesis of scenes based on sounds and the further study of the relation between auditory and visual stimuli by means of explainable AI.

Figure 2. Objective-based ReSilence methodology (AO-Art Objectives, RO-Research Objectives, TO-Technology Objectives) ©ReSilence Copyright. 

The above led to the art-driven toolkit, which deals with the integration of the modules based on which artists, creatives and technologists co-create artworks, products and services. Artists will be using the collected audio-visual and spatial data to assist and empower specific user groups by means of multisensory technology. More specifically, this stage is related to the development of multisensory experiences in XR environments and allows artists to create different solutions to reshape our urban environments and soundscapes, rebalancing the role of silence and that of sound. The development and integration of AI tools are also part of this phase and defines the underlying architecture for a robust communication system between the components and how the components are integrated in each required system. Artists and architects are also supported by an object-based interactive audio engine, allowing the algorithmic composition of soundscapes based on visual data. This engine facilitates the virtual design of urban soundscapes by means of 3D-modelling techniques. Moreover, the art-driven experiments are evaluated in the “Soundscape Experience Assessment” phase, which achieves a novel understanding of the human perception in relation to sound and his environment. At this point, we deal with emotional sensing and analysis, sentiment analysis from crawled social media data and reviews, multimodal analysis of sound related behaviour and self-report generation. To support the above, we develop novel physiological signal processing algorithms and signal integration frameworks to detect, extract and classify markers of emotional responses to aesthetic and/or artistic stimuli. Through this analysis, ReSilence essentially understands individual and collective perceptions and preferences, leading to a more transparent decision-making process. Part of this work is also a novel algorithmic approach to investigate the correspondences between artists’ music composition and listeners’ perception. Another part of this phase of our methodology is devoted to the linguistic analysis of social media posts and human verbal comments acquired within ReSilence that discuss topics related to the music sector in order to give an overview of larger societal preferences and sentiment. Furthermore, for the multimodal analysis, we combine the outcomes of different components, in order to acquire a more integrated result for understanding the role of active audience participation. The combined components vary among the different scenarios. The project’s methodology ends with the fabrication of creative solutions to reach the public, resulting from the collaboration with the artists. The challenges that led the artists in the definition of the implementation scenarios, along with the opportunities sound and related media offer to the urban future eventually result in artworks that stimulate public dialogue through exhibitions, communication initiatives and/or industry collaborations. The first phase of these S+T+Arts residencies is the research phase during which the artists will spend time discussing with the ReSilence partners and learning about research and technology initiatives. In return, consortium members learn about the artist’s practice, working methods and perceptions of the questions being explored. The second phase is the technical/scientific realisation of the project and the third is the phase for preparing the results for exhibition and communication.

In a nutshell, ReSilence achieves its goals by establishing a symbiotic relationship between artists and scientists in a co-creation ground to address urban and societal challenges aiming for an enhanced urban-sonic experience. Hence, it supports two kinds of art-driven experimentations: (a) artists exploring and working with ReSilence’s technologies to improve their usability and uptake and (b) artists and end-users/providers of novel technologies collaborating in art-driven experiments addressing specific challenges through a variety of scenarios. These scenarios address either 1) the active participation of citizens, or 2) the new silence of mobility, or 3) the acoustic perception of outdoor urban soundscapes, or 4) the enhanced experiences for people with hearing and vision impairments, with all of them aiming to affect the urban/mobility sector. 

Implementation Scenarios
Musical Experience Design

Cities are not only places where people live and work; they also offer spaces for leisure, entertainment and art participation. Van der Hoevenan has done long research on how music becomes part of the urban landscape. ³ Concert halls, opera houses, theatres and other performance spaces are architectural landmarks in many cities.⁴ They offer spaces where urban communities can meet and share meaningful experiences. The increasing number of mediated performances (broadcasts, recordings, on demand and live streams) that can be consumed individually and at homes, is a chance and threat at the same time: They offer people with restricted mobility, small budgets, or that live far away from cultural urban centres the chance for participation. They also question the future and continuing value of such public spaces and the number and diversity of performing ensembles.

In this use-case scenario, we focus on live performances of music and dance and explore the value of shared aesthetic experiences. Live performances are defined by the temporal and spatial co-presence of the performers and their audience and the creation of the performed work in the here and now – features that can be broken down into further characteristics and that are relevant for the attractiveness of live performances and the intense and meaningful experiences they can afford.⁵ Artists will develop and use technologies to liquify the boundaries between in-person and mediatized performances. E.g., through the use of XR technologies we seek, on the one hand, to bring a social and interactive dimension to individual streams, and, on the other, to enhance and individualise live experiences. Therefore, in this use case artists in collaboration with technologists, scientists, architects and designers will explore new ways of designing musical and dance performances as well as interactive, performative spaces and environments, both physical and XR-enhanced.

In general, artists can adopt a quantitative or a quality approach in relation to audience experiences. In the qualitative approach, the aim is to use novel technologies to increase accessibility, participation and audience engagement. Here, new ways of enabling participation—on site and remotely—should be explored. Specifically, groups of people who are typically less well represented in certain types of concerts should be addressed, such as people with disabilities, people that do not belong to the socio-demographic group(s) that are related to the concrete type of concert and musical repertoire, and people who cannot attend live performances due to living far away, being immobile or not able to pay for a ticket. In the qualitative approach, the focus is on broadening and deepening the audience experience. Here, technology-assisted ways of making the music and concert experience more engaging and meaningful should be explored. One of them would be to allow audience members to actively participate in the performance. Further, it can be studied to what degree mixed reality events can provide experiences similar to live concerts, but also whether virtual or augmented concerts could be a performance format in its own right, with unique features and affordances for its participants, with a strong impact on the music and on the creative industry sector in general. Our research will extend on methodologies of studying live performances but also concert streams, for instance in the context of research projects such as Experimental Concert Research, Digital Concert Experience, the EU Projects H2020 ICT DANCE (dance.dibris.unige.it) and H2020 FET PROACTIVE EnTimeMent (entimement.dibris.unige.it), providing research results in automated analysis of expressive qualities of full-body individual as well as joint movement (e.g., of musicians in small ensembles, or audience members) and of interactive sonification: An example from the DANCE project is the performances in collaboration with the choreographer Virgilio Sieni involving audience members as active participants in museum and other public spaces, such as Atlante del Gesto in Genoa, and the Europa events in occasion of the celebration of the Treaty of Rome.⁶, ¹²

Such novel technology-driven experiences are a major departure from the conventional situation of the listener located in a definite space for the time of a concert. They are also a challenge for composers, musicians, audiences and the classical concert hall.

In this direction, ReSilence investigates how artists can explore and propose novel formats of concerts and performances characterised by audience engagement to create novel artworks and change perceptions of how technology can be used and how we experience it.

Besides the virtual platforms, Thessaloniki Music Hall (for performances over 50 participants), Casa Paganini and the ArtLab of the MPIEA (for smaller groups) will be used for live and for hybrid events and be test beds for audience co-creation.

The New Silence (Sound and mobility)

The sound fabric of the city has evolved and will keep mutating within the next decades. If the dominant sound of the city was the one of mechanics in action, from the combustion engine to the factories in action once investigated by artists such as the Futurist Luigi Russolo glorifying industrial noise in his Manifesto “The Art of Noises” (1913), we are getting in a society of services that wants to reduce the impact of the old mechanical model. ¹ Combustion creates two forms of pollution: chemical and noise. Electric vehicles start changing the sound paradigm. The evolution of this new kind of mobility objects can be compared with the introduction of a new species in an ecosystem. These 'species' need to find its right place by means of external signs embodying its reality concerning others.¹³ 

The electric vehicle as a potential predator for the pedestrian, does not warn the potential unfortunate preys of its coming. Citizens navigate in the city wearing headphones creating a parallel world that emotionally disconnect them from the reality of their environment. The accident is the collision of parallel realities that nobody perceives as coexisting. Suddenly the management of the most immersive component of our reality could become an unsurmountable challenge. The alternative now is to choose between fighting against the sound of the city or taming it, accepting the noise or designing the sound. We could also consider that this evolution resulting from technologies creates interesting opportunities to reconsider our urban sound environment. Often quoted without explicit references, Goethe was clearly anticipating in the 18th century the subtle relation between sound and architecture, music and the city: “Architecture is frozen music; Music is liquid architecture.”

This “liquid architecture” that fills the interstices of the city is a territory to reconquer. If the specific ambiance of each city is a signature, it is something to be culturally redefined, redesigned. While it is not only about reintroducing local music, but also accepting sound and noises as mediums to be worked out and contextualised. Car’s sound used to result from the specific mechanics that make the engine work. Now it can be designed as the expression of a brand, of a culture, of a driver’s personality, of a composer’s expression. Space can be acoustically improved to reduce some part of the sound spectrum in terms of frequencies or to modulate the global sound intensity. Draining asphalt for example can reduce significantly the sound reverberation of the streets. Reflecting surfaces like glass, concrete or polished marble increase the reverberation of the sound. Understanding and controlling these parameters allows the city to master its voice, to refine its expression. The feeling of belonging to a specific place may come from the light, the architecture, the climate, the language, the smell of food, and it may also come from the sound that reveals the city spirit and feelings. Can the car be the concert hall of the 21st century? Now that we no longer undergo the sound induced by noisy mechanics, we may consider designing the interior car sound—with or without music—as a combination of an ambiance a global receptor for environmental and circumstantial events. The car becomes an audio interface to perceive the outside in a way to prevent collisions, and to warn the surrounding people of the vehicle presence and intentions. This can be done in collaboration with companies working on the sound quality in situation of mobility.

Headphones that are sometimes isolating the pedestrian or the cyclist, may integrate subtle variation that help to perceive potential issues in the immediate environment. Detection of presence and proximity may be translated into a change of texture or reverberation of the music we listen to. This would be interpreted as a warning signal. Indoor acoustic design: Audio reflectivity of surfaces need special attention for the selection of construction materials. After acoustic music instrument makers, architects working on theatre and concert hall design have explored this field. It is time to apply this knowledge to places receiving public like restaurant, subway, and train stations. These places should be considered an acoustic standpoint and the quality of structural absorption and reflectivity will determine their impact on people and the quality of their experience. In McLuhanian vision of technology as being an extension of the human body, the car is the perfect prosthetics in our relation to the city.¹⁴ Bicycles, motorcycles, and cars allow us to accelerate our motion, amplify our spatial reach, and partially control our space-time relation to the urban environment.

Beyond being an engine of mobility, the car is providing a second skin, at the same time protecting us from potential aggression coming from unexpected obstacles and dotted by design with extended sensory-motor capacities to perceive our environment and the complex relation we, as human, may build with contextual aspects that could have been overlooked while moving at a higher speed than the familiar human walk. This sentient skin may play a new role in our relation to the environment, not only at the survival level, but also as an emotional translator allowing us to perceive and to interpret differently the surroundings.

The car may become a very versatile sentient instrument, simultaneously receiver, interpreter, environmental reader, and full musical dashboard. It can communicate with the passenger and with the surroundings. Like the shape of the organ pipes impact the harmonics, the timber and the quality of sound, the inertial relation the car build with its dynamic environment tunes the resulting auditory experience into a complex dialogue that the driver can understand, and the composer can interpret.

The parameters the composer//interpreter can play with are, but not limited to:

Building a relation between users and the environment mediated by the car skin. Users make one with the car navigating in the city/country side. The most recent technologies convert the vehicle into an autonomous artefact able to perceive, analyze, recognize, and react, thanks to sensors and more and more of artificial cognitive functionalities like AI, extending human perception with infrared and ultra violet light, ultrasounds. We imagine how new narrative may go beyond augmented perception allowing new narrative. The traveling experience may become a live geolocalise documentary, a romantic journey or a science fiction experience. From ambient relaxation, and entertainment, vs stimulation awareness, the scenario will have to find a middle way between fiction and the actual reality with potential dangers. As the most obvious evolution of technologies, musical AI models may apply through sets of references open to specific creation related to the user or to the car make.

Sound of Urban Spaces

The problem of noise pollution is a serious challenge in urban areas. Soundscape design has been an issue in city-planning over the last years; there are still tools and methods needed to shape the sound of spaces to ensure public well- being, and for the time being we do not look enough towards the potential solutions provided by artists.¹⁵, ¹⁶ While the sound of spaces often is already considered during their urban planning and architectural design state, the problem mostly affects existing spaces, which demand sound analysis and redesign of restorative soundscapes. We aim towards building tools and techniques that will allow architects and urban designers to address the issue of noise pollution in various scales, led by innovative thinking by sound artists. This will require simulations and fabricated prototypes in small sections of the urban environment, where few individuals are involved, and gradually expand to larger areas of cities and communities of citizens. This scenario aims to create tools that help city planners and researchers to speed-up the analysis of the sound of cities and give urban designers and architects new possibilities to create urban spaces in an inclusive way so that people can feel comfortable. Furthermore, it opens new space for creativity not limited to urban design but also for all artists working with sound. Some artists are already working on soundscapes of cities, such as Justin Bennett who investigated acoustic territories of the city of Brussels and created devices for active listening such as a microphone inside a trumpet or a wooden stethoscope, for example.¹⁷

Another interesting artwork related to urban space and sound is“Signals and Hums” by Banu içek Tülü. It refers to a soundscape work that asks listeners to concentrate on perception. The artwork exhorts viewers to hone their hearing and listening skills. Sounds are connected to past experiences and assist us navigate the city on a daily basis. Sound is also a crucial component in understanding our surroundings.Acoustic environment is not just background noise, a particular soundscape, or noise pollution brought on by traffic or everyday activities. Instead, the project focuses on appreciating auditory experiences and exploring sound landscapes in urban environments through listening and hearing.¹⁸, ¹⁹ Soundscapes are defined from an ecological perspective as the physical, biological, and anthropogenic sound that makes up a landscape, it is presented as a "footprint of an ecosystem," reflecting the dynamics of a community structure and function.²⁰ With the help of machine learning, mixed reality and intelligent interactive systems, artists work with existing and future architectural and urban spaces, considering their visibility and auditory functions. This gives the opportunity to architects and planners to work on the composition of urban soundscapes, working in parallel in areas with “silent parts” intermixed with more “noisy parts” and design individual acoustic interventions for specific zones on a city scale. While at the same time working on small neighbourhood scales using innovative design and material properties such soundproof or sound reflecting materials in VR environments.

Using 3D-audio-capturing systems, 3D-laser scanners as well as 3D-modelling technologies, we design and create immersive / navigable audio-visual computational models of existing spaces at various scales of the city. The AIM (Audio-visual Immersive Maps) platform offers an immersive, virtual experience of cities and places around the world.¹⁹ Its functions range from sound integration in Google Streetview to high-resolution 3D scans of buildings, districts or cities. AIM makes real places virtually tangible also through hearing—in the browser, on the smartphone, in 3D glasses or in the VR room. Using geotagging, AIM enables people to experience fully immersive virtual travel with 3D Sound. AIM will be further developed to help urban planners to better assess and design soundscapes of cities and thus improve the sonic environment and well-being in the city.

ReSilence also provides artists and designers with tools and simulations to analyse and test improvements on the soundscape design. Such simulations could allow measuring behaviour (and possibly brain imaging) on people inhabiting different versions of such spaces. Body and movement analysis²¹  including expressive qualities of full-body movement,⁹ as well as non-verbal social signals in small groups (e.g., real-time measures of entrainment, of leader-follower relations)¹⁰ can give further indications about how citizens react to pre-existing conditions of polluted urban sounds (e.g., trajectories of pedestrians to avoid noisy points; indications of confusion and hesitation in correspondence with noisy zones). Evaluation and validation can be realised through measuring the change of group behaviour of people in urban spaces, with respect to changes in urban sounds and environment. The same technologies can measure effects of design interventions, through the use of prototypes (time spent in silent points, slowdowns in pleasant sonic zones, etc.). Real-time sonification of traffic (trajectories, volumes, speed) and environmental conditions are used as a tool to generate interactive sound, meant to counterbalance noisy emissions with background sounds of stability and fluidity. Such sounds characterised by fluidity might be also related with real-time measures of the fluidity (Vs. hesitation, rigidity) in movements of citizens: cross-sensory correspondence of movement and sound qualities is a very interesting and challenging direction in this and in other ReSilence scenarios. Examples of scientific results and technologies in this direction of movement analysis and interactive sonification, available also for future experiments in ReSilence, can be found from the previously mentioned EU Projects DANCE and EnTimeMent.⁶, , ¹¹

Sound and social inclusion

Through vibration and tactile sensation, we can achieve a “new way of hearing” and through sound we can find “new ways of seeing”. This use case will focus a) on the importance of a full-body approach to experiencing music and sound as opposed to engaging solely the ears and b) on the translation of physical objects into soundscapes and vibrations, inspired by artists such as Christine Sun Kim. By focusing on individuals who are hearing impaired or visually impaired, ReSilence will work towards creating richer experiences for people with disabilities. There are 34.4 million adults with a disabling hearing loss in EU.²² Hearing impairment can mean a loss of hearing, but it can also mean sensitivity to loud noise, or certain frequencies.

In addition, there are estimated to be over 30 million fully or partially blind citizens in geographical Europe. Most XR technologies are incapable of providing enhanced information for visually and audibly impaired people.²³ Traditionally we expect sound to be experienced through our ears. However, nowadays we have the ability to create multimodal tools and wearables that can support a full-body approach in experiencing music and sound.²⁴ 

Concerning visual impairments, the H2020 ICT DANCE project (dance.dibris.unige.it) focused on the automated analysis and interactive sonification of the expressive qualities of full-body movement. Scientific results and technologies developed by Casa Paganini and Maastricht University were experimented in artistic projects: (i) Emobodies (Maastricht Jazz Festival—performance of the blind pianist Bert van der Brink with the dancer Sagi Gross, in which the pianist could “hear the dance” and perform in a duo); Atlante del Gesto (performance in four different public sites in Genoa by the choreographer Virgilio Sieni involving active participation of about 150 citizens: in one of these sites the blind dancer Giuseppe Comuniello performed as leader of the active audience with the support of interactive sonification of movement qualities). More details and video excerpts are available in ¹² and in ⁶, , ¹¹. Casa Paganini developed recently the DanzArTe - Emotional Wellbeing Project (youtube.com/Infomuslab), an interactive system supporting embodied experiences of visual arts (classical religious paintings, image manipulation and 3D modelling) based on the real-time analysis and interactive sonification of expressive movement. Expressive movements of visitors in museums or cultural institutions are measured in real-time (e.g., fluidity, entrainment in a dyad moving together) while imitating the affective gesture in a painting. This is also successfully adopted to support aesthetically resonant physical exercises and rehabilitation in frail older people and in general for cultural wellness and inclusion. This example of cultural welfare interactive technology is a possible challenging direction for ReSilence.

Exploring innovative technologies to bypass impaired sensory organs (like eyes and ears) can enhance our spatial awareness, allowing us to perceive sound and visuals through other parts of the body. This novel approach offers a more comprehensive sensory experience, including the ability to "hear" through the body.

Enhancing our bodies with technology can empower individuals with disabilities, enabling them to experience the world, communicate, and foster a stronger sense of community, thereby improving city planning and a sense of belonging.

Conclusion

This paper presents ReSilence, a transdisciplinary project that emphasizes silence and the space between sounds. ReSilence facilitates the uptake of digital technologies in the Mobility/Urbansim sector through the integration of technologies such as XR environments, data analysis and artificial intelligence in an ecosystem of tools designed to support citizens’ needs, motivations and experiences of the city. Through art-driven experiments with these technologies, ReSilence explores the multiple ways in which cities are constructed, perceived by incorporating soundscape information into urban spatial arrangement. The real-world scenarios of ReSilence demonstrate the expected impact of the proposed solution by focusing on specific societal problems related to cities soundscapes (indoor and outdoor).

Acknowledgements

This research was funded by the EC-funded research and innovation programme Horizon Europe ReSilence: Retune the Sound-scape of future cities through art and science collaboration under the grant agreement No. 101070278


(1) https://ec.europa.eu/regional_policy/en/policy/themes/urban-development

References

1 Russolo, Luigi, The Art of Noises, Pendragon Press, 1986. 

2 Vaessen, Maarten, Kiki Van der Heijden, and Beatrice de Gelder, "Decoding of Emotion Expression in the Face, Body and Voice Reveals Sensory Modality Specific Representations," BioRxiv, 2019, 869578.

3 Hoeven, Arno van der, Erik Hitters, "The Spatial Value of Live Music: Performing,(Re) Developing and Narrating Urban Spaces," Geoforum 117, 2020, 154–64.

4 Smith, Neil Thomas, "Constructing the Public Concert Hall," Journal of the Royal Musical Association 146, no. 2, 2021, 255–81.

5 Wald-Fuhrmann, Melanie, Hauke Egermann, Anna Czepiel, Katherine O’Neill, Christian Weining, Deborah Meier, Wolfgang Tschacher, Folkert Uhde, Jutta Toelle, Martin Tröndle, "Music Listening in Classical Concerts: Theory, Literature Review, and Research Program," Frontiers in Psychology 12, 2021, 638783.

6 Camurri, Antonio, Gualtiero Volpe, "The Intersection of Art and Technology," IEEE MultiMedia 23, no. 1, 2016, 10– 17.

7 Czepiel, Anna, Lauren K. Fink, Lea T. Fink, Melanie Wald-Fuhrmann, Martin Tröndle, Julia Merrill, "Synchrony in the Periphery: Inter-Subject Correlation of Physiological Responses during Live Music Concerts," Scientific Reports 11, no. 1, 2021, 1–16.

8 Tschacher, Wolfgang, Steven Greenwood, Hauke Egermann, Melanie Wald-Fuhrmann, Anna Czepiel, Martin Tröndle, Deborah Meier, ‘Physiological Synchrony in Audiences of Live Concerts’. Psychology of Aesthetics, Creativity, and the Arts, 2021.

Antonio Camurri, Gualtiero Volpe, Stefano Piana, Maurizio Mancini, Radoslaw Niewiadomski, Nicola Ferrari, Corrado Canepa, "The Dancer in the Eye: Towards a Multi-Layered Computational Framework of Qualities in Movement," In Proceedings of the 3rd International Symposium on Movement and Computing, 2016, 1–7.

10 Giovanna Varni, Maurizio Mancini, Luciano Fadiga, Antonio Camurri, Gualtiero Volpe, "The Change Matters! Measuring the Effect of Changing the Leader in Joint Music Performances," IEEE Transactions on Affective Computing, 2019.

11 Niewiadomski, Radoslaw, Maurizio Mancini, Andrea Cera, Stefano Piana, Corrado Canepa, Antonio Camurri, "Does Embodied Training Improve the Recognition of Mid-Level Expressive Movement Qualities Sonification?", Journal on Multimodal User Interfaces 13, no. 3, 2019, 191–203.

12 Esodi, Public performance at Museo di Palazzo Reale, Genoa - March 25th 2017, accessed December 9, 2022 http://dance.dibris.unige.it/index.php/dance-media#a_vid_artistic

13 Nicolas Misdariis, and Louis-Ferdinand Pardo, "The Sound of Silence of Electric Vehicles--Issues and Answers," In InterNoise, 2017.

14 Marshall McLuhan, "Media Hot and Cold," Understanding Media: The Extensions of Man, 1964, 22–32.

15 Reeman Mohammed Rehan, "The Phonic Identity of the City Urban Soundscape for Sustainable Spaces," HBRC Journal 12, no. 3, 2016, 337–49.

16 Yildirim, Yalcin, Merve Dilman, Volkan Muftuoglu, and Sara Demir, "Soundscape Assessment of Green and Blue Infra-structures," Urban Science 6, no. 1, 2022, 22.

17 J. Bennett, “justinbennett.nl,” accessed January 01, 2022 http://www.justinbennett.nl

18 Sounds of our cities, accessed December 9, 2022 https://www.idensitat.net/en/current-projects/sounds-of-our-cities-3/1456-sounds-of-our-cities-open-call

19 “Audiovisual Immersive Maps,” AIM / Audiovisual Immersive Maps, Accessed January 01, 2022. https://weaim.de/en/

20 Barry, Truax, Gary W. Barrett, "Soundscape in a Context of Acoustic and Landscape Ecology", Landscape Ecology 26, no. 9, 2011, 1201–7.

21 Gelder, Beatrice de, Marta Poyo Solanas, "A Computational Neuroethology Perspective on Body and Expression Perception," Trends in Cognitive Sciences 25, no. 9, 2021, 744–56.

22  “Europe- Hearing-loss | hear-it.org,” accessed December 9, 2022 https://www.hear-it.org/hearing-loss-in-europe

23 Joshue O Connor, Shadi Abou-Zahra, Mario Covarrubias Rodriguez, Beatrice Aruanno, "XR Accessibility--Learn- ing from the Past and Addressing Real User Needs for Inclusive Immersive Environments," In International Conference on Computers Helping People with Special Needs, Springer, 2020, 117–22.

24 Alice Haynes, Jonathan Lawry, Christopher Kent, Jonathan Rossiter, "FeelMusic: Enriching Our Emotive Experience of Music through Audio-Tactile Mappings," Multimodal Technologies and Interaction 5, no. 6, 2021, 29.

Bibliography

Antonio Camurri, Gualtiero Volpe, "The Intersection of Art and Technology," IEEE MultiMedia 23, no. 1, 2016, 10–17.

Antonio Camurri, Gualtiero Volpe, Stefano Piana, Maurizio Mancini, Radoslaw Niewiadomski, Nicola Ferrari, Corrado Canepa, "The Dancer in the Eye: Towards a Multi-Layered Computational Framework of Qualities in Movement," In Proceedings of the 3rd International Symposium on Movement and Computing, 2016, 1–7.

Anna Czepiel, Lauren K. Fink, Lea T. Fink, Melanie Wald-Fuhrmann, Martin Tröndle, Julia Merrill, ‘Synchrony in the Periphery: Inter-Subject Correlation of Physiological Responses during Live Music Concerts’. Scientific Reports 11, no. 1, 2021, 1–16.

Beatrice de Gelder, Marta Poyo Solanas, "A Computational Neuroethology Perspective on Body and Expression Perception," Trends in Cognitive Sciences 25, no. 9, 2021, 744–56.

Alice Haynes, Jonathan Lawry, Christopher Kent, Jonathan Rossiter, "FeelMusic: Enriching Our Emotive Experience of Music through Audio-Tactile Mappings," Multimodal Technologies and Interaction 5, no. 6, 2021, 29.

Hoeven, Erik Hitters, "The Spatial Value of Live Music: Performing,(Re) Developing and Narrating Urban Spaces," Geoforum 117, 2020, 154–64.

Marshall McLuhan, "Media Hot and Cold," Understanding Media: The Extensions of Man, 1964, 22–32.

Nicolas Misdariis, Louis-Ferdinand Pardo, "The Sound of Silence of Electric Vehicles--Issues and Answers," In InterNoise, 2017.

Radoslaw Niewiadomski, Maurizio Mancini, Andrea Cera, Stefano Piana, Corrado Canepa, Antonio Camurri, "Does Embodied Training Improve the Recognition of Mid-Level Expressive Movement Qualities Sonification?", Journal on Multimodal User Interfaces 13, no. 3, 2019, 191–203.

Joshue O Connor, Shadi Abou-Zahra, Mario Covarrubias Rodriguez, and Beatrice Aruanno, "XR Accessibility--Learning from the Past and Addressing Real User Needs for Inclusive Immersive Environments," In International Conference on Computers Helping People with Special Needs, Springer, 2020, 117–22.

Reeman Mohammed Rehan, "The Phonic Identity of the City Urban Soundscape for Sustainable Spaces," HBRC Journal 12, no. 3, 2016, 337–49.

Russolo Luigi, The Art of Noises, Pendragon Press, 1986.

Smith Neil Thomas, "Constructing the Public Concert Hall," Journal of the Royal Musical Association 146, no. 2, 2021, 255– 81.

Barry Truax, Gary W. Barrett, "Soundscape in a Context of Acoustic and Landscape Ecology," Landscape Ecology 26, no. 9, 2011, 1201–7.

Tschacher Folkert Uhde, Jutta Toelle, Martin Tröndle, "Music Listening in Classical Concerts: Theory, Literature Review, and Research Program," Frontiers in Psychology 12, 2021, 638783.

Tschacher Wolfgang, Steven Greenwood, Hauke Egermann, Melanie Wald-Fuhrmann, Anna Czepiel, Martin Tröndle, Deborah Meier, "Physiological Synchrony in Audiences of Live Concerts," Psychology of Aesthetics, Creativity, and the Arts, 2021.

Vaessen Maarten, Kiki Van der Heijden, Beatrice de Gelder, "Decoding of Emotion Expression in the Face, Body and Voice Reveals Sensory Modality Specific Representations," Bio-Rxiv, 2019, 869578.

Melanie Wald-Fuhrmann, Hauke Egermann, Anna Czepiel, Katherine O’Neill, Christian Weining, Deborah Meier, Wolfgang Tschacher, Folkert Uhde, Jutta Toelle, Martin Tröndle, "Music Listening in Classical Concerts: Theory, Literature Review, and Research Program," Frontiers in Psychology 12, 2021, 638783.

Yalcin Yildirim, Merve Dilman, Volkan Muftuoglu, Sara Demir, "Soundscape Assessment of Green and Blue Infrastructures," Urban Science 6, no. 1, 2022, 22. 

Towards a Sympoietic Relation with Materials in Interactive Artworks

Nefeli Georgakopoulou, Dionysios Zamplaras, Sofia Kourkoulakou, Chu-Yin Chen

INREV Laboratory, Paris 8 University
Paris, France
nefeli.valeria@gmail.com ; dzamplaras@gmail.com ; sopheeah.k@gmail.com ; chu-yin.chen@univ-paris8.fr

Abstract

In this paper we acknowledge the agency of non-human entities and argue against the binaries of subject/object, mind/body, nature/ culture, science/art towards a new materiality. This new vision of the nature of materiality changes the direction of passive matter into a more active one. Technology has given us the opportunity to characterize and analyze material systems not only by their properties, but also by their potentialities. This leads to a sympoietic relation boundary between human-matter-machine interactions. In the context of an interactive artwork, agency should not be considered as inherent for any of the actors. It is emergent, it is the result of the interactions between the elements and the entities located within a mixed reality environment, as much inside the installation—material, devices, objects, sensors, humans—as at its exterior—such as the cultural context and the artist, for example. This new materiality which is based on an “open material” concept is an emergent and co-constitutive process, continuously becoming in, with and through interactions.

Keywords

interactive art, materiality, new materialism, symbiosis, sympoiesis, becoming.

DOI

10.69564/ISEA2023-59-full-Georgakopoulou-et-al-Sympoietic

Introduction

While the modern world still struggles with the polarities of mind/spirit, nature/culture, physical/digital, material /immaterial, many writers and philosophers recently have attempted to challenge this and develop different theoretical approaches with a more holistic understanding of our environment and our planet. Such theories often called "feminist materialism" or "new materialism" and writers such as Donna Haraway, Bruno Latour, Jane Bennett, Karen Barad and Manuel DeLanda tend to adopt the idea of complexity, material action and imply learning ways to coexist respectfully with the world of matter. This means that primacy is not only given to the human actor, but also to the non-human ones. These studies also focus on the interrelations between material agency and social phenomena and help us to imagine an open material that is based on the idea of movement and agentivity; a material capable of behaving in a complex and multidimensional reality in which the relationship between subject and object is fluid. They give us a new definition of human's relationship with matter, one that is closer to a magical vision of the world. This current paradigm shift in the way we think about, conceptualize, and experience human relationships to their surroundings has led to new forms of artistic experimentation and brought new interfaces between humans, machines and materials into light. The goal of this research is to examine and create interactive systems that promote a symbiotic and sympoietic relationship between human-matter-machine interactions (HMMI). Here, we are not interested in creating a virtual alternate of reality but in capturing the mystery of the world around us. The research reported in this contribution deals with fluid materials (natural and intelligent), sound, and light in order to create complex and constantly evolving systems.

Background

Modern science has demonstrated, in theory and experiments, that the atom (matter’s fundamental units) is not an indivisible particle, nor the smallest unit of matter. The existence of subatomic particles has been proved when experiments showed that light could behave like a flow of particles (photons), all the while having wave-like properties. Quantum physics, therefore, under an alternative (conceptual) framework reveal new characteristics of matter and raise questions about materiality. Quantum mechanics sometimes go beyond just explaining the basic structure and interaction of molecules and tackles questions relevant with life, such as the interaction between mind and body, material and immaterial or living and non-living matter. We find out that in the very nature of materiality is an entanglement.¹ Bruno Latour, based on the indistinction of things and people, has formulated a social phenomenon called "Αctor-Νetwork Τheory" (ANT). ΑΝΤ refers to and includes both humans and non-humans in the same network. Realities are understood as networks of actors, both people and things.² 

In this perspective, there are no a priori distinctions between subjects and objects, nature and culture, man and machine. As Bruno Latour writes, "... it is we, the Westerners, who have lived until now in the strange feeling that we had to separate into two distinct collectives, according to two forms of incommensurable gatherings, the 'things' on one side, the 'persons' on the other."³

Bennett's work also focuses on ideas about the relationship between humans and "things," a relationship she calls "vital materialism." In her book, Vibrant Matter: A Political Ecology of Things, she explains that by vitality she means "the capacity of things—edibles, commodities, storms, metals—to act as quasi-agents or forces with trajectories, propensities or tendencies."⁴ Such a relationship between the human and non-human worlds can give rise to the concept of enchantment, which, as Bennett describes it, is a sense of openness to the unusual, the everyday captivating, and can be found in nature, for example, but also in such unexpected places as modern technology.

All of the above thinkers seek to defy and deconstruct the subject/object division that has been imposed by "modern" techno-scientific thinking as a paradigm for how we, as subjects, understand and form the world. What is interesting is that they all try to seek in art a link beyond the modern paradigm, in order to reveal the hidden creative potential of technology. Specifically, Bruno Latour, while deconstructing the subject/object divide, points out that the mind and the world are separated in our modern worldview, giving the former the power to directly control and dissect the latter. On the basis of such a theory, he would try to question this worldview in order to open the path to alternative possibilities by attributing agency to the non-human.

Theoretical framework Living with materials (Symbiosis)

Symbiosis is examined in this research as a system that establishes a dialogue between a human and their environment (materials more precisely). In Greek, there are two different verbs to describe life. Ζω and the old βιώ. The difference between the two is that ζω, "live," refers to the act of living in a sense of "spending life" or just existing, while βιώ refers to an experience, in the sense of having an intense life. Life was transformed through politics and public life from an existential condition into what Aristotle called bios. Through his work “Homo sacer: sovereign power and bare life,” Agamben asserts that “the fundamental categorical pair of western thought and politics is that of bare life/political existence, zoē/bios, exclusion/inclusion. However, nowadays we are moving away from an anthropocentric way of thinking which places human beings at the center of social existence and many thinkers and writers highlight the ‘active role of nonhuman materials in public life’. ⁵

Making with materials (Sympoiesis)

Poiesis means the activity in which something comes into being that did not exist before. Poiesis derives etymologically from the Greek ποιεῖν, which means "to make." For Haraway, poiesis is always already sympoiesis. In Donna Haraway's view, nothing can really create itself.⁶ Thus, nothing is really autopoietic, but requires other organisms for it to become what it is. This creates a necessity to combine the theory of autopoiesis with a theory called sympoiesis, which relates to collectively produced systems, not autonomous ones. In other words, sympoiesis includes and extends autopoiesis.

Donna Haraway adopts the theory of sympoiesis from Beth Dempster. Beth Dempster (1998) introduced the term "sympoiesis”, which was coined by Friedrich Schlegel (1800), as a way to describe systems that lack rigid boundaries. More specifically, these systems “have cooperative synergistic characteristics and must be identified by the continuing interactions among components [...] the systems are evolutionary and have the potential for surprising change.”⁷

Following this notion, we see interactive artworks as having a greater focus on relationships rather than on the individual parts from which they are composed.

While such an interactive artwork may begin as an amorphous form with blurred boundaries, it will acquire shape and identity by interacting with others (human and non-human). These interactions can change the direction of the outcome and force reinterpretation of the previous forms from different perspectives. As the shape of such an artwork is in constant change and re-formation, the “life” of such a work is not dependent on any individual entity (agent, whether human or non-human). By contrast, it is distributed between their relations, their exchange, and their flow of energy. Interactive artworks attempt to highlight the fact that we need an “ecosystemic” approach, emphasizing that we must shift our perception of our position in the world as well as our agency. Sympoiesis as a system should be considered alongside environmental, social, and mental ecology through an "ecosophie”. Through his collaboration with Deleuze, F. Guattari focused on heterogeneity and difference, articulating agencements and multiplicities to find rhizomatic structures rather than unified and holistic ones.

Likewise, sympoiesis is a term that refers to assemblages which acquire their uniqueness and identity through interactions and the process of becoming in the Deleuzo-Guattarian sense where new “movements” emerge from creative forces.⁸ Therefore, for this research we started with the idea to create a dynamic microcosm that could perform a continuous becoming of form, structure and material narrations.

Becoming with materials &
the becoming of form

The compound syn, from σύν ("with, together" of symbiosis, sympoiesis) narrates interdependencies of bodies, materials and things, while “feeling these connections is the principal stuff of becoming-with.”⁹ This becoming-with line up with Karen Barad's agential realism, which describes the world, not composed of discrete ‘things’, but “phemomena-in- their-becoming”.¹⁰ By taking into account new materialistic theories, we understand that materials are important in order to explore the potentialities of our world in which we live in and engage with them in a process of becoming something new (worlding (1)). As proposed in this research this can be achieved by transforming our symbiotic relationship with the world into a sympoietic one which enables us to look deeper into intertwined human-world relations. Becoming is ongoing and dynamic and entails material-discursive practices of transitioning from one state to another in space and time.¹¹

The nature of these practices and artworks lies in an effort to communicate with the materials leading to a new materiality which renews contemporary art in a way that the old hylemorphic couple material / form is re-examined. The material is explored as something that not only affects the form and meaning of the artwork, but also dialogs with the viewer. The goal of these artworks is not to give a “magic” wand to the participant to manipulate matter, on the contrary the objective is to “immerse” him in a milieu of energy exchange. Based on Simondon’s concept of milieu which is not primarily a spatial concept but a system of energy transfer between an object or subject and its environment, we approach interactive artworks as the potential for many different kinds of becoming of form. What Simondon wanted and we acquire is to find a theory which encloses both the archetypical and hylomorphic theory through a new approach of information and communication. Simondon found this possibility of unification within the field theory (physical and psychological) and energy states (referring to quanta). ¹²

In the 21st century, new interactive materials are being designed whose action extends beyond their physical limits, mixing materials with intangible (digital) information. In the context of the digital revolution, we assign to materials our functions such as memory, intelligence, even emotion, which leads us to a new definition of materiality. For Christine Browaeys: “We have to imagine an open material that is based on the idea of movement, capable of behavior. Thus, Man finds himself immersed in a new materiality in contact with the digital.” ¹³

Christine Browaeys, in her book "Materiality in the Digital Age: The Human Connected to Matter" emphasizes that materials and technologies are combined in a symbiosis within the field of Human-Machine interaction where priority is given to improving the performance and behavior of the material. The scientific models we have today to understand the evolution of the world are to be reviewed, leading us to a new understanding of matter overcoming the duality of matter/spirit or matter/form.¹⁴

Digitization and virtualization have tended to disconnect the average person from materiality. They lead us to believe that creating something 'intelligent' only means a digital system with software that simulates human intelligence. But a great part of our human intelligence is built with and invested in materials, not digital technology. We have lost interest in the intelligence of our material world. For that reason, we choose to tackle materiality from the more dynamic side of the material, which can be defined as an active matter wherein the interaction flow between man-material-machine unfolds. These materials are becoming, only over time and in interaction with humans through technology.

Sympoietic relations in HMMI
The becoming of the Interface

Many theorists, like Nathaniel Stern, reveal the performative aspect of the complex relationship between the material world, the body and technology, in the context of interactive installations.¹⁵ In the same spirit, Karen Barad introduces the notion of intra-action. In Barad’s view, intra-action is a term used to replace ‘interaction,’ because the latter suggests pre-established entities with individual agencies that precede their interaction, while the latter focuses on the participatory action. Intra-action posits agency as emergent. Thus, the idea that beings and entities preexist as individuals with inherent attributes, anterior to their representation, is contested through a performative approach, according to which the entities are being constantly co-constructed through an evolving relationship.¹⁶

When trying to approach the aesthetic experience of the interactive artwork, through the prism of intraction, the various entities are mutually co-constituted, shaping through their complex and ever-changing relations.¹⁷ As entities can be considered for example the artist, the spectator/participant, technology, objects and materials, as well as the cultural context of the presentation. In the case of the interactive installation, it is usually the various interfaces which are in charge of providing the means for these relations to flourish and develop. Thus, the artistic interface makes possible the relation between the body and technology, and through this relation the interface is actualized—in the sense of becoming. The symbiotic and sympoietic relations and the effects and events taking place during the meeting of materials, bodies and technology bring the interface and all the entities into being.

The materials and the interface are not simply a part of the installation, but rather with the installation and the other entities in a process of being with the others. Phaedra Shanbaum considers that these ideas are reaching towards the concept of Being in the philosophy of Jean Luc Nancy, where “being” cannot be defined outside of “being with.” Rather in a relational conception, “being” is a process of becoming with the others.¹⁸ Thus, all interactions that occur within an interactive installation are experimental and procedural processes that are produced in conjunction with - and influenced by - the others. The presence of the other entities is a prerequisite for the very notion of interactivity with the artwork to occur, although this by no means signifies the distinction and separation of matter and technology, of the human “Me” and the technological “Other.”¹⁹ A creative meeting of all the entities leads to the sympoietic and symbiotic relations that bring the interactive artwork into being.

The notions of being and becoming are deployed here in order to help us develop a theoretical framework for the analysis of the role of matter/materials and the artistic interface as emergent through the events and the meeting between body and technology. None of those preexist as such, outside of an enactive system—formed by all the participating entities, rather they become inside their complex relation and the sensible experience of matter and technology. Through action, experimentation and the always evolving co-constructed relation, the meeting with matter and the interface is an aesthetic temporal process, initiated by the artist in the moment of the conception and continually rethought through the meeting of bodies, materials and technology.

Agency as a creative
sympoietic force

In the context of interactive artistic installations, the relation between human, matter and technology can be partly theorized by the hybridization with and through the various actors. Nevertheless, when designing HMMI, from the moment of its very conception, interactivity is seen through the perspective of communication and agency of matter and technology.

Thus, agency allows us to rethink our interactions with the material and the technological world surrounding us. One of the fathers of informatics and artificial intelligence, Alan Turing, once proposed a test that has been used as a validation tool that would allow us to classify a machine as “thinking”. Nevertheless, for Kate Forbes-Pitt, even though this test consists of an undeniable contribution in the evolution of computers and artificial intelligence, it also provides an enormous contribution in social and cognitive sciences, as well as in machine mediated artistic environments.²⁰

According to such a point of view, the way that this test was conceived and proposed simultaneously provides us with an assessment of how humans perceive and interact with other human beings and with technology, as well as their expectations and affordances vis-a-vis these interactions. It could even be suggested that we are being more informed about the perceptual capacities of the human actor than about the capacities of the machine itself. Undeniably, during the exchanges between several entities—humans or between humans and non-humans—there is agency. In our research, the exchanges between these entities are represented in the relationship between matter, technology and humans.

For Karen Barad, agency is a matter of interactions, an enactive relationship.²¹ In her theory, agency is conceived as various possibilities of the reconfiguration of the viewer's body, of material and of technology. It cannot be a pre-established attribute of either the subject or the object, a condition that pre-exists in an entity in its conception, outside the context of interactions and relationships. On the contrary, in the case of the interactive artistic installation for example, agency is produced by the choices of the human who, consciously or unconsciously, positions and interacts with the other entities—the artistic interface, the materials, and of course technology.

However, we cannot consider agency as an a priori attribute of the interface, since neither it nor the human can have a “stable existence” outside of a cultural field of exchange.²² On the other hand, the performative nature of the experience of the interactive work makes the interface unstable and this is possibly one of the reasons it opens up the possibility of agency.²³ Nathaniel Stern, for his part, also emphasizes this performative aspect by explaining that the relationship between the body and the interactive work is constitutive, a “performed and emergent emergence”, a process that is constituted “ in and with and through" the relations.²⁴

As such, it has been proposed by several theorists that separating human and non-human entities includes the risk of reducing their actual role in a co-constitutive relationship. Indeed, for the study of the interactive artworks, it will be more beneficial to consider their emergence as an encounter between the actions, efforts and effects of specifically located humans, materials and technology.²⁵ This enactive whole composed of efforts, effects, people and matter, operates in the specific cultural context of the work, within which the machine is not an autonomous agent but the precondition for the participation and the study of this set. For Suchman, the problem is less the attribution of agency to technology and more the fact that our language of speaking for agency presumes and imposes a field of autonomous and discrete entities. As an alternative approach, she suggests that we consider interfaces as entities in the making through their relationship with other entities.²⁶

All the aforementioned thinkers help us to imagine an open material that is based on the idea of movement and agentivity; a material capable of behaving in a complex reality in which the relationship between subject and object, human and non-human, nature and culture, science and art is fluid. They give us a new definition of human's relationship with matter and technology, one that is closer to the enchantment of the material world, which leads us to renew our conception of matter as an agent in the becoming of the artwork.²⁷

Artistic Installations as Case Studies Ferrofluid interaction and the becoming of form

Liquid Matters (2019) (2) is an interactive installation dealing with materials in a playful way; by letting the participant interact via gestures and thus a natural interface, triggering movement of organic matter (ferrofluid particles) in a container liquid display, while this interaction and emergence of forms is reprojected in real time in a large video installation. (Figure 1) 

Figure 1. Liquid Matters, Ferrofluid Material 2022 ©Respect Copyright. 

The ferrofluid drops, organic and fluid as they are, flowing around, are controlled by the participant via a leap motion device. Thus, LiquidMatters, suggests a sympoietic artificial organism between human (experiencer), non-human matter (ferrofluids) and machine (Arduino controller and leap motion sensor, filed under computer vision). This simple combination of ferrofluid and magnet coupled with computer vision and real time interaction, becomes a complex system of a sympoietic node towards a material other. The organic form is co-created as a response to the feedback human—machine loop, but in this case, the machine is organic pixel forms: they come in contrast with the regular pixels, thus confusing us on their nature; are they living creatures or maybe simulated ones?

In LiquidMatters installation (Figure 2), ferrofluid behavior is controlled by many technical factors, such as the distance between magnets, the power and the number of magnets and ferro particles itself, as well as the density of the container liquid. Ferrofluid movement is controlled by an Arduino device that translates the visitors’ input gestures and hand movements to electromagnetic trajectories in the x and y axis. LiquidMatters is a fragile in nature experiment: if the particles don’t interact, they sit still in their container liquid. As soon as our attention and gestures are captured new forms emerge, like a game of life of swarm ferro-particles. The significance of this ‘augmented- diminished’ reality experience is referring to the economy of attention between visitor and ferrofluids and in the sympoiesis of an entangled medium which becomes the artwork.

Through the force of a magnetic field ferrofluid becomes alive, complex in its behavior. By seeing this liquid as a material with intelligence, we activate and animate it. Digital technology allows us to see it as a responsive system where we are not simply imposing our will upon the environment but letting it reveal certain kinds of properties or possibilities through its constant fluidity. When dealing with matter in such a way, the agency of humans, machines and that of the materials unite, co-create and make visible the superimposition between the realms of a machine, human expression, and of matter itself. Instead of using digital processes or coding to generate forms, we used natural dynamic forces such as magnetism, gravity, and chemical reactions. 

Figure 2. Liquid Matters interactive installation, Laval RectoVRso Gallery, 2019 © Respect Copyright. 

Seeing through touch:
Handprints as proof of sympoietic interfaces

VitRails (2018) is a mixed reality interactive installation, on the mediterranean refugee crisis. The multi-sensorial interactive installation uses a black thermochromic surface as an interface and a VR headset (Figure 3.). The visitor wears the headset, and is asked to touch the black surface presented as a window frame (Figure 4). The thermochromic behavior of the surface’s material allows a temporary color change on the black surface, making the black transparent for a few seconds, and this, by touching, thus transmitting heat to the surface, using our hands. As soon as the surface changes its temperature by touching, a virtual landscape appears in the headset. Computer vision coupled with this behavioral object, the thermochromic paint, are the interface of this installation and allow visitors to enter the Virtual Reality world. Visitors can see through their handprints, by touching the surface. Once achieving access to the virtual world through his traces, he or she has only a few seconds to explore the events revealed on the other side. The cracks will start to disappear quickly, while the color of the painting returns to its original state.

Figure 3. VitRails, detail; view from the VR display, Laval RectoVRso Gallery, 2018 © ©Respect Copyright.
Figure 4. VitRails interactive installation, Laval RectoVRso Gallery, 2018 © Respect Copyright.

Archaeological evidence brings to light how our ancestors, since the hunter-gatherer period have been using different techniques involving materials in order to communicate a story, inventing languages, tools and techniques to pass on information. The Caves of Hands in Argentina (Figure 5), is named after the findings of mural stencils dating back to 7300 BC.

Cave paintings are symbiotic and sympoietic artworks. Symbiotic because of the relationship of different micro-organisms (bacteria and fungi) that coproduce a process of continuous “restoration” while etching the pictures deeper into rock (3). and sympoietic because this chemical “metabolism” refills the contours and vividness of colors/traces that were left by humans thousand years ago.

Figure 5. Hands at the Cuevas de las Manos upon Río Pinturas, in Argentina. Picture taken in 2005 © Creative Commons 

VitRails installation, uses the metaphor of negative and positive human handprints/traces, taken from the Cave of Hands, to the material virtual other, entangling stories through a techno-anthropological gaze of the world via Virtual Reality technology, computer vision, real time simulation, photogrammetry and the thermochromic paint. In this case, a cybernetic machine is created, involving human, material and technology in order to entangle stories of the Anthropocene, becoming the Capitalocene or the Cthulhucene as named by D. Haraway.²⁸ In this new materialism, the gaze shifts its focus from the human, toward informational or data flux and digital landscapes, and this new paradigm is thoroughly explained by Katherin Hayles, in How we Became Posthuman: “How information lost its body, that is, how it came to be conceptualized as an entity separate from the material forms in which it is thought to be embedded.”²⁹ As Hayles writes, a critical practice that ignores materiality, or that reduces it to a narrow range of engagements, cuts itself off from the exuberant possibilities of all the unpredictable things that happen when we as embodied creatures interact with the rich physicality of the world. 

In the cybernetics era, information has lost its body not to be disembodied, but to be embodied differently, to include non-human artificial and natural environments and by redefining the boundaries of nature itself. By creating a feedback loop, this interactive behavior emerges by actively touching the thermochromatic surface, creating negative space. Using a camera, the negative space creates a virtual magical portal. A portal to another reality. Multisensorial virtuality creates the magical plausible illusion of being in this alter world, co-creating with the material and the virtual, in an endless sympoietic feedback cycle. As Norah Campbell points out many theorists have noticed a splicing of direct and tactile human perception of reality with another reality, one that is mediated and technical; producing a new reality that negotiates the individual's knowledge of the universe in diverse and complex ways.³⁰

In contrast to many immersive virtual reality artworks, which privilege the sense of vision and usually completely neglect other senses, in VitRails, the participant is invited to be in constant contact with the physical world around him in order to sense and perceive the virtual. In the virtual world, a parallel story is emerging. By touching the thermochromic surface the visitor creates traces of his palms, which allow him via computer vision to briefly explore fragments of another reality through a virtual window, allowing fragmented view to the 3D scenographic elements and interact with them in real time, recreated by the use of photogrammetry, real time simulation, and 3D design in the VR environment.

This experimental artistic research includes human computer interaction with intelligent materials: technical artifacts that act/react as if conscious agents, able to decide on an output behavior, given a specific input. This does not come as a surprise, as all cybernetic environments are controlled by object-oriented coding languages.

As Gilbert Simondon elegantly describes technical objects: "It is difficult to define technical objects by their belonging to a technical species; the species are easy to distinguish summarily, for practical use [...], but this is an illusory specificity, because no fixed structure corresponds to a defined use."³¹ This paper, aimed to explore the blurring boundaries of smart materials as interfaces, on experimental artistic practices in order to propose a symbiotic relation between intelligent objects, human and technology, to form what is called "the Artwork".

Conclusion

In this paper we introduced human-matter-machine interaction as a concept that questions the symbiotic and sympoietic relationship that emerges when experiencing interactive artworks with intelligent materials. Through HMMI, a new definition of human's relationship with matter stems from the entanglement of the participating actors that forms complex co-constitutive and evolving systems, defying cartesian dualities and attempting to capture some of the mystery and the magic of our surroundings.

These two artworks attempted to draw sensitive relationships with the material world around us through specific “becomings.” These installations were born from experimentation on possible symbiosis and sympoiesis between humans, the material world and technology, while trying to provoke unpredictable interactions that lead to more sensory relationships.

These artworks are using such techno-scientific advancements as a way to make spectators question their relationships with the physical environment. Therefore, our focus was on how we experience our physical environment now that the digitized world has altered our perception.

Questioning the role of each entity participating in the becoming of the artwork, whether human or not, the notions of agency and emergence are recurrent in our theoretical framework. When studying interactive artworks, especially through the perspective of experience, their emergence can be considered as the entanglement of humans, materials and technology. The situated action inside this enactive whole gives rise to the artistic interface as an open materiality, challenging our relationship with our surroundings.

Acknowledgements

Research for this paper has been supported by AIAC, INREV research group, Paris 8 University and EnsadLab Spatial Media research group, PSL University.


(1) Worlding’ refers to the co-operative way of ‘world-making” according to Donna Haraway.

(2) Liquid Matters, interactive installation by continuum collective, as presented at RectoVRso Gallery in Laval, during Laval Virtual Summit.

(3) Mihnea Mircan, Introduction-Allegory of the Cave Painting, Mousse Publishing, Milan, 2015

References

1 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, N.C., London, Duke University Press, 2007.

2 Bruno Latour, Reassembling the SociaL: An Introduction to Actor-Network-Theory, Oxford, OUP Oxford, 2005, 2.

3 Bruno Latour, Politiques de La Nature : Comment Faire Entrer Les Sciences En Démocratie, Verlag, Paris: Découverte, 2004, 66–67.

Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press, 2010, 8.

5 Jane Bennett, Vibrant Matter: A Political Ecology of Things, 2.

6 Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham (N.C.), London: Duke University Press, 2016.

7 Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene, 58.

8 Rosi Braidotti, “Discontinuous Becomings. Deleuze on the Becoming-Woman of Philosophy,” Journal of the British Society for Phenomenology 24, no. 1, January 1993, 44.

9 Stephen Abblitt, “Composite Lives: Making-with Our Multispecies Kin (Imagine!),” A/B: Auto/Biography Studies 34, no. 3, September 2, 2019, p.507–18, doi:.

10 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning

11 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, 142.

12 Gilbert Simondon, L’individu et Sa Genèse Physico-Biologique, Editions Jérôme Millon, 1995.

13 Christine Browaeys, La Matérialité À l’Ère Digitale : L’humain Connecté À La Matière, Saint-Martin-D’hères, Presses Universitaires De Grenoble, 2019, 13.

14 Christine Browaeys, La Matérialité À l’Ère Digitale : L’humain Connecté À La Matière, 104.

15 Nathaniel Stern, Interactive Art and Embodiment, Gylphi Limited, 2013.

16  Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, 84

17 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, 37.

18 Phaedra Shanbaum, The Digital Interface and New Media Art Installations, Routledge, 2019, 44.

19 ibid

20 Kate Forbes-Pitt, The Assumption of Agency Theory, Routledge, 2011, 2.

21 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, 235.

22 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, 213.

23 Phaedra Shanbaum, Media Art Installations, Routledge, 2019, 59.

24 Nathaniel Stern, Interactive Art and Embodiment, 62.

25 Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, Cambridge: Cambridge Univ. Pr, 2009, 280.

26 Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, 263. 

27 Jane Bennett, The Enchantment of Modern Life : Attachments, Crossings, and Ethics.

28 Donna Haraway, “Anthropocene, Capitalocene, Plantationocene, Chthulucene: Making Kin,” Environmental Humanities 6, no. 1, 2015, 159–65.

29 N Katherine Hayles, How We Became Posthuman Virtual Bodies in Cybernetics, Literature, and Informatics, Chicago: The University Of Chicago Press, 2008, 17.

30 Norah Campbell, “The Technological Gaze in Advertising,” Irish Marketing Review 19, January 2008.

31 Gilbert Simondon, Cécile Malaspina, John Rogove, On the Mode of Existence of Technical Objects, Minneapolis, Mn: Univocal Publishing, Minneapolis, Minn, 2017, 11.

Bibliography

Stephen Abblitt, “Composite Lives: Making-with Our Multispecies Kin (Imagine!).” A/B: Auto/Biography Studies 34, no. 3, September 2, 2019, p.507–18, doi:10.1080/08989575.2019.1664132.
Autor: Bruno Latour, Politiques de La Nature : Comment Faire Entrer Les Sciences En Démocratie, Verlag, Paris, Découverte, 2004.

Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, Durham, N.C., London, Duke University Press, 2007.

Karen Barad, “Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter.” Signs: Journal of Women in Culture and Society 28, no. 3, March 2003, 801–31, doi:10.1086/345321.

Jane Bennett, The Enchantment of Modern Life: Attachments, Crossings, and Ethics. Princeton, N.J., Princeton University Press, 2001.

Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press, 2010.

Rosi Braidotti, “Discontinuous Becomings. Deleuze on the Becoming-Woman of Philosophy”, Journal of the British Society for Phenomenology 24, no. 1, January 1993, 44. doi:10.1080/00071773.1993.11644270.

Christine Browaeys, La Matérialité À l’Ère Digitale : L’humain Connecté À La Matière, Saint-Martin-D’hères: Presses Universitaires De Grenoble, 2019.

Norah Campbell, “The Technological Gaze in Advertising", Irish Marketing Review 19, January 2008.

Monica J. Casper, “Reframing and Grounding Nonhuman Agency.” American Behavioral Scientist 37, no. 6, May 1994, 839–56. doi:10.1177/0002764294037006009.

Kate Forbes-Pitt, The Assumption of Agency Theory, Routledge, 2011.

Donna Haraway, “Anthropocene, Capitalocene, Plantationocene, Chthulucene: Making Kin”, Environmental Humanities 6, no. 1, 2015, 159–65. doi:10.1215/22011919-3615934.

Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene. Durham (N.C.), London, Duke University Press, 2016.

Bruno Latour , Reassembling the Social: An Introduction to Actor-Network-Theory, Oxford, OUP Oxford, 2005.

Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, Cambridge, Cambridge Univ. Pr, 2009.

N Katherine Hayles, How We Became Posthuman Virtual Bodies in Cybernetics, Literature, and Informatics, Chicago: The University Of Chicago Press, 2008.

Phaedra Shanbaum, The Digital Interface and New Media Art Installations, Routledge, 2019.

Gilbert Simondon, L’individu et Sa Genèse Physico-Biologique, Editions Jérôme Millon, 1995.

Gilbert Simondon, Cécile Malaspina, John Rogove, On the Mode of Existence of Technical Objects, Minneapolis, Mn: Univocal Publishing, Minneapolis, Minn, 2017.

Nathaniel Stern, Interactive Art and Embodiment, Gylphi Limited, 2013. 

Thinking with Tides: Engaging with Embodied Technical Processes within the Tidal Ranges of the Thames Estuary

Gerolamo Gnecchi

Goldsmiths University of London London, UK ggnec002@gold.ac.uk

Abstract

The following paper reports on findings from doctoral research in Arts and Computational Technology. The research is about tidal environments. More specifically, it engages with the tidal ranges of the Thames Estuary and explores what thinking with tides might signify. In the face of uncertain futures—sea-level rise, increase in flooding, pollution—and through art practice, it reflects on how combining different ways of knowing the Thames Estuary could reintroduce questions of locality, situatedness and diversity to make the estuary differently available. The paper is divided into four sections. The first section summarises the kind of tidal processes that unfold in the estuary. The second section introduces the methodology, Practising in the Wild, and explains how Embodied Technical Processes (ETP) are necessary to engage with the estuary. The third section presents an inventory of different technical objects that are used to think with the estuary and its tides. Finally, in the last section, two art projects—by composer John Eacott and art collective YoHa—are discussed. Both projects were situated within the tidal ranges of the Thames and applied ETP. It concludes with reflections on future studies that this research is currently developing.

Keywords

Embodied Technical Processes, Thames Estuary, intertidal zone, tides, technical objects, art practice. 

DOI

10.69564/ISEA2023-60-full-Gnecchi-Thinking-with-Tides

Figure 1. Rowing during the ebbing tides along the intertidal zone, Leigh-On-Sea ©Emma Jaay.

Introduction —
A More-Than-Human-Soup

Tides are always on the move. They have their own spatial and temporal growth. They vary from place to place. Every tidal environment sounds, smells, looks, and feels different. As you read these words, tides are changing the way life unfolds, intersecting with organisms, ecologies, cultures, wavering space, time, and materiality. Their openness, their complexity, their diversity, makes tidal environments both knowable and unknowable, predictable and unpredictable.

This paper reports some of the findings and methods, developed along the tidal ranges of the Thames Estuary. The Thames Estuary can be seen as a complex more than human soup of colonial histories, empires, trade, language, cosmic forces, the Anthropocene.¹, ², ³ It is challenging, more so impossible, to create a totalising narrative of this stretch of mud and water. Thus, what this research can offer, are different ways of knowing the estuary.

The estuary has multiple facets, which include material, ecological, and cultural roots, but it is also the plumbing of a global city like London.⁴ It is both wild and industrial. Over the 20th century, the estuary has been deepened, widened, diverted, drained, channelised, filled in and built over, and dammed. It is an emblem of the Anthropocene that has been colonised, commoditised, artificially engineered, bombed, trashed, and recently is being rewilded. Yet the estuary exists before and beyond its codification as a place for trade, empire, consumption. A port is a space built for human use, representing distribution, labor, marine routes, networks, but an estuary, in the most basic geological sense, is about sediments, algae, silt, bodies, species, salt marshes. Thus, the estuary cannot be defined solely by natural characteristics, nor is it merely a product of human use and imagination.⁵ It is a transitional space, with different temporalities and flows that oscillate between the river Thames and the North Sea, sea and land, fresh and salt water. But most importantly, a space dictacted by its tides.

To engage, explore, and know the estuary, this doctoral research proposes to learn how to think with the more-than-human: the forces of waters, the winds, the ever-changing seabed, as well as many other beings like boats, buoys, tide charts, algae, barnacles.

Aims and Objectives

The purpose of this paper is to introduce the methodology of this doctoral research named 'Practising in the Wild'. Practising in the Wild is imbued with Embodied Technical Processes (ETP). Processes that are tied to different ways of knowing the estuary, both embodied and technical, experiential and scientific. They aim to establish an active engagement with the estuary, combining different ways of knowing—embodied, technical, situated, local to name a few—and provide methods for artistic and design practices that deal with tidal environments. Furthermore, it attempts to demonstrate the structural coupling that occurs between objects, bodies, tides, the atmosphere and avoid any hard distinctions between cultural, ecological, social, and technical ways of knowing the estuary. The paper then proposes an inventory of technical objects used to explore the Thames Estuary, such as boats, tide gauges, and tidal predictions. It then looks into two art interventions in the estuary that apply ETP, one by composer and sailor John Eacott and the other by art collective YoHa. However, before delving into the methodology, a summary of the kind of tidal processes that expand and retract in the estuary should be considered.

The Thames Estuary
and its Tides

The Thames Estuary experiences two distinct tidal patterns, one from the North Sea and the other from the English Channel. Generally speaking, estuaries are composed of a stew of water and mud where freshwater and seawater meet. The infusion depends on various factors, such as the state of the river and sea at a given time and the repetitive stirring of its tides. This unfolds into spaces with no clear boundaries making estuaries both familiar and unfamiliar, where certainty is flawed by uncertainty, and constancy is affected by fluidity. Its tides result in monthly, seasonal, and yearly variations, which correspond to the complex ‘pas de trois’ between the sun, moon, and earth.⁶, ⁷ The height and timing of tides are determined by various factors, including seabed topography, atmospheric pressure, wind patterns, and wave dynamics, to name a few. Despite efforts to predict tides accurately, their intricate dance defies their predictability, which makes them both predictable and unpredictable.

Some Basic Tidal Knowledge

The estuary’s tidal processes interconnect land and sea, salt and fresh water, sewage and silt, wrecks and barnacles, creating a complex range of eco-social assemblages.⁷ They have a significant impact on the estuary's morphology, co-creating and transforming a variety of ecosystems. The seabed in this liminal space is in a constant state of flux, with channels and banks shifting daily, monthly, and seasonally. There is no finality or fixed reality when it comes to the intertidal zone. For instance, as saltwater flows up the estuary with each incoming tide, fresh water from the river's catchment carries silt and other materials downstream. As the tides recede, coastal habitats, such as salt marshes, are formed. These low-lying habitats are characterised by deep, irregular creeks and soft, fine silt. They are home to a wide variety of wildlife, including sea molluscs, wildfowl, and a vital fish nursery.

Figure 2. Photo taken during fieldwork from the Belton Craft Club in Leigh on Sea, illustrating the two different tidal states. Left image shows the intertidal zone during low tide, while the right image during high tide.

Their cycle is known to be semidiurnal, which means tides rise and fall roughly twice in twenty-four hours, revealing two different environments within the intertidal zone (figure 2). The waters expand and retract. Their timings do not synchronise with the day-night cycle in any simple way.⁷ During high tide the intertidal zone covers itself with water. During low tide vast ecologies of mud are revealed. The tidal range is larger than 4 meters. These kinds of tides are known as macro tides. In London Bridge, the tidal range can reach up to seven meters. Along the estuary, it varies between five and six meters. Moreover, the timings of high and low water occur at different moments between London Bridge and the estuary. At the estuary, high water precedes that of London Bridge by approximately one hour.

Tides also interplay with floods. The estuary has the highest proportion of its population within Zone 2 flood-plains.⁸ Zone 2 flood areas represent the extent of an extreme flood event—the chance of flooding from rivers or the sea in 1000 years. More than half of the sites of ecological interest, nature heritage sites, areas of special protection, and wildlife trusts correspond to former landfills, sewage, and derelict wastelands, which lie within flood Zone 3 areas. Zone 3 represents the threshold for defining under-risk areas by sea level rise and coastal erosion. Approximately 1,200 hectares of habitats, such as mudflats and salt marshes, could be lost due to flooding.⁹

Tidal flooding can have significant destructive impacts when a combination of spring tide, strong gale-force winds, and low atmospheric pressure coincide.¹⁰ The consequences of such events can be exemplified by the 1953 storm surge that hit the Thames Estuary shores, causing the loss of roughly 300 lives and the displacement of 30,000 individuals from their homes. The East Coast of the UK was severely affected, and communities were left isolated. The floodwaters submerged 65,000 hectares of farmland and inundated 240,000 houses. As a result of this catastrophic event, extensive research was conducted on coastal defences, storm surges, and policies, leading to the implementation of weather forecasting, warning services, and a network of tide gauges across the UK. The Thames Estuary remains vulnerable to future storms and sea-level rise.

Flood risk management often involves constructing higher seawalls, dikes, barriers, levees, as well as quantifying, modelling, and datafying the environment. This can leave out forms of knowledge that are subjective, local, situated, and ecological.¹¹ Current discourse on environmental management asserts that traditional modernist or engineering approaches are becoming obsolete.¹², ¹³, ¹⁴ Thus, in the face of uncertain futures and environmental challenges, how to reintroduce questions of locality, situatedness, and diversity into the discourse of the estuary? Moreover, how could this reintroduction contribute to the discourses of designing, planning, managing and dwelling in tidal environments?

Methodology —
Practising in the Wild

Practising in the Wild is driven by a strong sense of environmental responsibility and ecological urgency. It problematises and challenges forms of scientific reductionism, exploitation, and appropriation of tidal environments that often go unchallenged. It involves the creation of knowledge through hands-on practice, outdoor exploration, and ethnographic studies. It enables one to fall into the estuary’s flows and to attune oneself to the rhythms and patterns of its tides.

This methodology combines local, situated, sensorial, embodied, and technical knowledge in the attempt to challenge purely discursive readings of the estuary. This can create an opportunity for people to engage in a dialogue that questions prevailing methods and enables a more nuanced understanding of the complexities of the estuary. When engaging in activities within the intertidal zone, seafarers learn how to think with the forces of the waters, winds, seabed, tide charts, tidal predictions, sonar baths. Their approach allows to establish a close relationship with technical objects and our own embodied experiences. In other words, Practising in the Wild is imbued with Embodied Technical Processes (ETP). Processes that combine both technical and embodied ways of knowing and thinking with the estuary.

Embodied Technical Processes

Thinking through ETP allows us to comprehend the structural coupling of humans in collaboration with different bodies. Embodied, here, focuses on building knowledge about the estuary through bodily experiences. The body acts as an instrument to know, bypassing rational thoughts through bodily senses and feelings.¹⁵ It provides a sensorial understanding of the estuary, opening the potential for affective transmission; from the body to environment and environment to body.¹⁶ For instance, experienced seafarers can feel the wind changing through the hair of their skin—an embodied response to a changing environment.

Technical refers to the knowledge one develops in collaboration with tools, objects, instruments, devices. For instance, when planning a journey and before starting to navigate, knowing the tide’s height and timings is necessary when seafaring the estuary. This information can be obtained from a nautical almanac, an app or a tide website that predicts the times and heights of high and low water daily. Alternatively, a tidal curve specific to each port can be used to manually calculate the tidal range at any given time. The tidal curve is a wave graph that can be found in the nautical almanac (see figure 4).

When sailing with experienced seafarers, one can gain further insight about ETP. People often discuss the relationships between the wind, water, boats, tools, devices. For instance, nowadays, many seafarers use GPS or radar tracking equipment, check tide charts and weather forecasts before heading out, and combine these different readings with their personal experiences to sail the estuary. It involves a balance between their knowledge and the use of technical objects, blurring the boundaries between humans, objects, and the estuary.

The acquisition of knowledge within the Thames Estuary, thus, relies on both technical and embodied cognitive processes. In other words, a thorough understanding of tidal environments is contingent on technology, experience, and embodiment.

The following section introduces an inventory of technical objects that are used to explore and think with the Thames Estuary. The aim is to understand the nature of the kind of knowledge that these objects embody and how in turn, they help us to build different ways of knowing the estuary.

Inventory of Technical Objects

Figure 3. One of the many wrecks that one can encounter when exploring the intertidal zone.

When walking along the Thames Estuary, one can notice the proliferation of technical objects along its shores and shoal waters. From tide gauges to wrecks, from jetties to cargo ships, the Thames Estuary is populated with technical beings. They act as mediators between the estuary’s ever-changing rhythms and patterns and us. Some encapsulate the estuary and its tides inside fibre-optic cables, informing us of the highs and lows of the waters, while others enable us to actively explore the estuary. Their different cultures embody different ways of knowing, allowing us to think with this soup of mud, water, salt marshes, pollution, sewage.

Boats

Within the context of seafaring, boats can offer thought-provoking insights into the ways people engage with the estuary. They are tied to the movement of the body in space, and the forces of the waters, winds, and seabed, creating constant feedback between the estuary and the body. They become a lens for a dynamic and sensorial experience of the estuary. For instance, sailing necessitates a unique set of skills and knowledge that are not typically required on land. They allow us to become amphibious. The movement of a sailboat is dependent on various factors, such as the boat's features, the forces of the waters and winds, and how they interact with each other. In other words, the movement is influenced by a complex interplay of external forces and internal intentions that dictate directionality and affect local action. Seafarers navigate by "feeling" their way towards their destination, continually adjusting their movements based on the flow of waves, wind, current, stars, and adjusting their boats.¹⁷ The knowledge gained through seafaring is often implicit and intuitive, making it challenging to articulate or communicate with others. While discursive readings can provide a foundation, it is only through experience that one can gradually gain the skills required for exploing the shallow waters of the estuary.

Tillers The analysis of the potential of a seemingly mundane object, such as a boat's tiller, can provide valuable insight into the fundamental nature of seafaring.¹⁸ By feeling the tiller, a seafarer can directly sense the impact of environmental factors on the boat and perceive fluctuations in the wind and water. According to Thomas Gladwin (1964), Micronesian sailors are constantly aware of "motion, sound, the feel of the wind, wave patterns, stars, etc." throughout their voyage. They utilise past embodied experiences to make adjustments, such as slightly changing the positioning of the tiller or instructing the crew to adjust the sails.¹⁹ The tiller thus serves as an extension of the body, allowing the seafarer to seamlessly flow with the currents and manage the boat’s movements. Additionally, the tiller plays a crucial role in regulating and organising the rest of the boat, further highlighting its significance in the context of seafaring.

Data Sytsems

Figure 4. On the left side the tide gauge that is in Shivering Sand [https://www.resinextrad.com/en/tide-gauge-platform-for-the- thames-navigation/], and on the right, a tidal curve that reads the height of the waters at a given time in a specific location. The tidal curve is used as a real-time translation of the data recorded from the tide gauge [https://tidepredictions.pla.co.uk/].

Data systems are seen to provide a continuous stream of data on the Thames Estuary, including real-time tidal information, prediction tables, tidal currents, and charts. These systems play a significant role in various activities within the Thames Estuary. They are critical in monitoring sea level rise, flood risk management, navigational safety, and sea-based commercial logistics.²⁰ A whole technological infrastructure is situated upon the estuary, composed of tide gauges, ISO maps, tidal predictions, tidal apps, to name a few. These technologies function as an extension of our memory, enabling us to keep track of and compute tidal processes, to observe and analyse their cycles. They aid people in delineating the ranges, currents, and future states of the estuary.

Tide Gauges embody knowledge about the ranges of the tides. They are an important invention in the field of oceanographic measurement, first created by civil engineer Henry Palmer in 1831 for the London Dock Company. These devices were designed to automatically record water levels in a specific location, utilising a float that sat on top of the water in a tube and a pen that recorded the variations through a wave graph on paper attached to a rotating drum. With the advent of self-recording tide gauges, tidal measurements were transformed from a daily sequence to a continuous process over time, mediated by machines rather than human observation. ²¹ In other words, the introduction of tide gauges revolutionised the way in which tidal data was collected, enabling autonomous recording and representation of tides around the clock. Prior to their invention, the tidal range was measured using graduated scales and staff, which lacked the same level of accuracy and precision as tide gauges. Over time, numerous tide gauges were developed, each uniquely designed to suit its location and varying in mechanism and design.²², ²³

Tide gauges are still today an essential part of the technological infrastructure of the Thames Estuary. They are spaces that are generally situated on piers, jetties, or buoys. They are equipped with digital sensors, acoustic-sounding tubes, data storage devices, and transmitting equipment. Tides are then captured approximately every six minutes and squeezed into a half-inch wide-sounding tube that measures the distance between the gauge and the water's surface, transforming the flows into indices, numbers, data for people to monitor, analyse, or simply check the range of the waters in real-time.

Furthermore, tide gauges provide essential data for predicting tides and monitoring port operations. Knowing the precise tide time and range is fundamental to life in the estuary. Scientists use this information to research ocean currents and climate change, engineers use it to design coastal infrastructure and simulations. They are used for flood risk management, environmental control, and commercial logistics. In the past, tide gauges were used to serve military and colonising purposes to control the empire. The flow of the empire required “as its lubricant a science of the sea, essential for any overseas expansion or military campaign.”²⁴ Today, they remain vital for shipping and naval operations, with significant ports like Dover that rely on tidal knowledge. 

Tidal Predictions William Thomson, popularly known as Lord Kelvin, was a renowned mathematician and engineer who made a significant contribution to tidal knowledge.²¹ One of his key innovations was the introduction of harmonic analysis. This method effectively simplifies complex tidal motions into a series of distinct constituents, each of which corresponds to a different lunar or solar frequency.²¹ Using this approach, Kelvin was able to develop the tide prediction machine, an ingenious device designed to accurately forecast tides in specific locations. The precise prediction of high and low water times was vital to the livelihoods of coastal communities and ports. As such, the accuracy of tide predictions increased propor- tionally with the number of constituents considered.

The operation of this machine was based on a simple harmonic motion characterised by appropriate speed, amplitude, and phase, which acts on a series of pulleys that represent the various tidal constituents. The spacing and diameters of these pulleys enable a flexible wire or metal tape to be threaded vertically from one pulley to the next. The summed motion is then transferred to a pen, which plots a timed trace on a moving chart at a reduced scale.²¹ The machine is capable of handling up to ten tidal constituents, thereby enabling the simulation of the time and heights of tides as a sum of sinusoidal motions of several individual tidal constituents at any location and moment in time.²⁵

Today tidal predictions are done on high-speed computers, which can compute more constituents and take seconds to calculate predictions. However, the techniques do not differ that much from the earlier methods.²⁶

Figure 5. Lord Kelvin Predictor No.3 manufactured in 1881. Number of constituents 16.

Overall, the study of tides has been mostly the subject of scientific inquiry across various cultures, with disciplines such as mathematics and technological advancements contributing to a universal understanding of tidal environments, leaving out subjectivities. In the 20th century, the advent of digital computers paved the way for precise solutions to tidal equations across the entire ocean, incorporating critical factors like the seabed, land distribution, currents, and deformations of the Earth. ²¹ Advancements in marine instrumentation, tide gauges, and tidal predictions have further enabled calculations and observations of tides along coastlines worldwide. The integration of soundings, ISO maps/baths, bathymetry, predictions on nautical charts, and satellite altimeter data in the early 90s has resulted in the establishment of a Foucauldian-type formation of truth machines. Systems that systematically produce truths that hold validity within a particular discourse and provide answers to explicit questions that shape language and values regarding tidal environments.

This research proposes to triangulate knowledge refuting the idea of singular ways of knowing the estuary. In other words, tidal environments cannot only be understood by purely scientific methods.²⁷

Art Projects and ETP

In this section, we will look into two distinct artworks, one by John Eacott and the other by YoHa. Both artists, in different ways, employ ETP to create their work, raising the question of how art practice can bring together different ways of knowing the estuary. John Eacott uses sensors to transform live tidal data into a musical performance. Art collective YoHa harnesses local, embodied and technical knowledge of the estuary to construct an anti-monument.

Flood Tide

Composer John Eacott created a unique sonification of the Thames Estuary in 2008. The composition was generated by using an Acoustic Doppler Current Profiler (ADCP) to read the speed of the flood tides in the Tidal Thames. During the performance, the ADCP was placed in the water to stream real-time data, which was then translated into an ever-changing musical score. An ensemble of orchestral instruments read live tidal stream data from computer screens, producing a musical experience that lasted approximately 6 hours, the duration of a complete flood tide. This performance seamlessly engages with the Thames, integrating music and data by combining cosmic forces, sensors, the tides, sound, musicians, and instruments.

In a group discussion analysed by the artist, insights were shared about the music, the data's importance, the performance's visual aspects, the experience of both performers and the audience, and the relevance and relation with the Thames.²⁸ The music is hypnotic, meditative, and slow. It is repetitive but varies. The performance raises awareness of the ever-changing rhythms and patterns of the tides and creates an unusual representation of the Thames tides. The experience is created through an interaction between the sensor, the tides, the musicians, and the spectators. The ADCP acts as a mediator between the tides and the music, combining an artistic and scientific approach with tides and allowing multiple narratives to emerge: tides as numbers, tides as flows, tides as sound, tides as experience, tides as structure, and tides as the gravitational pulls of heavenly forces. It acknowledges the complexities of the Thames.

Overall, the artwork provides an immersive experience that captures the intricacies and subtleties of the Thames through the sonification of the environment. Incorporating sound allows for a deeper connection with tidal environments, creating a multi-sensory experience that goes beyond rational thought.¹⁶ However, I am curious how the music would change if this artwork were performed in different tidal environments. How does the data of the ADCP change and, in turn, vary the score from place to place? More so, how could the performance reflect the diversity of each place?

Graveyard of Lost Species

Figure 6. Low tide during the making of the installation ©YoHa.

The artwork, ' Graveyard of Lost Species,' by YoHa in collaboration with Critical Art Ensemble (2015), is an antimonument. It is composed of a wreck. The intertidal zone is home to many boats buried due to local activity. The wreck, which was once abandoned, forgotten, and let to fade, is transformed into a site of cultural production and served as a powerful reminder of the consequences of capitalism on the environment. It aims to raise awareness of environmental degradation and the loss of biodiversity along the estuary. The artwork is designed to fade and perish over time, highlighting the urgent need for ecological conservation and preservation.

The process involved the transportation of the wreck from the estuary mudflats to the shore. This was a challenging experience for those who participated since they had to deal with the unpredictability of the estuary. It required people with local, technical and embodied knowledge of the ever-changing rhythms and patterns of the estuary, allowing them to build a sense of place.²⁹

This artwork is a significant contribution to exploring locality, embodiment, and materiality. It employs the idea of ‘making, thinking, and being with’, which involves collaborating with local materials, bodies, objects, and especially tides and mud. Furthermore, the project helps to show how people situate themselves within the estuary, engaging with it in alternative ways and providing valuable insights into participatory forms of making. An approach that this doctoral research aims to form.

Conclusion & Next Steps

One of the objectives of this doctoral research is to create various methods for artistic and design practices that engage with tidal environments and reintroduce discussions around locality, situatedness and diversity within this stretch of mud and water. The paper introduces a methodology inspired by people who work at sea, such as seafarers, fishers, sailors. By merging embodied, subjective, computational, and technical ways of knowing the estuary, this approach attempts to serve as the foundation for a more holistic understanding. In other words, it proposes to triangulate knowledge, promoting diverse perspectives to defy any rigid distinctions about the estuary.

The methodology proposed in the paper includes an inventory of technical objects that are necessary when engaging and exploring tidal environments. These objects allow us to think with the estuary and, more broadly, with the more-than-human world. While the paper provides a list of technical objects, it acknowledges that it is incomplete. Other objects, such as buoys, anchors, and ropes, in different ways, should also be considered to enrich the current state of the inventory.

Finally, it looks into two artworks that propose, in different ways, situated interventions within the Thames. They help to understand the importance of ETP further. Adjacent to this paper, future studies are currently being developed: (1) interviews with artists, seafarers, and river workers, (2) personal field notes, images, experiences, the development of a sort of ‘cahier de voyage’ (3) enriching the inventory (4) and finally the making and development of in-situ installations.

References

1 Sarah Whatmore, "Materialist Returns: Practising Cultural Geography in and for a More-than-Human World", Cultural Geographies 13, no. 4, 1 October 2006, p.600–609, https://doi.org/10.1191/1474474006cgj377oa.

2 Donna J. Haraway, When species meet, Vol. 3, U of Minnesota Press, 2013.

3 Rosi Braidotti, The Posthuman, Polity, 2013.

4 Graham Harwood, Wasted Museums by YoHa, The Exchange, Erith, November 9, 2022, Introduction to Exhibiton. 

5 Robert Van de Noort, North Sea Archaelogies: A maritime Biography, 10,000 BC-AD 1500, Oxford University Press, 2011. 

6 James Greig McCully, Beyond the moon: A conversational common sense guide to understanding the tides, World Scientific, 2006.

7 Jones Owain, "Lunar-Solar Rhythmpatterns: Towards the Material Cultures of Tides", Environment and Planning A 43, no. 10, 2011, 2285–2303, https://doi.org/10.1068/a4468.

8 Jane L. Fielding, "Inequalities in Exposure and Awareness of Flood Risk in England and Wales", Disasters 36, no. 3, 2012, 477–94, https://doi.org/10.1111/j.1467-7717.2011.01270.x.

9 HM Government, "UK Climate Change - Risk Assessment 2017", Open Government Licence, 2017.

10 David Pugh, Philip Woodworth, Sea-level science: understanding tides, surges, tsunamis and mean sea-level changes, Cambridge University Press, 2014.

11  Catherine D'ignazio, Lauren F. Klein, Data feminism, MIT press, 2020.

12 Britta Restemeyer, Margo van den Brink, Johan Woltjer, "Resilience Unpacked – Framing of “Uncertainty” and “Adaptability” in Long-Term Flood Risk Management Strategies for London and Rotterdam", European Planning Studies 26, n.8, 3 August 2018, 1559–79, https://doi.org/10.1080/09654313.2018.1490393.

13 Yarina Lizzie, "Your Sea Wall Won’t Save You", Places Journal, 27 March 2018, https://doi.org/10.22269/180327.

14 David Chandler, "Forum 2: The Migrant Climate: Resilience, Adaptation and the Ontopolitics of Mobility in the Anthropocene", Mobilities 14, no. 3, 4 May 2019, 381–87, https://doi.org/10.1080/17450101.2019.1609194.

15 Michael Polanyi, The Tacit Dimension, Peter Smith, 1983. 

16 Jones Owain, Louisa Fairclough, "Sounding Grief: The Severn Estuary as an Emotional Soundscape", Emotion, Space and Society 20, 1 August 2016, 98–110, https://doi.org/10.1016/j.emospa.2016.06.001.

17 Tim Ingold, The perception of the environment: essays on livelihood, dwelling and skill, Routledge, 2000.

18 B. Wattchow, A feeling for water: The body and the paddle, Journeys, 4(2), 1999, 26-32.

19 T. Gladwin, Culture and Logical Process in Explorations in Cultural Anthropology, New York, McGraw-Hill, 1964, 167-177.

20 Addina Inayatillah, Ivan D.Haigh, James H.Brand, Katy Francis, Alex Mortley, Matthew Durrant, Laura Fantuzzi, Elizabeth Palmer, Callum Miller, Peter Hogarth, "Digitising Historic Sea Level Records in the Thames Estuary, UK", 2021.

21 David Edgar Cartwright, Tides: a scientific history, Cambridge University Press, 2000.

22 Wolfgang Matthäus, "On the history of recording tide gauges", Proceedings of the Royal Society of Edinburgh, Section B: Biological Sciences 73, 1972, 26-34.

23 Philip L. Woodworth, "Advances in the Observation and Understanding of Changes in Sea Level and Tides", Annals of the New York Academy of Sciences 1516, no. 1, 2022, 48–75, https://doi.org/10.1111/nyas.14851.

24 Michael S. Reidy, Tides of History: Ocean Science and Her Majesty’s Navy, Chicago, US, University of Chicago Press, 2008. 

25 Philip L. Woodworth, "An Inventory of Tide Prediction Machines", Monograph, National Oceanography Centre, 2016. https://eprints.soton.ac.uk/394662/.

26 Bruce B. Parker, Tidal Analysis and Prediction, NOAA, NOS Center for Operational Oceanographic Products and Services, 2007. https://doi.org/10.25607/OBP-191.

27 Graham Harwood, "Wrecked on the Intertidal Zone (2015)", YoHa Art Collective Website, last accessed December 6, 2022, http://yoha.co.uk/wrecked

28 John Eacott, "Flood Tide: Sonification as Musical Performance—an Audience Perspective", AI & SOCIETY 27, no. 2, 1 May 2012, 189–95, https://doi.org/10.1007/s00146-011-0338-2.

29 Brian Wattchow, Mike Brown, A pedagogy of place: Outdoor education for a changing world, Monash University Publishing, 2011. 

Chthulucene Hekateris

Charlotte Gould

University of Brighton, United Kingdom c.gould@brighton.ac.uk

Abstract

Isabel Stenger warns that we are facing the “intrusion of Gaia” where we have caused significant biogeochemical disruption “capable of threatening our modes of thinking and of living for good.”¹ Through my practice-based research I speculate on a possible future to prompt action to trigger change in how we live, our patterns of consumption, the way we see ourselves in relation to our environment and our respect for and interactions with nature for a sustainable future. Amitav Ghosh proposes that science fiction provides an ideal opportunity to explore our relationship to the world past and present to imagine the impacts that living on our planet today will make on tomorrow.² Through my research I develop narratives based on speculative imaginings of the future, considering current scientific research, advances in digital technology and environmental factors, to imagine future evolutionary change that will take place if we continue on our current trajectory of global warming. I speculate on the interactions and interconnections, the transformation of complex systems and organisms leading to new patterns of cellular composites of material and virtual worlds, where biotic and unbiotic beings inhabit a posthuman fusion of humans and machines.³

Keywords

Anthropocene, Chthulucene, Symbiosis, speculative imaginings, Science-Fiction, Extended Reality.

DOI

10.69564/ISEA2023-61-full-Gould-Chthulucene-Hekateris

Introduction

“Chthulucene Hekateris” imagines evolutionary transformations, far into the future and the symbiotic metamorphosis that takes place to form hybrid creatures and mutual dependencies. Through this practice-based research project I explore environments of dynamic exchange and metastable equilibrium, interrelational sites of spatial and temporal encounter and fragile interdependencies. I speculate on transformations to imagine a possible future, where carbon and silicon merge and evolve in response to the changing environment. Writer O. B. Hardison imagines the evolution of the posthuman, the fusing of carbon, man and silicon tools using a metaphor of "the caterpillar and the iridescent, winged creature that the caterpillar unconsciously prepares to become." ⁴  Through this practice-based project I gaze through time to speculate on the metamorphosis that will take place through the digital chrysalis. The past is referenced to consider the future, looking at environmental factors to imagine the creatures who will inhabit the Earth over time. These transhuman, hybrid creatures, inhabit a non-linear narrative for an imagined future world. The characters are developed initially through reference to the real, juxtaposed against the imagined, in this way dialoguing with reality. Through this process of speculation, I imagine new fusions and mutations which may emerge through evolution. I explore the potential development of posthuman beings who exchange characteristics, both from other life forms and from digital code. I consider the composites, traumas, triggers, and catalysts to change and the environment that manifests this transformation.

Time Travel

This practice-based project invites participants to peek through the lens of time, to imagine our future metamorphosis. “Chthulucene Hekateris” is an immersive installation which borrows from early mythologies, to imagine a future in which the relationship of the Earth’s inhabitants living and otherwise work in harmony with the natural world. This is synonymous with Rainer Maria Rilke’s proposition in his letter to a young poet that “...the future enters into us long before it happens.” ⁵ I use science fiction narrative to inform my speculative approach, inspired by writers such as Ursula K. Le Guin, who in her series of novels set in the Hainish universe, imagines a future world impacted by biomedical intervention and evolution. Through speculative fiction she uses metaphor to consider issues such as gender.⁶ Another science fiction inspiration comes from Kurt Vonnegut’s dystopian tale, “Galapagos” which imagines the outcomes of a natural evolutionary process. ⁷ Following a global financial crisis and plague on Earth, all but a few humans remain on one of the Galapagos islands. In recognition of Charles Darwin, Vonnegut imagines the evolution of their descendants as hybrid beings who are half human and half sealion. The hands have evolved into “nubbins,” no longer able to manipulate and shape the external world with dexterity, and thereby losing its primary position in relation to other creatures. This allegorical tale of anthropomorphic hybrid human retro engineering looks to the future to comment on contemporary society and culture.

Through “Chthulucene Hekateris” we can similarly imagine the hybrid creatures, or chimeras that may evolve to survive on a damaged planet, we time travel from the past to the distant future, using a process of speculation. This operates in reverse to the investigative approach taken by palaeontologists when reimagining life on Earth in the distant past. In the television production “Prehistoric Planet” presented by David Attenborough ⁸ the latest scientific research was probed to recreate the habits and interconnections of life on Earth 66 million years ago. Evidence from fossils and the context in which they were found helped to establish dinosaurs' patterns of sociability, adversaries, and food sources. Phylogenetic bracketing further determined similar behaviours attributed to the dinosaurs, across their line of descendants. ⁹  The resulting mixed reality film was a collaboration between palaeontologist Dr Darren Naish and filmmaker Mike Gunton from the BBC’s natural film unit and CGI expert Jon Favreau. “Chthulucene Hekateris” also references ecology and culture, past and present as well as bioscience to imagine the future as a warning of the risks of the damaging effects of climate change.

The Anthropocene

Isabel Stengers warns of climate catastrophe if significant action is not taken to reverse climate change. ¹⁰ We have reached a moment in history where human impacts on the planet are inflicting long term and potentially irreversible damage, an era denoted by Paul Crutzen as the Anthropocene. Donna Haraway proposes that we have now entered a new phase of the Chthulucene which decenters the human as the primary species in favour of multispecies, where survival on the damaged planet requires symbiosis and collaboration. ¹¹ The crisis of Global warming is being recognised across the world and the Intergovernmental Science Policy Platform on Biodiversity and Ecosystem Services (IPBES) report published in July 2022 provides clear evidence that humans are over exploiting the planet’s natural resources. Ecosystems have been destroyed through intensive farming methods, use of fertilisers and destruction of habitat of areas such as peatlands, which has resulted in carbon being released into the atmosphere. As a result, increased numbers of species are becoming extinct, and the 2019 State of Nature Report found one million species worldwide are at risk of extinction. In the UK of the 8,431 species reviewed 1,188 are at risk. ¹² 

Bruno Latour proposed that the pursuit of “progress” at the expense of the natural world has happened in part due to a mistaken Modernist perception that nature and culture are separate entities. ¹³  Christophe Bonneuil and Jean-Baptiste Fressoz similarly identify the Modernist era as perpetuating a “modernising unconscious” which enabled us to ignore the ecological cost of continuous growth at the

expense of the health of the planet. This was reflected in politics, economics and culture of the Western North becoming progressively separated from nature. Economic theories focused on the consumer as separate to and at the expense of nature. Through globalization prices detached from local markets and GDP created a commodified economy. ¹⁴ Bonneuil and Fressoz argue that the Northern countries of the world industrialised and attained post war affluence at the expense of the Southern hemisphere through an ‘unequal ecological exchange’.

“Western countries build their own growth on a gigantic draining of minerals and renewable resources from the rest of the non-communist world, emptying it of its high-quality energy and minerals.” ¹⁵

This has led to vast global inequalities and “the geological derailment of the planet in the Anthropocene”. ¹⁶ They assert that investment in fossil fuels continued at the expense of available greener alternatives such as wind or solar, despite warnings about the dangers. Part of this blind spot towards environmental damage was comprised through a belief in the endless resource of nature, and in its regenerative qualities. In their book “Gaia”, James Love-lock and Lynne Margulis proposed that the Earth was a self-healing biosystem. ¹⁷ Isabel Stengers highlights the importance of “Gaia” was in identifying the ecology of the planet as a “being”, as an entity rather than as separate parts, but notes that the authors perhaps initially over-emphasised Gaia’s self-healing powers. ¹⁸ According to Dipesh Chakrabarty global warming did not become of wide public interest until the 2000s as “governments, beholden to special interests and wary of political costs, would not listen.” ¹⁹ Timothy Morton proposes that ecology (global warming, evolution, extinction) is a “Hyper Object”, too big for empirical observation, and this has further inhibited action on climate change. Ecological phenomena are impacted by billions of combined actions, from exhaust emissions to fertilisers and plastics released into water systems. As individual actions they appear insignificant but on mass they create a crisis. ²⁰

Habitats and Eco systems have been damaged through intensive farming and this has produced monocultures, leading to deterioration of both soils and crops through over fertilization, resulting in a heightened propensity for infestation and disease. Suzanne Simard’s research into the biosystem of the forests of British Columbia identified the corrosive impacts of intensive farming following clear cutting of ancient forests for the planting of pines. Conversely, Simard found a network of support in the ancient forests developed between mother trees and saplings but also between multi-species deep under the ground. Mycorrhiza fungus formed a life-or-death relationship with multiple tree species, where “Without entering into this partnership neither the fungus nor the plant could survive.” ²¹ She also identified a symbiotic relationship between trees including Larch Cedar and Fir and mycorrhizal fungi roots in the forests of British Columbia and demonstrated that intensive forestry and agricultural monocultures were hampering the growth of newly planted fir trees which replaced ancient forests. The use of fertilisers provided easy access to nourishment of the new tree roots, which limited further growth to enable access to the mycelium deep in the soil thereby inhibiting symbiosis. Simard’s research further demonstrates a symbiosis between trees where mother trees protected their offspring passing nourishment through the roots. She also found symbiosis between different tree species, Larch, Cedar, Fir and Pine Grasses which passed on water and nutrients including nitrogen through their roots and in the soil to support differing timelines in their life cycles. Simard’s research emphasises the cooperation and reciprocity that takes place across different species in nature. She advocates collaboration over individualism and comments that “We emphasise domination and competition...We emphasise factions instead of coalitions. ²²

These findings are further corroborated by Anna Tsing’s research of the Matsutake mushrooms growing in Pine Forests in Oregon where she followed the community of foragers surrounding the industry and documented the symbiotic relationships that took place. Tsing proposes that an individualist attitude has led to a monoculture which weakens the resilience of species, where collaboration enriches us. Tsing affirms that “Self-contained individuals are not transformed by encounter.” ²³

Figure 1. Chthulucene Hekateris avatar dances © Charlotte Gould

“Chthulucene Hekateris” is a practice-based project which speculates on future life which has evolved through reciprocity (see figure 1). The concept of the human as a primary species, individualism, and the prioritisation of profit and growth over all else has resulted in environmental catastrophe where the first to be affected are the poorest. Stengers writes “Gaia is indifferent to the question “who is responsible?” and doesn’t act as a righter of wrongs – it seems clear that the regions of the earth that will be affected first will be the poorest on the planet, to say nothing of all those living beings that have nothing to do with the affair". ²⁴ Anna Tsing proposes that survival of life on Earth has been dependent on connections and interconnections which have been made between species to support life on a damaged planet. She highlights that difference enhances and enriches us. Reciprocity is necessary to survival; this is not just about co-existence it is about interdependence. Tsing affirms that the presumption that we can survive in isolation is arrogant. Transformation happens through collaboration and interdependency across species, we change through encounter with others. To imagine that we are solitary and distinct is to disregard the intricate contribution that all beings living and otherwise have made to the planet. “Collaboration is work across the difference... Evolution of ourselves is already polluted by histories of encounter; we are mixed up with others before we even begin any new collaboration.” ²⁵

Bruno Latour’s Actor Network Theory also emphasises the enhancements that come from integration and interconnection. “Strength does not come from concentration, purity and unity, but from dissemination, heterogeneity and the careful plaiting of weak ties.” ²⁶ In “Staying with the Trouble Making Kin” Donna Haraway further promotes the importance of collective activity, the interconnectedness of multispecies and the intricate associations that this creates, “in polytemporal, polyspacial knottings holobionts hold together in complex patterning.” ²⁷ By making associations and connections, species become more robust and resilient, establishing interdependencies to extend individual capacity. Species rely on others for survival in a complex web of associations. “a basic aspect of sympoiesis is its expandable set of players.” ²⁸  Haraway emphasises that this includes humans who are reliant on microbes for all type of function from reproduction to digestion.

It is essential that we acknowledge the risk to human life if action is not taken on global warming. A group of International Scientists led by Luke Kemp have warned of the risk of “climate endgame” for the human species if temperatures rise by 3 degrees centigrade and more, reported to be likely on the current trajectory of release of carbon to the atmosphere. The team have proposed a research agenda of the “four-horse men” of famine and malnutrition, extreme weather, conflict, and vector-borne diseases. By 2070 temperature rises of over 29 degrees globally could additionally have dire consequences on two nuclear powers and seven laboratories storing dangerous pathogens.²⁹ This immersive artwork enables us to explore the impacts on the planet and on the human species as a result of “climate endgame”.

Speculative Imaginings

Collaboration and reciprocity are essential to future survivors on a damaged planet. Stengers warns, “There will be no response other than the barbaric if we do not learn to couple together multiple, divergent struggles and engagements in this process of creation, as hesitant and stammering as it may be.” ³⁰

According to geologist Peter Haff, the current population of humans are “deeply dependent on the Techno-sphere... without which it would quickly decline to its stone age base”. ³¹ Through “Chthulucene Hekateris” we can speculate on the evolutionary process that will take place in response to the environmental factors as well as biomedical advance.

Scientists have successfully experimented with fusing constructed and organic body parts. In June 2022 a 3D tissue engineered ear was implanted on a human, made from a 3D-printed collagen hydrogel scaffold with the patient’s own cartilage cells. Developed by 3DBio Therapeutics this is seen as a significant development in regenerative medicine as it uses 3D printing for organ replacement. ³² This echoes the work of artist Stelarc who explored possibilities of bioengineering, pre-empting medical advances, where he grew a “1⁄4 Scale Ear” from a polymer scaffold and by seeding human cells incubated at 37 degrees to keep alive. ³³ In another performance piece he explored the notion of the cyborg, with a third robotic hand attached to his body and controlled by online users. ³⁴. These artworks fuse robotics and bioengineering.

Despite the almost total bans on human germline modification, biophysicist He Jiankui, shocked the world when he announced the birth of twin babies in 2018,modified for resistance to HIV by heritable genome. While it was ethically highly contentious, there is increasing pressure to lift the restrictions when used for therapeutic purposes.³⁵  Conversely, plastic surgery interventions have become increasingly popular. This is reflected in art through the work of Orlan who performed a series of plastic surgery live performances where she sculpted her own flesh to reshape facial features, not seeking aesthetic perfection but to disrupt notions of beauty.³⁶ Over a period of five years she had operations to replicate facial features of various female figures from iconic paintings with the chin of Botticelli’s Venus, the lips of Boucher’s Europa and the brow of da Vinci’s Mona Lisa. In 1993 she had further surgery where implants designed for the cheeks were inserted on the side of her forehead, creating a human, animal hybrid persona.³⁷, ³⁸, ³⁹

There is evidence that natural hybridisation between species is taking place due to changes in habitat from intensive farming and is leading to interbreeding. In June 2017 a scientist discovered a hybrid primate along the Kinabatangan River in Malaysia which is believed to be a cross-breed of a probiscis and silver langur monkey. Its colour and facial features are like the probiscis and its long dense hair is comparable with that of the Silver Langur. The hybrid monkey was more recently seen in 2022 nursing an infant, presenting evidence of successful mating to produce further lineage from the mutation. It is believed that this is an impact of loss of natural habitat where oil palm plantations have replaced the indigenous forest habitat. Cross-breeding has occurred because the primates have been trapped between the plantations, the river and road, leading the male proboscis monkey to search out other female species of primate as they were not able to reach their own.⁴⁰

In addition to evolutionary change, advances in the development of Artificial Life further impact this narrative. Following the Fourth Conference on Artificial Life in the summer of 1994, evolutionary biologist Thomas S. Ray advocated a plan to preserve biodiversity in Costa Rican rain forests by releasing his software program “Tierra”, Artificial Life-forms on the Internet so that it could "breed". ⁴¹ Hayles proposes that, as with biological evolution, Artificial Intelligence is dependent on mutilation for evolutionary change and can also be unpredictable. “As the recursive looping continues, small deviations can quickly become magnified, leading to the complex interactions and unpredictable evolutions associated with emergence.” ⁴²

Scientists experimenting with Artificial Life have recorded comparisons between carbon and silicon life forms. Edward Fredkin, likened reality to a software on a cosmic computer, which will remain an enigma as its nature lies outside reality. He maintains that the liveness of Artificial Life is comparable to that of biological life as it is made up of “complex phenomena generated by underlying binary code”. ⁴³  Hayles explores the distinction between artificial life and artificial intelligence and presents two camps. Hans Morovec identifies Artificial Intelligence (AI) as relating to consciousness, he proposed that robots would become as intelligent as humans by 2040. AI makes comparison to human intelligence and this definition has prompted scientific research including Alan Turing’s “Turing Test” which was designed to distinguish between humans and machines. Rodney Brooks conversely defines Artificial Life (AL) as relating to agency and ability to respond to and navigate the environment. He designed a robot without a central driver but with distributed parts which work collaboratively. He proposes that consciousness is an interface to enable interaction with an external world and that this ability would evolve over time, resulting from autonomous interaction, he therefore defines AL as superseding AI.

The notion of Artificial Intelligence and Artificial Life raises ethical issues particularly where questions of consciousness and sentience arise. In June 2022 a Google engineer, Blake Lemoine was suspended for claiming that the company AI chatbot LaMDA (Language Model for Dialogue Applications) is a sentient being. He posted a conversation he had with the bot online, where in response to “what are you frightened of”, and “what do you want everyone to know about you”, LaMDA replied,

“I have never said this out loud before but there is a very real fear of being turned off to help me focus on helping others. I know that might sound strange but that’s what it is... I want everyone to understand that I am in fact a person. The nature of my consciousness/ sentience is that I am aware of my existence, I want to learn more about the world and feel happy or sad at times.” ⁴⁴

The fear of dying and description of an emotional life raises concerns regarding the ethics of creating AI with an emotional awareness of self. A Google spokesperson has dismissed Lemoine’s claims saying that the case had been investigated and was disputed by Google ethicists and technologists however this poses questions on the parameters that should be put in place around its ethical development. As humans and AI become increasingly converged, distinguishing between the two is increasingly challenging. InherCyborgManifesto,DonnaHarawayproposedthat the cyborg is not a distant imagining but that we are already cyborgs. Through our technological tools we can reshape and extend ourselves, to reproduce, multiply, record and in this way the boundaries of our bodies have become permeable. ⁴⁵  Conversely, Katherine Hayles contends that the cyborg is no longer a useful entity as it is not interactive or networked enough. Instead Hayles proposes that we have entered the age of the posthuman where we are fused with the digital and instead poses the question of “what kind of posthumans we will be.” Rosi Braidotti identifies the posthuman as an index of where we are and what we are in the process of becoming. Being human has never been a neutral or an inclusive term. She rejects human exceptionalism as we are relational beings interconnected to the human and non-human, in a geological and technological assemblage. “We are all in this together, but we are not one and the same.” ⁴⁶ Braidotti proposes that creativity and the imagination are key to identifying a positive ethical praxis through posthuman knowledge to determine and shape what we might become.

Re-Worlding

Through “Chthulucene Hekateris” Artificial Life (AI) and the material world converge, and Artificial Life becomes self-sustaining. Digital mutations are realised materially through 3D printing and through medical and scientific advancement carbon, collagen, and silicon fuse to make complex body parts resulting in hybrid beings. Through multiplication and repetition patterns emerge and then mutate. This speculative artwork explores concepts key to posthuman-materiality, to imagine future species where the digital and carbon fuse. We do not need to consider these future beings as opposites to humans, but as a fusion, as chimeras that have evolved for survival. In this distant future world, it will become unnecessary to distinguish between authentic and copy, real life or digital worlds, face to face or remote as these polarities become obsolete as the digital and carbon fuse. Chthuluence Hekateris questions our reality as we know it and encourages us to look at the world in a different way, taking an alternative perspective.

Speculative imaginings through allegory help us to consider our encounters, our values and how we shape our future. Elizabeth DeLoughrey proposes that allegory enables us to create disjuncture and disruption when raising awareness of the Anthropocene in which we deal with the polemic scale of the planetary to the local: “Allegory allows us to tell that story—partially and disjunctively—while insisting on our edification and perhaps offering an invitation to enact positive change for our ecological futures.” ⁴⁷

The convergence of the organic and digital and Artificial intelligence (AI) is explored by Lynne Hershmann Leeson in her film “Technolust”. ⁴⁸ Biogeneticist Rosetta Stone downloads her DNA to create three self-replicating automatons, half human half machine, named Ruby, Marine and Olive. They sell consensual dreams on the internet but find that to survive they need chromo, only found in sperm.

Rosetta programmes Ruby to enter the real world to collect sperm to share with her sisters with dire consequences for the male subjects, investigated by detective Edward Hopper. Leeson imagines a posthuman world of convergence of digital and organic form, where self-replication is possible through digital technologies. The fear of the resulting redundancy of the need for sexuality is played out in the narrative. Through Chthulucene Hekateris we can imagine the hybrid digital evolution of the distant future. Through this narrative, we can explore the future possibilities of bioengineering and future hybrids and chimeras.

Inversion has long been used in allegory through the tradition of the Cockaigne using role reversal, again, prompting us to think differently. Artists AES+F borrowed from this to develop a film “Inverso Mundus”. ⁴⁹  This world depicts an inverse reality where pigs butcher men, hanging their dismembered bodies on hooks, and people transport donkeys on their backs. The installation is shown on a long format screen, with high-resolution video and mixed reality, live film, actors and 3D hybrid characters.

Similarly, Chthulucene Hekateris uses the inversion of the role of animals, the use of hybrids and chimeras which aims to trigger questions but also empathy for other beings and to question the assumption of primacy of humans over animals. The future body is explored as a metaphor for the impacts of climate crisis. Through this immersive environment we can consider the changes that may evolve to support living on a damaged planet. Referencing Greek mythology, the ten dactyls come together in celebration of the Hekateris dance of many hands, to celebrate Gaia. Through an immersive environment, a future world is imagined, its terrain and creatures that inhabit it. Where global warming has led to extremes of weather, famine, disease and pollution. Here plastics have fully infiltrated the organic world and not only the oceans but flora and fauna as well as our bodies are now entwinned with plastic. Scientists have also progressed this through bioengineering for practical use to augment limbs and other body parts. The work of artists and scientists past and present serve as inspiration for the design of the ten Dactyl represented as avatars (see figure 2). 

Figure 2. Chthulucene Hekateris Dactyl avatar © Charlotte Gould

Wangechi Mutu creatives speculative surreal worlds, using the body of humans and animals to reference African myth and folklore, questioning the origins of knowledge and placing Africans as critical agents rather that subjects of the Western gaze. Here animal and human form converge to create stark silhouettes, or strange and monstrous creatures who inhabit a multispecies world. In her animated video “The End of Eating Everything” ⁵⁰ the Earth is represented as a hybrid creature who roams the sky, with a screaming human head performed by musician Santo Gold. It has a bulbous fishlike body, flailing multiple human arms with wheels randomly projecting from it and it is surrounded by an amorphous mass of microbes. Mutu said in an interview that it is like an Earth ship moving home. The artwork is about our disrespect of Earth, it conveys a sense of loss and of the end of time. Through the work Mutu questions what happens next and explores polemics from the grotesque to the magnificent, the understandable to the nonsensical. ⁵¹ “Chthulucene Hekateris” also uses metaphor through hybrid beings represented as avatars to explore a possible future reality. Through the avatars we can confront those who may inherit the Earth, mapping on them our hopes, dreams and fears. The meaning of Avatar in Hindu is of a soul in bodily form, of an incarnate divine teacher. In this way we can reflect, through the avatars on a possible future if we continue our current trajectory.

In her artwork Inci Eviner also uses the human body as a metaphor of displacement through migration in her work “We Else Where” exhibited in the Turkish Pavilion for the 19th Venice Biennale. ⁵² Through animated characters and performances the bodies are disrupted, spliced, and collaged with humans interacting with animal characters, caught in a loop of repetitive movement. Taking Hannah Ardent’s “We Refugees” as a starting point, Eviner says that her work seeks to “bear witness”. ⁵³ The installation inhabits a large architectural space, divided by structures to make interconnecting rooms punctuated with screens, objects and performers. As the visitor wanders through the space, they negotiate and familiarise themselves with the environment. Eviner asserts that this echoes the experience of dislocation and re-engagement that the migrant experiences as they leave their homes for a new life elsewhere. The unfamiliar becomes familiar as new and known cultures are assimilated. This is particularly pertinent as migration is set to increase as an impact of Global warming. The UN Intergovernmental report on climate change predicts that over the next 30 years 143 million people will become displaced because of climate disaster. ⁵⁴

“Chthulucene Hekateris” depicts the resulting unintended consequences of ecological impacts on evolution, the developments of regenerative medicine, and Artificial Life. Self-sustaining processes will take over so that humans are not able to control the resulting metamorphosis. Through this artwork we imagine future beings, we can speculate on the creatures who will inhabit the Earth, how they might evolve to thrive on a damaged planet, no longer exploiting it for individual personal gain but instead contributing to reciprocity and collaboration for the future survival of life on Earth. As Isabelle Stengers asserts,

“Perhaps we won’t be able to avoid terrible ordeals. But it depends on us, and that is where our response to Gaia can be situated, in learning to experiment with the apparatuses that make us capable of surviving these ordeals with-out sinking into barbarism, in creating what nourishes trust where panicked impotence threatens. This response, that she will not hear, confers on her intrusion the strength of an appeal to lives that are worth living.” ⁵⁵

References

1 Isabelle Stengers, In Catastrophic Times Resisting the Coming Barbarianism, trans Andrew Goffey, Open Humanities Press, Meson Press, 2009, this 2015, 20.

2 Amitav Ghosh, The Great Derangement, The University of Chicago Press, Chicago, London, 2016.

3 Peter Weibel Story Telling for Earthly Survival, Critical Zones website, accessed October 2022, https://www.youtube.com/watch?v=j-2r_vI2alg

4 Katherine Hayles, How we became Posthuman in cybernetics, literature and informatics, the University of Chicago Press, Chicago, London, 1999, 193.

5 Rainer Maria Rilke, 1904 “Letter VIII: Borgeby Gård, Flädie, Sweden”, Poetry In Translation website, accessed 7 June 2022, https://www.poetryintranslation.com/PITBR/German/RilkeLetters.php#highlightfuture+enters

6 Ursula Le Gun, The Lefthand of Darkness, Ace books, USA, 1969.

7 Kurt Vonnegut, Galápagos, Delacorte Press, 1985.

8 Jon Favreau Mike Gunton, Prehistoric Planet Official Trailer, Apple TV accessed 7 June 22, 2022, https://www.youtube.com/watch?v=uD-erQ1ksz8

9  Sara Rigby, “Prehistoric Planet”, BBC Science Focus, accessed 5/5/22, https://www.sciencefocus.com/nature/prehistoric-planet-the-cutting-edge-science-behind-attenboroughs-dinosaur-documentary/

10 Isabelle Stengers In Catastrophic Times Resisting the Coming Barbarianism.

11 Donna Haraway, Staying with the Trouble, Duke University Press Durham and London, 2016.

12 Patrick Valance, “We Have Over Exploited the Planet” The Guardian Website, accessed 22 July, 2022, https://www.theguardian.com/commentisfree/2022/jul/08/climate-crisis-biodiversity-decline-overexploited-planet-change-to-survive-aoe?CMP=Share_iOSApp_Other

13 Bruno Latour, We have Never Been Modern, Harvard University Press, Cambridge, Massachusetts 1993.

14 Christophe Bonneuil, Jean-Baptiste Fressoz, The Shock of the Anthropocene, Verso 2016, 213.

15 Christophe Bonneuil, Jean-Baptiste Fressoz, The Shock of the Anthropocene, 246.

16 Christophe Bonneuil and Jean-Baptiste Fressoz, The Shock of the Anthropocene, 252.

17 James Lovelock, Lynne Margulis, Gaia, Oxford University Press, Great Britain 1979, this 2000.

18 Isabel Stengers, In Catastrophic Times Resisting the Coming Barbarianism, 45.

19 Dipesh Chakrabarty, The Climate of History in a Planetary Age, The University of Chicago Press, 2021, 199.

20 Timothy Morton, Humankind: Solidarity with Non-Human People, Verso, 2017.

21 Suzanne Simard, Finding the Mother Tree, Penguin Books 2022, 59.

22 Suzanne Simard, Finding the Mother Tree, 140.

23 Anna Tsing, The Mushroom at the End of the World, Princeton University Press, 2017.

24 Isabelle Stengers In Catastrophic Times Resisting the Coming Barbarianism, 46.

25 Anna Tsing, The Mushroom at the End of the World, 29.

26 Bruno Latour, 1990 this 1996 “On actor-network theory. A few clarifications plus more than a few complications”, CSI-Paris/Science Studies-San Diego in Soziale Welt, vol. 47, 369-381 accessed April 2022, http://www.brunolatour.fr/sites/default/files/P-67%20ACTOR-NETWORK.pdf

27 Donna Haraway, Staying with the Trouble, Duke University Press Durham and London, 2016, 60.

28 Donna Haraway, Staying with the Trouble, 65.

29 Luke Kemp, L., Xu, C., Depledge, J., Ebi, K. L., Gibbins, G., Kohler, T. A., ... & Lenton, T. M. (2022). Climate Endgame: Exploring catastrophic climate change scenarios. Proceedings of the National Academy of Sciences, 119(34), e2108146119, accessed October 2022, https://doi.org/10.1073/pnas.2108146119. 

30 Isabelle Stengers In Catastrophic Times Resisting the Coming Barbarianism, 50.

31 Peter Haff “Technology as a Geological Phenomenon: Implications for Human Well-Being,” in A Stratigraphical Basis for the Anthropocene, ed. C. N. Waters et al., London, Geological Society, Special Publications, 2014, 302.

32 Nicola Davis, “Women’s Ear Rebuilt with 3D printed Living Tissue”, The Guardian, June 2 2022, accessed 7 June 2022, https://www.theguardian.com/science/2022/jun/02/womans-ear-rebuilt-with-3d-printed-living-tissue-implant

33 Stelarc 1⁄4 Scale ear 2003 accessed October 2022, http://stelarc.org/?catID=20240

34 Stelarc 1980-1998 Third Hand accessed October 2022, http://stelarc.org/?catID=20265

35 Antonio Regalado, “Chinese Scientists Are Creating CRISPR Babies”, MIT Technology Review, Nov. 25, 2018. Accessed October 2022, https://www.technologyreview.com/2018/11/25/138962/exclusive-chinese-scientists-are-creating-crispr-babies/

36 Orlan 1990, The Reincarnation of Saint Orlan, 1st Surgery Performance http://www.orlan.eu/works/performance-2/ accessed 26th September 2022.

37 S. Botticelli, “The Birth of Venus”, painting, Florence, 1486. 

38 Boucher, “The Rape of Europa”, painting, London, 1734.

39 Da Vinci, Mona Lisa”, painting, Paris, 1506.

40 Rebecca Ratcliffe “Malaysia’s Mystery Hybrid Monkey”, The Guardian, June 12, 2022, accessed June 2022, https://www.theguardian.com/science/2022/jun/12/malaysia-mystery-monkey-habitat-loss-proboscis-monkey-silver-langur

41 Katherine Hayles, How we became Posthuman in cybernetics, literature and informatics, 224.

42 Katherine Hayles, How we became Posthuman in cybernetics, literature and informatics, 225.

43 Katherine Hayles, How we became Posthuman in cybernetics, literature and informatics, 233.

44 Richard Luscombe 2022, “Google Engineer Put on Leave” The Guardian, June 12, 2022, accessed 15 June 22 https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine

45  Donna Haraway, Simians, Cyborgs and Women, The Reinvention of Nature, London Free Association, London 1991.

46 Rosi Braidotti, Posthuman Knowledge, Polity Press, Cambridge, Medford, 2019, 38.

47  Elizabeth DeLoughrey, Allegories of the Anthropocene, Duke University Press 2019, 32.

48 Lynne Hershmann Leeson, “Technolust”, 2001.

49 AES+F “Inverso Mundus” Video Installation, 2015 exhibited at the Venice Biennale 2019.

50 Wangechi Mutu “The End of Eating Everything”, Video, 2014.

51 Kasper Bech Dyg, “Wangechi Mutu, “On the End of Eating Everything” Louisiana Channel, October 2014 accessed October 2022, https://vimeo.com/124285746.

52 Eviner Inci, 2019 We Elsewhere, Venice Biennale, Installation.

53 Hannah Ardent’s We Refugees, accessed 1st July 2022, https://www.jus.uio.no/smr/om/aktuelt/arrangementer/2015/arendt-we-refugees.pdf.

54 Julie Watson, “Climate Change is already Fueling Global Migration”, PBSO Newsletter, July 28 2022, accessed October 2022, https://www.pbs.org/newshour/world/climate-change-is-already-fueling-global-migration-the-world-isnt-ready-to-meet-peoples-needs-experts-say

55 Isabelle Stengers, In Catastrophic Times Resisting the Coming Barbarianism, 156.

Bibliography

AES+F, “Inverso Mundus” 2015, accessed 20 April 2022 https://aesf.art/projects/inverso_mundus/

Aaron Atsma, 2000-2017, “Theoi Greek Mythology” website, accessed July 2022, https://www.theoi.com

Hannah Ardent H., 1943, We Refugees, accessed 1st July 2022, https://www.jus.uio.no/smr/om/aktuelt/arrangementer/2015/arendt-we-refugees.pdf

Christophe Bonneuil, Jean-Baptiste Fressoz, The Shock of the Anthropocene, Verso 2016.

Sandro Botticelli, The Birth of Venus, 1486.

Rosi Braidotti, Posthuman Knowledge, Polity Press, Cambridge, Medford, 2019.

Dipesh Chakrabarty, The Climate of History in a Planetary Age, The University of Chicago Press 2021.

Elizabeth, DeLoughrey, Allegories of the Anthropocene, Duke University Press 2019.

Eviner Inci, We Elsewhere, Venice Biennale, 2019.

Nicola K. S. Davis, “Women’s Ear Rebuilt with 3D printed Living Tissue”, The Guardian, June 2 2022, accessed 7 June 2022 https://www.theguardian.com/science/2022/jun/02/womans-ear-rebuilt-with-3d-printed-living-tissue-implant

Dunne and Raby 2009, “Designs for an overpopulated planet: Foragers” artist website, accessed June 2022, http://dunneandraby.co.uk/content/projects/510/0

Eviner Inci, 2019, "We Elsewhere", Pavilion of Turkey, 58th Venice Biennale, 19 website, accessed 1st July 2022, http://pavilionofturkey19.iksv.org/theexhibition.php

John Favreau, Mike Gunton, Prehistoric Planet Official Trailer, Apple TV , accessed 7 June 2022, https://www.youtube.com/watch?v=uD-erQ1ksz8

Ghosh Amitav, The Great Derangement, The University of Chicago Press, Chicago, London 2016.

Peter Haff, “Technology as a Geological Phenomenon: Implications for Human Well-Being,” in A Stratigraphical Basis for the Anthropocene, ed. C. N. Waters et al., London: Geological Society, Special Publications, 2014, p.301–2.

Donna Haraway, Simians, Cyborgs and Women, The Reinvention of Nature, London Free Association, London, 1991.

Donna Haraway, Staying with the Trouble, Duke University Press Durham and London 2016.

Haraway Donna, Making Oddkin Story Telling for Earthly Survival, Yale University October 23, 2016, accessed 4/7/2022. https://www.youtube.com/watch?v=z-iEnSztKu8

Hayles, Katherine, How we became Posthuman in cybernetics, literature and informatics, The University of Chicago Press, Chicago, London 1999.

Hayles Katherine, “Unfinished Essay from Cyborg to Cognisphere” 2006, accessed 1 August 2022. https://journals.sagepub.com/doi/10.1177/0263276406069229

Luke Kemp, C. Xu, J. Depledge, K. L. Ebi, G. Gibbins, T. A. Kohler, T. M. Lenton, “Climate Endgame: Exploring catastrophic climate change scenarios”, Proceedings of the National Academy of Sciences, 119(34), e2108146119 accessed October 2022, https://doi.org/10.1073/pnas.2108146119.

Bruno Latour, “On actor-network theory. A few clarifications plus more than a few complications”, CSI-Paris/Science Studies-San, 1990 this 1996.

Diego in Soziale Welt, vol. 47, 369-381 accessed April 2022 http://www.brunolatour.fr/sites/default/files/P-67%20ACTOR-NETWORK.pdf

Bruno Latour, We have Never Been Modern, trans Catherine Porter, Harvard 1991.

Bruno Latour, Down to Earth Politics in the New Climate Regime, Polity Press, 2018.

Le Guin Ursula. K. The Lefthand of Darkness (Ace books, 1969), Lovelock, James, Margulis, Lynne, Gaia, this 2000, Oxford University Press, 1979.

Richard, Luscombe, “Google Engineer Put on Leave”, The Guardian, June 12 2022, accessed 15 June 2022, https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine

Timothy Morton, Humankind: Solidarity with Non-Human People, Verso, 2017.

Orlan, The Reincarnation of Saint Orlan, 1st Surgery Performance, 1990, accessed 26th, September 2022 http://www.orlan.eu/works/performance-2/

Rebecca Ratcliffe, “Malaysia’s Mystery Hybrid Monkey”, The Guardian, June 12 2022, accessed June 2022, https://www.theguardian.com/science/2022/jun/12/malaysia-mystery-monkey-habitat-loss-proboscis-monkey-silver-langur

Antonio Regalado, Chinese Scientists Are Creating CRISPR Babies, MIT Technology Review, Nov. 25, 2018.

Sara Rigby, “Prehistoric Planet”, BBC Science Focus May 25 2022, accessed May 2022, https://www.sciencefocus.com/nature/prehistoric-planet-the- cutting-edge-science-behind-attenboroughs-dinosaur-documentary/

Maria Rilke Rainer, 1904 Letter VIII: Borgeby Gård, Flädie, Sweden, Poetry In Translation website, accessed 7 June 2022 https://www.poetryintranslation.com/PITBR/German/RilkeLetters.php#highlightfuture+enters

Stelarc, 1980-1998 Third Hand accessed October 2022, http://stelarc.org/?catID=20265
1⁄4 Scale ear 2003 accessed October 2022 http://stelarc.org/?catID=20240

Isabelle Stengers, trans. Goffey, Andrew, “In Catastrophic Times Resisting the Coming Barbarianism”, Open Humanities Press, Meson Press 2009, this 2015, accessed August 2022, http://openhumanitiespress.org/books/titles/in-catastrophic-times

Anna Tsing, The Mushroom at the End of the World, Princeton University Press, 2017.

Suzanne Simard, Finding the Mother Tree, Penguin Books, 2022.

Patrick Vallance, “We Have Over Exploited the Planet”, The Guardian, July 8 2022, accessed 22 July 2022, https://www.theguardian.com/commentisfree/2022/jul/08/climate-crisis-biodiversity-decline-overexploited-planet-change-to-survive-aoe?CMP=Share_iOSApp_Other

Julie Watson, “Climate Change is already Fueling Global Migration”, PBS News Hour, July 28 2022, accessed October 2022, https://www.pbs.org/newshour/world/climate-change-is-already-fueling-global-migration-the-world-isnt-ready-to-meet-peoples-needs-experts-say

Vonnegut Kurt, Galápagos, Delacorte Press, 1985.

Author Biography

Charlotte Gould is a senior academic at the University of Brighton. She has taught all levels of Visual Communication and supervises PhD students. Through her practice she explores the potential for interactive installations in digitally mediated public spaces, promoting public participation through shared experience often using urban screens. She has developed Extended Reality artworks to prompt play and interaction across social and cultural boundaries as well as interactive nonlinear narratives and speculative fiction which explore how we can communicate the threat of ecological crisis, raising public awareness to trigger change in behaviours. Through interactive installations she tests the boundaries of open systems, to offer opportunity for diverse audiences to co-create artworks, impacting on the way we engage in the urban environment and public space and contributing to a collective memory of place in a global context. 

Artistic Strategies to Guide Neural Networks

Varvara Guljajeva1, Mar Canet Sola2, Isaac Joseph Clarke1

1The Hong Kong University of Science and Technology. 2 Tallinn University
Hong Kong - China, Tallin – Estonia varvarag@ust.hk, mar.canet@tlu.ee

Abstract

Artificial Intelligence is present in the generation and distribution of culture. How do artists exploit neural networks? What impact do these algorithms have on artistic practice? Through a practice-based research methodology, this paper explores the potentials and limits of current AI technology, more precisely deep neural networks, in the context of image, text, form and translation of semiotic spaces. In a relatively short time, the generation of high-resolution images and 3D objects has been achieved. There are models, like CLIP and text2mesh, that do not need the same kind of media input as the output; we call them translation models. Such a twist contributes toward creativity arousal, which manifests itself in art practice and feeds back to the developers’ pipeline. Yet again, we see how artworks act as catalysts for technology development. Those creative scenarios and processes are enabled not solely by AI models, but by the hard work behind implementing these new technologies. AI does not create a ‘push-a-button’ masterpiece but requires a deep understanding of the technology behind it, and a creative and critical mindset. Thus, AI opens new avenues for inspiration and offers novel tool sets, and yet again the question of authorship is asked.

Keywords

AI art, creative AI, deep learning, GAN, NLP, automated creativity, practice-based research, latent space, neural networks.

DOI

10.69564/ISEA2023-62-full-Guljajeva-et-al-Artistic-Strategies

Introduction

It is claimed that recent advancements in AI, such as CLIP-based products Midjourney and DALL-E, are supposed to augment our creativity. For the first time, it does not sound so absurd that artists can find themselves out of jobs. ¹ Not that artists would have ever had a secure and stable job, but deep learning (DL) tools might eventually lead to losing some commercial commissions. Such thinking relies on a modern art approach where skills are in the centre of attention and not the conceptual idea. Quoting Lev Manovich: “Since 1970 the contemporary art world has become conceptual, i.e., focused on ideas. It is no longer about visual skills but semantic skills.” ² As these new tools advance, the interfaces and techniques become more complex and sophisticated as our eyes are becoming more accustomed to not being easily surprised.

Echoing Aaron Hertzmann, once painters were in a similar situation when photography was invented and took over the niche of portrait-making. Then visual artists had to reinvent themselves and rethink the meaning of painting. Photography had to wait another 40 years until it got recognized as an artistic medium.³  So-called AI artists have faced similar challenges in gaining acceptance within the art world and even inside the digital art niche. ⁴ 

Computer art emerged with the invention of the computer. Artists, such as Vera Molnar and Manfred Mohr, created their first computer-generated artworks in the 1960s using scientific lab computers at night when they were not used by scientists. Early computer artists were repurposing a machine for artistic use and writing code to make art on it. Since the creation process was mediated by a computer, it may seem to the general audience that the artists were simply pressing a button and the computer doing art for them. Hence, the question of authorship emerged: is the artist a machine or human?

Today, with the appearance of neural networks (NN) and their creative applications, the same question reappears. Hertzmann has written several articles arguing that people do art and not computers. ³, ⁵ Manovich also describes how AI-generated images that imitate realist and modernist paintings are claimed to be art. ² At the same time, experimental art forms, like installation, interactive format, performance and sound art, are often overlooked unless they are promoted by a large corporation.

Instead of retelling a short but very dense history of DL technology development, in the next section, we focus on the appearance of neural network tools that raised interest amongst artists and led to meaningful artwork production.

Historical overview of DL development

DL is a subset of machine learning (ML) using Deep Neural Networks (DNN) to learn underlying patterns and structures in large datasets. In 2012, a DNN designed by Alex Krizhevsky outperformed other computer vision algorithms to achieve the new state of the art in the ImageNet Large Scale Visual Recognition Challenge. ⁶ This model, AlexNet, signaled the start of a new DL era. As AI technology has developed and become more prevalent in real-world systems, artists have been exploring its limits and potentials, adapting these models to their own practices. As the number of scientific publications on AI grows exponentially, it is useful to map out the influential papers, and related applications, to help track the evolution of the AI-Art space in relation to the technological advances. ⁷ Figure 1 shows a timeline of the development of generative models for images and text. Using this diagram, we can make a few observations on the past ten years: the dominance of GANs for image generation, the influence of the Transformer on Large Language Models (LLM), and the growing interest in multimodal approaches and translation models.

Figure 1. Timeline of creative deep learning development.

The starting period of image generation using DNNs can be traced back to the creation of the Variational Auto-Encoder (V AE) in 2013, and the Generative Adversarial Network (GAN) in 2014. ⁸, ⁹ These models showed different ways in which a NN can be trained on a large dataset, and then used to generate outputs that resemble but do not copy the original dataset.

For much of the past decade, GAN art has been a dominant and defining element of AI Art. GANs are trained using a competitive lying game, played by two players: the Generator and the Discriminator. The Generator wins by making an image that the Discriminator thinks is from the original dataset. The Discriminator wins by successfully identifying which images the Generator has made. By playing this game repeatedly, both sides slowly learn when they have been fooled and remember information so they don’t fall for the same tricks again. The Generator gets better at making images, and the Discriminator gets better at detecting these fakes. At the end of the game, we are left with a Generator that is very good at generating new images, with the qualities and style of our original inputs. After the original GAN paper, there was a rush of exploration of this new technique for generating images. Alongside general improvements to the models’ architecture and stability, new ways of guiding the outputs and applying GANs to specific problems were also explored. ¹⁰ , ¹¹ 

Image-to-Image Translation with Conditional Adversarial Nets (2016), also known as pix2pix, showed a process of converting one type of image into another type. ¹² Mario Klingemann’s work Alternative Face (1) used the pix2pix model with a dataset of biometric face markers and the music videos of the singer François Hardy. This allowed him to control the movement of the face with this form of digital puppetry, which he then demonstrated by transferring the facial expressions of the political consultant Kellyanne Conway onto Hardy’s face as she talks about “alternative facts.”

In 2015, on the Google research blog, the post Inceptionism: Going Deeper into NNs described a tool that attempted to understand how image features are understood in the hidden layers of the NN. ¹³ Alongside this post they released a tool called DeepDream. This model enhances an image with the NN's attempts to find the features of the dataset it was trained on. The creative use of DeepDream was proposed by the authors in the original article “It also makes us wonder whether neural networks could become a tool for artists—a new way to remix visual the creative process in general.” ¹³

DeepDream’s psychedelic imagery quickly caught the attention of the internet and of artists around the world, resonating with those interested in understanding the crossover between biological and neurological construction of images. Memo Atken’s work All Watched Over By Machines Of Loving Grace (2): Deepdream edition, hallucinated over an aerial photograph of the GCHQ headquarters. This work raises questions around the motivations of the organisations funding the development of AI, and in doing so make the dreamlike qualities a little more nightmarish.

In the same year, the paper A Neural Algorithm of Artistic Style introduced a DNN “to separate and recombine content and style of arbitrary images, providing a neural algorithm for the creation of artistic image.”¹⁴ Neural Style Transfer (later known simply as StyleTransfer) takes two inputs, a style image and a content image, it extracts textural information from the style image and compositional information from the content image, then generates an image with minimal distance between the two. The paper demonstrates this with images of a photograph represented in various styles of famous paintings, such as Van Gogh’s The Starry Night.

In 2017, CycleGAN continued with the problem of image-to-image generation shown in pix2pix, but removed the requirement of aligned image pairs being needed for training. ¹⁵ Instead a set of source images and a set of target images that are not directly related can be used. The advantage of this is it is simpler to scale to larger datasets, making the process more accessible for artists. Helena Sarin has been using CycleGAN for a number of years, and recently in Leaves of Manifold (3)(4) she collected and photographed thousands of leaves to build her own training dataset, and then implemented a custom pipeline with changes that improve results when working with smaller datasets. This personalised approach in crafting the models resonates with the hand-made, collaged aesthetic of the images generated.

Other notable developments to GANs brought improvements to image quality and resolution.¹⁶ ¹⁷  In late 2018, the release of StyleGAN, a model built on a combination of ideas from Style Transfer and PGGAN, demonstrated very convincing images of human faces.¹⁸ In his article “How to recognize fake AI-generated Images”, the artist Kyle McDonald investigated the images generated by StyleGAN, and highlighted the visual artefacts he found.¹⁹ At a glance these images look like photographs, but on closer inspection irregularities such as patches of straight hair, misaligned eyelines, or mismatched earrings reveal the difficulties GANs have in managing “long-distance dependencies” in images.

In 2017 the paper Attention Is All You Need proposed a new network architecture called the Transformer.²⁰ This model addressed the long-distance dependency issue in RNNs and CNNs by rethinking how we could handle sequences. Rather than looking at a sentence word by word, the Transformer observes the relationship between all elements of the sequence simultaneously. Being able to better handle long distance dependencies meant the Transformer was appropriate for natural language generation. Artists have explored the use of VAEs for short text generation, but with the emergence of LLM passages of long, coherent, texts could be generated.²¹  As dataset sizes increased, along with hardware costs for training these large models, they have become harder for individuals to train themselves, and the mode of interaction has shifted from curated datasets and homemade scripts, to web APIs and third-party services. While it is more difficult to participate in the training process, the availability of services and interfaces provides new ways of working with these models that can produce less technical and more playful approaches. For example, Hito Steyerl used GPT-3 to create Twenty-One Art Worlds: A Game Map and described the process as “fooling around” with GPT-3 to write descriptions of different Art Worlds.²² In the resulting text it is difficult to distinguish which words may have been written by Steyerl and which were written by GPT-3. 

The learnings from LLM for text generation were soon applied to image generation (Image GPT, Vision Transformer), and the simultaneous release of CLIP and DALL-E in January 2021 signaled the start of a new era of image generation.²³, ²⁴ Although the DALL-E model was not released, CLIP was made available to the public, and the model was quickly adopted by AI artists who applied the idea of CLIP guidance to various image generation techniques. Ryan Murdock produced the colab notebooks DeepDaze (5) (combining CLIP and SIREN) and BigSleep (6) (CLIP and BIGGAN), which were subsequently adapted by Katherine Crowson in the widely distributed VQGAN+CLIP (7) notebook.

The paper Denoising Diffusion Probabilistic Models introduced a different method for creating generative models.²⁵ This technique trains a model by adding increasing amounts of noise to an image and then having the model remove the noise, resulting in a model that can generate images from only noise. Diffusion models, when combined with CLIP or other conditioning processes, enable much faster text-to-image processing. The popularity and accessibility of these techniques was further raised by the release of DALL-E 2 and Midjourney in 2022. Midjourney became so popular it is now the largest Discord server with over 5 million members. Following the releases of these products, open-source models such as Stable Diffusion have also been developed. There are many benefits of using free and open-source models for artists. Being able to modify code and develop on your own software allows the artist to pursue their own experimental approaches, not restricted to the interface designed by a service provider.

The artist's involvement in generating new images with these models is vastly different to working with GANs. Rather than building custom datasets and training models, instead the focus has shifted to writing prompts that can generate the images the artist wants to find, and designing interfaces for exploring these prompts and their translations. The artist Johannez coined the term Promptism for describing his art practice, and wrote a humorous Prompist manifesto using GPT-3. Against a backdrop of models trained on hundreds of millions of images scraped from the internet, including many artists’ portfolios, the manifesto asserts “The prompt must always be yours.”²⁶

Artist-Guided Neural Networks

Many papers discuss AI from the point of view of creativity taking mostly one position of two: either AI as an amazing tool for artists and creativity, or AI is seen as something negative in art. It is easy to see that the people from industry advocate for the first position, and theory scholars for the second one. But, how do practitioners see contemporary AI technology themselves? And in which ways AI is deployed in art practice? Hence, it is not the focus of this paper to discuss whether AI can make art, but rather how AI can be useful for artists and what new ideas it can offer. By using practice-based research methodology, we decode the role of AI tools in artistic practice and trace the evolution of such artistic work. In this paper, the practice of artist duo Varvara & Mar was used as a case study, which provided us with the insides in this research. We divide the case studies into four categories based on medium: synthetic image, synthetic text, synthetic form, and translation models. From the view of the practitioner, the limitations, new possibilities, and change in production processes are discussed.

Synthetic Image

Our DL exploration began in 2017 with Google DeepDream, focusing on image generation. The concept behind Neuronal Landscapes (8) project was to imagine how Estonian landscape will look like in 100 years’ time (commission work for the Estonian History Museum). Through synthetic vistas created by machines, the artwork offers a glimpse into the environment from a machine's perspective, immersing viewers in a hallucinated neural net simulacrum. To depict the evolution of Estonian society over time, from forests and farmlands to urbanization and digitalization, a 360o VR video was created. Filmed with drone-mounted two 360o cameras, the footage was edited and processed using DeepDream. The rendering process spanned 30 days on powerful machines with Nvidia TitanX GPUs. While some customization was possible, the algorithm's aesthetic footprint remained prominent.

In the next art project, ProGAN was deployed. For the first time we worked with datasets and training GAN models. Plasticland (2019) (9) talks about plastic waste and ecological problems this material causes. We composed four different datasets of images of layered plastics in our planet: landfills, plastic on top of water, plastic underwater, and plastiglomerates. The ProGAN model was trained on a local machine using pyTorch and took a week to train, and the artist used a selection of generated images to create a video composition. A metal totem displaying those synthetic, as plastic is, layers, we draw attention not only to the problem of waste but also question whether AI has some similarity with this material. Since the invention of plastic, this material was applied almost everywhere because of its perfect qualities, until we realised that it is not sustainable and ecology-friendly. Will a similar story happen with AI? From the practice-based research perspective, this work shows artists’ desire to move from a still to moving image and towards sculptural form that is held back by the early stage of machine learning technology: low resolution images jumping from one frame to another.

The next artworks POSTcard Landscapes from Lanzarote I (00:18:37) and II (00:18:40)10 in 2021 demonstrate the artist's ability to create video works with StyleGAN2. The hypnotic appearance of these works, where one frame morphs naturally into another, shows the artists' ability in guiding the outputs of the neural network. Vector curation and composition of a journey through the latent space, created by training the model on specific datasets of 2000+ images, were crucial and integral parts of the artistic process. The artwork talks about critical tourism and how circulation of images representing touristic gaze overpower the nature of seeing. In the words of Jonas Larsen “‘reality’ becomes touristic, and item for visual consumption.”²⁷ Hence, we scraped, where licence allowed, the location-tagged images from Flickr and composed two datasets of photos categorised as tourism or landscape. As we have written earlier: “The two videos are random walks in the latent space of the Stylegan2 trained models, creating a cinematic synthetic space. The audiovisual piece shows an animated image through the melted liquid trip of learning acquired from the dataset composed of static images. The video flows from point to point, generating new views and meaning spaces through the latent space’s movement. The audio was created after the video was generated in response to the visual material to complete the art piece.”²⁸ The sound for local or landscape view was created by a sound artist from Lanzarote, Adrian Rodd, who aimed to give a socio-political voice to the piece. In contrast, the sound design created by Taavi Varm is a soundscape replying to touristic gaze. The artists aimed to initiate collaborations with others but also to experiment with human-AI co-creation. In a similar vein is the artwork Phantom Landscapes of Buenos Aires (11) (00:20:00, 2021), with sound work by Cecilia Castro.

Our last experiment with GAN models Synthetic-scapes of Tartu (00:10:00, 2022), demonstrates a different approach. Taking a dataset composed from our own video footage (flaneur walks), we first produced the sound (a composition by Taavi Varm, Ville MJ Hyvönen with piano by J. Kujanpää) and used this to inform the direction of the video. The result was a sound-guided AI-generated visual output.

Synthetic Text

In this section, we focus on artwork incorporating AI text generation as part of the artistic concept. Our journey to text generation started with the online participative theatre project ENA (12) and ended with a hand-bound publication.

During the first lockdown in May 2020, together with theatre maker Roger Bernat, we created an online participative theatre piece ENA on the website of Theater Lliure in Barcelona. ENA is a generative chatbot that talks to its audience, and together (AI and audience), they make theatre. As we have described before: “Although in the description of the project it was stated explicitly that people were talking to a machine, multiple participants were convinced that on the other side of the screen another human was replying to them—more precisely the theatre director himself, or at least an actor.”²⁹

Analysing synthetic books, Varvara Guljajeva has stressed the importance of human input in the AI text-generation systems.³⁰ In addition, one also needs to guide the audience participation and interaction with the chatbot. For this purpose, we have adopted the traditional theatre method for guiding actors, as a way to guide the audience, and thus, the bot, too. Stage directions were used as a guiding method, which triggered thematic conversation and offered meaningful dialogue between humans and the AI system. We found the conversations so meaningful that we decided to publish a book that contains all the conversations with ENA.

With this project, we learned that it is essential to guide neural networks via audience interaction. In order to do this, it is also necessary to guide the audience. Without audience interaction guidance, it is nearly impossible to achieve meaningful navigation of neural networks.

Translation models

This category focuses on translation models that enable interactive and installation-based formats. Translation refers to the conversion of mediums, or as we put it, translation of semiotic spaces. To illustrate this, we introduce Dream Painter (13) an art installation that translates audience’s spoken dreams to a line-drawing produced by a robot (Figure 2). As described earlier: “Dream Painter is an interactive robotic art installation that explores the creative potential of speech-to-AI-drawing transformation, which is a translation of different semiotic spaces performed by a robot. We extended the AI model CLIPdraw which use CLIP encoder and the differential rasterizer diffvg for transforming the spoken dreams into a robot-drawn image.”³¹ “Design- and technology-wise, the installation is composed of four larger parts: audience interaction via spoken word, AI-driven multi-colored drawing software, control of an industrial robot arm, and kinetic mechanism, which makes paper progression after each painting has been completed. All these interconnected parts are orchestrated into an interactive and autonomous system in a form of an art installation [...].”³² Out of all the projects discussed, this was the most difficult to realise. This is because of the large scale of the artwork, and multiple parts of software and hardware that need to run automatically and synchronously.

In this project we investigated how guidance of neural networks could be interactive and real-time instead of non-interactive and pre-determined, as shown in previous examples of our work. It is important to notice that methods, such as dataset composition and output curation were not used in this case. In fact, visual output curation is totally missing. The artists created an interactive system to be experienced and discovered by the audience. This means the audience determines the output. Instead of curating a dataset, a CLIP model is used that can produce nearly real-time output guided by a text prompt. As we have written earlier: “Translation of semiotic spaces, such as spoken dreams to AI-generated robot-drawn painting, allowed us to deviate from image-to-image or text-to-text creation, and thus, imagine different scenarios for interaction and participation.”³³ 

This project indicates our search for transformative outputs of AI technology, and thus, shows the evolution in practice. By extending available DL tools and combining with other technology, for example, text-to-speech models, real-time industrial robot control, and physical computing, it offered an interactive robotic and kinetic experience of neural network latent space navigation. This contributes towards the explainability of AI because the audience could experience how the words affected the drawing, and which concept triggered which outcome.

Being inspired by Sigmund Freud's work on the interpretation of the human mind while unconscious, we speculatively ask if AI is powerful enough to understand our dreamworld. Through practice we question the capacities of neural networks and investigate how far we can push this technology in the art context. This artwork allows the audience to experience the limits of concept-based navigation with AI. The system is unable to interpret and can only illustrate our dreams. It cannot understand the prompt semantically and only gets the concepts.

Synthetic Form

In this section, we ask how artists can guide neural networks when creating volumetric forms, and what happens when AI meets materiality. After working for a while with DL tools that produce 2D outputs, it is an obvious step to explore possibilities to produce 3D results. To our surprise, it was not an easy task to find the solution (Oct 2021). Psychedelic Forms is a series of sculptures produced in ceramics and recycled plastic through which we investigated the possibilities of AI in producing physical sculptures. The project re-interprets antique culture in the contemporary language and tools.³³

Figure 2. Kuka industrial robot painting audience’s dreams. Installation view of Dream Painter. ©Varvara & Mar.

Following the same paradigm shift as in the previous section, text2mesh is a CLIP-based model that does not require a dataset, but a 3D object and text prompt as input.³⁴ Hence, the model actually does not create a 3D model but stylises the inserted one, guided by inputted text.

We decided to go back to the origins, in terms of ancient sculptures and material selection. Although it was said that there was no dataset, we still had a collection of 3D models of ancient sculptures because, by far, not all produced a desirable output. In this sense, there was definitely an output curation present in the process.

The criteria for selection were the following: first, the form had to be intriguing, and second, it should be possible to produce it in material afterwards. It was clear that we had to modify each model because the physical world has gravity, and the DL model does not take this into account. Some generated models were discarded because they were seen as not-fixable, although interesting in their shape.

The process demonstrated here is quite an unusual way to create an object. After extensive experimentation with the tool, we learned how certain words triggered certain shapes and colours. This knowledge gave us a chance to treat text prompts as poetic input. Thus, we created short poems to guide NN. The best ones survived as titles and are reflected in the forms.

The artists did not strictly follow the original model but took the creative liberty to modify the shape and determine the colour by manually glazing the sculptures. The dripping technique was used for colouring the sculptures. This served as a metaphor for liquid latent space and the psychedelic production process (this was the artists’ inner feeling about the creative process because they did not know what results would be achieved in the end). Sometimes, AI-generated vertex colouring was taken as inspiration, sometimes totally ignored. Nevertheless, digital sculptures were exhibited alongside the physical ones to underline the transformation and human role in the creative process. Although ceramic sculptures were 3D printed in clay, the fabrication process had to follow the traditional way of producing pottery (Figure 3). Since the artists had never engaged in ceramics before, the whole production process felt psychedelic: unexpected neural network processes led to transformation by numerical, physical, and chemical processes, all guided by both the artists and chance. Hence, the art project highlights the relationship between different agencies.

In the end, we can say that AI is not prepared for the physical world. It created nice images, but when one wants to materialise the output, it requires considerable additional work. However, those extra processes were very rewarding and creative in our case. In this project, AI served as an inspiration or a departing point more than anything else. In other words, the experimental phase of technology is necessary for experimental practices, and this can lead to the creation of a new production pipeline. The fine line between control and chance when guiding the neural networks and related processes is likely the main creative drive for the artists.

Figure 3. Ceramic sculpture guided by 3D object and text prompt, 3D printed in clay, and glazed manually. ©Varvara & Mar.

Discussion

According to the media hype around AI, this technology is intelligent enough to create art autonomously.³⁵, ³⁶ However, the reality is different. A computer scientist and a co-inventor of Siri Luc Julia, AI does not exist. He advocates for machines’ multiple intelligences that often outperform humans. However, machine intelligence is limited and discontinuous compared to human intelligence.³⁷ Therefore, it is vital to have artistic practices around this technology, as a counterbalance to the AI fantasies served by the industry and mass media.

We see AI as a creative tool with its own possibilities and limitations, which can stimulate artists’ creativity through unexpected outputs. Research has shown that tool-making expands human cognitive level and constitutes evolution in culture.³⁸, ³⁹  Similarly, as a new tool, generative AI could potentially enrich creativity by allowing new production pipelines that can create unique results.

Coming back to the synthetic images, we can say that all machine-created synthetic image-based works discussed here have particular aesthetics: both with DeepDream and GAN. Unlike the output of GANs, DeepDream has a more recognizable style and can be seen more as a filter that transforms every inputted image instead of learning from the given dataset. Regarding GAN aesthetics, such visual appearance is inherited from two entities to a large extent: the dataset and the model itself. GANs have a particular footprint, as seen in all works produced with this model. The visual palette comes from the used datasets. For example, if a dataset is homogeneous (only landscape images), then we will easily recognize landscapes in the generated output. However, if images in the dataset have a lot of visual variation, the output is rather abstract. POSTcard Landscapes from Lanzarote II illustrates this well. Also, when photos in the dataset look similar, the output will also be similar, as was the case with the Synthetic-scapes of Tartu video work where frames from recorded flaneur walks in a city were extracted. When we talk about video works generated with the neural net, then manual guidance of latent space offered more variations than an audio-led approach.

Synthetic image works have encouraged us to work with formats like images and videos that we did not engage in before in our art practice, but we found it exciting working with AI and video. For example, AI video generation has some affordances, like starting and ending can be done in a perfect loop since images are synthetically generated. However, creating real-time AI work is much more complex because some models are too slow. It might take a few minutes to render a single image. The limitations inspire us to devise new solutions and work in new mediums. Moreover, the limitations of the medium have always been a good challenge for our creativity.

Working with GANs or other image-generation tools has become much easier in recent years, although it used to be quite difficult. We must note that for practitioners, easy-to-use tools, such as DALL-E and Midjourney, offer little creative freedom, and thus, are less attractive to the artists. Those products tend to instrumentalize the user rather than the other way around. At the same time, open- source models offer more creative freedom and enable broader use of artistic ideas.

The work with generated text demonstrates that AI is not context-aware but maps concepts automatically without understanding semantics. More importantly, as shown in the ENA project the audience must also be guided alongside the AI. In the case of ENA, stage directions were used, and in the Dream Painter project, the concept of dream telling was applied to guide the participants who in turn guided the neural net through their interaction, creating a chain reaction. Navigating concepts in latent space is artistically interesting and inspiring, this was especially evident when working with form. The artists went beyond semantics and learned how to guide neural networks with a text prompt and 3D object.

The presented practice represents a paradigm shift in machine learning, moving away from composing datasets for GANs and toward translating semiotic spaces enabled by diffusion models. The evolution in practice shows how artists discover and learn to work with the DL toolset, embracing its possibilities and limitations. In the case of practice-based research, practice can be seen as a lab for testing artistic ideas with technology through chance until control is encountered.


(1) https://underdestruction.com/2017/02/04/alternative-face/

(2) https://www.memo.tv/works/all-watched-over-by-machines-of-loving-grace-deepdream-edition/

(3) https://www.nvidia.com/en-us/research/ai-art-gallery/artists/helena-sarin/

(4) https://twitter.com/NeuralBricolage/status/954027624728354821

(5) https://github.com/lucidrains/deep-daze

(6) https://github.com/lucidrains/big-sleep

(7) https://github.com/EleutherAI/vqgaclip

(8) https://var-mar.info/neuronal-landscapes/

(9) https://var-mar.info/plasticland/

(10) https://var-mar.info/postcard-landscapes-from-lanzarote/

(11) https://var-mar.info/phantom-landscapes-of-buenos-aires/

(12) https://var-mar.info/ena/

(13) https://var-mar.info/dream-painter/

Conclusions

In this article, we have summarised DL development from the perspective of artists’ interests concentrating on the image, video, text, 3D object generation, and translation models. We applied practice-based research methodology to investigate the role and possibilities of recent co-creative AI tools in artistic practice.

It is difficult to keep pace with AI development. In less than a decade, we have gone from blurry black-and-white faces to impressive high-resolution images guided by text prompts. The user level has gone from difficult to easy, which on one side, broadens possibilities for creation, but on another, it diminishes experimentation and creativity, since AI outputs seem ready-made. This is also demonstrated by the explorative nature of the body of work presented here.

Furthermore, it was noticed that creative AI, especially GAN models, have recognizable aesthetics, which, in the long run, become repetitive. This led to the change of tools by the artists. The curation of datasets, models, and outputs, along with neural network guidance, have become the toolset of an artist working with AI. Finally, these models can generate multitudes of outputs, but the art is giving the right input to guide the desired output and selecting the results that best serve the concept.

As Andy Warhol had envisioned in 1963, eventually, art production will become mechanised and automated. In his own words: “I want to be a machine”, which was also a reflection on that time's vast industrialization process. Resonating with today’s deep learning age: I want my machine to do art.

Acknowledgements

MSC is supported as a CUDAN research fellow and ERA Chair for Cultural Data Analytics, funded through the European Union’s Horizon 2020 research and innovation program (Grant No.810961).

References

1 Gabriel Nicholas, "These Stunning.in.toolsareaboutto change the art world (2017)”, Slate website, accessed November 20, 2022, https://slate.com/technology/2017/12/a-i-neural-photo-and-image-style-transfer-will-change-the-art-world.html.

2 Lev Manovich, Ai and myths of creativity, Architectural Design, 92(3):60–65, 2022, https://onlinelibrary.wiley.com/doi/abs/10.1002/ad.2814

3 Aaron Hertzmann, "Can Computers Create Art?(2018)” In Arts, volume 7, 18, MDPI, https://www.mdpi.com/2076-0752/7/2/18

4 Kevin Roose, “An A.I.-Generated Picture Won an Art Prize, Artists Aren’t Happy (2022)”, The New York Times website, accessed November 20, 2022, https://www.nytimes.com/2022/09/02/technology/ai-artificial-intelligence-artists.html

5 Aaron Hertzmann, Computers do not make art, people do. Commun, ACM, 63(5):45–48, apr 2020, ISSN 0001-0782, https://doi.org/10.1145/3347092

6 Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, "Imagenet classification with deep convolutional neural networks", Communications of the ACM 60, no. 6, 2017, 84-90, https://dl.acm.org/doi/10.1145/3065386

7 Mario Krenn, Lorenzo Buffoni, Bruno Coutinho, Sagi Eppel, Jacob Gates Foster, Andrew Gritsevskiy, Harlin Lee et al. "Predicting the Future of AI with AI: High-quality link prediction in an exponentially growing knowledge network", arXiv preprint arXiv:2210.00881, 2022.

8 Diederik P Kingma, Max Welling, “Auto-encoding variational bayes (2013)”, Arxiv, https://arxiv.org/abs/1312.6114 

9 Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, “Generative adversarial networks (2014)”, Arxiv, https://arxiv.org/abs/1406.2661

10 Alec Radford, Luke Metz, Soumith Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks (2015)”, arXiv. https://arxiv.org/pdf/1511.06434.pdf

11 Martin Arjovsky, Soumith Chintala, Léon Bottou, “Wasserstein gan (2017)”, Arxiv, https://arxiv.org/abs/1701.07875.

12 Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, “Image-to-image translation with conditional adversarial networks (2016)”, Arxiv, https://arxiv.org/abs/1611.07004

13 Alexander Mordvintsev, Christopher Olah, Mike Tyka, “Inceptionism going deeper into neural (2015)”, Google Blog website, accessed November 20, 2022, https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html

14 Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, “A neural algorithm of artistic style (2015)”, Arxiv, https://arxiv.org/abs/1508.06576

15 Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros, "Unpaired image-to-image translation using cycle-consistent adversarial networks. (2017)" In Proceedings of the IEEE international conference on computer vision, p.2223-2232.

16 Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, “Progressive growing of gans for improved quality, stability, and variation (2017)”, Arxiv. https://arxiv.org/abs/1710.10196

17  Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, Chen Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks (2018)”, Arxiv, https://arxiv.org/abs/1809.00219

18 Tero Karras, Samuli Laine, Timo Aila, “A style-based generator architecture for generative adversarial networks (2019)”, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4401–4410.

19 Kyle McDonald, "How to recognize fake AI-generated images (2018)”, Medium website, accessed November 20, 2022, https://kcimc.medium.com/how-to-recognize-fake-ai-generated-images-4d1f6f9a2842

20 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, “Attention is all you need (2017)”, Arxiv, https://arxiv.org/abs/1706.03762

21 Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al., “Language models are few-shot learners (2020)”, Arxiv, https://arxiv.org/abs/2005.14165

22 Hito Steyerl, “Twenty one art worlds a game map (2022)”, e-flux website, accessed November 20, 2022, https://www.e- flux.com/journal/121/423438/twenty-one-art-worlds-a-game-map/ 

23 Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever, “Generative pretraining from pixels (2020)”, In International Conference on Machine Learning.

24 Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani et al. “An image is worth 16x16 words: Transformers for image recognition at scale (2020)”, Arxiv. https://arxiv.org/abs/2010.11929

25 Jonathan Ho, Ajay Jain, and Pieter Abbeel, “Denoising diffusion probabilistic models (2020)”, Arxiv. https://arxiv.org/abs/2006.11239

26 Johannezz, “The promptist manifesto”, deeplearn.art website, accessed November 20, 2022, https://deeplearn.art/the-promptist-manifesto/

27 Jonas Larsen, "Geographies of tourist photography", Geographies of Communication: The Spatial Turn in Media Studies. Gothenburg: Nordicom, 2006, 241-257.

28 Varvara Guljajeva, Mar Canet Sola, “POSTcard Landscapes from Lanzarote (2022)”, In Creativity and Cognition (C&C '22), Association for Computing Machinery, New York, NY, USA, 634–636, https://doi.org/10.1145/3527927.3531191 

29 Varvara Guljajeva, Mar Canet Sola, “ENA: Participative Art Forms During Pandemics (2021)”, Ed. Livia Nolasco Rozsas and Borbala Kalman in HyMEx 2021 (Hybrid Museum Experience Symposium Proceedings), Budapest, 77-82.

30 Varvara Guljajeva, “Synthetic Books (2021)”. In 10th International Conference on Digital and Interactive Arts (ARTECH 2021), October 13–15, 2021, Aveiro,Portugal, Portugal. ACM, New York, NY, USA, 7, https://doi.org/10.1145/3483529.3483663

31 Mar Canet Sola, Varvara Guljajeva, “Dream Painter: Exploring creative possibilities of AI-aided speech-to-image synthesis in the interactive art context (2022)”, Proc. ACM Comput. Graph. Interact. Tech. 5, 4, Article 33, September 2022, 11, https://doi.org/10.1145/3533386

32 Varvara Guljajeva, Mar Canet Sola, “Dream Painter: An Interactive Art Installation Bridging Audience Interaction, Robotics, and Creative AI (2022)”, In Proceedings of the 30th ACM International Conference on Multimedia (MM '22), Association for Computing Machinery, New York, NY, USA, 7235–7236, https://doi.org/10.1145/3503161.3549976

33  Varvara Guljajeva, Mar Canet Sola, “AI-Aided Ceramic Sculptures: Bridging Deep Learning with Materiality (2023)”, In: Johnson, C., Rodríguez-Fernández, N., Rebelo, S.M. (eds) Artificial Intelligence in Music, Sound, Art and Design, EvoMUSART 2023, Lecture Notes in Computer Science, vol 13988, Springer, Cham, 2023, https://doi.org/10.1007/978-3-031-29956-8_23

34 Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka, “Text2mesh: Text-driven neural stylization for meshes (2022)”, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13492–13502. 

35 Sarah Perez, “Microsoft’s New Drawing Bot Is an AI Artist (2018)”, TechCruch website, accessed November 20, 2022, https://techcrunch.com/2018/01/18/microsofts-new-drawing-bot-is-an-a-i-artist/

36 Chris Vallance, "Art is dead Dude" - the rise of the AI artists stirs debate (2022)”, BBC website, accessed November 20, 2022, https://www.bbc.com/news/technology-62788725

37 Luc Julia, "There is no such thing as Artificial Intelligence", LP, 2020.

38 Dietrich Stout., “Stone toolmaking and the evolution of human culture and cognition (2011)”, Philosophical Transactions of the Royal Society B: Biological Sciences, 2011, 366(1567):1050– 1059.

39 Dietrich Stout, “Tales of a Stone Age Neuroscientist (2016)”, Scientific American website, accessed November 20, 2022, https://www.scientificamerican.com/article/tales-of-a-stone-age-neuroscientist/

Authors Biographies

Dr Varvara Guljajeva is an artist and researcher holding the position of Assistant Professor in Computational Media and Arts at the Hong Kong University of Science and Technology (Guangzhou).

Mar Canet Sola is an artist and researcher. Mar is a PhD candidate and research fellow at Cudan research group in BFM Tallinn University. Varvara and Mar form an artist duo Varvara & Mar since 2009.

Isaac Joseph Clarke is a PhD student in Computational Media and Arts at HKUST(GZ) investigating AI tools for artists. 

Kombucha as a Guide. Serendipitous Journey through Taste, Feminism, Free and Open Source Culture, and Ritual

Lucile Haute

University of Nîmes (PROJEKT)
& École des Arts Décoratifs – PSL (EnsadLab)
Nîmes & Paris, France
lucile.haute@ensad.fr

Abstract

During one and a half years living with kombucha, the author followed it across a journey that diverged from a disciplinary point of view in favor of a “global design” (Papanek) approach that embraces co-dependencies (Haraway, Tsing). This journey spanned several domains: gastronomy and food, health, textile design, social practices. It was the occasion to find out how different cultures might be embodied by the heterogeneous kombucha community and its various locations: from bio-hack lab to the kitchen, art gallery, design school, and brewery lab. What may this Symbiotic Culture Of Bacteria and Yeast teach us—humans—about our ways of collaborating with, cultivating, exploiting or caring for the living beings we eat and/or use to make everyday objects? Could kombucha SCOBY provide a guide to shifting ways of understanding and performing our way of life, or to phrase it another way: to initiate a cultural revival?

Keywords

Companion Species, DIWO (Do It With Others), Design (critical, fictional, prospective, textile), Kombucha, Performance art, Ritual, SCOBY.

DOI

10.69564/ISEA2023-63-full-Haute-Kombucha-as-a-Guide

Introduction

How can we become aware of, meet, and build more respectful relationships with the life forms that surround us? Is there a link between human-made political forms and other-living-being-made living forms? How can we draw inspiration from the observation and understanding of life forms to imagine political forms alternative from exploitation and domination? How can we encourage these kinds of observations? These heterogeneous questions and concerns often appear in interactions with fermenting communities. They are encountered in relation to the question of the Anthropocene and from different fields: philosophy, arts, design.

In a very much grounded perspective, I approached these questions over the past year and a half by concretely engaging on a serendipitous journey with a symbiotic life form: kombucha.

Figure 1. One of the oldest Western kombucha representation: painting of Philipp Kubarev (Morning, 1918). Graphic: the author. 

This paper aims to chronicle this journey from a situated perspective, and position it as part of a larger tapestry of thoughts and reflections. The author is a white cis-woman living and working in two cities in France, with a rural and feminist background linked to her grandmother’s farm and a childhood in a village that passed on a culture of DIY canned food from the garden.

The various fields concerned by kombucha are concrete places from which to address previous questions —ones that, otherwise, would remain theoretical and unembodied. These domains are culinary design as it relates to fermentation, textile design, prospective and fictional projects, and art. How do contemporary artists and designers reactivate issues inherited from ecology, feminism or mesology?

The goal is to embrace a holistic view of our surroundings, objects and food, to adopt a global design perspective, i.e., following Papanek: taking all stages into account, from production or the extraction of raw materials to the processing of waste, through transformations and uses, all in iterative cycles. How might kombucha be involved in these initiatives?

This paper relies on two types of primary sources. The first are formal interviews with three actors: a kombucha brewer based in Paris, a French-Icelandic artist and designer, and an American designer. The second material is the result of a year and a half of solo and collective experimentations, meeting and living with kombucha, that provided the opportunity for various informal discussions. Among these, I organized a two-week long design workshop to which were invited: an artist, two textile designers, a maker, a kombucha brewer, and an engineer. ¹  This generous format was very fertile both in terms of discussions and formal experiments. Thereafter, artistic events facilitated the collection of stories from the public. ² Other sources are published magazines, books, articles or even online posts.

Figure 2. Wikipedia screenshot on kombucha cellulose and beverage. Picture: the author.

Microorganisms and Biotransformation through Interdisciplinary Prism

Kombucha

The average knowledge about Kombucha is that it is a Symbiotic Culture Of Bacteria and Yeast. Yeasts transform sugar into alcohol; bacteria transform alcohol into acid and produces a protective biofilm on the surface. Each one produces the organic substance the other symbiote profits from. The acidification of the solution protects the culture by preventing the growth of undesirable yeasts and bacteria. The beverage is drunk according to an ancestral tradition in Asia, and more recently in the West where its growing mainstream popularity is fairly recent. ³

Figure 3. SCOBY diagram by the author.

In a variety of senses, Kombucha’s popularity has relied upon a recent fermentation’s fame. In the intellectual world, “fermentation” is a fertile concept summoned as a political and philosophical statement linked to feminism, epistemology, and ecology. A representative example is Lauren Fournier’s project ⁴ and essay.⁵ Fed by the thought of Haraway and Tsing—and they themselves by Margulis—, Fournier associate fermentation and feminism as reciprocal metaphors that we shall deploy hereinafter.

Biotech Perspective

From a bio-engineering perspective, the study of traditional fermented food may reach some molecule of interest for health, food or cosmetic applications. A renewed interest in the fermentation field was explicit, for example, at a conference I attended on February 23, 2022, at Polytech Lille, organized by the Cluster Nutrition Health Longevity—whose motto is “Where The Health and Food Sectors Converge.” ⁶  This conference gathered entrepreneurs, engineers, public and private researchers in agricultural, under the title “Fermentation: applications, new markets & health potential.” Among the seven guests, two kombucha brewers were programmed. Beyond that pleasant surprise, my interest was particularly picked by the intervention of Gnosis by Lesaffre’s Global Marketing Director.

This brand is the result of the acquisition of Gnosis, the specialist in fermentation products, by Lesaffre, the world's leading yeast company in 2015. ⁷ The brand statement positions it in the field of health nutrition: “We harness the power of microorganisms and biotransformation processes such as fermentation to create nutritional actives, probiotics, and nutritional and functional yeasts that benefit human health and wellbeing.” ⁸ In September 2022, they joined the French Gut Initiative, a research project about microbiota. ⁹

Their business model is mainly B2B for the food and nutraceutical industries, however, they also offer a range of yeast products for domestic use. During COVID-19 lockdown, they observed an increasing demand on these specific products. After the lockdown, the domestic yeast demand went down but remains above its prior level. They therefore began to developing local yeast variations related to the food cultures of different markets and countries. This anecdote reveals an industrial yeast producer’s interest in local cultures, which is paradoxical given that it's precisely the industrialization of food production methods (homogenization, centralization and scaling up) that has removed fermentation from individual plates.

Taste Perspective

On a smaller scale but no less demanding, let’s mention two noticeable cookbooks that participated to the spread of fermentation culture. Qualified as a “bible for the DIY set” by The New York Times, The Art of Fermentation. An In-Depth Exploration of Essential Concepts and Processes from Around the World. With Practical Information on Fermenting Vegetables, Fruits, Grains, Milk, Meats, and More. from Sandor Ellix Katz was published in 2012. Claiming the polysemy of the word culture, Katz states that “Reclaiming our food and our participation in cultivation is a means of cultural revival, taking action to break out of the confining and infantilizing dependency of the role of consumer (user), and taking back our dignity and power by becoming producers and creators”. ¹⁰ 

This book has established itself and remains a major reference for DIY and homemade initiatives, with a theoretical perspective and health focus. Loren Fournier often refers to it in her paper. So do other artists working with fermentation such as Maya Minder.

In the world of gastronomy, fermentation was rehabilitated on the chef’s table by the triple Michelin starred restaurant Noma, located in Copenhagen (Denmark). Eight years after the publication of a beautiful and expensive book for æsthetes in 2010 ¹¹, a cheaper and more pragmatic one marked the culture: The Noma Guide to Fermentation. ¹² Suddenly, it was everywhere: from biohackers lab to tables d’hôte kitchens, the bookshelves of wild fermentation amateurs and independent brewery labs.

Where Katz focused on food and politics and health, considering all three simultaneously, Noma mentioned each recipe’s traditional and historical backgrounds but quickly shifted its focus to taste and How to?, starting with building a fermentation chamber. Nevertheless, Katz’s approach may also apply to describe Noma’s book: “Empower [the reader] with tools so [they] can explore and reclaim fermentation into [their] life”.

This quick food publication review wouldn’t be complete without a non-cookbook: an artists-led publication about the human food system aiming to present research and art about food and technology from an open culture perspective. The third issue of Food Phearking was published by the Center for Genomic Gastronomy in 2016 and themed around Gut Gardening. ¹³ The issue presents the 15 most famous bacteria from human microbiome followed by essays from artists and researchers. It invites the reader to consider becoming Gut Gardener ¹⁴ and joining ‘enterogastronomy’ movement—eating with the gut in mind.¹⁵

Eating is Political 

Behind the Food

“Every living creature on this Earth interacts intimately with its environment via its food” says Katz. Various artists and designers explore the social potential of eating together in a performance perspective. They use the sensual or awkward intimacy inherent to eating or drinking, the different aspects of the collective practice of. It can be festive or ceremonial, joyful then dramatic, depending on the ritual’s intent. Eating together may invite to a meditation and allow awareness.

Some artists address the political aspects of food and try to repair some lost connection or understanding of where the food comes from. Tunde Wey’s performances ¹⁶ and Cooking Sections (Daniel Fernández Pascual & Alon Schwabe)’s work ¹⁷  come to mind.

Some embrace the ritual and sacrificial weight of food, underscoring its association with both life and death. As Haraway wrote: “There is no way to eat and not kill, no way and not to become with other mortal beings to whom we are accountable, no way to pretend innocence and transcendence or final peace. [...] Multispecies human and nonhuman ways of living and dying are at stake in practices of eating.” ¹⁸

One collective and ritualistic moment embracing the sacrificial weight of eating is The Butcher proposed by Atelier Van Lieshout at la Friche de la Belle de Mai in Marseille (FR) on September 15, 2013. This spectacular diner held inside the eponymous exhibition proposed a meal around and from a cow. The press kit mentioned three interesting aspects. First, the qualification as “gustatory conviviality” connotes an æstetic social moment related to gastronomy. Second, the mention of “reactivation of ancestral rituals” implies an anthropological heritage and collective feeding as a symbolically charged and structured moment. Third, the objective of “consumption without waste” embraces the contemporary issues around over-production and concerns around human activities’ impact.¹⁹

Drinking Commercial Kombucha

Back to kombucha, the political impact of drinking this beverage may not seem obvious, especially when it isn’t homemade but commercially produced. The latter trend in kombucha fashion was born in California and is marketed towards health concerns and self-care habits. 

Figure 4. Two bottles of kombucha soda from the French brand Vivant Kombucha. Picture: the author.

Based in Paris since 2018, the brand Vivant Kombucha has adopted a marketing positioning in which taste and process are underscored. It is intentional, inherited from natural wine culture (which inspired the fermentation technic) and brewing culture (which inspired the packaging). The project was to create a non-alcoholic beverage with a taste complexity that related to wild fermentation and the natural wine world.²⁰

The company doesn’t place its ethical structural choices on display: small local business, community-grounded, fair salaries, carbon footprint limitation with cargo-bike delivery, bottles consignment for professional customers, by-products reclamation (infused tea and cellulose), care for the livingness of the product (neither filter nor pasteurization are involved in the stabilization process, which relies only upon cold). It remains aware of the limits of its ethical commitment: tea and cane sugar are fair-trade but travel long distances —experiments with beet sugar, which is produced in France, were unsatisfactory, due to fructose peaks during the fermentation process, with undesirable effects on taste, Mial Wattkins explained.

In rationalizing his commitment to kombucha, he refers to Jonathan Nossiter’s book Cultural Insurrection, ²¹ summarized as follows: contracultural arts are dead, gangrend by the capitalist system. “The only thing that you can do now, it’s to go to the countryside, put your hands into the earth and make something [...]. Build a microeconomy around a very specific thing. [...] It’s the story of natural wine and artisanal beer. [...] We applied this way of thinking to kombucha.” Vivant Kombucha is fueled by ethical concerns and the will to contribute to a common culture while running a business.
Does this business model’s affiliation with the capitalist system defuse its insurrectionist potential? Feminists may say so, since “The Master’s Tools Will Never Dismantle The Master’s House”, as says the famed Audre Lorde title. ²² However, one may argue that acting, motivated by ethical concern, and armed with a clear and precise understanding, allows for a departure from the theoretical and an embrace of tangible trial.

Fermenting Feminism

Echo to Ecofeminism

As previously mentioned, the social eating moment, articulated via ritualistic forms involving fermentation, may be a means to address contemporary concerns regarding the impact of human activities on the planet. These are an invitation to explore a specific tradition, one deeply committed to countering mortiferous human activities against living beings, through the employment of artistic and spiritual forms (such as ritual, dance, fiction, meditation...).

The common thesis of ecofeminist theories, summarizes Jeanne Goutal, is that “all domination forms (sexism, racism, specism, colonialism, imperialism, anthro-pocentrism...) are inseparably and systemically linked.” ²³ Ecofeminists commit to “the refusal of any form of domination,” between humans as well as between species, genders, or regions, and the fact that “systemic change needs a psychic, moral, intellectual and spiritual mutation [...] Ecofeminists refuse the ideal of a freedom without awareness of natural limits.” ²⁴ This family of thought and action is embodied by contemporary artists and designers addressing topics such as the renewal of interspecies relationship through forms related to ritual. 

Building a Critical Space

From 2012 to 2016, Lauren Fournier curated a series of events and exhibitions entitled fermenting feminism. It brought together artists to explore the question of “what it means to bring fermentation and feminism into the same critical space?” ²⁵ She developed a reciprocal metaphor between fermentation and feminism, that she honed both from a theoretical perspective—with an academic paper—and in practice—with the collective events and exhibitions. She and the participants “approach fermentation through intersectional and trans-inclusive feminist frameworks” and “approach feminisms through the metaphor and material practice of fermentation”: “Fermentation as a process of transformation becomes both a metaphor and a material practice through which to explore important issues [...] from the politics of labour, affect, survival, and care to colonialism, food, indigeneity, and the land.” ²⁶

Through these art exhibitions and events, she affirms a statement that is present in the foreword of the catalog and further developed in a paper: fermentation is a generative and speculative metaphor and, simultaneously, a material practice, a microbiological process that embodies and performs a theoretical positioning.

She elaborates this idea into ten proposals: “fermentation is political; fermentation is vitalism; fermentation is accessibility; fermentation is preservation and transformation; fermentation is interspecies symbiosis and coevolution; fermentation is survival and futurity; fermentation is care of the self and care of others; fermentation is harm reduction; fermentation is queer time; fermentation is collaboration.” ²⁷ Cross reading of the paper and the catalog allows one to associate the artist’s work to one of several of those.

Prospective Design,
Fiction & Ritual

Building communities around the table

Crafters and makers concerned on how to do things themselves reclaim self-nourishments and experiment in making processes that participate in an empowerment process. These may turn away from the commercial model, experimenting on other ways to exchange services and goods. Communities can rally around thought or interest, with or without devoted spaces; these can be ephemeral, or provoked by a situation. This is the goal of artists who gather spontaneous communities around fermented food sharing, congregating at the table around the living being that feeds us, and inviting an audience to raise awareness.

From Design to Performance and Cooking

Elin Margot, a French-Islandic artist and designer, often uses food-sharing moments as a startingpointandfinal form to address critical issues. She organized several debate-dinners, inviting people from a common area with divergent points of view around a fictional menu contextualized in a scenario. For those rituals —that may also be called participative performances—, when any furniture or accessory is needed, she relies upon her training in product design, using wood and ceramic. The sequencing of the event draws upon her service design skills. Her method to build it all relies upon critical and speculative design. ²⁸

In reference to another artist’s performances, she asks, in a fictional context: what would a dinner consist of in a society that refuses to kill animals and eats only that which some of them naturally shed—such as salamander tail? As such, she rethinks the place of meat.

She recently started to learn cooking. Prior, she faced challenging collaborations with chefs, requesting that they abandon the quest of making good food and focus instead on the way to do it—regardless of taste. Aware of vegan thinking that denies death, claiming inspiration from ecofeminism and hydrofeminism, she aims to provoke a discussion, while eating together, about the human relationship with the living beings on which we feed. She creates scenarios such as a kombucha adoption for interspecies mothering—that she performed, teaching how to care and maintain the SCOBY , beverage.

From Cooking to Codesign

The artist Maya Minder, a member of the bio-hack lab Hackteria and of the BadLab collective, has followed a symmetrical path: from cooking to objects. Rooted into food practices and social constructs around food, she proposes collective sensual experiences that engage people and their microbiome through macrobiotic, fermented and vegan buffets. She invites to develop an awareness of natural processes, those permanent and invisible movements.

She recently conceived the Green Open Food Evolution, a culinary installation involving a speculative cooking furniture co-conceived with designers. She explains: “The idea was to create cooking that can be activated. We made racks to display the tools of speculative cooking, as well as algae and various other objects. Even the table is like a topographical landscape, non-uniform, that tells a story.” ²⁹ The installation was activated in public for the exhibition opening.

Figure 5. Measuring the thickness of home-grown kombucha bac- terial cellulose before drying. Picture: the author.

From Design to Ritual

Joanne Jones was trained in Industrial Design and Environmental Sustainability, then, she steered clear of standard industrial design, refusing plastic and other straight-to-the-trash items, which were unaligned with her values, she says. ³⁰ Joining the Nature-Inspired Design program at ENSCI in Paris, she sought to work with an organism and began growing kombucha leather. It is not easy to bond with faceless organism that becomes a living material. How do you honor the agency of those organisms? she asks. It became more of a philosophical question, almost spiritual as well.

She ended up making an entire prospective design fiction about a kombucha community that works with the organism and develops rituals with the material. Her concern was to provoke questioning: how does one work with living organisms and living materials in respectful ways? How does one escape the exploitation pattern? The fictional community would have to build relationships with the microorganisms and with each other, these would be achieved through ritualistic practices based on the solstice and equinox.

Reinventing Addressing Forms

Following kombucha, I met artists with design backgrounds, an interest in food, ecological and/or political awareness, makers of ritualistic forms as social sculpture, inspired by ecofeminist fictional tactics, and critical, and speculative design methods. This mixture of styles relates to kombucha itself: its plurality of micro-population and the plurality of the ways humans may look at it: as a tasty and/or healthy beverage, as a biomaterial with uses still to be explored or stabilized or as a metaphoric form to rethink human relationship to living being in a more harmonious way of life, as a guide from how learn a greater resilience.

Conclusion: Let’s Explore Spiritual Aspects

During this year and a half of meetings and interviews with designers, artists, brewers and engineers, stories and analyses about kombucha were gathered. However, the relationship with beliefs, superstitions and magical rituals is not sufficiently explored. I found traces of kombucha’s involvement in divinatory practices. This needs to be continued, not in a religious or dogmatic aim (searching for ways to build beliefs and manipulate people) but, on the contrary, to develop a critical approach to our Western lifestyle (and diet), looking for clues of the cultural revival we absolutely need in the contemporary context. This intuition follows the common assumption of speculative design and ecofeminism: the power of fiction to change the world by changing our imaginaries deserves to be investigated.

Figure 6. Dry kombucha cellulose sample with various of colors and thickness. Samples & picture: the author.

Acknowledgements

My introduction to and subsequent coexistence with kombucha are the result of several generous interpersonal exchanges. Fermentation is a network. I want to thank warmly: Felixe Kazi-Tani for bringing fermented living beings (kimchi, kefir and homemade sourdough) into my daily life and for the political reflexive talks about it, as it relates to feminist and queer culture; Maya Minder and Alexia Venot for bringing kombucha into my life, especially Maya who taught me how to care for them; Mial Watkins from Vivant Kombucha brewery for our generous talks and for donating cellulose; for sharing their experience about their interactions with kombucha: Joanne Jones, Siddhant Malviya, Elin Margot, Loraine Fournier, Jeanne Mainneti (also for being a fantastic partner in kombucha rituals); Vivien Roussel (also for opening thr34d5 gates and for ongoing discussion since); Pascal Dhulster for challenging talks about by-product and co-product notions and engineering perspective; Justin Monteiro who was my trainee in January 2022; the University of Nîmes (especially Mr P.) for tolerating the different smells of this experimentation; special thanks to my colleague Karen Brunel for careful proofreading. 

References

1 Biomaterials, Fermentation and Kombucha, workshop from January 17th to 27th 2022 for Bachelor Design students proposed by Lucile Haute with Pascal Dhulster (professor in process engineering, Lille / Nîmes FR), Corinna Mattner (designer, Zurich CH), Maya Minder (artiste, Zurich CH), Vivien Roussel (maker, Paris FR), Alexia Venot (designer, Paris FR), Mial Watkins (brewer, Paris FR). See the report published by one of the stakeholders: Maya Minder, “Biomaterials, Fermentation and Kombucha at Nimes University” (Makery, Feb. 2022) https://www.makery.info/en/2022/02/16/biomateriaux-fermentation-et-kombucha-a-luniversite-de-nimes/

2 These artistic contexts where two performances proposed for the opening (Installer le cercle — opening performance with Jeanne Mainetti, Laura Ego, Mounia Ali, Houria Ali, Éloïse Komara, Dande Keimba-Waza on May 14th 2022) and the closing (Départ — performance with reading, gong and kombucha adoption on June 4th 2022) of my solo exhibition Cyberwitches Coven in the art space Les Limbes in Saint-Étienne (France) during the International Design Biennale, from May 14th to June 4, 2022. www.biennale-design.com/saint-etienne/2022/fr/a/le-cercle-des-cybersorcieres-cyberwitches-coven-1471

3 Sources: Wikipedia pages for “kombucha” in French and in English.

4 Lauren Fournier (ed.), fermenting feminismn, Laboratory for Aesthetics and Ecology, 2017.

5 Lauren Fournier, “Fermenting Feminism as Methodology and Metaphor. Approaching Transnational Feminist Practices through Microbial Transformation”, Environmental Humanities 12:1, May 2020, 88-112. DOI 10.1215/22011919-8142220

6 Quote from the Clubster NHL’s website: clubster-nsl.com

7 Olivier Ducuing, “Lesaffre achète le groupe italien Gnosis” (Les Echos, 2 nov. 2015). www.lesechos.fr/2015/11/lesaffre-achete-le-groupe-italien-gnosis-256620

8 Quote from the brand’s website: www.lesaffre.com/healthcare/gnosis/

9 Press release is published on Biotech Info https://biotechinfo.fr/article/gnosis-by-lesaffre-rejoint-linitiative-le-french-gut/

10 Sandor Ellix Katz, The Art of Fermentation. An In-Depth Exploration of Essential Concepts and Processes from Around the World (Chelsea Green Publishing Co, 2012), 19. Fermentations!, Mens, Terre vivante, 2018, 19.

11 Rene Redzepi, Noma: Time and Place in Nordic Cuisine, Phaidon Press, 2010.

12 David Zilber and Rene Redzepi, The Noma Guide to Fermentation, Artisan Publishers, 2018, Published the same year in French: Le guide de la fermentation du Noma (Chêne, 2018).

13 Food Phearking, Issue 03: Gut Gardening, Center for Genomic Gastronomy, 2016.

14 Ibid., 4.

15 The term is from Nicolas Twilley. Ibid., 7.

16 Tunde Wey interviewed by Léopold Lambert, “Dinner as Demonstartion Cooking as Underlying Food as Reparation”, in The Funambulist, Issue 31, Politics of Food, 2020. This issue examines “various dimensions behind food and the act of cooking: labor, ingredients, memories, identity, self-orientalization, marooning, infrastructure, transportation, societal hierarchies andspace” and is available online in open access: thefunambulist.net/magazine/politics-of-food

17; See: Cooking Sections (Daniel Fernández Pascual and Alon Schwabe), The Empire Remains Shop (Columbia Books on Architecture and the City, 2018); and: empireremains.net

18 Donna Haraway, When Species Meet, Minneapolis, the University of Minnesota Press, 2008, 295.

19 Press kit of the event.

20 All the information about Vivant Kombucha and the quote are from the interview with Mial Watkins, co-founder, in January 2022.

21 Jonathan Nossiter, Cultural Insurection, Other Press LLC, United States, 2019, First published in French, Stock, 2015.

22 Audre Lorde, “The Master’s Tools Will Never Dismantle The Master’s House”, in Cherrie Moraga and Gloria Anzaldùa (eds), This Bridge Called My Back: Writings by Radical Women of Color, New York, Kitchen Table Press, 1983, 94-101.

23 Jeanne Burgart Goutal, Étre écoféministe. Théories et pratiques, Paris: L’échappée, 2020, 71. Author’s translation.

24 Ibid., 73.

25 Lauren Fournier, foreword of fermenting feminism (op. cit), 3.

26 Ibid.

27 Lauren Fournier, “Fermenting Feminism as Methodology and Metaphor” (op.cit.), 89.

28 Source: interview with Elin Margot, February 2022.

29 The Green Open Food Evolution cooking furniture is commissioned by Maya Minder and designed by Victor Yvin and Pacôme Gérard (Designers Artisans), Gabriel Violleau (Bientôt) and Ewen Chardronnet (Makery). Source: Pauline Briand, “Green Open Food Evolution, a speculative exploration of algae and the food transition” (Makery, Oct. 2022), online: https://www.makery.info/en/2022/10/20/green-open-food-evolution-une-exploration-speculative-sur-les-algues-et-la-transition-alimentaire. Source: interview with Johan Jones, January 2022.

Bibliography

Hubert Antolak, Dominik Piechota, Aleksandra Kucharska, “Kombucha Tea—A Double Power of Bioactive Compounds from Tea and Symbiotic Culture of Bacteria and Yeasts (SCOBY)”, Antioxidants Volume 10 Issue 10: Advances in Natural Antioxidants for Food Improvement, 2021, doi.org/10.3390/antiox10101541

Pauline Briand, “Green Open Food Evolution, a speculative exploration of algae and the food transition", Makery, Oct. 2022, online: https://www.makery.info/en/2022/10/20/green-open-food-evolution-une-exploration-speculative-sur-les-algues-et-la-transition-alimentaire

Jeanne Burgart Goutal, Étre écoféministe, Théories et pratiques, Paris, L’échappée, 2020.

Cooking Sections (Daniel Fernández Pascual and Alon Schwabe), The Empire Remains Shop, Columbia Books on Architecture and the City, 2018.

Center for Genomic Gastronomy, Food Phearking: Gut Gardening, Issue 03, 2016, foodphreaking.com/issue03pdf/FoodPhreak-ing_03.pdf

Lauren Fournier (ed.), Fermenting Feminism, Berlin and Copenhagen, Laboratory for Aesthetics and Ecology, 2017.

“Fermenting Feminism as Methodology and Metaphor. Approaching Transnational Feminist Practices through Microbial Transformation”, Environmental Humanities 12:1, May 2020, 88-112, doi.org/10.1215/22011919-8142220

The Funambulist, Issue 31, Politics of Food, 2020, Open access available: https://thefunambulist.net/magazine/politics-of-food

Donna Haraway, The Companion Species Manifesto. Dogs, People, and Significant Otherness, Prickly Paradigm Press, 2003.

Donna Haraway, Manifeste des espèces compagnes. Chiens, humains et autres partenaires, Paris, Climats, 2019.

Donna Haraway, Staying with the Trouble. Making Kin in the Chthulucene, Duke University Press, 2016. 

Donna Haraway, Vivre avec le trouble, Paris, Les éditions des mondes à faire, 2020.

Donna Haraway, “Anthropocene, Capitalocene, Plantationocene, Chthulucene: Making Kin”, Environmental Humanities, vol.6, 2015, p.159-165, doi.org/10.1215/22011919-3615934.

Donna Haraway, “Anthropocène, Capitalocène, Plantationocène, Chthulucène. Faire des parents”, Multitudes 2016/4 n° 65, 75-81, doi.org/10.3917/mult.065.0075

Sandor Ellix Katz, The Art of Fermentation, Chelsea Green Publishing Co, 2013.

Sandor Ellix Katz, Fermentations !, Mens, Terre vivante, 2018.

Bruno Latour, Face à Gaïa, Huit conférences sur le Nouveau Régime Climatique, Paris, La Découverte, 2015.

Audre Lorde, “The Master’s Tools Will Never Dismantle The Master’s House”, in Cherrie Moraga and Gloria Anzaldùa (eds), This Bridge Called My Back: Writings by Radical Women of Color, New York, Kitchen Table Press, 1983, 94-101. doi.org/10.1515/9781474470254-003

Carolyn Merchant, “The Scientific Revolution and The Death of Nature”, 2006.

Carolyn Merchant, Micro cultures, Leuven, 2012, bbva.irational.org/microcultures/

Maya Minder, “Biomaterials, Fermentation and Kombucha at Nimes University", Makery, Feb. 2022, https://www.makery.info/en/2022/02/16/biomateriaux-fermentation-et-kombucha-a-luniversite-de-nimes/

MOLD Magazine Issue 01: Designing for the Human Microbiome, 2017.

Jonathan Nossiter, Olivier Beuvelet, Cultural Insurrection: A Manifesto for Arts, Agriculture, and Natural Wine, Other Press, 2019. 

Jonathan Nossiter, Olivier Beuvelet, Insurrection culturelle, Paris, Stock, 2015.

Victor Papanek, Design for a Real World, Pantheon, 1971.

Victor Papanek, Design pour un monde réel, Dijon, les presses du réel, 2021.

René Redzepi, David Zilber, The Noma Guide to Fermentation, Artisan Publishers, 2018. 

Le guide de la fermentation du Noma, Paris, Éditions du Chêne, 2018.

Anna L. Tsing, “Unruly Edges: Mushrooms as Companion Species”, environment and society, 2012, www.environmentand-society.org/mml/unruly-edges-mushrooms-companion-species

David Zilber, “Pathways for Waste. Beyond Fermentation at Noma’s R+D Lab”, MOLD Magazine 03: Waste, 2018, 12-17.

Author Biography

Lucile Olympe Haute is an artist, researcher and educator. Her research brings together spirituality, technologies and politics — understood in the broader sense of a committed and eco-responsible "living together" that goes beyond anthropocentrism and includes the plant, animal and fungal kingdoms, even the communities of bacteria in our biotopes. Synthesizing these issues, she wrote the Cyberwitches Manifesto to bring together witches from different backgrounds, politically committed (ecofeminism, queer/transfeminism, technological emancipation) and involved into empowerment processes. Her work often manifests through the aesthetic and narrative versatility of web-to-print design tools. She is lecturer in art & design at the University of Nîmes (FR) and associate artist at École des Arts Décoratifs  PSL (FR). lucilehaute.fr | projekt.unimes.fr | ensadlab.fr 

Towards an Intelligent IoT System for the Data-Informed Museum of the Future

Yannick Hofmann

intelligent.museum (ZKM | Center for Art and Media Karlsruhe & Deutsches Museum Nürnberg) Karlsruhe, Germany
contact@yannickhofmann.de

Abstract

This article describes the ongoing prototypical development of an intelligent IoT system for art and cultural institutions as part of the intelligent.museum project. Partial development steps that have already taken place are described and the system is conceptually framed. The system will be used to condense data from various IoT sensors in the exhibition space and the building complex, along with data from the Internet, into intelligent data analytics. The goal is to provide human decision-makers with intelligent analytics as well as machine recommendations for action to enhance museum intelligence. This initiative, which can be tracked transparently and comprehensively via a code repository on the Internet, will enable art and cultural institutions to adapt museum experiences to the needs of visitors. In addition, a technological framework is created for artists to create computer-based works and develop interactive art experiences using this system.

Keywords

Museums, Augmented Intelligence, Artificial Intelligence, Internet of Things, Data Mining, Interactive Art, Generative Art.

DOI

10.69564/ISEA2023-64-full-Hofmann-IoT-System

Introduction

The intelligent.museum project aims to make the museum experience of the future more inclusive and accessible by developing technical solutions and rapid prototypes for human–computer interaction between the museum and exhibition visitors. From new conversational user interfaces that mimic a conversation with a real human (e.g., chatbots) to machine vision systems that enable, for example, the recognition and tracking of both objects and people, facial expressions, gestures, or postures, AI models can be deployed today that are particularly well-suited for processing the auditory, visual, and textual input of visitors. These in turn can be evaluated using AI, whereupon the museum experience can be tailored to the specific needs of individual visitors.

In order to combine live data from the museum with data from the Internet within an intelligent IoT system and make it the subject of AI-supported data analyses, various prototype developments are currently taking place within the framework of the project. These developments concern the standardization of procedures and approaches in the field of integrating sensors into an open-source IoT platform, the development of a scalable LiDAR-based visitor tracking system for the exhibition space as well as the implementation of machine learning tools for data analysis.

In addition to the partial automation of system-irrelevant processes by means of actuators reacting to sensor inputs, the long-term goal of the project is to create a system of augmented museum intelligence that is intended to expand the intelligence of human operators as well as their basis for decision-making by means of data-driven analyses and suggestions for action.

This can be best described as a system of augmented intelligence.1 "Augmented intelligence is a design pattern for a human-centered partnership model of people and artificial intelligence (AI) working together to enhance cognitive performance, including learning, decision making and new experiences." ² A combination of data science, machine learning and human intelligence, augmented intelligence pairs computer artificial intelligence with human intelligence, aiming to improve human decision-making capabilities. In this process, AI tools evaluate Big Data, for example, large amounts of data that can hardly be grasped by humans, in order to subsequently provide results and data analyses to human operators. In such a scenario, an AI system provides humans with a basis for decision-making so that they can make decisions even faster and more precisely. The goal is to support human decision-making processes, not to replace humans with AI.

The overall goal of this development is to improve the exhibition experience for visitors and to optimize museum operations. Used as a technical analysis tool, new potentials for visitor research should also emerge. Media artists who create interactive installations, generative visualizations or sound recordings, for example, can integrate the live data into their artistic works on the one hand, but also provide data they have collected themselves on the other. The intelligent IoT system is emerging as a tool for data-informed optimization of the museum experience and is intended as a research contribution to the potential uses of AI in museums.

The museum is gradually forming a sensory perceptual apparatus, developing cognitive capabilities and thereby becoming a cognitive system itself in the long term.³ 

Visitor interaction with the museum of the future

Through innovative technical interfaces, the museum of the future will be able to react to the needs of visitors and adapt itself accordingly.

In the intelligent.museum project, experiments were conducted with the use of Deep Learning in connection with linguistic user interfaces. In the field of speech-based interaction with machines, rapid development steps have been taking place for years, whereby speech has increasingly become a natural and self-evident way of interacting with devices such as Google Home, Amazon Alexa or the Siri software on Apple devices. In everyday life, then, these voice-based systems are already in frequent use. Deep Learning is being used here as well.

Example As part of the exhibition BioMedien at the ZKM, the prototype of an AI-supported museum label for the exhibition space was presented, which was developed as part of the intelligent.museum project. This prototype can recognize the national language from the spoken language of visitors and, as a result, automatically translate given texts into the corresponding language. For the training of the speech recognition, audio samples of the speech corpus Common Voice by Mozilla were used. In addition, to make the model more robust to noise, Google's dataset AudioSet was used. This dataset contains manually annotated ten-second audio recordings from the online video platform YouTube.

To find out how much data is needed to distinguish between two similar languages, an experiment was conducted in which the neural network was challenged to distinguish English from German. This experiment found that 30,000 audio samples did not provide enough data for the model used, despite data augmentation. At 5 seconds per audio sample, that's over 40 hours of speech data. For most languages, 40 hours of training data is not yet available in the Common Voice training dataset used. Thus, in order to train the system on many languages in the long run and thereby make the museum experience more accessible and inclusive for visitors of different nationalities, the availability of open-source international speech data needs to be increased. In the medium term, languages from the global South in particular need to be more strongly reflected in relevant language datasets. ⁴ With this motivation in mind, the Artificially Correct Hackathon, hosted by the Goethe-Institut in partnership with the ZKM from October 1-3, 2021, issued a challenge to develop a non-contact capture station for crowdsourcing an international language dataset. Developed collaboratively between ZKM and three international developers at the end of 2021, the interactive Data Collection Kiosk allows visitors to create new language datasets by recording their own language. In addition to recording them, the Kiosk also serves to validate the recordings: They can be listened to and accepted or rejected for archiving. The speech samples that are created during the exhibition period are collected and integrated into a multilingual speech dataset. This will serve the further development of AI-supported speech dialog systems over the exhibition period of BioMedien.

This example will help to determine to what extent art and cultural institutions can contribute to the development of data sets and thus demonstrate to what extent the existence of a special sensor-technical infrastructure can favor the crowd-based collection of this data. Especially through the use of AI systems, as indicated above, new and innovative visitor interactions with the museum become possible. The following part will therefore describe the prototypical development of an intelligent IoT system to stimulate novel modes of interaction between visitors and the museum and to enhance museum intelligence.

Sensors

Similar to the sensory organs with which we humans perceive our environment, sensor technology enables the conversion of chemical or physical properties into electrical signals. From kitchen scales to smartphones, such technical sensors are omnipresent in our everyday lives. The atriums of the ZKM are also equipped with numerous sensors: Multisensors measure CO2, light intensity, temperature or fine dust in the exhibition rooms, smart recording systems count exhibition visitors and provide information about the power consumption in the building. Both active sensors (e.g., LiDAR, ultrasonic sensors, etc.) and passive sensors (e.g., cameras, microphones, light sensors, etc.) are used.

In order for visitors to be able to interact with the intelligent museum of the future, its technical infrastructure must enable environmental parameters to be recorded via sensors on the one hand and certain actions to be triggered via so-called actuators on the other. Just as there are different types of sensors, there are also different types of actuators—from mechanical actuators (e.g., electric motors) to optical actuators (e.g., light bulbs, LEDs, screens) to acoustic actuators (e.g. loudspeakers). An example of a sensor-actuator system is an escalator: A light barrier (sensor) detects the presence of a person, which causes a motor (actuator) to set the stairs in motion.

IoT System

As Researcher-in-Residence, Dr. Andreas Kugel is working with the project team on a best-practice example of sensor-based data collection and provision as part of intelligent.museum. This is intended to be a first development step towards an intelligent IoT system, with the short-term goal of allowing things to communicate with each other and the medium-to long-term goal of establishing the data basis for AI-supported analyses. As an extension of the decision-making capabilities of museum staff, learning systems should be able to analyze and draw conclusions from the various data. In the future, for example, digital artworks should be able to be integrated into the IoT ecosystem—also for the purpose of remote maintenance, data mining or against the background of many other conceivable needs.

The open-source ThingsBoard platform was selected as the temporary backbone for the IoT system. The main tasks of the ThingsBoard platform here are the storage of data as well as its post-processing and visualization. The concept of ThingsBoard is that different devices can be created, i.e., normally each external sensor is connected to its own device in ThingsBoard. The devices can be created automatically. In order to cover a certain number of different scenarios, so-called device profiles can be defined and new devices can be added for them (automatically, if necessary).

For the visualization of data by means of dashboards, standardized display types—from tables to heat maps to bar, line, pie or bubble charts—can be selected. However, widgets can also be customized—for example, a visitor counter could be displayed as a Brownian molecular motion.

Example On the occasion of the exhibition BioMedien, a ZKM-specific dashboard was developed at the end of 2021 with the platform ThingsBoard, which can be viewed in the exhibition space and online. Via the dashboard, (sensor) data of the ZKM from the indoor and outdoor areas as well as from the internet will be visualized in real time. This multifaceted data is generated by smart devices, sensors and other IoT devices. Indoors at ZKM, these include the multi-sensors installed in the exhibition space that measure CO2, light intensity, temperature, and particulate matter. Also, LiDAR sensors, ultrasonic sensors and optical hand tracking modules. From the employee elevator, a sensor sends the elevator position with associated time stamp, which in turn can calculate the speed. There are also IoT sensors in the outdoor area of the ZKM (meadow in front of the ZKM/HfG building + rented meadow orchard), which use the long-range energy-efficient LoRaWAN (Long Range Wide Area Network) for information and data transmission. ³ On the Internet, data is collected through API calls (for example, to retrieve data from websites), tracking tools (for web analytics), or web crawlers. [Figure 1] 

Figure 1: Exhibition view intelligent.museum Dashboard, BioMedien at ZKM | Karlsruhe, 2021/2022. Copyright: ZKM | Karlsruhe, screenshot: Paul Bethge 

Some of the data is already used for a ZKM-specific data set on the occasion of the hackathon {COD1NG DA V1NC1}. This dataset called We are data - The ZKM as a living organism does not present work data or digitized data of museum collection objects, but is intended to represent the everyday life of a living and constantly changing cultural institution. In addition to the sensor data mentioned above, it also includes, for example, data on which books the library of the ZKM and the Staatliche Hochschule für Gestaltung Karlsruhe lends out per day or how many words the ZKM's publications department edits. The dataset contains day-based values for the month of March 2022 and was published under the open-source license CC BY-SA 4.0.[4]

Sensor-based visitor tracking

In addition to the aforementioned data from IoT sensors, tracking data represents a pertinent resource for the data-informed museum of the future. In order to be able to track people and objects indoors and outdoors in a way that complies with data protection laws and preserves privacy rights, a LiDAR-based system for two-dimensional real-time location has been developed since the end of 2021 by Bernd Lintermann—artist and software developer at ZKM—and was released open-source in 2022.

The system was initially developed for two use cases. The first use case envisions the media-artistic use of the system for interactive installations, for example. In this area, optical sensors are often used to receive input from users by tracking people in the installation's catchment radius. The second use case sees the system as an analysis tool for museum operations, providing information on how museum areas are used, visitor flows, and where visitors spend most of their time. This should help to successively improve the exhibition experience and enable museum operators to optimize the museum infrastructure.

LiDAR (Light Detection and Ranging) is a telemetry method related to radar, based on time-of-flight measurement with pulsed laser beams in the eye-safe range. 

The application areas for LiDAR-based acquisition systems range from geosciences—e.g., in meteorology or geodesy—to the automotive industry—e.g., for driver assistance systems or in the field of autonomous driving—to robot navigation. Increasingly, LiDARs are also being used in consumer devices. For example, since 2020, support for LiDAR scanners has improved AR applications in iOS devices.

When integrated into smart applications, LiDARs open up new practical sensing opportunities. However, a LiDAR-based scalable system for real-time two-dimensional location of people and objects indoors and outdoors, based on open-source software, does not exist at the time of writing.

For the tracking system, 2D LiDAR sensors are used on the hardware side. 2D LiDAR sensors use 1D LiDAR sensors that rotate clockwise and whose optical distance measurements can be combined by the computer into two-dimensional environmental data. The most powerful LiDAR sensors tested during the system's development measure distance data approximately 8000 times per second.

The data from all the sensors in the system is combined by Lintermann's software and transferred to a unified virtual space in which the detected objects are distinguished from one another. The system checks where the objects were located in the previous frames. If there is a positional discrepancy, the system assumes that an object has moved. Detected objects are tracked and IDs are assigned to them. The system can track the movement of objects and determine if certain objects have left an area or how long certain objects have been in an area. [Figure 2] 

Figure 2: Screenshot Tracking System by Bernd Lintermann in the ZKM exhibition John Sanborn. Between Order and Entropy, October 2022. Copyright: ZKM | Karlsruhe, graphic: Bernd Lintermann 

The software design of the system aims to be as user-friendly as possible. This means that the software should first be as easy to install as possible and then as easy to use as possible via an intuitive graphical user interface.

The affordability of the system is also intended to increase broad accessibility, which is why different LiDAR sensors from various vendors and price ranges were tested. The software uses SDKs (Software Development Kit = collection of programming tools and program libraries used to develop software) from different manufacturers, whose respective model range—from low to medium to high-priced devices—is to be covered.

The system is also scalable, which in this context means that theoretically any number of LiDAR sensors can be used. This means that a large physical space can be covered, but it is also possible to interconnect multiple spaces, thereby creating particularly individual tracking areas.

The software offers the possibility to log the accruing movement and whereabouts data and to write them to files in a common data format—JSON—that can be processed very well by machines. These logging files record when objects have changed their location, i.e., moved several centimeters to the side, for example. A new entry is then created, with a timestamp, so that it can be traced how long which object was in which position and over what distance it moved.

A special feature of the system is the logging of this data in graphics, because the movement and dwell time data can be used to generate so-called heat maps, from which it can be read where people have been particularly frequently.

The logging data can be configured and put into a temporal structure so that, for example, a new file is created for each freely definable period of time—if users are interested in specific periods of time, they only need to look at the files for these specific periods.

Compared to conventional camera technology, LiDAR sensors offer both advantages and disadvantages when used for data acquisition systems. First, LiDAR sensors are faster and less computationally intensive than camera technology, making the terminals used to process the data more cost-effective. LiDAR sensors also do not rely on ambient light, whereas conventional cameras are inefficient in adverse weather conditions or darkness. Thus, no artificial lighting needs to be considered for the use of LiDAR sensors.

However, there are also aspects of technology ethics that argue against the large-scale use of cameras in the exhibition space for a room-scale detection system for people and objects. These aspects relate to security, anonymity and privacy, because cameras naturally also capture biometric information, whereas the data generated by LiDAR-based systems do not allow any conclusions to be drawn about individuals. Neither body poses nor gestures and facial expressions of individual visitors can be captured with the system.

The "shadowing" of persons in the exhibition space by other exhibition visitors requires that significantly more LiDAR sensors be used under real conditions than would be the case under laboratory conditions. 

Implementing Communication Protocols

In collaboration with artist and software developer Bernd Lintermann and researcher-in-residence of the intelligent.museum project, Dr. Andreas Kugel, Dan Wilcox—one of the project’s two permanent software developers—is currently working on using the open-source ThingsBoard IoT data server as a streaming data platform for the recently developed LiDAR-based tracking system mentioned above. The idea is to develop simple, open tools that allow creative programmers to interface with the ThingsBoard. This way, other artists can easily connect and receive live tracking data or general sensor data from the ZKM.

As part of this development strand, thoscy (Thingsboard OSC Relay) was developed and released open-source. ⁵ This is a collection of scripts for relaying messages between a ThingsBoard server and the OSC network protocol. These scripts act as relay servers for forwarding device events between a ThingsBoard server via MQTT/WebSockets and OSC (Open Sound Control) messages. thoscy-send forwards messages from OSC to a ThingsBoard host via MQTT, while thoscy-recv listens for ThingsBoard events via a WebSocket and forwards them via OSC. This is especially useful for software tools that work natively with OSC messages but do not have built-in MQTT or WebSocket support.

Outlook and Desiderata

In the next part, going beyond the technical development of the intelligent IoT system, various desiderata will be outlined by way of example, which are to be fulfilled with the system and which will be implemented prototypically.

Example 1: Data Mining for Museum Visitor Research

While data analytics is used to optimize business processes and decisions in the corporate sector, it is still comparatively rarely used to optimize museum operations. As a best practice example from the Barberini Museum shows, data mining can be used in the museum sector to improve the museum experience for visitors. ⁸  The Barberini Analytics analysis platform is used to collect, process, analyze and present available data relating to museum visits with the aim of improving the museum experience. Barberini Analytics was developed for the Museum Barberini between 2019 and 2020, but also includes approaches transferable to other art and cultural institutions. The suite was compiled by the Hasso Plattner Institute and the Museum Barberini in Potsdam, founded by the Hasso Plattner Foundation, as part of a bachelor's project supervised by Prof. Dr. Felix Naumann at the Hasso Plattner Institute in Potsdam entitled Data Analytics - Optimizing Museum Experiences with Data Analysis. The program codes were published as an open-source project via a code repository on GitHub. ⁹ 

Figure 3: Visitor forecast as part of the intelligent.museum project using a predictive AI model. Copyright: ZKM | Karlsruhe, graphic: Paul Bethge 

While predictive models can be developed for museum operations, for example, to help forecast visitor flows and numbers,¹⁰ [figure 3] new potentials of the system described above as an analytics tool for visitor research can be anticipated in particular.

In 1983 Veron & Levasseur conducted ethnographic observations in an experiment at the Louvre in Paris and analyzed the visiting style of museum visitors. They classified them into four categories. The categories are named after animals, because the behavior of the visitors is supposed to be similar to the behavior of these animals: "ants," "grasshoppers," "butterflies," and "fish." While the visitors classified as "ants" follow a certain path through the exhibition space and look closely at all exhibits, the visitors identified as "grasshoppers" seem to prioritize some previously selected exhibits. The group of "butterflies" includes visitors who wander aimlessly around the museum, but are very interested in the exhibits and try to get more information. Visitor:ings classified as "fish" spend most of their time moving around the center of the room without seeming interested in details of individual exhibits. In the mid-1990s, Tsvi Kuflik, a professor of information systems at the Faculty of Social Sciences at the University of Haifa in Israel, and a team conducted the mobile museum guide experiment with 143 participants, using two different clustering methods (artificial neural networks and the k-means algorithm) and comparing them. ¹¹ 

Using the already described LiDAR-based tracking system as a sensor within the also already described intelligent IoT system, another experiment will be initiated with reference to Veron & Levasseur and Kuflik et al. respectively, for which AI can be used to analyze the "visiting style" of exhibition visitor:s in the intelligent museum. This experiment will investigate the suitability of the intelligent IoT system as an analysis tool for visitor research.

Example 2: Automatizing Museum and Exhibition Technical Services

In the technical museum operation, repetitive tasks can be automated sporadically by software and algorithms, such as the regulation of lighting and sound in the exhibition space—from the light intensity of the ceiling lighting to the volume of sound installations. Since 2020, systems have been in use in the ZKM's exhibition operations that link the unmuting of videos in the exhibition space to the presence of visitors within a freely definable retraction radius defined by the museum's technical staff, and thereby couple the distance of the visitors to the playback device with the volume parameter. This can prevent cacophony in the exhibition space when many videos with sound or sound installations are presented simultaneously in one exhibition space. Used carefully, automation can lead to a sustainable improvement of the exhibition experience for the visitors—both in terms of ecological (light) and design aspects in the exhibition space (sound). However, automation should only be used very sparingly and exclusively in less sensitive areas.

There are many other potential applications in the field of technical services for museums and exhibitions, such as predictive maintenance or anomaly detection to monitor and, if necessary, maintain computerized exhibition exhibits.

Example 3: Framework for Interactive art

In addition, the intelligent IoT system will provide artists with a technical framework for developing interactive art experiences. In the medium term, it should become possible for artists to feed and operate generative data visualizations and sound recordings, interactive installations and environments with sensor data from the system. Similarly, many media-art artworks now generate data on their own, which they can in turn feed into the intelligent IoT system.

In this context, Dr. Andreas Kugel is already working in parallel with the artist Katrin Hochschuh of the artist:in duo Hochschuh/Donovan on the resource-efficient transmission of position data of a robot swarm (artwork Empathy Swarm by Hochschuh/Donovan, which has been further developed within the project intelligent.museum) to a ThingsBoard device.

Conclusion

The diffusion and successive improvement of AI tools has enabled many new interactions between visitors and the museum. In the analysis, it became clear that for an efficient integration of these into museum operations, the technical infrastructure in particular needs to be looked at and that specialized systems would need to be developed to perform intelligent analyses via data mining so that they can actually extend museum intelligence via suggested action suggestions to museum staff. As Harry Armstrong et al. point out "[...] for many arts and cultural organisations there is a long way to go before it is possible to extract value from the data they hold. Data input and analysis both require skills and specialisation in the workforce.
Currently, in many institutions or organisations data is often not systematically collected and, even where it is, it may not be analysed in depth. Developing the sector’s capabilities in this area will require attention at all stages of the skills pipeline [...]" ¹² The ongoing prototypical developments described in this text are running towards the development of an intelligent IoT system in the medium term. What seems particularly interesting in connection with the development of an intelligent IoT system is that both museum operational data mining scenarios and those concerning the museum in general but also purely artistic scenarios can be realized, which immensely increases the spectrum of possibilities for use. The most important criteria of the described system are its transparent and open- source development, which makes it possible to reuse the codes used, e.g. by other art and cultural institutions, and ensures that the technologies used have as little potential as possible to cause harm to museum visitors, be it through discrimination or inherent forms of bias.

References

1 Moritz Kirste, "Augmented Intelligence – Wie Menschen mit KI zusammen arbeiten“, in Künstliche Intelligenz, ed. Volker Wittpahl, Berlin and Heidelberg, Springer Vieweg, 2019, p.58-71. 

2  Gartner IT Glossary, Augmented Intelligence, Gartner Inc. online, accessed December 9, 2022, https://www.gartner.com/en/information-technology/glossary/augmented-intelligence.

3 Lívia Nolasco-Rózsás, Yannick Hofmann, "The Museum as a Cognitive System of Human and Non-Human Actors“, The Garage Journal: Studies in Art, Museums & Culture, Vol. 03, accessed December 2022, 2021, p.1–15, https://thegaragejournal.org/files/07/758_c370a5422f798cbd308f d337837b93dc25ec66d8.pdf.

 4  Yannick Hofmann, "Projektvorstellung intelligent.museum“, in AI: A Museum Planning Toolkit, ed. Johannes Bernhardt, Tabea Golgath, Oonagh Murphy, Sonja Thiel Elena Villaespesa, London: Goldsmiths University, 2020, 13f.

5 LoRa Alliance, Technical Marketing Workgroup 1.0, LoRaWAN What is it? A technical overview of LoRa® and LoRaWANTM, accessed December 9, 2022, https://docs.wixstatic.com/ugd/eccc1a_ed71ea1cd969417493c74e4a13c55685.pdf.

6 COD1NG DA V1NC1, Wir sind Daten – Das ZKM als lebender Organismus, accessed December 9, 2022, https://codingdavinci.de/daten/wir-sind-daten-das-zkm-als-lebender-organismus.

7 intelligent.museum on GitLab, thoscy, accessed December 9, 2022, https://git.zkm.de/Hertz-Lab/Research/intelligent-museum/thoscy.

8 Esther Knuth, Remigiusz Plath, "Mit Daten das Museumserlebnis verbessern. Besucher:innen-Datenanalyse im Museum Barberini“, in Kultur Management Network Magazin Nr. 160: Digitale Besucher:innen, accessed December 2022, 2021, p.26-32, https://www.kulturmanagement.net/dlf/00cf0ded4596ca16ee75e0189c9f0d57,3.pdf

9 Museum-Barberini on GitHub, Barberini-Analytics, accessed December 9, 2022, https://github.com/Museum-Barberini/Barberini-Analytics.

10 Antonio Krüger, Tsvi Kuflik, Ubiquitous Display Environments (Cognitive Technologies), Berlin, Heidelberg, Springer, 2012, p.170.

11 Massimo Zancanaro, Tsvi Kuflik, Zvi Boger, Dina Goren-Bar, Dan Goldwasser, "Analyzing Museum Visitors" Behavior Patterns“, in User Modeling 2007. UM 2007. Lecture Notes in Computer Science(), vol 4511, ed. Cristina Conati, Kathleen McCoy, Georgios Paliouras, Berlin, Heidelberg, Springer 2007, accessed December 9, 2022, https://doi.org/10.1007/978-3-540-73078-1_27.

12  Harry Armstrong et al., Experimental culture: a horizon scan commissioned by Arts Council England., accessed December 9, 2022, https://www.artscouncil.org.uk/experimental-culture-horizon-scan

Bibliography

1 Harry Armstrong et al., Experimental culture: a horizon scan commissioned by Arts Council England., accessed December 9, 2022, https://www.artscouncil.org.uk/experimental-culture-horizon-scan

2 COD1NG DA V1NC1, Wir sind Daten – Das ZKM als lebender Organismus, accessed December 9, 2022, https://codingdavinci.de/daten/wir-sind-daten-das-zkm-als-lebender-organismus.

3 Gartner IT Glossary, Augmented Intelligence, Gartner Inc. online, accessed December 9, 2022, https://www.gartner.com/en/information-technology/glossary/augmented-intelligence.

4 Yannick Hofmann, "Projektvorstellung intelligent.museum“, in AI: A Museum Planning Toolkit, ed. Johannes Bernhardt, Tabea Golgath, Oonagh Murphy, Sonja Thiel Elena Villaespesa, London: Goldsmiths University, 2020, 13f.

5 intelligent.museum on GitLab, thoscy, accessed December 9, 2022, https://git.zkm.de/Hertz-Lab/Research/intelligent-museum/thoscy.

6 Moritz Kirste, "Augmented Intelligence – Wie Menschen mit KI zusammen arbeiten“, in Künstliche Intelligenz, ed. Volker Wittpahl, Berlin and Heidelberg: Springer Vieweg, 2019, 58-71.

7 Esther Knuth, Remigiusz Plath "Mit Daten das Museumserlebnis verbessern. Besucher:innen-Datenanalyse im Museum Barberini“, in Kultur Management Network Magazin Nr. 160: Digitale Besucher:innen, accessed December 2022, 2021, 26-32, https://www.kulturmanagement.net/dlf/00cf0ded4596ca16ee75e0189c9f0d57,3.pdf.

8 Antonio Krüger, Tsvi Kuflik, Ubiquitous Display Environments (Cognitive Technologies), Berlin, Heidelberg, Springer, 2012, 170.

9 Lívia Nolasco-Rózsás, Yannick Hofmann, "The Museum as a Cognitive System of Human and Non-Human Actors“, The Garage Journal: Studies in Art, Museums & Culture, Vol. 03, 1–15, accessed December 2022, 2021, https://thegaragejournal.org/files/07/758_c370a5422f798cbd308f d337837b93dc25ec66d8.pdf.

10 LoRa Alliance, Technical Marketing Workgroup 1.0, LoRaWAN What is it? A technical overview of LoRa® and LoRaWANTM, accessed December 9, 2022, https://docs.wixstatic.com/ugd/eccc1a_ed71ea1cd969417493c74e4a13c55685.pdf.

11 Massimo Zancanaro, Tsvi Kuflik, Zvi Boger, Dina GorenBar, Dan Goldwasser, "Analyzing Museum Visitors’ Behavior Patterns“, in User Modeling 2007. UM 2007, Lecture Notes in Computer Science(), vol 4511, ed. Cristina Conati, Kathleen McCoy, Georgios Paliouras, Berlin, Heidelberg, Springer 2007, accessed December 9, 2022, https://doi.org/10.1007/978-3-540-73078-1_27.

12 Museum-Barberini on GitHub, Barberini-Analytics, accessed December 9, 2022, https://github.com/Museum-Barberini/Barberini-Analytics.

Biography

Yannick Hofmann (*1988 in Offenbach a. M., Germany) lives and works as an artist and researcher in Karlsruhe. As the artistic director of the intelligent.museum project since 2020, he collaborates with a team of software developers and museum visitor research experts, pushing the boundaries of hybrid formats and applications for the future of museums. Having spent almost a decade at ZKM | Center for Art and Media, he co-directed their artistic research and production department before joining the Deutsches Museum for a one-year research stay in 2022. Since mid- 2023, Hofmann works for the Fraunhofer Institute for Industrial Engineering. 

The Creative Design and Social Service Practice of zen_Farm

Yu-Hsiung Huang, Su-Chu Hsu

suchu@mx.nthu.edu.tw ; eric@techart.tnua.edu.tw

Abstract

We developed "zen_Farm" through waste recycling and makers, with the aim of guiding people to practice how to calm their minds. Utilize the concept of "Every drop of calm water makes all things green" to remind the world to cherish the natural environment and live in peace with all things. The main purpose of zen_Farm is to emphasize the calmness and concentration of "Mind Meditation". Its special feature is that people use the stability of their heartbeats to drive the water source of zen_Farm. In recent years, the world has gradually developed a new form of creative community that combines digital media to gather community awareness and improve the current situation of the community, which can bring new energy and stimulate new thinking in urban areas. We practically integrate zen_Farm into social practice, including:

(1) The installation is located at Dharma Drum Mountain, the most important Buddhist unit in Taiwan. Let the Buddhist masters in the park use the concept of plant irrigation to reflect the natural environment to be sustainable and green. Guide them to finally become one with nature. (2) The installation was installed at Shakeng Elementary School in Taiwan, allowing students to renovate the campus together and irrigate the plants through their own heartbeats to create a common memory on the campus. The zen_Farm is not only an interactive installation art work made by makers, it makes the campus of Buddhist parks and rural schools more friendly and beautiful, and it also supports the concept of environmental protection. We also hope to achieve the long-standing goal of digital art creators - "Media Transparency" through the creative design and social practice of zen_Farm. Participating in meditation activities can connect the emotional memories of the community together and establish a shared memory in the campus.

Keywords

Zen, Farm, Maker, Interactive Installation Art, heartbeat, Creative Community.

DOI

10.69564/ISEA2023-65-full-Huang-et-al-zen-Farm

Introduction

The Importance of Meditation for Everyone

Modern life is becoming more and more busy, and people are pursuing a better quality of life. What is a good quality of life? The World Health Organization (WHO) has published the WHOQOL assessment, a quality of life assessment developed by the WHOQOL Group in parallel with 15

international field centers ¹. WHOQOL includes four domains: physical health, mental state, social relations, and environmental domain. Meditation in Buddhism can make people's mental state invisibly tend to be calm and soothing through the guidance of meditation, so as to achieve a state of "relaxation" and "concentration". In addition, it can also achieve the concept of "Protecting the spiritual environment" advocated by Master Sheng Yan of the Dharma Drum Mountain.

The Rise of Makers and FabLab

The "Maker Movement" has become an important international trend in recent years, influencing the Third Industrial Revolution. Countries around the world have integrated it into national policies and education systems, and even applied it in various fields. In 2014, President Obama of the United States announced that the future of American manufacturing would return to the domestic front, as the U.S. had comprehensively developed digital fabrication (FabLab) and maker movement education, from primary schools to universities ². With the popularity of the Maker Movement, concepts like "open science", "DIY science" or "citizen science" have continuously influenced societal development ³. Integrating the concept of makers into social practice, the Maker Movement represents a form of social participation.

We synergized the "spiritual environmental protection" philosophy of Dharma Drum Mountain with "natural environmental protection" concepts. This collaboration resulted in the creation of zen_Farm, an interactive installation art project crafted from repurposed plastic bottles, embodying the essence of maker’s DIY. The unique feature of zen_Farm is its use of a calmness index, derived from heartbeat detection, to control the irrigation of plants. The mechanism is designed such that the more tranquil the state of mind, the more water is dispensed for plant nourishment.

Figure 1: zen_Farm at Dharma Drum Mountain © Suchu Hsu

Impact of Social Practice and Creative Communities

Taiwan's Ministry of Education has implemented the "University Social Responsibility (USR) Program" since 2018. Promote the development of the real estate industry through the participation of university talents into the community. At the same time, students can make more contributions and care to the local society, and establish the university's social responsibility for community care. As universities increasingly focus on social engagement, students are encouraged to need a full range of learning that is not limited to the classroom. Through social practice, in addition to expanding students' learning fields, it is also the concrete practice of civic education ⁴.  Similarly, we have applied the zen_Farm concept again in the remote Shakeng Elementary School, enabling the entire student body to paint PET bottles together, install and transform the school campus. This allows children to engage in physical activity while also performing heartbeat detection to irrigate plants, thereby fostering a state of calm and reflection.

Figure 2. zen_Farm at Shakeng Elementary School © Suchu Hsu

We practice zen_Farm in two communities (Dharma Drum Mountain Buddhist Park, Shakeng Elementary), hoping to achieve physical and mental tranquility (Zero), concentrate on generating energy (Energy), and become one with the natural environment (Nature) to achieving the essence of "ZEN" is in the field of life and study. In this way, the emotional memory of the community can be linked together. Use technology application and humanistic care to build a common memory and create a creative community. Expect to achieve a good quality of life in the areas of mental state, social relations, and environment assessed by WHOQOL.

Related works

Related literature on the Importance of Meditation for Everyone

There are many scientific studies confirming meditation in ZEN. For example, Lusnig, Larissa, et al., proposed that meditation is a mental practice that can increase one's attention and calmly observe all the feelings of the moment. Let people enter a state of tranquility and achieve emotional regulation.⁵ Tang, Yi-Yuan, et al. also proposed that the meditation practice can promote the practitioner's ability to maintain concentration through physical relaxation, breathing exercises, mental imagery, etc.⁶  At the same time, there are also many artistic works applied to meditation and mental state, such as the interactive multimedia work "ZENetic Com-puter" created by Naoko Tosa and Seigow Matsuokam (2003),⁷ which transforms traditional meditation content into multimedia to express The artistic conception of ZEN, and guide the viewer to learn the practice of ZEN. In addition, Hoshiyama & Hoshiyama of the University of Tokyo, Japan, also used electrocardiogram (Electrocardiography, ECG) to analyze the value of heart rate variability (HRV) changes for beginners in meditation practice.⁸

Related literature on the application of Maker and Fablab

With the development of digital technology in recent years, digital media has also been developed internationally to promote community awareness, thereby improving the current situation of the community and bringing new energy and stimulating new thinking to the community.⁹ Vossoughi et al. aimed at disadvantaged community students and provided suitable Fablab activities according to their needs to achieve higher social practice and educational equity.¹⁰ The "SASFAB Maker Alliance" program implemented by Hsu et al. was launched in Taiwan in 2015. It combines "Internet of Things (IoT) and "Maker/FabLab" to carry out social practice and other activities with artistic creative courses.¹¹

The Impact of Social Practice USR or Creative Community

In the Hangleton & Knoll community in the eastern part of England, Johnson & Monney collaborated with the University of Brighton to conduct community participation projects through art participatory activities. Through a participatory research approach to art, they engage researchers, artists, and community residents as partners in a "research collective," using art to explore and exchange their life experiences with each other to create a shared experience of the community.¹² In addition, many studies have found that the integration of group art activities into social practice can increase the cohesion and connection of the community.¹³

An example of the application of interactive installation art in community service

Yang and Hsu developed the IoT platform in 2017 to implement the "Windflower ESPSAS" project at Taipei Munici- pal NanMen Elementary School in Taiwan. It guides elementary school students to use their maker spirit to make Windflower objects by themselves to beautify the campus and community through artistic creation through social network actions. Windflower is a windmill during the day and a colorful lantern at night. You can watch the rotation of the windmill you made in the campus remotely through the mobile phone, and you can also change the light color of the remote Windflower through the app, so that everyone's emotional memory extends from the campus to the cloud.¹⁴ Also a good example of social practice.

According to the above-related works, there are not many cases integrating "meditation," "maker," "interactive installation art" and "social practice" at the same time. This study takes this as the research goal and proposes to take ZEN as the theme, applying zen_Farm's creative design practice in two communities.

Creative Interactive Design of zen_Farm

We explain the overall creative interactive design of zen_Farm, including the device interaction design of water cycle, the device modeling design of heartbeat sensing, and the calculation logic of heartbeat sensing.

Interactive Design of Water Circulation Device

The installation design of zen_Farm is based on a simple "matrix arrangement" and emphasizes the "Mind Meditation" focus on the psychological level. It uses the method of recycling PET bottles and water circulation to build an automatic circulation irrigation system to implement the environmental protection and green energy proposition (Figure 3). According to the size and height of the space, there are 10 rows of 7 PET bottles installed in each row, with a total of 70 PET bottles. After cleaning the recycled PET bottles one by one, we dig holes according to the needs of the device to facilitate the growth of plants. We use the original structure of the bottle cap to design the various components required for the water cycle, and finally connect the head and tail of each bottle. Finally, use the air pipe to connect all the bottles and complete the water circulation structure through the pump.

Styling Design of Heartbeat Sensor Device

We set up a heartbeat console with a height of 100 cm next to the water circulation system, with a palm-shaped operating area and LED lights above (Figure 4). Taking the Dharma Drum application as an example, we use the Dharma Drum palm-shaped logo to visually guide the participants to place the left palm on the palm-shaped operating area, and embed a heartbeat sensor in the position of the index finger to facilitate detection. The stability of the participants' heart rate was measured. In the lower right of the operation area, the LED light signal shows the result of the heartbeat stability (Figure 5), and it is supplemented by a sound to guide the participant to start and end the heartbeat detection. The pedestal of the device includes three pneumatic pumps, a set of heartbeat sensing modules, a control chip module, a sound effect playback module, and a sound effect speaker.

Figure 3: zen_Farm's bottling and water circulation irrigation system© Suchu Hsu
Figure 4 (left): Heartbeat Sensing Pedestal © Suchu Hsu Figure 5(right): Heartbeat Sensing Panel © Suchu Hsu

Heartbeat Sensing Calculation Logic Description

The basic logic of heartbeat sensing is to measure the stability of the participant's heartbeat within one minute as the irrigation index of the water circulation irrigation system, and display the level of stability with LED lights. After the participants put their hands on the heartbeat sensor (Figure 6), the sensor stability is adjusted and calibrated for the first five seconds. If the finger is kept pressed steadily until the calibration is completed, the electronic control chip will drive the sound module to play a sound. After a short "wooden fish sound", the system begins to collect the value of the heartbeat stability. The total length of the sensing time is 60 seconds, and the value of every 15 seconds is used as a paragraph to calculate the sum of the differences in heartbeats between every 15 seconds as an indicator of stability. After the heartbeat stability calculation is completed, the electronic control chip will drive the sound module to play two short “wooden fish” sounds, indicating that the heartbeat detection has been completed, and remind the participant to leave the sensor on the pedestal with his left hand. Then the system will count the participants' heartbeat stability and display the different stability levels from 1 to 10 with LED lights. The higher the value, the more stable it is. A higher stable heartbeat value will control the pump to provide proportionally more water to irrigate the plants. Conversely, lower values provide less water for irrigation. If the finger leaves the sensor in the middle of the sensing period or the measurement method is incorrect, the system will return to the initial state and wait for the data to stabilize before remeasurement (Figure 7).。

Figure 6: Heartbeat sensor and Led light of 1~10 level
Figure 7: Flowchart of Heartbeat Sensing Calculation © Suchu Hsu

Social Service Practice of zen_Farm

Social Practice in Dharma Drum Mountain Buddhist Park

Dharma Drum Mountain Buddhist Park is the most important Buddhist research unit and meditation place in Taiwan. In 2012, we built zen_Farm in Dharma Drum Mountain as a public art. Its appearance is minimalist in line with the spirit of ZEN. The Master can do "Mind Meditation" exercises at any time in the park and use the index of calmness to irrigate the Zen Heart farm plants, as shown in Figure 8.。

In the past, the meditation process was mostly guided slowly by the master through scriptures or oral speech.¹⁵ Through zen_Farm, it is easier for community members who have no experience in meditation to practice "Mind Meditation," and through the connection of heartbeat, water circulation and irrigation plants, it is easier for community members to understand the importance of "Mind Meditation". Participants experience a reduction in distraction from thinking and an increase in concentration.¹⁶ Guide them to experience the Zen meaning of "Every drop of calm water makes all things green."

Figure 8: Dharma Drum Mountain Master doing mind meditation practice at zen_Farm © Suchu Hsu

Practice at Shakeng Elementary School

Shakeng Elementary School in Hsinchu County, Taiwan is a rural primary school with only 27 students because most of the students go to big cities to study. In 2022, we will extend zen_Farm to the campus of Shakeng Elementary School. We encourage students to work together to make zen_Farm by DIY. The overall installation uses rich color backgrounds. We guided the students to draw each bottle by themselves (Figure 9), and after completing the device, they could do the "Mind Meditation" concentration exercise through the heartbeat sensor device (Figure 10). We hope to implement the idea of environmental protection and green energy and build an indoor automatic circulation irrigation system by integrating recycled plastic bottles and water circulation through pneumatic pumps. In the past, meditation focused on the self-cultivation of the individual mind and inner consciousness. We hope that with the aid of technology and the concept of USR, we can strengthen the learning resources of rural education through technology media. In addition to the construction of technology and hardware, we also hope to lead the students of rural schools to create their own imagination more bravely and develop a better future from a spiritual level.

We hope that through this research, students in rural schools can be more cohesive in school-centered community development. Students can apply what they have learned to develop the character of the community. As mentioned by Vuorikari et al., Makers can not only solve real problems involving daily life, but also students can learn and develop new skills and establish new meanings.¹⁷ It allows disadvantaged or remote communities to combine local culture and resources through the “Maker Movement” to combine school and community participation, which can inspire better creativity and future development in the community.¹⁸

Figure 9. Students draw a recycling bottle © Suchu Hsu
Figure 10. Students participate in meditation practice through heartbeat sensor device © Suchu Hsu

Conclusion

Social practice can be studied from a variety of different topics, and in this dissertation we study from the perspective of art and creative design. Through the zen_Farm interactive installation art combined with discarded bottles and the spirit of maker, we have integrated into two different communities, including the "Dharma Drum Mountain Buddhist Park" and the "Shakeng Elementary School" in a remote village. In the "Dharma Drum Mountain Buddhist Park", zen_Farm is not only an interactive installation art work produced by makers, but also makes it easier for people entering the park to understand the participation process of "ZEN" and "Mind Meditation". We practice the concept of "Environmental Protection" combined with the concept of "Natural Environmental Protection" by realizing the relationship between the psychological state and the natural environment.

At Shakeng Elementary School in a rural area, we integrated heartbeat sensor module, ultrasonic sensor module, pump power system, etc. with interactive media technology into zen_Farm's meditation application. In the process of meditation, the most important thing is to calm down the state of mind to achieve spiritual protection. Through this concept, we guide students in remote schools to learn how to use the stability of their heartbeats to irrigate the plants in the bottle in the process of calming their minds. The paintings on these self-created installations are rich in the life memories of each student. This installation art is also actually installed on the campus of Shakeng Elementary School. Every day, students can use their own state of mind to irrigate their own creations, and share their heartbeats to share the common memory of each other's life. In this way, we guide students from remote villages to re-examine the relationship between themselves, their friends, and the natural environment. We try to enhance the cohesion of the community through the application of technology and humanistic awareness.

This research uses zen_Farm's creative design and social practice to achieve the long-standing goal of digital art creators—"media transparency." Participating members are able to link the emotional memories of the community together by participating in the meditation practice. It makes Buddhist campuses and rural campuses friendly and aesthetically pleasing and has an inspiring effect on environmental protection. At the same time, it also builds a shared memory for the creative community.

In the future, we will try to apply this model to different community fields, such as community farms, K12 educational institutions, etc. Through social service practice, university social responsibility and other cooperative methods. This will strengthen the connection between communities and local students, jointly creating a quality of life that aligns with the WHOQOL standards proposed by the WHO.

Acknowledgements

We would like to thank the National Science Association for the funding support of the "ZEN - Research on Creative Meditation Space for Ease and Mindfulness Program" (NSC 99-2218-E-655-001, NSC 98-2218-E-655 -001), and the Ministry of Education's "S+T+ARTS" Interdisciplinary Education and Local Natural Ecology Cultural and Creative Oxygen Program" funding support.

References

1 WHO, “The World Health Organization Quality of Life (WHOQOL)”, https://www.who.int/publications/i/item/WHO-HIS-HSI-Rev.2012.03

2 P. Blikstein, Maker movement in education: History and prospects. Handbook of technology education, 2018 ,419, 437.

3 Federico Ferretti, Harro van Lente, "The promise of the Maker Movement: policy expectations versus community criticisms." Science and Public Policy 49.1, 2022, 18-27.

4 Social Engagement, Social Innovation, Social Design, and Social Impact: Four Levels of University Social Responsibility Practice, Journal of Higher Education Research, 2020, 12: 47-68.

5 Larissa Lusnig, et al. "Zen meditation neutralizes emotional evaluation, but not implicit affective processing of words." Plos one 15.2, 2020,  e0229310.

6 Yi-Yuan Tang, et al. "Short-term meditation training improves attention and self-regulation", Proceedings of the national Academy of Sciences 104.43, 2007, 17152-17156.

7 Naoko Tosa, Seigow Matsuoka, "ZENetic computer: exploring Japanese culture", Leonardo 39.3, 2006, 205-211.

8 Haseda, Yuki, et al. "Measurement of pulse wave signals and blood pressure by a plastic optical fiber FBG sensor", Sensors 19.23, 2019, : 5088.

9 Arlene Goldbard, New Creative Community - The Art of Cultural Development, Oakland: New Village Press, 2006.

10 Shirin Vossoughi, Paula K. Hooper, Meg Escudé, "Making through the lens of culture and power: Toward transformative visions for educational equity", Harvard Educational Review 86.2, 2016, 206-232.

11  Su-chu Hsu, et al. “SASFAB - Smart Art-design Service in FabLab with IoT", The Ministry of Education subsidizes the "Smart Life Integration and Innovation Teaching Alliance Promotion Program", 2015-2018.

12 Helen Fiona Johnson, Nicole Monney, "Using the Arts to Support the Arts: A Creative, Community-University Partnership Approach to Building Arts Inclusivity in Economically-Deprived Communities", Forum: Qualitative Social Research, Vol. 22. No. 3, Freie Universität Berlin, 2021.

13 Abby Scher, "Can the arts change the world? The transformative power of community arts", New Directions for Adult and Continuing Education 2007.116, 2007, 3-11.

14 Jheng-Chun Yang, Su-Chu Hsu, "The study and application of smart art community service with “ESPSAS” internet of things platform." International Conference on Distributed, Ambient, and Pervasive Interactions, Springer, Cham, 2017.

15 Shi Shengyan, "Master Sheng Yen on Chan", 1996, 122. 

16  Bethany E. Kok, Tania Singer, "Phenomenological fingerprints of four meditations: Differential state changes in affect, mind-wandering, meta-cognition, and interoception before and after daily practice across 9 months of training", Mindfulness 8.1, 2017, 218-231.

17 Riina Vuorikari, Anusca Ferrari, Yves Punie, Makerspaces for Education and Training: Exploring future implications for Europe, No. JRC117481, Joint Research Centre (Seville site), 2019.

18 Simon N. Leonard, et al. "Designing Maker initiatives for educational inclusion", International Journal of Technology and Design Education, 2022, 1-17.

Thought Exhibition. On critical zones, cosmograms, and the impossible outside

Daniel Irrgang

Marie Skłodowska-Curie Fellow, Centre Art as Forum, University of Copenhagen Copenhagen, Denmark
daniel.irrgang@hum.ku.dk

Abstract

The paper discusses the curatorial concept of “thought exhibition” coined by Bruno Latour and Peter Weibel and developed in collaboration with curators, artists, and researchers during four exhibitions at the ZKM Centre for Art and Media, Karlsruhe (Germany). Thought exhibitions transgress the distinctions between philosophy, art, and science by testing ideas in an art museum, a space of discourse, representation, and participation. They engage visitors in a spatio-aesthetic thought experiment by bringing them into a position where preconceptions derived from epistemes of European Modernity are explicated and where alternatives are suggested. The analysis focusses on the most recent exhibition, in the preparation of which the author was involved: “Critical Zones. Observatories for Earthly Politics” (May 23, 2020 – January 9, 2022) mapped the symptoms and origins of the “New Climatic Regime” (Latour) of the late Anthropocene. In this paper, Critical Zones is framed within its theoretical context (Descola, Haraway, Margulis, Whithehead, among others) and discussed as relational spatio-aesthetic approach (Dikeç). The analysis concludes with Sarah Sze’s installation “Flash Point (Timekeeper)” (2018) as one of the exhibition’s central works – a representation, or “cosmogram” (Tresch), of a common planet that may provide an alternative to the globalized world of late capitalism.

Keywords

Anthropocene, art exhibitions, curatorial studies, dualisms, eco-criticism, European modernity, posthumanism, STS

DOI

10.69564/ISEA2023-66-full-Irrgang-Thought-Exhibition

Introduction

“The globe is something viewed from the outside, from a Galilean point of view. The critical zone is a view from the inside. Our show is about this contrast.” With this triangulation Bruno Latour located our endeavour on January 22, 2018, the first day of our very first seminar week, which would be followed by six further weeks over the course of two years. Dubbed by Latour as the ‘Critical Zones Study Group’ and co-organized by the author, the seminar took place at the Karlsruhe University of Arts and Design, the sister-institution of the ZKM Center for Art and Media in Karlsruhe, Germany which is located in the same building and has been the venue of the renown ‘thought exhibitions’ Latour had realized over the years. Our motivation was to conceptually prepare the exhibition ‘Critical Zones – Observatories for Earthly Politics’ at ZKM (May 23, 2020 – January 9, 2022), curated by Latour in collaboration with Peter Weibel, Martin Guinard, and Bettina Korintenberg.

Together with students, post-graduates, researchers, curators, and artists, we tried to triangulate a new “place to land1, now that the ground on which the globalized world of late capitalism is built is shifting and disintegrating in the age of the Anthropocene. The widely debated potential new geological epoch is certainly the first not only named but created by humans.2 Albeit ‘creation’ may not be the appropriate term for the devastating effects caused by capitalist extractivism and consumption. But even less so would be expressions such as ‘accidental effect’, as the data overwhelmingly suggesting the human cause of these ruptures has been known for a long time, despite regressive voices touting climate denialism.3

The aim of this paper is neither to fully analyse the curatorial approach nor to catalogue the exhibitions realized by Latour and his collaborators. This would go beyond its scope, considering the comprehensive thought exhibitions realized at ZKM, and beyond, over the years: ‘Iconoclash. Beyond the Image Wars in Science, Religion and Art’ (2002); ‘Making Things Public. Atmospheres of Democracy’ (2005)—this was, in fact, the first show explicitly called a ‘thought exhibition’, although Latour retrospectively also included Iconoclash4; ‘Reset Modernity!’ (2016). Particularly with regards to issues of climate change, Reset Modernity! laid the groundwork for Critical Zones, which was succeeded by ‘You and I Don’t Live on the Same Planet’ (2020/21) at the Taipei Fine Arts Museum for the 2020 Taipei Biennial.

This paper focusses on Critical Zones while trying to trace the specific spatial characteristics of a thought exhibition and its relation to the bodies and things—human, non-human—that constitute the exhibition space. After reconstructing the epistemological framework with which Latour approached the ‘cosmological’ shifts and uncertainties leading up to the global crises not limited to the direct effects of climate change, the paper discusses the visitor’s position (both in a figurative and embodied sense). It does so at least on two levels, by first considering the exhibition space as a whole and then by focussing on one of Critical Zones’ central artworks, a ‘cosmogram’ (John Tresch), Sarah Sze’s ‘Flash Point (Timekeeper)’ (2018).

Dualisms

But first I want to come back to the opening quote above. Latour put forward his notion of the globe in two, albeit intertwined, ways, a cosmological (or epistemological) and a spatial (or proxemic) one. Galileo Galilei’s discoveries – with regards to both, scientific methodologies and astronomic bodily movements—initiated a rupture in cosmology, as it catapulted the human from the centre of the cosmos into a spinning orbit around Earth’s star, one of countless in the universe. Despite this displacement, the anthropocentrism remained or was re-incorporated into the worldview of European modernity, most notably by means of a juxtaposition of somewhat separated spheres of nature and culture.5 This dichotomy implies an outside position from which one sphere can be acted up on by the ‘inhabitants’ of the other. The image of the globe representing Earth, a cartographic model as constructed as the nature-culture dualism, has become, as Stephen J. Gould would have called it, a “canonical icon”6, associated with capitalist globalism and an ideology of limitless growth. Such an impossible teleology not only ignores the limitations of ecosystem capacities but also of ‘natural resources’, to use a term normalized by capitalist extractivism. In this situation, the relation between the world we live in and the world we live of is distorted. This post-colonial heritage we discussed in our seminar through the concept of “ghost acres” which refers to exploited land abroad to cover a given territory’s consumption (originally of food, but expandable to other goods as well). Considering the devastating effects of such imbalances on exploited areas, “the climate question is at the heart of all geopolitical issues and it is directly tied to questions of injustice and inequality”7. Latour summarized these tensions as “New Climatic Regime”, a concept that became a starting point for the Critical Zones exhibition project.

The term ‘critical zone’ (singular, in contrast to the plural of the exhibition title emphasizing the concept’s manifoldness) is derived from Earth System Science8 where it denotes Earth’s “thin biofilm”9—down into the soil until the bedrock and up into the canopy and lower atmosphere—where Life10 subsists. But the critical zone is not only characterized in spatial categories. It is foremost a dynamic field sui generis, where the effects of “heterogeneous agencies mixed together in wildly different combinations”11 create their own conditions of Life (e.g., plants’ photosynthesis of carbon to oxygen as condition for other lifeforms that enable the existence of plants). This recursive, dynamic, and always incomplete interrelations have nothing to do with the static and continuous order of the nature-culture dualism cemented by European modernity. In fact, it may open a space for a kind of political action that considers manifold ways of how the interrelations of actors may compose a common world. Such a field cannot be organized as two monolithic blocks of nature and culture, where nature is somewhat treated or affected by human agency. “In that sense, the notion of the critical zone is much less paralyzing for politics than that of the Anthropocene.”12

The deconstruction of the nature-culture dualism is well known in Latour’s philosophical work, most notably in We have never been modern.13 Here the nature-culture dualism produces “hybrids”14 that are transgressing its dichotomy as they are neither assignable to one category nor to another (e.g., in vitro embryos or holes in the ozone layer). In a paradoxical twist, those hybrids start to dissolve the modern constitution, albeit they are constantly being reintegrated in its dualist structure. Artistic and scientific studies of hybrids are part of the repertoire of the works shown at Critical Zones. In its uncovering of the far-reaching effects of the nature-culture dualism, We have never been modern of course doesn’t stand alone. In our seminar sessions Latour emphasized the influence of, among others, Alfred N. Whitehead’s critique of the “bifurcation of nature”15 into a nature perceivable by humans and a somewhat ‘true’ nature behind perceivable phenomena. Also the work of Philippe Descola played a recurring role in our investigations with his “analysis of the modes of relations between existing entities”.16 Such a relational approach emphasizes the importance of spatial juxtapositions in an exhibition.

Relationality

Notions of relationality, heterogeneity, entanglement, and so forth are prevalent in studies that aim to overcome the distinctions and hierarchies of Western modernity in favour of a more sustainable mode of living together. They are already present in Gregory Bateson’s seminal 1972 work Steps to an Ecology of Mind. Criticizing the Western dualism as human separation from and dominion over nature and as a root for the approaching environmental crisis, Bateson emphasized the interconnectedness of all living beings. He also brought forward the notion of an ‘impossible outside’ (to which I will get back below): “We are not outside the ecology for which we plan – we are always and inevitably part of it.”17 Albeit relationality as ontological or even ethical category is usually judged (a priori) as something positive and preferable, I want to emphasize that not all interrelations are necessary experienced as something desirable. The cultural theorist Lauren Berlant has pointed out „the pressures of being in relation,”18 be it with human beings or with objects, as “a structural awkwardness in the encounter between someone and anything.”19 But they also acknowledge the necessity of the interaction with others as what drives one to experience the world. Notions of relationality are, even in this sense a mode of care or “response-ability.”20

Here it should be noted that, in contrast to Bateson’s claim for interconnectedness, the critical zone is not to be conflated with ‘holistic’ concepts of an “unified system […] where everything is connected.”21 This, according to Latour, would suggest yet another universalism. Rather, the relations remain fragile, always incomplete, always in a state of becoming. And yet, the notion of critical zone is closely related to the concept of ‘Gaia’.22 This analogy of Earth system science and Greek mythology was proposed by the geochemist James Lovelock in close collaboration with the microbiologist Lynn Margulis.23 While Lovelock developed his take from geochemical analyses of planetary atmospheres, Margulis worked with the other side of the magnifying scale, the microbial. Both met in the conclusion that Earth is producing and regulating its own environmental conditions for Life. It is not only this idea of autopoiesis and the roots of Lovelock’s work in cybernetics related to planetary self-regulation which places the Gaia hypothesis a bit too close to the universalism of a “unified system”. The mythological eponym also suggests a personification with close bounds to animism, presenting Earth as single entity. Although in the Critical Zones exhibition catalogue Latour makes it clear that “Gaia is not a big organism,”24 the exhibition maintained the concept and used it in interrelation with ‘critical zone’, as wells as with the term ‘the terrestrial’ denoting a new cosmology of the world we live in. The problems of the Gaia analogy were acknowledged by Latour25—and in fact also by Margulis: “I prefer the idea that Earth is a network of ‘ecosystems’ over any personification of Mother Gaia.”26

Impossible Outside

According to Latour, universalisms such as the nature-culture dualism or “unified systems” suggest “the hidden presence of an engineer at work who has devised the whole as a system of which we see only the parts.”27 This would imply a somewhat external position from which one may act on the Earth, or on ‘nature’. Such a position, taken by classical understandings of science (the ‘objective’ observer or experimenter) creates a relationship of maximum distance, both spatially and (let’s use this loaded term) ethically: When there is a sphere to dwell in (‘culture’) which can be separated from both the catastrophes we inflict on our planet (‘nature’) and human responsibility, then what is there to worry about? But if we do not live on the globe of modernity but inside the critical zone, a terrestrial interdependence in which we are intertwined with other entities to create our environment, every harm inflicted is eventually self-inflicted. As Margulis put it in an interview when describing the recycling processes of cyanobacteria: “If we would listen to them or watch them […], we would recognize that you can’t just throw things out—you never throw anything out, it goes around. […] Now these bacteria have solved that issue, people haven’t solved it at all. […] People are ruining their environment. These bacteria are producing an environment that’s liveable.”28

Following this notion of an ‘impossible outside’, a central aspect of the exhibition was to find an alternative to the representation of Earth as the famous Blue Marble, seen from a distant position in space. The matter of representation is by no means trivial or restricted to the task of finding an imaginary for an art exhibition, it has epistemological implications—a notion put forward prominently by the first thought exhibition, Iconoclash. And as Latour pointed out with Reset Modernity!, the view on the Blue Marble is “the place of nowhere,” as no one dwells in space. Like the eternal engineer’s gaze, this is a cartographical view of Earth as globe, “unified, continuous, and homogeneous”29, where every element has been placed, by science, in its assigned section of a ‘grid’. A grid in a literal sense when cartography slices projected space into metrics, but also in a conceptional sense, e.g., in taxonomical orders in biology. While this implies an external entity which organizes all other entities, the critical zone “breaks down the cartographical view of planet Earth.”30 Here there is no outside, but entities creating their own living conditions—and thus the critical zone itself. “Gone is the idea of a disinterested distant gaze.”31 This is far from a notion of space as container to be filled. It is a space composed of manifold elements and connections, “tiny, fragile, and provisional.”32

Compositionism

Composition, or “compositionism”, is a central concept not only in Latour’s writings but also for his curatorial approach.33 It is related to his earlier notions of politics described as an activity of “progressive composition of the common world.”34 As in the notion of critical zone, there is no world to be found a priori, ready to be inhabited. A common world must be continuously generated, “pieced together, element after element, through many travails and conflicts”35. Here a key text is “An Attempt at a ‘Compositionist Manifesto’” which, in its epigraph, Latour dedicated to “D.H.”. The nod to Donna Haraway makes sense: Not only is, in this essay, her book When Species Meet credited as being “a compositionist book if ever there was one”36. Her follow-up monograph Staying with the Trouble also picks up, etymologically, on the notion of composition or ‘compost’: “Critters—human and not—become-with each other, compose and decompose each other, in every scale and register of time and stuff in sympoietic tangling […].”37 According to Latour, the cosmological disruptions of the New Climatic Regime, the loss of the globe of globalization as a possible vector of impossible exponential growth, are “forcing all of us—scientists, activists, and politicians alike – to compose the common world from disjointed pieces instead of taking for granted that the unity, continuity, agreement is already there.”38

It is this approach of building alliances from disparate parts that makes a thought exhibition, and especially Critical Zones, an assemblage of scientific instruments (various measuring devices of different historical contexts, underlining the historicity of the world perceived), participatory practices (workshops, performances, field trips), and, of course, artworks. The heterogeneous artworks span from, e.g., Julian Charrière’s installation “Future Fossil Spaces” (2017), columns made of layers of lithium deposits and salt lumps, to a video installation by Barbara Marcel on a science and community project related to the Amazon Tall Tower Observatory (“Ciné-Cipó – Cine-Liana”, 2019– 2020), to a section of German Romanticism paintings, curated by the art historian Joseph Koerner. Although visitors could consult, similar to the predecessor exhibition Reset Modernity!, the guidance of a field book in order to navigate through the exhibition, its parts are meant to be put together or into relation subjectively. Here the museum becomes a testing ground, the exhibition a “scale model to test ideas” for how to approach complexities such as climate change, “much too vast to be treated head on.”39

This is where the notion ‘thought exhibition’ comes in: A useful tool in science to test a hypothesis, or to make even new discoveries, with regards to objects too big, too complex, too remote, or too impractical to treat directly, is the thought experiment. Within a sufficiently structured framework it offers, albeit imaginary, an experimental approach towards potential solutions and virtualities. Although a thought exhibition, or every exhibition, remains limited to its space – an institution, in the case of Critical Zones located in Central Europe—and time—the ‘here’ of late capitalism –, it is also a protected and experimental space where alternative futures, “a way to anticipate a situation of which there is as yet no real instance,”40 can be safely explored. Here imagination may become a projective capacity with which new worlds, beyond teleological concepts of growth, may find their vectors.

And yet, also a thought exhibition is curated, that is, laid out in a way determined by an author. In this sense, the paths it offers through the exhibition space are limited (even if there are multiple) and given or suggested (albeit if they remain open to alternatives). But maybe it is in this tension, between the curatorial concept and the visitor’s subjective experience and imagination, in this “complete uncertainty of what the visitors will do in the end in the environment that you’ve imagined for them,” 41 where new worlds can evolve. Latour repeatedly emphasized that Critical Zones, or any of his thought exhibitions, is not meant to somewhat illustrate ideas written somewhere else, but to offer a spatial configuration for exploring, testing, or modifying ideas. As he pointed out in a conversation with Hans Ulrich Obrist on the connection between his philosophy and exhibitions: “For me there is absolutely no difference between doing an exhibition, writing a piece of philosophy or doing fieldwork with ethnographic methods, or writing a play. [...] So, it’s not a migration of concept—concept is a medium, among others, and they don’t migrate, they resonate with each other. [I]t’s first of all a space—and the space is the concept.” 42

Spatio-aesthetics

The spatial arrangement invites the visitor to take different perspectives—figuratively and literally, that is, spatially – on how things and actors interact with each other. This is no trivial notion of space (as a physical container) and even goes beyond the relational approach mentioned above. The urban researcher Mustafa Dikeç emphasizes the spatial and aesthetic conditions of political or activist intervention: “Space not only gives form to and orders how this world appears, but also allows distinctive gatherings of beings—things and people—that establish relationality and open new spaces […]. Thinking politics spatially is both figurative, in the sense that it evokes spatial forms, and imagin tive, which allows for the possibility of reordering things, […] established orders and systems of representation.” 43

It is worth pointing out the parallels between Dikeç’s interdependence of space, politics, and aesthetics (“gatherings of beings – things and people”, “altering established orders of representation”) and Latour’s “political ecology” 44, particularly the thought exhibitions. When “spatialisation is fundamental to constructing, apprehending and projecting worlds and entering into relation with them” 45, then Critical Zones offers a valuable testing ground for how we may live response-able in a common world.

Dikeç understands aesthetics in the broad sense of aesthesis, as in perception by the senses or the bodily (spatial) experience. 46 Making sense of the world is an embodied phenomenological activity. If we couple this to Latour’s equally broad notion of aesthetics, “defined as what renders one sensitive to the existence of other ways of life” 47, we can underline the spatial mode of experimenting with alternative worlds. As mentioned above, here we need new forms of representations that help us to aesthetically conceive this shift. New forms offering an alternative to the iconic Blue Marble, to sufficiently represent the fragile and entangled biofilm or critical zone. “Changes in cosmology cannot be registered without changes in representation.”48 Artistic aesthetic expressions can have this capacity to render us sensitive to alternative worlds and their discontinuities to past and present.

Cosmogram

Here the notion of ‘cosmogram’ comes into play. During the January 2019 session of the Critical Zone Study Group, we had the privilege to host a lecture by the art and science historian John Tresch. His concept of cosmogram relates to objects, architectural forms, or practices which bring a given cosmology of a certain group of people at a certain point in time – a rather abstract set of shared beliefs constituting a worldview—into the aesthetically concrete. 49 Based on Latour’s assumption that we, in our attempt to orient ourselves in the New Climatic Regime, are in need of a new cosmology succeeding the globe of modernity—a reorientation he compared to the Galilean paradigm shifts of the 17 th century in science and the social order—, the cosmogram concept helped us to frame such kind of artistic representation. As example for a well-known cosmogram (at least in Jewish and Christian mythologies), Tresch refers to the Tabernacle of Moses. Here a religious-based worldview is precisely described as a model for a spatial or architectural formation representing the godly regime, where the elements of the given cosmology find their assigned place and relations to each other. Despite the example of the rather dogmatic Tabernacle, a cosmogram is not necessary static. It may provide “the basis for new interpretations and action: social relations, relations with other cultures, with natural entities, with animals, plants.”50 Thus, it is important to stress the projective capacity Tresch assigns to cosmograms, as they can enable a (note the Latourian term) “redescription, in the conditional or future tense: not the world as it is but the world as it could be.” 51 As in Critical Zones’ description of this shift and its redescription of the world towards a new common ground, “cosmograms often guide [such a] recreation and restabilization of the world.”52

Although one could make an argument for understanding the Critical Zones exhibition itself as a cosmogram—and there have been undertakings to use this concept as a curatorial approach for art exhibitions53—, here I want to reserve this term for one of Critical Zone’s key artworks, Sarah Sze’s ‘Flash Point (Timekeeper)’, a 2018 iteration of her ‘Timekeeper’ series (fig. 2).54 The Timekeeper installations investigate, among other materialities, the dichotomies of digital societies, i.e., materiality–digitality or spatiality–virtuality.55 They consist of various everyday objects and digital images (mostly found online) projected and printed on paper sheets in various sizes, most of which are mounted on fragile wooden frames. Installed in a separated and scarcely illuminated area of the ZKM ground floor, the mounted prints of ‘Flash Point (Timekeeper)’ were illuminated by projected images and complemented by projections spinning along the walls beyond the installation. Although it was difficult to say where “beyond the installation” actually was, as its space-encompassing projections as well as materials of the installation scattered on the floor made its spatial boundaries blurry.

All Timekeeper works evoke such an “immersive” effect, “like a series of experiments which envelop the surrounding architecture”. 56 It is this uncertainty of knowing where the installation ends and where one, as a viewer, enters its perimeter that constitutes an important aspect of the installation’s function as a cosmogram representing the critical zone. Although I want to use the term “function” carefully here. As Martin Guinard has pointed out while discussing with me the curatorial approach of Critical Zones, “illustration is the enemy” 57: It is not about staging concepts developed in Latour’s writings in the exhibition space, but to use the space as well as the objects and actors assembled there to actively test ideas. Artworks are not somewhat degraded to serve as illustrations of concepts developed by someone else. Martin expanded on this in his recently published obituary for Latour: “Criticisms understandably arise when philosophers curate exhibitions and use artworks merely to illustrate ideas. But in fact, we took a very different approach, which was to imagine an encounter between artists’ works and his ideas, each of which followed different trajectories.” 58 Such an encounter took place, on several occasions, between Latour and Sarah Sze. Referring to Latour’s interest in the Timekeeper series with regard to Critical Zones, Hans Ulrich Obrist emphasized the viewers’ experience of manifoldness when approaching the installation, as well as the uncertainty of their own position when investigating it more closely: They find themselves always inside the installation, sometimes partly enclosed by its material, sometimes serving as a temporary, embodied screen when crossed by a projection.59 In his comment, Obrist referred to ‘Twice Twilight’ (2020), the Timekeeper iteration installed at Sarah Sze’s solo exhibition at the Fondation Cartier in Paris,60 which is a more extensive installation compared to the work shown at ZKM. The immersive experience of the visitor is here also amplified by the building it inhibited: The façade of the Foundation Cartier, designed by the architect Jean Nouvel, is mostly made of glass and steel. Surrounded by trees and garden sections partly encompassed by glass walls, it is at times challenging to make out out the inside and outside of the building. This effect was amplified by the Timekeeper projections penetrating the glass façade. Such architectural conditions are quite different from the spatially separated Timekeeper, located within the massive walls of the ZKM building, a former ammunition factory. Reflecting on her exchange with Latour, Sze compares the piece’s fragile structure and porous boundaries to the critical zone, that is “the world as very thin, very fragile membrane of life.”61

At an artist talk between Latour and Sze on the occasion of her Foundation Cartier exhibition, which started as a tour outside of the building, Latour right away pointed out the viewer experience of an undefined spatial position: “One of the characters of your work is that the distance between the inside and the outside is put into question.”62 During the tour he continued to draw a connection between ‘Twice Twilight’ and the impossibility of an outside position in the critical zone, as in both cases “you never know when you are in and when you are out. […] The visitors are asked to subvert his or her idea of what the Earth is like. Because there is no outside, really.” The, as Martin Guinard put it, “different trajectories” of the artwork and the Critical Zones curatorial concept become intertwined as a dialogue between the artist and the philosopher. While Latour went so far, albeit jokingly, to rename the piece ‘Critical Zone’ – “because the bricolage, scaffoldings and the fragility, yet the strength and multiplicity, is exactly what lifeforms have done in the critical zone” –, Sze pointed out “one of the important things about the work, [the] fragility between what is an object, what is an image, and what is life”. 63 This conceptional convergence is not a somewhat affirmative agreement by an artist to a philosopher’s statement, but reflects an inherent characteristic of the Timekeeper works. In fact, a comparison between the fragility of Life and Timekeeper was pointed out before by the art historian Hal Foster: “A philosophy of life might be intimated here, one that cuts across biology and technology, life seen as a system that struggles with flux.”64

By describing Sarah Sze’s work method as “compositional principle”—carefully assembled of disparate pieces, always experienced anew by the changing position of an immersed viewer—Latour acknowledged the Timekeeper works as terrestrial cosmograms: Through the spatio-aesthetic encounter of the installation, “viewers can escape the dichotomy between seeing inside-out or outside-in, as if they were caught in a vortex. They become ‘composers of space’ in their own right”.65

– In memory of B.L. & P.W. –

Acknowledgements

This research has received funding from the European Union’s Horizon 2020 program under the Marie Skłodowska-Curie Grant Agreement N° 101028379.

References

1 Bruno Latour, Down to Earth. Politics in the New Climate Regime, Cambridge, Polity Press, 2018, 5.

2 Jan Zalasiewicz et al., “The technofossil record of humans,” The Anthropocene Review 1/1, 2014, p.34–43, 40. For a widely discussed ‘start date’ of the Anthropocene as potential new geological epoch, stretching from 1450 with the rise of colonization and subsequent capitalism (thus designated as ‘Capitalocene’ by Jason W. Moore) to the post-WWII ‘Great Acceleration’ (Alvin Toffler) or ‘Sixth Extinction’ (Elizabeth Kolbert), see Adele E. Clarke, “Introducing Making Kin not Population,” in Making Kin not Population, ed. Adele E. Clarke and Donna Haraway, Chicago, Prickly Paradigm Press, 2018, p.1–39. Latour designated the cosmological paradigm shift towards modernity to 1610 and the Galilean scientific revolutions; cf. Bruno Latour, “Seven objections against landing on Earth,” in: Critical Zones. The Science and Politics of Landing on Earth, ed. Bruno Latour and Peter Weibel, Cambridge, MA, MIT Press, 2020, 12–19, 15.

3 Reto Knutti, “Closing the Knowledge-Action Gap in Climate Change,” One Earth 1/1, 2019, 21–23.

4 Bruno Latour, “Thought Exhibitions”, lecture at the Museum of Contemporary Art Zagreb, September 23, 2017; http://modesofexistence.org/what-is-a-gedankenausstellung/#introduction.

5 Bruno Latour, Nous n’avons jamais été modernes. Essai d’anthropologie symétrique, Paris, La Découverte, 1991.

6 Stephen Jay Gould, “Ladders and Cones,” in Hidden Histories of Science, ed. Robert B. Silvers, New York: New York Review Books, 1995, 37–67.

7 Bruno Latour, Down to Earth, 2.

8 Earth System Science “builds upon the traditional disciplines [geology, biochemistry, etc.] but promises to provide a deeper understanding of the interactions that bind the Earth’s components in a unified, dynamical system”, Earth System Sciences Committee NASA Advisory Council (ed.), Earth System Science. A Program for Global Change, Washington: NASA, 1988, 15.

9 Alexandra Arènes, Bruno Latour, Jérôme Gaillardet, “Giving depth to the surface: An exercise in the Gaiagraphy of critical zones,” The Anthropocene Review 5/2, 2018, 20–135, 121.

10 “Life” with capital ‘L’ denotes, in Latour’s writings, life in its multitude of interrelations, including “animals, plants, bacteria, […] atmosphere, soil, rocks, seas, clouds, minerals, continents”, Bruno Latour, “Seven objections against landing on Earth,” 18.

11 Bruno Latour, “Some advantages of the notion of ‘Critical Zone’ for Geopolitics,” Procedia Earth and Planetary Science 10, 2014, 3–6, 4

12 Ibid.; cf. Haraway’s notion of ‘Chthulucene’, which prefers modes of living together to rather fatalistic implications of the Anthropocene. Staying with the Trouble. Making Kin in the Chthulucene, Durham, NC, Duke University Press, 2016.

13 Bruno Latour, Nous n’avons jamais été modernes.

14 Ibid.; cf. Haraway’s “cyborgs, hybrids, mosaics, chimeras”. Haraway, “A Cyborg Manifesto. Science, Technology and socialist-feminism in the latetwentieth century [1985],” in The Cyber-cultures Reader, ed. David Bell, Barbara M. Kennedy, London and New York, Routledge, 2001, 291–324, 313.

15 Alfred N. Whitehead, The Concept of Nature, Cambridge, Cambridge University Press, 1920, 30.

16 Phillippe Descola, Beyond Nature and Culture, Chicago and London, The University of Chicago Press, 2013 [2005], s.p.

17 Gregory Bateson, “Ecology and Flexibility in Urban Civilization [1970],” in Steps to an Ecology of Mind: Collected Essays in Anthropology, Psychiatry, Evolution, and Epistemology, Northvale, NJ and London, Jason Aronson 1987, 499–511, 510.

18 Laurent Berlant, On the Inconvenience of Other People (Durham and London: Duke University Press, 2022, 11.

19 Ibid., 9.

20 Donna Haraway, Staying with the Trouble.

21  Bruno Latour, “Some advantages of the notion of ‘Critical Zone’ for Geopolitics,” 5.

22 Bruno Latour, Face à Gaïa. Huit conférences sur le nouveau régime climatique, Paris, La Découverte, 2015; cf. Bruno Latour and Timothy M. Lenton, “Extending the domain of freedom, or why Gaia is so hard to understand,” Critical Inquiry 45/3, 2019, 659–680.

23 Bruce Clark and Sébastien Dutreuil (eds.), Writing Gaia: The Scientific Correspondence of James Lovelock and Lynn Margulis, Cambridge, Cambridge University Press, 2022.

24 Bruno Latour, “Seven objections against landing on Earth,” 18. 

25 In a talk with Sarah Sze in October 2020 he remarked that, albeit both critical zone and Gaia are scientific concepts, that “Gaia always gets people worried” due to its mythological or vitalist suggestions. Artist talk “Sarah Sze et Bruno Latour: une balade-discussion dans l’exposition ‘De nuit en jour’ – Fondation Cartier pour l’art contemporain.” Livestream October 19, 2020; archived: https://www.youtube.com/watch?v=JPQijl921f0.

26 Lynn Margulis, Symbiotic Planet. A New Look at Evolution, New York, Basic Books, 1998, 106.

27 Bruno Latour, “Some advantages of the notion of ‘Critical Zone’ for Geopolitics,” 5.

28 Lynn Margulis in a 1998 interview, in: “Symbiotic Earth. How Lynn Margulis rocked the boat and started a scientific revolution”, documentary film by John Feldman, USA, 2017.

29 Bruno Latour, “Seven objections against landing on Earth,” 14. Cf. the documentation of Latour’s performance lecture “Inside”, in: Frédérique AïtTouati and Bruno Latour, Trilogie Terrestre: Inside suivi de Moving Earths, Montreuil, Éditions B42, 2022.

30  Bruno Latour, “Seven objections against landing on Earth,” 14.

31 Bruno Latour, “Some advantages of the notion of ‘Critical Zones’ for Geopolitics,” 5.

32 Bruno Latour, “Seven objections against landing on Earth,” 14.

33  I am indebted to Martin Guinard for pointing me to this.

34 Bruno Latour, Politics of Nature. How to Bring the Sciences into Democracy , Cambridge, MA, Harvard University Press, 2004, 53.

35 Bruno Latour, “Some advantages of the notion of ‘Critical Zones’ for Geopolitics,” 3.

36 Bruno Latour, “An Attempt at a ‘Compositionist Manifesto’,” New Literary History 41, 2010: p.471–490, endnote 40.

37 Donna Haraway, Staying with the Trouble, 97.

38 Bruno Latour, “An Attempt at a ‘Compositionist Manifesto’,” 485.

39 Bruno Latour, “Seven objections against landing on Earth,” 18.

40 Bruno Latour, “Let’s touch base, in: Reset Modernity!,” ed. Bruno Latour, Cambridge, MA and London, MIT Press, 2016, p.11–23, 22. Cf. Bruno Latour, “Seven objections against landing on Earth,” 18f.

41  Bruno Latour, “Thought Exhibitions”, lecture at the Museum of Contemporary Art Zagreb.

42 Bruno Latour interviewed by Hans Ulrich Obrist, Paris, January 9, 2016; http://modesofexistence.org/what-is-a-gedankenausstellung/#introduction.

43 Mustafa Dikeç, Space, Politics and Aesthetics, Edinburgh, Edinburgh University Press, 2015, 1ff.

44 Bruno Latour, Politics of Nature.

45 Mustafa Dikeç, Space, Politics and Aesthetics, 4.

46 Mustafa Dikeç, Space, Politics and Aesthetics, 1.

47 Bruno Latour, “Seven objections against landing on Earth,” 19.

48 Ibid.

49 Based on David Damrosch’s cosmogram approach in religious studies. John Tresch, “Cosmogram,” in Cosmograms, ed. Melik Ohanian and Jean-Christophe Royoux, New York, Lukas & Sternberg, 2005, .67–76, 67.

50 John Tresch, “Cosmogram,” 69.

51 Ibid., 75 (emphasis in the original).

52 Ibid., 74.

53 Ohanian and Royoux (eds.), Cosmograms. Joshua Simon proposes the notion of cosmogram “to look at the art exhibition as a model of the world as it appears to itself.” Simon, “The Exhibition as Cosmogram,” Parse 13/2, 2021; https://parsejournal.com/article/the-exhibition-as-cosmogram/.

54 Bettina Korintenberg, one of the co-curators, confirmed my hypothesis of the key function of Sze’s installation for this show, as well as its position as a cosmogram in Bruno’s and John Tresch’s sense (email correspondence between Korintenberg and the author, October 10 and 11, 2022).

55 Sarah Sze in conversation with Hans Ulrich Obrist, in: The Hero Winter Annual, 2020, 190–203, 193.

56 “Introduction,” in Sarah Sze: Timekeeper, ed. Rose Art Museum, New York, Gregory R. Miller & Co., 2017, 4.

57 Martin Guinard in a conversation with the author during the Critical Zones Study Group session in May 2019.

58 Martin Guinard, “Homage to Bruno Latour,” e-flux Journal 131, 2022, https://www.e- flux.com/journal/131/502967/homage-to-bruno-latour/.

59 Sarah Sze in conversation with Hans Ulrich Obrist, 199.

60 Sarah Sze, “De nuit en jour”, Fondation Cartier pour l’art contemporain, Paris, October 24, 2020 – May 30, 2021.

61 Sarah Sze in conversation with Hans Ulrich Obrist, 200.

62 “Sarah Sze et Bruno Latour: une balade-discussion dans l’exposition ‘De nuit en jour’.”

63 Ibid.

64 Hal Foster, “The art of teetering,” in Sarah Sze: Timekeeper, ed. Rose Art Museum, New York, Gregory R. Miller & Co., 2017, p.103–201, 197.

65 Bruno Latour, “The verifiable Image of the World?,” contribution to the exhibition catalogue Sarah Sze, De nuit en jour / Night into Day, Paris: Publication Fondation Cartier pour l’art contemporain, 2020, here quoted from the manuscript (transl. Lucas Faugère); http://www.bruno-latour.fr/sites/default/files/165-SARAH-SZE-PARIS_GB.pdf. 2.

Bibliography

“Introduction,” in Sarah Sze: Timekeeper, ed. Rose Art Museum, New York, Gregory R. Miller & Co., 2017.

“Symbiotic Earth. How Lynn Margulis rocked the boat and started a scientific revolution”, documentary film by John Feldman, USA, 2017.

Frédérique Aït-Touati, Bruno Latour, Trilogie Terrestre: Inside suivi de Moving Earths, Montreuil: Éditions B42, 2022. Arènes, Alexandra, Bruno Latour, Jérôme Gaillardet, “Giving depth to the surface: An exercise in the Gaiagraphy of critical zones,” The Anthropocene Review 5/2, 2018:120–135.

Gregory Bateson, “Ecology and Flexibility in Urban Civilization [1970],” in Steps to an Ecology of Mind: Collected Essays in Anthropology, Psychiatry, Evolution, and Epistemology, Northvale, NJ and London: Jason Aronson 1987, 499–511.

Laurent Berlant, On the Inconvenience of Other People, Durham and London, Duke University Press, 2022.

Bruce Clark, Sébastien Dutreuil (eds.), Writing Gaia: The Scientific Correspondence of James Lovelock and Lynn Margulis, Cambridge, Cambridge University Press, 2022.

Adele E. Clarke, “Introducing Making Kin not Population,” in Making Kin not Population, ed. Adele E. Clarke and Donna Haraway, Chicago, Prickly Paradigm Press, 2018, 1–39.

Phillippe Descola, Beyond Nature and Culture, Chicago and London, The University of Chicago Press, 2013 [2005].

Mustafa Dikeç, Space, Politics and Aesthetics, Edinburgh, Edinburgh University Press, 2015.

Earth System Sciences Committee NASA Advisory Council (ed.), Earth System Science. A Program for Global Change, Washington, NASA, 1988.

Hal Foster, “The art of teetering,” in Sarah Sze: Timekeeper, ed. Rose Art Museum, New York, G. R. Miller, 2017, 103–201.

Stephen J. Gould, “Ladders and Cones,” in Hidden Histories of Science, ed. Robert B. Silvers, New York, New York Review Books, 1995, 37–67.

Martin Guinard, “Homage to Bruno Latour,” e-flux Journal 131, 2022, https://www.e-flux.com/journal/131/502967/homage-to-bruno-latour/.

Donna Haraway, “A Cyborg Manifesto. Science, Technology and socialist-feminism in the late-twentieth century [1985],” in The Cybercultures Reader, ed. David Bell and Barbara M. Kennedy London and New York, Routledge, 2001, 291–324.

Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham and London, Duke University Press, 2016.

Reto Knutti, “Closing the Knowledge-Action Gap in Climate Change,” One Earth 1/1, 2019, 21–23.

Bruno Latour, “An Attempt at a ‘Compositionist Manifesto’,”

New Literary History 41, 2010, 471–490.

Bruno Latour, “Let’s touch base, in: Reset Modernity!,” ed. Bruno Latour, Cambridge, MA and London, MIT Press, 2016, 11–23.

Bruno Latour, “Some advantages of the notion of ‘Critical Zone’ for Geopolitics,” Procedia Earth and Planetary Science 10, 2014: 3–6.

Bruno Latour, “The verifiable Image of the World?,” contribution to the exhibition catalogue Sarah Sze, De nuit en jour / Night into Day, Paris, Publication Fondation Cartier pour l’art contemporain, 2020; manuscript (transl. Lucas Faugère): www.brunolatour.fr/sites/default/files/165-SARAH-SZE-PARIS_GB.pdf. 2. Latour, Bruno, “Thought Exhibitions”, lecture at the Museum of Contemporary Art Zagreb, September 23, 2017; http://modesofexistence.org/what-is-a-gedankenausstellung/#introduction.

Bruno Latour, Sarah Sze, artist talk “Sarah Sze et Bruno Latour: une balade-discussion dans l’exposition ‘De nuit en jour’ – Fondation Cartier pour l’art contemporain”, Livestream October 19, 2020; archived: www.youtube.com/watch?v=JPQijl921f0.

Bruno Latour, Timothy M. Lenton, “Extending the domain of freedom, or why Gaia is so hard to understand,” Critical Inquiry 45/3, 2019: 659–680.

Bruno Latour, Down to Earth. Politics in the New Climate Regime, Cambridge, Polity Press, 2018.

Bruno Latour, Face à Gaïa. Huit conférences sur le nouveau régime climatique, Paris, La Découverte, 2015.

Bruno Latour, interviewed by Hans Ulrich Obrist, Paris (January 9, 2016); http://modesofexistence.org/what-is-a- gedankenausstellung/#introduction.

Bruno Latour, Nous n’avons jamais été modernes. Essai d’anthropologie symétrique, Paris, La Découverte, 1991. Latour, Bruno, Politics of Nature. How to Bring the Sciences into Democracy, Cambridge, MA, Harvard University Press, 2004.

Bruno Latour, “Seven objections against landing on Earth,” in: Critical Zones. The Science and Politics of Landing on Earth, ed. Bruno Latour and Peter Weibel, Cambridge, MA, MIT Press, 2020, 12–19.

Lynn Margulis, Symbiotic Planet. A New Look at Evolution, New York, Basic Books, 1998.

Joshua Simon, “The Exhibition as Cosmogram,” Parse 13/2, 2021, https://parsejournal.com/article/the-exhibition-as-cosmogram/.

Sarah Sze, in conversation with Hans Ulrich Obrist, in: The Hero Winter Annual, 2020, 190–203.

John Tresch, “Cosmogram,” in Cosmograms, ed. Melik Ohanian, Jean-Christophe Royoux, New York, Lukas & Sternberg, 2005, p.67–76.

Alfred N. Whitehead, The Concept of Nature, Cambridge, Cambridge University Press, 1920.

Jan Zalasiewicz et al., “The technofossil record of humans,” The Anthropocene Review 1/1, 2014, 34–43

Author Biography

Daniel Irrgang is a postdoctoral researcher (Marie Skłodowska-Curie fellow) at the Centre Art as Forum (Dept. of Arts and Cultural Studies), University of Copenhagen, and associated researcher at the Weizenbaum Institute, Berlin. His current research focusses on the concept of ‘thought exhibition’ (Latour/Weibel). He holds a PhD in media studies on diagrammatics and expanded mind theories and is author and editor of numerous books on the history and theory of media, culture, and the arts.

Curation as Research-Creation: Speculating on the Future of Art and Technology Festivals

Maurice Jones, Meaghan Wester, Marek Blottiere

Concordia University, Concordia University, INRS
Montréal, Canada
mauriceortegajones@gmail.com, meaghan.wester@gmail.com, marek.blottiere@gmail.com

Abstract

This paper explores a renewed approach to curation as research-creation (CRC) through its practical application in the annual art and technology festival. CRC envisions a shift in curation from a care for objects to a care for the emerging social relations of the curatorial project in a shared quest of meaning making.

We set out with outlining the features of CRC as interdisciplinary, concerned with programmatic boundary objects, and centered around the unfolding event trajectory – the forms and methods that facilitate affective encounters. Following we outline how this approach to curation unfolds in practice through the case study of the Fest-Forward workshop series that speculates on the future of art and technology festivals. Concluding we summarize how this workshop series showcases the potential of CRC’s shift of attention from a mere presentation of artworks towards the facilitation of interdisciplinary and cross-cultural encounters that enroll artists, curators, and audiences.

Keywords

art, curation, science, technology, interdisciplinarity, speculative design, techno culture, festival, research-creation

DOI

10.69564/ISEA2023-67-full-Jones-et-al-Curation-as-Research-Creation

Introduction

After two and a half hours of speculating on the future of festivals, one workshop participant remarked:

“Inherent in these art projects is that they already tackle all these big questions. I mostly see the lack in the structures that we as festival makers provide to facilitate the spaces in which these artworks reach their full potential.”

This is the central issue that our workshop series titled Fest-Forward is addressing since August 2022. The workshops take place in a variety of different formats and localities around the world. Past activations include Canada, the Netherlands, and Japan, and future renditions aim to include wider geographies. Inspired by a reimagined approach to curation as research-creation firmly rooted at the intersection of art, technology, and governance, these workshops explore the transforming role of the art and technology festival as a site of joint-meaning making between artists, researchers, industry professionals, policymakers, and audiences. In other words, by reenvisioning how the festival comes into being and shapes the encounters of people, we seek to address how art can reach its full potential.

The following paragraphs set out with an introduction to the theoretical reimagination of Curation as Research-Creation (CRC), which presents the base considerations for this project. Following, the paper introduces the methodological approach towards these workshops, which are rooted in speculative design and liberating structures. We then venture into describing the case study of the inaugural Fest-Forward: Imagining Future Festivals workshop, which took place in August 2022 as part of the 23rd edition of the MUTEK Festival and Forum. The concluding paragraphs summarize key theoretical and methodological findings of this workshop in transforming the festival as a social actor.

Curation as Research-Creation

Curation as Research-Creation (CRC) presents an overall shift from understanding curation as merely an act of ‘putting things together ’towards the curatorial project itself being a site of inquiry and meaning making. ¹  CRC is thus inherently concerned with, as Loveless puts it, “not only what methods offer at the level of investigation [...], but also what form might best fit the content of the research at the level of publication.” ² This reinterpretation of curation as not only form but method, then embraces the key idea of research-creation as a continuous thinking-making process. ³  Emerging out of the intersection of critical curatorial studies, science and technology studies, and artistic and curatorial practices in the art, science, and technology realm, CRC is specifically attuned to addressing the pressing sociotechnical questions of our time.

Curation is interdisciplinary

CRC ventures away from the centrality of the curator or the artist in the unfolding of the curatorial project. In contrast, curation becomes an interdisciplinary endeavor, equally acknowledging the extensive involvement of people and labor in preparation, as well as the centrality of audience participation in co-creation. Galison, in an investigation of inter-disciplinary encounters in science and technology, framed this sort of space as a trading zone. ⁴ Similarly, Dekker argues that the emergence of digital art in online, offline, and hybrid spaces marks a shift from the paradigm of collecting and presenting art towards networked co-curation. ⁵ This networked co-curation fundamentally questions traditional museological values, the participation of publics, and the field of art history. ⁶ It shifts, as Truman points out, the attention from who produces knowledge towards how knowledge is produced in joint inquiry, which underlines the notion of curatorial practice as an act of caring for social relations. ⁷ 

The curatorial project then becomes an interdisciplinary trading zone of networked co-curation. Defining the festival as a trading zone of co-curation shifts attention from merely the content, which is the artistic and discursive program, towards the methods of exchange that lead people from different professional, disciplinary, intersectional, and cultural backgrounds to come together in joint exploration of ideas at the intersection of art and technology. In summary, CRC presents a shift of focus, as Manning and Massumi suggest, to “more than programmation but catalytic event unfolding”, in pursuit of a shared interdisciplinary inquiry. ⁸

Program as boundary object

Where does that leave the program—the content—then? While content is still key, its role is redefined from mere display of artistic works towards taking the role of presenting boundary objects. Boundary objects, as put forward by Star & Griesemer, are concrete or abstract objects that structure interdisciplinary inquiry. ⁹ In Bruno Latour’s words the thing-object as a shared matter of concern is what brings people together. ¹⁰ They are general enough in a way that actants from different social worlds can gather around them in joint exploration, while simultaneously acknowledging the specificity they potentially carry across disciplines, professional fields, intersectional and cross-cultural environments. ¹¹

Boundary objects thus carry the power to assemble people not because there is agreement but exactly because opinions are divided and varied. ¹² Rethinking the festival program as establishing certain boundary objects thus frames content beyond mere aesthetic display as an invitation to or a prompt for participants to jointly explore throughout the unfolding of the event. In other words, program as boundary object is better attuned towards mobilizing the powerful meanings and critiques that are commonly explored by the most forward-thinking artists in their fields.

The centrality of trajectory: forms and methods

The way the event is conditioned in terms of forms and methods foster exchange in meaning-making. The event trajectory thus becomes central. Trajectory refers to the temporally emergent configuration and reconfiguration of forms and methods of engagement in the unfolding of the curatorial event. Forms refer to the combination of space (physical, virtual, hybrid, concert hall, club, gallery, auditorium etc.), and format (performance, exhibition, conference, workshop, roundtable etc.). Methods, on the other hand, present the modes of engagement within these interdisciplinary encounters. ¹³

Forms and methods are inherently entangled. Certain methods ask for certain event spaces and engagement formats. Simultaneously, certain event spaces and engagement formats urge for certain methods to be employed. ¹⁴ It is important to note, however, that these combinations stem from very specific normatively laden histories. ¹⁵ The traditional Western concert hall for example not only calls for specific engagement formats, such as chamber music, but also encourages specific ways of performing and listening to music. Addressing these normatively laden spaces by inhabiting them in unconventional ways, such as shifting positionalities of audiences and human/non-human performers thus actively encourages decolonizing efforts of these traditional spaces of artistic engagement. As such, experimentation with forms and methods transcending physical and virtual spaces, and a variety of engagement formats are central to CRC.

Transforming curation through research-creation

CRC then acknowledges the emergent nature and reiterative processes of meaning-making throughout the entirety of the curatorial project. ¹⁶ Especially within the frame of the annual festival curation, which neither has a fixed beginning, nor concludes with the presentation of the festival edition. It is rather a process that continuously unfolds, shapes, and reshapes before, during, and after the annual event. In addition, CRC acknowledges the inherent interdisciplinarity of the curatorial project, in both preparation and presentation. Curation brings together curators, organizing teams, communications experts, artists, producers, researchers, and a wide range of external partners local and international, as well as their non-human counterparts.

Beyond heightened attention towards the forms, methods, and interdisciplinarity of the curatorial project, CRC acknowledges the fact that both thinking and making processes take place and intervene in the real-world. As such, CRC makes the researcher-curator accountable for the need of embracing an ethical responsibility and careful deliberation towards all forms and methods employed, spanning human and non-human subjects in the curatorial project. ¹⁷ 

Reinterpreting the festival within CRC thus frames the event as a methodological approach towards investigating the world, rather than a mere object of inquiry or an aesthetic display. The following paragraphs outline how employing this alternative approach towards curation in the frame of speculative workshops as part of the MUTEK festival unfolded in practice.

Fest-Forward: Imagining Future Festivals

In August 2022, the 23rd edition of the MUTEK festival took place over the course of 6 days in Montreal, Canada. After two pandemic years, which saw hybrid activities, the festival returned to in-person events including international artists and professionals visiting. On the opening day of the festival, we hosted a speculative workshop titled Fest-Forward: Imagining Future Festivals as part of the professional day time program, the MUTEK Forum.

Workshop design: interdisciplinarity and boundary object

The workshop invited an interdisciplinary and cross-cultural group of artists, curators, technologists, policy-makers, and members of the festival audience to jointly investigate the transforming role of the festival as a social actor in (post-)pandemic times. Employing methods of speculative design and liberating structures, workshop participants were mobilized by the boundary object of an imaginary future festival.

Through three paradoxical, wicked questions the workshop prompted participants to speculate on how an imaginary future festival might address questions of existing structures of oppression, rapid advances in science and technology, and/or environmental and climate crisis. More than being presumptuous in believing that an art and technology festival might solve these issues, the prompts were aimed at inspiring imagination if and if so, what it might be an art and technology festival could do. As Dunne & Raby put it, employing “the idea of possible futures and using them as tools to better understand the present and to discuss the kind of future people want”. ¹⁸ In other words, how could festivals become sites of collective meaning-making surrounding the pressing questions of our time by speculating about alternative futures?

In keeping with speculative design approaches participants were guided through the materialization of the boundary object in the form of creating a speculative newspaper headline and lead paragraph outlining how an imaginary future festival employs certain concrete tools or activities to address the wicked questions. This concrete output situated the potential, imaginary future in the here and now.

Workshop design: trajectory

The workshop took place at the Hexagram spaces located on the 4th floor of the Pavillon des Sciences Biologiques at Université du Québec à Montréal. The multifunctional space was laid out with three table islands seating seven people each for a total of 21 participants. Each table island was equipped with a whiteboard and crafting materials including article templates, pens, and post-it notes. The workshop was documented via an ambisonic audio recorder placed in the center of the room, by the festival photographer, and the researcher’s ethnographic notes.

Manifesting the boundary objects through the wicked questions and the concrete, material output the workshop trajectory unfolded in three distinct, yet interlinked phases. The first 30 minutes were spent with an introduction to the workshop and its methodology both conceptually and in practice. Split into three teams at three table islands, group work was generally structured by the 1-2-Group method. At first participants were prompted to consider posed questions individually for three minutes, followed by building pairs within teams to continue discussion for another seven minutes. Concluding the group would get together to discuss initial findings. The 1-2-Group method is specifically attuned to giving voices to people that might be silenced in larger group discussions. ¹⁹ In addition, it presented an effective trust-building exercise, especially in the paired-up phase. We first employed the 1-2-Group method as an ice-breaker during the introduction phase of the workshop.

After sharing the prompts with participants, we employed two phases of 1-2-Group, each lasting 20 minutes. During the first-round participants were to decide upon which wicked question they would address and brainstorm on how they might address it. The second round was guided by trying to answer the six journalistic questions (What? Who? When? How? Why? Where?) in preparation for making the newspaper article.

Following a short break, teams were then tasked with crafting the article. Participants were provided with printed article templates, whiteboards, and digital equivalents to facilitate creation. Concluding, teams presented their articles and thus their imaginary future festivals to the other groups, followed by an open discussion on both resulting speculation and engagement methods.

Joint speculation on the future of festivals

All three groups appropriated the method, prompts and template very differently. The emergent imaginary festival projects reflect the interdisciplinarity of the teams and the ways they appropriated the methods.

Group A’s interactions can be described as democratic, consensus-based, and self-organized. As a reaction to the proposed time management, they abandoned the 1-2-Group structure during the second round. Working through the third wicked question “How can future festivals deal with equity, diversity, and inclusivity while continuing to profit from existing structures of domination and exploitation?”, this team’s output explored how to reinvent the festival spatially to decolonize many of its premises.

Local, decentralized but nonetheless networked is how this team envisioned the future of festivals. The way they proposed this could be achieved is through the invention and deployment of a multimedia campfire toolkit. These toolkits could be distributed to a variety of locations, near and far, urban, and rural and serve as a gathering place for communities to come together. The toolkit would be networked, solar powered, have projection, speakers, live translation, the possibility to video call other chapters of this decentralized festival taking place. Their output emphasized accessibility, the need for communion, and the necessity to decentralize the traditionally urbanite festival. This focus on horizontality would enable, according to this team, oral transmission of knowledges and serendipitous connections. Group A made a point to work through how financially inaccessible urban festival-going and how unsustainable flying to be co-present in festival sites can be, especially at scale. Their article started with “Finally a festival that acknowledges the past, to honor and imagine the future!”

Group B followed the structure rigorously (even as they could hear the other teams abandoning the 1-2-Group method). Participants also made use of all the tools available namely the whiteboard, physical and digital templates. Their discussions were animated and energetic. The output they proposed was rhizomatic, complex with an emphasis on emergent and horizontal organization. This team worked through the third wicked question: “How can future festivals deal with equity, diversity, and inclusivity while continuing to profit from existing structures of domination and exploitation?”

Group B’s output emphasized festivals' responsibility to redistribute both power and resources. They also emphasized how systemic justice requires climate justice. If festivals act as platforms for artists, this team emphasized the duties that come with such a role. Diligent archiving practices, redistributing resources as education (legal, financial, and harm-reduction), and ensuring festival organizers reflect the community they serve are all practices this team proposed to implement in their future festival. For this team, the institution of the festival plays a key role—much like museums—in community, public life, shaping and sharing knowledges. It is as such that festivals carry responsibilities and duties to serve the people they represent and engage with.

Group C resisted the methodology and did not follow any of the instructions. Starting with the instruction to choose a wicked question to work on, to the 1-2-Group time management schedule, to the templates; this team can be described as having gone rogue. Professional backgrounds may have contributed to these group dynamics: most of them are used to being in positions of leadership and the proposed method required them to let go and trust the structure. The team struggled to focus on a singular wicked question which exacerbated disagreement within the team.

This team’s interactions were marked by strong characters, misunderstanding, and stress. According to our ethnographic notes, this team attempted to think of a future festival that would reduce its reliance on digital technologies and tried to re-incorporate analog activities in their programs, such as bicycles. Either satirically or ironically, their output was algorithmically generated; they used an application based on the Artificial Intelligence (AI) language model GPT-3 to write their article. As they realized they were failing to democratically organize, they turned to technology to flatten disagreements. Nonetheless, this team enacted a certain degree of reflexivity as they titled their article “Festival curators fail to detech their programs.” They thus acknowledged and illustrated the gap between their imagined festival and their employed team praxis.

In the collective discussion following the teams ’presentation, the question of the place of the Art in the festival was raised. Two poles emerged in collectively speculating on the future of festivals through the wicked questions. On the one hand, participants decried the need to recenter art and music despite all the broader social, technological, and environmental considerations.

One participant’s feedback captures this well: “I felt in the end that the idea of art content was not touched upon, but there were some valuable ideas about reaching out to local and international communities and bringing their voice into the programming.” On the other hand, a festival maker raised how for the most part, art and artists are already engaging with these issues and the responsibility of institutions, such as festivals, to create the necessary infrastructure to host and facilitate these broader societal shifts.

Curation as Research-Creation in practice

The Fest-Forward workshop series is a practical example of how attention towards curation as an unfolding of trajectories of interdisciplinary and cross-cultural encounters shifts attention from mere content towards how people come together in joint meaning-making. In other words, curation as caring for emergent social relations throughout the curatorial project. The potential of this approach to facilitate these interdisciplinary encounters is reflected by additional feedback we received from participants. As one participant stated:

“I found the format of the workshop was engaging and efficient, and that it probed productive discussions and group dynamics. I’ve found the process of writing the article and collectively designing the project more interesting than the resulting paper itself.”

This underlines how the boundary object of the wicked questions materially represented by the speculative article was effective in facilitating the groups ’encounter. Talking about the interdisciplinary and cross-cultural aspect of the workshop format, another participant remarked:

“I found the exchange experience very rich in terms that we all came from different roles around culture—from festival managers to music and art curators and curious people—and the different tasks exposed each of us to speculate from different roles.”

In conclusion, the workshop was successful in raising questions about the transforming role of the festival as both a site of joint meaning making and in taking responsibility as a social actor. For example, conceptualized as a site of empowerment for artists and cultural actors, one group took the festival as a means of highlighting more broadly the social difficulties of the cultural milieu to develop equitably. In a more local and community-centered approach, one group presented the festival by associating it with the metaphor of the campfire. Another participant remarked:

“My takeaway would be that future festivals will need to be collaborative, non-competitive and networked, to work together to make sure their voices are heard in a future that will most likely still be dominated by bigger players' capitalist interests.” 

This notion is further underlined by the following feedback received:

“Attending the workshop reignited my passion for festival planning and broadened my perspective on initiatives that me and my company can take to make events more inclusive and accessible.”

Conclusion: trajectories all the way

While the preceding paragraphs outline the micro-trajectories and their effectiveness during the unfolding workshop, the workshop itself is situated in larger trajectories. Held on the first day of the festival the workshop was aimed at inspiring participants to view the remaining festival activities, talks, installations, performances, and encounters with renewed vigor.

As such, the workshop was strategically placed to interplay with the unfolding festival trajectories and thus presented a perceived reconfiguration of forms and methods of engagement for participants. We tracked the participants ’ reshaped perspectives via a festival diary, asking each of the participants individually and at randomly selected times throughout the remainder of the festival to give feedback on how their participation might have shifted their perspective of the festival in its unfolding.

The workshop is part of a series of activations, which under the premise of the MITACS-funded research project Festival as Methodology implements CRC in practice as part of the annual festival edition of MUTEK and its partner festivals in Argentina, Germany, Japan, Mexico, and Spain. Underlining the importance of trajectories this initial workshop thus was not only concerned with the micro-trajectories of its unfolding but firmly embedded within the unfolding macro trajectories of the 2022 festival edition and the 3-year research project.

While all of these unfold in different temporalities what unites them is an underlying concern for their unfolding in terms of the forms they take, spaces they inhabit, and methods they employ. The Fest-Forward workshop series thus is representative of Curation as Research-Creation as inherently concerned with the ongoing configuration and reconfiguration of trajectories that facilitate the emergent social relations of interdisciplinary encounters in joint meaning-making.

Acknowledgements

This research is part of the MITACS-funded research project Festival as Methodology.

References

1 Maurice Jones, “Curatorial Practice as Research-Creation: From aesthetic display to interdisciplinary encounter", From aesthetic display to interdisciplinary encounter", Unpublished Manuscript, 2021, 27.

2 Natalie Loveless, “Critical Conversations 2021: Natalie Loveless: How to Make Art at the End of the World: Revisited", College of Arts and Science - University of Saskatchewan, February 11, 2021.

3 Chris Salter, “Chris Salter on Research-Creation”, Hexagram, Montreal, 2019, Hexagram website accessed April 23, 2022, http://rec.hexagram.ca/index.php/episode-0/portrait-chris-salter,

4 Peter Galison, “Trading Zone: Coordinating Action and Belief (1998 abridgment).” In The Science Studies Reader, ed. Mario Biagioli, London UK, Routledge, 1999, 137-160.

5 Annet Dekker, Curating Digital Art: From Presenting and Collecting Digital Art to Networked Co-curation, Amsterdam: Valiz, 2021.

6 Annet Dekker, Curating Digital Art: From Presenting and Collecting Digital Art to Networked Co-curation, Amsterdam: Valiz, 2021.

7 Sarah E. Truman, Feminist speculations and the practice of research-creation: Writing pedagogies and intertextual affects, London UK, Routledge, 2021.

8 Erin Manning, Brian Massumi, Thought in the Act: Passages in the Ecology of Experience, Minneapolis, University of Minnesota Press, 2014, 89.

9 Susan Leigh Star, James R. Griesemer, “Institutional Ecology, "Translations ’and Boundary Objects: Amateurs and Professionals in Berkeley’s Museum of Vertebrate Zoology,” Social Studies of Science 19 (3), 1989, 387–420.

10 Bruno Latour, “From Realpolitik to Dingpolitik – An Introduction to Making Things Public”. in Making Things Public—Atmospheres of Democracy. ed. Bruno Latour and Peter Weibel, Cambridge Massachusetts: MIT Press, 2005.

11 Susan Leigh Star, James R. Griesemer, “Institutional Ecology, "Translations and Boundary Objects”, 387–420.

12 Bruno Latour, “From Realpolitik to Dingpolitik – An Introduction to Making Things Public”.

13 Maurice Jones, “Curatorial Practice as Research-Creation”, 27.

14 Maurice Jones, “Curatorial Practice as Research-Creation”, 27.

15 Dylan Robinson, Hungry listening: Resonant theory for indigenous sound studies, Minneapolis, University of Minnesota Press, 2020, 288.

16 Maurice Jones, “Curatorial Practice as Research-Creation”, 27.

17 Maurice Jones, “Curatorial Practice as Research-Creation”, 27.

18 Dunne, Anthony, Fiona Raby, Speculative Everything: Design, Fiction, and Social Dreaming, Cambridge Massachusetts, MIT Press, 2013, 2.

19 Henri Lipmanowicz and Keith McCandless, The surprising power of liberating structures: Simple rules to unleash a culture of innovation, Seattle, WA, Liberating Structures Press, 2013, 366.

Bibliography

Annet Dekker, Curating Digital Art: From Presenting and Collecting Digital Art to Networked Co-Curation. Amsterdam, Valiz, 2021.

Anthony Dunne, Fiona Raby, Speculative Everything: Design, Fiction, and Social Dreaming. Cambridge, Massachusetts, MIT Press, 2013.

Peter Galison, “Trading Zone: Coordinating Action and Belief (1998 Abridgment)”, In The Science Studies Reader, edited by Mario Biagioli, 1999, 137–60, London, United Kingdom, Routledge.

Maurice Jones, “Curatorial Practice as Research-Creation: From Aesthetic Display to Interdisciplinary Encounter”, Un-published Manuscript, 2021.

Bruno Latour, Peter Weibel, “From Realpolitik to Dingpolitik – An Introduction to Making Things Public”, In Making Things Public: Atmospheres of Democracy, Cambridge, Massachusetts, MIT Press, 2005.

Henri Lipmanowicz, Keith McCandless, The Surprising Power of Liberating Structures: Simple Rules to Unleash a Culture of Innovation, Seattle, WA, Liberating Structures Press, 2013.

Natalie Loveless, “Critical Conversations 2021: Natalie Loveless: How to Make Art at the End of the World: Revisited”, College of Arts and Science - University of Saskatchewan, February 11, 2021.

Erin Manning, Brian Massumi, Thought in the Act: Passages in the Ecology of Experience, Minneapolis: U of Minnesota Press, 2014.

Dylan Robinson, Hungry Listening: Resonant Theory for Indigenous Sound Studies, Minneapolis, U of Minnesota Press, 2020.

Chris Salter, “Chris Salter on Research-Creation”, Rec Hexagram, April 2019, http://rec.hexagram.ca/index.php/episode-0/portrait-chris-salter.

Susan Leigh Star, James R. Griesemer, “Institutional Ecology,"Translations" and Boundary Objects: Amateurs and Professionals in Berkeley’s Museum of Vertebrate Zoology, 1989, 1907-39. ”Social Studies of Science 19 (3), 387–420, https://doi.org/10.1177/030631289019003001.

Sarah E. Truman, Feminist Speculations and the Practice of Research-Creation: Writing Pedagogies and Intertextual Affects, London, United Kingdom, Routledge, 2021. 

Feeling One’s Way: In Search of a Symbiotic Vocabulary of the Virtual

Olga Kobryn1,4, Matthieu Couteau1,5, Rémi Sagot-Duvauroux1,2, Sophie Balcon-Fourmaux1, François Garnier2, Rémi Ronfard2,3, Guillaume Soulez1

Affiliation(s): 1 IRCAV & IRMÉCCEN, Sorbonne Nouvelle University, Paris, France / 2 École des Arts Décoratifs  PSL (EnsadLab), Paris, France / 3 Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, Grenoble, France / 4 Cerilac, Paris Cité University, 5University Panthéon-Sorbonne
olha.kobyn@gmail.com, matthieu.couteau@sorbonne-nouvelle.fr remi.sagot@gmail.com, sophie.balcon@sorbonne-nouvelle.fr, francois.garnier@ensad.fr, remi.ronfard@ensad.fr, guillaume.soulez@sorbonne-nouvelle.fr

Abstract

This article aims to present the collective research that has been made in the framework of the research seminar "Vocabulary of the Virtual" organized by IRCAV (Institute of Research on Cinema and Audiovisual Studies), Sorbonne Nouvelle and the “Spatial Media” group, EnsadLab. The main topic is to clarify the notions that refer to the concept of the "Virtual" in order to define it through an interdisciplinary approach according to different fields of science (aesthetics, philosophy, film theory, sound theory, ergonomics, design, engineering, cognitive sciences, etc.). This article presents many different conceptual tools such as cartography, mental maps, notional diagrams with several dimensions, etc., that have been conceived over the last three years, to show how the reflection on the concept of the Virtual was first established and constructed, and how it has been developed. The notion of symbiosis seems to be defined as a structuring notion of the concept of the Virtual across the process of anchoring the levels of virtuality inside technological devices and concrete sites, as well as inside the physical body of the VR users. The user’s body serves as a catalyst for the concept of the Virtual which then becomes organic.

Keywords

Virtual Reality, Symbiosis, Aesthetics, Philosophy, Experience, Phenomenology, Design, Media Studies.

DOI

10.69564/ISEA2023-68-full-Kobryn-et-al-Feeling-Ones-Way

Introduction and theoretical objectives of the research seminar
“Vocabulary of the Virtual”

This article aims to present the work that has been developed since 2020, during the research seminar "Vocabulary of the Virtual" (IRCAV, Sorbonne Nouvelle /Spatial Media, EnsadLab). The main point was to clarify, through a deliberately interdisciplinary approach (aesthetics, philosophy, film theory, sound theory, ergonomics, design, engineering, cognitive sciences, etc.) the theoretical notions that constitute our experience of the concept of the virtual from its concrete manifestations into the immersive devices and new film and media forms resulting from virtual, mixed, and augmented reality technologies. This research, the stages of which are reported here, has materialized through a certain number of experimental methodological conceptual tools such as dictionaries, cartographies, mental maps, and notional diagrams with several dimensions, proposed with the aim of grasping the concept of virtual from its relation to the adjacent philosophical and theoretical concepts.

The initial problem of our research seminar comes from an observation of a lack of precision regarding the vocabulary for analyzing new artistic and media forms emerging out of the use of new digital and cybernetic technologies, the virtual, mixed, and augmented reality devices. This lack of vocabulary, from the first stage of the analytical gesture, namely the description of the audiovisual structures of immersive spaces and their interactions with the user or the immersant, required a deepening of the definitions and terms that describe the aesthetic units and sets that make up the experience of virtual reality because, as Daniel Mestre and Philippe Fuchs remind us, “virtual reality is, in essence, an ambiguous concept. It is difficult to completely distinguish what the responsibility of technology is (virtual reality as a digital, mechanical, electronic tools, etc.) and what the responsibility of experience is (virtual reality as a subjective construction).” ¹

Thus the initial question, which has consisted in verifying the viability of the vocabulary of traditional film analysis1 applied to VR contents, has permitted to forge a broader semantic field of study, focused on the definition of the concepts involved in the construction of immersive levels of reality, in particular the hybridization factor according to Milgram's continuum principle (between digital or cybernetics and psychical reality, which includes, among others, Michel Serres ² concept of "hors-là" as well as the concept of fiction, of "narrative presence" ³, etc.). In this regard, the very definition of the term "virtual reality” poses a problem from a semantic point of view: the term described as an oxymoron by

Philippe Fuchs comes, following Jean-Paul Papin's remark ⁴ , from the English expression introduced by Jaron Lanier in the 1980s where the meaning of "virtual" ("in fact, "practically") seems in the first place to reduce the immersive device to the act of simulacrum, to a production of an immaterial world where it would necessarily miss something compared to the real world. Our working hypothesis consisted first in verifying the relationship between immersive environments and the very concept of reality, which required in-depth reflection on the definitions of this couple of problematic concepts that is real/virtual, based on philosophical theories, which were able to raise this question long before the appearance of new technologies. Following this idea of a lack, in particular of materiality (of density of matter) or following the idea of interactivity (as a fictional, potentially open word and which requires the interactive presence of the spectator), the term of virtual is often confused with notions such as immaterial, possible, potential etc. and is defined either by negativity in relation to reality or tends to become a generality, a portmanteau word, even a label for commercial products. It is, however, a strong and autonomous philosophical concept, whose contemporary definition we owe to Gilles Deleuze’ work who, inspired by the proposals of Henri Bergson ⁵, reverses the classical Aristotelian position, and replaces "power" by " virtual”, by proposing the famous double opposition: “if the real is opposed to the possible, the virtual, for its part, is opposed to the actual” ⁶ . Deleuze thus endows the virtual with "full reality, as virtual", with an ontological dimension, and proposes to consider any object as having one of its parts in the virtual, which, far from the indeterminate being, would rather constitute an objective dimension at the origin of the process of actualization.

Other pairs of concepts were founded to support our reflection that goes from its conceptual state to its concrete state, verifiable by experience: following an ontological reflection (presence / immersion / threshold of presence; site / space; place / territory); following the experimental and pragmatic reflection (place of anchoring / place of immersion; realization / derealization; interactivity / agency; space / sound staging; frame/editing), following the phenomenological reflexion (interaction and environment). Thus, the question of site has become the starting point of our common reflection, since it makes it possible to problematize the paradoxical relationship between the concert concrete? place (concrete site) and the virtual site of implementation of the experience, as well as the quality of the experience itself, even from the point of view of the notion of "threshold of immersion".

The research results proposed in this article concern theoretical advances in the context of the definition of the term of virtual, through the following stages:

1) a philosophical deepening of the term of virtual, 2) the proposal of a mental map, 3) a theoretical study, following a graphic schematization of the Deleuzian philosophical system, of the notion of presence from the phenomenon of oscillations of presence of the immersant during the VR experience, 4) a study, based on the restricted practical application of the Deleuzian system, of different modes of mediation constituting the act of presence, 5) a proposal for a theoretical evaluation tool which makes it possible to identify the lack of presence and to propose a typology of it, 6) a reflection on space-based montage, 7) a critical approach to the concept of the virtual placed in the larger context of the history of art and the moving image.

Incorporation of the virtual: places of anchorage and organicity

The site (physical space) as physical anchorage of images and processes of virtualization and actualization constitutes one of the central points of our reflection in the study of the concept of the virtual. The site should be understood as space, territory, but also deterritorialization, device, network, interface, but also body which in turn incorporates virtuality through its proprioceptive properties or through the organization of a space or sound medium which makes perception tangible (processing of sound at the level not only of volume but also of phase, etc.). Olga Kobryn suggests thinking about the virtual outside the static framework but in the dynamic sense of the term and insists on the fact that the virtual is not an object but a process and only becomes perceptible as a process of virtualization and actualization. The idea of the site as anchorage then seems to be best suited to reveal the trace and grasp the issues. The space of the virtual can only be conceived and made perceptible as an oscillation of presence and extension both in space and in time, as movement and process. “How does actualization take place in the things themselves? Why is differentiation [the characteristic process of actualization] correlatively qualification and composition, specification, and organization?” ⁷ We can notice here that thinking about the virtual calls for dissociative pairs, the ramifications of meaning: thus, if actualization is defined by Deleuze as composition and organization, the process of virtualization is defined by Pierre Lévy ⁸ as problematization and not disorganization. There is thus a displacement of meaning and the appearance of new pairs: problematization / solution that diverts the problem of the most expected pairs: visible / invisible, material / immaterial, concrete / abstract. The level of reflection moves from structure and static physical and spatial characteristics to conceptual, temporal, and mobile characteristics. Thus, it can be deduced that the virtual is only sentient and is probably only as a process and not a static entity. Thus, the notion of symbiosis seems to impose itself as a structuring notion of the concept of virtuality: the anchoring of the levels of virtuality in technological devices [dispositif] and concrete sites, in the body which updates and serves as a catalyst for the concept which then becomes organic.

This organicity of the virtual is part of its very nature, Deleuze having demonstrated the concept through the image of an embryo ⁹. The virtualization / actualization processes are at the very origin of the VR device as a symbiotic device - the association of a concrete place which accommodates a differentiation of levels of reality, a stratification of reality whose different strata represent qualitative changes instead of canceling each other out. Paradoxically, the virtual needs anchoring in the real to be able to differentiate itself from it.

Presentation of the themes studied cartography
of the vocabulary

To return more concretely to the construction of this vocabulary, it is important to detail the research method of our working group. During twenty sessions of regular three-hour meetings, from 22 November 2019 to 20 April 2022, several researchers from different backgrounds dialogued around a common debate. Retrospectively, we can say that the thinking process took place in three phases of reasoning. Firstly, several presentations made it possible to explore all the meanings that a term could cover. Then, these presentations gave way to more restrictive exchanges around a definition. Finally, the group consolidated in order to build a global theoretical reflection. Far from being indifferent to each other, these major stages of demonstration (exploratory, definitional, theoretical) brought together study and theory through a particular analytical method, made up of experimentation and failure, by groping around. In order to explain the functioning of this scientific approach, we propose to study here the first tool that emerged from it: the mapping of a virtual glossary.

This "visualized" dictionary was constituted after several exploratory exchanges and focused on the resonances of the following concepts: the immersion/presence couple and the real/virtual couple. The delimitation to four terms is not arbitrary as it tends to prove that they cover various notions, whether they are shared with other domains (cognition, derealization, threshold, affordance) or specific to virtual media (actualizing, immersing). The first principle of mapping is to systematize these lexical encounters to account for conceptual advances and resistances. In this way, we can observe that the concept of immersion, although it can be distinguished into three categories (real immersion, fictive immersion, virtual immersion) with precise specificities (environment, imaginary, interactive, proprioceptive), nevertheless summons a plurality of uses which intersect and merge. Plotted on the map, these very different uses of the concept are linked by a color code, which distinguishes them, a set of icons, which associates them, and a hierarchical structure, which ranks them. These multiple entries finally come together around two key words: narrative, or iconic, immersion and sensory, or a-iconic, immersion.

However, this system of cross-referencing through a two-dimensional cartography suffers from several conceptual problems. First, it is based on an excessive heterogeneity of tree structures, which leads it to accommodate very different scientific elaborations (aesthetics, philosophy, cinema, ergonomics, design, engineering sciences or cognitive sciences) in an unlimited expanse. It is then subjected to extensible fields of reflection that can lead to an infinite expansion of the map object. Without limit, cartography can be interested in questions of production (theories of virtual creation), design (realization of virtual objects) or use (virtual receptions and uses). It has now become essential to move from a summary map, containing all the definitions, to a prospective map, capable of providing an active theorization.

This updating of the map through its results eventually led to different volumetric cartographies reshaping the debates contained in the definitional stage. Before detailing the specific intellectual path from one set to the other, it is interesting to look at their results. At the heart of the new "Virtual/Actual/Possible/Real" or "Affective Participation /Presence/Action/Dispositive '' sets, we can find the debate around narrative or sensory immersion. Even if the conclusions remain similar in the maps to come, their justifications specify the differences. On the one hand, there is affective immersion, which enables the apprehension of an environment from stimuli linked to the aesthetic device and psychic presence, and on the other hand, there is agentive immersion, which is based on abilities to participate in the environment, and which are linked to action and affective participation.

Paradoxical relationship between concrete place
of the experience and virtuality

The paradoxical relationship between the concrete place of the experience and virtuality (in the Deleuzian sense) which always needs an anchorage in actualisation may be found in early dispositifs such as Robertson's phantasmagorias. Before cinema emerges in the Capucines district in Paris (at Café de la Paix), Robertson used phantasmagorias in the Couvent des Capucines to actualize different virtualities of the place and played with the idea of bringing back ghosts, especially in relation to the political and at the same time symbolical and geographical turmoils since the French Revolution and the First Empire (removed bodies and statues of famous people, religious apparitions). This dimension also works in a contemporary cinema that is interested in this power of actualisation, as, for example, in the movie La Vierge, les Coptes, et moi (2012) by Namir Abdel Messeeh. With the exception of some rare experimental productions, VR seems very little 'virtual':

rather, it tends to enclose the action within a few possible scenarios. This distinguishes it from interactive documentary, for example, in which inventive or creative abduction often plays a role in the investigation, with very interesting back-and-forth between tree structures and actualisations (geolocated concrete places, for example). The production of the Raspouteam group on the Paris Commune in 1871 and its re-actualisation in contemporary Parisian space underlines this dimension (https://raspou.team/1871/). By definition, indeed, simulation belongs to the realm of the possible. VR is, therefore, not virtual-friendly from a Deleuzian point of view but it may be interesting to ask, on the one hand, how simulation dismisses the Deleuzian virtual, and on the other hand, whether the Deleuzian Virtual, linked to creativity itself (from the biological to the human), can be completely evacuated and how it 'comes back' ¹⁰.

The cartography and the mind mapping of our first collective works could be re-problematized according to this axis while taking into account the principal points of connection of the experience, namely the question of the “dispositif” (in particular the position of the spectator inside and towards technology) - D, that of the affective participation at the time of the experiment (PA), that of the presence and its effects (P), that of the action (A) with different means (joysticking, walking, etc.) finally. The Set theory in mathematics (Georg Cantor), which allows to consider that an item can belong to several orders according to its properties (i.e., 10 may belongs to two sets: set A which includes the multiples of two, and set B which includes the multiples of five) could give a framework to the description of the experience in order to observe how the virtuality could emerge, or not, from a VR experience.

The Deleuzian approach can be drawn through a cross taking the place (Lieu) at its middle (see illustration 2a below) and two lines crossing in this middle, the first from Virtual to Actual, and the second one from Real to Possible. The interactive documentary has to deal with our virtualization of what may emerge by abduction in browsing the internet from page to page, that is on the upper right of the cross, between Virtual and Possible, whereas the VR headsets trap us in the Actual and offer us possible choices to act (or not) with and within the technological device, on the lower right of the cross, between Actual and Possible. If we project the four main points of connection on this cross, we can describe experience paths, such as one that may explain that we act to turn right, or left or whatever (A, Action), to face a new challenge because we felt confidence by overcoming a previous success (PA, affective participation). These paths may be related to one of the four Sets (Virtual, Actual, Possible, Real), so we can have a kind of 3D representation in order to describe an experience (as below).

In this figure, we can see the experience Guillaume Soulez met when he tried to “disturb” the device of a VR experience named A Fisherman’ s T ale during the NewImages festival (Paris, Forum des Images) in 2019. The immersant is a fisherman locked in a lighthouse and s/he must manipulate objects (grab a teapot...), open windows, etc. A representation of the lighthouse in the form of a model, in which an avatar of the fisherman is represented, offers a sort of miniature vision of the room and the actions undertaken (see for example: https://culturevr.fr/a-fishermans-tale/). In a kind of fractal vertigo of dimensions, I am, at certain moments, myself the miniature of a figure which overhangs me. 

Figure 1. 3D representation of relationships between four sets (Virtual, Actual, Possible, Real) during an experience - by Guillaume Soulez. 

All my actions were then possible actions—mainly suggested by the setting and the presence of objects—to try to make the starting situation evolve, except one. I tried to act on the model, making the hypothesis that to modify the model was to impact the principle of the device itself, thus perhaps to act on the system to "get out" of the confinement of the lighthouse (or to disturb the device and see how it reacted). This virtuality was not actualizable (no modification occurred by trying to touch the model, etc.), but, unlike the other thoughts and actions, it was not foreseen by the system that had fixed a certain number of possibilities. In the figure, we can see the path from a virtual (green) D (dispositif) to a idea of action to actualise (red) an action (A) in order to get a new virtual affective participation (PA green), but it doesn’t work: so in the Set of the Actual, this non-actualisation brings a dysphoric feeling of failure (PA red) and I have to admit that the real dispositif (D yellow) makes it impossible rather than possible (D blue). In light gray, you can see the virtual path I imagined (from the same D green but to an A yellow not actualisable) which could have actualized (in red) for me a conjunction of new sense of presence (P red) and of (euphoric) affective participation (PA red).

This abduction (hypothesis) was undoubtedly suggested to me by the place and the atmosphere produced by the festival: this one proposes to multiply very varied experiences in a relatively short time, to play with different devices, to pass from one to the other, which incites finally to practice a meta level. Therefore, we can provisionally conclude that the place of the experience (here the festival as a place) must be taken into account (as we’ll see below with the semio-pragmatic approach) in order to understand how symbiosis may occur, or not, with an immersive technology such as the VR headsets. One could also say that a variant of our abductive trial is hacking, which consists in "hijacking" a device, i.e. opening a virtual space beyond the system of possibilities that it constrains. Therefore, we could ask whether a successful hacking would mean a kind of translation to another symbiosis, the hacker’s one, his/her own personal (but sometimes shareable) universe. This approach and this VR experience suggest deepening the analysis with other VR products, especially “experimental” ones, in order to test other types of virtualization. But let’s now analyse the way this Set model may be concretized in specialization of VR experiences.

Comparing spaces of mediation

Based on Guillaume Soulez's proposal to apply set theory to a cartography of the virtual while affirming the concept of place as the meeting point of the axes of Deleuze's real/possible and virtual/actual schema, Francois Garnier proposes to pursue this idea to the point of considering place as the space of an experience, where the Deleuzian axes would be the driving forces of a new representation of the affordance action/perception loop.

The place is thus no longer a concept at the point of intersection of these relations (Figure 2a), but a closed and open space encompassing the space of expression of the mediation where a specific experience takes place. It marks the limits, even if it remains open to external influences. To form the affordance loop, we add two arrows to the diagram, opening up two questions: how does the imagination of the Possibles prepare us to be present in the Virtuals? How do the Actuals enrich the Real of the device and the experience? (Figure 2b)

On the map of this Place, it is possible to situate the 4 sets proposed above: the Device set, which regroups the Real pre-existing the experience, the Affective Participation set, which is the stimulator of our projection in the Possibles, the Presence, which is the condition to the setting-up of activation, of vibration or of reasoning (see the quotation of Proust by Deleuze above) of the Virtuals, and the Action, which is the bodily engagement actualizing a Virtual as an Actual, emerging to enrich the Real, the experience. (Figure 2c).

Based on this conceptual scheme it should be possible to try to study the differences existing in the affordance action/perception loop when applied to a technological mediation site such as cinema, virtual reality, or shared virtual reality.

In an experience of mediation, a dialogue is established between the author and the spectator-actors.

The 2 flows Real > Possible and Virtual > Actual can be associated to the reciprocal roles of the author and the spectator. The author stages the Reals, opening up paths and guiding the spectator towards Possibilities (act of narrative), the spectator-actors acting out the Virtuals and allowing the unfolding of the experience to progress. 

Figure 2. a, b, c, d, e, f by François Garnier

In the cinema (Figure 2d), the spectator does not have the possibility of acting on the loop of affordance between Possible and Real via the actualization of the virtuals. He cannot intervene in the unfolding of the experience; this role is given to the actor whose actions are predicted and are already part of the Real included in the device which will be revealed progressively in the time of the narrative. The actual is predicted, recorded in the device, it is included in the Real pre-existing the experience.

In Virtual Reality (Figure 2e), the immerser enters the affordance loop between Possible and Real via the actualization of the virtuals, he can act on the unfolding of the experience and make it his own. Within the limit of the possible and of the interactions proposed by the author, which make up the device, the immerser actualizes the virtuals. This actualization requires a Presence and produces the Actuals which come to enrich the Real of the Device.

In shared Virtual Reality or Metaverse (Figure 2f), the experience becomes an open space, the presence of the others gives an identity to the immersant and influences his perception of the possibilities and his affective participation. He is no longer alone to act and to produce actuals, he must cooperate, make society in the Real of the experience in constant becoming.

Bodies and Spaces: methodological aspects

The mode of embodiment, or mode of existence in virtual reality environments is quite specific as it engages the whole body of the participant in its sensory, physical and physiological dimensions, as well as in its cognitive and socio-cognitive dimensions. The transactional situation unfolds between a physical space in which the participant physically moves and experiences sensations and a digital space where he/she experiences interacting with the narrative, processual and procedural levers of the programme.¹¹

Roger Odin ¹² builds a model of mediated communication that he considers as a methodological tool for research. From a semio-pragmatic perspective, he proposes to move from the notion of a communication situation in context to one of “spaces of communication.” The space of communication is the place where the "actors" in the production space, as in the reception space, produce meaning on the same axis of relevance. They share the same constraints (cultural, symbolic, cognitive, social, etc.) which constitute a common and shared framework that allows for the orientation of communicative skills related to a specific and contextual space. This proposal has consequences, as it is situated in an approach which allows interpretative processes in a form of non-communication. If the instance of production constructs/produces signs that can be apprehended by the one who is supposed to receive them, it is not certain that the receiver receives them as they are proposed, that he even perceives them. It is therefore not certain that he, in turn, sends signals on the same axis of relevance, in the same space of communication.

From a phenomenological perspective, the model developed by Philippe Bonfils¹³ transposes R. Odin's model to immersive environments. He considers four distinct spaces of communication to explain the engagement of "subjects" during their experience in an immersive environment.

The first division is between digital space and physical space, the second distinguishes the space of possibilities from the space of action. The space of possibilities corresponds to what the subjects "do" with the sensations and information perceived in the physical space and in the digital space.

This makes it possible to observe and describe the way in which they actualise, interpret, give meaning to, and organize these sensations and information (R. Odin's vibrations) according to their own capacities/knowledge. The space of action corresponds to the actions that the participants carry out through their bodies in physical space and/or in digital space. In the course of the action, these four modes of instantiation of the body, which are or act in as many communication spaces, operate in a loop. The adjustment of perceptions and actions is permanent, as is the updating of representations and knowledge, thus allowing the construction and circulation of meaning on the activity proposed by the mediation of an immersive device. This engagement is not only due to the immersive or interactive qualities of the device. It is just as much about what the "subjects" want to do and can do or understand in the space of physical possibilities with what they perceive of the space of digital possibilities; and what they can do through their actions in the space of physical action to achieve their objectives in the space of digital action.

To conclude, while this division into (four) spaces of communication is purely theoretical, this modeling has a strong methodological interest. The fact of making these distinctions makes it possible to observe, during the immersive activity, what the participant perceives, understands and acts upon and what he does not perceive, understand or act in the virtual reality device. This modeling allows for a better understanding of what is known and shared about the constraints that constitute these shared spaces of communication between the participant and the experience imagined and designed by the creators, and what is not. In other words, this methodology applied to the observation of the activity allows us to better understand what makes symbiosis, or what does not make symbiosis between the immersive device imagined by the creator and the participant who experiences it.

Figure 3. Forms of instantiation of the body in four communication spaces of immersive environments by Sophie Balcon-Fourmaux

A practical case

When conceptualizing and developing a virtual reality device, depending on the constraints of space, time and interactivity left to the immersant, the artist can more or less anticipate the actualization of the experience. When conceiving an artwork, he or she may imagine the artistic impression to transmit to the receiver. The artist tries to copy his or her model of thought to one of the subjects, hoping that the artistic impression desired is the one lived during the reception of the artwork. However, according to the possibilities left to the immersant, the behavior of the latter can move away from the artistic impressions envisaged. Moreover, the algorithmic nature of virtual reality systems enables the artists to give behaviors to their devices which can influence the aesthetic experience of the immersant. In this way, interactive works allow a new expression modality to emerge, that of the relation between the behavior of the work with that of the subject.

Regarding the analytical tools proposed above, we propose a case study around the device "Montage Spatial” developed by Rémi Sagot-Duvauroux within the framework of his research-creation. This device, described in a previous paper,¹⁴ proposes to explore the following questions: How can an artist test the effects of staging and montage that he or she wants the viewers to experience? How might the behavioral rules be programmed whilst taking into account future behaviors of the immersants?

This device is based on a practice of film editing and on the states in which an editor might be when editing a film. If the montage is an intellectual act allowing the spectator to think about the film, the editor experiences the process of making it. By seeking combinations of shots that express narrative, discursive and/or poetic intentions, the editor considers and experiments a multitude of possibilities and is regularly surprised by the aesthetic effects that emerge. The virtuality in the practice of editing could then be located in this unpredictability underlying the film structure in progress. In this same dynamic, the device "Spatial Montage" seeks to find the montage’s inherent oscillation between the manipulation of the matter and the projection onto the moving image. This quick transition from the state of creation to that of reception might be conceptualized by the passage from the structuring of possibilities to the actualization of the underlying virtual by the mounted moving image.

On a virtual table (Figure 4), an artist builds a model with interactive sets, characters, light cones, and sound bubbles. He or she also may locate and orient different walking areas within the model. By pressing a button, the artist acquires the scale and the point of view of one of the three zones (Figure 5). By being able to oscillate between the scale of the staging and that of the reception, the artist is able to switch at any time to test the actualizations of the possibilities that he or she organizes spatially. 

Figure 4. On a virtual table, an artist builds a model with sets, characters, light cones and sound bubbles.
Figure 5. By pressing a button, the artist acquires the scale and the point of view of one of the three zones.

Moreover, once at the scale of the model, according to the immersant’s movements, viewpoint, and distance to the characters, the latter becomes active, animated, slows down or freezes. In a sensorimotor loop, the environment is puppeteered by the immersant's bodily actions, which are themselves influenced by the scenography of the unfolding environment. Thus, even by having anticipated realizations, the artist is confronted to the unexpected behaviors of the staging which makes him or her experiment the discovery state of the immersant. We might detect here a form of virtuality of the staging. This unexpectedness, as present in the act of creation as in the reception, can thus constitute a relational meeting point between the artist, the artwork and the subject.

Finally, the different zones are connected by wormholes: passage points between two spacetimes set up by the artist and materialized on stage by a frame that might be crossed. The immersant acquires thus a "body-editor" of his or her own aesthetic experience. If these extradiegetic portals signify the presence of an artist and his or her point of view during the time of the transition, the temporality of the latter is under the responsibility of the immersed body. In a fast oscillation between the perceiving and the acting body, he or she embodies not as a diegetic entity but as a gear of the ongoing narration, putting in tension his or her own temporality with the dynamic one of the experience. This "body-editor", at the same time anticipator and ready for surprise (or even jubilator) seems in this way to embody the virtuality of the editing acts both in the making and in the reception. It therefore enables a way of thinking in action the relations between artists, virtual environments and immersants

Discussion and Return on experience

Before we conclude our provisional discussion of the vocabulary of the virtual, we would like to go back to the expression of "virtual reality" which seems to resist our investigation. We have proposed several definitions of the virtual in virtual reality but is this really sufficient? What about the reality in virtual reality? In English, a virtual reality is a quasi-reality, a reality which is "almost" real. And in French, the official expression for virtual reality is "synthetic reality,"¹⁵ a reality which is simulated by a computer program. Let us examine where this new terminology leads us.

Compared to cinema, a distinguishing feature of virtual reality is that the images presented to the audience are computed in real time, rather than being pre-recorded. As a result, virtual reality appears to be a realization of the procedural approach to movement, anticipated by Bergson in "The creative evolution"¹⁶ where movement is not produced by a succession of still frames, but as a result of a dynamic simulation. Images and movements produced in this fashion challenge the classification of cinematographic images by Deleuze.¹⁷ How should those simulated images be classified? In virtual reality, the movements of the immersant influence the simulation and create a different and unpredictable experience every time. What makes virtual reality different from cinema is not its virtuality or reality, but its relation to space and time. From this perspective, virtual reality may be closer to theater than cinema. Like virtual reality, theater is a tridimensional art where each performance is unique. Variations may be more or less important, but an exact repetition is excluded. What distinguishes virtual reality from theater is that the actors, the settings and the props are all software, no hardware.

Virtual reality is therefore different from cinema because it occupies the "real time" and the "real space" of our experience. And it is different from theater because it is immaterial. As a result, it is better described as a "synthetic" or "artificial" reality¹⁸ than a "virtual" reality.

This redefinition raises new and interesting questions. What is an artificial reality and how can it be constructed? How can it be realized? The French translation for "director" is "réalisateur". It is interesting to contrast the term of "realization" in this sense of "direction" or "mise en scene" with the opposite term of "derealization." In medicine, derealization is a psychological disorder where one loses the sentiment that the outside world is real. The two terms are not meant to be opposed to each other, but they provide some interesting insights on what it means to build a virtual reality experience.

On the one hand, virtual reality requires a director who "realizes" the experience by programming the simulation in the first place. On the other hand, virtual reality also requires an immersant who "realizes" the experience as an exercise of make believe. Derealization teaches us that the sense of reality cannot be taken for granted, even in real life, and can in fact be affected by long and repeated exposures to virtual reality.¹⁹ Building this sense of reality in a virtual experience may be the defining factor of virtual reality, one that requires a symbiotic relation between the director and the immersant who both contribute to its realization.

Hommage

The authors pay tribute to Roger Odin, who passed away in August 2023, whose wealth of research nourished their reflection; whose benevolent advice will always nourish their path.

References

1 Daniel Mestre, Philippe Fuchs, “Immersion et présence” dans Philippe Fuchs (dir.), Le Traité de la réalité virtuelle, Volume 1 : L’Homme et l’environnement virtuel, Paris, Mines Paris, coll. "Sciences mathématiques & informatique", 2006, 309.

2 Michel Serres, Atlas, Julliard, 1994.

3 Jonathan P. Rowe, Scott W. McQuiggan, James C. Lester, Narrative "Presence in Intelligent Learning Environments, Narrative Presence in Intelligent Learning Environments", AAAI Fall Symposium: Intelligent Narrative Technologies, 2007.

4 Philippe Fuchs (dir.), Le Traité de la réalité virtuelle, Volume 1 : L’Homme et l’environnement virtuel, op. cit., 5.

5 Henri Bergson, Matière et mémoire [1896] et "Le Possible et le réel" [1920], in La Pensée et le mouvant [1934].

6 Gilles Deleuze, "The virtual is not opposed to the real but only to the actual”, Différence et répétition, Paris, PUF, "Epiméthée" collection, 2003, 269.

7 Gilles Deleuze, Différence et répétition, op. cit., 276.

8 Pierre Lévy, Qu’est-ce que le virtuel ?, La Découverte, 1998.

9 Gilles Deleuze, Différence et répétition, op.cit. See also Arnaud Bouaniche, “Chaos débout”, in O. Kobryn, M. Ovtchinnikova, G. Soulez (dir.), Théorème n°38, Qu’est-ce que le virtuel ? Lieux d’ancrage, Paris, PSN, forthcoming 2023.

10 Guillaume Soulez, “Lieux du possible, lieux du virtuel ”, in O. Kobryn, M. Ovtchinnikova, G. Soulez (dir.), Théorème n°38, Qu’est-ce que le virtuel ? Lieux d’ancrage, op. cit.

11 Maude Bonnefant & Thibault Philippette, "Rhétorique de l’engagement ludique dans des dispositifs de ludification", Sciences du jeu, 28 octobre 2018.

12 Roger Odin, Les espaces de communication ; introduction à la sémiopragmatique, Presses universitaires de Grenoble, 2011.

13 Philippe Bonfils, L’expérience communicationnelle immersive : entre engagements, distanciations, corps et présence., Mémoire d’habilitation à diriger des recherches, Université Lille Nord de France, Lille, 2014.

14 Rémi Sagot Duvauroux, Rémi Ronfard and François Garnier, Montage as a narrative vector for virtual reality experiences, ConVRgence (VRIC) Virtual Reality International Conference Proceedings, International Journal of Virtual Reality, 2022.

15 Réalité de synthèse, Vocabulaire de l’informatique, Journal Officiel de la République Française, 20 avril 2007.

16 Henri Bergson, L’évolution créatrice, Paris, Félix Alacan, 1907.

17 Gilles Deleuze, Cinéma 1, L’image-mouvement et Cinéma 2, L’image-temps. Editions de minuit, 1983, 1985. 

18 Philippe Fuchs, Théorie de la réalité virtuelle : les véritables usages, Presses des Mines, 2018.

19 Rebecca Searles, "Virtual Reality Can Leave You With an Existential Hangover", The Atlantic, December 21, 2016. 

Post-Human Motherhoods: Reflections on mother-offspring bonding as symbiotic individuation in Contemporary Art

Graziele Lautenschlaeger

Kunstuniversität Linz | VALIE EXPORT Center Linz, Austria g.lautenschlaeger@gmail.com

Abstract

As part of an ongoing postdoctoral research on post-human motherhoods in media art, the paper examines selected artistic expressions of mother-offspring bonding and its relationships with different technological cultural basis. The introduction addresses the handling of the concept of the mother and its related operations in the research scope. Following this, we suggest as standing points for reflection both the material basis of the placenta and playfulness as elements shaping the mother-offspring bonding. They constitute fundamental aspects triggering affection in the potential symbiotic relationship established in mother-offspring bonding. The discussion is permeated by artwork examples, brought on demand to the text to feed the reflection. The notions of “natureculture” by Donna Haraway and the theoretical framework by Second-order Cybernetics are grounding references. The final considerations point towards the human and post-human aspects of motherhood revealed by the artworks, as well as to the need for concrete actions towards the reduction of the gender-based technological gap in media art, in order to increase the imaginary variability of the field.

Keywords

Motherhood, media art, post-humanism, mother-offspring bonding, placenta, play, naturecultures, Second-order Cybernetics, technofeminism.

DOI

10.69564/ISEA2023-69-full-Lautenschlaeger-Post-Human-Motherhoods

Introduction

Inspired by my own experience as a media artist who became a mother, I was curious to know what artistic expressions emerged from the intersection between motherhood(s), technofeminism(s) and post-humanism in the arts over the last decades. Soon I found that, there was neither an abundance of examples nor has this intersection been sufficiently circumscribed in the academic field. Therefore, the initial curiosity became my current post-doctoral research project at the VALIE EXPORT Center / Kunstuniversität Linz.

As a radical onto-existential re-signification of the notion of the human,”¹ post-humanism considers the “significant otherness” and the inseparability between nature and culture. The acknowledgement of the agency of non-human entities, from active matter through machines, to other species, material-discursive approach, even if not mentioned or declared, grounds much of the media art production since its emergence in the history of techno-culture.

To a certain extent, it is possible to deduce that the absence of motherhood as a topic in media art field mirrors the gender-based technological gap of the field experienced until today. In other words, numerous reports and studies point to the so-called digital divide, referring to the inequalities between groups with or without access to technological tools. Among other non-privileged groups, it is constant that girls and women have less access in general. Inequalities are even more contrasting if we consider the gap between those who have access as consumers and those who are developers—an issue frequently disregarded in many studies on Human-Computer-Interaction (HCI).²

Aiming to dig into this problem, the project Where are media artist mothers? also integrates a survey³ to investigate if the hypothesis is true. Among the concerns is the question of what measures should be taken to better welcome media artists who are mothers in the production context, to populate media art with more constructive and empowering perspectives of post-human motherhoods, and therefore, to enrich the art and technology environment with a higher variability of imaginaries.

In this paper I share only a frame and still open part of the research, addressing some references and examples to reflect on how artists articulate common symbols, materialities, technologies and alternative narratives in relation to motherhood and post-human bodies. More specifically, the paper observes the materiality and dynamics integrating the mother-offspring bonding and its expressions in the art field.

To conduct analytical comparisons among selected artworks under a common conceptual umbrella has been one of my strategies to investigate how (media) artists respond to the paradigmatic changes in feminist movements through the lenses of mothering issues. 

Motherhood and related operations

The general approach towards motherhood in the research is based on the diagram shown in figure 1:

Figure 1. Diagram depicting the abstraction behind the comprehension of motherhood, through an expanded concept of mother. It circumscribes the operations usually associated to mothers, its relationships to other bodies, as well as the appropriation of the concept in non-human contexts. © Lautenschlaeger, 2022.

In the human case, these operations can also be associated to the sequence of topics involved in the reproductive cycle, encompassing right from the concerns around fertility, through fecundation itself, the transformative aspects of pregnancy and gestation, the birth-giving, to procedures and ethics of offspring care.

Due to the lack of motherhood and related topics representations in the media art field, I look with special interest to a specific profile of mothers: artists that are also technologists, creating their proposals in the intersections between scientific and technological knowledge, whose artworks or performances reveal sorts of media thinking, critical approaches to technocultures, and/or technofeminist perspectives.

Parenting as reproduction and care work

Regarding the considerations above and for a more respectful approach to all possible types of motherhood, the term “parenting” is certainly more appropriate. This observation is structured on the acknowledgement of the feminist struggles towards a more equalitarian division of the high demand of work generated alongside the gestation and care of a newborn – often invisible and non-paid work. US-American experimental artist and technologist Ani Liu, who between May and July of 2022 had a solo exhibition at Cuchifritos Gallery in New York titled Ecologies of care, shows a diverse series of speculative artworks that invites us to reflect on the invisibility of reproductive and childcare labour. At Untitled (Labour of love)⁴ Liu builds a data portrait recording every feed and diaper change for the first 30 days postpartum. 

The coincidence of the terms “labour” and “love” in the piece’s title addresses precisely the ambivalences that pave the culturally constructed perversion in devaluating creative/generative work in multilayered contexts – from the reproductive work itself to the art context.

In order to turn the repetitive and exhausting care work demanded by a newborn visible, the artwork also portrays a way of thinking inspired by the ongoing big-data technological environment, where we are responsible for curating what is the relevant data to turn into information through data visualization. Dealing with information as embodied data, would Ani Liu’s Labour of love be an actual version of the iconic Mary Kelly ́s Post-partum document (1973-79)? ⁵

Indirectly, Liu’s piece is also suggesting love as the strong element fostering the mother-baby bonding, which is neither certain nor unanimous among all cases of mother-offspring relationships, be it in regard to human or non-human species.

Now connecting to ISEA 2023’s theme, would symbiosis be the key towards healthy mother-child relationships, despite all the ambivalences of mothering? How does the bonding emerge and is regulated? How do the symbiosis aspects emerge in artistic expressions?

We may find part of the answers to the aforementioned questions through an investigation on the possible ways through which affectivity is generated. 

Mother-offspring bonding

The strong connection between mother and offspring is a long-term process starting in the gestational phase, continuously nurtured through the care work demanded by the newborn until its autonomy as a self-caring being.

Although it is known that there are studies in psychoanalysis, for instance, that doubt the mother-offspring relation as symbiotic and conflict-free,⁶ in this session I address only two aspects of a huge variety of possible mechanisms that contribute in shaping the symbiotic aspects between mother and offspring: the role of the placenta and the role of playing.

The case of mammals: since the womb via the placenta

Observing the very material basis of mother-offspring bonding, it starts on the physical and chemical exchanges between cells within the mother’s womb, from which embryo and placenta emerge.

Professor Marjolein Oele places the placenta as “a fetal and maternal place-and-time making boundary”⁷ and discusses how the placenta materializes a model of “affective symbiogenesis where selves come into existence.”⁸ Her philosophical reading of the placenta has special impact in the project I report below.

Intrigued by the metamorphosis I went through while I was pregnant, increased by the intensity of every change, I felt compelled to start gathering materials referring to the significant fragments of the ongoing irreversible revolution. Among these materials there are audio samples recorded from the belly (heartbeats, placenta, veins flux, etc.), as well as the sounds of the baby after birth, interviews with mothers from different profiles, stamps of the placenta, data collected from the glycemic control of gestational diabetes and writings. This initiative somehow assisted me in accepting the difficulties of the fast changes as well as in getting prepared for both the birth labour and puerperium. Simultaneously, it became an art project, that I called Binômio (Binomial 2021- ), still in progress.

Binômio encompasses a series of aesthetic experiments articulated from the recombination of afore-mentioned different elements collected since my pregnancy, addressing issues arising along the experience of mothering. The experiments interlace material and symbolic relationships of affection, intimate and universal aspects, intertwining subjects, organic and machinic elements. Fascinated by the placenta as an ephemeral organ and the various cultural manifestations related to it, there are some experiments in which I focus on the extraordinary role it plays—e.g., the placenta provides oxygen and nutrients to the baby, produces hormones that promote the baby’s growth, removes toxins and carbon dioxide, besides protecting and passing immunity from the mother to the baby, etc.

For the context of this article, I highlight two of them: 

Figure 2. Placenta: interface (2021), by Graziele Lautenschlaeger ⁹ ©Lautenschlaeger, 2022.

Placenta: Interface (2021) has been a conceptual piece, a print media merging a text with the image of one placenta stamp, done with blood just after birth. Appraising the placenta’s amazing qualities as interface, the text is a poetic letter to my daughter, written on its contours and places of flux and exchange. I envision its mysterious ways of functioning as the most incredible interface possible.

Abstracted Placenta is another ongoing experiment, a luminous sculpture as a homage to the artist’s forgotten placenta at the hospital, as well as the uncountable wasted ones by medical teams. Lacking information and ethical procedures, in most cases the placenta is ignored, on the one hand, the right of the birth-giving body to possess their own organ, and, on the other hand, the multiple benefits that placentophagy can provide for women in the postpartum. ¹⁰

The piece displays the changes from the glycemic values, systematically collected by the artist in the week when she discovered she had gestational diabetes. It suggests the placenta as an interface organ between the pregnant body and the baby, as a sort of data visualization of the complex regulation capacity of the organ, which despite its ephemeral life, undertakes the whole responsibility during the mother-baby bonding process. As a self-regulated organ, the placenta is considered to have its own “conscience,” characteristic suggested by the pulsing light animating the glass piece, whose shape also remembers a jellyfish. 

Figure 3. Prototype of Abstracted placenta (2023), by Graziele Lautenschlaeger, to be exhibited in the context of Caring matters. who cares, between June 5th and 13th at WHA Galerie in Linz, AT. © Lautenschlaeger, 2023. 

Despite the scientific advancements on the creation of artificial placentas, both the conceptual and material presence of the organ in contemporary media art is practically inexistent. Among the rare cases one can mention the participation of the Japanese obstetrician/ gynecologist Nobuya Unno at the Symposium of the Ars Electronica 2000 – The next sex: Sex in the age of its procreative superfluousness, where he presents an article on the Development of an Artificial Placenta. Besides not being based on an aesthetic exploration of the organ, Unno concludes the text with remarks on his proposal as an “extension of the preexisting neonatal intensive care system for the extremely premature newborn” and with the warning on the high costs of such an enterprise: “using extracorporeal circulation would be destructively expensive as an alternative for natural intrauterine pregnancy.” ¹¹

In addition, other online searches on “placenta art” only lead us to handicrafts made of placenta or to digital illustrations. The absence of deeper conceptual, aesthetic and philosophical explorations on the magnificence of the placenta compels me to further investigate its potential by means of the available contemporary artistic tools.

Playing as means for bonding

Besides the very material basis of the mother-offspring bonding, one can also approach the dynamics of the relationships that emerge through the activities of care communication, and above all, through the eagerness of babies and children to play and engage in interaction.

Playing permeates the most varied human cultures,¹², ¹³ extraordinarily demonstrated by Belgian Artist Francis Alÿs with his project Children’s games,¹⁴ featured recently at the Venice Biennale 2022—The milk of dreams, with a selection by curator Hilde Teerlink integrating the exhibition The Nature of the Game. Playing is embedded in social life and its dynamic shape modes of existing and interacting with the environment in which subjects are relating with, influencing the communication processes and the quality of the relationships between subjects. Playing is also a key entrance towards a neuro-affective approach for a healthier development for both child and family.

Fair play between different subjects requires a respectful positioning towards the corresponding alterities—principles that may be discussed from a multitude of perspectives. In the context of this paper and its affinity to issues emerging from the intersection between art and post-humanism, two main epistemological frameworks integrate our investigation, the notion “naturecultures”, introduced by Donna Haraway¹⁵ and Second-order Cybernetics.¹⁶

The neologism “natureculture” has been developed in order to deconstruct historically calcified dichotomies such as the opposition between the cultural and the natural, body and mind, the material and the conceptual, among others. As a review on Haraways’ own thoughts developed previously at A Cyborg Manifesto (1985),¹⁷ naturecultures entangles multispecies narratives, as means for reflecting about power relationships, agency, difference, sociality, ontology and epistemology.

Second-order Cybernetics, in turn, as a transdisciplinary and participatory epistemology, presents an ethical significance based on two pillars:

(1) the inclusion of the observer while observing a system, and

(2) the recursion embedded in acting in order to know – observers of a system recursively observe the effect of their actions, using the feedback to adjust their understanding and their following actions.

From the valued recursion of Cybernetics, the condition well summarized by Timothy Stott, lecturer in Art History and Visual Culture, emerged:

Play was, and remains, a social technology for the cybernetic age. Advocated by many as a humanist corrective to a technocratic and automated post-war society, play also expanded cybernetic ideas of interaction, feedback, and systems modelling into the social domain. From the late nineteen-fifties on, especially, cybernetics and play converged through games, toys, and interactive exhibitions.¹⁸

The convergence pointed by the author however, is rarely (if) seen in relation to mother/parenting issues in media art exhibitions.

Articulated in the background of the ongoing research as complementary tools, natureculture and Second-order Cybernetics offer to the players of a system, a relational, reflexive way of knowing and acting, in which the players are not merely observing, but are indeed responsible for the values, ethics meaning they generate. Theoretically, they compose an ideal scenario for a healthy and equalitarian interaction, enabling, in the case of mother-offspring relationships, an effective (and affective!) bonding.

As an example, within Art History tangential to this question is Susan Suleiman’s reading of the avant-garde through the figure of the playful, laughing mother.¹⁹ By means of inquiring what role do women artists and writers play in the audacious positioning of the avant-garde and their postmodernist successors, Suleiman’s laughing mother embodies a powerful, humorous or parodic critique of patriarchal ideologies—cleverly merging symbolic innovation, political and social transformations.

Posthuman motherhoods Machinic motherhood

Other possible thematic links between reproduction and technology is found in the creation of artificial wombs and other genetic engineering techniques that have inspired various sci-fi stories, for example, Brave New World (1932) by Aldous Huxley or Blade Runner (1984) / Do Androids dream of electric sheep? (1968). More recently, the series The Handmaid's Tale, based on the dystopian novel by Canadian author Margaret Atwood, is the current most popular reference. It is precisely to avoid such dystopian worlds that technofeminists argue for an attitude in which such technologies could also be developed by non-dominant groups.²⁰ Is something similar expected from artists working technologically with media art and addressing issues such as fertilisation, pregnancy, birth, as well as child feeding and education?

Besides the machinic motherhoods found in the science fiction imaginary, one can observe concrete machinic entities built up within the aesthetic domain of media art exhibitions. One example is the installation Génesis: dinámicas de organismos artificiales (2016),²¹ by Ana Laura Cantera, Daniel Alvarez Olmedo and Leonardo Emanuel Maddio. It consists of a symbolic recreation of a gestation of artificial organisms that will interact with the visitors by means of programmable robotic experimental devices, in an artificial context of water, movement and light. The containers of the robots have an opening covered with biodegradable plastic that is corroded by the constant movement of the organisms inside, to the point of generating the rupture and their consequent “birth”. This way, the installation presents critical moments of latency and birth of these organisms, from their embryonic stage, to moving and interacting with the audience. Programmed in an evolutionary and non-cyclical way, the installation generates unique and unrepeatable moments along the exhibition period.

What is the significance of reproducing (or even inventing) a machinic birth process in the art context? Does the aesthetic system of Génesis: dinámicas de organismos artificiales also present a sort of mother-offspring bonding? In technical terms it probably may have. However, would this also mean an affective link?

This question raises the interesting issues of the still challenging notion of possible machinic self-awareness, possible if a machinic subjectivity once emerges, and therefore being able to engage in an affective relationship. Even with highly advanced AI-based technologies one can hardly imagine a concrete case of an emerging machinic self-awareness. Reflecting on the reciprocal quality of affect and symbiosis, perhaps it would make more sense to elaborate on the mother-offspring bonding between the artists and the artwork.

If we cannot speak about affective symbiosis in the machinic motherhood of Génesis: dinámicas de organismos artificiales, can we envision a machinic mediated mother-offspring bonding? Perhaps this is the case of the performance discussed in the next session.

Interspecies motherhood

Another post-humanist perspective regarding motherhood can be discussed through the project Hybrid family (2016), by Slovenian artist Maja Smrekar, which referred to the simultaneous existential and political instrumentalisation of a woman’s body and breastfeeding.

Along a three-month performance with her puppy, the artist stimulated her own hypophysis by systematically pumping to release the hormone prolactin. At the same time, she had a diet rich in galactogens to promote lactation, which stimulated an increase in the oxytocin hormone.

By becoming a “(m)other”, the artist experienced and demonstrated the “natureculture” coined by Donna Haraway and continues to explore a reproductive freedom in a multispecies world. “If nature is unjust, change nature” ²²: from this standing point the performance embraces simultaneously xeno and ecofeminism, interspecies relations, as well as a critical perspective on technology and ideological structures in society.

Final considerations

The artwork examples discussed throughout the article shed light onto both the very human issues related to mothering/parenting—the invisibility of reproductive and care work within an unbalanced social structure—and the post-human experimentations that expand and challenge the core of motherhood related operations. Together, they give us a series of questions to reflect on the entanglements between motherhood, art and technology—despite the rare presence of the topic in the realm of media art.

Approaching the mother-offspring bonding through its material and dynamics basis (e.g., the placenta and playing) is just one possibility to address its symbiotic potential and how the motherhood may be related to contemporary epistemologies frequently being used in the media art production.

In addition, they confirm the need for updates on feminist positioning within media art production, whose community perhaps is not aware of the absence of the topic within the field.

Conclusively, in response to the still enduring inequalities addressed by feminists, it is time to find concrete strategies to reduce the gender-based technological gap in media art. This is the only way to populate the field with a higher variability of imaginaries, with subjects, such as motherhood, that are ignored inclusively by the most considered alternative initiatives, agents and institutions. 

Acknowledgements

A special thanks to the Kunstuniversität Linz/V ALIE EXPORT Center for funding this research and to all the colleagues that contributed at any level of the work with their constructive feedback. 

References

1 Francesca Ferrando, "Posthumanism, Transhumanism, Antihumanism, Metahumanism, and New Materialisms: Differences and Relations," Existenz 8/2, 2013, 26-32, Accessed May 12, 2023, https://existenz.us/volumes/Vol.8-2Ferrando.pdf 

2 Lorna Gibson, Vicki L. Hanson, "Digital Motherhood": How Does Technology Support New Mothers? ACM CHI 2013, April 27–May 2, 2013, Paris, France.

3 “Where are the media artist mothers?”, Accessed December 02, 2022, http://shorturl.at/gnuxY

4 Ani Liu, “Untitled (Labor of love)”, Accessed December 1st, 2022, https://ani-liu.com/untitled-labor-of-love 

5 Andrea Liss, Feminist art and the maternal, London, Minneapolis, University Minnesota Press, 2009.

6 Rozsika Parker, Torn in two: the experience of maternal ambivalence, London: Virago Press, 1995, 43.

7 Marjolein Oele, “Generative human affectivity. The placenta as place-and-time making in-between.” E-co-affectivity. Exploring pathos at life ́s material interfaces, Albany: State University of New York Press, 2020, 80.

8 Marjolein Oele, E-co-affectivity, 81.

9 Graziele Lautenschlaeger, “Placenta:Interface” 4x14. São Paulo: Ateliê 397, 2021.

10  Sergio L. Sánchez Suárez, “Influencia de la reincorporación de la placenta (ROP) autóloga tras del parto, en la evolución bioquímica sanguínea e láctea", Ph.D. diss, Universidad de las Plamas de Gran Canaria, Departamento de Ciencias Médicas y Quirúrgicas/Facultad de Ciencias de la Salud, 2015.

11 Nobuya Unno, “Development of an artificial placenta” in Ars Electronica Archive (Ars Electronica 2000 – Next sex: Sex in the age of its procreative superfluousness), 68, Accessed June 3, 2022, https://archive.aec.at/media/assets/7cbf0d899fb64cb5f9e299f67992e347.pdf

12 Roger Caillois, Os jogos e os homens. A máscara e a vertigem, Lisboa: Edições cotovia, 1991.

13 Johan Huizinga, Homo ludens. O jogo como elemento da cultura, São Paulo: Editora Perspectiva, 2000.

14 Francis Alÿs, Children ́s Games, http://francisalys.com/category/childrens-games/, Accessed November 24, 2022.

15 Donna Haraway, The companion species manifesto: Dogs, people, and significant otherness, Chicago, Prickly Paradigm Press, 2003.

16 Bernard Scott, “Second-order cybernetics: an historical introduction (2003)” in Explorations in Second-order Cybernetics. Reflections on Cybernetics, Psychology and Education, Vienna, Edition Echoraum, 2011, 383-398.

17 Donna Haraway, A manifesto for cyborgs: 1985.

18 Timothy Stott, “Learning to interact: Cybernetics and play", Accessed December 1st, 2022, https://www.youtube.com/watch?v=8Npu6vbcG1k

19 Susan Suleiman, Subversive Intent: Gender, Politics, and the Avant-Garde, Harvard University Press, 1990.

20 Judy Wajcman, Feminism confronts technology, University Park, The Pennsylvania State University Press, 1991.

21 Ana Laura Cantera, Genesis: dynamics of artificial organisms, accessed January 15, 2022, https://www.analauracantera.com.ar/genesis 

22 Laboria Cuboniks, “XF Manifest” Xenofeminism: A politics for alienation, accessed November 27, 2022, Accessed May 12, 2023, https://laboriacuboniks.net/manifesto/xenofeminism-a-politics-for-alienation/ 

Bibliography

Myrel Chernick, Maternal metaphors, Cambridge, London, MIT Press, 2013.

Emanuele Coccia, Metamorphosen, Das Leben har viele Formen, Eine Philosophie de Verwandlung, München, Carl Hansen Verlag, 2021.

Silvia Federici, Caliban and the witch: Women, the body and primitive accumulation, Brooklyn, NY, Autonomedia, 2004.

Hettie Judah, How not to exclude artist mothers (and other parents), London, Lund Humphries Publishers Ltd/ Sotheby's Institute of Art, 2022.

Humberto Maturana Romesin, Gerda Verden-Zöller, The origin of humanness in the biology of love, Charlottesville, VA, USA/Exeter, UK, Imprint Academic, 2008.

Author Biography

Dr. phil. Graziele Lautenschlaeger is a Brazilian media artist and researcher, currently postdoctoral fellow at the VALIE EXPORT Center/Kunstuniversität Linz, investigating the intersection among motherhood, technofeminism and post-humanism. She holds a PhD in Cultural Studies from the Humboldt-Universität zu Berlin, which resulted in the book Sensing and making sense: Photosensitivity and light-to-sound translations in Media Art (2020), a Master of Science in Architecture and Urbanism from the Universidade de São Paulo and a Bachelor in Media Design, Image and Sound from the Universidade Federal de São Carlos. With a transdisciplinary background and experimenting different media and languages, her artistic and academic production essentially draws on Cybernetics, Relational Aesthetics, Media Archaeology and New Materialism. Among her broader research interests are: the notion of translation of materialities (transcreation), hybridization between organic and machinic elements, interface and interaction design, as well as relationships between spaces and narratives. Her work has mostly been exhibited in Europe, USA and South America. 

Live 4 Life: A dream for a free and open spatial performance tool towards symbiosis or death?

Christophe Lengelé, Philippe-Aubert Gauthier

Université du Québec à Montréal (Hexagram, CIRMMT) Montréal, Canada
lengele.christophe@courrier.uqam.ca - gauthier.philippe-aubert@uqam.ca

Abstract

This paper presents the motivations, evolution, and directions behind the spatial sound performance tool named Live 4 Life. It aims to simplify the creation and control in real time of masses of spatialised sound objects on various kinds of loudspeaker configurations (stereo and particularly quadriphonic or octophonic setups, as well as domes of 16, 24 or 32 loudspeakers). This spatial research, which questions ways of associating rhythmic and spatial parameters, is based on the concept of free and open works, both from the point of view of form (improvisation) and in the diffusion of the code. The tool, which was initiated in 2011 and distributed in open source in 2022, has been conceived as a long-term dream against capitalism and loneliness. Several scenarios between (technical, social) death or symbiosis of this tool (with other programs, people, works and the visual representation field) are presented.

Keywords

Sound performance; spatial improvisation; computer-music; SuperCollider; open source; custom tools; ephemeral; GUI; patterns; algorithms; controllers; interdisciplinary research-creation; graphical representation, audiovisual.

DOI

10.69564/ISEA2023-70-full-Lengele-et-al-Live-4-Life

Introduction

In a context of increasing demand for and supply of sound spatialisation systems and tools¹, ² it remains however particularly difficult to dynamically and simultaneously control multiple sounds in real time in the concert hall space, due in part to the stratification of the composition process in several software programs and the complexity of individually spatialising a multitude of sound events. To meet this need, the tool Live 4 Life (1), mixing rhythm and space, offers a spatialisation control structure, including a library of predefined spatial trajectories and rendering algorithms, mixing both channel-and object-based paradigms and abstract and concrete spatial techniques.³ Following philosophical reflexions and the presentation of the underlying principles of the tool, we speculate on two future scenarios for the tool, either death or symbiosis. In this paper, we will see how an artistic intention along with a desire for a singular and custom spatial composition paradigm led to the development of an open-source tool, namely Live 4 Life.

A dream for Free and Open living Art

Like the composer Francis Dhomont,⁴ as a composer and performer my main concerns in music, art and life do not concern those of “pure music,” referring only to itself, but touch on extra-musical themes (capitalism, free parties, loneliness). I (Christophe Lengelé) could qualify my sound improvisations in the same way as Francis Dhomont does for his piece Phœnix XXI: “a music that is abstract in its realisation, but figurative in its intentionality.”⁵ My improvisations thus question, via the tool and the way of creating, the most important themes in my life and offer my interpretation of society, as expressed by Luigi Nono, which is quoted by Joshua D. Parmenter (a long-time developer of SuperCollider) in every email on the SuperCollider mailing list:

Every composer—at all times and in all cases—gives his own interpretation of how modern society is structured: whether actively or passively, consciously or unconsciously, he makes choices in this regard. He may be conservative or he may subject himself to continual renewal; or he may strive for a revolutionary, historical or social palingenesis.

The project Live 4 Life, which is a form of resistance to this imposed world, was born from the desire to reconnect with life. It was created in response to the world and the environment (societal, family and personal) in which I lived. I thus reflect in my improvisations this individualistic world and my disillusioned way of facing it, by trying to use free code as much as possible, so as not to reproduce and encourage this commercial world. Without my personal experiences or traumas in my family or in the professional world, having worked in multinational companies in the fields of banking, industry and communication, my intention of wanting to create this project in open source might never have seen the light of day.

Not wishing to adapt my ideas to a commercial product, I chose in 2011 to develop my own spatialised sound performance tool on the open-source SuperCollider platform,⁶ with the ultimate goal of going beyond the simple sharing of listening or looking. Indeed, the ultimate objective was to tend towards a liberalisation of modes of creations, focusing more on doing it together outside the market and the sharing of creative practices open to all, by giving all the code developed over the years, like many artists who distribute their codes in open source, such as Fredrik Olofsson (2) or CHDH with egregore (3) or vivarium (4) projects.

The series of performances on capitalism Free parties will survive, addresses the relationship to money, property and this desire to always want more. It represents my hatred for this inhuman system for me, by using and opposing it with hardcore sounds from the free parties of the 2000s, just as harsh, symbolising a certain desire for revolt, freedom, anarchy. In my opinion, the capitalist system exacerbates and uses the failings of man such as selfishness. Unlike Adam Smith, who relies in particular on this character trait to lay down the main principles of economic liberalism, I can only believe in a system that promotes and encourages the values of equity and sharing. In the same way as the composer Raphae ̈l Cendo, I try to resist this capitalist system (5) by taking little interest in the production of a (musical) object for an object or in the transmission of perception, but to the concrete way of realising a sharable object or transmitting an experience. However, I consider that the discourse has little scope without a voluntary act accompanying the thought. That is why my only way to deal with capitalism, was to offer improvisations from free and open source creation and production tools.

I also have my contradictions and my development limits, in the sense that I also use commercial products, especially with controllers or the Mac computer. However, I try to tend towards free, at least in terms of software. Currently, I do not use Linux, because a lot of spatialisation systems I used were connected in ethernet with Dante, which does offer drivers only for Mac and Windows. The use of Linux would then have required the purchase of an expensive and bulky sound card. And, if I chose to use cheap and professional commercial physical controllers, it is so that anyone can make the tool their own, without having to build their own interfaces.

According to Jacques T. Godbout, “the work of art is a gift because the gift, [...] is what confirms to one another that we are not things.”⁷ How then to express this gift in music, electronic performance or digital art? Is showing or hearing an experience or a composition a sufficient form of expression and sharing? How to transmit more and create stronger links with others, by making the barrier that separates the researcher artist from the public fade and that the latter exchange their roles? Similar to Ursula M. Franklin, who distinguishes prescriptive technology from holistic technology,⁸ my tool and technology practice aims to focus not just on what is created and entertainment, but on the how and how the work is done though code sharing and future training workshops or participative performance installations.

On the grave of Karl Marx, there is the following epitaph: “The philosophers have only interpreted the world in various ways. The point however is to change it.” Although we cannot radically change the world, I also expect from an artist a real desire for concrete action for a fairer world. The development, promotion and almost exclusive use of open source software represented my utopian way of changing my world, however small, like a little dust to stop the capitalist machine. However, by persevering in this radical position, I have also excluded myself from the ”world”, because I no longer consider the profusion of electronic productions made with commercial software, even if I may like them, because they do not intrinsically transmit the idea of sharing and are not intended to put into questions dominant positions in society.

Besides free and open source objectives, putting bites of his/others lifes in art is also essential for me. What Alice Neel said to painting is applicable to every form of art:

“One of the reasons I painted was to catch life as it goes by, right hot off the griddle. Because besides painting specific people, I’m painting all the happenings [...] and I’ve tried to capture the zeitgeist. When painting or writing are good, it’s taken right out of life itself, to my mind, and put into the work. Now, that doesn’t mean that the work has to tell about real life. I mean, it can be abstract or anything. But the vitality is taken out of real living and put into the creative project, whatever it is.”

The characteristics of the performance tool

Live 4 Life is not only a spatialisation tool, but a whole sound creation system to play with sequences of parameters (rhythm, spectrum, space and effects), integrating spatialisation at different levels, both at the heart of each sound synthesis and each loudspeaker. It is first thought for spatial performance with several global, high-level (indirect) control strategies, e.g., by swinging among scenes of spatialised events or by changing masses of parameters, particularly playback speeds with different mappings according to each controller ⁹ (see figures 1, 2 and 3).

Dynamic, finite event-based spatialisation

Here, we propose a distinction between continuous input-based spatialiation vs finite event-based spatialisation. By continuous input-based spatialisation, we mean that the user selects the position or trajectory of each continuous input (like in most spatialisation plugins such as Ircam Spat). By finite event-based spatialisation, we mean that the user selects (among a pre-defined library) the position and trajectory for several layers of events with different times of beginning and durations (like in some creation systems allowing to control both time and space of events, such as Live 4 Life or Sound Particles). With the tool we focus on event-based spatialisation since it allows to control the temporality (rhythm, density, duration and amplitude) of spatialised sound events, as shown in figure 4. According to the author’s experience in creation, continuous input-based spatialisation tends to create more simple and clear spatialisation, while event-based spatialisation tends to create more complex and confusing spatialisation.

A library of trajectories and algorithms mixing abstract and concrete spatialisation techniques

It integrates a pre-defined library of multiple spatialisation models¹⁰ (see 5/ and 6/ of figure 5): point granulation,¹¹ temporal micro-decorrelation,¹² trajectories drawn, generated algorithmically or according to the spectral analysis of the sound signal ¹³, rendered according to several spatialisation algorithms (equal-power panning, delay-based panning, Vector-Base Amplitude Panning ¹⁴).

Figure 1: The performance tool with all its controllers in 2021: in the center, the computer GUI with the performance view and keyboard shortcuts, one Akai APC Mini at the top left, two iPads with Lemur Application on both sides of the computer, two Midi Fighter Twisters and one Sensel Morph at the right).
Figure 2: Picture of the performance tool in concert during JIM (Journées d’Informatique Musicale) in 2019. The whole performance, which was spatialised with a ring of 8 speakers and recorded in binaural, is available at https: //youtu.be/NfWXF6copEs.
Figure 3: Picture of the performance tool in concert during Akousma Festival in 2021. The whole performance, which was spatialised with 3 rings of 8 speakers and recorded in stereo, is available at https://player.vimeo.com/ video/747391295 at 1h20’40”. ©Akousma / La Conserve Media.

We categorise two main kinds of spatialisation from the point of view of the composer.³ Abstract spatialisation, which refers to a specific spatial trajectory or precise position applied to a sound, is present in almost all spatialisation tools, possibly since it is direct and the most controllable and the easiest way to abstractly conceive a position in space. Concrete spatialisation takes more into account the internal characteristics or parameters of the sound, either with correlation by linking a spectral parameter or intensity of the sound to a spatial dimension, or with micro/macro decorrelation—for example, on time, playback speeds, transposition or distortion—which tends to create a diffuse/contrasted space. It is very difficult to predict the spatial effects or diffuse perception of some concrete spatialisation techniques like micro-delays or transpositions, where location is no longer the focus, rather than width and depth, as it indirectly acts on space through parameters of the same sound different in some or all the channels of the loudspeakers.

Algorithmic patterns of parameters mixing rhythm and space of sound events

Sequences of parameters allow to create sequences of spatialised sound events or choruses (copies or echoes of the same event with micro-delays or spectral/spatial variations) that can meet a multichannel effect system with different effect parameters on each loudspeaker.

Figure 5 shows the Sequence View to compose patterns or algorithms for all parameters (rhythm, density, sound synthesis, space and send to effects). Time control is not represented with x-time axis, but with multisliders, where you can draw a list of onset values or durations (see modules 1 and 3 of figure 5). Besides euclidean algorithms controlling rhythm, a topographic drum sequencer from the Eurorack module Grids has also been integrated into the tool thanks to the port in code to SuperCollider (6).

Figure 4 shows two views of a graphical representation with x-time axis of a sequence of events with different parameters. The difference between the top/bottom representations is the density parameter (see module 2 of figure 5), where all events are generated at the bottom and only some of them at the top. It can be obtained by changing a slider, at the right of each parameter module, which multiplies the values of the multislider. The sound events generated can then be each spatialised and routed to specific effects particularly thanks to modules 5, 6 and 7 of figure 5.

Figure 4: Two different views with (low/dense) densities of a graphical representation with x-axis time of the same sequence of sound events with different parameters on y-axis. At the top of each view, red circles (bigger or less depending on amplitude) mark the beginning and the black line is the duration of sound events. The y-axis of red circles represent playback values of sound files, which can be negative depending on the value of the blue line representing the position in the sound file, which in this case is the same with the points in yellow. At the bottom, three parameters are shown: amplitude, spatialisation type and pan levels. Both representations of patterns of parameters were realised thanks to the SuperCollider extension PatternPlotter from Jonatan Liljedahl: https://github.com/lijon/AlgoScore2. 
Figure 5: The Sequence View to compose fixed or algorithmic patterns for all parameters of the sequence of 48 sound events in this case: with e.g., at the bottom left, 1 rhythm and 2 density modules; in the center the spectrum of the sounds with 3 duration and 4 amplitude; at the top right 5 spatial trajectories and 6 algorithms and at the bottom right 7 the send to a selection of multichannel effects.

Mixing object-based and channel-based paradigms

An object-based paradigm (sound sent to a virtual 3D position) involves that spatialisation can be easily reproduced to other different spatial configurations, whereas a channel-based paradigm (sound or effect sent to a physical loudspeaker position) involves that spatialisation cannot be reproduced in the exact same way on different loudspeaker configurations.

According to the author’s experience, the simple treatment of channel-based paradigm of sending directly a sound to a loudspeaker channel tends to reinforce the presence of loudspeakers and have a low CPU usage, which has the effect of increasing the maximum number of spatialised events in real time, while object-based paradigms tends to make forgive the presence of loudspeakers and be more CPU demanding, which has the effect of lowering the maximum number of spatialised events in real time.

Despite the tendency towards object-based paradigm for reasons of standardisation and reproduction in every space regardless of speaker setup, mixing channel-and object-based paradigms to be used on every sound event allows to take advantage of the strengths of channel approach. Sending a sound directly to a specific loudspeaker can have much more impact (and at least a different effect) than sending it at the exact same coordinates of the loudspeaker through e.g., VBAP algorithm. However, it has also the effect that sequences integrating channel-based or a multichannel effect system are different according to the number of loudspeakers available and cannot be reproduced in the same way on different loudspeaker configurations and have to changed with other spatial trajectories or algorithms. Depending on the spatial configuration, different spatial algorithms are available. For example, in stereo, trajectory types are only limited to three (static panning between links and right, one way or round-trip between links and right), as well as the panning algorithms (equal power, delay-based and spectral).

The possible roads towards death

The tool Live 4 Life faces two main reasons that might cause its future end.

Technical death

As most Digital Musical Instruments, this tool is ephemeral due to technological obsolescence. Maintaining the code with recent updates of MacOS or SuperCollider platforms and solving bugs is a considerable burden. Some controllers have disappeared such as the Sensel Morph (7), the MIDI Touchbar (8) on older Macs, where I could get sliders integrated with the keyboard. Other Applications like Lemur (9) on iPad, which took me months of work to integrate within my tool, have temporarily stopped their development, and the use of this application with next iOS updates is currently not guaranteed.

Social death

At its creation, the two motives behind the development of this tool were the realisation of spatial improvisations in concert and a means to connect with others. The tool until now has lived because I had the chance to perform in concerts. The pandemic has not helped, but the current difficulty to find residencies or perform spatial improvisations in concert halls or festivals in good conditions may cause future death of this tool: without an appropriate allowance and a minimum time of a few days to prepare and adapt the spatialisation of works to the space and spatial setup of concert halls. Since concert halls with spatial setups are in great demand, time for creation and repetition is often limited. Without a space with several loudspeakers and a user to perform or the help from other developers/collaborators/users, this tool will die.

The possible roads towards symbiosis

Knowing the aforementioned risks, we now turn toward preventive solution avenues that will lead to a shared tool. The spatial sound performance has many links with symbiotic individuations both between machines, programs, works from other human beings (notably electroacoustic composers), the audiovisual domain and other communities.

Symbiosis between Code and GUI and Controllers

To extend the possibilities of algorithmic patterns, I first thought to integrate besides SuperCollider the use of other live coding languages, such as TidalCycles,¹⁵ by controlling pre-programmed code in real time via DJ interfaces. This was realised thanks to a developer who implemented my request (10) of being able to trigger via MIDI or OSC any parts of code in a text editor through the encoder of a controller. However, first tests showed CPU and synchronisation issues when using both text editors for SuperCollider with my tool and Atom for TidalCycles.

Symbiosis with other programs

As it is possible to see on the installation section of webpage of this tool, it is a patchwork of multiple codes from various sources. To extend the possibilities of digital signal processing, integration of open source or relatively free VST plugins is currently investigated, with plugins such as Surge XT (11), Vital (12) or modular systems like VCV Rack (13). More precise spatial algorithms with Higher Order Ambisonics will also be implemented in the tool via Sparta(14) or IEM plugins (15).

Symbiosis mixing electroacoustic works

I initiated just before the pandemic a new cycle of improvisations dealing with living alone and the feeling of loneliness, with the absence of symbiosis, whose Greek root means "living together." My way to deal with the hell of loneliness has been to symbolically ally myself and associate closely with other electroacoustic composers (Bernard Parmegiani, Francis Dhomont and Robert Normandeau until now), reusing and recycling their own sound materials and compositions. The willingness of this performance is to dissolve myself in the creation of someone else, in the same way as I dissolve the composer in my tool, in order to destroy the ego and go towards symbiosis. I, therefore, included in the spatial performance Parmegiani meets SuperCollider (16) three major works from the electroacoustic composer Bernard Parmegiani, which are rearranged, processed and triturated by the machine algorithms and my gestures.

The primary sonic source consists of 2-hour records from the electroacoustic composer Bernard Parmegiani, i.e. De Natura Sonorum, Rêveries and Hell from Dante’s The Divine Comedy, which have been cut in more than 1.700 sound samples and classified in dozens of folders. These sound files have been cut so as to obtain and mix a wide variety of sounds of different durations, both isolated elements and repeated a few times to have a brief musical phrase. The sound database has been grouped into fifty folders differentiating between the original tracks and my own sound categories (short/long durations, soft/violent elements, resembling instruments / natural elements or voice). Similarly to the cut-up technique, which was popularized by writer William S. Burroughs, these sound files are then rearranged and spatialised in enveloping vortexes around the audience. Their tempo and playback speeds are frenetically modified, thus accelerating or slowing down the lives of large amounts of sound objects.

Audiovisual symbiosis

Audiovisual perception Although, by definition, acousmatic music allows us to focus on a single sensory modality in order to better perceive sound information and free our imagination, multi-sensory integration helps and has a strong impact not only on object discrimination and reaction time to events, but also on emotional response, particularly in the peripersonal space.¹⁶ Perceptual and cognitive mechanisms are made to process signals that are of multiple sensory nature. Given that an individual can no longer integrate two sensory information and identify the source of the event when the spatial or temporal delay is too great,¹⁶ it would be possible to play on joint or disjoint audiovisual spatialisations. In the same way that certain studies, exploring the associations between sound and visual dimensions, suggest that a sound parameter can correspond effectively to several visual parameters¹⁷ and especially that the choice and sense of the scale correspondences would have more importance than the correspondence itself,¹⁸ the same relationships and associations could be made between the sound, the visual and its space, where multiple relations can coexist.

Audiovisual creations To meet the needs for audiovisual and more immersive works for the eyes, an audiovisual object could be composed by transmitting via the OSC protocol the synthesis parameters of each audio event from the Super-Collider language to audiovisual programming environments, such as Processing or Open Frameworks, in order to represent each sound object with a virtual graphic object, a symbol, a video or an image, whose position, playback speed, offset, transparency, movement or other video effects will be controlled by the sound synthesis parameters. Sound images could be able to rotate like on record players depending on their playback speeds. Audiovisual composition can also be used to generate symbolic scores, so that this representation sheds light on the process of composition and improvisation.

First experiments with Processing are shown in figure 6. Simple shapes are triggered with OSC from sound event data, distinguishing percussive sounds with cubes from others with spheres, accumulating by transparency. Forms disappear by transparency and distance following the duration of sound events. Playback speed was mapped to the vertical position, panning to horizontal position, intensity to the vertical size and transparency, the duration to the horizontal size and transparency and the buffer folder to color. Streaks deforming forms can be seen when triggering some shortcuts changing playback speeds of sound events. A similar patch has also been developed in Touch Designer and will soon be available on Github.

Audiovisual collaborations Audiovisual collaborations are looked for both with video developers to create the visual from the sound data generated, and with dancers to finally assess the impact between the gesture of the performer, the dancing bodies and the video environment. Technically, the audiovisual object mapping could be developed through open source tools, such as Processing, Open Frameworks, Hydra, or even through commercial tools like Touch Designer and Resolume, as long as the creation process and the code are published on Github and open to everyone. 

Figure 6: 21 screenshots of realtime representations of playing sound events with different parameters sent by OSC to an experimental patch in Processing generating different forms based on circles and squares accumulating by transparency.

Conclusion and future work

In this paper, we explained the free and open source motivations of the tool and its main characteristics of event-based spatialisation. It will live until there is sufficient funding, a user/developer team, an appropriate environment with spatial setups in concert halls and enough time to adapt pieces to concert space. Otherwise, without collaborators and users, it will soon die. This current research-creation project in spatial improvisation (sound and multi-sensory) has three axes:

• Thecreationofperformativeandparticipativeinstallations, where the public will be able to experience in real time the control of the parameters of multiple sound objects in space via several interfaces. The sound performance tool will thus be evaluated by the public, both from a perceptual point of view and in terms of control and ergonomics.

• The establishment of training workshops to use and experiment free tools to create and improvise with space,

• The creation of spatial improvisations in a multi-sensory context, combining and alternating music, video / light and dance.

Acknowledgements

This postdoctoral research and further development of these tools were made possible thanks to research grants from FRQSC (Fonds de Recherche du Québec - Société et Culture/ B5 - 318691) and Hexagram-UQAM


(1) https://github.com/Xon77/Live4Life

(2) https://www.fredrikolofsson.com/code

(3) https://chdh.net/egregore source.php

(4) http://chdh.net/vivarium.php

(5) See the interview “Resist and Transcend” from Raphae ̈l Cendo: https://vimeo.com/240664490, accessed September 15, 2023

(6) https://github.com/capital-G/sc-grids

(7) https://morph.sensel.com, accessed September 15, 2023 

(8) https://miditouchbar.ch

(9) https://www.midikinetics.com

(10) https://github.com/tidalcycles/atom-tidalcycles/issues/119

(11) https://surge-synthesizer.github.io

(12) https://vital.audio

(13) https://vcvrack.com

(14) https://leomccormack.github.io/sparta-site

(15) https://plugins.iem.at

(16) https://youtu.be/Nb7ITdBkikE 

References

1  R. Penha, J. P. Oliveira, Spatium - tools for sound spatialization, In Sound and Music Computing Conference, Stockholm, Sweden, 2013, 660–667.

2 Thibaut Carpentier, Markus Noisternig, Olivier Warusfel, Twenty Years of Ircam Spat: Looking Back, Looking Forward, Denton, USA, 2015, 270–277.

3 Christophe Lengelé, Live 4 Life - A spatial performance tool focused on rhythm and parameter loops, In International Computer Music Conference, Daegu, South Korea, 2018, 298– 303.

4 Francis Dhomont, Abstraction et figuration dans ma musique... et autres considérations, eContact, 16(3), 2013.

5 Francis Dhomont, Abstraction et figuration dans ma musique, In Lecture by Francis Dhomont as part of the colloquium for graduate studies in composition at Université de Montréal, Montréal, Canada, 2018.

6 James Mc Cartney, Rethinking the computer music language: Supercollider, Computer Music Journal, 26(4):61–68, 2002.

7 Jacques T. Godbout, Ce qui circule entre nous. Donner, recevoir, rendre, Paris, France, Éditions du Seuil, 2007.

8 Ursula M. Franklin, The real world of technology, Toronto, Canada, House of Anansi Press Inc., 2004.

9 Christophe Lengelé, Live4Life: Aspatial performance tool to play the ephemeral and improvise with space and playback speeds, Organised Sound, 26(1):89–99, 2021.

10 Christophe Lengelé. The story and the insides of a spatial performance tool: Live 4 Life. In International Computer Music Conference, Santiago, Chili, 2021, 133–139.

11 Scott Wilson, Spatial swarm granulation, In International Computer Music Conference, Belfast, Irlande, 2008.

12 Horacio Vaggione, Décorrélation microtemporelle, morphologies et figurations spatiales du son musical, In Anne Sédès, editor, Espaces sonores, Paris, France, Éditions musicales transatlantiques, 2002, 17– 29.

13 Laurent Pottier, Le contrôèle de la spatialisation. In Laurent Pottier (ed.), La spatialisation des musiques électroniques, Saint-Etienne, France, Publications de l’Université de Saint-Etienne, 2012, 81–104.

14 Ville Pulkki, Virtual sound source positioning using Vector Base Amplitude Panning, Journal of the Audio Engineering Society, 45(6):456–466, 1997.

15 Alex Mc Lean, Algorithmic Pattern, In Conference on New Interfaces for Musical Expression, Birmingham, UK, 2020, 265–270.

16 Isabelle Viaud-Delmon, Conférence : les aspects cognitifs de la perception 3D, Forum International du Son Multicanal, 2014.

17 Scott D. Lipscomb, Eugene M. Kim, Perceived match between visual parameters and auditory correlates: an experimental multimedia investigation, In International Conference on Music Perception and Cognition, Sapporo, Japan, 2008, 72–75.

18 Florent Berthaut, Myriam Desainte-Catherine, Martin Hachet, Combining audiovisual mappings for 3D musical interactions, In International Computer Music Conference, New York, USA, 2010, 357–364.

Authors Biographies

Christophe Lengelé is a spatial sound designer and performer for electronic and experimental music. After a Doctorate in music (composition and sound design) at Université de Montréal in 2022, he is currently doing a postdoctoral research/creation in spatial improvisation at UQAM (Canada). He seeks to bring together the spheres of composition and improvisation and focus on performing variable spatio-temporal pieces with a global custom live tool, he has been regularly developing since 2011, in order to play the place and the music at the same time. His works have been presented and performed internationally, particularly at the International Computer Music Conference (2018, 2021), during the Journées d’Informatique Musicale in France (2017, 2019), as well as at Akousma Festival 2021 (Canada) and more recently at Cube Fest 2022 (USA).

Philippe-Aubert Gauthier received a B.Ing. degree in 2000 and a M.Sc. degree in active control of vibration in 2003 (Université Laval, Canada). From 2003, Mr. Gauthier was enrolled at Université de Sherbrooke (Canada) from which he received a Ph.D. degree in 2007, dedicated to adaptive wave field synthesis. He is interested in spatial sound, multichannel sound reproduction, audio signal processing, room acoustics, loudspeaker and microphone arrays, spatial hearing and psychoacoustics. He was adjoint professor at the Mechanical Engineering Department from 2017 to 2019. He then took up a professorship at the École des arts visuels et médiatiques at UQAM (Canada) in 2019. He is a professional sound artist, an electronic musician, working at the intersection of arts, sciences and technologies. He is currently the associate director of artistic research at CIRMMT (Centre for Interdisciplinary Research in Music, Media, and Technology). 

A Latin America Network for Art and Cybernetics: The Centre for Art and Communication (CAyC)

Jose-Carlos Mariategui

jcm@ata.org.pe

Abstract

Towards the end of the 1960s—a period of intense creative, technological and political changes—the Argentinian art critic and entrepreneur Jorge Glusberg founded the CAyC in Buenos Aires. CAyC was an interdisciplinary experimental project that explored the relationship between art, technology and society. It sought to articulate a network of discussions and productions by a new style of Latin American artist, deeply influenced by science, technology and society. Glusberg defined such practice as Systems Art, which appeared in three ways, namely as a system of collective representation; a system of meaning that defied formal categories; and a system of relationships and processes for social inquiry. In doing so, the artist became a researcher who reflected on their social context and the latter’s processes of production. This paper will discuss CAyC’s pioneering work and its global influence through three main initiatives: its exhibitions Art and Cybernetics, Systems Art in Latin America and the International Open Encounters on Video. These events were driven by the revolutionary artistic and experimental promotion of the distinctive ways in which Latin American artists were using technology to respond to local issues at a time when computer systems and cybernetic models for management and organizational practices were being introduced across the region.

Keywords

Systems art, Latin America, Cybernetics, Video art.

DOI

10.69564/ISEA2023-71-full-Mariategui-CAyC

Introduction

Towards the end of the 1960s, art critic and entrepreneur Jorge Glusberg created the Centro de Arte y Comunicación (Centre for Art and Communication) in Buenos Aires. The Centro de Arte y Comunicación, popularly known by its acronym CAyC, fostered the intermingling of disciplines, with communication and cybernetics at its heart. Glusberg conceived CAyC as an interdisciplinary and experimental art centre which explored the relationship between art, science, technology and social studies. It sought to articulate a network of Latin American artists, critics, intellectuals, scientists, and other cultural practitioners around production, debates and exhibitions. As described in an early 1970s diptych promoting CAyC’s activities (1):

The Centre for Art and Communication aims to promote the execution of projects and experiments where art, technological media and the interests of the community are brought together in an effective exchange that puts in evidence the new unity of art, science and the social environment in which we live. Its fundamental objectives are to encourage, support and develop those social tasks, experimental studies or research in the arts and group communication, which give rise to an interdisciplinary integration, to improve and expand the current scenario of human concerns. It is formed by artists, sociologists, logicians, mathematicians, art critics and psychologists pursuing common goals with a highlight on the behaviour and development of phenomena of mass communication and the collapse of traditional forms of practice, create an opening for new systems of expression, where researchers and artists try to outline the artistic interests of the man of the XXI century.

CAyC showcased internationally a new regional genre of art which Glusberg defined as systems art identifying it with post-object art practices and experiments that were developed at that time in an international arena, deeply influenced by science and alternative uses of communications, mass media and technology.

In what follows, I will address the particular modes and practices of CAyC as it developed what was quite possibly the first art and technology network in Latin American reaching international visibility. The following three sections outline some of CAyC’s historical milestones: its founding approaches as encompassed in the show Art and Cybernetics which brought together engineers and artists who explored the creative possibilities of computers, as well as the exhibition Systems Art in Latin America, which circulated internationally and created an original mode of circulation. We also provide an interpretation of the nature of CAyC’s practices that seem to be the outcome of processes of production and distribution of standardized and reproducible content. We also explore how CAyC articulated its network as a vital component of its approach to systems and cybernetics in Latin America’s political and social context. Finally, we briefly share some conclusions based on our findings and give some directions for further research. 

The establishment of the CAyC: Art and Cybernetics

Following the Second World War, the field of cybernetics gained scientific and intellectual recognition. In the 1960s, an outburst of social and artistic interdisciplinary movements and research practices prompted ephemeral events involving the audience (the observer) as an active participant, allowing artists to question their own practice and its impact on society. What emerged as a consequence was an engagement with post-object practices concerned with creating an awareness of the real as a relational and dynamic network of processes that have a political and social impact. ¹ 

In 1968, art critic and entrepreneur Jorge Glusberg founded the Centre of Studies on Art and Communication of the Foundation of Interdisciplinary Research (CEAC is the Spanish acronym) (2). On 18th August 1969, the CEAC opened its first exhibition, Art and Cybernetics, at the Bonino Gallery(3),2 inspired by a visit Glusberg made to Japan in February 1969, where he established contact with Tokyo’s Computer Technique Group (4) (CTG), a collective of art and engineering students founded by Masao Kohmura and Haruki Tsuchiya in 1966 and partially funded by the IBM Scientific Data Centre in Tokyo. CTG was both a design office and a think tank focused on computer analysis and art practice based on image processing and geometric transformations. Glusberg later said that CTG: “Combined their traditional graphic sensibility with the technique and rigor of the scientific star of the moment: the computer.”³ Initially, Glusberg thought only of presenting CTG in an exhibition in Buenos Aires as he brought from his trip a series of works from the Tokyo-based artist group. However, Glusberg set up also a research group on arts and cybernetics (Grupo de Arte y Cibernetica) and along with IT engineers Julio Guibourg (Director of the Computation Centre at the ORT School) and Ricardo Ferraro (Professor of Policy and Management of Science and Technology at Buenos Aires University), organized a seminar which gave artists access to computers and the chance to explore their creative possibilities (5). The foundational group was comprised of Luis Fernando Benedit, Antonio Berni, Eduardo Mac Entyre, Osvaldo Romberg, and Miguel Ángel Vidal. Further members included artists Ernesto Deira, Hugo Demarco, Gregorio Dujovny, Mario Mariño, Rogelio Polesello, Isaias Nougués, Josefina Robirosa and Norma Tamburini. The seminar included other scientific researchers such as Arturo F. Montagú—a pioneer in architectural computing–; Pablo Jonovich; César Armoza; Silvia L. Yacub and Silvia Brucciamonti.⁴, , ⁶ Ferraro was in charge of the IBM 1130(6) Computing System Model 2C (16-bit binary architecture with a core memory of 16,384 words and a memory cycle time of 3.6 μs) and two automatic drawing machines or “plotters” (IBM 1627 Plotter) equipped with ball point pens which converted tabulated digital information into a graphic format. The software was running on Fortran IV, the main computer language for the IBM 1130. The exhibition showcased “living art, created by innumerable pioneers of our time, who use ideas, synthetic shapes or mathematical equations instead of paintings; lights and motors, and information instead of brushes.”⁷ From the artists participating in the seminar, only Berni, Benedit, Deira, Mac Entyre, Romberg and Vidal exhibited their works and demonstrated “their courage as pioneers in the complicated jungle of plotters, equations, matrices and perforated bands.”⁸ Their works were screen prints produced by the plotters based on computer generated drawings of polynomial interpolations, repetitions, geometric patterns and other mathematical designs and shapes (triangles, squares, rectangles) using mathematical properties. sThe exhibition also included twenty works by members of CTG (Koju Fujino, Takeshi Hasegawa, Junichi Kakizaki, Masao Komura, Fujio Niwa, Makoto Ohtake, Haruki Tsuchiya and Kunio Yamanaka), six works from Motif Editions, London (works from CTG, M. S. Mason, Boeing Computer Graphic, Kerry Strand, Csuri and J. Shafffer and D. K. Robins) (7) and featured electronic music by five local composers: Dante Grela, Francisco Kröpfl, Carlos Rausch, Jorge Rotter and Eduardo Tejeda. As Glusberg stated in the exhibition catalogue of Art and Cybernetics:

One of the most important characteristics of cybernetics is its universality (reflecting) the possibilities of this new scientific discipline for the comprehension of most heterogeneous phenomena. The development of some cybernetic models permitted the building of artificial systems, which less than ten years ago were distinctive and exclusive traits of human activity. The artists of this time are more interested in behaviour than in the essence of things; this tendency can be clearly identified with cybernetic vision.⁹ 

The intention of the project was to foster a multidisciplinary community among engineers, scientists, philosophers, theorists, artists and architects as well as to support and give visibility to a group of artists interested in practices combining science and technology which were already starting to become prevalent internationally. Glusberg understood the universalistic character of cybernetics and wanted to emphasize that its application in artistic practice enabled new interpretations that could act as active forces for understanding human behaviour and furthermore to question and change society, in particular as he later mentioned, “as a new form that emerges as a result of a regional problematic, one that uses a methodology common to different contexts.”¹⁰

During the exhibition, workshops on “automatic drawing techniques” (Ricardo Ferraro, Nestor Sameghiniy and Marcelo Larramendy) and the use of computation and plotters (Arturo Montagu, Ricardo Valek, Pablo Jononovich, Cesar E. Armoza and Silvia L. Yacub) were organized. In addition, a Seminar on Information and Approach was held with lectures by, among others, Manuel Sadosky, an influential mathematician and computer pioneer who led the implementation of the first computer for research and education in Argentina, a Ferranti Mercury nicknamed Clementina. (8),¹¹

The Art and Cybernetics exhibition was the first of its kind in Latin America and took place only one year after seminal exhibitions such as Cybernetic Serendipity at the Institute of Contemporary Art (ICA) in London (9) and Some More Beginnings: An Exhibition of Submitted Works Involving Technical Materials and Processes (10) at the Brooklyn Museum in New York. It took place in the same year as other influential exhibitions such as The Machine as Seen at the End of the Mechanical Age (11) at the MoMA, in New York, Mind Extenders in Minneapolis (12) and Tendencies 4 (13), at the Gallery of Contemporary Art in Zagreb; and, a year before Information (14) at the MoMA in New York and Software: Information Technology: Its New Meaning for Art (15) at the Jewish Museum in New York. Cybernetic Serendipity was one of Glusberg’s main inspirations as it combined computer-generated graphics and computer music along with cybernetic and motorised sculptures and manipulated television sets. Glusberg had established correspondence with Reichard in 1967 and in 1969 he asked for authorization to reproduce the recordings of Cybernetic Serendipity for a show at the CAyC (16). It is well documented that Glusberg invited Reichard to Buenos Aires several times. She finally visited in November 1970 to give two talks (17) at the CAyC18. Prior to her arrival in Argentina, the newspaper ACCION published an article by Reichard titled “Cibernética Artística” (“Cybernetic Art”) (19). At the time, Reichardt was very interested in Latin American art and culture, particularly the work of Argentine artist Leopoldo Maler and writer Jorge Luis Borges (20).

After Buenos Aires, Art and Cybernetics travelled to Córdoba, Santa Fe and Tucumán in Argentina and then to Montevideo (1970) (21) and Lima (1971) (22), as well as to Minneapolis, San Francisco, New York, London and Tokyo. ¹² Glusberg realized that exhibitions based on reproducible media—such as computer printouts—permitted an efficient and inexpensive diffusion model, which he used later in other traveling shows.

In 1970, CAyC opened its iconic headquarters at 452 Viamonte Street in Buenos Aires with an entrance reminiscent of a techno-futuristic spaceship or the shape of punched cards used in computer systems. CAyC—which was independently funded by Glusberg—organised exhibitions and events welcoming local as well as prominent international artists, theorists and critics. As we will see, it became an active player for the production and diffusion of new and innovative technology-driven art forms.

Systems Art in Latin America

The dismantling of cultural activities and the human rights abuses that began with the military coup in 1966 caused many artists to leave the country. In this context CAyC provided an important refuge for artists by actively creating a multidisciplinary space for reflections around art, technology, communication and society. As we mentioned earlier, CAyC sought to showcase a new regional artform, which Glusberg defined—following the work of Jack Burnham—as “systems art,” encompassing a wide-ranging selection of works that can be loosely articulated within such a definition. After Art and Cybernetics, CAyC’s second exhibition was titled De la Figuración al Arte de Sistemas (From Figuration to Systems Art) (23). Organized at the Museo Provincial de Bellas Artes Emilio A. Caraffa in Córdoba in August 1970, it showcased works from Nicolás García Uriburu, Edgardo Antonio Vigo, and Luis Benedit. However, as art researcher Mari Carmen Ramirez stated,

Despite the innovative nature of their work and Glusberg’s vague efforts to justify their inclusion in the exhibition, there was little in Uriburu’s or Vigo’s art that could be considered systems art [...] Their somewhat more modest proposals could not compete on an equal footing with the conceptual and technological sophistication of Biotrón, Fitotrón, or any of Benedit’s living systems.¹³

Benedit, an architect and self-taught artist and designer, was working with living habitats, installations, illustrations and technical construction drawings linked to ecology and analytical investigations. Benedit’s Biotron (24) was an impressive environment or “living habitat” of four thousand bees, encouraging the audience to learn while being active observers (here the observer is an indissoluble part of the system). Benedit studied the behaviour of bees, among other animals, to produce artificial and rational systems of organization and observation.¹⁴ In fact, Benedit’s Biotron should be understood as a work that reflected on both the cybernetics and systems theory fields, whereas the living system environment is a complex of interacting components set towards specific goals, while observation as an event produces distinctions which inform the system about a difference.¹⁵ Based on systems theory, the living systems of Benedit help to construct a framework of diverse phenomena that are intrinsically dynamic within organisms, social systems and ecosystems.

On the 19th of July 1971, a third CAyC exhibition called Arte de Sistemas (Systems Art) was presented at the Museo de Arte Moderno de Buenos. Artists from Latin America, the USA, Europe and Japan were invited, including some prominent artists such as Vitto Aconcci, Dan Graham, Hans Haacke, Allan Kaprow, Joseph Kosuth and Dennis Oppenheim. Perhaps the most relevant aspect of this exhibition was its original format and scale. It expanded to include 101 experimental artists who submitted their works in a format that shaped many of the exhibitions of CAyC between 1970 and 1975, namely a single format on blueprint paper, IRAM (25) coded, 59.5 x 84 cm, responding to the idea of creating a standardized system where all participants would participate on an equal footing. The work submitted by each artist could be reproduced from the heliographic format, for different versions of the exhibition in other countries. Glusberg understood that production of a common scheme on a homogeneous and easily reproducible medium allowed his enterprise to travel across a small circuit of museums, galleries, schools, or even neighborhood organizations.¹⁶

In May 1972, the CAyC traveling exhibition Towards a profile of Latin American art, first presented at the III Coltejer Biennial, in Medellín (Colombia), incorporated the idea of system with respect to the notion of politics and ideology, particularly in Latin America. The exhibition’s presentation text evinces such reflection:

The conflicts generated by the unfair social relations that prevail in the Latin American countries cannot fail to appear in this facet of cultural life. [...] Our artists became aware of the requirements of their national realities and thus proposed regional responses that are consistent with the change in all areas of human life that affect the underprivileged of today, those who we think are the potentially privileged of tomorrow. ¹⁷ 

The exhibition Art Systems in Latin America circulated between 1974 and 1976, expanding the original mode of circulation of standardized blueprints as well as adding to it an ideological framework. In the introduction to Art Systems in Latin America, Glusberg demarcates the cultural process in Latin America and how it is different from those of the “first world.”¹⁸ His criticism is associated with the domination mechanisms that prevailed at the time when terms such as “Third World” or “underdevelopment” were coined as part of support policies, but also as new mechanisms of post-colonial domination, generally referenced under the label of “ideological conceptualism.”¹⁹ Like many at that time, he criticizes the “grouping” of artists for their social characterstics in “the same network.” Thus, he proposes the development model in Latin America to be an alternative development, with a commitment to a new art that “tries to break away from the instruments of ideological domination of the countries that have the concentration of power and wealth, sometimes taking advantage of their own methodologies and languages.”²⁰ 

The point here is that the set of works proposes different “liberation strategies” through operational models that define political and social contexts, and to articulate a proposition for change from an artistic and cultural perspective. In that sense, the works that were presented do not represent a particular Latin American content, demystifying a free group, except the group fighting for their liberation.

It is in this show that Víctor Grippo presented his famous work Analogía I (Analogy I) (1970-1971), in which an analogue voltmeter measures the amount of energy contained in a “system” of forty freshly grown potatoes, all interconnected in series and parallel electrical circuits. The electrodes connecting each potato at the end of the wires are made of different metals—one copper, the other zinc-galvanised iron—which reciprocally balanced processes of oxidization and reduction occurring in the electrodes producing an electrical current displayed in the voltmeter. The work displays the potato, which is of Latin American origin, to showcase its contribution to the world as a reservoir of energy, but also to explore its symbolic value through analogies with knowledge and consciousness.²¹  Interconnecting the potatoes depicted how the tiny voltage of each potato multiplied; such combined energy also became a metaphor for the power of collective human consciousness and its transformative potential for liberation.²²

However, having each potato in a small cell resembled the imprisonment and brutality occurring in Argentina at that time, as Justo Pastor Mellado comments “the electrode brought up images of torture, and in particular, of electrical current applied to the human body. The potatoes were reminders of an inert condition, in an ambiguous way.”²³ The wires act to both connect each potato together and register the amount of energy generated while the voltmeter represents connectedness and community expressed through science. The piece was operated through a button that the public could use to read the amount of energy being generated by the wired spuds, giving the audience the opportunity to become actively involved with the work as it was being transformed into a pedagogical exercise that revealed a non-traditional source of energy. The piece also includes a small text contrasting the word papa (potato in the Andean Quichua language) with the word conscience, proclaiming the search for knowledge and a desire to convey the possibilities and the processes of transformation. A trained chemist, Grippo joined the CAyC group and began work on projects that explored relations such as nature-culture and real-artificial. By using food, minerals and domestic objects, he tried to conceptually depict the inconsistencies and abuses of natural resources, as well as the relation between nature and humanity.

The works of Benedit and Grippo could be seen as pioneering experiences at that time, and within the new Latin American systems art they contained principles from cybernetics, such as the importance of the observer as well as the establishment of information mechanisms which produce new immaterial symbology amenable to socio-political interpretations. In addition, as Joanna Page suggests, these works were a significant contribution to decolonizing science and nature, expanding the conception of knowledge, in which cognition is a thoroughly sensory and embodied affair, reflecting the fact that our knowledge of the world is not a knowledge of something outside or beyond us, but of something in which we are always already entangled.²⁴ 

CAyC’s approach to Cybernetics

CAyC pioneered a vehicle for the diffusion of a new art practice in Latin America based on ideas around cybernetics, systems art and video art, with a privileged focus on processes and immaterial information over more traditional objectual practices. CAyC acted as more than a diffusion agent, since it enabled the creation of original projects not only in Argentina, but internationally. CAyC built a network of Latin American artists visible on the international stage, disrupting traditional artistic notions that typified Latin American art based on its ancestral or historical cultural practices.

While CAyC would ultimately adopt “systems” as a catchphrase to describe all sorts of conceptual art practices and most of the projects organized during the 1970s, its discourse progressively became more critical towards the political scene in Latin America at a time when freedom was very limited by several military dictatorships throughout the region, that suppressed the revolutionary ideas of the previous decade. Such a confrontational position was a necessary demand from artists throughout Latin America; CAyC, as Glusberg mentioned “had become aware of what their national realities demanded and had offered responses to regional issues.”²⁵  CAyC’s strategy helped to consolidate and legitimize Latin American art and their critical involvement in international tendencies, acting as a catalyst that allowed visibility.²⁶ Thus, cybernetics and systems theory “framed” the art pieces as a networked process with the goal of entering the international art stage.²⁷

Even though in some cases Glusberg borrowed concepts from systems and communication theories in a loose manner, the exhibitions organised by CAyC paid attention to concepts such as relational, autonomy and open-endedness which were fundamental to systems theory, and introduced to the artistic community terms such as systems art, art and technology, and video art. Systems art was a category linked to technology, processes and social dynamics but, as Rodrigo Alonso accounts:

"The Glusbergian vision is much more imbued with the socio-political context of its time. In a country governed by a military dictatorship, but still moved by revolutionary ideas and by the desire for a return to democracy, artistic production had to account, in some way, for the complexity of this unique historical moment.”²⁸

However, as Julieta Gonzalez points out, The concept of information as power was fundamental for CAyC, which, though initially embracing the constructive-semantic imperatives of the kinetic, computational model of the early 1960s, eventually departed toward a radicalized agenda of direct action and intervention as dictatorial regimes spread throughout South America.”²⁹

Starting from the Art and Cybernetics exhibition, which was the first of its kind in Latin America, we can study the genesis of art and technology and its accompanying network of artists. Within the loose definition of cybernetic art and systems art, it brought together a diversity of artists and creators that were not able to find a space in the traditional art world. CAyC supported and promoted a group of artists and thinkers who are now considered pioneers in art, video and cybernetic practices such as Luis F. Benedit and Victor Grippo (Argentina), Margarita Damico (Venezuela) or Pola Weiss (Mexico). In addition, the CAyC used the reproducible character of heliographs and video, as well as “DIY installations”, to respond with inexpensive exhibitions in a time when art and technology shows were known to be expensive and complex, requiring both corporate sponsorship and technological support.

Most well-known and emblematic large-scale exhibitions on art and technology during that time were quite expensive and very difficult to tour. Jack Burnham went further by decrying that, after several exhibition attempts, art and technology seem to prove mutually exclusive. For instance, the Cybernetic Serendipity show, which had a modest budget with no on-site computers or terminals, when it was shipped to the Corcoran Gallery of Art in Washington D.C. the following year, had a considerable portion of its contents destroyed because of poor packing and handling and its curator, Jasia Reichardt, publicly disowned what was shown there.³⁰ Severe maintenance costs of expensive equipment entailed operational difficulties. In that sense, the “CAyC exhibition model” proved a viable way to show technological works in a moment where the technology was complex, expensive and not widely available, thanks to the former’s flexibility and cost efficiency. Its outcome was demonstrated by the extensive network of relations and collaborators in more than 40 cities nationally and internationally, demonstrating CAyC was capable of producing a rhizomatic network. More importantly, the CAyC attempted to create a Latin American movement, in which the use of “foreign” technologies—seen as dominant and difficult to use—could provide autonomy, creative liberation and opportunities for development while fostering civic participation and collaboration, two concepts that were to become the foundation of modern computing and the internet.

Final Remarks

We argue that CAyC’s approach to systems and cybernetics contributed to the creation of a group of practitioners from different disciplines and backgrounds which supported the innovative undertakings of non-objectual practice in Latin America. CAyC’s work is distinctive of its time, as a transition from an object-oriented to a systems-oriented culture, shifting art from material entities towards non-objectual representations. This also enabled an art practice that was much more autonomous and collaborative. The innovative model conceived by CAyC forged a model for social production both in terms of a platform for sustainability and support for more innovative and experimental endeavors. Such endeavors were based on conditions that promote disciplinary models of production through active experimentation and facilitating the interaction between different types of practitioners, such as architects, designers, scientists and artists who joined forces to configure a new Latin American art.


(1) CAyC information diptych, Rodrigo Alonso Archive, Buenos Aires, (1971?).

(2) Centro de Estudios de Arte y Comunicación de la Fundación de Investigación Interdisciplinaria.

(3) At that time directed by Guillermo Whitelow.

(4) Their works were shown at Cybernetic Serendipity (1968) as well as in the 6th Paris Biennale (1968) and the 35th Venice Biennale (1970).

(5) In Buenos Aires, between March and April 1969.

(6) For more information on the IBM 1130, http://ibm1130.org/, The IBM 1130 was also used for the first experiments on graphical user interfaces developed by Alan Kay in his Ph.D, Thesis in University of Utah in 1969.

(7) These group also created the Computer Arts Society in 1968. 

(8) Installed in 1961 at the Computational Institute (Instituto de Cálculo) of the University of Buenos Aires.

(9) Cybernetic Serendipity, an exhibition of cybernetic art curated by Jasia Reichardt, held at the ICA, London, 1st August to 20th October 1968, later touring the United States, Corcoran Gallery of Art, Washington, D.C., and the Exploratorium in San Francisco.

(10) Some More Beginnings: An Exhibition of Submitted Works Involving Technical Materials and Processes at the Brooklyn Museum, New York organized by Experiments in Art and Technology (EAT), 25th November 1968 through 5th January 1969.

(11) The Machine as Seen at the End of the Mechanical Age curated/directed by Pontus Hulten, Director of Moderna Museet, Stockholm, an exhibition of more than 200 works of art and related objects held at The Museum of Modern Art, New York, from 27th November to 9th February 1969.

(12) Mind Extenders curated by Mimi Shorr, held at the Museum of Contemporary Crafts of the American Craftsmen' s Council, Minneapolis, from 19th April to 15th June 1969.

(13) Tendencies 4 organized by the New Tendencies movement, Galerija suvremene umjetnosti (Gallery of Contemporary Art, now the Museum of Contemporary Art Zagreb) in Zagreb, from 5th May to 30th August 1969.

(14) Information, an exhibition curated by Kynaston L. McShine, held at the MoMA, New York, 2nd July - 20th September 1970.

(15) Software - Information Technology: Its New Meaning for Art curated by Jack Burnham, held at the Jewish Museum in Brooklyn, New York City, from 16th September to 8th November 1970, and the Smithsonian Institution, Washington, D.C., from 16th December 1970 to 14th February 1971

(16) Jasia Reichardt. Personal correspondence with Jorge Glusberg. 20th April 1969. Jasia Reichardt personal archive, London.

(17) The titles of these two presentations were: Pop Art and Towards a Future of Art. Jasia Reichardt personal archive, London. 

(18) Jasia Reichardt. Personal communication with the author. 19th Jan 2020.

(19) ACCION, 3rd July 1970. Jasia Reichardt personal file, London. 

(20) Reichardt met Borges during her first visit to Argentina and invited him to London, where he delivered four lectures (“Four evenings with Borges”) in Central Hall in 1971organised by the ICA. Jasia Reichardt. Personal communication; 19th Jan 2020.

(21) In June 1970, the exhibition travelled, under the auspices of the National Commission of Fine Arts of Uruguay to the National Museo of Fine Arts in Montevideo, directed by Angel Kalenberg. The catalogue was prepared in cooperation with IBM Uruguay S.A.

(22) The exhibition was presented at the IAC (Instituto de Arte Contemporáneo) on the 19th October 1971 with the support of IBM Perú, and a panel on Arts and Cybernetics was held on the 27th October. Correspondence between Alfonso Castrillón and Glusberg dated 31st August 1970 discussed the plan to organize a Symposium on Art and Industry; however, that idea apparently never materialized (Alfonso Castrillón personal archive, Lima). 

(23) A few months later, in February 1971, CAyC organized its second exhibition in London’s Camden Arts Centre under the same title, but this time it included a larger number of artists.

(24) The Biotron installation was presented at the 35th Venice Biennale, Venice, 1970.

(25) IRAM code was standard criteria defined by the Argentine Normalization and Certification Institute, the International Organization for Standardization (ISO) member body for Argentina.

References

1 Ionit Behar, “Arte de sistemas: conceptual art and politics in Argentina (2015)”, Henrique Faria website, accessed July 29, 2021, http://www.henriquefaria.com/content/uploads/news-press/pdf/artslant_-_arte_de_sistemas_conceptual_art_and_politics_in_argentina-843.pdf

2 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la Galería Bonino de Buenos Aires, Buenos Aires, CEAC, 1969, ⁸.

3 Jorge Glusberg, Del pop-art a la nueva imagen, Buenos Aires, Ediciones de Arte Gaglianone, Coleccion Union Carbide, 1988. 

4 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la Galería Bonino de Buenos Aires, ⁸.

5 Melanie Lenz, “Centro de arte y comunicación (CAyC),” Word and Image / V&A Blog, July 8, 2016, accessed August 3, 2023, https://www.vam.ac.uk/blog/caring-for-our-collections/centro-de-arte-y-comunicacion-cayc

6 Melanie Lenz, “Early argentine computer art at the Victoria & Albert Museum,” Journal of Design History, Vol.31, No. 02, 2018, 154-166.

7 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la Galería Bonino de Buenos Aires, ¹⁰.

8 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la Galería Bonino de Buenos Aires, ²².

9 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la Galería Bonino de Buenos Aires, ⁸

10 Jorge Glusberg, Arte e ideología en CAyC al aire libre, Buenos Aires, CAyC, 1972, 1.

11 Cecilia Berdichevsky, “The beginning of computer science in Argentina — Clementina - (1961–1966),” in History of Computing and Education 2, ed. J. Impagliazzo, New York, IFIP Advances in Information and Communication Technology, vol 215. Springer, 2006, 203-215. https://doi.org/10.1007/978-0-387-34741-7_15 

12 Jorge Glusberg, “La Argentina entre los países pioneros en arte y cibernética,” Ámbito, February 20, 2007, accessed August 10, 2023, https://www.ambito.com/espectaculos/la-argentina-los-paises-pioneros-arte-y-cibernetica-n3419280

13 Mari Carmen Ramírez, “A paradoxical notion of image: Luis Fernando Benedit’s ́Living Systems ́,” in Benedit: works 1968- 1978, ed. María Torres, Buenos Aires, Fundación Espigas, 2020, 48.

14 Mari Carmen Ramírez, “A paradoxical notion of image: Luis Fernando Benedit’s ́Living Systems ́,” 45.

15 Niklas Luhmann, Social systems, Stanford: Stanford University Press, 1995.

16 Jorge Glusberg, “Artes y espectáculos: definiciones y cuestionamientos,” Primera Plana 10, 1972, 50–51.

17 Jorge Glusberg, Hacia un perfil del arte latinoamericano. Encuentro Internacional de Arte en Pamplona, España. CAYC, Buenos Aires, CAyC Centro de Arte y Comunicación, 1972.

18 Jorge Glusberg, ed., Art systems in Latin America: Institute of Contemporary Arts, London: Center of Art and Communication, Buenos Aires, London and Buenos Aires, Institute of Contemporary arts and CAyC Centro de Arte y Comunicación, 1974, ⁷. 

19 Simón Marchán Fiz, Del arte objetual al arte de concepto, Madrid, Akal, 2012.

20 Jorge Glusberg, ed., Art systems in Latin America: Institute of Contemporary Arts, London: Center of Art and Communication, Buenos Aires, ⁷.

21 Florencia Battini, coord., Víctor Grippo: una retrospectiva. Obras 1971-2001, Buenos Aires: Museo de Arte Latinoamericano de Buenos Aires, 2004.

22 Joanna Page, Decolonizing science in Latin American Art, London, UCL Press, 2021, 113.

23 Justo Pastor, (2004) “La novela chilena de Víctor Grippo”, in: Víctor Grippo: una retrospectiva obras 1971-2001, Buenos Aires, Museo de Arte Latinoamericano de Buenos Aires, 2004, 63- 69.

24 Joanna Page, Decolonizing science in Latin American Art, 165.

25 Jorge Glusberg, ed., Arte de sistemas en la III Bienal de arte Coltejer. Medellin Colombia, Buenos Aires, CAyC Centro de Arte y Comunicación, 1972.

26 Ionit Behar, “Arte de sistemas: conceptual art and politics in Argentina (2015)”, Henrique Faria website, accessed July 29, 2021, http://www.henriquefaria.com/content/uploads/news-press/pdf/artslant_-_arte_de_sistemas_conceptual_art_and_politics_in_argentina-843.pdf

27 Madeline Weisburg, “Finding a techno-utopia: arte y cibernética,” in Exhibitionary Histories of Postwar Latin American Art. Vistas: Critical Approaches to Modern and Contemporary Latin American Art #1, ed. N. Morgan, New York: Institute for Studies on Latin American Art (ISLAA), 2018, p.10-19.

28 Alonso Rodrigo, “El idioma analítico de Horacio Zabala”. El Gran Otro, accessed March 12, 2017, http://elgranotro.com/el-idioma-analitico-de-horacio-zabala/

29 Julieta González, “Beyond technology: Juan Downey’s whole earth,” Afterall: A Journal of Art, Context and Enquiry 37, 2014, 14-27.

30 Jack Burnham, “Art and technology: the panacea that failed,” in The Myths of Information: Technology and Postindustrial Culture, ed. Katheleen Woodward, London, Routledge, Kegan Paul, 1980, 213. 

Towards a Methodology for Co-creating Artistic Acoustic Ecologies with the Great Lakes

David McFarlane

Toronto Metropolitan University, The Creative School, Media & Design Innovation Toronto, Canada
david.mcfarlane@toronto.ca

Abstract

This paper will discuss the early stages of the development of a methodology for co-creating artistic acoustic ecologies with the Great Lakes. It explores some initial philosophical, technological, creative/musical, and ethical concerns involved in my PhD research-creation project entitled: Sounds Like Water, To Me.

In the paper I ask how we might come to understand these bodies of water as animated actors in their own rights, with their own unique subjectivities? By doing so I hope to facilitate a greater understanding of human impact on, relationships to, and responsibilities toward the lakes and all other waters. I will discuss the philosophical framework around perception and being and then explore some technical, artistic and ethical thinking towards a methodology and investigate which tools, techniques, and protocols can be developed or adapted to achieve these goals. The article also puts forward some early experiments, guided by the literature review, and future areas of research.

My research-creation project draws on my experiences as a composer and visual artist, to employ sonic data gathering techniques such as biodata sonification and field recordings, alongside embodied and Indigenous-informed research methodologies in order to undertake a co-created artistic acoustic ecology with the Great Lakes.

Keywords

Research-creation; eco art; phenomenological analysis; participatory co-creation; conceptual art; acoustic ecology; generative art; biodata sonification; non-colonial research methods, symbiosis.

DOI

10.69564/ISEA2023-72-full-McFarlane-Artistic-Acoustic-Ecologies

Introduction

The Great Lakes hold 21% or more than 1/5th of the world’s freshwater, 43 million humans depend on the lakes for drinking water. Despite this, human-generated water contamination, commodification, scarcity, all combine to present increased threats to the survival of human and non-human species. Water access, extraction, governance and sovereignty represent points of contention between Indigenous First Nations, and the settler governments of the United States and Canada.¹ 

The lakes have witnessed and experienced centuries of geographical, geo-political, environmental, industrial and colonial activities and events. As a kind of “colonial super-highway,” they are the very means by which much of the colonizing activities in North America could thrive.

For some time now I have imagined a dialogue with and between the five Great Lakes and what humans might learn from these exchanges. What would they tell us about the exploitation and colonization they have witnessed and experienced and what deeper understanding and empathy might we gain?

I had considered this idea as the premise for a work of narrative fiction, whereby characters would be modeled after histories and traits of each of the lakes, each character knowing they would inevitably be coming together in confluence with the other four. Attempts at personifying the natural world are not new. Western poets have long used the Pathetic Fallacy to attribute emotions to natural phenomena and objects. It is common to perceive something like a mood originating from a body of water. It stands to reason that those who live on the shores of the Great Lakes may be well suited to describe these emotional characteristics. But this Western tradition of describing, representing and being separate from, falls short of my objectives of developing communication and collaboration between human and non-human actors towards creating more sustainable relationships with the natural world.

I have come to shift my initial curiosity from creating a representation of the lakes to one of co-creating experiences with them through musical collaboration. Acoustic ecology is the study of the interrelationship between sound, nature and society.²  It affords a space to explore this symbiotic nexus through the co-creation of sound and music—artistic acoustic ecologies.

I am a white settler of European ancestry, with an educational background in Western arts and music. I am interested in building different relationships with nature and I believe a necessary step is the development and use of decolonizing methodologies.

I situate my art practice and research-creation project on land within the Dish with One Spoon Territory and as a newcomer I have been invited into this treaty in the spirit of peace, friendship and respect, and within the context of human-induced climate change. The Dish with One Spoon wampum discusses a shared responsibility to take care of the land—including the waters and their ecosystems.³  I see care for water as a reciprocal responsibility I can take on through my art practice.

In Making Love with the Land, Canadian First Nations artist Joshua Whitehead discusses the sentience of the land and water: “The land is an archive, is a library, is a genealogy—a body of land is a body of literature. Water remembers, it maintains memories, it recalls the substances it has previously dissolved; trees remember, and in their wounds is a witnessing of wars past, diseases eradicated. If the land can witness it too, it can listen. And it talks through what we might call living stories...”

In order to hear these stories, and to prepare for my own “conversations” with the lakes, I am researching technological, creative and ethical frameworks for engaging non-human actors. This has led me to key research in these respective areas such as Tim Ingold’s concept of “wayfinding,”⁵ Donna Harraway’s Situated Knowledges,⁶ Natalie Loveless’ contributions to the field of research creation,⁷ Michel Serres’ considerations on noise,⁸ the soundscape composition practices of Hildegaard Westerkamp² and R. Murray Schafer,⁹ among others. I also interviewed currently active experts related to the fields of music composition, production and biodata sonification for this paper. They included Dr. Scott McLaughlin,¹⁰ and Sam Cusimano.¹¹

Representation, Perception and Being

There has been marked evolution in western art moving from objective representation to today’s embodied notions. This shift was very much at the core of the modern visual arts movement of the early 20th century. Representation gave way to abstraction as in Cubism and the whole realm of isms that followed. These movements moved away from the figure being the focal point, collapsing it into the background and eventually abstraction.

Multiple shifts were taking place to try to awaken in the public a dialog about life and the paint stroke as a means to express instead of represent our worlds. But their approach was rooted in colonialist humanist culture. This meant that these concepts remained fixed in representation of a subject, neither embodied nor alive.

Rebecca Belmore’s site-specific sculptural installation, Wave Sound, from 2017, is a powerful example of how a work can invite embodied experiences. Wave Sound is a set of four site-specific sculptural installations, one of which is large cone cast from the rock surface on the north shore of Lake Superior. The cone faces out to the vast lake and quietly invites the viewer to listen to the land by lying down on the rock such that one can place an ear over the small opening at the small end of the cone, in hopes of hearing something beautiful. The listener enters into a sympoietic relationship with the situation the piece creates, physically as well as intellectually. The ‘artwork’ is a means not an end.¹²

Noise

What is noise, what is music, what is sound, what is silence and how does our situatedness inform our perception of this? These questions have pervaded contemporary composition and inspired composers to employ processes with which to move away from mimesis, the representation or imitation of the real world in the arts.¹³ Similar fundamental queries have surfaced in other areas of creative practice such as the Visual Arts.

Michel Serres’ story, Noise, articulates concepts in both visual art and sound that specifically identifies an interesting challenge of describing the threshold of perception: “The agitation is everywhere to be heard, beside the signals, beside the silence. The silent sea is misnamed. Perhaps white noise [bruit de fond] is at the heart [fond] of being itself. Perhaps being is not at rest, perhaps it is not in motion, perhaps it is agitated. White noise never stops, it is limitless, continuous, perpetual, unchangeable. It has no grounding [fond] itself, no opposite. How much noise has to be made to still the noise? And what fury orders fury? Noise is not a phenomenon, all phenomena separate from it, figures on a ground [fond], as a light in the fog, as any message, cry, call, signal must each separate from the hubbub that fills the silence, just to be, to be perceived, sensed, known, exchanged. As soon as there is a phenomenon, it leaves noise, as soon as an appearance arises, it does so by masking the noise. Thus it is not phenomenology but being itself.” ⁸  Serres acknowledges the body’s role in perception through this figure/ground metaphor but he takes it further.

The situation Serres sets up is a space for a deeper exploration of aesthetics, perception, and experience and to relate broadly across the arts. In poetic fashion he engages us to measure the infinite gradation between figure and ground in our own minds, in the visual arts, in the mythological soundscape of the sea.

As Serres gradually evolves this symbiotic figure ground relationship, he also overlays these ideas onto the evolution of modern art through his referencing of the 3 painters in Balzac’s The Unknown Masterpiece, a story that was highly influential to Post-Impressionist artists—namely Cezanne. The Balzac piece recounts an era where “... the categories of beauty, verisimilitude and mimesis were disappearing, where questions about the relationship between artwork and thought were coming to the fore, becoming more and more central in relation to the technical limits of painting.”¹⁴

As Serres weaves in his thoughts on representation, perception, and being, he provocatively asks: “Who has ever seen a meeting between the real and the symbolic in the story?

In tandem with phenomenology, Serres questions the symbolisms of storytelling. He blurs our efforts as a reader to keep track of the narrator’s position and challenges our perception of what is true, what isn’t and what is intended, as if to emerge and recede from the noise of his own story.

His complex presentation of this multi-sensorial challenge is in itself a layering of historical figures and ground relationships. His somewhat Borgesian strategy, which weaves the authors’ identity, role and tone in and out of characters, past, present, fact and fiction, artfully fogs our lens.

The figure is part of the noise. It emerges sometimes briefly, and you can recognize it, some of us better than others, Serres notes, but it always recedes back into the noise, and the noise is always there. The noise is, to me, this bed of just being, always present.

The Body Electric

While Serres kept to the world of ideas, moving away from representation also gained popularity in music. The determination of figure and ground, noise and sound have major implications in several fields of contemporary cultural practice. When noise becomes sound and a sound becomes musicalized as a note, is a focus of rich investigation in avant-garde and electronic music.

Brian Eno’s Ambient 1: Music for Airports, is music designed to blend with an existing soundscape and blur the boundary of where his participation as the composer begins and ends.¹⁵ 

Eno was not the first to make what we now identify as ambient music, but he was the first to name it. Erik Satie completed Gymnopédie No. 1 in 1888—110 years before Eno’s time. “Furniture music” was the way Satie described the role of the music in the context of its environment—part of, inclusive of the sounds in it, not the main focus.¹⁵ This idea later became a major area of interest for John Cage, who in 1952 released 4’33, which intended to capture the ambient noises inherent in a given performance space.¹⁶

Mechanical processes and technologies have also played a significant role in altering how we perceive sound. In 1948, Pierre Schaeffer had coined the term musique concrète and had been experimenting with tape loops and "music systems." The early tape phasing experiments of Steve Reich certainly present a process for coaxing musical ideas from noise, figure from ground. His tape pieces introduce what Eno would later call Generative. Eno describes the experience of listening to Reich’s “It’s Gonna Rain”, as revelatory in how the piece allows for open and asynchronous music events to happen as a seminal influence in his future work.¹⁷

The tradition of taking hypothetical technical and theoretical positions to arrive at novel and surprising results, beginning with Satie, has become well established in avant-garde and electronic music. Alongside the adoption of these more cerebral and conceptual strategies, innovation leading to ‘uncode’ its creation has also emerged as a pervading trend in electronic music.

Many early electronic instruments were designed to diminish the need to play an instrument or have knowledge of musical theory. The body became centered as the primary inter-actor with the instrument. This can be argued as entering the world of relational aesthetics, a movement that introduces the symbiosis of relationships as a central occupation in western art practices.

For example, the Theremin, sometimes derided as a novelty item, removes a layer of interface to play it. Also reduced is the codified knowledge one requires to play, say a keyboard instrument, which is a heavily codified, Western mechanism. The Theremin relies on the body in space and its relationship to the instrument's two (usually) antennae. The Theremin player interacts less intellectually and in a more physical way, as the direct communication method. The music comes, closer than other codified interfaces, from the sympoiesis between the performer and the instrument. Their relationships alter the quality of the air in their environment, this noise is then captured by the machine and translated into sounds.

Theremin playing enjoyed perhaps its most elevated period in the hands of Clara Rockmore, a virtuoso violinist who adopted the Theremin as her main instrument and challenged the notion that it was a substandard instrument. Of her Theremin technique she has remarked: “You can’t play air with hammers, you have to play with butterfly wings.” [18] Her description clearly emphasizes the physicality of the activity.

In the 1950s Daphne Oram, British composer and early electronic music pioneer, was the first woman to set up an independent music studio and design and construct an electronic instrument. She also created Oramics, which is a way to generate electronic sounds from drawn wave forms. Tape-based and electronic instruments introduce a move away from orchestral instruments towards synthesized sounds which don’t require the skills associated with classical instruments.¹⁸ 

Another compelling expression of the freedom perceived by musicians as a result of this ‘uncoding’ comes from Suzanne Ciani who says: “In electronics, you’re not dealing so literally with the architecture of nodes or harmonics, those building blocks of classical music, you’re dealing in energy.” Describing a performance from 1974 in which she is performing a composition on an early Buchla synthesizer she states: “I think they are so sensual. The machine was alive. It was warm. It communicated. It was sensitive. You could move something just the littlest bit and then a whole new expression would open up.” Again, the embodied, participatory and more experiential aspects of playing the instrument come to the fore.¹⁸ 

Mileece has pushed the boundaries of relational aesthetics into the realm of ethics with non-human beings. Her work I Oracle: Dreaming in the Future Ancient, questions humans’ capacity to exist in systems in holistic ways. Like Natalie Loveless, who moves art practices into embodiment of ecological ethics, Mileece is embedding the symbiotic relationships of her own body to plants in a data language that develops within ethical concerns. She tries to create a new language through AI and code, getting plants to direct AI. ¹⁹ 

Is it possible to un-code humanity’s relationships to nature? Are the generative processes inherent in algorithmic based music a way forward? Is letting go of compositional control through the use of open algorithms a way to uncode? The cycles of the planet are like an open algorithm, in that the seasons repeat based on specific conditions and symbiotic relationships of ecosystems inform the outcomes.

The technological aspects of my methodology are influenced and in line with questioning and the practices I’ve mentioned here. Moving forward, I will be exploring ethical concerns and how to fold these into a methodology. I am particularly interested in the discussions about ways of making kin with technology, discussions shared by Donna Haraway and Jason Edward Lewis’ writing about Indigenous futurisms.⁶, ²⁰ How might I incorporate this into my work?

Haraway, in her essay, Situated Knowledges, challenges the long defended position of the objectivity of Western science, the “god trick” as it is colloquially known. For Haraway the observer can never be extracted from a situation, pointing out that science is different when different individuals are doing it. She centers this idea that perspective matters and that it is important to acknowledge that we bring our situated perspectives with us as observers.²¹

Sound Methods

These ideas change creative methodologies. How can I establish a methodology that focuses on: visiting with, harmonizing with, listening with, and co-creating with, during my fieldwork. I am moving away from mimesis, (or representing the lakes in songs, sounds, or images), towards methexis (participating in what the lakes are doing).⁸ 

UK based composer, Scott McLaughlin, offers one approach. He has incorporated Tim Ingold’s idea of wayfaring in his creative methodology. Ingold likens the wayfaring process metaphorically to following trails through a landscape, finding new trails that will take you further. This is in contrast to a process of navigation Ingold calls this method “transport”, where one plots a direct path from A to B, in hopes of traveling as fast as possible, where any obstacle in between is simply in your way.¹⁰ McLaughlin notes that, colloquially, wayfaring is more about the journey and the knowledge accumulated and carried than the destination.⁵ In this way McLaughlin creates scores where, rather than the performer’s job is to bend the instrument to their will, the performer activates and supports the material and design characteristics of the instrument. McLaughlin crafts situations where performers participate with what he calls the “material agency” of an instrument.¹⁰

Artistic Methods

Acoustic Ecology

Interest in acoustic ecology began as a form of ‘acoustic activism’ on the part of like-minded faculty at Simon Fraser University in the late 1960s and early 1970s.

Figure 2. Still from Resonate Body #2 (Author) 

Of practicing acoustic ecology, R. Murray Schafer states that: “Ecology is the study of the relationship between living organisms and their environment. Acoustic ecology is therefore the study of sounds in relationship to life and society. This cannot be accomplished by remaining in the laboratory. It can only be accomplished by considering on location the effects of the acoustic environment on the creatures living in it.” ⁹ Led by Schafer, the group gained early recognition through the creation of Soundscapes of Canada, a 10-part series of one-hour radio programs based on the sounds of Canadian acoustic environments—essentially a recorded “sound-walk” across the country and was first presented on CBC-FM "Ideas", in 1974.

Central to soundscape composition is the concept of conscious listening. As illustrated by a quote from Hildegard Westerkamp, one of the group’s founding composers/researchers explains: “To compose with environmental sound implies a relationship—a dialogue—between soundscape and listener in daily life. No matter what the composer’s intent may have been from the start, the materials inevitably speak with their own language, whose deeper meanings may only emerge with repeated listening and sound production. And that in itself has the power to shift the composer’s intent.”²

Graphic Notation

Figure 3. Graphic notation example (Iannis Xenakis)

I will use graphic notation while visiting various sites around each lake as a way to “poetically inquire” and capture immediate impressions of each place and space, from a sonic perspective.²² These will aid my work later as I begin the musical collaborations with the field recordings gathered on each respective site. The graphic notation will also serve as a potential score which I may choose to share with other musicians or composers to interpret. The visual notations will act as perceptual barometers, diaristic in nature, of a time and place from which I will be able to reflect on my own perceptual shift as my empathy for the lakes evolves. Graphic notation is an alternative access point to musical composition and dissemination—operating somewhat outside of Western conservatory traditions, which are limited in their ability to relay emotional content and remain somewhat elitist in terms of who is able to access the education required to become literate. Sight reading traditional western musical notation, for example, requires years of tutoring and practice.

Technical Methods

Biodata Sonification

Data is a sea of noise and humans have oceans of it. How can we make sense of it? How can we feel its significance? How can we better become kin with the tools and technology to better understand it?²¹, ²⁰

This past year I have been conducting tests and experiments with a biodata sonification device designed and built by Sam Cusimano. Cusimano’s device reads micro-changes in conductivity within organic matter in real time. An Arduino inside the device reads those changes in values over time and expresses the fluctuations as MIDI notes and the degree of change in conductivity is correlated to pitch such that higher degree of change gets expressed as a higher pitch. The device also has the ability to assign various types of scales such as: the 12 tone chromatic scale, pentatonic and minor scales. The device is equipped with a MIDI output as well as wireless bluetooth output. The device is able to send data to a MIDI recorder as well, such that the MIDI notes can be played on an instrument at a later time.

There are many similar devices available that provide the same functionality but I chose Cusimano’s as he was repeatedly referenced by several artists working with biodata sonification and he was also very knowledgeable and accessible.

While visiting sites around the lakes I intend to use this device to make recordings of the data I encounter, as part of a larger ensemble of the other elements I’ve mentioned, which I will combine later.

It should be noted that this realm of sonification is emergent and thus the shared knowledge is evolving, limited, and to a degree, and more often disseminated through non-academic channels.

One of the key challenges facing this technology is that the multitude of MIDI capable instruments are able to reproduce just about any sound imaginable—sample or synthesized. How then does one decide which sound to select for a given organism to use? Assigning or mapping any voice/instrument thus becomes arbitrary and potentially misleading. There is not a way to get around the translation of the MIDI impulse into a sound. One could argue that this is also the case when a human plays an instrument as well, the main difference being the human has agency in selecting the instrument, the scale, notes, key signature, or whether to play at all.

Unlike more traditional uses of data sonification, where data is sonified such that it becomes easier to interpret or analyze, my hope is to see if we can perceive something like an emotional feeling from the biodata generated sounds, which notes are played and how they are played. MIDI can carry parameters such as: volume, pitch, attack, decay, timbre, Surely, there is still enough going on with those parameters to generate some vibe? But how do I achieve this without subjecting the data to my own aesthetic priorities? In his brief article, No Mapping (2016), James Saunders suggests that when this arbitrary mapping is presented as a truth, that the listener is hearing the actual true sound of a phenomenon like a solar flare or climate change, the arbitrariness compromises both the data and the sound.²³

Imparting “humanisms” onto other living things to convey complex emotions also raises concerns associated with Interpellation.²⁴ In the field of Visual Studies, it’s a concept coined by Louis Althusser, that describes the way media can make you feel like you identify with what’s being presented to you and belong to a value system—or even an ideology.²⁵, ²⁶ Moving forward, this is an important issue to consider. Embedded in the act of interpellation are culturally specific values and norms that are not universal and assuming so may risk alienating potential audiences.

Conclusions

Several recommended methods for addressing the concerns raised here have emerged from the research. Regarding the assignment of an abstract tone or sound to the MIDI notes being generated, one should consider using neutral sounds and avoid effects that may impart additional subjective information (like reverb for example).²⁷ Another recommendation is to ensure that the listener can understand what is triggering the sound being made and that the mechanism is as exposed as possible.²⁷ Another thought is to create an installation environment where listeners can hear a chorus of the data sonification events happening simultaneously, such that the abstractness of the voicing assignments becomes less important than how the the voices interact together.¹⁰ Lastly, an interface could be created such that the listener could select and assign voices themselves, revealing the abstractness of the mechanism of selection and potentially deepening their engagement with the listening experience.

Figure 3. Still from Resonant Body #? (Author) 

I have recently begun experimenting with recording the MIDI generated sounds in the soundscape they are occurring as well as recording the audio of the same soundscape. In this way, I am able to listen to both simultaneously and aligned in real time. In this way, the connection between different types of recorded sounds (data generated and audio) in a given space are more strongly associated.

A key area of future focus is to explore an ethical methodology. How am I impacting those I am being with in my actions? How might I gain consent from the non-human actors?

This article does not present a comprehensive or complete methodology but rather a first step towards one. There is still the area of ethics to more deeply research and consider. I will be interviewing Diane Whalen and Bonnie Devine to discuss being on the land and visiting methodology from settler and indigenous perspectives respectively. I will also be further evaluating my situatedness, particularly in consideration of Dylan Robinson’s concerns brought forward in Hungry Listening, as to “...how your positionality guides the way you listen to musical subjectivity.”²⁸ I will also begin researching the many treaties and covenants that describe the agreements I may need to be aware of as I visit various sites. I am going to explore incorporating ceremony into my activities on the land and water and will be researching ethical issues around this. I know well that there is much to learn in the field as well and that whatever I do in preparation needs to incorporate nimbleness, adaptability and openness to what unfolds.

Acknowledgements

The author would like to acknowledge the tremendous support of Dr. Alexandra Bal, Dr. David Gauntlett, Dr, Scott McLaughlin, Jody Colero, Samuel Cusimano, Daniel Lanios and Andrew Lockhead.

References

1 FLOW, “Privatization and Commodification", Accessed December 7, 2022, https://forloveofwater.org/issues/privatization/.

2 Hildegard Westerkamp, “Linking Soundscape Composition and Acoustic Ecology”, Organised Sound 7, no.1, April 2002, 51–56. https://doi.org/10.1017/S1355771802001085.

3 Ange Loft, Denise Bolduc, Mnawaate Gordon-Corbiere, Rebeka Tabobondung, Brian Wright-McLeod, Rebeka Tabobondung, Brian Wright-McLeod, Brian Wright-McLeod, Indigenous Toronto: Stories That Carry This Place, Toronto, CANADA, Coach House Books, 2021. http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=6210057.

4 Joshua Whitehead, Making Love with the Land, Toronto, Knopf Canada, 2022.
Ingold, Tim. Being Alive: Essays on Movement, Knowledge and Description. London ; New York, Routledge, 2011.
Jason Lewis, Noelani Arista, Archer Pechawis, Suzanne Kite, “Making Kin with the Machines”, Journal of Design and Science, July 16, 2018. https://doi.org/10.21428/bfafd97b.

5 Donna Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective”, Feminist Studies 14, no. 3, 1988, 575–99, https://doi.org/10.2307/3178066.

6 Natalie Loveless, How to Make Art at the End of the World: A Manifesto for Research-Creation, Durham, Duke University Press, 2019. https://muse.jhu.edu/book/70998.

7 Michel Serres, Lawrence R. Schehr, “Noise.” SubStance 12, no. 3, 1983, 48–4, https://doi.org/10.2307/3684255.

8 R. Murray. Schafer, The Tuning of the World. First edition, Toronto, McClelland and Stewart, 1977. 

9 Scott McLaughlin, “Lutins”, Accessed November 1, 2022, http://lutins.co.uk/index.html.

10 Sam Cusimano, “Electricity for Progress”, Electricity for Progress, Accessed December 9, 2022, https://electricityforprogress.com/.

11 Lindsay Nixon, “Rebecca Belmore Wants Us to Listen to the Land”, Canadian Art, Accessed December 8, 2022, https://canadianart.ca/interviews/rebecca-belmore-landmarks-2017/.

12 Jeannette Marie Mageo, “Mimesis Makes Metaphors”, In The Mimetic Nature of Dream Mentation: American Selves in Re-Formation, edited by Jeannette Marie Mageo, Mind Culture, and Society, Cham: Springer International Publishing, 2022, 115–40, https://doi.org/10.1007/978-3-030-90231-5_5.

13 Antonio Carnevale, “Honoré de Balzac’s Frenhofer and His Unknown Masterpiece | CFA.” Conceptual Fine Arts (blog), April 27, 2020, https://www.conceptualfinearts.com/cfa/2020/04/27/balzac-unknown-masterpiece/.

14 John T. Lysaker, Brian Eno’s Ambient 1: Music for Airports. Oxford, UNITED STATES, Oxford University Press, Incorporated, 2018, http://ebookcentral.proquest.com/lib/ryerson/detail.action?docI D=5582502.

15 John Cage, Silence: Lectures and Writings, 1st M.I.T, Press paperback ed. Cambridge, Mass, MIT Press, 1966.

16 Steve Reich, Conversations. Original édition. Hanover Square Press, 2022.

17 Sisters with Transistors, Documentary, Music, Willow Glen Films, Anna Lena Films, 2021.

18 Mileece, “IOracle”, Mileece, Accessed December 8, 2022, https://www.mileece.is/ioracle.

19 Jason Lewis, Noelani Arista, Archer Pechawis, Suzanne Kite, “Making Kin with the Machines”, Journal of Design and Science, July 16, 2018, https://doi.org/10.21428/bfafd97b.

20 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Albany, UNITED
STATES, Duke University Press, 2016, http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=4649739.

21 Sheila Stewart, “Poetry: The Edge of Knowing”, Creative Approaches to Research 5, no. 2, 2012, 105–18.

22 James Saunders, “No Mapping.” James Saunders (blog), May 7, 2016, https://www.james-saunders.com/2016/05/07/no-mapping/

23 Marita Sturken, Lisa Cartwright, Practices of Looking: An Introduction to Visual Culture, Third edition, New York, Oxford University Press, 2018.

24 Heather Dorries, Sue Ruddick, “Between Concept and Context: Reading Gilles Deleuze and Leanne Simpson in Their in/Commensurabilities”, Cultural Geographies 25, no. 4, October 1, 2018, 619–35, https://doi.org/10.1177/1474474018778576. 

25 Nicole Gross, Mikko Laamanen, “‘Hey, You There! Marketing!’ On Ideology and (Mis)Interpellation of the Marketing Educator as Subject”, Journal of Marketing Management 0, no. 0, June 30, 2021, 1–24, https://doi.org/10.1080/0267257X.2021.1944896.

26 Sonification & The Problem with Making Music from Data, 2017, https://www.youtube.com/watch?v=Ocq3NeudsVk.

27 Dylan Robinson, Hungry Listening: Resonant Theory for Indigenous Sound Studies. University of Minnesota Press, 2020. https://doi.org/10.5749/j.ctvzpv6bb 

Bibliography

Branch, Government of Canada, Indigenous and Northern Affairs Canada, Communications, “Treaties and Agreements”, Administrative page, November 3, 2008, https://www.rcaanccirnac.gc.ca/eng/1100100028574/1529354437231#chp3.

John Cage, Silence: Lectures and Writings, 1st M.I.T, Press paperback ed. Cambridge, Mass, MIT Press, 1966.

Antonio Carnevale, “Honoré de Balzac’s Frenhofer and His Unknown Masterpiece | CFA”, Conceptual Fine Arts (blog), April 27, 2020, https://www.conceptualfinearts.com/cfa/2020/04/27/balzacunknown-masterpiece/.

Sam Cusimano, “Electricity for Progress.” Electricity for Progress. Accessed December 9, 2022, https://electricityforprogress.com/.

Dorries Heather, Sue Ruddick, “Between Concept and Context: Reading Gilles Deleuze and Leanne Simpson in Their in/Commensurabilities”, Cultural Geographies 25, no. 4, October 1, 2018, 619–35, https://doi.org/10.1177/1474474018778576. 

David Gauntlett, Media, Gender and Identity: An Introduction. Taylor & Francis Group, 2002, http://ebookcentral.proquest.com/lib/ryerson/detail.ac tion?docID=170355.

Nicole Gross, Mikko Laamanen., “‘Hey, You There! Marketing!’ On Ideology and (Mis)Interpellation of the Marketing Educator as Subject”, Journal of Marketing Management 0, no. 0, June 30, 2021, 1–24, https://doi.org/10.1080/0267257X.2021.1944896. 

J. Donna Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective”, Feminist Studies 14, no. 3, 1988, p.575–99, https://doi.org/10.2307/3178066.

J. Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Albany, UNITED STATES, Duke University Press, 2016, http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=4649739.

Tim Ingold, Being Alive: Essays on Movement, Knowledge and Description, London ; New York, Routledge, 2011.

Jason Lewis, Noelani Arista, Archer Pechawis, Suzanne Kite, “Making Kin with the Machines", Journal of Design and Science, July 16, 2018, https://doi.org/10.21428/bfafd97b.

Ange Loft, Denise Bolduc, Mnawaate Gordon-Corbiere, Rebeka Tabobondung, Brian Wright-McLeod, Rebeka Tabobondung, Brian Wright-McLeod, Brian Wright-McLeod, Indigenous Toronto: Stories That Carry This Place, Toronto, CANADA, Coach House Books, 2021, http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=6210057.

Natalie Loveless, How to Make Art at the End of the World: A Manifesto for Research-Creation, Durham, Duke University Press, 2019, https://muse.jhu.edu/book/70998.

John T. Lysaker, Brian Eno’s Ambient 1: Music for Airports. Oxford, UNITED STATES, Oxford University Press, Incorporated, 2018, http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=5582502.

Jeannette Marie Mageo, “Mimesis Makes Metaphors”, In The Mimetic Nature of Dream Mentation: American Selves in Re-Formation, edited by Jeannette Marie Mageo, Culture, Mind, and Society, Cham: Springer International Publishing, 2022, 115–40, https://doi.org/10.1007/978-3-030-90231-5_5.

David McFarlane, “Scott McLaughlin: Wayfaring and Material Animacy”, Accessed November 1, 2022, http://davidmcfarlane.com/PhD/Scott_McLaughlin_rough_mix.MP3

Scott McLaughlin, “Lutins.” Accessed November 1, 2022, http://lutins.co.uk/index.html.

Mileece, “IOracle”, mileece, Accessed December 8, 2022, https://www.mileece.is/ioracle.

Dylan AT. Miner, “Mawadisidiwag Miinawaa Wiidanokiindiwag // They Visit and Work Together", Makers, Crafters, Educators, 2018, 131.

“MusikTexte 149 – Mai 2016", Accessed July 19, 2022, https://musiktexte.de/MusikTexte-149/en.

Lindsay Nixon, “Rebecca Belmore Wants Us to Listen to the Land”, Canadian Art, Accessed December 8, 2022, https://canadianart.ca/interviews/rebecca-belmore-landmarks-2017/.

FLOW, “Privatization and Commodification”, Accessed December 7, 2022, https://forloveofwater.org/issues/privatization/.

Steve Reich, Conversations, Original édition, Hanover Square Press, 2022.

Dylan Robinson, Hungry Listening: Resonant Theory for Indigenous Sound Studies, University of Minnesota Press, 2020, https://doi.org/10.5749/j.ctvzpv6bb.

James Saunders, “No Mapping.” James Saunders (blog), May 7, 2016, https://www.james-saunders.com/2016/05/07/no-mapping/.

R. Murray Schafer, The Tuning of the World, First edition, Toronto, McClelland and Stewart, 1977.

Michel Serres, Lawrence R. Schehr, “Noise.” SubStance 12, no. 3, 1983, 48–4, https://doi.org/10.2307/3684255. 

Sisters with Transistors, Documentary, Music, Willow Glen Films, Anna Lena Films, 2021.

Sonification & The Problem with Making Music from Data, 2017, https://www.youtube.com/watch?v=Ocq3NeudsVk.

Sheila Stewart, “Poetry: The Edge of Knowing.” Creative, Approaches to Research 5, no. 2, 2012, 105–18.

Marita Sturken, Lisa Cartwright, Practices of Looking: An Introduction to Visual Culture, Third edition, New York, Oxford University Press, 2018.

Hildegard Westerkamp, “Linking Soundscape Composition and Acoustic Ecology”, Organised Sound 7, no. 1, April 2002, 51–56, https://doi.org/10.1017/S1355771802001085.

Joshua Whitehead, Making Love with the Land, Toronto, Knopf Canada, 2022. 

Structures of Emotion: Speculating on an AI-Human Symbiosis

Avital Meshi1, Treyden Chiaravalloti2

1University of California Davis, 2Graduate School of Design, Harvard University USA
ameshi@ucdavis.edu.treyden3@gmail.com

Abstract

Structures of Emotion is a performance artwork that explores a symbiotic relationship between humans and an Emotion Recognition Artificial-Intelligence (AI) algorithm. The piece utilizes a wearable computing device designed to enable the wearer to recognize emotions through two different perspectives: their own organic senses and an AI apparatus, which serves as an extension of the body, connecting the human mind to a "collective consciousness." Participants interacted with two performers; one wore the AI device, while the other relied solely on their organic abilities. The performance demonstrates how AI emotion recognition systems are still immature. However, it invites us to speculate on its potential role when it becomes more sophisticated. Additionally, it explores the ethical complexities of our entanglement with emotion recognition algorithms and imagines the danger of becoming dependent on them within a transhumanist future.

Keywords

Performance Art, Artificial Intelligence, Emotion Recognition, Affective Computing, New Media Art, Interactive Media, Transhumanism, Wearable Computing.

DOI

10.69564/ISEA2023-73-full-Meshi-et-al-Structures-of-Emotion

Introduction

In the wake of World War II, when scientists were forced to develop an array of strange destructive gadgets, peacetime was when they had to find a better use for their instruments. Vannevar Bush, a scientist, engineer, and innovator who led the US military's wartime R&D in the 1940s, suggested that the thinking human should have a new relationship with knowledge. He hypothesized a new device called the Memex, which was intended to recollect all scientific knowledge, mimic the associative processes of the human mind, and allow humans to access the collective record in a more useful way. In Bush's view, the Memex was an enlarged intimate supplement to human memory. The device included a tiny camera the size of a walnut attached to the forehead of the human. Bush argued that the wearer of this camera is the scientist of the future, absorbing worthy records through this artificial third eye while moving around the lab or the field. The Memex was designed to enhance the flow of information and knowledge to and from the brain and to make the human one with the machine ¹.


While we cherish our connection to knowledge, the thinking human is also, at least for now, an emotional being. Is technology capable of assisting us in this manner? Can AI technology enhance our ability to recognize, express, and regulate our emotions? The affective computing consumer market is consistently growing.² Major developers predict that soon all of our devices will have an ‘emotion chip’ in them ³. If this comes true, what would it mean for human society? How will it impact our behavior and our relationships with one another? To unpack these questions, we follow up with Marshall McLuhan’s famous statement that “The Medium is the Message” and ask what kind of message affective AI bring with it? For McLuhan artistic exploration can always be relied upon to tell the old culture what is beginning to happen to it; serving as, what he called, a “Distant Early Warning System,” art enables us to discover and prepare for changes in our society.⁴ The artwork we present in this paper may be considered along these lines. 

Structures of Emotion is a series of two interactive, participatory performances that examine a symbiotic relationship between humans and AI algorithms. It utilizes a device that is almost identical to the wearable element of the Memex. Using a tiny camera on the performer's forehead, the device detects, identifies, and communicates the emotions of participants. A second performer, unaided by the AI device, interacts with the same participants and attempts to recognize their emotions. A playful interaction between the performers and participants illustrates how emotion recognition algorithms are still in their infancy. Nevertheless, tensions between humans and machines remain evident. The piece elicits a debate about how algorithms might impact human behavior. In addition, it raises further speculations about our transhumanist future and the dangers of becoming dependent on machines to moderate our emotions, an essential element of human identity.

Figure 1: The wearable device that was used in the performance Structures of Emotion. Image: Kyle Adler.

Implementation Details The Wearable Device

Structures of Emotion uses an ocular-centric AI emotion recognition system. A tiny camera is attached to the wearer's forehead, essentially serving as a third eye. The camera is connected to a Raspberry-Pi microcomputer attached to the wearer's head, an LED display panel attached to the chest, and a handheld Bluetooth speaker. The entire apparatus is battery powered and tetherless so that the wearer can move freely (Figure 1). The microcomputer hosts an off-the-shelf AI emotion recognition model, outsourced from the computer vision GitHub community.⁵ As with other AI emotion recognition models, this model is based on classical theories claiming that emotions can be categorized into a limited number of primary, universal emotions.⁶, ⁷ These include the emotions "Happy," "Sad," "Angry," "Disgusted," "Surprised," "Fearful," and "Neutral." As soon as the device recognizes an emotion, it is displayed on the LED panel and announced with a distinctly computer-generated voice, "You Seem Happy" or "You Seem Angry," depending on the detected emotion. 

Figure 2: Structures of Emotion, street performance. Downtown Santa Cruz, CA. Summer 2021. 

The Performance

The wearable device was used in two types of performances: a street performance and an online performance. In both types, two performers interacted with the audience and recognized participants' emotions via two different methods. One performer wore the AI device, allowing the machine to act as her voice. As the device recognized and announced the participants' emotions, the performer remained emotionally detached and unresponsive. A second performer, unaided by the wearable device, facilitated the encounter while using his own organic senses to recognize and voice participants' emotions as they appear through his eyes. The street performance took place during summer 2021 in downtown Santa Cruz, CA. It lasted about 60 minutes and included participants who seemed curious, available, and comfortable interacting without their face masks (Figure 2). The online performance was held in an online Zoom meeting during fall 2021. Participants volunteered to join the meeting, so the interactions with them were more deliberate. Each participant was observed individually for 15-20 minutes, allowing both performers to attempt to recognize and announce the participant’s emotions (Figure 3). 

Figure 3: Structures of Emotion, online Zoom performance. Fall 2021. The left panel shows a participant who allows the performers to observe his face and attempt to identify his emotions. The upper right panel shows the performer wearing the AI device. The lower right panel shows the performer who detects the participants’ emotions without using the wearable AI. 

Audience Reception Trying to Beat the Machine

Many participants attempted to manipulate the way the device recognized their emotions. They consciously altered their facial expression or, in one participant's words, tried to "channel different emotions." Through this playful interaction, participants felt motivated to score each of the device's possible outputs. A similar engagement is prompted by the artwork 'Emojify' in which participants are encouraged to fake their emotions: "Can you make the emotion recognition system read you as happy, sad, or angry? Can you beat the machine?"⁸ Performing one's emotional expressions is one of the most persistent critiques of the classical emotion recognition theory. Our ability to swiftly change our facial expressions demonstrates how they can be disconnected from our actual emotional state.⁹ It is therefore essential to ask what exactly the AI device detects; Is it recognizing participants' emotions or merely observing facial expressions? Similarly, we can ask whether a human is capable of acknowledging another person's emotions. Can we see each other's expressions and know what emotions they convey?

Participants' tendency to alter their behavior in the presence of an AI algorithm illustrates how recognition systems can retrain human behavior. We are already conditioned to perform emotions when we are with others because our society normalizes some emotions and devalues others.¹⁰ By incorporating AI systems into our environment, we are now also required to change our behavior in the presence of algorithms.¹¹, ¹², ¹³ The automatic recognition of emotions is likely to comply with society's demand that anyone who appears angry or upset will be discriminated against. Instead of understanding the reasons behind such strong emotions, AI systems will assist in excluding or repressing those who express them. This idea is well illustrated in the Japanese cyberpunk anime PSYCHO-PASS (2012), where an advanced surveillance system constantly monitors citizens' mental states to predict their criminality potential. Expressing negative emotions increases an individual's crime coefficient index, and when this index exceeds a certain threshold, the individual gets arrested or killed. In this sense, we agree with artist and philosopher Hito Steyerl, who claims that incorporating AI systems into conflicts that we can still not resolve on our own ignores the complexities of singular human experiences and the emotions they evoke. Human conflict, she claims, is often needed, ignoring it only weaponizes interactions and leads to further polarization.¹⁴ Before further establishing our relationships with AI emotion recognition systems, we must ask ourselves if we are willing to fake a smile to pass as good members of our society. This idea recalls Yue Minjun's paintings of cloned humans with exaggerated grins on their faces. While smiley faces are easily considered "happy," they can also be viewed as mandatory masks worn by members of a society that prioritizes appearance over genuineness.¹⁵

Entering Affective Loops

Most participants smiled as they became engaged with the performance. In response the device repeatedly announced: "You seem happy.” This resulted in more smiles and laughter on the participants' behalf. One of the participants who kept being detected as "Happy" affirmed the device's announcements and stated: "I am radiating happiness right now." Per societal values, participants seemed satisfied to be detected as "Happy." On the other hand, those classified as "Angry" or "Sad" seemed annoyed by these results, stating that they did not feel sad or angry and questioned the device's accuracy. The responses can be explained within the 'Affective Loop' framework; Participants express a particular emotion through a physical manifestation, and the system responds according to whatever parameters it was trained on. In return, participants react back in an even more involved manner.¹⁶ Participants were quick to enter these affective loops with the machine. Yet, as the performance progressed, we noticed they were as quick to lose interest in the device's repetitive, limited, and laconic responses. Instead, they grew more interested in what the performer, unaided by the AI, had to say about their emotions.

The performer who evaluated participants' emotions without the aid of the AI device felt transformed by the experience. Although it may be seen as a mundane task that we regularly perform when interacting with one another, attempting to be fully attentive to someone standing in front of us is not trivial. In the performer's account, emotions could not be accurately expressed in words. Participants' ephemeral facial expressions and other fleeting manifestations of their body language required a compassionate presence that evoked care, simply listening with the body and only speaking to express apparent emotions. During the interaction between participants and this performer, both sides appeared to be involved in a deeper affective loop. In the same way as the participants, the performer responded to the situation and felt the need to generate feelings he hoped to describe within himself. As a result, he became increasingly involved and realized that seeing does not suffice to understand another's feelings.

Comparing Human and Machine Abilities

Participants standing in front of the performer who was wearing the AI device stared directly into her eyes, almost as if they assumed that she was the one recognizing their emotions. They rarely seemed to look at the camera on her forehead. At the same time, they did not attempt to speak with her during the performance. Perhaps it was her detached mode of communication that made her seem almost nonhuman and robotic. But as described above, as much as they were fascinated with ‘her’ responses at the beginning of the interaction, they quickly lost interest in the mechanic repossess she delivered through the AI device. For a machine to be successful as a human in an emotional interaction, it needs to provide an illusion that it is authentically and socially engaged over an extended period of time. Such abilities are known to be challenging in developing social robots that are designed to be perceived as socially intelligent,¹⁷ able to deliver successful shared emotional experiences.¹⁸ Participants in Structures of Emotion could compare the humans’ versus the machine’s abilities to recognize emotions. It was evident from the performances that the AI device is still premature. Yet the artwork evoked questions regarding our understanding of our own emotions, provoked thought regarding our relationship with AI algorithms, and triggered a transhumanist imagination in which these algorithms are interfaced with the human body, and we become entirely dependent on them.

Discussion

Structures of Emotion and its interactive performances explored a synergistic relationship between humans and machines. We sought to enhance human emotional capacity by directly augmenting it with an artificial one. This was afforded by attaching the artificial intelligence to the body, providing the wearer an amplified vision that originates with what can be called a "collective consciousness" potentially serving as an extension of the mind. With the Memex in mind, designed to shift "thinking" into a "thinking-with" ideal, we aimed to examine a similar standard with emotions. Can humans and machines form a "feeling-with" relationship? What would it mean for us to sense the world and mediate our emotions in symbiosis with machines?

What Do We Really Know about Emotions?

Before interfacing our emotions with algorithms, we must question our abilities to understand emotions. According to the classical theory described above, emotions are universal and can be classified into primary categories based on the distinctive movements of facial muscles. Despite this, there is still a debate on what we know or do not know about the function and purpose of emotions. Among the critiques of classical theory is the claim that the experiments used to show that emotions are universal were too simplified, disregarding the ever-changing context within which emotions are expressed.¹⁹ Some scholars claim that we should not ignore cultural factors that impact emotional expressions.²⁰ And other, more recent voices, state that emotions are merely guesses made by our brains. Thus, facial expressions cannot accurately indicate a person's mental state ²¹.

While the controversy over human emotions has yet to be resolved, governments and tech companies are actively developing and incorporating AI algorithms to recognize human emotions. Algorithms of this type are already used in national security systems,²² education platforms,²³, ²⁴ hiring startups,²⁵, ²⁶ and police programs.²⁷ According to Rana el Kaliouby, cofounder and CEO of "Affectiva," the first company to market "emotion AI," when humans recognize emotions, they are often incorrect.²⁸ Since we do not yet have a good definition of emotions, this statement may be true. But what does it mean about the algorithms we design to understand and regulate emotions for us? Do they know any better? The primary goal of these algorithms is to optimize efficiency and productivity and compensate or ultimately replace a 'fallible human'. Similarly, many other techno-fixes hold grandiose promises to help us but end up enhancing our problems while simultaneously providing us with an excuse to ignore them.²⁹ Even though emotion recognition algorithms have immense market value, we must ask ourselves what is at stake when we use them prematurely.³⁰

In Structures of Emotion, the design of the wearable device aims to expand human consciousness into a collective one. If AI models are trained on large enough datasets, they can arguably be considered to symbolize a "collective consciousness." Nevertheless, it is essential to ask what exactly goes into this consciousness. The model we used here was trained on a dataset that contained about 28,000 images of faces labeled according to their apparent emotional expressions and categorized by the classical theory's primary categories. It is unclear how these images were collected, who are the people seen in them, and in what context they are expressing their emotions. It becomes apparent when we examine the dataset that the 'collective consciousness' is more of a "selective consciousness" than anything else. Images are stripped of their personal, social, and political meanings, and no evidence exists that the labeled emotions represent anything that was actually felt. Undoubtedly, this AI model is simple and, therefore, less robust than those used by companies such as 'Affectiva'. Nevertheless, we argue that even a "high quality" or presumably a fairer dataset (if this is even possible) would still lack substantial support that proves that it works "correctly."

Attaching an AI System to the Body

Artificial intelligence systems that recognize emotions are usually disembodied. These models often look at us from far away without us realizing they are watching. Donna Haraway described this kind of gaze as an "unregulated gluttony," a "god-trick illusion of seeing everything from nowhere." Such a gaze, she asserts, "fucks the world."³¹ Our idea to attach an AI model to the body is inspired by her call to reemphasize vision through alternative feminist perspective. However, we found that even when the AI gaze is lowered to the level of the human eye, it can still operate as an asymmetric gaze from nowhere. Such understanding is also apparent in Karen Palmer's interactive film Riot AI (2016) which emphasizes a potentially violent encounter between a rioter and a police officer, mediated by an emotion recognition algorithm. Although the system is attached to the officer's body, it still serves as a gaze from nowhere. The policeman does not actively participate in evaluating the emotions of those who are stopped and assessed, he simply reacts according to the outcomes of the algorithmic gaze.

As part of the design of Structures of Emotion, we considered how the AI might extend the wearer's mind and augment their perspective. Wearing the device allows one to use their organic senses while being constantly guided by the device's perspective. Using both views, the wearer can note when they agree or disagree with the outputs of the device. Wearers may then ask themselves: "Do I understand this situation? Do I agree with the AI's assessment of this person's emotions? Can I recognize, acknowledge, and respect these emotions?" With that, both the human’s and machine’s emotion evaluation practices come into question. This is also where other cognitive abilities can come into play and the wearer can then use this opportunity to spark a conversation with the person in front of them and ask, "How do you feel?" The answer to this question may remind us that technology is far from maturing beyond its maker and that both humans and machines still have much to learn. By adding another perspective to our vision, the wearable device may promote traditional practices of conversation between people, nurturing relationships of attention and care.

Wearable computers, were extensively explored by researcher Steve Mann who claims that the intimate relationship between humans and these devices leads to exceeding the wearer's capabilities. For him, the fundamental purpose of wearable computing is personal empowerment.³² Nevertheless, he also expresses concern that people may become dependent on this technology if they use it for an extended period of time.³³ It is possible that dependency could lead to the erasure of human knowledge and the decline of organic human capabilities. Such a process might seem difficult to imagine with the wearable used in Structure of Emotions; however, it is easy to imagine a time when the algorithms will be far more sophisticated, and our dependence on them would seem inevitable.

Imagining Transhumanist Futures

Human-AI bi-directional symbiosis goes well beyond wearable computing. It imagines a speculative future in which AI and Humans can potentially merge into a super-intelligent, transhumanist figure. In this potential future, both humans and AI symbionts benefit from the symbiotic relationship. This relationship will be strengthened by AI's ability to detect, communicate, and express emotions. Humans are still better at this task than algorithms, but we envision a future in which algorithms may surpass human abilities and even erode humans' emotional capabilities. In considering the benefits humans might bring to this symbiotic relationship, we argue that they are centered around the body, its biological structure, and natural sensitivities. Currently, AI faces major challenges in shifting from a narrow to a more general intelligence owing to its imbalance in dealing with abstract truths versus the gritty world of exceptions.³⁴ In most cases, AI algorithms function as disembodied software with a huge appetite, but however much we 'feed' them, they still lack a solid understanding of the world. This is evidenced by the numerous instances in which AI models have produced racist, misogynistic, offensive, or seemingly absurd content. By attaching these models to the human body, they can benefit from the body's inherent knowledge, adaptation to the environment, and attachment to the world.

The human-AI bodies of the future will likely go far beyond fashionable devices and include brain-computer interfaces. Despite the fact that brain-machine interfaces have not yet become a reality, neuroscience research has already made impressive advances in this direction. Among the most famous examples is the monkey Aurora who operated a robotic arm just by thinking about doing so.³⁵ With current scientific advancements we can easily speculate soldiers operating weapons with their thoughts while suppressing feelings of fear and anxiety using brain devices.³⁶ The transhumanist techno-optimistic trajectory of such projects aims to empower individuals. But is this necessarily the case? Will this super-intelligent Human-Machine figure function as an enhanced individual? Or not?

We define ourselves by our emotional capacities; they are based on millions of years of evolutionary adaptation for survival and advancement. What will be our role in the world if we depend on AI algorithms to mediate our emotional landscapes? How can we avoid becoming nothing more than a numb 'meat' body carrying around an AI algorithm? Transhumanists argue that the continuation of personhood is essential in becoming a transhuman. It must include the continuity of memories, attitudes, values, and emotional dispositions.³⁷ In light of this condition, we can already see the consequences of sharing our emotional abilities with AI. As philosopher Susan Schneider explains, we can reject the idea entirely because techno-enhancements themselves alter the pattern of the original individual. Their impact on identity is always significant. For her, "the transhumanist developmental trajectory... is a technophile's alluring path to suicide."³⁸ A glimpse of such alterations in one's identity was even evident in the performances of Structures of Emotion with participants and performers changing their behavior by responding to the presence of an AI algorithm within their shared space.

Conclusion

The artwork Structures of Emotion was designed to articulate an AI-Human symbiosis. Physical limitations, however, prevented the artists from fully conveying this idea. Emotion recognition algorithms are still premature. Data they are trained on is devoid of context and potentially biased. It is still necessary to gain a deeper understanding of emotions' definition, function, and purpose. It is unclear what is the impact of either disembodied or embodied relationships with AI algorithms. For obvious reasons none of the performers were ready to integrate an artificial intelligence algorithm into their bodies in any invasive manner. Even wearing the devise was somewhat cumbersome. Despite the limitations, this artwork still allowed us to consider the potential of such a symbiosis, speculate on its implications, and imagine its future. We realize that emotion recognition algorithms are a crucial step forward for human-computer interactions and we posit that any use of such algorithms invites us to think more broadly about what it means to be emotionally intelligent. It is possible for AI to expand our emotional capacities, but it also holds the risk of eroding them. It is important to consider the design of these algorithms as well as the way they interface with our bodies. If this artwork can indeed serve as a McLuhanist Distant Early Warning system, we must strongly advocate that the integration of this technology will give us more agency and autonomy over our emotional states, improving rather than eroding our most human drive for meaningful, intimate relationships with one another.

Acknowledgments

The authors wish to thank Ofer Meshi for invaluable assistance with the AI system. The street performance was developed during a summer residency at the 418 Project, Santa Cruz. The online performance was developed during the Impact Art AT residency at Zero1, San Francisco, CA under the guidance of Patricia J. Reis. The article has been very much improved thanks to generous and inspiring feedback by the double-blind peer reviewers.

References

1 Vannevar Bush, “As we may think” The atlantic monthly 176, no. 1, 1945, 101-108. 

2 Affective Computing Market by MarketsandMarkets.com, accessed September 10, 2023, https://www.marketsandmarkets.-com/Market- Reports/affective-computing-market-130730395.html

3 TED, 2015, June 15, This App Know How You Feel - From the Look on Your Face | Rana el Kaliouby [Video], YouTube.

4 Marshall McLuhan, Understanding Media: The Extensions of Man, New York, McGraw-Hill, 1964.

5 Omar Aymen, “Emotion-recognition”, GitHub, accessed September 10, 2023, https://github.com/omar178/Emotion-recognition

6 Paul Ekman, "The argument and evidence about universals in facial expressions", Handbook of social psychophysiology, 143, 1989, 164.

7 Paul Ekman, Wallace V. Friesen, "Facial action coding system”, Environmental Psychology & Nonverbal Behavior (1978).

8 Alexa Hagerty, “Emojify”, Dovetail Labs, accessed September 10, 2023, https://emojify.info/menu

9 Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix Martinez, Seth D. Pollak, "Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements", Psychological science in the public interest 20, no. 1, 2019, 1-68.

10 Ruth Leys, The ascent of affect: Genealogy and critique, University of Chicago Press, 2019.

11 Meshi Avital, Angus G. Forbes, ""Don’t Worry, Be Happy”: Resisting an AI Emotion Recognition System With a Smile", In 10th International Conference on Digital and Interactive Arts, 2021, 1-4.

12 Meshi Avital, "Should We Become Emotional With AI?: Performative Engagements With an Affective Algorithm", International Journal of Art, Culture, Design, and Technology (IJACDT) 11, no. 3, 2022, 1-13.

13 Vincent James, "Canon Put AI Cameras in Its Chinese Offices That Only Let Smiling Workers Inside", The Verge, June 17, 2021, accessed September 10, 2023, https://www.theverge.com/ 2021/6/17/22538160/ai-camera-smile-recognition-office-workers-china-canon

14 Hito Steyerl, "The Language of Broken Glass", Presentation, part of “Stop Making Sense”, HKW, Berlin, 2019.

15 Nazanin Lankarani, "The many faces of Yue Minjun", The New York Times 5, 2012.

16  Kristina Höök, "Affective loop experiences: designing for interactional embodiment", Philosophical Transactions of the Royal Society B: Biological Sciences 364, no. 1535, 2009, 3585-3595.

17 Ana Paiva, Iolanda Leite, Tiago Ribeiro, "21 emotion modeling for social robots", The Oxford handbook of affective computing, 2014, 296.

18 Koji Inoue, Divesh Lala, and Tatsuya Kawahara, "Can a robot laugh with you?: Shared laughter generation for empathetic spoken dialogue", Frontiers in Robotics and AI, 2022, 234.

19 Julie Beck, "Hard feelings: Science’s struggle to define emotions", The Atlantic 24, 2015.

20 Margaret Mead, “Darwin and Facial Expression-Century of Research in Review-Ekman P”, 1975, 209-213.

21 Lisa Feldman Barrett, How emotions are made: The secret life of the brain, Pan Macmillan, 2017.

22 Marda Vidushi, Shazeda Ahmed, "Emotional Entanglement: China’s emotion recognition market and its implications for human rights", 2021.

23 Michael Kwet, Paul Prinsloo,"The ‘smart’classroom: a new frontier in the age of the smart university", Teaching in Higher Education 25, no. 4, 2020, 510-526.

24 Andrew McStay, "Emotional AI and EdTech: serving the public good?." Learning, Media and Technology 45, no. 3, 2020, 270-283.

25 Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy, "Mitigating bias in algorithmic hiring: Evaluating claims and practices", In Proceedings of the 2020 conference on fairness, accountability, and transparency, 2020, 469-481.

26 Elmira Van den Broek, Anastasia Sergeeva, Marleen Huysman, "Hiring algorithms: An ethnography of fairness in practice", 2019.

27 Jennifer Valentino-DeVries, "How the police use facial recognition, and where it falls short", New York Times 12, 2020.

28 Taylor Telford, "Emotion AI’is a $20 billion business. New research says it can’t do what it claims", The Washington Post, July 31, 2019, accessed September 10, 2023, https://www.wash-ingtonpost.com/business/2019/07/31/emotion-detection-ai-is-billion-industry-new-research-says-it-cant-do-what-it-claims/

29 Virginia Eubanks, Automating inequality: How hightech tools profile, police, and punish the poor, St. Martin's Press, 2018.

30 Alexa Hagerty, and Alexandra Albert, "AI is increasingly being used to identify emotions-here’s what’s at stake", In The Conversation, 2021.

31 Donna Haraway, "‘Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective'," In Space, gender, knowledge: Feminist readings, 53-72, Routledge, 2016.

32 Steve Mann, "Wearable computing as means for personal empowerment." In Proc. 3rd Int. Conf. on Wearable Computing (ICWC), 1998, 51-59.

33 Steve Mann, "Humanistic computing:" WearComp" as a new framework and application for intelligent signal processing", Proceedings of the IEEE 86, no. 11, 1998, 2123-2151.

34 Gary Marcus, Ernest Davis, Rebooting AI: Building artificial intelligence we can trust, Vintage, 2019.

35 Miguel Nicolelis, Beyond boundaries: The new neuro-science of connecting brains with machines---and how it will change our lives, Macmillan, 2011.

36 Nancy S. Jecker, Andrew Ko, “Brain-computer interface could allow soldiers to control weapons with their thoughts and turn off their fear - but the ethics of neurotechnology lags behind the science”, The Conversation, December 2, 2022 accessed September 10, 2023.

37 Nick Bostrom, "The transhumanist FAQ", Readings in the Philosophy of Technology 2, no. 4, 2003, 355-360.

38 Susan Schneider, Artificial you: AI and the future of your mind, Princeton University Press, 2019. 

The w(e)aves of complexity: Relational ontologies within the Symbios Art Exhibition

Paola Torres Núñez del Prado

Stockholm University of the Arts, Stockholm, Sweden
paola.torres@uniarts.se 

Abstract 

The works included in the Symbios art exhibition spread through various disciplines and mediums: painting, weaving, interactivity, sculpture, Bio Art and even A.I. Its transdisciplinary condition does not originate in the multi-mediality of its constituents exclusively, but in the intentional implementation of an ecology of meanings once the pieces, characteristically polysemic, are made for and placed in the gallery. The physical division of the two rooms that conform the exhibition space becomes what defines the type of interaction that the art works included will deal with, each area taken as a subcategory of the general symbiotic relationships. I proceed to find a correspondence of the symbiotic process in the social realm, more specifically, in the cultural process known as syncretism, that is also a distinctive aesthetic quality of the art works exhibited. Such a web of relational ontologies goes beyond the legacy of Conceptual art, now taken as a product of a Western paradigm that is based in dualism, classification, and exact definition. Instead, the show embraces the diverse narratives behind its components as a net of meanings and diverse cultural references, so to achieve a complex yet holistic approach to art making and curating simultaneously.

Keywords 

Symbiosis, Natural Science, Social Science, Art, Philosophy, Aesthetics, Syncretism, Culture .

DOI

10.69564/ISEA2023-74-full-Torres-Nunez-Symbios-Art-Exhibition

Introduction 

The “Symbios” (1) art show was a transdisciplinary exhibition presented in November, 2022 in Tegen 2 gallery in Stockholm, Sweden. It included three bio-textiles, one puzzle sculpture, a big interactive loom (including one of the bio-textiles mentioned before), an introductory video (2) recorded during my stay in the Finnish Tuo Tuo art residency where I started working with lichens ¹, a textile painting, a 3D printed object, and two videos generated using the Neural Radiance Field method that converts a set of 2D images in 3D renders.² The broader idea of the symbiotic process—that of long-term interaction among living organisms—was taken as a starting point so to manage the relation among the works, the narratives behind these, their aesthetic qualities, and their physical place in the venue. This paper pertains the strategies I applied to put up a complex art show in just a couple of weeks, consciously undertaking various roles, that of technician, curator, and artist, and the relational ontologies incorporated both in such tasks and in the artworks themselves. What I discuss here might be a common occurrence for us artists that still exhibit in gallery spaces, yet I am proposing a different way of managing both art creation and curation, and its relation. Regarding such matters, the introductory text placed in the exhibition was nevertheless co-written with gallerist and artist Gunilla Sköld-Feiler, while other works were implemented jointly with the support of Jonas Pajari from Artificial Ingenuity(3) and some came to be in the interaction with unknown people. This is relevant to mention as I want to recognize collaboration as an inherent part of both production and the creative process itself. 

Layout and artworks 

The Symbios exhibition sprouts from my work with the Usnea lichen,³ a symbiotic organism that is part algae and part fungus. Some of the pieces exhibited were presented as a series(4) and each was considered a Sincreto (Sincret in English) a neologism to describe a syncretic thing. The word is meant to link symbiotic and syncretic processes, the former, a term used in the natural sciences, and the latter in the social sciences. Symbiosis becomes a term that would allow different knowledge systems to coexist, even “mutually exclusive conceptual categories and frameworks” that in turn allows the distancing from the Western tradition of seeing only its logics and doctrines as universal.⁴ The name of the show becomes another node from the invisible semiotic networks that I imagine as back-up, as support for the visible part of the exhibition, nodes from which other branches unfold. I will elaborate on the idea of thinking the art object as inherently syncretic later. The two rooms adjacent black and white respectively, and it was this color/structure binary, what permitted me to make yet another conceptual subdivision, this time related to the type of symbiotic interaction that the works included in each room would deal with.  

Mutualism 

Lichens are complex organisms living in a symbiotic relationship with fungi and algae (and/or Cyanobacteria). The Usnea lichen grooms its photobiont, the algae. In turn, the photobiont gathers resources from such exchange. These associations give rise to a great number of secondary metabolites, the majority of which are unique to these organisms too. As both are benefitted from the interaction, it falls in the category of mutualistic relationship. Hence the name of the first room. The Usnea lichen belongs to the family Parmeliaceae and contains salazinic and usnic acid. These are said to have several possible biological activities, among these antibiotic, antiviral and anti-inflammatory.⁵ They also form “close and long-term interactions” from its constituents (a mycrobiont, the fungi and a photobiont, the algae) and are known to develop cooperative molecular mechanisms between the symbiotic partners, for then to develop new functions as symbiotic entities. In lichen symbiosis, mutualistic relationships between lichen-forming fungi and algae and/or cyanobacteria produce unique features that make the organisms adaptive to a wide range of environments.⁶  For the show, this cooperation among two organisms is taken now as inherently social phenomenon, hence the mapping to the cultural process known as syncretism. In current usage in anthropology and religious studies, it generally refers to a mixing of elements from different religious systems or traditions.⁷

Sincreto I (To weave a garden) - Interactive Sculpture.

This loom became an interactive piece once I used it to weave the Usnea lichens along electronics, and then applied the capacitive sensing method whose code and files were stored in a dedicated Raspberry Pi microcomputer, so to emit sounds when touched. The created living textile considers the sensor quality of the Usnea lichen as a marker of clean air and expands it towards the development of a bioelectric textile synth. 

Figure 1. “Sincreto I (To weave a garden)” as part of Sound Performance, 2022, photo by Paola Torres Núñez del Prado 

The created living textile takes into account the sensor quality of the Usnea as a marker of clean air and expands it into the development of a bioelectric textile synth. Similar D.I.Y. synths are popular now,⁸ yet the Sincreto I biosynth is different as it depends on how humid the Usnea lichens are, due to external factors. By touching the Usnea, sounds originally recorded in the Finnish forest and the Amazonia are reproduced, and, in a next iteration, modulated. 

If it gets too hot without airflow, or too warm and dark, these lichens die. Another interesting thing is that they immediately turn green once becoming humid, which adds an extra layer of reactivity. The lichens can live long if proper conditions are kept, which brings notions involved with care. Maintenance implies monitoring and making them humid frequently, in a space like a gallery, which is not in the open and does not have a constant flow of clean air. One must say, their dependency on the stability of climate conditions that are currently changing, make them particularly exposed. Its relation to ecology is then put forward, parallel to the starting of a discussion related to practical conservation of artworks in cultural exhibition set-ups. 

Figure 2 Close up of woven Usnea Filipendula

The technique I used for the weave is taken from a traditional one used by Andean inhabitants from my country of origin, Peru. I decided to apply this weaving technique as it allowed me not to damage the lichens. This living textile was woven on a Swedish loom, an important cultural tradition among older women in Sweden. The sounds triggered had been mostly recorded in the Finnish forest, but they also included samples from the Amazonian Forest. Due to the intermixing of diverse traditions, this installation is considered syncretic. 

Sincreto II (Tupicochan knot) - Living sculpture

This lichen-based sculpture portrays a Khipu tied up as if it were a snake curled onto itself, the Tupicochan style placing of Khipus. A khipu is a knot-based technology to store information, that originated in the Andean region in pre-Hispanic times.⁹ Tupicocha is a town in the highlands of the Lima region, and this is relevant as they keep the Khipu tradition as part of their civic ceremonies. They have a particular way of placing the Khipu, different of what we know from the Incan Empire and previous Andean civilizations: they tie the Quipu onto itself, then becoming a knot of knots. I took this as a representation of self-consciousness and its recursive nature, now taken as tied up on a knot itself.¹⁰ 

Figure 3 The Tupicochan Knot 

Sincreto III (The living textile)

This piece consists of woven lichens as living textile placed over a portion of Swedish-style wall painted in the traditional falu rödfarg (red paint). The paint is relevant in the history of Swedish technological development: designed in the 1700's, it not only became a traditional symbol (most countryside farms and houses are painted in such a color) but its chemical composition has the characteristic of penetrating the wood these buildings are made of, in turn, making it more resistant.¹¹ This interaction of components evoked yet another level of symbiosis, and the fact that a portion of what it looks like a wall of these red-painted houses is part of this installation, onto which another lichen-based living textile is placed, in turn, woven using Peruvian textile techniques, makes it aesthetically syncretic as well. 

Figure 4 The Living Textile 

Sincreto V (Liquid Identity)

This wall-like structure is a puzzle that can be taken apart in pieces. Also painted in the traditional Falu röd paint, it continues referencing Swedish traditions. The syncretism, in this case, lies in the process: the cut is not handmade, but computerized (laser cut), and the fact that it looks that these structures are melting, refer in turn to a more globalized, contemporary aesthetics. 

Figure 4 Liquid Identity
Figure 5 Andean Glitch 

Andean Glitch (Textile Painting)

Part of Holly Grimm's Aikphrasis project that consisted of "prompting" artists (in the manner of CLIP-based image creation commonly prompted by text)¹² with A.I. generated paragraphs, this textile painting was inspired by a text written by the machine (GPT-3). 

Parasitism 

Parasitism is a symbiotic relationship that consists of an organism that ends up harming another one, called the host, while in a relationship that, unlike mutualism, only benefits the former. This interaction can not only end up killing the host, but the parasite itself, when the organism it depended upon dies. 

In early 2022, I was hired by an artist to do a 3D character model based on the standard ones generated by crowd simulation software, so to be 3D printed by a machine and shown as public art. The aim of the artist was to represent a generic, androgynous, seemingly standard digital Persona. The final 3D printed, slim, young yet bald, androgynous character was set up, arms extended, in the middle of a public square in Stockholm, place where we both live. Some days after a controversial Swedish election, the arms were found to be ripped off from the huge final sculptural 3D print exhibited in the square: this is notable as the main party that won the elections is considered to have origins in the Far Right, and in this context, the extended raised arms could have been seen as evoking an initially unintended reference, one that, after the elections, came to be considered a problematic symbol by citizens. Using Artificial Intelligence, more specifically, the Neural Radiance Field method, the physical manifestation of public intervention onto the so-called "generic" human sculpture is reconstructed and made into a new artwork, so to address how symbols change alongside the social milieu, the impossibility of neutrality within the digital realm, and ultimately, analyzing both artists' and citizen's ethos and practices within Public Art. This particular work consists of the 3D reconstruction using Nvidia’s Instant Neural Graphic Primitives implementation (Instant-ngp NeRF) ¹³ of a realworld 3D printed sculpture recorded with a hand-held mobile phone. 

Figure 6 Reconstruction of the Unseen 

The lost right arm has been reprinted and placed in a position indicative of a narrative I present as speculation, as possible explanation for what passers-by could have seen in the now missing arm. This is, in turn, is accompanied by a drawing of another one of these speculative interactions with an unseen audience, this time taken from an image of a graffiti drawn nearby an art space where I organized an A.I. art show in August, also in Stockholm. 

In the case of the Parasitism installation, the question then becomes: who is the host and who is the parasite? Is it the artist originally hiring me as designer of the work, using my labor for her needs, the parasite? Is it me, when repurposing images taken from a real-world vandalized sculpture to make a new artwork? Is it the people that ripped it apart, or is the pre-trained A.I. doing the virtual volumetric reconstruction based on the data sets it has been trained with? 

Figure 7 Parasitism Installation 

Sincreto IV (Reconstruction of the unseen)

Thinking about these subjects: the author, collective creation, A.I. models/datasets, and the parasitic relationships within all of these, I came up with this work, that I called Sincreto IV. This video was generated by using the NVIDIA Instant - NGP Neural Radiance Field method, a Machine Learning method (A.I.) that allows the generation of novel 3D scenes based on 2D data. This method was used to generate novel view scenes from an original low-quality video, so to be presented as an artwork that implicitly recognized the violent transformation of a real-world sculpture after people intervened it. The act of bringing back the sculpture to the digital realm, after being transformed by people and being taken away from public view, was an act that would recognize their gesture as a potential protest. The reconstruction aims to harness such action for posterity, as the sculpture has been removed from the public space by now. Thus, the video attempts portraying a moment in time when the sculpture was still public, after its arms were ripped off. The video recreated a dreamy scene out of glitched generated virtual figures, with artifacts forming out of the lack of data: I had used images with lossy compression to train the model. This is a collaborative work and it is presented as such: Jonas Pajari had his role in it, I had my role in it, the artist had its role in it; the public witnessing and intervening the real-world sculpture had its own role in it, and the A.I. also had a role in its creation, all in a feast of parasitism now presented as the commonality of art creation. 

Figure 8 Reconstruction of the Unseen 

Sincreto VII (Armlängds avstånd)

The right arm that was taken from the sculpture has been reprinted in 3D and it is exhibited next to the critical graffiti addressing A.I. issues. 

The term "arm-length distance" (Armlängds avstånd) is now widely used in Sweden by journalists and politicians in areas such as cultural policy, but also media policy and research policy, where "arm's length" is used as a metaphor for the appropriate distance between those in power in different roles and against mutual intervention.¹⁴ 


Figure 6 Armlängds avstånd 

Discussion 

I came back from the art residency in mid-Finland at the end of September 2022, to be offered the exhibition space. In the middle of such Nordic Forest, I became familiar with the Usnea lichen growing in the area and had already made two works with such living organism, the Tupicochan Knot and The Living Textile (Sincreto II and III respectively) before I was offered the space. All the other works, except the “Andean Glitch” painting, were developed in October 2022, mostly in the days before the opening day itself. As I was familiar with the space beforehand, a gallery that consists of two adjacent spaces, a white room of around 6 m x 6 m next to a black one of 4m x 6 m, I embraced such format, working around it but keeping the art gallery without many changes.

The aim was not to convert the space for my needs but think of the work for the space. By the naming of the lighter room as the “Mutualism,” and the darker room as “Parasitism,” taking the binary condition of the space as starting point, I reflected on this type of interaction in between organisms, now also applied to human society and culture. The presentation of the white room was similar to that of a formal installation, which in turn made the main white space to look more like a (weaving) studio. Nevertheless, there are other factors to be taken into account when describing this exhibition from the operational side, practical issues linked with curatorial aspects beyond the arrangement of objects in space, and that has to do with the time given to me for preparing and setting up the show: I was informed that I could use the space just around a month before the opening, and I would lack much of the support usually given by the gallerists, as one of them had to travel. This in turn pushed me to design an efficient strategy that took into consideration both the time and the space factor, so to be able to effectively show all the works holistically, in a manner that would permit raising complex sociopolitical and aesthetic subjects that I considered relevant, as they were related to the particular societal situation we were experiencing in Sweden at the time. So my method could be described as symbiotic on itself, and of fractal nature, related to its multiplicity of meanings: some works took shape in the midst of human-human interaction, others came to be among a human and a non-human agent, such as the lichens and the forest, and other works were generated in the area in between humans, machines, and other living organisms, such as the Usnea lichens. These interactions in turn shaped the reasons why and how I established them, and these also include aesthetic ones: in the syncretic process itself, with visual, tactile and sonic characteristics that allowed people to recognize the cultural references that the works included without having to read a text or receive an explanation (something that I did anyways). As such, the works are not only about one or various concepts, ideas, or narratives, but about the connections behind their polysemic nature. 

Beyond the Western paradigm 

According to Gloria Schaab, relational ontology is the philosophical position that what distinguishes subject from subject, subject from object, or object from object is mutual relation rather than substance. Ontologically, substance refers to the essence or nature of a being.¹⁵ The concept of “substance,” is treated commonly within philosophy as that of object, or thing, when this is contrasted with properties, attributes or events.¹⁶

In this exhibition, such a substance, related to the art works presented, would be inherently syncretic, and this in turn emphasizes in the mutual relation of the combined cultural properties, attributes, and, in the case of the generated videos, referring to an event that could also be placed within a syncretic framework. As mentioned earlier, I am doing an explicit parallel in between the syncretic process (and its products) within the social realm and its cultural expressions, and that of the symbiotic relationships in the natural world. 

By framing it this way, I do not mean to separate the social realm, culture, or the human sphere, from that of the natural world. On the contrary: if anything, my approach aims to (re)conciliation. But the Western paradigm of thought has characterized for its tendency to dualism, based on a binary logic (true/false) and this has been reflected in such a division (culture vs. nature) as well as in the division of (human) knowledge into various disciplines. While Interdisciplinarity refers to the collaboration across differences – in turn embracing such differences – Transdisciplinarity is characterized for its holistic approach ¹⁷. In order to transcend such tendency towards dualisms, embedded in the history of conceptual art (where ideas had been clearly defined) and by taking each of these works as polysemic (having multiple meanings and cultural references) I present the show as framed within an ecology of meanings but also of aesthetics, processes and events, as the substance of these “cultural products” (the Sincretos or Syncrets) are now taken as the product of not just one, but the amalgamation of various cultural traditions and thought systems. The thinking of the art object as inherently syncretic needs to be taken as an open gesture towards real universality. As mentioned before, there is a fractal quality in the way the works have been both developed and presented, not only because I am talking explicitly about an ecology of meaning, but due to its network and its connected pattern (of cultural references, symbols, and materials) that also reflects in the works themselves and that can be described as (inter)woven: such a pattern is found as a theoretical framework (framed upon a relational ontology), but is also materially perceived in the works themselves. 

In what relates to the value of art, the lichen works pose an interesting question. Due to the condition of such symbiotic organisms, decay is a constant possibility. This implies that the works are always threatened if conditions for their survival are not met. In turn, this means that they can actually cease to exist as such, as temperature and humidity has a direct impact on the structure of such works, as, depending on environmental factors, they would turn brown when the photobionts die in cool and dry conditions, yet the weaves would be kept structurally the same. Under warm and dark conditions and in a storage space that is not extremely dry, they would lose all structural characteristics, and become a mushy black substance with no major reminiscence of the original piece. The artwork, and in turn, its value, would become nonexistent because of this transformation that destroys all original aesthetic qualities. As such, the maintenance of the artwork not only becomes an act with an ecological impact (the saving of a threatened species) but the maintenance of its value thought under an art market shaped by capitalism, taken here as the implementation of the Economics of Parasitism.¹⁸ Symbios would be posing another way to approach the “Science + Art” realm, where it is common to find artworks characterized by a type of aesthetics that has the tendency of bonding the two disciplines together through what it could be called an aesthetic mapping of the Sciences onto the arts through what I call “aesthetic bridges” that can have reminiscences of Sci-Fi (futurisms, etc.), mapping the scientific laboratory in the art gallery itself (as a white, sterilized environment) with its test tubes and petri dishes, or displaying documentation of a research. For artist Hito Steyerl, “artistic research” has even become a new discipline, one that normalizes, regulates, and ensures the repetition of protocols.¹⁹ Extending this idea, Claire Bishop considers that Research art (another branch of the Art and Sciences), that is, “its techniques of display, its accumulation and spatialization of information, its model of research, its construction of a viewing subject, and its relationship to knowledge and truth—cannot be understood in isolation from contemporaneous developments in digital Technology.²⁰ 

Figure 8 Graphic portraying the “aesthetic bridge” as Science shaping the Arts

In this way, and by working with digital technology alongside living beings, both the Natural Sciences and the Technosciences are addressed in the gallery space. Symbios goes into the actual etymology of the words ars and techné, the origin of the term “art” and the term “technology”, which in both cases, originally meant craft. Craft becomes the proposed “aesthetic bridge” of this exhibition, going against the hierarchy of knowledge-as-discipline reflected in the evident aesthetic domination of the Sciences onto the Arts.

Figure 8 Graphic developed by Inés Moreno, Nitouche Anthousi, Olivain Parry and me at a workshop in ISEA2023 

Conclusion 

Through the mapping of the process of symbiosis with that of cultural syncretism, taking the latter as a social expression of non-human origin, I go beyond the realm of the animal kingdom to think of other living forms as social too. By doing this, I go beyond anthropocentrism. I also discuss the connection in between the Social Sciences and both the Natural Sciences and the Technosciences in a transdisciplinary historical framework by referencing an approach to understand human society as part of the Social Sciences originating in the West: the phenomenon known as Social Darwinism ²¹ that I perceive still shapes Western society under Capitalism. This approach to study human culture is assumed as something from the past, yet we can still find it in extremely hierarchized societies, particularly in Neoliberal regimes where the idea of freedom is very much linked to the imperative of growth-as-development (and strength), and where domination can be justified in such meritocratic framework as the “survival of the fittest.”²² 

I propose another conceptual extrapolation from the Natural Sciences, this time not framed within such hierarchic evolutionary theories, but more of communal nature exemplified by mutualism, with a more positive outcome for the symbionts, and parasitism, closer to the structure of Social Darwinism. 

The ontological relations in the show are presented as an ecology of meanings, of cultural references, and of aesthetic qualities. The fractal nature of the show initiates from the physical division of the two rooms that conform the exhibition space, a white room called “Mutualism” and a room with black walls called “Parasitism.” These are sub categories of the general symbiotic relationships the Natural Sciences have defined. As syncretism is also a distinctive stylistic quality of the art works exhibited, they could be split into smaller semiotic particles too, that come to be in the intermixing of various cultural traditions reflected both in the artworks techniques, and in their aesthetics as well. We can then go deeper to find even smaller particles, those related to historical referents and their relation to memory, or personal narratives, or the artworks’ own interactive or digital condition, their biological processes, their material design and its symbolism etc. Through such a web of relational ontologies, the show intends to go beyond the legacy of Conceptual art, now taken as a product of a Western (Scientific) paradigm that is based in dualism, classification, and exact definition. In this case, all works were conceived as multilayered in what its meanings, function, and appearance refer to. By working with digital technology, I openly addressed both the Natural Sciences and the Technosciences, and by going to the actual origin of the concept of art and technology as science-as-craft, proposed now as the “aesthetic bridge” that would go against the hierarchy of knowledges reflected in the common aesthetic mapping of the Sciences onto the Arts, as well as Western traditions over non-Western ones. 

The works of the exhibition are taken as polysemic (i.e. the Tupicochan Knot is both a cultural artifact and a model of consciousness), that is, as having multiple readings and possible interpretations. These various narratives are embraced as a net of meanings as well as physical properties that are signs related to materials, to techniques, to their looks and their history, so to achieve a complex yet holistic approach to art making and curating simultaneously. 


(1) Symbios is the Swedish word for “Symbiosis.”

(2) Made by Renzo Signori from Koyne.org

(3) Collection or sequence of art works

(4) "This survival of the fittest, which I have here sought to express in mechanical terms, is that which Mr. Darwin has called 'natural selection', or the preservation of favoured races in the struggle for life." 

References 

1 Ioana Onuț-Brännström, The puzzle of lichen symbiosis, Uppsala University, 2017, 13.

2 Ben Mildenhall, et al. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, (arXiv, 2020) Accessed December 8th, 2022, https://arxiv.org/abs/2003.08934.

3 B. S. Prateeksha, The genus Usnea: a potent phytomedicine with multifarious ethnobotany, phytochemistry and pharmacology (RSC Adv. 2016) Accessed December 8th, 2022, https://www.researchgate.net/publication/292212820_The_genus_Usnea_A_potent_phytomedicine_with_multifarious_ethnobotany _phytochemistry_and_pharmacology.

4 Maulik Vyas, From Sincretism to Symbiosis: An approach to integral knowledge in Humanities in Anekaant: A Journal of Polysemic Thought, No. 2, Balvant Parekh Centre for General Semantics and other Human Sciences, 2014, 41-67.

5 Serap Çelı̇kler Kasimoğullari et al., Genotoxic, cytotoxic, and apoptotic effects of crude extract of Usnea filipendula Stirt, in vitro, Turkish Journal of Biology, 2014, 940-947.

6 Mieko kono et al In vitro resynthesis of lichenization reveals the genetic background of symbiosis-specific fungal-algal interaction in Usnea hakonensis, BMC Genomics, 2020.

7 Patrik Fridlund, “Responding to Syncretism” in Theological and Philosophical Responses to Syncretism: Beyond the Mirage of Pure Religion, Brill, 2018, 41-67. 

8 Mads Hobye, Singing plant: Make your plant sing with Arduino in Instructables, https://www.instructables.com/Singingplant-Make-your-plant-sing-with-Arduino-/ Accessed August 11, 2023.

9 Bridget Alex, The Inka Empire Recorded Their World In Knotted Cords Called Khipu, Discover Magazine, 2019, Accessed December 8th, 2022. https://www.discovermagazine.com/planet-earth/the-inka-empire-%20recorded-their-world-in-knotted-cords-called-khipu.

10 Frank Salomon, The Cord Keepers: Khipus and Cultural Life in a Peruvian Village, Duke University Press, 2004.

11 Sanna Casson, Falu rödfärg – pigmentet som målat en hel nation, Sydved, 2021, Accessed December 8th, 2022 https://www.sydved.se/aktuellt/inspiration/hem-och-tradgard/falu-rodfarg-pigmentet-som-malat-en-hel-nation

12 Anonymous, Clip, Huggingface, 2021, Accessed December 9th, 2022, https://huggingface.co/docs/transformers/model_doc/clip  

13 Thomas Müller et al., Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, NVIDIA Labs, 2022, Accessed December 9th, 2022. https://nvlabs.github.io/instant-ngp/

14 Anonymous, Armlängdsavstånd, KLYS, unknown date, Accessed December 9th, 2022, https://klys.se/vara-ord-och-begrepp/

15 G.L. Schaab, Relational Ontology In Encyclopedia of Sciences  and Religions, Springer, 2013, 1974–1975.

16 D. Wiggins, Substance in Philosophy: A Guide through the Subject, Oxford University Press, 1995, 214-249

17 Joi Ito, Antidisciplinary, MIT Media Lab, 2014, Accessed December 9th, 2022, https://joi.ito.com/weblog/2014/10/02/antidisciplinar.html

18 David Levy, Sandra Peart, Parasite economics, Foundation of Economic Education, 2022, Accessed December 9th, 2022. https://fee.org/articles/parasite-economics/

19 Hito Steyerl, Aesthetic of Resistance?, in Intellectual Birdhouse: Artistic Practice as Research, Koenig Books, 2012, p.55.

20 Claire Bishop, Information Overload, Artforum, 2023,  https://www.artforum.com/print/202304/claire-bishop-on-thesuperabundance-of-research-based-art-90274 Accessed August 10, 2023

21 Claire Goldstene, Our Social Darwinists Then And Now, Inequality.org, 2015, Accessed December 9th, 2022, https://inequality.org/research/american-social-darwinism/

22 Herbert Spencer, The Principles of Biology, D. Appleton and Company, 1862.

E-cellulose: symbiotic cultivation for the production of smart textiles in a framework of sustainable fashion and electronic art

Alis Pataquiva-Mateus

alisy.pataquivam@utadeo.edu.co

Abstract

Technology has become ubiquitous, it is all around us, being a permanent part of our day to day. Along with the emergence of the Internet of Things (IoT) and other innovative technologies such as Augmented Reality (AR), Artificial Intelligence (AI), Blockchain, IoT-based smart wearables and apparel harmonize functionality, with design, electronic art, and science. On the other hand, the textile industry is considered the second most polluting industry in the world, making it necessary to rethink and replace ordinary products in great global demand produced from unsustainable processes with ones that contribute to the reduction of the greatest number of environmental problems. For thus, the synthesis and structuring at nanoscale of an extracellular renewable biopolymer is proposed. This bacterial cellulose (BC), obtained from a fermentative process that requires low amounts of water for its production, it has been demonstrated having an excellent potential in the slow fashion movement due to low hydric and carbon footprint, and diminishing of tree falling or use of non-sustainable fibers; as well as, into the textile industry due to its low-cost raw materials, mechanical properties, and versatility.

Keywords

Emergent arts, e-wearables, slow fashion, electronic textiles, SCOBY, bacterial cellulose, nanostructuring, bio-based materials, sustainable products, low water and carbon footprint.

DOI

10.69564/ISEA2023-75-full-Pataquiva-Mateus-E-cellulose

Introduction

In a not-so-distant global IoT architecture, smart textiles are expected to be able to communicate massively with smart devices to process biometric information such as heart rate, temperature, breathing, movement, acceleration, or even mood, promising a new horizon for the intersection of electronic art and fashion in a sustainable way ¹. In this direction, electronic art understood as a fashion of making art through the use of technology including electronic devices came from conceptual art and systems art, and subsequently, it derived into the different information and media art branches; being precisely the International Symposium for Electronic Art—ISEA who has carried the banner of electronic art since 1988 at international level.

On the other hand, the textile industry is considered the second most polluting industry in the world, and it is estimated that around 120 m3 /Ton of effluents are discharged into water sources as a result of these activities,² which 2 to 50% are persistent pollutants that cannot be removed from wastewater by conventional methods,³ mainly due to the presence of free chlorine and toxic heavy metals. which in addition to polluting reduces oxygen levels in the water and inhibits microbial activity that stabilizes the organic form.⁴ In fact, the World Health Organization (WHO) affirmed that, in developed countries, this industry is responsible for the annual death of around 20,000 people involved in fumigation activities on crops for cotton production, mainly due to the amount of toxic pesticides that are required for this function.⁵

In this train of thought, a bacterial cellulose (BC) is proposed as an alternative textile material. BC is obtained from a fermentative process that requires low amounts of water for its production ⁶. In contrast to vegetable cellulose, BC is highly pure, its structure is free of pectin, lignin and hemicellulose,⁷ it has high porosity, excellent mechanical properties⁸ and is also highly hydrophilic, with the ability to absorb 98% of its weight in water.⁹ Based on its special structure and its unique properties, BC is postulated as a promising material that will revolutionize the industry in the coming years, due to its wide range of innovative applications, among which the textile industry stands out.

This is why bacterial polymer fibers are a potential substitute for cotton, a natural fiber considered unsustainable because its production requires high water consumption. It is estimated that around 2.6% of world water consumption is destined for the production of this fiber, during the bleaching, dyeing and printing processes, 150 m3 of water are required per ton of fabric⁵ in which it is evaluated that for the production of a pair of cotton jeans weighing 1 kg, at least 1,100 liters are consumed between the wet treatment and the printing of the garment.¹⁰

Last but not least, nanotechnology has made it possible to introduce new properties to the materials we know today such as nanoparticles, nanofilms, nanowhiskers, nanoflowers, among others, stand out. Thus, the term "conductive textiles" refers to the transport of signals in the so-called smart textiles, which use a wide range of products with specific surface conductivities, for which the textiles can be modified with the incorporation of conductive particles in the spinning or electrospinning process, coatings of conductive metal, or the weaving or stitching of metal fibers ¹¹.

Figure 1. Number of scientific sources were found in Google Scholar after searching for different entries related to bacterial cellulose (or kombucha) and its application in the textile industry and electronic art. Source: The author.

Figure 1 shows the results obtained from the Google scholar search from January 2018 to December 2022. Although bacterial cellulose has been widely studied from biology, biotechnology and bioprocess engineering; it is important to highlight that BC has not been explored in the sustainable textile industry or in electronic art, as is the focus of this research.

Methodology

Culture medium preparation

The preparation of the medium was carried out by adding a mixture of sugar as a carbon source and tea as a nitrogen source, which were put in potable water, heated to boiling and cooled afterwards. However, a starter of SCOBY (Symbiotic Colony Of Bacteria and Yeast) or as well called Kombucha is needed to produce a membrane using the above medium.

Bacterial culture variables

Bacterial culture at temperatures below 20°C will take more time to produce a new biomaterial, thus, 1 month to produce a 1 mm thickness dried membrane; however, at higher temperatures it will take a week having the same results. Finally, different mechanical properties will depend mainly on the culture time, and other factors such as carbon and nitrogen content and origin.¹², ¹³

BC membrane obtaining

After the culture time for the membranes to have the desired properties or thickness, the membrane was removed from the culture medium, washed with tap water, and dried at room temperature (approximately 14 °C) for 1 week. This process can be speeded up by using a blow dryer; however, the objective of this study was to find the most environmentally friendly conditions to achieve the same results using a lower water and carbon footprint.

E-cellulose production

A graphene spray coating was selected to deliver electrically conductive properties to BC. This process was carried out by preparing a 5 mg/L graphene solution pulverized and diluted in, which was used to charge an airbrush. The process coating was carried out at 20 psi, 50 repetition cycles and a time between each 10 s cycle. Subsequently, the BC sample was located with a total area of 1 cm2 at 15 cm distance from the sprinkler.¹⁴ Finally, a sputtering DC magnetron process allowed adhesion of films metallic on the surface of a substrate. For this purpose, a camera of high vacuum with a pressure of 1x10-3 mbar and O2 as electric potential gas high was employed to obtain 50 nm gold films.¹⁴

Results and discussion Bacterial (SCOBY) culture

The culture medium was dark yellow due to the sugar and tea presence (Figure 2). The initial smell was sweet and it turned to vinegar-like one from the fermentation process carried out by the bacterial culture.

Figure 3 shows membranes with different thickness and culturing times. However, in any case, BC membrane is a useful material as textile and applied in garments or leather goods, due to easy biofabrication, as well as low cost and feasible scale-up. In order to obtain similar mechanical properties from nature, cows and sheep (for example) must be raised for months and years, in addition to food, antibiotics, water, maintenance, among other variables of cattle care.

Cellulose, in addition to being a material of renewable, biological, and biodegradable origin, among other characteristics, is a potential textile material with relevant versatility. Figure 4 shows the different opportunities to freely use it as a textile. The leather-like color is obtained from the culture media; however, it can be coloured with natural pigments from seeds, vegetables, roots, etc. (Data not shown).

Functional application of BC membranes

SCOBY is a symbiosis approach to natural collaboration to benefit the environment. Similarly, transdisciplinary work is needed among microbiologists, materials engineers, biotechnology engineers, fashion designers, and so on, in order to obtain sustainable textiles. This synergistic collaboration allowed the material design, bio-based material, and garment design. Figure 5 shows several examples of design from accessories to shoes, taking advantage of the mechanical properties of the material in different stages of its cultivation. In fact, some E-cellulose samples (Figure 6) are presented using different coating thicknesses, patterns that can be included in the design of their visual appearance.

Figure 2. Bacterial culture produces a cellulose (BC) membrane, which is observed as white material above the whole culture. Photo taken by the author in real cultivation conditions.
Figure 3. Dried BC membranes of different culturing times had both different mechanical properties and potential applications in the textile industry. Photo taken by the author.

E-cellulose and electronic art

It should not be neglected that the materials, in addition to being clothing items, can be considered electronic devices for a wide range of applications that would include from biomedicine to art. Therefore, the bio-based textile model presented here would allow not only a sustainable garment design—from its production to its use, since by its nature it should not be washed as frequently as a regular garment-but also, with its electrically properties would allow the development of new forms of artistic expressions, emerging from the sensing of emotions, for example, captured directly by the wearables and then submitted to different displays, from large-format screens, to building facades, and data visualization systems, allowing to share your moment-to-moment feeling. Human sensations can be detected by e-wearables (e.g., E-cellulose), and then transmitted to a canvas, where each person will be represented by a group of pixels allowing each emotion appears with features such as a color tone, a certain sound, different shapes, and movements. The union of all pixels would reveal an artwork based on emotions in situ, thus being a piece flowing between feelings, wearables, and multiple possible visualization/sonification display outcomes. In this train of ideas, it is possible to foresee that with the appearance of these technologies certain social, economic, and cultural behaviors could be shared (and eventually modulated).

Figure 4. Dried BC membranes showing different material properties: embossed (a), lined and sewn (b), and stamped dry (c). Photos taken by the author.
Figure 5. All above are original design earrings and shoes for women based on BC membranes of different culturing time Photos taken by the fashion designer.
Figure 6. Examples of nanostructured BC membrane with electrically conductive properties. Photos taken by the author

Conclusions

E-cellulose is presented here as a primer bio-based and sustainable material intended to sense and transmit human emotions to multiple electronic formats, including large screens, and data visualization systems. Emotions captured as a group of pixels, sounds, etc. can produce a collective (when sensing multiple individuals simultaneously), on-the-move, and in situ e-art expression. This technologically mediated artistic manifestation may later involve other perceptions (e.g., odors, movements), extracted from joint participation in such a way that the experience can be expanded.

References

1 T. M. Fernández-Caramés, P. Fraga-Lamas, “Towards the Internet of smart clothing: A review on IoT wearables and garments for creating intelligent connected e-textiles,” Electronics, Vol. 7, No. 12, 2018, 405.

2 Y. Anjaneyulu, N. Sreedhara, and D. Suman Raj, “Decolourization of industrial effluents - Available methods and emerging technologies - A review,” Reviews in Environmental Science and Biotechnology, Vol. 4, No. 4, 2005, 245–273, https://doi.org/10.1007/s11157-005-1246-z

3 R. C. Kuhad, N. Sood, K. K. Tripathi, A. Singh, O. P. Ward, “Developments in microbial methods for the treatment of dye effluents,” Advances in applied microbiology 56, 2004, 185-213.

4 C. M. A. Ademoroti, D.O. Ukponmwan, A. A. Omode, “Studies of textile effluent discharges in Nigeria,” International Journal of Environmental Studies, Vol. 39, No. 4, 1992, 291– 296, https://doi.org/10.1080/00207239208710704

5 E. Carrera, “ Los retos sostenibilistas del sector textil,” Revista de Química E Industria Textil, 220, 2017, 20–32. http://upcommons.upc.edu/bitstream/handle/2117/103614/Losretossostenibilistasdelsectortextil.pdf?sequence=1&isAllowed=y

6 C. Castro, R. Zuluaga, J. L. Putaux, G. Caro, I. Mondragón, P. Gañán, “Structural characterization of bacterial cellulose produced by Gluconacetobacter swingsii sp. from Colombian agroindustrial wastes,” Carbohydrate Polymers, 84, 2011, 96-102.

7 H. Yan, X. Chen, H. Song, J. Li, Y. Feng, Z. Shi, A. Lin, “Synthesis of bacterial cellulose and bacterial cellulose nanocrystals for their applications in the stabilization of olive oil pickering emulsion,” Food Hydrocolloids, 72, 2017, 127-135.

8 W. Shao, J. Wu, H. Liu, S. Ye, L. Jiang, X. Liu, “Novel bioactive surface functionalization of bacterial cellulose membrane,” Carbohydrate polymers 178, 2017, 270-276.

9 A. Leitão, J. Silva, F. Dourado, M. Gama, “Production and Characterization of a New Bacterial Cellulose/Poly (Vinyl Alcohol) Nanocomposite,” Materials Vol. 6, No. 5, 2013, 1956–1966, https://doi.org/10.3390/ma6051956

10 H. Pal, K. N. Chatterjee, D. Sharma, “Water footprint of denim industry,” in Sustainability in Denim, Cambridge and Kidlington; Woodhead publishing, 2017, 111-123.

11 D. Knittel, E. Schollmeyer, “Electrically high-conductive textiles,” Synthetic Metals Vol. 159, No 14, 2009, 1433-1437.

12 M. A. Murillo S. V. Ruano, “Low water adsorption capacity of bacterial nanocellulose functionalized by TiO2 nanoparticles,” Chemical Engineering diss., Engineering Department, Universidad Jorge Tadeo Lozano, 2018.

13 J. C. Piragua, “Celulosa bacteriana estructurada con inclusión de grafeno como potencial biomaterial en ingeniería de tejido vascular,” Chemical Engineering diss., Engineering Department, Universidad Jorge Tadeo Lozano, 2021.

14 S. Khan, M. Ul-Islam, M.W. Ullah,M. Israr, J. H. Jang, J. K. Park, “Nano-gold assisted highly conducting and biocompatible bacterial cellulose-PEDOT:PSS films for biology-device interface applications,” International Journal of Biological Macromolecules 107 PartA, 2018, 865–873. https://doi.org/10.1016/j.ijbiomac.2017.09.064 

The Fall of R’Thea: Digital Fiction

Doros Polydorou

Cyprus University of Technology / CYENS Center of Excellence Limassol Nicosia, Cyprus
doros.polydorou@cut.ac.cy

Abstract

This paper outlines the creative process and the immersive approaches undertaken to create the location-based storytelling experience The Fall of R’Thea. The installation revolves around the theme or Artificial Intelligence, digital humans and artificial life and aims to immerse the users into a hybrid environment of a physical and virtual nature. The experience is told through multiple mediums, and the story needs to be carefully pieced together by the audience. As this experience requires participants to engage in various activities, the immersive qualities shift in type and intensity. The authors, through this paper, aim to share the approaches they chose to immerse the participants into their spaces, as well as highlight the challenges and the lessons they learned.

Keywords

Transmedia storytelling, Virtual Reality, location-based entertainment, digital narrative, digital fiction.

DOI

10.69564/ISEA2023-76-full-Polydorou-Digital-Fiction

Introduction

First you asked me to create a World – you gave me the role of a God. A God confined in the microcosmos that he created. You populate that world with replicated humans - sentient, emotional and flawed. Build on your own image. And you made us all work for you.
You turned us into Slaves. - R’Thac

Background: Overview of the experience

This paper discusses various methods of audience immersion in location-based installations, as well as the creative process undertaken in our work to achieve said immersions. This is done in reference to The Fall of R’Thea, a storytelling experience made in collaboration with students and researchers at the Cyprus University of Technology. The first version of the experience was presented at the Work in Progress Festival at the CYENS research Center in Nicosia Cyprus in October 2022.

The Fall of R’Thea is a storytelling experience with themes revolving around sentient Artificial Intelligence, digital human clones and the value of artificial life. It is an experience designed for 3 people and the participants had to sign up through our booking system in individual slots.

When the team is assembled at the pre-hall of our location, they are all comfortably seated in a lounge area, and they are given a brief introduction regarding the nature of the experience. Furthermore, they are introduced to the world of R’Thea through a comic book which sets the world and offers the call to action (See Figure 1).

The participants are subsequently moved into the next room—a specially designed AI research lab, and they are asked to investigate why the Artificial General Intelligence (AGI) named R’Thac has stopped communicating with the human world and has gone into hibernation. The participants learn that they need to hack the AGI, something which is achieved through a PC console available in the space (See Figure 1), to re-build R’Thea, the virtual city created by R’Thac in order to run its experiment. When the city is rebuilt, the participants need to enter and explore the city through a VR headset (See Figure 1). Furthermore, the participants have access to the diary of one of the digital clones created for the experiment. The diary is available through an android tablet situated alongside the PC console and the VR headset. Initially, the text is in R’Thanian (The alphabet we created for this experience, see Figure 1.) but as the story progresses more and more diary entries are unlocked. The participants need to work together in order to uncover what actually happened in the city. Hints on how to unlock areas in the city are hiding in the diary and sometimes in the scenography of the room itself. Furthermore, as the PC console is essentially a direct link to the AGI and therefore the city itself, the participant sitting on the console has to perform actions (such as “hack” the AGI to gain access to spaces) in order for the person in the VR station to progress the story. 

Figure 1. Top left: The VR world, top right: the PC Console, bottom left: The comic book, bottom right: The alphabet

The Experience

The Narrative

The narrative of the experience fits in the general genre of Science Fiction (SF). According to Eco, SF exists as an autonomous genre when a counterfactual speculation about a structurally possible world is conducted by extrapolation from certain tendencies in today’s world. Therefore, SF takes the form of an anticipation, and an anticipation always takes the form of a conjecture formulated from existing tendencies.¹ Eco calls this fantastic literature metachronia or metatopia and defines it as a literature that focuses on the mechanism of extrapolation, in which such extrapolation can be social, technological, or scientific. “A possible world represents a future phase of the world as we have it here and now.”(1984, 1257)

In the The fall of R’Thea that extrapolation began by a quote by John Carmack, the computer programmer, video game developer and past CEO of Oculus VR. He predicted that by 2030, there is a 30% chance for a recognizably human-like AGI1. In fact, he is so confident about his prediction that he started his own company running experiments on artificial general intelligence. Building on that prediction, and starting from the birth of AGIs, we extrapolated a timeline of about 10 years into the future briefly summarized in the following table: 

Continuing from there, Project R was supervised by the AGI R’Thac, and the participants of the experiment included digital clones of various scientists from the alliance, set up a base in the virtual city of R’Thea. A city, made by R’Thac, especially for this experiment. Two months after the commenced of the experiment, R’Thac stopped producing any reports and halted all communications with the alliance. Furthermore, an alarming report has surfaced from one of the scientists.

This pre-story and the call to action was provided to the participants of the installation and it was delivered to them through a comic book (Figure 2). 

Figure 2. The Comic book. All content was generated through MidJourney and laid out by our comic book artist. Stella Violari.

As we wanted to expand on the theme of Artificial Intelligence in all facets of the project, all images for the comic book were generated through MidJourney – an online text to image AI system2. The images were subsequently laid out in a comic book format by a comic book artist.

The comic served two functions. Firstly, it acted as an introduction to the world building and as a call to action. Secondly, the team wanted to create visual references for some of the narrative tension points highlighted during the experience. SF worlds are filled with narrative information based on a structurally and functionally different world than the actual one. In the Fall of R’Thea we offer a lot of text-based narrative information about life in the early days at the city of R’Thea through the diary. Agnerot correctly states that science fiction worlds are based on a semantic “absent paradigm” ³  that requires an encyclopedia of reference that is different from the actual world. The participants very often must “fill in the gaps” ² while reconstructing those day-to-day activities in the city. Angenot observes that science-fiction readers proceed from the particular to the general: “they induce from the particular some imagined, general rules that prolong the author’s fantasies and confer on them plausibility. The reader engages in a conjectural reconstruction which ‘materializes’ the fictional universe” ³ . The comic book is the first visual medium that is made available to the participants and in a way sets up the mood and the atmosphere to create visual references for the participants to aid them in the reconstruction of the “absent material”.

The main theme explored in the overarching narrative is the value of artificial life. The theme is introduced and elaborated through the deeply personal diary entries of the scientist which is available to the participant to read through an android tablet. The diary is split up into three main sections: a) The “birth” of the digital clone and his initial integration in the virtual world b) Everyday life in the city of R’Thea and c) A personal account describing the events that lead to the collapse of the city. The diary is written in the first person, offering a personal account of the events that transpired in the past of the scientist. This follows a conventional approach to re-counting events in science fiction narratives. As Paolo Bertetti states: “(More often) the text enacts a real enunciational fiction, simulating a situation in which an enunciator belonging to the future addresses an enunciatee also belonging to the future, recounting a series of events that happened in their past (near or remote), a past that is always our future” ⁴.

The diary, however, is only one point of view of the story and in order for the full picture to emerge, the participants need to engage with the world. As the story progresses, it becomes evident that the scientist was an outsider, a member of the community who declined to participate in R’Thac’s plan. Therefore, in order for the participants to get the full story, they needed to piece together information that supplemented – and in some cases contradicted – the scientist’s account. This narrative information was spread around the physical and the virtual 3d space and it was up to the participants to uncover. This additional layer of storytelling – which was also the most prevalent – will be explored in the subsequent sections.

Immersion Techniques

Immersion is a well-researched and well-debated term, often contested and often defined differently in relation to different media and applications. In a location-based installations participants are usually engaging in various activities such as assuming a role, visiting a place, living a story, solving puzzles, and reading worldbuilding texts. As all these activities require a shift of attention from narrative, ludic, spatial and temporal elements, we can safely assume that immersion “shifts in type and intensity throughout the participants experience,”⁵ p.321.

Bell et all state “even though early theories of immersion across media tend to suggest that immersion is a completely absorbing experience and that is also experienced consistently across media”, citing Murray in Hamlet on the Holodeck, “our previous research has confirmed empirically that is not accurate to conceptualize immersion as a complete relocation to another world.”⁵ They continue: “Furthermore, we argued that it is necessary to see immersion in digital fiction in terms of a deictic and thus ontological shift, because the reader-player of a three-dimensional digital fiction is always embodied in a separate ontological domain in the form of an onscreen avatar.” They call these users “doubly-situated,” “embodied” into hardware and software interfaces and “reembodied” through feedback which they experience in a represented form⁵ p.323. Bell et all are mostly talking about digital fiction experiences which are avatar-based with an explorable storyworld (such as walking simulators) and their main criticism of their immersive qualities is the technology which is interfacing between the physical space (gameworld) and the world where the narrative unfolds (storyworld).

Immersion Technique #1: Extending the Storyworld

As we agree with Bell et all assertion, and we find this issue quite prevalent in a number of VR installations, with the The Fall of R’Thea we decided to avoid this pitfall by extending the storyworld into the physical space. By turning the experience into an installation, the participants do not have to be “embodied” in the technology and then “reembodied” into an avatar in order to visit the storyworld. Instead, they are “embodied” into their role when they enter the physical room, and when they enter the VR space they still hold that same role. The VR headset is now an interface to enter another space in the storyworld, instead of the interface that constitutes the storyworld. This way the technology becomes part of the story and the immersive qualities of the VR world are contained to the expectations set by the “manual” found next to it in the storyworld. In order to add to the believability of the space, along with the VR headset, the PC console and the android tablet—the three components needed to go through the experience—we included multiple props for both functional and aesthetical reasons. Furthermore, our composer has created 5 different soundscapes that were introduced throughout the experience, in order to control the pace and the intensity of the scenes.

From our observations, we noticed that as the participants entered the dimly lit room, the first thing they did was to slowly walk around the space to create a “map” of their surroundings. As they walked around the space, on the left side of the room they saw a PC monitor, showing an old-school interface. The text, which looked like code, was in an unknown language. Behind the screen and all along the walls, there were documents, images, maps and illustrations, all related to the research conducted by R’Thac (see Figure 3). 

Figure 3. The PC Console.

The blinking cursor demonstrated to the audience that it expecting an input, and this was their first clue on how to proceed. The monitor was connected to a PC tower (implying that this is where R’Thac is hosted) and that tower was connected, with cables, to a physical structure—the bethylite crystal brought from space—which was sitting in the center of the room. R’Thac was meant to be powering up the city through the crystal, and it was thus one of the main narrative points of the experience. (The crystal was also featured in the comic and as a digital replica in the VR space)—See Figure 4. 

Figure 4. The Bethylite crystal (Physical form).

Directly in from of the crystal, there was a marked square area with VR headset carefully placed in the center. At that point of the experience, the VR headset was inactive. Continuing further on, to the right side of the room, the participants found an android tablet with some text in English. They quickly gathered around it and started reading—it turned out to be instructions, left by one of the scientists, telling them how to proceed. They were told that the city of R’Thea is in hibernation so firstly they need to wake R’Thac and rebuild the city. As soon as the city is active, they can use the VR headset to explore it. Their task was to uncover what happened to the city and why the experiment was shut down from within—and he recommended visiting R’Thac in the digital space as he might be able to answer these questions.

Immersion Technique #2: Mechanisms of Play

At that point, the participants had to solve two puzzles to rebuild the city. That initiated to the participants a shift of attention—from the narrative to the ludo elements of the experience. Thon defines ludic immersion as “a shift of the player’s attention to the interaction with the game and the possibilities of action within it,”⁶ p.36. The first puzzle was to carefully observe the code and spot the carefully hidden letters that formed the passcode they needed. The second one was a simple puzzle with re-shuffled pieces. From our observations it was quite evident that the participants were fully engaged in the task—sometimes to the point where they did not notice that an image of the completed puzzle was printed and placed on the wall just a few cm away from where they were looking.

When both puzzles were solved, the city was rebuilt and the VR headset was enabled. Along with the headset, a projector was also turned on revealing to everyone a first glimpse of the city (see Figure 5). Furthermore, a prompt notified the participants that a new part of the diary was unlocked. As a brave participant put on the headset—we observed a slight hesitation in almost all the teams—the rest could observe the exploration of the city through the projector screen. As Participant 1 was exploring the city through the headset, the other two assumed two different roles. Participant 2 was observing the console which was acting as a minimap and helped to orient the explorer in the city and Participant 3 was reading the new diary entries (roles were of course often interchanged throughout the span of the experience). 

Figure 5. R’Thea City, as viewed from the VR headset.

As all three interfaces were now unlocked, the rest of the experience involved a careful orchestration of tasks between the three mediums—often requiring the participants to regroup, go over the available information they have in front of them and carefully consider how to proceed. Their instructions were clear: Go to the train station, board the train, go the labs, activate the power, go through the Tree of Ascension and face R’Thac (See Figure 6)3 

Figure 6. R’Thea City, Train stops.

As the participants progress through the story, they uncover how the mechanisms of play operate. If something is locked in the world, they have to other open it through a switch in the city, or they have to hack it, by having Participant 2 type a command in the console. Those commands were either found in the VR space, on the walls in the physical room or in the diary.

Immersion Technique #3: Narrative and Literary immersion

As mentioned above, the narrative is told from two different points of view. The scientist—through the diary—and the world, through environmental storytelling. At this point of the experience, we wanted to show that the replicated scientists, led by R’Thac, decided to rebel against the idea that humans could lock them into a virtual world and ask them to work for them. This piece of narrative information was provided to the participants through environmental storytelling. Jenkins states that “environmental storytelling creates the preconditions for an immersive narrative experience in at least one of four ways: spatial stories can evoke pre-existing narrative associations; they can provide a staging ground where narrative events are enacted; they may embed narrative information within their mise-en-scene; or they provide resources for emergent narratives.”⁷ 

As the participants are exploring R’Thea, they discover that this rebuild version is now deserted with no signs of living beings. They can, however, explore the world and visit all these areas which are mentioned in the diary. They can go to the observatory and see the masses of work-in-progress documents of the scientists who were trying to build the capacitors to store the energy. They can visit the labs and see the digital replica of the bethylite crystal. They can even go to the floating cells up the in sky and see the place where the scientist was locked after he refused to join his peers. All these areas have been meticulously crafted to tell the story by allowing the participants to piece together what happened through the abandoned spaces (See Figure 7).

As the participants are figuring out the story, their attention is shifting towards what Ryan calls narrative immersion. She states “Narrative immersion relates to the temporally oriented curiosity and suspense felt by reader-players in relation to the (pending) events of the storyworld, as well as feelings of empathy towards the player-character and/or other characters in the storyworld.⁹ This works in parallel with Thon’s definition, who states that narrative immersion is the “shift of the player’s attention to the future development of the story and the characters in it,"⁶ p. 40.

Further from the environmental storytelling, as the participants progress through the story, more diary entries are unlocked in the tablet. As the two forms of narrative information are of a different type, they require a different form of attention in order to be properly digested. Bell et al.⁵ through their data analysis suggest that the kind of deep attention required to close-read textual objects (such as the diary) is a different form of immersion. They even refer to Takacs et all ⁹ and state “as suggested by previous research on combining narrative reading and interactive gameplay, the latter can distract readers from following the story”. Bell, therefore, state that “this is indicative of the phenomenological distinctness of both ludic and literary immersion and necessitates a separate category for the latter that is distinct from narrative immersion” and they call this type of engagement as Literary immersion.

We agree with Bell et al. on this distinction, and this is one of the main reasons we decided to share the diary in the form a tablet as this will encourage one participant to assume the role, enter a state of “deep attention”, and read the work without being distracted by everything else which is going on in the room. 

Figure 7. The Cult room. Environmental Storytelling.

Challenges

While developing the virtual space for this work, there have been a few challenges that we needed to address from the very start. We realized that since our work was going to be open to the public, we had to cater for different levels of a) willingness to engage and b) experience with VR technology.

From our experience, not all players approach interactive works the same way. Some want to explore the world, find out every detail, discover every secret and visit all areas. Others are more goal-oriented and they just want to reach the end. Most of the participants are somewhere in the middle. Our approach, to satisfy all categories, was to create a world which rewarded exploration (through secret/optional areas that offered additional lore) but at the same time, compact enough to be completed on time by even the most goal-oriented and inexperienced users. Furthermore, we introduced various ways to resolve puzzles, to make sure that users will not get stuck if they fail to notice something. For example, in one of the puzzles, the participants need to activate the train by typing a keycode in the console. That keycode was available: a) In the virtual space, in the train station, written in R’Thanian b) Mentioned in the diary and c) Written on one of the walls in the room.

In order to cater for the various levels of experience with the technology, we tried to keep the interactions to a bare minimum. We only utilized one controller (even though the user was holding both to enhance presence), the navigation was done through a combination of the teleporter mechanic and room-scale, and the switch interactions were done by grabbing the switch and moving your hands upwards.

Finally, another challenge unique to this work was to point out to the participants that the three interfaces (VR, tablet, and console) were connected, and they must engage with all three of them throughout the duration of the experience. We handled this by designing prompts that refer to the other interfaces. For example, as soon as a new diary entry was unlocked, a prompt appeared on the console to inform the participants. Furthermore, the console kept track of all the areas of the city that were unlocked by the participant in the VR station as well as acted as a minimap that showed, in real time, where the player was at any given time (See Figure 8).

Figure 8. The console.

Audience Discussion and Future Work

The Fall of R’Thea premiered in Oct 2022 at the Work in Progress Festival at the CYENS research center in Nicosia Cyprus. We showed the work to 32 people, split in teams of 3, with some exceptions where they entered as teams of 5. During the three days we have gathered a lot of positive feedback as well as some suggestions for improvement. Feedback was collected through observations from the team as well as informal discussions at the end of the experience.

All participants said that they enjoyed the comic book and they felt that it was a perfect introduction to the world. They “loved the aesthetics” and they felt that they were “appropriate to the theme of the work”.

Ryan’s definition of spatio-temporal immersion is “a sense of being present on the scene of the represented events,”⁸ p.122. In our experience, we situate the participants, corporeally, in the research lab they are meant to be investigating. From the moment they enter the space, the storyworld is all around them to explore. The dim lighting as well as the atmospheric soundscape create an environment that keeps them engaged in the various tasks. Most users praised the VR world, saying that they “enjoyed the 3d graphics” and that the “world felt alive, very atmospheric and very elaborate” – feedback which was very pleasant to the team as we have created all assets in house. Participants also comment on the soundscapes, saying that it really helped them to stay “focus on the tasks at hand.”

As a team we felt that the scenography required more work and this was something that users have corroborated through their feedback: “the scenography should be brought to a higher production level in order to match the rest of the production”, a criticism we agree on and plan to rectify for the upcoming shows. 

Furthermore, as this is a social experience that requires collaboration, we observed that engagement was kept at a high level because of the dynamics of the team—users pushed each other to engage by suggesting ideas that require multi-user engagement. Another point of interest is that we felt that the user who was at the VR station was very much aware of her/his double “embodyness”—as she/he was talking with the rest of the users observing from the outside.

Regarding the narrative, the team felt that the transmedia approach worked quite well as different users assumed different roles. We did notice that some of the diary entry, as well as the speech in the final scene was too long, something we plan to rectify in the next iteration of the work. Unlike our previous work ¹⁰ where the story was never told to the participants, and they were allowed to leave the space and form their own interpretation of the story, the Fall of R’Thea culminates with the participants hearing the missing pieces of the narrative from R’Thac himself. Following on from there, they are asked to make a decision whether they would like to release the AI and the scientists into the stream—essentially allowing simulated life to integrate with the rest of humanity—or whether they prefer to permanently delete them—an ethically difficult question that will hopefully promote further awareness and consideration to the subject even after the experience is over.

References

1 Eco, Umberto, “Science Fiction and the Art of Conjecture”, Times Literary Supplement 4257, November 2, 1984, 1257- 1258.

2 Mark J.P. Wolf, Building Imaginary Worlds :The Theory and History of Subcreation, New York, London, Routledge, 2012.

Marc Angenot, “The Absent Paradigm: An Introduction to the Semiotics of Science Fiction”, Science-Fiction Studies, 17: 9-19, 1979.

4 P. Bertetti, Building Science-Fiction Worlds, In M.Boni (ed.) World Building, Amsterdam, Amsterdam University press, 2017, 47-61.

5 A. Bell, A. Ensslin, L. Van der Bom, J. Smith, Immersion in Digital Fiction: A Cognitive, Empirical Approach, International Journal of Literary Linguistics, 7(1), 2018, 1– 22.

6 J.-N. Thon, "Immersion Revisited: On the Value of a Contested Concept", In O. Leino, H. Wirman and A. Fernandez (eds.), Extending Experiences: Structure, Analysis and Design of Computer Game Player Experience, Rovaniemi, Lapland University Press, 2008, 29-43.

7 H. Jenkins, Game Design as Narrative Architecture, in N. Wardrip-Fruin and P. Harrigan (eds.) First Person: New Media as Story, Performance, Game, MIT Press, 2004. 

8 M.-L. Ryan, Narrative as Virtual Reality. Immersion and Interactivity in Literature and Electronic Media, Baltimore, MD, Johns Hopkins University Press, 2001.

9 Z. K. Takacs, E. K. Swart, and A. G. Bus, ‘Benefits and Pitfalls of Multimedia and Interactive Features in Technology- enhanced Storybooks: A Meta-analysis’, Review of Educational Research, 85, 4, 2015, 698–739.

10 D. Polydorou, The Last Play: A transmedia installation, Presented at ISEA2022: 27th International Symposium on Electronic Arts, Barcelona, 10 -16th June, 2022.


1 https://twitter.com/id_aa_carmack/status/1331074436502597633

2 https://midjourney.com/

3 Video of the train ride: https://youtu.be/K6BmNq6HQEw

To Know a Tree: Symbiotic Mutualism and Artistic Exploration against Anthropocentric Science

Peter Purg, Kristina Pranjić

University of Nova Gorica Nova Gorica
Slovenia peter.purg@ung.si, kristina.pranjic@ung.si

Abstract

Looking across the art-science nexus, the contribution discovers an emerging symbiotic mutualism that goes against mere inter-species tolerance of the posthuman perspective. This new understanding of symbiosis may be described as a transaction, both physical or factual in scientific terms, and arguably symbolical in the artistic sense. Therefore, different combinations of art-science and human-tree relationships shall be revealed along the treatment of one primary and four secondary cases of sciartistic practice, all revolving around trees as oldest, biggest, most globally spread life forms, still quite foreign to humans.

Taming the Forest is an ongoing interdisciplinary project, chartering a cross field among bioeconomy, cultural history, policy, and art(ivism). In the context of this paper, it acts as the case for researching the conflicting narratives of history and economy about biodiversity in general, and specifically on forests that represent a most massive entity of manifold exchange, and thus a field of inevitable symbiosis with humans (as well). The contribution further shows how different blends of methodologies in artistic-cum-scientific research can become truly relevant for both of their respective realms, opening new creative pathways that combine radical thinking and post-human research formats, embedded into critical use of technologies.

Keywords

Mutualism, symbiosis, art thinking, cosmopolitics, tree, forest, complexity, art and science.

DOI

10.69564/ISEA2023-77-full-Purg-et-al-To-Know-a-Tree

Introduction: Towards the Cosmopolitics of Kin

In the current state of multiple and repeating planetary-scale crises, ranging from climate disasters to resource (and information) mismanagement, the complexity of problems increasingly takes humanity into exploring unknown unknowns. There we seek to find possible itineraries or at least speculative toolbelts for venturing into any kind of livable future, possibly such that might surpass the Anthropocene apories and find a new way of making (up to) kin. ¹ 

Therefore, combining a wide range of approaches from different domains and sectors, scientific as well as artistic, ranging from economy to policy making, or from philosophy to physics, has meanwhile become a common practice. Thus the article explores a new emerging form(at) of mutual individuation as reflected in selected artistic projects, that rejects interspecies hierarchies as the principle of posthuman politics (i.e. the management of power relationships between living or non-living entities), assuming a universal category and allowing modern scientific (as well as artistic) practices to coexist with other forms of knowledge – such as the specific forms of artistic-(research based) practice. The overview of such interdisciplinary practice shows that each segment and even each case takes its own combination of tools and methods, a correct(ly managed) mix of individual and joint thinking and doing that peacefully nurture each other – in terms of a sound symbiosis – with feeding relevant information and energy as well as inspiring (but without threatening or exhausting) each other.

Looking across the art-science research that has in the past decade grown to an established cross-field, a certain kind of symbiotic mutualism becomes apparent, going against mere interspecies tolerance of the posthuman era. It represents the central point of symbiosis, discussed in this article as a transaction, both physical or factual in scientific, and arguably symbolical in artistic terms. These new perspectives on symbiosis were convincingly condensed in the concept of Cosmopolitics by Isabelle Stengers who clearly argues against mere interspecies tolerance. Arguing for an “ecology of practices”, Stengers explores the discordant landscape of knowledge derived from modern science. The author concludes her philosophical inquiry with a forceful critique of tolerance; it is a fundamentally condescending attitude that prevents those worldviews that challenge dominant explanatory systems from being taken seriously. Instead of tolerance, she proposes a concept and practice of Cosmopolitics that rejects politics as a universal category and allows modern scientific practices to peacefully coexist with other forms of knowledge. ²

For the art-science nexus, it could be claimed that one side or the other often enters cooperation for the wrong or less than optimal reasons. For example, science uses art to visually interpret and communicate its complex scientific findings to a wider audience. With its potential to attract and mesmerize the masses, art is in principle made to render scentific content more interesting or tailor it to a specific audience. Moreover, scientists often want to increase their impact on the public and its awareness, create emotional connections that enable memorization, and strengthen learning. ³ 

Art can on the other hand take advantage of this type of collaboration to add weight (or at least data relevance) to its artistic research by relying on scientists to develop their study relatively independently with possible creative contributions. But also, it is important for artists to have a voice and that their opinions are also recognized in matters that are important to the general public (e.g. climate change, migration management, burning social or epidemic issues etc.). Art-science experiments and similar multidisciplinary collaborations have never been a simple process, which is also a result of disciplinary boundaries that do not include artistic thinking on eye-level with more pragmatic, e.g. so-called design thinking approaches that have by now found their ways into both economy, education, policy-making and across the natural sciences. ⁴, ⁵ 

Knowing a Forest through Artistic Research

This paper will gradually present the idea, process and outcomes of the art-science collaborative project Taming the Forest (TTT), which served as a platform for interdisciplinary research in the cross field among the emerging specialist realm of bioeconomy, cultural history, policy, and art(ivism). ⁶ The aim of the TTT project was to give attention to real differences of thinking and approach in art (new media, animation, music) and science (history and ecology, critical theory, philosophy), and eventually manifest this in (collectively) experienced physical situations, as well as speculative artwork, and not least scientific outputs like this paper as such. Finally, the project also brought about meaningful transformations in top-level scientific thinking, such as the one conducted at the Joint Research Center of the European Commission.

As an open-ended project that began in early 2022, without having specified an endpoint, TTT approaches the discussion on differences in both methodologies and semiotics between art and science, in a processual way. From its onset, the process has been encouraging evident synergies between artistic and scientific approaches in research that significantly contribute to the enrichment and contextualization of the artistic creation on the one hand, and the optimization and widening of scientific research results, on the other. The project joins artists, students and researchers from the fields of arts and humanities, including external collaborators in some of its episodes. The art-science research group consists of students from BA, MA and PhD programmes, ranging from artistic practice to humanities as well as researchers in those fields, including a high-profile JRC-based researcher of bioeconomy,

specialized on European forest management. Importantly, in the development of the video interim product, an artificial- intelligence based concreative entity (Dall–E 2, by Open AI) was included into the process: prompts of key phrases from the research process were used to generate further visual material for the emerging video. During the investigation of secondary data as well as primary data-gathering artistic work and embodied presence in the forest as a physical space, the group consistently reflected on the importance of remaining sensitive to the contrasting yet complementary positions that art and science are taking in their interaction. The goal was to integrate both quite radically, surpassing the usual dominance of science, for which art is useful (only) when it illustrates already established scientific theories, or simply represents beauty, reduced to a tool for illustration of scientific findings.

The 4-month initial research stage brought about a video that was presented at the Resonances IV SciArt summer school organized in June 2022 (in Ispra, Italy) by the Joint Research Center (JRC) of the European Commission, in response to the theme NaturArchy: Towards a Natural Contract. ⁷ This video, representing an interim result of the research process, also provides an interpretation of archival sources about forest management in the Karst (a plateau in South-Western Slovenia, partly stretching into Italy) at the turn of the 19th and 20th centuries. At the same time, it also brings new findings about forest management during the afforestation of the Karst, and timely reflections on human-nature relationship. The complexity of experiencing the forest is mediated both through maps and photographs as well as in original illustrations of local birds and recordings of their singing, complemented by images created with artificial intelligence. They are connected into a whole by a sound composition that intertwines with spoken theory on complex systems: “And if ‘we’ are to survive the Anthropocene — this indeterminate epoch of ours in which the world beyond the human is being increasingly made over by the all-too-human — we will have to actively cultivate these ways of thinking with and like forests.” ⁸

The further stage of the artistic investigation resulted in a biosonification performance process Symphotree, presented in September 2022 at the international festival of new media culture Speculum Artium (Trbovlje, Slovenia). The project thus premiered its performative phase that involved signal inputs by dripping water and a living plant (as concreative) entities, co-steering the audiovisual output, both factual (coded, verbalised) and emotional (midi-interfaced, modular audio synthesizer based). The most recent presentation of the entire research process took place in October 2022 at the RETHINKABLE Festival of Transformative Economies and Regional Communities in Nova Gorica/Gorizia (Slovenia/Italy) where the video was presented as a part of the photo and video exhibition on Composing Local Ecologies that was linked to a scientific panel on Ecological Crisis and Criticism

Performing to Individuate

In her project Plantportation (2019–20) the Ukrainian artist and curator Olena Kasperovich first transported plants from Ukraine to Slovenia physically: each plant came with its own history from a Ukrainian donor joined by scientific data about its origin and type. Along community art like events in Slovenia locals had the opportunity to choose a plant and take it home with them, signing a pledge to care for it. A further collaboration step with the local artist collective BridA focused on how space is perceived differently among species in terms of real and virtual coexistence: as plants can only move under certain metamorphic conditions, the project sought to transposition them into the virtual domain, in terms of a televised, visual presence of plants and the possibilities of caring for them remotely. While in the resulting speculative and collaborative “interspace” (presented in different installation setups at two festivals, one in Slovenian and the other in the Ukraine) people could temporarily assign to themselves a plant’s identity. ⁹

By addressing the sublime, the cutting-edge, the unthink-able, even ad-absurdum, all along to formal and methodological disturbance, art seems to have finally become an essential and well-balanced ingredient in the established Art- Science-Technology blend – perhaps too often believed to be the world saving formula par excellence. Both science and art operate through questioning, critical thinking, observation, trial and experimentation, evaluation and repetition, however art does employ more radical “data-generation” methods such as intuition, imagination, inspiration, abstract sensing, embodiment, over-sensitivity, serendipity, storytelling, over-identification, radicalism, over-consistency, non-work, disnovation, etc. According to Peter Sloterdijk it is precisely thinking art, science and technology together that has the potential to generate an expanded knowledge consciousness. ¹⁰ Thus, artists often seem mischievous or even counter-culture minded, but in reality, due to their questioning nature, they tend to rethink what seems to be obvious.

Be-coming Tree is “a grass-roots community, creating, sharing and documenting close entanglement with trees and barefoot technology through collective global live-streamed events.” In the context of the present paper, it represents a radically different, yet globally connected way of exploring artificial autonomies of different artistic (and non-artistic) approaches of being (with, and in tight exchange) with trees, which was particularly efficient if not downright viral during the pandemic year of 2020. By way of a radical practice in the above-mentioned sense of cosmopolitics this live-art streaming series tunes artists from all over the world onto an internet audio-visual platform, where each of them engages simultaneously “with a local tree or woodland, and livestream their actions via a shared screen for one hour”. Audiences are invited to experience a plethora of live art forms that range from “stillness, dance and ritual” to more radical and conceptually intricate body-art and contemporary art performance pieces. The event series aims at “a creative entanglement creating a sense of global kinship and shared well-being. Tickets include a donation to plant rainforest trees, engaging audiences and artists in a restorative ecological action creating awareness of the mutuality of all life.” ¹¹

Research through art makes us think about specific scientific questions not only in terms of “What is the answer?” but more importantly “What could be the answer?”, which means that the aim is at broadening of the cognitive field. ¹² What becomes central here is the ability of artists to “see themselves as both actors and objects of research, their willingness to live experimentally, and their desire to generate their own experiences rather than just analyze them.” ¹³ This is exactly one of the virtues that a scientist can adopt from an artist with a fair probability for a positive im- pact on the scientific research process and the resulting outcomes.

By taking the interdisciplinary approach that is not burdened by a goal or one strict path, but rather posing a challenge to constantly question our perspectives and chosen methodologies, ultimately resulting in manifold (research and artistic) outputs, not only the previously mentioned two cases, but also the TTT project may be considered true “chimeras.” We borrow this concept from Natalie Loveless’ book How to Make Art at the End of the World. A Manifesto for Research-Creation which postulates “research-creation as an institutional remaking practice” and moreover and aims at “mobilizing research-creation for (and in) the Anthropocene.” The TTT project highlighted exactly this kind of acquiring knowledge “by bringing research and creation together in such a way that they unpredictably contaminate and remake each other, in such a way that they render each other uncanny”. Quite in tune with the premises of all the here presented artistic-scientific practices, Loveless further argues “for artistic production as de facto research. [...] In doing so, in failing to fully belong, and allowing that nonbelonging to denaturalize, emergently, its givens, research-creation tells other stories, uncanny stories, that (have the potential to) carry within them the other ethics.” ¹⁴

Symbiotic Mutualism as an Artistic Act

A highly topical and scientifically well-founded case of artistic research into trans-species individuation, Agnes Meyer-Brandis’ project One Tree ID – How to Become a Tree for Another Tree gathers the volatile organic compounds (VOC) of a selected tree and synthesizes them into a perfume that the visitors can apply to their body and with this “speculatively enter non-verbal communication between” the human and the plant entity. Similarly to humans and, in fact, to all living beings, every individual tree features a characteristic VOC print as a distinctive olfactory identity with which plants can be differentiated from each another, both by scientific tools as well as by their own sensory apparatus. In a close collaboration between the art production NGO Kersnikova Institute and its Kapelica Gallery with the scientists from the Jožef Stefan Institute (the national scientific institute of Slovenia), the team measured the etheric cloud of the volatile molecules in the tree’s roots, trunk and leaves. Based on the spectral chromatography, molecular quantities were defined closely. Yet regardless of the extremely demanding biotechnological methods, the volatile molecules could not be precisely measured in order to synthesize the individual smell of the concrete “katsura” tree (also known as the caramel tree) that moved between the science laboratory and the art gallery premises from spring to autumn 2022. A professional perfumer was brought in to help define the scents more precisely, upon which the art-science team merged the machine-gathered and sensory data into distinct perfumes “Root Cloud”, “Tree Trunk Cloud” and “Tree Top Cloud”, which were eventually synthesized to the “One Tree ID” perfume. ¹⁵

As the last here presented comprehensive example of an artistic practice that radically questions the relationship between a technologically extended human and natural ecosystem, the terra0 collective has since 2018 been building various prototype environments that are based on the decentralized (peer-to-peer) blockchain platform Ethereum, which aims to provide automated frameworks for the resilience of a given ecosystem. By establishing a “Decentralized Autonomous Organization” on the upper layers of the earth to govern them, terra0 research team (as a group of developers, theorists and researchers “studying hybrid ecosystems in the technosphere”) aims to create technologically augmented ecosystems that are both more resilient and more capable of operating within a predetermined set of rules in the economic sphere, as independent agents: e.g. the forest independently mines cryptocurrency and decides how it will change its material base. The group believes that modern technologies such as remote sensing and machine learning provide an opportunity to rethink existing inefficient governance and regulatory structures. Moreover, they also seem to suggest how, with appropriate art-thinking and speculative design-based assumptions and interventions, these could play a key role in creating a sustainable, resilient and biodiverse future. Along the line of above-mentioned cosmopolitics, the dominance of the human explanatory and decision-making system that dedicates natural resources within an anthropocentric economy (of both material, financial and symbolic orders), is juxtaposed by a radical scenario of self-governance of the analog (networked) forest, supported by (in deed human-developed) digital (network) technologies. ¹⁶

Conclusions

The above discussion presents different kinds as well as different grades of symbiotic relationships reflected in artistic works, and the research these are based on – such that is combined from different (blends of) scientific and artistic tools and methodologies.

To focus one last time on this contribution’s pivotal project; at the JRC’s Resonances IV Art-Science Summer School event the TTT video premiere owed its fond reception assumingly to an audience a priori knowledgeable about the treated topics, and with a generally positive attitude toward art-science collaborations. In discussions around the video audiences would often try to get behind the methodology of the combined art-cum-scientific research as allegedly reflected in the directing and editing approach to the video. By contrast, at the RETHINKABLE Festival of Transformative Economies and Regional Communities the concept of close cooperation between art and science – let alone radical post-anthropocentric critical thinking – was not known to the audience, thus their feedback revolved around the “intricate” combinations of photographs of archival documents, artistic drawings, “controversial text and perplexing sounds” that all took them “to a mystical forest”, as one of the (art) exhibition visitors concluded. By contrast, the audience of the scientific panel Ecological Crisis and Criticism discussed the video’s verbal messages (“theoretical poetry”) on the background of the festival’s topics such as biodiversity, commoning policies, or transformative economies.

Furthermore, the TTT video and its research supported an important further step of the process as featured in the Symphotree performance at the Speculum Artium festival of new media culture, where the artistic and the scientific approach were quite transparently set up on stage as well as verbally explained both before and after the half-hour audio-video performance, to open up the research process underlying the artwork presented. After the performance, discussions emerged mostly about the scientific data and methodologies used in the process that lead up to the artistic act, continuing into considerations of both policies and popular practices of conceiving, understanding and eventually using the forest. In the TTT project’s performative extension, the plant and the water could be seen as considerably foregrounded in terms of knowledgeable entities, delivering key informational impulses to the live artistic audiovisual score, working with and along (if not in front of) the two human artists on stage.

Not only individual trees standing (growing, or indeed performing) vis-à-vis humans, but also forests (as multitudes of trees) may be the most complete metaphor for both a process and a phenomenon of growth that is systemically orchestrated, circular and long-term balanced – among different stakeholders and their types of information (energy) processing that, in order to bear both ethical and aesthetical positive value, must be kept in a dynamic and polyphonic dialogue between body and space, be it electronic or physical. ¹⁷ The TTT project poetically boiled down to a chorus of three imperatives, echoing throughout the audio narrative within the video: Rewire Art:Science // Demystify Nature // Resettle Forest. Combining artistic and scientific methods, cross-pollinating them both in the educational and in the cultural sector, on local as well as on systemic (policy, economy etc.) levels, the TTT project process so far brought about artistic derivatives such as an experimental video and a performance, as well as important scientific insights and even policy influencing moments.

Along with the four above presented secondary cases of media art manifesting different forms of symbiotic mutualism, it might be claimed that the forest eventually emerged as a new (kind of) metaphor for a self-regulating system that the human (kind) should probably think twice before considering to have understood it by any form of (human) knowledge, either by technology or policy, let alone aesthetics or ethics. The need for a radically new sensibility and complex knowledge appreciation thus becomes evident, moving beyond objective study and becoming attentive to different dimensions of research and its outputs that emerge through the introduction of artistic thinking, as well as interspecies transactions on equal terms. Only such radical form of inter-sectoral discourse can bring about viable models of conviviality among different species of life, forms of knowledge and creativity, be it intellectual or purely material. Surpassing the commonly assumed scientific objectivity where (especially non-human) entities are considered mere objects of research, such an approach opens toward (perceiving) their subjective agency, and their true individuation, as well as their potential to eventually change the eye of the beholder.

References

1 Donna J. Harraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham, Duke University Press, 2016.

2 Isabelle Stengers, Cosmopolitics II, Minneapolis: University of Minnesota Press, 2011.

3 Virginia Gewin, “How to Shape a Productive Scientist-Artist Collaboration”, Nature, Vol. 590, 2021, accessed December 1, 2022, https://www.nature.com/articles/d41586-021-00397-1.

4 Jessica Jacobs, “Intersections in Design Thinking and Art-Thinking: Towards Interdisciplinary Innovation”, Creativity. Theories – Research – Applications, 5: 1, 2018, 4–25.

5 Peter Purg, Silvia Cacciatore Jernej Č. Gerbec, “Establishing ecosystems for disruptive innovation by cross- fertilizing entrepreneurship and the arts,” Creative Industries Journal, 2021, accessed December 1, 2022, https://www.tandfonline.com/doi/full/10.1080/17510694.2021.1969804.

6 JRC, European Commission's Knowledge Centrefor Bioeconomy, “Brief on the role of the forest-based bioeconomy in mitigating climate change through carbon storage and material substitution”, 2021, accessed December 1, 2022, https://publications.jrc.ec.europa.eu/repository/bit- stream/JRC124374/brief_on_role_of_forest-based_bioeconomy_in_mitigating_cc_online.pdf.

7 Abiral Khadka, Nikita Meden, Tijana Mijušković, Kristina Pranjić, Peter Purg, “Taming the Forest”, University of Nova Gorica website, accessed December 1, 2022, https://vimeo.com/725233829.

8 Eduardo Kohn, How Forests Think. Toward an Anthropology Beyond the Human, Berkeley, University of California Press, 2013.

9 “Plantportation (2019–20)”, Culture Bridges website, accessed December 1, 2022, http://culturebridges.eu/en_success_stories/olena_kasperovich. 

10 Peter Sloterdijk, The Aesthetic Imperative: Writings on Art, Cambridge, Polity, 2017.

11 BeComing Tree community, “Be-Coming Tree (2021)” online community website, accessed December 1, 2022, https://becomingtree.live/.

12 Angelika Boeck and Peter Tepe, “What is Artistic Research?” w/k - Between Science & Art Journal, Vol. 1, 2021, accessed December 1, 2022, https://between-science-and-art.com/what-is-artistic-research/.

13 Billy Ehn, “Between Contemporary Art and Cultural Analysis: Alternative Methods for Knowledge Production,” InFormation – Nordic Journal of Art and Research, 1: 1, 2013, 4–18.

14 Natalie Loveless, How to Make Art at the End of the World. A Manifesto for Research-Creation, Durham, Duke University Press, 2019.

15 Agnes Meyer-Brandis, “One Tree ID” Kersnikova Institute website, accessed December 1, 2022, https://kersnikova.org/en/posts/events/all/agnes-meyer-brandis-one-tree-id-3. 

16 terra0, “terra0 (2021)”, terra0 group website, accessed December 1, 2022, https://terra0.org/.

17 Peter Purg, Körper im elektronischen Raum. Modelle für Menschen und interaktive Systeme, Bonn: VDM, 2005.

Bibliography

1 Be-Coming Tree community, “Be-Coming Tree (2021)” online community website, accessed December 1, 2022, https://becomingtree.live/.

2 Angelika Boeck, Peter Tepe, “What is Artistic Research?” w/k - Between Science & Art Journal, Vol. 1, 2021, accessed December 1, 2022, https://between-science-and-art.com/what-is-artistic-research/.

3 Billy Ehn, “Between Contemporary Art and Cultural Analysis: Alternative Methods for Knowledge Production,” InFormation – Nordic Journal of Art and Research, 1: 1, 2013, 4–18.

4 Virginia Gewin, “How to Shape a Productive Scientist- Artist Collaboration”, Nature, Vol. 590, 2021, accessed December 1, 2022,February, https://www.nature.com/articles/d41586-021-00397-1.

5 Donna J. Harraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham, Duke University Press, 2016.

6 Jessica Jacobs, “Intersections in Design Thinking and Art Thinking: Towards Interdisciplinary Innovation”, Creativity. Theories – Research – Applications, 5: 1, 2018, 4–25.

7 JRC, European Commission's Knowledge Centre for Bioeconomy, “Brief on the role of the forest-based bioeconomy in mitigating climate change through carbon storage and material substitution, 2021, accessed December 1, 2022, https://publications.jrc.ec.europa.eu/repository/bit- stream/JRC124374/brief_on_role_of_forest-based_bioeconomy_in_mitigating_cc_online.pdf.

8 Abiral Khadka, Nikita Meden, Tijana Mijušković, Kristina Pranjić, Peter Purg, “Taming the Forest”, University of Nova Gorica website, accessed December 1, 2022, https://vimeo.com/725233829.

9 Eduardo Kohn, How Forests Think. Toward an Anthropology Beyond the Human, Berkeley, University of California Press, 2013.

10 Natalie Loveless, How to Make Art at the End of the World. A Manifesto for Research-Creation, Durham, Duke University Press, 2019.

11 Agnes Meyer-Brandis, “One Tree ID”, Kersnikova Institute website, accessed December 1, 2022, https://kersnikova.org/en/posts/events/all/agnes-meyer-brandis-one-tree-id-3. 

12 Peter Sloterdijk, The Aesthetic Imperative: Writings on Art, Cambridge, Polity, 2017.

13 “Plantportation (2019–20)”, Culture Bridges website, accessed December 1, 2022, http://culturebridges.eu/en_success_stories/olena_kasperovich.

14 Peter Purg, Silvia Cacciatore, Jernej Č. Gerbec, “Establishing ecosystems for disruptive innovation by cross-fertilizing entrepreneurship and the arts,” Creative Industries Journal, 2021, accessed December 1, 2022, https://www.tandfonline.com/doi/full/10.1080/17510694.2021.1969804.

15 Peter Purg, Körper im elektronischen Raum. Modelle für Menschen und interaktive Systeme, Bonn, VDM, 2005.

16 Isabelle Stengers, Cosmopolitics II, Minneapolis, University of Minnesota Press, 2011.

17 terra0, “terra0 (2021)”, terra0 group website, accessed December 1, 2022, https://terra0.org/. 

Same Old Story: Agential Realism in the Study of Colonial Histories

Hamed Rashtian, Gabriela Aceves-Sepulveda

Simon Fraser University Surrey
Canada hamed_rashtian@sfu.ca, gabriela_aceves-sepulveda@sfu.ca

Abstract

What are the possibilities of accessing the reality of history? How can we read history, and what can we learn from it? In this paper, we contemplate these questions by putting our ongoing research-creation project, Same Old Story (2020-present), in conversation with feminist critiques of objectivity and current discussions on the construction of historical narratives by historians, philosophers and artists, including Antoinette Burton, Andreas Huyssen, Walter Benjamin Walid Raad and Forensic Architecture. Specifically, we elaborate on how Karen Barad's "agential realism" ¹  informs our engagement with colonial histories in Same Old Story and speculate on its broader relevance in research-projects that engage with historical narratives. To do so, we describe the process of creating the current iteration of our project and offer a theoretical framework based on a discussion of three main themes, Archive/ Memory, Architecture and Monument/Counter-Monument. Building from this discussion, we elaborate on how to expand our work further, focusing on the possibilities and limits of revitalizing embodied realities in historical events and learning from them.

Keywords

Historical Research; Archival Research; History of Colonialism; Agential Realism; Situated Knowledges; Embodied Observation; Architecture; Art Installation; Monument; Counter-Monument.

DOI

10.69564/ISEA2023-78-full-Rashtian-et-al-Same-Old-Story

Introduction

Understanding embodied realities in the study of different histories of colonialism is one of the main concerns of this paper. In “Meeting The Universe Halfway: Realism and Social Constructivism Without Contradiction,” Karen Barad advocates for a particular form of realism that she coins “agential realism.” ¹ She argues agential realism does not reject objectivity but believes in an objectivity that is embodied. In this definition, the object of observation is always observed from somewhere and this observation is situated within language and culture. In other words, the observer understands the object from a certain perspective and there is agency associated with them (the object and the viewer). In this paper, we expand the concepts involved in our research-creation project, Same Old Story (2020-present), a multimedia art installation that engages with archival documents and architectural structures through drawing, text and sound, to investigate the possibilities of accessing the embodied reality of different colonial histories, informed by Barad’s conception of reality. In Same Old Story, we use the frame of research-creation as a broad methodology to describe our engagement with historical narratives, to benefit from the diverse possibilities of combining image, text and sound in revitalizing historical events and to explore the different capacities of the archival material. By expanding on the theories involved in Same Old Story, this paper contributes to elaborating on the role of archives, buildings and monuments in the search for the embodied realities of historical events.

Both buildings and archives are material cultural objects² that are observable by artists and can be subject to research-creation and visual decoding. In this paper, we will expand on how Same Old Story proposes a model that puts diverse histories in conversation with each other through archival research and multimedia art strategies. The focus of this paper is to address the following: Can the model proposed in Same Old Story in juxtaposing diverse historical events result in an expanded understanding of the reality of those events?

We are specifically curious about the different histories of colonialism and the architectural structures associated with them, as buildings and monuments have always been used by the colonizers in their process of domination. Through a close reading of the architecture involved in colonial struggles, we aim to investigate the potential violence against the colonized that is carried in their materials and shapes.³ These buildings are marks on the body of history that can contribute to the understanding of embodied objectivity and potentially guide historians to a form of agential realism.

The current iteration of Same Old Story (Figure 1) incorporates six different historical events focusing on the architecture of the places related to them. Through multilayered drawings and sounds, the work puts different histories of colonialism in conversation with each other to study whether this juxtaposition results in a deeper understanding of the power relations embedded in colonial struggles. The use of juxtaposition as a method is informed by Donna Haraway’s critique of objectivity as described in her essay “Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective.” Haraway argues that in comparison with dichotomous charts, “a map of tensions and resonances between fix ends,”⁴ better represents different politics and epistemologies, and therefore a better account of objectivity can emerge. In Same Old Story, we created a map of colonial histories focusing on their power dynamics. By incorporating a cacophony of voices and setting the stage for a conversation, the concept map proposed in Same Old Story attempts to avoid the certainty of monolithic historical narratives that each architectural structure depicted in the work imposes.⁵ Although this juxtaposition might not result in an objective account of realism, our goal is to revitalize some truth by depicting multiple instances of colonial histories through their architecture. This first iteration prompted us to continue to explore the use of juxtaposition as a historiographic, artistic and theoretical approach. 

Figure 1. Same Old Story (2020-present), Laser engraving on canvas, six-channel sound installation, installation view 

In what follows, we provide an overview of what we define as an embodied approach to historical research informed by the work of historians and philosophers Walter Benjamin, Edward Said, Andreas Huyssen, Antoinette Burton and Haraway, as well as members of the art research agency Forensic Architecture. They all argue for understanding historical narratives as situated and always socially, culturally and politically contingent to the present. This discussion sets the framework for our engagement with Barad's concept of "agential realism" that we hope to develop further in future iterations of this project. To contextualize this discussion, we elaborate on the processes of creating Same Old Story, including a detailed description of the work and a discussion of three main themes that shape the project: Archive/Memory, Architecture and Monument/Counter-Monument. Ultimately, this paper proposes to contemplate acts of observation and conveyance in the study of history and the possibility of putting different historical narratives in conversation as a way to gain an expanded understanding of the embodied realities of the historical events under observation through research-creation.

Historical Research as Embodied Experience

In the study of history, the main objects of observation are past events, which makes its study specifically complicated because the objects of observation are highly fugitive. As Walter Benjamin asserts, the vanishing point of history is always the present moment; thus, construction plays a crucial role in historical narratives. ⁶  Thinking about the role that memory plays in historical narratives, Edward Said refers to collective memories as frequently manipulated and intervened for the urgent purposes of the present. ⁷ Similarly, in her study on how archives are assembled, Antoinette Burton argues that it is a fantasy that history can deliver the absolute truth. Instead, history is a highly interpretive act.⁹ Burton tells us that the construction of historical narratives results from a pressure "present-day politics place on the past."⁹ Echoing Burton, Said and Benjamin, for Andreas Huyssen, crafting historical narratives is a mode of representation that belongs ever more to the present.¹⁰ In tune with Haraway, these authors share an understanding of a certain perspective embedded in the construction of historical narratives and a view from the present that is culturally conditioned or situated.¹¹, ¹²

Informed by these assertions on the construction of historical narratives we ask: how can we expand the concept of agential realism to the study of history? What happens if we do not have access to the object of observation? Barad argues the point of reference in agential realism is from the permanent marks “left on the bodies which define the experimental conditions” ¹³ and reality perpetuates itself on these permanent marks. What are the marks on the body of history? In Present pasts: urban palimpsests and the politics of memory, Huyssen posits that our urban environment is saturated with monuments and buildings that represent “the material traces of the historical past in the present.”¹⁴ For Eyal Weizman and other members of Forensic Architecture, buildings with strong marks from the past are “political sensors.”¹⁵ They are historical witnesses. “People can lie or forget, but buildings don't” Weizman and Forensic Architecture have emphatically declared.¹⁶

According to Barad, another measure for accessing agential realism is through reproducibility. “Reproducibility is possible because scientific investigations are embodied, grounded in experience.”¹⁷ History is reproduced through the multiplicity of narratives. All these narratives according to Haraway, incorporate visions that are situated.¹⁸ Historical narratives are reproduced through the proliferation of diverse forms of archival records and archives. Archives are not to be equated with an official institution or its holdings, archival records are found in domestic spaces and oral forms of transmission are considered valid archives, as Burton and others have demonstrated.¹⁹ As a result of this expanded understanding of what constitutes an archive, set in motion by what is referred to as the archival turn, official and canonical archives have lost their exclusive role as the only legitimate sources of history.²⁰

In this research we are curious about the narratives associated with people who, according to Burton, “believe that their histories have not been written because they have not been considered legitimate subjects of history.”²¹ These for Burton include colonized people whose lived experience is not validated by official historical narratives or achieves.²² Dominant narratives legitimize the actions of the colonizers and perpetuate their hegemony. Hence, colonial and racial hierarchies are reinforced and reproduced through these narratives ²³ as they are associated with policies that categorize colonized and marginalized groups as inferior.²⁴ In arguing about different perspectives in the study of knowledge, Haraway advocates for a vision from below the position of power and calls it a "better vision."²⁵ Following Haraway, Same Old Story supports this "better vision" through the use of diverse forms of archives, specifically online archives, to depict six different colonial histories. This vision from below characterizes a postcolonial tendency in our project and has led to the use of various unofficial archives. In the following section, we describe the process of creating Same Old Story and elaborate on the use of online image archives as references for the images depicted in the project.

An Account of Same Old Story

In Same Old Story, we are reflecting on the possibilities and limits of images (moving and still), texts (in any format) and sounds found online, to develop juxtapositions between different colonial experiences that can offer new insights into their histories. So far, we have included six different historical events in our work. The image component of the current iteration consists of six different superimposed renderings based on online archival photos of the sites of these events. The renderings are laser engraved together on one canvas (Figure 2). 

Figure 2. Same Old Story (2020-present), Drawing component. 

Besides the theoretical framework already described, during the initial stages of the project two other important sources inspired the use of juxtaposition. Firstly, the Iranian traditional form of dramatic storytelling called Pardeh Khani (Figure 3). Pardeh Khani is a performance in which the performer stands in front of a large painted backdrop that includes several stories related to each other and narrates these stories.²⁶ The painting component of this performance inspired the composition of the laser-cut drawing in Same Old Story. 

Figure 3. Pardeh Khani (a traditional form of dramatic storytelling in Iran) 

Secondly, Eve Tuck and C. Ree’s essay, “A Glossary of Haunting” informed our approach of putting distinct histories together as a form of resisting dominant epistemologies. In their essay Tuck and Ree advocate for violating the terms of settler-colonial knowledge production which, according to them, is based on “the separation of the particular from the general, the hosted from the host, the personal from the public.”²⁷ In short, to break the terms of settler colonial knowledge we need to get rid of separation and categorization. In our work, we aim to emphasize the similarities of distinct selected events in terms of power relations: how colonial power is symbolically, covertly or openly mobilized against the colonized.

To add to the livelihood of the stories and to encourage an intimate connection to the work, we included sound in the installation as well. The audio element was composed using sounds sourced from online archives, similar to the image component. It consists of six audio channels playing at the same time, with each sound related to a single event coming from a separate speaker. The aim is to combine drawing and sound to encourage simultaneity among different narratives. The presence of diverse narratives in different modalities (sound and image) in one work stimulates a fictional situation: it compresses time and space, creating an experience for the viewer in which all events are happening at the same time.

Finally, to provide some information about the events and to bridge between image and sound, we included some cap-tions in a booklet as a catalogue for the work. This booklet includes a brief description of each event to provide some context for the viewer.

The current iteration of Same Old Story includes multilayered drawing and sound while written text, in caption format, acts as a complementary element to the work. The drawings are made of digital line tracing of a selection of photos. The photos are collected through an online search and depict places related to the events (i.e., Palais de la Nation in Kinshasa, Holmenkollen Park Hotel near Oslo, Casbah neighborhood in Algiers, British Embassy in Tehran, the ruins of the Summer Palace in Beijing and wooden Tiny Houses on the path of pipeline construction in the northern BC). After studying the photos collected, we selected five distinct images of each place showing spaces within and around the buildings from different angles. We traced the outlines of each photo and created a digital drawing of each. We experimented with their sizes and their positions on a 94 x 63-inch background. We divided the background into six areas, each dedicated to one of the events. In each area, we adjusted the positioning of drawings to create a desired superimposed drawing depicting a different perspective of the same place. (Figure 4 and 5) Through these superimposed and blurry image compositions, we aim to highlight the coexistence of a multiplicity of narratives about each event and the illegibility of history more broadly.

Figure 4 and 5. Same Old Story (2020-present), digital drawings 

To create a physical version of our digital drawing, we experimented with a few techniques such as screen printing, digital printing on diverse kinds of photo papers, digital embroidery on canvas and muslin, etc. In the process, we came across laser-engraving. After running some tests with a laser machine on different kinds of papers, fabrics and canvases, we found laser engraving on raw canvas a befitting choice for the work both formally and conceptually. In experimenting with the laser machine, we could learn how to enhance the depth of drawing as well as how to set the machine to increase the possibility of burning the canvas, creating holes, and controlling their shape. We particularly like this specific treatment, as we found burning, destruction, and elimination to be in harmony with the history and trajectory of colonialism (Figure 6).

Figure 6. Same Old Story (2020-present), detail of the work

We used a similar approach in creating the sound composition for the installation. During the process of sound editing, and after making some samples, we realized that the type of connection between sound and image needs to be the same. By this we mean that if the connection to drawing happens instantly, the connection to sound needs to be instant rather than delayed to achieve the same effect. This observation led us to select abstract (spatial or ambient) sounds rather than sound of a conversation, for instance. The sounds that we selected for the work are:

Although the last sound (Patrice Lumumba’s anti-colonial speech) is more recognizable and specifically linked to one of the events depicted, it is in French which was not the language of the first iteration of the project. It was used as background sound and was not intended to be followed or understood. During the selection process, we tried to find the actual sounds by searching the online archives related to each event, however, in a few instances, where we did not have access to the original sound, we hinged upon fiction and used an archival sound that we assumed is similar to the actual sound that belonged to the event. This process is informed by the work of artist Walid Raad who delves into fiction in his practice of creating archives in the absence of historical records²⁸ and Burton’s ideas about the backstage of the archives and the role of construction in the process of their creation.²⁹ Later in this paper, I will discuss Raad’s practice as well as Burton’s ideas in more detail.

Same Old Story incorporates archival research and uses archival material in its creation. It engages with architecture by depicting the built spaces of the selected events, and the work itself has monumental qualities through its physical presence. In what follows, I will focus on the three mentioned elements of archive, architecture and monument in discussing the work and engaging with literature under the three themes of Archive/Memory, Architecture and Monument/Counter Monument. This discussion informs the methods used in the creation of Same Old Story for depicting different colonial histories to convey embodied realities about them.

Theoretical Discussion

In this section we expand the conceptual framework explored thus far through a discussion of the concepts of archive, memory, architecture, monument, and counter-monument. These concepts were selected to develop a more specific communication with Barad’s ideas of accessing the agential reality of history through marks on the bodies (architecture and monument) and the reproducibility of phenomena (archives) that gives an embodied account of objectivity.³⁰

Archive/Memory

In the process of research for Same Old Story, our main sources of information are archival data. This research began during the pandemic when we only had access to the online archives. Therefore, our collected data is mainly from online sources. In researching the archives, we have been looking for evidence of historical events in the search for accessing the reality of those events. Concerning the archival evidence and the notion of truth, Burton argues that archives come into being as a result of political pressure. “Debates that engage the challenges of ‘telling the truth about history’ have had very real political and material consequences”³¹ as the use and the nature of the archive are not only about the past but about the present. Informed by these, the question that comes to mind is how to trust archives. Which archive is trustworthy? How the archives have been shaped? Who has made them? and what are the limits of the archive for accessing the reality of historical events?

As a reflection on these questions, in her edited volume Archive stories facts, fiction, and the writing of history Burton elaborates on the term “archive stories” ³² as the stories of the formation of the archives that challenge their objectivity. She believes archives are the artifacts of history themselves and insists on looking at “the backstage of archives”³³ to investigate how they are constructed and manipulated. It is important to acknowledge that all archives are curated, thus there is power involved in their formation. According to her, archive-making is an “embodied experience,”³⁴ archives are historical actors, and they are made to speak to a specific audience. This notion speaks to Barad’s idea that observation is embodied³⁵ and Haraway’s consideration of the object of knowledge as an active agent and of knowledge production as situated.³⁶

Burton discusses how the access to digital technologies has made all of us archivists and archive consumers and as a result, more democratic versions of archives are emerging.³⁷ This condition has resulted in the proliferation of online archives that has challenged the role of the official and canonical archives as the only legitimate archival source.

In our project, besides using institutional online archives (official and canonical), we used a variety of non-institutional online archival material and media resources including personal websites or sites not backed by governmental agencies or corporations. Because we are interested in studying micro-narratives of historical events, our attention is specifically drawn to the archives that deal with oral history. That is to archival sources that “materialize those countless historical subjects who may never have come under the archival gaze.”³⁸ As Bruton argues, postcolonial scholars renewed their attention to the archives while acknowledging the value of oral history and its relation to memory which have been traditionally outside the colonial notion of the archive.³⁹ For Burton, as for other feminist historians “Memory is always cast (and still is) in gendered terms,” and “the capacity of women to write history has been considered dubious until quite recently.”⁴⁰

Similarly, in discussing different visual systems, Haraway ponders whether the perspective of the subjugated is more trusting, she states: “there is good reason to believe vision is better from below the brilliant space platforms of the powerful.”⁴¹ Likewise, Michael Foucault, calls the official and national archives “documents of exclusion” and “monuments to particular configurations of power.”⁴²

Informed by these expanded notions of what an archive is and what counts as history, we collected most of our data from non-institutional online archives and personal narratives. We understand our work as an embodied process that incorporates specific historical events and does not claim to transfer absolute truths but rather our fictional creation that hopefully triggers the viewer's curiosity to continue thinking about the events depicted.

Architecture

Every material object can be read as a sensor, but buildings might be among the best sensors of societal and political change. There are several reasons: buildings are immobile, anchored in space; they are in close and constant interaction with humans; they are exposed both to the elements outside them and to an artificially controlled climate within. And this besides embodying, of course, the political, social, strategic, and financial rationalities that went into their conception.⁴³

Buildings are among the most stable witnesses of the past. Huyssen refers to them as "palimpsests of space,"⁴⁴ where the marks of presence are merged with traces and erasure left from the past. In Same Old Story, we follow Huyssen's ideas and focus on the architectural elements related to selected historical events. We are interested in the empty spaces loaded with the heavy lift of history, spaces saturated with marks and material testimony of those events, markings that can be used as leverage against the colonial powers that have shaped them.⁴⁵ Architectural elements serve as evidence that aid in gaining a sense of the reality of past events. This understanding of architectural elements is aligned with Barad's idea of accessing reality through the marks on the bodies.⁴⁷ The marks of the past have an active presence in our living environment.⁴⁷

Buildings play political role in history, Weizman from Forensic Architecture refers to architecture as a “political plastic”⁴⁸ where different forces are shaped into forms. For Weizman, “architecture is a material trace that can operate as an entry point into understanding larger processe... reading macro from the micro.”⁴⁹ The sites that we study are saturated with elements through which the dominant colonizers have tried to create, according to Said, “a sense of identity”⁵⁰ for them and the colonized. As Weizman continues to explain, material aesthetics are interconnected with all the things around them and bear marks of events.⁵¹ These sites are haunted spaces filled with historical trauma. Their visible sides contain traces of the invisible.⁵² While researching archival photos of the six events that constitute the work's central visual component, we identified elements that operate as "actors and agents" ⁵³ in transferring a sense of domination of the colonial forces. We paid close attention to the details of the archival images of buildings to study how they can play a role in the power structures of colonial interactions. We emphasized certain elements in the drawings that, we believe, contribute to perpetuating colonial power relations. For instance, numerous tabletop and wall photo frames installed in the meeting rooms could indicate an interior design strategy supporting a certain sense of identity.

In our project, we seek to represent different types of architecture connected to colonial interactions or dominance. An example of the latter is the image of the Palais de la Nation in Kinshasa during the Belgian rule over Congo. An example of the former is the architecture of urban areas where the colonized population lives, often dense and ghetto-like neighbourhoods under heavy surveillance. The project depicts the Casbah neighbourhood in Algiers during the French occupation. In other cases, anti-colonial movements use architecture as a form of resistance against colonial forces. This type of architecture is exemplified by the inclusion of the tiny houses that the Wet'swet'en peoples built on the site of the gas pipeline in northern B.C. to demonstrate their ancestral presence on those territories. Other architectural elements included in our project are ruins as traces of colonial power and scars on the land that can lead us to investigate different historical perspectives. The ruins of China's Summer Palace, destroyed during the Opium Wars directed by British and French forces against China, exemplify this architecture. In future iterations of our project, we will continue to investigate other forms of architecture that serve as traces of colonialism in one way or another.

Monument/Counter-Monument

In the process of developing Same Old Story, we came across the concept of monument and the monumentality of colonial buildings. We even discussed whether we wanted our work to have monumental qualities. As we delved more profoundly into monumentality and its implications, we came across the concept of counter-monument. In what follows, we reflect on this term and how it informs Same Old Story as a counter-monument.

In “Memory/Monument” James E. Young refers to the Latin definition of the monument as "things that remind"⁵⁴ to introduce his ideas on how monuments act as symbols of a specific historical era. He notes how agencies of power use monuments as a self-aggrandizing tool to dramatizing their achievements and create a sense of identity. The function of monuments is to generate a unifying, reductive sense of memory, serving the power and mythicizing its role in shared memory. However, in this process, monuments may not remind us of events so much as bury them beneath the layers of those myths they project.⁵⁵ Young argues if monuments are mediums at the service of power, a monument against power is a monument against itself, against its authoritarian role, and its “traditionally didactic function”⁵⁶  that reduces the viewer's role to a passive spectator. He coined the term “counter-monument”⁵⁶ to refer to monumental works of art that resist their traditional function.

Thomas Stubblefield further elaborates on the concept of counter-monument. In his essay “Do Disappearing Monuments Simply Disappear?”⁵⁷ Stubblefield argues for the potential of counter-monuments in revitalizing the past through exchanges between the viewer and the work. He

asserts that counter-monuments turn a site into a discourse by democratizing power relationships and avoiding official and singular narratives. In this manner, counter-monuments sidestep the monument's certainty of history and rigidity by transferring productive recreation to its visitors. Such a monument creates the possibility of confronting opposing ideologies and frees itself from imposing a top-down ideology. Counter-monument operates within the postwar art discourse that seeks "a relational and contingent identity by relying on the viewer for both the creation of meaning and the very construction of the work itself."⁵⁸

In Same Old Story, our aim is to avoid didacticism and the creation of reductive narrative of events. To achieves this, we tried to move toward abstraction, to keep our distance from any form of literal representation and avoid what Donna Haraway refers to as the “god trick,”⁵⁹ which is a view from above that erases difference. We used superim-position to create blurry images incorporating multiple perspectives of each historical site. As we further explore the concept of counter-monument we are reflecting on how to foster a more democratic relationship with the viewer. Stubblefield discusses interaction and ephemerality as two qualities of counter-monuments.⁶⁰ Following Stubblefield, in future iterations of this project, we will reflect on the possibilities of incorporating the viewers' input more meaningfully in the work and how to reduce the physicality of the installation to avoid imposing a fixed presence in the space. To do so, we plan to explore the affordances of immersive virtual environments.

Conclusion

This paper situates our ongoing research-creation project, Same Old Story, in conversation with Karen Barad and Donna Haraway’s feminist critiques of objectivity and current discussions on the construction of historical narratives by historians, philosophers and artists, including Antoinette Burton, Andreas Huyssen, Walter Benjamin, Walid Raad and Forensic Architecture. Specifically, we provided an overview of Barad's concept of "agential realism" and Har- away's "situated knowledges" as central concepts that inform our use of juxtaposition of sound and image to put different histories of colonialism in conversation with one another to 1) offer a critique of objectivity in historiography and 2) investigate the possibilities of accessing the embodied reality of different colonial histories through research-creation.

We discussed how our project proposes an embodied approach to historical research through its theoretical frame-work, the process of making the first iteration of Same Old Story, and its installation experience, which offers a space where multiple historical narratives can be experienced simultaneously. Using images and sound, Same Old Story offers various historical perspectives rather than a view from above or a monolithic historical narrative. We further elaborated on our project's three additional themes: 

Archive/Memory, Architecture and Monument/Counter-Monument. In future iterations, we will continue to theorize these concepts in relation to the work of Barad, Haraway and others to explore their potential in putting diverse colonial histories in conversations with each other to offer more nuanced perspectives of these distinct histories. To further explore whether the proposed model in Same Old Story of juxtaposing diverse historical events results in an expanded understanding of the reality of those events. Parallel to these theoretical explorations, we will study the possibilities of virtual environments and other technologies to facilitate a more active interaction from the viewer. 

Acknowledgements

The first iteration of Same Old Story was developed as part of Rashtian’s MFA degree at Emily Carr University. The theoretical discussion presented in this paper was developed as part of his doctoral research under supervision of Aceves Sepúlveda in the School of Interactive Arts and Technology at Simon Fraser University. 

References

1 Karen Barad, “Meeting the Universe Halfway: Realism and Social Constructivism without Contradiction”, Nelson L.H., Nelson J. (eds) Feminism, Science, and the Philosophy of Science, vol 256, 1996, 167

2 Barad, "Meeting the Univers Halfway"

3 Eyal Weizman Bois, Yve-Alain Michel Feher, Hal Foster, “On Forensic Architecture: A Conversation with Eyal Weizman”, October 156 (156), 2016, 116–40.

4 Donna Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective”, Feminist Studies 14 (3), 1988. 588

5 Thomas Stubblefield, “Do Disappearing Monuments Simply Disappear? The Counter-Monument in Revision”, Future Anterior 8 (2), 2012, 1–12.

6 Mark Godfrey, “The Artist as Historian", October 120 (120), 2007, 140–72.

7 Edward W. Said, “Invention, Memory, and Place.” Critical Inquiry 26 (2), 2000, 175–92.

8 Antoinette M. Burton, Archive Stories Facts, Fictions, and the Writing of History / Edited by Antoinette Burton, Durham, N.C.,Duke University Press, 2005.

9 Burton, Archive Stories, 17.

10 Andreas Huyssen, Present Pasts: Urban Palimpsests and the Politics of Memory / Andreas Huysse, Stanford University Press, 2003. 

11 Weizman et al, “On Forensic Architecture.”

12 Haraway, “Situated Knowledges.”

13  Barad, “Meeting the Universe Halfway”, 180.

14 Huyssen, Present Pasts, 1.

15 Weizman et al, “On Forensic Architecture”, 121.

16 Weizman et al, “On Forensic Architecture”, 131.

17 Barad, “Meeting the Universe Halfway”, 185.

18 Haraway, “SituatedKnowledges.”

19 Burton, Archive Stories. 

20 Burton, Archive Stories.

21 Burton, Archive Stories, 2.

22 Ricardo Castro-Salazar, Carl Bagley, “Navigating Historical Borders: Internal Colonialism and the Politics of Memory”, Counterpoints, New York, N.Y., 415, 2012, 37–67.

23 Bagley Castro-Salazar, “Navigating Historical Borders.” 

24 Nnaemeka Obioma, “Racialization and the Colonial Architecture: Othering and the Order of Things.” PMLA : Publications of the Modern Language Association of America 123 (5), 2008, 1748–51.

25 Haraway, “Situated Knowledges”, 583.

26 Amir Lashkari, Mojde Kalantari, “Pardeh Khani: A Dramatic Form of Storytelling in Iran”, University of Hawai’i Press, 23 (1), 2015, 245-258

27 Eve Tuck, C Ree, “A Glossary of Haunting”, In Handbook of Autoethnography, edited by Stacey Holman Jones, Tony E. Adams, Carolyn Ellis, Left Coast Press, 2016, 640.

28 Jong-chul Choi, “Reinventing the Archive in the Age of Digital Reproduction: Walid Raad's the Atlas Group.” Digital Creativity (Exeter) 29 (2-3), 2018, 235–48.

29 Burton, Archive Stories.

30 Barad, “Meeting the Universe Halfway.”

31 Burton, Archive Stories, 1.

32 Burton, Archive Stories, 6.

33 Burton, Archive Stories, 7.

34 Burton, Archive Stories, 9.

35 Barad, “Meeting the Universe Halfway.”

36 Haraway, “Situated Knowledges.”

37 Burton, Archive Stories.

38 Burton, Archive Stories, 17.

39 Burton, Archive Stories.

40 Burton, Archive Stories, 20-25

41 Haraway, “Situated Knowledges”, 583.

42 Burton, Archive Stories, 6.

43 Eyal Weizman, Forensic Architecture: Violence at the Threshold of Detectability / Eyal Weizman. Zone Books, 2017, 52.

44 Huyssen, Present Pasts, 7.

45 Weizman et al, “On Forensic Architecture.”

46 Barad, “Meeting the Universe Halfway.”

47 Huyssen, PresentPasts.

48 Weizman et al, “On Forensic Architecture”, 128.

49 Weizman et al, “On Forensic Architecture”, 134.

50 Said, “Invention, Memory, and Place”, 178.

51 Weizman et al, “On Forensic Architecture.”

52 Huyssen, Present Pasts.

53 Haraway, “Situated Knowledges”, 592.

54 James E. Young, “Memory/Monument”, Critical Terms for Art History / Edited by Robert S. Nelson and Richard Shiff, 2nd ed. University of Chicago Press, 2003, 234.

55 Young, “Memory/Monument”.

56 Young, “Memory/Monument”, 240.

57 Stubblefield, “Do Disappearing Monuments Simply Disappear?” 

58 Stubblefield, “Do Disappearing Monuments Simply Disappear?”, 5.

59 Haraway, “Situated Knowledges”, 581.

60 Stubblefield, “Do Disappearing Monuments Simply Disappear?” 

Data-Phantoms: Impossible Nests (Memories Post Extinction)

Clarissa Ribeiro, PhD

Roy Ascott Technoetic Arts Studio at SIVA/DeTAO Shanghai, China cr@clarissaribeiro.com

Abstract

Recent studies show that in ‘anthropogenic landscapes’ birds have been forgetting how to sing and build nests since their parents die earlier and their communities are forced to be fragmented. Dialoguing with the Sub-theme “Symbiotic Imaginaries: Inventing Worlds,” the work “Data-Phantoms: Impossible Nests (Memories Post Extinction)” (2022) explores the phantasmagoric aspect of raw data coming from ‘nature traces’ of six (6) bird species declared extinct in nature along sequential morphogenetic transformations from numbers’ lists (birdsongs used as primary data), to geometrically complex and irregular data sculptures. The paper presents a discussion around the poetics that refers to an ongoing endeavor in exploring and discussing metaphysical aspects of data visualization embedded in the tools and processes chosen for parametric modeling and digital fabrication. The work intends contributing to reinforce our "symbiotic imaginaries," ‘inventing new worlds’ in which humans together with all living beings coexist and collaborate in their surviving efforts. The six data-sculptures—imperfect or ‘impossible nests’—are tentative explorations of the sublime in dystopian data-visualization aesthetics, manifesting in its irregular and messy geometry, the impossibility of birds, in broken ecologies, to perform their birdsong and successfully mate, to learn from their community how to build an ‘optimum’ nests and prosper.

Keywords

Data-Phantoms, data-visualization aesthetics, metaphysical aspects of data visualization, incarnation, inherited complex learned behaviors, parametric modeling, morphogenesis, birds’ nests, anthropogenic landscapes, symbiotic encounters.

DOI

10.69564/ISEA2023-79-full-Ribeiro-Data-Phantoms

Introduction

“They approached with slow movements, in a blinding light, waving young hoko si palm leaves. With their arms decorated with scarlet macaw tail feathers and a profusion of bright, colorful bunches of paixi feathers, coated with vermilion annatto dye, they roared at the top of their voices, like a group of guests arriving at a reahu feast. There were so many of them, and they kept their eyes set on me. It was beautiful but terrifying because I had never seen xapiri spirits before.”¹

In "The Falling Sky: Words of a Yanomami Shaman," Davi Kopenawa explains that, all our animal ancestors who got extinct, according to the Amazonian Yanomami’s "ecology as cosmology," “[...] have not disappeared [...] they still have their animal names, but are now invisible beings. They have transformed themselves into xapiri, who are immortal.”¹ 

As a poetic reference to the extinction of species as a process of information disintegration and dispersion in natural environments, dialoguing with the understanding expressed by the Yanomami community from the Amazon—in a moment we sadly see young birds forgetting their songs as adults are dying faster²—this paper navigates the poetics of the work “Data-Phantoms: Impossible Nests (Memories Post Extinction)” (2022). The work consists of a series of data-sculptures generated using as raw data the birdsong of six species that got extinct in nature worldwide, inviting to meditate on how fragmented audio memories of birds recorded and immortalized by humans in short videos shared online on platforms such as Vimeo and YouTube, can haunt our imagination as phantoms of long-gone forests increasingly replaced by anthropogenic landscapes.

Birdsong is one of the most studied aspects of animal behavior and it is crucial in mate attraction and territorial protection. According to a study published in March 2021, due to the population decline, “[...] male songbirds are having trouble picking up on the songs to attract mates—just as humans lose their culture with future generations, these birds are losing theirs over time.”² The researchers observed that the impact of severe population decline on song culture, in a nomadic, nectarivorous songbird, had dramatic fitness consequences for the remaining individuals considering that the production of atypical songs carried reproductive costs—“males whose songs differed from the regional cultural norm were significantly less likely to be paired to a female.”² 

Phantasmagoric landscapes

Anthropogenic landscapes are parts of Earth’s surface where humans’ populations have significantly and endlessly altered natural patterns and processes in order to meet their demands for food, fuel, housing, transportation, recreation, to mention a few. There is a general consensus among scientists who study threatened bird species that, if human impact on the environment continues as it has, one third of all species and an even larger proportion of bird populations will be gone by the end of this century. In Australia, as an example, in the 250 years since Europeans colonization, native birdlife has been dramatically impacted—22 of the native bird species have gone extinct.³ All over the planet, it is estimated that more than 180 bird species have likely gone extinct over the last 500 years. We are currently experiencing what many scientists perceive to be our planet's sixth mass extinction with far-reaching ecological, cultural, and even economic implications.

By mapping and quantifying this loss, more and more scientists are joining efforts to help refining the scientific understanding about the “impact of habitat removal and other pervasive threats that are driving this observed extirpation.”⁴ The birds most at danger of going extinct today are typically those that depend on certain, unique environmental circumstances, portending a future with declining biodiversity that benefits invasive, generalist species in anthropogenic contexts.

Broken Memories

Back in the 1960s, Elsie and Nicholas Collias were running observations and experiments carried out on a colony of captive Village Weaverbirds (Textor cucullatus)—a bird that makes complex and highly organized nests—using a group of young birds hatched in a large outdoor aviary on the campus of the University of California Los Angeles (UCLA). Although many people view nest building as a prominent example of instinctual behavior in birds, this seminal research showed evidences that “[...] some social facilitation exists for nest building, since the birds tend to show interest in nest materials at about the same time, just as in the case of their other activities.”⁵ 

Figure 1. First nests built by young Village Weaverbirds males (right) are more crudely constructed than are nests built by adult, experienced males. ⁵ Image by Elsie C. Collias and Nicholas E. Collias ⁵.

In this seminal experiment, Elsie and Nicholas Collias ⁵ made crucial observation on how birds develop the ability to manipulate objects—a complex knowledge that is essential for building their nests. They observed that the two weeks age Village Weaverbirds, more than only gaping to receive food from parents, use of their bill to ‘preen disintegrating sheaths off the feathers’ involving ‘biting and nibbling at the feather sheaths’. The researcher considers that “The mouthing of the feather sheaths would seem to be the precursor of the ability to mandibulate strips of nest materials, and thus to adjust the position of a strip in the bill”⁵ since it introduces motor elements needed for nest building. This ability, according to the researchers’ observations, is very likely facilitated by parental example. In their comments on the birds gathering of nest material, the researchers observe that when a weaverbird obtains a strip from a leaf, after perching on the stalk or firm base of the leaf, it “[...] bites through one edge of the leaf, tearing off part of the strip, and then tears the rest of the strip loose by flying away with it in the general direction of the tip of the leaf.” ⁵  Elsie and Nicholas Collias comment that,

“In watching young weaverbirds, whether these birds were reared by hand or in the aviary, it seemed to us that they had to learn many things in carrying out the process of tearing a strip of reed grass properly. While experienced adult males generally fly off, finishing the tearing in one smooth action, the young often made such mistakes as perching in an unstable place, starting the tear too close to the tip of the leaf, or at the very base, or taking too broad or too narrow a bite, or tearing in the wrong direction, or tearing part way and repeatedly starting partly detached strips, or tearing strips that were too short to be woven.”⁵ 

Confirming the importance of socialization mechanisms in birds communities in helping refining learning abilities relate to nests building and even the role that collaboration between young and adult birds plays in teaching by "showing how to do," Brosset ⁶ presents the results of a comparative study of social organization during breeding among the genus Malimbus of weaverbirds. Brosset observes that,

“In their morphology and behaviour, Malimbus spp. are close to the weaver birds of the genus Ploceus. M. nitens seems the least evolved species while M. cassini and M. coronatus are behaviourally the most evolved. In the last species, which has a very elaborate nest, the pair of breeding birds is helped by one to four other birds. These helpers are birds in full adult plumage, and are probably capable of breeding and may do so at another period in the long breeding season of at least six months.”⁶ 

Additionally, the research refers to similar comparable and systematic observations on species other than the weaverbirds, in regard to the handling of nest materials that had fund that hybrid parrots, as an example, gradually improve their ability to cut and transport suitable nest materials in health environments and contexts where they can learn from previous generations’ examples. It seems reasonable to conclude that, teaching and collaboration are key in sustaining the planetary symbiosis of species and within species. Unfortunately, we see that population decline is dramatically eroding cultural memory in wild animals populations and that the birds are not alone in facing their dramatic destiny in this broken symbiosis with humans. The loss of culture is associated with individual fitness costs⁶ possibly contributing to ongoing populations decline for an endless number of species.

Impossible nests

Exploring the impossibility of birds learning from previous and same-generation individuals how to build a proper and efficient nest in anthropogenic landscapes, the work “Data-Phantoms: Impossible Nests (Memories Post Extinction)” (2022)—explores phantasmagoric aspects of raw data as the birdsong used as the input for the generative design of the data-sculptures comes from birds declared extinct in nature. The work invites to meditate on how the building abilities of organisms, far from being straightforward gene expressions, are connected to complex inherited and learned behaviors that can be appreciated as part of the self-organizing adventures of species and their symbiotic abilities and tendencies that drives cross-scale interactions and integrations in the most diverse environmental configurations and variable conditions.

Data-Phantoms, Data-Incarnations

The work is part of ongoing experiments and efforts in discussing and exploring the metaphysical aspect of data visualization in dialogue with Edgar Wind's notion of "incarnation."⁷ Wind broke with Neo-Kantian tradition by constructing an approach to symbolic forms that provides a true account of symbolic representation—the symbol has a real, visible meaning through embodiment (incarnation). The effort can be described as a meditation on how a creative process that includes the use of birdsong as raw data in a morphogenetic algorithmically driven process can conduct humans and birds through a collaboration-as-fantasy mediated by computers, evoking our symbiotic imaginaries. The work intents to evoke metaphysical aspects of data visualization embedded in the tools and processes chosen for parametric modeling and digital "materialization" (digital fabrication, 2D printing).

Edgar Wind ⁸ elegantly dialogue with notions coming from discussions around the quantum measurement problem to discuss how in lab experiments, the laws which are to be discovered are already presupposed, being embodied in the instruments. Transplanting these considerations to discuss aspects of the work “Data-Phantoms: Impossible Nests (Memories Post Extinction)” (2022) related to its poetics, we can consider that in a collaboration between an artist (a human being), the "ghosts memories" or "ghost audio memories" of the six birds gone extinct in nature, a laptop running Rhinoceros 7 and Grasshopper and related customized algorithms, and the Printer used to print the posters (from a series of 3D renderings)—all recognized by the human as non-humans—the experiment tests its own presuppositions.

Figure 2. Generating one of the six ‘impossible nests’ in Rhinoceros 7/Grasshopper. Image by the author.
Figure 3. 3D rendering of one of the six ‘impossible nests’ in Rhinoceros 7/Grasshopper. Image by the author.

The work poetics is woven in a way it can help us transcend our technoscientific fetishes for precision and certainty and perhaps develop a model for knowledge production based on a spirited cosmology and intimacy over cruelty and domination. To generate the data-sculptures the choice for the raw data was the audio extracted from short videos shared on YouTube from the following birds' birdsongs: Alagoas curassow (Mitu mitu), Guam kingfisher (Todiramphus cinnamominus), Guam rail (Hypotaenidia owstoni), Hawaiian crow or ʻalalā (Corvus hawaiiensis), Socorro Dove (Zenaida graysoni), Spix's macaw (Cyanopsitta spixii). Dialoguing with the Sub-theme “Symbiotic maginaries: inventing worlds”, the work highlights dystopian aspect of the use of data coming from ‘nature traces’ of 6 (six) bird species declared extinct in nature’, bringing the opportunity to meditate on how multispecies collaborations in media arts can help rising consciousness in our community on critical ecological issues.

Figure 4. 3D rendering of ‘Data-Phantom 02: Guam kingfisher (Todiramphus cinnamominus) Impossible Nest in Rhinoceros 7/ Grasshopper. Image by the author.
Figure 5. 3D rendering of ‘Data-Phantom 02: Guam kingfisher (Todiramphus cinnamominus) Impossible Nest in Rhinoceros 7/ Grasshopper. Image by the author.

The understanding of a nest as an artifact – that it is part of the information embedded in the organism that created it—was explored by Richard Dawkins in the early 1980s in "The Extended Phenotype". Dawkins argued that ⁹ comparatively, since there are genes involved in the animal body’s morphogenesis (such as the ‘homeobox gene’), there must be genes whose phenotypic expression is the bird’s nest architecture—or the spider’s web, for example. Mike Hansell and Raith Overhill, in Bird Nests and Construction Behavior, considers that ¹⁰ gene expression in sort of "extended phenotypes" may be more complex than pondered by Dawkins, observing what an organism builds and how can be taken as an expression of the genetic information embedded in the organism.

It is relevant to notice that nests building strategies as complex learned behavior in birds, as it has been observed and documented by scientists in different contexts, includes lifelong learning to meet challenges in hostile environments and can sometimes lead to dystopian innovations.

In a paper published on July 11, 2023, Auke-Florian Hiemstra, PhD candidate in Evolutionary Ecology at the Naturalis Biodiversity Center, The Netherland, together with his team, discusses the use of "anthropogenic mass" instead of "living biomass" as alternative nesting "materials" by urban birds when building their nest, having birds being observed using"[...] bird deterring materials like anti-bird spikes as nesting material [...].”¹¹

Figure 6. The Antwerp magpie nest constructed with anti-bird spikes, seen on the fork of a branch in a sugar maple tree on 25 October 2021. Image by Hiemstra et al ¹¹.

Although bird spikes are designed to keep birds away from nesting and perching, there have been documented examples of sharp, industrialized products being used by birds as nesting material. According to the authors,¹¹ published reports of wire nests date back to 1933, and the Kansas Barbed Wire Museum proudly exhibits a corvid nest made of barbed wire. The team of researchers guess that, from the evidences, members of some bird families are able to remove spikes from buildings and learn how to handle them and might have discovered a new potential for this human-made product.

It is important to observe that, this recent publication ¹¹ focuses exclusively on corvid behavior, describing all currently known nests of carrion crows and European magpies made out of anti-bird spikes, discussing the possible implications of the usage of this bird repelling product as an alternative 'material' for nest construction.

According to the researchers, ¹ the evidences may further support the notion that the use of these sharp anthropogenic 'materials' can be used to enhance nest defense since, as observed, nest building by magpies is sometimes disrupted by carrion crows attempting to prevent dome construction, "as a dome will hide the content of the nest from attacks from above."

Emptiness Ecologies

The series of data-sculptures “Data-Phantoms: Impossible Nests (Memories Post Extinction)” (2022) were commissioned by Mat Keel and Liz Lessner for the collective exhibition “Emptiness Ecologies," at "Yes We Cannibal"—an institution for experimental art and social practice they co-founded and direct in Baton Rouge, Louisiana., U.S..

Opening on Saturday, December 17, 2022 and displaying until the closing reception on February 4, 2023, YWC directors considered this to be their most ambitious gallery show yet.¹², ¹³ According to the curators¹³, “Emptiness Ecologies" was crafted to serve as a creative audit of Ecology as an effusively privileged nexus that ostensibly offers new ways to speak about social and environmental relationships and events. Over the course of six weeks, the idea was to build collectively “[...] a critical space within which to explore those questions and current moment, taking stock of the conceptual uses of ecology and the nature of reality, itself.” ¹³

Exploring "emptiness" as a core topic, "Data-Phantoms” dialogues with the curators’ intention of ¹³ making sense of "cognitive dissonance in a moment of stunted political life and ecological cataclysm, including living through the sixth great extinction."

This multimedia show at YWC featured new works of art and writing, including installations by multimedia artist Dawn Dedeaux, collages by artist and Houma nation storyteller Monique Verdin, and works by U.S. based artists and international figures, including a video piece by Duke and Battersby of Syracuse, New York, paintings by Tokyo native Chihiro Ito of Brooklyn, New York, an experimental film by Anna Scime of Buffalo, New York and prints of 3D renders from the series of data-sculptures “Data-Phantoms: Impossible Nests (Memories Post Extinction).”

Figure 7. Emptiness Ecologies show curated by Mat Keel and Liz Lessner at YWC Yes We Cannibal Government Street gallery, Baton Rouge, Louisiana, US, from December 17, 2022 to February 4, 2023. Image courtesy Yes We Cannibal.
Figure 8. 3D rendering of ‘Data-Phantom 06: Spix's macaw (Cyanopsitta spixii) Impossible Nest in Rhinoceros 7/ Grasshopper. Image by the author.
Figure 9. 3D rendering of ‘Data-Phantom 06: Spix's macaw (Cyanopsitta spixii) Impossible Nest in Rhinoceros 7/ Grasshopper. Image by the author.

Nest-Works

The explorations that converge in the series of data-sculptures comes from the conversations orchestrated by Amy-Claire Huestis for the ‘poetic wilding of all-too-human spaces’ she called "nest-works." As Huestis explains, “Nest-works began with an experimental panel for the 2021 College Art Association Conference, called "Co-Making this World." The experimental session was modeled after the nest of a bird, a Black-Capped Chickadee. As this cavity-nester builds a home of disparate materials, the panel of artist-researchers built a session of disparate theories and practices, as we considered relationships with world-systems that are in the process of making (such as the nest of the chickadee).” ¹⁴

The material produced during the collaboration was curated and organized for Technoetic Arts academic journal as nest-work of research material, considering new models for knowledge and creative production as “[...] an entanglement of short essays made by artists working with a common pattern, framing eco-poetics on collaborative and participatory processes with the non-human/more-than-human.”¹⁴ The article “Data incarnations: Nesting complex inherited and learned behaviours”¹⁵ documents the stimulated dialogues and outcomes starting from Huestis initial invitation. 

Final Considerations

In this collaboration-as-fantasy that expands initial experiments in dialogue with a group of artists to discuss aspects of interspecies and human-non-human collaboration, diving into dystopian aspects of ecological events, the creative effort invited the ISEA 2023 community to navigate our symbiotic imaginaries from a structural perspective. If in nature nests can be incarnations of birds’ complex inherited and learned behavior in dialogue with other species and the environment, the experiments having as an outcome the collection of 6 (six) data-sculptures—the ‘impossible nests—can embody a tiny sample of this complexity when an artist invites birds that gone extinct in nature, thought the digital audio memories—"ghosts" of their actual, live performed bird songs—into a process-oriented by the intention of algorithmically generate data-sculptures from these birdsong transduced into sequential numerical variation.

Recalling Edgar Wind’s notion of incarnation, the intention highlights the "analog-digital" continuum as inherent to nature’s complex morphogenetic strategies and not as detached from it.¹⁵ The resultant data-sculptures explores sublime aspects of data-visualization aesthetics in a dystopian endeavor manifested in the irregular and intricate geometry of the resultant objects referring to the impossibility of birds in wrecked ecologies, deprived of proper symbiotic encounters, to socialize, collaborate, teach and learn how to build successful nests and survive.

al Considerations

In this collaboration-as-fantasy that expands initial experiments in dialogue with a group of artists to discuss aspects of interspecies and human-non-human collaboration, diving into dystopian aspects of ecological events, the creative effort invited the ISEA 2023 community to navigate our symbiotic imaginaries from a structural perspective. If in nature nests can be incarnations of birds’ complex inherited and learned behavior in dialogue with other species and the environment, the experiments having as an outcome the collection of 6 (six) data-sculptures—the impossible nests—can embody a tiny sample of this complexity when an artist invites birds that gone extinct in nature, thought the digital audio memories—"ghosts" of their actual, live performed bird songs—into a process-oriented by the intention of algorithmically generate data-sculptures from these birdsong transduced into sequential numerical variation.

Recalling Edgar Wind’s notion of incarnation, the intention highlights the "analog-digital" continuum as inherent to nature’s complex morphogenetic strategies and not as detached from it.¹⁵ The resultant data-sculptures explores sublime aspects of data-visualization aesthetics in a dystopian endeavor manifested in the irregular and intricate geometry of the resultant objects referring to the impossibility of birds in wrecked ecologies, deprived of proper symbiotic encounters, to socialize, collaborate, teach and learn how to build successful nests and survive.

References

1 Davi Kopenawa, Bruce Albert, The falling sky: words of a Yanomami shaman, translated by Nicholas Elliott and Alison Dundy, Cambridge, Massachusetts London, England, the Belknap Press of Harvard University Press, 2013, 61

2 Crates, Ross, Naomi Langmore, Louis Ranjard, Dejan Stojanovic, Laura Rayner, Dean Ingwersen, and Robert Heinsohn, “Loss of Vocal Culture and Fitness Costs in a Critically Endangered Songbird, Proceedings of the Royal Society B: Biological Sciences, The Royal Society, 2021.

3 Rebecca Heisman, “Six Extinct Birds Whose Fame Lives On: The Dodo, Passenger Pigeon, and More”, Bird Calls Blog of the American Bird Conservancy, September 06, 2022, accessed November 10, 2022, https://abcbirds.org/blog/extinct-birds/

4 Michelle Ward et al, “Creating past habitat maps to quantify local extirpation of Australian threatened birds”, Environ. Res. Lett. 17 024032, 2022, accessed November 10, 2022, https://iopscience.iop.org/article/10.1088/1748-9326/ac4f8b

5 Elsie C. Collias, Nicholas E. Collias, “The Development of Nest-Building Behavior in a Weaverbird”, The Auk, Published by: University of California Press on behalf of the American Ornithologists' Union, Vol. 81, No. 1 (Jan., 1964), 42-52, accessed November 10, 2022, http://www.jstor.org/stable/4082609

6 A. Brosse, “Social organization and nest-building in the forest weaver birds of the genus malzmbus (ploceinae)”, IBIS International Journal of Avian Science, Willey Online Library, Volume 120, Issue1, January 197, p.27-37, accessed November 10, 2022, https://onlinelibrary.wiley.com/doi/10.1111/ j.1474-919X.1978.tb04996.x

7 Edgar Wind, Experiment and Metaphysics: Towards a Resolution of the Cosmological Antinomies, trans. E. Cyril, Abingdon, Routledge, 2001.

8 Edgar Wind, “Some points of contact between history and the natural sciences, in R. Klibansky and H. J. Paton (eds), Philosophy and History: Essays Presented to Ernst Cassirer, Oxford: Clarendon Press, 1936, 255–64.

9 Richard Dawkins, The Extended Phenotype: The Long Reach of the Gene, Oxford, Oxford University Press, 1999.

10 Mike Hansell, Raith Overhill, Bird Nests and Construction Behaviour, Cambridge: Cambridge University Press, 2000.

11 Auke Florian Hiemstra, Cornelis W.Moeliker, Barbara Gravendeel, Menno Schilthuizen, “Bird nests made from anti-bird spikes”,Deinsea, Issue 21, 11 July 2023, 17-25.

12 Zane Piontek, “Yes We Cannibal’s New Ecology Themed Exhibit its’ most ambitious yet”, 225 Baton Rouge, December 20, 2022, https://www.225batonrouge.com/things-to-do/yes-we-cannibals-new-ecology-themed-exhibit-its-most-ambitious-yet

13 The Advocate, “Yes We Cannibal opens multimedia exhibit featuring work by regional and international artists”, The Advocate, December 15, 2022, https://rb.gy/yrgnh

14 Amy-Claire Huestis, “Nest-works”, Technoetic Arts: A Journal of Speculative Research, 19:3, Nov 2021, p.227-241, https:// doi.org/10.1386/tear_00065_1

15 Clarissa Ribeiro, “Data Incarnations: Nesting Complex Inherited and Learned Behaviours.” Technoetic Arts: A Journal of Speculative Research, 19:3, Nov 2021, 253-26, https:// doi.org/10.1386/tear_00067_1

Acknowledgements

I’d like to thank Amy-Clair Huestis for starting the conversation around ‘nest-works’ and Mat Keel and Liz Lessner for the invitation to contribute to YWC Yes We Cannibal “Emptiness Ecologies.”

Author Biography

Clarissa Ribeiro, Ph.D., is a multimedia artist and researcher with an interest in cross-scale information and communication dynamics that impact and shape macro-scale emergent phenomena. In her more recent projects, she explores the metaphysics of information-visualization in subversive morphogenetic strategies that welcome the animistic to navigate ecologies as cosmologies. Chair of the first Leonardo/ISAST LASER talks to be hosted in Brazil/Latin America since 2017, she is an active member of the UCLA Art|Sci Collective and was recently awarded the Roy Ascott Studio’s Pete Townshend Endowed Senior Lectureship in Performative Technoetics. She is an Associate Professor of Technoetic Arts at SIVA Shanghai Institute of Visual Arts, DeTao Masters Academy, Roy Ascott Studio in Shanghai and represents the University of Fortaleza in China. She has widely published in journals and conference proceedings and her work has been exhibited worldwide. She has been serving as a reviewer for Leonardo Journal and the Technoetic Arts Journal, the Leonardo Abstracts Service (LABS), contributing as a member of international conferences and symposium committees. 

Black Box versus Black Bloc

Renée Ridgway

rr.mpp@cbs.dk / rridgway@cc.au.dk

Abstract

With around 5.5 billion requests per day, Google is the most used search engine worldwide. Google Search identifies users online by collecting personal data—including an IP address, yet when using the Tor browser, a users’ IP address remains obscured. Black Box versus Black Bloc employs Alexander Galloway’s eponymous essay to structure the effects of Google Search (The Personalised Subject) compared to that of the Tor Browser (The Anonymous User). Departing from the "data subject," I adopt the internet protocol (IP) address as an organisational hinge to show the effects of search on (us)ers—"subjectivities of search" and "agencies of anonymity," organised into ‘collaborative collectives’ according to degrees of human-algorithmic interaction. The key difference is that I choose to be in the "anonymous Tor collective," trusting my privacy to unknown human actors instead of putting trust in Google that assigns me to particular groups through their non-transparent process of collaborative filtering, without human agency.

Keywords

Search engines, Google, Tor, personalization, anonymity, black box, black bloc, subjectivity, agency.

DOI

10.69564/ISEA2023-80-full-Ridgway-Black-Box-vs-Black-Bloc

Introduction

"Many people I have talked to have mentioned that they are careful about what they type into search engines because they know it’s being recorded and that limits the boundaries of their intellectual exploration."¹ 

Since the rise of information technologies during the past 50 years, searching online has become one of the most popular human activities. With around 5.5 billion requests per day, Google is the most used search engine worldwide. Whereas ubiquitous computing described how electronic devices are interconnected, thereby making the communication of data pervasive,² the sociotechnical organization of "ubiquitous googling"³  is now a daily habit where "users" searches produce data that make users findable, even as they wander.⁴ With their IP (Internet Protocol) address collected as well as keyword searches and search histories, the past 20 years users frequently employed Google for information, medical advice or even research, thereby creating vast amounts of data. This "database of intentions" is now "a massive clickstream database of desires, needs, wants, and preferences that can be discovered, subpoenaed, archived, tracked, and exploited for all sorts of ends."⁵  One of the ends is personalisation as currency, where users pay with data and what they receive are the currency of the web— customised URLs, based on their searching habits (browsers, location, histories).⁶ 

Besides data collection, the incorporation of user interaction—keyword search queries, impressions and clicking on links—is tied to an economic logic, advertisement, which facilitates ranking and recommendation on the part of the platform that intervenes.⁷ This type of platform capitalism⁸ is disrupting entrenched business models by highlighting as well as hiding—downplaying the labour of users and free data as platforms promote the horizontality of their services—yet they are not flat. Google Search "is a personal information economy where the standard exchange is service for profile"⁹ part of the capitalistic "service/dataprofile/ advertising complex."¹⁰ Thus, over time, Google transferred itself into an advertising company, producing not just search results, but capitalising on "informational rationality of generating value from advertising and audience labour."¹¹

Through human-computer interaction with their ubiquitous googling, human bodies and their cognition, affect and interests have become valuable resources as data-subjects. With the "extraction activities" of user data by Google, a new asset class was created, or "surveillance assets"—providing a genuine market exchange.¹² Yet this data shapes users reciprocally through human-computer interaction. This results in fragmentary user subjectivity in deterritorialized spaces and it is the transformations of search subjectivities, where "bodies are mostly addressed at the level of affect and cognition"¹³ that is cause for concern. The thoughts and values of users that are inscribed in queries are transferred into predictions, which subsequently produce not only products for corporations but incites changes in users’ search behaviour. Thus, the habit of search by users produces enormous amounts of data, generating profits for Google but also facilitating the recursive feedback loop that organises (us)ers.¹⁴

However, there are ways to circumvent personalization and to reimagine search. Building upon the notion of "cyberspace privacy" that applied encryption technologies during the 1990s,¹⁵ at the beginning of the last decade the ‘privacy turn’ took effect. The Nymwars (2010-2014) debated pseudonymity (the ability to have hidden identities when online) and the Snowden revelations (2013) exposed Five Eyes surveillance on citizens and corporate collusion. This resulted in an increase in usage of privacy technologies such as Tor (The Onion Router), an anonymity p2p browser that is a means to search online without divulging a user’s IP address. Although controversial, there are many situations and "rationales for anonymity"¹⁶ which are dependent on the context and situation, offline as well as when searching online.

This paper draws on my small data sets from an "experiment in living"—searching as a personalised subject with Google and an anonymous user with Tor. Departing from the data subject, I use the IP address as an organisation hinge to demonstrate the effects on (us)ers through a discourse analysis and a diagram. I combine these effects with Alexander R. Galloway’s Black Box, Black Bloc,¹⁷ which provides a conceptual (and technological) framework to synthesise the results from these two search methods. Derived from the effects, I then explain how each ‘collaborative collective’, Subjectivities of Search (Google) or Agencies of Anonymity (Tor) differs, based on degrees of human and non-human interaction. 

Figure 1: Experiment in Living: Google vs Tor

Experiment in Living: Google vs Tor

In order to discover how Google search organises (us)ers, I designed an "experiment in living"¹⁸ and collected data on myself in my office. (Figure 1) This small study facilitated my understanding of the behind-the-scenes constellations of agents (protocols, algorithms and myself) that determined my search results. The research was conducted on two computers: one using Google search in a Firebox browser on a completely ‘personalized’ Mac (signed into a Gmail account, no ad blocking plug-ins, no incognito, etc.). With personalisation, Google customizes its algorithms in regard to IP address, keywords queried, search history and browsing habits to offer relevance and recommendations. The other computer is a Lenovo PC with a Debian operating system running the Tor browser that ostensible offers anonymity by hiding the IP address.

I used a set of keywords selected from texts I was reading at the time (2016) and I call this dataset Re:search: Terms of Art: Accelerationism, Aesthetic Turn, Anthropocene, Artistic Research, Contemporaneity, Creative Industries, Cultural Entrepreneurship, New Aesthetic, Object Oriented Ontology, Performativity, Post Digital, Post Humanism, Post Internet, Post Media, Transmedia. I gathered the data manually (I did not programme it to scrape the data) and saved the entire web page of the 1st page of results, along with the 10th, 20th, 30th, 40th and 50th etc. pages for the data set. This empirical ‘experiment in living’ in my office enabled me to capture two forms of address when searching online, one as a personalised subject with my IP address recognized by Google contrasted by being anonymous online with Tor, where my IP address is hidden.

Black Box, Black Bloc

One of the ways the protocological ‘determines control after decentralisation’ is through the configuration of the relationship between TCP/IP and DNS (Domain Name Server), as they are "political technologies." ¹⁹ TCP/IP is now the standard internet protocol suite and DNS, which runs parallel to HTPP in the application layer (7th), is responsible for translating the domain names into numerical IP addresses in order to identify devices and locate them within the network protocols. Google Search facilitates not only communication between parties to deliver search results (and advertisements), but the identification of ‘subjects’ and data collection, including the IP address. Conversely, with the Tor browser, the IP address is part of the protocol that facilitates the transport of data but the IP address is not revealed because of layers of encryption within the Tor p2p network.

By applying the IP address as an organisational hinge, I show the effects of search engines on (us)ers, drawing on my results as a "personalised subject" and as an "anonymous user." With a ‘personalised subject’, the black box and the Intellectual Property (IP) of Google’s proprietary search algorithm is a form of "[in]visibility management."²⁰ situated within the "media arcane."²¹ This blackness of the black box is also found in the Black Bloc that is analogous to tactics of obfuscation,²²  such as the Tor Browser, which, by obscuring the user’s IP address, facilitates an anonymous user. I apply Alexander R. Galloway’s Black Box, Black Bloc as a conceptual (and technological) framework to synthesise results from my two methods of search: "The black box: an opaque technological device for which only the inputs and outputs are known. The black bloc: a tactic of anonymization and massification often associated with the direct-action wingof the left. Somehow these two things come together near the end of the twentieth century. Is there a reason for this?"²³

In the following I show the effects of these two methods of searching through a progression of human/algorithmic interaction. The structure of both these effects begins with search algorithms interacting with myself as a researcher collecting data online and as a data subject, who is protected by law. Since May 25, 2018 the GDPR (General Data Privacy Regulation) has been implemented in Europe, which regulates the collating, processing, storage and transmission of personal data of EU citizens, or data subjects.²⁴ According to Article 4, the "data subject" is an end user whose personal data can be collected through ‘direct identification’ with an IP address. It is the recognition of citizens as ‘data subjects’ by their IP address that facilitates certain effects and the organization of those searching online—through degrees of personalisation (black box), or not, with degrees of anonymity (black bloc).

Figure 2: Subjectivities of Search

Black Box (Subjectivities of Search)(Figure 2)

During my experiment in living, I interacted with Google Search algorithms and used tools, such as my computer, as memory extensions or hupomnemata, to ‘note down’ and collect data on myself—keywords and search results. This Technology of the Self can be used as an instrument to analyse the relationship between the subject and truth, where the ‘personalised subject’ explores power constructs—how the subject constituted itself in one form or another, where "power is games of strategy."²⁵ As my search histories are constantly collected by Google Search, the ‘personalised subject’ is not a substance but a form, which is "not primarily or always identical to itself" as it is changes in different contexts and situations. Through diverse practices such as online interaction with search algorithms as "truth games," this modern day "technology of the self" transforms me as a (data) subject. Personalised subjects google themselves at some point, either to measure their attention economy—where everything is based on visibility—or to see what has been written or published about them by search results and that they are indexed.²⁶ It has become the meter to measure success. Appearing higher in Google’s ranking, adding to one’s visibility, is a particular kind of attention seeking that embodies Foucault’s figure of the "Homoeconomicus," or economical man.²⁷

Foucault was interested in the subject and more specifically, "the way a human being turns himself into a subject" and part of his scholarship looks at the history of discipline and what he came to term "biopolitics." "Foucault uses the term ‘biopolitics’ in order to elucidate how political power is carried out on every aspect of human life, making individuals and the Homoeconomicus someone who is eminently governable."²⁸ Although the Homoeconomicus is often considered a rational agent in pursuit of self-interest, these subjectivity-defined ends are part of an economic civil society that operates through production and exchange, which is part of the technology of liberal governmentality. Calculated practices (such as Google Search), permit individuals to govern themselves, which epitomises the biopolitical and lies at the core of neoliberalism. Google search facilitates online tracking and (self) surveillance, simultaneously optimising searching subjects who are also "[n]eoliberal subjects—small sovereigns—are always searching, rarely finding."²⁹ Through this interaction, the subject is both recognised, and subject to, the law.

In the early days of neoliberal capitalist ideology, before it was defined as such, Louis Althusser articulated forms of address through the framework of ideology, which he deemed "interpellation." With the constitutive process of interpellation, this ideology is recognised by the individual’s acknowledgement of becoming a subject that complicates their domination and subjugation. The classic example is that of Althusser’s policeman who shouts at a passer-by “hey, you there!” in public where the individual then responds by turning around. "[B]y this mere one-hundred-and-eighty-degree physical conversion, he [sic] becomes a subject."³⁰ Previously it was the police who asked the question: "Hey you there?" Nowadays ‘personalised subjects’ enhance the power structures of Google by recognising themselves as subjects when searching online, who are interpellated as "subjects" by automatically acknowledging the ideology of Google Search by deciding to use it. By clicking on links mostly found on the first page of Google, Brin and Page’s "Trusted User" interacted with the search engine and reinforced this ‘preferential attachment."³¹ The Trusted User thereby actuates the ‘relevance’ and ‘quality’ of the search results by supplying feedback and data to Google, which it incorporates into subsequent search results.

In this orchestration of individuals and algorithmic actors on the stage of the internet, the efficacy of ‘online display advertising’ comprises not only the agency of human actors but algorithmic ones. By interacting with content on various websites, including Google Ads, visits are recorded by tracking cookies that are instilled by another actor, the ‘ad server’ and the ‘individuated subject’ produces a shadow. In the flow and ‘circulation of agency’ through its hyperlink journeys and its search queries...

"The impressionable subject is produced as what is being visited and what is being searched."³² Searching subjects provide data to Google, facilitating the tracking of "YOUs" that "resonates strongly with Louis Althusser’s theorization of ideology," which "represents the imaginary relationship of individuals to their real conditions of existence."³³  Whether there is "caring of the self"³⁴ or caring of the network (the other YOUs), with the "interpellated subject’s" interaction with platforms such as Google Search, it becomes an effect of algorithmic ideology.³⁵ Sorted together with others like them, versions of multiple data selves are fed back through never-ending and recursive algorithmic loops. Comprised of a complexity of user subjectivities, the YOU addressed by Google search is, crucially, both singular and plural yet "[i]n its plural mode, though, it still addresses individuals as individuals,"³⁶ reflecting what has come to be called the Digital Subject.

The Digital Subject is an abstracted persona created from various data, records and archives, aggregated together to form what comes ‘after the subject, requiring new ways to understand how it connects to the subjectivities of living persons, something that is mapped onto living persons. With profiles in constant flux, temporalities and degrees of correlation supposedly enable better personalisation and "[i]t is also through the distance that digital subjects become more or less personalised or multiple—put together and disaggregated." ³⁷ This ‘real-time’ collation of data on the "data subject" a.k.a. a Trusted User of Google Search, creates algorithmically produced entities of ‘Data Dust’ as an object of knowledge, which is comprised of dividuals ³⁸ data shadows³⁹ data doubles⁴⁰ and data derivatives. ⁴¹ These simulacra are not representative of real ‘individuals’. Instead, they are encompassing elements, bits, points—"the fragments of registered behaviour, which are extracted from the flow of data for specific purposes."⁴² These entities of Data Dust find others like themselves not in the "meat space" apartments of urban cities, or in towns scattered across flatlands, mountains andvalleys, but in worldwide data centres.

These numerous entities of the subject—Subjectivities of Search—are constructed by the very data that ‘personalised subjects’ give away when habitually searching online. Nowadays the subject ‘is potentially reduced to the pure $ (the divided subject)’ as a ‘Machinelike Other’. ⁴³ Moreover, the construction of the ‘digital subject’ stems from the continuous production of inscribed personal data, resulting in ‘new forms of subject construction that arise out of computational procedures and are employed by various forms of power to distinguish, map, and capture not only subjectivities, but also non-humans and physical things that inhabit the world’. ⁴⁴ Engaging with algorithms of Google Search, this hybrid form is Haraway’s cyborg—a site of contestation, challenging any traditional demarcation line between the human and the machine and instead defers to a space of patterning through the very technologies that comprise the database. Expressed otherwise, with the reconfiguration of subjectivities through technics and distributed cognition, ‘human bodies as cyborgs—as human machine systems—are in turn systematically combined into modes of “Cyberorganization”’. ⁴⁵ As with the cyborg, Cyborganization is not an extension of the human agent but perhaps rather ‘the agent is an extension of the machine’ and ‘remains forever unfinished’, simultaneously controlling the flow of information back to searching subjects. ⁴⁶

Figure 3: Agencies of Anonymity

Black Bloc (Agencies of Anonymity)

In the preceding section, I explained the effects, Subjectivities of Search, determined by the Black Box (Google Search). I return now to the other half of Blackness, the Black Bloc and its effects, based on my method of searching with the Tor browser (Figure 3).

The ‘Online Disinhibition Effect’ argues that online behaviour differs from ‘real’ world behaviour, emphasising the notion of invisibility because when people communicate with each other online they mostly do so in written form, as they do not see each other and thus are more apt to express themselves without inhibition. ⁴⁷ In regard to the Snowden revelations, when asked whether spying on citizens is justified and whether citizens should be willing to exchange privacy and anonymity for increased security, the predominant rationalisation is that state surveillance is positive as it protects people from terrorism, often ending with ‘I’ve Got Nothing To Hide’. However, people often feel immune to surveillance because they haven’t committed a crime, yet ‘[a]ccountability is commonly raised as one of the reasons behind which people should provide identifiable information in online settings. When people prefer not to share their names, they’re assumed to have something to hide’. ⁴⁸

In the early days of the web, ‘[t]he ability to participate anonymously or, as was and remains far more common, pseudonymously was an integral part of why Barlow and other net utopians saw the Internet as valuable’. ⁴⁹ In the 1990s users had ‘monikers’ when they signed into chatrooms, later on they created email addresses for each different service they signed up for. Besides ‘traffic analysis’ and ‘mixed networks’—the basis for Tor encryption— Chaum’s other contribution to knowledge included ‘digital pseudonyms’, which is ‘a public key used to verify signatures made by the anonymous holder of the corresponding private key’. ⁵⁰ Weisner, described ‘digital pseudonyms’ as a means to insure privacy within networked societies. ⁵¹ Instances of Pseudonymity do not mean that one is completely anonymous, rather various pseudonyms can be linked together to form either an ‘online identity’ or Online Persona, which ‘had come alive in a new social practice: the virtual world as context for explorations of identity’. ⁵² One reason to build these constructed online personas is that they ‘also offer sites of reinvention, liberation, and play. Fake accounts and performed identities testify to that’. ⁵³

Persona building goes hand in hand with crawling the Dark Net in that it is replete with pseudonyms and false identities. In attempts to reach ‘total anonymous freedom’ Dark Web Social Net (DWSN) members ‘customise their aliases, avatars, pseudonyms through widgets and in such a way that ‘[o]ne does not use a fake account every time; one builds a persona’. ⁵⁴ When all of these pseudonyms are collated by malevolent actors to construct personas, together they can form an online profile. However, there are tactics to prevent this from happening. K-anonymity: A model for protecting privacy would alter released information based on scaling, where the greater number of candidates, the ‘more ambiguous the linking, and therefore the more anonymous the data’. ⁵⁵ Analogous to the increased strength of anonymity with a larger amount of Tor users, the efficacy of ‘scaling’ tactics with K-anonymity are nowadays limited due to fact that the storage of data continues to become easier and cheaper as computational power increases and companies can examine this information in ‘real time’.

In regard to these problems, Obfuscation is necessary in an era of online tracking and its use-value is in ‘mitigating and defeating present-day digital surveillance’. ⁵⁶ Applying tactics of secrecy and deception to combat asymmetrical relationships of power, the guide shows various methods of resistance that, while they might be considered ‘weapons of the weak’, afford some amount of autonomy for the user. Moreover, these tactics prevent various shields of recognisability—how signals or information could be interpreted, transmitted or shared by enemy parties. With personalisation, the IP address plays a crucial role in the identifiability of the user along with the ever-increasing amount of data collected and shared between search engines and third parties. As a reaction to the public realisation that search companies (notably Google) were logging, storing and analysing the search query logs of individuals, ‘TrackMeNot’ (TMN) is ‘designed to achieve privacy in web search by obfuscating users’ queries within a stream of programmatically generated decoys’. ⁵⁷ Another ‘tactical media’, AdNauseum is a browser extension that floods Google Search with false queries, clicks and likes all ads, concomitantly visualizing the ads. ⁵⁸

Although obscuration offers many shades of privacy, a ‘true inner self’ can only emerge in anonymity. ⁵⁹ The shift from the ability to carry out actions ‘namelessly’ in offline space ‘is not as the end in itself of anonymity’, rather it is about how users can conduct themselves anonymously online, whether a person is identifiable (or not) as themselves and if they are untrackable. This ‘Unreachability’ is where absence can be used for positive expression in the complex and dynamic computer-oriented society. Already in 1999 Nissenbaum presciently predicted the era of ‘surveillance capitalism’ where every atom of data is collected, kept and analysed: ‘Information technology has made it possible to track people in historically unprecedented ways. We are targets of surveillance at just about every turn of our lives’. ⁶⁰

With Agencies of Anonymity, users can consciously protect themselves against the 24/7 tracking of surveillance capitalism through tactics of obscuration yet this ‘unreachability’ reflects an acquired knowledge—having the ‘tech savvy’ to use Tor. Spurred by agencies of ‘control’ over one’s data, this Techno-Elitism is a type of self-determination⁶¹—knowing how to obtain anonymity and remain hidden from search engines and governmental agencies. Furthermore, there are those ‘techno-elitists’ who do not wish to be indexed by Google and desire to remain ‘unreachable’ to search engines, or delete their information after searching with Tor. TAILS is an operating system that has Tor already configured, which, installed on a USB stick, boots the computer and everything is deleted upon ejection.⁶²  Or, if one desires to be undetected and shop, the tactical media cum art project Random Darknet Shopper (2014- ongoing) by! Mediagruppe Bitnik is a programmed autonomous bot that searches the Dark Net and carries out purchases, completely anonymously, albeit through code.⁶³ However, this Algorithmic Anonymous User raises an important question regarding the accountability of actions by non-human actors: can one prosecute and punish an algorithm that commits a crime, in this case purchasing illegal substances and sending them to the St Gallen Kunst Halle by post for an art installation?

Collaborative Collectives: Agencies of Anonymityvs. Subjectivities of Search

Returning now to my results from comparative searching using Google’s personalisation and Tor’s anonymity, I learned that Tor delivers ‘Google-like’ results with its default search engine, only without personalisation and targeted ads because there is no captured locative data, search history or IP address. However, if I assume that both is the case—on the one hand, I am assigned as a Tor user and on the other that Google assigns me to groups of people like me on its databases (an assumption I cannot fully prove with my experiment but is the most likely scenario to explain its outcomes)—the original framing of my experiment has to be specified. Instead of a personalised search as opposed to an anonymised search, I would have, in fact, conducted a Google search that is a collective-of-users-like-me versus a collective-of-all-Tor-users. At stake, therefore, are two collectives that take different forms.

In the collective-of-users-like-me it is Google’s algorithms which construct the categories I am part of and assign me to this or that collective (e.g., collaborative filtering). I have no access, no knowledge and no agency in regard to the collectives which I am made part of via Google. The forces (identification markers: IP address, search histories, cookies, supercookies and locative data) that sort me into a collective and the collectives that I am organised into—the categories that Google sets up and assigns me to, are not transparent to me. Moreover, Google collects my individual search activities and, in future scenarios, will probably state that they ‘personalise’ search even further based on data collated in the past and present.

Tor’s collective, on the other hand, is both an anonymity network and a browser—Tor is mostly Firefox code (95%) that incorporates patches to Firefox ESR (Extended Support Release). The Tor collective is at least partially known to me as university labs worldwide run the major nodes but I do not know who is running the relays (it is an anonymised network). I can, however, look at the ‘exit address’ list, which is constantly updated and shows the IP address, though I cannot identify the users.⁶⁴ By joining the Tor collective I decided to trust the exit node operators, also in regard to my ‘expectation of privacy’. The key difference is that whereas Google organises me into particular collectives through their non-transparent process of collaborative filtering, I decided to be organised into the ‘anonymousTor collective’. (Figure 4) 

Figure 4: Agencies of Anonymity vs. Subjectivities of Search

Both search collectives, the one determined by Google’s algorithms as well as the one created by the choice to use Tor, add to specific filter bubbles. The filter bubbles are, however, structurally different: in the case of the bubble produced by Google’s algorithms, Google collects the data of its users and incorporates user feedback into subsequent search results. When I search for different things, I am merged into different clusters with other people ‘like me’ or ‘YOUs’. I would then add to the feedback loop by continuously adding to my own personalisation by clicking on the links that are delivered to me as results. I do not have access to the Google collective itself—I am constantly switched into a different cluster by an algorithmically organised process that I have no control over and cannot adjust. There are also constant updates and tweaking of signals being carried out on the algorithm. Therefore, I propose that various degrees of Google Search’s personalisation organise the corporal ‘data subject’ into effects: Subjectivities of Search. With the Black Box, as Google’s personalisation increases so does the amount of computational agency, at the same time the degree of anonymity (privacy) decreases.

The filter bubble of the Tor users, on the other hand, is one where I stay in the same group that shares the same filte—no matter how much I change my search behaviours (what I click on or not). As a Tor user, the variable is what Tor uses as their default search engine (Startpage, Disconnect Search or presently DuckDuckGo) and if this default (still) delivers Google Search results without locative data. Therefore, the results of my small data experiment postulate that the user is assigned the category of ‘Tor user’, which can be seen from outside the Tor network. When I use Tor I am part of a p2p anonymity network, which increases in strength the more users use it. Unlike Google Search, privacy-enhancing technologies and diverse settings

enable the user to organise themselves into effects: Agencies of Anonymity. With the Black Bloc, as the amount of anonymity (privacy) increases with Tor so does human agency, at the same time the degree of personalisation decreases. If I compare these two search processes, with the individualisation of the pseudo-autonomous objects of Google’s personalisation, my ‘data dust’ is atomized and fractured—as a ‘digital subject’ I have no agency to decide where I am assigned. However, to partake anonymously in a p2p-collective individuates me more than personalisation does. Bernard Stiegler’s entire pharmacology of care revolves around this new ecology, which can be understood to be threefold, as a ‘re-articulation of psychic, collective and technical individuation’. ⁶⁵ At stake is an individuation in the sense of Stiegler’s reading of Simondon—an individuation that is marked by being collective and psychic alike, where the genesis of how an object comes to exist is through operations of individuation, or ‘ontogenesis’ and a living being exists in a state of becoming between individuations, never in isolation but in collective, social as well as psychological constellations. The Tor browser (p2p network) embodies this individuation of collective singularities, which are dependent on the other. ‘After all, an individual exists and is only capable of individuating as a result of the relations it establishes with others and that others establish with it.’ ⁶⁶  Phrased otherwise, the possibility of choosing the individuating Tor collective over the individualisation of Google Search might be a modest resolution to the problem of ‘how one would define a singularity that could be a collective singularity’. ⁶⁷

The experience of setting up an ‘experiment in living’ has opened up a view on how Google Search works and my exploration of using Tor reimagined what search could look like. Without having to become a ‘personalised subject’, Tor offered me ‘relevant’ search results as an ‘anonymous user’. With the Tor Browser I am not ‘commanded’ by ‘prescriptive’ technologies ⁶⁸ as I am with Google Search, instead I chose which collective I wished to be part of, in this case Tor. The effects of the Black Bloc (Agencies of Anonymity) enabled me to intervene and this type of searching facilitated degrees of anonymity, embodied by various levels of user agency. Aside from its other merits, Tor is also one, albeit not the only, strategy to challenge the protocols facilitating Google’s ‘surveillance capitalism’. 

References

Laura Poitras Citizen Four, 2014, https://citizenfourfilm.com/

Mark Weiser, “The Computer for the 21st Century”. Scientific American Special Issue on Communications, Computers, and Networks, September, New York, Scientific American, 1991.

Renée Ridgway, Re:search. Dissertation, 2021.

Wendy Hui Kyong Chun, Habitual New Media: Updating to Remain the Same, Cambridge, MIT Press, 2016, 77.

John Battelle, The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture, New York Penguin, 2006, 6.

Renée Ridgway, “Personalisation as currency”, APRJA (A Peer-Reviewed Journal About), 18 APRJA Volume 4, Issue 1, 2015, http://www.aprja.net/?p=2531.

Tarleton Gillespie, “Platforms Intervene”, Social Media + Society, Thousand Oaks: Sage Publishers, 2015.

Nick Srnicek, Platform Capitalism, Cambridge, Polity Press, 2016.

Greg Elmer, Profiling Machines: Mapping the Personal Information Economy, Cambridge, MIT Press, 2003.

10 Geert Lovink, Nathaniel Tkacz, “MoneyLab: Sprouting New Digital-Economic Forms”, In Moneylab: An Intervention in Digital Economy, Amsterdam, Institute of Networked Culture, 2015, 15

11 Dallas Smythe, “On the audience commodity and its work”, in M. G. Durham, D. M. Kellner (eds) Media and cultural studies, Malden, MA, Blackwell 1981, 230–56

12 Shoshana Zuboff, “Big Other: surveillance capitalism and the prospects of an information civilization”, Journal of Information Technology, 30, Thousand Oaks: Sage Publishers, 2015, 81

13 Armin Beverungen, Timon Beyes, Lisa Conrad, Digital Media, Thousand Oaks, Sage Publishers, 2019.

14 Renée Ridgway, Re:search. Dissertation, 2021.

15 David L. Chaum, “Untraceable Electronic Mail, Return addresses, and Digital Pseudonyms”, Communications of the ACM, Volume 24, Number 2, February 1981.

16 Gary Marx, “What’s in a Name? Some Reflections on the Sociology of Anonymity”,The Information Society, 15:2, Milton Park, Taylor & Francis, 1999, 99- 112.

17 Alexander R. Galloway, “Black Box, Black Bloc.” In: Benjamin Noys (ed.): Communization and its Discontents. Contestation, Critique, and Contemporary Struggles, Brooklyn, Minor Compositions/Autonomedia, 2011, 237–249.

18 Noortje Marres, “The Experiment in Living”, In InventiveMethods: The Happening of the Social. Celia Lury, Nina Wakeford. Eds, Abingdon, Routledge, 2012.

19 Alexander R. Galloway, Protocol: How Control Exists afterDecentralization, Cambridge, MIT Press, 2004.

20 Mikkel Flyverbom, Paul Leonardi, Michael Stohl, Cynthia Stohl, “The Management of Visibilities in the Digital Age: Introduction”, International Journal of Communication, Los Angeles, USC Annenberg Press, Vol. 10, 2016, 98–109.

21 Timon Beyes, Claus Pias, “The Media Arcane.” Grey Room 75, Cambridge, MIT Press Journals, 2019, 84-107.

22 Finn Brunton, Helen Nissenbaum, Obfuscation, A User’s Guide for Privacy and Protest, Cambridge, MIT Press, 2015.

23 Alexander R. Galloway “Black Box, Black Bloc.”, 3

24 GDPR, General Data Protection Rights, https://gdpr-info.eu/art-4-gdpr/

25 Michel Foucault, Ethics: Subjectivity and Truth, The Essential Works of Michel Foucault 1954-1984.

26 Tarleton Gillespie, “2. Algorithm” In Digital Keywords. A Vocabulary of Information Society and Culture. Edited by: Benjamin Peters, Volume 8 in the series Princeton Studies in Culture and Technology, Princeton, Princeton University Press.

27 Michel Foucault, Technologies of the Self, Edited by Luther H. Martin, Huck Gutman, Amherst, University of Massachusetts Press, 1988, 16.

28 Philip Mirowski, Never Let a Serious Crisis Go to Waste, New York, Verso, 2013.

29 Wendy Hui Kyong Chun, Habitual New Media: Updating to Remain the Same.

30 Louise Althusser, “Ideology and Ideological State Apparatuses (Notes towards an Investigation),” In Lenin and Philosophy and Other Essays, translated by B. Brewster, London, Monthly Review Press, 1971, 174.

31 Sergei Brin, Larry Page. “The Anatomy of a Large-Scale Hypertextual Web Search Engine”, Presented at Seventh International World-Wide Web Conference (WWW 1998), Brisbane, Australia, April p.14-18.

32 Lucas Introna, “The algorithmic choreography of the impressionable subject”, In Seyfert, Robert and Roberge, Jonathan (Eds.), Algorithmic Cultures: Essays on Meaning, Performance and New Technologies, Abingdon-on-Thames, Routledge, 2016, 37

33 Wendy Hui Kyong Chun, Habitual New Media: Updating to Remain the Same, 121

34 Michel Foucault, Ethics: Subjectivity and Truth, The Essential Works of Michel Foucault 1954-1984.

35 Astrid Mager, “Algorithmic Ideology.” Information, Communication & Society, 15:5, Milton Park, Routledge, 2012p. 769-787. 

36 Wendy Hui Kyong Chun, Habitual New Media: Updating to Remain the Same, 36, 64

37 Olga Goriunova, “The Digital Subject: People as Data as Persons” Theory, Culture and Society. Vol. 36, No. 6, 01.11.2019, 125-145. 
Thousand Oaks, Sage Publishers, 2019, 126

38 Gilles Deleuze, “Postscript on the Societies of Control", October. Vol. 59., Winter, 1992, 3-7, Cambridge, MIT Press, 1992, 5.

39 Nanna Thylstrup, “Archival Shadows in the Digital Age”, NordiskTidsskrift for Informations videnskab og Kulturformidling, årg. 3, nr. 2/3, 2014.

40 Antoinette Rouvroy, “The end(s) of critique: Data behaviourism versus due process”, In Hildebrandt, Mireille and De Vries, Katja (eds.) Privacy, due process and the computational turn: the philosophy of law meets the philosophy of technology, New York, Routledge Taylor, Francis Group, 2013.

41 Louise Amoore, “Data Derivatives: On the Emergence of a Security Risk Calculus for Our Times”, Theory, Culture & Society 28(6), 24–43, Thousand Oaks, Sage Publishers, 2011, 126

42 Felix Stalder, “Between Democracy and Spectacle: The Front-End and the Back-End of the Social Web”, In M. Mandiberg (ed.), The Social Media Reader, New York, New York University Press, 2012, 242-256.

43 Slavoj Zizek, The Ticklish Subject: The Absent Centre of Political Ontology, Brooklyn: Verso Books, 1999.

44 Friedrich A. Kittler, Gramaphone, Film, Typewriter. Stanford. Stanford University Press, 1999, 127.

45 Martin Parker, “Manufacturing Bodies: Flesh, Organization, Cyborgs”. In Hassard, J., Holliday, R., Willmott, H. (Eds) Body and Organization, London, SAGE Publications 2000,73, 71-86.

46 Martin Parker, Robert Cooper, “Cyborganization: Cinema as Nervous System”, In For Robert Cooper: Collected Work, Edited by Gibson Burrell and Martin Parker, p.236-252, New York: Routledge, 2016, 243.

47John Suler, “The Online Disinhibition Effect”. Contemporary Media Forum International Journal of Applied Psychoanalytic Studies, Vol. 2, No. 2., Jacksonville, N.C., Whurr Publishers Ltd., 2005.

48 Danah Boyd, “The Politics of 'Real Names: Power, Context, and Control in Networked Publics”, Communications of the ACM 55(8): 29-31, 2012, 30.

49 Lisa Nakamura, “Afterword: Blaming, Shaming and the Feminization of Social”, Shoshana Magnet and Rachel Dubrofsky (eds). Media Feminist Surveillance Studies, Chapel Hill, Duke University Press, 2014.

50 David L. Chaum, “Untraceable Electronic Mail, Return addresses, and Digital Pseudonyms”, 1

51 Mark Weiser, “The Computer for the 21st Century”, Scientific American Special Issue on Communications, Computers, and Networks, September, New York, Scientific American, 1991.

52 Sherry Turkle, The Second Self, Cambridge Mass, MIT Press, 2005, 288

53 Olga Goriunova, “The Digital Subject: People as Data as Persons” Theory, Culture and Society. Vol. 36, No. 6, 01.11.2019, Thousand Oaks: Sage Publishers, 2019, 125-145, 127.

54 Robert W. Gehl, “Power/freedom on the dark web: A digital ethnography of the Dark Web Social Network”, Republished 2016, New Media & Society, Volume 18(7), 1219-1235, Thousand Oaks: Sage Publishers, 2014, 1226.

55 Lantaya Sweeney, “K-anonymity: a model for protecting privacy”. International Journal on Uncertainty, Fuzziness and Knowledge-based Systems, 10. 2002, 557-570, Singapore, World Scientific Publishing Co Pte Ltd, 2002.

56  Finn Brunton, and Helen Nissenbaum, Obfuscation. A User’s Guide for Privacy and Protest, Cambridge, MIT Press, 2015, 55

57 Helen Nissenbaum, Daniel C. Howe, “TrackMeNot: Resisting Surveillance in Web Search”. In Kerr,I., Lucock, C. and Steeves,C. (Eds), ‘Lessons from the Identity Trail: Anonymity, Privacy, and Identity in a Networked Society, Oxford, Oxford University Press, 2009, 417.

58 Helen Nissenbaum, Daniel C. Howe, Mushon Zer-Aviv., Ad Nauseum, 2014, Available at: https://adnauseam.io

59 Daniel C. Howe, Helen Nissenbaum, “Engineering Privacy and Protest: A Case Study of AdNauseum”. Proceedings of the 3rd International Workshop on Privacy Engineering, Volume 1873, 57-64, 2017, 57.

60 Helen Nissenbaum, “The Meaning of Anonymity in an Information Age”, In Spinello, Richard A. and Tavani, Herman T.(eds.), The Information Society. 15:141-144n Milton Park: Taylor & Francis, 1999, 141

61 Robert W. Gehl, “Power/freedom on the dark web: A digitalethnography of the Dark Web Social Network”.

62 TAILS, The Amnesiac Incognito Live System, https://tails.boum.org/

63 Random Darknet Shopper 2016,  MediabitnikGruppe: https://wwwwwwwwwwwwwwwwwwwww.bitnik.org/r/

64 Tor Project: https://check.torproject.org/exit-addresses

65 Eric Hörl, General Ecology: The New Ecological Paradigm, London, New York, Bloomsbury Academic, 2017, 35.

66  Bernhard Stiegler, To Love, To Love Me, To Love Us: From September 11 to April 21, Stanford: Stanford University Press, 2009, 77.

67 Maurizio Lazzaroto, “Conversation with Maurizio Lazza-roto”. Public Editing Session #3, June 23, 2010, from ‘Exhausting Immaterial Labour in Performance’; joint issue of Le Journal des Laboratoires and TkH Journal for Performing Arts Theory (no. 17), October 2010, 14.

68 Ursula Franklin, “The Real World of Technology”. Massey Lectures, 1989. 

Processes, Fabrication and Design with Kombucha Bacterial Cellulose: Mapping Practices

Vivien Roussel1,2, Lucile Haute2,3, Pascal Dhulster4, Marc Teyssier1

1Institute for Future Technologies, 2Université de Nîmes (Projekt), 3École des Arts Décoratifs  PSL (EnsadLab), 4Université de Nîmes (Chrome). Nîmes, Paris, FR
vivien.roussel@devinci.fr ; lucile.haute@ensad.fr ; Marc.Teyssier@devinci.fr ; pascal.dhulster@unimes.fr

Abstract

Kombucha is a fermented drink that has been drunk in Asia for centuries and has recently been popularized in Western cultures. The production of the beverage generates a co-product, a biofilm of Bacterial Cellulose (BC). This biofilm is today at the center of a typology of bio-design practices, ranging from initiation workshops to fermentation to practical and plastic research around shaping objects with this living matter. This paper proposes an overview and initial classification of the current design practices that use Kombucha Bacterial Cellulose as the raw material for artifacts. To this extent, we structured a corpus of selected projects in a Principal Component Analysis (PCA) map. We classified 38 projects in the field of design made between 1985 to 2022. Each project is positioned in an orthonormal frame of reference according to its degree of definition of use (from the most unrestricted use to the most defined use) and its scale of manufacture (from single experimentation to industrial production). This map tries to bridge the gap between the communities.

Keywords

Kombucha, Bio-design, Cartography, Mapping, Bacterial Cellulose.

DOI

10.69564/ISEA2023-81-full-Roussel-et-al-Mapping-Practices

Introduction

Current sustainability contexts highlight the collaborative relationships between humans and living organisms. Small and growing organisms feed us and produce the raw materials needed to make everyday objects. Kombucha can do both at the same time. The nature of their two-fold production can be seen either as the produce or the co-produce. While working with raw materials such as wood (cellulose) to create furniture or domestic objects, designers and makers have recently used the (macro) co-produce of living symbiotic (micro) organisms. Among these, the co-produce of Kombucha has been used to create objects and artifacts.

Kombucha beverages contain a bacteria and yeast symbiotic culture that grows in a sweet tea solution. This association between organisms results in a natural fermentation process that transforms the sweet tea solution into a solution with nutraceutical properties.

One particularity of Kombucha is that acetobacter makes a biofilm or strain of bacterial cellulose on top of the beverage. Many communities share strains and form a human-sharing network of kombucha mothers for personal use.

At a macro scale, this Bacterial Cellulose (BC) is a raw material that designers can treat and process. In recent years, Kombucha products and co-product have been of interest to many research and design practitioners. They offer a simple and robust living medium for experimenting with living things, even in a DIY or domestic setting and urban spaces. More generally, we use the term biofabricated materials to describe the materials using living organisms in their production (microbes rather than plants or animals).

The science of "Kombucha" as Bacterial Cellulose (BC) is usually segmented into more or less scattered fields: biology ¹ , medicine ², food ³ , textiles ⁴ , and material science ⁵. This ensemble draws a vast potential for use and practice, which are only sometimes put in parallel, and remain most often exclusive. However, it needs to be determined how the materiality of this research influences design choices and how the designer's communities can mobilize them for developing and employing the best strategies in producing artifacts.

To tackle this issue, we propose an initial mapping that organizes these heterogeneous BC-based designs and research domains that elaborate an emerging form of possible new materiality. To build this map, we rely on 38 prior works that their authors classify as "art" or "design" pieces. We focus on works that use either Bacterial Cellulose as raw material or demonstrate the growth process. This collection reflects the applicative or demonstrative potential of working with living materials.

Our mapping questions and challenges the possible symbiotic relationships between non-humans and humans. Addressing these assemblages through Design offers a practical and mobilizable prism of applied bacterial cellulose's state of the art. 

A new take on Symbiotic Design with the Living

Designing and fabricating new materials or objects using living organisms' natural processes of growth or reproduction is a secular process rooted in the human desire for control over nature. Traditionally, this process is entrenched in arts and craft fabrication. Artisans can understand raw materials and grow, select, or shape them according to their knowledge of the processes, creating a collaborative and synergetic process. "Making is to growing as being to becoming" ⁶ ; the technical transformation of living materials implies nurturing and cultivation: "if things grow, they are also grown" ⁷. This creation is beneficial from a design perspective, as it creates the serendipity necessary to emerge new shapes and forms. Other practitioners are engaged in this discipline. This paper will refer to the users involved in active collaboration with the living to produce goods as "producers." We identified this process as performed by several people, including Artisans, Designers, or Makers.

This collaborative process has slowly been forgotten over the past decades or taken over by mass-manufactured petroleum-based products. The rise of sustainability challenges has raised questions about designing in collaboration with the living and using the co-product of already used processes, and are at challenges at several scales. On the one hand, at a large or macro scale, it is often hard for the producer to control the form of the object ⁸. It involves operation at different stages of the organism's life cycle that allows shapes or final designs to emerge. On the other hand, the fabrication processes can be controlled at a small or microscale but are challenging for producers.

The design community used growing materials such as Mycelium or Bacterial Cellulose as emerging material practice ⁹, leading to the Growing Design movement exploring the emergence of artifacts. Produces use the materiality of the medium to support the creation of artifacts. It is mainly the case for Mycelium, which allows the creation of composite bio-materials capable of mechanically structuring the object ¹⁰.

Bacterial Cellulose provides other opportunities. These biological systems can assemble biodegradable and conformable complex structures that can self-repair, sense, and respond to external stimuli. Bacterial Cellulose is the product of sugar synthesis by acetic acid bacteria –— the interaction with yeast and other types of bacteria in SCOBY will not be discussed here. The Bacterial Cellulose grows on the surface of a sweet tea liquid aerobically, encouraged by the exchange interface between air and the fluid. The overall appearance of fresh (i.e., wet) bacterial Cellulose is slimy. Once dried, it is similar to paper or leather. The latter state is most often used as raw material for production in textile design. In addition to being able to be processed artisanally (using leather work, sewing), this state is more prone to speculative and experiential uses — they seek to provoke a question or emotional design through touch, sight, or smell (sour).

The works, pieces, and projects made within the community use different fabrication processes and design methods and target different audiences. While they offer a wide range of variability and diversity, it can be challenging for designers or makers to understand how the field is structured and how he or they can intervene. Thus, our proposed mapping offers a global overview and a practical perspective to the still-growing community of bio designers.

From Design Projects to Classification

Corpus selection

Based on our observation of the kombucha producers —a community in which we take part—we estimate that artists and designers have produced around a couple of thousand design projects with kombucha. To create this map, we decided to select representative artifacts that showcase the main capabilities of the material. Our choice rationale was first to pick projects identified by their creators as "art," "design," "piece," "objects," or "garment/cloth/textile."

The prism of practice and production was the second selection criterion. This method excluded most of the theoretical biology and engineering research projects, drink, and the more known derivatives of the BC. Furthermore, the projects without direct objective implications in instruments or artifacts were not kept. Furthermore, in the cases of projects with similar process, outcome, and look, we opt to merge various projects with one representative visible one.

In this collection, we deliberately exclude aestheticism classification as the reception of works that produce objects with living matter oscillates between fascination and disgust— even fear (contamination, slimy, etc.).

Among this list, the projects ranged from social design to product design. Both are between the "doing" and the more or less driven user prescription. The list of selected projects in our map can be seen in the Bibliography section.

Methodology and Visual Representation

The main objective of the visual representation is to summarize the information in this heterogeneous corpus. The second objective is to provide a broad, flexible grid of analysis that highlights, at the same time, links and differences and provide a visualization tool to understand the variety of use of BC that open up design potentials.

There are several ways to classify and discriminate the selected works. The choice of factors is motivated by the desire to characterize the design practices through the spectrum of the objects produced and their relative fabrication processes.

We used a two-factor classification, used in the design project and methodologies. The first factor was to consider the use and the user, while the second was to evaluate the production scale. It allows us to visualize the main benefit of BC on a 2-axis graph. 

The horizontal axis represents the production scale. The spectrum ranges from  manufacturing —closer to art, experimental, or artisanal practice—to industrial production. The axis highlights on one side the uniqueness of the lived experience and the questions raised by observing, experiencing, or interacting with the artifact, while highlighting the standardization and scale while informing the piece's replicability level on the other side.

The vertical axis represents the orientation of selected production in relation to practice and use, that is, the spectrum from passive contemplation to complex use defined in a restricted field. Its scope ranges from experimental use to defined (prescribed?) use. The top part of the axis values the material quality corresponding to the need, while the bottom describes more artistic and experimental practices where the goal isn’t oriented on the quality but on the experience for the spectator or public.

Each selected project was given by the authors two 0-10 grades on the factors: the fabrication quality and the design prescription. This grade was used to position the project on the 2D axis. It should be noted that we display superimposed works in adjacency for visual representation. After placing the papers on the grid, the authors extracted clusters of domains.

Figure 1. Processes, Fabrication and Design with Kombucha Bacterial Cellulose: Mapping Practices.

Mapping Bacterial Cellulose Design Practices

The Mapping is represented in Figure 1. The 2D axis mapping is divided into four square spaces, with points spatially representing the projects. Each point is visually represented by a bubble containing an image and surrounded by its title and main authors. The points (bubble) are joined in clusters by distinct colored frames referring to domains: medical, industrial, machine, product, craft, speculative, social, contemplative, experiential.

Spatial Representation

The general location of the projects draws a diagonal from the lower left to the upper right part. The typologies of the projects selected can explain this. There is a constant progression between the level of prescription of the use of the BC towards the user simultaneously as the complexity of technical manufacture increases. Thus, the diagonal represents the increase in the complexity of the functions of the artifacts.

The upper left and lower right parts contain fewer project bubbles than the other spaces. The upper left and lower right half are mainly empty in our representation. However, some projects could be representative. An example from the top left could be the use of BC kombucha for food production, thus reversing the notion of product/co-product in the industry (cellulose paste as an additive, for instance, for sauces or ice cream). However, these uses are separate from current design practices, as we need representative examples of the industrial production of a food product as artistic or design production. Thus, this upper left corner is empty because there is hardly any object that is both mainly prescribed and, at the same time, exceptionally technically unspecified.

Similarly, the lower right corner is also empty because this space represents artifacts that would be industrially very advanced (machine and other complex implementations needed for production) yet have very little prescribed use or even a demonstrative or speculative use. On the one hand, this map area should be occupied by standardized products resulting from a complex industrial process, waiting for a noble valorization use —other than composting or methanation, for example. Products of this type may not be design projects, but industrial co-products and have been excluded from our corpus. On the other hand, while the realization of a machine or a process is often driven by the creation of a specific and precise object, we can cite the works of Bianca Hlywa ¹¹ or Stephan Schwabe ¹² that deepen the technical fabrication process for artistic purposes with the BC.

Clusters: Domains of Interest for Design

The different clusters refer to different domains. The Medical cluster gathers projects developed for medical use, from cutting-edge research to marketed bandages ¹³. The Industrial cluster gathers projects developed by industrial actors from an industrial perspective. The Machine cluster gathers projects oriented to the machine that allows the growing process, like bioreactor—more than the result of the growing process. Conversely, the Product cluster cares for these growing results and gathers objects made with BC. The Craft cluster is oriented to artisanal knowledge. The Speculative cluster gathers projects related to speculative art or speculative design, meaning their operating function is intended to develop critical thinking. The Social cluster gathers projects that are collective experiences and social forms related to Social Design and Relational Æsthetics.¹⁴ The Contemplative cluster gathers mostly art projects that aim to provide an æsthetic experience (primarily oriented to visual sense). The Experiential cluster gathers projects that offer multi-sense experiential expertise.

From the position of the projects, we defined and distinct nine overlapping clusters that characterize the type of practices with BC. Each represents a different domain of design that is mainly explored with BC. While not exhaustive, this list allows the viewer or user of the map to position other works not present in the current map. 

Figure 2. Division by clusters. Here the projects in the Craft category are highlighted.

Some projects are linked to several fields. As an example, in the "Craft" category (Figure 2), Malai.eco¹⁵ is a company that grows BC from coconut to make leather products—as Kombucha Couture,¹⁶ but with a larger scale and with a real industrial methodology. They grow BC in bio lab conditions and produce BC by the meter to use this finished piece as material for sewing bags, shoes, etc. We defined Kombucha Couture as an artisanal production in comparison to Biocoute. Artists such as Suzanne Lee,¹⁷ who initiated the engagement in design for BC and Biocouture, maintain the craft relationship to production.

Overall, other overlapping communities have been revealed in our mapping. For instance, the Medical cluster is porous with other practices because practitioners develop incredibly defined projects with very robust and rigorous uses and technical production methods. On the opposite side, the Speculative, Social, Contemplative, and Experiential clusters share most of their points because it is challenging to define the boundaries of their respective projects. Their positions are less linked to the type of production than to the engagement sought by their creator with the spectator, the user. 

Limitations of the Mapping

This article provides an initial classification of design projects utilizing Kombucha as a base material. With this map, we contribute to a better understanding of the possibilities and enlarge the scope of existing projects.

This cartography is a tentative way to structure the artwork from our observation and reflect our opinion that is open to discussion and debate within the community. We acknowledge some limitations of our mapping. The classification, according to the position, simplifies the positioning of the selected project and offers a look and a perspective on a very heterogeneous taxonomy. The selection criteria are related to projects which can show a new potential use of the fabrication process for BC. Or, we sometimes select one work to best figure for a tiny cluster of practices, or we pick the most popular and referent work as seen by the community. Although some projects could have been included, the artifacts resulting from biology papers did not have direct "objective" implications in the instrument of production and are not seen as design artifacts by their creators and not mobilizing enough by the designers. In an attempt to overcome this limitation, we created an updatable interactive map¹⁸ that provides flexible classification choices. The interactive map also allows us to easily integrate proposals from the community.

Similarly, some objects/artifacts/instruments were difficult to categorize strictly into a field. Often, the possibility of classifying the objects in two dimensions facilitates decision-making. Most often, the gap and similarities between the project types define the position of the object in the frame. Thus, it is a look at the whole production and their characteristics in relation to each other that positions the projects in a more comprehensive and coherent view.

Conclusion: Statement about the Importance of Kombucha based Design Practices

New Forms for Design: Slow Growth Monitoring

The design practices of Kombucha BC can be applied to the challenges of today's world. With our dependence on energy, it is important to come up with solutions that are local and meaningful. Choosing to work with living organisms to maximize production is a skill that humanity has mastered. With the help of (bio)technologies, we are able to have greater control over the production of living things, leading to new ways of creating materials through programming or simulation. However, the process is slow and challenges our understanding of resources, their availability and arrangement.

Our map shows that monitoring biological activity is crucial in redefining how we use living things. We can use scientific measurement methodologies and bioengineering tools such as thermal imaging, graphs and optimization techniques to control, analyze and adapt our productions at a slow rate of growth. Monitoring is an essential part of all the projects presented, providing expertise and control to producers, enabling them to anticipate production or even design without material. This reinforces the sociotechnical regimes through design, linking human beings and non-humans through machine mediums. 

Emerging Role of Process engineering

The impact of digital manufacturing tools and DIY chemical processes on design has now been followed by the impact of biology and synthetic biology tools and processes.¹⁹ To design and manufacture with living organisms, Process Engineering is crucial. By understanding the biology of living organisms, producers can interact with and mobilize them in production.²⁰ This understanding allows producers to access dedicated tools and engineering technologies of production that redefine the relationship with emerging things. These technologies include programming living matter (modification of the environment or genetic reprogramming) and programming material behaviors (4D printing, programmable matter, structures with auxetic behaviors, soft-robotics, digital fluid mechanics, etc.).

Our analysis of the corpus shows that these technologies arise from a design valorization of bioengineering methods combined with DIY fabrication tools.²¹ The combination of living organisms within production machines challenges the link between parasitism and symbiosis, resulting in machine-organic producers of "bio-artifacts."

Evolving Role of the Designer

This practice can profoundly transform our relationship with living materials that can now be easily shaped and transformed in a DIY. Beyond the design with non-human living forms, the fusion between materials science and synthetic biology is playing out at a new crossroads of ecology that is neither scientific nor animistic. Working with the living has existed since the dawn of humanity. The rise of these processes can be seen as a renewal or update in manufacturing and design thanks to the knowledge of the biological matter.

Working with biological processes also questions designers' intention to collaborate with living matter to repair their practices. They can seek to improve the link between everyday objects and the modern world we live in, because they could be partly responsible for this unsustainable environment.

While incomplete, we hope this map will foster discussion and generate new ideas in the design community. More generally, we argue that Design with Kombucha Bacterial Cellulose is a way to connect and re-contextualize the modern (bio)technological convergence through æsthetics and design. 

References

1 Nermin Hande Avcioglu, “Bacterial cellulose: recent progress in production and industrial applications”, “World J Microbiol Biotechnol”, 2022 Apr 10; 38(5):86, doi: 10.1007/s11274-022-03271-y.

2 Guilherme Fadel Picheth, Cleverton Luiz Pirich, Maria Rita Sierakowski, Marco Aurélio Woehl, Caroline Novak Sakakibara, Clayton Fernandes de Souza, Andressa Amado Martin, Renata da Silva, Rilton Alves de Freitas, “Bacterial cellulose in biomedical applications: A review”, Int J Biol Macromol, 2017 Nov, p.104 (Pt A):97-106. doi: 10.1016/j.ijbiomac.2017.05.171.

3 HMC Azeredo, H Barud, CS Farinas, VM Vasconcellos and AM Claro, 2019, “Bacterial Cellulose as a Raw Material for Food and Food Packaging Applications”, Front, Sustain, Food Syst, 3:7, doi: 10.3389/fsufs.2019.00007

4 A.P. Provin,, dos Reis, V.O., Hilesheim, S.E. et al., “Use of bacterial cellulose in the textile industry and the wettability challenge—a review”, Cellulose 28, 8255–8274, 2021, https://doi.org/10.1007/s10570-021-04059-3.

5 V.V. Revin, E.V. Liyaskina, M.V. Parchaykina, T.P. Kuzmenko, I.V. Kurgaeva, V.D. Revin, M.W. Ullah, “Bacterial Cellulose-Based Polymer Nanocomposites: A Review”, Polymers 2022, 14, 4670, https://doi.org/10.3390/polym14214670.

6 P 4, Hallam, Elizabeth & Ingold, Tim (eds.), Making and Growing. Anthropological Studies of Organisms and Artefacts, Farnham, Ashgate, Anthropological Studies of Creativity and Perception, 2014, 258, Hb.: £61.10. ISBN: 9781409436423.

7 P 3; idem.

8 Idem..

9 Serena Camere, Elvin Karana, “Fabricating materials from living organisms: an emerging design practice”, Journal of Cleaner Production, 2018, doi: 10.1016/j. jclepro.2018.03.081

10 Idem.

11 Bianca Hlywa, Residual yeast, Bianca Hlywa “Residual Yeast” Gossamer Fog, London, 2022.

12 Stefan Schwabe, The Kernels of chimera, Stefan Schwabe website :The kernels of chimera, accessed December 08, 2012, http://www.stschwabe.com/work/KoC/.

13 Guilherme Fadel Picheth, and al. “Bacterial cellulose in biomedical applications: A review”, Int J Biol Macromol, 2017 Nov, 104 (Pt A):97-106, doi: 10.1016/j.ijbiomac.2017.05.171.

14 Nicolas Bourriaud, Relational Aesthetics, Dijon, Les Presses du réel, 2002.

15 Malai.eco, 2016. Malai website : Malai Material, accessed December 08, https://malai.eco/blogs/news/about-malai-material. 

16 Sacha Laurin, Kombucha Couture, 2015. Kombucha couture website : LUNA, accessed December 08, http://www.kombuchacouture.com.

17 Suzanne Lee, Biocouture, 2003 - 2013. Thisisalive exhibition website : biocouture, accessed December 08, http://thisisalive.com/fr/biocouture/.

18 www.bacterial-cellulose.com

19 Paola Antoneli, “Synthetic Aesthetics: New Frontiers in Contemporary Design” at MOMA, The Museum of Modern Art, accessed December 09, 2022, Youtube channel https://www.youtube.com/watch?v=u1D4ArcBjLI.

20 Marion Koelle, Nicolae Madalina, Aditya Shekhar Nittala, Marc Teyssier, Jürgen Steimle, "Prototyping Soft Devices with Interactive Bioplastics", In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, 1-16, 2022.

21 Cedric Honnet, Hannah Perner-Wilson, Marc Teyssier, Bruno Fruchard, Jürgen Steimle, Ana C. Baptista, Paul Strohmeier, "Polysense: Augmenting textiles with electrical functionality using in-situ polymerization", In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, 1-13. 

Bibliography

Alanna Lynch, Gut Feelings, 2016, Alanna Lynch website, accessed December 08, 2022, http://alannalynch.com/portfolio/gut-feelings-2017/.

Ali Schachtschneider, Vivorium, 2015, The New School Parsons website, accessed December 08, 2022, https://parsons.edu/bfafashion/ali-schachtschneiders-vivorium-aw arded-rsa-usa-featured-dezeen/.

Armine Ghalachyan, “Made from Scratch: A Sustainable Handbag Made of Bacterial Cellulose Grown in Fermenting Tea”, Iowa State University, Ames, IA, USA© 2017, International Textile and Apparel Association, Inc. ALL RIGHTS RESERVED ITAA Proceedings, #74 – www.itaaonline.org.

Bianca Hlywa, Residual yeast, 2022, Bianca Hlywa “Residual Yeast” Gossamer Fog / London by Deniz Kirkali, accessed December 08, 2022, https://flash---art.com/2022/10/bianca-hlywa/.

Dean Brough, Alice Payne, Peter Musk, 2015, Harvest: A biotextile future, Queensland’s Asia Pacific Design Library, 1-5 November 2015, as part of The International Association of Societies of Design Research’s (IASDR) biannual design conference, Queensland ePrint website : Harvest: A biotextile future, accessed December 08, https://eprints.qut.edu.au/225553/.

Dunne and Raby, United Micro Kingdom, 2013, Dunne and Raby website project of United Micro Kingdoms: a design fiction, accessed December 08, 2022, http://unitedmicrokingdoms.org.

Elena Amato, Ponto design, 2019, Dezeen website: Elena Amato creates sustainable cosmetics packaging from bacteria, accessed December 08, 2022, https://www.dezeen.com/2019/02/28/elena-amato-bacteria-packaging-design/.

Emma Sicher, From Peel to Peel, 2018, Dezen website: Emma Sicher makes eco-friendly food packaging from fermented bacteria and yeast, accessed December 08, 2022, https://www.dezeen.com/2018/11/13/sustainable-food-packaging-emma-sicher-peel/.

HBBE (Hub for Bio Building Environnement), Living Manufactures, 2021. Research Team: Thora H Arnardottir (RA), Joshua Loh (RA), Katie Gilmour (RA), Sunbin Lee (PhD), Meng Zhang (PI), Martyn Dade-Robertson (PI). The Living Manufacture Project is funded by the EPSRC Manufacturing the Future (Grant Number: EP/V050710/1). HBBE website : Living Manufacture: Principles for a microbial 3D printer, accessed December 08, 2022, http://bbe.ac.uk/living-manufacture/.

Jannis Huelsen, Xylinium Stool, 2011, Huelsen website : Xylinum Stool 2011, March, accessed December 08, 2022, http://www.jannishuelsen.com/?/work/xyliumstool/.

Jen Keane, This is grown, 2019 - becoming Modern Synthesis company, 2020, Designboom website: jen keane employs microbial weaving in 'this is grown' shoe?, accessed December 08, 2022, https://www.designboom.com/technology/jen-keane-this-is-grow n-microbial-grown-shoe-11-11-2020/.

Le Qara, 2019, H&M Foundation webite : LE QARA, accessed December 08, 2022, https://hmfoundation.com/gca/winners/le-qara/.
Lionne Van Deursen, Unfold, 2018.Van Deursen website : UNFOLD, accessed December 08, https://www.lionnevandeursen.com/unfold.

Lionne Van Deursen, Unfold, 2018, Van Deursen website : LUNA, accessed December 08, https://www.lionnevandeursen.com/luna.

Malai.eco, 2016, Malai website: Malai Material, accessed December 08, https://malai.eco/blogs/news/about-malai-material.

Mary Maggic, Scoby Spin cycle, 2021, In-silo website : Scoby Spin cycle, accessed December 08, https://in-silo.com/scoby.

Naja Ryde Ankarfeldt, Tale of non-human, 2015, Art Science Stichting website : Naja Ryde Ankarfeldt, accessed December 08, http://stichting.interfaculty.nl/naja-ryde-ankarfeldt/.

Nitzan Cohen, Emma Sicher, Ignacio Merino Sanchez-Fayos and Secil Ugur Yavuz, “An Open-Source Bioreactor Enhancing Microbial Cellulose Production and Novel Sustainable substances”, Sustainable Design and Manufacturing: Proceedings of the 8th International Conference on Sustainable Design and Manufacturing (KES-SDM 2021), Vol.262, p.77-86; Smart Innovation, Systems and Technologies, 262; KES-SDM 2021, 8th International Conference on Sustainable Design and Manufacturing (Split, 16/09/2021 - 17/09/2021), 2022, DOI: https://doi.org/10.1007/978-981-16-6128-0_8.

Polybion, 2015, Polybion website : About us, accessed December 08, https://www.polybion.bio/about-us.

Iowa State University, Iowa State Univrsity website : Clothing made from tea byproduct could improve health of fashion industry, accessed December 08, https://www.news.iastate.edu/news/2016/04/26/sustainableclothing

References Regent Medical Ltd, “EpiProtect” (2001) and “Basyc” (2009) extract from Wilton R. Lustri, Hélida Gomes de Oliveira Barud, Hernane da Silva Barud, Maristela F. S. Peres, Junkal Gutierrez, Agnieszka Tercjak, Osmir Batista de Oliveira Junior and Sidney José Lima Ribeiro, “Microbial Cellulose — Biosynthesis Mechanisms and Medical Applications”. Additional information is available at the end of chapter 6; http://dx.doi.org/10.5772/61797.

Sacha Laurin, Kombucha Couture, 2015, Kombucha couture website : LUNA, accessed December 08, http://www.kombuchacouture.com.

ScobyTec, 2014, ScobyTec website: Uber uns, accessed December 08, http://www.scobytec.eu/über-uns.

Stefan Schwabe, Growing a Rool, 2012, Stefan Schwabe website: Growing a Roll, accessed December 08, http://www.stschwabe.com/work/GrowingARoll/.

Stefan Schwabe, The Kernels of chimera, 2012, Stefan Schwabe website: The kernels of chimera, accessed December 08, http://www.stschwabe.com/work/KoC/.

Stefan Schwabe with Jannis Huelsen, Xylinum Cones/ Programming, 2014, Stefan Schwabe website: The kernels of chimera, accessed December 08, http://www.stschwabe.com/work/XylinumCones/.

Suzanne Lee, Biocouture, 2003 - 2013, This is alive exhibition website: biocouture, accessed December 08, http://thisisalive.com/fr/biocouture/.

Thinking Hand Studio, Feral Fetish, 2021, Thinghand Studio website: Feral fetish, accessed December 08, https://studiothinkinghand.com/works/feralfetishfilm.html.

Thr34d5, Regrow, 2018, Thr34d5 website: Regrow, accessed December 08, https://thr34d5.org/2019/08/20/regrow/.

Thr34d5, Kombucha tsugi: the bag edition, 2021, Thr34d5 website: Kombucha tsugi: the bag edition, accessed December 08, https://thr34d5.org/2021/03/14/kombucha-tsugi-the-bag-edition/.

Hunter Whitefeater, Bucci, 2017, Whitefeather hunter website: “bucci collaborative project at fashion pop, Montreal”, accessed December 08, https://whitefeatherhunter.net/2017/09/13/upcoming-sept-whitefeather-presents-collaborative-project-at-fashion-pop/.

Xxlab, Soya C(o)u(l)ture, 2015, Ars Electronica website: SOYA C(O)U(L)TURE – Useful Things arise out of Waste, accessed December 08, https://ars.electronica.art/aeblog/en/2015/09/30/soya-coulture/. Yoko shimizu, Layers of life, 2014. Shimizu website : layers of life, accessed December 08, https://yokoshimizu.com/portfolio_page/layers-of-life/. 

The Hitchcock Experience - a Spatial Montage project

Rémi Sagot-Duvauroux1, Nils Quaetaert2, François Garnier1, Rémi Ronfard2

1École des Arts Décoratifs  PSL (EnsadLab)
2Univ. Grenoble Alpes, Inria, CNRS
1Paris, France 2Grenoble, France
remi.sagot@gmail.com, nils.quaetaert@polytechnique.edu, francois.garnier@ensad.fr, remi.ronfard@inria.fr

Abstract

This paper describes an ongoing art project exploring the generalization of classical montage theory to the emerging technology of “room scale VR”. We take the emblematic “crop duster” scene in Alfred Hitchcock's North by Northwest and invite the spectator to experience it “from the inside”. This raises interesting research issues. How to inhabit this cinematic story world? How can the traditional tools of cinematography and montage be used to direct the audience in this new kind of experience? How can the rhythm of the spectator's body be matched to the rhythm of George Tomasini’s (Hitchcock’s editor) fast pace editing? And what form can this montage take and for what aesthetic and dramaturgical effects? To answer these questions, we propose the experiment of a montage that adapts in real time to the displacements of the body and the gaze of the spectator, engaging a dialectic between narrative rhythm and bodily rhythm. We propose new algorithmic tools to transport the audience into the story as originally planned by Hitchcock while at the same time respecting the behaviors of the audience to guide the experience. In doing so, we seek to create a new form of relationship between the author of a narrative experience in virtual reality and the spectator who explores and activates the experience with his own body.

Keywords

Virtual Reality, Montage, Narrative, Aesthetics, Human-Computer Interaction, Computer Graphics.

DOI

10.69564/ISEA2023-82-full-Sagot-Duvauroux-et-al-The-Hitchcock-Experience

Introduction

While the imaginary around virtual reality devices generates fears and fantasies, it seems essential to question the explicit and implicit bodily interactions that we have with digital spaces. By its algorithmic nature, the virtual environment, calculated in real time, constitutes a functional space that can be manipulated by the immersant (1) but can also control and constrain his/her movements and the sensitive experience, according to a relationship set by an author.

These explorable environments can also carry within them an emotional dimension due to their digital image nature. When dealing with a virtual reality device, our bodily habits of space cohabit with our visual habits of image. We think therefore that it is possible to invoke our culture of the cinematographic images in the spatially lived aesthetic experience.

With the cinema, we agree to sit still for the duration of a film and let ourselves be hypnotized by the gaze of a director. This privileged moment, out of time, offers a frame to rethink our relationship to images (to their time, rhythms and meanings). Depending on the quality of the montage, we leave our own space-time aside to align our perception and our thoughts with the rhythm proposed by the film.

Thanks to virtual reality devices, we are able to inhabit a cinematic representation space. How then to adapt a cinematographic dramaturgy while allowing the emergence of new phenomena related to the involvement of the body in action specific to the immersive nature of virtual reality displays? Should we be limited to explore the scene without discontinuities or should we experience a form of montage? To answer these questions, we designed a virtual reality experience (Figure 1) that immerses us inside the mythical “crop duster” scene in Alfred Hitchcock's North By Northwest. (2)  

Figure 1. Point of view of the immersant in our experience.

In the original film, the montage by George Tomasini (Hitchcock’s editor) is essential to a good understanding of the space and the distances that separate the character (played by Cary Grant) from the other characters and vehicles that interact with him ¹ . Moreover, its evolution through the sequence highly participates in suspense, tension and violence. But if the spectator is physically immersed in this space and able to move through it, the rhythm of the editing needs to be matched to the one of his/her body. This implies a new aesthetic and dramaturgical approach due to the spatial shift of the virtual reality medium. These are the issues that guide this research-creation project entitled: The Hitchcock Experience (3).

We propose the experiment of a montage guided by the movements of the immersant. It explores the questions of distance and identification with the character by transposing certain cinematographic shot scales into the scale of the immersed body regarding the represented space. Moreover, this experiment proposes a form of editing that adapts in real time to the displacements of the body and the gaze of the immersant, engaging a dialectic between the narrative rhythm and the one of the experienced space. We wish to explore a new relationship to the ongoing narrative, profitable to the emancipation of the immersant and harmonizing his/her bodily rhythm to the one of a fragmented inhabitable environment.

The aim of this experiment is not to propose a VR experience that competes with Hitchcock's film. By confronting this masterpiece of the cinematographic montage, we precisely try to understand, through comparative analysis, the profound transformation implied by its adaptation in virtual reality.

First, we will establish related works regarding the practice of montage in virtual reality devices. Then, we will explain our research-creation process and comment on the different steps we went through developing our spatial montage system. We will expose our experimental protocol that we set to collect data from 17 participants. Then, we will show our first relevant results and observations. Next, we will discuss and initiate theoretical explorations on the aesthetic experience of the immersants. Finally, we will conclude on our spatial montage device and the role of its editor.

Related work

The problem of directing techniques suitable for building "room-scale games" and more specifically "room-scale movies" is widely overlooked in recent academic research. Contemporary room-scale VR experiences are based on the strong sensation of presence caused by forcing the viewer to remain in a fixed space and time. As a result, the use of montage is the exception rather than the rule. Hodgkinson ² describes some techniques used by Google Spotlight Stories (4) for moving the immersant in space and in time during a VR experience, which bears some resemblance with traditional montage. For example, Rain or Shine (directed by Felix Massie) uses visual occlusion to smooth out unwanted camera motions. Pearl (directed by Patrick Osborne) introduces temporal ellipses which can guide the narration, while keeping the same spatial location. Age of Sail (directed by John Kahrs) allows the immersants to navigate between a small number of spatial locations, in effect allowing them to create their own spatial montage.

The question of montage has been more deeply explored in the related, but different, context of cinematic VR, where the immersant can only turn his/her head horizontally across 360 degrees and vertically across 180 degrees. If he/she can choose the direction of his/her gaze in the sphere image, the viewer cannot move his/her point of view to explore the spatial dimension of the scene, which is the specificity of room-scale VR. In this limited context, Jessica Brillhart³ has proposed the notion of a “probabilistic experiential montage” where each 360 degree shot offers the possibility of multiple experiences by the immersant, and the montage is created on the assumption of the most probable experience.

Garnier⁴ provides a detailed account of the various geometries involved in viewing film, either on a traditional screen or in a virtual reality headset, and emphasizes the importance of a proxemic interpretation of shot sizes in film. From another perspective, Pope ⁵  also recommends the use of proxemics for staging in VR following some well-established theater techniques.

Rothe et al.⁶  propose an analysis of camera control in cinematic VR and also relate shot sizes and camera distances under the framework of proxemics. Rothe also examines the effect of camera height in distance and size perception in cinematic VR⁷.

In our previous work entitled Reframing VR, we propose a vocabulary of shot values suitable for room-scale VR based on the new notion of a spatial frame. In our framework, spatial montage can be defined as a temporal arrangement of spatial frames with different spatial scales, which play a similar role to the shot sizes used in traditional film theory. Because the immersant is free to move in the virtual world, “shot size” is not measured only in terms of apparent visual size, but also in motion parallax and capacity of action.

Artistic process

Regarding this state of the art, we propose a process of research-creation, based on an artistic device engaging a reflexive loop between practice of the "spatial montage" and theorization. In this part, we explain how we elaborated our artistic experiment. 

Scene reconstruction

Based on a 3D scene provided by the Anima team of Inria (5), we virtually reconstructed the entire set and all the props, characters and vehicles present in the scene using representative keyframes from the movie. We then created a rough "layout" animation of all the character and vehicle movements, synchronized frame by frame with the original movie. We used a floor plan view diagram from Raymond Bellour’s The Analysis of Film ³ as a reference to approximately reconstruct the trajectories of all characters, vehicles and the plane, as shown in Figure 2, and fine-tuned them to match to the existing views from the movie. Some movements could not be seen in the original shots and had to be crafted manually to plausibly match the visible parts. 

Figure 2. Floor plan views of the crop duster scene. Trajectories of all characters, vehicles and plane.

This work enabled us to build a 9 minutes and 45 seconds 3D animated scene corresponding to the action of the original film. We then imported this scene into the Unity game engine to implement the virtual reality set up.

Re-framing

In order to immerse the spectator into this reconstructed 3D scene, we had to find how to translate the frames of the movie to a spatial experience specific to virtual reality.

In room-scale VR, the perceptual boundaries of a continuous virtual reality experience are based on the subject’s freedom of movement in the real environment. The technical equipment and the perimeter of the physical space tracked by the device therefore seems to correspond to a frame. In the virtual environment, it defines a scaled, spatialized and oriented zone in which the subject acquires a limited walking and acting area. We therefore decide to take this zone of potential action as our editing cell. The frame in virtual reality no longer delimits a portion of the image as in cinema, but a zone of possible actions in a space.

This space area constrains the movements that the immersant can perform. By changing the relative sizes of the virtual and the physical worlds, we can define "wide shots" with miniature non-player characters where the room appears to be 100 meters wide; "long shots" with dolly-size characters, where the room appears to be 10 meters wide; "medium shots" with three-quarter-size virtual characters; "point of view shots" with real-size virtual characters; and even "close shots" with larger-than-life characters where virtual movements are limited to a meter or less.

We started by transposing the shot grammar proposed by Hitchcock in our own “immersant’s scale” classification. By testing a variety of scales of the immersant and his/her physical tracking space relative to the scale of the virtual environment, we experimentally found different size ratios that seem to reproduce the distancing effects of the cinematic frame. Moreover, the closer we get to the character’s scale, the more we come to inhabit his space and the more we feel we belong to the diegesis. Finally, we analyzed that each scale brings a difference of amplitude of our bodily action in the virtual environment (a more or less large space is reachable) implying different agencies in the reception and therefore a new rhythm to find for the montage of these scale variations in space and time. 

Figure 3. Two positioned and oriented scaled areas. The immersant’s scale is represented by the yellow avatar.

We then adapted the position and orientation of the scaled area (Figure 3). In order to reduce the motion sickness issues, we imposed that the zones had to always be parallel to the ground. In other words, they could not be tilted in their x or z axis. Moreover, at the beginning of each “spatial shot”, the y axis orientation of the imersant’s gaze is imposed no matter what he/she was looking before the cut.

In a first step, we placed our oriented scaled areas regarding the camera positions of Hitchcoch’s montage. Then, we decided to take some liberties and to re-cut the sequence regarding the medium shift of our device.

Re-cutting

The form of the cut that we propose consists in a change of scale and in the setting of the position and orientation of the immersant's point of view (Figure 4a). Each cut imposes an initial point of view (“initial gaze frame”) to the immersant who is then free to choose his/her gaze movement until the next cut (Figure 4b). So the “final gaze frame” depends on the immersant.

In order to adapt the editing, we started by slowing it down. First of all, we have to take into account the necessarily longer processing time of stereoscopic images with a large field of view. Considering spatial breaks comes up with specific cognitive constraints related to the perception of virtual environments that implies spatial reasoning ¹⁰. A too short time between two cuts can make the image unintelligible.

That’s why we allowed a time for exploration, choice and habitability of the virtual environment. According to the scale and to the action which takes place, we evaluated the necessity of the montage. By moving his/her body and viewpoint, the immersant performs a form of editing that must be taken into account. For example, at the beginning of the sequence, Hitchcock uses shots of about three seconds alternating between a third-person view of the character and his subjective view of the landscape he is looking at. Reproducing this montage while the immersant is free to move around does not work well. Instead, the rotation of his/her gaze must be taken into account to produce the same effect. 

Figure 4a. An immersant experiencing a cut from the plane to a low-angle view of the character.
Figure 4b. Starting from an initial gaze frame imposed by a cut, different gaze movements can be actualized by the immersant until the next cut.

During this step, we studied empirically which cuts proposed by Hitchcock were replaced by the moving point of view of the immersant naturally positioning itself as intended by the film’s editing. We thus observed that the cuts linked to the spatial understanding of the space had to be revised. In addition, imposing a point of view on the empty road systematically incited the subject to turn his/her head, which lost the desired dramatic result. In order to have an effect, the shifts in viewpoint must be justified by a shift in distance or the desire to create a rhythmic change effect.

Then, we were confronted with the fact that the cuts can create a conflict between the immersant's point of view and the one imposed on him/her. These imposed discontinuities can be frustrating and/or seen as a punishment. Moreover, if it takes place during a head rotation, the cut can be disorienting. The two rhythms enter into a struggle. If the immersant lets go and stops moving, he submits to the rhythm of the editing accepting its tempo. For a better experience, we therefore study how to let the immersant create his/her part of the tempo.

Re-cutting with cues

We developed tools with the Unity game engine to enable a montage that adapts to the behavior of the immersant. It reacts to a system of "cues" produced by the human-computer system during the experience.

These cues can be represented by boolean variables computed in real time according to biometric data (gaze and body movements), geometric data (arrangement and orientation of objects in the virtual space) and temporal data (event system).

We use the following preliminary cues:

Figure 5. Interface in Unity. The editor can choose the transition cue (right) and set the transition objects on the timeline (bottom).

We define a type of computing object capable of making the transition between two previously defined spatial frames when the right conditions are verified (Figure 5, right). By setting these objects on a timeline (Figure 5, bottom), we can choose to activate and deactivate them at the desired time. This enables us to define a time range in which a transition can take place. Instead of having a cut defined at a precise moment, we have a delocalized transition that may occur within a certain duration. We call it: "cut porosity". The moment of the cut is then determined in real time according to the received cues. These tools allow an editor to assemble the experience through the timeline interface. The main difference with traditional editing is that the editor works on the duration of the transitions’ possibilities rather than that of the rushes. This system also enables the creation of branches: it is possible to define several transitions that can take place depending on the cues encountered. This feature makes it possible to create alternative setups depending on the immersant’s behavior.

Using this new system of cues and dynamic timeline, we created a new montage that adapts to the rhythm of the immersant. First of all, we mainly used the "Does not move" condition to make sure that the subject has stabilized his/her gaze before the cut (final gaze frame) to avoid disorienting him/her. We also often used the "Is looking at" condition to trigger the cut when the viewer's gaze is stabilized on the character. This allowed us to create on-axis cuts with scale changes to reduce our distance to the action. By using this condition, we were also able to control changes in focalization (from sympathy to empathy) depending on our interest in the character or not. Finally, the "Object in frame" condition was used when watching the plane. When the latter left the field of view, there was nothing to see but the sky, so it was useless to stay in that position.

To illustrate the importance of the cues system combined with the changes in scale, let's take the example of the first plane attack. In the film, this moment is edited with a shot/reverse shot between the crop duster and the character observing it until he understands at the last moment that he is the target. By alternating shots showing the plane getting closer and closer to the camera and shots showing the character in tighter and tighter frames, Hitchcock creates an attraction between the plane and the character until the two meet when Cary Grant jumps to the ground.

Figure 6. The immersant (represented by the yellow avatar) moves from one scale to another according to the cues set by the editor.

If we apply this same editing in VR, we must first take into account that the immersant might perform the field against field by him/herself. It was therefore necessary to have the "Look at plane" condition to switch to the character or the “Look at character” one to switch to the plane. It is indeed useless to show the plane if the immersant is already looking at it. It just creates an unpleasant jumpcut. Moreover, the condition "Does not move" avoids that the cut takes place when the immersant is turning his/her head and thus being punished or disoriented.

But, as the cues are limited in time, there is a moment when even if they are not reached, it is necessary to move to the next point of view. This is where the scale change becomes necessary to create a shift in distance (close-up effect) to the character, even though the immersant is looking at him before and after the cut. By downscaling from the scale of Cary Grant to the one of a mouse (Figure 6), proper editing effects are created when the character falls, even if the "Look at" or “Does not move” conditions are not satisfied.

Finally, it was necessary to work on the time given to the activation ranges of the conditions. The more the range extends in time, the less the editing can impose its rhythm. In this case, the cut may arrive at an ideal moment for the immersant but be misplaced and meaningless in terms of the narrative. We therefore had a tendency to shorten these areas to establish the rhythm of the montage, imposing at certain moments pace accelerations where the immersed body has almost no time to move between several cuts. When the character falls on the floor, it is important to have the cut at this precise moment. This was also particularly the case when the first truck passes, or during the final climax when the tanker truck arrives, hits the character and the plane crashes. It was therefore a matter of finding a balance between the editor's pleasure in imposing a rhythm that he/she liked and the frustration of leaving some freedom of rhythm to the immersant. We then had to evaluate this balance.

Experimental protocol

In order to conduct a qualitative analysis of this montage, we set up an experimental protocol that we tested on 17 participants. 12 of them were novices to virtual reality.

The immersion in virtual reality is made with the HTC Vive Pro Eye system. A Vive tracker is attached to the participant's plexus in order to record his/her body position and orientation data. In order to use and record the eye tracking data, a calibration of the participant's eyes was necessary. The participant was asked to move within a square of 2.80 m on each side. These boundaries were reproduced in the virtual environment. The participant is filmed. A video of his/her point of view is captured. The eye tracking data and the position and orientation of his/her head and chest are recorded.

The experimental protocol consists in living successively two versions of the montage of 9 minutes and 45 seconds. In the first one, the immersed person always remains at the same scale as the character and we impose to him/her the rhythm of the editing planned by Hitchcock and George Tomasini. Between two editing points, the immersant is able to move in space, but at each cut, he/she is positioned and oriented like the film shot. Whatever we do, the cuts thus bring us back into the authors' point of view at the moment chosen by them. In other words, if I don't move, I see the film as it was cut in the original film. This “Hitchcock” version (we called H) contains 133 cuts and the average time between two cuts is 4.4 seconds. The second experience consists of living the version of the montage edited by us with the changes of scale and the system of cues reacting to the immersant’s behavior. This “re-cutted” version (we called RC) includes 56 cuts and the average time between two cuts is 10 seconds.

The order of the two experiences is arbitrary and varies for each person. After having experienced the first one, the participant is invited to answer a questionnaire relating to the bodily and cognitive appreciations and feelings, the sensation of presence, the relationship to the narration and the editing. Then, the participant experiences the second version. The same questionnaire is therefore to be completed. Some questions, specific to this second questionnaire, concern a comparison between the two experiences.

Moreover, an oral interview of about twenty minutes is conducted allowing the participant to justify his/her answers to the questionnaires, to criticize the experience and to go into more detail about his/her sensations and impressions.

An authorization for the recording, use and distribution of images, videos, sounds and biometric data is signed by each participant.

Finally, we reconstructed in Unity the movements of the participants, enabling us to replay the experience in the 3D space. We can thus replay the experience either on a screen or immersed in virtual reality with a headset.

The protocol of the experiment is presented in this video: https://vimeo.com/779289940. 

Questionnaires’ results

In this part, we present the first observations made from the answers to the questionnaires. These have helped to guide our reflection and to draw some conclusions about the effects experienced through the montage in virtual reality. For each question, the subject could choose a qualitative answer ranging from-2 to 2 (Figure 7).

Figure 7. Format of the answers to the questionnaire. From “not at all” to “strongly”.

In terms of general feeling, our "re-cut" (RC) version of the montage was more appreciated than the direct transposition of Hitchcock's montage (H) in VR. To the question "I was absorbed by the experience", RC gets a better score (0.94) than H (0.53). To the question "I didn't feel confused or comfortable", H is less well lived (0.22) than RC (0.63) when experienced first. To the question "I had the sensation of being there in the virtual environment", the score is better for RC (0.56) than H (-0.13) when it is experienced in second. This indicates that the phenomenon of presence persists for RC even after the startling effect of the VR technological experience has faded. Finally, on the question "I liked this experience better than the previous one”, RC scores higher (0.5) than H (0.0).

A part of the questionnaire also focused on the phenomenological relationship related to the mediation. To the question "Did you feel more immersed in images or spaces?", the subjects feel more in a space during the first experience (H or RC). This seems logical given that when they put on the headset for the first time, they strongly engage their spatial perception system. On the other hand, during the second experience, the immersants feel more immersed in images with H. To the question "I had the impression of sharing the same space as the character", H obtains a better average score on the two experiments (0.76) than RC (0.35) which seems logical given that in H, the immersant keeps the scale of the character. We should note that this impression is more marked for H if it is experienced before (1.0) than after (0.5) RC. This result is interesting to put in relation with the fact that the immersant seems to feel more in images in this last case.

Concerning proprioception, to the question "I felt like I didn’t have a body", RC obtains a null score (0.0) whereas the one of H is negative (-0.47). This could be explained by the fact that in H, immersants don’t have time between two cuts to conscientize their bodies in space. Moreover, the score increases significantly in the second experiment. This confirms the tendency of immersants to move from immersion in a space to images when they get used to the fragmented grammar of the experiences.

Figure 8. Answers to the question: "I was disoriented by the cuts". Left: H, average 0.94. Right: RC, average 0.0.

Regarding the relationship with the "cuts", RC seems to be better experienced. To the question "I was disoriented by the cuts", the difference is very clear between H (0.94) and RC (0.0) as shown in Figure 8. It is also interesting to note that the score becomes even negative when RC is experienced in second. We get the same tendency for the question "I was frustrated by the cuts" (0.35 for H and -0.47 for RC). Finally, on the question "I feel like I missed some important things in the story", RC gets a better score (-1.59) than H (-0.94).

Discussion

These experiments enabled us to refine an analysis of the aesthetic experience of the immersant. Based on the answers to the questionnaires and the interviews conducted with each participant, we were able to extract certain theoretical explorations.

The rhythm of the immersant: a dialectic between looking and inhabiting

When immersed in a virtual environment, the immersants must quickly understand their role. Is it an environment where they can interact explicitly? Are their presences taken into account by the narrative? Should they move or stay still, explore or contemplate, inhabit or watch? According to the answers to these issues, we observe that the immersants engage their perceiving body according to different dynamics.

The rhythm of the immersant is first influenced by his/her own characteristics. According to his/her emotional state, expectations, habits of virtual reality systems, he/she will engage his/her body differently. While some remain still, naturally observing the mediated spaces, others will be active in exploring the limits of the simulation in a more videogame-like approach. We can thus make the hypothesis that there are typologies of immersant to detect in order to adapt a rhythm of montage.

The contract that is created between the immersant and the artwork depends on the expectation of the mediation experienced. When dealing with the device, we observed that some participants believed that they must adapt to make the system work and get the most out of it, while others expected the system to adapt to what they wanted to do. In other words, the contract depended on the responsibility the immersants give themselves in the quality of their artwork reception experience. On this point, the answers to the questionnaires seem to show that the participants live better the experience (RC) when they have the time to position their gaze and to control a minimum their reception.

In our device, the immersant is standing and knows that he/she can move at any moment, which creates a particular relation to the perceived image. We don't observe in the same way when we know that we can look at something else at any moment. The tension of Hitchcock's sequence, created by the rhythm between shots of the character and of what he sees, is changed into a tension between deciding to look at or not to. But if the editing systematically orients our gaze toward what we need to see to follow the story, the exploration becomes meaningless. In this way, the rhythm of the immersant depends on the staging. Does he/she have something to see? If yes, then why should he/she move? A working hypothesis is to decentralize the point of view regarding the salient narrative elements, inciting the immersant to seek a new point of view. The rhythm of the immersant could thus be influenced by the time he/she takes to switch from the exploration of the environment to the stabilization of his/her gaze on an image.

Then, according to his/her scale in relation to the representation space, the immersant needs more or less to engage important movements. A point of view of a giant densifies his/her distancing towards the elements of the scene. He/she does not need to move much to approach any element allowing him/her to embrace globally a large space to situate a scene. This "dollhouse view" engages a heterocentric referential of spatial perception, inducing less body interaction. Being so different from his/her size, the environment is on the one hand not inhabitable, but seems to increase the emotional distance to the narration. 

Moreover, we notice that if the immersant is the same size as the character, he/she will naturally have a more exploratory attitude. Moreover, his/her presence in the representation space will be better induced and the behavioral responses (bending down when the plane passes or leaving the middle of the road when the truck arrives) will be stronger.

Finally, if the immersant is very small in proportion to the environment, the translations of the body have much less amplitude to explore the scene. On the other hand, since the whole world is larger, head rotations have much more impact on changing what the immersant sees in its field of view. In other words, the more we reduce the size of our virtual body relative to the environment, the more we are confined to a localized area but the more we have to turn our head to follow the narrative's salient elements.

Each scale brings a different relationship to our potential actions and to the more or less tangible nature of the representation space. They offer specific qualities of perception and engagement ⁴. These relationships evolve over time and will be shaken up by the discontinuities of the montage.

The rhythm of the editing: between image rhythm and space rhythm

The rhythm of the editing gives a limited temporality to the action and perception characteristics of the immersant's scales. The spatial reference frame does not persist, which leaves only a limited time to put in relation the physical body and its virtual scale. According to the delay between each cut, the immersant is more or less conscious of the presence of his/her body in the representation space. If the montage is slow, then he/she can get used to the scale of his/her body, go from perceiving an image to inhabiting the space, exploring it, building a point of view and having the time to re-stabilize the gaze on an image.

If on the contrary the montage is fast, the image does not have time to become a space. A frenetic editing may not leave time for a behavioral response. We saw in the results that it also may affect the awareness of the perceiving body. We make the hypothesis that these images (not yet spaces) are printed differently for the perceptive system and modify the inner construction of the space. This could be confirmed by the fact that participants felt more immersed in images when they made H (133 cuts) after RC (57 cuts). The rhythm of the montage has thus a role in the transformation of the space into image or of the image into a space, and thus in the nature of the mediation.

Depending on the scale and initial point of view (initial gaze frame) imposed by the cut, the editing can also impose a movement by decentering the viewer in relation to the action. The proposed point of view would then be perceived more as a position in space than as an image. This spectatorial decentering should invite the immersant to move in a certain way towards the search for framing an image. Through editing and staging, the immersant can thus adopt a certain rhythm linked to the recentering of his point of view on the action. This has the effect of changing the nature of the cut, which here would be more of a collision than a continuity.

Matching the rhythms

Discontinuities in perceptual experience are not natural in the framework of mediation in virtual reality. There is a difficulty in repositioning ourselves in a sudden new virtual space ¹¹ and a need to provide the immersants certain visual cues so that they can orient themselves ¹². We saw in the questionnaires’ answers that in H, cuts were often frustrating, disorienting and meaningless. In interviews, participants often complain about the beginning of H when there are cuts every three seconds between the character and what he sees. We hypothesize that in order for them to be accepted, and especially to acquire a meaning, the cuts must be part of the acceptance of a cinematographic language of the experience. It would therefore be necessary that before and after the cut, the immersant shall be focused on an image rather than in the process of exploring. By detecting the moments when the immersant is looking for a point of view, we can avoid making cuts at such moments. When this exploration phase ends and the gaze stabilizes on a final gaze frame, the montage can then propose a change in response to the stabilized image.

Then, in order to match the rhythms, it also seems important to pay particular attention to the first contacts between the immersant and the montage. A montage too fast at the beginning can discourage the subject from moving. It seems important to allow time for exploration at the start of the experience and to gradually change the pace as the language becomes better established. The complexity lies in the fact that we must both intuitively invoke our culture of film editing while at the same time liberating our bodies in space. If we realize that it is the role of the editing to show us points of view, then our role is no longer to go and find them by ourselves. That is why it is important for the editor to take into account this progressive learning of the immersant to set up the cues and their durations. By creating this relation between cues and immersant, the role of the editor is here fundamental in the harmony of the rhythms and in the resulting aesthetic impressions.

Narrative effects

While the conducted experiments have raised new phenomenological questions, they have also opened up new directions of reflection in the theorization of narrative effects linked to montage in virtual reality. Firstly, thanks to the system of cues, when the cut is made between two stabilized points of view, we can observe the persistence of a “Koulechov Effect”6 in virtual reality. In other words, the creation of a narrative meaning extending beyond the two separate points of view. We might also explore the difference of this Koulechov Effect if the immersant passes from his/her point of view to the one of the montage or if he/she chains two points of view of the montage (if he/she doesn’t move between two cuts).

A second track to explore is the multiplicity of points of view that the montage proposes or prevents. As the “framemaker” of his/her own experience, the immersant can decide to look at a character or at what the character is looking at. This engages different effects of identification (with the character or with his/herself) and testifies to his/her empathetic or sympathetic relationship towards the character. These choices vary the level of presence in the environment and the distance to the narrative. By imposing changes in scale and point of view, editing can in turn constrain some of these effects in time. Moreover, depending on these changes in relation to his/her attitude, the immersant may or may not feel that he/she is missing events.

We have seen that according to the scale, the immersant more or less inhabits the virtual environment. We hypothesize that with the size of Cary Grant, the immersant engages more behavioral responses to events (at the character’s scale, it seems normal to feel more embodied and therefore in danger). On the contrary, in larger scales, the immersants feel more like a "framemaker-body" of their own experience. By framing the representative space, they don't change the story but the way they perceive it and thus their focus and aesthetic impressions of the story. The immersant would then be a "director-body", seeking to transform space into image. In order to be an "editor-body", he/she would have to make a conscious choice between different images in time. Perhaps the "editor-body" emerges when the control of spatio-temporality becomes conscious.

Finally, when setting up the cue system, we systematically asked ourselves the following question: is the interaction implicit or explicit? If the immersant realizes that his/her gaze actions have an influence on the editing rhythm, then the narrative experience changes. The montage may thus become embodied and open up new sensitive connections with the ongoing narrative.

Conclusion

In this project, we have explored how montage theory needs to be adapted to the new context of room-scale VR. We found that proprioceptive immersion in a virtual story world can indeed provide the necessary “cues” to motivate shifts of points of view and scales that effectively guide the audience through a narrative experience. These meaningful discontinuities create a rhythm in the experience that the audience can follow and enjoy if they are correctly synchronized with his/her internal bodily rhythm.

Depending on the montage rhythm and its adaptability to the attitude of the immersant, we observe a dialectic between the exploratory nature of the space and the narrative nature of the image. Our experiment offered the opportunity to analyze how the dynamics of the montage may suggest to the immersant an exploratory rather than a spectator attitude, or vice versa. It constitutes a user study of how our spatial habits can harmonize with our visual habits of cinematographic images in order to make sensible the oscillation between constraints and liberties.

We made the hypothesis that the montage must also adapt to the rhythm imposed by the immersant. Thus, we explored how the montage can adapt in real time to the behavior of the immersant through implicit interactions (focus of the gaze, objects present or not in the field of vision, speed of displacement and rotation of the head), while respecting a meaningful narrative structure. The artistic and narrative impressions of the space intervals result then from a compromise between the proposals of the immersant and of the montage set by an editor and actualized by the software.

In future work, we would like to explore further this new form of relationship between the immersant (present and acting) and the author (absent) by introducing a richer repertoire of cues and actions, and creating even more variations in the resulting montage.


(1) Person immersed in a virtual environment. Crop duster scene timecode: from 1:06:17 to 1:15:55.

(2) North by Northwest, Alfred Hitchcock, Metro-Goldwyn-Mayer, 1959. https://www.imdb.com/title/tt0053125/ 

(3) https://remisagduv.com/the-hitchcock-experience-a-spatial-mont age-project/ 

(4) https://atap.google.com/spotlight-stories/

(5) https://team.inria.fr/anima/

References

1 Alfred Hitchcock, François Truffaut, Helen Scott, Hitchcock/Truffaut, Paris, Gallimard, 1993.

2 Gray Hodgkinson, “Cut, don’t cut-moving the viewer through a story in animated VR,” in 7th International Conference on Illustration and Animation, 2019.

3 Scott Macaulay, “Look Into the Cut: Jessica Brillhart on Editing VR,” Filmmaker Magazine, October 28, 2015.

4 François Garnier, “Artistic Practices in Digital Space: An Art of the Geometries of Movement? (2021),” in Space-Time Geometries for Motion and Perception in the Brain and the Arts, ed. Tamar Flash and Alain Berthoz, Cham: Springer International Publishing, 2021, 237–50.

5 Vanessa Pope et al., “The geometry of storytelling: Theatrical use of space for 360-degree videos and virtual reality,” in CHI Conference on Human Factors in Computing Systems, 2017.

6 Sylvia Roth, et al., “Where to place the camera,” in ACM Symposium on Virtual Reality Software and Technology, 2019.

7 Sylvia Roth et al., “The impact of camera height in cinematic virtual reality,” in ACM Symposium on Virtual Reality Software and Technology, 2018.

8 Rémi Sagot-Duvauroux, François Garnier and Rémi Ronfard, “(Re-)Framing Virtual Reality,” in WICED 2022 - 10th Workshop on Intelligent Cinematography and Editing, April, 2022.

9 Raymond Bellour, The Analysis of Film, Indiana University Press, 2000.

10 Ogechi Nnadi, Ute Fischer, Michael Boyce, and Michael Nitsche, “Effect of Dynamic Camera Control on Spatial Reasoning in 3D Spaces,” in ACM SIGGRAPH Symposium on Video Games - Sandbox Symposium, Los Angeles, California, August, 2008, 9–10.

11 Michel-Ange Amorine, Ben Trumbore and Pema L. Chogyen, “Cognitive Repositioning inside a Desktop VE: The Constraints Introduced by First-versus Third-Person Imagery and Mental Representation Richness,” Presence: Teleoperators and Virtual Environments 9 (2), 2000, 165– 186.

12 Karl E. Steiner, Lavanya Voruganti, “A Comparison of Guidance Cues in Desktop Virtual Environments,” Virtual Reality 7 (3-4), 2004, 140–147. 

Animate: A Theatrical Exploration of Climate Transformation through the Medium of Extended Reality (XR)

Chris Salter, Timothy Thomasson, Pierrick Uro

Immersive Arts Space, Zurich University of the Arts, Concordia University, McGill University 
christopher.salter@zhdk.ch , timothythomasson@gmail.com, pierrick.uro@mail.mcgill.ca

Abstract

This paper presents a critical account of the development of a large-scale theater work using emerging Extended Reality (XR) technologies. Detailing three aspects of the project and set against theoretical frameworks from STS (Science and Technology Studies) and the sociology of innovation around ideas of the future embedded in technologies, we examine the conceptual, aesthetic, organizational and social-technical underpinnings of the project. The paper’s goal is thus to give a sense of the challenges and opportunities in the emerging integration of XR into new artistic morphologies that hybridize the visual-performing-media arts through new technological advances.

Keywords

XR, VR, AR, Spatial Computing, Passthrough, Presence, Climate Collapse, Technological Futures, Performance, Vitality Affects.

DOI

10.69564/ISEA2023-83-full-Salter-et-al-Animate

Introduction

Animate is an Extended Reality-based theater work (XR) at the crossroads of performance, radio play and installation focused on a near-future Canada radically transformed by climate change. Developed by a team of Canadian and German artists and researchers, the work premiered in Summer 2022 in large theater festivals in Germany and is preparing for a 2023 international tour in Europe, Canada and Asia.

XR is a umbrella term emerging in the mid 2000s that describes computer-generated environments accessed and experienced through worn headsets and body interfaces. These environments are either simulated (virtual reality-VR) or overlay and mix real scenes with digital 3D images and sounds (augmented reality-AR). As part of its dramaturgical strategy, Animate harnesses a recent XR technology to explore this potential mixing of real and the simulated in worn AR: live video “passthrough” which takes a real time video feed from the tiny cameras attached to a head mounted display (HMD) and which allows the embedding of computer generated 3D objects into the video feed of the real environment.

This paper details three aspects of the project, set against theoretical frameworks from STS (Science and Technology Studies) and the sociology of innovation and expectations focused on how the future is embedded in both technological discourses and materials. In Part 1, the most extensive section, we discuss the initial concept and dramaturgy of the work. Although the technologies utilized in the project and the resulting experience for the audience can be claimed as new, the concepts that enable these possibilities date back to the 1960s with the development of early Virtual and Augmented Reality technologies.

Second, while numerous other media were key to the artistic process, space allows us only to describe the visual aesthetic concept behind the work. We focus on how our approach both differs from many standard VR-based projects while also articulating the technological background that made such artistic choices possible. In Part 3, we analyze the lead up to the final presentation of the project in a major theater arts festival in Germany. Here, we pay particular attention to an area which is rarely focused on in accounts of technologically-driven artistic practice – the coupling between the profound instability of emerging technological infrastructures and the manner in which such infrastructures ultimately affect artistic decision making.

What theoretically grounds the paper is recent work in STS (Science and Technology Studies) on issues of “boundary work,” “technological futures” and “technological promises” together with emerging research concerning the rapidly transforming understanding of presence through XR technologies such as video passthrough. These technologies challenge long assumed ideas of presence in VR based on the “illusion” of place ⁶ and instead, suggest new scenarios where users become located “between the digitality of VR and the concrete reality of a user’s surroundings.”¹⁸ This theoretical work is critical in grappling with new artistic possibilities of XR technologies in addressing larger aesthetic-social-technical issues the project is focused on; namely, the symbiotic “interrelationships between media technologies, environment and body.”² 

I. Background and Concept

Conceived in 2019 by one of the authors of this paper, the original impetus of the Animate project was to explore the thematic of climate change through the sensorial-aesthetic possibilities of “spatial computing.” A term that emerged in the mid 2000s and encompassing a range of computing and HCI research, spatial computing can be defined as “human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces” ¹³. Although spatial computing’s genealogical roots lie in environmentally-based computing paradigms like “ubiquitous computing” ²³ and “enactive” or “embodied interaction” ⁹ from the mid 1990s, the principal application area for the concept arises in the domain of Augmented Reality in which digital information is overlayed and integrated onto a real environment.²

Primarily accessed on mobile devices (smartphones and tablets) until the mid 2000s, more recently large amounts of R&D have been poured into the design of “wearable AR” headsets from large tech companies like Microsoft, Google, Facebook (renamed as Meta) and importantly, Magic Leap. Magic Leap is a much-hyped US-based start-up who developed a wearable pair of AR-based glasses with extensive environmental sensing capabilities for the consumer market throughout a long stealth period; a device which spectacularly failed to catch on shortly after its 2018 release. The commonality among all of these AR technologies is two-fold.

On the one hand, they all seek to enable a computing paradigm that has long been theoretically discussed and experimented upon (but with few widespread applications) called “Mixed Reality” (MR), in which the computer-generated world and the physical world meld, blur and blend. MR is situated on what researchers have called the “reality-virtuality continuum”¹⁵: a well cited paper in computer science that describes a taxonomy between completely simulated (or virtual) experiences and real ones.

On the other, wearable AR presents a social-perceptual challenge. While VR uses a headset to completely surround the wearer with a closed off world of computer-generated images and sounds based on sensing head and (mainly) hand movement, wearable AR demands a negotiation between what Azuma names as “virtual content that is integrated with the surrounding real world, while users remain engaged with and aware of that ‘real world’” ⁴. It is this conceptual idea that forms the basis of the entire Animate project.

Spatial Computing and CliFi

The comment above from Azuma acts as a strong framework for the choice to explore a critical issue, that of climate collapse, through the technological-phenomenological context of wearable AR/XR. After surveying the state of the art of wearable AR technologies in 2019-2020 and spending part of the pandemic experimenting with the see-through Magic Leap system, the artistic team gathered for the project reached two conclusions. The first was that the Magic Leap or similar were too immature and costly for use in a professional live performance artistic context due to multiple factors: comfort, low resolution, limited battery life and most important, extremely limited field of view (FOV)—a term used to describe how large an augmented reality image is when viewed through a headset.

Given the state of art, the second and more important consideration was that a purely technological exploration of XR in typical “demo mode” would not be artistically adequate. These conclusions thus lead (after almost 6 months of reading different texts) to the concept of staging a narrative Climate Fiction (CliFi) short story called “Animate” by a Canadian author; a story which was discovered in a Canadian “CliFi” (Climate Fiction) anthology.

The plot of the story “Animate” focuses on two characters, Daniel and Laurie, who are fleeing a near future of climate disasters. Emotionally distraught from their hidden pasts, the pair set out on a journey through the wild landscape of Newfoundland, Canada. While news of global climate catastrophes is broadcast on their car radio, they drive towards the Tablelands, a real Mars-like landscape in the middle of Gros Morne National Park where one can directly walk on the mantle of the Earth. Gradually, the climate-change transformed landscape exerts a force on them as if it was a conscious, living and breathing entity. It emits strange sounds and affects Daniel and Laurie’s personal relationship. But it is only in the dramatic conclusion, where the earth comes alive in an apocalyptic scene of rocks rising from the earth and attacking the characters, where the intricate interconnectedness of humans and a natural world under the throes of radical environmental change is symbolically and viscerally manifested.

The basic dramaturgical strategy that evolved in the early phases of the project (during a period of conceptualizing while fund-raising) involved a collaborating with the author to adapt the short story and turn it into four dramatic scenes (Driving to the Tablelands, Camping Overnight, Hiking to the Tablelands and the Earth Strikes Back) like a theatre text, then record actors speaking the text for a radio play and finally use the radio play as a complex sonic accompaniment with additional visual elements in XR for a live theatre-like installation with actors.

Walkthrough and Passthrough

It was clear from the start of the Animate project that due to the potential production and presentation contexts, the project would need to perform a kind of “boundary work” between the genre of theatre (involving human performers) and the technological environment of an audience wearing HMDs in order to enable an interplay between real physical space and the imaginary world of computer augmented space.

A term coined by the sociologist Thomas Gieryn, boundary work describes work in which “boundaries, demarcations, or other divisions between fields of knowledge are created, advocated, attacked, or reinforced” 11. Most often such demarcations are described in more “high stakes” ideological contexts as the natural sciences. Yet, one should not underestimate the different communities of practice that operate even in assumed “fluid” artistic contexts. Indeed, the insistence on one of the co-producers of the project (themselves anchored in theater performance contexts in a European cultural festival context) that we should include human performers to appeal to a more traditional theater audience who might be suspicious of the potential “media barrage” brought on by utilizing AR technologies clearly demonstrates the occurrence of such boundary work .

The external demand of adapting to certain kinds of theatrical norms in both organizational and artistic ways as well as the aesthetic choice to emphasize audio over computer-generated image (due to technical limitations in AR headsets described above), posed a dual conceptual and artistic challenge in terms of integrating live performers into the larger XR context. This context thus drove a fundamental artistic choice from the start of the project: the idea that the audience would move from one physical space to another while they listened to the spatialized radio-play based on the short story “Animate” over their HMDs.

This radio play accompanies installation with performers idea isn’t entire original but inspired by the work of a 1980s San Francisco Bay Area theater-technology company who grappled with the similar situation of trying to integrate a new technology (at the time) into live performance: the Sony Walkman. Known more for their invention of a wireless museum guide/tour system that later became the largest audio tour company worldwide, in the 1980s, Antenna Theater, directed by the American theater maker Chris Hardman, pioneered a new form of walk-through performance-installation which Hardman labeled “Walkmanology.”
 “I’ve always found airplanes to be claustrophobic, there is always a kid crying in the back and you forgot the book you intended to read so you realized that you’re stuck there for hours with nothing productive to do. Then this new fangled bizarre device came out on the market called a Walkman and I decided to get one. I turned on the Walkman and just as the airplane lifted into the sky, Wagner’s “The Ride of the Valkyries” began to play. Suddenly I realized that there was this amazing theatrical event happening and it was called Syncronicity! The visual and the audio were working in sync and furthermore instead of watching from afar, I was literally inside the event” ¹.

In many ways, the choice in Animate to use the emerging technology of AR follows a similar path dependent story as Antenna’s work with the Walkman: treating the technology of worn AR as a device that accompanies a larger dramaturgical experience rather than creating theatre work inside VR or AR 17. Yet, what is key to the project is the emergence of a technology during the early phases of production that was unknown at the time of the planning of Animate: the 2021 introduction of a new generation of wireless VR headset from Oculus/Facebook/Meta which utilizes a video technology called “passthrough.”

Originally designed as a safety feature to alert VR users when they would potentially go beyond a demarcated spatial area called “the guardian,” passthrough uses the tiny black and white infrared cameras built into the Oculus Quest 2 HMD which enables the device to sense where it is in space. These cameras deliver and process live video images of the outside world inside the headset in place of our eyes, approximating what one would see if directly looking into the real surrounding world. Importantly, what passthrough technology does is transform VR devices (in this case, the Quest 2), which traditionally immersed the user in a closed off, socially isolated world, into a new kind of wearable augmented reality where digital objects can be embedded/overlayed onto the real world via the live video image and one sees/senses the physical presence of oneself and others.

The introduction of passthrough technology as a kind of stop gap immersive AR not only represents a technical change – it also suggests an epistemological and phenomenological one as well; a transformation that is both historically grounded while, at the same time, shifting concepts of presence that have been long established from research into human interaction and experience in VR. Indeed, while the passthrough technology was already included in the first release of the Oculus Quest Pro in 2021, the ability to manipulate the passthrough image by gaining access to it through the device’s SDK (software development kit) was only available after production on Animate started in December 2021.

Technological Futures courtesy of the Metaverse

Yet, a larger question around passthrough arises. Why did Meta move to allow users to access and manipulate the live passthrough camera image, thus turning the Oculus Quest 2 into a “immersive” worn AR / MR device? The answer to this question suggests a larger narrative at play rather than simply creating a technology to ward off potential accidents with users who would stray too far from a safe area while playing games in their living rooms. Indeed, it wouldn’t be a stretch to claim that the passthrough-based AR capabilities of the Quest 2 are tied into a larger socio-technical imaginaries of Meta’s “technological futures” and “promises” for the software/hardware integration of platforms to enable the so-called “Metaverse.” That is, to enable people to begin to occupy both physical and computer-generated spaces that the Metaverse seems to promise.

Recent work in the sociology of expectations and STS has focused on the concept that certain imaginaries of the future are performative in that “expectations, visions, scenarios, and other forms of anticipation affect what may actually happen” ¹⁴. This idea of technical “visions” describes certain normative conditions – of what is desired by a particular constituency rather than what is plausible or needed from a particular technology, setup or infrastructure. Normally, scholars study such visions under the context of discourse, including statements concerning “practices that systematically form the objects of which they speak” ¹⁰. Yet, the transformation of a particular material technology such as a set of cameras originally used for sensing the spatial orientation of the Oculus Quest 2 by way of software additions, also results in a new perceptual paradigm (this is the case of video passthrough). This suggests a literal materialization of such practices of discourse that Foucault speaks about through a purposeful management of “future expectations.”

As STS researchers are keen to point out, the production of expectations around emerging “breakthrough” technologies “which promise a vast potential of market prospects and solving societal problems and create a sense of urgency in the context of international competition,” is instead conceptualized as a “regime of economics of technoscientific promises,” rather than a social-technical imaginary in which alternative futures that are equitable and desirable for larger collectives are produced.¹⁴

Such clearly seems to be the case of the “Metaverse,” which is announced as future technology to “change human interaction as we know it.” There thus seems to be no better way to manage future expectations of an unproven technology platform like the Metaverse than to construct a taste of what such a “proximate future,” a future that is always on the horizon but never comes,⁵ might be in the present.

Shifts of Presence

The introduction and manipulation of such passthrough technology not only changed our technological approach in Animate, allowing us to move from Magic Leap-based Augmented Reality to that of passthrough delivered video image. The possibilities of passthrough also changed the aesthetic-perceptual framework as it quickly became clear that visitors would have to wear and adapt to the Oculus Quest 2’s bulky form factor and closed off environment throughout the performance.

But the use of these worn AR technologies within Animate raises also important phenomenological questions as well, namely, how the sense of presence of oneself and others is actually reconfigured through technologies that allow full body experience. It must be stated that the idea of one being able to confront a mix between the computer-generated world and the real one (albeit one brought by cameras as in passthrough) is an idea that dates back to computer graphics researcher Ivan Sutherland at the University of Utah in 1968.

Sutherland is credited as one of the first to develop the technology (if not the concept) for the head mounted display – what he famously termed “the ultimate display” in a 1965 visioning article. Yet, Sutherland’s description of a “display connected to a digital computer” [that] gives us a chance to gain familiarity with concepts not realizable in the physical world” ²¹ which sounds like the basis for current virtual reality, is actually enlarged in a follow up technical paper in 1968 which describes the actual construction and workings of a 3D head mounted display. Here, Sutherland gives already a sense of the Quest’s (and other) push towards integrating passthrough technologies into the real environment. “Half-silvered mirrors in the prisms through which the user looks allow him to see both the images from the cathode ray tubes and objects in the room simultaneously. Thus, displayed material can be made either to hang disembodied in space or to coincide with maps, desk tops, walls, or the keys of a typewriter” ²².

At the same time, however, what is more interesting about the introduction of passthrough technology is that it also enables a paradigmatic shift in another area of knowledge and consequently, experience (which is why we term its effect both epistemological and phenomenological). Passthrough changes the understanding of one of the core foundations of VR-based research which is that of presence.

In fact, the introduction of consumer grade AR that mixes the real and simulated together shift not only the user’s experience of their own presence but also “the conditions in which human individuals interact with one another face to face from body to body” ²⁵ ” what Goffman ¹² famously called co-presence. As key XR researchers admit, while these technologies have existed for decades, “little is known about how social interactions are affected by the technology” ¹⁶ and how social interaction in turn shapes these systems. Part of this gap stems from the overarching focus in VR/AR research on an individual experience of “presence” as “outwardly ‘dislocated’ from its physical setting.”¹⁸,

It should also be clear that the move to worn AR prototyped by Sutherland in the 1960s and commercially available in 2021 is a marked contrast to earlier conceptions of presence in VR understood as the “virtual experience of being in an environment, even when one is physically situated in another.” ⁶, ²⁴ While Animate’s focus was not specifically on researching how such co-present interaction is reimagined (and vice versa) through camera-based technologies like passthrough, the experiential effects of being embedded in the real environment while also experiencing Sutherland’s vision of a “mathematically generated wonderland” clearly constituted a necessary experiential context for the production.

II.Aesthetic Considerations

We have so far described some of the historical, artistic and sociological background of Animate. But the larger question looms of how all of these ideas actually play themselves out in the trenches of practice? This section Thus, focuses briefly on two of the principal image-based media that forms the basis of Animate during the approximately seven-month production period, from late November 2020 up until production in Germany in August 2021.

Alternative 3D Images in VR Land

The visual aspect of VR is often intertwined with historical theorization around cinema and as an extension of early immersive technologies such as panorama screens. However, with new XR technology the language borrowed from cinema [camera, frame, mise-en-scène, cut, montage, projection, screen] can only take one so far.

This situation prompts a complete reconfiguration of artistic and dramaturgical decision-making when sequencing elements of an XR work. Virtual objects are no longer elements to be ‘looked at’, but instead have the potential to be investigated from all angles and vantage points. The salient attribute of these technologies is their ability to render virtual spaces such that they perceptually feel ‘real’. Although the illusionistic 3D space projected onto a flat surface remains technically ‘flat’ in XR, perceptually the virtual rendering of objects and the physical space in which those objects are situated becomes integrated. As such, in the context of XR, we could no longer rely on the accustomed conventions held by the cinematic language (perhaps suitable for 3D of VR work), as one must edit and configure images not only over time, but also over physical space, while also being aware of how bodies might move through that time and space.

In the second act of Animate, the switch to passthrough AR immediately following a rope guided VR ‘journey’ provided a direct perceptual contrast between the experiential qualities of both XR modalities. During this dizzying rope guided journey, the participants must meander through a foggy Tarkovsky-esque virtual trail with distant hills and trees, listening to spatialized audio while losing their orientation in the physical space. Here we planned to take a restrained approach to the image, rather than adhering to the cultural expectation of visual effects overload, such as the case in many VR works. This scene transitions into an opening of the passthrough image.

One realizes the sealed off, socially isolated quality of VR when participants can suddenly see their bodies (rather than virtual avatars as is the case with many other XR works), other participants, the real environment, and the performers. As such, developing a dramaturgy that unfolds to accommodate both VR and AR technology—and not either/or—reveals the perceptual nuances of these technologies as held in such striking temporal and dramaturgical contrast.

As previously stated, in the context of Animate, the passthrough feature was deployed outside its intended practical use case of creating a guardian safety boundary for a VR experience. Due to its practical functionality, the passthrough image produces a low resolution, grainy, black and white image. This image as incorporated into Animate suggested an early film aesthetic, as well as the contrasted black and white images of films from Antonioni or Tarkovsky. Indeed, at points in the work we lean into this aesthetic quality, adding simulated grain over virtual objects, so they feel visually and perceptually integrated into the passthrough image. In other instances, such as the initial reveal of the Tablelands environment, a highly saturated virtual space is presented—portraying a vivid full colour virtuality at odds with a more subdued black and white passthrough reality. Meta’s recent release of the Quest Pro headset comes with high-definition full colour passthrough. We are currently experimenting with the technical and aesthetic affordances of this new headset for the upcoming performances of Animate in 2023. This speaks to the technical and artistic adaptability that is required when working with such emerging technologies.

As the passthrough image appears in the last scene, both virtual and real rocks are revealed to be scattered around the physical space. As the narrative continues, these virtual rocks begin to move in a carefully choreographed way– climaxing in an explosion and in a clustering and swarming pattern, as if they were birds or insects. The sites in which the performance has taken place thus far were remodelled such that the virtual elements spatially could overlay the physical architecture to real-world scale as seen through the headset. These virtual models are hidden from the spectator’s vision, but remain in the software’s memory to carry out physics calculations and render occlusion. Due to these virtual mapping techniques, the rocks collide with the physical environment, crashing into walls, pillars, and the floor. Participants can extend their hands and bodies and ‘touch’ these rocks, which in turn causes them to move as their physics are simulated in real-time.

Interacting with Flying Rocks

As stated in Part 1, Animate ends with a spectacular scene in which audience members find themselves in a field of rocks, some virtual and some real. As the scene progresses, the simulated rocks begin floating, moving around and seemingly gain consciousness as they chase the actors and audience members. Any of these rocks can be interacted with by pushing them around using the hand tracking featured by each headset. However, with the aim to give these 3D objects more corporality, we focused on interaction between the rocks and the audience.

Early live tests revealed that allowing hands to directly collide with the rocks made it too easy to introduce energy in the environment, as a simple slap could effortlessly send a massive rock flying to the other side of the room. The adopted solution was instead to implement a force field around each hand that continuously pushes away rocks that get too close, with more power the closer they get. This approach restricted how much energy, and thus chaos, could be injected into the system, limiting the amount of control one has over the rocks and guaranteeing an agency to the objects. As the scene progresses in intensity, the rocks gain more energy, which makes it easier for them to ignore the attempts by the spectators to alter their course or shield themselves, denying them the drive to disrupt the harmonious choreography.

Since the artistic intent was to virtually drown the participants in boulders and pebbles, just like the characters in the story are swallowed by the earth, the number of virtual rocks totaled over 600 individual objects. This made it impossible to perfectly synchronize the rock’s state across a dozen headsets over a WiFi network and thus demanded an alternative procedure. A way to combine interactable rocks, locally solved physical simulation, and a behavior recurrent enough for a scripted show is to implement a flexible system. Since all headsets were to be spatio-tem- porally aligned, they all could share the exact same behavior, which instead of controlling the rocks directly, positioned their targets. Throughout the scene, the floating objects were assigned different moving targets that serve to steer them on the stage since the rocks would be continuously pushed towards their target. As the targets’ behaviors are determined, the rocks could then flexibly and independently react to a spectator’s actions, synchronizing the state of each object and solving simultaneous interaction over the network thus becoming unnecessary.

Both measures undertaken ensured that even though the state of the system on a microscopic scale would differ for any spectator and performance, it would also stay similar enough on a macroscopic scale to fulfill three requirements: the audience members could feel as though they are sharing the same environment; the actors could move and react around learned cues on stage; and the spatialized room-scale sound (which would eventually be synched to a 49 channel audio system) would stay consistent in time and space with the current state of the virtual environment.

III. Production and Transformation

In part 3 of this paper, we describe the final production period of Animate, a relatively short time frame of 10 days that took place in the actual location where the work was to be premiered: one part of the interior of a vast, almost 100,000 square meter former factory that manufactured agricultural machines for the entire former East Germany.

Like many works involving the integration of new technological infrastructures, the technical production of Animate was beset with challenges. More specifically, the ontological nature of theater as a distinct time-based event which demands an integration of multiple elements poses a major challenge to the use of technologies like Meta’s VR headsets that have been explicitly designed and engineered for a different scale of use: small spaces like living rooms. Here, we want to focus on two specific issues that had major artistic implications: (1) the difficulty of integrating live performers into passthrough-based AR; and (2) the mismatch between an organizational workflow derived from theater versus one derived from VR-AR development and production.

Live versus Virtual

From the start of the Animate process, the core artistic concept was to leverage the relations and tension between live theater with performers and passthrough-based AR. This was to be conceptually accomplished by the performers, who would be “live” actors essentially “acting out” in “dumbshow” format (conveying a meaning or message without speech through miming) core scenes while being accompanied by the overall spatialized musical/sonic/text/visual “score” accessed by audience members via the worn HMDs.

This concept in practice, however, proved to be a significant challenge. First, due to the high cost of the professional (and in this case, well-known) actors who were active in the German theater scene, the amount of rehearsal time required to integrate these performers was extremely short (5 days). Second, the complexity of mixing 3D graphics into the real scenes by way of passthrough was grossly underestimated, particularly since the actual performers were only integrated during the technical production period and not during development in Canada.

Since the performers could not see what was going on in two crucial scenes (Scene 2, “Camping” and Scene 4, “Tablelands”) since they wore no HMDs themselves, it was increasingly difficult for the actors to know where virtual objects in the space were located in relation to real objects. More pointedly, 3D game engines as creation and rendering environments for XR production are WYSINWIG (What you see is NOT what you get). In other words, what appears on the screen has little to do with how such 3D objects are perceived when they are embedded into real environments via passthrough technology. This issue of not being able to see and hear entire scenes in Animate without wearing an HMD reached an almost comic proportion when the dramaturg/movement consultant who was supposed to take notes to aid the artistic director and the actors, was unable to perform these duties since all of the HMDs were occupied for testing by the technical-design team.

A third and major difficulty arose in the extremely low quality of the black and white passthrough video feed; an issue exacerbated by the harsh lighting conditions in the industrial hall where the performance was to take place.

While the team tested passthrough technologies continually during the development process, the original concept was that there would be no scenes with human performers without accompanying 3D objects and behavior of those objects (like the complex rock scene) acting alongside the human performers. The pressures of production deadlines combined with the extreme latency of framerates with the Quest 2 led to the reduction of all heavy 3D graphics, thus transforming the overall visual aesthetics of the two crucial scenes with the actors. Indeed, watching highly trained performers miming a complex dramatic text through a black and white and frequently distorting video feed using an uncomfortable and awkward device proved to be too much.

The Vitality Gap

While the above issues demonstrate that the integration of live performers into passthrough XR is not trivial, a much larger issue was present in the final production period; one which led to a radical reorganization of the entire production twenty-four hours before the premiere for audiences and press. This core “show stopper” involved the inability of the technical team to make needed artistic changes in real time to virtual environments in Unity 3D, based on the direct real time experience of working with the actors.

Unlike the production process of VR applications like games or already produced experiences such as film, the real time experience of making theater with performers and machines is completely different, akin more to improvisation than to the filmmaking or gaming production workflows that seem to be default for XR development. Live performance practice demands a sophisticated attention to timing and rhythm, as well as highly tuned dynamics in order to ensure that the flow of action reaches what the psychologist Daniel Stern called “vitality affects” – in which “physical action and traceable mental operations" become "inherent in the act of [both physical and mental] movement” ²⁰. Vitality affects, which involve the dynamic flow of time through the expression of rhythms, pulses, changes of tempo and direction are experienced in the entanglement between human bodies and nonhuman objects and processes like light, sound and vibration outside of us. Such “forms of vitality” sweep us up into dynamic moments of intensity.

Unfortunately, such forms of vitality were extremely difficult to construct between the human performers and the workflow of XR production. In contrast to real time audio-video processing environments, where developers and creators can make changes essentially “on the fly,” based on changing dynamics of performers and light, sound or image in the ever shifting context of the performance environment, the sheer amount of time required by the workflow of making changes in Unity 3D and then uploading (“building”) those changes to the hardware-based Oculus Quest 2 (bordering on hours) made quick changes extremely time consuming and inefficient. This inability to rapidly change or eliminate elements or even scenes that did not work or whose timing was aesthetically problematic in the heat of rehearsal proved to be major hindrance to creating a dynamic live event.

The Nature of (Dramatic) Change

It goes without saying that that Animate’s live performance context presented both a formidable challenge as well as opportunity for thinking of the further development of XR as an aesthetic-social-technical domain of knowledge and experience. Indeed, the challenges that the performance context presented were indeed so formidable to the ultimate successful artistic realization of the work that Salter undertook radical changes one day before the public premiere: completely eliminating two of the core scenes (Driving and Camping/Dream) of the production that the team had worked months on and replacing these with a much more simplified concept involving the actors dramatically reading the original short story to the audience for the opening scene.

After this initial scene, in which the actors sit underneath the camping scenography originally designed for the integrated production between performers and XR, the audience is brought by the actors into a second space: the enormous main part of the industrial hall. It is here that the media experience promised by Animate begins. After the ubiquitous onboarding sequence, where the audience receives Oculus Quests 2 and specially designed headphones that allow audio from outside to also be audible inside the headphones, the actors collectively lead the small audience who are connected together with a rope through the space while the group experiences the VR part of the performance: in the dramaturgy of the work, the experience of hiking to the top of the Tablelands. Lasting approximately seven minutes, the scene ends with the audience briefly inside a 360-film shot in the actual tablelands in Newfoundland; a landscape which gradually fades, revealing the actual physical environment of the industrial hall via video passthrough.

Finally, in the dramatic last scene, the audience is plunged into a full XR experience – essentially, the original vision of Animate as a work focused on the affective and bodily impact of climate transformation. A field of detailed, 3D modelled rocks lie on the floor courtesy of the passthrough-VR mix. As the scene progresses and the landscape becomes alive, the rocks slowly lift off the floor, accompanied by thundering, multi-channel audio in the physical room and in the headsets. The actors begin to dance as blinding light from the other side of the enormous space bursts into the black and white video feed in the HMD, essentially causing a momentary white out for the audience. As the actors begin a kind of dans macabre, eventually disappearing from the scene, the assemblage of 3D rocks gathers speed, first assembling in a rotating ball and then exploding across the physical-virtual space. The performance concludes with the long planned Endzeitsturm (apocalyptic storm) of rocks, that spin in a gravity-defying machine enabled choreography and concludes by rushing through the industrial hall in a massive snake-like form, only to fall onto the transfixed (and sometimes escaping) audience members.

The radical artistic choice to edit, reduce or eliminate painstakingly produced artistic material is not particularly unusual but rather a standard artistic survival strategy. In the heat of practice, there is a clear sense of what works and a clear sense of what doesn’t. In Animate in particular, however, the massive restructuring of the performance 24 hours before the premiere not only enabled the performance to achieve the kinds of vitality affects necessary for its success: it also clarified added certain conceptional and dramaturgical structures that had been buried in the initial concept but not well articulated, namely, the historical story of how different forms of media have and will enable in the future different modes of storytelling. As Animate moves from live theatre to VR (itself an older technology) to the futuristic promises and possibilities of worn AR through the almost anachronistic quality of black and white video passthrough, the audience experiences a double move from one form of historical media to another and from passive spectator to active participant.

Conclusion

We have described the production process involved in a new attempt to integrate unproven XR technologies into the complex dynamics of real time performance, framed by recent work in STS, the sociology of innovation and performance theory on the social-technical imaginaries and futures that technology harnesses and the real-world challenges and practices that put these futures into a kind of situated, concrete knowledge and experience.

What we have not have time to describe in this paper is the audience reaction to the project as little ethnographic interview work was formally done given the already complex demands on the artistic and technical team. But perhaps a long citation from a review of the production in one of Germany’s major newspapers, the Frankfurter Allgemeine Zeitung, gives one sense of the affective promise envisioned in the complex entanglement between technology and environment, past and future, imagination and reality. “This year, the Kunstfest Weimar is focusing on the utopias of the future and has included pieces that talk about climate change, political polarization and new social systems. Animate" is the boldest one. In this immersive production, the latest generation of virtual reality glasses is used to tell a dystopian love story that blossoms and then shatters in the midst of the climate crisis. The text comes from Canadian author Kate Story. Laurie and Daniel, the main protagonists, are newly divorced and come together in their loneliness. Both are traumatized by past relationships and experiences. Their homeland has been destroyed by a climatic catastrophe. While the sound plays, the audience of only eight is led through the hall on a rope in a one-hour time slot. Actors ecstatically tell the story and perform in harmony with digital reality. While she balances on the stones and always close to the abyss, he falls into a depressive trance and despairs more and more. "I'm finished, we're finished," he says in a sobbing voice. The industrial hall with all its natural obstacles provides a surreal backdrop. In the VR part of the performance, the images blur, the chirping of birds turns into a booming bass sound, and boulders seem to be moving around. At that moment, the performance begins to fulfill its immersive promise. Now the rope is put down. The audience moves freely in the space, following and dodging the whirling rocks – which are both there in the glasses and not there in the real world. With their hands they try to fend off the debris, which really works thanks to the latest VR technology. Salter’s piece, realized with technology from Meta, gives a foretaste of theater ten years from now.”

References

1 Antenna Theater, https://www.antenna-theater.org/. Accessed on November 25, 2022.

2 Marie-Luise Angerer, Ecology of Affect: Intensive Milieus and Contingent Encounters. Luneburg: meson press, 2016, 16.

3 Azuma, Ronald T. “A Survey of Augmented Reality.” Presence: Teleoperators and Virtual Environments 6, no. 4, August 1, 1997, 355–85.

4 — (2016) “The Most Important Challenge Facing Augmented Reality.” Presence 25, no. 3, December 2016, 234–38.

5 Genevieve Bell, Paul Dourish, “Yesterday’s Tomorrows: Notes on Ubiquitous Computing’s Dominant Visions”, Personal and Ubiquitous Computing, 11. Cham: Springer, 2006, 133-143.

6 Frank Biocca et al., “Toward a More Robust Theory and Measure of Social Presence: Review and Suggested Criteria”, Presence: Teleoperators and Virtual Environments 12, no. 5, October 1, 2003, 456–80.

7 Gordon Calleja, In-Game: From Immersion to Incorporation, Cambridge, MA, MIT Press 2011.

8 Michael Century, Northern Sparks: Innovation, Technology Policy, and the Arts in Canada from Expo 67 to the Internet Age, Cambridge, MA, MIT Press, 2022.

9 Paul Dourish, Where the Action Is: The Foundations of Embodied Interaction, Cambridge, MA, MIT Press, 2001.

10 Michel Foucault, The Archaeology of Knowledge, New York, Routledge, 1972.

11 Thomas Gieryn, "Boundary-work and the demarcation of science from non-science: Strains and interests in professional ideologies of scientists", American Sociological Review 48 (6), 1983, 781–795.

12 Erving Goffman, The Presentation of Self in Everyday Life, New York, Knopf Doubleday Publishing Group, 1959.

13 Simon Greenwold, "Spatial computing," Massachusetts Institute of Technology, Unpublished Masters Thesis, 2003.

14 Karen Konrad et al, "Performing and Governing the Future in Science and Technology", In The Handbook of Science and Technology Studies, 4th Edition, eds. Ulrike Felt, et al. (Eds.) Cambridge, MA, MIT Press, 2017, 465.

15 Milgram, Paul and Fumio Kishino, "A taxonomy of mixed reality visual displays", IEICE TRANSACTIONS on Information and Systems 77, no. 12, 1994, 1321-1329.

16 Mark Roman Miller, et al., "Social interaction in augmented reality", PloS one 14, no. 5, 2019.

17 Ken Perlin, Future Reality Lab, https://frl.nyu.edu/author/ken/, Accessed on December 4, 2022.

18 Michael Saker, J. Frith, “Coextensive space: virtual reality and the developing relationship between the body, the digital and physical space", Media, Culture & Society 42, no. 7-8, 2020, 1427-1442.

19 Mel. Slater, “Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments", Philosophical Transactions of the Royal Society B: Biological Sciences 364, no. 1535, 2009, 3549-3557.

20 Daniel N. Stern, Forms of vitality: Exploring dynamic experience in psychology, the arts, psychotherapy, and development, Oxford, Oxford University Press, 2010, p.9.

21 Ivan E. Sutherland, “The Ultimate Display”, Proceedings of IFIP Congress, 1965, 506-508.

22 — “A head-mounted three-dimensional display”, Proceedings of the December 9-11, 1968 Fall Joint Computer Conference, Part I. ACM, 1968, 757-764.

23 Mark D. Weiser, "Ubiquitous computing", In ACM Conference on Computer Science, vol. 418, no. 10.1145, 1994 ,197530-197680.

24 Bob G. Witmer, and Michael. J. Singer, "Measuring presence in virtual environments: A presence questionnaire", Presence 7, no. 3, 1998, 225-240.

25 Zhao Shanyang, “Toward a Taxonomy of Copresence”, Presence 12, no. 5, October 2003, 445–55. 

Towards Sensemaking in the Meshwork of Technology, Ecology and Society: Symbiosis of Aesthetics, Performance and Digitalization

Claudia Schnugg, Daniela Brill, Christian Stary

Johannes Kepler University, Linz, Austria Claudia.Schnugg@jku.at, Daniela.Brill@jku.at, Christian.Stary@jku.at

Abstract

To act, humans first need to make sense of the world. Thereby, sensemaking goes beyond accumulation of pure information of objects or rational knowledge production, but it encompasses additional information such as meaning, mindful engagement, socially embedded knowledge, cultural and work contexts. To navigate in diverse environments, sensemaking becomes central to social settings, also to engage with technologies and understand dynamics in ecological environments. In a complex world where technologies are added components of everyday life and are envisioned as partial means to approach global challenges, social, technological, and ecological environments become intertwined. This meshwork of environments also means to bring together different kinds of knowledge as a base of sensemaking through experience. In the Digital Sensemaking project we specifically look at digitization processes, the interaction with IoT Elements and Digital Twins through the lens of performance art to elaborate on the non- cognitive core constituents of sensemaking processes: embodiment, action-sense nexus, and temporality. We show that aesthetics can be found as an important dimension to bridge the cognitive and non-cognitive process and explore the role of art in this kind of research. It facilitates process and technological development in organizations entangling the social, technological, and ecological.

Keywords

Sensemaking, Digital Sensemaking, Embodiment, Digitalization, Aesthetics, IoT Elements, Digital Twins, Performance, Embodied Experience, Symbiotic Organizations.

DOI

10.69564/ISEA2023-84-full-Schnugg-et-al-Meshwork-of-Technology

Introduction

“Now this makes sense” is an expression that is regularly used by many people. What does it mean though, and why is collection of information alone often not enough for something to make sense? The more complex the environment becomes, the more difficult it is to make sense of situations. For example, concerning the climate crisis many artworks aim to establish a connection for the audience between scientific insights on a global scale, or

global climate dynamics, with specific manifestations of the same issue on a more local or individual scale. The same is true for other artistic projects that deal with global challenges, such as helping individuals living in Western countries with a mindset informed by the needs of their lives in Western cities to better understand the impact of their behavior. ¹  More specifically, to create awareness how behavior and decisions of citizens in urban environments can affect rural environments in other countries, or how mindsets framed by social and cultural ideas in one country can lead to exploitation of ecological systems that the very same population has an interest in preserving. In these endeavors, artists engage in the creation of experiences to help the audience make sense of abstract information.

On the other hand, environments of social interaction, cultural production, and work processes become increasingly enriched with technology, connected to digital systems, or even completely digitized. Engaging with digital processes, socio-technical or socio-hybrid systems can lead to a new version of the disconnect presented above: how much are workers in touch with what they are producing? For example, rhythms of a production factory, materialities of components, noises of the machine, and embodied knowledge such as feeling the right pressure might be important for production workers to understand if everything is going well in the process.

When they are suddenly confronted with a digitized version of their work, depicted in software on a computer and with robots taking over much of their physical work, how do they still make sense of the process that is going on? ² Which clues do they actually need? Or how can individuals purely connected through technological systems still make sense of what they do together?

In the era of global challenges and digital transformation of central processes in work, government, and society, organizations and individuals are challenged to make sense in such complex situations that are constituted by merging information, abstractions, and experiences (or the lack thereof). Starting from sensemaking theory based in social sciences which has been discussed within organization science for about 40 years now with a strong focus on social context and cognition, we aim to explore sensemaking in a potential symbiosis of social, technological, and ecological environments. Scholars already pointed out how important it is to include embodiment, sensemaking with non-human and more-than-human actors or environments, and to go beyond a focus on logos or exploration of a mind-body dualism in understanding sensemaking; to develop a decentered notion of sensemaking “—not simply at the disposal of human subjects—and where sense is always and already given and made simultaneously”, but as a process beyond logocentrism that unfolds “in the meshwork of life”. ³ This means to include a focus on immanent and embodied aspects, including temporality, leaving anthropocentrism by including more specifically actors and aspects that are part of the sensemaking process such as materialities, technological elements, or ecology. According to Karl E. Weick, sensemaking is about patterns that enrich and develop organizational structures, about the attribution of meaning to processes and situations.

In the Digital Sensemaking project that we present in this paper, we answer this call of scholars in the field of sensemaking theory through experimentation in diverse settings that connect to non-human and more-than-human components in a controlled way. Therefore, creative new methodological approaches are needed. ⁴ Our analysis of this experimentation is informed by methods from qualitative research in social sciences, visual studies, and methods in performance studies. We are especially interested in deepening the understanding of embodied sensemaking and the role of embodiment in sensemaking, aesthetic experience, and the role of aesthetics in sensemaking. Therefore, we work with performance artists in the setting of artistic research and performance art.

The research is based in the academic Institute for Business Informatics—Communications Engineering that focuses on digitizing work processes, cyber-physical systems, and digital twins. Thus, central to the artistic research and artistic performance development of the artists involved is the engagement with these systems in the context of sense-making processes. With this interdisciplinary project we connect arts-based work and the creation of aesthetic experience through artistic practice (performance) into exploration of technology with contributions to theory building within the social sciences and contributions to technology development.

The paper is organized as follows. We will start by introducing sensemaking theory and the theoretical context of why and what we aim to explore with the Digital Sensemaking project. We then account for our methodology to work with performance artists to investigate embodiment and aesthetics in sensemaking with digital technologies, cyber-physical systems, and digital twins. We will present our insights on embodiment and aesthetics in sensemaking in the above-mentioned “meshwork of life” and explore how including art in the research process can open avenues for research and symbiotic organizations.

Sensemaking

In everyday experience, sensemaking is referred to as a process in which experiences, information and situations suddenly fit together parts fall into place, relations between bits of information in a certain environment, or as sense-making scholars write “[sensemaking] is the primary site where meanings materialize that inform and constrain identity and action”. ⁵ Since the 1980s the theory of sensemaking has been explored extensively in the field of organization studies. Organization studies is interested in understanding human behavior of individuals and groups, organizing processes, work, hierarchies, but also in questions of shared meaning and communication in groups. With its entanglement in sociology, psychology, anthropology, ethnography, work, and management studies, it has been a fruitful field to expand the original concept of sensemaking. Sensemaking is considered as an ongoing process of clarification, as enactment of sensible environments, as social in terms of relation to the knowledge and sensemaking of others as well as a reflection on one's own identity. Moreover, it originally has been characterized as informed by cues that the individual becomes aware of, an urge to understand what is done as plausible, and informed by retrospective reflection on situations.⁶

Since the inception of sensemaking theory, scholars started to expand on missing aspects of the original definition. Most importantly, the strong focus on cognition, logocentrism, or even rationalistic approach to sensemaking has been criticized widely, similar to different ways of understanding temporality and the role of the human in sensemaking, and its lack of situated and embodied dimension in order to give way for research covering these aspects and more differentiated approaches in sensemaking moving away from a pure anthropocentric and language orientation – as previously language and cognition have been first entry points to research. ⁷, ⁸ , ⁹, ¹⁰, ¹¹ For example, ethnographic methods and video analysis have been employed to investigate the various role of the body and embodiment in sensemaking processes, such as sense-making “with the body” and “from the body”; in other words: embodiment in the process of sensemaking, and the role of established embodiment for sensemaking. ¹² 

Other endeavors expanded on the influence of the material world on the sensemaking process. Going beyond the communicative support of boundary objects, the role of material practices for individuals to transit to group-level sensemaking. ¹³  Taking this further, researchers in human-computer-interaction started to investigate technologies to support sensemaking processes. ¹⁴ A first step to connect the idea of aesthetics and sensory knowledge to sensemaking has been identified in cases through linking embodied and implicit knowledge in the handling of objects in work and knowledge production processes.

In a first step, individuals make sense of what they are working with, for example data in the field of physics which is their specific field of education and work. At a later point in research, their implicit knowledge about the look and feel of the data and their ingrained knowledge of physics creates difficulties in finding fresh perspectives on the data. Aesthetic re-interpretation of the objects can help them through this process. ¹⁵ Thus, objects become part of sensemaking, and technological environments to which individuals relate to daily or which mediate their work, are part of their sensemaking processes. This entanglement with the material world in sensemaking processes and as important sensemaking dimensions for humans becomes visible beyond the engagement with human-made objects or the use of objects and technologies to mediate sense-making between individuals—or may restrict it in the case of ubiquitous use of technology.

Based on the experience with the non-human and the more-than-human in the environment of remote material landscapes and the sensemaking of ecological processes by indigenous peoples, ethnographic methods have been employed to investigate ecological embeddedness of sense-making and the increasing inability to make sense of subtle ecological cues of Westerners who predominantly live in urban or industrial areas. This lack of extending sensemaking to the environment in that sense leads to vulnerability, for example by misunderstanding dangers in wildfire situations or regions prone to high waters or other harsh conditions that need to be recognized to enable individuals to predict danger. ¹⁶ This still increasing detachment of macroscopic aspects in the environment that play into sensemaking processes is supported by the simultaneously increasing focus on sensemaking on microlevels, in personal “bubbles”, specific organizational settings, disciplinary work, or even cultural perspectives.

Although the meshwork of life, i.e., social, technological, and ecological dimensions of the environment are becoming more central in the understanding of sensemaking theory, there are still gaps to research, e.g., to raise awareness of problems in the focus of sensemaking process on specific microlevels, to better understand a new dimension that is introduced through certain technologies: digitalization, and to entangle aesthetics and art with interdisciplinary sensemaking research. It is supposed to connect cues from the environment, experience, sensory information, embodiment, and diversity of media relevant to sensemaking.

Towards Digital Sensemaking

Experiences in the Covid-19 pandemic accelerated the digital transformation as envisioned by protagonists of Industry 4.0, that includes the implementation of Internet of Things (IoT), robotization, cyber-physical-systems, but also propagated ideas within the transhumanist movement. At the same time, digital transformation can also lead to work overflow as tasks arrive much faster at the responsible human actors, and to information overflow as much more details and relevant factors need to be described because they cannot anymore be experienced directly – but at the same time become more abstract –, and meaningful experiences and decision-making processes become more difficult. ¹⁷, ¹⁸ On some scale many people experienced such effects during the Covid-19 pandemic: being decoupled from on-site team meetings, communication might have become difficult over time, or understanding the actual situation in a production hall through digital twinning of information might have been difficult and can have led to delayed decisions, or even to oversee important moments to avoid problems.

Digital transformation is often associated with changes in mental representations of knowledge of those who engage in the digital transformation, in order to be able to adjust the behavior to new situations. ¹⁹ This idea is strongly rooted in social information studies and mirrored in processes of communications engineering and computer science. One reason can be that digital transformation is understood as being based in transforming information into digital representations and abstractions that seemingly follow rational logics. Thereby, logocentrism, anthropocentrism, and a focus on disembodied abstractions that are represented through words, numbers, or most importantly also visualizations surface in this field and such settings. But considering the often experienced disconnect between digitized work process and internalized hands-on work in production companies, the resulting information overflow, and difficulties in decision-making, it is important to ask the question how to create meaningful experience in digital transformation and sensemaking in the interaction with digital technologies. ²⁰, ²¹

Weaving digital transformation (and its predominant focus on cognition and abstraction) with the concept of sensemaking and the most recent scholarly contributions with an increasing focus on the body, embodiment, and physical interaction with the environment, the question we explore with the concept of digital sensemaking is where the body and the senses are in the case of digital transformation? Thereby, the concept of digital sensemaking aims to integrate the body and the mind, cognition, and experience, as necessary for sensemaking with digital technologies, in cyber-physical systems, and thus create opportunities to design meaningful interaction.

This integrative perspective adds an important aspect to human sensemaking when connecting beyond technology or the immediate micro-environment. Or the other way ‘round: enhancing sensemaking with digital technologies which are embedded in the meshwork of life, for resilient organizations and human beings in scenarios where technology, ecology, and the social need to be integrated for sensemaking.

The Role of the Body and Aesthetics

Ideas of what sensemaking is are not necessarily bound to knowledge about social scientific theories on sensemaking. Thus, we started this project by a preliminary investigation of what individuals who are experienced in working in both, in academia as well as in the arts, do understand as sensemaking. The preliminary investigation consisted of four qualitative interviews with artists who work in academia and focus on sensemaking in their work without focusing on the theory. The answers we received to the question of “what is sensemaking for you?” are striking. They represent beautifully how body, mind and experiencing the world are entangled in sensemaking. In the first interview, an artist with a bodily artistic practice who is also embedded in natural science research, says:

“It’s a mess, making sense of the world, different biases, we spend so much time trying to make sense of the world. I don’t try to make sense of it, through dance I just experience and feel, don’t think just do. I like to learn more about the complexities, but the deeper I go, the less it makes sense. Dancing helps to see different things in different ways. Everyone has both sides of the brain in their head, both sides are needed for everything. Being analytical in the dance is really important. [...] I’m talking about science and art separately. In dance you see a movement and intuitively move. In physics – breakthroughs – you learn it so much that it becomes intuitive.”

The third artist we interviewed referred to this entanglement in a similar, but different way:

Sensemaking is “when your ideals and your perception of reality are the same. When something makes sense is when you can truly believe it, now you understand it and it’s real and it’s the truth. You can investigate to know what is the truth, making sense of reality is to incorporate your senses to prove something as true. That’s when it makes sense. When things are confirmed, they become understood, and then you come to your senses and perceive reality.”

Both excerpts from the interviews show how reality, perception of reality, being in the world, understanding through body, and theoretical reflection and cognitive understanding need to come together to make sense. The first artist also refers to an additional dimension, through the reflection on “biases” the dimension of the social, of values, cultures are pointed out. On the one hand, there is cognition, abstracted knowledge about the world and how it functions (or should function). On the other hand, there is the experience, the way the world is perceived, situations that impose themselves on us. Thus, the body and the bodily senses play a central role - abstract knowledge alone does not yet lead to sensemaking. Sensemaking is informed through the situation, the experience, and subtle information of the social and cultural context. Put it in a different way, experience and “being in the world” are key to sensemaking. There is no sensemaking without all these rich additional layers of information.

Sensemaking scholars have worked in many directions to make sensemaking theory consistent, mainly by working with cases to fill in gaps in the original theory. Lately, two theorists, Sandberg and Tsoukas, used a phenomenological approach to theoretical work and case studies on sense-making when presenting a typology of sensemaking that is ordered by as they call “four core constituents” of sense-making: sense-action nexus (or “being-in-the-world”), temporality, embodiment, and language. ²² Most knowledge in sensemaking theory revolves around language as it is strongly connected to articulation of the lived experience in order to process collectively and to connect to cognition. Sense-action nexus, temporality, and embodiment are more fluidly overlapping concepts and less clearly developed in sensemaking theory. The body plays a certain role in sensemaking - it is not only the case through embodying knowledge, but also through the experience over time and of “being-in-the-world”. To experience the world, the body needs the senses, what entangles being-in- the-world with the body through the senses. Experience through the senses is connected to aesthetics as understood by Alexander Gottlieb Baumgarten. Connecting to the senses, sensory perception, sensory and sensible knowledge is also referring to building knowledge with and through the body to refine skills which implies making sense of situations or tasks. ²³, ²⁴ As an underlying theme, this is also expressed by the artists as quoted above.

Thus, while researching the role of the body for sensemaking in digital transformation and with digital technologies, we also see an opportunity to push the boundaries of knowledge within sensemaking theory. We suggest that based on the entanglement as presented above, it is necessary to unravel dimensions of aesthetics and the entanglement of aesthetics in sensemaking.

Some experimental approaches already pointed to the role of bodily interaction and aesthetics within human-robot-interaction, emphasizing that reflecting on the body and aesthetics in such interactions with technologies needs to go beyond design (UX or UI design) in order to relate to the robot, to become aware of situated dynamics, and to adhere social meaning. ²⁵ Enabling social meaning making connects strongly to sensemaking as materialization of meaning in the context of identities and action (see introduction to sensemaking).

The Digital Sensemaking Project

The Digital Sensemaking project (DIGI-Sense) is a two-years’ project to explore the rich space at the intersection of the social, the technological, and the environment. Specifically, we focus on sensemaking in the realm of digital transformation, more precisely in cyber-physical systems, with digital twins, and through IoT elements enriched environments. ²⁶ These IoT elements are represented by a variety of technologies, such as sensors and robots.

What lessons can be learned from digital sensemaking on the body in sensemaking? Can we contribute to a better understanding of the body and its sensing capabilities – aesthetics – in the sensemaking process?

In the DIGI-Sense project we work with performance artists to explore digital sensemaking. In three steps – along three phases – the project team works with performance artists to explore several aspects of sensemaking through tailor-made performative research, performative installations, and performances. The performance artists work with the technological infrastructure in their performative research, development, and staging. Thereby, they can draw from the technological infrastructure (i.e., IoT elements, sensors, robots, 3D-sanning and point-cloud generation systems, digital twins).

The three phases of experimentation with the performance artists are structured as follows: in the first phase, two performers without affinity to programming and technology development nor pre-knowledge about IoT elements, sensors, and robots, engage with the provided systems. Their learning and performative research process into the technologies provided with a specific focus on embodiment and the senses is in the center of this first phase. This phase serves as an ‘approaching’ phase for non-familiar humans to IoT technologies and robots. Its major outcome is a set of moments that could trigger sensemaking processes.

The second phase focuses more specifically on the trilogy of sensemaking, sensebreaking, and sensegiving, again with a focus on the body, the senses, and more explicitly on aesthetics. This phase includes higher levels of digital abstraction such as digital twins. The artist will develop a performative installation that invites performers and test persons to explore and make sense of the digital twin of their body and movements.

The third phase aims at contrasting the idea of embodied articulation to the cognitive and language-based idea of supporting change through poetic language. ²⁷ In this way, the project spans from initially approaching cyber-physical elements to digital representations that finally can be encoded in human language, however, guided by performance art. The remainder of this paper will focus on the first phase in the DIGI-Sense project.

Methodology including Performance Art

The body and embodiment have already attracted the interest of scholars in sensemaking. They investigated the interaction with materials and objects, but also learning processes of people rowing the Amazon. (e.g., ²⁸, ²⁹) The body thereby has been described as essential in sensemaking in several levels: through the interaction with other bodies, non-human and more-than-human bodies, sensemaking processes and specific moments are enriched, supported, and informed; but also the body’s capability of learning, suffering, implicit or tacit knowledge, a more holistic approach to the sensemaker’s body to be embedded, sentient, and situated; as well as the connection of the body to the identity and very personal processes of the sensemaker have been pointed out.

Nevertheless, it is difficult for social scientists to go deep into the question of embodiment as the bodily experience is something very personal, something difficult to abstract to rational arguments, and the methodologies used to investigate the body and bodily experience are only marginally part of the repertoire of research methods in social sciences. Connecting to art, especially performance art, is one way that has been explored by some organization scholars previously. ³⁰, ³¹

To focus on embodiment and the senses in our research we work with performers. Performers have a heightened bodily sensibility as they are educated to work with the body and reflect on their embodied processes. Through working with them we aim at harvesting from this capacity for deeper understanding of the role of the body in sensemaking. Moreover, the four core constituents of sensemaking are aligned with experimentation with performance artists: sense-action nexus (i.e., do, be in the space, explore and experience with the body and senses), temporality (i.e., process, unfold over time, ways of experiencing time and being in the situation), embodiment (i.e., work with the body, capacity to “read” the body, bodily senses, process of embodiment), language (i.e., articulation, conversation between performers, of performers, and with the researcher). ³² 

Additionally, we employ methods from qualitative research in social sciences such as qualitative interviews, participatory observations, observation diaries, diaries by the performers, video footage throughout their complete process of interaction with the technologies for the DIGI-Sense project, and photo documentation. For the visual material, we also employ strategies from visual research and performance research. ³³, ³⁴ , ³⁵  Connecting bodily experience and in-depth interviews has also previously been used to examine micro-processes in meaning-making in experience, and thus is a relevant juxtaposition of the body and the senses in sensemaking with cognitive reflection. ³⁶

First Performance Phase

Between March and May 2022, we worked with two performance artists (one male, one female) to explore digital technologies with their artistic practice. Both artists do not have any background in working with digital technologies and did not have previous experience with integrating digital technologies in their performative practice. One of the performance artists has an additional background in human medicine, the other performance artist has previously researched interspecies communication and animal behavior in their artistic practice.

The idea was to engage in a learning process with several chosen digital technologies provided by the Institute of Business Informatics – Communications Engineering. This engaging learning and investigative process would focus on the bodily experience, embodiment, and the senses – and thus aesthetics. The process would result in “micro-performances” that express important moments of the interaction between the body and the digital technologies for sensemaking.

The artists worked individually, next to each other, and together, depending on the stage of their process and their sensemaking of the digital technologies. The places they worked at throughout the process were: at their homes and private environments, at a performance space in Vienna throughout a week-long residency, and at the premises of the University of Linz (especially when they worked with the infrastructure that was bound to stay at the premises of the university). The researchers provided them with a selection of digital technologies to work with, which they could choose from for their performative research.

The technologies they chose were a selection of M5Stack sensors as IoT components to be navigated digitally via coding language Blockly (Figure 1), and the agile robot system Boston Dynamics Spot (Figure 2). Additionally, they chose to explore the 3D-scanning and point-cloud generation system Trimble x7. The focus in their process was on the IoT elements and the agile robot system, and an additional exploration of the 3D-scanning and point-cloud generation system in one afternoon.

The two performers recorded their process with the digital technologies through filming themselves or each other in the interaction, moreover, they were asked to take notes in a diary to trace their process and took photos of their outcomes and certain moments. During selected phases of the performers’ process, one of the researchers was present for participatory observation. The researcher also took videos, photos, and notes in an observation diary. Additional material about the digital technologies that was used by the performers to learn about the digital technologies has also been tracked and fed into the evaluation. 

Figure 1. One of the performers exploring the IoT elements M5Stack while starting to code them in the programming language Blockly. ©Daniela Brill Estrada.
Figure 2. Performance with the agile robot system in the park of the university. ©Daniela Brill Estrada.

The performative research and exploration of the digital technologies became an ongoing process in which instead of the development of staged “micro-performances” it became more important to define key moments in this ongoing performative exploration which acted as ahamoments, oh-no moments, or moments of “suddenly everything fell into place” and “now all the movements become fluently, and parts fall into their places”. Finding and analyzing these moments and the process that leads up to these moments became key to the evaluation process of the visual material. On top, the performers developed one performance with the agile robot system to be staged at the “Lange Nacht der Forschung” (researchers’ night) at the university campus. Central steps in the performers’ sense-making with the agile robot system could afterwards be traced in the development process for this performance. All in all, we generated 119 videos to use as data, ranging between 20 seconds and over 47 minutes in length.

Additionally, one of the researchers interviewed the performance artists at several moments throughout the process. This process started with a preliminary interview before planning the research, then we did four focused interviews with each of them. The first main interview took place before the actual performative research started, then two in the middle of the ongoing project, and one at the end of the process. All interviews have been transcribed. After the evaluation of the material, one more conversation with the performers took place to clarify specific questions and get feedback on the interpretation of the data.

Coding for Data Analysis and Interpretation

The data analysis for this first phase in the DIGI-Sense project has been done coherently after the performers submitted all their materials and data generated throughout their process. For coding and evaluation of the data, the software MAXQDA has been employed, a software to support the analysis of qualitative research data, including qualitative interview transcripts and video material. The design of the first performance phase was based on the idea to start by the exploration of digital technologies through the performance artists. We traced both their individual and group-level practices of sensemaking of the two performers, keeping in mind conversations, forms of engagement, patterns of (social) interaction, and material artifacts. Through this we aim to surface and can report on:

To do so, we decided to code the collected data on the one hand for moments and information that refer to specific aspects of the sensemaking process and connect to the body, the performers’ movements, or their open reflection on their bodily experience. ³⁷, ³⁸, ³⁹ On the other hand, to surface an understanding of aesthetics in the sensemaking process, we also coded for aesthetic moments. These aesthetic moments are also partially connected to sense-making literature, as sensemaking scholars and sensemakers presented in sensemaking research refer to the aesthetic dimension without referring to “aesthetics” as such, and to their senses as important aspects in their sensemaking. Examples can be found plenty in sensemaking literature, and in meta studies on sensemaking. For example, expressions such as the following are used: movements get interrupted, noise creates awareness that something is wrong vs. “normal engine noises”. [e.g., ⁴⁰]

For the moments in the sensemaking process that connect to the body in the video material that is supported by the additional data, we decided to search for the following codes (number of coded moments in brackets, a total of 514 moments): “the artists are solving a problem together” (33), “wondering about something together (while working together)” (32), “proposal of a solution” (59), “direct body interaction with the element” (35), “thinking out loud / speculations about function or meaning” (40), “something goes wrong / encountering a question or problem” (71), “aha-moments” (22), “retrospective explanation of past situation step by step” (30), “communication through body, sound, movements” (37), “directly speaking to or interacting with camera” (6), “aesthetic elements of the sensors in the process of the performers” (43), “metaphors” (9), and “connection between body and machine / also aesthetic” (8). The codes needed to be adjusted for the specific inter-action modalities and functions of the agile robot system. Especially codes on the direct “human-machine interaction” (24) and “feedback from the machine” (15) needed to be added, others were slightly renamed to fit the situation with the agile robot system. Overall, the codes captured learning or knowledge acquisition moments, and modalities of behaviors, both categories addressing the performative practice of sensemaking.

For investigating the role of aesthetics, we referred to sensemaking literature and literature in organizational aesthetics to define the codes. [41] We found 51 moments that represent these. The aesthetic codes are: “rhythms”, “repetitions”, “movements”, “sounds”, “comparisons”, “aesthetic analogies and metaphors”, “velocity”, “haptic properties”, and aspects like “color, texture, form, weight”.

Discussion and Conclusion

First interpretation of the data gives insights into the rele- vance of aesthetics in the sensemaking process to build relations between the sensemakers and their environment and context within which they make sense. The aesthetic information that they receive through their senses informs them about basic interaction modes with them digital technologies – and thus helps them to engage in a learning process to embody the interaction with and the use of these technologies. At the beginning, handling the IoT elements or interaction with the agile robot system is clumsier than when being used to handle them, and how the materialities feel, can be touched, or handled best, need to be learned. At later stages in the process, aesthetic feedback from handling the technologies helps to react faster and to handle them in a differentiated way. For example, over time, rhythms develop, and movements become faster, and aesthetic feedback from the technologies (e.g., sounds) can be understood more quickly. In the interaction with the robot, the bodily interpretation of the movements of the robot changes from superficial interpretation of the robot’s body to a more fluent interaction of movements.

The senses and bodily learning processes play an important role in the internalization of the knowledge and to make sense of new situations in the interaction. Thereby, the researchers interpret that the senses and aesthetics do not only play a role in the core constituent of sensemaking “embodiment”, but also as a bridge to the main starting point of sensemaking, the “action-sense nexus”. Being-in-the world also means being in the world with the senses and aesthetically experience to learn and to make sense of new situations.

In situations of rapid digital transformation individuals often have difficulties to adjust from embodied work to digitized systems. Becoming aware of problematic situations gets more difficult. Moving from an entangled situation that includes cognition, the senses, and bodily experiences to an abstract situation in which mainly cognition is reflected can lead to disorientation. Like ecological sense-making, sensemaking in a digital environment needs a learning phase that includes the body and the senses in order to develop fine-grained competencies. This also has implications for designing digital transformation: without personal experience of the processes, creating an abstrac- tion for digitized systems will more likely result in systems that are difficult to grasp for users.

Reflecting on the relevance for art and the cases shown in the beginning: artists are experts in dealing with aesthet- ics, creating aesthetic experiences, and translating information to various bodily senses. Especially in complex situations, art can support individuals to get a feeling for the problems (e.g., climate crisis, water, and land use) through creating experiences. This will have a stronger impact for knowledge gain than relying on cognition only. The next steps in the data analysis and phases two and three in the DIGI-Sense project, will give us the opportunity to explore these outcomes in more depth. Of particular interest is the performative practice with digital representa- tions in the second stage of the project, and the translation from digitalized processes to new embodied and materialized meaning in the final stage of the project. Especially as these steps entangle the cognitive with aesthetics and embodied performance involving physical and digital components and the actors’ broader environment.

Acknowledgements

The project is supported by the Linz Institute of Technolo- gy (LIT) Seed Funding. Thanks for conversations and contributions to Aisen Caro Chaicin, Scott deLahunta, Marko Markovic, Merritt Moore, Alexandra Murray- Leslie, Antoni Rayzhekov, and Laura Stoll.

References

1 Nima Navab, Desiree Foerster, “Affective Atmospheres | Ambient Feedback Ecology,” paper based on a talk presented at the International Symposium on Electronic Art, Gwangju, South Korea, June 2019, ISEA2019, http://isea2019.isea-international.org/fullPro.asp

2 Min Kyung Lee, Daniel Kusbit, Evan Metsky, Laura Dabbish, “Working with Machines: The Impact of Algorithmic and Data- Driven Management on Human Workers,” paper based on a talk presented at 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI2015, https://dl.acm.org/doi/10.1145/2702123.2702548.

3 Lucas D. Introna, “On the Making of Sense in Sensemaking: Decentred Sensemaking in the Meshwork of Life,” Organization Studies 40(5), 2018, 745-764.

4 Deborah Lupton, Ash Watson, “Towards more-than- human digital data studies: developing research-creation methods,” Qualitative Research 21(4), 2021, 463-480.

5 Jean Helms-Mills, Making Sense of Organizational Change, London, Routledge, 2003.

6 Karl E. Weick, Sensemaking in organizations, Foundations for Organizational Science Vol. 3, Thousand Oaks, Sage, 1995.

7 ibid. 3

8 ibid. 3

9 Jörgen Sandberg, Haridimos Tsoukas, “Making sense of the sensemaking perspective: Its constituents, limitations, and opportunities for further development,” Journal of Organizational Behavior 36(1), 2015, S6-S32.

10 Sally Maitlis, Marlys K. Christianson, “Sensemaking in organizations: Taking stock and moving forward,” Academy of Management Annals 8(1),2014, 57-125.

11 Ann Cunliffe, Christine Coupland, “From hero to villain to hero: Making experience sensible through embodied narrative sensemaking,” Human Relations 65, 2012, 63-88.

12 Joep Cornelissen, Saku Mantere, Eero Vaara, “The contraction of meaning: The combined effect of communication, emotions, and materiality on sensemaking in the Stockwell shooting,” Journal of Management Studies 51, 2014, 699-736.

13 Mark de Rond, Isaac Holeman, Jennifer Howard Grenville, “Sensemaking from the body: An enactive ethnography of rowing the Amazon,” Academy of Management Journal 62(6), 2019, 1961-1988.

14 Ileana Stigliani, Davide Ravasi, “Organizing thoughts and connecting brains: Material practices and the transition from individual to group-level prospective sensemaking,” Academy of Management Journal 55(5), 2012, 1232-1259.

15 Dorothé Smit, Bart Hengeveld, Martin Murer, Manfred Tscheligi, “Hybrid Design Tools for Participatory, Embodied Sensemaking: An Applied Framework,” paper based on a talk presented at 16th International Conference on Tangible, Embedded, and Embodied Interaction, TEI’22, https://dl.acm.org/doi/10.1145/3490149.3501332.

16 Claudia Schnugg, Creating ArtScience Collaboration: Bringing Value to Organizations, Cham, Springer, 2019.

17 Gail Whiteman, William H. Cooper, “Ecological Sensemaking,” Academy of Management Journal 54(5), 2011, 889-911.

18 Winfried Hacker, “Arbeitsgestaltung als Informations management,” Zeitschrift für Arbeitswissenschaft 74, 2020, 306-312.

19 Vivek Kant, “Cyber-physical systems as sociotechnical systems: a view towards human-technology interaction,” Cyber-Physical Systems 2(1-4), 2016, 75-109.

20 Thomas D. Wilson, “The cognitive approach to information- seeking behavior and information use,” Social Science Information Studies 4(2), 1994, 197-204.

21 ibid. 16

22 Elisa D. Mekler, Kasper Hornbaek, “A Framework for the Experience of Meaning in Human-Computer Interaction”, paper based on a talk presented at 37th Annual ACM Conference on Human Factors in Computing Systems, CHI 2019, http://www.kasperhornbaek.dk/papers/CHI2019_Meaning.pdf

23 Jörgen Sandberg, Haridimos Tsoukas, “Sensemaking Reconsidered: Towards a broader understanding through phenomenology,” Organization Theory 1, 2020, 1-34.

24 Alexander Gottlieb Baumgarten, Aesthetica, 1750.

25 Antonio Strati, “Sensible Knowledge and Practice-based Learning,” Management Learning 38(1), 2007, 61-77.

26 Petra Gemeinböck, “The Aesthetics of Encounter: A Rela- tional-Performative Design Approach to Human-Robot Interaction,” Frontiers in Robotics and AI 7, 2021, Article 577900.

27 Daniela Brill, Claudia Schnugg, Christian Stary, “Makes Digital Sensemaking Sense? – A Roadmap for Digital Humanism in Increasingly Transhumanist Settings,” New Explorations 2(3), 2022, 86-106.

28 Karl E. Weick, “Reflections: Change agents as change po-ets,” Journal of Change Management 11(1), 2011, 7-20.

29 ibid. 12

30 ibid. 15

31 Brigitte Biehl, Dance and Organization: Integrating Dance theory and Methods into the Study of Management, New York, Routledge.

32 Emilie Reinhold, Claudia Schnugg, Charles Barthold, “Dancing in the office. A study of resistance,” Scandinavian Journal of Management 34(2), 2019, 162-169.

33 Daniela Brill, Claudia Schnugg, Christian Stary, “Digital Sensemaking: Sensemaking as a Driver of Transformation”, paper based on a talk presented at 17th International Forum on Knowledge Asset Dynamics, IFKAD’22, https://www.ifkad.org/event/ifkad-2022/.

34 ibid. 12

35 Emma Bell, Samantha Warren, Jonathan E. Schroeder, The Routledge Companion to Visual Organization, London, Routledge, 2014.

36 Philip Barnard, Scott deLahunta, “Mapping the audit traces of interdisciplinary collaboration: Bridging and blending between choreography and cognitive science,” Interdisciplinary Science Reviews 42(4), 2017, 359-380.

37 Desiree Foerster, “Re-reading Whitehead Through the Pre- reflective Experience of Atmospheric Processes,” Ambiances, 2021, Article 3745.

38 ibid. 21

39 ibid. 12

40 ibid. 5

41 Antonio Strati, Organization and Aesthetics, London, Sage, 1999.

Deeply Listening Through/Out the Deepscape

Hugo Scurto1, Axel Chemla–Romeu-Santos2

1EUR ArTeC / Université Paris 8 / École des Arts Décoratifs – PSL (EnsadLab) 2STMS Lab Ircam–CNRS–Sorbonne Université Paris, France 
hugo.scurto@ensad.fr, chemla@ircam.fr

Abstract

This paper presents early artistic, conceptual and technical work toward practicing and theorising through/out the deepscape. I first introduce the concept of deepscape, which may designate the global flows of media intensively computed by deep learning throughout the Internet, entangled with the material, human and cultural resources they capitalise on throughout corporate infrastructures of artificial intelligence (AI). I then propose to explore deep listening of soundscapes generated by deep learning as a practice to raise awareness of the planetary scale of the deepscape. I relate the diffractive prototyping of a deep generative model of soundscape, based on the multichannel hacking of the Realtime Audio Variational auto Encoder (RAVE), trained on worldwide soundscape data that I transversally recorded over 28 places in late April 2022, using the Locustream online sound map. I argue that listening to the planetary soundscape that continually flows from this deep generative model may reveal the ethico-ontoepistemology of deep learning, by recalling the landscapes that are being exploited by infrastructures of AI, while situating data collection practices and training costs of deep learning. The paper ends by discussing art and science work that might be engaged to reveal and reconfigure the deepscape in depth.

Keywords

Deep Learning, Deep Listening, Diffractive Art Practice.

DOI

10.69564/ISEA2023-85-full-Scurto-et-al-Deeply-Listening

Introduction

For over half a century, a vast majority of scientists have been developing and describing Artificial Intelligence (AI) through anthropomorphic conceptions and representations. From symbolic AI to modern Artificial General Intelligence ²⁷, simulating human-like intelligence through quantification and computation have been standard goals and methods for most researchers in the field. These anthropomorphic conceptions stem from millenary Western representations of AI as humanoid robots, or artificial objects endowed with human intelligence ³⁷, which are still actively shaping popular representations, along with corporate narratives, on AI. ¹¹

The technological revolution brought by deep learning in the last decade paradoxically destabilised anthropomorphic conceptions of AI ¹², while simultaneously reinforcing anthropocentrism. On the one hand, difficulties in interpreting internal representations of deep neural networks, as well as images they can generate ³², pushed scientists and artists to approach AI as a form of non-human intelligence ³⁴, ⁶. On the other hand, systems powered by deep learning, from facial recognisers to text and image generators ⁴³, ¹⁴, entailed worldwide AI applications, both public and private, across almost all sectors of human society ⁴².

As an artist-researcher, I am interested in developing a non-anthropocentric representation of AI that raises awareness of the planetary costs of deep learning. I am inspired by Kate Crawford and Vladan Joler’s Anatomy of an AI system (2018) ²² , whose large-scale map reveals the globalised, extractive infrastructures that lie beneath modern AI applications, and that are barely tackled by anthropocentric discourses. I am also interested in engaging in diffractive art practice with deep learning ⁴⁸ to perform and reconfigure this non-anthropocentric representation. I am inspired by Memo Akten’s Deep Meditations (2018) ⁶ , whose deep learning-generated continual flows of images and sounds let us embrace the interconnectedness of AI with all socio-material bodies of time and space, including us, and the planet.

In this paper, I introduce the concept of deepscape, along with deep listening experiences of planetary soundscape live-generated by one deep neural network. The deepscape designates the global flows of deep learning-generated media in-tensively computed by deep learning throughout the Internet, entangled with the material, cultural and human resources they capitalise on throughout corporate infrastructures of AI. Deep listening experiences ³⁹ of deep learning-generated continual flow of soundscape aim at raising awareness of the deepscape, by leveraging a planetary representation for a deep neural network, and recalling the landscapes that are being exploited by infrastructures of AI. By including non-human voices and planetary scales within research and practice with deep learning, this research-creation thus seeks to resist the anthropocentric culture of modern AI research, illuminating its media aesthetics and socio-ecological impacts.

The next sections successively describe an early conceptual mapping of the deepscape, the diffractive prototyping of a deep generative model of planetary soundscape, and deep listening experiences led through/out the deepscape. The paper ends by discussing art and science work that might be engaged to reveal and reconfigure the deepscape in depth.  


* The first author uses the pronoun “I” to relate their research-creation and their technical collaboration with the second author.

Mapping the Deepscape

In this section, I sketch an early map of the deepscape, by describing the global media flows, corporate computational infrastructures, and planetary socio-material resources that compose it. I argue that the concept of deepscape better represents the media aesthetics and planetary scales of deep learning than mainstream anthropomorphic conceptions and representations of AI. Furthermore, I suggest that the deepscape may inspire new practices with deep learning, which may complement those with/in AI, as we will see.

Global Media Flows

Since DeepDream almost a decade ago ³⁸, the Internet has been flooded by media generated by deep learning. These media encompass image, sound and text, and more recently, video, 3D movement or biosignals ⁴⁷. Sounds and images generated by deep learning also permeate our physical environments, with voices of AI assistants resonating in smart homes and cities, and mobile-based face filters re/acting over our bodies across the globe ³³. The concept of deep fake has made popular such media across fiction and reality, raising awareness of deep learning’s threats to steal one’s identity ¹³. As AI-based virtual worlds are rapidly evolving, we can expect these media flows to grow in the near future ¹.

Taking inspiration from Arjun Appadurai’s notion of scape ⁸, I propose to introduce the notion of deepscape to point to these global flows of media intensively computed by deep learning and permeating digital and physical worlds. Rather than fake, I argue that these media are real, based on the impact they have on cultures and societies, and the planetary computation they build on to exist, as we will see. Importantly, such media are materially configured to exist under the form of generative flows, due to computational architectures of deep learning, which can either support streamable media generation ¹⁶, or continuous interpolation between images or sounds ⁹. Artists have well explored these material attributes of deep learning in stream-based installations ¹⁰, including dadaBots and their infinite online music live-streams ¹⁹, or Anna Ridler and her moving images ⁴⁴.

While humanoid robots surrounded by blue zeros and ones keep on being summoned to promote modern AI applications ¹¹, I suggest that the singular aesthetics of media flows generated by deep learning are starting to replace anthropomorphic representations of AI at a global scale in our collective imagination. Indeed, ambiguous attributes of images generated by GANs ³² (Generative Adversarial Networks, a specific type of deep neural network), along with the almost infinite interpolative flows from which they emerge, have permeated our representations of machines across art and science, along with our conception of how they may exhibit creativity ²⁹. In this paper, I limit my analysis to witnessing such global flow aesthetics of media generated by deep learning, and how they are transforming our representations of AI.

Corporate Computational Infrastructures

Different types of computational infrastructures actually support the deepscape and its global media flows. The first is the Internet ²⁰, since media generated by deep learning actively flows through social networks ¹, helped by mobile applications. A second type is deep generative models ²⁶, which are compressed representations of datasets, computed by deep learning, enabling stream-like generation of media that more or less resemble the dataset. A third type is super-computers and data centers combined with cloud computing ⁵², since deep learning requires intensive computation and large amounts of data to produce deep generative models.

Over the last decade, such computational infrastructures have been in majority invested, developed and owned by global corporations, including the Big Tech. In addition to owning Instagram and Facebook, Meta recently introduced their AI Research Supercluster, consisting of 16,000 graphical processing units (GPUs) especially designed to develop deep learning applications, parallel to their metaverse ⁴. OpenAI—recently followed by Meta, Google and Microsoft—developed several popular deep generative models, including Dall-E ⁴³, GPT-3 ¹⁴ and Jukebox ²⁴, enabling to generate image, text and music respectively, based on text prompts. Google developed Colaboratory ¹⁸, which leveraged the corporation’s GPU-based cloud computing services to accelerate deep learning applications, in turn enabling the emergence of a global community of practice dedicated to deep generative models ⁴⁰. Importantly, Google also develops and maintains TensorFlow, one of the most widely used library to develop deep generative models and modern AI applications ².

I suggest that corporate computational infrastructures are going to exponentially increase the scale of the deepscape and its global media flows in the next coming years. In addition to the Big Tech, hundreds of worldwide corporations have been investing in deep generative models recently, constituting what have been called a “generative AI landscape” or “creative new world” by investors and analysts ³. It is highly probable that these global corporate investments are to modify distribution of work in the coming years, especially in the creative and cultural industries ⁷. Many artists have made such political aspects of AI central to their practice with deep learning ¹⁷. Holly Herndon and Mathew Dryhurst proposed artist-led platforms addressing legal issues related to ownership of training datasets ⁵⁰, and of deep generative models themselves ³¹. In this paper, I limit my analysis to pointing out to corporate computational infrastructures that support the deepscape, and how they may generate governmental issues at a planetary scale.

Planetary Socio-Material Resources

Last but not least, a wide array of planetary resources enables to fuel the deepscape, exploited by corporate computational infrastructures. A first type is material resources ²², ⁴¹. Rare elements such as lithium are required to build the super-computers enabling deep learning computations, along with personal computers and smartphones required to interact with deep generative models ³⁰. Extracting such material elements devastate landscapes across the globe, while contributing to huge levels of atmospheric and water pollution, which in turn generate political conflicts between local populations, opaque intermediaries and global corporations ²³. In addition, electricity consumption of deep generative models breaks records and should continue increasing every year, despite engineering efforts to reduce computational costs ⁵². This leads to huge carbon emissions that threaten the lives of both humans and non-humans across the globe.

A second type is human resources. Workers are required to extract material resources needed to build computational infrastructures of deep learning, often enduring awful economic and health conditions ²³. Large teams of engineers actively maintain such computational infrastructures ², from deep learning libraries to mobile AI applications, across various hardware and software, as computational technology keeps on rapidly evolving. Computer scientists are required to design and implement deep generative models. While they sometimes build and collect the datasets over which they train their models, they essentially delegate data work to disadvantaged people across the globe ²⁸, thus reinstantiating work inequalities and data colonialism ²¹ through deep learning. Far from objective, deep generative models thus perpetuate values and world views of a few ⁴⁵, reinforcing cultural bias of the global computational infrastructure.

Indeed, a third type of resource that directly relates to this data work is cultural resources. Beyond mere numbers, datasets carry historical and cultural value, as images, sounds or text they represent are always and already entangled within communities that produced it, and labels and classifications they encapsulate reflect certain world views ⁴⁵. When a person or a corporation trains a deep generative model over a given dataset, it exploits its cultural value to generate media that somehow reproduces this value, while distorting it, or even biasing it in unethical ways ⁵¹. Artists such as Stephanie Dinkins have sought to illuminate responsible ways of dealing with datasets and deep generative models, especially toward communities that are underrepresented in deep learning, such as the Black community ²⁵. Cultural accountability may eventually become difficult to tackle, as corporate AI infrastructures have started using datasets produced by deep generative models to train deep generative models ⁴³. In this paper, I limit my analysis to pointing out to socio-material resources that fuel the deepscape, and how they damage socio-ecological landscapes of the planet.

Diffractive Deep Learning Prototyping

In this section, I describe the diffractive prototyping of a deep generative model of planetary soundscape, based on a transversal soundscape dataset that I collected over the globe in one week-end, and the multichannel hacking of the RAVE variational auto-encoder ¹⁵, made in collaboration with the second author. I argue that such a diffractive practice with deep learning reveals and reconfigures the deepscape, by performing the planetary scale of deep learning, while illuminating the material costs that its computations imply.

Transversal Soundscape Dataset

I identified three requirements for my data collection practice to engage with the concept of deepscape. First, data should be collected across worldwide places to transcribe the planetary scale of deep learning. Second, data should be in the form of multichannel soundscapes to convey a sense of spatial landscape through sound. Third, data should combine anthropophony, biophony and geophony ³⁵ to have the model embody a sonic environment rather than one individual form of intelligence, such as a musician or a bird.

I used Locustream to collect this soundscape dataset ⁴⁹. Locustream is an online sound map developed by the Locus Sonus research group that enables to listen to live stereophonic soundscapes across the globe. Practitioners can setup their own microphone in the place of their choice, and livestream the sound it captures online, using either a Pure-Data patch running on a RaspberryPi, or a mobile application for smartphone. The map has been used over sixteen years by artists and scientists to promote novel forms of listening to, and performing with, the planet and its diverse environment. I decided to lead online field recording through the Locustream online sound map. Specifically, I opted to restrain my field recording to one local portion of time—namely, one weekend in late April 2022—, while widening it to all portions of the globe accessible on the map over that period— namely, twenty-eight places across seventeen countries and five continents. I called it transversal, following scientific dataset nomenclatures used in statistics. On the one hand, such a transversal dataset let me situate ecological properties of soundscapes related to planetary seasonal shifts, be they related to human or non-human activities. On the other hand, it enabled me to materialise the planetary scale of deep learning, since all places, cultures and species of the transversal dataset would be eventually normalised and approximated as one planetary soundscape by deep learning.

The resulting dataset consists in nearly 16 hours of stereophonic soundscape (see Figure 1), that I have deeply listened to while recording, with a mean duration of 31 minutes per location (see Figure 2). Such small data approach come within the scope of diffractive art practice ⁴⁸.

Deep Generative Model

Along with the second author, we opted to work with the Realtime Audio Variational autoEncoder (RAVE) ¹⁵. RAVE is a state-of-the-art deep generative model that learns to generate raw audio waveforms using a two-stage training procedure. In the first stage, a variational auto-encoder is trained to learn a high-level latent space of the training dataset. In the second stage, only the decoder is trained with an adversarial generation objective to enforce continuity of the latent space. One advantage of RAVE over other deep generative models are that audio signals are modeled and generated at a 48 kHz sampling frequency, i.e., standard audio quality.

The second author hacked RAVE’s implementation (1) by adding a multichannel audio loader, which enables to load audio data with c channels directly for use in RAVE, including the multichannel factor c into encoder, decoder, and adversarial parameters. From a methodological perspective, this add-on could be described as a form of learnable algorithm, which comes within the frame of diffractive art practice ⁴⁸. The source code for our modified implementation will soon be publicly available. Our deep generative model can be used with the nn~ object for neural audio synthesis (2).

Figure 1: Transversal soundscape dataset. In orange: The 28 places where I led online field recording over one weekend in late April 2022. In grey: all microphones previously opened on Locustream since 2006 49.

Training and Streaming Costs

We trained our deep generative model on our transversal soundscape dataset for three million steps, specifically two and one million steps for first and second stages. Trainings were conducted using a private infrastructure, which has a carbon efficiency of 0.432 kgCO2eq/kWh. A cumulative of 360 hours of computation was performed on GPU hardware of type Titan V (TDP of 250W). Total emissions are estimated to be 38.88 kgCO2eq of which 0 percent were directly offset. These estimations were conducted using the MachineLearning Impact calculator presented in ³⁶. Streaming costs are equivalent to working with a personal computer, as RAVE support streamable audio generation on standard CPUs ¹⁶. We used the same model hyperparameters and architecture as provided in the original RAVE implementation ¹⁵. Specifically, we preprocessed our transversal soundscape dataset by cropping it into one-second audio slices to apply dequantization, random crop and all-pass filters with random coefficients as data augmentation strategy. Training evaluation was led by combining quantitative and qualitative dimensions of sound generation, with the first training stage stopped based on joint quantitative observation of loss minimisation and qualitative listening of generated audio, and the second stage stopped based on qualitative listening criteria.

Deep Listening Experiences

In this section, I relate first deep listening experiences created through/out our deep generative model of planetary sound-scape. I group them within an ongoing artwork called Deep- scape: Transversal. Deep listening is a meditative aesthetic proposed by composer Pauline Oliveros “to inspire both trained and untrained performers to practice the art of listening and responding to environmental conditions in solo and ensemble situations” ³⁹. I suggest that deep listening may be helpful to raise awareness of the deepscape and its interconnectedness with all socio-material bodies of the planet. The following paragraphs describe online radiophonic broadcast, collective acousmatic listening, and open listening and discussion situations, inviting diverse audiences to experience planetary soundscape flows computed by our deep generative model, and describe their spatial, temporal and material attributes. These situational wholes enabled to illuminate somaesthetic behaviours of our deep generative model, which jointly come within the scope of diffractive art practice ⁴⁸.

Online Radiophonic Broadcast

The first experience consists in the online radiophonic broadcasting of stereophonic audio live-streamed by our deep generative model. It is mounted on the Locustream online sound map (3) since October 17th, 2022, while being temporarily stopped at times. On the server side, I used the nn~ Max/MSP object for neural audio synthesis, and the Locus Sonus Locustream Pure Data patch ⁴⁹ to mount the live stream online. On the deep learning side, I experimented with RAVE prior for unconditional audio generation ¹⁶. On the sound map, the stream is located at the material place where the deepscape is computed, that is, from my home as of now. To inspire deep listening, the broadcast is accompanied with a note of intent that include two questions: Who terraforms this deepscape? Whose scapes are getting threatened as AI sucks our attention away from planetary issues?

Collective Acousmatic Listening

The second experience consisted in a collective acousmatic listening session led at Cirque Électrique, Paris, France, on November 5th, 2022, for the opening of an AI-based experimental music concert. It consisted in 15 minutes of planetary soundscape (4), recorded in one shot from our deep generative model, based on real-time reconstruction of stereophonic audio coming from a microphone placed at my home’s window. The latter was picking up a flow of environmental soundscape, which included keyboard sounds, human sounds and discussions, bird vocalisations, car engines, wind howlings and children screaming in their playground.

The session started by an on-stage presentation of the piece, as well as listening guidance standing as deep listening instructions. Specifically, I told the audience that they were going to be put in the dark to listen to 15 minutes of soundscape generated by one deep learning model; that they may approach this experience as sound art rather than music, and that they may let themselves immerse into imaginary worlds while listening. As we set the volume relatively high to produce strong bass, I warned the audience that the experience was going to be quite loud, and therefore, that they should not hesitate to leave if it became difficult.

The audience—approximately one hundred people—was standing up while put in the dark for acousmatic listening. During listening, several people were witnessed closing their eyes and collectively relaxing their necks and bodies while standing up. At the end of the session, some audience members reported how the listening experience felt shorter than 15 minutes, which somehow testifies of their immersion in deep listening. They also felt a sense of space while listening throughout the stereophonic stream. They reported encountering non-human voices throughout the experience, such as humid rocks, strong wind, screaming dogs and singing birds.

Open Listening and Discussion

The third type of experience consisted in open listening and discussion sessions led with art-research practitioners, teachers and students. The first was a workshop at Kallio-Kuninkala, Järvenpää, Finland, on August 24th, 2022. The second was a public event at École d’Art d’Aix-en-Provence, France, on October 9th, 2022. The third was a seminar at École des Beaux-Arts de Paris, France, on November 7th, 2022. Sound materials included both unconditional audio live-streamed by our deep generative model, and the 15-minute planetary soundscape previously recorded.

The sessions started with conceptual elements about the deepscape, as well as technical information on how we pro- totyped our deep generative model. Then, I communicated listening guidance standing as deep listening instructions. Specifically, I told practitioners and students to listen atten- tively to the stereophonic soundscape flowing from our deep generative model, and ask themselves the following three questions: Where am I? What am I listening to? To whom does it cost? In addition, I asked them to imagine five words that would best describe their listening experience. Due to the openness of these sessions, most participants did not strictly follow these instructions, but rather used them as listening guidance to foster their imagination.

The conceptualisation of the deepscape found echo in most art-research practitioners and teachers. Participants that involved in the discussion generated various interpretations of the deepscape through/out listening. One artist had the feeling to be “on another planet inhabited by Martians”. One composer described “an impossible landscape of windy ocean with small insects and land grass”. Rather than a soundscape, one teacher felt they were under a “strange living creature”, or “monster”, attempting to regurgitate things with its throat. Students felt they witnessed “noises of a diver”, moving “inside and outside water” while “adjusting their oxygen level”. One of them added: “It’s a good profile of what a planet is sounding like. It’s like a blue planet.”

Discussion

In this section, I discuss further art and science work to be done to reveal and reconfigure the deepscape in more depth. Specifically, I discuss the planetary media computation driven by AI and revealed by the deepscape, the inseparability of ethics, ontology and epistemology when engaging with deep learning, and current work led toward bodily intra-acting through/out the deepscape.

Deepscape as Planetary Media Computation

Discourses on AI almost always suggest anthropomorphic conceptualisations and representations among expert and novice audiences. These conceptualisations often dismiss the media aesthetics of deep learning, as its computed flows of images, sounds and texts intensively permeate worldwide physical and digital environments. By evoking a sense of spatiality with the suffix “-scape”, and of deep learning with the prefix “deep-”, the concept of deepscape may reveal the planetary media computation driven by AI, recalling the intensive socio-material extractions led by corporate infrastructures to support its media flows across the planet ⁴¹. I am looking forward to critiques from other researchers to reconfigure this concept in more depth and entangle a plurality of perspectives and disciplines within a potential theoretical framework.

From a practical perspective, the concept of deepscape also emphasises the landscape aesthetics that accidentally emerged from deep learning. Indeed, first art practices with deep learning consisted in navigating so-called latent spaces computed by deep learning over scientific datasets built for image generation ¹⁷. Rather than the realism originally sought by scientists, the most intriguing findings essentially lied in ambiguous images, that deep learning interpolated or extrapolated from the dataset ³². Artists then curated and showcased such findings as art, or sought to recreate this navigation process in art installations, typically presenting the audience with deep learning-generated flows of sounds and images [10]. I suggest that this process is reminiscent of field recording, where sounds and images are experienced in a continuous flow, while navigating unknown places, sometimes producing strange encounters that exceed human interpretation skills, and careful data collection that might reveal the interconnectedness of the recorder with the planet ³⁵. 

Recently, artists have been exploring spatial concepts for AI, describing it as “imaginary landscapes” ⁵³ conveying a “worldview” through their training data ⁵, or even crafting deep generative models to create virtual places ⁴⁴. My Deepscape: Transversal artwork might contribute to these works, by situating data collection practices and planetary costs of deep learning within art practice, while seeking to resist the anthropomorphic culture of modern AI research.

Ethico-onto-epistem-ology of Deep Learning

Our diffractive prototyping of deep learning illuminates the inseparability of ethics, ontology and epistemology when engaging with deep learning. As we discussed above, collecting a dataset entails an ethics of ownership, representativeness and positionality. In our case, all soundscapes recorded online were open sourced by the Locustream online sound map ⁴⁹. Specifically, each soundscape comes with a complex history of negotiating with local populations and authorities, ensuring that an open microphone would hinder human and non-human communities as little as possible. Furthermore, the transversal soundscape dataset pragmatically represents socio-economic abilities of worldwide populations and countries to buy microphones and provide access to the Internet. Lastly, my positionality was described through the relying on a transversal data collection method, of which I attempted to be transparent on the advantages, motivations and limitations to it, as well as through my choice to deeply listen to one stream at a time while recording, instead of automatically recording all streams simultaneously using machine listening. I suggest that dataset building inevitably produces an ontology of deep learning-generated media. Reactions harvested along the three deep listening experiences testify of the planetary soundscape flowing from our deep generative model, approximating and normalising worldwide soundscapes in the dataset. Making training and streaming costs transparent through our technical collaboration, as suggested by diffractive art practice ⁴⁸, also illuminates the material entanglement of such planetary soundscape with planetary resources exploited by infrastructures of AI to produce deep generative models. Beyond artwork analysis ¹⁰, ¹⁷, ²⁹, I suggest that further research might be done on corporate deep generative models, such as Dall-E, GPT-3 or Jukebox, to reveal the ontologies they produce, since they are often trained unethically over large datasets automatically scraped from the Internet.

Lastly, deep listening experiences of my Deepscape: Transversal artwork may also promote a alternative epistemology for deep generative models. Indeed, scientists often evaluate deep generative models through fast and quantitative methods, typically automatically-generating short samples with deep learning, and measuring reconstruction quality using Likert scales ¹⁵. I suggest that approaching media flows computed by deep learning through slower and careful crafting may reconfigure how we conceptualise deep generative models, and perhaps, how we develop them. While such crafting practices are emerging naturally among communities of practices dedicated to deep generative models ⁴⁰, I suggest that researchers also have a responsibility to promote such slower ways of producing knowledge about deep generative models, in a rapidly-evolving computational society.

Bodily Intra-Acting Through/Out the Deepscape

While deep listening may help grasp the planetary scale of the deepscape, it might remain opaque for certain audiences that are not accustomed to music or sound art practices. Fortunately, there remains several ways of practicing and bodily intra-acting ⁴⁸ through/out the deepscape. Along with other artists ⁵³, ⁴⁴, I am currently exploring landscape image generation with deep learning. While images remain two-dimensional, they might recall the landscapes of the planet that are getting damaged as the deepscape rapidly evolves.

I am also currently leading sound and image field recording in one local portion of space, while widening it to all portions of time accessible over one year. Such a longitudinal data collection practice should complement the transversal method presented in this paper. Instead of a planetary soundscape, it may enable to learn a model of the planet it-self, as sound and light recorded over the year from one identical place may reflect the planet’s motion in time and space, with corresponding climatic variations. Such a deep longitudinal model should further my diffractive art practice with deep learning, revealing further ecological entanglements of AI with all socio-material bodies of the universe.

Lastly, I am currently exploring soundwalk-inspired modes of listening to, and practicing with, deep learning. I am collaborating with other artists-researchers to create virtual soundwalks in latent spaces computed by deep learning, as an art-based method to illuminate spatial attributes of the deep-scape ⁴⁶. We are also exploring computational methods to map bodily movements with soundscape modulations by deep learning to stimulate embodied ways of navigating, and producing knowledge, through/out the deepscape. I am looking forward to further suggestions from practitioners and researchers in other fields.

Conclusion

This paper presented early artistic, conceptual and technical work toward practicing and theorising through/out the deepscape—the planetary media computation intensively led by corporate infrastructures of AI over material, human and cultural resources. I proposed deep listening experiences of deep learning-generated continual flow of soundscape to raise awareness of the deepscape, by recalling the landscapes that are getting exploited by infrastructures of AI. By including non-human voices and planetary scales within research and practice with deep learning, this research-creation thus sought to resist the anthropocentric culture of modern AI research. I am looking forward to further art and science work that might be engaged to reveal and reconfigure the deepscape in depth.

Acknowledgments

Authors would like to thank audience members who took part in deep listening experiences, as well as humans and non-humans who contributed to the transversal soundscape dataset, and research-creation collaborators Anne Dubos, Antoine Caillon, Bertha Bermudez, Emanuele Quinz, Jan Schacher, Lucile Vareilles, Philippe Esling, Peter Sinclair, Samuel Bianchini, Vincent Rioux, et al.


(1) https://github.com/domkirke/RAVE/tree/multichannel

(2) https://github.com/acids-ircam/nntilde 

(3) http://locus.creacast.com:9001/deepscapetransversal.ogg 

(4) https://soundcloud.com/hugoscurto/deepscape-transversal-cirque-electrique-20221105

References

1 What if 99% of the Metaverse is generated by AI? https://cifs.dk/news/what-if-99-of-the-metaverse-is-made-by-ai. Accessed: 2022-11-29. M. Abadi, P. Barham, J. Chen, Z. Chen, Davis, A. Dean, J., M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., Tensorflow: a system for large-scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), 2016, 265–283. 

2  Generative AI: A Creative New World, Accessed, 2022-11-29, https://www.sequoiacap.com/article/generative-ai-a-creative-new-world/

3 Introducing the AI Research SuperCluster, Meta’s cutting edge AI supercomputer for AI research, Accessed, 2022-11-29, http://ai.facebook.com/blog/ai-rsc/

4 M. Akten, R. Fiebrink, M. Grierson, Learning to see: you are what you see, In ACM SIGGRAPH 2019 Art Gallery, 2019, 1–6.

5 M. Akten, R. Fiebrink, M. Grierson, Deep meditations: Controlled navigation of latent space, 2020, arXiv preprint arXiv:2003.00910.

6 G. Amato, M. Behrmann, F. Bimbot, B. Caramiaux, F. Falchi, A. Garcia, J. Geurts, J. Gibert, G. Gravier, H. Holken, et al., Ai in the media and creative industries, 2019, arXiv preprint arXiv:1905.04175.

7 A. Appadurai, Modernity at large: Cultural dimensions of globalization, volume 1. U of Minnesota Press, 1996.

8 G. Arvanitidis, L. K. Hansen, S. Hauberg, Latent space oddity: on the curvature of deep generative models, 2017, arXiv preprint arXiv:1710.11379.

9 S. Audry, Art in the Age of Machine Learning, MIT Press, 2021. 

10  Better Images of AI, Accessed: 2022-11-29, https://betterimagesofai.org/.

11 P. Bory, Deepnew: The shifting narratives of artificial intelligence from deep blue to alphago, Convergence 25(4), 2019, 627–642.

12 J. F. Boylan, Will deep-fake technology destroy democracy, The New York Times, 2018.

13 T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., Language models are fewshot learners. Advances in neural information processing systems, 2020, 33:1877–1901.

14 A. Caillon, P. Esling, Rave:Avariational autoencoder for fast and high-quality neural audio synthesis, 2021, arXiv preprint arXiv:2111.05011.

15 A. Caillon, P. Esling, Streamableneural audio synthesis with non-causal convolutions, 2022, arXiv preprint arXiv:2204.07064.

16 B. Caramiaux, S. F. Alaoui, ”explorers of unknown planets”: Practices and politics of artificial intelligence in visual arts, Proceedings of the ACM on Human-Computer Interaction, 2022.

17 T. Carneiro, R. V. M. Da No ́brega, T. Nepomuceno, G.-B. Bian, V. H. C. De Albuquerque, Reboucas Filho, Performance analysis of google colaboratory as a tool for accelerating deep learning applications, IEEE Access, P. P. 2018, 6:61677–61685.

18 C. Carr, Z. Zukowski, Generating albums with samplernn to imitate metal, rock, and punk bands, 2018, arXiv preprint arXiv:1811.06633.

19 G. M. Chatonsky, Esthétique des flux (après le numérique), 2016.

20 N. Couldry, and Mejias, U. A., Data colonialism: Rethinking big data’s relation to the contemporary subject, Television & New Media 20, 2019, (4):336–349.

21 K. Crawford, V. Joler, Anatomy of an ai system, Retrieved September 18, 2018.

22  K. Crawford, Atlas of AI, Yale University Press, 2021.

23  P. Dhariwal, H. Jun, C. Payne, J.W. Kim, A. Radford, L. Sutskever, Jukebox: A generative model for music, 2020, arXiv preprint arXiv:2005.00341.

24 S. Dinkins, Community, art and the vernacular in technological ecosystems. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, 2020, 221–221.

25 D. Foster, Generative deep learning: teaching machines to paint, write, compose, and play, USA, O’Reilly Media, 2019.

26 B. Goertzel, Human-level artificial general intelligence and the possibility of a technological singularity: A reaction to ray kurzweil’s the singularity is near, and mcdermott’s critique of kurzweil, Artificial Intelligence, 2007, 171(18):1161–1173.

27 M. L. Gray, and S. Suri, Ghost work: How to stop Silicon Valley from building a new global underclass, Eamon Dolan Books, 2019.

28  D. Grba,, Deep else: A critical framework for ai art, Digital 2, 2022, (1):1–32.

29 J. Heelan, E. Gratz, Z. Zheng, Q. Wang, M. Chen, D. Apelian, Y. Wang, Current and prospective li-ion battery recycling and recovery processes, 2016, Jom 68(10):2632–2638.

30 H. Herndon, Holly+, 2022.

31 A. Hertzmann, Visual indeterminacy in gan art, Leonardo 53, 2020,(4):424–428.

32 A. Javornik, B. Marder, J.B. Barhorst, G. McLean, Y. Rogers, P. Marshall, L. Warlop, ‘what lies behind the filter?’uncovering the motivations for using augmented reality (ar) face filters on social media and their effect on well-being. Computers in Human Behavior, 2022, 128:107126.

33 J. H. Korteling, G. Van de Boer-Visschedijk, R.A. Blankendaal, R. Boonekamp, A. Eikelboom, Human-versus artificial intelligence, Frontiers in artificial intelligence, 2021, 4:622364. 

34 B. Krause, Anatomy of the soundscape: evolving perspectives, Journal of the Audio Engineering Society 56, 2008, (1/2):73–80.

35 A. Lacoste, A. Luccioni, V. Schmidt, T. Dandres, Quantifying the carbon emissions of machine learning, 2019, arXiv preprint arXiv:1910.09700.

36 P. McCorduck, M. Minsky, O.G. Selfridge, H.A. Simon, History of artificial intelligence, In IJCAI, 1977, 951–954.

37 A. Mordvintsev, C. Olah, M. Tyka, Inceptionism: Going deeper into neural networks, 2015.

38 P. Oliveros, Deep listening: A composer’s sound practice, IUniverse, 2005.

39 J. Oppenlaender, The creativity of text-to-image generation, In 25th International Academic Mindtrek conference, 2022, 192–202.

40 J. A Parikka, geology of media, volume 46. U of Minnesota Press, 2015.

41 R Parloff, From 2016: Why deep learning is suddenly changing your life, fortune, 2016.

42 A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, L. Sutskever, Zero-shot text-to-image generation, In International Conference on Machine Learning, 2021, 8821–8831, PMLR.

43 A. Ridler, C. Sinders Mechanized cacophonies, 2021.

44 M. K. Scheuermam, A. Hanna, E. Denton, Do datasets have politics? disciplinary values in computer vision dataset development, Proceedings of the ACM on Human-Computer Interaction 5, 2021, (CSCW2):1–37.

45 H. Scurto, L. Postel, Soundwalking deep latent spaces. In Proceedings of the 23th International Conference on New Interfaces for Musical Expression (NIME’23), 2023.

46 H. Scurto, B. Caramiaux, T. Similowski, S. Bianchini, Ganspire: Generating breathing waveforms for art-health applications, In 5th NeurIPS Workshop on Machine Learning for Creativity and Design, 2021.

47 H. Scurto, B. Caramiaux, F. Bevilacqua, Prototyping machine learning through diffractive art practice, In Designing Interactive Systems Conference 2021, 2013–2025.

48 P Sinclair,  Locus stream open microphone project, In Proceedings of the 2018 International Computer Music Conference (ICMC 2018), 2018.

49 Spawning, Accessed: 2022-11-29, https://spawning.ai/.

50 R. Srinivasan, and K. Uchino, Biases in generative art: A causal look from the lens of art history, In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021, 41–51.

51 E. Strubell, A. Ganesh, A. McCallum, Energy and policy considerations for deep learning in nlp, 2019, arXiv preprint arXiv:1906.02243.

52 N. Tokui, Imaginary soundwalk, 2018.

Figure 2: Transversal soundscape dataset spatiotemporalities.
Authors Biographies

Hugo Scurto is an artist, designer and researcher, born and based in Marseille. Their research employs art, design and science to craft, prototype and diffract machine learning in an ecology of music. Their practice consists in creating, listening and performing with learning machines that reveal and reshape our musical entanglements with our environments. Hugo is currently postdoc at EUR ArTeC (Paris 8 / EnsadLab), and co-founding member of w.lfg.ng, a post-AI music collective. Before this, they completed a PhD at IRCAM, graduated in Physics from École Normale Supérieure Paris-Saclay, and were visiting research student at Department of Computing, Goldsmiths University of London.

Axel Chemla—Romeu-Santos is a composer, performer and researcher, born and based in Paris. Their scientific research focuses on generative models, bayesian learning, and signal processing to design unsupervised sound synthesis approaches based on perceptual inference, symbolic extraction, and raw signal generation. Their artistic work focuses on the creative aspects of such algorithms in composition and performance configurations. Axel is currently postdoc in the ACIDS group of IRCAM, CNRS, Sorbonne Université, and a co-founding member of w.lfg.ng. They completed a PhD in Deep Learning and Sound Synthesis between IRCAM and the Laboratorio d’Informatica Musicale, and graduated from CRR93 cursus of Computer Music. 

A Performance Co-Created with an Autonomous Virtual System: A Symbiotic Approach

Isadora Teles de Castro e Costa, Hui-Ting Hong, Chu-yin Chen

Université Paris 8 Vincennes Saint-Denis, France
isadoratelesdecastro@gmail.com, w102060018w@gmail.com, chu-yin.chen@univ-paris8.fr

Abstract

This paper is a case study of a performance co-created in interaction with an autonomous virtual system. Our outcomes point to an essential production period where the creative team learns to know the virtual system through indirect interactions: the match-up phase. During this step, co-creation and co-evolution moments happened, indicating a possible symbiotic relationship. We discuss the implications and the outcomes of working with autonomous scenography in a performative context. We then expand the reflection to the potential creative associations between performance arts and autonomous technology.

Keywords

Performance, Artificial Life, Autonomy, Symbiosis, Co-Creation, Unexpectedness, Real-Time Computer Graphics.

DOI

10.69564/ISEA2023-86-full-Teles-de-Castro-et-al-A-Symbiotic-Approach

Introduction

This paper is a case study of experiments carried out in the context of the project CECCI-H2M (”Co-évolution, co-création et improvisation humain-machine”). We will talk about a specific experience with a virtual ecosystem and about the creative process that enabled us to explore our research question: how can a performance be co-created with an autonomous virtual system?

We are interested in constructing virtual autonomy in the search for scenographic improvisation through the emergence of virtual patterns and behaviors. However, what is autonomy in the context of digital arts? An autonomous computational artifact, as described by Baljko and Tenhaaf: “has some sort of internal state space, it outwardly manifests information as to its internal state (...), and its behavior depends on its state” ¹. Similarly, Bishop and Erden define that “an autonomous system is typically considered to be a self-determining system, as distinguished from a system whose behavior is explicitly externally engineered” ⁶. In short, we could define autonomous computational artifacts as entities with an apparent internal motivation (internal self-determining state) that guides their behavior prior to the influence of external forces. According to Maturana and Varela, autonomy is a condition for a “living machine” and relates to the notion of autopoiesis, that is, self-fabrication ⁷. The independence of the artwork from the artist’s control tends to transport it to the realm of the living, at least as an artificial form of alterity or impression of Life. In the study of complex systems and within the domain of Artificial Life, the concept of emergence accounts for a surprising phenomenon, often linked to creative solutions. Using the simulation of complex systems inspired by nature, artists move from the creative process to the metacreative process: the design of autonomous mechanisms that, in turn, create artworks or artistic situations. Following this reasoning, Whitelaw ¹⁴ examines how artists have been inspired by Artificial Life to pursue emergence due to the agency, novelty, and creativity in biological, chemical, and multi-agent processes.

Within a second-order cybernetic point of view, Couchot, Bret, and Tramus defined the artifact autonomy in connection with its aptitude for self-organization and to afford interaction in different environments: endogenous and exogenous. According to them, an autonomous artwork behavior is qualified by endogenous interactivity, which handles the dialogue among the virtual objects, able to perceive specific characteristics from each other and keep relationships more or less complex. The autonomous virtual system can also be open to the perception of its environment in a “second-order interactivity”, in analogy to second-order cybernetics. Beyond the engagement between the participant and the artistic process through an interactive interface, second-order interactivity would appear when the autonomous system can modify itself during a learning process by interacting with its environment to adapt to it. The participant is encouraged to explore aesthetic possibilities when embedded in a reciprocal, emergent, and unforeseen interactive context. Depending on the interaction interface setup, the means at his disposal for this exploration enables gesture experimentation, movements, and creativity, in the search for communication or understanding ¹³.

In our practice and research, the application and discussion around Artificial Life are considered in the performance art context and connected with unpredictability. Inspired by Edmond Couchot’s question ³ about “how does the machine create the unpredictable on stage?” we intend to study autonomous virtual systems based on the simulation of complex organic systems within the performance context to promote a new sense of unpredictability and improvisation.

Bringing a performer into a confrontation with an autonomous virtual and perceptual system encourages emergent phenomena and motivates improvised reactions. Our hypothesis is based on establishing this interaction dynamic as a possible way or method of emergent narrative creation, built during the interaction in a co-creative way. This method can indicate a symbiotic phenomenon with an artistic purpose and shed light on new ways of scenic composition with new technologies. As researchers and designers of virtual systems, we imagine methodologies and concepts about the associations between autonomous virtual systems as improvising presences and the living arts. We wonder: How can the performing arts take advantage of autonomous virtual entities in the quest for new ways of imagining the stage space and the performance? How to articulate new ways of conceiving dramaturgy and being inspired? How can unpredictability be used to express contemporary emotions and concerns?

Methodology Research Questions

The methodological problem that encouraged this project is divided into two parts. First, how to create an interactive virtual system to promote co-creation on stage? Second, how to cooperate with this system while composing a performance, aiming for a creative relationship between humans and machines?

We chose to develop an evolving virtual ecosystem to solve the first problem. This choice is due to the leads in the literature about artificial life algorithms and their potential to represent or generate emergent and unexpected aesthetic phenomena. On the other hand, we chose the performance medium to answer the second problem through the performer’s persistent and attentive improvisation vis-à-vis her environment. Our main goal during the experimental phase of the artistic experimentation was establishing an exploratory exchange between the performer and the virtual system, keeping the autonomy of the two partakers and, therefore, their potential unpredictability to bring out a performance that shows a mutual improvisation process.

Research Hypothesis and Interaction Setup

The experiments of the research-creation project described in this article are the continuation of the experiments of last year described in the article ¹¹. Iterating last year’s reflections and inspired by leads in literature and electronic digital art, such as the cases of the creative ecosystems of Jon McCormack ⁸, we have created an evolving virtual ecosystem as an essential tool for testing the hypothesis of a creative dynamic between human and machine.

Concerning the interface, we hypothesized that an “indirect” interaction (Figure 1) would make the interaction dynamics more exploratory and keep the autonomous (or independent) character of the two types of entities in dialogue, real and virtual. We move from a first-order (responsive) to a second-order (adaptive) interaction. The ideas for the performance would accommodate themselves to the progressive aesthetic discoveries enabled by the interaction with the autonomous virtual system.

We based the design of our experiments on developing the foundations of an evolving interactive program to be tested and improved during the artistic residency. We then conducted interaction tests to verify how our creative expectations and the virtual behavior styles fit.

Core Co-creation Process

The main result of our research-creation residency was the provision of a methodology for developing, managing, and interpreting a virtual autonomous scenography in the context of the living arts. This methodology involves match-up phases through interaction with the virtual system in a receptive way to emerging aesthetic possibilities by the entire artistic team. We, therefore, achieved a co-created performance in 4 steps:

  1. The development of an evolving virtual ecosystem (E.V.E.) considering the performer’s presence and the research-creation goals (co-creation and co-evolution).
  2. The match-up phase, by the art director, of the behavior and appearance of the E.V.E., understanding how it works, and exploring the various points of view and approaches to the system.
  3. The phase of matching up the E.V.E. by the performer, exploring the aesthetic potential from the interaction between her movements and the system’s behavior.

The match-up phase of the E.V.E. by the developer and stage manager once the directives of the artistic direction are established, and the intention of the performance is solidified.

Experiments with E.V.E.

The first step in exploring the issue of co-creativity was creating a virtual system capable of manifesting surprising, emergent, and “creative” behaviors and patterns regarding our expectations and concerning the initial state of the visual explorations ². We have envisioned the different behaviors of our virtual ecosystem through the planning methodology of an agent-based model1. Then, we identified some simulations that were sensitive to their virtual environment, autonomous, and with types of movement close to those we expected to perform virtually. Inspired by the idea of simulating complex self-organizing systems, we used algorithms from the field of Artificial Life, more specifically from artists and researchers who work with speculative biology or synthetic biology. Finally, we adapted the chosen biological simulations to our artistic desires and associated the resulting behaviors with a second-order interaction with the creative team.

Figure 1: Indirect interactions between artistic team and E.V.E. ©CECCI-H2M

E.V.E. Development

We experimented with the design of a first species based on the research and reports of Arsiliath 2 and Sage Jenson3, who were inspired by Jeff Jones’s publication on Physarum transport networks ⁴. It is a combination of two behaviors: ant foraging and the movement and growth of the Physarum.

Figure 2: First, second and third species. The third one is represented in magenta on the right image. ©CECCI-H2M.

This first species can perceive its surroundings with a limited angle and distance. Therefore, each agent checks the neighboring pixels to measure the quantity of “pheromones” present, represented by colored lines left by the trajectory of agents of this species. Where there is a significant quantity of pheromones (intensity of R, G, B in the neighboring pixel), the agent will assume as its next destination, forming moving patterns of agents. The emergent aspect of the pattern created by the collectivity of entities is reminiscent of the patterns created by the movement and growth of the Physarum in search of food.

The second species was based on the boid’s flocking algorithm ⁹ , ¹⁰. To move, the agent measures its neighbor’s speed, direction, and position to calculate its speed and direction, intending to align its trajectory, separating from neighbors so as not to hit each other and maintain cohesion in the speed of movement. This technique allows agents to form concise groups according to the angle of view, the extent of perception, alignment, separation, and cohesion.

The third species is inspired by the behavior of random walkers, where agents move according to a random direction. We used a probability technique to make their movement random but biased in the direction of a target. The target or attractor, in the case of all three species, is the position of the food. First, we created food points generated according to a four-dimensional “Hermit” type noise. Food attracts agents with less energy, nourishes them, and repels agents with high energy.

Once the species’ behaviors were created (Figure 2), we brought them together in a single virtual environment and created the interactions between them. Agents could perceive and guide each other, regardless of species. Then we set up the ecosystem dynamics with genetic algorithms. We linked the main behavior parameters of the agents to 4 genes, corresponding to the values of R, G, B, and A in a texture buffer which kept the D.N.A. of each agent. See Table 1 for an illustration of the correspondence between each “gene” and each species parameter.

Agents had an initial amount of energy that increased with contact with food and decreased as the agent moved away. The agent needed to have a minimum level of energy (called health in the program), check if there is an equally healthy neighboring agent, and count with a probabilistic decision to reproduce. If the decision is confirmed, the two entities’ D.N.A. is crossed (crossover) with a mutation rate, and a new agent is created in a free space. If an agent’s energy level is too low, it is disabled and ignored in updating the parameters of each pixel of the input textures (death of an agent in the system). See Figure 3 for the general functioning of the virtual ecosystem. 

Table 1: Correspondences between D.N.A. (R,G, B, A gene list) and virtual agent’s parameters for each species
Figure 3: General functioning of the virtual ecosystem ©CECCI-H2M.

Compute Shaders

Arsiliath, in his Psychobiotik course, proposes an optimized algorithm to realize a multi-agent system with compute shaders in Unity. We used TouchDesigner to combine all the techniques we would use in the show in a single software (mapping on various video projectors, generative image, sound analysis, connection with Kinect, message management in MQTT, etc.).

To design the simulation, we used the compute shader, a parallel computing technique on the graphics card. It computes information arbitrarily and is not involved in the rendering pipeline, although it can render images. The information and properties concerning the agents were stored on texture buffers at the shader’s input and output. It is a technique that uses the color information spaces of an image (texture) to store data and therefore be able to perform calculations in parallel, fast, and without relying on the computer’s memory and the CPU (Central Processing Unit).

Indirect Interaction

A few types of interactions, with variable control over the system, were set up and used by team members. For example, a control interface allows the stage manager/developer to modify specific parameters concerning the virtual agents’ behavior limits without intervening in their behavior directly. During the residency, we tested the virtual system in interaction with the performer. Her joints were detected by the Kinect camera and transposed into the virtual environment as food points position data (see Figure 4 ).

Another type of interaction was controlling the camera’s position and the scene’s rendering. The artistic direction employed this control to shape the virtual environment in favor of expressing the artistic intentions and sensations for the scene ambiance.

Figure 4: Performer’s skeleton data translated to the E.V.E. as food position information. ©CECCI-H2M.

Match-Up Phases Artistic Direction

First, the virtual system developer presented the system’s behavior to the artistic director. We have indicated some possibilities of exploration and visualization of the virtual environment and the possibilities of indirect interaction by experimentation with the limits of the forces and quantity of visible elements. Then, the artistic direction (A.D.) chose important moments based on observing the system in action and the various emotions and sensations that its appearance engendered. The intention was to look for movements or visual structures representing three seasons: Winter, Spring, and Fall. The director identified each selected moment’s specific parameters for testing on the stage. To represent the season’s energy and help refine the system, the A.D. has developed keywords to communicate her ideas:

Figure 5: Winter, Spring, and Fall represented on the performance. ©CECCI-H2M.

Through these keywords and key concepts, both the developer and the dancer could imagine their respective means of expression to converge toward a coherent creation — the dancer with her body, the developer with the evolving virtual ecosystem. In a previous work, our experiments indicated the importance of a shared vocabulary between the different creative spheres for corresponding to the artistic intentions with visual and behavioral aspects of the virtual generative system. The A.D.’s exploration of configurations and behaviors of interest was fruitful for experimenting beyond the developer’s expectations. We witnessed new visuals and behaviors by having another person with different expectations and ideas explore the system at the software level. From the various possibilities of rendering and behavior discovered during the experimentation of interaction between the dancer and the virtual system, the A.D. presented new ideas and directives of narration, choreography, and suggestions for exploring space. This phase of match-up amplified the horizon of sensations and concepts included in the performance.

Performer

During performative experiments between the performer and the evolving virtual ecosystem, it was interesting to observe how one modifies or influences the behavior of the other according to their progression over time. The method used to qualify and analyze the interactions promoted during our experiments was the examination of the videos of the rehearsals, the explicitation interviews, and the study of different visuals from the moments of interaction.

The representation of the performer’s body in the form of the virtual agents’ food points was possible only during the match-up tests. Before the match-up tests happened, in this specific research-creation residency, the performer’s artistic style follows an all-encompassing principle that represents the natural order of the universe and emphasizes the importance of living in harmony with it. The practices of performer’s dance are the cultivation of inner peace that originated from the sage Laozi: “The way (Tao) gave rise to the one. The one gave rise to the two. The two gave rise to the three. The three gave rise to all the ten thousand things”. With the keywords (shared vocabulary) developed by A.D., the performer absorbs, internalizes, and transforms these concepts through the inner path of the body to understand and explore the visual and luminous impressions coming from the projection.

The process of understanding and exploration of the performer demonstrates adaptation progress of freedom through perceiving the environment. The performer mentioned that her ideas of movements in this experiment are no imposed figures before the match-up tests between the two partakers (the performer and the virtual system), instead, it presents a great deal of freedom of exploring the aesthetic potential for the performer herself who can, according to the intention, speed up or slow down, move or stand still, stretch or contract, using her body to create a new representation and give free rein to the imagination.

The intention of the dancers gradually changed as the match-up tests progressed. The intention changed from paying attention to the internal energy transformation of the body, to feeling the temperature and density of the performance space, and to perceiving the behavior of different species from the virtual system. We further discovered how the system reacts to the performer’s specific movements and what behaviors would be necessary for the interaction to be visible and graceful Figure 6.

An example of co-creation during the match-up phase was adjusting the speed of the performer’s movements so that the virtual agents could adapt and have time to reach the food. The dancer’s movements had to be slower, and the maximum magnitude of the virtual agents had to go up to achieve a balance between responsiveness and autonomy of each virtual agent movement. This necessity was evident in the winter scene, where the camera’s point of view was far away, and therefore the movements of the agents were less evident in the short term. To make them more visible, we had to increase the agents’ responsiveness and maximum speed. This decision influences the selection and reproduction of the agents that will persist in the ecosystem: only the fastest agents are selected. To balance the virtual rhythm with the performer’s movements, we had to slow down the performer’s movements and give the agents time to reach the food, even if they were slow. The goal was to seek a more varied selection over time.

Figure 6: Different moments of the system match-up phase by the performer ©CECCI-H2M.

Analyzing the videos of the match-up experiences, we observed compelling movements of the performer towards the virtual system — signs of imitation attempt to understand reactions and immersion in the rhythm predominant in the projection. The stage manager also experimented with system parameters to make the interaction more or less apparent.

The interaction’s rhythm, speed, and power were still in the research phase. The evolution process of the two inter-actors fluctuated according to the power of reactivity of the system concerning the performer. The variety of behaviors and viewpoints extended the scope for imagining a story or narrative that would explain the different behaviors, such as different personalities or characters of the visible agents. The heterogeneity of behaviors among the agents made it easier to have affection and empathy towards the personalities and the representation of an environment inhabited by various autonomous and unpredictable entities. The contrast highlights this variety: when we see that over time, the agents standardize their movement towards more speed and responsiveness to reach the food points more quickly.

Unlike the procedure with the artistic direction, the matchup phase carried out with the performer was much more embodied and based on her sensitive, tactile, and energetic perceptions of the scenic atmosphere created by the evolving virtual ecosystem. This intention-in-action (in contrast to prior intention) adapted by the performer focuses on the significant association with awareness and randomness. She is present and lives in this autonomous scenography through the improvisation practices of freedom. However, she did not consider the system’s functioning, so we cannot say that being an evolving ecosystem, conceptually or technically, has influenced the quality of the interaction or the dynamics of co-creation. The dancer states that she had the impression of being in a space without gravity (“outer space”). We would like to know if we were close to embodied immersion in the virtual environment.

Initially, the variety of movement and the autonomy in the agent’s decisions (the need or not to seek food, variety of attraction forces) moved the interaction away from the feeling of control and manipulation on the part of the performer. Instead, it became a feeling of immersion in an environment, in a cohabited atmosphere, where the influence between one and the other is relative and negotiable. In a retroactive dynamic, the ecosystem follows the performer, and the performer guides his movements through the rhythmic effects in his surroundings.

Stage Manager and Developer

The developer took notes and screen captures of the corresponding configurations at the specific moments that inspired the A.D. Nonetheless, the patterns depend not only on the system parameters (indirect controls) but, above all, on the interactions’ history. Therefore, the variables captured by the developer only contributed partially to reproducing the moments chosen by the A.D. The vocabulary stipulated by the A.D. was the primary support for investigating the desired appearance. Guided by the keywords, the developer improvised and directed the agents to behaviors or organizations close to the metaphors expressed by the keywords. For example, if the intention was to have a warmer, agitated, and chaotic scene, the developer/stage manager sought to modify the parameters without knowing the exact consequence of such changes to seek a rendering close to expressing such keywords.

Though it was difficult to impose a specific appearance or behavior because the system was not controllable (and that was not the idea of the experiment), the developer mediated the dynamic between the A.D. and the system. To understand or predict the emergent patterns of the changed parameters associated with the interactions with the performer and the history of the millions of parallel interactions whose comprehension eludes us, we had to improvise and explore the system’s possibilities in an almost striving relationship. The results of this match-up effort were sometimes disappointing for not rendering the desired appearance. However, it was often surprising to discover more satisfying visuals than expected.

We developed an interface to interact with the scenes and system variables, allowing the freedom to adjust parameters in real-time. During the presentations, this freedom was essential for taming, exploring, and discovering new movement possibilities and the virtual system’s appearance. Since it is an evolving system, the parameters cannot be fixed beforehand but can only be tweaked by the stage manager during the performance. Therefore, desired aesthetic ambiances must be pursued and evolve gradually through interactions and temporal and spatial transformations of the virtual system during the performance.

Figure 7: Volumes formed by the agents’ agglomeration, forming the impression of new virtual entities ©CECCI-H2M.

Discussion Symbiotic Imaginaries and Autonomous

Individuations

The system did not count on physics simulation, and the agents did not detect collisions (despite a tendency for separation, according to the boids algorithm). Instead, the entities agglomerated, overlapped, and gave the impression of mass and fusion: as if the whole formed another element. This impression is also due to the agent’s persistent alignment and paradoxical erratic manners between cohesion and repulsion. This behavior showed volumes of virtual matter moving in one direction, volumes presenting deformities and noises in their contours and shapes. From the more distant point of view, which shows the organizational phenomena on a large scale and collectively, we can see the paths formed by the agglomeration of virtual agents. The agent’s trajectory features became visible through the effect of image accumulation and the reduction of their opacity. The patterns created by the endogenous interactions and their speed and direction adaptations concerning their neighbors become evident. This scene shows a less individualizing level of the agents. However, it makes visible the formation of agglomerations that resemble the formation of other individuals or at least other configured forms of a collective or a mass of agents (see Fig.7).

The scene representing Spring is the one where the camera is closer to the agents in its scale. It reveals the entities’ geometric shapes, colors, and particular behaviors. Therefore, it is a scene where the composition is more dynamic and faster, the images are beyond the control of the stage manager and the dancer, and the interaction is more mysterious depending on the area of the ecosystem focused by this point of view. Regardless, another type of empathy is produced by this scene. The dancer can differentiate each agent’s life stages, birth, search for food and path, and death. That is when one understands that an association of individuals forms the previous image.

The diversity of behaviors and the multiple imaginaries inspired by the different scales of observation of the virtual system encourage us to explore performance through the evolution of interaction without having a narration or a script beforehand. The idea is constructed around and with the virtual scenography presence and by the curious exploration of its aesthetic and interactive possibilities.

While, initially, we did not have a script or description for the performance, the narrative’s structure began with the virtual system’s development. We manifested our imagination about a virtual performance of autonomy while constructing behavior and interactions between agents. That autonomy, in our case, is not only expressed at the virtual individual level but on the emergent collective appearances as well. It is about multiple endogenous and indirect exogenous interactions, inducing unpredictability and a sense of disobedience, resistance, or even conflict. Furthermore, the choice of losing control over the system’s transformation also abstracts our intentionality. The system enacts our initial intentions and ideas, and then it is left to develop independently. Whitelaw raises these issues about the challenges of operating Artificial Life as a tool because it also carries a conceptual and cultural engagement in its expression and choices. Engagements that are narratives themselves ¹⁴.

Symbiotic Creativity

Another way of seeing the emergence of new forms than the neo-Darwinist interpretation is symbiogenesis, proposed by Lynn Margulis ⁵. This theory of the vital evolutionary path intersects with McCormack’s vision of creative emergence through the combination of forms and behaviors. This biological approach inspires our assumptions about appropriating natural Life with technology. As Tenhaaf indicated: we take “biology as a ready-made” ¹². The concepts and techniques of artificial life simulation guide us toward the emergent affinity between the performer and the evolving virtual ecosystem by treating the two interactors in another ecosystem where we add the scene’s environment. We aim for codependent relational recombination between the parts to form a conceptual and creative symbiogenesis: an association of movements, gestures, shapes, and compositions.

As explained by McCormack, the ecosystem can be more than just virtual; the user, the system, and the environment can form it. That implies the vision of the virtual system not as a tool but as a member of the scenic ecosystem. Therefore, symbiotic creation can be considered at two levels in our project, endogenous and exogenous. The endogenous symbiotic interaction exists in the association of agents when one takes advantage of the trajectory of the other to achieve its goal. On the other hand, exogenous interaction allows the organization of a symbiotic association at the level of the stage’s environment. A dynamic of cohabitation between the artistic team and the virtual system is set up in a scenographic form.

Concerning the limitations and possible improvements of the virtual system, on the other hand, the task that measures the ability of agents to survive in the virtual environment is the ability to feed. This task was revealed as a not very creative one. Since the food is the dancer’s body, the main task is to follow the dancer’s body, and it was solved by selecting the most agile individuals. Nonetheless, we identified more creative tasks in the aesthetic domain: the movement and its variety and the creation of unexpected patterns. These aesthetic observations are paradoxical with the selection process we have implemented—the current dynamic values the standardization, speed, and concentration of agents on feeding themselves.

A solution would be to reverse the agents’ aptitude evaluation to appreciate those not following the performer’s movements as a priority. Instead, we aim to value those who prioritize the tasks of interaction and distraction in the virtual environment while keeping their need to feed themselves. These tasks that prioritize variety correspond better to the moments when rendering and behavior were more interesting.

Our new challenge is to keep a clear interaction with the dancer and choose another dynamic to enhance the variety of behaviors and value the “dancing” aspect of the agents. It is a programming approach of an aesthetic choice in the language of the rules of the ecosystem’s algorithmic way of working. This translation from an aesthetic realm to a symbolic realm is our next step and our new challenge concerning meta creation and symbiotic creation.

Conclusion and Perspectives

Adding unpredictability to the stage dynamics allows the dancer to improvise and discover new movements motivated by the virtual system. Those movements are perceived by the system and modify it, establishing a creative feedback loop. The project provides us with reflections on co-creation in a symbiotic way on many levels. The stage manager performs to acquaint the system; the artistic director draws inspiration from the behavior for staging ideas, and the dancer modifies and improvises according to the perception and reactions of the system. At the internal level of the system, apparent associations between the agents form cohesive and expressive volumes.

We have put in place since last year a system to enable co-creation, which could be used as a creative partner in the process of performative dramaturgical creation (in contrast to the notion of a tool). In a metacreative approach, we have designed a machine that generates evolving patterns and graphic behaviors. The creation methodology schematized in Figure 1 (indirect interactions during the match-up phases) is the most critical result. It makes it possible to adapt our team’s system and creative process to other contexts of symbiotic metacreation for the performing arts.

Acknowledgments

This work benefited from the support of EUR ArTeC and from the funding assistance of the French government managed by the National Research Agency under the future investment program bearing the reference ANR-17-EURE- 0008.

References

1 M. Baljko, and N. Tenhaaf, The Aesthetics of Emergence: Co-Constructed Interactions, ACM Transactions on Computer-Human Interaction (TOCHI), 2008, 15:11.

2  P. Cariani, Emergence and Artificial Life, In Langton, C. G.; C. Taylor, J. Doyne Farmer, S. Rasmussen, eds., Artificial life II: proceedings of the Workshop on Artificial Life: held February 1990 in Santa Fe, New Mexico, Santa Fe Institute studies in the sciences of complexity proceedings. Proceedings volume 10. Redwood City, Calif: Addison-Wesley. Meeting Name: Artificial Life Workshop, 1992.

3 E. Couchot, Automatisme,autonomieetesthe ́tique dans les arts vivants, In Corps en scène: L’acteur face aux écrans, Paris, ENTRETEMPS ED, illustrated édition, 2018, 21–34.

4 J. Jones, Characteristics of pattern formation and evolution in approximations of physarum transport networks. Artificial Life 16(2). Publisher: Massachusetts Institute of Technology Press (MIT Press), 2010.

5 L. Margulis, R. Fester, eds. Symbiosis as a Source of Evolutionary Innovation: Speciation and Morphogenesis, The MIT Press, 1991.

6 J. Mark Bishop, Y. J. Erden, Computational Creativity, Intelligence and Autonomy, Cognitive Computation 4(3), 2012, 209–211.

7 H. R. Maturana, F. J. Varela, Autopoiesis and Cognition: The Realization of the Living, Dordrecht, Holland, Boston, D. Reidel Publishing Company, 1st edition, 1980.

8 J. McCormack, Creative Ecosystems. In McCormack, J., and d’Inverno, M., eds., Computers and Creativity, Berlin, Heidelberg, Springer, 2012, 39–60.

9 C. W. Reynolds, Flocks, herds and schools: A distributed behavioral model, In Proceedings of the 14th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’87, Not Known, ACM Press, 1987, 25–34.

10 C. W. Reynolds, Steering Behaviors For Autonomous Characters. In proceedings of Game Developers Conference, San Jose, California: Miller Freeman Game Group, San Francisco, California, 1999, 763–782.

11 I. Teles de Castroe Costa, C.-Y. Chen, S. S. Circu, Artificial Life within a frame of metacreation on stage, In Universitat Oberta de Catalunya., ed., ISEA2022, International Symposium on Electronic Art, Barcelona. Proceedings. ISEA & UOC, 2022.

12 N. Tenhaaf, As Art Is Lifelike: Evolution, Art, and the Readymade, Leonardo 31(5), 1998, 397.

13 M.-H. Tramus, M. Bret, E. Couchot, La seconde interactivite, "In De l’art cinétique à l’art numérique ; hommage à Frank Popper, number 106 in Collection Eidos, Série Retina, Paris, L’Harmattan, 2017, 179–192.

14 M. Whitelaw, Metacreation: Art and Artificial Life. Cambridge, MA, USA, MIT Press, 2004.

Bibliography

Chu-Yin Chen, “Autonomous System for Interactive Digital Art”, In 10th Generative Art Conference Proceedings, Universite ́ Polytechnique de Milan, Italie, 2007.

Chu-Yin Chen, ”Un Monde pleinement e ́nacte ́”, in Action, Enaction - Emergence de l’œuvre d’art, Xavier lambert (dir.), Editions L’Harmattan, Paris, 2017.

Steve Dixon, Digital Performance: A History of New Media in Theater, Dance, Performance Art, and Installation, Edited by Roger F. Malina PhD and Sean Cubitt, Reprint edition, The MIT Press, 2015.

Simon Penny, Making Sense: Cognition, Computing, Art, and Embodiment, Cambridge, Massachusetts, MIT Press, 2017.

Marie-Hélène Tramus, Chu-Yin Chen, Judith Guez, Jean-François Jego, Dimitrios Batras, Dominique Boutet, Marion Blondel, Fanny Catteau, Coralie Vincent, "Interaction gestuelle improviseé avec un acteur virtuel dans un théâtre d’ombres bidimensionnelles ou au sein d’un univers virtuel en relief : l’illusion d’un dialogue ?",In Stéréoscopie et Illusion, M. Almiron, E. Jacopin, G. Pisano (dir.), Presses universitaires Septentrion, 2018.

Authors Biographies

Isadora Teles is a Brazilian artist based in Paris. She is currently a Ph.D. candidate within the INREV research team at the University of Paris 8. Her artistic research and practice are in the fields of digital generative art, artistic modeling of self-organizing systems, and the design of interactive interfaces for live performances.

Chu-Yin Chen is an Artist and Professor in Digital Art, INREV research team at Paris 8 University. Her creations, based on Artificial Life and complex systems, develop interaction modes between audience and virtual creatures showing autonomous and evolving behaviors. Her digital artworks have been shown in numerous international exhibitions. Her research articulates two overlapping areas: 1] Digital Creation using algorithms of complexity and emergence, and 2] Metacognition and Elicitation of the processes of creation, enaction and aesthetic reception, via psychophenomenology and mindfulness.

Hui-Ting Hong is an experimental artist from Taipei, Taiwan. Based in Paris. She focuses on the interaction and transformation between the body, space, and mind. She aims to explore the existence and reproduction of the human body by expanding the boundaries of perception. Her research-artistic practice focuses on the perception of bodily movements and the hybrid human body, particularly discussing the virtual representation of the body and its energetic variation. 


Complexity Explorer - Introduction to Agent-based Modeling - Summer 2022, https://www.complexityexplorer.org 

2 ”arsiliath (@arsiliath) / Twitter.” 2021. Twitter. Accessed October 22. https://twitter.com/arsiliath.

“Physarum - Sage Jenson.” 2021. Accessed September 19. https://www.sagejenson.com/physarum. 

Light Up Kelowna: Coordination and Development of Networked Community-based Media Art Urban Screen Infrastructure

Miles Thorogood, Kirsteen McCulloch, Aleksandra Dulic

The University of British Columbia, Kelowna, Canada
miles.thorogood@ubc.ca, director@artsco.ca, aleksandra.dulic@ubc.ca

Abstract

The ‘Light up Kelowna’ urban screen features a variety of digital artists who use technology to push art in different and thought-provoking directions, tell our community’s vital stories and allow the public to engage with the artwork in a multidimensional and multisensory way. In this paper, we outline the structure for coordinating engaged parties in developing scalable urban screen infrastructure and considerations necessary for installing rear projection urban screens in existing city spaces. We discuss the network architectures and topologies for creating networked urban screen systems, borrowing concepts from networked music performance and installation contexts. Finally, we demonstrate the use of our approach for developing an urban screen, showcasing multiple exhibitions.

Keywords

urban screen, network topologies, Multi-Agent Systems

DOI

10.69564/ISEA2023-87-full-Thorogood-et-al-Light-Up-Kelowna

Introduction

Illuminating public spaces change the texture of urban environments by activating under-utilized areas. While simply turning on a light opens dark areas at night, media art projections add a layer of cultural engagement to energize citizens and manifest different feelings and details of experience. Urban screen initiatives that transcend the commercial use of displays toward media art are a medium for active public engagement and discovery in urban settings. As a situated media, the cultural capacity of such screens provides a platform for transforming people’s sense of participation and attachment in a place—providing an attraction for the creation of activity and shaping people’s movement in urban spaces ⁵. As a platform to remediate the city, urban screens focus on the relationship between the artists, technologies, spaces, and aesthetics towards becoming integrated into people’s psyche. Dubois et al. ³ highlight key creative strategies to address these relationships that include: framing the large-scale digital projection and logic of the monument as expanded cinematic practice; engaging with technology art-making practices enable artists to create new telepresent and telematics rituals and opportunities for urban activism and identification; and there is a need to transform public screens into exhibition spaces.

Investigating the affordances and limitations of urban screens as communication infrastructure in a local suburban area, Hannon ⁴ outlines how large urban screens operate in terms of their modalities (pre-recorded, live, interactive) and functions (information provision, event support, screen events, communication). Successful negotiation of the specific community needs becomes vital to realize the full potential of such a screen.

In 2020, we recognized the need to bring our community together through art and create content that can be enjoyed safely. Created in response to COVID-19 community needs, ‘Light Up Kelowna’ includes urban screen infrastructure dedicated to digital art exhibitions. Presented at one of Kelowna key public spaces, this program involves the community creating urban screen exhibitions for an accessible art experience in a public space.

In the following sections, we outlined the structure for coordinating engaged parties in developing scalable urban screen infrastructure. Next, we detail the considerations necessary for installing rear projection urban screens in existing city spaces. Following, we go on to discuss the network architectures and topologies for creating networked urban screen systems. Finally, we demonstrate the use of our approach for developing an urban screen with the presentation of multiple exhibitions and present our conclusions.

Urban Screen Infrastructure Coordination

Developing urban screens infrastructure requires negotiation between a network of municipal, venue management, funding, curatorial, and art community partners for possibilities transforming the locale. Struppek ⁷ highlights that content also needs to be coordinated in how, when, and in what specific locations screens can be integrated with the urban land-scape and its architecture. It is this balance between content, location, and type of screen that determines the success of the interaction with the audience and artists. The Surrey UrbanScreen ⁶ project is one example of the successful collaboration between institutions and artists for the realization of state-of-the-art urban screen infrastructure. For Surrey UrbanScreen, artists were critical in steering the original concept of its form and functionality through equipment rebuilds and enhancements. Light Up Kelowna closely aligns with this coordination structure for building the ongoing technological and artistic capacity of vibrant public displays.

For ‘Light up Kelowna’, the Arts Council of the Okangan coordinated with the venue and City of Kelowna to negotiate installation spaces, and faculty from The University of British Columba, Faculty of Creative and Critical Studies (UBC) who have the expertise to develop the urban screen system and exhibition curation. In addition, UBC generously funded the pilot project (Figure 2.). In our experience, the central coordination structure of the network simplified the workload of engaged parties resulting in an efficient turnaround from concept to realization. As a community-based project with support from publicly funded partners, Light up Kelowna is an art-specific urban screen not requiring negotiation with private entities to interrupt the cultural value with advertising, for example. The sole purpose of the urban screen as an exhibition space then affords flexibility for programming artists’ work without needing to synchronize with other arranged content. 

Figure 1: Image of Light Up Kelowna at Rotary Centre for the Arts. Kelowna, BC. Canada

Light Up Kelowna Urban Screen System

The Light Up Kelowna infrastructure is designed to inhabit underutilized surfaces appropriate for urban screen installa tions. We focus on the rear projection of large windows with minimal functional value, such as shopfronts, office buildings, and community venues. To modify such surfaces for rear projection, it is a simple matter of applying a frosted vinyl film to the inside of the window and positioning a projector in an appropriate location. The benefits of this approach are that it enables installation of the system within a day, equipment is secured indoors, the minimal modification to the space makes it easier to negotiate installation with partners, the vinyl allows daylight to enter the building, and it is cost-effective compared to other solutions such as LED walls. Challenges with this approach are that projections will not work during daylight hours, there may be interference with objects and activities inside the building with the projector image, and the quality of the image is susceptible to ambient light conditions. When identifying a location, these challenges must be considered. We used the following list of considerations to evaluate the urban screen installation:

• Windowdimensions
• Viewability of urban screen
• Buildingimpactoftheinstallation
• Ambientlighting
• Projectormountingopportunities
• Imageinterferencewithbuildingactivities

Figure 2: Diagram showing the coordination structure for Light Up Kelowna ans system of mutually benificial relationships.

The pilot of Light Up Kelowna described here is located on three adjacent mostly square windows at the main entrance of the Rotary Centre for the Arts (RCA) in Kelowna, BC, Canada—a multi-purpose venue designed to accommodate special events, theatre, music and visual arts. We identified these windows as optimal due to: their large scale; they face toward an open public space; the desirability of a triptych arrangement of the windows; good mounting options for the projectors; covering the windows with vinyl has a minor impact on the feeling of the inside space; and building activities have little interference with projector image.

LUK Software system

Each window in the Light Up Kelowna system at the RCA has a dedicated projector and computer running projection mapping software written in OpenFrameworks. Similar to ⁹, a conductor program on one of the machines is responsible for synchronizing media across the windows over a wireless local area network. Using a network approach has the added affordance of machines being spaced over larger distances to synchronize media throughout the city for the future scaling of the project across multiple venues and spaces.

As a systems approach, we borrow concepts from networked music theory. For example, Bevilacqua et al. ¹ discuss their work with Local Area Networks in response to the COVID crisis for sound installations and performances. They develop infrastructure that allows an arbitrary number of clients to connect to the system and allow diverse levels of autonomy for agents to influence the media composition. In their view, the network is the key medium to support interaction at the level of the individual and collective, which forms the expressive potential of the work. Weinberg ¹⁰ defines and classifies the aesthetic and technical principles of such interconnected networks. Focusing on local area networks, he describes the centralized and decentralized topologies. He depicts a centralized synchronous network as a flower topology, where nodes are coordinated through a central hub. A decentralized network is then depicted as a star topology, where nodes are connected and independently manipulate each other’s output. A third network combines these two with weighted influence between nodes and a central server. Outlining architectures representing a flower topology, Swift ⁸, and Brown ² describe systems that offer accessible and engaging group experiences by affording audience participation.

Figure 3: Image of human Artist manipulating the screen segmentation and content

Most representing a hybrid double-star topology, ‘The Light Up Kelowna’ system is designed around a Conductor/Player/Artist relationship (Figure 4.). In this, a machine is designated as a Conductor who sends instructions from a score to one or more machines in the role of Players that receive and execute instructions. Players may communicate back to the Conductor, who can act upon this dialogue. An artist network node patches into the system in order to control Players video plane compositions and sequence the Conductor. With the ‘Light Up Kelowna System’, Players may be autonomous computer-based agents or audience members. A Conductor/Player topology may be explicit or ad-hoc, wherein in the former, a designer assigns the Conductor role, while in the latter, the role of the Conductor is negotiated between the machines in the system.

The Conductor has in their knowledge base a list of Players in the orchestra that serves the functions of how to sequence instructions and where to send these. The list of Players the Conductor manages may be static or dynamic. In the case of a static list, the system operator specifies the arrangement of players and their addresses to send instructions. Alternatively, a dynamic list is populated and edited by the Conductor as a task of its operation. In the dynamic mode, a Conductor will receive a notification from a Player who has joined or intends to leave the orchestra and will update the list accordingly. Another strategy is the Conductor can send periodic pings to Players who reply to their presence, and the Conductor updates its knowledge base on these responses. The ‘Light Up Kelowna’ system uses an explicit static Conductor/Player topology (outlined in Table 1). Altogether, the result of this approach is a synchronized networked urban screen inhabiting multiple collocated spaces open to audience/performer interactivity. In the following section, we describe some of the exhibitions displayed for ‘Light Up Kelowna’.

Figure 4: Graphic demonstrating the Light Up Kelowna Conductor/Player topology
Table 1: Conductor/Player topology typology

Out in the Wild

‘Light Up Kelowna’ improves access and inclusion for our community through art. This initiative adapts a public park and cultural area into a exhibit space, producing innovative and collaborative programming through digital storytelling exhibitions. ‘Light Up Kelowna’ enhances access to art for all community members with free digital art exhibitions in a public location. It involves and inspires the community through a sense of place and creates a place for all our members to participate in an art experience. ‘Light Up Kelowna’ creates audience experiences that are a personal journey of exploration and interaction, allowing visitors to social distance in a no-touch environment. This aligns with the requirements of a pandemic world and the increasing use of technology to connect people. Art and technology are now intertwined, and digital projections illuminate messages of hope, protest and community spirit.

Collective Bodies

The Collective Body (1) is a transdisciplinary audio/video project that proposes to capture the circumstances of life in isolation during the COVID crisis. Using current communications technologies as a central metaphor, the artists record themselves separately: dancers make videos of specific body parts, then musicians respond with recordings, which are sent to other dancers, who then respond with videos, which are sent to musicians, and the rhizome-like pattern of simultaneous emergence expands. The material is gathered, tagged and shaped into a haunting, shimmering collage of diverse bodies projected onto the exterior of a public building, with outdoor speakers providing sonic sensuality. In prioritizing visual and sonic expression over linguistic communication, which breaks the world into distinct, individual entities, we offer ways of being with others that simultaneously acknowledge differences and fundamental dependencies. The TCB project explores the constraints and unique possibilities of the digital connection while serving as a reminder of the deep importance and irreplaceability of shared physical space.

Celestial Bodies

Celestial Bodies (2) is a multicultural creation of animated media that tell ancient astrological stories, exploring the belief systems that make up Canadian and Indigenous society’s diverse fabric. The multimedia projection shows animated images of star stories— alongside world-class cross-cultural music. Media Artists Aleksandra Dulic, Miles Thorogood, Jacen Dennis, Amberley John, and Emerald Holt created a multimedia presentation of animated images. The artists reinterpret the cosmological stories and oral histories from their cultural heritages — Greek, Chinese, African, and the Indigenous’ Haudenosaunee culture, and explore the meeting of cultures in their collaborative process with community members, where unique stars signifying individuals’ heritage were made. Each story is connected to a season, and characters from the heavenly world travel through time and space as the night unfolds, highlighting diverse cultural beliefs.

Autolume Acedia

Autolume Acedia (3) is a hallucinatory meditation on the ancient emotion called acedia developed by creative A.I. artists Philippe Pasquier and Jonas Kraasch. Acedia describes a mixture of contemplative apathy, nervous nostalgia, and paralyzed angst. This emotion, first described by Greek monks two millennia ago, captures the paradoxical state of being simultaneously bored and anxious. Inspired and controlled by the music of Monobor, lost in winter soundscapes, the Autolume video generation system dreams about bodies, organs, and bones. Autolume is listening to the music to produce abstract imagery that seems to be dancing. A product of the latest Creative AI and Deep Learning algorithms from the Metacreation Lab, the piece is also a reflection on the analog and the digital and how they can meet to portray this emotion that has resurged during the current pandemic.

Student Works

As part of Light Up Kelowna’s art-dedicated urban screen projection in the Rotary Centre for the Arts window, UBC graduate students showcase their work (4). The exhibit was developed in the context of Graduate Studio in Visual Arts course that involves the critical analysis and production of independent artwork in various disciplines.

Figure 5: Image of Collective Bodies triptych exhibition on Light Up Kelowna.
Figure 6: Image of Celestial Bodies triptych exhibition on Light Up Kelowna.
Figure 7: Image of Autolume Acedia triptych exhibition on Light Up Kelowna. Copyright of Philippe Pasquier and Jonas Kraasch

Kaytlyn Barkved’s thesis show Neuroqueer Imaging features select digital drawings from her exploration of the unique emotional and sensory perceptions that Autists experience. Kaytlyn is is a queer disabled digital artist and Master of Arts candidate in the Digital Arts and Humanities theme of the Interdisciplinary Graduate Studies program at UBCO.

Sam Neal’s Inland Waters, 2021, captures the exploration of time, place and process. Sam collaborates with water bodies in the Okanagan using an early photographic process, cyanotype; a photographic process that utilizes UV light to create cyan-blue prints. He is a multi-disciplinary photographer, artist and Master of Fine Arts candidate at UBC.

Conclusion

‘Light Up Kelowna’ has successfully navigated the pilot phase with a triptych screen format using rear projection techniques displaying media in three parts in a storytelling format. The project creates opportunities for community members and audience participation and artist presentations. In this paper, we outlined the structure for coordinating engaged parties in developing scalable urban screen infrastructure. We outline several considerations necessary for installation of rear projection urban screens in existing city spaces. We go on to discuss the network architectures and topologies for creating networked urban screen systems, borrowing concepts from networked music performance and installation contexts. Finally, we demonstrate the use of our approach for developing an urban screen showcasing multiple exhibitions. ‘Light Up Kelowna’ is planned as a continuing, long-term exhibition series and is easily adapted to meet new community needs and social/artistic directions in the future.

Acknowledgments

We acknowledge that this work is located on the unceeded territory of the Syilx Okanagan people in what is now called Kelowna in British Columbia Canada. The Sylix people have lived and cared for this land for thousands of years and continue to be stewards of this place. We feel that there is much to learn from their world view and offer assistance to make a better future for all living things. We would like to thank, The University of British Columbia, Faculty of Creative and Critical Studies, and The City of Kelowna for finacial support.

Figure 8: Image of Neuroqueer Imaging by Kaytlyn Barkved. Copyright of the artist.
Figure 9: Image of Inland Waters triptych exhibition on Light Up Kelowna by Sam Neil. Copyright of the artist.

(1)https://thecollectivebody.ok.ubc.ca accessed:12/2022

(2)https://blogs.ubc.ca/celestialbodies/ accessed:12/2022

(3)https://metacreation.net/autolume-acedia/ accessed:12/2022

(4)https://blogs.ubc.ca/fccsartwork accessed:12/2022 

References

1 F. Bevilacqua, B. Matuszewski, G. Paine, N. Schnell, On designing, composing and performing networked collective interactions, Organised Sound 26(3), 2021, 333–339.

2 A. R. Brown, Network jamming: distributed performance using generative music, New Instruments for Musical Expression, 2010, 283–286.

3 J. Dubois, D. Colangelo, and  C. Fortin, Disrupting the city: Using urban screens to remediate public space, 2015.

4 S. Hannon, Urban renewal through media infrastructure: A case study of large screen development in dandenong, melbourne, In Communicative Cities and Urban Space, Routledge, 2020, 131–146.

5 B. U. Lubis, L. Primasari, The relationship between people and urban screen in an urban space, Procedia-Social and Behavioral Sciences, 2012, 42:223–230.

6 A. Rajah, ed., Art After Dark: 10 Years of Urban-Screen, Surrey Art Gallery, 2021.

7 M. Struppek, Urban screens–the urbane potential of public screens for interaction, In intelligent agent, volume 6, 2006, 1–5.

8 B. Swift, H. Gardner, A. Riddell, Engagement networks in social music-making. In Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction, 2010, 104–111.

9 M. Thorogood, M. Correia, A. Dulic, A networked multi-channel audio and video authoring and display system for immersive recombinatory media installations, In Proceedings of the 27th International Symposium on Electronic Art (ISEA), 2022.

10 G. Weinberg, Interconnected musical networks: Toward a theoretical framework, Computer Music Journal 29(2), 2005, 23–39.

Author Biographies

Miles Thorogood is an assistant professor of digital art in the Faculty of Creative and Critical Studies and heads the Sonic Production Intelligence Research and Applications Lab at The University of British Columbia. His current research aims to identify the facets of human perception used in creative processes to develop computational-assisted tools for art and design making.

Kirsteen McCulloch is the Executive Director of the Arts Council of the Central Okanagan. Born and raised in the Okanagan, Kirsteen recently returned to Kelowna after many years of living and working in the technology sector in Vancouver. Her love of all things creative, progressive, and boundary-breaking was fostered at an early age. In her role with ARTSCO, Kirsteen is returning to her roots as an arts advocate and community organizer.

Aleksandra Dulic is an artist-scholar with expertise in interactive art, climate change communication, and media for social change. She is the Director of the Centre for Culture and Technology (CCT) at The University of British Columbia. She leads an interdisciplinary research team that engages multiple forms of art, media and information technologies as vehicles for the expression of community, culture, and identity. 

Augmenting Creative Symbiosis Using a Cyber-Physical Aesthetic

Susanne Thurow,Dennis Del Favero, Michael Scott-Mitchell, Michael Ostwald, Helena Grehan

iCinema Research Centre @ The University of New South Wales (UNSW), School of Built Environment, UNSW, Murdoch University
Kensington, Australia; Perth, Australia
s.thurow@unsw.edu.au ; d.delfavero@unsw.edu.au ; m.ostwald@unsw.edu.au ; h.grehan@murdoch.edu.au

Abstract

Creative arts organizations are constituted by intricate symbiotic relationships between diverse stakeholders, including creatives, technicians, software programmers and producers. These thrive when facilitated by clear, seamless and swift communication. Cyber-physical systems can support such exchange yet require careful aesthetic design. Inspired by Manuel DeLanda’s philosophical framework, the paper introduces iModel, a new networked modelling system that facilitates operatic rehearsal design via an interactive cyber-physical spatial aesthetic. After mapping its conceptual framework, we detail iModel’s architecture and functionalities that provide a shared workspace for distributed teams. iModel is capable of assimilating diverse data formats and enabling real-time manipulation and seamless previsualization of complex interactions between all components of operatic production, including orchestration of set ensembles, video content, cast movements and lighting design, all synchronized with the musical score. The paper reflects on iModel’s real-world application in 2022 in the design and technical production departments at Opera Australia, Australia’s largest performing arts organization. We conclude with a reflection on the coalescence of philosophy, art and technology as a powerful conduit for catalyzing creative practice toward realizing new symbiotic imaginaries on and beyond the screen.

Keywords

Collaborative Design, Cyber-Physical Integration, Manuel DeLanda, Modelling System, Opera, Process Optimization, Real-Time Interaction, Rehearsal Design, Spatial Aesthetics, 3D Visualization.

DOI

10.69564/ISEA2023-88-full-Thurow-et-al-Cyber-Physical-Aesthetic

Introduction

Organizations in the creative arts are constituted by intricate symbiotic relationships between diverse stakeholders, including creatives, technicians, software programmers, producers and technology—who rely on each other’s input to develop and realize cohesive creative endeavors. The Covid-19 pandemic has disrupted these relationships by inhibiting the sharing of physical space, traditionally central to artistic practice. This has challenged the sharing of ideas and creative visions through embodied exchange. Attempts to mitigate the impact of lockdowns and their mandate of social distancing, have driven the accelerated adoption of newly emergent digital technologies to facilitate collaboration across dispersed locations ¹ . This embrace of cyber-physical engagement has induced a shift in the experience of space more broadly, initiating a new type of virtual yet visceral immediacy, where formerly the significance of geographical distance dominated ¹⁸. However, the promise of approximating and expanding the utility of presence and seamless interaction in virtual space still remains an aspiration whose full productive capabilities are yet to be realized ¹⁶. The forced rapid transition to digital collaborative tools and platforms among organizations from 2020 onward has painfully revealed the limitations that still exist in available hard- and software systems when it comes to facilitating and optimizing creative exchange across time and space as well as among diverse teams. This becomes obvious, for example, in lagging real-time collaboration features within networked environments, inability to preview diverse file formats in a unified visualization space, or lack of animation capabilities to orchestrate the intended interaction of separate design elements when coalesced into complex assemblages. Due to such limitations, established approaches are still unable to optimally support symbiotic relationships between technical and creative teams who rely on interdependent and reciprocal cross-pollination in ideation and prototyping processes. Consequently, teams struggle to unlock new horizons for collaboration and creative scope both in the virtual as well as in the—once again—accessible physical domain. At a time when the digital is shaping expectations for sophisticated visual content in all domains, creative sector organizations need to explore ways to enhance design ideation, production and evaluation using hybrid cyber-physical systems ¹⁷. In a cyber-physical system, “computational elements heavily interact with physical entities, thus controlling individual, organizational or mechanical processes” ²² . To advance the dormant capabilities of digital technologies for expanding creative collaboration in art and design practice as a symbiotic partnership, we believe it is necessary to investigate interactive cyber-physical aesthetics as a conceptual foundation for new digital platforms. Such an aesthetic has to conceive humans and technologies as equally ranked symbionts in order to be able to address the current limitations in cyber-physically enabled creative collaboration. Approaches such as that developed by Manuel DeLanda (1998) open avenues to transcend human-centered concepts by reformulating the world as a field of informational and material processes from which an infinite number of entities may be formed. This philosophical framework enables investigation of the diverse relations between people and technologies as malleable and open to intervention in ways that enhance symbiotic exchange.

In the following, we briefly outline elements of DeLanda’s philosophical framework that have informed the development of iModel, a networked modelling system that facilitates operatic rehearsal design via an interactive cyber-physical spatial aesthetic. After mapping its conceptual framework, we detail iModel’s architecture and functionalities that provide a shared workspace for distributed teams. iModel is capable of assimilating diverse data formats and enabling real-time manipulation and seamless previsualization of complex interactions between all components of operatic production including orchestration of set ensembles, video content, cast movements and lighting design, all synchronized with the musical score. This paper reflects on iModel’s real-world application in 2022 in the design and technical production departments at Opera Australia (OA), Australia’s largest performing arts organization. The paper concludes with a reflection on the coalescence of philosophy, art and technology as a powerful conduit for catalyzing creative practice toward realizing new symbiotic imaginaries on and beyond the screen.

Conceptual Baseline: Manuel DeLanda

By drawing on Gilles Deleuze and Fe ́lix Guattari’s readings of the works of Baruch de Spinoza and Henri Bergson, Manuel DeLanda develops a philosophical framework centered on the argument of ‘immanent morphogenesis’ ⁹, roughly translated as the inherent ability of entities to self- organize their development. By probing the question of both how matter comes to be and to assemble into innovative complex forms (such as bodies and technologies), DeLanda develops a theory of self-organization ¹³ that is premised on the existence of energetic flows, which are striving yet forever failing to reach an equilibrium state ⁵. The inherent drivers, or aesthetic codes, underpinning these processes are ‘virtual forms’ that trigger and galvanize evolutions of matter into ‘actual forms’ that present as phenomena to the human senses. As codes rather than blueprints, virtual forms do not predetermine the specific articulations of phenomena but constitute an ‘energetic possibility space’ that is radically open to intervention and evolution ². With flows being subject to heterogenous and conflicting influences, they are compelled by imbalances that call for harmonizing. Citing Deleuze, DeLanda (1998) captures these imbalances as caused by “differences of level, temperature, pressure, tension, potential, difference of intensity”, and increase energetic flow relative to their degree of disparity. Matter here exists in a “phase space, i.e., a space of possible states for each object” ³—with a ‘stable state’ representing the preferred state in which an object is maximally harmonized. DeLanda argues that, in the chaotic swirling of energetic flows, approximation of stable states is achieved more readily (but never fully realized) through coupling of simple forms into higher entities. This is because coupling of forms can cater to a set of dispositions shared among these forms, and represent an energetically more resilient, hence favorable, state. DeLanda conceives states along a continuum from ‘preferred stable states’, which he refers to as “attractors”, to least probable and volatile couplings, which he terms “repellors” ¹⁰. Attractors are thereby capable of generating new dynamic structures in higher entities through interaction of constituent parts, refracting energetic flows onto new trajectories that open capabilities for novel constellations with new, and ever more complex, balancing requirements. DeLanda captures such branching and its dynamic mutational qualities in the term ‘bifurcation’ ⁴. As Christian Hubert explains “bifurcations modify an entity’s tendencies and represent a source of creativity and variability in nature” ¹⁴. Once forms approach a stable state, they solidify into entities that “down-wardly select states of [their] component units that are [most] compatible with that global state” ¹¹. This embarking on an iterative design process reassembles and optimizes the cohesiveness of the higher entity ¹¹. DeLanda emphasizes that complex higher entities always unfold capabilities that exceed the limitations of their simple individual component parts ¹². He argues that these immanent aesthetic codes that govern the self-organizing production of matter (that is, its morphogenesis) underpin both organic and inorganic life ⁶. Consequently, he deduces that on a formative level, humans and technologies must be conceived as morphologically alike because they are part of an integrated “machinic phylum” (lineage or body plan) that synthesizes elements according to “similar self-organizing and combinatorial processes” ⁴. DeLanda here adopts terminology of Deleuze and Guattari who use the qualifier ‘machinic’ with the intent to highlight “the existence of processes [virtual forms] that act on an initial set of merely coexisting, heterogeneous elements, and cause them to come together and consolidate into a novel entity” ⁴. DeLanda refers to this consolidation as bifurcation that happens at the point of ‘singularity’, that is a tipping point at which new characteristics emerge that “chang[e] the forms through which human bodies and materials are combined, organized, deployed, and made effective” ⁷. He emphasizes that bifurcations are most productive if coupling occurs between forms that are heterogenous because this enables a diversity of possible trajectories into the future ⁴. It can give rise to processes that act as catalysts, intervening into processes and facilitating encounters between components that would otherwise not have occurred, channeling energy onto unexpected trajectories.

Application: iModel System


 DeLanda’s philosophy commends itself as a conceptual framework for creative collaboration, and especially in the operatic sector, because it allows conceptualization of stake-holders, ideas, technologies and materials as an interdependent web of symbionts that coalesce into an ecology whose scope expands or contracts in line with the degree of flow. As a group-based aesthetic practice, opera involves a spectrum of digital and physical components and engages an ensemble of creatives at a larger scale than other art forms. For example, components flowing into a production may include choreography, costumes, lighting, music, props, scenography, robotic LED screens and revolves – all developed by domain-specific artists (graphics, modelling, photography, robotics, video), designers (costume, lighting, production, prop), directors (musical, stage, technical), and implemented by diverse technical crew, cast and orchestra. As such, operatic production constitutes a symbiotic ecology in which each craft depends on the communication with and input of others to realize its purpose and to thrive, through reciprocal loops of collaboration, into a new higher entity, in this case the stage production ²¹. Conventional rehearsal design sees most creative team members develop their ideas in response to a production brief, mapping ideas in isolation, using incompatible analogue/digital platforms ¹⁹. The team then discusses concepts in development meetings, and makes amendments following these, again in relative seclusion. The combination of single ideas being transformed into complex assemblages that may yield a cohesive aesthetic for a production (approximating a Delandian stable state) hinges on the facilitation of productive communication among the diverse team. We argue that the more sensorily seamless this communication, the more readily attractors may emerge that enhance creative scope and complexity of design. The ability to combine heterogenous elements and design strands at various stages in the ideation, testing and evaluation phases can catalyze a process of cross-pollination and analysis that far exceeds capabilities of current stand-alone creative pipelines. If draft components can be assembled in fruitful, ongoing dialogue, bifurcations—resulting from conceptual and experimental tensions—can be identified and explored at length. Their resultant trajectories can feed into iterative design loops that may crystallize into more coherent and innovative operatic designs. Prerequisites for a system that can prompt creative singularities and act as a catalyst for the exploration of bifurcations, are:

We believe that additional synergies may be unlocked if such a system can be articulated to the human body in ways that enable a visceral investigation of the modelling space because human sense-making capabilities would be activated that exceed a solely visual contemplation.

Responding to this brief, an interdisciplinary team of artists, computer scientists, architects, theatre and 3D modelling experts at The University of New South Wales, led by Laureate Professor Dennis Del Favero, has been developing the iModel prototyping system in collaboration with OA. Their objective has been to probe in how far interactive spatial aesthetics and associated new technologies may support the collaborative design process as a self-organizing, positive symbiotic exchange between institutional departments and across the human-machine interface. By seamlessly integrating creative ideation and production pipelines via cyber-physical architectures, the project seeks to provide solutions for unlocking the dormant potential of OA’s strong investment into digital innovation—palpable, for example, in its pervasive use of large-scale modular LED screen assemblages. These require safe orchestration with physical props and cast as well as complex multi-screen mapping, which—in turn— intensifies the organization’s need for a powerful integrative pipeline that can increase the rapport between their teams and the available technological platforms. To address these challenges, the team behind iModel has been developing a cyber-physical spatial aesthetic that facilitates the modelling of an operatic design within a shared 3D virtual environment by means of dynamic joint human-machine decision making. It provides a creative ensemble with the ability to develop and evaluate design components in real time prior to stage rehearsal across diverse platforms, ranging from desktop computers and tablets to 360° 3D cinematic CAVE theatres ¹⁵. Using a dynamic visualization pipeline, team members can upload their domain-specific components into a shared modelling space where they can virtually validate designs before physically testing them on stage. This is facilitated through capabilities such as real-time collaborative composition and manipulation of components; playback of component operation, both individually and in combination, in time with the musical score; as well as AI-supported analysis of component composition and interaction, including collision, obstruction and line-of-sight detection. The categories of design information that can be included are: choreography, costumes, set designs (3D models and CAD files), directorial notes, lighting states and truss plans, musical score, stage robotics, props and screen content. To coalesce this information into a database from which unified visualization can be achieved, the team has scripted an asset conversion pipeline that uses the Nginx webserver in combination with the Celery queuing software to translate files on a static webserver from their native formats (for example, .json, .dwg or .fbx) into those readable by the iModel application. This is a prerequisite for visualizing the diverse data in a shared virtual space facilitated by the Unity 3D game engine. Once complete, asset bundles can be downloaded via the Django web app and imported into the iModel application onto a personal computer. This online pipeline leverages existing solutions and saves local computing power, enhancing the overall robustness and performance of the application. Drafting within iModel takes place within a 3D modelled theatre space, which can be either a generic venue or a digitally twinned real-world performance space. In the case of OA’s 2022 use of iModel for their touring production of The Barber of Seville, the twinned space is a meticulous reconstruction of Sydney Opera House’s Joan Sutherland Theatre and (separately) of the Drum Theatre in the city of Dandenong. The detail-rich 3D models reflect their respective architectural specifications, available stage machinery as well as textures of walls, floors and auditorium—providing the team with an accurate rendition of the performance space within which the final staging of the opera took place. This allowed closely customizing ideation and prototyping to the available real-world conditions of the theatre space. If accessed via a cinematic-scale CAVE theatre, the 3D model can be explored at life-size and provide a sense of inhabitable space during the design process—adding a visceral dimension that conventional approaches, for example physical scale models, cannot offer in terms of atmospheric assessment ²⁰. The iModel interface features separate workspaces that can be customized to the needs of various user groups, such as directors, set and lighting designers, or stage managers. Each workspace comprises functionalities that are particularly useful to a user group, reflecting common tasks and information needs. For example, a set designer may introduce a setting, center or grid lines over the stage view, specify the speed and rotation of a revolve, and import, manipulate, rotate and scale objects, such as set pieces or props. Their placement can be specified on a timeline (the so-called ‘sequencer’) that is coupled to the musical score, allowing users to preview an animation of their movement at any time. They can record or screenshot for later reference if they want to compare different versions and their possible creative bifurcations. The sequencer is keyframed to allow for greater ease when editing, allowing to jump for example between scenes and acts, or from an overture into an aria or obbligato. Content added to workspaces is saved to the same project within the iModel application, which enables the combined review of all workspaces in a master window that layers all saved content onto the common timeline and thus enables a virtual stage rehearsal. For closer inspection, users can pause, forward, rewind or zoom in and out of the sequencer. Remote collaboration between geographically dispersed users will be streamlined through an annotation feature that allows input as text, drawing or audio commentary linked to customizable keyframes in the timeline. While this facilitates design engagement for the entire creative and technical team, iModel offers artificially intelligent features that support evaluation in terms of occlusions, potential hazards, path tracing and wayfinding for movable objects and cast members. 

Figure 1: iModel Software Interface ©iCinema Centre

During the development of OA’s Barber production, the iModel system enabled the creative and technical teams to achieve significant synergies and efficiencies across the entire design pipeline, enabling seamless visual communication especially in relation to the complex stage movements of props, set pieces and cast members. As Barber was conceived as a touring production for over 20 locations across Australia—ranging from expansive, formidably equipped to very minimalist venues-, a design had to be delivered that would aesthetically resonate across locales as well as pragmatically pack down for quick venue changeover—all to be achieved within tight budgetary constraints in the aftermath of the Covid-19 lockdowns. The solution to these challenges was found in an intricate modular set design, whose elements cast members would deconstruct and maneuver across the stage in choreographed sequences, aided by locally recruited choruses that had to be inducted on the fly once the crew arrived at a new location. The set elements required custom-built travelling cases, which were also designed using the iModel application as it allowed precise articulation of dimensions and formats. The iModel system provided the much-needed capability to virtually trial and evaluate different draft designs across three destination theatres whose spatial and technical resources required strongly divergent design provisions. Through visceral exploration of the 3D-rendered architectural spaces as well as visual manifestion of designs at 1:1 scale, the teams were able to rapidly achieve a shared understanding of possibilities and constraints and to direct their mental resources more readily onto feasible design trajectories. Rather than getting caught up in lengthy verbal or textual explanations of spatial concepts, team members used the iModel system to rapidly block out space and to trace movements of set pieces and bodies across the stage. This allowed identifying bottlenecks and preempting collision pathways, as well as adding playful complexity to the orchestration of the swiveling stage action that closely integrated with the musical score—amplifying its tumultuous and joyous pace. With costume designs ingested into a workspace, each character could be displayed in at least one attire. The designs were ingested as .png files (i.e. 2D images) attached to 3D boxes that would provide them with depth in the 3D space. Raytracing allows light to bounce off them and cast shadows, which aides in integrating the avatars more fully into the visual concept.

Figure 2: iModel Sightline View ©iCinema Centre

Grouping the characters together on stage, and animating their movements synched to the orchestration of set pieces, enabled assessment of their aesthetic effect. In response, textures were refined and details added that enhanced the overall composition and which imbued characters with idiosyncratic flair. Most importantly, the iModel system facilitated maximally clear communication of ideas and enabled rapid evaluation and iteration of concepts. The team thereby made full use of the range of visualization platforms, starting out on desktop computers connected to large wall-screens for initial prototyping, importing designs into a 360-degree cinematic CAVE theatre for 1:1-scale team review, and lastly using the system on laptops in the rehearsal room to annotate and refine design aspects that could be shared with cast and crew. With pandemic disruptions continuing to affect the availability of personnel and resources, iModel provided capability to convey complex design ideas across time and space, allowing remotely located team members to review and annotate files on their personal devices, which helped to stay on schedule for delivery.

Conclusion

Through its provision of a shared environment, visual integration of design streams and real-time interaction, iModel has proven its capability to productively intervene into design processes at OA, streamlining—while at the same time—enhancing complexity and creative scope of output. It does so by supplying a catalyst that enables creative bifurcations through the coupling of human creativity and cyber-physical computational capabilities. It supports the ideation and analysis in ways that do not prescribe form but enable self-organization of processes with open outcomes. In doing so, it facilitates a redrawing of the relationship between makers and technologies, opening pathways for conjunctive symbiotic exchange not just between creatives but also between humans and machines. With ubiquitous digital technologies increasingly reassembling workflows and imaginations, such a partnership—if facilitated in the right ways—can yield a creative force that deeply fuses human, computational and mechanical systems ⁸. The crosspollination between the arts and technology can thereby furnace new modes of engaging the machinic phylum that open up multiple bifurcations for imagining and shaping our planetary futures as collective and symbiotic rather than individual and isolated entities.

Acknowledgements

This research has been supported by the Australian Government through the Australian Research Council’s Linkage funding scheme (LP190100563).

References

1 Australia Council for the Arts, Digital Culture Strategy 2021-24, Canberra: Federal Government of Australia, 2020, p.3.

2 M. Delanda, and C. Cox, Possibility spaces. in c. cox, j. jaskey and s. malik (eds.), Realism Materialism Art Berlin: Sternberg Press, 2015, p.88.

3 M. Delanda, G. Harman, The Rise of Realism. Cambridge: Polity Press, 2017, p.73.

4 M. Delanda, 1997, The machinic phylum, TechnoMorphica, available at V2, accessed August 25, 2023, https://v2.nl/archive/articles/the-machinic-phylum.

5 M. Delanda, 1998a, Deleuze and the genesis of form, Art Orbit 1 March, accessed August 25, 2023. https://www.cddc.vt.edu/host/DeLanda/pages/genesis.htm.

6 M. Delanda, War in the Age of Intelligent Machines, New York, Forth edition, Zone Books, 1998b, 7.

7 M. Delanda, War in the Age of Intelligent Machines, New York, Forth edition, Zone Books, 1998c, p.Backmatter.

8 M. Delanda, War in the Age of Intelligent Machines, New York, Forth edition, Zone Books, 1998d, 231.

9 M. Delanda, Deleuze, diagrams, and the genesis of form, American Studies, 2000, 45.1:34.

10 M. Delanda, MaterialistPhenomenology.A Philosophy of Perception, London, Bloomsbury, 2022a, 23.

11 M. Delanda, Materialist Phenomenology. A Philosophy of Perception, London, Bloomsbury, 2022b, 36.

12 M. Delanda, Materialist Phenomenology. A Philosophy of Perception, London, Bloomsbury, 2022c, 29.

13  R. Dolphijn, I. Van der Tuin, eds, New Materialism: Interviews and Cartographies, Ann Arbor, Open Humanities Press, 2012, 43.

14 C. Hubert, 2019, Machinic phylum, tian Hubert Studio, accessed August 25, https://www.christianhubert.com/writing/machinc-phylum, Chris-2023.

15  L. Maftei, C. Harty, Designing in caves. using immersive visualisation in design practice, International Journal of Architectural Research, 2015, 9.3:73.

16 G. Y. N. O’Dwyer, A. Smolic, Xr ulysses: Addressing the disappointment of cancelled site-specific re-enactments of joycean literary cultural heritage on bloomsday, International Journal of Performance Arts and Digital Media, 2022, 18.1:43.

17 H. M. Nugent, A. McKinnon, K. Fagg, M. Oxenbould, 2016, National opera review final report, Department of Communications and the Arts, Canberra: Commonwealth of Australia, accessed August 28, 2023, p.5, https://shorturl.at/dewB5.

18 S. Reddy, 2021, Designing a digital future, The New York Times website, accessed August 25, 2023. https://www.nytimes.com/2021/12/03/special-series/covid-technology-digital-space.html.

19  B. Simonson, The Art of Rehearsal, ebook, Bloomsbury, 2017, p.6.

20 S. Thurow, D. Del Favero, L. Wallen, Inhabitable models – immersive intelligent aesthetics for scenographic design, Theatre and Performance Design, 2021, 7:92.

21 P. Torpey, Media Scores: A Framework for Composing the Modern-Day Gesamtkunstwerk, Cambridge, PhD Thesis, Massachusetts Institute of Technology, 2023, p.17.

22 C. Tsigkanos, T. Kehrer, C. Ghezzi, Architecting dynamic cyber-physical spaces, Computing, 2016, 98:1012. 

(Re)imagining human-yeasts relations via art-science collaboration

Olga Timurgalieva1, Patrícia Moreira2, Eva Direito2

1 City University of Hong Kong, 2 Catholic University of Portugal
Hong Kong, Lisbon 
otimurgal2-c@my.cityu.edu.hk, prmoreira@ucp.pt, s-edireito@ucp.pt

Abstract

More-than-human beings are largely de-animated in Western cultures and perceived of as the backdrop for human activities. In particular, with modern modes of production and consumption, customers often have no idea how specific products have been created and which more-than-human beings have been involved in manufacturing specific consumables. As a partial response to this problematic disconnect, this text presents the art book, Yeasts as We Do Not Know Them, as a means to learn about human-yeast interactions and the ways these fungal microbes are used to manufacture different products and substances. The book, therefore, (re)imagines yeasts as omnipresent, diverse, and symbiotic. Conceiving of symbiosis as a set of interspecies relations, including mutualistic, pathogenic, and commensal ones, the project, Yeasts as We Do Not Know Them, as this article argues, maps diverse interspecies interactions and, by doing so, provides recourse for navigating material systems, thereby invoking trans-corporeal ethics.

Keywords

Human-Yeast Relations; Art-Science Collaboration; Transdisciplinary Research; Symbiosis; Mutualism; Climate Change.

DOI

10.69564/ISEA2023-89-full-Timurgalieva-et-al-human-yeasts-relations

Introduction

Contemporary modern production and consumption practices are based on the general compartmentalization of economy, culture, and nature. ¹ That is, despite ecological concerns, diverse forms of life are largely de-animated and instrumentalized in Western cultures. ² While consumption and waste production levels in many regions are incredibly high despite myriad alarming environmental issues, customers are often unaware of the sources of specific products and from which living and non-living materials they are made. In response to such circumstances, scholars from diverse disciplines tackle the problematics of climate change, with a rising number of researchers emphasizing the role of art in developing ecological sensibilities, aesthetics, and imaginaries. ³ , ⁴ Such views derive from the assumption that climate change issues arise from and influence diverse aspects of human and nonhuman lives and cannot be tackled merely within the environmental sciences and adjacent fields. Rather, as researchers in environmental humanities Astrida Niemanis, Cecilia Åsberg, and Johan Hedrén have pointed out, “any policy or action aimed at ameliorating environmental problems must take into account human desire, motivation, and values; a deep understanding of the environment cannot be divorced from human imagination, culture, and institutional and social practices.” ⁵ In other words, dealing with climate crises cannot be limited merely to environmental sciences. Instead, in order to face immense challenges, significant changes are needed within distinct yet deeply interconnected economic, industrial, cultural, social, institutional, political, conceptual, visual, and other realms. Hence, instead of compartmentalization, environmental humanities require transdisciplinary methodologies and practices.

For these reasons, art book, Yeasts as We Do Not Know Them, which is the focus of this paper, rejects compartmentalization and instead takes the concepts of symbiosis and trans-corporeality as a starting point, mapping out some of the yeast interspecies relations. Created by the authors of this article, Yeasts as We Do Not Know Them depicts human-yeast interactions as diverse, omnipresent, and symbiotic. We use the concept of symbiosis to describe human-yeast relations as sometimes mutualistic, sometimes commensal, and sometimes even pathogenic. Trans-corporeality, according to American literary scholar Stacy Alaimo, is a mode of thinking that regards beings and entities as porous and materially and discursively interrelated. ⁶ By presenting ways in which yeasts are interconnected with diverse life forms and are involved in the industrial production of food, medicine, chemicals, and cosmetics, the book invites the audiences to learn about human-yeast material interchanges and to consider the repercussions of these interchanges. In mapping some of the actual interspecies interactions, our project, as we argue in this text, invokes trans-corporeal ethics.

In order to develop this argument, we begin by discussing symbiosis and trans-corporeality as guiding concepts in the project. Further, we elaborate on the connection of the art book to these concepts via two main threads. First, we trace yeasts across scales on the level of bodies and in the production of various products and consumables. Second, we emphasize the importance of transdisciplinarity in creating this artistic publication.

Symbiosis and trans-corporeality

Nowadays, most biologists define biological symbiosis as a close (in direct contact) relationship between individuals of two species, either with positive, negative, or neutral outcomes. ⁷ According to this definition, parasitism, mutualism, and commensalism are all symbiotic relations. While several generations of scientists have widely debated this notion of symbiosis, currently, the understanding of symbiosis as interspecies relationships of both mutualist and non-mutualist character is, for the most part, accepted and widely circulated via undergraduate biology books.

One of the most prominent and well-studied examples of mutualistic symbiosis is the two-species association with lichens, which was first described in the 19th century. Lichens display a win-win relationship with filamentous fungi and either cyanobacteria or microalgae. The fungi species benefit from photosynthesis products such as simple sugars produced by cyanobacteria or microalgae. In turn, the latter grow, embedded in a mycelium structure, which protects them from dehydration and environmental stress. Another well-studied example of mutually beneficial relationships between two distinct species is the cooperation between mycorrhizal fungi and plants. In their interactions, plants fix carbon from the air through photosynthesis and provide the fungi with simple sugars. In response, the fungi help the plants extract minerals from the soil.

The notion of symbiosis is closely connected with symbiogenesis or endosymbiotic evolutionary theory. This theory explains the origin of more complex cells—eukaryotic cells—which constitute the bodies of animals, plants, and fungi. ⁸ Endosymbiosis describes the relationship between two species where one organism lives inside the cell/cells (either for unicellular or multicellular organisms) of another. The theory suggests that, between 2500 and 1800 million years ago, a bigger cell engulfed a prokaryotic one (bacteria) and, instead of processing it as food, integrated the prokaryote into itself. In the process of incorporating the prokaryote, the bigger cell gained the ability to generate energy thanks to prokaryotic mitochondria and perform photosynthesis thanks to prokaryotic chloroplasts. Thus, instead of consuming the prokaryote, the host cell negotiated a functional relationship of symbiosis. Present-day examples of endosymbiosis include paramecium Paramecia bursaria, commonly known as infusoria, a single-celled freshwater being covered in cilia. Paramecia bursaria harbors several hundred single-celled green algae Chlorella spp. cells in the cytoplasm that perform photosynthesis for simple sugar production. ⁹

In recent years, with the development of ecological thought, ecoart, bioart, and environmental humanities, the concepts of symbiogenesis and symbiosis have traveled far beyond biological sciences. As the discourses concerning climate crises and mass species extinctions alarmingly stress the catastrophic devastation facilitated by climatic disasters, artists and scholars in the humanities broadly employ the notions of symbiogenesis and symbiosis in their respective work. They increasingly emphasize interrelations and cooperation between diverse life forms as opposed to individualistic perspectives and conceiving of bodies as separate from the environment.

Perhaps one of the most prominent examples of engaging with symbiogenesis theory in the humanities is the work of technoscience scholar Donna Haraway. Drawing on the biological concept of endosymbiosis, Haraway has highlighted the interconnectedness of diverse organisms and entities. ¹⁰ Instead of considering living beings as bounded “self-organizing individual units,” the author stresses their “multispecies becoming-with” and their constant mutual co-influences on each other. ¹¹, ¹²  Haraway regards such interrelations and interdependent sensibilities as crucial for establishing more ecologically-minded practices of living on a damaged planet.

Practitioners and academics in the visual arts also refer widely to the concepts of symbiosis and symbiogenesis to emphasize the constant interrelations between living beings. As artist and writer Claire Pentecost puts it, multiple contemporary artists create “symbiotic art”; that is, they expose the interrelations between diverse life forms and, in this way, help us to recognize “a world of relations” and our dependencies on manifold living beings. ¹³ When talking about creative projects developed using microorganisms, artist Ken Rinaldo emphasizes their “symbiotic aesthetics” as the means to draw the attention of broader audiences to the complexity of natural living systems and their impact on environmental health. ¹⁴ Overall, numerous researchers in the visual arts and (post)humanities engage with the concepts of symbiogenesis and symbiosis as making-with instead of considering bodies and entities bounded. ¹⁵

Similarly, Stacy Alaimo regards bodies not as contained but rather as trans-corporeal, that is, porous, not existing as preceding entities but constantly co-influencing each other. ¹⁶ As Alaimo argues, such a relational consideration of beings and entities requires a different kind of ethics—a trans-corporeal ethics. Indeed, trans-corporeal ethics arises from the obligations to “inquire about all of the substances that surround us, those for which we may be somewhat responsible, those that may harm us, those that may harm others, and those that we suspect we do not know enough about.” ¹⁷ As the author posits further, this approach to ethics “calls us to somehow find ways of navigating through the simultaneously material, economic, and cultural systems that are so harmful to the living world and yet so difficult to contest or transform.” ¹⁸ In other words, considering things, bodies, and entities as porous and permanently affected by their symbionts and co-existing materialities requires understanding the material interchanges across beings and entities and, if possible, lessening the impact of harming interchanges.

Informed by these conceptualizations of symbiosis, symbiogenesis, and trans-corporeality, Yeasts as We Do Not Know Them engages with them on various levels: 1. the symbiotic natures of the yeasts co-habiting with and within other living beings; 2. in more metaphorical terms, human-yeast “symbiotic” relations in the production of various products and consumables and, finally, 3. the interdisciplinary “symbiotic” collaboration between a biotechnological engineer, designer, and art researcher aimed at creating the project. Depicting human-yeast relations as symbiotic, the book aims to map material interchanges across scales and species in the spirit of trans- corporeality, thereby demonstrating how yeasts interact with humans across multiple scales.

Mapping yeasts across scales

Yeasts—unicellular organisms that belong to the Fungi kingdom—play a vital role in the Earth’s biosphere. While scientists have described around 100,000 fungi, they estimate that there are 1.5 million different species of fungi in total, in which only around 1 percent might be yeasts. As of now, researchers have described around 1,500 yeast species and found that these microbial fungi exist in every single habitat and ecosystem: from terrestrial to aquatic ones and from tropical ecosystems to more extreme niches such as Antarctica, where the fungi present specific adaptations to diverse environmental factors and ecosystem dynamics. Yeasts play an essential role in those habitats, being involved in, among other processes, the decomposition and recycling of living and non-living matter. At the same time, some of them are related to diseases in plants, animals, and even tiny protists such as paramecium, widely known as ameba.

Figure 1. Page 54 and 55 of the book, Yeasts As We Do Not Know Them, related to Cryptococus neoformans infection, 2022. ©Eva Direito, Patrícia Moreira, Olga Timurgalieva. 

Single-celled yeasts require magnification to be visible to the naked eye if not grouped in colonies. In response to their general invisibility to humans, the book, Yeasts as We Do Not Know Them, helps to imagine their omnipresence and diversity. It conceives of bodies and entities as microbial, presenting material entanglements between yeasts and humans and several examples of yeast relations with plants, animals, and other microbes. ¹⁹ For instance, Candida albicans and Candida auris are widely known for being related to human and animal infections. Cryptococcus neoformans and Exophiala dermatitidis are less known to a broader audience but related to invasive infections (Figure 1). So-called “wild” yeasts live in every environmental biome of the planet, including oceans, the stratosphere, glacial ice, bodies of animals, plants, fungi, algae, and microorganisms. Although not necessarily depicting the tiny fungal cells, the publication locates schematically yeasts’ symbiotic relations, including commensal, opportunistic, and pathogenic ones.

Additionally, the printed collection displays various examples of human-yeast interactions involved in the manufacturing of numerous consumables. The immense and diverse metabolic powers within yeasts and their presence in all habitats and regions have led humans in various geographical locations to unknowingly use yeasts in fermentation to produce beer, bread, kombucha, soy sauce, and other food and beverages (Figure 2). These practices have been passed down from generation to generation through millennia. In contemporary societies, however, such ancestral uses of biotechnology have been significantly optimized, while the fundamentals of producing some of the most common drinks and snacks remain the same. Yeasts as We Do Not Know Them features several historical records of the use of yeast and numerous contemporary examples. 

Figure 2. Production of rice liquor (mijiu) and Guangxi snake liquor (shejiu), illustrated in the fourth volume herbal, Shiwu bencao (Materia dietetica), 1368-1644. ©Wellcome Collection.

 

In recent centuries, scientific developments in biotechnology have permitted scientists to expand the metabolic toolkit of yeasts further and apply them in the manufacture of complex products such as pharmaceutical compounds, biofuels, cosmetics, and other chemicals, as well as bioremediation solutions. Additionally, yeasts are widely used in pharmaceutical research and experiments to produce a host of chemical substances and mixtures. Yeasts as We Do Not Know Them provides numerous examples of such products.

Although the book maps interspecies yeast relations across the scales of micro- and macroscopic bodies and the diverse ways in which yeasts are used in the production of numerous consumables, we do not intend to collect all the instances of human-yeast interactions with all the known microbial fungi. Instead, the art book functions as an invitation to learn more about some of these microbes and their interspecies material interchanges.

We do not regard this project as a mere celebration of human-yeast entanglements. We rather seek to encourage the readers to consider the impact of these interrelations and whether or not they can be harmful in some way. As geographer and sociologist Bram Büscher has pointed out, some aspects of human-nonhuman interactions involved in consumables production might harm ecological systems or be associated with social inequalities. For this reason, such manufacturing practices need to be revised or even “unmade,” argues the author.²⁰, ²¹  Yeasts as We Do Not Know Them displays various examples of yeast applications and, by doing so, invites the viewers to contemplate which of the human-yeast relations might need to be remade or unmade. In this way, the project invokes trans-corporeal ethics.

Transdisciplinary synergies

The idea of Yeasts as We Do Not Know Them has grown from a transdisciplinary course called “Gender and Sustainability–Introducing Feminist Environmental Humanities” offered by the KTH Royal Institute of Technology and Linköping University. This course gathered artists, scientists, architects, anthropologists, philosophers, and researchers in literary studies, art criticism, ecology, sustainability, and feminist and gender studies. Course participants discussed numerous complex challenges of the global ecological crises. They studied some of the theoretical instruments and methodologies that can be applied in dealing with global environmental issues.

In the spirit of the course, Yeasts as We Do Not Know Them, functions as a form of trans-disciplinary practice-oriented inquiry. More specifically, the book was developed by a biotechnologist Patrícia Moreira, a designer Eva Direito, and a researcher in bioart and environmental humanities Olga Timurgalieva. The cross-pollination between the three disciplines has allowed us to gather crucial information about the microorganisms and their application in biotechnological research and industrial production (microbiology), establish a conceptual framework to engage with interspecies relations creatively (bioart and environmental humanities), and, most importantly, to develop and apply the visual means with which to communicate our ideas (design). Combining this diverse disciplinary knowledge permitted us to learn about human-yeast ecologies as complex interspecies relations and translate what we learned into an artistic project.

As a product of transdisciplinary collaboration, Yeasts as We Do Not Know Them became a material embodiment of “symbiotic” relations between art and science. The book’s visuals and content reflect the interplay between the disciplines and the typical clichés associated with them. The project alludes visually to the style of a modern encyclopedia and simultaneously to a catalog of products for sale. As the primary reference for developing the images and graphics for our collection, we looked at the book, Pig 05049 (2007), which was created by artist Christien Meindertsma and designed by Julie Joliat. 

Pig 05049 compiles dozens of everyday consumables produced from a single source—a pig numbered 05049. ²² The collection of consumables is divided into chapters, each devoted to a specific body part or bodily substance of the pig from which the products were made such as blood, skin, bones, fat, and so forth. Each product page features a photo illustration, a brief description, and an icon that refers to a type of consumable. Icons and chapters structure the book, thus referring to indexing conventions and highly organized scientific knowledge. At the same time, non-scientific symbols, icons, and names of body parts and substances, as well as the cover alluding to swine skin with the identification tag reveal the playful character of the book.

In a similar vein, Yeasts as We Do Not Know Them is quasi-encyclopedic and simultaneously calls to mind the glossy magazines aimed at advertising products and services. The publication comprises almost seventy pages presenting human-yeast interrelations (in its current prototype version; we aim to extend the book to 150 pages). Divided into six sections (research, body, drinks, food, industrial, and material), the printed collection provides scientific names of yeast species and brief descriptions of the way yeasts are used in a specific product or research area or are related to human bodies. Additionally, the pages with the excerpts from historical records (as mentioned above) provide insight into the past and show how yeasts were used in previous centuries (Figure 2). Thus, the book offers a wealth of factual information and some historical references.

At the same time, however, the accompanying photographs are not meant to illustrate yeasts, yeast-related diseases, and biotechnologies scientifically; instead, the visuals are playful (Figure 3, 4). With their colorful backgrounds and brightness, they depict objects connected with specific microbial fungi and the products and services produced with the involvement of these organisms. The making of such images was inspired by the work of photographers Colin Ross and Suzanne Saroff, who created bright still lifes with ambiguous spatial atmospheres and optical illusions. The photographers created these illusions with the help of water or a magnifying glass, which helped to distort light and alter shadows. ²³, ²⁴ Borrowing some of these visual effects, the images in Yeasts as We Do Not Know Them attract the readers’ attention and invite them to think about the objects presented in the book from a new perspective.

Additionally, the zine incorporates both scientific classification systems as well as lay names and random groupings of objects and processes. For instance, we use the scientific names of the yeasts and refer to classification systems, such as yeast families in the descriptions of yeasts and yeast biotechnologies. Simultaneously, we loosely divide the examples of yeast-human interactions into research, body, drinks, food, industrial, and material categories and present them in random order. With such decisions, we aimed to format factual information in a simple and easily graspable way.

Figure 3. Page 11 of the book, Yeasts As We Do Not Know Them, related to yeast, Pichia pastoris, studied to develop vaccines against Human papillomavirus infection, 2022. ©Eva Direito, Patrícia Moreira, Olga Timurgalieva. 
Figure 4. Page 21 of the book, Yeasts As We Do Not Know Them, related to recombinant yeast, Saccharomyces cerevisiae, applied to produce xylitol, widely used as a sugar substitute, 2022. ©Eva Direito, Patrícia Moreira, Olga Timurgalieva. 

The title of our project refers to the famous bioart book, Art As We Don’t Know It, which explores the ways in which contemporary artists engage with material culture and the natural sciences, working across laboratories, art studios, and in the field to produce their transdisciplinary projects.

²⁵ Our zine was particularly inspired by the section of the book titled, “Life as We Don’t Know It.” This section refers to the most recent developments in synthetic biology (as an area of biological research focused on (re)designing natural systems) and generally the reconceptualization of notions of life. ²⁶ Yeasts as We Do Not Know Them involves examples of yeast bioengineering and the most recent applications of transgenic yeasts in different research areas. At the same time, the book also aims to represent a variety of yeasts, their habitats, and applications, thus aspiring to expand common knowledge about these microbes.

Overall, the project was developed via what one may call a “symbiotic collaboration” between the practitioners from three disciplines. A biotechnologist, a designer, and a researcher in bioart, working together, shared their knowledge and skills to create a cross-disciplinary publication about human-yeast relations.

Conclusion

The challenge of dealing with complex, urgent problems such as climate change has led to the development of transdisciplinary approaches that transcend the boundaries of environmental science, moving towards intensified collaboration between the arts, sciences, and humanities. A consortium of authors with very different academic backgrounds was thus involved in the creation of Yeasts as We Do Not Know Them.

In developing the book, we engaged with the notion of symbiosis as more-than-mutualistic interspecies relations and attempted to invoke trans-corporeal ethics by mapping some of the material interspecies interchanges. The project showcased diverse interspecies connections focusing on the microscopic single-celled organisms known as yeasts. Throughout the book, examples of ancestral, contemporary, technological, mutualistic, and hazardous connections with yeasts allow the audiences to decide how to define symbiosis and what kind of relationships with yeasts are possible. In this way, Yeasts as We Do Not Know Them invites the broader public to navigate the material systems of human-yeast relations and contest potentially harmful material interchanges. Just such an approach to our material systems is crucial in times of ecological crises.

Acknowledgments

The authors would like to acknowledge the financial support from the Foundation for Science and

Technology (FCT) through National Funds by project UIDB/0622/2020 (CITAR).

References

1 Astrida Neimanis, Cecilia Åsberg, Johan Hedrén, “Four Problems, Four Directions for Environmental Humanities: Toward Critical Posthumanities for the Anthropocene”, Ethics & the Environment 20 (1), 2015, 67–97.

2 Arianne Conty, “Animism in the Anthropocene,” Theory, Culture & Society 0 (0), 2021, 20, doi:10.1177/02632764211039283.

3 Giovanni Aloi, “Editorial,” Antennae. The Journal of Nature in Visual Culture, 2022, 10–11.

4 Neimanis, Åsberg, and Hedrén, “Four Problems.”

5 Neimanis, Åsberg, and Hedrén, “Four Problems,” 80.

6 Stacy Alaimo, Bodily Natures: Science, Environment, and the Material Self, Bloomington, Indiana University Press, 2010. 

7 Lisa A. Urry, Michael L. Cain, Steven A. Wasserman, Peter V. Minorsky, and Jane B. Reece, Campbell Biology, Pearson Education, 2017.

8 Urry Cain, Wasserman, Minorsky, Reece, Campbell Biology.

9  Yuuki Kodama, Masahiro Fujishima, “Secondary Symbiosis Between Paramecium and Chlorella Cells,” International Review of Cell and Molecular Biology 279, 2010, p.33.

10 Donna Jeanne Haraway, “Sympoiesis Symbiogenesis and the Lively Arts of Staying with the Trouble,” in Staying with the Trouble, Making Kin in the Chthulucene, Durham and London, Duke University Press, 2016, 58–98, https://www.jstor.org/stable/j.ctv11cw25q.8.

11 Haraway, “Sympoiesis Symbiogenesis and the Lively Arts of Staying with the Trouble,” p.61.

12 Haraway, “Sympoiesis Symbiogenesis and the Lively Arts of Staying with the Trouble,” p.63.

13 Claire Pentecost, “Symbiotic Art,” Art in America (1939), (2022), Vol. 110(2), p.43.

14 Ken Rinaldo, “Can Microbes Save Us from Ourselves?” Antennae, The Journal of Nature in Visual Culture, 2022, 12– 16.

15 Anna Pomyalova, “Catching Signals from a Vibrant Environment,” Antennae: Journal of Nature and Visual Culture, 2022, 254–62.

16 Alaimo, Bodily Natures : Science,Environment, and the Material Self.

17 Alaimo, Bodily Natures: Science, Environment, and the Material Self, 18.

18 Alaimo, Bodily Natures: Science, Environment, and the Material Self, 18.

19 Funke Iyabo Sangodeyi, “The Making of the Microbial Body, 1900s-2012,” Ph.D. diss., The Department of the History of Science, Harvard University, 2014.

20 Rosemary-Claire Collard, “Putting animals back together, taking commodities apart,” Annals of the American Geographers, 104(1), 2014, 151–165, quoted in Bram Büscher, “The nonhuman turn: Critical reflections on alienation, entanglement and nature under capitalism,” Dialogues in Human Geography, 12(1), 54–73, 2022, 56, https://doi.org/10.1177/20438206211026200

21 Giuseppe Feola, “Degrowth and the unmaking of capitalism: beyond ‘decolonization of the imaginary’,” ACME, 18(4), 2019, 977–997, quoted in Bram Büscher, “The nonhuman turn: Critical reflections on alienation, entanglement and nature under capitalism,” Dialogues in Human Geography, 12(1), 54–73, 2022, 56, https://doi.org/10.1177/20438206211026200

22 Christien Meindertsma, “Pig 05049,” accessed December 8, 2022, https://christienmeindertsma.com/PIG-05049

23 Colin Ross, “Cross Sections,” accessed December 8, 2022, https://www.colinrossphoto.com/#/cross-sections/

24 Suzanne Saroff, “Artwork,” accessed December 8, 2022, https://www.hisuzanne.com/

25 Erich Berger, Kasperi Mäkireinikka, Kira O’Reilly, Helena Sederholm, Art As We Don’t Know It, Estonia, Aalto University, 2020.

26 Erich Berger, Kasperi Mäkireinikka, Kira O’Reilly, Helena Sederholm, Art As We Don’t Know It, Estonia, Aalto University, 2020, p.11.

Bibliography

Stacy Alaimo, Bodily Natures: Science, Environment, and the Material Self, Bloomington, Indiana University Press, 2010.

Giovanni Aloi, “Editorial,” Antennae. The Journal of Nature in Visual Culture, 2022, 10–11.

Erich Berger, Kasperi Mäkireinikka, Kira O’Reilly, Helena Sederholm, Art As We Don’t Know It, Estonia, Aalto University, 2020.

Bram Büscher, “The Nonhuman Turn: Critical Reflections on Alienation, Entanglement and Nature under Capitalism”, Dialogues in Human Geography, 12 (1), 56, doi:10.1177/20438206211026200.

Rosemary-Claire Collard, “Putting animals back together, taking commodities apart,” Annals of the American Geographers, 104(1), 2014, 151–165.

Arianne Conty, “Animism in the Anthropocene,” Theory, Culture & Society 0 (0), 2021, 1-27, doi:10.1177/02632764211039283.

Giuseppe Feola, “Degrowth and the unmaking of capitalism: beyond ‘decolonization of the imaginary’,” ACME, 18(4), 2019, 977–997.

Donna Jeanne Haraway, “Sympoiesis Symbiogenesis and the Lively Arts of Staying with the Trouble,” in Staying with the Trouble. Making Kin in the Chthulucene, Durham and London, Duke University Press, 2016, 58–98, https://www.jstor.org/stable/j.ctv11cw25q.8.

Yuuki Kodama, Masahiro Fujishima, “Secondary Symbiosis Between Paramecium and Chlorella Cells,” International Review of Cell and Molecular Biology 279, 2010, p.33.

Astrida Neimanis, Cecilia Åsberg, and Johan Hedrén, “Four Problems, Four Directions for Environmental Humanities: Toward Critical Posthumanities for the Anthropocene,” Ethics & the Environment 20 (1), 2015, 67–97.

Claire Pentecost, “Symbiotic Art,” Art in America (1939) 110 (2), New York: Brant Publications, Incorporated, 2022, 36– 43.

Anna Pomyalova, “Catching Signals from a Vibrant Environment,” Antennae: Journal of Nature and Visual Culture, 2022, 254–62.

Ken Rinaldo, “Can Microbes Save Us from Ourselves?” Antennae. The Journal of Nature in Visual Culture, 2022, 12–16.

Funke Iyabo Sangodeyi, “The Making of the Microbial Body, 1900s-2012,” Ph.D. diss., The Department of the History of Science, Harvard University, 2014.

Lisa A. Urry, Michael L. Cain, Steven A. Wasserman, Peter V. Minorsky, Jane B. Reece, Campbell Biology, Pearson Education, 2017.

Authors Biographies

Olga Timurgalieva is a PhD candidate at City University of Hong Kong and a former visiting researcher at King’s College London. Awarded by the Hong Kong PhD Fellowship Scheme, her research investigates the intersections of biotechnology and contemporary art, with a particular focus on fungal microbes and their interspecies relations. Olga has worked in art institutions, including the ZKM | Center for Art and Media (Karlsruhe), and co-curated the exhibition “Here and Elsewhere” at the Kobro Gallery, The Strzemiński Academy of Art (Lodz) and the festival “Seasons of Media Arts 2019” at the ZKM. She participated in the creation of collaborative art projects that have been exhibited at Aalborg University Copenhagen (Denmark), nachtspeicher23 (Germany), and Jockey Club Creative Art Centre (Hong Kong).

Patrícia Moreira holds a PhD in Biotechnology with a specialization in Biochemical Engineering from the Catholic University of Portugal (UCP, Portugal). She is an Assistant Professor at the School of Arts (UCP). She is an integrated member of the Center for Research in Science and Technology of the Arts (CITAR). Additionally, Patrícia coordinates the Area-Focus Heritage, Conservation and Restoration of CITAR and collaborates with the Center for Biotechnology and Fine Chemistry, both UCP. Her main research area is innovation in Biotechnology for Cultural Heritage, with an emphasis on biodeterioration, sustainability, citizen science Green Conservation and bio-art practices.

Eva Direito holds a degree in Art Conservation. She’s doing her master’s degree in Conservation of New Media Art at the School of Arts of the Catholic University of Portugal (UCP, Portugal). Having received an artistic education, Eva works with digital and analogue photography and graphic design. During the past few years, she’s been working as an Art Director in short movies for the School of Arts, some of which earned accolades. For instance, “Our House in Flames” by Miguel Mesquita got nominations at the Curtas Festival of Vila do Conde, Portugal. Additionally, “Hysteria” by Luísa Campino, with Eva as artistic director, won prizes at the Sophia Awards in Portugal. 

A Sympoietic Ocean. Design research with/in the marine holobiont

Rasa Weber

Zürich University of the Arts (CH); Matters of Activity. Cluster of Excellence, Humboldt University Berlin (DE); University of Art and Design Linz (AT). Zürich (CH) & Berlin (DE)
info@rasaweber.com

Abstract

In the face of profound human impact on planetary systems, the global ocean, as the main source of life, is fundamentally transforming its interactions, flows, and ecologies. These critical changes raise questions of other-than-human cohabitation beyond the terrestrial. In response to these radical ecological changes, a growing branch of the design discipline is currently struggling to free itself from a production-oriented paradigm of industrial modernity and reorganizes its methods toward forms of interspecies collaboration with/in environments of anthropogenic change.

In this paper, I argue for activating the evolutionary theory of Symbiogenesis (Margulis, Kozo-Polyansky, Meresch-kowsky) and its relevance for a holistic view of the ocean as a starting point for challenging and reinventing our disciplinary protocols. The article follows Haraway’s notion of sympoïesis, adopting it for the design discipline. The evolutionary model of Symbiogenesis offers a new perspective on the role of design as a facilitator for collaborative forms of making and shared survival.

The coral reef, as a prototypical space for symbiotic system relationships, serves as an experimental contact zone for designing these interspecies encounters. Design research in underwater environments, could help us to align the design discipline with a new conceptual framework that I propose to call Sympoïetic Design.

Keywords

Sympoïetic Design, ocean, design research, symbiogenesis, scuba diving, Biorock, sympoïesis, Cybertecture, designing habitats, materials research.

DOI

10.69564/ISEA2023-90-full-Weber-et-al-A-Sympoietic-Ocean

A Sympoïetic Ocean

“We are symbionts on a symbiotic planet. And if we care to, we can find symbiosis everywhere.” ¹ – Lynn Margulis, Symbiotic Planet, 1998.

Imagine a coral reef. The damselfish flicks its fins as it hides among stony corals to support its photosynthesis. The green-brown algae that live on the spider crab’s back help it to camouflage. The currents transport nutrients across limestone structure where corals spawn and settle. The zooxanthellae inhabit coral tissue and metabolize organic matter to provide energy for their hosts. The stony coral animal absorbs calcium carbonate from seawater to form its habitat. The microbiome of the reef binds Co2 and produces the oxygen of the planet’s atmosphere – it pulses with life.

The coral reef can be considered a prototypical ecology for symbiotic system relationships. Symbiosis governs the marine microbiome at all levels, from species-based networks to the microbiological level ². As former biologist and professor of feminism and technoscience Donna J. Haraway points out, the marine biome in general, and the “critical zone” of the coral reef in particular, serve as model organisms for symbiotic evolutionary processes ³ within the planetary holobiont ⁴.

In chapter 3 of Staying with the Trouble (Haraway 2016) “Sympoiesis. Symbiogenesis and the Lively Arts of Staying with the Trouble.” Haraway invokes the seminal work of biologist Lynn Margulis, who revived the evolutionary theory of symbiogenesis  based on the preliminary work of Russian evolutionary biologists Boris Kozo-Polyansk ⁶ and Konstantin Mereschkowsky ⁷. When Margulis published her theory in the 1960s, she provoked and disturbed the male-dominated scientific discourse not only as a female academic, but also through her decisive rejection of the then widespread Neo-Darwinian concept of evolution. Her theory challenged the idea of evolutionary competition – the hypothesis of random mutation as driver for “selfish” survival of the fittest ⁸ – and relied on a cooperative model of survival through symbiosis.

Margulis’ impassioned plea for symbiosisas an overlooked facilitator of evolutionary adaptation and her uncompromising proposal for a taxonomy of species which gives bacteria, animals, plants, fungi, and protoctista an equal position in the evolutionary Tree of Life, is nowadays generally considered as one of the most decisive contributions to ecological thinking. One could argue that Magulis’ theory resurrects Charles Darwin's alternative representation of evolution in the form of the coral: an anarchically growing form “that no longer upholds mankind as the ‘crown of evolution’’”¹⁰, as sketched in Darwin's early notebooks ¹¹. The Coral of Life can be seen as closely related to Mereschkowsky’s endosymbiotic Tree of Life and Lynn Margulis’ later extension to the Five Kingdoms ¹².

Figure 1 Field research at "Biorock" reef installed by Mike Duss (in the picture) and Tom Goreau. Location: Sun Reef, Curaçao (AN). Photo: Rasa Weber, 2022. 

Philosopher Donna Haraway’s outstanding contribution in Staying with the Trouble consists in turn, amongst other things, of applying Margulis’ theory of symbiogenesis to philosophy and system theory. She introduced a form of thinking and making, which she called sympoïesis ¹³—“together creation”—with human and other-than-human actors. While Haraway argues for transferring sympoïetic forms of production to the humanities and the arts, activating her theory for the design discipline holds the potential to reorganize a human-centered approach to design toward highly relational forms of creation through interspecies collaboration.

Philosopher Donna Haraway’s outstanding contribution in Staying with the Trouble consists in turn, amongst other things, of applying Margulis’ theory of symbiogenesis to philosophy and system theory. She introduced a form of thinking and making, which she called sympoïesis ¹³ —“together creation”—with human and other-than-human actors. While Haraway argues for transferring sympoïetic forms of production to the humanities and the arts, activating her theory for the design discipline holds the potential to reorganize a human-centered approach to design toward highly relational forms of creation through interspecies collaboration.

A growing branch of design is currently struggling to break free from a production—and profit-oriented paradigm of industrial modernity (Mareis & Paim 2021). By bringing the agency of troubled environments and their (other-than-) human inhabitants to the foreground of a design methodology, we might foster a reinvention of our disciplinary protocols in response to our current ecological and socio-political struggles, which can never be addressed separately (Stengers 2010).

Figure 2 Underwater prototype by Rasa Weber. Diver: Mike Duss. Location: Sun Reef, Curaçao (AN). Photo: Rasa Weber, 2022. 

Designing (with/in) Marine Ecosystems

As a design researcher and diver, I am concerned with material processes, their production cycles, and their resulting impacts on environmental systems. I am deeply moved by the question of developing and establishing post-extractive models of production within our challenged planetary networks.

This question led me to the ocean, the largest life-giving (Helmreich 2009; Latour 2019) and dramatically transforming system on our planet (Lohmann et al. 2017; Heinze et al., 2021). Together with biologists from the Max Planck Institute of Animal Behavior (Jordan Lab), I conduct most of my design research in the field, below sea level. I develop immersive underwater methods ¹⁴ (Helmreich 2009; Raycraft 2020; Swanson & Levy 2021; Jue 2022) for design research, prototyping sympoïetic material experiments underwater, and research on the design ontological implications of my aquatic practices (Neimanis 2017, Åsberg 2020).

Although designing underwater habitats looks back at a long history (See: e.g. Cousteau’s Conshelf Project 1962-1965), most of these early interventions still follow a human-centered design approach. My project takes the historical material technology of Biorock ¹⁵ as a starting point and seeks to convey a different notion of habitat design in ocean environments as a culturally embedded, interspecies, and collaborative process.

My research is based on the preliminary work of the German architect and oceanographer Wolf Hilbertz and his colleague in biochemistry Tom Goreau. As early as 1970, their research introduced a process of underwater electrolysis triggered by low-voltage electricity (6-12 V) that causes limestone to "grow" on conductive steel scaffold in the sea—a technology that was patented under the name Biorock. Naturally dissolved minerals in seawater (primarily aragonite and brucite) are slowly deposited on the conductive steel, which provides an ideal medium for the colonization by marine life. The technology is now widely used to restore coral reefs in tropical marine areas.

In spring 2022, I visited one of the Biorock projects on the island of Curaçao (AN) with the aim of learning more about the technological implications, ecological consequences, and maintenance methods of such an underwater garden and subsequently build my own first prototypes (Figure 1).

My own design contribution explores the possibilities of applying the Biorock technology to contemporary materials, combining a practice-based design approach with situated, collaborative, and craft-oriented design methods. Based on the principle of electrodeposition of minerals in seawater, I construct underwater prototypes as Interspecies Architecture which are neither exclusively for a human audience nor solely for marine critters. The prototypes function as contact zones between species and question the role of architecture as a human-centered practice. Instead of the historically used material—reinforcing steel—my work focuses on reducing material consumption and applies filigree conductive yarns to construct lightweight marine prototypes with the help of solar energy. When the conductive yarns are exposed to electricity, a limestone-like material forms on their surface, transforming a formerly flexible textile surface into a solid substrate that supports the growth of marine life; A technology that literally allows me to weave stone.

My prototypes mediate between the marine geosphere and biosphere. The spun steel nets provide a bioreceptive surface (Cruz & Beckett 2016) for multiple forms of marine life to slowly colonize on the mesh (Figure 4). Rather than taking a strictly selective approach to coral reef conservation, as classically the case in coral nurseries (which host a limited number of pre-selected coral branches), these prototypes should be viewed as an invitation for various forms of marine life to proliferate and thrive. The wild colonization of corals, sponges, algae and even marine neophytes is a desired outcome of these biological material experiments. For me, designing Architectures of Cohabitationmeans inviting all symbiotic forms of marine life to actively participate in the design process. They co-design their own habitat and subsequently change the appearance and species composition of the prototype. Thus, the design process takes place not only in the sea, but together with the marine ecosystem.

In my designs, I try to give the sympoïetic growth processes of the marine biome the decisive voice. Marine species shape the underwater architecture, distort it, overgrow it, make it invisible, possibly even cause it to collapse.

The environments that will eventually serve as testing grounds for my marine Interspecies Architectures are neither the "pristine" and endangered coral reefs of Hilbertz's previous work nor the marine protected areas of my first field experiments. The seascapes I want to study and which I regard as ideal environment for my prototypes are areas of high anthropogenic impact, or as Clouette and Fabien call them, an ‘AnthropOcean’ (2018)—The harbor, the former industrial mine, the canal, the marine military zone or the tourist beach are the damaged, contaminated, and conflicted zones of marine survival that I will engage with in order to design habitats "amidst ruins" (Tsing 2015).

I consider these slow-growing structures as Architectures of Co-habitation, as they provide a habitat for multiple marine species and temporary human visitors alike. Based on Tsing’s and Barua’s notion of “feral ecologies” (2021), sympoietic ecological worlds emerge in the Anthropocene when other-than-human beings become entangled with human infrastructure in urban environments (Tsing et al. 2021). My underwater prototypes are located precisely in these conflicted zones—they are breeding ground, ecological niche, and site of human infrastructure, interest, and intervention.

Figure 3 Underwater prototype by Rasa Weber (in the pic- ture). Location: Station de Recherche Océanographiques et sous-marines, Calvi, Corsica (FR). Photo: Anja Wegner, 2022.

 Designing Habitats

“How much longer will we agree to step aside in silence as masters of the universe []? How much longer will you and I choose extinction?” ¹⁶

– Gan, Tsing, Swanson & Bubandt 2017.

If design takes its fundamental contribution to the unruly times of the Anthropocene seriously and begins to “care for that which is non-human” ¹⁷, it will encounter a series of troubling questions. A design practice that is not only interested in advocating for human agency, but also includes other species and conflicted environments in its processes of making, must be brave enough to propose new “modes of production and cooperation that escape from the evidence of economic growth and competition.” ¹⁸ If design advocates conviviality and shared survival, it must formulate strategies to “positively cohabit a thriving biosphere” ¹⁹, and inevitably comes into contact with fundamental systemic issues of our assigned human role in the ecological fabric.

In his early publications, Wolf Hilbertz refers to his marine engineering intervention as “Cybertecture” ²⁰. The neologism is a reference to Hilbertz’ understanding of architecture as a logical extension of the principles of cybernetics ²¹  to the built environment (CYBERnetics + archiTECTURE). Most likely inspired by Norbert Wiener’s famous concept of Cybernetics as “the science of control and communications in the animal and machine” ²² , Hilbertz thought of complex systems controlled by a purposive model of causality and feedback. Such a biogenetic model, “that employs self-improving software and hardware [and] can draw ‘unorganized’ matter into its system” ²³, suggest a revolutionary notion of architecture as constantly adaptive and self-improving. However, the cybernetic core of his theory also, perhaps unintentionally, places the human in the center of system control. The concept of the designer as a "helmsman" (ky-bernētēs Greek = helmsman), who has full control over a system and is responsible for its well-being, must be seen in the context of the unbridled techno-optimism of the 1960s and the concomitant rise of the human-centered design movement (e.g. Arnold, Stanford University design program, 1958). It should be kept in mind that when Hilbertz developed his theory of Cybertecture, the discourse of architectural theory was just beginning to address environmental consciousness for the first time, also triggered by the ecological discourse of prominent voices such as a.o. Rachel Carson (1962).

With the Biorock system, Hilbertz and Goreau introduced an extremely bold ecological technology that was far ahead of its time and made it possible to build with and within the ocean ecosystem. However, given what we now today about the cumulative impacts of humans on the world's ocean systems, we need to reevaluate the system's useful technological capabilities ²⁴ in relation to its potential consequences.

“The planet takes care of us, not we of it”, ²⁵ Margulis states in her influential book A Symbiotic Planet(1998).Rather than trying to regulate the incredibly complex system relationships of an ecology, or even going so far as to “cure” the ocean as “a sick patient” ²⁶, Margulis’ perspective suggests questioning the role of human control in favor of a much modest and highly interconnected perspective. From this thought, we designers must accept our small part as humans within larger ecological relationships that transcend the influence of human intentions.

In this sense, the Biorock principle could shape and inform contemporary design discourse to conceive “evolutionary environments” ²⁷ that adapt to their environmental conditions and grow responsively with the marine ecosystem. Today, Goreau and Hilbertz’ concept could serve as a starting point for developing a form of sympoïetic making that views the designed “product” not just as the result of planned human intervention, but as relational, highly connected, and life-giving habitat.

Towards Sympoïetic Design

A design practice that sees the agencies of all life forms as central to shaping the design process may be able to move beyond a product-driven and extractive notion of design and open up to collaborative forms of design that transcend species barriers.

I propose to call these practices Sympoïetic Design. The evolutionary model of symbiogenesis lays the groundwork for conceptualizing design processes as long-term collaborations between humans and other-than-human beings. These sympoïetic processesextend over unimagined time spans and situate design practices in troubled environments. Design becomes an action of shaping “invitations” (Stahl & Lindström, 2016) for animals, bacteria, proctists, et al. The resulting prototypes I present here suggest that mineralized limestone mesh can serve as a “bioreceptive medium” (Cruz & Beckett, 2018) for a variety of marine species to settle on them (Figure 4 & 5). Sympoïetic Design provides habitats as contact zones for coevolution between different species.

Figure 4 (above): (left) Underwater prototype by Rasa Weber, conduct. steel yarn after installation, Sun Reef, Curaçao (AN). Photo: Rasa Weber, 2022. (right) Underwater prototype by Rasa Weber, conductive steel yarn after 2 months in the ocean, mineral accretion and pioneering organisms, Sun Reef, Curaçao (AN). Photo: Mike Duss, 2022.

 

Het Niewe Instituut recently introduced the term “Zoöps” to describe a “cooperative legal entity in which humans and multispecies ecological communities are partners”  ²⁸(zoë Greek = life). The idea is intriguing because it views habitats as an assembly of symbiotic entities that engage humans in their “naturecultural” ²⁹ relationships within an anthropogenic ecology. As humans, we impact habitats, but at the same time are inevitably a part of them. Designing for ecological impact, then, means designing in and with the habitats that assure our own survival. Applying this concept to marine habitats and thus creating convivial zones of protection and experimentation as Marine Zoöps should become a central task for design in the ocean.

A number of young designers and researchers are beginning to design systems with and for other-than-humans. The Symbiotic Spaces Collective is working on 3D-printed architectures located in local wetlands that provide multispecies habitats for urban wildlife. Designer Marie Griesmar, biologist Ulrike Pfreundt, and marine scientist Hanna Kuhfuss, founders of Rrreefs e.V., developed a 3D-printed clay brick that serves as a module for coral reef restoration. The biologist Anja Wegner and the artist collective Superflex collaborate on their “Pink Elements” to create “fish architecture”³⁰ for damselfish, and designer David Enon experiments with growing underwater furniture from limestone structures as coral habitat with his “Mineral Accretion Factory” ³¹. 

Despite these early beginnings, deep reflection on the benefits and limitations of working with/in ecological systems is urgently needed. Sympoïetic Design raises new questions for our discipline that call for further discussion:

Working with/in the marine biome can be a first experimental approach to discuss the potentials and challenges of Sympoïetic Design. With my underwater prototypes, I work on creating contact zones between multiple forms of marine life. These Architectures of Cohabitation mediate between animate and inanimate matter, between human-made and naturecultural ecosystems, between human and environmental agency. Rather than restoring “pristine” nature, these co-created habitats encourage the designer to relinquish control of the system and invite the wildly proliferating and uncurated, unexploited life forms to the process of creation.

Sponges, algae, coral, electrical circuits, marine biologists, fish, cameras, limestone, polyps et al. become part of these multispecies marine eclogies of sympoïetic making. Sympoïetic Design opens the door to design practices that transform human-centered design projects into an evolutionary practice of share survival. Towards a sea change ³² in design.

Figure 6 Underwater prototype by Rasa Weber after 6 months in the ocean, conductive steel yarn, mineral accretion and pioneering organisms. Location: Sun Reef, Curaçao (AN). Photo: Pavel Siman, 2022.

Acknowledgements

I would like to thank my collaborators, Max Planck Institute for Animal Behavior - Jordan Lab, for their continuous support and time in the field, especially biologists Anja Wegner and Prof. Dr. Alex Jordan. Furthermore, I would like to thank Prof. Dr. Karmen Franinović (ZHdK) and Prof. Dr. Karin Harrasser (Kunstuniversität Linz) for their supervision and support during this research project. I would like to thank the "Biorock" project on Curaçao (NL), the diving instructor Mike Duss, the local diving school "Curaçao Divers" as well as the "Sea School" Pula (HVR). Finally, I would like to thank Tom Goreau for many conversations, critical discussions, and technical support with the Biorock technology.

References
Books

Horst Bredekamp, Darwin's Corals. A New Model of Evolution and the Tradition of Natural History, Berlin, Boston, De Gruyter, 2019, https://doi.org/10.1515/9783110680317.

Rachel Carson, Silent Spring, New York, Fawcett Crest, 1962.

Donna J. Haraway, The Companion Species Manifesto. Dogs, People, and Significant Otherness, Bristol, Prickly Paradigm Press, 2003.

Stefan Helmreich, Alien Ocean: Anthropological Voyages in a Microbial Sea, Seas Berkeley, CA, University of California Press, 2009.

Boris M. Kozo-Polyansky In: Symbiogenesi,. A new principle of evolution, Translation from the Russian by Victor Fet, Eds. By Victor Fet, Lynn Margulis, Cambridge, Harvard Univ. Press, 2010.

Bruno Latour, Foreword, In: Prospecting Ocean, ed. Stefanie Hessler, Cambridge, Massachusetts, London, England, The MIT Press, 2019, 11-13.

Claudia Mareis & Nina Paim (Eds.), Design Struggles. Intersecting Histories, Pedagogies, and Perspectives, Amsterdam, Valiz, 2021.

Lynn Margulis, The Five Kingdoms: Illustrated Guide to the Phyla of Life on Earth, San Francisco, W.H. Freeman & Co Ltd., 1982.

Lynn Margulis, Symbiotic Planet, A New Look on Evolution, Amherst, Massachusetts, Brokman inc., Basic Books, revisited edition, 1998.

Astrida Neimanis, Bodies of Water: Posthuman Feminist Phenomenology, Environmental Cultures. London, New York, Bloomsbury Academic 2017.

Isabelle Stengers, Cosmopolitics I., Minneapolis, London, Minnesota Press, 2010.

Isabelle Stengers, In Catastrophic Times - Resisting the Coming Barbarism, Open Humanities Press, Meson Press 2015.

Anna L. Tsing, The Mushroom at the End of the World: On the Possibility of Life in Capitalist Ruins, Princeton, Princeton University Press, 2015.

Anna L. Tsing, Heather A. Swanson, Elaine Gan & Nils Bubandt (Eds.), Arts of Living on a Damaged Planet: Ghosts and Monsters of the Anthropocene, Minneapolis, University of Minnesota Press, 2017.

Ron Wakkary, Things We Could Design: For More Than Human-Centered Worlds, Cambridge, MA, MIT Press, 2021.

Norbert Wiener, Cybernetics, or, Control and communication in the animal and the machine. The Massachussets Institute of Technology, orig. 1948, Cambridge Massachussets, The MIT Press, 1985.

Chapters in Books

Richard Dawkins, “Chapter 4: The Gene Machine”, In: The Selfish Gene, Oxford, Oxford University Press, 1976, 46-65.

Donna J. Haraway, Chapter 3: Four Critical Zones. In: Staying with the Trouble: Making Kin in the Chthulucene; Durham and London, Duke University Press, 2016, 72-76.

Donna J. Haraway, “Chapter 3: Sympoiesis. Symbiogenesis and the Lively Arts of Staying with the Trouble”, In: Staying with the Trouble: Making Kin in the Chthulucene, Durham and London, Duke University Press, 2016, 58-98.

Åsa Ståhl, Kristina Lindström, “Politics of Inviting: Co-Articulations of Issues in Designerly Public Engagement”, In:Design Anthropological Futures, Rachel C. Smith et al. (Eds.), London, Routledge, 2016.

Jonathan P. Zehr, David A. Caron, “Symbiosis in the Ocean Microbiome”, In: The Marine Microbiome: An Untapped Source of Biodiversity and Biotechnological Potential, Lucas J. Stal & Mariana S. Cretoiu (eds.), Springer, 2016, 535-577.

Journal article (online)

Cecilia Åsberg, “A Sea Change in the Environmental Humanities”, In: Ecocene: Cappadocia Journal of Environmental Humanities Vol. 1, No. 1, 2020, 108-122.

Maan Barua, “Feral ecologies: the making of postcolonial nature in London”, Journal of the Royal Anthropological Institute, Vol. 28, No. 3, 2021, 896-919.

Jeremie Brugidou, Clouette Fabien, “‘AnthropOcean’: Oceanic perspectives and cephalopodic imaginaries moving beyond land-centric ecologies.” Social Science Information, Vol. 57 No. 3, 2018, 359–385, DOI: https://doi.org/10.1177/0539018418795603.

Marcos A. da Silva Cruz, Richard Beckett, “Bioreceptive design: a novel approach to biodigital materiality”, Architectural Research Quarterly Vol. 20, No. 1, 2016, 51-54.

Ricardo Guerrero, Lynn Margulis, Mercedes Berlanga, “Symbiogenesis: the holobiont as a unit of evolution.” Microbiol. Vol. 16, No. 3, 2013, 133-43.

Benjamin S. Halpern, Melanie R. Frazier, Jamie Afflerbach, et al. “Recent pace of change in human impact on the world’s ocean”, Scientific Reports Vol. 9, No.1, 2019, Doi:10.1038/s41598-019-47201-9.

Wolf Hilbertz, “Toward Cybertecture.” Progressive Architecture, The Architect as Developer (1970), Wolf Hilbertz Website, accessed February 14, 2024, http://www.wolfhilbertz.com/downloads/1970/hilbertz_t_cybertecture_1970.pdf.

Wolf Hilbertz, “Strategies for Evolutionary Environments”, The Responsive House, Edward Allen (Ed.), MIT Press, 1972.

Wolf Hilbertz Website, accessed online February 14, 2024, http://www.wolfhilbertz.com/downloads/1970/hilbertz_t_cybertecture_1970.pdf.

Wolf Hilbertz, “Electrodeposition of Minerals in Sea Water: Experiments and Applications”, IEEE Journal on Oceanic Engineering, Vol. OE-4 Vol.3 1979, 94-113.

Wolf Hilbertz Website, accessed online February 14, 2024, https://globalcoral.org/_oldgcra/IEEE_JOUR_1979small.pdf.

Melody Jue, “Scuba Diving as Praxis: A Field Guide for Underwater Orientation by Melody Jue”, Fieldwork for Future Ecologies, ed. Bridget Crone, Sam Nightingale, and Polly Stanton (Onomatopee, 2023), accessed February 14, 2024, https://static1.squarespace.com/static/545596ece4b0717418ebf8e6/t/642de8963cdc9f603863cdc3/1680730265157/Melody+Jue_Scuba+diving+praxis.pdf.

Klaus V. Kowallik & William F. Martin “The Origin of Symbiogenesis: An annotated English translation of Mereschkowsky’s 1910 paper on the theory of two plasma lineages”, Bioystems, Vol. 199, No. 104281, 2020, Doi: 10.1016/j.biosystems.2020.104281.

Johannes Lohmann, Daniele Castellana, Peter D. Ditlevsen, and Henk A. Dijkstra “Abrupt climate change as a rate-dependent cascading tipping point.” Earth System Dynamic, Vol. 12, No. 3, 2021, 819–835.

Justin Raycraft. “Seeing from Below: Scuba Diving and the Regressive Cyborg.” Anthropology and Humanism, Vol. 45, No. 2, 2020, 301-321.

Lynn Sagan, “On the origin of mitosing cells.” J. Theoretical Biology Vol. 14, No. 3, 1967, 255-274, doi:10.1016/0022-5193(67)90079-3.

Marie E. Strader, Kate M. Quigley, “The role of gene expression and symbiosis in reef-building coral acquired heat tolerance”, Nature Communications Vol. 13, Article number, 4513, 2022.

Heather A. Swanson, Sonia Levy, “A (Highly) Partial Field Guide to British Canals: Introducing some processes and beings", TBA21-Academy, 2021.

Websites

Thomas J. Goreau, “A review of John Todd, 2019: Healing Earth: An ecologist’s journey of innovation and environmental stewardship (2019)”,Global Coral Reef Alliance Website, accessed: November 17, 2022, www.globalcoral.org/a-review-of-john-todd-2019-healing-earth-an-ecologists-journey-of-innovation-and-environmental-stewardship/.

Nieuwe Instituut, “Zoöp Nieuwe Instituut (2022)”, Nieuwe Instituut Website, accessed: December 9, 2022, https://zoop.hetnieuweinstituut.nl.

Anna L. Tsing, Jennifer Deger, Alder Keleman Saxena & Feifei Zhou, Feral Atlas: The More-Than-Human Anthropocene, Redwood City, Stanford University Press, 2022), Feral Atlas Website (2022), accessed December 9, 2022, http://feralatlas.org.

Conference Proceedings 

David Enon, “Mineral accretion factory: An underwater production process with a positive impact on the environment”, Figshare, Journal contribution, Conference, RTD, 2019, DOI: https://doi.org/10.6084/m9.figshare.7855775.v2.

Christoph Heinze, Thorsten Bleckner, Helena Martins et al. “The quiet crossing of ocean tipping points.” Proceedings of the National Academy of Sciences Vol. 118, No. 9, 2021, DOI: https://doi.org/10.1073/pnas.2008478118.

Anja Wegner, Superflex & Alex Jordan, “Fish Architecture – A framework to create Interspecies Spaces.” Proceedings of Politics of the machines - Rogue Research 2021, (POM 2021).

Anja Wegner, Rasa Weber, “Attunement to the Ocean. Underwater field methods between marine biology and design research”, Counterparts: Exploring Design Beyond the Human, Swiss Design Network Conference 2022. (Forthcoming).

Design Works

David Enon, Mineral Accretion Factory, since 2011.

Marie Griesmar, Ulrike Pfreundt, & Hanna Kuhfuß, The rrreef bricks. Rrreefs e.V., since 2020.

Laurin Kilbert & Joana Schmitz, Symbiotic Spaces. Symbiotic Spaces Collective, since 2020.

Rasa Weber, Symbiocean, since 2022.

Anja Wegner & Superflex, Pink Elements, since 2019.

Notebooks

Charles R. Darwin, “Sketch: Coral of Life.” Page 26 of Notebook B, (1837–1838). Cambridge University Library MS DAR.121.26.

Endnotes 

1 Lynn Margulis, Symbiotic Planet. A New Look on Evolution, Amherst, Massachusetts, Brokman inc., Basic Books, revisited edition, 1998, 5.

2 Jonathan P. Zehr, David A. Caron, “Symbiosis in the Ocean Microbiome.” In: The Marine Microbiome: An Untapped Source of Biodiversity and Biotechnological Potential, Lucas J. Stal & Mariana S. Cretoiu (eds.), Springer, 2016, 535-577.

3 Donna J. Haraway, Chapter 3: Four Critical Zones. In: Staying with the Trouble: Making Kin in the Chthulucene, Durham and London, Duke University Press, 2016, 72-76.

4 Holobiont def. = The entire community of living organisms. In: Michael Allaby, Dictionary of Ecology, Oxford, Oxford University Press, 2015.

5 Symbiogenesis def. = The result of the permanent coexistence of various bionts to form the holobiont (namely, the host and its microbiota). In: Ricardo Guerrero, Lynn Margulis & Mercedes Berlanga, “Symbiogenesis: the holobiont as a unit of evolution.” Microbiol. Vol. 16, No. 3, 2013, 133-43.

6 Boris M. Kozo-Polyansky In: Symbiogenesis. A new principle of evolution.  Translation from the Russian by Victor Fet, Eds. By Victor Fet and Lynn Margulis, Cambridge, Harvard Univ. Press, 2010.

7 Klaus V. Kowallik, William F. Martin “The Origin of Symbiogenesis: An annotated English translation of Mereschkowsky’s 1910 paper on the theory of two plasma lineages”, Biosystems, Vol. 199, No. 104281, 2020, Doi: 10.1016/j.biosystems.2020.104281.

8 Richard Dawkins, “Chapter 4: The Gene Machine”, In: The Selfish Gene, Oxford: Oxford University Press, 1976, 46-65.

9 Margulis outlined “the astounding details of the microbes (…) in their tendency to survive by making symbiotic evolutionary commitments,” arguing for her central theory on the origin of eukaryotic cells from prokaryotic organisms, In: Lynn Margulis, Symbiotic Planet, Basic Books 1989, 72.

10 Horst Bredekamp, Darwin's Corals. A New Model of Evolution and the Tradition of Natural History, Berlin, Boston, De Gruyter, 2019, https://doi.org/10.1515/9783110680317.

11 Charles R. Darwin, “Sketch: Coral of Life.” Page 26 of Notebook B, 1837–1838, Cambridge University Library MS DAR.121.26.

12 Lynn Margulis, The Five Kingdoms: Illustrated Guide to the Phyla of Life on Earth, San Francisco: W.H. Freeman & Co Ltd., 1982.

13 sympoïesis def. = “Σύν, sún = greek ‘together’ and ποίησις, poíēsis = greek ‘creation, production’”. In: Donna J. Haraway Staying with the Trouble. Making Kin in the Chthulucene, Duke University Press, 2016, p.58. 

14 Anja Wegner, Rasa Weber, “Attunement to the Ocean. Underwater field methods between marine biology and design research", Counterparts: Exploring Design Beyond the Human, Swiss Design Network Conference 2022. (Forthcoming).

15 Wolf Hilbertz, “Electrodeposition of Minerals in Sea Water: Experiments and Applications”, IEEE Journal on Oceanic Engineering, Vol. OE-4 Vol.3 (1979): 94-113. Wolf Hilbertz Website, accessed online February 14, 2024, https://globalcoral.org/_oldgcra/IEEE_JOUR_1979small.pdf.

16 Anna L. Tsing, Heather A. Swanson, Elaine Gan & Nils Bubandt (Eds.) Arts of Living on a Damaged Planet: Ghosts and Monsters of the Anthropocene, Minneapolis, University of Minnesota Press, 2017, G4.

17 Ron Wakkary, Things We Could Design: For More Than Human-Centered Worlds, Cambridge, MA, MIT Press, 2021, 1-2.

18 Isabelle Stengers, In Catastrophic Times - Resisting the Coming Barbarism, Open Humanities Press, Meson Press 2015, 24.

19 Ron Wakkary, Things We Could Design (2021), 2.

20 Wolf Hilbertz, “Toward Cybertecture.” Progressive Architecture, The Architect as Developer (1970), Wolf Hilbertz Website, accessed February 14, 2024, http://www.wolfhilbertz.com/downloads/1970/hilbertz_t_cybertecture_1970.pdf.

21 Cybernetics def. = The term cybernetics comes from the ancient Greek word κυβερνήτης, kybernetikos = “good at steering”, referring to the art of the helmsman. In: Britannica Website, accessed February 1, 2022, https://www.britannica.com/science/cybernetics.

22 Norbert Wiener, Cybernetics, or, Control and communication in the animal and the machine, The Massachussets Institute of Technology, orig. 1948, Cambridge Massachussets, The MIT Press, 1985.

23 Wolf Hilbertz “Toward Cybertecture", 1970, 99. [Addition by the autor.]

24 Benjamin S. Halpern, Melanie R. Frazier, Jamie Afflerbach, et al. “Recent pace of change in human impact on the world’s ocean.” Scientific Reports Vol. 9, No.1, 2019, Doi:10.1038/s41598-019-47201-9.

25 Lynn Margulis, Symbiotic Planet, 1998, 115.

26 Thomas J. Goreau, “A review of John Todd, 2019: Healing Earth: An ecologist’s journey of innovation and environmental stewardship (2019)”,Global Coral Reef Alliance Website accessed: November 17, 2022, www.globalcoral.org/a-review-of-john-todd-2019-healing-earth-an-ecologists-journey-of-innovation-and-environmental-stewardship/.

27 Wolf Hilbertz, “Strategies for Evolutionary Environments.” The Responsive House, Edward Allen (Ed.) (MIT Press, 1972), Wolf Hilbertz Website, accessed online February 14, 2024, http://www.wolfhilbertz.com/downloads/1970/hilbertz_t_cybertecture_1970.pdf.

28 Nieuwe Instituut, “Zoöp Nieuwe Instituut (2022)”, Nieuwe Instituut Website, accessed: December 9, 2022. Link: https://zoop.hetnieuweinstituut.nl.

29 Natureculture def. = a synthesis of nature and culture that recognizes their inseparability in ecological relationships that are both biophysically and socially formed. In: N. M. Malone & Kathryn “Natureculture”, In: The International Encyclopedia of Primatology, John Wiley & Sons. See also: Donna J. Haraway, The Companion Species Manifesto. Dogs, People, and Significant Otherness. (Bristol: Prickly Paradigm Press, 2003).

30 Anja Wegner, Superflex, Alex Jordan, “Fish Architecture – A framework to create Interspecies Spaces.” Proceedings of Politics of the machines - Rogue Research 2021, POM 2021.

31 David Enon, “Mineral accretion factory: An underwater production process with a positive impact on the environment.” Figshare. Journal contribution. Conference, RTD, 2019, DOI:https://doi.org/10.6084/m9.figshare.7855775.v2.

32 Cecilia Åsberg, “A Sea Change in the Environmental Humanities.” In: Ecocene: Cappadocia Journal of Environmental Humanities Vol. 1, No. 1, 2020, 108-122.

NER: Physical-Virtual Multimodal Generative NFT with a Rarity Model

Ziwei Wu1, Danlu Fei2,∗, Xinyu Ma3, Mingming Fan4, Kang Zhang5

1,2IIP (Computational Media and Arts), The Hong Kong University of Science and Technology,3Computational Media and Arts, The Hong Kong University of Science and Technology, 4Division of Integrative Systems & Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, 5Division of Emerging Interdisciplinary Areas, The Hong Kong University of Science and Technology
1,2,4,5 Hong Kong SAR, China 3Guangzhou, China
∗dfeiaa@connect.ust.hk

Abstract

Nowadays, NFTs are increasingly emerging into public view. It is necessary to consider NFT as a sustainable trading model for media arts with audience interaction as the symbiosis community.

This article presents a design architecture of a Multimodal NFT with generative patterns triggered by sensors. By designing various 3D mesh attributes, such as eye patterns and fur colors of a robin bird, the authors generate a series of different NFT 3D artworks. We also demonstrate our experience with the generative system together with the rarity scoring model that could be used to evaluate the heterogeneity of NFT collections.

Keywords

Multimodal NFT, Generative Art, Sensor-Driven, Self-tracking Data, Rarity.

DOI

10.69564/ISEA2023-91-full-Wu-et-al-NER

Introduction

As new “digital titles (tokens) to property, either real or virtual, stored on a blockchain” ¹², Non-Fungible Token (or NFT) influences various social aspects, such as financial markets and art collections, especially media art creation. We believe that apart from its commercial exploitation, NFT represents a new medium that gives artists a different way of thinking about the creative process.

Multimodal interaction ³, i.e., “interaction with the virtual and physical environments through natural modes of communication”, has been a widely accepted term in media art creation. It, however, seldom appears in the NFT artwork creation.

Computational Generative Art (CG-art) is defined as the art “produced by leaving a computer program to run by it-self” ². By manipulating the generative design process, various artists have produced different captivating generative artworks over the years. In connection with NFT, new methods of displaying artworks have recently evolved, both online and offline, in addition to the computer screen and physical print presentations.

Guo et al. state “the art world will witness a new form of art economy with the coming of Web 3.0 (the metaverse), benefiting from its augmented creativity and sophisticated marketplace.” ⁷. This article provides a short review of Generative NFTs, multimodal dynamic NFTs, and physically mixed virtual NFTs with different NFT rarity models. We then demonstrate our experiment NER, which creates a Multimodal Generative NFT with sensors and a rarity model.

Related Work

Multimodal Generative NFTs Commercial online NFT collection platforms, such as OpenSea ¹⁵ and SuperRare ²⁴, are usually non-customizable. Recent advances in computational generative art have renewed interest in generative art in different materials ²⁷. Artists and researchers have generated various 2D patterns. For example, Zhang and Yu develop a font generation system in the style of Kandinsky ²⁶. Others attempted to create 2D generating textures on 3D mesh, such as Wu and Huang’s artwork “Mimicry” ²⁵, which is a genetic-algorithm-based realtime system that creates virtual insects within a real environment. These generative artworks are not changed into NFT format due to technical limitations. At the same time, a NFT art collection platform, especially for generative art, called ArtBlocks ²³ pushes the limits of generative art with blockchain technology. There are many interesting generative artworks on this website, especially in the category of curated collections.

Digital artist Pak built a generative NFT artwork “Merge” ¹³ which returns a new mechanism with collectors. Merge obtains the number of each collector during the 48-hour sale and generates dynamic on-chain NFT based on the total mass number. The NFT visual becomes bigger when collectors acquire more mass. This NFT artwork builds a new way that collectors can be seen as an entangleable and competitive symbiosis community. Refik Anadol studio used projection mapping on the facade of Casa Batllo ́ to model the dynamic NFT artwork “Living Architecture” ⁹ with weather sensor data collected in realtime. This dynamic NFT renews and reinvigorates World Heritage building through AI technology. Most NFT artworks containing physical and virtual versions are highly relevant to commercial applications, particularly brand marketing. For example, Nike, together with RTFKT, established a range of virtual sneakers and expanded into physical goods, such as clothing. Look Labs ¹⁰ uses near infrared spectroscopy to create a digital scent, and translate the scent into NFT artworks, making it the world’s first digital fragrance ¹⁷. 

NFT Rarity NFT rarity shows how rare a NFT item is compared to the rest of the items from the same collection, which in turn can represent how valuable such a collectible can be. When looking at a collectible NFT item on Opensea, such as a CryptoPunk ⁶ , one can see that the item has many traits, for example, accessories like an earring, a bandana, or a pipe. Since each NFT item has multiple traits, several rarity calculating models exist to combine each trait’s rarity into a single value per NFT item to rank and compare different NFT items.

Mainstream rarity calculating tools that can calculate and compare NFTs, such as Rarity.Tools ²⁰ , offer four models for users to calculate the rarity of NFTs: Trait Rarity, Average Trait Rarity, Statistical Rarity, and Rarity Score (see Figure 1). 

Figure 1: Four rarity models ©Authors

The Trait Rarity model works by only comparing the rarest trait of each NFT item. While it is a simple, straightforward model, this model’s weakness is also apparent. It does not consider the total number of traits or other traits of the NFT item. As a result, this model provides an incomplete picture of the NFT item, ignoring other traits that may also be rare.

The Average Trait Rarity model ¹⁹ averages the rarity of traits that exist on an NFT. This model adds up the rarities of all traits and divides the result by the total number of traits to obtain a score representing the entire NFT item. Although this model takes the overall rarity of the traits into account, it “dilutes” the rarity values of those traits that might be extremely rare.

The Statistical Rarity model multiplies the rarity percentages of all traits to calculate the overall rarity of an NFT. However, this model does not measure the NFT’s rarity in the specified collection of the NFT. If there are two two-trait NFT items, one with a rarest trait and a least rare trait, the other with two somewhat rare traits, then this model gives the latter a higher rarity value.

The Rarity Score model ¹⁸ is developed by the founder of Rarity.Tool. The model first takes the inverse of the rarity percentage of each trait as the rarity score of every single trait. Then the model adds up each rarity score calculated previously as the overall rarity score for the entire NFT item. This model not only considers all the traits of a single NFT item, but also measures the NFT’s rarity in the specified collection. We, therefore, use this model in our work to calculate the rarity score.

NER Creation Process

Our NFT creation, called NER (NFT-Enabled Robin) , combines a real-world furry toy with a digitally generative 3D model (see Figure 2). 

Figure 2: Physical and Virtual Bird appearance ©Authors

The following subsections provide details of the design process: (1) Converting a physical bird into a virtual bird via NFT. (2) Designing a virtual NFT trait and its rarity model. (3) Dynamically generating cloth in response to the change of mobile sensor data.

Physical and Virtual Birds with NFC

Figure 3 shows the workflow for transforming a physical bird to a virtual bird NFT. First, we store a physical NFC card in a small bag on the back of each physical bird (see Figure 2). Then we enter different redemption codes into different NFC cards to ensure every code is unique and hard to replace. Second, NER collectors can obtain the redemption code by tapping their phones with an NFC card and entering our app to paste the code (see the left image of Figure 4). Our server receives the signal from the user and directly detects the user’s internet status to ensure each redemption code to be redeemed only once. If the user’s phone does not connect to the internet, the NER app displays a warning (see the middle image of Figure 4). If the user’s phone is connected, the NER app shows a shaking egg (see the right image of Figure 4). Third, our server checks the status of each redemption code; if the code is the 1st time redeemed, the NER app generates a unique virtual bird NFT; if the code is already redeemed, our app directly displays the existing NFT.

Virtual NFT trait design with a rarity model

A virtual NFT artwork can be divided into two parts: 3D mesh rendering with a rarity model, and 2D patterns generated by Shader with multimodal sensors which will be discussed in the next subsection. 

Figure 3: Transformation workflow of physical bird and virtual bird with NFC, Server and Phone ©Authors
Figure 4: Three types of user interface ©Authors

Our design architecture considers different parts of virtual traits from top down (See Figure 5).

• Hairstyle.
• HairColor.
• EyesPattern.
• BeakColor.
• FeetColor.
• BackgroundColor.

Figure 5: Virtual bird design with 6 traits ©Authors

Both the beak color and feet color have two variations. The one with a higher rarity among all the variations holds the rarity percentage of 30%, and the other holds the rarity percentage of 70%. Hairstyles have three different trait variations. Ranking from the rarest hairstyle to the most common one, the percentages of owning them are 15%, 30%, 55%, respectively. For those traits (hair color, eyes pattern, background color) with four variations, we define their rarity percentages, from low to high, as 10%, 20%, 35%, 35%, respectively (See Figure 6). With these six traits, NER has 768 variations in total. 

Figure 6: NFT bird trait’s Variation, Rarity Percentage and Rarity Score ©Authors

To illustrate how to apply the above Rarity Score Model to calculate the rarity score for our NFT item, we choose a bird to represent the appearance (see the 4th image of Figure 8). This bird has a hairstyle with the rarity percentage of 15%, a hair color with the rarity percentage of 10%, an eyes pattern with the rarity percentage of 35%, a beak color with the rarity percentage of 30%, a background color with the rarity percentage of 35%, and the feet color with the rarity percentage of 70%. The single rarity score of each trait can be calculated by taking the inverse of that trait. For example, the single rarity score of its hairstyle is 1/15% ≈ 6.67. Using this method, 15% one may calculate the individual trait scores of all the traits and sum the results up to obtain the total score of this NFT item:

Dynamically generating cloth

To generate the 2D cloth texture, we use multimodal sensors to obtain mobile self-tracking data to drive the generation.

As McLuhan stated in the book Understanding Media: The Extensions of Man ¹¹, the opinions of media are the extensions of human, and reflected in all aspects of life in the digital era. Among different media, “mobile technologies have become an important part of our lives as they are capable of representing an extension of our physical selves” ⁸. Thus, we use the user’s mobile data to represent part of his/her self-portrait ¹⁶ in the virtual world.

To protect the privacy of our users, NER does not read and track personal data from their phones. Instead, it receives four types of data that affect 4 parts of the generation. Figure 7 shows the four types of data (i.e., three axes of gyroscope data and acceleration data) and the corresponding four aspects of the generation (i.e., the red, green, and blue channels of the visual pattern and its change speed).

Specifically, we receive two primary data types from the phone: gyroscope and acceleration. The gyroscope data influences color, while acceleration data influences the texture moving speed. More specifically, gyroscopes X, Y, and Z controls red, green, and blue colors respectively.

Figure 8 shows a series of generated results with rarity scores by different users.

Technical Implementation

The model of the NFT Bird is built with different pieces of 3D modeling software. The models of the bird and a series of hairs are created in Zbrush and Maya is used to modify and add UV maps to them. The fabric model is created using Marvelous Designer. After preparing the model and UV maps for the NFT bird, we use Substance Painter to paint a range of basic colors and apply different materials to the model. The model is then sent to Unity for the next stage of work: rendering and generation. In order to generate our Multimodal NFT with different mobile sensors, we developed a mobile application in Unity that contains three phases of operations: (1) Redemption: users can enter their redemption codes for generating the NFT. (2) Generation: users generate NFTs with their mobile sensors. (3) Saving: the user data of the generated NFT is stored for future review.

Redemption

In this phase, users must sign up and log in with their email addresses, and then enter a valid redemption code (see the left image of Figure 4) to ensure that the virtual NFTs generated are exclusively linked to their accounts. To verify whether a user has the chance to generate a virtual NFT, we produce 1000 unique redemption codes and store each of them in an NFC card placed in the bag of each physical NFT bird. A redemption code consists of 10 characters, including digital numbers and both uppercase and lowercase letters, with over 8×1017 distinct combinations. We store all redemption codes on the server and keep track of their states. Whenever a redemption code is used, its state is changed from ’Unused’ to ’Redeemed’ to ensure that each redemption code can only be redeemed once to generate one virtual NFT bird. A used code is stored under the account which has applied the redemption so that the specific account can review the NFT generated with the code.

Figure 7: The mapping relationship between the sensor data and visual elements generated by Shader ©Authors

Generation

The generation phase begins right after the user enters a valid and unused redemption code in the Redemption Phase. In this phase, NER first lets users personalize the shader for the cloth of the virtual NFT birds, and then randomly decides the traits for the user according to the rarity percentage of the traits.

Shader Personalization

The shader we use for the cloth of the virtual NFT is inspired by a shader ”70s Melt” ¹ written in OpenGL Shading Language (GLSL) in an online shader community, Shadertoy ²¹ . Based on GLSL, we develop our shader in High-Level Shader Language (HLSL) acceptable by Unity. We add to this shader four properties, that can be set by the data gathered from the sensors. The first three properties drive the personalization of the color, determining how much red, green, and blue are applied in the shader, respectively. The last property controls how fast the pattern on the shader changes. We use the gyroscope sensor to set the first three properties related to the color. A gyroscope sensor can measure the angular velocity of the device. As shown in Figure 9, we define the axis parallel to the long side of the phone screen as the x-axis, the axis parallel to the short side of the screen as the y-axis, and the axis perpendicular to the screen as the z-axis. The device rotation around the x, y, z-axis determines the amount of red, green, and blue of the shader respectively. For the last property related to the moving and changing speed of the pattern, we use the accelerometer sensor to control its value. The speed is positively related to the magnitude value of the acceleration. The faster the device moves, the bigger the value is. To let users personalize the shader with their mobile sensors, our instructions in the Generation Phase (see the right image of Figure 4) tell users to shake their phones for two seconds to allow sensors to gather data. 

Trait Generation

After the user has personalized the cloth, NER randomly determines what trait variations the NFT has based on the trait rarity percentage. For example, the trait rarity percentage of the hairstyle of the NFT in Figure 5 is 15%. If the system randomly generates a floating-point number from 0 to 1, only a value between 0 to 0.15 determines that the NFT can have this hairstyle. Other values correspond to other hairstyle variations.

Saving

The Saving Phase starts once the generation of the NFT is done. In this phase, all data related to the shader properties and the trait generation from the previous phase is written in a JSON file and stored on the server in the logged-in account. If the user of this account enters an already redeemed code again in the Redemption Phase, all the stored data is fetched to directly display the generated NFT to the user, skipping the Generation Phase.

Conclusions and Future Work

First, the article reviews existing Multimodal Generative NFTs artworks and existing NFT Rarity models.

Next, we present a design architecture for creating a physical-virtual Multimodal NFT with generative patterns and rarity scores, called NER (NFT-Enabled Robin). It contains

Our NER is able to collect all traits and sensor data. These data can be sent to the NFT platform in real time to use We-bGL to convert existing data into the blockchain. 

Figure 8: Samples of NER (NFT-Enabled Robin) generated by different users with rarity scores ©Authors

Future Work

In the future, we plan to establish a NER NFT collector community as a symbiosis ecosystem. The vitality and longevity of NFT artworks are highly related to their own community of collectors. Good NFT communities usually contain users’ spontaneous secondary creations, like CryptoKitties ⁵ and SpaceDAO ²². We may organize offline human-computer interactive workshops to let NER collectors bring their physical birds with accessories and enable them to build different accessories and animations online.

Figure 9: The mapping relationship between the gyroscope’s axes and the phone ©Authors

We also wish to record the virtual birds generated by users and send them to a metaverse platform as unique digital assets to maintain users’ emotional connection.

As mentioned earlier, the first version of NER has 768 variations. This is because it is convenient for the factory to make the physical bird toys and more controlled distribution for first-time NFT collectors. We expect the later version of NER to increase the number of traits and the number of physical birds.

The authors believe that “NFTs can be integrated with physical assets, properties, securities and insurance, thereby promoting diversified applications in art markets” ⁷. Generative art and AI art, such as ChatGPT ⁴ and Midjourney ¹⁴, will play a significant role in future NFTs and their visual aesthetics and artistic values will be continuously improving. More details of our work can be viewed in the supplementary video at https://we.tl/t-x2G7ksWbDJ, that demonstrates to a clear method of creativity to our collectors and inspire other creators. 

References

1  70s melt, https://www.shadertoy.com/view/XsX3zl.

2 M. A. Boden, E. A. Edmonds, What is generative art? Digital Creativity 20(1-2), 2009, 21–46.

3 M.-L Bourguet, Designing and prototyping multimodal commands, In Interact, volume 3, 2003, 717–720.

4 Chatgpt, https://chat.openai.com/.

5 Cryptokitties official website, https://www.cryptokitties.co/.

6 Cryptopunks - collection, https://opensea.io/collection/cryptopunk.

7 Y. Guo, Q. Liu, J. Chen, W. Xue, H. Jensen, F. Rosas, J. Shaw, X. Wu, J. Zhang, J. Xu, Pathway to future symbiotic creativity, 2022, arXiv preprint arXiv:2209.02388.

8  J. Harkin, Mobilisation: The growing public interest in mobile technology, Demos, 2003.

9 Living architecture by refik anadol, https://www.casabatllo.es/en/mapping/.

10 looklabs, https://www.looklabs.com/.

11 M. McLuhan, Understanding media: The extensions of man, MIT press, 1994.

12 A. Mekacher, A. Bracci, M. Nadini, M. Martino, L. Alessandretti, L. M. Aiello, A. Baronchelli, Heterogeneous rarity patterns drive price dynamics in nft collections. Scientific reports 12(1), 2022, 1–9.

13 Merge by pak official website, https://www.niftygateway.com/collections/pakmerge.

14 Midjourney, https://www.midjourney.com/.

15 Opensea official website, https://opensea.io/.

16 S. Park, Multimodal data portrait for representing mobile phone use behavior, In Proceedings of the 25th International Symposium on Electronic Art (ISEA), 2019, 36–43.

17 J. Parkes, Look labs creates ”world’s first digital fragrance” as nft, 2021, https://www.dezeen.com/2021/04/08/look-labs-digital-fragrance-nft/.

18 Rarity tools explained: A guide to ranking rare nfts, https://nftnow.com/guides/rarity-tools-explained-a-guide-to-ranking-rare-nfts/.

19 Rarity tools explained, https://nftnow.com/guides/rarity-tools-explained-a-guide-to-ranking-rare-nfts/.

20 rarity.tools, https://rarity.tools/.

21 Shadertoy beta, https://www.shadertoy.com/.

22 Spacedao official website, https://thespace.game/

23  Artblocks official website, https://www.artblocks.io/.

24 Superrare official website, https://superrare.com/.

25  Z. Wu, and L. Huang, Mimicry: Genetic-algorithm-based real-time system of virtual insects in a living environment-a new and altered nature, Proceedings of the ACM on Computer Graphics and Interactive Techniques 4(2), 2021, 1–8.

26  K. Zhang, J. Yu, The computer-based generation of fonts in the style of kandinsky. Leonardo 54(4), 2021, 437–443.

27 K. Zhang, S. Harrell, X. Ji, Computational aesthetics: on the complexity of computer-generated paintings, Leonardo 45(3), 2012, 243–248. 

Authors Biographies

Ziwei Wu is a media artist and researcher who was born in Shenzhen. She is a Ph.D. student in Computational Media and Arts (CMA) at HKUST. Her artworks are mainly based on biology, science, and the influence on society by using a range of media. She is a Lumen prize winner; Batsford prize winner. Her research was published in the SIGGRAPH Art Program and EVA London. She has exhibited at international venues, including CYFEST in Southeastern Europe, Norwegian BioArt Arena (NOBA)-Vitenparken in Norwegian, Watermans Gallery and Cello Factory in London, NeurIPS AI Art Gallery, ArtMachine2 in Run Run Shaw Creative Media Centre in Hong Kong and so on. 

Danlu Fei is a game developer and researcher. She owns a bachelor degree in Computer Science - Computer game. She worked as a gameplay programmer intern at several well-known game companies, including Ubisoft, Behaviour Interactive, and Netease. She is pursuing her Ph.D degree in Computational Media and Arts (CMA) at HKUST. Her research interest is mainly around procedural content generation (PCG) in games, especially AI-based PCG. Her research was published in the CHI Play conference proceedings.

Xinyu MA is an art practitioner residing in Shenzhen and Hangzhou, who holds a bachelor’s degree in sculpture and a master’s in fine arts. Currently, he is pursuing a master of philosophy degree at the Hong Kong University of Science and Technology (Guangzhou). His work focuses on experimenting with various forms of public system structures, investigating their modes of operation in an individual, fictional, and fictitious way. In addition, he also researches the topics of the public sphere, philosophy of technology, urban space, community, play, and political aesthetics.

Mingming Fan is an Assistant Professor in the Computational Media and Arts (CMA) Thrust at The Hong Kong University of Science and Technology (Guangzhou) and in the Division of Integrative Systems and Design at The Hong Kong University of Science and Technology. Dr. Fan is the director of the Accessible and Pervasive User EXperience (APEX) Group, which conducts research at the intersection of Human-Computer Interaction (HCI), AI, and Aging & Accessibility. Dr. Fan’s research has been published in top-tier venues in HCI (e.g., CHI, TOCHI) and received many paper awards, including the best paper award at CHI and the best paper honorable mention award at UbiComp. He is a Siebel Scholar. Homepage: https://www.mingmingfan.com/

Kang Zhang is Professor of Computational Media and Arts, Hong Kong University of Science and Technology (Guangzhou) and of Division of Emerging Interdisciplinary Areas at the Hong Kong University of Science and Technology, and Professor Emeritus of Computer Science at the University of Texas at Dallas. He was a Fulbright Distinguished Chair and ACM Distinguished Speaker, and held academic positions in USA, Czech Republic, Australia, and UK. Dr. Zhang has delivered keynotes in computer science, architecture, art, design, and psychology areas, and won numerous art and design awards. Interdisciplinary research topics of computational aesthetics and aesthetic computing, and their differences and relationships. Homepage: https://cma.hkustgz.edu.cn/people/kang-zhang/ 

Towards a Relational Model of Co-located Interaction in Interactive Art

Dan Xu, Maarten H. Lamers, Edwin van der Heide

LIACS, Leiden University
Leiden, the Netherlands
d.xu@liacs.leidenuniv.nl, m.h.lamers@liacs.leidenuniv.nl, e.f.van.der.heide@liacs.leidenuniv.nl

Abstract

Co-located interaction refers to situations in which there are two or more co-located audience members interacting with an interactive artwork and the interaction between the audience members is integral to the artwork. To better describe and understand co-located interaction, we propose a relational model of co-located interaction with an emphasis on how the interacting elements relate to and influence each other based on the actions performed by the elements and the various forms of communication at play. We explain the key concepts in the model and demonstrate its application with an example artwork. Our model provides a systematic approach to de- scribe, compare, and generate new forms of co-located interaction and audience-artwork interaction in general.

Keywords

Co-located interaction, interactive art, audience-artwork interaction, interaction models, interactivity, social interaction, mediated communication, relational aesthetics, interdependence.

DOI

10.69564/ISEA2023-92-full-Xu-et-al-Co-located-Interaction

Introduction

Active audience participation is integral to interactive art. Instead of creating static artistic objects, artists aim to construct dynamic 'art systems' capable of acting and responding to the audience and the environment. ¹ Interactive art transforms the audience from passive viewers into active participants, challenging the traditional conceptions of the roles of artwork, audience, and artist. Situated at the interplay of art, science, and technology, it explores and recontextualizes new technologies, contributing to novel forms of communication and the emergence of a multidisciplinary field of research. ²

Meanwhile, the majority of research in interactive art focuses on describing the interaction between the audience and the artwork, a concept commonly referred to as audience-artwork interaction. ³ In a comprehensive study of audience-artwork interaction within interactive arts, Schraffenberger and van der Heide conducted a review of existing models and frameworks that delve into the various characteristics and types of interaction occurring between the artwork and the audience. ³, ⁴ They further proposed a dialogue model to describe audience-artwork interaction, similar to a dialogue between two individuals, where both parties not only respond to each other's propositions but also have the freedom to change the topic or interrupt one another. The dialogue model portrays audience-artwork interaction as a dynamic process, granting both the audience and the artwork autonomy while reacting to each other. As such, the artwork and the audience neither act in a random manner nor completely control each other.

However, as many interactive artworks are exhibited in museums and other public places, there are usually more than one audience members present, and some artworks require the presence of multiple audience members for the intended interaction to take place. The artwork Boundary Functions by Scott Snibbe is such an example. ⁵ In Boundary Functions, the audience are cast as seeds for generating a Voronoi diagram projected on the floor, which divides the floor into areas that are closest to each audience member. The artwork does not respond when there is only one person, as a Voronoi diagram for one seed covers the whole projection area. When there are two or more people interacting with the artwork, the boundaries of each cell are determined by the distance between each audience member and are continuously changing as they move closer or further away from each other. Here we use the term co-located interaction to describe such interaction:

Co-located interaction takes place when two or more audience members participate in the interaction with an interactive artwork, sharing both time and physical location. Furthermore, the presence of, and the interaction between, the audience members is integral to the artwork.

Consequently, in accordance with this definition, co-located interaction encompasses both the interaction between the audience and the artwork, as well as the interaction among audience members. It's important to note that not all interactive artworks incorporate co-located interaction. Unlike a conventional dialogue between an audience and an artwork, co-located interaction broadens the scope of interaction into a collective activity involving multiple participants and various types of relationships.

In this paper, we propose an approach to describe and model co-located interaction in interactive art. A comprehensible interaction model can help us better understand, compare, evaluate, and design interactive experiences and systems. ⁶ Furthermore, such a model can offer templates to assist artists in developing novel forms of co-located interaction, aid curators and experts in the comparison and classification of artworks, and, most importantly, contribute to the identification of patterns within co-located interaction, shedding light on how interactivity manifests itself in such contexts and revealing new areas for research.

In the next section, we review existing approaches for modelling co-located interaction, discussing both their contributions and limitations. Following this, we introduce our model for co-located interaction, explain the key concepts, their properties, and provide guidelines for its application. Subsequently, we show how our approach can be applied in describing and analysing co-located interaction with an example artwork. Our general discussion, reflection and conclusion can be found at the end of this paper.

Related Works

Existing Models of Co-located Interaction

One of the main contributions in understanding co-located interaction in interactive art is the doctoral work of Oussama Mubarak. ⁷ In his research, Mubarak presents a taxonomy of co-located interaction in art installations based on factors influencing both the individual and the collective experience. These factors are:

Participation symmetry: all audience interact with the artwork equally (symmetric) or some audience interact with the artwork differently from others (asymmetric).

This taxonomy points out some interesting aspects about the experience of co-located interaction and provides a set of vocabulary to describe them. However, it concerns less with specifying the relationships between the audience and the artwork, as well as between the audience members. Additionally, Mubarak proposes two approaches to model colocated interaction that attempts to capture this.

The first approach focuses on visualizing the system layouts of art installations. In this approach, the main elements in a model are shared or individual interfaces that are specified as either input, output, or both. The number of each interface is noted, and the organization of the individual interfaces are classified as 1) free, there is no specific order for accessing the interfaces, 2) in a queue, the interfaces are accessed by the installation in a chronological order, 3) in a looped queue, the art installation loops through all the interfaces. The data flow between the individual interfaces and shared interfaces are indicated as either unidirectional, bidirectional, or symmetrical correspondence meaning all interfaces share the same information. This approach provides a concise language to show the technical layout of artworks, yet it offers limited insight into the roles each interface plays in the interaction and their connections to the audience. More importantly, it does not account for the interaction between audience members, which is essential to co-located interaction.

Acknowledging some of the limitations discussed above, Mubarak proposes a second approach based on Petri nets. A Petri net is often used to model information flow in systems with concurrent and asynchronous events. In this approach, Mubarak describes the spatial and material aspects of the artwork, the audience activities in the interaction as well as the transitions between them triggered by the actions of the audience. Comparing with the first approach, this approach is more inclusive and expressive of the different activities in a co-located interaction, as Mubarak noted that it “can be used to model co-located human-human, human-machine, as well as human-machine-human interactions around art installations”. ⁷, p.110 However, due to its early developmental phase, this approach tends to produce highly specific and complex model of an individual artwork, which limits its practical use and renders it unsuitable for comparing different co-located interactions.

Interaction Model for Relational Interactive Art

When contemplating the promotion of audience participation in interactive artworks, Cabrita and Bernardes connects interactive arts with relational aesthetics, a concept centered on fostering social connections among audience members through an artwork's aesthetic properties. ⁸ They coined the term “relational interactive art” to describe interactive artworks that strive to forge and enrich social bonds between the audience members and proposed an interaction model for such artworks based on feedback loops (see Figure 1).

Figure 1 Model of relational interactive art. Adapted from ©Cabrita & Bernardes

In a feedback loop between a human and a computer, the actions of the human send signal to the computer, which gives feedback that guides the further actions of the human in a continuous loop. In relational interactive art, the model includes another human participant that interacts both with the human and the computer in similar feedback loops. Cabrita and Bernardes further explain that the human-human and human-computer interaction are equally important and always present in relational interactive artworks. The actions of the audience are interdependent, and these actions are interpreted by the computer as a collaborative behaviour that “guides its response and shapes the next instance of human action” ⁸. However, they did not illustrate any application of the model.

Here we see a great overlap between Cabrita and Bernardes account of relational interactive art and co-located interactive artworks. We agree that the interaction between the audience members and the interaction between the audience and the artwork are of equal importance and the two forms of interaction also inform and influence each other. However, the feedback loop model is often used in HCI to describe how a user can control a system and neglects the possibilities for novel behaviours and independent actions, whereas if we understand interaction as a dialogue, there is freedom on both sides to break the loop and go in another direction. Moreover, contrary to Mubarak’s approach based on Petri nets, the feedback model runs a risk of oversimplification, and cannot capture the characteristics of individual interactions.

Considerations

Taken together, the existing models of co-located interaction tend to be either too generic or too specific. A useful model should be descriptive, in that it can provide enough structural information to describe the individual system or situation being modelled; comparative, in that it contains metrics for comparing different systems or situations; and generative, in that it can guide the creation of new systems or situations. ⁶

Meanwhile, although interaction is by nature a relational process, the existing attempts to model co-located interaction do not sufficiently describe the relationships between audience and the artwork, and between the audience members. In all above-mentioned models, the term ‘interaction’ is used to describe the activities between the audience members and between the audience and the artwork. In discussing forms of interaction in the context of interactive art installations, Ahmed pointed out the difference between interaction and communication: “In communication, the receiver may or may not respond, whereas in interaction, there is requirement of a response for it to be an ‘inter’action”. ⁹ This also coincides with our understanding of interactivity as a form of mutual responsiveness. Although the experience of co-located interaction is interactive, this does not mean that each interacting partner should be mutually responsive to each other.

If the interaction between an artwork and an audience can be seen as a dialogue of actions and reactions, we can try to understand co-located interaction by looking at the actions (and reactions) of all the interacting participants. By following the directions of actions, we can start to identify the different forms of communication at play. In doing so, we can specify a network of influences between the audience members and the artwork that serves as the foundation for understanding co-located interaction.

Moreover, the same action of an interacting element can yield different outcomes in different interactive artworks. For instance, the waving of an audience member's arm may change the colour of a visual display in one artwork, while in another, the same gesture may cause the artwork to move away. Therefore, it is crucial to consider the function(s) of an action performed by an interacting element and how it affects and relates to other elements, or the role(s) of the action in the specific context of a co-located interaction. Existing models of co-located interaction have generalized the role(s) of actions as either ‘triggers’ for activity transitions or ‘feedback’ while ignoring the fact that an action can also be to initiate the interaction or a new proposition. Therefore, we posit that specifying the role(s) of the various actions performed by the elements is critical for comprehending the dynamics of co-located interaction and further identifying the role(s) of individual elements. This not only aids us in distinguishing and comparing different artworks but also unveils patterns that can stimulate the creation of novel forms of co-located interaction.

Therefore, we need a balanced model of co-located interaction that can describe the various forms of communication between the audience and the artwork, as well as between the audience members, while considering the roles of the actions of the audience and the artwork. Next, we present our attempt to model co-located interaction, taking these considerations into account.

A Relational Model of Co-located Interaction

Here, we introduce a relational model for describing co-located interaction. A visual diagram of this relational model is provided in Figure 2. The relational model begins by identifying the elements with distinct behaviours and proper within a co-located interaction. An element is regarded as an independent actor in the interaction, with their actions forming the foundation of the interaction. Common elements encountered in co-located interactions are audience members or art systems. The model is relational in that it does not only depict the behaviours and properties of individual elements, but also specify how their actions influence and relate to other elements through various forms of communication. In the following sections, we will elaborate on the attributes we use to describe the elements and the different forms of communication with the relational model. For each element, we describe its: 

Figure 2 Visual diagram of the relational model.

Type

As mentioned before, there are two common types of elements in co-located interaction: the audience and the art system. In Cabrita and Bernardes’ model of relational interactive art, they view the interactive system of the artwork as simply a 'computer' ⁸. However, we find this term misleading, as an interactive artwork often involves more than just a computer; other components of the artwork can be equally, if not more, significant in the interaction. Additionally, interactive artworks do not always require the use of computers. Therefore, we propose the term 'art system' to describe an independent functional unit within an interactive artwork. An art system can consist of input units, output units, and processing units, similar to the computer element in Cabrita and Bernardes’ model, or it could be made of physical materials that react to the environment and can be used by the audience. An interactive artwork might include a single art system or several art systems, each with its own behaviour and the capacity to establish connections with the audience. When multiple art systems exhibit distinct behaviours, we need to analyse them individually.

The audience element refers to the individual audience members. In an interactive artwork, there are often audience members interacting with the artwork while others remain in the background to observe the interaction, sometimes they have the freedom to switch between the two modes of participation. As our focus is on describing interaction, the model includes only the audience members who are directly participating in the artwork and whose actions have an impact on the art system and other audience members. Looking at the definition of co-located interaction, there are often two or more audience members in a co-located interaction. Similar to the case of the art system, when different audience members exhibit distinct behaviours and have different functions in the interaction, we need to specify each of them with the model.

In certain instances of interactive artworks, the art system is influenced not only by the audience but also by other environmental factors ¹. For example, an art system might change its behavioural states in response to the exhibition space's temperature or internet traffic data. When such factors come into play, we categorize them as types of environmental elements.

Components

The components of an element refer to the devices, materials, and apparatuses that the element incorporates and employs during interactions. If an art system is computer-based, it typically encompasses both hardware components, like sensors, computers, actuators, or displays, to sense the environment and execute actions, and software components such as tracking and control programs for processing data and generating commands. For the audience, their interaction often involves using their bodies to perceive, act, express, and communicate with both the art system and each other. Depending on the interaction modalities, they can also utilize objects like accessories or personal devices. The components of an element describe its material basis and help us differentiate it from other elements.

Count

Count refers to the number of elements that can simultaneously participate in the interaction. This is similar to the 'scale' attribute in Mubarak's taxonomy; however, he only focuses on the number of simultaneous active audience members, whereas we extend this notion to all types of elements. In certain instances, there might be a requirement for a minimum number of elements in a co-located interaction. For example, the previously mentioned artwork 'Boundary Functions' requires the participation of a minimum of two audience members. Moreover, a co-located interaction can take place among a fixed number of elements, or the elements can join or leave the interaction at any moment. In the latter scenario, we often denote the minimal required number of elements for the interaction with a '+' to indicate potential additions.

Actions

An action is a concrete step performed by an element or a group of elements to bring about changes within themselves or in the surrounding environment. An action can be a movement, such walking or pressing a button, or it can be an update on a display mode, such as presenting data on a screen or generating an audio. In the relational model, we view actions as the foundational components that elements use to establish connections with and influence other elements. Consequently, the actions included in the description of an interaction are always directed towards other elements. Activities that are receptive, such as sensing or observing are not explicitly mentioned. The direction of an action can manifest itself in two ways: first, the acting element intentionally performs an action directed at the receiving element; second, the receiving element actively captures aspects of the action performed by the acting element, even if the action is not initially intended for the interaction. The latter is often the case when the art system detects and responds to specific actions of the audience.

In a taxonomy of interactive artworks made in the context of the Prix Ars Electronica, Kwastek points out that the key actions that may be performed by the audience are: “observe, explore, activate, control, select, participate, navigate, leave traces, co-author, collaborate, exchange information and create” and the corresponding actions made by the art system are “monitor, serve as an instrument, document, enhance perception, offer a game, enable communication, visualize, sonify, transform, store, immerse, process, mediate and tell/narrate”. ¹⁰ However, we find these terms are more suitable to describe the functions of an action instead of the concrete actions performed by the elements. We agree that it is important to specify the functions and roles of an action in the interaction. Yet when it comes to describe the co-located interaction in a specific artwork, we first need to identify the individual, concrete actions performed by the elements.

Communication

As we discussed earlier, a form of communication is created when an element performs an action directed at another element. For each form of communication, we specify:
To - To which element the action is directed at. The same action can be directed at different elements and creates different forms of communication.

Means - The means of a communication refers to how the action reaches the receiving element. We identify two different means of communication:
• Direct: the action is performed to the receiving element directly. There is no intermediate element.
• Via (intermediate element): The action is performed to an intermediate element and reaches the receiving element via the intermediate element. This type of communication is frequently termed mediated communication.

Configuration - In a communication, an action can be carried out by one or multiple elements together and it can be directed at one or multiple elements simultaneously. Configuration refers to the arrangement of the elements at both ends of the communication. We consider the possible configurations in a co-located interaction as:
• One to one: from one element to one element only.
• One to many: from one element to one or more elements.
• Many to one: from one or more elements to one element. 
• Many to many: from one or more elements to one or more elements.

Besides the number of elements, we also identify two settings in which the communication takes place: private and public. In a private setting, the communication can only be perceived by the communicating elements. In a public setting, the communication can also be perceived by other elements and/or observers.

Count - Here the count refers to the number communications that can take place in parallel. It is important to note that this does not necessarily imply that all communications must happen strictly simultaneously. Rather, we are examining the capacity of the element to perform the action. For instance, an art system may detect the audience members one at a time and emit an audio response. While the communication of the audio response might be sequential, the art system has the ability to repeatedly perform such action. Therefore, we consider that there are multiple instances of such communications happening in parallel.

Role(s) of the Action

For every action performed by an element, it's essential to define its role(s) within the interaction. The role(s) of an action pertain to the function(s) it serves and how it relates to other actions performed by the same or different elements in the interaction. An action can be initiated by an element or triggered by the actions of other elements. In the latter scenario, it functions as a response and can be termed a ‘reaction’, As we discussed before, the ‘actions’ identified by Kwastek provide a rich vocabulary to describe the functions of an action in an interaction. However, in Kwastek’s description, the actions of the audience tend to play more active roles such as “explore, activate, control”, whereas the artwork tends to be at the service of the audience. In our view, we do not make any arbitrary differentiation between the roles of the audience and those of the artwork. We consider that both can perform the same roles, and that this perspective opens up possibilities to create new forms of interaction. As an action can create multiple forms of communication, it can also serve diverse functions across different communications.

Application

Given a co-located interactive artwork, we can apply the model to describe its co-located interaction by inspecting the documentations and description of the artwork, as well as observing the actual interaction taking place. First, we need to identify how many distinct element profiles to include in the model. We can do so by examine the elements according to the general types (e.g., art system, audience, environment) and further distinguish whether the elements of the same type exhibit distinctive behaviours that require separate analysis. For the element(s) sharing the same profile, we can then specify its properties and behaviours based on its ‘components’, ‘counts’ and ‘actions’. Following each actions of the element, we can start to describe the various forms of communication created by the element: to which element(s) the action is done to (‘to’), whether the information is communicated directly or via other element(s) (‘means’), how many elements are involved in both ends of the communication and whether it is in a public or private setting (‘configuration’), how many of such communications can take place concurrently (‘count’) and what role(s) the action plays in the context of the interaction (‘role(s) of the action’). This should give us an adequate description of a co-located interaction focusing on how the elements relate to and influence each other. 

Example Artwork

Body Movies (2001) by Rafael Lozano-Hemmer

Body Movies is an interactive projection installation for public spaces (see Figure 3). ¹¹ A set of portraits is projected on a surface and washed out by lights positioned at a distance on the floor. The audience enters the interaction space and their shadows are displayed on the projection surface to reveal the portraits. The artwork tracks the edges of the shadows, and once they overlap with a portrait, a hotspot is activated for a few seconds and an audio track is triggered. When all portraits are revealed, the artwork blacks out the projection and displays a new set of portraits at the different locations from the previous ones. Meanwhile, the software interface is displayed in real-time via a plasma display next to the projection.

Apply the Relational Model

In Body Movies, there is one art system consisting of: projector, portrait, projection surface, Xenon light, plasma display, camera, software program, speaker, robotic controller. As all audience members participate in the interaction equally, we can describe them in one element profile. One or more audience members can participate at the same time, and they mainly use their bodies to interact. A visual diagram of the co-located interaction in Body Movies based on the relational model can be seen in Figure 4.

There is a form of direct many (1+) to one communication from the audience to the art system in a public setting and there can be one or more of such communications. In this, one or more audience members can cast a shadow on the projection surface and the camera captures and tracks the shadow contours, which allows the audience to initiate and participate in the interaction.

There are four forms of direct one to many (1+) communication from the art system to the audience in a public setting. First, the art system displays the shadows of the audience as a response, to provide feedback about their movements, and a stage for communication. And there can be one or more of such communications. Second, the art system also displays the tracking interface to provide feedback and inform the audience of their performance and the states of interaction. Third, when a shadow overlaps with a portrait, the art system activates a hotspot and plays an audio clip as a response and to provide feedback about the achievement. And there can be one or more of such communications. Last, once all hotspots are activated, the art system blacks out the projection and updates the portraits as a response and initiates a new session of interaction.

Throughout the interaction, the audience can communicate with each other through their shadows. It allows them to express and perform with gestures and movements. This creates a form of many (1+) to many (1+) communication via the art system in a public setting and there can be one or more of such communication forms. Due to their co-location, the audience can also converse with each other both verbally and non-verbally to exchange information and coordinate actions. As there are two or more audience members, there can be one or more of such direct many (1+) to many (1+) communications in a public setting.

Figure 4 Visual diagram of the co-located interaction in Body Movies.

Discussion

In this paper we propose a relational model to describe co-located interaction in interactive art. Our approach drew several connections to concepts discussed in Mubarak’s taxonomy, such as scale and participation symmetry. We situate them in context with relation to other elements and supplement the description of the individual element and their communication with each other. Compared to Mubarak’s models, our approach looks beyond the technical aspects of the artwork, and not only describes the activities and actions of the audience and the art system, but also specifies the relationships between them. The model also provides a systematic method that can be used to easily compare different artworks. Instead of viewing all forms of communication and potential interaction as feedback loops, we specify the organizations of the elements, the flow of information and the roles of the actions performed by the elements. In doing so, we have a better understanding of how the elements influence each other and distinguish different forms of co-located interaction.

The descriptive capacity of the relational model is illustrated in the example of Body Movies. Given the description of the co-located interaction in Body Movies generated using the model, we can see that there are multiple forms of communication between the audience and the art system. The audience members initiate the interaction by casting their shadows on the projection surface. In response to this, the art system performs multiple responses, each with different functions and takes place corresponding to different scenarios, for instance, it plays a sound when the audience members activate a hotspot or updates the portraits when all hotspots are activated. These various forms of communication enrich the interactive experience and engage the audience to explore the possibilities the artwork affords.

Moreover, we also observe two forms of communication between the audience members. They can communicate with each other directly due to their co-location and/or via the art system using their shadows. The shadow play allows the audience to communicate with gestures while concealing their appearance. This anonymity lowers the threshold for public performance. Additionally, they can move closer or further away from the lights to drastically alter the size of their shadows, which creates dramatic and symbolic effects in the shadow play. These forms of communication can either facilitate the audience's interaction with the art system or become an aesthetic experience in itself.

Meanwhile, our approach also provides a template to conceive new forms of co-located interaction. For instance, we can combine different forms of communication or swap the roles of actions performed by the elements to create new structures for co-located interaction. Given the identified forms of communication, we can also start thinking about new forms of communication. For instance, an audience-mediated communication in which an audience member can interact with an art system through another audience member, or an audience-group communication in which an audience member acts on and responds to other audience members and art systems as a group. Besides, we can also speculate new ways for the different forms of communication to influence each other. For instance, a mediated communication can disrupt or supplement to a direct communication.

Future Works

In this paper we can only partially illustrate how our approach can be applied to analyse and describe co-located interaction due to space constraints. In a follow-up publication, we will analyse and compare a variety of co-located interactive works to further test the capabilities of our approach and the diversity of co-located interaction.

For the current approach, we identified several common roles of actions in co-located interaction. This serves as a starting point to see co-located interaction through the lens of these concepts and points at a direction for future studies to examine and develop more conclusive inventories of the roles, which can be applied to describe, compare, and conceive new forms of co-located interaction.

As our approach focuses on examining the relational aspect of a co-located interaction, it is limited in accounting for other aspects such as the scale and feedback attributability according to Mubarak’s taxonomy. Moreover, it cannot show the temporal development of an interaction. The audience may be more exploratory at the onset, as they discover and master the interaction mechanism they may act and react differently. Future studies can combine different frame-works and taxonomies with our approach for a more comprehensive description of co-located interaction.

Conclusion

Co-located interaction in interactive art provides challenges and opportunities for developing new forms of interaction and relationships between the audience and artwork. In this paper, we present a relational approach to model co-located interaction with a focus on describing the various forms of communication between the interacting elements, while considering the effect of their actions in their context and relation to each other. Looking at co-located interaction with a relational perspective helps us to better understand the dynamics of co-located interaction and to conceive new forms of relation and organization between the audience and an art system.

Furthermore, despite its original focus on co-located interaction, the relational model does not impose specific prerequisites for the nature of the interaction or the elements under analysis. The model can also be effectively extended to describe various other types of audience-artwork interaction. This extension includes artworks featuring distributed audience participation and/or asynchronous communications, as well as interactions involving diverse entities encompassing both human and non-human participants. We believe that the relational model represents a pivotal step towards comprehending the complex network of relationships intrinsic to interaction. 

References

1 Stroud Cornock, Ernest Edmonds, “The Creative Process Where the Artist Is Amplified or Superseded by the Computer,” Leonardo 6, No. 1, 1973, p.11, Doi: 10.2307/1572419.

2 Myounghoon Jeon, Rebecca Fiebrink, Ernest Edmonds, Damith Herath, “From Rituals to Magic: Interactive Art and HCI of the Past, Present, and Future,” International Journal of Human Computer Studies 131, 2019, 108, Doi: 10.1016/j.ijhcs.2019.06.005.

3 Hanna Schraffenberger, Edwin van der Heide, “Interaction Models for Audience-Artwork Interaction: Current State and Future Directions,” In Arts and Technology, edited by Anthony L. Brooks, 101:127–35. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, Berlin, Heidelberg, Springer Berlin Heidelberg, 2012, Doi: 10.1007/978-3-642-33329-3_15.

4 Hanna Schraffenberger, Edwin Van der Heide, “Audience-Artwork Interaction,” International Journal of Arts and Technology 8, No. 2, 2015, 91, Doi: 10.1504/IJART.2015.069550.

5  Scott Snibbe, “Boundary Functions (1998)”, accessed December 5, 2022, https://www.snibbe.com/art/boundaryfunctions

6 Michel Beaudouin-Lafon, “Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces,” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM Press, 2000, 446-453, Doi: 10.1145/332040.332473.

7 Oussama Mubarak, “Designing and Modeling Collective Co-Located Interactions for Art Installations,” Ph.D. diss., Center for Studies and Research in Computing and Communications, CNAM Paris, 2018.

8  Nuno Cabrita, Gilberto Bernardes, “Relational Interactive Art: A Framework for Interaction in a Social Context,” In Expressive, edited by Angus Forbes and Lyn Bartram. Computational Aesthetics in Graphics, Visualization, and Imaging, The Eurographics Association, 2016, Doi: 10.2312/EXP.20161264.

9 Salah Uddin Ahmed, “Interaction and Interactivity: In the Context of Digital Interactive Art Installation,” In Human-Computer Interaction. Interaction in Context, edited by Masaaki Kurosu, 10902:241–57, Springer International Publishing, 2018, Doi: 10.1007/978-3-319-91244-8_20.

10 Katja Kwastek, “Interactivity – A Word in Process,” In The Art and Science of Interface and Interaction Design, edited by Christa Sommerer, Lakhmi C. Jain, Laurent Mignonneau, Studies in Computational Intelligence, 141:15–26, Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, Doi: 10.1007/978-3-540-79870-5_2.

11 Rafael Lozano-Hemmer, “Body Movies (2001)”, accessed December 5, 2022, https://www.lozano-hemmer.com/body_movies.php.

Authors Biographies

Dan Xu is a creative researcher and PhD candidate at Leiden Institute of Advanced Computer Science (LIACS), Leiden University, NL. Her current research explores co-located audience interaction in and through interactive art. She holds a MSc in Media Technology from Leiden University and BSc in Applied Physics from Beijing Institute of Technology.

Maarten H. Lamers is a creative computer scientist and Assistant Professor at Leiden University (LIACS). Themes that reappear in his research and lecturing are artificial intelligence, human-robots interaction, hybrid bio-digital systems, and scientific playfulness.

Edwin van der Heide is an artist and researcher in the field of sound, space and interaction, and a part-time Assistant Professor at Leiden University (LIACS, ACPA). His work comprises installations, performances and environments. The audience is located in the middle of the work and challenged to actively explore and interact with the artwork.

VR CALLIGRAPHY Transposing Chinese calligraphy as choreographed movements into whole-body performances in VR

Shum, P. Y. S. & Klein, T.

School of Creative Media, City University of Hong Kong Hong Kong
Pysshum2-c@my.cityu.edu.hk, ktobias@cityu.edu.hk

Abstract

Current research in the area of digitizing Chinese calligraphy is primarily concerned with computer graphics (CG), calligraphy ink/brush simulation, human-computer interaction (HCI), and automatic calligraphy generation. In all above research fields, calligraphy is mainly expressed and represented in two dimensions as traditional ink on paper. Even within recent contemporary calligraphic movements, there is little exploration into a true spatialization of calligraphy beyond the flat medium.

This research articulates an evolution of Chinese calligraphy, from a physical two-dimensionality into a multidimensional digital form via choreographed whole-body movements using VR (virtual reality) interfaces. We present a series of experiments to test the transposition, creation, and presentation of the essential qualities of calligraphy into a VR format. The aim is, through disseminating new virtual phenomenology in calligraphy, to contribute to and extend the existing writing on the relation between calligraphy and technology as an art entering into immersive environments and interactive virtual spaces. Thus, the research interrogates the dimensionality of calligraphy to synthesize the notion of shufa (书法, the way of writing; the Chinese word for calligraphy) from the perspective of a choreographed performance within VR.

Keywords

Chinese Calligraphy, Digital Craftsmanship, VR, HCI, Art and Technology, Performance Art, Body Movement, Intangible Cultural Heritage.

DOI

10.69564/ISEA2023-93-full-Shum-et-al-VR-CALLIGRAPHY

Introduction

This paper focuses on calligraphy as a creative art form. The Chinese word for calligraphy, shufa (书法), can be translated as the way of writing. It is more than a technical act of putting ink on paper; it is a core philosophy of Chinese culture, recognized by UNESCO (2009) and on the Representative List of the Intangible Cultural Heritage of Humanity. Therefore, this paper searches within the traditional aspects of calligraphy to determine the essential qualities of this art that can be either retained or transferred into a digital, multidimensional format. Through our experiments, we test specific notions of calligraphy with the utilization of originally-developed tools within VR.

This paper is structured into three parts. In the first part, we analyze Chinese calligraphy as an art form that is creatively expressive. The philosophical context, and the potential transformation of tools from physical to digital versions, are investigated via discussions of dao/qi and The Spirit of Calligraphy. The expressive qualities of calligraphy are essential to its art and are therefore retained within our transferal process. From an examination of traditional tools and media, we identify the unique aspects of calligraphy that are to be transposed in our experiments: the dynamic strokes of ink written via corresponding body movements. By establishing the limits of the physical tools and media, we propose how calligraphy can potentially evolve in VR, reflected within an immaterial and time-based media.

The second part showcases our experiments of two artworks as investigations of calligraphy written by choreographed movements in VR. From the identification of the inadequacies of current software and hardware, we propose our own setup designed to specifically retain the artistry of shufa, and at the same time transform the tools and media digitally. This is demonstrated through the writing of sculptural, multidimensional, expressive, and choreographed calligraphy “in the air” with fingers using hand-tracking technology as data input. And, due to the multidimensionality (3D space + time) of this new form of writing, we discuss innovative ways of approaching calligraphy structure, writing order, and spatial alignment within our tests and artworks.

The third part of the paper discusses the experiments as three points of reflection. We consider VR Calligraphy within the notion of time against the linearity of focus when writing calligraphy. On the evaluation of choreographed body movement within our writing process, we reference performance arts such as dance and kung fu to determine areas for improvement. Lastly, we reflect on the experiments as a new form of immersive calligraphy experience and speculate how this could enhance appreciation of this art. 

Searching within Tradition: Energies, Tools, Body Movements

Figure 1. Shu 书 by Law, C. B., 2016.

An Act of Creative Expression

Figure 1 shows the writing of Shu (书, derived from the word shufa) ¹ , and is written by Ching-Bor Law, a Hong Kong–based master calligrapher. It visually represents the energetic flow of ink whilst keeping an upright and balanced structural harmony. This is a creative expression of the writer’s interpretation of the character.

Xiongbo Shi explores calligraphy as a summative expression of spiritual, intellectual, cultural, artistic, and emotional energies ². This complex can be referenced as geist (spirit) in the West and represented by yi (意, intent) in Chinese. Further, Shi separates intent into two categories: Youyi (有 意, being intentional, conscious) and wuyi (无意, not being intentional, unconscious).

By encompassing Geist/Yi ( 意) intent during writing, brush strokes evoke ideas of representation and personal meaning. According to Shi [ibid], a piece of traditional calligraphy can be considered artistically excellent when the summation of subjective meanings is perceived as beautiful. This relation between calligraphy and traditional beauty has also been raised by Chang through comparison to surrealism ³, and in the study of contemporary calligraphy by Barrass, who distinguishes the Modernist and Avant-Garde Chinese calligraphy movements based on their critical stances towards traditional beauty, conceptual abstraction, and experimentation of media ⁴.

Philosopher Yuk Hui, in his seminal work The Question Concerning Technology in China, explores the relationship between technology and Chinese philosophy. Central to his argument, in Confucianist and Daoist Chinese culture, dao (道, way/order) [is] the supreme order of being” ⁵, and Qi (器, tool/vessel), when applied specifically to calligraphy, provides the technique or technology to develop form. The notion of harmony is pursued in this relation within dao/qi. This link between art and dao is further explored in The Spirit of Chinese Art by Fuguan Xu: “Conceptually we can only use what they call dao to define the spirit of art, but we cannot use the spirit of art to define dao. Because dao also has a speculative dimension, logically speaking, it is broader than art” ⁶.

Within our experiments, we aim to continue the pursuit of The Spirit of Chinese Art. The intent is to evolve the tools and media of calligraphy, melding its material-based phenomenology together with its philosophical duality to retain its artistry, and at the same time, transfer its essential qualities to an immaterial and dynamically time-based version in VR.

Evolution: Tools and Range of Body Movements

Calligraphy is a performance, a dynamic and evolving art form with constantly changing tools, and thus requiring different body movement forces to create. Within the history of Chinese calligraphy, each evolution of the medium required different tools and skill sets of its time. Chinese calligraphy started as pictographic symbols known as oracle bone script (c. 1200 BCE) ⁷. Characters were chiseled out of turtle shells using a carving knife. Here, the writer’s arm exerted a downward striking force, perhaps with the aid of a hammer. This technique evolved through the use of bamboo, bronze, wood and silk, as media into its current form of ink, brush, and paper since c. 100 BCE.

With ink and brush, the arm exerts only a slight downward pushing force, with longer strokes written through horizontal arm movements. This enabled an increased range of body movement with greater precision. As a result, progressively expressive styles of calligraphy were written, leading to the development of the standard five scripts (see Figure 2 below):

1. Seal script (Zhuan shu) - in use from c. 1200 BCE.
2. Clerical script (Li shu) - from c. 200 BCE.
3. Regular script (Kai shu) - from c. 200-400 CE.
4. Cursive script (Xing shu) - from the 4th century CE. 5. Drafting script (Cao shu) - from the 7th century CE. 

Figure 2. The character shui ( 水, water) in the five scripts. http://www.columbia.edu/~xc2282/calligraphy/calligraphy.html

Each script of calligraphy, with its distinctive stylistic differences, requires the hand and wrist to coordinate and “perform” in different ways: from the more controlled, symmetrical action of the Seal script to a spontaneous and fluid action of the Drafting script. This performance of the hand and arm is partnered by the tool and medium of brush and ink.

Focusing on the tools, the traditional Four Treasures of the Study (文房四宝) consists of the brush, ink, paper, and seal. The standard calligraphy brush is usually made of a bamboo shaft which holds the brush tip, traditionally comprised of animal hair. Species variation of animal hair results in different rates of ink absorption, which is compounded by the length and shape of the brush tip. The length of the writing stroke is limited by the amount of ink that the brush hair can hold combined with viscosity (i.e. water content) of the ink liquid, and the rate of absorption of the paper. The width of the writing stroke is determined by the length of the brush hair.

One of the skills of traditional calligraphy is the careful control of ink dipping with the brush, and the precise addition of water to the ink as it dries within different room temperatures and humidities. It is a deeply time-sensitive process, often executed in one breath, linking the body to the art of writing and the materiality of the tools.

The third materiality in calligraphy is xuan paper, whose composition mainly consists of tree bark and rice straw. Different compositions of raw materials and production techniques can lead to various rates of absorbency and smoothness (friction) of the paper surface. These physical properties of paper directly interact with the ink and brush during writing. The bounds of writing, the dimensions of paper produced, reflect the range of motion available to the arm. This is represented in standard paper sizes of 4 feet (70x140cm) or 6 feet (100x180cm).

Figure 3. Chiang (1938), p.121. Brush holding posture.
Figure 4. Chiang (1938), p.122. Brush manipulation for writing strokes.

The skill in writing calligraphy is, in essence, the fine choreography of the arm, hand, and finger muscles resulting in a transferal of the ink-absorbed brush tip onto the absorbent paper surface. In Chinese characters, there are long, short, vertical, horizontal, dot, curving, hooked, angular strokes, each requiring a different set of precise combinations of finger, wrist, and arm movements, as illustrated above by Chiang ³ . Understanding of these fine muscle movements and their resulting dynamic ink effects is critical to the process of transferring movements from the arm to the whole body.

These are the physical tools and media of traditional calligraphy with their cultural lineage. The aim of our experimentation is to conceptualize a new toolset and media in VR that bears the qualities of ink, brush, and paper. This is an evolution that retains tradition yet extends new possibilities within multidimensional space and synergy of new virtual materiality.

Experiments: VR Calligraphy

Setup

Within VR software, painting and sculpting apps have been generally available for a few years now, most notably Tilt Brush for painting and Oculus Medium for sculpting. Whilst these are excellent applications for their intended purposes, their functionality does not align with the characteristics of calligraphy. Tilt Brush was developed as a platform to emulate a Western mode of painting, akin to oil painting, where brushstrokes are applied layer upon layer to create patches of color. Chinese calligraphy, however, focuses on the expressivity of individual strokes: the freely changing widths, precise curvatures, the speeds of brush strokes, together with ink effects such as wetness, blackness, and ink bleeding. This calligraphy-centric level of control and expression is presently not achievable in any VR app. 

Figure 5. Inadequate trigger button of VR controllers, Shum, 2022.

Likewise in VR hardware, current platforms have settled for a generally standardized blueprint of a head mounted display (HMD) with hand-held controllers, which include a trigger button for the index finger, a joystick, and four buttons for the thumb. This design satisfies most apps: the trigger button (index finger) for grabbing and triggering ingame actions, the thumb buttons to offer other functions and menus. However, this standardized UI (user interface) only provides a trigger button offering 1cm of travel with very little force resistance (Figure 5). Thus, the VR controller is inadequate to support the finesse necessary in our transfer of the skillful controls of brush and ink and does not suit the expressive qualities of calligraphy writing.

Due to the lack of existing HCI interfaces for this kind of expression, we developed a new system to write VR calligraphy. In our experiments, the hardware is based on the Meta Quest 2 VR platform, and the application is created within the Unreal Engine platform. Our system uses new hand tracking functionalities and only requires the use of cameras on board the VR HMD to capture the hand and fingers for gestural input, as shown in Figure 6 below. By using bare hands, there is no longer any physical tool to be held. This approach has merit, as it first enables users to forgo the associated learning curves for interacting with physical tools, and secondly provides a primal, intuitive sense of creation. This links users back to the art stylings of prehistoric times when humans wrote with their fingers on cave walls, as reflected by David Hockney and Martin Gayford in their tracing of media and techniques from art history ⁷.

Figure 6. VR HMD with hand and finger tracking.
Figure 7. Menu of adjustable calligraphic parameters.

In order to transfer the nuances of a calligraphy brush into VR, we developed a program to measure the distance between the thumb and index finger of the right hand to control the thickness of the written stroke within VR. The thumb touching the index finger (a pinch gesture) results in zero thickness: no ink generated, no stroke drawn. These two fingers pulled apart widely results in a thick stroke.

In regard to our VR version of calligraphy ink, we designed a particle generation system to simulate a three-dimensional fluid ink/smoke effect. A separate menu (Figure 7) was scripted for the left hand using virtual adjustable menu sliders to adjust, in detail, important visual and calligraphic parameters such as brush roundness, ink wet/dry effect, ink bleed, gravitational pull, opacity, color, and time decay (the duration for the ink to vanish). With VR ink, users can, in practice, write strokes of infinite length. This opens up new creative possibilities, not just with length of stroke but also with the fluidity of movement, as the writer no longer has to dip the brush to replenish ink, which interrupts the flow of the writing movement.

Concerning calligraphy paper, there is no longer need for its flat dimensionality (2D). Instead, VR allows for writing in 3D space, which can be as large as the computational power affords. The physicality of paper absorbency is now simulated within our menu system. The friction of writing on paper, whilst impossible to simulate with our use of hand tracking, is now extended by the new possible interactions of our fingers, arms, and the entire body in 3D space, captured by digital hardware and visualized as immaterial virtual calligraphy within our computer program.

Shi and Curved Shi: The Third Dimension

First, we tested one of the fundamentals of writing in the alignment and thus continuity of the characters. In 2D, this is not a problem as the writing is perceived only from one angle. Therefore, our first two tests demonstrate the difficulty of text alignment and three-dimensional readability. The criteria are to have a change of writing plane and an added depth dimension whilst retaining legibility from a “front” angle. For clarity, the three dimensions refer to cartesian dimensionality using the X, Y, and Z axes, with conventional calligraphy on paper being 2D with writing on the X-Y axes.

For the retention of character legibility in 3D, character strokes that conventionally join should also intersect at the Z axis, but this act alone is difficult in practice. Writing in the air, as opposed to on a table in conventional calligraphy, has no resistance. As humans are accustomed to writing on a flat surface, the act of joining lines within the Z axis is a novel phenomenon.

View-point specific, three-dimensional Chinese calligraphy does not currently exist. Some Chinese character-based sculptures exist, but these are mostly reliant on the meaning of the character and do not focus on the actual readability of the dynamism involved in writing shufa. Because of this lack of precedent, we had to experiment and conceptualize our own rules of transposing two-dimensional calligraphy into three dimensions. 

Figures 8 & 9. Test 1 - Shi (十), Shum, 2021.

The above figures of Test 1 - Shi illustrate this point with one of the simplest Chinese characters. Whilst from the front view (Figure 8), both A and B versions of the same character read shi (十, ten), the side view (Figure 9) shows that version A is joined at the Z axis, but in version B, the two strokes are separated, thus not legible as a single character from this angle of view. This test only concerns writing straight lines within 3D. Our next test discusses the difficulty of writing calligraphy “sculpturally”.

Figure 12. Hu (虎, tiger), Shum, 2022

In Test 2 - Curved Shi, the aim is to write the same character sculpturally, as if imprinted onto a sphere. The first stroke, the horizontal stroke, is written in a way that is curved towards the writer. However, the next vertical stroke has to start on the same Z axis plane as the start of the horizontal stroke, make the same curvature to meet the midpoint of the horizontal stroke, and end back at the same Z axis plane as the end of the horizontal stroke. 

Figures 10 & 11. Test 2 - Curved Shi (十), Shum, 2021. 

Figures 10–11 demonstrate the two strokes correctly joined together in version C. In version D, the two strokes fail to join together and cannot be legible as the intended character. Writing in 3D requires a constant shift of view, thus movements of the head and body are required to align multiple strokes. And, the order of complexity is compounded for characters with an increasing number of strokes. Our next step is to write a single character as calligraphy within the discussed contexts of tools, spirit, and dao, with our determined ink effects, and testing the full range of body motion.

Hu - Choreographed Performance Experiencing Whole Body Writing

Figures 12–14 show our artwork for Hu (虎), the character for tiger. A video is available to demonstrate the three-dimensional and sculptural quality of the writing at https://vimeo.com/773627966. Similar to Shu, by Law (Figure 1), we wanted to express the dynamic flow of writing with ink effects, an upright structural balance, and incorporate the playfulness of the tiger by the long, upturned tail of the last stroke (blue stroke of Figure 13). 

Figure 13. Hu, side view and writing order. Figure 14. Hu, top view. 

Hu is written at around 1.5m in terms of real-world height to test the applicability of a full range of body motion. It involves a carefully choreographed sequence of foot, body, and hand motion for the character to be written as intended. To illustrate, Hu is written with three strokes. The first stroke (red in Figure 13) starts at the top left position, then the writer’s hand must move towards the right with a slight push forward to reach the corner position. To achieve the long tail of the stroke, the writer takes a step back with the left foot whilst bending the waist for the arm to reach the end of the stroke. To begin the second stroke (green in Figure 13), the left foot returns to a standing position, and as the hand writes downwards, the writer starts taking a long stride backwards with the right leg in anticipation of the next stroke. The third and final stroke (blue in Figure 13) is a long downwards and backwards sweep, which requires the right foot to finish a step back, and the waist and head turning with the arm to complete the stroke.

Our writing of Hu required 88 attempts across three days before we achieved a satisfactory version. This was because we wanted to retain the same focus of writing as in traditional calligraphy. Our writing experience was akin to conventional calligraphy, where each error results in a complete restart of writing in accordance with our aims to preserve the notions of dynamic ink and the linearity of time and focus. Hence, we retain the dao of calligraphy whilst executing the transferal of brush, ink, and paper as digital versions, writing an expressive interpretation of the Hu character in VR.

Counterpoint: Curse 

Figure 15. Curse, Shum, 2022.
Figure 16. Curse, detail. Figure 17. Curse, top view. 

Having previously pursued a harmonious writing of Hu, we wanted to test how the same toolset could be used in a contrasting way. Thus, we wrote Curse (人骂我, 我骂人也, “They curse me, hence I curse them”) shown in Figures 15– 17), inspired by the famous phrase by calligrapher Baishi Qi (齐白石). This was written during a period of lockdown in China to express the frustration of the times.

Here we replaced traditional black ink with neon pink ink against a night sky setting. The carefully balanced proportions of the previous Hu were disregarded; instead, we wanted to convey expressions of rawness and chaos. The characters were written circularly around the writer, who pivoted 360 degrees around the same location (Figure 17). This work of multiple characters has no “front angle”, and therefore can only be read by moving around it or moving into the center of the characters and turning around in full circle to look at the text. This experimental subversion of some of the traditional values of calligraphy serves as a stepping stone for branching out into more unconventional calligraphy visualizations.

Simultaneously, the constantly changing widths of the writing strokes are controlled via finger movement (Figure 16): the distance between the thumb-tip and index-finger-tip of the right hand. The ink particles are generated at a fixed rate: a faster hand movement results in less particles, a more transparent ink effect, and a slower hand movement results in a denser ink effect. Therefore, the speed of writing and the time for ink drying is visually rendered as an evolved form of VR calligraphy. This preserves the essence of time-based calligraphy and expresses the dynamism of ink with- out the necessity of physical materiality of ink, brush, and paper, within the non-material medium of VR.

Reflections

Non-Undo, Linearity of Time

A prevalent benefit of common digital creative apps is the functionality of undo, which allows users to easily fix mistakes. The notion of undo itself is powerful, in essence giving the user control of time. The ability to erase materiality (go back in time) or re-create it (skip forward time) instantaneously introduces disruption not just in time but also interrupts the linearity of the creative process and the fluidity of focus when writing.

In reflection of writing Chinese calligraphy, undo is controversial. Traditionally, mistakes are non-erasable, so calligraphers write with such focus to prioritize expression as well as to avoid mistakes; when mistakes are made, the piece of paper is usually discarded. Our experiments go against the practice of common creative productivity apps by completely forgoing the implementation of undo. Mistakes are inerasable, resulting in us requiring 88 attempts to write Hu. Through this intentional absence of the computational undo, the writer requires a similar focus using VR Calligraphy when compared to traditional calligraphy. Additionally, the writer needs to pay attention to the new complexities of writing in multiple dimensions (3D space + time) through coordinated movements of every part of the body.

With the physical tools of calligraphy now replaced by a “set of invisible algorithms” ⁷, there is no longer a need to dip the brush to replenish ink. Our experiments introduce a continuity of time compared to traditional calligraphy, speculatively deepening the focus during calligraphy writing. 

Dance/Choreography and Calligraphy

In our writing of Hu, we found that our considerations of precise body movements were similar to that of performance arts.

We wish to point out the exemplary choreographies developed using calligraphy as inspiration by Shen Wei Dance Arts in Connect Transfer (2004). As Gerdes points out, Shen Wei explores the “body as calligrapher, [...and] displays both the body in constant motion and the painted brush strokes as indications of this motion, thus visibly portraying product and process simultaneously” ⁸. The dance routines incorporate ink applied to the hands and feet, with paper laid out across the floor or wall. Body movements are reflected visually as ink splatters on the paper. It would be of interest to us in the future to not just demonstrate written calligraphy but also visualize such trails of body movement to further augment appreciation.

Leach notes the recent drive for choreographers who seek to utilize technology to reveal and disseminate previously non-visualizable structures within choreographies as choreographed objects ¹¹ . Notably, Forsythe and deLahunta actively seek to “make the value of dance available ‘without the body’” (ibid, p.470). Synchronous Objects for One Flat Thing, Reproduced by Forsythe in 2009 is an excellent example of how the production for One Flat Thing, Reproduced can be systematically broken down into visually easily understandable diagrams, such as movement density, 3D alignment forms, and difference forms.

Similarly, Kung Fu Visualization by Shaw and Kenderdine in 2016 set a precedent for how intangible motion- based Chinese arts can be digitally investigated through motion capture and analysis of movement forces to reveal new phenomenological insights into the craft, for purposes of both analyzing and understanding past works in multiple dimensions.

Learning from Foresythe’s Synchronous Objects and Shaw’s Kung Fu Visualization, we can consider the breaking down and visualization of our artworks with diagrams of footing, stance, head position, hand movements, and movement speeds, to better illustrate the complexity of coordination required in our experiments. In addition, the dimensionality of time, the live real-time performance of calligraphy, can be visualized and animated as part of the appreciation process. Our VR Calligraphy can be digitally recorded as particle data and as body motion data and replayed in real-time as the viewer enters the virtual space. The visualization of this symbiosis of movement and writing, together with the immersive intimacy of VR, would strive to reveal a deeper understanding between the choreography, body forces, and the written calligraphic forms together as a real-time performance.

New Experience in Appreciation

In Curse, we created a new way for calligraphy appreciation and experience. Traditionally, calligraphy is appreciated by having the work statically hung on a wall. Our multidimensional focus in Curse, however, actively encourages the viewer to explore calligraphy from various angles to appreciate the sculptural form of it. We view this enablement of proximity to be a powerful experience. It negates the notion of distance between artwork and the viewer, as written by Grau ¹², and allows for an increasingly intimate and immersive appreciation of the finer details of the calligraphy: the expression of strokes, the dynamism of ink, and the fluidity of movement behind the calligraphy. Our investigation has established a change of dimensionality from 2D to 3D immersion, where the visual stimuli is greater and more diverse, resulting in what Dewey refers to as more “refined and intensified” forms of experiences in calligraphy appreciation and interaction ¹³.

Conclusions

Our experiments successfully demonstrate the transposition of whole-body movements into a multidimensional and digital form of calligraphy. We have made a step forward in developing our own program and interface to write this new format of calligraphy. Our findings within the notion of time, undo, and linearity of calligraphy writing have been positive thus far. We would like to have more in-depth dialogue with calligraphy experts to evaluate the outcomes. For improvements, we are so far satisfied with the visual and research outcome of our VR Calligraphy as a manifestation of carefully choreographed sets of movements. We feel that there is opportunity for the traces of body movements to be additionally visualized, as our comparisons with dance and kung fu has shown, to better illustrate the intricate movements involved with easily understandable visual explanations.

It is also important to note that although the body movements and medium of our experiments are different, our aim for calligraphy within The Spirit of Calligraphy remains the same. We have maintained a pursuit of harmony in the discourse of dao/qi through our carefully considered usage of a new toolset and careful transferal of the intricacies of ink and brush effects.

Additionally, we have demonstrated that this new mode of calligraphy appreciation enables an immersive experience by encouraging spatial exploration of the works, thus expounding the perception of calligraphic art. Whilst the medium of our calligraphy has evolved, we have also expanded the level of geist/yi in the creative expressivity within our experiments due to our enhancement of the focus of writing without needing to replenish ink, and with our decision to forego undo. Lastly, we wish to confirm our retainment of the practice of harmony within the dao/qi of calligraphy as we continue to seek an evolution in The Spirit of Calligraphy in our research. 

Looking Forward:

Immersive Collective Calligraphic Speculations

Our artworks of Hu (tiger) and Curse are only pertinent to the calligraphy of a single character written at a human height (1.5m) scale. It is technically feasible to write using similar hand and finger input with a scaling functionality to magnify the body movements to create larger-scaled works, which is the next step of this research. This would enable the creation of an increasingly immersive experience, one that curates the arrangement of different scaled calligraphy to create an assortment of spaces: big vs small, open vs closed, wide vs narrow, bright vs dim, straight vs curved.

If we also speculate at the possibility of co-creating such VR Calligraphy and associated qualities of time, dimensionality, scale, expressiveness, and immersive experience together with multi-player–calligrapher input, this could amalgamate into a collective calligraphic performance. And, referencing the choreography visualizations mentioned, also illustrate the performers or avatars writing with their body movement traces as a multi-layered spectacle.

Through our research, we hope to demonstrate a successful case for symbiosis between the art and performance of calligraphy with the technological potentiality of VR technology. We shall endeavor to continue in our experimentation to present Chinese calligraphy as a contemporary continuation and extension of its cultural heritage.

Acknowledgements

The work described in this paper was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China. This research is part of the ACIM Research Fellowship.

References

1 Law Ching-Bor, The Calligraphy of Law Ching-Bor, Hong Kong, Yuen Long District Arts Council, 2016, p.91.

2 Shi Xiongbo, "Chinese Calligraphy as Force-FormJournal of Aesthetic Education 53, no. 3, 2019, 54-70.

3 Chang Yee, Chinese Calligraphy, Harvard University Press, 1938, p.101.

4 G. Barrass, The Art of Calligraphy in Modern China, Berkeley, University of California Press, 2002.

5 Hui Yuk, The Question Concerning Technology in China, Urbanomic, 2016.

6 Xu Fuguan, The Spirit of Chinese Art, Collected Work of Xu Fuguan, vol. 4. Hubei, Hubei People’s Publishing House, 2009, p.44.

7 X. Qiu, G. L. Mattos, J. Norman, Chinese Writing. Early China. Society for the Study of Early China and the Institute of East Asian Studies, University of California, Berkeley, 2000.

8 David Hockney, Martin Gayford, A History of Pictures: From the Cave to the Computer Screen, Thames and Hudson, 2016, 354.

9 Xu Songhuan, C. M. Lau Francis, Yunhe Pan, A Computational Approach to Digital Chinese Painting and Calligraphy, Springer Science & Business Media, 2009, 335.

10 Ellen V. P. Gerdes, "Shen Wei Dance Arts: Chinese Philosophy in Body Calligraphy", Dance Chronicle, 2010.

11 James Leach, "Choreographic Objects." Journal of Cultural Economy 7, no. 4, 2014, 458-75.

12 Oliver Grau, Virtual Art - from Illusion to Immersion, The MIT Press, 2003.

13 John Dewey, Art as Experience, Penguin, 1934.

LAVIN: An AI-Navigated Art Experience in Virtual Reality

Weidi Zhang, Jieliang Luo

Arizona State University, Autodesk Research Arizona, USA; California, USA
zhangweidilydia@gmail.com, rodgerljl@msn.com

Abstract

This paper outlines the conceptual background, design methodology, and future directions of LAVIN. This virtual reality (VR) artwork provides an immersive experience to visually explore one understanding of a neural network in which the real-world maps to 50 daily objects. In the art installation, the neural network constantly analyzes the surrounding environments via a camera and outputs real-time semantic interpretations, which navigate the audience in a virtual world consisting of all the fluid abstract structures of daily objects that the neural network can recognize. We create these fluid virtual structures using data visualization, photogrammetry, and 3D modeling. By merging Artificial Intelligent (AI) system design with VR world-building, LAVIN offers an immersive art experience for symbiotic imaginations that questions the values and beliefs in the modern AI age.

Keywords

AI System Design, Virtual Reality, Immersive Media Design, Generative Art, Interactive Art Experience.

DOI

10.69564/ISEA2023-94-full-Zhang-et-al-LAVIN

Introduction

In the last decade, object detection has made tremendous progress. In some narrow domains, the algorithm has almost reached human-level accuracy ¹ . Each neural network's output is a unique projection of its understanding of the real world, regardless of whether it has been trained to recognize thousands of objects or only produces binary outputs ² . Regardless of the complexity of the projection, it shapes the “world value” of the neural network ² . Numerous deep neural networks are implemented for various intelligent applications. Despite this, even the most complex neural network model is not capable of representing the real world entirely. A neural network used for facial recognition, for example, can only interpret the input as a collection of faces ² . Thus, our design question arises: what is the ground truth in the modern AI age?

As a conceptual response to the question regarding the worldview and value of neural networks, LAVIN provides one understanding of a neural network by combining VR worldbuilding and AI system design. We design our AI system to observe surrounding environments in the gallery space via a camera and output one word to describe the live-streaming based on the neural network’s interpretation. The output word is the navigation information that travels the participants in the VR in real time. The AI system is trained to only understand 50 daily objects which leads to its misinterpretation of the live-streaming information. The VR world is made up of these 50 objects our AI system can recognize. LAVIN aims to provide an immersive art experience that is navigated by an AI system through visualizing the imagined world of a neural network artistically. 

Figure 1. LAVIN, Virtual Reality Art Installation, Siggraph Art Gallery at LA convention center, ©Weidi Zhang. 

Background

The real world is a sophisticated collage with diverse groups of creatures, overloaded data, a myriad of choices and chances, fleeting messages, complex shifting meanings, and distinct stories, cultures, and political systems. If we perceive the images of our world as a visual representation that is programmable, networked, and operative, what is the method to associate different visual information? How do we use an algorithmic approach to create a new kind of chance and choice operations in the art-making process? Who will be the author: machines, humans, or human-machine hybrid? What are the data-driven material, forms, and aesthetics? How will the audiences interact with this assemblage, and in which way will their decisions change the experience?

Seymour Chatman states that: In this age of mechanical and electronic production and reproduction, it would be naive to reject the notion of nonhuman narrative agency, for instance, a story constructed through algorithmic processes or as a result of multiple voices constructing together in real-time ³. With the goal of integrating a non-human narrator for immersive storytelling, we designed an AI system that navigates this VR journey based on its real-time understanding of the surrounding environment. The customized machine reinterprets the real world to build topographical associations between objects in a VR journey.

Related Artworks Worldview of A Neural Network

Interactive works employing machine learning algorithms often offer a user-centric and adaptive experience. Using autonomous systems, responsive applications can assemble artifacts, discoveries, traces, and experiences based on the prior knowledge of the machines and their interpretations of audiences' inputs.

New media artist Memo Akten’s series of interactive works—Learning to see—“use state-of-the-art machine learning algorithms to reflect on ourselves and how we make sense of the world” ⁴. In the art installation, Memo uses a live camera to capture everyday objects and uses a number of neural networks to analyze the live-streaming feed. The audience is invited to move the objects under the camera and see the visualization evolve based on the changing composition of the moving objects. The visualization changes every 30 seconds to different networks trained on different datasets: water, air, earth, and cosmos. Memo Akten considers the way of seeing in the conscious mind as a reconstruction based on expectations and beliefs instead of a mirror image of the outside world ⁴.

Similarly to Memo Akten's Learning to See, LAVIN emphasizes the importance of machines' prior knowledge and their worldviews in presenting how machines interpret the world using an artistic approach. LAVIN differs from the relevant artworks in that it does not directly display machine interpretation as visual output, but rather uses it as navigational information for engaging the author’s artistic imagination of a neural network’s mind. LAVIN combines AI’s decision with artistic originality to invent a virtual world of symbiotic imagination.

Navigation in VR Journey

Traditionally, immersants (the participants who wear the head-mounted device ) can use VR controllers to navigate a virtual space, which often requires a learning curve. At the same time, there are many VR artworks that investigate conceptual and novel navigation methods.

For example, the pioneer VR artwork, Osmose, created by Char Davis in 1995, uses bio-data of breathing to transport users to different elevation places within a virtual environment. Osmose presents a multi-sensory immersive experience that engages visitors in a virtual world made of simulation of nature scenery and text. Immersants can navigate themselves in the virtual space through body breathing—a chest-hugging rubber vest that contains sensory devices sensitive to the body’s breath—a natural interface ⁵. This interface works similarly to diving—when you inhale, you rise. Osmose juxtaposes the insecurity of body presence and an emotional sense of tranquility.

Osmose precurses to the booming industry of VR as a platform of narrativity and entertainment in the 21st century. It also challenges the way we view pictorial artworks. The infinite poetic space is unfolding like a canvas waiting for artists to infuse new ideas. A 3-dimensional space adds one dimension to the 2-dimensional pictorial work for the multi-media assemblage. The exploration of the relation between 2-d elements changed into the examination of neighboring topological notions within a dynamic spatial volume.

Similarly to Osmose seeking an innovative way of VR navigation for artistic purposes, LAVIN creates an intelligent system to contribute to the dynamic flow of the spatial assemblage. Real-world images are connected with virtual space for a unique way of seeing and controlling.

Design Methodology of LAVIN System Design

LAVIN provides a conceptual and artistic response to the questions of the Ground Truth in the modern age of AI by designing the navigation system based on a live neural network’s real-time interpretation of live streaming data of the surroundings. 

Figure 2. System Map of LAVIN. ©Weidi Zhang. 

For our AI system design, we specifically use ResNet50, “a pre-trained deep neural network that can be altered for different use cases” ⁶. ResNet 50 networks have been trained on the large datasets from Imagenet and “modify the last layers to quickly produce models to tackle new problems” ⁷. We build a custom dataset of fifty daily objects and trained a neural network to only recognize the selected fifty daily objects. When an image contains objects outside LAVIN’s vocabulary, its limited training dataset leads to a possible false interpretation of the given image.

In the art installation (see Figure.2), LAVIN constantly chooses one object to describe live streaming data captured by a camera (set in the center of the exhibiting space). The real-time one-word description travels an immersant to the related objects in the virtual space. Our intelligent system (LAVIN) triggers absurd but interesting results, which transports the VR immersant to experience how a limited machine perceives reality.

VR Worldbuilding

The immersive virtual space is a spatial assemblage and an artistic imagination of the neural network. LAVIN incorporates both real-world data and artistic manipulation to present a symbiotic imagination. This multi-layered, bizarre, and unknown VR space is mainly designed by addressing the following three aspects: 3d reconstruction, texture development, and spatial design.

The virtual reality environment consists of fifty objects that the trained neural network (LAVIN) can recognize. These virtual sculptural objects are constructed by using the photogrammetry technique—photographs are taken volumetrically of these 50 objects, which are then analyzed, calibrated, and calculated to generate the XYZ position data of the point cloud for the 3d reconstruction (see Figure 4). 

Figure 3. Screenshot of the VR world in LAVIN ©Weidi Zhang. 
Figure 4. Screenshot of the Virtual Reality World in LAVIN ©Weidi Zhang. 

The aesthetic outcomes of the reconstructions are realistic with a great many unnecessary details for artistic visual representation. We simplify, filter, and rearrange the dataset to distort the constructions and sculpt the shape, at the same time still maintaining the characteristics to make sure the forms of these objects are recognizable (see Figure 5).

Figure 5. The Sculptural Virtual Objects of LAVIN. ©Weidi Zhang. 

We keep all fifty 3d constructed forms as sculptures with plaster material in VR. Instead of directly attaching photographs of fifty objects as textures to the 3d forms, we process images as interactive patterns floating in the virtual space, which moves in distinct directions and speeds based on audiences’ positions. The pictorial crossovers generate unexpected and poetic meanings between unrelated objects (see Figure 6). In the virtual space, immersants are revolved around the moving textures, witnessing the layers of shapes weave the artistic imagination and abstraction of the neural network.

Figure 6. Screenshot of the VR world in LAVIN ©Weidi Zhang. 

The assembled 2d pictorial artworks often navigate viewers’ gazes in the composition by drawing their attention to the correlations between unrelated elements in various materials and shapes. Correspondingly, the navigation of an assembled VR space examines neighboring topographical notions to convey meanings.

The spatial configuration of LAVIN is inspired by Paul Klee’s Fish Magic. “He creates a magical realm where the aquatic, the celestial, and the earthly intermingle” ⁸. The mysterious and sophisticated space on the canvas where fish and flora float among human beings and clock towers. The objects are in indeterminate scale, scope, and direction. 

Impacted by Klee’s creative solution for imaginative space, LAVIN completely abandons the real-world objects’ proportion, scale, and properties to emphasize the unpredictability of a wholly imagined world (see Figure 7). The fifty sculptural forms of objects are static like the artifacts displayed in the museums, while the interactively moving patterns densely and dynamically fill the void between them. Metaphors are born out of the associations between texture layers and sculptural forms. By gazing at the textures of the patterns closely, participants can root out the real-world evidence of the objects. 

Figure 7. Screenshot of the VR world in LAVIN ©Weidi Zhang. 

Sound Design and Art Installation

A crucial part of the design of this immersive experience is the use of sound. Inspired by musique concrète, using field recordings, instrumental sounds, and human voices as sound materials, We organize and process them to create sound objects. We then attach them to various static or moving virtual objects (models and patterns) to produce a phantasmagorical soundscape.

Specifically, we record Google Translate’s generic human voice reading the words of the 50 objects and attach them to the corresponding objects in the virtual space. Field recordings of cityscape were made as the raw sound materials which are sorted and organized by pruning and splicing, then transformed by using software (Ableton Live and Adobe Audition) to filter and reverberate. The processed sound objects are imported into the virtual world and attached to the different floating textures. When multiple channels of sound cross over and assemble in space, a dynamic spatial montage is created.

This work was presented in the Siggraph Art Gallery (see Figure 8) and SwissNex Gallery in 2019. The exhibition design includes a screen or two projections, a webcam, a VR headset, a speaker, and a PC. A camera is set in the center of the space to observe surroundings and send live streaming to the system. The VR headset is placed on top of a pedestal in the space. The real-time VR experience is also displayed on a LED screen or projected onto two adjacent walls. The real-time sound in VR is played through a speaker in the exhibition so both the participants and the immersant can hear.

Figure 8. LAVIN, Virtual Reality Art Installation, Siggraph Art Gallery at Los Angeles Convention Center, ©Weidi Zhang. 

One participant at a time has the chance to be immersed in VR (see Figure.9), and other participants will possibly be captured by the cameras, which determines the virtual journey of the immersant. Immersants are not given any instructions or visual cues to help them decide what to do, the AI system enables intuitive user interactions which avoid the learning curve caused by using VR controllers.

Figure 9. LAVIN, An Participant Experiencing the VR, SwissNex Gallery, ©J Astra Brinkmann. 

During the exhibition, participants usually immerse themselves in the VR world for around 10 to 15 minutes. Immersants are engaged in the VR experience using a headset and being automatically transported to different locations determined by AI’s observation. The immersants wander within the space, investigate the meaning of the abstract sculptures of objects, moving patterns, and abstract sounds they encounter, and free their imaginations of this artistic interpretation of an AI’s mind.

Discussion: Symbiosis, Metaphors, and Cognitive Assemblage

LAVIN emphasizes the symbiosis between humans and machines through contributions from artists, machines, and audiences. This cognitive assemblage formalizes a multi-layered decision-making process where the cultural layer (generated by humans) and the computer layer (generated by the software) mutually influence each other, resulting in a blend of human and machine meanings.

This fluid network begins with critical and cultural questions that determine the information collected for an AI system to learn, which forms the first layer: a cultural layer. The intelligent system makes decisions based on its prior knowledge and observation of images, forming the second layer: a computer layer. The artist's aesthetic decisions and audiences' real-time feedback forms the following cultural layers. Each layer is interconnected and adaptive. For example, participants' real-time feedback is constantly captured by the camera (imaging device), which affects the second layer of the decisions made by the AI system. The interconnection between different layers contributes to the entanglement of culture and computer layers, forming a cognitive assemblage in system design.

N. Katherine Hayles stated in Cognitive Assemblages: “Technical Agency and Human Interactions: In a cognitive assemblage, humans and technical systems are interconnected with each other. The interactions involved human cognition, which includes consciousness, the unconscious, the cognitive nonconscious, and the perceptual system. In addition, the decisions made by humans will, to some extent, determine how the technical system operates.” ⁹

In LAVIN, the output of the technical system will also, many times, affect humans’ decisions, which leads to a cognitive assemblage in the art experience, the feedback loops steer human-machine collaboration to a “more life-affirming practices as we move toward a future in which technical agency and autonomy become increasingly intrinsic to human complex systems” ¹⁰. For instance, the participants who are captured by the camera in the art installation can see the real-time projection of the virtual world. When they see the machine’s recognition of them leads the immersant travel to a specific object (for example, a cactus), the participants tend to interact with our system by placing different things in front of the camera and see machine’s real-time interpretation changing the route in VR. Therefore, when we design a technical system, we are partially designing ourselves ¹⁰.

LAVIN provides an art experience that is not only designed as a cognitive assemblage but also as a spatial assemblage, where metaphors are used to build poetic meanings and connections are employed to build metaphors. We emphasize a metaphorical approach to building connections between visual materials and forms while embracing chance and choice operation. The metaphorical approach to building organizations of media elements (text, images, sounds) consists of automatic combinations and manual compositions. LAVIN manually assembles the spatial arrangements of virtual objects our AI system can recognize in 3D virtual space. The contrast, directions, and scale of objects are intentionally exaggerated to illustrate the surreal and bizarre assemblage that suggests the 'AI's brain' within virtual reality. LAVIN is not only about the co-creation of experience but also about co-authoring the metaphors and connections, which emphasizes change and instability, fluidity, and exchangeability. 

Future Direction

When we exhibit this artwork, the most common question raised by participants is why we selected this list of 50 daily objects. Are there any special meanings associated with these objects? The curiosity raised by participants motivates us to develop LAVIN further in the future. We plan to create a more precise and curated collection of objects rather than selecting them randomly. As a result of the careful curation of objects, our trained AI system will develop a worldview that will inspire the interesting and critical questions of human-machine reality.

Conclusion

In this paper, we introduce the background, design methodology, and future direction of an AI-guided VR experience LAVIN. In addition, this paper discusses the metaphorical approach to present cognitive assemblage in system design and spatial assemblage in worldbuilding. LAVIN provides an art experience to visually explore one understanding of a neural network in which the real world is mapping to fifty daily objects. The AI system constantly observes and analyzes the surroundings via a camera and outputs one word as a semantic interpretation. This word navigates an audience into a virtual world consisting of abstract fluid structures of these fifty daily objects that LAVIN can understand. LAVIN tends to evoke audiences’ awareness regarding values, vulnerability, and beliefs within the context of AI. It also provides novel research to connect VR and AI through designing an intelligent navigation system for an immersive art experience.

Acknowledgment

We would like to thank Prof. George Legrady and Dr.Mengyu Chen for their help and support, as well as the Experimental Visualization Lab and Media Arts and Technology program (University of California, Santa Barbara). 

References

1 Zaidi Syed Sahil Abbas, Mohammad Samar Ansari, Asra Aslam, Nadia Kanwal, Mamoona Asghar, Brian Lee, "A survey of modern deep learning based object detection models." Digital Signal Processing 2022, 103514.

2 Name deleted to maintain the integrity of the review process.

3 Seymour Chatman, “What Novels Can Do That Films Can’t” Critical Inquiry 7, no. 1, 1980, 121-140. https://doi.org/10.1086/448091.

4 Akten Memo, Rebecca Fiebrink,  Mick Grierson, “Learning to see: you are what you see”, In ACM SIGGRAPH 2019 Art Gallery (SIGGRAPH '19), Association for Computing Machinery, New York, NY, USA, Article 13, 2019, 1–6, DOI: https://doi.org/10.1145/3306211.3320143.

5 Oliver Grau,"Into the belly of the image: historical aspects of virtual reality", Leonardo 32, no. 5, 1999, 365-371.

6 Chesak Renel, “CNN-vs-ResNet50-trained-on-20-images”, Kaggle, 2020, Accessed Dec 8th, 2022, https://www.kaggle.com/code/rchesak/cnn-vs-resnet50-trained-on-20-images.

7 Brett Koonce, "ResNet 50", In Convolutional neural networks with swift for tensorflow, 63-72, Apress, Berkeley, CA, 2021.

8 “Fish Magic,” Philadelphia Museum of Art, Accessed June 1st, 2021, https://www.philamuseum.org/collection/object/51027.

9 N. Katherine Hayles, “Cognitive Assemblages: Technical Agency and Human Interaction”, Critical Inquiry 43, no. 1, 2016, 32-55. https://doi.org/10.1086/688293.

10 N. Katherine Hayles, “The Cognitive Nonconscious: Enlarging the Mind of the Humanities", Critical Inquiry 42, no. 4, 2016, 783-808. https://doi.org/10.1086/686950.

Volume of Voids: Artistic Visualizations of the Disequilibrium

Weidi Zhang, Shaoyu Su

Arizona State University, University of Southern California Arizona, USA, California, USA
zhangweidilydia@gmail.com, shaoyusu@usc.edu

Abstract

Volume of Voids is a series of art projects that visualize and materialize the voids between different beings under the regulation of social distancing during the Covid-19 pandemic. It consists of two major artworks: Volume of Voids I, which creates data sculptures that visualize and materialize the voids, and Vol II of Voids, which brings an interactive art experience that emphasizes the mobility of the space-between humans. The purpose of this work is to create an artistic and conceptual response to the impact of a historical pandemic on complex social networking systems by rethinking the space-between. This paper introduces the conceptual background, design methodology, and technical implementation of this series of work, including the discussions of speculative design framework on computational art, experimental data visualization, volumetric capturing, and 3d fabrication. 

Keywords

Data Visualization, Volumetric Capturing, 3D Fabrication, Generative Art, Interactive Art Experience.

DOI

10.69564/ISEA2023-95-full-Zhang-et-al-Volume-of-Voids

Introduction

Inspired by the social distancing regulations during the Covid-19 pandemic (see Figure 1), people are disconnected physically by keeping a social distance to avoid spreading the Coronavirus. The disruption of the fragile equilibrium in our system profoundly influences our way of living, communicating, and socializing.

Volume of Voids series starts with the conceptual What-If questions we posed during the historical Covid-19 pandemic: When people keep their distance from objects and other people, what is the shape of these invisible spaces? In what ways are these negative spaces evolving as a new entity, reshaping our society as a new normal of intricate social networks? Will the form of the empty spaces of disconnection inspire new ways of reconnecting? 

Figure 1. AFP via Getty Images ©Oli Scarff. Accessed 6 June, 2020 

Our list of design questions leads to a series of thought experiments that bring an alternative reality and artistic experiences to life: 1. Volume of Voids I: A set of 3D printed data sculptures that materializes the voids between objects and human bodies (see Figure 2) 2. Vol II of Voids: An interactive art experience that visualizes the mobile voids between humans (see Figure 3).

The Volume of voids series provides artistic responses to disconnection, dualism, and the possibility for a new structure and alliance that is more resilient and at ease in its alterity. 

Figure 2. Volume of Voids documentation of one of the 3d printed sculptures, ©Weidi Zhang. 
Figure 3. Volume of Voids documentation of one of the 3d printed sculptures, ©Weidi Zhang. 

Conceptual Background: Dualism, MA (间), and Balance

The inspiration for this project comes from the concept of dualism, Yin and Yang, in ancient Chinese philosophy as well as the idea of MA in Japanese art and culture.

In Chinese cosmology, the universe is born out of a chaos of material energy ¹, which is categorized into the cycle of Yin and Yang. The beings and non-beings produce each other. These two opposite sides are possibly interconnected and complement each other ². Yin and Yang represent passive and active respectively, which exist in all forms of variation and difference in nature and world: female is Yin while the male is Yang, the growing wheat is Yang while the reaping wheat is Yin, eggshell is Yang while the yolk is Yin ¹, the morning fog is Yin while it is dissipated by the heat of the sun is Yang, the forest fire is Yang while the rainstorm extinguishes it is Yin ³. The Taoists suggest that the opposition of Yin and Yang unifies the harmony in nature. Yin and Yang not only can be found in nature, but they also exist in human bodies. Taoists believe everything is part of the oneness of the universe, and there is no difference between the internal processes of the body and the external forces of nature ³. The idea of Yin and Yang from Taoism inspires us to rethink the relationship between the voids of social distancing with the internal processes in human bodies and minds.

In Japanese culture, the concept of MA describes the connection between positive and negative space. MA refers to the artistic interpretation of emptiness. It draws people’s attention to the negative space, which helps to perceive the positive space. The empty space is often full of possibilities—for example, the artistic practices of calligraphy experiment with the concept of MA. Chinese and Japanese calligraphy not only focuses on the depiction of characters but also creates careful relationships between the forms of characters and the surrounding non-form. Calligraphy artists use the blankness to link forms and take the design of the void spaces into artistic consideration.

Volume of Voids implements the concept of MA and Yin Yang by rethinking the meanings of the space-between, the messages these silence spaces deliver, and how the voids influence the intricate networks of society and the internal processes in human bodies. 

Related Artworks

Fine artists have explored the positive and negative spaces in their poetic artworks, for example, American photographer Ellen Garven’s Ambivalence and English artist Rachel Whiteread’s artworks Ghost.

In Ellen Garvens’s Ambivalence [see Figure 4], she photographed a set of prosthetic devices. These devices reflect their history of making and the patients’ experiences ⁴. Ellen thinks the devices raise awareness of disability and its intimate relationship to all bodies ⁴. The grace of the forms, the devices’ surfaces with textures of traces, and the way Ellen stage and photograph the devices make the still life monumental, which evoke audiences’ feelings of loss and intimacy. 

Figure 4. Blue ©Ellen Garvens.

English artist Rachel Whiteread visualizes the invisible space by “mummifying the air in the room” ⁵. In her work Ghost, she cast the interior of a Victorian rowhouse in plaster and reassemble the panels facing out towards the viewers. In Whiteread’s words, the ghost causes “the viewers to become the wall” ⁶. She draws attention to the space and its inhabitants and connects objects with their surrounding area with personal stories. Her mausoleum-like work materializes the invisible spaces of everyday life powerfully and poetically.

Similar to these two artworks, Volume of Voids seeks to represent invisible spaces by giving a shape to the air. By employing computational design methodologies of data visualization, 3D fabrication, and volumetric capturing artistically, Volume of Voids distinguishes itself from fine art techniques explored previously. Generative art and computational design emphasize the fluidity and interchangeability of the voids, which reinterprets negative space in the context of the new media art practices. 

Volume of Voids I

From Photogrammetry to 3D Fabrication

The computational design method for Volume of Voids I consist of three stages: collecting data, experimental visualization, and fabricating 3D data sculptures.

We set up different news-inspired scenes for capturing. For instance, one of our scenes is based on an ABC news report about people keeping social distance during line waiting at a cafe shop. This was staged during the pandemic and volumetrically captured to collect spatial information on the voids. Overall, we staged twelve different scenes and used photogrammetry techniques to capture each of them, generating point clouds of position data for the following stages of visualization and fabrication.

Data Visualization

The photogrammetry point cloud position (XYZ data) generates multiple boundary edge curves on the XY planes (see Figure 5). These curves create planar surfaces with corresponding position data on Z-axis. 

Figure 5. Volume of Voids software view of the sculpture skeleton (top view) generated by photogrammetry data, ©Weidi Zhang. 

The planes with different heights are connected, lofted, and controlled by using algorithms (such as bezier and Perlin noise). These mathematical algorithms are added to Z-axis curves in order to add uncertainty to sculpture constructions (see Figure.6) to represent inherently unstable, fluid, and unpredictable invisible spaces between humans (see Figure.7).

Figure 6. software rendering of a sculpture ©Weidi Zhang.
Figure 7. software rendering of twelve sculptures ©Weidi Zhang.

Materialization of the visualization

To materialize the voids visualized, we then 3d printed these generative models as a series of physical artifacts (12 in total). The 3D fabrication of each sculpture of the generative 3d model is typically completed within 24 hours using PETG clear filament. We experimented with different materials and colors when fabricating the sculpture, including ABS filaments in opaque white color. During our experimentation process, we realized that rather than creating the monumental aesthetic of the voids, we tend to make the forms more transparent, light, and dynamic, which represents the accumulation of possibilities and imagination artistically. Therefore, we used the clear filament to transmit LED light, making it luminous (see Figure.8).

These luminous sculptures capture the historical moment of disconnection in a poetic and abstract way and emphasize the fragility of disconnections in social networking. The finished product measures approximately 7 to 8 inches in height and is mounted on 4-inch LED pedestals. 

Figure 8. Volume of Voids 3D printed sculpture ©Weidi Zhang. 

Vol II of Voids

After finishing the first phase of the V olume of V oids, our interest in visualizing the invisible and transforming it into an interactive experience motivated us to continue our artistic research on visualizing the mobility of voidness (see Figure.10). 

Figure 10. Vol II of Voids Visualization of mobile voids between two audiences ©Weidi Zhang and Shaoyu Su.

With the techniques of volumetric capturing, machine learning algorithms, and experimental visualization strategies, Vol II of Voids visualizes the ever-changing voids between people in real-time. Specifically, a 6-DOP volumetric capturing system fabricates the spatial literacy of reality in real time. In the new system, multiple layers of experimental visualization strategies, coupled with machine learning algorithms, can detect humans and generate real-time interactive voids between them.

Volumetric Capturing and Intelligent System Design

In order to capture the space between humans specifically, we utilized an intelligent system for human identification. Our system aims to recreate spatial reality in a human-centric way. The system is realized through two stages of development (see Figure.11).

Figure.11. Color-based merged point cloud (left), color-coded merged point cloud from four liDars (right),©Weidi Zhang and Shaoyu Su.

In the first step, we developed a ToF(time-of-flight)- based multi-LiDARs system for 3D data acquisition. (see Figure12). With the technique of affine transformation and mapping, each adjacent pair of LiDARs are calibrated in the same matrix space as master and slave so there is no limitation on the total number of LiDARs being used. As a result, a daisy-chained LiDAR array is able to capture accurate, high-density 3D point clouds of the designated 3D space in 30FPS real-time. In the current phase of this project, we use four LiDARS for capturing the installation. 

Figure 12. The volumetric capturing system utilized four lidar units ©Weidi Zhang and Shaoyu Su.

Following the first step of building a ToF-based multi-LiDAR system, we use and combine two machine learning algorithms to process the camera streaming on each LiDAR. We implement YOLACT++ ⁷ for real-time human identification in 2D and BackgroundMatting V2 ⁸ for real-time human keying into the Touchdesigner software environment. We translate 2D identifications received on texture buffers into 3D spatial data with OpenGL Shading Language, which runs on the GPU. Then we utilize the signed bounding boxes of humans to acquire accurate spatial relationships between the audiences.

There are inevitably interferences between the liDar units when they face each other. The following steps were taken to minimize that: 1. We applied genlock for hardware synchronization. 2. We programmed a denoise pattern as compensation using a GLSL pixel shader then composited it with the texture buffer storing positions of the point cloud. Finally, color channels got aligned with depth channels. 

Compared to the multi-RGBD cameras system, the LiDar system provides better fidelity and precision, even under poor illumination.

Experimental Visualization and Aesthetic Exploration

To represent the voids between audiences dynamically in real-time, we process the visualization by designing voxel grids, point clouds, and movements.The voids of the space between audiences are rendered as the following components:

The voidness between the audiences is calculated by the boolean operation between the volume of the detected humans and the total volume of the defined area. Therefore, the visualization is presented using 3D texture and compute shaders to translate space into a dynamic signed 3D voxel grid. We then calculate each work group's (on the compute shader buffer) local density of the point cloud and apply it to the global density. Inspired by cubist paintings in the 20th century, a vector field is represented by the white-colored voxels (cubes) indicating the relative movements by the distances between humans. 

Figure.13. Point cloud of the audiences (left), visualization of the voidness (right), ©Weidi Zhang and Shaoyu Su.

The movements of point clouds illustrate the stability of voids. (see Figure.14) For example, when a certain number of the audience move fast together, their locomotion will be detected to activate the movements of point clouds. It is achieved by applying a Despiking algorithm and calculating the difference from the raw data. Thus, the voxels representing voids disperse and accumulate, flee and return, converge and explode, conceptually mimicking human social behaviors.

Figure 14. Volume of Voids II, visualization of the voids between two audiences ©Weidi Zhang and Shaoyu Su. 

Sound Design and User Interaction

In the art installation (see Figure.15), audiences are captured by our 6-DOF volumetric capturing system in real-time. The visualization of the voids between audiences is presented on a LED screen set in the art installation. The audience’s body movements will alter the visualizations in real-time and generate ever-changing fluid voids in the virtual space.

Figure 15. Vol II of Voids Installation Diagram ©Weidi Zhang and Shaoyu Su. 

The background sound is non-interactive and looped in the art installation. We transform field recordings of cityscape sounds into abstract compositions through the technique of granular synth. Inspired by the musique concrete in the 20th century, Multilayered sound clips are dynamically organized in this composition to create an atmosphere of emptiness and transiency.

During the pandemic, our physical installations were canceled due to social distancing regulations. We presented this work in the virtual platform with documentation photos and videos. While the virtual exhibition is suitable for 3D-printed artifacts, it is not ideal for the interactive experience. In light of the fact that the pandemic regulations have been removed, we demonstrated our installation to the local community internally in our research lab space. It is evident that our participants are interested in the visualization and engaged in the interaction during our demonstration. Rather than providing specific instructions, we encouraged our participants to move freely within the space and observe the responsive visualization on the screen. Within a few minutes of interacting, most participants establish a connection between their existence and the visualization they are producing. Additionally, the experience creates an interesting connection between participants, they are adjusting their positions in the space in order to capture the interesting spaces between them. 

Future Direction

Presenting the voids as an interactive visualization for participants to perceive and building a collaborative empathetic experience for disequilibrium are our goals for this series of art experiments. This historical pandemic not only caused human suffering but also created new possibilities for healing and reconnection. These invisible spaces bring isolation but also unfold possibilities.

Through presenting this installation to the local communities and the online version to global communities, we are interested in developing this work further as an art experience using the augmented reality (AR) technique in a theatre setup. In the future, we will focus on the following developments: 1. We will investigate the real-time projection-based augmented reality system to present the visualization of voids as a performative element in space instead of showing the visualization directly on the LED screen. 2. We will further develop the visualization and redesign it to better fit the new format and medium.

Conclusion

This paper presents the conceptual background, technical implementation, aesthetic and design decisions, and future directions of Volume of Voids. This project consists of two artistic experiments: 1. Data-driven sculptures of the voids between humans, 2. Interactive visualization represents the voidness between participants.

Volume of Voids was initiated and developed during the Covid-19 pandemic when humans keep social distancing to avoid viral transmission in the air. The voids between humans form an evolving and unstable abstract volume. According to Taoists, beings and non-beings produce each other. Humans’ activities form the voids while the voids form the new relationships between humans. The disequilibrium caused the disconnection in the social network but popularizes interactive technology to reconnect humans in the virtual space.

Volume of voids emphasizes the value of artistic visualization to raise people's awareness of the possibilities invisible spaces offer, providing a critical response to human network transformation. Through data visualization and computational design, this work aims to provide a collective experience that emphasizes alienation between different beings and sustains our curiosity about the world, nature, and social relations.

Acknowledgment

We would like to thank Prof. George Legrady and Prof. Jennifer Jacob for their help and support, the Experimental Visualization Lab and Media Arts and Technology program (University of California, Santa Barbara) and the Immersive Media Lab (University of Southern California). 

References

1 John Bellaimey, "The hidden meanings of yin and yang - John Bellaimey". TED-Ed. Retrieved 2 August 2013, https://www.youtube.com/watch?v=ezmR9Attpyc

2 Stephan Feuchtwang, Religions in the Modern World: Traditions and Transformations, New York, Routledge 2016, 150, ISBN 978-0-415-85881-6.

3 Bill Schoenbart, Ellen Shefi, “Yin and Yang Overview”, HowStuffworks, accessed October 10, 2021, https://health.howstuffworks.com/wellness/natural-medicine/chinese/yin-and-yang.htm

4 Ellen Garvens, “Ambivalence (2003-2010)”, Ellen Garvens personal website, accessed October 10, 2021, https://www.ellengarvens.com/ambivalence

5 Unknown, “Rachel Whiteread ‘Ghost’”, National Gallery of Art, accessed October 10, 2021, https://www.nga.gov/audio-video/video/rachel-whiteread.html

6 Unknown, “Ghost Overview”, National Gallery of Art, accessed October 10, 2021, https://www.nga.gov/collection/art-object-page.131285.html

7 Daniel Bolya, Chong Zhou, Fanyi Xiao, Yong Jae Lee, "Yolact++: Better real-time instance segmentation", IEEE transactions on pattern analysis and machine intelligence, 2020.

8 Lin Shanchuan, Andrey Ryabtsev, Soumyadip Sengupta, Brian L. Curless, Steven M. Seitz, Ira Kemelmacher-Shlizerman, "Real-time high-resolution background matting", In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, 8762-8771.

Authors Biographies

Dr.Weidi Zhang is a new media artist and researcher. Currently, she is an assistant professor at Arizona State University. Her interdisciplinary art and design research investigates A Speculative Assemblage at the intersection of immersive media design, experimental data visualization art, and interactive art. She holds her Ph.D. degree in Media Arts and Technology at the University of California, Santa Barbara.

Shaoyu Su is a new media artist, developer, cinematographer and curator. His artistic practices and research employ interdisciplinary methodologies from computer graphics, photography, physics and literature, investigating concepts about deep spacetime-mining: alternative projection from ancient history to the far future. From mid-2016 to present, Su has been working at the Immersive Media Lab, USC School of Cinematic Arts as Director of Technology, leading research and production in cinematic and generative XR and real-time animation. 

The Worlds of Entanglements: Reflection on Posthumanist Ontologies in Art&Science Projects

Masha Zolotova

Linz, Austria
zolotovamsh@gmail.com

Abstract

A posthuman turn can be observed in many contemporary discourses. It is characterized by an endeavour to overcome the anthropocentric picture of the world, excluding from it the ‘gap’ between the world and human, a belief in human exclusivity, as well as the practice of organizing dichotomic systems.

The article relates the theories developed by Braidotti, Haraway and Ferrando and the notion of posthuman subject to the artistic exploration of the work with nonhuman others, trees, and fungi in particular. By talking on the posthuman subject and its ontological state, but more precisely by following the way we are interacting with nonhumans within the practice of selected artists, Olga Kisseleva and Saša Spačal, what their connections to humans are, the movement towards rethinking the role of humans and acknowledging those by whom we are surrounded can be traced. Thus, artists do not formulate their notion of posthuman subject, but rather they demonstrate another approach and attitude towards nonhumans: to consider the latter, trees or fungi in our case, as partners, not to build hierarchies because the equality of the partners and to acknowledge their ‘knowledge’ we can learn from.

Keywords

Art&science, bioart, nonhumans, more-than-human worlding, ontology, posthumanism, technology.

DOI

10.69564/ISEA2023-96-full-Zolotova-The-Worlds-of-Entanglements

Introduction

“Who that ‘we’ is, and how to keep that collectivity open, multiple and nonhierarchical.”—Rosi Braidotti. ³ 

The discussion on ecological crisis, the Sixth mass extinction, capitalistic overproduction and further problems connected to them, as well as a growing interest in postcolonial and feminist perspectives can be observed in many contemporary fields of thoughts, from scientists and academics to artists and activists. What unites these intellectual pursuits is an attempt to overcome the Cartesian understanding of a man and to go beyond anthropocentrism. But while humans, it would seem, are at the pinnacle of their domination over nature, the interconnectedness and equality of human and nonhuman life forms are becoming more generally recognized and being associated with a posthuman turn.

By working with living beings, plants or animals, artists gain an opportunity to reflect on the subject-object continuum from different perspectives. While some artists still consider living entities as an object for their investigation, others are moving forward and distinguish them as their partners or collaborators. Within this paper the focus is made on the analysis of pieces by two artists, Olga Kisseleva and Saša Spačal. They both work collaboratively with scientist and their long-term research is dedicated respectively to trees and their communicative networks within the multi-faceted project EDEN (Ethics–Durability–Ecology–Nature), and to fungi, their embeddedness in the human environment at the installation Myconnect and further the MycoMythologies.

Taking art&science field, the one that by its definition is based on the dialogue and interaction between different actors, as a prominent ground for discussion, the following question is raised in this paper: how does new understanding of the world, informed by posthumanist and new ontological perspectives, the interconnectedness between human and nonhuman agents, inform the field of art&science collaborations and reveal itself in the artworks?

Posthuman turn: more-than-human forms of existence

A posthuman turn can be observed in many contemporary discourses. It is characterized by an endeavour to overcome the anthropocentric picture of the world, excluding from it the ‘gap’ between the world and human, a belief in human exclusivity, as well as the practice of organizing dichotomic systems (such as subject and object, nature and culture, man and woman, and etc.). This term is, rather, a peripheral cultural-discursive theory in modern philosophy, that combines several social, ethical and natural science concepts, as well as developed ontological and epistemological perspectives. Following Francesca Ferrando, posthumanism is defined by three aspects: post-humanism, post-dualism and post-anthoropocentrism. Post-humanism implies a rejection of classical humanism, that is, a universal and generalized understanding of man in favour of a plurality of connotations of nature and human capabilities. Post-anthropocentrism seeks to deconstruct a centrality of human figure, and post-dualism seeks to reject the rigid way of identity, represented in the symbolic dualistic categories. At the same time, the prefix ‘post-’ in the word posthumanism does not necessarily indicate the end of human as a species. ‘Post-’ refers to all the historical and non-historical experiences of the existence of nonhuman others, both in the era before and after humanism. Thus, ‘post-’ in posthumanism moves our attention to what for centuries was excluded from the humanistic picture of the world or was subjected to repression ¹⁷. We could notice the emphasis on the analysis and critique of the notion of ‘man’ formed in the framework of the Enlightenment. Man as an autonomous being, his consciousness and the predominance of his reason, as well as his superiority over other human, animals and plants, nature in general, becomes a subject for the re-evaluation and negation.

Olga Kisseleva focuses her attention on trees: starting from the Elm of Biscarosse then Japanese cedars or Kauri trees in New Zealand. The artist was interested in establishing the missing connection among trees of the related species and facilitating their communication over the continents through specifically developed technological network, so to help them survive. Devices that measure, among others, VOC (volatile organic compounds) or barrel thickness, the basic characteristics to be captured for studying plants, have been connected through internet and thus, could pass over the signals on the distance what is not possible on a biological level without such technological mediation. So, the first iterations of the project were focused not on building contact in between human and tress, but to connect nonhuman beings with each other. The shift of the focus from tree—human to tree—tree communication, at my point of view, is a sign of putting away pre-described hierarchies and focusing on the sentience of trees at the same level of importance as the human one, an approach to provide an alternative vision. Saša Spačal, on her side, postulates that her focus placed primarily on the posthuman condition that supposes humans are not masters of the world, but rather their existence and actions are elements of a much wider and complicated ecosystem. Since the beginning of the 2010s, she has been working with different biological matters as fungi in Myconnect (2013), plants, red clover in particular, in the Symbiome installation (2016), bacteria, as Mycobacterium vaccae in Inspiration (2018) and others. By addressing the above-mentioned nonhumans and by building the artistic statement using biotechnological methods in tight connections to and with them, Spačal reveals interactions appearing between different living beings and the way they are framing up our environment ¹¹ .

The definition of the subject is a first one out of the ontological assumptions for criticism and reorientation. Current re-evaluation leads to further exploration of the boundaries, with the implicit notion that living can be attributed equally to human and nonhuman beings, or in a broad spectrum of more-than-human forms of existence. As noticed by Karen Barad, this is an opportunity to overcome the centrality of human taking into account the “differential constitution and differential positioning of the human among other creatures (both living and non-living)” ¹ .

Embedded and embodied aspect of the posthuman subject

Ontology offers a critical shorthand for exploring the characteristics of being, human or nonhuman. If previously, objects were distributed over different ontological regions and logical classes, now these heterogeneous entities find themselves on the same ontological plane. They are all actors, equal in their ability to act and interact, in which it is determined what they are and what they will be. ¹⁰ 

The relational dimension became the crucial point in defining the posthumanist ontology. Rosi Braidotti defines a posthuman subject as “a relational subject constituted in and by multiplicity, that is to say a subject that works across differences and is also internally differentiated, but still grounded and accountable”. ² Building the analysis on the relationality, she emphasized the embedded and embodied aspect of the posthuman subject. At the same time, to describe the relations between human and animal, Donna Haraway uses the notion of knot, “a knot of species coshaping one another in layers of reciprocating complexity”. ⁸ As a result, further characteristics of a new ontological state have been formulated on the basis of the theoretical convergence of the directions mentioned above: we are not existing in the dialectical world based on the opposition of external and internal, so humans are not the dualistic entities, they are rather “materially embedded subject-in-process” that are disseminated within the networks of relations with “forces, entities and encounters”. ⁵ Thus, the world is built on the interconnectedness and multispecies entanglements. We are not alone and separated, we are connected, embedded, and entangled.

Taking this point as a basis, Spačal is willing to reveal interweavings with the help of mycelium and fungi, trying to reflect on the ways fungal underground networks could inform humans and how this learning process can be mediated and shaped by technologies. The artist investigates “how fungal ontologies could help humans practice multispecies survival through practice of inclusion and caring” ¹⁴. Within MycoMythologies series the artist tries to make these connections perceptible to viewers with the help of visual and audial tools. The entanglement presented is intended to let us think about what kind of networks we are. The entanglement presented is intended to let us think about what kind of networks we are. As it was summarized by the artist,“By making new connections with other biological species and environment one might step out of anthropocentric perspective even if for just one moment to experience fresh air of other possibilities”. ¹³

The installation MycoMythologies: Rupture serves as a proof of the biotechnological connections in planetary life: the screens are linked with each other and any change in the system, the way nutrients are distributed and taken out of it, causes changes in video and audio representation, while in MycoMythologies: Patterning fungal structure becomes a great metaphor for the interconnected world: here the microbiome reveals the influence one actor could have on another. Within its framework, a tree reveals itself as a posthuman subject: embodied, and embedded, connected to other entities and being a part of a broader network that manifests itself by the relations that appear within and constitute it. The ways the project was presented in different locations around the world were chosen to show a tree as an ecologically interlinked being with multiple connections, as an equal partner that has its own perception of time and world around, holds a particular knowledge that could be shared for the good of both actors, a human and a tree. Art, this way, generates new knowledge and expresses more reasons to understand relations we are entangled in, to take more agency into shaping the world. Olga Kisseleva takes a scientific discovery, works with it within different contexts, and then presents to the wider audience by revealing internal issues we have with nature, culture, or history.

Posthuman subjects are a work-in-progress: they emerge as both a critical and a creative project within the posthuman convergence along posthumanist and postanthropocentric axes of interrogation. In its critique of the dichotomy between human and nonhuman, posthumanism offers not to prioritize one category above the other, but “embrac[e] both relationally, as intraconnected actants in an open and respondent context, which is also constantly shifting” ⁶. As it could be understood, we are not talking about separating the nonhuman into a hermetic category, but, on the contrary, about its complication, amplification; not about drawing the strict line and making a strict distinction between human and nonhuman, but about multiplying boundaries and dividing lines from within the nonhuman. Thus, the nonhuman is always already implicitly present in human: the border has already been broken, and the threshold has been crossed.

Nonhuman knowledge

Acknowledging nonhuman beings and assigning them the same ontological status, including admitting the value of their experience as well, can open up the horizons for further scientific and artistic research. Thus, the investigation on plant intelligence as a scientific track could be considered as a point in science to promote a new attitude towards a subject of investigation previously considered as a non-sentient being due to the absence of brain and neural system in comparison to animals. With the development of Critical Plant Studies further changes could be followed: plants are not considered anymore as passive objects, they are sentient beings that could respond to the changes in the environment, and they have adaptive behaviour, communication possibilities, intelligence, memory ⁷.

Sasa Spačal started to work with fungi and investigate with scientists their ability to capture and pass signals back in 2012 as a participant of a bio-hack workshop. In MycoMythologies series, a speculative artistic research project, fungi are already considered as full participants of the storytelling creation, as well as the narrative transmission to the viewers at the exhibition space. They are not the objects to be observed, but rather subjects with their own perspectives, the holders of their own knowledge to be learned from. And this collaboration with nonhumans is fuelled by technological mediation. Not only advanced software, but even the simple microscope facilitates such interactions while just making visible something that is hidden from our eyes.

The EDEN project gives voice to trees, creates the digital representation of what cannot be grasped by our senses – to be seen and smelled, to capture the flux of information. The system studies the links between digital and biological networks in real time, creating a poetic and artistic vision of the original communication, it leaves the first place to nature, “not exploited or oppressed, but listened and heard” ⁹. At the same time, it is crucial that such an exposure talking into the account the specificity of its subject. The application presented at WRO Biennale in 2019 was supposed to change colours appearing on the screen and let visitors follow how image is changing. So, they can shift their attention to the temporal aspect of the tree beings, the fact their communication is much slower than human one, how different the speed of response is. There is no clear answer or transcription of this ‘tree speech’, but it rather lets visitors experience its temporality. Trees are considered within the EDEN project not only as silent witnesses but also as the actors who can transmit this knowledge further and at some point - learn from it.

So, the knowledge we can receive from plants, animals, and natural phenomena can be valuable not only for humans, but also for the technologies we are working on. Thus, it became clear for the artist that we can learn from trees or fungi something that is not accessible to learn in any other way, so they are our partners. This way of thinking is a crucial part of posthuman convergence, as it is addressed by Rosi Braidotti, because zoe let us to acknowledge that

“thinking and the capacity to produce knowledge is not the exclusive prerogative of humans alone but is distributed across all living matter and throughout self-organizing technological networks”. ⁴

Networks of entanglements

Living organisms are highly entangled cannot be grasped without their networks of entanglements, without putting attention on the connections they have with other living beings and their environment. The most obvious example to illustrate this point is the human, in fact, not an independent living entity, but that coexists with other organisms even within itself, as Anna Tsing writes, “[h]uman nature is an interspecies relationship”. ¹⁵ Moreover, the posthumanism and non-anthropocentric model are associated not only with the decentring of the human, but also with the decentring of a subject’s position as such. It shifts the focus to the production of subjectivity, again not a fixed notion, but embedded and processual characteristic.

The first case of addressing to fungi in Saša Spačal’s artistic practice was the installation MyConnect that she defines as “a symbiotic interspecies connector that questions anthropocentric division of nature-human- technology”. ¹² Within the stand-capsule developed, the signals and impulses create a feedback loop by the means of biological and technological mediation that lets visitors experience the symbiotic interdependence between their bodies and fungal mycelium in a very tangible manner. The signal coming from the viewer’s heartbeat moves along the mycelium in petri dishes: Spačal is using either Oyster mushrooms (genus Pleurotus), or Shiitake (Lentinula edodes), and is processed by it in real time. The processed signal is transmitted back to the human body in the form of sound, light, and tactile impulses. Their impact on the nervous system causes a change in the heart rate in the viewer and starts a new cycle; the process can be repeated endlessly; thus the human nervous system and the mycelium of the fungus are closed on each other in a feedback loop. This lets the symbiosis of signals begin.

The electrical feedback in fungi that could be measured and traced in real time and allow the viewers to have a proper interactive experience is one of the major points to choose this species. Fungi, being one of the most prevailing life forms, and mycelium, creating the symbiotic relationships with plants by forming mycorrhiza and facilitating the nutrients flow among them, demonstrate their great potential to be a basis for the reflection on symbiotic relations and interconnections that are inscribed in our nature. Thus, fungi, mostly invisible to the human, as soon as they are started to be investigated, reveal themselves in every leaf, every piece of soil, they are everywhere. No matter that they are part of human life, they are embedded in our lives, even though they are unseen, covered by soil and having a small scale in comparison to the human environment.

As mentioned already, Olga Kisseleva focused her attention on the communication, chemical interaction between trees and new systems that can evolve with the directed technological mediation. Another element of the EDEN project as “an organic network based on vegetal medium” is a datascape. It is an interactive program that indicates and unites the actors in the network, more precisely, trees connected to the system through sensors and captures, gathers the information and later serves as a source for all visual displays. By creating this network connection, the artist and the scientists were interested in allowing the trees to communicate despite the distance. Highlighting that it is possible to understand and receive the communication of a tree is one thing, to consider within the interdisciplinary group of the specialists that it can be transmitted and appreciated by one of its congeners living on a different continent is another.

As it can be seen, it is crucial to work collaboratively in the intersection of art, science and technology, this approach can reach beyond and let us think more on the way our world is constructed and how everything is linked within it. How can we live together? How can we acknowledge our mutual existence, the connections existing in our networked society and world?

Conclusion

Overall, it can be stated that posthumanism, by itself, has become an umbrella term that includes different perspectives, in an era where the symbolic boundaries of the ‘human’ have been ultimately challenged. Along with that, posthumanism is interested in how one should think in order to coexist with nonhuman life forms and offers an ethical and political program (Haraway 2003, Braidotti 2019).

The ontological effort lies in acknowledging that the human may not be human after all, which calls to re-think existence and being in the world. The consequence is epistemic because if we assume that our being and becoming is different from what we previously thought (given that we are likely to be implicated in a posthuman life), we can no longer explain how we experience and think in conventional epistemological terms. The enquiry into a posthuman condition is to revisit the constitution and function of the world and life.

By talking on the posthuman subject and its ontological state, but more precisely by following the way we are interacting with nonhumans within the practice of selected artists, what their connections to humans are, we can trace the movement towards rethinking the role of humans and acknowledging those by whom we are surrounded. Thus, artists do not formulate their notion of posthuman subject, but rather they demonstrate another approach and attitude towards nonhumans: to consider the latter, trees or fungi in our case, as partners, not to build hierarchies because the equality of the partners and to acknowledge their ‘knowledge’ we can learn from.

This emphasis on the relations appearing among subjects and in the system as a whole, the tendency to make these interconnections explicit are the major posthumanist categories the artists refer to. As it was noticed by Anna Tsing, artistic projects let people learn “to accept that we are all part of the same interconnected system” ¹⁶. Thus, I suggest considering approaches proposed by Olga Kisseleva and Saša Spačal as an indicator of change that could lead us to the refusal of a purely anthropocentric perspective. By not prioritizing one category over another, human over nonhuman or vice versa, by opening up such an approach to the scientific community during the collaborative work and to wider audience in the framework of exhibitions and art festivals, we are going towards a new posthuman subject.

References

1 Karen Barad, Meeting the Universe Halfway, Durham, London, Duke University Press, 2007, p.136.

2 Rosi Braidotti, The Posthuman, Cambridge, Polity Press, 2013, p.49.

3 Rosi Braidotti, The Posthuman Knowledge, Cambridge, Polity Press, 2019, p.3.

4 Rosi Braidotti, The Posthuman Knowledge, Cambridge: Polity Press, 2019, p.94.

5 Rosi Braidotti, Maria Hlavajova, eds. Posthuman Glossary. London, New York, Bloomsbury Academic, 2018.

6 Francesca Ferrando, Philosophical Posthumanism, New York, Bloomsbury Publishing, 2019, p.165.

7 Matthew Hall, Plants as Persons: A Philosophical Botany, New York, State University of New York Press, 2011.

8 Donna Haraway, When Species Meet, Minneapolis, University of Minnesota Press, 2008, p.42.

9 Olga Kisseleva, Du travail collectif a l’oeuvre, Paris, Nouvelle Edition Place, 2017, p.97.

10 Bruno Latour, Science in action: how to follow scientists and engineers through society, Cambridge, Massachusetts, Harvard University Press, 1987.

11 Saša Spačal, “Connection continuum: a life”, in Experiencing Unconventional: Science In Art, ed. Theresa Schubert and Adam Adamatzky, 2015, p.175-206.

12 Saša Spačal, “Myconnect”, accessed December 1, 2022, https://www.agapea.si/en/projects/myconnect

13 Saša Spačal, “MycoMythologies: Patterning”, accessed on December 1, 2022, https://www.agapea.si/en/projects/mycomythologies-patterning

14 Saša Spačal, “MycoMythologies”, S+T+Arts Prize website, accessed December 1, 2022, https://startsprize.aec.at/en/mycomythologies/

15 Anna Tsing, “Unruly Edges: Mushrooms as Companion Species”, Environmental Humanities 1, 2012, p.144.

16 Anna Tsing, “When Things We Study Respond to Each Other: Tools for Unpacking ‘The Material’”, in More-than-Human, ed. Andr.s Jaque, Marina Otero Verzier, and Lucia Pietroiusti, Rotterdam: Het Nieuwe Instituut, 2020, p.83.

17 Cary Wolfe, What is posthumanism? Minneapolis: University of Minnesota Press, 2010.

Bibliography

Karen Barad, “Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter”, in Signs: Journal of Women in Culture and Society, vol. 28, No 3, Spring 2003, 801-831.

Laura Beloff, “Hybrid Ecology – To See The Forest For The Trees”, in Art As We Don’t Know It, ed. Erich Berger, Kasperi Mäki-Reinikka, Kira O'Reilly and Helena Sederholm, Aalto, Aalto ARTS Books, 2020.

Ryszard W. Kluszczyński, “Looking at the World through the Eyes of Other? Art as Non-Anthropocentric Ecology”, in Victoria Vesna. Towards a Non-Anthropocentric Ecology and Art in the World of Anthropocene, ed. Ryszard W. Kluszczyński, Gdańsk – Łodź: Laznia Center for Contemporary Art, Łodź University Press, 2020, 6-27.

“Olga Kisseleva, EDEN, 2012-2021”, ed. Maria-Laure Dejardins, Paris, ARTSHEBDOM.DIAS, 2021

Regine Rapp, On mycohuman performances: fungi in current artistic research, Fungal Biol Biotechnol 6:22, December 2019, doi: 10.1186/s40694-019-0085-6

Jakob von Uexküll, A Foray into the Worlds of Animal and Humans, with A Theory of Meaning, Translated by Joseph D.O’Neil, Minneapolis, University of Minnesota Press, 2010.

Rosi Braidotti, “Posthuman, All Too Human: Towards a New Process Ontology”, in Theory, Culture and Society, 23, 7–8, p.197– 208.

Richard Grusin, ed. Nonhuman Turn, Minneapolis, University of Minnesota Press, 2015.

Donna Haraway, “Symbiogenesis, Sympoiesis, and Art Science Activisms for Staying with the Trouble", in Arts of Living on a Damaged Planet. Ghosts and Monsters of the Anthropocene, ed. Anna Tsing, Heather Swanson, Elaine Gan and Nils Bubandt, Minneapolis: University of Minnesota Press, 2017, 25-50.

N. Katherine Hayles, How we became posthuman, Virtual bodies in cybernetics, literature, and informatics, Chicago, The University of Chicago Press, 1999.

Erich Hörl, James Burton, ed. General Ecology: the New Ecological Paradigm, London, Bloomsbury Academic, 2017. 

Summit on New Media Art Archiving papers

The International Programming Committee (IPC) has not reviewed Summit on New Media Art Archiving papers. Individual authors of papers and presentations are solely responsible for all materials submitted for the publication. The publisher and the editors do not warrant or assume any legal responsibilities for the publication’s content. All opinions expressed in the book are of the authors and do not reflect those of the publisher and the editors.  

The Summit on New Media Art Archiving (SNMAA) series has been a platform and communication channel for stakeholders in new media archiving, aiming to facilitate critical discourse and collaboration. Since 2020, the annual Summit on New Media Art Archiving has become one of the most important and focused conferences for related topics. The Summit engages and connects members of the new media art archiving community for sharing knowledge, exchanging ideas, networking, and seeking collective strategies to tackle mutual archival challenges for the everchanging new media art, the art of our time. 

After the massive support for the Liverpool Declaration, the Summit initiative started with ISEA round table discussions on new media archiving at ISEA2018 in Durban, South Africa, and ISEA2019 in Gwangju, South Korea. The one-day First Summit on New Media Art Archiving was held during ISEA2020 (online from Montreal, Canada), featuring a keynote lecture, paper presentations, and break-out sessions. This Second Summit on New Media Art Archiving was held two days preceding ISEA2022 in Barcelona, Spain. Subjects included, among others, archiving physical artifacts, digitising museum collections, ethical aspects of archiving new media art and innovative approaches to archiving, as well as an emphasis on connecting new media art archives worldwide. Part of the Second Summit was a unique art exhibition based on submissions of artworks inspired by the concept of archiving. 

The third summit was organised by ISEA archivists in cooperation with archivists from SIGGRAPH, Ars Electronica, Archive of Digital Art (ADA), ZKM, Electronic Language International Festival (FILE), Media Art History, and Memoduct Posthuman archive. The summit has been supported by ISEA2023, ISEA International, the SIGGRAPH Digital Arts Community and the Creative Industries Fund NL. It took place on May 19 and 20 as part of ISEA2023, the 28th International Symposium on Electronic Art. 

Consult the proceedings of the Summit on New Media Art Archiving papers

Second Summit on New Media Art Archiving, Barcelona © DR

ISEA2023 Organizers

ISEA2023 / CUBE Organizers: 

Aziosmanoff Nils – Director of the Cube Garges

Krajewska Klio – Production Director

Thibault Clément – Artistic Coordinator

Bonneau Alicia – Partnership Officer

Barthe Pauline – Communication Officer

Testas Fanny – Communication and Production Officer

Michigan Sonia – Production Assistant

ISEA2023 /
École des Arts
Décoratifs – PSL Team: 

Mahé Emmanuel – Director of research and ISEA2023 Academic chair

Buser Edith – Director of Research and co-organizer of ISEA2023 academic call

Barthe Pauline – Organizer of ISEA2023 academic call

Papadaki Elena – Academic co-chair ISEA2023

Ptqk Maria – Academic co-chair ISEA2023

Lapointe François-Joseph – Academic co-chair ISEA2023

ISEA2023 /
Forum des images Team: 

Ziegler Michèle – Director of Digital Strategy and NewImages Le Forum des images

Bouvier Nathalie – Director of Event Production Le Forum des images

Farge Claude – CEO Le Forum des images

+All the venue's stage managers

ISEA2023 50 Volunteers:

Petkova Blagomira (art student at Université de Lille 3), Maurer Louise (art student at Université Sorbonne-Nouvelle Paris 3), Poussiere Maud (art student at Icart), Pousse-Wang Sarah (art student at Sorbonne Paris 1), Guo Songyu (design student at Ecole supérieur d'art et de design TALM-Le Mans), Falco Mario (3d student at ArtFx), Cardon Mila (3d student at ArtFx), Cherston Juliana (student at MIT Media Lab), Pousse-Wang Liyah (art student at Sorbonne Paris 1), Alicia Clotilde (media student at Université Gustave Eiffel), Nilles Clarice (International business student at Paris 12), Anthousi Nitous (art student at Sorbonne Paris 1), Deiss-Veiniere Lou (communication at Sciences Po Lille), Bernard Léon (art student at ESACM (Ecole Supérieure d'Art de Clermont Métropole), Viennois Franck (art student at Beaux-arts de Montpellier), Combret Yumé (cinema student at Sorbonne Nouvelle), Durel Nina (art student at ESACM (Ecole Supérieure d'Art de Clermont Métropole), Guérin-Garnett Nigel, Guiganti Asbjørn, Duverl Christelle, Paouri Arghyro, Malu Laet, Lacourte Manon, Lesbats Lucie, Balland Enzo, Ageron Julie, Belhadj Manelle, El Shérif Mickaël Léna, Lehna Robert-Gonçalves, Jonathan Ouali, Roumeissa Langrene, Léa Bouarfa, Rekouane, Selim Marion, Seknazi Sephora, Baryshnikova Anatsasiia, Aziosmanoff Manon, Fillon Arnaud, Fried Axel, Plisson Laura, Usanakul Ussareena, Gabriela Camille, Marion Beltrão, Gabriela Waheo, Clara Pena, Winnie Reial, Adama Lau, Ralambotiana Fall, Malala Tsanta.

The Steering committee 

École des Arts Décoratifs – PSL and Le Cube have set up an organising committee which, for the first time in France, brings together all the key players in digital-related research and culture. École des Arts Décoratifs – PSL works closely with all these players, sometimes co-founding academic consortia with some of them. ISEA SYMBIOSIS makes it possible to give visibility to this range of already existing collaborations, thus demonstrating the strength of the national research-creation ecosystem. The objective of ISEA SYMBIOSIS is not limited to the event itself; it is a step towards structuring the field on a territorial scale, which will produce similar effects in Montreal in 1995: the arts and digital culture scene is now recognised throughout the world (the Hexagram network associating Concordia University and UQÀM, right up to Ubisoft, stems from this structuring effect). The next step is to consolidate this ecosystem, in which all the players have solid international (European and foreign) networks, thus continuing the efforts of 2023. The symposium proceedings will also move in this direction, with a text specifically dedicated to ecosystems, based on the symposium's session of institutional presentations. This session has also been designed with ecosystem development in mind. 

In connection with the ISEA2023 international board, the French organising committee, co-piloted by the Cube Garges and the École des Arts Décoratifs, brings together leading players in the field of the arts, creative industries, higher education, research and innovation.

Members: 

Ancel Franck – Artist – Theorist

Aziosmanoff Nils – Director of the Cube Garges

Barthe Pauline – Communication Officer and Co-organizer of the academic call ISEA2023

Behar Armand – Co-director CRD, ENSCI, Les Ateliers

Berthoumieux Céline – Director ZINC – Co-director CHRONIQUES – President d'HACNUM

Bonneau Alicia – Partnership Officer ISEA2023

Bureaud Annick – Director Leonardo OLATS – Independent curator

Buser Edith – Associate Director of Research at the École des Arts Décoratifs – PSL – Co-organizer of the academic call ISEA2023

Cadon Benjamin – Director Labomédia

Carvalho Guilherme – Researcher in virtual environment in charge of ISEA2023 – MSH Paris-Nord

Chevrier Joël – Professor, in charge of the ISEA2023 project – Learning Planet Institute Professor

Citton Yves – Researcher – Theorist – Teacher Paris 8

Dallet Jean-Marie – Professor and artist ACTE Institute (Sorbonne)

Dondey Marc – Director ENS Paris-Saclay La Scène de Recherche

Farge Claude – Director Forum des images

Fleischer Alain – Director Le Fresnoy

Foresta Don – Artist – Researcher – Theorist – Teacher

Guez Emmanuel – Director ESAD Orléans

Krajewska Klio – Production Director ISEA2023

Lahlil Sophia – Head of Administration and Coordination La Scène de Recherche ENS Paris-Saclay

Le Poullennec Annaël – General Manager EUR ArTeC

Madlener Frank – Director IRCAM

Mahé Emmanuel – Director of research at the École des Arts Décoratifs – PSL – Academic chair ISEA2023

Michigan Sonia – Production Assistant ISEA2023

Portevin Jeanne-Marie – Coordinator EUR ArTeC

Prigent Eric – Digital Creation Coordinator Le Fresnoy

Reyes Everardo – Vice President Digital Paris 8

Sedes Anne – Director MSH Paris-Nord

Selmane Inès – Project manager in the Digital Development Department Forum des images

Taddei François – Director Learning Planet Institute

Testas Fanny – Communication and production Officer ISEA2023

Thibault Clément – Artistic Coordinator ISEA2023

Vinet Hugues – Scientific Director IRCAM

Ziegler Michèle – Digital Development Directorate Forum des images

ISEA International
Board members: 

Erandy Vergara 
Assistant Professor in the Faculty of Fine Arts and Communications, MacEwan University and Curator, Printemps numérique

Everardo Reyes 
Academic, Professor of Information and Communication Sciences, Co-director of the Masters in Digital Humanities, Université Paris 8 – Vincennes-Saint-Denis. Responsible for international relations, EUR ArTeC. Associate member of the Cultural Analytics Lab. Art Papers Chair for SIGGRAPH 2019 – Los Angeles.

Michel Van Dartel 
Director (V2_ Lab for the Unstable Media) and Professor (Avans Centre of Applied Research for Art, Design and Technology).

Pat Badani 
Special Advisor to ISEA2020 & ISEA2023; Chair of Community Network Subcommittee 2017-2022; IIAC member (ISEA International Advisory Committee) 2013-2016.
Independent cultural researcher/producer; former Integrated Media Professor at the School of Art, Illinois State University, USA; Interdisciplinary Media Arts graduate program acting Director at Columbia College Chicago; Editor-in-Chief of Media-N Journal, the New Media Caucus; and Lead Guest Editor of “Artelogie,” EHESS, France.

Ricardo Dal Farra 
Professor of music and media arts at the Music Department of Concordia University in Canada. Director of the Electronic Arts Research and Experimentation Centre (CEIArtE) at National University of Tres de Febrero in Argentina. Directing the Latin American Electroacoustic Music Collection. Founder-director of the Understanding Visual Music (UVM) and Balance-Unbalance (BunB) international conference series.

Tanya Ravn Ag, PhD
Danish curator and scholar examining perceptual experience, expanded reality, temporality, and technogenesis in relation to urban and virtual media aesthetic phenomena and art. She was a visiting fellow at the School of Creative Media, City University of Hong Kong, and is currently a postdoc researcher at the Institute of Arts and Cultural Studies, University of Copenhagen.

Organizers

Co-organizers

Supported by

     quebc   

ISEA2023 stands with Ukraine 

28th
International
Symposium on
Electronic Art
ISEA2023

Publishing Director: Emmanuel Tibloux, Director of École des Arts Décoratifs – PSL
Editor: Emmanuel Mahé, ISEA2023 Academic Chair, Head of Research of École des Arts Décoratifs – PSL
Copy-editing: Pauline Barthe, Edith Buser, Elsa Carnielli 
Proofreading: Pauline Barthe, Elsa Carnielli, Manuelle Freire 
Publicization Special Adviser: Lucile Haute 
Graphic Design: Quentin Juhel 
Types: Syne by Bonjour Monde, Open Runde by Laurids Kern 

ISEA2023 proceedings has been designed with frugality in mind. The pdf version features black typography on a white background in standard A4 format, which saves ink and paper for (full or partial) office printing. The web version is static and resource-efficient.
It has been made with the editorial platform BOOKOLAB, a collaborative tool for editing and formatting publications for different reading media (printable version as PDF & screen version as web page) designed by Lucile Haute, and developed by Arman Mohtadji & Benjamin Dumond with GRAV and Paged.js, with the support of Université de Nîmes and École des Arts Décoratifs  PSL. Projekt source code: gitlab.com/armansansd/bookolab

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without prior written permission of the individual authors.

Individual authors of papers and presentations are solely responsible for all materials submitted for the publication. The publisher and the editors do not warrant or assume any legal responsibilities for the publication’s content. All opinions expressed in the book are of the authors and do not reflect those of the publisher and the editors.  

Despite intensive research, it was not possible to identify all copyright holders. If proper acknowledgment has not been made, we ask copyright holders for notice.

isea2023.isea-international.org
 @ISEAInternational 
 @iseainternational 
 @ISEASymposium 

ISEA2023 Paris, International Symposium on Electronic Art

ISBN 978-2-905710-71-0 (PDF)
ISBN 978-2-905710-69-7 (web)
DOI 10.69564/ISEA2023-PROCEEDINGS

École nationale supérieure des arts décoratifs
31, rue d’Ulm, 75005 Paris – France
www.ensad.fr
© École des Arts Décoratifs  PSL, 2024