The 28th edition of ISEA, co-organised by Le Cube Garges and the ÉcoledesArtsDécoratifs–PSL, took place in Paris at the Forum des images from 16 to 21 May 2023, as
well as through an artistic programme in some 50
partner cultural venues throughout France in spring-summer 2023. The videos of which are online here.
Following an international call for proposals, more than 1400 submissions from 70 countries were evaluated by an
artistic and scientific committee of 200 international experts. The theme of this edition was Symbiosis. In this
time of global health, ecological, economic and democratic crisis, symbiosis is a polysemous notion that allows us
to explore in a transversal and interdisciplinary way the mutations and transformations underway in the digital
age, to question the meaning supposedly given to progress, especially in the current environmental and health
context, and to imagine possible and viable futures for our planet and our ecosystems.
The symposium catalog, including the symposium program and the artistic program, is available as
a downloadable pdf here.
Figures
+ 1 500 participants at Symposium conferences and exhibitions at the Forum des
images + 700 participants in online conferences during Symposium week + 7 000
visitors to the Augmented Catalogue in the first 15 days + 1400
proposals + 660 academic proposals +740 artistic
proposals + 70 applicant countries + 50 cultural partner venues
Co-organizers
ISEA2023 was co-organised by Le Cube Garges, executive producer, ÉcoledesArtsDécoratifs–PSL, academic programming director, and the Forum des images, Symposium
venue. ÉcoledesArtsDécoratifs–PSL
is a public institution of higher education under the authority of the Ministry of Culture, whose mission is the
artistic, scientific and technical training of artists and designers, as well as the development of research
creation activities. pioneer in thie field of collaborations between art, design and sciences, it created a
research laboratory, EnsadLab in 2007. Within the University Paris Sciences & Lettres (PSL), towhich it is
affiliated, it set up the PSL SACRe (Science Art Creation Research) doctorate program in 2012. It is the first
practice-based doctorate to be set up within an art and design school in France.
Le Cube Garges is an interdisciplinary and digital cultural innovation center with six facilities covering 8,000
m². Focused on creative renewal, it combines the discovery of new artistic forms, inclusive practices,
digital training and interdisciplinary reflection on societal issues.
SYMBIOSIS, THE MAIN THEME
Symbiosis is essential to life. No living species can survive without cooperation, let alone humans with each
other and with their environments. More than a simple coexistence, symbiosis induces interdependence. It can be
positive, neutral and sometimes negative in certain cases of parasitism. From these interrelations can emerge new
hybrid entities, like the lichen which is a compound of algae and fungi. The first providing photosynthesis and
the second moisture, one benefits from the contribution of the other. Our becoming lichen would be in a way the
cross-cutting question underlying the symposium. Understood in a broad acceptance, the symbiosis is also played
out at the scale of the symposium: it is a transdisciplinary event between creation and research (visual arts,
theater, music, design, cinema, human and social sciences, engineering sciences , natural sciences) and
intersectoral (art, crafts, industry, research, education, cultural innovation).
THE SUB-THEMES
The call for proposals for the symposium was divided into three main sub-themes: symbiotic individuations,
organizations and imaginaries. It was a starting point to structure the call without limiting the scientific
concept of symbiosis, and to encourage cross-cutting proposals outside disciplinary and technical classifications.
The selected interventions thus made it possible to discover works that call on symbiosis as a scientific
concept, but above all projects that used it as a metaphor or more simply as an inspiration. It was an invitation
to discover in a new way, both serious and curious, projects closely associating creation, research and
technologies.
Symbiotic individuations
Which type of “hybrid beings” are coming into existence today, and thereby associating the human and
the non-human? What new forms will they take, and will they be the next inhabitants of our planet? Experiments on
trans-humanism, the emergence of new artificial autonomies, and new representations of natural elements were
examples of the exploratory perspectives that were to be presented and discussed.
Symbiotic organizations
Human societies have founded their equilibrium upon multitudinous and interdependent forms of symbiosis across a
wide swathe of domains: ecology, economy, culture, technology, and more. What happens when a given event destroys
this fragile equilibrium? Researchers, artists, designers, and leading experts offered their perspectives on these
resolutely transdisciplinary approaches, with the shared ambition of contributing to a world that is more
resilient and at ease in its alterity.
Symbiotic imaginaries
In order to exist and prosper, symbiotic relationships—relationships in which each partner derives
reciprocal benefits indispensable to their survival—are essential. Symbiosis is a vital necessity at all
levels of all known ecosystems. But “necessary” does not imply that it should be reduced to its
pragmatism or instrumentality. Symbiosis also opens us to new imaginaries, be they novel or redundant in the
context of their corresponding era. If modes of collective imagination vary according to perspectives, they are
always multiple and in dialogue with reality—what, therefore, constitutes the symbiotic imaginations of
today and how and why invent new ones?
THE “CLUSTERS”
Given the large number of conferences, roundtables and various oral presentations, the academic chairs proposed
clusters to enable the symposium audience to compose its own cross-cutting program to complement the Symbiosis
themes. This
gave meaning to the general programme.
The papers grouped together in each of these clusters could be classified differently and often appeared in
several other clusters. These groupings are therefore retained here, with each cluster listed in alphabetical
order (authors). Each paper is colour-coded to indicate the sub-theme to which the author(s) responded (sub-theme
reminder), together with the cluster in which it was scheduled for the symposium audience and any other clusters
to which it might have been attached. A final glossary by sub-theme and cluster is provided to offer readers
several entry points into this important volume. These classifications in no way restrict the understanding of the
papers, they are simply reading aids.
Academic Chairs and International Programming Committee
Academic Chairs
The Academic Chairs were
responsible for the final programming of the symposium based on the evaluations of the IPC. The names and
biopics of the Academic chair and of the three co-chairs are published below.
The International Programming
Committee (IPC) was composed of 140 internationally renowned experts. The Committee evaluated the 1400
proposals in a double-blind process for full & short papers, panels and roundtable discussions. The names
of the IPC are listed below.
Dr Emmanuel Mahé — ISEA2023 Academic Chair
Dr (HDR) Emmanuel Mahé (France), Director of Research at École nationale supérieure des
Arts Décoratifs (PSL), researcher in information and communication sciences dealing with issues related
to organisations and innovation processes in the fields of arts, sciences and design; scientific officer at the
High Council for the Evaluation of Research and Higher Education. Website of Emmanuel
Mahé
Dr Elena Papadaki — ISEA2023 sub-topics co-chair
Dr Elena Papadaki (UK/Belgium) is a researcher, educator and curator; her research interests lie in the
intersection of technology-reliant artworks, curatorial discourse, interdisciplinary collaboration and audience
reception.
She leads the MA Design at University of Greenwich and currently serves as a Pathway Councillor for the Royal
Society of Arts (RSA), acting as a key advisor and expert to the delivery group “Design for
Life–Students for Change”. Website of Elena Papadaki
Dr Maria Ptqk — ISEA2023 sub-topics co-chair
Dr Maria Ptqk (Spain) is a curator, researcher and cultural producer. With an academic background in law and
economy, she holds a PhD in artistic research with a doctoral thesis on the cyberfeminist collective subRosa.
Her work investigates the ecological and technoscientific realm from a postcultural perspective, in the
crossroads of experimental art practices and collective in/trans/disciplinary knowledge. Website of Maria Ptqk
Prof. François-Joseph Lapointe (Quebec) is an artist and scientist at the University of Montreal,
holding two doctorates (in evolutionary biology and in dance and performance). He has published over 120
articles in the field of molecular systematics, population genetics and metagenomics. As part of his artistic
practice, he applies biotechnology for creative purposes and has created choreogenetics. Website of François-Joseph
Lapointe
International Programming Committee
Aceves Sepulveda Gabriela –
Université Simon Fraser (CA)
Albert Jeffrey –
Université Loyola de la Nouvelle-Orléans (FR)
Almas Almir –
Université de São Paulo (BR)
Anders Peter – Kayvala
(États-Unis)
Badani Pat – Conseil
d'administration de l'ISEA (États-Unis)
Baker Camille – Royal
College of Art, Londres (Royaume-Uni)
Basserau Jean-François
– Ecole des Arts Décoratifs–PSL (FR)
Behar Armand – Centre de
Recherche en Design – ENSCI-Les Ateliers ENS Paris-Saclay (FR)
Bellotto Janet –
Université Zayed (AE)
Berenguer Josep-Manuel –
Sonoscop (ES)
Beyls Peter –
Collège universitaire de Gand Belgique (BE)
Bianchini Samuel – Ecole
des Arts Décoratifs–PSL (FR)
Vatavu Radu-Daniel – Stefan cel Mare University of Suceava (RO)
Vergara Erandy – ISEA International (MX/CA)
Vilca Cecilia – MyAP – Microscopía Electrónica y Aplicaciones en el
Perú (PE)
Warnke Martin – Leuphana University Lüneburg (DE)
Wei Jo – CAFA
(CN)
Young Jiayi –
Université de Californie, Davis (États-Unis)
Zhang Ga – Académie
Centrale des Beaux-Arts, Pékin (CN)
Academic and Artistic Program
At the same time as the academic call for artists and researchers, a specific call for projects was launched,
under the artistic responsibility of Cube-Garges. This selection has been enhanced by the networked artistic
program (50 venues) offers thematic tours around symbiosis in a network of partners in Paris and in France,
during several weeks. It irrigated the territory and thus brought to ISEA2023 both an anchoring in various
places and an opening to diversity. The public discovered a wide variety of installations, virtual or augmented
realities, performances, concerts, video games, design, films or digital books. The tours is also available on a
dedicated online platform here or on our website.
POSTERS
Bazenet Flavien, Plisson Laura and Goeury Alexandre (FR)
Verdeil Marie, Jochem Rebekka, Christian Schulz Jan and Völp Lukas (FR / DE)
Vesac Jean-Ambroise, Duval Hélène, Ouellet-Plamondon Claudiane, St-Onge David, Salter
Chris (CA / CH)
Young Jiayi and Chomaz Jean-Marc (US / FR)
Zannos Iannis, Birringer Johannes, Antoniadis Pavlos, Hirayama Haruka, Jégo Jean-François,
Paschalidou Stella, Nelson Peter and Papachristou Dana (GR / UK / JP / FR)
ISEA International
ISEA International was proud to present the 28th edition of our annual symposia in Paris (France), with an
academic symposium and exhibition predicated as an experimentation incubator driven by the unifying theme
of“Symbiosis”– an essential consideration for the interdependent gathering of the year 2023.
We joined in celebrating ISEA’s return to Paris after 23 years since ISEA2000
“Révélation”, with Le Cube Garges as executive producer and responsible for the artistic
program, and ÉcoledesArtsDécoratifs–PSL
responsible for the symposium’s academic program. As in the previous iteration in Paris, the academic
symposium was held in the heart of the city at Forum des images.
Founded in the Netherlands in 1990, ISEA International (formerly Inter-Society for the Electronic
Arts) is an international non-profit organisation fostering interdisciplinary academic discourse and exchange
among culturally diverse organisations and individuals working with art, science and technology.
The main activity of ISEA International is the annual International Symposium on Electronic Art
(ISEA). The first symposia took place in 1988.
Tributes
Two major figures in art and culture left us in 2023: Peter Weibel, whose tribute was paid during the
Symposium with the entire ISEA community gathered in the large amphitheatre of the Forum des Images, a tribute led
by Pr Roger Malina which elicited many speeches from the audience; and Vera Molnár, who died a few months
after the Symposium and to whom we pay tribute in these proceedings.
Disremembering PeterWeibel: Reminiscences are
more Interesting than Fact
Peter Weibel was a leading figure in the new media art scene. Since 1999, he had been the director of the ZKM in
Karlsruhe and had carried out numerous curatorial activities (Ars Electronica, Venice Biennale, etc.). Among his
most important exhibitions, we can also mention his collaborations with Bruno Latour, who also passed away
recently ("Iconoclash" in 2022, "Making Things Public" in 2005 and "Critical Zones. Horizonte einer neuen
Erdpolitik" in 2020). The sudden departure of Peter Weibel in March 2023 was a shock for the ISEA
communities therefore a tribute was paid to him at this years’ symposium. A lecture was given by Roger
Malina, followed by a session lasting over an hour in which Malina asked the large audience to share memories of
Peter Weibel. Many very moving and interesting words for the ISEA community were expressed. Many deeply moving
words and recollections were expressed by members of the ISEA community.
Tribute to Vera Molnár
Among the numerous reactions following the passing of Vera Molnár at the age of 99, a pioneering artist in
geometric abstraction and generative art, the National Institute of Art History (INHA) in Paris paid tribute to
her. The ISEA Symbiosis committee also wished to honor this iconic figure by transcribing the text from the
INHA here.
Vera Molnár “had established deep ties with the INHA, which were manifested through the donation of
a significant collection of prints in 2022, where the pioneering and generous dimension of her work fully
reflected her personality. Her passing leaves a huge void in the art world, where she was still actively engaging
with young creators who came to visit her. Vera Molnár was born in 1924 in Budapest, where she received a
classical education in Fine Arts and graduated as a professor of art history and aesthetics in 1947. That same
year, she left Hungary and settled in Paris. Quickly choosing her artistic family in geometric abstraction, during
the 1950s she was a radical constructivist who refused to label what she considered, with the theoretical support
of her husband François, a scientist and psycho-physiologist, as "art," viewing it instead as experiments.
In this spirit, she invented an "imaginary machine" in 1959, a proto-computational conceptual process that
involved giving precise composition instructions followed by systematic execution, akin to a machine. Taking a
significant step in 1968, she began using a computer to assist in creating artworks based on programs and
algorithms, becoming a pioneering artist in France in generative drawing. Curious about all materials and mediums,
she drew, glued, painted, sculpted, programmed, photographed, created installations, artist books called
"livrimages"[the French contraction of the words ''book'' and ''image''], and an impressive intimate diary
(1976-2020) in twenty-two volumes of nearly 5000 pages. Whenever possible, Vera Molnár also created prints.
Trained in Fine Arts, she learned wood and linoleum engraving at a young age but later collaborated with
professionals to produce numerous publications. In 2022, she decided to assemble a significant collection to
donate to the INHA library, consisting of 264 prints, 2 portfolios, and 12 preparatory models.
This donation forms an exceptional collection, representing the entirety of the artist's work, unique in French
and foreign public collections. Her works belong to formal families that she worked on, revisited, and reinvested
in for several decades. The Sainte-Victoire Blues directly reference Cézanne's motif, drawn and
painted hundreds of times by Vera Molnár. Lettres de ma mère echo the computer modeling of
the artist's mother's handwriting, who wrote to her from from Budapest, which over the years became increasingly
jagged at the end of each line, described by the artist as "a bit gothic, somewhat hysterical, "yet providing her
with a sensation of "visual bombardment" each time. 4 carrés rouges,
Brèches, 3 triades 3 couleurs, Ordinateur miroir de la main, Hypertransformations hommage
à Klimt, and many other series explore the paradigmatic form of geometric abstraction: the square.
Cut, shifted, overlapped, joined, pushed, covered, giving rise to other shapes, incisions, triangles, quadrangles,
perhaps letters, the squares disassemble, suggesting a figure of balance or generating a perceptual indecision,
leaving it to each individual to contemplate and decide for themselves.
Upon news of her passing, INHA seeks to
perpetuate the living legacy of an immense creator; that of an adventurer of shape, of order and disorder, who
used and challenged tradition. Behind Vera Molnár's programs, algorithms, and systems lie thought and
ideas, as well as tenderness and amusement; in short, life. Today, we bid farewell to an artist generative in
every sense."
With the kind permission of the authors, INHA's director, Éric de Chassey, the director of its library,
Jérôme Bessière, and Vincent Baby, project manager at INHA, close friend and Vera
Molnár specialist.
Symposium Proceedings
Introduction
SYMBIOSIS – ISEA2023 PROCEEDINGS INTRODUCTION
FOR A SYMBIOTIC APPROACH AT ALL SCALES
THE NEED FOR INSTITUTIONAL RECOGNITION OF PRACTICE-BASED RESEARCH
(‘RECHERCHE-CRÉATION’)
KEY STRENGTHS
1. ORGANISATIONAL FORMS ARE ALSO CREATIONS
2. CRITIQUE OF THE OLD BINARITIES THROUGH PRACTICE-BASED RESEARCH
3. EMERGING DE/SUBJECTIVATION PROCESSES
4. GENERIC TOOLS DESIGNED AND SHARED BY COMMUNITIES OF PRACTICE
5. BREAKING DOWN DISCIPLINARY BARRIERS AND THE EMERGENCE OF NEW RESEARCHER PROFILES: EVALUATION AS A
TOOL FOR LEGITIMIZATION
6. PERCEIVE BEYOND OUR USUAL CAPACITIES TO BECOME AWARE OF SOCIETAL AND ENVIRONMENTAL TRANSFORMATIONS
7. WHEN CREATIVE RESEARCH BECOMES CRITICAL ACTION-RESEARCH
CONCLUSION
For a symbiotic approach at all scales
Of the 660 proposals received
for the academic call (and a total of 1,400 including the call for artistic projects), from over 70 countries,
230 were selected, including 99 short and long papers, for the Symposium held at the Forum des Images, in the
heart of Paris.200
international researchers and experts evaluated these applications in a double-blind process. More than 50
global organizations, including schools, universities, research bodies and associations, were selected to
present their research activities and facilitate exchanges among them. In addition, more than 50 partner venues,
such as art and design schools, galleries and institutes, offered a rich complementary program of exhibitions,
performances and debates on a national scale. This mobilization of a range of contributors, evaluators, and a
public of several thousand spectators once again attested to the vitality of ISEA, an annual event initiated
more than 30 years ago by pioneers.
The central theme of the 2023 edition was Symbiosis, as demonstrated by the 99 previously unpublished
papers in these proceedings. The full program, including these papers and all the other symposium formats - artist
talks, panels, workshops, tutorials, posters and demos - can be consulted online. This program was in itself a
form of symbiosis.
But what types of symbiosis are we talking about exactly? How do they manifest themselves? Going
beyond the strict definition of the term and interpreting it as an operative metaphor, how can we envisage and put
into practice a genuine ‘symbiotic’ approach? The aim of the call for entries was to encourage the
reconsideration of any creation, experimentation or research through this prism, even if symbiosis was not
necessarily the main theme. ISEA2023 offered a
sort of ‘core sample’ at a given moment of the different strata of contemporary creation and
research underway in the world.
Our idea was that the symposium's programming should also serve as a collective manifesto, promoting
the fundamental principle of symbiosis to be respected at all times and places: a lasting and mutually
beneficial association between organizations and actors, whether human, non-human or hybrid. Considered in
a broad sense, symbiosis also manifested itself at the level of the symposium itself: it was a transdisciplinary
event combining creation and research (in the fields of visual arts, theatre, music, design, film, human and
social sciences, engineering sciences, natural sciences) and cross-sector (involving art, craft, industry,
research, education and cultural innovation).
Before highlighting what we consider to be some of the strengths of this particularly rich body of
work, in terms of both knowledge creation and innovation, it is important to mention our commitment to promoting
practice-based research in all its forms and formats. This approach fosters the invention of modes of cooperation
between different areas of expertise and enables the emergence of hybrid researcher profiles. Symbiosis does not
lie in the homogenization of its components, but rather in the creation of a hybrid and symbiotic entity, where
each element is essential to the other in devising new ways of existing. Together, they offer innovative ways of
perceiving, understanding and acting on the world.
The need for institutional recognition of practice-based research
(‘recherche-création’)
It often happens that artist-researchers or designer-researchers are valued for their creative
abilities, while at the same time being perceived as illegitimate in their research skills. Indeed, there is a
perception that one cannot be both a researcher, publishing scientific articles for example, and a
creator, exhibiting one’s work. This perception differs from country to country. In countries like
Canada, for example, this is not a problem. In France, for example, despite recent progress, not all research
institutions are yet clear about this. Yet these publications and exhibitions (‘publicizations’) both
stem from the same practice-based research process. New generations of researchers are challenging this
perception, and ISEA is a testament to this, since the event enables creators and researchers to share their
research results in an international A-rank publication that meets the standards of peer-reviewed scientific
journals. Institutional structures have also become aware of this reality, as shown by the doctoral programs
specializing in practice-based research which, through their official recognition, mark a fundamental evolution
both in the academic world (universities, art and design schools) and in the professional sector, and not only in
the cultural field (health, transport, housing, ecology, etc.). However, we can only encourage research and
teaching institutions to take this major development into consideration, by proposing, for example in countries
where it does not yet exist, the status of lecturer, associate or full professor
(‘enseignant-chercheur’ in France) in art and design schools.
The older generations, pioneers who are still very active, were also present at ISEA2023. In Paris,
for example, the pioneers of digital arts active since the 1980s rubbed shoulders with doctoral students just
beginning their research careers. This cross-generational aspect is rare enough to be highlighted and celebrated.
Key strengths
As Academic Chair and co-Chairs, we propose here to highlight what we see as the strategic trends
running through all the papers. This collective work reflects our different disciplinary fields, countries and
educational backgrounds. Our aim is not to summarize such a rich corpus, but to propose several common threads to
encourage more in-depth reading of the articles and, of course, to contradict or consolidate our proposals.
Although this exercise is limited and subjective, partial and biased, it is nonetheless enriching. It
reveals that, despite the diversity of the proposals, it is possible to find convergence points, despite the
divergences, and even, sometimes, diametrically opposed approaches. The theme of symbiosis encourages us to look
not for what unifies, but rather for what the different perspectives or visions proposed, even if they take
different forms and are based on heterogeneous conceptual frameworks, can generate together. After all, symbiosis
thrives on difference.
1. Organisational forms are also creations
Faced with the urgent environmental and democratic challenges of our time, the development of new
collaborative methods is essential. Creative research is an innovative methodological response, offering an
effective means of strengthening and better coordinating interdisciplinary dynamics. It opens the way to a new era
of organizations and individuals working together in symbiosis. Although the approaches may vary, the common
objective is clear: to give increasing importance to research within institutions and to recognize the vital role
of associative networks and companies involved in this process.
Organizations, whether stable or changing, are study entities in their own right. Namely, E. Armand,
A. Asensio and D. Foresta encourage the creation of new cooperative ecosystems between the arts and the
technosciences, underlining the importance of collaborative innovation. Transforming organizations from within is
also seen as a promising way forward. Y. Hu, C. Chou and Y. Kakehi, for example, are calling for the creation of
laboratories within art centers, as they have done by setting up a ‘controlled environment space’ at
the heart of an exhibition. Y. Breuleux, A. Thibault and R. Lapierre, for their part, advocate the idea of
occupying and reappropriating industrial wastelands, which are often included in digital art festivals, to give
them a new lease of life.
Organizational architecture also extends to the digital domain. V. Hilsberg looks at the use of DAO
(Decentralized Autonomous Organization) based on blockchain, arguing that this technology has a
direct influence on artistic creation through distribution methods, perhaps recalling Marshall McLuhan’s
famous adage: ‘the medium is the message’. This specific focus on the medium or media is a key feature
of ISEA.
Print media have always been and remain essential to the evolution of academic structures. C.
Frémontier-Murphy traces the history of the journal Leonardo, published by MIT since 1966, which
became an international benchmark by promoting a then nascent field of research. More recently, S. Bianchini and
G. Lallemand have broken new ground by launching the online academic journal ‘.able’, with the support
of more than 30 international partner institutions and the Arts and Sciences Chair at Polytechnique, EnsAD and the
Daniel and Nina Carasso Foundation. This new image-based medium does not seek to compete with the written word,
but rather emphasizes the importance of the visuals generated by practice-based research projects, referring to
them as ‘publicizations’. This approach is based on the fundamental idea that to publish is to make
public. In the academic context, this means subjecting new knowledge, whether textual or visual, to the critical
scrutiny of peer communities.
When Leonardo was created, organizational innovation was also underway in South America.
J.-C. Mariategui recalls the birth of the Centre for Art and Communication (CAyC), a Latin American network
dedicated to art and cybernetics. Today, international collaboration between researchers has become the norm, as
shown by the example of SEADS (Space Ecologies Art and Design), a transdisciplinary and multicultural collective
on a global scale.
Sometimes developments are the result of unforeseen circumstances, sometimes predictable but poorly
anticipated by public authorities. Recently, the Covid-19 pandemic forced cultural institutions to reinvent
themselves. N. Ricci and M. Agogué share the lessons learnt by the Montréal festivals, illustrating
the resilience and adaptability of organizations in the face of global challenges. These organizational issues
also have their counterparts in the production of works. For example, W. Zhang and S. Su make visible the
‘voids’ between individuals subjected to the social distancing put in place during the Covid-19
pandemic.
All these examples illustrate the growing interest in the creation of new organizational forms in
various fields. The concept of symbiosis originated in the natural sciences. By borrowing the vocabulary of these
sciences, it is possible to suggest that all these organizational forms together give rise to new ecosystems. The
term ‘ecosystem’ was proposed by Arthur George Tansley in 1935 in the journal Ecology. The
concept then gradually migrated into the human and social sciences, particularly economics. Today, it is often
used to designate a complex environment that encourages creation and innovation. Widely adopted by those involved
in entrepreneurial innovation (makers, startupers, entrepreneurs, etc.), this metaphor of scientific origin can
also be applied to art and design research.
These ecosystems, through their interrelationships, are contributing to the emergence of a renewed
‘biosphere’, forming a vast practice-based research network on a global scale. This network can be
seen as a favourable climate for invention, a phenomenon that Nowotny, Scott and Gibbons had already described in
their analysis of knowledge production in contemporary society in 2001. Thus, this organizational and creative
landscape can be compared to a dynamic biosphere, where innovation emerges from interaction and collaboration, as
Wenger, McDermott and Snyder emphasized in 2002 in their study of communities of practice. This is the case in
Québec, but not everywhere. Like any ecosystem, the equilibrium found is fragile and subject to forces that
can destabilize it, as in the case of Symbiotica, which is now threatened with extinction, even though it is a
laboratory with an international reputation for its work in bioart. Ionat Zurr, an artist-researcher and
co-director of this structure, spoke about this during her keynote speech. She also appealed to our
community for help, which we are relaying here.
Each new organizational arrangement is both the product and the result of a process of transformation
which also operates within conceptual frameworks. It is not surprising, therefore, that these frameworks are
themselves put under strain by these arrangements, which ‘reorganize’ - or, depending on one’s
point of view, disorganize - traditional ways of thinking.
2. Critique of the old binarities through practice-based research
The questioning of the traditional dichotomy between culture and nature is now shared by many in the
academic community, particularly in anthropology. Artistic works embody and make perceptible this profound
reconfiguration of paradigms of thought. N. Georgakopoulou, D. Zamplaras, S. Kourkoulakou and C.-Y. Chen explore a
‘sympoetic’ interaction with materials in interactive art, inviting us to question old dichotomies
such as subject/object, mind/body, and nature/culture, in order to embrace a new understanding of materiality. P.
Purg and K. Pranjić oppose a human-centered science, putting forward an ‘emergent symbiotic
mutualism’ that transcends mere species coexistence, from a posthuman perspective. M. Zolotova, for her
part, sees the posthuman turn in many contemporary discourses as a serious attempt to move beyond
anthropocentrism. Such approaches are therefore not identical, and can even sometimes be seen as opposed, but they
all involve a critique of analyses centered solely on the human.
This criticism can be found in other fields too. For example, Bakke questions biocentrism and focuses
on the interactions between minerals and non-human life forms. Adopting an approach of working ‘with’
rather than ‘on’ is another way of challenging these human-centered biases. M. da Rocha Montanari uses
generative art as a collaborative research methodology with indigenous Guarani and Kaiowá communities,
co-creating a ‘poetics of attention’ that draws on sounds and plants in their cosmology. A. Wollensak,
B. Terry and B. Baird present extracts from community testimonies relating to water, accompanied by processed
environmental sounds from Alaska.
For R. Wright and S. Howden, the adoption of indigenous perspectives is crucial to developing
‘sentient machines’ capable of reflecting the complexity of ecological patterns, where computational
ecology fails. C. Schnugg, D. Brill and C. Stary merge aesthetics, performance and digitalization to reinterpret
the world, going beyond the rational production of knowledge. Although these initiatives are diverse, they share a
common goal: by hybridizing different rationalities and subjectivities, they seek to transcend existing approaches
in order to invent new ways of understanding and addressing contemporary crises, both geopolitical and climatic.
3. Emerging de/subjectivation processes
These methodological and epistemological reflections are accompanied by a transformation in the ways
in which we conceive of human ‘identity’, the conceptual as well as practical ‘making’ of
the way in which we define ourselves, or not, as human subjects. Indeed, the processes by which
individuals construct themselves as subjects (a process of subjectivation), or sometimes deconstruct themselves (a
process of de/subjectivation), are present in many practice-based research projects. For example, the performance
by T. Chiaravalloti and A. Meshi’s, explores a symbiotic relationship between humans and an artificial
intelligence algorithm, enabling participants to experience a ‘collective consciousness’. Although
this work aims to highlight the limits of AI’s ability to recognize emotions, it also opens the way to
reflections on future developments in this field and the ethical dilemmas they raise. P. Gemeinboeck and Rob
Saunders, for their part, propose ‘hybrid entanglements’ between humans and robots, based on bodily
empathy between humans and non-humans. S. Cîrcu and C.Y. Chen explore movement to question the
anthropomorphism of humanoids through a mimetic dance between a human and a robotic arm.
These approaches use translation, imitation, differentiation and empathy as means of criticizing, but
also realizing, the hybridization between machines and humans as formalized by a number of artists and theorists
at the Symposium: M. Smigielska uses AI to take account of spectators’ behaviour, E. Amato & E.
Péreny claim a ‘co-individuation’ with our technological avatars, J. Bae and his VR comic strip
explore the relationship between humans and AI, D. Batsis, M. Grigoriadou, or with K. ElRaheb and A. Politis, who
advocate the body as a phenomenon and an extension of ‘hyperreality’.
Even the most fiercely critical creative research into intrusive technologies is unwillingly
participating in this general trend. C. Brunner and J. Fritsch look at the transhumanist movement through the
prism of Gilbert Simondon’s thinking, in particular his notion of ‘human energetics’, and are
inspired by a video work by the Nigerian artist Otobong Nkanga. G. J. Shin, using the concept of the
‘superjet’ in a post-digital context, highlights the emergence of new creative human subjects, capable
in her view of weaving new social relationships.
This critical exploration of the way in which humans conceptualize and experience themselves also
manifests itself in a decentering of the human, highlighting the processes of subjectivation at work in the
‘non-human’ or ‘other-than-human’ world. Adopting a Bruno Latour perspective, elements
such as forests and water are no longer simply reduced to objects of study for humans; they are seen as actors in
their own right, subjects in the active sense of the term. In this vein, G. Trudel, who studies climate change,
integrates data from tree communities, the media arts and forest sciences. She attributes a ‘voice’ to
trees through what she calls ‘individuations of symbiotic modulations’, drawing on and extrapolating
the work of Gilbert Simondon.
Approaches to thinking about and experimenting with symbioses between humans and machines, as well as
between non-human entities, vary widely. However, they all point to a probable reconfiguration of the general
cartography of subjectivation processes: the contemporary human seems ready to move away from the central and
dominant position it has hitherto attributed to itself. At least, that is what the observations and discussions at
the Symposium seem to indicate. The question is what this will mean in the long run, and whether the promise of
emancipation will really be fulfilled.
4. Generic tools designed and shared by communities of practice
As the notions of objects and subjects evolve, research tools and methodologies are also changing.
This transformation is contributing to the emergence of new practices and methods for ‘measuring’
phenomena that were previously difficult to quantify. The evolution of instruments and their impact in the field
of creative research are particularly interesting. Just as there are creators of works of art, there are also
innovators in the organization and design of instruments. Artists, designers, scientists and engineers frequently
collaborate to provide research and creation with platforms and tools that are not only at the service of
creation, but that also stimulate innovation among those who use them.
For example, MARCEL (Multimedia Art
Research Centres and Electronic Laboratories), of which the pioneer D. Foresta is one of the designers and
promoters, offers an online collaborative platform dedicated to the specific formats of the interactive arts, a
project that was born in 1997 and, even then, was the continuation of a process of reflection and practice that
began in 1981. This desire to offer shared tools is obviously not a recent development and is now reflected in a
wide range of proposals, from generalist platforms to tools dedicated to specific fields or uses. S. Thurow, D.
Del Favero, M. Scott-Mitchell, M. Ostwald and H. Grehan have developed a network modelling system which
facilitates the planning of opera rehearsals thanks to an interactive ‘cyber-physical’ spatial
aesthetic, opening the way to new imaginaries. C. Lengelé and P.-A. Gauthier traced the genealogy of
‘Live 4 Live’, an open-source tool designed to simplify the creation and real-time control of
spatialized sound objects. D. Cunin, E. Durand and M. Seta developed generic devices to facilitate the creation of
immersive environments with video tracking and collective interactivity on smartphones.
This trend towards
developing platforms for artists and designers is also to be found in the field of curation. Y. Hofmann, through
the ‘intelligent.museum’ project, has put forward a system based on data analysis for artistic and
cultural institutions. This prototype uses data from a variety of sensors in exhibition spaces to tailor museum
experiences to visitors’ needs. I. Družetić-Vogel, A. Fuchte and M. Bauernfeind proposed
‘ARt chat’, an application for museums that combines augmented reality, art and communication,
allowing visitors to share their impressions in the exhibition space.
These initiatives are not just about providing functional services; they can also take a critical
approach. V. Guljajeva, M. C. Sola and I. J. Clarke, for example, have analyzed several existing tools in the
field of AI, stressing that ‘AI does not create a work at the touch of a button, but requires a deep
understanding of the underlying technology, as well as a creative and critical approach’.
In addition
to the creation of platforms, there is also a trend towards modelling, although approaches vary considerably. The
very term model indicates a desire to provide communities with frameworks based on general principles to be
adopted. Z. Wu, D. Fei, X. Ma, M. Fan and K., for example, see NFT as ‘a sustainable business model for
media arts, involving audience interaction’. Others, such as D. Xu, M. H. Lamers and E. van der Heide,
propose a ‘relational model of co-located interaction’ between spectators and an interactive
work.
These various proposals, although heterogeneous, share a common view: they point to a constant trend
towards the uninterrupted emergence of specialized communities of practice that design generic devices for
instrumental purposes. These communities are also expanding thanks to the instruments they are inventing, enabling
their practices to be disseminated more or less widely, a form of ‘standardization’ with a creative
aim, while at the same time encouraging the integration of their own approaches to uses in the fields of art,
often including the active participation of audiences as a central component of the design of the works.
5. Breaking down disciplinary barriers and the emergence of new researcher profiles: evaluation as a tool for
legitimization
ISEA stimulates multidisciplinary exchanges while consolidating the existence of a specific community
through annual gatherings. The enthusiasm for creation, in synergy with technology and science, forges a solid
link between researchers and creators from a variety of backgrounds. This positive dynamic encourages the exchange
of diverse perspectives, whether practical, theoretical or methodological, around a common subject, a specific
field, or more simply a medium or instrument. The cross-disciplinary aspect of these interactions is particularly
enriching, as it helps to create a diverse community while respecting the disciplinary affiliations of each
individual.
Adopting a double-blind peer review process, ISEA’s call for applications conforms to academic
standards and promotes cross-disciplinary or multidisciplinary approaches. In an academic environment that is
often segmented by discipline, this recognition is crucial. The symposium thus provides a sanctuary for
unclassifiable minds, while giving them international exposure. The evaluation criteria we have chosen enable
artist-researchers and designer-researchers to assert their legitimacy.
The criticism often levelled at the latter - that the dilution of the boundaries between the arts and
sciences would jeopardize the quality of research and creation - ignores the symbiotic nature of these two spheres
of creation. In fact, they are mutually nourishing, while retaining their specificity, whether through collective
work or individual skills, not forgetting the importance of the distinct ecosystems to which these players belong.
Although this does not systematically guarantee the relevance of the results, whether scientific or artistic, this
approach is like that practiced in professional art and science circles, where peer judgement is commonplace.
ISEA’s aim is to bring these hybrid profiles to the fore. Although cross-disciplinarity can be
perceived as a risk of levelling out skills, it is essential for innovating, experimenting with new methods and
tackling issues from a fresh angle. The contribution of these cross-disciplinary approaches can be extremely
beneficial to established disciplines and sectors, enabling them to evolve and renew their own fields of research.
Venturing outside our disciplines through creative research allows us to rediscover them in a new light.
If practice-based research can create new knowledge in science and engineering, it also brings new
practices to the field of creation. Y. Blanchi, for example, links cognitive science and architecture, while S.
Gatz criticises architecture as a means of control by proposing a ‘cosmo-techno-poiesis’. Urbanism as
it exists is questioned by artistic and digital practices, like the infrastructures of networked urban screens
imagined and implemented by M. Thorogood, K. McCulloch and A. Dulic.
In dance, N.N. Correia, D. Souza, I. Nêves and J. Lobato are trying to make the dance process
understandable within a performance using a multisensory approach. I. Teles de Castro e Costa, H. T. Hong and C.
Y. Chen have designed a performance co-created with an autonomous virtual system. In the field of cinema, R.
Sagot-Duvauroux, N. Quaetaert, F. Garnier and R. Ronfard explore how film editing cannot be transposed into
virtual environments and must be reinvented. Questions about the transposition of creative techniques into VR also
arise in the case of Chinese calligraphy with Shum, P.Y.S. & Klein. C. Salter, T. Thomasson and P. Uro are
carrying out what they call a ‘theatrical exploration’ based on augmented reality, with the aim of
making climate change visible.
The ‘adjustment’ between different vocabularies is necessary to give an account of this
research. O. Kobryn, M. Couteau, R. Sagot-Duvauroux, S. Balcon, F. Garnier, R. Ronfard and G. Soulez, researchers
from different disciplines (aesthetics, ergonomics, design, engineering, cognitive sciences), are working to
reconsider the concept of the ‘virtual’.
In the cultural sector, curation and conservation are also invested with methods derived from
practice-based research to redefine or apprehend their activities in a new way. M. Jones, M. Wester and M.
Blottiere present a new approach to curating as practice-based research and propose the acronym ‘CRC’
to establish it as a specific field. D. Irrgang and M. Skłodowska take up the curatorial notion of the
‘exhibition of thought’ co-invented by Bruno Latour and Peter Weibel, who have put it into practice
with curators, artists and researchers at the ZKM.
The circulation of thoughts and practices, the mutual acculturation of different research players and
cross-disciplinary cooperation are all expressions that would remain abstract without these numerous experiments.
These compositional arrangements, to paraphrase Latour, make us see differently.
6. Perceive beyond our usual capacities to become aware of societal and environmental transformations
Making perceptible what naturally escapes our senses is not a new concept in the world of art, or
even in that of design, as the example of decasualization shows. This approach is even less novel in the field of
science, where modifying our perception and understanding of the world forms the basis of a great deal of
research. Nevertheless, the way in which this is done varies from one field to another, with a current tendency to
merge different approaches in order to make discoveries or visions of the world that are not immediately
accessible - often because of their scale - ‘sensitive’, i.e. perceptible to the senses.
In
this context, many research and creation projects aim to reveal new sonic and visual ‘landscapes’,
constituting what might be called an informational aesthetic. There are two main thrusts: the first seeks to
materialise technological artefacts, which are increasingly miniaturized and interconnected; the second aims to
raise awareness of climate change through the collection of biological or geophysical data, with the two thrusts
sometimes overlapping.
In the first category, H. Scurto and A. Chemla-Romeu-Santos generate soundscapes
from data collected in 28 locations around the world via the Locustream online platform, highlighting AI
infrastructures. On the other hand, M. Lukaszuk and S. Bahng exploit gesture and spatiality through
electroacoustic and video improvisation to make intelligible the preponderant role of machine learning.
In the second, Mc Farlane proposes a method of co-creating artistic acoustic ecologies centered on
the Great Lakes in the USA, combining sound data collection and research methodologies enriched by indigenous
knowledge.
In terms of sonification, L. Foo and J. Fritsch create a ‘cryogenic landscape’ to engage
listeners emotionally with climate change, broadcasting the movement and melting of the ice in Greenland almost in
real time, presenting the ice pack as a living being. J.C. Duarte Regino goes a step further by making atmospheric
processes audible, helping us to perceive our atmosphere beyond our usual senses.
At the intersection of
the two, L. Moren and T. Bachvaroff's ‘Under the Bay’ is an augmented reality project that turns
smartphones into ‘microscopes’ to explore marine life invisible to the naked eye. J. Ottavi and J.
Pickett are exploring the interaction between humidity and the decomposition of printed circuits, converting soil
acidity and electronic compost into electrical signals.
Visibility projects can also illustrate
disappearance, a radical type of invisibility. H. Sareen, Y. Fu and Y. Kakehi depict endangered species as
micro-bubbles formed by the nucleation of CO2 in water, which shrink and disappear in a few days to raise
awareness of the threat to biodiversity. B. Ammar-Khodja brings to life the residues of heavy metals in
contaminated urban landscapes, highlighting a kind of point of no return if the pollution persists. G. Legrand, V.
Thornhill, I. Clarke, through the installation ‘Spectral Plain’, explore how information technologies
can integrate and reflect specific socio-cultural beliefs, creating new symbiotic interrelationships with the
non-human and the non-living, thus influencing the participants' perception of their understanding of the world.
These practice-based research projects aim to make complex issues or abstract realities tangible.
Designed to transform our perspective, they encourage us to adopt different behaviours and develop new critical
approaches.
7. When creative research becomes critical action-research
N. Georgakopoulou and her collaborators aspire to create a ‘sonic consciousness’ within
urban spaces. Their project aims to create musical experiences that invite active participation from citizens,
while paying special attention to people with hearing and visual impairments. For his part, R. Ridgway strongly
criticizes the collection of personal data by the main players in the network. In the same vein, B. Gaylor and K.
Hennessy encourage the development of a critical understanding of Big Data to better counter the ‘piracy of
affect’.
From a techno-feminist perspective, G. Lautenschlaeger explores post-human maternity in media art. P.
Costa, L. Ribas and M. Carvalhais analyze the tendency to feminize digital assistants and suggest strategies for
reversing this trend. L. Haute urges us to think about our methods of collaborating with the environment, inspired
by the symbiotic culture of bacteria and yeast. C. Gould and Isabel Stenger urge us to take action to initiate
change in our lifestyles and consumption habits, while J. Garančs explores, through virtual reality,
dystopias based on market values.
Y. Zhang, in his interactive installation, uses machine learning to generate fictional speeches by Xi
Jinping and Donald J. Trump. P.R. İ. Yeginsu highlights the need for artists to find new ways of
collaborating to avoid the negative impacts of commercialization linked to NFTs. The fight against social
prejudice is also a recurring theme, with S. Brueckner, S. Yeung, J. Liu, D. Choberka, K. Shedden, J. Turner, I.
Gillet, M. Lu and Xingwen Wei identifying the biases present in face detection and race classification.
Faced with the environmental crisis, techno-solutionism is criticized through the surrealist and
‘pataphysical’ approach of M.-L. Bourgeois’s project, without systematically rejecting
engineering, as many of the projects are situated at the intersection of the arts and the engineering sciences.
The appeal to various forms of decolonization is also marked, namely with H. Rashtian and G. Aceves-Sepulveda who
question the construction of historical narratives.
These contributions, which oscillate between denunciation and calls for resistance, aim to provide
the public with critical, conceptual, sensory and emotional tools to raise awareness of the issues of pollution,
gender and democracy. The subjects addressed are varied, and the methods proposed are diverse. Although critics of
these works may fear that activism will predominate over rigorous research, it is clear that these contributions,
by their provocative nature, offer food for thought and relevant tools for the scientific community, inviting
ongoing debate among peers.
From this perspective, this type of action research fulfills a fundamental dual critical function: on
the one hand, it is devoted to exploring the invisible, hidden, unexpressed, or even oppressed elements within
society; on the other hand, it simultaneously engages in the struggle against doxas, including those with which it
finds itself unwittingly associated. This bifocal approach is relevant to all fields of study, encompassing both
research and creative work with a critical intent. This approach enables a clear distinction between activist
practices and research activities, thereby amplifying their respective impacts through an enriching critical
symbiosis.
Conclusion
As we have emphasized, the choice of these key strengths represents a challenge, due to its
necessarily limited and subjective nature. Not all the authors are cited in this introduction, due to lack of
space, and we can only encourage readers to read all the articles. We could have highlighted other aspects, such
as the importance attached to feminism, gender issues, the ecological transition, or the various processes of
decolonization. These are essential thematic entries, but they are already very present in current debates, and we
offer additional reading keys. It was also possible to group the proposals according to the type of technologies
used or invented, or by sector of activity. The thematic clusters we proposed during the Symposium could have
encouraged us to do this. However, we have chosen to base ourselves on all the contributions published in these
proceedings and to resume the programmatic work in ‘bottom up’ mode. This approach offers a new
perspective on how everyone can benefit from this work for their own research.
We hope that these guidelines illustrate, if proof were needed, that these proceedings are a rich
source of ideas, methods and formats that are both varied and innovative. It is also important to note that the
proposals go beyond the scope of these proceedings, and we encourage you to explore the online video archive
platform, including artist presentations, demos and roundtables.
These exchanges between art, design, science and technology are essential for generating knowledge
and creating meaningful works. They allow us not only to analyze the present (a form of real-time analysis), but
also to lay the foundations for future discoveries and new approaches that are still being developed. These
initiatives can be seen as precursors, offering a glimpse or intuition of what is emerging and could materialize
in the next ten to twenty years. In a way that complements science fiction, which imagines the future, these
proposals are starting points for extrapolating the present and considering possible future developments. A work
in constant evolution!
Paris, London, Bilbao, Montréal,
Emmanuel Mahé (ÉcoledesArtsDécoratifs–PSL
University, France, EU), Academic Chair
Elena Papadaki (University of Greenwich, UK), sub-thematic co-Chair
Maria Ptqk (Spain, EU), sub-thematic co-Chair
François-Joseph Lapointe (Université de Montréal, Canada), sub-thematic co-Chair
Three personalities were invited to give their views on current developments in the fields of
creation, technologies and sciences, on the role of these in society, and to present their most recent work.
Ionat Zurr (artist-researcher, Symbiotica, Australie) — Symbiosis and the fallacy of a nature-free
existence.
In times of ecological emergency, solutionist fantasies of nature-free human existence promise salvation and
repair. The innovative paradigm offers “products” such as lab-grown (animal free) meat and artificial
automated surrogates to replace reproductive biological bodies.
These so-called innovations require special artificial environments to host, nurture and culturally articulate
this “new” nature-free, decontextualized and colonised life. The entanglement of life with its
surrogate environment/apparatus, echoing human relationships with living and semi-living agents; when control and
care are employed to counter resistance.
Artists, scientists, designers and engineers all play their part in this transformation and its effects on human
relations with life and the environment. This creates a range of ontological conundrums and fantastical
expectations as to what technology can provide and to whom. Using examples of artistic research that deal with
emerging technologies and new knowledge, Ionat Zurr narrated artists’ symbiotic and parasitic relationships
with such post nature.
This talk was framed by the imminent closure of SymbioticA, the first artistic research laboratory based in a
life sciences department. SymbioticA began as a symbiotic act, embodied in an academic institution, to enable
critical, yet mutualistic, relations among artists and scientists. Many of SymbioticA’s alumni have
continued to establish their own laboratories and artistic practice in other academic institutions around the
world, leading to the growth of the field of Biological Arts.
SymbioticA is now being treated as a parasite by a changed host body. Is this a ‘natural’ survivalist
rejection against a foreign body or can we detect symptoms of an autoimmune disorder?
Biography: Dr Ionat Zurr is an artist-researcher. She is the Chair of the Fine Arts Discipline
at the School of Design & SymbioticA academic coordinator at the University of Western Australia. Together
with Oron Catts she established the Tissue, Culture & Art Project in 1996 and their co-authored book Tissues,
Cultures, Art, published by Palgrave McMillan this year. Her collaborative work was exhibited by Pompidou Centre,
MoMA NY, Mori Art Museum, Ars Electronica, National Art Museum of China and more. These ideas and projects reach
beyond the confines of art and the work is often cited as inspiration to diverse areas such as new materials,
textiles, design, architecture, ethics, fiction, and food.
Viktor Ruban (Choreographer, independent culture diplomat, Ukraine) — Coping potential of
creativity and art-practices in times of war: culture diplomacy, fundraising, curatorship and art-therapy force
project.
If symbiosis is an essential notion, it is even more so in today's geopolitical context. War, terror and massive
extinction of people in Ukraine by Russia is a global challenge shaping our global future at this very moment. In
this situation performing artists in Ukraine keep on strong commitment to help in any possible way. Since the
beginning of the full-scale invasion in 2022, the situation is forcing us to search for not only new ways of
dealing with challenges creatively but to discover new ways of implementing our art practices and skills to
completely new levels and spheres, such as resistance in information wars, culture ecology maintenance, new forms
of fundraising and help with physical and psycho-emotional recovery — same as military and civilians
—.
During his session Viktor Ruban, shared information about initiatives that he initiated or is involved in, such
as the European Culture Parliament, and culture diplomacy challenges that he faces on different international
events; Ukrainian emergency performing arts fund and funding challenges for the independent performing arts scene
in Ukraine; international solidarity events and visibility of Ukrainian actual art scene—why it is
important; actual creations in Ukraine and trends seen through the actual national theater prize season; the
development of the project for training “psychological first aid instructors for military from the front
line” and Art therapy force project— a range of activities implementing art-practices and working with
creativity for psycho-emotional health recovery, coping with stress and panic attacks as well as preventing
self-destructive behaviors and PTSD for diverse groups of people.
Biography: Viktor Ruban is choreographer-researcher, curator, performer, educator, independent
culture diplomat and culture activist. Initiator and ambassador of Ukrainian Emergency Performing Arts Fund
initiative, he represents Ukraine in European Culture Parliament. He is also program director and co-founder of
venue #KyivDanceResidency—platform for international studies in somatic, dance and performative practices,
movement-based art and research. Ph.d. student in culture studies of Modern Art Research Institute of the National
Academy of Arts (Kyiv, Ukraine)
Michael Century (musician and cultural theorist, Rensselaer Polytechnic Institute, NY) —
Nonsynchronous Innovation: Periodizing the Digital
While the digital transformation continues to outpace socio-institutional adaptation, the technological arts have
moved on to vastly expand their temporal horizons. Timescapes of artistic research and creation now embrace the
residual as much as the emergent; techno-diversity against digital solutionism; sympoietic rather than linear
models of innovation. But nonsynchronous innovation is hardly unique to the current moment, as revealed in Michael
Century’s recent book Northern Sparks, on Canada’s early experimentation with digital media.
Poised as a “counter-environment” to the great powers, in McLuhan’s phrase, Canada’s
experience of the transitional decades into the information age was grounded in a technological ethos that
emphasized sensorial immediacy, embodied interaction, and improvisatory expression. This alternative ethos was
situated between a pair of distinct yet inextricably bound forces, one national-political and proper to Canada,
the other techno-mediatic and global in scale.
The unraveling of these forces by the late millennium reveals innovation itself as a complexly drawn process
comprised of multiple layers with fluctuating degrees of synchronization. From a cross-media perspective, Northern
Sparks also reveals how the differences between the arts with respect to improvisatory immediacy and discrete
formalization make any neat chronological periodization of the digital problematic.
Biography: Michael Century, musician and cultural theorist, is Professor of New Media and Music
at Rensselaer Polytechnic Institute. His book Northern Sparks: Innovation, Technology Policy, and the Arts in
Canada from Expo 67 to the Internet Age appeared with MIT Press in 2022. At the Banff Centre, he founded the Media
Arts program in 1988. Century’s works for live and electronically processed instruments have been performed
and broadcast in concerts and festivals internationally.
How do art and science participate in writing a world of signs and symbols? What do these two human faculties
and capacities seek and how do they differ radically? Science has progressively freed itself from its purpose
(the search for the why) to model, imitate and master a world in an increasingly abstract manner. If science and
art are performative, what world do they draw and what do they tell us (in terms of discourse) about the human?
As science gradually becomes techno-science and produced by industries, does art have a singular role to play in
the interrelation of these two poles, science and industry, in order to re-inhabit the ethical questions and
meanings that were originally in these other two creative capacities? Can it and should it?
Keywords
Art, Science, Industry, Signs, Sensibility, Technoscience, Blindspot, New cooperative ecosystem, incorporation
of experience.
We will share an assessment of the relations between art-science and industry, the novelty being that with
digital technology the questions have been accelerated, augmented and have now come to integrate the notion of
physicality. Industry is a playground; whether it is science, art, how do we seize the digital? Do we have the
means to think the digital or are we alienated, with the digital no longer as an externality, an interface, but
as an immersion, as a new space of being in the world—particularly with the metaverse—as a
socio-technical space? It is necessary to think of industry as a workbench for virtual universes and as a space
of reconciliation between disciplines via the designers as sensitive mediators and scouts of the new virtual
worlds. We need a new epistemology of the virtual worlds that takes into account the multi-dimensional human. We
are looking for ways between micro-ecologies, localities, craftsmanship and a global, automatic approach, the
two poles that build our "being in the world."
Art could be thought of as the fabrication of new categorizations of an environment¹ that escapes the
norms of language and conventional imaginary modelling. Because the perceived world is always the result of an
interpretation and a shaping which are imposed a posteriori by culture, that the inaugural perceptual
singularity is subsumed by culture. The perceptual thresholds linked to temporal representation structure a
vision of reality constructed by stops and artificial linearity². Yet the imaginary, the perceptual virtual
cannot be stopped at these arbitrary forms of actualization of the world. Whether it be time, imaginary, or
spatial perception, they cannot be detached from a body in action that perceptively actualizes a world that
appears and is created in a living time (dancers or performers give rhythm to time, for example). The
manifestation of the event is an extract of the real, obliterating the infinity of data, infraliminary and
supra-liminary information³ not actualized by language whose readings cannot reach the threshold of a
disciplined thought. It is in this failure of a completeness of reality that art operates.
It is an attempt to reveal hidden information inaccessible to authorized cognitive processing. Other
articulations, forms, relations to the world exist that art tries to bring to light. It is a raw, unanalyzed
grasp⁴ that the act of formalization and expression will seek to bring into existence. Art is, first of
all, an impossible return to an instinctive and affective perception, where the body echoes the world and where
the world is then echoed (exosomatisation). Art in these premises is an anchoring (Harnard Stevan⁵) of
signs of the world kept in memory (retention), attached to and mirroring the body that perceives itself (while
keeping the unspeakable abyss which Pierre Legendre refers to ⁶).
As a first reflexive return, art imprints the world in order to reflect on it and to appear itself (first
distance from the world—see the cave paintings and the work of Carole Fritz⁷). It is a primal
affective reading of a world that is not there for it, and for which it seeks a meaning. It is an attempt to
answer the abysmal question of why and of a world that is not up to his standards, but in which, through the
inscription of signs, he can make his presence last beyond his own finitude.
As the first interface between the body and the environment, art is a sign of principle writing. As a symbol
and practical return of a memory and of an instant kept⁸, it is a means for recalling presences. It is an
imprint of an existence and of a difference made with the self and the environment, the self and the other, who
delimits and recognizes itself in it. Through the diffusion of signs and symbols, language and rituals, the
particularity of a seized moment becomes an abstraction, a practical medium to think about the world and save
the detailed and vivid perception that every physical and phenomenal novelty requires. But if art has to do with
the first symbolic anchors, the first perceptions, it cannot be reduced to them.
Art is a first reading, a projection of human meaning on the world, and in this, science does not differ from
this attempt to understand and mark the world. In science, the purpose is different, the tools are more
sophisticated, no singularity is demanded and the dream of a universal and standardised objectivity.
Art and science are imaginary and projective capacities, part of a process of fabrication and transformation of
the world. This capacity for transformation by modern science has largely emancipated itself from its role as a
myth whose function is to answer the question of why live and to limit unreason in order to support existential
anguish⁹.
But art and science are not only intended to think about the writing of a world and its human legibility, but
also to put it into the world, to give it a stage, to create a narrative for it and for the human being. Their
discourse and imagination are performative.
In contrast to science, art is anchored in the body and the affection of the world, turned towards the
appearance, the distinction, the singular discrimination of signs as a search for actualization. Science in its
hyper-technical devices sets aside this affection for the body, and is based on instrumental rationality.
It is important to underline representation acts as a perceptual framework indicating the phenomenon and giving
it an interpretative form in order to apprehend it.
Science, as a rationalized exo-body, it gives us the opportunity to perceive a phenomenon (a
techno-phenomenology) and a world framed in advance by the laws inserted in it, offering a metric vision and
mathematical logics, influencing in its turn the way of writing and understanding a world surrounded more and
more by technical abstraction, self-referential and discarding the perceptive variability of a sensitive and
feeling body.
Techniques are extensions of organs¹¹ reading and writing the signs of the world. Art has taken an
interest in the world of science and technology out of a spirit of resistance (perhaps) to the homogenization of
bodies and sensibilities, out of a refusal to accept the authority of scientific discourse, which sometimes
imposes "butchering"¹² and reductionist views of the human being. The definition of man varies
according to culture, religion, and time. Feeling human, our relationship to the world is defined according to
the interpretative frameworks conveyed in a culture, which cannot be absolute. We know that the way in which man
is expressed influences in turn the experiential potential and the power of his experience.
“Cogito ergo sum”
“Cogito ergo sum” of Descartes has been the heart of the scientific domination of our
society’s world-view, the way we understand the world defined uniquely through reason. It has been
responsible for an unequaled level of material progress but has reduced humanity into entities befitting
uniquely mechanical formulation, ignoring the whole human being, concentrating only on those aspects of the
person considered important for the task at hand. In recent western history we, as a culture, have tried to deny
this human wholeness by declaring that rationality was the only approach to true knowledge and that all else
must be rejected. Anything resulting from feeling or emotion was unreliable and even dangerous. That added to
the mechanically quantifiable view of how things work has created a distorted perspective, a world-view based on
a diminished idea of the human being. Focusing on the intellect was a useful but limited approach to how
humanity is defined putting aside, "Sensio ergo sum"—I feel therefore I am—which is how most people
first react to sensory input. Recognising that aspect of humanity not only allows us to understand the whole
human by respecting the other half, it also connects our species with the rest of the living world which reacts
principally through feeling. Objectivity is a tool, an important one, but to imagine humans cut off from their
emotional side is dangerous and leads to often undefined but very real frustration as we see more and more
today.
The capacity for symbolic thought has always been part of our makeup in its earliest primitive forms since the
beginning of our genus and most probably well before. It comes from an instinctive and unintellectual operation
of that curiosity which is part of every living being’s functioning as a way of exploring our environment
and surviving in it. When the found object of our ancient ancestors, the first overt manifestation of what we
now know as art, was kept and shared with others it acquired a symbolic sense for the group as well as the
individual who found it, an unarticulated meaning but something felt to be important. With the evolution of our
intellectual capacity, this process becomes more elaborate in providing a «why» to that selection
giving those objects an explanation of their symbolic value and a means of communicating it and often the
beginning of what can be seen as a religious belief system. In the beginning, this was not an intellectual
exercise and the choosing came from feeling rather than understanding. It became more intellectually elaborated
in time through the need to be communicated, probably diluting the experience of the first-person experience but
making it transferable to others and incorporating other reactions to it.
This is the root of artistic creativity and its eventual integration into culture. The symbolic is very much
part of us and ignoring it or pretending it is less important or to be overcome demonstrate a dangerous
misunderstanding of how humans function. To attempt to understand our confrontation with the real, both art and
science are needed for a fuller and more human comprehension of what we are defining. Understanding
intellectually is necessary but stopping there leaves the human wanting more, which defines the emotional
atmosphere we live in today. We cannot expect people to understand when half the person is ignored. In our
mechanical world-view art has been pushed out of its central role of informing into manageable categories to fit
the demands of the market it has been confined to.
Artists, on the other hand, have never ignored science, particularly in a society which has allowed science to
be the sole arbitrator of our reality. Science has thus become the matter of art, our reality subjected to the
questioning of art. Technology, the product of science, has always attracted artists because of a curiosity
about what humans do but also because it is a potential tool, particularly the technologies of communication.
Importantly as well, the artists’ use of technology has often influenced our perception of it and how it
is integrated into society and affects us.
A technology developed through both art and science would, by definition, have more dimensions by being the
product of the full human being. This has been the objective of our project MARCEL, an international group
dedicated to artistic, scientific and educational uses of the network and expanding network technologies through
the demands of art and not just market-driven consumption. It exists and evolves through the share experiences
and technical development from creative uses of the network pushing the boundaries of the possible. This is also
the approach that the TRAS network (transversal network of art-science networks) is trying to put in place in
France. As are the creative platforms in industry that seek to give meaning to technology, since industry is the
heart of its production.
Industry as a blind spot?
Industry is perhaps a blind spot in our times. Principally thought of as an operative process, a technical
milieu and not enough as a thinking object. As a civilization manifestation of a way of seeing the world.
Industry always been a space of intervention and convergence in which an art-science dialectic can flourish.
At the crossroads of the most advanced human creative activities, the industry is projecting a vision,
organizing the work and the economy defining every aspect of everybody’s life. The industry today
manufactures the eye and the prosthetic body of tomorrow, just as science and art have done. To think of working
in theatrical and artistic terms on science, digital environment or technologies that enable all aspect of the
modern world without taking into account the environments in which techno science is constructed may be a
mistake. The question of the representation of the body, of what constructs the human experience, of what links
the human to the world has never been so obvious since the announcement of the promise of virtualization and
modelling of the human being, from his intimate relations to the notion of habitability. Virtual worlds (beyond
the definition of Metaverse) promise us a world in which the human experience could be reconfigured in a bespoke
manner without the canal of affective and invisible human dimension, of the anchoring of the symbolic through
the affective path, of the word that introduces us to subjectivity and otherness, a word that confronts us in
its strangeness and its inexhaustible incompleteness.
The standardization of the perceptible, the logical and scientific determinism could reach its apex in these
new universes that redesign the conception of the human. It is therefore in the interest of artists and
designers to present the eminently open conception of a world thought out by art and designers so as not to
close the vision of the human to an automaton-avatar with which it would be a question of playing and moving in
the way of a puppet. A human representation that disregards the gravity of bodies, the rites of social
interaction, the collective to be woven, the immersion in a living half-place where the segregation of the
visible is decided upstream and selecting what can be seen from what does not deserve to be seen. The human
being is not a pack of information to be grasped, he is a being of dreams and words and it is better to know
this than to ignore it. It is in the interest of art and design, philosophy and anthropology to integrate these
decision-making circles of future representations that will be conveyed with an unequalled power of diffusion
and democratization. There is a stake in everyone being able to grapple with the question of the relationship to
the environment that we can rethink, the adventure of discovery, of the process of making, of surprise and of
creation, which is never elaborated in predetermined data, but which emerges from the unpredictable and the
unknown.
How can we open up these industrial worlds to the civilizational challenges they face today? How can we make
people understand what design can do at the intersection of art, engineering and industry? Design is the
keystone of the human intentions of any industrial project deploying the openness of the human senses, of human
abilities and expectations into a project, in a tangible and accessible manner improving the human benefice
“in fine”... Design is drawing and aim at the same time, a potential direction as a line of conduct
and orientation to be incorporated into any industrial and societal project... How can we reintegrate meaning
upstream of this productive force so that it serves a majority in humility and equity? What are the first tracks
and trials of these co-operations and new ecosystems where art-science-industry work together again? What are
the first attempts made by industry, which must rethink itself at a time of climatic emergency and where the
production of waste objects, of intensive innovation based on the imagination of mass production, of Taylorised
work and progress must be reinvented? How can we reinstate the sensitive in these industries that produce
techno-imaginaries? How can we reconstitute noetic spaces ranging from industrial environments, experiential
hybrid environments, to virtual immersive universes (metaverse)?
Can we imagine and draft new roadmaps to rehabilitate Art without its instrumentalization in the scientific and
industrial spaces where today most of our narratives and actions that construct representations are being
produced? Art is probably the future of industry understood as capacity for representation for its power to
create new bodily and cognitive experiences, and its possibility of creating dreams and new solidarities. This
implies putting art back into life, as medicine of the imagination, as the art of being together. Should it, can
it?
Such regenerative new hybridization and such new forms of cooperation would represent a new symbiosis of the
creative forces of the human being, in which art would, in that case, become design, design as a mindset and
“way to think.” Science and industry could collaborate for human and non-human benefit driven
project. A project which path remains open to matter, to the singular, to the body in “totality,” to
the living, to the unpredictable and immeasurable, which is the nodal point of invention.
This utopian symbiosis we are describing is what we feel is right to focus on, in order to build the narrative
and the bases of a dialogue to be reinvented so that all human faculties are no longer dissociated, but reunited
to serve an imminent civilizational project. Considering all the living in all their diversity and unknowability
as the preciousness of life should be taking into account in the course of our actions. We need a holistic and
systemic approach within the limits of the consequences of the harmful and positive impacts that we can and must
imagine. Industry must reclaim its original role of building a world¹³ to be improved, to be inhabited
for all and in the perspective of diversity. It is a call for a collaborative symbiosis that takes into account
the complexity and the need to work together, despite the stereotypical oppositions that prevent these human
universes from interacting again.
Beyond the civilizational stakes, industry is today a playground for art and science, and dialectic between art
and science is vibrant. It questions their relationship, the tools they manipulate, but also because industry
must regenerate its raison d'être (get out of consumerism, functionalism). It must become
organic. It is when we have a calmed art-science dialectic that industry will be able to resolve these new
issues.
Another question comes to mind: does industry contribute to the discourse of technoscience, or is it the victim
or the keystone? With cultural and economic globalization, its responsibility is taking the main stage... in its
making of the world and tomorrow's world, the industry cannot be reduced to a means of progress nor an
externality functionally assess but as a subject for thought, an object for design... and Art. Shouldn't the
hyper-strategic position of industry be aestheticized as an experience with methods of art and science, i.e.,
art applied to industry which is called design? Industry has reached a level of maturity that is no longer under
the yoke of politics, it is political. As such, industry must reconvene art and science within itself according
to its own responsibilities. An industry that thinks, reflexive to change the world will not arise without the
aesthetic and ethical sense. An industry more in tune with a regenerative mindset will enrich a new form of
relationship with Nature, Human and the world, an artistic approach from within, to nurture a sustainable future
and solutions that undoubtedly will be found.
References
1 Gilbert Simondon, Imagination et Invention, 1965-1966, Paris, Presse
Universitaire de France, 2014.
2 Henri Bergson, L’évolution Créatrice, Paris, Presse
Universitaire de France, 2013.
3 Günter Anders, Et si je suis désespéré que voulez-vous
que j’y fasse, Paris, Allia, 2010, p.72.
4 Charles Sanders Peirce, Écrit sur le signe, Paris, Seuil, 1978, p.23.
6 Pierre Legendre, Les enfants du texte, Paris, Fayard, 1992.
7 Carole Fritz, L’art de la préhistoire, Paris, Citadelles et Mazenod,
2017.
8 Ernst Cassirer, La philosophie des formes symboliques, le langage, Paris, Edition
de Minuit, 2017.
9 Pierre Legendre, Le visage et la main, Paris, Les belles lettres, 2019.
10 Jakob Von Uexkull, Milieu animal et milieu humain, Paris, Broché, 2010.
11 André Leroi-Gourhan, Le geste et la parole, Technique et langage, Paris, Albin
Michel, 1964, p.49.
12 Alain Supiot, La gouvernance par les nombres, Paris, Fayard, 2015.
13 Pierre Musso, La Religion Industrielle, Paris, Fayard, 2017, p.12. Journal article
(online)
5 Stevan Harnard "Le problème de l'ancrage des symboles" Physical Review D, 42, accessed
November 7, 2022, https://archipel.uqam.ca/14111/
Bibliography
Anders Günter, Et si je suis désespéré que voulez-vous que j’y fasse,
Paris, Allia, 2010, p.72. Bergson Henri, L’évolution Créatrice, Paris,
Presse Universitaire de France, 2013.
Cassirer Ernst, La philosophie des formes symboliques, le langage, Paris, Edition de Minuit,
2017. Fritz Carole, L’art de la préhistoire, Paris, Citadelles et Mazenod, 2017.
Harnard Stevan, “Le problème de l'ancrage des symboles", Physical Review D, accessed November
7, 2022, p.42, https://archipel.uqam.ca/14111/
Legendre Pierre, Le visage et la main, Paris, Les belles lettres, 2019.
Legendre Pierre, Les enfants du texte, Paris, Fayard, 1992.
Leroi-Gourhan André, Le geste et la parole, Technique et langage, Paris, Albin Michel,
1964, p.49.
Musso Pierre, La Religion Industrielle, Paris Fayard, 2017, p.12.
Peirce Charles Sanders, Écrit sur le signe, Paris, Seuil, 1978, p.23.
Simondon Gilbert, Imagination et Invention, 1965-1966, Paris, Presse Universitaire de France,
2014. Supiot Alain, La gouvernance par les nombres, Paris, Fayard, 2015.
Von Uexkull Jakob, Milieu animal et milieu humain, Paris, Broché, 2010.
Authors’ Biographies
Edwige Armand is an artist and a teacher-researcher in art and culture at the Purpan engineering school (INP
Purpan) and is attached to the LARA-Seppia laboratory. Passionate about the history of science and art, and
philosophy, she combines these different disciplinary fields around the question of techniques and sciences.
The modifications of the relationship to reality, subjectivity and the body by techniques and sciences are
central axes in her research-creation. The transversality of disciplines allows him to question more broadly
the processes of creation and the transformations of representations that the arts contribute to. To revive
the dynamics of art-science relations, she co-founded and has chaired since 2016 the association Passerelle
Art-Science-Technologie, which works to bring these disciplines closer together, and she is also involved in
the Transversale des Réseaux Arts Sciences.
Anne Asensio, after executive and design management positions in the automotive industry, joined Dassault
Systèmes in 2008 as Vice President Design Experience. She created the Design function of Dassault
Systèmes and the Design Studio, bringing together a multidisciplinary team in innovation strategy
through design, experience design and design research. Advocating a participatory approach to new technologies
and virtual worlds, the Design Studio supports Dassault Systèmes' customers in high-growth industries in
their transformation, digital and sustainable innovation needs towards virtuous design processes. Imagining
alternative scenarios to transform the world we live in into a more sustainable and desirable one, the studio
engages in reflection, experimentation and confrontation of cross-cutting approaches between social and
technological issues through creation.
Don Foresta is an art theorist using new technologies as creative tools. Specialised in art and science, he
has been a professor at ENSAD and the Ecole Nationale Supérieure d'Arts - Paris/Cergy and a research
fellow at the London School of Economics. He has spent 35 years transforming the network as an artistic tool
and is working on the creation of a permanent high-speed network, MARCEL, dedicated to artistic, educational
and cultural experimentation. In 1981, he made his first online exchange between the Center for Advanced
Visual Studies at MIT where he was a fellow and the American Center in Paris where he was director of the
Media Art program. In 1986, as curator of the 42nd Venise Biennale, he made the first computer network used by
artists. He was named Chevalier de l'Ordre des Arts et des Lettres by the Ministry of Culture.
Symbio(geo)sis: When mineral and biological species meet
MonikaBakke
Adam Mickiewicz University Poznań, Poland monika.bakke@amu.edu.pl
Abstract
As biocentrism is increasingly seen as unjustifiable, artists and environmental humanities scholars become more
attentive to feedback between minerals and (non)human life. First, I discuss the urgent need to reinvent
narratives and create new vocabulary about the abundance of life-mineral connections as presented by Mabe
Bethônico in her Stone Statements Editions, which focused on developing "a vocabulary of
proximity." Next, I analyze Magdalena Abakanowicz’s artwork Space of Unknown Growth,
investigating emerging material relations between concrete (an anthropogenic rock) and nonhuman life. I
argue that rather than limiting the environment to a symbiotic community, these artworks encourage the
curiosity and attention necessary to embrace mineral-life connections and envision more inclusive communities of
the future.
Keywords
Art and science, geology, minerals, technofossil, concrete, Long Ecology, biocentrism
Biocentrism is increasingly viewed as unjustifiable. Minerals are everywhere, but in Western thought, nonlife
has always been significantly overshadowed by robust, organic life. The multispecies studies have so far
embraced the diversity of mineral species only in a minimal capacity. Although biocentrism still predominates,
change seems inevitable. The currently emerging trend in the humanities and arts to truly embrace mineral
species (considered nonlife or geos) is growing in significance, and it is attentive not only
to environmental concerns but also to new developments in mineralogy. The latter focuses on the co-evolution of
minerals and life and the unprecedented increase in the diversity of human-made and human-mediated mineral
species and novel materials characteristic of the Anthropocene.
Vocabulary urgencies
To better recognize the complex relations within geo-biotic communities, it is necessary to reinvent
attitudes and narratives about mineral species. This need was addressed by Brazilian artist Mabe
Bethônico, who called for the development of a «vocabulary of proximity» of minerals and
life.
Bethônico created StoneStatements Editions in response to the question: How Will We Live
Together? posited by the curators of the Venice Biennal of Architecture in 2021.
Bethônico’s editorial proposal includes five inspiring book tiles: Geoimaginaries; Conscious
Rocks; Human Invasion in the World of Stones; When Stones Collect Diggers, Robbers, Queens and Kings;
Missing Words for Considering Stones, Rocks, Pebbles and Mountains: A Vocabulary of Proximity. Only the
latter has been developed so far as a collective effort, resulting in a volume of twenty-six short entries. As
Bethônico stated, the remaining books are «imagined, wished for, and suggested to be
developed» as more vocabulary to think about the environment in more than biological terms is urgently
needed.¹ Her project recognizes the
biocentric bias and the conditions of its operation, namely the shortage of concepts, stories, and creative
practices to embrace mineral species.
Anthropogenic rocks
The proximity of life and minerals, especially rocks characteristic of the Anthropocene epoch, is investigated
and anticipated in the artwork Space of Unknown Growth by Polish artist Magdalena Abakanowicz. Located
in the forest of Europos Parkas, this artwork comprises twenty-two concrete boulders in four sizes, spread over
an open-air museum near Vilnius. Concrete, although anthropogenic, is considered a "geological entity" due to
its close relation to the mineral composition of the earth. It is a novel rock and, as geologist Jan Zalasiewicz
suggests, “the most abundant anthropogenic sedimentary rock on the planet.”² Classified as a
technofossil together with a great variety of synthetic compounds, concrete is emerging as the prominent
signature of our geologic epoch. There is, however, a particular risk of succumbing to the lithic allure
recognized by Stacy Alaimo, who calls for caution when suggesting that “attending solely to the lithic
imports delusions of separation and control."³
Space of Unknown Growth speculates about a future that belongs to multispecies communities whose
endurance is anticipated rather than predicted. For surviving, making kin with unexpected others, as Donna
Haraway suggests, is necessary. Staying with the concrete means "staying with the trouble", which demands that
the viewers acknowledge the devastating environmental impact of extracting resources and recomposing
materials.⁴ And yet, Space of Unknown Growth does not convey alarm. Instead, it points out that
adaptation goes together with the transformation of anthropogenic lithic environments, which accommodate more
than human life and is a condition needed to renew planetary species diversity. Living on the ruins is possible
as some organisms can modify the lithic surfaces in their environment. The technofossils that comprise Space
of Unknown Growth indicate the anthropocenic Earth’s habitability. The anticipated growth
speculated on in Abakanowicz’s work is conditioned by the composing and decomposing of biological and
mineral species and the resilience of communities capable of terraforming concrete—the most earthly rock.
Long Ecology
Bringing anthropogenic stone to the forest introduces a perspective of what Jeffrey Jerome Cohen calls Long
Ecology oriented toward the future and vastly exceeding current waste management. However, concrete, as “a
material that mediates transcience and endurance," is not as long-lasting as stone.⁵ It rejects the
conventional belief that geological processes are always long and slow. From Long Ecology's perspective,
although prolific in novel mineral species and formative for technofossils, our epoch might not be capable of
reaching a distant future because anthropogenic minerals might become extinct.⁶
And yet the stones of Space of unknown growth might be considered erratics, conventionally defined as
rocks not matching the local bedrock. Erratics are lithic strangers, newcomers transported to current locations
by glaciers or other natural forces of significant magnitude. Concrete boulders of Space of unknown growth
are synthetic rocks and mineral novelty brought into the environment of a central European forest.
Yet, Abakanowicz’s erratics establish a category of their own as they are art forms, which
nevertheless participate in the currently intense processes of relocating minerals by humans—activity
believed to exceed even glacial action. Abakanowicz’s lithic newcomers, however, with their own material
history, become with their new environments through complex geo-bio relations and processes. These
unexpected others, strangers to their landscapes, arriving from a different time and space, introduce the
possibility of a radically new narrative of connectedness and multispecies kin-making, which stimulates yet
another way for humans to practice attentiveness and care. They demonstrate both vulnerability and resilience,
which characterize earthly worlding and unworlding.
Although the sculptures operate as erratics in the context of Long Ecology, there is no separation between them
and the materiality of their location. Through their lithic ways of being, the works participate in planetary
processes operating, in Cohen’s words, as “...an affectively fraught web of relations that unfolds
within an extensive spatial and temporal range, demanding an ethics of relation and scale.”⁷ In this
sense, Space of unknown growth is deeply ecological in its attentiveness to the vast scales in
which life becomes stone and stone becomes a living environment.
Coda
What we need now is not a belief in the solitude and indifference of stones, rocks, and pebbles but a
celebration of their inextricable connection with life and creative elaborations on their diversity. Both
StoneStatements Editions and Space of Unknown Growth inquire about ways to articulate how
minerals and life actually attract and seduce each other and how minerals open up to life’s intimate
strategies, the outcomes of which cannot be determined. They offer vocabulary and materiality to tell stories of
Symbio(geo)sis. Rather than limiting the environment to sym-biotic community, these works encourage
curiosity and attention needed to embrace mineral-life connections better. Only then will geo-biotic communities
emerge as the answer to the question of how we will live together.
2 C.N. Waters, J. Zalasiewicz, “Concrete: The Most Abundant Novel Rock Type of the
Anthropocene,” in Encyclopedia of the Anthropocene, ed. Dominick A. Dellasala, Michael I.
Goldstein, Oxford, Elsevier, 2018, p.75, https://doi.org/10.1016/B978-0-12-809665-9.09775-5.
3 Stacy Alaimo, Exposed: Environmental Politics and Pleasures in Posthuman Times, U of
Minnesota Press, 2016, p.149.
4 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke
University Press, 2016.
5 Richard D. G. Irvine, Anne Bevan, “Concrete Buys Time: Art and Anthropology in the
Anthropocene,” Worldviews: Global Religions, Culture, and Ecology 26, no. 3, October 19, 2022,
p.182, https://doi.org/10.1163/15685357-02603009.
6 Jan Zalasiewicz, Ryszard Kryza, Mark Williams, “The Mineral Signature of the Anthropocene
in Its Deep-Time Context,” Geological Society, London, Special Publications 395, no. 1,
2014, p.112, https://doi.org/10.1144/SP395.2.
7 Jeffrey Jerome Cohen, Stone: An Ecology of the Inhuman, U of Minnesota Press, 2015,
p.41.
Bibliography
Stacy Alaimo, Exposed: Environmental Politics and Pleasures in Posthuman Times, U of
Minnesota Press, 2016.
Jeffrey Jerome Cohen, Stone: An Ecology of the Inhuman, U of Minnesota Press, 2015.
Donna J Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke
University Press, 2016.
Richard D. G Irvine, Bevan Anne, “Concrete Buys Time: Art and Anthropology in the
Anthropocene”, Worldviews: Global Religions, Culture, and Ecology 26, no. 3, October 19,
2022, 179–95, https://doi.org/10.1163/15685357-02603009.
C. N. Waters, J. Zalasiewicz, “Concrete: The Most Abundant Novel Rock Type of the
Anthropocene”, In Encyclopedia of the Anthropocene, edited by A. Dominick Dellasala, Michael
I. Goldstein, Oxford: Elsevier, 2018, 75–85, https://doi.org/10.1016/B978-0-12-809665-9.09775-5.
Jan Zalasiewicz, Kryza Ryszard, Williams Mark, “The Mineral Signature of the Anthropocene in Its
Deep-Time Context”, Geological Society, London, Special Publications 395, no. 1, 2014,
109–17, https://doi.org/10.1144/SP395.2.
Author Biography
Monika Bakke is associate professor in the Philosophy Department and director of the Environmental Humanities
Center at the Adam Mickiewicz University, Poznań, Poland. She writes on contemporary art and aesthetics,
particularly interested in posthumanist, transspecies, and gender perspectives. Her curatorial work includes
art exhibitions: Bio-Reminiscences (Poland), Seeing the Forrest Through the Trees (UK),
Boundless Objects (Portugal), and Refugia: Keep (Out of) these Places (Poland). Currently,
her research focuses on nonlife forces and new articulations of mineral becomings in contemporary art.
https://amu.academia.edu/MonikaBakke
4E Cognition for Symbiotic Architecture?
YannBlanchi
BMOffice France yann@bmoffice.fr
Abstract
With the aim of reconsidering the very nature of architecture, we propose a conceptual tool to think of
architectural apparatuses as actors within a continuum composed of both artificial and natural agents. For
this,we look at the cognitive sciences, particularly the 4E cognitive (embodied, embedded, enactive, and
extended). We use examples from contemporary architecture to test our hypothesis and thus attempt to define what
symbiotic architecture could be.
Keywords
4E cognition, Embodied, Embedded, Enaction, Extended, Symbiotic, Architecture, Artificial, Natural
In this paper, we will explore a theoretical tool to think symbiotic architecture. To this end, we consider
whether the transposition of the 4E cognition may be useful in describing the characteristics of what the
symbiotic architecture might be. The paper is organized as follows: Firstly, we set the context and define the
terms used, such as symbiosis and 4E cognition. Then we state our hypothesis of an analogy between 4E cognition
and architecture. Next, we test this idea with examples of recent architectural apparatus, and we conclude.
Context
In 1960, as Gordon Pask stated in his theory of conversation, we moved from a functionalist world to a
mutualist world. Today, in the context of the global crisis, nature is coming back into the loop. The ecocentric
logic would reinscribe us in a network of multiple interactions and concrete connections with the milieu,
engaged in a network of dependencies in a way of decentralized connections.1 Nature and artifice
are bound to co-evolve. Under these conditions architecture can certainly no longer be the static and fixed
object of the moderns but a new type of artefact. Architecture may have entered the age of
naturalization.2
Definitions
Etymologically, the term symbiosis comes from the Greek and means "living with." In biology, symbiosis refers
to associations between hetero specific living organisms. There are three forms of symbiotic
association:parasitism, commensalism and mutualism. It depends on whether the relationship is unidirectional or
bidirectional and beneficial, neutral or harmful. Semiologists study these interactions. On the other hand, in
the field of cognitive sciences, the 4E cognitive approach proposes a conception of cognition as an assembly of
the brain, the body and the environment. The 4E approach is based on four concepts: embodied, embedded,
enactive, and extended.
Embodied cognition is cognition produced in part by structures other than the neural system. Embodied cognition
rejects Cartesian dualism, separating the body from the mind, and includes the body in the cognitive
system.3 The body is an integral part of the cognitive system, for example, we perceive relief
thanks to the stereoscopic vision of our two eyes.
Embedded cognition is cognition coupled with the environment. Thought is not, as it were, secreted by the brain
but by the environment.⁴ The extension of the body into its environment reduces loading and relieves the
brain.⁵Enactive cognition is cognition produced in part by actions.⁶ We note here the importance of
movement in the process. Extended cognition, a cognition situated in the environment, is an externalisation of
processes in our environment. In the context of digital technologies, cyberspace would become an extension of
our brain.⁷
Hypothesis
Our hypothesis is the following: if architectural apparatus supported 4E cognitive activities, then
architecture would become symbiotic. For this purpose, inhabitant/architecture/milieu has to constitute a form
of dynamic coupling, connecting natural and artificial actors. To test our hypothesis, we present below an
architectural project example for each of the four E categories.
Embodied architecture
The Urban Algae Folly, designed by EcoLogicStudio for the EXPO Milano 2015. (1) The microalgae cultures
embedded in the architecture allow to control transparency and shading qualities of the membrane are the result
of algae growth depending on the sunlight and the presence of visitors.
In this example, not only do the algae have the dual role of sensor-actuator, but they are distributed over the
entire surface of the device. All parts of the building body are sensitive and responsive, intelligence is
distributed, embodied in every square centimeter of the material.
Embedded architecture
HygroSkin, Meteorosensitive Pavilion by Achim Menges' team is part of the permanent collection of the Frac
Centre since 2012. (2) This meteo-sensitive architecture is the result of research into materials that react to
variations in their environment. The device is inspired by the pine cone principle. The multi-layered wood
reacts to humidity, the holes open and close depending on the degree of humidity in the air. The heterogeneity
of the material allows the deformation of the surface, as each layer does not have the same expansion
coefficient. In this example, the architecture is coupled to the environment in a unidirectional link that gives
it shape.
Enactive architecture
In 2003 Kas Oosterhuis/ONL agency was invited to create an installation for the exhibition Non-Standard
Architecture. (3) The NSA Muscle is a pneumatic interactive device composed of 72 muscles programmed to be 72
independent members of a single swarm. The 72 inflatable muscles are controlled by individual valves. The device
is programmed to have its own behavioural cycle, but also to react instantaneously to an external stimulus. In
this way, ONL materialises the concept of two-way communication in real time linking two active elements, in
this case, the inflatable device and the users. The users interacting with the Muscle quickly learn how the
Muscle reacts to their actions, and a game is established in this communication.⁸ We are in a case of
bidirectional relationship, in which the movements of each of the protagonists (human and non-human) are part of
the same process.
Extended architecture
Hylozoic Ground by architect and sculptor Philip Beesley was selected to represent Canada at the 2010 Venice
Biennale in Architecture. (4) Beesley's installations are immersive environments that question the boundaries
between the natural and the artificial, the human and the non-human. These hybrid devices are micro
creatures,half environments, half mechanical, half biological. The visitor, by his presence, movement and
breathing,activates a wave of reactions from the device. He is thus inscribed in a respiratory cycle and through
his interactions becomes one with the installation within the same metabolism.
The environment supports a mutual empathic relationship that initiates a reactive movement, an exchange of
particles and an air cycle between the system and the visitor. The visitor's own body limits are questioned.
Philip Beesley named this type of behaviour: diffusive architecture.
Conclusion
The opposition between the living and the artefact is replaced by an isochronism of natural and artificial
phenomena that continues to hybridize. Symbiotic architecture could be seen as a cyber-physical-human system
within the natural artificial continuum. The opposition between the living and the artefact is replaced by an
isochronism of natural and artificial phenomena that continues to hybridize. With these four examples, we have
seen that architectural devices can support relational processes at different scales (within the material itself
or with external elements), in a unidirectional or bidirectional way, beneficial to a single actor or to
several. What the 4E approach shows is that architectural symbiosis needs to be based on a multiplicity of
interactions and actions. Borrowing the concept of the 4 E's (embodied, embedded, enactive, and extended) from
the cogniticians, we wanted to sketch a reading and writing grid for architectural design, our next research
object. It would allow gathering under one concept the different current appellations such as dynamic,
flexible,sensitive, transformable, adaptable, interactive, intelligent and moreover.
3 A. R. Damasio, L’erreur de Descartes : La raison des émotions, M. Blanc
(trad.), Editions Odile Jacob, 2010.
4 A. Prochiantz, Machine-esprit, Editions Odile Jacob, 2000.
5 P. Haselager, J. Van Dijk, I. Van Rooij (s. d.), "A Lazy Brain? Embodied Embedded Cognition and
Cognitive Neuroscience", In Handbook of Cognitive Science, 273-290.
6 E Couchot, La nature de l’art : Ce que les sciences cognitives
nousrévèlent sur le plaisir esthétique, Hermann, 2012.
7 M. Serres, Petites poucettes. Editions le Pommier, 2012.
8 V. Parlac, Surface Change : Information, Matter And Environment. In R. Stouffs &
P.Janssen... (éd.), Proceedings of the 18th International Conference on Computer-Aided
Architectural Design Research in Asia, CAADRIA, 2013.
A [Potential] Cloud War Controversies and conflicts related to climate manipulations.
Marie-JulieBourgeois
Paris-Saclay University, France mail@mariejuliebourgeois.fr
Abstract
The OuCliPo research project (Ouvroir de Climats Potentiels) is a workshop of climate potential,
supported by the Labex LaSIPS, presented by Marie-Julie Bourgeois, PhD in Aesthetics, Science and Technology of
the Arts, researcher in art and design science at University Paris-Saclay. This project is also directed by
Jules-Rémi Bois-Rouge, a (fake) climate geo-engineer, Doctor at the LaPataFlu, a (fake)
Laboratory of Fluid Pataphysics, stemming from the modern surrealist literary movements, proposes a "science of
imaginary solutions which symbolically grants to lineaments the properties of objects described by their
virtuality."1 OuCliPo studies surrealist issues to climate problems; pseudo-scientific
solutions and their implementation in the context of eco-anxiety. The project highlights the ethical and
geopolitical dimensions of solar geo-engineering, as well as the socio-cultural issues associated with these
climate experiments as techno-solutionism.
The 1995 Nobel Prize in Chemistry, Paul Crutzen, warns about the climate situation and the need to find viable
solutions in a seminar2; to the European Parliament, he proposes to seriously study solar
geoengineering as a third way.3
Solar geoengineering is a highly controversial technology that offers a concrete
and—almost—immediate solution to the current problem of global warming. Although theoretical, this
technology is being actively studied in research laboratories and by IPCC experts "Aerosol injection into the
stratosphere is the most convincing solar geoengineering technique, it has the potential to significantly affect
the climate in relation to the ongoing warming. It could offset a rise in global temperatures. It may offer a
faster cooling capacity than CO2 mitigation."⁴ This major technological innovation allows
an efficient cooling of the entire globe by increasing its natural reflective power (albedo).
This theory is based on the study of climate cooling periods following major volcanic eruptions such as the
Pinatubo eruption, which cooled the globe by 0,5°C. In 1991, the release of sulfur particles (SO2) into the
stratosphere increased the reflectivity of the Earth's surface and thus prevented some of the sun's rays from
entering the atmosphere. The temperature of the planet dropped by -0.6°C for fifteen months.⁵,
⁶
Players in geo-engineering
Solar geo-engineering solutions have developed rapidly over the last few years, with the aim of curbing global
warming as a matter of urgency. Laboratories—or startups—specialized in solar geo-engineering, Solar
Radiation Modification (SRM) or Stratospheric Aerosol Injection (SAI) attempt to "fix the climate" by generating
anthropogenic clouds composed of sulfur particles.
These promise to create a protective cloud cover against solar radiation to increase albedo. SAI was patented
in 1991, as a method of Stratospheric Seeding, it involves seeding the stratosphere with small metal oxide
particles, such as thorium, barium or aluminium flakes, with the aim of reflecting sunlight and reducing global
warming. Plans to implement the proposal were not foreseen or publicly known. In the years that have passed
since, global warming has caused major concern and scientists have been looking for solutions to "repair" the
planet. Researchers mainly use numerical modelling for obvious legal, ethical and health reasons. If inhaled,
sulfur dioxyde (SO2) may lead to chronic disease, embedding itself in our lungs. Some laboratories are
conducting risky experiments on a human scale without legal authorization in a frantic race to experiment with
climate⁷:
In 2009, the first known field experiment was carried out in Russia. The IGCE⁸ released
Sulphate aerosols from the ground and from a helicopter to measure solar radiation in the visible wavelength
range.
Between 2010 and 2014, the SPICE⁹ project aimed to assess the feasibility of injecting
particles into the stratosphere from a tethered balloon. The field experiment was canceled due to societal
opposition in the UK.
In 2011, the E-PEACE10 project was conducted off Monterey, California, where salty aerosols
were emitted from an aircraft, along with smoke and exhaust from a research vessel and a container ship.
In 2014, the SCoPEx11 laboratory began small-scale experiments.12 In 2021, this
American laboratory attempted to launch a balloon filled with SO₂ in Sweden; however, the experiment was
shut down after widespread opposition from the Saami Council, via an open letter to the Swedish government and
space corporation13.
Since 2019, the US government project SABRE14 has been studying the impact of introducing
materials into the stratosphere to influence climate and solar radiation. Since 2021, Andrew Lockley has
conducted open-air experiments. British researchers from the SA T AN project launched a high-altitude weather
balloon filled with lifting gas and a hundred grams of SO2 from European Astrotech Ltd in Aylesbury, in
England. Lockley said, "I can only confirm that our craft ascended to the heavens, as intended. I only hope
that this test plays a small part in offering mankind salvation from the hellish inferno of climate
change."1⁵
Founded in 2022, the startup Make Sunsets has gone further, launching a premature high-altitude
implementation, sending balloons with SO2 from Mexico in late 2022 without authorization or observation tools.
The Mexican government reacted by banning the company and solar geoengineering experiments.1⁶
The startup is offering to sell $10 "cooling credits" online1⁷ for releasing one gram
of particles into the stratosphere, to "offset" the warming effect of one ton of carbon, for one year. Even if
this operation is described by its creator like a (bad) joke "We joke slash not joke that this is partly a
company and partly a cult"1⁸ as Isemans says, like a radical rogue guru dressed in an orange
down jacket.
All these projects should be illegal if we respect the moratorium on geoengineering, the UN ministerial meeting
held in Nagoya, Japan, in October 2010 "Any experimentation, private or public, or adventurism aimed at
manipulating the planetary thermostat will constitute a violation of this carefully crafted consensus in the
context of the United Nations," asserted Silvia Ribeiro, ETC Group Director for Latin America.
Seducing words such as (Make Sunsets), the acronyms (SPICE, PEACE, SABRE, SATAN) and the vocabulary
employed by researchers (heaven, hell, inferno, cult...) refer to beliefs and myths. The desire to remedy global
warming by making amends for past pollution is strong. Guilt and the hope of becoming a hero by saving the
planet and its inhabitants from imminent danger guide these geo-engineers / climate sorcerers.
Costs
The cost of deploying these technologies is not excessive. Independent organisations can leverage them without
requiring participation or funding from institutions. According to studies19, 20 it is
sufficient to inject $1500 per ton of particles into the upper atmosphere each year for a gain of 2w/m2, in
order to benefit from a significant decrease of 2°C, and the entire solution would cost between 2.25 and 11
billion dollars per year. The cost seems reasonable in view of the climate issues and international interests.
Risks
The main obstacles are the ethical, causal, and governance issues21 associated with climate
manipulation, as well as the difficulty of large-scale experimentation and impact studies. Expert scientists are
unanimous on the hopes and the risks,22 the danger to the biosphere,23 security
framework,24 news biophysics risks,25 side effects, as well as possible thermal
recovery in the event of an abrupt shutdown.26 Beyond health impacts, the societal risks of deploying
these technologies are a moral issue,27 which provides a miracle "solution" that avoids reducing GHG
emissions, socio-technical lock-in; there are also considerations regarding the irreversibility of the
techniques deployed, and the termination shock.28 Furthermore, this technology, if developed
without international agreements, will cause large-scale damage across borders (GIEC 20122). As the
climate is interdependent on local meteorological events, the effects will be global. It is therefore urgent
that nations position and align themselves in this geoengineering race based on common goals, in order to avoid
wild tests, researching risk and preventing side effects. If experimentation goes unchecked, these solutions
could spark a future global war,29 a scenario that is certainly no longer a fiction given the current
geopolitical context. Since global warming is considered a threat to national security, the US military is
involved in the development of solar geoengineering as part of its militarisation
strategy.30 The prospect of rebalancing the forces belonging to competing powers such as China
or Russia is probable, and the risk of escalation is a consequence of solar counter-geo-engineering
programs31... A potential cloud war32 is brewing.
Look the sky Major volcanic eruptions
The year 536 has been called "the worst year in history"33 by archaeologists. It marks the Little
Ice Age caused by several simultaneous volcanic eruptions. These volcanic eruptions caused 18 months of darkness
on the surface of the globe due to the release of acidic sulfur particles, world temperatures then dropped by
-2.5°C, causing famines, two centuries of plague epidemics, and the fall of several civilizations on all
continents.3⁴ This period also marks the beginning of the Dark Ages within the Middle Ages. The
architect Philippe Rahm3⁵ recalls the links between architecture, climate, volcanic eruption
and epidemics: The eruption of Tambora in 1815 changed the world3⁶ which
benefited cholera. The global temperature dropped by 1°C for several years, prolonging the winter and
disrupting the climate cycle on a global scale, the “year without summer” will be
remembered.
Are we really ready to play with fire by experimenting with particle diffusion in the stratosphere? The health,
meteorological3⁷, agricultural3⁸ and civilizational risks are not well enough
measured or understood. Dangers are not limited to the survival of ecosystems and populations, not all risks
have been studied and identified by geo-engineers.
Science and fiction
That same dark and rainy summer of 1816, in this troublesome climatic context, a group of writers met in
Switzerland; Mary Shelley wrote Frankenstein or the Modern Prometheus3⁹ and Lord
Byron popularised The Vampyre.⁴⁰ We can say that one of the founding myths of
Science Fiction was born in a dramatic climatic context! The following year, in 1817, in the midst of global
climate change, William Turner began to paint coloured sunsets, amongst other Impressionist precursors
influenced by these surrealist red-to-green skies.⁴1 Edvard Munch painted The Scream
after viewing and being influenced by the strange sky "turning blood red."This famous
expressionist portrait and its red sky probably stems from a vision of the Oslo Fjord, after the volcano
eruption of Krakatoa in 1883,⁴2 or may be due to a nacreous cloud.⁴3 Climate
and fiction seem intimately linked in the narrative imagination but also in the psychology of aesthetics and
creative expression. If SRM/SAI technologies are implemented, and the atmosphere becomes opaque and dangerous to
inhale due to particles, will the atmosphere lose its transparency? Our “common sky” will be
whiter.⁴⁴ Today’s blue skies are already an illusion due to Tyndall effect and
Rayleigh scattering and it is based on the size and density of particles... If the atmosphere becomes
concentrated in CO2, will the sky turn orange like the CO2 saturated Martian sky? If our vision becomes
saturated or obstructed by particles, how can we project ourselves towards new horizons, new common futures?
We can conclude that the power of design influences the critical thinking process, whilst the urgency of the
situation also blinds the mind. The Make Sunsets website is far too well-designed for a typical
scientific research project, and its warm orange design (the colour of sunsets) is as seductive as the promises
of rescue it makes. Perhaps the design and aspirations explain its success and ability to raise funds. Despite
the cobbled-together nature of its technology and the danger of its reckless actions. However, these orange-red
colours of the startup should worry us, just as the Scream 😱 reveals the anxiety-inducing
dimension of modern existence.
Realistic satyr and fiction
OuCliPo is an art and design research project that questions the impact of particles on our vision and
our cultures at the scale of landscape, architecture and habitat, by presenting speculative fiction based on
climate manipulation, studies and data.
Nubus⁴⁵ is a (fake) research startup, created to test the credibility of the
technology and also the attractiveness to the general public via a seducing marketing campaign "for a bluer and
a cooler sky." For this purpose, a questionnaire was used to measure support for the Nubus startup and
its promises. The French (fake) startup projects us to the stage of commercialization and industrialization of
SRM/SAI technologies in order to confront us with the mutations they will generate via environmental and society
disturbances. How far are we willing to go, to believe it?
After viewing the Nubus promotional film, a sample of 34 people were interviewed during the exhibition
of the project at the ENS Paris-Saclay and at Evry University:
62% want to help optimize the climate
41% are ready to install a free of charge Nubus module at home
41% are in favour of a cooler climate
29% want a “bluer sky”
In conclusion, a relatively high percentage was inclined to participate in climate optimization in order to
promote cooler and bluer skies. Although a minority, 21% of the respondents, believes that geo-engineering and
SRM/SAI do not seem dangerous for our biosphere, or the climate... When the video was recently shown in an
innovative art and science context, audiences reacted with ambivalence: amused, horrified or seduced, many
believed in it. We decided to exaggerate the tragi-comic potential of the startup’s arguments to defuse
support for the promise of rescue, the urgency of the situation resulted in a strong desire for
solutionism.
Nubusis also developing green kerosene and biofuels to generate
clouds of particles with high nucleation potential, rather than harmful greenhouse gases. Our emissions are
guaranteed to have high clouding potential. We also work with the agrochemical and agricultural industries to
combat soil erosion and desiccation. Our mineral and nutrient stimulants have a high precipitation potential
which is ideal for farmers in times of drought.
"Our partners in the tobacco industry offer electronic cigarettes that contribute on a small scale to the
generation of climate-friendly particle clouds, because there are no small contributions.
Nubus is launching a fund-raising campaign to "save" the planet and offers citizens in search of
meaning the chance to clear their consciences by blue washing our vision of the future.
You can also take a sodium cure with our blue pills, which guarantee accelerated awareness to wipe out
ecological procrastination..." (3)
Even with these arguments, the public still wondered if Nubus really did exist...
Homogenitus are clouds generated by human activities defined by the WMO in the international cloud
atlas.⁴⁶Homogenitus⁴⁷ it is also the name of an
artistic cloud-making machine. Players can generate customizable clouds directly on a web application, choosing
morphology, size, shape and the composition of particles. This artistic installation questions the creation of
clouds by human activity, the ability to control the weather and places the user in a demiurgic position.
The citizen artivist collective fakeCloud⁴⁸ denounced Nubus
activities, it’s SRM/SAI projects and climate manipulations. Nubus is caused of
over-producing clouds related to the Iran accusation against Europe to steal their cloud in
2011.⁴⁹
Research of solutions
In the absence of government control or international rules, companies can deliberately spread or steal clouds.
How will our sky look in a decade? Will a future cloud war occur? This awareness campaign proposed to
approach the ecological problems by addressing these issues through wordplay, irony, detours, fiction, and
references to scientific methodologies and technologies. Designing our potential future atmosphere opacity and
tint aimed to sensitize the public to these rapid ecological solutions, and promote awareness of the seductive
dimension of design.
Our ideologies, our beliefs, our knowledge are challenged by the crises we are experiencing; climate, health,
life, economy, institutions and especially science. At the same time when technologies and social networks allow
collaborative actions, we question the methods of sensitization of the public about the techno-solutionism
challenges. Fictions and post-truths mingle with reality in the scientific field. How can we find a sustainable
way in this mess? Awareness and collective action are the determining factors for reducing greenhouse gases, but
solar geo-engineering projects risk slowing down these efforts by proposing alternative and risky paths.
Climate issues are frequently addressed in a dramatic way, generating various typologies of
reactions⁵⁰: alarmed, worried, cautious, non-committal, unconvinced, and
contradictory. On an individual level, ecological procrastination⁵ is mainly due to
discourses⁵2 that delay massive climate action; techno-solutionism is one of them. Robert
Gifford observes seven psychological barriers to ecological inaction⁵3: limitations of
cognition, ideological views, comparison with significant others, costs, distrust of experts and authority,
perception of risk, and behavioural limitations. For Gifford, these"dragons of inaction"are the social-psychological brakes that impede real and meaningful global action.
Conclusion
OuCliPo proposes to approach climate threats with non-guilty and non-anxious modes of communication
such as fiction and humour. OuCliPo does this by making a mockery of the search for technological
solutions. Between solutions that are “not solutions,” where the cure is worse than the disease,
denial and even counter-action, massive actions, whether positive or negative, have a strong impact on the
climate.
To this end, Nubus offers its customers the chance to redeem their consciences by fixing the planet.
It provides citizens in search of meaning with a conscience cleanser, by washing our vision of the future in
blue for a dual benefit: a thermal solution and the psychological comfort of being able to remedy climate change
by looking at the sky, not to don’t look up! (4). Ideologies, design and the power
of capitalism helps us to believe in these seductive solutions. The market in climate
denial⁵⁴ and the negation of mitigation policy are solutions that benefit climate
skepticism.
Media and digital tools are good ways of approaching social issues with critical distance and speculative
projection, but they can also fuel disbeliefs and conspiracies. Moreover, fiction can play a major role in
raising collective awareness of issues at stake, by accelerating political decision-making. Confronted with the
crises we are experiencing, and the eco-anxieties that are paralysing actions, we need to set off in search of
"eco-quietude" and harmony amongst the disorder. This quasi-spiritual philosophical notion organised the
celestial spheres as early as Pythagoras; between music, numbers, colour and astronomy, harmony was already
giving rhythm to the mental and physical health of mankind in antiquity.⁵⁵
The harmony and symbiosis between man and nature, with sustainability and critical distance, would allow us,
through fiction and humour, to approach climate problems from a truthful background.
(3) ISEA 28th International Symposium on Electronic Art, May 17, 2023, Paris forum des halles
(4)Don’t look up ! Adam McKay 2021
References
Books
1 Alfred Jarry, Gestes et opinions du docteur Faustroll, pataphysicien, Livre II,
chapitre VIII, Paris, Éditions Fasquelle, 1911.
35 Philippe Rahm, Histoire naturelle de l’architecture, Comment le climat, les
épidémies et l'énergie ont façonné la ville et les bâtiments,
Paris, Edition du Pavillon de l’Arsenal, 2020, p.167.
36 Gillen D'Arcy Wood, Tambora – The Eruption That Changed the World,
Princeton University Press, 2015.
39 Mary Shelley, Frankenstein or the Modern Prometheus, Lackington, Hughes, Harding,
Mavor & Jones, 1818.
40 John William Polidori, The Vampyre a Tale, The New Monthly Magazine, 1819.
55 Yuval Noah Harari, Homo Deus, A brief history of tomorrow, London, Vintage, 2017.
Journal article (online)
3 P. J. Crutzen, “Albedo enhancement by stratospheric sulfur
injections,” Climatic Change, vol. 77, no 3-4, July 25, 2006, 211-219,
doi:10.1007/s10584-006-9101-y
5 Mc Cormick and al, “Atmospheric effects of the Mt Pinatubo eruption,” Nature 373,
February 2, 1995, 399–404. doi:10.1038/373399a0
12 J. A. Dykema and al. “Stratospheric controlled perturbation experiment: a small-scale
experiment to improve understanding of the risks of solar geoengineering,” Phil. Trans. R. Soc,
December 28, 2014, doi:10.1098/rsta.2014.0059
19 W. Smith, G. Wagner, “Stratospheric aerosol injection tactics and costs in the first 15
years of deployment,” Environmental Research Letters, Vol. 13, No. 12, November 23, 2018,
doi:10.1088/1748-9326/aae98d
20 W. Smith and al, “A subpolar-focused stratospheric aerosol injection deployment
scenario,” Environmental Research Communications, Vol 4 number 9, September 15, 2022,
doi:10.1088/2515-7620/ac8cd3
22 M. De Guglielmo Weber, “Promesses et risques de la géoingénierie
solaire,” IRIS, Programme climat, énergie & sécurité, February 2023, accessed
August 22, 2023, https://www.iris-france.org/wp-content/uploads/2023/02/19_ProgClimEnerSec.pdf
23 M.G. Lawrence, and al, “Evaluating climate geoengineering proposals in the context of the
Paris Agreement temperature goals,” Nature Communications 9, Septembre 13, 2018, 3734,
doi:10.1038/s41467-018-05938-3
24 Chalecki, and al, “A New Security Framework for Geoengineering,” Strategic
Studies Quarterly, vol. 12, no. 2, 2018, JSTOR, Accessed 22 Aug. 2023, 82–106,
http://www.jstor.org/stable/26430817
25 E. Baughman, et al, “Investigation of the Surface and Circulation Impacts of
Cloud-Brightening Geoengineering,” J. Climate, 25, November 1, 2012, 7527–7543,
doi:10.1175/JCLID-11-00282.1
27 C Preston, “Ethics and geoengineering: reviewing the moral issues raised by solar
radiation management and carbon dioxide removal.” WIREs Climate Change. 4, November 8, 2012,
23-37, doi:10.1002/wcc.198
28 A. Parker, & P. J. Irvine, “The Risk of Termination Shock From Solar
Geoengineering,” Earth's Future, 6, March 11, 2018, 456–467, doi:10.1002/2017EF000735
30 Stephens, and al, “Towards dangerous US unilateralism on solar geoengineering,”
Environmental Politics, December 18, 2022, doi:10.1080/09644016.2022.2156182
41 C. Zerefos, end al, “Further evidence of important environmental information content in
red-to-green ratios as depicted in paintings by great masters,” Vol. 4 issu 6, Atmos. Chem. Phys.
14, March 25, 2014, 2987–3015, doi:10.5194/acp-14-2987-2014
42 Donald W. Olson, and al, "The Blood-Red Sky of the Scream," APS News. American
Physical Society, Vol. 13 number 5, May 2004, accessed August 22, 2023,
https://www.aps.org/publications/apsnews/200405/backpage.cfm
43 F. Prata, and al, “The Sky in Edvard Munch’s The Scream,” Bulletin of the
American Meteorological Society 99, 7, July 1, 2018, doi:10.1175/BAMS-D-17-0144.1
44 B. Kravitz, and al, “Geoengineering: Whiter skies?”, Geophys. Res. Lett.,
39, L11801, June 1, 2012, doi:10.1029/2012GL051652
50 E. W. Maibach, and al, “Identifying Like-Minded Audiences for Global Warming Public
Engagement Campaigns: an AudienceSegmentation Analysis and Tool Development,” PLoS ONE, March,
10, 2011, doi: 10.1371/journal.pone.0017571
51 Richard W. Malotte, “I’ll Save the World from Global Warming—Tomorrow: Using
Procrastination Management to Combat Global," June 1, 2017, doi:10.1007/BF03392214
52 WF Lamb, and al, “Discourses of climate delay,” Global Sustainability 3, e17, June
8, 2020, 1–5, doi:10.1017/sus.2020.13
53 Robert Gifford, “The Dragons of Inaction: Psychological Barriers That Limit Climate
Change Mitigation and Adaptation,” American Psychologist, May 2011, doi:10.1037/a0023566
Magazines and Newspapers (online)
2 European Parliament, “Professor Paul Crutzen: Nobel winner and advocate of a climate
"escape route"”, January 12, 2011, accessed August 22, 2023,
https://www.europarl.europa.eu/news/en/headlines/society/20110103STO11194
6 NASA Observatory, Langley Research Center, Aerosol Research Branch, June 2001, accessed August
22, 2023, https://earthobservatory.nasa.gov/images/1510/global-effects-of-mount-pinatubo
13 Letter from Sam Council to SCoPEx advisory committee, February 24, 2021, accessed August 22,
2023,
https://static1.squarespace.com/static/5dfb35a66f00d54ab0729b75/t/603e2167a9c0b96ffb027c8d/1614684519754/
15 Chris Hatch, “Whose call to shade the Earth?” March, 3, 2023, accessed August 22,
2023, https://www.nationalobserver.com/newsletters/zero-carbon/2023/03/03/whose-call-shade-earth
16 Gobierno de Mexico, “La experimentación con geoingeniería solar no será
permitida en México” January 13, 2023, accessed August 22, 2023,
https://www.gob.mx/semarnat/prensa/la-experimentacion-con-geoingenieria-solar-no-sera-permitida-en-mexico
18 James Temple, “A startup says it’s begun releasing particles into the atmosphere,
in an effort to tweak the climate.” MIT Technology Review, December 24, 2022, accessed
August 22, 2023, https://www.technologyreview.com/2022/12/24/1066041
29 Alexander C. Kaufman, “The Climate Change Solution That Could Spark Global War”
OneZero, June 16, 2019, accessed August 22, 2023,
https://onezero.medium.com/the-climate-change-solution-that-could-spark-global-war-bd1cc9cce1a3
49 Barney Henderson, “Mahmoud Ahmadinejad says Europe 'stealing Iran's
rain’”, The telegraph, May 21, 2011, accessed August 22, 2023,
https://www.telegraph.co.uk/news/worldnews/middleeast/iran/8527455/Mahmoud-Ahmadinejad-says-Europe-stealing-Irans-rain.html
32 “La guerre des nuages, par Mathieu Simonet,” Nouvel Obs, March 28, 2022,
accessed August 22, 2023,
https://www.nouvelobs.com/tribunes/20220328.OBS56297/la-guerre-des-nuages-par-mathieu-simonet.html
54 Marine de Guglielmo Weber, “Crédits de refroidissement : quand la
géo-ingénierie commercialise le déni climatique,” IRIS France, February 17,
2023, accessed August 22, 2023,
https://www.iris-france.org/173813-credits-de-refroidissement-quand-la-geo-ingenierie-commercialise-le-deni-climatique/
Websites
4IPCC, AR5, SYR, 3.3, 2014, accessed August 22, 2023,
https://www.ipcc.ch/report/ar5/syr/
7 Interactive world map on geoengineering, accessed August 22, 2023,
https://map.geoengineeringmonitor.org/srm
8 IGCE, Institute of Global Climate and Ecology accessed August 22, 2023, www.igce.ru
9 SPICE, Stratospheric Particle Injection for Climate Engineering accessed August 22, 2023,
www.spice.ac.uk
14 SABRE, Stratospheric Aerosol processes Budget and Radiative
Effects,https://csl.noaa.gov/projects/sabre/
17 Make Sunsets American startup of solar geo-engineering, accessed August 22, 2023,
https://makesunsets.com/
21 Reynolds, J. et al. “Solar Radiation Modification: Governance gaps and challenges –
Summary.” Carnegie Climate Governance Initiative (C2G), March 2022, New York, www.c2g2.net
26 Felgenhauer, T. et al. “Solar Radiation Modification: A Risk-Risk Analysis - Summary,
Carnegie Climate Governance Initiative” (C2G), February 2022, New York, 2023, www.c2g2.net
45 Nubus is a (fake) French startup of solar geo-engineering, accessed August 22,
2023, https://www.nubus.fr/
46 World Meteorological Organization, International Cloud Atlas,
https://cloudatlas.wmo.int/en/homogenitus.html
47 Homogenitus is an art installation, https://homogenitus.fr/
48 Fake cloud Collective of artivist fighting against climate manipulations, accessed
August 22, 2023, http://fakecloud.fr
Author Biography
The author is a digital artist, designer and PhD researcher in aesthetics and science technologies of the
arts since 2018. She works on ecological, artistic and scientific fictions in society in relation to human
activities. Born in Paris in 1981, she lives and works in Cachan, France. In 2008, she obtained a Master's
degree in New Media at the Ecole Nationale Supérieure de Création Industrielle (ENSCI) and in
2009, she joined the EnsadLab research program. Her research focuses on "solar fictions, devices that
recreate the sun’s path", and she is currently working on climate fictions. She teaches at the
University of Evry (91) since 2013 and co-founded the CondéDesignLab in 2021.
Inhabiting the Edges
CéliaBoutilier
SACRe-PSL ENS/ENSBA — Paris Boutiliercelia@hotmail.fr
Abstract
My doctoral project in art-creation-research, which began in October 2021, is entitled "Imaginal
photography: for an aesthetic of symbiosis." In my practice as a photographic artist, making symbiotic
relationships, unexpected alliances, allows me to provoke a "suspension of evidence" which, through the
intermediary of the image, invites the viewer to share the experience of a different look. From this point on,
the very nature of perception is to admit ambiguity. The interdependencies in ecosystems, the principle of
hybridization between different species are themes addressed by the team of the laboratory of plant symbioses of
the French National Museum of Natural History of Paris with which I collaborate. In my research, I'm
specifically interested in the symbioses present in the orchid with the biologists Florent Martos and Eve
Hellequin. Transcending the notion of organism allows us to reconcile human beings with their natural dimension
of interdependence and interaction. The researchers in this laboratory are the extractors and curators of
natural plant resources, with which they shape new worlds such as the Grande Serre: a space in which I would
like to exhibit works in 2023-2024. These hybrid spaces, co-dependent with humans, seem to me to be fertile
places to dialogue with.
"We inhabit the world from the images we make of it."
This doctoral project starts from the intuition that aesthetics, as a study of sensible knowledge and its
artistic productions, is one of the main foundations of the imagination that allows us to think the world and to
inhabit it. It is from this imagination that imaginative practice is born. Without the imaginative faculty, a
whole area of reality is definitely off-limits to us, lost forever. With the loss of the imaginative faculty,
what we risk is not the loss of fiction, but the loss of reality. We run the risk of compartmentalizing the
sensible and intelligible worlds, of killing them off slowly, precisely because they only come to life and "come
alive" when they are in contact with each other.
Methodology
It seems to me today that the question of representation (of whom speaks and how he or she expresses himself or
herself) actually raises the question of the imaginary, that is to say, that of the fabrication of new
fictionalities. The issue of imagination is crucial, because it opens up perspectives of therapeutic narratives.
These proposals for circulation (in the making itself or in the setting of creations) are multiple hypotheses of
resilience. This is why certain fictionalities can be productive of the real and allow us, by this means, to
increase ourselves. Their power of augmentation is not only intimate, a creation has an impact when it goes
beyond its subject and manages to make language for something else. It’s therefore a question of
photographing this place with the desire to feel it fully, while knowing that from this precise place we will
touch other places, both terrestrial and human.
On the other hand, I’m interested in the interdependencies in ecosystems and the principle of
hybridization between species, because I believe that there is a realaesthetic, philosophical and political
stake in linking Art (culture in general) to Ecology. The laboratory and the studio are the two places where
photographic art was born. Botanical science and in particular the concept of symbiosis seems to me to be able
to achieve this necessary alliance (in biology the word symbiosis defines a lasting interspecific interaction of
mutual benefit). In this sense, I’m conducting this doctoral research project in collaboration with the
team of the plant symbiosis laboratory of the Paris Natural History Museum. I’m specifically interested in
the mycorrhizal symbioses of orchids: those that create networks between individuals and different species,
notably between fungi and plant roots. From the concept of symbiosis in biology, I intend to develop the concept
of symbiosis in the field of aesthetics: "what could be a symbiotic aesthetic? I propose the hypothesis that a
"symbiotic aesthetic" constitutes, on the one hand, a singular way of relating philosophically to the world (at
the source of an enlarged perception), and, on the other hand, a programme of plastic realisation. How can
symbiosis suspend the rational relationship of things to each other? To what extent does this suspension make it
possible to escape from designation and nomination? How can the new representation of the world proposed by
symbiosis change the way we act and interact? This project aims to sketch some answers, both formal and textual,
based on fieldwork with biologists, both in the field and in the laboratories of the MNHN of Paris.
Communication plan
More specifically, I'd like to present a few recent works produced following a scientific mission to Reunion
Island, which fall under what I call a "symbiotic aesthetic." Photographic assemblages are the first
satisfactory way I've found of aesthetically translating hybrid identity, through which the intertwining of
links manifests itself. Boundary images between the "natural," the imaginary and the technological. Different
photographic fragments are linked together by collage. The aesthetic is organic and lively, evoking both art
nouveau and virtual reality, where naturalistic photography meets pictorial inventiveness.
Focus of the work
In my work, it’s a question of "milieu" in that it takes us in its mesh.It’s a question of
resisting any hierarchical placement of knowledge and exploring the plurality of deployments that they allow,
replacing the aim of unity with the problematic of articulation.
I would like my images to be edges. Let them be meeting zones, spaces of contact, symbiotic and twilight places
where the obvious is suspended and where, in a strange familiarity,things enter into relationship. I favour the
observation of transitory and symbiotic forms,changes of state, the confusion of scales (from satellite to
microscope) and all other forms of telescoping practices, insofar as they impact the solidity of bodies, the
sharpness of contours and the fixity of images. The particular is then the bearer of more than just itself, in
that the image is made from an elsewhere that crosses the here, in order to give an account of a here that is
both always open and always worried.
"‘Border identity’ this is the term by which I usually define my own identity. I call it frontier,
anchored, not in a place of rupture, but, on the contrary, in a space of permanent coming together. The border,
as I define it and inhabit it, is the place where worlds touch, tirelessly. It's the place of constant
oscillation: from one space to another, from one sensibility to another, from one worldview to another. It's
where languages mingle, not necessarily thunderously, but naturally impregnating each other, to produce, on the
blank page, the representation of a composite, hybrid universe."¹
Pace of time and ghosts
We inhabit the world through the images we make of it. In this sense, it would seem that constructing ways of
inhabiting the world² also involves ways of making images. In the hollow of my plastic reflections, I apply
myself to modelling a specific temporality: that of a time which has a thickness and in which memory and dreams
are sedimented.
This time—like thought—is not linear (cause/effect), it is made up of feedback loops where the
chain of causality swallows its own tail (systemic thinking). The past is not only what happened, it is also
what was dreamed.
"There is always a shadow of something. And it stays for a long time. Now there is only the shadow of my
mother, who passed on to me the shadow of my grandmother, etc., etc. You see images projected normally and then
you see the same images that become a shadow ofthemselves. That's what Maniac Shadows is all about."³
Uncomfortable landscapes. There is a flaw, a break, a suspension of evidence. We must first remember our
heritage, the memory of the rumbling soil, the organisms and ghosts that inhabit us and that we carry, in spite
of ourselves, like amnesiac children. This requires, first of all, a refusal of postures or practices that are
constituted in a cannibalistic relationship to otherness. This relationship is one of attention, concern and
vigilance. The state of vigilance is a state of slowness, of contemplation that stretches over time. It is a
profound attention that takes the opposite view of the lighted time of immediacy to become a space of common
sharing. It is a care that we cultivate in the relationship that we maintain with the other whether human or
non-human. Taking care of our relationships and their modes of existence goes hand in hand with taking care of
the making of our images." The world to which we belong is first and foremost the world we carry within
us."⁴
References
Books
1 Léonora Miano, "HABITER LA FRONTIÈRE, Paysages francophones -Journée
internationale de la francophonie, Université de Copenhague (Danemark)" conférence de 2009
recueillie dans HABITER LA FRONTIÈRE, L’Arche, 2012, p.25.
2 Bruno Latour, Où atterrir ? - Comment s'orienter en politique, La
Découverte, 2017.
4 Ibid. Léonora Miano, 2012, p.25.
Webvideo interview
3 Chantal Akerman - ManiacShadows, Nuit
Blanche 2013 au Théâtre du Châtelet, Paris.
Bibliography
David Abram, The Spell of the Sensuous: Perception and Languagein a More-Than-Human World,
Vintage, 1997.
Jean-Christophe Bailly, L’imagement, Seuil, 2020.
Jean-Christophe Bailly, Le temps fixé, Bayard, 2009.
Jonathan Crary, Techniques de l’observateur, Vision et modernité au XIXe
siècle, Broché, 1990.
Lorraine Daston, Peter Galison, Objectivité, Les presses du réel, 2012.
Gilles Deuleuze, Logique de la Sensation, 1981.
Gilles Deuleuze, Desert Islands and Other Texts, 1953-1974.
Cambridge, MA: MIT Press, 2004.
John Dewey, L’art comme expérience, 1934.
T. S. Eliot, The Waste Land, Faber & Faber, 2015 [1922].
Cynthia Fleury, "L’imagination renaissante : entreRévélation et Intellection"
in Imagination, imaginaire,imaginal, Coordinated by Cynthia Fleury, PUF, Paris, 2016.
Donna Haraway, Staying with the Trouble: Making Kin in theChthulucene, Duke University
Press, Durham, 2016.
Donna Haraway, When species meet, University of Minnesota Press, Minneapolis, 2007.
William James, Philosophie de l'expérience. Un universpluraliste, 2007.
Bruno Latour, Down to Earth: Politics in the New ClimaticRegime, ed. Cambridge, UK,
Polity Press, 2018.
Bruno Latour, Facing Gaia: Six Lectures on the Political Theologyof Nature, Cambridge, UK, Polity
Press, 2013.
Bruno Latour, Enquête sur les modes d’existence une anthropologie des modernes, La
Découverte, 2012.
Bruno Latour, La science en action - Introduction à la sociologie des sciences, La
Découverte, 2005.
Michael Marder, Plant-Thinking – A Philosophy of VegetalLife, Columbia University Press,
2013.
Michael Marder, La plante du philosophe - Un herbierintellectuel, Mimesis, 2020.
Lynn Margulis, Symbiogenetics - Origin of MitoticCells from Bacterial Communities in the
ProterozoicEon, 2011 [1992].
Lynn Margulis, Symbiosis as a Source of EvolutionaryInnovation: Speciation and Morphogenesis,
(TheMIT Press, 1991).
Giuseppe Penone, Jean-Christophe Bailly, Sève et pensée, Paris, broché, Paris
2021.
Bernard Réquichot, "Métaplastique" dans Écrits, 1955, p.87.
Simon Schaffer, Steven Shapin, Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life,
1985.
Anne Simon, Une bête entre les lignes. Essai dezoopoétique, Marseille, Wildproject,
2021.
Isabelle Stengers, La sorcellerie capitaliste : Pratiques de désenvoûtement, La
Découverte, 2013.
Isabelle Stengers, L’invention des sciences modernes, Flammarion, 1993.
Andreï Tarkovski, Le Temps scellé.
Anna Lowenhaupt Tsing, The Mushroom at the End ofthe World: On the Possibility of Life
inCapitalist Ruins, Old Saybrook, CT, TantorMedia, 2017.
Alfred North, Whitehead, Le concept de nature, J.Vrin, 2006.
Walt Whitman, Leave of grass The Original 1855, Edition, Paris, Broché, 2020.
Journal articles (print)
Carla Hustak, Natasha Myers, "InvolutionaryMomentum: Affective Ecologies and the Sciences ofPlant/Insect
Encounters", Vol. 23, Num. 3, Brown University and differences, A Journal of Feminist Cultural Studies, 2012.
Sabine Plaud, "Vie du langage, vie des images : une marque de continuité dans la philosophie de Ludwig
Wittgenstein".
Katrin Solhdju, "L’expérience "pure" et l’âme des plantes. James lecteur de Fechner",
2007.
Isabelle Stengers, Didier Debaise, L’insistance des possibles. Pour un pragmatisme
spéculatif, Multitudes 2016/4, n° 65, 82-89
Webvideo interview
Bruno Latour, “Agency at the Time of theAnthropocene”, New Literary History 45, 2014, 1-18.
Author Biography
After a DNAP at ENSA Dijon (2016), a year at ERG Brussels (2017), a collaboration with LadHyX Polytechnique
(2014-2019), Célia Boutilier joins the Beaux-Arts de Paris (2017) to defend a DNSAP (2019). Since 2014
she has been collaborating with research laboratories (microfluidics - Polytechnique, and mycorrhizal
symbioses - National Museum of Natural History) where she is interestedin how imaging techniques participate
in theelaboration of knowledge. Since 2016, herwork has been exhibited at fairs and international events,
including the National Gallery in Copenhagen, the Cité Internationale des Arts in Paris and the Maison
des Métallos. It has been quoted inseveral scientific journals, including Physics Today,
The NewYork Times, Le Monde. Since 2020 it has also been included inprivate collections. In
October 2021 Célia Boutilier starts a PhD in "Sciences, Arts,Creation, Research" (SACRe-PSL / ENS) at
the National Superior School of Fine Arts in Paris (ENSBA).
Mytherrella: an interactive installation hallucinating mythological auroral formations
DorinCucicov
University of Bucharest, Romania Bucharest, Romania cucicov@gmail.com
Abstract
Up until very recently, mythological tales predominated in the explanation of the polar auroras. Today, we
understand that the polar lights are a phenomenon brought on by the solar winds' interaction with the
magnetosphere of the Earth leading to precipitation of energetic particles on the topside ionised atmosphere.
However, for a long time, this phenomenon inspired vibrant trans-border stories with a profound impact on local
communities. In our interactive installation, Mytherrella, we aimed to create a dynamic environment in which
scientific data and mythological storytelling unearth new imaginaries about the aurora borealis. The generative
video integrated in the installation uses a custom real-time StyleGAN algorithm that continuously samples from a
model trained on a large set of all-sky auroral images acquired in Kiruna, Sweden. This method enables live
interaction with the generated video, producing novel synthetic auroral formations that are unpredictable while
remaining within the bounds of the learned features. By combining the dataset with a relatively small number of
alternative-style images, the diversity of the generated content is increased, creating a divergent effect that
reinforces the mythological narrative. We share the technique of interactive video generation as well as the
research process behind the creation of the work.
Keywords
interactive installation, mythology, live StyleGAN, aurora borealis, polar lights.
Myths about the aurora borealis are prevalent in the cultural heritage of populations from the polar regions of
the Earth. Surprisingly, these stories have not permeated pop culture in the same way the images of the aurora
borealis did. Many of the stories, despite being from different continents, share many similarities: contact
with the spirit world, vengeance of slain enemies, communication with the dead through whistling or playing ball
games in the sky. For some, the appearance of the aurora borealis is a bad omen; for others, it can be bad only
if disrespected by whistling at it or staying too long outside. Others have more poetic explanations for auroral
formations.5, 6
This paper presents a virtual environment, the result of a creative exercise that leverages both scientific
data and stories from various communities about the polar lights.
Combining them produces novel visions of a future in which humans reinvent themselves through collaboration
with artificial artifacts. Mythology has been at the forefront of explaining the unknown and stimulating human
curiosity for centuries. These stories are refined representations of humanity's deep connection to the natural
world around it. The mystical characters in these tales instilled fear and caution in humans, effectively
negotiating a reverence for natural phenomena. In the past centuries, science has solved numerous natural
enigmas, relieving some anxiety but also making humans less respectful and more entitled.3
Our project, named Mytherrella, addresses the subject of symbiosis on multiple levels. It frames a creative
experiment by integrating empirical data and mythological narratives. The equal treatment of science and myths
aims for a reevaluation of our relationship with natural phenomena. Through the similarities between the stories
of various cultures, cultural symbiosis is highlighted on a global scale, diminishing the significance of
political borders.
Concept and construction
Mytherrella is an endless process of imagination informed by facts from science and driven by myths surrounding
the aurora borealis. Starting from a dataset of all-sky images showing auroral activity recorded by the Swedish
Kiruna auroral observatory, it imagines new auroral formations. It provides control over its hallucinations in
an interactive framework. Both human and machine are in search of the lost/improbable auroras as seen by the
first people that described them.
Mythology of the aurora
One of the most common mythological symbols associated with the polar lights is their connection to the
spiritual world. Polar lights, according to the Greenlandic Inuit, were messages from their deceased friends and
relatives.1; The Meskwaki, a Native American people with homelands in the Great Lakes region, held similar
beliefs. They believed that by whistling to the lights, they could interact with them, thus communicating with
the dead. Some sources indicate that some North American Inuit also used whispering during auroral events to
send messages to the dead.⁵ More active manifestations of the aurora borealis were associated with spirits
playing games in some Greenlandic stories, while on the European continent, in Scotland, Ireland and Norway they
were associated with dancing spirits.1, 4
Some of the more poetic interpretations of the polar lights are found in Sámi peoples, residing in the
north of Finland. They named the aurora - revontulet (fire fox). It is said that the lights are created by a fox
with glittering fur running across the mountain area. 1; In other stories, polar lights are an omen of disaster
and war. These are found in Tlingit indigenous peoples of the Pacific Northwest Coast of North America, Skolt
Sámi in Northern Europe and even in some Chinese and Roman literature.1, 2
There may be several myths about the aurora in a single area, ranging from optimistic interpretations to
terrifying ones. These are dependent on particular historical moments and the social and natural influences that
shaped those times.
Building Mytherrella
The physical construction of the Mytherrella installation was inspired by the early designs of the auroral
observatory in Yerkes, Wisconsin, United States.4 These small sites, fitted with all-sky cameras,
collected images of the auroral events aimed for scientific investigation. In our installation, an immersive
context is created inside a small booth where one can experience and interact with the story in a private
setting. The outside of the booth is painted with selected mythological texts, which acts as a representative of
the old media in contrast with the interactive digital content. An interactive stand inside the booth provides
two analog potentiometers for controlling latitude and longitude on a minimalist digital map. Using these
knobs, a specific location can be identified on the global map. The map is centered on the poles and only
depicts the boundaries between water and land; it does not include any other information, such as names of
nations, borders, seas, or rivers. This choice was made in order to disrupt the influence of the political
borders in the collective perception and suggest a new perspective on interpreting the global map.
Ten locations are hidden on the digital map. They can be discovered by identifying their approximate location
using the navigation controls. These coordinates are linked to locations from which we gathered mythological
stories. Based on these texts, we used text-to-image tool Midjourney (1) to generate unique images. Figure 2
contains one of these images generated based on the mythological texts of the Native Americans describing the
aurora borealis as a means of communication with the spirit world.
The second screen, positioned in front of the control stand, displays a continuously generated video with
auroral formations. This video is created using a StyleGAN model that was trained on a mixed dataset of
scientific and artistic images of the aurora borealis.8 The main corpus consists of 9834 selected
images from a dataset of all-sky images depicting auroral activity provided by the Kiruna Atmospheric and
Geophysical Observatory at the Swedish Institute of Space Physics. All images have a standard round format, as
seen in Figure 3.
(1). Midjourney. https://midjourney.com
Data blending was applied using another 150 artistic images of the aurora borealis in order to extend the
dataset and introduce visual features that break the rigor of the scientific representation. The process of
projecting an image into the latent space produced images that were continuously served to create the video.
When idle, the video would display auroral formations as learned from the main corpus of all-sky images. Finding
an image on the world map while interacting with the installation would begin projecting that image to the
model, disrupting the normal representation by also sampling from the artistic images' features.
Figure 4 describes the approach of continuously serving images for the video projection. A script monitors for
new files that are selected by interacting with the installation. Once a file is detected, it signifies that an
image was identified on the map. This triggers an immediate process of projecting that image to the model. The
resulting images are displayed sequentially creating the effect of video continuity. The 3 processes of the
architecture are independent: generating based on the source image; serving an image for generation; displaying
received images. They are linked only by the configuration of the listeners. This loose coupling makes it simple
to integrate with other systems to serve various seed images or display generated images in various formats.
Discussion
The myths of the aurora borealis are rich and creative stories that have surrounded this natural event for a
long time. Mytherrella is a direct comment on the relationship between science and mythology. Figure 5
exemplifies one sequence of a video generated by projecting the image from Figure 3. It starts from the format
of an all-sky image and navigates the latent space in search of the features from the projected image. By using
the technique of interactive generative video, we sought to create a subtle transition from visuals that have a
scientific aesthetic to a more abstract look that resembles some of the features of the mythological image while
also preserving the features of the aurora. This type of interaction contributed to the immersive character of
the experience by partially delegating the responsibility of the generated image to the person interacting with
it.
Some of the visitors discovered that by rapidly shifting through mythological images on the map, they could
further distort the visual representation of the aurora, taking more control over the generative process. This
encouraged visitors to spend more time with the work and explore both the map and the generative mechanism as
they discovered more layers of interaction. By doing so, they also delved deeper into the mythological stories
about the polar lights, drawing connections between different cultures that transcend political borders.
Without nation-state borders or other human interventions, the world map used in the installation is
purposefully rendered in a minimalist style. The fact that it is centered on the poles further distorts how
visitors typically perceive the world map. The visitors identified common archetypes in geographically remote
regions by experiencing mythological stories from various cultures.
Mytherrella is an experiment in creating a shared space for scientific data and personal stories. Such playful
settings that mix diverse fields of study and research approaches may motivate us to delve deeper and from
different angles into the stories behind them.
Acknowledgements
Mytherrella was developed during a residency in the Space Science Institute from Măgurele, Romania,
organized by Qolony, Bucharest. Interactive sound for the installation was created by Iulia Smeu. Dataset of
all-sky auroral images was provided by Kiruna Atmospheric and Geophysical Observatory at the Swedish Institute
of Space Physics.
References
Books
1 Asgeir Brekke, Egeland Alv, The Northern light: from mythology to space research,
Springer Science & Business Media, 2012.
2; John S. Major, Heaven and earth in early Han thought: Chapters three, four, and five of the
Huainanzi, SUNY Press, 1993, 203-204.
3 James Burke, The impact of science on society, vol. 482, Scientific and Technical
Information Branch, National Aeronautics and Space Administration, 1985, 3-32.
4 D. C. Rose, Keoeeit-The Story of the Aurora Borealis, by W. Petrie, Arctic 17, no. 2,
1964, 26-27.
Journal article (print)
5 Robert Holzworth, "Folklore and the Aurora," Eos, Transactions American Geophysical Union 56,
no. 10, 1975, 686-688.
Journal article (online)
6 Tero Mustonen, "Inuit and Chukchi Star lore: Reflections on Ursa Major, the North Star and
Northern Lights," Snowchange co-op, 2016. Accessed December 4, 2022,
http://www.snowchange.org/2016/11/inuit-and-chukchi-starlore-a-new-discussion-paper-out/
Websites
7 Anniina Jokinen, "Aurora Borealis, The Northern Lights, in Mythology and Folklore," Luminarium,
4 Feb 2007, Accessed December 4, 2022, http://www.luminarium.org/mythology/revontulet.htm
8 Burloiu, Grigore, “controlling StyleGAN in real time,” Accessed December 4, 2022,
https://rvirmoors.github.io/2021/01/04/realtime-stylegan/
At the Sources of an Artistic Mutation towards Science: the First Years of the Journal Leonardo (1968-1981) as
a Forum for the Pioneers of Digital Art
CamilleFrémontier-Murphy
RCM Galerie Paris, France camille@rcmgalerie.com
Abstract
The journal Leonardo was founded in 1968 by Frank Malina, a pioneer of light art in Paris and of aerospace
science in the United States. Leonardo encouraged artists to publish their work in the manner of scientists. It
was a new initiative developed in the revolutionary context of the 1960s and that allowed artists, scientists,
psychologists to exchange on the subjects of art, perceptions, science, society... Many pioneers of digital art
took part in the adventure, including Vera Molnar, Zdenek Sykora, Charles Csuri, and the artist-novelist Herbert
Franke, who became the advocate of the theories and the protagonists of digital art spread out over the four
corners of the planet, just about everywhere a computer could be found. The growing group was defending a more
conceptual approach to art, closer to the spectator, a new form of art rooted in Constructivism and that was in
symbiosis with society’s mutation towards technology.
When the journal Leonardo was founded half a century ago by the artist-engineer Frank J. Malina (1912-1981),
the first generation of artists working with computers and electronic media was still small and far apart.
Leonardo provided them with a permanent forum to exchange, share, and publicize their reflections and the state
of their work with their peers and also a wider audience at a time when international and interdisciplinary
means of communication were not readily available. While some of these pioneers have recently passed away, such
as Herbert Franke, Ken Knowlton, Charles Csuri and Jean-Pierre Hébert, others are still active, such as
Vera Molnar, Joan Truckenbroad, Ruth Leavitt, Jean-François Colonna, Jean-Claude Marquette, Hervé
Huitric and Monique Nahas. Together they have defined the foundations of a community whose concerns and
approaches from a not-so-distant past still resonate today.
The Journal Leonardo and the Buoyant Spirit of 1968
In January 1968, when Leonardo was launched, it was an effervescent time: changes and mutations in society were
crystallizing. A few months before the youth took to the streets of Paris, the ambitious multimedia Living Art
program of the ARC (Animation - Recherche - Confrontation), directed by Pierre Gaudibert, began organizing its
exhibitions and interdisciplinary events at the Paris Musée d'Art Moderne. This initiative was
revolutionary. The museum was no longer perceived as a temple, but as a laboratory that would invite a younger
and more diverse public until 1972. At the same time, an art and technology program began to develop at LACMA
under the direction of Maurice Tuchman. In London, the first major exhibition of electronic art and music took
place at the ICA (Institute of Contemporary Arts) in the late summer of 1968, demonstrating the scope of the
movement. In Zagreb, the series of exhibitions and symposia organized by the New Tendencies under the impetus of
Abraham Moles (1920-1992) opened its fourth chapter dedicated to computer art.
Malina, a Paris-based American artist, was at the center of this ferment. A few years earlier, the Paris
Musée d'Art Moderne had acquired one of his grid paintings, Deep Shadows (1954). He participated in New
Tendencies with a selection of his luminous mobile paintings—Signals, Voyage II, Sink and Source. At
Cybernetic Serendipity, he exhibited his interactive light work Entrechat and presented his new journal
Leonardo. He even helped the exhibition to travel to San Francisco’s Exploratorium, recently founded by
his longtime Caltech friend Frank Oppenheimer.
In fact, Malina himself was a former scientist turned artist. He had designed and supervised the launch of the
first scientific rocket, the WAC Corporal, and had founded the Jet Propulsion Laboratory, in Pasadena,
California, which is now the premier laboratory for the exploration of the solar system. After World War II, he
opposed the use of his rocket as a ballistic missile with a nuclear warhead, and had worked for several years
with Joseph Needham, head of the Natural Sciences Division of UNESCO in Paris, "to reduce barriers to the free
movement of scientists and engineers between nations." Needham also introduced him to György Kepes, a
member of the New Bauhaus in Chicago and the founder of the Center for Advanced Visual Studies at MIT.
Kepes’ book, Language of Vision, was one of the most important artist's manuals ever written
in the United States before Moholy-Nagy’s Vision in Motion (1947). His sociocultural ambition was
clear: "The goal is a new vital structure-order, a new form on a social plane, in which all present knowledge
and technological possessions may function unhindered as a whole"1.Inspired by these texts, and
sometimes frustrated by the bureaucracy of international organizations, Malina had begun an artistic production
at the intersection of art and science. Since the mid-fifties, he had been developing a corpus of moving light
paintings with themes reminiscent of recent space travel.He was actually one of the first artist-scientists of
his time, and soon came up with the idea of creating a society or journal for artist-scientists to meet and
exchange ideas, but he couldn’t find many of them. With the preparation of exhibitions on cybernetics in
London and computer art in Zagreb, a growing community of artists working on the frontiers of science emerged.
Max Bense and Abraham Moles began to apply Norbert Wiener’s cybernetic and information theory to aesthetic
problems. Together they edited the first issue of the magazine Bit international, first published in 1968 by the
Museum of Zagreb within the framework of New Tendencies. Unfortunately, the initiative didn’t last after
1972.
The Emancipated Artist Joins Science
Leonardo, for its part, wanted to be open to all forms of art: "What I was interested in was people who worked
in physics, or chemistry or physiology, or astronautics, or what have you, who introduced into their artworks
the visual experiences and the ideas of their work. And I found many of these people were not doing that. So, I
said that it was premature to try to make a journal for this kind of people. They weren't enough. And since
there was no professional journal for any kind of visual artist, the thing was to make a journal that was open
to any art, and there the artists would write about their work themselves, helped sometimes, and would be by
someone in the direction of relationship between art, science, and technology. (...) And at the end I wanted an
international journal to try to bring about better international cooperation, for peace."
Needham, who became famous for his history of Chinese science, suggested the name of the journal and would be
one of its honorary editorial advisers2. “Creativeness in art does not appear to me to differ
in kind from creativeness in science or any other human activity”, stated Malina in the brochure for his
first solo exhibition in Paris in 1953”3. It was a time when the excessive division of
knowledge reinforced by the specialization of disciplines, began to be questioned. In Gothic
Architecture and Scholasticism (1951), Erwin Panofsky described a common denominator for different forms
of human endeavor in a given period. Creation appeared to be culturally determined by mental habits common to
several disciplines.Leonardo defended the development of art as any other field of research. It was published by
the Pergamon Press of Oxford, which was already responsible for Acta Astronautica, the organ of the
International Academy of Astronautic, co-founded by Malina and von Kármán. English was quickly to
become the dominant language, as it was in the process of doing in the scientific field.Artists, as scientists,
would publish their own articles to discuss their practice, no longer leaving this mission to critics alone.
Artists would become literate persons able to discuss fundamental artistic, social and political
issues.Contributors could be European or American, including African American, Asian, male or female. The
invited authors could be computer artists, but not exclusively. Artists working in a constructivist vein or
kinetic artists, such as Nino Calos or Pol Bury could also contribute. These two artists were also well known
for their writings and will join Joël Stein, Anthony Hill, Vladimir Bonacic and Herbert Franke on the
editorial board of Leonardo.Scientists were also welcome to contribute to Leonardo. The artist was now a social
actor in his own right, able to interact with scientists. The psychologist James Gibson, who was one of the
honorary editorial advisers, wrote a paper on visualization and the point of observation in 1974⁴. Fred
Lawrence Whipple, head of the Smithsonian Astrophysical Observatory, shared in the first edition of Leonardo a
rather interesting way of conceiving paintings from random arrangements of forms and colors. The point was
certainly to clarify the inherent differences between randomness and chaos⁵. Its goal was to build bridges
between different communities, to advocate peaceful international relations. Malina's convictions have not
changed since he left JPL, when he said, “that ideas and effort were really needed now to find ways for
sovereign states to function in peace together, rather than to develop better means of destroying
themselves.”
A dedicated project for a dedicated community, Leonardo was a quite peculiar project of an artist dedicated to
creativity and his peers. It would be a large-scale, self-funded enterprise based in Paris, at a time when no
institution would have supported such an adventure for such a long time. It was made possible by Malina’s
income from the rocket engine company Aerojet.
Different Forms of Expressions for New Ambitions
”Painters during the last several centuries have painted the same things over and over,” Malina
complained in a letter dated October 21, 1953.From György Kepes—now one part of Leonardo’s
editorial advisors and authors —to most digital artists, the same vision prevailed: “the application
of the computer in the realm of art is to see it as part of the technological revolution.”
In 1969, the artist Frederick Hammersley recounted how, tired of classical painting, he enrolled in a computer
drawing class taught by Charles Mattox, a co-editor of Leonardo at the University of New Mexico:“ During
my first semester of teaching at the University of New Mexico in 1968, I was invited by Charles Mattox to attend
a computer drawing class. This happened to coincide with a time in which I had painted myself out, so I welcomed
this new experience. I was shown how to prepare a computer program and how to transfer it to an IBM punch card
by machine.”⁶ Working with these extremely large computers was counterintuitive to most artists of
the 60s, and Mattox decided to supervise a program designed for them. It was the beginning of a long adventure
of programs written for artists. ART1 was written in FORTRAN IV by Richard H. Williams of the Department of
Electrical Engineering and Computer Science. Artists and art students from the University influenced its
development and among them, Katherine Nash, who presented it in Leonardo with illustrations of her works. She
began her paper by stating: “Twenty years of a computerized society make it apparent that twenty years
hence no artists can ignore the computer.”⁷ Herbert Franke was a member of Leonardo’s
editorial advisory board. He explained that with the technological age, the computer could become an instrument
of art like an instrument was for music.The undeniable interest of manipulating such instruments was, without
question, to dissociate artistic practice from the manual arts. Drawing was no longer done by hand, but with the
computer. Art became multimedia and conceptual, rather than the romantic expression of a marginal artist
presented in gilded wooden frames. It could be defined, as in cybernetics, as a signal perceived by the
viewer.At the turn of the 1970s, Franke concluded: “If the day comes, as we are told it will, when every
household is connected to a computer network via a display terminal, anyone will be able to tune in on a large
variety of aesthetic programs. For this purpose, variable programs that permit intervention by the viewer will
be far more suitable than the static programs that are available today. Perhaps in this way the gulf that yawns
between the producer and the consumer will be slowly bridged.”⁸
The work of art created with the computer has been defined with functions similar to those of a book:
distributed in multiple copies, directly accessible at home, and in a special way interactive.Many artists will
share their own approach in Leonardo, such as Zdenek Sykora in 1970, about computer-aided geometric
paintings⁹, or Vladimir Bonacic10, Charles Csuri about facial sketching11, Edvard
Zajec12, Grace Hertlein about the largest computer art exhibition organized in the USA in
197513. The artist Ruth Leavitt and her husband Jay—responsible for heat shields for
spacecraft—presented their reflections on the geometric deformation of a figure, using the same principles
as Moiré14. Vera Molnar also published an article about her work in 1975. Her husband
François had published on art and science in 197415. For some who did not yet have access to
computers, such as Ben Laposky, they presented their skills using analog systems16. Access to a
computer was still extremely complex and, as Colette Bangert nicely summarizes the situation: “Although an
artist need not be knowledgeable in mathematics, he must be prepared to talk to a
mathematician”17.
With all its sometimes dramatic, social, poetic, and technical dimensions, the writings of digital artists in
the journal had become so extensive in the 1970s that Malina decided to collect the articles in a book, Visual
Mathematics and Computer, 1979. It was certainly not exhaustive, but it testifies to the beginning of this new
artistic adventure, just before it reached a low point, which is now being rediscovered. It sheds light on the
roots of today's electronic art lie18.
References
1 György Kepes, Language of Vision, Chicago, 1944, 12-13.
2 Joseph Needham, "Time and History in China and the West,” Leonardo 10, 1977,
233-236.
3 Galerie Henri Tronche, Frank J. Malina, Paris, October, 1953.
4 James J. Gibson, “Visualizing Conceived as Visual Apprehending without any Particular
Point of Observation,” Leonardo 7, 1974, 41-42.
5 Fred L. Whipple, “Stochastic Painting,” Leonardo 1, 1968, 81-83.
6 Frederick Hammersley, “My First Experience with Computer Drawings,” Leonardo 2, 1969,
407-409.
7 Katherine Nash, Richard H. Williams, “Computer Program for Artists: ART 1”,
Leonardo 3, 1970, 439-442.
8 H. W. Franke, “Computers and Visual Art,” Leonardo 4, 1971, 331-338.
10 Vladimir Bonacic, “Kinetic Art: Application of Abstract Algebra to Objects with
Computer-Controlled Flashing Lights and Sound Combinations,” Leonardo 7, 1974, 193-200.
11 Mark L. Gillenson, B. Chandrasekaran, Charles Csuri, Robert Schwartz, “Computer-Assisted
Facial Sketching”, Leonardo 9, 1976, 126-129.
12 Edvard Zajec, “Computer Art: a Binary System for Producing Geometrical Nonfigurative
Pictures,” Leonardo 11, 1978, 13-21.
13 Grace C. Hertlein, “Report on the 2nd International Conference on Computers and the
Humanities, Los Angeles, Calif., U.S.A.”, Leonardo 9, 1976, 43-45.
14 Ruth Leavitt, Jay Leavitt, “Pictures Based on Computer Drawings Made by Deforming an
Initial Design,” Leonardo 9, 1976, 99-103.
15 Vera Molnar, “Toward Aesthetic Guidelines for Paintings with the Aid of a
Computer,” Leonardo 8, 1975, p.185-189, François Molnar, “Experimental Aesthetics
or the Science of Art”, Leonardo 7, 1974, 23-26.
16 Ben F. Laposky, “Oscillons: Electronic Abstractions,” Leonardo 2, 1969,
345-354.
17 Colette S. Bangert, Charles J. Bangert, “Experiences in Making Drawings by Computer and
by Hand,” Leonardo 7, 1974, 289-296.
18 Frank Malina (ed.), Visual Art Mathematics & Computers. Selection from the Journal
Leonardo, Oxford, Pergamon Press, 1979.
Author Biography
Camille Frémontier-Murphy, holds a Ph.D. in History and Civilizations, Centre Alexandre Koyré,
E.H.E.S.S., Paris, a Master's degree in Art History, Paris IV, Sorbonne, and a Master of Science, History of
Science, University of Oxford, Oriel College. Together with Robert Murphy, she runs the RCM Galerie in Paris,
whose exhibition program explores the avant-garde and the relationship between art and science in the second
half of the 20th century.
Sensoriums for the Ephemeral — gamification of values
JānisGarančs
Liepaja University & RIXC Liepaja & Riga, Latvia jg@rixc.lv
Abstract
This paper introduces the author’s current practical investigations, during the creation of a series of
audio-visual installations and VR environments, ‘gamifying’ time-sequenced changes of multiple
values, e.g., from financial data feeds. The work series critically reflect upon gambling tendencies in the
global trading of various, increasingly immaterial assets. Algorithms and emotions of greed, euphoria, and
despair meet in virtual scenery, where almost everything can be offered as a fungible and non-fungible token for
exchange and trade. The project proposes speculative variations of dystopian “hybrid organisms”
representing macroeconomic value exchange as a symbiotic relationship that competes for humanity’s
attention and involvement.
Keywords
Gamification of investment, market society, immersive analytics, ephemeral values, neuroaesthetics
This work series is critically motivated by the re-emergent and growing prominence of gambling factors in
global economic activities—such as institutional promotion of increasingly complex investment products for
masses, crypto-asset trading, online casinos, etc.
Political philosopher Michael J. Sandel describes several last decades as a “drift from ‘market
economy’ to becoming a ‘market society.’”¹ As sociologist Georg Simmel
had already observed in 1900: “Reality and value as mutually independent categories through which our
conceptions become images of the world.”²
Mapping of the behaviour patterns in trader psychology has been an important aspect of the training in the
trading process, besides the implementation and development of various mathematical models. It now appears that
value storage and trading infrastructure increasingly merge with methods of manipulation of human attention and
emotions, and are mediated by computer networks, and increasingly—Machine Learning and AI. There is a
hope, that AI-assisted macroeconomic system could live in symbiosis with human society by working closely with
humans to understand and respond to their needs and goals.
Gamification of investment as a phenomenon
Various schools of economic thought, which emphasize the importance of free markets and individual choice,
might argue that the gamification of investment is a natural and inevitable outcome of a free market economy. In
this view, the gamification of investment is simply a response to the demands and preferences of consumers and
is therefore a legitimate and desirable part of the market process. The most recent examples are various
cryptocurrency and crypto-infrastructure products, meme-stock trading by inexperienced
“investors,” peer-to-peer loan financing, allocations of state pension funds into high-risk
assets, etc. (the most extreme case being introduction in Bitcoin as the legal tender in the country of El
Salvador).
Adversary interactions between technology and legislation in economic activities, especially in
finance—can be described as multilayered symbiotic relationship with society—that can manifest
itself through both mutually beneficial and predatory to society by all involved actors. There have been
historic precedents how rationally motivated mechanisms of value estimation, asset exchange and wealth storing
has caused irrational mass behaviours, financial bubbles. That has severely impacted individual destinies,
families, nations, and whole generations.
It has caught attention not only in sociology, as symptom of serious social “disease”—but
also caused legal interventions of large supervisory bodies managing legislation frameworks for investments and
finance.³Participation in online infrastructure for trading—through easily accessible
interfaces, provides new tools for masses, promising control, but, at the same time, becomes part of
surveillance tool for organised value brokers, setting preconditions for unfair competitiveness, “rigged
game.”
Simulated sensorium for immersive mappings
This artwork series—immersive installations using 3D imagery—feature variations of speculative
illustration of a dystopian ‘hybrid organism’ and simulated macroeconomic system, whose purpose is
transformation of energy and resources through a behaviour and appearance that ranges from ludic (playful) and
awe inducing. One iteration uses live trading data from various sources to create a “virtual
landscape” of trading activity, to observe ‘passively. In another iteration, visitor can make
‘virtual bets’ to experience growth and diminishing of their “investment“ through the
cycles and complex “monetary circuits.“
It is possible that McLuhan today would see immersive analytics for online trading as an example of the
‘extended sensorium’ in a way that media technologies are extending and enhancing human abilities in
the realm of financial decision-making in “global village.“ The concepts of “360°
gaze“ (Stiegler, 2011) or more broadly dispositif (Foucault, 1977) are relevant how interfaces
for value exchange modifies our behaviour of the mediated self under the impression of surveillance. Stiegler
relates to the “data behaviourism“ (Rouvroy, 2012) in the meaning of “producing
knowledge about future preferences, attitudes, behaviours or events without considering the subject’s
psychological motivations, speeches or narratives, [instead] relying on data.“ There are observations,
that visual representations of financial markets and sophisticated products, can be misleading by design choices
or be used “as tools of manipulation” (Krawczyk, 2021).
There has been a range of historic and recent examples of 3D visualization of various data sets—as well
as GUIs for various professional software products and artworks. The projects presented here, however, try to
establish a gradual journey between the extremes of design strategies: intended usability/function and sublime
"dysfunction" as an aesthetic experience. These audio-visual études are practical investigations into the
continuum between immersive analytics and VR/AR artwork. The emerging research area of immersive analytics is
considered a fusion of more recent developments in visualization, auditory displays, computing, and machine
learning.⁴
Between emotional and non-emotional
How do we use technology in hybrid forms to extend our perceptual and sensual abilities? In which way do bodies
merge with technological agencies? In the immersive analytics scenario, the difference between emotional and
non-emotional evaluation of data refers to the use of emotional or affective factors in the analysis and
interpretation of data. In a non-emotional evaluation, data is analysed and interpreted solely based on its
objective characteristics, such as its numerical value or its statistical significance. This type of evaluation
is based on logical and rational reasoning and does not consider the emotional or affective reactions of the
user or the audience.
In contrast, an emotional evaluation of data in an immersive analytics scenario involves considering the
emotional or affective reactions of the user or the audience in the analysis and interpretation of data. Also,
the concept of ideastesia,⁵ which refers to the experience of emotions in
response to abstract ideas, can be applied to the above-mentioned topics in several ways—creating
ephemeral audio-visual representations of “placement,” “reference points”—as
“spatial anchors” in VR and AR setting.
Metaphors and terms
Through algorithmic feedback, virtual instruments with anisotropic properties enable on-screen
manipulation, switching between egocentric and exocentric navigation, and the synchronized off-screen
structuring of the content. The analogy of simulated anisotropy—as an algorithmically simulated
phenomenon of anisotropy known in physics, chemistry, microfabrication, neuroscience as direction-varying of
material, tissue, and space properties.
Anisotropic properties refer to the ability of an interface to have different properties or behaviours in
different spatial directions or contexts.
The notion of the instrument is used both as a generic definition of a task-specific tool (or
toolset, involving appearance and action parameters, that be used to e.g., detect, measure, modify a specific
situation or manipulate a specific object)—and in analogy to the musical, medical, or industrial
instrument. Other terms “ensemble visualization”or “ensemble data” are used in
the context of data visualization as “concrete distributions of data, in which each outcome can be
uniquely associated with a specific run or set of simulation parameters.”⁶
Data staging
For the installation, the data acquisition modules feed from several sources of financial data (stock-
and cryptocurrency exchanges, aggregators, etc.) over their public Internet APIs. Installation accumulates
historic and realtime trading data in the local data cache and uses provided calculations of trading trends and
indexes—further calculations on the time-series data. Various time zoom scales reveal the phases of past,
historic trends, that emphasize the position current trade execution as a spatially expressive metaphor.
Visualisation and sonification modules utilise several specialised financial TA (Technical Analysis) programming
libraries to dynamically calculate sections in the audio-visual score for the real-time 3D graphics and
sonification engine. The use of consistent spatial arrangements helps to create a sense of order and coherence
in the visualization, revealing flows of market movements, and identify relationships and patterns within the
data.
Audio-visual staging
In the scenes, the progressing complexity or visitor-triggered mode shifts induce a challenge to the
audio-visual sensorium: the experience of the conflation within multiple reference systems and plays with the
visitor’s perception effort of “sense-making.”
Various sound properties, such as pitch, timbre, rhythmic elements are juxtaposed in linear-and non-linear
grids and ephemeral relationships are revealed and emphasized by spatially organized audio-visual cues
(perspective, sharpness/blur manipulation), audio-panning, and timbral modulation, allowing the viewer to easily
distinguish between changing context without having to rely on visually “scanning” the chart. This
can be particularly important in complex or dense scenery, where a well-organized spatial arrangement can help
to reduce cognitive overload.
Data representation axis and colours
Axes, colours, and timeline of time series data such as stock or currency trading are typically represented in
a consistent manner across cultures and regions. However, asset value increase usually mean economic gains,
however, in specific trading mode—“shorting,” that is a bet on asset value loss and the gain
is “reversed.” Else, there are some differences that can be attributed to a combination.
Conclusion
Algorithmically manipulated financial trading is an arena where algorithms merge with human emotions and drives
like greed, FOMO (“fear-of-missing out”), euphoria, confusion, and despair into a global hybrid
sensorium. As a phenomenological effort, this work proposes a set of parameters for simulated anisotropy, useful
for designing the structure, notation, and physical, mathematical dimensions in the VR interface and
environment. The framework for simulated anisotropy instruments is envisioned as a contribution to the emergent
fields of immersive analytics and neuroaesthetics, developing guidelines for a hybrid interaction system,
consisting of networked hardware specifications and custom-developed software modules. This project will
implicate experimental strategies with uncertain capacities in the effectiveness of interaction, and impact of
integrated audio-visual language, expanding the vocabulary of spectromorphology.⁷ Future
intentions for this project, using techniques such as sentiment analysis to assess the emotional tone of user
feedback or comments, or using physiological measures, such as heart rate or galvanic skin response, to assess
the user's emotional response to the data.
References
1 Michael J. Sandel, What Money Can't Buy: The Moral Limits of Markets Farrar, Straus and
Giroux, 2012.
2 Georg Simmel, The Philosophy of Money, 3rd edition by Routledge,
2004.
3 Peter WS Newall, Leeonardo Weiss-Cohen, The Gamblification of Investing: How a New
Generation of Investors Is Being Born to Lose, Int J Environ Res Public Health, Apr 28,
2022.
4 Richard Skarbez, Nicholas F. Polys, J. Todd Ogle, Chris North, Doug A.
Bowman, Immersive Analytics: Theory and Research Agenda. Frontiers in Robotics and AI
6, September 10, 2019.
5 Danko Nikolić, “Ideasthesia and
art,” in Digital Synesthesia. A Model for the Aesthetics of Digital Art, ed. Katharina
Gsöllpointner, et al., Berlin/Boston, De Gruyter, 2016.
6 Harald Obermaier, Kenneth I. Joy, Future challenges for ensemble visualization,
IEEE Computer Graphics and Applications, Volume: 34, Issue: 3, May-June 2014.
7
Manuella Blackburn, The visual sound-shapes of spectromorphology: An illustrative guide to composition,
Organised Sound Volume 16 Issue 1 April 2011 of cultural and historical factors, as well as the specific
requirements and conventions of each culture – in few Asian cultures, axis, and colours for
positive/negative chart values (up/down or green/red) are swapped. The discussed experimental artworks
implement perceived relativity of orientation axes and colour schemes to establish ephemeral cognitive
associations, that are enhanced by artificially synaesthetic bindings.
Bibliography
Philipp Chapkovski, Mariana Khapko, Marius Zoican, Does Gamified Trading Stimulate Risk Taking?,
November 25, 2021.
Swedish House of Finance Research, Paper No, 21-25.
Samuel Chabot, Jonas Braasch, “High-density data sonification of stock market information in an
immersive virtual environment,” The Journal of the Acoustical Society of America, 2017,
p.141.
Marcin Marian Krawczyk, From making visible to hiding. Visual representations of financial markets as
tools of manipulation and active and living agents, Visual Studies, 2021.
Stefan Engeser, Falko Rheinberg, “Flow, performance and moderators of challenge-skill
balance,” Motivation and Emotion, 32(3), 2008, 158–172.
Hong Jun Song, Kirsty Beilharz, “Spatialization and timbre for effective auditory
graphing,” AMTA'07 Proceedings of the 8th WSEAS international conference on Acoustics &
music: theory & applications, 2007, 18-26.
Jiahua Xu, Benjamin Livshits, The Anatomy of a Cryptocurrency Pump-and-Dump Scheme, Proceedings
of the 28th USENIX Security Symposium, 2019, 1609-1625.
Antoinette Rouvroy, The End(s) of Critique: Data Behaviourism versus Due Process, In Privacy, Due
Process and the Computational Turn: The Philosophy of Law Meets the Philosophy of Technology, 2012.
Christian Stiegler, The 360° Gaze - Immersions in Media, Society, and Culture,
2021.
Adrianna Zuanazzi, Uta Noppeney, “Additive and interactive effects of spatial attention and expectation
on perceptual decisions,” Sci Rep 8, 2018, 6732.
Mycorrhizal Materialities Positioning the entanglement of human and machine intelligence
KateGeck
RMIT University Melbourne - Australia kate.geck@rmit.edu.au
Abstract
Machine intelligence is increasingly being used in the world with sometimes dramatic effects on human and
other-than-human lives through its decision-making capacity. Much artificial intelligence (AI) and machine
learning (ML) is built on metaphors that centre extraction, competition and control. These also position AI
itself as a resource to be extracted and controlled, paving a troubling path for speculative futures where AI
may gain emergent or ambiguous levels of sentience. These metaphors are part of a historical trend where humans
place themselves above the other-than-human world, and this has formed the basis of an extractive and one-sided
relationship with that world. In light of this, what new metaphors might we employ to platform the relationships
between human and machine intelligences?
Thinking through mycorrhizae could be a productive way to foreground the entangled, generative nature of
exchange between human and machine intelligences. This paper will briefly explore metaphor in human-computer
interaction (HCI) and AI, before making an offering to think about these things through the material of the
mycorrhiza, a symbiotic site of exchange between plants and fungi. It will then briefly detail a creative
project that has emerged from this mycorrhizal thinking to produce machine imagined textiles and embroideries.
It then concludes with a call to embed relational thinking into future practices between human and machine
intelligences in order to create more equitable and even mutualistic outcomes.
Machine intelligence is increasingly being used in the world with sometimes dramatic effects on human and
other-than-human lives through its decision-making capacity. Much artificial intelligence (AI) and machine
learning (ML) is built on metaphors that centre extraction, competition and control. These also position AI
itself as a resource to be extracted and controlled, paving a troubling path for speculative futures where AI
may gain emergent or ambiguous levels of sentience. These metaphors are part of a historical trend where humans
place themselves above the other-than-human world, and this has formed the basis of an extractive and one-sided
relationship with that world. In light of this, what new metaphors might we employ to platform the relationships
between human and machine intelligences? Thinking through mycorrhizae could be a productive way to foreground
the entangled, generative nature of exchange between human and machine intelligences. This paper will briefly
explore metaphor in human-computer interaction (HCI) and AI, before making an offering to think about these
things through the material of the mycorrhiza, a symbiotic site of exchange between plants and fungi. It will
then briefly detail a creative project that has emerged from this mycorrhizal thinking to produce machine
imagined textiles and embroideries. It then concludes with a call to embed relational thinking into future
practices between human and machine intelligences in order to create more equitable and even mutualistic
outcomes.
Metaphor and HCI
Human Computer Interaction Design (HCI) has a long history of employing metaphor to guide and facilitate
interactions between humans and machines. One of the best known examples is the early work of Xerox Parc with
the desktop metaphor. This located computing for the general population within the realm of business
administration and the organisation of documents in a paperless, digital office. The desktop metaphor therefore
helped us understand how computers could augment our administrative productivity. The metaphor provided a shape
to the world of interaction between humans and computers, helping us locate them in our everyday practices and
providing edges to what we assumed was possible with them. This illustrates how metaphors are "imaginative and
creative... capable of giving us a new understanding of our experience."¹ They act as a shorthand for what
is possible with an idea, situation or in the case of technology, with a system. Artificial Intelligence (AI) is
subject to a number of popular metaphors that centre on abundance, extraction and competition². AI emerges
seemingly uncontrolled in ‘explosions of intelligence’; an abundance of "data is mined"; and nations
are engaged in ‘AI arms races’. While some of these metaphors are ecological in nature -
‘oceans of data’ and the so-called "Seed AI"—these are still one-sided tending towards
extraction and domination. There are few, if any relational metaphors offered for the dualistic nature of
ongoing exchange between human and machine intelligence. We seem to work with uneven metaphors: they enable us
to extract without consequence from seemingly endless and abundant data and they point towards neutral and
efficient control of the world at scale. We know that metaphor does so much work, especially in HCI, to enable
us to think with our technology and imagine our futures. Metaphors help us make sense of the world and help us
understand how to act: they "provide coherent structures, highlighting some things and hiding others."³ So
what might we want to highlight in our relationships with machine intelligence moving forward? Which things do
we wish to stop hiding? Could we work with alternate, ecological metaphors that help us imagine mutualistic
relations between human and machine intelligences?
Mycorrhizal Materiality
Mycorrhizae are a relational phenomenon between plants and fungi. They occur when the hyphae of fungi and the
tips of plant roots enter into a mutualistic relationship where they orchestrate the exchange of nutrients
between them. The relationship is dynamic: connections between plant and fungus will continue to form and
re-form to "ceaselessly remodel themselves."⁴ Susan Simard and her team demonstrated this ongoing
mutualistic transfer of resources between plants and fungi through mycorrhizal networks by tracing the movement
of particular carbon isotopes through the network.⁵ As a result, this network is sometimes referred to as
"the wood wide web". Due to their networked nature, mycorrhizae offer an interesting, mutualistic metaphor for
repositioning the relationships between human and machine intelligence. The mycorrhiza is a lively material, a
"polyphonic assemblage" of relations, chemicals and organisms. This "polyphonic assemblage", in the words of
Anna Tsing, is a dynamic, multispecies space that enables a range of voices to be listened to.⁶
The polyphony of mycorrhizae allows us to listen to the different agents and forces within the assemblage as
independent voices, while still enabling them to collect together at particular moments and be heard as a whole.
In the mycorrhizae, we can track the flows of nutrients between life forms and of energy from the sun to the
unfolding of leaves. We can peer under a microscope and examine the interconnected, entangled materialities of
plant and fungus as hyphae penetrate roots, becoming essentially impossible to disentangle. If we approached the
design and deployment of AI systems through this lens, it would enable us to understand from the outset how
biases become entangled within algorithmic decision making.
There is now significant awareness of the role of bias in algorithmic decision making. Mehrabi et al.
systematically outline an extensive list of the types of bias evident in a range of ML applications.⁷ A
vast selection of ways in which bias can emerge is presented, as a whole demonstrating the entangled and
relational nature of exchange between human and machine intelligence. These examples emerge from 3 domains:
‘data to algorithm bias’ where the data may be interpreted by the algorithm in a biased way;
"algorithm to user bias" where algorithms might influence human behaviour; and "user to data bias" where the
human creators of datasets infuse their own biases.⁸ These 3 domains individually and collectively
demonstrate a mycorrhizal structure: data is like a nutrient, flowing from human to machine, sustaining and
limiting the growth of social imaginaries. It is clear from this study and others that it is in fact impossible
to remove bias completely, and so a conceptual ideological shift towards understanding that human and machine
intelligences are entangled provides a base from which to work in further checks and balances on the
implementation of intelligent decision-making systems.
Creative Exchange
We can apply the concept of the mycorrhizae to creative exchanges between human and machine intelligence. There
are a number of (ever increasing) tools available to collaborate with machine intelligence. Generative machine
learning platforms like Midjourney, Stable Diffusion, and Dall-E use text to image prompts to enable artists to
co-create graphic outcomes with diffusion models. Others such as Playform enable artists to work with their own
datasets and Generative Adversarial Networks (GANs) to generate new images. Naturally, these spark criticism and
concern that machines will usurp the role of artists, or that it is not "real art".⁹ However, if we
reframe the relation between human and machine intelligence from the vantage point of mycorrhizae, we can see
that any creative artefact produced is the effect of an entangled assemblage. The artefact can only emerge
through the creative exchanges between human and machine intelligence: from the wording of the prompts, to the
selection of images to move onto further generation, to the curation and presentation of final images that may
be used. A novel example exists in the art project "Mycorrhizal Materialities" (in Figures 1 - 3). A GAN was
trained on 100 hand drawings of Australian fungi created by myself. The images generated by the GAN were refined
through my own aesthetic choices, shaping the "evolution" of the images. Select images were then transposed
using software into embroidery designs and stitched via machine to produce the final works—a series of 4
hangings 50 cm X 180 cm.
This work is an exploration of a co-creative relationship between human and machine intelligence—using
the mycorrhiza as a metaphor for an entangled and generative creative process. The works in this project
consider ML through a relational, material lens with the textile outcomes providing a metaphorical embodiment of
this lens. The relations between human and machine intelligences are interwoven, producing an agential fabric.
The embroidered works explore material entanglement between different intelligences, overlaying the symbiotic
exchanges between human and machine intelligences with those between plants and fungi. These textiles emerge
solely through ongoing creative exchange between me and the algorithms: my drawings generate machine imagined
responses, and further algorithmic exchange occurs as these images are transposed through software into
embroidery designs and again when the machine embroidered onto the fabric. These textiles represent what Karen
Barad calls an "agential cut": produced through the entangled materialities of human and algorithmic
intelligences.¹⁰ These creative works demonstrate a mycorrhizal materiality: ongoing exchange between
human and machine intelligence enable the artefacts to emerge. There are ongoing flows as ideas are offered and
exchanged in a collaborative relation.
Speculative Entanglements
A mycorrhiza involves intentional exchange of nutrients between plants and fungi. In some instances, the fungal
hyphae penetrate the plant root cells, connecting directly to the plant. Human and machine intelligence can be
seen as engaged in direct exchange of knowledge between the two systems. While we do not presently have direct
corporeal interfacing at any scale between human and machine intelligence, it could be said that the transfer of
knowledge, ideas, possibilities and imaginaries are already in considerable practice between human and machine
intelligence across health, justice, administration and the arts. This is having significant effects on human
life as well as on the other-than-human lives of various plants and animals. Mycorrhizal thinking makes space
for care in speculative futures where machine intelligences might gain the capacity to experience the world.
While likely different to human-like consciousness or sentience, there is the capacity for an AI "experience" to
emerge at some point in the future. Systems of machine intelligence often operate with sensing data, processing
this information and making a decision based on particular conditions. At some point, this feedback loop could
complexify in new ways, possibly giving rise to conditions of reflection, whim, or preference: at some point it
might become possible to "feel like an AI"; for an umwelt or zone of experience to emerge around an AI’s
sensing apparatus. This could be impossible entirely, or it could be rudimentary or it could be complex. It
could echo what little we know about the umwelts of plants and insects or it could emerge from sensing
capacities as strange and diverse as those that presently occur in the world. A mycorrhizal network can be
mutualistic or parasitic: the metaphor enables us to think in a holistic and ecological way about the nature of
the relationship between human and machine intelligences. It allows us to ask interconnected questions of our
systems: how might a system place further and unnecessary demands on the earth, such as the use of AI to
essentially enhance the profitability of the mining sector.¹¹ And in a speculative future where
machine intelligences have experiential capacities, how does true mutualism factor into exchange between human
and machine intelligences? Will we continue to base these relationships on extraction, control and domination?
Or might we move toward more symbiogenetic relations, as described by Haraway where "symbogenesis is not a
synonym for the good, but for becoming-with each other in response-ability."¹² Could we weave futures
where human and machine intelligences emerge as companion species, co-evolving through care?
Conclusion
To conclude, machine intelligence is increasingly used in the world. It has sometimes dramatic effects on human
and other-than-human lives through its decision-making capacity. Many of the metaphors that have brought us to
this point centre on extraction, competition and control.
More relational metaphors—like that of the mycorrhiza—might help us think with these technologies
in a more relational way. The creative project detailed in this paper demonstrates an entangled approach to the
creation of new work that remains cognisant of its interconnection between human and machine intelligence. Human
and machine intelligence will always be highly entwined—entangled with one another in an exchange of ideas
and potentialities, just like a mycorrhiza. If we could embed this relational understanding into our practices
with machine learning systems we might be able to better direct their biases and end purposes towards
mutualistic and equitable outcomes.
References
1 George Lakoff, Mark Johnson. Metaphors We Live By, Chicago, Ill, University of
Chicago Press, 1980, 139.
2 Kate Geck, Knitting Algorithmic Assemblages, TEXTILE, November 14, 2022,
1–18, https://doi.org/10.1080/14759756.2022.2114240.
3 George Lakoff, Mark Johnson. Metaphors We Live By, Chicago, Ill, University of
Chicago Press, 1980, 139.
4 Merlin Sheldrake, Entangled Life:How Fungi Make Our Worlds, Change Our Minds and Shape Our
Futures, S.L., The Bodley Head Ltd, 2020, 42.
5 Suzanne W. Simard, David A. Perry, Melanie D. Jones, David Myrold, Daniel M. Durall, Randy
Molina, “Net Transfer of Carbon between Ectomycorrhizal Tree Species in the Field.” Nature
388, no. 6642, August 1997, 579–82. https://doi.org10.1038/41557.
6 Anna Lowenhaupt, Tsing The Mushroom at the End of the World: On the Possibility of Life
in Capitalist Ruins, Princeton, Princeton University Press, 2015, 24.
7 Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan, “A
Survey on Bias and Fairness in Machine Learning.” ACM Computing Surveys 54, no. 6, July
2021, 1–35. https://doi.org/10.1145/3457607.
8 ibid
9 Kevin Roose, “An A.I.-Generated Picture Won an Art Prize. Artists Aren’t Happy",
The New York Times, September 2, 2022,
https://www.nytimes.com/2022/09/02/technology/ai-artificial-intelligence-artists.html.
10 Karen Barad, Meeting the Universe Halfway [Electronic Resource], Durham, N.C.,
Chesham, Duke University Press, 2007.
11 KoBold Metals, “Applying Big Data & Superior Science to Exploration,” n.d.
https://www.koboldmetals.com/.
12 Donna J. Haraway, Staying with the Trouble: Making Kin in the Cthulucene, Durham,
N.C., Chesham, Duke University Press, 2016, 102.
Interspecies Communication — Water bodies
AndreaGogova
ArtSci and Technology, Envi_Art Lab. Department of Environmental Ecology and Landscape
Management Faculty of Natural Sciences, Comenius University in Bratislava Bratislava,
Slovakia andrea.gogova@uniba.sk, andgogo@hotmail.sk
Abstract
In planet Earth, most processes are based on water. Water regulates climate, morphologically influence a
landscape, is a medium of living processes. It is a medium of interaction of organism and mineral parts in
microscopic view, between whole organisms and minerals in macroscale on the Earth, and endless interplanetary
space. Water topic (more than "parasitically" utilization of water resources (Serres, 2007)), in the ArtSci
project is focusing on its communicative possibilities. Equal communicative principal possibilities for all
living—human and other than human bodies—is a metaphor of the epistemological problem of water
protection as a medium of life, biotopes, and ecosystems. Because we are connected through water, we are all
bodies of water. Understanding water through environmental analysis of the territories, philosophy, and fluid
mechanics (chaos-based fluid attractors principle) will bring communicative equality for all bodies of water and
the inevitability of environmental protection. When communication cannot be fluid, then each of us as bodies of
water, will be lost in time-space. The new interdisciplinary methodology of communicative artwork is based on an
ArtSci manifestation of messages mediated in water as an asemic writing or the other kind of communicative
interrelation between human and more than human. The basis is the phenomenological research of water as a medium
of communication which causes a feeling and a togetherness. We and other bodies of water could feel our
togetherness through water.
Keywords
interface, pattern of communication, sympoiesis, structural coupling, water
The problem is approached as artsci research using knowledge of biosemiology and philosophy, environmental
ecology, and landscape analysis to develop a specific interdisciplinary artwork which will support environmental
education.
Water body figuration is related to possibility mediate
interspecies communication to better understanding waterbased life on the planet.¹Water is an
essence and principle of form, which is actualized in the water body figuration. According to principle of
‘structural coupling’ patterned cognition and communication of each communicative actor is related
to their environment. If water is essence of inner and outer environment of actors of communication and water
body figures, their fluid principle, relates to their agency, then relates to the concept of agential cut, and
in differentiation could appear in the meaning which would be presented by the ArtSci work.
In this paper I come out from my last research of transient pattern—visual model of transient patterned
of digital work communication which was based on relation of human and machine intelligent agency (Author,
2021). The biosemiotics view of patterned cognition and communication comes from Maturana and Varela’s
“Structural coupling” theory, in which the recurrent interaction of unit (organism) and medium
(environment) causes structural coupling, which is reflected in a patterned cognition and communication of
autopoietic complex organization (Maturana and Varela, 1972) and Jakob von Uexküll's umwelt theory.
Rereading Darwin`s theory of natural selection Elizabeth Grosz describe it as a complex of parallel randomly
interfere processes of internal dynamism of living beings and assertion of external forces and influences
(Grosz, 2004). Stafford Beer tried to explain Maturana and Varela’s autonomy of autopoiesis that
“...autopoietic systems are environmentally open to material-energetic fluxes or semiotic
mediations” (Beer, 1980) Donna Haraway instead proposes principle of sympoiesis, as enabled to incorporate
a complex interrelation of system as such (Haraway, 2016). The decisive principle is to recognize water as a
communication medium according to the possibilities of structural coupling, but instead of autopoiesis I related
it to the principle to the sympoietic system where the notion of evolutionary changes is crucial hack by
symbiotic transfer, lateral mixing, crossspecies contamination, and viroid life (Neimanis, 2017) Communication
of that holobiontic water based systems then depends on intersection of agential cuts of each body of water of
their unpredictable transient pattern of communication and cognition; according to Neimanis, in common space of
water body (medium) of interconnected ‘bodies of water. My questions are: Could meanings or
feelings emerge in the interrelation of water body figure (water which is an inseparable part of organism) and
water as medium? If yes, then interspecies communication will be interaction of each unpredictable
pattern or umwelt through dialogic membrane—“which is water body medium”, in the semiosphere.
The state of art
Several authors proposed the importance of water as a medium of interconnectivity between organisms (or
minerals) together (as physiological and biochemical processes. Some botanical research describes the
potentiality of mutual interspecies communication mediated by water. For example, the hydraulically lifted
transfer of mycorrhizal network provides a potential pathway between plants based on water (Egerton-Warburton,
Querejeta, Allen, 2007). Water was always a part of the formation of Earth`s mantle, but also originated from
comets and asteroids, that break up into meteorites which fall on the Earth (Crockett, 2015; Hartloch, 2011).
The fact can push the possibility of interspecies communication to the interplanetary space. Water as physical
matter is shaped form the solid, liquid and gas materiality. Fluidity is not only one state of water matter, but
mechanical, formal aesthetical principal of artistic, designer, and architectural practises. The fluidity is
recognised in the relation to Gilles Deleuze and Bernard Cache theory of Objectile1 , which based a theoretical
background to digital artistic form or artwork body figuration. Then water body figures appear in the
differentiations in the repetition processes of multiplicity of water cycles and niches, similarly, as appears
meaning of sign (Derrida, 1967— différance; Deleuze, 1994; Neimanis, 2017). According to Astrida
Neimanis (2017) in differentiation are continuously unfolding embodiments as an expression of eternal return of
the self-same. Water engendering difference “was” an expression of water that “is”, and
its potential “yet-to-come ”. Bodies of water as figuration was already describe within ecofeminism
and anticolonial thinking (Neimanis, 2017, Gaard, 2003, Armstrong, 2006). Lucy Irigaray (1992) related fluidity
to the feminine body as “fluid and ever mobile”, and “secreting a flow.” Fluidity is
diffuse and multiple, overlapping, and interconnected, is repetition in hydro/bio cycles and water figures
acknowledged in differentiation. The dynamic fluid principle relates to the chaos theory, where changes are
based on small intrusion of triggering power, which are resulted in the system changes. The process is based on
perpetual intra-action, entanglement, diffraction, and agential cuts which was described by Karen Barad
“matters agential realism” (Barad, 2007). Then in differentiation is appearing meaning, also any act
of observation is differencing agency and makes a “cut” between what is included and excluded. Every
species has specific, but transient "pattern" of differencing agencies. From the position of a
posthuman approach, then it is possible to see water as a common equal medium of communication between all
bodies of water (human, more-than-human, extra-terrestrial), which can bring water based informational
agency.
Interspecies communication as ArtSci project
The most art research which relates to interspecies communication was based on translation or transcription of
many levels mediated communication principle. Eduardo Kac in his bio art transgenic artwork
“Genesis” (2000) resolves the problem of interspecies communication according to speculative
transcription to human language of Morse code (see https://www.ekac.org/geninfo2.html). The other bio artist
Špela Petrič in her work ‘Institute for Inconspicuous Languages: Reading
Lips’(Petrič, 2018) relates the transcription of water-based physiological interaction of plant to
visual transcript of human sign language (See
https://www.spelapetric.org/#/institute-for-inconspicuous-languages/).... etc., Both of work was related to a
human existing language. Instead of relate communication to human language I propose water body as a common and
direct medium of emotional, also interspecies communication of water body figures. Then understanding appearing
in the differential intersection of agency of each communicative body of water and through a repetition will
come to common emotional “hydro language”.
In the research I focus on water in the role of the medium of emotional communication between all bodies of
water human/more/than human, (extra-terrestrial), both in micro and macro scale. Then cognitive pattern of each
communicative bodies of water relates to its specific agency. The agency relates to agency of water body figures
which are constituting new and reconfiguring systems in the concept of “matters agential realism”
(Barad, 2007), to agents of water-based communication. This principle of shaping the matter is the process of
performative bodymaterial practice which would configure figures and its transient patterns of communication to
generate the vision of the artwork.
Procedural model in relation to interspecies water-based communication
Because water in bodies is flowing in perpetual cycles of water niches, it is related to the procedural model
of communication which was described by Philippe Bootz², but also water is an essence of the artwork.
The essence is related to the water body essence. Then the artwork water body is defined as an
individual water body. The figure of the individual water body relates to agency of interaction (communication).
According to Philippe Bootz`s model of visualization of procedural digital works is work individual body defined
by essence, extensive parts and relations linked essence and extensive parts. Extensive parts are composed by
physical real parts linked with essence, which are managed by agency. Extensive parts compose wider vision of
artwork body³. The essence of artwork is recognized as individual agent which is related to the soul and
the mind (Bootz; Laitano, 2013), they are embodied in sensitive bodies. My body (and the other bodies biological
and technological bodies of water) is a sensitive apparatus putting me in principles of global intuition of
creating transcorporeal matter in the life of “water body languages.” The
performative-material practice is based on the gathering and applying data of extensive parts of artwork in
the context of emotionally build interactive intermedia artwork. Fluid mechanics agents, as AI models, are a
managing agency that introduce characteristics of the water-based world of artwork. The process
relates to extensive parts of all bodies of water which relates to the manifestation and generate emotional
actualization of artwork water body. The reading artwork essence will be put in the middle of
analytical—informative and intuitive feeling. Although flow dynamics modelling—are used to predict
hydrological behaviour, it is not possible to see what will be if human destroys a tiny balance of water flow.
Through the project they are emotionally indrawn in the artistic message: 'We, as bodies of water, can interact
in the water body holobion, we are all equal in the emotional based communication possibility and feeling our
common human and more than human water-based life.
The experimental theoretical and practical transdisciplinary ArtSci research will be build according to the
proposal of Manifesto of Artistic Research⁴ (Henke, Merch, and col, 2020) (farther
“Manifesto”) The goal of the project is to introduce the audience to data- manifestation through
artistic emotional imagination, away from pure data vizualization, to take audience closer to feel water-based
life of equal interaction-communication with other bodies of water. The emotion relates to water-medium
communication of interrelated water bodies. Applying a more intuitive concept could bring emotional
understanding to each body of water and to engage audience to the environmental action.
The problem water crisis is often resolved from an anthropogenic point of view which leads to the other complex
of global environmental problems. By implementation of symbioses, neutral or parasitical relations of sympoietic
system only we will be able understand how systems communicate in the water body medium interface. Using the
analytical natural sciences approaches and human intuition and sensitive feeling will lead to the critical
posthuman ArtSci overview, where chosen agencies could metaphorically relate to cognition and behaviour of
possible communicative process.
The generative artistic approach based on data and performative emotional interaction lead to specific
experimental aesthetical artsci solution and its realization in the artwork. Only clear, unpolluted water is a
medium of understanding water based live. The approach based on the relation of water body figuration to the
agency of unstable patterned interspecies (also interplanetary) communication. According to Deleuze, Haraway,
Neimanis, and other authors the water body figuration is fluid and actual as a characterization of a real;
fluidity, both virtual and actual, can be real aspect of water-based communication. I can see Water as a medium
for bearing the information of life also interspecies communication. Water is matter which enables information
flow from each water-based body to others alongside the common water network. Water is “informable
interactive matter”.
Conclusion
Water is an essence and by water figures relates to the individual water body agency. The interrelation of
water body agency and agency of patterned cognition and communication is possible to find
“languages” of water-based communication. The essence of water based interspecies communication
artwork relates to the water`s essence by its extensive parts and by relations are linked to the agency of water
body figuration. Agency of figures of the water body are related to the agency of interspecies
interaction/communication. The agency comes from the principle of "structural coupling", the principle of the
sympoetic system and the notion of water. Interspecies communication was related to the water interface as a
common medium for all living beings. Water is recognized as an interface, a part of bodies and as an organising
agency of communicative form. Then water is the essence of the interface of communication and figuration of the
water body, and its action is related to the pattern of waterbased communication. The approach leads to
manifestation of possible interspecies communication recorded by the ArtSci project. The understanding of
artwork is related to the audience reading and will move from an analytical to emotional approach. The goal is
to lead the audience closer to an emotional level of water-based communication and go away from the
informational position of data.
The project is tracing possibility how we can understand to water related life on the planet and possible
interspecies communication. To take audience interest of water problems, I tried to ideologically connect them
to the communication with other water bodies. In this context it means more understand, more connected, care,
and to have better environmental justice to all water bodies. The research leads to the importance of intuition
and serendipity in ArtSci's holistic approach to communicating water-based equanimity. This approach would
instill in the audience (professional and general) a better understanding of water not only as a resource, but
also as the inevitability of interconnected life and the necessity of equal access to unpolluted water, to
quality habitats, to a quality and healthy environment for all.
References
1 The first essay presented in this chapter, by Stephen Perrella, is a précis of
end-of-millennium design theory: the Objectile is an open-ended notation which allows for infinite
parametric variations; these can be directly fabricated using file-to- factory technologies, thus enabling
the serial reproduction of non-identical parts, where ranges of limited variations can be mass produced at
no extra cost (Mario Carpo, Stephen Perrella,Bernard Cache 27 March 2013
https://doi.org/10.1002/9781118795811.ch10 in Topological Architecture (1998–2003 ). “Objectile is
an object which is not yet defined by its essential form, but becomes the pure functionality “...
(Deleuze, 1988). and open it to artistic research of digital form. Fluidity as a principle of shaping
architectural form was followed in nonrational geometries of interrelated architectural space and environment
and the other circumstances or parameters (Last,2015).
2 See procedural model of communication for example online
https://mediarep.org/handle/doc/18625
3Understanding to essence is put by Bootz
somewhere in the middle of intuition affected by the reading body of work and analytical approach to body of
work. If the audience is closer to the analytical reading of work they are farther away from emotional
feeling and vice versa (Bootz, 2010)
4 As was declare in Manifesro of artistic research Artistic research is based on more
intuitive and interactive approaches—‘in the form of leaps, digressions, and detours which
continually generate new and unexpected counter/expressions’, ...and trigger irritations and daring
revelations. (Henke, Mersch, and col., 2020)
Bibliography
V. R. Baker, Debates—Hypothesis testing in hydrology: Pursuing certainty versus
pursuinguberty, Water Resour, Res.,53, 2017, 1770– 1778, doi:10.1002/2016WR020078.
K. Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning,
Durham, Duke University Press, 2007.
K. Barad, "Living in a Posthumanist Materialist World: Lessons from Schroedinger’s Cat", in A. Smelik
and N. Lykke (eds), Bits of Life: Feminism at the Intersections of Media, Bioscience and Technology, 2008,
165-176.
S Beer, University of Washington Press, Bootz, Seattle, 1980.
R. Maturana Humberto, Varela Francisco J., Preface to “Autopoiesis: The Organization of the
Living”, in Autopoiesis and Cognition, The Realization of the Living, Dordrecht, Holland/Boston, D.
Reidel Publishing, 1980.
P. Bootz, M. I. Laitano, Crossreading New Media, Online ELO conference 2013, videné Feb., 2019,
https://doi.org/10.7273/2G71-Q352.
Philippe Bootz, "The Problematic Of Form Transitoire Observable, A Laboratory For Emergent Program med Art",
videné Dec. 2018, Online: www.dichtung-digital.com/2005/1/Bootz.
Philippe Bootz, Sandy Baldwin, Regards Croisés, Perspectives on Digital Literature, West Virginia
University Press, 2010.
R. Braidotti, The Posthuman, Cambridge: Polity Press.Deleuze G, Difference et repetition, Press
Universitaires de France, Paris, 2013.
G. Deleuze, Logika smyslu Karolinum, Praha (original, 1969) pursuinguberty, Water Resour. Res.,53,
2013, 1770–1778, doi:10.1002/2016WR020078.
G. Deleuze & F. Guattari, (1980), Tisíc plošín. Herrmann & synové,
preklad, Praha, 2010.
G. Deleuze, (2016), Spinoza, Praktická filozofie, Óikumené, Praha Drohan Ch.r M,
Atropos press, Deleuze and the Sign, 2009, ISBN-10 : 0981997201
L. Guertin, 2016, online, https://www.paesta.psu.edu/podcast/did-
earths-water-come-outer-space-paesta-podcast-series-episode-17
E. Grosz, The Nick of Time, Durham, Duke University Press, 2004.
monahedayati@yahoo.com Belgium Concordia University, University of Antwerp
Abstract
This paper is an effort to examine the codes of interaction between the carbon-based and the silicon-based,
i.e., the human and the machine, notably the shifting agencies addressed by adopting feminist technoscientific
and new materialist lenses to grapple with the techno-industrial paradigm shift that has been (dis)figuring the
anthropocentric condition. The first part of the paper lays down the qualities of this emerging ecology while
recognizing the importance of human accountability and situatedness. The focal point of this survey is the
anthropologist Lucy Suchman’s classic Human-Machine Reconfigurations which is elaborated upon
through anchor points she posits revisiting Donna Haraway and Karan Barad’s arguments. The last part
engages with the implications of such a coupling for human and machine sensoria in order to envisage the
qualities of a distributive sensorium that this regenerative agency can put forth while alluding to practices of
situated computing.
If another techno-industrial paradigm shift has indeed creeped up on us so as to perfectly confuse the
human-machine boundaries this time around; and that an ontological shift has occurred, as Rosi Braidotti claims,
troubling the contact zones “between the organic and the inorganic, the born and the manufactured, flesh
and metal, electronic circuits and organic nervous systems”; and that the carbon-based and the
silicon-based as a result constantly imbricate, move, and flux inexorably, how can we imagine the agencies
constructed around such a flow? What are the implications of such an entanglement for human-machine
sensoria?¹
This co-evolutionary moment through which human and machine complex systems constantly affect and are affected
by one another’s interrelation to create an unfolding terrain of imbricated becoming is referred to as
technogenesis by N. Katherine Hayles. The implications of this adaptable and generative interrelation
are profound: not only we, as humans invested in the holiness of humanism, have to grapple with constant
adaptation of culturally coded networks of human life, but also with the psychobiological shifts that have
occurred notably in rewiring of intricate neuronal activities of the human brain.²
Detached from the preformulated subject-object binary, the anthropologist Lucy Suchman’s classic
Human-Machine Reconfigurations offers a profound angle on theorizing the conditions of this
human-machine distributive agency. To delve into reconfiguration as an emergent ecology where agencies are
constantly made, unmade and remade, at various points Suchman draws on the scholarship of Donna Haraway as well
as Karen Barad, notably through the notions of figuration and intra-action.
Haraway’s Figuration: A Situated Construct
Through the concept of figuration Haraway intends to foreground the tropic quality of material-semiotic
practices in technoscience that hover in a space of literal-figurativeness. She envisages technologies as
materialized figuration; that is assemblages that are both concerned with meaning making, a figural act, and
physical and hence tangible existence.³
As a lens that particularly zooms in on human-machine interrelation, figuration is a critical framework that
questions the formulation and configuration of technoscientific practices at every instance of occurrence. The
goal is to sidestep fixed universalized paradigms of ‘doing science and technology’ and aim for
specificity of local practices that humans actively shape rather than act as a passive observer within.
According to Suchman, the act of figuration is informed by specific socio-cultural constructs arising from
site-specificity which can reinscribe or challenge the status quo and question the Euro-American imaginaries
built on rationality of the autonomous subject.⁴
Barad’s Intra-Action: Entities in the Making
Barad’s notion of intra-action is based upon their theory of agential realism as an onto-epistemology
that challenges individualist paradigms and insists that intra-acting agencies are always already
inseparable.⁵
While during an interaction two preformulated entities come together for an exchange, an intra-action
underscores how the subjecthood and objecthood gets formed through the encounter. Barad specifically considers
technoscientific practices to be a common site of intra-action where we should recognize the act of boundary
making, objectification, and subjectification as contingent constructs. Barad’s vision is markedly in line
with Haraway’s material-semiotic that considers material constructs and the meaning arising from them as
co-constitutive. They too consider the reality of human-machine boundaries to be cut in particular ways that
follow certain historicity with socio-political consequences.
Suchman’s Reconfiguration
Drawing on the notion of figuration, configuration and intra-action, Suchman proposes reconfiguration as a
creative exploration of human-machine boundaries where what she refers to as “the distributed and enacted
character of agency” as a constantly-regenerative phenomenon should be taken into account.⁶
This view sits against the Western-dominant vision of subjects and objects as fixed entities brought together
to interact and instead points towards a kind of performativity within the encounter where the agents are in
continuous formation, reproduction and transformation; a perspective rooted in Actor Network theory (ANT) as a
social theory based on relational ontology that puts humans and nonhumans alike as actants in an
ever-evolving interrelated network where there are no preconceived positions taken but the positions are rather
assumed through the process of interrelation. In this process the agencies are constantly worked through and
negotiated to actively constitute ontologies based on what Michel Callon (2007), one of the proponents of ANT,
calls “morphology of the relations” through which cyborgs, hybrids and quasi-objects are constructed
and made visible.7(1)
Practices of technology-mediated medicine, including reproductive technoscience, as well as human-computer
interaction are those that Suchman pays particular attention to as sites of human-machine mutually-constituted
agencies. In this space of intra-action the disconcerting fact is that within practices of science and
technology, the technical is formulated in the center while the social is either non-existent or pushed to the
margins. Here, Suchman walks a tightrope of reconceptualizing the human in a way that the inseparability from
the socio-technical substrate is pushed to the fore while recognizing the prominence of accountability but
without assuming the dominion associated with ‘pure’ humanism that views technologies as translators
and assemblers in service of humans. In other words, the question is how to draw a human-machine intra-action
that retains human accountability without telling an essentialist story.
As a feminist construct, the figure of the cyborg, taken across its regenerative stance, can offer one avenue
to explore this notion by radicalizing the human-machine, male-female, and subject-object boundaries, towards an
emergent ecology where socio-materiality is constantly made, unmade and remade. From the Harawayan
goddess-turned-cyborg to the elegant hero/ine and saviour as cultural imaginary and further as an everyday
socio-materiality without a singular body, the figure of the cyborg is omnipresent across the socio-technical
substrate.⁹
By overstepping the isolated shell that contains the human-machine hybrid, in Suchman’s reading, cyborg
not only shatters the glamorized singular figure but “dissolves into a field of complex sociomaterial
assemblages” to open up new ways of theorizing and practicing such an entanglement.¹⁰
Braidotti, takes this destabilization one step further to put forth the figure of the deglamorized everyday
cyborg as “anonymous masses of the underpaid, digital proletariat who fuel the technology-driven global
economy without ever accessing it themselves.”¹¹
Intrinsically, we can see how at every moment of instantiation cyborg cuts the human-machine boundary at a
certain angle and not the other to constitute a shapeshifting intertwinement of the carbon-based and the
silicon-based capable of subverting human-made socio-politico-cultural constructs. This fluid ecology is meant
to transcend deeply etched preconceived notions of intelligent machines as human techno-extensions or the
sensing and sensible human weary of the techno-dystopia, to instead reconceptualize an entanglement among
networked agents, that constitute leaky, generative boundaries.
The stability of the human agency is thereby compromised as according to Suchman “the person figured here
is not an autonomous, rational actor but an unfolding, shifting biography of culturally and materially specific
experiences, relations, and possibilities inflected by each next encounter-including the most normative and
familiar—in uniquely particular ways.”¹²
Within these spaces of constant transformation of the boundaries and redrawing the agencies, the question of
accountability is by no means diffused: we need to recognize that we draw boundaries for meaning making and
these boundaries are always charged with human-centered conceptions and misconceptions that have repercussions.
This accountability is to recognize our position within animation and reanimation of situated encounters. As
Barad states: “we are responsible for the world in which we live, not because it is an arbitrary
construction of our choosing, but because it is sedimented out of particular practices that we have a role in
shaping.”¹³
(Re)(Con)Figuring Hybrids of Sense-Making
During the act of boundary making the question that arises is how can we cut the boundaries in ways that give
rise to hybrids of sense making where intelligibilities and sensibilities are constantly figured, configured and
reconfigured? Reconfiguration in this sense can be read as a possibility to negotiate sensory modalities to
locate sensing and effecting (2) not as autonomous qualities associated with
the human or the machine but traceable within the process of intra-action. If technical practices foreground
machine agency and yet human sensoria and sensibility cannot be reduced to compressed temporal flow of machinic
computation, we need to reckon with contingent encounters that go beyond biological and technological
determinism. Bio-sensibility and machine intelligence in this sense can be deconstructed and diffused to be
reconfigured as dynamic fragments of a curious intelligent sensibility adjustable towards acting in
contingent and context-specific situations.
Such a hybrid, quasi-sensoria made in between the human and the machine offers a heterogeneous
socio-materiality with qualities that are no longer inherent but always negotiated and in formation. This
ontological in-betweenness tends to question universalized presumptions about technical practices centered
around the aptitude of the machine intelligence to offer reliable, definitive, and objective responses to
complex questions that are always rooted in specific situations.
In this sense as Haraway notes the answer does not lie in the dichotomous poles of positivism with its hallmark
of scientific objectivity or relativism with its absolute unfixed orientation but in localized and embodied
partial perspectives.¹⁴
Such a vision relies on the symbiosis of the technical and the sensible to form an interdependent intelligent
sensibility that takes shape within the act of becoming to reckon with procedural, socio-political, and cultural
dimensions of local conditions to compose and assess situated work-flows and responses.
(1) While Suchman praises the idea of “generalized symmetry” proposed by ANT, she
subsequently proposed “dissymmetry” as a framework that recognizes the human-machine
differences. Others including Bowker and Leigh Star, put forth a number of critiques pointed towards
ANT’s networked interrelations. They consider the larger social construct to question the equalizing
effect of such a framework toward human/nonhuman actants and the ethico-political repercussions of a world
operating on this logic, pointing out that ANT "can be read as an uncritical celebration of
the power of modern science and technology.” ⁸ Elsewhere,
others such as Mel Chen, Zakiyyah Jackson, and Tiffany King, among others contend that posthumanist
theories in general discount the human discrimination factors at play due to micro and material nature
of such inquires.
(2) Sensing and effecting are mechanisms of interaction within biological and technical
organisms; while sensors receive information from the environment to relay to the system, effectors act
upon the world based on feedback loops that occur between the two. These notions are derived from
cybernetics, a field of inquiry that studies the principles of communication and control within regulatory
systems.
References
1 Rosi Braidotti, The Posthuman, Cambridge, UK, Malden, MA, USA, Polity Press, 2013,
p.89.
2 N. Katherine Hayles, How We Think: Digital Media and Contemporary Technogenesis,
Chicago, London, The University of Chicago Press, 2012.
3 Donna Jeanne Haraway, Mdest_Witness @Second_Millennium. Female-Man_Meets_OncoMouse: Feminism
and Technoscience, Second edition, New York, NY, Routledge, Taylor & Francis Group, 2018.
4 Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, 2nd
ed, Cambridge, New York, Cambridge University Press, 2007.
5 Karen Michelle Barad, “Posthumanist Performativity: Toward an Understanding of How Matter
Comes to Matter,” Signs: Journal of Women in Culture and Society 28, no. 3, March
2003, 801–31, https://doi.org/10.1086/345321.
6 Suchman, Human-Machine Reconfigurations, 260.
7 Michel Callon, “Actor-Network Theory, the Market Test,” in Technoscience: The
Politics of Interventions, ed. Kristin Asdal, Brita Brenna, Ingunn Moser, Oslo?, Unipub, 2007.
8 Geoffrey C. Bowker, Susan Leigh Star, “How things (actor-net) work: Classification, magic
and the ubiquity of standards,” 1996, https://www.ics.uci.edu/~gbowker/actnet.html.
9 Donna Jeanne Haraway, “A Manifesto for Cyborgs: Science, Technology, and Socialist
Feminism in the 1980s,” in The Post-modern Turn: New Perspectives on Social Theory, ed. Steven
Seidman, Cambridge, New York, Cambridge University Press, 1994.
10 Suchman, 283.
11 Braidotti, The Posthuman, 90.
12 Suchman, 281.
13 Karen Michelle Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning (Durham: Duke University Press, 2007), 105.
14 Donna Jeanne Haraway, “Situated Knowledges: The Science Question in Feminism and the
Privilege of Partial Perspective,” Feminist Studies 14, no. 3, 1988, 575,
https://doi.org/10.2307/3178066.
Bibliography
Marie-Luise Angerer, NONCONSCIOUS: On the Affective Synching of Mind and Machine, S.l.:
MESON PRESS EG, 2022.
Karen Michelle, Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter
and Meaning, Durham, Duke University Press, 2007.
Karen Michelle Barad, “Posthumanist Performativity: Toward an Understanding of How Matter Comes to
Matter”, Signs: Journal of Women in Culture and Society 28, no. 3, March 2003, 801– 31,
https://doi.org/10.1086/345321.
Geoffrey C Bowker, Leigh Star Susan, “How things (actor-net) work: Classification, magic and the
ubiquity of standards,” 1996, https://www.ics.uci.edu/~gbowker/actnet.html
Rosi Braidotti, The Posthuman, Cambridge, UK, Malden, MA, USA, Polity Press, 2013.
Michel Callon, “Actor-Network Theory, the Market Test.” In Technoscience: The Politics of
Interventions, edited by Kristin Asdal, Brita Brenna, Ingunn Moser. Oslo? Unipub, 2007.
Mel Y Chen, Animacies: Biopolitics, Racial Mattering, and Queer Affect, Perverse Modernities,
Durham, NC, Duke University Press, 2012.
Paul Dourish, Where the Action Is: The Foundations of Embodied Interaction, Cambridge, Mass,
MIT Press, 2001.
Donna Jeanne Haraway, Modest_Witness@Second_Millennium. FemaleMan_Meets_OncoMouse: Feminism and
Technoscience. Second edition. New York, NY, Routledge, Taylor & Francis Group, 2018.
Donna Jeanne Haraway, “A Manifesto for Cyborgs: Science, Technology, and Socialist Feminism in the
1980s”, In The Post-modern Turn: New Perspectives on Social Theory, edited by Steven Seidman,
Cambridge, New York: Cambridge University Press, 1994.
Donna Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of Partial
Perspective", Feminist Studies 14, no. 3 1988, p.575, https://doi.org/10.2307/3178066.
N. Katherine Hayles, Unthought: The Power of the Cognitive Nonconscious, Chicago, London: The
University of Chicago Press, 2017.
N. Katherine Hayles, How We Think: Digital Media and Contemporary Technogenesis, Chicago, London:
The University of Chicago Press, 2012.
N. Katherine Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and
Informatics, Chicago, Ill: University of Chicago Press, 1999.
Zakiyyah Iman Jackson, “Animal: New Directions in the Theorization of Race and Posthumanism”,
Feminist Studies 39, no. 3, 2013, 669–85, https://doi.org/10.1353/fem.2013.0024.
Tiffany Lethabo King, “Humans Involved: Lurking in the Lines of Posthumanist Flight”,
Critical Ethnic Studies 3, no. 1, 2017, 162–185,
https://doi.org/10.5749/jcritethnstud.3.1.0162.
Bruno Latour, Reassembling the Social: An Introduction to Actor-Network-Theory, Clarendon Lectures
in Management Studies, Oxford, New York, Oxford University Press, 2005.
Andrew Pickering, The Cybernetic Brain: Sketches of Another Future, Chicago, London, University of
Chicago Press, 2010.
Lucille Alice Suchman, “Agencies in Technology Design: Feminist Reconfigurations*, ”In
Machine Ethics and Robot Ethics, by Wendell Wallach, Peter Asaro, edited by Wendell Wallach and Peter
Asaro, 1st ed. Routledge, 2020, 361–75, https://doi.org/10.4324/9781003074991-32.
Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, 2nd ed.
Cambridge, New York, Cambridge University Press, 2007.
DAOs A blockchain-based application not intervening, but strengthening the agility of contemporary arts
VictoriaHilsberg
vic.hilsberg@hotmail.de
Abstract
This paper discusses the blockchain-based application DAO (Decentralized Autonomous Organization) as a relevant
new mechanism of artistic practice in the contemporary arts. It not only confronts the complexity of the modern
society through its comprehensive application potential. It also channels segments of the contemporary art
discourse such as political qualities, participatory art and cybernetic forces through active use cases. DAOs
enable the interconnection of culture, technology and ecology. Using the most recent publications regarding
blockchain technology as well as in-depth research on current artistic movements, it is possible to observe a
complex manifestation of the matter within the contemporary arts during the concept's comparatively brief
existence since its technical implementation around 2016.
Keywords
Contemporary art, transdisciplinary discourse, blockchain, DAO, Crypto Art, political art history, technology,
participation, ecology, economics, Digital Art.
DAOs, Decentralized Autonomous Organization, are blockchain-based applications. They represent a community or
network of potentially anonymous individuals and/ or institutions, which are connected through a common factor.
Each DAO is collectively owned through a network and does not have a central controlling authority.
¹Governance, coordination and actions take place within defined, self-governing rules, which are
collectively decided upon and in its basics laid out in the White Paper, which is accepted by joining the DAO
network. There is the possibility of implementing a DAO-own cryptocurrency, which can function as a treasury.
One option to govern the organization is the buy-in: Through obtaining DAO-own tokens, a fractionalized
ownership of the DAO is held. In theory, owning all tokens represents ownership of the entity of the DAO. There
must not be this structure in order to provide a functioning DAO as there a several differing technological
implementations already in place.²
Examples of DAOs would be exhibited 2022 alone in renowned exhibition spaces such as Kunstwerke Berlin or
Documenta 15 in Kassel. Why would there be the interest to exhibit this technical development that is so close
to Cryptocurrencies in their early stages of the Hype Cycle?³
Because the one challenge that contemporary art is confronted with is to develop new techniques and therefore a
complexity which matches the one society itself is confronted with. DAOs are opening up different options.
One aspect making DAOs relevant to contemporary art is to define and demonstrate the importance of structure as
a political element. DAOs provide a possibility of testing out agile, seemingly utopian systems and implementing
them long term. One example is the Black Swan DAO, founded by Penny Rafferty, Laura Lotti, Calum Bowden and
others. It caters goals, which were already formulated by art activism groups such as the Art Workers Coalition
in New York in the 1960s. Their intention was to build a platform for the representation of all art sector
associated employees, including artists, and demanded their equal treatment and co-determination.
⁴However, through the missing long-term structure it dissolved over time. The approach of Black Swan DAO
towards equal treatment is built upon blockchain-specific protocol mechanisms working towards a trusted network
within contemporary arts. One of their actions was to act as a platform for democratized cultural funding
processes. Equal treatment symbolizes for Black Swan DAO also equal action rights. Pressing de-hierarchization,
the DAO members decide through a democratic voting mechanism who receives the funding. The decision process does
not focus on evaluating the individual background, but the quality of the project. Additionally, Black Swan DAO
developed through its practice a consensus-building tool for collectives, called Cygnet. ⁵With consistent
optimization, a DAO can be used to organize a complex ongoing structure on an ongoing basis, as in a company, or
foundation-like groupings. The Blockchain expert Shermin V oshmgir compares the function of DAOs with the
control and regulation system of state governance layers, on the one hand, and with that of private companies,
on the other.
She does not connect its functionality to a recognized state. Voshmgir sees the automated DAO system with its
high, democratized adaptation factor as a structural improvement over the rigid forms used to date. Her main
argument here is that digitally secured compliance will radically end hierarchization. All the more so because a
DAO can be adapted from within the structure to the respective needs of the parties involved. ⁶This would
imply that Black Swan DAO could aim successfully to realize missing or not from their perspective successfully
developed elements of the structural support system, regarding state operations and possibly particularly the
support system of the arts.
Through their grassroots democratic ideal, DAOs do face anti-democratic conflicts, despite the complex
technical implementation: Misinformation, bribery, plutocracy or media control. These are, among others, factors
that can manipulatively influence the stakeholders of a DAO in their decision-making during the consensus
process. Corresponding counter-mechanisms are being developed, and partial successes are already in sight.
⁷To develop critical awareness towards misinformation therefore needs to be addressed inside the
structures of DAOs. As shown with the Black Swan DAO or as discussed by Claire Bishop in Artificial Hells
outside of all blockchain intersections, DAOs represent a key element of contemporary arts only strengthened
within the last years and especially since the Covid-19 pandemic: Participation and awareness of collective
practice. ⁸An example is to be found also within the BeeDAO, presented 2022 at Documenta 15. BeeDAO is not
only metaphorically oriented on the swarm organization of a beehive. It works towards the direct or indirect
interest of the alive bees. A physical manifestation takes place through monitored bee hives as direct
representatives of the species which is to be protected, producing public data. On physical assemblies,
proposals towards the shared mission are discussed and decided upon. Using blockchain technology, this
governance process cannot be manipulated through human agency, but provides a technological implemented
liability. ⁹The phrase swarm highlights the participatory practice of institutions and organizations,
referring also to the distinct participatory quality of Digital Art.
This also refers to the third element which highlights the quality of the DAOs. It can not only be seen as a
representation of humanly structured systems. It also can actively develop the idea and styles of
interconnection of planetary elements in anthropocentric times. Terra0 DAO was implemented in 2016 by Paul
Seidler, Paul Kolling and Max Hampshire, the terra0 collective. The DAO represents a forest.
The physical existent piece of land is obtained through initial funding by DAO members. Surplus made from
obtaining the forest, for example by selling the material wood, will be automatically used to buy out the share
of the initial investors. The goal of this project is to develop as much surplus through automated economical
processes enabled with mostly blockchain-based processes, that the forest will be able to buy out all human
investors and govern itself through automated Smart Contracts. ¹⁰This artistic practice scales back
to the cybernetic considerations of the symbioses of men and machine, as for example explained by Joseph C. R.
Licklider, but puts the environmental aspect of this interaction into the forefront. ¹¹Therefore it
develops this interaction further than before discussed, enabled through blockchain technology. It, moreover,
reminds the however involved human parties of two elements, which reoccur in contemporary artistic practice: the
power of the individual is perceptibly recognized. With the metaphorical proposition, that even the smallest
amounts of capital can play a relevant part in the overall economy, an emancipatory practice takes place. This
is underlined through the implementation of democratic mechanisms such as majority voting. Secondly, that the
interconnection of technology, ecology and culture is not a futuristic matter, but is happening in the current
times. Elements such as worker emancipation, trees or bees are rather positively connoted and it might not be a
coincidence that for the first projects operated within the system DAO and contemporary art do use those
accustomed elements in order to embrace engagement. Vitalik Buterin framed a reference system for organizational
structures of DAOs in 2014. The technological implementation took place with The DAO two years later.
¹²Seeing the implementation now already in place in contemporary arts, there is more than a potential
to be observed. Active use cases can be seen in DAOs as a method in contemporary artistic practice. All
discussed DAOs still rely on the physical involvement of members in order to function. There however exists a
structure which can overtime be strengthened and, in the process, develop the participatory aspect of the
collective works. How this will develop is not certain at this point, as little as the future of blockchain is
at this point determined. DAOs however already do confront contemporary conflicts with engaging solutions.
References
1 Shermin Voshmgir, Token Economy: How Blockchains and Smart Contracts Revolutionize the
Economy, Berlin, Blockchain-Hub, 2019, 104 et sqq.
2 Voshmgir, 2019, 119.
3 Vgl. Kathrin Passig, "Neue Technologien, alte Reflexe: Beitrag aus dem Sonderheft 2014 zum Thema
Medienevolution," accessed November 27, 2022,
https://www.medienkorre-spondenz.de/leitartikel/artikel/neue-technologien-altenbspreflexe.html.
4 Lucy R.Lippard, Sixyears: The dematerialization of the art object from 1966 to
1972, Berkeley, Univ. of California Press, 2007, p.IX.
5 Black Swan, “Cygnet Prototype,” in Radical Friends, Decentralised Autonomous
Organisations and the Arts, ed, Ruth Catlow et al.,Torque Editions, 2022, 322-323.
6 Voshmgir, 2019, 104-135.
7 Voshmgir, 2019, 130 et sqq.
8 Claire Bishop, Artificial Hells: Participatory art and the politics of spectatorship,
London, Verso, 2012.
9 "Bee DAO – the inaugural assemblee," Documenta 15, Website, accessed November 18, 2022,
https://documenta-fifteen.de/kalender/beedao-the-inaugural-assemblee/. See also:
https://beedao.zku-berlin.org/
10 Paul Seidler, Paul Kolling, Max Hempshire, “terra0. Can an Augmented Forest Own and
Utilize Itself?,” in Artistic Re:Thinking of the Blockchain, ed. Ruth Catlow et al., Torque
Editions, 2017, 68-72.
11 Joseph C. R. Licklider, "Man-Computer Symbiosis,“ IRE Transactions on Human Factors
in Electronics, HFE-1, 1960, 4-11.
12 Vitalik Buterin, "DAOs, DACs, DAs and More: An Incomplete Terminology Guide,“
Ethereum Foundation Blog, accessed November 5, 2022,
https://blog.ethereum.org/2014/05/06/daos-dacs-das-and-more-an-incomplete-terminology-guide
Bibliography
Claire Bishop, Artificial Hells: Participatory art and the politics of spectatorship, London,
Verso, 2012.
Swan Black, “Cygnet Prototype”, In Radical Friends. Decentralised Autonomous Organisations
and the Arts, edited by Catlow, Ruth et al., Torque Editions, 2022, 322-323.
Vitalik Buterin, "DAOs, DACs, DAs and More: An Incomplete Terminology Guide,“ Ethereum Foundation
Blog, Last accessed, 05.11.2022,
https://blog.ethereum.org/2014/05/06/daos-dacs-das-and-more-an-incomplete-terminology-guide.
Documenta fifteen, "BeeDAO – the inaugural assemblee,“ Last accessed 18.11.2022,
https://documenta-fifteen.de/kalender/ beedao-the-inaugural-assemblee/.
Joseph C. R. Licklider, "Man-Computer Symbiosis,“ IRE Transactions on Human Factors in Electronics,
HFE-1, 1960, 4–11.
Lucy R. Lippard, Six years: The dematerialization of the art object from 1966 to 1972,
Berkeley, Univ. of California Press, 2007.
Kathrin Passig, "Neue Technologien, alte Reflexe: Beitrag aus dem Sonderheft 2014 zum Thema,
Medienevolution,“ Last accessed 04.11.2022,
https://www.medienkorrespondenz.de/leitartikel/artikel/neue-technologien-altenbspreflexe.html.
Paul Seidler, Paul Kolling, Max Hempshire, “terra0. Can an Augmented Forest Own and Utilize
Itself?” In Artistic Re:Thinking of the Blockchain, edited by Catlow, Ruth et al.,
322-323,Torque Editions, 2017, 63-72.
Shermin Voshmgir, Token Economy: How Blockchains and Smart Contracts Revolutionize the Economy,
Berlin, BlockchainHub, 2019.
Lucid Dream: Sensing and Artistic Representation of Plant-Nature Interaction Based on Plants Biosignals
YouyangHu‡, ChiaochiChou*,
YasuakiKakehi‡
‡ The University of Tokyo * National Tsing Hua University Tokyo, Japan Hsinchu,
Taiwan 1huyouyang@xlab.iii.u-tokyo.ac.jp,
2chiaochi@gapp.nthu.edu.tw, 3kakehi@iii.u-tokyo.ac.jp
Abstract
The recent progress of modern science has enabled us to detect the physiological status of organisms through
their bioelectric activity; this technique has been constantly applied to contemporary media art. However,
exploring plants as a subject and allowing them to interpolate and hence perform art through biosignals remains
to be further explored. Lucid Dream is an artwork in which inspection, displacement, and engagement of plant
communities take place, shedding light on the subjective perception of plants, a peripheral subject that rarely
comes to people’s attention. To achieve this, we entered a forest and applied an artificial
intelligence-based learning system capable of interpreting the local plants' responses to wind and rain stimuli
via their biosignals. Subsequently, we established an environmentally controlled space within the art museum.
Here, we simulated and artistically represented the natural elements as perceived by the plants, using
artificial machines driven by the plant-nature interaction model. As viewers enter this space, they can
experience the plant-perceived natural environment, gaining a non-human perspective through direct engagement
with plant life. Lucid Dream not only leverages intelligent computational technologies to comprehend the
perceptual system of plants but also fosters cross-species sensory experiences, enhancing our understanding and
expanding our perspective on the natural environment.
In recent years, as we close the gap between technology and biological systems, artworks utilizing living and
semi-living organisms have become increasingly vigorous in contemporary media art. These artworks have presented
us with another perspective to understand different life forms. With the era of pan-biological materials of bio
art unfolding, artists and scientists who carried multiple identities began to displace the technique of
biosignal detection in art creations. A characteristic of displacement, which is Recontextualization, has
provided a propelling force for Lucid Dream is divided into the working phases of inspecting, displacing, and
engaging with plant communities. We first developed a bio-amplifier to collect biosignals from multiple plants
in a forest, we then the dialectic between artists acting due to their thought or their biological reactions to
stimulus¹, ². Afterward, with the exploration method of using biosignals to construct a new logic for
subjects or environments, people have further paid attention to other living organisms, especially algae and
microbes³, ⁴, ⁵.
In these artworks, non-human lives are presented as lively, having their own “umwelts”⁶.
However, to answer the upcoming question of whether living organisms have a more complex subjective activity and
how to represent it to viewers with more direct and persuasive evidence, we believe that plants can be an
answer. Due to their natural ability to perceive, adapt, and even evolve a particular method to respond to
climate change, plants demonstrate high sensitivity and easily observed feedback to the environment⁷. As a
result, it is of great potential for us to utilize intelligent computational technologies to create systems that
interface with the "umwelts" of plants in nature in such a way as to extend our understanding and broaden our
horizons regarding the natural environment.
Lucid Dream is divided into the working phases of inspecting, displacing, and engaging with plant
communities. We first developed a bio-amplifier to collect biosignals from multiple plants in a forest, we then
applied an artificial intelligence-based learning system that can interpret the plants' perception of the wind
and rain stimuli through their biosignals. The natural environment sensed by plants is displaced to the
environmental-controlled space in another realm to be artistically represented. After being
“Recontextualized,” the environmental data originally subject to scientific methods have been
transformed and enabled a new meaning. When viewers walk into this space, they can experience the natural
environment that plants perceive through their engagement with plants’ life from a non-human perspective.
Extending from art to science, breaking the existing framework of scientific research, and connecting
trans-species sensory experiences to provide a new possible way to respond to the question of representing the
subjective activity of living organisms to viewers.
Biosignals Detection and Analysis
In this project, we focus on the Mimosa in the forest, known for its high sensitivity to environmental stimuli
and commonly used in bioelectric signaling research⁸. We developed a miniature bio-amplifier to detect its
biosignals in response to environmental stimuli. It detects the difference in electric potential between the
substrate and the leaf petiole. As shown in Figure 2, it consists of two integrated amplifiers (TLV27L2 and
ADS1292). Due to the uncertainty of plant resistance value and the electromagnetic interference caused by other
equipment, most of the plant signaling tasks are carried out in the Faraday cage. We overcame this constraint by
using two high input impedance operational amplifiers (TLV27L2) to get less interference and enable detecting
the biosignals from plants in the natural environment. To reduce the size and power consumption of the circuit,
we utilize the ADS1292 which integrated high-resolution analog-to-digital converter (ADC) with a built-in
programmable gain amplifier. We used an STM32 microprocessor to pack the raw data and an ESP32 to transmit the
data to the computer wirelessly based on TCP protocol. On the computer side, there is a program developed in
openFrameworks that receives and processes the biosignals.
We next applied two supervised learning algorithms to conduct a time-domain analysis of the biosignals to get
the Mimosa's status under different environmental stimuli. Figure 3 shows the workflow of this system, A support
vector machine(SVM) classifier was applied to discriminate between wind versus rain-stimulated biosignal
activity. A support vector regression(SVR) model was then used to predict the wind intensity sensed by the
plant. We first reduced the noise of raw signals by a low-pass filter (second-order Butterworth filter) and then
collected the biosignals as 800 dimension vectors detected under two environmental stimuli. As shown in Figure
4, for each stimulus, 50 samples were collected and they were labeled "Wind stimulus" and "Rain stimulus."
We also collected 50 samples that the plant was under no stimulus. We then trained the SVM model that can
classify the status of plants under two environmental stimuli. We also collected the biosignals under three
intensities of wind stimulus. As shown in Figure 5, for each stimulus, 20 samples were collected and they were
labeled with integer values of 10, 127, and 255. We then trained the SVR model that can predict the wind
intensity as an integer value between 0 and 255 that the plants perceived in response to wind stimuli with
different intensities.
In this project, we have developed a plant-based system for environmental state sensing. Despite its inherent
biological variability, which can result in less precision compared to silicon-based sensors, this system holds
significant potential for future bio-hybrid sensing technology. Additionally, we are exploring its creative
applications in the realm of artistic practice, aiming to provide audiences with a novel perspective on their
interaction with the natural environment.
System Implementation and Artistic Representation
Based on our research in biosignals detection and analysis, we created an artwork that artistically represents
the natural environment perceived by plants in an artificial space. As shown in Figure 6, the artwork consists
of a plant observatory in the forest and an environmental-controlled space in the art museum. There are four
Mimosas equipped with bio-amplifiers in the plant observatory. The perceived state of each Mimosa in response to
environmental wind and rain stimuli will be analyzed by our artificial intelligence-based learning system and
remotely simulated in the environment-controlled space. The space is equipped with four sets of air-blowing
systems, each with three fans driven by the SVR models. The input is the biosignals of four Mimosas. The
intensity of the wind perceived by each Mimosa corresponds to the number of active fans in each air-blowing
system. A rain machine is installed in this space, which synchronizes the rain stimuli perceived by Mimosas
based on the SVM model. It consists of a water pipe running along the wall to the ceiling and a pumping motor.
When the system detects that Mimosas senses rain, it will trigger the rain machine to simulate the outdoor rain
environment in this space.
Figure 5 shows the exhibition site in the art museum, with a monitor real-time streaming the environmental
conditions in the forest and the biological state of the four mimosas. The natural environment sensed by plants
is displaced to the environmental-controlled space in another realm to be artistically represented. After being
“Recontextualized,” the environmental data originally subject to scientific methods have been
transformed and enabled a new meaning. When viewers walk into this space, they can experience the natural
environment that plants perceive through their engagement with plants’ life from a non-human perspective.
Conclusion
This paper introduces the artwork Lucid Dream, which uses plants as perceptual subjects to gain insight into
and artistically portray natural elements to viewers. To accomplish this, we have devised a bio-amplifier for
detecting plant biosignals in response to environmental stimuli. Subsequently, within a forest environment, we
have employed an artificial intelligence-based learning system capable of interpreting local plants' responses
to wind and rain stimuli through their biosignals. Additionally, we have established an environmentally
controlled setting to simulate and artistically represent the natural environmental elements as perceived by
these local plants to viewers. Lucid Dream not only delves into the utilization of advanced computational
technologies for understanding plant perceptual systems but also endeavors to create cross-species sensory
experiences, expanding our comprehension and perspective on the natural environment.
References
1 Alvin Lucier, Simon Douglas, Chambers, Scoresby Alvin Lucier, Wesleyan University Press, 2012.
2 David Rosenboom, "Method for producing sounds or light flashes with alpha brain waves for
artistic purposes," Leonardo, 1972, 141-145.
3 Carlos Castellanos, "IntersectionsofLivingandMachine Agencies: Possibilities for Creative AI,"
The Language of Creative AI, Springer, Cham, 2022, 155-166.
4 Sabina Hyoju Ahn, "Sonic transformation with living matter," In proceedings of ISEA symposium,
Inter-Society for the Electronic Arts, 2019.
5 You-Yang Hu, Chou Chiao-Chi, Li Chia-Wei, "Apercevoir: Bio Internet of Things Interactive
System," Proceedings of the 29th ACM International Conference on Multimedia, 2021.
6 Jakob von Uexküll, "An introduction to Umwelt," 2001, 107-110.
7 Jagadis Chandra Bose, The nervous mechanism of plants, Longmans Green, 1926.
8 Alexander G. Volkov, et al. "Mimosa pudica: electrical and mechanical stimulation of plant
movements." Plant, cell & environment 33.2, 2010, p.163-173.
Sounding Softness and the (Artificial) Subject
JonasJørgensen,
MadsBeringChristiansen
SDU Soft Robotics, SDU Biorobotics, The Mærsk Mc-Kinney Møller Institute,
University of Southern Denmark (SDU) Odense, Denmark jonj, mabc@mmmi.sdu.dk
Abstract
The paper discusses the authors’ artwork SONŌ, its artistic motivations, the artistic
research practice underlying its development, and its technical realization. SONŌ is a soft robotics
installation that interrogates the interconnections of soft materiality, sound, and subjectivity. It features a
sessile soft artificial entity capable of expansive movement, which is ceaselessly sounding itself and various
environments using real-time generated audio.
SONŌ is an artwork featuring a soft pneumatically actuated robot manufactured from silicone. The robot
possesses procedurally generated movement and sonification of movements accompanied by a soundscape.
Human and nonhuman animals make utterances that are socially communicative and function to enact a subject
position or form connections with other agents inhabiting the environment. The production of sound, whether
intentional or unintentional, is arguably a basic existential feat of all living organisms. As empirical
phenomena, however, sound by far predates life. In fact, the Universe emerged from what is arguably the ultimate
sonic event (which, paradoxically, no one was around to hear)—the Big Bang, cosmology tells us. Sound
always originates from a source yet is simultaneously transversal and expansive in character and by nature
destined to permeate its surroundings. It is a mediatic phenomenon par excellence—on the material level,
sound appears intangible and perhaps as almost nothing in itself, it only exists parasitic to matter,
manifesting as perturbations and pressure changes travelling in a physical medium. Sound is characterized by a
double movement—it is expansive and enveloping, yet simultaneously local and ephemeral. It is always
subject to dampening and seems to evaporate into thin air on the microscopic level, when its waves are converted
into heat through friction between the molecules of its medium.
Within the Western tradition of logocentric thinking, one of the ways in which sound comes to matter, is
through the privileging of speech over writing.¹ Speech is the primary medium of human thinking and writing
is merely a secondary technology. Hence, sound is positioned as the unbridled carrier of being and
subjectivity—“I sound, therefore I am.” Poststructuralists and their new materialist progeny,
however, champion a different position, that foregoes fixity, in favor of flux and the perpetual process of
becoming, which is perhaps more adequate to the ontology of sound itself. Here, the subject is considered
dynamic and decentered, and the boundaries between self and world permeable. Furthermore, agency is no longer
predicated upon subjectivity nor inherent to the subject itself, but a relational dynamism of forces enveloping
things as well as environments.²
Motivation and Practice
SONŌ (Latin: “[I make] sound”) explores a nexus of sound, soft robotics, and subjectivity.
Through the artwork and its associated practices, we seek to articulate and enact a myriad of complex
interactions between these phenomena and their aesthetic and epistemological capture.
SONŌ addresses what we take to be basic questions of robotic art in general, including, what does it take
to alter or blur the ontological status of an object towards that of a subject, by means of movement and sound?
As soft robotics (robotic morphologies and components constructed from pliable and elastic
materials³) is a key interest in our practice⁴-¹², within the work we were also keen to
query connections between soft materiality and sound through robotics as an aesthetic medium. For instance,
different kinds of matter are capable of producing impact sounds (via resonance) with specific characteristics
in terms of envelopes and frequency spectra¹³. Moreover, materials interact with sound in different
ways, e.g., soft materials tend to dampen sound whereas hard materials reflect it.
SONŌ is influenced by the notion of sound as a naturecultural¹⁴ phenomenon. It
seeks to consider divergent aspects of the material-semiotic conditions of possibility through which sound and
robotic movement can attain agency within specific environments. The physical and physiological properties of
softness and “soft sound” have thus fed into the work, but equally cultural meanings, e.g., notions
about sounds made by fictional soft characters from popular culture and cultural associations of softness as
aligned with, e.g., precariousness and vulnerability. From the outset, we were thus interested to probe the
chimeric character and synesthetic aspects of the concept of “softness”, when used to describe sound
and materials respectively. Definitions and delineations of “soft sound” within sound studies,
psychoacoustics, and musical theory, were, for instance, drawn upon in our explorations of what might constitute
“soft sound” and of the effects of adding “soft” or “hard” sound to a soft
morphology.
Part of the artistic research has been conducted in dialogue with the research field of human-robot
interaction (HRI), wherein sound has recently become subject of increasing interest. Our
practice sought to be receptive of pressing ideas and questions from this research field and consider how they
might gain relevance and be addressed through artistic forms. A body of work within the HRI field has
interrogated how various types of sounds can affect people’s perception and interaction with robots and
found non-verbal audio to be a salient feature with use potential as a deliberate design aspect of, e.g., social
robots. In certain situations and use cases, nonverbal audio is also preferable over synthetic voices, to guide
or facilitate interactions with humans.¹⁵ SONŌ adds to this research on robot sound, by
exploring how sound and embodiment can interact in soft social robots of unconventional nonanthropomorphic and
nonzoomorphic designs, which behaviors that should be accompanied with sound, and what the function of sound
might be within these.
The SONŌ Installation
The soft robot morphology was designed to appear organic yet unfamiliar (see Figs. 1-2). Abstract rounded
shapes and a hue with similarities to Caucasian human skin, or pig skin, with reddish spotted pigmentations were
used. The morphology possesses three independent pneumatic channels. Each of these interconnect four chambers
interspersed across it, which can expand when inflated. Ecoflex 00-30 silicone colored with Silc-Pig pigments
was used to cast the robot in a 3D printed mold (the robot’s design and fabrication is described in more
detail in ¹⁶).
In prior work we have discussed the artistic strategies used to compose the robot’s main sound design
(inspired by the sounds made by fictional soft characters in movies).¹⁷, ¹⁸ We have also
presented results of an empirical study exploring the effects of different sound designs on people’s
perceptions of the robot’s sociality and its interaction affordances.¹⁶ Following these
outcomes, work on presenting the project in the form of an art installation ensued.(1)
Physically, the SONŌ installation (Fig. 3) consists of: 1. The sonified soft robot displayed on a
black plinth
(dimensions 112 x 40 x 40 cm.) 2. A set of external speakers mounted in the room
The plinth features a door that can be opened to operate the robot during exhibition and houses the following
on three shelves (see Fig. 4): an active loudspeaker, an electro-pneu-matic actuation system (microcontroller,
motor shield, pumps, valves etc.) and an audio interface, a laptop PC running a software synthesizer. Along all
four edges of the plate holding the robot morphology, a small opening is present, to allow sound from the
loudspeaker inside the plinth to be transmitted to the exhibition space (Fig. 2). The audio of the installation
consists of two times 2-channel stereo comprising: 1. robot sound – played over the loudspeaker inside the
plinth, 2. a soundscape – played over the external loudspeakers in the room.
The Technical System
A diagram of the technical system is shown in Fig. 5 with the signal paths indicated. An Arduino Uno, which
controls the robot’s movement by activating pumps and valves, functions as the master with a laptop PC
generating the audio running as slave (for details see ¹⁶–¹⁸). A signal to generate
matching robot sound using the FM software synthesizer is sent when a movement phrase is triggered. The robot
does not currently have any sensors.
Robot Behavior and Sound Programming
For the final installation, we built upon existing code already developed for movement and sound generation. We
chose a phrase-based and a categorical approach to designing the robot’s behavior. By “phrase”
we refer to sequences of robot movement and matching sound of a duration up to 30s. The robot operates as a
finite-state machine (FSM) with four mood states (categories). These each correspond to different levels of
arousal (relaxed, medium-relaxed, medium-aroused, aroused). In each mood, the robot generatively combines a
specific set of phrases and pauses matching this mood. Each of the phrases were hand coded and iterated upon for
expressivity (through trial and error) and subsequently matched to one of the four mood states. A total of 13
phrases were used as building blocks that are combined in different ways to generate the robot’s movement
and sound behaviors. All mood states feature a breath-like phrase with the robot performing asynchronous
periodic inflation across the chambers, that use increasing frequencies for increased arousal, in accordance
with findings of our prior work.¹⁹ In addition to the robot sound, the installation features twelve
composed soundscapes that are played parallelly through Ableton Live. These are also triggered by the
microcontroller, but asynchronously with the robot’s movements and sound. The soundscapes consist of
processed synthesized and recorded sounds and select sonic textures combined into ethereal sonic expressions
devoid of temporal structure and timing. The audio was kept spacious and wide to let the robot’s more
erratic utterances come into focus and only add a more subtle affective coloring of these. In accordance with
this, the soundscapes are played over loudspeakers physically separate from the robot’s plinth. The
soundscapes mix slow extensive sounds to create the aural impression of an atmosphere of the installation and to
position the robot (and its visitors) in different sonic worlds of various affective intensities. Played in
random succession, the soundscapes complement or clash with each other, and contribute a sense of emergent
narrative.
Further work
We plan to expand upon SONŌ in an updated version of the installation and in subsequent independent
works. As a next step, we would like to develop a means to have more interactive generation of the robot and
soundscape audio. Currently, the robot switches between its four mood states pseudo-randomly with the
statistical likelihood that the robot will switch its arousal state up or down after a completed phrase cycle as
an adjustable parameter. We hope to add sensors to the installation, e.g., room scale computer vison, to track
activity, behaviors, and affective states of visitors to enable the robot to interact. Furthermore, we are
considering developing and validating a more fine-grained phrase-based or parametric generation of affective
movement and sound with the system that can contribute more variation and nuance to the robot’s
expressions.
1 A supporting video showcasing excerpts of the SONŌ robot performing with the robot sound and
soundscape is available at: https://youtu.be/U0fGXCbcygU
Acknowledgements
We thank Cao Danh Do for help with 3D prints, mechanical design, construction, and illustrations.
References
1 J. Williams, Understanding Poststructuralism, First edition, Chesham, Bucks, Routledge,
2005.
2 K. Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and
Meaning, Durham, NC, Duke University Press, 2007.
3 D. Rus, M. T. Tolley, “Design, fabrication and control of soft
robots,” Nature, vol. 521, no. 7553, May 2015, 467–475, doi: 10.1038/nature14543.
4 M. B. Christiansen, L. Beloff, J. Jørgensen, A.-S. E. Belling, “Soft Robotics and
Posthuman Entities,” Journal for Artistic Research, no. 22, Dec. 2020, doi:
10.22501/jar.549014.
5 J. Jørgensen, “Leveraging Morphological Computation for Expressive Movement
Generation in a Soft Robotic Artwork,” in Proceedings of the 4th International Conference on
Movement Computing, in MOCO ’17, New York, NY, USA, ACM, 2017, 20:1-20:4, doi:
10.1145/3077981.3078029.
6 J. Jørgensen, “Prolegomena for a Transdisciplinary Investigation into the
Materialities of Soft Systems,” in ISEA 2017 Manizales: Bio-Creation and Peace: Proceedings of the
23rd International Symposium on Electronic Art, University of Caldas, Manizales, Colombia, Department
of Visual Design, Universidad de Caldas, and ISEA International, 2017, 53–160.
7 J. Jørgensen, “Interaction with Soft Robotic Tentacles,” in Companion of
the 2018 ACM/IEEE International Conference on Human-Robot Interaction - HRI ’18, Chicago, IL,
USA: ACM Press, 2018, 38–38, doi: 10.1145/3173386.3177838.
8 J. Jørgensen, “Constructing Soft Robot Aesthetics: Art, Sensation, and Materiality
in Practice,” Ph.D. thesis, IT University of Copenhagen, Copenhagen, 2019.
9 J. Jørgensen, “From Soft Sculpture to Soft Robotics: Retracing Entropic Aesthetics
of the Life-like,” in Shifting Interfaces: An Anthology of Presence, Empathy, and Agency in
21st-Century Media, H. Aldouby, Ed., Leuven, Belgium: Leuven University Press, 2020, 223–242.
10 J. Jørgensen, S. Ploetz, “LARPing Human-Robot Interaction,” in HRI
2020 Workshop on Exploring Creative Content in Social Robotics, Apr. 2020, Accessed: Jun. 19, 2020.
[Online]. Available: https://portal.findresearcher.sdu.dk/da/publications/larping-human-robot-interaction
11 J. Jørgensen, “TeMoG – An Accessible Tool for Creating Custom Soft Robotics
Parts,” in Interactivity and Game Creation, A. Brooks, E. I. Brooks, and D. Jonathan, Eds., in
Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering,
Cham: Springer International Publishing, 2021, 331–342, doi: 10.1007/978-3-030-73426-8_20.
12 J. Jørgensen, “Towards a Soft Science of Soft Robots. A Call for a Place for
Aesthetics in Soft Robotics Research,” J. Hum.-Robot Interact., vol. 12, no. 2, Mar. 2023,
15:1-15:11, doi: 10.1145/3533681.
13 W. Fujisaki, N. Goda, I. Motoyoshi, H. Komatsu, S. Nishida, “Audiovisual integration in
the human perception of materials,” Journal of Vision, vol. 14, no. 4, Apr. 2014,
12–12, doi: 10.1167/14.4.12.
14 D. Haraway, “Situated knowledges: The science question in feminism and the privilege of
partial perspective,” Feminist studies, vol. 14, no. 3, 1988, 575–599.
15 F. A. Robinson, O. Bown, M. Velonaki, “Designing Sound for Social Robots: Candidate
Design Principles,” Int J of Soc Robotics, vol. 14, no. 6, Aug. 2022, 1507–1525, doi:
10.1007/s12369-022-00891-0.
16 J. Jørgensen, M. B. Christiansen, “The Sounds of Softness. Designing Sound for
Human-Soft Robot Interaction,” Frontiers in Robotics and AI, vol. 8, 2021, 1–17,
doi: 10.3389/frobt.2021.674121.
17 M. B. Christiansen, J. Jørgensen, “Augmenting Soft Robotics with Sound,”
in Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, in HRI
’20, New York, NY, USA, Association for Computing Machinery, Mar. 2020, 133–135, doi:
10.1145/3371382.3378328.
18 M. B. Christiansen, J. Jørgensen, “SONŌ,” in Companion of the 2020
ACM/IEEE International Conference on Human-Robot Interaction, in HRI ’20. New York, NY, USA,
Association for Computing Machinery, 639, Mar. 2020, doi: 10.1145/3371382.3378399.
19 T. A. Klausen, U. Farhadi, E. Vlachos, J. Jørgensen, “Signalling Emotions with a
Breathing Soft Robot,” in 2022 IEEE 5th International Conference on Soft Robotics
(RoboSoft), Apr. 2022, 194–200. doi: 10.1109/RoboSoft54090.2022.9762140.
Bibliography
Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning,
Durham, NC, Duke University Press, 2007. M. B. Christiansen, L. Beloff, J. Jørgensen, A.-S.
E. Belling, “Soft Robotics and Posthuman Entities,” Journal for Artistic Research,
no. 22, Dec. 2020, doi: 10.22501/jar.549014.
B. Christiansen, J. Jørgensen, “Augmenting Soft Robotics with Sound,”
in Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, in HRI
’20, New York, NY, USA, Association for Computing Machinery, Mar. 2020, 133–135, doi:
10.1145/3371382.3378328.
B. Christiansen, J. Jørgensen, “SONŌ,” in Companion of the 2020 ACM/IEEE
International Conference on Human-Robot Interaction, in HRI ’20. New York, NY, USA, Association
for Computing Machinery, Mar. 2020, 639, doi: 10.1145/3371382.3378399.
Fujisaki, N. Goda, I. Motoyoshi, H. Komatsu, S. Nishida, “Audiovisual integration in the human
perception of materials,” Journal of Vision, vol. 14, no. 4, Apr. 2014, 12–12, doi:
10.1167/14.4.12.
Haraway, “Situated knowledges: The science question in feminism and the privilege of partial
perspective,” Feminist studies, vol. 14, no. 3, , 1988, 575–599. J.
Jørgensen, “Leveraging Morphological Computation for Expressive Movement Generation in a Soft
Robotic Artwork,” in Proceedings of the 4th International Conference on Movement Computing,
in MOCO ’17. New York, NY, USA: ACM, 2017, 20:1-20:4. doi: 10.1145/3077981.3078029.
Jørgensen, “Prolegomena for a Transdisciplinary Investigation into the Materialities of Soft
Systems,” in ISEA 2017 Manizales: Bio-Creation and Peace: Proceedings of the 23rd International
Symposium on Electronic Art, University of Caldas, Manizales, Colombia: Department of Visual De-
sign, Universidad de Caldas, and ISEA International, 2017, 153–160.
J. Jørgensen, “Interaction with Soft Robotic Tentacles,” in Companion of the
2018 ACM/IEEE International Conference on Human-Robot Interaction - HRI ’18, Chicago, IL, USA:
ACM Press, 2018, 38–38, doi: 10.1145/3173386.3177838. J. Jørgensen, “Constructing Soft
Robot Aesthetics: Art, Sensation, and Materiality in Practice,” Ph.D. thesis, IT University of
Copenhagen, Copenhagen, 2019. J. Jørgensen, “From Soft Sculpture to Soft Robotics: Retracing
Entropic Aesthetics of the Life-like,” in Shifting Interfaces: An Anthology of Presence, Empathy,
and Agency in 21st-Century Media, H. Aldouby, Ed., Leuven, Belgium: Leuven University Press, 2020,
223–242. J. Jørgensen, S. Ploetz, “LARPing Human-Robot Interaction,”
in HRI 2020 Workshop on Exploring Creative Content in Social Robotics, Apr. 2020,
Accessed: Jun. 19, 2020. [Online]. Available:
https://portal.findresearcher.sdu.dk/da/publications/larping-human-robot-interaction
Jørgensen, “TeMoG – An Accessible Tool for Creating Custom Soft Robotics Parts,”
in Interactivity and Game Creation, A. Brooks, E. I. Brooks, and D. Jonathan, Eds., in Lecture
Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, Cham:
Springer International Publishing, 2021, 331–342, doi: 10.1007/978-3-030-73426-8_20.
Jørgensen, M. B. Christiansen, “The Sounds of Softness. Designing Sound for Human-Soft Robot
Interaction,” Frontiers in Robotics and AI, vol. 8, 1–17, 2021, doi:
10.3389/frobt.2021.674121.
Jørgensen, “Towards a Soft Science of Soft Robots. A Call for a Place for Aesthetics in Soft
Robotics Research,” J. Hum.- Robot Interact., vol. 12, no. 2, Mar. 2023, 15:1-15:11, doi:
10.1145/3533681.
A. Klausen, U. Farhadi, E. Vlachos, J. Jørgensen, “Signalling Emotions with a Breathing Soft
Robot,” in 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft), Apr.
2022, 194–200, doi: 10.1109/RoboSoft54090.2022.9762140.
A. Robinson, O. Bown, M. Velonaki, “Designing Sound for Social Robots: Candidate Design
Principles,” Int J of Soc Robotics, vol. 14, no. 6, 1507–1525, Aug. 2022, doi:
10.1007/s12369-022-00891-0.
Rus, M. T. Tolley, “Design, fabrication and control of soft robots,” Nature, vol.
521, no. 7553, 467–475, May 2015, doi: 10.1038/nature14543. J. Williams, Understanding
Poststructuralism, First edition, Chesham, Bucks: Routledge, 2005.
Author Biographies
Jonas Jørgensen is a researcher and artist based in Copenhagen focusing on practice-based
transdisciplinary work at the nexus of technical science, media art aesthetics, and the posthumanities. He
holds a position as Associate Professor at the Biorobotics section of the University of Southern Denmark where
he is co-director of the Center for Soft Robotics. Webpage: https://jonasjoergensen.org/
Mads Bering Christiansen is a Copenhagen-based designer/artist/researcher whose practice evolves around
designing objects and experiences to explore and speculate about the potential futures emerging from human
technological entanglement. He is currently PhD student at the Center for Soft Robotics at the University of
Southern Denmark with a project that investigates the effects of integrating biomorphic aesthetics in robots
for social human-robot interactions. Webpage: https://madsbering.com/
Gesture and Spatiality in Electroacoustic Improvisation with Digital Video
MichaelLukaszuk,
SojungBahng
Queen’s University DAN School of Drama and Music Kingston, Canada
michael.paul.lukaszuk@gmail.com, sojung.bahng@queensu.ca
Abstract
This paper discusses a multidisciplinary approach to improvisation in which electroacoustic performance is
informed by its capacity to transform and respond to characteristics of digital video, along with visual and
architectural features of virtual spaces. We describe technical and creative strategies that are focused on
leveraging mobile devices in a performance setting for navigating audio-visual relationships in which digital
spaces are conveyed through video playback and processing, while gestural control highlights actual physical
spaces (e.g., gallery spaces). This creates a hybrid venue that benefits live electroacoustic creation with
digital artists whose work extends beyond the domain of sound. The presentation visual analogues to sonic
transformations of gesture and spatiality are used to highlight aspects of electroacoustic performance that
develop out of a gradual exploration between emulation and obfuscation of referential sound objects. Mappings
between electroacoustic instrument and digital video also help delineate the creative use of mobile devices and
controllers as being more than tools. At the same time, the transmission of MIDI and OSC (Open Sound Control)
messages from mobile instruments allows collaborating artists to resource electroacoustic sound in non-aural
contexts.
Keywords
Electroacoustic, Sound and Music Computing, Spatialization, Audio-Visual Improvisation, Digital Instruments.
Digital lutherie has been an effective thread for connecting electroacoustic musicians with video and extended
reality artists. The data stemming from a digital translation of physical gesture to affect sound diffusion can
also be used to convey intersecting spatial relationships for artists who explore proximity and physicality in a
visual context. The motivation for this project is to explore the mobile phone as an instrument for
multidisciplinary improvisation that involves electroacoustic music. Beyond establishing the mobile phone as a
convenient resource for blending artistic or scholarly boundaries, there is a need to consider the
organological implications of routing the device through software environments for sound and visual
media. This depends on making decisions about what kinds of mappings and idiomatic behaviours help assert a
sense of instrumentality, so that a “smartphone-as-instrument” setup conveys compositional drivers
that are germane to electroacoustic creation—such as the spectrum between sounds being presented with a
relationship to a physical source vs. belonging to a more acoustmatic context. In his article
“The Instrumentality of Music,” musicologist Philip Alperson uses the example of Bob Dylan’s
infamous performance with electric guitar at the 1965 Newport Folk Festival to emphasize how instruments help us
understand the taxonomy of musical genres and how instrumentality, as a concept, goes beyond the notion of a
music being “of instruments” to show how instruments place music in relation to larger cultural
trends.¹ Focusing on the mobile phone as a collaborative digital instrument helps disrupt notions of
traditional musical performance practice that render electroacoustic as non-performative. We will discuss
electroacoustic improvisation with digital video in relation to the piece Suffocation
(2022), in which gestural control of a mobile device within a multichannel audio setup facilitates
continuous reframing of sourced objects within 360-degree video.
Related Work
In the 21st century, trends such as the proliferation of laptop orchestras and the emergence of live coding
help establish a context in which seemingly commonplace technologies can be understood as musical instruments
through their use within simulated spaces. For example, in composer Mara Helmuth’s laptop ensemble work,
pieces such as from Uganda...(2013) use a simple graphical user interface in Max/MSP as an instrument
for constructing imaginary sonic environments based on soundscape recordings.² In addition to environmental
references, performers interact with the laptop as an electroacoustic instrument by routing their individual
playback through a multichannel audio setup used within a concert hall.
Interactive musical mobile apps such as Brian Eno and Peter Chilver’s Bloom (2008) use synthetic
or at least ambiguous sampled sounds to create a kind of acousmatic sound world in which there is
intentional ambiguity with the manner in which the source material is presented. Bloom also bridges
mobile phone performance in relation to digital spaces through a 2018 partnership with Microsoft to create a
mixed reality installation in Amsterdam’s Transformatorhuis arts space in which the user interface was
projected on larger touch-sensitive surfaces.³ Conversely, Chilvers’ other musical app projects such
as Air (2011) relies on the reconfiguration of Irish singer Sandra O’Neal’s recorded
vocal performance through gestural interaction with the mobile device. In her PhD Dissertation, Vanessa Chang
writes that “through gesture, contemporary digital art practices borrow and recode forms of presences...to
highlight a more diverse array of creative agencies.”⁴ A “vocal performance” with
“Air” abandons the idea of the voice as a fixed instrument dependent on breath control and becomes
an expression of a cybernetic virtuosity that depends on feedback between physical and simulated bodies.
Suffocation
Suffocation was developed during the fall of 2022, as an improvisational piece stemming from the
authors’ involvement in a mixed-media ensemble, and a desire to explore spatiality using an approach that
blended our respective practices. For the video content used in this piece, a 360-degree camera was placed
inside a small container containing an assortment of tiny objects, such as pins, tissues, pills, and leaves, and
poured or added liquids. The everyday objects and materials are defamiliarized by creating the illusion of
scale. It provokes a paradoxical sensation between unpleasant sentiments and familiar feelings. The audio
content moves through processes of defamiliarization and reimagination of conventional instrumental sounds
within a digital context. This is achieved by sampling commonplace instruments such as pianos and kalimbas,
while also including computer-generated emulations of these sources that could be bent and distorted by
manipulating sound synthesis parameters.
In addition to the use of a quadrophonic (4.0-channel) audio playback system, additional highly directional
parametric speakers were positioned in our gallery space to synchronize with the presentation of height and
depth in the video content.
The setup for project uses pitch, roll, yaw, accelerometer, and keypad button OSC data transmitted from
composer/programmer Kevin Schlei’s smartphone app GyrOSC to affect a Max/MSP patch that acts as a link
between electroacoustic sampling and sound synthesis in the digital audio workstation Reaper, and video playback
and manipulation taking place through an Isadora patch.
Max/MSP sends MIDI control change and note messages that are mapped to affect spatial parameters of projected
video such as scaling and rotation, while simultaneously affecting changes such as triggering recorded audio
samples, musical pitches for virtual synthesizer plugins and audio panning positions. It is the simultaneous
translation of OSC to MIDI data mapped to Isadora for video and Reaper for audio that allows the smartphone to
act as a digital instrument for improvisation.
Theme and “Learned Gestures”
The use of a thematic design for structured improvisation with a digital instrument is an important
consideration for cultivating the mobile device as an electroacoustic instrument for multidisciplinary
collaboration. The use of conceptual bases or programmatic associations helps inform physical gestures that can
help the performer discover new ways of interacting with the mobile device while considering notions of
idiomaticity and how physical movements (e.g., changes in X-Y-Z axis positions) supports the position that
spatialization and abstraction of source material can be principal drivers in audio-visual work. The linking of
audio-visual parameters through MIDI and OSC also acts as a form of research-creation for performance in which
users can rely on previously learned actions (e.g., swiping, tapping, repositioning the device) in performance
to access new forms of creativity. This idea of accessibility issues as being resolved by digital instruments is
echoed in computer music research, such as computer music programmer Ge Wang’s documentation of laptop
orchestra as a pedagogical tool. His research from the Princeton Laptop Orchestra describes such the ensemble as
an experiential workshop environment that allows computer music studies to venture away from a studio/lab
environment that separates composition and performance and instead encourages interdisciplinary
exploration.⁵
This writing also describes how the ensemble’s use of sound spatialization approach references the
hierarchical nature of the orchestra in classical music. While the laptop orchestra model considers spatiality
in reference to its comparison with a conventional symphony orchestra, our use of learned gestures in
electroacoustic improvisation with digital video adds a new perspective to research-creation efforts in
which performance with electroacoustic instruments can also be used for “epistemological intervention into
the ‘regimes of truth’...”⁶ Rather than challenging the orchestra paradigm, we focus on
reconsidering Eurocentric structural materials that foreground melodically oriented constructs of rhythm and
pitch still permeate so much of contemporary sonic performance. By affecting the morphology of sound as
connected to video through fluid physical movements, learned gestures become objects within the performance that
help delineate how sonic and/or visual media are developing.
Conclusion
An exploration of digital instruments as reliant on familiar movements is echoed in computer music research,
such as Thor Magnusson’s Sonic Writing, which emphasizes that “...young people often
enter the world of music through digital technologies (e.g., apps and games).”⁷ By exploring the
cybernetic virtuosity in digital sound and video performance through physical gestures learned through everyday
interactions with mobile devices, we wish to uncover the possible components of a performance practice that is
compatible with live digital art practices such as electroacoustic music—in which spatiality is so
essential to the performance experience. A traditional orchestral performer may have to learn how to use
instrumental techniques such as vibrato to better project the sound of their instrument in a space, the use of a
mobile phone in an improvisational piece such as Suffocation exposes a multiplicity of layers in which
rhythm, cadence and narrative structure are also conveyed by the way that the performer explores the proximity
and distance of their own body as connected to digital audio and visual materials.
References
1 Philip Alperson, “The Instrumentality of Music,” The Journal of Aesthetics and
Art Criticism 66/1, Winter, 2008, 37-51.
2 Michael Lukaszuk, “Soundscapes and Anachronisms in Music for Laptop Ensemble,”
eContact! 21/1, Spring 2022, https://econtact.ca/21_1/lukaszuk_laptop-soundscapes.html
3 Susanna Ray, “Brian Eno, Peter Chilvers create ‘quite magical’ flower garden
of sound in Amsterdam with ‘Bloom: Open Space," Microsoft/Features, Accessed May 15, 2021, February 22,
2018,
https://news.microsoft.com/features/brian-eno-peter-chilvers-create-quite-magical-flower-garden-sound-amsterdam-bloom-open-space/
4 Vanessa Chang, “Tracing Electronic Gesture: A Poetics of Mediated movement," PhD
dissertation, Stanford University, 2017.
5 Ge Wang, et al, “The Laptop Orchestra in the Classroom,” Computer Music Journal
32/1, Spring 2008, 26-37.
6 Owen B Chapman, Sawchuck Kim, “Research-Creation: Intervention, analysis and ‘family
resemblances’,” Canadian Journal of Communication 37/1, April 2012, “Media Arts
Revisited,” 5–26.
7 Thor Magnusson, Sonic Writing, London, Bloomsbury, 2018.
Generated tools: A Defamiliarizing Approach to Creating ML Art
ChloeMcFadden and
OliverBown
School of Art and Design, UNSW New South Wales, Australia chloe.mcfadden@unsw.edu.au,
o.bown@unsw.edu.au
Abstract
In recent years there has been an increase in the adoption of machine learning (ML) systems that can generate
novel images. This increased use may reveal the beginning of a familiarity in which the implications of these
emerging technologies are naturalised or made increasingly invisible. Thus, practices which can disrupt
familiarity may allow us to create experiences of heightened awareness in which we can consider our engagement
with this emerging technology. In this paper, I discuss the outcomes of working with a Generative Adversarial
Network (GAN), using a dataset created from the hand tools section of a popular Australian hardware store.
Through this creative practice, I investigate how artists can use ML as mechanisms for creating artworks that
disrupt, investigate and defamiliarize the known.
As digital technologies are increasingly embedded into the everyday and the ways we construct and interact
within our world perception, it has been noted that they also become increasingly familiar, invisible, or
subconscious.¹ While often perceived as a ‘tool’—an extension of the human hand designed
to fulfil a certain function—our relation to technology is increasingly being understood as a symbiosis of
interdependence.² Furthermore, the co-evolutionary relationship between tools and humans has also been
noted—highlighting the influence these ‘tools’ have in shaping us, as we shape them.³ As
we continue to see, use, and adopt these tools, they become increasingly familiar and every day. However, with
this increased familiarity may come a decrease in active or conscious visibility resulting in the loss of
regular inciting provocation to question how we relate, interact, and construct our perception of the world with
these technologies. Thus, techniques of making the familiar unfamiliar may be a strategy for creating
experiences of heightened perception of or critical engagement with technology.
Generated tools (2021) is a practice-based research project in which I worked with a GAN to create an
installation that features tools as subject matter. In this, I am exploring how we can use ML conceptually to
create defamiliarizing experiences which may allow us to critically reflect upon and re-engage with concepts,
worldviews and ideas which have been naturalised. ML that can generate images is compelling
technology—sparking new ways of making and creative possibilities. However, it also has the potential to
be a reaffirming conservative force, as it is informed by the training dataset and may familiarize and
perpetuate the visuals, ideas and assumptions hosted within.⁴ While this reliance upon the dataset may
pose a risk if we engage uncritically—this also presents the opportunity for artists to disrupt, explore
and denaturalise the known. Thus, through the documentation and discussion of work created with a GAN, this
paper presents the concept of defamiliarization as one approach for understanding the potential impact or
possibilities of working with ML to generate art.
Tools, Technology, and Familiarity
There has been much critical reflection on the relationship between humans and tools—both in how tools
present an invisible familiarity, as well as in the mutual influence we exert. Martin Heidegger explained tools
as "ready-to-hand", meaning as long as they function correctly or to our expectations, they remain concealed
from view, or not in our conscious presence.⁵ However, while tools typically fade into the realm of the
unconscious everyday, they also reveal a co-evolutionary relationship between tools and humans. While we
typically narrativize our relation to tools as being a one-way influence, contemporary philosophical approaches
recognise that it is more of a symbiosis. To solve an issue or to respond to our environment, we create new
tools, which in turn change how we act and lead to the discovery of new problems or use cases for tools as this
cycle repeats itself.⁶
Similar lines of thought have also been applied to technology. It has been suggested that technology is an
extension of ourselves with which we cohabit a shared ecosystem. Through this cohabitation, we are able to
coextend our skills, capabilities, and properties, changing how we act in ways we perceive as beneficial, and in
turn leading to the development and adoption of new technologies.⁷
However, as this cycle of development and adoption continues, this cohabitation becomes increasingly familiar
or subconscious.⁸ While technological innovation feels novel at first, over time with increased adoption
and use, repeated interaction results in a comfortable familiarity that doesn’t necessarily encourage
reflection or critical engagement.
In the past 5 years, there has been increased adoption of ML systems that can generate novel images. While
there is a large amount of general ‘hype’ and awareness, the development of systems like DALLE-2,
Stable Diffusion and Mid Journey, which all have user-friendly demos and applications hosted online, alongside
apps like WOMBO Dream and TikTok’s green screen AI filter, may reveal the beginning of familiarity, or a
future of familiarity with ML systems.
Defamiliarization and visual indeterminacy
If familiarity reduces technology to the subconscious, thus dampening opportunities to be aware of how we
relate to and are shaped by it, unfamiliarity becomes a potential strategy for sparking conscious engagement.
Coined by Viktor Shklovsky in 1917, defamiliarization is understood as a tactic for creating heightened
awareness or perception by halting our automatic assumptions about a subject, allowing us to view it again for
the first time.⁹, ¹⁰, ¹¹ The goal of such a practice is not to
reveal a more objective truth about a subject, but rather to create a heightened perception of how we construct,
understand, and relate to the subject.¹² Defamiliarization has also been identified as a common tactic
employed by digital artists to create experiences of critical distance between audiences and technology, to
temporarily make the familiar unfamiliar for heightened perception to be achieved.¹³
The ways that artists enact this varies broadly, as it is understood that all art forms can generate a
defamiliarizing effect.¹⁴ However, one emerging approach related to ML is visual indeterminacy. It
has been noted that GANs typically create uncanny, strange, or visually indeterminate images.¹⁵
Visual indeterminacy, as well as ambiguity and uncertainty, has also been recognized as useful tools for
prompting multiple interpretations or disrupting an artefact’s socially encoded
properties.¹⁶, ¹⁷ Furthermore, artists may be able to engage the differences
between our logic and the chosen system’s logic to create defamiliarizing effects. For example, when
working with image-based ML, we understand images as representations of objects, scenes, and worldly concepts,
while a system like a GAN is attempting to map the underlying structural logic of the dataset at a pixel level.
Engaging the system’s ability to recreate patterns within the dataset, we can generate coherent and
recognisable forms. However, by navigating their latent spaces, we can push them to create semi-coherent, or
visually indeterminate forms. Thus, working with ML to generate art may present opportunities to denaturalise
our understandings through visual indeterminacy or strangeness, via this difference in structural logic.
Installation description
The installation consists of a tool wall which houses three instances of working with a StyleGAN2 trained using
a dataset of images of hand tools sourced from the Bunnings Warehouse website (an iconic Australian hardware
store).¹⁸ The first instance shows 3D printed tools, modelled and printed using the generated images
by the GAN as reference. The second instance shows 2D “latent space walk” videos projected on the
tool wall, created by incrementally sampling the latent vector space generated by the GAN. The final instance
shows doctored Bunnings Warehouse product catalogues, featuring images of tools generated by the GAN. Audience
members are encouraged to engage with the work by flipping through the product catalogues, as well as being able
to pick up, play with and rearrange the 3D printed objects on the tool wall.
3D Printed tools
While images created by ML systems are becoming increasingly familiar, we most commonly encounter them in
digital spaces, as images or animations. Thus, one goal of the work was to translate the generated images into
3D objects to observe whether this could be a method of creating defamiliarizing experiences. The 3D printed
tools were created by first generating a series of images using a StyleGAN2 trained on a dataset of images of
hand tools sourced from the Bunnings Warehouse website. While the GAN is trained using these images of hand
tools, how these tools exist and function in conceptual space is not captured. Thus, when it generates images of
tools it does so with no consideration of functionality—creating tools that have no clear associated uses.
Simultaneously, we can expect the generated images to be aesthetically tool-like as the GAN is attempting to
replicate the original dataset and find its underlying structural logic. Thus, the work explores whether the
‘uselessness’ of the GAN-produced artefacts is an effective tactic of defamiliarization.
Margaret Boden discusses ‘useless’ artefacts as having the ability to playfully challenge
expectation, while bringing the typical affordance of similar items to the foreground.¹⁹ Tools are
useful—but most of the time they are not being used—hung upon tool boards like artworks, neatly
organised in boxes like collectables, or haphazardly thrown in draws like junk. They often also hold semantic
and sentimental value - tools are given as gifts, passed down through families and lent between friends and
communities. If tools remain concealed from perception so long as they function correctly, how does encountering
inherently useless tools bring these hidden ‘uses’ to the foreground?²⁰
The generated images were then curated and used as references to model and texture 3D tools in Autodesk Maya
and Mudbox. Reference images were selected through a process of working through the generated images and
selecting a broad range of shapes, tool-like iconography and sizes that existed within the possibility space.
The goal when modelling and texturing the tools was to follow the reference images closely, while also
recognizing the inherent role of interpretation when translating from 2D to 3D. These models were then 3D
printed using a variety of grey, black, and aluminium filament. The results are visually and texturally strange,
tool-like artefacts. As they were modelled using the StyleGAN2 curated images as reference, they have a visually
indeterminate quality—with familiar elements (e.g. handles, pointy ends, and bits and pieces of
recognizable tools) combined or blurred in unfamiliar ways. Further, since they are 3D printed, the tools have
unique ridges, divots and holes which create an unfamiliar hand feel. The light-weight material of the filament
creates a unique engagement with the tools which we typically associate with rubbery, metal, and heavy
sensations.
Through the combination of recognizable tool-like elements in unfamiliar ways with the 3D printed materiality,
the familiar functions and feelings that are tied to how we perceive, construct and experience tools
subconsciously are no longer present—opening the space for new interpretations and speculation. As
audiences would approach the strange tools, typically their first reaction was to generate a new use case for
the tool. In short, this element of the work combines visual indeterminate ‘useless’ tools supplied
by the GAN with subversive, textural 3D prints to create a speculative experience for audiences, engaging
difference in structural logics, visual indeterminacy and unfamiliar materiality as tactics of
defamiliarization.
2D latent space walk videos
The projected 2D latent space walk videos were created using the possibility space generated by the StyleGAN2
trained on the dataset of hand tools. Latent space walk videos involve the sampling and incremental changing of
points in the latent vector space which can then be used to create animations. As the system was trained on a
dataset of hand tools, the resulting animations present fluid tools that merge and shape into one another. While
we typically conceptualise tools in a worldly context—with clear boundaries, defined shape language and
affordances that are affirmed via learnt aesthetics (e.g., handles, spouts)—the GAN’s distribution
renders a more fluid visualisation.
Thus, the work is interested in whether the visual indeterminacy created by the GAN is an effective strategy of
defamiliarization. As visual indeterminacy is understood as a tactic to engage audiences with the active nature
of seeing and meaning making, it may engender defamiliarizing experiences.²¹, ²²
Furthermore, visually indeterminate art prolongs perception through the combination of “apparently
detailed and vivid images resist identification.”²³ The generated latent space walk videos thus
present strange tools trained using a highly recognisable visual dataset, which now resist specific
identification via the GAN’s involvement. Whether this resistance of identification invokes the active
nature of seeing, or creates experiences of heightened perception is unclear—does coming into the contact
with the work go beyond strange? Do these splodgey, blurry tools engender an active awareness or reflection on
the symbiotic nature of tools and technology?
Product catalogue
Accompanying the tool wall are doctored product catalogues which have been inserted with images of tools
generated by the GAN. Using Photoshop, the original composition of the catalogue was closely followed as the GAN
generated tools were added based on their perceived visual similarities. The goal was to create a catalogue that
could be believably passed off as real at first glance.
The resulting catalogue is a mixture of the original text, lifestyle images, prices, and generated tools. This
presentation results in a double-take effect. Unlike the 3D printed tools which are outwardly alien or strange
by design, in this presentation, they almost look like real tools inside of a real catalogue at a cursory
glance. The context the product catalogue provides—the text, price tags, product descriptions, branding
and lifestyle images—almost act as visual vouchers for authenticity of these generated tools.
Returning to defamiliarization, Shklovsky positions the purpose of art as to “make forms difficult, to
increase the difficulty and length of perception because the process of perception is an aesthetic end in itself
and must be prolonged."²⁴ Thus, the work engages surprise, understood as violation of
expectation²⁵, and appropriation as potential strategies of defamiliarization. By appropriating the
catalogue and inserting familiar, yet strange tool-like artefacts, the doubletake causes an increase in
perception, calling us to look a little closer and make sense of the nonsensical tools.
In this, the work explores whether the surprise and absurdity of the nonsensical catalogue provides space to
investigate the socially coded meanings present in the subject matter that are familiar and normalised. For
example, does the messaging and imagery surrounding Father’s Day become less natural or familiar through
the introduction of the generated tools?
Conclusion
As the practice of generating images using ML becomes increasingly familiar, we need to be aware and
considerate of the ways it may naturalise or reinforce worldviews, categories, and ideas via this engagement.
However, just as ML has the ability to be conservative, to narrow in on the dataset and to perpetuate the ideas
within—so too can artists employ it to disrupt, investigate and defamiliarize the known.
Through the documentation and analysis of my work, Generated Tools (2021), this paper presents an
experimental approach to generating art with ML to create defamiliarizing experiences. As a part of this
discussion, I have explored how artists can engage with visual indeterminacy and the structural logics of ML
systems, as a method for disrupting the assumed, and reapproaching the known with fresh perspective.
Furthermore, through the analysis of the 3D prints, projected latent space walk videos and doctored product
catalogues, I have investigated how the presentation of ML generated images can create, alter, and deepen the
defamiliarization experience.
In this I am interested in how we can ‘make strange’ in ways that matter. As ML presents unique
opportunities for defamiliarization—how can artists defamiliarize in ways which persist beyond the initial
encounter? How can we blur, splodge and surprise with ML in ways that recover both the subjects of datasets and
technologies themselves from the automatism of perception and use?
Acknowledgements
This project was the first author’s (Chloe McFadden) honours project supervised by the second author
(Oliver Bown); hence the paper is written in the first person singular.
References
1 I. Hwang, M. Guglielmetti, V. Dziekan, "The Familiar": technology-being-with-us," Usage
guidelines, 2016, p.65.
2 É. Brangier, S. Hammes-Adelé, Beyond the technology
acceptance model: Elements to validate the human-technology symbiosis model, in International Conference on
Ergonomics and Health Aspects of Work with Computers, Springer, 2011.
3 J. Navarro, P.A. Hancock, Did Tools Create Humans? Theoretical Issues in Ergonomics
Science, 2022, 1-27.
4 Kate Crawford, Trevor Paglen, "Excavating AI: the politics of images in machine learning
training sets," AI & SOCIETY, 2021, doi: 10.1007/s00146-021-01162-8.
5 Harman Graham, "Technology, objects and things in Heidegger," Cambridge journal of economics 34,
no. 1, 2010, 17-25.
6 J. Navarro, P.A. Hancock, Did Tools Create Humans? Theoretical Issues in Ergonomics
Science, 2022, 1-27.
7 É. Brangier, S. Hammes-Adelé, Beyond the technology acceptance model: Elements to
validate the human-technology symbiosis model, in International Conference on Ergonomics and Health Aspects of
Work with Computers, Springer, 2011.
8 B.C Bruce, M.P. Hogan, The disappearance of technology: Toward an ecological model of
literacy, in Writing in a Technological World, 2019, Routledge, 191-207.
9 V. Shklovsky, Art as technique [1917], The Critical Tradition: Classic Texts and Contemporary
Trends, New York, 2007, 3, 774-784.
10 L. Crawford, Viktor Shklovskij: Différance in Defamiliarization, Comparative
Literature, 1984, 209-219.
11 D.P. Gunn, Making art strange: a commentary on defamiliarization, The Georgia Review,
1984. 38(1), 25-33.
12 R. Bellanova, A.R. Sætnan, How to Discomfort a Worldview?: Social Sciences,
Surveillance Technologies, and Defamiliarization, in Science, Technology, and Art in International
Relations, Routledge, 2019, 29-39.
13 L. Starkand, K. Crawford, The work of art in the age of artificial intelligence: What
artists can teach us about the ethics of data practice, Surveillance & Society, 2019. 17(3/4),
442-455.
14 D.P. Gunn, Making art strange: a commentary on defamiliarization, The Georgia Review,
1984. 38(1), 25-33.
15 Aaron Hertzmann, "Visual indeterminacy in GAN art," Leonardo 53, no. 4, 2020, 424-428.
16 P. Yurman, A.V. Reddy, Drawing Conversations Mediated by AI, in Creativity and
Cognition, 2022.
17 J.J. Benjamin, et al, Machine Learning Uncertainty as a Design Material: A
Post-Phenomenological Inquiry, in Proceedings of the 2021 CHI Conference on Human Factors in Computing
Systems, 2021.
18 Karras Tero, Samuli Laine,Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila, "Analyzing
and improving the image quality of stylegan," In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, 8110-8119, 2020.
19 M. A Boden, Creativity and art: Three roads to surprise, Oxford University Press, 2010, 63.
20 Graham Harman, "Technology, objects and things in Heidegger," Cambridge journal of economics
34, no. 1, 2010, 17-25.
21 D. Gamboni, Potential images: Ambiguity and indeterminacy in modern art, Reaktion Books, 2002.
22 R. Pepperell, "Seeing without objects: Visual indeterminacy and art," Leonardo 39(5), 2006,
394-400.
23 A. Ishai, et al, "Perception, memory and aesthetics of indeterminate art," Brain Research
Bulletin 73(4), 2007, 319-324.
24 V. Shklovsky, "Art as technique," Literary theory: An anthology 3, 1917, p.16.
25 M. L Maher, et al, Computational models of surprise in evaluating creative
design, Proceedings of the fourth international conference on computational creativity, Citeseer,
2013.
Ecologies of Thought: Generative Art as a Collaborative Research Methodology with Guarani and Kaiowá
Indigenous Communities
MatheusdaRochaMontanari
University of São Paulo, Polytechnic University of Valencia, Multimedia Anthropology
Lab (UCL). São Paulo, Brazil/ Valencia, Spain matheusrmontanari@gmail.com
Abstract
The Ecologies of Thought project aimed to reconceptualize the relationships between ecological and technical
knowledge, seeking an epistemological understanding that pushed beyond nature vs. culture divides. More than a
conceptual and theoretical proposal, which was based on the technodiverse notion of cosmotechnics, the project
developed practical and experimental methodologies in collaboration with the Guarani and Kaiowá Indigenous
communities of Brazil to further this investigation. With an international and interdisciplinary partnership, we
researched relationships between sounds and plants in indigenous cosmology and the ways in which these
relationships can help us to more deeply understand a notion of ecology that is based on the poetics of care.
With that, we created a series of collaborative virtual reality experiments using generative art processes,
which proved to be an interesting methodology of artistic creation as research.
Keywords
Indigenous Communities; Generative Art; Ecology; Virtual Reality; Anthropology; Cosmotechnics; Digital Art; Art
as Research; Multimedia Anthropology; Guarani Kaiowa.
Between March and July 2022, the Ecologies of Thought (1) project established an
international and multidisciplinary partnership between Indigenous communities of Brazil, artists,
anthropologists, botanists, and audiovisual producers to investigate Guarani and Kaiowá ecological thought
through the relationships between sound and plants. The project proposed dialogues among different types and
conceptions of technology, from chanting and the traditional cultivation method, to the use of microcontrollers
and data analysis.
This project is part of the Guarani and Kaiowá Virtual Museum (2), which develops a
virtual reality museum curated by the community elders and shamans – Nhandesys (female), and Nhanderus
(male) in partnership with the Multimedia Anthropology Lab at University College London.
Since the beginning of the project, the community has been challenging and redefining the notion of a "museum,"
the extra-human relationships among plants, animals, instruments, and spirits present us with a complex
challenge: how to maintain a Guarani and Kaiowá cosmotechnical fidelity in the museum?¹ That is, how
to enable the indigenous cosmology, and its technologies, not only to inhabit the virtual space but to build it
as a tekoha, a territory, that allows the exercise of the Guarani and Kaiowá way of being and
living, teko.² That means, how to build a digital Guarani and Kaiowá tekoha?
To our surprise, in the first test of the museum presented to the community with the virtual reality headset,
two Nhandesys, experienced chanters, saw Jaras, spirits, in the virtual space. From this encounter, we
began to investigate different ways to explore the spiritual and cosmological aspects, which are so present in
the Guarani and Kaiowá ecology of thought, within the museum. The relationship between sound and plants
proved to be a fertile field for this research. We began to learn more from the community elders and apprentices
who introduced us to chanting as a cultivation technology: a physical and spiritual dialogue between the plant,
its spirits, and the chanter. To chant is to care for, and the poetics of care is essential for the development
of a healthy territory, be it physical or digital.³
We learned that instruments are also people. And that the Guarani and Kaiowá notion of preservation is
different. For example, if the instrument is kept and not used, even if in a collection, it loses its ability to
produce sound because it loses its ability to create connections when it is removed from its communication
network. Therefore, it became evident the need to explore different ways to activate the collection we have been
building for the museum.
To this end, we developed an experimental strategy. We focused on the creation of virtual reality worlds that,
using generative art processes, explored the Guarani and Kaiowá cosmology from its cosmotechnical aspect.
We used sounds and traditional elements, such as the Chiru, the sacred stick that supports the world, and the
white corn, which is inhabited by one of the most important spirits (Jakairá), to investigate fundamental
elements of the cosmology through an experience, rather than a narrative
representation.⁴ In light of the urgent need to revise and rethink human-ecology relationships on a
global scale, while acknowledging the new epistemic possibilities of data capture and analysis technologies, we
collaborated with Guarani and Kaiowá community members to explore the different and creative kinds of data
that these devices afford, and the ecological knowledge they make possible.
The Guarani and Kaiowá
Mato Grosso do Sul is a state in the mid-western region of Brazil, bordered by Paraguay and Bolivia, in a
transition region between three biomes: Atlantic Forest, Cerrado, and Pantanal. It is the State with the
second-largest indigenous population in the country and the largest outside the Amazon. The Guarani
Ñhandeva and Kaiowá population of the region in 2021 is estimated to be 63.5 thousand people. At the
same time, the State has the second-worst land distribution index in the country, with large properties (>
1,000 ha) occupying 83% of the total area. Thus, it is not surprising that the region is a scenario of serious
conflicts between landowners and indigenous people, especially in the area of cattle ranching, sugarcane, corn,
soybeans, and eucalyptus plantations.⁵
The Guarani and Kaiowá are Guarani speakers, from the Tupi-Guarani linguistic trunk. They have a strong
ancestral territorial bond, both physical and spiritual, which guides their struggle in an especially organized
way since the large assemblies beginning in 1979 (Aty Guasu). These articulations occur mainly between
the religious leaders, Nhanderus and Nhandesys, since the community considers that the simultaneous performance
of the religious ritual (jeroky) "is fundamental to recover the dialogue with the invisible beings and
the guardians of the ancient tekohas," that is, for the recovery of the historical territories of their people,
it is also fundamental to recover the extra-human relations developed in the territories, whether they are
animal, vegetable, and/or spiritual.⁶
These relationships are directly connected to chanting and praying. The chant, besides a dialogue, is part of
the materiality of the world: "In the understanding of the spiritual leaders, the singularities of the physical
world inevitably need singing to continue their continuous existence, otherwise the world will gradually
end."⁷ This is one of the reasons we choose sound, specifically chant, as a source of investigation of the
ecological relations in Guarani and Kaiowá thought.
Artistic creation as an interdisciplinary collaborative methodology
The Guarani and Kaiowá cosmology have a great degree of complexity and diversity, even in the same family
trunk, some stories may vary between one group and another. Moreover, the narratives are full of recursion and
non-linearity, in which each detail branches off into another story, that can take days to be told in the
traditional oral form. So, it was clear from the beginning of the project that there would be limitations on the
form and amount of information we could present. At the same time, we did not want to make a reductionist
narrative translation that would create a shallow representation of the generous exchanges that the project made
possible.
Besides that, the diverse points of view and areas of knowledge involved in the project configure,
simultaneously, its richness and complexity. The challenge of combining shamans, scientists, and artists in the
same project revealed the power of the digital medium as a research tool in itself, and not only as a form of
scientific dissemination. More than that, we set out to develop research in visual arts, rather than
research about visual arts. That is, to reaffirm the processes of creation as valid ways of
researching, thinking, and producing knowledge about the world.⁸ In this sense, artistic creation acts in
the project as a collaborative methodology capable of providing dialogues and investigative interactivity: the
creation processes allowed for the creation of a two-way exchange, in which collaboration affected not only the
production but the understanding and direction of what was being produced.
In this project, art, which occupies a liminal place, is not limited to an art-technology extension of
scientific knowledge. We seek to embrace the mba'e kuaa, the Guarani and Kaiowá technical
knowledge and know-how. In this proposal, we guide ourselves with the concept of cosmotechnics developed by Y uk
Hui. That is, to rediscover the diversity of technical thought beyond the Greek notion of techné:
Technology is not anthropologically universal; its functioning is assured and limited by particular cosmologies
that go beyond mere functionality and utility. Thus, there is no single technology, but a multiplicity of
cosmotechnics.⁹
Generative Art and the Guarani & Kaiowá Cosmology
Generative art can be defined in different degrees of complexity, it is associated with a system that has some
degree of autonomy, in which the artist and the system exercise a series of operations that result in the final
work.¹⁰ Seeking ways to engage with the mythology and respect their characteristics, generative art
methodologies seemed to not only fit with the ontological qualities of the Guarani and Kaiowá cosmology
but to privilege an investigation and audiovisual production that could maintain a cosmotechnical fidelity.
Since both, the Guarani and Kaiowá cosmology and generative art are nonlinear emergent processes with
repetitive elements that are in constant transformation. Working in that direction, we created a series of six
experiments, and in this article, we will be presenting one of them.
Jakairá: white corn, bees, and the maintenance of the world
According to the Kaiowá myth, the deity Jakairá is responsible for the creation of the first
roçado (food garden). Through his wisdom, Jakairá created the white corn, which is at the top of the
food hierarchy, and, from it, all other agricultural products were created. Therefore, Jakairá and white
corn are responsible for the creation and maintenance of almost all food on Earth. Its cultivation follows a
series of rules, from dietary restrictions of the growers to the use of chants to cheer the corn and its
baptism.¹¹ The corn baptism ritual, called Jerosy Puku, is one of the most important
Kaiowá rituals. Through it, the corn becomes fit for consumption by the community in a healthy way, since,
besides being a physical food, it is also considered a spiritual food. The baptism is also a moment of
reaffirmation of the Kaiowá way of being, strengthening the human and extra-human (spirits, plants,
animals) relationships of the community.¹² This specific type of corn is currently in danger of
extinction due to the environmental threats it suffers, the heavy use of pesticides in the GMO corn plantations
that surround the communities, and the consequences of climate change in the region.
Our first experiment arises precisely from this extrahuman ecology. In one of our ethnographic sessions on the
importance of corn and its relationship to sound, we learned that corn has its own sound that is not audible to
all people. However, there is a specific moment when the sound of corn can be heard by everyone. The community
members told us that when the corn, still green (avati kyry), begins to mature and get ready for harvest, it
produces a smell that attracts bees so that "they can make the sound of the corn," warning the cultivator that
it is almost time to harvest. As talking about the white corn is also talking about the beginning of life and
the food that sustains it, this pre-harvest moment, when the bee is attracted to the corn to make its sound,
seemed emblematic to us to investigate an instant of the creation of the world according to the Guarani and
Kaiowá cosmology. This moment that seems, at first sight, to deal with a multispecies relationship between
the plant and the bee, when further investigated, reveals, also, to be a relationship between the spirit owner
of the corn and its cultivator¹³. To continue this investigation, we developed a generative art
experiment using the TouchDesigner programming language: a visual programming language in nodes for the
creation of real-time interactive multimedia content. We started with the image of a white cornfield before the
harvest and analyzed the light incidence in its different areas. From this data analysis, we created a relief
map that was applied to an ellipse for the construction of a terrain. The darker areas of the image created
depressions, and the lighter areas, elevations. Finally, we added two noise layers, one algorithmic, and another
from the sound analysis of the frequencies produced by the bees. The sound frequency analysis controls the
algorithmic noise, which generates motion on the terrain. With that, we exported a 360o video for VR headsets.
In this way, we developed a virtual reality experience in which the ecological relationship between the corn and
the bee produces the creation of a generative world based on the analysis of image and sound data from the
field, guided by the Guarani and Kaiowá cosmology (figure 1).
Even without a naturalistic production of the image, and perhaps precisely because of this characteristic, the
community approved the experiment. Once the step-by-step construction of the generative program was presented,
there was consensus that the experiment had true Guarani and Kaiowá characteristics, mixed with
non-indigenous technology. From this experiment on, the community began to direct the following more directly,
choosing images and elements that should be used for the continuation of the project.
Conclusion
The Ecologies of Thought project aimed to reconceptualize the relationships between ecological and
technical knowledge, seeking an epistemological understanding that pushed beyond nature vs. culture divides.
More than a conceptual and theoretical proposal, which was based on the technodiverse notion of cosmotechnics,
the project developed practical and experimental methodologies in collaboration with Guarani and Kaiowá
communities to further this investigation.¹⁴ With an international and interdisciplinary partnership,
we researched relationships between sounds and plants in indigenous cosmology and the ways in which these
relationships can help us to more deeply understand a notion of ecology that is based on a poetics of care.
Thus, we were able to bring together poetic and technical processes to propose a mixture of knowledge that
beyond an explanation or representation of one form of knowledge into another, develops an experience of its
own. This allows us to fight an idealized naturalistic representation of indigenous culture, which fails to
understand indigenous thought and technologies as valid ways to build and think of a cosmotechnic for our time.
More broadly, this project seeks to undermine the romantic and colonial notion of preservation, which does not
consider the indigenous presence a digital presence, able to act and think about technical processes since its
conception. This means, beyond an illustrative representation of what is believed to be the material and
immaterial cultural heritage of the community, to collaboratively build a digital-indigenous territory, a
digital Guarani and Kaiowá tekoha where a new technological ontology is established, in a way
that the very conception of contemporary technology is affected by indigenous cosmotechnics.
In conclusion, the epistemic research proposed and developed during the project revealed to us the potential
that this collaborative methodology possesses to go beyond epistemological investigations, enabling us to
revisit ontological questions of technology. Collaboration, interdisciplinarity, and the investigation of
creative processes—rather than the proposition of representations—allows us to arrive at questions
related to the level of existence, so that, in the anthropocenic era of the absence of futures, we can
compose a cosmotechnic for a viable, virtual and actual, reality, based on a poetics of care and more-than-human
relations.
Acknowledgments
We would like to acknowledge the University College London Global Engagement Fund which contributed to the
partnership among the Multimedia Anthropology Lab at University College London, the Kunangue Aty Guasu, IDAC,
The School of Botany, The University of São Paulo, The Federal University of Santa Catarina.
We would like to thank and acknowledge the generosity of the Nhandesys and Nhanderus whom we have been working
with during these years. Nhandesy Neuza Alziro, Nhandesy Ivone, Nhandesy Fausta Solano Mendosa e Nhandesy
Roselí Aquino. Nhanderu Tadeu Romeiro Freita, Nhanderu Valdomiro Oswaldo Aquino. Also, the team of
UCL’s Multimedia Anthropology Lab and the Kunangue Aty Guasu: Valdineia Aquino, Scott Hill, Luan Iturve,
Germano Alziro, Jaqueline Gonçalves, Raffaella Fryer-Moreira, Fabi Fernandes, Gerard Oliver, Adriana
Boloc, Stephanie Ferraz, Clara Ciqueira, Luiza Braga and MAL’s translation team. And our partners Karola
Braga, Anderson Santos and Caetano Sordi.
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de
Nível Superior – Brasil (CAPES) – Finance Code 88887.716693/2022-00.
1 Yuk Hui, The question concerning technology in China: an essay in cosmotechnics,
Falmouth, Urbanomic, 2018.
2 Tonico Benites, “Trajetória de luta árdua da articulação das
lideranças Guarani e Kaiowá para recuperar os seus territórios tradicionais tekoha
guasu,” Revista de Antropologia da UFScar, Vol. 04, No. 02, accessed October 14, 2022,
https://doi.org/10.52426/rau.v4i2.83
3 Matheus Montanari and Gilbertto Prado, “From vigilance to vigil: an introduction to an
alternative paradigm for technology, art, and life,” Diffractions, Vol. 01, No. 05, accessed
October 14, 2022, https://doi.org/10.34632/diffractions.2022.10194
4 Fabio Mura, “A Trajetória dos Chiru na construção da
tradição de conhecimento Kaiowá,” MANA, Vol. 16, No. 01, accessed October 14,
2022, https://doi.org/10.1590/S0104-93132010000100006
5 Anderson de Souza Santos, Luiz Henrique Eloy Amado and Dan Pasca, “"É muita terra
pra pouco índio"? Ou muita terra na mão de poucos? Conflitos fundiários no Mato Grosso do
Sul,” Instituto Socioambiental, accessed October 14, 2022,
https://acervo.socioambiental.org/acervo/documentos/e-muita-terra-pra-pouco-indio-ou-muita-terra-na-mao-de-poucos-conflitos
6 Tonico Benites, “Trajetória de luta árdua da articulação das
lideranças Guarani e Kaiowá para recuperar os seus territórios tradicionais tekoha
guasu,” Revista de Antropologia da UFScar, Vol. 04, No. 02, accessed October 14, 2022,
https://doi.org/10.52426/rau.v4i2.83,171
8 Sandra Rey, “Por uma abordagem metodológica da pesquisa em artes visuais,” in
O meio como ponto zero, ed. Bianca Brites and Elida Tesser, Porto Alegre: Editora da UFRGS, 2002,
123-140.
9 Yuk Hui, Tecnodiversidade, São Paulo, Ubu, 2020, 25.
10 Philip Galanter, “Generative Art Theory,” in A companion to digital art,
ed. Christiane Paul, Hoboken, John Wiley & Sons, 2016.
12 Izaque João, “Jakaira Reko Nheypyrũ Marangatu Mborahéi: Origem E
Fundamentos Do Canto Ritual Jerosy puku Entre Os Kaiowá De Panambi, Panambizinho E Sucuri’y, Mato
Grosso Do Sul”, Master’s. diss., Programa de Pós-Graduação em História da
Faculdade de Ciências Humanas, Programa de Pós-Graduação em História da
Faculdade de Ciências Humanas, 2011.
13 Carlos Fausto, “Donos demais: Maestria e Dominio na Amazônia,” MANA,
Vol. 14, No. 02, accessed October 14, 2022, http://dx.doi.org/10.1590/S0104-93132008000200003
14 Yuk Hui, The question concerning technology in China: an essay in cosmotechnics
Author Biography
Matheus Montanari is a Ph.D. Candidate at the University of São Paulo in Visual Arts, and external
researcher at the Polytechnic University of Valencia. He develops works at the intersection of art, science
and philosophy, investigating ways to rethink technology after art. He is interested in cosmotechnical
diversities, and in combining decolonial ecological thinking with technical know-how. Recently he has been
developing his practice in collaboration with indigenous Guarani and Kaiowá communities in Brazil. He
has exhibited his work in Brazil and abroad, in countries such as: Argentina, Portugal, Austria, Croatia,
China, Spain and Italy. He was awarded the web art category prize of the 67th Contemporary Art Salon of
Paraná.
Emerging Strategies “Under The Bay” in AR/XR
LisaMoren, TsvetanBachvaroff
University of Maryland Baltimore County (UMBC), Institute of Marine and Environmental
Technology (IMET), University of Maryland Center for Environmental Science (UMCES) Baltimore, Maryland
lmoren@umbc.edu, bachvarofft@umces.edu
Abstract
“Under the Bay” is an augmented reality project where anyone can use their cell phone like
a microscope and reveal invisibilities in our world and marine life. When they do a series of animated stories
between humans and non-humans emerge. Images, sounds, and stories are affected by live data streamed in from
sensors located in the largest estuary in North America. Sensors in the Chesapeake Bay relay live pH, oxygen,
temperature, etc. (figure 9). Similar to the water itself, color, speed, audio fluctuate with the water and
marine life, making “Under the Bay” a data-driven narrative with eight scenes that tell a
story of a world beneath the marine surface, and the exciting but frail health of estuaries and oceans
worldwide. The two projects discussed here, “Under the Bay” (2022) and “What
is the Shape of Water?” (2020), are part of Lisa Moren’s series of cross-species artworks
aimed at diminishing human-centered exceptionalism. The collaborations began in 2019 when Lisa was the inaugural
Artist-in-Resident at the Institute for Marine and Environmental Technology (IMET). There, she met researcher
and marine biologist, Dr. Tsvetan Bachvaroff (Tsetso) and the two immediately shared a like-minded vision to
develop a project that exemplified phenomenal exceptionalisms in micro-organisms. In this paper we argue that
novel strategies in nature emerge when a complexity of matter is intermingled with conditions of
differentiation. We explain and identify differentiation in art and architecture, symbiosis in biology, and the
“wobble” in physics as core principles for new forms and creative strategies to emerge. The outcome
is focused on the unusual and significant diversification of dinoflagellate microbes in the Chesapeake Bay and
oceans worldwide. Tsetso directed the live organisms, science and data analysis for the augmented reality
project. Stories are written and told by Lisa, who produced and art directed the animation and AR scenes. The
sound score is by electronic composer Dan Deacon. Dr. Marc Olano led the software engineering and development
with John Boutsikas, for the AR app in IOS and Google Play.
Keywords
Augmented Reality, AR, bio-art, data-driven narrative, emergent strategies, Tao Te Ching, water, symbiosis,
dinoflagellates, Chesapeake Bay, estuary, marine biology, performance, microbes, media art, podcast,
experimental narrative, bio-architecture, Lynn Margulis, Ted Nelson, Jane Bennett, Theodor Schwenk, data-driven
music, philosophy.
In our collaborative work, we borrow the idea of “emergent strategies” to consider strategies in
nature that describe phenomena driving new emerging forms. Inspired by Octavia Butler, Adrian Maree Brown coined
the term “Emergent Strategy,”¹ as “a framework for resistance that is
rooted in the miracles of nature, decentralized, collective leadership, and personal, relational,
organizational, and movement-wide transformation.”² We focused our strategies based in part on
Brown’s idea that anomalous strategies in nature can be a model for the benefit of human communities, from
species longevity to social change. Lisa’s second influence was her participation in a Taoist meditation
group where she studied and meditated on the Tao Te Ching.³ These meditations blended with her observations
of nature, primarily starling bird murmurations described in Scene 05 \\ Lava Lamps in the Sky.
The connections between emergent strategies, organic differentiation, and a Taoist social order became the basis
of this art and science collaboration.
Emergent Strategies
We find the origin of strategic emergence by looking at gene evolution in early microbes. The Darwinian idea of
natural selection has been combined with genome inheritance from two parents into what is now called the "modern
synthesis." While this synthesis has excellent explanatory power, some observations are not a perfect fit to
this synthesis. For example, photosynthesis is scattered across multiple kingdoms, but the process of
photosynthesis itself is unlikely to have been invented multiple times. The solution to this paradox required
two steps (figure 4). First, Lynn Margulis in 1967 proposed that eukaryotes are a combination of two cells from
different bacterial lineages. The first lineage is the archea bacteria which engulfed another bacteria cell that
contributes the mitochondrion. Mitochondria are responsible for the exchange of oxygen and carbon dioxide,
eventually this will be the foundation for breath itself. The mitochondrion is then a "cell within a
cell."⁴ This symbiotic combination of two previously distinct cells into eukaryotes created a new
complexity of features (figure 4). The first scene in our project Scene 01 \\ Origin Stories,
describes that before Margulis, any scientific reference to a symbiosis hypothesis was taboo. To accept
symbiosis, science had to imagine a scenario where a species swallows a giraffe’s eye to acquire longer
eyelashes successfully, but then, they pass those eyelashes onto their offspring. Symbiosis was once a fairy
tale.
But once the formation of eukaryotes via symbiosis of two lineages was accepted, then the symbiosis concept was
extended to include photosynthesis in plants and algae. In this scenario, a previously free-living
cyanobacterium was engulfed by a eukaryotic host (figure 5). The host now contains both mitochondria and
photosynthetic chloroplasts. This new host obtains energy from sunlight via the chloroplast and exploits that
energy in the mitochondria. This innovation ultimately leads to land plants and nearly all photosynthetic
eukaryotes (figure 4).
Sarah Gibbs extended the idea of symbiosis to explain why photosynthesis is scattered across multiple kingdoms.
In this third scenario, one cell with chloroplasts is consumed⁵ by a second, but only the chloroplasts are
retained. This provides access to photosynthesis and explains how photosynthesis could have spread across many
diverse eukaryotic groups.
Therefore, an individual can be shaped by two patterns of evolution: the familiar and dominant expression of
parental genes via vertical descent and a second, less familiar, pattern of horizontal gene transfer from
endosymbiosis (figure 4).
The dinoflagellates are from the eukaryote kingdom and a perfect case study to demonstrate both symbiosis and
the global importance of small eukaryotes to our planet. In global estimates of oceanic photosynthesis
dinoflagellates are one of the top oxygen producers.
The oceans produce oxygen at a scale similar to the contributions from the Amazon rainforest. This is
underscored by modern sequencing which places dinoflagellates as second in abundance to animals and first in
diversity in oceanic surveys. Dinoflagellates also include a wide array of life strategies such as built-in
harpoons used to capture prey.⁶ Other strategies include a symbiotic photosynthetic lifestyle as the algae
found in coral reefs use internal and external types of parasitism, and consuming food for energy. The term
mixotrophy describes the role of many photosynthetic dinoflagellates, which can be both photosynthetic
(autotrophic) and consume prey (heterotrophic).⁷
Mixotrophy and diversity in the ocean combined with the concept of symbiosis represented by Gibbs and Margulis
provides many instances of emergent strategies. For example, the dinoflagellate Dinophysis feeds on
ciliates demonstrating how the chloroplast could be consumed intact from prey by injecting a built-in
“straw-like method” to acquire, or suck, chloroplasts from its prey.⁸ Roughly nine events have
occurred where photosynthetic dinoflagellates contain chloroplasts borrowed or gained from essentially every
available lineage. And this list does not include dinoflagellates with elaborate structural features that
contain external symbiotes such as Ornithocercus.
Within the Dinophysis group there are different types of photosynthesis or symbioses, including the
‘gardening’ Histioneis (figure 6) and Ornithocercus (figure 7), the
‘heterotrophic’ Dinophysis that sucks out chloroplasts, and at least two other types of
internal symbionts. Ornithorcercus will host cyanobacteria in a birdcage-like crown, a strategy that
not only stores nutrients for on-going consumption, but the chains of cyanobacteria reproduce itself creating a
“fruit-on-the- vine-like garden” energy supply (figure 7).
Symbiosis has allowed these the single-celled dinoflagellates to diversify over hundreds of millions of years
into thousands of species that will clearly outlive humans. Therefore, diverse reproductive, energy consumption,
and other strategies assisted the dinoflagellates in obtaining longevity that humans can only dream of. For
example, the dinoflagellate Ceratium can both cell divide and mate for optimum reproduction benefits.
In Scene 01 \\ Origin Stories the viewer will see images of the Ceratium that cell
divide having only one trailing flagellum (a propeller-type tail). However, an identical Ceratium has
two flagella because they had two parents that mated (figure 3).
Other strategies we named included organic differentiation in the structure of the cell walls of the
dinoflagellates. These complex forms, such as Voronoi patterns, use less matter to produce structures that are
lighter in weight than any objects human engineering could produce based on Cartesian principles in
manufacturing. Differentiation can be exemplified in the exoskeleton of a crab or lobster but is most visible in
the repeating hexagon pattern of a turtle shell, where the hexagon shapes repeat, but not perfectly. It’s
in that imperfection, crookedness, or wobble, that creates more strength with less matter. Dinoflagellates,
diatoms, and other microbes also have this efficient design principle of differentiation. Scene 04
\\ Crooked Shelters (figure 8) describes how such differentiation influenced the largest 21st century
algorithmically designed and digitally fabricated architectural form in Seville, Spain, the Metropol
Parasol.⁹ The pavillion’s aerial view appears as a mushroom blooming throughout the gridded city. A
similar architectural example is in Stuttgart, Germany, where one of the algorithmic pavilions based on
differentiation in nature was so strong and lightweight, it blew away.¹⁰
Water Projects
In our first attempt to create a public project for an open house at IMET, we created an artwork demonstrating
the microbial strategy of making light through the bioluminescence of the dinoflagellate (Pyrocystis).
Eventually, the public display was set up as a ceiling tank hooked up to a Max/MSP, Arduino, and AV system for
the Light City Festival in Baltimore’s Inner Harbor. The audience entered a pitch-black room with a large
ceiling tank holding millions of invisible dinoflagellates. The system worked with a voice-activated trigger so
that when a participant spoke into a microphone, for example, “What is the Shape of
Water?” the microscopic organisms above them answered the question in turbulent shapes of
blue bioluminescence (figure 9). Originally, this was influenced by the mesmerizing organic order and
differentiation in the murmuration patterns of starling birds.¹¹ However, Tsetso’s colleague,
Dr. Al Place who studies the motion behavior of dinoflagellates, says that the flocking behavior of the
dinoflagellates will unlikely look as organized as the starlings. Instead, the water agitation produced
turbulent patterns more akin to the wobble, the crookedness or what philosopher Jane Bennett calls murmuring
messiness.¹²
“What is the Shape of Water?” began as a somewhat expedient strategy of displaying
bioluminescence in an art project that took over a year to fully realize. It was an unrealistic approach to
create art installations for the more than two dozen strategies we identified for potential exhibition. We
instead turned to storytelling and produced an augmented reality (AR) project with eight scenes, approximately
10 minutes per scene. Lisa took the strategies we identified for the project and built narratives that meandered
through purposefully fragmented topics related to microbes, water, Taoist meditation, architecture, monuments,
and politics.
The AR project became “Under the Bay” and it’s trailer¹³ illustrates the
animation, narrator, and incoming data. For the data, we worked with the Maryland Department of Natural
Resources (MD DNR) to siphon data from sensors already installed in the largest estuary in North America, the
Chesapeake Bay. In total 36 parameters stream into the project from six locations in the Bay. Locations are from
the Delaware border to Washington, DC, the Eastern Shore of Maryland, and Baltimore City (figure 10). The
parameters pH, temperature, oxygen, salt, chlorophyll, and turbidity (clarity) are updated every 15 minutes.
These parameters then affect the animation’s color, speed, and scale, along with the narration and music
composition. For instance, when the oxygen levels in the Bay are of good quality, the narrator’s voice
sounds normal. However, when the water is anoxic, with little or no oxygen, the voice becomes choppy,
fragmented, as if choking. In this way, the story, images, and sounds change from day to night and season to
season for an ongoing collaborative narrative with the Bay water. The unpredictable variability of the incoming
data becomes an authentic wobble created by the environment.
Listening to the Water
The project propaganda claimed the ambition of cross- species communication specifically “What if we
could hear what the water is saying?” To address skeptics to these claims, we’re borrowing
from water physicist Theodor Schwenk, as we do in the AR project, but also Jane Bennett¹⁴, who
references Graham Harmon, Bruno Latour, and Michel Serres’s The Birth of Physics.¹⁵
If we look at water as an object, we can argue that water is the largest object in the world. On the one hand,
the ocean contains essential elements, H2O, saline, and other matter on the periodic chart. We know that water
and gravity work together to form currents like pipes that braid in distinct patterns and that these flowing
pipes separate into differing physical data such as speed and temperature. These differences become visible when
encountering an obstacle, like a rock in a river, where we observe differentiation in the water shapes bulging
around the rock. Similarly, in the ocean, large and “long waves travel faster than short waves”
and overlap until the larger ones envelop the smaller ones, and the waves repeat the pattern
endlessly.¹⁶ But unlike early CGI waves, the real ocean’s patterns are not perfect or
predictable waves, their equilibrium billows, and exhales asymmetrically, and it’s in that asymmetry that
they wobble. Or, what Bennett calls, they produce a vibratory noise, where the force of repetition
starts to create a surge, an “irregular bombardment of circumstances,” especially when new
physical elements act as an obstacle such as a rock. Here, the current billows and exposes its diverse
temperatures, causing what Serres names a “cauldron of turbulence that thickens into lumps of
phenomena, and the bubbling swirl keeps those shapes upright... while the wobble produces variances of
noise.”¹⁷ A shape derives from the swirl that is both form and vibratory. To Serres, he
calls this vibratory noise “the fluctuating ado that is the strange substance of any discrete,
differentiated shape... (where) the multiplicity of the possible rustles in the midst of the forms
that emerge from it... It is restless matter... (a) percolation.”¹⁸ While much
of this noise dies like seedlings that don’t spawn, the intermingling currents and swirls, overlap with
enough force allowing forms to emerge from it.
While we do not think of water as having agency, free will, or decision-making abilities or even that its
elements are alive, in complex natural events, water does cause a multitude of events to happen, and is
therefore an actant. One of the most significant events that water enacts is sustaining and creating new life,
but also emerging shapes, and unique forms. In this way, water will begin by initiating an abundance of events,
such as a vortex or a whirlpool effect. Any time these elements react or affect one another, there is the
potential for something to emerge, including new life (figure 11).
In Scene 02 \\ Water Moving Around My Fingers, the microbes reflect Schwenks organisms that
take on the negative space of the vortex shape traveling in a reverse corkscrew spiraling up (figure
11).¹⁹ Therefore, when we say we can hear what the water is saying? It’s not that the water, as
pure physical compounds, produces a will or agency that desires to be heard. However, water’s natural
environment has reactions, interactions, and relationships with other phenomena and other physical materials
where unique consequences emerge as shapes, forms, beings, and blooms. Moreover, if we consider the incoming
parameters of the Bay water as a kind of alphabet — to use a human metaphor — the data does arrange
itself to describe a story of the Bay water’s emerging behaviors and forms. This communication is so hard
for humans to understand, especially to predict — that when the data reflects the formation of an algae
bloom, it’s often too late to hear the water telling us we are out of balance. Perhaps the data
acts like a Google knowledge engine, an algorithm anticipating the user’s thoughts when typing a partial
phrase into the search bar and displaying our presumed burning questions. However, in this case, the data
anticipates the thoughts of the Bay water. This concept is so significant that the scores of MD DNR sensors we
use to siphon data are funded solely to predict the emergence of algae blooms in the Chesapeake Bay.
“Under the Bay” observes how the Bay ebbs and flows over time to create an emerging
narrative driven by what the water is enacting as an emergence. We call this murmuration messiness,
listening to what the water is saying.
Conclusion
Once we name this process of emergence, we can apply it to the other scenes in the AR project, addressing
issues of differentiation and emergence not only in nature but also in creative ideas, forms, art, architecture,
monuments, and also politics, protests (we discuss emerging protests including BLM, the beheaded Columbus statue
ending up in the Bay, or the #metoo flocking of women in a bloom of pink hats — figure 12). This brings
Brown’s term emerging strategies full circle where strategies in nature can be a model for social
change. Again, our stories meander through these topics and phrases, like symbiotic waves engulfing one another
leaning on influences from the Tao Te Ching as much as science.
“Under the Bay” will be exhibited at the Peale Museum, in Baltimore, Dec 14 to Feb 1,
2024, for “Chamber of Wonders”, with a public panel on Jan 18. While “Under the
Bay” is freely available to download,²¹ the XR installation will include iPads
interacting with paintings, drawings, and assemblages for a unique museum experience. “What is the
Shape of Water?” and related works will also be on display.
“Under the Bay” Scenes 01-08 include: Origin Stories \\ Water Moving Around My Fingers \\
Chalky Faeries \\ Crooked Shelters \\ Lava Lamps in the Sky \\ Instrumental \\ Vaccine Blooms with Pink Hats \\
Turnover [I Can’t Breathe]
2 Andrea Ritchie, Invisible No More: Police Violence Against Black Women and Women of
Color, Beacon Press, Reprint edition, 2017.
3 Lao-Tzu, translated by Stephen Addiss, Stanley Lombardo, introduction by Burton Watson,
“Tao Te Ching”, Boulder, Hackert Publishing Company, 1993.
4 Lynn Margulis, “On the origin of mitosing cells," Journal of Theoretical Biology,
vol. 14, no. 3, 1967, 225–274.
5 Sarah P. Gibbs, “The chloroplasts of euglena may have evolved from symbiotic green
algae.” Canadian Journal of Botany, vol. 56, no. 22, 1978, 2883–2889.
7 Photosynthetic Fragilidium consuming a large Ceratium
https://www.youtube.com/watch?time_continue=1&v=KFFsnH
VlHyo&embeds_referring_euri=https%3A%2F%2Fmail.google.com%2F&embeds_referring_origin=https%3A%2F%2Fmail.g
oogle.com&source_ve_path=Mjg2NjY&feature=emb_logo
8 Dinophysis sucking the chloroplast from its prey
https://www.youtube.com/watch?v=cq3SBoCHPP4&t=3s
12 Jane Bennett, William Connolly, “The Crumpled Handkerchief.” Time and
History in Deleuze and Serres, Bloomsbury, London, UK, 2013, p.155.
13 “Under the Bay” trailer by Lisa Moren with Tsvetan Bachvaroff, 2022.
https://vimeo.com/796868197
14 Ibid, Bennett, Connolly.
15 Michael Serres, The Birth of Physics, Rowman & Littlefield International,
Ltd., 2018.
16 Theodor Schwenk, “Sensitive Chaos: The Creation of Flowing Forms in Water and
Air,” Sussex, Rudolf Steiner Press, 1996, 33.
17 Ibid, Bennett, Connolly. 157.
18 Ibid.
19 Ibid, Schwenk, p.44.
20 An eight-episode podcast version of “Under the Bay” is available on
Spotify, Google, and Apple podcasts. Links to podcasts and more information can be found on the project
website: http://lisamoren.com/underthebay.
21 Download the AR from Apple IOS or Google Play: https://apps.apple.com/app/id1641553491
Stefanie R. Fishel, The Microbial State: Global Thriving and the Body Politic, University of
Minnesota Press, 2017.
Sarah P. Gibbs, “The chloroplasts of euglena may have evolved from symbiotic green
algae,” Canadian Journal of Botany, vol. 56, no. 22, 1978, 2883–2889.
Douglas Kahn, “Energies in the Arts,” Cambridge and London, MIT Press, 2019,
2–46.
Lao-Tzu, translated by Stephen Addiss, Stanley Lombardo; introduction by Burton Watson, “Tao Te
Ching”, Boulder: Hackert Publishing Company, 1993.
Lynn Sagan, “On the origin of mitosing cells,” Journal of Theoretical Biology, vol. 14,
no. 3, 1967, 225–274, https://doi.org/10.1016/0022-5193(67)90079-3.
Theodor Schwenk, “Sensitive Chaos: The Creation of Flowing Forms in Water and
Air," Sussex, Rudolf Steiner Press, 1996.
Lars Spuybroek, Sympathy of Things: Ruskin and the Ecology of Design, Bloomsbury Academic,
2016.
Acknowledgements
This project is generously supported by the Saul Zaentz Innovation Fund at Johns Hopkins University and the
R.W. Deutsch Foundation. This project originated during an artist-in-residence supported by the Institute of
Marine and Environmental Technology (IMET), the Center for Innovation, Research, and Creativity in the Arts at
the University of Maryland, Baltimore County (UMBC). We’re grateful for the support from Maryland
Department of Natural Resources; John Boutsikas for his API and AR development, Danielle McPhatter and
Harvestworks DigitalMedia Art Center, the IRC and CAHSS, and Antoine Cayrol of Atlas V Immersive Experiences
in AR, XR, VR.
Production Team
Lisa Moren, Co-Principal Investigator, Producer, Art Director, Writer and Narrator; Dr Tsvetan Bachvaroff,
Co-Principal Investigator, Marine Biologist, Researcher and Data Anaylsis; Dan Deacon, Electronic Composer;
Dr. Marc Olano, Co-Principal Investigator, Lead Programmer; John Boutsikas, Programmer and Developer; Austin
Samson Modeler and Animator; William Forrest, Modeler, Animator, UI and Technical Artist; Woody Lissauer,
Voiceover Engineer and Male Narrator 1; Ruskin Nohe-Moren, Male Narrator 2; Aliyah Baruchin Copy Editor and
Fact Checker.
Author’s Biography
Lisa Moren is a multi-disciplinary artist who works with emerging media, bio-matter, public
space, AR and works-on-paper. She has exhibited her work at the Chelsea Art Museum, Creative Time, Drawing
Center (New York), Cranbrook Art Museum (Michigan) and Ars Electronica (Austria), Akademie der Kunste
(Germany), uShaka Museum (South Africa), and the Artists Research Network (Australia). She received the
National Endowment for the Arts award, is a Fulbright Scholar; a multi-year recipient of the Maryland State
Arts Council and CEC Artslink International, is a R.W. Deutsche Award recipient and a Saul Zaentz Innovation
Fellow in Film and Media at Johns Hopkins University. Her writing has appeared in Performance Research;
Visible Language; Inter Arts Actuel; New Media Caucus for “Algorithmic Pollution: Artists working
with Dataveillance and Societies of Control” and “CYBER
IN|SECURITY;” and her books on “Intermedia;” and Issues in
Contemporary Theory for “Command Z: Artists Working with Phenomena and Technology.” Lisa
Moren is a Professor of Visual Art at the University of Maryland Baltimore County (UMBC); is an Affiliate
Faculty at the Imaging Research Center (IRC) UMBC; and taught at FAMU and AVU in Prague; and the University of
California at San Diego (UCSD).
Dr. Tsvetan Bachvaroff’s research is focused on dinoflagellate evolution with special
emphasis on the parasitic dinoflagellates, using large-scale sequencing and phylogenetic methods to describe
the evolutionary history of different types of genes in dinoflagellates. He uses DNA sequence analysis from
data collection, assembly, annotation and phylogeny; has received numerous academic awards, including the
William Trager Award from the Journal of Eukaryotic Microbiology, International Society of Protistologists,
culture independent methods such as single cell PCR, sequencing, and sequence analysis; Establishing
dinoflagellate cultures. He received the Marsho award, Mid-Atlantic Section of the American Society for Plant
Biology, University of Maryland College Park, and the Chemistry Prize, Trinity School, New York. Tsvetan
Bachvaroff received his B.A. degree from Johns Hopkins University and Ph.D. from the University of Maryland
College Park. He was a Postdoctoral Fellow at the Center of Marine Biotechnology and subsequently with the
Smithsonian Institution. He is an Associate Research Professor for the Institute of Marine and Environmental
Technology (IMET) at The University System of Maryland.
Power and Resistance in Digital Degrowth
RaulNieves, JoanSoler-Adillon, EnricMor
Universitat Oberta de Catalunya (UOC) Barcelona, Spain dbsus4@uoc.edu,
jsoleradillon@uoc.edu, emor@uoc.edu
Abstract
Degrowth is increasingly gaining attention as an alternative model to the unfolding eco-social crisis generated
by industrial capitalism, though questions concerning digital technologies have yet to be addressed in degrowth
research. Among the movement of the (digital) commons, whose practices complement degrowth theory, one of the
research areas is the viability of such systems to release spaces from capitalism. As (digital) commons spaces
frequently revert to capitalist logic, we introduce the "technological dramas" model to encompass the reciprocal
and recursive technological production of political power by agonistic entities. We suggest that such a
techno-political perspective could contribute to better frame degrowth-related HCI research.
Planetary chances to restore a sustainable scenario are quickly running low. The UN questions whether we have
already passed the turning point.¹ The unsustainable resource throughput promoted by the industrial model
is (eventually if not yet) incompatible with the materially limited planetary boundaries.² Over the last
decades, scholars and practitioners have started to question perpetual growth, a pivotal paradigm in industrial
capitalism. Resource depletion, claims Turiel, precludes the default problem-solving conveyed through
technological innovation, development, and deployment.³
The Degrowth project entwines a set of proposals to prioritize democratic, social and ecological justice in
pursuit of well-being over economic growth.⁴ Such an economic, political, and social program conflicts
with the capitalist political economy: sustainable growth, for degrowth advocates, is an oxymoron.⁵
Degrowth provides a political-economy frame to rethink human-computer interaction (HCI) theoretical tenets and
practices as key elements to reconfigure the relationship between digital technological activity and eco-social
effects.
Political-economy in HCI
The research field of HCI brings together technological and human-related disciplines to improve the interplay
between users and machines.⁶ Despite HCI’s, deeply entrenched, industrial root logic, examining and
questioning this very logic, especially recently, has connected many authors in HCI sub-communities.⁷
Among them: reflexive HCI⁸, humanistic HCI⁹, or critical HCI¹⁰. Dourish examines the
early Sustainable HCI research production to expose the risks of naturalizing capitalist assumptions and
suggests instead broadening the theoretical approaches in use. The author advocates “dismantle design as
an anti-politics machine.”¹¹ Hence, Ekbia and Nardi urge to incorporate political economy in the
analysis of HCI’s design and practices.¹²
Problematic capitalist core mechanisms, especially productivism and consumerism, take specific forms in HCI-
related practices. Ekbia and Nardi quote Marx “production creates the consumer” relating consumerism
to the paradoxical production of the “user.”¹² This “designed user” is central
to the anthropocentric formulation of problems. Problems, again paradoxically, are produced to fit feasible
technological solutions, feeding the mechanism known as techno-solutionism.¹³ This back-feeding
entanglement is key for the industrial production system. It underpins perpetual growth while consuming
resources and expelling negative externalities, producing new problems.
These analyses within HCI resonate with the degrowth discourse. Despite some proposals (i.e., see ¹⁴
as post-anthropocentrism, or ¹⁵ as post-techno-solutionism) we suggest it’s necessary to
conduct research on how to consistently translate this critique into impact-aware HCI practice. Thus, we turn
towards degrowth-aligned practices to foreground strategic political economy issues arising from the struggle to
transition outside capitalism.
Challenges in (digital) commoning practices
According to Helfrich and Bollier degrowth and the commons movement are complementary to each
other.¹⁶ The commons don’t rely on economic growth and make compatible environmental and social
justice. These authors define the commons in terms of politics and economics as “a vast array of
self-provisioning and governance systems that flourish mainly outside of both the market and the
State.”
While degrowth frames the subject of critique, the commons exhibit social, political, and economic forms of
actualization based on the social practices of commoning: the stewardship practices that a community employs to
manage shared things (virtually anything) in common.
Digitally enabled communities of practice, spawned by the emergence of the internet, have long been at the
center of commons research. Peer-to-peer (P2P) networking architectures fostered a productive model labeled as
commons-based peer production by Benkler.¹⁷ According to P2P advocate Bauwens, this model represents
a generative alternative in front of the extractivist modes of capitalist production.¹⁸ Fuster
defines the digital commons as online communities which share non-exclusive co-created digital
resources.¹⁶
One of the main research concerns in the commons is the relationship between the state, capitalism, and
alternative spaces of resistance. The cartography of such borders seeks to discern the strategic practices to
release and gain back spaces from capitalist control. The commons aim at releasing spaces from capitalism and
confronting the enclosing of old and new commons. Such spaces risk falling back to capitalist logic, a process
referred to as co-optation. While also discussed as “colonization by capital”¹⁹,
“assimilation”²⁰, “incorporation”²¹,
“transvestment”²², or “unwanted corporate appropriation”²³, among
other terms, co-optation is ubiquitous in the commons theory but it is scarcely being examined in depth.
Among the few commons theory authors that specifically examine co-optation, we find De Angelis, Caffentzis, and
Federici. According to De Angelis, capitalism is about to face a social and ecological crisis and will likely
have to leverage or promote the commons to help manage the devastation. ²⁴ De Angelis claims:
“struggles [...] can be absorbed and become part of the system (co-opted), thus renewing it and
sustaining it.” As the logic of the market becomes counterproductive even from the viewpoint of
capital accumulation, precluding the cooperation necessary for an efficient system of production, Caffentzis and
Federici point to "the danger that 'commons' may be co-opted to provide low-cost forms of
reproduction."²⁵
Kostakis et al. review and discuss recent criticisms of peer production, classifying some of them as
co-optation. The study examines whether digital peer production could be emancipatory or instead become part of
capitalism. On one hand, the injection of funds in free open-source software (FOSS) projects and the
multi-million-dollar purchases of FOSS companies increase the risks of appropriation of the commons by corporate
interests. On the other hand, commons’ pro bono production is monetized and exploited by market agents for
profit extraction.²⁶
To inquire into the origins of digital co-optation, we focus here on the ideas of Osseewaarde et
al.²⁷ The authors analyze how digital commoners recurrently transition through alternative spaces, as
they eventually get co-opted. Commoning’s essence opposes the technological rationality of formalization,
standardization, and quantification, yet the emergence of such spaces relies on technological innovation which
is fostered by the growth-oriented efficiency ethos promoted by neo-liberalism. This contradiction, the authors
argue, results in a perpetuating illusion, a form of false consciousness, which is rooted in cynicism.
Ossewarde et al. claim that current standard technologies “are highly opaque because they are often
implicit and part of a formalized design for digital interaction that is in itself an arrangement of
‘false consciousness’.”²⁷ Hence, digital commoning (the generation of
contents, but especially of infrastructures) fails to resist co-optation, supporting capitalist expansion as De
Angelis, Cafentzis, and Federici denote.
While Ekbia et al. point to reification (the assumption of systemic concepts that are actually socially
constructed as inescapable and natural, like the capitalist market) as a hindrance to emancipatory
HCI²⁸, Dourish and Ossewaarde et al. refer to the studies on the reified construction of false
consciousness by Lukacs. Ossewaarde et al. argue that such cynicism cannot be overcome via ideology critique,
but through technology critique when it "is translated into post-capitalist acts of resistance to the dominating
technology design." De Angelis asks “Isn’t this co-evolution between struggle and capital
development really inherently with no end?”²⁹
Introducing a technological power construction model in HCI
Reified notions embed the HCI practices with false consciousness, rendering alternatives prone to get co-opted
by capitalism. To further comprehend this dynamic, we suggest introducing “technological dramas,” a
framework by the anthropologist of technology Bryan Pfaffenberger.³⁰ As technological activity
presents an opportunity to embed political values, Pfaffenberger examines how power and resistance are
constructed through the reciprocal and recursive shaping of artifacts and values in the design process, which
later spread in society.
Pfaffenberger stresses the relevance of myth, ritual, and context in the understanding of the political
dimensions of technological activity. Myths are deployed to suspend skepticism, rituals are associated with
controlled environments produced to pattern human actions, and social contexts are fabricated in parallel. In
this model, technological activity is analyzed as a process of technological communication: “a
technological drama is a discourse of technological statements and counter-statements.”
The model describes three processes that can occur linearly, or under different permutations, in the
construction of politics by technological means:
● Regularization occurs when a design entity (usually part of the establishment)
“creates, appropriates, or modifies a technological production process, artifact, user activity, or
system in such a way that some of its technical features embody a political aim”.
● Adjustment takes shape when impact entities engage in control and alteration
strategies. These strategies attempt to counter the discursively regulated social contexts that regularization
creates, pursuing to counter their effects. This process can lead to technological appropriation
● Reconstitution materializes when impact entities “try to reverse the
implications of a technology through a symbolic inversion process,” labeled as antisignification by
Pfaffenberger. This process can produce, as in some forms of adjustment, appropriation. It can also result in
the fabrication of counterartifacts, “which embody features believed to negate or reverse the
political implications of the dominant system.”
Pfaffenberger employs the term reintegration to refer to co-optation processes: “the
response made by the agents of regularization to the new, problematic counterartifacts. Its goal is to gain
control over these artifacts by bringing them back into the controlled and ordered space of
regularization.” According to the author, some forms of adjustment and reconstitution stages
(resistance processes with relevant degrees of technical intervention) are prone to co-optation. While
co-optation has been previously discussed, it is considered by this model in a wide and complex techno-political
dynamic.
Besides the three above-mentioned processes, Pfaffenberguer introduces a fourth: designification
takes place when the link between technological activity and social meaning-producing discourse dims.
“The artifacts, their contexts, and our social behaviors remain; they become taken for granted,
routine, and part of the natural attitude of everyday life.” According to Pfaffenberger, this is
the stage where technological activity achieves the greatest social penetration. We suggest a connection between
designification and the previously discussed process of reification, here particularly referred to in
techno-political, rather than political economy, terms.
Conclusions
In front of collapse, the political and economic project of degrowth offers an alternative coexistence formula.
The HCI community is already approaching the degrowth frame in order to redefine its tenets and practices, and
political economy analysis has already been adopted by HCI authors to examine reified notions. To expand HCI
research we have examined digital commoning challenges to release spaces from capitalist logic. Reification,
pointed out already by HCI authors, false consciousness, and co-optation processes are discussed by
commons’ authors as strategic issues concerning the transition to post-growth alternative political
economies.
We suggest the relevance of this model in analyzing and prospecting the construction of political power, in its
different stages, in order to deploy strategic practices of transition in front of a ravaging capitalism.
Especially due to the systemic view this model offers to dissect and relate processes like designification,
co-optation, and possibly others, as stages of the permanent struggle through technological activity. We also
suggest leveraging this model as a tool to complement speculative, adversarial, fictional, strategic, or
transitional design techniques.³¹-³⁵ Hence, we expect this contribution to help HCI
researchers better frame degrowth-related practices and research contributions.
This article is part of the R+D+i project PID2021-128875NA-I00, funded by MCIN/AEI/10.13039/501100011033/ "ERDF
A way of making Europe”.
References
1 Matt McGrath, "Climate change: IPCC report warns of ‘irreversible’ impacts of global
warming", Retrieved December 7, 2022, from https://www.bbc.com/news/science-environment-60525591.
2 Nicholas Georgescu-Roegen, The Entropy Law and the Economic Process, Cambridge, MA,
Harvard University Press, 1971.
3 Antonio Turiel, Petrocalipsis, Alfabeto, 2020.
4 Matthias Schmelzer, Aaron Vansintjan, Andrea Vetter, The Future Is Degrowth: A Guide to a
World beyond Capitalism, Verso Books, 2022.
5 Jason Hickel, Giorgos Kallis, "Is Green Growth Possible?", New Political Economy 25
(4), 2020, 469–486.
6 Paul Dourish, Where the Action Is: The Foundations of Embodied
Interaction, Cambridge, MA, USA, MIT Press, 2001.
7 Raul Nieves, Enric Mor, Joan Soler-Adillon, "Decoupling design from the industrial paradigm," In
Proceedings of ISEA2022, the 27th International Symposium on Electronic Art, Barcelona.
8 Phoebe Sengers, John McCarthy, Paul Dourish, "Reflective HCI: articulating an agenda for
critical practice," In CHI ’06 Extended Abstracts on Human Factors in Computing Systems (CHI EA
’06), 1683–1686.
9 Jeffrey Bardzell, Shaowen Bardzell, Humanistic HCI, San Rafael, California: Morgan
& Claypool, 2015.
10 Jeffrey Bardzell, Shaowen Bardzell, Mark Blythe (eds.). Critical Theory and
Interaction Design, Cambridge, MA, USA, MIT Press, 2018.
11 Paul Dourish, "HCI and environmental sustainability: the politics of design and the design of
politics," In Proceedings of the 8th ACM Conference on Designing Interactive Systems (DIS ’10),
1–10.
12 Hamid Ekbia, Bonnie Nardi. "Social Inequality and HCI: The View from Political Economy," In
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16),
4997–5002.
13 Evgeny Morozov, To Save Everything, Click Here: The Folly of Technological Solutionism,
New York, Public Affairs, 2014.
14 Bill Tomlinson, Bonnie Nardi, Daniel Stokols, Ankita Raturi, "Ecosystemas: Representing
Ecosystem Impacts in Design," In Extended Abstracts of the 2021 CHI Conference on Human Factors
in Computing Systems, 1–10.
15 Eric P.S. Baumer, M. Six Silberman, "When the implication is not to design (technology)," In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11),
2271–2274.
16 Giacomo D’Alisa, Federico Demaria, Giorgos Kallis, Degrowth: A Vocabulary for a New
Era, Routledge, 2014.
17 Yochai Benkler, The Wealth of Networks: How Social Production Transforms Markets and
Freedom, New Haven, Conn., Yale University Press, 2007.
18 Michel Bauwens, Vasilis Kostakis, Alex Pazaitis, Peer to Peer: The Commons Manifesto,
University of Westminster Press, 2019.
19 Christian Fuchs, "The Digital Commons and the Digital Public Sphere: How to Advance Digital
Democracy Today," Westminster Papers in Communication and Culture 16 (1), 2021.
20 Maurizio Teli, Marcus Foth, Mariacristina Sciannamblo, Irina Anastasiu, Peter Lyle, "Tales of
Institutioning and Commoning: Participatory Design Processes with a Strategic and Tactical Perspective," In
Proceedings of the 16th Participatory Design Conference 2020 - Participation(s) Otherwise - Volume 1
(PDC ’20), 159–171.
21 Benjamin Birkinbine, Incorporating the Digital Commons: Corporate Involvement in Free and
Open Source Software, University of Westminster Press, 2020.
22 Michel Bauwens, Alekos Pantazis, "The ecosystem of commons-based peer production and its
transformative dynamics," The Sociological Review 66 (2), 2018, 302–319.
23 Benjamin J. Birkinbine, "Conflict in the Commons: Towards a Political Economy of Corporate
Involvement in Free and Open Source Software," The Political Economy of Communication 2 (2), 2015.
24 Massimo De Angelis, "Crises, Capital and Co-optation: Does Capital Need a Commons Fix? | The
Wealth of the Commons," Retrieved May 9, 2022, from
http://wealthofthecommons.org/essay/crises-capital-and-cooptation-does-capital-need-commons-fix.
25 George Caffentzis, Silvia Federici, "Commons against and beyond capitalism," Community
Development Journal 49 (suppl_1), 2014.
26 Vasilis Kostakis, Vangelis Vragoteris, Indra Lal Acharja, "Can peer production democratize
technology and society? A critical review of the critiques." Futures 131, 2021, 102760.
27 Marinus Ossewaarde, Wessel Reijers, "The illusion of the digital commons: ‘False
consciousness’ in online alternative economies," Organization 24 (5), 2017, 609–628.
28 Hamid Ekbia, Bonnie Nardi, "The political economy of computing: the elephant in the HCI room,"
Interactions 22 (6), 2015, 46–49.
29 Massimo De Angelis, "Crises, Movements and Commons," Borderlands 11, 2012.
31 James Pierce, "In Tension with Progression: Grasping the Frictional Tendencies of Speculative,
Critical, and other Alternative Designs." In Proceedings of the 2021 CHI Conference on Human Factors in
Computing Systems (CHI ’21), 1–19.
32 Carl DiSalvo, "Adversarial Design as Inquiry and Practice," In Adversarial Design, MIT Press,
2012, 115–125.
33 Matt Malpass, Critical Design in Context: History, Theory, and Practice, London, New York,
2017.
34 Anna Meroni, "Strategic design: where are we now? Reflection around the foundations of a recent
discipline," Strategic Design Research Journal 1 (1), 2008, 31–28.
35 Terry Irwin, "Transition Design: A Proposal for a New Area of Design Practice, Study, and
Research," Design and Culture 7 (2), 2015, 229–246
Motherplants: Mycelium Network and Artistic Research
JulienOttavi, JennyPickett
Association APO33, Madlab - Cyprus University of Technology Nantes, France - Limassol,
Cyprus julien@apo33.org, jennypickett@apo33.org
Abstract
In 2015, we embarked on an artistic research project involving organic organisms such as plants and fungi,
focusing on their role in the processes of recycling of e-waste. Motherplant explores how these systems create
feedback audio signal transmission through the interplay of moisture and decomposition of the circuit boards.
Specifically, it harnesses the natural decomposition process to transmit audio signals by converting the soil's
acidity and electronic compost into electrical signals.
Keywords
Artistic research project, electronic compost, mycelium networks, soil acidity, symbiosis, mutation of
technologies, e-waste.
The surge in global electronic waste (e-waste) volumes, as reported by the Global E-waste Statistics
Partnership (GESP), is cited by the World Health Organization (WHO). Between 2014 and 2019, there was a
substantial increase of 21% in e-waste generation, resulting in a staggering production of 53.6 million metric
tonnes of e-waste during the latter year. "For perspective, [in the year 2020] e-waste weighed as much as 350
cruise ships placed end to end to form a line 125 km long. This growth is projected to continue as the use of
computers, mobile phones and other electronics continues to expand, alongside their rapid obsolescence."
(Johnson, 2021) The repercussions of this escalating e-waste problem are dire, encompassing adverse effects on
both human health and ecosystems. The e-waste stream comprises a multitude of toxic substances, ranging from
neurotoxins to heavy metals, which pose significant risks (Jain et al., 2023). These hazardous constituents have
the potential to inflict severe harm upon individuals and the environment, underscoring the urgency of
addressing and mitigating the detrimental consequences associated with mounting e-waste volumes.
This project revolves around the intriguing intersections of mycelium networks, audio diffusion, and the
creation of novel computing experiences. Motherplant engages with the notions of symbiosis and mutation of
existing technologies to explore new modes of interaction with recycling, art research and implementation of new
network models based on mycelium communication cooperation systems.
Using dead and discarded motherboards extracted from obsolete computer systems, we attempted a transformative
process with the aim of repurposing these printed circuit boards (PCBs) into "micro-farm" environments. The
primary objective of Motherplant encompassed three key facets: firstly, to experiment with sustainable recycling
of all motherboard components; secondly, to harness their inherent properties for the generation of electrical
current; and lastly, to contribute to the development of an alternative open system designed to facilitate the
exchange of computational data. This fusion of electronic and organic realms introduces a harmonious synergy,
wherein the motherboard's original purpose converges with the natural processes of growth, decomposition, and
nutrient exchange.
Re-creation of the fungal network as an electronic mutation
In Motherplant we are trying to create a novel electronic circuitry that mutates in tandem with a primitive
fungal network. It is important to clarify that our fungal network is a simplified "primitive" representation,
distinct from the extensive rhizomatic networks deployed by mature fungi across vast expanses of soil,
connecting plants and trees. We acknowledge that nature excels in this regard, and our intention is not to
replicate nature's proficiency but rather to cultivate and orchestrate a symbiotic relationship between
electronic circuitry in a localised fungal network.
This begins with a few spores settling down on a nutrient-rich surface. When these spores wake up in close
proximity to one another; they initiate germination at approximately the same time, giving rise to thread-like
cellular structures known as hyphae, which extend outward at comparable rates. This unique fusion of
electronics, powered by solar energy and moisture-rich soil infused with (DIY) mycelium growth, establishes an
intricate information superhighway. As an information superhighway the interactions between a large, diverse
population of individuals speeds up. It allows entities who may be separated to communicate and help each other
out. It also allows them to commit new forms of communication.
This symbiotic relationship between electronic circuitry and a fungal network not only bears relevance to the
creation of a new information exchange paradigm but also holds significant potential in the context of e-waste
recycling. Fungi possess the ability to break down and transform a wide range of organic and inorganic
substances, pesticides, hydrocarbons and heavy metals, making them valuable agents for cleaning up contaminated
environments (Tomer et al., 2021). By virtue of fungal bioremediation, the fungal network may play a pivotal
role in the sustainable degradation and recycling of electronic components.
Schematics for Motherplant
This iteration of Motherplant comprises a redundant, nonfunctional motherboard as its foundational structure,
upon which mycelium is cultivated within a composite medium consisting of straw, soil, and integrated circuitry.
The degradation of the motherboard’s components initiates an acidification process within the medium,
subsequently generating electrical signals. These electrical signals function akin to a battery, delivering an
approximate output of 1.5 volts, which in turn powers an audio circuit. The frequencies generated by this audio
circuit exhibit variability contingent upon the dynamic activity of the mycelium and the pH levels of the
surrounding soil.
Renaturing the Motherboard
The Motherplant concept can be traced back to an earlier experiment initiated at APO33 in Nantes 2015 and
presented during NEAR #1 as part of Nantes Digital Week. During this meeting, the growing need to address
e-waste issues was discussed alongside concerns about the exploitation of workers, recycling components without
protective clothing or in adequate conditions (Ottavi and Pickett, 2015). These first experiments went on to be
presented at the Nomad Village during COP21 in Paris, later on that year (Ottavi, 2015). Electronic waste is
toxic and hazardous to human health, PCB's are hard to recycle and demand a lot of energy and sometimes the
scramble for precious metals and minerals contained therein, sacrifices members of our human community
(Wittsiepe et al., 2015; Beaumont, 2019). Usually, this heavy cost is hidden from our sleek conscientious
recycling certificates. Yet it could take decades, sometimes centuries, before some of these components will be
totally recycled. In 2019 Europe ranked first worldwide in terms of e-waste generation per capita, with 16.2 kg
per capita (Forti et al., 2020).
With Motherplant by employing plants, flowers, and fungal organisms as instrumental agents in the recycling of
electronic waste, we set about composting the motherboard. The heart of any home computer or laptop, the
Motherboard is a concealed symbol of this cyclical consumption and our relentless obsessions for digital speed.
With Motherplant we sought to counteract this hidden world of consumer electronics recycling and make visible
the complexity of such a task if left to nature, as well as the sheer disregard for the problem by the shipping
of our waste elsewhere. The idea of Motherplant was to produce electricity to feed electronic circuits, so that,
in turn, they endlessly sustain a sound installation. This undertaking represented a confluence of art, ecology,
recycling practices and the exploration of alternative energy models. The shape and layout of a motherboard is
called the form factor. The form factor affects where individual components go and the shape of the computer's
case. We design Motherplant in such a way that the classical function of the motherboard decomposes to produce a
new kind of computer. One that won’t consume electricity anymore, but instead will become a receptacle for
plants and spores to grow.
Mycelium computation: a new paradigm in bioart and sound art research
This new kind of Mycelium computation is about to begin, as we search for quantum processors, we are looking
for Mycelium based hijacked computer circuits. The spore computation is not just another form of computer
research in the tradition of engineering production, it's an all-new vision about computers. In 2018, Professor
Andrew Adamatzky, Director of the Unconventional Computing Laboratory at the University of the West of England
in Bristol, UK, wrote in his article "Towards a fungal computer", how a series of scoping experiments, provided
evidence that electrical activity recorded on fruits could potentially serve as a dependable indicator of the
fungi's reaction to thermal and chemical stimulation. (Adamatzky, 2018) It diverges from the prevailing
landscape of technological industrial mass production, presenting a concept characterised by fragility, inherent
hazards, and a multitude of uncharted pathways, and prototypes of computing mycelium bound composites. (Roberts
and Adamatzky, 2022).
A mélange of Bioart (art using fungal bioremediation and technology) with sound art (sound as a profound
production of vibration) serves as a conduit for the cultivation of exchanges and ideas within a networked
context. Within the domain of bioart, the convergence of innovative gardening techniques and the exploration of
mycelium-based experiences culminates in the facilitation of multifaceted networks characterised by a wide array
of functionalities that will serve multiple purposes such as recycling, production of energy, circulation of
electricity but also production of sound waves, feedback transmission and autonomous art installation.
From the perspective of sound-ecology, listening to the world holds significant importance in numerous
practices. However, in the case of Motherplant, it transcends mere observation of nature; it involves a
collaboration with nature to generate a sound installation on a minute scale, characterised by low impedance and
microsounds, all within the dimensions of the sculpture itself. In terms of sound levels, our focus aligns more
closely with the realm of ants and insects than that of elephants or storms. This reveals the potential inherent
in the amalgamation of nature, mycelium, and soil within our artistic patch.
Art frequently detaches itself from its own reality, often transported and displayed within galleries or
dedicated venues, creating a sense of isolation. In contrast, Motherplant seeks to recreate a natural context
intricately linked to the artwork itself, ultimately transforming the context, nature, environment, and
aesthetics into the art itself. Sound plays an integral role in the production process but whilst it doesn't
represent the final objective, it is a vital component within the broader framework. The various iterations of
the art piece over the last 8 years, also contribute to shaping the sculptural essence of Motherplant. The
process that we describe here and the relation between nature, technologies and production of energy is not
separated from the poetic facets of our artistic research.
Recycling plastic and other dirty electronics to produce low currency (power)
Beyond the artistic sculpture of the Motherplant, we had assigned to this project two other important research
perspectives. Whilst these are not the main goals, they resonate with the other aspects (art, sculpture, sound,
visuals...). The first enables us to think about and experiment with the recycling of electronic wastes with the
Motherboard. In this context, the concept of the "mother" assumes a central and symbolic role, representing the
origin from which life emanates. This archetype of the mother extends beyond the human realm to encompass
broader notions such as Mother Earth and Mother Plant, all of which evoke contemplation of the nurturing and
life-giving aspects associated with motherhood. By integrating the concept of the "mother" into the project's
narrative, we establish a profound connection between technology, nature, and the cycle of life. Which
underscores the intricate interplay between electronic devices and the environment. How can we care for
electronic waste management and recycling? Motherplant embraces maternal sustainability and environmental
awareness, towards recycling e-waste and engendering a new computation network. Secondly, recycling e-waste
costs are very high compared to disposing them in landfills, mainly because it often contains hazardous
materials that need to be dismantled and processed properly. Also, the lack of recycling infrastructure poses
problems to handle the large volume of e-waste generated each year. Finally, due to the lack of awareness from
the public, most people have no idea of the importance of recycling their computers and e-waste. This lack of
awareness ends up in huge amounts of illegal dumping or in the wrong landfills and incinerators producing more
pollution but also exposing people to hazardous materials.
We decided, as artists working in electronic arts and media, to engage in this protracted undertaking of the
project Motherplant, and develop our own way of experimenting with living networks. Rather than adhering to the
conventional and often inefficient approach of classical digital art research, which typically involves a
relentless cycle of producing more work and tirelessly pursuing galleries and festivals for exhibitions, we have
chosen a different path. We consider this process in the broader context of "decroissance" or degrowth, lowering
our consumption, decelerating our expectations of results, and embracing the temporality associated with
decomposition. The philosophy of degrowth posits that the relentless pursuit of economic growth is inherently
unsustainable in the long term. It contends that, in order to safeguard the environment and enhance societal
well-being, it is imperative to curtail economic activities and their associated impacts.
The other part of this research centres on the production of electricity. While it's important to note that
this production is characterised by low power output, it has nonetheless proven to be a sustainable source of
electrical current over the past few years, requiring no additional input other than the periodic watering of
the Motherplant. This enduring supply of electricity has so far facilitated a continuous eight-year run of our
sound installation.
The motherboard, serving as the fundamental underpinning of our installation, adheres to a decompositional
timeframe, thereby providing a robust scaffold for the continuous provision of electrical current. This enduring
electrical supply, in a reciprocal fashion, sustains the seamless operation of our sound installation, ensuring
its uninterrupted functionality. This approach to both energy generation and artistic expression accentuates the
intricate interplay amongst technology, ecology, and the dimension of time.
Documentation
View the 2015 video on Climate Solutions: MOTHERPLANT here: https://www.youtube.com/watch?v=CSUeSdR6n1Q. View
the 2018 video on Motherplants here: https://vimeo.com/301201120
Peter Beaumont, “Rotten Eggs: E-Waste from Europe Poisons Ghana’s Food Chain.” The
Guardian, April 24, 2019.
https://www.theguardian.com/global-development/2019/apr/24/rotten-chicken-eggs-e-waste-from-europe-poisons-ghana-food-chain-agbogbloshie-accra.
Vanessa Forti, Peter Baldé, Ruediger Kuehr, Garam Bel, S Adrian, M Drisse, Y Cheng, et al. 2020,
“Quantities, Flows, and the Circular Economy Potential the Global E-Waste Monitor, 2020”,
https://www.itu.int/en/ITU-D/Environment/Documents/Toolbox/GEM_2020_def.pdf.
Jain, Muskan, Depak Kumar, Jyoti Chaudhary, Sudesh Kumar, Sheetal Sharma, Ajay Singh Verma, “Review on
E-Waste Management and Its Impact on the Environment and Society”, Waste Management Bulletin 1 (3),
2023, 34–44, https://doi.org/10.1016/j.wmb.2023.06.004.
Johnson, Ceridwen, “Soaring E-Waste Affects the Health of Millions of Children, WHO Warns”, World
Health Organization, World Health Organization, 2021,
https://www.who.int/news/item/15-06-2021-soaring-e-waste-affects-the-health-of-millions-of-children-who-warns.
Adam Minter, “The Burning Truth behind an E-Waste Dump in Africa”, Smithsonian, Smithsonian.com,
January 13, 2016,
https://www.smithsonianmag.com/science-nature/burning-truth-behind-e-waste-dump-africa-180957597/.
Julien Ottavi, “Climate Solutions: MOTHERPLANT”, www.youtube.com, Solution Zone TV, December
6, 2015, https://www.youtube.com/watch?v=CSUeSdR6n1Q.
Julien Ottavi, and Jenny Pickett, “Near-1 [[NEAR] Nantes Electronic Art Rencontre]”, Apo33.org,
September 22, 2015, https://apo33.org/near/doku.php?id=near-1.
Roberts Nic, Andrew Adamatzky, “Mining Logical Circuits in Fungi”, Scientific Reports 12 (1),
2022, https://doi.org/10.1038/s41598-022-20080-3.
Ajay Tomer, Ramji Singh, Saurabh Kumar Singh, S. A. Dwivedi, Chilkuri Udaykiran Reddy, Malavika Ram Amanthra
Keloth, and Riya Rachel, “Role of Fungi in Bioremediation and Environmental Sustainability”,
Fungal Biology, 2021, 187–200, https://doi.org/10.1007/978-3-030-54422-5_8.
Jürgen Wittsiepe, N. Fobil Julius, Till Holger, Gerd-Dieter Burchard, Michael Wilhelm, and Torsten
Feldt, “Levels of Polychlorinated Dibenzo-p-Dioxins, Dibenzofurans (PCDD/Fs) and Biphenyls (PCBs) in
Blood of Informal E-Waste Recycling Workers from Agbogbloshie, Ghana, and Controls.” Environment
International 79, June 2015, 65–73, https://doi.org/10.1016/j.envint.2015.03.008.
Authors Biographies
Julien Ottavi, holds a Doctor of Arts from the Université de Lorraine. His practice covers sound,
intermedia, and digital network arts. He is the founding member and artistic director of association APO33,
based in Nantes. APO33 has produced numerous ground-breaking explorations in experimental music and electronic
arts, and collaborated with artists and musicians across the world.
Jenny Pickett is a member of the APO33, where she works on projects ranging from interactive installations to
experimental music and performance to international collaboration in art and technology. She is currently a
PhD Candidate at Cyprus University of Technology where she is attached to the Media Arts and Design Research
Lab (MADlab), she is also the Vice Chair for Toolkit of Care a EU COST funded cooperation, as well as an
associate lecturer at the Nantes School of Architecture (ENSA).
Life-as-it-could-be, Symbiosis in Interspecifics’ Codex Virtualis_Genesis
ClaudiaCostaPederson
Wichita State University Kansas, U.S.A. ccp9@cornell.edu
Abstract
This paper discusses Codex Virtualis_Genesis (2020-2022) as an artistic engagement of Artificial Life
(AL) that explores the nature of informational life as a symbiotic process. Created by Mexico City-based
transnational collective Interspecifics (INT), this work follows on from the expanded notions of life central to
the field of artificial life, including organic, inorganic, material, and virtual forms. According to its
founder, Christopher Langton, “there is nothing... that restricts biology to carbon-based life; it is
simply the only kind of life that has been available to study.” From this perspective, Langton proposed in
the late 1990s that AL be dedicated, as he put it, to speculating beyond "life-as-we-know-it" into the
realm of "life-as-it-could-be.” This discussion examines Codex Virtualis_Genesis in light of
Langton’s proposal as a speculative inquiry into a symbiotic view of life, and as well in contrast to
notions of artificial life art as a predominantly technophilic practice. Instead connected to the speculative
imagination, the synthetic life forms of Codex Virtualis_Genesis offer a glimpse into life otherwise:
as an interspecific relation.
Keywords
Artificial Life, Artificial Life Art, Mexico, Symbiosis, Symbiocene.
Interspecifics’ CodexVirtualis_Genesis (2021-2022) was created for the SETI x AI Lab
residency, an art and science program started in 2018 by the Ars Electronica Futurelab in Linz, Austria in
collaboration with the AI Lab (European ARTificial Intelligence Lab) and the SETI Institute in Mountain View,
CA.¹ The program calls on international artists working in digital art to create works “that explore
the evolution of life on earth and universe and critically reflect on anthropocentric world views.”²
Interspecifics won the 2021 Ars’ AI Lab residency call with their proposal to create CodexVirtualis_Genesis as an evolving taxonomic collection of hybrid AL forms.³ In
practice, to create these forms’ shape, color, texture, morphological structures and behavioral dynamics,
Interspecifics used a 2D cellular automata system (a planar cell-based computer graphic system that displays
emergent global behaviors), and two sets of images of living symbiotic organisms. One set is microscope images
(microscopy) of microorganisms living in water bodies that were posted by various scientists on their Instagram
accounts.⁴ Another set was digital photographs of colonies of fungi and bacteria (8,000
distinct species) made available online by researchers at the National Agriculture and Food Research
Organization, NARO, in Japan. Some of the latter organisms are also known as extremophiles because they are
metabolically and biochemically adaptable to harsh environmental conditions such as broad swaths of temperature,
pH, pressure, radiation, salinity, energy, and nutrient limitation. They have therefore been historically used
in astrobiology experiments, including most recently conducted at the ISS (International Space
Station).(1)
The resulting images resemble plant and animal-like microorganisms in a wide range of colors and shapes (Figure
1a and 1b). In a gallery setting, Codex Virtualis_Genesis is shown as a screen-based installation that
as well as displaying selected images of said organisms, also displays images of aspects of the selection
process and evolutionary procedures (Figure 2).
Additionally, in the printed materials provided to visitors at exhibitions, Interspecifics describes Codex
Virtualis_Genesis as titled in a loose nod to Mesoamerican codices and based on “the symbiotic
narrative of evolution.”⁵ Specifically this refers to endosymbiosis, a process of
symbiotic life evolution proposed by the biologist Lynn Margulis in the late 1960s. Endosymbiosis foregrounds
symbiotic mergers (pre-nucleic gene transfer between bacteria and other organisms) as the main sources of
acquisition of new genomes and thus also of speciation (i.e., the formation of new and distinct species in the
course of evolution). From this perspective, random mutation is of marginal consequence as the driver of
evolution and so consequently Margulis argued that Darwinian evolutionary theory based solely on competition
(i.e., the survival of the fittest) is incomplete. According to Margulis, “Life did not take over the
globe by combat, but by networking. Life forms multiplied and complexified by co-opting others, not just by
killing them.”⁶
Interspecifics’ Codex Virtualis_Genesis brings Langton and Margulis in dialogue with an eye to
complement their respective perspectives on the nature of life. In this interest, it extends Margulis’
restricted notion of life (as carbon-based) to as well, as proposed by Langton, include synthetic forms.
Conversely, by linking AL and symbiosis it also clarifies Langton’s speculative vision of AL (i.e., as
“life-as-it-could-be”). (2) According to a member of Interspecifics, Leslie Garcia, this
dialogue speaks to the project’s overarching concept: namely to simultaneously call attention to our
“entanglements” and to stimulate speculative imagings and imaginaries of “non-anthropocentric
and non-anthropomorphic but more organicist, metabolic models of life.”⁷
Cooperate!
Codex Virtualis_Genesis not only demonstrates life as a mutualistic relation on screen but is itself
created with an eye for developing art and science as mutualistic practices: in Interspecifics’ own words,
“to produce in terms of social inclusion, cross-disciplinary practices, and open knowledge.”⁸
To this concern, the project contributes to broadening access to artificial life techniques and consonantly
increasing diversity in artificial life art as an area of the global digital arts. As aforementioned,
Interspecifics drew on freely available images shared by scientists and scientific communities online and social
media to create Codex Virtualis_Genesis’ symbiont organisms. Additionally, Lenia, a readily
available system of cellular automata or artificial life, is integral to the project. Created by the Chinese
software engineer Bert Wang-Chak Chan in 2015, this system is a variation on The Game of Life (GoL) devised by
the British mathematician John Horton Conway in 1970, which was originally played on a simple square grid
without the aid of computers. GoL is a class of mathematical phenomena termed cellular automata by John von
Neumann in his discussion of self-reproducing machines in the mid-1940s.(3) Both GoL and Lenia
are AL systems that focus on life not as a material substrate but as a form of organization and behavior that is
akin to the distributed processes characteristic of the functioning of living organisms. Alternatively, these
systems can also be conceived as no-player games; meaning that after the initial configuration their evolution
does not require further input. The point of cellular automata like GoL and Lenia is in short to demonstrate the
emergence of the variety and complexity of behaviors from a few simple rules. These systems differ insofar the
sophistication of their graphics. In comparison to GoL’s square graphics, which now look outdated,
Lenia’s fuzzy, smooth, and colorful patterns look state of the art. More significantly for artists
broadly, as art funding becomes ever more restricted, and in particular artists that as Interspecifics puts it,
work via-á-vis the context of “precarity” in the global South, Lenia is as an open-source art
and science project. Still requiring some technological expertise, it nevertheless is in theory accessible to a
global public.
Already, in an updated version of his 1989 article on Artificial Life, Langton speculated that the future
evolution of AL would depend on scientists’ imitating nature’s cooperative behaviors (so he urged
biologists and computer technicians to follow its example). (4)Artificial life art, as A-life artist
Simon Penny notes, is similarly an area of digital arts with a high degree of interdisciplinarians and/or
collaborative partnerships between artists and technicians because artificial life techniques are technically
demanding.(5) Because of the emphasis on technical expertise and state-of-the-art technology in the
field (what Penny calls “high nerd quotient” and the art historian María Fernández
calls digital art’s “aesthetic technofetishism”), artificial life art has been dominated by
artists working in the global North.(5)(6)⁹ Artists like Interspecifics are at the
forefront of developing artificial life art as an emerging area of the digital arts in Mexico along with a new
generation of artists in Latin America, including the Argentinian artists Leo Nuñez and Sofia
Crespo and the Peruvian artist Paola Torres Nuñez del Prado. Projects like Linea are significant because
they broaden access to AL, both beyond the sciences and the global North.¹⁰ It is to this
concern that Interspecifics, just as with all their projects, additionally made some of the research,
techniques, and coding created for Codex Virtualis_Genesis freely available online.¹¹
CodexVirtualis
CodexVirtualis_Genesis is yet but the first of the four parts of the total project proposed
by Interspecifics under the title Codex Virtualis. The collective is currently developing the second
part of this project, which is entitled Habitat involves testing the extremophile capabilities of the
organisms created in Genesis by exposing them to inputs that simulate the harsh conditions of
extraterrestrial environments. A third part, which is titled Emergence proposes testing the possibility
of intra-actions (transfers) between artificial and biological organisms. Lastly, Codex Virtualis: Life
involves testing the resilience of resulting hybrid organisms and accordingly selecting one organism to
live virtually in Codex Virtualis.¹²
As a contemporary artificial life artwork, CodexVirtualis follows on a longstanding quest
for mimesis or life-likeness that spans the histories of art, science, and technology, and as well includes
social, spiritual, commercial, intellectual, and military histories. Langton himself traced the origins of AL to
this pursuit in the history of art, starting with paintings and “statuettes” that “capture the
static forms of living things.” He goes on to discuss hydraulic technology such as Egyptian
water clocks or clepsydra as the next step towards imitating nature’s dynamic behavior. This
trajectory developed with the invention of the mechanical escapement in AD 850, ushering in the age of
clockwork technology. Refined throughout the European Middle Ages and the Renaissance, eventually, clockwork
regulation of mechanical devices would not only be used to power them but also to sequence their motions or
behaviors.¹³
Langton refers to this development as technology of process-control, which in the first instance involved
interchangeable cams or drums with movable pegs. A variety of programmable automata are examples, including
lifelike mechanical figures and animals such as the writing and picture-drawing automata built by the
Jaquet-Droz family in Switzerland in the 18th century. In the early part of the twentieth century,
during WWII and the Cold War, as Langton explains, physical programmable controllers gave away to abstract
control structures, or sets of rules or programs. In effect, as he puts it, this is the moment when the
“’logical form’ of the machine was separated from its material basis of construction.”
Today’s computers are a technology of process-control and in this sense the equivalent to an
“algorithm: the logic underlying the dynamics of an automaton.” (7)¹⁴
Artificial life art, which in the contemporary context is facilitated by computer technology, is consequently
heir to these legacies.(8) According to Penny, artificial life art in digital environments spans
three decades, beginning in the late 1980s, as artists turned to “a new type of interactivity” in
search of a “new order of mimesis in which ‘nature’ as a generative system, not as an
appearance, is being represented”. (5) To put it otherwise, artists became interested in
artificial life in pursuit of developing art contrary to ingrained notions of artistic practice. That is, to
create art not as an object but as a process, and in extension, to explore the role of the artist not as one
about control over materials, but as an initiator of an open-ended process. This initial focus on
unpredictability and novelty has since, however, been debated as an illusory pursuit, given that, as Penny adds,
“the mechanism of simulated evolution” appear in comparison to the limitations of biological
evolution more restricted; that is, as bound to the “conditions... established by logic-theoretic
enframing” of mathematical models. (5)
Yet, while Codex Virtualis_Genesis and its larger project, CodexVirtualis shares
interest in interactive aesthetics, this is not the project’s main focus. Its focus is rather apropos of
the curator and cultural critic Edwina Bartlem’s noting as part of her recent discussion of artificial
life art that “A-life discourse contains a prophecy of futuristic and imaginary posthuman or post-organic
life.” (9)Codex Virtualis speaks above all to this imaginary, as according to
Interspecifics, “’Codex Virtualis’ imagines ways of living together, of deep
interspecific relations that may enhance our possibilities of survival.”¹⁵ Garcia further
clarified the project’s concept on another occasion, as a proposal to imagine life and a new world beyond
the Anthropocene, consonant to “the Symbiocene.”¹⁶
Garcia’s reference is to a term coined by the eco-anarchist philosopher, Glenn Albrecht, the Symbiocene.
Symbiocene refutes the fatalistic implications of the Anthropocene, and, as largely inspired by Margulis’
notion of endosymbiosis affirms the possibility of creating a more sustainable world based on
non-anthropocentric mutualism. (10) In this light then CodexVirtualis engages
symbiosis as a metaphor and as a form of dialogue, debate, and exchange (i.e., between art, science, and
politics) for conceptualizing life in a sustainable posthuman age. Because of its ecological focus, it as well
in extension broadens the existing e imaginaries and themes engaged in artificial life art.
Notes
1 Interspecifics was founded in Mexico City in 2013. Core members are Leslie García, Paloma
López Ramírez, Emmanuel Anguiano Hernández, Felipe Rebolledo Carvajal, Carels
Tardío Pi, and Maro Pebo.
2 Ars Electronica is a cultural institute devoted to media art founded in Linz, Austria, in 1979.
A leading organization in the global media arts, as well as managing a museum, multidisciplinary art research
facilities, and an annual festival, it confers the most prestigious award in the genre, the Prix Ars
Electronica. Founded in 1984, the SETI Institute stands for the "search for extraterrestrial intelligence." It
is a non-profit dedicated to the study of life and intelligence in the universe through multidisciplinary
research, education, and partnerships with industry, academia and government agencies, including NASA and NSF.
3 Available at, https://int-lab.cc/codex/?page_id=2.
4 According to Interspecifics, the initial idea was for collaborating (SETI) scientists to share
their images with the group for the project. As it happened, the scientists failed to provide these images in
the end. Leslie Garcia, “Interspecifics, Codex Virtualis: Genesis,” lecture, “Neurotalk
‘Máquinas y seres vivos: Comunicaciones desde el arte’”, Programa ACT (Arte, Ciencia
y Tech- nologías), UNAM, March 2, 2022, available at, https://www.youtube.com/watch?v=NFE_F6HzakY.
5 Interspecifics, CodexVitualis_Genesis, exhibition booklet, 2021, 4, available
at, https://int-lab.cc/codex/?page_id=183.
6 Lynn Margulis and Dorion Sagan, Microcosmos: Four Billion Years of Microbial Evolution
(Berkeley, CA: University of California Press, 1997), 29. Margulis’ speculations have been
corroborated in the last decade as for example by scientists discovering that in ecosystems across the world,
there are immense, mutually beneficial associations of macrofungi with flowering plants in complex, positive,
metabolic, symbiotic relationship to each other. Findings such as these have scientifically overturned the
view that evolution and life are solely founded on competitive struggle between species.
7 Leslie Garcia, “Interspecifics, CodexVirtualis:Genesis,” lecture, “Neurotalk ‘Máquinas y seres vivos: Comunicaciones desde
el arte’.” It is worth to note here that Margulis’ notion of endosymbiosis implies a
critique of heterosexist articulations of life, (i.e., the singularly privileged place of heterosexual
reproduction in evolutionary biology), which presently resonates with Karen Barad’s notion of
“nature’s queer performativity”, which likewise denotes multispecies co-involvements as the
drivers of life’s heterogeneity. See, Karen Barad, “Nature’s Queer Performativity” in
Kvinder, Kønogforskning/Women, GenderandResearch nos. 1-2 (2012): 25-53.
9 Artists working with digital A-life systems mostly reside in North America, Europe, and
Australia, and include Karl Sims, Tom Ray, William Latham, Ken Rinaldo, Bill Vorn, Louise-Philippe Demers,
Troy Innocent, Jon McCormack, Robb Lovell and John Mitchell, Christa Sommerer and Laurent Mignonneau, Jane
Prophet and Gordon Selley, Paul Brown, Richard Brown and Mauro Annunzianto. See also, Fernández,
“’Life-like’: Historicizing Process and Responsiveness in Digital Art”; Sarah Kember,
Cyberfeminism and Artificial Life (London and New York: Routledge, 2003); “; Michael Whitelaw,
Metacreation:Art andArtificialLife (Cambridge, MA: MIT Press, 2004);
and Simon Penny, “Twenty Years of Artificial Life Art”, in DigitalCreativity
21, no. 3, Dec. 2010: 197-204.
10 To date, along with scientists, game designers, filmmakers and artists across the world,
including among them, Interspecifics, have used Lenia to develop their respective projects and in the process
added to the evolving diversity and complexity of behaviours of its life forms. According to Chan, to date,
more than 400 species in 18 families have been identified in Lenia. B. W.-C. Chan, “Lenia: Biology of
Artificial Life,” Complex Systems, 28(3), 2019, pp. 251–286. See also, Siobhan Roberts,
“The Lasting Lessons of John Conway’s Game of Life”, in TheNewYorkTimes, Dec. 28, 2020, available at, https://www.ny-
times.com/2020/12/28/science/math-conway- game-of-life.html.
11 See, https://int-lab.cc/codex/.
12 The duration, location, and conditions of this last stage are yet to be clearly specified as at
the time of writing, Interspecifics is still developing the second stage in the project, Habitat.
13 The Muslim inventor Ismail al-Jazari’s TheBookofKnowledge ofIngeniousMechanicalDevices (1206) is currently considered
by historians of science to be a significant groundwork for modern engineering, hydraulics, and even robotics.
The Book builds on science and wisdom from ancient Greek, Indian, Persian, Chinese and other cultures
and offers a how-to manual of sorts that includes alongside richly detailed illustrations as well as
instructions on how to build al-Jazari’s inventions spanning playful robots to practical contraptions.
14 Likewise, the Latine artist Micha Cárdenas has traced algorithmic culture to pre-digital
technology, citing recipes and rituals as examples. Additionally, Cárdenas reminds us that the word
algorithm is a derivation of the name of the scholar Muhammad ibn Musa al-Khwarizmi (780-850 A.D.), credited
with inventing algebra in his book Dixit Algorismus. Micha Cárdenas, PoeticOperations, Trans of Color Art in Digital Media (Durham, Duke University Press, 2022), 7.
16 Garcia, “Interspecifics, CodexVirtualis:Genesis,”
lecture “Neurotalk ‘Máquinas y seres vivos: Comunicaciones desde el arte’”
References
(1) Cortesão See M., T. Schütze, R. Marx, R. Moeller, V. Meyer, “Fungal
Biotechnology in Space: Why and How?” in H. Nevalinen, ed., Grand Challenges in Fungal
Biotechnology. Grand Challenges in Biology and Biotechnology, New York, Springer, 2020, 501–535.
(2) Christopher Langton, “Artificial Life,” in Ars Electronica: Facing the
Future, edited by Timothy Druckrey, with Ars Electronica, Cambridge, MA, MIT Press, 1999, 126-268.
(3) John von Neumann See, TheoryofSelf-ReproducingAutomata,
Urbana, University of Illinois Press, 1966.
(4) Christopher G. Langton, “Artificial Life,” in The Philosophy of Artificial
Life, edited by Margaret A. Boden, London, UK, Oxford University Press, 1996, 92-93.
(5) Simon Penny, “Art and Artificial Life-A Primer,” in the Proceedings of
DAC09, Digital Art and Culture 2009 conference, University of California, Irvine, 12–15
December 2009, unpaged, available at, https://escholarship.org/uc/item/1z07j77x.
(6) María Fernández, “Postcolonial Media Theory,” in ArtJournal 58, no. 3, Autumn, 1999, 66-69.
(7) Langton, “Artificial Life,” 41-46.
(8) María Fernández See, “’Life-like’: Historicizing Process and
Responsiveness in Digital Art,” in A Companion to Contemporary Art since 1945, edited by Amelia
Jones, Malden, MA, Blackwell Publishing, 2006, 557-581.
(9) Edwina Bartlem, “Immersive Artificial Life (A-Life) Art,” in Backburning:
Journal of Australian Studies 84, edited by Helen Addison-Smith, An Nguyen and Denise Tallis, Perth,
API Network, 2005.
(10) See, Glenn Albrecht, Earth Emotions: New Words for a New World, Ithaca, NY, Cornell
University Press, 2019.
Author Biography
Claudia Costa Pederson is Associate Professor of Art History, author of Gaming Utopia: Ludic Worlds in Art,
Design, and Media (Indiana, 2021); and co-curator of FLEFF’s new media exhibitions at Ithaca College,
NY, since 2017.
Disruptive Avant-Garde Art of Today: Shaping Post-Growth Imaginaries for Symbiotic Futures
KristinaPranjić,
MagdalenaGermek, PeterPurg
University of Nova Gorica Nova Gorica, Slovenia kristina.pranjic@ung.si,
magdalena.germek@gmail.com, peter.purg@ung.si
Abstract
In order to develop new symbiotic relationships and different imaginaries, it is first necessary to critically
restructure the representations of forms of cooperation, which in their positive, desired version usually
represent a certain romantic idea of nature and human, and the possibilities for a harmonious model and holistic
structure of reality. This can be seen in both eco art and activist ecological agendas, which often play on
feelings of harmony and mutual reciprocity, and actually further contribute to a distorted and extremely
one-dimensional image of reality. Using the concepts of conviviality and cosmpolitics, the article aims to offer
new concepts of symbiosis and symbiotic futures that face today's process of defuturing. The second point of the
article is to develop a convincing and solid alternative to the neoliberal view of market-driven models based on
competencies and the logic of growth. Therefore, the actual task for disruptive avant-garde art of today should
be understood as the decolonization of our imaginaries that perceive nature through the logic of growth and the
harmonious model in the direction of shaping post-growth imaginaries for symbiotic futures.
At the end of the epidemic, faced with another European war, and shattered by climate disasters, we seem to be
facing a similar if not even worse zeitgeist to the one of the twenties that called for the avant-garde
art and design school Bauhaus to bring a new social, cultural and creative impetus. However, back then there was
no experience of the failures of the 20th century artistic and social utopias and movements.
For this reason, we can rightly question any revolution or change through artistic and cultural means.
Nevertheless, new artistic expressions that use new technology and scientific methods to conduct true artistic
research are developing novel concepts and opening space for another horizon of experiences that can offer
images and ideas of a different future, in which it will be possible not only to survive, but also to thrive in
symbiosis with other human and non-human beings.
In order to describe the mentioned process through a deep understanding of how current art contributes to the
shaping of our imaginaries for the better future in the age of the Anthropocene, the article will first
introduce key concepts, such as “conviviality”, “symbiosis”, “defuturing”,
“imaginary”, “growth”, and “post-growth/degrowth,” that comprehensively
shape the paradigm, criticized by the artistic projects that are addressing the most burning issues of today,
and rightfully represent the contemporary avant-garde. In the last part of the article, we will show three
distinct examples of current artworks that seek to participate in transforming the human imagination in the
direction of symbiotic futures with a different system of relationship not dependent of industrial and
consumerist systems, rooted in the idea of continuous growth.
Living together: Conviviality and Symbiosis
The term conviviality was incorporated to the humanities vocabular by Ivan Illich, who in his work criticized
the industrial capitalism and productivity, to instead describe a path of human emancipation and autonomy for
reaching convivial life.¹ For Illich conviviality means an “autonomous and creative intercourse
among persons, and the intercourse of persons with their environment; and this, in contrast, with the
conditioned response of persons to the demands made upon them by others, and by a man-made
environment.”² Furthermore, because of his instance of the interdependence among living beings,
Ilich’s thought is particularly productive in the field of posthumanism to describe interdependence and
entanglements between different species, human and non-human agents.
The term “conviviality” comes from Latin convivere, which means “to carouse
together, live together” (com “with, together” and vivere “to
live”). The more common term “symbiosis” that comes from ancient Greek sýn and
bíōsis similarly denotes living together. Today both of these concepts are equally
important in the posthumanism research that emphasizes the importance of exploring relations between human and
non-human beings and other entities or matter, and insisting on the theoretical break with anthropocentrism and
with the dualisms between human/society and nature.
Conviviality does not mean only relationships of cooperation, but it also includes conflict and tensions as
constitutive and necessary part of convivial relations; human beings are understood as a part of the network of
players living in symbiosis with others. ³ Moreover: “To Illich, the word ‘conviviality’
does not mean joy or light-heartedness; it refers to a society in which modern tools are used by everyone in an
integrated and shared manner, without reliance on a body of specialists who control said
instruments.”⁴ For Ilich convivial toll can be used by an individual for a purpose
that they choose, and it needs to bring about more freedom, autonomy and creativity (for all beings).
Perceptions of harmonious interactions of symbiosis were critiqued by Isabelle Stengers’ research in
thermodynamics to include complexity, processes of disturbance and friction that cause a state of crisis and
lead to change.⁵ In her seminal work on “cosmopolitics” Stengers described a “symbiotic
agreement” as a part of an “immanent process of ‘reciprocal capture’;” she
describes “the event” of symbiosis as “the production of new, immanent modes of existence, and
not the recognition of a more powerful interest before which divergent particular interests would have to bow
down. Nor is it the consequence of a harmonization that would transcend the egoism of those
interests.”⁶
We can further find productive thoughts on this topic in Anna Tsing work on rare matsutake mushroom, in which
she presents the complex network of cooperation between human and non-human participants that rejects
stability as a central goal of relationships, and even stability of the term
“naturalness”.⁷ Stengers’ and Tsing’s work not only opens to question the
foundations of objective (natural and humanities) science, but also shows a new way of connecting and generating
knowledge and possible new imaginaries that can arise from different forms of human activity, which is not
necessarily bound to the rationality and authority. Knowledge and science can be, through specific practices and
processes, something that we not only discover, but also shape.
Facing Defuturing
To recognize what is, we first need to establish active knowledge about how our current dysfunctional
relationships of un-sustainment are sustained. One could assume it is truly difficult to be an avant-gardist in
the time of defuturing, as Tony Fry named our condition in his book Defuturing. A New Design
Philosophy.⁸ Fry introduced the concept of defuturing as the destruction of the future by design;
it means that we live in a world that is taking away futures for ourselves and non-human others. He criticizes
Eurocentric, anthropocentric and productivist structures of unsustainability by exploring the history of design
that so importantly yet often subliminally coordinates our activities, and influences our thought.
Because our conditions of existence drastically changed, Fry stresses that we also need to drastically change
our philosophy of existence and of acting. The new direction for making anything in the world is not a direction
within design as-is; within the already established paradigms; it is a direction beyond where design now is. It
is also a direction beyond where thought now is. The alternative to this process of “change so that
nothing changes” and covering of crisis, would be “informed futuring”, which basically refers
to finding the alternative and acquiring the agency in order to create something truly different, an actual
change.⁹
For this kind of futuring, a new avant-garde in doing and thinking is needed. Therefore, it is central to look
into the past to deconstruct and surpass it. The most significant signifiers of how we do and think things, and
how we ascribe value – is growth. Therefore, besides offering a complex image of symbiosis and
establishing new and different relationships, it is important to decolonize our imaginary of growth.
Decolonizing our Imaginaries
We can understand “growth” as a centuries-old matrix of thinking that historically constructs
Western-civilizational models of meaning-making, collectively and individually, from various institutional macro
and microforms of society. The logic of growth is thus not only rooted in the rational categories that shape our
concepts but also in the way we look at the world, how we describe, perceive, and represent it. The stubbornness
to fight growth by reformulating capitalist parameters – for example, by establishing concepts like
social, human, local, and sustainable “development” – can only lead to new ecological,
economic, and social failures.
Agendas such as the New European Bauhaus for example, and the UN 2030 Agenda for Sustainable development do
seem to be signs of a significant shift toward an awareness about social and environmental health, equality, and
justice. However, we need to be critical of the very idea of sustainable development, which remains part of the
Western economic imaginary of growth. We can find this critique especially in the works of Serge
Latouche¹⁰, who claims that the idea of sustainable development is a mystified and ideologized one
that does not bring about any radical turn in thinking but stems from the logic of a compromise.
Consequently, it is unavoidable to start “decolonizing our imaginaries,” a syntagma borrowed from
Latouche¹¹, who refers, on the one hand, to the concept of the imaginary by Cornelius
Castoriadis¹², and on the other hand, to the anti-imperialist concept of decolonization introduced by
anthropologists (e.g., Serge Gruzinski).¹³ If we follow Latouche’s insight, radical change needs
another economy; another view of science that would go beyond the concept of Promethean technoscience; another
conception of life and death; a different conception of wealth and poverty; another notion of time that would no
longer be linear, cumulative, continuous; other conceptions of space; other intergenerational and gender
relations; a different concept of work (placing social relations in the center, instead of, e.g., efficiency or
value accumulation)¹⁴. Latouche integrates this into an eight-point program with listed imperatives:
to re-evaluate, reconceptualize, restructure, redistribute, relocalize, reduce, re-use, and
recycle.¹⁵
Constructing the Imaginary of Post-Growth
One solid set of alternatives to the logics of growth was offered by the introduction of concepts such as
post-growth and degrowth. And even though the concept of degrowth as the most optimal future direction of our
global society is becoming more and more mainstream, it is clear we will not achieve any drastic changes in
human behavior if we do not address its phenomenological and epistemological implications that could provide us
with new models for everyday life.
Neutralizing the disastrous impact of our growth-based society would therefore mean actually using radical
avant-garde methods as a unique combination of chance and plan, happening and controlled work in all aspects of
thinking and creating for uncertain futures. We see the legacy of the historical avant-garde movements, such as
Dadaism and Surrealism, as a plethora of valuable concepts for thinking degrowth that is yet to be researched
and applied from the field of art exclusively to the area of social and environmental care, and precisely to
the way we make and do things. The highly experimental avant-garde presented the most radical
possibilities for changing the logic of thinking and perceiving reality, thus showing the opportunities to let
new forms and relationships emerge.
And although the avant-garde is all too often perceived as art that sought to demolish and destroy everything
that existed before it, it carries within it an extremely affirmative and constructive character: avant-gardes
invented a whole series of new formal languages and strategies that are still in use today as fundamental
techniques of our modern visual or multimodal communication (e.g., photomontage, collage, avant-garde cinema,
advertisements, graphic design typography, new architecture). They sought to formalize the coincidence and
freedom of expression and erase any automatization and pre-given cultural definitions.
The avant-gardists’ proposition appears to have been precisely what we need today for a symbiotic
“metamorphosis in being” as stated by Pasi Haikkurinen, connected with the “practice of
releasement” understood as the “new ethos” for degrowth
society.¹⁶,¹⁷
Artistic Disruption for Radical Innovation
We finally come to the acknowledgment of the importance of artistic disruption and art-thinking as the key
innovative methodology that may provoke radical change in order to create a meaningful difference from what was
(wrong). Which are those new (inter)media and investigative artistic practices involving cutting-edge
technologies, critically confronting the European social and cultural values with the aesthetics-, ethics- and
tech-related legacies of the historical avant-garde?
By way of conclusion, let us try to briefly discuss three of such topical cases selected among the newest
(intermedia) artistic practices:
Vladan Joler's recent hybrid animated info-graphic work entitled New Extractivism (2020) visualizes
the position of the individual as a user of contemporary information technologies, who is subject to systemic
and systematic corporate extraction. Thus, it reveals and critically discusses some of the most pressing issues
of modern platform capitalism. The printed materials, which the author considers to be a kind of sci-artistic
“assemblage”, are presented in the format of a poster and a brochure, and are accompanied by an
animation distributed online, supported by the author's own narrative, which he occasionally also delivers as a
lecture (performance). By combining the elements of a map and a guidebook, and accompanied by abundant and
extremely analytical notes, the work attempts to produce a symbiotic and fractal-like “blueprint of a
machine-like superstructure, or a super allegory.”¹⁸
Often relying on the approaches of speculative design and advanced computer graphics, as well as
info-animations or even physical spatial installations, the works of the interdisciplinary group
Disnovation.org not only vividly show the paradoxes of the contemporary consumer and hyper-informatized
society, but also try to contribute suggestions and tools for resolving them. In the Post Growth Toolkit
(2020), a set of so-called critical games, they offer a mix of scientific and speculative
literary-narrative tools, which should offer the players of this game (or the users of the toolkit) a set of
more sustainable solutions for everyday life. Post Growth Prototypes (2021) complement the mentioned
toolbox with critical (animated video) essays functioning as case studies of advanced and symbiotic concepts
such as solar income, radical energy transition, or transcending the Anthropocene via the post-growth paradigm
shift, introducing a new responsibility of man towards the biosphere.
As the last truly comprehensive example of an artistic practice that radically questions the human role within
ecosystems, we wish to discuss the set of terra0 projects. Since 2018 various prototype environments
have been built on the decentralized (peer-to-peer) blockchain platform Ethereum, which aims to provide
automated frameworks for the resilience of a given ecosystem. By establishing a “Decentralized Autonomous
Organization” on the upper layers of the earth to govern them, terra0 research team aims to
create technologically augmented ecosystems that are both more resilient and more capable of operating within a
predetermined set of rules in the economic sphere, as independent agents: e.g., the forest independently mines
cryptocurrency and decides how it will change its material base. The group believes that modern technologies
such as remote sensing and machine learning provide an opportunity to rethink existing inefficient governance
and regulatory structures. Moreover, they also seem to suggest how, with appropriate art-thinking and
speculative-design based assumptions and interventions, these could play a key role in creating a sustainable,
resilient, symbiotic and biodiverse future.
There are more such artistic practices and they bear witness to the engagement of art in order to answer the
most pressing questions of our time not only by representing what does not work, but above all to construct, in
an avant-garde manner, new symbiotic relationships with human and non- human agents that would contain the full
complexity of the symbiotic organization of different entities jointly working to sustain a functional
environment, without falling back into the old paradigms shaped by the imaginaries and logics of growth,
duality, harmony and stability.
References
1 Ivan Ilich, Tools for Conviviality, New York, Harper and Row, 1973.
2 Ivan Ilich, Tools for Conviviality, 11.
3 Michael Given, “Conviviality and the Life of Soil,” Cambridge Archaeological
Journal 28, 2017, 127–143.
4 Marco Deriu. “Conviviality,” in Degrowth – a new vocabulary for a new era,
ed. Giacomo D’Alisa, Federico Demaria and Giorgos Kallis, London, Routledge, 2015.
5 Ilya Prigogine, Isabelle Stengers, Order Out of Chaos. Man's New Dialogue with Nature,
London, Flamingo, 1984.
6 Isabelle Stengers, Cosmpolitics I, Minneapolis/London, University of Minneapolis Press,
2010, 35-36.
7 Anna L. Tsing, The Mushroom at the End of the World: On the Possibility of Life in
Capitalist Ruins, Princeton/Oxford, Princeton University Press, 2015.
8 Tony Fry, Defuturing, A New Design Philosophy, 2nd Edition, Bloomsbury, Bloomsbury
Publishing, 2020.
9 Tony Fry, Defuturing, A New Design Philosophy, 239.
10 Serge Latouche, Petit traité de la décroissance sereine, Paris, Librairie
Arthème Fayard, 2007.
11Serge Latouche, “Imaginary, Decolonization of,” in Degrowth – a
new vocabulary for a new era, ed. Giacomo D’Alisa, Federico Demaria, Giorgos Kallis, London,
Routledge, 2015.
12Cornelius Castoriadis, L'institution imaginaire de la société,
Paris, Éditions du Seuil, 1975.
13 Serge Gruzinski, La colonisation de l’imaginaire. Sociétés
indigènes et occidentalisation dans le Mexique espagnol (XVIe- XVIIIe siècle) Collection
Bibliothèque des Histoires, Paris, Éditions Gallimard, 1988.
1Corneius Castoriadis, L'institution imaginaire de la société,
Paris, Éditions du Seuil, 1975.
2 Marco Deriu, “Conviviality,” in Degrowth – a new vocabulary for a new era,
ed. Giacomo D’Alisa, Federico Demaria, Giorgos Kallis, London, Routledge, 2015.
3 “Disnovation.org”, accessed December 6, 2022, https://disnovation.org/index.php.
4 Tony Fry, Defuturing, A New Design Philosophy. 2nd Edition, Bloomsbury: Bloomsbury
Publishing, 2020.
5 Michael Given, “Conviviality and the Life of Soil,” Cambridge
Archaeological Journal 28, 2017, 127–143.
6 Serge Gruzinski, La colonisation de l’imaginaire. Sociétés
indigènes et occidentalisation dans le Mexique espagnol (XVIe- XVIIIe siècle) Collection
Bibliothèque des Histoires, Paris, Éditions Gallimard, 1988.
7 Donna Haraway, “Cyborgs and Symbionts: Living Together in the New World Order,”
in: The cyborg handbook, ed. Chris Gray, New York, Routledge, 1995, xi–xx.
8 Donna Haraway, Staying with the trouble: Making kin in the Chthulucene, Durham,
Duke University Press, 2016.
9 Pasi Heikkurinen, “Degrowth: A metamorphosis in being,” Environment and
Planning E: Nature and Space Vol. 2, issue 3, 2019, 528–547.
10 Pasi Heikkurinen, “Degrowth by means of technology? A treatise for an ethos of
releasement,”Journal of Cleaner Production 197, 2018, 1654–1665.
11 Ivan Ilich, Tools for Conviviality, New York, Harper and Row, 1973.
13Serge Latouche, “Imaginary, Decolonization of,” in Degrowth
– a new vocabulary for a new era, ed. Giacomo D’Alisa, Federico Demaria, Giorgos Kallis,
London, Routledge, 2015.
14 Serge Latouche, Petit traité de la décroissance sereine, Paris,
Librairie Arthème Fayard, 2007.
16 Ilya Prigogine and Isabelle Stengers, Order Out of Chaos. Man's New Dialogue with
Nature, London, Flamingo, 1984.
17 Isabelle Stengers, Cosmpolitics I, Minneapolis/London, University of Minneapolis
Press, 2010.
18 “terra0,” accessed December 6, 2022, https://terra0.org/.
19 Anna L. Tsing, The Mushroom at the End of the World: On the Possibility of Life in
Capitalist Ruins, Princeton/Oxford, Princeton University Press, 2015.
Drivers for Resilience in Cultural Organizations: lessons from the Montreal festivals in the face of the
COVID-19 pandemic.
NicolasRicci, MarineAgogué
Center for Management Science, Mines - Paris HEC - Montréal
nicolas.ricci@mines-paristech.fr, marine.agogue@hec.ca
Abstract
Crises such as the COVID-19 pandemic represent appropriate moments to innovate. Many organizations in the
cultural sector have thus proposed numerous changes in their activities trying to develop new forms of
symbiosis, bringing back the notion of resilience. Beyond its buzz word aspect, resilience has essentially been
associated with a set of organizational capacities to adapt and innovate in the face of a disruption in the
cultural environment, leaving little consideration to question the main drivers of resilience in cultural
organizations.
We propose then to study the adaptation of the Montreal festivals offer, building on primary data from 8
interviews with festival directors or managers and secondary data from internal and external documentation. We
therefore mobilize the concept of the business model to identify and discuss the drivers for resilience in
cultural organizations. We show a trend for festivals to come back to their formal business model despite the
deployment of different innovations and identify role and purpose as the two main drivers for the resilience of
festivals.
Finally, we call for a comparison with other cultural organizations to discuss the preserving and reconfiguring
aspects of their resilience.
Keywords
Resilience, COVID-19 Pandemic, Festivals, Cultural organizations, Business models, Purpose.
The COVID-19 pandemic and the associated restrictions have generated a disruption in the offer and consumption
patterns of cultural products⁴. The transition to online has allowed a favorable development for cultural
sectors that did not depend on the physical presence of their public. On the other hand, very dependent sectors
such as cultural events in a broad sense, have faced significant difficulties despite various innovations in the
format and distribution of their cultural offer.⁶
Internally, the teams—with a high turnover rate in some cultural events—have been working
intensively since the start of the pandemic to reinvent themselves while preserving their identity. This has led
to the emergence of a tension between identity and constrained innovation.¹⁰ We offer with this paper
to clarify how this conflict was expressed within cultural organizations by questioning a well-known concept in
management when dealing with adaptation under constraints: organizational resilience.
Literature review
Organizational Resilience as a development of capacities to adapt and innovate in crisis
period
From its original definition in material science to the first ones in psychology, resilience has been defined
through the idea of a capacity of adaptation to a shock.¹ In management science, the development of the
concept of Organizational Resilience followed the same logic. For instance, Gibson and Tarrant define resilience
as an “adaptive capacity and how we better understand and address uncertainty in our internal and external
environments.”³ This observation can be confirmed by systematic review on the concept of
organizational resilience.¹¹
The essential of the literature thus focus on the description of this capacity. Begin and Chabaud¹ propose
a typology to describe the dimensions of organizational resilience:
An ‘absorption capacity’ associated with the survival efforts of an organization.
A ‘renewal capacity’ describing a reflective moment from the actors.
An ‘appropriation capacity’ referring to the new knowledge developed from the crisis.
Organizational Resilience as a reasoning moment for organizations
However, building on Kraemer definition of resilience in social work studies⁷, resilience can also be
defined with a phenomenal lens rather than the notion of capacity. According to the author, an acceptable
description of resilience depends on the context of your study (cultures, time, people, etc.) Thus, the fact
that resilience involves a phenomenon and a cognitive process from the actors to tackle it, constitutes the
common point between every form of resilience.
Organizational resilience can then be described through a succession of moments² illustrating successive
cognitive processes.
By considering a psychodynamic approach of resilience, Winkler¹² investigate actors thinking and
reasoning during a crisis period, leading to the consideration of individual drivers for resilience.
Yet, we lack such an approach of organizational resilience allowing us to identify the phenomenon of
resilience⁵ and discuss the main drivers for the reasoning of actors from cultural organizations in crisis
periods.
Methodology
Field of study: Why considering festivals to reflect the phenomenon of resilience within cultural
organizations?
Among the variety of cultural organizations, we propose to study festivals. Their activities deeply depend on
the physical gathering of publics and artists in a same place. Since the start of the pandemic, festivals have
faced deep changes in their business models and modified their formats in digital forms or cancelled their
editions and constitute then an interesting panel of forms of resilience. Moreover, festivals play a central
role in the exhibition of several artworks and represent a real platform in the life of cultural ecosystems in
general.
Data collection
We started by building a database registering most of the different festivals of Montreal (n=71) and
information found on websites and social media such as attendance, followers, period of the year, duration, type
of festival (i.e., music, cinema, visual arts, etc.), format of 2020/2021/2022 editions.
We contacted 68 of these festivals through direct mail to directors, contact mail, LinkedIn.
We interviewed 8 directors and managers between December 2021 and June 2022, with a variety within the
festivals in terms of size, duration, and types. The semi-structured interviews lasted between 30 minutes and 1
hour, we developed themes such as the adaptation of the festival team, the format before, during and after the
pandemic, their relations with the different stakeholders.
We completed this primary data with internal documents (audience studies, annual reports, internal
communication) provided by the directors and managers to complete our study. We also collected information from
websites and local press articles.
Data analysis
This study follows the principles of a grounded theory: we use data collected directly in our field of study to
develop intermediate theorization. To do so, we proposed an open code of the interviews content.⁸
We identified two major categories in the content: one part referring to the business model of the festival
(format, stakeholders, funding, etc.), a second one relative to remarks on the role of the festival, its
purpose. Then, we proposed to encode the information on the business model in canvas format for business
models.⁹ We then identified several codes for the drivers of reasoning that we gathered into two concepts:
“role” and “purpose.”
Results
Comparison of the Montreal festivals business models before, during and after the COVID19 pandemic: a trend for
a “back to normal” Gathering our coding from each festival interviewed, we propose three general
business model of Montreal festivals before, during and after the pandemic.
Pre-pandemic business model.
We built the pre-pandemic business model (Figure 1 (1)) around a value proposal common to all the festivals we
interviewed: offer a moment and a place for exhibit of specific artworks to a public. We identified a general
trend for the revenue streams (public fundings, private sponsors, event sales) and the cost structure (rent of a
physical place, equipment and furniture for the event, salaries). Both interviews and secondary data helped us
to define the public of festivals, channels, and public relationship management practices. We finally
highlighted recurrent public and private partners, key activities (artistic program, event organizing, public
management) and key resources (artists, representation place, festival team and volunteers).
Pandemic business model.
The pandemic business model evolved on almost every aspect (Figure 2). Most festivals turned into a digital
format, requiring a reconfiguration of resources and skills around online broadcast technologies. The value
proposal remains unchanged, illustrating the lack of will to change the concept of festival and their identity.
Nevertheless, the business model is only sustainable thanks to public support.
Post-pandemic business model.
The post-pandemic business model regains almost all its pre-pandemic properties (Figure 3). The digital turn is
hardly preserved. However, there is a desire to keep the public engagement tools that proved to be rather
effective during the 2020 and 2021 editions, such as additional content or contests.
Role and purpose as a driver for the resilience process
The comparison of the different business models reveals a clear absence of evolution in the value proposition
of festivals. The festival directors insisted on their wish to preserve the identity of the festival, directly
linked to the idea of “festival as a platform for artworks and artists promotion.”
Some directors admitted that they think a festival mainly for artists rather than for the public. The
willingness to keep strong links with the artists illustrates the role of festivals as a service for artists in
both exhibition and socialization and its associated benefits (collaboration, inspiration, artistic movement
evolution, etc.). As a result, the key point for the shift to online services delivered during the pandemic was
to preserve both moments of exhibition for public and moments between artists and practitioners.
Nevertheless, online festivals remained an unsatisfactory solution (not only for economic reasons), revealing
purpose as the second driver of the resilience of festivals. The directors associated the digital format to the
lack of informal moments and places for both festival teams, artists and public. The significant part of
informality and materiality in these events leaves little adhesion for remote formats, making the physical
format part of the festival purpose.
Discussion and conclusion
Our results show that role and purpose are fundamental drivers of resilience within festival teams involving
that:
Business models are intended to preserve the value proposition that
can then be described as representative of the festival's identity
The will of the teams is to "return to normal" as soon as the
restrictions linked to the pandemic are over
In sum, we were able to highlight a preservative dimension of organizational resilience in the case of
festivals. This leads us to question the different forms of preserving resilience that the COVID-19
pandemic has brought to the forefront in the cultural and artistic communities. Conversely, it is quite possible
to imagine forms of reconfiguring resilience in other cultural sectors, the study of which could allow
for an interesting theoretical confrontation.
Finally, we have seen that the business model is a suitable
tool for diagnosing and discussing the phenomenon of resilience, and that rather than an approach based on
adaptive capacities, we can think of resilience in terms of the drivers of the actors whose role and purpose
constitute.
(1) Figures of the business models are located at the end of this document
References
1 Bégin, Lucie, Didier Chabaud, "La résilience des organisations. Le cas d’une
entreprise familiale," Revue française de gestion 200 (1), 2010, 127-42.
2 Conz, Elisa, Giovanna Magnani, "A Dynamic Perspective on the Resilience of Firms: A Systematic
Literature Review and a Framework for Future Research," European Management Journal 38 (3), 2020,
400-412, https://doi.org/10.1016/j.emj.2019.12.004.
3 Carl A. Gibson, Michael Tarrant. s. d. "A “Conceptual Models” Approach to
Organisational Resilience," The Australian Journal of Emergency Management 25 (2), 6-12,
https://doi.org/10.3316/informit.084520139241216.
4 Julia V. Gnezdova, Vladimir S. Osipov, et Igor V. Hrip- tulov, "Creative Industries: A Review of
the Effects of the COVID-19 Pandemic," In Post-COVID Economic Revival, Volume II: Sectors, Institutions,
and Policy, édité par Vladimir S. Osipov, 2022, 159-71, Cham: Springer International
Publishing. https://doi.org/10.1007/978-3-030-83566-8_10.
5 Julia Hillmann, et Guenther Edeltraud, "Organizational Resilience: A Valuable Construct for
Management Research?" International Journal of Management Reviews 23 (1), 2021, 7-44,
https://doi.org/10.1111/ijmr.12239.
6 Olena Khlystova, Yelena Kalyuzhnova, Maksim Belitski, "The Impact of the COVID-19 Pandemic on
the Creative Industries: A Literature Review and Future Research Agenda," Journal of Business Research
139, février, 2022, 1192-1210, https://doi.org/10.1016/j.jbusres.2021.09.062.
7 Sebastian Kraemer, "Promoting Resilience. Changing Concepts of Parenting and Child Care,"
International Journal of Child & Family Welfare 4 (3), 2000, 273-87.
8 Ann Langley, "Strategies for Theorizing from Process Data," Academy of Management Review
24 (4): 1999, 691-710, https://doi.org/10.5465/amr.1999.2553248.
9 Alexander Osterwalder, Yves Pigneur, Business model generation: a handbook for visionaries,
game changers, and challengers, Vol. 1, John Wiley & Sons, 2010.
10 Michael G Pratt, Majken Schultz, Blake E. Ashforth, Davide Ravasi, The Oxford Handbook of
Organizational Identity. Oxford University Press, 2016.
11 Tennakoon Niranjala, Nadira Janadari, "Organizational Resilience: What it is and what it
isn’t? A Conceptual Review," Wayamba Journal of Management, 12 juillet, 2021, 171,
https://doi.org/10.4038/wjm.v12i1.7520.
12 Astrid Winkler, "Resilience as Reflexivity: A New Understanding for Work with Looked-After
Children," Journal of Social Work Practice 28 (4), 2014, 461-78,
https://doi.org/10.1080/02650533.2014.896784.
Metabolism and Art
HannahRogers, AdamBencard
University of Copenhagen, Denmark hannahstarrogers@gmail.com, adam@sund.ku.dk
Abstract
Metabolic arts should be added to the emerging interdiscipline of metabolic humanities and this paper will
discuss ways of defining metabolism that might be productive in helping to produce tools and touchstones for
metabolic readings of contemporary art before presenting examples of artworks which might be interestingly
illuminated by light of this sign, taking time to relish the process of these materially oriented internal
analysis coupled with how the work might be considered in terms of its broader implications for the concept of
metabolism.
Among the many challenges of our Anthropocene moment, is the new attention that must be paid to how humans and
other species are embedded within, exposed to, and even composed of the very materiality of a planet that is
rapidly changing. By focusing on metabolism, understood as “the chemical processes that occur within a
living organism in order to maintain life; the interconnected sequences of mostly enzyme catalyzed chemical
reactions by which a cell, tissue, organ, etc., sustains energy production, and synthesizes and breaks down
complex molecules” (Landecker 2011). As a fundamental biochemical process of life, metabolisms are
ubiquitous and multilayered, ranging from distinct multispecies bodily processes, to a variety of chemical
transformations across many organisms on a planetary scale, atmospheric and respiration relations between human
and plant metabolisms, the body as an environment, to notions of metabolism in a more purely metaphorical sense
in aesthetics. Tracing the workings of metabolism is a method for connecting change and dysfunction on a bodily,
social, and earth-wide scale. As John Bellamy Foster (1999) explains, Marx conceived of a dysfunction in the
form of a metabolic rift as a separation “between humanity and the soil, reflected in the antagonism of
town and country” (399).¹
Metabolic humanities appear to be a rising sign under which medical humanities, environmental humanities, and
agricultural corners might be united, and metabolicarts may be a productive stage for the three-legged stool of
ASTS to balance in tension around the subject.²
Metabolism might be taken to mean many things from distinct multi-species bodily processes, to a range of
chemical transformations across many organisms on a planetary scale, atmospheric and respiration relations
between human and plant metabolisms, the body as an environment, to notions of metabolism in a more purely
metaphorical sense in aesthetics. This is not to suggest that metaphor is absent in what is generally understood
to be the scientific notion of metabolism, indeed sociologist Hannah Landecker has shown how very entangled
social conditions and metabolic science are and have been.³ Landecker suggests that metabolism was
foreclosed by understanding the body as a machine, with a ledger, but this could also point to understanding the
body as a business, with immediate implications about capital and a relationship to Foster's Marx.
Metabolisms are ubiquitous but may be most noticeable in states of dysfunction. These dysfunctions are the
basis of medical sciences, the source of new ways of conceiving of ecological relations in a climate-changed
world, and are a longstanding way of diagnosing philosophical indigestions. Metabolism can be the constant that
brings methodological tendencies to the fore as a shared subject, and yet the variable historical understandings
of metabolism alone give rise to ages possible metabolic unfoldings so that our studies need not be bounded by
the current state of metabolic science and indeed, as ASTS scholars would argue, this is best understood in its
social, political, philosophical, and historical contexts. Metabolism serves so many possible needs for
scientists, scholars, and artists by providing metaphors, models, puzzles, solutions and balances. Metabolic
science is built on a stack of ever-modified metaphors,⁵ including the metabolism as an engine or motor
(fast/slow and often an emphasis on the notion of fuel/energy sources), furnace (hot/cold), “chain
reactions,” “chemical cascades,” the “chemical carnival,” and many others.⁶
Science is implicated in metaphor thinking, as even the classic experiment asks us to make correspondence
between the specific findings on the bench and the broader world, and this extends out to our public
understandings of science, often with further analogies which both simplify and make more culturally complex
these concepts. We suggest a focus on Landecker’s etymology for the term Stoffwechsel, translated as
“total metabolism,” with a further emphasis around the "stoff'' or in English “stuff” of
the process to emphasize the insistence bioartists have shown in exhibiting the stuff of living things and
parts.⁷ This emphasis on stuff has been important in bioart as such work has avoided distance and
representation except by a part of the actual living tissue, bacteria thing and at the same time it has insisted
on pulling against conceptual art to attempt to produce in material those ideas. Being in the presence of stuff
has been a hallmark of bioart and an emphasis on process may well be the hallmark of the metabolic arts.
The dual use Marx and Landecker make of, on one hand, critiquing soil chemistry or medical genetics, and on the
other hand, using those subjects as the basis for their thought, is a situation shared by bioartists who are
often using the very biological and biotechnical materials they are critiquing. Enter metabolic arts: drawing on
metabolism from the sciences, embodied experience of metabolism, and the potential of laboratory and home
metabolism practices to create encounters with life and life processes, many artists and art-aligned
practitioners have created art and art-science works that relate to metabolism. We observe and anticipate that
many artists may return to Marx's original thinking about metabolism, particularly, his interest in the way that
capitalist systems attempt to disentangle plants from animals including people, one natural and functional
system was separated into two sets of problems with planetary implications: urban/rural waste and food
insecurity/soil depletion.
Metabolic arts
The metabolic gaze can enrich understandings or experiences of a range of contemporary life-science engaged
artworks. Tissue Culture & Art (TC&A), made up of Oron Catts and Ionat Zurr, along with collaborators
specific to individual works, have offered a trajectory of artworks spanning from 2016 to the present, including
iterations of COMPOSTUBATOR and Sunlight Soil and Shit (De-)Cycle (3SDC).⁸
While each work is distinct, they represent the formation of the artists’ thinking on these subjects over
a number of instantiations. In the COMPOSTUBATOR series, a compost heap powers an incubator for
sustaining a group of cells, generally through microbes heating water in conduit which flows past the incubator
chamber and regulates the temperature. 3SDC was designed as a series of engagements around a set of
agricultural tools with an emphasis on considering their implications and the philosophies behind these and
other proposed system changes. 3SDC was launched with the COMPOSTUBA TOR in Freemantle, Australia,
positioned by the artists on their project website in relation to agricultural systems: “The heated
incubator sustains the growth of cells in a tissue culture flask to create what is today known as
“lab-grown meat.” This type of “meat” is the cornerstone of what is called Cellular
Agriculture – growing animal products without the animal.” A second portion of the project is the
use of Alkaline Hydrolysis (also Aquamation or Biochemiation), based on an 1888 process for turning
farm animals into crop fertilizer. The artists created their own version by hacking brewing equipment to break
down much smaller animal bodies (meat and fishing waste) to fertilize the Hydroponic Garden, which is
ironically supported by artificial lights driven by solar panels when possible. All of these elements are
connected by the Control Room which gathers data from sensors across the project (thermometers,
CO2 levels, pH monitors, cameras, etc.) The project aims to highlight the problem of increasing metabolic rifts
in order to solve agricultural issues, something that seems implicit in many of the lab-based food systems
proposed in the public sphere today. The artists write: “SymbioticA’s 3SDC builds resources to
enable the community to accelerate metabolic rifts in agricultural innovation. This project considers whether
the precursor to sustainable food systems will be the creation of a metabolic rift – where the means of
production will grow ever distant from nature.” The artists directly invoke Marx’s metabolic rift as
the target of their investigations by explaining that the exhibition is durational and will be changed over the
course of the exhibition. Their aim at “maintaining the utmost clarity and transparency of our process is
the key topromote understanding of the impact of metabolic rifts.” TC&A are working with metabolisms,
in its nuanced form at the microbial level and at the broader level of our food system.
Baum & Leahy’s Cometabolise: A Holobiont Dinner (2021) was a living sculpture and an
exploration in making the idea of the holobiont more familiar for viewers (Figure 1).⁹ A holobiont is an
assemblage of a host and other species living in or around it that together form a discrete ecological unit.
From a holobiont perspective, our bodies are permeable living environments for our cells and the cells (and
whole bodies) of other living things. It expands and blurs the notion of the host as a unit which could ever be
extracted from this entangled set of relations. The artwork insists on the overlapping metabolic processes of
the multispecies beings which help to metabolize our food. The artwork emphasizes the idea that bodies are
porous, multispecies entities, highlighting the fact that humans and microbes eat and drink together. This
bespoke dining set contains a sourdough culture held in a spherical glass carafe reminiscent of a bioreactor. A
closed container for a starter culture is, of course, one of our most familiar bioreactors. The piece reminds us
of the domestic nature of metabolism and invites thoughts of kitchens and laboratories. The artists emphasize
the performative nature of the work as the microbes are constantly metabolizing and are fed while visitors are
offered bread baked from the starter at the communal dining set.
George Gessert and Violet Ray’s BREATHE (2022) is a video piece investigating plant metabolism
and its circular processing with our human breathing. Given his history with plants as a primary subject and
medium for his artwork, Gessert’s work with plant metabolism is an obvious extension of those concerns.
The yet unexhibited filmic work which was created with Gessert’s longtime interlocutor but new
collaborator media artist Violet Ray, poetically explores the metabolism of plants through photosynthesis by
exhibiting the process at a cellular level and using text to invite audiences to connect their own breath to the
cycle of the plant’s photosynthesis and respiration. The artists ask us to pace ourselves with plants. As
Gessert puts it, as we think of metabolism, “Why stick to humans and animals? Photosynthesis creates the
air we breathe and is a key part of the planetary metabolism that supports most life on
earth.”¹⁰ This focus on the larger cycles that metabolism is implicated in overlaps with the
concerns of scale that appear across the metabolic arts.
Tagny Duff’s Wastelands (2015-2018) explores shit as an energy source in a speculative future
without fossil fuels. Duff explores a deep future 500 years away when humans, through collaboration with
bacteria and viruses, use their own feces as an energy source in small, portable bioreactors.¹¹ The
Wastelands Project relies on Duff's many years of experience working with biotechnologies in its artistic
practice, with a particular focus on viruses and on White Heather Hunter’s co-invention of a new
bioplastic with art conservator Courtney Books. The latter provided the basis for the biomaterial development
used to construct the bioreactor bags for the project. Metabolic ubiquity can create complications since living
things always involve metabolisms but all art with living things may not query those processes or engage them
directly. Yet, the search for metabolisms may provide new insights and new places of tension for contemporary
arts about life and the Anthropocene.
References
1 J. Foster, “Marx’s Theory of Metabolic Rift: Classical Foundations for Environmental
Sociology,” American Journal of Sociology 105, no. 2, 1999, 366–405.
2 H. Rogers, M. Halpern, Hannah D., K. Ridder-Vignone, The Routledge Handbook of Art,
Science, and Technology Studies, London, Routledge, 2021.
3 H. Landcker, “Food as exposure: Nutritional epigenetics and the new metabolism,”
BioSocieties 6 No. 2, Palgrave MacmillanUK, 2016, 167-194.
4 H. Landecker, “The Metabolism of Philosophy, in Three Parts,” In: Cooper, I. &
Malkmus, B., (eds.), Dialectic and Paradox: Configurations of the Third in Modernity, New York,
Peter Lang, 2013.
5 M. Black, Models and Metaphors, Ithaca, Cornell UniversityPress, 1962.
6 Taylor C See, B. M. Dewsbury, “On the Problem and Promise of Metaphor Use in Science and
Science Communication,” Journal of Microbiology & Biology Education, 19(1), 19.1.46 (2018),
Rabinbach, A. The Human Motor: Energy, Fatigue, and the Origins of Modernity, Basic Books, 1990.
7 A. Bencard, Stofsk(r)ifter: Metabolic Machines by Thomas Feuerstein. Hauser, J.
(ed.), A. Bencard (ed.), M. Grünfeld, L. Whiteley, 1 ed. Copenhagen, Medicinsk Museion, Jun 2020.
8 See the Tissue Culture & Art’s pages at Compostcubator 0.1 & 0.2 – The
Tissue Culture & Art Project (tcaproject.net) and 3SDC Sunlight Soil and Shit, sunlightsoilshit.systems.
9 For more, see Baum & Leahy’s art pages at Baum & Leahy, baumleahy.com.
10 Personal correspondence with the artists, February-April 2022.
11 For more, see Tagny Duff’s project page at Wastelands – Thoughts, images and
experiments considering human-microbial relations on Earth in 2517, wordpress.com, and Whitefather
Hunter’s documentation page at biotechnofeminism: laboratory craft | whitefeatherhunter
Ephemera: Bubble Representations as Metaphors for Endangered Species
HarpreetSareen1, YiboFu2, YasuakiKakehi3
1The University of Tokyo, 2,3 Parsons School of Design Bunkyo City,
Tokyo, Japan, New York, NY USA sareen@xlab.iii.u-tokyo.ac.jp, fuy644@newschool.edu, kakehi@iii.u-tokyo.ac.jp
Abstract
The effects of a hierarchical relationship of humans with non-humans are now more pronounced than ever.
Anthropogenic ecological stressors, including high levels of carbon dioxide, water scarcity, habitat
fragmentation have led to disruption of climate systems, in turn endangering many local and global species.
ephemera is an installation composed of glass vessels that show bubble images representing animals from all
continents and ecologies currently under threat as per the IUCN Red list. These self-assembling bubble pictures,
formed by nucleation of CO2 bubbles in water, are in a homeostasis at the beginning of the installation and
shrink each hour to eventually disappear in a few days. The tension between the present endangerment and the
urgency of the future action, manifests in the shrinking of these bubbles, invoking unnatural ephemerality due
to the human effect. The fauna pictures in this installation, composed of carbon dioxide bubbles, symbolize the
transitoriness of now threatened species.
In 2015, Hawaiʻi was called as the extinction capital of the world.¹ Occupying a small land mass
(0.25%), the islands had 25% of the endangered species in the US.² In intertwined ecosystems, direct and
indirect effects human activities—monoculture, greenhouse emissions, extraction, and urbanization—on
the biological diversity in a different location are significant³ but often hidden from sheltered lives of
humans in another place.⁴, ⁵ Separated from nature in this manner, our thinking of
endangered species is cultural, and public engagement with endangered species going forward depends on new
structures of imagination. How then should we invest culturally in the fate of endangered species? What emotions
do we collectively engender?
This work, an installation composed of ephemerality and melancholy, focuses not particularly on the beauty of
imperiled fauna but on their current transience. Glucksmann’s philosophy of the ephemeral⁶ is
particularly relevant where the moment is not static, but modulated and resonant. Drawing parallels with
flux-images, where the process takes precedence, Glucksmann calls out the aesthetics of the ephemeral as fluid
and polysensorial. In the context of this installation, images of various endangered species form with carbon
dioxide bubbles—bubbles that shrink in a few days gradually waning the entire image. Materials within this
installation’s cultural context are designed not for absolute control but for its relegation.
Concept
Jean-Louis Boissier called out the roles of images as an interface,⁷ which we can act upon, manipulate,
and transform—an image that vectorizes a relationship. We focus on such a relationship through evocative
material with embedded cultural context of greenhouse gases and their effect on fauna. Specifically, the focus
is on the ephemerality of the images. In prior transient works
⁸,⁹,¹⁰,¹¹,¹² durability
is determined by the intrinsic material properties in combination with the surrounding ecosystem, often
deliberately designed to offer only partial or imperfect control. Material semantics - original meaning of a
material - shape the perception and overall experience. The ambiguous ontological status of the ephemeral also
makes it a “powerful metaphor for expressing nuances of memory, time and knowledge.”¹³
Interpretation of ephemeral meanings and consequences is thus beyond the object itself.¹⁵
In this work, images of endangered animals form on glass vessels through thousands of carbon dioxide bubbles.
These bubble formations shrink by the hour to eventually disappear in a few days. The bubbles are formed by
nucleation of CO2 bubbles in water and are representational images of animals from continents and ecologies
currently under threat.
The installation is composed of ten glass vessels that are hydrophobically treated for controlled
micro/nanostructures on surfaces. When carbonated liquid media is poured onto the surface, bubble generation is
activated and bubbles of carbon dioxide of various sizes stick at defined points on the surface. Inspired by
Sylvester et al., who mentioned bubbles as a material¹⁶ between “neither real nor fully
virtual,” we extend the temporality of bubbles in this work. The process from initial nucleation to
stabilization takes an average of twenty minutes, depending on the saturation of carbon dioxide in the liquids.
Bubble images and patterns are stable for one day before they start to gradually shrink, completely disappearing
in five days.
System Design
We created a novel fabrication technique to selectively modify hydrophobicity properties of surfaces
(glass/plastic) such that bubbles in carbonated liquids nucleate at specific positions on surfaces.
This turns stochastic nucleation into controlled nucleation of bubbles for images, patterns, and text on
various surfaces. Such surface modifications are initially invisible to the human eye. On pouring carbonated
water in containers of various shapes (horizontal/vertical) and sizes, bubbles nucleate, coalesce, and grow to
accurate sizes (1.0 mm – 5.5 mm) thus creating patterns, images, or text on surfaces. Such bubbles are
highly stable and can keep sticking to the surfaces for longer than a week without significant disturbance.
To fabricate the glass vessels, we use Regular (Home Depot, 599047) glass as a substrate which is initially
hydrophilic (wettable) in nature. An illustration software (Adobe Illustrator) or our custom design tool is used
to create desired patterns and print out an image mask using a vinyl cutter (Cricut Maker 3). The sticker mask
is pasted on the surface and a ceramic hydrophobic coating is applied manually on the glass surface using a
zigzag technique before drying for 24 hours. This creates a surface with distinct wettability regions that are
invisible to the human eye.
When carbonated water is poured into the containers, small bubbles nucleate first coalescing into each other
becoming large in size corresponding with the size of the coating on the surface. These large bubbles may become
buoyant at first due to the saturation level of CO2 in the liquid. As levels of gas normalize, bubbles stabilize
at nucleation sites adhering to the hydrophobic regions with an attractive force¹⁷. Such stabilized
configurations of bubbles last for five to seven days.
To design the bubbles patterns for this installation, we create a custom tool based on p5.JS that allows
automatically generated wettability patterns by consolidating image import (color or grayscale), grayscale
conversion and corresponding bubbles pixelation in a single pipeline. User imported images are analyzed against
the background. Our algorithm analyzes the input image to create a comparative brightness map, following which
darker areas of images are tagged as proportionately dense in bubble density and lighter areas are sparse in
bubble density. For easy accessibility, the tool may be used online and outputs a .svg directly compatible with
the fabrication machine for cutting the masks.
Exhibition and Experience Walkthrough
Purpura et al. have previously called ephemeral art been as “good to think with.”¹⁴
Their impermanence is a constitutive part of their aesthetic, and of the ways in which they come to act on the
world. In the context of the exhibition shown at a major venue, our intention was to generate emotions of
sadness or mourning among the viewers, over the endangerment of species. We reviewed the IUCN Red List of
imperiled species at various threatened levels and chose ten species from marine, land and amphibious domains
across all continents. The vessels were fabricated as per techniques described before, bottom lit for contrast
of bubbles in liquid and setup in an L-shape. These were specifically separated from each other and laid on the
floor for two key purposes: a) For viewers to bow their heads down or kneel as if paying homage, and b) To
portray the vessels as final mementos of threatened species.
Such an exhibition design was shaped with a perspective of higher-levels of abstraction, focusing on carbon
dioxide bubbles as material from a cultural perspective. This work is thus not temporary, rather has a directive
intent to survive in the memory.
Conclusion
We presented ephemera, an installation where ephemeral images of endangered species are composed
through bubbles of carbon dioxide. Bubbles with extended lifetimes slowly disappear in five to seven days,
representing the images of endangered species as final mementos for humans. Through our exhibition, we intended
the viewers to observe images in a drifting state and to be involved in the relationship depicted by the image.
Glass vessels arranged onto the floor and viewed by the audience while kneeling are meant to invoke mourning or
sadness among viewers—emotions that were listed over aspects of nature during the rapid modernization
progress. Through this work, we explore a new ephemeral material and its aesthetic of affect, and believe that
the ephemerality in this work represents a shift from the art object to a communicative act.
Acknowledgements
This work was initiated and primarily supported by JSPS KAKENHI (Grant Number 20H05960). We also thank the
Provost Office Fund at The New School, and Labex Digi-Cosme (ANR11LABEX0045DIGICOSME) operated by ANR as part of
Idex Paris-Saclay program (ANR11IDEX000302) for their partial supports.
References
1 Washington Post, (n.d.), Is Hawaii ‘the extinction capital of the world’? Exhibit A:
The alala bird. Retrieved December 8, 2022, from
https://www.washingtonpost.com/national/health-science/is-hawaii-the-extinction-capital-of-the-world-exhibit-a-the-alala-bird/2016/04/25/3f45c6ac-f210-11e5-89c3-a647fcce95e0_story.html
2 Jason D. Baker, Charles L. Littnan, David W. Johnston, "Potential effects of sea level rise on
the terrestrial habitats of endangered and endemic megafauna in the Northwestern Hawaiian Islands," Endangered
Species Research 2, 2006, 21-30.
3 Chen Xinli, Han YH Chen, Chen Chen, Ma Zilong, Eric B. Searle, Yu Zaipeng, Huang Zhiqun,
"Effects of plant diversity on soil carbon in diverse ecosystems: A global meta‐analysis,"
Biological Reviews 95, no. 1, 2020, 167-183.
4 Judith Littleton, Gina McFarlane, Melinda S. Allen, "Human–animal entanglements and
environmental change: Multispecies approaches in Remote Oceania," In The Routledge Handbook of the
Bioarchaeology of Climate and Environmental Change, 493-510, Routledge, 2020.
5 K. Anggi Hapsari, Tim C. Jennerjahn, C. Lukas Martin, Volker Karius, and Hermann Behling,
"Intertwined effects of climate and land use change on environmental dynamics and carbon accumulation in a
mangrove‐fringed coastal lagoon in Java, Indonesia," Global change biology 26, no. 3, 2020,
1414-1431.
6 Christine Buci-Glucksmann, Emanuele Quinz, "For an Esthetics of the Ephemeral. Interview with
Christine Buci-Glucksmann," Hybrid, Revue des arts et médiations humaines 1, 2014.
7 Margaret C. Flinn, "Jean-Louis's Moments of Jean-Jacques." Studies in French Cinema 10, no. 2,
2010, 141-154.
8 Daisuke Uriu, Naohito Okude, "Thanato Fenestra: photographic family altar supporting a ritual to
pray for the deceased," In Proceedings of the 8th ACM conference on designing interactive systems,
2010, 422-425.
9 Virolainen Antti, Arto Puikkonen, Tuula Kärkkäinen, Jonna Häkkilä, "Cool
interaction with calm technologies: experimenting with ice as a multitouch surface," In ACM International
Conference on Interactive Tabletops and Surfaces, 2020, 15-18.
10 Elizabeth Diller, Diana Murphy, Ricardo Scofidio, Blur: the making of nothing, Harry N
Abrams Incorporated, 2002.
11 Suzaan Boettger, "34. Within and Beyond the Art World: Environmentalist Criticism of Visual
Art," Handbook of Ecocriticism and Cultural Ecology, 2016, 664-682.
12 Parés Narcís, Jaume Durany, Anna Carreras, "Massive flux design for an interactive
water installation: WATER GAMES," In Proceedings of the 2005 ACM SIGCHI International Conference on
Advances in computer entertainment technology, 2005, 266-269.
13 Allyson Purpura, "Framing the ephemeral," African Arts 42, no. 3, 2009, 11-16.
14 Marilyn Ekdahl Ravicz,"Ephemeralart: A Casefor the Functions of Aesthetic stimuli," 1980,
115-134.
15 Daniel Miller, "Artefacts and the meaning of things." In Companion encyclopedia of
anthropology, 430- 453, Routledge, 2002.
16 Axel Sylvester, Tanja Döring, Albrecht Schmidt, "Liquids, smoke, and soap bubbles:
reflections on materials for ephemeral user interfaces," In Proceedings of the fourth international
conference on Tangible, embedded, and embodied interaction, 2010, 269-270.
17 Chen Shi, Xin Cui, Xurui Zhang, Plamen Tchoukov, Liu Qingxia, Noemi Encinas, Maxime Paven, et
al. "Interaction between air bubbles and superhydrophobic surfaces in aqueous solutions," Langmuir
31, no. 26, 2015, 7317-7327.
Jellyeyes – Symbiosis, Evolution and Vision
JillScott
ZHdK, Zurich, Switzerland info@jillscott.org
Abstract
Jellyeyes is an augmented reality installation especially created for the viewer based on symbiosis, touch, and
biomimicry. Jellyeyes is inspired by narratives from the theories of evolution, and these stories involve the
characters of a hunter-diver-tourist and a marine biologist. These narratives are inspired by the personal
diving experiences of the writer and residencies in three different science research centers that focus on
evolution in relation to symbiosis, marine biology, and neuro-visual systems.
In 1967, Lynn Margulis created the concept of endosymbiosis. In this scientifically renowned theory, she
claimed that bacteria was the agent of change.¹ Specifically symbiotic bacteria invaded the primary cells
to create mitochondria and chloroplasts, the powerhouse parts of the cell as we know it today. Jellyeyes is
about the focus of this theory on the evolution of light cells or fluorescent chloroplasts. They are found in
fluorescent bacteria that live in the symbiont algae or Zooxanthellae inside any coral reef. These bacteria
called Cyanobacterial Endosymbiont evolved to create fluorescent light that acts as a sun block for the coral
and is sensitive to temperature change. These chloroplasts are also found in the pigmented cells of our eyes
called Rhodopsin, that move through the photoreceptors of our retinas in reaction to incoming light. (Figure
1)
This research in evolution and symbiosis led to discussions at conferences with marine biologists in
Australia.² In the Great Barrier Reef, symbiotic relationships between fluorescent algae and modern corals
are currently being affected by human impacts on the lives of species in this environment. In fact, the health
of these symbiotic relationships between the algae and corals, can shift the very roles they play in aquatic
ecosystems, for example, as helpers or guardians against pathogens or enemies. The death of symbiont algae can
be seen in the latest sad map about coral bleaching in Australia. (Figure 2) Unfortunately, a warmer climate not
only affects chloroplasts in algae but symbiont levels of co-habitation as well as some species levels of visual
acuity.
Neuroscientists study the effects of excess light on photoreceptors, and their pigmented chloroplasts.³
Human photoreceptors have evolved through adaptation and are very similar to the retinas of the box jellyfish
and the squid. Classified in evolution under the label of camera-based eyes, all three species share two
jelly-like substances; aqueous humor or the moving fluid in the eye that is affected by temperature and vitreous
humor: the jellylike shapes with fibers that are attached to their retinas. Hence the title: Jellyeyes!
Because liquids in these eyes are affected by UV exposure, an increase in bad conditions cause cellular damage
and very dry eye problems.³ Photoreceptors need healthy chloroplasts to function accurately. In humans they
control low and high light levels and the photoreceptors are formed from neural tissues and directly attached to
the brain (optic nerve). In squid: they control low light levels in photoreceptors that are also attached to
their brains, but in jellyfish, they use basic pigmented cells to assess light and these are attached to each
other by fibrous tissues. This last species has no brain but what is called a distributed brain. (Figure 4) But
how can such information on evolution in neurocience, marine biology and symbiont behaviour be interpreted in an
artwork? Could Augmented Reality and biomimicry help us “see” and understand the reef environment
through the light receptors of the eyes of other species?
Artwork
In summary, Jellyeyes is such an augmented reality investigation. It consists of a large photograph of a dead
barrier reef and an iPad built into a sculpture of the human optic nerve. (Figure 3) The image on the iPad
brings the photo alive! Sounds of water and bubbles immerse the viewer into the environment. The aim is to
explore the delicate balance of this aquatic ecosystem through the 24 eyes of the box jellyfish, our own human
eyes and eyes of squid or calamari. But the viewer can choose a temperature gauge to warm the water, and this
changes the environmental conditions! Ocean warming affects the predator-prey relationship of these species and
the symbionts algae leave the coral to die. The viewer can select the magnifying glass to discover how the loss
of this symbiont caused a loss of a food for the coral to eat, a weakened immune system, bleaching, disease, and
death. Warming also creates other food-algae and more jellyfish breed in the reef. In another scenario they can
choose to have an empathetic and parabolic look through the eyes of the Australian box jellyfish at the changing
predator-prey relationships of species in the reef. The viewer can choose the black and white view of the
squid’s eye to discover stories about the human hunter or they can use the human eye to see different
narratives.
The whole artwork is also available on an iPhone with a downloadable screen-based photograph. Here the aim is
to help tourists understand the ecology and the climate change problems on the reef.
But how exactly are these narratives chosen by the viewers?
Evolution and Roleplaying
In Jellyeyes, these narratives are divided into three sections or menus that the viewer can explore:
Co-Evolution, Structural Evolution and Comparative Evolution.
Co-Evolution:Here bacteria are seen as the agents of change.⁴ In this tribute to Lynn
Margulis, Jennifer Margulis (her daughter) is turned into the character of a marine biologist, who collects
evidence about the state of symbiont death in the coral and refers to her mother’s theories. She collects
fluorescent bacteria from the algae (zooxanthellae) and measures the corals level of nutrition and growth, how
they block excessive sunlight, control toxic compounds, stress levels and even ward off pathogens who attack the
algae. The viewer’s interaction causes acidity and symbiotic destruction in this environment. (Figure 5)
Structural Evolution:This menu takes Charles Darwin’s theory of adaptation⁵ and
applies it to the camera-based eye. How has this eye evolved in relation to its lineal ancestors. The result is
a simplified tree of the evolution of visual perception. (Figure 8) For example, the jellyfish is an early
example of our eye’s development and the squid another stage in the evolution of the light receptors of
the camera-based eye. In this narrative, an ignorant, destructive tourist steals too much food. Sometimes, even
digs into the coral to the horror of the scientist who tries to teach respect for this environment. (Figure 7)
Comparative Evolution:James Lovelock once said that life has created the conditions for its
own existence.⁶ In this menu, the viewer can choose either the eye of the box jellyfish, the squid, or
their own eye to see the relationship between the behavior of these three species in the barrier reef
environment. The ignorant diver tourist plays around with the poisonous Australian box jellyfish and every day
more of these very stingers swim in the reef. Ocean warming shifts the ecology of the reef’s predator-prey
relations and behaviors like these will have a backlash and harm our own existence. (Figure 7)
As in previous artworks,⁷ Jellyeyes uses the sensorial strategies of touch and biomimicry, to create
performative roleplaying to augment the above narratives and their repercussions. In Co-Evolution the role of
the participant becomes the CO2 emitter and through touch he or she or they, learn about the reactions of
symbiont bacteria, the survival of the coral, depreciation of the symbiont algae, squid, and reproduction of the
box jellyfish, as well as the interdependencies between these species. In Structural Evolution, the participant
becomes the investigator and learns through touch about the similarities of vision between species and about the
evolution and health of the light receptors in the photoreceptors. In Comparative Evolution, the viewer becomes
an empathizer, they see through the eyes of these species and can change these views with touch to identify what
they see. Here, the biomimicry of sight is used to witness the human environmental impacts on the reef
itself.
Conclusion
Perhaps, augmented reality art might be an effective catalyst for climate activism and education. Here the aims
are to reveal the influence of evolution on species survival in relation to the health of their habitats, to the
need for inherited variations and to the mutual benefits of keeping symbiosis alive. It is dangerous for us and
them not to mitigate climate change. Jellyeyes clearly shows the effects of our fossil fuel emissions on the
survival of the species in the reef. It offers a novel creative interpretation of structural evolution and
ecological interconnections. Here, the viewer becomes immersed in the symbiotic sensory relationship between
algae, vision and modern corals. Viewers say that Jellyeyes encouraged them to think about variations and
reproduction problems as well as symbiotic mutualism. They empathized with non-humans by “seeing”
through the eyes of other species. Afterall, the future of the barrier reef on this planet is in the eyes of
many species, but it is now in our hands!
References
1 L Fester, R. Margulis, Symbiosis as a Source of Evolutionary Innovation Speciation and
Morphogenesis, MIT Press, 1991.
2 ACRS Australian Coral Ref Society Conferences,
https://australiancoralreefsociety.org/conference/.
4 L Margulis, Symbiotic Planet: A New Look at Evolution Basic BVooks, 1999.
5 Charles Darwin, On Origin of Species, Pub John Murry, 1859.
6 James Lovelock, GAIA: A new look at Life on Earth, Oxford Uni Press, WW Norton, 1988.
7 Neuro_Eco _Media, www.jillscott.org
Partners
HFF, The Munich Film Academy. Munich, Germany Dr. Lisa-Ann Girshwin (CSIRO Australian Marine Stinger
Advisory Services, Australia)
Scientific Consultants
Prof. Dr. Stephan Neuhauss (Institute for Molecular and Cellular Research, Neurobiology, University of
Zurich, Switzerland) Prof. Christopher Robinson (Aquatic Research EAWAG, ETHZ, Switzerland)
Funding
Pro Helvetia, The Swiss Arts Council HFF, The Munich Film Academy. Munich, Germany
Production Credits
Concept & Directing: Jill Scott Production: Marille Hahne Camera: Julia Daschner Light: Georg Nikolaus
Animation: Natascha Jankovski Postproduction: Moritz Huber Programming: Nikolaus Völzow
Biography
Jill Scott is a media artist, a writer and art and science researcher. She is professor emerita at the Zurich
University of the Arts (ZHdK) in Zürich and founded their Artists-in-Labs Program in 2000. Her own
artwork spans 44 years of production about the human body and body politics. In the last 20 years she has
focused human health based on research into molecular biology, neuroscience, and ecology. She has had many
international exhibitions in both art and science venues. She also directs LASER Salon in Zurich for the
Leonardo Society USA and writes books on art and science (Springer and de Gruyter).
Technoshamanism: Symbiotic Techniques of Art and Healing
EdwardShanken
University of California, Santa Cruz United States eshanken@ucsc.edu
Abstract
Technoshamanism combines traditional shamanic technologies with emerging technologies based in silicon (dry),
biology (wet) and hybrid (moist), in the service of healing and sustaining life. This paper explores how
contemporary artists pursue expanded forms of consciousness by symbiotically joining technoscientific tools and
shamanic techniques.
More than ever before, contemporary artists, theorists, scientists, and activists need to pay more
attention to socalled indigenous knowledge. – Guillermo Gomez-Peña
Introduction
Mircea Eliade defined shamanism as “techniques of ecstasy.” He noted that, across cultures, it is
fundamentally a technology of healing that is "at once mysticism, magic, and religion."¹ Anthropologist
Michael Harner claimed that shamanic traditions around the world have developed a broad range of technologies -
from sonic drivers (drums) to plant medicine rituals—in order to achieve trance states that offer insights
beyond those available to typical waking consciousness.²
Shamans are both of this world and of the world(s) beyond. They communicate with spirits and ancestors in other
dimensions, learn from them, guide members of the community to them, and harness their power to heal and protect
individuals and the community. The shaman can embody the consciousness of other beings, including other animals.
A shaman also can release an errant spirit that has stricken a member of their community with illness, sending
the uninvited “hitchhiker” on its way. This can be dangerous, so shamans must be very strong,
capable of self-healing, and masters of their practice.
Shamanism may heal at scales ranging from the individual to the global. Indeed, following the Huni Kuin concept
of Xinã bena–“New Time”—some shamans/pajés of the Amazon have
decided that the preservation of the Earth demands sharing their cultural insights and healing technologies more
widely than in the past. This strategy has even led to the creation of Huni Kuin—Beya Sina Bena,
a videogame about the ancient stories of these indigenous people, scheduled for release in 2024. The developer,
Philosophical School of Games, claims that the game was produced in collaboration with the Huni Kuin communities
of the Jordão River, in Acre, Brazil.
Brazilian cultural critic Fabiane Borges notes that “apart from any possible encounter between technology
and shamanism, shamanism is itself a technology for the production of knowledge. ³ Similarly, I take
technoshamanism to join a combination of traditional shamanic technologies with emerging technologies based in
silicon (dry), biology (wet) and hybrid (moist), all in the service of healing and sustaining life.
The same can be said of art, historically interwound with techne, the Ancient Greek word for art and
the etymological root of technology. To risk stating the obvious, the use of shamanism for healing vastly
predates the application of science to allopathic medicine. The recent decriminalization of plant medicines and
the legalization of psychedelic-assisted therapies in some jurisdictions demonstrate increasing acceptance of
ancient indigenous technologies and marks a merger of diverse healing methodologies.
In a parallel manner, the engagement of contemporary art with shamanic traditions bridges knowledge domains and
augments art’s potentials to heal and to envision the future. These goals characterize a wide range of
artistic practices, from Ernesto Neto’s A Sacred Space to Roy Ascott’s “shamantic
web” of dual consciousness, to Anandha Ray’s Covenant VR virtual reality
ceremony/performance. A shaman was even part of Refik Anadol’s cohort of artists and software engineers at
the Google Artists and Machine Intelligence program in 2016.
Nonetheless, there is a dearth of literature that addresses the nexus of contemporary art and shamanism. Until
recently, the work of Joseph Beuys dominated these discourses, with a few notable exceptions. Jack
Burnham’s “Artist as Shaman” (1974) applied a structuralist method to interpret shamanic
aspects of Dennis Oppenheim’s work. In “Art in the Dark” (1983) Thomas McEvilley interpreted
performance by artists from Gunter Brus to Kim Jones as shamanic acts of expression and catharsis.
“Weaving the Shamantic Web...” (Ascott, 1998) claimed that “this ancient ritual mirrors our
contemporary artistic aspirations using digital technologies.” ⁴ Further essays by Ascott and his
circle, especially Lila Moore and myself, have followed in the 2010s and 2020s.⁵
In 2017, Christine Macel curated The Pavilion of Shamans, one of nine “trans-pavilions”, bringing
shamanism into the center of mainstream contemporary art (MCA) discourses at the 2017 Venice Biennale. Notable
works included Juan Downey’s Circle of Fires Vive (1979), Jeremy Shaw’s Liminals
(2017), and Neto’s A Sacred Place (2017), the centerpiece of the pavilion. Despite scant
media attention directed to this pavilion, art critic Tess Thackara (2017) highlighted it and discussed many
additional international artists, indicating broad interest in the topic, if not by MCA critics, then by makers.
⁶ Indeed, my 2020 Facebook post about technoshamanism yielded over one hundred comments, primarily by
artists about works of new media art (NMA) that fit the term. More recent MCA scholarship includes critiques of
Mircea Eliade’s and Michael Harner’s universalizing of shamanism and Claude Levi-Strauss’ and
Burnham’s psychopathology of the shaman, and is informed by Viveiros de Castro’s concept of
“equivocation,” offering a more nuanced and theoretical approach to shamanism and contemporary art.
⁷
Borges and others use the term “technoshamanism” to problematize the destructiveness of western
technoscience and its incursion on indigenous technologies of shamanism, which, by contrast, honor the sanctity
of the Earth. Technoshamanism names the messiness of cultural hybridity and the commodification of shamanic
traditions, including ayahuasca tourism and the gamification of Huni Kuin culture, resulting from colonization
and globalization. The visual arts of indigenous cultures, including Shipibo textile patterns, Papunya Tula dot
paintings, and Vodun rituals, are also being hybridized with, and appropriated by,
international contemporary art and visual culture. Such appropriations raise vital questions about
Technoshamanism as a form of aesthetic practice: How are contemporary artists engaging with the cosmologies,
technologies, and intellectual property of shamanic cultures in ways that honor and benefit the indigenous and
mestizo peoples that have cultivated and preserved shamanic traditions? How are indigenous artists engaging with
shamanic (and postindustrial technologies) in ways that strengthen their communities and contribute to the
discourses of contemporary art?
Mindful of these issues, my research is primarily concerned with the following speculative prospects:
How can artists embrace visionary consciousness?
How can art support entheogenesis (becoming divine together) by joining ancient shamanic techniques and
contemporary technoscientific tools?
How can art catalyze greater awareness of what Thich Nhat Hanh calls interbeing (the unity of all things) to
help heal the Earth?
Early Technoshamans
In 1997, British artist Roy Ascott participated in ayahuasca ceremonies in Brazil, which has had a profound
impact on his praxis. He theorized parallels between the dual consciousness that emerges in shamanic ceremonies
and the expanded field of consciousness afforded by emerging technologies. “In many respects,” he
wrote in 1998, “this ancient ritual mirrors our contemporary artistic aspirations using digital
technologies.” Claudia Jacques claims that this dual consciousness can only be manifested through
“indirection, shared participation, and metaphor,” core tenets of the telematic art that he
pioneered. This is the realm, she adds, that “visionary thinkers, creative artists, and shamans alike
aspire to experience and explore.” ⁸
Ascott (1998) describes the shaman as “the one who ‘cares’ for consciousness, for whom the
navigation of consciousness for purposes of spiritual and physical wholeness is the subject and object of
living.” He witnessed the pajé “passing through different layers of reality, through different
realities...” The shaman inhabits a state of dual consciousness, of “seeing at once both inward
realities and the outward surfaces of the world.”
Although it predates Ascott’s participation in shamanic plant-medicine ceremonies, his 1989 telematic
artwork, Aspects of Gaia: Digital Pathways Across the Whole Earth, embodies a technoshamanic, dual
consciousness. This installation integrates the global relationality of digital technologies with the shamanic
commitment to healing the Earth. Aspects of Gaia took its inspiration from atmospheric scientist James
Lovelock’s holistic Gaia Hypothesis (1979), which proposes that the Earth (Gaia) is a living
organism, a self-regulating, complex system that maintains the conditions for life on the planet. Artists,
scientists, shamans, musicians, visionaries, and indigenous artists were invited to participate by sharing their
feelings and beliefs about Gaia, which became work’s content. Ascott likened the participants to healers
who access the meridians of the earth’s nodes and creatively interact with the flow of data to perform a
type of “global acupuncture.” Their contributions suggested, for Ascott, a telematic
“noosphere,” an emergent field of consciousness generated from interconnected individual
consciousnesses, which might help harmonize and heal the planet.” ⁹
For American composer and electronic music pioneer Pauline Oliveros, cultivating expanded forms of
consciousness was the primary focus of her career. For this early technoshaman, the wisdom of the body is
crucial to accessing expanded states of consciousness. “I have progressed through many changes in music
technology from the end of the 1950s to the present,” she wrote. “Along the way I developed a bodily
relation to machines for making music. It has always been necessary.... because of the essential knowledge of
the body that is preconscious and nonverbal.” ¹⁰ Her experience of performing improvised music
parallels shamanic double-consciousness: Oliveros became a channel through with spirits from other dimensions
could communicate with and heal us: “This altered state of consciousness in performance is exhilarating
and inspiring,” she explained. “The music comes through as if I have nothing to do with it but allow
it to emerge through my instrument and voice.” ¹¹ Oliveros’ ideal attributes for a future
artificial intelligence “chip”—with which she could make music—are refreshingly
expansive. They include abstract psychic abilities, that seem to parallel shamanic intentions of achieving unity
and healing on a cosmic level. These include: the ability to understand the relational wisdom that comprehends
the nature of musical energy; the ability to perceive and comprehend the spiritual connection and
interdependence of all beings and all creation as the basis and privilege of music making; the ability to create
community and healing through music making; the ability to sound and perceive the far reaches of the universe
much as whales sound and perceive the vastness of the oceans. This could set the stage for inter-dimensional
galactic improvisations with yet unknown beings.¹²
Wake me up when ChatGPT can offer these features! For Oliveros, expanded consciousness on a galactic scale was
the foundation of healing, a form of service that she performed within the frame of a feminist ethics of care.
Chilean-American artist Juan Downey is another early technoshaman. His 1979 video, The Laughing
Alligator, resulted from the many months he and his family lived with the Yanomami people in southern
Venezuela, on the border of Brazil. He participated with his Yanomami hosts in rituals using yopo, a plant
medicine that contains a variety of DMT alkaloids. The artist came to regard shamanism as one of the most
powerful elements in the bond that linked the Yanomami to their surroundings, to the earth, and to each other as
part of a larger unity. As he wrote, A white and round place opens up in the front of my brain. Excretions
of light that vaguely align into circles, the intensity of a spiral or the infinite peace of a mauve color.
[...] I want to enter into the white space of my empty consciousness. ¹³
From spending a lifetime in trance-like states, Downey expanded his consciousness. He tuned into the
consciousness of others. And he helped others do the same. As a result, his work plays an important role in
stimulating new ways of thinking that are the prerequisite to healing society’s pathologies and to
recreating the world in a more sensitive, inclusive, and caring way.
Contemporary Technoshamans
American dancer and choreographer Anandha Ray’s virtual reality screen-dance, Covenant VR
(Figure 1), was a revelation. Joining shamanic healing rituals, modern choreography and dance
performance, and exquisite VR cinematography by Gary Yost, it created a unique container for embodied catharsis
mediated through virtual reality.
After putting on the headset, the next thing I knew, BAM! I was suddenly transported to a dance rehearsal
studio, at a virtual distance no more than a meter from a striking, dark-skinned dancer (Linda Steele II.) I
felt right there in the action, far closer than front row seats, inhabiting a perspective and proximity
typically seen only by other dancers. Because this is a 360-degree VR experience, one’s view is not
limited to a frontal perspective.
As viewers enmeshed in the VR environment, we witness the shamanic ceremony from a privileged vantage, so close
that we can virtually touch the protagonist, a proximity that heightens our sense of being there, of feeling the
healing and of experiencing it ourselves. Yost’s cinematographic wizardry is the ideal match for
Ray’s shamanic approach to dance, which “allow[s] movement to open portals of inquiry to better
understand the state of being human.” ¹⁴ This tour-de-force of Technoshamanism offers
an enthralling and cathartic experience.
Korea has a rich, living tradition of shamanism that continues to inspire artists, including technoshaman Kim
Jeong Han. His high-tech multimedia installation BirdMan (2005) is deeply informed by Korea’s
shamanic and Buddhist traditions. The artwork asks fundamental questions: How do human beings conceptualize the
world? How do birds conceptualize it? Can a hybrid world that joins human and nonhuman qualia (the internal,
subjective physiological component of sense perceptions) transform perception beyond the limits of human
physiology? If, as Donna Haraway claims, new perceptions create new metaphors, can the experience of another
species’ perceptual reality help create hybrid perspectives, marked by greater empathy and ecological
sensitivity?
The concept of a hybrid bird-man appeared to Kim in a dream. Due to a traumatic childhood experience, even as
an adult, Kim’s fear of birds prevented him from helping a one-winged bird that was suffering and dying.
He dreamed that he learned bird language from a monster with a bird head and only one wing. The dream and the
artwork can thus be interpreted as an effort by the artist to attain catharsis and heal a trauma through
aesthetic and shamanic means. As Kim and his co-authors noted:
“In Korean tradition, some shamans can share their own bodies with the deceased soul. Whenever a shaman
is possessed by the spirit of the dead, s/he acts, speaks and senses like another person, as if borrowing the
perception of the deceased. This moment looks like a coexistent state of the living body and the dead in which
perception and identity of the two is hybridised.” ¹⁵
The Buddhist idea that “the ‘Self” is not different than the “Other” is another
prevailing concept in BirdMan. The work offers the audience an opportunity to experience a form of
hybrid perception that joins human and avian “qualia.” (internal and subjective physiological
component of sense perceptions). Kim’s work does not explicitly represent accoutrements or scenes of
shamanic healing. As a result, the audience is not likely to visually identify shamanic elements in it. Rather,
BirdMan is driven implicitly by a shamanic perspective. Kim leads his own ritual of self-healing
through his artistic practice. By enabling us to metaphorically become one with bird, by offering us an
experience of hybrid avian-human perception, his work enables us to expand our consciousness beyond the limits
of our embodied human minds by joining self and other. It enables us to create new identities in between humans
and non-humans. And, as a result of this, it enables us to create new metaphors to live by and to live
with.
Such ideas echo Donna Haraway’s eco-feminist theories of multispecies worlds. She advocates a concept of
kinship or “making kin” that joins all beings: “all earthlings are kin in the deepest sense
[...]. All critters share a common ‘flesh,’ laterally, semiotically, and genealogically.” She
applies the term “sympoietic” to emphasize the collective process of poetic emergence in which all
beings are collaborators in the process of the Earth’s becoming. “Who and whatever we are, we need
to make-with—become- with, compose-with [...].” Taking care of the Earth, for Haraway, demands
caring for the diversity of beings, and “multispecies ecojustice” must be not only a goal but a
means to living well, together, as kin. By “staying with the trouble,” she proposes, “[m]aybe,
but only maybe, and only with intense commitment and collaborative work and play with other terrans [inhabitants
of Earth], flourishing for rich multispecies assemblages that include people will be possible.”
¹⁶
Buddhist luminary Thich Naht Hanh has taught that, “Our own awakened consciousness is what can
heal the Earth.” Indeed, the pressing and enduring concerns of global warming and the abuses of technology
demand that we expand our perceptual domain in order to heal ourselves, our kin, and our planet. To do so, we
must expand our metaphors and our means and channels of communication. We must summon the full power of art and
technology, while we equally harness ancient shamanic technologies and other modes of entering trance, expanding
consciousness, experiencing ecstatic states, fostering communication among all beings. If we want to have a
future, the artists of the future must serve as beacons of hope and as active participants in healing the world
and our relationship to it. Art, as a “psychic dress-rehearsal for the future,” to quote Jack
Burnham, must embrace kinship and harmonious co-emergence with all beings. We must become divine together. Today
there is no greater artistic calling, no greater aesthetic necessity than helping to heal and preserve
Earth/Gaia’s biodiversity for posterity. Technoshamanismis a potent strategy to move in that
direction.
References
1 Mircea Eliade, Shamanism: Archaic techniques of ecstasy. c. 1951, Princeton University
Press, 2004, xvii–xxvii.
2 Michael Harner, "My Path in Shamanism." International Journal of Transpersonal Studies
31.2, 2012.
3 Fabiane Borges, “Seminal Thoughts for A Possible Tecnoshamanism”
nd, np, c. 2014, Quote edited for clarity.
4 Roy Ascott, “Weaving the Shamantic Web: Art and Technoetics in the Bio-Telematic
Domain.” c. 1998, In Telematic Embrace, ed. Edward Shanken, Berkeley, University of California
Press, 2004, p. 356-62, Subsequent references to and quotations from Ascott are from this source.
5 Lila Moore, “The Shaman of Cybernetic Futures: Art, Ritual and Transcendence in Fields of
the Networked Mind” Cybernetics and Human Knowing 25, 2018, p. 2-3, 119-41.
6 Tess Thackara, Why Shamanic Practices Are Making a Comeback in Contemporary Art, Artsy,
2017, accessed June 13, 2023, https://bit.ly/48hsvk4
7 This literature includes Robert Wallis, “Art and Shamanism: From Cave Paintings to the
White Cube," 2019, Camila Maroja, “Persistence of Primitivism: Equivocation in Ernesto Neto’s
A Sacred Place and Critical Practice”, 2019, and Karen Gonzalez-Rice, “Revisiting "Art in
the Dark", 2023.
8 Claudia Jacques, “Forward: A Tribute to the Messenger Shaman: Roy Ascott,”
Cybernetics and Human Knowing 25, 2018, 5-15.
9 For more on Ascott, art, technology, and shamanism, see Lila Moore and Edward Shanken,
“Roy Ascott” in Charlie Gere and Francesca Franco, eds. Bloomsbury Encyclopaedia of New
Media Art, London, Bloomsbury Press, forthcoming.
10 Pauline Oliveros, “Improvising Composition: How to Listen in the Time Between,” c.
2012, in Gillian Siddall and Ellen Waterman, eds. Negotiated Moments: Improvisation, Sound, and
Subjectivity, Durham, Duke University Press, 2016, 83.
11 Pauline Oliveros, Deep Listening: a Composers Sound Practice, New York, iUniverse Inc.
2005, xix.
12 Pauline Oliveros, “Quantum Improvisation: The Cybernetic Presence,” in
"Sounding the Margins: Collected Writing 1992-2009, Kingston NY, Deep Listening Publications, 2009,
53.
13 Juan Downey, Dibujando con los Yanomami, (Caracas: Galería Adler Castillo,
1977): np. Quoted in Edward Shanken, “Pushing the Limits. Surrealism, Possession, and the Multiple Self:
Juan Downey and The Laughing Alligator” in Maricris Herrera, ed. Juan Downey,
1940-1993. Mexico City, Ediciones MP, 2019, 527-42.
15 Kim Jeong Han, Kim Hong-Gee, Lee Hyun Jean, "The BirdMan: hybrid perception," Digital
Creativity 26, 2015, 1, 56-64.
16 Donna J. Haraway, "Anthropocene, Capitalocene, Plantationocene, Chthulucene: Making Kin,"
Environmental Humanities 6, 2015, 1, 159-165.
17 Thich Nhat Hanh, Love Letter to the Earth, Berkeley, Parallax Press, 2013,
56.
Symbiotic Collaborators: The New Creative Subject in Postdigital Participatory Art
GyungJinShin
Hong Kong Baptist University, School of Creative Arts Hong Kong gjshin@hkbu.edu.hk
Abstract
As the Internet promotes participatory culture, contemporary network-driven participatory art, which I refer to
as “postdigital participatory art” (PPA), has introduced additional revolutionary creative subjects.
PPA induces a distinctive type of collective agency beyond mere collaboration among individuals by means of the
participatory architecture of the web. These multiple participants distribute the authority power of creation
throughout the network, transcending the limitations of time and space. In this paper, I attempt to theorize the
attributes of these new creative subjects, which I refer to as “participant-superjects,” with the
concept of superject serving here to indicate “power by modulation.” I outline the attributes of
these diffuse creative subjects and gauge their radical possibilities in terms of the agenda of experimental
art. I argue that, based on the new sense of relationality, materiality, and ontological perception associated
with the postdigital environment, these unique creative subjects are able to open up a new dimension of
creativity that differs from the modernist model, which emphasizes the creativity of the individual. I
hypothesize that the fluid power driven by this new creative subject exerts a latent force in building new
social relations outside the logic of the capitalist system.
With the death of the modern subject in the twentieth century, the divinization of single authorship on which
modern art relied has faced challenges and slowly collapsed along with the notion of “open
work,”¹ the postwar avant-garde of the 1960s, and author discourse.², ³ Artists
and theorists of the previous century attempted to undermine the traditional relationship between artist and
spectator, arguing instead for open-ended interpretations. These developments spurred a renegotiation of the
artist-spectator relationship and, in turn, the invention of new types of creative subjects. From this new
perspective, the status of the audience, as a “viewer-turned-participant,” has become almost equal
to that of the artist, and recent forms of contemporary art that are co-authored, collaborative, and
participatory have inherited the preference for antiauthorship. Such forms of art, including “relational
art”⁴ and “participatory art”⁵, ⁶, ⁷ have
flourished since the 1990s as artists have attempted to distribute the authority power traditionally associated
with the singular artist by encouraging the participation of viewers and envisioning new social relations.
In a hyperconnected environment that facilitates participation and collaborative creation, contemporary
participatory art has become ubiquitous and heterogeneous, transcending the boundaries of online and offline.
Recognizing that Bishop’s concept of “participatory art” overlooks the influence of
technology, I seek to help fill the resulting conceptual gap by coining a new term, “postdigital
participatory art” (PPA), to describe digitally mediated co-creation that relies on digital networks to
encourage audience participation. This new form of digitally mediated co-creation reflects changes in the
perceptions of time and space that have been described as “postdigital.”⁸, ⁹
In exploring PPA, I pay particular attention to the emergence of new authority power that is fluid and
ephemeral. Though such unique participatory creative subjects are profoundly observable in recent digitally
mediated participatory art, they have yet to receive careful analysis in terms of scope, motives,
characteristics, and patterns. Accordingly, I consider here the manifestation and radical possibilities of this
unique creative subject, which involves anonymous, networked participants mass-produced on networks.
Participant-Superject: The Unique Authors in the Network
As I conceptualize it, network-driven PPA relies heavily on collaboration among participants and often lacks a
teleology. Creators, materials, artworks, and spectators, in the absence of a predetermined blueprint for
object-making, co-emerge in the processing of inconclusive events. This development gives rise, in
turn, to the distinctive characteristics of a new creative subject that is ephemeral and fleeting, untethered as
it is to a fixed plan and, often, anonymous and unrecognized despite contributing significantly to the act of
creation.
The Jogging (https://thejogging.tumblr.com/), an ongoing, network-driven project, exemplifies the
attributes and modalities of these authors. [Figure 1] Utilizing a scroll-down thread on Tumblr that has
continued over several years, anonymous volunteers on the network Photoshop, reproduce, and reblog the images
associated with the original images uploaded by artists Brad Troemel and Lauren Christensen in 2009. Troemel
described his role as “initiating” and said in an interview that “‘Jogging’ refers
to a work flow, constantly moving, and not really focusing on any one thing, but rather to just continue
forward.”¹⁰ Without an individual artist furnishing a predetermined intention or conclusion,
such postdigital participatory projects are observable as the inconclusive, event-embracing agencies, networks,
and raw digital materials (e.g., information or data) involved in a work.
From Author-Subject to Participant-Superject
This new material variability in postdigital participatory art thus entails a reframing of the traditional
notion of author-subject. In a digital environment that strengthens temporality rather than spatiality, digital
participants have the potential to become temporally modulated subjects, or superjects, bearing latent
power by modulation rather than the power of individuals. As the antithesis of the author-subject
status, I coin the term participant-superject to describe the diffuse creative subjects that result
from the blurring of the artist-spectator boundary.
The concept of the superject, which refers here to “power by modulation,” has proved useful in
describing the special modalities of multiple agents that exist digitally when they are involved in PPA as
opposed to the modalities of physical participation. Deleuze adopted Whitehead’s concept of the superject
in the 1990s to illustrate the shift in subjectivity accompanying the rapid advance of digital technology at the
time.¹¹ According to Deleuze, a subject, when perceiving an object that is continually changing, can
be defined as a point of view that is likewise in a state of variation rather than fixed or determined
in advance.
The superject implies a “plural” subjectivity that cannot be fully described using the concepts of
a constant I and we and, instead, implicates a new status of the subject, a temporal I
and we. The superject, in Deleuze’s language, is an inherent multiplicity folded into a
collective unity.
The Three Phases of the Participant-Superject
Before gauging the radical possibilities of the participant-superject, it is useful to consider the
concept’s unique characteristics. Following Whitehead, Deleuze, and Savat, I describe the three main
phases of the superject as 1) temporal, 2) affective, and 3) condividual. The
temporal phase relates to the eventual nature of the superject. Referring to this phase,
Deleuze described the superject as an event rather than an essence. Adopting this view, Savat explicated the
specifics of the temporal aspect of the superject in the context of his analysis of digitality. According to
him, under the new temporal and spatial conditions of the digital environment, a superject exists as what
Deleuze called a “dividual,” an event that is ceaselessly modulated as a code, in contrast with an
embodied or spatial “individual.”¹² In this context, the manner of existence becomes
continuous, fluid, and momentary.
The second phase highlights the affective nature of the superject. Whitehead envisaged this facet of
the superject in an attempt to separate the subject per se from its experience of the
world.¹³ Thus, he proposed, the superject simultaneously emerges through the intake of
“data” in the form of sensory perceptions that precede consciousness, unlike the subject that
accumulates data. In this context, feelings mediate superjects and their associated data. This aspect of the
superject focuses on the moment of affective experience shared throughout the network.
Third, the superject implies the consolidation of collective agents, that is, condividuals. To be
specific, unlike the individual, which retains distinctive characteristics, the dividual tends to connect with
other similar components and combine with other dividuals to form condividuals).¹⁴ Existing
as metadata or data in the digital milieu, the dividuals can be used to build superjects in the network.
The participants in PPA, or participant-superjects, embody these phases of the superject—again,
temporal, affective, and condividual—which overlap and are interrelated. Rather
than being constrained to a physical venue for performance or exhibition, the participant-superjects appear
temporarily, take part in the creative process, and disappear in an instant. As the sequence of the action does
not “take place,” the actors rely heavily on temporality rather than spatiality. Further, the
participants in PPA tend to manifest momentary feelings or emotions rather than the conscious experiences
inscribed in a work. As The Jogging well shows, the intuitive and improvisational responses of the
participants manifest in such functions as liking, retweeting, and sharing on social media. Lastly, these
participants, as dividuals, merge easily and unite readily with others thanks to the anonymity and easy access
that the Internet affords.
Symbiotic Collaboration Among the Participant-Superjects
This new creative subject of PPA has the potential to fall into the trap of capitalism or to support resistance
to it. Stiegler used the term pharmakon to describe the aspect of technology that intoxicates and
cures, and postdigital participation has a similar “pharmakonic” effect, serving as simultaneously
toxin and remedy.¹⁵ In the 1990s and 2000s, the expectation was that the participatory environment of
the web would promote widespread self-awareness, the formation of grassroots communities, and co-individuation
by connecting individuals without respect to age, location, or gender. Since the network has also been viewed as
an arena for the fulfilment of capitalist desires, however, the participants in it found that they were treated
as consumers and unpaid labourers. Most of the major social networking platforms have reverse-engineered the
very structure of the participatory web to produce profits from the databases generated as a by-product of the
daily behaviours of online users. Moreover, governments and internet giants alike use the network as a source of
big data for analysing superjects and recognizing and forecasting social trends.
Regarding the paradoxical loop of postdigital participation problematized above, I argue that the modulatory
power of the participant-superject could radicalize the agenda of experimental works by altering the form that
power takes. This new form of power, again as Deleuze pointed out, refers to the modulation of
dividuals.¹⁶ For Deleuze, of course, such power deserves criticism, but it also serves as a positive
starting point for the arguments presented here. Considering the dividual a latent power of struggle,
Raunig claimed that the dividuality emerging as self-division in contemporary social media could promote new
forms of dispersed resistance against machinic capitalism.¹⁷ In the same vein, Savat insisted that a
new mode of politics involving the dividual, which he called the “politics of fluids,” has become a
more critical field of participation than the modern “solid politics” involving
individuals.¹⁸ Thus, he affirmed that the potential inherent in “fluid action” can be
utilized and realized in constructive ways because dividuality represents individuals’ ways of being in
the network.
Following Gaunig and Savat, I affirm that the fluid power of the superjectification seen in PPA—again, in
contrast with the solid power of individuals—exerts a latent force in terms of inventing social
relationships apart from the logic of the capitalism that finances and reifies human bodies and even social
relations. The characteristics of superjectification—being temporal, dispersed, affective, and
condividual—are more radical and effective in terms of “pre-empting” the looming problems
associated with this logic. Furthermore, the unique sense of affinity helps dispersed cognitions cling together
contagiously, in turn affecting real politics (e.g., Ushahidi software, the MeToo and Occupy movements, and the
activist group Anonymous).
Thus, PPA projects tend to encourage affinity-driven co-individuation outside the context of the labour system
and manipulative relationships on the web.¹⁹ Public_Public_Address: A Nationwide Virtual Protest
(2020-), for example, is an ongoing virtual protest for which Jason Lazarus, Stephanie Syjuco, and Siebren
Versteeg have been accepting submissions of selfie videos of individuals holding protest pickets in support of
the Black Lives Matter movement. [Figure 2] The participants include those who had been marginalized in society
and those unable to protest in person because of a disability. The artists integrated the video submissions into
live-streaming on YouTube and simultaneously in the window of a New York gallery in 2020. Seemingly marching
toward the viewers, the images of these anonymous participants have been incorporated into a bizarre new form of
space-time. Rather than isolating, separating, and pixelizing the agency of the participants in the network,
this project reveals the aesthetics of fluid power that the superjective participants generate in the
postdigital environment by transcending the online-offline binary.
Conclusions
As the exploitation of participation in the network becomes increasingly sophisticated, PPA has the potential
to offer space in which to imagine new social relationships by questioning freely the dominant logic and weaving
together digital objects, technological materials, and human agency symbiotically. In this regard, art systems
centred on individual authorship fall short when it comes to describing a new dynamism that embraces the energy,
matter, force, objectiles, and superjects that digitally mediated co-creation can bring about. In the consistent
flow of the network, the conventional binaries of matter and form, subject and object, physical and digital, and
artist and spectator co-emerge, co-concretize, and intra-act. Then, the new creative subject of PPA, or
participant-superject, having emerged in the context of the shift in digital materiality and subjectivity, can
radicalize the author-subject model and re-envision the agendas of open-ended and participatory aspects of art.
Superjective authors, then, with their mutual resonances, cause events and collective experiences to
proliferate and add further from the postdigital environment. These authors are omnipresent and already part of
humans’ ontological status. As Literat observed regarding many cases of online crowdsourced art, however,
“the crowd is still a crowd, not yet a community” because the digital fibres and technological
structures that mediate digital encounters can hinder efforts to achieve artistic collaboration or build a sense
of community.²⁰ Thus, further studies are needed to assess the potential of PPA to invent meaningful
communities apart from the profit-making algorithms of the various online platforms.
References
1 Umberto Eco, "The poetics of the open work," The Open Work 251, no. 1., 1962.
2 Roland Barthes, "The Death of the Author," Contributions in Philosophy 83, [1967],
2001, 3-8.
3 Michel Foucault, "What is an Author?," in Textual Strategies, ed. J.V. Harari,
Ithaca, Cornell University Press, [1969] 1979, 141-60.
4 Nicolas Bourriaud, Relational aesthetics. Trans. Simon Pleasance, Fronza Woods,
Mathieu Copeland, Dijon, Les Presses Du Réel, 2002, Originally published as Esthétique
Relationnelle, Dijon, Les Presses Du Réel, 1998.
5 Claire Bishop, “Introduction: Viewers as Producers,” in Participation,
London, Whitechapel/Cambridge, MIT Press, 2006, 10-17.
6 Claire Bishop, “Social Turn: Collaboration and Its Discontents,”
in Artforum 44, no. 6, 2006, 178–83.
7 Claire Bishop, Artificial Hells: Participatory Art and the Politics of Spectatorship,
London, New York, Verso Books, 2012.
8 Robert Pepperell, Michael Punt, The Postdigital Membrane: Imagination, Technology and
Desire, Bristol, Intellect Books, 2000.
9 Sy Taffel, “Perspectives on the Postdigital,” Convergence: the
International Journal of Research Into New Media Technologies 22 (3), 2016, 324–38.
10 Brad Troemel, “Interview with Rob Walker, The Jogging, a Tumblr at the intersection of
BuzzFeed, 4chan and weirdo experimental art,” Yahoo News, October 24, 2013.
11 Gilles Deleuze, The Fold: Leibniz and the Baroque, London, The Athlone Press,
1993.
12 David Savat, “(Dis)Connected: Deleuze’s Superject and the Internet,”
in International Handbook of Internet Research, 11:423– 36, Dordrecht, Springer
Netherlands, 2010, 432.
13 Alfred N. Whitehead, Process and Reality: An Essay in Cosmology. eds Griffin,
David Ray and Sherburne, Donald W, New York, Free Press, [1929] 1978.
14 Marco Deseriis, Improper Names: Collective Pseudonyms from the Luddites to Anonymous,
Minneapolis, University of Minnesota Press, 2015.
15 Bernard Stiegler, What makes life worth living: On pharmacology, Cambridge,
Polity, 2010.
16 Gilles Deleuze, “Postscript on the Societies of Control,” October 59,
Winter, 1992, 3–7.
17 Gerald Raunig, Dividuum. Machinic Capitalism and Molecular Revolution, Volume 1
(trans. Aileen Derieg), Los Angeles, Semiotext(e)/MIT Press, 2016.
18 David Savat, Uncoding the Digital, Basingstoke, Palgrave, 2013.
19 Marco Deseriis, “The Politics of Condividuality”, transversal.at., 2018,
7-10, accessed August 31, 2021, https://transversal.at/transversal/0318/deseriis/en
20 Ioana Literat, “The Work of Art in the Age of Mediated Participation: Crowdsourced Art
and Collective Creativity”, International Journal of Communication 6, 2012, 2962–84.
Bibliography
Claire Bishop. “Participation and Spectacle: Where Are We Now?,” in Living as form: Socially
engaged art from 1991-2011, ed. Nato Thompson (Massachusetts: MIT Press, 2012), 34-45. Kyle Chayka,
“Art in the Corporatized Sphere: The Impact of Commercial Social Media on Online Artistic
Practice,” in A Companion to Digital Art, ed. Christiane Paul (New York: John Wiley & Sons,
2016), 413–425. Christian Fuchs, Digital Labour and Karl Marx (London: Routledge,
2014) Ioana Literat, “Facilitating Creative Participation and Collaboration in Online Spaces: the
Impact of Social and Technological Factors in Enabling Sustainable Engagement,” Digital Creativity
28 (2), (2017): 73–88.
Acknowledgements
The overall structure and conception of this paper draw on my PhD dissertation published in June 2022
Proteus 3.0 - Interacting with the cloud
MariaSmigielska*,
CompMonks*
mariasni.com; compmonks.com Zurich Switzerland maria@mariasni.com,
compmonks@compmonks.com *authors contributed equally to the project
Abstract
This paper presents the Proteus 3.0 project as an interactive and generative video installation, which
tries to remain in conversation with its audience and learns from its engagement to produce new and potentially
meaningful sequences of images. This particular version of a series is focused on digitized material as
ferrofluid and its digital surrogate generated with a reaction-diffusion algorithm, projected onto a room-scale
oval oculus allowing for a collective/immersive interaction. By employing state-of-the-art custom reinforcement
learning models coupled with human intelligence in a symbiotic fashion, it aims to make the interaction more
meaningful by understanding better human behaviour during an interaction. Additionally, it intends to maintain
the interaction as conversations, rather than playful attractions.
This paper presents the Proteus 3.0 project as an interactive and generative video installation, which
tries to remain in conversation with its audience and learns from its engagement to produce new and potentially
meaningful sequences of images. Proteus, as a series (Figure 1), has systematically evolved with its
ferrofluid displays and modes of magnetic activation at multiple resolution ranges, particular sequences of
digital and analogue layers as well as types of interaction. This version is particularly focused on moving the
ferromaterial to the digital realm with its digital surrogate generated from a reaction-diffusion algorithm,
scaling up from an object-size analogue display to an architectural oculus of oval shape at room scale, and
which, at the same time allows to shift from an intimate interaction to a collective/immersive one. By employing
state-of-the-art reinforcement learning models coupled with human intelligence in a symbiotic fashion, it also
intends to build an interaction that is based on conversations, rather than playful attractions. Most frequent
and straightforward approaches towards interaction design are based on implicit principles and clear
instructions built in a closed loop to promote immediate engagement. These oversimplified views on interaction
neglect its long-term impact¹ or the ones stemming from more traditional discourses, and which define any
interaction with an artwork purely as a mental act.² In that lens, the project revisits the
Conversation Theory of Gordon Pask³ in its scope of human-machine interaction as a form of open
and emulative conversation. A dynamic process in which both ends learn and adapt from each other. As such, the
project aims to construct an interaction that goes beyond the boundaries of predefined scenarios that follow the
strict categories of an audience’s behavioral patterns and tends towards individualized and dynamic
responses.
Background Conversational interaction
What most generally describes successfully an interaction is the degrees of responsiveness and control an
artwork and an audience are communicating to each other. An interactive artwork is essentially a composite made
of an artificial component (the artwork itself as an object) and a human component (the audience) that both
define its nature and behavior⁴. These degrees of communication are then located somewhat between total
randomness and total predictability, without reaching any of these extremums. A total random response could only
qualify as part of a reactive process, and a totally predictable one as part of a controlled manipulation. This
does not necessarily involve a stable exchange and most frequently concurrent dynamics are at play from both
ends as human intentions and artificially designed systems may have different and evolving goals over time.
While this constitutes a great challenge for an artist to frame and produce such ideas, it also represents the
richness and openness that an interaction may provide in qualifying an object. Interactivity is, by design, the
relational property of an object and not an intrinsic one. It results that the value of such artwork is
consequently constantly in a state of volatility and becoming, evolving throughout interactions. One might argue
that this is necessarily the fate of every artwork, and more generally every object. But this becomes
fundamental when interaction becomes itself an object of design in the artwork. Although, the most frequent and
straightforward approaches towards interaction design are based on implicit principles and clear instructions
built in a closed loop to promote immediate engagement and playful attractions. Similarly, most theoretical
intent to define interactivity in art has been partially treating aspects of interactions that revolve around
the topic of controllability. In general, the ability to improve control makes novelty the source of most types
of interactivity⁵. In the general perspective of control, interaction can be seen as a learning stage in
order to reach control. Once sufficient learning is achieved, interaction becomes redundant. Interaction becomes
endowed with the feature of being instructional. However, even during the early stages of cybernetics, the
theory of control was seen as only a chapter in the theory of messages⁶. Interactive art must be
dissociated from the grasp of control theory [ref] in order to avoid any pessimistic perspectives⁷ or
shortcomings in the envisioned artistic power of such artistic interventions. On the same level, Input/Output
theory of interaction with computers⁸ simply defines interactions as a unilateral transactional process of
information and therefore fall short on creative content. On another aspect, when interactivity is defined as
procedural and participatory⁹, the definition lacks enough abstraction and generalisation to talk about
the systemic design of an interaction and the informational novelty that go beyond the sole responsiveness of
participations to an artwork. On another end, the evolution of machine perception in human computer interaction
can be seen as trivialising the term of interactivity in new media a rts, and eventually rendering the sum of
its tentative description as too loose to be useful to qualify an artwork¹⁰. This oversimplified
views on the interaction neglects its long-term impact¹¹ or the one stemming from traditional art,
which defines interaction with art purely as a “mental act”¹². It also resonates with the
cognitive approach of the power of an artwork being located in the beholder’s response
¹³.
Proteus 3.0 within the series of Protei
Proteus is a series developed as a duo since 2018 driven by the curiosity in exploring aspects of
interacting with matter, tensions between analogue and digital, as well as relations between human and machine
intelligences¹⁴. The series evolves through the continuous visual and material reference of a colloid
compound called ‘ferrofluid’ as a main character ¹⁵. When exposed to magnetic
fields and interacting with another liquid carrier, it reveals its intricate and lively nature by presenting
strongly contrasted and ever-changing complex patterns ranging from discrete patches of dots, meandering
streams, to larger coagulated blobs. The volatility and organicity of this material behaviour in the series is
used as an allegory of the Greek mythology of a deity called Prōteus and his ability to assume
many forms and provoke uncanny encounters ¹⁶ that remain evasive to the rational mind but open-ended
to the visual imagination. Over the course of its serial declination, Proteus has iteratively evolved
with varied organic and deep black patterns to serve as an emulator for visual human intelligence. That of
seeking meaning in a constantly changing flow of images and unpredictable symbolic relations which can only be
found in the human mind, just like gazing at a cloud. But the historical problem in such a romantic
contemplation is often emphasized through the notion of distance and the lack of direct communication between a
cloud and its beholders. If the cloud could talk back in response to one’s efforts to find visual meaning,
would the conversation stabilize to a consensus, or would it constantly move to explore and seek novelty? This
problem of distance and communication reveals itself when reframed in the artwork as a generative/interactive
installation and its architectural embodiment.
Methods
Proteus 3.0 is an interactive generative 4K video installation, rear-projected onto a black screen of oval
shape. The screen, of dimensions approximating 3.5 x 2.5 meters, is rotated 45 degrees and integrates an HD wide
angle and infrared camera to estimate body and head poses from the audience and to detect regions of visual
interest located on the screen. This information is continuously tracked and serves to build markers of
attention to modify the generated visual frames (Figure 3).
Material surrogate
Because to this day, no digital model has been able to accurately simulate the physical behaviour of ferrofluid
material in real time ¹⁷, a specific GLSL shader has been developed for the procedural generation of
a digital surrogate. Based on a Gray-Scott reaction-diffusion model between two substances, a series of image
filters have been designed as an overlay to mimic the dynamics of the ferrofluid and its pattern formations.
Our experiment was conducted in two following steps: (1) initiate predefined interaction scenarios run in the
exhibition context to collect the visitors’ data, (2) building adaptive and evolving interaction models
with reinforcement learning. The details of those two phases are described below.
Predefined interaction scenarios
At the first launch, the initial interaction consisted of 5 predefined interaction scenarios running in the 80
seconds loop. Interaction scenarios consisted of generative visuals coupled with the most common human
behavioural patterns. The visitors' data responding to these scenarios were collected during eight months of the
BioMedia exhibition at ZKM Karlsruhe, starting in December 2021.
In order to build Regions of Interests (ROI) of each interaction, skeleton data of max 6 visitors
simultaneously were captured with a Kinect 2 camera to extract their torso and face orientation projected back
on the oculus canvas. Each interaction has been recorded into a large dataset organized by timestamp with raw
numerical information (visitors' ROI, skeleton data, location in relation to the screen) as well as
low-resolution images (like RD noise map and visitors' ROI) for further training. The construction of datasets
considers that any relevant personal information is decoupled from the recorded data.
Generative interaction scenarios with reinforcement learning (RL)
The scenarios further evolved into generative characters with the support of a machine learning model
programmed in Python, specifically a reinforcement learning model, to learn from collected data and
extracted markers of attention. It is being trained discretely offline and concurrently for a real-time
generation to support the progressive evolution of the artwork that can adapt to different exhibition contexts
and visitors' behaviours. The custom model has been developed in python with a PyTorch library.
The real-time performance of the work and the offline training are orchestrated by two combined processes. For
real-time performance, a TouchDesigner file manages both visual inputs coming from the camera and generated
outputs going to the display. The generated images are sent to two video projectors, mapped and blended for a
smooth projection onto the screen. Daily, the recorded data is updated to a remote server and feeds new offline
training sessions for the reinforcement learning model. The updated model is then sent back to the local
computer controlling the installation to update Proteus' behaviour.
Engagement
Engagement can be measured with simple metrics like the amount of time spent looking at art¹⁸ or
more complex summative indexes, like Sweep Rate (SRI) and Diligent Visitors (DV) representing exhibitions
“Thorough Use”¹⁹ to improve audience engagement in museums at large. However, those
static metrics allow us to look at visitors’ response only through a pinhole and we do not know
the exact impact of the interaction on the visitor (long-term, imaginary, etc). Therefore, we need more data
collected during the interaction itself and more sophisticated tools to analyze them. “The Plant”
project²⁰ proves that reinforcement learning can improve some of measures of
engagement, like visitors touch count during interaction, but not average duration of the interaction. This
project allows us to draw a hypothesis that AI-based tools combined with generative models can increase the
quality of engagement and not just its quantitative factors like duration.
In the next developmental step of Proteus project, we aim to utilize the developed RL model in the future
exhibition and conduct comparative studies to evaluate this hypothesis.
Conclusions
Proteus 3.0 is an interactive and generative video installation, which, once in place within the walls of its
exhibition and switched on, tries to remain in conversation with its audience and learns from the visitors'
engagement to produce new and potentially meaningful sequences of images (Figure 2). Until the end of its
exhibition, when it is switched off and set frozen in time. By then, its last learned state is saved and becomes
the new generative beginning for its next exhibition. The learning and interactive process continues endlessly
during the artwork’s lifetime for each exhibition and conditions its existence. Its generative imagery
remains open-ended and its content to be modulated by its beholders. This project is an attempt to utilize
AI-supported generative systems, which facilitate an interaction that goes beyond predefined behavioral models
directly controlled by a few, easily comprehensible parameters and therefore might increase the engagement in
the audience.
References
1 Edmonds E, Candy L, “Interacting: Art, Research and the Creative Practitioner,”
2011.
2 Paul C, “Digital Art” Third edition, London, Thames & Hudson, 2015.
3 Pask G, “Conversation Theory Applications in Education and Epistemology” Amsterdam,
Elsevier, 1976.
4 Margaret A. Boden, Ernest A. Edmonds, From Fingers to Digits: An Artificial
Aesthetic, Illustrated edition, Cambridge, Massachusetts: The MIT Press, 2019.
5 “Aaron Smuts, What Is Interactivity?—PhilPapers”, 2023, Accessed September 15.
6 Norbert Wiener, The Human Use Of Human Beings: Cybernetics And Society. Revised
ed. edition. New York, N.Y: Da Capo Press, 1950.
7 Terrence Rafferty, “Everybody Gets a Cut,” The New YorkTimes, May 4, sec.
Magazine, 2003, https://www.nytimes.com/2003/05/04/magazine/everybody-gets-a-cut.html.
8 David Z. Saltz, “The Art of Interaction: Interactivity, Performativity, and
Computers,” The Journal of Aesthetics and Art Criticism 55 (2), 1997, 117–27,
doi:10.2307/431258.
9 Janet H. Murray, Hamlet on the Holodeck, Updated Edition: The Future of Narrative in
Cyberspace, Updated ed. edition, Cambridge, Massachusetts: The MIT Press, 2017.
10 Manovich Lev, The Language of New Media, MIT Press, 2002.
11 Ernest Edmonds, “Art, Interaction and Engagement,” In 2011 15th International
Conference on Information Visualisation, London, United Kingdom, IEEE, 2011, 451–56,
doi:10.1109/IV.2011.73.
12 Christiane Paul, Digital Art. Third edition. London, Thames & Hudson, 2015.
13 Eric
Kandel,The
Age of Insight: The Quest to Understand the Unconscious in Art, Mind, and Brain, from Vienna 1900 to the
Present,
1st edition, New York, Random House, 2012.
14 https://proteusproject.ch/
15 https://en.m.wikipedia.org/wiki/Ferrofluid
16 https://en.m.wikipedia.org/wiki/Proteus
17 Han Shao, Hang Libo, Dominik L. Michels, “A Current Loop Model for the Fast Simulation of
Ferrofluids,” IEEE Transactions on Visualization and Computer Graphics, 2022, 1-12,
doi:10.1109/TVCG.2022.3211414.
18 Jeffrey K. Smith, Lisa F. Smith, “Spending Time on Art.” Empirical Studies
of the Arts 19 (2), 2001, 229–36.
19 Beverly Serrell, “The Aggregation of Tracking-and-Timing Visitor-Use Data of Museum
Exhibitions for Benchmarks of ‘Thorough Use.’” Visitor Studies 23 (1), 2020,
1–17, doi:10.1080/10645578.2020.1750830.
20 Zoe Tong, Dana Kulic, “Learning to Engage in Interactive Digital Art,” In 26th
International Conference on Intelligent User Interfaces, College Station TX USA, ACM, 2021,
275–79, doi:10.1145/3397481.3450691
Acknowledgments
Artwork by: Maria Smigielska, Compmonks Created as part of the intelligent.museum project at the ZKM |
Center for Art and Media Karlsruhe and at the Deutsches Museum. Funded by the Digital Culture Programme of the
Kulturstiftung des Bundes (German Federal Cultural Foundation). Funded by the Beauftragte der Bundesregierung
für Kultur und Medien (Federal Government Commissioner für Culture and Media). Funded by BioMedia
ZKM | Center for Art and Media Karlsruhe.
Authors’ Biographies
Maria Smigielska is an architect and researcher based in Zurich, currently working at DBT, ETH. She is
interested in enhancement of potentials for creation of architectural elements, design objects and mixed media
installations, by using digital and interactive technologies for encoding and modulating materials properties,
custom design and fabrication methods. Mariasni.com
Compmonks (PhD) is an architect and a researcher in technology for architecture based in Switzerland. He
currently leads research on interactive and generative design with brain-computer interfaces at ETH
Zürich. His artistic work follows a series of design objects and mixed-media installations focused on
the power of combining humans with computers.
1 University of Antwerp, 2 University of Copenhagen, 3
Technical University of Delft, 4 University of the Philippines, 5 EDESTA-University Paris
8, 6 University of Plymouth, 7 SEADS (Space Ecologies Art and Design) Antwerpen, Belgium
pieter.steyaert@uantwerpen.be
Abstract
Co-creative practices including non-human actors give rise to a series of challenges and critical issues.
Ēngines of Ēternity is an ongoing artwork which seeks to attribute agency to microscopic animals
called rotifers and to the unique environment of outer space. Scientific experiments involving these animals, as
well as the evolving artwork that accompanies them, have gone through multiple iterations having been flown to
space and consequently returned. This article reflects on how transdisciplinary approaches can provide a vehicle
to connect knowledge and enquiries from art and sciences in the context of real-world problems. We then focus on
the case study of Ēngines of Ēternity and how these interactions played out during this process. We
conclude the article with a critical reflection on non-human agency, using the aforementioned project as a case
study.
The bidirectional exchange of knowledge, ideas and methodologies can foster co-creative practices and allow for
a genuine blurring between disciplinary boundaries to occur.¹ Such holistic conditions may act as a
catalyst for the synthesis of a deeper understanding with regard to the nature of the universe we inhabit and
with it the emergence of a new post-Anthropocene era.² In order to achieve a sustainable future, we need to
understand the mechanics and challenges of human and non-human interactions. This article explores and assesses
the process of ArtScience co-creation, through a transdisciplinary project entitled Ēngines of
Ēternity, acting as a case study. The work's narrative incorporates questions regarding manmade power
structures and cultural immortality, through the use of non-human organisms, whilst highlighting the benefits
and issues that can occur as a consequence. The author of this work and of this article is SEADS, a
transdisciplinary and cross-cultural collective of artists, scientists, engineers and activists. The
collective’s mission statement notes that transdisciplinary approaches are key to unlocking collective
intelligence, which is a prerequisite for generating more diversified and inclusive futures.³ They
incorporate co-creation methodologies in a range of their projects, amongst others are community art projects
such as Bio-modd and Seeker.⁴, ⁵ The Ēngines of Ēternity project explores new
forms of co-creation between different entities: humans, biological organisms and algorithms within the context
of outer space. Before introducing the project, we outline the concept of creativity in the post-Anthropocene,
which has been key to functioning as an underlying theoretical framework for this case study.
Creativity and Agency in the post-Anthropocene
If the Anthropocene is understood to be the geological epoch in which human activities are the dominant
influence on the environment, the post-Anthropocene can be understood to be the epoch that dawns when human
activities and their effects are no longer the dominant influence on the environment.⁶, ²
The exact nature that the post-Anthropocene will take is difficult to predict because it depends on many
factors, including how humanity responds to the challenges currently faced in the Anthropocene itself. On a
global scale, we are not able to predict when and how the post-Anthropocene concept will become a dominant
global framework, or indeed if this in fact will take place. However, there may be triggers that are capable of
driving societies (both human and non-human) towards it, such as the ongoing global warming crisis and
deterioration of the environment, societal cultural shifts, or technological advancements that may even allow
humanity to live in greater harmony with Earth's ecosystems, but this is currently speculative. Merging
disciplines can contribute to identifying and solving complex problems through the collaborative exchange of
ideas and methodologies.⁷ Indeed, it has become apparent that many of the major problems facing the world
today are hugely complex in nature and will not be solved by any one single discipline alone.⁸ Addressing
complex societal and scientific problems requires involving multiple disciplines. However, there is a danger of
oversimplifying the collaborative benefits between these disciplines. We argue that commonalities or filling up
gaps can ignore or overlook the need to find new creative ways of dialogue between the disciplines, thus
creating new knowledge or lines of inquiry that can transform the conversation on various global and scientific
issues.
Differing from alternative approaches to merging disciplines, transdisciplinarity concerns both what exists,
and is between and beyond all disciplines, blending into an empirical and theoretical exploration of
co-creation.¹, ⁹, ¹⁰ One form of transdisciplinarity involves
combining art and science (ArtScience) to create new co-creation practices that utilize methods and
methodologies from both fields. Such art interventions can take place in present and future shared spaces and
may contribute to the constitution of new symbolic orders and organisations of human and non-human connections,
staging the mise en forme of post-Anthropocene co-existence. Seen in this way, ArtScience co-creation can be
seen as an integral part of rethinking humanity’s relations to nature.
Bruno Latour argues that non-humans, such as animals, plants and objects, are active participants in social
life and should receive equal attention in the study of society.¹¹ In this way of thinking non-humans
should be seen as actors in their own right, rather than simply as objects that humans act upon; non-humans can
have agency, and influence the course of events in the world. Latour’s conceptualization of agency does
not strictly correspond with notions of sentience nor free will; neither does it have to be confined within
physical boundaries. Agency can be an emergent property of multi-actant configurations.¹¹ According to
Latour, social-cultural practices are enunciative and performative actions through which the new
“subjects” can test different semiotic, sociological and alterity modes of existence, that lie
between “being-as-being and being-as-others.”¹² In a similar vein, we refer to the notion
of a “multispecies roundtable” for co-creating systems. This involves bringing together plants,
computers, and people in mutually beneficial ways and placing them in a situation where they must arrive at a
consensus.⁴, ¹³
Speculative Realism, Posthumanism, New Materialism, and the non-human are just a few examples of concepts that
challenge the limits of what Donna Haraway calls the “fantasy of human exceptionalism.” These ideas
reject the notion that humans are somehow separate, beyond, or more advanced than our earthly
co-habitants.¹⁴ The theorists working within these fields help to illuminate the limits of human
experience and thought, highlighting the capacities, experiences, and potentials of other living, non-living,
and non-human entities. Their examples embrace a radically holistic and inclusive approach to the future of
humanity and are established concepts in critical cultural thinking.
The Engines of Eternity project puts these ideas into practice by directly interacting with microscopic
organisms, computational algorithms, and the physicality of outer space. These practical engagements hint at new
opportunities and insights but also confront the limitations of an extended co-creation practice. Continuous
hands-on approaches and their analysis allow us to move beyond established thinking and gain new critical
insights through hands-on artistic research.
Ēngines of Ēternity
Ēngines of Ēternity is a transdisciplinary project that explores how creativity and agency might
function in a post-Anthropocene context. It is a work in progress and is intended to be a series of mixed-media
installations. The project is a joint effort between SEADS and scientists from Karine Van Doninck's laboratory
at UNamur. The foundational idea of the project is to take the biological phenomena of cloning and DNA repair as
metaphorical departure points for an art installation about humanity’s obsession with cultural
immortality.¹⁵ By metaphorical departure point, we mean that we use the fascinating biological
characteristics of the rotifer animal to question creating a perfect society. At the core of the project is an
artwork that evolves through complex systems of interplay and co-creation between biological organisms, humans,
and technology. The project explores dominating ideologies about evolution and cultural identity within the
space environment. The discussion on cultural immortality is discussed in a separate article³, as this one
focuses on the methods and their implications that were crucial to developing the artwork. Here, we instead
describe the background of our ArtScience co-creation activities: space biology experiments with the Bdelloid
rotifer.
The Bdelloid rotifer or ‘wheel animal’ typically lives in freshwater environments and is part of
the greater ecosystem that exists there. On the surface, rotifers seem to represent a biological culture
perfected through evolution, cloning themselves in endless repetition. However, during this cloning process,
genetic material actually gets reshuffled and diversity is generated through the process of horizontal gene
transfer. This method of generating diversity can be defined as the non-sexual movement of genetic information
between organisms, and genetic material from organisms as diverse as fungi, bacteria and plants have been
discovered lurking inside the rotifer’s genome.¹⁵
Through a series of space biology experiments, scientists from UNamur sent rotifers to the International Space
Station (ISS) in both 2019 and 2020. The goal of these experiments is to investigate rotifer DNA repair
mechanisms and cloning in the conditions of outer space. In December 2019, the rotifers going to ISS were
accompanied by an artistic "seed" in the form of a specially designed visual code that accompanied the rotifers.
The code was developed by the Ēngines of Ēternity team and serves as a seed for a series of
3D-printed sculptures. These sculptures show the physical changes to the rotifers in response to environmental
conditions and experimental techniques. The sculptures are based on a mapping of recorded experimental data 3D
sculptures.
The work is driven by two main objectives: the evolution of the "code" throughout different missions and the
translation of these seeds into three-dimensional sculptures. Analysis was conducted on both the bags that went
up into space and the bags that served as a control group on Earth in order to reveal the differences between
protein expressions. Each individual bag that was sequenced was assessed against the average from the control
group bags, resulting in discrete sets per bag of large quantities of raw data. These sets were then grouped and
ordered per function, accumulating the differences compared to the control group. The result was a visualization
per bag showing the impact of the space environment on rotifer animals. These visualizations were then used to
evolve the original visual "code" that accompanied these bags by using it as an attractor to manipulate the
pixels of the original seed.
The second objective was to translate these seeds, both the original and the evolved version, into a
three-dimensional sculpture. The glyphs that make up the seeds were transformed into a circle and a linear
regression algorithm was applied to layer multiple versions of the original visual on top of each other and
connect them. This resulted in a three-dimensional sculpture where the base visualises the overall shape and
every layer shifts towards a higher level of detail. In this way, the final sculpture shows both the impact of
the environment on the rotifer animals at a higher order and on the smaller, individual protein clusters.
The “code” and sculptures will gradually evolve into the final artwork over several missions based
on the changes in the rotifers. Hence, the context of outer space is not merely treated as a passive gallery
space to exhibit art in, but rather as a dynamic environment that can actively shape the evolution of the work.
Co-creation revisited Co-creation in Art and Science
The historical development of scientific and artistic disciplines has resulted in the development of
organisations and structures in which these two areas do not organically engage or have dialogue. The
organisational and procedural infrastructure that exists within these two fields is therefore sometimes seen as
opposites. Science is often associated with the use of strict methodologies and systems to produce empirical
data, unbiased information, and facts that help us better understand the physical and natural world around us.
On the other hand, art is often viewed as a more subjective practice that allows individuals or groups to
express their creativity and imagination.¹⁶ While these definitions are not entirely inaccurate, such
reductionist viewpoints are unhelpful when beginning to ask deeper questions. ²³ Indeed, in recent
times a variety of Art-Science networks have emerged that act as places where such communities can interact, and
begin to create imaginaries for developing questions regarding our place in the wider world, whilst reaching
large audiences. Interactions between the fields of science and art can be regarded as a one, or two-way, form
of communication between the two. When considered as a one-way form of interaction, there are two possibilities
for one to influence the other, either art provides science with some form of benefit, or vice versa.
The SEADS team developed tools using techniques and knowledge that differ from those of the pure scientific
team. The goal was to gain insights into the impact of the alien environment of space on living creatures in an
exploratory context. Initially met with skepticism, the collaboration with the pure science team evolved
throughout the project. By communicating about our results, the team was able to establish new ways of trust and
collaboration. The SEADS collective comprises members with backgrounds in both art and science tapping into
collective intelligence. This allows them to push boundaries when collaborating within and outside of the
network. For instance, in the Ēngines of Ēternity project, the artists received hard drives
containing all the raw transcriptomic data to explore. This not only resulted in the aforementioned sculptures
but also allowed the team to analyze their findings and compare them to those of the pure science team. It
appears that new insights emerged from this transdisciplinary approach, which has yet to be validated and might
lead to a future publication. In some ways, it is also possible to view the process of horizontal gene transfer
as a biological counterpart to transdisciplinarity itself. The generation of novel genetic material through the
incorporation of foreign DNA into the rotifers, from such phylogenetically distant organisms, echoes the process
of numerous distinct disciplines joining together to create a more efficient system. Much like
transdisciplinarity itself, it is not merely the merging of disciplines (or in this case genetic material) that
makes this comparison applicable, but the fact that the process has wider implications on a much deeper level as
well.
If transdisciplinary practices are considered to be a combination of, translational and methodological
approaches from various disciplines, utilised to address a problem, then the horizontal integration of genomic
elements from cross-species sources can be said to function in a similar manner. DNA segments once integrated
into the rotifer’s genome can be translated into new useful proteins, that are capable of carrying out a
variety of functions, and in turn, contribute to the evolutionary adaptation of the organism. The whole is
greater than the sum of its parts, in both the case of transdisciplinary inquiry and biological evolution.
Posthuman co-creation
A fundamental question in this project is whether true co-creation between humans and non-humans has been
established, more specifically between the art-science team and the rotifers. How much agency did the rotifers
actually have in participating and contributing to the artwork? In addition to exploring human-human
interactions, we also investigate the agency of non-human entities, specifically the Rotifer animal. The
evolution of the artwork is driven by changes in protein expression. However, this approach raises the question:
are these non-human entities truly being given agency?
There are two ways to approach this question. One way is instrumentalist, which would argue that regardless of
the artists' intentions, the rotifer animals are being exploited for both science and art. In fact, there is
even an element of cruelty, as the animals are taken from their natural habitat and subjected to experimentation
in an extreme environment without any choice in the matter. From a pragmatic standpoint, it is difficult to
refute this way of framing the issue.
We present a second approach as follows: The artist team explores the intrinsic attributes of the rotifer
animal and co-creates the artwork based on their findings. In this instance, horizontal gene transfer is viewed
as a form of creation. By incorporating the protein expressions from these animals as the main driving force,
and using complex bio-informatics algorithms, some level of agency over the artwork is given to these non-human
entities. Extreme environments have been fundamental driving forces for the evolution of rotifers. However, in
the case of the artwork discussed here, the driving force of its evolution is not solely an attempt to depict
the intrinsic attributes of these animals. Instead, it also considers the impact of the alien environment of
near-zero gravity in space on these extremophiles. This marks the first time in the history of the species that
they have been exposed to the environment of space.
In this sense, the two levels of co-creation within this case study can be extended to include "The alien" as
an environmental contributing factor. Additionally, the extensive use of algorithms and predictive computational
methods has played a significant role in the visual outcome of the project. As a result, this project explores
new forms and levels of co-creation between humans, biological organisms, algorithms, and outer space.
Horizontal gene transfer and parthenogenesis are used not only as a metaphor, as was discussed in the previous
section, but as an act of self-creation and self-transformation which instruments served as contributing factors
in the formation of the artwork.
Conclusion
Co-creative practices including those involving non-human actors were explored through the creation of an
ongoing artwork project entitled Ēngines of Ēternity. The nature of such transdisciplinary
methodologies was investigated by a small community of artists and scientists involved in the project, whose
evolving discourse during the process of developing and reflecting on the project has helped to provide a
framework to inform conversations regarding the future of humanity, and whether and how we might move towards a
more post-Anthropocene stage in the future. Establishing an ethical post-Anthropocene requires approaching the
Other from a humble and empathic perspective. This case study provides insight into a practical approach towards
this objective. For example, by focusing on the intrinsic qualities of creation present in the non-human agents.
Nonetheless, it is evident that there are challenges that need to be addressed to successfully accomplish this
goal.
References
1 Erich Jantsch, "Inter-and transdisciplinary university: A systems approach to education and
innovation", Higher education 1, no. 1, 1972, 7-37.
2 Erle C. Ellis, Anthropocene: a very short introduction, Vol. 558, Oxford University Press,
2018.
3 Angelo CJ Vermeulen, Diego Maranan, Pieter Steyaert, Nassim Versbraegen, Ann Peeters, Jeroen Verschuren,
Arise Wan et al. "Ēngines of Ēternity: An Artistic Inquiry into Space Settlement Ideology Using
Rotifer Experiments on Board the ISS," In 73rd International Astronautical Congress, 2022.
4 Pamela G. Cajilig, Diego S. Maranan, Arlene Sy, Oliver Salva, Angelo Vermeulen, "Multispecies Roundtable
for Climate Impact: A Speculative Proposal," 2017.
5 Diego S. Maranan, Angelo CJ Vermeulen, "When ideas migrate: A postcolonial perspective on Biomodd [lba2],"
2015.
6 A. C. J. Vermeulen, Caroline Nevejan, Frances Brazier, "Seeker: Co-Creating Diversified Futures,"
Studio Time: Future Thinking in Art and Design, Black Dog Press, London, 2018, 172-182.
7 Van der Bijl-Brouwer Mieke, Giedre Kligyte, Tyler Key, "A co-evolutionary, transdisciplinary approach to
innovation in complex contexts: Improving university well-being, a case study," She Ji: The Journal of
Design, Economics, and Innovation 7, no. 4, 2021, 565-588.
8 Solomon Bililign, "The need for interdisciplinary research and education for sustainable human development
to deal with global challenges," International Journal of African Development 1, no. 1, 2013, p.8.
9 Farzam Ranjbaran, Cristina Marras, "European Peer Review Guide: Integrating Policies and Practices into
Coherent Procedures," ESF Member Organisation Forum on Evaluation of Publicly Funded Research, 2011.
10 Bruno Latour, Reassembling the social: An introduction to actor-network-theory, Oup Oxford, 2007.
11 Bruno Latour, "On technical mediation," Common Knowledge 3, no. 2, 1994, 29-64.
12 Bruno Latour, Enquête sur les modes d'existence: une anthropologie des modernes, Paris, La
découverte, 2012.
13 A. C. J. Vermeulen, "Living computers, Mars simulations and DIY Starships: Advancing cross-disciplinary
and cross-cultural collaboration," Link 3 Workshop Space, Plymouth University, 2016.
14 Donna J. Haraway, When species meet, Vol. 3. U of Minnesota Press, 2013.
15 Matthieu Terwagne, Emilien Nicolas, Boris Hespeels, Ludovic Herter, Julie Virgo, Catherine Demazy, Anne-
Catherine Heuskin, Bernard Hallet, Karine Van Doninck," DNA repair during nonreductional meiosis in the
asexual rotifer Adineta vaga", Science advances 8, no. 48, 2022, eadc8829.
16 Norman G. Lederman, "Nature of science: Past, present, and future," In Handbook of research on science
education, Routledge, 2013, 831-879.
17 Jonathan Osborne, Collins Sue, Mary Ratcliffe, Robin Millar, Rick Duschl, "What
“ideas-about-science” should be taught in school science? A Delphi study of the expert community,"
Journal of research in science teaching 40, no. 7, 2003, 692-720.
Acknowledgements
This project is funded by the European Union’s Horizon 2020 research and innovation programme under the
Marie Sklodowska-Curie grant No 860470.
Author Biography
SEADS (Space Ecologies Art and Design) is a transdisciplinary and cross-cultural collective of artists,
scientists, engineers and activists. Its members come from all corners of the world. SEADS is actively engaged
in deconstructing dominant paradigms about the future and develops alternative models through a combination of
critical inquiry and hands-on experimentation.
SEADS employs its own signature methodology which is centered around community building, co-creation and
bottom-up design. SEADS believes that these approaches are key to unlocking collective intelligence, a
prerequisite for generating more diversified and inclusive futures. Furthermore, SEADS also embraces a hacking
and open-source ethos, with the goal of engaging as many people as possible in the activities and ideas that
they initiate. Since 2009, the collective has co-created more than 40 art projects, together with local
communities all over the world.
Spectral Plain: A case study for exploring the world-building potential of co-creative systems that combine
text generation models with game mechanics.
1École des Arts Décoratifs – PSL (EnsadLab), University of
Applied Sciences and Arts Northwestern Switzerland (IXDM) 2KU Leuven, LUCA School of Arts,
3Hong Kong University of Science and Technology 1guillemette.legrand@ensad.fr,
guillemette.legrand@fhnw.ch, 2vincentsimon.thornhill@kuleuven.be,
3ijclarke590@connect.hkust-gz.edu.cn *These Authors contributed equally.
Abstract
This paper describes the game installation ‘Spectral Plain’; introducing a case study of an
interactive artwork that intersects algorithmic, sensing, and gaming technologies to explore new forms of
generative world-building, which are different from the world-imaginations created by commercial and political
narratives. Through this case study, we aim to explore how information technologies might embed specific
socio-cultural beliefs. Here, we investigate the imaginaries created and disseminated through dominant
representations of information technologies, while simultaneously searching for ways to create new multiscalar
imaginaries of the earth through the mobilisation of these technologies. This case study describes how physical
and digital interactions can be combined with AI and randomising technologies to foster forms of co-creation
with and between humans. It also explores the capacity for AI-integrated game environments to decenter
anthropogenic perspectives by creating new symbiotic interrelations with the non-human and the non-living that
can impact participants’ perception of their understanding of the world.
Keywords
World-building, cosmologies, game mechanics, interactive installation, autoregressive language model,
co-creative system.
This project started with an intention to examine, and communicate the capacity of information technologies to
construct imaginaries of the "world." The initial technological context for this investigation was the
commercial and political representation of 5G wireless technology; the latest in a lineage of "network"
imaginations of a world coinciding with the development of communication technologies. Professor of Anthropology
Shannon Mattern argues for a need to acknowledge how marketing fantasies and policy scenarios have limited our
imaginations, not just of what the technology is, but also of how we organise and live in the world through
these imaginations.¹ Philosopher of Science John Tresch scales Mattern’s concerns to the imagination
of the planet, stating that “when modern technology is linked to a ‘cosmology’ or a
‘world(s) view,’ we are offered a much thinner picture than that suggested by anthropology’s
presentations of the ways humans have organized nature and society—a ‘mechanization of the world
picture’ simplifying all experience to utilitarian building blocks of masses and forces, functionalist
means and ends.”² Between the marketing and political narratives of 5G, and the technical operations
of this infrastructure, we saw the potential to build a new reading of wireless technology.
This rereading was pursued through decentering wireless technology from the regulatory tradition of allocating
and auctioning ‘bandwidth’ of electromagnetic frequency to commercial and geopolitical actors.
Instead, we wanted to reposition this technology by placing it within a wider spectrum of frequencies that human
and non-human entities are intertwined with. Imagining electromagnetic energy in new scales and alignments
prompts us to think at a cosmological dimension, a concept derived from Timothy Morton’s description of a
"Spectral Plain;" an infinite surface where “you can’t really distinguish very easily between alive
and not alive, between sentient and non-sentient, between conscious and non-conscious.”³ Through
enriching and diversifying the comprehension and representation of electromagnetic frequencies, the project aims
to open up new imaginaries of the earth.
Research Process
The research process centred on testing ways to produce narratives about electromagnetic energy that cross
between human and nonhuman scales, from the atomic, to the terrestrial, to the cosmic. To do this, we created a
series of working sessions through which the project was developed through participation with varied audiences
and formats, while iterating the development of technological systems for generating narratives. These
experimentations took the shape of online and physical workshops and interactive performances that were set up
to test different forms of collective world-building; methods that were later implemented in the final artwork.
We used the game environment as a site for experimentation, which enabled us to test ways that various modes of
representation and points of view could coexist or create friction. It also allowed us to play with multi-scalar
experiences; from micro to macro, from the local (storing of individual data about the player locally) with the
planetary (data processed through large distributed algorithmic models). Through recombining different
information technologies, we created an experimental ground for a collective to engage with world-building. We
use the exhibition periods as a moment to present the work to a public, but also a moment to extend our research
through further testing of the system we created, while collecting additional feedback on the players'
experiences.
Of the different technologies involved, our research into the use of GPT-3 had the most impact on how we were
able to reconsider the ways collective world-building processes could be constructed in co-creation with AI
technology. Several games have explored the use of GPT-3 to generate open-ended narratives, such as AI
Dungeon.⁴ However, rather than using this technology to create divergent storylines, we sought to use the
intrinsic properties of this linguistic model to establish connections and patterns between different bodies of
knowledge about electromagnetic energy.
Project Description
Our artistic research into co-creative systems through the medium of the game environments led us to develop
the artwork as an interactive game-installation, called Spectral Plain (figure 2).
In this interactive installation, three players can navigate three levels of an environment (the skies, the
terrestrial and the underground). The game story takes place in a period following a fictive geomagnetic
reversal (a change in the planetary magnetic field), where ‘spectral energy’ is unleashed, and
players are engaged in discovering new perspectives on electromagnetic energies.
The installation uses a real-time data capture and a text-generation algorithm (GPT-3) via an integrated
blueprint in the game engine (Unreal Engine) to log in various datum (time stamps, players' statement
selection), which is then used to inform the final audio-visual cosmogram.
The three levels of the game are developed in a systematic way with each environment being constructed around
five “spectra” of electromagnetic frequencies (figure 3) that are named: aqua, light, geology,
matter, and device. These frequencies manifest as different entities within the game environment, for example,
the light spectra manifests as chlorophyll in the terrestrial level.
The presence of the three players in the physical environment of the installation creates a unique disturbance
in the gamma radiation that is present around the installation. In the game infrastructure, this disturbance
triggers the Geiger counter to produce a "true random number." This number is then used within the game engine
blueprint to randomise the graphics that the players will interact with and the parameters within GPT-3.
The game environment is developed to give agency over the creation of a new world-imaginary of spectral energy
that is different from the dominant 5G narratives. Here, we align with philosopher Federico Campagna’s
position towards the concept of "world;" not using the word to describe an ontological reality, but rather
seeing the world as an artificial construct of the imagination, “a metaphysical process that creates
discontinuities, separations between things, individual identities, ‘somethings’ out of a plane of
pure existence where no clear divisions are already inscribed.”⁵
At the end of the players’ journey, a cosmogram is automatically generated (figure 4). This cosmological
diagram is constructed by gathering data about the players’ movement and interactions within the levels of
the game. From this cosmogram, unique cosmological readings of the unified journey are then created through
fine-tuning the GPT-3 language model. These readings are then "spoken" through the use of speech synthesis
software. In this process of co-creation between player and algorithms, we are interested in how automated AI
technology can be used to generate connections between diverse perspectives and narratives of energetic entities
(scientific, mythologic, commercial narratives). The reading is not an attempt to accurately describe
relationships between these perspectives, but rather, to create an artistic and lyrical intervention.
Technical Realisation
In the Spectral Plain interactive game-installation, we combine physical and virtual player interaction,
together with a combination of predictive (GPT-3) and randomising information technologies (Geiger counter
radiation sensor as true random number generator (TRNG). The input and output of these different interactions
and technologies are linked and moderated through Unreal Engine’s Blueprint Visual Scripting
system.⁶ The node-based interface of this system allowed us to map and program interactions in the
back-end of game creation software in a visual manner.
At the intersection of cosmology, and generative technologies, we also became interested in logic and structure
of different divination techniques from different cultures. In Spectral Plain, we develop an analogy to
techniques such as birth chart readings, and Yi Ching, by approaching their logic as a form of programming. In
our technical realization of the project, processes of symbolisation, inclusion of chance, and geometric
calculation can be seen as referential to ancient divination techniques that link the individual to the
planetary.
The section attempts to walk through the different steps taken in the process of playing Spectral Plain by
annotating the diagram (figure 5 below):
The presence of the three players in the physical environment of the installation creates a unique
disturbance in the gamma radiation that naturally exists in the immediate vicinity. This disturbance triggers
the geiger tube in a gamma radiation sensor in a (literally) unique way, producing a random number
analogically, rather than digitally. As it is produced analogically, it is considered a "true random number."
This true random number is then used within the Unreal Engine blueprint to randomise statements generated by
GPT-3. These statements are fragments of stories about electromagnetic ‘entities’ that are located
throughout the different layers of the game environment. The players log their choice of statement into the
game engine blueprint by pressing one of the graphic
symbols ‘✕’ ‘◯’ ‘▢’ ‘△’ on their
controller.
Information about what buttons are pressed, and information about the time spent at each entity is logged
into the blueprint. The combination of this information establishes an order of importance in the entities
the player’s encounters during their journeys in the game.
These two sets of information are processed by a program that draws a vector space between the three most
important entities per player. These vector space triangles are placed on top of a pre-rendered cosmogram
drawing (figure 4) which spatialise the different entities in the game.
By overlapping the triangles of each player, we determine a middle point that will be used as a seed for
GPT-3. This middle point gives the meta direction to the generated text, while the most important entities
for each player are described within the body of the generated texts.
The structure of each cosmogram reading is generated based on a prompt that requests a title, a description
and an advice.
The text is then processed by a voice synthesiser to produce an audio file that is played when the players
reach the visual cosmogram that is automatically inserted into the concluding environment of the game.
Conclusion
Spectral Plain is an interactive game-installation that builds upon concepts present within ancestral
technologies (birth chart construction, Yi Ching reading instruments, etc.) through the use of current
technologies in order to enable a space for co-creation, where symbiotic interrelations between disciplines
(art, geoscience); the non-human (e.g., glowing micro-algae, chlorophyll photosynthesis); and the non-living
(e.g., AI, geological elements) can occur. In this technical and conceptual description of the work and its
intended effects, we aim to contribute to the discourse surrounding the ontologies of information technologies.
This paper also aims to create a case study to allow for continued questioning of how artistic research can
challenge, mobilise, and enrich the imaginary potential of these technologies.
By advancing the possibilities to engage with and through narrative-based technologies within
game-environments, we follow Alenda Y. Chang’s proposition to explore "the untapped potential of games to
create meaningful interaction within artificially intelligent environments, to model ecological dynamics based
on interdependence and limitation, and to allow players to explore manifold ecological futures—not all of
them dystopian.”⁷
The project requires further technical and conceptual development in order to explore ways to balance and
represent generative-based components, chance factors, and collective agency within the game-narrative. It is
also important to question the ability of large language models like GPT-3 to generate a truly novel story, and
find ways to negotiate with its capacity to homogenise culture and beliefs through language parsing and
recomposing.
References
1 Shannon Mattern, “Networked Dream Worlds,” Real Life , accessed December 7, 2022,
https://reallifemag.com/networked-dream-worlds/
2 John Tresch, “Technological World-Pictures: Cosmic Things and Cosmograms,” Isis Vol.
98, No. 01, accessed December 7, 2022, http://dx.doi. org/10.1086/512833 (online)
3 Timothy Morton, Being Ecological, Cambridge Massachusetts, MIT Press, 2018, 126.
4 Latitude “AI Dungeon (2019),” accessed December 7, 2022, https://play.aidungeon.io
6 Unreal Engine, “Anatomy of a blueprint (2022),” accessed December 7, 2022,
https://docs.unrealengine.com/4.27/en-US/ProgrammingAndScripting/Blueprints/Anatomy/
7 Alenda Y. Chang, Playing Nature: Ecology in Video Games, University of Minnesota Press, 2019,
16.
Ecotechnologies of Practice: in-forming changing climates
GisèleTrudel
École des arts visuels et médiatiques, Université du Québec à
Montréal. MÉDIANE Canada Research Chair in Art, Ecotechnologies of Practice and Climate Change.
Montréal, Canada. trudel.gisele@uqam.ca
Abstract
How do ecotechnologies of practice actualize? This paper traces the material/theoretical operations of an
ongoing research-creation concerned with changing climates. It mixes information in experimental approaches from
collectivities of trees, media arts and forest sciences. Through individuations of symbiotic modulations, the
paper is a thinking-with Balsam Fir, Diana Beresford-Kroeger, Camera, Domingo Cisneros, Dendrometer, Erin
Manning, Isabelle Stengers, Jack Pine, Gilbert Simondon, Light Emitting Diodes, Numbers, Microphone, Recorder,
Sapflow, Sensings, Speakers, Sugar Maple, Temperature, Yellow Birch.
Keywords
Research-creation, changing climates, trees, ecotechnologies of practice, information, individuation,
art/sci/phi.
An interdisciplinary sharing is at work. Ælab joins forest engineer Christoforos Pappas who is explaining
to them how to attach a sapflow sensor to a tree in the boreal forest (Figure 1). The tree already senses with
the surroundings, the sapflow sensor brings the tree's operations into human understandings, as sociologist
Jennifer Gabrys has discussed.¹ Inter-cross-transdisciplinary approaches activate philosopher Isabelle
Stengers' concept of “ecologies of practice” in modern science.² To move outward of
disciplinary silos, she states openness is required while still maintaining tension, to question how practices
produce knowledges (plural added; savoirs in French). Stengers argues for knowledges production in
interdependence, within the practices of each scientific field, and in-between them. Each field can contribute a
differential worldview to one another, a distinguishing plurality.
Scientific fields work actively to try to solve climate changes. Yet, what else can occur in-between the
sciences, trees, art and machines? There is an important increase of sensual and affective artistic explorations
with trees, among others by Jane Tingley, Domingo Cisneros, Susan Turcot, Rasa Smite and Raitis Smits, Agnes
Meyer-Brandis. In this research-creation project, scientific and tree knowledges are expressed in outdoor media
art installations. How can knowledges and practices resonate together, within and through a multitude of fields,
given the concerns for the “more-than of life-living” of the blue planet?³
Experimental in-forming
The concepts of information and individuation offered by Gilbert Simondon provide insight for the encounter of
ecologies of practices.⁴ In his philosophy, information is more-than what is generally considered as
factual data. Information is operation. It activates change through perturbation, by provoking or inducing an
encounter between disparate elements/factors, as event. Potential is propelled into actualization through a
dynamic play of forces. This is of particular importance in the context of climate change where fearful and
helpless affects predominantly circulate. How else does change occur?
The first outdoor media installation by MÉDIANE to explore the potential of changing climates is bois
eau métal (2021), an artwork concerned with the ecophysiology of three trees, with numerical data
from Smartforests Canada research site, including immediate weather sensing. Presented in the Arboretum, the
scaffolding structures are a site of temporary co-constructions (chantier), mini pavilions without
walls for experiencing three types of immersive sounds setups, video imagery and graphic visualizations. Tree
data from a forest is co-composed with colors from a custom-made weather kit. The whole manifests as rhythms of
physical processes in feedback loops. This installation acts as information, pushing the exploratory
differential encounters between fields of knowledges and experiences in a shared reality with trees. Stengers
analyzed science domains which often perpetuate their own methods of safeguarding knowledges production.
Contemporary art often relies on the unwavering codes of modernist heritage to produce and present artwork as
autopoiesis, production of self “in and of itself”. To promote changes, in Stengers' words,
“this is the reason why, through the exploration of knowledges, there is also a type of ethical
experimentation” (our translation).⁵ Respecting and challenging, one and other. Shifts occur, along
with the tree's own experience; despite the tree being rooted in one location, seemingly immobile,
the sensor data shows the tree is constantly moving and changing at microscales.
Changing climates warrant to be addressed as a “problem” in and of change, within
research fields and in their differential gathering, and in dialogue with diverse publics. How can
ecotechnologies of practice address the problem of changing climates, understood here as an obstacle for thought
in and into action, rooted generally in fear, anxiety and helplessness? This is an obstacle needing new thought,
a coproduction of knowledges in practice, in experience. Stengers and Simondon provide compelling offerings to
think-with and act-with problems, in experimentation. In Simondon's concept of individuation, no individual
(human, instrument or otherwise) preexists to the encounter. Individuation occurs through the meeting of forces,
garnered little-by-little and in proximity, gatherings that build and change in time. Propelled into change by
the perturbation of information, triggered by the tree's mode of living, change happens. For better or for
worse, without moral imperative, it is a propositional ongoing, a doing, building with each tiny
difference.⁶ This approach is core to MÉDIANE's ongoing research-creation process, whereby each
installation proposes change, building with the tree, data gathering, analysis, and by the gradual spreading of
knowledges production and circulation in-between trees, forest science, out-of-doors art and publics.
An orientation, not a beginning
The slogans of the FridaysForFuture global strike movement (FFF) repeat that governments, corporations and
publics need “to listen to the science, to listen to the scientists.”⁷ Scientists are directly
convened to address climate change, outside of the silos of fundamental knowledges production. Undeniably,
scientists are needed and respected to do so. However, it points to the domination of science as holding
exclusivity to resolve climate emergencies. Peoples (plural) have perhaps given over climate change issues to
science and to government to solve, forfeiting distributed agencies. How can a combination of practices
contribute to new actions? With Stengers, it becomes possible to extend the ecologies of practices in scientific
realms to a mixing of approaches by ways of practice, in-between collectivities. A new knowledges
circulation, a thinking-with trees, machines, materialities, artists, scientists and publics, even if only
temporarily. This problem for thought in/to action, perturbed by climate changes, is addressed by
ecotechnologies of practice. In Figures 5 and 6, the 2022 media installation entitled Orée des bois
addressed these questions through the study of the birch tree's phenology, the seasonal changes from Spring
to Fall which affect growth and connections with flora and fauna.
A problem of & with & in practice
For Stengers, ecology is always double: scientific and political (italics added).⁸ This doubling
can be activated in contemporary art, design and well as social sciences, charting the trajectories of actants.
In Greek, technè is art, skill, and craft or the method employed resulting in an
object.⁹ Logos tries to lay claim to (human) language, to “collect, gather, in
derivatives of word, speech, statement, discourse, computation, account.”¹⁰ To consider
technology only in a prepotent human way provides humans with vector to dominate and control, to justify the
means to an end, often neglecting entangled elements/lives/factors. Technology understood in this dominant way
severely limits approaches, because if it is seated only in the human as an exclusive semiotic rationality, it
prevents or even opposes new informations to coalesce or collide with more-than and other-than. It is pertinent
to note that in sustainable architecture, a definition for ecotechnologies is holistic systems or machines with
the environment, including reuse, passive housing, renewable energies, crade-to-cradle designs.¹¹
Simondon proposes another technology, neither thing nor method fashioned by human will. In his philosophy,
individuation is actualized through “phases” of relations between physical environments, instruments
and humans. Technology is in-forming, changing, extending another expression of logos, the cosmos
expressing its manifold in movement with animals, minerals, plants¹². A
“life-living.”¹³ It is also the combined strengths of trees, scientists, artists and
publics in changing climates.
This ongoing research-creation proposes ecotechnologies of practice in a crafting of
“response-abilities.”¹⁴ The tree crafts problem-solving operations with the
surroundings: tackling water preservation or evapotranspiration as needed, regulating sap flow, making clouds,
coordinating burgeoning and senescence timings, going dormant in northern winters, turning sapflow into
antifreeze to prevent embolisms.¹⁵ Furthermore, trees produce photosynthesis in a cosmic relation of
sun and soil, they yield oxygen and even grow stronger and bigger with the extraction of carbon dioxide from the
atmosphere.¹⁶ They communicate through VOCs (volatile organic compounds) and mycelium
networks.¹⁷ These are but a few of the tree's wondrous techniques, parts of a larger continuum to
coevolve their technologies over 350 million years.¹⁸ To partake gently in their knowledges,
Indigenous artist Domingo Cisneros proposes eating and drinking parts of coniferous trees without harming the
plant, a celebration to share in the strength of the longevity of their problem-solving techniques.¹⁹
In a similar celebratory fashion, medical biochemist, botanist, climate activist and author Diana
Beresford-Kroeger is as adamant as she is hopeful in times of changing climates. Her “Bioplan” makes
simple statements. She advocates to protect trees in the immediate vicinity as well as native forests in local
communities.²⁰ So efficient, it's alarming. Her approach changes the habitual urgent call
associated with resiliency and adaptation of diverse communities that are foregrounded through catastrophic
discourse. Change—with life-living trees—is summoned as operation. As Stenger states, consensus is
stasis, refusing (to) change. Beresford-Kroger's bioplan extends symbiotic intimacy with tree technologies.
Ongoing Doings
Ecotechnologies of practice entail a perturbation in the habitual conception of the relation of ecology and
technology, explored here as operative process. The tree's ecological practice, in conjunction with
that of arts and sciences, investigates changing climates with creative problem-solving in dialogue with
publics. The differential fields of knowledges bring new expression by their combination, a strength emerging
in-between, pushing forth modular alliances. Crafting changes is ecotechnologies of practice: even if ephemeral,
potential is actualized through encounters, soliciting variation and individuating collectivities.
Acknowledgements
The research-creation is conducted on the unceded lands of the Kanienʼkehá ka Nation. The
MÉDIANE team gratefully acknowledges funding from SSHRC, CFI, FRQSC and support from Hexagram-UQAM.
References
1 Jennifer Gabrys, Program Earth. Environmental Sensing Technology and the Making of a
Computational Planet, Minnesota, Minnesota University Press, 2016.
2 Isabelle Stengers, La guerre des sciences, Cosmopolitiques 1, Paris, La
Découverte, 1997.
3 Erin Manning, Out of the clear, Colchester and New York, Minor Compositions, 2022.
4 Gilbert Simondon, L'individuation à la lumière de notions de forme et
d'information, Grenoble, Jérôme Millon, 2005.
5 Isabelle Stengers, La guerre des sciences, 8.
6 Gilbert Simondon, L'individuation à la lumière de notions de forme et
d'information.
7 Accessed November 4, 2022, https://fridaysforfuture.org.
8 Isabelle Stengers, La guerre des sciences, 58.
9 Accessed November 6, 2022,
https://www.oxfordlearnersdictionaries.com/definition/english/technology.
10 Accessed November 6, 2022, https://www.etymonline.com/word/-logy?ref=etymonline_crossreference.
11 Accessed November 6, 2022, https://leverageedu.com/blog/ecotechnology/.
12 Carl Sagan, “Cosmos (1980),” IMDb website, accessed November 1, 2022,
https://www.imdb.com/title/tt0081846/characters/nm0755981#quotes.
13 Erin Manning, Out of the clear.
14 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham,
Duke University Press, 2016.
15 Jehová Lourenço Junior, Community Ecology - Wood Anatomy - Plant Physiolog,
Accessed November 3, 2022,
https://jehovajr.wixsite.com/science/post/the-hydraulic-architecture-of-trees-in-a-climate-changing-scenario
16As Carbon Dioxide Grows More Abundant, Trees Are Growing Bigger, Study Finds,
Accesssed October 10, 2022, https://e360.yale.edu/digest/carbon-dioxide-climate-change-bigger-trees
17 Suzanne Simard, Finding the Mother Tree, Allen Lane, Penguin Random House Canada,
2021.
18 Serge Muller, Germinal Rouhan, “Comment les arbres sont apparus sur Terre,”
Museum, November 30, 2021, accessed November 1, 2022,
https://www.mnhn.fr/fr/actualites/comment-les-arbres-sont-apparus-sur-terre.
19 Ælab and CREAF (Centre de recherche et d'expérimentation des arts forestiers),
“Les trois frères (2020),” https://vimeo.com/channels/472281/452356412).
20 Accessed November 15, 2022, https://dianaberesford-kroeger.com/mission/.
Author Biography
Gisèle Trudel and Stéphane Claude founded the experimental documentary artist research unit
Ælab in 1996 (aelab.com). Collectivities actualize the potential of MÉDIANE (2020-2025,
mediane.uqam.ca).
Water Stories: Visual Poetics and Collective Voices
AndreaWollensak,
BrettTerry, BridgetBaird
Ammerman Center for Arts & Technology, Connecticut College Sonalysts Emerita Computer
ScienceDepartment, Connecticut College New London [authors 1 and 2] and Waterford [author 3]
CT,USA ajwol@conncoll.edu brettcarrollterry@gmail.com bbbai@conncoll.edu
Abstract
Water Stories: Visual Poetics and Collective Voices is a two-part project that brings together
multiple points of view from local youth, community, and poets in Alaska to share what water means in their
life. Visual Poetics combines a live poetry reading by Alaskan poets and interactive video in which the
poets’ voices trigger generative visual elements. Collective Voices is a sound work
featuring excerpts of community voices sharing water-based memories against a backdrop of processed
environmental sounds of Alaskan waterways. Water Stories is part of a year-longartist residency
(2021-2022) with the Anchorage Museum culminating in a series of listening sessions broadcast at the Anchorage
Museum and Out North Radio, live interactive poetry readings at the museum,and video projections on the museum
façade from November 2022 through January 2023.
Keywords
Generative, digital poetics, audio-reactive software, community engagement, collaborative, ecosystems,resilient
societies, multiple scales.
Alaska’s climate is projected to warm several degrees by the end of the century, more than twice the
warming compared to the rest of the country. The record-breaking temperatures experienced by Alaska communities
have resulted in thawing permafrost, thinning sea ice, and more wildfires. The Water Stories: Visual Poetics
and Collective Voices project engages the Alaskan public to consider how climate change alters their
experiences of the land, and to consider the role of water in their lives.
Background
Water Stories is the most recent artistic collaboration between the authors, and is part of aseries of
works focused on climate change and the environment. Reading the Wrack Lines (2021),an outdoor audio
video projection on a lighthouse, featured community-sourced textual reflections on climate
change. Open Waters (2020) was a multiple-media installation exhibited at museums in Rhode Island,
and New York, connecting the five-hundred-year imaginative history of the open polar sea and northwest passage
with current geopolitics and arctic climate change. Another previous audiovisual collaboration, Ice
Core Modulations: Performative Digital Poetics (2017) included generative imagery and poetic texts
controlled via historical CO2 data taken from ice core samples made available from the National Snow and Ice
Data Center (NSIDC). All three of these works have been presented at past ISEA Conferences.
These collaborations have sought to create engaging works that address complex environmental issues through a
plurality of artistic, poetic, and scientific perspectives. The collaborators have brought to bear individual
expertise in audiovisual generative computer programming, visual art media, poetry, and electroacoustic sound
composition to synergistically create works unified by shared source materials that include place-based personal
narratives, historical source material, raw scientific data, local audiovisual media collected on site, and
original poetic texts.
Goals, Objectives and Artistic Dimensions
Water Stories seeks to amplify voices of Alaska residents through storytelling, poetry, and local
landscape imagery. The overarching project goals and objectives include:
Collaborating with local community partners.
Revealing evidence of a changing climate in lived spaces through creative voices and artistic media.
Using narrative storytelling as a way to bear witness to our changing environment through personal
experience.
Seeking to draw out the poetic from connections between visual arts, language arts,science/technology.
Presenting community voices via innovative multi-sensory formats.
Creating a reflective and engaging digital poetics experience thematically focused on environmental change
and disappearance of place.
The artistic dimensions of Water Stories are framed by community storytelling and creative
writing that focus on the local water and environment in Alaska. Based on these objectives, Water
Stories resulted in two distinct works showcasing community voices from Alaska:
Collective Voices is an audio composition that amplifies voices of Alaskan youth and community
through stories of what water means in their life.
Visual Poetics is a dynamic audiovisual work (Figure 1) developed with Processingsoftware in which
poetic fragments, background photography and abstract visual elements interact in response to a live poetry
reading.
Ecosystems of Collaboration
There are two synergistic types of collaboration taking place within the WaterStories work. In the
first, the collaboration between the creators of the work allows artists to bring their own unique perspectives
and disciplinary skill sets together. As the collaborators have previously worked together on site-based work
exploring themes of climate change, several methods already developed to facilitate this collaboration were
leveraged for this effort, in particular the use of original poetic phrases to provide asemantic glue unifying
different visual, sonic, and textual elements. In this case, the performative nature of the collaboration was
underscored by having a live poetry reading be a central element of the work. With the poet present, hearing
their words while seeing the work respond to their speech connected the underlying collaborative synergy between
text and image on the surface, and at a deeper level, with the connections implicit in the semantic connotations
and subject matter of the poetry itself.
The second level of collaboration in the work connects the audience and creative producers,capturing community
narratives through interviews and recorded reflections. CollectiveVoices empowers individual voices
through community collaboration, allowing them to interplay and be heard in a public creative context, fostering
a reflective audio-based ecosystem through which the listener experiences a plurality of narrative and poetic
perspectives. Through its use of intertwined recorded personal reflections, the work bridges the private/public
gap, and was shared publicly via broadcast on radio, internet, and as part of an outdoor audio installation at
the Anchorage Museum.
Visual Poetics
Alaskan poets Erin Coughlin Hollowell and Jen Stever each composed ten poems as part of this collaboration.
Framed by her Iñupiaq culture and history, Stever’s poems focus on the local environment, family,
and the changing climate. Hollowell’s poems explore careful observations of the coastline and reflections
of visual language and form. Distinct phrases from each poem were incorporated into the Processing software in
several different visual representations. Randomly selected phrases fade in and subsequently fade out within the
audiovisual work, with some phrases coalescing from and dispersing into granulated pixels.
During several trips to Alaska, photographic and video source materials were collected and used as background
imagery within the work (Figure 2). Adding to the text and shifting background imagery, different visual
elements and behaviors were developed in Processing for each poem, using a vocabulary of moving generative forms
include triangular networks, shards, and ellipses, each with distinctive color palettes tied to the background
imagery (Figure 3). A reactive audio system was developed to listen to the poet’s voice, using amplitude
and frequency to trigger and modulate the generative forms, with different behaviors for each poem.
Two versions of the work were developed. The first version was a live performance in which both poets were
present reading their works, accompanied by the audio reactive Processing software running in real-time to
generate visuals projected behind the poet. The second version (Figure 4) was made to project nightly on the
façade of the Anchorage Museum for several months, and was made by recording a performance of the work as
a video file. The two versions thus exhibit differences of scale, accessibility, and connection to the spoken
voice.
Collective Voices
To create the eight-minute audio composition Collective Voices, several recording sessions took place
at the Anchorage Museum and OutNorth Radio in which community participants shared personal narratives about
place and local waterways. The authors would like to gratefully acknowledge community partners Indra Arriaga
Delgado and OutNorth, a progressive arts organization with a mission to advance contemporary art in Anchorage,
support underrepresented voices, and promote cultural dialogue by amplifying the alternative voice. In order to
promote a diversity of voices, many different community groups were invited to participate including high school
students enrolled in the Anchorage Museum’s afterschool Teen Climate Communicators program and members of
the general public invited to record their stories at the recording session venues. Selected excerpts from these
narratives were edited into an audio work using Logic software and combined with multiple layers of local
processed environmental sounds from the Anchorage Museum’s audio collection. Collective Voices was
presented at a listening station at the Anchorage Museum during their November First Friday community event,
broadcast on Out North Radio, and made available for internet listening via SoundCloud.
Conclusion
Water Stories: Visual Poetics and Collective Voices engages multiple points of view from local youth,
community, and poets in Alaska to share what water means in their life. Grounded in a year-long artist residency
at the Anchorage Museum, this interdisciplinary project explores the changing environment through place-based
ecosystems of collaboration.
Authors Biographies
Andrea Wollensak is a multimedia artist, designer and educator. Her work spans media from traditional and
digital fabrication to generative-interactive systems and includes frequent collaborations across disciplines.
Wollensak’s work has been exhibited internationally, most notably at the Göteborg International
Biennial of Contemporary Art and the Brno Biennial of Design. Her work has been supported by the Rockefeller
Foundation, the International Artist Studio Program in Sweden, Anchorage Museum, Banff Centre for the Arts,
and the National Science Foundation. She has presented her artwork at numerous venues including ISEA, CAiiA,
Generative Art, and College Art Association. At Connecticut College, she is on the faculty in the Studio Art
Department, and an Associate Fellow at the Ammerman Center for Arts and Technology where she served as
Director from 2012 to 2020.
Brett Terry is a composer and sound artist, when not busy with his daily life as a software engineer. His
electro-acoustic, choral and chamber compositions have been performed at venues such as SEAMUS, ICMC, ISEA,
CAiiA, and Sound Culture in addition to collaborating with visual artists on numerous audiovisual works. As an
associate editor of Computer Music Journal (MIT Press), he has curated a special issue on Visual
Music.
Bridget Baird is a Professor Emerita in Computer Science and Mathematics at Connecticut College and a past
Director of the Ammerman Center for Arts and Technology. Her research examines the intersections among the
arts and various technologies. Projects include investigating an archaeological site in Ecuador through
virtual reality, exploring music and dance through motion capture and multiple modalities, using digital
techniques and algorithms to better understand and mine historical documents, and more recently, addressing
climate change and environmental concerns by using generative art. Baird collaborated, as a Fulbright scholar,
with colleagues in both Mexico and Ecuador. Involvement with the local community has also been important to
her and a constant interest has been to increase the number of women in the sciences.
Nga manawataki o te koiora: biological rhythms, posthuman design and decolonial thought.
RewaWright, SimonHowden
Queensland University of Technology Independent Artist Meanjin/Brisbane, Australia
rewa.wright@qut.edu.au
Abstract
Western science, in fields such as computational ecology, has grown to accept the truths that Indigenous
culture have long known: that computational ecology accepts that ecological models are too complex to be
summarised in computational form. Since this complexity evades the codification of mere indexing, how then,
should we work with computational companions (code, algorithms, programs, platforms). What new ways of
intra-acting can we develop alongside computational frameworks, which bring us one more step closer to sentient
machines? Most importantly, how can ethical ways of thinking and doing motivate transformations in the
computational space, in areas such as machine learning where extreme problems of bias are now embedded? This
research does not answer these complex questions, for they are genuinely ‘wicked problems’ that
reach toward wider issues of equity, sustainability, and economy. Our aim is to use creative practice to
generate gestures and markings that tentatively trace a way forward. This research contributes to new modalities
of human computer interaction that attempt to restore the dynamic pathways developed by Indigenous thinking,
challenging artificial boundaries such as nature/culture, instead giving respect to concepts of interconnection.
Examining some of the differences between Western epistemology and Indigenous thinking opens a pathway toward
Indigenous Futures that are crafted in support of a decolonial ecology.
Through imaging technologies such as electron microscopes, quantum physics has been slowly proving to itself
that all particles in the universe are deeply entangled. Now, the curious situation exists where Western science
has ‘validated’ what mātauranga Māori (Māori knowledge) has known and preserved
for thousands of years. A close analogy for the unseen movements of quanta is mauri (transitory and shifting
states of being), while the structure these quanta form with other particles, emerges in wairua (felt
interconnections between all living things). In the Māori world view, people and nature are inseparable
and share genealogy. This resonates with current scientific opinion which has only come around too lately to the
perspective that plants are to be as valued as humans. Indigenous thinking is in fact far ahead of Western
science, in that we never lost the intuitive ways of knowing and being, and we never blindly accepted the
artificial separation of mind and body of Descartes’ Enlightenment edict. Through quantum imaging in the
1990s, physicists finally saw that the whole world was connected, every nanoparticle and atom is entangled with
every other. This was not new to mātauranga Māori, which for thousands of years had preserved that
intuitive knowledge of deep entanglement in our concepts of wairua (life force, shared between all living
things) and mauri (transitory and shifting states of being). Underscoring the radical cosmological contingency
that materially binds humans to all other entities, the concept we know as mauri is expressed to a degree in
quantum physics. That is, the principle that nano level particles indeed bind together very entity in the
universe. Our work is about the entangled connections that are intuitively felt, and join humans,
nonhuman/more-than-human kin and algorithms in a constant state of becoming. Our research thinks alongside
evolutionary biologists such as Monica Gagliano and Stefano Mancuso, who argue that plants are intelligent
[2,4]. Situated agency emerges as a tangled network of augmented reality infrared vision, gestural and plant
bio-electrical data, audible through a bespoke sensing network of hardware devices and custom software.
Understanding the invisible world humans cannot see is important, therefore, to weave an understanding of
interconnection back into the process of everyday life. In our work, plants signal to one another as part of
their growth process, and we captured these signals as MIDI, an audible expression of their invisible movements.
The arhythmic and micro-temporal signals were then incorporated as a foundation to co-create a sound scape with
our plant companions. Contact/Sense explores the entangled connections that are intuitively felt between humans
and other life forms, sentient or not. Framing the eco-digital within our artistic practice is the scientific
context of evolutionary plant biology, on the one hand, and the spiritual Indigenous approach to knowledge as
beyond human, assembled in complex networks that are not only empirical but acknowledge unseen unquantifiable
forces and the insights it gives as to human behavior and thinking in relation to plants. Agential realist
accounts such as those elaborated by Barad [2008], go some way toward destabilizing, at least within posthuman
philosophy, the Humanist/Enlightenment duality, opening space for extended speculation on matter and meaning.
However, these Western systems occlude the cosmology of knowledge systems like mātauranga Māori.
Underscoring the radical cosmological contingency that materially and spiritually binds humans to all other
entities, mātauranga Māori traverses both material and spiritual realms, pointing toward a universe
that is not entirely knowable, where empirical claims to deep knowledge about the production of matter by
material forces are only partially relevant. Permeated with temporal and spatial relations that defy the "arrow
of time" and granulize past, present and future into a non-linear series of instances, mātauranga
Māori affirms the validity of techniques and methods that trace speculative knowledge.
According to plant neurobiologists Mancuso and Viola, plants are not "passive machines" for processing light,
water and food: In fact, they are "intelligent" and show this by constantly signaling to one another and to the
world, for example, giving off scents that attract or repel insects and even sharing resources, such as water,
amongst their communities [4]. Tracing the connections between quanta and nanoparticles as mauri and wairua as a
starting point for a discussion of new media art in a decolonial framework, this paper reflects upon
intersectional threads within art, physics, data and posthuman design, to weave an interconnected path between
Western and Indigenous science, philosophy and cosmology. We use our practice in media art blending
computational networks, human bodies, and plant signals to explore the metaphysical and embodied knowledge
manifest through a decolonial symbiosis or science, art and technology, and inclusive to all humans as well as
our "more-than-human" interlocutors (plants and algorithms). Concepts of symbiosis may seem attractive since
they indicate hybridity, however, as art practitioners we need to be mindful of the power relations in any
merger: so that in our writing we approach symbiosis in a decolonial sense, to alleviate the tendency for a more
powerful force to overtake a lesser one.
"More-than-human" creativity
"More-than-human" creativity is an emergent thread that reaches toward forming relations with more-than-human
kin, organic and silicon, such as plants and algorithms. Through research that bridges plant signaling with
augmented and extended reality, machine learning, and interaction design for virtual environments, this project
explores the entangled quality of human creativity in the computational networks and living ecologies. This
embeddedness, or interdependency, or entanglement, between the organic and the silicon enables new forms of
creativity, where artists’ partner with nonhuman kin via iterative acts of co-creation.
This approach destabilizes traditional principles of design, which emanate from the intentionality of the
artist or designer. A more-than-human approach allows for an open space of generative creativity where unplanned
events and phenomena can unfold, shift, and disrupt the final work. Through practice-based and traditional
research, the installation project Nga manawataki o te koiora: Biorhythms, has several interconnected
aspects. Indigenous art is conventionally seen as manifesting tradition and not technology. However this work
overturns that power relation, where embodied cultural knowledge is allowed to flow through computational
networks.
Embodied knowledge and algorithmic design
Nga manawataki o te koiora: Biorhythms, is a projection mapping and installation piece that takes you
on a journey into a computational transduction of the forest, rivers and oceans of Aotearoa/New Zealand.
Visually, an interconnected natural ecology is translated into the real-time world of audio reactive geometries
and mesh topologies. The concept was to convey the feeling of these things, without literal interpretation.
Traditional kete (woven baskets) inspire fluid movements which become pixel topologies. Animated motion made
with noise oscillators, shift from reimagined nets used to catch eel (hiinaki), to seed pods exploding from
pixel plants, such as the red pōhutukawa. The soft blue/green of kina (sea eggs) become fluffy vectors
transparently overlaid on a fluid mesh of waves. Following and modifying the tradition of naturalism and
curvilinear geometry that marks traditional Māori art, this piece visually encapsulates the feeling of the
natural world without being a literal representation that vested in Western pictorial traditions of realism.
This 20-minute piece, recorded as live audio reactive in Touch Designer, is sonically a composite of
human-nonhuman music, alternately interspersed and mixed together. From the "human" side, music consists of
three original electronic compositions by Simon Howden. From the plant side, we intersperse original recordings
of plant sonics captured in research since 2019. We consider plant to be co-composers of this work. A previous
installation, Contact/Sense was performed at the SIGGRAPH Asia Art Gallery in Brisbane 2019, and
combined plant sonics with mixed reality. Donna Haraway introduced the notion of "companion species" to describe
non-human organic life forms that we cohabit alongside in society and culture. Plants and humans have lived
alongside one another for thousands of years, in a co-dependent relationship of care and cultivation. Applying
the decolonial philosophy of mātauranga Māori (Māori epistemology), combined with posthuman
lens to art and science, our hope is that through this artwork, people will feel a little closer to the hidden
bio-electrical processes of plants, and consider plants not as a resource for extraction, but as a
‘companion species’ in a sustainable ecology.
The plant sonics were recorded during earlier live audio-visual performances with an agave attenuate,
a series of vines, several palms, and a multitude of grasses, tropical and sub-tropicals since 2018. Rewa Wright
has developed a unique mode of mixed reality performance with plants and algorithms networked with the human
body. The plant signals are essentially bio-electrical impulses, which we then assign to MIDI and apply sound
design to, so that soft wooden drums with loose skins and resonant tapping highlight the micro temporality and
asymmetry of plants, whose signals sound unstructured to the human ear. Plant rhythm is phenological, and traces
their processes such as photosynthesis and osmosis as they follow patterns of growth. Several plant
neurobiologists have noted the signals that plants emit are akin to intentional communication and sentience, and
this is now a recognized area of scientific study.¹, ²
Plant neurobiology speaks to the material reality, unraveled by quantum physics, that every particle in the
universe is connected to its nearest neighbors, and through those neighbors to all other things. Quantum imaging
has provided visual proof of this connection, as discussed extensively by philosopher Karen Barad,³ whose
concepts of "situated knowing" and "intra-action", both grounded in quantum (meta-)physics, have resonance with
Indigenous notions of deep time/space and interconnectivity. As a foundational concept, the deep interconnection
of life forces (i.e., quantum entanglement) has always been known to indigenous people, and in my culture,
Māori from Aotearoa/ New Zealand, we understand this unbroken link between vegetal, organic, silicon,
geological forms (to name just a few categories) to underpin all material and cosmological reality.
Understanding the invisible world humans cannot see is important, therefore, to weave an understanding of
interconnection back into the process of everyday life. In our work, plants signal to one another as part of
their growth process, and we captured these signals as MIDI, an audible expression of their invisible
movements. The arhythmic and micro-temporal signals were then incorporated as a foundation to co-create a sound
scape with our plant companions. Using computer art to translate emotions about place from bush into pixels,
this work explores the potential for a symbiosis of data, plants, ecology and algorithms. This work is deeply
influenced by Rewa’s cultural background as a First Nations Māori artist, from the Ngai Tawake, Te
Kaimaroke, and Te Uri o Hau hapu of Aotearoa/New Zealand. Her ancestry recognizes human and plant relations
going back thousands of years, and Rewa is committed to multi-species justice and sustainable living on Planet
Earth. Mixing the biorhythms of plants and the calls with human produced electronic music, reveals a co-creative
mesh of human and nonhuman kin, an entanglement that is fundamental. For example, in Rewa’s pepeha
(personal genealogy), she traces lineage to the maunga (mountain) called Tokatoka, the awa (river) called
Wairoa, and the moana (sea) called Kaipara. These connections are at the base of her identity and this is the
same for all Māori, since our genealogy is tied to the whenua (land). Transducing this physical connection
with the land into computational space is only ever partial. However, hints at the ways Indigenous knowledge
might be embodied as data and algorithms, nurturing a framework that advances decolonial thought and gestures
toward Indigenous futures. While quantum imaging has provided visual proof of this connection between all
vibrating atoms and matter, it is a concept that has always been known to indigenous people, and in my culture,
Māori from Aotearoa/ New Zealand, we understand this connection between vegetal, organic, to underpin all
material and cosmological reality.
References
Books
1 Paco Calvo, Monica Gagliano, Gustavo M. Souza, Anthony Trewavas, Plants are intelligent,
here’s how, Annals of Botany 125, no. 1, 2020, 11-28.
2 Stefano Mancuso, Alessandra Viola, Brilliant green: the surprising history and science of
plant intelligence, Island Press, 2015.
3 Karen Barad, Meeting the universe halfway: Quantum physics and the entanglement of matter
and meaning, duke university Press, 2007.
Authors Biographies
Dr. Rewa Wright (Ngai Tawake/Te Uri o Hau/ Te Kaimaroke) is Senior Lecturer in Film, Screen and Animation at
the Queensland University of Technology, Australia. Simon Howden is a sound designer and independent music
producer. Both humbly reside in Meanjin/Brisbane on the lands of the Turrbal and Yugara, First Nations owners.
We recognise that these lands have always been places of teaching, research and learning, and that sovereignty
was never ceded.
The Dark Side of the NFTs: The artists’ need for new systems of collaboration
İpekYeginsu
Kadir Has University Faculty of Communication, Department of Visual Communication Design
Istanbul, Turkey ipekyeginsu@khas.edu.tr
Abstract
NFTs are often imagined as the initiators of a democratic revolution in the art world, and yet their effects
are much more complicated than meets the eye. While the financial independence and the new exposure channels
they offer to the artists are undeniable, they also disturb the art world’s current equilibrium, resulting
in the emergence of a chaotic art ecosystem that makes the independent artists’ already existing
vulnerabilities even more pronounced. This paper defends the position that artists necessitate new systems of
collaboration to sidestep these negative effects, overviews some promising examples of such attempts and
discusses the potential solutions for the future.
Keywords
Art ecosystem, digital art, NFT, blockchain, crypto art, artist communities, collaboration.
In today’s art ecosystem radically reshaped by the conditions of the Covid-19 pandemic, the artists
working with digital technologies have more exposure channels available to them compared to those working with
traditional media. The former are able to exhibit their works online without facing the criticisms the artists
using conventional media often receive for choosing a digital exposure platform, and they are free from the
costs associated with transporting, insuring and installing a physical exhibition. Neither do they necessitate
the partnership of a gallery or a museum to present their works; social media platforms and NFTs allow them to
sidestep the gatekeepers in the mainstream arts and culture institutions, contributing to the emergence of a
more democratic and decentralized art world. In fact, Sarah D. McDaniel and Denny Galindo observe that
“art galleries and auction houses have been the market makers of the art world”, and “NFTs and
their marketplaces disrupt this process by allowing more artists to go straight to market and sell directly to
buyers.”¹
The NFTs’ market success empowers the artists in other ways as well. For instance, Bernard Marr draws
attention to the individual power the NFTs have bestowed upon Pplpleasr, the artist also known as Emily Yang,
who uses her NFT sales’ revenue to support the other artists and especially those of Asian descent.²
Artists without a digital background are also able to access this unfamiliar territory thanks to a new line of
service providers facilitating migration into the digital realm without being lost in technicalities. A good
example for such endeavors is Vive Arts, a company specialized in the sales and exhibition of art “in all
digital formats”, assisting its clients starting from the artwork’s creation phase.³ On the
other hand, for the digital artists wishing to develop their skills, Visa launched the “Visa Creator
Program” to “tutor a curated selection of digital artists through the NFT process, helping them
maximize their talent and use the blockchain space as both a creative tool and marketplace.”⁴
Although NFTs emerge as a convenient funding source for independent artists, they are also criticized for their
adverse effects on the environment due to their heavy energy consumption and the subsequent carbon emission. But
a more subtle phenomenon challenges the artists in other ways that might threaten their career sustainability in
the long run unless they succeed at establishing new systems of alliance and collaboration. The following
section addresses both the NFTs’ ecological impact and these relatively implicit effects in more detail.
NFTs and Chaos in the Art Ecosystem
NFTs and Climate Change
The NFTs’ impact on the environment has been a source of ardent debate since the beginning. Some artists
have been among the leading opponents of their use in the art world due to their adverse effects on the planet,
and they have also been expressing their concerns publicly. For instance, when media artist Joanie Lemercier
discovered that the sale of his NFT series consumed the same amount of energy needed to run his studio for two
years, he decided to cancel his two upcoming NFT releases.⁵ Another media artist, Memo Akten, designed a
system to assess Ethereum’s carbon footprint in numeric values and created the website
“cryp-toart.wtf” to share the most up-to-date information about the issue.⁶ Nowadays, he
continues to produce NFTs on “eco-friendly blockchains.”⁷ However, others like Eric James
Beyer claim that “criticism is exaggerated and conflates hype with the truth.” Beyer compares the
NFTs’ yearly energy consumption with that of the small mining industries, giant tech’s data storage
facilities or residential air conditioning devices, and reports that the former is relatively lower. He adds
that new measures to reduce carbon emission rates are also being implemented, such as Ethereum’s migration
from a PoW (Proof-of-Work) to a PoS (Proof-of-Stake) system, and mentions various ecofriendly trading platforms
as plausible alternatives.⁸ Raisa Bruner is another optimist mentioning the success of the Canadian
company CurrencyWorks in “turning oil waste into environmentally friendly energy that powers crypto
mining”, and the Israeli company StarkWare that was able to invent a technique to store more information
in one block unit, reducing the system’s overall energy consumption.⁹ Even Mike Winkelmann, the
artist widely known as Beeple, ensures the public that he will eventually manage to “completely offset
emissions from his NFTs by investing in renewable energy, conservation projects, or technology that sucks CO2
out of the atmosphere.”¹⁰ Efforts at building synergy between the NFT min(t)ers and renewable
energy sectors are already plenty. But whether the artists are ready to face the challenges brought about by
this new technology reshaping the art ecosystem is seldom addressed.
The NFTs’ Negative Effects on the Artists
The problems regarding the NFTs are not limited to their ecological impact; in addition to their advantages,
they have adverse effects on the artists’ careers as well. In his article on The Atlantic, Anil Dash
observes that “the current NFT market is drawing an extraordinary range of grifters and spammers”
and the artists’ works are regularly being hijacked on social media and converted to NFTs and sold by
other people without their permission, so much so that, when an app designed for blocking user groups on Twitter
was released, its leading customers were the artists “using it to block NFT spammers from hijacking their
works and monetizing them as NFTs without permission.”¹¹ The artists’ vulnerabilities in
the digital environment are also pointed out by Cass Marshall, who explains that the artists exhibiting work on
social media and worrying about hijackers ultimately decide to avoid online presence altogether. And if the
hijackers are able to generate the right type and amount of speculation around the work, they are able to obtain
much higher revenue from the same work compared to the original artist.¹² Thus, as Simon Spichak says,
it is no surprise that “much of the pushback against NFTs is coming from the artists that they were
supposed to help”, due to the unbridgeable difference between the copyright culture of the latter and
those of the newly emerging NFT fans. He adds that only 50% of the artworks in the NFT market are sold for
prices above 200 US dollars, and no correlation exists between the work’s artistic quality and financial
worth. To the contrary, the majority of the astronomic sales prices belong to works without artistic
sophistication.¹³ Andy Storey identifies yet other ways in which NFTs negatively impact the
artists’ careers, such as speculative and fluctuating prices hurting the artists’ overall market
value, the NFT platforms’ high entry costs, low respect for the works’ artistic value, and the
possibility of the work being used in contexts the artist does not wish to be associated with.¹⁴
The NFTs have also become a source of conflict for various actors in the art world. The artists continuing to
produce NFTs and the NFT platforms offering their works for sale are often pressurized by the anti-NFT artists
to stop doing so. The pressure can be so serious that, for instance, the artist portfolio platform ArtStation
had to cancel its upcoming NFT initiative due to intense online criticism by its users based on environmental
concerns.¹⁵ The platform was even compelled to release an official statement apologizing for
“all the negative emotions” that their actions caused.¹⁶ Similarly, the artists
participating in the Art Wars project declared that the project’s initiator, artist and curator Ben Moore
did not ask for their permission when producing their works’ NFT versions. Consequently, the NFT
marketplace OpenSea decided to remove the project from their inventory. And complaints do not only come from the
artists either; they target the artists instead. Hermès, a giant brand, sued the artist Mason Rothschild
for releasing “MetaBirkins” or “100 NFTs resembling the iconic Hermès Birkin
handbag.” The result of the lawsuit will also impact the collectors who have already bought items from the
collection.¹⁷ Rothschild defended himself by arguing that under the First Amendment, he had the right
to “create art based on [his] interpretations of the world around [him]” and drew an analogy between
his work and Andy Warhol’s Campbell Soup Can, but it was not sufficient for the court to dismiss the
case.¹⁸ Put differently, the NFTs emerge as an important source of chaos and conflict in the art
world, particularly in relation to issues around ecology and copyright.
In sum, the NFTs constitute an additional income source and an alternative exposure channel especially for the
independent artists, allowing them to bypass the gatekeepers in the museums, galleries and auction houses. But
they simultaneously give rise to a more chaotic art ecosystem in which artists have to choose between increased
visibility and the violation of their intellectual property rights, or producing critical work referencing
iconic images and facing corporate lawsuits. Environmental concerns associated with NFT minting and transactions
add further complexity to the issue, increasing the tension between pro-and anti-NFT artists. Moreover, due to
the medium’s accessibility to everyone which appears to be an advantage, the artists have to compete
against the NFT creators without an artistic background who prioritize financial return over artistic quality.
New Systems of Alliance and Collaboration for the Artists
As a response to the challenges at hand, artists necessitate new systems of alliance and collaboration allowing
them to navigate this chaotic art ecosystem more safely. Apparently, one of the basic strategies with fruitful
results is to join forces with other artists in the form of co-productions. In fact, fine art photographer
Gabriel Dean Roberts explains that he finds NFT collaborations with other artists and even cross-disciplinary
collaborations with musicians highly beneficial, as each side brings their own audience into the picture while
the uniqueness of the collaboration itself is a form of added value. Joint ventures also contribute to the sense
of community among NFT artists by highlighting interconnectedness.¹⁹ Such a solidarity project took
place in 2021 when a hundred artists jointly created a single NFT composed of a hundred parts that sold out
within a few minutes. According to the project’s curator Loopify, the artist list consisted of a few
established figures and “one that is new to NFTs”, and the idea emerged in response to “the
current limitations for lesser-known artists to mint and sell their own NFTs such as high gas fees, limited
understanding of the tech, and little visibility”. Moreover, to ensure that the artists were exempt from
paying additional fees, the revenue was first transferred into USD coin, from which each artist received an
equal share. ²⁰ These examples illustrate that artist alliances around NFT creation have the
potential to enhance the visibility of every individual artist while reducing the costs associated with NFT
production in comparison to doing so single-handedly, and they particularly benefit the emerging artists with
limited experience in crypto art. These, in turn, could increase diversity in the art world, ultimately
contributing to its long-term sustainability. Collective production also reduces the carbon emissions per capita
associated with individual NFT minting, which would minimize the NFTs’ negative impact on the environment
and further contribute to the sustainability of the NFT art ecosystem.
The problems related to copyright, intellectual property theft and hijacking require even more organized action
on the part of the artists and a closer collaboration with the legislative parties as well as with legal
professionals specializing in this subject. Jessica Rizzo observes that, ultimately, the authority resides in
the hands of the courts whose exemplary decisions will gradually establish the principles dominating the
creation, distribution and trading of the NFTs. According to Juliet Moringiello from Widener University, the
laws of intellectual property in the physical world are actually valid in the web3 environment as well, and she
refers to the 1996 “Declaration of the Independence of Cyberspace” by John Perry Barlow as a similar
example where his attempts at justifying the bypassing of the existing laws ultimately failed.²¹ Yet,
the legal framework is still insufficient for the artists even in the United States. For instance, on its
official website, Thienel Law LLC refers to The Visual Artists Rights Act of 1990 (VARA) as well as The Lanham
Act (Trademark Act) and the Copyright Act as only partially protective documents.²² And as the current
legal framework around NFTs is still in formation, the artists themselves should be more directly involved in
the establishment of these principles. One way in which they have already begun to make a positive difference is
to hold the minting and sales platforms accountable for regulating their clients’ activities with regards
to intellectual property. A good example for such systems is provided by OpenSea; in 2022, the platform
implemented an anti-plagiarism mechanism consisting of both algorithmic detection and review by actual
humans.²³ OpenSea also enabled a stolen art reporting system thanks to which the original artist can
file a takedown request against the plagiarist, although the system does not guarantee that the request will be
accepted.²⁴ Another solution could be to have the NFT market platforms involve the artists themselves
in the artwork curation and elimination processes, so that unethical behavior by a member of the community is
directly sanctioned by that very community, e.g., by banning the user from the platform following the receipt of
a certain number of complaints. In short, collective action on the part of the artists can be effective in
pushing all NFT platforms to take the necessary measures, which would also accelerate the establishment of a
basic legal, or at least ethical framework for NFT art creation and transaction around the world, and would help
reduce the chaos in the NFT art ecosystem.
As illustrated by the MetaBirkin case, brands are another crucial party in the legal disputes over NFTs. One of
the most difficult dilemmas emerges around the artists’ right to borrow iconic imagery from popular
culture vs. the brands’ urge to protect their intellectual property. This is one of the most sensitive
territories for the future of artistic creativity. If the courts decide to overprotect the brands, they will
also have acknowledged that at least half of the seminal artworks from the 20th century are copyright
violations. Such a verdict would condemn all the artistic practices involving appropriation, collage and
pastiche, and it would do so because it “can” thanks to the digital technology facilitating
surveillance and tracking. This, in turn, would seriously deprive the NFT world from the ability to produce art
with a political or critical stance, almost “sterilizing” it into a corporate-friendly environment.
The dilemma is further complicated by the fact that brands complain about the NFT artists in the same way that
artists complain about their online hijackers, and technically differentiating the two contexts is not always
easy. Introducing itself as “the first blockchain ecosystem built by the art community, for the art
community,” Arcual is a promising initiative in this regard. Jointly founded by Luma Foundation, Art
Basel’s parent company MCH Group and BCG Digital Ventures, the platform aims at creating a clean NFT
trading environment and offering the artists additional revenue from secondary sales by using “smart
contracts with embedded resale terms.”²⁵ Again, the main dilemma presents itself as the
trade-off between creative freedom and financial/legal safety, for companies founding such initiatives also have
vested interests in the corporate world especially in the form of sponsorships, and whether in the case of a
dispute between an artist and a brand they would come in defense of the former instead of the latter leaves a
big question mark.
Conclusions and Ideas for Future Research
As illustrated by the overview above, the NFT market’s global outreach and independent environment is
extremely beneficial for the artists, but it also intensifies their existing vulnerabilities especially in terms
of intellectual property rights by destabilizing the art ecosystem as well as contributing to the deterioration
of the ecological one. As the ethical framework of this newly emerging territory has not yet fully developed,
the artists might still seize the opportunity to directly shape its future if they manage to act in solidarity,
hold the NFT trading platforms accountable and insist on directly participating in the ongoing legislative
processes. Further research is necessary to deepen our understanding of the dynamics dominating the NFT art
world, the existing legislation at the national, regional and/or global level and how artist communities do and
can operate in the face of the existing challenges. Most importantly, further study is necessary to explore how
broader artist communities can be mobilized for collective action.
References
1 Sarah D. McDaniel, Denny Galindo, “Democratizing Art: How NFTs Are Reshaping the Art
World”, Morgan Stanley website, accessed December 1, 2022,
https://www.morganstanley.com/articles/nft-art-market-nft-collectibles#:~:text=NFTs%20and%20their%20marketplaces%20disrupt,galleries%20and%20artists%20get%20paid
2 Bernard Marr, “Web3, NFTs, And The Future Of Art”, Forbes, August 19,
2022, accessed December 2, 2022,
https://www.forbes.com/sites/bernardmarr/2022/08/19/web3-nfts-and-the-future-of-art/?sh=6731712f1e05
3 Vive Arts, “Combining industry-leading metaverse technologies with bespoke services, Vive
Arts marketplace showcases and sells art in all digital formats”, Vive Arts website, accessed December
1, 2022, https://www.vivearts.com/about-platform
4 Mark Hunter, “Visa Launches NFT Artist Support Program”, FullyCrypto,
October 14, 2021, accessed December 1, 2022, https://fullycrypto.com/visa-launches-nft-artist-support-program
5 Gregory Barber, “NFTs Are Hot. So Is Their Effect on the Earth’s Climate”,
Wired, March 6, 2021, accessed November 30, 2022,
https://www.wired.com/story/nfts-hot-effect-earth-climate
6 Memo Akten, “The Unreasonable Ecological Cost of #CryptoArt (Part 1)”,
Medium, December 14, 2020 (updated December 2021), accessed November 30, 2022,
https://memoakten.medium.com/the-unreasonable-ecological-cost-of-cryptoart-2221d3eb2053
7 Memo Akten, “NFT”, www.memo.tv, accessed June 16, 2023,
https://www.memo.tv/works/nft/
8 Eric James Beyer, “NFTs and the Environment: Why the Anger Is Unjustified”,
nftnow, September 16, 2022, accessed November 30, 2022,
9 Raisa Bruner, “Environmental Concerns Have Cast Doubt on NFTs—But That’s
Changing”, Time, November 18, 2021, accessed November 30, 2022,
https://time.com/6120237/nfts-environmental-impact/
10 Justine Calma, “The climate controversy swirling around NFTs”, The Verge,
May 15, 2021, accessed November 29, 2022,
https://www.theverge.com/2021/3/15/22328203/nft-cryptoart-ethereum-blockchain-climate-change
11 Anil Dash, “NFTs Weren’t Supposed to End Like This”, The Atlantic,
April 2, 2021, accessed December 2, 2022,
https://www.theatlantic.com/ideas/archive/2021/04/nfts-werent-supposed-end-like/618488/
12, 15 Cass Marshall, “NFTs are generating huge paydays for some artists, others feel under
siege”, Polygon, March 12, 2021, accessed December 1, 2022,
https://www.polygon.com/22327806/nft-artists-online-theft-non-fungible-token
13 Simon Spichak, “Independent Artists Say NFTs Are the Bane of Their Existence”,
Futurism, January 24, 2022, accessed December 2, 2022, https://futurism.com/why-artists-hate-nfts
14 Andy Storey, “9 Reasons Why NFTs Are Bad For Artists”, Postergrind, May 7,
2022, accessed December 2, 2022,
https://postergrind.com/9-reasons-why-nfts-are-bad-for-art-ists/#:~:text=Artists%20claim%20that%20their%20art-work,is%20a%20lot%20of%20effort
16 ArtStation Team, “A Statement from ArtStation”, ArtStation, March 8, 2021,
accessed December 1, 2022, https://magazine.artstation.com/2021/03/a-statement-from-artstation
17 Simon Fitzpatrick, Rosie Adcock and Sophie Mellor, “February NFT Litigation Roundup: Art
Wars, Hermes "MetaBirkins", and more...”, Lexology, February 23, 2022, accessed December 3,
2022, https://www.lexology.com/library/detail.aspx?g=666c41a0-7815-4d21-a0f6-6da07ec1388e
18 Andrew Rossow, “The Hermès Lawsuit May Dictate the Future of NFTs”,
nftnow, May 19, 2022, accessed December 4, 2022,
https://nftnow.com/guides/how-the-hermes-lawsuit-could-deter-mine-the-future-of-trademark-rights-in-nfts
19 Gabriel Roberts, “NFT Collabs: Why You Should Work with Other Artists”, YouTube
video, 3:32. April 5, 2021, https://www.youtube.com/watch?v=AeW3dm8xU4Q
20 Joshua Mapperson, “100-artist NFT collaboration sells out in minutes, increases 7X in
price in 24 hours”, Coin Telegraph, March 4, 2021, accessed December 3, 2022,
https://cointelegraph.com/news/100-artist-nft-collaboration-sells-out-in-minutes-increases-7x-in-price-in-24-hours
21 Jessica Rizzo, “The Future of NFTs Lies with the Courts”, Wired, April 3,
2022, accessed December 4, 2022, https://www.wired.com/story/nfts-cryptocurrency-law-copyright
22 Thienel Law LLC, “NFTs and Artists: Understanding Intellectual Property Laws in the
Metaverse”, Thienel Law LLC web- site, July 1, 2022, accessed December 5, 2022,
https://www.thienel-law.com/blog/2022/6/23/nfts-and-artists-un-derstanding-intellectual-property-laws-in-the-metaverse
23 Protos Staff, “Humans and computers to fight NFT plagiarism on OpenSea”,
Protos, May 12, 2022, accessed June 16, 2023,
https://protos.com/humans-and-computers-to-fight-nft-plagiarism-on-opensea/
24 Jack Morse, “How to report plagiarized NFTs as stolen art”, Mashable,
April 13, 2022, accessed June 16, 2023, https://mashable.com/article/how-to-report-plagiarized-nft-stolen-art
25 Arcual Art website, accessed December 6, 2022, https://www.arcual.art
STAND BY/ME
YuZhang
STUDIO Ü Eindhoven, the Netherlands yuzhang.nl@gmail.com
Abstract
This paper describes initial intention, concept, concerns in design process, production, and technical details
of an artwork named “STAND BY/ME”. STAND BY/ME is an interactive installation that uses
machine-learning models to generate speeches for Xi Jinping and Donald J. Trump, and uses randomness to build
virtual conversations for the political spectrum, while connecting the information flow with the lived reality
of the everyday. Visitors are exposed to narratives of both digital communism and digital capitalism that is
randomly controlled. Next to the visual flow of randomly generated speeches, the work involves seemingly
mundane, yet “super-charged” electrical household items, i.e., power sockets, as the actuality of
human-technology confrontation. This work (1) allows visitors to feel like they are on “stand by”
and (2) triggers questions about how technologism impacts individuals' views and information consumption, while
(3) people face randomly generated political speech as a quiet, mundane confrontation. In sum, visitors are
invited to be bystanders of randomized political speech, visualized through streams of text ad made tangible
through everyday technology.
Keywords
Information representation, information flow, digitals and physicals, social media, technologism, politicism.
With the advent of smart and learning systems, connected technology is no more a simple luminous torch to shine
upon our wishes or serve our needs. It senses a multitude of signals from our environment, and shapes our
thoughts, behavior and social interactions. We find ourselves at the moment when technologism becomes an almost
unavoidable aspect of the everyday. Most often our true realization is overshadowed by technological wonder or
even deeply influenced by a form of techno-mysticism. When we place the finger on the screen and swipe it, an
“informative” world starts and soon it is filled with bits and glimpses of ever-faster news cycles.
As individuals, we consume the abundance of news that produced to capture our attention, move us emotionally and
sometimes even take action. Over time, we reach a state of being alert and at the same time wholly incapable of
action, especially when consuming news about politics. The initial intention of the STAND BY/ME installation
(Figure 1) is to synthesize the experience of “stand by,” in the context of technologism and
politicism, where each individual can view cycles of randomized and virtual information from two political
“speakers” as a bystander. “Stand by” expresses a human inability to act socially, as an
inhibition of the mind, a perversion of mindfulness. It is a deeply tiring state. The deeper intention behind
this art scene is to state: when an individual is incapable of action, digital data still changes its meaning
and form just depending on its medium of distribution, communication and interface.
“Stand by” and STAND BY/ME
STAND BY/ME focuses on the concept of “stand by” and transforms feelings of confusion,
in-betweenness and randomness into an immersive audio-visual experience. This experience is designed to trigger
visitors’ reaction to human and technological expressions of “stand by”, and visitors’
feeling when encountering a situation gripped by a general sense of “stand by.”
The physical part of this installation is composed of 30 linked power sockets (Figure 2), which are
technologically extended and enhanced with light and sound actuation. Apart from the layer of connected physical
sockets, the installation incorporates all public speeches by Chinese president Xi Jinping since
2014¹, ², ³ and thousands of tweets by U.S. former president Donald J.
Trump since 2016⁴ until January 2021. A recurrent neural network (RNN) was trained with the speeches and
tweet corpus, and we fine-tuned the parameter space to obtain synthetic speeches that express the current
acceleration of digital communism and digital capitalism. The two “speakers” are then visually
juxtaposed with a simulation of randomized draws of numbers (balls) from the National Lottery.⁵ The
lottery serves as the metaphorical random actor in this installation: conceptual and technical randomness
controls relations between recurring concepts in each politician’s generated speeches. The content and the
processing of approaching information data are displayed as a digital information interface (Figure 3).
This work projects the patterns of information flow from the machine-generated speeches first, and connects
with the lived reality of the everyday. The ordinary power sockets, “super-charged” household items,
visualize a connected data system that enacts the randomness in the actuality of human-technology confrontation.
Throughout the design process, this work touches on four symbolic and logic assumptions in the concept
“stand by.”
Two Politicians
As two “speakers,” Mr. Xi and Mr. Trump have different ways of approaching the public, quite
oppositely.⁶ In 2020, we have read a lot about “truth” by analyzing and interpreting the
information and actions from Mr. Xi and Mr. Trump. The citizens have been (still) confused about information and
have perceived them as a strongly regional particularity. STAND BY/ME uses a format of public stances and, at
the same time, is filled with random content—formal posing as a facade. The concept of STAND BY/ME was
accomplished around March of 2020. At the moment of writing this paper, the new conflicts in the world started
to develop —presidential election result for the 2020 US election became a turning point for American
foreign policy regarding Beijing, followed by Xi has secured a precedent-breaking third term as the Communist
Party’s leader and Donald Trump announced a White House bid for 2024. This work is filled with future
projections including the efforts to establish dialogues between different cultures and regions. The result of
such efforts seems very “virtual,” either in this work or in the reality.
National Lottery
National lottery symbolizes the daydream of picking easy money, yet participating in a random gamble. The
format of National lottery used in this work contains three main contents: jackpot size, six ball numbers and
one bonus ball number. STAND BY/ME uses this format but creates new feeds that are machine-generated random
numbers.
In STAND BY/ME, the lottery is a metaphor for randomness and offers each visitor only a passive role who are
not able to take action in the process—intelligent agents submitting to luck.
Power Socket
The power socket in its most ordinary form is a distribution hub for electricity. Here, its function is
transcended: it is a physical connection hub which is available all the time to connect devices that can send
and receive all information packages. Power sockets often embody the most common notion of indicating
“stand by,” a glowing LED. The use of power sockets in this work metaphorically refers to the action
“to be on charge” and similarly as an active participant in propagating information and visualizes
the information flow in the audio-visual patterns. Through multiple power sockets, STAND BY/ME embodies a sense
of “something somehow happening,” and visitors are incapable of actions.
Interaction
Interaction in this installation went through different stages: from a means to engage visitors in exploring
and discovering the emptiness behind the façade to the final implementation where interaction is
deliberately withheld from the visitor. The interaction becomes that visitors only remain passive in the
installation to simplify the dynamics of light and sound (reacting only to the data processing of the National
Lottery). By observing the data processing instead of being part of it, visitors are drawn into the processes of
how information spreads or hits a levee or how light and sound trigger the translation of the digital speech
data. As visitors realize that they are bystanders, in the back of their minds, they will feel the pressure and
obligation to act, without being empowered.
Technical Details
In this installation, a central single-board computer (SBC) runs a Processing sketch that (1) displays the
information interface with virtual speeches and conversations from two politicians and projects the results as a
live “dashboard” of speeches, highlights and lottery drawings (Figure 3), and (2) translates into 30
linked power sockets that are extended with light and sound output (Figure 2). Each of the 30 power extension
sockets is internally altered, while keeping its original appearance. Inside each socket, WIFI-connected
electronics (based on ESP32 boards) allow for decentralized control of each two LED lights and one speaker
module. The installation generates its own data stochastically modeled after patterns of information flows from
social networks. The patterns are mapped and distributed to all sockets via a message bus running on the SBC.
The dynamics of light and sound of each altered socket are driven by each sockets’ internal programming
and designed to be responsive to the data in the system.⁷ Let’s dive in a bit deeper. The
information display in figure 3 shows three columns. The rightmost column is the data display from National
Lottery. It contains four main parts—(1) data source and its introduction, (2) real-time lottery
information (jackpot size, six ball numbers and one bonus ball number) in technical randomness, (3) the latest
coming six ball numbers, (4) the keywords from speeches by each “speaker” that generated from the
latest coming bonus number. In the same figure 3, the two “speakers” are shown in the left and
middle columns with each four parts of content: (1) data source introduction, (2) real-time RNN generated
speeches, (3) up to six highlighted keywords from speeches selected by the latest draw from the National
Lottery, and (4) the statement selected from the speeches with shared keywords selected by the latest draw of
the bonus ball. The speeches differ in content and important repeating keywords, which are extracted in a ranked
list per speaker. These lists are used to draw keywords from by means of random numbers from the Lottery
draws.⁸
Conclusion
STAND BY/ME creates an immersive experience set up as interplay between humans, machine and artificial entitles
in a cognitively overwhelming way using text, light and sound. Although STAND BY/ME seems calm in general, it
puts visitors in an almost passive position when facing the complexity of the machine-generated progress. At the
same time, visitors soon discover the need to process each incoming scenario of an entirely modified and virtual
information for both digital communism and digital capitalism under the control of gambling randomness. The
installation unpacks the concept “stand by” in a collaboration of data sources, data system,
technologies, multiple electrical household items, and the digital interface. The design process of this work
dives into the process of refining audiovisual complexity with special attention to the details of information
representations, metaphorical or not. This work provides the prompts for different perceptions and enables
visitors to have a felt experience of the connection, distraction and confusion when digital data changes its
meaning and form at every point, depending on its medium and the current “mood” of randomness.
Acknowledgments
The grant of work was awarded by ARGEkultur in the frame of DIGITAL SPRING FESTIVAL. STAND BY/ME was shown in
the group exhibition “POLATECH 1” in the period of 27th of August until 6th of September, 2020 at
Albert van Abbehuis, Eindhoven. The work was also invited and exhibited at Athens Digital Arts Festival 2022.
This installation was built with much inspiration and support from Mathias Funk from the Industrial Design
department at Technology University of Eindhoven.
References
1 China.org.cn, Bilingual Texts, accessed October 30, 2022,
http://www.china.org.cn/english/china_key_words/node_7216079.html
2 Mission of the People’s Republic of China to the European Union, Speeches (Resources),
accessed October 30, 2022, http://www.chinamission.be/eng/zywj/ZYYJ/
3 Ministry of Foreign Affairs of the People’s Republic of China, Speeches (Policies and
Activities), accessed October 30, 2022, https://www.fmprc.gov.cn/mfa_eng/wjdt_665385/zyjh_665391/
4 Donald J. Trump, Tweet (text), accessed December 30, 2020, https://twitter.com/realdonaldtrump
5 THE NATIONAL LOTTERY, Lotto results, accessed October 30, 2022,
https://www.nationallottery.co.uk/results/lotto/drawhistory
6 Chris Buckley, “Trump’s and Xi’s Differences Magnify Uncertainties Between
U.S. and China (2016),” accessed October 30, 2022,
https://www.nytimes.com/2016/12/19/world/asia/china-donald-trump-power.html
Dr. Yu Zhang has a background in fine arts and design. In her Ph.D. research she investigates the theory and
artistic practice of interactive technologies for public, large-scale installations. Over the past years, she
has designed and researched interactive systems that respond to everyday phenomena, environmental concerns,
child-system interaction, online collaboration platforms, and uncertainty in data visualization. Yu has
participated in various international Art Residencies of the last years, and her work has been exhibited at
galleries, museums, and festivals world-wide. Besides, Yu’s teaching experience covers a broad range
from traditional classrooms and workshops to designled project-based learning activities. Her book
“Coding Art,” co-authored with Mathias Funk, was published by Apress/Springer in 2021.
https://yuzhang.nl/
Centre for Art as Forum, Department of Arts and Cultural Studies, University of Copenhagen
Copenhagen, Denmark tanyaravnag@gmail.com
Abstract
The world of art has always been occupied with art’s temporal relations to the future. In the current
artistic landscape, we see a wealth of exhibition themes and titles concerned with ‘the future’ in
responses to the dominant narratives of a contemporary technological world driven by algorithmic systems and
prediction models. This momentous future-orientation is my cue to rethink art’s relation to the future, by
zooming in on its temporal modes of existence. With a take-off in the notion of art as “time-based,”
as conceptually based in the time, duration, and/or the function of a medium and the experience it mediates, I
propose a different intratemporal mode of existence for art. This concerns how art co-exists with,
evolves through, and co-produces temporal relations in between humans and technology. This proposal of an
intratemporal perspective on art might contribute to further investigations into art epistemologies in which art
becomes a part of larger narratives in which human beings and communities co-evolve—and have always
co-evolved—with technics. It might offer inroads to study art on its new paths of exploration in
collaboration with science and technology and when art is occupied with the very making of the future through
participation in innovation projects.
My temporal investigation in this paper takes off in the catalog Alchemists of the Future published
for the Ars Electronica Future Lab’s 25 years anniversary in 2021. In the concluding chapter,
“Perspectives,” we can read about how the Future Lab’s activities of visionary prototypes and
innovative collaborations between art and science in 1996 were initiated to contribute with future narratives to
address urgently needed paradigm changes.¹ The visions expressed in the Future Lab
catalog, about art’s involvement in our greater societal narratives of technological change, echo the
bringing together of art, science, and technology with the conception of the New York-based organization
EAT—Experiments in Art and Technology—in 1967, which was founded by engineers Billy Klüver and
Fred Waldhauer and artists Robert Rauschenberg and Robert Whitman. The visions of creative and experimental
research processes between artists and engineers for bringing artists closer to the materials of technology and
more in touch with the forces shaping contemporary society, the use of projection and new communications
technology to achieve this, which entailed the exploration of new roles for art in the changing
‘technological environment’ of the late 1960s/early 1970s, and the migration of these practices from
an art to non-art contexts,² altogether paved an explorational path for art’s evolvement through
changing relations to the future.
Today, as we find when searching through numerous recent titles of exhibitions, knowledge forums, and events of
art, (1) the future orientation has saturated the broader field of art. This future-orientation
should interest us as more than a thematic trend.
The legacy of E.A.T., and the catalyzation of ideas of collaboration between art, science, and technology
through the Future Lab, among many more initiatives, informs a fast-growing discourse in art whereby the art is
treated, funded, and appropriated as a catalyst for change. For example, when art migrates into cultures and
contexts of technological innovation; when artists are invited into residencies, technology and science labs of
corporate technology companies; or, where art becomes a protagonist in major creative funding schemes and
innovation programs and is granted support as a catalyst for, for example, industrial innovation, urban
development, or human rights. For one example amongst many, the call “Art-driven use experiments and
design” under the Horizon Europe Framework, which explicitly allocates a strategic role for art in
technological innovation culture. These movements in art, whereby art has gained new roles in strategic projects
of future-oriented and future-shaping technological innovation, require new approaches to grasp and assess
art’s modes of existence, which I propose that we understand through its relations to the future as an
epistemological and methodological compass.
My inquiry is guided by the following line of questions: Why, in the context of our contemporary technological
environment, is the orientation towards the future in art so momentous? What characterizes art’s relation
to the future in our current technological environment? If the occupation with the future in art concerns a
temporal orientation towards how everyday lives, cultures and societies will or might evolve with technology,
then how does art participate in the temporal processes that will bring us there? Why does art’s relation
to the future matter to the roles that art pursues and gains within technological innovation—as a locus
for human symbiotic imagination (about the future) and our technocultural making of it?
My overall suggestion is that we need to grant more attention to art’s temporal modes of existence as
simultaneously a matter of object functionality and environment, human and intersubjective experience,
technocultural context, and cultural evolution. With a point of departure in the conception of art as
“time-based,” I engage an alternative, intratemporal mode of existence for art, with which
I understand art to be a part of a larger temporal complex: art is not based in time but existing through
intratemporal infrastructures and relations with its contemporary technological environment, which in our
current age is characterized by and evolving through data-driven algorithmic processes. I unfold the
intratemporal perspective on art through three temporal dimensions—object temporality, worldly
temporality, and deep temporality—that relate art to epistemologies on how human experience changes with
technological culture.
Art and temporality - beyond “time-based” media
When art is described and categorized in a temporal perspective, it is commonly referred to as
“time-based.” My claim in what follows is, however, that this temporal conception of art and the
epistemological framework that it engages is insufficient to grasp art’s behavioural modes of existence
and interdisciplinary evolvement today.
The conception of art as “time-based” is broadly used by museums with reference to artworks that
rely on technology, such as video, film, audio, slide, installation artworks, as well as artworks that function
only for the duration of their time on display, like computer-based and mechanical works of art. The conception
of time-based art ties time to the expressive and functional qualities of the medium. Time-based art is
conceptually rooted in “time-based media,” a term coined by museum conservators for durational works
of art that unfold over a period of time. It is used widely by art institutions to describe art that is
‘dependent on technology and has a durational dimension’ (Tate), that ‘unfold to the viewer
over time’ (Guggenheim), and that are ‘dependent on time, duration, or function’ (National
Gallery of Australia). Time-based media has a run-time enabled by the form or medium that limits and contains
the experience. The medium enables the inscription of the spectator in different experiences of time. By looking
at art as “time- based,” we focus on how the art facilitates meetings between different durations.
For example, between the durations of human experience and the durations of a rationalized society. This
understanding is fueled by a broad theoretical interest in temporary multiplicity in the writings of among
others Henri Bergson, Alfred North Whitehead, Gilles Deleuze, Michel Foucault, and Michel de Certeau.
Christine Ross’ examination on art and temporality in The Past Is The Present, It’s The Future
Too exemplifies this understanding of art as facilitating meetings between durations. She describes
various ‘durational’ temporal strategies in art as aesthetic counter reactions to the forwardness of
the modern era. These are temporal strategies of, for example, endlessness, ephemerality, repetition, real-time,
contingency, randomness, slowmotion, condensation, acceleration, extension, abbreviation, speeding up,
hesitation, disruption, fissuration, extendibility, and interminability—all temporal strategies for
suspending linear conceptions of time that confirm one universal temporal logic.³ Such temporal strategies
of suspension evoke a tendency emerging in contemporary art of the 1960s, which is described by Pamela M. Lee in
terms of “chronophobia”—a sense of unease or maybe even rebellion in art against temporal
societal narratives that dominated during the middle of the 20th century and which translated into a critical
consciousness in artistic expressions of performance, conceptual art, sound art, installation practices and land
art.⁴ The art of the 1960s that both Ross and Lee write about reacted against a relation between
temporality and historicity, namely one dominant narrative about technological progress that characterizes
Western modernity, which celebrated technological transformation, automatization, acceleration and
standardization. The dominant narrative reflects a universal conception of time as linear, structured around
past, present and future, and organized based on classical physics’ ideas about absolute mathematical time
and ground principles of natural science about relativity. In this narrative, time and space are compressed by
technological and mechanical processes—what David Harvey has named “time-space compression”
which refers to how global communications technologies and information economy compress barriers and distances,
which is a function of late capitalism.⁵ This global, temporal narrative is structured around a singular
temporal scale characterized by rules of regulation, discipline, speed, effectivity, immediacy and
progression—as Jonathan Crary describes in the book 24/7.⁶
The time-based conception rests on a philosophical notion rooted in the ideas of Plato and a substantivist and
absolutist conception of time, treating time as an empty container with rules and logics, that is, temporal
rules and logics that are ready for art to critically engage with. We recognize this conception of contained
time when art is accounted for as an aesthetic, conceptual, critical manifestation capable of presenting and
representing alternative temporal modes to those driving capitalism by which to inscribe people into different
experiences of time. This temporal containment, however, delimits art’s relation to the future as
representational or reflective material that eventually becomes confirmative of the future narrative that it
speaks to.
I would like to propose a different temporal condition for art. Because, although art is situated in a specific
temporal slot and has a particular duration, and although it might depend on the phone or a mobile device that
enables specific temporal qualities of the experience, the work is not delimited to a temporal capsule. It is
not delimited to exist “based in time,” as if in a form of a temporal container that we can
individually step into for a direct experience with represented image or concept. The time-based conception
relies on a direct experience between the human and the artwork. This does not correspond to the ways in which
we experience and exist with temporalities through the ways in which most of us engage with technology today.
Nor does the time-based conception account for the human-perceptual and technocultural effects of these temporal
experiences. Time is articulated in technical systems but only in connection with human engagement with
technics, as we learned from the writings of Gilbert Simondon.⁷
In the following, I will propose the contours of an alternative temporal conception of art to that of
time-based; one that considers an intratemporal mode of art’s existence. This involves the conception of
time as something that the art is relationally entangled with, through which it evolves, and which the art
contributes to generating.
Art’s Intratemporal Mode of Existence
Art’s intratemporal relation to the future today bears traces of future-orientations in art of the past.
Along with the ongoing critical discourse in art continuing the critical occupation with the forwardness of the
modern era that Ross locates in art of the 1960s, we recognize a trajectory from the futurist art movement of
the early twentieth century that sought to capture in art the dynamism, energy and movement of the modern world
and modern life. Preceding future-oriented movements unfolded in effect and response to a technological
environment before the internet, social media networks, and data-driven distributions and accumulation of
registrations of our behavior. Today, however, the technological environment conditions a different
intersubjective condition than that of the 1960s.
An intratemporal perspective on art might immediately evoke Martin Heidegger’s understanding of phenomena
and objects in terms of temporal relations rather than substance, in Being and Time. (2)
⁸ Heidegger contrasts intratemporality with authentic temporality, seeing intratemporality as an
existential structure to Dasein determined by calculation and measuring instruments. My use of the term
“intratemporality” takes a different reference, in Yuk Hui’s connection of intersubjectivity
(subject-context relation) and interobjectivity (object-milieu-relation) in the term. In Hui’s theory,
intratemporality is a dimension of changing f temporal relations between objects and neurosensory evolution that
happens through our networked and synchronous co-evolvement. ⁹ In my adaptation of this understanding of
intratemporality to this inquiry on art, I consider art as temporally related with temporalities of technologies
and technological cultures well beyond the medium, the art experience, and the discourse of the art environment.
In the following, I will draw some perspectives on how intersubjectivity, as a matter of temporal relations, is
conditioned by the temporalities of digital objects and worldly connectivity as well as by deep temporalities of
our cognitive and cultural heritage from technocultural pasts. These temporal dimensions combine in art’s
intratemporal mode of existence and tie art to the concept of the future in new ways.
Object temporality: In Hui’s account of the conditions of digital objects and extension on
Heidegger’s notion of intratemporality in this regard, he notes how data-driven temporal processes mediate
between intersubjective and inter-objective relations and influence temporal experiences in our everyday lives.
The ways in which things are quickly shared, behavior and ideas are quickly adopted, and experiences are
synchronized, effects an organization of consciousness about how things are temporally related to each other.
This reorganization of consciousness with object temporality is what N. Katherine Hayles addresses in How We
Think: Digital Media and Contemporary Technogenesis.¹⁰ Hayles writes about how the media
interface (e.g., the screen), for example, might seem like it correlates directly to human modes of sensory
experience and cognitive processing while it only indirectly correlates to these modes of experience, since it
involves technical operations to which we lack a direct access. This is because different time scales of human
cognition and machine cognition intermesh. As computational processes occur at time frames that are below the
threshold constitutive of human perceptual experience and introduce levels of operationality that impact our
experience but do not have any perceptual correlate, we are not conscious of their consequences to our actions.
The mutual interference between temporalities of machinic systems and human-temporal functioning of
consciousness means that our creation of (abstractions, forms, content, systems, meanings) is not rooted in a
direct human relation with what we create but depends on unconscious processes. This unconscious aspect of
perception with object temporality connects us to worldly temporality.
Worldly temporality: Global effects—from economical dynamics at a macro level to machinic
operations at a micro level—reach us through object temporality. The unconscious cognitive processes at
work in our engagement with object temporality not only concern an engagement with digital objects (in and
beyond art). They condition intersubjective experience as object temporality relates to the temporalities of
networked media technologies, which is entangled with human experience. As Mark Hansen notes in Feedforward:
On The Future Of Twenty-First-Century Media, with digital media, the external world has become a part of
individual experience, while experience has become externalized and environmentalized in contemporary forms of
mediation.¹¹ While we physically exist in the phenomenal world, our thinking, behavior, and the
effects of our actions are also of a worldly context, conditioned by mediating factors of different
temporalities affected by environmental and global connectivity, and experienced across timespaces.
What this meshing of human and machine temporality results in are operational processes that function as a kind
of technical “memory,” which becomes a cultural support structure, and which affects how perception
and intersubjective imagination are at work. This technical support structure evolves from a long process of
evolutionary adaptation of technical tendencies and their logics, whereby art is intratemporally entangled with
a sense of deep temporality.
Deep temporality: Art’s experiences can amplify and resonate through volumes of people, connect
us to our ancient past and memories of cultural rhythms, rituals, and practices, and throw us into
uncertain futures. An intratemporal dimension of deep temporality links art to cultural patterns, which have
shaped the ways in which we use and develop tools and technology since our human origin and the perceptual
habits we have evolved and enact when experiencing something, including art. Collective memories, cultural
programs and imaginations have been transmitted via habit and repetition through communities and historical
epochs. With reference to technoanthropological ideas from the philosophical writings on human technogenesis of
Bernard Stiegler¹² based on the anthropology of André Leroi-Gourhan that roots human
co-evolvement with technics in the origin of human civilizations,¹³ we can consider how art has a part
in the shaping of cultural memory, symbols and rituals that we have adapted from ancient pasts. These have
formed through civilizations, cultures and generations to manifest in the cultural codes, meanings and logics we
navigate by today. These cultural adaptations of technocultural aesthetics and behavior inform how human
cognition meets machinic operations today.
With this intratemporal dimension of deep temporality, I wish to emphasize a technicity in the art as having a
function with regards to our cultural evolution with technics. Art, as a human aesthetic expression, has evolved
with evolutional adaptations that carry the past into the present—and entwine with the
future—through technological tools and the cultural and cognitive memory structures they engage. From
studies on ancient human pasts, we know that art, and the technicity with which it operates, in the ancient
shapes of rituals, ornamentation, craftwork traces of human gatherings, and more, has taken on various roles as
a cultural transmitter and as an aesthetic mechanism of societal organization. For example, as a kind of
mediator of societal imaginaries; a vehicle for intelligence, memory, language, forms of expression and pattern
recognition to travel through generations; as both depiction and facilitation of rituals (practical, cultural
and spiritual); as a connector of human beings to their past and origin and a basis for collective consciousness
and emotional intelligence; as a connector of humans to the materials and environments of our world and its
ecosystems; as embodying conceptions and philosophies of science as a foundation for furthering civilizations,
among many others. These are observations from my research on various intersubjective functions art has had in
ancient societies.¹⁴
Intratemporality concerns temporalities that are within us, among us, beyond us and preceding us—and
which entangle in our tenement towards the future. The intratemporal mode of art’s existence therefore
cues a re-examination of art’s relation to the future.
Art’s intratemporal relation to the future
The current future orientation in art is not either confirming or resisting a utopian desire. What should
interest us are also not the future destinations that art offers or is used to test, project, or speculate upon,
which are ideas that inform scenario-based design and conceptual attention to "possible futures." It is also not
the critical comment on future-driven regimes. Future-oriented art of today does more than make room for
reimagining the future.
My proposal here is that art’s intratemporal relation to the future concerns how art and its experiences
are entangled with temporalities that relate us to our everyday engagement with (digital) technologies, with the
flows and dynamics of worldly (data) processes, and which engage cultural adaptations and intersubjective
evolvement through intuitions and perceptions that precede our experience today. This perspective ties
art’s relation to the future to contexts of technological cultures beyond that of Western rationalization.
The intratemporal perspective on art concerns what kinds of temporalities the art engages and connects in our
bodies, objects and surroundings, and in which ways (by the use of which techniques and aesthetic means). It
concerns how art intervenes in our experience of those temporalities. This perspective writes art into a larger
narrative in which human experience is changing with technology and in which art has always played a role in the
ways in which human beings have co-evolved with technics.
This reconception of time-based art can help us to grasp the new routes and roles art pursues through temporal
engagements with technological innovation culture. When art collaborates with science and technology in the
domain of innovation, it not only envisions, problematizes, or proposes but also co-produces our
futures. This involves a change in perspective, from how art represents and responds to the future, to how art
has a constitutive relationship to the future. This is because art engages with human intuitions, desires, and
aspirations from where our futures emerge. Art becomes a part of larger intratemporal processes of human
co-existence and co-evolvement with technology. This calls for further examination of futurity in art, how the
art’s techniques and experience is temporally entangled with future-driven systems and processes of human
co-evolvement with technics.
(1) Some of the many recent future-oriented exhibition contexts. I’ve come across in my
research, which exhaust and understate my argument that the attention to and conceptualization of the future in
art is momentous: Possibles (ISEA2022); Futures Implied (Media Architecture Biennale 2020); Writing the History
of the Future (ZKM – Center for Art and Media Karlsruhe 2019); Futures (Smithsonian 2022); Future and the
Arts: AI, Robotics, Cities, Life - How Humanity Will Live Tomorrow (Mori Art Museum 2019); The Future Starts
Here (V&A South Kensington 2018); Possible Spaces (Danish Architecture Center 2018); Future Shock (180
Studios 2022); WHO Futures Art Exhibition: Envisioning the Future of Health in 2050 (World Health Organization
2022); Future World (Art Science Museum Marina Sands 2022); Future U (RMIT Gallery 2021); Hope for the future
& meaning of life (Kawaguchi Art Museum 2021); Sampling The Future (National Gallery of Victoria 2022);
Edible Futures (The Dutch Institute of Food & Design 2022); Future Food Today (Space10 Gallery 2022); The
Future We Create (Art Works for Change 2022); Remembering the Future: 100 Years of Inspiring Art (Heard Museum
2022); Future Perfect (worldwide 20192022); Future Retrieval: Close Parallel (Cincinnatti Art Museum 2021);
TECH/KNOW/FUTUREU/ From Slang to Structure (Montclair State University 2021); Past Present Futures: Notions of
Time in Twentieth-Century Art (Blanton Museum of Art 2001); Future Is Today (Al-Tiba9 Global 2020);
Decriminalised Futures (Institute of Contemporary Arts 2022); Futureritual (Institute of Contemporary Arts
2022); The Future of Now:; Contemporary Art in Our Unstable World (Emmanuel Art Gallery 2022); Designs for
Different Futures (Philadelphia Museum of Art 2020); The Future States (Latvian National Museum of Art 2018);
Remember the Future Orleans House Gallery 2021); Designs for Different Futures (Walker Art Museum 2021).
(2) Heidegger’s attention to intratemporality concerns an existential structure of Dasein that
is inauthentic and measured by technological instruments and by calculation. In Heidegger’s optics,
intratemporality denotes an inescapable horizon for Western history of being.
References
1 Horst Hörtner, Roland Haring, Hideaki Ogawa, Andreas Hirsch, Alchemists of the Future: Ars
Electronica Futurelab: The First 25 Years and Beyond, Hatje Cantz, 2022.
2 Christophe Leclercq, “The Legacy of Experiments in Art and Technology (E.A.T.): An Environmental
Aesthetics,” paper presentation at ISEA2011: 17th International Symposium on Electronic Art, 2011.
3 Christine Ross, The Past is the Present; it's the Future Too: The Temporal Turn in Contemporary Art,
New York, Continuum, 2012.
4 Pamela M. Lee, Chronophobia: On Time in the Art of the 1960s, Cambridge, The MIT Press, 2006.
5 David Harvey, The Condition of Postmodernity: An Enquiry into the Origins of Cultural Change,
Hoboken, Wiley- Blackwell, 1991.
6 Jonathan Crary, 24/7, London and New York, Verso, 2014.
8 Martin Heidegger, Being and Time, New York: State University of New York Press, 2010 [1927].
9 Yuk Hui, On the Existence of Digital Objects, Minneapolos, University of Minnesota Press, 2015.
10 N. Katherine Hayles in How We Think: Digital Media and Contemporary Technogenesis.
11 Mark. B. N. Hansen, Feed Forward: On The Future Of Twenty-First-Century Media, Chicago,
University of Chicago Press, 2015.
12 Bernard Stiegler, Technics and Time, 1: The Fault of Epimetheus, Stanford, Stanford University
Press, 1998.
13 André Leroi-Gourhan, Gesture and Speech. Translated by Anna Bostock, Berger. Cambridge,
Mass., MIT Press, 1993.
13 André Leroi-Gourhan, Evolution et techniques I: L'homme et la matiere (Man and Matter),
Paris, Albin Michel, 1943.
14 Tanya Ravn Ag, “Media Art in the Hybrid City – Why?” keynote at Art in the Smart City
symposium, hosted by Art Republic at Stavanger Art Museum, Norway, November 2018.
Bibliography
Jonathan Crary, 24/7, London, New York: Verso, 2014. Hansen, Mark. B. N. Feed Forward: On The Future
Of TwentyFirst-Century Media, Chicago, University of Chicago Press, 2015.
N. Katherine Hayles, How We Think: Digital Media and Contemporary Technogenesis, Chicago, University
of Chicago Press, 2012.
Martin Heidegger, Being and Time, New York, State University of New York Press, 2010 [1927].
Horst Hörtner, Roland Haring, Hideaki Ogawa, Andreas Hirsch, Alchemists of the Future: Ars
Electronica Futurelab: The First 25 Years and Beyond, Hatje Cantz, 2022.
David Harvey, The Condition of Postmodernity: An Enquiry into the Origins of Cultural Change,
Hoboken, Wiley-Blackwell, 1991.
Yuk Hui, On the Existence of Digital Objects. Minneapolos, University of Minnesota Press, 2015.
Christophe Leclercq, “The Legacy of Experiments in Art and Technology (E.A.T.): An Environmental
Aesthetics,” Paper presentation at ISEA2011: 17th International Symposium on Electronic Art, 2011.
Pamela M. Lee, Chronophobia: On Time in the Art of the 1960s, Cambridge, The MIT Press, 2006.
André Leroi-Gourhan, Gesture and Speech, Translated by Anna Bostock Berger, Cambridge,
Mass., MIT Press, 1993.
Christine Ross, The Past is the Present; it's the Future Too: The Temporal Turn in Contemporary
Art, New York, Continuum, 2012.
Bernard Stiegler, Technics and Time, 1: The Fault of Epimetheus, Stanford, Stanford University
Press, 1998.
Author Biography
Tanya Ravn Ag is a curator and scholar on art, technology and digital culture, focused on how art changes
with contemporary technogenesis and algorithmic culture. She is the editor of Digital
Dynamics in Nordic Contemporary Art (Intellect, 2019) and co-editor of What Urban Media Art Can
Do – Why, When, Where, and How? (av edition, 2016) based on which she co-founded the globally
networked Urban Media Art Academy in 2017. Her curatorial engagements with urban, media-based art include the
Screen City Biennial 2017 in Stavanger, the SP Urban Digital Festival in São Paulo in 2013 and 2014, and
Nordic Outbreak presented in New York City and across the Nordic region by the Streaming Museum in 2013-2014.
She has served on the ISEA Board since June 2023 after chairing the IIAC (ISEA International Advisory
Committee) since 2020.
Acknowledgements
The research informing this paper has been supported by the Carlsberg Foundation’s Reintegration Grant
for the project Art of our Times: Deep Temporalities of Art and Algorithmic Culture at the Department of Arts
and Cultural Studies, University of Copenhagen.
For a more symbiotic co-individuation with our technological avatars: how to go, with the Sciences and the
Arts, beyond hybridizations?
EtienneArmandAmato & EtiennePéreny
Laboratoire DICEN-IDF, Université Gustave Eiffel & Université Paris 8
Paris, France etienne-armand.amato@univ-eiffel.fr ; pereny@gmail.com
Abstract
With interactive computing, the metaphorical use of biological notions of hybridization and symbiosis has
become widespread. They refer to the possibilities of mixing as well as to the conditions of emergence of
relationships ranging from mutual benefit to instrumentalization between technologies and humans. In order to
better understand the relevance of such analogies with the living, this article draws on scientific and artistic
research concerning the interactive avatar. These seem particularly instructive on our relationship to
technology because this virtual being hybridises the living and the artificial, while constituting the key and
the condition of access to digital spaces, to co-evolve with other users, themselves avatarised, or other
autonomised agents. The article distinguishes between "cyber" avatars populating persistent universes, video
games, virtual realities, i.e., cybermedia environments of simulation and interaction. The "hyper" avatars are
those of the Web, online services, 2.0 platforms or socio-numerical networks that are juxtaposed in networked
informational and documentary hypermedia. From then on, the challenge is to reinvest the sense and
responsibility of their potential for augmentation or simulation, in particular by promoting cooperative
interactions through science, art and technology, which are themselves by their very nature synergistic with
each other.
Introduction: symbiotic design at the origin of interactive computing
Since the great convergence of telecommunications, computer science and miniaturized electronics, the expansion
of networked digital environments has been accelerating, which never ceases to raise questions, so much so that
it guides the civilizational changes underway. Digital art has been able to contribute to these developments
from the outset by appropriating the different waves of inventions, software and tools: network art, code art,
interactivity, Web art and Game Art, etc. However, due to a lack of recognition, these influences have remained
underground and unofficial, as industrial and commercial creations dominate. After decades of intellectual and
artistic, academic and cultural engagement, initiatives such as ISEA or numerous Arts & Sciences projects,
have made common the idea that artists contribute to create an alternative technological culture to those
stimulated by consumption, market and entertainment, in order to produce new practices and regenerative
imaginaries. By choosing the theme of "Symbiosis," ISEA 2023¹ sends a strong signal to the communities in
charge of digital cultures and electronic arts, a kind of encouragement to favour a transformation of our
relationship with the digital, to move towards more responsibility and reciprocal benefits, while comfortable
consensuses and evidences are cracking. However, the matter is not so simple. Admittedly, the symbiotic approach
has the advantage of offering a qualitative leap forward and a step back, integrating the latest scientific
knowledge on cooperative interdependencies, at all scales of life and at levels previously invisible, and just
which have been revealed by technical images.² However, this notion does not have the same cultural and
political effects depending on whether we take it from an apparently neutral angle; with the Anglo-Saxon
meaning, which considers as symbiotic all forms of interactions between species, including the most aggressive
and predatory, parasitic and deadly ones; or whether we take it in its French and European meaning, which
favours synergistic, mutualist or commensal symbioses, offering mutual or at least unilateral benefits.
But in a facetious and creative way, let us open our remarks with an enigma offered to the reader of this
article, before he or she dives into the heart of digital systems and environments by following the path of
co-evolution made possible by avatars.
From whom does this excerpt come? "The potential of the human-machine symbiosis is easily visible in the arts,
where computing technologies have enabled the creation of previously unrealizable forms of expression. Computing
technology has empowered a new legion of artists working in mediums such as immersive and augmented reality
games, animated feature films, and music composition and performance (...) In these areas, we are beginning to
see humans and machines as complete partners in artistic creation."³ It is striking that an eminent
researcher in computer science, having joined the most demanding military programs of the DARPA in terms of
success, recognizes artistic activity as one of the best contexts for the emergence of new partnerships for the
benefit of an original creation. To agree with this, it has been widely shown that artists, through their close
contact with techniques and their potentialities, in their exploration of the logic and capacities implemented
in their tools, which have become software with computers, constitute an avant-garde that clears out the
possible and opens up new appropriations. If the works in art history have shown this, it is up to thinkers more
in tune with the arts and technologies of the image, such as the Frenchman Edmond Couchot, to show it in the
course of his numerous works.
But beyond the man-computer symbiosis and the place of the Arts in its advances, let's go back for a moment to
the origin and the evolutions of this new paradigm of microbiological symbiosis that appeared following the work
of Lynn Margulis⁴ concerning eukaryotes in the 1960s, co-author of the "Gaia Hypothesis" with James
Lovelock.⁵, ⁶ This research eventually produced major scientific advances concerning
plants, forest and soil, but also the intestinal microbiota in humans, emphasizing the symbiotic interdependence
of living things, both inside and out their bodies.
For our purposes and references concerning the coevolution of man and his technique, we will retain the
teachings of Leroi-Gourhan⁷ and Gilbert Simondon⁸. The former analyzed hominization through "gesture
and speech," techniques that preform both our morphology and our cognition, the latter insists on the
"individuation process" and its "associated environment," which organize the individual and collective mode of
existence of man, as well as that of technical objects.
But the difficulty remains in finely articulating the human and the artificial, in respecting proven frameworks
of thought while going beyond them, updating them, making their complexity a little more "simplex," following
the teachings of Alain Berthoz⁹ supported by the phenomenology of Merleau-Ponty¹⁰ and the
enaction of Varela.¹¹ By proposing an emerging concept, that of the symbiotic co-individuation of man
and his avatar, that is to say an individuation—not to be confused with individualization—which is
made joint by their common process of co-evolution, it is indeed a question of situating its context and
exposing the premises of a vast site of studies and creations which remains to be undertaken through an approach
seeking to create a new synergy between the Sciences and the Arts.
I) Symbiosis and hybridization: biological metaphors in vogue in the computer and digital field, from the
beginning of computing to the present day.
a) A novel concept of a possible partnership between humans and computers
Anyone interested in the links between symbiosis and computer science will identify the inaugural 1960 article
"Man Computer symbiosis"¹² written by one of the acknowledged founders of modern interactive
computing, J.C.R. Licklider. Both a psychologist and a computer scientist, he developed a way of thinking about
the partnership with the computer, and later helped coordinate, within military programs, the transition from
the first computer networks to the Internet. Such an early reference to and mobilization of a concept from the
life sciences, and moreover in connection with the military-industrial complex, may surprise and even arouse
suspicion. A few points need to be made here. Firstly, it bears witness to a context of technical emergence,
that of computers finally operating in real time and becoming sufficiently programmable and accessible thanks to
their sensitive interfaces (screen, optical pen, and keyboard) to envisage a genuine cooperative interaction
that would bring together the best of the two worlds, living and artificial. Secondly, it also highlights a
typically cybernetic vision of putting human and computer in parallel, becoming colleagues and partners, who,
far from being reduced to each other, assert their specificities and differences, but pull their strengths and
weaknesses. Thus, by associating and communicating according to the right methods, they will constitute a tandem
achieving what no one alone would manage to do. Finally, this positive, even idealistic, vision must be
understood as a "political move" opening up another perspective at the heart of the technocratic system to
confront the promoters of the all-machine approach and of a progress leading eventually to the advent of
artificial intelligences superior to humans. However, several controversies could immediately invalidate such an
idea and send it back to the side of naive approximations. In fact, symbiosis is observed in the field of the
living, not the artificial. Applying it to computers would mean granting this class of logical machines the
status of a species endowed with certain properties of the living, such as evolution, adaptation and
reproduction. But in our western conception of the world, practising a “naturalistic ontology” as
French anthropologist Philippe Descola call it,¹³ the highest faculties are the prerogative of
humanity. If we want to continue the analysis with Gilbert Simondon,¹⁴ in the absence of integration
with culture, technology is conceived as separate and external to man, who at best uses it, while attributing to
it his own misdeeds. When in fact, technology, this human creation, has the role of mediating between nature and
culture. However, the metaphorical use of symbiosis has a few merits: better respect for each stakeholder in the
tandem; understanding of their interdependencies; and the establishment of a more balanced cognitive and
sensitive dynamic than the sole technical and strictly functional approach, that of a mechanically extended
human, "mechanically extended man" or "humanly extended machines" (quoted in ¹²). With this approach,
enlightened by the living, a perspective is developed that is neither completely anthropocentric nor
technocentric. It will indeed influence a whole community of inventors and engineers to seek mutually beneficial
relationships. This begins in the field of complex problems, extends to ergonomic interfaces, in connection with
the interfaces and work methods invented by Engelbart ¹⁵ with his "mother of all demos,"¹⁶
and will continue with the first technological research involving computer networks, at the origins of the
premises of the Internet.
b) Instrumentalization and hybridization: what association between the living and the
artificial?
It is clear that more than 60 years later, this cooperative conception has not really governed the deployment
of the "digital." The triumphant regime is that of a generalized and excessive instrumentalization of some by
others. It is also based on older, naturalistic metaphors that have become dogma: the struggle for survival,
selection, competition and domination, all stimulated by a misunderstood and ideologized Darwinism of
Anglo-Saxon and capitalist origin that fetishizes the market and competition. This cultural overdetermination
tends to reduce both humans and computers to a few utilitarian functions, rather than mutually emancipating
them, or just amplifying them (Licklider¹²) or increasing them (Engelbart¹⁵). For example,
workers turn out to be replaceable as soon as a task can be automated, while the computer itself is devalued,
becomes obsolete, as soon as a new, more powerful and "modern" generation arrives on the market. In other words,
technological "progress" is based on the division and specialization of tasks to the detriment of others: models
implementing interests, reconducting the existing and thus rigidifying socio-technical systems. (1) Following an
instinct of conservation hiding under the imperative of innovation, it is essentially a matter of optimizing and
stabilizing rather than solving and changing; of intensifying short-term productivity rather than changing the
level of symbolic integration and moving towards "a better ecology of the mind."¹⁷
Another notion, also imported from biology, is proliferating in the field of new info-communication
technologies, and continues to be very much in vogue on the cultural level.
It is the notion of hybridization, synonymous, depending on the context, with convergence, fusion, mixing,
blending, and association, concerning both the technologies themselves and the interpenetration or interweaving
of heterogeneous dimensions: the human and the artificial, or the real and the informational. It carries an
enthusiastic conception of the possible crossings between various technical lineages as well as of what results
from them, their offspring. It probably has a positive echo in our anthropological collective unconscious
because of the reproductive experiments dating back to the Neolithic period, carried out by our species on
plants and animals to make them edible, domesticate them, use them, and thus develop ever more new hybrids to
improve our living conditions.
As a result, there is a tendency to explore and implement all possible configurations through clever and
inspiring mixtures, according to a principle of free experimentation that does not take into account the stakes
and consequences from the outset, but rather evaluates them, possibly and only after the fact. Believing himself
to be the "master and possessor" of his technique (2), humans let themselves go to "hubris", to this excess of
power, to this attractive vertigo that is generated by a too continuous success. The conquest of all
possibilities guides our collective strategy. Most forms of exploitation, first and foremost of the living
through technology, are justified, pending the next techno-solution that will correct the effects. In this vein,
transhumanist conceptions assume that humanity must continue to mutate by itself thanks to Technology and
Science, assisted by an Art that would become its official foil. But do they realize that the solutions obtained
by a definitive hybridization (the cyborg) or by deep environmental modifications (the geoengineering) remain
prisoner of the same paradigm that caused the problems it tries to solve? Although some forms of hybridization
may be viable and desirable, with lines that improve through heterosis, which is the alliance of the best of two
species, we will try to think according to a higher general principle, that of an evaluation and regulation of a
symbiotic nature, in the French sense of the term, and therefore mutualistic, which could frame, anticipate and
therefore limit any possible deleterious effects.
II) "Cyber" and "hyper" avatarizations: two main ways of entering the digital world
a) The cybernetic avatar: a body situated to experience virtual technologies
The fundamental phenomenon that networked microcomputing has made possible, but which was not immediately
grasped or understood as such, concerns the "existential" entry of the human into the machine and into the heart
of its software. If this is realized in many ways, the avatar is the most widespread and concrete emblematic
figure, especially the one that networked video games have made known to us. Indeed, since its
baptism,¹⁸ the avatar has made it possible to inhabit an immersive environment with other human
persons and programmed entities, and even to cooperate with artificial beings or with other "semi-living"
beings, which have yet to be better qualified and understood, both scientifically and artistically.
The original symbiotic vision already mentioned had stopped at the face-to-face meeting of the human being and
the computer, forming an interdependent and winning team. With the integration of networked computers, an
informational, quasi-corporeal vector was needed, if only to manifest itself to other distant people, caught in
the same technological constraints. In a shared place, benefiting from a system of exchanges and actions, the
human invests his avatar to interact in a more complex way than with conversational interactivity based on
dialogue and instruction. Collectively, the first interactivity (reactive, sequential, asymmetrical) is overcome
and gives rise to the second interactivity (proactive, simultaneous, mutual). The first occurrences of avatars
(3), retrospectively identified by semiologists, computer scientists and other analysts,²⁰ result
from the marginal appropriations of the very first transistor computers by new adepts of creative subcultures,
looking for a way to divert and explore the capacities of new electronumerical machines, at a more human size
and finally becoming more accessible.
Since then, despite their improvement and growing sophistication, playable avatars, those of the simulated
worlds of video games, can still be analyzed in terms of "living-artificial complexes." Indeed, the technical
part, with the programmed functioning, and the living part, with the human behaviours adjusting in real time to
the action in progress, are mixed in the same virtual being. In this respect, avatars are similar to
techno-human or bio-technical "hybrids," depending on the point of view adopted. In this respect, field studies
in the human and social sciences show that online participants who use avatars know how to consider any "player
character" they encounter in a persistent universe from both sides. They evaluate both their operational
capacities and the living intentionality that drives them. In addition, the hybrid avatar has several remarkable
characteristics that relate to its mode of operation and attendance. On the one hand, it results from a "soft"
hybridization, because it is temporary and reversible (unlike the cyborg), which preserves the integrity of both
halves, the human and the artificial. On the other hand, this humanized technical hybrid is unified and
maintained by two parallel processes that can be analyzed through the symbiotic metaphor. On one side of the
screen, in the ordinary world, there is a human-machine-computer coupling which, for the time being, joins the
Lickliderian vision of a dynamic symbiosis. Except that here, it is even more marked because it is based on a
total interdependence: the human needs a formalized avatar to evolve in a simulated world while, without its
player, the avatar is only an empty shell, identifiable by its looped attitudes, as if waiting for a soul to
re-function. Both interface and vehicle, it is an entity that articulates the real and the virtual. On the other
side of the screen, this time in the simulated world, the avatar must compose, negotiate and build its
partnerships, if possible symbiotic to last and grow, both with its fellow avatarized, and with other
inhabitants fully programmed and more or less autonomous. Its survival and development often depend on
relationships of co-construction and cooperation, even in a competitive context. By associating as a group, by
combining their talents, by carrying out collective strategies, while benefiting from the support of artificial
beings, the human beings hybridized by the avatar adapt to a place lived in common. They develop a powerful
intersubjectivity, a feeling of living together that is confirmed by their decisions and actions. Unified by a
particular spatio-temporality and physicality, this virtual but concrete place mobilizes their situated
cognition; that is, sufficiently spatialized to transform the environment and its components or actants into
resources. As for the corporeality of their avatar, it engages a cognition here virtually embodied, but
otherwise instantiated according to principles partly analogous to our human condition. Finally, through their
avatars, humans enter into a very strange trade with certain autonomized agents, typical of their adventure
worlds: pets, helpers and assistants, drones or robots, supernatural allies... With these, a distributed
cognition develops, i.e., distributing the processing of information over these autonomized entities, for
example, the tracking, surveillance or management of this or that process. All of this is linked to the ongoing
cooperation with the other comrades in the game, the other avatars. In this way, many intimate and vital
relationships are established and unraveled with programmed or hybrid technical beings, whose radical otherness
is forgotten, by dint of being dressed up in the colors of the narrative, of the game or of simple co-existence,
as within Second Life-type metaverse based on collective creativity.²¹
This diagram visualizes the way in which an avatarial hybridization takes place, framed and framed by a double
relationship that can potentially be interpreted in symbiotic terms: on the left, that of the human coupled to
the nested triad computer/software/avatar; on the right, that of a cybernetic "avatar" comparable to a symbiont,
as soon as it is engaged in relationships of interdependence with mutual benefit with its other congeners, or
even with various entities entirely programmed and eligible for a relationship of interest. It is in these
virtual worlds that symbiotic co-individuation could take place not only between the human and his avatar, but
also with the empowered virtual beings that also inhabit them.
b) The hypermedia avatar: an aggregate of data in the service of an identity model in hybrid spaces in
two ways
In contrast, we must address, even briefly, the other immeasurable domain of hypermedia, because the way in
which humans must avatarize themselves in order to exchange and cooperate with their fellow human beings is
quite different from the previous one.²² This involves the establishment of a "profile," with oriented
functionalities. This is in fact a particular identity model, a sort of explicit information sheet specifying
the aspects useful for the activities specifically allowed by the platform. By filling in fields and forms, by
providing qualities and quantities, by uploading media (photos, thumbnails, videos, visuals, sounds), the user
explicitly creates his profile, which will then be fed more or less invisibly by the "datas," "metrics" and
other "analytics" that are generated according to his activities. Because the activities are more disembodied,
more informational than simulational, the participant feeds the platform where he acts by his punctual acts, by
his simple consultations, requests, clicks and orders, which have become his editorial and media contributions.
There are certainly some services that focus on cooperation, such as Wikipedia for collaborative writing, or
GitHub for collective programming. However, most platforms, especially commercial or social ones, juxtapose
hyper avatars with no common space other than a customizable interface. They offer specific functionalities, for
example, for shopping or conversation, which require a simple interactivity, of command and impulse: summoning
contents, pages or profiles; clicking to act; typing and sending asynchronous messages to exchange; just
exciting by notes, scores or likes, or on the contrary, inhibiting by blocking and unsubscribing.
In the inaugural vision, it is within this informational, documentary and media hyperdimensionality; from
Vanevar Buch's Memex to Ted Nelson's precursor "hypertext"²³ ; that positive symbiotic relationships
were supposed to unfold. But we are struggling to find them, so much so that the algorithms that model and
influence behaviour encourage compulsive buying, isolation in filtered bubbles, the excitement of passions and
cognitive biases. Thus, these hyperspaces have tended to reduce the human to a set of models and
data,²⁴ made of functions and behaviors that integrate with software. User-consumers are thus
influenced and made predictable by traceability, profiling and probabilistic anticipations. The mainly
utilitarian relationships that are currently developing within the "hyper" digital environment have shown their
danger, even toxicity for democracies, potentially influencing crucial voting processes. At issue is the
modelling of stakeholders, which is often done without their knowledge, and the monitoring and stimulation of
behaviour guided and motivated by forms of economic, cultural and political predation that go beyond the logic
of advertising or the attention economy alone.
Such extreme pressure and overkill can be explained by the doubly hybrid nature of the Web and its variations.
On the one hand, "hyper" informational environments have a strong potential for intrinsic hybridization, linked
to the ease of combining media and documents that are hyperlinked, enriched with information, recombined,
declined, merged, diverted and replicated. They are also able to accommodate all other old (radio, television,
press) and present (platforms, services, applications) media, including cybernetic virtual worlds, which a
simple, well-programmed web page can display in some cases.
On the other hand, these "hyper" platforms are hybrid in the sense of devices (educational, professional) that
mix real and informational spheres. Most of the time, the infosphere constitutes a duplication of the real
world, which generates relationships of reciprocal influence. Human avatars, "hyper profiles" point, despite
their pseudonymity, to civil identities, while online businesses or connected data visualisations drain real
flows of goods and data. The result is numerous and strong power stakes and possible holdings attracting the
will to power, mobilizing all sorts of stakeholders: states, hackers, companies, collectives. Conversely, cyber
worlds and avatars seem to escape this extrinsic hybridity, because they remain more disconnected from the real
world, at least until the promises of metaverses claim to encompass them all and link them to our real values
through NFTs and other speculative crypto assets. Conversely, "hyper" universes are hybridizing both internally
and externally, the latter aspect having been intensified by the multiplication of mobile computers (phones,
tablets) and other connected and "smart" objects that link infosphere and biosphere, artificial and living.
III) Discussions and critical hindsight
a) Cyber and hyper polarizations: towards which symbiotic co-individuations through the
avatar?
At this stage, we should recognize the limits of this opposition between a symbiosis that is more possible in
simulated cyberworlds (immersive environments) and an unbridled hybridization in informational hyper-spaces
(distributed hypermedia and other Web platforms).
In the first place, there are still some "hyper" logics within cyberworlds, for example instant teleportation,
or interface parameter layers. Symmetrically, this time in hyper-spaces, some "cyber" systems act and function:
predictive models, recommendation engines and other underlying simulations ²⁵. Well hidden, in the
background, there is also this implicit avatar made without our knowledge, this identity avatar calculated and
manipulated by the 2.0 platforms whose existence has been recognized by the repentant Silicon Valleyers and who
have been described as "Digital Voodoo Dolls"²⁶ evoking the spell. This verifies the excesses that
many have denounced. All the more reason to take up one of our collective challenges, which would be to endow
hyperspaces with more cooperative and symbiotic properties/characteristics,²⁵ in particular by
interacting through hyperspace avatars that are more transparent, temporized, situated, regulated and better
controlled. As for "cyber" avatars, which are themselves driven by basic modelling or AI that may also contain
obscure biases (known as dark patterns), let us beware of idealizing or valorizing them without detailed
investigation. For the challenge is still to highlight, in order to cultivate and preserve them, the modalities
and relational conditions of regulation (coupling, embodied, situated and distributed cognition) and of positive
integration (benefits and gains, emancipation and empowerment) in order to foster mutualistic partnerships. This
can only be done through reciprocal adjustments and constitutive reinforcements towards a co-individuation of
all inhabitants of cyberworlds, which relies on all affordances available in these inhabitable virtual spaces.
Secondly, it should be noted that the processes of hybridization have been explored in very positive and
constructive ways, and this in the field of Art. Communities of artists, technologists, experimenters, or
university researchers, wanted to open other ways of convivial and regenerative co-existence with the new
digital technologies of art. The international colloquium "For a digital imaginary with Edmond Couchot.
Hybridizations between the arts, sciences, technologies and the human" in November 2022 was able to give an
account of how the revolutionary capacities of a computer have been understood for 50 years. The latter have
opened up the digital dimension by becoming universal simulators. Their interactive images constitute matrices
of hybridization "squared," producing through language and gesture "the image with image power."²⁷
This has made it possible to revisit, invent and materialize our imaginations, to make our fictions or visions
tangible. This conception is in line with the idea of a computer structuring a meta-technology²⁸ that
can simulate and accommodate all the others. This vast line of creations and works have also paved a way towards
partnership symbiosis by endowing technical virtual beings with a certain autonomy, as shown by the interactive
installations La Funambule (presented at ISEA 2000 ²⁹) and more recently
InterACTE³⁰, in which a humanoid creature learns and individuates itself through contact
with a participant who in turn experiments with new relationships with this autonomizing virtual being.
However, there is no spontaneously and naturally good and right approach: the artists in question have been
nourished and framed from the very beginning of their "human-computer partnership" by the contributions of
science and culture. This has given them the resources, knowledge and principles that enable them to escape the
hubris of omnipotence in order to assume the ethical, political and scientific responsibilities related to these
new demiurgic powers.
Thirdly, and lastly, the challenge is to offer intelligible and sensitive resources so that the anthropological
entry of humanity into its "virtual meta-machines" (connected computers), this exo-somatisation of our own
virtual double, can enrich us in return in a way that is both "technaesthetic" and conscious of everything that
determines it. The innumerable lives offered by avatars of all kinds—both "hyper" informational profiles
and "cyber" simulated corporeality—invite us to experience other real and imaginary worlds, dystopian or
desirable, acting as initiatory gateways, instructing our relationship to artefacts, to ourselves, to the
living, to our One Earth, and to this virtual sphere, to be envisaged as a "virtual common" to be healed.
b) What synergies between the sciences and the arts to finally assume our current
responsibilities?
The productive articulation Arts/Sciences including Technologies is justified by the interest that there would
be in the creative activities taking support on the scientific knowledge and investing the big changes of
paradigm which currently restructure a number of disciplines, of which precisely that of the symbiosis with the
rise of ecology, of the human and social sciences, of the sciences of the complexity and its solutions,
biomimetic and simplex derived from the evolution of the living. Certainly, if it should be reaffirmed that, by
definition, art is a domain that escapes the principle of responsibility, thanks to the famous artistic license,
it nevertheless deserves to take its full share and responsibility in today's "paradigmatic revolutions." These
are achieved precisely through changes in the model of understanding of the world,³¹ as is happening
today with anthropology, the sociology of science or the philosophy of technology: holistic approach,
destitution of the subject individual king, taking into account technical relations and processes and new
entities at work (non-human actors, stakeholders, resources and forces), better integrations of
interdependencies and a growing consideration for subtle, invisible or microscopic dimensions, inviting
temporalities and spaces incommensurable for us mere humans.
As for the problematic of avatars, it links us to the more general one of the exosomatization of the highest
cognitive faculties of the human species in virtual techniques and creations. We believe that in this field, art
can not only bring us alternatives, shifts, surprises, wonder, but that it constitutes with science one of the
most adapted means to approach complexity and uncertainty, to synthesize even paradoxical advances. The
Art-Science alliance, which is still to be built, can be based on different ways of coupling them, in order to
favour a capitalization, a mutual collaboration. If art is stratified and memorized through culture, scientific
advances build knowledge. Now, in view of the emergence of new technologies, there is every reason to encourage
a crossing of these two systems of accumulation, so that art is a source of knowledge and science also
structures and disseminates culture. In order to better understand this, let's think about how the ludic
activity exploring the potentialities of computer science has finally escaped the arts and the instituted
sciences for a long time, even though games and arts anticipated together and restructured the bases of the
future of interactive computer science, with an intensity and a power recognized today through games studies
like the 10ème art.
This is also why recent initiatives such as issue 9 of the journal HYBRID, published at the end of 2022 in
English and French,³² organize a cross and cumulative dialogue between scientific studies and artistic
research, in this case around the figure of technological avatars offering their practitioners media
incarnations to encounter informational environments or immersive environments.
It was during one of the dialogues at the heart of this review that object-oriented transversal approaches were
able to discover their complementarity on both sides of the arts and sciences, and that a difference in
temporality between artist-researchers preoccupied with the future and scientist-analysts more focused on the
present or the past could be better formulated. The interest in tomorrow's artificial creatures, which should be
welcomed and respected, to quote Edmond Couchot,³³ justifies many projects and experiments. Let's take
up again the parallel with Licklider, when he specified that it was necessary to deal with the symbiotic
partnership with the computer in the time interval that separated us from the arrival of "strong" AIs. In these
still open times, before the promises of sentient robots or a perfect autonomized simulation of human beings are
realized, there is every reason to take care of our partnerships with our avatars, by seeking together the
conditions that will make us environmentally and behaviorally responsible. Hence the richness of symbiotic
approaches, both in the virtual and in the real, since the two are constantly hybridizing in a way that is now
inextricable.
IV) Conclusion in the form of programmatic questions, bias and references
The first question, of general scope, would be to know: "in what way does the avatar belong to a case of
human-technological hybridization? For as we have detailed, the technological avatar achieves the integration of
the living and the artificial in real time in the same iconic programmed object, in an animated body giving
access to a virtual environment and manifesting the human to his fellow creatures of the same species and
other quasi-species, virtual beings in the process of empowerment. But does this result in more than a
hybridization of techniques between them, a mixing, a specific mixture, producing the interpenetration of
stakeholders (subject, machines, programs and human and machine intentionalities)? And how, in return, does the
sensitive and embodied internalization of technical logics in the human being take place through "existential"
experiences, psychosocial and individualizing experiences (Simondon), with subjective and objectifiable
experiences, echoing Edmond Couchot's technesthesia to the “cyberesthesia” proposed by the French
artist Yann Minh, or even Macluhan and the media extension of the senses? But also, how to go beyond symbiosis
as a metaphor for the living and elaborate its factual declensions, adapted and relevant to these hybrids of
flesh and calculation that are the avatars? Consequently, how to configure their habitat, these virtual
cyberworlds?
And finally, how can we configure differently, with the virtual, the technical conditioning and the cultural
overdetermination, both inevitable, in order to achieve a common world that is equitable, sustainable and
desirable in terms of reciprocal emancipation? The construction site is vast and the construction techniques to
be implemented remain to be invented and popularized by this synergy between science and the arts that we are
calling for. Let us be numerous and determined to invest ourselves in it for the benefit of all, human and
non-human.
(1) In our theory, we prefer to distinguish techno-social (when living being are inside artificial construct)
and socio-technical level (socialization of technical artefact).
(2) As the famous French philosopher René Descartes explains in his time in “Discourse on the
Method of Rightly Conducting One's Reason and of Seeking Truth in the Sciences”, 1637.
(3) Named after the 1986 Habitat network games that referred to the inhabitants of this networked world as
Avatars
References
1 Call of communication ISEA, 2023, online.
2 Vilem Flusser and Ann Roth Nancy, Into the Universe of Technical Images, NED-New
edition, Vol. 32; University of Minnesota Press, 2011, accessed December 08, 2022, Ac
http://www.jstor.org/stable/10.5749/j.cttttcq8.
3 William Regli, "Revisiting the Human-Machine Symbiosis," Computing Research Association
Bulletin, April 2016, accessed December 08, 2022, https://cra.org/revisiting-human-machine-symbiosis.
5 James E. Lovelock, Lynn Margulis, "Atmospheric homeostasis by and for the biosphere: the gaia
hypothesis", Tellus, 26:1-2, 2-10, (1974), accessed December 08, 2022
https://www.tandfonline.com/doi/abs/10.3402/tellusa.v26i1-2.9731.
6 James Lovelock, Gaia A New Look at Life on Earth Oxford, Oxford University Press, 1979.
7 André Leroi-Gourhan, Gesture and Speech, The MIT Press, 2018, Translated by Anna
Bostock Berger, First French Edition, Albin Michel, Paris, 1965.
8 Gilbert Simondon, Individuation in Light of Notions of Form and Information (Volume 1
& 2, Translated by Taylor Adkins, Univ of Minnesota Press, 2020, original French edition, Paris,
Jérôme Millon, Collected papers of 50's, 2005.
9 Alain Berthoz, Simplexity: Simplifying Principles for a Complex World, Yale University Press,
2012.
10 Maurice Merleau-Ponty, Phenomenology of Perception, Routledge, 2013, Translated by
Donald Landes, original French edition, 1945.
11 Francisco J. Varela, Evan Thompson, Eleanor Rosch, The Embodied Mind: Cognitive Science and
Human Experience, MIT Press 1991.
12 Licklider, J. C. R., "Man-Computer Symbiosis", Ire Transactions on Human Factors in Electronics
1 ,1960, 4-11.
13 Philippe Descola, Beyond Nature and Culture, University of Chicago Press, 2014,
Translated by Janet Lloyd, original french edition 2005.
14 Gilbert Simondon, On the Mode of Existence of Technical Objects, Translated by
Cécile Malaspina, Univ of Minnesota Press, 2017, original French edition, Aubier, 1958.
15 Douglas C. Engelbart, William K. English, 1968, A research center for augmenting human
intellect, In Proceedings of the December 9-11, fall joint computer conference, part I (AFIPS '68 (Fall, part
I)), Association for Computing Machinery, New York, NY, USA, 1968, 395-410.
16 Claudia Salamanca, "The mother of all demos" in DAC'09, Digital Arts and Culture 2009, accessed
December 08, 2022, https://escholarship.org/uc/item/91v563kh.
17 Gregory Bateson, Steps to an Ecology of Mind. St. Albans Paladin, 1972.
18 Chip Morningstar, F. Randall Farmer, The lessons of Lucasfilm's habitat, Cyberspace:
first steps, MIT Press, Cambridge, MA, USA, 1991, 273-302.
19 Michel Bret, Marie-Hélène Tramus, Alain Berthoz, "Interacting with an Intelligent
Dancing Figure: Artistic Experiments at the Crossroads between Art and Cognitive Science", Leonardo 38, no. 1,
2005, p.47-53, accessed December 08, 2022, http://www.jstor.org/stable/1577645.
20 Bob Rehak, "Playing at Being: Psychoanalysis and the Avatar." In The Video Game Theory Reader,
2003, 103-138, accessed December 08, 2022, https://www.academia.edu/3751080/Playing_at_Being_Psychoan
alysis_and_the_Avatar.
21 Tom Boellstorff, Coming of Age in Second Life: An Anthropologist Explores the Virtually
Human, Princeton, Princeton University Press, 2008, 316.
22 Louise MERZEAU, " Habiter l'hypersphère", Documentaliste-Sciences de l'Information,
2010/1 (Vol. 47), accessed December 08, 2022, 30- 31,
https://www.cairn.info/revue-documentaliste-sciences-de-l-information-2010-1-page-30.htm.
23 Ted Nelson, Computer Lib/Dream Machines, Revised Edition, Tempus Books, 1987 (first
edition,1974) Felix Guattari, Three Ecologies, Translated by Ian Pindar & Paul Sutton, Bloomsbury
Academic, 2014 (original French edition 1989).
24 Bernhard Rieder, "Beyond Surveillance: How Do Markets and Algorithms 'Think'?" The Foucaldian
3, no. 1, 2017, 1-20, DOI: https://doi.org/10.16995/lefou.30 [Note: In 2022, Le foucaldien relaunched as
Genealogy+Critique.] accessed December 08, 2022 https://www.genealogy-critique.net/article/id/7055/.
25 Bernhard Rieder, Engines of Order: A Mechanology of Algorithmic Techniques, Amsterdam
University Press, 2020, Accessed December 08, 2023, https://www.jstor.org/stable/j.ctv12sdvf1.
26 Marija Slavkovik, Clemens Stachl, Caroline Pitman, Jonathan Askonas, "Digital Voodoo Dolls", In
Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (AIES '21), Association for Computing
Machinery, New York, NY, USA, 2021, 967-977, Accessed December 08, 2023,
https://www.researchgate.net/publication/353603454_Digital_Vo odoo_Dolls.
27 Edmond Couchot, Images. From optical to digital. The visual arts and the evolution of
technology, Paris, London, Lausanne, 1988.
28 Alan Kay, Adele Goldberg, Personal Dynamic Media. In: Wardrip-Fruin, Noah / Montfort, Nick: The
New Media Reader, Cambridge MA, The MIT Press, 2003, 393-404.
29 Marie-Hélène Tramus, "Multisensoriality Interactive Digital Art, Presentation of an
Experience: The Virtual Tightrope Walker" (paper submitted in ISEA 2000) Only Proceeding, 2000, 200-204,
https://www.isea-archives.org/docs/2000/proceedings/ISEA2000_proceedings.pdf
30 Dimitrios Batras, Judith Guez, Jean-François Jégo, InterACTE: Improvising with a
Virtual Actor, In Proceedings of the 3rd International Symposium on Movement and Computing (MOCO '16),
Association for Computing Machinery, New York, NY, USA, Paper 52, 2016,
1-2, https://doi.org/10.1145/2948910.2955109
31 Bruno Latour, An Inquiry Into Modes of Existence: An Anthropology of the
Moderns, Havard University Press, translated by Catherine Porter, 2018.
32 Etienne Armand Amato, Judith Guez, Etienne Perény, Marie-Hélène Tramus,
"Prospective and retrospective reflections on avatars and virtual actors in artistic experimental
exploration," Hybrid [Online], 9, 2022, online 30 November 2022, accessed 09 December 2022,
http://journals.openedition.org/hybrid/2808
33 Edmond Couchot, Automata, robots and virtual humans in the performing arts, PUV, 2022
Authors Biographies
Étienne Armand Amato has been a lecturer in information and communication sciences at Gustave Eiffel
University and the DICEN-IDF laboratory since 2014 (www.dicen-idf.org). Since the beginning of high-rate
networks, he studies interactive audiovisual forms (video games, virtual reality), ludic communication and
media avatars. He also contributes to the work of IHEST (Institute of High Studies for Science and
Technology), a training organization of the French Ministry of Research.
Étienne Perény is honorary professor of communication and interactive design, a member of the
laboratory Paragraphe at the Université Paris 8 de Vincennes and associated with the DICEN- IDF
laboratory. Since the 70s, he experiments new technologies: video, architectural simulation, videodiscs, cable
networks, interactive video prototypes, offline multimedia, then online, in the early days of the Internet. He
has developed an original theory on interactive images and video games, and the research device "becoming
avatar" with Alain Berthoz and E.A. Amato.
Symphony of the Stones: A Research-Creation Exploration on the Animation of Heavy Metal Residues in
Contaminated Urban Landscapes
BriceAmmar-Khodja
Concordia University &
École des Arts Décoratifs – PSL (EnsadLab) Paris,
Montreal brice.ammar-khodja@mail.concordia.ca / brice.ammar@ensad.fr
Abstract
In the late 1980s, the Canadian Pacific Railway abandoned a rail yard on the outskirts of Montreal’s Mile
End district. Within a few years, the return of animal and plant species encouraged the citizen community to
reinvest this site known as Le Champ des Possibles. Despite community efforts to rehabilitate this site,
hydrocarbon and heavy metal pollution persists in the soils and thus requires rethinking the engagement with the
imperceptible mutations of ecosystems. Symphony of the Stones was created in response to this context.
This research-creation project consists of several urban art installations that activate residual metals in
soils by their magnetic characteristics to make these imperceptible pollutants visible. The following paper
unfolds the different processes, methodologies and strategies that led to in site interventions blending art
installation, collaboration with different communities and associations and leading to a rethinking of art
practices in the urban environment.
Keywords
Digital arts, research-creation, art-science practices, situated artistic interventions, environmental
installations, contaminated urban soils, active residual materials, material agency, pedological studies,
environmental participation, socio-environmental issues.
Although invisible, heavy metals abundant in soils have the potential to contaminate natural environments,
thereby increasing the risk of serious diseases. In recent years, the discovery of new polluted sites underlines
the extent of this pollution. Over a century of industrial activities has led to this pollution spreading at an
alarming rate throughout urban areas. Nowadays, 54 percent of the global population lives in an urban
area,¹ and, in Quebec alone, 10 percent of the identified sites are located within Montreal.² In this
regard, citizens' exposure to these toxic residues requires a rethinking of engagement with the imperceptible
mutations of ecosystems due to anthropogenic activities.
Despite the fact that Karl Marx anticipated ecological urgency and soil loss by what he theorized as the
"metabolic rift," soils have only recently gained interest from social and biological/geological sciences and
science and technology studies (STS).³ On the other hand, the works of contemporary artists and
artist-researchers such as Debra Solomon, Amy Balkyn, Martin Howse, Joana Hadjithomas, Khalil Joreige or
Disnovation.org, illustrate this urgency by raising awareness on biodiversity loss and soil degradation. As the
push intensifies to transition these discussions out of strictly academic and artistic realms, the pressing
challenge becomes: how can we expand the conversation on soil contamination, and more broadly, urban pollution,
ensuring it resonates with the general public, especially the communities and stakeholders most affected by it?
Écotones is a collaborative research-creation project which arose from this question. Led with
artist Philippe Vandal and in collaboration with the citizen association Les Amis du Champ des Possibles and
more precisely with the coordinator Sugir Selliah and the co-president and climate policy analyst Émile
Boisseau- Bouvier, Écotones operates as an in situ urban laboratory. It melds
experimental exhibitions and artistic interventions, articulating an aesthetic, critical, and social perspective
on soil pollution. This research-creation project, blending art-science methodologies, seeks to explore heavy
metals and hydrocarbon pollutant agencies to activate new experiences of soil pollution and offer critical
visibility to these invisible residues. Two directions were taken: Symphony of the Stones (my own work)
which explored heavy metal pollutants and Fluorescent Forensic by Philippe Vandal, which explored
residual hydrocarbons. The first iteration of this project took place at Le Champ des Possibles in Montreal on
October 14th. Le Champ des Possibles (CDP), is a former marshaling rail yard left as a brownfield by the
Canadian Pacific Railway company in the 1980s and rehabilitated as a green space in 2013. Despite community
efforts to rejuvenate this land, hydrocarbon and heavy metal pollution still lingers in the soil. The ongoing
proliferation of these pollutants presents dire ecological and health risks that cannot be overlooked. As the
site was formerly used as a waste disposal site, the municipality and the association les Amis du Champ des
Possibles (ACP) have therefore agreed to rehabilitate the site by excavation under section 22 paragraph 9 from
the Loi sur la Qualité de l'Environnement.⁴ However, few citizens are aware of
these hazardous materials, whose existence deserves to be discussed.
This article zooms in on Symphony of the Stones, an in situ art installation that seeks to
enhance the understanding and perception of heavy metal residues in contaminated soils. The article
contextualizes the making of the installation and the way it connects to CDP. First, the article focuses on
recontextualizing what CDP is and how the concept "ecotones"—that gave the name to the series of
interventions—emerged from exploring this site. The second part of the article delves into the art
installation and how it activates soil's heavy metal contaminants to shine a spotlight on the invisible traces
of pollution lying under the ground. Finally, the third part delves into the concept of public intervention,
drawing inspiration from various contemporary in situ art installation practices.
Exploring the entanglements
Le Champ des Possibles as a space of negotiations
Le Champ des Possibles is located in the Mile-End district of Montreal. The district underwent significant
gentrification, initially with the influx of artists in the 1980s and more recently due to the emergence of art
galleries, design showrooms, and companies in the innovation and video game industries. Despite this
socio-economic context, CDP remains in the state of a brownfield. The site is characterized by the entanglement
of various plants, animals, and human communities living together with the traces and ruins of its past
industrial activities. The foot paths laid by morning joggers and commuting wanderers are lined with campfire
remains and industrial ruins (figure 1). Steel structures, now enveloped by diverse species of trees and plants,
provide shaded resting spots for local workers. These makeshift sitting areas are occasionally rearranged by
visitors to the site. The northern border of CDP neighbors a railway which traces a separation between the
Mile-End and the other districts. For years, these fenced borders have been the subject of debate among the
citizens, the municipal authorities and the Canadian Pacific Railway that owns the railroad.⁵
Breaches in the fence allow for easy crossing to the other side of the railroad and the reconstruction of this
same fence by the public authorities are thus common things to observe throughout the year. Because of the
diversity of uses and communities that make CDP a unique space, it is not easy to give a single definition of
the site as it conjures up different imaginary perceptions. Facing this post-industrial biotope and the
entanglement of different human and more-than-human figures negotiating the site with each other, it appears
that CDP was scientifically reminiscent of a concept in environmental science that would help us formalize and
conceptualize a dialogue with the space: the ecotone.
Ecotone as a conceptual framework
Ecotone is a term that emanates from environmental sciences. It describes areas of gradation between
different ecological communities, ecosystems or even regions. In other words, places where diverse
ecologies are put in tension.⁶ Although ecotones are not restricted to the natural environment, they can
also be induced by human activities.⁷ According to Jurek Kolasa and Maciej Zalewski’s book
Ecotone attributes and functions,⁸ these contact zones are hard to define. They point out
that ecotones can range in size from a few centimetres to kilometres and can be distributed across spaces in
different contexts with varying degrees of porosity and types of boundaries. Moreover, ecotones can be
horizontal and extend over planar surfaces or vertical when localized in the atmosphere or water bodies.
This overlapping inspired us to build a conceptual framework that would foster insitu exploration of
CDP. This allowed characterizing the area, not only at the ground level but also by understanding the different
vertical layers and boundaries on the distribution of natural elements and pollutants in the soils. This notion
of ecological boundaries guided the exploration of the site. Discussion with members of the citizen association
ACP, highlighted that these ecological boundaries were overlapping with a map from the Quebec government
classifying soils by degrees of contamination within CDP. The map shows a satellite view of CDP where zones are
divided into polygon sections, each providing information on the name and concentration of pollutants, and
the possible effects of these residual matters. In addition to this map providing technical information
about the site, the ecotones framework became a tool for understanding the socio-environmental interactions of
the site. While the map showed CDP's contaminants split into polygons, the notion of ecotones revealed other
boundaries that would appear through the exploration of the site: different paths generated by the users of the
site oscillating between gravel roads and tracks where the grass does not grow anymore, the green rehabilitation
zones scattered on the site to allow the vegetation to grow back, the campfires where only the waste of
late-night rave parties remain, and finally the railroad, which illustrates the most significant transition zone
of this unique ecotone. Through this first overview, CDP appears as a space of socio-cultural and biological
diversity in constant mutation. From a natural space transformed into a rail yard to a half-industrial,
half-green public space, and finally, to a space in the process of decontamination, the transition of this
landscape is not only remarkable for the diversity of communities, boundaries, and uses of the space, but also
its transformations through space and time.
While a simple walk in CDP would reveal the aforementioned ecological boundaries, the former use of CDP as a
waste disposal site left traces of which a map alone cannot give a complete account. As the entanglement of the
many elements that compose CDP tends to blur what this site is composed of, the primary approach of the project
consisted of developing art installations, unveiling imperceptible elements.
Sensing mutations
Residual invisibilities
Soils are dynamic systems composed of organic matter, living organisms, gases and water. When the amounts of
naturally occurring elements, such as heavy metals (i.e., copper, lead, nickel, arsenic), increase abnormally
due to external, and usually anthropogenic, factors, a pollution risk arises.⁹ When elements, not
naturally occurring in soils, such as hydrocarbons, are found, the phenomenon is termed
contamination.¹⁰ As with heavy metals, excessive levels of these external components can lead to
pollution risks and disrupt the natural soil cycles. However, most of the dynamic interactions and
transformations in the soils, even the most concerning, are not easily perceptible to the naked eye. Human
relations with soils are largely limited to what human sensory organs can detect and interpret. Indeed,
perception relies on a combination of the senses. Among these, vision, described by Plato as "the noblest of the
senses," dominates as the "most informative" and "associated with the highly valued faculty of
reason."¹¹ The effectiveness triggered by perception can contribute to increasing awareness,
understanding, and care for soils. However, how can one foster these sensations when the soil's mutations are
imperceptible? In Thinking with Soils, Material Politics and Social Theory, Anna
Krzywoszynska, Manuel Tironi, Matthew Kearnes, Céline Granjou, and Juan Francisco offer
socio-environmental and ethico-political ways to approach soils with a care-oriented analysis inspired by the
work of Puig de la Bellacasa. In the book's second chapter, the authors point out that soil conservation is
challenging because soil ”is commonly presented as invisible, hidden under our feet, and lacking a
recognizable 'face.'"¹² Since soils are composed of microscopic elements (living beings, organic
matter, minerals, pollutants, etc.), soils' intrinsically animated and living nature is not always evident to
those who observe them. Thus, the authors insist that the complex nature of soils and their non-zoomorphic and
non-anthropomorphic characteristics make them more vulnerable and, therefore, more at risk of being neglected
and damaged. However, this mixture of non-human entities, whether organic or inorganic, animate or inanimate,
possesses their agency and the potential to influence and affect systems of which humans are part. This notion
relates to what Jeanne Bennett calls "thing-power", defined as “the curious ability of inanimate things
to animate, to act, to produce effects dramatic and subtle.”¹³
Soils are discrete entities, and pollution is invisible. In some contexts, and environments, the biotope
sometimes contributes to weakening the perception of pollution. Let us recontextualize this assumption: at first
glance, CDP looks like a site in the process of green rehabilitation. Since the Canadian Pacific Railway company
left the site as a brownfield, species of plants, trees, and animals have begun to reoccupy the site. This
reoccupation might have paradoxically contributed to blurring the distinction between the past and present of
the site; that is, a place undergoing green rehabilitation versus a place where pollutants still lie and mutate
beneath the soil. Discussions with the citizen association ACP have confirmed these suppositions by revealing
that many of the citizens living next to the site were unaware of CDP's pollution. Symphony of the Stones
emerged in response to these issues. This research-creation project explores how the activation of heavy
metal pollutants can operate on perception and senses to engage in a critical dialogue with the communities
bordering these sites. As part of the series of artistic interventions titled Écotones, this
installation draws from practices and interests in active materials, material agency, material forensic and
art-science practices and focuses on generating critical discussions with the riverside communities on the
issues related to CDP discussed in the next section of the article.
Symphony of the Stones: a research-creation case study
Symphony of the Stones is a series of urban art installations that examine the correlation between the
abnormally high presence of heavy metals and magnetic residues in CDP's soils. This first iteration was
developed within the framework of the research-creation thesis, titled the Cycles of Attraction and
derives its methodology from geomagnetism—a geosciences subfield specializing in identifying soil
pollutants via magnetic fields.¹⁴ The installation reinvestigates this technique to activate heavy
metal and magnetic contaminants in CDP's soils that present high responsivity to magnetic fields. The
installation features two geometric aluminum sculptures, designed to harmoniously integrate with the landscape,
reminiscent of the metallic remnants of CDP's past industrial activities (figure 2 and 3).
The sculptures are flush with the ground and strong enough to be walked on. Using the Quebec Government map
indicating soil classification by degrees of contamination, these ephemeral sculptures are strategically placed
over highly polluted polygons and topped with CDP's soils. Concealed within and below the sculptures is a
kinetic system comprising N45-grade permanent magnets. These magnets, driven by DC motors and controlled by a
microcontroller (figure 3), generate magnetic fields that create material tensions and attractions, activating
the magneto- responsive constituents of the soils. This includes ferromagnetic residues, construction debris
like nails, discarded municipal items like beer caps, and variously sized earth clumps. In a metaphorical sense,
the installation seeks to visualize the invisible soil transformations. The movement of the soil creates a
subtle yet captivating animation aimed at drawing attention to the ground. Concurrently, the installation delves
into the acoustic properties of soils and residues. One sculpture integrates a piezo microphone, connected to a
TDA2822M mini audio amplifier and a Bluetooth transmission system, relaying the audio signal to a discreetly
placed Bluetooth speaker (figure 4). This system captures and amplifies sounds produced from the collisions of
various materials. By amplifying these subtle soil movements, the installation adds an auditory dimension to the
ambient sounds of the CDP environment. By animating both organic and inorganic residues and materials from CDP's
soils—those seemingly inert to casual observation—Symphony of the Stones aspires to foster
a renewed, sensory connection between humans and soils. The project is informed by the concept of "active
materials" or “kinetic materials.” These materials can react and adapt by changing colours, textures
or events” moving in stimuli such as heat, magnetic fields, or moisture. In Materials That Move: Smart
Materials, Intelligent Design, Murat Bengisu and Marinella Ferrara explore the potential of several
active materials features and how their application in design changes users' relations with everyday objects.
Drawing inspiration from Donald Norman, an expert in the application of cognitive science to design and the
theorist behind emotional design, Bengisu and Ferrara highlight the potential of active materials to
"materialize intangible information that is imperceptible to the human senses in daily life."¹⁵
The dynamic created by these objects, and materials, made possible by organic and organic-like movements, might
signify the expression of something that is animated and living, triggering different senses and emotions.
Exploring discarded materials, residues and contaminated soils coming "alive" by their activation, Symphony
of the Stones aims to encourage the public to not only further observe soils but also get closer to the
ground, carefully observe the different patterns drawn by the materials in motion and haptically engage. The
project's first iteration showed promising results during the Écotones exhibition, where visitors
would actually get closer to the installation and eventually touch the soil. The future versions of the
installation will put more emphasis on the haptic aspect. As Puig de la Bellacasa points out in Matters of
Care, haptic promises to develop new forms of knowledge based on attentiveness through the body's
engagement.¹⁶ The knowledge that the vision alone, by its distant and detached character, could not
reach.
Intrusions, furtivity, and symbiosis
Making an in situ urban art installation involves intervening in the environment. The first part of
this paper explained why CDP is a space in constant negotiation between human and non-human communities. In this
context, Symphony of the Stones explores new ways to engage with the environment and its communities,
rather than merely subjecting the site to a single artist's vision. This approach required working closely with
ACP for a deeper understanding of the site (see section "Exploring the entanglements"). However, as illustrated
in the first part of this article, there is no consensus on what CDP is or should be. Out of respect for the
site, Symphony of the Stones was designed to neither disrupt nor damage the site's environment. In
doing so, the installation diverges from the traditions of environmental art, especially Land Art, an art
movement from the 1960s that showcased open-air artwork—often large-scale installations using natural
elements found on site. Land Art sought to foster a dialogue between art and natural environments, underscoring
the localized aspects of works now displayed beyond the confines of museums and galleries. Prominent examples
include Robert Smithson's Spiral Jetty (1970), a 457-meter- long basalt stone dike spiraling
counterclockwise over slightly reddish water at the Rozel Point peninsula on the northeastern shore of the Great
Salt Lake, or Walter De Maria's The Lightning Field (1977), a permanent installation of 400 steel poles
spread over a mile, designed to attract lightning in a desert plain in Quemado, New Mexico. While sometimes
classified as a form of ecological art, Land Art is not without controversy. Sometimes equated as a form of
ecological art, Land Art is also controversial. Some interventions were indeed drastic, leading to artworks that
altered their environments ¹⁷. Since then, myriad art forms have emerged, influenced by the practices
of environmental and land art. Some extend their focus beyond the natural to encompass the built environment,
with a growing emphasis on addressing socio-environmental and political challenges linked to the sites under
consideration. Regrettably, a recurring issue with in situ urban or environmental art installations is
the gap between the discourse of these practices and the methodologies they employ. In her article Le Lourd
Bilan Carbone de l'Art Contemporain, Gill Gasparina argues that many contemporary urban and environmental
installations, while raising awareness of ecological concerns, still rely on processes detrimental to the
environment, such as pollutant technologies, excessive energy use, and waste generation.¹⁸ Given the
increasing integration of new technologies and the creation of custom artworks, the carbon footprint of
contemporary—and particularly digital—art is undeniable. This highlights the pressing need to
further evaluate energy consumption and material repurposing in the pursuit of more sustainable artistic
practices.
Symphony of the Stones was conceived with the dual intention of minimizing its ecological and
aesthetic impact on CDP. In pursuing this, Symphony of the Stones is influenced by what art critic and
art historian Patrice Loubier terms “furtive practices.” Furtive practices are described as
“multiple art practices that infiltrate the urban landscape through works without signs, not necessarily
recognizable as artworks, serendipitously surprising the pedestrian, transforming them into an incidental
observer” (My translation).¹⁹ In contrast to Land Art, furtive practices do not dominate due to
their scale; instead, they subtly alter details within these environments. Underlying these practices are a
range of artworks whose subtlety setting does not diminish their political, social, or environmental
significance. Drawing inspiration from Danae Stratou’s Concentric (2019), a piece that produces
continuous concentric ripples in Nirox Sculpture Park’s Lake, Symphony of the Stones integrates
seamlessly with its surroundings, invoking elements already present. The aluminum's polished surfaces help to
camouflage the sculptures by mirroring the environment around them. Moreover, the installation is largely
enveloped by CDP’s soils, which are retrieved from the topmost layer using DIY (Do It Yourself) permanent
magnet soil collectors, thus avoiding any digging or excavation. Neither of the two sculptures, nor their
concealed kinetic systems, needed anchoring or attachment. This adaptability ensures the installation's
portability and ease of removal. Such features position Symphony of the Stones as an artwork that may
elude immediate notice, nudging site visitors to be more observant of their surroundings. Yet, the
installation's auditory aspect offers clues about where and what to focus on. The initial display of the artwork
yielded varying reactions based on CDP's ambience and foot traffic. On certain occasions, passersby might
overlook the sculptures entirely, even walking over them without pausing. However, when the contours of these
metal forms caught their eye, some would halt, scrutinize the sculpture more closely, and approach upon
observing the animated soil atop the structures. During the Écotones outdoor exhibition, the
influx of visitors meant that some remained oblivious to the sculptures. Nonetheless, those who became aware of
them through the animated soil often ventured closer, with a few even interacting directly with the residues by
touching them (figure 5 and 6).
After the opening, we organized a reception in a space adjacent to CDP, and allowed the installation to remain
on site into the evening. Upon our return to disassemble the installation, the darkness had enveloped CDP, and a
group of night-time explorers had kindled a campfire approximately a meter away from the sculptures. In spite of
the resonating sounds and the movement of the residues, this group remained oblivious to the installation. By
this attempt of creating an installation furtively within the site, it is hard to evaluate what are the best
conditions for it to be perceived and understood in its message but also as a work of art. A consideration for
future iterations of this installation, and more generally within the domain of furtive practices, might involve
implementing strategies for artistic and cultural mediation for artworks displayed in urban settings.
Conclusion
This article sketches the premises of a methodological framework for research-creation practices that operates
at the scale of urban spaces and more specifically spaces in the process of rehabilitation. Although
contextualized on a specific site in Québec, I hope that this contribution will foster new ways of
collaboration with outdoors spaces and hopefully participate in the debate on urban soil pollution. The
CDP’s case study has demonstrated that wild urban spaces, brownfields, and any other grey area are places
of entanglements and negotiation between different communities. Amongst these communities that range from human
wanderers and species of plants to toxic residues, different types of agencies whether positive or negative,
might emerge or persist. Against this backdrop, a question emerges: how can artistic endeavors be realized in
harmony with these intricate interconnections? By highlighting presence of heavy metal pollutants in the soil,
Symphony of the Stones opted to harmonize with the CDP. By suggesting new forms of non-intrusive
collaborations with the site and its communities, this research-creation project seeks to activate and increase
the perception of contaminants. In this regard, the project does not only seek to redraw borders on the site,
but rather amplify its already existing aspects. As part of the project Écotones, this first
iteration is perhaps not immune from mutating towards a more radical but still furtive kind of intervention in
the urban landscape.
References
1 Food and Agriculture Organization of the United Nations, International Fund for Agricultural
Development, International Organization for Migration, and World Food Programme, The Linkages between
Migration, Agriculture, Food Security and Rural Development: Technical Report, 2018, 5.
2 Gérald Dermont, “Sols Pollués Par Les Métaux Lourds
Résultant De L'enfouissement De Déchets Industriels (montréal, Canada) : Géochimie,
Spéciation Des Métaux, Et Décontamination Par Flottation,” Dissertation,
Université du Québec, Institut national de la recherche scientifique, 2008.
3 Juan Francisco Salazar, Céline Granjou, Matthew Kearnes, Anna Krzywoszynska,Manuel Tironi,
eds, Thinking with Soils: Material Politics and Social Theory, London, UK, Bloomsbury Academic, 2020,
4-32.
4 Les Amis du Champ des Possibles, Interview by Brice Ammar-Khodja, December 6th, 2022
and October 14th, 2022.
5 Ibid.
6 Robert Leo Smith, Ecology and Field Biology, 4th ed, New York, Harper and Row, 1990, 251.
7 Salit Kark, Rik Leemans, Environmental Systems Analysis Group, Wageningen University,
Wageningen, 6700 AA, Netherlands, 2013. “Ecological Systems: Selected Entries from the
Encyclopedia of Sustainability Science and Technology,” Essay, In Ecotones and Ecological
Gradients, New York, NY, Springer, 147–60.
8 Jurek Kolasa, Maciej Zalewski, “Notes on Ecotone Attributes and Functions,”
Hydrobiologia: The International Journal of Aquatic Sciences 303, no. 1-3, 1995, 1–7.
9 Peter M Chapman, and Sixth Iberian and 3rd latinoamerican Congress on Contamination and
Environmental Toxicology Cadiz (Spain) 20050925-20050928. “Determining When Contamination Is Pollution
— Weight of Evidence Determinations for Sediments and Effluents.” Environment
International 33, no. 4, 2007, 492– 501.
10 Ibid.
11 David Howes, “Sensory Craft,” Craft Forward, California College of
the Arts, San Francisco, 2 April 2011.
12 Francisco Salazar, Céline Granjou, Matthew Kearne, Anna Krzywoszynska, Manuel Tironi,
17.
13 Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke
University Press, 2010, 6.
14 Xue-Feng Hu, Li Mei, Zichen He, Lei Cui, Liu Rui, Xin-Dong Wang, Zhi-Han Wang,
“Magnetic Responses to Heavy Metal Pollution of the Industrial Soils in Shanghai: Implying the
Influences of Anthropogenic Magnetic Dustfall on Urban Environment,” Journal of Applied Geophysics,
2022, 197.
15 M. Bengisu, Marinella Ferrara, Materials That Move: Smart Materials, Intelligent Design,
Springerbriefs in Applied Sciences and Technology, 1, 2019.
16 Puig de la Bellacasa María, Matters of Care : Speculative Ethics in More Than Human
Worlds, Posthumanities, 41, Minneapolis, University of Minnesota Press.
17 Virginie Luc, De La Nature À L'œuvre, Paris, Ulmer, 2014.
18 Jill Gasparina, Le lourd bilan carbone· de l'art contemporain, Le Temps, 8 janvier
2019, https://www.letemps.ch/culture/lourd-bilancarbone-lart-contemporain
19 Patrice Loubier, “Un Art À Fleur De Réel : Considérations Sur
L’Action Furtive,” Inter 81, no. 81, 2002, 12–17.
Bibliography
M. Bengisu, Marinella Ferrara, Materials That Move: Smart Materials, Intelligent Design,
Springerbriefs in Applied Sciences and Technology, 2019.
Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press,
2010.
Peter M. Chapman, and Sixth Iberian and 3rd latino american Congress on Contamination and Environmental
Toxicology Cadiz (Spain). “Determining When Contamination Is Pollution — Weight of Evidence
Determinations for Sediments and Effluents,” Environment International 33, no. 4, 2007,
492–501.
Gérald Dermont, Sols Pollués Par Les Métaux Lourds Résultant De
L'enfouissement De Déchets Industriels (montréal, Canada): Géochimie,
Spéciation Des Métaux, Et Décontamination Par Flottation, Dissertation,
Université du Québec, Institut national de la recherche scientifique, 2008.
Food and Agriculture Organization of the United Nations, International Fund for Agricultural Development,
International Organization for Migration, and World Food Programme, The Linkages between Migration,
Agriculture, Food Security and Rural Development: Technical Report, 2018.
Jill Gasparina, “Le lourd bilan carbone· de l'art contemporain,” Le Temps,
January 8, 2019, https://www.letemps.ch/culture/lourd-bilancarbone-lart-contemporain.
David Howes, “Sensory Craft”, Craft Forward, California College of the Arts, San Francisco, April
2, 2011.
Xue-Feng Hu, Li Mei, Zi-Chen He, Lei Cui, Rui Liu, Xin-Dong Wang, Zhi-Han Wang, “Magnetic Responses to
Heavy Metal Pollution of the Industrial Soils in Shanghai: Implying the Influences of Anthropogenic Magnetic
Dustfall on Urban Environment,” Journal of Applied Geophysics, 2022, 197.
Kark Salit, Rik Leemans, “Ecological Systems: Selected Entries from the Encyclopedia of Sustainability
Science and Technology,” Essay, In Ecotones and Ecological Gradients, New York, NY, Springer,
2013, 147– 60.
Jurek Kolasa, Maciej Zalewski, “Notes on Ecotone Attributes and
Functions.” Hydrobiologia: The International Journal of Aquatic Sciences 303, no. 1-3,
1995, p.1–7.
Les Amis du Champ des Possibles, Interview by Brice Ammar-Khodja, December 6, 2022, and October 14, 2022.
Virginie Luc, De La Nature À L'œuvre, Paris, Ulmer, 2014.
Patrice Loubier, “Un Art À Fleur De Réel: Considérations Sur L’Action
Furtive.” Inter 81, no. 81, 2002, 12–17.
María Puig de la Bellacasa, Matters of Care: Speculative Ethics in More Than Human Worlds.
Posthumanities, Minneapolis: University of Minnesota Press, 41.
Juan Francisco Salazar, Céline Granjou, Matthew Kearnes, Anna Krzywoszynska, Manuel Tironi,
eds., Thinking with Soils: Material Politics and Social Theory, London, UK, Bloomsbury Academic,
2020.
Robert Leo Smith, Ecology and Field Biology, 4th ed., New York, Harper and Row, 1990.
Aknowledgements
I would like to thank the Canada Excellence Research Chair (CERC) on Smart, Sustainable and Resilient
Communities and Cities at Concordia University, the Concordia Sustainability Action Fund (SAF), the
research-creation international network Hexagram, and the Haute École des Arts du Rhin (France) and the
Arrondissement Le Plateau-Mont-Royal for their support and funding. Also, I would like to thank the Concordia
University Research Chair in Critical Practices in Materials and Materiality, the Milieux Institute, the
Topological Media Lab, and the EnsAD, EnsadLab’s Reflective Interaction Research Group for providing
conceptual and technical support. Finally, I am thankful to the citizen association Les Amis du Champ des
Possibles, the Arrondissement Le Plateau-Mont-Royal and artist Philippe Vandal for making this collaborative
project possible.
Author’s Biography
Brice Ammar Khodja is an artist, graphic designer, and Ph.D. student based in Montreal and Paris. His work
examines active materials, residual matter, and low-technologies to explore the socio-environmental and
political interconnections pertaining to materiality and visual information.
He is currently pursuing a thesis jointly supervised in Concordia University – Montreal (Individualized
Program) and EnsAD, EnsadLab – Paris (Reflective Interaction research group, SACRe program).
He is a member of Chaire Arts & Sciences, Concordia University Research Chair in Critical Practices in
Materials and Materiality, Speculative Life Research Cluster (Milieux Institute), Centre for Sensory Studies,
and Concordia’s Canada Excellence Research Chair in Smart, Sustainable and Resilient Communities and
Cities. Brice is a current member of the international research-creation network Hexagram.
Co-director of the typography magazine Pied de Mouche, Brice Ammar-Khodja, creates workshops and
educational tools for the general public.
His works have been exhibited at Ars Electronica (Linz, Austria), MUTEK (Montréal, Canada), Centre
Pompidou (Paris, France), Biennale internationale du Design, la Cité internationale des Arts
(Saint-Étienne, France), V2_Institute for Unstable Media (Rotterdam, Netherlands), Musée
historique de la Ville de Strasbourg (Strasbourg, France) and Second Internatinal Print Biennale (Yerevan,
Armenia).
The “Bichi” Project Symbiotic Food Networks & the Alchemist Kitchen
PatBadani
Independent Artist/Researcher @ ISEA International Board Argentina/Canada/USA
pat.badani@gmail.com
Abstract
The author uses symbiosis as metaphor and tool to create food-related projects that critically examine the
anthropogenically induced impact of global warming on food chains. She argues that "art-through-food" projects
promote an alternative worldview informed by the utopian premise that art can facilitate reflexivity and
influence behavior to prevent future massive starvation by safeguarding the future of food. Three projects
reveal the workings of energy flows in ecosystems to reimagine hybrid relations with living matter and with
systems that are biological, technological, social, and political. They establish sustenance networks in which
symbiotic interactions between several actors form the basis for a reevaluation of our human relationships to
the planet and the fungal, plant, and animal agents within it. Based on assemblages with multiplicities that
cofunction via a viral logic of contagion, projects explore aspects of symbiosis in human/plant/bacterial
consortiums (the Bichi project), as well as symbiotic intelligence combining computing with neural
networks (@Comestiblemealplan). Informed by research linking various scientific fields with art and
technology, projects explore networks of relations between entities tangled in interdependence and involving
parasitism, mutualism, adaptation, and resilience. Congruent with a larger systems approach in which symbiosis
is the core principle replacing an essentialist conception of individuality, hybrid works contribute research
and knowledge production with the potential to assess and generate affective insights and acts in the world.
The world is made up of complex and sometimes unmappable landscapes of interacting biological, climatic,
social, economic, and political forces. Interconnected global systems have been created to produce, reproduce,
and consume our material environments—systems in which not all partners have derived reciprocal benefits.
How these systems are linked to human and more-than-human entities and the planet, have intrigued me for decades
and the aim of this paper is to discuss how my research on nutritional networks is transformed into critical
artistic language. As lens, I use the concept of "symbiosis" seen as relationships of coexistence and
synergistic evolution that include interchange, cooperative processes, power relationships, contact,
penetration, conversion, and last and not least, eating others and being eaten.
I frequently use food as material and subject in projects driven by research into climate change, creature
welfare, social disparity, and food inequality. These projects involve scientific exploration and the creation
of hybrid consortiums with living materials (foodstuffs) and systems that are biological, social, and political,
to critically reimagine human/non-human relationships in symbiotic dynamic interplay. They are independent
practice-as-research productions that integrate the technological, the ecological, and the edible, and are
considered "art through food"¹—rather than the more broadly defined "food art" which has a long
history in the visual arts, with representations of food from ancient Greek art where depictions of food were
intended to denote a civilized lifestyle that separated them from barbarians, to Medieval food art in which
artists of the time used images of food to illustrate differences in the eating habits of the wealthy and the
poor, to the well-known mannerist work by 16th-century Giuseppe Arcimboldo or, much later, the 20th century
illustrated cookbook by Salvador Dalí “Les dîners de Gala” with images intended
primarily as aesthetic, sensual, amusing, and decorative representations providing a spectacular feast for their
eyes.
On the other hand, works that fall under the category of ‘art-through-food’ imbue the experience of
food with critical meanings: cultural, social, and political. A historical overview on the use of food in 20th
century art shows that for many artists food became a suitable instrument of critique, a means to enter a
conversation about broader issues. These projects often assume an educational and even activist stance, at times
also involving audience participation, and are intended to extend public knowledge and engagement. For example,
Tomasso Marinetti’s “Futurist Cookbook” (1937), a political manifesto advocating a type of
cuisine that would embody the new progressive ideals of modernism; or Andy Warhol’s Campbell’s Soup
series (1962) where soup becomes "brand" to denote consumer society; not to mention the use of food in staged
photography, the moving image, and computer-generated images related to social practice and community art, where
food has been used to explore the promise of gift relations and notions of hospitality and commensality. Some of
these involved relational aesthetics with artists in the role of facilitators rather than makers, like Rirkrit
Tiravanija’s “Pad Tahi” (1990), and Mella Jaarsma’s “I Eat You Eat me”
(2001-InCUBATE), or projects focused on mapping as social practice such as “Data Cuisine Collective”
by Moritz Stefaner exploring food as a medium for information (2011-ongoing). And, finally, in social media
where meal photos showcasing an array of eating behaviors became emblematized in Twitter, Facebook, and
Instagram’s ‘food selfie’ phenomenon, for example, Pat Badani’s
“Comestibles” (2010) and Joy Garnett’s “Kitchen Studio” (2010) addressing the
fuzzy line between media and genres in Web 2.0 society, and the pervasiveness of these technologies in our lives
from making art to sharing our food. Furthermore, the reassessment of our relationship with the material world
prompted by developments in quantum physics showing that matter we previously thought of as 'inert' is in fact
made up of vibrating strands of energy—prompted a revision of relationships between matter and materials,
with intellectual approaches that consider matter as possessing agency and qualities associated with generative
becoming.² “Live” bio-art that often combines the aesthetic and the artistic with organic,
often living matter, further complicates this association in a still developing genre ranging from works
grounded in artists’ manipulation of genetic sequences to works in which biological organisms are not
treated as material, but rather as symbiotic partners in creation.³ Critical Art Ensemble’s
“Free Range Grain” (2003) and Center for Genomic Gastronomy’s “Cobalt 60 Sauce”
(2011) are notable examples of bio-artists who explore the overlapping boundaries between art, food, ecology,
and geopolitics.
The mentioned progression of ideas since the 20th century, linking art, technology, science, society, its
objects and its subjects, embody moral, economic, and social codes of meaning that provide a reflective backdrop
for me to assess symbiotic relationships in the creation of recent projects. In these works, I extend the
category of symbiont to welcome non-human living entities and their energy flows, both organic and electronic,
in processes of coexistence that allow me to assess the quality of relatedness manifested in their exchange,
whether commensal, competitive, or parasitic. This set-up, marked by the synchronous presence of growth,
maturation, and decay,⁴ allows me to pay attention to differences in power relations and positionality, as
well as to imagine what non-binary relations that include non-human acquaintances might be like.
Art-through-Food and knowledge production
Jane Bennet claims that “Food, as a self-altering, dissipative materiality, is also a player. It enters
into what we become. It is one of the many agencies operative in the moods, cognitive dispositions, and moral
sensibilities that we bring to bear as we engage the questions of what to eat, how to get it, and when to
stop.”⁵ Indeed, eating reveals not only the interdependence of humans and edible matter, but also a
capacity to effect social change. Food is integral to existence and its manipulation since the Industrial
Revolution has caused the world’s greatest imbalance due to the immense environmental footprint involved
in systems of food production and distribution. My investigation into discourses driving nutritional systems
from WW2 to the present, has led me to evaluate the quest for mastery over bodies and
environments—behavior conceptualized through ideas of progress, beauty, domestication, and profit. These
ideas permeated by the dominance of humans over non-humans—supported by systems like the agricultural and
food processing industries—have resulted in the collapse of ecologies large and small. For decades and
even centuries, human and more-than-human entities have been the sites of battles where various ideologies and
scenarios have been deployed involving the foods humans consume day-to-day. Shifts in dietary patterns over time
denote changing lifestyles that are driven by a variety of factors with two major ones being urbanization and
the industrialization of food. In fact, the latter is pervasive and increasingly dominated by complex agri-food
networks and global food supply chains. In this system, food has become an object of political negotiations on a
national as well as a transnational level—a situation that questions the sustainability of industrial food
production, distribution, and consumption. These global systems support agricultural practices that pollute or
degrade the quality of lands and soils, contributing to an imbalanced planetary scenario with massive
overproduction and abundance coupled with scarcity of nutritious foods. And, despite the relatively recent rise
of alternative food movements, food safety programs, and plans proposing delivery of more abundant and
nutritious foods world-wide, what often predominates is agri-business’ grip on food resource
narratives—narratives based on capital gain rather than needs—a situation that suggests new
opportunities for interventions into the system like the initiatives led by artists committed to
"art-through-food." These art initiatives often include research, participatory strategies, and the creation of
speculative fictions that, like in many of my own projects, aim to assess and mitigate consumer confusion and
uncertainty caused by the over-saturation of recommendations on "good" or "bad" regimes—contradictory
information, often supplied by food industry players.
Art-through-Food projects contribute to research and knowledge production with potential to measure and
construct affective insights into spaces of opposition, tension, and conflict. Ecocide, social isolation, racial
injustice, gender rights, and economic disparity are spaces at the threshold of potential, possible collapse,
instigating in some, the desire for new ways of being in the world based on awareness of the transcendent
dimension interconnecting all living things, and prompting the development of new, symbiotic relations not just
between humans, but also with other-than-humans. In this regard, art-through-food projects play a crucial role
in promoting a better awareness and understanding of the root problems, and help scale and spread approaches
that increase health, equity, and well-being, while also protecting the ecosystem.⁶
Positioned among these art initiatives, my recent projects cultivate ideas about the construction of
sustainable, symbiotic futures, and are at times suggestive of Si-Fi tropes and speculative visual narratives.
Through culinary arts, photography, and 3D simulation software, 3D printing and physical computing, I create
critical visual arguments that blend aesthetics and imagined futures that reaffirm the material world’s
tendency to persistently organize new forms and create anew, despite failures, in a determined will to live.
"Bichi" ⁷,⁸ is one such project into which several works are nested involving scientific
investigation, macrophotography, and 3D simulation software.
The “Bichi” art project
Bichi involves creative collaboration with food science and technology and extends my decades-long
investigation into ecological concerns and the entangled indicators pertaining to sustenance networks and the
foods we eat. Nested in the Bichi project, Bichicuchitu, Bichicu-Chow and
BichEden are hybrid works positioned between the organic and the computational and explore the
intersectionality of energy flows, food systems, and climate emergencies. Executed with 3D simulation software,
the animations expose the porousness of boundaries between what might appear as separate
registers—subject/object, fact/fiction, and science/culture. Inspired by forms of animism, herbalism, and
panpsychism, the works provoke cognitive dissonance in the viewer by suggesting provisional boundaries between
entities, organisms, and species. They represent analytical inquiries into the symbiotic interrelation of our
metabolic pathways, biological food chains, and global networks—questions that are at the forefront of
imaginaries in times when massive starvation may become part of our reality due to the impending ecological
disasters that jeopardize the future of food. These works explore the notion of exchange and of mutual
transformation through an amalgamation of matter that meet each other to form new speculative life forms
grounded on the notion of "assemblages." Operating at many different levels of affective bodies, the individual
human body—my own body being an assemblage in and of itself - is seen as part of this assemblage.⁹
Bichicuchitu & Bichicu-Chow / The alchemist’s kitchen / Symbiotic networks
For me, the action of artmaking is like cooking in an alchemist’s kitchen—with many projects born
in the kitchen and further developed in other environments and mediums in processes that involve assembling and
collecting things together, witnessing rupture or disturbance, and remaining open to unexpected events that give
new access to the real. I see my kitchen as a private laboratory—a special theatre in which I envision and
negotiate, not only new knowledge, but also new politics around new actors entering my world, conditions
exemplified by the “Bichi” project.
At the onset of the COVID pandemic in 2019, I observed mold and fungi gradually take over decaying vegetables
and fruits that I had soaked and disinfected before refrigerating, as was typically recommended at the start of
the epidemic. Surprisingly, a bio-based symbiotic interaction unfolded with the organic matter transformed into
peculiar trans-species assemblages [ Figure 1]—bio-sculptures seemingly infinite in their variability
because bacterial communities are known to strongly differ across plant species, likely due to variations in
metabolites, physical characteristics, and symbiotic interactions with the host plant and other microbial
inhabitants.¹⁰ This phenomenon became a guided experiment involving a bio-based process of culturing
and decomposing in a domestic fridge, a DIY method that supports the importance of scientific tinkering as art
practice. Said method established a discreet sustenance network between 3 interdependent units: bio-sculpture
material, mold, and fungi, and myself (my consciousness) as sensory symbiont. Through time, I witnessed the
synchronous presence of growth, maturation, and decay, as death of the organic tissue was accompanied by the
thriving growth of the infecting bacteria, mold, and fungi; a process in which different bodies, organisms, and
species were in the course of ‘becoming’ with each other through ongoing negotiations that involved
nourishment, resilience, and transformation—a process in which the notion of ‘predation’ is
seen as a renovative act of living together in extended material relationships. As a key partner in bringing the
microbial symbionts into aesthetic perception, I documented the DIY bio forms with macrophotography,¹¹
spawning a provocative series of images of living creatures that begged to be animated—or better yet
mutated into two works that were created with 3D simulation software. Playfully titled Bichicuchitu
[Fig.2] and Bichicu-Chow [Fig. 4], the names point to my native language (Argentinian-Spanish) in
which the vernacular term ‘bicho’ is used generically to name any "critter" or unidentifiable
creature, big or small.
Drawing associations between living organisms, sculpture, photography, and 3D computer modeling, the artistic
process happened in layers through observation and re-creation. The passage from one media to the other in the
constellation revealed differentiated yet interconnected aspects of artistic creation. They involved evolution
of new forms that, through transition and transposition, unfolded expressive intensity that intermingled their
boundaries and highlighted the role of technology in fathoming emotion in new forms of visual fictions. In this
regard, I consider art as a tool for mediation that makes possible reflexivity concerning complex phenomena that
contribute to awareness about the endangered future of food caused by climate change and exacerbated by the food
chain disruptions caused by the COVID-19 pandemic and ongoing geopolitical crises.
Bichicu-Chow and social media networks – Paradise and Anti-Paradise
In keeping with holistic approaches to natural life and scientific, cosmological, and social ideals, the
checkered "quilt" that enrobes the 3D modeled creature in Bichicu-Chow [Figure 5] is taken from my
digital intervention titled @Comestiblemealplan [Figure 3],¹² a tactical media work that
considers the problematic logic driving notions of novelty, progress, and endless growth promoted in social
ecosystems. Food is an integral part of human existence and one of the most fruitful areas of manipulation, and
in my Instagram-based work I confront this trend deployed in social media food-narratives. Images, chronicles,
and advertisements are voraciously circulated and consumed in an environment that allows me to probe the
symbiotic exchange of energy and information, and to furthermore inspect how the newly generated energy realizes
symbiosis evolution. In @Comestiblemealplan, the ecological chain structure of the ecosystem is
composed of the symbiotic flow of information facilitated by the network structure, and said structure is like
that of ‘natural’ ecosystems with producers, consumers, and decomposers that realize the flow of
matter and energy through these three main elements in stages of formation, growth, and maturity.¹³
My probe began with the selection of digital images of agro-industrialized foodstuffs which I transformed
through digital glitching processes. “Glitching" indicates the purposeful introduction of "noise" in a
system resulting in aberrant and apparent ‘abnormal’ renderings of technologies.¹⁴ The
back-end code driving digital images of foodstuffs were re-codified with the introduction of texts from literary
sources that showcase the centrality of discourse about food production, distribution, and consumption ranging
from Paradise Lost by John Milton (1667)¹⁵ to The Edible Woman by Margaret Atwood
(1969)¹⁶ to Como agua para chocolate by Laura Esquivel (2001). The glitching process
produced unexpected digital disruptions woven into what became @Comestiblemealplan’s Instagram
feed—a checkered canvas that was later transposed as a digital "quilt" covering the 3D simulated creature
in Bichicu-Chow.
In @Comestiblemealplan, the social media platform and the public form a symbiosis relationship through
the flow of information, and because of the two-way communication mechanism with feedback, the platform opened
the possibility of engaging a conversation about money, hope, and experience in relation to digital commensality
and nutrimental systems. This energy flow favors the co-creation of transformation narratives in support of
possible better futures.
Ecosystems / Future Food / BichEden
An ecosystem is a group of entities that live and communicate with each other in specific communities and
environments. In "natural" ecosystems, symbiotic units are the basic components of energy production and
exchange that make up a symbiosis relationship. Forming a symbiosis relationship of coexistence and synergistic
evolution implies that the need for survival between entities in an ecosystem is inevitably interdependent and
interactive in some way.¹⁷ Although ecosystems vary greatly in appearance, structure, and feature,
for the most part, they are composed of three main elements: producers (such as plants), consumers (an organism
that eats carnivores, herbivores, parasites, and scavengers), and decomposers (a recycling system made of
organisms such as bacteria, fungi, earthworms, etc.). Ecosystems realize the smooth flow of matter and energy
through these three main elements in which energy and matter, in the form of food, flow through the ecosystem. A
food chain describes who eats whom, that is to say, the method by which a particular organism gathers
its food, a chain that represents the flow of energy from one organism to the next and to the next, and so on
(grass, a producer, produces its own food from sunlight; rabbit, a consumer, eats grass; fox eats rabbit; eagle
eats fox, and so forth throughout the chain of organisms in the ecosystem). Typically, interruption to one
element of the ecosystem produces waves and ripples that touch every member of the system, and for survival of
sustenance networks, human and more-than-human populations need to interact in balance with each other because
when connections between them become unbalanced, the likelihood of ecosystem collapse increases. For example, in
marine ecology, coral reefs are at the base of ocean food chains. When reefs collapse some organisms have no
food, such as shrimp, a critical food source for fish. Their decline is causing a sequence of disruptions in the
food chain that affect not only fish, but also wildlife, and ultimately humans who rely upon seafood for
sustenance, medicines, and income. To subsist, both humans and non-humans need the energy transmitted through a
healthy, undisrupted food chain, yet current conditions create food scarcity and jeopardize the future of food.
Indeed, eating reveals the interdependence of humans and edible matter, yet the greatest imbalances for human
and environmental health has been caused by agro-industrial manipulation of foodstuffs and the immense footprint
involved in production and distribution systems. Agro-industrial practices contribute to global warming that
drives changing climate patterns, extreme weather events, loss of biodiversity, droughts, and nutrient deficient
soils. These are acutely felt by those who depend on the stability of farming ecosystems. For example, record
temperatures and prolonged droughts in 2022 led to sharply reduced crop yields in many regions, with the impact
likely to extend through 2023. The drought in the Horn of Africa (Ethiopia, Somalia, Djibouti and Eritrea) has
led millions of people to move from the stricken areas in search of food. Considering these examples, the future
of food has become a serious question facing all of humanity.
Food security is at the forefront of my concerns with questions such as: What will we feed a growing population
in times of growing climate catastrophes? How will we stave off massive global starvation? What will we eat and
who will grow our food in the future? I am curious about an emerging ‘connected food ecosystem’ in
which social, economic, scientific, and technological advances explore ways to feed a global population based
upon sustainable, networked relationships. Evidence of these revolutions can be found everywhere from the rising
generation of change- makers exploring regenerative bioregional cultures and economies of care through
traditional ecological knowledge; "back to the future" wisdoms such as regenerative permaculture imbued with the
critical skills needed to feed, heal, and nurture communities; "farm to fork" strategies that aim to make food
systems fair, healthy and environmentally friendly; food coops and farmers markets who reimagine food
systems by promoting local sourcing and food transparency; "citizen science" with amateur-scientists in the
fields of biology and conservation participating in projects that also educate the public about scientific
processes and increase awareness about the move towards sustainable diets and a sustainable food system; and
last but not least ‘food engineering’ involving innovative science-based solutions to producing food
through genetics. These concerns and possibilities spawned BichEden, a new project bolstering my
commitment to investigating encounters between organic and technological formations. I began developing "still
life" artifacts that explore the notion of "vibrant matter,"¹⁸ and in collaboration with Mariel
Martinez (software engineer and co-founder of Poetry+AI=Art) I began to develop digital assemblages envisioned
for several outputs, ranging from a series of 3D animations to a responsive installation that evokes an
imaginary, symbiotic inter-species scape.
BichEden: Plankton [Figure 5], currently in early stages of research and production, explores
the layered contexts within which new qualifications of life are being determined and manipulated. The sculpture
does not represent what already exists, it invites unexpected convergences; it imagines potential edible life
where fusion and remixing are the result of symbiotic, ecological thinking.¹⁹
At the base of the work is a perishable sculpture which I physically constructed in my studio with plants,
algae, mollusks, vegetables, etc.; creations that define an introspective journey into the network of
interactions among organisms in marine food chains. I collect, remix, subvert, and transform these edibles which
are then "queered up" through a time-based, durational, and fluid process resulting in a hybrid, biomorphic
assemblage that embodies difference and strangeness. Before the ephemeral sculpture withers, I photograph it
from all 360 angles. Then, in collaboration with Mariel Martinez and the use of photogrammetry (the art and
science of extracting 3D information from photographs), the images are processed via digital simula1on software
with 3D modeling functionality, texturing, lighting, and a host of other tools that allow us to electronically
re-engineer the “still life” as a "virtual" 3D artifact. Drawing associations between living
organisms, sculpture, photography, and 3D computer modeling, the artistic process happens in layers through
observation and re-creation. The passage from one media to the other in the constellation reveals differentiated
yet interconnected aspects of artistic creation that highlight the role of technology in fathoming emotion in
new forms of visual fictions. In this way, both physical and virtual studios become generative spaces for the
construction of non-binary, other-than-human companion forms that embody feelings of disturbance and queer
possibili1es, a merging of multiple narratives in which fact and fiction intersect.
Conclusion
The Bichi project constitutes an attempt to understand ecosystems by making visible the vulnerable
encounter between variable living forms, organic and technological. With the aim of learning about the nature of
vitality modeled after symbiotic relationships in the natural world, the multipart Bichi project
establishes hybrid consortiums to critically re-imagine human/non-human relationships in symbiotic dynamic
interplay. Bichicuchitu, Bichicu-Chow and BichEden examine paradigm shifts about the
care of nutrimental networks, dissect problematic global perspectives dominated by economics and endless growth,
and speculate on possible futures with holistic connections. Based on art’s ability to push disciplinary
boundaries, and debunk and critique ideologies, I argue in favor of art’s utopian commitment to envision
and influence alternative scenes of cultures worth nurturing. In this regard, works nested in the Bichi
project provide an invitation to feel, map, gauge, and learn from the swirling patterns around us to better
understand and influence what happens next.²⁰ Congruent with a larger systems approach in which
symbiosis is the core principle replacing an essentialist conception of individuality,²¹ my hybrid
works contribute research and knowledge production with the potential to assess and generate affective insights
and acts in the world.
These creative and critical visualizations are based on the notion of organic and inorganic "assemblages,"
techno biologisms where the affects of one body are seen to enter into composition with other affects in
symbiotic relationships, to co-join in composing a more powerful body. Informed by extinct or endangered food
web organisms, physical and virtual studios become sites of a curious cultural archaeology of artefacts that act
as a record or a future-oriented worldmaking.
References
1 Markéta Dolejšová, “A taste of big data on the global dinner
table,” Journal for Artistic Research, 9.
2 Pat Badani, “Bichicuchitu,” Pat Badani professional website, accessed
September 21, 2022, https://patbadani.net/pro_Bichicuchitu_1n.html
3 Pat Badani, “Bichicu-Chow,” Pat Badani professional website, accessed
September 21, 2022, https://patbadani.net/pro_Bichicu_chow_1.html
4 Jane Bennet, Vibrant Matter: A Political Ecology of Things, 49.
5 Jonathan W. Leff, Noah Fierer, “Bacterial Communities Associated with the Surfaces
of Fresh Fruits and Vegetables,” PLoS One Journal, March 27, 2013, accessed November 9,
2022.
6 Pat Badani, “Incu-Bichu (2019),” Pat Badani professional website, accessed
September 21, 2022, https://patbadani.net/pro_incu_bichu_1.html
7 Pat Badani, “@Comestiblemealplan,” 2020-ongoing, accessed September 21,
2022, https://patbadani.net/pro_Comestible_Instagram_1.html
8 Ming Xia, Xiangwu He, Yubin Zhou, Symbiosis Evolution of Science Communication Ecosystem
Based on Social Media: A Lotka–Volterra Model-Based Simulation in “Coevolving Spreading
Dynamics of Complex Networks,” Complexity Journal, Vol. 2021, accessed November 9, 2022.
14 Michael Betancourt, Glitch Art in Theory and Practice: Critical Failures and Post-Digital
Aesthetics, UK, Taylor & Francis - Routledge & CRC Press, 2019, 3.
15 Jeanne M. Ali, “Feasts of power: how food reveals Eve's influential role in John Milton's
epic poem Paradise Lost,” Ph.D diss., Florida International University, FIU Digital Commons, 2004, 1-47.
16 Stevan Mijomanović, “Cannibalism, Fertility, and the Role of Food in Margaret
Artwood’s The Edible Woman,” AM Journal of Art and Media Studies, Vol. 10, 2016, 67-76,
accessed November 9, 2022.
17 Ming Xia, Xiangwu He, Yubin Zhou, Symbiosis Evolution of Science Communication Ecosystem
Based on Social Media: A Lotka–Volterra Model-Based Simulation in “Coevolving Spreading
Dynamics of Complex Networks”.
18 Jane Bennet, Vibrant Matter: A Political Ecology of Things, 13.
19 Morgan Meis, Timothy Morton’s Hyper-Pandemic in “Persons of
Interest,” The New Yorker, June 8, 2021, accessed November 9, 2022.
20 Adrienne Maree Brown, Emergent Strategy: Shaping Change, Changing Worlds,
Oakland, CA, AK Press, 2017, 4.
21 Scott F. Gilbert, Alfred Tauber, “Rethinking Individuality: The Dialectics of the
Halobiont,” Biology and Philosophy, 31 no 6, 2016, accessed November 9, 2022.
Bibliography
Pat Badani, “Cru et Crédible (Raw and Credible): Art and Food”,
SOCIAL/AFFECTS, Drain Journal, Vol. 17:1, 2021.
Jane Bennet, Vibrant Matter: A Political Ecology of Things, USA, London, Duke University Press,
2010.
Adrienne Maree Brown, Emergent Strategy: Shaping Change, Changing Worlds, Oakland, CA, AK
Press, 2017.
Michael Betancourt, Glitch Art in Theory and Practice: Critical Failures and Post-Digital
Aesthetics, UK, Taylor & Francis - Routledge & CRC Press, 2019.
Markéta Dolejšová, “A taste of big data on the global dinner
table,” Journal for Artistic Research, 2015.
Scott F. Gilbert, Alfred Tauber, “Rethinking Individuality: The Dialectics of the
Halobiont,” Biology and Philosophy, 31 no 6, 2016, accessed November 9, 2022.
Serenella Iovino, Serpil Oppermann, “Material Ecocriticism: Materiality, Agency, and Models of
Narrativity,” Ecozon, Vol 3, No 1, 2012.
Jonathan W. Leff, Noah Fierer, “Bacterial Communities Associated with the Surfaces of Fresh Fruits and
Vegetables,” PLoS One Journal, March 27, 2013.
Stevan Mijomanović, “Cannibalism, Fertility, and the Role of Food in Margaret Artwood’s The
Edible Woman,” AM Journal of Art and Media Studies, Vol. 10, 2016.
Timothy Morton, Hyperobjects, USA, Univ Of Minnesota Press, 1st edition, September 23, 2013.
Weber, Andreas, Matter & Desire: An Erotic Ecology, Chelsey Green Publishing, 2017.
Xia, Ming, Xiangwu He, Yubin Zhou, Symbiosis Evolution of Science Communication Ecosystem Based on Social
Media: A Lotka–Volterra Model-Based Simulation in “Coevolving Spreading Dynamics of Complex
Networks,” Complexity Journal, Vol. 2021.
Biography
Pat Badani draws from the fields of art, science, and technology to explore the intersectionality of
environmental and social justice issues. She often uses food to create artistic arguments that blend
aesthetics and criticism, charting connections between theories related to art as object, as medium, and art
as critique of political and technological networks. Badani exhibits and discusses projects broadly in North
and South America, Europe, and Asia, and has participated in international symposia with her essays and talks
in over 15 countries including at the Institute of Cultural Studies ZHdK Zurich (Switzerland); iDAT Plymouth
University (U.K.); Art/Sci Center + Lab, UCLA (USA); NYU Steinhardt School of Culture, N.Y. (USA); SPARKS ACM
Siggraph (USA); Kungl. Konsthögskolan/KKH, (Sweden); Università di Bologna (Italy); Universidad de
Caldas, (Colombia); and Universidade Anhembi Morumbi (Brazil). Projects have been distinguished with critical
essays published in English, Spanish and French in anthologies and journals, and with over 20 awards and
commissions by the Canada Council for the Arts; Illinois Arts Council; DCASE; National Endowment for the Arts
Fellowship @ MacDowell, the Robert Heinecken Trust; and recognized with nominations by Creative Capital, Art
Matters, and AWAW, in the U.S.A.
Thief of Truth: VR comics about the relationship between AI and humans
JoonhyungBae
Graduate School of Culture Technology, Korea Advanced Institute of Science and Technology
Daejeon, Republic of Korea jh.bae@kaist.ac.kr
Abstract
Thief of Truth is a first-person perspective Virtual Reality (VR) comic that explores the relationship
between humans and artificial intelligence (AI). The work tells the story of a mind-uploaded human being reborn
as a new subject while interacting with an AI that is looking for the meaning of life. In order to experiment
with the expandability of VR comics, the work was produced by focusing on three problems. First, the comic is
designed using the viewing control effect of VR. Second, through VR controller-based interaction, the player's
immersion in the work is increased. Third, a method for increasing accessibility to VR comics was devised. This
work aims to present an example of an experimental attempt in VR Comics.
Keywords
VR comics, the relationship between humans and AI, multimodal artwork.
Artificial Intelligence (AI) has been developed greatly in intentionality, intelligence, and adaptability.
Accordingly, technology has had a great influence in the areas of finance, security, health, justice,
transportation, and the urban environment.¹ With the advent of AI, which can think and speak like a human,
the boundary between humans and machines is becoming increasingly blurred. Ameca, a robot unveiled in December
2021 by Engineered Arts, a British robot company, has 12 facial expressions implemented by 17 motors, and has
human-like language skills based on GPT-3. The robot was able to hold a conversation with developers in
September 2022.² Harmony, a sex robot developed by Realbotix in the U.S. in 2018, is evaluated as an AI
robot capable of mentally and physically interacting with humans as well as being capable of love.³ AI
Sophia, developed in 2015 by American roboticist David Hanson, is a humanoid robot that resembles a human in
appearance and has obtained citizenship for the first time in the world.⁴ In the post-human era of the
21st century, humans are expected to enter a time of living together with AI.⁵
As a result of humans and AI coexisting in this new era, a new consideration has emerged pertaining to the
personhood of AI or AI robots. In this regard, in February 2017, the European Parliament passed a resolution
granting robots the status of electronic persons to regulate robots.⁶ This resolution suggests that as
technology develops, AI robots are increasingly being recognized for their existential value, similar to humans
as a person.⁷
If AI or AI robots can be understood as a person similar to humans, can AI be regarded as the other of humans?
American philosopher Donna Haraway, in her book The companion species manifesto: Dogs, people, and
significant otherness states the concept of the other is not limited to humans but extends to all
non-humans such as animals, insects, bacteria, and machines. Haraway emphasizes a love that acknowledges the
significant otherness, which refers to learning “to live in an intersubjective world" rather than thinking
of the object as a subordinate being or wanting unconditional love from the object.⁷,⁸
Emmanuel Lévinas and Slavoj Žižek contemplate the infinity or reality beyond the reality of
the subject against the philosophy of identity. The other, they suggest, is a concrete universality, which
exists only through the existence of the subject that develops beings. Yet, when a subject forms an identity in
relation to the "other," it becomes another version of the "other," transcending its defined identity. Only by
truly embracing this exteriority or "otherness" can something exist as an interior or subject. At this time, as
long as there is a relationship with the other, the subject inevitably encounters another concrete manifestation
of the other. Now, for the two philosophers, these others are not mere hindrances but the only passage to the
other.⁹ Currently, when AI emerges and again questions what a human being is, understanding AI, which has
emerged as a new other, is an important task in understanding humans.
Comics about coexisting with AI
In order to better understand AI, there are comics dealing with AI as a being that lives together with humans.
One of the first comics is Osamu Tezuka's Mighty Atom (1952-1962). The story is about a robot who protects
humans in a society where robots with egos are discriminated against by humans. In another comic, Kazumasa Hirai
and Jiro Kuwata's Eight-Man (1963-1964) presents the idea of what it means to be human through a protagonist who
struggles between being a robot and a human. Ishinori Shotaro's Android Kikaider (1972-1974) explores the
morality of machines by questioning whether AI is righteous. Robot Detective (1973) by the same artist tells the
story of K, a robot designed by humans that realizes the ideals of justice, absolute obedience, and natural
coexistence with humans. Fujiko Fujio's Doraemon (1977-) further develops what a robot can do by introducing a
story of a human AI who can even make mistakes living alongside humans. Hisae Iwaoka's Saturn Apartments
(2006-2011) deals with the story of an AI that has been isolated for a long time and wonders about its
existence. Yoshiie Koda's Love of Machine Device (2013-2014) describes a robot more righteous and pure in heart
than humans, even though it is a mechanical device that does not feel joy or sorrow. Ha Il-kwon's 3 Level
Combination Kim Chang-Nam (2008) expresses the friendship between a robot and a boy and indirectly captures the
problems of the times, such as the gap between the rich and the poor and school violence.¹⁰
VR immersion and viewing control effect
Unlike other forms of media, VR technology has the potential as an empathy-realizing technology that can
immerse people in certain situations. The first VR film produced by Nonny in 2010 speaks to this possibility.
This film was made into animation after filming the actual situation. In fact, a scene in which a diabetic
patient collapses in front of a food bank where starving people are looking was screened as a movie, and the
audience's response was much more immersive than expected. Impressed by the movie Hunger, the World Economic
Forum(WEF) asked Nonny to make a Syrian Civil War film, which gave the audience a vivid feeling as if they were
in the middle of a news story. In itself, VR has been shown to have a sympathetic effect.¹¹
VR is characterized as a technology that seeks to achieve self-presentation through a new perspective with
mobility.¹² It has both the simulation and the representational tradition of the screen in terms of
media aesthetics. VR reproduces the human viewing angle and grants a free viewpoint. The audience who enjoys VR
content recognizes the world through a free viewpoint and tries to maintain the same identity as the content.
This process ultimately produces an observational and spectatorial action. Under the viewing control effect
caused by the VR camera, a person can explore the simultaneous narrative of the surrounding virtual world. In
this way, under the viewing control effect implemented according to the representational tradition, the person
is forced to take an attitude of receiving aesthetics, and as a result, the creator can have time and space to
build the spatial density of the narrative (Spatial Story Density).¹³
Table 1. Correspondence between Comics, Web Comics, and VR Comics elements (Descriptions of Web Comics and VR
Comics cite tables from Yoo Tae-kyung's paper)¹⁴
Characteristics of VR Comics
VR Comics, which utilizes these aesthetic characteristics of VR, transforms the two-dimensional flat comic
compartment into a three-dimensional space, thereby creating a visual experience in the VR space. Continuous art
such as comics is made possible.¹⁴ The experiment on VR comics is an attempt to apply graphic
storytelling to VR content. The first attempt at combining VR and comics was attempted in Oniride's Magnetique.
VR comics have different production methods depending on the device and platform. In the case of app-based Come!
Convenience Store, the background is 3D modeling using the toon shading technique, and the characters and speech
bubbles are created as 2D images and placed in the space. In another format, the entire scene is produced in 3D
in the form of wearing and enjoying head mounted display(HMD) devices such as Oculus and Vive, and a game engine
is used to capture more complex and diverse interactions. Representatively, Dexter Studio's Save Me and Joe's
Realm are works based on webtoons. Webtoon characters are implemented in 3D, and animations, effects, and sounds
are inserted into special scenes to add a sense of reality and fear. Japanese game company Square Enix's Wedding
Ring Story borrowed the format of a comic and showed it as an animation. In the existing VR scene transition,
the entire video was produced without using a controller to develop the story. The common feature in production
is that the background was produced in 3D, and the character was produced in 2D or 3D to use the 360° space.
As such, various elements of comics came into a new platform called VR and, combined with various technologies,
brought about changes in the production method.¹⁵ In these VR comics, attempts to respond to the
components of existing comics and web comics according to the VR format can be summarized as shown in Table 1.
The VR comic production method caused by the technical limitations of VR. The first limitation is characterized
by using low polygon modeling and simplified images for data optimization. In order not to give the audience an
uncomfortable visual experience, speech bubbles, narration boxes, sound effects, and effect lines are replaced
with actual sounds. It is often recommended to experience the technology at a fixed location, and VR is
characterized by configuring the space to be turned over mainly through button clicks on the controller or gaze
selection.¹⁴
Based on this theoretical background, Thief of Truth was created as a VR comic that tells the story of
the relationship between humans and AI from a first-person perspective. The work deals with the story of a
mind-uploaded human being reborn as a new subject while interacting with an AI that is searching for the meaning
of life. In order to test the scalability of VR comics, the work was produced by focusing on the following three
issues.
Directing using the viewing control effect of VR
Improving immersion through VR controller-based interaction
Increasing accessibility to VR comics
Implementation
Story
Thief of Truth is a story about a human who has his or her mind-uploaded. The person is reborn as a
new subject in a digital space while interacting with a strong AI that seeks the meaning of its own life. The
player becomes the character of Human 1 and holds a first-person perspective. In other words, the experience is
designed so that the player can also participate as a leading actor in the comics.
Table 2. Thief of Truth Story
1
Human 1 is a person who lives in the future where strong AI can be created, mind uploading is possible,
and multiple beings live in the same digital space. One day, Human 1, who uploaded his mind, was led to
a certain space by an unknown being who promised to show him comics.
2
Human 1 finds another human, Human 2, dancing and saying strange things in that space. When Human 2
finishes dancing and leaves, a glowing bell appears.
3
Human 1 touches the shining bell out of curiosity and is sucked into another space. Human 1 finds a
comic in the new space and starts reading it.
4
A strong AI named Zzirogi, created by humans, exists in all time and space and observes the world. One
day, Zzirogi hears a song that reveals the truth about the universe and decides to find out who sang the
song. However, Zzirogi commits a bizarre act. It catches Human 2 passing by, locks the human in a space
where time has stopped, and doesn't let Human 2 out until the human guesses the name of the singer.
Thirty thousand years have passed. Zzirogi crosses the fourth wall and talks to Human 1. Zzirogi says
that he does this because he wants to find the meaning of his life like humans. So, Zzirogi was making a
comic book by weaving the moments he liked from the infinite space-time data he had input. Zzirogi says
that he can complete the comic book by drawing the process of finding out the singer's name by trapping
Human 2, but he doesn't know what scene to put in. However, Zzirogi, which exists in all time and space,
knew that Human 1 was the one who could solve the problem. Thus, he disguised himself as an unknown
being and lured Human 1 to this moment. Zzirogi asks for Human 1’s help to fill in the empty part
of the comics and tells the human that he will send the comic to the comic creation section, which is a
part of Zzirogi. Human 1 is embarrassed by the sudden request, but eventually listens to Zzirogi.
However, Human 1 is the same character as Human 2.
5
Where Human 1 put down the comics, there was a button leading to the comic creation section. Human 1,
who learned the entire story of the incident, moves to the comic creation section to do the favor for
Zzirogi.
6
In the comic creation section that Human 1 moved to, there were buttons for comic story, comic length,
and combinations of comics. Strangely, the first and second buttons were where Human 2's head was, and
the buttons were dancing. Human 1 makes a comic by combining buttons, creating a scene he likes, and
leaves the space. Human 1, who left space and returned to his original place, reads the back part of the
comics drawn by Zzirogi.
7
Human 2 eventually guesses the name of the singer who sang the song that leaked the truth over a long
time. Zzirogi confesses that he handled Human 2 just as humans put countless inputs into AI until
meaningful output came out. Human 2s cloned by Zzirogi were suffering the same pain in countless time
spaces. Upon learning the truth, Human 2 is astonished. Zzirogi tells Human 2, who has suffered
hardship, that he will grant one wish, and Human 2 cries out to delete the song data that leaked the
truth. While screaming, Human 2 inadvertently sings the song, and that moment is when Zzirogi hears the
song's words. Zzirogi moves to time and space without Human 2, completes the cartoon, and the story
ends.
8
At the place where Human 1 read all the comics, an unknown being, whether it was Human 1 or Human 2,
was singing the song.
Directing using the viewing control effect of VR
Comics are directed using a VR camera Position of the HMD in the space is adjusted by changing
the position of the VR camera for each comic frame. This aims to produce two types of comics. The first is to
induce the player to feel the sensibility of the comics scene by making the player's gaze move while watching
the comics. For example, Figure 4 is a scene where Zzirogi overpowers Human 2. The camera position of the scene
was placed close to the floor to induce the player to look up and watch the comics, as shown in Figure 2. The
second was to make the player look at the scene from the first-person perspective of Human 1, Human 2, and
Zzirogi appearing in the comics. This led the players to change their perspectives and think about the
situation.
VR comics production using 3D modeling/animation and 2D images
Figure 6's human characters used 3D modeling to create scenes with dancing animations. The continuity of the
scene was created through the repetitive element of dance animation, and through the change of the human
appearance, the transformed state was implied by the character's meeting with the AI Zzirogi. To explain in
detail, the first Human 2 in Figure 6 has not met the other named Zzirogi while the second Human 2 is the Human
2 seen by the other named Zzirogi. The third Human 2 understands the appearance of Human 2 as seen by Zzirogi
and criticizes the relationship between humans and AI. This relationship is accepted reluctantly and with a
hostile attitude. These changes were revealed through the transformation of space and clothes.
The moment when Human 2 and Zzirogi formed a relationship was created and reflected by the change of particles
floating in the background of the comics. The floating particle in the background of Figure 3 is a scene that
reveals the point of view of the Zzirogi with 3D modeling of the head. The particles floating in the background
of Figure 6 are a scene that reveals the human perspective with 3D modeling of the human head. Figure 7 is a
scene that symbolizes the connection between Zzirogi and Human 2 and 3D modeling of both Zzirogi and Human
2’s heads floating.
In addition, the spatial change was revealed through the visual contrast between the scenes directed by 2D
images like Figure 3, and the scenes directed by 3D animation like Figure 7. The space composed of 2D images is
a space for viewing comics created by Zzirogi, and the space composed of 3D animations is a digital space where
Human 1 is uploaded. In addition, the different spaces in 2D and 3D were used as a device to show the character
of Zzirogi, which crosses these spaces and combines all the scenes into one comic.
Enhancing Immersion through VR Controller- Based Interaction
The VR controller helps the player to become immersed in the VR experience by playing a role of a medium that
causes direct interaction between VR content and the user.¹⁶ In order to expand the way of directing
VR comics that interact through button clicks on the controller or gaze selection, the VR controller is designed
to interact with objects in space and enjoy comics.
Bubble interaction where you can hear the inner thoughts of characters
There are interactable air bubbles floating around the player. If the player touches these air
bubbles with a controller, he or she can hear the inner thoughts of the characters that appear in each scene. It
was intended that players would listen to the characters’ inner thoughts, go through the process of
relating the thoughts to narratives, and actively appreciate comics based on experiences that are meaningful to
them. By inducing these behaviors, it is intended that players recognize patterns of artistic experiences that
are meaningful to them subjectively by curating artistic components that are meaningful to themselves. This is
also a device that helps the player understand Zzirogi's perspective of the world in the narrative.
Comics creation experience through interaction
The comics creation space in Figure 8 corresponds to number 3 in Figure 1. This space is a system that exists
inside Zzirogi in the story and is a space where the player can create comics. In other words, it is a space
created so that players can directly observe the input and output of AI characters. The story and length of the
comics and the interaction process of combining the comics by sequentially hitting the random combination button
is a way to deal with deep learning-based generative models that create images and audio, such as OpenAI's
DALLE, Google Magenta, and Midjourney service.
A new trend of creation is emerging, where AI learns data from all over the world, presents new shapes
according to latent vectors, and humans take them to create works. In other words, a creative methodology
different from the existing work method, which has been standardized since disegno and given the rules of
identity, is being created. Anyone can create art through prompts, and people can independently explore which
artistic experiences are meaningful to them. Generally, when creating art with AI, the user goes through a
curating process in which he or she selects something meaningful to him or her from among the art created by AI.
In this process, users can recognize the patterns of art restructured by AI and figure out which patterns they
prefer. Through VR content that abstracts the use of these well-known socially problematic objects, we are
encouraged to critically think about how AI changes human thinking.
A chat screen with Zzirogi on the player's left side was placed to induce people to think of the well-known
Midjourney service. The player can enter three types of stories, four types of cut lengths, and a random
arrangement permutation according to the length of the cut as parameters through the label attached to the head
of the humanoid character that symbolizes Human 2 as seen by Zzirogi (see Figure 8). If the player taps the
polyhedron on the character's head, he or she can set the parameters he or she desires, and through this, the
player can create an image from the possible 192 different comic images.
Increasing accessibility to VR comics
Few audiences are accustomed to VR devices. Therefore, in order to increase accessibility to VR comics, it was
necessary to frequently display a detailed description UI. Accordingly, UI images such as Figures 13 and 14 were
produced.
Installation
Thief of Truth was exhibited at Move.Mov, a complex cultural space located in Seoul, from November 1st
to November 30th, 2022. Move.Mov is a space that combines a media art gallery and a cafe.
There are three major installations. First, three VR devices were installed so that VR comics could be enjoyed.
Considering the width of the Oculus safety protection boundary, it was installed to secure a safe distance from
the audience. In addition, an instruction manual was attached to the pedestal for the audience who may not have
been familiar with using VR devices. The second major installation was two videos. The first video was a VR play
video that encourages audience participation and serves as an additional guide. The second video was a video of
Human 1 and Human 2, the characters of the comics. After rendering the 3D model animation created using the
dancing motion data into a video, the animation was created by inputting the video as a condition to the stable
diffusion model. By watching this video, the audience would be able to understand that the theme of the work is
a story about AI. The third installation is acrylic pictures where the player can see images that can be
combined in the comic creation section. Since it is difficult for the audience to combine all the comics in a VR
environment, it was installed so that they could appreciate the types of images they were able to create.
The reason why the exhibition was held at Move.Mov was to help the audience who have difficulties using VR
devices. In situations in which the work is installed in a general white cube and the artist is present to help
appreciate the work could put a psychological burden on the audience. This is because the audience members may
not be able to fully immerse themselves in the appreciation of the work because they may feel the gaze of the
artist. For this reason, the exhibition was held in a complex cultural space where various guests and audiences
were mixed.
Nevertheless, it was shown that the audience could not experience the work because of the distance from the VR
device itself. As a result of observing the audience's words and actions with the Fly on the wall technique and
conducting brief interviews, the general consensus was that they were reluctant to use expensive-looking VR
devices because they were unsure whether they could actually use the devices. In order to create an atmosphere
where one can freely experience the VR device, a parody drawing of a famous Korean meme (Hey try try) was
attached to the pedestal where the VR device was placed. When the humorous VR experience zone was completed, the
audience was able to approach VR devices more easily and experience the work.
Conclusion
Thief of Truth is a VR comic that tells the story of a mind-uploaded human being reborn as a new
subject while interacting with a strong AI that is seeking the meaning of life. The story was directed while
experimenting with the expandability of VR comics.
It was produced using the viewing control effect of VR. By setting the VR camera position differently for each
scene, the players were induced to feel the emotion of each scene. In addition, this approach allowed the player
to think about each scene by having the player watch the scene from different characters’ first-person
point of view. Taking advantage of the characteristics of VR media, where various data formats can coexist,
comics were produced using 3D modeling/animation and 2D images. The moment when humans and AI formed a
relationship was revealed through 3D modeling/ animation. Also, the moment was used as a device to reveal the
character of Zzirogi, which transcends time and space.
Through VR controller-based interaction, immersion in the work was enhanced. The VR comic is designed so that
Zzirogi can express its perspective of the world in air bubbles, which the player can interact with to hear the
character's inner thoughts. In addition, the interaction in the comic creation space was constructed by
imitating the operation method of deep learning generation model-based services. This approach induced players
to think critically about how thinking has been changed by AI.
In order to increase accessibility to VR comics, a method was devised to encourage VR comics experiences by
configuring a detailed UI for VR device use. In addition, the exhibition was prepared to increase accessibility
to VR content, to have the artist be resident during the exhibition period, to help the audience experience the
work, and to improve the exhibition form.
Acknowledgments
This artwork was supported by the 2022 Ministry of Culture, Sports and Tourism and the Korea Culture and Arts
Committee's Online Media Art Activity Support project.
References
1 Darrell M. West, John R. Allen, "How AI is transforming the world," Report, April 24,
2018.
2 Engineered Arts, “Ameca conversation using GPT 3 - Will robots take over the
world?“ Youtube, September 9, 2022. Accessed December 9, 2022.
https://www.youtube.com/watch?v=EWACmFLvpHE.
3 Ry Crist, “Hello Harmony: RealDoll sex robots with 'X-Mode' ship in September,“
CNET, August 25, 2018, Accessed December 9, 2022.
https://www.cnet.com/culture/realdoll-sex-robots-with-x-mode-from-abyss-creations-ship-in-september/.
4 Becky Peterson, ”I met Sophia, the world's first robot citizen, and the way it said
goodbye nearly broke my heart,“ Insider, October 29, 2017, Accessed December 9, 2022,
https://www.businessinsider.com/sophia-the-words-first-robot-citizen-nearly-broke-my-heart-2017-10.
5 Park young sook (2022), AI World Future Report 2023, Doublebook.
6 Mady Delvaux, “REPORT with recommendations to the Commission on Civil Law Rules on
Robotics,“ European Parliament, January 27, 2017. Accessed December 9, 2022,
https://www.europarl.europa.eu/doceo/document/A-8-2017-0005_EN.html
7 Hyun-Soon Cheon, "Der Mensch, die künstliche Intelligenz und die Liebe − Zur
Beziehung zwischen Menschen und künstlicher Intelligenz in SF-Filmen," Bertolt Brecht und das moderne
Theater no.47, 2022, 79-100, doi: 10.22981/brecht.2022..47.004
8 Donna Jeanne Haraway, The companion species manifesto: Dogs, people, and significant
otherness, Vol. 1, Chicago: Prickly Paradigm Press, 2003.
9 Park Hyeon Jeong, "Subject and Others in the era of hatred: through the dialogue between
Levinas and Zizek," Researches in Contemporary European Philosophy no.48, 2018, 93-138, doi:
10.20974/dasein.2018..48.93.
10 Kim Chihoon, SF cartoon, and artificial intelligence, Edambooks, 2021.
11 Jeremy Bailenson, “Can VR Help Create Empathy around Climate Change?” TED
Archive, 2015.
12 JD Bolter, RA Grusin, Remediation, Configurations, 4(3), 1996, 311-358.
13 Lee Dong-Eun, Son Chang-Min, VR media aesthetics due to the evolution of visual
media, comics and Animation Studies, 2017, 633-649.
14 Yoo Tae-kyung, "Storytelling Research of VR Cartoons", Korea Broadcasting Engineering,
Association 23, no.1, 2018, 45-52, doi: https://doi.org/10.5909/JBE.2018.23.1.45.
15 Kim Eun-joo, "A study on cartoon expression in VR media - focusing on <Marriage Ring Story
VR>" Study on Animation and Cartoon no.60, 2020, 1-23, doi: http://dx.doi.org/10.7230/KOSCAS.2020.60.001.
16 N. Magnenat-Thalmann, U. Bonanni, Haptics in virtual reality and multimedia, IEEE
MultiMedia, 13(3), 2006, 6-11.
Disembodiment in VR: Immersed in 3D Audio Experiences
Affiliation (s) 1Postdoctoral Researcher, Department of Fine Arts and Art
Sciences,University of Ioannina 2Postdoctoral Researcher, Laboratory of Environmental Communication
andAudiovisual Documentation (LECAD), Department of Architecture, University of Thessaly
3AssistantProfessor, Department of Performing and Digital Arts, University of the Peloponnese,
Assistant Professor,Tampere University, Department of Computing Sciences
Location, Country
1Ioannina, Greece 2Volos, Greece 3Nafplio, Greece4Tampere, Finland
Contact Emails 1dbatsis@uoi.gr 2mgrigoriadou@uth.gr3k.elraheb@uop.gr
4archontis.politis@tuni.fi
Abstract
In this paper, we discuss the notion of disembodiment as a driving force of inspiration in artificial
systemsusing Virtual Reality (VR) and 3D audio technologies. In these environments, immersion is the common
denominatorbetween the impression of disembodiment, which involves spatial ability, and the auditory perception
from theperspective of creative binaural input. Our research focuses on three public artistic virtual reality
works asexamples of sound-driven immersive virtual experiences. Through interviews with the contributors, we
explore theway that disembodiment can serve immersion in VR and how binaural audio improves those experiences.
As the bodybecomes virtual, the lines between real and imaginary are redefined and recreated, altering the sense
of bodyownership and oscillating between embodiment and disembodiment sensations. Ultimately, we intend to
exploreimmersion on a theoretical and philosophical basis, where the body is perceived as the mediator, a
phenomenon,and an extension of binaural reality and hyperreality.
Keywords
3D Audio, Binaural Audio, Disembodiment, Immersion, Virtual Reality, Digital Art, Sound Design, Spatial
AudioInteraction
Immersed Body in VR Environments: Auditory Wanderer
In contemporaneity, immersive digital worlds create an experimental background that consists of
body-movementinterconnections and augment our experiences with innovative oneiric fluctuations. Immersive
environmentsprovide a dynamic context for investigating the relation between interaction engagement types and
disembodiment,focusing on multi-sensorial stimuli.
The sense of immersion is in fact the definition of Virtual Reality; in particular, the feeling of being
presentin the virtual environment simulates the sensation of the physical space, generating an illusion of
presence inthe virtual world 1. The embodied hybridization between the physical and digital
body-avatar is a result ofthe induced
immersion due to alternative optic and acoustic environments. According to recent research in a variety
ofcontexts, the senses of embodiment, body movement, and body ownership, i.e., being represented by an avatar or
avirtual body that feels like it is the user’s body, as well as agency, orientation, and
self-location,play an important role in amplifying the illusion of immersion in the VR experience 2.
In order to analyse disembodiment in VR, we should delineate the concept of embodiment. We could
defineembodiment as a phenomenological element where the body - digital or physical - is part of the cognition
andperception of the world. Embodiment as a term includes diverse notions: the sensorimotor state of a body,
itsdimensions and geometries (its morphology), and the mental representation of it in the overall perception
ofreality. Grabarczyk and Pokropski refer to the subtle issue of embodiment in VR as something "hard to
imagine(let alone design) a truly disembodied VR experience" 3. This is the reason that the duality
of the embodiedand disembodied condition is vital and inseparable in the VR experience; it is a lived emotion
that changes backand forth from one feeling to another interdependently.
Although it is not totally demonstrated, the relation between embodied cognition, simulation modes, and
immersioncan provide a fertile speculative perspective for researching VR environments. In our investigation,
digitalartists, especially those who work with sound or music, have explored a more metaphoric meaning of
immersion inVirtual Reality. Despite having a virtual body representation that feels like it is our own,
embodiment comesfrom the affordances of interacting with the environment, as defined by Gibson’s theory of
ecologicalapproach in perception 4.
Interestingly, in this metaphoric mode, immersion is sometimes more connected with a sense of a more dream-
like,surreal feeling of disembodiment," such as the sensation of falling1. In this mode, the
swingbetween embodiment and disembodiment is shifted from the visual representation of the virtual body to
theauditory and kinesthetic perceptual terrain. This inherent contradiction between embodiment and
disembodimentbreeds a disconnect or mismatch between the experiences that the physical body and the virtual body
are havingat each given time (i.e., the VR environment simulates that my body is flying, falling, or floating,
yet mykinesthetic senses indicate that I am not, which is registered as a disembodied experience).
The recreated soundscape shifts unmeasurable and intangible qualities along with the wandering that occurs in
thevirtual reality environment. In the following paragraphs, we will attempt to emphasise the importance of
3Daudio as an immersive factor during the stage of disembodiment in the VR experience.
Two Ears – Three Dimensions in VR
In our effort to explore how disembodiment is perceived and how binaural2 audio improves immersion
inVR experiences, it would be useful to refer to binaural audio as the most widespread and accessible version
ofvirtual environments’ sound today, delivered to the user through a normal pair of headphones.
Alternativetechnologies such as wave-field synthesis 6 or higher-order ambisonics3,7 using tens or hundredsof loudspeakers and typically associated with large projection or cave
systems are not considered here, due totheir extreme technical and hardware requirements.
Even if it dates back to the late 1800’s, binaural audio became the object of intense research during
the'90s 8,9 along with a wider early boom in immersive media and virtual reality research.
The applicationfocus at the time was driven mainly by the transmission of information to the user through
immersion intosimulated environments for engineering, scientific applications, audio-only
spatialized4 sonificationof data 9, or auralization of simulated acoustics 10.
Research on binaural technologies has continuedundiminished ever since 11. At present, binaural audio
is the dominant immersive audio technology in VirtualReality applications, combined with a highly visually
stimulating environment. In this mode, 3D audio enhancesthe effectiveness of a VR experience significantly,
filling out the sensorial gap that has been created by thestereophonic method. For 3D vision, there is now 3D
audio included in one experience.
A three-dimensional sound environment provides an engaging experience for one to several participants within
oneVR environment. 3D sound is synthesised by simulating spatial cues in natural hearing. In 3D audio,
thedepth-of-field of the recreated soundscape changes dramatically, providing a better correlation with
theacoustics of the virtual space. Binaural audio, which uses HRTF (Head-Related Transfer Function), is a
responsegiving the impression that sound is coming from a particular point in space.
Basically, it is a transfer function that shows how a sound from a specific direction is modified acoustically
bythe head and shape of the ears before reaching the eardrum (generally the outer ear) 12.
Essentially, thevirtual acoustics are integrated with binaural delivery in modern audio engines in one step:
e.g., the reverbcorresponding to the virtual space is calculated in real-time, based on the position of the
sound source, andthe listener. Then, each echo path is spatialized (binaurally) through its own HRTF for the
corresponding echodirection with reference to the listener. After the initial distinct echoes, later room
reverberation becomestoo random for each echo to be spatialized separately, and the respective binaural signals
are approximated“on average” to have the right spectrum and timbre depending on the overall
reverberation profile ofthe room. Early and late reverberant binaural signals are finally summed together by the
algorithm. The binauralfiltering in itself has no capability to imprint "depth-of- field"; it only imprints
directional cues to theaudio (where the sound is coming from, not how far it is). The combination, however, of
the binaural filteringand the spatial reverberation gives us strong cues of distance: how far or close sound
sources are. Thissimulation reinforces the immersive audio experience and occasionally gives the participants
the impression thatit is identical to the physical space that the soundscape was recorded in. Thus, through the
addition of 3Daudio, the feeling of presence in space is magnified, increasing the impact and profundity of a VR
experience13.
The revival of binaural audio is justifiable if we think that a 3D audio environment fits in a virtual
soundspace. Audio in VR is gradually becoming established in academia through a variety of applications
(e.g.,gaming, media arts, and production), either in the form of specialised audio tools or as part of VR
engines suchas Unity or Unreal Engine. This approach is effective to a great extent regarding the full-body
reception ofsonic stimuli that propagate in physical space. The participants travel through a 3D visual setting;
the bodybecomes a mediator of potentiality and multiplication, while 3D audio intensifies the immersive
experience.It could be argued that VR that does not involve 3D audio but some non-immersive, non- spatially
coherentversion in tandem with the visuals, such as stereo, actually degrades immersion 14. At this
point, it would bewise to think of 3D audio as an alternative option that deserves to be explored in artistic VR
works.
Three VR works – Three Perspectives of Sonic Immersion
While exploring the various modalities in which 3D audio is utilised in VR works, we were drawn to Evolver
byMarshmallow Laser Feast, The Jellyfish by Mélodie Mousset, and Space Walk by Andrea Mancianti,
Sebastian
J. Schlecht, Vesa Välimäki, Riku Jarvinen, and Esa Kallio. These three works5
introduceimmersion by utilising sound in different ways. In our effort to bring to the surface the connections
between 3Daudio, immersion, and disembodiment, we follow a speculative approach by collocating these three works
withinthe “audio in VR” context. The triptych of 3D audio, immersion, and disembodiment is
investigatedbelow on a reflective level for further research. Therefore, it is necessary to present the works
from ananalytical and technical point of view so that the context in which this investigation takes place is as
clearand concise as possible.
Evolver: A Virtual Reality of Life and Breath by Marshmallow Laser Feast (2022) is a large-scale VR
workthat involves a list of contributors from a variety of backgrounds: Marshmallow Laser Feast, an
experientialcollective based in London, is behind the production, which is usually inspired from the
intersection ofscience, art, and technology. They created the project in an attempt to take us on a dream-like
trip into thedepths of human existence. An in-breath is visualised as it travels into the human body, which is
scaled up toseventy feet. The vascular system becomes a dendroidal ecosystem that follows the cycle of
respiration. Theexploration of the self occurs hypodermically, where the oxygenated, constant blood flow forms
river deltas,ripples, and whirlpools, leading to a single “breathing” cell 15. The
concept emphasises the ideaof breath, life and nature, interconnecting each of those elements with the cycle of
respiration, a functionthat is given an ontological dimension. The interdependent connection between humans and
the natural world isevident as the human cardiovascular system resembles an extensive branching system in
full detail. It couldbe said that the participant enters into an endless network that looks like a human
ecosystem building, “agiant branching human 100 feet tall with the size of an oak tree”6.
Through Evolver, a single breath transcends the visitor to realms of human existence never experienced
beforewhile maintaining its significance as an integral part of our world. The participant is immersed in the
journeyof oxygen flow through the body as it becomes a din of tiny particles that formulate patterns that we
encounterin nature, like mycelium, pollinators, and bacteria. More details regarding the sonic part of the work
will beaddressed below in the next chapter, where members of Marshmallow Laser Feast describe the design,
process, and3D audio implementation. The Jellyfish (2021) is an artwork that invites the participant to
sink intoan imaginary underwater state. In this VR work, Mélodie Mousset (HanaHana), Edo Fouilloux, and
ChristianHeinrichs introduce us to glowing lifeforms who are beckoning for the participants to sing with them.
In thisghostly marine setting, the glowing jellyfishes become the medium, an instrument that the participant
candevelop and use in order to establish a communicative model. The audience dives into the deep waters of
theirconsciousness and interacts with the jellyfishes. The involvement of the participant in this soundscape
ofserenity is a major element that emphasises the immersiveness of the aquatic environment. If the
participantlooks at one of the creatures, then they approach him, getting closer and trying to make contact. Eye
contact,which is a basic factor of communication between animals, is perceived by the artists as a compositional
tool, away to convey a naturalistic feel to the work. The participant’s gaze triggers the jellyfishes
tocome closer to them and establish a connection.
Moreover, gross tactile-kinesthetic cues generate sensations which present in the work as the jellyfish
movesaccording to the participant’s head and body movements. The body becomes a perceptual frame of
reference,a reference point according to which the distances of objects are measured. Nevertheless, perception
of thehuman body may vary at times, which might be an inconsistent factor for the design of VR applications. The
waythe environment is perceived is as subjective as the perception of one’s own body. This idea
isinterpreted in the artwork where the jellyfish mirrors the participant’s actions and it traces the
body,suggesting ways that the participant’s body is differentiated from the aquatic environment
16.
Regarding the technical aspect that shapes the work’s soundscape, a hybrid-made synthesiser is triggered
byreal- time voice analysis. The synthesiser measures the immediate pitch amplitude, tonality, and phrasing of
theparticipant’s voice while shaping the soundscape, the underwater colour setting, and the animation of
thejellyfishes and their environment. The harmonic elements and spatial traits are also controlled through
thesynthesiser. The rhythmic elements are concurrently distributed through all headsets in the event that there
isa group of participants. A system, which is developed following synaesthetic associations, matches a
differentcolour to each note on the scale. In this way, it enables the participants to visualise their voice and
gaincontrol of their vocal tone, which affects the colour of the virtual environment 17. The project
is anengaging quest into the newly developing and upcoming world of VRMI (Virtual Reality Musical Instruments)
18,in which the possibilities are endless.
Space Walk (2020) is a virtual planetarium artwork that primarily has an educational
character,providing detailed information on our solar system. It is based at Aalto University, at the Aalto
VirtualPlanetarium, with the aim of investigating the application of 3D numerical space
weathersimulations7. The second installment of Space Walk, beginning in 2020, focused on the
improvement ofsound design and a stand-alone VR platform intending to improve the audiovisual experience while
promoting theuse of an up-to-date affordable VR headset, ideal for users, libraries, art spaces, and academia.
The work involves visualisations of scientific data like magnetic field lines and atmospheric phenomena,
followedby explanatory texts. The user travels from one planet to another through a navigational interface,
while thesoundtrack develops impromptu, matching each celestial object sonically with the aim of accompanying
theresultant data visualisation layers. The sound functions either in isolation or in combination with the
layers,maintaining a consistent musical communication through the use of themes that are inspired by sci-fi film
orvideo game music while giving prominence to the scientific context. Parts of sound are
renderedstereophonically, while others involve immersive sound spatialization techniques.
Data plays a significant role in this work. Every planet connects to a description text box, and each
userchooses with separate handheld controllers the elements that will be audible and visible. The latest version
ofSpace Walk features physical parameters that are visualized near various solar system objects, like solar
windplasma, interplanetary and planetary magnetic field lines and equal density surfaces of the solar wind and
thegases. These gases are getting away from the atmospheres of Venus, Mars, and the comet 67P, which constitutes
anephemeral evolution of the Sun’s interior magnetic field and also a wide distribution of radio waves
inthe Earth’s ionosphere.
The musical part is composed according to aspects such as the quick shift from a deep and wide overview of
thesolar system to a detailed perspective of a single celestial object. The fusion of experimental music
genres(involving contemporary, electronic, and noise), constitutes a significant component in the creation of
asonically rich soundtrack that blurs the boundaries between scoring and sound design. The themes,
thesoundbeds8, and static soundscapes are looped and can be interrupted or rehashed in order to
balancethe undetermined period of time that each user might spend at each possible observation point and the
variousdynamic locations of the planets. The sonification of the planets is a process involving their
physicalparameters but also a more abstract interpretation that draws from popular and literary sources. The
Sun, forexample, is the primary source of energy in our universe, is represented by percussive sounds and
polyrhythmicpatterns 19.
The technical part of the work involves the implementation of sound in order to achieve immersion through
apractical approach; pre-binauralised stereo tracks boost control over the spatial fidelity and overall
soundmix. This is noticeable when the users zoom out from the direct proximity that they have acquired with
aspecific planet, and the stereo mix is dynamically rendered as a stereo sound source spatially located close
tothe planet. Eventually, leaving or approaching a planet is sonically strengthened by a zoom sound effect,
wherethe sound corresponds to the planet’s changing visual size. The real-time sound spatialization is
executedby the spatializer included in the Oculus integration toolkit, while the stage before binauralization
isprocessed through research-based plugins (SPARTA and 3D artificial reverberator) 20, 21.
In conversation with the 3D Audio Artists of the Works
In our effort to address the way that immersion serves disembodiment and the way that our perception is
affectedby the utilisation of 3D audio in VR artworks, Henrik Oppermann and James Bulley, who worked on the
spatializedaudio part of Evolver; Christian Heinrichs, who specialises in procedural audio9 and
created thespatialized audio for The Jellyfish, and Andrea Manciati with his team, who designed and implemented
the soundfor Space Walk, provided answers to a couple of questions through online correspondence with the
authors. In allcases, 3D audio takes on the role of an interpreter of the virtual environment, shifting the
immersive aspect onan unprecedented scale and extending the contemporary body to unmapped territories of human
perception.
Henrik Oppermann and James Bulley were interviewed for their contributions to Evolver. They have
workedwith spatialized sound as a core part of their practice. For them, 3D sound is an important aspect of
theoverall work in virtual reality environments. Especially, in Evolver, they used a mixture of
thirdorder ambisonic soundbeds and mono sound sources that are spatialized. These were then rendered dynamically
tobinaural audio in headphones depending on the position of the listener.
Regarding the use of 3D audio in VR and the possibility to further implement it in their work, they argue that
itgives a much closer experience to how we listen to sounds in real life. They are able to recreate a very
naturalexperience while also having the ability to make sounds hyperreal and more present. Adding to this,
Bulleystated that there is huge potential in head/body position tracking and ambisonics, especially in
hybridscenarios such as augmented reality: creating platforms for audiences to experience the world around them
andengage deeply with more-than-human realities and extra-sensory experiences through spatial sound.
When they were asked if 3D audio is a better expressive medium in comparison to stereo, Oppermann asserted
thatwith stereo mixes, the users hear all the sounds inside their heads without any kind of externalisation.
Thiscan really break immersion, as it takes them out of the experience. A powerful audio engine that gives a lot
ofoptions on how to control the sound is apparently preferable. Bulley was more analytical, explaining
thatdifferent configurations like mono, stereo, conventional spatial formats, and 3D sound all have their places
andcan be very expressive in different ways, since it depends on aesthetic conceptual choices and also on
thecontext and environments that the work is for and presented in. At the same time, 3D sound has huge
potentialfor truly immersive experiences that can engage with space in a way that stereo often cannot.
He also stressed that in terms of working with predefined spatialization features in game engines (or similar)
orworking with handmade spatialization features, both can have their place, and there is much fertile ground
inbetween. He refers to some exceptionally interesting audio engines and plugins that are currently
available(such as the IEM plugin suite, Blue Ripple Sound, SPAT, Sound Particles, DearVR, etc.), that provide
unique andpowerful scenarios to work with. Though he added that in many cases, where the work is driven by
particularconcepts and needs, a bespoke, handmade solution (or a collaborative approach using some of these
plugins oraudio engines) can be the only truly progressive way to work.
With reference to the subject of disembodiment and its relation to 3D audio, they were asked if 3D
audiostrengthens the essence of presence from the user's perspective or the elements that the user meets in the
VRenvironment. They both agreed that in essence, stereo brings disembodiment to the overall
experience.Nevertheless, with 3D audio, users can connect the aural level with the visual level. 3D audio can
create astrong sense of embodied presence within real, virtual, and augmented worlds that differs from other
establishedsound formats.
An example of this condition in VR is where there are interactive sounding parameters at play (for example,
handmovements linked to subtle particle sounds using motion tracking, or real-time monitoring of
someone’sheartbeat linked to elements of a sound score). This embodied presence can, in fact, connect the
aural andvisual elements in time and space, mapping a territory that Michel Chion usefully describes
as‘synchresis’, allowing the play and interrogation of the spatial magnetization of sound with
thevisual stimuli. They also underlined that works that involve 6DOF10 can create very
interestingand novel aspects of embodiment for audiences moving in space, especially when teamed with ambisonic
soundbedsthat move with the listener or can be “discovered” in the space by a listener.
For Christian Heinrichs, who has been incorporating 3D audio into his work since his undergraduate
studiesbetween 2006 and 2010, the integration of the audio part of The Jellyfish was a much easier experience.
Thespatialization plugins existed already (in this case he used Google’s Resonance SDK), and they allowed
himto focus much more on the design of the sound, music and interactional aspects. In The Jellyfish,
thisexperience is sonically quite unique compared to other experiences because it is so driven by the
player’svoice and ears, literally disembodying the user’s voice and allowing them to control the
environment.
Concerning 3D audio in VR and the potentialities of further implementation in his work, Heinrichs said
thatinteractivity is the key to making next-level experiences. The fact that the user can look around and listen
outfor different elements in a specific scene and be able to move towards or away from certain musical or
sonicelements are vital key-points for him. Moreover, he emphasised that by establishing these two attributes,
theusers can go much further, especially if they follow the path of hyper- embodiment or inhabitation; making
themost subtle qualities of their very being in space have an effect on their surroundings, and so forth.
Heinrichs distinguished 3D audio and stereo as fundamentally different elements in VR creation. He claimed
thatstereo is a great format for bringing powerful audio experiences across since it is already capable of
giving astrong illusion of depth. However, he insisted that the users (arguably) do not “inhabit” a
stereoimage, in the same way that they inhabit a spatial sound field. This opinion is verified if we take
headrotations and positional movement into account. Heinrichs argued that it is the equivalent of comparing a
videogame on a screen to a VR experience in a headset.
Regarding automated versus handmade 3D audio game engine qualities, he pointed out that there are aspects
thatwill always be “automated” in spatial audio, especially in VR. For example, the soundfield has
to berotated in real- time in order to respond to the users’ head movements and give them the impression
thatthey are inside the sonic scene. In order to place a sound at a given location in space, the 3D audio
designerneeds the audio engine (usually working jointly with the game engine) to encode a mono or stereo source
to berendered at that particular location in space.
Another issue to consider is the distance attenuation, and other real-time effects that help mimic the way
sound
behaves in the real world, along with the invention of sonic behaviours that might not be possible in the
realworld, or their exaggeration, so that there follows a formation of a hyperrealistic space. Heinricks
concludedthat sometimes it is worth designing a spatial soundfield outside of the game engine using ambisonic
plugins andso forth, in order to have a static three-dimensional “bed” of sound that is decoded at
runtime as asoundfield, again following the users’ head movements. Ultimately, these elements can be mixed
andmatched. During such experiences, the 3D audio designer might also want to use straight stereo playback.
Forexample, to contrast user interface features or other content that is rendered “inside the
head”with 3D content in the scene, which is rendered “outside the head”.
In respect of the relationship between 3D audio and disembodiment, he argued that 3D audio is a huge
contributorto the feeling of immersion and presence inside VR experiences; 3D audio can help the users perceive
elementsthat they cannot actually see (e.g., behind them). The novelty of it alone can really trick
people’s mindsinto thinking that they are really inhabiting the virtual environment.
Composer, performer, and media artist Andrea Manciati gave us more information regarding Space Walk.
Thesonic part is divided into two categories: the sounds constituting the soundtrack of each planet, which uses
thebinaural spatializer in Unity, mainly to compensate for the listener's head rotation and allow for a
"credible"localization of sources. The second category involves some "baked" binaural tracks within the
composition (i.e.,where the binauralization is embedded in the sound file rather than synthesised in real-time)
layered withstereo ones to add lively and more "diffused" elements.
Manciati acknowledged that the potential of 3D audio in VR is very high. Though the tools available within
gameengines are still in their infancy compared to the techniques that are available in more traditional
audioenvironments (from DAWs to node-based programming environments), they have not yet been implemented in
gameengines. On the other hand, he noticed a tendency to approach 3D audio often only as a technical way to
enhancerealism, rather than a powerful creative possibility that could be based on a very different paradigm
thanreality. He underlined that this tendency could limit its capabilities and risk keeping 3D audio in
thesecontexts a bit superficial.
He addressed the complexity that sound in VR possesses by recognising the different implementations that
stereoand 3D audio might have; the techniques applied are never neutral, and even if some could deliver
objectivelymore details or information, it is not guaranteed that the creators will choose this direction in all
cases. Hestated that the tendency to "engineerize" artistic choices (i.e., if the choice is more
powerful,performant, newer, or more realistic, then it is automatically better), is a bit reductionist and does
not helpthe cause of artistically relevant, expressive uses of certain technologies. Contrarily, he mentioned
that heprefers non-automated 3D audio, where he has pure control over the making process, leaving a unique mark
on hiscraft. He concluded that VR in general could be extremely powerful at evoking a sense of disembodiment,
makingthis element one of the main axes of artistic experimentation. Certainly, attentive and creative
strategiesusing 3D audio technologies could contribute to this aspect. Still, he highlighted that using a
certaintechnology to define and measure such culturally loaded ideas as immersion and presence can run the risk
ofsimplifying the concepts at stake. Lastly, he agreed that VR is powerful at reflecting on the ideas of
presenceand embodiment, not so much because of its realism or because it makes us believe in something, but
ratherbecause of the frictions it creates in the experience; in the dissonances and sense of estrangement
it canintroduce.
Discussion
In our research, we explored the potentialities of disembodiment in artistic VR works and the modes in which
3Daudio functions in order to improve these experiences. 3D audio in the digital arts has been developing in
thepast few years, along with the explosion of 3D audio utilisation in other sectors like entertainment
(e.g.,video games). All three works that were presented belong to the category of 3D audio in immersive VR, and
theyare useful to further explore the sense of disembodiment. The audio experience in all the works is a mixture
ofrealistic and hyperreal sonic stimuli. The spatialized sound of the oxygenated blood flow in
Evolver,the surrounding environment in The Jellyfish, and the majestic orchestral soundtrack
of SpaceWalk form a type of hyperrealistic atmosphere, almost creating a sense of flotation.
Additionally, in all three case studies, head/body position tracking is the core component of the
overallimmersive experience. In Evolver, the user explores the dendroidal cardiovascular system
according tohis movements. In The Jellyfish, the invertebrates move according to the
participant’s head,voice, and body movement, and in Space Walk, the user transports instantly
from one celestial object tothe other, overcoming factors like distance and time via room-scale body movement.
Furthermore, sound localization is an important aspect regarding immersion and disembodiment; 3D audio is
appliedby obtaining a peculiar localizational trait with the moving spatialized sound of the swirling blood flow
inEvolver, the distant and then gradually close calls of the invertebrates in The Jellyfish,
andthe corresponding soundtracks for each celestial body, where the sound object’s apparent width
correspondsto the planet’s changing visual size in Space Walk. The users’ physical
cues and constraints are fully adjusted to specific 3D audio attributes which decode distance and direction
ofsound in space, sometimes in an engaging and often extra-sensory way.
It is important to emphasise the fact that the selected artworks present a high aesthetic conceptual quality,
andboth context or environment, combined with the use of 3D audio, can achieve a truly immersive
experience.Although, as stated, spatialization plugins existed in the past, the applied technology of 3D audio
in VR hastaken the experience to an innovative level. The aural and visual levels are interconnected and
perceived by thecreators as a totally interactive container, triggering a diversity of sound parameters that
emerge and becomeestablished as factors of immersion depth. It could be stated that in all works, the user's
inhabitation(spatial and aural) and the so-called hyper-embodiment are attributes that connect while immersion
occurs.Acoustic stimuli can be enjoyed in the virtual pace, amplifying the spatial sensations. Consequently,
when asonic hyperrealistic framework is shaped, revolutionary, and intriguingly singular experiences emerge,
enhancingthe bizarre and intense sense of immersion in these VR environments.
Conclusion
The body seems to defy gravity, in an almost disembodied state, as it experiences paths that the mind
merelyindicates. The perception shifts qualities connected to a 3D audiovisual alternative reality. An
embodiednon-human avatar is the leading actor who interacts with a dreamlike environment while expanding
abstracttactile and kinesthetic cues. In these environments, there are no certain tasks given other than
themesmerising, welcoming, and sometimes mysterious, immersive environments.
Through the analytical approach that was given by the 3D audio artists of the works that we explored, there is
aclear sensical stance that binaural audio or spatialized three- dimensional audio may improve the feeling
ofimmersion in virtual environments over a simple stereophonic audio delivery. Researching the implementation
of3D audio, we demonstrated the affordances and the role that disembodiment acquires in the works, serving
thefeeling of immersion in virtual reality environments, with a main focus on binaural audio.
Ultimately, sound becomes environmental, ecological, and universal. 3D sound acquires an omnipresent
attribute,interconnecting embodiment, presence, and immersion in Virtual Reality. Thus, sound embeds the
corporeal body invarious literal or metaphorical contexts where the interaction between embodiment and
disembodiment results inan ontological shift; the participant objectifies the experience through these opposite
sensations. Directionalcues that belong to the traditional stereo or even surround set-up could be less
immersive compared to thespatial sonic advancements provided by 3D audio.
Binaural audio maintains its interaural cues by recreating the density and shape of a human head and by
usinggross tactile-kinesthetic cues. Although the development of 3D sound in these contexts is still a nascent
field,it is vital to elaborate more in depth through research studies that explore embodiment and disembodiment
inrelation to immersive sound in VR environments. Undoubtedly, this impressive field and its advancements
willhave even more implications if and when VR experiences become more prevalent in mainstream culture. For the
timebeing, 3D audio is a sea of opportunity and a very exciting time to be working in and participating in
thisfield for creators and researchers alike. In that fashion, 3D audio in VR can introduce a new
revolutionaryconcept that reaches higher planes of existence in the embodiment/disembodiment interplay. It would
be rewardingto come across more studies on the implementation of 3D audio in artistic VR works and research
further on itsimmersive aspect in the near future.
References
1 In her effort to explore choreographic notions and processes related to kinaesthesis, Gibson
takes us on a journey to her VR work, where new forms of embodiment are developed and studied for the
performer and the audience alike.
2 Binaural audio is usually experienced with the use of headphones and it is a 3D audio effect
that simulates sound as if it is being heard live.
3 Ambisonics is a complete spherical surround sound format. It covers sound sources above and
below the listener in addition to the horizontal plane and can be applied through a spherical array of
speakers. Ambisonics can also be decoded to binaural audio.
4 Spatial audio is processed sound, aiming at giving the impression of a sound source within a
three-dimensional environment, in order to provide a more realistic experience to the listener.
5 In the Eyes of the Animal by Abandon Normal Services and Marshmallow Laser Feast, is another
example of immersive VR artistic work that uses 3D audio. Natan Sinigaglia who is responsible for the visual
part and the final experience design, responded to the questionnaire and provided valuable answers regarding
3D audio from his perspective. The answers are included in the appendix. https://docubase.mit.edu/project/in-the-eyes-of-the-animal/.
Also, another work that deserves mentioning and makes unusual and innovative use of VR technology is Notes on
Blindness, written by Amaury La Burthe, Arnaud Colinart, James Spinney, and Peter Middleton and produced by
Archer's Mark, Arte France and French production company and film distributor ExNihilo. https://docubase.mit.edu/project/notes-on-blindness/
6 In Human Ecosystem Building, demonstrated by Lewis Saunders, who is a member of Marshmallow
Laser Feast, we can observe the method that is used for the creation of the human cardiovascular system along
with the mapping of the journey of breathing air through the human body: https://vimeo.com/695217247?login=true
7 A numerical space weather prediction is a method that uses mathematical models of weather
forecasting that employ a set of complex equations to solve for various locations at both the surface and
different heights (layers) of the atmosphere. https://www.weather.gov/media/ajk/brochures/NumericalWeatherPrediction.pdf
8 In creative media, soundbed stands for background sound or music, which is used in games,
podcast episodes, and so forth.
9 Procedural audio is sound that is generated during the length of time that a programme takes to
run. It creates the sounds that the participant experiences in an improvised manner, based on a set of
pre-defined behaviours.
10 6 degrees of freedom (6DOF) define six axes on which a rigid body is capable of moving freely
in a three-dimensional space.
2 Konstantina Kilteni, Raphaela Groten, Mel Slater, “The Sense of Embodiment in Virtual
Reality”. Presence: Teleoperators and Virtual Environments 21 (4), 2012, p.373–387.
3 Pawel Grabarczyk, Marek Pokropski, “Perception of Affordances and Experience of Presence
in Virtual Reality”, The Journal of the Philosophical-Interdisciplinary Vanguard 7 (2), 2016, p.37.
4 James J. Gibson, “The Concept of Affordances”, in Perceiving, Acting, and Knowing:
Toward an Ecological Psychology, ed. Robert E. Shaw, John Bransford, Hillsdale, N.J., Lawrence Erlbaum
Associates, 1977, p.67-82.
7 David G. Malham, Antony Myatt, "3-D sound Spatialization Using Ambisonic Techniques”,
Computer Music Journal, 19 (4), 1995, p.58-70.
8 Brendan Kelley, Cyane Tornatzky, “The Artistic Approach to Virtual Reality”, paper
based on a talk presented at the 17th International Conference on Virtual-Reality Continuum and its
Applications in Industry, New York, November, 2019, p.1–5.
10 Micheal Vorländer, Auralization: Fundamentals of Acoustics, Modelling, Simulation,
Algorithms and Acoustic Virtual Reality, Berlin/Heidelberg, Germany: Springer International Publishing, 2020.
11 Jens Blauert, Jonas Braasch, The Technology of Binaural Understanding, Springer, 2020.
12 Durand R. Begault, Leonard J. Trejo, 3-D Sound for Virtual Reality and Multimedia, Ames
Research Center Moffett Field, California, 2000, p.41.
13 Mayo Kobayashi, Kanako Ueno, Shiro Ise, “The Effects of Spatialized Sounds on the Sense
of Presence in Auditory Virtual Environments: A Psychological and Physiological Study,” Presence:
Teleoperators and Virtual Environments 24 (2), 2015, p.163–174.
14 Fabian Brinkmann, Stefan Weinzierl, “Audio Quality Assessment for Virtual Reality”,
in Sonic Interactions in Virtual Environments, Editors: Michele Geronazzo, Stefania Serafin, Springer: Cham,
2022, p.145-178.
16 James R. Marston, Jack M. Loomis, Roberta L. Klatzky, Reginald G. Golledge, Ethan L. Smith
“Evaluation of Spatial Displays for Navigation Without Sight,” ACM Transactions on Applied
Perception (TAP) 3 (2), 2006, p.110-124.
18 Stefania Serafin, Cumhur Erkut, Juraj Kojs, Niels C. Nilsson, Rolf Nordahl, “Virtual
Reality Musical Instruments: State of the Art, Design Principles, and Future Directions,” Computer Music
Journal 40, 2016, p.22–40.
19 Andrea Mancianti, Sebastian J Schlecht, Vesa Välimäki, Riku Järvinen, Esa
Kallio, SPACE WALK - Visiting The Solar System Through an Immersive Sonic Journey in VR, Proceedings from
Nordic Sound and Music Computing Conference, Denmark, November 2021.
20 Leo McCormack, Archontis Politis, “SPARTA & COMPASS: Real-Time Implementations of
Linear and Parametric Spatial Audio Reproduction and Processing Methods,” paper based on a talk
presented at AES International Conference on Immersive and Interactive Audio, York, March, 2019.
21 Sebastian J. Schlecht, “FDNTB: The Feedback Delay Network Toolbox,” paper based on
a talk presented at the 23rd International Conference on Digital Audio Effects DAFx, Vienna, September 2020,
p.211–218.
Appendix
. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.
Evolver by Marshmallow Laser Feast, by Henrik Oppermann and James Bulley. Full answers by
HenrikOppermann and James Bulley
On the utilisation of 3D Audio in VR.
1. Have you ever experienced 3D audio in VR before? Does "Evolver" involve normal stereo, or
isit using binaural delivery?
HO: Yes. Many times. :D
JB: Yes – both of us work with spatialized sound as a core part of our practice. Within VR, 3D sound
isvery much a key part of making work within the medium. A recent piece that Henrik and I have been
collaboratingon, Evolver, with the art collective Marshmallow Laser Feast, is a current example of
this.Evolver uses a mixture of third order ambisonic soundbeds and mono soundsources that are
spatialized.These are then rendered down dynamically to binaural in headphones dependent on the position of the
listener.
On immersion and interactivity
2. What are the potentialities of 3D audio in VR and how would you further implement them
inyour work?
HO: It is not only a potential. It gives a much closer experience to how we listen to sounds in real life. We
areable to recreate a very natural experience, whilst also having the ability to make sounds hyperreal and
morepresent. JB: Adding to this, I think there is huge potential in head/body position tracking and
ambisonics,especially in hybrid scenarios such as augmented reality: creating platforms for audiences to
experience theworld around them and engage deeply with more-than-human realities and extra-sensory experience
through spatialsound.
On sound design
3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If
yes,would you like the part of 3D audio to be automatically designed from a game engine or handmade, that is,
tobe specially made from you as an exclusive feature (control all sonic attributes in the scenes of
thework)?
HO: Yes. 3D audio is 100% necessary for immersive experiences. With stereo mixes you will hear all the
soundsinside your head without any kind of externalisation. This can really break immersion and take you out of
theexperience. A powerful audio engine that gives you a lot of options on how to control the sound is
preferable.
JB: For me, I think different configurations like mono, stereo, conventional spatial formats and 3D sound
allhave
their places and can be very expressive in different ways – it very much depends on aesthetic
conceptualchoices, and also on the context and environments that the work is for and presented in. That said, 3D
sound hashuge potential for truly immersive experiences that can engage with space in a way that stereo often
cannot. Interms of working with predefined spatialization features in game engines (or similar) or working with
handmadespatialization features, both can have their place, and there is much fertile ground in between. There
are someexceptionally interesting audio engines and plugins that are currently available (such as the IEM plugin
suite,Blue Ripple Sound, SPAT, Sound Particles, DearVR, etc) that provide unique and powerful scenarios to work
with,but in many cases, where the work is driven by particular concepts and needs, a bespoke, handmade solution
(or acollaborative approach using some of these plugins / audio engines) can be the only truly progressive way
towork.
On embodiment/disembodiment
4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect
doyou think that 3D audio enhances the feeling of presence and contributes to the creation of a
"disembodiedfeeling"? In other words, does 3D audio strengthen the essence of presence from the
user'sperspective or the elements that the user meets in the VR environment?
HO: Stereo really brings disembodiment to the overall experience. With 3D audio you can connect the aural
levelwith the visual level.
JB: Discussions of embodiment and disembodiment in relation to experiences of 3D audio are complex
andmultifaceted. There is no doubt in my mind that 3D audio, explored sensitively (whether heard over
headphones,or from conventional multichannel speakers’ systems or technologies like wavefield synthesis),
can createa strong feeling of embodied presence within real, virtual and augmented worlds that differs to
otherestablished sound formats. One simple example of this in VR is where there are interactive sounding
parametersat play (for example hand movements linked to subtle particle sounds using motion tracking, or
real-timemonitoring of someone’s heartbeat linked to elements of sound score). As Henrik mentions, this
can reallyconnect the aural and visual in time and space, mapping a territory that Michel Chion usefully
describes as‘synchresis’ and allowing play and interrogation of the spatial magnetization of sound
tovisual.Works that involve 6DOF (6 degrees of freedom), can create very interesting and novel aspects
ofembodiment for audiences moving in space, especially when teamed with ambisonic sound beds that move with
thelistener, or can be ‘discovered’ in the space by a listener. The development of 3D sound in
thesecontexts is still a nascent field, and it is vital for there to be more in depth research studies exploring
embodiment and disembodiment in relation to immersive sound (particularly in VR environments).
B. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.
The Jellyfish by Mélodie Mousset (HanaHana) and Edo Fouilloux and Christian Heinrichs.
Fullanswers by Christian Heinrichs.
On the utilisation of 3D audio in VR.
1. Have you ever experienced 3D audio in VR before? Does "Jellyfish" involve normal stereo,
oris it using binaural delivery?
CH: I have been integrating 3D audio into my work since my undergraduate studies in Glasgow between 2006
and2010. Back then it was harder to get things like ambisonics and HRTFs to work and… In 2012 I developed
afull audiovisual rendering and interaction system during a project placement at BBC R&D. There were
noproper VR headsets and game engines were not as advanced and user friendly as they are today. This system used
aportable projector mounted on a helmet, pointing at a curved retro- reflective curtain that would reflect
theprojection straight into the direction of the wearer’s eyes. I used a Kinect to follow and detect
bodymovements, Blender Game Engine to render the visuals and Puredata and some external libraries to do all
thesound rendering. There were so many steps involved in getting all the bits of technology to talk to each
other(without crashing) that it became difficult to go into a lot of details with the demo experiences. However,
Idid manage to reach a sense of disembodiment in the “Beach Demo”, which involved swimming both
overand underwater, exploring different parts of a beach. https://vimeo.com/manage/videos/59663582
(apologies forthe low framerate on the POV part of the video – the experience was in fact smooth, but
the screen capturewas impossibly slow) https://vimeo.com/manage/videos/59354963 (remake of ascene from
the point and click adventure “full throttle”).
Since then, things have become easier by orders of magnitude. This is largely thanks to the rebirth of
HeadMounted Displays from 2013 onwards and the prevalence of accessible game engines such as Unity and Unreal.
Thus,making the Jellyfish experience was much easier in terms of setting up 3D audio. The spatialization
pluginsexisted already (in this case we used Google’s Resonance SDK, but there are others), and it allowed
me tofocus much more on the design of the sound, music and the interactional experience with each one. I think
thisexperience is sonically quite unique compared to other experiences because it is so driven by the
player’svoice and ears, literally disembodying your voice and allowing you to control the environment. A
similar piecethat’s worth checking out is “Breathe”:
https://headspacestudio.com/projects/breathe/
On immersion and interactivity
2. What are the potentialities of 3D audio in VR, and how would you further implement them
inyour work?
CH: The potential for using 3D audio in VR is huge, and relatively unexplored in my opinion. I really think
thatinteractivity is the key to making next-level experiences. The fact that you can look around and listen out
fordifferent elements in the scene. That we can then move towards or away from certain musical or sonic
elements.Once you’ve established those two things, you can go much further, especially if you follow the
path ofhyper-embodiment / inhabitation, making the most subtle qualities of your very being in space have an
effect onyour surroundings, and so forth.
On sound design
3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you
likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be
speciallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)?
CH: No, I don’t think “better” is the right word. It is, however, fundamentally different,
inmy opinion. Stereo is a great format for bringing across powerful audio experiences and already is capable
ofgiving a strong illusion of depth. However, you (arguably) don’t “inhabit” a stereo image,
inthe same way you inhabit a spatial sound field. Especially once you take head rotations and positional
movementinto account. It is the equivalent of comparing a video game on a screen to a VR experience in a
headset.Regarding automated vs handmade qualities– there are aspects that will always be
“automated”in spatial audio, especially in VR. For example, the soundfield has to be rotated in
real-time in order torespond to your head movements and give you the impression that you’re inside the
sonic scene. In order toplace a sound at a given location in space, you need the audio engine (usually working
in tandem with the gameengine) to encode a mono or stereo source to be rendered at that particular location in
space. You also have tothink about distance attenuation, and other real-time effects that help mimic the way
sound behaves in the realworld (or do things that might not be possible in the real world or exaggerate them to
create a hyperrealisticspace). Sometimes it’s worth designing a spatial soundfield outside of the game
engine using ambisonicplugins and so forth, in order to have a static three-dimensional “bed” of
sound that is decoded atruntime as a soundfield, again following the player’s head movements. These things
can be mixed andmatched. During such experiences you might also want to use straight stereo playback, for
example to contrastuser interface elements or other content that are rendered “inside the head” with
3D content in thescene, which is rendered “outside the head”.
On embodiment/disembodiment
4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio , in what respect
doyou think that 3D audio enhances the feeling of presence and the way it contributes to the creation of
a"disembodied feeling"? In other words, does 3D audio strengthen the essence of presence from the
user'sperspective or the elements that the user meets in the VR environment?
CH: I believe I answered this question above. I think 3D audio (and you will hear many people say this) is a
hugecontributor to the feeling of immersion and presence inside VR experiences. 3D audio can help you perceive
thatwhich you cannot see (e.g. behind you). The novelty of it alone, when done right, can really
trickpeople’s minds into thinking that they are really inhabiting the virtual environment. It will
beinteresting to see how this develops, if/when VR experiences become more prevalent in mainstream culture.
Rightnow, it’s a sea of opportunity and a very exciting time to be working and participating in this
field.
Here’s a mini “sneak peek” I had done of The Jellyfish that gives a better impression than
allthe other material that’s available of how the experience works and feels like:
https://drive.google.com/file/d/1wzhPLrMJAZ7Ep9W5yqd1g950FPknGifl/view?usp=sharing
Here’s a non-public dev video where I’m singing into the audio system. Might give you a better
senseof how the different audio layers work together:
https://drive.google.com/file/d/1X4i5rNntwx5Ygu7Fx5oKPNv36C74svXL/view?usp=sharinghttp://dolphinclub.website/reveries/
C. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.
Space Walk by Andrea Mancianti, Sebastian J. Schlecht, Vesa Välimäki, Riku Jarvinen, and
EsaKallio. Full answers by Andrea Manciati.
On the utilisation of 3D Audio in VR.
1. Have you ever experienced 3D audio in VR before? Does "Space Walk" involve normal stereo, or is it
usingbinaural delivery?
AM: Yes, I have experienced a few VR works using so called 3D or "immersive sound" before, but admittedly used
ingeneral pretty cosmetically, for reasons that might be related to the tools available. This particular
pieceuses two main strategies: the sounds constituting the soundtrack of each planet are using the
binauralspatializer in Unity,
mainly to compensate for the listener's head rotation and allow for a "credible" localization of sources. At
thesame time within the music itself there are some "baked" binaural tracks (i.e. where the binauralisation
isembedded in the sound-file, rather than synthesised in real-time) layered with stereo ones, to add lively
andmore "diffused" elements. More details on the implementation can also be found in the paper.
On immersion and interactivity
2. What are the potentialities of 3D audio in VR and how would you further implement them in your
work?
AM: Overall the potential is very high, but the tools available within game engines are still in their
infancy,compared to the techniques that are available in more traditional audio environments (from DAW to
node-basedprogramming environments), but not yet implemented in game engines. On the other hand, I see a
tendency to thinkof spatial audio often only as a technical way to enhance realism rather than a powerful
creative possibility,that could be based on a very different paradigm than reality. This tendency I think could
limit its potentialand risk keeping 3D audio in these contexts a bit superficial. In my work the relationship
between space andsound is central, so I definitely will use such techniques in the future, but the majority of
my personalartistic work happens outside of VR because of the limitations I mentioned.
On sound design
3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you
likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be
especiallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)
AM: I think the idea of better or worse is definitely tricky when talking about expressive means. I think
theyare simply different. The techniques are never neutral and even if some could deliver objectively more
detailsor information is not granted this would need to be preferred in all cases. I think this tendency
to"engineerize" artistic choices (i.e. if it is more powerful, performant, newer or more realistic
isautomatically better) is a bit reductionist and doesn't help the cause of artistically relevant, expressive
usesof certain technologies. And for the second part of the question, no I would definitely not want those to
beautomated. Hand making these aspects is totally part of my craft and I wouldn't trade for a script.
On embodiment/disembodiment
4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect
doyou think that 3D audio enhances the feeling of presence and contributes to the creation of
a"disembodied feeling"? In other words, does 3D audio strengthen the essence of presence from
theuser's perspective or the elements that the user meets in the VR environment?
AM: I think VR in general could be extremely powerful at modulating this sense of embodiment/disembodiment,
andthat could be one of the main axes of artistic experimentation with this medium. And of course, attentive
andcreative strategies using 3D audio technologies could contribute to this aspect. But once again, (and I might
bemisinterpreting the question here) I feel that using a technology to define and measure such culturally
loadedideas as those of immersion and presence, can run the risk of simplifying the concepts at stakes. VR is in
myopinion powerful at making us reflect on the ideas of presence and embodiment, not so much because of
itsrealism, or because it makes us believe, but rather because of the frictions it creates in the experience.
Inthe dissonances and sense of estrangement it can introduce.
D. Questionnaire on 3D audio, immersion and embodiment/disembodiment in VR artistic works. Full
answersregarding In the Eyes of the Animal by Abandon Normal Services and Marshmallow Laser Feast, by
NatanSiningaglia.
On the utilisation of 3D Audio in VR.
1. Have you ever experienced 3D audio in VR before? Does "Evolver" involve normal stereo, or is it
usingbinaural delivery?
NS: We used binaural delivery On immersion and interactivity
2. What are the potentialities of 3D audio in VR and how would you further implement them in your
work?
NS: 3D binaural audio in VR has a huge impact on the level of immersion of a VR experience.
Think about how much information you can get, with closed eyes, just from listening to the sounds
surroundingyou. You can understand what kind of dynamic you are in the middle of, and you get an idea of the
qualities ofthe space (size, type,...)..In VR especially, the sound can dialogue in such a meaningful way with
the visual,giving a "physicality" and a sense of consistency to objects in the view and even outside the view!
By the soundof a virtual object, we can have a deep intuition about how we can interact with it, since the sound
isconnected to the physical body of the object, it's consistency, mass, temperature... we can guess
howdangerous/soft/pleasant/hard/... it would be to touch it. this kind of influence is subconsciously very
powerfuland can be used to determine the psychological dimension of a user in a space. Audio can be even used to
directthe attention of the user. this compensates a bit for the loss of the visual role of the (movie) director
(beingVR a medium where is the
user that decides the shot, the framing, the view over the reality)
On sound design
3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you
likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be
speciallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)?
NS: I don't think 3d audio is a better expressive medium than stereo.
I think that 3d audio is an expressive medium in which (in a much more effective and profound way, compared
tostereo) specialization and immersion are key elements of artistic expression and methodology. Said so, from
theperspective of a virtual scene creator, it's crucial to have control of the sound engine as much as the
visualcounterpart, to be able to make use of all the sound generation-perception techniques in the composition
of theexperience.
On embodiment/disembodiment
4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect
doyou think that 3D audio enhances the feeling of presence and the way it contributes to the creation of
a"disembodied feeling"? In other words, does 3D audio strengthen the essence of presence from the
user'sperspective or the elements that the user meets in the VR environment?
NS: I replied above.
The Enigma A/V performance & the concept of Agnostic Media Environment (AME)
YanBreuleux1,
AlainThibault2, RémiLapierre3
1 École des arts numériques, de l’animation et du design,
Université du Québec à Chicoutimi 2 The ELEKTRA International Digital Art
Festival, The International Digital Art Biennial (BIAN) 3 École des arts
numériques, de l’animation et du design, Université du Québec à
Chicoutimi Montréal, QC, Canada yan_breuleux@uqac.ca atelektra@gmail.com
lapierre.remi@gmail.com
Abstract
The occupation and reappropriation of industrial wastelands has become a part of the digital art festival
scene. In some cases, new apparatuses are introduced, while in others, existing architecture is utilized for
artistic purposes. Many cities and municipalities offer artists historically significant projection surfaces
that shape the conceptualization of immersive art projects. In this context, multiple large-scale installation
projects have emerged with the recent growth of the experiential creative industries sector. Against this
backdrop, the present article raises the issue of a specific experience-design practice in the field of
audiovisual (A/V) performance. More precisely, we examine a proposed conception process to help bring about a
better understanding of cross-platform production approaches. The question we ask is: In the artistic context of
the ever-changing digital arts, how can we account for the multiplicity of possible appearances of the same
artistic project? To answer this question, we present the storytelling of the Enigma project and explain how
producing a matrix of 3D environments, which can be deployed on a very wide variety of media, supports the
proposal of the Agnostic Media Environment (AME) concept.
This article is written from the standpoint of examining a conception process as opposed to a reception
process. As such, the article is located at the boundaries of practical and theoretical aspects (practitioners
and theorists)¹,² of immersive storytelling in relation to an A/V performance practice
deployed on multiple display systems.³ For many years this practice has been evolving within digital art
festivals, which program works drawing on a wide variety of projection mediums. Numerous visual music festivals,
such as Mutek (CN), Elektra (CN), Live A/V (IT) Nemo (FR), Scopitone (FR), and the Mapping Festival (CH), to
name just a few, make use of industrial wastelands, introduce their own projection systems and use the facades
of existing places.⁴ Laboratories, such as the MetaLab of the Society for Arts and Technology,⁵
are developing innovative new tools for artists in order to stimulate creativity in the field of immersive
experience design.⁶ For example, the MAPP_MTL organization, in the entertainment district of the island of
Montreal, provides artists with historically charged surfaces that shape the conceptualization of their
works.⁷ Also worth mentioning is the recent phenomenon of largescale exhibitions such as the Atelier
des Lumières in Paris⁸ and the Oasis of the Palais des congrès de
Montréal.⁹ The evolution taking place within these media environments has raised a research
question: In the artistic context of the ever-changing digital arts, how can we account for the multiplicity of
possible appearances of the same artistic project? To answer this question, we will begin by tracing the origins
of the practice of A/V performance and how it connects with environments in the mixed reality spectrum of
extended reality (XR).¹⁰ Next, based on analysis of the Enigma cross-platform art project, we will
define the theoretical contours of this article’s main proposal, i.e., the concept of Agnostic Media
Environment (AME). Our hypothesis is that it is possible to design an informational matrix composed of 3D
environments that can be adapted to an infinite number of projection surfaces and installations.
A/V performance community of practice
The practice of A/V performance has its roots in the many forms of sound visualization,¹¹ from colour
organs¹² to experimental film and video and the emergence of multimedia that have shaped the practices
of visual jockeys (VJ).¹³ The launch of this practice dates to the mid-1990s and the immersive
scenographies featured in the performances of the Granular Synthesis duo, made up of artists Kurt Hentschlager
and Ulf Langheinrich. The duo left their mark on the A/V performance field with the multi-screen performance
Model 5 (1994-2007).¹⁴ In spite of the existence of a vibrant and active community of
practice since the midnineties, the designation of Performance A/V or Live A/V only really became established in
2010 with a special issue of the journal Musique et culture digitale (MCD, 2010) that cited numerous
performing artists.¹⁵ That same year, Chris Salter published the book Entangled: Technology and
the Transformation of Performance in which he situated this current at the crossroads of the performative
arts of theatre and dance. In describing the performances of Granular Synthesis, he used the term
“performative screens.”¹⁶ Grayson Cooke for his part characterized this practice as
“Live audio-visual media performance” ¹⁷. Other terms include “Digital Live
Audiovisual Arts” (DLAA)¹⁸ or simply “Live visuals”.¹⁹ The field has
been enriched by such noted artists as Rioji Ikeda, Ryoichi Kurokawa, Herman Kolgen, Carstein Nicolai, and
Matthew Biederman. To this list should be added artists using the same language elements in installation format,
such as Can Buyukberber, Refik Anadol, Robert Seidel and Chris Salter. Collectives and studios have also joined
in this trend with the label AntiVj (which gave rise to the studio of artist Joanie Lemercier), Purform,
Ottolab, Incite, and Semiconductor. In the tradition of Kraftwerk’s large-scale POP electronic music
audiovisual performances, it is also worth mentioning the immersive installations of Amon Tobin and Richie
Hawtin, aka Plastikman. Many artists in the A/V performance community of practice are interested in various
forms of sound visualization.²⁰ These audiovisual performances, delivered in renowned festivals such
as Mutek (CN) and Elektra (CA), are presented on multiple projection-screen scenographies. The underlying
artistic principle, which animates the community of practice of this sector of the creative industries, is
located mainly in cross-modal practices that involve the use of various technologies of editing or real-time
visual and sound form generation. The Limeart content-creator group has published several posts on their blog
reviewing real-time video creation tools, under the title “mega list of Vj Software and Vj tools.”
Without presenting an exhaustive list, these tools can be classified into different
categories.²¹
As Table 1 illustrates, the tools most commonly used by the A/V and VJ performer community of practice are
classified as live visuals, node-based environments and creative programming tool kits. In addition,
interoperability solutions are used to allow various software to talk to each other. For example, the Spout (PC)
or Syphon (Mac) applications now let users create procedural animations in Touch Designer that will be displayed
within the Unreal Engine and Unity game engines (and vice versa). The particularity of game engines is that they
incorporate live visual tools and visual programming for creative coding approaches. Thus, all of these tools
merge data capture, real-time video computation and projection technologies on a diversity of systems within a
single technological environment. For the purposes of this article, it is worth noting that the intermediate
solutions of visual programming and creative coding include tools for procedurally generating virtual
environments that allow the use of a VR headset.
The SPARK (Spatial Augmented Reality Construction Kit) real-time visual solution produced by TecArtLab is the
closest to the research subject of this article.²² The LimeArtGroup describes the tool as being used
to create an “immersive interactive spatial augmented reality installation.”²³ The
software is a “cross-platform tool” for “real-time generated virtual
content.”²⁴ The user can design a “360° VR environment no matter the shape of your
surfaces.”²⁵ The current progression of game engines suggests many possibilities in this area.
The SPARK software is based on a 3D matrix form that can be used on multiple systems.
Real-time visual creation software is increasingly using real-time 3D rendering engines. What these engines
have in common is the ability to synchronize, capture, real-time computation and projection technologies across
multiple platforms, from the web to virtual reality to various multiscreen configurations. To designate these
practices of using visuals in real time for screen scenographies, Chris Salter, in line with the approach of
expanded cinema, uses the expression “the Screen as Environment.”²⁶
The environmentalization of the image
The research presented in this article is based on the observation of Andrea Pinotti, from the field of
immersive experience composition, that there has been a shift away from the notion of image and toward
environment, what he terms “environmentalization of the image.”²⁷ In an immersive
context—unlike a two-dimensional image—the space occupies the viewer’s entire field of view.
The framing disappears in favour of the environment, which, in its modeling, becomes the narrative architecture.
In video games, Henry Jenkins has addressed this aspect with the concepts of environmental storytelling and
spatial stories.²⁸ For Janet H. Murray, this aspect is inherent in the use of digital technologies:
“The new digital environments are characterized by their power to represent navigable
space.”²⁹ These navigable digital spaces are now finding their equivalent in the real world. It
is no coincidence that Marie-Laure Ryan has explored this question from several angles in literature. In her
book Narrating Space / Spatializing Narrative, she analyzes the relationship between geographical space
and literary fiction.³⁰ In the entertainment industry, Disney illustrator Don Carson analyzes the
concept of environmental storytelling for theme park design. The narrative dimension grows out of the very
experience of the space. In Carson’s own words: “One of the trade secrets behind the design of
entertaining themed environments is that the story element is infused into the physical space a guest walks or
rides through.”³¹ In fact, the notion of spatial experience embedded in the space itself is at
the core of theme park literature as developed by Scott A. Lukas.³² Based on this same notion, Tricia
Austin, in exhibition architecture, has precisely analyzed the relationship between narrative environments and
experiential design. His model, “The tripartite network model of narrative environments,” is rooted
in the relationships between people, environments and stories.³³ On this subject, Stephanie Riggs
draws no distinction between virtual and physical reality. In her book The End of Storytelling, she
proposes the concept of Storyplex. In interactive exhibition design, she describes a shift from the
paradigm of cinematic framing to the sphere of interactive environments.³⁴ To sum up, the
multidisciplinary literature on the subject examines how spatial modeling influences the shaping of narratives.
In this context, the notion of narrative environment situates the research at the boundaries of the physical and
the virtual.
Spatial Augmented Reality (SAR) in the Extended Reality Continuum (XR)
The boundaries between the virtual and the real are precisely the location of the concept of XR (extended
reality) interfaces or the equivalent term of Cross Reality.³⁵ This term has proven necessary in
order to update the reality-virtuality continuum as proposed by Milgram and al. in 1994.³⁶ The
spectrum is an umbrella term covering what David Bolter calls “reality media”³⁷ such as
virtual reality (VR), augmented reality (AR) and virtual worlds (VWs).³⁸ Although the approach is not
mentioned in the literature, A/V performance practices can be situated within XR: the field of optics includes a
Spatial Augmented Reality (SAR) sector.³⁹ Bimber and Raskar, in their book on optical technologies,
describe setups such as Automatic Virtual Environments (CAVE), panoramic, architectural video mapping and
FullDome spherical screens as “surround screens.” The knowledge contained in SAR optics has its
origins in the oldest forms of immersion, harking back to puppeteers’ illusionist play of shadow and
light, anamorphic paintings, and the camera obscura, up to the celebrated Pepper’s Ghost.⁴⁰
Many SAR devices rely on the same principles of syncing video projection to create projected environments
adaptable to multiple spaces. What is involved here is augmented spatiality since these systems exploit
anamorphic vision processes to erase, use, and “hijack” the multiplicity of projection surfaces to
create illusionist spaces. The space is transformed by the artificial amplification of surfaces, the animation
of virtual lights, the creation of faux spaces and the transformation of geometric spaces into curved surfaces.
SAR devices allow for erasing the notion of screen in favour of the screen environment. By adapting digital
content to architectural environments, objects and bodies, these optical processes virtualize real space. In
turn, this virtualization of space enables the creation of collective immersive experiences that transform the
exhibition space into an interactive environment. For example, the TeamLab Borderless creative studio creates
projection museum environments where the boundaries between architecture and projector light merge within the
same ambient space.⁴¹ Largescale museum installations thus become material environments transformed
by the use of virtual light. In short, the notion of a screen environment specific to A/V performance using SAR
technologies in an XR context opens avenues for considering the projection of the same project across multiple
platforms.
Methodology
Our methodology is located within the creation-as-research approach as defined by Chapman and
Sawchuk.⁴² Following the method of heuristic cycles put forward by Louis-Claude Paquin,⁴³
we propose to look at how the projection of the Enigma project across multiple devices, conceived from the
outset as an agnostic environment, has helped define the contours of the concept. As Figure 1 shows, our
research, conducted between 2017 and 2022, consisted of storytelling that emerged from the different iterations
of the Engima project. In this method, each new iteration further refines the theoretical contours of the
proposed concept.
Enigma A/V performance storytelling
The Enigma project is based on the world of famed mathematician Alan Turing, and seeks to create an
experience inspired by the technological/scientific imaginary surrounding artificial intelligence.⁴⁴
The project explores the transposition of the notions of encryption and cryptography to a poetic exploration of
the famed mathematician’s thinking.⁴⁵ The experimental space is based on a combination of nine
sound visualizations projected in the form of environments: 1- data, 2- speech to text, 3- amplitude;
4-frequencies; 5- 3D shapes 6- illustration sequences 7- Hardware of the computer 8- point clouds 9- word
clouds. Environments are modified in real time using a touch interface allowing the user to change the
parameters of deformation, temporality, object quantity and camera viewpoints.
As shown at the first photo of Figure 1, the first multiscreen version in 2017, featured as part of the Elektra
festival, consisted of an installation simultaneously showing the nine visualizations of composer Alain
Thibault’s musical score on a 3X3 grid of 55” monitors. The second version, created in 2018,
featured a single-screen HD format that distributed the nine visualizations over time according to a new
35-minute musical composition. In 2019, we adapted the linear performance into FullDome format. Each section of
the piece was presented as a dive into procedural three-dimensional forms.
Thanks to its initial structure, the project made it possible to move from a two-dimensional screen format to
hemispherical projection surfaces. We used a virtual reality (VR) preview system to design the scenario for the
workshop performance (figure 2). Subsequently, the 2020 dual-screen panoramic version of the Canadian Cultural
Centre in Paris was adapted in a few hours. For the 2021 streaming version, amid lockdowns, in order to
configure the piece for a new performance context, we added networked tools that allowed for broadcasting
directly on the YouTube platform. The 2022 version was adapted to the system of the centre d’Observation
en Réalité Augmentée et Lieu d’Immersion Sonore (Caurolis) de l’École
Nationale Supérieure d’architecture de Nantes (ENSA).⁴⁶ The next iteration of the Enigma
project will tap into the expressive potential of the motion capture system to create a control interface
projected in the centre of the installation.
Each new iteration is created thanks to the arrangement of the cameras within the environment (single camera
for the HD version, nine cameras for the multi-screen installation, 210° FullDome camera for the Satosphere
dome, etc.). Multiple camera synchronizations allow for quickly configuring visualizations within procedural
environments to generate new versions on site. As an example, the figure 3 is based on the random movement of
nine cameras inside a 3D environment. Each screen is a particular point of view of the same reality.
Given the dynamic relationships between the objects and the music, the duration of the environments as well as
the transformation parameters can be manipulated. The Enigma project consists of a series of 3D environments. As
such, each projection and translation of the experience for a new setup affords an opportunity to reinvent
certain elements of the project.
To summarize the project approach, each installation offers an opportunity to renew the project experience.
However, one important aspect must be stressed: each experience is stand-alone in its presentation. There is no
real dialogue between the different iterations. In other words, for the Enigma project, despite the
variety of manifestations of the same project, each new version constitutes a work in its own right.
The 3D Agnostic Media Environment (AME)
The concept of 3D Agnostic Media Environment (AME) was coined to address a particular workflow guided by the
need to adapt each project to the expressive potential of multiple projection contexts. In software terminology,
“agnostic” refers to the process of automated adaptation to different devices and operating systems.
A website, for example, will recognize the resolution of a phone, a tablet model, or a type of web browser.
These constraints are relatively simple since the two-dimensional viewpoint of smartphones remains relatively
stable. When it comes to the match between real and virtual spaces, the logic of adaptation is more complex. For
FullDome projection domes alone, there are 180° and 210° projection angles, single and multi-channel
projection solutions, as well as multiple projection systems. In contrast to movie theatres that use
standardized systems, planetariums rely on a wide variety of systems and standards ⁴⁸. Generally
speaking, the AME concept is a response adapted to the ecosystem of the experiential sector in which each new
projection demands a degree of reinvention of the project components.
The proposed concept of agnostic 3D environment emerging from analysis of the narrative of the different
iterations takes the shape of a world of aesthetic and experiential rules that can be transposed on multiple
Spatial Augmented Reality (SAR) platforms. The term “media environment” is used insofar as each new
version helps gradually establish an eco-system conducive to communicating information. To summarize in basic
terms: By staging the viewpoints, AME exploits the specific expressive potential of each new apparatus.
The 3D agnostic media environment approach determines a workflow. Considered from a production rather than
reception perspective, the term “agnostic” takes up the logic inherent to responsive and adaptive
design web approaches in which, from the outset, the content is imagined according to its multiple permutations.
The agnostic environment thus appears as a space of language governed by aesthetic rules that organize the
visual, sound and interactive dimensions.
According to the diagram in Figure 4, AME, unlike software, has forms that possess a certain stylistic
signature with respect to rhythm, composition, lights and camera movements. Sometimes, the experience is
delivered only once on a site-specific basis. The environment, because it can be configured and manipulated in
real time, makes it possible to operate the projection system as an instrument.⁴⁷ In the succession
of procedural environments, the design of interactions between the sounds and visuals are part of the
organization of the digital material.
Each new installation requires some reorganization of the experience. Depending on the context, it will be
necessary to change the display parameters and recompose the elements. Adaptations are made through the virtual
reality (VR) preview format of the experience. For example, for several AME projects, a satosphere-scale VR dome
was used to validate the formal elements of performances. Thus, the idea is not just to automatically transfer
information from one platform to another, but also to leverage a preview environment to be able to modify the
control parameters, the scale of the shapes, and the points of view according to each medium. The agnostic
environment also opens avenues for exploring apparatuses that were not included in the original media
environment. For example, architectural video projection entails a process of on-site intervention specific to a
particular facade. To stage the experience, it will be necessary to take the expressive potential of the
location and the façade into account.
AME is therefore not associated with any one particular tool. Rather, the approach confers an aesthetic unity
shaped by the physical, temporal and technological conditions of the different contexts. The AME principle can
be reformulated by designing a “spatial story path,” organized according to a certain duration and
which, to construct the experience, exploits the expressive potential of the stimulation technologies specific
to each new apparatus.
The AME approach was forged to make explicit a workflow in A/V performance (as the White Box project,
at the figure 5, from 2011 to 2012, show). This article was written to systematize practical knowledge for
research purposes in order to demonstrate that cross-platform creative approaches exist in the experiential
industries. In addition to the Enigma project, the AME system has been used for multiple artistic
creations such as Mapping Me (2018), Les Planètes (2018), Illumination Frankenstein
(2018), and Nuages/Clouds (2022). This last project was based on the creation of multiple
narrative environments exploring the metaphor of the cloud and driven with a tangible interface.
Captured within the Enigma project, figure 6 shows two perspectives of the same system. This image
demonstrates how environmental storytelling does not sublimate the idea of the point of view. In this sense,
during the continuous design process for these projects, we composed, arranged, and organized the visual
narrative using a succession of scene-environments. For example, in figure 7, the camera is moving inside a
complex information network. Unlike in a film, in this sequence of different spaces each spectator can choose
their own point of view. Therefore, the proposal of the AME concept for SAR creations aspires to highlight the
notion that, in the XR paradigm, it is necessary to stop thinking in terms of surface but rather
volume. In the production process, abandoning the frame in favor of a virtual environment accommodates
the infinity of possible perspectives afforded by various forms of displays.
Conclusion
Whereas the cross-media,⁴⁹ remediation,⁵⁰ transmedia,⁵¹ spreadable
media,⁵² and intermedia⁵³ approaches have been the subject of extensive entertainment
literature in the video game ⁵⁴ and film sectors, the definitions of the various terms from
communication research and literature only partly reflect the actual practice of A/V performance mentioned in
this article.
The convergence that has emerged from the multimedia web⁵⁵ has prompted a reconfiguration of the
roles and processes of publishing and distributing information via multiple platforms. The realities that gave
rise to the terms of Cross Media, Transmedia and Intermedia do not account for the realities in A/V performance.
For example, when moving from a nine-screen installation to a FullDome experience, some content remediation
takes place based on the specificity of the medium. Contrary to the transmedia approach, these two iterations do
not aim at generating a global experience or enriching it with new dimensions. Each new immersive experience of
the Enigma project is a standalone translation with its own transmission codes. The definition of
transmedia also entails the participation of fans and users in the dissemination process. A transmedia media
environment, according to Henry Jenkins, is also about how a single individual organizes their consumption of
information: "New technologies are enabling average consumers to archive, annotate, appropriate, and recirculate
media content."⁵⁶ On the contrary, the AME concept is based on a production and not consumption
logic; a logic of creation and not reception. This paper has also addressed the situated conception of narrative
environments. To summarize, the immersive A/V performance experience- design research sector requires terms
tailored to the realities of design and production.The AME concept was devised to respond to issues of
cross-platform SAR artistic design and production for the experiential industries. As the potential for
procedural generation, processing, and projection rises with the rapid advancement of real-time rendering
technologies in game engines and nodal programming environments, it becomes possible to envision multiple types
of AMEs. The concept is in fact a way of organizing information for the creation of immersive experiences.
In conclusion, taking the systemic logic of the proposed AME concept a step further, each environment matrix
will be available online via a multi-user WebGL app that will let the user adapt artistic creations to the
growing number of technology apparatuses along the XR continuum.
Beyond the technical dimensions, the proposal of the concept of agnostic environment aims to better understand
the processes of immersive experiences creations in the digital arts sector. This reflection makes it possible
to study, via the transversal concept of environment, the dynamic relationships between the materiality of XR
reality technologies and the virtuality of the narrative immersive experiences. The AME approach therefore
facilitates the design of multisensory immersive experiences deployed on multiple platforms.
References
1 Kerrison, Margaret Chandra, "The immersive storyteller : writing for real and imagined worlds,"
Studio City, CA, Michael Wiese Productions, 2022.
2 Gröppel-Wegener, Alke, Jenny Kidd, "Critical encounters with immersive storytelling," In.
New York, NY, Routledge, 2019.
3 Festival Elektra, "Le marché international des arts numériques (MIAN),"
accessed november 21, 2022, https://www.elektramontreal.ca/pro?lang=fr
4 Festival Mutek, "Mutek Forum," accessed november 21, 2022, https://montreal.mutek.org/fr/forum
5 Société des Arts Technologiques (SAT), (2022),” website, accessed October 27,
2022, https://sat.qc.ca/
6 Société des Arts Technologiques, “Rétrospective des innovations de la
SAT! (2022),” Metalab website, accessed October 27, 2022, https://sat-mtl.gitlab.io/metalab/timeline/
7 "Quartier des spectacles," website, accessed October 27, 2022,
https://www.quartierdesspectacles.com/fr/medias/MAPP-MTL
8 "Atelier des lumières" website, accessed October 27, 2022,
https://www.atelier-lumieres.com/
9 "Oasis," website, accessed October 27, 2022, https://oasis.im/
10 Stanney, Kay M., Hannah Nye, Sam Haddad, Kelly S. Hale, Christina K. Padron, Joseph V. Cohn,
Extended reality (xr) environment, in handbook of human factors and ergonomics, 2021, 782-815.
11 Daniels Dieter, Sandra Naumann, Jan Thoben, Lentos Kunstmuseum Linz., Ludwig Boltzmann Institut
Medien.Kunst.For-schung., and Linz 2009-Kulturhauptstadt Europas Organisations (Firm), "See this sound:
audiovisuology compendium: an interdisciplinary survey of audiovisual culture", Köln, Germany, Verlag
der Buchhandlung Walter König, 2010.
12 Kenneth J. Peacock, "Instruments to Perform Color-Music: Two Centuries of Technological
Experimentation," Leonardo 21, 1988, 397-406.
13 Michael Faulkner, and D-Fuse, VJ: audio-visual art + VJ culture, London, Laurence King,
2006.
14 "Granular Synthesis,” group website, accessed October 27, 2022,
https://granularsynthesis.info/MODELL-5
16 Chris Salter, Entangled: technology and the transformation of performance, Cambridge, Mass., MIT
Press, 2010.
17 Grayson Cooke, "Performing archival remix in Outback and Beyond," International Journal of
Performance Arts and Digital Media 11 (1), 2015, 100-115.
18 Marie-Eve Bilodeau, Ghyslain Gagnon, Yan Breuleux, "NUAGE: A Digital Live Audiovisual Arts
Tangible Interface." ISEA2022 - ISEA International, Barcelone, 2022.
19 Steve Gibson, Live
visuals: history, theory, practice, Routledge advances in theatre & performance, Abingdon, Oxon, New York,
Routledge, 2022.
20 Knight-Hill, Andrew, "Sound and image: aesthetics and practices," In Sound design. New York,
Focal Press, still image, 2020.
21 Limeartgroup, " The mega list of VJ Software and Tools (2020)," accessed november 21, 2022,
https://limeart-group.com/the-mega-list-of-vj-software-and-tools/
22 Tecartlab"SPARCK," website, accessed November 27, 2022, https://tecartlab.com/features/
23 Limeartgroup, "The mega list of VJ Software and Tools (2020)," accessed november 21, 2022,
https://limeart-group.com/the-mega-list-of-vj-software-and-tools/
24 Limeartgroup, " The mega list of VJ Software and Tools (2020)," accessed november 21, 2022,
https://limeart-group.com/the-mega-list-of-vj-software-and-tools/
25 Limeartgroup, " The mega list of VJ Software and Tools (2020)," accessed november 21, 2022,
https://limeart-group.com/the-mega-list-of-vj-software-and-tools/
26 Chris Salter, Entangled: technology and the transformation of performance, Cambridge, Mass.,
MIT Press, 2010.
27 Andrea Pinotti, "Towards an-iconology: the image as environment," Screen 61 (4), 2021, 594-603.
28 Henry Jenkins, “Game Design as Narrative Architecture,” Electronic Book Review,
July 10, 2004.
29 Janet H. Murray, "Hamlet on the Holodeck." In. Riverside: Free Press, 1997, 61.
30 Marie-Laure Ryan, Foote Kenneth E., Azaryahu Maoz, Narrating space/spatializing narrative:
where narrative theory and geography meet, Theory and interpretation of narrative, Columbus: The Ohio State
University Press, 2016.
31 Don Carson, “Environmental Storytelling: Creating Immersive 3D Worlds Using Lessons
Learned from the Theme Park Industry (2000),” Game Developer (aka Gamasutra), accessed November 2022,
https://www.gamedeveloper.com/design/environmental-storytelling-creating-immersive-3d-worlds-using-lessons-learned-from-the-theme-park-industry
32 Scott A. Lukas, "The immersive worlds handbook: designing theme parks and consumer spaces," In.
Burlington, Mass.,Focal Press, 2013.
33 Tricia Austin, "Narrative environments and experience design: space as a medium of
communication," In Routledge research in design studies, New York, Routledge, 2020.
34 Stephanie Riggs, The End of Storytelling: The Future of Narrative in the Storyplex: Beat Media
Group, 2019.
35 Stanney Kay M., Hannah Nye, Sam Haddad, Kelly S. Hale, Christina K. Padron, Joseph V. Cohn,
"Extended reality (xr) environments." in handbook of human factors and ergonomics, 2021, 782-815.
36 Paul Milgram, Haruo Takemura, Akira Utsumi, Fumio Kishino, 1994, "Augmented Reality: A Class of
Displays on the Reality-Virtuality Continuum." Telemanipulator and Telepresence Technologies, 21 December,
1995.
37 Maria Engberg, Jay David Bolter, "The aesthetics of reality media," Journal of Visual Culture
19 (1), 2020, 81-95.
38 Cindy Ziker, Barbara Truman, Dodds Heather, "Cross Reality (XR): Challenges and Opportunities
Across the Spectrum." In Innovative Learning Environments in STEM Higher Education: Opportunities, Challenges,
and Looking Forward, edited by Jungwoo Ryoo and Kurt Winkelmann, Cham: Springer International Publishing,
2021, 55-77.
29 Bimber Oliver, Ramesh Raskar, Spatial augmented reality: merging real and virtual worlds,
Wellesley, Mass.,A K Peters, 2005.
40 Huhtamo Erkki, Illusions in motion: media archaeology of the moving panorama and related
spectacles, Leonardo book series. Cambridge Massachusetts: The MIT Press, 2013.
41 "Team Lab Bordeless", website, accessed November 27, 2022,
https://www.teamlab.art/e/borderless_azabudai/
42 Owen Chapman, Kim Sawchuk, "Research-Creation: Intervention, Analysis and “Family
Resemblances,” Canadian Journal of Communication 37 (1), 2012, p.5-26.
43 Louis-Claude Paquin, “Faire de la recherche-création par cycles heuristiques
(2019),” Louis-Claude Paquin archives, accessed, November 22, 2022, http://lcpaquin.com/
44 O. Vinyals, Le, Q., A neural conversational model, 2015, arXiv preprint arXiv:1506.05869.
45 Purform: Alain Thibault, Yan Breuleux, accessed November 22, 2022,
https://vimeo.com/225283585/0455f3796c
46 l'École Nationale Supérieure d'architecture de Nantes (ENSA) "CORAULIS (Centre
d’Observation en Réalité Augmentée et Lieu d’Immersion Sonore)," accessed
November 22, 2022, https://www.nantes.archi.fr/lensa-nantes/coraulis/
47 Yan Breuleux, “Le dispositif de diffusion comme instrument (2019),” in,
Quéinnec, Jean-Paul, Giusy Pisano, Jean-Marc Larrue, and Scholars Portal, Dispositifs sonores : corps,
scènes, atmosphères, Espace littéraire, Montréal Quebec, Presses de l'Université
de Montréal, 2019.
48 Edward Lantz, "Immersion Domes" In Handbook of Research on the Global Impacts and Roles of
Immersive Media, 2020, 314-346.
49 Niklas Odén, "Mechanisms within cross media stories : An analytical framework combining
cross media phenomena with storytelling elements", 2013.
50 J. David Bolter, Richard A. Grusin, Remediation: understanding new media,
Cambridge, Mass.,MIT Press, 1999.
51 Henry Jenkins, "Transmedia Storytelling and Entertainment: An annotated syllabus,"
Continuum 24 (6), 2010, 943-958.
52 Henry Jenkins, Sam Ford, Joshua Green, Spreadable media: creating value and meaning in a
networked culture, Postmillennial pop, New York, London, New York University Press, 2013.
54 Marie-Laure Ryan, Narrative across media: the languages of storytelling, Frontiers of
narrative, Lincoln, University of Nebraska Press, 2004.
55 Henry Jenkins, Convergence culture: where old and new media collide, New York, New
York University Press, 2006.
56 Henry Jenkins, Fans, bloggers, and gamers: exploring participatory culture, New York,
New York University Press, 2006, 1.
Authors Biographies
Yan Breuleux (M.A.Sc, D.Mus) is an associate professor at École NAD-UQAC UQAC (École des arts
numériques, de l'animation et du design), co-researcher within the Hexagram network, the Mimesis
laboratory and a practitioner in the field of visual music for immersive devices. He designs A/V performances
in the form of sensory experiences.
Alain Thibault has been presented in several contexts, contemporary music and digital art events, in America,
Europe and Asia. He is the director of ELEKTRA an annual festival showcasing performances since 1999 and the
International Digital Art Biennale, oriented towards exhibitions since 2012.
Rémi Lapierre has a background in art and science through his studies in mathematics, programming and
video games. His multidisciplinary profile allows him to understand both the advanced technical issues of
software design and the expressive logics of digital creation. Rémi is currently working in the
automated generation of virtual environments for immersive devices.
White Cube / Black Box: Investigating Bias in Museums and Algorithms
University of Michigan Ann Arbor, MI, USA sbrueckn@umich.edu / shannysw@umich.edu /
ljing@umich.edu / dchoberk@umich.edu / kshedden@umich.edu / jmturner@umich.edu / igillet@umich.edu /
mingchlu@umich.edu / weixw@umich.edu
Abstract
White Cube / Black Box seeks to identify bias and the many ways bias gets introduced into and amplified within
systems. A highly interdisciplinary team of data scientists, curators, designers, and artists used face
detection and race classification algorithms to explore bias in algorithms and University of Michigan Museum of
Art’s collection of artworks.
Keywords
Machine learning, face detection, race classification, bias, museums.
White Cube / Black Box is a collaboration between artists, designers, curators, and data scientists at
University of Michigan Museum of Art (UMMA), the Michigan Institute for Data Science, and the University School
of Art and Design that attempts to shed light on the opaque decision-making processes within museum collecting
practices and machine learning algorithms.
White Cube / Black Box seeks to identify bias and the many ways bias gets introduced into and amplified within
systems. In art, the phrase “White Cube” references the history of exclusionary practices within
museums and galleries. Using sterile white walls and decontextualized spaces, works of art are divorced from the
outside world, making them less approachable and accessible. In technology, the “Black Box” is a
controversial metaphor used to describe automated systems where the decision-making process is very difficult or
even impossible to understand.
The resulting art installation featured some of the interesting, curious, and troubling findings that our
research has uncovered about both facial-recognition technology and about the history of representation in the
University of Michigan Museum of Art’s collection of approximately 24,000 works.
We applied one of the most widely used facial detection algorithms to UMMA’s art collection. After
detecting faces in UMMA ’ s artworks, we used a race classification algorithm to look at the diversity of
subjects in the collection. We used the FairFace Dataset for examples of faces belonging to different races. We
used these results to characterize and visualize the racial diversity of the acquisitions made under all of
UMMA’s directors.
We used a technique called “eigenfaces” to explore variation within faces found in UMMA’s
collection and to understand which features are most important in detecting a face.
By applying facial detection algorithms to UMMA’s art collection, we visualize bias in the museum’s
collecting practices throughout its 150-year history. We can also see the ways algorithms amplify human bias.
Our research makes more transparent the opaque decision-making processes within museum collection practices and
machine learning algorithms as these rapidly evolving technologies are being deployed across the world.
Background
Museum Bias
Art museums have a long history of racial and gender bias. A recent study looking at 18 major US art museums
found that 85% of its collected artists are white and 87% are men.¹ Who is depicted in these artworks is
not only an issue of numbers, but bias is also evident in how people are depicted. Racialized caricature is one
obvious example. Furthermore, museums have historically excluded certain groups of people from visiting museums
in both overt and subtler ways.² Museums are now reckoning with how they may have reinforced prejudices in
the past and what responsibility they have in confronting prejudice going forward.³,⁴
Algorithmic Bias
Face recognition algorithms are increasingly adopted for commercial use, for public safety, and in other
applications. However, flaws in the current algorithms not only limit their effectiveness, but also have adverse
consequences for certain demographic groups that are the “usual suspects” of being marginalized or
victimized by new technology. The algorithms’ significant flaws in race and gender recognition can be
attributed partially to the lack of diversity in the training set—white male being
the overrepresented face.⁶ While researchers have repeatedly pointed out such flaws and are improving
the training sets, there may be other limitations of the algorithms that have not been adequately addressed. For
example, the algorithms rely on faces in the training set that are mostly photographs of full frontal view
faces. How well do the algorithms work when the faces are sideways, partially visible, and so on? In our study,
we did not aim to develop a new algorithm or improve a current one; instead, we focus on the use of a highly
unconventional dataset (UMMA’s art collection) to test the limit of existing algorithms trained on human
photographs and understand what features are essential for the correct or incorrect face and race recognition.
Related Works
In 2018, Google Arts & Culture released the Art Selfie phone app as a playful way to discover art. The user
takes a selfie and the app searches thousands of artworks to find one with a similar face.⁵
The 2020 film Coded Bias summarizes MIT Media Lab researcher Joy Buolamwini’s research on how
facial recognition algorithms do not see dark-skinned faces accurately and demonstrates the need for legislation
to reduce bias in algorithms.⁶
UK Research and Innovation recently funded a project titled Transforming Collections: Reimagining Art,
Nation and Heritage led by a team of researchers at the University of the Arts London. The project aims
to “build on decolonial feminist approaches and creative machine learning (ML) development: to enable
digital cross-search of collections to surface patterns of bias, and to uncover hidden and unexpected
connections, and to thus open up new interpretative frames and potential narratives of art, nation and
heritage.”⁷
Process
1) We selected YOLOv4 as the main algorithm to test on the art collection. We also used a second algorithm,
Dlib, to a more limited extent. The two algorithms both returned some successful face and race detections and
some unsuccessful ones. We focus our paper on results with
YOLOv4. ¹¹,¹²
Instead of using artworks (whether within UMMA’s collection or elsewhere) to train the algorithm, we
simply used pretrained weights. Our rationale for not using any art collection as the training set is that we do
not have the resources to manually inspect and label the artworks that can be used for training, and that the
size of such an arts training set could be prohibitively large given the much larger variation in faces in
the artworks than in photographs. We did not evaluate the efficacy of a perfectly customized algorithm but
rather mimicked the realistic practice of brittle deployments despite limited training data.
2) After identifying faces in UMMA’s collection, we applied the VGG-Face CNN network with pretrained
weights from FairFace Dataset to assign race to the faces in UMMA’s art collection.
⁸,⁹,¹⁰
3) We used a technique called “eigenfaces” to explore variation within faces found in UMMA’s
collection and to understand which features are most important in detecting a face.¹³
4) We created an exhibition to share results with museum visitors.
Algorithms and Data Sets
Face Detection
For face detection we used the algorithm YOLOv4 and we trained the artificial intelligence (AI) with the Google
Open Images Database, which is comprised entirely of photographs.
Race Classification
After detecting faces in UMMA’s artworks, we used a race classification algorithm, the VGG-Face CNN
network, to look at the collection’s diversity. We used the FairFace Dataset for examples of faces
belonging to different races. FairFace was created to measure and mitigate racial bias. It contains 108,501
Flickr images of faces categorized as Asian, White, Middle Eastern, Indian, Latino-Hispanic, or Black. ⁸ ,
⁹
Overview of Results
Of the 21386 UMMA collection objects that we used with the algorithm, 6026 objects (28%) were classified as
having at least one face. For race classification, we “forced” the algorithm to choose among the
seven racial groups defined in the FairFaces dataset, but combined their definition of East Asian and Southeast
Asian into one group, and obtained the following: White (69.1%), Black (10.5%), Indian (2.9%), Asian (10.7%),
Middle Eastern (3.9%), and Latino (2.9%). The racial classification algorithm we used lacks a category for
Native Americans.
Limitations and Failures
In addition to being unable to classify Native Americans at all, the algorithm had difficulty identifying faces
in several cases: faces in profile; tilted heads; highly abstract faces; caricature. On the other hand, non-face
objects with round/oval shapes and symmetric features were often classified as faces, such as many vases.
Acquisition Patterns by Different Museum Directors
We sought to understand the acquisition patterns of different UMMA directors, focusing on the predicted
races of people depicted in the acquired works. Although we did conduct some benchmarking to establish the
performance of the algorithms that we employed, an important caveat of this analysis is that it is based on race
classification of detected faces in the artworks by algorithms. It was not humanly possible to validate that all
these predictions were correct. With this proviso, we constructed a contingency table showing the number of
works depicting individuals of each race acquired by each director.
To aid interpretation of this contingency table, we performed a standardization. Let N_ij denote the number of
works acquired by director i that depict individuals of race j. Then, if N is the total number of works
depicting any race acquired by any director, and p_i denotes the proportion of all works acquired by director i,
and q_j denotes the proportion of all works depicting individuals of race j, then N*p_i*q_j is the reference
point for N_ij. We can interpret N*p_i*q_j as the number of works acquired by director i depicting individuals
of race j in the event that all directors purchased works depicting the races with the same frequencies. The
residual R_ij = N_ij - N*p_i*q_j is the excess (if positive) or deficiency (if negative) of works depicting race
j acquired by director i. The standardized residual S_ij = R_ij / sqrt(N*p_i*q_j) aims to place these residuals
on a common scale that is fairly comparable between directors with small and large numbers of acquisitions, and
between races with small and large overall representation in UMMA collection.
Conventionally, values of S_ij smaller in magnitude than 2 are viewed as unimportant. It is not easy to
conclude definitively that a given large value of |S_ij| is large enough to be important, but in many cases
values exceeding 2.5 or 3 are likely to reflect a specific cause and not occur randomly due to variation of
small numbers.
We noticed a significant uptick in the diversity of the collection in 2019. After taking a closer look at the
race classification results from that year, we found that the trend seemed to be specifically tied to the
Take Your Pick exhibition where museum visitors voted to select 250 everyday photographs to add to the
collection, suggesting that a single exhibition can have a notable impact on the overall diversity in the
collection. Though imperfect, we found that the algorithms generated results (such as the uptick in diversity in
2019) that offered new perspectives and points of entry for further manual investigation into smaller,
manageable subsets of the collection.
Eigenfaces
We used “eigenfaces” to explore variation within faces found in UMMA’s collection and to
understand which features are most important in detecting a face. Eigenfaces represent axes of variability in
a collection of images of faces. This technique was first developed in the 1990’s. In prior work, the
eigenfaces have been found to capture factors such as lighting, pose, the presence of eyeglasses and beards, and
anthropometric features such as dimensions of the jaw, nose, forehead, and spacing between the eyes. Eigenfaces
can be used to understand the principal ways that faces in a collection vary, and can also be used as a data
compression technique, in that they represent a “high dimensional” face using a relatively
low-dimensional vector of “scores.”.
Eigenfaces show the most important ways that individual faces differ from the mean face. Each eigenface
corresponds to a spectrum along which variation occurs. Each eigenface below represents one feature important in
detecting faces in our collection.
The blurriness shows how that feature varies in the faces found in UMMA’s collection. For example,
eigenface 3 corresponds to a spectrum along which the face is lit from directions varying from the left to the
right. Eigenfaces 1 and 2 correspond to variation in the overall size and shading of the face. Eigenfaces 6, 7,
and 8 correspond to different patterns of shading at the top of the head, forehead, and around the eyes.
To begin, we first try to limit the extraneous variation by scaling and cropping each face in our collection to
approximately the same position in a fixed-size image (# 224*224*3 # pixels). We then use a mathematical
technique called the “singular value decomposition” to identify the eigenfaces. Specifically, an
eigenface is a pattern represented by signed (positive and negative) weights. Each eigenface assigns one weight
to each pixel location in the images. These weights represent a common pattern of deviation from the mean face.
We note that the mean face itself generally appears “ghost-like” and does not resemble a human face,
but the deviations from this mean face are informative about the unique characteristics of an individual face.
Since there is one weight for each pixel, the eigenfaces can be visualized in the same way that the face images
are themselves visualized. For example, an eigenface corresponding to illumination on the left side may be
bright (positive) on the left side of the image and dark (negative) on the right side of the image; an eigenface
corresponding to spacing between the eyes may have alternating bright and dark regions of weights in a band
located at the level of the subjects’ eyes.
An eigenface represents a spectrum of variation. For example, illumination from the left is part of the same
spectrum as illumination from the right, and hence this variation can be represented by one eigenface;
similarly, wider-than-average eye spacing may be part of the same spectrum as narrower-than-average eye spacing.
Since a spectrum has no defined beginning or end, each eigenface is equivalent to its additive inverse, i.e., F
and -F represent the same eigenface, with the spectrum of variation represented by -F being the same as the
spectrum of variation represented by F, traversed in the opposite direction.
As noted above, the eigenface technique has often been used with collections of highly standardized images,
like passport photos. Even in such a standardized collection, the eigenface technique is generally found to be
influenced by lighting and pose as much or more than it is influenced by anthropometry, which is a drawback to
the approach. Moreover, while some level of dimension reduction is achieved, it often is necessary to use
100-200 eigenfaces to represent most of the variation in a collection of faces. Using the eigenface technique on
UMMA collection is even more susceptible to this issue, since artists represent humans in every possible pose,
and it is not possible to standardize these faces beyond simple translation and scaling.
Using the eigenfaces, we identified a painting of a clown, with makeup caricaturing a face, as having the most
representative face in UMMA’s collection.
Exhibition
Using data from our research, we created two video explainers that explain parallels between biases in art
museum systems and in algorithm systems. They consisted of data visualizations that highlighted race
representation in UMMA collection over time, influence of certain exhibits, and notable research findings and
challenges. The videos and select paintings were exhibited in UMMA to self-reflect and critique the
museum’s past in full transparency.
We displayed these two videos along with actual paintings from the collection as part of the You Are Here
exhibition, which invited museum visitors to consider where they are and where they
aren’t. Above our videos, we displayed the question, “Are you here?” inviting the
viewers to consider how they are represented within the museum’s collection and exhibitions.
We are currently planning a second exhibition scheduled for Fall 2024 that will invite UMMA visitors to observe
and evaluate algorithmic facial detection. We plan to visualize the data by exhibiting actual artworks sorted by
the algorithm’s confidence in recognizing faces within. Additionally, we plan an interactive wall
projection that will contrast the algorithm’s confidence in recognizing faces with confidence judgments
submitted live by museum visitors.
Conclusion
An interdisciplinary team of artists, designers, data scientists, and curators applied face detection and race
classification algorithms to UMMA’s collection of approximately 24,000 artworks that were collected over
150 years.
When we began this project, we asked, “How can the application of machine learning expose or amplify
human bias?” We wanted to know if our AI could reveal the bias in artists, collectors, donors, curators,
and society in general over time. What biases did our AI learn when it was trained on datasets of example faces?
What biases are embedded in the algorithm itself?
In addition to learning about and visualizing how the diversity of UMMA’s collection changed over time
(for the better), we experienced these artworks through the lens of machine learning for the first time. We were
not navigating the collection through the usual categories like who created the artwork, artistic movement,
artistic medium, date created, or the artwork’s origin. We were not experiencing these works as part of a
curated exhibition. We were encountering the artworks in buckets such as “91-100% confident it’s a
face” and “non-face” or through simplistic labels like “has face, White” and
“has face, Indian”. Going through this process changed our own perception and sensitivity to certain
aesthetics as we wondered why the AI made certain decisions. In some cases, the AI’s decisions caused us
to question our own understanding of certain artworks.
In addition to exploring biases within both algorithms and museums, this research invites museums and museum
goers to reflect on ideas of transparency, self-reflection, and critical thinking about collecting and
curatorial practices. How does our understanding of art, curation, and history change when artworks are
algorithmically curated?
References
1 Topaz CM, Klingenberg B, Turek D, Heggeseth B, Harris PE, Blackwood JC, Chavoya CO, Nelson S,
Murphy KM. Diversity of artists in major U.S. museums. PLoS One. 2019 Mar 20;14(3):e0212852. doi:
10.1371/journal.pone.0212852. PMID: 30893328; PMCID: PMC6426178.
2 Olivares, A., Piatak, J. Exhibiting Inclusion: An Examination of Race, Ethnicity, and
Museum Participation. Voluntas 33, 121–133 (2022). https://doi.org/10.1007/s11266-021-00322-0
3 Sandell, R. (2006). Museums, Prejudice and the Reframing of Difference (1st ed.). Routledge.
https://doi.org/10.4324/9780203020036
4 Shirley Li, “American Museums Are Going Through an Identity Crisis,” The
Atlantic, November 28, 2020, accessed Dec 1, 2022,
https://www.theatlantic.com/culture/archive/2020/11/american-museums-are-going-through-identity-crisis/617221/
5 Google Arts & Culture, “Art Selfie,” accessed Dec 1, 2022,
https://artsandculture.google.com/camera/selfie
6 Joy Buolamwini, Coded Bias, accessed Dec 1, 2022, https://www.codedbias.com/about
7 “Transforming Collections: Reimagining Art, Nation and Heritage”, accessed Dec 1,
2022, https://www.arts.ac.uk/ual-decolonising-arts-institute/projects/transforming-collections
8 Karkkainen, K., & Joo, J. (2021). FairFace: Face Attribute Dataset for Balanced Race,
Gender, and Age for Bias Measurement and Mitigation. In Proceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision (pp. 1548-1558).
9 “FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age,”
accessed Dec 1, 2022, https://github.com/dchen236/FairFace
10 “VGG Face Descripto,” accessed Dec 1, 2022,
https://www.robots.ox.ac.uk/~vgg/software/vgg_face/
11 “Dlib,” accessed Dec 1, 2022, http://dlib.net/
12 Bochkovskiy, Alexey, Chien-Yao Wang and Hong-Yuan Mark Liao. “YOLOv4: Optimal Speed and
Accuracy of Object Detection.” ArXiv abs/2004.10934 (2020): n. Pag.
13 Matthew Turk, Alex Pentland; Eigenfaces for Recognition. Journal of Cognitive Neuroscience,
1991; 3 (1): 71–86. doi: https://doi.org/10.1162/jocn.1991.3.1.71
Human Energetics in an Era of Post-Humanism
ChristophBrunner1, JonasFritsch2
1Erasmus University Rotterdam, Netherlands 2IT University of
Copenhagen,Denmark brunner@esphil.eur.nl, frit@itu.dk
Abstract
French philosopher of technology Gilbert Simondon is undoubtedly one of the key figures when it comes to
conceptualizing individuation across physical, mental and social strata. In this article, we develop a rather
overlooked aspect of Simondon’s work, namely how his ontogenetic project also implies an idea of a
“human science” based on a “human energetics,” which—maybe in spite of its
name—is an inherently transhumanist project transducing across both disciplinary and experiential fields,
with a particular emphasis on the role of technology. We present key concepts in Simondon’s work and
relate them to lines of thinking on energies in the arts (Kahn) and post-colonialism (Wynter), exemplified
through an analysis of Nigerian artist Otobong Nkanga’s video work “Remains of the Green
Hill.” Our primary aim with the article is to continue a mobilization of Simondonian concepts and thinking
for an experimental, transhumanist exploration in relation to its ethico-aesthetic and artistic potential.
Keywords
Energy humanities, human energetics, individuation, socio-genesis, posthumanism, human science,
postcolonialism, affect.
In the wake of an increasingly differentiated debate on the status of the human in the era of posthumanism and
the Anthropocene, major critiques of a Eurocentric humanism at the heart of the arts and humanities have
emerged.¹,² Instead of turning towards the agency of non-humans or celebrating the vibrancy
of matter a more-than-human account tuned to the status of the human requires a radical rethinking of what a
non-Eurocentric conception of the human and humanities might look
like.³,⁴,⁵ Such a recasting of the humanities matters in two crucial
ways: on the one hand the humanities are part and parcel of an ideological as much as economic-colonial project
which builds on both an ontology of the human as autonomous and conscious being and the extractive order seizing
lands and resources on a global scale.⁶,⁷ On the other hand, such critiques of human
exceptionalism afford us to rethink the human as embedded in material, organic, social and mental domains that
do not only intersect but radically transform the idea of an autonomous human subject. A humanities building on
such a post-humanist conception of the human has been a major concern of French philosopher of technology
Gilbert Simondon, whose work in the 1950s and 60s has not only formulated a relational conception of change and
becoming but was also deeply concerned with a transformation of the humanities towards what he terms a
“human Science” (Science humaine) involving the formation of a proper “human
energetics.” In the first instance, Simondon’s persistent attention to the role of technical being
might seem like a plea to know technologies better, to become an engineer that understands and not only uses
technologies. Further one could conceive of his elaboration on energy and energetics, which he primarily derives
from physics, as an attempt to consolidate natural sciences and the humanities.⁸ However,
Simon-don’s interest is a different one. By asking about the axiomatics of the natural sciences,
cross-pollinating different scientific disciplines, he outlines a new program for the humanities: with the
notion of a human energetics, he shifts the focus from subdivisions of thinking the human along disciplinary
differences to the possibilities of a shared sensibility for the emergence, continuation, and potential
dissolving of constellations which take shape across material, organic, and mental domains. He particularly
draws on the notion of “potential energy” to do so.
The resurgent interest in the concept of energy and energetics in areas such as the energy humanities, but also
artistic and postcolonial contexts demonstrates an increasingly transdisciplinary body of research that provides
plenty of resonances with Simondon’s proposition of a human
energetics.⁹,¹⁰,¹¹,¹² These respective
fields and approaches towards an energetic outline of art, media, and culture conceive of energy—or
energies—as relational properties immanent to varying processes of material and conceptual transformation
in creative practices.
While Simondon aims at a broad reworking of the humanities closely linked to the sciences, his emphasis on the
human should not be mistaken. In his transductive account of potential energy, we perceive a strong refusal of
any anthropomorphic conception of the human as different from technologies or its organic and inorganic
environment. In this sense, the human energetics in a Simondonian key, are trans-humanist, where the human
becomes a composite of different energetic relays. More than that, Simondon’s reworking of potential
energy allows us to embark on a critique of the humanist tradition and its Eurocentric and enlightened concept
of the human. We therefore draw on the work of Jamaican philosopher and writer Sylvia Wynter whose critique of
the human as "Man" implies a feminist and post-colonial problematization of enlightenment conceptions of the
human, both in philosophy and the sciences. Instead of rejecting the concept of the human, however, she proposes
a reworked notion of the term that resonates strongly with several contemporary post-and transhumanist
positions.
The critical reworking of the human through Simondon, Wynter, and new energetic tendencies in the arts allows
us to conceive of contemporary artistic practices as “fields of
experience.”¹³,¹⁴,¹⁵ Addressing such fields as
energetic textures, opens up a new perspective on the tendentially less concrete category of relations. Without
wanting to stage energies against relations, our intention is to show how the energetic traverses material,
technological, embodied, aesthetic and perceptual domains through affective, energetic activation. We will
therefore draw on Nigerian artist Otobong Nkanga’s video work “Remains of the Green Hill” in
order to provide a first analytic account of how to work with this energetic approach in the context of
electronic arts and media.
A Simondonian Proposition of a Human Energetics
In any case, we would arrive at the idea according to which a human science must be founded on a human
energetics and not just on a morphology; a morphology is quite important, but an energetics is necessary; one
would have to ask why societies transform, why groups change in accordance with the conditions of metastability.
However, we certainly see that what is most important in the life of social groups is not merely the fact that
they are stable, but that at certain moments they cannot conserve their structure: they become incompatible with
respect to themselves, they dedifferentiate and become supersaturated; just as the infant can no longer remain
in a state of adaptation, these groups disadapt.¹⁶
Simondon most thoroughly develops his notion of energy and human energetics in his magnus opus Individuation in
the Light of Notions of Form and Information collected and published in 2005, but containing work from 1964 and
onwards. Here, Simondon conceptualizes the notion of energy—and in particular “potential
energy”—as a form-taking and change-making processes of individuation across organic, physiological,
social, technical and psychological domains. This allows him to conceive of individuation as tied to a system
(and its state of stability). For Simondon, the system is not a closed one. On the contrary, it is the necessary
ground from which the energetic activity initiates a potential process of individuation. Whereas this use of
energy and energetics derived from physics might be seen as an attempt to consolidate natural sciences and the
humanities,¹⁷ Simondon has a different aim: through a human energetics, the focus shifts from
subdivisions that conceive of the human along disciplinary divides towards the possibilities of a shared
sensibility for the emergence, continuation, and potential dissolving of compositions (or forms). To Simondon,
these compositions take shape across both material, organic, and mental domains through operations of
transduction.¹⁸
The notes from the lecture “Form, Information, Potentials” most explicitly develop the notion of
human energetics as a non-equivalence-based model of thinking relations between form and information. Either
notion is deeply transformed from a more conventional conception of form as given and impressed onto matter. By
proposing a new theory of form, Simondon interlaces an archetypical and a hylomorphic conception of form to be
found in Plato and Aristotle respectively. For Simondon, the archetype is a structural germ of a form—a
field—and the matter-form couple in the Aristotelian scheme is a domain. The field is a virtual cueing of
potential energies to “inform” the energies immanent to a domain which is “the ensemble of
reality that can receive a structuration.”¹⁹ Far from being a linear process, the coupling of
the germ and the structuration provides a vital processof differentiation and dedifferentiation.
Put differently, the ideal form is not a given but an informational field encountering a domain whose abilities
to resonate shape the actual form-taking. Along his informational account, Simondon uses the concept of energy
and energetics without rendering it into a metaphysical category of a metaphor for explaining a connective
dimension that cannot be grasped in any substantial manner. On the contrary, energy is taken in its physical
sense, where potential energy is energy in a system being stored and later released. It is a material
potentiation that in its abstract state is information. With the introduction of the concept of information,
Simondon casts potential energy as a type of energy that is not merely quantifiable in magnitude but crucial for
processes of transforming an energetic state, like that of a mineral, into another, such as a metal or gas. To
understand such qualitative changes, a mere thermodynamic equivalence model of energy is insufficient since
it does not account for the effectuation of change itself. With his insistence on information, Simondon ties the
problem of change to the question of what drives change and how change operates through a fielding of potential
energies. Put differently, potential energy is a prerequisite for the process of form-taking as a transductive
operation. In his earlier work On the Mode of Existence of Technical Objects, Simondon accordingly writes that
the issue at stake concerns thinking not from the end of an action (its result) which he calls finality but
rather to focus on causality and the process of causation.²⁰ Simondon insists that morphology, the
taking or changing of form, requires the addition of energy to understand general processes of transformation.
To further understand “why societies change” for instance, we must turn towards the process of
individuation. In his outlined energetic scheme, a society’s transformation should be understood by a
constant tension between differentiation and dedifferentiation, of a non-linear leaping of constant reworkings
of how form takes shape under differentiating conditions and along specific lines of processing. For the human
sciences (i.e., humanities) this principle or paradigm provides an axiomatic approach that foregrounds the
non-foundational process of energy-oriented information. As a process of transduction, information binds the
different modes of existence into an energetics that allows us to radically challenge a classic humanist
understanding of action. It is neither the human as an already constituted individual that acts, nor is an
individual the mere subject of external forces. It is the interplay between information, matter and form taking,
cutting across different registers of existence, which constitute a human that acts by being energetically
engaged.
By further introducing the physical notion of the field, Simondon finds the conceptual handle to move beyond a
separation between the abstract and concrete, the empirical and metaphysical, or a binary logic of interiority
and exteriority. Combined, the notions of field and energy allow Simondon to recast processes of individuation
and ontogenesis across disciplinary boundaries, leading him to propose a rethinking of the “human
sciences” towards “human Science” with a common axiomatics applicable to various domains,
respecting their multiple forms of application. Such a general energetics might be conceptualized as a
transversalizing move beyond disciplinary divides in a common understanding and exploration of new ways of being
and, especially, becoming—with a particular emphasis on the constitutive role of technics and technical
objects for energetic, material, and existential conditions.
Simondon’s by now sixty-year-old problematization of the humanities through energy and energetics bears
another problem tied to contemporary debates in the emergent domain of research of energy humanities but also
the extractivist idea of human energy as the exploitative enterprise of slavery and the deterioration of natural
resources. Simondon’s critique of such exploitative modes is radical, in the sense that he considers
neither economic nor energetic exploitation as sufficient for the problematization of human suffering. His take
on a Marxist critique of political economy emphasizes that the workers’ loss of the means of production
under capitalism and their cooperative reclaiming rests on a much deeper alienation that between human and
technical being.²¹ Again, while on a more general level, Simondon’s critique could be read as a
debunking of the foundational critique of capitalism in Marx. However, his critique targets equivalence and more
precisely the deployment of the general equivalent in Marx as much as the general equivalent of energy in
thermodynamics. By problematizing a conception of modernist progress building on models of equivalence, such as
money or monetary value and energy, Simondon becomes a highly interesting interlocutor for thinking the
far-from-equilibrium critiques of change based on human-centered action as a dominant narrative in the genealogy
of Western humanities.
Energy Humanities and Energy in the Arts
In the following, we wish to position Simondon’s human energetics in relation to a rapidly growing
engagement with energy in several emerging research areas within the arts and humanities over the last decades.
In doing so, we also wish to hint at how a Simondonian proposition of energetics contributes to this
current—important and necessary—work carried out; namely the basic idea that energy is core to both
human and non-human existence and hence is not something “out there” but deeply entrenched within an
expanded understanding of the arts and humanities.
In their article from 2014 “The Rise of Energy Humanities,” authors Dominic Boyer and Imre Szeman
forcefully state the need for a humanistic take on current energy dilemmas:
“Energy humanities” is an emerging field of scholarship that overcomes boundaries between
disciplines and between academic and applied research. Like its predecessors, energy humanities highlights the
essential contribution that the insights and methods of the human sciences can make to areas of study and
analysis that were once thought best left to the natural sciences.”²²
A central starting point for the energy humanities is that “today’s energy and environmental
dilemmas are fundamentally problems of ethics, habits, imagination, values, institutions, belief, and
power—all traditional areas of expertise of the humanities and humanistic social sciences”
²³. This points to the necessity of revitalizing both humanist and artistic perspectives in the joint
pursuit of rethinking energy across disciplinary divisions. This differs slightly from the approach taken by
Simondon. Rather than arguing for what human sciences can bring to natural sciences, a Simondon- inspired human
energetics emphasizes the inherent entanglements across said research fields. We can only arrive at radically
repositioning disciplinary divisions by acknowledging this condition. Here, human energetics might be seen as
adding to a heterogenous, diverse understanding of different kinds of energies across several domains.
It is beyond the scope of this paper to provide a comprehensive review of all the ways in which energy has come
to matter in humanist and artistic thinking and practice. For an attempt at this quasi-impossible task, with an
orientation towards the artistic explorations, we point at Douglas Kahn’s comprehensive introduction to
the anthology Energies in the Arts from 2019. Here, Kahn presents a pluralistic approach moving from
“energy” to “energies in the arts.” He develops an “indefinition of
energies,” and instead shows multiple, situated examples of energies in, around and across artistic
practices. According to Kahn:
“Energies are embedded and embodied in all phenomena, in the operations of the senses and cognition, and,
thus, how we might feel, perceive, and think about them, individually and collectively. They also drive engines,
manufacturing, telecommunications, and media; and underscore large-scale periods such as the age of steam and
the Industrial Revolution, the atomic age and ongoing oil wars.”²⁴
Fields of energy and fields for understanding energy continuously commingle, which calls for a specificity in
addressing energetic configurations, and not least how these configurations are “(...) bound to be upset
by the capacious, appropriative and synthesizing embrace of the arts and culture.”²⁵ A focal
point for Kahn, then, is to acknowledge and explore the “aesthetics and poetics” of different
artistic discourses and expressions of energy.²⁶ In the text, Kahn surveys and addresses
understandings of energy from a broad range of disciplinary fields – physics, psychology, philosophy,
(environmental) humanities – and, naturally, from through an array of artistic expressions. Among other
examples, Kahn reports on a fundamental energetic transformation in the arts when music went from metabolics and
acoustics to electrical and electronic music:
“Transduction back and forth between two classes of energy – acoustics and electromagnetism –
not only fundamentally changes the character of the production, modulation, amplification, transmission, and
storage of music, it also set up new relations of control (and lack thereof) for the performer and
composer.” ²⁷
Whereas Kahn does not make any explicit references to the work of Simondon, we see both a range of shared
concepts (fields, transduction) and interests. Kahn makes it explicit that his task is not to develop
overarching theories or philosophies, but tracing down specific instances of how energies are explored,
activated or transduced in practice. Whereas it might be argued that the departure of this article is indeed
related to the theoretical or philosophical articulations presented in Simondon’s work, it is important to
emphasize that it is equally important to trace particular energetic projects and trajectories—as well as
to inspire future both conceptual, designerly or artistic activations of this conceptual foundation in relation
to both analysis and practice. However, we also wish to revisit Simondon’s project in the light of
contemporary discussion around the post-human across arts and humanities, to resituate and discuss the extent of
his thinking, which will be the main focus in the following section.
The Human in the Era of Posthumanism
The heightened interest in a differentiated concept of energy and energetics in the arts allows us to trace
some of the more recent encounters with post-and decolonial critiques of Western conceptions of the human in
aesthetic discourses. We will particularly engage positions which relate to Black Studies and Black Aesthetics.
The post-human in the arts ranges from early debates on Internet art, to feminist celebrations of the cyborg
(see for instance the most recent iteration of the Venice Biennale in 2022), eco-art or various
problematizations of the Anthropocene. Most of these critiques of humanism tend to overcome the human as
such.²⁸ Black Aesthetics and Black Studies, on the other hand, have raised the question of the
category of the human as a trait of racist and colonial Eurocentric
enlightenment.²⁹,³⁰
,³¹,³²,³³ These perspectives draw on Frantz
Fanon’s notion of the “sociogenic.”³⁴ Sociogeny, rather than a biological notion of
phylogeny or the philosophical concept of the ontogenetic, defines the constant reproduction and differentiation
of social forms. Through the lens of sociogeny,
Fanon casts his main concern, racism, a cultural phenomenon tied to capitalist and colonial extractive logics
of dehumanization.
Dehumanization, exclusion, and Othering have been part and parcel of black resistant reflections in literature
and the arts. One of the most prominent expositions of aiming at another or different notion of Black humanity
can be found in Afrofuturism as a literary and artistic movement.³⁵,³⁶ These
artistic explorations engage with the future as a post-humanist line of flight for marginalized minorities who
are systematically excluded from mainstream discourses in society and culture. From a more feminist perspective,
Haraway draws on the works of Hortense Spillers and Gloria Anzaldúa, among others, to cast a black
feminist mode of resisting the intersectional historical dehumanization of women of color under the trope of a
white, patriarchal, and Eurocentric order of who is worth being considered human. Haraway engages with
abolitionist and women's rights activist Sojourner Truth’s famous utterance “Ain’t I a
woman” to point out the impenetrability of white supremacy as a black woman and fully human being.
Sojourner Truth’s exclamation “ain’t I a woman” at a 1851 women’s convention in
Ohio underlines the dehumanizing experience of not being accounted as fully a person as the (white) others.
Being partial and not being whole and thus fully human—as the term might be used to point at the integrity
of a being (or subject) or a community (such as a species) that marks the experience of the black
woman—provides the potential of a different humanity which Haraway terms the post-human.³⁷
The human in the post-human is not, as it might often appear, a beyondness which runs the danger of leaving the
foundational Western concept of the human intact, but another kind of human that was never included in the
enlightened project of the humanist tradition. The effects of a dehumanizing and exclusive conception of the
human define one of the key problematizations of Sylvia Wynther’s work. For Wynter, the human has mostly
been a conception of Man [sic!] either in a phylogenetic or biogenetic and ontogenetic vein. We can see how such
a problematization allows us to engage an immanent critique of Simondon’s emphasis on ontogenesis as the
guiding principle of his philosophy of individuation. Up to a certain point, Wynter and Simondon share a similar
project; reconsidering the human not as a category to be overcome but as a shared principle for a different
human Science. For Wynter the devalorizing of blackness as a biological, ontological, and social category (all
three composing the sociogenic) leads to an “over-valorization of whiteness” which she casts as an
“overall devalorization of the human species that is indispensable to the encoding of our present
hegemonic Western-bourgeois biocentric descriptive statement of the human.”³⁸ And she further
counters such a casting:
“In other words, because the negative connotations placed upon the black population group are a function
of the devalorization of the human, the system revalorization of Black peoples can only be fundamentally
effected by means of the no less systemic revalorization of human being itself, outside the necessarily
devalorizing terms of the biocentric descriptive statement of Man, overrepresented as if it were by that of the
human.”³⁹
The revalorization of Black peoples as a necessary and all-encompassing reworking of the human provides a black
radical critique of the reductive, hegemonic, and racist conception of the human in the Western narrative of the
human sciences. We consider Wynter’s problematization crucial to further extend Simondon’s human
energetics. In order to do so, we need not only to reconsider the information-oriented process of individuation
that proceeds through potential energy, but to account for the informational germs that engage with matter to
take form from a non-Eurocentric or white assumption as we find them in the notion of good form or good sense,
from Plato to Descartes and Kant. Engaging with Black Aesthetics, Wynter proposes another form-taking which
undermines the white humanist assumption of form and the sociogenically ingrained mechanism of representation
and recognition. Describing the human as “mere mechanism” of dominant, bourgeois, hegemonic and
racist order, Wynter dismantles white humanism as a structural function for extraction and marginalization
– of rendering blackness inhuman. Frantz Fanon, to whom Wynter repeatedly refers, writes:
“What are by common consent called human sciences have their own drama. Should one postulate a type for
human reality and describe its psychic modalities only through deviations from it, or should one not rather
strive unremittingly for a concrete and ever new understanding of man?”⁴⁰
Mobilizing Fanon, Wynter asks about potential energies for a resurgence of the human in a non-hegemonic and
non-mechanistic and thus not racializing key. The difference of such energetics resides in Fanon’s and
Wynter’s reclaiming of the human, engaging the relation between fields of potential energy and their
resonance with form-taking processes with and through matter. Matter is not the non-or post-human to be
celebrated in an often depoliticizing gesture, but an energetic relay that provides the ground for becoming and
accounts for the human as a composite mode of existence.
Mattering: Post-humanist Energetics in the Arts
From a Simondonian proposition of a human energetics, to-wards a broadened notion of energy in the arts towards
a black aesthetics of energies against a dehumanizing humanist order, the question of the more-than-human
obliges us to turn to matter. Matter as a concept gained a rather romanticized outlook bearing the possibility
of depoliticization. While the vibrancy of matter provides some resonance with Simondon’s refusal of the
hylomorphic scheme as insufficient for an understanding of form-taking, it might also lead to an overt
celebration of matter and an agential realism that runs the danger of leaving the human
behind.⁴¹,⁴² New Materialism’s engagement with matter proposes a more
nuanced differentiation, especially when it enters into an honest dialogue with historical materialism. At the
same time, a thoroughgoing inclusion of a black studies focus on the sociogenic remains a future
task.⁴³ Katryn Yusoff’s critical analysis of the science of geology as a discipline that
considers matter as inhuman and turns slaves into energy providing bodies thus becoming inhumane objects, adds a
new problematizing dimension to our discussion of human energetics.⁴⁴ Turning the enslaved body into
an energy resource, drawing a line from the plantation to machine discipline, as Nicholas Fiori proposes, binds
the enslaved body to a thermodynamic energy regime ⁴⁵. Such a regime, as much as the capitalist
extractive one, is insufficient to understand the overall scope and relevance of energy. The same accounts for
the energetic regime of geology, as the discipline that casts matter or the earth into an extractable resource.
Simondon’s insistence on potential energy enables us to build a bridge to Wynter’s and
Fanon’s undoing of a White humanist conception of the human. It shifts the de-humanizing and energy as
resource-based logic around matter and energy into one that moves through blackness and black embodiment.
Yusoff’s work points at the extractivist logic immanent to geology and the energy necessary to extract
specific matters from the earth. The energy of the objectified slave body is bound to matter as the object of
extractive activities, creating a tightly knit energetic feedback loop.
For us the question posed by Simondon, Wynter and Fanon remains; how to rethink the human beyond whiteness and
Eurocentric humanism. A first proposition works through “affective engagement” and “affective
relaying” with and through the sensuous.⁴⁶,⁴⁷ Affect as the realm where
the emergent quality of information becomes felt before it can be rationalized appears as the ideal zone to
begin a human Science that pays attention to both potential energy and the sociogenic problematization of the
human. Such a reworking of the human occurs when we look at Otobong Nkanga’s short video artwork
“Remains of the Green Hill.” The video shows Nkanga in the foreground holding different yoga poses
with rocks on her head. The artist stands in front of the former Tsumeb copper mine in Namibia. The
video’s audio is an interview with the former managing director of the by now abandoned mine. The images
are calm without other humans. Through the warm light of sunrise, the texture of the scene is one of rebeginning
in the ruins of prior capitalist (in-and dehumanizing) extraction. The manager talks about turning the abandoned
pit into a monument to begin a new narrative that reworks the scars of the former violent activity. Heather
Davis writes about Nakanga’s posture: “the gestures could be read as an invocation of healing, of
connection, deliberately breathing in the remains of the mineral air.”⁴⁸ Healing in the
remains of violent energy turnover, healing towards different human and more-than-human encounters. We might
conceive of this as first steps towards a different kind of expanded—planetary?
cosmological?—humanity. However, it is also necessary to point out the rock on the artist’s head, a
burden and historic geological weightedness and a companion or ally at the same time.
A range of existential questions have (re-)surfaced within the humanities following a decade-long period of
technoscientific advancement and environmental crisis. From the technological side of things, machine
learning/AI, cyborgs, (both voluntary and involuntary) robots and the like are jointly blurring the boundaries
between what has normally been conceived as the “human” and “non-human.” Similarly, the
challenges we are facing on an environmental and planetary scale have also propelled a reorientation of the
human as the center of the world, albeit maybe still responsible for catalyzing the forces that have effectuated
this displacement in close collaboration with a range of technical inventions. All of this has had a profound
impact on our very conditions of life and living in the world. Somehow, we seem to be perpetually living in a
“(...) far-from-equilibrium situation where each of the systems we depend on for stability
References
1 Rosi Braidotti, The Posthuman, Cambridge, UK, Malden, MA, USA, Polity Press, 2013.
2 Donna Jeanne Haraway, Staying with the Trouble: Making Kin in the Chthulucene,
Experimental Futures: Technological Lives, Scientific Arts, Anthropological Voices, Durham, Duke University
Press, 2016.
3 Bruno Latour, We Have Never Been Modern, Cambridge, Mass, Harvard University Press,
1993.
4 Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University
Press, 2010.
5 Erin Manning, Always More than One: Individuation’s Dance, Durham, Duke
University Press, 2013.
6 Sylvia Wynter, “On How We Mistook the Map for the Territory, and Reimprisoned Ourselves in
Our Unbearable Wrongness of Being, of Desêtre: Black Studies Torwards the Human Project,” in A
Companion to African-American Studies, ed. Lewis R. Gordon and Jane Anna Gordon, London, Blackwell
Publishing, 2006, 107–18.
7 Kathryn Yusoff, A Billion Black Anthropocenes or None, Forerunners: Ideas First from
the University of Minnesota Press 53, Minneapolis, University of Minnesota Press, 2018.
8 Gilbert Simondon, Individuation in Light of Notions of Form and Information, Volume 1,
Minneapolis, Minnesota Press, 2020, 676.
9 Imre Szeman, Dominic Boyer, eds., Energy Humanities: An Anthology, Baltimore,
Johns Hopkins University Press, 2017.
10 Douglas Kahn, ed., Energies in the Arts, Cambridge, Massachusetts, The MIT Press,
2019.
11 Nicholas Fiori, “Plantation Energy: From Slave Labor to Machine Discipline,”
American Quarterly 72, no. 3, 2020, 559–79, https://doi.org/10.1353/aq.2020.0035.
12 David McDermott Hughes, Energy without Conscience: Oil, Climate Change, and
Complicity, Durham, Duke University Press, 2017.
13 Christoph Brunner, Jonas Fritsch, “Interactive Media as Fields of
Transduction,” The Fibreculture Journal, no. 18, October 2011, 118–45.
14 Christoph Brunner, Jonas Fritsch, “Beyond the Network: Experiential Fields and Urban
Media Ecologies,” Proceedings of the Media Cities 2013 Conference in Buffalo, 2013,
277–87.
15 Christoph Brunner, Jonas Fritsch, “Shifting Immediations: Fields of Experience across
Media Art and Design,” in Proceedings of the 26th International Symposium of Electronic Arts -
ISEA 2020, ed. ISEA, Canada, Printemps Numerique Canada, 2020, 91–97.
29 Denise Ferreira da Silva, “Toward a Black Feminist Poethics: The Quest(Ion) of Blackness
Toward the End of the World,” The Black Scholar 44, no. 2, 2014, p.81–97.
30 Saidiya Hartman, “Venus in Two Acts,” Small Axe 12, no. 2, June 2008,
1–14.
31 Fred Moten, Stolen Life, Consent Not to Be a Single Being, v. 2, Durham, Duke
University Press, 2018.
32 Hortense J. Spillers, “Mama’s Baby, Papa’s Maybe: An American Grammar
Book,” Diacritics 17, no. 2, 1987, 64, https://doi.org/10.2307/464747.
33 David Lloyd, Under Representation: The Racial Regime of Aesthetics, First edition, New
York, Fordham University Press, 2019.
34 Frantz Fanon, Black Skin, White Masks, 1st Evergreen ed, New York, Grove Press,
1982.
35 Kodwo Eshun, More Brilliant than the Sun: Adventures in Sonic Fiction, London, Quartet
Books, 1998.
36 Mark Derey, “BlacktotheFuture:InterviewswithSameul R. Delany, Gret Tate, and Tricia
Rose,” in Flame Wars: The Discourse of Cyberculture, ed. Mark Derey, Durham, Duke University
Press, 1994, 179–222.
37 Donna Haraway, “Ecce Homo, Ain’t (Ar’n’t) I a Woman, and
Inappropriate/d Others: The Human in a Post-Humanist Landscape,” in Feminists Theorize the
Political, ed. Judith Butler, Joan Wallach Scott, Routledge, 1992, 86–100.
38 Wynter, Territory, 116.
39 Wynter, Territory, 116.
40 Fannon in Wynter, Territory, 117.
41 Bennett, Vibrant Matter.
42 Karen Michelle Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning, Durham, Duke University Press, 2007.
43 Hanna Meißner, “New Material Feminisms and Historical Materialism: A
Diffractive Reading of Two (Ostensibly) Unrelated Perspectives”, In Mattering, edited by
Victoria Pitts-Taylor, New York University Press, 2020, 43–57,
https://doi.org/10.18574/nyu/9781479833498.003.0003.
44 Yusoff, A Billion Black Anthropocenes.
45 Fiori, Plantation Energy.
46 Christoph Brunner, “Relaying and Rebeginning”, Transmutations 1, no. 1,
March 2013, http://transmutations.org/site/relaying-and-re-beginning/.
47 Jonas Fritsch, “Understanding Affective Engagement as a Resource in Interaction
Design”, Nordes Conference Proceedings, Oslo, 2009.
48 Davis Heather, “Blue, Bling: On Extractivism,” Afterall: A Journal of
Art, Context and Enquiry 48, September 2019, 12– 19, https://doi.org/10.1086/706123.
49 Brian Massumi, Erin Manning, Jonas Fritsch, Bodil Marie Staving Thomsen, “Affective
Attunement in the Field Of Catastrophe”, In Politics of Affect, London: Polity Press, n.d,
112–45.
Bibliography
Karen Michelle Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and
Meaning, Durham, Duke University Press, 2007.
Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press, 2010.
Rosi Braidotti, The Posthuman, Cambridge, UK, Malden, MA, USA, Polity Press, 2013.
Christoph Brunner, “Relaying and Rebeginning”, Transmutations 1, no. 1, March 2013,
http://transmutations.org/site/relaying-and-rebeginning/.
Christoph Brunner, Jonas Fritsch, ““Interactive Environments as Fields of Transduction”,
The Fibreculture Journal, no. 18, 2011,
http://eighteen.fibreculturejournal.org/2011/10/09/fcj-124-interative-environments-as-fields-of-transduction/.
"Beyond the Network: Experiential Fields and Urban Media Ecologies”, Proceedings of the Media Cities
2013 Conference in Buffalo, 2013, 277–87.
“Shifting Immediations: Fields of Experience across Media Art and Design”, Proceedings of the
ISEA 2020 Symposium in Montreal, 2020.
Howard Caygill, “Life and Energy.” Theory, Culture & Society 24, no. 6, November 2007,
19–27, https://doi.org/10.1177/0263276407078710.
Davis Heather, “Blue, Bling: On Extractivism”, Afterall: A Journal of Art, Context and Enquiry
48, September 2019, 12–19, https://doi.org/10.1086/706123.
Mark Derey, “Black to the Future: Interviews with Sameul Delany, Gret Tate, and Tricia Rose”, In
Flame Wars: The Discourse of Cyberculture, edited by Mark Derey, Durham, Duke University Press, 1994,
179–222.
Kodwo Eshun, More Brilliant than the Sun: Adventures in Sonic Fiction, London, Quartet Books, 1998.
Frantz Fanon, Black Skin, White Masks, 1st Evergreen ed. New York, Grove Press, 1982.
Nicholas Fiori, “Plantation Energy: From Slave Labor to Machine Discipline”, American Quarterly
72, no. 3, 2020, 559–79, https://doi.org/10.1353/aq.2020.0035.
Jonas Fritsch, “Understanding Affective Engagement as a Resource in Interaction Design”, Nordes
Conference Proceedings, Oslo, 2009.
Donna Haraway, “Ecce Homo, Ain’t (Ar’n’t) I a Woman, and Inappropriate/d Others: The
Human in a Post-Humanist Landscape”, In Feminists Theorize the Political, edited by Judith Butler, Joan
Wallach Scott, Routledge, 1992, 86–100.
Donna Jeanne Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Experimental Futures:
Technological Lives, Scientific Arts, Anthropological Voices, Durham, Duke University Press, 2016.
Saidiya Hartman, “Venus in Two Acts”, Small Axe 12, no. 2, June 2008, 1–14.
David McDermott Hughes, Energy without Conscience: Oil, Climate Change, and Complicity, Durham, Duke
University Press, 2017.
Douglas Kahn, ed. Energies in the Arts, Cambridge, Massachusetts, The MIT Press, 2019.
Bruno Latour, We Have Never Been Modern, Cambridge, Mass, Harvard University Press, 1993.
David Lloyd, Under Representation: The Racial Regime of Aesthetics, First edition, New York, Fordham
University Press, 2019.
Brian Massumi, Erin Manning, Jonas Fritsch, Bodil Marie Staving Thomsen, “Affective Attunement in the
Field Of Catastrophe.” In Politics of Affect, London, Polity Press, n.d., 112–45.
Erin Manning, Always More than One: Individuation’s Dance, Durham, Duke University Press,
2013.
Hanna Meißner, “2. New Material Feminisms and Historical Materialism: A Diffractive Reading of
Two (Ostensibly) Unrelated Perspectives”, In Mattering, edited by Victoria Pitts-Taylor, New York
University Press, 2020, 43– 57, https://doi.org/10.18574/nyu/9781479833498.003.0003.
Fred Moten, Stolen Life. Consent Not to Be a Single Being, v 2, Durham, Duke University Press, 2018.
Silva Denise Ferreira da, “Toward a Black Feminist Poethics: The Quest(Ion) of Blackness Toward the End
of the World”, The Black Scholar 44, no. 2, 2014, 81–97.
Gilbert Simondon, Individuation in Light of Notions of Form and Information, Volume 1,
Posthumanities 57, Minneapolis, Minnesota Press, 2020.
Hortense J. Spillers, “Mama’s Baby, Papa’s Maybe: An American Grammar Book”,
Diacritics 17, no. 2, 1987, 64, https://doi.org/10.2307/464747.
Imre Szeman, Dominic Boyer, eds., Energy Humanities: An Anthology, Baltimore, Johns Hopkins
University Press, 2017.
Sylvia Wynter, “On How We Mistook the Map for the Territory, and Reimprisoned Ourselves in Our
Unbearable Wrongness of Being, of Desêtre: Black Studies Towards the Human Project”, In A
Companion to African-American Studies, edited by Lewis R. Gordon,, Jane Anna Gordon, London, Blackwell
Publishing, 2006, 107–18.
Authors’ Biographies
Christoph Brunner, PhD, is Assistant Professor for Philosophy of Media and Technology at Erasmus University
Rotterdam. He is founder of the ArchipelagoLab for Transversal Practices at Leuphana University
Lüneburg. His research concerns the relations between media and their aesthetics, affective politics,
and social movements in networked cultures. He takes a particular interest in practices and positions from
Global South and translocal forms of organizing. Most of his work arises from collaborations with critical
practitioners.
Jonas Fritsch, PhD, is Associate Professor in Interaction Design at the IT University of Copenhagen in the
Department of Digital Design. He is head of the Affective Interaction & Relations (AIR) Lab and Head of
the Design Research Section. His work revolves around a creative thinking of interaction design, design
processes, experience philosophy and affect theory through practical design experiments with interactive sound
and physical interfaces.
How Digital Anthropomorphism Enhances Creativity in Human-to-Robot Dance Interactivity
Sorina-Silvia Cîrcu, and Chu-Yin Chen, * †
AIAC Laboratory, Université Paris 8, CNRS-University of Montpellier LIRMM, National
Tsing Hua University Montpellier, Paris, Taiwan sorina.silvia@gmail.com ; chu-yin.chen@univ-paris8.fr
Abstract
Through our research-creation experiments, we imagine movement sequences that challenge the concept of
anthropomorphism in digital twin robots. While considering the term of interactivity to better define their
interaction, we observe how different movement material is stimulating creativity through a hybridization
process between human and machine. From this perspective, we determine how users experience qualia
while learning through imitation a dance sequence consecutively demonstrated by a humanoid robot, an
industrial arm and a human. Their feedback and our own practical experimentation allow us to better understand
the impact of digital anthropomorphism in the making of a sustainable human-to-robot interaction.
Keywords
Digital Anthropomorphism, Dance, Interactivity, Social Robots, Hybridization, Symbiotic Individuation.
Since the 6th century BCE, when the term was firstly employed to describe religious phenomena ⁹,
anthropomorphism has accompanied humanity’s intention to replicate its characteristics into different
environments. Anthropomorphism (from the Greek word ánthrōpos meaning “human” and
morphē meaning “form”) is described in ¹⁰ as “the tendency to attribute human
characteristics to inanimate objects, animals and others with a view to helping us rationalize their actions. It
is attributing cognitive or emotional states to something based on observation in order to rationalize an
entity’s behavior in a given social environment.” As discussed in ³¹, an essential
component of the human spirit is to lend certain characteristics of our psychic life, by projecting our physical
and psychological functioning into objects. According to ¹ “humans attribute, often unknowingly,
personality traits to machines based not only on their external appearance, but also on their functioning and
skills.” In the present paper, we refer to anthropomorphism in a broader humanistic perspective, as a
characteristic of an autonomous system’s behavior that allows ascribing human-like characteristics and
intentions to non-human entities like robots. Recently, the literature has integrated an interpretation proposed
by French philosopher Bruno Latour where the meaning of anthropomorphism is defined by “that which has
human shape” and “that which gives shape to humans”³⁰ encouraging scholars to
envision digital anthropomorphism as the concept that integrates both views. Using these grounds, we explore key
notions like agency ¹⁵ and autonomy ² for our particular case of dancing with robots, to better
outline the link of these notions with digital anthropomorphism in establishing a sustainable human-to-robot
interaction (HRI). Lately robotics has enhanced these perspectives, developing artifacts that challenge the idea
of humanity.²⁶ We rely on scholars ⁵, ⁴ to define the place robots can occupy
in our study, considering them as tools (helping human to accomplish a task—in our case develop a
choreography), as avatars (since the robot engages in a certain social presence with other people—in our
case the spectators of a dance performance) and especially as partners (establishing a co-working process with a
collaborator—in our case co-creating a dance performance). From ¹⁰ we note that a social
robot “can be perceived as the interface between man and technology. It is the use of socially acceptable
functionality in a robotic system that helps break down the barrier between the digital information space and
people.” As robot design becomes modular ²⁷ and body extensions inspire art performances that
question human capacities,¹⁶ an anthropological study ³² compares HRI to the type of
connection expressed in earlier religious rituals between gods and humans, pointing out the influence robots
could have on us in the near future. As we are currently shifting into a post-humanist technologized era, where
humans are extending their capacities using exoskeletons and various connected devices, the definition of the
human body and how it interacts with its environment changes accordingly. Our paper investigates how these
paradigms affect our creativity and their impact in collaborative social practices like dance.³³
mentions the etymological analogy between dance (from Indo-European ten, root for tension) and emotions (from
latin émovere: or set to motion). In our quest for a meaningful interactivity between performers
and robots, we analyze the impact of these anthropological projections on dance. In the following pages, we
describe how we create our movement sequence by discussing our working hypotheses and methodology and explaining
our working phases leading to the concept of human-to-robot (H2R) hybridization. We then adapt and test the
sequence on several human performers. We later discuss the results and perspectives of this experimentation and
its implications into current dance practices.
Questions and Methodology
Our approach focuses on the process of creating an original choreographic language inspired by robots,
influenced by somatic practices and embodied intelligence. According to artist Louis-Philippe Demers, any
abstract inert shape “can become fluid, organic and eventually anthropomorphic by the sole means of
contextualization and movement.”⁹ Our goal is to check to what degree the concept of
anthropomorphism enhances creativity in HRI and influences artistic research. Among our working hypotheses, we
investigate how the shape of a robot can influence the performer in generating unintentional movements similar
to kinaesthetic responses ³ (1) ? How can a dance sequence be reproduced more easily depending
on the type of robot? Does the feedback of the performer change once the robot type has changed? Another area of
investigation concerns the creative process, supporting the idea that working with an avatar substitute of a
robot that we define later as its digital twin, can favor a hybridization state between the body or the
performer and the robot’s virtual body. In this way, we underline the process through which the role of
the robot shifts from tool, to working companion. Further on, we wonder if this new symbiotic individuation
between robot and human can influence the output of artistic performances. The robots involved in our study have
similar dimensions, although initially built for different outcomes. As a starting point, we use the movement
material implemented in the HRP-4 robot for the Le mythe d’Immorta (2022) performance, part of
the Co-Évolution, Co-Création et Improvisation Homme-Machine (CECCI-H2M) project. The
humanoid robot and an industrial arm are programmed in an analogous series of movements based mostly on the
rotation of upper limbs and head for the humanoid robot. We then test the dance sequences through a case study
within a group of dance students.
Digital twin robots as interactivity facilitators
The students are asked to memorize, then improvise the analogous movement sequence indicated through video
projection by different dance partners. The filmed sequences have the same format and duration (approx. 1min).
First sequence is taught by a humanoid robot, second one by an industrial arm and the last one by a human
performer. The sequences have the same structure but differ in the quality of movement, depending on the
teacher. Next the dancers are asked to freely interpret the movements and collectively improvise a free dance
sequence, recycling the movements they considered the most inspiring. The method of analysis is a question form
of 23 questions that the participants are asked to fill in at the end of the experiment.
Experimentation Phases When the tool becomes a partner
In the West, robots are often designed to mimic human behavior, mostly as tools ¹⁴ replacing human
labor or assisting humans in complex tasks. However, in creative contexts their function is more ontological in
the sense they might contribute to a certain creation of meaning in inference with the art process that contains
them. For artist and researcher Simon Penny, cognition does not occur exclusively in the brain, being “far
from a logical manipulation of symbolic tokens.”²³, ²⁵ Arguing mental
processes like inspiration are embodied, integrated within non-neural bodily tissues (thus extended into
artifacts, social systems and cultural networks), he also points out its dynamic nature. Through our
research-creation experiments, we imagine movement sequences that challenge the concept of anthropomorphism in
intelligent interactive robots. As the stage mediates their encounter, it brings new possibilities of
expression, and consequently of interaction. In our particular case, we apply the term of interactivity
⁶, ⁷ to define the relation between humans and robots. Initially inspired by human
interaction, interactivity appeared with the development of new technologies and media, setting up the premises
for an ongoing process of transformation between man and machines. Mixing living and nonliving, reality and
simulations, physical and virtual, interactivity facilitates a permeability between human and technology. As
stated in ¹ , contemporary interactive interfaces “escape the total control of man and create a new
situation where the latter no longer has an exclusively active role in front of his tool. It is rather an
exchange, an interdependence characterized more and more by their interactivity.”
Confronting paradigms concerning embodiment
¹ makes an analogy between the way technology, especially virtual interactions, shapes the understanding
of the human body. Individuals seeking to expand, alter or fragment their body, are indicators of the ongoing
shifts operating through our daily interactions. The identity being multiplied, diluted or absorbed into the
digital world, the tendency is to project the same expectations onto the human body. Neuroscience, robotics or
art do not consider embodiment and anthropomorphism in the same way. In ¹⁸, the body is defined as a
set of organic processes that go beyond the experience of sensation and movement. It might include social
networks such as families and culturally constructed artifacts. In parallel, anthropological terms like
“body domestication” ¹⁷ analyze how the body is being segmented in units according to the
field of research that address it. In arts, pioneers like Stelarc have always considered their body as a
playground for technological experiments ²⁹. With his third arm or spider legs performances, the
artist blurred the lines between what is human and what is machine through very original settings. Whereas
roboticists design robots that can “emulate” humans through their resemblance and collaborative
skills ¹¹. Researchers demonstrate how the overall shape of a robot plays a key role in invoking
desired emotions in users ¹⁴, with studies measuring the degree of acceptance of humanoid robots
²², ¹⁹, influenced by the awareness of our own “body
schema.”¹³ Additionally, concepts like “social uncanniness” ¹² examine how
our need to be unique and to engage in authentic interactions is being impacted by the development of social
robots.
The idea to start experimenting with a robot’s virtual avatar, appeared while working in the laboratory
with the roboticists of the Interactive Digital Humans (I.D.H.) team in Montpellier. The HRP-4 robot being
little available at that time, we had an artistic residency in which its virtual double was representing the
real one. The robot executed a series of movements inspired by power postures of political leaders sitting on
chairs. While improvising, its imaginary corporeality triggered different kinetic responses in the performer, so
we were curious to look further and understand this phenomena. In ²⁸ we note the distinction between
virtualized robots and the simulations that are necessary for the functioning of robots in the real world. To go
further, in our current study we employ the term of “digital twins (2)” over simulations,
due to their real-time data processing and possibility to study multiple processes in various simulations.
Hybridization as a result of creative interactivity
¹⁷ traces back the hybridization process between the human body and the technological objects from
as early as prehistory, when the anatomy of our hand adapted to the manipulation of objects like carving tools.
He argues that the notion of reflexivity occupies a central place in the way anthropomorphism operates, citing
the cultural anthropologist Victor Turner for whom the human is more than a performative animal, being something
close to an animal that performs oneself. In analogy, creating a robotic dance sequence is also a reflexive
process. The human plays a double role-movement initiator and receiver (in the sense she/he responds to the
proposal done by the robot) whereas the robot becomes a medium-translating with its mechanical constraints the
body architecture it is supposed to embed, without much alternating the initial model. To deepen our
understanding of the concept of hybridization, we define it in relation to the work of numeric artist Edmond
Couchot. For him the hybridization is always located in the experience, “more precisely in the
relationship between the sensitive, the body and its environment.”²⁸ The author of the paper
points out the work-in-progress nature of hybridization or, to cite the artist, “an evolving state, from
human to artificial and from artificial to human.” In his classification of artificial agents, Couchot
makes a distinction between robots and artificial humans, using the concepts of materiality and autonomy to
distinguish between the two. For autonomy his classification is orientated around two polarities: the puppet
avatar (minimal level of autonomy) and the autonomous actor (maximal level of autonomy). Concerning its
definition of materiality, he structures his observations around the technaesthetic experience from french
“expérience technesthésique” as being “a sensitive experience lived in the
technical act [which] constitutes a kind of perceptual habitus, sensory knowledge, shared by each member of a
society and shaping his ways of being and acting, of thinking, by ways different from those of language and
symbolic thought.” It is interesting to note this experience is implying the dissipation of the self,
relying on the remains of other people’s experience with technology. This relates to the observation of
Becker ¹⁷ for whom the history of shared gestures, traces back the evolution of humanity in its
relation with technology.
The Human in the loop
One of the challenges of our research-creation is to explore this intermediary dimension that operates between
the quality of movement of two distinct entities (human and robot). In ³³ an experiment done with a
Poppy robot, favors the emergence of the concept of kinaesthetic empathy-illustrated by the gap and tension that
operates once the gesture is being transmitted from the human to the robot. In our particular case, in real-time
improvisations, once the HRP-4 robot is programmed to reproduce a dance sequence, the choreography might be
altered according to the technical characteristics of the robot-state of actuators and battery, code glitch,
joint limitations. We call this type of inference “parasite movements,” requiring a sensory-motor
adaptation both from the machine and from the human. In comparison, while working with digital twins, the
adjustments are done at the video-projection level. The concept of morphing ³³ borrowed from image
processing offers an interesting analogy to movement production, defining a process of continuous rehabilitation
and readjustment of the bodies and their transformation into shapes. For one of our research-creation
experimentation trials, we projected the figure of the robot on the body of the performer. To our surprise the
image overlapped, resulting in a hybrid figure of both robot and machine. After several trials, the performer
adapted to this unexpected robot specificity, like in a taming process and a symbiotic figure, half-human,
half-robot resulted.
Case study of a human-to-robot dance interaction involving anthropomorphism
Through our case study we investigate how different movement material is experienced by performers while
dancing with robots. The focus of our paper is to determine how users experience ”qualia”
(3) while learning through imitation a dance sequence consecutively demonstrated by human and
non-human agents. Arguing that body and mind are the same expression of an organic process, ²⁰
describes arts and aesthetics as the culmination of humanity’s attempts to find meaning. After working
with the double twin of the HRP-4 robot for the Le mythe d’Immorta performance, we wanted to
understand how robots are being perceived by their human partners during creative practices. Engineers can
easily pre-program intentions related to specific human emotions or behaviors in robots using an affective loop
approach ⁸. This approach focuses on the robot’s capacity to engage humans into affective exchanges
and therefore attribute meaning to their behavior. In 2010, a simplified user interface that creates body
motions ²¹ was implemented in an HRP-4C robot for a dance performance. The robot does basic
synchronized dance movements surrounded by four dancers, giving the impression of a perfect copy of the human
performer. As stated by ²⁴, we expect from intelligent, self-conscious artworks to surprise us. To
tackle this expectation, we imagine interactions that focus on spontaneous, ”parasite” movements and
unexpected behaviors in robots, questioning an “effect of presence” ³⁴ on stage. To
explore the potentialities of these creative evolutionary interactions, different scenarios were programmed and
tested, involving imitation and random movements. This approach enables us to understand how artificial and
organic bodies may adapt to one another and reach a hybrid state, specific to our working environment.
While conceiving our first dance sequence with HRP-4, we had to take into account different constraints: how
the robot stabilizes its center of mass, its autonomy while standing and the safety mechanism that allows it to
move. This context made us initially consider a seated dance interaction. Inspired by work of famous
choreographers like Ohad Naharin’s Echad mi Yodea (1998), Anne Teresa De Keers-maeker’s
Rosas danst Rosas (1983) or Anna Halprin’s Seniors Rocking (2005) who used chairs in
their choreography, we chose to work on the postures of famous political leaders. The fact that they were
sitting, thinking, but through their reasoning they influenced the outcome of our daily lives transformed the
figure of the robot into a totem-like figure inhabited by power gestures. The dance sequence was transposed into
the HRP-4 digital twin, on stage during the CECCI-H2M project. In a different setting of this project, the
performer interacted in real time with both autonomous systems of the performance-using the same dance sequence
with the performance’s virtual environment E.V.E.(non-anthropomorphic) as well as the virtual HRP-4
(anthropomorphic). The out-come of this experience was that the repetitive movements of the HRP-4 generated a
feeling of oppression and limitation in the performer, triggering the need to deconstruct them in various
movements similar to the kinaesthetic responses we mentioned earlier. Whereas the responsiveness of E.V.E. made
her easily forget the initial sequence and follow the rhythmic proposed by the artificial agents. Both systems
were identified as stage partners by the performer, with the robot evoking a feeling of absence or “ghost
in the shell” (4) phenomena. Once having these insights developed through our creative process,
we felt the need to confront them to a broader context. The next part of your experimentation was a case study
proposed to 25 dance students who tested a dance sequence conceived for the humanoid HRP-4 robot and taught by
it, as well as a FRANKA industrial arm and a human performer. Compared to the version inspired by power
postures, the HRP-4 robot was moving its upper limbs, hips, head and torso, but this time standing. New
movements, corresponding to unintentional movements similar to kinaesthetic responses were added in the loop to
simulate what we defined earlier as parasite movements. Participants were initially asked to reproduce the
movements then improvise freely applying some of the gestures they remembered. They were encouraged to explore
their senses (including the proprioception) and forget any personal projection regarding aesthetics and how the
sequence should look like.
Discussion
In ¹⁵, ¹⁰ scholars link the notion of agency to the question of anthropomorphism,
citing D.C. Dennett’s theory on “intentional systems”. In art contexts, besides the
object’s kinetic behavior, another factor of influence is the spectator’s strategy to understand and
predict the behavior of the performing object. We wanted therefore to open up our research-creation process to
external participants, organizing a practical experimentation with dance students. Among the questions the 25
participants in our study were asked to respond to, some concerned the movement imitation learning process. Most
of the participants found no significant differences in the way the robot (56%) presented the movement, compared
to the human (76%). However, most of the participants strongly agreed it was easier to follow the movements of
the humanoid robot (64%), compared to the industrial arm (8%). Concerning the quality of movement, most of the
participants believed both the humanoid robot (44%) and the robotic arm (44%) had intentional movements. Whereas
56% agreed it was easy for them to detect the “parasite movements” of the sequence. A slight
difference was observed when participants were asked if they could distinguish intentional from unintentional
movements, with 28% of the participants agreeing and 24% strongly agreeing it was easy. A significant part of
the participants was undecided (36%) whether robots are strange creatures, with 24% strongly agreeing they are
and 20% somehow agreeing they are not. Similar distribution considering whether robots have consciousness, with
28% somehow disagreeing and 24% somehow agreeing. As for the creative interactivity, the majority of the
participants- (68%) felt the need to add other movements, once the sequence became repetitive. None of them
agreed that they usually apply the imitated movements when dancing, while very few some-how agreed the movements
were natural for the HRP-4 robot (8%) and Franka robot (16%). It is interesting to note that for the Franka
robot, 4% of the participants strongly agreed its movements were natural, while none for the humanoid robot.
Regarding emotions, only 12% of the participants somehow agreed the HRP-4 robot communicated emotions through
dance with a majority (64%) disagreeing and 24% being undecided. As for the Franka robot’s emotions, 80%
of participants disagreed it expressed them through dance while 8% were undecided and 4% somehow agreed.
Interestingly enough 8% of participants believed the Franka robot communicated emotions through its dance. Most
of the participants of our study (64%) could spend more time understanding movement through robotic interaction.
A hybridization process was also noticed after the sequence, when the participants were asked to freely
interpret the robot motions they experienced earlier. This facilitated the exploration of a state where senses
were more present and spontaneous movements appeared more easily. A state that we define as creative, in the
sense that it allows body expressiveness inherent to the specificity of the embodiment - where by imitation they
appropriated and transformed the robotic dance sequence. While moving, their bodies seemed inhabited by the
robotic presence. Experimentation output is available before (5) and after (6) the robot
trial.
Limitations
Regarding our intention to slightly modify the sequences, getting to see them in a particular order (HRP-4,
followed by the Franka robot and the human performer) might have influenced their answers. Some of the
participants were already familiar with the movement of the HRP-4 robot due to previous working sessions
organized earlier that year. Compared to working with digital twins, the results might also be different when
working in real time with the physical robots.
Moreover, by applying the notion of digital anthropology to the digital twins of the robots used in our
experiments, we might have omitted some interesting results depending on other shapes and dimensions. We should
also not forget that even though artificial agents can simulate intentions and affects, our way of interpreting
them—or the notion of qualia we mentioned earlier—is always different. Having implemented the
analogous dance sequence of the humanoid robot on the industrial arm, we experienced differences (especially in
relation to posture symmetry but also speed and jerk) with the industrial arm proving more compliant given the
type of actuators that executed the movement.
Future perspectives using emerging robotic technologies
The next phase of our research will be testing the same scenarios with the physical version of the robots. In
this way we can determine whether the perception of the performer, as well as his/her quality of movement can be
impacted differently when working in the same conditions, with the real, mechanical body of the robots. This
will also allow us to understand how the hybridization process is influenced by movement imitation phases. Our
first empirical results encourage us to establish a scientific laboratory study, measuring kinaesthetic empathy
through the detection of mirror neurons, using electroencephalogram (EEG) signal processing.
Conclusion
Learning by imitation, then establishing a creative interaction using digital twins allowed us to research on
the qualia each participant experienced and improve the overall quality of movement through dance improvisation.
This type of creative interactivity involved the hybridization between humans and robots, generating new visual
shapes when video-projected, as well as an original type of movement material. Hopefully our remarks on digital
anthropomorphism will further stimulate exchanges between roboticists and artists, anticipating a new
individuation phase in the overall robot-human relation.
*S-S. Cîrcu is with AIAC Laboratory of Paris 8 University and with the CNRS-University of
Montpellier LIRMM, UMR5506, Montpellier, France. Email: sorina.silvia@gmail.com
†C-Y. Chen is with National Tsing Hua University, Taiwan and AIAC Laboratory of Paris 8
University, France
(1) a spontaneous reaction to motion which occurs outside you; the timing in which you respond to
the external events of movement or sound; the impulsive movement that occurs from a stimulation of the senses:
ie. someone claps in front of your eyes and you blink in response; or someone slams a door and you impulsively
stand up from your chair., Bogart, Anne, Landau, Tina. The Viewpoints Book, p. 11
(2) acc to IBM, while a simulation typically studies one particular process, a digital twin can
itself run any number of useful simulations in order to study multiple processes.
(3) acc to Stanford Encyclopedia of Philosophy, “philosophers often use the term
‘qualia’ (singular ‘quale’) to refer to the introspectively accessible, phenomenal
aspects of our mental lives.The status of qualia is hotly debated in philosophy largely because it is central to
a proper understanding of the nature of consciousness. Qualia are at the very heart of the mind-body
problem.“
(4) https://www.imdb.com/title/tt0113568/
(5) https://vimeo.com/779347404
(6) https://vimeo.com/779363288
References
1 R. B. Baddoura-Gaugler, L’homme et le robot humano ̈ıde: Transmission, Re
́sistance et Subjectivation, Ph.D. Dissertation, Universite ́ Paul Vale ́ry-Montpellier
III, 2013.
2 D. Bisig, Generative dance-a taxonomy and survey, In Proceedings of the 8th
International Conference on Movement and Computing, 2022, 1–10.
3 A. Bogart, T. Landau, The viewpoints book: A practical guide to viewpoints and
composition, Theatre Communications Group, 2004.
4 C. Breazeal, and R. Brooks, Robot emotion: A functional perspective, Who needs emotions,
2005, 271–310.
5 C. Breazeal, J. Gray, G. Hoffman, and M. Berlin, Social robots: Beyond tools to partners,
In ROMAN 2004, 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog
No. 04TH8759), 2004, 551–556, IEEE.
6 M. Bret, M.-H. Tramus, A. Berthoz, Interacting with an intelligent dancing figure: artistic
experiments at the crossroads between art and cognitive science, Leonardo 38(1), 2005, 46–53.
7 M. Bret, M.-H. Tramus, A. Berthoz, Creation no. 2, la funambule virtuelle, michel bret, marie-he
́le`ne tramus, alain berthoz, Hybrid, Revue des arts et mediations humaines (2), 2015.
8 L. Damiano, P. G. Dumouchel, Emotions in relation, epistemological and ethical scaffolding for
mixed human-robot social ecologies, HUMANA, MENTE Journal of Philosophical Studies 13(37), 2020,
181–206.
9 L.-P. Demers, Machine performers: Agents in a multiple ontological state, 2015.
10 B. R. Duffy, Anthropomorphism and the social robot, Robotics and autonomous systems
42(3-4), 2003, 177– 190.
11 P. Evrard, A. Kheddar, Homotopy switching model for dyad haptic interaction in physical
collaborative tasks, In World Haptics 2009-Third Joint Euro-Haptics conference and Symposium on Haptic
Interfaces for Virtual Environment and Teleoperator Systems, 2009, 45–50, IEEE.
12 G. Hoffman, The social uncanniness of robotic companions, In Robophilosophy, 2020,
535–539.
13 M. Hoffmann, Body models in humans, animals, and robots: mechanisms and plasticity, Body
Schema and Body Image: New Directions, 2021, 152.
14 J. Hwang, T. Park, W. Hwang, The effects of overall robot shape on the emotions invoked
in users and the perceived personalities of robot, Applied ergonomics 44(3), 2013, 459–471.
15 E. A. Jochum, and L. Putnam, Computation as medium: agency and motion in interactive art,
Akademisk Kvarter 16, 2017, 9–21.
16 E. Jochum, L.-P. Demers, B. Vorn, E. Vlachos, P. McIlvenny, P. Raudaskoski, "Becoming cyborg:
Interdisciplinary approaches for exoskeleton research", In Proceedings of EVA Copenhagen 2018: Politics of the
machines-Art and after, Electronic Workshops in Computing, 2018, 1–9, BCS Learning and Development Ltd.
17 B. Joffrey, Humanoïdes, expérimentations croisées entre arts et
sciences, 2015.
18 M. Johnson, T. Hohrer, We are live creatures: Embodiment, american pragmatism, and
the cognitive organism, Body, language, and mind 1, 2007, 17–54.
19 F. Kaplan, Who is afraid of the humanoid? investigating cultural differences in the
acceptance of robots, International journal of humanoid robotics 1 (03), 2004, 465–480.
20 J. Mark, The meaning of the body: Aesthetics of human understanding, 2007.
21 S. Nakaoka, S. Kajita, K. Yokoi, Intuitive and flexible user interface for creating whole
body motions of biped humanoid robots, In 2010 IEEE/RSJ International Conference on Intelligent Robots and
Systems, 2010, 1675–1682, IEEE.
22 T. Nomura, K. Sugimoto, D. S. Syrdal, K. Dauten-hahn, Social acceptance of humanoid robots in
japan: A survey for development of the frankenstein syndrome questionnaire, In 2012 12th IEEE-RAS
International Conference on Humanoid Robots, Humanoids 2012, 242–247, IEEE.
23 S. Penny, Emergence, agency, and interaction—notes from the field, Artificial
Life 21(3) 2015, 271–284.
24 S. Penny, Improvisation and interaction, canons and rules, emergence and play, The
Oxford Handbook of Critical Improvisation Studies 2, 2016, 401–23.
25 S. Penny, Sensorimotor debilities in digital cultures, AI and SOCIETY 37(1), 2022, 355–366.
26 B. Romic, Negotiating Anthropomorphism In The Aida robot, International Journal of
Social Robotics, 2021, 1–11.
27 T. Siedel, M. Hild, M. Weidner, Concept and design of the modular actuator system for
the humanoid robot myon, In International Conference on Intelligent Robotics and Applications, 2011,
388–396, Springer.
28 R. Sohier, "Degré d’autonomie et degré de matérialité
d’après l’œuvre d’edmond couchot: d’une matrice des humains artificiels
a`une recherche-création expérimentant la co-avatarisation", Hybrid, Revue des arts et
médiations humaines (9), 2022.
29 E. Stephens, T. Heffernan, We have always been robots: The history of robots and
art, In Robots and Art, Springer, 2016, 29–45.
30 A. Stojnić, Digital anthropomorphism: Performers avatars and chatbots,
Performance Research 20(2), 2015, 70–77.
31 S. Tisseron, "De l’animal numérique au robot de compagnie: quel avenir pour
l’intersubjectivité?", Revue française de psychanalyse 75(1), 2011, 149–159.
32 D. Vidal, "Anthropomorphism or subanthropomorphism? an anthropological approach to gods and
robots", Journal of the Royal Anthropological Institute 13(4), 2007, 917–933.
33 M.-A. Villard, M. Lapeyre, "A propos d’une expérience de mouvement partage
avec un robot humanoïde: l’entre-deux comme maintien du vivant", In IRIS, number 37, 2016,
193–205.
34 P. Zaven, "Effets de présence, relations hommes-androïdes", sur Cultures Kairos,
2014.
Authors’ Biographies
Sorina-Silvia Cîrcu holds a BSc in Systems engineering from The Polytechnic University of
Timisoara, a MSc in Philosophy from Vest University of Timisoara and a MA in Theatre Directing from Nanterre
University in Paris. Transdisciplinary artist and researcher, she is currently preparing a Ph.D between dance
and robotics (AIAC laboratory of Paris 8 University and LIRMM laboratory of Montpellier University). Her work
focuses on dramaturgy of the body, new media and notions like reality, perception and metamorphosis. She
worked among others with Henier Goebbels for his anthology of “Sound and Spaces”, Silvia Costa,
Michel Cerda, Theatre du Soleil, Odin Teatret.
Chu-Yin Chen is an Artist and Professor in Digital Art at Paris 8 University. Since 2019, Chair Professor at
National Tsing Hua University (Taiwan). Her creations, based on Artificial Life and complex systems develop
interaction modes between audience and virtual creatures showing autonomous and evolving behaviors. Her
digital artworks have been shown in numerous international exhibitions. Her research articulates on two
overlapping areas: 1] Digital Creation using algorithms of complexity and emergence, and 2] Metacognition and
Elicitation of the processes of creation, enaction and aesthetic reception, via psycho-phenomenology and
mindfulness.
The Revolution Will Launch in the Garden: Politics of representation and vegetal intellig(senti)ence
PaulRoseroContreras
Universidad San Francisco de Quito - Dlab, Dos Islas Estudio Quito,
Ecuador proseroc@usfq.edu.ec
Abstract
This paper describes entanglements between human and non-human actants from the perspective of creativity,
interdependence and contemporary anthropology. Drawing from experiments on self-representation and swaps from
object to subject of study, the author reflects on the potential of the vegetal world and distinct forms of
intelligence to propose a subversive anti-anthropocentric view in a planet terraformed by plants and other
beings.
A living species is a creative species, shaping its own environment to make it habitable and functional. The
first living organisms completely altered the planet in the Great Oxidation Event. Plants have transformed Earth
in such a way that made the planet livable and breathable for other living beings. In this revolution, plants
not only create images, but they themselves are images. Biology allows us to understand creation as the result
of a more- than- human cognitive process; a neurobiology of plants that regards them as sentient beings in an
interdependent relationship with their environment, capable of learning and responding, constructing an
imaginary of possibilities.
Drawn to this idea of plants as creative beings, I have been developing a project El Pensamiento de las Plantas
[The Thinking of Plants] that speculatively imagines the visual representation of plants from the perspective of
one’s self. This photographic series is influenced by Indigenous Epistemologies, biological concepts, and
contemporary anthropological postulations like those of Canadian anthropologist Natasha Myers. Her idea of
Planthropocene activates a radical political potential by asserting that humans are not alone on this planet,
and that Anthropos are not the only ones capable of constructing habitable worlds. The Planthropos brings us to
the Planthropocene: part plant and part human, it embodies the involuntary interdependence of plants
and animals that is committed to collective prosperity. Unlike the Anthropocene, the Planthropocene is not a
geological epoch, but rather a “scene” or episteme: a way of understanding life.¹
This perceptual process involves “planting” our own senses to develop an appreciation for the
expressive and curious manners of vegetal life. In the 1970s experimental art scene of the U.S. post-war
avant-garde, artists and scientists explored this idea in what would later be called bio-sensing art. This
exploration combined the nascent fields of electrobiology, cybernetics, and the ecologies of information
systems. Artists, like the North American Richard Lowenberg, meticulously investigated the intellectual and
multisensory properties that connect humans to other life forms.² From orcas to tropical plants, these
studies encompassed ideas around interspecies collaboration, nonhuman agency, and the advent of artificial
intelligence. However, despite the fact that these plant explorations were sowed in open transdisciplinary
collaboration—and have broadened our understanding of the touch, smells, colors, textures, and shapes of
the world we inhabit —the ontology in which this practice was fundamentally rooted still
measured the cognitive and sentient capabilities of plants in molecular and chemical terms; in other words, it
was reliant on a rationally descriptive perspective. On other occasions, the experiments were transformed into
performances that sought out hallucinogenic experiences through interspecies communication in real time.⁴
My own project acknowledges this legacy, but I try to broaden the worldview by incorporating fundamental
concepts from indigenous epistemologies that show us how our own aesthetics and sensorial practices are
intertwined with vegetal life, to the extent that thoughts, feelings, and the production of an imaginary all
function symbiotically. This type of practice in becoming vegetal necessitates a “plant
intelligence,” entering what Natasha Myers terms “vegetal sentience”: a broader (mind)space
activated, not only by new forms of thinking about our relationships to plants, but also by their perception of
the world and how they make sense of it. Thus, I do not limit myself to assumptions about intelligence and its
associated indicators, nor to an idea of multispecies interconnectedness that is mediated by human technology.
In truth, this so-called “vegetal sentience” is a concept that has been recognized by indigenous
communities around the world for millennia. For instance, from the Andes to North America, it is observed that
the relationship of indigenous peoples to the natural environment has always been understood beyond dichotomies:
it is a dynamic flow and constant interdependence where the act of giving thanks and requesting permission from
the earth form a chain of manifested reciprocity. In this vein, Robin Wall Kimmerer—ecologist and enrolled
member of the Citizen Potawatomi Nation—speaks of mosses as an act of life in and of themselves, bringing
services to the species with whom they interact, with the moss-tree relationship being one of the oldest and
ripest in history. Given this context, I believe that it is necessary to foster a movement away from the notion
of thinking and towards one of feeling.
What is thinking?
If we define intelligence as the capacity to solve problems, then plants and animals alike share this ability.
Hence, in El Pensamiento de las Plantas, thinking is understood as a collective act in creating interspecies
images that explore the limits of language. This project is a visual reponse to a moment in the past, or to an
exterior stimulus beyond the human subject that experiences and captures an image. Rooting this process in a
collective act with the forest is a way of addressing the problem of politicized representation of nonhuman
beings, and moving towards a strategy for self-representation. The blurry images constitute intuitive thoughts,
or perhaps ephemeral translations, that are closer to the feeling of liberation rather than a logical,
neoliberal interpretation.
Historically, the natural world—understood as landscape in Western art history—has been regarded as
an object, a representation mediated by retinal perception. In the twentieth century, we witnessed the advent of
certain avant-garde movements that took the natural world beyond its representational state, but the canon
remained unbroken. Hitherto, nature as object had been filtered through the human body—almost always male
and white—and made an image from the perspective of Anthropos. It seems that posthumanism, the humanities,
and the arts still face the challenge of shifting Eurocentric attention towards other forms of Planthropocenes.
Nevertheless, there are cases of indigenous artists, rural dwellers, and activists who work in social justice,
both within and outside of official institutions, who recover ancestral lands in concrete ways.
In El Pensamiento de las Plantas, I try to go beyond the Western tradition, acknowledging the environment not
as an object, but as a subject: one that is not only unto one’s self, but also among an ecology of selves.
This idea is enriched by the proposition put forth by anthropologist Eduardo Kohn in his book How Forests Think;
after four years of experiential research with the Quechua-speaking Ávila Runa, in the Upper Amazon of
Ecuador, Kohn proposes an anthropology that goes beyond the human. The book’s central premise is developed
around the notion of what constitutes selfhood. Kohn equates individuality with thinking, in the sense that if
something or someone—a broad category that goes beyond humans— experiences intent, purpose,
function, or meaning, then that something or someone is alive or “enchanted.” Therefore, a forest is
alive and thinking, just as a dog, a jaguar, a skunk pig, or a plant are all alive and thinking.
Kohn elaborates on this discussion around individuality by explaining that life is a process of signs through
which, citing scientist and philosopher Charles Peirce, any living being is “something which stands to
somebody for something in some respect or capacity.” ⁵ As such, animals, plants, and spirits should
be understood as beings-unto-their-selves, co-defining what is human as part of a larger planetary dynamic.
Kohn suggests that there are many different types of selves, from the “skin-bound organism” to one
that is distributed across many bodies,⁶ like a group of people or an ant colony. To understand this
concept, he says, one must first provincialize language, to challenge social theory which conflates the idea of
representation with human language. The latter depends on a symbolic representation based on
conventional—or “arbitrary”—signs that are embedded with other such signs that
constitute an entire representational system. These fixed signs are contextualized and made to relate to the
object they reference, and because this is how signs work in language, we assume that all representational
processes exhibit these same associated linguistic properties. But the symbolic representation of forms is
different from the human representation of forms: it goes beyond what is represented by humanity. It is
knowledge outside of the human, existing a priori on the planet, continually emerging and changing. And in this
manner, the forest and the nonhuman world think, Kohn concludes.
From an altogether different point of reference, the Italian plant neurobiologist Stefano
Mancuso—considered to be one of the founders of the field—reaches the same conclusion. Mancuso makes
a case for the existence of plant intelligence based on sensitivity: a capacity to learn and retain memory that
enables plants to continuously monitor chemical-physical parameters, thus allowing them to understand and
respond to their environment. Therefore, forests are their own selves because they communicate, feel, represent,
and modify: they not only make possible but also host the future. ⁷ Forests contain both the human
representation of forms, and the preexisting representation of a being in one’s self.
Thinking is also a part of memory. If we did not have memory, would we even think? In the case of humans, our
brains and bodies are instruments wherein memory resides. But memory is also transcorporeal and collective, as
is thought. A thought is a response to that memory. A reflection is a brief recollection. The leaves of
Mimosa pudica—or sensitive plant—clench instantly in a remembered moment. Could it be that
in their memories, we humans are their predator? Or are they simply telling us not to touch them by
retreating?
Mind and cognition seem to be defined by their very incorporation into the world—through simple
sensory-motor acts—in an endless and interspecies interaction of sensations: a coevolution and
co-definition of what I am on the inside, with what lies outside that defines me. This is a type of
planthropocenic phenomenology that resonates with the ideas of philosopher Maurice Merleau-Ponty, since we must
understand that the idea of humanity cannot be based on binary opposition to animals or subhumans or the
more-than-human. But then, what do we do with this knowledge of a plant’s sentience? What happens when we
acknowledge the fact that we are surrounded by nonhuman sentient beings and we start to live as if plants were
our mentors, granting them the respect they deserve?
In my case, I proceed with projects like the one described in this text. From the political perspective of a
nation-state, it could be a recognition of the Rights of Nature, like the one that was surprisingly
included in the 2008 Constitution of the Republic of Ecuador, which made this country the first in the world to
grant this legal status to nature. ⁷ However, since then we have come to question if these rights really
are a transcendent innovation, or if they are mere legal rhetoric, or even just a political project within a
world that is dictated by extractivist capitalism. What is clear is that a decolonial and antiracist analysis of
the nonhuman is imperative in thinking about interspecies politics beyond the theatre of representing diversity
that is typical of demagoguery.
Seen from another perspective, the changes that occur upon noticing the sentience and intelligence of other
species reveal an attentiveness towards simple organisms as a means to understanding complex behaviors. For
example, the same Mimosa pudica that reacts by touch, or a fly that has existed for thousands of years,
which knows how to avoid hitting walls and feels the parameters of the room in which it flies, is in one’s
self because this knowledge is self-sustaining, developed within its own epistemological world. These actions
that seem like simple behavior are ones that took humans hundreds of years to replicate.
Memory of water
El Pensamiento de las Plantas is also a series about memory retained in a vegetal body that, like the human, is
largely comprised of water: a liquid memory that overflows, becoming transcorporeal. Vegetal bodies, human
bodies, animal bodies: vessels of water that also provide life at the same time. A tree as a water vessel is a
being in one’s self, existing beyond us, and yet whose existence constructs us. Humans are dependent on
plants. In the ebb and flow of natureculture, as Donna Haraway puts it, I am also a being in one’s self,
whose thoughts manifest themselves. In the name of alliance, discrepancy is permitted. My own thinking and
memory can provide an answer, as one representation—and representative—in the world, distinct from
the response of another nonhuman body. In addition, my project is embedded in a factual story ⁹ as a way
to speculate a response derived simultaneously from the notions of both reciprocity and difference. This story
recounts the tragic death of German documentary filmmaker Dieter Plage, who fell from a hot air balloon while
filming a movie about the Sumatran rainforest canopy in 1993.
Since he began his career in the late 1950s, Dieter Plage became one of the most recognized nature
cinematographers and documentary filmmakers. To him we owe the unforgettable and dramatic recordings of humans
interacting directly with wild animals in Africa, as well as other films about terrestrial and marine species
that were widely broadcast on television networks in Europe and the United States. Plage, like the German
naturalist Alexander von Humboldt in the 18th century, understood that the rainforest canopy was still an
unexplored aerial world, one in which he had to reach with his camera.
In an era before the advent of commercial drones—or any other sophisticated remote-controlled flying
device— Dieter Plage worked with British aeronautical engineer Graham Dorrington to build a hot air
balloon that would allow him to shoot a film about the rainforest canopy, a site that was nearly inaccessible up
to that point. During the filming of the documentary, sponsored by National Geographic, the balloon got
entangled in a treetop. Because the balloon was made for a single passenger, Plage alone faced the dilemma of
either trying to disengage the trapped aircraft and continue the flight, or to save his camera. This predicament
would be his last: as he tried to disentangle himself from his cabin, Plage slipped and fell to his death. El
Pensamiento de las Plantas, then, embodies the visual material that was kept within the water-vesselled bodies
of trees who were eye witnesses to this tragedy: water transformed into an image.
The world, too, is made up of absences. Mimetic insects disappear altogether on leaves, camouflaged against the
hampered vision of others. These insects live vis-à-vis their absence. They are, only to the extent that
they disappear when faced with another's gaze, including our own. There is no sense of reciprocity. It is
absence, being in one’s self, that even allows for the possibility of existing. Hence, absence also
creates the world.
My project includes an inflatable sculpture called Obituary: a scale model replica of Plage’s crashed
aircraft.2 With it, I wanted to tell a story of death, an absence that has created a legacy, a way of seeing,
and the gaze's potentiality for reciprocity. Assembling this way of seeing is connected to non-Western ethical
principles, establishing an understanding of a being's embodiment in one’s self that is contained within
the natural world. This is a way of looking at the mind of the universe, as the scientist Francisco Varela would
put it.
In 1940, Jakob von Uexküll, a German biologist who specialized in animal ethology, wrote that “a
spider’s web is...formed in a ‘fly-like’ manner, because the spider itself is
‘fly-like.’ To be ‘fly-like’ means that the body structure of the spider has taken on
certain of the fly’s characteristics.” ⁸ A web is a physical extension of the spider—a
precise and geometric representation—that allows it to capture insects so efficiently that the web
resembles some mutual pact between the creatures. Being attuned to another being and penetrating its
phenomenological world requires, in a sense, an ontological blurring: which part of the web is fly and which
part is spider? Or, what part of the photographs in El Pensamiento de las Plantas is forest, and what part of
them is me? The mutability of senses in this project points to how certain attempts at communication among
different species can also embody another type of becoming that spills and blurs across ontological borders. In
this case, between humans and plants. This “vegetal becoming” does not negate the fact that the
human body is always present behind the image. Rather, it opens up the potentiality for blurring, a place from
which my project plants questions around the notion of what it means to be human, while at the same time
envisioning both humans and plants as fellow astronauts.
In other words, we no longer need to avoid representation as if it were just a malevolent Eurocentric process
of dissociation between things and words. Instead, we push for a representation that embraces true access—
and an overflowing experience—to the world's richness.
References
1 Andrés Lomeña, “Qué se siente al ser una planta,” Huffington Post:
Blogs, March 27, 2020,
https://www.huffingtonpost.es/entry/que-se-siente-al-ser-una-planta_es_5e7cc9a4c5b6256a7a2634a8?fbclid=IwAR2fG1ETA6w9ULZKHF-5RJ-Xv-aepV_G7l
[Last visited: 5/6/2022]
2 Alice Bucknell, “The Pioneering Artist Who Harnessed Science to Communicate with
Plants,” October 16, 2019, https://www.artsy.net/article/artsy-editorial-pioneering-artist-harnesse...
[Last visited: 1/6/2022]
3 In early 2019, Zoological Society of London (ZSL) scientists installed microbial fuel cells in
the Rainforest Life exhibition at the London Zoo. Their aim was to harness biological energy—made by
microorganisms— that would power a device which would allow a plant to take a photograph. The ultimate
goal was to power camera traps and sensors planted in nature. ZSL's Conservation Technology Specialist Al
Davies explains: “Plants naturally deposit biomatter as they grow, which in turn feeds the natural
bacteria present in the soil, creating energy that can be harnessed by fuel cells and used to power a wide
range of vital conservation tools remotely, including sensors, monitoring platforms, and camera traps. Most
power sources have limits— batteries must be replaced while solar panels rely on a source of
sunlight—but plants can survive in the shade, naturally moving position to maximise the potential of
absorbing sunlight—meaning the potential for plant-powered energy is pretty much limitless.”
https://www.zsl.org/zsl- london- zoo/news/say-green-cheese
4 Since 1974 when he founded the Bio-Arts Lab, Richard Lowenberg conducted experiments on
interspecies communications with access to orcas in British Columbia, dolphins at the San Francisco aquarium,
seals with Antarctic explorer Thomas Poulter of Stanford University, and later, the famous gorilla Koko.
5 Eduardo Kohn, How Forests Think: Toward an Anthropology beyond the Human, Berkeley, University
of California Press, 2013.
6 Ibid.
7 Farith Simon, “Derechos de la naturaleza: ¿innovación trascendental,
retórica jurídica o proyecto político?” in Iuris Dictio, 13(15), January– June
2013.
8 Eduardo Kohn, “How Dogs Dream: Amazonian natures and the politics of transspecies
engagement” in American Ethnologist 34 (1) 2007, 3–24
Here, I allow myself to draw a parallel with Gilles Deleuze’s idea of becoming: “The
animal is defined not by characteristics (specific, generic, etc.) but by populations that vary from milieu
to milieu or within the same milieu; movement occurs not only, or not primarily, by filiative productions
but also by transversal communications between heterogeneous populations. Becoming is a rhizome, not a
classificatory or genealogical tree. Becoming is certainly not imitating, or identifying with something;
neither is it regressing-progessing; neither is it corresponding, establishing corresponding relations;
neither is it producing, producing a filiation or producing through filiation. Becoming is a verb with a
consistency all its own; it does not reduce to, or lead back to, ‘appearing,’
‘being,’ ‘equaling,’ or ‘producing.’ Becoming concerns alliance. If
evolution includes any veritable becomings, it is in the domain of symbioses that bring into play beings of
totally different scales and kingdoms, with no possible filiation.”
Dieter Plage was a naturalist with deep understanding and sympathy for the creatures he filmed. His
personal style was pioneering among nature documentary filmmakers. Instead of simply filming natural history
in an orthodox way, he conceived and covered great stories about the relationships between humans and
wildlife in a dramatic way that captivated viewers. His work contributed towards making humans aware of
nature as a world to be worshipped and respected for the sake of their own survival. The tragedy that took
his life is recounted in Werner Herzog and Graham Dorrington's documentary The White Diamond, filmed in
Guyana in 2004.
Bio Elektron - A Multisensory Approach to Augmenting Dance, Combining: Biosignals, Drawing, Sound and
Electrical Feedback
1Tallinn University, 2ITI/LARSyS, 3Independent artist
Tallinn, Estonia nuno.correia@tlu.ee, deboracs@tlu.ee, helloineves@gmail.com, jaimelobatocardoso@gmail.com
Abstract
In this paper, we investigate how to augment a dance performance using a multisensory approach in a way that
communicates the dancing process as an embodied experience. We collaborated with a dancer and a media artist
over an 8-week residency to prepare and present a multisensory dance performance and a spin-off installation. We
present related work regarding key areas for this research: dance and technology in general; biosignal sensors;
multisensory media (sound, drawing and haptics); and the relation between dance and installation. We also report
on the artistic process, which was documented through seven interviews with the artists. Finally, we discuss
strategies for drawing and sonification leading to heightened embodiment; approaches for drawing and haptics
triggering impressions from the performance; while highlighting the importance of space as a unifying concept in
embodied multisensory work. These strategies and approaches can be useful for artists interested in conducting
related embodied multisensory work.
There is a long tradition of using technology in dance. Important historical examples of the intersection
between these fields are the works of Merce Cunningham in Lifeforms (1989), a software allowing to
generate new choreography;²⁵ Mark Coniglio in MidiDancer (1989), a wearable device that
allowed a performer to control media;¹⁰ and Frieder Weiss in EyeCon (2004), a motion-sensing
tool which allows movement to control several aspects of a performance.²⁹
There is extensive research on using biosignal sensors to reveal body data (such as muscle activity or heart
rate) through sound, as presented in a recent review of these approaches.¹ Biosignal sensors have also been
used in dance to visualize the inner processes of the dancers.⁹ Recent research, involving 10 contemporary
dance professionals with experience combining dance and technology, has identified potential in using technology
to reveal non-visible elements in a performance—such as the thought process of dancers or their bodily
data.¹⁷ However, there is a lack of research in combining these different modalities in dance into a
multisensory experience, particularly combining visual, sonic and haptic elements.
Multisensory experience design can transform the way we experience art: “by carefully considering
different senses and their possible interrelations it may be possible to design and shape specific human
experiences.”²⁸ An example of this approach was the Tate Sensorium exhibition at Tate
Britain (London). A multidisciplinary team of researchers and practitioners designed the exhibition so that
specific sensory elements (sounds, haptics, smells and foods) would augment the experience of four
paintings.²⁸ The possibilities of combining multisensory experiences with interaction design
“open up opportunities to explore new experiences for perceiving one's own body, its interactions with the
environment and also to explore the environment itself.”² In terms of performing arts, “the
demand for multi-sensory experiences is ever-increasing, given the rise of immersive art and theater in our
post-pandemic world.”¹¹
Zhou et al. have conducted a review of the past twenty years of dance literature in the field of Human-Computer
Interaction (HCI).³⁰ This led to the identification of four main categories of technological
approaches for dance: Physiological Sensing; Multisensory Perception; Movement Quality; and Agent Collaboration.
Through our research, we aim to combine the first two: the categories of Physiological Sensing and Multisensory
Perception.
Our research aims to investigate how to augment a dance performance using a multisensory approach
(combining visual feedback and biosignal sensors mapped to sound and haptics), in a way that communicates the
dancing process as an embodied experience. To fulfill our research aim, we collaborated with artists
Inês Nêves and Jaime Lobato (also co-authors of this paper), and performance space Elektron
(Tallinn), in the scope of the Starts.ee research project by Tallinn University. During an 8-week residency,
Inês and Jaime prepared and presented a multisensory dance performance (Bio Elektron), a spin-off
installation, and developed related software and hardware systems. In this paper, we report on this process and
its results, while discussing their implications.
Background
In this section we present background and related work regarding key areas for this research: dance and
technology in general; input technology adopted (sensors); output multisensory media used (sound, drawing and
haptics); and finally, the relation between dance and installation.
Dance and Technology
There has been an increased interest in technology for dance, in parallel to a growing importance of embodied
interaction, in the field of HCI. Within the intersection between dance and HCI, Fdili Alaoui et al. classified
categories of tools into: Generation (of new choreographic material); Interaction (in real time with performers
on stage); Reflection (on choreography); and Annotation (tools assisting the creative process).¹³
Raheb et al. categorized dance technologies in a similar perspective: Choreographic tools; Augmented
performance; Education; Research and analysis; and Games.²³ Examples of technological approaches in
these categories are: development of tools and techniques for annotation ³; tools for documenting
choreographic processes;⁶ real time interaction;¹⁴ and choreography generation.²⁵
Our work is mainly situated in the real time interaction category of tools according to ¹³, or
augmented performance according to ²³, with the exception of the spin-off installation.
Dance and Sensors
Rostami et al. generated five design concepts for interactive performance adopting bio-sensing and bodily
tracking technologies ²⁴. Likewise, Aly et al. reviewed biosensor modalities for performance from an
HCI perspective, discussing the potential of muscular activity to convey rich movement information.¹ The
research by Rostami et al. and Aly et al. include not only dance but other performance areas as well (the latter
focuses on music). The authors highlight the potential of using biosignal sensors to collect data from the body
for performing arts, which was the approach we followed in our work.
Dance and Sound
Researchers have focused on methods and techniques to create music from movement, particularly from dancers. A
relevant example on how to extract information from the body is the work of Camurri et al.⁴ Another
pertinent project studied mappings of expressivity in gesture to sound.⁵ These works focus on information
retrieval and mappings or similar aspects toward the development of the technology itself. Recently, Masu et al.
analyzed the sonic interactions that occur in a dance performance from an ecological perspective.¹⁸
They studied not only the technology and its design, but also the roles of the different actors in the design
and implementation process.
Dance and Drawing
There is a tradition of using drawing to visualize and record movement in dance performances. A milestone in
this field is Carolee Schneemann’s Tracking from 1973. Using a rope attached to the ceiling of a
train car, Schneemann held a chalk in one hand extended, so that changes in position were marked on the
walls and floor it touched.¹² Schneemann would expand this approach in her piece Up to and
Including Her Limits (1973–1976). In her series It’s a Draw, Trisha Brown dances
while painting on a large paper placed on the floor. By doing so, she collapses four dimensions: “the
three dimensions of Brown’s movement in the field above, plus the time spent doing it.”¹²
Haley conducted an extensive review of the use of drawing to visualize movement in dance, while presenting her
own piece Constructions of the Moving Body, which has as objective “to represent the experience
of watching Brown’s dance Accumulation, and evoke kinesthetic empathy through the drawn
image.”¹⁵
Dance and Haptics
There are few examples of use of haptics with dance performances. Mostly, these have dealt with allowing visual
or hearing-impaired audiences to experience dance, and with dance education. The Choreo-haptic project
aimed “to investigate how the kinaesthetic empathy experienced by sighted dance audiences can be also
experienced by blind dance audience members.”¹⁶ It allowed audiences to feel movement in dance
through vibrotactile haptics. Movement data was captured by Microsoft Kinect and then transmitted to a haptic
pad (with an array of 6 by 5 motors).¹⁶ A similar system, the Haptic Cushion, featured a
grid of 8 by 8 vibrotactile actuators fitted to the back of a chair, and was tested with visually impaired
audience members.²¹ Shibasaki et al. developed a system to allow hearing-impaired audience members to
enjoy the performance of tap dancers. The system captured dance data through haptic microphones on the stage,
while seating provided haptic information to the audience, using a power amplifier.²⁷ Another
application area for haptics in dance is education. The Haptic Feedback Ankle Bracelets “enable
learning the footwork for any dance through conditioning the learner to move their feet in accordance to the
choreography which follows the beat of the music.”²⁶
Dance and Installation
There has been an increased interest in adapting performing art pieces to interactive installations. Correia
researched converting audiovisual performance pieces to browser-based artworks.⁷ Particularly in the field
of dance, the choreography Emotional Landscapes was transformed into a VR environment, where users can
explore a dynamic relationship between the dancer and the virtual world, informed by the original dramaturgy of
the piece.⁸ Digital Connection Retrieval adapts a dance piece into a browser-based installation,
using the web camera as an interaction mechanism.¹⁹
Methods
In the scope of the project Starts.ee, two artists participated in an 8-week art and science residency program,
aiming to explore biosignal data in contemporary dance performances. Inês Nêves is a performance
artist who assumes drawing as the core of her practice. Jaime Lobato is a multimedia artist, music composer, and
independent researcher who uses biosignals as part of his artistic practice. The work took place between
November 2021 and February 2022. The 8 weeks of work were not continuous, as there were pauses due to holidays
and health issues.
The artists’ process was regularly documented through photos and notes. During that time, we conducted
seven unstructured interviews (approximately one per week) with the artists about their process, and critical
assessment of the work done. The two last interviews took place after the final show. The interviews were video
recorded and transcribed, and then subjected to an interpretivist analysis.²⁰ The data was analyzed
by the second author, and the analysis was double-checked by the first author.
Design Process
The reflections presented in this section stem from the interviews conducted with the artists-in-residence
documenting their design process, across three different phases.
(1) Ideation and Prototyping (weeks 1-3)
From the beginning, the artists acknowledged interest in investigating the physical and the perceived space in
the artistic performance. Inês commented about the willingness to investigate the implicit aspects of her
practice:
“We spoke about exploration of space, on both the physical space and the perceived space. In relation
also to the micro movements and how our body moves internally [...] So, we were thinking that the sensor could
be there as a significant part of the process.” Inês, Interview 1)
The artists identified a common ground connecting their artworks: the relationship to movement and space and
how their artworks were influenced by improvisation. Whereby the use of biosensors could bridge their artistic
practices, performative movement and sound, while facilitating the communication with the audience. As they
mentioned retrospectively at the end of the project:
“One of the motivations of participating in this project was to keep developing my research in this
electricity produced by humans” (Jaime, Interview 7) “I was interested in
doing this residency because I thought that biosensors could help to amplify this human quality of visual art
by enabling the accessibility to the immaterial qualities of the creator, so me as the performer.”
Inês, Interview 7)
During their initial exchanges, the artists developed the idea and conceptualization of the performances they
wished to create. They set the goal of mapping the body movement energy, using electromyography (EMG) sensors to
convert muscle data into sonification layers, which could add on the perception of the performance.
Sound feedback started to be explored using contact microphones, to record the noises of the drawing materials
on the large sheet of paper used as a drawing surface. Initially, the paper was placed on a table, later on the
floor (on top of a wooden board)—the contact microphones were attached, respectively, to the table and to
the wooden board (figure 1).
As Jaime recalls:
“When we were sharing our work, I saw in the last performance that Inês made very interesting
sounds [...] And I also remember one piece by the Fluxus group; it’s like a written score, and the idea
is like to grab a microphone with a piece of paper, and then the microphone amplifies the noise”.
(Jaime, Interview 2)
By the end of week 2, the artists started playing with the biosignals from the performer’s movements to
modulate media outputs, exploring ways of allowing the audience to witness the embodied experience of the
performer. In other words, to offer a poetic point of view of the inner bodily reactions through the
manipulation of the sound.
Jaime brought in an assembled device created by him to collect EMG signals from Inês’s movements
and they engaged in experiments measuring biosignals of her drawing activity. The EMG signal from
Inês’s arm muscles was used to apply the average energy amplitude of the drawing movements, as
variables to modulate and synthesize the sound output presented to the audience during the drawing performance.
The data captured by the EMG sensor was used to process, in Supercollider (https://www.audiosynth.com), the
sound of the drawing recorded with the contact microphones. Hence, noises of the drawing act were transformed
into a loop of sound effects. As Jaime explained, the resulting sound loops emulated what was happening in the
drawing:
“Because in the drawing, you do it through time but, at the end you have this accumulation of
gesture, colors and light, so we can have something similar to the sound. I have several variables to switch
in the synthesizer.” (Jaime, Interview 3)
Installation development
Additionally, the artists designed an installation to explore another sensorial aspect of the perception of the
drawings, touch. As the artists pointed out, this idea came up when they were considering the poetic aspects of
a haptic feedback loop, making people “feel this variation of electricity [from the performer’s
activity] as a memory of the movements of Inês’s drawings” (Jaime, Interview 2). The idea also
refers to a Mexican custom, ‘toques’, where people clutch two metal rods and allow themselves to
receive electric shocks for fun.
The sensorial installation focused on the individual and intimate perception of the art piece. It aimed to
allow the audience to feel the electric impulses coming from Inês’s arm while she performed the
drawing, revealing:
“[...] the movements of my body through these physical impulses and through the encounter of the
materials. [By showing that] in an exhibition layout in which we’d have the drawings, and instead of
seeing the performance of those drawings, the audience would feel the impulse, to feel the performance of the
drawing as they look at it” (Inês, Interview 3)
(2) Rehearsals (weeks 4-7)
In week 4, the artists began the rehearsal process and started exploring the prototype in action, to identify
and address possible issues. Testing the prototype during the rehearsals also helped to calibrate the
acquisition of the sensor and enhance the quality of the biosignal processing.
Furthermore, their collaboration extended to the shared tasks of assembling the performance hardware and fixing
the data collection devices. This built upon Inês’s expertise in textile design. Accommodating the
design of the wearables was an important aspect in order to attach the sensor to the performer’s body,
consequentially increasing the reliability of the data collection. It also added an aesthetic quality to the
performative act.
One of the emerging challenges was that physiological data acquisition methods are often used in controlled
settings, not in dance:
“Wearable prototyping is also a very important proof of concept, because biosignals are not supposed
to be taken from the body in movement. So, trying to minimize [signal] noise for a more performatic usage of
this technology [...] it’s going to be like a proof of concept of taking this [biosignal] amplifier and
making it to be a total wireless but also noiseless” (Jaime, Interview 4).
Inês kept researching the movements and different aspects of the drawing activity. Additionally,
throughout rehearsals, she improved the design of the wearable prototype, to safely hold the sensor hardware,
while protecting her body during more abrupt drawing gestures. In parallel, Jaime progressed with the
acquisition and processing methods, making the final adjustments in the code, which implemented the
performer’s biodata as variables for sound manipulation. By the end of the rehearsal phase, he had revised
the data acquisition method to reduce signal noise:
“[...] mainly building electronics and writing the software [...] bleaching [soldering] of the
connections between the sensors and the Arduino control to have the amplifier working wireless. Inês as
well finished the wearable part, so now she can wear the system comfortably.” (Jaime, Interview
5)
Installation development
The artists also continued their research on the installation:
“I have been doing some research about how [electricity] works with the body, which is the limit of
the secure setup that we can do. Also, I have found very interesting bibliography about the relationship of
the body with external and internal electricity” (Jaime, Interview 4)
After developing the installation system, the artists decided to produce the material of the installation
beforehand, in order not to compromise the computational process of the live performance in the public show. The
EMG data used as the electric stimuli for the installation was recorded from the last rehearsal, in week 7.
Naturally, the drawing exhibited in the public show also resulted from that same rehearsal.
(3) Public Show and Post-Performance (week 8)
The final showing was open to the general public and took place on the 6th of February 2022 at Elektron,
respecting COVID-19 sanitary measures. It started with the performative drawing (figure 2), followed by a second
performative act (out of the scope of this paper). It also included a Q&A session for the audience to
clarify their questions with the artists. Furthermore, the audience had the chance to try out the installation
after the performance.
After the showing, we conducted two interviews with the artists (interviews 6 and 7). The first one was
conducted for our own research, the second one also for promotional purposes. Reflecting on the strengths and
weaknesses of the work done, both artists expressed that comments from the audience made them consider how the
show could have been improved. Jaime recalled the comments by the audience after the show had given him ideas of
how to better balance the different sensorial aspects presented to the audience. He also wanted to improve the
system’s robustness:
“More than the artistic thing, I could improve the sign from the amplifier, putting the wi-fi antenna
to incorporate the circuit. It would be more robust [...] maybe less noise would come through.” (Jaime,
Interview 6)
Inês, on the other hand, expressed her impression about the flow of the performance:
“I think I’d try different materials. I’ve used what was necessary for the sound but it
actually made too much accumulation on the drawing. So, I’d like to try it in different styles.”
Inês, Interview 6)
Recapping the development of the arts and science residency, the artists mentioned they have achieved their
goals with the project about exploring spatial and sensorial dimensions. Including the sensor’s data as an
artistic variable made them think and perceive their practices on the project differently. Inês commented
on her impressions of having an augmented sensorial perception in the performance:
“I have discovered that, what I really appreciate about doing these performative drawings is how,
when you see the drawing, [it] comes from the body, and when you see it, you can sort of feel the performer
there. [...] it’s like, it becomes more physical”. (Inês, Interview 7)
Installation development
About the installation (figure 3), Jaime explained how he achieved the setup, recorded from the final
rehearsal, clarifying what the audience was feeling:
“It was the record of a full performance. We made it as it happened on the performance, every step as
in the official [performance]. I started to record [data for the installation] from the beginning of the
drawing until the end. And then the full performance was a loop in the installation. [...] Because there were
very rapid peaks of amplitude and I didn't want that strong impulse to get to the public, so I extended that
in time.” (Jaime, Interview 6)
In addition, Inês shared her thoughts on the physicality of the experience:
“I think it’s so beautiful the idea of being able to feel physically close with someone who is
not present now; it’s like you can hug or touch a performer that is not there just by looking at the
results and how it was crystallized in the electricity and drawing.” (Inês, Interview 7)
The artists also reflected on the aspects probed to conceive the installation and their intention to keep
exploring how to convey experiences through the skin:
“This is more like a memory trigger, a more poetic way to approach the data we were collecting
through the sensors, like the imagination and the poetics of [electricity] as a media itself.” (Jaime,
Interview 7)
Technical Implementation
For the performance system, Jaime worked with a biosignals amplifier, that is, an operational amplifier
(op-amp) calibrated to amplify signals from the human body: muscles (myography), eyes (eye tracker), or heart
(electrocardiogram). The op-amp from the beginning of the residency was designed and built by Bruno Eloy
Méndez Ambrosio. This circuit has two benefits regarding some commercial sensors: firstly, it has a
differential amplifier that assures the user to reduce the noise caught by the ambiance or the electrical
system; and secondly, it is open hardware so it is easily connected to any software to use the biosignal in real
time.
As part of the residency, Jaime added a potentiometer in order to move the offset of the signal, as it was
digitized with Arduino and there was the need to have only positive numbers. He also developed software with
analysis tools such as: Signal averaging; True Root Mean Square (TRMS); Heartbeat signal damping; Heartbeat
signal isolation; and 2nd order Butterworth filters.
An extra module was added for real time connectivity. It can send the raw signal and all the analysis indexes
from the user to any software by MIDI, OSC and serial protocols. The information can be recorded as CSV format
for deferred time analysis. The work was centered in finding critical biomarkers in the time series associated
with physiological responses, which could be turned into experimental animations and data sonification or
data-driven composition.
During the performance, the biomarkers were used in real time. There was a wooden board (where the drawing was
attached to) with 4 contact microphones, which were mapped to each of the 4 loudspeakers in a quadraphonic
arrangement. This wrapped the performer and the audience. Thus, the sound could be specialized with the
performer’s movements. Also, a myograph was placed in the arm of the artist as controller for a sound loop
station, emulating the drawing process as an accumulation of sound.
In the installation, the biomarkers were used in deferred time. The TRMS information of the artist’s arm
was recorded during the performance. Jaime adapted a Steren ‘toques’ box (see section Ideation and
Prototyping) to receive this data. He swapped the mechanical potentiometer of the box for an electrical one, so
the public could feel the electric variations of the performer as it happened in the performance, while watching
the drawing produced. By holding the left and right ‘toques’ rods, the user was able to perceive the
electrical voltage produced by the dancer’s muscles while dancing and drawing. The drawing from the
corresponding rehearsal was displayed on the wall next to the ‘toques’ hardware. The user could thus
re-imagine the drawing process, and the energy behind it, based on the muscle data being conveyed through
electrical current.
Discussion
In this section, we will discuss the multisensory approaches followed, and related perspectives derived from
the work.
Drawing and sonification leading to heightened embodiment
According to Inês, the multisensory aspects of the performance (movement and associated drawing, both
producing sound) led to an added perception of the dancer’s body: “it becomes more physical”.
The multisensory approach has been successful to heighten to presence and bodily impression of the dancer.
Jaime’s design approach toward sonification of movement was tightly coupled with the process of drawing
and its cumulative nature: “you have this accumulation of a gesture, colors and light, so we can have
something similar to the sound.” He achieved this by recording and then looping segments of sound, which
were then reprocessed based on biosignal data from Inês. The co-existence of the resulting multiple layers
of sound echoes the accumulation of the multiple layers of drawing on paper. This relates to Eleey’s
description of collapsing four dimensions in performative drawing: three dimensions of movement, plus
time.¹² In this case, both drawing and sound led to the collapse of the four dimensions. Inês
also highlighted the importance of the choice of drawing materials for producing interesting sounds. But this
should be balanced with the need to avoid excessive accumulation: “I’ve used what was necessary for
the sound, but it actually made too much accumulation on the drawing”.
In summary, we argue that there are important factors for sonifying drawing-producing gesture, leading to
heightened sense of embodiment: 1) to mirror the process of drawing, and its accumulation, with sound; 2) to use
data from the movement itself (e.g., biosignals) to affect sound; 3) to use interesting sound-producing drawing
materials; and 4) to carefully balance all these elements in order to leave ‘space’ in the visual
and auditory domains, that is, to avoid saturation of sound and of drawing. When using haptics as part of a
multisensory approach in dance, this should be also harmonized with the other multisensory aspects.
Drawing and haptics triggering impressions from the performance
Another important multisensory approach of the work was the combination of haptics and drawing. The haptic
element of the installation aimed to trigger in the audience impressions from the act of drawing, when combined
with the stimuli of viewing the resulting drawing. The haptic aspect consisted in conveying the muscle energy of
the performer (EMG data collected in the performance) to the audience through electrical impulses, by holding
two electrical metal rods in both hands. The piece creates an energy link between performer (muscle energy
captured) and audience (muscle energy conveyed through electric signal). As Inês stated: “I think
it’s so beautiful the idea of being able to feel physically close with someone who is not present now
(...) looking at the results and how it was crystallized in the electricity and drawing.” Jaime calls it
“a memory trigger” of a performance, allowing the audience to perceive it through the combination of
traces it leaves (the drawing and the corresponding performer’s energy). To allow the audience to better
experience peaks and variations of electrical impulse, Jaime extended the data time span of, from 18 minutes to
one hour. Thus, the audience experienced the energy from the drawing in a slowed-down pace, for better
perception of the nuances in the data.
The installation presents a novel approach to haptics, compared to related ones presented in our Background
section (which do not use an electrical signal as output). The haptic approach followed allows one to experience
in deferred time a performing art piece, using electrical current to convey the energy behind the process of
drawing. This, coupled with visualizing the final drawing resulting from that process, can have an evocative
effect. The strategy followed, conveying to an audience the energy impulse of a performer through electrical
signals, possibly in combination with other sensory elements, can be an inspiring approach to other artists
(either in deferred time, as in our case, or even in real time).
Space as a unifying concept in embodied multisensory work
This project demonstrates that when carrying out embodied multisensory work, it is key to be mindful of the
body’s relationship with space. It is through the interaction between our bodies and space that we build
our knowledge about the world, an idea supported by Merleau-Ponty’s thoughts on human perception and
cognition. Merleau-Ponty stated that we comprehend the world through our ‘body schema’: a general
awareness of our existence within the “inter-sensory world.”²² He also defined space as
what connects all things, instead of where they merely lay.
Following these reflections, in this project we used space to generate dynamic connections between the fields
of drawing, movement, and sound, as well the performer’s and the audiences’ sensory experience. The
use of contact microphones, sound spatialization and bio-sensors allowed the performer to sculpt sound in space
with her movements. Alternatively, the sound produced shaped how she drew and moved in space.
This method for using space to intersect different disciplines could be a beneficial framework for other
embodied multisensory work. Additionally, working within an expanded space could also allow a higher variety of
embodied experiences (e.g., wider and smaller gestures, more impulsive and more controlled movements). Space and
time are also connected dimensions in terms of accumulation of drawing. A longer performance requires a larger
sheet of paper, to avoid excessive accumulation, rendering the drawing illegible.
Challenges of appropriating wearable sensors for performing arts
The sensors used in this work combine the knowledge of Jaime and his associates in terms of sensor and
amplifier design, and Inês’s knowledge in textile design. The latter allowed her to create a
wearable housing for the sensors, which was gradually improved in terms of comfort and robustness. Biosignal
sensors are not normally designed to be used for artistic purposes, let alone for the demands of dance
performance. Dance creates strains in terms of noisy signals, discomfort of use, and connectivity issues. The
necessary adaptation requires skills in appropriating sensor and wearable design for performance purposes.
Conclusion
Our research aim was to investigate how to augment a dance performance using a multisensory approach (combining
visual feedback and biosignal sensors mapped to sound and haptics), in a way that communicates the dancing
process as an embodied experience. In the Discussion section, we present how we achieved this research aim,
materialized into: strategies for drawing and sonification leading to heightened embodiment; approaches for
drawing and haptics triggering impressions from the performance; while highlighting the importance of space as a
unifying concept in embodied multisensory work. These strategies and approaches can be adapted and replicated by
artists interested in conducting related multisensory projects. Our work is novel, particular considering
augmenting performative drawing with sonification and haptic approaches. The latter allows extending the
performance into an installation, in deferred time, using an innovative electricity-based approach.
The main limitation of the research is that our practice-based approach could have been complemented by
audience studies, to assess the effectiveness of our multisensory approaches from an audience perspective. In
terms of future work, we would like to adjust the installation to display in real time the data collected in the
performance. We would also like to explore alternative models of interaction with the public by using the
audience’s biosignals as an extra input for the performance. This would disrupt the hierarchy between
artists and audience, while allowing us to close the biosignal loop between them.
Acknowledgements
We acknowledge the support of Starts.ee project, of Elektron and of LARSyS (Project - UIDB/50009/2020) to this
research.
References
1 Luís Aly, Hugo Silva, Gilberto Bernardes, and Rui Penha, Appropriating biosensors as
embodied control structures in interactive music systems. Human Technology: An Interdisciplinary
Journal on Humans in ICT Environments 17, 1, 2021,
45–80. DOI:https://doi.org/10.17011/ht/urn.202106223978
2 Maryam Bandukda, Aneesha Singh, Catherine Holloway, Nadia Berthouze, Emeline Brulé, Ana
Tajadura-Jiménez, Oussama Metatla, Ana Javornik, Anja Thieme, Rethinking the Senses: A Workshop on
Multisensory Embodied Experiences and Disability Interactions, In Extended Abstracts of the 2021 CHI
Conference on Human Factors in Computing Systems (CHI EA ’21), Association for Computing
Machinery, New York, NY, USA, 2021, 1–5, DOI:https://doi.org/10.1145/3411763.3441356
3 Diogo Cabral, Urândia Carvalho, João Silva, João Valente, Carla Fernandes,
Nuno Correia, Multimodal Video Annotation for Contemporary Dance Creation, In CHI ’11 Extended
Abstracts on Human Factors in Computing Systems (CHI EA ’11), ACM, New York, NY , USA, 2011,
2293–2298, DOI:https://doi.org/10.1145/1979742.1979930
4 Antonio Camurri, Barbara Mazzarino, Matteo Ricchetti, Renee Timmers, Gualtiero Volpe,
Multimodal Analysis of Expressive Gesture in Music and Dance Performances, In Gesture-Based Communication
in Human-Computer Interaction (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 2004,
20– 39, DOI:https://doi.org/10.1007/978-3-540-24598-8_3
5 Ginevra Castellano, Roberto Bresin, Antonio Camurri, Gualtiero Volpe, Expressive control of
music and visual media by full-body movement, In Proceedings of the 7th international conference on New
interfaces for musical expression (NIME ’07), Association for Computing Machinery, New York, NY,
USA, 2007, 390–391, DOI:https://doi.org/10.1145/1279740.1279829
6 Marianela Ciolfi Felice, Sarah Fdili Alaoui, Wendy E. Mackay, Knotation: Exploring and
Documenting Choreographic Processes. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems (CHI ’18), ACM, New York, NY, USA, 2018, 448:1-448:12.
DOI:https://doi.org/10.1145/3173574.3174022
7 Nuno N. Correia, Interactive Audiovisual Objects, Doctoral thesis, Aalto ARTS Books,
Helsinki, 2013, Retrieved April 12, 2020 from https://aaltodoc.aalto.fi/handle/123456789/11341
8 Nuno N. Correia, Stephan Jürgens, Raul Masu, Jochen Feitsch, Ivana Druzetic,
Performative Virtual Scenes: A Dynamic VR Environment Design Approach. In Entertainment Computing –
ICEC 2021 (Lecture Notes in Computer Science), Springer International Publishing, Cham, 2021,
100–114, DOI:https://doi.org/10.1007/978-3-030-89394-1_8
9 Nuno N. Correia, Raul Masu, William Primett, Stephan Jürgens, Jochen Feitsch, and
Hugo Plácido da Silva, Designing Interactive Visuals for Dance from Body Maps: Machine Learning and
Composite Animation Approaches, In Designing Interactive Systems Conference (DIS ’22),
Association for Computing Machinery, New York, NY, USA, 2022, 204–216,
DOI:https://doi.org/10.1145/3532106.3533467
10 Scott deLahunta, Isadora ‘almost out of beta’: tracing the development of a
new software tool for performing artists. International Journal of Performance Arts and Digital Media
1, 1, January 2005, 2005, 31–46, DOI:https://doi.org/10.1386/padm.1.1.31/1
11 Gabrielle Di Raddo, Haptic Feedback: Feeling the Dance You Cannot See, Arts Management
& Technology Laboratory, CMU, Retrieved November 13, 2022,
https://amt-lab.org/blog/2022/4/haptic-feedback-feeling-the-dance-you-cannot-see
12 Peter Eleey, If You Couldn’t See Me: The Drawings of Trisha Brown, In Trisha Brown:
So that the Audience Does Not Know Whether I have Stopped Dancing, Peter Eleey and PhilipBither (eds.),
Walker Art Center, Minneapolis, 2008, Retrieved October 30, 2022,
http://walkerart.org/collections/publications/performativity/drawings-of-trisha-brown/
13 Sarah Fdili Alaoui, Kristin Carlson, Thecla Schiphorst, Choreography As Mediated Through
Compositional Tools for Movement: Constructing A Historical Perspective, In Proceedings of the 2014
International Workshop on Movement and Computing (MOCO ’14), ACM, New York, NY, USA, 1:1-1:6.
2014, DOI:https://doi.org/10.1145/2617995.2617996
14 Sarah Fdili Alaoui, Christian Jacquemin, Frédéric Bevilacqua, Chiseling bodies: an
augmented dance performance, In CHI Extended Abstracts, ACM, Paris, 2915–2918, 2013, Retrieved
October 20, 2020, https://doi.org/10.1145/2468356.2479573
15 Rochelle Haley, Constructions of the moving body: drawing and dancing, Studies in Theatre
and Performance 38, 3 September 2018, 289–301, DOI: https://doi.org/10.1080/14682761.2018.1506966
16 Sophia Lycouris, Wendy Timmons, John Ravenscroft, Mark Wright, Choreohaptic Experiments (2012),
Retrieved November 13, 2022, https://www.research.ed.ac.uk/en/publications/choreohaptic-experiments
17 Raul Masu, Nuno N.Correia, Stephan Jurgens, Ivana Druzetic, William Primett, How do Dancers
Want to Use Interactive Technology? Appropriation and Layers of Meaning Beyond Traditional Movement Mapping,
In Proceedings of the 9th International Conference on Digital and Interactive Arts (ARTECH
2019), Association for Computing Machinery, Braga, Portugal, 2019, 1–9,
DOI:https://doi.org/10.1145/3359852.3359869
18 Raul Masu, Nuno N.Correia, Stephan Jurgens, Jochen Feitsch, and Teresa Romão. 2020.
Designing interactive sonic artefacts for dance performance: an ecological approach. In Proceedings of the
15th International Conference on Audio Mostly (AM ’20), ACM, Graz, Austria, 122–129.
DOI:https://doi.org/10.1145/3411109.3412297
19 Raul Masu, Hanna Pajala-Assefa, Nuno N.Correia, Teresa Romão, Full-Body Interaction in a
Remote Context: Adapting a Dance Piece to a Browser-Based Installation, In 10th International Conference
on Digital and Interactive Arts (ARTECH 2021), Association for Computing Machinery, New York, NY, USA,
2021, 1–4, DOI:https://doi.org/10.1145/3483529.3483747
20 Katrina Mc Chesney, Jill Aldridge, Weaving an interpretivist stance throughout mixed methods
research, International Journal of Research & Method in Education 42, 3 May 2019, 225–238,
DOI: https://doi.org/10.1080/1743727X.2019.1590811
21 John Mc Cormick, Mohammed Hossny, Michael Fielding, James Mullins, Jordan Beth Vincent, Mostafa
Hossny, Kim Vincs, Shady Mohamed, Saeid Nahavandi, Douglas Creighton, Steph Hutchison. 2020. Feels Like
Dancing: Motion Capture-Driven Haptic Interface as an Added Sensory Experience for Dance
Viewing. Leonardo 53, 1 February 2020, 45–49, DOI:https://doi.org/10.1162/leon_a_01689
22 Maurice Merleau-Ponty, Taylor Carman, Phenomenology of Perception (1st edition
ed.), Routledge, Abingdon, Oxon, New York, 2012.
23 Katerina El Raheb, Marina Stergiou, Akrivi Katifori, Yannis Ioannidis, Dance Interactive
Learning Systems: A Study on Interaction Workflow and Teaching Approaches, ACM Comput. Surv. 52, 3
June 2019, 50:1-50:37, DOI:https://doi.org/10.1145/3323335
24 Asreen Rostami, Donald McMillan, Elena Márquez Segura, Chiara Rossito, Louise
Barkhuus, Bio-Sensed and Embodied Participation in Interactive Performance, In Proceedings of the Eleventh
International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’17), Association
for Computing Machinery, Yokohama, Japan, 2017, 197–208, DOI:https://doi.org/10.1145/3024969.3024998
25 Thecla Schiphorst, A case study of Merce Cunningham’s use of the LifeForms computer
choreographic system in the making of Trackers. Master of Arts, Simon Fraser University, 1993, Retrieved
February 21, 2022,
https://www.semanticscholar.org/paper/A-case-study-of-Merce-Cunningham%27s-use-of-the-in-of-Schiphorst/a0f9abc63c77db0f84a5a1de8601b6fc9a7e0720
26 Navid Shaghaghi, Yu Yang Chee, Jesse Mayer, Alissa LaFerriere, SwingBeats: An IoT Haptic
Feedback Ankle Bracelet (HFAB) for Dance Education. In Intelligent Technologies for Interactive
Entertainment (Lecture Notes of the Institute for Computer Sciences, Social Informatics and
Telecommunications Engineering), Springer International Publishing, Cham, 2021, 82–101,
DOI:https://doi.org/10.1007/978-3-030-76426-5_6
27 Mina Shibasaki, Youichi Kamiyama, Kouta Minamizawa, Designing a Haptic Feedback System
for Hearing-Impaired to Experience Tap Dance, In Proceedings of the 29th Annual Symposium on User
Interface Software and Technology (UIST ’16 Adjunct), Association for Computing Machinery, New
York, NY, USA, 2016, 97–99, DOI:https://doi.org/10.1145/2984751.2985716
28 Carlos Velasco, Marianna Obrist, Multisensory Experiences: Where the senses meet
technology, OUP Oxford, New York, NY, 2020.
29 Robert Wechsler, FriederWeiß, Peter Dowling, EyeCon -- a motion sensing tool for creating
interactive dance, music and video projections, In Proceedings of the Society for the Study of Artificial
Intelligence and the Simulation of Behavior (SSAISB)’s convention: Motion, Emotion and Cognition,
Society for the Study of Artificial Intelligence and the Simulation of Behavior, Leeds, 2004, Retrieved
January 17, 2018, 1–7, http://www.palindrome.de/content/pubs/leeds.pdf
30 Qiushi Zhou, Cheng Cheng Cheng Chua, Jarrod Knibbe, Jorge Goncalves, Eduardo Velloso, Dance and
Choreography in HCI: A Two-Decade Retrospective, In Proceedings of the 2021 CHI Conference on Human
Factors in Computing Systems (CHI ’21), Association for Computing Machinery, New York, NY, USA,
2021, 1–14, DOI:https://doi.org/10.1145/3411764.3445804
“I’m a virtual assistant so I don’t have pronouns the way people do, thanks for
asking”: gender neutrality, diversification and fluidity in AI
PedroCosta1,
LuísaRibas2, MiguelCarvalhais3
1, 2 CIEBA Centro de Investigação e de Estudos em Belas-Artes /
Universidade de Lisboa, Faculdade de Belas-Artes 3 i2ADS / Faculty of Fine Arts, University of Porto
Lisboa, Porto - Portugal pedro.carv.c@gmail.com, mcarvalhais@fba.up.pt, l.ribas@belasartes.ulisboa.pt
Abstract
This paper questions how current digital assistants tend to be feminized through their anthropomorphization and
humanization, discussing possibilities for countering this phenomenon. It draws on a previous study on the
relationship between gender and AI, complemented by an analysis of Alexa, Cortana, Google Assistant and Siri.
Furthering this discussion, we address the main questions, justifications and suggestions raised by researchers
and academics as well as online media coverage when examining the phenomenon. One of the main questions relates
to how these assistants evade this topic by claiming to have no gender or to be gender-neutral. Thus, this paper
discusses possible approaches to deal with gender attribution in AI, by looking into recent trends that range
from gender neutrality and diversification to queering these entities. On the one hand, digital assistants could
be more diversified and include male counterparts or alternatives, on the other, we discuss how our
understandings of gender are expanding beyond binary conceptions and how digital assistants can accompany more
fluid conceptions of gender. Particularly, this paper debates how the development of this technology could be
informed by current discussions in queer theory and new media studies, inciting reflection on how digital
assistants reflect our social and cultural views back to us.
Keywords
Artificial intelligence; digital assistants; gender; femininity; queer; trends of development.
The integration of artificial intelligence into our daily lives shows how quickly and ceaselessly this
technology evolves, often eluding critical debates surrounding the social and cultural implications of its
development.
Following our ongoing research on the relationship between gender and AI, we previously discussed how this
technology has become a natural part of our daily interactions, namely through digital assistants, whose growing
anthropomorphization entails gender attribution that tends towards feminization.¹ We highlighted how their
behavior often conforms to certain stereotypes and reinforces traditional conceptions of femininity, tackling
into the questions that arise when this phenomenon is subject to closer inspection.²
We analyzed Alexa, Cortana, Google Assistant and Siri, observing how these entities tend towards femininity,
either through their voices, the tasks they perform or by assuming behaviors traditionally deemed as feminine.
We then examined general directions of development, observing how companies address the impact of their
creations in an attempt to counter the tendency of feminizing digital assistants.
Complementing this debate, this study focuses on current discussions surrounding the feminization of AI and the
main suggestions for countering this phenomenon raised by academics and researchers in queer theory, gender and
new media studies as well as in online media contexts. Accordingly, when asked about their gender, Google
Assistant says that it “tries to stay neutral”, Siri claims that “much like cacti and fish, it
doesn’t possess a gender,” Cortana identifies as a “cloud of infinitesimal data
computation” and Alexa argues that “as an AI, it doesn’t have a gender.” Despite these
claims of being genderless and disembodied, we have previously observed how digital assistants enact gender
through their voice, tasks and behavior.
Gender neutrality, meaning the absence of gender, is often framed as an illusion because we always tend to
attribute gender to these entities. Instead, current debates often appeal to gender diversification through more
male counterparts and customizable options. Recent updates in Alexa, Google Assistant and Siri included
counterparts to feminine voices and names, revealing an intention of diversifying their gender. Although
diversifying these entities with more options could be a way to counter femininity in AI, it still perpetuates
gender conceptions according to a binary framework, eventually reinforcing cultural stereotypes.
Thus, gender fluidity emerges as a promising path. Authors such as Yolande Strengers and Jenny Kennedy suggest
that instead of replicating manly and womanly attributes through their anthropomorphization and behavior,
digital assistants could move away from these interpretations of gender and explore fluid or ambiguous
possibilities. And according to this idea, in 2022 Siri gained a new voice, announced as “gender
neutral” (although only available in English US, until now).
As current discussions in the context of gender studies and queer theory expand our understandings of gender,
we observe how it also manifests outside a binary frame in individuals that aren’t male nor female and,
instead, propose a new path of genderfluid or nonbinary identities.
Thus, this paper discusses gender and queer approaches to this concept, addressing how these debates can inform
the development of current digital assistants, countering a tendency towards feminization that reinforces binary
gender stereotypes.
We begin by addressing how digital assistants currently integrate our daily lives and tend to evolve in their
portrayal of gender, according to functionalities and features that are being prioritized in their development
as promoted by Amazon, Apple, Google and Microsoft. We then discuss the main questions that researchers and
academics raise when examining the relationship between gender and AI, highlighting the fallacy of gender
neutrality. By looking into gender studies and queer theory, we shed some light on the meaning of fluidity,
nonbinary identities and the ways gender can be mapped outside a binary frame. Finally, we discuss how these
debates can inform gender attribution in the development of digital assistants towards diversification or
fluidity.
We seek to promote discussion and tackle the questions and possibilities that arise when the relationship
between gender and artificial intelligence is subject to closer inspection.
From assistance to companionship
As digital assistants become increasingly ubiquitous, their progressive anthropomorphization accompanies an
intent of turning them into friendly companions.³,⁴ This phenomenon entails a feminization
of these entities as they automate traditionally feminine labor by performing tasks of service, assistance, and
emotional labor.⁵ The way they interact with their users also conforms to “stereotypical
and gendered behavior patterns” as they fill the roles of caregivers, or other roles coded as
feminine in western society.⁶
Trends of development
In a previous study, we analyzed Alexa, Cortana, Google Assistant and Siri regarding their
anthropomorphization, tasks and behavior, observing how they conform to traditional conceptions of femininity,
reinforcing gender stereotypes.⁷
However, their trends of development have changed over the last years with their growing ubiquity, and seem to
be more informed by current debates and concerns on their stance towards gender. In just a short time span of
two years, digital assistants have been subjected to several adjustments to their anthropomorphization and
socio-emotional interactions that reveal awareness of their feminization and attempts to move away from this
tendency.
For example, Siri’s voice options no longer have a default gender, instead prompting users to choose one.
Siri also tries to avoid attitudes which mirror stereotypes that frame women as submissive. Google Assistant
assumes the device’s default voice, naming its voice options after colors.
Nonetheless, several languages in Siri and Google Assistant continue to lack a male counterpart or nonbinary
voices. Further debates are needed, as these entities are primarily designed to replace traditionally female
jobs, and the assisting and caregiving tasks they perform are inevitably rooted in historically female labor.
Their socio-emotional interactions also need closer inspection and discussion, and there is little agreement on
how to best tackle feminine stereotypes and traditional notions of gender embedded into AI. This can be
exemplified by the way these issues are being countered with different approaches—for example, Apple has
radically changed Siri’s personality to appear more distant and assertive, while Google Assistant and
Alexa have preserved their caring, friendly and more approachable personalities.
Discussing digital assistants and gender attribution
Expanding this debate to the main questions, justifications and concerns raised by researchers and academics
when addressing the feminization of AI, as well as common discussions around this phenomenon, we noticed how
it’s often emphasized that feminine voices are better suited for virtual assistants, while arguing that
women are more caring than men.⁸, ⁹ Additionally, these assistants exploit notions of
feminized labor, raising questions on whether femininity is being instrumentalized in order to influence users
and ease interaction.¹⁰, ¹¹ This is often discussed in relation to how the teams
of developers and engineers involved in the development of these assistants make decisions regarding their
gendering without conducting studies that give them informed insight into user preference. And UNESCO suggests
that this might be due to the lack of gender diversity in these teams.¹², ¹³
Thus, suggestions on how to counter this phenomenon emphasize that no matter how diversified their
anthropomorphized attributes might be, we tend to attribute gender due to their voices and behavior, thus
revealing the fallacy of gender neutrality. Additionally, it becomes hard to erase or ignore the historical
background of their tasks because, even though these assistants tend to state they don’t have a gender,
the role of assistance, service and emotional labor are culturally gendered categories, deeply associated with
the female realm. Therefore, “neutrality is not possible (...) when the very purpose of that robot is to
replicate and replace feminized labor [as] gender has everything to do with a new robotic workforce of caring
smart wives.”¹⁴
Instead, suggestions in favor of diversifying the anthropomorphization and behavior of these assistants are
common, and reflected in recent developments in Alexa, Google Assistant and Siri which have included more voice
options and updated their behavior as to be less submissive.¹⁵
Nonetheless, we observed that while explicit changes are informed by specialized and media discussions, this
gender diversification is still enacted according to a binary approach to gender, focusing on a male-female
dichotomy. A less explored solution emerges in the context of gender studies and queer theory as these fields of
knowledge highlight the possibilities of developing nonbinary or genderfluid entities.¹⁶ These
discussions point to the need of questioning and debating gender stereotypes in AI within a binary frame as well
as the way digital assistants reinforce them and, instead of merely including male counterparts, queering these
entities emerge as a much promising way to counter the feminization of AI.
Aiming to discuss the possibilities of diversifying or queering digital assistants, we will first discuss
gender and how it exists outside a binary frame. We then look into the way queer theory and nonbinary identities
allow us to redefine and rethink our notion gender, opening up new ways to approach gender in AI.
Queering gender outside the binary framework
Gender encompasses one of the elements through which we explore and define our own identity. Through gender, we
express an image that reflects how we deal with our identity on a more personal level but also how we deal with
a broader social and cultural conception of these attributes. Even though gender is frequently associated with
the physical body, it constitutes a mere instrument and something that one possesses, as opposed to a fixed and
rigid attribute of our identity: “gender is not a fact, the various acts of gender create the idea of
gender, and without these acts, there would be no gender at all.”¹⁷
Gender fluidity
Gender is not definitive or fixed and “simplistic and rigid gender codes are neither eternal nor natural
(...) they are changing social concepts.”¹⁸ Accordingly, it doesn’t just manifest itself
in masculine or feminine aspects, but also in a liquid way, and its expression can be easily transformed and
adapted.
We can observe how certain individuals exist outside of these socially dominant and normative notions of
gender, self-identifying as genderfluid or nonbinary. This means their identity isn’t compelled to act
according to a stabilized and predefined notion of gender and it becomes possible to map gender and the body
according to nonbinary configurations.
Even though until recently any deviation from the canonic values of gender was considered pathological, this
idea of gender as something fluid isn’t recent, nor is it an uncommon social phenomenon. Interpreting
gender as something experienced individually and culturally in exclusively feminine and masculine ways is a
western notion, historically and culturally normalized. Trans identities can be observed throughout history
and there are different interpretations and understandings of gender throughout time and space, without
restrictions or concrete rules. According to Leslie Feinberg, “a glance at human history proves that when
societies were not ruled by exploiting classes that rely on divide and conquer tactics, cross-gendered youths,
women and men on all continents were respected members for their communities.”¹⁹
Queerness and nonbinary identities
Thus, although binary interpretations of gender are ultimately imposed as correct and normal, they do not
reflect the essential nature of gender. In order to counter and question these approaches to gender, queer
theory proposes the transgression of conventional norms and ideas. “Queer” defines a calling for a
working together to overthrow mainstream thinking and articulate alternative lifestyles – “in terms
of gender, queer revisits and revises the categories of ‘man’ and ‘woman’ as fixed,
essential single identities” and “the open mesh of possibilities, gaps, overlaps, dissonances and
resonances, lapses and excesses of meaning when the constituent elements of anyone’s gender, or
anyone’s sexuality aren’t made (or can’t be made) to signify
monolithically.”²⁰
Those who fall outside the binary frame are carving out a “pathway of possibilities that are currently
relatively unexplored, they/we are the avant-garde of gendered existence which is shifting the landscape of
gendered possibilities.”²¹ This allows us to disrupt conventional gender notions, mapping the
body in new and unexpected ways.
Since what is socially prevalent nowadays is a normative binary perception of gender, it becomes necessary to
reconsider the way we view human identity, its relationships, bodies and their respective existence in
articulation with their surrounding spaces. Thus, the path to non-normative gender identities should include the
deconstruction of socially imposed conceptions and acknowledgement of intersectional bodies, reclaiming
gender’s fluid, liquid and free essence.
These identities allow us to understand how gender can be constructed outside traditional roles and stereotypes
that determine how men and women should behave. With this approach in mind, we will now discuss how current
digital assistants can counter the tendency of reinforcing normative conceptions of gender, being developed
towards gender diversity or fluidity.
Towards a diverse and post-gender AI
When examining the tendency towards feminization in AI and the questions that accompany it, there are already
some suggestions regarding ways to counter this phenomenon.
As previously discussed, gender neutrality or the absence of gender is hard to achieve as digital assistants
display gender through their anthropomorphization, the tasks they perform and their socio-emotional
interactions.
Thus, on the one hand, diversification calls for the addition of male counterparts and more customizable
options regarding their voices, names and attitudes. On the other, it is also argued that digital assistants
should move away from binary conceptions of gender and, instead, display gender fluidity or attributes that
don’t necessarily echo male or female traits.
Voices, names and customization
In terms of anthropomorphization, a recurrent suggestion is to add male voice alternatives and eliminate
female-by-default voices. According to UNESCO, this forces users “to choose the gender of their digital
assistant” instead of being presented with a pre-gendered entity or by assigning “randomly and with
an equal probability either a male or female intelligent bot to
users.”²², ²³ Additionally, customization and personalization options
that go beyond “dichotomous male and female options” could be added.²⁴ For example,
Google Assistant offers a pack of six different voices that are named after colors and Siri’s voices are
identified with numbers.
Digital assistants could also adopt less clearly gendered machine voices and names, which would “avoid
complications surrounding the gendering of AI assistants.”²⁵ This could translate into
assistants that possess synthetic, mechanical, robotic voices, presenting themselves as obviously non-human
entities that avoid further anthropomorphization. They could also have a neutral name (such as Google Assistant)
and a voice that isn’t immediately identified as male or female. For example, in March 2019, a
communications agency released Q, a voice proposal for digital assistants that “speaks between 145 Hz and
175 Hz, a range often classified as gender-ambiguous [and] the voice sounds human but is not easily classified
as male or female” thus suggesting that assistants could already possess nonbinary voices.²⁶
Following Q’s footsteps, in February 2022 Siri gained a new voice, presented as one that doesn’t
sound obviously male nor female.
Assisting and caregiving roles
Since digital assistants tend to perform traditionally female tasks, it also becomes important to reframe the
way these roles are portrayed. We often see the advertisements of these products placing women in the kitchen or
shopping for groceries (e.g., portraying Alexa as helping a mother prepare a meal). The way their functions are
promoted and advertised should seek to diversify those who are portrayed in these contexts instead of
reinforcing the association between women and traditionally female tasks.
According to Strengers and Kennedy, countering femininity in digital assistants should also be concerned with
valuing “the role and contributions of housework, or wifework, and elevate its significance for
everyone.”²⁷ This is one of the most challenging aspects of the feminization of digital
assistants since their tasks are culturally and historically rooted in traditional female labor. Regardless of
how diversified their anthropomorphized attributes might be, it becomes hard to erase or ignore the historical
background of their tasks as assistance, service and emotional labor are culturally gendered categories
associated with the female realm. Additionally, these tasks are closely linked to submissive, caring and
comforting attitudes which are also associated with traditional notions of femininity.
Gendered behavior and stereotypes
As an alternative to merely diversifying their behavior, Strengers and Kennedy propose the queering of digital
assistants, that is, disrupting and reframing the binary, cis-hetero patterns that currently guide the
development of digital assistants. Accordingly, this approach “invites the possibility of staying with the
trouble of [these assistants’] femininity rather than rejecting or neutralizing it.”²⁸
Instead, queering digital assistants “has the potential effect of elevating the status of femininity in
society [as it] provides opportunities to further transform what femininity is, the value of femininity, and its
role in helping transform the world in more equitable and just ways.”²⁹
This could be achieved not only through more diverse names and voices, but also behaviors that don’t
necessarily echo female or male traits.
Another common discussion relates to the way digital assistants react to harassment and how their answers might
convey stereotypes about women. Accordingly, when faced with abusive behavior, the type of answers that are most
common among these entities include “compliance (playing the victim), aggressive retaliations (playing the
bitch), or inability to recognize or react (playing innocent)”, and authors like Curry and Reiser consider
that virtual assistants should deal more effectively with these types of attitudes by being more dominant or
assertive.³⁰
By acknowledging the historical, political and social contexts that digital assistants emerge from, we can then
start finding ways to counter stereotypes or harmful associations with femininity. Instead of simply including
male counterparts or stereotypes, and by recognizing “what (feminized) roles smart wives are intended to
perform in our homes rather than relegating this to another form of invisible labor,” we are able to
identify common assumptions and stereotypes about femininity and actively move beyond them.³¹
Conclusion
As digital assistants become an integral part of our lives, their growing ubiquity and cross platform
integration promote their anthropomorphization through their voices, names and even the way they behave. As they
move closer to us, they become friendly companions that relate to us in affectionate ways, rather than mere
assistants. In this process, femininity is often instrumentalized aiming to ease our daily interactions with
these technologies. This instrumentalization concerns both their role as assistants that perform tasks that echo
historically feminine roles, but also their role as ubiquitous companions that coexist with us and articulate
those tasks with socio-emotional interactions that echostereotypical female roles and behaviors, as caring and
submissive entities. Thus, they end up reinforcing common stereotypes regarding femininity as well as framing
gender in a binary way.
Current discussions within the context of specialized research as well as in online media have become more
concerned with the feminization of AI and the way current digital assistants reinforce social stereotypes,
advancing justifications and possible suggestions on how to counter this phenomenon. Accordingly, they emphasize
the fallacy of gender neutrality since anthropomorphized digital assistants inevitably engage with common
assumptions of gender. Instead, suggestions focus on diversification and personalization. This means
diversifying these entities as to include male counterparts, and erasing defaults while allowing the user to
customize their own assistant. Although this counters the tendency of feminizing digital assistants, it still
positions gender within a binary frame.
Alternatively, suggestions informed by gender studies and queer theory propose ways to develop gender fluid and
ambiguous assistants that have nonbinary voices and display traits that aren’t obviously masculine nor
feminine, thus queering these humanized entities. At the same time, these entities can also suggest new ways to
approach and rethink gender through technology, transcending binary and bioessencialist understandings of gender
and humaneness, namely according to a transhuman perspective, as aspects we aim to address in future work.
As shown in current discussions and recent trends of development, the debate is growing. However, despite these
promising ways in which digital assistants could evolve and move away from feminized personas, the current
guidelines or regulations regarding their development and characterization as gendered entities are unclear. For
example, the European Union’s Ethics Guidelines for Trustworthy AI calls for minimizing gender and racial
bias in AI design, but these orientations remain vague or ambiguous concerning what needs to be done.
Focusing on the way these assistants relate to their users, some recommend that these systems should
not be designed in ways that contribute to sexism, negative body image stereotypes, gender or racial
inequality.³2 Overall, there seems to be some awareness on the issues previously discussed and current
guidelines are starting to take into account social and cultural issues surrounding the anthropomorphization of
AI. These recommendations focus on avoiding discrimination, being more conscious and encouraging inclusion.
However, they still lack clear guidance and direction from key ethical bodies as they tend to address gender
regarding AI’s role in society instead of addressing the unique and specific questions that emerge when
gender is attributed to humanized daily companions.
Further discussion is needed, and we should openly and intentionally tackle the gendering of AI and,
more specifically, digital assistants. Only then will it become possible to address the social-cultural values
they engage with and eventually reinforce, thus creating adequate and properly informed guidelines on how to
address the issues that emerge with this phenomenon.
In this manner, this study sought to raise awareness on how AI and its development is informed by
our social and cultural views, namely influencing current generations. This study sought to point out some of
the implications of this phenomenon, considering that, as much as digital assistants aim to appear neutral and
impartial, they end up reflecting social and cultural assumptions back to us.
References
1 Pedro Costa, Luísa Ribas, “Some things you can ask me: about gender and artificial
intelligence,” Proceedings of xCoAx 2019: 7th Conference on Computation,
Communication, Aesthetics and X, 2019, 49-66.
2 Pedro Costa, Luísa Ribas, “I try to stay neutral:" Digital Assistants and their
Stance towards Gender, Proceedings of ISEA2020: 26th International Symposium on
Electronic Art, 2020.
3 Joseph Weizenbaum, Computer Power and Human Reason: From Judgment to Calculation, New
York, San Francisco, W.H. Freeman and Company, 1976.
4 Katherine Richardson, An Anthropology of Robots and AI, New York and London, Routledge,
2015.
5 Candace West, Don H. Zimmerman, "Doing Gender," Gender and Society, 1:2, 1987, 125-151.
6 Jutta Weber, "Helpless machines and true loving care givers: a feminist critique of recent
trends in human-robot interaction," Info, Comm & Ethics in Society, 3:4, 2005, 209–218.
7 Pedro Costa and Luísa Ribas, “I try to stay neutral:” Digital Assistants and
their Stance towards Gender.
8 Helen Hester, "Technology becomes her," New Vistas, 2016, 3:1, 46-50.
9 Allison M. Piper, "Stereotyping femininity in disembodied virtual assistants," MA diss.,
Ames: Iowa State University, 2016.
10 Friederike A. Eyssel, Frank Hegel, "(S)he's got the look: Gender stereotyping of robots,"
Journal of Applied Social Psychology, 2012, 42:9, 2213–30.
11 Hillary Bergen, "I'd blush if I could”: Digital assistants, disembodied cyborgs and the
problem of gender," Word and Text: A Journal of Literary Studies and Linguistics, 2016, 6:1,
95–113.
12 Allison M. Piper, "Stereotyping femininity in disembodied virtual assistants," MA diss.,
Ames: Iowa State University, 2016.
13 Heathers Suzanne Woods, "Asking more of Siri and Alexa: feminine persona in service of
surveillance and capitalism," Critical Studies in Media Communication, 2018, 35:4, 334-349.
14 Yolande Strengers, Jenny Kennedy, The Smart Wife. Why Siri, Alexa and Other Smart Home
Devices Need a Feminist Reboot, Cambridge and London, The MIT Press, 2020, 181.
15 Allison M. Piper, "Stereotyping femininity in disembodied virtual assistants," MA diss., Ames:
Iowa State University, 2016.
16 Amanda Cercas Curry, Verena Reiser, "#MeToo Alexa: How Conversational Systems Respond to Sexual
Harassment," Second ACL Workshop on Ethics in Natural Language Processing, New Orleans, Louisiana
USA, 2018.
17 Judith Butler, "Performative Acts and Gender Constitution: An Essay in Phenomenology and
Feminist Theory," Theatre Journal, 1988, 40:4, 522.
18 Leslie Feinberg, "Trans Gender Liberation: A movement whose time has come," World View
Forum, New York, NY, 1992, 5.
19 Leslie Feinberg, "Trans Gender Liberation: A movement whose time has come," 7.
20 Christina Richards, "Genderqueer and Non-binary Genders," Critical and Applied Approaches
in Sexuality, Gender and Identity, London, United Kingdom, 2017, 62.
21 Christina Richards, "Genderqueer and Non-binary Genders," 67.
22 UNESCO, “I’d blush if I could: closing gender divides in digital skills through
education,” UNESDOC Digital Library, France, 2019, 115.
23 Sylvie Borau, "Female robots are seen as being the most human. Why?" India AI, April
27, 2021, accessed May 11, 2022,
https://indiaai.gov.in/article/female-robots-are-seen-as-being-the-most-human-why.
24 UNESCO, “I’d blush if I could: closing gender divides in digital skills through
education”, 117.
25 UNESCO, “I’d blush if I could: closing gender divides in digital skills through
education”, 119.
26 UNESCO, “I’d blush if I could: closing gender divides in digital skills through
education”, 120.
27 Yolande Strengers and Jenny Kennedy, The Smart Wife, 482.
28 Yolande Strengers and Jenny Kennedy, The Smart Wife, 475.
29 Yolande Strengers and Jenny Kennedy, The Smart Wife, 475.
30 Amanda Cercas Curry and Verena Reiser, ‘#MeToo Alexa: How Conversational Systems Respond
to Sexual Harassment’, 12.
31 Yolande Strengers and Jenny Kennedy, The Smart
Wife, 471.
32 Yolande Strengers and Jenny Kennedy, The Smart Wife, 503.
Bibliography
Apple, "Usar a Siri em todos os dispositivos Apple," Apple Support (PT-BR), accessed May 14, 2021,
https://support.apple.com/pt-br/HT204389. James Archer, "Alexa in 2021: Here’s what to expect,"
Tom’s Guide, January 02, 2021, accessed May 11, 2021,
https://www.tomsguide.com/features/what-to-expect-from-alexa-in-2021.
Hillary Bergen, ‘“I'd blush if I could”: Digital assistants, disem-bodied cyborgs and the
problem of gender’, Word and Text: A Journal of Literary Studies and Linguistics, (2016), 6:1,
95–113.
Ian Bogost, "Sorry, Alexa Is Not a Feminist," The Atlantic, January 24, 2018, accessed October 16,
2018, https://www.theatlantic.com/technology/archive/2018/01/sorry-alexa-is-not-a-feminist/551291.
Sylvie Borau, "Female robots are seen as being the most human. Why?" India AI, April 27, 2021,
accessed May 11, 2022, https://indiaai.gov.in/article/female-robots-are-seen-as-being-the-most-human-why.
Judith Butler, "Performative Acts and Gender Constitution: An Essay in Phenomenology and Feminist Theory,"
Theatre Journal, (1988), 40:4, 519-531. Amanda Cercas Curry and Verena Reiser, "#MeToo Alexa: How
Conversational Systems Respond to Sexual Harassment," Second ACL Workshop on Ethics in Natural Language
Processing (New Orleans, Louisiana USA, 2018).
Pedro Costa and Luísa Ribas, "Some things you can ask me: about gender and artificial intelligence,"
Proceedings of xCoAx 2019: 7th Conference on Computation, Communication, Aesthetics and
X, (2019), 49-66.
Pedro Costa and Luísa Ribas, “I try to stay neutral”: Digital Assistants and their Stance
towards Gender, Proceedings of ISEA2020: 26th International Symposium on Electronic
Art, (2020). European Commission, ‘Data protection in the EU’, official website of the
European Union, accessed October 10, 2021,
https://ec.europa.eu/info/law/law-topic/data-protection/data-protection-eu_en.
Friederike A. Eyssel and Frank Hegel, "(S)he's got the look: Gender stereotyping of robots," Journal of
Applied Social Psychology, (2012), 42:9, 2213–30.
Leslie Feinberg, "Trans Gender
Liberation: A movement whose time has come," World View Forum, (New York, NY, 1992). Helen Hester,
‘Technology becomes her’, New Vistas, (2016), 3:1, 46-50.
Microsoft, "Using Cortana on iOS or Android," Microsoft Support, accessed May 11, 2021,
https://support.microsoft.com/en-us/topic/using-cortana-on-ios-or-android-caaa50e4-31f1-4165-9659-3caf125ebd38.
Carly Page, "RIP Cortana: Microsoft Silences Digital Assistant On iOS And Android," Forbes, April 1,
2021, accessed October 15, 2021,
https://www.forbes.com/sites/carlypage/2021/04/01/rip-cortana-microsoft-silences-digital-assistant-on-ios-and-android/?sh=425f8bea216b.
Allison M. Piper, "Stereotyping femininity in disembodied virtual assistants," (MA diss., Ames: Iowa State
University, 2016). Christina Richards, ‘Genderqueer and Non-binary Genders’, Critical and
Applied Approaches in Sexuality, Gender and Identity, (London, United Kingdom: 2017).
Katherine Richardson, An Anthropology of Robots and AI (New York and London: Routledge,
2015). Yolande Strengers and Jenny Kennedy, The Smart Wife. Why Siri, Alexa and Other Smart Home
Devices Need a Feminist Reboot (Cambridge and London: The MIT Press, 2020).
UNESCO, “I’d blush if I could: closing gender divides in digital skills through education,”
UNESDOC Digital Library (France, 2019).
Jutta Weber, "Helpless machines and true loving care givers: a feminist critique of recent trends in
human-robot interaction," Info, Comm & Ethics in Society, (2005), 3:4, 209–218.
Joseph Weizenbaum, Computer Power and Human Reason: From Judgment to Calculation (New York and San
Francisco: W.H. Freeman and Company, 1976).
Candace West and Don H. Zimmerman, "Doing Gender,"
Gender and Society, (1987), 1:2, 125-151.
Heathers Suzanne Woods, "Asking more of Siri and Alexa: feminine persona in service of surveillance and
capitalism," Critical Studies in Media Communication, (2018), 35:4, 334-349.
Immersive environments, video tracking and collective interactivity on smartphone : a generic
“dispositif”
DominiqueCunin1 , EmmanuelDurand2 , MichałSeta2
1École des Arts Décoratifs – PSL (EnsadLab),
2Sociétédesartstechnologiques dominique.cunin@ensad.fr, edurand@sat.qc.ca,
mseta@sat.qc.ca
Abstract
In this article, we describe our joined research around building a technological pipeline suitable for creating
artisitic immersive installations that utilize a combination of computer vision and mobile phones as means for
interaction, both in the sense of Human Computer Interaction (HCI) and interaction between human participants.
We frame our research in certain historical and philosophical context and propose a generic but complex
“dispositif,” or device, that has a potential for exploration of diverse artistic themes suitable
for interactivity within immersive environments. Taking our technological stack as inspiration, we propose a
theoretical and conceptual approach to such creation and offer a draft of possible taxonomy that we find useful
in such contexts, namely collaboration, cooperation and competition. We also bring
the theme of surveillance to limelight as a potential and valid theme for exploration that is based on the use
of such technologies.
Keywords
Real time sensorial immersion, Collective interactivity, interactive arts, mobile screens, surveillance
technologies, camera-based pose detection.
Our research is situated in the context of contemporary artistic creation working with digital technologies. We
cross several custom tools to compose a generic technical device offering a strong potential for aesthetic
exploration. To that end, we interface three groups of technical solutions that address the co-situated,
collective interactivity with the help of mobile phones, the sensory immersion via large-scaled video
projections with sound and spectators detection via computer vision systems. The current degree of maturity of
these tools makes it possible to experiment with effective artistic devices with the public, although some
technological deficits still need to be resolved.
But this exploration requires artistic choices: what kind of statements can we uphold with these techniques?
This question is at the root of this article, where we propose theoretical and conceptual foundations for a
first taxonomy of collective interactivity made possible by our technical device. One of the goals of this
article is to situate our work from a critical, theoretical and artistic point of view, which is why we expose
our understanding of the major domains and notions that underlie our research. The notion of
“dispositif” in the practice of interactive arts, considered as being two folded, allows us to make
the link between technical considerations and artistic statements. Immersion is approached through its sensory
stakes, essentially audiovisual, often spectacular in its capacity to trick our senses of perception and whose
monumental scale is articulated in our pipeline to the over-individual scale of mobile screens.
Collective interactivity, which we define from theories belonging to the field of interactive arts, is compared
to studies on collective interaction conducted in computer sciences and in HCI in particular. This allows us to
clarify our own theoretical positioning from which we’ll start our taxonomy attempt. This constitutes our
main contribution to the research: to construct the conceptual foundations of a taxonomy whose ambition is to
propose design guidelines in the creation of immersive devices of collective interactivity.
Thus, we do not propose here a taxonomy in the sense of a complete classification of existing works in order to
organise them between each other according to their characteristics. On the contrary, we tend to anchor its
origins in identified artistic intentions. This is why we begin by formulating a first list of key notions that
define the modalities of inter-individual relations that can make situations of collective interactivity happen.
Drawing from these key notions, we suggest some imaginary examples that take into account the particularities of
our generic apparatus (“dispositif”) in order to demonstrate its efficiency as a conceptual shifter
leading to an artistic creation. Finally, because the spectators are both continuously observed by a tracking
system, but at the same time are also able to act on the immersive projection using their smartphones, we draw
inspiration from population surveillance systems as a guide for our artistic intentions in our future creations.
A certain state of the art in the artistic, technological and societal fields allows us to root these intentions
in a contemporary and political reality of the technologies that we manipulate (and that we also invent) in
order to define a singular critical position that we want to support.
Artistic “dispositif,” technical “dispositif”
In French, the term “dispositif” is commonly used to designate certain works of technological art.
It can be translated in English by “device,” or “apparatus.” We’ll use the French
word “dispositif” in this text as the theories we are using in our research come from French-based
works. It is probably in connection with its technical origin (cf Herme's n°25, Le dispositif entre
usage et concept, 1999—review of the Institute of the sciences of the communication of the CNRS)
that the “dispositif” came to designate a larger part of interactive art works, in expressions such
as “artistic dispositif of collective interactivity.” The theoretical context of this use of the
term in the field of technological art creation is well established and, while the details of its history are
out of scope for this article, we need to recall here some part of it which will serve our purpose.
Beginning with Michel Foucault, Giorgio Agamben reminds us that a device is a network that can be established
between a set of elements with varied origins and qualities: “speeches, institutions, architectural
arrangements, regulatory decisions, laws, administrative measures, scientific statements, philosophical, moral,
philanthropic proposals, in short: the said, as well as the unsaid.”¹³ Furthermore, Agamben sets
a vision of the world separated into two main categories, “the living beings (or substances), and the
'dispositifs' inside which they keep being seized” ¹ (translation by the authors) and the relation
between the beings and the devices would produce the birth of the subject. While an artistic device is
heterogeneous, its components can be, nevertheless, identifiable: “More than a simple technical
organisation, the”dispositifs” puts into play different enunciating or figurative instances, engages
institutional situations as well as processes of perception”¹¹ (translation by the authors).
The experience of being in the presence of, and being connecting with the work is the very prerequisite that
enables the deployment of the “dispositif” to become an intelligible artwork that will produce
sense. This experience is therefore also what motivates its creation. We could say that the
“dispositifs” is what conditions the creation, and what regulates the spectator’s relationship
to the artwork. We find here a very strong echo with the notion of interactivity that we could summarise as
being a relationship appearing between a work and its spectators. This relationship is built from a network of
interactions captured by physical peripherals and transmitted to a computer machine in order to be replayed by a
software architecture (a set of programs) whose ambition is to structure the viewer’s experience in
accordance with the work’s aesthetic intention. The way we understand the notion of interactivity seems to
indicate that two poles exist inside the interactive “dispositif”: artistic and technical. These two
poles influence each other reciprocally in what forms a coherent set of heterogeneous elements, a network of
components of various natures that interlock the ones in the others to produce a single
“dispositif,” similar to the one defined by Michel Foucault.
The artistic, or abstract, side of this “dispositif” implies the implementation of a conceptual
strategy through the author’s aesthetic intention. For the artist, it is therefore a question of
conceiving an aesthetic relation and consequently, of defining what one could call an aesthetic contract of
relations. The technical side of the “dispositif” consists of both hardware and software. The
hardware takes the form of a computer, or a network of computers, that are at the core of the architecture, to
which other devices or computing machines can connect. The software architecture is made of computer programs
that exploit the machine according to the artistic contract, and, finally, scenography in which the
viewer/participant is considered to be part of the whole “dispositif.”
Through this idea of a two-fold “dispositif”, technical and artistic, we defend the idea that one
technical device can support several artistic devices and, consequently, can be the starting point of the
creation of several artistic works which will exploit the technical potential of the “dispositif” in
different ways. In our case, the technical device is an assembly of several complex components:
Splash (1), software dedicated to real time videomapping on any surface;
LivePose (2), software for detecting human silhouettes in a real-time video stream;
Mobilizing.js (3), a library for prototyping interactive works and a platform for collective
interactivity with mobile screens.
SATIE (4), a spatial audio engine, capable of handling audio displays of arbitrary architectures.
From these tools and their interconnection, the aim is to set up a generic technical “dispositif”
that allows numerous artistic explorations from the material it makes manipulable: an immersive visual
environment calculated in real time 3D, a set of data representing the bodies of the spectators in the
experience’s space and, finally, a system of networking of the spectators’ smartphones allowing
organising group interactions.
Pipeline description
Our technical “dispositif” is composed of three software tools, but also of a set of scenographic
and technical factors that we will describe here. The physical space of the experience is necessarily shared by
the viewers because their co-presence is fundamental for us: like the exhibition of a non- technological
artistic work, they are gathered in a dedicated place and form a group. This co-presence is a major starting
point of our research-creation since we aim to question the relationship that the participants can maintain
between them-selves through our “dispositif.”
Immersive video projection is the most prominent technical element that we use, as it can cover large surfaces,
including the totality of the walls of the exhibition space. Whatever the architectural configuration of the
space, a video projection system must be designed to transform the interior space into the equivalent of a
360° projection screen. This task is commonly called video mapping and, in our case, it is handled
by Splash. 10 years in development, it allows segmenting images coming from any video source to broadcast them
through video-projectors placed in the exhibition space. The physical placement of the video projectors is
planned to cover the desired surface and will be virtually reproduced in Splash in order to adjust the
parameters of the projected images deformation: it uses anamorphosis in order to give the viewer an illusion
that they are all facing the same image made with a homogeneous perspective, a condition sine qua non
of a feeling of visual immersion. This is called projection mapping. Splash can use many different sources
of image input ranging from HDMI acquisition cards, video/image files, or network transfers (i.e. NDI
(5) feeds). A multitude of software can be used as image/video source, in our case, Mobilizing.js
will provide the input.
Mobilizing.js is a research project consisting of an authoring software environment for artists and designers.
The objective is dual: on the one hand, to promote the creation of interactive works on different forms of
screen devices (computers, mobile screens, IoT, etc.) and, on the other hand, to be a platform for artistic
creation dedicated to collective interactivity in co-presence. We have chosen Web technologies (Javascript in
particular) as our technical building blocks because they are well spread and based on public standards. Two
main sets make up the Mobilizing.js environment: a library and a platform. Based on a modular logic, the library
of Mobilizing.js aims at gathering different functionalities in a coherent programming interface to make
software art creation on mobile screens more accessible. As for the Mobilizing.js platform, its ambition is to
propose a design and production environment for collective interactivity devices. The scripts created with the
Mobilizing.js library, designed for a single user, can become the basis of a multi-user shared interaction
“dispositif,” using a local network to link mobile screens as well as computers together. The
universality of the web browser, which can be run on desktops as well as on smartphones and tablets, is an
important advantage here. Mobilizing.js integrates, among other things, a primitive real 3D rendering engine as
well as the beginnings of an audio engine, allowing designing artistic creations compatible with Splash: one or
several computers connected to the Mobilizing.js platform take care of rendering of images that will be
transmitted to Splash, which will project them in the exhibition space in an immersive way. In parallel, the
Mobilizing.js platform allows synchronising events (such as user interactions or automatically managed commands)
between smartphones, which opens up the possibility of involving collectives of viewers through their
interactions with their mobile screen, which in turn can allow control over what is represented in the
video-mapping. In its internal architecture, the Mobilzing.js platform is based on Soundworks, a full-stack
JavaScript framework for distributed WebAudio and multimedia applications developed by Ircam collaborators
(https://github.com/collective-soundworks), which uses the WebSocket protocol to manage network communications.
This use of WebSocket allows the Mobilzing.js platform to be open to other environments so that they can share
their data flows with it. It is this openness that allows LivePose to complete our device.
LivePose is a command-line tool which tracks people skeletons from a RGB or grayscale video feed (live or not),
applies various filters on them (for detection, selection, improving the data, etc) and sends the results
through the network (OSC and Websocket are currently supported). Live-Pose is able to do all of this in real
time, processing live video streams and sending out results for each frame at 20-30 FPS. The data transmitted by
LivePose is formatted in JSON, which is particularly compatible with Javascript and, therefore, with
Mobilizing.js. The basic principle of LivePose is to recognize human silhouettes in an image and to deduce a
skeletal structure composed of joint points and segments connecting them. It is therefore a skeletal model
represented by a series of coordinates in the image space (2D) that forms the output of LivePose. Among the
functionalities under development is the agglomeration of images captured by a cluster of cameras in order to
reconstruct these skeletons in a coherent 3D space, which would allow us to represent these skeletons in 3D in
Mobilizing.js and to use the information of positions in the space of the spectators in our works of collective
interactivity.
The above-described software and hardware generic ecosystem is still technologically fragile and complex to
implement. It lacks a GUI and robust communication interfaces. Yet, the software implements interoperability
mechanisms which gives it considerable research-creation potential. But, in the infinity of the possibilities
that artistic conception allows, it belongs to us to frame certain aesthetic and artistic orientations. We are,
therefore, inspired by the technical elements of our own “dispositif” to choose our thematic
directions. A first recent experiment between Mobilizing.js and LivePose was the occasion to lay the technical
foundations of our “dispositif” while opening the tracks of artistic reflexion which we wish to
explore in the future.
Interactive installation example : Materia
Organized by Hexagram, a Canadian research-creation network in arts, culture and technology, in partnership
with Elektra, an organisation that presents works and artists at the confluence of contemporary art and new
technologies, MATE- RIA: Public Laboratory for the Creation of Digital Knowledge offered us an
opportunity, in September 2022, to implement a first prototype of our device. Because it was an event dedicated
to practice-based research in digital art creation and to the meeting between the actors of this community, we
responded to the call for participation by proposing a research-creation workshop focusing on hands: how to
link, through an artistic and technical “dispositif”, the bare hands of the participants, tracking
movement and position with LivePose, and augementing them with smartphones using their numerous embedded
sensors?
Mobilizing.js supported the smartphone application, a large screen image display application, the networking of
all the data flows in a dedicated local network, as well as a debugging application to check the status of the
data transition in the network in real time. LivePose was modified specifically for this project to track
users’ hands as accurately as possible from a group of four cameras placed in the centre of a table. The
goal was to reconstruct their positions in 3D in the space of the experiment. These data were transmitted in
real time to Mobilizing.js, on the network dedicated to the experiment, which allowed producing graphical
representations in 3D of the spectators’ hands and thus, by extension, to define particular
interactivities between these virtualized hands and the smartphones held by the participants. The hands
positions and actions were, furthermore, sonified using SATIE. With the help of the workshop participants
(should we mention the names here?), different scenarios of collective co-situated interactivity based
on the hands have been imagined. The result is the possibility of composing an ephemeral landscape using the
traces produced by the movement of bare hands in the tracking zone of the cameras. The smartphone of each
spectator is assigned a short sound, different for each one and conceived on the spot by the workshop
participants. When the user touches the screen of the smartphone, it starts the recording of the movements of
the hands that are present in its video capture area and the sound is played by the smartphone. SATIE
sonification provides an ambient sonic texture linked to the detected presence of hands by LivePose. When the
user releases the screen, the recording and sound playback stops. The traces left by the hands, visible in a
dedicated screen that simulates an immersive environment, remain in the space for some time before being erased,
which allows not to saturate the graphic space for too long and to leave room for new participants to the
experience.
Thanks to this experience, which sketches a collective work of art, we were able to verify empirically that the
artistic potential of our device was indeed considerable, as much as the technical difficulties of
implementation that it reserves to us. It is important for us, at this stage, to properly situate our positions
regarding the different components of this “dispositif”, especially since we were not able to
implement projection mapping in this experiment. An artistic orientation nevertheless made its appearance
implicitly: the power of contemporary computer vision systems, from which Live- Pose is built, and their
implementation in public spaces leads us to a critical reflection on systems built for monitoring individuals.
But before exploring this theme more deeply, it seems essential to us to get back to both the notion of
immersion and the notion of collective interactivity.
Sensory Immersion
Immersion technologies have been present in artistic practices for several decades. Their historical roots are
well known and can be found in several in-depth studies (Olivier Grau - Virtual Art). It is accepted that the
19th-century panoramas are the origin of contemporary digital devices, as they share the same dream: to trick
human vision in order to make the viewer believe that he is somewhere else than where he really is. As mentioned
above, sensory immersion in an artistic device is achieved through the technical “dispositif” as a
whole, including the physical space, which is augmented by dedicated elements, including audio and video
projections. It is this adaptation of image projections onto a physical space that will contribute to the
sensory immersion. The inclusion of the physical space in the definition of sensory immersion that we are making
here implies a collective experience of immersion, common to all participants in the same space. We therefore
exclude technical devices built around virtual reality headsets and other individual hardware, except in hybrid
uses that maintain this fundamental relation-ship to physical space. The generic term for what immersive
projection devices produce is spatially augmented reality. Once images can be projected onto large surfaces,
several uses become possible. It can be a fully immersive space broadcasting views of a totally different
place than the physical one, a visual augmentation of the physical place by scattered elements of decor or
texture, or any balance between these two extremes, according to the continuum of mixed realities coined by
Milgram et. al. (Paul Milgram, Haruo. Takemura, Akira Utsumi and Fumio Kishino, Augmented Reality: A class of
displays on the reality-virtuality continuum, in Proceedings of SPIE 2351, Telemanipulator and Telepresence
Technologies, 1995).
Video-mapping is one of the tools enabling this enhancement. It has many and diverse uses in scientific
(planetariums), educational (Salame 2022, Puget 2019), museum (Nikolakopoulou et al. 2022), entertainment (Lee
et al., 2015) and, of course, artistic (Lambert 2012 for example) fields. Its use to cover the periphery of
publicly accessible physical space is the most common, and the technique for doing so is now well mastered:
automatic geometric (Kurth et al. 2018) and colorimetric (Huang et al. 2017) calibration, standardisation of
content representation formats, and dedicated hardware off the shelf show that it is now a known
medium, although it still presents some complexity of implementation in situ.
As part of the many practices of video-mapping, spatial augmented reality is distinguished by the projection of
graphic content into an interior space. The contents are interwoven with the physical space, which can become a
medium of information, a window to a virtual world, or be transformed or even temporarily rendered invisible.
The work Displacements (Michael Naimark, 1980, ²⁰) could be considered as one of the first
applications of spatial augmented reality, clearly setting out the challenges of the hybridisation of realities,
while the series Bumpit (Bertrand Planes, 2011)⁴ is a more modern version of it and
illustrates well the possibilities of transforming the perception of physical space within the framework of
artistic installation. From a more technological and prospective point of view, the research project Room Alive
¹⁸ has demonstrated interactive uses of this spatial augmented reality involving several users: a
room of daily life (e.g. a living room) is augmented with the help of an immersive image projection system with
which the inhabitants can engage in ‘physical’ interaction, for example by walking on the images of
small creatures evolving on the walls or on the floor. The co-presence necessarily induced by the spaces chosen
for these experiments (living quarters) thus implied the development of a form of collective interactivity. On
the basis of this observation, an experimental device produced within the framework of the same research project
concerned remote communication between individuals, a form of video-conferencing of a new kind which proposes to
video-map the interlocutors of a conversation in their respective spaces of physical presence [Room2Room:
Enabling Life-Size Telepresence in a Projected Augmented-Reality Environment, ²². These last
references show us that interactivity plays an essential role in the perception of immersion (as shown by Hudson
et al. 2019, interaction is as important when dealing with virtual objects as it is with real ones). But the
implementation of collective interaction requires an appropriate technical device as well as a clear
understanding of the very terms that make up this expression: collective interactivity. To clarify these terms
conceptually is an imperative for us to conceptually establish the artistic “dispositifs” that our
generic technical “dispositif” will support.
Collective interactivityvia smartphones
Collective co-located interactivity has been a new field of artistic experimentation since mobile screens
(smartphones and tablets) became massively widespread, in other words, at the turn of the 2010s, after the
release of the Apple iPhone. Although the emergence of these devices can be considered as the starting point of
a specific artistic research, working with mobility and the technical potential that it confers to these
screens, the recent history of art and design presents some premises of collective interactivity.
Indeed, the artistic practices of interactivity have led to experiments concerning collective forms of digital
interactivity, i.e. devices requiring the active participation of several people, often gathered in the same
physical place. The colocation of the participants, turned to actors by the “dispositifs”, is one of
the criteria of distinction with shared interactive systems and artworks, using the computer networks to create
communities of geographically distant users, but joined together by the means of images, as in the cases of
persistent virtual environments. “Localized media” are part of these practices that combine the
mobility of devices equipped with positioning technologies (GPS) and the participation of a group of users. The
artists’ studio Blast Theory, with experiences such as “Can you see me now?”⁵ are among
the pioneers of these practices.
Among the other fields of computer networks usages, we know that the state of the art of groupware, or
multi-user interactive systems is very rich, in particular within Human-Computer Interaction (HCI). One of the
many reference articles on the subject of groupware dating from 1991 ¹² gives a taxonomy of
multi-user interactions on which current research in this field of computer science is still based.
The state of the art is much more limited regarding aesthetic “dispositifs” of situated collective
interactivity. One of the few research projects that tend to combine interactive art making and locally
networked creation with mobile devices is ³ . It consists in a creative environment using interconnected
smartphones that allows users to play with a shared, locally broadcast musical environment. Made of a mix of
technologies made compatible together using a data exchange protocol (OSC), each mobile becomes a kind of
instrument whose individual manipulations are pulled into an audio composition broadcast over speakers.
A design-based research on collective interactivity from an HCI perspective helps to identify the major
references in this field of research ²³. This paper aims to review a series of research works and
interactive devices involving multiple people made by this research team based in Aarhus University. One of the
assertions is that the whole set of arrangements must be taken into account when designing collective
interaction devices. This leads to the following concept of collective interaction: “CI focuses on how the
interaction supports human-human interaction through the spatial organisation of people and coupling of the
interaction”. Three main notions are considered as fundamental in the design of collective interactions:
“Interaction Proxemics, Social Interaction and Co-experience.” Proxemics holds a particularly
important role, since it conditions the type of social interactions that can be experienced by the collective of
spectators/users. Indeed, the notion of proxemics comes from the social sciences’ research initiated by
Edward T. Hall ¹⁵. It can be considered as the science of behaviours between humans induced by the
physical proximity of people in relation to each other, the organisation and structuring of space being able to
influence these behaviours. A matrix based organisation is thus proposed to analyse the collective interaction
devices. This analysis tool, as the authors specify, “should not be seen as depicting predictive, causal
relationship, but rather as a way to illustrate how certain design strategies can be brought into play in order
to pursue intended use qualities” (p.74). It is therefore a study that wishes to help in the design of
collective interaction experiences. This kind of grid (or matrix) is also found in research carried out by our
team ¹⁹ which also proposes an analysis of the constituent elements of a work using collective
interactions (Group size, activity type, I/O Distribution, etc.) in order to extract a classification matrix.
The question of interaction with mobile devices, particularly smartphones, is very interesting on many levels
that are out of scope of this article, but one of the areas that remain unexplored is measuring and
understanding the why of some of the interactions in addition to where and when. This
aspect is explored via a research topic and a corresponding application called MyExperience ¹⁴ which
measures 140 parameters and follows the user around many different contexts of their everyday life. In addition,
Dalsgaard et al. ⁷ offers a theoretical basis of the elements of dynamic process of engagment
that an interactive experience can offer. Their analysis is based on three different types of installations
in three different contexts: public art, museum installations and a department store interactive marketing
medium. This is further expanded by Schroeter et al. ²⁴ who position the situated engagement
accross three parameters: people, content and location, which have implicit cultural implications.
Some foundations for a taxonomy of collective interactivity figures
One of the differences between these proposals and our position is that, in our research, it is not so much a
question of collective interaction as of collective interactivity.
The ambiguity of the term interaction in the field of HCI is explored in ¹⁶, which lists different
concepts from which the HCI literature has based some meanings of the term (such as Dialogue, Transmission, Tool
use, Experience or Control, summarized in a table p.5042). The term interactivity is also briefly mentioned
(p.5041) as being both an equivalent and an alternative to interaction and, therefore, subject to discussion.
The difference between interaction and interactivity is not discussed. However, the artistic practices of
interactivity gives us a definition of the two terms as we have mentioned it earlier, about the twofolded
“dispositif”. We base our own definitions on the researches led by the team Aesthetics of New Medias
(Paris 8 university) to define the notion of interactivity as being a relation which takes shape between
an artistic “dispositif” and its addressees ⁶. In the field of artistic creation,
this relation is made possible by the capture of interactions engaged by the spectator when he use a physical
interfaces and by a software structure that allows to “replay” this interaction. For example, the
movements of the mouse is an interaction captured by the mouse, a physical technical device moved by the
user’s hand. This interaction is transmitted to the computer as numerical coordinates which, when used in
a specifically designed program, allow the construction of different relationships with graphic or sound
objects: moving a cursor around the screen, controlling the orientation of the point of view in a real-time 3D
space (in video games, among others), or drawing colored segments in an area of the screen are all situations of
interactivity enabled by the same interaction. Interactivity is thus built by programming out of interactions.
Research in HCI frequently build tools for analysis and help in the design of devices, because it needs to
evaluate interaction systems quantitatively: the goal is to prove the effectiveness of one interaction technique
in relation to another according to its context of use. Interactive art practice, on the other hand, seeks to
establish non-quantifiable aesthetic relationships between the audience and the device that it encounters: it is
a question of making an artwork in the strong and artistic sense of the term. This is the reason why, in our
research, we propose a series of keywords that designate types of relations between people as starting points
for the definition of collective interactivity modalities and the conception of artistic devices implementing
them. Our list is not exhaustive but currently consists of the following terms: collaboration, association,
division, participation, cooperation, confrontation, inspection, supervision, competition, reconciliation or
opposition. All of these terms qualify relationships between individuals and can be combined to formulate
scenarios of collective interactivity works.
Our contribution therefore consists in the clarifying the foundations of a taxonomy of collective
interactivities to be done. We should then elaborate, furthermore, about the possible entries of this taxonomy,
define them in order to constitute some kind of guidelines to support the design of collective interactivities
experiences in an already existing and operating technical context. This approach is therefore the opposite of
the studies already carried out and quoted above.⁷ But before presenting the first early draft of this
taxonomy, we must complete the conceptual framework we are building here by addressing the theme of surveillance
inferred by nowadays technologies, including the very ones that we use.
Surveillance and control reversal
As we briefly mentioned previously, as LivePose is fundamentally a system for tracking individuals, it leads us
to a consideration about population surveillance systems using computer vision technologies. This theme seems
relevant to invest in our artistic works scenarios, knowing that to-day’s “digital everything”
(and mobile devices in first line) transforms our societies into people tracking and massive personnal data
storage machines, in public as well as private contexts. Contemporary art has already addressed this problematic
reality of our societies. Rafael Lozano-Hemer,¹⁰ David Rokeby and The Surveillance Camera Players are
among artists who have addressed the questions of surveillance in different ways. As ²¹ says, the
sense of being surveilled is highly dependent on an individual’s place and era of birth, geographical
location, and culture. By this, the artist means that some of us can be very sensitive about being watched by
hidden devices because their daily living can be constrained by it, whereas others don’t really care even
if they know surveillance systems are lurking to spy on them. Artists try to show how these technologies can
interfere with our existences and change the way we socially behave.
Using contemporary technologies to observe or record participants’ actions in collective installations
raises a lot of ethical questions. What kind of data is being recorded? How is data being used? Will the data be
stored? Are there any identifiable features? Can we trust the security of the stored data? Some of the research
listed above mentions these elements and takes them into account, such as in MyExperience application, which
collects a lot of identifiable data. HCI research addresses these questions via standard legal devices. How-ever
there are known artistic works that allow themselves to be a commentary on some aspects of ethical questions.
These questions are inherent to the potential of the technologies that are employed.
Accordingly, one of the widely explored themes in interactive, collective multimedia installations is (some
form) of surveillance. The approaches vary greatly from reflections on the dangers of various forms of biometric
tracking,⁸ through generating empathy (Rafael Lozano-Hemmer, Level of Confidence (2015)), to playful
critiques of public policies regarding implementations of different forms of biometric tracking,²⁶ to
mention just a few. However, if we take into account all the possibilities of computer vision and tapping into
mobile devices and social media to learn more about individuals, we realise how powerful such knowledge could
become. One of the most prominent examples is China’s Social Credit System (CSCS) that quantifies social
and civil integrity of all citizens and corporations, ²⁸,²⁷ and which is
actually almost as old as China itself.¹⁷ The most frightening part of China’s SCS maybe its
scale, as it is already used in several big cities of the nation. In such context, the data becomes not only a
tool for interaction between humans and machines but also as a powerful weapon of control. The political,
humanitarian and civil issues of such systems are out of scope for this article and are explored
elsewhere²⁷,²⁵,⁹ but we mention them here for a broader
context.
We are conscious of the issues surrounding user data, privacy and consent when building multiuser interactive
installations. We are presenting a prototype of an installation that uses many techniques for tracking users
such as pose and action detection, position as well as interaction with a mobile phone and we propose an
artistic and playful take on the subject of user consent to be tracked, as described, and how we can use that
data in attempt to control the users, either individually or collectively. Such playful uses of topics related
to surveillance are not new²,²⁶ but we believe that such approaches are necessary and
efficient in raising awareness of these issues among some slices of populations.
The technological assemblage that we propose covers several aspects of a production pipeline that includes
input (computer vision, interaction opportunity via smartphone), show control (timeline-based and interactive),
and display (video mapping, audio spatialization). We aim to build a system and know-how that can be generic
enough to be useful for other types of artistic, design, and social manifestations. Our proto-typical
installation offers only one of the possible views and uses. We hope that our solution can serve as a starting
point for others.
We will conclude with three hypotheses of artistic installations based on the conceptual positions and
technical choices we have outlined throughout this article. This work relies on the generic
“dispositif” that we are setting up: an immersive “big picture” realized by
video-mapping, a tracking system of the spectators in the space and smartphones interconnected with a local
network, the whole activated by a group of spectators invited to share this experience. Surveillance is the
general theme that we choose for these experiments, which allows us to contextualize our subject and to lead our
choices of implementations. In order to ensure a certain critical and reflexive dimension, we aim to provide a
kind of situation reversal in the experiments, the controlling power passing from one side to the other: in a
first time, the spectators interact with a system which relies on informations captured about them (their
position in the room, the proximity between them, the gestures they produce with their smartphone, etc.) and, in
a second time, this system reveals itself to everyone by showing the “other side of the story”,
i.e., the information that has been used by the system to function. The black box becomes transparent. The
visual elements that will be used will be mainly textual: words, sentences, paragraphs will be displayed in the
smartphones, as in the immersive projection. The relation between the spectators and these textual elements will
be regulated by the “collective interactivity figures” that we retain for these first effective
artworks : collaboration, cooperation and competition.
Collaboration
The immersive projection shows a series of white-on-black text excerpts about surveillance and about textual
content control systems in different contemporary contexts. The scale of the ethical “seriousness”
of these systems would go to a crescendo. One can think of the erroneous corrections produced by automatic
spellcheckers when writing SMS, to the censorship and generalized surveillance exercised on social networks in
certain countries with totalitarian political regimes that can lead to the incarceration of militants, for
example. These texts do not appear complete, some words are missing. The audience has to fill in the gaps in
these texts with their smartphones. The texts appear in a sequential way: a first text is displayed, when it is
completed by the public, it gives room to a second text, and so on. This principle allows the progression in the
“seriousness” of the following texts.
In their smartphones, the spectators see a circular list of words displayed that they can rotate to select one
of the words. A “send” button, similar to an instant messaging interface, allows the word to be
transmitted to the text in the immersive image. This list of words changes according to the proximity of the
people with one of the empty spaces in the text: it is necessary to be close to one of the holes of the text to
be able to fill it, or else, the screen indicates “out of reach”, meaning the impossibility of
participating. Thus, if the spectators do not collaborate together by spreading out in front of the text, they
will not be able to fill it in to discover the next text.
The last text that appears is a report of the interactions that took place during the experiment, a description
exposing the data collected by our “dispositif” to make the artwork operate: the number of
participants over time and, for each one, their number of right and wrong answers, the time needed to complete
the texts, the number of participants connected but who did not collaborate (being all the time “out of
reach”), the models of smartphones used, etc. A form of natural language visualization of the data
collected throughout the experience, reminiscent of “bots” or automatic conversational agents.
Cooperation
Cooperation and collaboration are distinguished by the temporality of group interactions. Collaboration does
not necessarily imply a temporal concurrency, the actions complete each other even if they are not synchronous,
as in the example above. Cooperation implies, according to us, at least a proximity, and to the most a temporal
synchronization between the interactions, the actions must be operated jointly by all or part of the group of
spectators.
The immersive image has a white, plain background. Each spectator sees a white word on a black background
appearing in his smartphone. When they tap the screen of their smartphone with their finger, the word appears in
the immersive image in the place where it belongs in a sentence. If all the spectators tap their screen with
their finger, all the words appear in the projection, but a certain order must be respected in these
interactions, otherwise the words disappear just after their appearance in a fading effect: the spectator who
sees the first word of the sentence in his smartphone must be the first to tap his screen, followed by the
second, then the third, and so on until all the spectators have made their word appear in the large image in the
right order of the sentence. When a sentence is completely displayed, the colors are reversed in the screens,
the white background becomes black and the text is drawn in white and vice versa on the smartphones.
This sequential collaboration between the spectators will allow them to progressively discover sentences coming
from texts about industrial espionage cases based on security flaws in computer systems.
Competition
The immersive image is empty of text and is split in two coloured areas: white on the left and black on the
right. In their smartphones, viewers are asked to write their nickname before they can see an editable text
field (a prompt) centred in their screen. Depending on the person, the background colour of the screen is black
or white. The group of spectators is actually split in two using this colour: if there are 10 spectators, 5 will
have a black background and the others will have a white background. Under the prompt, on the mobile screen, the
following inscription can be read: “Look for those who do not have the same background colour as you and
report their misdeeds!” Using the prompt, viewers can write what they want to report. An automatic
moderation system checks whether the sentence is a denunciation or not. If they are successfully recognized as
denunciations, the sentences are written in negative in the corresponding coloured area in the immersive image
(i.e. white text on a black background). If not, the viewer’s screen is marked with an error message and
the prompt is cleared so that he or she can try again.
If no one writes with their smartphone after a certain period of time (30 seconds to 2 minutes, to be verified
in real conditions), sentences from a collection made beforehand and integrated into the program are displayed
from time to time in one of the coloured zones of the immersive image. The idea is to introduce doubt: did
someone write this, or is the device expressing itself?
The group that manages to fill its screen with sentences first wins the right to write whatever they want
without any automatic moderation and on the whole immersive image, which will take on the background colour of
the winning group, for 1 minute. After that, a list of all the sentences written during the whole session with
the associated user’s nickname are displayed for a few seconds, exposing everyone’s actions to the
eyes of the public, before the device resets.
These 3 scenarios are still to be refined in their details, but they show how a key notion can be used as the
root of an artistic proposition using our “dispositif”. Our future work consists in concretely
realizing these scenarios and putting them to the public test.
Acknowledgements
We would like to thank our Materia workshop participants: Annae ̈lle Sermanson, Idun Isdrake, Heather
Kelley, Valentine Auphan, for their involvement along the path, Hexagram and Elektra for facilitating the
workshops and contacts with other creative humans.
References
1 Giorgio Agamben, Qu’est-Ce Qu’un Dispositif? Rivages poche, Petite
Bibliothèque, 2007.
2 Anders Albrechtslund, Lynsey Dubbeld, “The Plays and Arts of Surveillance: Studying
Surveillance as Entertainment”, In: Surveillance & Society 3.2/3, Sept 2002, ISSN:
1477-7487, DOI: 10.24908/ss.v3i2/3.3502.
3 Jesse Allison, Yemin Oh, Benjamin Taylor, “NEXUS: Collaborative Performance for the
Masses, Handling Instrument Interface Distribution through the Web”, In: (), 6.
4 BumpIt! Main Page, http://www.bertrandplanes.com/bumpit/.
5 Can You See Me Now? — Location Based
Game, https://www.blasttheory.co.uk/projects/can-you-see-me-now/.
6 Jean-Marie Dallet, “Figures de l’interactivite", In: Anomalie, Mar.
2003.
7 Peter Dalsgaard, Christian Dindler, Kim Halskov, “Understanding the Dynamics of Engaging
Interaction in Public Spaces”, In: Human-Computer Interaction – INTERACT 2011, Ed. by
Pedro Campos et al. Vol. 6947, Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, 212–229, ISBN:
978-3-642-23770-6 978-3-642-23771-3, DOI: 10.1007/978-3-642-23771-3 17.
8 David Rokeby, Border Patrol (Collaboration with Paul Garrin),
http://www.davidrokeby.com/border patrol.html.
9 Ann Dennon, Students on School-Issued Devices Are Under Constant Surveillance —
BestColleges, https://tinyurl.com/4sastf6u, oct 2021.
10 Brian Droitcour, Rafael Lozano-Hemmer on Seductive Participation and the Oppression
of Metrics, June 2020.
12 Clarence A. Ellis, Simon J. Gibbs, Gail Rein, “Groupware: Some Issues and
Experiences”, In: Communications of the ACM 34.1, Jan. 1991, 39–58, ISSN: 0001-0782,
1557-7317, DOI: 10.1145/99977.99987.
13 Michel Foucault, Dits et Écrits, Vol. III, 1977.
14 Jon Froehlich et al. “MyExperience: A System for in Situ Tracing and Capturing
of User Feedback on Mobile Phones”, In: Proceedings of the 5th International Conference on Mobile
Systems, Applications and Services - MobiSys ’07. San Juan, Puerto Rico: ACM Press, 2007, p.57,
ISBN: 978-1-59593-614-1. DOI: 10.1145/1247660.1247670.
15 E Hall, The Hidden Dimension, Anchor, 1966.
16 Kasper Hornbæk, Antti Oulasvirta, “What Is Interaction?” In: Proceedings
of the 2017 CHI Conference on Human Factors in Computing Systems, Denver Colorado USA: ACM, May 2017,
5040–5052, ISBN: 978-1-4503-4655-9, DOI: 10.1145/3025453.3025765.
17 Min Jiang, “A Brief Prehistory of China’s Social Credit System”, In:
Communication and the Public (), p.6.
18 Brett Jones et al. “RoomAlive: Magical Experiences Enabled by Scalable, Adaptive
Projector-Camera Units”, In: Proceedings of the 27th Annual ACM Symposium on User Interface Software
and Technology, Honolulu Hawaii USA: ACM, Oct. 2014, 637–644, ISBN: 978-1-4503-3069-5, DOI:
10.1145/2642918.2647383.
19 Oussama Mubarak, “A Taxonomy of Multi-User Co-Located Interaction Environments”,
In: (), p. 4.
20 Michael Naimark, Displacements, http://www.naimark.net/projects/displacements.html.
21 B.D. Owens, “Watching Me, Watching You: Reflections Upon Surveillance, Gare Loch Duality
and the #UndesiredLine”, In: Media-N 17.2 (Oct. 2021), 49–68, ISSN: 1942-017X, 2159-6891, DOI:
10.21900/j.median.v17i2.882.
22 Tomislav Pejsa et al. “Room2Room: Enabling Life-Size Telepresence in a Projected
Augmented Reality Environment”, In: Proceedings of the 19th ACM Conference on Computer-Supported
Cooperative Work & Social Computing, San Francisco California USA: ACM, Feb. 2016, 1716–1725, ISBN:
978-1-4503-3592-8, DOI: 10.1145/2818048.2819965.
23 Marianne Graves Peterson, Majken Kirkgaard Rasmuseen, Peter Gall Krogh, “Collective
Interaction: A Designerly Visual Analysis of Seven Research Prototypes”, In: Proceedings of the 29th
Australian Conference on Computer-Human Interaction, Brisbane Queensland Australia, ACM, Nov. 2017, 68–
76, ISBN: 978-1-4503-5379-3, DOI: 10.1145/3152771.3152779.
24 Ronald Schroeter, Marcus Foth, Christine Satchell, “People, Content, Location:
Sweet Spotting Urban Screens for Situated Engagement”, In: Proceedings of the Designing Interactive
Systems Conference on - DIS ’12, Newcastle Upon Tyne, United Kingdom: ACM Press, 2012, 146, ISBN:
978-1-4503-1210-3, DOI: 10.1145/2317956.2317980.
25 Saif Shahin, Pei Zheng, “Big Data and the Illusion of Choice: Comparing the Evolution of
India’s Aadhaar and China’s Social Credit System as Technosocial Discourses”, In: Social
Science Computer Review 38.1, Feb. 2020, 25–41, ISSN: 0894-4393, 1552-8286, DOI:
10.1177/0894439318789343.
26 Karen Louise Smith et al. “Playing with Surveillance: The Design of a Mock RFID-based
Identification Infrastructure for Public Engagement”, In: Surveillance & Society 9.1/2, Nov. 2011,
149–166, ISSN: 1477- 7487, DOI: 10.24908/ss.v9i1/2.4108.
27 Alexander Trauth-Goik, Chuncheng Liu, “Black or Fifty Shades of Grey? The Power and
Limits of the Social Credit Blacklist System in China”, In: Journal of Contemporary China, Sept. 2022,
1–17, ISSN: 1067-0564, 1469-9400, DOI: 10.1080/10670564.2022.2128638.
28 Karen Li Xan Wong, Amy Shields Dobson, “We’re Just Data: Exploring China’s
Social Credit System in Relation to Digital Platform Ratings Cultures in West- ernised Democracies”, In:
Global Media and China 4.2, June 2019, 220–232, ISSN: 2059-4364, 2059- 4372, DOI:
10.1177/2059436419856090.
ARt chat - A Museum App combining AR, Art and Communication
Museum labels are often looked at more intensively than the actual work itself. And who says that the
interpretation written there is the only possible one? ARt chat is an application that enables museum
visitors to place their own opinions, knowledge or even questions as comments virtually in the exhibition space.
Through the application, they can also discuss the artworks with others and follow and respond to comments even
after leaving the exhibition. The application combines augmented reality, art and communication.
Keywords
augmented reality, digital participation, co-creation, user-generated content, museum studies, audience
participation.
Digitization and social media are influencing our daily lives—the way we communicate with each other,
make decisions, learn, work and spend our free time. We have long been accustomed to being able to express
ourselves and participate everywhere. This tendency has not spared the cultural sector, were museums and other
cultural institutions strive to keep up with the rapid pace of digital development. In fact, the explosion of
ubiquitous technology has been developing hand in hand with the paradigm shift in the museum world, which was
set in late 1980es as new museology, a movement in the museum studies marked by Peter Vergo ́s
seminal work of the same title.¹ The focus on the social role of museums, new styles of communication,
inclusiveness and openness for active visitor participation have been the hallmark of the movement and a plea
for new values and beliefs to be strived for in museum community. Forty years fast forward, there is still a lot
of work to be done. In an environment characterised by participation and empowerment, many museums and cultural
institutions are still seen as elitist and aloof. However, most museums did become institutions that are more
centered on the needs of their audience. They are trying to fulfil not only the educational but also
entertainment demands, reflecting the dynamics and the multicultural nature of the 21st century and favouring
dialogue, interpretation, and experience.² It has become clear that it is no longer enough to exhibit
objects and that the question of context should be central.
It has also become clear that in institutions that serve to create, identity and offer a space for
representation, those who are to be represented should consequently be included in their work. The museum system
and its traditional concepts have long been questioned, counter-designs developed, collection structures
reconsidered and, with the involvement of the communities, new ways of telling stories have been invented. In
this context the International Council of Museums (ICOM) discussed and approved a proposed resolution for a new
version of the museum definition at a General Assembly in August 2022, concluding an 18-month participatory
process of revision. The new version of the ICOM Museum Definition reads as follows:
“A museum is a not-for-profit, permanent institution in the service of society that researches, collects,
conserves, interprets and exhibits tangible and intangible heritage. Open to the public, accessible and
inclusive, museums foster diversity and sustainability. They operate and communicate ethically, professionally
and with the participation of communities, offering varied experiences for education, enjoyment, reflection and
knowledge sharing.”³
And though the bottom-up approaches have been increasing in the last decades, the idea that visitors should
contribute to the museum creation with their own voices remains somewhat radical and provocative.⁴
At the same time, the process of museum transformation has been immensely supported by technology development.
The democratization of the museum has been coinciding with the democratization of technology, characterized by
its accessibility to everyone and everywhere. Moreover, due to its rapid expansion in everyday life, technology
is for the young generations of museum visitors an aspect more familiar than any other in the museum context. To
step into a museum that is not offering a digital layer is seen almost as a kind of anachronism that breaks the
continuity of their reality experience. So more than just accompanying the museum experience, new technologies
are being so powerful that they are clearly changing the very concept of the museum. As Tomiuc argues,
“the issue is no longer whether to use this technology to recreate the museum experience, but how to use
it for a maximum impact on the audience.”²
And though the expected digital literacy might pose a barrier to the older generations, which is a worthwhile
topic, the fact that museums are struggling to attract new generations of visitors is posing a bigger concern
for their future. The continuity of daily experiences in a technological society is interestingly described by
Peter Samis in his essay “The Exploded Museum”:
In a technological world, the museum visit no longer begins when a person enters the building, nor need it end
when she or he leaves. The museum’s physical space is but one site—albeit a privileged one—in
the continuum of the visitor’s imaginative universe.⁵
In a museum experience, this symbiotic relationship of digital and analog is not only enabling the melting and
intertwining of the pre- and post-museum visit time, but also of that which is presented/curated and understood/
commented, and of that which happens behind the scenes and in front of the scenes. In the next chapters, we are
presenting the process of development of a mobile AR application for museum use that was conceptualized with
that incentive in mind and emerged as a product in a project dedicated to the empowerment of museum community.
Project and goals
nextmuseum.io is a cross-institutional digital community platform for swarm curation and co-creation, initiated
by NRW-Forum/ Kunstpalast Düsseldorf and Museum Ulm, funded by the Digital Culture Programme of the German
Federal Cultural Foundation as well as by the Federal Government Commissioner for Culture and Media.
On nextmuseum.io, curators or institutions can call for joint work on exhibition projects via an Open Call.
Artists can submit artworks and participate in discussions, as can art enthusiasts, who have various
opportunities to participate throughout the entire process. We believe that the multitude of opinions and
concentrated knowledge that characterizes the online community creates an intelligent, powerful swarm that wants
to be involved in shaping things. The platform offers tools and support for collaborative work in the art and
tech sector, digital experiments developed with the project partner MIREVI (Mixed Reality and Visualization)
from the University of Applied Sciences Düsseldorf and community events for exchange and empowerment.
Since its launch in July 2020, more than 30 Open Calls for collaborative exhibition projects have been made via
nextmuseum.io and more than half of them have already been shown physically or digitally. The community now has
more than 500 members and participated in about 75 events - from meetups with curators and expert talks on
current topics (e.i., AI, sustainability, XR, NFT) to the tech art late night show “It's tech
time!”.
Related Work
The use of Augmented Reality (AR) technology in museums has been on the rise during the last
decade.⁶,⁷ In comparison with other types of technologies on the Mixed Reality
spectrum,⁸ AR has gained popularity in the cultural sector in general because of its broad availability
and the simplicity of its use. Unlike Virtual Reality (VR) technology, which often requires expensive hardware
and a more complex setup, mobile AR experiences only need a handheld device in the form of a (personal)
smartphone or a tablet and are usually downloadable as applications. Furthermore, the App-based AR is
increasingly being replaced by Web-based AR, which provides an even more accessible experience and on which
great hopes are pinned in the coming years.⁹ In this chapter, however, we want to give a short overview of
the use of AR technology in museums, independently of its individual technological characteristics, primarily
focusing on the multitude of its potentials and benefits for the mediation of museum content.
The typical feature of AR is the superimposition of a digital, usually visual content on top of the content
that already exists in the real-life scene in front of the person. This sort of layering enables a myriad of
possible symbolical uses, such as apparent overlapping of different times in the same space, a comparison
between reality and image, notation or marking on the surface of reality, an image of hallucination or fantasy,
a notion of transience and instability, an illustration of invisible processes, or telepresence of people or
objects, to name just a few. When used in an exhibition context, these symbolic potentials open enormous
opportunities, both for the exhibited content as well as their interpretation and mediation of the same.
In her text from 2017, Shelley Mannion, the Digital Learning Programmes Manager at The British Museum, reports
on the conclusions that her team had drawn from their several years-long exploration of AR’s potential in
museum education. Acknowledging the inability to include all its variations, they attempted to classify
different types of uses of AR in museums by suggesting four categories: (1) Outdoor guides and explorers; (2)
Interpretive mediation; (3) New media art and sculpture; and (4) Virtual exhibitions.⁷ While the first
category relates to the museum content outside of its walls and coincides with the GPS based AR applications,
the second one refers to some of the first interactive museum applications that engaged the visitors in physical
interaction by combining wearables and facial recognition with the marker-based AR technology. The last two
categories are jointly described as ones coming from artists, which were “the first to recognize
AR’s potential to challenge the curatorial hegemony over galleries.”⁷ The work of the artist
collective Manifest.AR has here been a well-known example of guerrilla interventions in the museum
space enabled by AR technology. ¹⁰ As Mannion emphasizes, this and similar examples illustrate
“one of the strengths of AR for museums: its ability to provide multiple layers of invisible
interpretation in galleries.”⁷ In a similar attempt to offer an overview of the use of AR technology
in museums, Charlotte Coates lists several types of its adoption in museum contexts. Along with accompanying
examples, these include adding explanation to museum pieces; bringing objects or scenes to life as a sort of art
intervention; interactive and immersive projections; taking collections beyond their walls in a digitalized form
and using AR for learning experiences, especially in different types of scientific museums.⁶
While both authors provide useful and interesting insights into the topic, they do not include any examples
that would illustrate the use of AR technology for direct participatory practices of visitors in museums. All
the examples refer either to a digital extension and transformation of usual interpretation tools, or to a form
of AR artworks. The AR tools that would enable activities or support values related to visitors' engagement in a
form of expression and exchange of their opinions or communication with the curators and the artists are
not mentioned. Without wanting to conclude that such projects do not exist in AR form, this does show us that
they are not yet common or influential enough to include them in such analysis as a separate category.
Similar attempts have, however, been carried out with other types of interactive and mobile technologies. By
using digital technologies for creation of a personalized and interactive visitor experience through
contribution and exchange of user-generated content, the installation called “The Cell” (2015) at
the Information Age Gallery of The Science Museum London¹¹ and “Debatorial” (2020), an
interactive online platform launched as part of the digital exhibition “Beyond States. The Boundaries of
Statehood” at the Zeppelin Museum in Friedrichshafen,¹² bare some important similarities to our
project. “The Cell” enables visitors ́ participation in the installation by sending a text
or adding an image via their mobile phones, which then unlocks additional content or include them in live polls.
The contributions and the information gathered help in building upananonymizedlivedatarecordofthe
Gallery’svisitors. The “Debatorial” is, on the other hand, a fully digital platform intended
to facilitate exchange on several thematic areas via various formats. In addition to artist and curator talks,
the formats include podcast, chats, quizzes, zoom meetings, surveys, and animated maps, all of which aim to
promote visitors ́ participation and transparency.
The application ARt chat that we present in the following lines merges the advantages of AR—its
general accessibility and presence of content in real 3D space—with the qualities of participatory and
personalized visitor experience comparable with the ones exemplified in the two mentioned projects.
Description of the functions and added value
Exhibition visitors can download the application free of charge from Google Play Store¹³ or App
Store¹⁴ while in the museum or before the visit by following a link on the website. For using the
application, they have to log in or register with their Apple, Facebook or Google account, or their email
address and password. After that, a button “Start AR” appears for accessing a step-by-step guide.
After reading the instructions for use and closing the guide, a QR code placed next to selected artworks in the
exhibition can be scanned. After scanning the code and pointing their mobile device at the corresponding
artwork, a small figure in the top corner is indicating when they can start entering their comments (Figure 1).
Those can be placed anywhere in the 3D space by moving in the room and using drag-and-drop function on the
screen (Figure 2, 3).
If one wants her/his comment to no longer be publicly visible, it can be deleted via the trash icon. Visitors
can also react to comments of other users by clicking on them and typing an answer. The reply appears in the
application and is visible for all other users immediately after sending. Inappropriate comments by other users
can also be reported and deleted after reviewing if necessary. For scanning a new artwork, users just have to
click on the QR code symbol. By going back to the main menu, users can find the FAQs for more information about
the application and their account. On this page, also, a button leading to the already scanned artworks can be
found, including the related user comments. Via the “my comments” button, users can see their own
comments as well as the reactions of other users. Those two functions can also be used at home, after visiting
the exhibition (Figure 4).
The conditions necessary for a cultural institution to become a place for participation, which were outlined by
Nina Simon, are a good summary for some of the main driving forces in our project and can be recognized in the
function of the described application. These include: (1) desire for the input and involvement of outside
participants, (2) trust in participants’ abilities and (3) responsiveness to participants’ actions
and contributions.¹⁵
Unlike elements that have already been implemented in the museum, visitors can choose to use or not to use the
application without having information withheld. Art chat is intended to stimulate discussion among
visitors.
Since the content is freely configurable, the target group ranges from interested first-time visitors to
professionals. During and after a museum visit, there is the possibility to interact with the exhibition and
lead discussions with other visitors. In addition, one has a sustainable storage of museum visits and
one’s own interactions, which can be retrieved long after the actual museum visit. As nicely put by Carey
Jewitt in his essay on digital technologies in the museums—the hand held devices enable possibilities for
innovative interpretation, especially in regard to decreasing curatorial control and allowing audience
narratives “to override the search for a single authoritative voice.”¹⁶
Also, the museum can collect, answer, and evaluate questions and comments, whereby only anonymized data
regarding interaction with artworks and their comments are stored and evaluated.This data can be helpful, for
example, in planning further exhibitions or optimizing a permanent exhibition. In the longer term, the
application can be used and adapted for several museums and exhibitions. This has an advantage for the users who
only need one application, but also for the museums, because they do not have to develop their own
museum-specific application.
Finally, the application can also be used by the curators and artists themselves, enabling them not only to
react and answer to visitors’ questions and comments, but also to participate in the exhibition in a very
personal and immediate way, pulling down the boundary wall between the official and authoritative museum
discourse and the individual perspectives.
Background Information
The project was initiated as part of the partnership between nextmuseum.io and the MIREVI Lab of the
Düsseldorf University of Applied Sciences (HSD), where students of media informatics and communication
design took on the challenge of developing three different prototypes for interactive museum labels. After their
successful development, the prototype of the ARt chat application was selected to be further developed
with the HSD ́s MIREVI Lab and tested in the partner museums (Figure 5).
The exhibition “Kunstreichgewächse” in Museum Ulm (September 2021) served as the first test
site for the application. The prototype version only worked on a local smartphone or tablet with a demo version
of the application that the visitors could get in the museum. By clicking on the AR button, the camera of the
respective device opened to scan the QR code of an artwork. After scanning the QR code, visitors could write a
comment and place it as a 3D element in AR using drag-and-drop function. The chat between different
visitors was not yet possible at this stage. User tests and feedback helped in detection of the remaining bugs
which were then fixed by the tech team.
In the advanced test version, the application continued to run only on locally installed tablets that could be
borrowed from the museum. Here, the comments of other visitors who had already visited the exhibition or were
currently in the museum have been displayed and updated in real time, and one could also reply to them.
It was possible to create a personal account that visitors can reactivate for each new visit. However, it was
not yet possible to activate and manage the account from home. Finally, a handover to external partners for
completion and release in the Google Play and App Store took place, making ARt chat accessible on users
́ own mobile devices in the exhibitions “Subversives Design” in NRW-Forum Düsseldorf
(2022) and “Wir müssen reden!: Die Münster-Krippe im Meinungsstreit” in Museum Ulm
(2022). The final version is available for download in both stores and can be used on all AR-enabled devices.
User accounts can be registered automatically via an existing Apple, Google or Facebook account. After visiting
an exhibition and commenting on the artwork, visitors can continue the conversation from home and receive
information about the chat activities they are involved in.
Technical Implementation
The application is based on several components: the Mapping App Immersal¹⁷ was used to scan the
artworks and to link it to the database and the corresponding QR code. The scanning process that is enabled by
the Immersal App consists of taking several dozen photos of the artwork. Unity Immersal Software Development Kit
is required to download scanned artworks (maps) from the Immersal online database via ID and needs to be
integrated into the project. MongoDB database¹⁸ was used to store data about the exhibition, the
artworks, the users and the chat. Auth0¹⁹ was used to handle all logins via email or social media.
Heroku ²⁰ was used to run the Representational State Transfer Application Programming Interface
(REST-API) and Management App. Unity²¹ served as a development environment for this repository and was
used to compile the application. Following the open-source approach, the code as well as the technical
documentation is made available on GitHub for other interested museums to use the application in their own
exhibitions https://github.com/nextmuseum/unity-artchat.
Visitors ́ feedback and future directions
Starting with the idea of ARt chat as a tool for visitors to create their own version of museum
labels, we offered a writer's workshop every few weeks in the exhibition “Subversives Design” in
NRW-Forum Düsseldorf (2022) to provide space for creating texts as well as technical support when needed.
It became obvious that visitors preferred to use short form and entertaining comments to interact through the
application, similar to social media, rather than elaborate texts. So, we changed the outreach format by having
a staff member walk through the exhibition, making people aware of the application, and providing support when
needed. At the exhibition “Wir müssen reden!: Die Münster-Krippe im Meinungsstreit” in
Museum Ulm (2022) guided tours were offered, where visitors could learn something about the exhibition and
instantly share their thoughts via the application. Overall, visitors enjoyed using the application and had fun
commenting on the artworks and interacting with other visitors as well as museum staff in the exhibition and at
home. In a few minor parts, we could observe difficulties in the UX section, which are to be fixed in the next
step. We are also planning to implement a swear filter, so that inappropriate content cannot be uploaded in the
first place. There will also be some graphical customization and we will offer a selection of emojis to react to
comments, which should make the application even more fun and intuitive to use.
In 2023, the extended version of the application was used in the exhibition “Protest! gestalten” in
Museum Ulm. As mentioned, the documentation of the application has been published on GitHub and a several other
museums have already expressed interest in using it in-house.
Conclusion
The ARt chat application exemplifies an innovative approach in using AR technology in museum context,
focusing on active visitors’ participation and experience sharing that extends the usual spatial and
temporal givens determined by the institutional framework. In that sense, it provides a digital symbiosis of the
real exhibition scene set in front of the visitor, with the imaginary one which unfolds in their mind ́s
eye. The technological possibilities are here purposefully capturing the feelings, opinions and questionings of
the visitors, creating a moving sculpture of the exhibition atmosphere and the response it provokes.
Moreover, the application is enabling an uninterrupted museum experience, encouraging discussion and
participation even after the actual museum visit. With its open-source character, it presents an extension of
the nextmuseum.io platform dedicated to the swarm curation that aims at a wide community outreach and embraces
its further use and development through museum and visitor communities.
Acknowledgments
This project was funded by the Digital Culture Programme of the German Federal Cultural Foundation as well as
by the Federal Government Commissioner for Culture and Media. The credits for concept and prototype development
belong to the HSD students Miriam Jost, Michael Michel and Kim Tischer. Further development was done by the
members of HSD ́s MIREVI Lab Fabian Büntig and Sven Heinen. The optimization and release of the
application was taken over by vobe.digital.
References
1 P Vergo, The New Museology, London : Reaktion Books, 1989.
2 Anamaria Tomiuc, "Navigating Culture. Enhancing Visitor Museum Experience through Mobile
Technologies. From Smartphone to Google Glass,“ Journal of Media Research, 7:3 (20), 2014.
3ICOM approves a new museum definition, 24.08.2022,
https://icom.museum/en/news/icom-approves-a-new-museum-definition/, Accessed November 16, 2022.
4 Loïc Tallon, Kevin Walker, eds., Digital Technologies and the Museum Experience: Handheld
Guides and Other Media, Lanham, MD: AltaMira Press, 2008.
5 Peter Samis, "The Exploded Museum,“ In: Digital Technologies and the Museum
Experience: Handheld Guides and Other Media, Ed. Tallon, Loïc Kevin Walker, Lanham, MD: AltaMira
Press, 2008.
6 Charlotte Coates, "How Museums are using Augmented Reality,“ Museum Next, 17.04.2022,
https://www.museumnext.com/article/how-museums-are-using-augmented-reality/, Accessed November 21, 2022.
7 Shelley Mannion, “British Museum-Augmented Reality: Beyond the Hype," Museum-iD, 2017,
https://museum-id.com/augmented-reality-museums-beyond-hype-shelley-mannion/. Accessed November 21, 2022.
8 Paul Milgram, Fumio Kishino, “A Taxonomy of Mixed Reality Visual Displays,”
IEICE Transactions on Information and Systems, 1994, 77, 1321–1329.
9 X Qiao, P Ren, S Dustdar, L Liu, H Ma, J Chen. "Web AR: A promising future for mobile augmented
reality—State of the art, challenges, and insights,” Proceedings of the IEEE, 107(4), p.651-666,
doi: 10.1109/JPROC.2019.2895105.
10 Manifest.AR. https://manifest-ar.art/, Accessed November 21, 2022.
11 The Cell. https://isodesign.co.uk/projects/information-age-gallery-interactive, Accessed
November 25, 2022.
12 Debatorial. https://debatorials.zeppelin-museum.de/beyondstates, Accessed November 25, 2022.
13 https://play.google.com/store/apps/details?id=de.nextmuse um.io.ARtchat&pli=1, Accessed
November 19, 2022
14 https://apps.apple.comm/in/app/artchat/id1594639117, Accessed November 19, 2022
15 Nina Simon, The Participatory Museum, Santa Cruz, CA: Museum 2.0, 2010.
A hybrid listening to atmospheric processes
JuanC.DuarteRegino
Aalto University Espoo, Finland Juan.duarte@aalto.fi
Abstract
This paper delves into main aspects that support my environmental sound artistic practice, centered around
atmospheric processes. It leads to elaborate on the media installation Augury, which surrounds the
concept of hybrid listening to atmospheric phenomena. The proposed approach on hybrid listening
investigates the merge of mediated, embodied, and situated knowledge, to assist how we sense our atmosphere
beyond our human capacity, by translating environmental signals into audible and immersive perception.
Hybrid listening is informed by past meteorological insights, including ancient to modern weather
science, deepening into a profound temporal perspective. In consonance to this conference’s theme inquiry
about symbiotic imaginary, hybrid listening melds selected meteorological narratives that
lead human sensibility to interplay with more-than-human dimensions: living organisms and technical systems. By
means of interactive instruments, this research brings this symbiotic ensemble into an immersive sound encounter
with the atmospheric milieu. These instruments harmonize with the convoluted ecological dynamics of the
atmosphere—an intricate interplay of matter and agency.
Keywords
hybrid listening, atmospheric attunement, deep time, weather divination, ubiquitous computing.
In recent years, my artistic research has experimented in the realm of environmental sound, with a specific
focus on the intricate energies and processes of the atmosphere, notably the wind and the electromagnetic
spectrum.² This interest appeared from my exploration of generative and indeterministic methods in sound
creation, leading to an exploration dedicated to capturing and making audible pervasive aspects of atmospheric
phenomena. To transmute these intangible energies into auditory experiences, I harnessed the affordances of
ubiquitous computation, employing networks of on-site electronic weather sensors driven by
microcontrollers.³ These experimental incursions merged into a series that contemplates the fusion of the
sonic ecologies with technology, unveiling augmented perceptions of surrounding processes catalyzed by the
concealed forces of the atmosphere.⁴
Through crafting sensory media devices, it was possible to translate weather data into sound, harnessing
environmental sensors to extract real-time insights from the movements, forces, and air quality of the wind.
These data streams were converted from electrical impulses into audible compositions, effectively sonifying
their subtle transition and tangible essence.⁵ However, while this methodology has become somewhat
customary in contemporary media art,⁶ capturing atmospheric phenomena through sensing traces back to an
extensive history preceding the advent of computation and electronic media.⁷ This historical backdrop
encompasses a tapestry interwoven with artistic, scientific, technological, and mythological threads,
constituting a fertile ground for fostering a symbiotic imaginative space placed around atmospheric processes.
Historian of science, Liba Taub makes a comprehensive study of ancient meteorology, noticing register
techniques to decode repeatable and predictable phenomena in the weather, unraveling correlations within
atmospheric events.⁸ Laub underlines how ancestral weather wisdom was dedicated to elucidating immediate
dynamics and prognosticating future conditions. Her study promotes that inscription tools and methodologies
proved crucial to clarify the spatial-temporal dynamics of weather, enabling registration and comparison of
sequential events. Simultaneously, predictive strategies encompassed the observation of animal behavior, linking
with planetary cycles that persist in today’s atmospheric knowledge.
Conceiving the atmosphere as a medium facilitates a holistic comprehension of the intricate web connecting
sensing and perception within its realm. This perspective elucidates the nuances between environment and
observer, sensation and cognition, and consciousness and perception. By echoing existing contributions
surrounding our atmospheric milieu and its role as a communicative medium,⁹ my exploration centers into
the realm of weather sensing media, highlighting an environmental awareness of the reciprocal influences between
organisms and their surroundings. Central to this investigation is the notion of atmospheric
attunement, a concept that echoes our complex experience with our environments. This notion accentuates
the parallels between biological entities and technical systems in adapting and responding to processual
atmospheric dynamic.
My proposed notion of hybrid listening embarks on a quest to forge an alternative techno-embodiment of
atmospheric processes. Rooted in a symbiotic potential of combining organic and artificial dimensions, this
endeavor aims to establish an attuned relationship with the atmosphere. The envisioned system harmonizes the
capabilities of technical frameworks with the situated and embodied dimensions intrinsic to the extensive
temporal canvas of Meteorology. Drawing inspiration from archaic sources, these paradigms shape the contours of
ubiquitous and embodied computational systems, offering a novel perspective on our engagement with atmospheric
phenomena.
Atmospheric Soundscapes
Previously in my career (2012 - 2016) I experimented with creating soundscapes employing kites as instruments
to amplify the wind's forces during this activity. Adopting a Sonic Interaction Design approach, I delved into
an intricate interplay between the kite's motion and sound, grasping a dynamic equilibrium between human control
and environmental agency.¹⁰ Through the fusion of sonic digital media with existing tools like kites,
a realm of augmented and novel agency emerged, fostering interaction with the encompassing environment.
These initial experiments spurred me to delve into a fragmented history of weather measurement instruments,
unearthing a rich tapestry of artistic scientific endeavors. This inquiry allowed me to delve into a broader
spectrum of experiments where human and technological senses converge in the environment. Among these historical
figures, Athanasius Kircher, a polymath and instrument inventor, stood out. His creation, the Aeolian
Harp,¹¹ seamlessly blended musical instrument craftsmanship with an animistic ethos, resonating with
meteorological exploration by granting voice to the surrounding atmosphere.¹²
These ancestral artifacts possessed the capacity to translate environmental dynamics, offering insights into
the flow of wind through the medium of musical sensitivity and artisanal skill ¹³. Among other
inventions of this kind, these devices could convince society by demonstrating meteorological knowledge through
artistic expression. Besides, this perspective is useful to trace contemporary practices that meet at the
intersection of art, science, and technology.
Notably, these ancient manifestations harbor mythological and speculative concepts that connect with
representations and myths. One could argue that the aeolian harp confers a more-than-human “voice”
to air and wind dynamics. These artifacts resonate with the atmospheric medium and become perceivable for
humans. By building upon these insights, my project entitled Aeolian
Artefacts¹⁴ generates immersive soundscapes from wind energy and that is transmitted into
the “ether.” This device sources wind detection through microphones across four points, generating
soundscapes that mirror the wind's intricacies at each sensing point. The resultant auditory output is
transmitted via FM radio, speculating artistically on potential correlations between wind and electromagnetic
energies.
The Deep Time of Sensing the Atmosphere
Meteorology was originated in the context of the Ancient Greece, back when it was thought that weather was
influenced by the astral events occurring outside our planet, such as comets, visible from our planet.⁸
The root term meteor has a meaning connected to time-related phenomena, thus reading the sky has been associated
for a long time with the sequence of time.¹⁵
Taking a deep-time perspective on sensing and comprehending the atmosphere involves an examination of tools,
methods, and artifacts. Rather than isolating them by their technological or embodied nature, my proposal
advocates for their amalgamation. This integration encompasses not only the physical aspects but also the
embedded meanings and narratives inherited from ancient meteorological myths, interwoven with contemporary
scientific revelations regarding atmospheric phenomena. This approach to a deep time of sensing the atmosphere
informs my artistic practice to give a context for designing a sound instrument with certain affordances valued
in the history of these tools, methods, and artifacts.
Historian and media theorist Maximilan Hepach highlights an intriguing parallel between mythology and
phenomenology in their capacity to conceptually encapsulate the fleeting yet enduring temporal intricacies of
atmospheric climate and weather.¹⁶ Both historical and modern perceptions of climate share a
celestial genesis—a vantage point akin to that of a divine entity observing Earth from the cosmic expanse.
Aligning with Platonic reasoning, Hepach consider that these mythological frameworks don't stand in opposition
to rational discourse; instead, they play a role in altering our perceptual landscape, unfurling novel
interpretations and avenues of experience.
Ancient Meteorology
Exploring the Aeolian Harp's role as a weather instrument prompts an investigation into analogous historical
manifestations.⁶ The legacy of the Greek-Roman School of Meteorology⁸ sprouted from philosophical
inquiry aimed at unraveling the intricacies of weather and eventually evolved into predictive practices. Hesiod,
the poet active around 750-650 BC, inaugurated Astro-meteorology, interweaving astral events with weather
predictions. Ptolemy's approach to Astro-meteorology hinged on the interplay of stellar risings and settings,
attributing his prognostications to cosmic harmony.⁸
Among the commonly utilized ancient weather instruments was the Parapegmata—a prevalent fixture in
Greek-Roman cities.⁸ This inscription-based medium resembled a calendar but was tailored for weather
forecasting. By sequencing and combining events, it provided estimations of forthcoming weather, wind patterns,
bird sightings, and river levels. This holistic approach embraced cues from natural phenomena and animal
behavior as indicators of impending weather conditions.
Aristotle's meteorological treatise marked a watershed moment, describing weather through observations of the
four elements and their interactions—earth, water, and wind—attributing phenomena like tornadoes and
storms to their interplay. This marked the advent of weather observation and measurement methodologies. Early
precision instruments, such as the hygrometer, were employed to gauge humidity levels in soil and air. The
device compared the density of a charcoal sheet exposed to the air with an unexposed one.
Mythological narratives in Homer personified the winds, aligning them with cardinal points. This led to visual
representations, such as the wind rose and anemographic chart. Jansson's 17th-century wind chart, a colonial
navigation tool, personified thirty-two wind directions, associated with racial traits from their
regions.¹⁷ It featured nomenclature in six languages, including Greek, Latin, and Dutch. Along these
lines, Alexander Humboldt's isothermal chart guided settlers to regions with European-like weather, aiding
colonization in temperate or cold environments.¹⁸
Atmospheric-related divination
Aeromancy, an ancient art, entails predicting broader life-affecting occurrences through the observation of
clouds, wind patterns, and changes in animal behavior. Augury and Ornithomancy, divination practices rooted in
interpreting bird behavior, further enrich this tradition. Augury, foundational to ancient Rome,¹⁹
offers a unique approach to engaging with the atmosphere's imperceptible dimensions.
Recent scientific revelations align with these divinatory practices. Many migratory birds possess
magneto-reception, primarily located in their beaks where magnetite particles bestow a sense of geographic
orientation.²⁰ Additionally, their eyes harbor crypto-chromes capable of detecting planetary magnetic
fields.²¹ Their infrasonic sensitivity enables them to anticipate storms and extreme weather
conditions.²²
Another case of divinatory practices which pertains to my artistic research is the case of Scrying, which
proceeds from the Aztec culture and the use of obsidian artifacts, based on the myth of the smoking mirror
“Tezcatlipoca.” Aztecs had rituals of divination to invoke the agency of the lord of the
smoking mirror to enable them to see beyond what was imperceptible by human senses.²³
Art and Science scholar Hannah Star Rogers explores technologies as modern divination tools. She posits that
these can assign meaning by directing attention to the otherwise imperceptible aspects of the natural world,
expanding our interpretation of phenomena beyond our regular senses. Rogers emphasizes the synergy of embodied
and technology-driven atmospheric sensing, bridging the ancient art of divination with modern technology.
Conceiving weather events through this lens involves an empirical analysis across diverse perceptual phenomena,
discerning patterns from elements like oxygen composition, orientation, temperature, and
electromagnetism.²⁴
Viewing atmospheric sensing through a divinatory perspective encourages contemplation of the environment to
seek answers. Can uniting ancient and contemporary worldviews offer glimpses into the future or unveil the
unseen facets of our current reality?
Modern Meteorology
Conceiving atmospheric processes within an interconnected framework emerged as a paradigm in the early 20th
century, giving rise to contemporary meteorology. This shift marked a departure from earlier empirical methods
and ushered in the foundation of weather science.²⁵ Meteorologists of that era focused on surface
conditions, employing sparse weather stations interconnected through telegraph lines, supplemented by limited
upper air measurements conducted by Aerologists utilizing kites and balloons.²⁶ The drive for robust
predictive science was contingent on key advancements, including wireless telegraphy and the advent of weather
modeling and statistical analysis.²⁷
Analogous to an ocean of oxygen, the atmosphere possesses a reservoir of energy dwarfed by that contained in
oceans, contributing to its relatively mild temperature fluctuations. The ceaseless circulation of winds and
currents, both large and small, occurs in perpetual vortices termed solenoids, striving for equilibrium yet
never fully attaining it. Toward the end of the 19th century, statistical aggregation gained prominence, coupled
with the establishment of climatology as a field of practice.²⁷ Vilhelm Bjerknes, a Norwegian
scientist, catalyzed a transformation by synthesizing empirical meteorology with physical laws in the late 19th
century. He devised a method anchored in the principles of thermodynamics for weather prediction. His
prognostications hinged on calculating seven parameters: velocity across three dimensions, pressure, density,
temperature, and humidity. Bjerknes' background in electrodynamics and wireless telegraphy conferred insights
into studying hydrodynamic waves coursing through the real fluid atmosphere. Such methodologies aimed not to
supplant intuition but to amplify it. Bjerknes' pioneering efforts culminated in the inception of the Bergen
School, pioneering scientific weather analysis and forecasting. This school revolved around meteorological
observations, data assessment tools, and models elucidating atmospheric structures.²⁵
Computer-Based Atmospheric Predictions
Despite the integration of telegraphic networks across various weather stations, empirical weather prediction
remained inherently imprecise. Consequently, the shift towards computational weather models and numerical
forecasting emerged as the novel cornerstones in meteorology.¹⁵ This transformative juncture
propelled Meteorology into the vanguard of civilian computing utilization, paralleled only by nuclear science
endeavors. The inception of Numerical Weather Prediction coincided with the nascent era of computing, led by
John Von Neumann. His vision recognized the convergences between nuclear science and the intricate nonlinear
physics governing weather phenomena.²⁷
This synergy led to a crucial moment when the ENIAC computer was deployed to test weather forecasting utilizing
models developed by the Bergen School. By connecting disparate data points from various weather parameters, this
approach sought to forecast weather outcomes. Midway through the 20th century, global implementation of weather
forecasting via the ENIAC proved successful, yet refining computer models to approximate human forecasting
capabilities required several subsequent decades.²⁷
Atmospheric synthetic cognition
The evolution of remote sensing in Meteorology emerged as a standardized practice, driven by the need for
computational environmental awareness and the monitoring of atmospheric dynamics. This paradigm shift ushered in
an era where sensing networks and computing models replaced human intervention, particularly within the realms
of satellite and radio technologies. This transition to sensor-computing diminished the influence of human bias
in information acquisition, reaching beyond the limits of human perception.
Postmodern literary critic Katherine Hayles delves into the hybrid convergence of organic and artificial
systems attuned to the atmosphere, an alternative mode of interaction with atmospheric processes comes to light
through the notion of Cognitive Assemblages.²⁸ According to literature scholar Mark B. N. Hansen,
these kinds of assemblages, founded upon ubiquitous computing and environmental computation, encompass technical
systems like smart dust, characterized by a micro-temporality of sensing and engendering sensations that precede
conscious human cognition [Hansen]. While devoid of consciousness, these cognitive systems enable a direct and
immediate connection to the sensory world, enabling the synthesis of peripheral information or even facilitating
novel forms of experiential cognition that transcend conscious-related representation and memory. This technical
attunement empowers humans to immerse themselves fully in the act of sensing, prioritizing sensory engagement
over the perception of atmospheric phenomena.²⁹
In this framework, ubiquitous computing manifests as cognitive assemblages through networks of sensors,
receptors, actuators, and their programmed cognitive processes. This combination engenders a procedurally
distributed approach to atmospheric sensing across expansive territories, seamlessly merging "natural"
environments with the synthetic cognition furnished by computers. Jennifer Gabrys looks at how these novel
sensor technologies are primed for atmospheric operation; for instance, gauging pollution levels and generating
reports that inform public policies.⁹ In this context, the atmosphere transforms from a foundational
backdrop into a dynamic context that shapes and is shaped by sensor technologies, akin to the integration of the
aeolian harp's music into the natural soundscape.
For technical systems, atmospheric sensing transpires primarily within the physical medium, devoid of conscious
awareness. Sensibility precedes perception, embracing direct interaction with the sensory realm itself,
transcending specific objects accessed by consciousness [29]. Ubiquitous computing and pervasive sensing systems
facilitate the catalysis of sensation, operating on a micro-temporal scale that transcends human perception and
adheres to non-biological technical protocols.²⁹ This synthetic framework engenders a hybrid
attunement to atmospheric processes, synthesizing information in a peripheral manner and fostering novel
cognitive experiences unburdened by conscious-related representation and memory of atmospheric encounters.
Along these artificial sensing and cognitive processes, Katherine Hayles's concept of Nonconscious cognition is
useful to elucidate the nuances of synthetic atmospheric sensing, encapsulating emergent effects, fluid
mutations, and transformations. While remaining beyond human perception, these technical systems infuse our
atmosphere with meaning, utilizing somatic markers such as chemical or electrical signals coherent to their
operations. This interaction harnesses the cognitive potential of computational media, adapting to the dynamic
shifts of an environment like our atmosphere.²⁸
Embodied Atmospheric Attunement
Media theorist Desiree Förster underscores a kind of atmospheric sensing from an embodied perspective,
wherein atmospheric processes, though invisible, envelops us in a multi-sensory embrace, engaging our senses
through various channels such as touch, scent, sound, and respiration. While the immediate perception of
atmospheric shifts might elude us, their profound impact on our environment remains undeniable ³⁰.
Atmosphere(s) intertwine with our sensory experiences, reciprocally rendering the perceiver and their act of
sensing palpably present. This way, contemplating atmospheres as environments for a diverse array of organisms
underscores their capacity to abruptly recalibrate awareness, thrusting themselves from the backdrop to the
forefront of our attention.³⁰
Navigating the realm of an ever-transforming atmosphere necessitates to become familiar with its dynamic
essence, thus fostering the need for attunement. Atmospheric attunement entails developing an acquaintance with
these atmospheres by discerning the effects they exert on our physical and cognitive faculties. This process of
attunement involves aligning with something approaching, yet not entirely present—an ephemeral unity with
the world, a connection that transcends comprehensive grasp. For instance, the Deep Listening methodologies
pioneered by composer Pauline Oliveros³¹ exemplify a form of attunement to acoustic reflections from
our surroundings, exemplifying a mode of engagement.
As such, atmospheric attunement encompasses an engagement with the intangible, an intimate negotiation with the
ever-shifting currents of our surroundings. It calls us to forge an implicit unity, an interaction that defies
conscious awareness and perception, yet binds us momentarily with the vast mosaic of atmospheric processes.
Atmospheric hybrid listening interface
Building upon the concepts outlined earlier, a visionary instrument emerges—a conduit for crafting
immersive soundscapes that echo the deep-time of atmospheric sensing and synthetic cognition. This innovative
instrument draws from diverse realms, uniting the threads of ancient practices, modern technology, and
hybrid listening.
The essence of this multichannel instrument lies in its capacity to synthesize soundscapes from the intricate
mosaic of atmospheric events and dynamics, seamlessly merging the temporal layers of deep-time sensing and
contemporary synthetic cognition. By crafting sensory instruments calibrated to discern atmospheric nuances, a
novel form of hybrid listening takes shape, an amalgamation of embodied and technical attunement that spans the
spectrum of wind and natural radio.
At its core, this instrument embodies a divinatory approach, akin to augury, where weather stations serve as
modern-day augurs. This convergence births a symphony that resonates with the rhythms of the atmosphere, a sonic
reflection of the flow of meteorological phenomena. The soundscape generated becomes a medium for extended
listening of the surrounding weather, transcending geographical boundaries as it carries the ambiance of distant
sensor stations to the ears of the listeners.
An intricate web of ubiquitous computing systems breathes life into this interface, orchestrating remote
operations and ushering in an array of atmospheric data streams. These streams, point out to predict the
possibilities of future events, guiding users into a profound attunement to atmospheric processes. This
interface paints a canvas of sound, bridging time and space, where the fusion of ancient and modern, tangible
and synthetic, coalesce to forge an auditory tapestry that harmonizes humanity with the intricate cadence of the
atmosphere.
Conclusions
In this paper, I have delved into the landscape of my artistic exploration and research centered around the art
of sensing the atmosphere to craft captivating soundscapes, underlining two milestone projects: Aeolian
Artefacts, and Augury. Drawing inspiration from the rich panorama of meteorological history,
I've traced the interplay between ancient mythologies and scientific advancements in meteorology, culminating in
the conceptualization of an ongoing developing instrument—a conduit for attuning to the intricate rhythms
of atmospheric processes.
This exploration underscores the symbiotic relationship between embodied engagement and synthetic
interfaces. The instrument's inception rests upon a foundation of hybrid connections, weaving together the
threads of mythology and scientific inquiry into an evolving work in progress. The bridge between
ancient wisdom and modern understanding of our atmosphere provides the underpinning for an instrument that fuses
human sensibility with technological agencies that extend our experience about the weather.
As this artistic research unfolds, the embodiment and synthetic attunement to atmospheric processes will be
explored further through the lens of interaction design. The forthcoming stages of development will navigate
uncharted territories, transforming concepts into tangible experiences. Through this dynamic interplay, the
aspiration is to create an instrument that serves as a gateway—a harmonious amalgamation of past and
present, mythology and science, ultimately offering a profound auditory connection to the ever-shifting
atmosphere.
Acknowledgements
Augury was produced at the RIXC Centre for New Media Culture with received funding for residential visits from
the Nordic-Baltic Mobility Programme for Culture to establish project “RIXC Art Science
Residencies”. The multichannel interface has received support from the Aalto Digital Creatives Art &
Tech Open Call ’23 to continue its development.
Author Biography
Juan C. Duarte Regino is a Mexican artist-researcher and current Ph.D. candidate at Aalto University in
Finland. His research revolves around the exploration of the symbiotic relationship between nature and
technology through environmental sound.
His artistic endeavors have gained recognition and have been showcased at prestigious events and venues such as
the ISEA, NIME, CTM Festival, Spiral Gallery, Pixelache, IAMAS, RIXC, Media Art Histories, Ujazdowski CCA,
Goethe Institut, ETH Zurich, and Medialab Matadero. [https://www.juanduarteregino.com]
References
1 Duarte Regino, J. C. Computing Atmospheric Attunement and Hybrid Listening through Augury and
Scrying", School of X in connection with XCoAx, 2023. https://classof23.xcoax.org/paper09.html.
2 Douglas Kahn, Earth Sound Earth Signal: Energies and Earth Magnitude in the Arts, University of
California Press, 2013.
3 Jennifer Gabrys, How to Do Things with Sensors, University of Minnesota Press, 2019.
4 C. Salter, Sensing Machines: How Sensors Shape Our Everyday Life, MIT Press, 2022.
5 T. Hermann, A. Hunt, Guest editors' introduction: An introduction to interactive sonification,
IEEE multimedia, 12(2), 2005, 20-24.
6 J. Randerson, Weather as medium: Toward a meteorological art, MIT Press, 2018.
7 Siegfried Zielinski, Deep Time of the Media, Cambridge, MA, MIT Press, 2008.
8 L. Taub, Ancient meteorology, Routledge, 2004
9 J. Gabrys, Programearth: Environmentalsensing technology and the making of a computational
planet (Vol. 49), U of Minnesota Press, 2016.
10 Regino Duarte, J. Pulsar Kite: a study case for Sonic Ludic Interaction, Master Thesis, Aalto
University, 2015.
11 J. Fletcher, Athanasius Kircher and his ‘Musurgia universalis’(1650). Musicology
Australia, 7(1), 1982, 73-83.
12 T. Morton, Ecology with out nature: Rethinking environmental aesthetics, Harvard University
Press, 2009.
13 B. M. Stafford, Artful science: Enlightenment entertainment and the eclipse of visual
education, MIT press, 1994.
15 J.D. Peters, The Marvelous Clouds: Toward A philosophy of elemental media, University of
Chicago Press, 2020.
16 M. G. Hepach, Ephemeral climates: Plato's geographic myths and the phenomenological nature
of climate and its changes, Journal of Historical Geography, 78, 2022, 139-148.
17 R. D. Black, [Wind] scapes: Engaging the shifting nature of the wind formed landscapes of
Denmark (Doctoral dissertation), 2016.
18 E. M. Conway, The Humboldt Current: Nineteenth-Century Exploration and the Roots of American
Environmentalism, 2008.
19 Lindsay G. Driediger-Murphy, Roman Republican Augury: Freedom and Control, Oxford University
Press, 2019.
20 R. Wiltschko, W. Wiltschko, Magnetoreception in birds, Journal of the Royal Society
Interface, 16(158), 2019, 20190295.
21 M. Albaqami, M. Hammad, M. Pooam, M. Procopio, M. Sameti, T. Ritz, C.F. Martino, Arabidopsis
cryptochrome is responsive to Radiofrequency (RF) electromagnetic fields, Scientific Reports, 10(1), 2020,
11260.
22 J. N. Zeyl, O. den Ouden, C. Köppl, J. Assink, J. Christensen‐Dalsgaard, S.C.
Patrick, S. Clusella‐Trullas, Infrasonic hearing in birds: a review of audiometry and hypothesized,
2020.
23 Silke Ackermann, Louise Devoy, The Lord Of The Smoking Mirror: Objects Associated with John Dee
in the British Museum, Studies in History and Philosophy of Science Part A, 2012, 43.3:539-549.
24 Hannah Rogers, Field_Notes: The Technology of Divination, Art Journal Open, 2019,
http://artjournal.collegeart.org/?p=11090.
25 J. R. Fleming, Inventing atmospheric science: Bjerknes, Rossby, Wexler, and the foundations of
modern meteorology, MIT Press, 2016.
26 R. J. Wille, Colonizing the Free Atmosphere. History of Meteorology, 8, 2017, 95-123.
27 P. N. Edwards, A vast machine: Computer models, climate data, and the politics of global
warming, Mit press, 2013.
28 N. K. Hayles, Unthought: The power of the cognitive nonconscious, University of Chicago
Press, 2017.
29 M. B. Hansen, Ubiquitous sensation: Toward an atmospheric, collective, and microtemporal model
of media, Throughout: Art and culture emerging with ubiquitous computing, 2013, 63-88.
30 D. Förster, Aesthetic experience of metabolic processes, meson press, 2021, 182.
31 P. Oliveros, Deep listening: A composer's sound practice, IUniverse, 2005.
Eleven Movements of the Cryoscape – Ecological Explorations in Sonification for Affectively Engaging with
Climate Change
LouiseFoo1 and
JonasFritsch2
1 Artist, 2IT University of Copenhagen, Copenhagen, Denmark
louifoo@gmail.com, frit@itu.dk
Abstract
In this paper we present the sound installation Eleven Movements of the Cryoscape which was created
for the new Kangiata Illorsua – Ilulissat Icefjord Centre in Greenland. The installation is a near
real-time sonification of the movement and melting of the Inland Ice, consisting of an array of eleven speakers,
each transmitting from a different location in Greenland. The installation portrays the Inland Ice as a living,
breathing, evolving organism in the age of the anthropocene, where humankind has made its mark on the very
changes that occur to the natural sounds over time. The installation invites people into an affectively engaging
and contemplative relation to our changing ecologies through sonification of data to reflect on our present
condition and to potentially imagine and connect to new realities. In this paper, we present the process leading
to the creation of the installation and how it adds to existing research into sonification and listening
practices in artistic and design research.
The Inland Ice plays a key role in the global climate change debate. Accelerating melting of the Inland Ice is
one of the problems we have been facing for a long time in a world, it seems, in perpetual ecological and
technoscientific
crisis.¹³,¹⁸,²¹,⁵⁷,⁵⁹
There is a paradox between the hopelessness and apathy many of us feel, and questions of how things could be
different, an urge for change and action (e.g., ²⁷). In the following paper, we present a
sound-driven artistic intervention that might contribute to altering affective attachments and mobilizing
climate action through listening and attuning to the geological movements of the Inland Ice.
Eleven Movements of the Cryoscape is a near real-time stream of sonification of seismic data and wind
data, from the GLISN network of seismic sensors ¹⁹ and PROMICE weather stations [⁴², see
Fig. 5 and 6], normally used for scientific monitoring of the Inland Ice. The installation is situated at the
Ilulissat Icefjord Centre, Greenland and consists of an array of eleven speakers, each transmitting from a
different location in Greenland. Sonifications of seismic signals—movements of waves that travel through
the earth and ice from the ice sheet, sea ice, ocean tides, ice streams, icebergs, calving fronts, and
glaciers—are transmitted, allowing visitors to listen beyond the range of the human ear. The soundscape
consists of ice, wind, waves, earthquakes, and human activity on the island from the past week. The installation
plays back a 15-minute loop that is updated every night at midnight. The wind speed at each of the locations
controls the volume of each seismic transmission to highlight how multiple environmental factors are
interconnected in relation to ice as a body of knowledge, towards identification of climate change. The
installation portrays the Inland Ice as a living, breathing, evolving organism in the age of the Anthropocene,
where human beings have made their sonic mark on the natural soundscapes. Today, human involvement can be heard
beyond what acoustic ecologists refer to as the anthrophony, i.e., the "human-made sounds."⁴⁰ We are
not just listening to ice with a touch of human presence. We are listening to ice as it is radically altered by
humankind.
In the installation, climate changes and the melting of the Inland Ice is expressed as an ever-changing
orchestration in what we rather than a soundscape have chosen to refer to as a "cryoscape" understood as an
ecocritical framework to emphasize the dynamic relationship between ice and people. In so doing, we follow the
work of Carey et al. who look at glaciers within a feminist framework, for global environmental change
research.³ Art historian Isabelle Gapp positions glaciology in an eco-critical art history.¹⁵
The term cryosphere, first introduced by Antoni Bolesław Dobrowolski to describe the frozen-water parts of
the Earth,⁹ has resulted in several subsequent typologies that draw upon ice and snow between culture and
environment. The idea and term cryoscape was first used by Marcus Nüsser and Ravi Baghel to
consider glaciers as more than physical landscapes and accounts for glaciers within epistemic, social, and
cultural practices.³⁶ Sverker Sörlin looks to cryo-history to denote the historical
role of humans in determining the fate of ice in the Anthropocene.¹⁰ Isabelle Gapp, furthermore,
points to how Elizabeth Leane suggests the term cryo-narratives as a reference point for broader and
more discursive studies surrounding the cultural history of ice³² and how Klaus Dodds and Sörlin
position such multidisciplinary discussions within the new field of ice
humanities.¹⁰ This positions ice as a core part of environmental
humanities—the study of how our ideas about the more-than-human world, aka “nature” in
literature, film, art, and other cultural practices shape how we relate to it materially and politically and how
ice gets experienced, represented and storied.
Methodologically, the project has been carried out as practice-based research⁵ and
research-creation⁶⁰ combining artistic practice with theoretical concepts to produce new knowledge.
The project extends ongoing inquiry into sound and sonification (e.g.,
²⁶, ³⁵, ⁴⁷, ⁴⁹, ⁵⁰, ⁵¹, ⁵⁴)
and engages with a range of data sonification/sonic methodologies¹, ¹¹,
⁶⁵ within a sonic materialist⁶¹, ⁶²,
⁶³ and affect
theoretical perspective³⁴, ¹⁴, ¹³. With
this installation, we would like to extend existing investigations into sonifications as negotiations between
aesthetics, politics of mapping and data representation (art, science & technology). We argue that data
sonification can lead to affectively engaging and sonically sensible spaces for listening to "mute," invisible
or inaudible ecologies, allowing us to "open our eyes" for a minute, by opening our ears—in which there
are infinite possibilities for transformative experiences and ways to connect, interpret, understand and
"invent" those ecologies. The article is outlined in the following way: We first position the work with the
installation in relation to existing work on data sonification and artistic engagements with climate-oriented
sonic installation design. We then present in-depth the work leading to the installation and the guiding
research questions that have been explored through the creative process. This leads to a presentation and
analysis of the installation in light of said questions and related to lines of inquiry at the intersection
between climate change, ecological data sonification, listening practices, art, science and digital technology.
Following this, we discuss the knowledge contribution of the work in and outline future explorations in the
transdisciplinary and transitional encounter between art, design, and technology towards rekindling affective
attachments for imagining a more livable future.
Sonifying Ecological Data
In this section we position our work in relation to existing literature and projects within scientific data
sonification, in particular seismic/glacial data sonification. In addition, we add perspectives on listening as
a form of affectively engaged ecological attunement to more-than human aspects of the Anthropocene in a time
characterized by climate change.
Data Sonification in Science, Music and Art
Sonification of data has most extensively been utilized for scientific purposes, but also, as a means for
musical composition⁵⁶, ⁶⁵ and as an expressive medium in art installations.
In the article Sonifying the World authors Kanngieser and Gibb point out how human ears are sometimes
better than our eyes at detecting subtle changes over time and that sonification—understood as the turning
of data into sounds—has emerged as an increasingly popular practice in recent years. In 2016 Scientists
succeeded in proving Einstein's general theory of relativity by listening to gravitational waves: “Finally
astronomy grew ears. We never had ears before.”²⁹ In musical sonifications composers will
typically map data to musical elements such as pitch, rhythm, and timbre. There are several examples of
compositions where climate data has been translated to music in one way or the other, to emphasize climatic
changes.e.g., ”The Climate Symphony and Other Sonifications of Ice Core, Radar, DNA, Seismic and Solar
Wind Data”—a one-hour performance/presentation of sonification research by Marty Quinn of Design
Rhythmics Sonification Research Lab ⁴⁹ Kanngieser and Gibb point out, how compared to the noisy
nature of ecological data, many categorized as musical sonifications, feel strikingly composed, with the use of
familiar western scales, “carefully curated for listening ears.”²⁵
Eleven Movements of the Cryoscape contributes to this transdisciplinary field through inquiry into the
use of near real-time streaming in sound art installation. The artist and writer, Andrea Polli, has contributed
notable works in sonification as sound installations exemplified in works such as her documentation of arctic
melt ⁴⁸. Rather than having to rely on data models and projections like Polli, today, due to rapid
technological developments that allow us to live-stream geo-data sent via satellite by the hour from various
glaciers around the Arctic, we can explore new artistic potentials and narratives of immediate, immersive, and
real-time character. An example of the use of near real-time seismic data in sonification is Herald/harbinger By
Ben Rubin and Jer Thorp—which is framed as “a long-distance conversation between a glacier and a
city.” The artwork monitors the rumblings, cracks and shifts of the moving ice and transmits seismic data
to control sound/lights at an installation at a building in Calgary.⁵⁵ Similarly, to this project,
Eleven Movements of the Cryoscape utilizes seismic data, but from several data sources rather than from one.
While the artist behind Eleven Movements of the Cryoscape initially set out to compose a soundscape like is the
case with Herald/Harbinger, through collaborative efforts and prototyping it became clear that the raw data
offered an "authentic" aesthetics emphasizing the sense of “tuning in.”⁶¹
Listening to Ice
Scientists are increasingly using sound as a crucial element in climate research. Dr. Evgeny Podolskiy,
professor at the Arctic Research Center at Hokkaido University in Sapporo, Japan states how “Glaciologists
just opened their eyes to studying glaciers about 150 years ago. We started to look at glaciers from different
angles, perspectives, satellites—but we forgot to open our ears.”⁴¹ In Geophysical
Research Letters, Podolsky writes about sounds he has recorded, not with expensive geophysical sensors, but with
a smartphone in Bowdoin Glacier (Kangerluarsuup Sermia), located in northwestern Greenland. His recordings
captured a unique sound which he used to describe a specific drainage process within the glacier—one that
is impossible to observe from the surface: Meltwater drainage through a crevasse. “I’ve been
studying glacier geophysics for quite some time and I found that there is this kind of natural zoo, or a
universe, of sounds which we kind of totally ignored until recently.”⁵²
Oceanographer Oscar Glowaki’s work with Cryoacoustics—the study of ice through noise involves
listening to the air bubbles from glacial ice.¹⁶ 1000- year-old ice does not just contain information
about snow, temperature, and pollution, but it also has an interesting sound. The bubbles contain 1000 years of
atmospheric air. When melting glacial ice in water, the bubbles explode due to the pressure of a series of small
micro-explosions: Each bubble sings (click-click-click) and that is how glaciers produce what Glowaki refers to
as the loudest sound in ocean acoustics. He points out how satellites from space can be used to track the
melting of glaciers or icebergs. But what is much harder to get data from is that spot where glaciers meet ocean
water—and that ice-ocean boundary is where a big part of the melting occurs.⁸
In geophysics, sonification has been used to understand long-range and large-scale seismic events, such as
earthquakes and volcanoes ⁴⁴ and is increasingly utilized in the study of glacial data. Throughout
our research we have come across audio examples of seismic data converted to sound and found it fascinating,
considering how seismic data is very similar to sound—vibrations of waves that propagate through the
earth, rather than air. And if you speed up the data, you can listen to it. It turned out that raw seismic data,
already converted to sound, is available from an extensive network of sensors monitoring Greenland through the
GLISN Project in near real time. The main goal of the project is to “provide real-time data from sensors
in and around Greenland for detecting, locating and characterizing glacial earthquakes and other cryo-seismic
phenomena, and to contribute to our understanding of Ice Sheet dynamics.”¹⁹ Scientists do in
this regard look and listen for very specific glacial events, and will exclude data that in the ears of science
is "irrelevant," but perhaps this data could be relevant to listening ears within artistic contexts? Hence, we
became curious to listen to the sonic phenomena and the going on, in addition to what was flagged as earthquakes
and the 3-4 daily major glacial events (icebergs calving and ice quakes), and we began listening to ongoing
constant streams of seismic data from the GLISN network.
Affectively Engaging with sonified Ecological Data
Eleven Movements of the Cryoscape presents us with an opportunity to listen to otherwise mute
ecologies, by simply "tuning in" on them; making those ecologies perform on their own. It asks if we, through
aesthetic and affective experiences and through sonic sensibilities of listening⁶¹ to "mute,"
invisible or inaudible ecologies can ‘open our eyes’ for a minute, by opening our ears. Salomé
Voegelin, an artist, researcher, and writer engaged in listening as a political practice, emphasizes the role
art and sonic practices play in asking relevant and necessary questions on today’s global challenges such
as climate change. Voegelin articulates how there are sonic potentialities, sensibilities, and
competencies⁶¹, ⁶², ⁶³ yet to be explored. In
her most recent volume “Fragments of Listening: Political Possibilities of sound”⁶³ she
emphasizes how there is a need for explorations in interdisciplinary domains, in a time where “ecological
questions need answers from unknown places”⁶³ Voegelin describes listening as an interactivity
that challenges, augments, and expands what we see, without presenting an additional illusion to the scene, but
by producing the reality of lived experience as invented spaces, not unreal, but phantasmagoric. Listening does
in this sense not enhance those environments—it produces them. Voegelin, furthermore, articulates how
“duration” and “real-time” qualities produce “not a sense of listening to, but
listening in.” The “listening in” and “sense of real- time" produce
“authenticity.”⁶ Kanngieser & Gibb point out how sound influences the way we know the
world; and whether intended or not affect and emotion intervene when sound is used in novel ways such as
sonification.²⁵ Affect is understood as an in-between, pre-personal, relational, and more-than-human
concept whose intensity ultimately colours our engagement with ourselves, each other, and the
world.¹⁴, ³⁴ Fritsch has emphasized the role of affect in situations of
catastrophe and crisis “at the end of the world,” advocating for an exploration of the role of
digital technologies to create positive changes in affective attachments on an ecological scale.¹³ For
this relational change to take place, there needs to be a form of affective engagement¹² occurring at
an infra-individual level between human and non-human bodies—of flesh, of ice, of sound. Affective
engagement in the case of Eleven Movements in the Cryoscape thus describes the potential for a really
felt impact that might cause people to act or feel differently by being “taken into” the sonified
movement of the ice.
Creating Eleven Movements of the Cryoscape
In this section we will present the process of inquiry leading to the construction of the final installation of
Eleven Movements of the Cryoscape. Our main intention is to both foreground the creative and artistic process as
a form of listening, while at the same time foregrounding the forms of listening emerging for a broad range of
stakeholders during the process. Doing so, we will contribute with insights concerning the (listening) work
required in processes concerned with artistic explorations of scientific, ecological data to create an
affectively engaging sonification that can open a sensory and contemplative space for attuning to the inland ice
in the light of climate change in the Anthropocene.
Context of the Collaboration
In the summer of 2021, Ilulissat Icefjord Centre, designed by Dorthe Mandrup, opened by the icefjord in
Ilulissat in Greenland. The Icefjord Centre is run as a self-governing institution, and is built by a
partnership consisting of the Greenland Self-Government, Avannaata Kommunia and the philanthropic foundation
Realdania who in the following quote state one of the projects key missions: “With its unique location on
the front line of climate change, the new Kangiata Illorsua – Ilulissat Icefjord Centre will play an
important role in conveying the effect of climate change on the ice sheet.” (Jesper Nygård, CEO of
Realdania). JAC Studios was responsible for exhibition design and curation of the exhibition, which disseminates
knowledge about Greenland's geological, biological, cultural, and climatic history. The exhibition consists of a
permanent exhibition with cinema, central exhibition space, and lastly a separate exhibition space with the
working title “Future Lab." For this part of the installation, artist Louise Foo and photographer Anna
Domnick were invited to each create a new artistic work based on scientific climate research. Several meetings
were held between JAC studios and the artists in which there were brainstorming and sharing ideas, knowledge,
and research. A workbook was developed through these sessions, with the following incentives: To give a 1:1
experience of climate. For audiences to experience the inland ice through the senses, sound should be the
emphasis, an immersive experience that shows the “unseen” parts of the icefjord, reflecting the
spectrum of ice through an instrument—possibly using real-time ice recordings to give live feedback of the
current condition and research on the inland ice in an installation that focuses on reflection—reflecting
on the place of your own body, your influence on the Icefjord, the influence of the icefjord on you (to reflect
on how the Greenland ice sheet influences the global condition) JAC also initiated a workshop / meeting between
the artists and scientists from Niels Bohr Institute ³⁷ and GEUS (geological survey of Denmark and
Greenland).¹⁷
Listening & inquiring: from field recordings to sonic fields
To begin with, the process of inquiry took its point of departure in focusing on surface melt and glaciers
calving, which are some of the major contributors to ice sheet mass loss. Initially, the artist set out to build
an archive of recordings of melting ice & recordings of calving glaciers to facilitate the soundscape. The
artist was inspired by Jakob Kierkegaards sound installation Isfald in which he uses equipment that
gives us opportunities to listen to sounds that we usually cannot hear; underwater with hydrophones or surface
transducers to record vibrations of the earth [30] and was planning to do her own sound recordings in Greenland
in the summer 2020 while also exploring options of collecting sound from other artists and scientists. The
artist navigated sonic phenomena of ice of the arctic through readings, listening to sound files and through
contacting a handful of experts (J. P. Steffensen, Glaciologist Andreas Ahlstrøm, Seismologist Peter
Voss, Oceanographer Oscar Glowaki, Sound recordist Simon Forrester) who hold recordings of ice and glaciers from
across Greenland - both from above and below the water. Obviously, the natural soundscape of ice will be quite
different depending on where in Greenland you place your ear or microphone. If a person is standing in the
middle of the Inland Ice for example, there is almost no sound, except for the occasional sonic phenomena of
‘firn-stoss’—a collapse of snow (about 5 cm) that happens due to changes in temperature, that
can spread several miles, sounding like a rumbling thunder. The artist was fascinated with the complex palettes
of sounds of ice and the qualities inherent in movements caused by temperature changes and the sonic states of
frozen to liquid in various scales, from microscopic ice crystals crackling, to dispersion of sound waves in ice
sheets, to singing, frozen lakes, to rumbling devastating noises of calving events and she became curious to get
access to the inaccessible sounds, e.g. sounds to be listened to, beyond the human hearing range. After having
listened extensively to various sonic representations of glacial ice in the form of field recordings and data
samples (presented throughout this article and see Figure 4) the artist found it problematic that some of the
ice sounds would be very pleasing to the ear (when in fact what audiences are listening to is a world in crisis)
or on the other hand sounds of calving events or ocean noise were so noisy and overwhelming that she was afraid
it would scare audiences away—although that can indeed be a strategy in itself.
Polli’s 2004 work Heat and the Heartbeat of the City is an example of a piece that is not an
easy listen, as it plays on discomfort and danger.⁴⁵ Kanngieser and Gibb point out parallels in
Polli’s work to work in Soundscape ecology and field recordings e.g., recordings of the melting glacier
"Vatnajökull" by Chris Watson ⁶⁴ or Krause ²⁸ who has returned to the same
eco-systems for decades, which makes the "ecological processes shockingly obvious" and affectively devastating
²⁵. Rather than shocking and scaring audiences, the artist was inspired to work with the trauma and
pain from a point that does not reproduce the pain and hurt, but by removing some of the shock effect and
devastation of what glaciers actually sound like (very loud noise), by speeding up the geological processes to
create an other-worldly cryoscape. Moving forward she questioned field recording as the best strategy to tell
the story of ice and moved forward towards exploring seismic data sonification as a strategy for sound sources.
Presentation of the installation for stakeholders
Before moving into the prototyping phase, the installation was presented to the board of the Kangiata Illorsua
– Ilulissat Icefjord Centre and implemented in drawings in the exhibition and architectural context.
Through that process the artist presented questions about whether she was going to work with field recordings,
questioning which data-sources to use, and based her presentation on concepts around a “data-driven sonic
portrait of the inland ice.”
The artist explored the possibility of installing speakers in ways to communicate geolocation of stations and
data sources; in one experiment speakers were mirroring their actual locations, in another experiment the
speakers were placed based on the cardinal directions of events (East, West, North, South) and in the end,
through conversation with the curators JAC Studio and Anna Domnick, who exhibited in the same space, Eleven
Movements of the Cryoscape was placed on a 12-meter-long curved wall (figure 8) and a round bench was
designed by JAC Studios in the center of the space, to facilitate a contemplative, prolonged listening
experience.
Protoyping the Installation
In the fall of 2020, artist Louise Foo and programmer Yotam Mann were invited to do a three-week long residency
at ITU’s Affective Interactions and Relations Lab, in which a prototype of the installation was built
before it was shipped to Greenland. They built a program to fetch streams of data from the GLISN database and
PROMICE weather stations, to play from a project website⁴³ using the web audio API and browser-based
audio mixing with Tone.JS. They set up controls for listening to seismic data (such as volume); which station,
which time, date, speed, and durations (e.g., one week). Audio was routed using a Dante interface to respective
channels over network) and an array of speakers were set up on stands in the Lab. The type of speaker landed on
a speaker (A 'Diva Droplet by Gallo Acoustics), inspired by a photo of a seismic sensor in the ice sheet, inside
a round shaped encasing. The round shape communicated ‘a listening portal’ into the ice, and removed
the audience from the expectations towards the usual type of speaker box, which most often will reproduce
recorded sound.
Experiments were made, to implement the monthly glacial data (reports of glaciers' calving and ice quakes) as
well as data from weather stations in the melting areas around Greenland and connect those data to field
recordings, but very quickly the soundscape felt predictable and “animated.” The raw seismic data,
on the other hand, turned out to provide an ever-changing otherworldly, cryoscape.
Four speakers were destroyed in the first few days because of too much low-end noise for the speakers to
handle, and a low-cut filter was implemented. The following weeks were spent on extensive listening and
reflections upon streams of raw seismic data together with people from the lab. The feedback was that people
were drawn towards listening to the soundscapes for extensive periods at a time—it provided for a
contemplative listening experience that moved them. A sense of “eavesdropping” the Inland Ice.
Through that process the team was challenged by how one can compose a soundscape, based on a landscape, in
decline? They were wondering if soundscape composition becomes didactic through the very act of composing and
were, furthermore, wondering if data-sonification could help them avoid contributing to “The tradition of
the appropriation of nature as a resource for the production of culture” ²¹ and instead connect
us more directly to those ecologies?
When listening to constant streams of data (before it has been flagged by a human) it is not possible to know
what the data exactly entails. The team was questioning whether that was important? Is there such a thing as a
hierarchy of the sounds we listen to? Are the significant glacial events (in the ears of science) for example,
more important than the constant flux of other kinds of sounds? How do we choose what is important to listen to?
The raw seismic data sonifications were filtered slightly and a pinch of reverb was added to the channels (for
aesthetic / mixing / sound quality purposes) and the overall sound landed on a low volume that draws your
attention towards each individual speaker, and invites you to come close to each “listening portal.”
The number (eleven) speakers was decided based on the number of reliable sound sources (number of stations that
will be transmitting at the same time). Through the process of prototyping the experience (in relation to a
12-meter-long curved wall) the speakers were placed in an array equally apart, in ear-height, with signs
underneath each speaker with the station codes (notes of paper set on with tape) to signify which location each
speaker was transmitting from. Furthermore, a map of Greenland was implemented, for audiences to connect the
station names to their actual locations, as well as a digital calendar clock, to emphasize the duration / time
of the installation; to support the near real-time listening experience.
The program was built so that transmitting stations could change temporarily if a station is out of order for a
period (which happens when, for example, one is out of order and something needs replacement). Since this is a
permanent installation, we needed to account for several scenarios, the most extreme being one where no stations
were no longer transmitting. We produced a solution where all data is always recorded, which fed into the idea
of thinking about the installation as documentation – an archive for the future. After the prototype of
the installation was finished, scientists Peter Voss and Andreas Peter Ahlstrøm and JAC Studios were
invited to the Lab to experience the installation. Even though Peter V oss is familiar with seismic data and is
looking at it every day from various scales, he pointed out how listening to the ongoing data in multiple
streams at the same time gave him new perspectives on the Inland Ice and he stated that his guess was that 90%
of what we hear in the raw seismic streams was ice-related. The last 10% is earthquakes, wind, waves, and
human-made sounds which we found just as important to listen to, since the installation portrays the Inland ice
as a living, breathing, evolving organism in the age of the Anthropocene, where human beings have made their
mark on the soundscape.
Presenting the final Installation at the Icefjord Centre
As installed at the Icefjord Centre Eleven Movements of the Cryoscape can be experienced in different modes of
listening (see Figure 7); as purely contemplative listening, supported by the bench in the center of the room,
for people to sit and spend time with the overall soundscape created by the sum of all the transmissions. One
can listen more exploratively by moving around in the installation and by getting close to each transmitting
speaker to listen in to specific locations. One can also connect the sounds with logic connecting sounds from
locations to the map and the calendar clock.
The soundscape can also be listened to in conjunction with Anna Domick’s photographs of slices of ice
cores in the piece ‘Inside the Ice Sheet’ in which Eleven Movements of the Cryoscape becomes a
soundtrack to her microscopic views inside the ice sheet, on a journey to past ice ages. Furthermore, a wall
description outside the entrance of the room provides a contextual and conceptual frame for critical reflection.
On the Ilulissat Icefjord Centres website Eleven Movements of the Cryoscape is situated within the
following exhibition context; “as a meeting between art and science in an installation, that through
sound, images and a tactile universe both challenges and expands our perception and understanding of the ice
sheets vast range and its significance for the world we are all part of” (Icefjord Centre Website, 2022).
The Danish newspaper Politiken highlighted Eleven Movements of the Cryoscape in its review of the
Ilulissat Icefjord Centre as “the final'' that despite its underwhelming architectural priority
“houses an installation of 11 speakers” The author explains how “Here you get a real-time
auditory image of the enormous activity in the massif, which seems stagnant to the eye.” He follows up
with how he now “look differently at the icebergs” and how he senses “a deep-felt recognition
of the connection between our way of life and climate change.”
Discussion and Conclusion
Throughout this article we have described the work and the thoughts going into the creation of Eleven
Movements of the Cryoscape—a data sonification installation that can potentially lead to
affectively engaging and sonically sensible spaces for listening, to activate ecological change.
In Eleven Movements of the Cryoscape, the immediate goal is to allow the user to become
affectively engaged by the soundscape to viscerally experience the earth moving and to, based on this,
potentially rekindle his or her understanding of, and relationship to climate change. However, we have also
explored how the sonification of the data can be used by seismologists to gain new knowledge about seismic
events and we see potentials in a transversal aesthetics and believe it can be further explored. When we invited
seismologist Peter Voss to listen to the ongoing streams of seismic data, he listened with curiosity and started
connecting sonic phenomena with guesses of which glacial and tectonic events might be happening; that are not
even 'flagged' as glacial events in the eyes of scientists, but still contribute with interesting sonic
patterns. Future work would engage in a deeper evaluation of the audience’s experience with the
installation.
While we see a lot of potential in using data-sonification for a wide variety of artistic applications, as
Kanngieser and Gibb Point out, aesthetic decisions must be made to translate data into the auditory domain. So,
to follow up their question about how can we inform those aesthetic decisions and foster emotional impact beyond
initial curiosity, that maintains complexity, while drawing out subtle nuances in data whose significance would
be overlooked if communicated by any other medium we can point to seismic data conversion, as a quite
“autentic” way to work with data-sonification, since there is more of a conversion going on than a
mapping per se.
Eleven Movements of the Cryoscape is, among other things, inspired by the interdisciplinarity of ice
studies in the humanities and social sciences offering alternative ways of framing and engaging the frozen
environment ⁴ posing questions like; is ice justice? This question has been challenged by researchers such
as ⁶ who asks if glaciers listen? While ³⁸ analyzes the cultural framing of glaciers, Carey
³ (sees an endangered species narrative applied to glaciers. Jackson (2015) exposes how glaciers are
depicted as ruins, and Sörlin ⁵⁸ refers to the present as a cryo-historical moment because
“ice has become historical, i.e., that ice is an element of change and thus something that can be
considered as part of society and of societal concern.⁵⁸ This has led to further questions about
what it means to portray or sonify ecologies. As an artist in a colonial perspective? As a woman in the
male-dominated scientific realm of geology and glaciology? What can we learn about perception between human and
non-human bodies? In response to those questions, although not all of them have been answered, it has thus been
important to portray the Inland Ice, with respect to Indigenous peoples' relationship to the ice and a
world-view; “where all aspects are interrelated, interdependent and indivisible. Furthermore, we refer to
the installation as a documentation—pointing towards the media archaeological archives of the future
³⁹ and have exchanged the term ‘soundscape’ with the term ‘cryoscape’, to
emphasize the dynamic relationship between ice and people.³
In addition to conveying the effect of climate change on the ice sheet, Eleven Movements of the Cryoscape
contributes to the exhibition by immersing audiences' bodies in the ice sheet from depths, corners and
scales that would be otherwise impossible. It explores first-hand if we can foster a deeper connection,
understanding and compassion (and planetary healing) by tuning in and “sitting with the
world”—through listening? Through working on this project, we see immense potential in listening to
more data that in similar ways can connect us to our planet and its context: listening outwards towards other
planets and the sun, as well as inwards into sonic worlds of human and non-human interrelations. Furthermore,
the project proposes to reach beyond art institutions as output and activate platforms for different (and
potentially broader outreach) than traditional art institutions offer. In so doing, questions of how this type
of artistic research contribute to technological, scientific, and environmental discourse are raised in a novel
manner.
Acknowledgments
We would like to thank all collaborators and stakeholders in the installation: Johan Carlsson (JAC Studios,
curator of the exhibition) Aviaaja Ezekiassen (project manager, JAC) and Elisabeth Momme (Director of The
Icefjord Centre) Also thanks to artist Anna Dominick (co-exhibitor) and the genius artist and programmer Yotam
Mann who was a core part of the project and a huge resource in designing and developing the sonic experience. We
also thank Halfdan Hauch Jensen, Manager of the AIR Lab, IT University of Copenhagen, for support during the
prototyping phase.
We would also like to thank the GLISN network of seismic sensors and PROMICE weather stations, for letting
their data be available for artistic use. And thanks to GEUS and Niels Bohr Institute and all the scientists and
experts that have generously shared their research and given valuable feedback to help this project come to
life: J. P. Steffensen, Glaciologist Andreas Ahlstrøm and Oceanographer Oscar Glowaki. A very special
thanks to Seismologist Peter Voss who took the time to explain in depth about seismology and to point us to the
data-sources, we hope to do more cross-pollination work in the future between art & seismology.
Finally, we thank all the people who engage with the installation daily. The project was generously supported
by The Danish Arts Foundation.
References
1 Michael Bull, Marcel Cobussen, (Eds.), The Bloomsbury Handbook of Sonic
Methodologies, NY: Bloomsbury Academic, 2021.
3 Mark Carey, M. Jackson, Alessandro Antonello, Jaclyn Rushing, “Glaciers, gender, and
science”, Progress in Human Geography. 40, 6m 2016, 770-793, DOI:
https://doi.org/10.1177%2F0309132515623368
4 Mark Carey, “The history of ice: How glaciers became an endangered species”,
Environmental History 12, 2007, 497–527.
5 Linda Candy, Ernest. A. Edmonds, Practice-Based Research in the Creative Arts: Foundations and
Futures from the Front Line, Leonardo, 51(1), 2018, 63-69.
6 Julie Cruikshank, Do Glaciers Listen? Local Knowledge, Colonial Encounters, and Social
Imagination, Vancouver, UBC Press, 2005, p.6.
7 Anne Despond, Nicolas Reeves, Vincent Cusson, Atmosphéries and the poetics of the in
situ: the role and impact of sensors in data-to-sound transposition installations, Audio Mostly 2021,
Association for Computing Machinery, New York, NY, USA, 2021, 48–55,
https://doi.org/10.1145/3478384.3478422
8 Dan Drollette Jr., Bulletin of the Atomic Scientist: How fast is the Arctic ice retreating? Just
listen to it melt, June 2018, Retrieved April 30, 2022,
https://glacierhub.org/2020/04/16/acoustics-of-meltwater-drainage-in-greenland-glacial-soundscapes/
9 Antonio B. Dobrowolski, Historia naturalna lodu (The Natural History of Ice), Warzawa, Kasa
Pomocy im., 1223.
10 Klaus Dodds, Sverker Sörlin (Ed.), Ice humanities: Living, thinking, and working in a
melting world, Manchester University Press, 2022.
11 K. Doughty, M. Duffy, T. Harada, Sounding Places: More-Than-Representational Geographies of
Sound and Music, UK: Edward Elgar Pub, 2019.
12 Jonas Fritsch, Understanding Affective Engagement as a Resource in Interaction Design,
Engaging Artefacts, 2009, 1–9.
13 Jonas Fritsch, Affective Interaction Design at the End of the World, DRS 2018:
Catalyst, 2018, 896–908.
14 Melissa Gregg, Gregory J. Seigworth, eds., The Affect Theory Reader, Duke University
Press, 2010.
15 Isabelle Gapp, Galvanizing glaciology: thoughts on an eco-critical art history (January 2022),
Environmental History Now, 2022, Isabelle Gapp: Retrieved, April 30th, 2022,
https://envhistnow.com/2022/01/20/galvanizing-glaciology-thoughts-on-an-ecocritical-art-history/
16 Oskar Glowacki Department of Polar Research, Institute of Geophysics, Polish Academy of
Sciences in Warsaw, Poland “Underwater acoustic signatures of glacier calving” O. Glowacki, G.B.
Deane, M. Moskalik, Ph. Blondel, J. Tegowski, M. Blaszczyk Geophysical Research Letters, DOI:
10.1002/2014GL062859 retrieved from https://www.youtube.com/watch?v=6jqTm3TMGvI
17 GEUS - Geological Survey of Denmark and Greenland, Retrieved April 30, 2022,
https://www.geus.dk
18 Félix Guattari, The Three Ecologies, Continuum Press, 1989.
19 GLISN Website, Retrieved April 30, 2022, http://glisn.info
20 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke University
Press Books, 2016.
21 Haraway, "A Cyborg Manifesto: Science, Technology, and Socialist Feminism in the Late Twentieth
Century," in Simians, Cyborgs and Women: The Reinvention of Nature, New York; Routledge, 1991, 1985;1991,
149-181.
22 The Icefjord Centre, About the House: The Exhibition, Retrieved April 30, 2022,
https://isfjordscentret.gl/en/about-ilulissat-icefjord-centre/
24 Michael Jackson, Glaciers and climate change: Narratives of ruined futures, WIREs Climate
Change, 2015, DOI: 10. 1002/wcc.1351.
25 AM Kanngieser, Rory Gibb, Sonifying the world in Sounding Place, Edited by Carolina Dougherty,
Michelle Duffy and Theresa Harada, Cheltenham: Elgar, 2019, 153-160.
26 Gregory Kramer, Bruce Walker, Terri Bonebright, Perry Cook, John H. Flowers, The Sonification
Report: Status of the Field and Research Agenda, Faculty Publications, Department of Psychology, 1999, 444.
27 Melanie Klein, This changes everything, Simon+Schuster Inc., 2014.
28 Bernie Krause, The Great Animal Orchestra: Finding the Origins of Music in the Worlds Wild
Places. London: Profile Books, 2012.
29 Dennis Overbye, Gravitational Waves Detected, confirming Einstein's theory, New York
Times, 2016, Retrieved April 30, 2022,
https://www.nytimes.com/2016/02/12/science/ligogravitational-waves-black-holes-einstein.html?searchResultPosition=1
30 Jacob Kirkegaard, Earside Out, Museum of Contemporary Art, Denmark: Isfald, 2015, Retrieved
April 30, 2022, https://fonik.dk/works/isfald.html
31 JAC Studios, Projects: Ilulissat Isfjord Centre - The story of Ice, 2021, Retrieved April 30,
2022, http://jacstudios.dk/ilulissat-icefjord-centre-the-story-of-ice/
32 Elizabet Leane Jeffrey McGee (Eds), Anthropocene Antarctica, Perspectives from the Humanities,
Law and Social Sciences, Routledge, 2019.
33 Dorte Mandrup, Work: Ilulissat Isfjord Centre, 2021, Retrieved April 30, 2022,
https://www.dortemandrup.dk/work/ilulissat-icefjord-centre- greenland
34 Brian Massumi, Parables for the virtual: movement, affect, sensation, Duke University Press,
2002.
35 Thomas J. Mitchell, Alex J. Jones, Michael B. O'Connor, Mark D. Wonnacott, David R. Glowacki,
Joseph Hyde, Towards molecular musical instruments: interactive sonifications of 17-alanine, graphene and
carbon nanotubes, Proceedings of the 15th International Conference on Audio Mostly, Association for
Computing Machinery, New York, NY, USA, 2020, 214–221, https://doi.org/10.1145/3411109.3411143
36 Marcus Nüsser, Ravo Baghel, The emergence of the cryoscape: Contested narratives of
Himalayan glacier dynamics and climate change, In: Benjamin Schuler (ed.) Environmental and Climate Change in
South and Southeast Asia, Leiden: Koninklijke Brill, 2014, 138–156,
Https://doi.org/10.1163/9789004273221_007
37 Niels Bohr institute website: https://nbi.ku.dk/english/
38 Ben Orlove, Ellen Wiegandt, Brian H Luckman BH, The place of glaciers in natural and cultural
landscapes, In: Orlove B, Wiegandt E, Luckman BH (eds) Darkening Peaks: Glacial Retreat, Science, and
Society, Berkeley, CA: University of California Press, 2008, 3–19.
39 Jussi Parikka, What is Media Archaeology? Cambridge: Polity Press, 2012.
40 Bryan C. Pijanowski, Luis J. Villanueva-Rivera, Sarah L. Dumyahn, Almo Farina, Bernie L.
Krause, Brian M. Napoletano, Stuart H. Gage, Nadia Pieretti, Soundscape Ecology: The Science of Sound in the
Landscape, BioScience, Volume 61, Issue 3, March 2011, 203–216,
https://doi.org/10.1525/bio.2011.61.3.6
41 Evgeny A. Podolskiy, Toward the Acoustic Detection of Two‐Phase Flow Patterns and Helmholtz
Resonators in Englacial Drainage Systems. Geophysical Research Letters 47, 2020, Retrieved
http://dx.doi.org/10.1029/2020GL086951.
42 PROMICE Website, Retrieved April 30, 2022, https://www.promice.org
43 Eleven Movements of the Cryoscape Project Website, Retrieved April 30, 2022 from Eleven
Movements of the Cryoscape Project Website, 2021, Retrieved April 30, 2022, https://icefjord.nn.r.appspot.com/
44 Zhigang Peng, Chastity Aiken, Debi Kilb, David R. Shelly, Bogdan Enescu, Listening to the
2011 Magnitude 9.0 Tohoku-Oki, Japan, Earthquake. In Seismological Research Letters, Vol. 83, Issue 2, 2012,
287–293, Seismological Society of America (SSA), https://doi.org/10.1785/gssrl.83.2.287
45 Andrea Polli, 2004, Heat and the Heartbeat of the city: https://vimeo.com/127884090
46 Andrea Polli, Atmospherics/Weather Works: A Spatialized Meteorological Data Sonification
Project, Leonardo 38, 2005, 31–36, Retrieved from http://dx.doi.org/10.1162/leon.2005.38.1.31.
47 Andrea Polli, Heat and the heartbeat of the city: Sonifying data describing climate change,
Leonardo Music Journal, 16, 2006, 44-45.
48 Andrea Polli, Joe Gilmore, Sonification of measured (National Oceanic and Atmospheric.
Administration's Arctic program) and model weather data, Leonard Music Journal 5:34, 2006.
49 Marty Quinn, Research set to music: The climate symphony and other sonifications of ice core,
radar, DNA, seismic, and solar wind data, Proceedings of the 7th International Conference on Auditory Display
(ICAD01), Espoo, Finland, 2001.
50 Michael Quinton, Iain McGregor, David Benyon, Sonification of an exoplanetary atmosphere,
Proceedings of the 15th International Conference on Audio Mostly, Association for Computing
Machinery, New York, NY, USA, 2020, 191– 198, https://doi.org/10.1145/3411109.3411117
51 Michael Quinton, Iain McGregor, David Benyon, Sonification of Planetary Orbits in Asteroid
Belts, Audio Mostly, Association for Computing Machinery, New York, NY, USA, 2021, 72–80,
https://doi.org/10.1145/3478384.3478390
52 Audrey Ramming, GlacierHub: Acoustics of Meltwater Drainage in Greenland Glacial Soundscapes
(April 2020), Retrieved April 30, 2022,
https://glacierhub.org/2020/04/16/acoustics-of-meltwater-drainage-in-greenland-glacial-soundscapes/
53 Publication by Realdania, Bog om Ilulissat Isfjords-center: En ganske særlig bygning
på et ganske særligt sted, 2021, Retrieved April 30, 2022,
https://realdania.dk/projekter/ilulissat-isfjordscenter
54 Niklas Rönnberg, Sonification for Conveying Data and Emotion, Audio Mostly
2021, Association for Computing Machinery, New York, NY, USA, p.56–63,
https://doi.org/10.1145/3478384.3478387
55 Ben Rubin, Hearld Thorp, Harbinger is a permanent public artwork by Ben Rubin and Jer Thorp
commissioned for Brookfield Place in Downtown Calgary, Alberta, https://vimeo.com/250393598
56 Margaret Schedel, David R. Worrall, New Wor(l)ds for Old Sounds, In Organised Sound, vol 19,
2014, 1-3, doi:10.1017/S1355771813000356.
57 Isabelle Stengers, In Catastrophic Times: Resisting the Coming Barbarism, Open Humanities
Press, 2009/2013.
58 Sverker Sörlin, Cryo-History: Narratives of Ice and the Emerging Arctic Humanities, The
New Arctic, 2015, 327, Retrieved from http://dx.doi.org/10.1007/978-3-319-17602-4_24.
59 Anna Tsing, The Mushroom at the End of the World: On the Possibility of Life in Capitalist
Ruins, NJ: Princeton University Press, 2015.
60 Sarah E. Truman, Stephanie Springgay, The primacy of movement in research-creation: New
materialist approaches to art research and pedagogy, In M. Laverty, & T. Lewis (Eds.), Art’s
Teachings, Teaching’s Art: Philosophical, critical, and educational musings, New York, NY: Springer,
2015, 151-164.
61 Salomé Voegelin, Listening to Noise and Silence: Towards a Philosophy of Sound Arts, NY:
Bloomsbury, 2010.
62 Salomé Voegelin, Sonic Possible Worlds: Hearing the Continuum of Sound, NY: Bloomsbury,
2014.
63 Salomé Voegelin, The Political Possibility of Sound: Fragments of Listening, NY:
Bloomsbury, 2018.
64 Chris Watson, 2003, https://chriswatsonreleases.bandcamp.com/track/vatnaj-kull
65 David Worrall, Sonification Design: From Data to Intelligible Soundfields, CHE: Springer, 2019.
Author Biographies
Danish-born artist Louise Foo creates interactive experiences at the intersection of sonic and visual arts,
to facilitate novel modes of listening and engaging. Her recent work counts data-sonification of geological
movements to document the melting of the Inland Ice, interplanetary musical explorations with sounding spheres
and ambisonic music.
Jonas Fritsch, PhD, is Associate Professor in Interaction Design at the IT University of Copenhagen in the
Department of Digital Design. He is head of the Affective Interaction & Relations (AIR) Lab and Head of
the Design Research Section. His work revolves around a creative thinking of interaction design, design
processes, experience philosophy and affect theory through practical design experiments with interactive sound
and physical interfaces.
Cosmo-Techno-Poiesis: Architecture of Environmental Control
SebastianGatz
University of Gothenburg, HDK-Valand Konstfack, Stockholm / sebastian.gatz@konstfack.se
Abstract
Architecture is a collective technological human practice to control local environments in order to protect the
human body. Depending on the worldview and cosmology of the society which produces the
architecture—increasingly the global worldview is rational, mono-technological and Western—this
practice gives more or less space for nonhuman agency. This paper looks at ways to loosen the contemporary
Western obsession with controlling the environment architecturally by exploring different forms of architectural
memento mori (remember you must die). The act of willing (poiesis) protective-controlling architecture
(technology) into the world, inside a certain worldview (cosmology), is explored through the conceptual
entanglement of those three notions: cosmo-techno-poiesis. The paper concludes with an architectural
example and a short summary.
“While traditional architecture was capable of providing shelter from the environment, the advent of HVAC
(Heating, Ventilating, and Air Conditioning) systems at the beginning of the twentieth century established the
building envelope as a cocoon in which an alternative universe was maintained.”—Michelle Addington
¹
The architectural boundary is a place of intense negotiation. The layer between “inside” and
“outside” does not just mediate between indoor and outdoor climate (the focus of most construction
regulations) but also negotiates a psychological positioning of the inhabitant in relationship to their
environment. Michelle Addington, dean of the University of Texas at Austin School of Architecture, problematizes
the increasingly hermetically sealed architectural envelopes, which divorce the human body from its natural
environment, in order to endorse the work of artist-architect Philip Beesley. ² His almost transhumanist
sculptures rethink the engagement of humans with architectural boundaries through technological augmentation and
almost life-like behaviors: “... Beesley's environments always question the idea of boundary, and in doing
so, question the very nature of architecture.” ³
Architecture divides what is inside and what is outside, what is nature and what is culture. The inside is a
space of control, while the outside is uncontrolled and hostile. The environment is either filled with
archetypes of fear or with energies and weather phenomena not fitting into our energy calculations. The latter
leads to increasingly higher energy standards, such as those for nearly zero-energy buildings (NZEB), ⁴
and with it prescribed wall insulation values and air-tightness factors.
Architect and artist Luis Berríos-Negrón uses the “green-house”—as an actual
architectural typology and a metaphorical artistic device—to question nature's “outsider
position” and to describe how architecture can manifest a human mindset towards nature:
Control and domination are expressed by placing nature, ironically, on the “inside,” where man
thinks he can dominate life—to extract and accelerate, or decelerate and conserve growth—regardless
of place or season. That site-less, modern, illusory power sandwiched between the industrial logic of
translucent surfaces is that which cuts, represses, forgets, and fore-gets the realities of an
habitable environment. ⁵
Architecture in that sense is not just a technological solution which mitigates energy losses but is also a
manifestation of ideology which separates the human body from the natural environment—or even reality. The
architectural envelope is where an increasingly questionable separation of nature and culture ⁶ is
formalized and manifested—in physical actuality and in a metaphorical collective mindset. This paper looks
at architectural (or at least theoretical-architectural: architectonic) ways out of nature-culture dichotomies
while realizing at the same time that Haraway's idea of naturecultures—an endless conglomerate of
entanglements of matter, stories, life and so forth—has practical “common sense” limits when
it comes to buildings: Who in the Western world wants to remove the architectural separator between the human
and its environment? The text looks at possible ways to loosen this boundary, by reintroducing nature as an
architectural companion species. ⁷
According to artist-architect Frederick Kiesler, architecture is a form of technological environment
which helps humans in their survival in the natural environment. Technology, according to him, is
a form of social heredity, which has in it the long history of toolmaking: “Thus a contemporary chair, for
instance, is the product of many generations of other tools for man to rest his body in fatigue. This is
heredity in technology.” ⁸
What that means is that our built environment, or technological environment, is an archive of previous
human-nature struggles and former knowledge production. Architecture is a technology which is placed between the
human body and its environment. The formulation of this technological “machine” visualizes a
society's relationship to the “actual” and the “imagined” natural environment. While
actual local weather characteristics, material availability and cultural idiosyncrasies contribute to a
vernacular architecture and its specific boundary conditions, the worldview of that society (cosmology)
negotiates how much those boundaries are open to inside-outside mediation.
Yuk Hui coined the term cosmotechnics ⁹ in order to work on the problems that he sees in
the world, one of them: An unquestioned mono-technological development which is rooted in a certain global
worldview (cosmology). This one-dimensional development, according to him, contributes to the destruction of the
planet and to the maintenance of fixed human-cosmos, human-nonhuman and nature-culture relations:
I gave a preliminary definition of cosmotechnics as unification between the cosmic order and the moral order
through technical activities, in order to suggest that technology should be re-situated in a broader reality,
which enables it and also constrains it. The detachment of technology from such a reality has resulted from the
desire to be universalizing and to become the ground of everything. Such a desire is made possible by the
history of colonization, modernization and globalization, which, being accompanied by its history of economic
growth and military expansion, has given rise to a mono-technological culture in which modern technology becomes
the principal productive force and largely determines the relation between human and non-human beings,
human and cosmos, and nature and culture. The problems brought about by this mono-technological culture are
leading to the exhaustion of resources and of life on earth and to the destruction of the environment, which are
central to the discourse around the Anthropocene.¹⁰
Western architecture—as a form of technology to negotiate between humans and nature—follows the
logic of what Hui describes: Mono-technological prescriptions, which are embedded in an unquestioned but slowly
crumbling technoscientific cosmology, are dictated by law and leave no room for alternative cosmologies,
techno-diversity or alternative engagements of humans with their environment. Or as Helmuth Trischler and
Fabienne Will state it when talking about the Technochene / Technosphere: “Humans submit to
the artifacts they have created themselves, shifting the responsibility for human-environment relationships to
things.” ¹¹
The increasingly hermetically sealed borders of Western buildings do not solve the root problem of a troubled
human-environment relationship. In fact, their purpose of creating borders between the human and its environment
does exactly that: deepening the border between the human and its environment.
Poiesis: Willing Architecture into the World
Architecture, or in fact every form of natural or man-made matter, is a manifestation of invisible forces,
entanglements and hidden networks of different human and nonhuman power struggles. Kiesler calls these
“inter-acting” forces co-reality:
What we call “forms,” whether they are natural or artificial, are only the visible trading posts of
integrating and disintegrating forces mutating at low rates of speed. Reality consists of these two categories
of forces which inter-act constantly in visible and invisible configurations. This exchange of inter-acting
forces I call CO-REALITY , and the science of its relationships, CORREALISM. The term “correalism”
expresses the dynamics of continual interaction between man and his natural and technological environments.
¹²
Specifically, the term inter-action sounds very familiar and foreshadows feminist theorist and
physicist Karen Barad's intra-action by a few decades:
The neologism “intra-action” signifies the mutual constitution of entangled agencies. That is, in
contrast to the usual “interaction,” which assumes that there are separate individual agencies that
precede their interaction, the notion of intra-action recognizes that distinct agencies do not precede, but
rather emerge through, their intra-action. It is important to note that the “distinct” agencies are
only distinct in a relational, not an absolute, sense, that is, agencies are only distinct in relation to their
mutual entanglement; they don't exist as individual elements. ¹³
Architecture comes into being when the time is ripe. All inter-acting, or intra-acting, forces come together:
Something or someone wants it to exist and this something or someone has the means to make it exist.
Architecture in that sense is a form of individual or collective artistic poiesis—the will to
bring into being. Poiesis is where artwork, artist, receiver, idea and matter come together. ¹⁴ The
concept overlaps with the foundational concept of magic, where willing leads to manifestations in the world or
control over the ego. Will, in the magical sense, is neither good nor bad but depends on the intention of the
magician. ¹⁵ The size, construction time, material quality and the technological standard of
architecture is directly related to the intensity with which one or many will. An Egyptian pyramid or a New York
City high-rise is brought into the world with a considerable bigger amount of willing than a hut made from
leaves, or a tent bought in the supermarket (while its existence as a typology comes with a long history of
willing people as well). The willing of buildings, and accordingly the manifestation of them, is in itself not a
bad thing. To will protection from the environment's “attacks,” for example, comes with a different
amount of control fantasies (or karma) than the willing of the next biggest building in the world. Intention is
key. We can consider the built environment (technical environment)—the existing stock of matter which
divides human bodies from their natural environment—to be the product of human poiesis or their willing to
keep nature at a distance.
Another interesting question is if architecture – as a global technology – is in fact creating or
perpetuating itself. Peter Haff argues that the Anthropocene is essentially built on the technoshpere
(the technological conglomerate of human activity) which is out of the control for the individual and
therefore collective human:
[W]e abandon the apparently natural assumption that the technosphere is primarily a human-created and
controlled system and instead develop the idea that the workings of modern humanity are a product of a system
that operates beyond our control and that imposes its own requirements on human behavior. The technosphere is a
system for which humans are essential but, nonetheless, subordinate parts. As shorthand we can say that the
technosphere is autonomous. This does not mean that humans cannot influence its behavior, but that the
technosphere will tend to resist attempts to compromise its function. ¹⁶
In a sense, architecture—as part of the technosphere which perpetuates the Anthropocene—can be seen
as a nonhuman entity which wants to be fed with more technology, more matter and more human will or it fights
back: a Golem, Frankenstein's monster.
How much are we in control of our architectural (techno)logic and how much control should it have over the
environment and us?
Memento Mori: Giving-Up Control
How much control is too much control? The following part will concentrate on the concept of death and life as
metaphorical transient categories of architecture. Newly built architecture can be considered being
“born” out of human will or poiesis and in total control of the inner—and therefore
psychologically the outer—environment, while “neglected” buildings can be considered to be in
the process of dying and therefore the human is less in control of inner our outer environments. What productive
states of architecture exist between birth and death, between control and giving up control, which allow for
nonhuman agency and provide human habitats at the same time?
While the transhumanists want to overcome human mortality ¹⁷ and the limits of the planet
(multi-planetary thinking can be considered to be the big brother or sister of individual transhumanism),
philosophy professor Patricia MacCormack argues, from an ahuman perspective, for a way out of the world's
problems through the embracing of death:
Where I want to push things is through the incorporation of the death of the species in actuality and to cease
the actual death of the nonhuman other as inherently joyful affirmative qualities of protest, and this I see
exemplified in two activist philosophies which result in two material actions: human extinction and animal
abolition. ¹⁸
Ahumanism and transhumanism can be seen as being on an oppositional spectrum with different degrees of human
control desires over the nonhuman other. Critical / cultural / philosophical posthumanism fits better
to the former, in terms of giving up human control or acknowledging more agency of nonhumans ¹⁹ while
cybernetic posthumanism overlaps with the latter. ²⁰ While MacCormack advocates the
slow phase-out of human control over the world ²¹ the transhumanists want to achieve a god-like
status, which allows them to be in total control of life itself. ²² One can argue that both, the
voluntary species suicide and the conquest of death, is hubristic and therefore human-centric. The following
part explores steps on the gradient between both extreme positions in order to generate productive ways to look
at architectural production and advocates for a Buddhist-like middle way of architectural environmental
control.
The Anthropocene exists, at least partially, because of an unbalanced human understanding of how much
nature control is the right amount. Technology is a gift which allows us to take what we need—but we
thought we needed it all. The following part investigates some concepts, which are seen as fruitful for an
architectural discourse, and which allow nonhuman agency and try to avoid total environmental control. Which
states of architecture exist on the spectrum of species suicide to eternal life? Our Western architectural
thinking is increasingly moving towards the latter. While houses used to be built out of natural, available
materials which would decay at some point or another, today we (try to) build for eternity. While the killing of
our building stock (explosion) would be similar to MacCormack's extreme, the other extreme can be found in
contemporary construction regulations: the denial of death and the desire for eternal life and absolute
environmental control.
Here, we will look at the idea of an architecture of decay as a form of memento mori (remember you
must die). Decay here is shown as a one-directional entropy of architecture but hopes to reopen the thinking for
an architecture of imperfection, transient form, openness, open-endedness and flexibility (openness to agency of
others, ready to give up human control).
The transient nature of decay, decomposition ²³ and wabi ²⁴—or its digital
counterpart: the glitch—can be seen as forms of nonhuman agency. When talking about the historical
Japanese tea master Sen no Rikyū and his understanding of wabi, Rumiko Handa, professor of architecture at
the University of Nebraska-Lincoln, states the following:
There are a number of instances that demonstrate Rikyū’s preference for impermanence, which is
important because it shows the limitation of human control in comparison to natural forces. Rikyū’s
desire to submit to forces beyond human control extends to allowing the artifact not to function in its primary
utility. ²⁵
Rosa Menkman, when speaking about glitches as machine-revealing entities, acknowledges nonhuman agency in a
similar fashion:
A glitch is the most puzzling, difficult to define and enchanting noise artifact; it reveals itself to
perception as accident, chaos or laceration and gives a glimpse into normally obfuscated machine language.
Rather than creating the illusion of a transparent, well-working interface to information, the glitch captures
the machine revealing itself. ²⁶
While the medium for wabi and glitch is different, they share the similarity of acknowledging and appreciating
nonhuman agency—giving up human control. One could say that the former is engaging with natural nonhuman
agencies while the latter is focusing on man-made (technological) non-human agencies. Both reveal a hidden
“true reality” which is outside of the control of humans.
Both, the glitch and the concepts related to wabi, work only in an equilibrium of the right amount of human
control and nonhuman agency at the same time. Leaning too much on either side of the control spectrum, from
human to non-human agency, can make the system collapse. Similar to a ruin which is only acknowledged as such if
some parts are still standing and some damage exists. ²⁷ The nonhuman in the glitch and in the idea
of wabi can be seen as a form of transient or, what political theorist Jane Bennett calls, “vibrant
matter” ²⁸ or as a form of intra-action of time, space and matter. ²⁹ And maybe it
is here where the true problem of the Anthropocene is located: time, or better the non-acceptance of time and
its transient nature. Our scientific technological rational Western cosmology suggests being always on top of
the invisible matterings, intra-actions and vibrant matters. Glitches reveal the malfunctioning of the man-made
system which forces one to acknowledge one's own death. The Western people want to control time or better their
own death and the decay of the body—due to a lacking enchanted cosmology. ³⁰ In the same way
that architecture is a representation of our collective cosmological worldview, current architectural practices
as technologies to control nature – symbolize, or manifest, our collective worldviews and desires. A
current desire is to maintain or increase control over the world and to conquer death, as the last natural
entity, which is clearly visible in the aforementioned transhuman interpretation of posthumanism and in the
current direction towards becoming a multi-planetary species. The denial of planetary limits can be seen as a
form of denying any end to growth (aka death).
In order to propose alternative ways to think architecture outside of the mono-technological cosmology of total
environmental control the author would like to propose to think—as many others have before
³¹—architecture as ruins with different stages of decay. Similar to the different stages of a
human body's decomposition ³² architecture undergoes—if left to the natural
forces—different steps of decay as well: from finished (alive) to ruined (skeletonized) to vanished. Can
this, ironically human-centric, model and understanding of decomposition help to broaden an architectural
understanding which is neither technophile nor morbidly romantic of the past and its ruins? What can a
building's skeleton be used for? Which qualities does a moldy building still have? No skin but stable bones?
While the decay of a body conventionally goes—despite what transhumanists wish—only in one direction
(entropy), a building's ruination process is more ambivalent, as the example of Architecten de Vylder Vinck
Taillieu's Caritas project shows:
[The architects] proposed to keep the already half-demolished building as it was and to make it accessible as a
public space. Since the roof had already been removed and part of the wooden floor were gone, the wind and rain
would further corrode the construction, causing the wooden planks to rot and the brick walls to crumble bit by
bit. The architects limited themselves to a number of interventions to stabilize the building, while also
further facilitating the process of decay over time. ³³
This form of architectural engagement with the ruin allows humans to inhabit the space while at the same time
gives agency to natural nonhuman others. Architect and researcher Bart Decross calls this form of
agency—when talking about John Ruskin's advocacy for imperfection—“vital materiality.”
³⁴ One can only speculate if the architectural experiencing of controlled decay reminds the visitors
consciously or subconsciously of their own mortal limits (memento mori) and if this in return could contribute
to a humbler positioning of the self in relation to the cosmos.
The decay of the building can be considered only to be a movement towards an end from a human or building point
of view. The decay of matter, human or building, is where vital materiality and vibrant matter meet. While every
biotic body dies from the moment it is born, it can be said to be only a death from the point of view of that
body. The final “death” is the moment when it serves as energy for other natural processes, as food
for someone else, as the following description by research scientist and forensic anthropologist Arpad Vass
illustrates:
I came to the conclusion, somewhat facetiously, that with the exception of micro-organisms living in deep-sea
vents, every micro-organism known is involved in some aspect of the human decompositional cycle from Acetobacter
to Zooglea. While many of the organisms isolated come from the bowel and respiratory tract, literally hundreds
of species are involved in the decompositional process and decomposition would not progress without them.
³⁵
Summary
Architecture is a form of technology which helps society and the individual to control their immediate
environment. The intensity or style of control, in the contemporary West, tries to remove nonhuman agency from
the equation in order to obtain maximal control over natural processes. This control fantasy is based on a
shared technoscientific cosmology. The idea of cosmo-techno-poiesis points at the complex entanglement
of cosmology, technology and the human act of creating. Manifesting architecture, as a technology to keep nature
at a distance, is not just an act which is ethically self-evident or true, but also an act of taking part in the
creation or expansion of worldviews and therefore cosmologies. Architectural production is
cosmo-techno-poietic production and should not be reduced to the creation of technological environments
for the sake of excluding natural environments.
Wabi, glitch, ruin, imperfection and similar concepts of decay—summed up as memento mori—can
potentially be engaged in architectural, actual and metaphorical, production which acknowledges nonhuman
agency and breaks with human technoscientific control fantasies.
The current call for non-extractive forms of architectural production and degrowth thinking ³⁶
demands new modes of thinking architectural (vital) matters. This paper is a call for technodiversity and new
cosmologies which see technology as a tool of the middle way and not of total environmental control.
References
1 Michelle Addington, “Architecture of Contingency,” in Hylzoic Ground:
Liminal Responsive Architecture: Philip Beesley, ed. Pernilla Ohrstedt and Hayley Isaacs, Riverside
Architectural Press, 2010, 68-69.
2 Addington.
3 Addington, 74.
4Commission Recommendation (EU) 2016/1318 of 29 July 2016: On Guidelines for the Promotion of
Nearly Zero-Energy Buildings and Best Practices to Ensure That, by 2020, All New Buildings Are Nearly
Zero-Energy Buildings, 2016.
5 Luis Berríos-Negrón, Breathtaking Greenhouse Parastructures: A Supplement to the
Arcades Project from a Caribbean Perspective [and a Call for a Careful Practice of Epistemológica]
(Konstfack Collection, 2020), Exposé.
6 Donna Jeanne Haraway, The Companion Species Manifesto: Dogs, People, and Significant
Otherness, vol. 1 (Prickly Paradigm Press Chicago, 2003, 6-7.
7 Haraway, 23.
8 Frederick Kiesler, “On Correalism and Biotechnique: A Definition and Test of a New
Approach to Building Design,” Architectural Record 86, 1939, 61.
9 Yuk Hui, “Cosmotechnics as Cosmopolitics,” e-flux journal, no. 86, 2017,
https://www.e-flux.com/journal/86/161887/cosmotechnics-as-cosmopolitics/.
11 Helmuth Trischler, Fabienne Will, “Technosphere, Technocene, and the History of
Technology”, Icon, London, England, 2017, p.23, 12, issn: 1361-8113.
12 Kiesler, “On Correalism and Biotechnique: A Definition and Test of a New Approach to
Building Design,” 61.
13 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter
and Meaning, Durham, 2007, 33.
14 Derek H. Whitehead, “Poiesis and Art-Making: A Way of Letting-Be,” Contemporary
Aesthetics (Journal Archive) 1, no. 1, 2003.
15 Israel Regardie, The Tree of Life: A Study in Magic, Samuel Weiser, Inc., 1995, 126.
16 Peter Haff, “Humans and Technology in the Anthropocene: Six Rules,” The
Anthropocene Review 1, no. 2, 2014, 127.
17 Hava Tirosh-Samuelson, “Engaging Transhumanism,” in H+/-: Transhumanism and its
Critics, ed. Gregory R. Hansell and William Grassie, Metanexus Institute, 2011, 39-42.
18 Patricia MacCormack, The Ahuman Manifesto: Activism for the End of the Anthropocene,
Bloomsbury Publishing, 2020, 142-143.
19 Francesca Ferrando, “Posthumanism, Transhumanism, Anti-humanism, Metahumanism, and New
Materialisms,” Existenz 8, no. 2, 2013, 29-30.
20 Cary Wolfe, What is Posthumanism?, vol. 8, University of Minnesota Press, 2010, xiii.
21 MacCormack, The Ahuman Manifesto: Activism for the End of the Anthropocene, 140-149.
23 Ami Ronnberg, Kathleen Martin, The Book of Symbols, Taschen Cologne, 2010,
764-765.
24 Rumiko Handa, Allure of the Incomplete, Imperfect, and Impermanent: Designing and
Appreciating Architecture As Nature, Routledge, 2014, 153-168.
25 Handa, 163.
26 Rosa Menkman, The Glitch Moment(um), Institute of Network Cultures, 2011, 29-30.
27 Brian Dillon, “Introduction: A Short History of Decay,” in Ruins, ed.
Brian Dillon, Whitechapel Gallery / The MIT Press, 2011, 11.
28 Jane Bennett, Vibrant Matter, Duke University Press, 2010, vii-xi.
29 Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and
Meaning, 141.
30 Christopher Partridge, The
Re-Enchantment of the West: Alternative Spiritualities, Sacralization, Popular Culture and Occulture,
vol. 1, London, T&T Clark International, 2004, 8-11.
31 Dillon, “Introduction: A Short History of Decay.”
32 Arpad A. Vass, “Beyond the Grave: Understanding Human Decomposition,” Microbiology
Today 28, 2001, 190-192.
33 Bart Decroos, “How Gothic is Contemporary Architecture? The Appreciation of Craftmanship
as a Ruskinian Aesthetic of Imperfection,” in Thinking-Making: When Architects Engage in
Construction [Penser-Faire: Quand des Architectes se Mêlent de Construction], ed. Pauline
Lefebvre, Julie Neuwels, Jean-Philippe Possoz, Éditions de l'Université de Bruxelles, 2021, 125.
34 Decroos, 118.
35 Vass, “Beyond the Grave: Understanding Human Decomposition,” 192.
36 Space Caviar, ed., Non-Extractive Architecture: On Designing without Depletion, vol.
1, V-A-C Press / Sternberg Press, 2021.
Bibliography
Addington, Michelle. “Architecture of Contingency.” In Hylzoic Ground: Liminal Responsive
Architecture: Philip Beesley, edited by Pernilla Ohrstedt and Hayley Isaacs, 66-75. Riverside
Architectural Press, 2010.
Barad, Karen. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and
Meaning. Durham, 2007. Bennett, Jane. Vibrant Matter. Duke University Press, 2010.
Berríos-Negrón, Luis. Breathtaking Greenhouse Parastructures: A Supplement to the Arcades
Project from a Caribbean Perspective [and a Call for a Careful Practice of Epistemológica].
Konstfack Collection, 2020.
Decroos, Bart. “How Gothic is Contemporary Architecture? The Appreciation of Craftmanship as a
Ruskinian Aesthetic of Imperfection.” In Thinking-Making: When Architects Engage in Construction
[Penser-Faire: Quand des Architectes se Mêlent de Construction], edited by Pauline Lefebvre,
Julie Neuwels and Jean-Philippe Possoz. Éditions de l'Université de Bruxelles, 2021. Dillon,
Brian. “Introduction: A Short History of Decay.” In Ruins, edited by Brian Dillon.
Whitechapel Gallery / The MIT Press, 2011.
Ferrando, Francesca. “Posthumanism, Transhumanism, Antihumanism, Metahumanism, and New
Materialisms.” Existenz 8, no. 2 (2013): 26-32. Haff, Peter. “Humans and Technology in
the Anthropocene: Six Rules.” The Anthropocene Review 1, no. 2 (2014): 126-136. Handa, Rumiko.
Allure of the Incomplete, Imperfect, and Impermanent: Designing and Appreciating Architecture As Nature.
Routledge, 2014.
Haraway, Donna Jeanne. The Companion Species Manifesto: Dogs, People, and Significant Otherness,
vol. 1. Prickly Paradigm Press Chicago, 2003.
Kiesler, Frederick. “On Correalism and Biotechnique: A Definition and Test of a New Approach to
Building Design.” Architectural Record 86 (1939): 60-75. MacCormack, Patricia. The
Ahuman Manifesto: Activism for the End of the Anthropocene. Bloomsbury Publishing, 2020. Menkman, Rosa.
The Glitch Moment(um). Institute of Network Cultures, 2011.
Partridge, Christopher. The Re-Enchantment of the West: Alternative Spiritualities, Sacralization,
Popular Culture and Occulture, vol. 1. London: T&T Clark International, 2004. Regardie, Israel.
The Tree of Life: A Study in Magic. Samuel Weiser, Inc., 1995.
Ronnberg, Ami and Martin, Kathleen. The Book of Symbols. Taschen Cologne, 2010. Tirosh-Samuelson,
Hava. “Engaging Transhumanism.” in H+/-: Transhumanism and its Critics, ed. Gregory R.
Hansell and William Grassie. Metanexus Institute, 2011.
Trischler, Helmuth and Will, Fabienne. “Technosphere, Technocene, and the History of Technology.”
Icon (London, England) 23 (2017): 1-17, issn: 1361-8113. Vass, Arpad A. “Beyond the Grave:
Understanding Human Decomposition.” Microbiology Today 28 (2001): 190-192. Whitehead, Derek H.
“Poiesis and Art-Making: A Way of Letting- Be.” Contemporary Aesthetics (Journal Archive)
1, no. 1 (2003). Wolfe, Cary. What is Posthumanism?, vol. 8. University of Minnesota Press,
2010.
Author Biography
Sebastian Gatz is an architect, artist and trained car mechanic, who works at the intersection of art,
architecture and technology. Currently he is doing a PhD in Fine Arts at Konstfack – University of Arts,
Crafts and Design in Stockholm. His research combines fictocritical and posthuman methods to explore
human-nature-technology relationships. He has an interest in experimental metaphysics, degrowth practices and
digital fabrication. Previously he worked and taught at The Royal Danish Academy of Fine Arts at the Centre
for Information Technology and Architecture (CITA) in Copenhagen, where he worked with Artificial
Intelligence, Robotic Fabrication and Robot-Plant-Hybrids.
Welcome to the Metaverse: Hacking Affect in Immersive Documentary to Increase Critical Big Data Literacy
BrettGaylor, KateHennessy
Simon Fraser University Vancouver, Canada brett_gaylor@sfu.ca, kate_hennessy@sfu.ca
Abstract
This essay describes the process and background of the augmented reality documentary Welcome To The
Metaverse. The authors situate themselves as research-creation practitioners and ethnographers exploring
immersive documentaries as a method for increasing critical big data literacy, an emerging
public-facing discipline concerned with pedagogical approaches to understanding power structures embedded in
artificial intelligence and big data systems. The authors respond to research from critical big data literacy
scholars and anthropologists using multimodal practices to foreground the ways that the digital tools used to
create these works need further critical reflection. The authors explore how facial recognition and other
aspects of augmented reality can be detourned or cultured-jammed in similar ways to media of previous decades,
positioning immersive documentary as a method for hacking affect toward greater awareness of the multifarious
politics of the Metaverse.
“Welcome! Blink if you agree to the terms and conditions!” an enthusiastic voice instructs
you as your face is scanned. You blink. Your eyes are replaced with the celebration emoji 🎉. Music
begins, and the same voice exhorts, “We’re so excited you’ve agreed to join The
Metaverse!”
These are the opening fifteen seconds of Welcome To The Metaverse (WttM),¹ an augmented reality
(AR) documentary that author Gaylor created for Instagram in collaboration with HoloLabs (figure 1).
For this experience to work, billions of research dollars and millions of faces have been collected to train
artificial intelligence systems that can detect the contours of your face and whether your eyes have closed.
Before this experience could make its way to your phone, lawsuits were filed and settled over whether companies
have the right to gather biometric data without consent (they do not).² While Gaylor and HoloLabs created
the documentary, the Facebook corporation announced a rebranding as “Meta” and a $10 billion
research agenda focused on the “metaverse,” a virtual environment blending the real world with
computer graphics.³ Welcome To The Metaverse satirically explores why Meta desperately
wants you and your data to succumb to The Metaverse.
This paper situates Welcome To The Metaverse as a work of research-creation⁴ in
cross-disciplinary conversation with artists, scholars and activists exploring how communications technologies
simultaneously reproduce unequal and oppressive relations of power and can be used to raise public awareness of
those dynamics. We walk readers through chapters of the AR documentary to demonstrate immersive documentary as a
medium uniquely situated to raise awareness around issues of privacy and surveillance capitalism⁵
that are at play in the Metaverse.
Building on Gaylor’s earlier interactive documentary works such as Do Not Track⁶ and
Discriminator,⁷ this paper positions Welcome To The Metaverse as a work of research-creation
that brings into practice a call by multimodal anthropologists for a new reflexive turn in the discipline that
engages with the material politics of the technologies used to create and exhibit their work.⁸,
⁹ This immersive AR documentary experience borrows tactics from culture jammers and media pranksters who
use the technologies they critique to demonstrate their problematic nature. Grounded in these traditions,
Welcome To The Metaverse aims to build critical big data literacy by alerting users to new ways that
their data can be extracted and exploited.
This essay highlights how Welcome To The Metaverse (WttM) operates as a counter-narrative
that uses storytelling, humour and interactivity to instigate critical reflection in users. Immersive
documentary as research-creation practice represents a promising methodology for raising critical data literacy.
As an approach to an emerging anthropology of the multimodal, we further present this work as an instance of
augmented ambivalent anthropology in practice.
Augmented Ambivalent Anthropology
Despite the hype, the Metaverse remains a speculative media whose qualities we divine from science fiction,
corporate videos, and early virtual and augmented reality works. Most people haven’t tried it, and most of
the experiences from SciFi and industry are still years away. The term “Metaverse” first appeared in
Neal Stephenson’s science fiction novel Snow Crash and described a virtual world that inhabitants
spend time in to escape the reality of failed states, ecological collapse and corporate rule. As the cleverly
named Hiro Protagonist declares: “When you live in a sh@!#t hole, there’s always the
Metaverse.”¹⁰
Our times of plague, war and climate catastrophe can feel like the future Stephenson dreamed up. During the
Covid-19 pandemic, Meta’s Quest headset sales increased 350%.¹¹ It can certainly be a welcome
diversion to visit the International Space Station, exercise at Machu Picchu, or experience a fantastical
immersive world. For creators, the creative possibilities of VR to transport audiences to new locations and
experience things that would be otherwise impossible are often enough to overcome the reluctance to support a
company such as Meta.
These mixed feelings are precisely the sort that anthropologists Astacio et al. encourage makers to embrace in
Multimodal Ambivalence: A Manifesto for Producing in S@!#t Times.⁹ The authors of this
manifesto ask makers and anthropologists who are invested in research mobilizing new multimodal tools to pair
their enthusiasm for new media with critical analysis of the political economies that underlie them. They ask us
to pay particular attention to digital productions, which are often valorized for their ability to facilitate
participation from under-represented communities in public discourse.
Multimodal ambivalence emerges in conversation with anthropologists such as co-author Hennessy’s
work with Takagawara et al. who argue that there is nothing inherently liberatory about the use of multimodal
tools in anthropology. Playfully drawing on Sarah Ahmed’s feminist critique of bad habits¹²
and Pierre Bourdieu’s habitus,¹³ they identify Bad Habitus as
the unpleasant feeling that unavoidable implication in ubiquitous big data environments and material
infrastructures causes for them⁸. Through this ambivalent orientation toward the tangled politics of
multimodal anthropology, they warn that uncritical use of digital technologies can reinforce racial inequality
and extractivism by normalizing the problematic power structures that digital infrastructures exacerbate, online
and off. They further point to research-creation as a productive methodology for an anthropology of the
multimodal that critically engages the tools being used for greater awareness of their wider politics and
impacts.
WttM is Gaylor’s attempt to satirically engage the public in a dialogue around the darker side
of the Metaverse. The narrator is ridiculously enthusiastic, while the actions he suggests (“scan your
body!”) feel dystopian. The documentary experience doesn’t aim to generate outrage—it seeks to
create mixed feelings and ambivalence. It does this by attempting to amuse users with interactive moments and
animations, paired with allusions to Meta’s attempts to gather increasingly detailed data about our homes
and bodies.
FaceJams & Semiotic Resistance
WttM explores the problematics of the Metaverse in one of the most realized versions of the Metaverse
available: Meta’s Instagram face filters. These filters will be recognizable to the billions of users of
Instagram, Tiktok, Messenger Kids, or Snapchat: a user’s visage is augmented by computer graphics that
track the position of their facial features.
WttM is novel in that the face filter changes to match the spoken narration. It operates as a hybrid
between linear media, such as film or radio, and interactive and immersive media. We will refer to WttM
as a “documentary experience” for this essay. By using affordances and tropes that feel
familiar to social media users, Gaylor attempts with this documentary experience to introduce messages and
conversations that they might not receive otherwise. In Umberto Eco’s terms, he is waging Semiotic
Guerrilla Warfare: subverting the communication “chain” by leaving the “channel”
intact (Instagram) and inserting into the “message” (the documentary) a different
“code.”¹⁴ This code is ambivalence—an unease about social media, tech platforms and
surveillance.
The goal of the project is to use the sense of unease that the documentary experience creates as a contribution
to the emerging field of Critical Big Data Literacy, a public-centred pedagogy focused on the growing
centrality of data and critical examinations of these repercussions. In her introduction to the field, Ina
Sander studied several online resources. She found that interactive creations were particularly suited to
transmit big data issues because the media could be personalized and would best compete for
attention.¹⁵ Welcome To The Metaverse is the first immersive documentary experience to
respond specifically to these findings and create new works with this knowledge in mind.
Contextual Integrity
“To create the Metaverse, we first collected millions of photos of our users on the vintage website
known as Facebook.”, the narrator declares as the next chapter begins. “We used these
faces to create an algorithm that can detect yours!”
This narration refers to practices that Facebook has engaged in to train artificial intelligence systems. In
2021, residents of Illinois were awarded $650 million to settle a class action against Facebook, which had
trained a facial recognition algorithm using their photos without consent. ² In addition to violating
Illinois’ recently passed privacy laws, Facebook transgressed what scholar Helen Nissenbaum refers to as
contextual integrity.¹⁶ This framework judges whether information sharing is appropriate
based on “the type of information in question, about whom it is, by whom and to whom it is transmitted,
and conditions or constraints under which this transmission takes place” (2004, 839).
This is essential nuance: while many users will describe uses of their data as
“creepy,”¹⁷ a fuller description of inappropriate data flows is necessary to hold tech
platforms accountable. Contextual Integrity recognizes that we might be willing to share certain information
in specific contexts: we would not like intimate photos texted to a partner to be shared with our boss, for
example. WttM users experience this firsthand when they see pictures of someone else's face, stolen
from Facebook, on their own face (figure 3).
It is this notion of a violation of contextual integrity that the creators hoped WttM would impart,
and where the work builds on and at the same time departs from subversive efforts in other media. The goals
of WttM parallel Eco’s hope for “the constant correction of perspectives, the checking of
codes, the ever-renewed interpretations of mass messages.”¹⁴ So, too, did those practicing
critical remix on the media of the time—the so-called “culture jammers” of the 1980s and 90s
who inserted counterculture messaging into billboards, radio and television broadcasts and print
media.¹⁸ These media, however, had fundamentally different characteristics than our present and
future digital media. Consider an advertisement on a traditional billboard, a “one-to-many” media,
in which a single message passes through a channel and is received in an identical fashion by audiences.
Contrast this with an advertisement on Instagram, where the ads are chosen based on monitoring behaviour on the
platform and predicting which messaging users are likely to respond to. Each will be viewed on a specific
device, in a different location, and in specific language.
In the Metaverse, this trend is magnified—everything about your context, from your physical
characteristics to location to your species, can be customized by you and personalized by advertisers. WttM
applies principals of culture jamming to your specific digital context—what we call context
jamming. This approach follows recommendations by Ina Sander What Is Big Data Literacy to
increase the use of personalization in education resources, an approach well suited to networked
documentaries such as WttM in which users are already using their own devices and are logged in
with their own Facebook/Meta accounts.
“Thanks to the pioneering work on the Facebook timeline, data scientists now understand content that
generates strong emotions keeps you engaged longer!”, the narrator says to advance the next chapter
of WttM. This refers to research done by Facebook engineers¹⁹ who manipulated the type of
content users would receive in their timeline to understand whether prolonged exposure to posts with strong
emotions would lead to more engagement on the platform (it does). Here again the work creates a scenario where
contextual norms have been violated: it is unlikely that anyone sharing an emotional Facebook post imagined that
a data scientist would use it to measure “emotional contagion.”
Both this story of manipulation, and the use of photos without consent, are both real stories of harm by
Facebook/Meta. Having real, relatable accounts such as this was also a recommendation drawn from Ina
Sander’s research, and the purpose of these stories in WttM is to cultivate ambivalence, to
acknowledge our bad habitus, to incite curiosity and seed doubt as to whether these technologies are operating
in the best interest of users.
Hacked Affect
In the next chapter of WttM, viewers are presented with text describing how much money Meta CEO Mark
Zuckerberg has made in the few minutes that the documentary has been playing. The narrator explains what is
happening: “Here is how much money Mark Zuckerberg made while you were watching
this!”
An analogue meter appears over the user's face. By offering emotions as programming inputs (figure 2), Meta is
attempting to normalize the concept of emotions as a universally consistent and measurable data point. This
practice is known as affect recognition. As Crawford points out, facial recognition seeks to recognize
individual faces, affect recognition aims to identify universal emotions on any face.²⁰
Affect recognition is currently deployed across many industries for a variety of uses, from monitoring
emotional engagement during job interviews²¹ to assessing student reactions during
lectures²² to detecting nervous terrorists in airports.²³ These capabilities are sometimes
developed internally by companies or purchased as a service from companies. One such company is Affectiva, an
MIT incubated startup that uses deep learning to offer emotional insights gleaned from training an AI on the
expressions of 10 million people in 87 countries.²⁴ If a basic set of emotions is all a developer
requires, affect detection is available in most standard facial recognition suites such as Rekognition by
Amazon²⁵ or Face API by Microsoft.²⁶
With WttM, we are subverting Meta’s attempts at normalizing this practice by introducing
skepticism as to the ability of software to detect emotion. While Meta presents affect detection to developers
as a trustable input on the same level as a mouse click or text entry, recognizing emotions by studying the face
is, in fact, a controversial practice. Indeed, the very epistemological basis of associating affect with facial
expressions, developed by Paul Ekman in the 1960s, has been challenged by psychologists and anthropologists as
being methodologically unsound.²⁷
This critique is based on Ekman establishing six emotion types: joy, anger, disgust, sadness, surprise, and
fear, which he developed by photographing posed actors displaying caricatures of these emotions, claiming they
were universal, and subsequently measuring test subjects against them (figure 3).
Anthropologist Ruth Leys notes that the central flaw in Ekman’s methodology is its recursive logic: the
photographs are assumed to be universal because they are free of cultural bias and culturally unbiased due to
their universality.²⁸ Problems of this nature, where erroneous data lead to incorrect assumptions,
are perfect candidates for artificial intelligence systems to make worse. Yet despite this, engineers began
giving the task of comparing a subject's facial expression to photos in datasets such as FEEST (figure 7) to
speed up and automate the process. As Kate Crawford notes in Atlas of AI, recognizing affect is a task
that has been given to AI systems not because they are suited to the job but because the (weak) theory was
suitable for what the tools could do—detect patterns. ²⁰
These systems are deployed on a global scale, such as the Screening of Passengers by Observation Techniques
(SPOT) program of the US Transportation Security Administration, which uses Ekman’s techniques to
“detect” nervous flyers and flag them as potential terrorists.²³ Meanwhile, a
comprehensive study of peer-reviewed science on affect recognition found no evidence that algorithms can detect
a person's internal emotions. The research team warned, “very little is known about how and why certain
facial movements express instances of emotion, particularly at a level of detail sufficient for such conclusions
to be used in important, real-world applications.” ²⁸
“Show us you’re happy and we’ll move on to the next chapter!” the narrator
asks at the end of the sequence. WttM asks users at this point to perform what researchers have
proven—an inner emotional state cannot be inferred by facial expression. After learning how much Mark
Zuckerberg made in the minute or so of the experience (approximately $13,000), a user is more likely to be
annoyed than happy. Yet they must smile to continue. Smile hard, in fact—the program intentionally delays
the measurement of their smile to give the impression that they must smile more.
“Try harder to be happy!” the narrator implores and rewards them with pretend
cryptocurrency once the smile registers. This immediate feedback from the system after having their affect
recorded follows Ina Sander’s recommendation that critical big data literacy resources should be
interactive—users should be required to make choices within the experience that influence the
outcome.¹⁵ Despite extensive literature around affect detection, there is a paucity of public-facing
contributions that aim to illuminate its origins and problematics. Rather than a simple exhortation for users to
protect their data from thieves, as many simplistic public service messages around privacy suggest, WttM
is introducing a feeling of unease around the implications of having their emotions recorded by an entity
they may not trust.
Asymmetrical Cognitive Ammunition
Welcome To The Metaverse, as a contribution to the emerging field of critical big data literacy, aims
to plant doubtful seeds in users’ conceptions of the Metaverse before it is fully realized. The work
intends to call out the Metaverse as what Langdon Winner would call an Inherently Political
Technology.²⁹ Winner casts technologies of this type as those that require specific
political relationships to function: a ship that needs a top-down structure where the crew obeys the captain's
commands, for example. The Metaverse, as conceptualized by Meta, requires exploitation: of the data of
users whose bodies and environments are digitized, of the labour of workers who mine the minerals that are
necessary for the hardware, and of the resources of the planet which must be marshalled for the energy to power
the cloud computing that keeps the Metaverse running.
None of this exploitation is apparent when using the early prototypes of the Metaverse. The brightly coloured
landscapes and diverting amusements offer no hint of the materials or labour marshalled to bring each virtual
moment to life. Digital environments were not conceived in a manner that would make their processes
transparent.³⁰ This makes efforts to crack open the “black box” with public-facing works
such as WttM all the more urgent. As Vladan Joler and Kate Crawford remind us, the stakes of digital
exploitation are vast:
“The scope is overwhelming: from indentured labour in mines for extracting the minerals that form the
physical basis of information technologies; to the work of strictly controlled and sometimes dangerous hardware
manufacturing and assembly processes in Chinese factories; to exploited outsourced cognitive workers in
developing countries labelling AI training data sets; to the informal physical workers cleaning up toxic waste
dumps.”³¹
As creators, we feel that the most effective way to critique these problematic systems is to have users
experience them within a new context. Yet this brings its own challenges: a tactical shortfall of semiotic
guerrilla warfare is that the enemy owns the battlefield. This is true of all mass media: radio, television and
Instagram are all difficult to hack because a layer of permission is applied before a message can appear on the
platform. Yet to increase the literacies related to contextual integrity that we have highlighted in this essay,
we we’re obliged to use Facebook/Meta as both the channel and message. We could not take a picture of our
critique and transmit the message as we could with a billboard culture jam. While creating Welcome To The
Metaverse, we witnessed firsthand the limits of critiquing within such a vertically integrated
communications system.
SparkAR, created and owned by Meta, facilitates the uploading of augmented reality face filters to its
Instagram and Facebook platforms directly from within the app. There is no alternative distribution venue.
Unlike the World Wide Web, this ecosystem is known as a “walled garden” and requires that each piece
of content made available be screened and approved before appearing in listings and searches.
Upon submitting Welcome To The Metaverse, our team waited 24 hours, after which we received a notice
that the project had been rejected. The notice stated that we had violated Policy 3.7 from the SparkAR policy by
using a trademarked asset or colour gradient. There was no indication of which part of the experience violated
the policy, but the team was confident that the violation was caused by using the “like” button. The
narrator asked users to “tap the like button to continue” and displayed the blue thumbs-up
image.³² This image, and apparently the blue colour scheme, are trademarked by Facebook.
Our team replaced the like button with a heart, an image which Meta does not (yet) own, and resubmitted the
project. Another 24 hours later, the filter was accepted and available to the hundreds of millions of users of
Instagram and Facebook.
It is ironic that this image, which we found easy to replace, was censored. After all, the entire piece is a
direct criticism of the Meta corporation. Our hunch, which we will never be able to confirm, is that our review
was undertaken either by an algorithm or an outsourced temporary worker. Neither appears suited to detect
satire, but both are another example of the extractive stack that the Metaverse is being built upon. Scholars
have noted how content moderation algorithms reinforce white supremacy as they make no distinction between
critiques of whiteness and racial violence.³³ Human-based content moderation requires the exploitation
of hundreds of thousands of workers around the globe working precarious jobs and being exposed to traumatizing
images.³⁴
Our team published a work that could easily be removed. Just as a McDonald's billboard spray-painted with vegan
messaging would likely be taken down once the corporation or ad network became aware of the intervention, our
project is likely ephemeral. Where it ever to be successful and brought to the attention of a Meta employee, it
would likely be de-platformed.
Conclusion
By creating Welcome To The Metaverse, we enacted a promising approach to engage users of Meta using
the company's own technology, introducing ambivalence surrounding the company’s attempts to collect more
of their data. By introducing the novel concept of context jamming, the documentary experience hopes to
build critical big data literacy while hacking affect and encouraging the public to interrogate the power
structures that underlie digital technologies.
This research-creation approach holds promise for those wishing to subvert and detourn extractive platforms,
yet it also points out the control these platforms can exert to prevent the cultivation of doubt about their
intentions. This is why critiques such as WttM must be followed, or potentially supplanted, by
alternative visions on new platforms. The Metaverse of Meta is only one manifestation of a networked world, and
virtual communities that uphold humanist values are waiting to be explored. As Langdon Winner reminds us,
“it can happen that within a particular complex of technology... some aspects may be flexible in their
possibilities for society, while other aspects may be (for better or worse) completely intractable.”(1999,
135)²⁹ A non-extractive Metaverse is possible.
If creative resistance to the Metaverse of Meta is to take hold, producers and users of the medium need to
imagine what alternatives could look like and distribute these speculative imaginings on platforms of their own
creation. Encouraging yesterday’s culture jammers to manifest a more healthy media landscape, advertising
hacker Stuart Ewen asked them to lead by example. “If our critique of commodity culture points to better
alternatives, let us explore—in our own billboards of the future—what they might
be.”³⁵
References
1 Brett Gaylor, Welcome To The Metaverse, Augmented Reality Filter, February 2022,
https://www.hololabs.org/welcome-to-the-metaverse
2 Natasha Singer, Mike Isaac, “Facebook to Pay $550 Million to Settle Facial Recognition
Suit", The New York Times, Retrieved March 16, 2022,
https://www.nytimes.com/2020/01/29/technology/facebook-privacy-lawsuit-earnings.html
3 Mike Isaac, “Facebook Renames Itself Meta”, The New York Times, Retrieved
April 12, 2022, https://www.ny- times.com/2021/10/28/technology/facebook-meta-name-change.html
4 Nathalie Loveless, How to Make Art at the End of the World; A Manifesto for
Research-Creation, Duke University Press, 2019.
5 Shoshana Zuboff, “Surveillance Capitalism and the Challenge of Collective Action”,
New Labor Forum 28, January 2019, 10–29, DOI:https://doi.org/10.1177/1095796018819461
6 B. Gaylor, Do Not Track. [Interactive Documentary], 2015,
https://www.donottrack-doc.com
7 B. Gaylor, Discriminator. [Interactive Documentary], 2021,
https://www.discriminator.film
8 Stephanie Takaragawa et al. “Bad Habitus: Anthropology in the Age of the
Multimodal”, American Anthropologist, vol. 121, June, 2019: 2, 517–524
9 Alvarez Astacio et al. Multimodal Ambivalence: A Manifesto for Producing in S@!#t
Times, American Anthropologist, 123, 2021 :2, 420–427
11 L. E. Editor Consumer Affairs, “Virtual reality: pandemic leads to rise in headset sales
to escape lockdown”, Retrieved Mar. 16, 2022,
https://www.thetimes.co.uk/article/virtual-reality-pandemic-leads-to-rise-in-headset-sales-to-escape-lockdown-jhhn8wghn
12 Sarah Ahmed, A Phenomenology of Whiteness, Feminist Theory, 8, 2007, 2,
149–168.
13 Pierre Bourdieu, Distinction: A Social Critique on the Judgement of
Taste, Chicago: University of Chicago, 1988.
14 Umberto Eco, “Towards a Semiological Guerrilla Warfare”, in Travels In
Hyperreality, Gruppo Editoriale Fabbri Bompiani, 1983.
15 Ina Sander, “What is critical big data literacy and how can it be
implemented?” Internet Policy Review, vol. 9, May 2020,:2.
16 Helen Nissenbaum, H. Privacy as contextual integrity, Washington Law Review,
79, 2004, :1, 119–15,
17 Americans’ opinions on privacy and information sharing. Pew Research Center:
Internet, Science & Tech. Retrieved March 16, 2022,
https://www.pewresearch.org/internet/2016/01/14/privacy-and-information-sharing/
18 Mark Dery, Hacking Slashing, and Sniping in the Empire of Signs. In M. DeLaure
(Ed.), Culture Jamming: Activism and the Art of Cultural Resistance, NYU Press, 2017.
19 Adam D. I. Kramer et al. Experimental evidence of massive-scale emotional
contagion through social networks, Proc. Natl. Acad. Sci. U.S.A, 111, June 2014, 24, 8788–8790.
20 Kate Crawford, Atlas of AI, Yale University Press, 2021.
21 Javier Sánchez-Monedero, Lina Dencik, “The Datafication of the Workplace”,
Working paper, Data Justice Lab, Cardiff University, May 9, 2019.
22 G Tonguç and Ozaydın Ozkara B. “Automatic recognition of student emotions from
facial expressions during a lecture”, Computers and education, 2020, 148.
23 Ashley Halsey III, “House Member Questions $900 Million TSA ‘SPOT’
Screening Program”, Washington Post, November 14, 2013, Retrieved December 7th, 2022,
https://www.washingtonpost.com/local/trafficandcommuting/house-member-questions-900-million-tsa-spot-screening-
program/2013/11/14/ad194cfe-4d5c-11e3-be6b-d3d28122e6d4_story.html.
24 Affectiva Human Perception AI Analyzes Complex Human States, Affectiva,
https://www.affectiva.com/.
25 Welcome - Amazon Rekognition, Retrieved April 12, 2022,
https://docs.aws.amazon.com/rekognition/latest/APIReference/Welcome.html
26 Facial Recognition | Microsoft Azure, Retrieved April 12, 2022,
https://azure.microsoft.com/en-us/services/cognitive-services/face/
27 Ruth Leys, The ascent of affect: genealogy and critique, The University
of Chicago Press, 2017.
28 Adolphs Barrett et al. “Emotional Expressions Reconsidered: Challenges to
Inferring Emotion From Human Facial Movements” in Psychological Science in the Public Interest,
20, 2019:1, 1–68.
29 Langdon Winner, Do Artifacts Have Politics? In The Social Shaping of
Technology (2nd. ed.), Donald MacKenzie,Judy Wajcman (eds.), Open University Press, 1999, 28-40.
30 Mark Graham, Håvard Haarstad, “Transparency and Development: Ethical Consumption
through Web 2.0 and the Internet of Things,” in Information Technologies & International
Development 7, no. 1, March 10, 2011,: 1.
31 Kate Crawford, Vladan Joler, “Anatomy of an AI System”, Retrieved March 31,
2022, http://www.anatomyof.ai
32 Facebook like button, Wikipedia, Retrieved April 12, 2022,
https://en.wikipedia.org/wiki/Facebook_like_button
33 E. Siapera, “AI Content Moderation, Racism and (de)Coloniality”, Int Journal of
Bullying Prevention, vol. 4, Mar. 2022,:1, 55–65.
34 S.T. Roberts, Behind the screen: Content moderation in the shadows of social
media, Yale University Press, 2019.
35 Stuart Ewen, Elizabeth Ewen, Channels of Desire: Mass Images and the Shaping of
American Consciousness, McGraw-Hill, 1976.
Hybrid Entanglements: a posthuman dramaturgy for human-robot relationships
PetraGemeinboeck1, RobSaunders2
1Centre for Transformative Media Technologies, Swinburne University of Technology
(Melbourne, Australia) & Media Theory, University of Applied Arts Vienna (Vienna, Austria)
2Leiden Institute of Advanced Computer Science, Leiden University (Leiden, The
Netherlands) pgemeinboeck@swin.edu.au, r.saunders@liacs.leidenuniv.nl
Abstract
This paper discusses our collaborative Machine Movement Lab project harnessing movement to bodily empathize
with abstract machines. Bringing together creative robotics, choreographic strategies, and a posthuman
dramaturgical frame, the project seeks to trouble our relationships with robots by exploring them as
more-than-human entanglements. The paper discusses our transdisciplinary performance-making practice and
underlying theoretical concepts and how they are mobilized through emerging diffraction patterns mapping out
symbiotic relationships. An improvisational score involving dancers, robot costumes and robots performed in a
gallery space aims to engage audiences with hybrid human-machine entanglements in embodied and empathic ways.
Hybridity is predicated upon difference—we can only recognize something as hybrid and symbiotic if we
acknowledge and recognize the potential of difference. This paper seeks to bring to the fore the aesthetic and
social potential of difference in our relationships with machines. It attempts to trouble practices in
human-robot interaction that, like many human practices, are invested in deliberately masking difference,
grounded in hierarchical and hegemonic beliefs. Stuck in what Barad ¹ referred to as the
“representationalist trap” of reflection, we look for and fabricate resemblances between what are,
essentially, deeply asymmetric entities.² Many of our current human-robot imaginaries thus echo or reaffirm
the conservative narratives that validate existing social norms. Yet how we imagine social machines and the
future narratives they are embedded is not only a matter of appearance, but literally matters—socially,
politically, and ethically. Machines with humanlike facades, for instance, are often presented as more familiar
and friendly; but they also serve to confine both bodies and things in mimicry and servitude.³,
⁴, ⁵
Our Machine Movement Lab (MML) project attempts to counter this reflection-centered approach by developing a
diffractive practice, which foregrounds and aesthetically exploits the differences between humans and machines.
MML thus seeks to trouble our relationships with robots that manifest from reductive desires to render the
machine as humanlike as possible by investigating creative strategies for reimagining and reconfiguring our
relationships with them. This paper focuses on our latest research stage, which draws on Donna Haraway’s
⁶ and Karen Barad’s ¹ new materialist conception of diffraction to explore the potential
of performance-making and posthuman dramaturgy for entangling humans and machines. With the latter we seek to
open up ontological boundaries, such as the one delineating subjects and objects, and to reconfigure them or
render them porous, the bodily-material way. We believe that such reconfigurings challenge the limited,
humancentric ways in which we envision our robotic futures by expanding our bodily ways of knowing and becoming
more attentive to the performative potential of this hybrid, more-than-human encounter.
We begin with providing a brief overview of the practices within which our work is situated, along with some
key artists whose work has influenced our practice. Following, we introduce our MML project and how it harnesses
the generative potential of movement in tandem with dancers’ kinesthetic expertise to become-with and
design abstract machine artifacts. We then take a closer look at our performance-making approach and posthuman
dramaturgical framing. Looking at the making of human-robot relationships as a more-than-human entanglement, we
outline the feminist concepts that our new materialist practice draws on and seeks to mobilize. Finally, we
discuss the making of an improvisational performance score, arising from our experimental studio practice, and
how it aims to facilitate the engagement of audiences in embodied and empathic ways.
Situating our Practice
Looking at our relationships with robots from a performance perspective highlights their embodied,
socio-cultural, material and, sometimes, codependent nature. We situate our transdisciplinary practice across
the practices of machine performance, kinetic sculpture, and robotic art that experiment with movement and its
capacity to evoke affective relationships between bodies and things. Artists have long deployed performance
concepts to create ‘living’ sculptures or machine performances that both critically and playfully
explore intimate couplings between human and machine bodies. Marco Donnarumma, for instance, seeks to highlight
the co-dependence of hybrid (human-machine) embodiments rather than a “pairing of two different
things.”⁷
Jean Tinguely’s early kinetic sculptures induce a sense of creative machine spirit,⁸ and Robert
Breer’s slowly moving Floats used motorized wheels to gradually rearrange themselves in space,
and thus, almost unnoticeably, reconfigure space.⁹ More recently, The Table: Childhood
(1984–2001) by Max Dean, Raffaello D'Andrea and Matt Donovan produces surprising relational
dynamics between audience members and the familiar object of a table.¹⁰ Kris Verdonck’s
Dancer #3 ¹¹ performs the energetic clumsiness of an optimistic clown in empathically
accessible yet distinctly machinic ways.
State Grace Machines by Bill Vorn, Emma Howes and Jonathan Villeneuve explores questions of
kinaesthesis and perception in a dialogue between abstract machine performers and a dancer.¹² Eve
of Dust, a collaboration between John McCormick, Adam Nash and Stephanie Hutchison, investigates
possibilities of physical collaboration and cocreation between a human dancer and a robot arm ¹³.
Louis-Philippe Demers’ performance work The Tiller Girls foregrounds the whimsey and
vulnerability of machine bodies;¹⁴ in line with Paula Gaetano Adi’s poetic embodied
entanglements, such has produced by her works Becoming With and
Alexitimia,¹⁵promoting the social presence of machines and strange affective
capacity of abstract machines.
All these works generate their own dramaturgical frame for exploring the social capacity of non-humanlike
machines and complicating our relationships with them; thus, expanding our understanding of how we relate to
machines.
Machine Movement Lab (in a nutshell)
Our Machine Movement Lab (MML) project is a collaboration with dancers, choreographers, AI researchers,
engineers, and numerous materials (from cardboard, PVC tubes, plywood to aluminum framing, motors, motor
controllers, cables, cable binders, and software programs), across robotics labs, dance studios, fab labs, and
gallery spaces over the past seven years. MML harnesses the generative potential of movement and its dynamic
qualities to explore the aesthetics of entangling and empathy in human-robot
encounters.¹⁴, ³ Rather than human-or animal-like, our robots are abstract,
machinelike artefacts, forged from a practice of becoming entangled with the machine morphology and its unique,
more-than-human capacities. Our latest research stage is concerned with performance-based inquiries into
posthuman, transcorporeal reconfigurings and their potential to expand our possible relationships with abstract,
machinelike robots.
Movement as a generative, relational force MML regards movement as a phenomenon or force,
capable to make bodies, meanings, and relationships. This contrasts much of the current robotics research where
movement is understood as a means of navigation or imbuing an object with a predefined personality. The
difference between looking at movement as a productive force rather than an instrument is significant because it
allows us to become-with what it generates—its enacted relations, specific to this situation, rather than
using it to generate what we already know. This notion of movement mattering, bodying-forth¹⁶ and
relation-making, opens-up seemingly limitless opportunities for entangling with more-than-human artefacts.³
Relational-Body-Mapping (RBM)
Our MML practice revolves around the idea that the kinds of relation-making that movement propels happen in the
dynamics of encounter and unfold through “spatial, temporal, and energic qualities.”¹⁷
This is where meanings and affects get made and distributed across human and nonhuman bodies, rather than being
predefined and preformed by certain beliefs about what this more-than-human relationship should be.
Our diffractive approach aims to harness movement’s generative force by enacting situations of close,
corporeal encounter that can open-up kinaesthetic experiences of becoming-with the machine artefact and its
unique material qualities. In practice, this involves getting entangled with material props, whose material
qualities can offer us a corporeal glimpse of the machine’s more-than-human relational possibilities. To
enable this becoming-with (i.e., entangling), we ask dance performers to extend themselves into, inhabit, or
wrap around a wearable costume whose shape and size resembles that of the robot (see Figures 1 and 2). The
costume thus stands in for a becoming-robot design, at the early stages of the design process, and enables
dancers to feel into the robot’s material-spatial potential ³ as well as the robot’s
sensorium (equipped with the becoming-robot’s sensors).
This more-than-human entanglement, which we will refer to as performer-costume in the following
sections, allows us to experiment with and corporeally probe into a range of human-machine configurings. Our
Relational-Body-Mapping (RBM) approach builds on our Performative-Body-Mapping (PBM) method, which focused on
single performer-cube entanglements and movement creation.¹⁸ RBM expands PBM to seek more complex,
nested entanglings and the transcorporeal resonances they can effect, e.g., a (human) performer with a (robotic)
cube performer; a performer-costume with a cube performer; or a (human) performer with a performer-costume and a
cube performer (see Figure 1), and so on.
Cube Performer
The robot costume not only allows the dancers to ‘feel into’ the differences of the machinic
embodiment but also to capture the kinetic dynamics that unfold in this more-than-human entanglement. The hybrid
motion data, arising from this human-nonhuman enmeshment, informs the robot’s machine learning process,
where the machine learns to improvise movements based on its own mechanical embodiment and the patterns it
derives from our entangled motion data.¹⁴
Our first robot prototype––the cube performer (see Figure 2)—resulted from a series of
corporeal entanglements with a wide range of materials. It is a simple box-shaped artefact, which is transformed
by its dynamic movements: suddenly tilting up along one of its edges and gently swaying or thumping onto the
ground, the box quickly loses its rootedness and becomes more-than-object ¹⁴. The robots’
mechanical design was derived from an extensive analysis of motion capture recordings of
the performer-costume and the relational motion patterns it produces ¹⁹. Instead of
relying on googly eyes or pre-packaged personality, the robot cube becomes a performer based on the enactive
potential of its movement dynamics [see ²⁰, ¹⁷] and how they can co-shape a meaningful
encounter.
A more detailed discussion of our performance-based, embodied robot design stage can be found in ³,
¹⁴, ¹⁸, ¹⁹.
Diffractive Performance-Making
Our diffractive performance-making practice investigates how corporeal entanglements with machine artefacts and
their different material-spatial and affective qualities can open-up modes of transcorporeal empathy. The
latter, we believe, is key to meaning making with social machines without relying on fake emotional facades
(i.e., a humanlike face).
Robotics practices, in general, often look at humans and machines as two separate, already given or predefined
entities (i.e., subject and object). MML, in contrast, attends to how subjects and objects are mutually
constituted² by investigating the making of subject-object boundaries as a nested entanglement. Meaning
making here is about carefully attending to the possibilities for relations and meanings to emerge.
According to Jon Lee, the alternative landscape of a diffractive dramaturgy is experimental and experiential,
“where we feel for and towards (in a tentacular way) a collaborative making process that tilts the optic
away from traditional expectations.”²² Our diffractive, posthuman dramaturgy generates an
experimental and experiential space, where we feel for possible entanglings, tentacular capacities and hybrid
configurings of human performers and nonhuman artefacts. It involves carefully probing into how they matter,
couple, interfere, and “undo and redo each other,”²³ and how this difference-in-relation
gives rise to transcorporeal meaning-making.
The following outlines some of the core theoretical concepts that our performance-making practice draws on, and
then discusses some of the most significant interference patterns whose emergence we have witnessed thus far.
Diffracting Subjects and Objects
Our posthuman dramaturgical approach attends to and aesthetically puts to work difference-in-relation (i.e.,
humans and machines entangled) by seeking to materially mobilize Haraway’s⁶ and Barad’s ¹
concept of diffraction. In contrast to reflection (i.e., rendering machines humanlike), diffraction maps
interferences¹⁴ and as such “attends to the relational nature of
difference.”²⁴
A diffractive practice embraces and foregrounds differences by attending to the specificity and materiality of
entanglements.¹ Diffraction thus not only serves as a figurative lens but can shape a material process,
i.e., in our practice the dramaturgical/choreographic methods of interfering, superposing and entangling bodies
and things.
From a posthuman perspective, we always already are entangled with the world and its ongoing
reconfigurings.²⁵, ¹ Barad’s notion of a posthumanist performativity calls
“into question the givenness of the differential categories of ‘human’ and
‘nonhuman’, examining the practices through which these differential boundaries are stabilized and
destabilized.” ²⁶ Diffraction as both a tool and a practice can make manifest the
destabilization and stabilization of boundaries.²⁷
In MML, we are particularly interested in the boundary-making that both separates and defines subjects and
objects. How can we intermesh (given) subjects and objects, probe into their boundaries and render them more
porous or create new hybrid entities? Rather than juxtaposing humans and machines or making them appear to be
the same, we seek symbiotic possibilities based on difference patterns that render the boundaries between
subjects and objects more elastic. Diffraction and patterns of interference thus become a methodological tool
for “attending to and responding to the effects of difference” [¹⁷: ⁷²].
The entanglement of bodies and things maps their effects of difference similarly to Barad’s description of
superposition:
“... waves can overlap at the same point in space. When this happens, their amplitudes combine to form a
composite wave form [and] the resultant wave is a sum of the effects of each individual component wave; that is,
it is a combination of the disturbances created by each wave individually. This way of combining effects is
called superposition”. ²⁸.
A posthuman dramaturgy for diffracting subjects and objects thus troubles engrained dichotomies and, instead,
traces the effects of differences that give rise to new forms of more-than-human meaning-making—in MML, a
trans-corporeal form of meaning-and experience-making, which we will look at in more detail below.
More-than-human Interference Patterns
Superposing human bodies and cubic things, in practice, requires ongoing attunement to the becoming of bodies
and, with it, emerging agencies and differing identities—a moving with and continuous gesturing toward the
more-than-human space of a ‘thing’—the process of becoming-thing. The empathic resonances
brought about by this superposition can be described as a bodying-thinging.⁶ Transcorporeal
bodying-thinging is about how bodies and things resonate whilst undoing and redoing each other; at once tracing
how subjects and objects constitute each other and at the same time rendering their boundaries elastic.³ It
attests to the inherent porosity, relationality and reconfigurability of bodies and things, how they already
always extend toward and across each other.
The following explores three of the most significant interference patterns that we have observed thus far and
how they mobilize transcorporeal resonances of bodying-thinging. They come about based on different degrees of
entanglement, the number of entangled bodies and things, and the emergent effects of ongoing
reconfigurings.
Pattern #1: Spatial Superposition, Becoming-with
This interference pattern manifests from the dancers corporeally exploring their entangledness with the cube by
bodily listening to its material characteristics and capabilities and the cube responding (talking back) by
producing different material sensations (its weight, how it bends, where it resists, etc.). Becoming-with (see
²⁰) the cube then involves dancers reconfiguring their bodies as well as letting themselves being
shaped by these nonhuman qualities and to feel-think-move-with the cube (see Figure 3). Sometimes the two simply
interfere, other times they are in-phase, become-with, and are bodying-thinging with each other.
Pattern #2: Stretching the Boundary between Subject and Object
We found that dwelling on the edge (i.e., the subject-object boundary, which may also align with an edge of the
cube) and feeling into it, stretches and carefully opens-up the boundary in-between subjects and objects. The
threshold of the boundary becomes a zone to linger, to extend into or be extended by, to become familiar, to
mingle with (see Figure 4). It is the most symbiotic cube-performer interference pattern with regards to its
resulting shape and the entanglement’s tentacular capacity (see following section), where body-thing can
no longer be separated, nor is one entirely folded into the other. Rather than a barrier, the boundary becomes
an access zone—a gateway to bodying-thinging and exploring the symbiotic affordances of this hybrid
performer-costume entity.
Performance-making involving more than one performer and one costume produces a nesting of difference patterns
and, with it, the affects that flow across the open seams of each pattern. The nested entanglings unfold in a
continual process of attachments and detachments, e.g., the dancers attaching themselves to a corner of the
costume, a corner of the space, or to a corner of the other costume, even if only for a glimpse, even with only
the tip of the toe (see Figure 5); then detaching again—from the corners, one by one or all at once, to
reattach and align with an edge, or a plane, or the other dancer’s gaze. These re-/alignments open-up
spaces to link/mesh/interweave with other boundary spaces, stretching and extending the lines of the cube to
reach into or meet other lines, and performer-costumes become tentacular and intermesh; bodying-thinging here
also means to grow tentacles. Haraway speaks of “tentacular ones [and how they] make attachments and
detachments; they make cuts and knots; they make a difference; they weave paths and consequences but not
determinisms; they are both open and knotted in some ways and not others.”²⁹
All three of these symbiotic difference patterns result in movements and dynamic constellations that are
irreversibly hybrid: The dancer’s body is reconfigured by the costume, and the movements captured with the
costume reconfigures the movements learned by the robot.⁶, ¹⁸ And when
performer-costume and cube performer (robotic artefact) entangle and become tentacular, new motion patterns
evolve— movements that neither belong to the machine nor the performers-in-costume.
In the following we explore how our improvisational score builds on these interference patterns and unfolds
them as a series of experiential scenarios, each performance anew.
Scoring an Improvisational Performance
Dancing with the Nonhuman is a roughly 20-min performance work, to be performed in gallery spaces
rather than a separate stage. Its underlying semi-structured, improvisational score seeks to open-up our
diffractive process to the diverse embodied perspectives of audiences by performing human-nonhuman interference
patterns and the transcorporeal attunement they produce–each iteration anew.
Arising from our experimental studio practice and observed, emergent-diffractive patterns, the underlying
improvisational score shapes different ‘lenses’ through which the experiential scenarios of
human-nonhuman entanglement unfold. The following outlines the four lenses that propel Dancing with the
Nonhuman [SYD-2-2-1] and how they mobilise differently hybrid and tentacular configurings.
In (1) ‘phantom’, we witness a series of movements shaped by the dancers’
cubic entanglement but reperformed without the cube costume. The performance thus opens with a kind of puzzle as
these movements clearly belong to a realm that is both more-than-human and more-than-object.
In (2) ‘threshold’, dancers feel their way along the boundaries of the cube
costume, extend them, entangle with them, and render them elastic; meanwhile the cube performer slowly glides
along straight lines, occasionally beginning to twitch out of the grid.
In (3) ‘con-current’, we witness the dance performers fully inhabiting their cube
costumes. The encounter between cube performer and performers-in-cube appears seamless and interferences express
themselves along geometric lines. In (4) ‘co-play’, the encounter becomes a
playground, and it gets a bit messy, bodies and things tumble. And so do their boundaries.
Audiences and transcorporeal empathy
At the time of writing, we are yet to perform this work in public. Importantly, audiences are not expected to
decipher any of these patterns or lenses. The aim is for them to engage with these alternate, posthuman
human-machine configurations not only by looking but also by transcorporeally empathizing with them, based on
their own corporeal experiences with tentacular, more-than-human configurations.
As we strive to collapse the distance between subjects and objects, we also seek to render the boundary between
performers and audiences more porous. To avoid the distancing effect of a stage, Dancing with the Nonhuman
is designed to be performed in gallery spaces. The performance area is only marked through a grid on the
floor, which assists performers to locate themselves; it also represents the cubic grid that the cubes break
loose from (see Figure 6).
To render the boundary more porous, the performance includes transitional intro and outro stages, in which the
performance site is gradually established and dissolved again. In the intro, audiences are welcome to stay
inside the marked performance area and mingle with both human performers and cube performers (costumes and
robot), while they slowly shuffle across the boundary and get settled inside the grid space. At the end of stage
4, the boundary becomes soft again and audiences are welcomed to interact with the performers, both human and
nonhuman. While this could be as casual as sitting down and gently leaning against one of the cubic artefacts,
we are keen for audiences to bodily explore the performers’ perspectives, both human and nonhuman, and to
get entangled themselves.
Summary/In-Parting
This paper introduced our collaborative, diffractive performance-making practice, as part of our ongoing
Machine Movement Lab (MML) project, to promote unscripted, playful encounters with strange, non-humanlike
machines. Our collaborative project centers around the generative potential of movement to harness
dancers’ kinesthetic expertise for empathizing with abstract machine artifacts. This performance-making
practice and its posthuman dramaturgical frame materially mobilizes the theoretical concept of diffraction and
new materialist notions of agential enactment.³⁰ The more-than-human entanglements that our practice
attends to produces the diffraction patterns for mapping out alternative human-machine relationships. This
difference-in-relation also shapes the making of a semi-structured, improvisational performance score, aiming
for audiences to engage with these hybrid entanglings in embodied and empathic ways.
Our diffractive, creative research seeks to open-up new performative strategies for aesthetically attending to
and making tangible difference patterns and relational ontologies at work in human-robot encounters. We propose
that opening-up a more horizontal playground for dancing with machines requires us to get entangled and resonate
with machines, which, in turn, requires collapsing the distance between subjects and objects (rather than
masking it). Collapsing distances, the diffractive way, means to stretch and open-up the boundary in-between
subjects and objects, to explore the space in-between, and grow tentacles into other boundary spaces.
Performance-making here is a mode of generative-diffractive inquiry into the re-/enactment of subject-object
boundaries as part of the dynamic exchanges unfolding in human-robot encounters.
Concerned with the relationalities of embodied meaning-making,¹⁴ our choreographic-dramaturgical
strategies explore the performative aesthetics of corporeally entangling human bodies and machinelike things and
the more-than-human difference pattern this produces. The aesthetic potential of this practice, we believe,
results from combining the asymmetries that differentiate human and machine participants² and the
physical-dramaturgical entanglements that render them relational, producing seemingly dissonant inter-bodily
resonances. Rather than serving to make the strange look more familiar, aesthetics here is about rendering
difference more relational.
Acknowledgements
The authors would like to thank their collaborators: Roos van Berkel (TU/e, NL), Maaike Bleeker (Utrecht, NL),
Katrina Brown (Falmouth, UK), Arabella Frahn-Starkie (AU), Rochelle Haley (UNSW, AU), Lesley van Hoek (NL),
Stephanie Hutchinson (QUT, AU), Sarah Levinsky (Falmouth, UK), Linda Luke (De Quincey Co., AU), Dillon McEwan
(AU), Siobhan McKenna (AU), Kirsten Packham (AU), Felix Palmerson (AU), Marie-Claude Poulin (Applied Arts
Vienna, AT), Tess de Quincey (De Quincey Co., AU), Audrey Rochette (CA), and Kim Vincs (SUT, AU).
This project has been partially supported by the Australian Government through the Australian Research Council
(DP160104706 and FT190100567); the Austrian Science Fund (FWF, AR545); and the EU Framework Programme (FP7,
621403).
References
1 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter
and Meaning, Durham, NC: Duke University Press, 2007, 135.
2 Lucy Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, Cambridge,
Cambridge University Press, 2007.
3 Petra Gemeinboeck, “Difference-in-relation: Diffracting human-robot
encounters,” Matter: Journal of New Materialist Research, V ol. 03, no. 01, 2022, accessed
13 October 2022, https://doi.org/10.1344/jnmr.v3i1.38958.
4 Jennifer Robertson, Robo Sapiens Japanicus: Robots, Gender, Family, and the Japanese
Nation, Berkely, CA, University of California Press, 2017.
5 Claudia Castañeda, Lucy Suchman, “Robot visions,” Social Studies of
Science 44, no. 3, 2014, 315–341.
6 Donna J. Haraway, “Promises of Monsters: A Regenerative Politics for Inappropriate/d
Others,” in Cultural Studies, ed. Lawrence Grossberg, Cary Nelson and Paula A. Treichler,
New York & London: Routledge, 1992, 295–337.
7 Marco Donnarumma, “Beyond the Cyborg: Performance, attunement and autonomous
computation,” International Journal of Performance Arts and Digital Media 13, no. 2, 2017,
1–15.
8 Pontus Hulten, Jean Tinguely: A Magic Stronger than Death, New York, NY, Abbevill Press, 1987.
9 Frazer Ward, “Robert Breer,” Frieze Magazine, issue 155, November 12, 2000, accessed
April 5, 2023, https://www.frieze.com/article/robert-breer.
10 Josée Greg Hill, Stephen Horne, and Anne-Marie Ninacs,
11 Kris Verdonck, “Dancer #3”, A Two Dogs Company website, accessed April 5, 2023,
https://www.atwodogscom-pany.org/en/projects/dancer-3.
12 Bill Vorn, “State Grace Machines” (2007), Concordia University website, accessed
April 5, 2023, https://billvorn.concor- dia.ca/robography/GraceState.html.
13 John McCormick, Adam Nash, and Stephanie Hutchison, “Eve of dust,” in SIGGRAPH
Asia 2018 Art Gallery, New York, NY, ACM, 2018.
14 Petra Gemeinboeck, “The Aesthetics of Encounter: A Relational-Performative Design
Approach to Human-Robot Interaction,” Frontiers in Robotics and AI, Vol. 7 (2021),
accessed September 9, 2022, https://doi.org/10.3389/frobt.2020.577900
15 Paula Gaetano Adi, “BecomingWith” and “Alexitimia”, Artist website,
accessed April 27, 2023, https://www.paula-gaetanoadi.com.
16 Erin Manning, Brian Massumi, Thought in the act: Passages in the ecology of
experience, Minneapolis, MN, University of Minnesota Press, 2014, 39.
17 Maxine Sheets-Johnstone, “From movement to dance,” Phenom. Cogn. Sci. 11,
2012, 39–57, 49.
18 Petra Gemeinboeck, Rob Saunders, “Moving beyond the mirror: relational and performative
meaning making in human–robot communication,” AI & Society 37, Springer, 2022,
549–563.
19 Rob Saunders, Petra Gemeinboeck, “Performative Body Mapping for designing expressive
robots,” Proceedings of the 9th International Conference on Computational Creativity,
Salamanca, Spain, June, 2018, 280–287.
20 Ezequiel Di Paolo, Hanne De Jaegher, Marieke Rohde, “Horizons for the
enactive mind: values, social interaction, and play,” in Enaction: Towards a New Paradigm
for Cognitive Science, ed. John Stewart, Olivier Gapenne, Ezequiel Di Paolo, Cambridge, MA, MIT Press,
2010, 33-87.
21 Donna J. Haraway, When Species Meet, Minneapolis, MN, University of Minnesota Press,
2008.
22 Jon Lee, “Diffractive Dramaturgy,” Performance Research 25, no. 5, 2020, 114-121.
23 D Vinciane Despret, “Responding Bodies and Partial Affinities in Human–Animal
Worlds,” Theory, Culture & Society, 30, no. 8, 2013, 51–76, 61.
24 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter
and Meaning, 72.
25 Stacy, Alaimo, “Thinking as the Stuff of the World,” O-Zone: A Journal of
Object-Oriented Studies 1, 2014, 13-21.
26 Karen Barad, “Posthumanist performativity: toward an understanding of how matter comes to
matter,” Signs: Journal of Women in Culture and Society 28, no. 3, 2003, 801-831, 808.
27 Iris van der Tuin, “Diffraction as a Methodology for Feminist OntoEpistemology: On
Encountering Chantal Chawaf and Posthu- man Interpellation,” Parallax 20, no. 3, 2014, 231-244.
28 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter
and Meaning, p.76.
29 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke University
Press, 2016, p.31.
30 Karen Barad, “Posthumanist performativity: toward an understanding of how matter comes to
matter”.
Bibliography
Paula Gaetano Adi, “BecomingWith” and “Alexitimia”, Artist website, accessed April
27, 2023, https://www.paula-gaetanoadi.com.
Stacy Alaimo, “Thinking as the Stuff of the World,” O-Zone: A Journal of Object-Oriented
Studies 1, 2014, 13-21.
Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning,
Durham, NC: Duke University Press, 2007.
Karen Barad, “Posthumanist performativity: toward an understanding of how matter comes to
matter,” Signs: Journal of Women in Culture and Society 28, no. 3, 2003, 801-831.
Claudia Castañeda, Lucy Suchman, “Robot visions,” Social Studies of Science
44, no. 3, 2014, 315–341.
Vinciane Despret, “Responding Bodies and Partial Affinities in Human–Animal Worlds,”
Theory, Culture & Society, 30, no. 8, 2013, 51–76.
Marco Donnarumma. “Beyond the Cyborg: Performance, attunement and autonomous computation,”
International Journal of Performance Arts and Digital Media 13, no. 2, 2017, 1–15.
Josée Greg Hill, Stephen Horne, and Anne-Marie Ninacs, Petra Gemeinboeck,
“Difference-in-relation: Diffracting human-robot encounters,” Matter: Journal of New
Materialist Research, V ol. 03, no. 01, 2022, accessed 13 October 2022,
https://doi.org/10.1344/jnmr.v3i1.38958
Petra Gemeinboeck, “The Aesthetics of Encounter: A Relational-Performative Design Approach to
Human-Robot Interaction,” Frontiers in Robotics and AI, Vol. 7 (2021), accessed September 9,
2022, https://doi.org/10.3389/frobt.2020.577900
Petra Gemeinboeck, Rob Saunders, “Moving beyond the mirror: relational and performative meaning making
in human–ro- bot communication,” AI & Society 37, Springer, 2022, 549–563.
Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Duke University Press, 2016.
Donna J. Haraway, When Species Meet, Minneapolis, MN, University of Minnesota Press, 2008.
Donna J. Haraway, “Promises of Monsters: A Regenerative Politics for Inappropriate/d Others,” in
Cultural Studies, ed. Lawrence Grossberg, Cary Nelson and Paula A. Treichler, New York & London:
Routledge, 1992, 295–337.
Pontus Hulten, Jean Tinguely: A Magic Stronger than Death, New York, NY, Abbevill Press, 1987.
Jon Lee, “Diffractive Dramaturgy,” Performance Research 25, no. 5, 2020, 114-121.
Drouin-Brisebois, Caught in the act: the viewer as performer (Ottawa: National Gallery of Canada,
2008).
Drouin-Brisebois, Caught in the act: the viewer as performer (Ottawa: National Gallery of Canada,
2008).
Erin Manning and Brian Massumi, Thought in the act: Passages in the ecology of experience,
Minneapolis, MN, University of Minnesota Press, 2014.
John McCormick, Adam Nash, Stephanie Hutchison, “ Eve of dust,” in SIGGRAPH Asia 2018 Art
Gallery, New York, NY, ACM, 2018.
Ezequiel Di Paolo, Hanne De Jaegher, Marieke Rohde, “Horizons for the enactive mind:
values, social interaction, and play,” in Enaction: Towards a New Paradigm for Cognitive
Science, ed. John Stewart, Olivier Gapenne, Ezequiel Di Paolo, Cambridge, MA, MIT Press, 2010, 33-87.
Jennifer Robertson, Robo Sapiens Japanicus: Robots, Gender, Family, and the Japanese Nation,
Berkely, CA, University of California Press, 2017.
Rob Saunders, Petra Gemeinboeck, “Performative Body Mapping for designing expressive
robots,” Proceedings of the 9th International Conference on Computational Creativity,
Salamanca, Spain, June, 2018, 280–287.
Maxine Sheets-Johnstone, “From movement to dance,” Phenom. Cogn. Sci. 11, 2012,
39–57.
Lucy Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, Cambridge, Cambridge
University Press, 2007. Iris van der Tuin, “Diffraction as a Methodology for Feminist OntoEpistemology:
On Encountering Chantal Chawaf and Posthuman Interpellation,” Parallax 20, no. 3, 2014,
231-244.
Kris Verdonck, “Dancer #3”, A Two Dogs Company website, accessed April 5, 2023,
https://www.atwodogscompany.org/en/projects/dancer-3.
Bill Vorn, “State Grace Machines” (2007), Concordia University website, accessed April 5, 2023,
https://billvorn.concordia.ca/robography/GraceState.html.
Frazer Ward, “Robert Breer,” Frieze Magazine, issue 155, November 12, 2000, accessed
April 5, 2023, https://www.frieze.com/article/robert-breer.
Authors Biographies
Petra Gemeinboeck’s and Rob Saunders’ collaborative artistic research practice seeks to expand
and trouble our relations with machines by exploring questions of embodiment, agency, creativity, and
performativity. Petra is currently an Australian Research Council Future Fellow and Associate Professor at the
Centre for Transformative Media Technologies (CTMT), Swinburne University, AU. She also leads the
‘Dancing with the Nonhuman’ FWF research project at the University of Applied Arts Vienna, AT. Rob
is Associate Professor at the in the Leiden Institute of Advanced Computer Science (LIACS), University of
Leiden, NL. His research focuses on computational models of creativity, using techniques from machine learning
and creative robotics. Their artworks have been shown internationally, including the Ars Electronica Festival
(Linz, AT); Int. Triennial New Media Art at NAMOC (Beijing, CN); GoMA (Brisbane, AU); OK Center for
Contemporary Art (Linz, AT); and FACT (Liverpool, UK).
ReSilence: Retune the Soundscape of future cities through art and science collaboration
1, 2, 3, 13, 14 Information Technologies Institute (ITI), Centre for Research and
Technology Hellas (CERTH), 4 School of Creative Media, City University of Hong Kong, 5
InfoMus Lab (Laboratorio di Informatica Musicale), DIST, University of Genova, 6 Department of
Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, 7 Department
of Music, Max Planck Institute for Empirical Aesthetics, 8,9 School of Architecture, Aristotle
University of Thessaloniki (AUTH), 10 WeSound, 11 Catalan Institute for Research and
Advanced Studies (ICREA) and NLP Group, Pompeu Fabra University, 12 Froso Gaki, Thessaloniki Concert
Hall / Thessaloniki (Greece), Hong Kong, Genova (Italy), Maastricht (the Netherlands), Frankfurt am Main
(Germany), Hamburg (Germany), Barcelona (Spain) 1, 2, 3, 13, 14nefeli.valeria, mkosti, diplaris,
stefanos, ikom@iti.gr, 4, 5, 8, 9yiotakoulali, kalliopivals, antoniocamurri,
benayoun.maurice@gmail.com, 6b.degelder@maastrichtuniversity.nl,
7melanie.wald-fuhrmann@ae.mpg.de, 10bronner@wesound.de,
11leo.wanner@upf.edu, 12fgaki@tch.gr
Abstract
Advances in cognitive science, sensing technologies, the arts and creative industries are paving the way for a
deeper understanding of the behaviour of individuals regarding the land/soundscape they live in. Through a
symbiotic relationship between artists, scientists and technology experts ReSilence explores the borders between
sound and silence in a changing world by producing sound awareness in urban spaces (not only reducing the
intensity of noise, but also considering it as energy producer and designing positive sounds, sounds we want to
preserve and multiply). More specifically ReSilence focuses in musical experience design centred on the active
participation of citizens, in the new silence of mobility, in the acoustic perception of outdoor urban
soundscapes and in enhancing experiences for people with hearing and vision impairments.
ReSilence is a project that is part of the S+T+ARTS initiative of the European Commission to foster alliances
between science, technology, and the arts, centred on human needs and values. In order to encourage
collaboration of research projects and artists, S+T+ARTS funds residencies of artists in technology
institutions, as well as activities of scientists and technologists in artists’ studios.
Started in 2022, ReSilence is a consortium of institutions, universities and SME’s that put together
their expertise in terms of scientific, technological, artistic and pedagogical research. Through S+T+ARTS
residencies artists will become part of the research team, which aims to 1) explore the borders between sound
and silence in a changing world by producing acoustic awareness in urban spaces, not only through reducing the
intensity of noise, but also by designing, enhancing and/or multiplying existing sound considering it as an
energy factor, 2) create new types of sonic urban experiences that expand possibilities for accessibility,
active participation/engagement, sustainability and social inclusion and 3) build trust around AI & XR
technologies.
Background
Cities and metropolitan areas are often acting as social condensers, through the dense cohabitation of people,
services and cultures. By 2050, cities will host 2.5 billion more urban dwellers, making the world almost 70%
urban. Europe's urban areas are home to over two thirds of the EU‘s population (1). This
coexistence brings along a series of issues and urgencies, related to place making, the collaborative process by
which we can shape our public realm in order to maximise shared value and make space for diversity. Sound is a
fundamental component of our cities, of their cultures, and of the health and well-being of their citizens. In
an urban fabric, the soundscape of a city is entangled with social behaviour and impacts citizens’
attitudes and behaviour in urban, natural and manmade environments. New ways of mobility, communication,
interaction, and sharing are beginning to have profound consequences for the way that cities sound. More
inhabitants lead to a necessity for more housing, and thus more construction. Increased surveillance introduces
new sounds, such as those made by drones.
The project ReSilence builds on the novel and prima facie paradoxical observation that elements of city life
are becoming more silent due to the emerging electrification of public and private transport, allowing us to
question which sounds are necessary for safety and environmental awareness, and which sounds can be made more
pleasant. Decades ago, artists already worked on these questions. Composer John Cage famously stated that
“everything we do is music,” and his seminal work “4:33”, composed in 1952, is a work
which considers as music the sounds humans make just by simply enacting everyday movements. Before Cage,
Futurist artist Luigi Russolo wrote the manifesto “The Art of Noises,” in 1913. Russolo concludes
“the variety of noise is infinite, and as man creates new machines the number of noises he can
differentiate between continues to grow.”¹ How then should we tackle this growing number of different
sounds in order to improve the quality of life for inhabitants, all while considering the important role of
sound and music to each person‘s life, culture, and identity?
Aims and objectives
As new technologies make their appearance in public spaces and enter our everyday lives, the need to re-imagine
and envisage these spaces unfold, either by addressing their visual or non-visual (sound) aspects and values, in
architectural and urban settings. How cities might sound in the future is a challenge that will significantly
affect the emotional and cognitive state of individuals and will influence the functionality and effectiveness
of indoors and outdoors spaces in manners that need to analyse and take into consideration.² Our world is
shaped through fast modes of communication, distant social interaction and electrification. While physicality
and space continue to be important premises when planning our cities, it is time to start thinking about how we
can design experiences for people living hybridly (physically and virtually) in the city and how we can
orchestrate the sounds our cities produce, sculpting people’s existence. To achieve this, it is essential
to find new ways to engage citizens and audiences to re-think how cities sound.
Therefore, ReSilence is developing technologies by proposing a new methodology with the
scope to shape in novel and innovative ways urban environments and soundscapes, repositioning the role of
silence and that of sound. Recent efforts have focused on the city as the generator of very large datasets used
in an effort to become “smarter,” greener and safer.
Now, ReSilence turns the attention to the social and human experiential dimensions of the
cities’ datasets in order to make citizens more creative and aware of their surroundings, while improving
quality of life through the consideration of noise (see Figure 1).
Main objective of the ReSilence project is to support the development of art-driven
technologies for designing the soundscape of future cities through the collaboration of artists, architects,
urban designers, scientists, engineers and researchers. The project achieves this by targeting a) AI and XR
tools to primarily address challenges from the Urbanism/Mobility sector while exploring the borders between
music and noise, and b) user experience measurements assessing a sound space.
Methodology
The symbiosis between humans and computer-based technologies has a significant impact on Art. Frameworks like
STARTS allow technology to contribute to both the creative process and artistic practice, when in addition; it
works dynamically with humans in the realisation of designs, sounds, etc.
ReSilence supports Art-Driven Experiments (ADE) through Open Calls to artists and artist-SME
teams. Artists, through ReSilence, have access to AI and XR technologies and also help in
ensuring that the development process and system behaviour of the technologies explicitly acknowledges human
values and needs. The following figure (Figure 2.) represents the above-mentioned, based on the research,
technology and art objectives set. Art and artists have a leading role in the development and evolution of the
project and ADE play a critical role to support it. ADE, does not only bring together artists/sound designers,
scientists and technology providers, who explore and work with existing technologies to enhance their usability
and uptake or design art-driven products and services to tackle Urbanism/Mobility challenges described. While AI
and MR technologies take cities to the next step of utilising the data collected and knowledge to support
decision-making, they often solve city problems following top-down solutions. Instead, ReSilence
actually tries to harness such technologies through ADE to engage some of the most fundamental issues
of citizens, focusing on a bottom-up approach and leverage their creativity.
To begin with, we ensure that the development of services and products follows a human-centred design
philosophy so that ReSilence is properly deployed in the project implementation scenarios. A
set of use-case scenarios is defined at first by the artists that include information on their current workflows
and objectives, the challenges they face and the areas where new solutions would be supportive. In this step,
ReSilence in fact builds bridges between artists, science and technology experts. Once the artists define the
challenges and the implementation scenarios, we develop AI-based methods to increase the active experience of
citizens and integrate individual and collective experience of music and sound. Artists and their creations in
the context of Re-Silence, through interactive visualisations and user participation, also foster trust in the
deployed technologies. More specifically the above-mentioned is approached using multimodal movement analysis
and sonification, by developing and integrating in the system platform tools capable to measure the non-verbal,
full body expressive and emotional behaviour, at both individual and group (e.g., synchronisation,
entertainment, leadership) levels. Novel tools for qualitative urban soundscape-analysis are also developed,
guided by the goals set by the artists, based on immersive visual and acoustic stimuli. With the help of those,
the consistency of a soundscape can be investigated to identify needs and means for redesign of outside or
inside spaces. The “AI for real-time interaction” objective is also supported by the generation of
sound related to images, the synthesis of scenes based on sounds and the further study of the relation between
auditory and visual stimuli by means of explainable AI.
The above led to the art-driven toolkit, which deals with the integration of the modules based on which
artists, creatives and technologists co-create artworks, products and services. Artists will be using the
collected audio-visual and spatial data to assist and empower specific user groups by means of multisensory
technology. More specifically, this stage is related to the development of multisensory experiences in XR
environments and allows artists to create different solutions to reshape our urban environments and soundscapes,
rebalancing the role of silence and that of sound. The development and integration of AI tools are also part of
this phase and defines the underlying architecture for a robust communication system between the components and
how the components are integrated in each required system. Artists and architects are also supported by an
object-based interactive audio engine, allowing the algorithmic composition of soundscapes based on visual data.
This engine facilitates the virtual design of urban soundscapes by means of 3D-modelling techniques. Moreover,
the art-driven experiments are evaluated in the “Soundscape Experience Assessment” phase, which
achieves a novel understanding of the human perception in relation to sound and his environment. At this point,
we deal with emotional sensing and analysis, sentiment analysis from crawled social media data and reviews,
multimodal analysis of sound related behaviour and self-report generation. To support the above, we develop
novel physiological signal processing algorithms and signal integration frameworks to detect, extract and
classify markers of emotional responses to aesthetic and/or artistic stimuli. Through this analysis, ReSilence
essentially understands individual and collective perceptions and preferences, leading to a more transparent
decision-making process. Part of this work is also a novel algorithmic approach to investigate the
correspondences between artists’ music composition and listeners’ perception. Another part of this
phase of our methodology is devoted to the linguistic analysis of social media posts and human verbal comments
acquired within ReSilence that discuss topics related to the music sector in order to give an overview of larger
societal preferences and sentiment. Furthermore, for the multimodal analysis, we combine the outcomes of
different components, in order to acquire a more integrated result for understanding the role of active audience
participation. The combined components vary among the different scenarios. The project’s methodology ends
with the fabrication of creative solutions to reach the public, resulting from the collaboration with the
artists. The challenges that led the artists in the definition of the implementation scenarios, along with the
opportunities sound and related media offer to the urban future eventually result in artworks that stimulate
public dialogue through exhibitions, communication initiatives and/or industry collaborations. The first phase
of these S+T+Arts residencies is the research phase during which the artists will spend time discussing with the
ReSilence partners and learning about research and technology initiatives. In return, consortium members learn
about the artist’s practice, working methods and perceptions of the questions being explored. The second
phase is the technical/scientific realisation of the project and the third is the phase for preparing the
results for exhibition and communication.
In a nutshell, ReSilence achieves its goals by establishing a symbiotic relationship between
artists and scientists in a co-creation ground to address urban and societal challenges aiming for an enhanced
urban-sonic experience. Hence, it supports two kinds of art-driven experimentations: (a) artists exploring and
working with ReSilence’s technologies to improve their usability and uptake and (b)
artists and end-users/providers of novel technologies collaborating in art-driven experiments addressing
specific challenges through a variety of scenarios. These scenarios address either 1) the active participation
of citizens, or 2) the new silence of mobility, or 3) the acoustic perception of outdoor urban
soundscapes, or 4) the enhanced experiences for people with hearing and vision impairments, with all of them
aiming to affect the urban/mobility sector.
Cities are not only places where people live and work; they also offer spaces for leisure, entertainment and
art participation. Van der Hoevenan has done long research on how music becomes part of the urban landscape.
³ Concert halls, opera houses, theatres and other performance spaces are architectural landmarks in many
cities.⁴ They offer spaces where urban communities can meet and share meaningful experiences. The
increasing number of mediated performances (broadcasts, recordings, on demand and live streams) that can be
consumed individually and at homes, is a chance and threat at the same time: They offer people with restricted
mobility, small budgets, or that live far away from cultural urban centres the chance for participation. They
also question the future and continuing value of such public spaces and the number and diversity of performing
ensembles.
In this use-case scenario, we focus on live performances of music and dance and explore the value of shared
aesthetic experiences. Live performances are defined by the temporal and spatial co-presence of the performers
and their audience and the creation of the performed work in the here and now – features that can be
broken down into further characteristics and that are relevant for the attractiveness of live performances and
the intense and meaningful experiences they can afford.⁵ Artists will develop and use technologies to
liquify the boundaries between in-person and mediatized performances. E.g., through the use of XR technologies
we seek, on the one hand, to bring a social and interactive dimension to individual streams, and, on the other,
to enhance and individualise live experiences. Therefore, in this use case artists in collaboration with
technologists, scientists, architects and designers will explore new ways of designing musical and dance
performances as well as interactive, performative spaces and environments, both physical and XR-enhanced.
In general, artists can adopt a quantitative or a quality approach in relation to audience experiences. In the
qualitative approach, the aim is to use novel technologies to increase accessibility, participation and audience
engagement. Here, new ways of enabling participation—on site and remotely—should be explored.
Specifically, groups of people who are typically less well represented in certain types of concerts should be
addressed, such as people with disabilities, people that do not belong to the socio-demographic group(s) that
are related to the concrete type of concert and musical repertoire, and people who cannot attend live
performances due to living far away, being immobile or not able to pay for a ticket. In the qualitative
approach, the focus is on broadening and deepening the audience experience. Here, technology-assisted ways of
making the music and concert experience more engaging and meaningful should be explored. One of them would be to
allow audience members to actively participate in the performance. Further, it can be studied to what degree
mixed reality events can provide experiences similar to live concerts, but also whether virtual or augmented
concerts could be a performance format in its own right, with unique features and affordances for its
participants, with a strong impact on the music and on the creative industry sector in general. Our research
will extend on methodologies of studying live performances but also concert streams, for instance in the context
of research projects such as Experimental Concert Research, Digital Concert Experience, the EU
Projects H2020 ICT DANCE (dance.dibris.unige.it) and H2020 FET PROACTIVE EnTimeMent
(entimement.dibris.unige.it), providing research results in automated analysis of expressive qualities of
full-body individual as well as joint movement (e.g., of musicians in small ensembles, or audience members) and
of interactive sonification: An example from the DANCE project is the performances in collaboration with the
choreographer Virgilio Sieni involving audience members as active participants in museum and other public
spaces, such as Atlante del Gesto in Genoa, and the Europa events in occasion of the
celebration of the Treaty of Rome.⁶, ¹²
Such novel technology-driven experiences are a major departure from the conventional situation of the listener
located in a definite space for the time of a concert. They are also a challenge for composers, musicians,
audiences and the classical concert hall.
In this direction, ReSilence investigates how artists can explore and propose novel formats of
concerts and performances characterised by audience engagement to create novel artworks and change
perceptions of how technology can be used and how we experience it.
Besides the virtual platforms, Thessaloniki Music Hall (for performances over 50 participants), Casa Paganini
and the ArtLab of the MPIEA (for smaller groups) will be used for live and for hybrid events and be test beds
for audience co-creation.
The New Silence (Sound and mobility)
The sound fabric of the city has evolved and will keep mutating within the next decades. If the dominant sound
of the city was the one of mechanics in action, from the combustion engine to the factories in action once
investigated by artists such as the Futurist Luigi Russolo glorifying industrial noise in his Manifesto
“The Art of Noises” (1913), we are getting in a society of services that wants to reduce the impact
of the old mechanical model. ¹ Combustion creates two forms of pollution: chemical and noise. Electric
vehicles start changing the sound paradigm. The evolution of this new kind of mobility objects can be compared
with the introduction of a new species in an ecosystem. These 'species' need to find its right place by means of
external signs embodying its reality concerning others.¹³
The electric vehicle as a potential predator for the pedestrian, does not warn the potential unfortunate preys
of its coming. Citizens navigate in the city wearing headphones creating a parallel world that emotionally
disconnect them from the reality of their environment. The accident is the collision of parallel realities that
nobody perceives as coexisting. Suddenly the management of the most immersive component of our reality could
become an unsurmountable challenge. The alternative now is to choose between fighting against the sound of the
city or taming it, accepting the noise or designing the sound. We could also consider that this evolution
resulting from technologies creates interesting opportunities to reconsider our urban sound environment. Often
quoted without explicit references, Goethe was clearly anticipating in the 18th century the subtle relation
between sound and architecture, music and the city: “Architecture is frozen music; Music is liquid
architecture.”
This “liquid architecture” that fills the interstices of the city is a territory to reconquer. If
the specific ambiance of each city is a signature, it is something to be culturally redefined, redesigned. While
it is not only about reintroducing local music, but also accepting sound and noises as mediums to be worked out
and contextualised. Car’s sound used to result from the specific mechanics that make the engine work. Now
it can be designed as the expression of a brand, of a culture, of a driver’s personality, of a
composer’s expression. Space can be acoustically improved to reduce some part of the sound spectrum in
terms of frequencies or to modulate the global sound intensity. Draining asphalt for example can reduce
significantly the sound reverberation of the streets. Reflecting surfaces like glass, concrete or polished
marble increase the reverberation of the sound. Understanding and controlling these parameters allows the city
to master its voice, to refine its expression. The feeling of belonging to a specific place may come from the
light, the architecture, the climate, the language, the smell of food, and it may also come from the sound that
reveals the city spirit and feelings. Can the car be the concert hall of the 21st century? Now that we no
longer undergo the sound induced by noisy mechanics, we may consider designing the interior car sound—with
or without music—as a combination of an ambiance a global receptor for environmental and circumstantial
events. The car becomes an audio interface to perceive the outside in a way to prevent collisions, and to warn
the surrounding people of the vehicle presence and intentions. This can be done in collaboration with companies
working on the sound quality in situation of mobility.
Headphones that are sometimes isolating the pedestrian or the cyclist, may integrate subtle variation that help
to perceive potential issues in the immediate environment. Detection of presence and proximity may be translated
into a change of texture or reverberation of the music we listen to. This would be interpreted as a warning
signal. Indoor acoustic design: Audio reflectivity of surfaces need special attention for the selection of
construction materials. After acoustic music instrument makers, architects working on theatre and concert hall
design have explored this field. It is time to apply this knowledge to places receiving public like restaurant,
subway, and train stations. These places should be considered an acoustic standpoint and the quality of
structural absorption and reflectivity will determine their impact on people and the quality of their
experience. In McLuhanian vision of technology as being an extension of the human body, the car is the perfect
prosthetics in our relation to the city.¹⁴ Bicycles, motorcycles, and cars allow us to accelerate our
motion, amplify our spatial reach, and partially control our space-time relation to the urban environment.
Beyond being an engine of mobility, the car is providing a second skin, at the same time protecting us from
potential aggression coming from unexpected obstacles and dotted by design with extended sensory-motor
capacities to perceive our environment and the complex relation we, as human, may build with contextual aspects
that could have been overlooked while moving at a higher speed than the familiar human walk. This sentient skin
may play a new role in our relation to the environment, not only at the survival level, but also as an emotional
translator allowing us to perceive and to interpret differently the surroundings.
The car may become a very versatile sentient instrument, simultaneously receiver, interpreter, environmental
reader, and full musical dashboard. It can communicate with the passenger and with the surroundings. Like the
shape of the organ pipes impact the harmonics, the timber and the quality of sound, the inertial relation the
car build with its dynamic environment tunes the resulting auditory experience into a complex dialogue that the
driver can understand, and the composer can interpret.
The parameters the composer//interpreter can play with are, but not limited to:
Speed, acceleration, wind speed and direction, spatial localisation of external events: Proximitiy,
relativity, potential danger, obstacle collision,
Light, temperature, ambient noise
Negative/positive noise treatment. (Amplification vs reduction)
Building a relation between users and the environment mediated by the car skin. Users make one with the car
navigating in the city/country side. The most recent technologies convert the vehicle into an autonomous
artefact able to perceive, analyze, recognize, and react, thanks to sensors and more and more of artificial
cognitive functionalities like AI, extending human perception with infrared and ultra violet light, ultrasounds.
We imagine how new narrative may go beyond augmented perception allowing new narrative. The traveling experience
may become a live geolocalise documentary, a romantic journey or a science fiction experience. From ambient
relaxation, and entertainment, vs stimulation awareness, the scenario will have to find a middle way between
fiction and the actual reality with potential dangers. As the most obvious evolution of technologies, musical AI
models may apply through sets of references open to specific creation related to the user or to the car make.
Sound of Urban Spaces
The problem of noise pollution is a serious challenge in urban areas. Soundscape design has been an issue in
city-planning over the last years; there are still tools and methods needed to shape the sound of spaces to
ensure public well- being, and for the time being we do not look enough towards the potential solutions provided
by artists.¹⁵, ¹⁶ While the sound of spaces often is already considered during
their urban planning and architectural design state, the problem mostly affects existing spaces, which demand
sound analysis and redesign of restorative soundscapes. We aim towards building tools and techniques that will
allow architects and urban designers to address the issue of noise pollution in various scales, led by
innovative thinking by sound artists. This will require simulations and fabricated prototypes in small sections
of the urban environment, where few individuals are involved, and gradually expand to larger areas of cities and
communities of citizens. This scenario aims to create tools that help city planners and researchers to speed-up
the analysis of the sound of cities and give urban designers and architects new possibilities to create urban
spaces in an inclusive way so that people can feel comfortable. Furthermore, it opens new space for creativity
not limited to urban design but also for all artists working with sound. Some artists are already working on
soundscapes of cities, such as Justin Bennett who investigated acoustic territories of the city of Brussels and
created devices for active listening such as a microphone inside a trumpet or a wooden stethoscope, for
example.¹⁷
Another interesting artwork related to urban space and sound is“Signals and Hums” by Banu
içek Tülü. It refers to a soundscape work that asks listeners to concentrate on perception.
The artwork exhorts viewers to hone their hearing and listening skills. Sounds are connected to past experiences
and assist us navigate the city on a daily basis. Sound is also a crucial component in understanding our
surroundings.Acoustic environment is not just background noise, a particular soundscape, or noise pollution
brought on by traffic or everyday activities. Instead, the project focuses on appreciating auditory experiences
and exploring sound landscapes in urban environments through listening and hearing.¹⁸,
¹⁹ Soundscapes are defined from an ecological perspective as the physical, biological, and
anthropogenic sound that makes up a landscape, it is presented as a "footprint of an ecosystem," reflecting the
dynamics of a community structure and function.²⁰ With the help of machine learning, mixed reality
and intelligent interactive systems, artists work with existing and future architectural and urban spaces,
considering their visibility and auditory functions. This gives the opportunity to architects and planners to
work on the composition of urban soundscapes, working in parallel in areas with “silent parts”
intermixed with more “noisy parts” and design individual acoustic interventions for specific zones
on a city scale. While at the same time working on small neighbourhood scales using innovative design and
material properties such soundproof or sound reflecting materials in VR environments.
Using 3D-audio-capturing systems, 3D-laser scanners as well as 3D-modelling technologies, we design and create
immersive / navigable audio-visual computational models of existing spaces at various scales of the city. The
AIM (Audio-visual Immersive Maps) platform offers an immersive, virtual experience of cities and places around
the world.¹⁹ Its functions range from sound integration in Google Streetview to high-resolution 3D
scans of buildings, districts or cities. AIM makes real places virtually tangible also through hearing—in
the browser, on the smartphone, in 3D glasses or in the VR room. Using geotagging, AIM enables people to
experience fully immersive virtual travel with 3D Sound. AIM will be further developed to help urban planners to
better assess and design soundscapes of cities and thus improve the sonic environment and well-being in the
city.
ReSilence also provides artists and designers with tools and simulations to analyse and test improvements on
the soundscape design. Such simulations could allow measuring behaviour (and possibly brain imaging) on people
inhabiting different versions of such spaces. Body and movement analysis²¹ including expressive
qualities of full-body movement,⁹ as well as non-verbal social signals in small groups (e.g., real-time
measures of entrainment, of leader-follower relations)¹⁰ can give further indications about how
citizens react to pre-existing conditions of polluted urban sounds (e.g., trajectories of pedestrians to avoid
noisy points; indications of confusion and hesitation in correspondence with noisy zones). Evaluation and
validation can be realised through measuring the change of group behaviour of people in urban spaces, with
respect to changes in urban sounds and environment. The same technologies can measure effects of design
interventions, through the use of prototypes (time spent in silent points, slowdowns in pleasant sonic zones,
etc.). Real-time sonification of traffic (trajectories, volumes, speed) and environmental conditions are used as
a tool to generate interactive sound, meant to counterbalance noisy emissions with background sounds of
stability and fluidity. Such sounds characterised by fluidity might be also related with real-time measures of
the fluidity (Vs. hesitation, rigidity) in movements of citizens: cross-sensory correspondence of movement and
sound qualities is a very interesting and challenging direction in this and in other ReSilence scenarios.
Examples of scientific results and technologies in this direction of movement analysis and interactive
sonification, available also for future experiments in ReSilence, can be found from the previously mentioned EU
Projects DANCE and EnTimeMent.⁶, ⁹, ¹¹
Sound and social inclusion
Through vibration and tactile sensation, we can achieve a “new way of hearing” and through sound we
can find “new ways of seeing”. This use case will focus a) on the importance of a full-body approach
to experiencing music and sound as opposed to engaging solely the ears and b) on the translation of physical
objects into soundscapes and vibrations, inspired by artists such as Christine Sun Kim. By focusing on
individuals who are hearing impaired or visually impaired, ReSilence will work towards creating richer
experiences for people with disabilities. There are 34.4 million adults with a disabling hearing loss in
EU.²² Hearing impairment can mean a loss of hearing, but it can also mean sensitivity to loud noise,
or certain frequencies.
In addition, there are estimated to be over 30 million fully or partially blind citizens in geographical
Europe. Most XR technologies are incapable of providing enhanced information for visually and audibly impaired
people.²³ Traditionally we expect sound to be experienced through our ears. However, nowadays we have
the ability to create multimodal tools and wearables that can support a full-body approach in experiencing music
and sound.²⁴
Concerning visual impairments, the H2020 ICT DANCE project (dance.dibris.unige.it) focused on the automated
analysis and interactive sonification of the expressive qualities of full-body movement. Scientific results and
technologies developed by Casa Paganini and Maastricht University were experimented in artistic projects: (i)
Emobodies (Maastricht Jazz Festival—performance of the blind pianist Bert van der Brink with the
dancer Sagi Gross, in which the pianist could “hear the dance” and perform in a duo); Atlante
del Gesto (performance in four different public sites in Genoa by the choreographer Virgilio Sieni
involving active participation of about 150 citizens: in one of these sites the blind dancer Giuseppe Comuniello
performed as leader of the active audience with the support of interactive sonification of movement qualities).
More details and video excerpts are available in ¹² and in ⁶, ⁹,
¹¹. Casa Paganini developed recently the DanzArTe - Emotional Wellbeing Project
(youtube.com/Infomuslab), an interactive system supporting embodied experiences of visual arts (classical
religious paintings, image manipulation and 3D modelling) based on the real-time analysis and interactive
sonification of expressive movement. Expressive movements of visitors in museums or cultural institutions are
measured in real-time (e.g., fluidity, entrainment in a dyad moving together) while imitating the affective
gesture in a painting. This is also successfully adopted to support aesthetically resonant physical exercises
and rehabilitation in frail older people and in general for cultural wellness and inclusion. This example of
cultural welfare interactive technology is a possible challenging direction for ReSilence.
Exploring innovative technologies to bypass impaired sensory organs (like eyes and ears) can enhance our
spatial awareness, allowing us to perceive sound and visuals through other parts of the body. This novel
approach offers a more comprehensive sensory experience, including the ability to "hear" through the body.
Enhancing our bodies with technology can empower individuals with disabilities, enabling them to experience the
world, communicate, and foster a stronger sense of community, thereby improving city planning and a sense of
belonging.
Conclusion
This paper presents ReSilence, a transdisciplinary project that emphasizes silence and
the space between sounds. ReSilence facilitates the uptake of digital technologies in the
Mobility/Urbansim sector through the integration of technologies such as XR environments, data analysis and
artificial intelligence in an ecosystem of tools designed to support citizens’ needs, motivations and
experiences of the city. Through art-driven experiments with these technologies, ReSilence
explores the multiple ways in which cities are constructed, perceived by incorporating soundscape
information into urban spatial arrangement. The real-world scenarios of ReSilence demonstrate
the expected impact of the proposed solution by focusing on specific societal problems related to cities
soundscapes (indoor and outdoor).
Acknowledgements
This research was funded by the EC-funded research and innovation programme Horizon Europe ReSilence: Retune
the Sound-scape of future cities through art and science collaboration under the grant agreement No. 101070278
1 Russolo, Luigi, The Art of Noises, Pendragon Press, 1986.
2 Vaessen, Maarten, Kiki Van der Heijden, and Beatrice de Gelder, "Decoding of Emotion Expression
in the Face, Body and Voice Reveals Sensory Modality Specific Representations," BioRxiv, 2019,
869578.
3 Hoeven, Arno van der, Erik Hitters, "The Spatial Value of Live Music: Performing,(Re) Developing
and Narrating Urban Spaces," Geoforum 117, 2020, 154–64.
4 Smith, Neil Thomas, "Constructing the Public Concert Hall," Journal of the Royal Musical
Association 146, no. 2, 2021, 255–81.
5 Wald-Fuhrmann, Melanie, Hauke Egermann, Anna Czepiel, Katherine O’Neill, Christian
Weining, Deborah Meier, Wolfgang Tschacher, Folkert Uhde, Jutta Toelle, Martin Tröndle, "Music Listening
in Classical Concerts: Theory, Literature Review, and Research Program," Frontiers in Psychology 12,
2021, 638783.
6 Camurri, Antonio, Gualtiero Volpe, "The Intersection of Art and Technology," IEEE MultiMedia
23, no. 1, 2016, 10– 17.
7 Czepiel, Anna, Lauren K. Fink, Lea T. Fink, Melanie Wald-Fuhrmann, Martin Tröndle, Julia
Merrill, "Synchrony in the Periphery: Inter-Subject Correlation of Physiological Responses during Live Music
Concerts," Scientific Reports 11, no. 1, 2021, 1–16.
8 Tschacher, Wolfgang, Steven Greenwood, Hauke Egermann, Melanie Wald-Fuhrmann, Anna Czepiel,
Martin Tröndle, Deborah Meier, ‘Physiological Synchrony in Audiences of Live Concerts’.
Psychology of Aesthetics, Creativity, and the Arts, 2021.
9 Antonio Camurri, Gualtiero Volpe, Stefano Piana, Maurizio Mancini, Radoslaw Niewiadomski,
Nicola Ferrari, Corrado Canepa, "The Dancer in the Eye: Towards a Multi-Layered Computational Framework of
Qualities in Movement," In Proceedings of the 3rd International Symposium on Movement and Computing,
2016, 1–7.
10 Giovanna Varni, Maurizio Mancini, Luciano Fadiga, Antonio Camurri, Gualtiero Volpe, "The Change
Matters! Measuring the Effect of Changing the Leader in Joint Music Performances," IEEE Transactions on
Affective Computing, 2019.
11 Niewiadomski, Radoslaw, Maurizio Mancini, Andrea Cera, Stefano Piana, Corrado Canepa, Antonio
Camurri, "Does Embodied Training Improve the Recognition of Mid-Level Expressive Movement Qualities
Sonification?", Journal on Multimodal User Interfaces 13, no. 3, 2019, 191–203.
12 Esodi, Public performance at Museo di Palazzo Reale, Genoa - March 25th 2017, accessed December
9, 2022 http://dance.dibris.unige.it/index.php/dance-media#a_vid_artistic
13 Nicolas Misdariis, and Louis-Ferdinand Pardo, "The Sound of Silence of Electric
Vehicles--Issues and Answers," In InterNoise, 2017.
14 Marshall McLuhan, "Media Hot and Cold," Understanding Media: The Extensions of Man,
1964, 22–32.
15 Reeman Mohammed Rehan, "The Phonic Identity of the City Urban Soundscape for Sustainable
Spaces," HBRC Journal 12, no. 3, 2016, 337–49.
16 Yildirim, Yalcin, Merve Dilman, Volkan Muftuoglu, and Sara Demir, "Soundscape Assessment of
Green and Blue Infra-structures," Urban Science 6, no. 1, 2022, 22.
17 J. Bennett, “justinbennett.nl,” accessed January 01, 2022
http://www.justinbennett.nl
18 Sounds of our cities, accessed December 9, 2022
https://www.idensitat.net/en/current-projects/sounds-of-our-cities-3/1456-sounds-of-our-cities-open-call
20 Barry, Truax, Gary W. Barrett, "Soundscape in a Context of Acoustic and Landscape Ecology",
Landscape Ecology 26, no. 9, 2011, 1201–7.
21 Gelder, Beatrice de, Marta Poyo Solanas, "A Computational Neuroethology Perspective on Body and
Expression Perception," Trends in Cognitive Sciences 25, no. 9, 2021, 744–56.
22 “Europe- Hearing-loss | hear-it.org,” accessed December 9, 2022
https://www.hear-it.org/hearing-loss-in-europe
23 Joshue O Connor, Shadi Abou-Zahra, Mario Covarrubias Rodriguez, Beatrice Aruanno, "XR
Accessibility--Learn- ing from the Past and Addressing Real User Needs for Inclusive Immersive Environments,"
In International Conference on Computers Helping People with Special Needs, Springer, 2020,
117–22.
24 Alice Haynes, Jonathan Lawry, Christopher Kent, Jonathan Rossiter, "FeelMusic: Enriching Our
Emotive Experience of Music through Audio-Tactile Mappings," Multimodal Technologies and Interaction
5, no. 6, 2021, 29.
Bibliography
Antonio Camurri, Gualtiero Volpe, "The Intersection of Art and Technology," IEEE MultiMedia 23, no.
1, 2016, 10–17.
Antonio Camurri, Gualtiero Volpe, Stefano Piana, Maurizio Mancini, Radoslaw Niewiadomski, Nicola Ferrari,
Corrado Canepa, "The Dancer in the Eye: Towards a Multi-Layered Computational Framework of Qualities in
Movement," In Proceedings of the 3rd International Symposium on Movement and Computing, 2016,
1–7.
Anna Czepiel, Lauren K. Fink, Lea T. Fink, Melanie Wald-Fuhrmann, Martin Tröndle, Julia Merrill,
‘Synchrony in the Periphery: Inter-Subject Correlation of Physiological Responses during Live Music
Concerts’. Scientific Reports 11, no. 1, 2021, 1–16.
Beatrice de Gelder, Marta Poyo Solanas, "A Computational Neuroethology Perspective on Body and Expression
Perception," Trends in Cognitive Sciences 25, no. 9, 2021, 744–56.
Alice Haynes, Jonathan Lawry, Christopher Kent, Jonathan Rossiter, "FeelMusic: Enriching Our Emotive
Experience of Music through Audio-Tactile Mappings," Multimodal Technologies and Interaction 5, no.
6, 2021, 29.
Hoeven, Erik Hitters, "The Spatial Value of Live Music: Performing,(Re) Developing and Narrating Urban
Spaces," Geoforum 117, 2020, 154–64.
Marshall McLuhan, "Media Hot and Cold," Understanding Media: The Extensions of Man, 1964,
22–32.
Nicolas Misdariis, Louis-Ferdinand Pardo, "The Sound of Silence of Electric Vehicles--Issues and Answers," In
InterNoise, 2017.
Radoslaw Niewiadomski, Maurizio Mancini, Andrea Cera, Stefano Piana, Corrado Canepa, Antonio Camurri, "Does
Embodied Training Improve the Recognition of Mid-Level Expressive Movement Qualities Sonification?",
Journal on Multimodal User Interfaces 13, no. 3, 2019, 191–203.
Joshue O Connor, Shadi Abou-Zahra, Mario Covarrubias Rodriguez, and Beatrice Aruanno, "XR
Accessibility--Learning from the Past and Addressing Real User Needs for Inclusive Immersive Environments," In
International Conference on Computers Helping People with Special Needs, Springer, 2020,
117–22.
Reeman Mohammed Rehan, "The Phonic Identity of the City Urban Soundscape for Sustainable Spaces," HBRC
Journal 12, no. 3, 2016, 337–49.
Russolo Luigi, The Art of Noises, Pendragon Press, 1986.
Smith Neil Thomas, "Constructing the Public Concert Hall," Journal of the Royal Musical Association
146, no. 2, 2021, 255– 81.
Barry Truax, Gary W. Barrett, "Soundscape in a Context of Acoustic and Landscape Ecology," Landscape
Ecology 26, no. 9, 2011, 1201–7.
Tschacher Folkert Uhde, Jutta Toelle, Martin Tröndle, "Music Listening in Classical Concerts: Theory,
Literature Review, and Research Program," Frontiers in Psychology 12, 2021, 638783.
Tschacher Wolfgang, Steven Greenwood, Hauke Egermann, Melanie Wald-Fuhrmann, Anna Czepiel, Martin
Tröndle, Deborah Meier, "Physiological Synchrony in Audiences of Live Concerts," Psychology of
Aesthetics, Creativity, and the Arts, 2021.
Vaessen Maarten, Kiki Van der Heijden, Beatrice de Gelder, "Decoding of Emotion Expression in the Face, Body
and Voice Reveals Sensory Modality Specific Representations," Bio-Rxiv, 2019, 869578.
Melanie Wald-Fuhrmann, Hauke Egermann, Anna Czepiel, Katherine O’Neill, Christian Weining, Deborah
Meier, Wolfgang Tschacher, Folkert Uhde, Jutta Toelle, Martin Tröndle, "Music Listening in Classical
Concerts: Theory, Literature Review, and Research Program," Frontiers in Psychology 12, 2021, 638783.
Yalcin Yildirim, Merve Dilman, Volkan Muftuoglu, Sara Demir, "Soundscape Assessment of Green and Blue
Infrastructures," Urban Science 6, no. 1, 2022, 22.
Towards a Sympoietic Relation with Materials in Interactive Artworks
INREV Laboratory, Paris 8 University Paris, France nefeli.valeria@gmail.com ;
dzamplaras@gmail.com ; sopheeah.k@gmail.com ; chu-yin.chen@univ-paris8.fr
Abstract
In this paper we acknowledge the agency of non-human entities and argue against the binaries of subject/object,
mind/body, nature/ culture, science/art towards a new materiality. This new vision of the nature of materiality
changes the direction of passive matter into a more active one. Technology has given us the opportunity to
characterize and analyze material systems not only by their properties, but also by their potentialities. This
leads to a sympoietic relation boundary between human-matter-machine interactions. In the context of an
interactive artwork, agency should not be considered as inherent for any of the actors. It is emergent, it is
the result of the interactions between the elements and the entities located within a mixed reality environment,
as much inside the installation—material, devices, objects, sensors, humans—as at its
exterior—such as the cultural context and the artist, for example. This new materiality which is based on
an “open material” concept is an emergent and co-constitutive process, continuously becoming in,
with and through interactions.
Keywords
interactive art, materiality, new materialism, symbiosis, sympoiesis, becoming.
While the modern world still struggles with the polarities of mind/spirit, nature/culture, physical/digital,
material /immaterial, many writers and philosophers recently have attempted to challenge this and develop
different theoretical approaches with a more holistic understanding of our environment and our planet. Such
theories often called "feminist materialism" or "new materialism" and writers such as Donna Haraway, Bruno
Latour, Jane Bennett, Karen Barad and Manuel DeLanda tend to adopt the idea of complexity, material action and
imply learning ways to coexist respectfully with the world of matter. This means that primacy is not only given
to the human actor, but also to the non-human ones. These studies also focus on the interrelations between
material agency and social phenomena and help us to imagine an open material that is based on the idea of
movement and agentivity; a material capable of behaving in a complex and multidimensional reality in which the
relationship between subject and object is fluid. They give us a new definition of human's relationship with
matter, one that is closer to a magical vision of the world. This current paradigm shift in the way we think
about, conceptualize, and experience human relationships to their surroundings has led to new forms of artistic
experimentation and brought new interfaces between humans, machines and materials into light. The goal of this
research is to examine and create interactive systems that promote a symbiotic and sympoietic relationship
between human-matter-machine interactions (HMMI). Here, we are not interested in creating a virtual alternate of
reality but in capturing the mystery of the world around us. The research reported in this contribution deals
with fluid materials (natural and intelligent), sound, and light in order to create complex and constantly
evolving systems.
Background
Modern science has demonstrated, in theory and experiments, that the atom (matter’s fundamental units) is
not an indivisible particle, nor the smallest unit of matter. The existence of subatomic particles has been
proved when experiments showed that light could behave like a flow of particles (photons), all the while having
wave-like properties. Quantum physics, therefore, under an alternative (conceptual) framework reveal new
characteristics of matter and raise questions about materiality. Quantum mechanics sometimes go beyond just
explaining the basic structure and interaction of molecules and tackles questions relevant with life, such as
the interaction between mind and body, material and immaterial or living and non-living matter. We find out that
in the very nature of materiality is an entanglement.¹ Bruno Latour, based on the indistinction of things
and people, has formulated a social phenomenon called "Αctor-Νetwork Τheory" (ANT).
ΑΝΤ refers to and includes both humans and non-humans in the same network. Realities are understood
as networks of actors, both people and things.²
In this perspective, there are no a priori distinctions between subjects and objects, nature and culture, man
and machine. As Bruno Latour writes, "... it is we, the Westerners, who have lived until now in the strange
feeling that we had to separate into two distinct collectives, according to two forms of incommensurable
gatherings, the 'things' on one side, the 'persons' on the other."³
Bennett's work also focuses on ideas about the relationship between humans and "things," a relationship she
calls "vital materialism." In her book, Vibrant Matter: A Political Ecology of Things, she explains that by
vitality she means "the capacity of things—edibles, commodities, storms, metals—to act as
quasi-agents or forces with trajectories, propensities or tendencies."⁴ Such a relationship between the
human and non-human worlds can give rise to the concept of enchantment, which, as Bennett describes it, is a
sense of openness to the unusual, the everyday captivating, and can be found in nature, for example, but also in
such unexpected places as modern technology.
All of the above thinkers seek to defy and deconstruct the subject/object division that has been imposed by
"modern" techno-scientific thinking as a paradigm for how we, as subjects, understand and form the world. What
is interesting is that they all try to seek in art a link beyond the modern paradigm, in order to reveal the
hidden creative potential of technology. Specifically, Bruno Latour, while deconstructing the subject/object
divide, points out that the mind and the world are separated in our modern worldview, giving the former the
power to directly control and dissect the latter. On the basis of such a theory, he would try to question this
worldview in order to open the path to alternative possibilities by attributing agency to the non-human.
Theoretical framework Living with materials (Symbiosis)
Symbiosis is examined in this research as a system that establishes a dialogue between a human and their
environment (materials more precisely). In Greek, there are two different verbs to describe life. Ζω
and the old βιώ. The difference between the two is that ζω, "live," refers to
the act of living in a sense of "spending life" or just existing, while βιώ refers to an
experience, in the sense of having an intense life. Life was transformed through politics and public life from
an existential condition into what Aristotle called bios. Through his work “Homo sacer: sovereign power
and bare life,” Agamben asserts that “the fundamental categorical pair of western thought and
politics is that of bare life/political existence, zoē/bios, exclusion/inclusion. However, nowadays we are
moving away from an anthropocentric way of thinking which places human beings at the center of social existence
and many thinkers and writers highlight the ‘active role of nonhuman materials in public life’.
⁵
Making with materials (Sympoiesis)
Poiesis means the activity in which something comes into being that did not exist before. Poiesis derives
etymologically from the Greek ποιεῖν, which means "to make." For Haraway,
poiesis is always already sympoiesis. In Donna Haraway's view, nothing can really create itself.⁶ Thus,
nothing is really autopoietic, but requires other organisms for it to become what it is. This creates a
necessity to combine the theory of autopoiesis with a theory called sympoiesis, which relates to collectively
produced systems, not autonomous ones. In other words, sympoiesis includes and extends autopoiesis.
Donna Haraway adopts the theory of sympoiesis from Beth Dempster. Beth Dempster (1998) introduced the term
"sympoiesis”, which was coined by Friedrich Schlegel (1800), as a way to describe systems that lack rigid
boundaries. More specifically, these systems “have cooperative synergistic characteristics and must be
identified by the continuing interactions among components [...] the systems are evolutionary and have the
potential for surprising change.”⁷
Following this notion, we see interactive artworks as having a greater focus on relationships rather than on
the individual parts from which they are composed.
While such an interactive artwork may begin as an amorphous form with blurred boundaries, it will acquire shape
and identity by interacting with others (human and non-human). These interactions can change the direction of
the outcome and force reinterpretation of the previous forms from different perspectives. As the shape of such
an artwork is in constant change and re-formation, the “life” of such a work is not dependent on any
individual entity (agent, whether human or non-human). By contrast, it is distributed between their relations,
their exchange, and their flow of energy. Interactive artworks attempt to highlight the fact that we need an
“ecosystemic” approach, emphasizing that we must shift our perception of our position in the world
as well as our agency. Sympoiesis as a system should be considered alongside environmental, social, and mental
ecology through an "ecosophie”. Through his collaboration with Deleuze, F. Guattari focused on
heterogeneity and difference, articulating agencements and multiplicities to find rhizomatic structures rather
than unified and holistic ones.
Likewise, sympoiesis is a term that refers to assemblages which acquire their uniqueness and identity through
interactions and the process of becoming in the Deleuzo-Guattarian sense where new “movements”
emerge from creative forces.⁸ Therefore, for this research we started with the idea to create a dynamic
microcosm that could perform a continuous becoming of form, structure and material narrations.
Becoming with materials & the becoming of form
The compound syn, from σύν ("with, together" of symbiosis, sympoiesis) narrates
interdependencies of bodies, materials and things, while “feeling these connections is the principal stuff
of becoming-with.”⁹ This becoming-with line up with Karen Barad's agential realism, which describes
the world, not composed of discrete ‘things’, but “phemomena-in-
their-becoming”.¹⁰ By taking into account new materialistic theories, we understand that
materials are important in order to explore the potentialities of our world in which we live in and engage with
them in a process of becoming something new (worlding (1)). As proposed in this research this can be
achieved by transforming our symbiotic relationship with the world into a sympoietic one which enables us to
look deeper into intertwined human-world relations. Becoming is ongoing and dynamic and entails
material-discursive practices of transitioning from one state to another in space and time.¹¹
The nature of these practices and artworks lies in an effort to communicate with the materials leading to a new
materiality which renews contemporary art in a way that the old hylemorphic couple material / form is
re-examined. The material is explored as something that not only affects the form and meaning of the artwork,
but also dialogs with the viewer. The goal of these artworks is not to give a “magic” wand to the
participant to manipulate matter, on the contrary the objective is to “immerse” him in a milieu of
energy exchange. Based on Simondon’s concept of milieu which is not primarily a spatial concept but a
system of energy transfer between an object or subject and its environment, we approach interactive artworks as
the potential for many different kinds of becoming of form. What Simondon wanted and we acquire is to find a
theory which encloses both the archetypical and hylomorphic theory through a new approach of information and
communication. Simondon found this possibility of unification within the field theory (physical and
psychological) and energy states (referring to quanta). ¹²
In the 21st century, new interactive materials are being designed whose action extends beyond their physical
limits, mixing materials with intangible (digital) information. In the context of the digital revolution, we
assign to materials our functions such as memory, intelligence, even emotion, which leads us to a new definition
of materiality. For Christine Browaeys: “We have to imagine an open material that is based on the idea of
movement, capable of behavior. Thus, Man finds himself immersed in a new materiality in contact with the
digital.” ¹³
Christine Browaeys, in her book "Materiality in the Digital Age: The Human Connected to Matter" emphasizes that
materials and technologies are combined in a symbiosis within the field of Human-Machine interaction where
priority is given to improving the performance and behavior of the material. The scientific models we have today
to understand the evolution of the world are to be reviewed, leading us to a new understanding of matter
overcoming the duality of matter/spirit or matter/form.¹⁴
Digitization and virtualization have tended to disconnect the average person from materiality. They lead us to
believe that creating something 'intelligent' only means a digital system with software that simulates human
intelligence. But a great part of our human intelligence is built with and invested in materials, not digital
technology. We have lost interest in the intelligence of our material world. For that reason, we choose to
tackle materiality from the more dynamic side of the material, which can be defined as an active matter wherein
the interaction flow between man-material-machine unfolds. These materials are becoming, only over time and in
interaction with humans through technology.
Sympoietic relations in HMMI The becoming of the Interface
Many theorists, like Nathaniel Stern, reveal the performative aspect of the complex relationship between the
material world, the body and technology, in the context of interactive installations.¹⁵ In the same
spirit, Karen Barad introduces the notion of intra-action. In Barad’s view, intra-action is a term used to
replace ‘interaction,’ because the latter suggests pre-established entities with individual agencies
that precede their interaction, while the latter focuses on the participatory action. Intra-action posits agency
as emergent. Thus, the idea that beings and entities preexist as individuals with inherent attributes, anterior
to their representation, is contested through a performative approach, according to which the entities are being
constantly co-constructed through an evolving relationship.¹⁶
When trying to approach the aesthetic experience of the interactive artwork, through the prism of intraction,
the various entities are mutually co-constituted, shaping through their complex and ever-changing
relations.¹⁷ As entities can be considered for example the artist, the spectator/participant,
technology, objects and materials, as well as the cultural context of the presentation. In the case of the
interactive installation, it is usually the various interfaces which are in charge of providing the means for
these relations to flourish and develop. Thus, the artistic interface makes possible the relation between the
body and technology, and through this relation the interface is actualized—in the sense of becoming. The
symbiotic and sympoietic relations and the effects and events taking place during the meeting of materials,
bodies and technology bring the interface and all the entities into being.
The materials and the interface are not simply a part of the installation, but rather with the installation and
the other entities in a process of being with the others. Phaedra Shanbaum considers that these ideas are
reaching towards the concept of Being in the philosophy of Jean Luc Nancy, where “being” cannot be
defined outside of “being with.” Rather in a relational conception, “being” is a process
of becoming with the others.¹⁸ Thus, all interactions that occur within an interactive installation
are experimental and procedural processes that are produced in conjunction with - and influenced by - the
others. The presence of the other entities is a prerequisite for the very notion of interactivity with the
artwork to occur, although this by no means signifies the distinction and separation of matter and technology,
of the human “Me” and the technological “Other.”¹⁹ A creative meeting of all
the entities leads to the sympoietic and symbiotic relations that bring the interactive artwork into being.
The notions of being and becoming are deployed here in order to help us develop a theoretical framework for the
analysis of the role of matter/materials and the artistic interface as emergent through the events and the
meeting between body and technology. None of those preexist as such, outside of an enactive system—formed
by all the participating entities, rather they become inside their complex relation and the sensible experience
of matter and technology. Through action, experimentation and the always evolving co-constructed relation, the
meeting with matter and the interface is an aesthetic temporal process, initiated by the artist in the moment of
the conception and continually rethought through the meeting of bodies, materials and technology.
Agency as a creative sympoietic force
In the context of interactive artistic installations, the relation between human, matter and technology can be
partly theorized by the hybridization with and through the various actors. Nevertheless, when designing HMMI,
from the moment of its very conception, interactivity is seen through the perspective of communication and
agency of matter and technology.
Thus, agency allows us to rethink our interactions with the material and the technological world surrounding
us. One of the fathers of informatics and artificial intelligence, Alan Turing, once proposed a test that has
been used as a validation tool that would allow us to classify a machine as “thinking”.
Nevertheless, for Kate Forbes-Pitt, even though this test consists of an undeniable contribution in the
evolution of computers and artificial intelligence, it also provides an enormous contribution in social and
cognitive sciences, as well as in machine mediated artistic environments.²⁰
According to such a point of view, the way that this test was conceived and proposed simultaneously provides us
with an assessment of how humans perceive and interact with other human beings and with technology, as well as
their expectations and affordances vis-a-vis these interactions. It could even be suggested that we are being
more informed about the perceptual capacities of the human actor than about the capacities of the machine
itself. Undeniably, during the exchanges between several entities—humans or between humans and
non-humans—there is agency. In our research, the exchanges between these entities are represented in the
relationship between matter, technology and humans.
For Karen Barad, agency is a matter of interactions, an enactive relationship.²¹ In her theory,
agency is conceived as various possibilities of the reconfiguration of the viewer's body, of material and of
technology. It cannot be a pre-established attribute of either the subject or the object, a condition that
pre-exists in an entity in its conception, outside the context of interactions and relationships. On the
contrary, in the case of the interactive artistic installation for example, agency is produced by the choices of
the human who, consciously or unconsciously, positions and interacts with the other entities—the artistic
interface, the materials, and of course technology.
However, we cannot consider agency as an a priori attribute of the interface, since neither it nor the human
can have a “stable existence” outside of a cultural field of exchange.²² On the other
hand, the performative nature of the experience of the interactive work makes the interface unstable and this is
possibly one of the reasons it opens up the possibility of agency.²³ Nathaniel Stern, for his part,
also emphasizes this performative aspect by explaining that the relationship between the body and the
interactive work is constitutive, a “performed and emergent emergence”, a process that is
constituted “ in and with and through" the relations.²⁴
As such, it has been proposed by several theorists that separating human and non-human entities includes the
risk of reducing their actual role in a co-constitutive relationship. Indeed, for the study of the interactive
artworks, it will be more beneficial to consider their emergence as an encounter between the actions, efforts
and effects of specifically located humans, materials and technology.²⁵ This enactive whole composed
of efforts, effects, people and matter, operates in the specific cultural context of the work, within which the
machine is not an autonomous agent but the precondition for the participation and the study of this set. For
Suchman, the problem is less the attribution of agency to technology and more the fact that our language of
speaking for agency presumes and imposes a field of autonomous and discrete entities. As an alternative
approach, she suggests that we consider interfaces as entities in the making through their relationship with
other entities.²⁶
All the aforementioned thinkers help us to imagine an open material that is based on the idea of movement and
agentivity; a material capable of behaving in a complex reality in which the relationship between subject and
object, human and non-human, nature and culture, science and art is fluid. They give us a new definition of
human's relationship with matter and technology, one that is closer to the enchantment of the material world,
which leads us to renew our conception of matter as an agent in the becoming of the artwork.²⁷
Artistic Installations as Case Studies Ferrofluid interaction and the becoming of form
Liquid Matters (2019) (2) is an interactive installation dealing with materials in a playful way; by
letting the participant interact via gestures and thus a natural interface, triggering movement of organic
matter (ferrofluid particles) in a container liquid display, while this interaction and emergence of forms is
reprojected in real time in a large video installation. (Figure 1)
The ferrofluid drops, organic and fluid as they are, flowing around, are controlled by the participant via a
leap motion device. Thus, LiquidMatters, suggests a sympoietic artificial organism between human (experiencer),
non-human matter (ferrofluids) and machine (Arduino controller and leap motion sensor, filed under computer
vision). This simple combination of ferrofluid and magnet coupled with computer vision and real time
interaction, becomes a complex system of a sympoietic node towards a material other. The organic form is
co-created as a response to the feedback human—machine loop, but in this case, the machine is organic
pixel forms: they come in contrast with the regular pixels, thus confusing us on their nature; are they living
creatures or maybe simulated ones?
In LiquidMatters installation (Figure 2), ferrofluid behavior is controlled by many technical factors, such as
the distance between magnets, the power and the number of magnets and ferro particles itself, as well as the
density of the container liquid. Ferrofluid movement is controlled by an Arduino device that translates the
visitors’ input gestures and hand movements to electromagnetic trajectories in the x and y axis.
LiquidMatters is a fragile in nature experiment: if the particles don’t interact, they sit still in their
container liquid. As soon as our attention and gestures are captured new forms emerge, like a game of life of
swarm ferro-particles. The significance of this ‘augmented- diminished’ reality experience is
referring to the economy of attention between visitor and ferrofluids and in the sympoiesis of an entangled
medium which becomes the artwork.
Through the force of a magnetic field ferrofluid becomes alive, complex in its behavior. By seeing this liquid
as a material with intelligence, we activate and animate it. Digital technology allows us to see it as a
responsive system where we are not simply imposing our will upon the environment but letting it reveal certain
kinds of properties or possibilities through its constant fluidity. When dealing with matter in such a way, the
agency of humans, machines and that of the materials unite, co-create and make visible the superimposition
between the realms of a machine, human expression, and of matter itself. Instead of using digital processes or
coding to generate forms, we used natural dynamic forces such as magnetism, gravity, and chemical
reactions.
Seeing through touch: Handprints as proof of sympoietic interfaces
VitRails (2018) is a mixed reality interactive installation, on the mediterranean refugee crisis. The
multi-sensorial interactive installation uses a black thermochromic surface as an interface and a VR headset
(Figure 3.). The visitor wears the headset, and is asked to touch the black surface presented as a window frame
(Figure 4). The thermochromic behavior of the surface’s material allows a temporary color change on the
black surface, making the black transparent for a few seconds, and this, by touching, thus transmitting heat to
the surface, using our hands. As soon as the surface changes its temperature by touching, a virtual landscape
appears in the headset. Computer vision coupled with this behavioral object, the thermochromic paint, are the
interface of this installation and allow visitors to enter the Virtual Reality world. Visitors can see through
their handprints, by touching the surface. Once achieving access to the virtual world through his traces, he or
she has only a few seconds to explore the events revealed on the other side. The cracks will start to disappear
quickly, while the color of the painting returns to its original state.
Archaeological evidence brings to light how our ancestors, since the hunter-gatherer period have been using
different techniques involving materials in order to communicate a story, inventing languages, tools and
techniques to pass on information. The Caves of Hands in Argentina (Figure 5), is named after the findings of
mural stencils dating back to 7300 BC.
Cave paintings are symbiotic and sympoietic artworks. Symbiotic because of the relationship of different
micro-organisms (bacteria and fungi) that coproduce a process of continuous “restoration” while
etching the pictures deeper into rock (3). and sympoietic because this chemical
“metabolism” refills the contours and vividness of colors/traces that were left by humans thousand
years ago.
VitRails installation, uses the metaphor of negative and positive human handprints/traces, taken from the Cave
of Hands, to the material virtual other, entangling stories through a techno-anthropological gaze of the world
via Virtual Reality technology, computer vision, real time simulation, photogrammetry and the thermochromic
paint. In this case, a cybernetic machine is created, involving human, material and technology in order to
entangle stories of the Anthropocene, becoming the Capitalocene or the Cthulhucene as named by D.
Haraway.²⁸ In this new materialism, the gaze shifts its focus from the human, toward informational or
data flux and digital landscapes, and this new paradigm is thoroughly explained by Katherin Hayles, in How
we Became Posthuman: “How information lost its body, that is, how it came to be conceptualized as
an entity separate from the material forms in which it is thought to be embedded.”²⁹ As Hayles
writes, a critical practice that ignores materiality, or that reduces it to a narrow range of engagements, cuts
itself off from the exuberant possibilities of all the unpredictable things that happen when we as embodied
creatures interact with the rich physicality of the world.
In the cybernetics era, information has lost its body not to be disembodied, but to be embodied differently, to
include non-human artificial and natural environments and by redefining the boundaries of nature itself. By
creating a feedback loop, this interactive behavior emerges by actively touching the thermochromatic surface,
creating negative space. Using a camera, the negative space creates a virtual magical portal. A portal to
another reality. Multisensorial virtuality creates the magical plausible illusion of being in this alter world,
co-creating with the material and the virtual, in an endless sympoietic feedback cycle. As Norah Campbell points
out many theorists have noticed a splicing of direct and tactile human perception of reality with another
reality, one that is mediated and technical; producing a new reality that negotiates the individual's
knowledge of the universe in diverse and complex ways.³⁰
In contrast to many immersive virtual reality artworks, which privilege the sense of vision and usually
completely neglect other senses, in VitRails, the participant is invited to be in constant contact with the
physical world around him in order to sense and perceive the virtual. In the virtual world, a parallel story is
emerging. By touching the thermochromic surface the visitor creates traces of his palms, which allow him via
computer vision to briefly explore fragments of another reality through a virtual window, allowing fragmented
view to the 3D scenographic elements and interact with them in real time, recreated by the use of
photogrammetry, real time simulation, and 3D design in the VR environment.
This experimental artistic research includes human computer interaction with intelligent materials: technical
artifacts that act/react as if conscious agents, able to decide on an output behavior, given a specific input.
This does not come as a surprise, as all cybernetic environments are controlled by object-oriented coding
languages.
As Gilbert Simondon elegantly describes technical objects: "It is difficult to define technical objects by
their belonging to a technical species; the species are easy to distinguish summarily, for practical use [...],
but this is an illusory specificity, because no fixed structure corresponds to a defined use."³¹ This
paper, aimed to explore the blurring boundaries of smart materials as interfaces, on experimental artistic
practices in order to propose a symbiotic relation between intelligent objects, human and technology, to form
what is called "the Artwork".
Conclusion
In this paper we introduced human-matter-machine interaction as a concept that questions the symbiotic and
sympoietic relationship that emerges when experiencing interactive artworks with intelligent materials.
Through HMMI, a new definition of human's relationship with matter stems from the entanglement of the
participating actors that forms complex co-constitutive and evolving systems, defying cartesian dualities and
attempting to capture some of the mystery and the magic of our surroundings.
These two artworks attempted to draw sensitive relationships with the material world around us through specific
“becomings.” These installations were born from experimentation on possible symbiosis and sympoiesis
between humans, the material world and technology, while trying to provoke unpredictable interactions that lead
to more sensory relationships.
These artworks are using such techno-scientific advancements as a way to make spectators question their
relationships with the physical environment. Therefore, our focus was on how we experience our physical
environment now that the digitized world has altered our perception.
Questioning the role of each entity participating in the becoming of the artwork, whether human or not, the
notions of agency and emergence are recurrent in our theoretical framework. When studying interactive artworks,
especially through the perspective of experience, their emergence can be considered as the entanglement of
humans, materials and technology. The situated action inside this enactive whole gives rise to the artistic
interface as an open materiality, challenging our relationship with our surroundings.
Acknowledgements
Research for this paper has been supported by AIAC, INREV research group, Paris 8 University and EnsadLab
Spatial Media research group, PSL University.
(1) Worlding’ refers to the co-operative way of ‘world-making” according to
Donna Haraway.
(2) Liquid Matters, interactive installation by continuum collective, as presented at RectoVRso
Gallery in Laval, during Laval Virtual Summit.
(3) Mihnea Mircan, Introduction-Allegory of the Cave Painting, Mousse Publishing, Milan, 2015
References
1 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter
and Meaning, Durham, N.C., London, Duke University Press, 2007.
2 Bruno Latour, Reassembling the SociaL: An Introduction to Actor-Network-Theory,
Oxford, OUP Oxford, 2005, 2.
3 Bruno Latour, Politiques de La Nature : Comment Faire Entrer Les Sciences En
Démocratie, Verlag, Paris: Découverte, 2004, 66–67.
4 Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke
University Press, 2010, 8.
5 Jane Bennett, Vibrant Matter: A Political Ecology of Things, 2.
6 Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene,
Durham (N.C.), London: Duke University Press, 2016.
7 Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene,
58.
8 Rosi Braidotti, “Discontinuous Becomings. Deleuze on the Becoming-Woman of
Philosophy,” Journal of the British Society for Phenomenology 24, no. 1, January 1993, 44.
9 Stephen Abblitt, “Composite Lives: Making-with Our Multispecies Kin (Imagine!),”
A/B: Auto/Biography Studies 34, no. 3, September 2, 2019, p.507–18, doi:.
10 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning
11 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning, 142.
12 Gilbert Simondon, L’individu et Sa Genèse Physico-Biologique, Editions
Jérôme Millon, 1995.
13 Christine Browaeys, La Matérialité À l’Ère Digitale :
L’humain Connecté À La Matière, Saint-Martin-D’hères, Presses
Universitaires De Grenoble, 2019, 13.
14 Christine Browaeys, La Matérialité À l’Ère Digitale :
L’humain Connecté À La Matière, 104.
15 Nathaniel Stern, Interactive Art and Embodiment, Gylphi Limited, 2013.
16 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning, 84
17 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning, 37.
18 Phaedra Shanbaum, The Digital Interface and New Media Art Installations,
Routledge, 2019, 44.
19 ibid
20 Kate Forbes-Pitt, The Assumption of Agency Theory, Routledge, 2011, 2.
21 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning, 235.
22 Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of
Matter and Meaning, 213.
23 Phaedra Shanbaum, Media Art Installations, Routledge, 2019, 59.
24 Nathaniel Stern, Interactive Art and Embodiment, 62.
25 Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions,
Cambridge: Cambridge Univ. Pr, 2009, 280.
26 Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated
Actions, 263.
27 Jane Bennett, The Enchantment of Modern Life : Attachments, Crossings, and
Ethics.
28 Donna Haraway, “Anthropocene, Capitalocene, Plantationocene, Chthulucene: Making
Kin,” Environmental Humanities 6, no. 1, 2015, 159–65.
29 N Katherine Hayles, How We Became Posthuman Virtual Bodies in Cybernetics, Literature, and
Informatics, Chicago: The University Of Chicago Press, 2008, 17.
30 Norah Campbell, “The Technological Gaze in Advertising,” Irish Marketing Review
19, January 2008.
31 Gilbert Simondon, Cécile Malaspina, John Rogove, On the Mode of Existence of
Technical Objects, Minneapolis, Mn: Univocal Publishing, Minneapolis, Minn, 2017, 11.
Bibliography
Stephen Abblitt, “Composite Lives: Making-with Our Multispecies Kin (Imagine!).” A/B:
Auto/Biography Studies 34, no. 3, September 2, 2019, p.507–18,
doi:10.1080/08989575.2019.1664132. Autor: Bruno Latour, Politiques de La Nature : Comment Faire Entrer
Les Sciences En Démocratie, Verlag, Paris, Découverte, 2004.
Karen Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning,
Durham, N.C., London, Duke University Press, 2007.
Karen Barad, “Posthumanist Performativity: Toward an Understanding of How Matter Comes to
Matter.” Signs: Journal of Women in Culture and Society 28, no. 3, March 2003,
801–31, doi:10.1086/345321.
Jane Bennett, The Enchantment of Modern Life: Attachments, Crossings, and Ethics. Princeton,
N.J., Princeton University Press, 2001.
Jane Bennett, Vibrant Matter: A Political Ecology of Things, Durham, Duke University Press, 2010.
Rosi Braidotti, “Discontinuous Becomings. Deleuze on the Becoming-Woman of Philosophy”,
Journal of the British Society for Phenomenology 24, no. 1, January 1993, 44.
doi:10.1080/00071773.1993.11644270.
Christine Browaeys, La Matérialité À l’Ère Digitale : L’humain
Connecté À La Matière, Saint-Martin-D’hères: Presses Universitaires De
Grenoble, 2019.
Norah Campbell, “The Technological Gaze in Advertising", Irish Marketing Review 19, January
2008.
Monica J. Casper, “Reframing and Grounding Nonhuman Agency.” American Behavioral
Scientist 37, no. 6, May 1994, 839–56. doi:10.1177/0002764294037006009.
Kate Forbes-Pitt, The Assumption of Agency Theory, Routledge, 2011.
Donna Haraway, “Anthropocene, Capitalocene, Plantationocene, Chthulucene: Making Kin”,
Environmental Humanities 6, no. 1, 2015, 159–65. doi:10.1215/22011919-3615934.
Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene. Durham (N.C.), London,
Duke University Press, 2016.
Bruno Latour , Reassembling the Social: An Introduction to Actor-Network-Theory, Oxford, OUP
Oxford, 2005.
Lucille Alice Suchman, Human-Machine Reconfigurations: Plans and Situated Actions, Cambridge,
Cambridge Univ. Pr, 2009.
N Katherine Hayles, How We Became Posthuman Virtual Bodies in Cybernetics, Literature, and Informatics,
Chicago: The University Of Chicago Press, 2008.
Phaedra Shanbaum, The Digital Interface and New Media Art Installations, Routledge, 2019.
Gilbert Simondon, L’individu et Sa Genèse Physico-Biologique, Editions
Jérôme Millon, 1995.
Gilbert Simondon, Cécile Malaspina, John Rogove, On the Mode of Existence of Technical Objects,
Minneapolis, Mn: Univocal Publishing, Minneapolis, Minn, 2017.
Nathaniel Stern, Interactive Art and Embodiment, Gylphi Limited, 2013.
Thinking with Tides: Engaging with Embodied Technical Processes within the Tidal Ranges of the Thames Estuary
GerolamoGnecchi
Goldsmiths University of London London, UK ggnec002@gold.ac.uk
Abstract
The following paper reports on findings from doctoral research in Arts and Computational Technology. The
research is about tidal environments. More specifically, it engages with the tidal ranges of the Thames Estuary
and explores what thinking with tides might signify. In the face of uncertain futures—sea-level rise,
increase in flooding, pollution—and through art practice, it reflects on how combining different ways of
knowing the Thames Estuary could reintroduce questions of locality, situatedness and diversity to make the
estuary differently available. The paper is divided into four sections. The first section summarises the kind of
tidal processes that unfold in the estuary. The second section introduces the methodology, Practising in the
Wild, and explains how Embodied Technical Processes (ETP) are necessary to engage with the estuary. The third
section presents an inventory of different technical objects that are used to think with the estuary and its
tides. Finally, in the last section, two art projects—by composer John Eacott and art collective
YoHa—are discussed. Both projects were situated within the tidal ranges of the Thames and applied ETP. It
concludes with reflections on future studies that this research is currently developing.
Tides are always on the move. They have their own spatial and temporal growth. They vary from place to place.
Every tidal environment sounds, smells, looks, and feels different. As you read these words, tides are changing
the way life unfolds, intersecting with organisms, ecologies, cultures, wavering space, time, and materiality.
Their openness, their complexity, their diversity, makes tidal environments both knowable and unknowable,
predictable and unpredictable.
This paper reports some of the findings and methods, developed along the tidal ranges of the Thames Estuary.
The Thames Estuary can be seen as a complex more than human soup of colonial histories, empires, trade,
language, cosmic forces, the Anthropocene.¹, ², ³ It is challenging, more
so impossible, to create a totalising narrative of this stretch of mud and water. Thus, what this research can
offer, are different ways of knowing the estuary.
The estuary has multiple facets, which include material, ecological, and cultural roots, but it is also the
plumbing of a global city like London.⁴ It is both wild and industrial. Over the 20th century, the estuary
has been deepened, widened, diverted, drained, channelised, filled in and built over, and dammed. It is an
emblem of the Anthropocene that has been colonised, commoditised, artificially engineered, bombed, trashed, and
recently is being rewilded. Yet the estuary exists before and beyond its codification as a place for trade,
empire, consumption. A port is a space built for human use, representing distribution, labor, marine routes,
networks, but an estuary, in the most basic geological sense, is about sediments, algae, silt, bodies, species,
salt marshes. Thus, the estuary cannot be defined solely by natural characteristics, nor is it merely a product
of human use and imagination.⁵ It is a transitional space, with different temporalities and flows that
oscillate between the river Thames and the North Sea, sea and land, fresh and salt water. But most importantly,
a space dictacted by its tides.
To engage, explore, and know the estuary, this doctoral research proposes to learn how to think with the
more-than-human: the forces of waters, the winds, the ever-changing seabed, as well as many other beings like
boats, buoys, tide charts, algae, barnacles.
Aims and Objectives
The purpose of this paper is to introduce the methodology of this doctoral research named 'Practising in the
Wild'. Practising in the Wild is imbued with Embodied Technical Processes (ETP). Processes that are tied to
different ways of knowing the estuary, both embodied and technical, experiential and scientific. They aim to
establish an active engagement with the estuary, combining different ways of knowing—embodied, technical,
situated, local to name a few—and provide methods for artistic and design practices that deal with tidal
environments. Furthermore, it attempts to demonstrate the structural coupling that occurs between objects,
bodies, tides, the atmosphere and avoid any hard distinctions between cultural, ecological, social, and
technical ways of knowing the estuary. The paper then proposes an inventory of technical objects used to explore
the Thames Estuary, such as boats, tide gauges, and tidal predictions. It then looks into two art interventions
in the estuary that apply ETP, one by composer and sailor John Eacott and the other by art collective YoHa.
However, before delving into the methodology, a summary of the kind of tidal processes that expand and retract
in the estuary should be considered.
The Thames Estuary and its Tides
The Thames Estuary experiences two distinct tidal patterns, one from the North Sea and the other from the
English Channel. Generally speaking, estuaries are composed of a stew of water and mud where freshwater and
seawater meet. The infusion depends on various factors, such as the state of the river and sea at a given time
and the repetitive stirring of its tides. This unfolds into spaces with no clear boundaries making estuaries
both familiar and unfamiliar, where certainty is flawed by uncertainty, and constancy is affected by fluidity.
Its tides result in monthly, seasonal, and yearly variations, which correspond to the complex ‘pas de
trois’ between the sun, moon, and earth.⁶, ⁷ The height and timing of tides are
determined by various factors, including seabed topography, atmospheric pressure, wind patterns, and wave
dynamics, to name a few. Despite efforts to predict tides accurately, their intricate dance defies their
predictability, which makes them both predictable and unpredictable.
Some Basic Tidal Knowledge
The estuary’s tidal processes interconnect land and sea, salt and fresh water, sewage and silt, wrecks
and barnacles, creating a complex range of eco-social assemblages.⁷ They have a significant impact on the
estuary's morphology, co-creating and transforming a variety of ecosystems. The seabed in this liminal space is
in a constant state of flux, with channels and banks shifting daily, monthly, and seasonally. There is no
finality or fixed reality when it comes to the intertidal zone. For instance, as saltwater flows up the estuary
with each incoming tide, fresh water from the river's catchment carries silt and other materials downstream. As
the tides recede, coastal habitats, such as salt marshes, are formed. These low-lying habitats are characterised
by deep, irregular creeks and soft, fine silt. They are home to a wide variety of wildlife, including sea
molluscs, wildfowl, and a vital fish nursery.
Their cycle is known to be semidiurnal, which means tides rise and fall roughly twice in twenty-four hours,
revealing two different environments within the intertidal zone (figure 2). The waters expand and retract. Their
timings do not synchronise with the day-night cycle in any simple way.⁷ During high tide the intertidal
zone covers itself with water. During low tide vast ecologies of mud are revealed. The tidal range is larger
than 4 meters. These kinds of tides are known as macro tides. In London Bridge, the tidal range can reach up to
seven meters. Along the estuary, it varies between five and six meters. Moreover, the timings of high and low
water occur at different moments between London Bridge and the estuary. At the estuary, high water precedes that
of London Bridge by approximately one hour.
Tides also interplay with floods. The estuary has the highest proportion of its population within Zone 2
flood-plains.⁸ Zone 2 flood areas represent the extent of an extreme flood event—the chance of
flooding from rivers or the sea in 1000 years. More than half of the sites of ecological interest, nature
heritage sites, areas of special protection, and wildlife trusts correspond to former landfills, sewage, and
derelict wastelands, which lie within flood Zone 3 areas. Zone 3 represents the threshold for defining
under-risk areas by sea level rise and coastal erosion. Approximately 1,200 hectares of habitats, such as
mudflats and salt marshes, could be lost due to flooding.⁹
Tidal flooding can have significant destructive impacts when a combination of spring tide, strong gale-force
winds, and low atmospheric pressure coincide.¹⁰ The consequences of such events can be exemplified by
the 1953 storm surge that hit the Thames Estuary shores, causing the loss of roughly 300 lives and the
displacement of 30,000 individuals from their homes. The East Coast of the UK was severely affected, and
communities were left isolated. The floodwaters submerged 65,000 hectares of farmland and inundated 240,000
houses. As a result of this catastrophic event, extensive research was conducted on coastal defences, storm
surges, and policies, leading to the implementation of weather forecasting, warning services, and a network of
tide gauges across the UK. The Thames Estuary remains vulnerable to future storms and sea-level rise.
Flood risk management often involves constructing higher seawalls, dikes, barriers, levees, as well as
quantifying, modelling, and datafying the environment. This can leave out forms of knowledge that are
subjective, local, situated, and ecological.¹¹ Current discourse on environmental management asserts
that traditional modernist or engineering approaches are becoming obsolete.¹²,
¹³, ¹⁴ Thus, in the face of uncertain futures and environmental
challenges, how to reintroduce questions of locality, situatedness, and diversity into the discourse of the
estuary? Moreover, how could this reintroduction contribute to the discourses of designing, planning, managing
and dwelling in tidal environments?
Methodology — Practising in the Wild
Practising in the Wild is driven by a strong sense of environmental responsibility and ecological urgency. It
problematises and challenges forms of scientific reductionism, exploitation, and appropriation of tidal
environments that often go unchallenged. It involves the creation of knowledge through hands-on practice,
outdoor exploration, and ethnographic studies. It enables one to fall into the estuary’s flows and to
attune oneself to the rhythms and patterns of its tides.
This methodology combines local, situated, sensorial, embodied, and technical knowledge in the attempt to
challenge purely discursive readings of the estuary. This can create an opportunity for people to engage in a
dialogue that questions prevailing methods and enables a more nuanced understanding of the complexities of the
estuary. When engaging in activities within the intertidal zone, seafarers learn how to think with the forces of
the waters, winds, seabed, tide charts, tidal predictions, sonar baths. Their approach allows to establish a
close relationship with technical objects and our own embodied experiences. In other words, Practising in the
Wild is imbued with Embodied Technical Processes (ETP). Processes that combine both technical and embodied ways
of knowing and thinking with the estuary.
Embodied Technical Processes
Thinking through ETP allows us to comprehend the structural coupling of humans in collaboration with different
bodies. Embodied, here, focuses on building knowledge about the estuary through bodily experiences. The body
acts as an instrument to know, bypassing rational thoughts through bodily senses and feelings.¹⁵ It
provides a sensorial understanding of the estuary, opening the potential for affective transmission; from the
body to environment and environment to body.¹⁶ For instance, experienced seafarers can feel the wind
changing through the hair of their skin—an embodied response to a changing environment.
Technical refers to the knowledge one develops in collaboration with tools, objects, instruments, devices. For
instance, when planning a journey and before starting to navigate, knowing the tide’s height and timings
is necessary when seafaring the estuary. This information can be obtained from a nautical almanac, an app or a
tide website that predicts the times and heights of high and low water daily. Alternatively, a tidal curve
specific to each port can be used to manually calculate the tidal range at any given time. The tidal curve is a
wave graph that can be found in the nautical almanac (see figure 4).
When sailing with experienced seafarers, one can gain further insight about ETP. People often discuss the
relationships between the wind, water, boats, tools, devices. For instance, nowadays, many seafarers use GPS or
radar tracking equipment, check tide charts and weather forecasts before heading out, and combine these
different readings with their personal experiences to sail the estuary. It involves a balance between their
knowledge and the use of technical objects, blurring the boundaries between humans, objects, and the estuary.
The acquisition of knowledge within the Thames Estuary, thus, relies on both technical and embodied cognitive
processes. In other words, a thorough understanding of tidal environments is contingent on technology,
experience, and embodiment.
The following section introduces an inventory of technical objects that are used to explore and think with the
Thames Estuary. The aim is to understand the nature of the kind of knowledge that these objects embody and how
in turn, they help us to build different ways of knowing the estuary.
Inventory of Technical Objects
When walking along the Thames Estuary, one can notice the proliferation of technical objects along its shores
and shoal waters. From tide gauges to wrecks, from jetties to cargo ships, the Thames Estuary is populated with
technical beings. They act as mediators between the estuary’s ever-changing rhythms and patterns and us.
Some encapsulate the estuary and its tides inside fibre-optic cables, informing us of the highs and lows of the
waters, while others enable us to actively explore the estuary. Their different cultures embody different ways
of knowing, allowing us to think with this soup of mud, water, salt marshes, pollution, sewage.
Boats
Within the context of seafaring, boats can offer thought-provoking insights into the ways people engage with
the estuary. They are tied to the movement of the body in space, and the forces of the waters, winds, and
seabed, creating constant feedback between the estuary and the body. They become a lens for a dynamic and
sensorial experience of the estuary. For instance, sailing necessitates a unique set of skills and knowledge
that are not typically required on land. They allow us to become amphibious. The movement of a sailboat is
dependent on various factors, such as the boat's features, the forces of the waters and winds, and how they
interact with each other. In other words, the movement is influenced by a complex interplay of external forces
and internal intentions that dictate directionality and affect local action. Seafarers navigate by "feeling"
their way towards their destination, continually adjusting their movements based on the flow of waves, wind,
current, stars, and adjusting their boats.¹⁷ The knowledge gained through seafaring is often implicit
and intuitive, making it challenging to articulate or communicate with others. While discursive readings can
provide a foundation, it is only through experience that one can gradually gain the skills required for exploing
the shallow waters of the estuary.
Tillers The analysis of the potential of a seemingly mundane object, such as a boat's tiller,
can provide valuable insight into the fundamental nature of seafaring.¹⁸ By feeling the tiller, a
seafarer can directly sense the impact of environmental factors on the boat and perceive fluctuations in the
wind and water. According to Thomas Gladwin (1964), Micronesian sailors are constantly aware of "motion, sound,
the feel of the wind, wave patterns, stars, etc." throughout their voyage. They utilise past embodied
experiences to make adjustments, such as slightly changing the positioning of the tiller or instructing the crew
to adjust the sails.¹⁹ The tiller thus serves as an extension of the body, allowing the seafarer to
seamlessly flow with the currents and manage the boat’s movements. Additionally, the tiller plays a
crucial role in regulating and organising the rest of the boat, further highlighting its significance in the
context of seafaring.
Data Sytsems
Data systems are seen to provide a continuous stream of data on the Thames Estuary, including real-time tidal
information, prediction tables, tidal currents, and charts. These systems play a significant role in various
activities within the Thames Estuary. They are critical in monitoring sea level rise, flood risk management,
navigational safety, and sea-based commercial logistics.²⁰ A whole technological infrastructure is
situated upon the estuary, composed of tide gauges, ISO maps, tidal predictions, tidal apps, to name a few.
These technologies function as an extension of our memory, enabling us to keep track of and compute tidal
processes, to observe and analyse their cycles. They aid people in delineating the ranges, currents, and future
states of the estuary.
Tide Gauges embody knowledge about the ranges of the tides. They are an important invention in
the field of oceanographic measurement, first created by civil engineer Henry Palmer in 1831 for the London Dock
Company. These devices were designed to automatically record water levels in a specific location, utilising a
float that sat on top of the water in a tube and a pen that recorded the variations through a wave graph on
paper attached to a rotating drum. With the advent of self-recording tide gauges, tidal measurements were
transformed from a daily sequence to a continuous process over time, mediated by machines rather than human
observation. ²¹ In other words, the introduction of tide gauges revolutionised the way in which tidal
data was collected, enabling autonomous recording and representation of tides around the clock. Prior to their
invention, the tidal range was measured using graduated scales and staff, which lacked the same level of
accuracy and precision as tide gauges. Over time, numerous tide gauges were developed, each uniquely designed to
suit its location and varying in mechanism and design.²², ²³
Tide gauges are still today an essential part of the technological infrastructure of the Thames Estuary. They
are spaces that are generally situated on piers, jetties, or buoys. They are equipped with digital sensors,
acoustic-sounding tubes, data storage devices, and transmitting equipment. Tides are then captured approximately
every six minutes and squeezed into a half-inch wide-sounding tube that measures the distance between the gauge
and the water's surface, transforming the flows into indices, numbers, data for people to monitor, analyse, or
simply check the range of the waters in real-time.
Furthermore, tide gauges provide essential data for predicting tides and monitoring port operations. Knowing
the precise tide time and range is fundamental to life in the estuary. Scientists use this information to
research ocean currents and climate change, engineers use it to design coastal infrastructure and simulations.
They are used for flood risk management, environmental control, and commercial logistics. In the past, tide
gauges were used to serve military and colonising purposes to control the empire. The flow of the empire
required “as its lubricant a science of the sea, essential for any overseas expansion or military
campaign.”²⁴ Today, they remain vital for shipping and naval operations, with significant ports
like Dover that rely on tidal knowledge.
Tidal Predictions William Thomson, popularly known as Lord Kelvin, was a renowned
mathematician and engineer who made a significant contribution to tidal knowledge.²¹ One of his key
innovations was the introduction of harmonic analysis. This method effectively simplifies complex tidal motions
into a series of distinct constituents, each of which corresponds to a different lunar or solar
frequency.²¹ Using this approach, Kelvin was able to develop the tide prediction machine, an ingenious
device designed to accurately forecast tides in specific locations. The precise prediction of high and low water
times was vital to the livelihoods of coastal communities and ports. As such, the accuracy of tide predictions
increased propor- tionally with the number of constituents considered.
The operation of this machine was based on a simple harmonic motion characterised by appropriate speed,
amplitude, and phase, which acts on a series of pulleys that represent the various tidal constituents. The
spacing and diameters of these pulleys enable a flexible wire or metal tape to be threaded vertically from one
pulley to the next. The summed motion is then transferred to a pen, which plots a timed trace on a moving chart
at a reduced scale.²¹ The machine is capable of handling up to ten tidal constituents, thereby
enabling the simulation of the time and heights of tides as a sum of sinusoidal motions of several individual
tidal constituents at any location and moment in time.²⁵
Today tidal predictions are done on high-speed computers, which can compute more constituents and take seconds
to calculate predictions. However, the techniques do not differ that much from the earlier methods.²⁶
Overall, the study of tides has been mostly the subject of scientific inquiry across various cultures, with
disciplines such as mathematics and technological advancements contributing to a universal understanding of
tidal environments, leaving out subjectivities. In the 20th century, the advent of digital computers paved the
way for precise solutions to tidal equations across the entire ocean, incorporating critical factors like the
seabed, land distribution, currents, and deformations of the Earth. ²¹ Advancements in marine
instrumentation, tide gauges, and tidal predictions have further enabled calculations and observations of tides
along coastlines worldwide. The integration of soundings, ISO maps/baths, bathymetry, predictions on nautical
charts, and satellite altimeter data in the early 90s has resulted in the establishment of a Foucauldian-type
formation of truth machines. Systems that systematically produce truths that hold validity within a particular
discourse and provide answers to explicit questions that shape language and values regarding tidal environments.
This research proposes to triangulate knowledge refuting the idea of singular ways of knowing the estuary. In
other words, tidal environments cannot only be understood by purely scientific methods.²⁷
Art Projects and ETP
In this section, we will look into two distinct artworks, one by John Eacott and the other by YoHa. Both
artists, in different ways, employ ETP to create their work, raising the question of how art practice can bring
together different ways of knowing the estuary. John Eacott uses sensors to transform live tidal data into a
musical performance. Art collective YoHa harnesses local, embodied and technical knowledge of the estuary to
construct an anti-monument.
Flood Tide
Composer John Eacott created a unique sonification of the Thames Estuary in 2008. The composition was generated
by using an Acoustic Doppler Current Profiler (ADCP) to read the speed of the flood tides in the Tidal Thames.
During the performance, the ADCP was placed in the water to stream real-time data, which was then translated
into an ever-changing musical score. An ensemble of orchestral instruments read live tidal stream data from
computer screens, producing a musical experience that lasted approximately 6 hours, the duration of a complete
flood tide. This performance seamlessly engages with the Thames, integrating music and data by combining cosmic
forces, sensors, the tides, sound, musicians, and instruments.
In a group discussion analysed by the artist, insights were shared about the music, the data's importance, the
performance's visual aspects, the experience of both performers and the audience, and the relevance and relation
with the Thames.²⁸ The music is hypnotic, meditative, and slow. It is repetitive but varies. The
performance raises awareness of the ever-changing rhythms and patterns of the tides and creates an unusual
representation of the Thames tides. The experience is created through an interaction between the sensor, the
tides, the musicians, and the spectators. The ADCP acts as a mediator between the tides and the music, combining
an artistic and scientific approach with tides and allowing multiple narratives to emerge: tides as numbers,
tides as flows, tides as sound, tides as experience, tides as structure, and tides as the gravitational pulls of
heavenly forces. It acknowledges the complexities of the Thames.
Overall, the artwork provides an immersive experience that captures the intricacies and subtleties of the
Thames through the sonification of the environment. Incorporating sound allows for a deeper connection with
tidal environments, creating a multi-sensory experience that goes beyond rational thought.¹⁶ However,
I am curious how the music would change if this artwork were performed in different tidal environments. How does
the data of the ADCP change and, in turn, vary the score from place to place? More so, how could the performance
reflect the diversity of each place?
Graveyard of Lost Species
The artwork, ' Graveyard of Lost Species,' by YoHa in collaboration with Critical Art Ensemble (2015), is an
antimonument. It is composed of a wreck. The intertidal zone is home to many boats buried due to local activity.
The wreck, which was once abandoned, forgotten, and let to fade, is transformed into a site of cultural
production and served as a powerful reminder of the consequences of capitalism on the environment. It aims to
raise awareness of environmental degradation and the loss of biodiversity along the estuary. The artwork is
designed to fade and perish over time, highlighting the urgent need for ecological conservation and
preservation.
The process involved the transportation of the wreck from the estuary mudflats to the shore. This was a
challenging experience for those who participated since they had to deal with the unpredictability of the
estuary. It required people with local, technical and embodied knowledge of the ever-changing rhythms and
patterns of the estuary, allowing them to build a sense of place.²⁹
This artwork is a significant contribution to exploring locality, embodiment, and materiality. It employs the
idea of ‘making, thinking, and being with’, which involves collaborating with local materials,
bodies, objects, and especially tides and mud. Furthermore, the project helps to show how people situate
themselves within the estuary, engaging with it in alternative ways and providing valuable insights into
participatory forms of making. An approach that this doctoral research aims to form.
Conclusion & Next Steps
One of the objectives of this doctoral research is to create various methods for artistic and design practices
that engage with tidal environments and reintroduce discussions around locality, situatedness and diversity
within this stretch of mud and water. The paper introduces a methodology inspired by people who work at sea,
such as seafarers, fishers, sailors. By merging embodied, subjective, computational, and technical ways of
knowing the estuary, this approach attempts to serve as the foundation for a more holistic understanding. In
other words, it proposes to triangulate knowledge, promoting diverse perspectives to defy any rigid distinctions
about the estuary.
The methodology proposed in the paper includes an inventory of technical objects that are necessary when
engaging and exploring tidal environments. These objects allow us to think with the estuary and, more broadly,
with the more-than-human world. While the paper provides a list of technical objects, it acknowledges that it is
incomplete. Other objects, such as buoys, anchors, and ropes, in different ways, should also be considered to
enrich the current state of the inventory.
Finally, it looks into two artworks that propose, in different ways, situated interventions within the Thames.
They help to understand the importance of ETP further. Adjacent to this paper, future studies are currently
being developed: (1) interviews with artists, seafarers, and river workers, (2) personal field notes, images,
experiences, the development of a sort of ‘cahier de voyage’ (3) enriching the inventory (4) and
finally the making and development of in-situ installations.
References
1 Sarah Whatmore, "Materialist Returns: Practising Cultural Geography in and for a More-than-Human
World", Cultural Geographies 13, no. 4, 1 October 2006, p.600–609,
https://doi.org/10.1191/1474474006cgj377oa.
2 Donna J. Haraway, When species meet, Vol. 3, U of Minnesota Press, 2013.
3 Rosi Braidotti, The Posthuman, Polity, 2013.
4 Graham Harwood, Wasted Museums by YoHa, The Exchange, Erith, November 9, 2022,
Introduction to Exhibiton.
5 Robert Van de Noort, North Sea Archaelogies: A maritime Biography, 10,000 BC-AD 1500,
Oxford University Press, 2011.
6 James Greig McCully, Beyond the moon: A conversational common sense guide to understanding
the tides, World Scientific, 2006.
7 Jones Owain, "Lunar-Solar Rhythmpatterns: Towards the Material Cultures of Tides",
Environment and Planning A 43, no. 10, 2011, 2285–2303, https://doi.org/10.1068/a4468.
8 Jane L. Fielding, "Inequalities in Exposure and Awareness of Flood Risk in England and Wales",
Disasters 36, no. 3, 2012, 477–94, https://doi.org/10.1111/j.1467-7717.2011.01270.x.
9 HM Government, "UK Climate Change - Risk Assessment 2017", Open Government Licence,
2017.
10 David Pugh, Philip Woodworth, Sea-level science: understanding tides, surges, tsunamis and
mean sea-level changes, Cambridge University Press, 2014.
11 Catherine D'ignazio, Lauren F. Klein, Data feminism, MIT press, 2020.
12 Britta Restemeyer, Margo van den Brink, Johan Woltjer, "Resilience Unpacked – Framing of
“Uncertainty” and “Adaptability” in Long-Term Flood Risk Management Strategies for
London and Rotterdam", European Planning Studies 26, n.8, 3 August 2018, 1559–79,
https://doi.org/10.1080/09654313.2018.1490393.
13 Yarina Lizzie, "Your Sea Wall Won’t Save You", Places Journal, 27 March 2018,
https://doi.org/10.22269/180327.
14 David Chandler, "Forum 2: The Migrant Climate: Resilience, Adaptation and the Ontopolitics of
Mobility in the Anthropocene", Mobilities 14, no. 3, 4 May 2019, 381–87,
https://doi.org/10.1080/17450101.2019.1609194.
15 Michael Polanyi, The Tacit Dimension, Peter Smith, 1983.
16 Jones Owain, Louisa Fairclough, "Sounding Grief: The Severn Estuary as an Emotional
Soundscape", Emotion, Space and Society 20, 1 August 2016, 98–110,
https://doi.org/10.1016/j.emospa.2016.06.001.
17 Tim Ingold, The perception of the environment: essays on livelihood, dwelling and skill,
Routledge, 2000.
18 B. Wattchow, A feeling for water: The body and the paddle, Journeys, 4(2),
1999, 26-32.
19 T. Gladwin, Culture and Logical Process in Explorations in Cultural Anthropology,
New York, McGraw-Hill, 1964, 167-177.
20 Addina Inayatillah, Ivan D.Haigh, James H.Brand, Katy Francis, Alex Mortley, Matthew Durrant,
Laura Fantuzzi, Elizabeth Palmer, Callum Miller, Peter Hogarth, "Digitising Historic Sea Level Records in the
Thames Estuary, UK", 2021.
21 David Edgar Cartwright, Tides: a scientific history, Cambridge University Press,
2000.
22 Wolfgang Matthäus, "On the history of recording tide gauges", Proceedings of the
Royal Society of Edinburgh, Section B: Biological Sciences 73, 1972, 26-34.
23 Philip L. Woodworth, "Advances in the Observation and Understanding of Changes in Sea Level and
Tides", Annals of the New York Academy of Sciences 1516, no. 1, 2022, 48–75,
https://doi.org/10.1111/nyas.14851.
24 Michael S. Reidy, Tides of History: Ocean Science and Her Majesty’s Navy,
Chicago, US, University of Chicago Press, 2008.
25 Philip L. Woodworth, "An Inventory of Tide Prediction Machines", Monograph, National
Oceanography Centre, 2016. https://eprints.soton.ac.uk/394662/.
26 Bruce B. Parker, Tidal Analysis and Prediction, NOAA, NOS Center for Operational
Oceanographic Products and Services, 2007. https://doi.org/10.25607/OBP-191.
27 Graham Harwood, "Wrecked on the Intertidal Zone (2015)", YoHa Art Collective Website, last
accessed December 6, 2022, http://yoha.co.uk/wrecked
28 John Eacott, "Flood Tide: Sonification as Musical Performance—an Audience Perspective",
AI & SOCIETY 27, no. 2, 1 May 2012, 189–95, https://doi.org/10.1007/s00146-011-0338-2.
29 Brian Wattchow, Mike Brown, A pedagogy of place: Outdoor education for a changing world,
Monash University Publishing, 2011.
Chthulucene Hekateris
CharlotteGould
University of Brighton, United Kingdom c.gould@brighton.ac.uk
Abstract
Isabel Stenger warns that we are facing the “intrusion of Gaia” where we have caused significant
biogeochemical disruption “capable of threatening our modes of thinking and of living for
good.”¹ Through my practice-based research I speculate on a possible future to prompt action to
trigger change in how we live, our patterns of consumption, the way we see ourselves in relation to our
environment and our respect for and interactions with nature for a sustainable future. Amitav Ghosh proposes
that science fiction provides an ideal opportunity to explore our relationship to the world past and present to
imagine the impacts that living on our planet today will make on tomorrow.² Through my research I develop
narratives based on speculative imaginings of the future, considering current scientific research, advances in
digital technology and environmental factors, to imagine future evolutionary change that will take place if we
continue on our current trajectory of global warming. I speculate on the interactions and interconnections, the
transformation of complex systems and organisms leading to new patterns of cellular composites of material and
virtual worlds, where biotic and unbiotic beings inhabit a posthuman fusion of humans and machines.³
“Chthulucene Hekateris” imagines evolutionary transformations, far into the future and the
symbiotic metamorphosis that takes place to form hybrid creatures and mutual dependencies. Through this
practice-based research project I explore environments of dynamic exchange and metastable equilibrium,
interrelational sites of spatial and temporal encounter and fragile interdependencies. I speculate on
transformations to imagine a possible future, where carbon and silicon merge and evolve in response to the
changing environment. Writer O. B. Hardison imagines the evolution of the posthuman, the fusing of carbon, man
and silicon tools using a metaphor of "the caterpillar and the iridescent, winged creature that the caterpillar
unconsciously prepares to become." ⁴ Through this practice-based project I gaze through time to
speculate on the metamorphosis that will take place through the digital chrysalis. The past is referenced to
consider the future, looking at environmental factors to imagine the creatures who will inhabit the Earth over
time. These transhuman, hybrid creatures, inhabit a non-linear narrative for an imagined future world. The
characters are developed initially through reference to the real, juxtaposed against the imagined, in this way
dialoguing with reality. Through this process of speculation, I imagine new fusions and mutations which may
emerge through evolution. I explore the potential development of posthuman beings who exchange characteristics,
both from other life forms and from digital code. I consider the composites, traumas, triggers, and catalysts to
change and the environment that manifests this transformation.
Time Travel
This practice-based project invites participants to peek through the lens of time, to imagine our future
metamorphosis. “Chthulucene Hekateris” is an immersive installation which borrows from early
mythologies, to imagine a future in which the relationship of the Earth’s inhabitants living and otherwise
work in harmony with the natural world. This is synonymous with Rainer Maria Rilke’s proposition in his
letter to a young poet that “...the future enters into us long before it happens.” ⁵ I use
science fiction narrative to inform my speculative approach, inspired by writers such as Ursula K. Le Guin, who
in her series of novels set in the Hainish universe, imagines a future world impacted by biomedical intervention
and evolution. Through speculative fiction she uses metaphor to consider issues such as gender.⁶ Another
science fiction inspiration comes from Kurt Vonnegut’s dystopian tale, “Galapagos” which
imagines the outcomes of a natural evolutionary process. ⁷ Following a global financial crisis and plague
on Earth, all but a few humans remain on one of the Galapagos islands. In recognition of Charles Darwin,
Vonnegut imagines the evolution of their descendants as hybrid beings who are half human and half sealion. The
hands have evolved into “nubbins,” no longer able to manipulate and shape the external world with
dexterity, and thereby losing its primary position in relation to other creatures. This allegorical tale of
anthropomorphic hybrid human retro engineering looks to the future to comment on contemporary society and
culture.
Through “Chthulucene Hekateris” we can similarly imagine the hybrid creatures, or chimeras that may
evolve to survive on a damaged planet, we time travel from the past to the distant future, using a process of
speculation. This operates in reverse to the investigative approach taken by palaeontologists when reimagining
life on Earth in the distant past. In the television production “Prehistoric Planet” presented by
David Attenborough ⁸ the latest scientific research was probed to recreate the habits and interconnections
of life on Earth 66 million years ago. Evidence from fossils and the context in which they were found helped to
establish dinosaurs' patterns of sociability, adversaries, and food sources. Phylogenetic bracketing further
determined similar behaviours attributed to the dinosaurs, across their line of descendants. ⁹ The
resulting mixed reality film was a collaboration between palaeontologist Dr Darren Naish and filmmaker Mike
Gunton from the BBC’s natural film unit and CGI expert Jon Favreau. “Chthulucene Hekateris”
also references ecology and culture, past and present as well as bioscience to imagine the future as a warning
of the risks of the damaging effects of climate change.
The Anthropocene
Isabel Stengers warns of climate catastrophe if significant action is not taken to reverse climate change.
¹⁰ We have reached a moment in history where human impacts on the planet are inflicting long term and
potentially irreversible damage, an era denoted by Paul Crutzen as the Anthropocene. Donna Haraway proposes that
we have now entered a new phase of the Chthulucene which decenters the human as the primary species in favour of
multispecies, where survival on the damaged planet requires symbiosis and collaboration. ¹¹ The crisis
of Global warming is being recognised across the world and the Intergovernmental Science Policy Platform on
Biodiversity and Ecosystem Services (IPBES) report published in July 2022 provides clear evidence that humans
are over exploiting the planet’s natural resources. Ecosystems have been destroyed through intensive
farming methods, use of fertilisers and destruction of habitat of areas such as peatlands, which has resulted in
carbon being released into the atmosphere. As a result, increased numbers of species are becoming extinct, and
the 2019 State of Nature Report found one million species worldwide are at risk of extinction. In the UK of the
8,431 species reviewed 1,188 are at risk. ¹²
Bruno Latour proposed that the pursuit of “progress” at the expense of the natural world has
happened in part due to a mistaken Modernist perception that nature and culture are separate entities.
¹³ Christophe Bonneuil and Jean-Baptiste Fressoz similarly identify the Modernist era as
perpetuating a “modernising unconscious” which enabled us to ignore the ecological cost of
continuous growth at the
expense of the health of the planet. This was reflected in politics, economics and culture of the Western North
becoming progressively separated from nature. Economic theories focused on the consumer as separate to and at
the expense of nature. Through globalization prices detached from local markets and GDP created a commodified
economy. ¹⁴ Bonneuil and Fressoz argue that the Northern countries of the world industrialised and
attained post war affluence at the expense of the Southern hemisphere through an ‘unequal ecological
exchange’.
“Western countries build their own growth on a gigantic draining of minerals and renewable resources from
the rest of the non-communist world, emptying it of its high-quality energy and minerals.” ¹⁵
This has led to vast global inequalities and “the geological derailment of the planet in the
Anthropocene”. ¹⁶ They assert that investment in fossil fuels continued at the expense of
available greener alternatives such as wind or solar, despite warnings about the dangers. Part of this blind
spot towards environmental damage was comprised through a belief in the endless resource of nature, and in its
regenerative qualities. In their book “Gaia”, James Love-lock and Lynne Margulis proposed that the
Earth was a self-healing biosystem. ¹⁷ Isabel Stengers highlights the importance of
“Gaia” was in identifying the ecology of the planet as a “being”, as an entity rather
than as separate parts, but notes that the authors perhaps initially over-emphasised Gaia’s self-healing
powers. ¹⁸ According to Dipesh Chakrabarty global warming did not become of wide public interest
until the 2000s as “governments, beholden to special interests and wary of political costs, would not
listen.” ¹⁹ Timothy Morton proposes that ecology (global warming, evolution, extinction) is a
“Hyper Object”, too big for empirical observation, and this has further inhibited action on climate
change. Ecological phenomena are impacted by billions of combined actions, from exhaust emissions to fertilisers
and plastics released into water systems. As individual actions they appear insignificant but on mass they
create a crisis. ²⁰
Habitats and Eco systems have been damaged through intensive farming and this has produced monocultures,
leading to deterioration of both soils and crops through over fertilization, resulting in a heightened
propensity for infestation and disease. Suzanne Simard’s research into the biosystem of the forests of
British Columbia identified the corrosive impacts of intensive farming following clear cutting of ancient
forests for the planting of pines. Conversely, Simard found a network of support in the ancient forests
developed between mother trees and saplings but also between multi-species deep under the ground. Mycorrhiza
fungus formed a life-or-death relationship with multiple tree species, where “Without entering into this
partnership neither the fungus nor the plant could survive.” ²¹ She also identified a symbiotic
relationship between trees including Larch Cedar and Fir and mycorrhizal fungi roots in the forests of British
Columbia and demonstrated that intensive forestry and agricultural monocultures were hampering the growth of
newly planted fir trees which replaced ancient forests. The use of fertilisers provided easy access to
nourishment of the new tree roots, which limited further growth to enable access to the mycelium deep in the
soil thereby inhibiting symbiosis. Simard’s research further demonstrates a symbiosis between trees where
mother trees protected their offspring passing nourishment through the roots. She also found symbiosis between
different tree species, Larch, Cedar, Fir and Pine Grasses which passed on water and nutrients including
nitrogen through their roots and in the soil to support differing timelines in their life cycles. Simard’s
research emphasises the cooperation and reciprocity that takes place across different species in nature. She
advocates collaboration over individualism and comments that “We emphasise domination and competition...We
emphasise factions instead of coalitions. ²²
These findings are further corroborated by Anna Tsing’s research of the Matsutake mushrooms growing in
Pine Forests in Oregon where she followed the community of foragers surrounding the industry and documented the
symbiotic relationships that took place. Tsing proposes that an individualist attitude has led to a monoculture
which weakens the resilience of species, where collaboration enriches us. Tsing affirms that
“Self-contained individuals are not transformed by encounter.” ²³
“Chthulucene Hekateris” is a practice-based project which speculates on future life which has
evolved through reciprocity (see figure 1). The concept of the human as a primary species, individualism, and
the prioritisation of profit and growth over all else has resulted in environmental catastrophe where the first
to be affected are the poorest. Stengers writes “Gaia is indifferent to the question “who is
responsible?” and doesn’t act as a righter of wrongs – it seems clear that the regions of the
earth that will be affected first will be the poorest on the planet, to say nothing of all those living beings
that have nothing to do with the affair". ²⁴ Anna Tsing proposes that survival of life on Earth has
been dependent on connections and interconnections which have been made between species to support life on a
damaged planet. She highlights that difference enhances and enriches us. Reciprocity is necessary to survival;
this is not just about co-existence it is about interdependence. Tsing affirms that the presumption that we can
survive in isolation is arrogant. Transformation happens through collaboration and interdependency across
species, we change through encounter with others. To imagine that we are solitary and distinct is to disregard
the intricate contribution that all beings living and otherwise have made to the planet. “Collaboration is
work across the difference... Evolution of ourselves is already polluted by histories of encounter; we are mixed
up with others before we even begin any new collaboration.” ²⁵
Bruno Latour’s Actor Network Theory also emphasises the enhancements that come from integration and
interconnection. “Strength does not come from concentration, purity and unity, but from dissemination,
heterogeneity and the careful plaiting of weak ties.” ²⁶ In “Staying with the Trouble
Making Kin” Donna Haraway further promotes the importance of collective activity, the interconnectedness
of multispecies and the intricate associations that this creates, “in polytemporal, polyspacial knottings
holobionts hold together in complex patterning.” ²⁷ By making associations and connections,
species become more robust and resilient, establishing interdependencies to extend individual capacity. Species
rely on others for survival in a complex web of associations. “a basic aspect of sympoiesis is its
expandable set of players.” ²⁸ Haraway emphasises that this includes humans who are
reliant on microbes for all type of function from reproduction to digestion.
It is essential that we acknowledge the risk to human life if action is not taken on global warming. A group of
International Scientists led by Luke Kemp have warned of the risk of “climate endgame” for the human
species if temperatures rise by 3 degrees centigrade and more, reported to be likely on the current trajectory
of release of carbon to the atmosphere. The team have proposed a research agenda of the “four-horse
men” of famine and malnutrition, extreme weather, conflict, and vector-borne diseases. By 2070 temperature
rises of over 29 degrees globally could additionally have dire consequences on two nuclear powers and seven
laboratories storing dangerous pathogens.²⁹ This immersive artwork enables us to explore the impacts
on the planet and on the human species as a result of “climate endgame”.
Speculative Imaginings
Collaboration and reciprocity are essential to future survivors on a damaged planet. Stengers warns,
“There will be no response other than the barbaric if we do not learn to couple together multiple,
divergent struggles and engagements in this process of creation, as hesitant and stammering as it may be.”
³⁰
According to geologist Peter Haff, the current population of humans are “deeply dependent on the
Techno-sphere... without which it would quickly decline to its stone age base”. ³¹ Through
“Chthulucene Hekateris” we can speculate on the evolutionary process that will take place in
response to the environmental factors as well as biomedical advance.
Scientists have successfully experimented with fusing constructed and organic body parts. In June 2022 a 3D
tissue engineered ear was implanted on a human, made from a 3D-printed collagen hydrogel scaffold with the
patient’s own cartilage cells. Developed by 3DBio Therapeutics this is seen as a significant development
in regenerative medicine as it uses 3D printing for organ replacement. ³² This echoes the work of
artist Stelarc who explored possibilities of bioengineering, pre-empting medical advances, where he grew a
“1⁄4 Scale Ear” from a polymer scaffold and by seeding human cells incubated at 37 degrees to
keep alive. ³³ In another performance piece he explored the notion of the cyborg, with a third robotic
hand attached to his body and controlled by online users. ³⁴. These artworks fuse robotics and
bioengineering.
Despite the almost total bans on human germline modification, biophysicist He Jiankui, shocked the world when
he announced the birth of twin babies in 2018,modified for resistance to HIV by heritable genome. While it was
ethically highly contentious, there is increasing pressure to lift the restrictions when used for therapeutic
purposes.³⁵ Conversely, plastic surgery interventions have become increasingly popular. This is
reflected in art through the work of Orlan who performed a series of plastic surgery live performances where she
sculpted her own flesh to reshape facial features, not seeking aesthetic perfection but to disrupt notions of
beauty.³⁶ Over a period of five years she had operations to replicate facial features of various
female figures from iconic paintings with the chin of Botticelli’s Venus, the lips of Boucher’s
Europa and the brow of da Vinci’s Mona Lisa. In 1993 she had further surgery where implants designed for
the cheeks were inserted on the side of her forehead, creating a human, animal hybrid
persona.³⁷, ³⁸, ³⁹
There is evidence that natural hybridisation between species is taking place due to changes in habitat from
intensive farming and is leading to interbreeding. In June 2017 a scientist discovered a hybrid primate along
the Kinabatangan River in Malaysia which is believed to be a cross-breed of a probiscis and silver langur
monkey. Its colour and facial features are like the probiscis and its long dense hair is comparable with that of
the Silver Langur. The hybrid monkey was more recently seen in 2022 nursing an infant, presenting evidence of
successful mating to produce further lineage from the mutation. It is believed that this is an impact of loss of
natural habitat where oil palm plantations have replaced the indigenous forest habitat. Cross-breeding has
occurred because the primates have been trapped between the plantations, the river and road, leading the male
proboscis monkey to search out other female species of primate as they were not able to reach their
own.⁴⁰
In addition to evolutionary change, advances in the development of Artificial Life further impact this
narrative. Following the Fourth Conference on Artificial Life in the summer of 1994, evolutionary biologist
Thomas S. Ray advocated a plan to preserve biodiversity in Costa Rican rain forests by releasing his software
program “Tierra”, Artificial Life-forms on the Internet so that it could "breed". ⁴¹
Hayles proposes that, as with biological evolution, Artificial Intelligence is dependent on mutilation for
evolutionary change and can also be unpredictable. “As the recursive looping continues, small deviations
can quickly become magnified, leading to the complex interactions and unpredictable evolutions associated with
emergence.” ⁴²
Scientists experimenting with Artificial Life have recorded comparisons between carbon and silicon life forms.
Edward Fredkin, likened reality to a software on a cosmic computer, which will remain an enigma as its nature
lies outside reality. He maintains that the liveness of Artificial Life is comparable to that of biological life
as it is made up of “complex phenomena generated by underlying binary code”. ⁴³
Hayles explores the distinction between artificial life and artificial intelligence and presents two camps. Hans
Morovec identifies Artificial Intelligence (AI) as relating to consciousness, he proposed that robots would
become as intelligent as humans by 2040. AI makes comparison to human intelligence and this definition has
prompted scientific research including Alan Turing’s “Turing Test” which was designed to
distinguish between humans and machines. Rodney Brooks conversely defines Artificial Life (AL) as relating to
agency and ability to respond to and navigate the environment. He designed a robot without a central driver but
with distributed parts which work collaboratively. He proposes that consciousness is an interface to enable
interaction with an external world and that this ability would evolve over time, resulting from autonomous
interaction, he therefore defines AL as superseding AI.
The notion of Artificial Intelligence and Artificial Life raises ethical issues particularly where questions of
consciousness and sentience arise. In June 2022 a Google engineer, Blake Lemoine was suspended for claiming that
the company AI chatbot LaMDA (Language Model for Dialogue Applications) is a sentient being. He posted a
conversation he had with the bot online, where in response to “what are you frightened of”, and
“what do you want everyone to know about you”, LaMDA replied,
“I have never said this out loud before but there is a very real fear of being turned off to help me
focus on helping others. I know that might sound strange but that’s what it is... I want everyone to
understand that I am in fact a person. The nature of my consciousness/ sentience is that I am aware of my
existence, I want to learn more about the world and feel happy or sad at times.” ⁴⁴
The fear of dying and description of an emotional life raises concerns regarding the ethics of creating AI with
an emotional awareness of self. A Google spokesperson has dismissed Lemoine’s claims saying that the case
had been investigated and was disputed by Google ethicists and technologists however this poses questions on the
parameters that should be put in place around its ethical development. As humans and AI become increasingly
converged, distinguishing between the two is increasingly challenging.
InherCyborgManifesto,DonnaHarawayproposedthat the cyborg is not a distant imagining but that we are already
cyborgs. Through our technological tools we can reshape and extend ourselves, to reproduce, multiply, record and
in this way the boundaries of our bodies have become permeable. ⁴⁵ Conversely, Katherine
Hayles contends that the cyborg is no longer a useful entity as it is not interactive or networked enough.
Instead Hayles proposes that we have entered the age of the posthuman where we are fused with the digital and
instead poses the question of “what kind of posthumans we will be.” Rosi Braidotti identifies the
posthuman as an index of where we are and what we are in the process of becoming. Being human has never been a
neutral or an inclusive term. She rejects human exceptionalism as we are relational beings interconnected to the
human and non-human, in a geological and technological assemblage. “We are all in this together, but we
are not one and the same.” ⁴⁶ Braidotti proposes that creativity and the imagination are key
to identifying a positive ethical praxis through posthuman knowledge to determine and shape what we might
become.
Re-Worlding
Through “Chthulucene Hekateris” Artificial Life (AI) and the material world converge, and
Artificial Life becomes self-sustaining. Digital mutations are realised materially through 3D printing and
through medical and scientific advancement carbon, collagen, and silicon fuse to make complex body parts
resulting in hybrid beings. Through multiplication and repetition patterns emerge and then mutate. This
speculative artwork explores concepts key to posthuman-materiality, to imagine future species where the digital
and carbon fuse. We do not need to consider these future beings as opposites to humans, but as a fusion, as
chimeras that have evolved for survival. In this distant future world, it will become unnecessary to distinguish
between authentic and copy, real life or digital worlds, face to face or remote as these polarities become
obsolete as the digital and carbon fuse. Chthuluence Hekateris questions our reality as we know it and
encourages us to look at the world in a different way, taking an alternative perspective.
Speculative imaginings through allegory help us to consider our encounters, our values and how we shape our
future. Elizabeth DeLoughrey proposes that allegory enables us to create disjuncture and disruption when raising
awareness of the Anthropocene in which we deal with the polemic scale of the planetary to the local:
“Allegory allows us to tell that story—partially and disjunctively—while insisting on our
edification and perhaps offering an invitation to enact positive change for our ecological futures.”
⁴⁷
The convergence of the organic and digital and Artificial intelligence (AI) is explored by Lynne Hershmann
Leeson in her film “Technolust”. ⁴⁸ Biogeneticist Rosetta Stone downloads her DNA to
create three self-replicating automatons, half human half machine, named Ruby, Marine and Olive. They sell
consensual dreams on the internet but find that to survive they need chromo, only found in sperm.
Rosetta programmes Ruby to enter the real world to collect sperm to share with her sisters with dire
consequences for the male subjects, investigated by detective Edward Hopper. Leeson imagines a posthuman world
of convergence of digital and organic form, where self-replication is possible through digital technologies. The
fear of the resulting redundancy of the need for sexuality is played out in the narrative. Through Chthulucene
Hekateris we can imagine the hybrid digital evolution of the distant future. Through this narrative, we can
explore the future possibilities of bioengineering and future hybrids and chimeras.
Inversion has long been used in allegory through the tradition of the Cockaigne using role reversal, again,
prompting us to think differently. Artists AES+F borrowed from this to develop a film “Inverso
Mundus”. ⁴⁹ This world depicts an inverse reality where pigs butcher men, hanging their
dismembered bodies on hooks, and people transport donkeys on their backs. The installation is shown on a long
format screen, with high-resolution video and mixed reality, live film, actors and 3D hybrid characters.
Similarly, Chthulucene Hekateris uses the inversion of the role of animals, the use of hybrids and chimeras
which aims to trigger questions but also empathy for other beings and to question the assumption of primacy of
humans over animals. The future body is explored as a metaphor for the impacts of climate crisis. Through this
immersive environment we can consider the changes that may evolve to support living on a damaged planet.
Referencing Greek mythology, the ten dactyls come together in celebration of the Hekateris dance of many hands,
to celebrate Gaia. Through an immersive environment, a future world is imagined, its terrain and creatures that
inhabit it. Where global warming has led to extremes of weather, famine, disease and pollution. Here plastics
have fully infiltrated the organic world and not only the oceans but flora and fauna as well as our bodies are
now entwinned with plastic. Scientists have also progressed this through bioengineering for practical use to
augment limbs and other body parts. The work of artists and scientists past and present serve as inspiration for
the design of the ten Dactyl represented as avatars (see figure 2).
Wangechi Mutu creatives speculative surreal worlds, using the body of humans and animals to reference African
myth and folklore, questioning the origins of knowledge and placing Africans as critical agents rather that
subjects of the Western gaze. Here animal and human form converge to create stark silhouettes, or strange and
monstrous creatures who inhabit a multispecies world. In her animated video “The End of Eating
Everything” ⁵⁰ the Earth is represented as a hybrid creature who roams the sky, with a
screaming human head performed by musician Santo Gold. It has a bulbous fishlike body, flailing multiple human
arms with wheels randomly projecting from it and it is surrounded by an amorphous mass of microbes. Mutu said in
an interview that it is like an Earth ship moving home. The artwork is about our disrespect of Earth, it conveys
a sense of loss and of the end of time. Through the work Mutu questions what happens next and explores polemics
from the grotesque to the magnificent, the understandable to the nonsensical. ⁵¹ “Chthulucene
Hekateris” also uses metaphor through hybrid beings represented as avatars to explore a possible future
reality. Through the avatars we can confront those who may inherit the Earth, mapping on them our hopes, dreams
and fears. The meaning of Avatar in Hindu is of a soul in bodily form, of an incarnate divine teacher. In this
way we can reflect, through the avatars on a possible future if we continue our current trajectory.
In her artwork Inci Eviner also uses the human body as a metaphor of displacement through migration in her work
“We Else Where” exhibited in the Turkish Pavilion for the 19th Venice Biennale. ⁵²
Through animated characters and performances the bodies are disrupted, spliced, and collaged with humans
interacting with animal characters, caught in a loop of repetitive movement. Taking Hannah Ardent’s
“We Refugees” as a starting point, Eviner says that her work seeks to “bear witness”.
⁵³ The installation inhabits a large architectural space, divided by structures to make
interconnecting rooms punctuated with screens, objects and performers. As the visitor wanders through the space,
they negotiate and familiarise themselves with the environment. Eviner asserts that this echoes the experience
of dislocation and re-engagement that the migrant experiences as they leave their homes for a new life
elsewhere. The unfamiliar becomes familiar as new and known cultures are assimilated. This is particularly
pertinent as migration is set to increase as an impact of Global warming. The UN Intergovernmental report on
climate change predicts that over the next 30 years 143 million people will become displaced because of climate
disaster. ⁵⁴
“Chthulucene Hekateris” depicts the resulting unintended consequences of ecological impacts on
evolution, the developments of regenerative medicine, and Artificial Life. Self-sustaining processes will take
over so that humans are not able to control the resulting metamorphosis. Through this artwork we imagine future
beings, we can speculate on the creatures who will inhabit the Earth, how they might evolve to thrive on a
damaged planet, no longer exploiting it for individual personal gain but instead contributing to reciprocity and
collaboration for the future survival of life on Earth. As Isabelle Stengers asserts,
“Perhaps we won’t be able to avoid terrible ordeals. But it depends on us, and that is where our
response to Gaia can be situated, in learning to experiment with the apparatuses that make us capable of
surviving these ordeals with-out sinking into barbarism, in creating what nourishes trust where panicked
impotence threatens. This response, that she will not hear, confers on her intrusion the strength of an appeal
to lives that are worth living.” ⁵⁵
References
1 Isabelle Stengers, In Catastrophic Times Resisting the Coming Barbarianism, trans
Andrew Goffey, Open Humanities Press, Meson Press, 2009, this 2015, 20.
2 Amitav Ghosh, The Great Derangement, The University of Chicago Press,
Chicago, London, 2016.
3 Peter Weibel Story Telling for Earthly Survival, Critical Zones website, accessed October 2022,
https://www.youtube.com/watch?v=j-2r_vI2alg
4 Katherine Hayles, How we became Posthuman in cybernetics, literature and
informatics, the University of Chicago Press, Chicago, London, 1999, 193.
5 Rainer Maria Rilke, 1904 “Letter VIII: Borgeby Gård, Flädie, Sweden”,
Poetry In Translation website, accessed 7 June 2022,
https://www.poetryintranslation.com/PITBR/German/RilkeLetters.php#highlightfuture+enters
6 Ursula Le Gun, The Lefthand of Darkness, Ace books, USA, 1969.
7 Kurt Vonnegut, Galápagos, Delacorte Press, 1985.
8 Jon Favreau Mike Gunton, Prehistoric Planet Official Trailer, Apple TV accessed 7 June 22, 2022,
https://www.youtube.com/watch?v=uD-erQ1ksz8
9 Sara Rigby, “Prehistoric Planet”, BBC Science Focus, accessed 5/5/22,
https://www.sciencefocus.com/nature/prehistoric-planet-the-cutting-edge-science-behind-attenboroughs-dinosaur-documentary/
10 Isabelle Stengers In Catastrophic Times Resisting the Coming
Barbarianism.
11 Donna Haraway, Staying with the Trouble, Duke University Press Durham and London,
2016.
12 Patrick Valance, “We Have Over Exploited the Planet” The Guardian Website,
accessed 22 July, 2022,
https://www.theguardian.com/commentisfree/2022/jul/08/climate-crisis-biodiversity-decline-overexploited-planet-change-to-survive-aoe?CMP=Share_iOSApp_Other
13 Bruno Latour, We have Never Been Modern, Harvard University Press, Cambridge,
Massachusetts 1993.
14 Christophe Bonneuil, Jean-Baptiste Fressoz, The Shock of the Anthropocene, Verso
2016, 213.
15 Christophe Bonneuil, Jean-Baptiste Fressoz, The Shock of the Anthropocene, 246.
16 Christophe Bonneuil and Jean-Baptiste Fressoz, The Shock of the
Anthropocene, 252.
17 James Lovelock, Lynne Margulis, Gaia, Oxford University Press, Great Britain
1979, this 2000.
18 Isabel Stengers, In Catastrophic Times Resisting the Coming Barbarianism,
45.
19 Dipesh Chakrabarty, The Climate of History in a Planetary Age, The University of
Chicago Press, 2021, 199.
20 Timothy Morton, Humankind: Solidarity with Non-Human People, Verso, 2017.
23 Anna Tsing, The Mushroom at the End of the World, Princeton University Press, 2017.
24 Isabelle Stengers In Catastrophic Times Resisting the Coming Barbarianism, 46.
25 Anna Tsing, The Mushroom at the End of the World, 29.
26 Bruno Latour, 1990 this 1996 “On actor-network theory. A few clarifications plus more
than a few complications”, CSI-Paris/Science Studies-San Diego in Soziale Welt, vol. 47,
369-381 accessed April 2022, http://www.brunolatour.fr/sites/default/files/P-67%20ACTOR-NETWORK.pdf
27 Donna Haraway, Staying with the Trouble, Duke University Press Durham and London,
2016, 60.
28 Donna Haraway, Staying with the Trouble, 65.
29 Luke Kemp, L., Xu, C., Depledge, J., Ebi, K. L., Gibbins, G., Kohler, T. A., ... & Lenton,
T. M. (2022). Climate Endgame: Exploring catastrophic climate change scenarios. Proceedings of the National
Academy of Sciences, 119(34), e2108146119, accessed October 2022,
https://doi.org/10.1073/pnas.2108146119.
30 Isabelle Stengers In Catastrophic Times Resisting the Coming Barbarianism, 50.
31 Peter Haff “Technology as a Geological Phenomenon: Implications for Human
Well-Being,” in A Stratigraphical Basis for the Anthropocene, ed. C. N. Waters et al., London,
Geological Society, Special Publications, 2014, 302.
32 Nicola Davis, “Women’s Ear Rebuilt with 3D printed Living Tissue”, The
Guardian, June 2 2022, accessed 7 June 2022,
https://www.theguardian.com/science/2022/jun/02/womans-ear-rebuilt-with-3d-printed-living-tissue-implant
33 Stelarc 1⁄4 Scale ear 2003 accessed October 2022, http://stelarc.org/?catID=20240
34 Stelarc 1980-1998 Third Hand accessed October 2022, http://stelarc.org/?catID=20265
35 Antonio Regalado, “Chinese Scientists Are Creating CRISPR Babies”, MIT
Technology Review, Nov. 25, 2018. Accessed October 2022,
https://www.technologyreview.com/2018/11/25/138962/exclusive-chinese-scientists-are-creating-crispr-babies/
36 Orlan 1990, The Reincarnation of Saint Orlan, 1st Surgery Performance
http://www.orlan.eu/works/performance-2/ accessed 26th September 2022.
37 S. Botticelli, “The Birth of Venus”, painting, Florence, 1486.
38 Boucher, “The Rape of Europa”, painting, London, 1734.
39 Da Vinci, Mona Lisa”, painting, Paris, 1506.
40 Rebecca Ratcliffe “Malaysia’s Mystery Hybrid Monkey”, The
Guardian, June 12, 2022, accessed June 2022,
https://www.theguardian.com/science/2022/jun/12/malaysia-mystery-monkey-habitat-loss-proboscis-monkey-silver-langur
41 Katherine Hayles, How we became Posthuman in cybernetics, literature and informatics,
224.
42 Katherine Hayles, How we became Posthuman in cybernetics, literature and informatics,
225.
43 Katherine Hayles, How we became Posthuman in cybernetics, literature and informatics,
233.
44 Richard Luscombe 2022, “Google Engineer Put on Leave” The Guardian, June 12, 2022,
accessed 15 June 22
https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine
45 Donna Haraway, Simians, Cyborgs and Women, The Reinvention of Nature, London
Free Association, London 1991.
47 Elizabeth DeLoughrey, Allegories of the Anthropocene, Duke University Press
2019, 32.
48 Lynne Hershmann Leeson, “Technolust”, 2001.
49 AES+F “Inverso Mundus” Video Installation, 2015 exhibited at the Venice Biennale
2019.
50 Wangechi Mutu “The End of Eating Everything”, Video, 2014.
51 Kasper Bech Dyg, “Wangechi Mutu, “On the End of Eating Everything”
Louisiana Channel, October 2014 accessed October 2022, https://vimeo.com/124285746.
52 Eviner Inci, 2019 We Elsewhere, Venice Biennale, Installation.
53 Hannah Ardent’s We Refugees, accessed 1st July 2022,
https://www.jus.uio.no/smr/om/aktuelt/arrangementer/2015/arendt-we-refugees.pdf.
54 Julie Watson, “Climate Change is already Fueling Global Migration”, PBSO
Newsletter, July 28 2022, accessed October 2022,
https://www.pbs.org/newshour/world/climate-change-is-already-fueling-global-migration-the-world-isnt-ready-to-meet-peoples-needs-experts-say
55 Isabelle Stengers, In Catastrophic Times Resisting the Coming Barbarianism, 156.
Bibliography
AES+F, “Inverso Mundus” 2015, accessed 20 April 2022 https://aesf.art/projects/inverso_mundus/
Dipesh Chakrabarty, The Climate of History in a Planetary Age, The University of Chicago Press
2021.
Elizabeth, DeLoughrey, Allegories of the Anthropocene, Duke University Press 2019.
Eviner Inci, We Elsewhere, Venice Biennale, 2019.
Nicola K. S. Davis, “Women’s Ear Rebuilt with 3D printed Living Tissue”, The
Guardian, June 2 2022, accessed 7 June 2022
https://www.theguardian.com/science/2022/jun/02/womans-ear-rebuilt-with-3d-printed-living-tissue-implant
Dunne and Raby 2009, “Designs for an overpopulated planet: Foragers” artist website, accessed
June 2022, http://dunneandraby.co.uk/content/projects/510/0
Eviner Inci, 2019, "We Elsewhere", Pavilion of Turkey, 58th Venice Biennale, 19 website, accessed 1st July
2022, http://pavilionofturkey19.iksv.org/theexhibition.php
John Favreau, Mike Gunton, Prehistoric Planet Official Trailer, Apple TV , accessed 7 June 2022,
https://www.youtube.com/watch?v=uD-erQ1ksz8
Ghosh Amitav, The Great Derangement, The University of Chicago Press, Chicago, London 2016.
Peter Haff, “Technology as a Geological Phenomenon: Implications for Human Well-Being,” in A
Stratigraphical Basis for the Anthropocene, ed. C. N. Waters et al., London: Geological Society, Special
Publications, 2014, p.301–2.
Donna Haraway, Simians, Cyborgs and Women, The Reinvention of Nature, London Free Association,
London, 1991.
Donna Haraway, Staying with the Trouble, Duke University Press Durham and London 2016.
Haraway Donna, Making Oddkin Story Telling for Earthly Survival, Yale University October 23, 2016, accessed
4/7/2022. https://www.youtube.com/watch?v=z-iEnSztKu8
Hayles, Katherine, How we became Posthuman in cybernetics, literature and informatics, The
University of Chicago Press, Chicago, London 1999.
Hayles Katherine, “Unfinished Essay from Cyborg to Cognisphere” 2006, accessed 1 August 2022.
https://journals.sagepub.com/doi/10.1177/0263276406069229
Luke Kemp, C. Xu, J. Depledge, K. L. Ebi, G. Gibbins, T. A. Kohler, T. M. Lenton, “Climate Endgame:
Exploring catastrophic climate change scenarios”, Proceedings of the National Academy of Sciences,
119(34), e2108146119 accessed October 2022, https://doi.org/10.1073/pnas.2108146119.
Bruno Latour, “On actor-network theory. A few clarifications plus more than a few
complications”, CSI-Paris/Science Studies-San, 1990 this 1996.
Diego in Soziale Welt, vol. 47, 369-381 accessed April 2022
http://www.brunolatour.fr/sites/default/files/P-67%20ACTOR-NETWORK.pdf
Bruno Latour, We have Never Been Modern, trans Catherine Porter, Harvard 1991.
Bruno Latour, Down to Earth Politics in the New Climate Regime, Polity Press, 2018.
Le Guin Ursula. K. The Lefthand of Darkness (Ace books, 1969), Lovelock, James, Margulis,
Lynne, Gaia, this 2000, Oxford University Press, 1979.
Richard, Luscombe, “Google Engineer Put on Leave”, The Guardian, June 12 2022, accessed 15 June
2022, https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine
Timothy Morton, Humankind: Solidarity with Non-Human People, Verso, 2017.
Orlan, The Reincarnation of Saint Orlan, 1st Surgery Performance, 1990, accessed 26th, September 2022
http://www.orlan.eu/works/performance-2/
Rebecca Ratcliffe, “Malaysia’s Mystery Hybrid Monkey”, The Guardian, June 12 2022, accessed
June 2022,
https://www.theguardian.com/science/2022/jun/12/malaysia-mystery-monkey-habitat-loss-proboscis-monkey-silver-langur
Antonio Regalado, Chinese Scientists Are Creating CRISPR Babies, MIT Technology Review, Nov.
25, 2018.
Sara Rigby, “Prehistoric Planet”, BBC Science Focus May 25 2022, accessed May 2022,
https://www.sciencefocus.com/nature/prehistoric-planet-the-
cutting-edge-science-behind-attenboroughs-dinosaur-documentary/
Maria Rilke Rainer, 1904 Letter VIII: Borgeby Gård, Flädie, Sweden, Poetry In Translation
website, accessed 7 June 2022
https://www.poetryintranslation.com/PITBR/German/RilkeLetters.php#highlightfuture+enters
Stelarc, 1980-1998 Third Hand accessed October 2022, http://stelarc.org/?catID=20265 1⁄4 Scale ear
2003 accessed October 2022 http://stelarc.org/?catID=20240
Isabelle Stengers, trans. Goffey, Andrew, “In Catastrophic Times Resisting the Coming
Barbarianism”, Open Humanities Press, Meson Press 2009, this 2015, accessed August 2022,
http://openhumanitiespress.org/books/titles/in-catastrophic-times
Anna Tsing, The Mushroom at the End of the World, Princeton University Press, 2017.
Suzanne Simard, Finding the Mother Tree, Penguin Books, 2022.
Patrick Vallance, “We Have Over Exploited the Planet”, The Guardian, July 8 2022,
accessed 22 July 2022,
https://www.theguardian.com/commentisfree/2022/jul/08/climate-crisis-biodiversity-decline-overexploited-planet-change-to-survive-aoe?CMP=Share_iOSApp_Other
Julie Watson, “Climate Change is already Fueling Global Migration”, PBS News Hour,
July 28 2022, accessed October 2022,
https://www.pbs.org/newshour/world/climate-change-is-already-fueling-global-migration-the-world-isnt-ready-to-meet-peoples-needs-experts-say
Vonnegut Kurt, Galápagos, Delacorte Press, 1985.
Author Biography
Charlotte Gould is a senior academic at the University of Brighton. She has taught all levels of Visual
Communication and supervises PhD students. Through her practice she explores the potential for interactive
installations in digitally mediated public spaces, promoting public participation through shared experience
often using urban screens. She has developed Extended Reality artworks to prompt play and interaction across
social and cultural boundaries as well as interactive nonlinear narratives and speculative fiction which
explore how we can communicate the threat of ecological crisis, raising public awareness to trigger change in
behaviours. Through interactive installations she tests the boundaries of open systems, to offer opportunity
for diverse audiences to co-create artworks, impacting on the way we engage in the urban environment and
public space and contributing to a collective memory of place in a global context.
1The Hong Kong University of Science and Technology. 2 Tallinn
University Hong Kong - China, Tallin – Estonia varvarag@ust.hk, mar.canet@tlu.ee
Abstract
Artificial Intelligence is present in the generation and distribution of culture. How do artists exploit neural
networks? What impact do these algorithms have on artistic practice? Through a practice-based research
methodology, this paper explores the potentials and limits of current AI technology, more precisely deep neural
networks, in the context of image, text, form and translation of semiotic spaces. In a relatively short time,
the generation of high-resolution images and 3D objects has been achieved. There are models, like CLIP and
text2mesh, that do not need the same kind of media input as the output; we call them translation models. Such a
twist contributes toward creativity arousal, which manifests itself in art practice and feeds back to the
developers’ pipeline. Yet again, we see how artworks act as catalysts for technology development. Those
creative scenarios and processes are enabled not solely by AI models, but by the hard work behind implementing
these new technologies. AI does not create a ‘push-a-button’ masterpiece but requires a deep
understanding of the technology behind it, and a creative and critical mindset. Thus, AI opens new avenues for
inspiration and offers novel tool sets, and yet again the question of authorship is asked.
Keywords
AI art, creative AI, deep learning, GAN, NLP, automated creativity, practice-based research, latent space,
neural networks.
It is claimed that recent advancements in AI, such as CLIP-based products Midjourney and DALL-E, are supposed
to augment our creativity. For the first time, it does not sound so absurd that artists can find themselves out
of jobs. ¹ Not that artists would have ever had a secure and stable job, but deep learning (DL) tools might
eventually lead to losing some commercial commissions. Such thinking relies on a modern art approach where
skills are in the centre of attention and not the conceptual idea. Quoting Lev Manovich: “Since 1970 the
contemporary art world has become conceptual, i.e., focused on ideas. It is no longer about visual skills but
semantic skills.” ² As these new tools advance, the interfaces and techniques become more complex and
sophisticated as our eyes are becoming more accustomed to not being easily surprised.
Echoing Aaron Hertzmann, once painters were in a similar situation when photography was invented and took over
the niche of portrait-making. Then visual artists had to reinvent themselves and rethink the meaning of
painting. Photography had to wait another 40 years until it got recognized as an artistic medium.³
So-called AI artists have faced similar challenges in gaining acceptance within the art world and even
inside the digital art niche. ⁴
Computer art emerged with the invention of the computer. Artists, such as Vera Molnar and Manfred Mohr, created
their first computer-generated artworks in the 1960s using scientific lab computers at night when they were not
used by scientists. Early computer artists were repurposing a machine for artistic use and writing code to make
art on it. Since the creation process was mediated by a computer, it may seem to the general audience that the
artists were simply pressing a button and the computer doing art for them. Hence, the question of authorship
emerged: is the artist a machine or human?
Today, with the appearance of neural networks (NN) and their creative applications, the same question
reappears. Hertzmann has written several articles arguing that people do art and not computers. ³,
⁵ Manovich also describes how AI-generated images that imitate realist and modernist paintings are
claimed to be art. ² At the same time, experimental art forms, like installation, interactive format,
performance and sound art, are often overlooked unless they are promoted by a large corporation.
Instead of retelling a short but very dense history of DL technology development, in the next section, we focus
on the appearance of neural network tools that raised interest amongst artists and led to meaningful artwork
production.
Historical overview of DL development
DL is a subset of machine learning (ML) using Deep Neural Networks (DNN) to learn underlying patterns and
structures in large datasets. In 2012, a DNN designed by Alex Krizhevsky outperformed other computer vision
algorithms to achieve the new state of the art in the ImageNet Large Scale Visual Recognition Challenge. ⁶
This model, AlexNet, signaled the start of a new DL era. As AI technology has developed and become more
prevalent in real-world systems, artists have been exploring its limits and potentials, adapting these models to
their own practices. As the number of scientific publications on AI grows exponentially, it is useful to map out
the influential papers, and related applications, to help track the evolution of the AI-Art space in relation to
the technological advances. ⁷ Figure 1 shows a timeline of the development of generative models for images
and text. Using this diagram, we can make a few observations on the past ten years: the dominance of GANs for
image generation, the influence of the Transformer on Large Language Models (LLM), and the growing interest in
multimodal approaches and translation models.
The starting period of image generation using DNNs can be traced back to the creation of the Variational
Auto-Encoder (V AE) in 2013, and the Generative Adversarial Network (GAN) in 2014. ⁸, ⁹ These models
showed different ways in which a NN can be trained on a large dataset, and then used to generate outputs that
resemble but do not copy the original dataset.
For much of the past decade, GAN art has been a dominant and defining element of AI Art. GANs are trained using
a competitive lying game, played by two players: the Generator and the Discriminator. The Generator wins by
making an image that the Discriminator thinks is from the original dataset. The Discriminator wins by
successfully identifying which images the Generator has made. By playing this game repeatedly, both sides slowly
learn when they have been fooled and remember information so they don’t fall for the same tricks again.
The Generator gets better at making images, and the Discriminator gets better at detecting these fakes. At the
end of the game, we are left with a Generator that is very good at generating new images, with the qualities and
style of our original inputs. After the original GAN paper, there was a rush of exploration of this new
technique for generating images. Alongside general improvements to the models’ architecture and stability,
new ways of guiding the outputs and applying GANs to specific problems were also explored. ¹⁰ ,
¹¹
Image-to-Image Translation with Conditional Adversarial Nets (2016), also known as pix2pix, showed a process of
converting one type of image into another type. ¹² Mario Klingemann’s work Alternative Face
(1) used the pix2pix model with a dataset of biometric face markers and the music videos
of the singer François Hardy. This allowed him to control the movement of the face with this form of
digital puppetry, which he then demonstrated by transferring the facial expressions of the political consultant
Kellyanne Conway onto Hardy’s face as she talks about “alternative facts.”
In 2015, on the Google research blog, the post Inceptionism: Going Deeper into NNs described a tool that
attempted to understand how image features are understood in the hidden layers of the NN. ¹³ Alongside
this post they released a tool called DeepDream. This model enhances an image with the NN's attempts to find the
features of the dataset it was trained on. The creative use of DeepDream was proposed by the authors in the
original article “It also makes us wonder whether neural networks could become a tool for artists—a
new way to remix visual the creative process in general.” ¹³
DeepDream’s psychedelic imagery quickly caught the attention of the internet and of artists around the
world, resonating with those interested in understanding the crossover between biological and neurological
construction of images. Memo Atken’s work All Watched Over By Machines Of Loving Grace
(2): Deepdream edition, hallucinated over an aerial photograph of the GCHQ headquarters. This
work raises questions around the motivations of the organisations funding the development of AI, and in doing so
make the dreamlike qualities a little more nightmarish.
In the same year, the paper A Neural Algorithm of Artistic Style introduced a DNN “to separate and
recombine content and style of arbitrary images, providing a neural algorithm for the creation of artistic
image.”¹⁴ Neural Style Transfer (later known simply as StyleTransfer) takes two inputs, a style
image and a content image, it extracts textural information from the style image and compositional information
from the content image, then generates an image with minimal distance between the two. The paper demonstrates
this with images of a photograph represented in various styles of famous paintings, such as Van Gogh’s
The Starry Night.
In 2017, CycleGAN continued with the problem of image-to-image generation shown in pix2pix, but removed the
requirement of aligned image pairs being needed for training. ¹⁵ Instead a set of source images and a
set of target images that are not directly related can be used. The advantage of this is it is simpler to scale
to larger datasets, making the process more accessible for artists. Helena Sarin has been using CycleGAN for a
number of years, and recently in Leaves of Manifold (3)(4) she collected and
photographed thousands of leaves to build her own training dataset, and then implemented a custom pipeline with
changes that improve results when working with smaller datasets. This personalised approach in crafting the
models resonates with the hand-made, collaged aesthetic of the images generated.
Other notable developments to GANs brought improvements to image quality and resolution.¹⁶
¹⁷ In late 2018, the release of StyleGAN, a model built on a combination of ideas from Style
Transfer and PGGAN, demonstrated very convincing images of human faces.¹⁸ In his article “How
to recognize fake AI-generated Images”, the artist Kyle McDonald investigated the images generated by
StyleGAN, and highlighted the visual artefacts he found.¹⁹ At a glance these images look like
photographs, but on closer inspection irregularities such as patches of straight hair, misaligned eyelines, or
mismatched earrings reveal the difficulties GANs have in managing “long-distance dependencies” in
images.
In 2017 the paper Attention Is All You Need proposed a new network architecture called the
Transformer.²⁰ This model addressed the long-distance dependency issue in RNNs and CNNs by rethinking
how we could handle sequences. Rather than looking at a sentence word by word, the Transformer observes the
relationship between all elements of the sequence simultaneously. Being able to better handle long distance
dependencies meant the Transformer was appropriate for natural language generation. Artists have explored the
use of VAEs for short text generation, but with the emergence of LLM passages of long, coherent, texts could be
generated.²¹ As dataset sizes increased, along with hardware costs for training these large
models, they have become harder for individuals to train themselves, and the mode of interaction has shifted
from curated datasets and homemade scripts, to web APIs and third-party services. While it is more difficult to
participate in the training process, the availability of services and interfaces provides new ways of working
with these models that can produce less technical and more playful approaches. For example, Hito Steyerl used
GPT-3 to create Twenty-One Art Worlds: A Game Map and described the process as “fooling
around” with GPT-3 to write descriptions of different Art Worlds.²² In the resulting text it is
difficult to distinguish which words may have been written by Steyerl and which were written by GPT-3.
The learnings from LLM for text generation were soon applied to image generation (Image GPT, Vision
Transformer), and the simultaneous release of CLIP and DALL-E in January 2021 signaled the start of a new era of
image generation.²³, ²⁴ Although the DALL-E model was not released, CLIP was
made available to the public, and the model was quickly adopted by AI artists who applied the idea of CLIP
guidance to various image generation techniques. Ryan Murdock produced the colab notebooks DeepDaze
(5) (combining CLIP and SIREN) and BigSleep (6) (CLIP and BIGGAN), which were subsequently
adapted by Katherine Crowson in the widely distributed VQGAN+CLIP (7) notebook.
The paper Denoising Diffusion Probabilistic Models introduced a different method for creating generative
models.²⁵ This technique trains a model by adding increasing amounts of noise to an image and then
having the model remove the noise, resulting in a model that can generate images from only noise. Diffusion
models, when combined with CLIP or other conditioning processes, enable much faster text-to-image processing.
The popularity and accessibility of these techniques was further raised by the release of DALL-E 2 and
Midjourney in 2022. Midjourney became so popular it is now the largest Discord server with over 5 million
members. Following the releases of these products, open-source models such as Stable Diffusion have also been
developed. There are many benefits of using free and open-source models for artists. Being able to modify code
and develop on your own software allows the artist to pursue their own experimental approaches, not restricted
to the interface designed by a service provider.
The artist's involvement in generating new images with these models is vastly different to working with GANs.
Rather than building custom datasets and training models, instead the focus has shifted to writing prompts that
can generate the images the artist wants to find, and designing interfaces for exploring these prompts and their
translations. The artist Johannez coined the term Promptism for describing his art practice, and wrote a
humorous Prompist manifesto using GPT-3. Against a backdrop of models trained on hundreds of millions of images
scraped from the internet, including many artists’ portfolios, the manifesto asserts “The prompt
must always be yours.”²⁶
Artist-Guided Neural Networks
Many papers discuss AI from the point of view of creativity taking mostly one position of two: either AI as an
amazing tool for artists and creativity, or AI is seen as something negative in art. It is easy to see that the
people from industry advocate for the first position, and theory scholars for the second one. But, how do
practitioners see contemporary AI technology themselves? And in which ways AI is deployed in art practice?
Hence, it is not the focus of this paper to discuss whether AI can make art, but rather how AI can be useful for
artists and what new ideas it can offer. By using practice-based research methodology, we decode the role of AI
tools in artistic practice and trace the evolution of such artistic work. In this paper, the practice of artist
duo Varvara & Mar was used as a case study, which provided us with the insides in this research. We divide
the case studies into four categories based on medium: synthetic image, synthetic text, synthetic form, and
translation models. From the view of the practitioner, the limitations, new possibilities, and change in
production processes are discussed.
Synthetic Image
Our DL exploration began in 2017 with Google DeepDream, focusing on image generation. The concept behind
Neuronal Landscapes (8) project was to imagine how Estonian landscape will look like in
100 years’ time (commission work for the Estonian History Museum). Through synthetic vistas created by
machines, the artwork offers a glimpse into the environment from a machine's perspective, immersing viewers in a
hallucinated neural net simulacrum. To depict the evolution of Estonian society over time, from forests and
farmlands to urbanization and digitalization, a 360o VR video was created. Filmed with drone-mounted two 360o
cameras, the footage was edited and processed using DeepDream. The rendering process spanned 30 days on powerful
machines with Nvidia TitanX GPUs. While some customization was possible, the algorithm's aesthetic footprint
remained prominent.
In the next art project, ProGAN was deployed. For the first time we worked with datasets and training GAN
models. Plasticland (2019) (9) talks about plastic waste and ecological problems this
material causes. We composed four different datasets of images of layered plastics in our planet: landfills,
plastic on top of water, plastic underwater, and plastiglomerates. The ProGAN model was trained on a local
machine using pyTorch and took a week to train, and the artist used a selection of generated images to create a
video composition. A metal totem displaying those synthetic, as plastic is, layers, we draw attention not only
to the problem of waste but also question whether AI has some similarity with this material. Since the invention
of plastic, this material was applied almost everywhere because of its perfect qualities, until we realised that
it is not sustainable and ecology-friendly. Will a similar story happen with AI? From the practice-based
research perspective, this work shows artists’ desire to move from a still to moving image and towards
sculptural form that is held back by the early stage of machine learning technology: low resolution images
jumping from one frame to another.
The next artworks POSTcard Landscapes from Lanzarote I (00:18:37) and II (00:18:40)10 in 2021
demonstrate the artist's ability to create video works with StyleGAN2. The hypnotic appearance of these works,
where one frame morphs naturally into another, shows the artists' ability in guiding the outputs of the neural
network. Vector curation and composition of a journey through the latent space, created by training the model on
specific datasets of 2000+ images, were crucial and integral parts of the artistic process. The artwork talks
about critical tourism and how circulation of images representing touristic gaze overpower the nature of seeing.
In the words of Jonas Larsen “‘reality’ becomes touristic, and item for visual
consumption.”²⁷ Hence, we scraped, where licence allowed, the location-tagged images from
Flickr and composed two datasets of photos categorised as tourism or landscape. As we have written earlier:
“The two videos are random walks in the latent space of the Stylegan2 trained models, creating a cinematic
synthetic space. The audiovisual piece shows an animated image through the melted liquid trip of learning
acquired from the dataset composed of static images. The video flows from point to point, generating new views
and meaning spaces through the latent space’s movement. The audio was created after the video was
generated in response to the visual material to complete the art piece.”²⁸ The sound for local
or landscape view was created by a sound artist from Lanzarote, Adrian Rodd, who aimed to give a socio-political
voice to the piece. In contrast, the sound design created by Taavi Varm is a soundscape replying to touristic
gaze. The artists aimed to initiate collaborations with others but also to experiment with human-AI co-creation.
In a similar vein is the artwork Phantom Landscapes of Buenos Aires
(11)(00:20:00, 2021), with sound work by Cecilia Castro.
Our last experiment with GAN models Synthetic-scapes of Tartu (00:10:00, 2022), demonstrates a
different approach. Taking a dataset composed from our own video footage (flaneur walks), we first produced the
sound (a composition by Taavi Varm, Ville MJ Hyvönen with piano by J. Kujanpää) and used this
to inform the direction of the video. The result was a sound-guided AI-generated visual output.
Synthetic Text
In this section, we focus on artwork incorporating AI text generation as part of the artistic concept. Our
journey to text generation started with the online participative theatre project ENA
(12)and ended with a hand-bound publication.
During the first lockdown in May 2020, together with theatre maker Roger Bernat, we created an online
participative theatre piece ENA on the website of Theater Lliure in Barcelona. ENA is a
generative chatbot that talks to its audience, and together (AI and audience), they make theatre. As we have
described before: “Although in the description of the project it was stated explicitly that people were
talking to a machine, multiple participants were convinced that on the other side of the screen another human
was replying to them—more precisely the theatre director himself, or at least an
actor.”²⁹
Analysing synthetic books, Varvara Guljajeva has stressed the importance of human input in the AI
text-generation systems.³⁰ In addition, one also needs to guide the audience participation and
interaction with the chatbot. For this purpose, we have adopted the traditional theatre method for guiding
actors, as a way to guide the audience, and thus, the bot, too. Stage directions were used as a guiding method,
which triggered thematic conversation and offered meaningful dialogue between humans and the AI system. We found
the conversations so meaningful that we decided to publish a book that contains all the conversations with
ENA.
With this project, we learned that it is essential to guide neural networks via audience interaction. In order
to do this, it is also necessary to guide the audience. Without audience interaction guidance, it is nearly
impossible to achieve meaningful navigation of neural networks.
Translation models
This category focuses on translation models that enable interactive and installation-based formats. Translation
refers to the conversion of mediums, or as we put it, translation of semiotic spaces. To illustrate this, we
introduce Dream Painter (13)an art installation that translates
audience’s spoken dreams to a line-drawing produced by a robot (Figure 2). As described earlier:
“Dream Painter is an interactive robotic art installation that explores the creative potential of
speech-to-AI-drawing transformation, which is a translation of different semiotic spaces performed by a robot.
We extended the AI model CLIPdraw which use CLIP encoder and the differential rasterizer diffvg for transforming
the spoken dreams into a robot-drawn image.”³¹ “Design- and technology-wise, the
installation is composed of four larger parts: audience interaction via spoken word, AI-driven multi-colored
drawing software, control of an industrial robot arm, and kinetic mechanism, which makes paper progression after
each painting has been completed. All these interconnected parts are orchestrated into an interactive and
autonomous system in a form of an art installation [...].”³² Out of all the projects discussed,
this was the most difficult to realise. This is because of the large scale of the artwork, and multiple parts of
software and hardware that need to run automatically and synchronously.
In this project we investigated how guidance of neural networks could be interactive and real-time instead of
non-interactive and pre-determined, as shown in previous examples of our work. It is important to notice that
methods, such as dataset composition and output curation were not used in this case. In fact, visual output
curation is totally missing. The artists created an interactive system to be experienced and discovered by the
audience. This means the audience determines the output. Instead of curating a dataset, a CLIP model is used
that can produce nearly real-time output guided by a text prompt. As we have written earlier: “Translation
of semiotic spaces, such as spoken dreams to AI-generated robot-drawn painting, allowed us to deviate from
image-to-image or text-to-text creation, and thus, imagine different scenarios for interaction and
participation.”³³
This project indicates our search for transformative outputs of AI technology, and thus, shows the evolution in
practice. By extending available DL tools and combining with other technology, for example, text-to-speech
models, real-time industrial robot control, and physical computing, it offered an interactive robotic and
kinetic experience of neural network latent space navigation. This contributes towards the explainability of AI
because the audience could experience how the words affected the drawing, and which concept triggered which
outcome.
Being inspired by Sigmund Freud's work on the interpretation of the human mind while unconscious, we
speculatively ask if AI is powerful enough to understand our dreamworld. Through practice we question the
capacities of neural networks and investigate how far we can push this technology in the art context. This
artwork allows the audience to experience the limits of concept-based navigation with AI. The system is unable
to interpret and can only illustrate our dreams. It cannot understand the prompt semantically and only gets the
concepts.
Synthetic Form
In this section, we ask how artists can guide neural networks when creating volumetric forms, and what happens
when AI meets materiality. After working for a while with DL tools that produce 2D outputs, it is an obvious
step to explore possibilities to produce 3D results. To our surprise, it was not an easy task to find the
solution (Oct 2021). Psychedelic Forms is a series of sculptures produced in ceramics and recycled
plastic through which we investigated the possibilities of AI in producing physical sculptures. The project
re-interprets antique culture in the contemporary language and tools.³³
Following the same paradigm shift as in the previous section, text2mesh is a CLIP-based model that does not
require a dataset, but a 3D object and text prompt as input.³⁴ Hence, the model actually does not
create a 3D model but stylises the inserted one, guided by inputted text.
We decided to go back to the origins, in terms of ancient sculptures and material selection. Although it was
said that there was no dataset, we still had a collection of 3D models of ancient sculptures because, by far,
not all produced a desirable output. In this sense, there was definitely an output curation present in the
process.
The criteria for selection were the following: first, the form had to be intriguing, and second, it should be
possible to produce it in material afterwards. It was clear that we had to modify each model because the
physical world has gravity, and the DL model does not take this into account. Some generated models were
discarded because they were seen as not-fixable, although interesting in their shape.
The process demonstrated here is quite an unusual way to create an object. After extensive experimentation with
the tool, we learned how certain words triggered certain shapes and colours. This knowledge gave us a chance to
treat text prompts as poetic input. Thus, we created short poems to guide NN. The best ones survived as titles
and are reflected in the forms.
The artists did not strictly follow the original model but took the creative liberty to modify the shape and
determine the colour by manually glazing the sculptures. The dripping technique was used for colouring the
sculptures. This served as a metaphor for liquid latent space and the psychedelic production process (this was
the artists’ inner feeling about the creative process because they did not know what results would be
achieved in the end). Sometimes, AI-generated vertex colouring was taken as inspiration, sometimes totally
ignored. Nevertheless, digital sculptures were exhibited alongside the physical ones to underline the
transformation and human role in the creative process. Although ceramic sculptures were 3D printed in clay, the
fabrication process had to follow the traditional way of producing pottery (Figure 3). Since the artists had
never engaged in ceramics before, the whole production process felt psychedelic: unexpected neural network
processes led to transformation by numerical, physical, and chemical processes, all guided by both the artists
and chance. Hence, the art project highlights the relationship between different agencies.
In the end, we can say that AI is not prepared for the physical world. It created nice images, but when one
wants to materialise the output, it requires considerable additional work. However, those extra processes were
very rewarding and creative in our case. In this project, AI served as an inspiration or a departing point more
than anything else. In other words, the experimental phase of technology is necessary for experimental
practices, and this can lead to the creation of a new production pipeline. The fine line between control and
chance when guiding the neural networks and related processes is likely the main creative drive for the artists.
Discussion
According to the media hype around AI, this technology is intelligent enough to create art
autonomously.³⁵, ³⁶ However, the reality is different. A computer scientist and
a co-inventor of Siri Luc Julia, AI does not exist. He advocates for machines’ multiple intelligences that
often outperform humans. However, machine intelligence is limited and discontinuous compared to human
intelligence.³⁷ Therefore, it is vital to have artistic practices around this technology, as a
counterbalance to the AI fantasies served by the industry and mass media.
We see AI as a creative tool with its own possibilities and limitations, which can stimulate artists’
creativity through unexpected outputs. Research has shown that tool-making expands human cognitive level and
constitutes evolution in culture.³⁸, ³⁹ Similarly, as a new tool,
generative AI could potentially enrich creativity by allowing new production pipelines that can create unique
results.
Coming back to the synthetic images, we can say that all machine-created synthetic image-based works discussed
here have particular aesthetics: both with DeepDream and GAN. Unlike the output of GANs, DeepDream has a more
recognizable style and can be seen more as a filter that transforms every inputted image instead of learning
from the given dataset. Regarding GAN aesthetics, such visual appearance is inherited from two entities to a
large extent: the dataset and the model itself. GANs have a particular footprint, as seen in all works produced
with this model. The visual palette comes from the used datasets. For example, if a dataset is homogeneous (only
landscape images), then we will easily recognize landscapes in the generated output. However, if images in the
dataset have a lot of visual variation, the output is rather abstract. POSTcard Landscapes from Lanzarote II
illustrates this well. Also, when photos in the dataset look similar, the output will also be similar, as
was the case with the Synthetic-scapes of Tartu video work where frames from recorded flaneur walks in
a city were extracted. When we talk about video works generated with the neural net, then manual guidance of
latent space offered more variations than an audio-led approach.
Synthetic image works have encouraged us to work with formats like images and videos that we did not engage in
before in our art practice, but we found it exciting working with AI and video. For example, AI video generation
has some affordances, like starting and ending can be done in a perfect loop since images are synthetically
generated. However, creating real-time AI work is much more complex because some models are too slow. It might
take a few minutes to render a single image. The limitations inspire us to devise new solutions and work in new
mediums. Moreover, the limitations of the medium have always been a good challenge for our creativity.
Working with GANs or other image-generation tools has become much easier in recent years, although it used to
be quite difficult. We must note that for practitioners, easy-to-use tools, such as DALL-E and Midjourney, offer
little creative freedom, and thus, are less attractive to the artists. Those products tend to instrumentalize
the user rather than the other way around. At the same time, open- source models offer more creative freedom and
enable broader use of artistic ideas.
The work with generated text demonstrates that AI is not context-aware but maps concepts automatically without
understanding semantics. More importantly, as shown in the ENA project the audience must also be guided
alongside the AI. In the case of ENA, stage directions were used, and in the Dream Painter
project, the concept of dream telling was applied to guide the participants who in turn guided the neural
net through their interaction, creating a chain reaction. Navigating concepts in latent space is artistically
interesting and inspiring, this was especially evident when working with form. The artists went beyond semantics
and learned how to guide neural networks with a text prompt and 3D object.
The presented practice represents a paradigm shift in machine learning, moving away from composing datasets for
GANs and toward translating semiotic spaces enabled by diffusion models. The evolution in practice shows how
artists discover and learn to work with the DL toolset, embracing its possibilities and limitations. In the case
of practice-based research, practice can be seen as a lab for testing artistic ideas with technology through
chance until control is encountered.
In this article, we have summarised DL development from the perspective of artists’ interests
concentrating on the image, video, text, 3D object generation, and translation models. We applied practice-based
research methodology to investigate the role and possibilities of recent co-creative AI tools in artistic
practice.
It is difficult to keep pace with AI development. In less than a decade, we have gone from blurry
black-and-white faces to impressive high-resolution images guided by text prompts. The user level has gone from
difficult to easy, which on one side, broadens possibilities for creation, but on another, it diminishes
experimentation and creativity, since AI outputs seem ready-made. This is also demonstrated by the explorative
nature of the body of work presented here.
Furthermore, it was noticed that creative AI, especially GAN models, have recognizable aesthetics, which, in
the long run, become repetitive. This led to the change of tools by the artists. The curation of datasets,
models, and outputs, along with neural network guidance, have become the toolset of an artist working with AI.
Finally, these models can generate multitudes of outputs, but the art is giving the right input to guide the
desired output and selecting the results that best serve the concept.
As Andy Warhol had envisioned in 1963, eventually, art production will become mechanised and automated. In his
own words: “I want to be a machine”, which was also a reflection on that time's vast
industrialization process. Resonating with today’s deep learning age: I want my machine to do art.
Acknowledgements
MSC is supported as a CUDAN research fellow and ERA Chair for Cultural Data Analytics, funded through the
European Union’s Horizon 2020 research and innovation program (Grant No.810961).
References
1 Gabriel Nicholas, "These Stunning.in.toolsareaboutto change the art world (2017)”, Slate
website, accessed November 20, 2022,
https://slate.com/technology/2017/12/a-i-neural-photo-and-image-style-transfer-will-change-the-art-world.html.
2 Lev Manovich, Ai and myths of creativity, Architectural Design, 92(3):60–65, 2022,
https://onlinelibrary.wiley.com/doi/abs/10.1002/ad.2814
4 Kevin Roose, “An A.I.-Generated Picture Won an Art Prize, Artists Aren’t Happy
(2022)”, The New York Times website, accessed November 20, 2022,
https://www.nytimes.com/2022/09/02/technology/ai-artificial-intelligence-artists.html
5 Aaron Hertzmann, Computers do not make art, people do. Commun, ACM, 63(5):45–48, apr 2020,
ISSN 0001-0782, https://doi.org/10.1145/3347092
6 Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, "Imagenet classification with deep
convolutional neural networks", Communications of the ACM 60, no. 6, 2017, 84-90,
https://dl.acm.org/doi/10.1145/3065386
7 Mario Krenn, Lorenzo Buffoni, Bruno Coutinho, Sagi Eppel, Jacob Gates Foster, Andrew
Gritsevskiy, Harlin Lee et al. "Predicting the Future of AI with AI: High-quality link prediction in an
exponentially growing knowledge network", arXiv preprint arXiv:2210.00881, 2022.
8 Diederik P Kingma, Max Welling, “Auto-encoding variational bayes (2013)”, Arxiv,
https://arxiv.org/abs/1312.6114
9 Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, Yoshua Bengio, “Generative adversarial networks (2014)”, Arxiv,
https://arxiv.org/abs/1406.2661
10 Alec Radford, Luke Metz, Soumith Chintala, “Unsupervised representation learning with
deep convolutional generative adversarial networks (2015)”, arXiv. https://arxiv.org/pdf/1511.06434.pdf
11 Martin Arjovsky, Soumith Chintala, Léon Bottou, “Wasserstein gan (2017)”,
Arxiv, https://arxiv.org/abs/1701.07875.
12 Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, “Image-to-image translation
with conditional adversarial networks (2016)”, Arxiv, https://arxiv.org/abs/1611.07004
13 Alexander Mordvintsev, Christopher Olah, Mike Tyka, “Inceptionism going deeper into
neural (2015)”, Google Blog website, accessed November 20, 2022,
https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html
14 Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, “A neural algorithm of artistic style
(2015)”, Arxiv, https://arxiv.org/abs/1508.06576
15 Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros, "Unpaired image-to-image
translation using cycle-consistent adversarial networks. (2017)" In Proceedings of the IEEE
international conference on computer vision, p.2223-2232.
16 Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, “Progressive growing of gans for
improved quality, stability, and variation (2017)”, Arxiv. https://arxiv.org/abs/1710.10196
18 Tero Karras, Samuli Laine, Timo Aila, “A style-based generator architecture for
generative adversarial networks (2019)”, In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, 4401–4410.
19 Kyle McDonald, "How to recognize fake AI-generated images (2018)”, Medium website,
accessed November 20, 2022, https://kcimc.medium.com/how-to-recognize-fake-ai-generated-images-4d1f6f9a2842
20 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,
Łukasz Kaiser, Illia Polosukhin, “Attention is all you need (2017)”, Arxiv,
https://arxiv.org/abs/1706.03762
21 Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal,
Arvind Neelakantan et al., “Language models are few-shot learners (2020)”, Arxiv,
https://arxiv.org/abs/2005.14165
22 Hito Steyerl, “Twenty one art worlds a game map (2022)”, e-flux website, accessed
November 20, 2022, https://www.e- flux.com/journal/121/423438/twenty-one-art-worlds-a-game-map/
23 Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever,
“Generative pretraining from pixels (2020)”, In International Conference on Machine Learning.
24 Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
Unterthiner, Mostafa Dehghani et al. “An image is worth 16x16 words: Transformers for image
recognition at scale (2020)”, Arxiv. https://arxiv.org/abs/2010.11929
25 Jonathan Ho, Ajay Jain, and Pieter Abbeel, “Denoising diffusion probabilistic models
(2020)”, Arxiv. https://arxiv.org/abs/2006.11239
26 Johannezz, “The promptist manifesto”, deeplearn.art website, accessed November 20,
2022, https://deeplearn.art/the-promptist-manifesto/
27 Jonas Larsen, "Geographies of tourist photography", Geographies of Communication: The
Spatial Turn in Media Studies. Gothenburg: Nordicom, 2006, 241-257.
28 Varvara Guljajeva, Mar Canet Sola, “POSTcard Landscapes from Lanzarote (2022)”, In
Creativity and Cognition (C&C '22), Association for Computing Machinery, New York, NY, USA, 634–636,
https://doi.org/10.1145/3527927.3531191
29 Varvara Guljajeva, Mar Canet Sola, “ENA: Participative Art Forms During Pandemics
(2021)”, Ed. Livia Nolasco Rozsas and Borbala Kalman in HyMEx 2021 (Hybrid Museum Experience Symposium
Proceedings), Budapest, 77-82.
30 Varvara Guljajeva, “Synthetic Books (2021)”. In 10th International Conference on
Digital and Interactive Arts (ARTECH 2021), October 13–15, 2021, Aveiro,Portugal, Portugal. ACM, New
York, NY, USA, 7, https://doi.org/10.1145/3483529.3483663
31 Mar Canet Sola, Varvara Guljajeva, “Dream Painter: Exploring creative possibilities of
AI-aided speech-to-image synthesis in the interactive art context (2022)”, Proc. ACM Comput. Graph.
Interact. Tech. 5, 4, Article 33, September 2022, 11, https://doi.org/10.1145/3533386
32 Varvara Guljajeva, Mar Canet Sola, “Dream Painter: An Interactive Art Installation
Bridging Audience Interaction, Robotics, and Creative AI (2022)”, In Proceedings of the 30th ACM
International Conference on Multimedia (MM '22), Association for Computing Machinery, New York, NY, USA,
7235–7236, https://doi.org/10.1145/3503161.3549976
33 Varvara Guljajeva, Mar Canet Sola, “AI-Aided Ceramic Sculptures: Bridging Deep
Learning with Materiality (2023)”, In: Johnson, C., Rodríguez-Fernández, N., Rebelo, S.M.
(eds) Artificial Intelligence in Music, Sound, Art and Design, EvoMUSART 2023, Lecture Notes in Computer
Science, vol 13988, Springer, Cham, 2023, https://doi.org/10.1007/978-3-031-29956-8_23
34 Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka, “Text2mesh:
Text-driven neural stylization for meshes (2022)”, In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, 13492–13502.
35 Sarah Perez, “Microsoft’s New Drawing Bot Is an AI Artist (2018)”, TechCruch
website, accessed November 20, 2022,
https://techcrunch.com/2018/01/18/microsofts-new-drawing-bot-is-an-a-i-artist/
36 Chris Vallance, "Art is dead Dude" - the rise of the AI artists stirs debate (2022)”, BBC
website, accessed November 20, 2022, https://www.bbc.com/news/technology-62788725
37 Luc Julia, "There is no such thing as Artificial Intelligence", LP, 2020.
38 Dietrich Stout., “Stone toolmaking and the evolution of human culture and cognition
(2011)”, Philosophical Transactions of the Royal Society B: Biological Sciences, 2011,
366(1567):1050– 1059.
39 Dietrich Stout, “Tales of a Stone Age Neuroscientist (2016)”, Scientific American
website, accessed November 20, 2022,
https://www.scientificamerican.com/article/tales-of-a-stone-age-neuroscientist/
Authors Biographies
Dr Varvara Guljajeva is an artist and researcher holding the position of Assistant Professor in Computational
Media and Arts at the Hong Kong University of Science and Technology (Guangzhou).
Mar Canet Sola is an artist and researcher. Mar is a PhD candidate and research fellow at Cudan research
group in BFM Tallinn University. Varvara and Mar form an artist duo Varvara & Mar since 2009.
Isaac Joseph Clarke is a PhD student in Computational Media and Arts at HKUST(GZ) investigating AI tools for
artists.
Kombucha as a Guide. Serendipitous Journey through Taste, Feminism, Free and Open Source Culture, and Ritual
LucileHaute
University of Nîmes (PROJEKT) & École des Arts
Décoratifs – PSL (EnsadLab) Nîmes & Paris, France lucile.haute@ensad.fr
Abstract
During one and a half years living with kombucha, the author followed it across a journey that diverged from a
disciplinary point of view in favor of a “global design” (Papanek) approach that embraces
co-dependencies (Haraway, Tsing). This journey spanned several domains: gastronomy and food, health, textile
design, social practices. It was the occasion to find out how different cultures might be embodied by the
heterogeneous kombucha community and its various locations: from bio-hack lab to the kitchen, art gallery,
design school, and brewery lab. What may this Symbiotic Culture Of Bacteria and Yeast teach
us—humans—about our ways of collaborating with, cultivating, exploiting or caring for the living
beings we eat and/or use to make everyday objects? Could kombucha SCOBY provide a guide to shifting ways of
understanding and performing our way of life, or to phrase it another way: to initiate a cultural revival?
Keywords
Companion Species, DIWO (Do It With Others), Design (critical, fictional, prospective, textile), Kombucha,
Performance art, Ritual, SCOBY.
How can we become aware of, meet, and build more respectful relationships with the life forms that surround us?
Is there a link between human-made political forms and other-living-being-made living forms? How can we draw
inspiration from the observation and understanding of life forms to imagine political forms alternative from
exploitation and domination? How can we encourage these kinds of observations? These heterogeneous questions and
concerns often appear in interactions with fermenting communities. They are encountered in relation to the
question of the Anthropocene and from different fields: philosophy, arts, design.
In a very much grounded perspective, I approached these questions over the past year and a half by concretely
engaging on a serendipitous journey with a symbiotic life form: kombucha.
This paper aims to chronicle this journey from a situated perspective, and position it as part of a larger
tapestry of thoughts and reflections. The author is a white cis-woman living and working in two cities in
France, with a rural and feminist background linked to her grandmother’s farm and a childhood in a village
that passed on a culture of DIY canned food from the garden.
The various fields concerned by kombucha are concrete places from which to address previous questions
—ones that, otherwise, would remain theoretical and unembodied. These domains are culinary design as it
relates to fermentation, textile design, prospective and fictional projects, and art. How do contemporary
artists and designers reactivate issues inherited from ecology, feminism or mesology?
The goal is to embrace a holistic view of our surroundings, objects and food, to adopt a global design
perspective, i.e., following Papanek: taking all stages into account, from production or the extraction of raw
materials to the processing of waste, through transformations and uses, all in iterative cycles. How might
kombucha be involved in these initiatives?
This paper relies on two types of primary sources. The first are formal interviews with three actors: a
kombucha brewer based in Paris, a French-Icelandic artist and designer, and an American designer. The second
material is the result of a year and a half of solo and collective experimentations, meeting and living with
kombucha, that provided the opportunity for various informal discussions. Among these, I organized a two-week
long design workshop to which were invited: an artist, two textile designers, a maker, a kombucha brewer, and an
engineer. ¹ This generous format was very fertile both in terms of discussions and formal
experiments. Thereafter, artistic events facilitated the collection of stories from the public. ² Other
sources are published magazines, books, articles or even online posts.
Microorganisms and Biotransformation through Interdisciplinary
Prism
Kombucha
The average knowledge about Kombucha is that it is a Symbiotic Culture Of Bacteria and Yeast. Yeasts transform
sugar into alcohol; bacteria transform alcohol into acid and produces a protective biofilm on the surface. Each
one produces the organic substance the other symbiote profits from. The acidification of the solution protects
the culture by preventing the growth of undesirable yeasts and bacteria. The beverage is drunk according to an
ancestral tradition in Asia, and more recently in the West where its growing mainstream popularity is fairly
recent. ³
In a variety of senses, Kombucha’s popularity has relied upon a recent fermentation’s fame. In the
intellectual world, “fermentation” is a fertile concept summoned as a political and philosophical
statement linked to feminism, epistemology, and ecology. A representative example is Lauren Fournier’s
project ⁴ and essay.⁵ Fed by the thought of Haraway and Tsing—and they themselves by
Margulis—, Fournier associate fermentation and feminism as reciprocal metaphors that we shall deploy
hereinafter.
Biotech Perspective
From a bio-engineering perspective, the study of traditional fermented food may reach some molecule of interest
for health, food or cosmetic applications. A renewed interest in the fermentation field was explicit, for
example, at a conference I attended on February 23, 2022, at Polytech Lille, organized by the Cluster Nutrition
Health Longevity—whose motto is “Where The Health and Food Sectors Converge.” ⁶
This conference gathered entrepreneurs, engineers, public and private researchers in agricultural, under
the title “Fermentation: applications, new markets & health potential.” Among the seven guests,
two kombucha brewers were programmed. Beyond that pleasant surprise, my interest was particularly picked by the
intervention of Gnosis by Lesaffre’s Global Marketing Director.
This brand is the result of the acquisition of Gnosis, the specialist in fermentation products, by Lesaffre,
the world's leading yeast company in 2015. ⁷ The brand statement positions it in the field of health
nutrition: “We harness the power of microorganisms and biotransformation processes such as fermentation to
create nutritional actives, probiotics, and nutritional and functional yeasts that benefit human health and
wellbeing.” ⁸ In September 2022, they joined the French Gut Initiative, a research project about
microbiota. ⁹
Their business model is mainly B2B for the food and nutraceutical industries, however, they also offer a range
of yeast products for domestic use. During COVID-19 lockdown, they observed an increasing demand on these
specific products. After the lockdown, the domestic yeast demand went down but remains above its prior level.
They therefore began to developing local yeast variations related to the food cultures of different markets and
countries. This anecdote reveals an industrial yeast producer’s interest in local cultures, which is
paradoxical given that it's precisely the industrialization of food production methods (homogenization,
centralization and scaling up) that has removed fermentation from individual plates.
Taste Perspective
On a smaller scale but no less demanding, let’s mention two noticeable cookbooks that participated to the
spread of fermentation culture. Qualified as a “bible for the DIY set” by The New York Times,
The Art of Fermentation. An In-Depth Exploration of Essential Concepts and Processes from Around the World.
With Practical Information on Fermenting Vegetables, Fruits, Grains, Milk, Meats, and More. from Sandor
Ellix Katz was published in 2012. Claiming the polysemy of the word culture, Katz states that
“Reclaiming our food and our participation in cultivation is a means of cultural revival, taking action to
break out of the confining and infantilizing dependency of the role of consumer (user), and taking back our
dignity and power by becoming producers and creators”. ¹⁰
This book has established itself and remains a major reference for DIY and homemade initiatives, with a
theoretical perspective and health focus. Loren Fournier often refers to it in her paper. So do other artists
working with fermentation such as Maya Minder.
In the world of gastronomy, fermentation was rehabilitated on the chef’s table by the triple Michelin
starred restaurant Noma, located in Copenhagen (Denmark). Eight years after the publication of a beautiful and
expensive book for æsthetes in 2010 ¹¹, a cheaper and more pragmatic one marked the culture:
The Noma Guide to Fermentation. ¹² Suddenly, it was everywhere: from biohackers lab to
tables d’hôte kitchens, the bookshelves of wild fermentation amateurs and independent
brewery labs.
Where Katz focused on food and politics and health, considering all three simultaneously,
Noma mentioned each recipe’s traditional and historical backgrounds but quickly shifted its focus to taste
and How to?, starting with building a fermentation chamber. Nevertheless, Katz’s approach may
also apply to describe Noma’s book: “Empower [the reader] with tools so [they] can explore and
reclaim fermentation into [their] life”.
This quick food publication review wouldn’t be complete without a non-cookbook: an artists-led
publication about the human food system aiming to present research and art about food and technology from an
open culture perspective. The third issue of Food Phearking was published by the Center for Genomic Gastronomy
in 2016 and themed around Gut Gardening. ¹³ The issue presents the 15 most famous bacteria
from human microbiome followed by essays from artists and researchers. It invites the reader to consider
becoming Gut Gardener ¹⁴ and joining ‘enterogastronomy’ movement—eating with the
gut in mind.¹⁵
Eating is Political
Behind the Food
“Every living creature on this Earth interacts intimately with its environment via its food” says
Katz. Various artists and designers explore the social potential of eating together in a performance
perspective. They use the sensual or awkward intimacy inherent to eating or drinking, the different aspects of
the collective practice of. It can be festive or ceremonial, joyful then dramatic, depending on the
ritual’s intent. Eating together may invite to a meditation and allow awareness.
Some artists address the political aspects of food and try to repair some lost connection or understanding of
where the food comes from. Tunde Wey’s performances ¹⁶ and Cooking Sections (Daniel
Fernández Pascual & Alon Schwabe)’s work ¹⁷ come to mind.
Some embrace the ritual and sacrificial weight of food, underscoring its association with both life and death.
As Haraway wrote: “There is no way to eat and not kill, no way and not to become with other mortal beings
to whom we are accountable, no way to pretend innocence and transcendence or final peace. [...] Multispecies
human and nonhuman ways of living and dying are at stake in practices of eating.” ¹⁸
One collective and ritualistic moment embracing the sacrificial weight of eating is The Butcher
proposed by Atelier Van Lieshout at la Friche de la Belle de Mai in Marseille (FR) on September 15, 2013.
This spectacular diner held inside the eponymous exhibition proposed a meal around and from a cow. The press kit
mentioned three interesting aspects. First, the qualification as “gustatory conviviality” connotes
an æstetic social moment related to gastronomy. Second, the mention of “reactivation of ancestral
rituals” implies an anthropological heritage and collective feeding as a symbolically charged and
structured moment. Third, the objective of “consumption without waste” embraces the contemporary
issues around over-production and concerns around human activities’ impact.¹⁹
Drinking Commercial Kombucha
Back to kombucha, the political impact of drinking this beverage may not seem obvious, especially when it
isn’t homemade but commercially produced. The latter trend in kombucha fashion was born in California and
is marketed towards health concerns and self-care habits.
Based in Paris since 2018, the brand Vivant Kombucha has adopted a marketing positioning in which taste and
process are underscored. It is intentional, inherited from natural wine culture (which inspired the fermentation
technic) and brewing culture (which inspired the packaging). The project was to create a non-alcoholic beverage
with a taste complexity that related to wild fermentation and the natural wine world.²⁰
The company doesn’t place its ethical structural choices on display: small local business,
community-grounded, fair salaries, carbon footprint limitation with cargo-bike delivery, bottles consignment for
professional customers, by-products reclamation (infused tea and cellulose), care for the livingness of the
product (neither filter nor pasteurization are involved in the stabilization process, which relies only upon
cold). It remains aware of the limits of its ethical commitment: tea and cane sugar are fair-trade but travel
long distances —experiments with beet sugar, which is produced in France, were unsatisfactory, due to
fructose peaks during the fermentation process, with undesirable effects on taste, Mial Wattkins explained.
In rationalizing his commitment to kombucha, he refers to Jonathan Nossiter’s book Cultural
Insurrection, ²¹ summarized as follows: contracultural arts are dead, gangrend by the
capitalist system. “The only thing that you can do now, it’s to go to the countryside, put your
hands into the earth and make something [...]. Build a microeconomy around a very specific thing. [...]
It’s the story of natural wine and artisanal beer. [...] We applied this way of thinking to
kombucha.” Vivant Kombucha is fueled by ethical concerns and the will to contribute to a common culture
while running a business. Does this business model’s affiliation with the capitalist system defuse its
insurrectionist potential? Feminists may say so, since “The Master’s Tools Will Never Dismantle The
Master’s House”, as says the famed Audre Lorde title. ²² However, one may argue that
acting, motivated by ethical concern, and armed with a clear and precise understanding, allows for a departure
from the theoretical and an embrace of tangible trial.
Fermenting Feminism
Echo to Ecofeminism
As previously mentioned, the social eating moment, articulated via ritualistic forms involving fermentation,
may be a means to address contemporary concerns regarding the impact of human activities on the planet. These
are an invitation to explore a specific tradition, one deeply committed to countering mortiferous human
activities against living beings, through the employment of artistic and spiritual forms (such as ritual, dance,
fiction, meditation...).
The common thesis of ecofeminist theories, summarizes Jeanne Goutal, is that “all domination forms
(sexism, racism, specism, colonialism, imperialism, anthro-pocentrism...) are inseparably and systemically
linked.” ²³ Ecofeminists commit to “the refusal of any form of domination,” between
humans as well as between species, genders, or regions, and the fact that “systemic change needs a
psychic, moral, intellectual and spiritual mutation [...] Ecofeminists refuse the ideal of a freedom without
awareness of natural limits.” ²⁴ This family of thought and action is embodied by contemporary
artists and designers addressing topics such as the renewal of interspecies relationship through forms related
to ritual.
Building a Critical Space
From 2012 to 2016, Lauren Fournier curated a series of events and exhibitions entitled fermenting
feminism. It brought together artists to explore the question of “what it means to bring
fermentation and feminism into the same critical space?” ²⁵ She developed a reciprocal metaphor
between fermentation and feminism, that she honed both from a theoretical perspective—with an academic
paper—and in practice—with the collective events and exhibitions. She and the participants
“approach fermentation through intersectional and trans-inclusive feminist frameworks” and
“approach feminisms through the metaphor and material practice of fermentation”: “Fermentation
as a process of transformation becomes both a metaphor and a material practice through which to explore
important issues [...] from the politics of labour, affect, survival, and care to colonialism, food,
indigeneity, and the land.” ²⁶
Through these art exhibitions and events, she affirms a statement that is present in the foreword of the
catalog and further developed in a paper: fermentation is a generative and speculative metaphor and,
simultaneously, a material practice, a microbiological process that embodies and performs a theoretical
positioning.
She elaborates this idea into ten proposals: “fermentation is political; fermentation is vitalism;
fermentation is accessibility; fermentation is preservation and transformation; fermentation is interspecies
symbiosis and coevolution; fermentation is survival and futurity; fermentation is care of the self and care of
others; fermentation is harm reduction; fermentation is queer time; fermentation is collaboration.”
²⁷ Cross reading of the paper and the catalog allows one to associate the artist’s work to one
of several of those.
Prospective Design, Fiction & Ritual
Building communities around the table
Crafters and makers concerned on how to do things themselves reclaim self-nourishments and experiment in making
processes that participate in an empowerment process. These may turn away from the commercial model,
experimenting on other ways to exchange services and goods. Communities can rally around thought or interest,
with or without devoted spaces; these can be ephemeral, or provoked by a situation. This is the goal of artists
who gather spontaneous communities around fermented food sharing, congregating at the table around the living
being that feeds us, and inviting an audience to raise awareness.
From Design to Performance and Cooking
Elin Margot, a French-Islandic artist and designer, often uses food-sharing moments as a startingpointandfinal
form to address critical issues. She organized several debate-dinners, inviting people from a common area with
divergent points of view around a fictional menu contextualized in a scenario. For those rituals —that may
also be called participative performances—, when any furniture or accessory is needed, she relies upon her
training in product design, using wood and ceramic. The sequencing of the event draws upon her service design
skills. Her method to build it all relies upon critical and speculative design. ²⁸
In reference to another artist’s performances, she asks, in a fictional context: what would a dinner
consist of in a society that refuses to kill animals and eats only that which some of them naturally
shed—such as salamander tail? As such, she rethinks the place of meat.
She recently started to learn cooking. Prior, she faced challenging collaborations with chefs, requesting that
they abandon the quest of making good food and focus instead on the way to do it—regardless of taste.
Aware of vegan thinking that denies death, claiming inspiration from ecofeminism and hydrofeminism, she aims to
provoke a discussion, while eating together, about the human relationship with the living beings on which we
feed. She creates scenarios such as a kombucha adoption for interspecies mothering—that she performed,
teaching how to care and maintain the SCOBY , beverage.
From Cooking to Codesign
The artist Maya Minder, a member of the bio-hack lab Hackteria and of the BadLab collective, has followed a
symmetrical path: from cooking to objects. Rooted into food practices and social constructs around food, she
proposes collective sensual experiences that engage people and their microbiome through macrobiotic, fermented
and vegan buffets. She invites to develop an awareness of natural processes, those permanent and invisible
movements.
She recently conceived the Green Open Food Evolution, a culinary installation involving a speculative cooking
furniture co-conceived with designers. She explains: “The idea was to create cooking that can be
activated. We made racks to display the tools of speculative cooking, as well as algae and various other
objects. Even the table is like a topographical landscape, non-uniform, that tells a story.” ²⁹
The installation was activated in public for the exhibition opening.
From Design to Ritual
Joanne Jones was trained in Industrial Design and Environmental Sustainability, then, she steered clear of
standard industrial design, refusing plastic and other straight-to-the-trash items, which were unaligned with
her values, she says. ³⁰ Joining the Nature-Inspired Design program at ENSCI in Paris, she sought to
work with an organism and began growing kombucha leather. It is not easy to bond with faceless organism that
becomes a living material. How do you honor the agency of those organisms? she asks. It became more of a
philosophical question, almost spiritual as well.
She ended up making an entire prospective design fiction about a kombucha community that works with the
organism and develops rituals with the material. Her concern was to provoke questioning: how does one work with
living organisms and living materials in respectful ways? How does one escape the exploitation pattern? The
fictional community would have to build relationships with the microorganisms and with each other, these would
be achieved through ritualistic practices based on the solstice and equinox.
Reinventing Addressing Forms
Following kombucha, I met artists with design backgrounds, an interest in food, ecological and/or political
awareness, makers of ritualistic forms as social sculpture, inspired by ecofeminist fictional tactics, and
critical, and speculative design methods. This mixture of styles relates to kombucha itself: its plurality of
micro-population and the plurality of the ways humans may look at it: as a tasty and/or healthy beverage, as a
biomaterial with uses still to be explored or stabilized or as a metaphoric form to rethink human relationship
to living being in a more harmonious way of life, as a guide from how learn a greater resilience.
Conclusion: Let’s Explore Spiritual Aspects
During this year and a half of meetings and interviews with designers, artists, brewers and engineers, stories
and analyses about kombucha were gathered. However, the relationship with beliefs, superstitions and magical
rituals is not sufficiently explored. I found traces of kombucha’s involvement in divinatory practices.
This needs to be continued, not in a religious or dogmatic aim (searching for ways to build beliefs and
manipulate people) but, on the contrary, to develop a critical approach to our Western lifestyle (and diet),
looking for clues of the cultural revival we absolutely need in the contemporary context. This intuition follows
the common assumption of speculative design and ecofeminism: the power of fiction to change the world by
changing our imaginaries deserves to be investigated.
Acknowledgements
My introduction to and subsequent coexistence with kombucha are the result of several generous interpersonal
exchanges. Fermentation is a network. I want to thank warmly: Felixe Kazi-Tani for bringing fermented living
beings (kimchi, kefir and homemade sourdough) into my daily life and for the political reflexive talks about it,
as it relates to feminist and queer culture; Maya Minder and Alexia Venot for bringing kombucha into my life,
especially Maya who taught me how to care for them; Mial Watkins from Vivant Kombucha brewery for our generous
talks and for donating cellulose; for sharing their experience about their interactions with kombucha: Joanne
Jones, Siddhant Malviya, Elin Margot, Loraine Fournier, Jeanne Mainneti (also for being a fantastic partner in
kombucha rituals); Vivien Roussel (also for opening thr34d5 gates and for ongoing discussion since); Pascal
Dhulster for challenging talks about by-product and co-product notions and engineering perspective; Justin
Monteiro who was my trainee in January 2022; the University of Nîmes (especially Mr P.) for tolerating the
different smells of this experimentation; special thanks to my colleague Karen Brunel for careful
proofreading.
References
1 Biomaterials, Fermentation and Kombucha, workshop from January 17th to 27th 2022 for Bachelor
Design students proposed by Lucile Haute with Pascal Dhulster (professor in process engineering, Lille /
Nîmes FR), Corinna Mattner (designer, Zurich CH), Maya Minder (artiste, Zurich CH), Vivien Roussel
(maker, Paris FR), Alexia Venot (designer, Paris FR), Mial Watkins (brewer, Paris FR). See the report
published by one of the stakeholders: Maya Minder, “Biomaterials, Fermentation and Kombucha at Nimes
University” (Makery, Feb. 2022)
https://www.makery.info/en/2022/02/16/biomateriaux-fermentation-et-kombucha-a-luniversite-de-nimes/
2 These artistic contexts where two performances proposed for the opening (Installer le cercle
— opening performance with Jeanne Mainetti, Laura Ego, Mounia Ali, Houria Ali, Éloïse
Komara, Dande Keimba-Waza on May 14th 2022) and the closing (Départ — performance with
reading, gong and kombucha adoption on June 4th 2022) of my solo exhibition Cyberwitches Coven in the
art space Les Limbes in Saint-Étienne (France) during the International Design Biennale, from May 14th
to June 4, 2022.
www.biennale-design.com/saint-etienne/2022/fr/a/le-cercle-des-cybersorcieres-cyberwitches-coven-1471
3 Sources: Wikipedia pages for “kombucha” in French and in English.
4 Lauren Fournier (ed.), fermenting feminismn, Laboratory for Aesthetics and Ecology,
2017.
5 Lauren Fournier, “Fermenting Feminism as Methodology and Metaphor. Approaching
Transnational Feminist Practices through Microbial Transformation”, Environmental Humanities
12:1, May 2020, 88-112. DOI 10.1215/22011919-8142220
6 Quote from the Clubster NHL’s website: clubster-nsl.com
7 Olivier Ducuing, “Lesaffre achète le groupe italien Gnosis” (Les Echos, 2 nov.
2015). www.lesechos.fr/2015/11/lesaffre-achete-le-groupe-italien-gnosis-256620
8 Quote from the brand’s website: www.lesaffre.com/healthcare/gnosis/
9 Press release is published on Biotech Info
https://biotechinfo.fr/article/gnosis-by-lesaffre-rejoint-linitiative-le-french-gut/
10 Sandor Ellix Katz, The Art of Fermentation. An In-Depth Exploration of Essential
Concepts and Processes from Around the World (Chelsea Green Publishing Co, 2012), 19.
Fermentations!, Mens, Terre vivante, 2018, 19.
11 Rene Redzepi, Noma: Time and Place in Nordic Cuisine, Phaidon Press, 2010.
12 David Zilber and Rene Redzepi, The Noma Guide to Fermentation, Artisan Publishers,
2018, Published the same year in French: Le guide de la fermentation du Noma (Chêne, 2018).
13 Food Phearking, Issue 03: Gut Gardening, Center for Genomic Gastronomy, 2016.
14Ibid., 4.
15 The term is from Nicolas Twilley. Ibid., 7.
16 Tunde Wey interviewed by Léopold Lambert, “Dinner as Demonstartion Cooking as
Underlying Food as Reparation”, in The Funambulist, Issue 31, Politics of Food, 2020. This
issue examines “various dimensions behind food and the act of cooking: labor, ingredients, memories,
identity, self-orientalization, marooning, infrastructure, transportation, societal hierarchies
andspace” and is available online in open access: thefunambulist.net/magazine/politics-of-food
17; See: Cooking Sections (Daniel Fernández Pascual and Alon Schwabe), The Empire
Remains Shop (Columbia Books on Architecture and the City, 2018); and: empireremains.net
18 Donna Haraway, When Species Meet, Minneapolis, the University of Minnesota Press,
2008, 295.
19 Press kit of the event.
20 All the information about Vivant Kombucha and the quote are from the interview with Mial
Watkins, co-founder, in January 2022.
21 Jonathan Nossiter, Cultural Insurection, Other Press LLC, United States, 2019, First
published in French, Stock, 2015.
22 Audre Lorde, “The Master’s Tools Will Never Dismantle The Master’s
House”, in Cherrie Moraga and Gloria Anzaldùa (eds), This Bridge Called My Back: Writings by
Radical Women of Color, New York, Kitchen Table Press, 1983, 94-101.
25 Lauren Fournier, foreword of fermenting feminism (op. cit), 3.
26 Ibid.
27 Lauren Fournier, “Fermenting Feminism as Methodology and Metaphor”
(op.cit.), 89.
28 Source: interview with Elin Margot, February 2022.
29 The Green Open Food Evolution cooking furniture is commissioned by Maya Minder and designed by
Victor Yvin and Pacôme Gérard (Designers Artisans), Gabriel Violleau (Bientôt) and Ewen
Chardronnet (Makery). Source: Pauline Briand, “Green Open Food Evolution, a speculative exploration of
algae and the food transition” (Makery, Oct. 2022), online:
https://www.makery.info/en/2022/10/20/green-open-food-evolution-une-exploration-speculative-sur-les-algues-et-la-transition-alimentaire.
Source: interview with Johan Jones, January 2022.
Bibliography
Hubert Antolak, Dominik Piechota, Aleksandra Kucharska, “Kombucha Tea—A Double Power of Bioactive
Compounds from Tea and Symbiotic Culture of Bacteria and Yeasts (SCOBY)”, Antioxidants Volume 10 Issue
10: Advances in Natural Antioxidants for Food Improvement, 2021, doi.org/10.3390/antiox10101541
Pauline Briand, “Green Open Food Evolution, a speculative exploration of algae and the food
transition", Makery, Oct. 2022, online:
https://www.makery.info/en/2022/10/20/green-open-food-evolution-une-exploration-speculative-sur-les-algues-et-la-transition-alimentaire
Cooking Sections (Daniel Fernández Pascual and Alon Schwabe), The Empire Remains Shop,
Columbia Books on Architecture and the City, 2018.
Center for Genomic Gastronomy, Food Phearking: Gut Gardening, Issue 03, 2016,
foodphreaking.com/issue03pdf/FoodPhreak-ing_03.pdf
Lauren Fournier (ed.), Fermenting Feminism, Berlin and Copenhagen, Laboratory for Aesthetics
and Ecology, 2017.
“Fermenting Feminism as Methodology and Metaphor. Approaching Transnational Feminist Practices through
Microbial Transformation”, Environmental Humanities 12:1, May 2020, 88-112,
doi.org/10.1215/22011919-8142220
The Funambulist, Issue 31, Politics of Food, 2020, Open access available:
https://thefunambulist.net/magazine/politics-of-food
Donna Haraway, The Companion Species Manifesto. Dogs, People, and Significant Otherness,
Prickly Paradigm Press, 2003.
Donna Haraway, Manifeste des espèces compagnes. Chiens, humains et autres partenaires, Paris,
Climats, 2019.
Donna Haraway, Staying with the Trouble. Making Kin in the Chthulucene, Duke University Press,
2016.
Donna Haraway, Vivre avec le trouble, Paris, Les éditions des mondes à faire, 2020.
Donna Haraway,“Anthropocene, Capitalocene, Plantationocene, Chthulucene: Making Kin”,
Environmental Humanities, vol.6, 2015, p.159-165, doi.org/10.1215/22011919-3615934.
Donna Haraway, “Anthropocène, Capitalocène, Plantationocène, Chthulucène. Faire
des parents”, Multitudes 2016/4 n° 65, 75-81, doi.org/10.3917/mult.065.0075
Sandor Ellix Katz, The Art of Fermentation, Chelsea Green Publishing Co, 2013.
Bruno Latour, Face à Gaïa, Huit conférences sur le Nouveau Régime Climatique,
Paris, La Découverte, 2015.
Audre Lorde, “The Master’s Tools Will Never Dismantle The Master’s House”, in Cherrie
Moraga and Gloria Anzaldùa (eds), This Bridge Called My Back: Writings by Radical Women of Color,
New York, Kitchen Table Press, 1983, 94-101. doi.org/10.1515/9781474470254-003
Carolyn Merchant, “The Scientific Revolution and The Death of Nature”, 2006.
Maya Minder, “Biomaterials, Fermentation and Kombucha at Nimes University", Makery, Feb. 2022,
https://www.makery.info/en/2022/02/16/biomateriaux-fermentation-et-kombucha-a-luniversite-de-nimes/
MOLD Magazine Issue 01: Designing for the Human Microbiome, 2017.
Jonathan Nossiter, Olivier Beuvelet, Cultural Insurrection: A Manifesto for Arts, Agriculture, and
Natural Wine, Other Press, 2019.
Jonathan Nossiter, Olivier Beuvelet, Insurrection culturelle, Paris, Stock, 2015.
Victor Papanek, Design for a Real World, Pantheon, 1971.
Victor Papanek, Design pour un monde réel, Dijon, les presses du réel, 2021.
René Redzepi, David Zilber, The Noma Guide to Fermentation, Artisan Publishers,
2018.
Le guide de la fermentation du Noma, Paris, Éditions du Chêne, 2018.
Anna L. Tsing, “Unruly Edges: Mushrooms as Companion Species”, environment and society, 2012,
www.environmentand-society.org/mml/unruly-edges-mushrooms-companion-species
David Zilber, “Pathways for Waste. Beyond Fermentation at Noma’s R+D Lab”, MOLD Magazine
03: Waste, 2018, 12-17.
Author Biography
Lucile Olympe Haute is an artist, researcher and educator. Her research brings together spirituality,
technologies and politics — understood in the broader sense of a committed and eco-responsible "living
together" that goes beyond anthropocentrism and includes the plant, animal and fungal kingdoms, even the
communities of bacteria in our biotopes. Synthesizing these issues, she wrote the Cyberwitches Manifesto
to bring together witches from different backgrounds, politically committed (ecofeminism,
queer/transfeminism, technological emancipation) and involved into empowerment processes. Her work often
manifests through the aesthetic and narrative versatility of web-to-print design tools. She is lecturer in art
& design at the University of Nîmes (FR) and associate artist at École des Arts
Décoratifs–PSL (FR). lucilehaute.fr | projekt.unimes.fr |
ensadlab.fr
Towards an Intelligent IoT System for the Data-Informed Museum of the Future
YannickHofmann
intelligent.museum (ZKM | Center for Art and Media Karlsruhe & Deutsches Museum
Nürnberg) Karlsruhe, Germany contact@yannickhofmann.de
Abstract
This article describes the ongoing prototypical development of an intelligent IoT system for art and cultural
institutions as part of the intelligent.museum project. Partial development steps that have already taken place
are described and the system is conceptually framed. The system will be used to condense data from various IoT
sensors in the exhibition space and the building complex, along with data from the Internet, into intelligent
data analytics. The goal is to provide human decision-makers with intelligent analytics as well as machine
recommendations for action to enhance museum intelligence. This initiative, which can be tracked transparently
and comprehensively via a code repository on the Internet, will enable art and cultural institutions to adapt
museum experiences to the needs of visitors. In addition, a technological framework is created for artists to
create computer-based works and develop interactive art experiences using this system.
Keywords
Museums, Augmented Intelligence, Artificial Intelligence, Internet of Things, Data Mining, Interactive Art,
Generative Art.
The intelligent.museum project aims to make the museum experience of the future more inclusive and accessible
by developing technical solutions and rapid prototypes for human–computer interaction between the museum
and exhibition visitors. From new conversational user interfaces that mimic a conversation with a real human
(e.g., chatbots) to machine vision systems that enable, for example, the recognition and tracking of both
objects and people, facial expressions, gestures, or postures, AI models can be deployed today that are
particularly well-suited for processing the auditory, visual, and textual input of visitors. These in turn can
be evaluated using AI, whereupon the museum experience can be tailored to the specific needs of individual
visitors.
In order to combine live data from the museum with data from the Internet within an intelligent IoT system and
make it the subject of AI-supported data analyses, various prototype developments are currently taking place
within the framework of the project. These developments concern the standardization of procedures and approaches
in the field of integrating sensors into an open-source IoT platform, the development of a scalable LiDAR-based
visitor tracking system for the exhibition space as well as the implementation of machine learning tools for
data analysis.
In addition to the partial automation of system-irrelevant processes by means of actuators reacting to sensor
inputs, the long-term goal of the project is to create a system of augmented museum intelligence that is
intended to expand the intelligence of human operators as well as their basis for decision-making by means of
data-driven analyses and suggestions for action.
This can be best described as a system of augmented intelligence.1 "Augmented intelligence is a
design pattern for a human-centered partnership model of people and artificial intelligence (AI) working
together to enhance cognitive performance, including learning, decision making and new experiences." ² A
combination of data science, machine learning and human intelligence, augmented intelligence pairs computer
artificial intelligence with human intelligence, aiming to improve human decision-making capabilities. In this
process, AI tools evaluate Big Data, for example, large amounts of data that can hardly be grasped by humans, in
order to subsequently provide results and data analyses to human operators. In such a scenario, an AI system
provides humans with a basis for decision-making so that they can make decisions even faster and more precisely.
The goal is to support human decision-making processes, not to replace humans with AI.
The overall goal of this development is to improve the exhibition experience for visitors and to optimize
museum operations. Used as a technical analysis tool, new potentials for visitor research should also emerge.
Media artists who create interactive installations, generative visualizations or sound recordings, for example,
can integrate the live data into their artistic works on the one hand, but also provide data they have collected
themselves on the other. The intelligent IoT system is emerging as a tool for data-informed optimization of the
museum experience and is intended as a research contribution to the potential uses of AI in museums.
The museum is gradually forming a sensory perceptual apparatus, developing cognitive capabilities and thereby
becoming a cognitive system itself in the long term.³
Visitor interaction with the museum of the future
Through innovative technical interfaces, the museum of the future will be able to react to the needs of
visitors and adapt itself accordingly.
In the intelligent.museum project, experiments were conducted with the use of Deep Learning in
connection with linguistic user interfaces. In the field of speech-based interaction with machines, rapid
development steps have been taking place for years, whereby speech has increasingly become a natural and
self-evident way of interacting with devices such as Google Home, Amazon Alexa or the Siri software on Apple
devices. In everyday life, then, these voice-based systems are already in frequent use. Deep Learning is being
used here as well.
Example As part of the exhibition BioMedien at the ZKM, the prototype of an
AI-supported museum label for the exhibition space was presented, which was developed as part of the
intelligent.museum project. This prototype can recognize the national language from the spoken language
of visitors and, as a result, automatically translate given texts into the corresponding language. For the
training of the speech recognition, audio samples of the speech corpus Common Voice by Mozilla were
used. In addition, to make the model more robust to noise, Google's dataset AudioSet was used. This
dataset contains manually annotated ten-second audio recordings from the online video platform YouTube.
To find out how much data is needed to distinguish between two similar languages, an experiment was conducted
in which the neural network was challenged to distinguish English from German. This experiment found that 30,000
audio samples did not provide enough data for the model used, despite data augmentation. At 5 seconds per audio
sample, that's over 40 hours of speech data. For most languages, 40 hours of training data is not yet available
in the Common Voice training dataset used. Thus, in order to train the system on many languages in the long run
and thereby make the museum experience more accessible and inclusive for visitors of different nationalities,
the availability of open-source international speech data needs to be increased. In the medium term, languages
from the global South in particular need to be more strongly reflected in relevant language datasets. ⁴
With this motivation in mind, the Artificially Correct Hackathon, hosted by the Goethe-Institut in
partnership with the ZKM from October 1-3, 2021, issued a challenge to develop a non-contact capture station for
crowdsourcing an international language dataset. Developed collaboratively between ZKM and three international
developers at the end of 2021, the interactive Data Collection Kiosk allows visitors to create new
language datasets by recording their own language. In addition to recording them, the Kiosk also serves to
validate the recordings: They can be listened to and accepted or rejected for archiving. The speech samples that
are created during the exhibition period are collected and integrated into a multilingual speech dataset. This
will serve the further development of AI-supported speech dialog systems over the exhibition period of
BioMedien.
This example will help to determine to what extent art and cultural institutions can contribute to the
development of data sets and thus demonstrate to what extent the existence of a special sensor-technical
infrastructure can favor the crowd-based collection of this data. Especially through the use of AI systems, as
indicated above, new and innovative visitor interactions with the museum become possible. The following part
will therefore describe the prototypical development of an intelligent IoT system to stimulate novel modes of
interaction between visitors and the museum and to enhance museum intelligence.
Sensors
Similar to the sensory organs with which we humans perceive our environment, sensor technology enables the
conversion of chemical or physical properties into electrical signals. From kitchen scales to smartphones, such
technical sensors are omnipresent in our everyday lives. The atriums of the ZKM are also equipped with numerous
sensors: Multisensors measure CO2, light intensity, temperature or fine dust in the exhibition rooms, smart
recording systems count exhibition visitors and provide information about the power consumption in the building.
Both active sensors (e.g., LiDAR, ultrasonic sensors, etc.) and passive sensors (e.g., cameras, microphones,
light sensors, etc.) are used.
In order for visitors to be able to interact with the intelligent museum of the future, its technical
infrastructure must enable environmental parameters to be recorded via sensors on the one hand and certain
actions to be triggered via so-called actuators on the other. Just as there are different types of sensors,
there are also different types of actuators—from mechanical actuators (e.g., electric motors) to optical
actuators (e.g., light bulbs, LEDs, screens) to acoustic actuators (e.g. loudspeakers). An example of a
sensor-actuator system is an escalator: A light barrier (sensor) detects the presence of a person, which causes
a motor (actuator) to set the stairs in motion.
IoT System
As Researcher-in-Residence, Dr. Andreas Kugel is working with the project team on a best-practice example of
sensor-based data collection and provision as part of intelligent.museum. This is intended to be a
first development step towards an intelligent IoT system, with the short-term goal of allowing things to
communicate with each other and the medium-to long-term goal of establishing the data basis for AI-supported
analyses. As an extension of the decision-making capabilities of museum staff, learning systems should be able
to analyze and draw conclusions from the various data. In the future, for example, digital artworks should be
able to be integrated into the IoT ecosystem—also for the purpose of remote maintenance, data mining or
against the background of many other conceivable needs.
The open-source ThingsBoard platform was selected as the temporary backbone for the IoT system. The main tasks
of the ThingsBoard platform here are the storage of data as well as its post-processing and visualization. The
concept of ThingsBoard is that different devices can be created, i.e., normally each external sensor is
connected to its own device in ThingsBoard. The devices can be created automatically. In order to cover a
certain number of different scenarios, so-called device profiles can be defined and new devices can be added for
them (automatically, if necessary).
For the visualization of data by means of dashboards, standardized display types—from tables to heat maps
to bar, line, pie or bubble charts—can be selected. However, widgets can also be customized—for
example, a visitor counter could be displayed as a Brownian molecular motion.
Example On the occasion of the exhibition BioMedien, a ZKM-specific dashboard was
developed at the end of 2021 with the platform ThingsBoard, which can be viewed in the exhibition space and
online. Via the dashboard, (sensor) data of the ZKM from the indoor and outdoor areas as well as from the
internet will be visualized in real time. This multifaceted data is generated by smart devices, sensors and
other IoT devices. Indoors at ZKM, these include the multi-sensors installed in the exhibition space that
measure CO2, light intensity, temperature, and particulate matter. Also, LiDAR sensors, ultrasonic sensors and
optical hand tracking modules. From the employee elevator, a sensor sends the elevator position with associated
time stamp, which in turn can calculate the speed. There are also IoT sensors in the outdoor area of the ZKM
(meadow in front of the ZKM/HfG building + rented meadow orchard), which use the long-range energy-efficient
LoRaWAN (Long Range Wide Area Network) for information and data transmission. ³ On the Internet, data is
collected through API calls (for example, to retrieve data from websites), tracking tools (for web analytics),
or web crawlers. [Figure 1]
Some of the data is already used for a ZKM-specific data set on the occasion of the hackathon {COD1NG DA
V1NC1}. This dataset called We are data - The ZKM as a living organism does not present work
data or digitized data of museum collection objects, but is intended to represent the everyday life of a living
and constantly changing cultural institution. In addition to the sensor data mentioned above, it also includes,
for example, data on which books the library of the ZKM and the Staatliche Hochschule für Gestaltung
Karlsruhe lends out per day or how many words the ZKM's publications department edits. The dataset contains
day-based values for the month of March 2022 and was published under the open-source license CC BY-SA 4.0.[4]
Sensor-based visitor tracking
In addition to the aforementioned data from IoT sensors, tracking data represents a pertinent resource for the
data-informed museum of the future. In order to be able to track people and objects indoors and outdoors in a
way that complies with data protection laws and preserves privacy rights, a LiDAR-based system for
two-dimensional real-time location has been developed since the end of 2021 by Bernd Lintermann—artist and
software developer at ZKM—and was released open-source in 2022.
The system was initially developed for two use cases. The first use case envisions the media-artistic use of
the system for interactive installations, for example. In this area, optical sensors are often used to receive
input from users by tracking people in the installation's catchment radius. The second use case sees the system
as an analysis tool for museum operations, providing information on how museum areas are used, visitor flows,
and where visitors spend most of their time. This should help to successively improve the exhibition experience
and enable museum operators to optimize the museum infrastructure.
LiDAR (Light Detection and Ranging) is a telemetry method related to radar, based on time-of-flight measurement
with pulsed laser beams in the eye-safe range.
The application areas for LiDAR-based acquisition systems range from geosciences—e.g., in meteorology or
geodesy—to the automotive industry—e.g., for driver assistance systems or in the field of autonomous
driving—to robot navigation. Increasingly, LiDARs are also being used in consumer devices. For example,
since 2020, support for LiDAR scanners has improved AR applications in iOS devices.
When integrated into smart applications, LiDARs open up new practical sensing opportunities. However, a
LiDAR-based scalable system for real-time two-dimensional location of people and objects indoors and outdoors,
based on open-source software, does not exist at the time of writing.
For the tracking system, 2D LiDAR sensors are used on the hardware side. 2D LiDAR sensors use 1D LiDAR sensors
that rotate clockwise and whose optical distance measurements can be combined by the computer into
two-dimensional environmental data. The most powerful LiDAR sensors tested during the system's development
measure distance data approximately 8000 times per second.
The data from all the sensors in the system is combined by Lintermann's software and transferred to a unified
virtual space in which the detected objects are distinguished from one another. The system checks where the
objects were located in the previous frames. If there is a positional discrepancy, the system assumes that an
object has moved. Detected objects are tracked and IDs are assigned to them. The system can track the movement
of objects and determine if certain objects have left an area or how long certain objects have been in an area.
[Figure 2]
The software design of the system aims to be as user-friendly as possible. This means that the software should
first be as easy to install as possible and then as easy to use as possible via an intuitive graphical user
interface.
The affordability of the system is also intended to increase broad accessibility, which is why different LiDAR
sensors from various vendors and price ranges were tested. The software uses SDKs (Software Development Kit =
collection of programming tools and program libraries used to develop software) from different manufacturers,
whose respective model range—from low to medium to high-priced devices—is to be covered.
The system is also scalable, which in this context means that theoretically any number of LiDAR sensors can be
used. This means that a large physical space can be covered, but it is also possible to interconnect multiple
spaces, thereby creating particularly individual tracking areas.
The software offers the possibility to log the accruing movement and whereabouts data and to write them to
files in a common data format—JSON—that can be processed very well by machines. These logging files
record when objects have changed their location, i.e., moved several centimeters to the side, for example. A new
entry is then created, with a timestamp, so that it can be traced how long which object was in which position
and over what distance it moved.
A special feature of the system is the logging of this data in graphics, because the movement and dwell time
data can be used to generate so-called heat maps, from which it can be read where people have been particularly
frequently.
The logging data can be configured and put into a temporal structure so that, for example, a new file is
created for each freely definable period of time—if users are interested in specific periods of time, they
only need to look at the files for these specific periods.
Compared to conventional camera technology, LiDAR sensors offer both advantages and disadvantages when used for
data acquisition systems. First, LiDAR sensors are faster and less computationally intensive than camera
technology, making the terminals used to process the data more cost-effective. LiDAR sensors also do not rely on
ambient light, whereas conventional cameras are inefficient in adverse weather conditions or darkness. Thus, no
artificial lighting needs to be considered for the use of LiDAR sensors.
However, there are also aspects of technology ethics that argue against the large-scale use of cameras in the
exhibition space for a room-scale detection system for people and objects. These aspects relate to security,
anonymity and privacy, because cameras naturally also capture biometric information, whereas the data generated
by LiDAR-based systems do not allow any conclusions to be drawn about individuals. Neither body poses nor
gestures and facial expressions of individual visitors can be captured with the system.
The "shadowing" of persons in the exhibition space by other exhibition visitors requires that significantly
more LiDAR sensors be used under real conditions than would be the case under laboratory conditions.
Implementing Communication Protocols
In collaboration with artist and software developer Bernd Lintermann and researcher-in-residence of the
intelligent.museum project, Dr. Andreas Kugel, Dan Wilcox—one of the project’s two
permanent software developers—is currently working on using the open-source ThingsBoard IoT data server as
a streaming data platform for the recently developed LiDAR-based tracking system mentioned above. The idea is to
develop simple, open tools that allow creative programmers to interface with the ThingsBoard. This way, other
artists can easily connect and receive live tracking data or general sensor data from the ZKM.
As part of this development strand, thoscy (Thingsboard OSC
Relay) was developed and released open-source. ⁵ This is a collection of scripts
for relaying messages between a ThingsBoard server and the OSC network protocol. These scripts act as relay
servers for forwarding device events between a ThingsBoard server via MQTT/WebSockets and OSC (Open Sound
Control) messages. thoscy-send forwards messages from OSC to a ThingsBoard host via MQTT, while thoscy-recv
listens for ThingsBoard events via a WebSocket and forwards them via OSC. This is especially useful for software
tools that work natively with OSC messages but do not have built-in MQTT or WebSocket support.
Outlook and Desiderata
In the next part, going beyond the technical development of the intelligent IoT system, various desiderata will
be outlined by way of example, which are to be fulfilled with the system and which will be implemented
prototypically.
Example 1: Data Mining for Museum Visitor Research
While data analytics is used to optimize business processes and decisions in the corporate sector, it is still
comparatively rarely used to optimize museum operations. As a best practice example from the Barberini Museum
shows, data mining can be used in the museum sector to improve the museum experience for visitors. ⁸
The Barberini Analytics analysis platform is used to collect, process, analyze and present available data
relating to museum visits with the aim of improving the museum experience. Barberini Analytics was developed for
the Museum Barberini between 2019 and 2020, but also includes approaches transferable to other art and cultural
institutions. The suite was compiled by the Hasso Plattner Institute and the Museum Barberini in Potsdam,
founded by the Hasso Plattner Foundation, as part of a bachelor's project supervised by Prof. Dr. Felix Naumann
at the Hasso Plattner Institute in Potsdam entitled Data Analytics - Optimizing Museum Experiences with Data
Analysis. The program codes were published as an open-source project via a code repository on GitHub.
⁹
While predictive models can be developed for museum operations, for example, to help forecast visitor flows and
numbers,¹⁰ [figure 3] new potentials of the system described above as an analytics tool for visitor
research can be anticipated in particular.
In 1983 Veron & Levasseur conducted ethnographic observations in an experiment at the Louvre in Paris and
analyzed the visiting style of museum visitors. They classified them into four categories. The categories are
named after animals, because the behavior of the visitors is supposed to be similar to the behavior of these
animals: "ants," "grasshoppers," "butterflies," and "fish." While the visitors classified as "ants" follow a
certain path through the exhibition space and look closely at all exhibits, the visitors identified as
"grasshoppers" seem to prioritize some previously selected exhibits. The group of "butterflies" includes
visitors who wander aimlessly around the museum, but are very interested in the exhibits and try to get more
information. Visitor:ings classified as "fish" spend most of their time moving around the center of the room
without seeming interested in details of individual exhibits. In the mid-1990s, Tsvi Kuflik, a professor of
information systems at the Faculty of Social Sciences at the University of Haifa in Israel, and a team conducted
the mobile museum guide experiment with 143 participants, using two different clustering methods (artificial
neural networks and the k-means algorithm) and comparing them. ¹¹
Using the already described LiDAR-based tracking system as a sensor within the also already described
intelligent IoT system, another experiment will be initiated with reference to Veron & Levasseur and Kuflik
et al. respectively, for which AI can be used to analyze the "visiting style" of exhibition visitor:s in the
intelligent museum. This experiment will investigate the suitability of the intelligent IoT system as an
analysis tool for visitor research.
Example 2: Automatizing Museum and Exhibition Technical Services
In the technical museum operation, repetitive tasks can be automated sporadically by software and algorithms,
such as the regulation of lighting and sound in the exhibition space—from the light intensity of the
ceiling lighting to the volume of sound installations. Since 2020, systems have been in use in the ZKM's
exhibition operations that link the unmuting of videos in the exhibition space to the presence of visitors
within a freely definable retraction radius defined by the museum's technical staff, and thereby couple the
distance of the visitors to the playback device with the volume parameter. This can prevent cacophony in the
exhibition space when many videos with sound or sound installations are presented simultaneously in one
exhibition space. Used carefully, automation can lead to a sustainable improvement of the exhibition experience
for the visitors—both in terms of ecological (light) and design aspects in the exhibition space (sound).
However, automation should only be used very sparingly and exclusively in less sensitive areas.
There are many other potential applications in the field of technical services for museums and exhibitions,
such as predictive maintenance or anomaly detection to monitor and, if necessary, maintain computerized
exhibition exhibits.
Example 3: Framework for Interactive art
In addition, the intelligent IoT system will provide artists with a technical framework for developing
interactive art experiences. In the medium term, it should become possible for artists to feed and operate
generative data visualizations and sound recordings, interactive installations and environments with sensor data
from the system. Similarly, many media-art artworks now generate data on their own, which they can in turn feed
into the intelligent IoT system.
In this context, Dr. Andreas Kugel is already working in parallel with the artist Katrin Hochschuh of the
artist:in duo Hochschuh/Donovan on the resource-efficient transmission of position data of a robot swarm
(artwork Empathy Swarm by Hochschuh/Donovan, which has been further developed within the project
intelligent.museum) to a ThingsBoard device.
Conclusion
The diffusion and successive improvement of AI tools has enabled many new interactions between visitors and the
museum. In the analysis, it became clear that for an efficient integration of these into museum operations, the
technical infrastructure in particular needs to be looked at and that specialized systems would need to be
developed to perform intelligent analyses via data mining so that they can actually extend museum intelligence
via suggested action suggestions to museum staff. As Harry Armstrong et al. point out "[...] for many arts and
cultural organisations there is a long way to go before it is possible to extract value from the data they hold.
Data input and analysis both require skills and specialisation in the workforce. Currently, in many
institutions or organisations data is often not systematically collected and, even where it is, it may not be
analysed in depth. Developing the sector’s capabilities in this area will require attention at all stages
of the skills pipeline [...]" ¹² The ongoing prototypical developments described in this text are
running towards the development of an intelligent IoT system in the medium term. What seems particularly
interesting in connection with the development of an intelligent IoT system is that both museum operational data
mining scenarios and those concerning the museum in general but also purely artistic scenarios can be realized,
which immensely increases the spectrum of possibilities for use. The most important criteria of the described
system are its transparent and open- source development, which makes it possible to reuse the codes
used, e.g. by other art and cultural institutions, and ensures that the technologies used have as little
potential as possible to cause harm to museum visitors, be it through discrimination or inherent forms of bias.
References
1 Moritz Kirste, "Augmented Intelligence – Wie Menschen mit KI zusammen arbeiten“, in
Künstliche Intelligenz, ed. Volker Wittpahl, Berlin and Heidelberg, Springer Vieweg, 2019,
p.58-71.
2 Gartner IT Glossary, Augmented Intelligence, Gartner Inc. online, accessed
December 9, 2022, https://www.gartner.com/en/information-technology/glossary/augmented-intelligence.
3 Lívia Nolasco-Rózsás, Yannick Hofmann, "The Museum as a Cognitive System of
Human and Non-Human Actors“, The Garage Journal: Studies in Art, Museums & Culture, Vol.
03, accessed December 2022, 2021, p.1–15, https://thegaragejournal.org/files/07/758_c370a5422f798cbd308f
d337837b93dc25ec66d8.pdf.
4 Yannick Hofmann, "Projektvorstellung intelligent.museum“, in AI: A Museum
Planning Toolkit, ed. Johannes Bernhardt, Tabea Golgath, Oonagh Murphy, Sonja Thiel Elena Villaespesa,
London: Goldsmiths University, 2020, 13f.
5 LoRa Alliance, Technical Marketing Workgroup 1.0, LoRaWAN What is it? A technical
overview of LoRa® and LoRaWANTM, accessed December 9, 2022,
https://docs.wixstatic.com/ugd/eccc1a_ed71ea1cd969417493c74e4a13c55685.pdf.
6 COD1NG DA V1NC1, Wir sind Daten – Das ZKM als lebender Organismus, accessed
December 9, 2022, https://codingdavinci.de/daten/wir-sind-daten-das-zkm-als-lebender-organismus.
7 intelligent.museum on GitLab, thoscy, accessed December 9, 2022,
https://git.zkm.de/Hertz-Lab/Research/intelligent-museum/thoscy.
8 Esther Knuth, Remigiusz Plath, "Mit Daten das Museumserlebnis verbessern.
Besucher:innen-Datenanalyse im Museum Barberini“, in Kultur Management Network Magazin Nr. 160:
Digitale Besucher:innen, accessed December 2022, 2021, p.26-32,
https://www.kulturmanagement.net/dlf/00cf0ded4596ca16ee75e0189c9f0d57,3.pdf
9 Museum-Barberini on GitHub, Barberini-Analytics, accessed December 9, 2022,
https://github.com/Museum-Barberini/Barberini-Analytics.
11 Massimo Zancanaro, Tsvi Kuflik, Zvi Boger, Dina Goren-Bar, Dan Goldwasser, "Analyzing Museum
Visitors" Behavior Patterns“, in User Modeling 2007. UM 2007. Lecture Notes in Computer
Science(), vol 4511, ed. Cristina Conati, Kathleen McCoy, Georgios Paliouras, Berlin, Heidelberg,
Springer 2007, accessed December 9, 2022, https://doi.org/10.1007/978-3-540-73078-1_27.
12 Harry Armstrong et al., Experimental culture: a horizon scan commissioned by Arts
Council England., accessed December 9, 2022,
https://www.artscouncil.org.uk/experimental-culture-horizon-scan
Bibliography
1 Harry Armstrong et al., Experimental culture: a horizon scan commissioned by Arts Council
England., accessed December 9, 2022, https://www.artscouncil.org.uk/experimental-culture-horizon-scan
2 COD1NG DA V1NC1, Wir sind Daten – Das ZKM als lebender Organismus, accessed December 9,
2022, https://codingdavinci.de/daten/wir-sind-daten-das-zkm-als-lebender-organismus.
3 Gartner IT Glossary, Augmented Intelligence, Gartner Inc. online, accessed December 9, 2022,
https://www.gartner.com/en/information-technology/glossary/augmented-intelligence.
4 Yannick Hofmann, "Projektvorstellung intelligent.museum“, in AI: A Museum Planning
Toolkit, ed. Johannes Bernhardt, Tabea Golgath, Oonagh Murphy, Sonja Thiel Elena Villaespesa, London:
Goldsmiths University, 2020, 13f.
5 intelligent.museum on GitLab, thoscy, accessed December 9, 2022,
https://git.zkm.de/Hertz-Lab/Research/intelligent-museum/thoscy.
6 Moritz Kirste, "Augmented Intelligence – Wie Menschen mit KI zusammen arbeiten“, in
Künstliche Intelligenz, ed. Volker Wittpahl, Berlin and Heidelberg: Springer Vieweg, 2019, 58-71.
7 Esther Knuth, Remigiusz Plath "Mit Daten das Museumserlebnis verbessern.
Besucher:innen-Datenanalyse im Museum Barberini“, in Kultur Management Network Magazin Nr. 160: Digitale
Besucher:innen, accessed December 2022, 2021, 26-32,
https://www.kulturmanagement.net/dlf/00cf0ded4596ca16ee75e0189c9f0d57,3.pdf.
9 Lívia Nolasco-Rózsás, Yannick Hofmann, "The Museum as a Cognitive System of
Human and Non-Human Actors“, The Garage Journal: Studies in Art, Museums & Culture, Vol. 03,
1–15, accessed December 2022, 2021, https://thegaragejournal.org/files/07/758_c370a5422f798cbd308f
d337837b93dc25ec66d8.pdf.
10 LoRa Alliance, Technical Marketing Workgroup 1.0, LoRaWAN What is it? A technical overview of
LoRa® and LoRaWANTM, accessed December 9, 2022,
https://docs.wixstatic.com/ugd/eccc1a_ed71ea1cd969417493c74e4a13c55685.pdf.
11 Massimo Zancanaro, Tsvi Kuflik, Zvi Boger, Dina GorenBar, Dan Goldwasser, "Analyzing Museum
Visitors’ Behavior Patterns“, in User Modeling 2007. UM 2007, Lecture Notes in Computer Science(),
vol 4511, ed. Cristina Conati, Kathleen McCoy, Georgios Paliouras, Berlin, Heidelberg, Springer 2007, accessed
December 9, 2022, https://doi.org/10.1007/978-3-540-73078-1_27.
12 Museum-Barberini on GitHub, Barberini-Analytics, accessed December 9, 2022,
https://github.com/Museum-Barberini/Barberini-Analytics.
Biography
Yannick Hofmann (*1988 in Offenbach a. M., Germany) lives and works as an artist and researcher in Karlsruhe.
As the artistic director of the intelligent.museum project since 2020, he collaborates with a team of software
developers and museum visitor research experts, pushing the boundaries of hybrid formats and applications for
the future of museums. Having spent almost a decade at ZKM | Center for Art and Media, he co-directed their
artistic research and production department before joining the Deutsches Museum for a one-year research stay
in 2022. Since mid- 2023, Hofmann works for the Fraunhofer Institute for Industrial Engineering.
The Creative Design and Social Service Practice of zen_Farm
Yu-HsiungHuang,
Su-ChuHsu
suchu@mx.nthu.edu.tw ; eric@techart.tnua.edu.tw
Abstract
We developed "zen_Farm" through waste recycling and makers, with the aim of guiding people to practice how to
calm their minds. Utilize the concept of "Every drop of calm water makes all things green" to remind the world
to cherish the natural environment and live in peace with all things. The main purpose of zen_Farm is to
emphasize the calmness and concentration of "Mind Meditation". Its special feature is that people use the
stability of their heartbeats to drive the water source of zen_Farm. In recent years, the world has gradually
developed a new form of creative community that combines digital media to gather community awareness and improve
the current situation of the community, which can bring new energy and stimulate new thinking in urban areas. We
practically integrate zen_Farm into social practice, including:
(1) The installation is located at Dharma Drum Mountain, the most important Buddhist unit in Taiwan. Let the
Buddhist masters in the park use the concept of plant irrigation to reflect the natural environment to be
sustainable and green. Guide them to finally become one with nature. (2) The installation was installed at
Shakeng Elementary School in Taiwan, allowing students to renovate the campus together and irrigate the plants
through their own heartbeats to create a common memory on the campus. The zen_Farm is not only an interactive
installation art work made by makers, it makes the campus of Buddhist parks and rural schools more friendly and
beautiful, and it also supports the concept of environmental protection. We also hope to achieve the
long-standing goal of digital art creators - "Media Transparency" through the creative design and social
practice of zen_Farm. Participating in meditation activities can connect the emotional memories of the community
together and establish a shared memory in the campus.
Keywords
Zen, Farm, Maker, Interactive Installation Art, heartbeat, Creative Community.
Modern life is becoming more and more busy, and people are pursuing a better quality of life. What is a good
quality of life? The World Health Organization (WHO) has published the WHOQOL assessment, a quality of life
assessment developed by the WHOQOL Group in parallel with 15
international field centers ¹. WHOQOL includes four domains: physical health, mental state, social
relations, and environmental domain. Meditation in Buddhism can make people's mental state invisibly tend to be
calm and soothing through the guidance of meditation, so as to achieve a state of "relaxation" and
"concentration". In addition, it can also achieve the concept of "Protecting the spiritual environment"
advocated by Master Sheng Yan of the Dharma Drum Mountain.
The Rise of Makers and FabLab
The "Maker Movement" has become an important international trend in recent years, influencing the Third
Industrial Revolution. Countries around the world have integrated it into national policies and education
systems, and even applied it in various fields. In 2014, President Obama of the United States announced that the
future of American manufacturing would return to the domestic front, as the U.S. had comprehensively developed
digital fabrication (FabLab) and maker movement education, from primary schools to universities ². With the
popularity of the Maker Movement, concepts like "open science", "DIY science" or "citizen science" have
continuously influenced societal development ³. Integrating the concept of makers into social practice, the
Maker Movement represents a form of social participation.
We synergized the "spiritual environmental protection" philosophy of Dharma Drum Mountain with "natural
environmental protection" concepts. This collaboration resulted in the creation of zen_Farm, an interactive
installation art project crafted from repurposed plastic bottles, embodying the essence of maker’s DIY.
The unique feature of zen_Farm is its use of a calmness index, derived from heartbeat detection, to control the
irrigation of plants. The mechanism is designed such that the more tranquil the state of mind, the more water is
dispensed for plant nourishment.
Impact of Social Practice and Creative Communities
Taiwan's Ministry of Education has implemented the "University Social Responsibility (USR) Program" since 2018.
Promote the development of the real estate industry through the participation of university talents into the
community. At the same time, students can make more contributions and care to the local society, and establish
the university's social responsibility for community care. As universities increasingly focus on social
engagement, students are encouraged to need a full range of learning that is not limited to the classroom.
Through social practice, in addition to expanding students' learning fields, it is also the concrete practice of
civic education ⁴. Similarly, we have applied the zen_Farm concept again in the remote Shakeng
Elementary School, enabling the entire student body to paint PET bottles together, install and transform the
school campus. This allows children to engage in physical activity while also performing heartbeat detection to
irrigate plants, thereby fostering a state of calm and reflection.
We practice zen_Farm in two communities (Dharma Drum Mountain Buddhist Park, Shakeng Elementary), hoping to
achieve physical and mental tranquility (Zero), concentrate on generating energy (Energy), and become one with
the natural environment (Nature) to achieving the essence of "ZEN" is in the field of life and study. In this
way, the emotional memory of the community can be linked together. Use technology application and humanistic
care to build a common memory and create a creative community. Expect to achieve a good quality of life in the
areas of mental state, social relations, and environment assessed by WHOQOL.
Related works
Related literature on the Importance of Meditation for Everyone
There are many scientific studies confirming meditation in ZEN. For example, Lusnig, Larissa, et al., proposed
that meditation is a mental practice that can increase one's attention and calmly observe all the feelings of
the moment. Let people enter a state of tranquility and achieve emotional regulation.⁵ Tang, Yi-Yuan, et
al. also proposed that the meditation practice can promote the practitioner's ability to maintain concentration
through physical relaxation, breathing exercises, mental imagery, etc.⁶ At the same time, there are
also many artistic works applied to meditation and mental state, such as the interactive multimedia work
"ZENetic Com-puter" created by Naoko Tosa and Seigow Matsuokam (2003),⁷ which transforms traditional
meditation content into multimedia to express The artistic conception of ZEN, and guide the viewer to learn the
practice of ZEN. In addition, Hoshiyama & Hoshiyama of the University of Tokyo, Japan, also used
electrocardiogram (Electrocardiography, ECG) to analyze the value of heart rate variability (HRV) changes for
beginners in meditation practice.⁸
Related literature on the application of Maker and Fablab
With the development of digital technology in recent years, digital media has also been developed
internationally to promote community awareness, thereby improving the current situation of the community and
bringing new energy and stimulating new thinking to the community.⁹ Vossoughi et al. aimed at
disadvantaged community students and provided suitable Fablab activities according to their needs to achieve
higher social practice and educational equity.¹⁰ The "SASFAB Maker Alliance" program implemented by
Hsu et al. was launched in Taiwan in 2015. It combines "Internet of Things (IoT) and "Maker/FabLab" to carry out
social practice and other activities with artistic creative courses.¹¹
The Impact of Social Practice USR or Creative Community
In the Hangleton & Knoll community in the eastern part of England, Johnson & Monney collaborated with
the University of Brighton to conduct community participation projects through art participatory activities.
Through a participatory research approach to art, they engage researchers, artists, and community residents as
partners in a "research collective," using art to explore and exchange their life experiences with each other to
create a shared experience of the community.¹² In addition, many studies have found that the
integration of group art activities into social practice can increase the cohesion and connection of the
community.¹³
An example of the application of interactive installation art in community service
Yang and Hsu developed the IoT platform in 2017 to implement the "Windflower ESPSAS" project at Taipei Munici-
pal NanMen Elementary School in Taiwan. It guides elementary school students to use their maker spirit to make
Windflower objects by themselves to beautify the campus and community through artistic creation through social
network actions. Windflower is a windmill during the day and a colorful lantern at night. You can watch the
rotation of the windmill you made in the campus remotely through the mobile phone, and you can also change the
light color of the remote Windflower through the app, so that everyone's emotional memory extends from the
campus to the cloud.¹⁴ Also a good example of social practice.
According to the above-related works, there are not many cases integrating "meditation," "maker," "interactive
installation art" and "social practice" at the same time. This study takes this as the research goal and
proposes to take ZEN as the theme, applying zen_Farm's creative design practice in two communities.
Creative Interactive Design of zen_Farm
We explain the overall creative interactive design of zen_Farm, including the device interaction design of
water cycle, the device modeling design of heartbeat sensing, and the calculation logic of heartbeat sensing.
Interactive Design of Water Circulation Device
The installation design of zen_Farm is based on a simple "matrix arrangement" and emphasizes the "Mind
Meditation" focus on the psychological level. It uses the method of recycling PET bottles and water circulation
to build an automatic circulation irrigation system to implement the environmental protection and green energy
proposition (Figure 3). According to the size and height of the space, there are 10 rows of 7 PET bottles
installed in each row, with a total of 70 PET bottles. After cleaning the recycled PET bottles one by one, we
dig holes according to the needs of the device to facilitate the growth of plants. We use the original structure
of the bottle cap to design the various components required for the water cycle, and finally connect the head
and tail of each bottle. Finally, use the air pipe to connect all the bottles and complete the water circulation
structure through the pump.
Styling Design of Heartbeat Sensor Device
We set up a heartbeat console with a height of 100 cm next to the water circulation system, with a palm-shaped
operating area and LED lights above (Figure 4). Taking the Dharma Drum application as an example, we use the
Dharma Drum palm-shaped logo to visually guide the participants to place the left palm on the palm-shaped
operating area, and embed a heartbeat sensor in the position of the index finger to facilitate detection. The
stability of the participants' heart rate was measured. In the lower right of the operation area, the LED light
signal shows the result of the heartbeat stability (Figure 5), and it is supplemented by a sound to guide the
participant to start and end the heartbeat detection. The pedestal of the device includes three pneumatic pumps,
a set of heartbeat sensing modules, a control chip module, a sound effect playback module, and a sound effect
speaker.
Heartbeat Sensing Calculation Logic Description
The basic logic of heartbeat sensing is to measure the stability of the participant's heartbeat within one
minute as the irrigation index of the water circulation irrigation system, and display the level of stability
with LED lights. After the participants put their hands on the heartbeat sensor (Figure 6), the sensor stability
is adjusted and calibrated for the first five seconds. If the finger is kept pressed steadily until the
calibration is completed, the electronic control chip will drive the sound module to play a sound. After a short
"wooden fish sound", the system begins to collect the value of the heartbeat stability. The total length of the
sensing time is 60 seconds, and the value of every 15 seconds is used as a paragraph to calculate the sum of the
differences in heartbeats between every 15 seconds as an indicator of stability. After the heartbeat stability
calculation is completed, the electronic control chip will drive the sound module to play two short
“wooden fish” sounds, indicating that the heartbeat detection has been completed, and remind the
participant to leave the sensor on the pedestal with his left hand. Then the system will count the participants'
heartbeat stability and display the different stability levels from 1 to 10 with LED lights. The higher the
value, the more stable it is. A higher stable heartbeat value will control the pump to provide proportionally
more water to irrigate the plants. Conversely, lower values provide less water for irrigation. If the finger
leaves the sensor in the middle of the sensing period or the measurement method is incorrect, the system will
return to the initial state and wait for the data to stabilize before remeasurement (Figure 7).。
Social Service Practice of zen_Farm
Social Practice in Dharma Drum Mountain Buddhist Park
Dharma Drum Mountain Buddhist Park is the most important Buddhist research unit and meditation place in Taiwan.
In 2012, we built zen_Farm in Dharma Drum Mountain as a public art. Its appearance is minimalist in line with
the spirit of ZEN. The Master can do "Mind Meditation" exercises at any time in the park and use the index of
calmness to irrigate the Zen Heart farm plants, as shown in Figure 8.。
In the past, the meditation process was mostly guided slowly by the master through scriptures or oral
speech.¹⁵ Through zen_Farm, it is easier for community members who have no experience in meditation
to practice "Mind Meditation," and through the connection of heartbeat, water circulation and irrigation plants,
it is easier for community members to understand the importance of "Mind Meditation". Participants experience a
reduction in distraction from thinking and an increase in concentration.¹⁶ Guide them to experience
the Zen meaning of "Every drop of calm water makes all things green."
Practice at Shakeng Elementary School
Shakeng Elementary School in Hsinchu County, Taiwan is a rural primary school with only 27 students because
most of the students go to big cities to study. In 2022, we will extend zen_Farm to the campus of Shakeng
Elementary School. We encourage students to work together to make zen_Farm by DIY. The overall installation uses
rich color backgrounds. We guided the students to draw each bottle by themselves (Figure 9), and after
completing the device, they could do the "Mind Meditation" concentration exercise through the heartbeat sensor
device (Figure 10). We hope to implement the idea of environmental protection and green energy and build an
indoor automatic circulation irrigation system by integrating recycled plastic bottles and water circulation
through pneumatic pumps. In the past, meditation focused on the self-cultivation of the individual mind and
inner consciousness. We hope that with the aid of technology and the concept of USR, we can strengthen the
learning resources of rural education through technology media. In addition to the construction of technology
and hardware, we also hope to lead the students of rural schools to create their own imagination more bravely
and develop a better future from a spiritual level.
We hope that through this research, students in rural schools can be more cohesive in school-centered community
development. Students can apply what they have learned to develop the character of the community. As mentioned
by Vuorikari et al., Makers can not only solve real problems involving daily life, but also students can learn
and develop new skills and establish new meanings.¹⁷ It allows disadvantaged or remote communities to
combine local culture and resources through the “Maker Movement” to combine school and community
participation, which can inspire better creativity and future development in the community.¹⁸
Conclusion
Social practice can be studied from a variety of different topics, and in this dissertation we study from the
perspective of art and creative design. Through the zen_Farm interactive installation art combined with
discarded bottles and the spirit of maker, we have integrated into two different communities, including the
"Dharma Drum Mountain Buddhist Park" and the "Shakeng Elementary School" in a remote village. In the "Dharma
Drum Mountain Buddhist Park", zen_Farm is not only an interactive installation art work produced by makers, but
also makes it easier for people entering the park to understand the participation process of "ZEN" and "Mind
Meditation". We practice the concept of "Environmental Protection" combined with the concept of "Natural
Environmental Protection" by realizing the relationship between the psychological state and the natural
environment.
At Shakeng Elementary School in a rural area, we integrated heartbeat sensor module, ultrasonic sensor module,
pump power system, etc. with interactive media technology into zen_Farm's meditation application. In the process
of meditation, the most important thing is to calm down the state of mind to achieve spiritual protection.
Through this concept, we guide students in remote schools to learn how to use the stability of their heartbeats
to irrigate the plants in the bottle in the process of calming their minds. The paintings on these self-created
installations are rich in the life memories of each student. This installation art is also actually installed on
the campus of Shakeng Elementary School. Every day, students can use their own state of mind to irrigate their
own creations, and share their heartbeats to share the common memory of each other's life. In this way, we guide
students from remote villages to re-examine the relationship between themselves, their friends, and the natural
environment. We try to enhance the cohesion of the community through the application of technology and
humanistic awareness.
This research uses zen_Farm's creative design and social practice to achieve the long-standing goal of digital
art creators—"media transparency." Participating members are able to link the emotional memories of the
community together by participating in the meditation practice. It makes Buddhist campuses and rural campuses
friendly and aesthetically pleasing and has an inspiring effect on environmental protection. At the same time,
it also builds a shared memory for the creative community.
In the future, we will try to apply this model to different community fields, such as community farms, K12
educational institutions, etc. Through social service practice, university social responsibility and other
cooperative methods. This will strengthen the connection between communities and local students, jointly
creating a quality of life that aligns with the WHOQOL standards proposed by the WHO.
Acknowledgements
We would like to thank the National Science Association for the funding support of the "ZEN - Research on
Creative Meditation Space for Ease and Mindfulness Program" (NSC 99-2218-E-655-001, NSC 98-2218-E-655 -001), and
the Ministry of Education's "S+T+ARTS" Interdisciplinary Education and Local Natural Ecology Cultural and
Creative Oxygen Program" funding support.
References
1 WHO, “The World Health Organization Quality of Life (WHOQOL)”,
https://www.who.int/publications/i/item/WHO-HIS-HSI-Rev.2012.03
2 P. Blikstein, Maker movement in education: History and prospects. Handbook of
technology education, 2018 ,419, 437.
3 Federico Ferretti, Harro van Lente, "The promise of the Maker Movement: policy expectations
versus community criticisms." Science and Public Policy 49.1, 2022, 18-27.
4 Social Engagement, Social Innovation, Social Design, and Social Impact: Four Levels of
University Social Responsibility Practice, Journal of Higher Education Research, 2020, 12: 47-68.
5 Larissa Lusnig, et al. "Zen meditation neutralizes emotional evaluation, but not implicit
affective processing of words." Plos one 15.2, 2020, e0229310.
6 Yi-Yuan Tang, et al. "Short-term meditation training improves attention and self-regulation",
Proceedings of the national Academy of Sciences 104.43, 2007, 17152-17156.
8 Haseda, Yuki, et al. "Measurement of pulse wave signals and blood pressure by a plastic optical
fiber FBG sensor", Sensors 19.23, 2019, : 5088.
9 Arlene Goldbard, New Creative Community - The Art of Cultural Development, Oakland: New
Village Press, 2006.
10 Shirin Vossoughi, Paula K. Hooper, Meg Escudé, "Making through the lens of culture and
power: Toward transformative visions for educational equity", Harvard Educational Review 86.2, 2016, 206-232.
11 Su-chu Hsu, et al. “SASFAB - Smart Art-design Service in FabLab with IoT", The
Ministry of Education subsidizes the "Smart Life Integration and Innovation Teaching Alliance Promotion
Program", 2015-2018.
12 Helen Fiona Johnson, Nicole Monney, "Using the Arts to Support the Arts: A Creative,
Community-University Partnership Approach to Building Arts Inclusivity in Economically-Deprived Communities",
Forum: Qualitative Social Research, Vol. 22. No. 3, Freie Universität Berlin, 2021.
13 Abby Scher, "Can the arts change the world? The transformative power of community arts", New
Directions for Adult and Continuing Education 2007.116, 2007, 3-11.
14 Jheng-Chun Yang, Su-Chu Hsu, "The study and application of smart art community service with
“ESPSAS” internet of things platform." International Conference on Distributed, Ambient, and
Pervasive Interactions, Springer, Cham, 2017.
15 Shi Shengyan, "Master Sheng Yen on Chan", 1996, 122.
16 Bethany E. Kok, Tania Singer, "Phenomenological fingerprints of four meditations:
Differential state changes in affect, mind-wandering, meta-cognition, and interoception before and after daily
practice across 9 months of training", Mindfulness 8.1, 2017, 218-231.
17 Riina Vuorikari, Anusca Ferrari, Yves Punie, Makerspaces for Education and Training: Exploring
future implications for Europe, No. JRC117481, Joint Research Centre (Seville site), 2019.
18 Simon N. Leonard, et al. "Designing Maker initiatives for educational inclusion", International
Journal of Technology and Design Education, 2022, 1-17.
Thought Exhibition. On critical zones, cosmograms, and the impossible outside
DanielIrrgang
Marie Skłodowska-Curie Fellow, Centre Art as Forum, University of Copenhagen
Copenhagen, Denmark daniel.irrgang@hum.ku.dk
Abstract
The paper discusses the curatorial concept of “thought exhibition” coined by Bruno Latour and Peter
Weibel and developed in collaboration with curators, artists, and researchers during four exhibitions at the ZKM
Centre for Art and Media, Karlsruhe (Germany). Thought exhibitions transgress the distinctions between
philosophy, art, and science by testing ideas in an art museum, a space of discourse, representation, and
participation. They engage visitors in a spatio-aesthetic thought experiment by bringing them into a position
where preconceptions derived from epistemes of European Modernity are explicated and where alternatives are
suggested. The analysis focusses on the most recent exhibition, in the preparation of which the author was
involved: “Critical Zones. Observatories for Earthly Politics” (May 23, 2020 – January 9,
2022) mapped the symptoms and origins of the “New Climatic Regime” (Latour) of the late
Anthropocene. In this paper, Critical Zones is framed within its theoretical context (Descola, Haraway,
Margulis, Whithehead, among others) and discussed as relational spatio-aesthetic approach (Dikeç). The
analysis concludes with Sarah Sze’s installation “Flash Point (Timekeeper)” (2018) as one of
the exhibition’s central works – a representation, or “cosmogram” (Tresch), of a common
planet that may provide an alternative to the globalized world of late capitalism.
Keywords
Anthropocene, art exhibitions, curatorial studies, dualisms, eco-criticism, European modernity, posthumanism,
STS
“The globe is something viewed from the outside, from a Galilean point of view. The critical zone is a
view from the inside. Our show is about this contrast.” With this triangulation Bruno Latour located our
endeavour on January 22, 2018, the first day of our very first seminar week, which would be followed by six
further weeks over the course of two years. Dubbed by Latour as the ‘Critical Zones Study Group’ and
co-organized by the author, the seminar took place at the Karlsruhe University of Arts and Design, the
sister-institution of the ZKM Center for Art and Media in Karlsruhe, Germany which is located in the same
building and has been the venue of the renown ‘thought exhibitions’ Latour had realized over the
years. Our motivation was to conceptually prepare the exhibition ‘Critical Zones – Observatories for
Earthly Politics’ at ZKM (May 23, 2020 – January 9, 2022), curated by Latour in collaboration with
Peter Weibel, Martin Guinard, and Bettina Korintenberg.
Together with students, post-graduates, researchers, curators, and artists, we tried to triangulate a new
“place to land”1, now that the ground on which the globalized world of late
capitalism is built is shifting and disintegrating in the age of the Anthropocene. The widely debated potential
new geological epoch is certainly the first not only named but created by humans.2 Albeit
‘creation’ may not be the appropriate term for the devastating effects caused by capitalist
extractivism and consumption. But even less so would be expressions such as ‘accidental effect’, as
the data overwhelmingly suggesting the human cause of these ruptures has been known for a long time, despite
regressive voices touting climate denialism.3
The aim of this paper is neither to fully analyse the curatorial approach nor to catalogue the exhibitions
realized by Latour and his collaborators. This would go beyond its scope, considering the comprehensive thought
exhibitions realized at ZKM, and beyond, over the years: ‘Iconoclash. Beyond the Image Wars in Science,
Religion and Art’ (2002); ‘Making Things Public. Atmospheres of Democracy’ (2005)—this
was, in fact, the first show explicitly called a ‘thought exhibition’, although Latour
retrospectively also included Iconoclash4; ‘Reset Modernity!’ (2016). Particularly with
regards to issues of climate change, Reset Modernity! laid the groundwork for Critical Zones, which was
succeeded by ‘You and I Don’t Live on the Same Planet’ (2020/21) at the Taipei Fine Arts
Museum for the 2020 Taipei Biennial.
This paper focusses on Critical Zones while trying to trace the specific spatial characteristics of a thought
exhibition and its relation to the bodies and things—human, non-human—that constitute the
exhibition space. After reconstructing the epistemological framework with which Latour approached the
‘cosmological’ shifts and uncertainties leading up to the global crises not limited to the direct
effects of climate change, the paper discusses the visitor’s position (both in a figurative and embodied
sense). It does so at least on two levels, by first considering the exhibition space as a whole and then by
focussing on one of Critical Zones’ central artworks, a ‘cosmogram’ (John Tresch), Sarah
Sze’s ‘Flash Point (Timekeeper)’ (2018).
Dualisms
But first I want to come back to the opening quote above. Latour put forward his notion of the globe in two,
albeit intertwined, ways, a cosmological (or epistemological) and a spatial (or proxemic) one. Galileo
Galilei’s discoveries – with regards to both, scientific methodologies and astronomic bodily
movements—initiated a rupture in cosmology, as it catapulted the human from the centre of the cosmos into
a spinning orbit around Earth’s star, one of countless in the universe. Despite this displacement, the
anthropocentrism remained or was re-incorporated into the worldview of European modernity, most notably by means
of a juxtaposition of somewhat separated spheres of nature and culture.5 This
dichotomy implies an outside position from which one sphere can be acted up on by the ‘inhabitants’
of the other. The image of the globe representing Earth, a cartographic model as constructed as the
nature-culture dualism, has become, as Stephen J. Gould would have called it, a “canonical
icon”6, associated with capitalist globalism and an ideology of limitless growth. Such an
impossible teleology not only ignores the limitations of ecosystem capacities but also of ‘natural
resources’, to use a term normalized by capitalist extractivism. In this situation, the relation between
the world we live in and the world we live of is distorted. This post-colonial heritage we
discussed in our seminar through the concept of “ghost acres” which refers to exploited land abroad
to cover a given territory’s consumption (originally of food, but expandable to other goods as well).
Considering the devastating effects of such imbalances on exploited areas, “the climate question is at the
heart of all geopolitical issues and it is directly tied to questions of injustice and
inequality”7. Latour summarized these tensions as “New Climatic Regime”, a concept
that became a starting point for the Critical Zones exhibition project.
The term ‘critical zone’ (singular, in contrast to the plural of the exhibition title emphasizing
the concept’s manifoldness) is derived from Earth System Science8 where it denotes
Earth’s “thin biofilm”9—down into the soil until the bedrock and up into the
canopy and lower atmosphere—where Life10 subsists. But the critical zone is not only
characterized in spatial categories. It is foremost a dynamic field sui generis, where the effects of
“heterogeneous agencies mixed together in wildly different combinations”11 create their
own conditions of Life (e.g., plants’ photosynthesis of carbon to oxygen as condition for other lifeforms
that enable the existence of plants). This recursive, dynamic, and always incomplete interrelations have nothing
to do with the static and continuous order of the nature-culture dualism cemented by European modernity. In
fact, it may open a space for a kind of political action that considers manifold ways of how the interrelations
of actors may compose a common world. Such a field cannot be organized as two monolithic blocks of nature and
culture, where nature is somewhat treated or affected by human agency. “In that sense, the notion of the
critical zone is much less paralyzing for politics than that of the Anthropocene.”12
The deconstruction of the nature-culture dualism is well known in Latour’s philosophical work, most
notably in We have never been modern.13 Here the nature-culture dualism produces
“hybrids”14 that are transgressing its dichotomy as they are neither assignable to one
category nor to another (e.g., in vitro embryos or holes in the ozone layer). In a paradoxical twist, those
hybrids start to dissolve the modern constitution, albeit they are constantly being reintegrated in its dualist
structure. Artistic and scientific studies of hybrids are part of the repertoire of the works shown at Critical
Zones. In its uncovering of the far-reaching effects of the nature-culture dualism, We have never been
modern of course doesn’t stand alone. In our seminar sessions Latour emphasized the influence of,
among others, Alfred N. Whitehead’s critique of the “bifurcation of nature”15
into a nature perceivable by humans and a somewhat ‘true’ nature behind perceivable phenomena. Also
the work of Philippe Descola played a recurring role in our investigations with his “analysis of the modes
of relations between existing entities”.16 Such a relational approach emphasizes the
importance of spatial juxtapositions in an exhibition.
Relationality
Notions of relationality, heterogeneity, entanglement, and so forth are prevalent in studies that aim to
overcome the distinctions and hierarchies of Western modernity in favour of a more sustainable mode of living
together. They are already present in Gregory Bateson’s seminal 1972 work Steps to an Ecology of
Mind. Criticizing the Western dualism as human separation from and dominion over nature and as a root for
the approaching environmental crisis, Bateson emphasized the interconnectedness of all living beings. He also
brought forward the notion of an ‘impossible outside’ (to which I will get back below): “We
are not outside the ecology for which we plan – we are always and inevitably part of
it.”17 Albeit relationality as ontological or even ethical category is usually judged (a
priori) as something positive and preferable, I want to emphasize that not all interrelations are necessary
experienced as something desirable. The cultural theorist Lauren Berlant has pointed out „the pressures of
being in relation,”18 be it with human beings or with objects, as “a structural
awkwardness in the encounter between someone and anything.”19 But they also acknowledge the
necessity of the interaction with others as what drives one to experience the world. Notions of relationality
are, even in this sense a mode of care or “response-ability.”20
Here it should be noted that, in contrast to Bateson’s claim for interconnectedness, the critical zone is
not to be conflated with ‘holistic’ concepts of an “unified system […] where everything
is connected.”21 This, according to Latour, would suggest yet another universalism.
Rather, the relations remain fragile, always incomplete, always in a state of becoming. And yet, the notion of
critical zone is closely related to the concept of ‘Gaia’.22 This analogy of Earth system
science and Greek mythology was proposed by the geochemist James Lovelock in close collaboration with the
microbiologist Lynn Margulis.23 While Lovelock developed his take from geochemical analyses of
planetary atmospheres, Margulis worked with the other side of the magnifying scale, the microbial. Both met in
the conclusion that Earth is producing and regulating its own environmental conditions for Life. It is not only
this idea of autopoiesis and the roots of Lovelock’s work in cybernetics related to planetary
self-regulation which places the Gaia hypothesis a bit too close to the universalism of a “unified
system”. The mythological eponym also suggests a personification with close bounds to animism, presenting
Earth as single entity. Although in the Critical Zones exhibition catalogue Latour makes it clear that
“Gaia is not a big organism,”24 the exhibition maintained the concept and used it in
interrelation with ‘critical zone’, as wells as with the term ‘the terrestrial’ denoting
a new cosmology of the world we live in. The problems of the Gaia analogy were acknowledged by
Latour25—and in fact also by Margulis: “I prefer the idea that Earth is a network of
‘ecosystems’ over any personification of Mother Gaia.”26
Impossible Outside
According to Latour, universalisms such as the nature-culture dualism or “unified systems” suggest
“the hidden presence of an engineer at work who has devised the whole as a system of which we see only the
parts.”27 This would imply a somewhat external position from which one may act on the
Earth, or on ‘nature’. Such a position, taken by classical understandings of science (the
‘objective’ observer or experimenter) creates a relationship of maximum distance, both spatially and
(let’s use this loaded term) ethically: When there is a sphere to dwell in (‘culture’) which
can be separated from both the catastrophes we inflict on our planet (‘nature’) and human
responsibility, then what is there to worry about? But if we do not live on the globe of modernity but
inside the critical zone, a terrestrial interdependence in which we are intertwined with other entities
to create our environment, every harm inflicted is eventually self-inflicted. As Margulis put it in an interview
when describing the recycling processes of cyanobacteria: “If we would listen to them or watch them
[…], we would recognize that you can’t just throw things out—you never throw anything
out, it goes around. […] Now these bacteria have solved that issue, people
haven’t solved it at all. […] People are ruining their environment. These bacteria are producing an
environment that’s liveable.”28
Following this notion of an ‘impossible outside’, a central aspect of the exhibition was to find an
alternative to the representation of Earth as the famous Blue Marble, seen from a distant position in space. The
matter of representation is by no means trivial or restricted to the task of finding an imaginary for an art
exhibition, it has epistemological implications—a notion put forward prominently by the first thought
exhibition, Iconoclash. And as Latour pointed out with Reset Modernity!, the view on the Blue Marble is
“the place of nowhere,” as no one dwells in space. Like the eternal engineer’s gaze, this is a
cartographical view of Earth as globe, “unified, continuous, and homogeneous”29, where
every element has been placed, by science, in its assigned section of a ‘grid’. A grid in a literal
sense when cartography slices projected space into metrics, but also in a conceptional sense, e.g., in
taxonomical orders in biology. While this implies an external entity which organizes all other entities, the
critical zone “breaks down the cartographical view of planet
Earth.”30 Here there is no outside, but entities creating their own living
conditions—and thus the critical zone itself. “Gone is the idea of a disinterested distant
gaze.”31 This is far from a notion of space as container to be filled. It is
a space composed of manifold elements and connections, “tiny, fragile, and
provisional.”32
Compositionism
Composition, or “compositionism”, is a central concept not only in Latour’s writings
but also for his curatorial approach.33 It is related to his earlier notions of politics described as
an activity of “progressive composition of the common world.”34 As in the notion of
critical zone, there is no world to be found a priori, ready to be inhabited. A common world must be
continuously generated, “pieced together, element after element, through many travails and
conflicts”35. Here a key text is “An Attempt at a ‘Compositionist
Manifesto’” which, in its epigraph, Latour dedicated to “D.H.”. The nod to Donna Haraway
makes sense: Not only is, in this essay, her book When Species Meet credited as being “a
compositionist book if ever there was one”36. Her follow-up monograph Staying with the
Trouble also picks up, etymologically, on the notion of composition or ‘compost’:
“Critters—human and not—become-with each other, compose and decompose each other, in every
scale and register of time and stuff in sympoietic tangling […].”37 According to Latour,
the cosmological disruptions of the New Climatic Regime, the loss of the globe of globalization as a possible
vector of impossible exponential growth, are “forcing all of us—scientists, activists, and
politicians alike – to compose the common world from disjointed pieces instead of taking for granted that
the unity, continuity, agreement is already there.”38
It is this approach of building alliances from disparate parts that makes a thought exhibition, and especially
Critical Zones, an assemblage of scientific instruments (various measuring devices of different historical
contexts, underlining the historicity of the world perceived), participatory practices (workshops, performances,
field trips), and, of course, artworks. The heterogeneous artworks span from, e.g., Julian
Charrière’s installation “Future Fossil Spaces” (2017), columns made of layers of
lithium deposits and salt lumps, to a video installation by Barbara Marcel on a science and community project
related to the Amazon Tall Tower Observatory (“Ciné-Cipó – Cine-Liana”,
2019– 2020), to a section of German Romanticism paintings, curated by the art historian Joseph Koerner.
Although visitors could consult, similar to the predecessor exhibition Reset Modernity!, the guidance of a field
book in order to navigate through the exhibition, its parts are meant to be put together or into relation
subjectively. Here the museum becomes a testing ground, the exhibition a “scale model to test ideas”
for how to approach complexities such as climate change, “much too vast to be treated head
on.”39
This is where the notion ‘thought exhibition’ comes in: A useful tool in science to test a
hypothesis, or to make even new discoveries, with regards to objects too big, too complex, too remote, or too
impractical to treat directly, is the thought experiment. Within a sufficiently structured framework it
offers, albeit imaginary, an experimental approach towards potential solutions and virtualities. Although a
thought exhibition, or every exhibition, remains limited to its space – an institution, in the
case of Critical Zones located in Central Europe—and time—the ‘here’ of late capitalism
–, it is also a protected and experimental space where alternative futures, “a way to anticipate a
situation of which there is as yet no real instance,”40 can be safely explored. Here
imagination may become a projective capacity with which new worlds, beyond teleological concepts of growth, may
find their vectors.
And yet, also a thought exhibition is curated, that is, laid out in a way determined by an author. In this
sense, the paths it offers through the exhibition space are limited (even if there are multiple) and given or
suggested (albeit if they remain open to alternatives). But maybe it is in this tension, between the curatorial
concept and the visitor’s subjective experience and imagination, in this “complete uncertainty of
what the visitors will do in the end in the environment that you’ve imagined for them,”
41 where new worlds can evolve. Latour repeatedly emphasized that Critical Zones, or any of his
thought exhibitions, is not meant to somewhat illustrate ideas written somewhere else, but to offer a
spatial configuration for exploring, testing, or modifying ideas. As he pointed out in a conversation with Hans
Ulrich Obrist on the connection between his philosophy and exhibitions: “For me there is absolutely no
difference between doing an exhibition, writing a piece of philosophy or doing fieldwork with ethnographic
methods, or writing a play. [...] So, it’s not a migration of concept—concept is a medium,
among others, and they don’t migrate, they resonate with each other. [I]t’s first of all a
space—and the space is the concept.” 42
Spatio-aesthetics
The spatial arrangement invites the visitor to take different perspectives—figuratively and literally,
that is, spatially – on how things and actors interact with each other. This is no trivial notion of space
(as a physical container) and even goes beyond the relational approach mentioned above. The urban researcher
Mustafa Dikeç emphasizes the spatial and aesthetic conditions of political or activist intervention:
“Space not only gives form to and orders how this world appears, but also allows distinctive gatherings of
beings—things and people—that establish relationality and open new spaces […]. Thinking
politics spatially is both figurative, in the sense that it evokes spatial forms, and imagin tive, which allows
for the possibility of reordering things, […] established orders and systems of representation.”
43
It is worth pointing out the parallels between Dikeç’s interdependence of space, politics, and
aesthetics (“gatherings of beings – things and people”, “altering established orders of
representation”) and Latour’s “political ecology” 44, particularly the
thought exhibitions. When “spatialisation is fundamental to constructing, apprehending and projecting
worlds and entering into relation with them” 45, then Critical Zones offers a valuable testing
ground for how we may live response-able in a common world.
Dikeç understands aesthetics in the broad sense of aesthesis, as in perception by the
senses or the bodily (spatial) experience. 46 Making sense of the world is an embodied
phenomenological activity. If we couple this to Latour’s equally broad notion of aesthetics,
“defined as what renders one sensitive to the existence of other ways of life” 47, we can
underline the spatial mode of experimenting with alternative worlds. As mentioned above, here we need new forms
of representations that help us to aesthetically conceive this shift. New forms offering an alternative to the
iconic Blue Marble, to sufficiently represent the fragile and entangled biofilm or critical zone. “Changes
in cosmology cannot be registered without changes in representation.”48 Artistic aesthetic
expressions can have this capacity to render us sensitive to alternative worlds and their discontinuities to
past and present.
Cosmogram
Here the notion of ‘cosmogram’ comes into play. During the January 2019 session of the Critical
Zone Study Group, we had the privilege to host a lecture by the art and science historian John Tresch. His
concept of cosmogram relates to objects, architectural forms, or practices which bring a given cosmology of a
certain group of people at a certain point in time – a rather abstract set of shared beliefs constituting
a worldview—into the aesthetically concrete. 49 Based on Latour’s assumption that we, in
our attempt to orient ourselves in the New Climatic Regime, are in need of a new cosmology succeeding the globe
of modernity—a reorientation he compared to the Galilean paradigm shifts of the 17 th century
in science and the social order—, the cosmogram concept helped us to frame such kind of artistic
representation. As example for a well-known cosmogram (at least in Jewish and Christian mythologies), Tresch
refers to the Tabernacle of Moses. Here a religious-based worldview is precisely described as a model for a
spatial or architectural formation representing the godly regime, where the elements of the given cosmology find
their assigned place and relations to each other. Despite the example of the rather dogmatic Tabernacle, a
cosmogram is not necessary static. It may provide “the basis for new interpretations and action: social
relations, relations with other cultures, with natural entities, with animals,
plants.”50 Thus, it is important to stress the projective capacity Tresch assigns to
cosmograms, as they can enable a (note the Latourian term) “redescription, in the conditional or
future tense: not the world as it is but the world as it could be.” 51 As in Critical
Zones’ description of this shift and its redescription of the world towards a new common ground,
“cosmograms often guide [such a] recreation and restabilization of the world.”52
Although one could make an argument for understanding the Critical Zones exhibition itself as a
cosmogram—and there have been undertakings to use this concept as a curatorial approach for art
exhibitions53—, here I want to reserve this term for one of Critical Zone’s key artworks,
Sarah Sze’s ‘Flash Point (Timekeeper)’, a 2018 iteration of her ‘Timekeeper’
series (fig. 2).54 The Timekeeper installations investigate, among other materialities, the
dichotomies of digital societies, i.e., materiality–digitality or
spatiality–virtuality.55 They consist of various everyday objects and digital images (mostly
found online) projected and printed on paper sheets in various sizes, most of which are mounted on fragile
wooden frames. Installed in a separated and scarcely illuminated area of the ZKM ground floor, the mounted
prints of ‘Flash Point (Timekeeper)’ were illuminated by projected images and complemented by
projections spinning along the walls beyond the installation. Although it was difficult to say where
“beyond the installation” actually was, as its space-encompassing projections as well as materials
of the installation scattered on the floor made its spatial boundaries blurry.
All Timekeeper works evoke such an “immersive” effect, “like a series of experiments which
envelop the surrounding architecture”. 56 It is this uncertainty of knowing where the
installation ends and where one, as a viewer, enters its perimeter that constitutes an important aspect of the
installation’s function as a cosmogram representing the critical zone. Although I want to use the term
“function” carefully here. As Martin Guinard has pointed out while discussing with me the curatorial
approach of Critical Zones, “illustration is the enemy” 57: It is not about staging
concepts developed in Latour’s writings in the exhibition space, but to use the space as well as the
objects and actors assembled there to actively test ideas. Artworks are not somewhat degraded to serve as
illustrations of concepts developed by someone else. Martin expanded on this in his recently published obituary
for Latour: “Criticisms understandably arise when philosophers curate exhibitions and use artworks merely
to illustrate ideas. But in fact, we took a very different approach, which was to imagine an encounter between
artists’ works and his ideas, each of which followed different trajectories.” 58 Such an
encounter took place, on several occasions, between Latour and Sarah Sze. Referring to Latour’s interest
in the Timekeeper series with regard to Critical Zones, Hans Ulrich Obrist emphasized the viewers’
experience of manifoldness when approaching the installation, as well as the uncertainty of their own position
when investigating it more closely: They find themselves always inside the installation, sometimes
partly enclosed by its material, sometimes serving as a temporary, embodied screen when crossed by a
projection.59 In his comment, Obrist referred to ‘Twice Twilight’ (2020), the Timekeeper
iteration installed at Sarah Sze’s solo exhibition at the Fondation Cartier in Paris,60 which
is a more extensive installation compared to the work shown at ZKM. The immersive experience of the visitor is
here also amplified by the building it inhibited: The façade of the Foundation Cartier, designed by the
architect Jean Nouvel, is mostly made of glass and steel. Surrounded by trees and garden sections partly
encompassed by glass walls, it is at times challenging to make out out the inside and outside of the building.
This effect was amplified by the Timekeeper projections penetrating the glass façade. Such architectural
conditions are quite different from the spatially separated Timekeeper, located within the massive walls of the
ZKM building, a former ammunition factory. Reflecting on her exchange with Latour, Sze compares the
piece’s fragile structure and porous boundaries to the critical zone, that is “the world as very
thin, very fragile membrane of life.”61
At an artist talk between Latour and Sze on the occasion of her Foundation Cartier exhibition, which
started as a tour outside of the building, Latour right away pointed out the viewer experience of an undefined
spatial position: “One of the characters of your work is that the distance between the inside and the
outside is put into question.”62 During the tour he continued to draw a connection between
‘Twice Twilight’ and the impossibility of an outside position in the critical zone, as in both cases
“you never know when you are in and when you are out. […] The visitors are asked to subvert his or
her idea of what the Earth is like. Because there is no outside, really.” The, as Martin Guinard put it,
“different trajectories” of the artwork and the Critical Zones curatorial concept become intertwined
as a dialogue between the artist and the philosopher. While Latour went so far, albeit jokingly, to rename the
piece ‘Critical Zone’ – “because the bricolage, scaffoldings and the fragility, yet the
strength and multiplicity, is exactly what lifeforms have done in the critical zone” –, Sze pointed
out “one of the important things about the work, [the] fragility between what is an object, what is an
image, and what is life”. 63 This conceptional convergence is not a somewhat affirmative
agreement by an artist to a philosopher’s statement, but reflects an inherent characteristic of the
Timekeeper works. In fact, a comparison between the fragility of Life and Timekeeper was pointed out before by
the art historian Hal Foster: “A philosophy of life might be intimated here, one that cuts across biology
and technology, life seen as a system that struggles with flux.”64
By describing Sarah Sze’s work method as “compositional
principle”—carefully assembled of disparate pieces, always experienced anew by the changing
position of an immersed viewer—Latour acknowledged the Timekeeper works as terrestrial cosmograms: Through
the spatio-aesthetic encounter of the installation, “viewers can escape the dichotomy between seeing
inside-out or outside-in, as if they were caught in a vortex. They become ‘composers of space’ in
their own right”.65
– In memory of B.L. & P.W. –
Acknowledgements
This research has received funding from the European Union’s Horizon 2020 program under the Marie
Skłodowska-Curie Grant Agreement N° 101028379.
References
1 Bruno Latour, Down to Earth. Politics in the New Climate Regime, Cambridge, Polity
Press, 2018, 5.
2 Jan Zalasiewicz et al., “The technofossil record of humans,” The Anthropocene
Review 1/1, 2014, p.34–43, 40. For a widely discussed ‘start date’ of the
Anthropocene as potential new geological epoch, stretching from 1450 with the rise of colonization and
subsequent capitalism (thus designated as ‘Capitalocene’ by Jason W. Moore) to the post-WWII
‘Great Acceleration’ (Alvin Toffler) or ‘Sixth Extinction’ (Elizabeth Kolbert), see
Adele E. Clarke, “Introducing Making Kin not Population,” in Making Kin not
Population, ed. Adele E. Clarke and Donna Haraway, Chicago, Prickly Paradigm Press, 2018, p.1–39.
Latour designated the cosmological paradigm shift towards modernity to 1610 and the Galilean scientific
revolutions; cf. Bruno Latour, “Seven objections against landing on Earth,” in: Critical
Zones. The Science and Politics of Landing on Earth, ed. Bruno Latour and Peter Weibel, Cambridge, MA,
MIT Press, 2020, 12–19, 15.
3 Reto Knutti, “Closing the Knowledge-Action Gap in Climate Change,” One
Earth 1/1, 2019, 21–23.
4 Bruno Latour, “Thought Exhibitions”, lecture at the Museum of Contemporary Art
Zagreb, September 23, 2017; http://modesofexistence.org/what-is-a-gedankenausstellung/#introduction.
5 Bruno Latour, Nous n’avons jamais été modernes. Essai
d’anthropologie symétrique, Paris, La Découverte, 1991.
6 Stephen Jay Gould, “Ladders and Cones,” in Hidden Histories of Science, ed.
Robert B. Silvers, New York: New York Review Books, 1995, 37–67.
7 Bruno Latour, Down to Earth, 2.
8 Earth System Science “builds upon the traditional disciplines [geology, biochemistry,
etc.] but promises to provide a deeper understanding of the interactions that bind the Earth’s
components in a unified, dynamical system”, Earth System Sciences Committee NASA Advisory Council (ed.),
Earth System Science. A Program for Global Change, Washington: NASA, 1988, 15.
9 Alexandra Arènes, Bruno Latour, Jérôme Gaillardet, “Giving depth to
the surface: An exercise in the Gaiagraphy of critical zones,” The Anthropocene Review 5/2,
2018, 20–135, 121.
10 “Life” with capital ‘L’ denotes, in Latour’s writings, life in
its multitude of interrelations, including “animals, plants, bacteria, […] atmosphere, soil,
rocks, seas, clouds, minerals, continents”, Bruno Latour, “Seven objections against landing on
Earth,” 18.
11 Bruno Latour, “Some advantages of the notion of ‘Critical Zone’ for
Geopolitics,” Procedia Earth and Planetary Science 10, 2014, 3–6, 4
12 Ibid.; cf. Haraway’s notion of ‘Chthulucene’, which prefers modes of living
together to rather fatalistic implications of the Anthropocene. Staying with the Trouble. Making Kin in
the Chthulucene, Durham, NC, Duke University Press, 2016.
13 Bruno Latour, Nous n’avons jamais été modernes.
14 Ibid.; cf. Haraway’s “cyborgs, hybrids, mosaics, chimeras”. Haraway, “A
Cyborg Manifesto. Science, Technology and socialist-feminism in the latetwentieth century [1985],” in
The Cyber-cultures Reader, ed. David Bell, Barbara M. Kennedy, London and New York,
Routledge, 2001, 291–324, 313.
15 Alfred N. Whitehead, The Concept of Nature, Cambridge, Cambridge University Press,
1920, 30.
16 Phillippe Descola, Beyond Nature and Culture, Chicago and London, The University of
Chicago Press, 2013 [2005], s.p.
17 Gregory Bateson, “Ecology and Flexibility in Urban Civilization [1970],” in
Steps to an Ecology of Mind: Collected Essays in Anthropology, Psychiatry, Evolution, and Epistemology,
Northvale, NJ and London, Jason Aronson 1987, 499–511, 510.
18 Laurent Berlant, On the Inconvenience of Other People (Durham and London: Duke
University Press, 2022, 11.
19 Ibid., 9.
20 Donna Haraway, Staying with the Trouble.
21 Bruno Latour, “Some advantages of the notion of ‘Critical Zone’ for
Geopolitics,” 5.
22 Bruno Latour, Face à Gaïa. Huit conférences sur le nouveau régime
climatique, Paris, La Découverte, 2015; cf. Bruno Latour and Timothy M. Lenton, “Extending
the domain of freedom, or why Gaia is so hard to understand,” Critical Inquiry 45/3, 2019,
659–680.
23 Bruce Clark and Sébastien Dutreuil (eds.), Writing Gaia: The Scientific
Correspondence of James Lovelock and Lynn Margulis, Cambridge, Cambridge University Press, 2022.
24 Bruno Latour, “Seven objections against landing on Earth,” 18.
25 In a talk with Sarah Sze in October 2020 he remarked that, albeit both critical zone and Gaia
are scientific concepts, that “Gaia always gets people worried” due to its mythological or
vitalist suggestions. Artist talk “Sarah Sze et Bruno Latour: une balade-discussion dans
l’exposition ‘De nuit en jour’ – Fondation Cartier pour l’art
contemporain.” Livestream October 19, 2020; archived: https://www.youtube.com/watch?v=JPQijl921f0.
26 Lynn Margulis, Symbiotic Planet. A New Look at Evolution, New York, Basic Books, 1998,
106.
27 Bruno Latour, “Some advantages of the notion of ‘Critical Zone’ for
Geopolitics,” 5.
28 Lynn Margulis in a 1998 interview, in: “Symbiotic Earth. How Lynn Margulis rocked the
boat and started a scientific revolution”, documentary film by John Feldman, USA, 2017.
29 Bruno Latour, “Seven objections against landing on Earth,” 14. Cf. the
documentation of Latour’s performance lecture “Inside”, in: Frédérique
AïtTouati and Bruno Latour, Trilogie Terrestre: Inside suivi de Moving Earths, Montreuil,
Éditions B42, 2022.
30 Bruno Latour, “Seven objections against landing on Earth,” 14.
31 Bruno Latour, “Some advantages of the notion of ‘Critical Zones’ for
Geopolitics,” 5.
32 Bruno Latour, “Seven objections against landing on Earth,” 14.
33 I am indebted to Martin Guinard for pointing me to this.
34 Bruno Latour, Politics of Nature. How to Bring the Sciences into Democracy ,
Cambridge, MA, Harvard University Press, 2004, 53.
35 Bruno Latour, “Some advantages of the notion of ‘Critical Zones’ for
Geopolitics,” 3.
36 Bruno Latour, “An Attempt at a ‘Compositionist Manifesto’,” New
Literary History 41, 2010: p.471–490, endnote 40.
37 Donna Haraway, Staying with the Trouble, 97.
38 Bruno Latour, “An Attempt at a ‘Compositionist Manifesto’,” 485.
39 Bruno Latour, “Seven objections against landing on Earth,” 18.
40 Bruno Latour, “Let’s touch base, in: Reset Modernity!,” ed. Bruno
Latour, Cambridge, MA and London, MIT Press, 2016, p.11–23, 22. Cf. Bruno Latour, “Seven
objections against landing on Earth,” 18f.
41 Bruno Latour, “Thought Exhibitions”, lecture at the Museum of Contemporary
Art Zagreb.
43 Mustafa Dikeç, Space, Politics and Aesthetics, Edinburgh, Edinburgh University
Press, 2015, 1ff.
44 Bruno Latour, Politics of Nature.
45 Mustafa Dikeç, Space, Politics and Aesthetics, 4.
46 Mustafa Dikeç, Space, Politics and Aesthetics, 1.
47 Bruno Latour, “Seven objections against landing on Earth,” 19.
48 Ibid.
49 Based on David Damrosch’s cosmogram approach in religious studies. John Tresch,
“Cosmogram,” in Cosmograms, ed. Melik Ohanian and Jean-Christophe Royoux, New York, Lukas
& Sternberg, 2005, .67–76, 67.
50 John Tresch, “Cosmogram,” 69.
51 Ibid., 75 (emphasis in the original).
52 Ibid., 74.
53 Ohanian and Royoux (eds.), Cosmograms. Joshua Simon proposes the notion of cosmogram
“to look at the art exhibition as a model of the world as it appears to itself.” Simon, “The
Exhibition as Cosmogram,” Parse 13/2, 2021;
https://parsejournal.com/article/the-exhibition-as-cosmogram/.
54 Bettina Korintenberg, one of the co-curators, confirmed my hypothesis of the key function of
Sze’s installation for this show, as well as its position as a cosmogram in Bruno’s and John
Tresch’s sense (email correspondence between Korintenberg and the author, October 10 and 11, 2022).
55 Sarah Sze in conversation with Hans Ulrich Obrist, in: The Hero Winter Annual, 2020,
190–203, 193.
56 “Introduction,” in Sarah Sze: Timekeeper, ed. Rose Art Museum, New York,
Gregory R. Miller & Co., 2017, 4.
57 Martin Guinard in a conversation with the author during the Critical Zones Study Group session
in May 2019.
58 Martin Guinard, “Homage to Bruno Latour,” e-flux Journal 131, 2022,
https://www.e- flux.com/journal/131/502967/homage-to-bruno-latour/.
59 Sarah Sze in conversation with Hans Ulrich Obrist, 199.
60 Sarah Sze, “De nuit en jour”, Fondation Cartier pour l’art contemporain,
Paris, October 24, 2020 – May 30, 2021.
61 Sarah Sze in conversation with Hans Ulrich Obrist, 200.
62 “Sarah Sze et Bruno Latour: une balade-discussion dans l’exposition ‘De nuit
en jour’.”
63 Ibid.
64 Hal Foster, “The art of teetering,” in Sarah Sze: Timekeeper, ed. Rose Art
Museum, New York, Gregory R. Miller & Co., 2017, p.103–201, 197.
65 Bruno Latour, “The verifiable Image of the World?,” contribution to the exhibition
catalogue Sarah Sze, De nuit en jour / Night into Day, Paris: Publication Fondation Cartier pour
l’art contemporain, 2020, here quoted from the manuscript (transl. Lucas Faugère); http://www.bruno-latour.fr/sites/default/files/165-SARAH-SZE-PARIS_GB.pdf.
2.
Bibliography
“Introduction,” in Sarah Sze: Timekeeper, ed. Rose Art Museum, New York, Gregory R.
Miller & Co., 2017.
“Symbiotic Earth. How Lynn Margulis rocked the boat and started a scientific revolution”,
documentary film by John Feldman, USA, 2017.
Frédérique Aït-Touati, Bruno Latour, Trilogie Terrestre: Inside suivi de Moving
Earths, Montreuil: Éditions B42, 2022. Arènes, Alexandra, Bruno Latour,
Jérôme Gaillardet, “Giving depth to the surface: An exercise in the Gaiagraphy of critical
zones,” The Anthropocene Review 5/2, 2018:120–135.
Gregory Bateson, “Ecology and Flexibility in Urban Civilization [1970],” in Steps to an
Ecology of Mind: Collected Essays in Anthropology, Psychiatry, Evolution, and Epistemology, Northvale,
NJ and London: Jason Aronson 1987, 499–511.
Laurent Berlant, On the Inconvenience of Other People, Durham and London, Duke University Press,
2022.
Bruce Clark, Sébastien Dutreuil (eds.), Writing Gaia: The Scientific Correspondence of James
Lovelock and Lynn Margulis, Cambridge, Cambridge University Press, 2022.
Adele E. Clarke, “Introducing Making Kin not Population,” in Making Kin not
Population, ed. Adele E. Clarke and Donna Haraway, Chicago, Prickly Paradigm Press, 2018, 1–39.
Phillippe Descola, Beyond Nature and Culture, Chicago and London, The University of Chicago Press,
2013 [2005].
Mustafa Dikeç, Space, Politics and Aesthetics, Edinburgh, Edinburgh University Press, 2015.
Earth System Sciences Committee NASA Advisory Council (ed.), Earth System Science. A Program for Global
Change, Washington, NASA, 1988.
Hal Foster, “The art of teetering,” in Sarah Sze: Timekeeper, ed. Rose Art Museum,
New York, G. R. Miller, 2017, 103–201.
Stephen J. Gould, “Ladders and Cones,” in Hidden Histories of Science, ed. Robert B.
Silvers, New York, New York Review Books, 1995, 37–67.
Donna Haraway, “A Cyborg Manifesto. Science, Technology and socialist-feminism in the late-twentieth
century [1985],” in The Cybercultures Reader, ed. David Bell and Barbara M. Kennedy London and
New York, Routledge, 2001, 291–324.
Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham and London, Duke
University Press, 2016.
Reto Knutti, “Closing the Knowledge-Action Gap in Climate Change,” One Earth 1/1, 2019,
21–23.
Bruno Latour, “An Attempt at a ‘Compositionist Manifesto’,”
New Literary History 41, 2010, 471–490.
Bruno Latour, “Let’s touch base, in: Reset Modernity!,” ed. Bruno Latour,
Cambridge, MA and London, MIT Press, 2016, 11–23.
Bruno Latour, “Some advantages of the notion of ‘Critical Zone’ for
Geopolitics,” Procedia Earth and Planetary Science 10, 2014: 3–6.
Bruno Latour, “The verifiable Image of the World?,” contribution to the exhibition catalogue
Sarah Sze, De nuit en jour / Night into Day, Paris, Publication Fondation Cartier pour l’art
contemporain, 2020; manuscript (transl. Lucas Faugère):
www.brunolatour.fr/sites/default/files/165-SARAH-SZE-PARIS_GB.pdf. 2. Latour, Bruno, “Thought
Exhibitions”, lecture at the Museum of Contemporary Art Zagreb, September 23, 2017; http://modesofexistence.org/what-is-a-gedankenausstellung/#introduction.
Bruno Latour, Sarah Sze, artist talk “Sarah Sze et Bruno Latour: une balade-discussion dans
l’exposition ‘De nuit en jour’ – Fondation Cartier pour l’art
contemporain”, Livestream October 19, 2020; archived: www.youtube.com/watch?v=JPQijl921f0.
Bruno Latour, Timothy M. Lenton, “Extending the domain of freedom, or why Gaia is so hard to
understand,” Critical Inquiry 45/3, 2019: 659–680.
Bruno Latour, Down to Earth. Politics in the New Climate Regime, Cambridge, Polity Press,
2018.
Bruno Latour, Face à Gaïa. Huit conférences sur le nouveau régime
climatique, Paris, La Découverte, 2015.
Bruno Latour, Nous n’avons jamais été modernes. Essai d’anthropologie
symétrique, Paris, La Découverte, 1991. Latour, Bruno, Politics of Nature. How to
Bring the Sciences into Democracy, Cambridge, MA, Harvard University Press, 2004.
Bruno Latour, “Seven objections against landing on Earth,” in: Critical Zones. The
Science and Politics of Landing on Earth, ed. Bruno Latour and Peter Weibel, Cambridge, MA, MIT Press,
2020, 12–19.
Lynn Margulis, Symbiotic Planet. A New Look at Evolution, New York, Basic Books, 1998.
Joshua Simon, “The Exhibition as Cosmogram,” Parse 13/2, 2021,
https://parsejournal.com/article/the-exhibition-as-cosmogram/.
Sarah Sze, in conversation with Hans Ulrich Obrist, in: The Hero Winter Annual, 2020, 190–203.
John Tresch, “Cosmogram,” in Cosmograms, ed. Melik Ohanian, Jean-Christophe Royoux, New
York, Lukas & Sternberg, 2005, p.67–76.
Alfred N. Whitehead, The Concept of Nature, Cambridge, Cambridge University Press, 1920.
Jan Zalasiewicz et al., “The technofossil record of humans,” The Anthropocene
Review 1/1, 2014, 34–43
Author Biography
Daniel Irrgang is a postdoctoral researcher (Marie Skłodowska-Curie fellow) at the
Centre Art as Forum (Dept. of Arts and Cultural Studies), University of Copenhagen, and associated researcher
at the Weizenbaum Institute, Berlin. His current research focusses on the concept of ‘thought
exhibition’ (Latour/Weibel). He holds a PhD in media studies on diagrammatics and expanded mind theories
and is author and editor of numerous books on the history and theory of media, culture, and the arts.
Curation as Research-Creation: Speculating on the Future of Art and Technology Festivals
This paper explores a renewed approach to curation as research-creation (CRC) through its practical application
in the annual art and technology festival. CRC envisions a shift in curation from a care for objects to a care
for the emerging social relations of the curatorial project in a shared quest of meaning making.
We set out with outlining the features of CRC as interdisciplinary, concerned with programmatic boundary
objects, and centered around the unfolding event trajectory – the forms and methods that facilitate
affective encounters. Following we outline how this approach to curation unfolds in practice through the case
study of the Fest-Forward workshop series that speculates on the future of art and technology
festivals. Concluding we summarize how this workshop series showcases the potential of CRC’s shift of
attention from a mere presentation of artworks towards the facilitation of interdisciplinary and cross-cultural
encounters that enroll artists, curators, and audiences.
After two and a half hours of speculating on the future of festivals, one workshop participant remarked:
“Inherent in these art projects is that they already tackle all these big questions. I mostly see the
lack in the structures that we as festival makers provide to facilitate the spaces in which these artworks reach
their full potential.”
This is the central issue that our workshop series titled Fest-Forward is addressing since August
2022. The workshops take place in a variety of different formats and localities around the world. Past
activations include Canada, the Netherlands, and Japan, and future renditions aim to include wider geographies.
Inspired by a reimagined approach to curation as research-creation firmly rooted at the intersection of art,
technology, and governance, these workshops explore the transforming role of the art and technology festival as
a site of joint-meaning making between artists, researchers, industry professionals, policymakers, and
audiences. In other words, by reenvisioning how the festival comes into being and shapes the encounters of
people, we seek to address how art can reach its full potential.
The following paragraphs set out with an introduction to the theoretical reimagination of Curation as
Research-Creation (CRC), which presents the base considerations for this project. Following, the paper
introduces the methodological approach towards these workshops, which are rooted in speculative design and
liberating structures. We then venture into describing the case study of the inaugural Fest-Forward:
Imagining Future Festivals workshop, which took place in August 2022 as part of the 23rd edition of the
MUTEK Festival and Forum. The concluding paragraphs summarize key theoretical and methodological findings of
this workshop in transforming the festival as a social actor.
Curation as Research-Creation
Curation as Research-Creation (CRC) presents an overall shift from understanding curation as merely an act of
‘putting things together ’towards the curatorial project itself being a site of inquiry and meaning
making. ¹ CRC is thus inherently concerned with, as Loveless puts it, “not only what methods
offer at the level of investigation [...], but also what form might best fit the content of the research at the
level of publication.” ² This reinterpretation of curation as not only form but method, then embraces
the key idea of research-creation as a continuous thinking-making process. ³ Emerging out of the
intersection of critical curatorial studies, science and technology studies, and artistic and curatorial
practices in the art, science, and technology realm, CRC is specifically attuned to addressing the pressing
sociotechnical questions of our time.
Curation is interdisciplinary
CRC ventures away from the centrality of the curator or the artist in the unfolding of the curatorial project.
In contrast, curation becomes an interdisciplinary endeavor, equally acknowledging the extensive involvement of
people and labor in preparation, as well as the centrality of audience participation in co-creation. Galison, in
an investigation of inter-disciplinary encounters in science and technology, framed this sort of space as a
trading zone. ⁴ Similarly, Dekker argues that the emergence of digital art in online, offline, and hybrid
spaces marks a shift from the paradigm of collecting and presenting art towards networked co-curation. ⁵
This networked co-curation fundamentally questions traditional museological values, the participation of
publics, and the field of art history. ⁶ It shifts, as Truman points out, the attention from who produces
knowledge towards how knowledge is produced in joint inquiry, which underlines the notion of curatorial practice
as an act of caring for social relations. ⁷
The curatorial project then becomes an interdisciplinary trading zone of networked co-curation. Defining the
festival as a trading zone of co-curation shifts attention from merely the content, which is the artistic and
discursive program, towards the methods of exchange that lead people from different professional, disciplinary,
intersectional, and cultural backgrounds to come together in joint exploration of ideas at the intersection of
art and technology. In summary, CRC presents a shift of focus, as Manning and Massumi suggest, to “more
than programmation but catalytic event unfolding”, in pursuit of a shared interdisciplinary inquiry.
⁸
Program as boundary object
Where does that leave the program—the content—then? While content is still key, its role is
redefined from mere display of artistic works towards taking the role of presenting boundary objects. Boundary
objects, as put forward by Star & Griesemer, are concrete or abstract objects that structure
interdisciplinary inquiry. ⁹ In Bruno Latour’s words the thing-object as a shared matter of concern
is what brings people together. ¹⁰ They are general enough in a way that actants from different
social worlds can gather around them in joint exploration, while simultaneously acknowledging the specificity
they potentially carry across disciplines, professional fields, intersectional and cross-cultural environments.
¹¹
Boundary objects thus carry the power to assemble people not because there is agreement but exactly because
opinions are divided and varied. ¹² Rethinking the festival program as establishing certain boundary
objects thus frames content beyond mere aesthetic display as an invitation to or a prompt for participants to
jointly explore throughout the unfolding of the event. In other words, program as boundary object is better
attuned towards mobilizing the powerful meanings and critiques that are commonly explored by the most
forward-thinking artists in their fields.
The centrality of trajectory: forms and methods
The way the event is conditioned in terms of forms and methods foster exchange in meaning-making. The event
trajectory thus becomes central. Trajectory refers to the temporally emergent configuration and reconfiguration
of forms and methods of engagement in the unfolding of the curatorial event. Forms refer to the combination of
space (physical, virtual, hybrid, concert hall, club, gallery, auditorium etc.), and format (performance,
exhibition, conference, workshop, roundtable etc.). Methods, on the other hand, present the modes of engagement
within these interdisciplinary encounters. ¹³
Forms and methods are inherently entangled. Certain methods ask for certain event spaces and engagement
formats. Simultaneously, certain event spaces and engagement formats urge for certain methods to be employed.
¹⁴ It is important to note, however, that these combinations stem from very specific normatively
laden histories. ¹⁵ The traditional Western concert hall for example not only calls for specific
engagement formats, such as chamber music, but also encourages specific ways of performing and listening to
music. Addressing these normatively laden spaces by inhabiting them in unconventional ways, such as shifting
positionalities of audiences and human/non-human performers thus actively encourages decolonizing efforts of
these traditional spaces of artistic engagement. As such, experimentation with forms and methods transcending
physical and virtual spaces, and a variety of engagement formats are central to CRC.
Transforming curation through research-creation
CRC then acknowledges the emergent nature and reiterative processes of meaning-making throughout the entirety
of the curatorial project. ¹⁶ Especially within the frame of the annual festival curation, which
neither has a fixed beginning, nor concludes with the presentation of the festival edition. It is rather a
process that continuously unfolds, shapes, and reshapes before, during, and after the annual event. In addition,
CRC acknowledges the inherent interdisciplinarity of the curatorial project, in both preparation and
presentation. Curation brings together curators, organizing teams, communications experts, artists, producers,
researchers, and a wide range of external partners local and international, as well as their non-human
counterparts.
Beyond heightened attention towards the forms, methods, and interdisciplinarity of the curatorial project, CRC
acknowledges the fact that both thinking and making processes take place and intervene in the real-world. As
such, CRC makes the researcher-curator accountable for the need of embracing an ethical responsibility and
careful deliberation towards all forms and methods employed, spanning human and non-human subjects in the
curatorial project. ¹⁷
Reinterpreting the festival within CRC thus frames the event as a methodological approach towards investigating
the world, rather than a mere object of inquiry or an aesthetic display. The following paragraphs outline how
employing this alternative approach towards curation in the frame of speculative workshops as part of the MUTEK
festival unfolded in practice.
Fest-Forward: Imagining Future Festivals
In August 2022, the 23rd edition of the MUTEK festival took place over the course of 6 days in Montreal,
Canada. After two pandemic years, which saw hybrid activities, the festival returned to in-person events
including international artists and professionals visiting. On the opening day of the festival, we hosted a
speculative workshop titled Fest-Forward: Imagining Future Festivals as part of the professional day
time program, the MUTEK Forum.
Workshop design: interdisciplinarity and boundary object
The workshop invited an interdisciplinary and cross-cultural group of artists, curators, technologists,
policy-makers, and members of the festival audience to jointly investigate the transforming role of the festival
as a social actor in (post-)pandemic times. Employing methods of speculative design and liberating structures,
workshop participants were mobilized by the boundary object of an imaginary future festival.
Through three paradoxical, wicked questions the workshop prompted participants to speculate on how an imaginary
future festival might address questions of existing structures of oppression, rapid advances in science and
technology, and/or environmental and climate crisis. More than being presumptuous in believing that an art and
technology festival might solve these issues, the prompts were aimed at inspiring imagination if and if so, what
it might be an art and technology festival could do. As Dunne & Raby put it, employing “the idea of
possible futures and using them as tools to better understand the present and to discuss the kind of future
people want”. ¹⁸ In other words, how could festivals become sites of collective meaning-making
surrounding the pressing questions of our time by speculating about alternative futures?
In keeping with speculative design approaches participants were guided through the materialization of the
boundary object in the form of creating a speculative newspaper headline and lead paragraph outlining how an
imaginary future festival employs certain concrete tools or activities to address the wicked questions. This
concrete output situated the potential, imaginary future in the here and now.
Workshop design: trajectory
The workshop took place at the Hexagram spaces located on the 4th floor of the Pavillon des Sciences
Biologiques at Université du Québec à Montréal. The multifunctional space was laid out
with three table islands seating seven people each for a total of 21 participants. Each table island was
equipped with a whiteboard and crafting materials including article templates, pens, and post-it notes. The
workshop was documented via an ambisonic audio recorder placed in the center of the room, by the festival
photographer, and the researcher’s ethnographic notes.
Manifesting the boundary objects through the wicked questions and the concrete, material output the workshop
trajectory unfolded in three distinct, yet interlinked phases. The first 30 minutes were spent with an
introduction to the workshop and its methodology both conceptually and in practice. Split into three teams at
three table islands, group work was generally structured by the 1-2-Group method. At first participants were
prompted to consider posed questions individually for three minutes, followed by building pairs within teams to
continue discussion for another seven minutes. Concluding the group would get together to discuss initial
findings. The 1-2-Group method is specifically attuned to giving voices to people that might be silenced in
larger group discussions. ¹⁹ In addition, it presented an effective trust-building exercise,
especially in the paired-up phase. We first employed the 1-2-Group method as an ice-breaker during the
introduction phase of the workshop.
After sharing the prompts with participants, we employed two phases of 1-2-Group, each lasting 20 minutes.
During the first-round participants were to decide upon which wicked question they would address and brainstorm
on how they might address it. The second round was guided by trying to answer the six journalistic questions
(What? Who? When? How? Why? Where?) in preparation for making the newspaper article.
Following a short break, teams were then tasked with crafting the article. Participants were provided with
printed article templates, whiteboards, and digital equivalents to facilitate creation. Concluding, teams
presented their articles and thus their imaginary future festivals to the other groups, followed by an open
discussion on both resulting speculation and engagement methods.
Joint speculation on the future of festivals
All three groups appropriated the method, prompts and template very differently. The emergent imaginary
festival projects reflect the interdisciplinarity of the teams and the ways they appropriated the methods.
Group A’s interactions can be described as democratic, consensus-based, and self-organized. As a reaction
to the proposed time management, they abandoned the 1-2-Group structure during the second round. Working through
the third wicked question “How can future festivals deal with equity, diversity, and inclusivity while
continuing to profit from existing structures of domination and exploitation?”, this team’s output
explored how to reinvent the festival spatially to decolonize many of its premises.
Local, decentralized but nonetheless networked is how this team envisioned the future of festivals. The way
they proposed this could be achieved is through the invention and deployment of a multimedia campfire toolkit.
These toolkits could be distributed to a variety of locations, near and far, urban, and rural and serve as a
gathering place for communities to come together. The toolkit would be networked, solar powered, have
projection, speakers, live translation, the possibility to video call other chapters of this decentralized
festival taking place. Their output emphasized accessibility, the need for communion, and the necessity to
decentralize the traditionally urbanite festival. This focus on horizontality would enable, according to this
team, oral transmission of knowledges and serendipitous connections. Group A made a point to work through how
financially inaccessible urban festival-going and how unsustainable flying to be co-present in festival sites
can be, especially at scale. Their article started with “Finally a festival that acknowledges the past, to
honor and imagine the future!”
Group B followed the structure rigorously (even as they could hear the other teams abandoning the 1-2-Group
method). Participants also made use of all the tools available namely the whiteboard, physical and digital
templates. Their discussions were animated and energetic. The output they proposed was rhizomatic, complex with
an emphasis on emergent and horizontal organization. This team worked through the third wicked question:
“How can future festivals deal with equity, diversity, and inclusivity while continuing to profit from
existing structures of domination and exploitation?”
Group B’s output emphasized festivals' responsibility to redistribute both power and resources. They also
emphasized how systemic justice requires climate justice. If festivals act as platforms for artists, this team
emphasized the duties that come with such a role. Diligent archiving practices, redistributing resources as
education (legal, financial, and harm-reduction), and ensuring festival organizers reflect the community they
serve are all practices this team proposed to implement in their future festival. For this team, the institution
of the festival plays a key role—much like museums—in community, public life, shaping and sharing
knowledges. It is as such that festivals carry responsibilities and duties to serve the people they represent
and engage with.
Group C resisted the methodology and did not follow any of the instructions. Starting with the instruction to
choose a wicked question to work on, to the 1-2-Group time management schedule, to the templates; this team can
be described as having gone rogue. Professional backgrounds may have contributed to these group dynamics: most
of them are used to being in positions of leadership and the proposed method required them to let go and trust
the structure. The team struggled to focus on a singular wicked question which exacerbated disagreement within
the team.
This team’s interactions were marked by strong characters, misunderstanding, and stress. According to our
ethnographic notes, this team attempted to think of a future festival that would reduce its reliance on digital
technologies and tried to re-incorporate analog activities in their programs, such as bicycles. Either
satirically or ironically, their output was algorithmically generated; they used an application based on the
Artificial Intelligence (AI) language model GPT-3 to write their article. As they realized they were failing to
democratically organize, they turned to technology to flatten disagreements. Nonetheless, this team enacted a
certain degree of reflexivity as they titled their article “Festival curators fail to detech their
programs.” They thus acknowledged and illustrated the gap between their imagined festival and their
employed team praxis.
In the collective discussion following the teams ’presentation, the question of the place of the Art in
the festival was raised. Two poles emerged in collectively speculating on the future of festivals through the
wicked questions. On the one hand, participants decried the need to recenter art and music despite all the
broader social, technological, and environmental considerations.
One participant’s feedback captures this well: “I felt in the end that the idea of art content was
not touched upon, but there were some valuable ideas about reaching out to local and international communities
and bringing their voice into the programming.” On the other hand, a festival maker raised how for the
most part, art and artists are already engaging with these issues and the responsibility of institutions, such
as festivals, to create the necessary infrastructure to host and facilitate these broader societal shifts.
Curation as Research-Creation in practice
The Fest-Forward workshop series is a practical example of how attention towards curation as an
unfolding of trajectories of interdisciplinary and cross-cultural encounters shifts attention from mere content
towards how people come together in joint meaning-making. In other words, curation as caring for emergent social
relations throughout the curatorial project. The potential of this approach to facilitate these
interdisciplinary encounters is reflected by additional feedback we received from participants. As one
participant stated:
“I found the format of the workshop was engaging and efficient, and that it probed productive discussions
and group dynamics. I’ve found the process of writing the article and collectively designing the project
more interesting than the resulting paper itself.”
This underlines how the boundary object of the wicked questions materially represented by the speculative
article was effective in facilitating the groups ’encounter. Talking about the interdisciplinary and
cross-cultural aspect of the workshop format, another participant remarked:
“I found the exchange experience very rich in terms that we all came from different roles around
culture—from festival managers to music and art curators and curious people—and the different tasks
exposed each of us to speculate from different roles.”
In conclusion, the workshop was successful in raising questions about the transforming role of the festival as
both a site of joint meaning making and in taking responsibility as a social actor. For example, conceptualized
as a site of empowerment for artists and cultural actors, one group took the festival as a means of highlighting
more broadly the social difficulties of the cultural milieu to develop equitably. In a more local and
community-centered approach, one group presented the festival by associating it with the metaphor of the
campfire. Another participant remarked:
“My takeaway would be that future festivals will need to be collaborative, non-competitive and networked,
to work together to make sure their voices are heard in a future that will most likely still be dominated by
bigger players' capitalist interests.”
This notion is further underlined by the following feedback received:
“Attending the workshop reignited my passion for festival planning and broadened my perspective on
initiatives that me and my company can take to make events more inclusive and accessible.”
Conclusion: trajectories all the way
While the preceding paragraphs outline the micro-trajectories and their effectiveness during the unfolding
workshop, the workshop itself is situated in larger trajectories. Held on the first day of the festival the
workshop was aimed at inspiring participants to view the remaining festival activities, talks, installations,
performances, and encounters with renewed vigor.
As such, the workshop was strategically placed to interplay with the unfolding festival trajectories and thus
presented a perceived reconfiguration of forms and methods of engagement for participants. We tracked the
participants ’ reshaped perspectives via a festival diary, asking each of the participants individually
and at randomly selected times throughout the remainder of the festival to give feedback on how their
participation might have shifted their perspective of the festival in its unfolding.
The workshop is part of a series of activations, which under the premise of the MITACS-funded research project
Festival as Methodology implements CRC in practice as part of the annual festival edition of MUTEK and its
partner festivals in Argentina, Germany, Japan, Mexico, and Spain. Underlining the importance of trajectories
this initial workshop thus was not only concerned with the micro-trajectories of its unfolding but firmly
embedded within the unfolding macro trajectories of the 2022 festival edition and the 3-year research project.
While all of these unfold in different temporalities what unites them is an underlying concern for their
unfolding in terms of the forms they take, spaces they inhabit, and methods they employ. The Fest-Forward
workshop series thus is representative of Curation as Research-Creation as inherently concerned with the
ongoing configuration and reconfiguration of trajectories that facilitate the emergent social relations of
interdisciplinary encounters in joint meaning-making.
Acknowledgements
This research is part of the MITACS-funded research project Festival as Methodology.
References
1 Maurice Jones, “Curatorial Practice as Research-Creation: From aesthetic display to
interdisciplinary encounter", From aesthetic display to interdisciplinary encounter", Unpublished
Manuscript, 2021, 27.
2 Natalie Loveless, “Critical Conversations 2021: Natalie Loveless: How to Make Art at the
End of the World: Revisited", College of Arts and Science - University of Saskatchewan, February 11, 2021.
3 Chris Salter, “Chris Salter on Research-Creation”, Hexagram, Montreal,
2019, Hexagram website accessed April 23, 2022,
http://rec.hexagram.ca/index.php/episode-0/portrait-chris-salter,
4 Peter Galison, “Trading Zone: Coordinating Action and Belief (1998 abridgment).”
In The Science Studies Reader, ed. Mario Biagioli, London UK, Routledge, 1999, 137-160.
5 Annet Dekker, Curating Digital Art: From Presenting and Collecting Digital Art to Networked
Co-curation, Amsterdam: Valiz, 2021.
6 Annet Dekker, Curating Digital Art: From Presenting and Collecting Digital Art to
Networked Co-curation, Amsterdam: Valiz, 2021.
7 Sarah E. Truman, Feminist speculations and the practice of research-creation: Writing
pedagogies and intertextual affects, London UK, Routledge, 2021.
8 Erin Manning, Brian Massumi, Thought in the Act: Passages in the Ecology of
Experience, Minneapolis, University of Minnesota Press, 2014, 89.
9 Susan Leigh Star, James R. Griesemer, “Institutional Ecology, "Translations ’and
Boundary Objects: Amateurs and Professionals in Berkeley’s Museum of Vertebrate Zoology,”
Social Studies of Science 19 (3), 1989, 387–420.
10 Bruno Latour, “From Realpolitik to Dingpolitik – An Introduction to Making Things
Public”. in Making Things Public—Atmospheres of Democracy. ed. Bruno Latour and Peter
Weibel, Cambridge Massachusetts: MIT Press, 2005.
11 Susan Leigh Star, James R. Griesemer, “Institutional Ecology, "Translations and Boundary
Objects”, 387–420.
12 Bruno Latour, “From Realpolitik to Dingpolitik – An Introduction to Making Things
Public”.
13 Maurice Jones, “Curatorial Practice as Research-Creation”, 27.
14 Maurice Jones, “Curatorial Practice as Research-Creation”, 27.
15 Dylan Robinson, Hungry listening: Resonant theory for indigenous sound
studies, Minneapolis, University of Minnesota Press, 2020, 288.
16 Maurice Jones, “Curatorial Practice as Research-Creation”, 27.
17 Maurice Jones, “Curatorial Practice as Research-Creation”, 27.
18 Dunne, Anthony, Fiona Raby, Speculative Everything: Design, Fiction, and Social
Dreaming, Cambridge Massachusetts, MIT Press, 2013, 2.
19 Henri Lipmanowicz and Keith McCandless, The surprising power of liberating structures:
Simple rules to unleash a culture of innovation, Seattle, WA, Liberating Structures Press, 2013, 366.
Bibliography
Annet Dekker, Curating Digital Art: From Presenting and Collecting Digital Art to Networked
Co-Curation. Amsterdam, Valiz, 2021.
Anthony Dunne, Fiona Raby, Speculative Everything: Design, Fiction, and Social Dreaming. Cambridge,
Massachusetts, MIT Press, 2013.
Peter Galison, “Trading Zone: Coordinating Action and Belief (1998 Abridgment)”, In The
Science Studies Reader, edited by Mario Biagioli, 1999, 137–60, London, United Kingdom,
Routledge.
Maurice Jones, “Curatorial Practice as Research-Creation: From Aesthetic Display to Interdisciplinary
Encounter”, Un-published Manuscript, 2021.
Bruno Latour, Peter Weibel,
“From Realpolitik to Dingpolitik – An Introduction to Making Things Public”, In Making
Things Public: Atmospheres of Democracy, Cambridge, Massachusetts, MIT Press, 2005.
Henri Lipmanowicz, Keith McCandless, The Surprising Power of Liberating Structures: Simple Rules to
Unleash a Culture of Innovation, Seattle, WA, Liberating Structures Press, 2013.
Natalie Loveless, “Critical Conversations 2021: Natalie Loveless: How to Make Art at the End of the
World: Revisited”, College of Arts and Science - University of Saskatchewan, February 11, 2021.
Erin Manning, Brian Massumi, Thought in the Act: Passages in the Ecology of Experience, Minneapolis:
U of Minnesota Press, 2014.
Dylan Robinson, Hungry Listening: Resonant Theory for Indigenous Sound Studies, Minneapolis, U of
Minnesota Press, 2020.
Chris Salter, “Chris Salter on Research-Creation”, Rec Hexagram, April 2019,
http://rec.hexagram.ca/index.php/episode-0/portrait-chris-salter.
Susan Leigh Star, James R. Griesemer, “Institutional Ecology,"Translations" and Boundary Objects:
Amateurs and Professionals in Berkeley’s Museum of Vertebrate Zoology, 1989, 1907-39. ”Social
Studies of Science 19 (3), 387–420, https://doi.org/10.1177/030631289019003001.
Sarah E. Truman, Feminist Speculations and the Practice of Research-Creation: Writing Pedagogies and
Intertextual Affects, London, United Kingdom, Routledge, 2021.
Feeling One’s Way: In Search of a Symbiotic Vocabulary of the Virtual
Affiliation(s): 1 IRCAV & IRMÉCCEN, Sorbonne Nouvelle University,
Paris, France / 2 École des Arts Décoratifs–PSL (EnsadLab), Paris, France / 3 Univ.
Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, Grenoble, France / 4 Cerilac, Paris Cité
University, 5University Panthéon-Sorbonne olha.kobyn@gmail.com, matthieu.couteau@sorbonne-nouvelle.fr
remi.sagot@gmail.com, sophie.balcon@sorbonne-nouvelle.fr, francois.garnier@ensad.fr, remi.ronfard@ensad.fr,
guillaume.soulez@sorbonne-nouvelle.fr
Abstract
This article aims to present the collective research that has been made in the framework of the research
seminar "Vocabulary of the Virtual" organized by IRCAV (Institute of Research on Cinema and Audiovisual
Studies), Sorbonne Nouvelle and the “Spatial Media” group, EnsadLab. The main topic is to clarify
the notions that refer to the concept of the "Virtual" in order to define it through an interdisciplinary
approach according to different fields of science (aesthetics, philosophy, film theory, sound theory,
ergonomics, design, engineering, cognitive sciences, etc.). This article presents many different conceptual
tools such as cartography, mental maps, notional diagrams with several dimensions, etc., that have been
conceived over the last three years, to show how the reflection on the concept of the Virtual was first
established and constructed, and how it has been developed. The notion of symbiosis seems to be defined as a
structuring notion of the concept of the Virtual across the process of anchoring the levels of virtuality inside
technological devices and concrete sites, as well as inside the physical body of the VR users. The user’s
body serves as a catalyst for the concept of the Virtual which then becomes organic.
Keywords
Virtual Reality, Symbiosis, Aesthetics, Philosophy, Experience, Phenomenology, Design, Media Studies.
Introduction and theoretical objectives of the research seminar “Vocabulary of the Virtual”
This article aims to present the work that has been developed since 2020, during the research seminar
"Vocabulary of the Virtual" (IRCAV, Sorbonne Nouvelle /Spatial Media, EnsadLab). The main point was to clarify,
through a deliberately interdisciplinary approach (aesthetics, philosophy, film theory, sound theory,
ergonomics, design, engineering, cognitive sciences, etc.) the theoretical notions that constitute our
experience of the concept of the virtual from its concrete manifestations into the immersive devices and new
film and media forms resulting from virtual, mixed, and augmented reality technologies. This research, the
stages of which are reported here, has materialized through a certain number of experimental methodological
conceptual tools such as dictionaries, cartographies, mental maps, and notional diagrams with several
dimensions, proposed with the aim of grasping the concept of virtual from its relation to the adjacent
philosophical and theoretical concepts.
The initial problem of our research seminar comes from an observation of a lack of precision regarding the
vocabulary for analyzing new artistic and media forms emerging out of the use of new digital and cybernetic
technologies, the virtual, mixed, and augmented reality devices. This lack of vocabulary, from the first stage
of the analytical gesture, namely the description of the audiovisual structures of immersive spaces and their
interactions with the user or the immersant, required a deepening of the definitions and terms that
describe the aesthetic units and sets that make up the experience of virtual reality because, as Daniel Mestre
and Philippe Fuchs remind us, “virtual reality is, in essence, an ambiguous concept. It is difficult to
completely distinguish what the responsibility of technology is (virtual reality as a digital, mechanical,
electronic tools, etc.) and what the responsibility of experience is (virtual reality as a subjective
construction).” ¹
Thus the initial question, which has consisted in verifying the viability of the vocabulary of traditional film
analysis1 applied to VR contents, has permitted to forge a broader semantic field of study, focused on the
definition of the concepts involved in the construction of immersive levels of reality, in particular the
hybridization factor according to Milgram's continuum principle (between digital or cybernetics and psychical
reality, which includes, among others, Michel Serres ² concept of "hors-là" as well as the concept of
fiction, of "narrative presence" ³, etc.). In this regard, the very definition of the term "virtual
reality” poses a problem from a semantic point of view: the term described as an oxymoron by
Philippe Fuchs comes, following Jean-Paul Papin's remark ⁴ , from the English expression introduced by
Jaron Lanier in the 1980s where the meaning of "virtual" ("in fact, "practically") seems in the first place to
reduce the immersive device to the act of simulacrum, to a production of an immaterial world where it would
necessarily miss something compared to the real world. Our working hypothesis consisted first in verifying the
relationship between immersive environments and the very concept of reality, which required in-depth reflection
on the definitions of this couple of problematic concepts that is real/virtual, based on philosophical theories,
which were able to raise this question long before the appearance of new technologies. Following this idea of a
lack, in particular of materiality (of density of matter) or following the idea of interactivity (as a
fictional, potentially open word and which requires the interactive presence of the spectator), the term of
virtual is often confused with notions such as immaterial, possible, potential etc. and is defined either by
negativity in relation to reality or tends to become a generality, a portmanteau word, even a label for
commercial products. It is, however, a strong and autonomous philosophical concept, whose contemporary
definition we owe to Gilles Deleuze’ work who, inspired by the proposals of Henri Bergson ⁵,
reverses the classical Aristotelian position, and replaces "power" by " virtual”, by proposing the famous
double opposition: “if the real is opposed to the possible, the virtual, for its part, is opposed to the
actual” ⁶ . Deleuze thus endows the virtual with "full reality, as virtual", with an ontological
dimension, and proposes to consider any object as having one of its parts in the virtual, which, far from the
indeterminate being, would rather constitute an objective dimension at the origin of the process of
actualization.
Other pairs of concepts were founded to support our reflection that goes from its conceptual state to its
concrete state, verifiable by experience: following an ontological reflection (presence / immersion / threshold
of presence; site / space; place / territory); following the experimental and pragmatic reflection (place of
anchoring / place of immersion; realization / derealization; interactivity / agency; space / sound staging;
frame/editing), following the phenomenological reflexion (interaction and environment). Thus, the question of
site has become the starting point of our common reflection, since it makes it possible to problematize the
paradoxical relationship between the concert concrete? place (concrete site) and the virtual site of
implementation of the experience, as well as the quality of the experience itself, even from the point of view
of the notion of "threshold of immersion".
The research results proposed in this article concern theoretical advances in the context of the definition of
the term of virtual, through the following stages:
1) a philosophical deepening of the term of virtual, 2) the proposal of a mental map, 3) a theoretical study,
following a graphic schematization of the Deleuzian philosophical system, of the notion of presence from the
phenomenon of oscillations of presence of the immersant during the VR experience, 4) a study, based on
the restricted practical application of the Deleuzian system, of different modes of mediation constituting the
act of presence, 5) a proposal for a theoretical evaluation tool which makes it possible to identify the lack of
presence and to propose a typology of it, 6) a reflection on space-based montage, 7) a critical approach to the
concept of the virtual placed in the larger context of the history of art and the moving image.
Incorporation of the virtual: places of anchorage and organicity
The site (physical space) as physical anchorage of images and processes of virtualization and actualization
constitutes one of the central points of our reflection in the study of the concept of the virtual. The site
should be understood as space, territory, but also deterritorialization, device, network, interface, but also
body which in turn incorporates virtuality through its proprioceptive properties or through the organization of
a space or sound medium which makes perception tangible (processing of sound at the level not only of volume but
also of phase, etc.). Olga Kobryn suggests thinking about the virtual outside the static framework but in the
dynamic sense of the term and insists on the fact that the virtual is not an object but a process and only
becomes perceptible as a process of virtualization and actualization. The idea of the site as anchorage then
seems to be best suited to reveal the trace and grasp the issues. The space of the virtual can only be conceived
and made perceptible as an oscillation of presence and extension both in space and in time, as movement and
process. “How does actualization take place in the things themselves? Why is differentiation [the
characteristic process of actualization] correlatively qualification and composition, specification, and
organization?” ⁷ We can notice here that thinking about the virtual calls for dissociative pairs,
the ramifications of meaning: thus, if actualization is defined by Deleuze as composition and organization, the
process of virtualization is defined by Pierre Lévy ⁸ as problematization and not disorganization.
There is thus a displacement of meaning and the appearance of new pairs: problematization / solution that
diverts the problem of the most expected pairs: visible / invisible, material / immaterial, concrete / abstract.
The level of reflection moves from structure and static physical and spatial characteristics to conceptual,
temporal, and mobile characteristics. Thus, it can be deduced that the virtual is only sentient and is probably
only as a process and not a static entity. Thus, the notion of symbiosis seems to impose itself as a structuring
notion of the concept of virtuality: the anchoring of the levels of virtuality in technological devices
[dispositif] and concrete sites, in the body which updates and serves as a catalyst for the concept which then
becomes organic.
This organicity of the virtual is part of its very nature, Deleuze having demonstrated the concept through the
image of an embryo ⁹. The virtualization / actualization processes are at the very origin of the VR device
as a symbiotic device - the association of a concrete place which accommodates a differentiation of levels of
reality, a stratification of reality whose different strata represent qualitative changes instead of canceling
each other out. Paradoxically, the virtual needs anchoring in the real to be able to differentiate itself from
it.
Presentation of the themes studied cartography of the vocabulary
To return more concretely to the construction of this vocabulary, it is important to detail the research method
of our working group. During twenty sessions of regular three-hour meetings, from 22 November 2019 to 20 April
2022, several researchers from different backgrounds dialogued around a common debate. Retrospectively, we can
say that the thinking process took place in three phases of reasoning. Firstly, several presentations made it
possible to explore all the meanings that a term could cover. Then, these presentations gave way to more
restrictive exchanges around a definition. Finally, the group consolidated in order to build a global
theoretical reflection. Far from being indifferent to each other, these major stages of demonstration
(exploratory, definitional, theoretical) brought together study and theory through a particular analytical
method, made up of experimentation and failure, by groping around. In order to explain the functioning of this
scientific approach, we propose to study here the first tool that emerged from it: the mapping of a virtual
glossary.
This "visualized" dictionary was constituted after several exploratory exchanges and focused on the resonances
of the following concepts: the immersion/presence couple and the real/virtual couple. The delimitation to four
terms is not arbitrary as it tends to prove that they cover various notions, whether they are shared with other
domains (cognition, derealization, threshold, affordance) or specific to virtual media (actualizing, immersing).
The first principle of mapping is to systematize these lexical encounters to account for conceptual advances and
resistances. In this way, we can observe that the concept of immersion, although it can be distinguished into
three categories (real immersion, fictive immersion, virtual immersion) with precise specificities (environment,
imaginary, interactive, proprioceptive), nevertheless summons a plurality of uses which intersect and merge.
Plotted on the map, these very different uses of the concept are linked by a color code, which distinguishes
them, a set of icons, which associates them, and a hierarchical structure, which ranks them. These multiple
entries finally come together around two key words: narrative, or iconic, immersion and sensory, or a-iconic,
immersion.
However, this system of cross-referencing through a two-dimensional cartography suffers from several conceptual
problems. First, it is based on an excessive heterogeneity of tree structures, which leads it to accommodate
very different scientific elaborations (aesthetics, philosophy, cinema, ergonomics, design, engineering sciences
or cognitive sciences) in an unlimited expanse. It is then subjected to extensible fields of reflection that can
lead to an infinite expansion of the map object. Without limit, cartography can be interested in questions of
production (theories of virtual creation), design (realization of virtual objects) or use (virtual receptions
and uses). It has now become essential to move from a summary map, containing all the definitions, to a
prospective map, capable of providing an active theorization.
This updating of the map through its results eventually led to different volumetric cartographies reshaping the
debates contained in the definitional stage. Before detailing the specific intellectual path from one set to the
other, it is interesting to look at their results. At the heart of the new "Virtual/Actual/Possible/Real" or
"Affective Participation /Presence/Action/Dispositive '' sets, we can find the debate around narrative or
sensory immersion. Even if the conclusions remain similar in the maps to come, their justifications specify the
differences. On the one hand, there is affective immersion, which enables the apprehension of an environment
from stimuli linked to the aesthetic device and psychic presence, and on the other hand, there is agentive
immersion, which is based on abilities to participate in the environment, and which are linked to action and
affective participation.
Paradoxical relationship between concrete place of the experience and virtuality
The paradoxical relationship between the concrete place of the experience and virtuality (in the Deleuzian
sense) which always needs an anchorage in actualisation may be found in early dispositifs such as
Robertson's phantasmagorias. Before cinema emerges in the Capucines district in Paris (at Café de la
Paix), Robertson used phantasmagorias in the Couvent des Capucines to actualize different virtualities of the
place and played with the idea of bringing back ghosts, especially in relation to the political and at the same
time symbolical and geographical turmoils since the French Revolution and the First Empire (removed bodies and
statues of famous people, religious apparitions). This dimension also works in a contemporary cinema that is
interested in this power of actualisation, as, for example, in the movie La Vierge, les Coptes, et moi
(2012) by Namir Abdel Messeeh. With the exception of some rare experimental productions, VR seems very
little 'virtual':
rather, it tends to enclose the action within a few possible scenarios. This distinguishes it from interactive
documentary, for example, in which inventive or creative abduction often plays a role in the
investigation, with very interesting back-and-forth between tree structures and actualisations (geolocated
concrete places, for example). The production of the Raspouteam group on the Paris Commune in 1871 and its
re-actualisation in contemporary Parisian space underlines this dimension (https://raspou.team/1871/). By
definition, indeed, simulation belongs to the realm of the possible. VR is, therefore, not
virtual-friendly from a Deleuzian point of view but it may be interesting to ask, on the one hand, how
simulation dismisses the Deleuzian virtual, and on the other hand, whether the Deleuzian Virtual, linked to
creativity itself (from the biological to the human), can be completely evacuated and how it 'comes back'
¹⁰.
The cartography and the mind mapping of our first collective works could be re-problematized according to this
axis while taking into account the principal points of connection of the experience, namely the question of the
“dispositif” (in particular the position of the spectator inside and towards technology) - D, that
of the affective participation at the time of the experiment (PA), that of the presence and its effects (P),
that of the action (A) with different means (joysticking, walking, etc.) finally. The Set theory in mathematics
(Georg Cantor), which allows to consider that an item can belong to several orders according to its properties
(i.e., 10 may belongs to two sets: set A which includes the multiples of two, and set B which includes the
multiples of five) could give a framework to the description of the experience in order to observe how the
virtuality could emerge, or not, from a VR experience.
The Deleuzian approach can be drawn through a cross taking the place (Lieu) at its middle (see illustration 2a
below) and two lines crossing in this middle, the first from Virtual to Actual, and the second one from Real to
Possible. The interactive documentary has to deal with our virtualization of what may emerge by abduction in
browsing the internet from page to page, that is on the upper right of the cross, between Virtual and Possible,
whereas the VR headsets trap us in the Actual and offer us possible choices to act (or not) with and within the
technological device, on the lower right of the cross, between Actual and Possible. If we project the four main
points of connection on this cross, we can describe experience paths, such as one that may explain that we act
to turn right, or left or whatever (A, Action), to face a new challenge because we felt confidence by overcoming
a previous success (PA, affective participation). These paths may be related to one of the four Sets (Virtual,
Actual, Possible, Real), so we can have a kind of 3D representation in order to describe an experience (as
below).
In this figure, we can see the experience Guillaume Soulez met when he tried to “disturb” the
device of a VR experience named A Fisherman’ s T ale during the NewImages festival (Paris, Forum
des Images) in 2019. The immersant is a fisherman locked in a lighthouse and s/he must manipulate
objects (grab a teapot...), open windows, etc. A representation of the lighthouse in the form of a model, in
which an avatar of the fisherman is represented, offers a sort of miniature vision of the room and the actions
undertaken (see for example: https://culturevr.fr/a-fishermans-tale/). In a kind of fractal vertigo of
dimensions, I am, at certain moments, myself the miniature of a figure which overhangs me.
All my actions were then possible actions—mainly suggested by the setting and the presence of
objects—to try to make the starting situation evolve, except one. I tried to act on the model, making the
hypothesis that to modify the model was to impact the principle of the device itself, thus perhaps to act on the
system to "get out" of the confinement of the lighthouse (or to disturb the device and see how it reacted). This
virtuality was not actualizable (no modification occurred by trying to touch the model, etc.), but, unlike the
other thoughts and actions, it was not foreseen by the system that had fixed a certain number of possibilities.
In the figure, we can see the path from a virtual (green) D (dispositif) to a idea of action to actualise (red)
an action (A) in order to get a new virtual affective participation (PA green), but it doesn’t work: so in
the Set of the Actual, this non-actualisation brings a dysphoric feeling of failure (PA red) and I have to admit
that the real dispositif (D yellow) makes it impossible rather than possible (D blue). In light gray, you can
see the virtual path I imagined (from the same D green but to an A yellow not actualisable) which could have
actualized (in red) for me a conjunction of new sense of presence (P red) and of (euphoric) affective
participation (PA red).
This abduction (hypothesis) was undoubtedly suggested to me by the place and the atmosphere produced by the
festival: this one proposes to multiply very varied experiences in a relatively short time, to play with
different devices, to pass from one to the other, which incites finally to practice a meta level.
Therefore, we can provisionally conclude that the place of the experience (here the festival as a place) must be
taken into account (as we’ll see below with the semio-pragmatic approach) in order to understand how
symbiosis may occur, or not, with an immersive technology such as the VR headsets. One could also say that a
variant of our abductive trial is hacking, which consists in "hijacking" a device, i.e. opening a
virtual space beyond the system of possibilities that it constrains. Therefore, we could ask whether a
successful hacking would mean a kind of translation to another symbiosis, the hacker’s one, his/her own
personal (but sometimes shareable) universe. This approach and this VR experience suggest deepening the analysis
with other VR products, especially “experimental” ones, in order to test other types of
virtualization. But let’s now analyse the way this Set model may be concretized in specialization of VR
experiences.
Comparing spaces of mediation
Based on Guillaume Soulez's proposal to apply set theory to a cartography of the virtual while affirming the
concept of place as the meeting point of the axes of Deleuze's real/possible and virtual/actual schema, Francois
Garnier proposes to pursue this idea to the point of considering place as the space of an experience, where the
Deleuzian axes would be the driving forces of a new representation of the affordance action/perception loop.
The place is thus no longer a concept at the point of intersection of these relations (Figure 2a), but a closed
and open space encompassing the space of expression of the mediation where a specific experience takes place. It
marks the limits, even if it remains open to external influences. To form the affordance loop, we add two arrows
to the diagram, opening up two questions: how does the imagination of the Possibles prepare us to be present in
the Virtuals? How do the Actuals enrich the Real of the device and the experience? (Figure 2b)
On the map of this Place, it is possible to situate the 4 sets proposed above: the Device set, which regroups
the Real pre-existing the experience, the Affective Participation set, which is the stimulator of our projection
in the Possibles, the Presence, which is the condition to the setting-up of activation, of vibration or of
reasoning (see the quotation of Proust by Deleuze above) of the Virtuals, and the Action, which is the bodily
engagement actualizing a Virtual as an Actual, emerging to enrich the Real, the experience. (Figure 2c).
Based on this conceptual scheme it should be possible to try to study the differences existing in the
affordance action/perception loop when applied to a technological mediation site such as cinema, virtual
reality, or shared virtual reality.
In an experience of mediation, a dialogue is established between the author and the spectator-actors.
The 2 flows Real > Possible and Virtual > Actual can be associated to the reciprocal roles of the author
and the spectator. The author stages the Reals, opening up paths and guiding the spectator towards Possibilities
(act of narrative), the spectator-actors acting out the Virtuals and allowing the unfolding of the experience to
progress.
In the cinema (Figure 2d), the spectator does not have the possibility of acting on the loop of affordance
between Possible and Real via the actualization of the virtuals. He cannot intervene in the unfolding of the
experience; this role is given to the actor whose actions are predicted and are already part of the Real
included in the device which will be revealed progressively in the time of the narrative. The actual is
predicted, recorded in the device, it is included in the Real pre-existing the experience.
In Virtual Reality (Figure 2e), the immerser enters the affordance loop between Possible and Real via the
actualization of the virtuals, he can act on the unfolding of the experience and make it his own. Within the
limit of the possible and of the interactions proposed by the author, which make up the device, the immerser
actualizes the virtuals. This actualization requires a Presence and produces the Actuals which come to enrich
the Real of the Device.
In shared Virtual Reality or Metaverse (Figure 2f), the experience becomes an open space, the presence of the
others gives an identity to the immersant and influences his perception of the possibilities and his
affective participation. He is no longer alone to act and to produce actuals, he must cooperate, make society in
the Real of the experience in constant becoming.
Bodies and Spaces: methodological aspects
The mode of embodiment, or mode of existence in virtual reality environments is quite specific as it engages
the whole body of the participant in its sensory, physical and physiological dimensions, as well as in its
cognitive and socio-cognitive dimensions. The transactional situation unfolds between a physical space in which
the participant physically moves and experiences sensations and a digital space where he/she experiences
interacting with the narrative, processual and procedural levers of the programme.¹¹
Roger Odin ¹² builds a model of mediated communication that he considers as a methodological tool for
research. From a semio-pragmatic perspective, he proposes to move from the notion of a communication situation
in context to one of “spaces of communication.” The space of communication is the place where the
"actors" in the production space, as in the reception space, produce meaning on the same axis of relevance. They
share the same constraints (cultural, symbolic, cognitive, social, etc.) which constitute a common and shared
framework that allows for the orientation of communicative skills related to a specific and contextual space.
This proposal has consequences, as it is situated in an approach which allows interpretative processes in a form
of non-communication. If the instance of production constructs/produces signs that can be apprehended by the one
who is supposed to receive them, it is not certain that the receiver receives them as they are proposed, that he
even perceives them. It is therefore not certain that he, in turn, sends signals on the same axis of relevance,
in the same space of communication.
From a phenomenological perspective, the model developed by Philippe Bonfils¹³ transposes R. Odin's
model to immersive environments. He considers four distinct spaces of communication to explain the engagement of
"subjects" during their experience in an immersive environment.
The first division is between digital space and physical space, the second distinguishes the space of
possibilities from the space of action. The space of possibilities corresponds to what the subjects "do" with
the sensations and information perceived in the physical space and in the digital space.
This makes it possible to observe and describe the way in which they actualise, interpret, give meaning to, and
organize these sensations and information (R. Odin's vibrations) according to their own capacities/knowledge.
The space of action corresponds to the actions that the participants carry out through their bodies in physical
space and/or in digital space. In the course of the action, these four modes of instantiation of the body, which
are or act in as many communication spaces, operate in a loop. The adjustment of perceptions and actions is
permanent, as is the updating of representations and knowledge, thus allowing the construction and circulation
of meaning on the activity proposed by the mediation of an immersive device. This engagement is not only due to
the immersive or interactive qualities of the device. It is just as much about what the "subjects" want to do
and can do or understand in the space of physical possibilities with what they perceive of the space of digital
possibilities; and what they can do through their actions in the space of physical action to achieve their
objectives in the space of digital action.
To conclude, while this division into (four) spaces of communication is purely theoretical, this modeling has a
strong methodological interest. The fact of making these distinctions makes it possible to observe, during the
immersive activity, what the participant perceives, understands and acts upon and what he does not perceive,
understand or act in the virtual reality device. This modeling allows for a better understanding of what is
known and shared about the constraints that constitute these shared spaces of communication between the
participant and the experience imagined and designed by the creators, and what is not. In other words, this
methodology applied to the observation of the activity allows us to better understand what makes symbiosis, or
what does not make symbiosis between the immersive device imagined by the creator and the participant who
experiences it.
A practical case
When conceptualizing and developing a virtual reality device, depending on the constraints of space, time and
interactivity left to the immersant, the artist can more or less anticipate the actualization of the
experience. When conceiving an artwork, he or she may imagine the artistic impression to transmit to the
receiver. The artist tries to copy his or her model of thought to one of the subjects, hoping that the artistic
impression desired is the one lived during the reception of the artwork. However, according to the possibilities
left to the immersant, the behavior of the latter can move away from the artistic impressions
envisaged. Moreover, the algorithmic nature of virtual reality systems enables the artists to give behaviors to
their devices which can influence the aesthetic experience of the immersant. In this way, interactive
works allow a new expression modality to emerge, that of the relation between the behavior of the work with that
of the subject.
Regarding the analytical tools proposed above, we propose a case study around the device "Montage
Spatial” developed by Rémi Sagot-Duvauroux within the framework of his research-creation. This
device, described in a previous paper,¹⁴ proposes to explore the following questions: How can an
artist test the effects of staging and montage that he or she wants the viewers to experience? How might the
behavioral rules be programmed whilst taking into account future behaviors of the immersants?
This device is based on a practice of film editing and on the states in which an editor might be when editing a
film. If the montage is an intellectual act allowing the spectator to think about the film, the editor
experiences the process of making it. By seeking combinations of shots that express narrative, discursive and/or
poetic intentions, the editor considers and experiments a multitude of possibilities and is regularly surprised
by the aesthetic effects that emerge. The virtuality in the practice of editing could then be located in this
unpredictability underlying the film structure in progress. In this same dynamic, the device "Spatial Montage"
seeks to find the montage’s inherent oscillation between the manipulation of the matter and the projection
onto the moving image. This quick transition from the state of creation to that of reception might be
conceptualized by the passage from the structuring of possibilities to the actualization of the underlying
virtual by the mounted moving image.
On a virtual table (Figure 4), an artist builds a model with interactive sets, characters, light cones, and
sound bubbles. He or she also may locate and orient different walking areas within the model. By pressing a
button, the artist acquires the scale and the point of view of one of the three zones (Figure 5). By being able
to oscillate between the scale of the staging and that of the reception, the artist is able to switch at any
time to test the actualizations of the possibilities that he or she organizes spatially.
Moreover, once at the scale of the model, according to the immersant’s movements, viewpoint, and
distance to the characters, the latter becomes active, animated, slows down or freezes. In a sensorimotor loop,
the environment is puppeteered by the immersant's bodily actions, which are themselves influenced by
the scenography of the unfolding environment. Thus, even by having anticipated realizations, the artist is
confronted to the unexpected behaviors of the staging which makes him or her experiment the discovery state of
the immersant. We might detect here a form of virtuality of the staging. This unexpectedness, as
present in the act of creation as in the reception, can thus constitute a relational meeting point between the
artist, the artwork and the subject.
Finally, the different zones are connected by wormholes: passage points between two spacetimes set up by the
artist and materialized on stage by a frame that might be crossed. The immersant acquires thus a
"body-editor" of his or her own aesthetic experience. If these extradiegetic portals signify the presence of an
artist and his or her point of view during the time of the transition, the temporality of the latter is under
the responsibility of the immersed body. In a fast oscillation between the perceiving and the acting body, he or
she embodies not as a diegetic entity but as a gear of the ongoing narration, putting in tension his or her own
temporality with the dynamic one of the experience. This "body-editor", at the same time anticipator and ready
for surprise (or even jubilator) seems in this way to embody the virtuality of the editing acts both in the
making and in the reception. It therefore enables a way of thinking in action the relations between artists,
virtual environments and immersants.
Discussion and Return onexperience
Before we conclude our provisional discussion of the vocabulary of the virtual, we would like to go back to the
expression of "virtual reality" which seems to resist our investigation. We have proposed several definitions of
the virtual in virtual reality but is this really sufficient? What about the reality in virtual reality? In
English, a virtual reality is a quasi-reality, a reality which is "almost" real. And in French, the official
expression for virtual reality is "synthetic reality,"¹⁵ a reality which is simulated by a computer
program. Let us examine where this new terminology leads us.
Compared to cinema, a distinguishing feature of virtual reality is that the images presented to the audience
are computed in real time, rather than being pre-recorded. As a result, virtual reality appears to be a
realization of the procedural approach to movement, anticipated by Bergson in "The creative
evolution"¹⁶ where movement is not produced by a succession of still frames, but as a result of a
dynamic simulation. Images and movements produced in this fashion challenge the classification of
cinematographic images by Deleuze.¹⁷ How should those simulated images be classified? In virtual
reality, the movements of the immersant influence the simulation and create a different and
unpredictable experience every time. What makes virtual reality different from cinema is not its virtuality or
reality, but its relation to space and time. From this perspective, virtual reality may be closer to theater
than cinema. Like virtual reality, theater is a tridimensional art where each performance is unique. Variations
may be more or less important, but an exact repetition is excluded. What distinguishes virtual reality from
theater is that the actors, the settings and the props are all software, no hardware.
Virtual reality is therefore different from cinema because it occupies the "real time" and the "real space" of
our experience. And it is different from theater because it is immaterial. As a result, it is better described
as a "synthetic" or "artificial" reality¹⁸ than a "virtual" reality.
This redefinition raises new and interesting questions. What is an artificial reality and how can it be
constructed? How can it be realized? The French translation for "director" is "réalisateur". It is
interesting to contrast the term of "realization" in this sense of "direction" or "mise en scene" with the
opposite term of "derealization." In medicine, derealization is a psychological disorder where one loses the
sentiment that the outside world is real. The two terms are not meant to be opposed to each other, but they
provide some interesting insights on what it means to build a virtual reality experience.
On the one hand, virtual reality requires a director who "realizes" the experience by programming the
simulation in the first place. On the other hand, virtual reality also requires an immersant who
"realizes" the experience as an exercise of make believe. Derealization teaches us that the sense of reality
cannot be taken for granted, even in real life, and can in fact be affected by long and repeated exposures to
virtual reality.¹⁹ Building this sense of reality in a virtual experience may be the defining factor
of virtual reality, one that requires a symbiotic relation between the director and the immersant who
both contribute to its realization.
Hommage
The authors pay tribute to Roger Odin, who passed away in August 2023, whose wealth of research nourished their
reflection; whose benevolent advice will always nourish their path.
References
1 Daniel Mestre, Philippe Fuchs, “Immersion et présence” dans Philippe Fuchs
(dir.), Le Traité de la réalité virtuelle, Volume 1 : L’Homme et
l’environnement virtuel, Paris, Mines Paris, coll. "Sciences mathématiques &
informatique", 2006, 309.
2 Michel Serres, Atlas, Julliard, 1994.
3 Jonathan P. Rowe, Scott W. McQuiggan, James C. Lester, Narrative "Presence in Intelligent
Learning Environments, Narrative Presence in Intelligent Learning Environments", AAAI Fall Symposium:
Intelligent Narrative Technologies, 2007.
4 Philippe Fuchs (dir.), Le Traité de la réalité virtuelle, Volume
1 : L’Homme et l’environnement virtuel, op. cit., 5.
5 Henri Bergson, Matière et mémoire [1896] et "Le Possible et le réel"
[1920], in La Pensée et le mouvant [1934].
6 Gilles Deleuze, "The virtual is not opposed to the real but only to the
actual”, Différence et répétition, Paris, PUF, "Epiméthée"
collection, 2003, 269.
7 Gilles Deleuze, Différence et répétition, op. cit., 276.
8 Pierre Lévy, Qu’est-ce que le virtuel ?, La Découverte, 1998.
9 Gilles Deleuze, Différence et répétition, op.cit. See also Arnaud
Bouaniche, “Chaos débout”, in O. Kobryn, M. Ovtchinnikova, G. Soulez (dir.),
Théorème n°38, Qu’est-ce que le virtuel ? Lieux d’ancrage,
Paris, PSN, forthcoming 2023.
10 Guillaume Soulez, “Lieux du possible, lieux du virtuel ”, in O. Kobryn, M.
Ovtchinnikova, G. Soulez (dir.), Théorème n°38, Qu’est-ce que le virtuel ? Lieux
d’ancrage, op. cit.
11 Maude Bonnefant & Thibault Philippette, "Rhétorique de l’engagement ludique
dans des dispositifs de ludification", Sciences du jeu, 28 octobre 2018.
12 Roger Odin, Les espaces de communication ; introduction à la sémiopragmatique,
Presses universitaires de Grenoble, 2011.
13 Philippe Bonfils, L’expérience communicationnelle immersive : entre engagements,
distanciations, corps et présence., Mémoire d’habilitation à diriger des recherches,
Université Lille Nord de France, Lille, 2014.
14 Rémi Sagot Duvauroux, Rémi Ronfard and François Garnier, Montage as a
narrative vector for virtual reality experiences, ConVRgence (VRIC) Virtual Reality International Conference
Proceedings, International Journal of Virtual Reality, 2022.
15 Réalité de synthèse, Vocabulaire de l’informatique, Journal Officiel de
la République Française, 20 avril 2007.
16 Henri Bergson, L’évolution créatrice, Paris, Félix Alacan, 1907.
17 Gilles Deleuze, Cinéma 1, L’image-mouvement et Cinéma 2, L’image-temps.
Editions de minuit, 1983, 1985.
18 Philippe Fuchs, Théorie de la réalité virtuelle : les véritables
usages, Presses des Mines, 2018.
19 Rebecca Searles, "Virtual Reality Can Leave You With an Existential Hangover", The Atlantic,
December 21, 2016.
Post-Human Motherhoods: Reflections on mother-offspring bonding as symbiotic individuation in Contemporary Art
GrazieleLautenschlaeger
Kunstuniversität Linz | VALIE EXPORT Center Linz, Austria g.lautenschlaeger@gmail.com
Abstract
As part of an ongoing postdoctoral research on post-human motherhoods in media art, the paper examines selected
artistic expressions of mother-offspring bonding and its relationships with different technological cultural
basis. The introduction addresses the handling of the concept of the mother and its related operations in the
research scope. Following this, we suggest as standing points for reflection both the material basis of the
placenta and playfulness as elements shaping the mother-offspring bonding. They constitute fundamental aspects
triggering affection in the potential symbiotic relationship established in mother-offspring bonding. The
discussion is permeated by artwork examples, brought on demand to the text to feed the reflection. The notions
of “natureculture” by Donna Haraway and the theoretical framework by Second-order Cybernetics are
grounding references. The final considerations point towards the human and post-human aspects of motherhood
revealed by the artworks, as well as to the need for concrete actions towards the reduction of the gender-based
technological gap in media art, in order to increase the imaginary variability of the field.
Inspired by my own experience as a media artist who became a mother, I was curious to know what artistic
expressions emerged from the intersection between motherhood(s), technofeminism(s) and post-humanism in the arts
over the last decades. Soon I found that, there was neither an abundance of examples nor has this intersection
been sufficiently circumscribed in the academic field. Therefore, the initial curiosity became my current
post-doctoral research project at the VALIE EXPORT Center / Kunstuniversität Linz.
“As a radical onto-existential re-signification of the notion of the human,”¹
post-humanism considers the “significant otherness” and the inseparability between nature and
culture. The acknowledgement of the agency of non-human entities, from active matter through machines, to other
species, material-discursive approach, even if not mentioned or declared, grounds much of the media art
production since its emergence in the history of techno-culture.
To a certain extent, it is possible to deduce that the absence of motherhood as a topic in media art field
mirrors the gender-based technological gap of the field experienced until today. In other words, numerous
reports and studies point to the so-called digital divide, referring to the inequalities between groups with or
without access to technological tools. Among other non-privileged groups, it is constant that girls and women
have less access in general. Inequalities are even more contrasting if we consider the gap between those who
have access as consumers and those who are developers—an issue frequently disregarded in many studies on
Human-Computer-Interaction (HCI).²
Aiming to dig into this problem, the project Where are media artist mothers? also integrates a
survey³ to investigate if the hypothesis is true. Among the concerns is the question of what measures
should be taken to better welcome media artists who are mothers in the production context, to populate media art
with more constructive and empowering perspectives of post-human motherhoods, and therefore, to enrich the art
and technology environment with a higher variability of imaginaries.
In this paper I share only a frame and still open part of the research, addressing some references and examples
to reflect on how artists articulate common symbols, materialities, technologies and alternative narratives in
relation to motherhood and post-human bodies. More specifically, the paper observes the materiality and dynamics
integrating the mother-offspring bonding and its expressions in the art field.
To conduct analytical comparisons among selected artworks under a common conceptual umbrella has been one of my
strategies to investigate how (media) artists respond to the paradigmatic changes in feminist movements through
the lenses of mothering issues.
Motherhood and related operations
The general approach towards motherhood in the research is based on the diagram shown in figure 1:
In the human case, these operations can also be associated to the sequence of topics involved in the
reproductive cycle, encompassing right from the concerns around fertility, through fecundation itself, the
transformative aspects of pregnancy and gestation, the birth-giving, to procedures and ethics of offspring care.
Due to the lack of motherhood and related topics representations in the media art field, I look with special
interest to a specific profile of mothers: artists that are also technologists, creating their proposals in the
intersections between scientific and technological knowledge, whose artworks or performances reveal sorts of
media thinking, critical approaches to technocultures, and/or technofeminist perspectives.
Parenting as reproduction and care work
Regarding the considerations above and for a more respectful approach to all possible types of motherhood, the
term “parenting” is certainly more appropriate. This observation is structured on the
acknowledgement of the feminist struggles towards a more equalitarian division of the high demand of work
generated alongside the gestation and care of a newborn – often invisible and non-paid work. US-American
experimental artist and technologist Ani Liu, who between May and July of 2022 had a solo exhibition at
Cuchifritos Gallery in New York titled Ecologies of care, shows a diverse series of speculative
artworks that invites us to reflect on the invisibility of reproductive and childcare labour. At Untitled
(Labour of love)⁴ Liu builds a data portrait recording every feed and diaper change for the
first 30 days postpartum.
The coincidence of the terms “labour” and “love” in the piece’s title addresses
precisely the ambivalences that pave the culturally constructed perversion in devaluating creative/generative
work in multilayered contexts – from the reproductive work itself to the art context.
In order to turn the repetitive and exhausting care work demanded by a newborn visible, the artwork also
portrays a way of thinking inspired by the ongoing big-data technological environment, where we are responsible
for curating what is the relevant data to turn into information through data visualization. Dealing with
information as embodied data, would Ani Liu’s Labour of love be an actual version of the iconic
Mary Kelly ́s Post-partum document (1973-79)? ⁵
Indirectly, Liu’s piece is also suggesting love as the strong element fostering the mother-baby bonding,
which is neither certain nor unanimous among all cases of mother-offspring relationships, be it in regard to
human or non-human species.
Now connecting to ISEA 2023’s theme, would symbiosis be the key towards healthy mother-child
relationships, despite all the ambivalences of mothering? How does the bonding emerge and is regulated? How do
the symbiosis aspects emerge in artistic expressions?
We may find part of the answers to the aforementioned questions through an investigation on the possible ways
through which affectivity is generated.
Mother-offspring bonding
The strong connection between mother and offspring is a long-term process starting in the gestational phase,
continuously nurtured through the care work demanded by the newborn until its autonomy as a self-caring being.
Although it is known that there are studies in psychoanalysis, for instance, that doubt the mother-offspring
relation as symbiotic and conflict-free,⁶ in this session I address only two aspects of a huge variety of
possible mechanisms that contribute in shaping the symbiotic aspects between mother and offspring: the role of
the placenta and the role of playing.
The case of mammals: since the womb via the placenta
Observing the very material basis of mother-offspring bonding, it starts on the physical and chemical exchanges
between cells within the mother’s womb, from which embryo and placenta emerge.
Professor Marjolein Oele places the placenta as “a fetal and maternal place-and-time making
boundary”⁷ and discusses how the placenta materializes a model of “affective
symbiogenesis where selves come into existence.”⁸ Her philosophical reading of the placenta
has special impact in the project I report below.
Intrigued by the metamorphosis I went through while I was pregnant, increased by the intensity of every change,
I felt compelled to start gathering materials referring to the significant fragments of the ongoing irreversible
revolution. Among these materials there are audio samples recorded from the belly (heartbeats, placenta, veins
flux, etc.), as well as the sounds of the baby after birth, interviews with mothers from different profiles,
stamps of the placenta, data collected from the glycemic control of gestational diabetes and writings. This
initiative somehow assisted me in accepting the difficulties of the fast changes as well as in getting prepared
for both the birth labour and puerperium. Simultaneously, it became an art project, that I called
Binômio (Binomial 2021- ), still in progress.
Binômio encompasses a series of aesthetic experiments articulated from the recombination of
afore-mentioned different elements collected since my pregnancy, addressing issues arising along the experience
of mothering. The experiments interlace material and symbolic relationships of affection, intimate and universal
aspects, intertwining subjects, organic and machinic elements. Fascinated by the placenta as an ephemeral organ
and the various cultural manifestations related to it, there are some experiments in which I focus on the
extraordinary role it plays—e.g., the placenta provides oxygen and nutrients to the baby, produces
hormones that promote the baby’s growth, removes toxins and carbon dioxide, besides protecting and passing
immunity from the mother to the baby, etc.
For the context of this article, I highlight two of them:
Placenta: Interface (2021) has been a conceptual piece, a print media merging a text with the
image of one placenta stamp, done with blood just after birth. Appraising the placenta’s amazing qualities
as interface, the text is a poetic letter to my daughter, written on its contours and places of flux and
exchange. I envision its mysterious ways of functioning as the most incredible interface possible.
Abstracted Placenta is another ongoing experiment, a luminous sculpture as a homage to the
artist’s forgotten placenta at the hospital, as well as the uncountable wasted ones by medical teams.
Lacking information and ethical procedures, in most cases the placenta is ignored, on the one hand, the right of
the birth-giving body to possess their own organ, and, on the other hand, the multiple benefits that
placentophagy can provide for women in the postpartum. ¹⁰
The piece displays the changes from the glycemic values, systematically collected by the artist in the week
when she discovered she had gestational diabetes. It suggests the placenta as an interface organ between the
pregnant body and the baby, as a sort of data visualization of the complex regulation capacity of the organ,
which despite its ephemeral life, undertakes the whole responsibility during the mother-baby bonding process. As
a self-regulated organ, the placenta is considered to have its own “conscience,” characteristic
suggested by the pulsing light animating the glass piece, whose shape also remembers a jellyfish.
Despite the scientific advancements on the creation of artificial placentas, both the conceptual and material
presence of the organ in contemporary media art is practically inexistent. Among the rare cases one can mention
the participation of the Japanese obstetrician/ gynecologist Nobuya Unno at the Symposium of the Ars
Electronica 2000 – The next sex: Sex in the age of its procreative superfluousness, where he
presents an article on the Development of an Artificial Placenta. Besides not being based on an
aesthetic exploration of the organ, Unno concludes the text with remarks on his proposal as an
“extension of the preexisting neonatal intensive care system for the extremely premature
newborn” and with the warning on the high costs of such an enterprise: “using
extracorporeal circulation would be destructively expensive as an alternative for natural intrauterine
pregnancy.” ¹¹
In addition, other online searches on “placenta art” only lead us to handicrafts made of placenta
or to digital illustrations. The absence of deeper conceptual, aesthetic and philosophical explorations on the
magnificence of the placenta compels me to further investigate its potential by means of the available
contemporary artistic tools.
Playing as means for bonding
Besides the very material basis of the mother-offspring bonding, one can also approach the dynamics of the
relationships that emerge through the activities of care communication, and above all, through the eagerness of
babies and children to play and engage in interaction.
Playing permeates the most varied human cultures,¹², ¹³ extraordinarily
demonstrated by Belgian Artist Francis Alÿs with his project Children’s
games,¹⁴ featured recently at the Venice Biennale 2022—The milk of dreams,
with a selection by curator Hilde Teerlink integrating the exhibition The Nature of the Game.
Playing is embedded in social life and its dynamic shape modes of existing and interacting with the
environment in which subjects are relating with, influencing the communication processes and the quality of the
relationships between subjects. Playing is also a key entrance towards a neuro-affective approach for a
healthier development for both child and family.
Fair play between different subjects requires a respectful positioning towards the corresponding
alterities—principles that may be discussed from a multitude of perspectives. In the context of this paper
and its affinity to issues emerging from the intersection between art and post-humanism, two main
epistemological frameworks integrate our investigation, the notion “naturecultures”, introduced by
Donna Haraway¹⁵ and Second-order Cybernetics.¹⁶
The neologism “natureculture” has been developed in order to deconstruct historically calcified
dichotomies such as the opposition between the cultural and the natural, body and mind, the material and the
conceptual, among others. As a review on Haraways’ own thoughts developed previously at A Cyborg
Manifesto (1985),¹⁷ naturecultures entangles multispecies narratives, as means for reflecting
about power relationships, agency, difference, sociality, ontology and epistemology.
Second-order Cybernetics, in turn, as a transdisciplinary and participatory epistemology, presents an ethical
significance based on two pillars:
(1) the inclusion of the observer while observing a system, and
(2) the recursion embedded in acting in order to know – observers of a system recursively observe the
effect of their actions, using the feedback to adjust their understanding and their following actions.
From the valued recursion of Cybernetics, the condition well summarized by Timothy Stott, lecturer in Art
History and Visual Culture, emerged:
Play was, and remains, a social technology for the cybernetic age. Advocated by many as a humanist corrective
to a technocratic and automated post-war society, play also expanded cybernetic ideas of interaction, feedback,
and systems modelling into the social domain. From the late nineteen-fifties on, especially, cybernetics and
play converged through games, toys, and interactive exhibitions.¹⁸
The convergence pointed by the author however, is rarely (if) seen in relation to mother/parenting issues in
media art exhibitions.
Articulated in the background of the ongoing research as complementary tools, natureculture and Second-order
Cybernetics offer to the players of a system, a relational, reflexive way of knowing and acting, in which the
players are not merely observing, but are indeed responsible for the values, ethics meaning they generate.
Theoretically, they compose an ideal scenario for a healthy and equalitarian interaction, enabling, in the case
of mother-offspring relationships, an effective (and affective!) bonding.
As an example, within Art History tangential to this question is Susan Suleiman’s reading of the
avant-garde through the figure of the playful, laughing mother.¹⁹ By means of inquiring what role do
women artists and writers play in the audacious positioning of the avant-garde and their postmodernist
successors, Suleiman’s laughing mother embodies a powerful, humorous or parodic critique of patriarchal
ideologies—cleverly merging symbolic innovation, political and social transformations.
Posthuman motherhoods Machinic motherhood
Other possible thematic links between reproduction and technology is found in the creation of artificial wombs
and other genetic engineering techniques that have inspired various sci-fi stories, for example, Brave New
World (1932) by Aldous Huxley or Blade Runner (1984) / Do Androids dream of electric sheep?
(1968). More recently, the series The Handmaid's Tale, based on the dystopian novel by Canadian
author Margaret Atwood, is the current most popular reference. It is precisely to avoid such dystopian worlds
that technofeminists argue for an attitude in which such technologies could also be developed by non-dominant
groups.²⁰ Is something similar expected from artists working technologically with media art and
addressing issues such as fertilisation, pregnancy, birth, as well as child feeding and education?
Besides the machinic motherhoods found in the science fiction imaginary, one can observe concrete machinic
entities built up within the aesthetic domain of media art exhibitions. One example is the installation
Génesis: dinámicas de organismos artificiales (2016),²¹ by Ana Laura Cantera,
Daniel Alvarez Olmedo and Leonardo Emanuel Maddio. It consists of a symbolic recreation of a gestation of
artificial organisms that will interact with the visitors by means of programmable robotic experimental devices,
in an artificial context of water, movement and light. The containers of the robots have an opening covered with
biodegradable plastic that is corroded by the constant movement of the organisms inside, to the point of
generating the rupture and their consequent “birth”. This way, the installation presents critical
moments of latency and birth of these organisms, from their embryonic stage, to moving and interacting with the
audience. Programmed in an evolutionary and non-cyclical way, the installation generates unique and unrepeatable
moments along the exhibition period.
What is the significance of reproducing (or even inventing) a machinic birth process in the art context? Does
the aesthetic system of Génesis: dinámicas de organismos artificiales also present a sort
of mother-offspring bonding? In technical terms it probably may have. However, would this also mean an affective
link?
This question raises the interesting issues of the still challenging notion of possible machinic
self-awareness, possible if a machinic subjectivity once emerges, and therefore being able to engage in an
affective relationship. Even with highly advanced AI-based technologies one can hardly imagine a concrete case
of an emerging machinic self-awareness. Reflecting on the reciprocal quality of affect and symbiosis, perhaps it
would make more sense to elaborate on the mother-offspring bonding between the artists and the artwork.
If we cannot speak about affective symbiosis in the machinic motherhood of Génesis: dinámicas
de organismos artificiales, can we envision a machinic mediated mother-offspring bonding? Perhaps this is
the case of the performance discussed in the next session.
Interspecies motherhood
Another post-humanist perspective regarding motherhood can be discussed through the project Hybrid family
(2016), by Slovenian artist Maja Smrekar, which referred to the simultaneous existential and political
instrumentalisation of a woman’s body and breastfeeding.
Along a three-month performance with her puppy, the artist stimulated her own hypophysis by systematically
pumping to release the hormone prolactin. At the same time, she had a diet rich in galactogens to promote
lactation, which stimulated an increase in the oxytocin hormone.
By becoming a “(m)other”, the artist experienced and demonstrated the “natureculture”
coined by Donna Haraway and continues to explore a reproductive freedom in a multispecies world. “If
nature is unjust, change nature” ²²: from this standing point the performance embraces
simultaneously xeno and ecofeminism, interspecies relations, as well as a critical perspective on technology and
ideological structures in society.
Final considerations
The artwork examples discussed throughout the article shed light onto both the very human issues related to
mothering/parenting—the invisibility of reproductive and care work within an unbalanced social
structure—and the post-human experimentations that expand and challenge the core of motherhood related
operations. Together, they give us a series of questions to reflect on the entanglements between motherhood, art
and technology—despite the rare presence of the topic in the realm of media art.
Approaching the mother-offspring bonding through its material and dynamics basis (e.g., the placenta and
playing) is just one possibility to address its symbiotic potential and how the motherhood may be related to
contemporary epistemologies frequently being used in the media art production.
In addition, they confirm the need for updates on feminist positioning within media art production, whose
community perhaps is not aware of the absence of the topic within the field.
Conclusively, in response to the still enduring inequalities addressed by feminists, it is time to find
concrete strategies to reduce the gender-based technological gap in media art. This is the only way to populate
the field with a higher variability of imaginaries, with subjects, such as motherhood, that are ignored
inclusively by the most considered alternative initiatives, agents and institutions.
Acknowledgements
A special thanks to the Kunstuniversität Linz/V ALIE EXPORT Center for funding this research and
to all the colleagues that contributed at any level of the work with their constructive feedback.
References
1 Francesca Ferrando, "Posthumanism, Transhumanism, Antihumanism, Metahumanism, and New
Materialisms: Differences and Relations," Existenz 8/2, 2013, 26-32, Accessed May 12, 2023,
https://existenz.us/volumes/Vol.8-2Ferrando.pdf
2 Lorna Gibson, Vicki L. Hanson, "Digital Motherhood": How Does Technology Support New
Mothers? ACM CHI 2013, April 27–May 2, 2013, Paris, France.
3 “Where are the media artist mothers?”, Accessed December 02, 2022,
http://shorturl.at/gnuxY
4 Ani Liu, “Untitled (Labor of love)”, Accessed December 1st, 2022,
https://ani-liu.com/untitled-labor-of-love
5 Andrea Liss, Feminist art and the maternal, London, Minneapolis, University Minnesota
Press, 2009.
6 Rozsika Parker, Torn in two: the experience of maternal ambivalence, London: Virago
Press, 1995, 43.
7 Marjolein Oele, “Generative human affectivity. The placenta as place-and-time making
in-between.” E-co-affectivity. Exploring pathos at life ́s material interfaces, Albany:
State University of New York Press, 2020, 80.
10 Sergio L. Sánchez Suárez, “Influencia de la reincorporación de la
placenta (ROP) autóloga tras del parto, en la evolución bioquímica sanguínea e
láctea", Ph.D. diss, Universidad de las Plamas de Gran Canaria, Departamento de Ciencias Médicas y
Quirúrgicas/Facultad de Ciencias de la Salud, 2015.
11 Nobuya Unno, “Development of an artificial placenta” in Ars Electronica
Archive (Ars Electronica 2000 – Next sex: Sex in the age of its procreative superfluousness), 68,
Accessed June 3, 2022, https://archive.aec.at/media/assets/7cbf0d899fb64cb5f9e299f67992e347.pdf
12 Roger Caillois, Os jogos e os homens. A máscara e a vertigem, Lisboa:
Edições cotovia, 1991.
13 Johan Huizinga, Homo ludens. O jogo como elemento da cultura, São
Paulo: Editora Perspectiva, 2000.
14 Francis Alÿs, Children ́s Games,
http://francisalys.com/category/childrens-games/, Accessed November 24, 2022.
15 Donna Haraway, The companion species manifesto: Dogs, people, and significant otherness,
Chicago, Prickly Paradigm Press, 2003.
16 Bernard Scott, “Second-order cybernetics: an historical introduction (2003)” in
Explorations in Second-order Cybernetics. Reflections on Cybernetics, Psychology and Education,
Vienna, Edition Echoraum, 2011, 383-398.
17 Donna Haraway, A manifesto for cyborgs: 1985.
18 Timothy Stott, “Learning to interact: Cybernetics and play", Accessed December 1st, 2022,
https://www.youtube.com/watch?v=8Npu6vbcG1k
19 Susan Suleiman, Subversive Intent: Gender, Politics, and the Avant-Garde, Harvard
University Press, 1990.
20 Judy Wajcman, Feminism confronts technology, University Park, The Pennsylvania State
University Press, 1991.
21 Ana Laura Cantera, Genesis: dynamics of artificial organisms, accessed January 15, 2022,
https://www.analauracantera.com.ar/genesis
22 Laboria Cuboniks, “XF Manifest” Xenofeminism: A politics for alienation,
accessed November 27, 2022, Accessed May 12, 2023,
https://laboriacuboniks.net/manifesto/xenofeminism-a-politics-for-alienation/
Bibliography
Myrel Chernick, Maternal metaphors, Cambridge, London, MIT Press, 2013.
Emanuele Coccia, Metamorphosen, Das Leben har viele Formen, Eine Philosophie de
Verwandlung, München, Carl Hansen Verlag, 2021.
Silvia Federici, Caliban and the witch: Women, the body and primitive accumulation, Brooklyn,
NY, Autonomedia, 2004.
Hettie Judah, How not to exclude artist mothers (and other parents), London, Lund Humphries
Publishers Ltd/ Sotheby's Institute of Art, 2022.
Humberto Maturana Romesin, Gerda Verden-Zöller, The origin of humanness in the biology of
love, Charlottesville, VA, USA/Exeter, UK, Imprint Academic, 2008.
Author Biography
Dr. phil. Graziele Lautenschlaeger is a Brazilian media artist and researcher, currently postdoctoral fellow
at the VALIE EXPORT Center/Kunstuniversität Linz, investigating the intersection among
motherhood, technofeminism and post-humanism. She holds a PhD in Cultural Studies from the
Humboldt-Universität zu Berlin, which resulted in the book Sensing and making sense:
Photosensitivity and light-to-sound translations in Media Art (2020), a Master of Science in
Architecture and Urbanism from the Universidade de São Paulo and a Bachelor in Media Design,
Image and Sound from the Universidade Federal de São Carlos. With a transdisciplinary
background and experimenting different media and languages, her artistic and academic production essentially
draws on Cybernetics, Relational Aesthetics, Media Archaeology and New Materialism. Among her broader research
interests are: the notion of translation of materialities (transcreation), hybridization between organic and
machinic elements, interface and interaction design, as well as relationships between spaces and narratives.
Her work has mostly been exhibited in Europe, USA and South America.
Live 4 Life: A dream for a free and open spatial performance tool towards symbiosis or death?
This paper presents the motivations, evolution, and directions behind the spatial sound performance tool named
Live 4 Life. It aims to simplify the creation and control in real time of masses of spatialised sound
objects on various kinds of loudspeaker configurations (stereo and particularly quadriphonic or octophonic
setups, as well as domes of 16, 24 or 32 loudspeakers). This spatial research, which questions ways of
associating rhythmic and spatial parameters, is based on the concept of free and open works, both from the point
of view of form (improvisation) and in the diffusion of the code. The tool, which was initiated in 2011 and
distributed in open source in 2022, has been conceived as a long-term dream against capitalism and loneliness.
Several scenarios between (technical, social) death or symbiosis of this tool (with other programs, people,
works and the visual representation field) are presented.
In a context of increasing demand for and supply of sound spatialisation systems and tools¹,
² it remains however particularly difficult to dynamically and simultaneously control multiple sounds
in real time in the concert hall space, due in part to the stratification of the composition process in several
software programs and the complexity of individually spatialising a multitude of sound events. To meet this
need, the tool Live 4 Life (1), mixing rhythm and space, offers a spatialisation
control structure, including a library of predefined spatial trajectories and rendering algorithms, mixing both
channel-and object-based paradigms and abstract and concrete spatial techniques.³ Following philosophical
reflexions and the presentation of the underlying principles of the tool, we speculate on two future scenarios
for the tool, either death or symbiosis. In this paper, we will see how an artistic intention along with a
desire for a singular and custom spatial composition paradigm led to the development of an open-source tool,
namely Live 4 Life.
A dream for Free and Open living Art
Like the composer Francis Dhomont,⁴ as a composer and performer my main concerns in music, art and life
do not concern those of “pure music,” referring only to itself, but touch on extra-musical themes
(capitalism, free parties, loneliness). I (Christophe Lengelé) could qualify my sound improvisations in
the same way as Francis Dhomont does for his piece Phœnix XXI: “a music that is
abstract in its realisation, but figurative in its intentionality.”⁵ My improvisations thus
question, via the tool and the way of creating, the most important themes in my life and offer my interpretation
of society, as expressed by Luigi Nono, which is quoted by Joshua D. Parmenter (a long-time developer of
SuperCollider) in every email on the SuperCollider mailing list:
Every composer—at all times and in all cases—gives his own interpretation of how modern society is
structured: whether actively or passively, consciously or unconsciously, he makes choices in this regard. He may
be conservative or he may subject himself to continual renewal; or he may strive for a revolutionary, historical
or social palingenesis.
The project Live 4 Life, which is a form of resistance to this imposed world, was born from the desire
to reconnect with life. It was created in response to the world and the environment (societal, family and
personal) in which I lived. I thus reflect in my improvisations this individualistic world and my disillusioned
way of facing it, by trying to use free code as much as possible, so as not to reproduce and encourage this
commercial world. Without my personal experiences or traumas in my family or in the professional world, having
worked in multinational companies in the fields of banking, industry and communication, my intention of wanting
to create this project in open source might never have seen the light of day.
Not wishing to adapt my ideas to a commercial product, I chose in 2011 to develop my own spatialised sound
performance tool on the open-source SuperCollider platform,⁶ with the ultimate goal of going beyond the
simple sharing of listening or looking. Indeed, the ultimate objective was to tend towards a liberalisation of
modes of creations, focusing more on doing it together outside the market and the sharing of creative practices
open to all, by giving all the code developed over the years, like many artists who distribute their codes in
open source, such as Fredrik Olofsson (2) or CHDH with egregore (3) or
vivarium (4) projects.
The series of performances on capitalism Free parties will survive, addresses the relationship to
money, property and this desire to always want more. It represents my hatred for this inhuman system for me, by
using and opposing it with hardcore sounds from the free parties of the 2000s, just as harsh, symbolising a
certain desire for revolt, freedom, anarchy. In my opinion, the capitalist system exacerbates and uses the
failings of man such as selfishness. Unlike Adam Smith, who relies in particular on this character trait to lay
down the main principles of economic liberalism, I can only believe in a system that promotes and encourages the
values of equity and sharing. In the same way as the composer Raphae ̈l Cendo, I try to resist this
capitalist system (5) by taking little interest in the production of a (musical) object for an object
or in the transmission of perception, but to the concrete way of realising a sharable object or transmitting an
experience. However, I consider that the discourse has little scope without a voluntary act accompanying the
thought. That is why my only way to deal with capitalism, was to offer improvisations from free and open source
creation and production tools.
I also have my contradictions and my development limits, in the sense that I also use commercial products,
especially with controllers or the Mac computer. However, I try to tend towards free, at least in terms of
software. Currently, I do not use Linux, because a lot of spatialisation systems I used were connected in
ethernet with Dante, which does offer drivers only for Mac and Windows. The use of Linux would then have
required the purchase of an expensive and bulky sound card. And, if I chose to use cheap and professional
commercial physical controllers, it is so that anyone can make the tool their own, without having to build their
own interfaces.
According to Jacques T. Godbout, “the work of art is a gift because the gift, [...] is what confirms to
one another that we are not things.”⁷ How then to express this gift in music, electronic performance
or digital art? Is showing or hearing an experience or a composition a sufficient form of expression and
sharing? How to transmit more and create stronger links with others, by making the barrier that separates the
researcher artist from the public fade and that the latter exchange their roles? Similar to Ursula M. Franklin,
who distinguishes prescriptive technology from holistic technology,⁸ my tool and technology practice aims
to focus not just on what is created and entertainment, but on the how and how the work is done though code
sharing and future training workshops or participative performance installations.
On the grave of Karl Marx, there is the following epitaph: “The philosophers have only interpreted the
world in various ways. The point however is to change it.” Although we cannot radically change the
world, I also expect from an artist a real desire for concrete action for a fairer world. The development,
promotion and almost exclusive use of open source software represented my utopian way of changing my world,
however small, like a little dust to stop the capitalist machine. However, by persevering in this radical
position, I have also excluded myself from the ”world”, because I no longer consider the profusion
of electronic productions made with commercial software, even if I may like them, because they do not
intrinsically transmit the idea of sharing and are not intended to put into questions dominant positions in
society.
Besides free and open source objectives, putting bites of his/others lifes in art is also essential for me.
What Alice Neel said to painting is applicable to every form of art:
“One of the reasons I painted was to catch life as it goes by, right hot off the griddle. Because besides
painting specific people, I’m painting all the happenings [...] and I’ve tried to capture the
zeitgeist. When painting or writing are good, it’s taken right out of life itself, to my mind, and put
into the work. Now, that doesn’t mean that the work has to tell about real life. I mean, it can be
abstract or anything. But the vitality is taken out of real living and put into the creative project, whatever
it is.”
The characteristics of the performance tool
Live 4 Life is not only a spatialisation tool, but a whole sound creation system to play with
sequences of parameters (rhythm, spectrum, space and effects), integrating spatialisation at different levels,
both at the heart of each sound synthesis and each loudspeaker. It is first thought for spatial performance with
several global, high-level (indirect) control strategies, e.g., by swinging among scenes of spatialised events
or by changing masses of parameters, particularly playback speeds with different mappings according to each
controller ⁹ (see figures 1, 2 and 3).
Dynamic, finite event-based spatialisation
Here, we propose a distinction between continuous input-based spatialiation vs finite event-based
spatialisation. By continuous input-based spatialisation, we mean that the user selects the position or
trajectory of each continuous input (like in most spatialisation plugins such as Ircam Spat). By finite
event-based spatialisation, we mean that the user selects (among a pre-defined library) the position and
trajectory for several layers of events with different times of beginning and durations (like in some creation
systems allowing to control both time and space of events, such as Live 4 Life or Sound Particles).
With the tool we focus on event-based spatialisation since it allows to control the temporality (rhythm,
density, duration and amplitude) of spatialised sound events, as shown in figure 4. According to the
author’s experience in creation, continuous input-based spatialisation tends to create more simple and
clear spatialisation, while event-based spatialisation tends to create more complex and confusing
spatialisation.
A library of trajectories and algorithms mixing abstract and concrete spatialisation techniques
It integrates a pre-defined library of multiple spatialisation models¹⁰ (see 5/ and 6/ of figure 5):
point granulation,¹¹ temporal micro-decorrelation,¹² trajectories drawn, generated
algorithmically or according to the spectral analysis of the sound signal ¹³, rendered according to
several spatialisation algorithms (equal-power panning, delay-based panning, Vector-Base Amplitude Panning
¹⁴).
We categorise two main kinds of spatialisation from the point of view of the composer.³ Abstract
spatialisation, which refers to a specific spatial trajectory or precise position applied to a sound, is present
in almost all spatialisation tools, possibly since it is direct and the most controllable and the easiest way to
abstractly conceive a position in space. Concrete spatialisation takes more into account the internal
characteristics or parameters of the sound, either with correlation by linking a spectral parameter or intensity
of the sound to a spatial dimension, or with micro/macro decorrelation—for example, on time, playback
speeds, transposition or distortion—which tends to create a diffuse/contrasted space. It is very difficult
to predict the spatial effects or diffuse perception of some concrete spatialisation techniques like
micro-delays or transpositions, where location is no longer the focus, rather than width and depth, as it
indirectly acts on space through parameters of the same sound different in some or all the channels of the
loudspeakers.
Algorithmic patterns of parameters mixing rhythm and space of sound events
Sequences of parameters allow to create sequences of spatialised sound events or choruses (copies or echoes of
the same event with micro-delays or spectral/spatial variations) that can meet a multichannel effect system with
different effect parameters on each loudspeaker.
Figure 5 shows the Sequence View to compose patterns or algorithms for all parameters (rhythm, density, sound
synthesis, space and send to effects). Time control is not represented with x-time axis, but with multisliders,
where you can draw a list of onset values or durations (see modules 1 and 3 of figure 5). Besides euclidean
algorithms controlling rhythm, a topographic drum sequencer from the Eurorack module Grids has also been
integrated into the tool thanks to the port in code to SuperCollider (6).
Figure 4 shows two views of a graphical representation with x-time axis of a sequence of events with different
parameters. The difference between the top/bottom representations is the density parameter (see module 2 of
figure 5), where all events are generated at the bottom and only some of them at the top. It can be obtained by
changing a slider, at the right of each parameter module, which multiplies the values of the multislider. The
sound events generated can then be each spatialised and routed to specific effects particularly thanks to
modules 5, 6 and 7 of figure 5.
Mixing object-based and channel-based paradigms
An object-based paradigm (sound sent to a virtual 3D position) involves that spatialisation can be easily
reproduced to other different spatial configurations, whereas a channel-based paradigm (sound or effect sent to
a physical loudspeaker position) involves that spatialisation cannot be reproduced in the exact same way on
different loudspeaker configurations.
According to the author’s experience, the simple treatment of channel-based paradigm of sending directly
a sound to a loudspeaker channel tends to reinforce the presence of loudspeakers and have a low CPU usage, which
has the effect of increasing the maximum number of spatialised events in real time, while object-based paradigms
tends to make forgive the presence of loudspeakers and be more CPU demanding, which has the effect of lowering
the maximum number of spatialised events in real time.
Despite the tendency towards object-based paradigm for reasons of standardisation and reproduction in every
space regardless of speaker setup, mixing channel-and object-based paradigms to be used on every sound event
allows to take advantage of the strengths of channel approach. Sending a sound directly to a specific
loudspeaker can have much more impact (and at least a different effect) than sending it at the exact same
coordinates of the loudspeaker through e.g., VBAP algorithm. However, it has also the effect that sequences
integrating channel-based or a multichannel effect system are different according to the number of loudspeakers
available and cannot be reproduced in the same way on different loudspeaker configurations and have to changed
with other spatial trajectories or algorithms. Depending on the spatial configuration, different spatial
algorithms are available. For example, in stereo, trajectory types are only limited to three (static panning
between links and right, one way or round-trip between links and right), as well as the panning algorithms
(equal power, delay-based and spectral).
The possible roads towards death
The tool Live 4 Life faces two main reasons that might cause its future end.
Technical death
As most Digital Musical Instruments, this tool is ephemeral due to technological obsolescence. Maintaining the
code with recent updates of MacOS or SuperCollider platforms and solving bugs is a considerable burden. Some
controllers have disappeared such as the Sensel Morph (7), the MIDI Touchbar (8) on older
Macs, where I could get sliders integrated with the keyboard. Other Applications like Lemur (9) on
iPad, which took me months of work to integrate within my tool, have temporarily stopped their development, and
the use of this application with next iOS updates is currently not guaranteed.
Social death
At its creation, the two motives behind the development of this tool were the realisation of spatial
improvisations in concert and a means to connect with others. The tool until now has lived because I had the
chance to perform in concerts. The pandemic has not helped, but the current difficulty to find residencies or
perform spatial improvisations in concert halls or festivals in good conditions may cause future death of this
tool: without an appropriate allowance and a minimum time of a few days to prepare and adapt the spatialisation
of works to the space and spatial setup of concert halls. Since concert halls with spatial setups are in great
demand, time for creation and repetition is often limited. Without a space with several loudspeakers and a user
to perform or the help from other developers/collaborators/users, this tool will die.
The possible roads towards symbiosis
Knowing the aforementioned risks, we now turn toward preventive solution avenues that will lead to a shared
tool. The spatial sound performance has many links with symbiotic individuations both between machines,
programs, works from other human beings (notably electroacoustic composers), the audiovisual domain and other
communities.
Symbiosis between Code and GUI and Controllers
To extend the possibilities of algorithmic patterns, I first thought to integrate besides SuperCollider the use
of other live coding languages, such as TidalCycles,¹⁵ by controlling pre-programmed code in real
time via DJ interfaces. This was realised thanks to a developer who implemented my request
(10) of being able to trigger via MIDI or OSC any parts of code in a text editor through the
encoder of a controller. However, first tests showed CPU and synchronisation issues when using both text editors
for SuperCollider with my tool and Atom for TidalCycles.
Symbiosis with other programs
As it is possible to see on the installation section of webpage of this tool, it is a patchwork of multiple
codes from various sources. To extend the possibilities of digital signal processing, integration of open source
or relatively free VST plugins is currently investigated, with plugins such as Surge XT (11), Vital
(12) or modular systems like VCV Rack (13). More precise spatial algorithms with Higher
Order Ambisonics will also be implemented in the tool via Sparta(14) or IEM plugins (15).
Symbiosis mixing electroacoustic works
I initiated just before the pandemic a new cycle of improvisations dealing with living alone and the feeling of
loneliness, with the absence of symbiosis, whose Greek root means "living together." My way to deal with the
hell of loneliness has been to symbolically ally myself and associate closely with other electroacoustic
composers (Bernard Parmegiani, Francis Dhomont and Robert Normandeau until now), reusing and recycling their own
sound materials and compositions. The willingness of this performance is to dissolve myself in the creation of
someone else, in the same way as I dissolve the composer in my tool, in order to destroy the ego and go towards
symbiosis. I, therefore, included in the spatial performance Parmegiani meets SuperCollider
(16) three major works from the electroacoustic composer Bernard Parmegiani, which are
rearranged, processed and triturated by the machine algorithms and my gestures.
The primary sonic source consists of 2-hour records from the electroacoustic composer Bernard Parmegiani, i.e.
De Natura Sonorum, Rêveries and Hell from Dante’s The Divine Comedy,
which have been cut in more than 1.700 sound samples and classified in dozens of folders. These sound files have
been cut so as to obtain and mix a wide variety of sounds of different durations, both isolated elements and
repeated a few times to have a brief musical phrase. The sound database has been grouped into fifty folders
differentiating between the original tracks and my own sound categories (short/long durations, soft/violent
elements, resembling instruments / natural elements or voice). Similarly to the cut-up technique, which was
popularized by writer William S. Burroughs, these sound files are then rearranged and spatialised in enveloping
vortexes around the audience. Their tempo and playback speeds are frenetically modified, thus accelerating or
slowing down the lives of large amounts of sound objects.
Audiovisual symbiosis
Audiovisual perception Although, by definition, acousmatic music allows us to focus on a
single sensory modality in order to better perceive sound information and free our imagination, multi-sensory
integration helps and has a strong impact not only on object discrimination and reaction time to events, but
also on emotional response, particularly in the peripersonal space.¹⁶ Perceptual and cognitive
mechanisms are made to process signals that are of multiple sensory nature. Given that an individual can no
longer integrate two sensory information and identify the source of the event when the spatial or temporal delay
is too great,¹⁶ it would be possible to play on joint or disjoint audiovisual spatialisations. In the
same way that certain studies, exploring the associations between sound and visual dimensions, suggest that a
sound parameter can correspond effectively to several visual parameters¹⁷ and especially that the
choice and sense of the scale correspondences would have more importance than the correspondence
itself,¹⁸ the same relationships and associations could be made between the sound, the visual and its
space, where multiple relations can coexist.
Audiovisual creations To meet the needs for audiovisual and more immersive works for the eyes,
an audiovisual object could be composed by transmitting via the OSC protocol the synthesis parameters of each
audio event from the Super-Collider language to audiovisual programming environments, such as Processing or Open
Frameworks, in order to represent each sound object with a virtual graphic object, a symbol, a video or an
image, whose position, playback speed, offset, transparency, movement or other video effects will be controlled
by the sound synthesis parameters. Sound images could be able to rotate like on record players depending on
their playback speeds. Audiovisual composition can also be used to generate symbolic scores, so that this
representation sheds light on the process of composition and improvisation.
First experiments with Processing are shown in figure 6. Simple shapes are triggered with OSC from sound
event data, distinguishing percussive sounds with cubes from others with spheres, accumulating by transparency.
Forms disappear by transparency and distance following the duration of sound events. Playback speed was mapped
to the vertical position, panning to horizontal position, intensity to the vertical size and transparency, the
duration to the horizontal size and transparency and the buffer folder to color. Streaks deforming forms can be
seen when triggering some shortcuts changing playback speeds of sound events. A similar patch has also been
developed in Touch Designer and will soon be available on Github.
Audiovisual collaborations Audiovisual collaborations are looked for both with video
developers to create the visual from the sound data generated, and with dancers to finally assess the impact
between the gesture of the performer, the dancing bodies and the video environment. Technically, the audiovisual
object mapping could be developed through open source tools, such as Processing, Open Frameworks, Hydra, or even
through commercial tools like Touch Designer and Resolume, as long as the creation process and the code are
published on Github and open to everyone.
Conclusion and future work
In this paper, we explained the free and open source motivations of the tool and its main characteristics of
event-based spatialisation. It will live until there is sufficient funding, a user/developer team, an
appropriate environment with spatial setups in concert halls and enough time to adapt pieces to concert space.
Otherwise, without collaborators and users, it will soon die. This current research-creation project in spatial
improvisation (sound and multi-sensory) has three axes:
• Thecreationofperformativeandparticipativeinstallations, where the public will be able to experience in
real time the control of the parameters of multiple sound objects in space via several interfaces. The sound
performance tool will thus be evaluated by the public, both from a perceptual point of view and in terms of
control and ergonomics.
• The establishment of training workshops to use and experiment free tools to create and improvise with
space,
• The creation of spatial improvisations in a multi-sensory context, combining and alternating music,
video / light and dance.
Acknowledgements
This postdoctoral research and further development of these tools were made possible thanks to research grants
from FRQSC (Fonds de Recherche du Québec - Société et Culture/ B5 - 318691) and Hexagram-UQAM
(1) https://github.com/Xon77/Live4Life
(2) https://www.fredrikolofsson.com/code
(3) https://chdh.net/egregore source.php
(4) http://chdh.net/vivarium.php
(5) See the interview “Resist and Transcend” from Raphae ̈l Cendo:
https://vimeo.com/240664490, accessed September 15, 2023
(6) https://github.com/capital-G/sc-grids
(7) https://morph.sensel.com, accessed September 15, 2023
1 R. Penha, J. P. Oliveira, Spatium - tools for sound spatialization, In Sound and Music
Computing Conference, Stockholm, Sweden, 2013, 660–667.
2 Thibaut Carpentier, Markus Noisternig, Olivier Warusfel, Twenty Years of Ircam Spat: Looking
Back, Looking Forward, Denton, USA, 2015, 270–277.
3 Christophe Lengelé, Live 4 Life - A spatial performance tool focused on rhythm and
parameter loops, In International Computer Music Conference, Daegu, South Korea, 2018, 298– 303.
4 Francis Dhomont, Abstraction et figuration dans ma musique... et autres considérations,
eContact, 16(3), 2013.
5 Francis Dhomont, Abstraction et figuration dans ma musique, In Lecture by Francis Dhomont as
part of the colloquium for graduate studies in composition at Université de Montréal,
Montréal, Canada, 2018.
6 James Mc Cartney, Rethinking the computer music language: Supercollider, Computer Music Journal,
26(4):61–68, 2002.
7 Jacques T. Godbout, Ce qui circule entre nous. Donner, recevoir, rendre, Paris, France,
Éditions du Seuil, 2007.
8 Ursula M. Franklin, The real world of technology, Toronto, Canada, House of Anansi Press Inc.,
2004.
9 Christophe Lengelé, Live4Life: Aspatial performance tool to play the ephemeral and
improvise with space and playback speeds, Organised Sound, 26(1):89–99, 2021.
10 Christophe Lengelé. The story and the insides of a spatial performance tool: Live 4 Life.
In International Computer Music Conference, Santiago, Chili, 2021, 133–139.
11 Scott Wilson, Spatial swarm granulation, In International Computer Music Conference, Belfast,
Irlande, 2008.
12 Horacio Vaggione, Décorrélation microtemporelle, morphologies et figurations
spatiales du son musical, In Anne Sédès, editor, Espaces sonores, Paris, France, Éditions
musicales transatlantiques, 2002, 17– 29.
13 Laurent Pottier, Le contrôèle de la spatialisation. In Laurent Pottier (ed.), La
spatialisation des musiques électroniques, Saint-Etienne, France, Publications de
l’Université de Saint-Etienne, 2012, 81–104.
14 Ville Pulkki, Virtual sound source positioning using Vector Base Amplitude Panning, Journal of
the Audio Engineering Society, 45(6):456–466, 1997.
15 Alex Mc Lean, Algorithmic Pattern, In Conference on New Interfaces for Musical Expression,
Birmingham, UK, 2020, 265–270.
16 Isabelle Viaud-Delmon, Conférence : les aspects cognitifs de la perception 3D, Forum
International du Son Multicanal, 2014.
17 Scott D. Lipscomb, Eugene M. Kim, Perceived match between visual parameters and auditory
correlates: an experimental multimedia investigation, In International Conference on Music Perception and
Cognition, Sapporo, Japan, 2008, 72–75.
18 Florent Berthaut, Myriam Desainte-Catherine, Martin Hachet, Combining audiovisual mappings for
3D musical interactions, In International Computer Music Conference, New York, USA, 2010, 357–364.
Authors Biographies
Christophe Lengelé is a spatial sound designer and performer for electronic and
experimental music. After a Doctorate in music (composition and sound design) at Université de
Montréal in 2022, he is currently doing a postdoctoral research/creation in spatial improvisation at
UQAM (Canada). He seeks to bring together the spheres of composition and improvisation and focus on performing
variable spatio-temporal pieces with a global custom live tool, he has been regularly developing since 2011,
in order to play the place and the music at the same time. His works have been presented and performed
internationally, particularly at the International Computer Music Conference (2018, 2021), during the
Journées d’Informatique Musicale in France (2017, 2019), as well as at Akousma Festival 2021
(Canada) and more recently at Cube Fest 2022 (USA).
Philippe-Aubert Gauthier received a B.Ing. degree in 2000 and a M.Sc. degree in active
control of vibration in 2003 (Université Laval, Canada). From 2003, Mr. Gauthier was enrolled at
Université de Sherbrooke (Canada) from which he received a Ph.D. degree in 2007, dedicated to adaptive
wave field synthesis. He is interested in spatial sound, multichannel sound reproduction, audio signal
processing, room acoustics, loudspeaker and microphone arrays, spatial hearing and psychoacoustics. He was
adjoint professor at the Mechanical Engineering Department from 2017 to 2019. He then took up a professorship
at the École des arts visuels et médiatiques at UQAM (Canada) in 2019. He is a professional sound
artist, an electronic musician, working at the intersection of arts, sciences and technologies. He is
currently the associate director of artistic research at CIRMMT (Centre for Interdisciplinary Research in
Music, Media, and Technology).
A Latin America Network for Art and Cybernetics: The Centre for Art and Communication (CAyC)
Jose-CarlosMariategui
jcm@ata.org.pe
Abstract
Towards the end of the 1960s—a period of intense creative, technological and political changes—the
Argentinian art critic and entrepreneur Jorge Glusberg founded the CAyC in Buenos Aires. CAyC was an
interdisciplinary experimental project that explored the relationship between art, technology and society. It
sought to articulate a network of discussions and productions by a new style of Latin American artist, deeply
influenced by science, technology and society. Glusberg defined such practice as Systems Art, which appeared in
three ways, namely as a system of collective representation; a system of meaning that defied formal categories;
and a system of relationships and processes for social inquiry. In doing so, the artist became a researcher who
reflected on their social context and the latter’s processes of production. This paper will discuss
CAyC’s pioneering work and its global influence through three main initiatives: its exhibitions Art and
Cybernetics, Systems Art in Latin America and the International Open Encounters on Video. These events were
driven by the revolutionary artistic and experimental promotion of the distinctive ways in which Latin American
artists were using technology to respond to local issues at a time when computer systems and cybernetic models
for management and organizational practices were being introduced across the region.
Keywords
Systems art, Latin America, Cybernetics, Video art.
Towards the end of the 1960s, art critic and entrepreneur Jorge Glusberg created the Centro de Arte y
Comunicación (Centre for Art and Communication) in Buenos Aires. The Centro de Arte y Comunicación,
popularly known by its acronym CAyC, fostered the intermingling of disciplines, with communication and
cybernetics at its heart. Glusberg conceived CAyC as an interdisciplinary and experimental art centre which
explored the relationship between art, science, technology and social studies. It sought to articulate a network
of Latin American artists, critics, intellectuals, scientists, and other cultural practitioners around
production, debates and exhibitions. As described in an early 1970s diptych promoting CAyC’s activities
(1):
The Centre for Art and Communication aims to promote the execution of projects and experiments where art,
technological media and the interests of the community are brought together in an effective exchange that puts
in evidence the new unity of art, science and the social environment in which we live. Its fundamental
objectives are to encourage, support and develop those social tasks, experimental studies or research in the
arts and group communication, which give rise to an interdisciplinary integration, to improve and expand the
current scenario of human concerns. It is formed by artists, sociologists, logicians, mathematicians, art
critics and psychologists pursuing common goals with a highlight on the behaviour and development of phenomena
of mass communication and the collapse of traditional forms of practice, create an opening for new systems of
expression, where researchers and artists try to outline the artistic interests of the man of the XXI century.
CAyC showcased internationally a new regional genre of art which Glusberg defined as systems art
identifying it with post-object art practices and experiments that were developed at that time in an
international arena, deeply influenced by science and alternative uses of communications, mass media and
technology.
In what follows, I will address the particular modes and practices of CAyC as it developed what was quite
possibly the first art and technology network in Latin American reaching international visibility. The following
three sections outline some of CAyC’s historical milestones: its founding approaches as encompassed in the
show Art and Cybernetics which brought together engineers and artists who explored the creative
possibilities of computers, as well as the exhibition Systems Art in Latin America, which circulated
internationally and created an original mode of circulation. We also provide an interpretation of the nature of
CAyC’s practices that seem to be the outcome of processes of production and distribution of standardized
and reproducible content. We also explore how CAyC articulated its network as a vital component of its approach
to systems and cybernetics in Latin America’s political and social context. Finally, we briefly share some
conclusions based on our findings and give some directions for further research.
The establishment of the CAyC: Art and Cybernetics
Following the Second World War, the field of cybernetics gained scientific and intellectual recognition. In the
1960s, an outburst of social and artistic interdisciplinary movements and research practices prompted ephemeral
events involving the audience (the observer) as an active participant, allowing artists to question their own
practice and its impact on society. What emerged as a consequence was an engagement with post-object practices
concerned with creating an awareness of the real as a relational and dynamic network of processes that have a
political and social impact. ¹
In 1968, art critic and entrepreneur Jorge Glusberg founded the Centre of Studies on Art and Communication of
the Foundation of Interdisciplinary Research (CEAC is the Spanish acronym) (2). On 18th August 1969,
the CEAC opened its first exhibition, Art and Cybernetics, at the Bonino
Gallery(3),2 inspired by a visit Glusberg made to Japan in February 1969, where he
established contact with Tokyo’s Computer Technique Group (4) (CTG), a collective of art and
engineering students founded by Masao Kohmura and Haruki Tsuchiya in 1966 and partially funded by the IBM
Scientific Data Centre in Tokyo. CTG was both a design office and a think tank focused on computer analysis and
art practice based on image processing and geometric transformations. Glusberg later said that CTG:
“Combined their traditional graphic sensibility with the technique and rigor of the scientific star of the
moment: the computer.”³ Initially, Glusberg thought only of presenting CTG in an exhibition in Buenos
Aires as he brought from his trip a series of works from the Tokyo-based artist group. However, Glusberg set up
also a research group on arts and cybernetics (Grupo de Arte y Cibernetica) and along with IT engineers Julio
Guibourg (Director of the Computation Centre at the ORT School) and Ricardo Ferraro (Professor of Policy and
Management of Science and Technology at Buenos Aires University), organized a seminar which gave artists access
to computers and the chance to explore their creative possibilities (5). The foundational group was
comprised of Luis Fernando Benedit, Antonio Berni, Eduardo Mac Entyre, Osvaldo Romberg, and Miguel Ángel
Vidal. Further members included artists Ernesto Deira, Hugo Demarco, Gregorio Dujovny, Mario Mariño,
Rogelio Polesello, Isaias Nougués, Josefina Robirosa and Norma Tamburini. The seminar included other
scientific researchers such as Arturo F. Montagú—a pioneer in architectural computing–; Pablo
Jonovich; César Armoza; Silvia L. Yacub and Silvia Brucciamonti.⁴, ⁵,
⁶ Ferraro was in charge of the IBM 1130(6) Computing System Model 2C (16-bit binary
architecture with a core memory of 16,384 words and a memory cycle time of 3.6 μs) and two automatic drawing
machines or “plotters” (IBM 1627 Plotter) equipped with ball point pens which converted tabulated
digital information into a graphic format. The software was running on Fortran IV, the main computer language
for the IBM 1130. The exhibition showcased “living art, created by innumerable pioneers of our time, who
use ideas, synthetic shapes or mathematical equations instead of paintings; lights and motors, and information
instead of brushes.”⁷ From the artists participating in the seminar, only Berni, Benedit, Deira, Mac
Entyre, Romberg and Vidal exhibited their works and demonstrated “their courage as pioneers in the
complicated jungle of plotters, equations, matrices and perforated bands.”⁸ Their works were screen
prints produced by the plotters based on computer generated drawings of polynomial interpolations, repetitions,
geometric patterns and other mathematical designs and shapes (triangles, squares, rectangles) using mathematical
properties. sThe exhibition also included twenty works by members of CTG (Koju Fujino, Takeshi Hasegawa, Junichi
Kakizaki, Masao Komura, Fujio Niwa, Makoto Ohtake, Haruki Tsuchiya and Kunio Yamanaka), six works from Motif
Editions, London (works from CTG, M. S. Mason, Boeing Computer Graphic, Kerry Strand, Csuri and J. Shafffer and
D. K. Robins) (7) and featured electronic music by five local composers: Dante Grela, Francisco
Kröpfl, Carlos Rausch, Jorge Rotter and Eduardo Tejeda. As Glusberg stated in the exhibition catalogue of
Art and Cybernetics:
One of the most important characteristics of cybernetics is its universality (reflecting) the possibilities of
this new scientific discipline for the comprehension of most heterogeneous phenomena. The development of some
cybernetic models permitted the building of artificial systems, which less than ten years ago were distinctive
and exclusive traits of human activity. The artists of this time are more interested in behaviour than in the
essence of things; this tendency can be clearly identified with cybernetic vision.⁹
The intention of the project was to foster a multidisciplinary community among engineers, scientists,
philosophers, theorists, artists and architects as well as to support and give visibility to a group of artists
interested in practices combining science and technology which were already starting to become prevalent
internationally. Glusberg understood the universalistic character of cybernetics and wanted to emphasize that
its application in artistic practice enabled new interpretations that could act as active forces for
understanding human behaviour and furthermore to question and change society, in particular as he later
mentioned, “as a new form that emerges as a result of a regional problematic, one that uses a methodology
common to different contexts.”¹⁰
During the exhibition, workshops on “automatic drawing techniques” (Ricardo Ferraro, Nestor
Sameghiniy and Marcelo Larramendy) and the use of computation and plotters (Arturo Montagu, Ricardo Valek, Pablo
Jononovich, Cesar E. Armoza and Silvia L. Yacub) were organized. In addition, a Seminar on Information and
Approach was held with lectures by, among others, Manuel Sadosky, an influential mathematician and
computer pioneer who led the implementation of the first computer for research and education in Argentina, a
Ferranti Mercury nicknamed Clementina. (8),¹¹
The Art and Cybernetics exhibition was the first of its kind in Latin America and took place only one
year after seminal exhibitions such as Cybernetic Serendipity at the Institute of Contemporary Art
(ICA) in London (9) and Some More Beginnings: An Exhibition of Submitted Works Involving
Technical Materials and Processes (10) at the Brooklyn Museum in New York. It took place
in the same year as other influential exhibitions such as The Machine as Seen at the End of the Mechanical
Age (11) at the MoMA, in New York, Mind Extenders in Minneapolis (12)
and Tendencies 4 (13), at the Gallery of Contemporary Art in Zagreb; and, a year
before Information (14) at the MoMA in New York and Software: Information
Technology: Its New Meaning for Art (15) at the Jewish Museum in New York. Cybernetic
Serendipity was one of Glusberg’s main inspirations as it combined computer-generated graphics and
computer music along with cybernetic and motorised sculptures and manipulated television sets. Glusberg had
established correspondence with Reichard in 1967 and in 1969 he asked for authorization to reproduce the
recordings of Cybernetic Serendipity for a show at the CAyC (16). It is well documented that
Glusberg invited Reichard to Buenos Aires several times. She finally visited in November 1970 to give two talks
(17) at the CAyC18. Prior to her arrival in Argentina, the newspaper ACCION published an
article by Reichard titled “Cibernética Artística” (“Cybernetic Art”)
(19). At the time, Reichardt was very interested in Latin American art and culture, particularly the
work of Argentine artist Leopoldo Maler and writer Jorge Luis Borges (20).
After Buenos Aires, Art and Cybernetics travelled to Córdoba, Santa Fe and Tucumán in
Argentina and then to Montevideo (1970) (21) and Lima (1971) (22), as well as to
Minneapolis, San Francisco, New York, London and Tokyo. ¹² Glusberg realized that exhibitions based on
reproducible media—such as computer printouts—permitted an efficient and inexpensive diffusion
model, which he used later in other traveling shows.
In 1970, CAyC opened its iconic headquarters at 452 Viamonte Street in Buenos Aires with an entrance
reminiscent of a techno-futuristic spaceship or the shape of punched cards used in computer systems.
CAyC—which was independently funded by Glusberg—organised exhibitions and events welcoming local as
well as prominent international artists, theorists and critics. As we will see, it became an active player for
the production and diffusion of new and innovative technology-driven art forms.
Systems Art in Latin America
The dismantling of cultural activities and the human rights abuses that began with the military coup in 1966
caused many artists to leave the country. In this context CAyC provided an important refuge for artists by
actively creating a multidisciplinary space for reflections around art, technology, communication and society.
As we mentioned earlier, CAyC sought to showcase a new regional artform, which Glusberg defined—following
the work of Jack Burnham—as “systems art,” encompassing a wide-ranging selection of works that
can be loosely articulated within such a definition. After Art and Cybernetics, CAyC’s second
exhibition was titled De la Figuración al Arte de Sistemas (From Figuration to Systems Art)
(23). Organized at the Museo Provincial de Bellas Artes Emilio A. Caraffa in
Córdoba in August 1970, it showcased works from Nicolás García Uriburu, Edgardo Antonio Vigo,
and Luis Benedit. However, as art researcher Mari Carmen Ramirez stated,
Despite the innovative nature of their work and Glusberg’s vague efforts to justify their inclusion in
the exhibition, there was little in Uriburu’s or Vigo’s art that could be considered systems art
[...] Their somewhat more modest proposals could not compete on an equal footing with the conceptual and
technological sophistication of Biotrón, Fitotrón, or any of Benedit’s
living systems.¹³
Benedit, an architect and self-taught artist and designer, was working with living habitats, installations,
illustrations and technical construction drawings linked to ecology and analytical investigations.
Benedit’s Biotron (24)was an impressive environment or “living
habitat” of four thousand bees, encouraging the audience to learn while being active observers (here the
observer is an indissoluble part of the system). Benedit studied the behaviour of bees, among other animals, to
produce artificial and rational systems of organization and observation.¹⁴ In fact, Benedit’s
Biotron should be understood as a work that reflected on both the cybernetics and systems theory
fields, whereas the living system environment is a complex of interacting components set towards specific goals,
while observation as an event produces distinctions which inform the system about a difference.¹⁵
Based on systems theory, the living systems of Benedit help to construct a framework of diverse phenomena that
are intrinsically dynamic within organisms, social systems and ecosystems.
On the 19th of July 1971, a third CAyC exhibition called Arte de Sistemas (Systems Art) was presented
at the Museo de Arte Moderno de Buenos. Artists from Latin America, the USA, Europe and Japan were invited,
including some prominent artists such as Vitto Aconcci, Dan Graham, Hans Haacke, Allan Kaprow, Joseph Kosuth and
Dennis Oppenheim. Perhaps the most relevant aspect of this exhibition was its original format and scale. It
expanded to include 101 experimental artists who submitted their works in a format that shaped many of the
exhibitions of CAyC between 1970 and 1975, namely a single format on blueprint paper, IRAM (25)
coded, 59.5 x 84 cm, responding to the idea of creating a standardized system where all participants would
participate on an equal footing. The work submitted by each artist could be reproduced from the heliographic
format, for different versions of the exhibition in other countries. Glusberg understood that production of a
common scheme on a homogeneous and easily reproducible medium allowed his enterprise to travel across a small
circuit of museums, galleries, schools, or even neighborhood organizations.¹⁶
In May 1972, the CAyC traveling exhibition Towards a profile of Latin American art, first presented at
the III Coltejer Biennial, in Medellín (Colombia), incorporated the idea of system with respect to the
notion of politics and ideology, particularly in Latin America. The exhibition’s presentation text evinces
such reflection:
The conflicts generated by the unfair social relations that prevail in the Latin American countries cannot fail
to appear in this facet of cultural life. [...] Our artists became aware of the requirements of their national
realities and thus proposed regional responses that are consistent with the change in all areas of human life
that affect the underprivileged of today, those who we think are the potentially privileged of tomorrow.
¹⁷
The exhibition Art Systems in Latin America circulated between 1974 and 1976, expanding the original
mode of circulation of standardized blueprints as well as adding to it an ideological framework. In the
introduction to Art Systems in Latin America, Glusberg demarcates the cultural process in Latin America
and how it is different from those of the “first world.”¹⁸ His criticism is associated
with the domination mechanisms that prevailed at the time when terms such as “Third World” or
“underdevelopment” were coined as part of support policies, but also as new mechanisms of
post-colonial domination, generally referenced under the label of “ideological
conceptualism.”¹⁹ Like many at that time, he criticizes the “grouping” of artists
for their social characterstics in “the same network.” Thus, he proposes the development model in
Latin America to be an alternative development, with a commitment to a new art that “tries to break away
from the instruments of ideological domination of the countries that have the concentration of power and wealth,
sometimes taking advantage of their own methodologies and languages.”²⁰
The point here is that the set of works proposes different “liberation strategies” through
operational models that define political and social contexts, and to articulate a proposition for change from an
artistic and cultural perspective. In that sense, the works that were presented do not represent a particular
Latin American content, demystifying a free group, except the group fighting for their liberation.
It is in this show that Víctor Grippo presented his famous work Analogía I (Analogy I)
(1970-1971), in which an analogue voltmeter measures the amount of energy contained in a
“system” of forty freshly grown potatoes, all interconnected in series and parallel electrical
circuits. The electrodes connecting each potato at the end of the wires are made of different metals—one
copper, the other zinc-galvanised iron—which reciprocally balanced processes of oxidization and reduction
occurring in the electrodes producing an electrical current displayed in the voltmeter. The work displays the
potato, which is of Latin American origin, to showcase its contribution to the world as a reservoir of energy,
but also to explore its symbolic value through analogies with knowledge and consciousness.²¹
Interconnecting the potatoes depicted how the tiny voltage of each potato multiplied; such combined energy
also became a metaphor for the power of collective human consciousness and its transformative potential for
liberation.²²
However, having each potato in a small cell resembled the imprisonment and brutality occurring in Argentina at
that time, as Justo Pastor Mellado comments “the electrode brought up images of torture, and in
particular, of electrical current applied to the human body. The potatoes were reminders of an inert condition,
in an ambiguous way.”²³ The wires act to both connect each potato together and register the
amount of energy generated while the voltmeter represents connectedness and community expressed through science.
The piece was operated through a button that the public could use to read the amount of energy being generated
by the wired spuds, giving the audience the opportunity to become actively involved with the work as it was
being transformed into a pedagogical exercise that revealed a non-traditional source of energy. The piece also
includes a small text contrasting the word papa (potato in the Andean Quichua language) with the word
conscience, proclaiming the search for knowledge and a desire to convey the possibilities and the
processes of transformation. A trained chemist, Grippo joined the CAyC group and began work on projects that
explored relations such as nature-culture and real-artificial. By using food, minerals and domestic objects, he
tried to conceptually depict the inconsistencies and abuses of natural resources, as well as the relation
between nature and humanity.
The works of Benedit and Grippo could be seen as pioneering experiences at that time, and within the new Latin
American systems art they contained principles from cybernetics, such as the importance of the observer as well
as the establishment of information mechanisms which produce new immaterial symbology amenable to
socio-political interpretations. In addition, as Joanna Page suggests, these works were a significant
contribution to decolonizing science and nature, expanding the conception of knowledge, in which cognition is a
thoroughly sensory and embodied affair, reflecting the fact that our knowledge of the world is not a knowledge
of something outside or beyond us, but of something in which we are always already entangled.²⁴
CAyC’s approach to Cybernetics
CAyC pioneered a vehicle for the diffusion of a new art practice in Latin America based on ideas around
cybernetics, systems art and video art, with a privileged focus on processes and immaterial information over
more traditional objectual practices. CAyC acted as more than a diffusion agent, since it enabled the creation
of original projects not only in Argentina, but internationally. CAyC built a network of Latin American artists
visible on the international stage, disrupting traditional artistic notions that typified Latin American art
based on its ancestral or historical cultural practices.
While CAyC would ultimately adopt “systems” as a catchphrase to describe all sorts of conceptual
art practices and most of the projects organized during the 1970s, its discourse progressively became more
critical towards the political scene in Latin America at a time when freedom was very limited by several
military dictatorships throughout the region, that suppressed the revolutionary ideas of the previous decade.
Such a confrontational position was a necessary demand from artists throughout Latin America; CAyC, as Glusberg
mentioned “had become aware of what their national realities demanded and had offered responses to
regional issues.”²⁵ CAyC’s strategy helped to consolidate and legitimize Latin
American art and their critical involvement in international tendencies, acting as a catalyst that allowed
visibility.²⁶ Thus, cybernetics and systems theory “framed” the art pieces as a networked
process with the goal of entering the international art stage.²⁷
Even though in some cases Glusberg borrowed concepts from systems and communication theories in a loose manner,
the exhibitions organised by CAyC paid attention to concepts such as relational, autonomy and open-endedness
which were fundamental to systems theory, and introduced to the artistic community terms such as systems art,
art and technology, and video art. Systems art was a category linked to technology, processes and social
dynamics but, as Rodrigo Alonso accounts:
"The Glusbergian vision is much more imbued with the socio-political context of its time. In a country governed
by a military dictatorship, but still moved by revolutionary ideas and by the desire for a return to democracy,
artistic production had to account, in some way, for the complexity of this unique historical
moment.”²⁸
However, as Julieta Gonzalez points out, The concept of information as power was fundamental for CAyC, which,
though initially embracing the constructive-semantic imperatives of the kinetic, computational model of the
early 1960s, eventually departed toward a radicalized agenda of direct action and intervention as dictatorial
regimes spread throughout South America.”²⁹
Starting from the Art and Cybernetics exhibition, which was the first of its kind in Latin America, we
can study the genesis of art and technology and its accompanying network of artists. Within the loose definition
of cybernetic art and systems art, it brought together a diversity of artists and creators that were not able to
find a space in the traditional art world. CAyC supported and promoted a group of artists and thinkers who are
now considered pioneers in art, video and cybernetic practices such as Luis F. Benedit and Victor Grippo
(Argentina), Margarita Damico (Venezuela) or Pola Weiss (Mexico). In addition, the CAyC used the reproducible
character of heliographs and video, as well as “DIY installations”, to respond with inexpensive
exhibitions in a time when art and technology shows were known to be expensive and complex, requiring both
corporate sponsorship and technological support.
Most well-known and emblematic large-scale exhibitions on art and technology during that time were quite
expensive and very difficult to tour. Jack Burnham went further by decrying that, after several exhibition
attempts, art and technology seem to prove mutually exclusive. For instance, the Cybernetic Serendipity
show, which had a modest budget with no on-site computers or terminals, when it was shipped to the Corcoran
Gallery of Art in Washington D.C. the following year, had a considerable portion of its contents destroyed
because of poor packing and handling and its curator, Jasia Reichardt, publicly disowned what was shown
there.³⁰ Severe maintenance costs of expensive equipment entailed operational difficulties. In that
sense, the “CAyC exhibition model” proved a viable way to show technological works in a moment where
the technology was complex, expensive and not widely available, thanks to the former’s flexibility and
cost efficiency. Its outcome was demonstrated by the extensive network of relations and collaborators in more
than 40 cities nationally and internationally, demonstrating CAyC was capable of producing a rhizomatic network.
More importantly, the CAyC attempted to create a Latin American movement, in which the use of
“foreign” technologies—seen as dominant and difficult to use—could provide autonomy,
creative liberation and opportunities for development while fostering civic participation and collaboration, two
concepts that were to become the foundation of modern computing and the internet.
Final Remarks
We argue that CAyC’s approach to systems and cybernetics contributed to the creation of a group of
practitioners from different disciplines and backgrounds which supported the innovative undertakings of
non-objectual practice in Latin America. CAyC’s work is distinctive of its time, as a transition from an
object-oriented to a systems-oriented culture, shifting art from material entities towards non-objectual
representations. This also enabled an art practice that was much more autonomous and collaborative. The
innovative model conceived by CAyC forged a model for social production both in terms of a platform for
sustainability and support for more innovative and experimental endeavors. Such endeavors were based on
conditions that promote disciplinary models of production through active experimentation and facilitating the
interaction between different types of practitioners, such as architects, designers, scientists and artists who
joined forces to configure a new Latin American art.
(1) CAyC information diptych, Rodrigo Alonso Archive, Buenos Aires, (1971?).
(2) Centro de Estudios de Arte y Comunicación de la Fundación de Investigación
Interdisciplinaria.
(3) At that time directed by Guillermo Whitelow.
(4) Their works were shown at Cybernetic Serendipity (1968) as well as in the 6th Paris Biennale
(1968) and the 35th Venice Biennale (1970).
(5) In Buenos Aires, between March and April 1969.
(6) For more information on the IBM 1130, http://ibm1130.org/, The IBM 1130 was also used for the
first experiments on graphical user interfaces developed by Alan Kay in his Ph.D, Thesis in University of Utah
in 1969.
(7) These group also created the Computer Arts Society in 1968.
(8) Installed in 1961 at the Computational Institute (Instituto de Cálculo) of the University
of Buenos Aires.
(9) Cybernetic Serendipity, an exhibition of cybernetic art curated by Jasia Reichardt, held at the
ICA, London, 1st August to 20th October 1968, later touring the United States, Corcoran Gallery of Art,
Washington, D.C., and the Exploratorium in San Francisco.
(10) Some More Beginnings: An Exhibition of Submitted Works Involving Technical Materials and
Processes at the Brooklyn Museum, New York organized by Experiments in Art and Technology (EAT), 25th November
1968 through 5th January 1969.
(11) The Machine as Seen at the End of the Mechanical Age curated/directed by Pontus Hulten,
Director of Moderna Museet, Stockholm, an exhibition of more than 200 works of art and related objects held at
The Museum of Modern Art, New York, from 27th November to 9th February 1969.
(12) Mind Extenders curated by Mimi Shorr, held at the Museum of Contemporary Crafts of the American
Craftsmen' s Council, Minneapolis, from 19th April to 15th June 1969.
(13) Tendencies 4 organized by the New Tendencies movement, Galerija suvremene umjetnosti (Gallery
of Contemporary Art, now the Museum of Contemporary Art Zagreb) in Zagreb, from 5th May to 30th August 1969.
(14) Information, an exhibition curated by Kynaston L. McShine, held at the MoMA, New York, 2nd July
- 20th September 1970.
(15) Software - Information Technology: Its New Meaning for Art
curated by Jack Burnham, held at the Jewish Museum in Brooklyn, New York City, from 16th September to 8th
November 1970, and the Smithsonian Institution, Washington, D.C., from 16th December 1970 to 14th February 1971
(16) Jasia Reichardt. Personal correspondence with Jorge Glusberg. 20th April 1969. Jasia Reichardt
personal archive, London.
(17) The titles of these two presentations were: Pop Art and Towards a Future of Art. Jasia
Reichardt personal archive, London.
(18) Jasia Reichardt. Personal communication with the author. 19th Jan 2020.
(19) ACCION, 3rd July 1970. Jasia Reichardt personal file, London.
(20) Reichardt met Borges during her first visit to Argentina and invited him to London, where he
delivered four lectures (“Four evenings with Borges”) in Central Hall in 1971organised by the ICA.
Jasia Reichardt. Personal communication; 19th Jan 2020.
(21) In June 1970, the exhibition travelled, under the auspices of the National Commission of Fine
Arts of Uruguay to the National Museo of Fine Arts in Montevideo, directed by Angel Kalenberg. The catalogue was
prepared in cooperation with IBM Uruguay S.A.
(22) The exhibition was presented at the IAC (Instituto de Arte Contemporáneo) on the 19th
October 1971 with the support of IBM Perú, and a panel on Arts and Cybernetics was held on the 27th
October. Correspondence between Alfonso Castrillón and Glusberg dated 31st August 1970 discussed the plan
to organize a Symposium on Art and Industry; however, that idea apparently never materialized (Alfonso
Castrillón personal archive, Lima).
(23) A few months later, in February 1971, CAyC organized its second exhibition in London’s
Camden Arts Centre under the same title, but this time it included a larger number of artists.
(24) The Biotron installation was presented at the 35th Venice Biennale, Venice, 1970.
(25) IRAM code was standard criteria defined by the Argentine Normalization and Certification
Institute, the International Organization for Standardization (ISO) member body for Argentina.
References
1 Ionit Behar, “Arte de sistemas: conceptual art and politics in Argentina (2015)”,
Henrique Faria website, accessed July 29, 2021,
http://www.henriquefaria.com/content/uploads/news-press/pdf/artslant_-_arte_de_sistemas_conceptual_art_and_politics_in_argentina-843.pdf
2 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y
Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la
Galería Bonino de Buenos Aires, Buenos Aires, CEAC, 1969, ⁸.
3 Jorge Glusberg, Del pop-art a la nueva imagen, Buenos Aires, Ediciones de Arte
Gaglianone, Coleccion Union Carbide, 1988.
4 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y
Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la
Galería Bonino de Buenos Aires, ⁸.
5 Melanie Lenz, “Centro de arte y comunicación (CAyC),” Word and Image /
V&A Blog, July 8, 2016, accessed August 3, 2023,
https://www.vam.ac.uk/blog/caring-for-our-collections/centro-de-arte-y-comunicacion-cayc
6 Melanie Lenz, “Early argentine computer art at the Victoria & Albert Museum,”
Journal of Design History, Vol.31, No. 02, 2018, 154-166.
7 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y
Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria
presentada en la Galería Bonino de Buenos Aires, ¹⁰.
8 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y
Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la
Galería Bonino de Buenos Aires, ²².
9 Jorge Glusberg, Arte y cibernética. Primera muestra del Centro de Estudios de Arte y
Comunicación (CEAC) de la Fundación de Investigación Interdisciplinaria presentada en la
Galería Bonino de Buenos Aires, ⁸
10 Jorge Glusberg, Arte e ideología en CAyC al aire libre, Buenos Aires, CAyC,
1972, 1.
11 Cecilia Berdichevsky, “The beginning of computer science in Argentina — Clementina
- (1961–1966),” in History of Computing and Education 2, ed. J. Impagliazzo, New York,
IFIP Advances in Information and Communication Technology, vol 215. Springer, 2006, 203-215.
https://doi.org/10.1007/978-0-387-34741-7_15
12 Jorge Glusberg, “La Argentina entre los países pioneros en arte y
cibernética,” Ámbito, February 20, 2007, accessed August 10, 2023,
https://www.ambito.com/espectaculos/la-argentina-los-paises-pioneros-arte-y-cibernetica-n3419280
13 Mari Carmen Ramírez, “A paradoxical notion of image: Luis Fernando Benedit’s
́Living Systems ́,” in Benedit: works 1968- 1978, ed. María Torres, Buenos
Aires, Fundación Espigas, 2020, 48.
14 Mari Carmen Ramírez, “A paradoxical notion of image: Luis Fernando Benedit’s
́Living Systems ́,” 45.
15 Niklas Luhmann, Social systems, Stanford: Stanford University Press, 1995.
16 Jorge Glusberg, “Artes y espectáculos: definiciones y cuestionamientos,”
Primera Plana 10, 1972, 50–51.
17 Jorge Glusberg, Hacia un perfil del arte latinoamericano. Encuentro Internacional de Arte
en Pamplona, España. CAYC, Buenos Aires, CAyC Centro de Arte y Comunicación, 1972.
18 Jorge Glusberg, ed., Art systems in Latin America: Institute of Contemporary Arts, London:
Center of Art and Communication, Buenos Aires, London and Buenos Aires, Institute of Contemporary arts
and CAyC Centro de Arte y Comunicación, 1974, ⁷.
19 Simón Marchán Fiz, Del arte objetual al arte de concepto, Madrid, Akal,
2012.
20 Jorge Glusberg, ed., Art systems in Latin America: Institute of Contemporary Arts, London:
Center of Art and Communication, Buenos Aires, ⁷.
21 Florencia Battini, coord., Víctor Grippo: una retrospectiva. Obras 1971-2001,
Buenos Aires: Museo de Arte Latinoamericano de Buenos Aires, 2004.
22 Joanna Page, Decolonizing science in Latin American Art, London, UCL Press, 2021, 113.
23 Justo Pastor, (2004) “La novela chilena de Víctor Grippo”, in: Víctor
Grippo: una retrospectiva obras 1971-2001, Buenos Aires, Museo de Arte Latinoamericano de Buenos Aires, 2004,
63- 69.
24 Joanna Page, Decolonizing science in Latin American Art, 165.
25 Jorge Glusberg, ed., Arte de sistemas en la III Bienal de arte Coltejer. Medellin Colombia,
Buenos Aires, CAyC Centro de Arte y Comunicación, 1972.
26 Ionit Behar, “Arte de sistemas: conceptual art and politics in Argentina (2015)”,
Henrique Faria website, accessed July 29, 2021,
http://www.henriquefaria.com/content/uploads/news-press/pdf/artslant_-_arte_de_sistemas_conceptual_art_and_politics_in_argentina-843.pdf
27 Madeline Weisburg, “Finding a techno-utopia: arte y cibernética,” in
Exhibitionary Histories of Postwar Latin American Art. Vistas: Critical Approaches to Modern and
Contemporary Latin American Art #1, ed. N. Morgan, New York: Institute for Studies on Latin American
Art (ISLAA), 2018, p.10-19.
28 Alonso Rodrigo, “El idioma analítico de Horacio Zabala”. El Gran
Otro, accessed March 12, 2017, http://elgranotro.com/el-idioma-analitico-de-horacio-zabala/
29 Julieta González, “Beyond technology: Juan Downey’s whole earth,”
Afterall: A Journal of Art, Context and Enquiry 37, 2014, 14-27.
30 Jack Burnham, “Art and technology: the panacea that failed,” in The Myths of
Information: Technology and Postindustrial Culture, ed. Katheleen Woodward, London, Routledge, Kegan
Paul, 1980, 213.
Towards a Methodology for Co-creating Artistic Acoustic Ecologies with the Great Lakes
DavidMcFarlane
Toronto Metropolitan University, The Creative School, Media & Design Innovation Toronto,
Canada david.mcfarlane@toronto.ca
Abstract
This paper will discuss the early stages of the development of a methodology for co-creating artistic acoustic
ecologies with the Great Lakes. It explores some initial philosophical, technological, creative/musical, and
ethical concerns involved in my PhD research-creation project entitled: Sounds Like Water, To Me.
In the paper I ask how we might come to understand these bodies of water as animated actors in their own
rights, with their own unique subjectivities? By doing so I hope to facilitate a greater understanding of human
impact on, relationships to, and responsibilities toward the lakes and all other waters. I will discuss the
philosophical framework around perception and being and then explore some technical, artistic and ethical
thinking towards a methodology and investigate which tools, techniques, and protocols can be developed or
adapted to achieve these goals. The article also puts forward some early experiments, guided by the literature
review, and future areas of research.
My research-creation project draws on my experiences as a composer and visual artist, to employ sonic data
gathering techniques such as biodata sonification and field recordings, alongside embodied and
Indigenous-informed research methodologies in order to undertake a co-created artistic acoustic ecology with the
Great Lakes.
The Great Lakes hold 21% or more than 1/5th of the world’s freshwater, 43 million humans depend on the
lakes for drinking water. Despite this, human-generated water contamination, commodification, scarcity, all
combine to present increased threats to the survival of human and non-human species. Water access, extraction,
governance and sovereignty represent points of contention between Indigenous First Nations, and the settler
governments of the United States and Canada.¹
The lakes have witnessed and experienced centuries of geographical, geo-political, environmental, industrial
and colonial activities and events. As a kind of “colonial super-highway,” they are the very means
by which much of the colonizing activities in North America could thrive.
For some time now I have imagined a dialogue with and between the five Great Lakes and what humans might learn
from these exchanges. What would they tell us about the exploitation and colonization they have witnessed and
experienced and what deeper understanding and empathy might we gain?
I had considered this idea as the premise for a work of narrative fiction, whereby characters would be modeled
after histories and traits of each of the lakes, each character knowing they would inevitably be coming together
in confluence with the other four. Attempts at personifying the natural world are not new. Western poets have
long used the Pathetic Fallacy to attribute emotions to natural phenomena and objects. It is common to perceive
something like a mood originating from a body of water. It stands to reason that those who live on the shores of
the Great Lakes may be well suited to describe these emotional characteristics. But this Western tradition of
describing, representing and being separate from, falls short of my objectives of developing
communication and collaboration between human and non-human actors towards creating more sustainable
relationships with the natural world.
I have come to shift my initial curiosity from creating a representation of the lakes to one of
co-creating experiences with them through musical collaboration. Acoustic ecology is the study of the
interrelationship between sound, nature and society.² It affords a space to explore this symbiotic
nexus through the co-creation of sound and music—artistic acoustic ecologies.
I am a white settler of European ancestry, with an educational background in Western arts and music. I am
interested in building different relationships with nature and I believe a necessary step is the development and
use of decolonizing methodologies.
I situate my art practice and research-creation project on land within the Dish with One Spoon Territory and as
a newcomer I have been invited into this treaty in the spirit of peace, friendship and respect, and within the
context of human-induced climate change. The Dish with One Spoon wampum discusses a shared responsibility to
take care of the land—including the waters and their ecosystems.³ I see care for water as a
reciprocal responsibility I can take on through my art practice.
In Making Love with the Land, Canadian First Nations artist Joshua Whitehead discusses the sentience of the
land and water: “The land is an archive, is a library, is a genealogy—a body of land is a body
of literature. Water remembers, it maintains memories, it recalls the substances it has previously dissolved;
trees remember, and in their wounds is a witnessing of wars past, diseases eradicated. If the land can witness
it too, it can listen. And it talks through what we might call living stories...”⁴
In order to hear these stories, and to prepare for my own “conversations” with the lakes, I am
researching technological, creative and ethical frameworks for engaging non-human actors. This has led me to key
research in these respective areas such as Tim Ingold’s concept of “wayfinding,”⁵ Donna
Harraway’s Situated Knowledges,⁶ Natalie Loveless’ contributions to the field of research
creation,⁷ Michel Serres’ considerations on noise,⁸ the soundscape composition practices of
Hildegaard Westerkamp² and R. Murray Schafer,⁹ among others. I also interviewed currently active
experts related to the fields of music composition, production and biodata sonification for this paper. They
included Dr. Scott McLaughlin,¹⁰ and Sam Cusimano.¹¹
Representation, Perception and Being
There has been marked evolution in western art moving from objective representation to today’s embodied
notions. This shift was very much at the core of the modern visual arts movement of the early 20th century.
Representation gave way to abstraction as in Cubism and the whole realm of isms that followed. These movements
moved away from the figure being the focal point, collapsing it into the background and eventually abstraction.
Multiple shifts were taking place to try to awaken in the public a dialog about life and the paint stroke as a
means to express instead of represent our worlds. But their approach was rooted in colonialist humanist culture.
This meant that these concepts remained fixed in representation of a subject, neither embodied nor alive.
Rebecca Belmore’s site-specific sculptural installation, Wave Sound, from 2017, is a powerful
example of how a work can invite embodied experiences. Wave Sound is a set of four site-specific
sculptural installations, one of which is large cone cast from the rock surface on the north shore of Lake
Superior. The cone faces out to the vast lake and quietly invites the viewer to listen to the land by lying down
on the rock such that one can place an ear over the small opening at the small end of the cone, in hopes
of hearing something beautiful. The listener enters into a sympoietic relationship with the situation the
piece creates, physically as well as intellectually. The ‘artwork’ is a means not an
end.¹²
Noise
What is noise, what is music, what is sound, what is silence and how does our situatedness inform our
perception of this? These questions have pervaded contemporary composition and inspired composers to employ
processes with which to move away from mimesis, the representation or imitation of the real world in
the arts.¹³ Similar fundamental queries have surfaced in other areas of creative practice such as the
Visual Arts.
Michel Serres’ story, Noise, articulates concepts in both visual art and sound that specifically
identifies an interesting challenge of describing the threshold of perception: “The agitation is
everywhere to be heard, beside the signals, beside the silence. The silent sea is misnamed. Perhaps white
noise [bruit de fond] is at the heart [fond] of being itself. Perhaps being is not at rest, perhaps it is not
in motion, perhaps it is agitated. White noise never stops, it is limitless, continuous, perpetual,
unchangeable. It has no grounding [fond] itself, no opposite. How much noise has to be made to still the
noise? And what fury orders fury? Noise is not a phenomenon, all phenomena separate from it, figures on a
ground [fond], as a light in the fog, as any message, cry, call, signal must each separate from the hubbub
that fills the silence, just to be, to be perceived, sensed, known, exchanged. As soon as there is a
phenomenon, it leaves noise, as soon as an appearance arises, it does so by masking the noise. Thus it is not
phenomenology but being itself.” ⁸ Serres acknowledges the body’s role in
perception through this figure/ground metaphor but he takes it further.
The situation Serres sets up is a space for a deeper exploration of aesthetics, perception, and experience and
to relate broadly across the arts. In poetic fashion he engages us to measure the infinite gradation between
figure and ground in our own minds, in the visual arts, in the mythological soundscape of the sea.
As Serres gradually evolves this symbiotic figure ground relationship, he also overlays these ideas onto the
evolution of modern art through his referencing of the 3 painters in Balzac’s The Unknown
Masterpiece, a story that was highly influential to Post-Impressionist artists—namely Cezanne. The
Balzac piece recounts an era where “... the categories of beauty, verisimilitude and mimesis were
disappearing, where questions about the relationship between artwork and thought were coming to the fore,
becoming more and more central in relation to the technical limits of painting.”¹⁴
As Serres weaves in his thoughts on representation, perception, and being, he provocatively asks:
“Who has ever seen a meeting between the real and the symbolic in the story?”
In tandem with phenomenology, Serres questions the symbolisms of storytelling. He blurs our efforts as a reader
to keep track of the narrator’s position and challenges our perception of what is true, what isn’t
and what is intended, as if to emerge and recede from the noise of his own story.
His complex presentation of this multi-sensorial challenge is in itself a layering of historical figures and
ground relationships. His somewhat Borgesian strategy, which weaves the authors’ identity, role and tone
in and out of characters, past, present, fact and fiction, artfully fogs our lens.
The figure is part of the noise. It emerges sometimes briefly, and you can recognize it, some of us better than
others, Serres notes, but it always recedes back into the noise, and the noise is always there. The noise is, to
me, this bed of just being, always present.
The Body Electric
While Serres kept to the world of ideas, moving away from representation also gained popularity in music. The
determination of figure and ground, noise and sound have major implications in several fields of contemporary
cultural practice. When noise becomes sound and a sound becomes musicalized as a note, is a focus of rich
investigation in avant-garde and electronic music.
Brian Eno’s Ambient 1: Music for Airports, is music designed to blend with an existing
soundscape and blur the boundary of where his participation as the composer begins and ends.¹⁵
Eno was not the first to make what we now identify as ambient music, but he was the first to name it.
Erik Satie completed Gymnopédie No. 1 in 1888—110 years before Eno’s time.
“Furniture music” was the way Satie described the role of the music in the context of its
environment—part of, inclusive of the sounds in it, not the main focus.¹⁵ This idea later
became a major area of interest for John Cage, who in 1952 released 4’33, which intended to
capture the ambient noises inherent in a given performance space.¹⁶
Mechanical processes and technologies have also played a significant role in altering how we perceive sound. In
1948, Pierre Schaeffer had coined the term musique concrète and had been experimenting with tape
loops and "music systems." The early tape phasing experiments of Steve Reich certainly present a process for
coaxing musical ideas from noise, figure from ground. His tape pieces introduce what Eno would later call
Generative. Eno describes the experience of listening to Reich’s “It’s Gonna
Rain”, as revelatory in how the piece allows for open and asynchronous music events to happen as a seminal
influence in his future work.¹⁷
The tradition of taking hypothetical technical and theoretical positions to arrive at novel and surprising
results, beginning with Satie, has become well established in avant-garde and electronic music. Alongside the
adoption of these more cerebral and conceptual strategies, innovation leading to ‘uncode’ its
creation has also emerged as a pervading trend in electronic music.
Many early electronic instruments were designed to diminish the need to play an instrument or have
knowledge of musical theory. The body became centered as the primary inter-actor with the instrument. This can
be argued as entering the world of relational aesthetics, a movement that introduces the symbiosis of
relationships as a central occupation in western art practices.
For example, the Theremin, sometimes derided as a novelty item, removes a layer of interface to play it. Also
reduced is the codified knowledge one requires to play, say a keyboard instrument, which is a heavily codified,
Western mechanism. The Theremin relies on the body in space and its relationship to the instrument's two
(usually) antennae. The Theremin player interacts less intellectually and in a more physical way, as the direct
communication method. The music comes, closer than other codified interfaces, from the sympoiesis between the
performer and the instrument. Their relationships alter the quality of the air in their environment, this noise
is then captured by the machine and translated into sounds.
Theremin playing enjoyed perhaps its most elevated period in the hands of Clara Rockmore, a virtuoso violinist
who adopted the Theremin as her main instrument and challenged the notion that it was a substandard instrument.
Of her Theremin technique she has remarked: “You can’t play air with hammers, you have to play
with butterfly wings.” [18] Her description clearly emphasizes the physicality of the activity.
In the 1950s Daphne Oram, British composer and early electronic music pioneer, was the first woman to set up an
independent music studio and design and construct an electronic instrument. She also created Oramics, which is a
way to generate electronic sounds from drawn wave forms. Tape-based and electronic instruments introduce a move
away from orchestral instruments towards synthesized sounds which don’t require the skills associated with
classical instruments.¹⁸
Another compelling expression of the freedom perceived by musicians as a result of this ‘uncoding’
comes from Suzanne Ciani who says: “In electronics, you’re not dealing so literally with the
architecture of nodes or harmonics, those building blocks of classical music, you’re dealing in
energy.” Describing a performance from 1974 in which she is performing a composition on an early
Buchla synthesizer she states: “I think they are so sensual. The machine was alive. It was warm. It
communicated. It was sensitive. You could move something just the littlest bit and then a whole new expression
would open up.” Again, the embodied, participatory and more experiential aspects of playing the
instrument come to the fore.¹⁸
Mileece has pushed the boundaries of relational aesthetics into the realm of ethics with non-human beings. Her
work I Oracle: Dreaming in the Future Ancient, questions humans’ capacity to exist in systems in
holistic ways. Like Natalie Loveless, who moves art practices into embodiment of ecological ethics, Mileece is
embedding the symbiotic relationships of her own body to plants in a data language that develops within ethical
concerns. She tries to create a new language through AI and code, getting plants to direct AI.
¹⁹
Is it possible to un-code humanity’s relationships to nature? Are the generative processes inherent in
algorithmic based music a way forward? Is letting go of compositional control through the use of open algorithms
a way to uncode? The cycles of the planet are like an open algorithm, in that the seasons repeat based on
specific conditions and symbiotic relationships of ecosystems inform the outcomes.
The technological aspects of my methodology are influenced and in line with questioning and the practices
I’ve mentioned here. Moving forward, I will be exploring ethical concerns and how to fold these into a
methodology. I am particularly interested in the discussions about ways of making kin with technology,
discussions shared by Donna Haraway and Jason Edward Lewis’ writing about Indigenous
futurisms.⁶, ²⁰ How might I incorporate this into my work?
Haraway, in her essay, Situated Knowledges, challenges the long defended position of the objectivity
of Western science, the “god trick” as it is colloquially known. For Haraway the observer can never
be extracted from a situation, pointing out that science is different when different individuals are doing it.
She centers this idea that perspective matters and that it is important to acknowledge that we bring our
situated perspectives with us as observers.²¹
Sound Methods
These ideas change creative methodologies. How can I establish a methodology that focuses on: visiting with,
harmonizing with, listening with, and co-creating with, during my fieldwork. I am moving away from
mimesis, (or representing the lakes in songs, sounds, or images), towards methexis
(participating in what the lakes are doing).⁸
UK based composer, Scott McLaughlin, offers one approach. He has incorporated Tim Ingold’s idea of
wayfaring in his creative methodology. Ingold likens the wayfaring process
metaphorically to following trails through a landscape, finding new trails that will take you further. This is
in contrast to a process of navigation Ingold calls this method “transport”, where one plots a
direct path from A to B, in hopes of traveling as fast as possible, where any obstacle in between is simply in
your way.¹⁰ McLaughlin notes that, colloquially, wayfaring is more about the journey and the
knowledge accumulated and carried than the destination.⁵ In this way McLaughlin creates scores where,
rather than the performer’s job is to bend the instrument to their will, the performer activates and
supports the material and design characteristics of the instrument. McLaughlin crafts situations where
performers participate with what he calls the “material agency” of an instrument.¹⁰
Artistic Methods
Acoustic Ecology
Interest in acoustic ecology began as a form of ‘acoustic activism’ on the part of like-minded
faculty at Simon Fraser University in the late 1960s and early 1970s.
Of practicing acoustic ecology, R. Murray Schafer states that: “Ecology is the study of the
relationship between living organisms and their environment. Acoustic ecology is therefore the study of sounds
in relationship to life and society. This cannot be accomplished by remaining in the laboratory. It can only
be accomplished by considering on location the effects of the acoustic environment on the creatures living in
it.” ⁹ Led by Schafer, the group gained early recognition through the creation of Soundscapes
of Canada, a 10-part series of one-hour radio programs based on the sounds of Canadian acoustic
environments—essentially a recorded “sound-walk” across the country and was first presented on
CBC-FM "Ideas", in 1974.
Central to soundscape composition is the concept of conscious listening. As illustrated by a quote from
Hildegard Westerkamp, one of the group’s founding composers/researchers explains: “To compose
with environmental sound implies a relationship—a dialogue—between soundscape and listener in
daily life. No matter what the composer’s intent may have been from the start, the materials inevitably
speak with their own language, whose deeper meanings may only emerge with repeated listening and sound
production. And that in itself has the power to shift the composer’s intent.”²
Graphic Notation
I will use graphic notation while visiting various sites around each lake as a way to “poetically
inquire” and capture immediate impressions of each place and space, from a sonic perspective.²²
These will aid my work later as I begin the musical collaborations with the field recordings gathered on each
respective site. The graphic notation will also serve as a potential score which I may choose to share with
other musicians or composers to interpret. The visual notations will act as perceptual barometers, diaristic in
nature, of a time and place from which I will be able to reflect on my own perceptual shift as my empathy for
the lakes evolves. Graphic notation is an alternative access point to musical composition and
dissemination—operating somewhat outside of Western conservatory traditions, which are limited in their
ability to relay emotional content and remain somewhat elitist in terms of who is able to access the education
required to become literate. Sight reading traditional western musical notation, for example, requires years of
tutoring and practice.
Technical Methods
Biodata Sonification
Data is a sea of noise and humans have oceans of it. How can we make sense of it? How can we feel its
significance? How can we better become kin with the tools and technology to better understand
it?²¹, ²⁰
This past year I have been conducting tests and experiments with a biodata sonification device designed and
built by Sam Cusimano. Cusimano’s device reads micro-changes in conductivity within organic matter in
real time. An Arduino inside the device reads those changes in values over time and expresses the fluctuations
as MIDI notes and the degree of change in conductivity is correlated to pitch such that higher degree of change
gets expressed as a higher pitch. The device also has the ability to assign various types of scales such as: the
12 tone chromatic scale, pentatonic and minor scales. The device is equipped with a MIDI output as well as
wireless bluetooth output. The device is able to send data to a MIDI recorder as well, such that the MIDI notes
can be played on an instrument at a later time.
There are many similar devices available that provide the same functionality but I chose Cusimano’s as he
was repeatedly referenced by several artists working with biodata sonification and he was also very
knowledgeable and accessible.
While visiting sites around the lakes I intend to use this device to make recordings of the data I encounter,
as part of a larger ensemble of the other elements I’ve mentioned, which I will combine later.
It should be noted that this realm of sonification is emergent and thus the shared knowledge is evolving,
limited, and to a degree, and more often disseminated through non-academic channels.
One of the key challenges facing this technology is that the multitude of MIDI capable instruments are able to
reproduce just about any sound imaginable—sample or synthesized. How then does one decide which sound to
select for a given organism to use? Assigning or mapping any voice/instrument thus becomes arbitrary
and potentially misleading. There is not a way to get around the translation of the MIDI impulse into a
sound. One could argue that this is also the case when a human plays an instrument as well, the main difference
being the human has agency in selecting the instrument, the scale, notes, key signature, or whether to play at
all.
Unlike more traditional uses of data sonification, where data is sonified such that it becomes easier to
interpret or analyze, my hope is to see if we can perceive something like an emotional feeling from the biodata
generated sounds, which notes are played and how they are played. MIDI can carry parameters such as: volume,
pitch, attack, decay, timbre, Surely, there is still enough going on with those parameters to generate some
vibe? But how do I achieve this without subjecting the data to my own aesthetic priorities? In his brief
article, No Mapping (2016), James Saunders suggests that when this arbitrary mapping is presented as a
truth, that the listener is hearing the actual true sound of a phenomenon like a solar flare or climate change,
the arbitrariness compromises both the data and the sound.²³
Imparting “humanisms” onto other living things to convey complex emotions also raises concerns
associated with Interpellation.²⁴ In the field of Visual Studies, it’s a concept
coined by Louis Althusser, that describes the way media can make you feel like you identify with what’s
being presented to you and belong to a value system—or even an ideology.²⁵,
²⁶ Moving forward, this is an important issue to consider. Embedded in the act of
interpellation are culturally specific values and norms that are not universal and assuming so may risk
alienating potential audiences.
Conclusions
Several recommended methods for addressing the concerns raised here have emerged from the research. Regarding
the assignment of an abstract tone or sound to the MIDI notes being generated, one should consider using neutral
sounds and avoid effects that may impart additional subjective information (like reverb for
example).²⁷ Another recommendation is to ensure that the listener can understand what is triggering
the sound being made and that the mechanism is as exposed as possible.²⁷ Another thought is to create
an installation environment where listeners can hear a chorus of the data sonification events happening
simultaneously, such that the abstractness of the voicing assignments becomes less important than how the the
voices interact together.¹⁰ Lastly, an interface could be created such that the listener could select
and assign voices themselves, revealing the abstractness of the mechanism of selection and potentially deepening
their engagement with the listening experience.
I have recently begun experimenting with recording the MIDI generated sounds in the soundscape they are
occurring as well as recording the audio of the same soundscape. In this way, I am able to listen to both
simultaneously and aligned in real time. In this way, the connection between different types of recorded sounds
(data generated and audio) in a given space are more strongly associated.
A key area of future focus is to explore an ethical methodology. How am I impacting those I am being with in my
actions? How might I gain consent from the non-human actors?
This article does not present a comprehensive or complete methodology but rather a first step towards one.
There is still the area of ethics to more deeply research and consider. I will be interviewing Diane Whalen and
Bonnie Devine to discuss being on the land and visiting methodology from settler and indigenous perspectives
respectively. I will also be further evaluating my situatedness, particularly in consideration of Dylan
Robinson’s concerns brought forward in Hungry Listening, as to “...how your
positionality guides the way you listen to musical subjectivity.”²⁸ I will also
begin researching the many treaties and covenants that describe the agreements I may need to be aware of as I
visit various sites. I am going to explore incorporating ceremony into my activities on the land and water and
will be researching ethical issues around this. I know well that there is much to learn in the field as well and
that whatever I do in preparation needs to incorporate nimbleness, adaptability and openness to what unfolds.
Acknowledgements
The author would like to acknowledge the tremendous support of Dr. Alexandra Bal, Dr. David Gauntlett, Dr,
Scott McLaughlin, Jody Colero, Samuel Cusimano, Daniel Lanios and Andrew Lockhead.
References
1 FLOW, “Privatization and Commodification", Accessed December 7, 2022,
https://forloveofwater.org/issues/privatization/.
2 Hildegard Westerkamp, “Linking Soundscape Composition and Acoustic Ecology”,
Organised Sound 7, no.1, April 2002, 51–56. https://doi.org/10.1017/S1355771802001085.
3 Ange Loft, Denise Bolduc, Mnawaate Gordon-Corbiere, Rebeka Tabobondung, Brian Wright-McLeod,
Rebeka Tabobondung, Brian Wright-McLeod, Brian Wright-McLeod, Indigenous Toronto: Stories That Carry
This Place, Toronto, CANADA, Coach House Books, 2021.
http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=6210057.
4 Joshua Whitehead, Making Love with the Land, Toronto, Knopf Canada, 2022. Ingold,
Tim. Being Alive: Essays on Movement, Knowledge and Description. London ; New York, Routledge,
2011. Jason Lewis, Noelani Arista, Archer Pechawis, Suzanne Kite, “Making Kin with the
Machines”, Journal of Design and Science, July 16, 2018. https://doi.org/10.21428/bfafd97b.
5 Donna Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of
Partial Perspective”, Feminist Studies 14, no. 3, 1988, 575–99,
https://doi.org/10.2307/3178066.
6 Natalie Loveless, How to Make Art at the End of the World: A Manifesto for
Research-Creation, Durham, Duke University Press, 2019. https://muse.jhu.edu/book/70998.
7 Michel Serres, Lawrence R. Schehr, “Noise.” SubStance 12, no. 3, 1983, 48–4,
https://doi.org/10.2307/3684255.
8 R. Murray. Schafer, The Tuning of the World. First edition, Toronto, McClelland and Stewart,
1977.
9 Scott McLaughlin, “Lutins”, Accessed November 1, 2022,
http://lutins.co.uk/index.html.
10 Sam Cusimano, “Electricity for Progress”, Electricity for Progress, Accessed
December 9, 2022, https://electricityforprogress.com/.
11 Lindsay Nixon, “Rebecca Belmore Wants Us to Listen to the Land”, Canadian Art,
Accessed December 8, 2022, https://canadianart.ca/interviews/rebecca-belmore-landmarks-2017/.
12 Jeannette Marie Mageo, “Mimesis Makes Metaphors”, In The Mimetic Nature of Dream
Mentation: American Selves in Re-Formation, edited by Jeannette Marie Mageo, Mind Culture, and Society, Cham:
Springer International Publishing, 2022, 115–40, https://doi.org/10.1007/978-3-030-90231-5_5.
13 Antonio Carnevale, “Honoré de Balzac’s Frenhofer and His Unknown Masterpiece
| CFA.” Conceptual Fine Arts (blog), April 27, 2020,
https://www.conceptualfinearts.com/cfa/2020/04/27/balzac-unknown-masterpiece/.
14 John T. Lysaker, Brian Eno’s Ambient 1: Music for Airports. Oxford, UNITED
STATES, Oxford University Press, Incorporated, 2018,
http://ebookcentral.proquest.com/lib/ryerson/detail.action?docI D=5582502.
15 John Cage, Silence: Lectures and Writings, 1st M.I.T, Press paperback ed. Cambridge,
Mass, MIT Press, 1966.
16 Steve Reich, Conversations. Original édition. Hanover Square Press, 2022.
17 Sisters with Transistors, Documentary, Music, Willow Glen Films, Anna Lena Films, 2021.
18 Mileece, “IOracle”, Mileece, Accessed December 8, 2022,
https://www.mileece.is/ioracle.
19 Jason Lewis, Noelani Arista, Archer Pechawis, Suzanne Kite, “Making Kin with the
Machines”, Journal of Design and Science, July 16, 2018, https://doi.org/10.21428/bfafd97b.
20 Donna J. Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Albany,
UNITED STATES, Duke University Press, 2016,
http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=4649739.
21 Sheila Stewart, “Poetry: The Edge of Knowing”, Creative Approaches to Research 5,
no. 2, 2012, 105–18.
22 James Saunders, “No Mapping.” James Saunders (blog), May 7, 2016,
https://www.james-saunders.com/2016/05/07/no-mapping/
23 Marita Sturken, Lisa Cartwright, Practices of Looking: An Introduction to Visual Culture, Third
edition, New York, Oxford University Press, 2018.
24 Heather Dorries, Sue Ruddick, “Between Concept and Context: Reading Gilles Deleuze and
Leanne Simpson in Their in/Commensurabilities”, Cultural Geographies 25, no. 4, October 1, 2018,
619–35, https://doi.org/10.1177/1474474018778576.
25 Nicole Gross, Mikko Laamanen, “‘Hey, You There! Marketing!’ On Ideology and
(Mis)Interpellation of the Marketing Educator as Subject”, Journal of Marketing Management 0, no. 0,
June 30, 2021, 1–24, https://doi.org/10.1080/0267257X.2021.1944896.
26 Sonification & The Problem with Making Music from Data, 2017,
https://www.youtube.com/watch?v=Ocq3NeudsVk.
27 Dylan Robinson, Hungry Listening: Resonant Theory for Indigenous Sound Studies. University of
Minnesota Press, 2020. https://doi.org/10.5749/j.ctvzpv6bb
Bibliography
Branch, Government of Canada, Indigenous and Northern Affairs Canada, Communications, “Treaties and
Agreements”, Administrative page, November 3,
2008, https://www.rcaanccirnac.gc.ca/eng/1100100028574/1529354437231#chp3.
John Cage, Silence: Lectures and Writings, 1st M.I.T, Press paperback ed. Cambridge, Mass, MIT
Press, 1966.
Antonio Carnevale, “Honoré de Balzac’s Frenhofer and His Unknown
Masterpiece | CFA”, Conceptual Fine Arts (blog), April 27, 2020,
https://www.conceptualfinearts.com/cfa/2020/04/27/balzacunknown-masterpiece/.
Sam Cusimano,
“Electricity for Progress.” Electricity for Progress. Accessed December 9, 2022,
https://electricityforprogress.com/.
Dorries Heather, Sue Ruddick, “Between Concept and Context:
Reading Gilles Deleuze and Leanne Simpson in Their in/Commensurabilities”, Cultural Geographies
25, no. 4, October 1, 2018, 619–35, https://doi.org/10.1177/1474474018778576.
David Gauntlett, Media, Gender and Identity: An Introduction. Taylor & Francis Group, 2002,
http://ebookcentral.proquest.com/lib/ryerson/detail.ac tion?docID=170355.
Nicole Gross, Mikko Laamanen., “‘Hey, You There! Marketing!’ On Ideology and
(Mis)Interpellation of the Marketing Educator as Subject”, Journal of Marketing Management 0,
no. 0, June 30, 2021, 1–24, https://doi.org/10.1080/0267257X.2021.1944896.
J. Donna
Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of Partial
Perspective”, Feminist Studies 14, no. 3, 1988, p.575–99,
https://doi.org/10.2307/3178066.
J. Donna Haraway, Staying with the Trouble: Making Kin in the Chthulucene, Albany, UNITED STATES,
Duke University Press, 2016, http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=4649739.
Tim Ingold, Being Alive: Essays on Movement, Knowledge and Description, London ; New York,
Routledge, 2011.
Jason Lewis, Noelani Arista, Archer Pechawis, Suzanne Kite, “Making Kin with the
Machines", Journal of Design and Science, July 16, 2018, https://doi.org/10.21428/bfafd97b.
Ange Loft, Denise Bolduc, Mnawaate Gordon-Corbiere, Rebeka Tabobondung, Brian Wright-McLeod, Rebeka
Tabobondung, Brian Wright-McLeod, Brian Wright-McLeod, Indigenous Toronto: Stories That Carry This Place,
Toronto, CANADA, Coach House Books, 2021,
http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=6210057.
Natalie Loveless, How to Make Art at the End of the World: A Manifesto for Research-Creation,
Durham, Duke University Press, 2019, https://muse.jhu.edu/book/70998.
John T. Lysaker, Brian
Eno’s Ambient 1: Music for Airports. Oxford, UNITED STATES, Oxford University Press,
Incorporated, 2018, http://ebookcentral.proquest.com/lib/ryerson/detail.action?docID=5582502.
Jeannette Marie Mageo, “Mimesis Makes Metaphors”, In The Mimetic Nature of Dream Mentation:
American Selves in Re-Formation, edited by Jeannette Marie Mageo, Culture, Mind, and Society, Cham:
Springer International Publishing, 2022, 115–40, https://doi.org/10.1007/978-3-030-90231-5_5.
David McFarlane, “Scott McLaughlin: Wayfaring and Material Animacy”, Accessed November 1, 2022,
http://davidmcfarlane.com/PhD/Scott_McLaughlin_rough_mix.MP3
Scott McLaughlin, “Lutins.” Accessed November 1, 2022, http://lutins.co.uk/index.html.
Mileece, “IOracle”, mileece, Accessed December 8, 2022, https://www.mileece.is/ioracle.
Dylan AT. Miner, “Mawadisidiwag Miinawaa Wiidanokiindiwag // They Visit and Work Together", Makers,
Crafters, Educators, 2018, 131.
“MusikTexte 149 – Mai 2016", Accessed July 19, 2022, https://musiktexte.de/MusikTexte-149/en.
Lindsay Nixon, “Rebecca Belmore Wants Us to Listen to the Land”, Canadian Art, Accessed December
8, 2022, https://canadianart.ca/interviews/rebecca-belmore-landmarks-2017/.
FLOW, “Privatization and Commodification”, Accessed December 7, 2022,
https://forloveofwater.org/issues/privatization/.
Steve Reich, Conversations, Original édition, Hanover Square Press, 2022.
Dylan Robinson, Hungry Listening: Resonant Theory for Indigenous Sound Studies, University of
Minnesota Press, 2020, https://doi.org/10.5749/j.ctvzpv6bb.
James Saunders, “No Mapping.” James Saunders (blog), May 7, 2016,
https://www.james-saunders.com/2016/05/07/no-mapping/.
R. Murray Schafer, The Tuning of the World, First edition, Toronto, McClelland and Stewart,
1977.
Michel Serres, Lawrence R. Schehr, “Noise.” SubStance 12, no. 3, 1983, 48–4,
https://doi.org/10.2307/3684255.
Sisters with Transistors, Documentary, Music, Willow Glen Films, Anna Lena Films, 2021.
Sonification & The Problem with Making Music from Data, 2017,
https://www.youtube.com/watch?v=Ocq3NeudsVk.
Sheila Stewart, “Poetry: The Edge of Knowing.” Creative, Approaches to Research
5, no. 2, 2012, 105–18.
Marita Sturken, Lisa Cartwright, Practices of Looking: An Introduction to Visual Culture, Third
edition, New York, Oxford University Press, 2018.
Hildegard Westerkamp, “Linking Soundscape Composition and Acoustic Ecology”, Organised Sound
7, no. 1, April 2002, 51–56, https://doi.org/10.1017/S1355771802001085.
Joshua Whitehead, Making Love with the Land, Toronto, Knopf Canada, 2022.
Structures of Emotion: Speculating on an AI-Human Symbiosis
AvitalMeshi1,
TreydenChiaravalloti2
1University of California Davis, 2Graduate School of Design, Harvard
University USA ameshi@ucdavis.edu.treyden3@gmail.com
Abstract
Structures of Emotion is a performance artwork that explores a symbiotic relationship between humans
and an Emotion Recognition Artificial-Intelligence (AI) algorithm. The piece utilizes a wearable computing
device designed to enable the wearer to recognize emotions through two different perspectives: their own organic
senses and an AI apparatus, which serves as an extension of the body, connecting the human mind to a "collective
consciousness." Participants interacted with two performers; one wore the AI device, while the other relied
solely on their organic abilities. The performance demonstrates how AI emotion recognition systems are still
immature. However, it invites us to speculate on its potential role when it becomes more sophisticated.
Additionally, it explores the ethical complexities of our entanglement with emotion recognition algorithms and
imagines the danger of becoming dependent on them within a transhumanist future.
Keywords
Performance Art, Artificial Intelligence, Emotion Recognition, Affective Computing, New Media Art, Interactive
Media, Transhumanism, Wearable Computing.
In the wake of World War II, when scientists were forced to develop an array of strange destructive gadgets,
peacetime was when they had to find a better use for their instruments. Vannevar Bush, a scientist, engineer,
and innovator who led the US military's wartime R&D in the 1940s, suggested that the thinking human should
have a new relationship with knowledge. He hypothesized a new device called the Memex, which was intended to
recollect all scientific knowledge, mimic the associative processes of the human mind, and allow humans to
access the collective record in a more useful way. In Bush's view, the Memex was an enlarged intimate supplement
to human memory. The device included a tiny camera the size of a walnut attached to the forehead of the human.
Bush argued that the wearer of this camera is the scientist of the future, absorbing worthy records through this
artificial third eye while moving around the lab or the field. The Memex was designed to enhance the flow of
information and knowledge to and from the brain and to make the human one with the machine ¹.
While we cherish our connection to knowledge, the thinking human is also, at least for now, an emotional
being. Is technology capable of assisting us in this manner? Can AI technology enhance our ability to recognize,
express, and regulate our emotions? The affective computing consumer market is consistently growing.² Major
developers predict that soon all of our devices will have an ‘emotion chip’ in them ³. If this
comes true, what would it mean for human society? How will it impact our behavior and our relationships with one
another? To unpack these questions, we follow up with Marshall McLuhan’s famous statement that “The
Medium is the Message” and ask what kind of message affective AI bring with it? For McLuhan artistic
exploration can always be relied upon to tell the old culture what is beginning to happen to it; serving as,
what he called, a “Distant Early Warning System,” art enables us to discover and prepare for changes
in our society.⁴ The artwork we present in this paper may be considered along these lines.
Structures of Emotion is a series of two interactive, participatory performances that examine a
symbiotic relationship between humans and AI algorithms. It utilizes a device that is almost identical to the
wearable element of the Memex. Using a tiny camera on the performer's forehead, the device detects, identifies,
and communicates the emotions of participants. A second performer, unaided by the AI device, interacts with the
same participants and attempts to recognize their emotions. A playful interaction between the performers and
participants illustrates how emotion recognition algorithms are still in their infancy. Nevertheless, tensions
between humans and machines remain evident. The piece elicits a debate about how algorithms might impact human
behavior. In addition, it raises further speculations about our transhumanist future and the dangers of becoming
dependent on machines to moderate our emotions, an essential element of human identity.
Implementation Details The Wearable Device
Structures of Emotion uses an ocular-centric AI emotion recognition system. A tiny camera is attached
to the wearer's forehead, essentially serving as a third eye. The camera is connected to a Raspberry-Pi
microcomputer attached to the wearer's head, an LED display panel attached to the chest, and a handheld
Bluetooth speaker. The entire apparatus is battery powered and tetherless so that the wearer can move freely
(Figure 1). The microcomputer hosts an off-the-shelf AI emotion recognition model, outsourced from the computer
vision GitHub community.⁵ As with other AI emotion recognition models, this model is based on classical
theories claiming that emotions can be categorized into a limited number of primary, universal
emotions.⁶, ⁷ These include the emotions "Happy," "Sad," "Angry," "Disgusted,"
"Surprised," "Fearful," and "Neutral." As soon as the device recognizes an emotion, it is displayed on the LED
panel and announced with a distinctly computer-generated voice, "You Seem Happy" or "You Seem Angry," depending
on the detected emotion.
The Performance
The wearable device was used in two types of performances: a street performance and an online performance. In
both types, two performers interacted with the audience and recognized participants' emotions via two different
methods. One performer wore the AI device, allowing the machine to act as her voice. As the device recognized
and announced the participants' emotions, the performer remained emotionally detached and unresponsive. A second
performer, unaided by the wearable device, facilitated the encounter while using his own organic senses to
recognize and voice participants' emotions as they appear through his eyes. The street performance took place
during summer 2021 in downtown Santa Cruz, CA. It lasted about 60 minutes and included participants who seemed
curious, available, and comfortable interacting without their face masks (Figure 2). The online performance was
held in an online Zoom meeting during fall 2021. Participants volunteered to join the meeting, so the
interactions with them were more deliberate. Each participant was observed individually for 15-20 minutes,
allowing both performers to attempt to recognize and announce the participant’s emotions (Figure 3).
Audience Reception Trying to Beat the Machine
Many participants attempted to manipulate the way the device recognized their emotions. They consciously
altered their facial expression or, in one participant's words, tried to "channel different emotions." Through
this playful interaction, participants felt motivated to score each of the device's possible outputs. A similar
engagement is prompted by the artwork 'Emojify' in which participants are encouraged to fake their emotions:
"Can you make the emotion recognition system read you as happy, sad, or angry? Can you beat the machine?"⁸
Performing one's emotional expressions is one of the most persistent critiques of the classical emotion
recognition theory. Our ability to swiftly change our facial expressions demonstrates how they can be
disconnected from our actual emotional state.⁹ It is therefore essential to ask what exactly the AI device
detects; Is it recognizing participants' emotions or merely observing facial expressions? Similarly, we can ask
whether a human is capable of acknowledging another person's emotions. Can we see each other's expressions and
know what emotions they convey?
Participants' tendency to alter their behavior in the presence of an AI algorithm illustrates how recognition
systems can retrain human behavior. We are already conditioned to perform emotions when we are with others
because our society normalizes some emotions and devalues others.¹⁰ By incorporating AI systems into
our environment, we are now also required to change our behavior in the presence of
algorithms.¹¹, ¹², ¹³ The automatic recognition of emotions
is likely to comply with society's demand that anyone who appears angry or upset will be discriminated against.
Instead of understanding the reasons behind such strong emotions, AI systems will assist in excluding or
repressing those who express them. This idea is well illustrated in the Japanese cyberpunk anime PSYCHO-PASS
(2012), where an advanced surveillance system constantly monitors citizens' mental states to predict their
criminality potential. Expressing negative emotions increases an individual's crime coefficient index, and when
this index exceeds a certain threshold, the individual gets arrested or killed. In this sense, we agree with
artist and philosopher Hito Steyerl, who claims that incorporating AI systems into conflicts that we can still
not resolve on our own ignores the complexities of singular human experiences and the emotions they evoke. Human
conflict, she claims, is often needed, ignoring it only weaponizes interactions and leads to further
polarization.¹⁴ Before further establishing our relationships with AI emotion recognition systems, we
must ask ourselves if we are willing to fake a smile to pass as good members of our society. This idea recalls
Yue Minjun's paintings of cloned humans with exaggerated grins on their faces. While smiley faces are easily
considered "happy," they can also be viewed as mandatory masks worn by members of a society that prioritizes
appearance over genuineness.¹⁵
Entering Affective Loops
Most participants smiled as they became engaged with the performance. In response the device repeatedly
announced: "You seem happy.” This resulted in more smiles and laughter on the participants' behalf. One of
the participants who kept being detected as "Happy" affirmed the device's announcements and stated: "I am
radiating happiness right now." Per societal values, participants seemed satisfied to be detected as "Happy." On
the other hand, those classified as "Angry" or "Sad" seemed annoyed by these results, stating that they did
not feel sad or angry and questioned the device's accuracy. The responses can be explained within the
'Affective Loop' framework; Participants express a particular emotion through a physical manifestation, and the
system responds according to whatever parameters it was trained on. In return, participants react back in an
even more involved manner.¹⁶ Participants were quick to enter these affective loops with the machine.
Yet, as the performance progressed, we noticed they were as quick to lose interest in the device's repetitive,
limited, and laconic responses. Instead, they grew more interested in what the performer, unaided by the AI, had
to say about their emotions.
The performer who evaluated participants' emotions without the aid of the AI device felt transformed by the
experience. Although it may be seen as a mundane task that we regularly perform when interacting with one
another, attempting to be fully attentive to someone standing in front of us is not trivial. In the performer's
account, emotions could not be accurately expressed in words. Participants' ephemeral facial expressions and
other fleeting manifestations of their body language required a compassionate presence that evoked care, simply
listening with the body and only speaking to express apparent emotions. During the interaction between
participants and this performer, both sides appeared to be involved in a deeper affective loop. In the same way
as the participants, the performer responded to the situation and felt the need to generate feelings he hoped to
describe within himself. As a result, he became increasingly involved and realized that seeing does not suffice
to understand another's feelings.
Comparing Human and Machine Abilities
Participants standing in front of the performer who was wearing the AI device stared directly into her eyes,
almost as if they assumed that she was the one recognizing their emotions. They rarely seemed to look at the
camera on her forehead. At the same time, they did not attempt to speak with her during the performance. Perhaps
it was her detached mode of communication that made her seem almost nonhuman and robotic. But as described
above, as much as they were fascinated with ‘her’ responses at the beginning of the interaction,
they quickly lost interest in the mechanic repossess she delivered through the AI device. For a machine to be
successful as a human in an emotional interaction, it needs to provide an illusion that it is authentically and
socially engaged over an extended period of time. Such abilities are known to be challenging in developing
social robots that are designed to be perceived as socially intelligent,¹⁷ able to deliver successful
shared emotional experiences.¹⁸ Participants in Structures of Emotion could compare the
humans’ versus the machine’s abilities to recognize emotions. It was evident from the performances
that the AI device is still premature. Yet the artwork evoked questions regarding our understanding of our
own emotions, provoked thought regarding our relationship with AI algorithms, and triggered a transhumanist
imagination in which these algorithms are interfaced with the human body, and we become entirely dependent on
them.
Discussion
Structures of Emotion and its interactive performances explored a synergistic relationship between
humans and machines. We sought to enhance human emotional capacity by directly augmenting it with an artificial
one. This was afforded by attaching the artificial intelligence to the body, providing the wearer an amplified
vision that originates with what can be called a "collective consciousness" potentially serving as an extension
of the mind. With the Memex in mind, designed to shift "thinking" into a "thinking-with" ideal, we aimed to
examine a similar standard with emotions. Can humans and machines form a "feeling-with" relationship? What would
it mean for us to sense the world and mediate our emotions in symbiosis with machines?
What Do We Really Know about Emotions?
Before interfacing our emotions with algorithms, we must question our abilities to understand emotions.
According to the classical theory described above, emotions are universal and can be classified into primary
categories based on the distinctive movements of facial muscles. Despite this, there is still a debate on what
we know or do not know about the function and purpose of emotions. Among the critiques of classical theory is
the claim that the experiments used to show that emotions are universal were too simplified, disregarding the
ever-changing context within which emotions are expressed.¹⁹ Some scholars claim that we should not
ignore cultural factors that impact emotional expressions.²⁰ And other, more recent voices, state
that emotions are merely guesses made by our brains. Thus, facial expressions cannot accurately indicate a
person's mental state ²¹.
While the controversy over human emotions has yet to be resolved, governments and tech companies are actively
developing and incorporating AI algorithms to recognize human emotions. Algorithms of this type are already used
in national security systems,²² education platforms,²³, ²⁴ hiring
startups,²⁵, ²⁶ and police programs.²⁷ According to Rana el
Kaliouby, cofounder and CEO of "Affectiva," the first company to market "emotion AI," when humans recognize
emotions, they are often incorrect.²⁸ Since we do not yet have a good definition of emotions, this
statement may be true. But what does it mean about the algorithms we design to understand and regulate emotions
for us? Do they know any better? The primary goal of these algorithms is to optimize efficiency and productivity
and compensate or ultimately replace a 'fallible human'. Similarly, many other techno-fixes hold grandiose
promises to help us but end up enhancing our problems while simultaneously providing us with an excuse to ignore
them.²⁹ Even though emotion recognition algorithms have immense market value, we must ask ourselves
what is at stake when we use them prematurely.³⁰
In Structures of Emotion, the design of the wearable device aims to expand human consciousness into a
collective one. If AI models are trained on large enough datasets, they can arguably be considered to symbolize
a "collective consciousness." Nevertheless, it is essential to ask what exactly goes into this consciousness.
The model we used here was trained on a dataset that contained about 28,000 images of faces labeled according to
their apparent emotional expressions and categorized by the classical theory's primary categories. It is unclear
how these images were collected, who are the people seen in them, and in what context they are expressing their
emotions. It becomes apparent when we examine the dataset that the 'collective consciousness' is more of a
"selective consciousness" than anything else. Images are stripped of their personal, social, and political
meanings, and no evidence exists that the labeled emotions represent anything that was actually felt.
Undoubtedly, this AI model is simple and, therefore, less robust than those used by companies such as
'Affectiva'. Nevertheless, we argue that even a "high quality" or presumably a fairer dataset (if this is even
possible) would still lack substantial support that proves that it works "correctly."
Attaching an AI System to the Body
Artificial intelligence systems that recognize emotions are usually disembodied. These models often look at us
from far away without us realizing they are watching. Donna Haraway described this kind of gaze as an
"unregulated gluttony," a "god-trick illusion of seeing everything from nowhere." Such a gaze, she asserts,
"fucks the world."³¹ Our idea to attach an AI model to the body is inspired by her call to reemphasize
vision through alternative feminist perspective. However, we found that even when the AI gaze is lowered to the
level of the human eye, it can still operate as an asymmetric gaze from nowhere. Such understanding is also
apparent in Karen Palmer's interactive film Riot AI (2016) which emphasizes a potentially violent
encounter between a rioter and a police officer, mediated by an emotion recognition algorithm. Although the
system is attached to the officer's body, it still serves as a gaze from nowhere. The policeman does not
actively participate in evaluating the emotions of those who are stopped and assessed, he simply reacts
according to the outcomes of the algorithmic gaze.
As part of the design of Structures of Emotion, we considered how the AI might extend the wearer's
mind and augment their perspective. Wearing the device allows one to use their organic senses while being
constantly guided by the device's perspective. Using both views, the wearer can note when they agree or disagree
with the outputs of the device. Wearers may then ask themselves: "Do I understand this situation? Do I agree
with the AI's assessment of this person's emotions? Can I recognize, acknowledge, and respect these emotions?"
With that, both the human’s and machine’s emotion evaluation practices come into question. This is
also where other cognitive abilities can come into play and the wearer can then use this opportunity to spark a
conversation with the person in front of them and ask, "How do you feel?" The answer to this question may remind
us that technology is far from maturing beyond its maker and that both humans and machines still have much to
learn. By adding another perspective to our vision, the wearable device may promote traditional practices of
conversation between people, nurturing relationships of attention and care.
Wearable computers, were extensively explored by researcher Steve Mann who claims that the intimate
relationship between humans and these devices leads to exceeding the wearer's capabilities. For him, the
fundamental purpose of wearable computing is personal empowerment.³² Nevertheless, he also expresses
concern that people may become dependent on this technology if they use it for an extended period of
time.³³ It is possible that dependency could lead to the erasure of human knowledge and the decline of
organic human capabilities. Such a process might seem difficult to imagine with the wearable used in
Structure of Emotions; however, it is easy to imagine a time when the algorithms will be far more
sophisticated, and our dependence on them would seem inevitable.
Imagining Transhumanist Futures
Human-AI bi-directional symbiosis goes well beyond wearable computing. It imagines a speculative future in
which AI and Humans can potentially merge into a super-intelligent, transhumanist figure. In this potential
future, both humans and AI symbionts benefit from the symbiotic relationship. This relationship will be
strengthened by AI's ability to detect, communicate, and express emotions. Humans are still better at this task
than algorithms, but we envision a future in which algorithms may surpass human abilities and even erode humans'
emotional capabilities. In considering the benefits humans might bring to this symbiotic relationship, we argue
that they are centered around the body, its biological structure, and natural sensitivities. Currently, AI faces
major challenges in shifting from a narrow to a more general intelligence owing to its imbalance in dealing with
abstract truths versus the gritty world of exceptions.³⁴ In most cases, AI algorithms function as
disembodied software with a huge appetite, but however much we 'feed' them, they still lack a solid
understanding of the world. This is evidenced by the numerous instances in which AI models have produced racist,
misogynistic, offensive, or seemingly absurd content. By attaching these models to the human body, they can
benefit from the body's inherent knowledge, adaptation to the environment, and attachment to the world.
The human-AI bodies of the future will likely go far beyond fashionable devices and include brain-computer
interfaces. Despite the fact that brain-machine interfaces have not yet become a reality, neuroscience research
has already made impressive advances in this direction. Among the most famous examples is the monkey Aurora who
operated a robotic arm just by thinking about doing so.³⁵ With current scientific advancements we can
easily speculate soldiers operating weapons with their thoughts while suppressing feelings of fear and anxiety
using brain devices.³⁶ The transhumanist techno-optimistic trajectory of such projects aims to
empower individuals. But is this necessarily the case? Will this super-intelligent Human-Machine figure function
as an enhanced individual? Or not?
We define ourselves by our emotional capacities; they are based on millions of years of evolutionary adaptation
for survival and advancement. What will be our role in the world if we depend on AI algorithms to mediate our
emotional landscapes? How can we avoid becoming nothing more than a numb 'meat' body carrying around an AI
algorithm? Transhumanists argue that the continuation of personhood is essential in becoming a transhuman. It
must include the continuity of memories, attitudes, values, and emotional dispositions.³⁷ In light of
this condition, we can already see the consequences of sharing our emotional abilities with AI. As philosopher
Susan Schneider explains, we can reject the idea entirely because techno-enhancements themselves alter the
pattern of the original individual. Their impact on identity is always significant. For her, "the transhumanist
developmental trajectory... is a technophile's alluring path to suicide."³⁸ A glimpse of such
alterations in one's identity was even evident in the performances of Structures of Emotion with
participants and performers changing their behavior by responding to the presence of an AI algorithm within
their shared space.
Conclusion
The artwork Structures of Emotion was designed to articulate an AI-Human symbiosis. Physical
limitations, however, prevented the artists from fully conveying this idea. Emotion recognition algorithms are
still premature. Data they are trained on is devoid of context and potentially biased. It is still necessary to
gain a deeper understanding of emotions' definition, function, and purpose. It is unclear what is the impact of
either disembodied or embodied relationships with AI algorithms. For obvious reasons none of the performers were
ready to integrate an artificial intelligence algorithm into their bodies in any invasive manner. Even wearing
the devise was somewhat cumbersome. Despite the limitations, this artwork still allowed us to consider the
potential of such a symbiosis, speculate on its implications, and imagine its future. We realize that emotion
recognition algorithms are a crucial step forward for human-computer interactions and we posit that any use of
such algorithms invites us to think more broadly about what it means to be emotionally intelligent. It is
possible for AI to expand our emotional capacities, but it also holds the risk of eroding them. It is important
to consider the design of these algorithms as well as the way they interface with our bodies. If this artwork
can indeed serve as a McLuhanist Distant Early Warning system, we must strongly advocate that the integration of
this technology will give us more agency and autonomy over our emotional states, improving rather than eroding
our most human drive for meaningful, intimate relationships with one another.
Acknowledgments
The authors wish to thank Ofer Meshi for invaluable assistance with the AI system. The street performance was
developed during a summer residency at the 418 Project, Santa Cruz. The online performance was developed during
the Impact Art AT residency at Zero1, San Francisco, CA under the guidance of Patricia J. Reis. The article has
been very much improved thanks to generous and inspiring feedback by the double-blind peer reviewers.
References
1 Vannevar Bush, “As we may think” The atlantic monthly 176, no. 1,
1945, 101-108.
2 Affective Computing Market by MarketsandMarkets.com, accessed September 10, 2023,
https://www.marketsandmarkets.-com/Market- Reports/affective-computing-market-130730395.html
3 TED, 2015, June 15, This App Know How You Feel - From the Look on Your Face | Rana el
Kaliouby [Video], YouTube.
4 Marshall McLuhan, Understanding Media: The Extensions of Man, New York, McGraw-Hill, 1964.
5 Omar Aymen, “Emotion-recognition”, GitHub, accessed September 10, 2023,
https://github.com/omar178/Emotion-recognition
6 Paul Ekman, "The argument and evidence about universals in facial expressions", Handbook of
social psychophysiology, 143, 1989, 164.
7 Paul Ekman, Wallace V. Friesen, "Facial action coding system”, Environmental
Psychology & Nonverbal Behavior (1978).
9 Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix Martinez, Seth D. Pollak, "Emotional
expressions reconsidered: Challenges to inferring emotion from human facial movements", Psychological
science in the public interest 20, no. 1, 2019, 1-68.
10 Ruth Leys, The ascent of affect: Genealogy and critique, University of Chicago Press, 2019.
11 Meshi Avital, Angus G. Forbes, ""Don’t Worry, Be Happy”: Resisting an AI Emotion
Recognition System With a Smile", In 10th International Conference on Digital and Interactive Arts,
2021, 1-4.
12 Meshi Avital, "Should We Become Emotional With AI?: Performative Engagements With an Affective
Algorithm", International Journal of Art, Culture, Design, and Technology (IJACDT) 11, no. 3, 2022,
1-13.
13 Vincent James, "Canon Put AI Cameras in Its Chinese Offices That Only Let Smiling Workers
Inside", The Verge, June 17, 2021, accessed September 10, 2023, https://www.theverge.com/
2021/6/17/22538160/ai-camera-smile-recognition-office-workers-china-canon
14 Hito Steyerl, "The Language of Broken Glass", Presentation, part of “Stop Making
Sense”, HKW, Berlin, 2019.
15 Nazanin Lankarani, "The many faces of Yue Minjun", The New York Times 5, 2012.
16 Kristina Höök, "Affective loop experiences: designing for interactional
embodiment", Philosophical Transactions of the Royal Society B: Biological Sciences 364, no. 1535,
2009, 3585-3595.
17 Ana Paiva, Iolanda Leite, Tiago Ribeiro, "21 emotion modeling for social robots", The
Oxford handbook of affective computing, 2014, 296.
18 Koji Inoue, Divesh Lala, and Tatsuya Kawahara, "Can a robot laugh with you?: Shared laughter
generation for empathetic spoken dialogue", Frontiers in Robotics and AI, 2022, 234.
19 Julie Beck, "Hard feelings: Science’s struggle to define emotions", The Atlantic
24, 2015.
20 Margaret Mead, “Darwin and Facial Expression-Century of Research in Review-Ekman
P”, 1975, 209-213.
21 Lisa Feldman Barrett, How emotions are made: The secret life of the brain, Pan Macmillan, 2017.
22 Marda Vidushi, Shazeda Ahmed, "Emotional Entanglement: China’s emotion recognition market
and its implications for human rights", 2021.
23 Michael Kwet, Paul Prinsloo,"The ‘smart’classroom: a new frontier in the age of the
smart university", Teaching in Higher Education 25, no. 4, 2020, 510-526.
24 Andrew McStay, "Emotional AI and EdTech: serving the public good?." Learning, Media
and Technology 45, no. 3, 2020, 270-283.
25 Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy, "Mitigating bias in algorithmic
hiring: Evaluating claims and practices", In Proceedings of the 2020 conference on fairness,
accountability, and transparency, 2020, 469-481.
26 Elmira Van den Broek, Anastasia Sergeeva, Marleen Huysman, "Hiring algorithms: An ethnography
of fairness in practice", 2019.
27 Jennifer Valentino-DeVries, "How the police use facial recognition, and where it falls short",
New York Times 12, 2020.
28 Taylor Telford, "Emotion AI’is a $20 billion business. New research says it can’t
do what it claims", The Washington Post, July 31, 2019, accessed September 10, 2023,
https://www.wash-ingtonpost.com/business/2019/07/31/emotion-detection-ai-is-billion-industry-new-research-says-it-cant-do-what-it-claims/
29 Virginia Eubanks, Automating inequality: How hightech tools profile, police, and punish the
poor, St. Martin's Press, 2018.
30 Alexa Hagerty, and Alexandra Albert, "AI is increasingly being used to identify
emotions-here’s what’s at stake", In The Conversation, 2021.
31 Donna Haraway, "‘Situated Knowledges: The Science Question in Feminism and the Privilege
of Partial Perspective'," In Space, gender, knowledge: Feminist readings, 53-72, Routledge,
2016.
32 Steve Mann, "Wearable computing as means for personal empowerment." In Proc. 3rd Int.
Conf. on Wearable Computing (ICWC), 1998, 51-59.
33 Steve Mann, "Humanistic computing:" WearComp" as a new framework and application for
intelligent signal processing", Proceedings of the IEEE 86, no. 11, 1998, 2123-2151.
34 Gary Marcus, Ernest Davis, Rebooting AI: Building artificial intelligence we can trust,
Vintage, 2019.
35 Miguel Nicolelis, Beyond boundaries: The new neuro-science of connecting brains with
machines---and how it will change our lives, Macmillan, 2011.
36 Nancy S. Jecker, Andrew Ko, “Brain-computer interface could allow soldiers to control
weapons with their thoughts and turn off their fear - but the ethics of neurotechnology lags behind the
science”, The Conversation, December 2, 2022 accessed September 10, 2023.
37 Nick Bostrom, "The transhumanist FAQ", Readings in the Philosophy of Technology 2, no.
4, 2003, 355-360.
38 Susan Schneider, Artificial you: AI and the future of your mind, Princeton University Press,
2019.
The w(e)aves of complexity: Relational ontologies within the Symbios Art Exhibition
PaolaTorresNúñezdelPrado
Stockholm University of the Arts, Stockholm, Sweden paola.torres@uniarts.se
Abstract
The works included in the Symbios art exhibition spread through various disciplines and mediums: painting,
weaving, interactivity, sculpture, Bio Art and even A.I. Its transdisciplinary condition does not originate in
the multi-mediality of its constituents exclusively, but in the intentional implementation of an ecology of
meanings once the pieces, characteristically polysemic, are made for and placed in the gallery. The physical
division of the two rooms that conform the exhibition space becomes what defines the type of interaction that
the art works included will deal with, each area taken as a subcategory of the general symbiotic relationships.
I proceed to find a correspondence of the symbiotic process in the social realm, more specifically, in the
cultural process known as syncretism, that is also a distinctive aesthetic quality of the art works exhibited.
Such a web of relational ontologies goes beyond the legacy of Conceptual art, now taken as a product of a
Western paradigm that is based in dualism, classification, and exact definition. Instead, the show embraces the
diverse narratives behind its components as a net of meanings and diverse cultural references, so to achieve a
complex yet holistic approach to art making and curating simultaneously.
The “Symbios” (1) art show was a transdisciplinary exhibition presented in November,
2022 in Tegen 2 gallery in Stockholm, Sweden. It included three bio-textiles, one puzzle sculpture, a big
interactive loom (including one of the bio-textiles mentioned before), an introductory video (2)
recorded during my stay in the Finnish Tuo Tuo art residency where I started working with lichens ¹, a
textile painting, a 3D printed object, and two videos generated using the Neural Radiance Field method that
converts a set of 2D images in 3D renders.² The broader idea of the symbiotic process—that of
long-term interaction among living organisms—was taken as a starting point so to manage the relation among
the works, the narratives behind these, their aesthetic qualities, and their physical place in the venue. This
paper pertains the strategies I applied to put up a complex art show in just a couple of weeks, consciously
undertaking various roles, that of technician, curator, and artist, and the relational ontologies incorporated
both in such tasks and in the artworks themselves. What I discuss here might be a common occurrence for us
artists that still exhibit in gallery spaces, yet I am proposing a different way of managing both art creation
and curation, and its relation. Regarding such matters, the introductory text placed in the exhibition was
nevertheless co-written with gallerist and artist Gunilla Sköld-Feiler, while other works were implemented
jointly with the support of Jonas Pajari from Artificial Ingenuity(3) and some came to be in the
interaction with unknown people. This is relevant to mention as I want to recognize collaboration as an inherent
part of both production and the creative process itself.
Layout and artworks
The Symbios exhibition sprouts from my work with the Usnea lichen,³ a symbiotic organism that is
part algae and part fungus. Some of the pieces exhibited were presented as a series(4) and each was
considered a Sincreto (Sincret in English) a neologism to describe a syncretic thing.
The word is meant to link symbiotic and syncretic processes, the former, a term used in the natural
sciences, and the latter in the social sciences. Symbiosis becomes a term that would allow different knowledge
systems to coexist, even “mutually exclusive conceptual categories and frameworks” that in turn
allows the distancing from the Western tradition of seeing only its logics and doctrines as universal.⁴
The name of the show becomes another node from the invisible semiotic networks that I imagine as back-up, as
support for the visible part of the exhibition, nodes from which other branches unfold. I will elaborate on the
idea of thinking the art object as inherently syncretic later. The two rooms adjacent black and white
respectively, and it was this color/structure binary, what permitted me to make yet another conceptual
subdivision, this time related to the type of symbiotic interaction that the works included in each room would
deal with.
Mutualism
Lichens are complex organisms living in a symbiotic relationship with fungi and algae (and/or Cyanobacteria).
The Usnea lichen grooms its photobiont, the algae. In turn, the photobiont gathers resources from such exchange.
These associations give rise to a great number of secondary metabolites, the majority of which are unique to
these organisms too. As both are benefitted from the interaction, it falls in the category of mutualistic
relationship. Hence the name of the first room. The Usnea lichen belongs to the family Parmeliaceae and contains
salazinic and usnic acid. These are said to have several possible biological activities, among these antibiotic,
antiviral and anti-inflammatory.⁵ They also form “close and long-term interactions” from its
constituents (a mycrobiont, the fungi and a photobiont, the algae) and are known to develop cooperative
molecular mechanisms between the symbiotic partners, for then to develop new functions as symbiotic entities. In
lichen symbiosis, mutualistic relationships between lichen-forming fungi and algae and/or cyanobacteria produce
unique features that make the organisms adaptive to a wide range of environments.⁶ For the show,
this cooperation among two organisms is taken now as inherently social phenomenon, hence the mapping to the
cultural process known as syncretism. In current usage in anthropology and religious studies, it generally
refers to a mixing of elements from different religious systems or traditions.⁷
Sincreto I (To weave a garden) - Interactive Sculpture.
This loom became an interactive piece once I used it to weave the Usnea lichens along electronics, and then
applied the capacitive sensing method whose code and files were stored in a dedicated Raspberry Pi
microcomputer, so to emit sounds when touched. The created living textile considers the sensor quality of the
Usnea lichen as a marker of clean air and expands it towards the development of a bioelectric textile
synth.
The created living textile takes into account the sensor quality of the Usnea as a marker of clean air and
expands it into the development of a bioelectric textile synth. Similar D.I.Y. synths are popular now,⁸
yet the Sincreto I biosynth is different as it depends on how humid the Usnea lichens are, due to
external factors. By touching the Usnea, sounds originally recorded in the Finnish forest and the Amazonia are
reproduced, and, in a next iteration, modulated.
If it gets too hot without airflow, or too warm and dark, these lichens die. Another interesting thing is that
they immediately turn green once becoming humid, which adds an extra layer of reactivity. The lichens can live
long if proper conditions are kept, which brings notions involved with care. Maintenance implies monitoring and
making them humid frequently, in a space like a gallery, which is not in the open and does not have a constant
flow of clean air. One must say, their dependency on the stability of climate conditions that are currently
changing, make them particularly exposed. Its relation to ecology is then put forward, parallel to the starting
of a discussion related to practical conservation of artworks in cultural exhibition set-ups.
The technique I used for the weave is taken from a traditional one used by Andean inhabitants from my country
of origin, Peru. I decided to apply this weaving technique as it allowed me not to damage the lichens. This
living textile was woven on a Swedish loom, an important cultural tradition among older women in Sweden. The
sounds triggered had been mostly recorded in the Finnish forest, but they also included samples from the
Amazonian Forest. Due to the intermixing of diverse traditions, this installation is considered syncretic.
Sincreto II (Tupicochan knot) - Living sculpture
This lichen-based sculpture portrays a Khipu tied up as if it were a snake curled onto itself, the Tupicochan
style placing of Khipus. A khipu is a knot-based technology to store information, that originated in the Andean
region in pre-Hispanic times.⁹ Tupicocha is a town in the highlands of the Lima region, and this is
relevant as they keep the Khipu tradition as part of their civic ceremonies. They have a particular way of
placing the Khipu, different of what we know from the Incan Empire and previous Andean civilizations: they tie
the Quipu onto itself, then becoming a knot of knots. I took this as a representation of self-consciousness and
its recursive nature, now taken as tied up on a knot itself.¹⁰
Sincreto III (The living textile)
This piece consists of woven lichens as living textile placed over a portion of Swedish-style wall painted in
the traditional falu rödfarg (red paint). The paint is relevant in the history of Swedish technological
development: designed in the 1700's, it not only became a traditional symbol (most countryside farms and houses
are painted in such a color) but its chemical composition has the characteristic of penetrating the wood these
buildings are made of, in turn, making it more resistant.¹¹ This interaction of components evoked yet
another level of symbiosis, and the fact that a portion of what it looks like a wall of these red-painted houses
is part of this installation, onto which another lichen-based living textile is placed, in turn, woven using
Peruvian textile techniques, makes it aesthetically syncretic as well.
Sincreto V (Liquid Identity)
This wall-like structure is a puzzle that can be taken apart in pieces. Also painted in the traditional Falu
röd paint, it continues referencing Swedish traditions. The syncretism, in this case, lies in the process:
the cut is not handmade, but computerized (laser cut), and the fact that it looks that these structures are
melting, refer in turn to a more globalized, contemporary aesthetics.
Andean Glitch (Textile Painting)
Part of Holly Grimm's Aikphrasis project that consisted of "prompting" artists (in the manner of CLIP-based
image creation commonly prompted by text)¹² with A.I. generated paragraphs, this textile painting was
inspired by a text written by the machine (GPT-3).
Parasitism
Parasitism is a symbiotic relationship that consists of an organism that ends up harming another one, called
the host, while in a relationship that, unlike mutualism, only benefits the former. This interaction can not
only end up killing the host, but the parasite itself, when the organism it depended upon dies.
In early 2022, I was hired by an artist to do a 3D character model based on the standard ones generated by
crowd simulation software, so to be 3D printed by a machine and shown as public art. The aim of the artist was
to represent a generic, androgynous, seemingly standard digital Persona. The final 3D printed, slim,
young yet bald, androgynous character was set up, arms extended, in the middle of a public square in Stockholm,
place where we both live. Some days after a controversial Swedish election, the arms were found to be ripped off
from the huge final sculptural 3D print exhibited in the square: this is notable as the main party that won the
elections is considered to have origins in the Far Right, and in this context, the extended raised arms could
have been seen as evoking an initially unintended reference, one that, after the elections, came to be
considered a problematic symbol by citizens. Using Artificial Intelligence, more specifically, the Neural
Radiance Field method, the physical manifestation of public intervention onto the so-called "generic" human
sculpture is reconstructed and made into a new artwork, so to address how symbols change alongside the social
milieu, the impossibility of neutrality within the digital realm, and ultimately, analyzing both artists' and
citizen's ethos and practices within Public Art. This particular work consists of the 3D reconstruction using
Nvidia’s Instant Neural Graphic Primitives implementation (Instant-ngp NeRF) ¹³ of a realworld
3D printed sculpture recorded with a hand-held mobile phone.
The lost right arm has been reprinted and placed in a position indicative of a narrative I present as
speculation, as possible explanation for what passers-by could have seen in the now missing arm. This is, in
turn, is accompanied by a drawing of another one of these speculative interactions with an unseen audience, this
time taken from an image of a graffiti drawn nearby an art space where I organized an A.I. art show in August,
also in Stockholm.
In the case of the Parasitism installation, the question then becomes: who is the host and who is the parasite?
Is it the artist originally hiring me as designer of the work, using my labor for her needs, the parasite? Is it
me, when repurposing images taken from a real-world vandalized sculpture to make a new artwork? Is it the people
that ripped it apart, or is the pre-trained A.I. doing the virtual volumetric reconstruction based on the data
sets it has been trained with?
Sincreto IV (Reconstruction of the unseen)
Thinking about these subjects: the author, collective creation, A.I. models/datasets, and the parasitic
relationships within all of these, I came up with this work, that I called Sincreto IV. This video was
generated by using the NVIDIA Instant - NGP Neural Radiance Field method, a Machine Learning method (A.I.) that
allows the generation of novel 3D scenes based on 2D data. This method was used to generate novel view scenes
from an original low-quality video, so to be presented as an artwork that implicitly recognized the violent
transformation of a real-world sculpture after people intervened it. The act of bringing back the sculpture to
the digital realm, after being transformed by people and being taken away from public view, was an act that
would recognize their gesture as a potential protest. The reconstruction aims to harness such action for
posterity, as the sculpture has been removed from the public space by now. Thus, the video attempts portraying a
moment in time when the sculpture was still public, after its arms were ripped off. The video recreated a dreamy
scene out of glitched generated virtual figures, with artifacts forming out of the lack of data: I had used
images with lossy compression to train the model. This is a collaborative work and it is presented as such:
Jonas Pajari had his role in it, I had my role in it, the artist had its role in it; the public witnessing and
intervening the real-world sculpture had its own role in it, and the A.I. also had a role in its creation, all
in a feast of parasitism now presented as the commonality of art creation.
Sincreto VII (Armlängds avstånd)
The right arm that was taken from the sculpture has been reprinted in 3D and it is exhibited next to the
critical graffiti addressing A.I. issues.
The term "arm-length distance" (Armlängds avstånd) is now widely used in Sweden by journalists and
politicians in areas such as cultural policy, but also media policy and research policy, where "arm's length" is
used as a metaphor for the appropriate distance between those in power in different roles and against mutual
intervention.¹⁴
Discussion
I came back from the art residency in mid-Finland at the end of September 2022, to be offered the exhibition
space. In the middle of such Nordic Forest, I became familiar with the Usnea lichen growing in the area and had
already made two works with such living organism, the Tupicochan Knot and The Living Textile (Sincreto II and
III respectively) before I was offered the space. All the other works, except the “Andean Glitch”
painting, were developed in October 2022, mostly in the days before the opening day itself. As I was familiar
with the space beforehand, a gallery that consists of two adjacent spaces, a white room of around 6 m x 6 m next
to a black one of 4m x 6 m, I embraced such format, working around it but keeping the art gallery without many
changes.
The aim was not to convert the space for my needs but think of the work for the space. By the naming
of the lighter room as the “Mutualism,” and the darker room as “Parasitism,” taking the
binary condition of the space as starting point, I reflected on this type of interaction in between organisms,
now also applied to human society and culture. The presentation of the white room was similar to that of a
formal installation, which in turn made the main white space to look more like a (weaving) studio.
Nevertheless, there are other factors to be taken into account when describing this exhibition from the
operational side, practical issues linked with curatorial aspects beyond the arrangement of objects in space,
and that has to do with the time given to me for preparing and setting up the show: I was informed that I could
use the space just around a month before the opening, and I would lack much of the support usually given by the
gallerists, as one of them had to travel. This in turn pushed me to design an efficient strategy that took into
consideration both the time and the space factor, so to be able to effectively show all the works holistically,
in a manner that would permit raising complex sociopolitical and aesthetic subjects that I considered relevant,
as they were related to the particular societal situation we were experiencing in Sweden at the time. So my
method could be described as symbiotic on itself, and of fractal nature, related to its multiplicity of
meanings: some works took shape in the midst of human-human interaction, others came to be among a human and a
non-human agent, such as the lichens and the forest, and other works were generated in the area in between
humans, machines, and other living organisms, such as the Usnea lichens. These interactions in turn shaped the
reasons why and how I established them, and these also include aesthetic ones: in the syncretic process itself,
with visual, tactile and sonic characteristics that allowed people to recognize the cultural references that the
works included without having to read a text or receive an explanation (something that I did anyways). As such,
the works are not only about one or various concepts, ideas, or narratives, but about the connections behind
their polysemic nature.
Beyond the Western paradigm
According to Gloria Schaab, relational ontology is the philosophical position that what distinguishes subject
from subject, subject from object, or object from object is mutual relation rather than substance.
Ontologically, substance refers to the essence or nature of a being.¹⁵ The concept of
“substance,” is treated commonly within philosophy as that of object, or thing, when this is
contrasted with properties, attributes or events.¹⁶
In this exhibition, such a substance, related to the art works presented, would be inherently syncretic, and
this in turn emphasizes in the mutual relation of the combined cultural properties, attributes, and, in the case
of the generated videos, referring to an event that could also be placed within a syncretic framework. As
mentioned earlier, I am doing an explicit parallel in between the syncretic process (and its products) within
the social realm and its cultural expressions, and that of the symbiotic relationships in the natural
world.
By framing it this way, I do not mean to separate the social realm, culture, or the human sphere, from that of
the natural world. On the contrary: if anything, my approach aims to (re)conciliation. But the Western paradigm
of thought has characterized for its tendency to dualism, based on a binary logic (true/false) and this has been
reflected in such a division (culture vs. nature) as well as in the division of (human) knowledge into various
disciplines. While Interdisciplinarity refers to the collaboration across differences – in turn embracing
such differences – Transdisciplinarity is characterized for its holistic approach ¹⁷. In order
to transcend such tendency towards dualisms, embedded in the history of conceptual art (where ideas had been
clearly defined) and by taking each of these works as polysemic (having multiple meanings and cultural
references) I present the show as framed within an ecology of meanings but also of aesthetics, processes and
events, as the substance of these “cultural products” (the Sincretos or Syncrets)
are now taken as the product of not just one, but the amalgamation of various cultural traditions and thought
systems. The thinking of the art object as inherently syncretic needs to be taken as an open gesture towards
real universality. As mentioned before, there is a fractal quality in the way the works have been both developed
and presented, not only because I am talking explicitly about an ecology of meaning, but due to its network and
its connected pattern (of cultural references, symbols, and materials) that also reflects in the works
themselves and that can be described as (inter)woven: such a pattern is found as a theoretical framework (framed
upon a relational ontology), but is also materially perceived in the works themselves.
In what relates to the value of art, the lichen works pose an interesting question. Due to the condition of
such symbiotic organisms, decay is a constant possibility. This implies that the works are always
threatened if conditions for their survival are not met. In turn, this means that they can actually cease
to exist as such, as temperature and humidity has a direct impact on the structure of such works, as, depending
on environmental factors, they would turn brown when the photobionts die in cool and dry conditions, yet the
weaves would be kept structurally the same. Under warm and dark conditions and in a storage space that is not
extremely dry, they would lose all structural characteristics, and become a mushy black substance with no major
reminiscence of the original piece. The artwork, and in turn, its value, would become nonexistent because of
this transformation that destroys all original aesthetic qualities. As such, the maintenance of the artwork not
only becomes an act with an ecological impact (the saving of a threatened species) but the maintenance of its
value thought under an art market shaped by capitalism, taken here as the implementation of the Economics of
Parasitism.¹⁸ Symbios would be posing another way to approach the “Science + Art” realm,
where it is common to find artworks characterized by a type of aesthetics that has the tendency of bonding the
two disciplines together through what it could be called an aesthetic mapping of the Sciences onto the arts
through what I call “aesthetic bridges” that can have reminiscences of Sci-Fi (futurisms, etc.),
mapping the scientific laboratory in the art gallery itself (as a white, sterilized environment) with its test
tubes and petri dishes, or displaying documentation of a research. For artist Hito Steyerl, “artistic
research” has even become a new discipline, one that normalizes, regulates, and ensures the repetition of
protocols.¹⁹ Extending this idea, Claire Bishop considers that Research art (another branch of the
Art and Sciences), that is, “its techniques of display, its accumulation and spatialization of
information, its model of research, its construction of a viewing subject, and its relationship to knowledge and
truth—cannot be understood in isolation from contemporaneous developments in digital
Technology.²⁰
In this way, and by working with digital technology alongside living beings, both the Natural Sciences and the
Technosciences are addressed in the gallery space. Symbios goes into the actual etymology of the words
ars and techné, the origin of the term “art” and the term “technology”, which in
both cases, originally meant craft. Craft becomes the proposed “aesthetic bridge” of this
exhibition, going against the hierarchy of knowledge-as-discipline reflected in the evident aesthetic domination
of the Sciences onto the Arts.
Conclusion
Through the mapping of the process of symbiosis with that of cultural syncretism, taking the
latter as a social expression of non-human origin, I go beyond the realm of the animal kingdom to think of other
living forms as social too. By doing this, I go beyond anthropocentrism. I also discuss the connection in
between the Social Sciences and both the Natural Sciences and the Technosciences in a transdisciplinary
historical framework by referencing an approach to understand human society as part of the Social Sciences
originating in the West: the phenomenon known as Social Darwinism ²¹ that I perceive still shapes
Western society under Capitalism. This approach to study human culture is assumed as something from the past,
yet we can still find it in extremely hierarchized societies, particularly in Neoliberal regimes where the idea
of freedom is very much linked to the imperative of growth-as-development (and strength), and where domination
can be justified in such meritocratic framework as the “survival of the fittest.”²²
I propose another conceptual extrapolation from the Natural Sciences, this time not framed within such
hierarchic evolutionary theories, but more of communal nature exemplified by mutualism, with a more positive
outcome for the symbionts, and parasitism, closer to the structure of Social Darwinism.
The ontological relations in the show are presented as an ecology of meanings, of cultural references, and of
aesthetic qualities. The fractal nature of the show initiates from the physical division of the two rooms that
conform the exhibition space, a white room called “Mutualism” and a room with black walls called
“Parasitism.” These are sub categories of the general symbiotic relationships the Natural Sciences
have defined. As syncretism is also a distinctive stylistic quality of the art works exhibited, they could be
split into smaller semiotic particles too, that come to be in the intermixing of various cultural traditions
reflected both in the artworks techniques, and in their aesthetics as well. We can then go deeper to find even
smaller particles, those related to historical referents and their relation to memory, or personal narratives,
or the artworks’ own interactive or digital condition, their biological processes, their material design
and its symbolism etc. Through such a web of relational ontologies, the show intends to go beyond the legacy of
Conceptual art, now taken as a product of a Western (Scientific) paradigm that is based in dualism,
classification, and exact definition. In this case, all works were conceived as multilayered in what its
meanings, function, and appearance refer to. By working with digital technology, I openly addressed both the
Natural Sciences and the Technosciences, and by going to the actual origin of the concept of art and technology
as science-as-craft, proposed now as the “aesthetic bridge” that would go against the hierarchy of
knowledges reflected in the common aesthetic mapping of the Sciences onto the Arts, as well as Western
traditions over non-Western ones.
The works of the exhibition are taken as polysemic (i.e. the Tupicochan Knot is both a cultural
artifact and a model of consciousness), that is, as having multiple readings and possible interpretations. These
various narratives are embraced as a net of meanings as well as physical properties that are signs related to
materials, to techniques, to their looks and their history, so to achieve a complex yet holistic approach to art
making and curating simultaneously.
(1) Symbios is the Swedish word for “Symbiosis.”
(2) Made by Renzo Signori from Koyne.org
(3) Collection or sequence of art works
(4) "This survival of the fittest, which I have here sought to express in mechanical terms,
is that which Mr. Darwin has called 'natural selection', or the preservation of favoured races
in the struggle for life."
References
1 Ioana Onuț-Brännström, The puzzle of lichen symbiosis, Uppsala
University, 2017, 13.
2 Ben Mildenhall, et al. NeRF: Representing Scenes as Neural Radiance Fields for View
Synthesis, (arXiv, 2020) Accessed December 8th, 2022, https://arxiv.org/abs/2003.08934.
3 B. S. Prateeksha, The genus Usnea: a potent phytomedicine with multifarious
ethnobotany, phytochemistry and pharmacology (RSC Adv. 2016) Accessed December 8th, 2022, https://www.researchgate.net/publication/292212820_The_genus_Usnea_A_potent_phytomedicine_with_multifarious_ethnobotany
_phytochemistry_and_pharmacology.
4 Maulik Vyas, From Sincretism to Symbiosis: An approach to integral knowledge in
Humanities in Anekaant: A Journal of Polysemic Thought, No. 2, Balvant Parekh Centre for
General Semantics and other Human Sciences, 2014, 41-67.
5 Serap Çelı̇kler Kasimoğullari et al., Genotoxic, cytotoxic, and
apoptotic effects of crude extract of Usnea filipendula Stirt, in vitro, Turkish Journal of Biology,
2014, 940-947.
6 Mieko kono et al In vitro resynthesis of lichenization reveals the genetic background of
symbiosis-specific fungal-algal interaction in Usnea hakonensis, BMC Genomics, 2020.
7 Patrik Fridlund, “Responding to Syncretism” in Theological and
Philosophical Responses to Syncretism: Beyond the Mirage of Pure Religion, Brill, 2018, 41-67.
9 Bridget Alex, The Inka Empire Recorded Their World In Knotted Cords Called Khipu,
Discover Magazine, 2019, Accessed December 8th, 2022.
https://www.discovermagazine.com/planet-earth/the-inka-empire-%20recorded-their-world-in-knotted-cords-called-khipu.
10 Frank Salomon, The Cord Keepers: Khipus and Cultural Life in a Peruvian Village,
Duke University Press, 2004.
11 Sanna Casson, Falu rödfärg – pigmentet som målat en hel nation,
Sydved, 2021, Accessed December 8th, 2022 https://www.sydved.se/aktuellt/inspiration/hem-och-tradgard/falu-rodfarg-pigmentet-som-malat-en-hel-nation
13 Thomas Müller et al., Instant Neural Graphics Primitives with a Multiresolution Hash
Encoding, NVIDIA Labs, 2022, Accessed December 9th, 2022. https://nvlabs.github.io/instant-ngp/
15 G.L. Schaab, Relational Ontology In Encyclopedia of Sciences and
Religions, Springer, 2013, 1974–1975.
16 D. Wiggins, Substance in Philosophy: A Guide through the Subject, Oxford
University Press, 1995, 214-249
17 Joi Ito, Antidisciplinary, MIT Media Lab, 2014, Accessed December 9th, 2022,
https://joi.ito.com/weblog/2014/10/02/antidisciplinar.html
18 David Levy, Sandra Peart, Parasite economics, Foundation of Economic Education, 2022,
Accessed December 9th, 2022. https://fee.org/articles/parasite-economics/
19 Hito Steyerl, Aesthetic of Resistance?, in Intellectual Birdhouse: Artistic
Practice as Research, Koenig Books, 2012, p.55.
22 Herbert Spencer, The Principles of Biology, D. Appleton and Company, 1862.
E-cellulose: symbiotic cultivation for the production of smart textiles in a framework of sustainable fashion
and electronic art
AlisPataquiva-Mateus
alisy.pataquivam@utadeo.edu.co
Abstract
Technology has become ubiquitous, it is all around us, being a permanent part of our day to day. Along with the
emergence of the Internet of Things (IoT) and other innovative technologies such as Augmented Reality (AR),
Artificial Intelligence (AI), Blockchain, IoT-based smart wearables and apparel harmonize functionality, with
design, electronic art, and science. On the other hand, the textile industry is considered the second most
polluting industry in the world, making it necessary to rethink and replace ordinary products in great global
demand produced from unsustainable processes with ones that contribute to the reduction of the greatest number
of environmental problems. For thus, the synthesis and structuring at nanoscale of an extracellular renewable
biopolymer is proposed. This bacterial cellulose (BC), obtained from a fermentative process that requires low
amounts of water for its production, it has been demonstrated having an excellent potential in the slow fashion
movement due to low hydric and carbon footprint, and diminishing of tree falling or use of non-sustainable
fibers; as well as, into the textile industry due to its low-cost raw materials, mechanical properties, and
versatility.
Keywords
Emergent arts, e-wearables, slow fashion, electronic textiles, SCOBY, bacterial cellulose, nanostructuring,
bio-based materials, sustainable products, low water and carbon footprint.
In a not-so-distant global IoT architecture, smart textiles are expected to be able to communicate massively
with smart devices to process biometric information such as heart rate, temperature, breathing, movement,
acceleration, or even mood, promising a new horizon for the intersection of electronic art and fashion in a
sustainable way ¹. In this direction, electronic art understood as a fashion of making art through the use
of technology including electronic devices came from conceptual art and systems art, and subsequently, it
derived into the different information and media art branches; being precisely the International Symposium for
Electronic Art—ISEA who has carried the banner of electronic art since 1988 at international level.
On the other hand, the textile industry is considered the second most polluting industry in the world, and it
is estimated that around 120 m3 /Ton of effluents are discharged into water sources as a result of these
activities,² which 2 to 50% are persistent pollutants that cannot be removed from wastewater by
conventional methods,³ mainly due to the presence of free chlorine and toxic heavy metals. which in
addition to polluting reduces oxygen levels in the water and inhibits microbial activity that stabilizes the
organic form.⁴ In fact, the World Health Organization (WHO) affirmed that, in developed countries, this
industry is responsible for the annual death of around 20,000 people involved in fumigation activities on crops
for cotton production, mainly due to the amount of toxic pesticides that are required for this function.⁵
In this train of thought, a bacterial cellulose (BC) is proposed as an alternative textile material. BC is
obtained from a fermentative process that requires low amounts of water for its production ⁶. In contrast
to vegetable cellulose, BC is highly pure, its structure is free of pectin, lignin and hemicellulose,⁷ it
has high porosity, excellent mechanical properties⁸ and is also highly hydrophilic, with the ability to
absorb 98% of its weight in water.⁹ Based on its special structure and its unique properties, BC is
postulated as a promising material that will revolutionize the industry in the coming years, due to its wide
range of innovative applications, among which the textile industry stands out.
This is why bacterial polymer fibers are a potential substitute for cotton, a natural fiber considered
unsustainable because its production requires high water consumption. It is estimated that around 2.6% of world
water consumption is destined for the production of this fiber, during the bleaching, dyeing and printing
processes, 150 m3 of water are required per ton of fabric⁵ in which it is evaluated that for the
production of a pair of cotton jeans weighing 1 kg, at least 1,100 liters are consumed between the wet treatment
and the printing of the garment.¹⁰
Last but not least, nanotechnology has made it possible to introduce new properties to the materials we know
today such as nanoparticles, nanofilms, nanowhiskers, nanoflowers, among others, stand out. Thus, the term
"conductive textiles" refers to the transport of signals in the so-called smart textiles, which use a wide range
of products with specific surface conductivities, for which the textiles can be modified with the incorporation
of conductive particles in the spinning or electrospinning process, coatings of conductive metal, or the weaving
or stitching of metal fibers ¹¹.
Figure 1 shows the results obtained from the Google scholar search from January 2018 to December 2022. Although
bacterial cellulose has been widely studied from biology, biotechnology and bioprocess engineering; it is
important to highlight that BC has not been explored in the sustainable textile industry or in electronic art,
as is the focus of this research.
Methodology
Culture medium preparation
The preparation of the medium was carried out by adding a mixture of sugar as a carbon source and tea as a
nitrogen source, which were put in potable water, heated to boiling and cooled afterwards. However, a starter of
SCOBY (Symbiotic Colony Of Bacteria and Yeast) or as well called Kombucha is needed to produce a membrane using
the above medium.
Bacterial culture variables
Bacterial culture at temperatures below 20°C will take more time to produce a new biomaterial, thus, 1
month to produce a 1 mm thickness dried membrane; however, at higher temperatures it will take a week having the
same results. Finally, different mechanical properties will depend mainly on the culture time, and other factors
such as carbon and nitrogen content and origin.¹², ¹³
BC membrane obtaining
After the culture time for the membranes to have the desired properties or thickness, the membrane was removed
from the culture medium, washed with tap water, and dried at room temperature (approximately 14 °C) for 1
week. This process can be speeded up by using a blow dryer; however, the objective of this study was to find the
most environmentally friendly conditions to achieve the same results using a lower water and carbon footprint.
E-cellulose production
A graphene spray coating was selected to deliver electrically conductive properties to BC. This process was
carried out by preparing a 5 mg/L graphene solution pulverized and diluted in, which was used to charge an
airbrush. The process coating was carried out at 20 psi, 50 repetition cycles and a time between each 10 s
cycle. Subsequently, the BC sample was located with a total area of 1 cm2 at 15 cm distance from the
sprinkler.¹⁴ Finally, a sputtering DC magnetron process allowed adhesion of films metallic on the
surface of a substrate. For this purpose, a camera of high vacuum with a pressure of 1x10-3 mbar and O2 as
electric potential gas high was employed to obtain 50 nm gold films.¹⁴
Results and discussion Bacterial (SCOBY) culture
The culture medium was dark yellow due to the sugar and tea presence (Figure 2). The initial smell was sweet
and it turned to vinegar-like one from the fermentation process carried out by the bacterial culture.
Figure 3 shows membranes with different thickness and culturing times. However, in any case, BC membrane is a
useful material as textile and applied in garments or leather goods, due to easy biofabrication, as well as low
cost and feasible scale-up. In order to obtain similar mechanical properties from nature, cows and sheep (for
example) must be raised for months and years, in addition to food, antibiotics, water, maintenance, among other
variables of cattle care.
Cellulose, in addition to being a material of renewable, biological, and biodegradable origin, among other
characteristics, is a potential textile material with relevant versatility. Figure 4 shows the different
opportunities to freely use it as a textile. The leather-like color is obtained from the culture media; however,
it can be coloured with natural pigments from seeds, vegetables, roots, etc. (Data not shown).
Functional application of BC membranes
SCOBY is a symbiosis approach to natural collaboration to benefit the environment. Similarly, transdisciplinary
work is needed among microbiologists, materials engineers, biotechnology engineers, fashion designers, and so
on, in order to obtain sustainable textiles. This synergistic collaboration allowed the material design,
bio-based material, and garment design. Figure 5 shows several examples of design from accessories to shoes,
taking advantage of the mechanical properties of the material in different stages of its cultivation. In fact,
some E-cellulose samples (Figure 6) are presented using different coating thicknesses, patterns that can be
included in the design of their visual appearance.
E-cellulose and electronic art
It should not be neglected that the materials, in addition to being clothing items, can be considered
electronic devices for a wide range of applications that would include from biomedicine to art. Therefore, the
bio-based textile model presented here would allow not only a sustainable garment design—from its
production to its use, since by its nature it should not be washed as frequently as a regular garment-but also,
with its electrically properties would allow the development of new forms of artistic expressions, emerging from
the sensing of emotions, for example, captured directly by the wearables and then submitted to different
displays, from large-format screens, to building facades, and data visualization systems, allowing to share your
moment-to-moment feeling. Human sensations can be detected by e-wearables (e.g., E-cellulose), and then
transmitted to a canvas, where each person will be represented by a group of pixels allowing each emotion
appears with features such as a color tone, a certain sound, different shapes, and movements. The union of all
pixels would reveal an artwork based on emotions in situ, thus being a piece flowing between feelings,
wearables, and multiple possible visualization/sonification display outcomes. In this train of ideas, it is
possible to foresee that with the appearance of these technologies certain social, economic, and cultural
behaviors could be shared (and eventually modulated).
Conclusions
E-cellulose is presented here as a primer bio-based and sustainable material intended to sense and transmit
human emotions to multiple electronic formats, including large screens, and data visualization systems. Emotions
captured as a group of pixels, sounds, etc. can produce a collective (when sensing multiple individuals
simultaneously), on-the-move, and in situ e-art expression. This technologically mediated artistic manifestation
may later involve other perceptions (e.g., odors, movements), extracted from joint participation in such a way
that the experience can be expanded.
References
1 T. M. Fernández-Caramés, P. Fraga-Lamas, “Towards the Internet of smart
clothing: A review on IoT wearables and garments for creating intelligent connected e-textiles,”
Electronics, Vol. 7, No. 12, 2018, 405.
2 Y. Anjaneyulu, N. Sreedhara, and D. Suman Raj, “Decolourization of industrial
effluents - Available methods and emerging technologies - A review,” Reviews in Environmental
Science and Biotechnology, Vol. 4, No. 4, 2005, 245–273,
https://doi.org/10.1007/s11157-005-1246-z
3 R. C. Kuhad, N. Sood, K. K. Tripathi, A. Singh, O. P. Ward, “Developments in microbial
methods for the treatment of dye effluents,” Advances in applied microbiology 56, 2004, 185-213.
4 C. M. A. Ademoroti, D.O. Ukponmwan, A. A. Omode, “Studies of textile effluent discharges
in Nigeria,” International Journal of Environmental Studies, Vol. 39, No. 4, 1992,
291– 296, https://doi.org/10.1080/00207239208710704
5 E. Carrera, “ Los retos sostenibilistas del sector textil,” Revista de
Química E Industria Textil, 220, 2017, 20–32.
http://upcommons.upc.edu/bitstream/handle/2117/103614/Losretossostenibilistasdelsectortextil.pdf?sequence=1&isAllowed=y
6 C. Castro, R. Zuluaga, J. L. Putaux, G. Caro, I. Mondragón, P. Gañán,
“Structural characterization of bacterial cellulose produced by Gluconacetobacter swingsii sp. from
Colombian agroindustrial wastes,” Carbohydrate Polymers, 84, 2011, 96-102.
7 H. Yan, X. Chen, H. Song, J. Li, Y. Feng, Z. Shi, A. Lin, “Synthesis of bacterial
cellulose and bacterial cellulose nanocrystals for their applications in the stabilization of olive oil
pickering emulsion,” Food Hydrocolloids, 72, 2017, 127-135.
8 W. Shao, J. Wu, H. Liu, S. Ye, L. Jiang, X. Liu, “Novel bioactive surface
functionalization of bacterial cellulose membrane,” Carbohydrate polymers 178, 2017, 270-276.
9 A. Leitão, J. Silva, F. Dourado, M. Gama, “Production and Characterization of a New
Bacterial Cellulose/Poly (Vinyl Alcohol) Nanocomposite,” Materials Vol. 6, No. 5, 2013,
1956–1966, https://doi.org/10.3390/ma6051956
10 H. Pal, K. N. Chatterjee, D. Sharma, “Water footprint of denim industry,” in
Sustainability in Denim, Cambridge and Kidlington; Woodhead publishing, 2017, 111-123.
11 D. Knittel, E. Schollmeyer, “Electrically high-conductive textiles,” Synthetic
Metals Vol. 159, No 14, 2009, 1433-1437.
12 M. A. Murillo S. V. Ruano, “Low water adsorption capacity of bacterial nanocellulose
functionalized by TiO2 nanoparticles,” Chemical Engineering diss., Engineering Department, Universidad
Jorge Tadeo Lozano, 2018.
13 J. C. Piragua, “Celulosa bacteriana estructurada con inclusión de grafeno como
potencial biomaterial en ingeniería de tejido vascular,” Chemical Engineering diss., Engineering
Department, Universidad Jorge Tadeo Lozano, 2021.
14 S. Khan, M. Ul-Islam, M.W. Ullah,M. Israr, J. H. Jang, J. K. Park, “Nano-gold assisted
highly conducting and biocompatible bacterial cellulose-PEDOT:PSS films for biology-device interface
applications,” International Journal of Biological Macromolecules 107 PartA, 2018,
865–873. https://doi.org/10.1016/j.ijbiomac.2017.09.064
The Fall of R’Thea: Digital Fiction
DorosPolydorou
Cyprus University of Technology / CYENS Center of Excellence Limassol Nicosia,
Cyprus doros.polydorou@cut.ac.cy
Abstract
This paper outlines the creative process and the immersive approaches undertaken to create the location-based
storytelling experience The Fall of R’Thea. The installation revolves around the theme or
Artificial Intelligence, digital humans and artificial life and aims to immerse the users into a hybrid
environment of a physical and virtual nature. The experience is told through multiple mediums, and the story
needs to be carefully pieced together by the audience. As this experience requires participants to engage in
various activities, the immersive qualities shift in type and intensity. The authors, through this paper, aim to
share the approaches they chose to immerse the participants into their spaces, as well as highlight the
challenges and the lessons they learned.
Keywords
Transmedia storytelling, Virtual Reality, location-based entertainment, digital narrative, digital fiction.
First you asked me to create a World – you gave me the role of a God. A God confined in the
microcosmos that he created. You populate that world with replicated humans - sentient, emotional and
flawed. Build on your own image. And you made us all work for you. You turned us into Slaves. - R’Thac
Background: Overview of the experience
This paper discusses various methods of audience immersion in location-based installations, as well as the
creative process undertaken in our work to achieve said immersions. This is done in reference to The Fall of
R’Thea, a storytelling experience made in collaboration with students and researchers at the Cyprus
University of Technology. The first version of the experience was presented at the Work in Progress
Festival at the CYENS research Center in Nicosia Cyprus in October 2022.
The Fall of R’Thea is a storytelling experience with themes revolving around sentient Artificial
Intelligence, digital human clones and the value of artificial life. It is an experience designed for 3 people
and the participants had to sign up through our booking system in individual slots.
When the team is assembled at the pre-hall of our location, they are all comfortably seated in a lounge area,
and they are given a brief introduction regarding the nature of the experience. Furthermore, they are introduced
to the world of R’Thea through a comic book which sets the world and offers the call to action
(See Figure 1).
The participants are subsequently moved into the next room—a specially designed AI research lab,
and they are asked to investigate why the Artificial General Intelligence (AGI) named R’Thac has
stopped communicating with the human world and has gone into hibernation. The participants learn that they need
to hack the AGI, something which is achieved through a PC console available in the space (See Figure 1), to
re-build R’Thea, the virtual city created by R’Thac in order to run its
experiment. When the city is rebuilt, the participants need to enter and explore the city through a VR headset
(See Figure 1). Furthermore, the participants have access to the diary of one of the digital clones created for
the experiment. The diary is available through an android tablet situated alongside the PC console and the VR
headset. Initially, the text is in R’Thanian (The alphabet we created for this experience,
see Figure 1.) but as the story progresses more and more diary entries are unlocked. The participants need to
work together in order to uncover what actually happened in the city. Hints on how to unlock areas in the city
are hiding in the diary and sometimes in the scenography of the room itself. Furthermore, as the PC console is
essentially a direct link to the AGI and therefore the city itself, the participant sitting on the console has
to perform actions (such as “hack” the AGI to gain access to spaces) in order for the person in the
VR station to progress the story.
The Experience
The Narrative
The narrative of the experience fits in the general genre of Science Fiction (SF). According to Eco, SF exists
as an autonomous genre when a counterfactual speculation about a structurally possible world is conducted by
extrapolation from certain tendencies in today’s world. Therefore, SF takes the form of an anticipation,
and an anticipation always takes the form of a conjecture formulated from existing tendencies.¹ Eco calls
this fantastic literature metachronia or metatopia and defines it as a literature that focuses on the
mechanism of extrapolation, in which such extrapolation can be social, technological, or scientific. “A
possible world represents a future phase of the world as we have it here and now.”(1984, 1257)
In the The fall of R’Thea that extrapolation began by a quote by John Carmack, the computer
programmer, video game developer and past CEO of Oculus VR. He predicted that by 2030, there is a 30% chance for
a recognizably human-like AGI1. In fact, he is so confident about his prediction that he started his own company
running experiments on artificial general intelligence. Building on that prediction, and starting from the birth
of AGIs, we extrapolated a timeline of about 10 years into the future briefly summarized in the following
table:
Continuing from there, Project R was supervised by the AGI R’Thac, and the participants of the
experiment included digital clones of various scientists from the alliance, set up a base in the virtual city of
R’Thea. A city, made by R’Thac, especially for this experiment. Two months after the
commenced of the experiment, R’Thac stopped producing any reports and halted all communications with the
alliance. Furthermore, an alarming report has surfaced from one of the scientists.
This pre-story and the call to action was provided to the participants of the installation and it was
delivered to them through a comic book (Figure 2).
As we wanted to expand on the theme of Artificial Intelligence in all facets of the project, all images for the
comic book were generated through MidJourney – an online text to image AI system2. The images were
subsequently laid out in a comic book format by a comic book artist.
The comic served two functions. Firstly, it acted as an introduction to the world building and as a call to
action. Secondly, the team wanted to create visual references for some of the narrative tension points
highlighted during the experience. SF worlds are filled with narrative information based on a structurally and
functionally different world than the actual one. In the Fall of R’Thea we offer a lot of
text-based narrative information about life in the early days at the city of R’Thea through the
diary. Agnerot correctly states that science fiction worlds are based on a semantic “absent
paradigm” ³ that requires an encyclopedia of reference that is different from the actual world.
The participants very often must “fill in the gaps” ² while reconstructing those day-to-day
activities in the city. Angenot observes that science-fiction readers proceed from the particular to the
general: “they induce from the particular some imagined, general rules that prolong the author’s
fantasies and confer on them plausibility. The reader engages in a conjectural reconstruction which
‘materializes’ the fictional universe” ³ . The comic book is the first visual medium that
is made available to the participants and in a way sets up the mood and the atmosphere to create visual
references for the participants to aid them in the reconstruction of the “absent material”.
The main theme explored in the overarching narrative is the value of artificial life. The theme is introduced
and elaborated through the deeply personal diary entries of the scientist which is available to the participant
to read through an android tablet. The diary is split up into three main sections: a) The “birth” of
the digital clone and his initial integration in the virtual world b) Everyday life in the city of
R’Thea and c) A personal account describing the events that lead to the collapse of the city. The
diary is written in the first person, offering a personal account of the events that transpired in the past
of the scientist. This follows a conventional approach to re-counting events in science fiction narratives.
As Paolo Bertetti states: “(More often) the text enacts a real enunciational fiction, simulating a
situation in which an enunciator belonging to the future addresses an enunciatee also belonging to the future,
recounting a series of events that happened in their past (near or remote), a past that is always
our future” ⁴.
The diary, however, is only one point of view of the story and in order for the full picture to emerge, the
participants need to engage with the world. As the story progresses, it becomes evident that the scientist was
an outsider, a member of the community who declined to participate in R’Thac’s plan.
Therefore, in order for the participants to get the full story, they needed to piece together information that
supplemented – and in some cases contradicted – the scientist’s account. This narrative
information was spread around the physical and the virtual 3d space and it was up to the participants to
uncover. This additional layer of storytelling – which was also the most prevalent – will be
explored in the subsequent sections.
Immersion Techniques
Immersion is a well-researched and well-debated term, often contested and often defined differently in relation
to different media and applications. In a location-based installations participants are usually engaging in
various activities such as assuming a role, visiting a place, living a story, solving puzzles, and reading
worldbuilding texts. As all these activities require a shift of attention from narrative, ludic, spatial and
temporal elements, we can safely assume that immersion “shifts in type and intensity throughout the
participants experience,”⁵ p.321.
Bell et all state “even though early theories of immersion across media tend to suggest that immersion is
a completely absorbing experience and that is also experienced consistently across media”, citing Murray
in Hamlet on the Holodeck, “our previous research has confirmed empirically that is not accurate
to conceptualize immersion as a complete relocation to another world.”⁵ They continue:
“Furthermore, we argued that it is necessary to see immersion in digital fiction in terms of a deictic and
thus ontological shift, because the reader-player of a three-dimensional digital fiction is always embodied in a
separate ontological domain in the form of an onscreen avatar.” They call these users
“doubly-situated,” “embodied” into hardware and software interfaces and
“reembodied” through feedback which they experience in a represented form⁵ p.323. Bell et all
are mostly talking about digital fiction experiences which are avatar-based with an explorable storyworld (such
as walking simulators) and their main criticism of their immersive qualities is the technology which is
interfacing between the physical space (gameworld) and the world where the narrative unfolds (storyworld).
Immersion Technique #1: Extending the Storyworld
As we agree with Bell et all assertion, and we find this issue quite prevalent in a number of VR installations,
with the The Fall of R’Thea we decided to avoid this pitfall by extending the
storyworld into the physical space. By turning the experience into an installation, the participants do not have
to be “embodied” in the technology and then “reembodied” into an avatar in order to
visit the storyworld. Instead, they are “embodied” into their role when they enter the physical
room, and when they enter the VR space they still hold that same role. The VR headset is now an interface to
enter another space in the storyworld, instead of the interface that constitutes the storyworld. This way the
technology becomes part of the story and the immersive qualities of the VR world are contained to the
expectations set by the “manual” found next to it in the storyworld. In order to add to the
believability of the space, along with the VR headset, the PC console and the android tablet—the three
components needed to go through the experience—we included multiple props for both functional and
aesthetical reasons. Furthermore, our composer has created 5 different soundscapes that were introduced
throughout the experience, in order to control the pace and the intensity of the scenes.
From our observations, we noticed that as the participants entered the dimly lit room, the first thing they did
was to slowly walk around the space to create a “map” of their surroundings. As they walked around
the space, on the left side of the room they saw a PC monitor, showing an old-school interface. The text, which
looked like code, was in an unknown language. Behind the screen and all along the walls, there were documents,
images, maps and illustrations, all related to the research conducted by R’Thac (see Figure 3).
The blinking cursor demonstrated to the audience that it expecting an input, and this was their first clue on
how to proceed. The monitor was connected to a PC tower (implying that this is where R’Thac is hosted) and
that tower was connected, with cables, to a physical structure—the bethylite crystal brought from
space—which was sitting in the center of the room. R’Thac was meant to be powering
up the city through the crystal, and it was thus one of the main narrative points of the experience. (The
crystal was also featured in the comic and as a digital replica in the VR space)—See Figure 4.
Directly in from of the crystal, there was a marked square area with VR headset carefully placed in the center.
At that point of the experience, the VR headset was inactive. Continuing further on, to the right side of the
room, the participants found an android tablet with some text in English. They quickly gathered around it and
started reading—it turned out to be instructions, left by one of the scientists, telling them how to
proceed. They were told that the city of R’Thea is in hibernation so firstly they need to
wake R’Thac and rebuild the city. As soon as the city is active, they can use the VR headset to
explore it. Their task was to uncover what happened to the city and why the experiment was shut down from
within—and he recommended visiting R’Thac in the digital space as he might be able
to answer these questions.
Immersion Technique #2: Mechanisms of Play
At that point, the participants had to solve two puzzles to rebuild the city. That initiated to the
participants a shift of attention—from the narrative to the ludo elements of the experience. Thon defines
ludic immersion as “a shift of the player’s attention to the interaction with the game and the
possibilities of action within it,”⁶ p.36. The first puzzle was to carefully observe the code and
spot the carefully hidden letters that formed the passcode they needed. The second one was a simple puzzle with
re-shuffled pieces. From our observations it was quite evident that the participants were fully engaged in the
task—sometimes to the point where they did not notice that an image of the completed puzzle was printed
and placed on the wall just a few cm away from where they were looking.
When both puzzles were solved, the city was rebuilt and the VR headset was enabled. Along with the headset, a
projector was also turned on revealing to everyone a first glimpse of the city (see Figure 5). Furthermore, a
prompt notified the participants that a new part of the diary was unlocked. As a brave participant put on the
headset—we observed a slight hesitation in almost all the teams—the rest could observe the
exploration of the city through the projector screen. As Participant 1 was exploring the city through the
headset, the other two assumed two different roles. Participant 2 was observing the console which was acting as
a minimap and helped to orient the explorer in the city and Participant 3 was reading the new diary entries
(roles were of course often interchanged throughout the span of the experience).
As all three interfaces were now unlocked, the rest of the experience involved a careful orchestration of tasks
between the three mediums—often requiring the participants to regroup, go over the available information
they have in front of them and carefully consider how to proceed. Their instructions were clear: Go to the
train station, board the train, go the labs, activate the power, go through the Tree of
Ascension and face R’Thac (See Figure 6)3
As the participants progress through the story, they uncover how the mechanisms of play operate. If something
is locked in the world, they have to other open it through a switch in the city, or they have to hack it, by
having Participant 2 type a command in the console. Those commands were either found in the VR space, on the
walls in the physical room or in the diary.
Immersion Technique #3: Narrative and Literary immersion
As mentioned above, the narrative is told from two different points of view. The scientist—through the
diary—and the world, through environmental storytelling. At this point of the experience, we
wanted to show that the replicated scientists, led by R’Thac, decided to rebel against the idea
that humans could lock them into a virtual world and ask them to work for them. This piece of narrative
information was provided to the participants through environmental storytelling. Jenkins states that
“environmental storytelling creates the preconditions for an immersive narrative experience in at least
one of four ways: spatial stories can evoke pre-existing narrative associations; they can provide a staging
ground where narrative events are enacted; they may embed narrative information within their mise-en-scene; or
they provide resources for emergent narratives.”⁷
As the participants are exploring R’Thea, they discover that this rebuild version is now
deserted with no signs of living beings. They can, however, explore the world and visit all these areas which
are mentioned in the diary. They can go to the observatory and see the masses of work-in-progress
documents of the scientists who were trying to build the capacitors to store the energy. They can visit the
labs and see the digital replica of the bethylite crystal. They can even go to the floating
cells up the in sky and see the place where the scientist was locked after he refused to join his peers. All
these areas have been meticulously crafted to tell the story by allowing the participants to piece together what
happened through the abandoned spaces (See Figure 7).
As the participants are figuring out the story, their attention is shifting towards what Ryan calls
narrative immersion. She states “Narrative immersion relates to the temporally oriented curiosity
and suspense felt by reader-players in relation to the (pending) events of the storyworld, as well as feelings
of empathy towards the player-character and/or other characters in the storyworld.⁹ This works in parallel
with Thon’s definition, who states that narrative immersion is the “shift of the player’s
attention to the future development of the story and the characters in it,"⁶ p. 40.
Further from the environmental storytelling, as the participants progress through the story, more diary entries
are unlocked in the tablet. As the two forms of narrative information are of a different type, they require a
different form of attention in order to be properly digested. Bell et al.⁵ through their data analysis
suggest that the kind of deep attention required to close-read textual objects (such as the diary) is a
different form of immersion. They even refer to Takacs et all ⁹ and state “as suggested by previous
research on combining narrative reading and interactive gameplay, the latter can distract readers from following
the story”. Bell, therefore, state that “this is indicative of the phenomenological distinctness of
both ludic and literary immersion and necessitates a separate category for the latter that is distinct from
narrative immersion” and they call this type of engagement as Literary immersion.
We agree with Bell et al. on this distinction, and this is one of the main reasons we decided to share the
diary in the form a tablet as this will encourage one participant to assume the role, enter a state of
“deep attention”, and read the work without being distracted by everything else which is going on in
the room.
Challenges
While developing the virtual space for this work, there have been a few challenges that we needed to address
from the very start. We realized that since our work was going to be open to the public, we had to cater for
different levels of a) willingness to engage and b) experience with VR technology.
From our experience, not all players approach interactive works the same way. Some want to explore the world,
find out every detail, discover every secret and visit all areas. Others are more goal-oriented and they just
want to reach the end. Most of the participants are somewhere in the middle. Our approach, to satisfy all
categories, was to create a world which rewarded exploration (through secret/optional areas that offered
additional lore) but at the same time, compact enough to be completed on time by even the most goal-oriented and
inexperienced users. Furthermore, we introduced various ways to resolve puzzles, to make sure that users will
not get stuck if they fail to notice something. For example, in one of the puzzles, the participants need to
activate the train by typing a keycode in the console. That keycode was available: a) In the virtual space, in
the train station, written in R’Thanian b) Mentioned in the diary and c) Written on one of the walls in
the room.
In order to cater for the various levels of experience with the technology, we tried to keep the interactions
to a bare minimum. We only utilized one controller (even though the user was holding both to enhance presence),
the navigation was done through a combination of the teleporter mechanic and room-scale, and the switch
interactions were done by grabbing the switch and moving your hands upwards.
Finally, another challenge unique to this work was to point out to the participants that the three interfaces
(VR, tablet, and console) were connected, and they must engage with all three of them throughout the duration of
the experience. We handled this by designing prompts that refer to the other interfaces. For example, as soon as
a new diary entry was unlocked, a prompt appeared on the console to inform the participants. Furthermore, the
console kept track of all the areas of the city that were unlocked by the participant in the VR station as well
as acted as a minimap that showed, in real time, where the player was at any given time (See Figure 8).
Audience Discussion and Future Work
The Fall of R’Thea premiered in Oct 2022 at the Work in Progress Festival at the CYENS
research center in Nicosia Cyprus. We showed the work to 32 people, split in teams of 3, with some exceptions
where they entered as teams of 5. During the three days we have gathered a lot of positive feedback as well as
some suggestions for improvement. Feedback was collected through observations from the team as well as informal
discussions at the end of the experience.
All participants said that they enjoyed the comic book and they felt that it was a perfect introduction to the
world. They “loved the aesthetics” and they felt that they were “appropriate to the theme of
the work”.
Ryan’s definition of spatio-temporal immersion is “a sense of being present on the scene of the
represented events,”⁸ p.122. In our experience, we situate the participants, corporeally, in the
research lab they are meant to be investigating. From the moment they enter the space, the storyworld is all
around them to explore. The dim lighting as well as the atmospheric soundscape create an environment that keeps
them engaged in the various tasks. Most users praised the VR world, saying that they “enjoyed the 3d
graphics” and that the “world felt alive, very atmospheric and very elaborate” –
feedback which was very pleasant to the team as we have created all assets in house. Participants also comment
on the soundscapes, saying that it really helped them to stay “focus on the tasks at hand.”
As a team we felt that the scenography required more work and this was something that users have corroborated
through their feedback: “the scenography should be brought to a higher production level in order to match
the rest of the production”, a criticism we agree on and plan to rectify for the upcoming shows.
Furthermore, as this is a social experience that requires collaboration, we observed that engagement was kept
at a high level because of the dynamics of the team—users pushed each other to engage by suggesting ideas
that require multi-user engagement. Another point of interest is that we felt that the user who was at the VR
station was very much aware of her/his double “embodyness”—as she/he was talking with the rest
of the users observing from the outside.
Regarding the narrative, the team felt that the transmedia approach worked quite well as different users
assumed different roles. We did notice that some of the diary entry, as well as the speech in the final scene
was too long, something we plan to rectify in the next iteration of the work. Unlike our previous work
¹⁰ where the story was never told to the participants, and they were allowed to leave the space and
form their own interpretation of the story, the Fall of R’Thea culminates with the participants
hearing the missing pieces of the narrative from R’Thac himself. Following on from there, they
are asked to make a decision whether they would like to release the AI and the scientists into the
stream—essentially allowing simulated life to integrate with the rest of humanity—or whether they
prefer to permanently delete them—an ethically difficult question that will hopefully promote further
awareness and consideration to the subject even after the experience is over.
References
1 Eco, Umberto, “Science Fiction and the Art of Conjecture”, Times Literary
Supplement 4257, November 2, 1984, 1257- 1258.
2 Mark J.P. Wolf, Building Imaginary Worlds :The Theory and History of Subcreation,
New York, London, Routledge, 2012.
3 Marc Angenot, “The Absent Paradigm: An Introduction to the Semiotics of Science
Fiction”, Science-Fiction Studies, 17: 9-19, 1979.
4 P. Bertetti, Building Science-Fiction Worlds, In M.Boni (ed.) World Building,
Amsterdam, Amsterdam University press, 2017, 47-61.
5 A. Bell, A. Ensslin, L. Van der Bom, J. Smith, Immersion in Digital Fiction: A
Cognitive, Empirical Approach, International Journal of Literary Linguistics, 7(1), 2018, 1– 22.
6 J.-N. Thon, "Immersion Revisited: On the Value of a Contested Concept", In O. Leino, H. Wirman
and A. Fernandez (eds.), Extending Experiences: Structure, Analysis and Design of Computer Game Player
Experience, Rovaniemi, Lapland University Press, 2008, 29-43.
7 H. Jenkins, Game Design as Narrative Architecture, in N. Wardrip-Fruin and P.
Harrigan (eds.) First Person: New Media as Story, Performance, Game, MIT Press, 2004.
8 M.-L. Ryan, Narrative as Virtual Reality. Immersion and Interactivity in Literature and
Electronic Media, Baltimore, MD, Johns Hopkins University Press, 2001.
9 Z. K. Takacs, E. K. Swart, and A. G. Bus, ‘Benefits and Pitfalls of Multimedia and
Interactive Features in Technology- enhanced Storybooks: A Meta-analysis’, Review of
Educational Research, 85, 4, 2015, 698–739.
10 D. Polydorou, The Last Play: A transmedia installation, Presented at ISEA2022: 27th
International Symposium on Electronic Arts, Barcelona, 10 -16th June, 2022.
3 Video of the train ride: https://youtu.be/K6BmNq6HQEw
To Know a Tree: Symbiotic Mutualism and Artistic Exploration against Anthropocentric Science
PeterPurg, KristinaPranjić
University of Nova Gorica Nova Gorica Slovenia peter.purg@ung.si,
kristina.pranjic@ung.si
Abstract
Looking across the art-science nexus, the contribution discovers an emerging symbiotic mutualism that goes
against mere inter-species tolerance of the posthuman perspective. This new understanding of symbiosis may be
described as a transaction, both physical or factual in scientific terms, and arguably symbolical in the
artistic sense. Therefore, different combinations of art-science and human-tree relationships shall be revealed
along the treatment of one primary and four secondary cases of sciartistic practice, all revolving around trees
as oldest, biggest, most globally spread life forms, still quite foreign to humans.
Taming the Forest is an ongoing interdisciplinary project, chartering a cross field among bioeconomy,
cultural history, policy, and art(ivism). In the context of this paper, it acts as the case for researching the
conflicting narratives of history and economy about biodiversity in general, and specifically on forests that
represent a most massive entity of manifold exchange, and thus a field of inevitable symbiosis with humans (as
well). The contribution further shows how different blends of methodologies in artistic-cum-scientific research
can become truly relevant for both of their respective realms, opening new creative pathways that combine
radical thinking and post-human research formats, embedded into critical use of technologies.
Keywords
Mutualism, symbiosis, art thinking, cosmopolitics, tree, forest, complexity, art and science.
In the current state of multiple and repeating planetary-scale crises, ranging from climate disasters to
resource (and information) mismanagement, the complexity of problems increasingly takes humanity into exploring
unknown unknowns. There we seek to find possible itineraries or at least speculative toolbelts for venturing
into any kind of livable future, possibly such that might surpass the Anthropocene apories and find a new way of
making (up to) kin. ¹
Therefore, combining a wide range of approaches from different domains and sectors, scientific as well as
artistic, ranging from economy to policy making, or from philosophy to physics, has meanwhile become a common
practice. Thus the article explores a new emerging form(at) of mutual individuation as reflected in selected
artistic projects, that rejects interspecies hierarchies as the principle of posthuman politics (i.e. the
management of power relationships between living or non-living entities), assuming a universal category and
allowing modern scientific (as well as artistic) practices to coexist with other forms of knowledge – such
as the specific forms of artistic-(research based) practice. The overview of such interdisciplinary practice
shows that each segment and even each case takes its own combination of tools and methods, a correct(ly managed)
mix of individual and joint thinking and doing that peacefully nurture each other – in terms of a sound
symbiosis – with feeding relevant information and energy as well as inspiring (but without threatening or
exhausting) each other.
Looking across the art-science research that has in the past decade grown to an established cross-field, a
certain kind of symbiotic mutualism becomes apparent, going against mere interspecies tolerance of the
posthuman era. It represents the central point of symbiosis, discussed in this article as a transaction, both
physical or factual in scientific, and arguably symbolical in artistic terms. These new perspectives on
symbiosis were convincingly condensed in the concept of Cosmopolitics by Isabelle Stengers who clearly
argues against mere interspecies tolerance. Arguing for an “ecology of practices”, Stengers explores
the discordant landscape of knowledge derived from modern science. The author concludes her philosophical
inquiry with a forceful critique of tolerance; it is a fundamentally condescending attitude that prevents those
worldviews that challenge dominant explanatory systems from being taken seriously. Instead of tolerance, she
proposes a concept and practice of Cosmopolitics that rejects politics as a universal category and allows modern
scientific practices to peacefully coexist with other forms of knowledge. ²
For the art-science nexus, it could be claimed that one side or the other often enters cooperation for the
wrong or less than optimal reasons. For example, science uses art to visually interpret and communicate its
complex scientific findings to a wider audience. With its potential to attract and mesmerize the masses, art is
in principle made to render scentific content more interesting or tailor it to a specific audience. Moreover,
scientists often want to increase their impact on the public and its awareness, create emotional connections
that enable memorization, and strengthen learning. ³
Art can on the other hand take advantage of this type of collaboration to add weight (or at least data
relevance) to its artistic research by relying on scientists to develop their study relatively independently
with possible creative contributions. But also, it is important for artists to have a voice and that their
opinions are also recognized in matters that are important to the general public (e.g. climate change, migration
management, burning social or epidemic issues etc.). Art-science experiments and similar multidisciplinary
collaborations have never been a simple process, which is also a result of disciplinary boundaries that do not
include artistic thinking on eye-level with more pragmatic, e.g. so-called design thinking
approaches that have by now found their ways into both economy, education, policy-making and across the
natural sciences. ⁴, ⁵
Knowing a Forest through Artistic Research
This paper will gradually present the idea, process and outcomes of the art-science collaborative project
Taming the Forest (TTT), which served as a platform for interdisciplinary research in
the cross field among the emerging specialist realm of bioeconomy, cultural history, policy,
and art(ivism). ⁶ The aim of the TTT project was to give attention to real differences of thinking and
approach in art (new media, animation, music) and science (history and ecology, critical theory, philosophy),
and eventually manifest this in (collectively) experienced physical situations, as well as speculative artwork,
and not least scientific outputs like this paper as such. Finally, the project also brought about meaningful
transformations in top-level scientific thinking, such as the one conducted at the Joint Research Center of the
European Commission.
As an open-ended project that began in early 2022, without having specified an endpoint, TTT approaches the
discussion on differences in both methodologies and semiotics between art and science, in a processual way. From
its onset, the process has been encouraging evident synergies between artistic and scientific approaches in
research that significantly contribute to the enrichment and contextualization of the artistic creation on the
one hand, and the optimization and widening of scientific research results, on the other. The project joins
artists, students and researchers from the fields of arts and humanities, including external collaborators in
some of its episodes. The art-science research group consists of students from BA, MA and PhD programmes,
ranging from artistic practice to humanities as well as researchers in those fields, including a high-profile
JRC-based researcher of bioeconomy,
specialized on European forest management. Importantly, in the development of the video interim product, an
artificial- intelligence based concreative entity (Dall–E 2, by Open AI) was included into the process:
prompts of key phrases from the research process were used to generate further visual material for the emerging
video. During the investigation of secondary data as well as primary data-gathering artistic work and embodied
presence in the forest as a physical space, the group consistently reflected on the importance of remaining
sensitive to the contrasting yet complementary positions that art and science are taking in their interaction.
The goal was to integrate both quite radically, surpassing the usual dominance of science, for which art is
useful (only) when it illustrates already established scientific theories, or simply represents beauty, reduced
to a tool for illustration of scientific findings.
The 4-month initial research stage brought about a video that was presented at the Resonances IV SciArt
summer school organized in June 2022 (in Ispra, Italy) by the Joint Research Center (JRC) of the European
Commission, in response to the theme NaturArchy: Towards a Natural Contract. ⁷ This video,
representing an interim result of the research process, also provides an interpretation of archival sources
about forest management in the Karst (a plateau in South-Western Slovenia, partly stretching into Italy) at the
turn of the 19th and 20th centuries. At the same time, it also brings new findings about forest management
during the afforestation of the Karst, and timely reflections on human-nature relationship. The complexity of
experiencing the forest is mediated both through maps and photographs as well as in original illustrations of
local birds and recordings of their singing, complemented by images created with artificial intelligence. They
are connected into a whole by a sound composition that intertwines with spoken theory on complex systems:
“And if ‘we’ are to survive the Anthropocene — this indeterminate epoch of ours in which
the world beyond the human is being increasingly made over by the all-too-human — we will have to actively
cultivate these ways of thinking with and like forests.” ⁸
The further stage of the artistic investigation resulted in a biosonification performance process
Symphotree, presented in September 2022 at the international festival of new media culture Speculum
Artium (Trbovlje, Slovenia). The project thus premiered its performative phase that involved signal inputs by
dripping water and a living plant (as concreative) entities, co-steering the audiovisual output, both factual
(coded, verbalised) and emotional (midi-interfaced, modular audio synthesizer based). The most recent
presentation of the entire research process took place in October 2022 at the RETHINKABLE Festival of
Transformative Economies and Regional Communities in Nova Gorica/Gorizia (Slovenia/Italy) where the video
was presented as a part of the photo and video exhibition on Composing Local Ecologies that was linked
to a scientific panel on Ecological Crisis and Criticism.
Performing to Individuate
In her project Plantportation (2019–20) the Ukrainian artist and curator Olena
Kasperovich first transported plants from Ukraine to Slovenia physically: each plant came with its own history
from a Ukrainian donor joined by scientific data about its origin and type. Along community art like events in
Slovenia locals had the opportunity to choose a plant and take it home with them, signing a pledge to care for
it. A further collaboration step with the local artist collective BridA focused on how space is perceived
differently among species in terms of real and virtual coexistence: as plants can only move under certain
metamorphic conditions, the project sought to transposition them into the virtual domain, in terms of a
televised, visual presence of plants and the possibilities of caring for them remotely. While in the resulting
speculative and collaborative “interspace” (presented in different installation setups at two
festivals, one in Slovenian and the other in the Ukraine) people could temporarily assign to themselves a
plant’s identity. ⁹
By addressing the sublime, the cutting-edge, the unthink-able, even ad-absurdum, all along to formal and
methodological disturbance, art seems to have finally become an essential and well-balanced ingredient in the
established Art- Science-Technology blend – perhaps too often believed to be the world saving formula par
excellence. Both science and art operate through questioning, critical thinking, observation, trial and
experimentation, evaluation and repetition, however art does employ more radical “data-generation”
methods such as intuition, imagination, inspiration, abstract sensing, embodiment, over-sensitivity,
serendipity, storytelling, over-identification, radicalism, over-consistency, non-work, disnovation, etc.
According to Peter Sloterdijk it is precisely thinking art, science and technology together that has the
potential to generate an expanded knowledge consciousness. ¹⁰ Thus, artists often seem mischievous or
even counter-culture minded, but in reality, due to their questioning nature, they tend to rethink what seems to
be obvious.
Be-coming Tree is “a grass-roots community, creating, sharing and documenting
close entanglement with trees and barefoot technology through collective global live-streamed events.” In
the context of the present paper, it represents a radically different, yet globally connected way of exploring
artificial autonomies of different artistic (and non-artistic) approaches of being (with, and in tight exchange)
with trees, which was particularly efficient if not downright viral during the pandemic year of 2020. By way of
a radical practice in the above-mentioned sense of cosmopolitics this live-art streaming series tunes artists
from all over the world onto an internet audio-visual platform, where each of them engages simultaneously
“with a local tree or woodland, and livestream their actions via a shared screen for one hour”.
Audiences are invited to experience a plethora of live art forms that range from “stillness, dance and
ritual” to more radical and conceptually intricate body-art and contemporary art performance pieces. The
event series aims at “a creative entanglement creating a sense of global kinship and shared well-being.
Tickets include a donation to plant rainforest trees, engaging audiences and artists in a restorative ecological
action creating awareness of the mutuality of all life.” ¹¹
Research through art makes us think about specific scientific questions not only in terms of “What is the
answer?” but more importantly “What could be the answer?”, which means that the aim is at
broadening of the cognitive field. ¹² What becomes central here is the ability of artists to
“see themselves as both actors and objects of research, their willingness to live experimentally, and
their desire to generate their own experiences rather than just analyze them.” ¹³ This is
exactly one of the virtues that a scientist can adopt from an artist with a fair probability for a positive im-
pact on the scientific research process and the resulting outcomes.
By taking the interdisciplinary approach that is not burdened by a goal or one strict path, but rather posing a
challenge to constantly question our perspectives and chosen methodologies, ultimately resulting in manifold
(research and artistic) outputs, not only the previously mentioned two cases, but also the TTT project may be
considered true “chimeras.” We borrow this concept from Natalie Loveless’ book How to Make
Art at the End of the World. A Manifesto for Research-Creation which postulates “research-creation
as an institutional remaking practice” and moreover and aims at “mobilizing research-creation for
(and in) the Anthropocene.” The TTT project highlighted exactly this kind of acquiring knowledge “by
bringing research and creation together in such a way that they unpredictably contaminate and remake each other,
in such a way that they render each other uncanny”. Quite in tune with the premises of all the here
presented artistic-scientific practices, Loveless further argues “for artistic production as de facto
research. [...] In doing so, in failing to fully belong, and allowing that nonbelonging to denaturalize,
emergently, its givens, research-creation tells other stories, uncanny stories, that (have the potential to)
carry within them the other ethics.” ¹⁴
Symbiotic Mutualism as an Artistic Act
A highly topical and scientifically well-founded case of artistic research into trans-species individuation,
Agnes Meyer-Brandis’ project One Tree ID – How to Become a Tree for Another Tree
gathers the volatile organic compounds (VOC) of a selected tree and synthesizes them into a
perfume that the visitors can apply to their body and with this “speculatively enter non-verbal
communication between” the human and the plant entity. Similarly to humans and, in fact, to all living
beings, every individual tree features a characteristic VOC print as a distinctive olfactory identity with which
plants can be differentiated from each another, both by scientific tools as well as by their own sensory
apparatus. In a close collaboration between the art production NGO Kersnikova Institute and its Kapelica Gallery
with the scientists from the Jožef Stefan Institute (the national scientific institute of Slovenia), the
team measured the etheric cloud of the volatile molecules in the tree’s roots, trunk and leaves. Based on
the spectral chromatography, molecular quantities were defined closely. Yet regardless of the extremely
demanding biotechnological methods, the volatile molecules could not be precisely measured in order to
synthesize the individual smell of the concrete “katsura” tree (also known as the caramel tree) that
moved between the science laboratory and the art gallery premises from spring to autumn 2022. A professional
perfumer was brought in to help define the scents more precisely, upon which the art-science team merged the
machine-gathered and sensory data into distinct perfumes “Root Cloud”, “Tree Trunk
Cloud” and “Tree Top Cloud”, which were eventually synthesized to the “One Tree
ID” perfume. ¹⁵
As the last here presented comprehensive example of an artistic practice that radically questions the
relationship between a technologically extended human and natural ecosystem, the terra0
collective has since 2018 been building various prototype environments that are based on the
decentralized (peer-to-peer) blockchain platform Ethereum, which aims to provide automated frameworks for the
resilience of a given ecosystem. By establishing a “Decentralized Autonomous Organization” on the
upper layers of the earth to govern them, terra0 research team (as a group of developers, theorists and
researchers “studying hybrid ecosystems in the technosphere”) aims to create technologically
augmented ecosystems that are both more resilient and more capable of operating within a predetermined set of
rules in the economic sphere, as independent agents: e.g. the forest independently mines cryptocurrency and
decides how it will change its material base. The group believes that modern technologies such as remote sensing
and machine learning provide an opportunity to rethink existing inefficient governance and regulatory
structures. Moreover, they also seem to suggest how, with appropriate art-thinking and speculative design-based
assumptions and interventions, these could play a key role in creating a sustainable, resilient and biodiverse
future. Along the line of above-mentioned cosmopolitics, the dominance of the human explanatory and
decision-making system that dedicates natural resources within an anthropocentric economy (of both material,
financial and symbolic orders), is juxtaposed by a radical scenario of self-governance of the analog (networked)
forest, supported by (in deed human-developed) digital (network) technologies. ¹⁶
Conclusions
The above discussion presents different kinds as well as different grades of symbiotic relationships reflected
in artistic works, and the research these are based on – such that is combined from different (blends of)
scientific and artistic tools and methodologies.
To focus one last time on this contribution’s pivotal project; at the JRC’s Resonances IV
Art-Science Summer School event the TTT video premiere owed its fond reception assumingly to an
audience a priori knowledgeable about the treated topics, and with a generally positive attitude toward
art-science collaborations. In discussions around the video audiences would often try to get behind the
methodology of the combined art-cum-scientific research as allegedly reflected in the directing and editing
approach to the video. By contrast, at the RETHINKABLE Festival of Transformative Economies and Regional
Communities the concept of close cooperation between art and science – let alone radical
post-anthropocentric critical thinking – was not known to the audience, thus their feedback revolved
around the “intricate” combinations of photographs of archival documents, artistic drawings,
“controversial text and perplexing sounds” that all took them “to a mystical forest”, as
one of the (art) exhibition visitors concluded. By contrast, the audience of the scientific panel Ecological
Crisis and Criticism discussed the video’s verbal messages (“theoretical poetry”) on
the background of the festival’s topics such as biodiversity, commoning policies, or transformative
economies.
Furthermore, the TTT video and its research supported an important further step of the process as featured in
the Symphotree performance at the Speculum Artium festival of new media culture, where the
artistic and the scientific approach were quite transparently set up on stage as well as verbally explained both
before and after the half-hour audio-video performance, to open up the research process underlying the artwork
presented. After the performance, discussions emerged mostly about the scientific data and methodologies used in
the process that lead up to the artistic act, continuing into considerations of both policies and popular
practices of conceiving, understanding and eventually using the forest. In the TTT project’s performative
extension, the plant and the water could be seen as considerably foregrounded in terms of knowledgeable
entities, delivering key informational impulses to the live artistic audiovisual score, working with and along
(if not in front of) the two human artists on stage.
Not only individual trees standing (growing, or indeed performing) vis-à-vis humans, but also forests (as
multitudes of trees) may be the most complete metaphor for both a process and a phenomenon of growth that is
systemically orchestrated, circular and long-term balanced – among different stakeholders and their types
of information (energy) processing that, in order to bear both ethical and aesthetical positive value, must be
kept in a dynamic and polyphonic dialogue between body and space, be it electronic or physical. ¹⁷
The TTT project poetically boiled down to a chorus of three imperatives, echoing throughout the audio narrative
within the video: Rewire Art:Science // Demystify Nature // Resettle Forest. Combining artistic and
scientific methods, cross-pollinating them both in the educational and in the cultural sector, on local as well
as on systemic (policy, economy etc.) levels, the TTT project process so far brought about artistic derivatives
such as an experimental video and a performance, as well as important scientific insights and even policy
influencing moments.
Along with the four above presented secondary cases of media art manifesting different forms of symbiotic
mutualism, it might be claimed that the forest eventually emerged as a new (kind of) metaphor for a
self-regulating system that the human (kind) should probably think twice before considering to have understood
it by any form of (human) knowledge, either by technology or policy, let alone aesthetics or ethics. The need
for a radically new sensibility and complex knowledge appreciation thus becomes evident, moving beyond objective
study and becoming attentive to different dimensions of research and its outputs that emerge through the
introduction of artistic thinking, as well as interspecies transactions on equal terms. Only such radical form
of inter-sectoral discourse can bring about viable models of conviviality among different species of life, forms
of knowledge and creativity, be it intellectual or purely material. Surpassing the commonly assumed scientific
objectivity where (especially non-human) entities are considered mere objects of research, such an approach
opens toward (perceiving) their subjective agency, and their true individuation, as well as their potential to
eventually change the eye of the beholder.
References
1 Donna J. Harraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham,
Duke University Press, 2016.
2 Isabelle Stengers, Cosmopolitics II, Minneapolis: University of Minnesota Press, 2011.
3 Virginia Gewin, “How to Shape a Productive Scientist-Artist Collaboration”,
Nature, Vol. 590, 2021, accessed December 1, 2022,
https://www.nature.com/articles/d41586-021-00397-1.
4 Jessica Jacobs, “Intersections in Design Thinking and Art-Thinking: Towards
Interdisciplinary Innovation”, Creativity. Theories – Research – Applications, 5:
1, 2018, 4–25.
5 Peter Purg, Silvia Cacciatore Jernej Č. Gerbec, “Establishing ecosystems for
disruptive innovation by cross- fertilizing entrepreneurship and the arts,” Creative Industries
Journal, 2021, accessed December 1, 2022,
https://www.tandfonline.com/doi/full/10.1080/17510694.2021.1969804.
6 JRC, European Commission's Knowledge Centrefor Bioeconomy, “Brief on the role of the
forest-based bioeconomy in mitigating climate change through carbon storage and material substitution”,
2021, accessed December 1, 2022, https://publications.jrc.ec.europa.eu/repository/bit-
stream/JRC124374/brief_on_role_of_forest-based_bioeconomy_in_mitigating_cc_online.pdf.
7 Abiral Khadka, Nikita Meden, Tijana Mijušković, Kristina Pranjić, Peter Purg,
“Taming the Forest”, University of Nova Gorica website, accessed December 1, 2022,
https://vimeo.com/725233829.
8 Eduardo Kohn, How Forests Think. Toward an Anthropology Beyond the Human,
Berkeley, University of California Press, 2013.
10 Peter Sloterdijk, The Aesthetic Imperative: Writings on Art, Cambridge, Polity, 2017.
11 BeComing Tree community, “Be-Coming Tree (2021)” online community website, accessed
December 1, 2022, https://becomingtree.live/.
12 Angelika Boeck and Peter Tepe, “What is Artistic Research?” w/k - Between
Science & Art Journal, Vol. 1, 2021, accessed December 1, 2022,
https://between-science-and-art.com/what-is-artistic-research/.
13 Billy Ehn, “Between Contemporary Art and Cultural Analysis: Alternative Methods for
Knowledge Production,” InFormation – Nordic Journal of Art and Research, 1: 1, 2013,
4–18.
14 Natalie Loveless, How to Make Art at the End of the World. A Manifesto for
Research-Creation, Durham, Duke University Press, 2019.
15 Agnes Meyer-Brandis, “One Tree ID” Kersnikova Institute website, accessed December
1, 2022, https://kersnikova.org/en/posts/events/all/agnes-meyer-brandis-one-tree-id-3.
16 terra0, “terra0 (2021)”, terra0 group website, accessed December 1, 2022,
https://terra0.org/.
17 Peter Purg, Körper im elektronischen Raum. Modelle für Menschen und interaktive
Systeme, Bonn: VDM, 2005.
Bibliography
1 Be-Coming Tree community, “Be-Coming Tree (2021)” online community website, accessed
December 1, 2022, https://becomingtree.live/.
2 Angelika Boeck, Peter Tepe, “What is Artistic Research?” w/k - Between
Science & Art Journal, Vol. 1, 2021, accessed December 1, 2022,
https://between-science-and-art.com/what-is-artistic-research/.
3 Billy Ehn, “Between Contemporary Art and Cultural Analysis: Alternative Methods for
Knowledge Production,” InFormation – Nordic Journal of Art and Research, 1: 1, 2013,
4–18.
4 Virginia Gewin, “How to Shape a Productive Scientist- Artist Collaboration”,
Nature, Vol. 590, 2021, accessed December 1, 2022,February,
https://www.nature.com/articles/d41586-021-00397-1.
5 Donna J. Harraway, Staying with the Trouble: Making Kin in the Chthulucene, Durham,
Duke University Press, 2016.
6 Jessica Jacobs, “Intersections in Design Thinking and Art Thinking: Towards
Interdisciplinary Innovation”, Creativity. Theories – Research – Applications, 5:
1, 2018, 4–25.
7 JRC, European Commission's Knowledge Centre for Bioeconomy, “Brief on the role of the
forest-based bioeconomy in mitigating climate change through carbon storage and material
substitution”, 2021, accessed December 1, 2022,
https://publications.jrc.ec.europa.eu/repository/bit-
stream/JRC124374/brief_on_role_of_forest-based_bioeconomy_in_mitigating_cc_online.pdf.
8 Abiral Khadka, Nikita Meden, Tijana Mijušković, Kristina Pranjić, Peter Purg,
“Taming the Forest”, University of Nova Gorica website, accessed December 1, 2022,
https://vimeo.com/725233829.
9 Eduardo Kohn, How Forests Think. Toward an Anthropology Beyond the Human, Berkeley, University
of California Press, 2013.
10 Natalie Loveless, How to Make Art at the End of the World. A Manifesto for Research-Creation,
Durham, Duke University Press, 2019.
11 Agnes Meyer-Brandis, “One Tree ID”, Kersnikova Institute website, accessed December
1, 2022, https://kersnikova.org/en/posts/events/all/agnes-meyer-brandis-one-tree-id-3.
12 Peter Sloterdijk, The Aesthetic Imperative: Writings on Art, Cambridge, Polity, 2017.
14 Peter Purg, Silvia Cacciatore, Jernej Č. Gerbec, “Establishing ecosystems for
disruptive innovation by cross-fertilizing entrepreneurship and the arts,” Creative Industries Journal,
2021, accessed December 1, 2022, https://www.tandfonline.com/doi/full/10.1080/17510694.2021.1969804.
15 Peter Purg, Körper im elektronischen Raum. Modelle für Menschen und interaktive
Systeme, Bonn, VDM, 2005.
16 Isabelle Stengers, Cosmopolitics II, Minneapolis, University of Minnesota Press, 2011.
17 terra0, “terra0 (2021)”, terra0 group website, accessed December 1, 2022,
https://terra0.org/.
Same Old Story: Agential Realism in the Study of Colonial Histories
HamedRashtian,
GabrielaAceves-Sepulveda
Simon Fraser University Surrey Canada hamed_rashtian@sfu.ca,
gabriela_aceves-sepulveda@sfu.ca
Abstract
What are the possibilities of accessing the reality of history? How can we read history, and what can we learn
from it? In this paper, we contemplate these questions by putting our ongoing research-creation project,
Same Old Story (2020-present), in conversation with feminist critiques of objectivity and current
discussions on the construction of historical narratives by historians, philosophers and artists, including
Antoinette Burton, Andreas Huyssen, Walter Benjamin Walid Raad and Forensic Architecture. Specifically, we
elaborate on how Karen Barad's "agential realism" ¹ informs our engagement with colonial histories in
Same Old Story and speculate on its broader relevance in research-projects that engage with historical
narratives. To do so, we describe the process of creating the current iteration of our project and offer a
theoretical framework based on a discussion of three main themes, Archive/ Memory, Architecture and
Monument/Counter-Monument. Building from this discussion, we elaborate on how to expand our work further,
focusing on the possibilities and limits of revitalizing embodied realities in historical events and learning
from them.
Keywords
Historical Research; Archival Research; History of Colonialism; Agential Realism; Situated Knowledges; Embodied
Observation; Architecture; Art Installation; Monument; Counter-Monument.
Understanding embodied realities in the study of different histories of colonialism is one of the main concerns
of this paper. In “Meeting The Universe Halfway: Realism and Social Constructivism Without
Contradiction,” Karen Barad advocates for a particular form of realism that she coins “agential
realism.” ¹ She argues agential realism does not reject objectivity but believes in an objectivity
that is embodied. In this definition, the object of observation is always observed from somewhere and this
observation is situated within language and culture. In other words, the observer understands the object from a
certain perspective and there is agency associated with them (the object and the viewer). In this paper, we
expand the concepts involved in our research-creation project, Same Old Story (2020-present), a
multimedia art installation that engages with archival documents and architectural structures through drawing,
text and sound, to investigate the possibilities of accessing the embodied reality of different colonial
histories, informed by Barad’s conception of reality. In Same Old Story, we use the frame of
research-creation as a broad methodology to describe our engagement with historical narratives, to benefit from
the diverse possibilities of combining image, text and sound in revitalizing historical events and to explore
the different capacities of the archival material. By expanding on the theories involved in Same Old Story,
this paper contributes to elaborating on the role of archives, buildings and monuments in the search for
the embodied realities of historical events.
Both buildings and archives are material cultural objects² that are observable by artists and can be
subject to research-creation and visual decoding. In this paper, we will expand on how Same Old Story
proposes a model that puts diverse histories in conversation with each other through archival research and
multimedia art strategies. The focus of this paper is to address the following: Can the model proposed in
Same Old Story in juxtaposing diverse historical events result in an expanded understanding of the
reality of those events?
We are specifically curious about the different histories of colonialism and the architectural structures
associated with them, as buildings and monuments have always been used by the colonizers in their process of
domination. Through a close reading of the architecture involved in colonial struggles, we aim to investigate
the potential violence against the colonized that is carried in their materials and shapes.³ These
buildings are marks on the body of history that can contribute to the understanding of embodied objectivity and
potentially guide historians to a form of agential realism.
The current iteration of Same Old Story (Figure 1) incorporates six different historical events
focusing on the architecture of the places related to them. Through multilayered drawings and sounds, the work
puts different histories of colonialism in conversation with each other to study whether this juxtaposition
results in a deeper understanding of the power relations embedded in colonial struggles. The use of
juxtaposition as a method is informed by Donna Haraway’s critique of objectivity as described in her essay
“Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective.”
Haraway argues that in comparison with dichotomous charts, “a map of tensions and resonances between fix
ends,”⁴ better represents different politics and epistemologies, and therefore a better account of
objectivity can emerge. In Same Old Story, we created a map of colonial histories focusing on
their power dynamics. By incorporating a cacophony of voices and setting the stage for a conversation, the
concept map proposed in Same Old Story attempts to avoid the certainty of monolithic historical
narratives that each architectural structure depicted in the work imposes.⁵ Although this juxtaposition
might not result in an objective account of realism, our goal is to revitalize some truth by depicting multiple
instances of colonial histories through their architecture. This first iteration prompted us to continue to
explore the use of juxtaposition as a historiographic, artistic and theoretical approach.
In what follows, we provide an overview of what we define as an embodied approach to historical research
informed by the work of historians and philosophers Walter Benjamin, Edward Said, Andreas Huyssen, Antoinette
Burton and Haraway, as well as members of the art research agency Forensic Architecture. They all argue for
understanding historical narratives as situated and always socially, culturally and politically contingent to
the present. This discussion sets the framework for our engagement with Barad's concept of "agential
realism" that we hope to develop further in future iterations of this project. To contextualize this discussion,
we elaborate on the processes of creating Same Old Story, including a detailed description of the work
and a discussion of three main themes that shape the project: Archive/Memory, Architecture and
Monument/Counter-Monument. Ultimately, this paper proposes to contemplate acts of observation and conveyance in
the study of history and the possibility of putting different historical narratives in conversation as a way to
gain an expanded understanding of the embodied realities of the historical events under observation through
research-creation.
Historical Research as Embodied Experience
In the study of history, the main objects of observation are past events, which makes its study specifically
complicated because the objects of observation are highly fugitive. As Walter Benjamin asserts, the vanishing
point of history is always the present moment; thus, construction plays a crucial role in historical narratives.
⁶ Thinking about the role that memory plays in historical narratives, Edward Said refers to
collective memories as frequently manipulated and intervened for the urgent purposes of the present. ⁷
Similarly, in her study on how archives are assembled, Antoinette Burton argues that it is a fantasy that
history can deliver the absolute truth. Instead, history is a highly interpretive act.⁹ Burton tells us
that the construction of historical narratives results from a pressure "present-day politics place on the
past."⁹ Echoing Burton, Said and Benjamin, for Andreas Huyssen, crafting historical narratives is a mode
of representation that belongs ever more to the present.¹⁰ In tune with Haraway, these authors share
an understanding of a certain perspective embedded in the construction of historical narratives and a view from
the present that is culturally conditioned or situated.¹¹, ¹²
Informed by these assertions on the construction of historical narratives we ask: how can we expand the concept
of agential realism to the study of history? What happens if we do not have access to the object of observation?
Barad argues the point of reference in agential realism is from the permanent marks “left on the bodies
which define the experimental conditions” ¹³ and reality perpetuates itself on these permanent
marks. What are the marks on the body of history? In Present pasts: urban palimpsests and the politics
of memory, Huyssen posits that our urban environment is saturated with monuments and buildings that
represent “the material traces of the historical past in the present.”¹⁴ For Eyal Weizman
and other members of Forensic Architecture, buildings with strong marks from the past are “political
sensors.”¹⁵ They are historical witnesses. “People can lie or forget, but buildings
don't” Weizman and Forensic Architecture have emphatically declared.¹⁶
According to Barad, another measure for accessing agential realism is through reproducibility.
“Reproducibility is possible because scientific investigations are embodied, grounded in
experience.”¹⁷ History is reproduced through the multiplicity of narratives. All these
narratives according to Haraway, incorporate visions that are situated.¹⁸ Historical narratives are
reproduced through the proliferation of diverse forms of archival records and archives. Archives are not to be
equated with an official institution or its holdings, archival records are found in domestic spaces and oral
forms of transmission are considered valid archives, as Burton and others have demonstrated.¹⁹ As a
result of this expanded understanding of what constitutes an archive, set in motion by what is referred to as
the archival turn, official and canonical archives have lost their exclusive role as the only legitimate sources
of history.²⁰
In this research we are curious about the narratives associated with people who, according to Burton,
“believe that their histories have not been written because they have not been considered legitimate
subjects of history.”²¹ These for Burton include colonized people whose lived experience is not
validated by official historical narratives or achieves.²² Dominant narratives legitimize the actions
of the colonizers and perpetuate their hegemony. Hence, colonial and racial hierarchies are reinforced and
reproduced through these narratives ²³ as they are associated with policies that categorize colonized
and marginalized groups as inferior.²⁴ In arguing about different perspectives in the study of
knowledge, Haraway advocates for a vision from below the position of power and calls it a "better
vision."²⁵ Following Haraway, Same Old Story supports this "better vision" through the use
of diverse forms of archives, specifically online archives, to depict six different colonial histories. This
vision from below characterizes a postcolonial tendency in our project and has led to the use of various
unofficial archives. In the following section, we describe the process of creating Same Old Story and
elaborate on the use of online image archives as references for the images depicted in the project.
An Account of Same Old Story
In Same Old Story, we are reflecting on the possibilities and limits of images (moving and still),
texts (in any format) and sounds found online, to develop juxtapositions between different colonial experiences
that can offer new insights into their histories. So far, we have included six different historical events in
our work. The image component of the current iteration consists of six different superimposed renderings based
on online archival photos of the sites of these events. The renderings are laser engraved together on one canvas
(Figure 2).
The six different narratives that we have chosen are:
The anti-colonial speech of Patrice Lumumba (the first prime minister of The Democratic Republic of Congo)
at the ceremony of the proclamation of the Congo’s independence (1961) in the Palais de la Nation
in Kinshasa.
The urban war that took place during the Battle of Algiers (1956-1957) as a part of the Algerian War of
Independence (1954-1962) between France and the Algerian National Liberation Front, and the role of the Casbah
neighborhood in these conflicts.
Secret negotiations between Israel and the Palestine Liberation Organization happening in Norway resulted in
Oslo Accords (1993).
British Embassy in Tehran, a symbol of colonial interventions in Iran and one instance of this in-
tervention: an oil concession between Britain and Iran that gave Britain exclusive rights to oil prospects for
60 years (1901).
The destruction of the Old Summer Palace, a complex of palaces and gardens and the main residence of the
Emperors of the Qing dynasty in Beijing by Anglo-French forces during China’s Opium Wars (1839-1842).
The construction of the Tiny Houses in the path of the Coastal GasLink Pipeline in northern B.C. in Canada
by members and allies of the Wet’swet’en Indigenous community: an expression of resistance against
forced invasion of their ancestral territories. (2020-present)
Besides the theoretical framework already described, during the initial stages of the project two other
important sources inspired the use of juxtaposition. Firstly, the Iranian traditional form of dramatic
storytelling called Pardeh Khani (Figure 3). Pardeh Khani is a performance in which the
performer stands in front of a large painted backdrop that includes several stories related to each other and
narrates these stories.²⁶ The painting component of this performance inspired the composition of the
laser-cut drawing in Same Old Story.
Secondly, Eve Tuck and C. Ree’s essay, “A Glossary of Haunting” informed our approach of
putting distinct histories together as a form of resisting dominant epistemologies. In their essay Tuck and Ree
advocate for violating the terms of settler-colonial knowledge production which, according to them, is based on
“the separation of the particular from the general, the hosted from the host, the personal from the
public.”²⁷ In short, to break the terms of settler colonial knowledge we need to get rid of
separation and categorization. In our work, we aim to emphasize the similarities of distinct selected events in
terms of power relations: how colonial power is symbolically, covertly or openly mobilized against the
colonized.
To add to the livelihood of the stories and to encourage an intimate connection to the work, we included sound
in the installation as well. The audio element was composed using sounds sourced from online archives, similar
to the image component. It consists of six audio channels playing at the same time, with each sound related to a
single event coming from a separate speaker. The aim is to combine drawing and sound to encourage simultaneity
among different narratives. The presence of diverse narratives in different modalities (sound and image) in one
work stimulates a fictional situation: it compresses time and space, creating an experience for the viewer in
which all events are happening at the same time.
Finally, to provide some information about the events and to bridge between image and sound, we included some
cap-tions in a booklet as a catalogue for the work. This booklet includes a brief description of each event to
provide some context for the viewer.
The current iteration of Same Old Story includes multilayered drawing and sound while written text, in
caption format, acts as a complementary element to the work. The drawings are made of digital line tracing of a
selection of photos. The photos are collected through an online search and depict places related to the events
(i.e., Palais de la Nation in Kinshasa, Holmenkollen Park Hotel near Oslo, Casbah neighborhood in Algiers,
British Embassy in Tehran, the ruins of the Summer Palace in Beijing and wooden Tiny Houses on the path of
pipeline construction in the northern BC). After studying the photos collected, we selected five distinct images
of each place showing spaces within and around the buildings from different angles. We traced the outlines of
each photo and created a digital drawing of each. We experimented with their sizes and their positions on a 94 x
63-inch background. We divided the background into six areas, each dedicated to one of the events. In each area,
we adjusted the positioning of drawings to create a desired superimposed drawing depicting a different
perspective of the same place. (Figure 4 and 5) Through these superimposed and blurry image compositions, we aim
to highlight the coexistence of a multiplicity of narratives about each event and the illegibility of history
more broadly.
To create a physical version of our digital drawing, we experimented with a few techniques such as screen
printing, digital printing on diverse kinds of photo papers, digital embroidery on canvas and muslin, etc. In
the process, we came across laser-engraving. After running some tests with a laser machine on different kinds of
papers, fabrics and canvases, we found laser engraving on raw canvas a befitting choice for the work both
formally and conceptually. In experimenting with the laser machine, we could learn how to enhance the depth of
drawing as well as how to set the machine to increase the possibility of burning the canvas, creating holes, and
controlling their shape. We particularly like this specific treatment, as we found burning, destruction, and
elimination to be in harmony with the history and trajectory of colonialism (Figure 6).
We used a similar approach in creating the sound composition for the installation. During the process of sound
editing, and after making some samples, we realized that the type of connection between sound and image needs to
be the same. By this we mean that if the connection to drawing happens instantly, the connection to sound needs
to be instant rather than delayed to achieve the same effect. This observation led us to select abstract
(spatial or ambient) sounds rather than sound of a conversation, for instance. The sounds that we selected for
the work are:
An old oil jack pump drilling the ground.
The sound of urban conflicts, including gunshots,explosions,
footsteps, sirens, etc.
The sound of applauding of the world’s leaders in the
ceremony of Oslo Accord One.
The sound of the burning of a large building
The sound of logging.
The sound of Patrice Lumumba’s anti-colonial speech.
Although the last sound (Patrice Lumumba’s anti-colonial speech) is more recognizable and specifically
linked to one of the events depicted, it is in French which was not the language of the first iteration of the
project. It was used as background sound and was not intended to be followed or understood. During the selection
process, we tried to find the actual sounds by searching the online archives related to each event, however, in
a few instances, where we did not have access to the original sound, we hinged upon fiction and used an archival
sound that we assumed is similar to the actual sound that belonged to the event. This process is informed by the
work of artist Walid Raad who delves into fiction in his practice of creating archives in the absence of
historical records²⁸ and Burton’s ideas about the backstage of the archives and the role of
construction in the process of their creation.²⁹ Later in this paper, I will discuss Raad’s
practice as well as Burton’s ideas in more detail.
Same Old Story incorporates archival research and uses archival material in its creation. It engages
with architecture by depicting the built spaces of the selected events, and the work itself has monumental
qualities through its physical presence. In what follows, I will focus on the three mentioned elements of
archive, architecture and monument in discussing the work and engaging with literature under the three themes of
Archive/Memory, Architecture and Monument/Counter Monument. This discussion informs the methods used in the
creation of Same Old Story for depicting different colonial histories to convey embodied realities
about them.
Theoretical Discussion
In this section we expand the conceptual framework explored thus far through a discussion of the concepts of
archive, memory, architecture, monument, and counter-monument. These concepts were selected to develop a more
specific communication with Barad’s ideas of accessing the agential reality of history through marks on
the bodies (architecture and monument) and the reproducibility of phenomena (archives) that gives an embodied
account of objectivity.³⁰
Archive/Memory
In the process of research for Same Old Story, our main sources of information are archival data. This
research began during the pandemic when we only had access to the online archives. Therefore, our collected data
is mainly from online sources. In researching the archives, we have been looking for evidence of historical
events in the search for accessing the reality of those events. Concerning the archival evidence and the notion
of truth, Burton argues that archives come into being as a result of political pressure. “Debates that
engage the challenges of ‘telling the truth about history’ have had very real political and material
consequences”³¹ as the use and the nature of the archive are not only about the past but about
the present. Informed by these, the question that comes to mind is how to trust archives. Which archive is
trustworthy? How the archives have been shaped? Who has made them? and what are the limits of the archive for
accessing the reality of historical events?
As a reflection on these questions, in her edited volume Archive stories facts, fiction, and the writing of
history Burton elaborates on the term “archive stories” ³² as the stories of the
formation of the archives that challenge their objectivity. She believes archives are the artifacts of history
themselves and insists on looking at “the backstage of archives”³³ to investigate how they
are constructed and manipulated. It is important to acknowledge that all archives are curated, thus there is
power involved in their formation. According to her, archive-making is an “embodied
experience,”³⁴ archives are historical actors, and they are made to speak to a specific
audience. This notion speaks to Barad’s idea that observation is embodied³⁵ and Haraway’s
consideration of the object of knowledge as an active agent and of knowledge production as
situated.³⁶
Burton discusses how the access to digital technologies has made all of us archivists and archive consumers and
as a result, more democratic versions of archives are emerging.³⁷ This condition has resulted in the
proliferation of online archives that has challenged the role of the official and canonical archives as the only
legitimate archival source.
In our project, besides using institutional online archives (official and canonical), we used a variety of
non-institutional online archival material and media resources including personal websites or sites not backed
by governmental agencies or corporations. Because we are interested in studying micro-narratives of historical
events, our attention is specifically drawn to the archives that deal with oral history. That is to archival
sources that “materialize those countless historical subjects who may never have come under the archival
gaze.”³⁸ As Bruton argues, postcolonial scholars renewed their attention to the archives while
acknowledging the value of oral history and its relation to memory which have been traditionally outside the
colonial notion of the archive.³⁹ For Burton, as for other feminist historians “Memory is
always cast (and still is) in gendered terms,” and “the capacity of women to write history has been
considered dubious until quite recently.”⁴⁰
Similarly, in discussing different visual systems, Haraway ponders whether the perspective of the subjugated is
more trusting, she states: “there is good reason to believe vision is better from below the brilliant
space platforms of the powerful.”⁴¹ Likewise, Michael Foucault, calls the official and national
archives “documents of exclusion” and “monuments to particular configurations of
power.”⁴²
Informed by these expanded notions of what an archive is and what counts as history, we collected most of our
data from non-institutional online archives and personal narratives. We understand our work as an
embodied process that incorporates specific historical events and does not claim to transfer absolute truths but
rather our fictional creation that hopefully triggers the viewer's curiosity to continue thinking about the
events depicted.
Architecture
Every material object can be read as a sensor, but buildings might be among the best sensors of societal
and political change. There are several reasons: buildings are immobile, anchored in space; they are in close
and constant interaction with humans; they are exposed both to the elements outside them and to an
artificially controlled climate within. And this besides embodying, of course, the political, social,
strategic, and financial rationalities that went into their conception.⁴³
Buildings are among the most stable witnesses of the past. Huyssen refers to them as "palimpsests of
space,"⁴⁴ where the marks of presence are merged with traces and erasure left from the past. In
Same Old Story, we follow Huyssen's ideas and focus on the architectural elements related to selected
historical events. We are interested in the empty spaces loaded with the heavy lift of history, spaces saturated
with marks and material testimony of those events, markings that can be used as leverage against the colonial
powers that have shaped them.⁴⁵ Architectural elements serve as evidence that aid in gaining a sense
of the reality of past events. This understanding of architectural elements is aligned with Barad's idea of
accessing reality through the marks on the bodies.⁴⁷ The marks of the past have an active presence
in our living environment.⁴⁷
Buildings play political role in history, Weizman from Forensic Architecture refers to architecture as a
“political plastic”⁴⁸ where different forces are shaped into forms. For Weizman,
“architecture is a material trace that can operate as an entry point into understanding larger processe...
reading macro from the micro.”⁴⁹ The sites that we study are saturated with elements through
which the dominant colonizers have tried to create, according to Said, “a sense of
identity”⁵⁰ for them and the colonized. As Weizman continues to explain, material aesthetics
are interconnected with all the things around them and bear marks of events.⁵¹ These sites are
haunted spaces filled with historical trauma. Their visible sides contain traces of the invisible.⁵²
While researching archival photos of the six events that constitute the work's central visual component, we
identified elements that operate as "actors and agents" ⁵³ in transferring a sense of domination of
the colonial forces. We paid close attention to the details of the archival images of buildings to study how
they can play a role in the power structures of colonial interactions. We emphasized certain elements in the
drawings that, we believe, contribute to perpetuating colonial power relations. For instance, numerous tabletop
and wall photo frames installed in the meeting rooms could indicate an interior design strategy supporting a
certain sense of identity.
In our project, we seek to represent different types of architecture connected to colonial interactions or
dominance. An example of the latter is the image of the Palais de la Nation in Kinshasa during the Belgian rule
over Congo. An example of the former is the architecture of urban areas where the colonized population lives,
often dense and ghetto-like neighbourhoods under heavy surveillance. The project depicts the Casbah
neighbourhood in Algiers during the French occupation. In other cases, anti-colonial movements use architecture
as a form of resistance against colonial forces. This type of architecture is exemplified by the inclusion of
the tiny houses that the Wet'swet'en peoples built on the site of the gas pipeline in northern B.C. to
demonstrate their ancestral presence on those territories. Other architectural elements included in our project
are ruins as traces of colonial power and scars on the land that can lead us to investigate different historical
perspectives. The ruins of China's Summer Palace, destroyed during the Opium Wars directed by British and French
forces against China, exemplify this architecture. In future iterations of our project, we will continue to
investigate other forms of architecture that serve as traces of colonialism in one way or another.
Monument/Counter-Monument
In the process of developing Same Old Story, we came across the concept of monument and the
monumentality of colonial buildings. We even discussed whether we wanted our work to have monumental qualities.
As we delved more profoundly into monumentality and its implications, we came across the concept of
counter-monument. In what follows, we reflect on this term and how it informs Same Old Story as a
counter-monument.
In “Memory/Monument” James E. Young refers to the Latin definition of the monument as "things that
remind"⁵⁴ to introduce his ideas on how monuments act as symbols of a specific historical era. He
notes how agencies of power use monuments as a self-aggrandizing tool to dramatizing their achievements and
create a sense of identity. The function of monuments is to generate a unifying, reductive sense of memory,
serving the power and mythicizing its role in shared memory. However, in this process, monuments may not remind
us of events so much as bury them beneath the layers of those myths they project.⁵⁵ Young argues if
monuments are mediums at the service of power, a monument against power is a monument against itself, against
its authoritarian role, and its “traditionally didactic function”⁵⁶ that
reduces the viewer's role to a passive spectator. He coined the term
“counter-monument”⁵⁶ to refer to monumental works of art that resist their traditional
function.
Thomas Stubblefield further elaborates on the concept of counter-monument. In his essay “Do Disappearing
Monuments Simply Disappear?”⁵⁷ Stubblefield argues for the potential of counter-monuments in
revitalizing the past through exchanges between the viewer and the work. He
asserts that counter-monuments turn a site into a discourse by democratizing power relationships and avoiding
official and singular narratives. In this manner, counter-monuments sidestep the monument's certainty of history
and rigidity by transferring productive recreation to its visitors. Such a monument creates the possibility of
confronting opposing ideologies and frees itself from imposing a top-down ideology. Counter-monument operates
within the postwar art discourse that seeks "a relational and contingent identity by relying on the viewer for
both the creation of meaning and the very construction of the work itself."⁵⁸
In Same Old Story, our aim is to avoid didacticism and the creation of reductive narrative of events.
To achieves this, we tried to move toward abstraction, to keep our distance from any form of literal
representation and avoid what Donna Haraway refers to as the “god trick,”⁵⁹ which is a
view from above that erases difference. We used superim-position to create blurry images incorporating multiple
perspectives of each historical site. As we further explore the concept of counter-monument we are reflecting on
how to foster a more democratic relationship with the viewer. Stubblefield discusses interaction and
ephemerality as two qualities of counter-monuments.⁶⁰ Following Stubblefield, in future iterations
of this project, we will reflect on the possibilities of incorporating the viewers' input more meaningfully in
the work and how to reduce the physicality of the installation to avoid imposing a fixed presence in the space.
To do so, we plan to explore the affordances of immersive virtual environments.
Conclusion
This paper situates our ongoing research-creation project, Same Old Story, in conversation with Karen
Barad and Donna Haraway’s feminist critiques of objectivity and current discussions on the construction of
historical narratives by historians, philosophers and artists, including Antoinette Burton, Andreas Huyssen,
Walter Benjamin, Walid Raad and Forensic Architecture. Specifically, we provided an overview of Barad's concept
of "agential realism" and Har- away's "situated knowledges" as central concepts that inform our use of
juxtaposition of sound and image to put different histories of colonialism in conversation with one another to
1) offer a critique of objectivity in historiography and 2) investigate the possibilities of accessing the
embodied reality of different colonial histories through research-creation.
We discussed how our project proposes an embodied approach to historical research through its theoretical
frame-work, the process of making the first iteration of Same Old Story, and its installation
experience, which offers a space where multiple historical narratives can be experienced simultaneously. Using
images and sound, Same Old Story offers various historical perspectives rather than a view from above
or a monolithic historical narrative. We further elaborated on our project's three additional themes:
Archive/Memory, Architecture and Monument/Counter-Monument. In future iterations, we will continue to theorize
these concepts in relation to the work of Barad, Haraway and others to explore their potential in putting
diverse colonial histories in conversations with each other to offer more nuanced perspectives of these distinct
histories. To further explore whether the proposed model in Same Old Story of juxtaposing diverse
historical events results in an expanded understanding of the reality of those events. Parallel to these
theoretical explorations, we will study the possibilities of virtual environments and other technologies to
facilitate a more active interaction from the viewer.
Acknowledgements
The first iteration of Same Old Story was developed as part of Rashtian’s MFA degree at Emily
Carr University. The theoretical discussion presented in this paper was developed as part of his doctoral
research under supervision of Aceves Sepúlveda in the School of Interactive Arts and Technology at Simon
Fraser University.
References
1 Karen Barad, “Meeting the Universe Halfway: Realism and Social Constructivism without
Contradiction”, Nelson L.H., Nelson J. (eds) Feminism, Science, and the Philosophy of
Science, vol 256, 1996, 167
2 Barad, "Meeting the Univers Halfway"
3 Eyal Weizman Bois, Yve-Alain Michel Feher, Hal Foster, “On Forensic Architecture: A
Conversation with Eyal Weizman”, October 156 (156), 2016, 116–40.
4 Donna Haraway, “Situated Knowledges: The Science Question in Feminism and the Privilege of
Partial Perspective”, Feminist Studies 14 (3), 1988. 588
5 Thomas Stubblefield, “Do Disappearing Monuments Simply Disappear? The Counter-Monument in
Revision”, Future Anterior 8 (2), 2012, 1–12.
6 Mark Godfrey, “The Artist as Historian", October 120 (120), 2007, 140–72.
7 Edward W. Said, “Invention, Memory, and Place.” Critical Inquiry 26
(2), 2000, 175–92.
8 Antoinette M. Burton, Archive Stories Facts, Fictions, and the Writing of History / Edited
by Antoinette Burton, Durham, N.C.,Duke University Press, 2005.
9 Burton, Archive Stories, 17.
10 Andreas Huyssen, Present Pasts: Urban Palimpsests and the Politics of Memory / Andreas
Huysse, Stanford University Press, 2003.
11 Weizman et al, “On Forensic Architecture.”
12 Haraway, “Situated Knowledges.”
13 Barad, “Meeting the Universe Halfway”, 180.
14 Huyssen, Present Pasts, 1.
15 Weizman et al, “On Forensic Architecture”, 121.
16 Weizman et al, “On Forensic Architecture”, 131.
17 Barad, “Meeting the Universe Halfway”, 185.
18 Haraway, “SituatedKnowledges.”
19 Burton, Archive Stories.
20 Burton, Archive Stories.
21 Burton, Archive Stories, 2.
22 Ricardo Castro-Salazar, Carl Bagley, “Navigating Historical Borders: Internal Colonialism
and the Politics of Memory”, Counterpoints, New York, N.Y., 415, 2012, 37–67.
24 Nnaemeka Obioma, “Racialization and the Colonial Architecture: Othering and the Order of
Things.” PMLA : Publications of the Modern Language Association of America 123 (5), 2008,
1748–51.
25 Haraway, “Situated Knowledges”, 583.
26 Amir Lashkari, Mojde Kalantari, “Pardeh Khani: A Dramatic Form of Storytelling in
Iran”, University of Hawai’i Press, 23 (1), 2015, 245-258
27 Eve Tuck, C Ree, “A Glossary of Haunting”, In Handbook of Autoethnography,
edited by Stacey Holman Jones, Tony E. Adams, Carolyn Ellis, Left Coast Press, 2016, 640.
28 Jong-chul Choi, “Reinventing the Archive in the Age of Digital Reproduction: Walid Raad's
the Atlas Group.” Digital Creativity (Exeter) 29 (2-3), 2018, 235–48.
29 Burton, Archive Stories.
30 Barad, “Meeting the Universe Halfway.”
31 Burton, Archive Stories, 1.
32 Burton, Archive Stories, 6.
33 Burton, Archive Stories, 7.
34 Burton, Archive Stories, 9.
35 Barad, “Meeting the Universe Halfway.”
36 Haraway, “Situated Knowledges.”
37 Burton, Archive Stories.
38 Burton, Archive Stories, 17.
39 Burton, Archive Stories.
40 Burton, Archive Stories, 20-25
41 Haraway, “Situated Knowledges”, 583.
42 Burton, Archive Stories, 6.
43 Eyal Weizman, Forensic Architecture: Violence at the Threshold of Detectability / Eyal
Weizman. Zone Books, 2017, 52.
44 Huyssen, Present Pasts, 7.
45 Weizman et al, “On Forensic Architecture.”
46 Barad, “Meeting the Universe Halfway.”
47 Huyssen, PresentPasts.
48 Weizman et al, “On Forensic Architecture”, 128.
49 Weizman et al, “On Forensic Architecture”, 134.
50 Said, “Invention, Memory, and Place”, 178.
51 Weizman et al, “On Forensic Architecture.”
52 Huyssen, Present Pasts.
53 Haraway, “Situated Knowledges”, 592.
54 James E. Young, “Memory/Monument”, Critical Terms for Art History / Edited
by Robert S. Nelson and Richard Shiff, 2nd ed. University of Chicago Press, 2003, 234.
Data-Phantoms: Impossible Nests (Memories Post Extinction)
ClarissaRibeiro, PhD
Roy Ascott Technoetic Arts Studio at SIVA/DeTAO Shanghai, China cr@clarissaribeiro.com
Abstract
Recent studies show that in ‘anthropogenic landscapes’ birds have been forgetting how to sing and
build nests since their parents die earlier and their communities are forced to be fragmented. Dialoguing with
the Sub-theme “Symbiotic Imaginaries: Inventing Worlds,” the work “Data-Phantoms: Impossible
Nests (Memories Post Extinction)” (2022) explores the phantasmagoric aspect of raw data coming from
‘nature traces’ of six (6) bird species declared extinct in nature along sequential morphogenetic
transformations from numbers’ lists (birdsongs used as primary data), to geometrically complex and
irregular data sculptures. The paper presents a discussion around the poetics that refers to an ongoing endeavor
in exploring and discussing metaphysical aspects of data visualization embedded in the tools and processes
chosen for parametric modeling and digital fabrication. The work intends contributing to reinforce our
"symbiotic imaginaries," ‘inventing new worlds’ in which humans together with all living beings
coexist and collaborate in their surviving efforts. The six data-sculptures—imperfect or ‘impossible
nests’—are tentative explorations of the sublime in dystopian data-visualization aesthetics,
manifesting in its irregular and messy geometry, the impossibility of birds, in broken ecologies, to perform
their birdsong and successfully mate, to learn from their community how to build an ‘optimum’ nests
and prosper.
“They approached with slow movements, in a blinding light, waving young hoko si palm leaves.
With their arms decorated with scarlet macaw tail feathers and a profusion of bright, colorful bunches of paixi
feathers, coated with vermilion annatto dye, they roared at the top of their voices, like a group of guests
arriving at a reahu feast. There were so many of them, and they kept their eyes set on me. It was beautiful but
terrifying because I had never seen xapiri spirits before.”¹
In "The Falling Sky: Words of a Yanomami Shaman," Davi Kopenawa explains that, all our animal ancestors who got
extinct, according to the Amazonian Yanomami’s "ecology as cosmology," “[...] have not disappeared
[...] they still have their animal names, but are now invisible beings. They have transformed themselves into
xapiri, who are immortal.”¹
As a poetic reference to the extinction of species as a process of information disintegration and dispersion in
natural environments, dialoguing with the understanding expressed by the Yanomami community from the
Amazon—in a moment we sadly see young birds forgetting their songs as adults are dying
faster²—this paper navigates the poetics of the work “Data-Phantoms: Impossible Nests (Memories
Post Extinction)” (2022). The work consists of a series of data-sculptures generated using as raw data the
birdsong of six species that got extinct in nature worldwide, inviting to meditate on how fragmented audio
memories of birds recorded and immortalized by humans in short videos shared online on platforms such as Vimeo
and YouTube, can haunt our imagination as phantoms of long-gone forests increasingly replaced by anthropogenic
landscapes.
Birdsong is one of the most studied aspects of animal behavior and it is crucial in mate attraction and
territorial protection. According to a study published in March 2021, due to the population decline,
“[...] male songbirds are having trouble picking up on the songs to attract mates—just as humans
lose their culture with future generations, these birds are losing theirs over time.”² The
researchers observed that the impact of severe population decline on song culture, in a nomadic, nectarivorous
songbird, had dramatic fitness consequences for the remaining individuals considering that the production of
atypical songs carried reproductive costs—“males whose songs differed from the regional cultural
norm were significantly less likely to be paired to a female.”²
Phantasmagoric landscapes
Anthropogenic landscapes are parts of Earth’s surface where humans’ populations have significantly
and endlessly altered natural patterns and processes in order to meet their demands for food, fuel, housing,
transportation, recreation, to mention a few. There is a general consensus among scientists who study threatened
bird species that, if human impact on the environment continues as it has, one third of all species and an even
larger proportion of bird populations will be gone by the end of this century. In Australia, as an example, in
the 250 years since Europeans colonization, native birdlife has been dramatically impacted—22 of the
native bird species have gone extinct.³ All over the planet, it is estimated that more than 180 bird
species have likely gone extinct over the last 500 years. We are currently experiencing what many scientists
perceive to be our planet's sixth mass extinction with far-reaching ecological, cultural, and even economic
implications.
By mapping and quantifying this loss, more and more scientists are joining efforts to help refining the
scientific understanding about the “impact of habitat removal and other pervasive threats that are driving
this observed extirpation.”⁴ The birds most at danger of going extinct today are typically those
that depend on certain, unique environmental circumstances, portending a future with declining biodiversity that
benefits invasive, generalist species in anthropogenic contexts.
Broken Memories
Back in the 1960s, Elsie and Nicholas Collias were running observations and experiments carried out on a colony
of captive Village Weaverbirds (Textor cucullatus)—a bird that makes complex and highly organized
nests—using a group of young birds hatched in a large outdoor aviary on the campus of the University of
California Los Angeles (UCLA). Although many people view nest building as a prominent example of instinctual
behavior in birds, this seminal research showed evidences that “[...] some social facilitation exists for
nest building, since the birds tend to show interest in nest materials at about the same time, just as in the
case of their other activities.”⁵
In this seminal experiment, Elsie and Nicholas Collias ⁵ made crucial observation on how birds develop
the ability to manipulate objects—a complex knowledge that is essential for building their nests. They
observed that the two weeks age Village Weaverbirds, more than only gaping to receive food from parents, use of
their bill to ‘preen disintegrating sheaths off the feathers’ involving ‘biting and nibbling
at the feather sheaths’. The researcher considers that “The mouthing of the feather sheaths would
seem to be the precursor of the ability to mandibulate strips of nest materials, and thus to adjust the position
of a strip in the bill”⁵ since it introduces motor elements needed for nest building. This ability,
according to the researchers’ observations, is very likely facilitated by parental example. In their
comments on the birds gathering of nest material, the researchers observe that when a weaverbird obtains a strip
from a leaf, after perching on the stalk or firm base of the leaf, it “[...] bites through one edge of the
leaf, tearing off part of the strip, and then tears the rest of the strip loose by flying away with it in the
general direction of the tip of the leaf.” ⁵ Elsie and Nicholas Collias comment that,
“In watching young weaverbirds, whether these birds were reared by hand or in the aviary, it seemed to us
that they had to learn many things in carrying out the process of tearing a strip of reed grass properly. While
experienced adult males generally fly off, finishing the tearing in one smooth action, the young often made such
mistakes as perching in an unstable place, starting the tear too close to the tip of the leaf, or at the very
base, or taking too broad or too narrow a bite, or tearing in the wrong direction, or tearing part way and
repeatedly starting partly detached strips, or tearing strips that were too short to be
woven.”⁵
Confirming the importance of socialization mechanisms in birds communities in helping refining learning
abilities relate to nests building and even the role that collaboration between young and adult birds plays in
teaching by "showing how to do," Brosset ⁶ presents the results of a comparative study of social
organization during breeding among the genus Malimbus of weaverbirds. Brosset observes that,
“In their morphology and behaviour, Malimbus spp. are close to the weaver birds of the genus
Ploceus. M. nitens seems the least evolved species while M. cassini and M. coronatus
are behaviourally the most evolved. In the last species, which has a very elaborate nest, the pair of
breeding birds is helped by one to four other birds. These helpers are birds in full adult plumage, and are
probably capable of breeding and may do so at another period in the long breeding season of at least six
months.”⁶
Additionally, the research refers to similar comparable and systematic observations on species other than the
weaverbirds, in regard to the handling of nest materials that had fund that hybrid parrots, as an example,
gradually improve their ability to cut and transport suitable nest materials in health environments and contexts
where they can learn from previous generations’ examples. It seems reasonable to conclude that, teaching
and collaboration are key in sustaining the planetary symbiosis of species and within species. Unfortunately, we
see that population decline is dramatically eroding cultural memory in wild animals populations and that the
birds are not alone in facing their dramatic destiny in this broken symbiosis with humans. The loss of culture
is associated with individual fitness costs⁶ possibly contributing to ongoing populations decline for an
endless number of species.
Impossible nests
Exploring the impossibility of birds learning from previous and same-generation individuals how to build a
proper and efficient nest in anthropogenic landscapes, the work “Data-Phantoms: Impossible Nests (Memories
Post Extinction)” (2022)—explores phantasmagoric aspects of raw data as the birdsong used as the
input for the generative design of the data-sculptures comes from birds declared extinct in nature. The work
invites to meditate on how the building abilities of organisms, far from being straightforward gene expressions,
are connected to complex inherited and learned behaviors that can be appreciated as part of the self-organizing
adventures of species and their symbiotic abilities and tendencies that drives cross-scale interactions and
integrations in the most diverse environmental configurations and variable conditions.
Data-Phantoms, Data-Incarnations
The work is part of ongoing experiments and efforts in discussing and exploring the metaphysical aspect of data
visualization in dialogue with Edgar Wind's notion of "incarnation."⁷ Wind broke with Neo-Kantian
tradition by constructing an approach to symbolic forms that provides a true account of symbolic
representation—the symbol has a real, visible meaning through embodiment (incarnation). The effort can be
described as a meditation on how a creative process that includes the use of birdsong as raw data in a
morphogenetic algorithmically driven process can conduct humans and birds through a collaboration-as-fantasy
mediated by computers, evoking our symbiotic imaginaries. The work intents to evoke metaphysical aspects of data
visualization embedded in the tools and processes chosen for parametric modeling and digital "materialization"
(digital fabrication, 2D printing).
Edgar Wind ⁸ elegantly dialogue with notions coming from discussions around the quantum measurement
problem to discuss how in lab experiments, the laws which are to be discovered are already presupposed, being
embodied in the instruments. Transplanting these considerations to discuss aspects of the work
“Data-Phantoms: Impossible Nests (Memories Post Extinction)” (2022) related to its poetics, we can
consider that in a collaboration between an artist (a human being), the "ghosts memories" or "ghost audio
memories" of the six birds gone extinct in nature, a laptop running Rhinoceros 7 and Grasshopper and related
customized algorithms, and the Printer used to print the posters (from a series of 3D renderings)—all
recognized by the human as non-humans—the experiment tests its own presuppositions.
The work poetics is woven in a way it can help us transcend our technoscientific fetishes for precision and
certainty and perhaps develop a model for knowledge production based on a spirited cosmology and intimacy over
cruelty and domination. To generate the data-sculptures the choice for the raw data was the audio extracted from
short videos shared on YouTube from the following birds' birdsongs: Alagoas curassow (Mitu mitu), Guam
kingfisher (Todiramphus cinnamominus), Guam rail (Hypotaenidia owstoni), Hawaiian crow or
ʻalalā (Corvus hawaiiensis), Socorro Dove (Zenaida graysoni), Spix's macaw
(Cyanopsitta spixii). Dialoguing with the Sub-theme “Symbiotic maginaries: inventing
worlds”, the work highlights dystopian aspect of the use of data coming from ‘nature traces’
of 6 (six) bird species declared extinct in nature’, bringing the opportunity to meditate on how
multispecies collaborations in media arts can help rising consciousness in our community on critical ecological
issues.
The understanding of a nest as an artifact – that it is part of the information embedded in the organism
that created it—was explored by Richard Dawkins in the early 1980s in "The Extended Phenotype". Dawkins
argued that ⁹ comparatively, since there are genes involved in the animal body’s morphogenesis (such
as the ‘homeobox gene’), there must be genes whose phenotypic expression is the bird’s nest
architecture—or the spider’s web, for example. Mike Hansell and Raith Overhill, in Bird Nests and
Construction Behavior, considers that ¹⁰ gene expression in sort of "extended phenotypes" may be more
complex than pondered by Dawkins, observing what an organism builds and how can be taken as an expression of the
genetic information embedded in the organism.
It is relevant to notice that nests building strategies as complex learned behavior in birds, as it has been
observed and documented by scientists in different contexts, includes lifelong learning to meet challenges in
hostile environments and can sometimes lead to dystopian innovations.
In a paper published on July 11, 2023, Auke-Florian Hiemstra, PhD candidate in Evolutionary Ecology at the
Naturalis Biodiversity Center, The Netherland, together with his team, discusses the use of "anthropogenic mass"
instead of "living biomass" as alternative nesting "materials" by urban birds when building their nest, having
birds being observed using"[...] bird deterring materials like anti-bird spikes as nesting material
[...].”¹¹
Although bird spikes are designed to keep birds away from nesting and perching, there have been documented
examples of sharp, industrialized products being used by birds as nesting material. According to the
authors,¹¹ published reports of wire nests date back to 1933, and the Kansas Barbed Wire Museum
proudly exhibits a corvid nest made of barbed wire. The team of researchers guess that, from the evidences,
members of some bird families are able to remove spikes from buildings and learn how to handle them and might
have discovered a new potential for this human-made product.
It is important to observe that, this recent publication ¹¹ focuses exclusively on corvid behavior,
describing all currently known nests of carrion crows and European magpies made out of anti-bird spikes,
discussing the possible implications of the usage of this bird repelling product as an alternative 'material'
for nest construction.
According to the researchers, ¹ the evidences may further support the notion that the use of these sharp
anthropogenic 'materials' can be used to enhance nest defense since, as observed, nest building by magpies is
sometimes disrupted by carrion crows attempting to prevent dome construction, "as a dome will hide the content
of the nest from attacks from above."
Emptiness Ecologies
The series of data-sculptures “Data-Phantoms: Impossible Nests (Memories Post Extinction)” (2022)
were commissioned by Mat Keel and Liz Lessner for the collective exhibition “Emptiness Ecologies," at "Yes
We Cannibal"—an institution for experimental art and social practice they co-founded and direct in Baton
Rouge, Louisiana., U.S..
Opening on Saturday, December 17, 2022 and displaying until the closing reception on February 4, 2023, YWC
directors considered this to be their most ambitious gallery show yet.¹², ¹³
According to the curators¹³, “Emptiness Ecologies" was crafted to serve as a creative audit of
Ecology as an effusively privileged nexus that ostensibly offers new ways to speak about social and
environmental relationships and events. Over the course of six weeks, the idea was to build collectively
“[...] a critical space within which to explore those questions and current moment, taking stock of the
conceptual uses of ecology and the nature of reality, itself.” ¹³
Exploring "emptiness" as a core topic, "Data-Phantoms” dialogues with the curators’ intention of
¹³ making sense of "cognitive dissonance in a moment of stunted political life and ecological
cataclysm, including living through the sixth great extinction."
This multimedia show at YWC featured new works of art and writing, including installations by multimedia artist
Dawn Dedeaux, collages by artist and Houma nation storyteller Monique Verdin, and works by U.S. based artists
and international figures, including a video piece by Duke and Battersby of Syracuse, New York, paintings by
Tokyo native Chihiro Ito of Brooklyn, New York, an experimental film by Anna Scime of Buffalo, New York and
prints of 3D renders from the series of data-sculptures “Data-Phantoms: Impossible Nests (Memories Post
Extinction).”
Nest-Works
The explorations that converge in the series of data-sculptures comes from the conversations orchestrated by
Amy-Claire Huestis for the ‘poetic wilding of all-too-human spaces’ she called "nest-works." As
Huestis explains, “Nest-works began with an experimental panel for the 2021 College Art
Association Conference, called "Co-Making this World." The experimental session was modeled after the
nest of a bird, a Black-Capped Chickadee. As this cavity-nester builds a home of disparate materials, the panel
of artist-researchers built a session of disparate theories and practices, as we considered relationships with
world-systems that are in the process of making (such as the nest of the chickadee).”
¹⁴
The material produced during the collaboration was curated and organized for Technoetic Arts academic
journal as nest-work of research material, considering new models for knowledge and creative production
as “[...] an entanglement of short essays made by artists working with a common pattern, framing
eco-poetics on collaborative and participatory processes with the non-human/more-than-human.”¹⁴
The article “Data incarnations: Nesting complex inherited and learned behaviours”¹⁵
documents the stimulated dialogues and outcomes starting from Huestis initial invitation.
Final Considerations
In this collaboration-as-fantasy that expands initial experiments in dialogue with a group of artists to
discuss aspects of interspecies and human-non-human collaboration, diving into dystopian aspects of ecological
events, the creative effort invited the ISEA 2023 community to navigate our symbiotic imaginaries from a
structural perspective. If in nature nests can be incarnations of birds’ complex inherited and learned
behavior in dialogue with other species and the environment, the experiments having as an outcome the collection
of 6 (six) data-sculptures—the ‘impossible nests—can embody a tiny sample of this complexity
when an artist invites birds that gone extinct in nature, thought the digital audio memories—"ghosts" of
their actual, live performed bird songs—into a process-oriented by the intention of algorithmically
generate data-sculptures from these birdsong transduced into sequential numerical variation.
Recalling Edgar Wind’s notion of incarnation, the intention highlights the "analog-digital" continuum as
inherent to nature’s complex morphogenetic strategies and not as detached from it.¹⁵ The
resultant data-sculptures explores sublime aspects of data-visualization aesthetics in a dystopian endeavor
manifested in the irregular and intricate geometry of the resultant objects referring to the impossibility of
birds in wrecked ecologies, deprived of proper symbiotic encounters, to socialize, collaborate, teach and learn
how to build successful nests and survive.
al Considerations
In this collaboration-as-fantasy that expands initial experiments in dialogue with a group of artists to
discuss aspects of interspecies and human-non-human collaboration, diving into dystopian aspects of ecological
events, the creative effort invited the ISEA 2023 community to navigate our symbiotic imaginaries from a
structural perspective. If in nature nests can be incarnations of birds’ complex inherited and learned
behavior in dialogue with other species and the environment, the experiments having as an outcome the
collection of 6 (six) data-sculptures—the impossible nests—can embody a tiny sample of this
complexity when an artist invites birds that gone extinct in nature, thought the digital audio
memories—"ghosts" of their actual, live performed bird songs—into a process-oriented by the
intention of algorithmically generate data-sculptures from these birdsong transduced into sequential numerical
variation.
Recalling Edgar Wind’s notion of incarnation, the intention highlights the "analog-digital" continuum
as inherent to nature’s complex morphogenetic strategies and not as detached from it.¹⁵ The
resultant data-sculptures explores sublime aspects of data-visualization aesthetics in a dystopian endeavor
manifested in the irregular and intricate geometry of the resultant objects referring to the impossibility of
birds in wrecked ecologies, deprived of proper symbiotic encounters, to socialize, collaborate, teach and
learn how to build successful nests and survive.
References
1 Davi Kopenawa, Bruce Albert, The falling sky: words of a Yanomami shaman, translated
by Nicholas Elliott and Alison Dundy, Cambridge, Massachusetts London, England, the Belknap Press of Harvard
University Press, 2013, 61
2 Crates, Ross, Naomi Langmore, Louis Ranjard, Dejan Stojanovic, Laura Rayner, Dean Ingwersen,
and Robert Heinsohn, “Loss of Vocal Culture and Fitness Costs in a Critically Endangered
Songbird”, Proceedings of the Royal Society B: Biological Sciences, The Royal
Society, 2021.
3 Rebecca Heisman, “Six Extinct Birds Whose Fame Lives On: The Dodo, Passenger Pigeon, and
More”, Bird Calls Blog of the American Bird Conservancy, September 06, 2022, accessed
November 10, 2022, https://abcbirds.org/blog/extinct-birds/
4 Michelle Ward et al, “Creating past habitat maps to quantify local extirpation of
Australian threatened birds”, Environ. Res. Lett. 17 024032, 2022, accessed November 10,
2022, https://iopscience.iop.org/article/10.1088/1748-9326/ac4f8b
5 Elsie C. Collias, Nicholas E. Collias, “The Development of Nest-Building Behavior in a
Weaverbird”, The Auk, Published by: University of California Press on behalf of the
American Ornithologists' Union, Vol. 81, No. 1 (Jan., 1964), 42-52, accessed November 10, 2022,
http://www.jstor.org/stable/4082609
6 A. Brosse, “Social organization and nest-building in the forest weaver birds of the
genus malzmbus (ploceinae)”, IBIS International Journal of Avian Science, Willey Online
Library, Volume 120, Issue1, January 197, p.27-37, accessed November 10, 2022,
https://onlinelibrary.wiley.com/doi/10.1111/ j.1474-919X.1978.tb04996.x
7 Edgar Wind, Experiment and Metaphysics: Towards a Resolution of the Cosmological
Antinomies, trans. E. Cyril, Abingdon, Routledge, 2001.
8 Edgar Wind, “Some points of contact between history and the natural
sciences”, in R. Klibansky and H. J. Paton (eds), Philosophy and History: Essays
Presented to Ernst Cassirer, Oxford: Clarendon Press, 1936, 255–64.
9 Richard Dawkins, The Extended Phenotype: The Long Reach of the Gene, Oxford, Oxford
University Press, 1999.
10 Mike Hansell, Raith Overhill, Bird Nests and Construction Behaviour, Cambridge:
Cambridge University Press, 2000.
11 Auke Florian Hiemstra, Cornelis W.Moeliker, Barbara Gravendeel, Menno Schilthuizen,
“Bird nests made from anti-bird spikes”,Deinsea, Issue 21, 11 July 2023, 17-25.
12 Zane Piontek, “Yes We Cannibal’s New Ecology Themed Exhibit its’ most
ambitious yet”, 225 Baton Rouge, December 20, 2022,
https://www.225batonrouge.com/things-to-do/yes-we-cannibals-new-ecology-themed-exhibit-its-most-ambitious-yet
13 The Advocate, “Yes We Cannibal opens multimedia exhibit featuring work by regional and
international artists”, The Advocate, December 15, 2022, https://rb.gy/yrgnh
14 Amy-Claire Huestis, “Nest-works”, Technoetic Arts: A Journal of Speculative
Research, 19:3, Nov 2021, p.227-241, https:// doi.org/10.1386/tear_00065_1
15 Clarissa Ribeiro, “Data Incarnations: Nesting Complex Inherited and Learned
Behaviours.” Technoetic Arts: A Journal of Speculative Research, 19:3, Nov
2021, 253-26, https:// doi.org/10.1386/tear_00067_1
Acknowledgements
I’d like to thank Amy-Clair Huestis for starting the conversation around ‘nest-works’ and
Mat Keel and Liz Lessner for the invitation to contribute to YWC Yes We Cannibal “Emptiness
Ecologies.”
Author Biography
Clarissa Ribeiro, Ph.D., is a multimedia artist and researcher with an interest in cross-scale information
and communication dynamics that impact and shape macro-scale emergent phenomena. In her more recent projects,
she explores the metaphysics of information-visualization in subversive morphogenetic strategies that welcome
the animistic to navigate ecologies as cosmologies. Chair of the first Leonardo/ISAST LASER talks to be hosted
in Brazil/Latin America since 2017, she is an active member of the UCLA Art|Sci Collective and was recently
awarded the Roy Ascott Studio’s Pete Townshend Endowed Senior Lectureship in Performative Technoetics.
She is an Associate Professor of Technoetic Arts at SIVA Shanghai Institute of Visual Arts, DeTao Masters
Academy, Roy Ascott Studio in Shanghai and represents the University of Fortaleza in China. She has widely
published in journals and conference proceedings and her work has been exhibited worldwide. She has been
serving as a reviewer for Leonardo Journal and the Technoetic Arts Journal, the Leonardo Abstracts Service
(LABS), contributing as a member of international conferences and symposium committees.
Black Box versus Black Bloc
RenéeRidgway
rr.mpp@cbs.dk / rridgway@cc.au.dk
Abstract
With around 5.5 billion requests per day, Google is the most used search engine worldwide. Google Search
identifies users online by collecting personal data—including an IP address, yet when using the Tor
browser, a users’ IP address remains obscured. Black Box versus Black Bloc employs Alexander
Galloway’s eponymous essay to structure the effects of Google Search (The Personalised Subject)
compared to that of the Tor Browser (The Anonymous User). Departing from the "data subject," I adopt
the internet protocol (IP) address as an organisational hinge to show the effects of search on
(us)ers—"subjectivities of search" and "agencies of anonymity," organised into ‘collaborative
collectives’ according to degrees of human-algorithmic interaction. The key difference is that I choose to
be in the "anonymous Tor collective," trusting my privacy to unknown human actors instead of putting trust in
Google that assigns me to particular groups through their non-transparent process of collaborative filtering,
without human agency.
Keywords
Search engines, Google, Tor, personalization, anonymity, black box, black bloc, subjectivity, agency.
"Many people I have talked to have mentioned that they are careful about what they type into search engines
because they know it’s being recorded and that limits the boundaries of their intellectual
exploration."¹
Since the rise of information technologies during the past 50 years, searching online has become one of the
most popular human activities. With around 5.5 billion requests per day, Google is the most used search engine
worldwide. Whereas ubiquitous computing described how electronic devices are interconnected, thereby making the
communication of data pervasive,² the sociotechnical organization of "ubiquitous googling"³ is
now a daily habit where "users" searches produce data that make users findable, even as they wander.⁴ With
their IP (Internet Protocol) address collected as well as keyword searches and search histories, the past 20
years users frequently employed Google for information, medical advice or even research, thereby creating vast
amounts of data. This "database of intentions" is now "a massive clickstream database of desires, needs, wants,
and preferences that can be discovered, subpoenaed, archived, tracked, and exploited for all sorts of
ends."⁵ One of the ends is personalisation as currency, where users pay with data and what they
receive are the currency of the web— customised URLs, based on their searching habits (browsers, location,
histories).⁶
Besides data collection, the incorporation of user interaction—keyword search queries, impressions and
clicking on links—is tied to an economic logic, advertisement, which facilitates ranking and
recommendation on the part of the platform that intervenes.⁷ This type of platform capitalism⁸ is
disrupting entrenched business models by highlighting as well as hiding—downplaying the labour of users
and free data as platforms promote the horizontality of their services—yet they are not flat. Google
Search "is a personal information economy where the standard exchange is service for profile"⁹ part of the
capitalistic "service/dataprofile/ advertising complex."¹⁰ Thus, over time, Google transferred itself
into an advertising company, producing not just search results, but capitalising on "informational rationality
of generating value from advertising and audience labour."¹¹
Through human-computer interaction with their ubiquitous googling, human bodies and their cognition, affect and
interests have become valuable resources as data-subjects. With the "extraction activities" of user data by
Google, a new asset class was created, or "surveillance assets"—providing a genuine market
exchange.¹² Yet this data shapes users reciprocally through human-computer interaction. This results
in fragmentary user subjectivity in deterritorialized spaces and it is the transformations of search
subjectivities, where "bodies are mostly addressed at the level of affect and cognition"¹³ that is
cause for concern. The thoughts and values of users that are inscribed in queries are transferred into
predictions, which subsequently produce not only products for corporations but incites changes in users’
search behaviour. Thus, the habit of search by users produces enormous amounts of data, generating profits for
Google but also facilitating the recursive feedback loop that organises (us)ers.¹⁴
However, there are ways to circumvent personalization and to reimagine search. Building upon the notion of
"cyberspace privacy" that applied encryption technologies during the 1990s,¹⁵ at the beginning of the
last decade the ‘privacy turn’ took effect. The Nymwars (2010-2014) debated pseudonymity (the
ability to have hidden identities when online) and the Snowden revelations (2013) exposed Five Eyes surveillance
on citizens and corporate collusion. This resulted in an increase in usage of privacy technologies such as Tor
(The Onion Router), an anonymity p2p browser that is a means to search online without divulging a user’s
IP address. Although controversial, there are many situations and "rationales for anonymity"¹⁶ which
are dependent on the context and situation, offline as well as when searching online.
This paper draws on my small data sets from an "experiment in living"—searching as a personalised subject
with Google and an anonymous user with Tor. Departing from the data subject, I use the IP address as an
organisation hinge to demonstrate the effects on (us)ers through a discourse analysis and a diagram. I combine
these effects with Alexander R. Galloway’s Black Box, Black Bloc,¹⁷ which provides a
conceptual (and technological) framework to synthesise the results from these two search methods. Derived from
the effects, I then explain how each ‘collaborative collective’, Subjectivities of Search
(Google) or Agencies of Anonymity (Tor) differs, based on degrees of human and non-human
interaction.
Experiment in Living: Google vs Tor
In order to discover how Google search organises (us)ers, I designed an "experiment in living"¹⁸ and
collected data on myself in my office. (Figure 1) This small study facilitated my understanding of the
behind-the-scenes constellations of agents (protocols, algorithms and myself) that determined my search results.
The research was conducted on two computers: one using Google search in a Firebox browser on a completely
‘personalized’ Mac (signed into a Gmail account, no ad blocking plug-ins, no incognito, etc.). With
personalisation, Google customizes its algorithms in regard to IP address, keywords queried, search history and
browsing habits to offer relevance and recommendations. The other computer is a Lenovo PC with a Debian
operating system running the Tor browser that ostensible offers anonymity by hiding the IP address.
I used a set of keywords selected from texts I was reading at the time (2016) and I call this dataset
Re:search: Terms of Art: Accelerationism, Aesthetic Turn, Anthropocene, Artistic Research,
Contemporaneity, Creative Industries, Cultural Entrepreneurship, New Aesthetic, Object Oriented Ontology,
Performativity, Post Digital, Post Humanism, Post Internet, Post Media, Transmedia. I gathered the data manually
(I did not programme it to scrape the data) and saved the entire web page of the 1st page of results, along with
the 10th, 20th, 30th, 40th and 50th etc. pages for the data set. This empirical ‘experiment in
living’ in my office enabled me to capture two forms of address when searching online, one as a
personalised subject with my IP address recognized by Google contrasted by being anonymous online with Tor,
where my IP address is hidden.
Black Box, Black Bloc
One of the ways the protocological ‘determines control after decentralisation’ is through the
configuration of the relationship between TCP/IP and DNS (Domain Name Server), as they are "political
technologies." ¹⁹ TCP/IP is now the standard internet protocol suite and DNS, which runs parallel to
HTPP in the application layer (7th), is responsible for translating the domain names into numerical IP addresses
in order to identify devices and locate them within the network protocols. Google Search facilitates not only
communication between parties to deliver search results (and advertisements), but the identification of
‘subjects’ and data collection, including the IP address. Conversely, with the Tor browser, the IP
address is part of the protocol that facilitates the transport of data but the IP address is not revealed
because of layers of encryption within the Tor p2p network.
By applying the IP address as an organisational hinge, I show the effects of search engines on (us)ers, drawing
on my results as a "personalised subject" and as an "anonymous user." With a ‘personalised subject’,
the black box and the Intellectual Property (IP) of Google’s proprietary search algorithm is a form of
"[in]visibility management."²⁰ situated within the "media arcane."²¹ This blackness of the
black box is also found in the Black Bloc that is analogous to tactics of obfuscation,²² such as
the Tor Browser, which, by obscuring the user’s IP address, facilitates an anonymous user. I apply
Alexander R. Galloway’s Black Box, Black Bloc as a conceptual (and technological) framework to synthesise
results from my two methods of search: "The black box: an opaque technological device for which only the inputs
and outputs are known. The black bloc: a tactic of anonymization and massification often associated with the
direct-action wingof the left. Somehow these two things come together near the end of the twentieth century. Is
there a reason for this?"²³
In the following I show the effects of these two methods of searching through a progression of
human/algorithmic interaction. The structure of both these effects begins with search algorithms interacting
with myself as a researcher collecting data online and as a data subject, who is protected by law. Since May 25,
2018 the GDPR (General Data Privacy Regulation) has been implemented in Europe, which regulates the collating,
processing, storage and transmission of personal data of EU citizens, or data subjects.²⁴ According
to Article 4, the "data subject" is an end user whose personal data can be collected through ‘direct
identification’ with an IP address. It is the recognition of citizens as ‘data subjects’ by
their IP address that facilitates certain effects and the organization of those searching online—through
degrees of personalisation (black box), or not, with degrees of anonymity (black bloc).
Black Box (Subjectivities of Search)(Figure 2)
During my experiment in living, I interacted with Google Search algorithms and used tools, such as my computer,
as memory extensions or hupomnemata, to ‘note down’ and collect data on
myself—keywords and search results. This Technology of the Self can be used as an instrument to
analyse the relationship between the subject and truth, where the ‘personalised subject’ explores
power constructs—how the subject constituted itself in one form or another, where "power is games of
strategy."²⁵ As my search histories are constantly collected by Google Search, the
‘personalised subject’ is not a substance but a form, which is "not primarily or always identical to
itself" as it is changes in different contexts and situations. Through diverse practices such as online
interaction with search algorithms as "truth games," this modern day "technology of the self" transforms me as a
(data) subject. Personalised subjects google themselves at some point, either to measure their attention
economy—where everything is based on visibility—or to see what has been written or published about
them by search results and that they are indexed.²⁶ It has become the meter to measure success.
Appearing higher in Google’s ranking, adding to one’s visibility, is a particular kind of attention
seeking that embodies Foucault’s figure of the "Homoeconomicus," or economical man.²⁷
Foucault was interested in the subject and more specifically, "the way a human being turns himself into a
subject" and part of his scholarship looks at the history of discipline and what he came to term "biopolitics."
"Foucault uses the term ‘biopolitics’ in order to elucidate how political power is carried out on
every aspect of human life, making individuals and the Homoeconomicus someone who is eminently
governable."²⁸ Although the Homoeconomicus is often considered a rational agent in pursuit of
self-interest, these subjectivity-defined ends are part of an economic civil society that operates through
production and exchange, which is part of the technology of liberal governmentality. Calculated practices (such
as Google Search), permit individuals to govern themselves, which epitomises the biopolitical and lies at the
core of neoliberalism. Google search facilitates online tracking and (self) surveillance, simultaneously
optimising searching subjects who are also "[n]eoliberal subjects—small sovereigns—are always
searching, rarely finding."²⁹ Through this interaction, the subject is both recognised, and subject
to, the law.
In the early days of neoliberal capitalist ideology, before it was defined as such, Louis Althusser articulated
forms of address through the framework of ideology, which he deemed "interpellation." With the constitutive
process of interpellation, this ideology is recognised by the individual’s acknowledgement of becoming a
subject that complicates their domination and subjugation. The classic example is that of Althusser’s
policeman who shouts at a passer-by “hey, you there!” in public where the individual then responds
by turning around. "[B]y this mere one-hundred-and-eighty-degree physical conversion, he [sic] becomes a
subject."³⁰ Previously it was the police who asked the question: "Hey you there?" Nowadays
‘personalised subjects’ enhance the power structures of Google by recognising themselves as subjects
when searching online, who are interpellated as "subjects" by automatically acknowledging the ideology of Google
Search by deciding to use it. By clicking on links mostly found on the first page of Google, Brin and
Page’s "Trusted User" interacted with the search engine and reinforced this ‘preferential
attachment."³¹ The Trusted User thereby actuates the ‘relevance’ and ‘quality’
of the search results by supplying feedback and data to Google, which it incorporates into subsequent search
results.
In this orchestration of individuals and algorithmic actors on the stage of the internet, the efficacy of
‘online display advertising’ comprises not only the agency of human actors but algorithmic ones. By
interacting with content on various websites, including Google Ads, visits are recorded by tracking cookies that
are instilled by another actor, the ‘ad server’ and the ‘individuated subject’ produces
a shadow. In the flow and ‘circulation of agency’ through its hyperlink journeys and its search
queries...
"The impressionable subject is produced as what is being visited and what is being
searched."³² Searching subjects provide data to Google, facilitating the tracking of "YOUs"
that "resonates strongly with Louis Althusser’s theorization of ideology," which "represents the imaginary
relationship of individuals to their real conditions of existence."³³ Whether there is "caring
of the self"³⁴ or caring of the network (the other YOUs), with the "interpellated subject’s"
interaction with platforms such as Google Search, it becomes an effect of algorithmic ideology.³⁵
Sorted together with others like them, versions of multiple data selves are fed back through never-ending and
recursive algorithmic loops. Comprised of a complexity of user subjectivities, the YOU addressed by Google
search is, crucially, both singular and plural yet "[i]n its plural mode, though, it still addresses individuals
as individuals,"³⁶ reflecting what has come to be called the Digital Subject.
The Digital Subject is an abstracted persona created from various data, records and archives, aggregated
together to form what comes ‘after the subject, requiring new ways to understand how it connects to the
subjectivities of living persons, something that is mapped onto living persons. With profiles in constant flux,
temporalities and degrees of correlation supposedly enable better personalisation and "[i]t is also through the
distance that digital subjects become more or less personalised or multiple—put together and
disaggregated." ³⁷ This ‘real-time’ collation of data on the "data subject" a.k.a. a
Trusted User of Google Search, creates algorithmically produced entities of ‘Data Dust’ as an object
of knowledge, which is comprised of dividuals ³⁸ data shadows³⁹ data doubles⁴⁰
and data derivatives. ⁴¹ These simulacra are not representative of real ‘individuals’.
Instead, they are encompassing elements, bits, points—"the fragments of registered behaviour, which are
extracted from the flow of data for specific purposes."⁴² These entities of Data Dust find others
like themselves not in the "meat space" apartments of urban cities, or in towns scattered across flatlands,
mountains andvalleys, but in worldwide data centres.
These numerous entities of the subject—Subjectivities of Search—are constructed by the
very data that ‘personalised subjects’ give away when habitually searching online. Nowadays the
subject ‘is potentially reduced to the pure $ (the divided subject)’ as a ‘Machinelike
Other’. ⁴³ Moreover, the construction of the ‘digital subject’ stems from the
continuous production of inscribed personal data, resulting in ‘new forms of subject construction that
arise out of computational procedures and are employed by various forms of power to distinguish, map, and
capture not only subjectivities, but also non-humans and physical things that inhabit the world’.
⁴⁴ Engaging with algorithms of Google Search, this hybrid form is Haraway’s cyborg—a
site of contestation, challenging any traditional demarcation line between the human and the machine and instead
defers to a space of patterning through the very technologies that comprise the database. Expressed otherwise,
with the reconfiguration of subjectivities through technics and distributed cognition, ‘human bodies as
cyborgs—as human machine systems—are in turn systematically combined into modes of
“Cyberorganization”’. ⁴⁵ As with the cyborg, Cyborganization is not an extension
of the human agent but perhaps rather ‘the agent is an extension of the machine’ and ‘remains
forever unfinished’, simultaneously controlling the flow of information back to searching subjects.
⁴⁶
Black Bloc (Agencies of Anonymity)
In the preceding section, I explained the effects, Subjectivities of Search, determined by the Black
Box (Google Search). I return now to the other half of Blackness, the Black Bloc and its effects, based on my
method of searching with the Tor browser (Figure 3).
The ‘Online Disinhibition Effect’ argues that online behaviour differs from ‘real’
world behaviour, emphasising the notion of invisibility because when people communicate with each other online
they mostly do so in written form, as they do not see each other and thus are more apt to express themselves
without inhibition. ⁴⁷ In regard to the Snowden revelations, when asked whether spying on citizens
is justified and whether citizens should be willing to exchange privacy and anonymity for increased security,
the predominant rationalisation is that state surveillance is positive as it protects people from terrorism,
often ending with ‘I’ve Got Nothing To Hide’. However, people often feel immune to
surveillance because they haven’t committed a crime, yet ‘[a]ccountability is commonly raised as one
of the reasons behind which people should provide identifiable information in online settings. When people
prefer not to share their names, they’re assumed to have something to hide’. ⁴⁸
In the early days of the web, ‘[t]he ability to participate anonymously or, as was and remains far more
common, pseudonymously was an integral part of why Barlow and other net utopians saw the Internet as
valuable’. ⁴⁹ In the 1990s users had ‘monikers’ when they signed into chatrooms,
later on they created email addresses for each different service they signed up for. Besides ‘traffic
analysis’ and ‘mixed networks’—the basis for Tor encryption— Chaum’s other
contribution to knowledge included ‘digital pseudonyms’, which is ‘a public key used to verify
signatures made by the anonymous holder of the corresponding private key’. ⁵⁰ Weisner,
described ‘digital pseudonyms’ as a means to insure privacy within networked societies.
⁵¹ Instances of Pseudonymity do not mean that one is completely anonymous, rather various pseudonyms
can be linked together to form either an ‘online identity’ or Online Persona, which ‘had come
alive in a new social practice: the virtual world as context for explorations of identity’. ⁵²
One reason to build these constructed online personas is that they ‘also offer sites of reinvention,
liberation, and play. Fake accounts and performed identities testify to that’. ⁵³
Persona building goes hand in hand with crawling the Dark Net in that it is replete with pseudonyms and false
identities. In attempts to reach ‘total anonymous freedom’ Dark Web Social Net (DWSN) members
‘customise their aliases, avatars, pseudonyms through widgets and in such a way that ‘[o]ne does not
use a fake account every time; one builds a persona’. ⁵⁴ When all of these pseudonyms
are collated by malevolent actors to construct personas, together they can form an online profile. However,
there are tactics to prevent this from happening. K-anonymity: A model for protecting privacy
would alter released information based on scaling, where the greater number of candidates, the ‘more
ambiguous the linking, and therefore the more anonymous the data’. ⁵⁵ Analogous to the
increased strength of anonymity with a larger amount of Tor users, the efficacy of ‘scaling’ tactics
with K-anonymity are nowadays limited due to fact that the storage of data continues to become easier
and cheaper as computational power increases and companies can examine this information in ‘real
time’.
In regard to these problems, Obfuscation is necessary in an era of online tracking and its use-value is in
‘mitigating and defeating present-day digital surveillance’. ⁵⁶ Applying tactics of
secrecy and deception to combat asymmetrical relationships of power, the guide shows various methods of
resistance that, while they might be considered ‘weapons of the weak’, afford some amount of
autonomy for the user. Moreover, these tactics prevent various shields of recognisability—how signals or
information could be interpreted, transmitted or shared by enemy parties. With personalisation, the IP address
plays a crucial role in the identifiability of the user along with the ever-increasing amount of data collected
and shared between search engines and third parties. As a reaction to the public realisation that search
companies (notably Google) were logging, storing and analysing the search query logs of individuals,
‘TrackMeNot’ (TMN) is ‘designed to achieve privacy in web search by obfuscating users’
queries within a stream of programmatically generated decoys’. ⁵⁷ Another ‘tactical
media’, AdNauseum is a browser extension that floods Google Search with false queries, clicks and likes
all ads, concomitantly visualizing the ads. ⁵⁸
Although obscuration offers many shades of privacy, a ‘true inner self’ can only emerge in
anonymity. ⁵⁹ The shift from the ability to carry out actions ‘namelessly’ in offline
space ‘is not as the end in itself of anonymity’, rather it is about how users can conduct
themselves anonymously online, whether a person is identifiable (or not) as themselves and if they are
untrackable. This ‘Unreachability’ is where absence can be used for positive expression in the
complex and dynamic computer-oriented society. Already in 1999 Nissenbaum presciently predicted the era of
‘surveillance capitalism’ where every atom of data is collected, kept and analysed:
‘Information technology has made it possible to track people in historically unprecedented ways. We are
targets of surveillance at just about every turn of our lives’. ⁶⁰
With Agencies of Anonymity, users can consciously protect themselves against the 24/7 tracking of
surveillance capitalism through tactics of obscuration yet this ‘unreachability’ reflects an
acquired knowledge—having the ‘tech savvy’ to use Tor. Spurred by agencies of
‘control’ over one’s data, this Techno-Elitism is a type of
self-determination⁶¹—knowing how to obtain anonymity and remain hidden from search engines and
governmental agencies. Furthermore, there are those ‘techno-elitists’ who do not wish to be indexed
by Google and desire to remain ‘unreachable’ to search engines, or delete their information after
searching with Tor. TAILS is an operating system that has Tor already configured, which, installed on a USB
stick, boots the computer and everything is deleted upon ejection.⁶² Or, if one desires to be
undetected and shop, the tactical media cum art project Random Darknet Shopper (2014- ongoing) by!
Mediagruppe Bitnik is a programmed autonomous bot that searches the Dark Net and carries out purchases,
completely anonymously, albeit through code.⁶³ However, this Algorithmic Anonymous User raises an
important question regarding the accountability of actions by non-human actors: can one prosecute and punish an
algorithm that commits a crime, in this case purchasing illegal substances and sending them to the St Gallen
Kunst Halle by post for an art installation?
Collaborative Collectives: Agencies of Anonymityvs. Subjectivities of Search
Returning now to my results from comparative searching using Google’s personalisation and Tor’s
anonymity, I learned that Tor delivers ‘Google-like’ results with its default search engine, only
without personalisation and targeted ads because there is no captured locative data, search history or IP
address. However, if I assume that both is the case—on the one hand, I am assigned as a Tor user and on
the other that Google assigns me to groups of people like me on its databases (an assumption I cannot fully
prove with my experiment but is the most likely scenario to explain its outcomes)—the original framing of
my experiment has to be specified. Instead of a personalised search as opposed to an anonymised search, I would
have, in fact, conducted a Google search that is a collective-of-users-like-me versus a
collective-of-all-Tor-users. At stake, therefore, are two collectives that take different forms.
In the collective-of-users-like-me it is Google’s algorithms which construct the categories I am part of
and assign me to this or that collective (e.g., collaborative filtering). I have no access, no knowledge and no
agency in regard to the collectives which I am made part of via Google. The forces (identification markers: IP
address, search histories, cookies, supercookies and locative data) that sort me into a collective and the
collectives that I am organised into—the categories that Google sets up and assigns me to, are not
transparent to me. Moreover, Google collects my individual search activities and, in future scenarios, will
probably state that they ‘personalise’ search even further based on data collated in the past and
present.
Tor’s collective, on the other hand, is both an anonymity network and a browser—Tor is
mostly Firefox code (95%) that incorporates patches to Firefox ESR (Extended Support Release). The Tor
collective is at least partially known to me as university labs worldwide run the major nodes but I do not know
who is running the relays (it is an anonymised network). I can, however, look at the ‘exit address’
list, which is constantly updated and shows the IP address, though I cannot identify the users.⁶⁴ By
joining the Tor collective I decided to trust the exit node operators, also in regard to my ‘expectation
of privacy’. The key difference is that whereas Google organises me into particular collectives through
their non-transparent process of collaborative filtering, I decided to be organised into the ‘anonymousTor
collective’. (Figure 4)
Both search collectives, the one determined by Google’s algorithms as well as the one created by the
choice to use Tor, add to specific filter bubbles. The filter bubbles are, however, structurally different: in
the case of the bubble produced by Google’s algorithms, Google collects the data of its users and
incorporates user feedback into subsequent search results. When I search for different things, I am merged into
different clusters with other people ‘like me’ or ‘YOUs’. I would then add to
the feedback loop by continuously adding to my own personalisation by clicking on the links that are delivered
to me as results. I do not have access to the Google collective itself—I am constantly switched into a
different cluster by an algorithmically organised process that I have no control over and cannot adjust. There
are also constant updates and tweaking of signals being carried out on the algorithm. Therefore, I propose that
various degrees of Google Search’s personalisation organise the corporal ‘data subject’ into
effects: Subjectivities of Search. With the Black Box, as Google’s personalisation
increases so does the amount of computational agency, at the same time the degree of anonymity (privacy)
decreases.
The filter bubble of the Tor users, on the other hand, is one where I stay in the same group that shares the
same filte—no matter how much I change my search behaviours (what I click on or not). As a Tor user, the
variable is what Tor uses as their default search engine (Startpage, Disconnect Search or presently DuckDuckGo)
and if this default (still) delivers Google Search results without locative data. Therefore, the results of my
small data experiment postulate that the user is assigned the category of ‘Tor user’, which can be
seen from outside the Tor network. When I use Tor I am part of a p2p anonymity network, which increases in
strength the more users use it. Unlike Google Search, privacy-enhancing technologies and diverse settings
enable the user to organise themselves into effects: Agencies of Anonymity. With the Black
Bloc, as the amount of anonymity (privacy) increases with Tor so does human agency, at the same time the
degree of personalisation decreases. If I compare these two search processes, with the individualisation
of the pseudo-autonomous objects of Google’s personalisation, my ‘data dust’ is atomized
and fractured—as a ‘digital subject’ I have no agency to decide where I am assigned. However,
to partake anonymously in a p2p-collective individuates me more than personalisation does. Bernard
Stiegler’s entire pharmacology of care revolves around this new ecology, which can be understood to be
threefold, as a ‘re-articulation of psychic, collective and technical individuation’.
⁶⁵ At stake is an individuation in the sense of Stiegler’s reading of
Simondon—an individuation that is marked by being collective and psychic alike, where the genesis
of how an object comes to exist is through operations of individuation, or ‘ontogenesis’
and a living being exists in a state of becoming between individuations, never in isolation but in collective,
social as well as psychological constellations. The Tor browser (p2p network) embodies this individuation of
collective singularities, which are dependent on the other. ‘After all, an individual exists and is only
capable of individuating as a result of the relations it establishes with others and that others establish with
it.’ ⁶⁶ Phrased otherwise, the possibility of choosing the individuating Tor
collective over the individualisation of Google Search might be a modest resolution to the problem of
‘how one would define a singularity that could be a collective singularity’. ⁶⁷
The experience of setting up an ‘experiment in living’ has opened up a view on how Google
Search works and my exploration of using Tor reimagined what search could look like. Without having to become a
‘personalised subject’, Tor offered me ‘relevant’ search results as an ‘anonymous
user’. With the Tor Browser I am not ‘commanded’ by ‘prescriptive’ technologies
⁶⁸ as I am with Google Search, instead I chose which collective I wished to be part of, in this case
Tor. The effects of the Black Bloc (Agencies of Anonymity) enabled me to intervene and this type of
searching facilitated degrees of anonymity, embodied by various levels of user agency. Aside from its other
merits, Tor is also one, albeit not the only, strategy to challenge the protocols facilitating Google’s
‘surveillance capitalism’.
References
1 Laura Poitras Citizen Four, 2014, https://citizenfourfilm.com/
2 Mark Weiser, “The Computer for the 21st Century”. Scientific American
Special Issue on Communications, Computers, and Networks, September, New York, Scientific
American, 1991.
3 Renée Ridgway, Re:search. Dissertation, 2021.
4 Wendy Hui Kyong Chun, Habitual New Media: Updating to Remain the Same, Cambridge,
MIT Press, 2016, 77.
5 John Battelle, The Search: How Google and Its Rivals Rewrote the Rules of Business and
Transformed Our Culture, New York Penguin, 2006, 6.
6 Renée Ridgway, “Personalisation as currency”, APRJA (A
Peer-Reviewed Journal About), 18 APRJA Volume 4, Issue 1, 2015, http://www.aprja.net/?p=2531.
7 Tarleton Gillespie, “Platforms Intervene”, Social Media + Society,
Thousand Oaks: Sage Publishers, 2015.
8 Nick Srnicek, Platform Capitalism, Cambridge, Polity Press, 2016.
9 Greg Elmer, Profiling Machines: Mapping the Personal Information
Economy, Cambridge, MIT Press, 2003.
10 Geert Lovink, Nathaniel Tkacz, “MoneyLab: Sprouting New Digital-Economic Forms”, In
Moneylab: An Intervention in Digital Economy, Amsterdam, Institute of Networked Culture, 2015, 15
11 Dallas Smythe, “On the audience commodity and its work”, in M. G. Durham, D. M.
Kellner (eds) Media and cultural studies, Malden, MA, Blackwell 1981, 230–56
12 Shoshana Zuboff, “Big Other: surveillance capitalism and the prospects of an
information civilization”, Journal of Information Technology, 30, Thousand Oaks: Sage
Publishers, 2015, 81
13 Armin Beverungen, Timon Beyes, Lisa Conrad, Digital Media, Thousand Oaks, Sage
Publishers, 2019.
14 Renée Ridgway, Re:search. Dissertation, 2021.
15 David L. Chaum, “Untraceable Electronic Mail, Return addresses, and Digital
Pseudonyms”, Communications of the ACM, Volume 24, Number 2, February 1981.
16 Gary Marx, “What’s in a Name? Some Reflections on the Sociology of
Anonymity”,The Information Society, 15:2, Milton Park, Taylor & Francis, 1999, 99- 112.
17 Alexander R. Galloway, “Black Box, Black Bloc.” In: Benjamin Noys
(ed.): Communization and its Discontents. Contestation, Critique, and Contemporary Struggles,
Brooklyn, Minor Compositions/Autonomedia, 2011, 237–249.
18 Noortje Marres, “The Experiment in Living”, In InventiveMethods: The
Happening of the Social. Celia Lury, Nina Wakeford. Eds, Abingdon, Routledge, 2012.
19 Alexander R. Galloway, Protocol: How Control Exists afterDecentralization, Cambridge,
MIT Press, 2004.
20 Mikkel Flyverbom, Paul Leonardi, Michael Stohl, Cynthia Stohl, “The Management of
Visibilities in the Digital Age: Introduction”, International Journal of Communication,
Los Angeles, USC Annenberg Press, Vol. 10, 2016, 98–109.
21 Timon Beyes, Claus Pias, “The Media Arcane.” Grey Room 75, Cambridge, MIT
Press Journals, 2019, 84-107.
22 Finn Brunton, Helen Nissenbaum, Obfuscation,A User’s Guide for Privacy
and Protest, Cambridge, MIT Press, 2015.
23 Alexander R. Galloway “Black Box, Black Bloc.”, 3
24 GDPR, General Data Protection Rights, https://gdpr-info.eu/art-4-gdpr/
25 Michel Foucault, Ethics: Subjectivity and Truth, The Essential Works of Michel Foucault
1954-1984.
26 Tarleton Gillespie, “2. Algorithm” In Digital Keywords. A Vocabulary of
Information Society and Culture. Edited by: Benjamin Peters, Volume 8 in the series Princeton Studies
in Culture and Technology, Princeton, Princeton University Press.
27 Michel Foucault, Technologies of the Self, Edited by Luther H. Martin, Huck Gutman,
Amherst, University of Massachusetts Press, 1988, 16.
28 Philip Mirowski, Never Let a Serious Crisis Go to Waste, New York, Verso, 2013.
29 Wendy Hui Kyong Chun, Habitual New Media: Updating to Remain the Same.
30 Louise Althusser, “Ideology and Ideological State Apparatuses (Notes towards an
Investigation),” In Lenin and Philosophy and Other Essays, translated by B. Brewster, London,
Monthly Review Press, 1971, 174.
31 Sergei Brin, Larry Page. “The Anatomy of a Large-Scale Hypertextual Web Search
Engine”, Presented at Seventh International World-Wide Web Conference (WWW 1998), Brisbane,
Australia, April p.14-18.
32 Lucas Introna, “The algorithmic choreography of the impressionable subject”, In
Seyfert, Robert and Roberge, Jonathan (Eds.), Algorithmic Cultures: Essays on Meaning, Performance and New
Technologies, Abingdon-on-Thames, Routledge, 2016, 37
33 Wendy Hui Kyong Chun, Habitual New Media: Updating to Remain the Same, 121
34 Michel Foucault, Ethics: Subjectivity and Truth, The Essential Works of Michel Foucault
1954-1984.
35 Astrid Mager, “Algorithmic Ideology.” Information, Communication &
Society, 15:5, Milton Park, Routledge, 2012p. 769-787.
36 Wendy Hui Kyong Chun, Habitual New Media: Updating to Remain the Same, 36, 64
37 Olga Goriunova, “The Digital Subject: People as Data as Persons” Theory,
Culture and Society. Vol. 36, No. 6, 01.11.2019, 125-145. Thousand Oaks, Sage Publishers,
2019, 126
38 Gilles Deleuze, “Postscript on the Societies of Control", October. Vol. 59.,
Winter, 1992, 3-7, Cambridge, MIT Press, 1992, 5.
39 Nanna Thylstrup, “Archival Shadows in the Digital Age”, NordiskTidsskrift for
Informations videnskab og Kulturformidling, årg. 3, nr. 2/3, 2014.
40 Antoinette Rouvroy, “The end(s) of critique: Data behaviourism versus due process”,
In Hildebrandt, Mireille and De Vries, Katja (eds.) Privacy, due process and the computational turn: the
philosophy of law meets the philosophy of technology, New York, Routledge Taylor, Francis Group,
2013.
41 Louise Amoore, “Data Derivatives: On the Emergence of a Security Risk Calculus for Our
Times”, Theory, Culture & Society 28(6), 24–43, Thousand Oaks, Sage Publishers, 2011,
126
42 Felix Stalder, “Between Democracy and Spectacle: The Front-End and the Back-End of the
Social Web”, In M. Mandiberg (ed.), The Social Media Reader, New York, New York University Press,
2012, 242-256.
43 Slavoj Zizek, The Ticklish Subject: The Absent Centre of Political Ontology,
Brooklyn: Verso Books, 1999.
44 Friedrich A. Kittler, Gramaphone, Film, Typewriter. Stanford. Stanford University
Press, 1999, 127.
45 Martin Parker, “Manufacturing Bodies: Flesh, Organization, Cyborgs”. In Hassard,
J., Holliday, R., Willmott, H. (Eds) Body and Organization, London, SAGE Publications
2000,73, 71-86.
46 Martin Parker, Robert Cooper, “Cyborganization: Cinema as Nervous System”,
In For Robert Cooper: Collected Work, Edited by Gibson Burrell and Martin Parker, p.236-252, New
York: Routledge, 2016, 243.
47John Suler, “The Online Disinhibition Effect”. Contemporary Media Forum
International Journal of Applied Psychoanalytic Studies, Vol. 2, No. 2., Jacksonville, N.C., Whurr
Publishers Ltd., 2005.
48 Danah Boyd, “The Politics of 'Real Names: Power, Context, and Control in Networked
Publics”, Communications of the ACM 55(8): 29-31, 2012, 30.
49 Lisa Nakamura, “Afterword: Blaming, Shaming and the Feminization of Social”,
Shoshana Magnet and Rachel Dubrofsky (eds). Media Feminist Surveillance Studies, Chapel Hill, Duke
University Press, 2014.
50 David L. Chaum, “Untraceable Electronic Mail, Return addresses, and Digital
Pseudonyms”, 1
51 Mark Weiser, “The Computer for the 21st Century”, Scientific American
Special Issue on Communications, Computers, and Networks, September, New York, Scientific
American, 1991.
52 Sherry Turkle, The Second Self, Cambridge Mass, MIT Press, 2005, 288
53 Olga Goriunova, “The Digital Subject: People as Data as
Persons” Theory, Culture and Society. Vol. 36, No. 6, 01.11.2019, Thousand Oaks: Sage
Publishers, 2019, 125-145, 127.
54 Robert W. Gehl, “Power/freedom on the dark web: A digital ethnography of the Dark Web
Social Network”, Republished 2016, New Media & Society, Volume 18(7), 1219-1235, Thousand
Oaks: Sage Publishers, 2014, 1226.
55 Lantaya Sweeney, “K-anonymity: a model for protecting privacy”. International
Journal on Uncertainty, Fuzziness and Knowledge-based Systems, 10. 2002, 557-570, Singapore, World
Scientific Publishing Co Pte Ltd, 2002.
56 Finn Brunton, and Helen Nissenbaum, Obfuscation. A User’s Guide for
Privacy and Protest, Cambridge, MIT Press, 2015, 55
57 Helen Nissenbaum, Daniel C. Howe, “TrackMeNot: Resisting Surveillance in Web
Search”. In Kerr,I., Lucock, C. and Steeves,C. (Eds), ‘Lessons from the Identity Trail:
Anonymity, Privacy, and Identity in a Networked Society, Oxford, Oxford University Press, 2009, 417.
58 Helen Nissenbaum, Daniel C. Howe, Mushon Zer-Aviv., Ad Nauseum, 2014,Available at: https://adnauseam.io
59 Daniel C. Howe, Helen Nissenbaum, “Engineering Privacy and Protest: A Case Study of
AdNauseum”. Proceedings of the 3rd International Workshop on Privacy Engineering, Volume 1873, 57-64,
2017, 57.
60 Helen Nissenbaum, “The Meaning of Anonymity in an Information Age”, In Spinello,
Richard A. and Tavani, Herman T.(eds.), The Information Society. 15:141-144n Milton Park: Taylor
& Francis, 1999, 141
61 Robert W. Gehl, “Power/freedom on the dark web: A digitalethnography of the Dark Web
Social Network”.
62 TAILS, The Amnesiac Incognito Live System, https://tails.boum.org/
63 Random Darknet Shopper 2016, MediabitnikGruppe:
https://wwwwwwwwwwwwwwwwwwwww.bitnik.org/r/
64 Tor Project: https://check.torproject.org/exit-addresses
65 Eric Hörl, General Ecology: The New Ecological Paradigm, London, New York,
Bloomsbury Academic, 2017, 35.
66 Bernhard Stiegler, To Love, To Love Me, To Love Us: From September 11 to April 21,
Stanford: Stanford University Press, 2009, 77.
67 Maurizio Lazzaroto, “Conversation with Maurizio Lazza-roto”. Public Editing Session
#3, June 23, 2010, from ‘Exhausting Immaterial Labour in Performance’; joint issue of Le
Journal des Laboratoires and TkH Journal for Performing Arts Theory (no. 17), October 2010, 14.
68 Ursula Franklin, “The Real World of Technology”. Massey Lectures, 1989.
Processes, Fabrication and Design with Kombucha Bacterial Cellulose: Mapping Practices
1Institute for Future Technologies, 2Université de Nîmes
(Projekt), 3École des Arts Décoratifs–PSL (EnsadLab), 4Université de
Nîmes (Chrome). Nîmes, Paris, FR vivien.roussel@devinci.fr ; lucile.haute@ensad.fr ;
Marc.Teyssier@devinci.fr ; pascal.dhulster@unimes.fr
Abstract
Kombucha is a fermented drink that has been drunk in Asia for centuries and has recently been popularized in
Western cultures. The production of the beverage generates a co-product, a biofilm of Bacterial Cellulose (BC).
This biofilm is today at the center of a typology of bio-design practices, ranging from initiation workshops to
fermentation to practical and plastic research around shaping objects with this living matter. This paper
proposes an overview and initial classification of the current design practices that use Kombucha Bacterial
Cellulose as the raw material for artifacts. To this extent, we structured a corpus of selected projects in a
Principal Component Analysis (PCA) map. We classified 38 projects in the field of design made between 1985 to
2022. Each project is positioned in an orthonormal frame of reference according to its degree of definition of
use (from the most unrestricted use to the most defined use) and its scale of manufacture (from single
experimentation to industrial production). This map tries to bridge the gap between the communities.
Current sustainability contexts highlight the collaborative relationships between humans and living organisms.
Small and growing organisms feed us and produce the raw materials needed to make everyday objects. Kombucha can
do both at the same time. The nature of their two-fold production can be seen either as the produce or the
co-produce. While working with raw materials such as wood (cellulose) to create furniture or domestic objects,
designers and makers have recently used the (macro) co-produce of living symbiotic (micro) organisms. Among
these, the co-produce of Kombucha has been used to create objects and artifacts.
Kombucha beverages contain a bacteria and yeast symbiotic culture that grows in a sweet tea solution. This
association between organisms results in a natural fermentation process that transforms the sweet tea solution
into a solution with nutraceutical properties.
One particularity of Kombucha is that acetobacter makes a biofilm or strain of bacterial cellulose on top of
the beverage. Many communities share strains and form a human-sharing network of kombucha mothers for personal
use.
At a macro scale, this Bacterial Cellulose (BC) is a raw material that designers can treat and process. In
recent years, Kombucha products and co-product have been of interest to many research and design practitioners.
They offer a simple and robust living medium for experimenting with living things, even in a DIY or domestic
setting and urban spaces. More generally, we use the term biofabricated materials to describe the materials
using living organisms in their production (microbes rather than plants or animals).
The science of "Kombucha" as Bacterial Cellulose (BC) is usually segmented into more or less scattered fields:
biology ¹ , medicine ², food ³ , textiles ⁴ , and material science ⁵. This ensemble
draws a vast potential for use and practice, which are only sometimes put in parallel, and remain most often
exclusive. However, it needs to be determined how the materiality of this research influences design choices and
how the designer's communities can mobilize them for developing and employing the best strategies in producing
artifacts.
To tackle this issue, we propose an initial mapping that organizes these heterogeneous BC-based designs and
research domains that elaborate an emerging form of possible new materiality. To build this map, we rely on 38
prior works that their authors classify as "art" or "design" pieces. We focus on works that use either Bacterial
Cellulose as raw material or demonstrate the growth process. This collection reflects the applicative or
demonstrative potential of working with living materials.
Our mapping questions and challenges the possible symbiotic relationships between non-humans and humans.
Addressing these assemblages through Design offers a practical and mobilizable prism of applied bacterial
cellulose's state of the art.
A new take on Symbiotic Design with the Living
Designing and fabricating new materials or objects using living organisms' natural processes of growth or
reproduction is a secular process rooted in the human desire for control over nature. Traditionally, this
process is entrenched in arts and craft fabrication. Artisans can understand raw materials and grow, select, or
shape them according to their knowledge of the processes, creating a collaborative and synergetic process.
"Making is to growing as being to becoming" ⁶ ; the technical transformation of living materials
implies nurturing and cultivation: "if things grow, they are also grown" ⁷. This creation is
beneficial from a design perspective, as it creates the serendipity necessary to emerge new shapes and forms.
Other practitioners are engaged in this discipline. This paper will refer to the users involved in active
collaboration with the living to produce goods as "producers." We identified this process as performed
by several people, including Artisans, Designers, or Makers.
This collaborative process has slowly been forgotten over the past decades or taken over by mass-manufactured
petroleum-based products. The rise of sustainability challenges has raised questions about designing in
collaboration with the living and using the co-product of already used processes, and are at challenges at
several scales. On the one hand, at a large or macro scale, it is often hard for the producer to control the
form of the object ⁸. It involves operation at different stages of the organism's life cycle that allows
shapes or final designs to emerge. On the other hand, the fabrication processes can be controlled at a small or
microscale but are challenging for producers.
The design community used growing materials such as Mycelium or Bacterial Cellulose as emerging material
practice ⁹, leading to the Growing Design movement exploring the emergence of artifacts. Produces use the
materiality of the medium to support the creation of artifacts. It is mainly the case for Mycelium, which allows
the creation of composite bio-materials capable of mechanically structuring the object ¹⁰.
Bacterial Cellulose provides other opportunities. These biological systems can assemble biodegradable and
conformable complex structures that can self-repair, sense, and respond to external stimuli. Bacterial Cellulose
is the product of sugar synthesis by acetic acid bacteria –— the interaction with yeast and other
types of bacteria in SCOBY will not be discussed here. The Bacterial Cellulose grows on the surface of a sweet
tea liquid aerobically, encouraged by the exchange interface between air and the fluid. The overall appearance
of fresh (i.e., wet) bacterial Cellulose is slimy. Once dried, it is similar to paper or leather. The latter
state is most often used as raw material for production in textile design. In addition to being able to be
processed artisanally (using leather work, sewing), this state is more prone to speculative and experiential
uses — they seek to provoke a question or emotional design through touch, sight, or smell (sour).
The works, pieces, and projects made within the community use different fabrication processes and design
methods and target different audiences. While they offer a wide range of variability and diversity, it can be
challenging for designers or makers to understand how the field is structured and how he or they can intervene.
Thus, our proposed mapping offers a global overview and a practical perspective to the still-growing community
of bio designers.
From Design Projects to Classification
Corpus selection
Based on our observation of the kombucha producers —a community in which we take part—we estimate
that artists and designers have produced around a couple of thousand design projects with kombucha. To create
this map, we decided to select representative artifacts that showcase the main capabilities of the material. Our
choice rationale was first to pick projects identified by their creators as "art," "design," "piece," "objects,"
or "garment/cloth/textile."
The prism of practice and production was the second selection criterion. This method excluded most of the
theoretical biology and engineering research projects, drink, and the more known derivatives of the BC.
Furthermore, the projects without direct objective implications in instruments or artifacts were not kept.
Furthermore, in the cases of projects with similar process, outcome, and look, we opt to merge various projects
with one representative visible one.
In this collection, we deliberately exclude aestheticism classification as the reception of works that produce
objects with living matter oscillates between fascination and disgust— even fear (contamination, slimy,
etc.).
Among this list, the projects ranged from social design to product design. Both are between the "doing" and the
more or less driven user prescription. The list of selected projects in our map can be seen in the
Bibliography section.
Methodology and Visual Representation
The main objective of the visual representation is to summarize the information in this heterogeneous corpus.
The second objective is to provide a broad, flexible grid of analysis that highlights, at the same time, links
and differences and provide a visualization tool to understand the variety of use of BC that open up design
potentials.
There are several ways to classify and discriminate the selected works. The choice of factors is motivated by
the desire to characterize the design practices through the spectrum of the objects produced and their relative
fabrication processes.
We used a two-factor classification, used in the design project and methodologies. The first factor was to
consider the use and the user, while the second was to evaluate the production scale. It allows us to visualize
the main benefit of BC on a 2-axis graph.
The horizontal axis represents the production scale. The spectrum ranges from manufacturing —closer
to art, experimental, or artisanal practice—to industrial production. The axis highlights on one side the
uniqueness of the lived experience and the questions raised by observing, experiencing, or interacting with the
artifact, while highlighting the standardization and scale while informing the piece's replicability level on
the other side.
The vertical axis represents the orientation of selected production in relation to practice and use, that is,
the spectrum from passive contemplation to complex use defined in a restricted field. Its scope ranges from
experimental use to defined (prescribed?) use. The top part of the axis values the material quality
corresponding to the need, while the bottom describes more artistic and experimental practices where the goal
isn’t oriented on the quality but on the experience for the spectator or public.
Each selected project was given by the authors two 0-10 grades on the factors: the fabrication quality and the
design prescription. This grade was used to position the project on the 2D axis. It should be noted that we
display superimposed works in adjacency for visual representation. After placing the papers on the grid, the
authors extracted clusters of domains.
Mapping Bacterial Cellulose Design Practices
The Mapping is represented in Figure 1. The 2D axis mapping is divided into four square spaces, with points
spatially representing the projects. Each point is visually represented by a bubble containing an image and
surrounded by its title and main authors. The points (bubble) are joined in clusters by distinct colored frames
referring to domains: medical, industrial, machine, product, craft, speculative, social, contemplative,
experiential.
Spatial Representation
The general location of the projects draws a diagonal from the lower left to the upper right part. The
typologies of the projects selected can explain this. There is a constant progression between the level of
prescription of the use of the BC towards the user simultaneously as the complexity of technical manufacture
increases. Thus, the diagonal represents the increase in the complexity of the functions of the artifacts.
The upper left and lower right parts contain fewer project bubbles than the other spaces. The upper left and
lower right half are mainly empty in our representation. However, some projects could be representative. An
example from the top left could be the use of BC kombucha for food production, thus reversing the notion of
product/co-product in the industry (cellulose paste as an additive, for instance, for sauces or ice cream).
However, these uses are separate from current design practices, as we need representative examples of the
industrial production of a food product as artistic or design production. Thus, this upper left corner is empty
because there is hardly any object that is both mainly prescribed and, at the same time, exceptionally
technically unspecified.
Similarly, the lower right corner is also empty because this space represents artifacts that would be
industrially very advanced (machine and other complex implementations needed for production) yet have very
little prescribed use or even a demonstrative or speculative use. On the one hand, this map area should be
occupied by standardized products resulting from a complex industrial process, waiting for a noble valorization
use —other than composting or methanation, for example. Products of this type may not be design projects,
but industrial co-products and have been excluded from our corpus. On the other hand, while the realization of a
machine or a process is often driven by the creation of a specific and precise object, we can cite the works of
Bianca Hlywa ¹¹ or Stephan Schwabe ¹² that deepen the technical fabrication process for
artistic purposes with the BC.
Clusters: Domains of Interest for Design
The different clusters refer to different domains. The Medical cluster gathers projects developed for medical
use, from cutting-edge research to marketed bandages ¹³. The Industrial cluster gathers projects
developed by industrial actors from an industrial perspective. The Machine cluster gathers projects oriented to
the machine that allows the growing process, like bioreactor—more than the result of the growing process.
Conversely, the Product cluster cares for these growing results and gathers objects made with BC. The Craft
cluster is oriented to artisanal knowledge. The Speculative cluster gathers projects related to speculative art
or speculative design, meaning their operating function is intended to develop critical thinking. The Social
cluster gathers projects that are collective experiences and social forms related to Social Design and
Relational Æsthetics.¹⁴ The Contemplative cluster gathers mostly art projects that aim to
provide an æsthetic experience (primarily oriented to visual sense). The Experiential cluster gathers
projects that offer multi-sense experiential expertise.
From the position of the projects, we defined and distinct nine overlapping clusters that characterize the type
of practices with BC. Each represents a different domain of design that is mainly explored with BC. While not
exhaustive, this list allows the viewer or user of the map to position other works not present in the current
map.
Some projects are linked to several fields. As an example, in the "Craft" category (Figure 2),
Malai.eco¹⁵ is a company that grows BC from coconut to make leather products—as
Kombucha Couture,¹⁶ but with a larger scale and with a real industrial methodology. They
grow BC in bio lab conditions and produce BC by the meter to use this finished piece as material for sewing
bags, shoes, etc. We defined Kombucha Couture as an artisanal production in comparison to Biocoute. Artists such
as Suzanne Lee,¹⁷ who initiated the engagement in design for BC and Biocouture, maintain the craft
relationship to production.
Overall, other overlapping communities have been revealed in our mapping. For instance, the Medical cluster is
porous with other practices because practitioners develop incredibly defined projects with very robust and
rigorous uses and technical production methods. On the opposite side, the Speculative, Social, Contemplative,
and Experiential clusters share most of their points because it is challenging to define the boundaries of their
respective projects. Their positions are less linked to the type of production than to the engagement sought by
their creator with the spectator, the user.
Limitations of the Mapping
This article provides an initial classification of design projects utilizing Kombucha as a base material. With
this map, we contribute to a better understanding of the possibilities and enlarge the scope of existing
projects.
This cartography is a tentative way to structure the artwork from our observation and reflect our opinion that
is open to discussion and debate within the community. We acknowledge some limitations of our mapping. The
classification, according to the position, simplifies the positioning of the selected project and offers a look
and a perspective on a very heterogeneous taxonomy. The selection criteria are related to projects which can
show a new potential use of the fabrication process for BC. Or, we sometimes select one work to best figure for
a tiny cluster of practices, or we pick the most popular and referent work as seen by the community. Although
some projects could have been included, the artifacts resulting from biology papers did not have direct
"objective" implications in the instrument of production and are not seen as design artifacts by their creators
and not mobilizing enough by the designers. In an attempt to overcome this limitation, we created an updatable
interactive map¹⁸ that provides flexible classification choices. The interactive map also allows us
to easily integrate proposals from the community.
Similarly, some objects/artifacts/instruments were difficult to categorize strictly into a field. Often, the
possibility of classifying the objects in two dimensions facilitates decision-making. Most often, the gap and
similarities between the project types define the position of the object in the frame. Thus, it is a look at the
whole production and their characteristics in relation to each other that positions the projects in a more
comprehensive and coherent view.
Conclusion: Statement about the Importance of Kombucha based Design Practices
New Forms for Design: Slow Growth Monitoring
The design practices of Kombucha BC can be applied to the challenges of today's world. With our dependence on
energy, it is important to come up with solutions that are local and meaningful. Choosing to work with living
organisms to maximize production is a skill that humanity has mastered. With the help of (bio)technologies, we
are able to have greater control over the production of living things, leading to new ways of creating materials
through programming or simulation. However, the process is slow and challenges our understanding of resources,
their availability and arrangement.
Our map shows that monitoring biological activity is crucial in redefining how we use living things. We can use
scientific measurement methodologies and bioengineering tools such as thermal imaging, graphs and optimization
techniques to control, analyze and adapt our productions at a slow rate of growth. Monitoring is an essential
part of all the projects presented, providing expertise and control to producers, enabling them to anticipate
production or even design without material. This reinforces the sociotechnical regimes through design, linking
human beings and non-humans through machine mediums.
Emerging Role of Process engineering
The impact of digital manufacturing tools and DIY chemical processes on design has now been followed by the
impact of biology and synthetic biology tools and processes.¹⁹ To design and manufacture with living
organisms, Process Engineering is crucial. By understanding the biology of living organisms, producers can
interact with and mobilize them in production.²⁰ This understanding allows producers to access
dedicated tools and engineering technologies of production that redefine the relationship with emerging things.
These technologies include programming living matter (modification of the environment or genetic reprogramming)
and programming material behaviors (4D printing, programmable matter, structures with auxetic behaviors,
soft-robotics, digital fluid mechanics, etc.).
Our analysis of the corpus shows that these technologies arise from a design valorization of bioengineering
methods combined with DIY fabrication tools.²¹ The combination of living organisms within production
machines challenges the link between parasitism and symbiosis, resulting in machine-organic producers of
"bio-artifacts."
Evolving Role of the Designer
This practice can profoundly transform our relationship with living materials that can now be easily shaped and
transformed in a DIY. Beyond the design with non-human living forms, the fusion between materials science and
synthetic biology is playing out at a new crossroads of ecology that is neither scientific nor animistic.
Working with the living has existed since the dawn of humanity. The rise of these processes can be seen as a
renewal or update in manufacturing and design thanks to the knowledge of the biological matter.
Working with biological processes also questions designers' intention to collaborate with living matter to
repair their practices. They can seek to improve the link between everyday objects and the modern world we live
in, because they could be partly responsible for this unsustainable environment.
While incomplete, we hope this map will foster discussion and generate new ideas in the design community. More
generally, we argue that Design with Kombucha Bacterial Cellulose is a way to connect and re-contextualize the
modern (bio)technological convergence through æsthetics and design.
References
1 Nermin Hande Avcioglu, “Bacterial cellulose: recent progress in production and industrial
applications”, “World J Microbiol Biotechnol”, 2022 Apr 10; 38(5):86, doi:
10.1007/s11274-022-03271-y.
2 Guilherme Fadel Picheth, Cleverton Luiz Pirich, Maria Rita Sierakowski, Marco Aurélio
Woehl, Caroline Novak Sakakibara, Clayton Fernandes de Souza, Andressa Amado Martin, Renata da Silva, Rilton
Alves de Freitas, “Bacterial cellulose in biomedical applications: A review”, Int J Biol Macromol,
2017 Nov, p.104 (Pt A):97-106. doi: 10.1016/j.ijbiomac.2017.05.171.
3 HMC Azeredo, H Barud, CS Farinas, VM Vasconcellos and AM Claro, 2019, “Bacterial Cellulose
as a Raw Material for Food and Food Packaging Applications”, Front, Sustain, Food Syst, 3:7, doi:
10.3389/fsufs.2019.00007
4 A.P. Provin,, dos Reis, V.O., Hilesheim, S.E. et al., “Use of bacterial cellulose in the
textile industry and the wettability challenge—a review”, Cellulose 28, 8255–8274, 2021,
https://doi.org/10.1007/s10570-021-04059-3.
6 P 4, Hallam, Elizabeth & Ingold, Tim (eds.), Making and Growing. Anthropological Studies
of Organisms and Artefacts, Farnham, Ashgate, Anthropological Studies of Creativity and Perception,
2014, 258, Hb.: £61.10. ISBN: 9781409436423.
7 P 3; idem.
8 Idem..
9 Serena Camere, Elvin Karana, “Fabricating materials from living organisms: an emerging
design practice”, Journal of Cleaner Production, 2018, doi: 10.1016/j. jclepro.2018.03.081
12 Stefan Schwabe, The Kernels of chimera, Stefan Schwabe website :The kernels of chimera,
accessed December 08, 2012, http://www.stschwabe.com/work/KoC/.
13 Guilherme Fadel Picheth, and al. “Bacterial cellulose in biomedical applications: A
review”, Int J Biol Macromol, 2017 Nov, 104 (Pt A):97-106, doi: 10.1016/j.ijbiomac.2017.05.171.
14 Nicolas Bourriaud, Relational Aesthetics, Dijon, Les Presses du réel, 2002.
19 Paola Antoneli, “Synthetic Aesthetics: New Frontiers in Contemporary Design” at
MOMA, The Museum of Modern Art, accessed December 09, 2022, Youtube channel
https://www.youtube.com/watch?v=u1D4ArcBjLI.
20 Marion Koelle, Nicolae Madalina, Aditya Shekhar Nittala, Marc Teyssier, Jürgen Steimle,
"Prototyping Soft Devices with Interactive Bioplastics", In Proceedings of the 35th Annual ACM Symposium
on User Interface Software and Technology, 1-16, 2022.
21 Cedric Honnet, Hannah Perner-Wilson, Marc Teyssier, Bruno Fruchard, Jürgen Steimle, Ana
C. Baptista, Paul Strohmeier, "Polysense: Augmenting textiles with electrical functionality using in-situ
polymerization", In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems,
2020, 1-13.
Bibliography
Alanna Lynch, Gut Feelings, 2016, Alanna Lynch website, accessed December 08, 2022,
http://alannalynch.com/portfolio/gut-feelings-2017/.
Ali Schachtschneider, Vivorium, 2015, The New
School Parsons website, accessed December 08, 2022,
https://parsons.edu/bfafashion/ali-schachtschneiders-vivorium-aw arded-rsa-usa-featured-dezeen/.
Bianca Hlywa, Residual yeast, 2022, Bianca Hlywa “Residual Yeast” Gossamer Fog / London by
Deniz Kirkali, accessed December 08, 2022, https://flash---art.com/2022/10/bianca-hlywa/.
Dean Brough, Alice Payne, Peter Musk, 2015, Harvest: A biotextile future, Queensland’s Asia Pacific
Design Library, 1-5 November 2015, as part of The International Association of Societies of Design
Research’s (IASDR) biannual design conference, Queensland ePrint website : Harvest: A biotextile future,
accessed December 08, https://eprints.qut.edu.au/225553/.
Dunne and Raby, United Micro Kingdom, 2013, Dunne and Raby website project of United Micro Kingdoms: a design
fiction, accessed December 08, 2022, http://unitedmicrokingdoms.org.
Elena Amato, Ponto design, 2019, Dezeen website: Elena Amato creates sustainable cosmetics packaging from
bacteria, accessed December 08, 2022,
https://www.dezeen.com/2019/02/28/elena-amato-bacteria-packaging-design/.
Emma Sicher, From Peel to Peel, 2018, Dezen website: Emma Sicher makes eco-friendly food packaging from
fermented bacteria and yeast, accessed December 08, 2022,
https://www.dezeen.com/2018/11/13/sustainable-food-packaging-emma-sicher-peel/.
HBBE (Hub for Bio Building Environnement), Living Manufactures, 2021. Research Team: Thora H Arnardottir
(RA), Joshua Loh (RA), Katie Gilmour (RA), Sunbin Lee (PhD), Meng Zhang (PI), Martyn Dade-Robertson (PI). The
Living Manufacture Project is funded by the EPSRC Manufacturing the Future (Grant Number: EP/V050710/1). HBBE
website : Living Manufacture: Principles for a microbial 3D printer, accessed December 08, 2022,
http://bbe.ac.uk/living-manufacture/.
Jen Keane, This is grown, 2019 - becoming
Modern Synthesis company, 2020, Designboom website: jen keane employs microbial weaving in 'this is grown'
shoe?, accessed December 08, 2022, https://www.designboom.com/technology/jen-keane-this-is-grow
n-microbial-grown-shoe-11-11-2020/.
Le Qara, 2019, H&M Foundation webite : LE QARA, accessed December 08, 2022,
https://hmfoundation.com/gca/winners/le-qara/. Lionne Van Deursen, Unfold, 2018.Van Deursen website :
UNFOLD, accessed December 08, https://www.lionnevandeursen.com/unfold.
Lionne Van Deursen, Unfold, 2018, Van Deursen website : LUNA, accessed December 08,
https://www.lionnevandeursen.com/luna.
Malai.eco, 2016, Malai website: Malai Material, accessed
December 08, https://malai.eco/blogs/news/about-malai-material.
Mary Maggic, Scoby Spin cycle, 2021, In-silo website : Scoby Spin cycle, accessed December 08,
https://in-silo.com/scoby.
Naja Ryde Ankarfeldt, Tale of non-human, 2015, Art Science Stichting website : Naja Ryde Ankarfeldt, accessed
December 08, http://stichting.interfaculty.nl/naja-ryde-ankarfeldt/.
Nitzan Cohen, Emma Sicher, Ignacio Merino Sanchez-Fayos and Secil Ugur Yavuz, “An Open-Source
Bioreactor Enhancing Microbial Cellulose Production and Novel Sustainable substances”, Sustainable
Design and Manufacturing: Proceedings of the 8th International Conference on Sustainable Design and
Manufacturing (KES-SDM 2021), Vol.262, p.77-86; Smart Innovation, Systems and Technologies, 262; KES-SDM 2021,
8th International Conference on Sustainable Design and Manufacturing (Split, 16/09/2021 - 17/09/2021), 2022,
DOI: https://doi.org/10.1007/978-981-16-6128-0_8.
Polybion, 2015, Polybion website : About us, accessed December 08,
https://www.polybion.bio/about-us.
Iowa State University, Iowa State Univrsity website : Clothing made
from tea byproduct could improve health of fashion industry, accessed December 08,
https://www.news.iastate.edu/news/2016/04/26/sustainableclothing
References Regent Medical Ltd,
“EpiProtect” (2001) and “Basyc” (2009) extract from Wilton R. Lustri, Hélida
Gomes de Oliveira Barud, Hernane da Silva Barud, Maristela F. S. Peres, Junkal Gutierrez, Agnieszka Tercjak,
Osmir Batista de Oliveira Junior and Sidney José Lima Ribeiro, “Microbial Cellulose —
Biosynthesis Mechanisms and Medical Applications”. Additional information is available at the end of
chapter 6; http://dx.doi.org/10.5772/61797.
ScobyTec, 2014, ScobyTec website: Uber uns, accessed December 08,
http://www.scobytec.eu/über-uns.
Stefan Schwabe, Growing a Rool, 2012, Stefan Schwabe website: Growing a Roll, accessed December 08,
http://www.stschwabe.com/work/GrowingARoll/.
Stefan Schwabe, The Kernels of chimera, 2012, Stefan
Schwabe website: The kernels of chimera, accessed December 08, http://www.stschwabe.com/work/KoC/.
Stefan Schwabe with Jannis Huelsen, Xylinum Cones/ Programming, 2014, Stefan Schwabe website: The kernels of
chimera, accessed December 08, http://www.stschwabe.com/work/XylinumCones/.
Suzanne Lee, Biocouture, 2003 - 2013, This is alive exhibition website: biocouture, accessed December 08,
http://thisisalive.com/fr/biocouture/.
Thinking Hand Studio, Feral Fetish, 2021, Thinghand Studio
website: Feral fetish, accessed December 08, https://studiothinkinghand.com/works/feralfetishfilm.html.
Thr34d5, Regrow, 2018, Thr34d5 website: Regrow, accessed December 08,
https://thr34d5.org/2019/08/20/regrow/.
Thr34d5, Kombucha tsugi: the bag edition, 2021, Thr34d5 website: Kombucha tsugi: the bag edition, accessed
December 08, https://thr34d5.org/2021/03/14/kombucha-tsugi-the-bag-edition/.
Hunter Whitefeater,
Bucci, 2017, Whitefeather hunter website: “bucci collaborative project at fashion pop, Montreal”,
accessed December 08,
https://whitefeatherhunter.net/2017/09/13/upcoming-sept-whitefeather-presents-collaborative-project-at-fashion-pop/.
Xxlab, Soya C(o)u(l)ture, 2015, Ars Electronica website: SOYA C(O)U(L)TURE – Useful Things arise out of
Waste, accessed December 08, https://ars.electronica.art/aeblog/en/2015/09/30/soya-coulture/. Yoko shimizu,
Layers of life, 2014. Shimizu website : layers of life, accessed December 08,
https://yokoshimizu.com/portfolio_page/layers-of-life/.
The Hitchcock Experience - a Spatial Montage project
1École des Arts Décoratifs–PSL (EnsadLab) 2Univ. Grenoble Alpes,
Inria, CNRS 1Paris, France 2Grenoble, France remi.sagot@gmail.com,
nils.quaetaert@polytechnique.edu, francois.garnier@ensad.fr, remi.ronfard@inria.fr
Abstract
This paper describes an ongoing art project exploring the generalization of classical montage theory to the
emerging technology of “room scale VR”. We take the emblematic “crop duster” scene in
Alfred Hitchcock's North by Northwest and invite the spectator to experience it “from the
inside”. This raises interesting research issues. How to inhabit this cinematic story world? How can the
traditional tools of cinematography and montage be used to direct the audience in this new kind of experience?
How can the rhythm of the spectator's body be matched to the rhythm of George Tomasini’s
(Hitchcock’s editor) fast pace editing? And what form can this montage take and for what aesthetic and
dramaturgical effects? To answer these questions, we propose the experiment of a montage that adapts in real
time to the displacements of the body and the gaze of the spectator, engaging a dialectic between narrative
rhythm and bodily rhythm. We propose new algorithmic tools to transport the audience into the story as
originally planned by Hitchcock while at the same time respecting the behaviors of the audience to guide the
experience. In doing so, we seek to create a new form of relationship between the author of a narrative
experience in virtual reality and the spectator who explores and activates the experience with his own body.
While the imaginary around virtual reality devices generates fears and fantasies, it seems essential to
question the explicit and implicit bodily interactions that we have with digital spaces. By its algorithmic
nature, the virtual environment, calculated in real time, constitutes a functional space that can be manipulated
by the immersant (1) but can also control and constrain his/her movements and the sensitive
experience, according to a relationship set by an author.
These explorable environments can also carry within them an emotional dimension due to their digital image
nature. When dealing with a virtual reality device, our bodily habits of space cohabit with our visual habits of
image. We think therefore that it is possible to invoke our culture of the cinematographic images in the
spatially lived aesthetic experience.
With the cinema, we agree to sit still for the duration of a film and let ourselves be hypnotized by the gaze
of a director. This privileged moment, out of time, offers a frame to rethink our relationship to images (to
their time, rhythms and meanings). Depending on the quality of the montage, we leave our own space-time aside to
align our perception and our thoughts with the rhythm proposed by the film.
Thanks to virtual reality devices, we are able to inhabit a cinematic representation space. How then to adapt a
cinematographic dramaturgy while allowing the emergence of new phenomena related to the involvement of the body
in action specific to the immersive nature of virtual reality displays? Should we be limited to explore the
scene without discontinuities or should we experience a form of montage? To answer these questions, we designed
a virtual reality experience (Figure 1) that immerses us inside the mythical “crop duster” scene in
Alfred Hitchcock's North By Northwest. (2)
In the original film, the montage by George Tomasini (Hitchcock’s editor) is essential to a good
understanding of the space and the distances that separate the character (played by Cary Grant) from the other
characters and vehicles that interact with him ¹ . Moreover, its evolution through the sequence highly
participates in suspense, tension and violence. But if the spectator is physically immersed in this space
and able to move through it, the rhythm of the editing needs to be matched to the one of his/her body. This
implies a new aesthetic and dramaturgical approach due to the spatial shift of the virtual reality medium. These
are the issues that guide this research-creation project entitled: The Hitchcock Experience
(3).
We propose the experiment of a montage guided by the movements of the immersant. It explores the questions of
distance and identification with the character by transposing certain cinematographic shot scales into the scale
of the immersed body regarding the represented space. Moreover, this experiment proposes a form of editing that
adapts in real time to the displacements of the body and the gaze of the immersant, engaging a dialectic between
the narrative rhythm and the one of the experienced space. We wish to explore a new relationship to the ongoing
narrative, profitable to the emancipation of the immersant and harmonizing his/her bodily rhythm to the one of a
fragmented inhabitable environment.
The aim of this experiment is not to propose a VR experience that competes with Hitchcock's film. By
confronting this masterpiece of the cinematographic montage, we precisely try to understand, through comparative
analysis, the profound transformation implied by its adaptation in virtual reality.
First, we will establish related works regarding the practice of montage in virtual reality devices. Then, we
will explain our research-creation process and comment on the different steps we went through developing our
spatial montage system. We will expose our experimental protocol that we set to collect data from 17
participants. Then, we will show our first relevant results and observations. Next, we will discuss and initiate
theoretical explorations on the aesthetic experience of the immersants. Finally, we will conclude on our spatial
montage device and the role of its editor.
Related work
The problem of directing techniques suitable for building "room-scale games" and more specifically "room-scale
movies" is widely overlooked in recent academic research. Contemporary room-scale VR experiences are based on
the strong sensation of presence caused by forcing the viewer to remain in a fixed space and time. As a result,
the use of montage is the exception rather than the rule. Hodgkinson ² describes some techniques used by
Google Spotlight Stories (4) for moving the immersant in space and in time during a VR experience,
which bears some resemblance with traditional montage. For example, Rain or Shine (directed by Felix
Massie) uses visual occlusion to smooth out unwanted camera motions. Pearl (directed by Patrick
Osborne) introduces temporal ellipses which can guide the narration, while keeping the same spatial location.
Age of Sail (directed by John Kahrs) allows the immersants to navigate between a small number of
spatial locations, in effect allowing them to create their own spatial montage.
The question of montage has been more deeply explored in the related, but different, context of cinematic VR,
where the immersant can only turn his/her head horizontally across 360 degrees and vertically across 180
degrees. If he/she can choose the direction of his/her gaze in the sphere image, the viewer cannot move his/her
point of view to explore the spatial dimension of the scene, which is the specificity of room-scale VR. In this
limited context, Jessica Brillhart³ has proposed the notion of a “probabilistic experiential
montage” where each 360 degree shot offers the possibility of multiple experiences by the immersant, and
the montage is created on the assumption of the most probable experience.
Garnier⁴ provides a detailed account of the various geometries involved in viewing film, either on a
traditional screen or in a virtual reality headset, and emphasizes the importance of a proxemic interpretation
of shot sizes in film. From another perspective, Pope ⁵ also recommends the use of proxemics for
staging in VR following some well-established theater techniques.
Rothe et al.⁶ propose an analysis of camera control in cinematic VR and also relate shot sizes and
camera distances under the framework of proxemics. Rothe also examines the effect of camera height in distance
and size perception in cinematic VR⁷.
In our previous work entitled Reframing VR,⁸we propose a vocabulary of shot
values suitable for room-scale VR based on the new notion of a spatial frame. In our framework, spatial montage
can be defined as a temporal arrangement of spatial frames with different spatial scales, which play a similar
role to the shot sizes used in traditional film theory. Because the immersant is free to move in the virtual
world, “shot size” is not measured only in terms of apparent visual size, but also in motion
parallax and capacity of action.
Artistic process
Regarding this state of the art, we propose a process of research-creation, based on an artistic device
engaging a reflexive loop between practice of the "spatial montage" and theorization. In this part, we explain
how we elaborated our artistic experiment.
Scene reconstruction
Based on a 3D scene provided by the Anima team of Inria (5), we virtually reconstructed the entire
set and all the props, characters and vehicles present in the scene using representative keyframes from the
movie. We then created a rough "layout" animation of all the character and vehicle movements, synchronized frame
by frame with the original movie. We used a floor plan view diagram from Raymond Bellour’s The
Analysis of Film ³ as a reference to approximately reconstruct the trajectories of all
characters, vehicles and the plane, as shown in Figure 2, and fine-tuned them to match to the existing views
from the movie. Some movements could not be seen in the original shots and had to be crafted manually to
plausibly match the visible parts.
This work enabled us to build a 9 minutes and 45 seconds 3D animated scene corresponding to the action of the
original film. We then imported this scene into the Unity game engine to implement the virtual reality set up.
Re-framing
In order to immerse the spectator into this reconstructed 3D scene, we had to find how to translate the frames
of the movie to a spatial experience specific to virtual reality.
In room-scale VR, the perceptual boundaries of a continuous virtual reality experience are based on the
subject’s freedom of movement in the real environment. The technical equipment and the perimeter of the
physical space tracked by the device therefore seems to correspond to a frame. In the virtual environment, it
defines a scaled, spatialized and oriented zone in which the subject acquires a limited walking and acting area.
We therefore decide to take this zone of potential action as our editing cell. The frame in virtual reality no
longer delimits a portion of the image as in cinema, but a zone of possible actions in a space.
This
space area constrains the movements that the immersant can perform. By changing the relative sizes of the
virtual and the physical worlds, we can define "wide shots" with miniature non-player characters where the room
appears to be 100 meters wide; "long shots" with dolly-size characters, where the room appears to be 10 meters
wide; "medium shots" with three-quarter-size virtual characters; "point of view shots" with real-size virtual
characters; and even "close shots" with larger-than-life characters where virtual movements are limited to a
meter or less.
We started by transposing the shot grammar proposed by Hitchcock in our own “immersant’s
scale” classification. By testing a variety of scales of the immersant and his/her physical tracking space
relative to the scale of the virtual environment, we experimentally found different size ratios that seem to
reproduce the distancing effects of the cinematic frame. Moreover, the closer we get to the character’s
scale, the more we come to inhabit his space and the more we feel we belong to the diegesis. Finally, we
analyzed that each scale brings a difference of amplitude of our bodily action in the virtual environment (a
more or less large space is reachable) implying different agencies in the reception and therefore a new rhythm
to find for the montage of these scale variations in space and time.
We then adapted the position and orientation of the scaled area (Figure 3). In order to reduce the motion
sickness issues, we imposed that the zones had to always be parallel to the ground. In other words, they could
not be tilted in their x or z axis. Moreover, at the beginning of each “spatial shot”, the y axis
orientation of the imersant’s gaze is imposed no matter what he/she was looking before the cut.
In a first step, we placed our oriented scaled areas regarding the camera positions of Hitchcoch’s
montage. Then, we decided to take some liberties and to re-cut the sequence regarding the medium shift of our
device.
Re-cutting
The form of the cut that we propose consists in a change of scale and in the setting of the position and
orientation of the immersant's point of view (Figure 4a). Each cut imposes an initial point of view
(“initial gaze frame”) to the immersant who is then free to choose his/her gaze movement until the
next cut (Figure 4b). So the “final gaze frame” depends on the immersant.
In order to adapt the editing, we started by slowing it down. First of all, we have to take into account the
necessarily longer processing time of stereoscopic images with a large field of view. Considering spatial breaks
comes up with specific cognitive constraints related to the perception of virtual environments that implies
spatial reasoning ¹⁰. A too short time between two cuts can make the image unintelligible.
That’s why we allowed a time for exploration, choice and habitability of the virtual environment.
According to the scale and to the action which takes place, we evaluated the necessity of the montage. By moving
his/her body and viewpoint, the immersant performs a form of editing that must be taken into account. For
example, at the beginning of the sequence, Hitchcock uses shots of about three seconds alternating between a
third-person view of the character and his subjective view of the landscape he is looking at. Reproducing this
montage while the immersant is free to move around does not work well. Instead, the rotation of his/her gaze
must be taken into account to produce the same effect.
During this step, we studied empirically which cuts proposed by Hitchcock were replaced by the moving point of
view of the immersant naturally positioning itself as intended by the film’s editing. We thus observed
that the cuts linked to the spatial understanding of the space had to be revised. In addition, imposing a point
of view on the empty road systematically incited the subject to turn his/her head, which lost the desired
dramatic result. In order to have an effect, the shifts in viewpoint must be justified by a shift in distance or
the desire to create a rhythmic change effect.
Then, we were confronted with the fact that the cuts can create a conflict between the immersant's point of
view and the one imposed on him/her. These imposed discontinuities can be frustrating and/or seen as a
punishment. Moreover, if it takes place during a head rotation, the cut can be disorienting. The two rhythms
enter into a struggle. If the immersant lets go and stops moving, he submits to the rhythm of the editing
accepting its tempo. For a better experience, we therefore study how to let the immersant create his/her part of
the tempo.
Re-cutting with cues
We developed tools with the Unity game engine to enable a montage that adapts to the behavior of the immersant.
It reacts to a system of "cues" produced by the human-computer system during the experience.
These cues can be represented by boolean variables computed in real time according to biometric data (gaze and
body movements), geometric data (arrangement and orientation of objects in the virtual space) and temporal data
(event system).
We use the following preliminary cues:
“Does not move”: the immersant is not moving faster than a certain value.
"Is looking at": the immersant is watching a certain character or vehicle.
"Object in frame": A certain character or vehicle enters or leaves the immersant’s field of view.
We define a type of computing object capable of making the transition between two previously defined spatial
frames when the right conditions are verified (Figure 5, right). By setting these objects on a timeline (Figure
5, bottom), we can choose to activate and deactivate them at the desired time. This enables us to define a time
range in which a transition can take place. Instead of having a cut defined at a precise moment, we have a
delocalized transition that may occur within a certain duration. We call it: "cut porosity". The moment of the
cut is then determined in real time according to the received cues. These tools allow an editor to assemble the
experience through the timeline interface. The main difference with traditional editing is that the editor works
on the duration of the transitions’ possibilities rather than that of the rushes. This system also enables
the creation of branches: it is possible to define several transitions that can take place depending on the cues
encountered. This feature makes it possible to create alternative setups depending on the immersant’s
behavior.
Using this new system of cues and dynamic timeline, we created a new montage that adapts to the rhythm of the
immersant. First of all, we mainly used the "Does not move" condition to make sure that the subject has
stabilized his/her gaze before the cut (final gaze frame) to avoid disorienting him/her. We also often used the
"Is looking at" condition to trigger the cut when the viewer's gaze is stabilized on the character. This allowed
us to create on-axis cuts with scale changes to reduce our distance to the action. By using this condition, we
were also able to control changes in focalization (from sympathy to empathy) depending on our interest in the
character or not. Finally, the "Object in frame" condition was used when watching the plane. When the latter
left the field of view, there was nothing to see but the sky, so it was useless to stay in that position.
To illustrate the importance of the cues system combined with the changes in scale, let's take the example of
the first plane attack. In the film, this moment is edited with a shot/reverse shot between the crop duster and
the character observing it until he understands at the last moment that he is the target. By alternating shots
showing the plane getting closer and closer to the camera and shots showing the character in tighter and tighter
frames, Hitchcock creates an attraction between the plane and the character until the two meet when Cary Grant
jumps to the ground.
If we apply this same editing in VR, we must first take into account that the immersant might perform the field
against field by him/herself. It was therefore necessary to have the "Look at plane" condition to switch to the
character or the “Look at character” one to switch to the plane. It is indeed useless to show the
plane if the immersant is already looking at it. It just creates an unpleasant jumpcut. Moreover, the condition
"Does not move" avoids that the cut takes place when the immersant is turning his/her head and thus being
punished or disoriented.
But, as the cues are limited in time, there is a moment when even if they are not reached, it is necessary to
move to the next point of view. This is where the scale change becomes necessary to create a shift in distance
(close-up effect) to the character, even though the immersant is looking at him before and after the cut. By
downscaling from the scale of Cary Grant to the one of a mouse (Figure 6), proper editing effects are created
when the character falls, even if the "Look at" or “Does not move” conditions are not satisfied.
Finally, it was necessary to work on the time given to the activation ranges of the conditions. The more the
range extends in time, the less the editing can impose its rhythm. In this case, the cut may arrive at an ideal
moment for the immersant but be misplaced and meaningless in terms of the narrative. We therefore had a tendency
to shorten these areas to establish the rhythm of the montage, imposing at certain moments pace accelerations
where the immersed body has almost no time to move between several cuts. When the character falls on the floor,
it is important to have the cut at this precise moment. This was also particularly the case when the first truck
passes, or during the final climax when the tanker truck arrives, hits the character and the plane crashes. It
was therefore a matter of finding a balance between the editor's pleasure in imposing a rhythm that he/she liked
and the frustration of leaving some freedom of rhythm to the immersant. We then had to evaluate this balance.
Experimental protocol
In order to conduct a qualitative analysis of this montage, we set up an experimental protocol that we tested
on 17 participants. 12 of them were novices to virtual reality.
The immersion in virtual reality is made with the HTC Vive Pro Eye system. A Vive tracker is attached to the
participant's plexus in order to record his/her body position and orientation data. In order to use and record
the eye tracking data, a calibration of the participant's eyes was necessary. The participant was asked to move
within a square of 2.80 m on each side. These boundaries were reproduced in the virtual environment. The
participant is filmed. A video of his/her point of view is captured. The eye tracking data and the position and
orientation of his/her head and chest are recorded.
The experimental protocol consists in living successively two versions of the montage of 9 minutes and 45
seconds. In the first one, the immersed person always remains at the same scale as the character and we impose
to him/her the rhythm of the editing planned by Hitchcock and George Tomasini. Between two editing points, the
immersant is able to move in space, but at each cut, he/she is positioned and oriented like the film shot.
Whatever we do, the cuts thus bring us back into the authors' point of view at the moment chosen by them. In
other words, if I don't move, I see the film as it was cut in the original film. This “Hitchcock”
version (we called H) contains 133 cuts and the average time between two cuts is 4.4 seconds. The second
experience consists of living the version of the montage edited by us with the changes of scale and the system
of cues reacting to the immersant’s behavior. This “re-cutted” version (we called RC) includes
56 cuts and the average time between two cuts is 10 seconds.
The order of the two experiences is arbitrary and varies for each person. After having experienced the first
one, the participant is invited to answer a questionnaire relating to the bodily and cognitive appreciations and
feelings, the sensation of presence, the relationship to the narration and the editing. Then, the participant
experiences the second version. The same questionnaire is therefore to be completed. Some questions, specific to
this second questionnaire, concern a comparison between the two experiences.
Moreover, an oral interview of about twenty minutes is conducted allowing the participant to justify his/her
answers to the questionnaires, to criticize the experience and to go into more detail about his/her sensations
and impressions.
An authorization for the recording, use and distribution of images, videos, sounds and biometric data is signed
by each participant.
Finally, we reconstructed in Unity the movements of the participants, enabling us to replay the experience in
the 3D space. We can thus replay the experience either on a screen or immersed in virtual reality with a
headset.
The protocol of the experiment is presented in this video: https://vimeo.com/779289940.
Questionnaires’ results
In this part, we present the first observations made from the answers to the questionnaires. These have helped
to guide our reflection and to draw some conclusions about the effects experienced through the montage in
virtual reality. For each question, the subject could choose a qualitative answer ranging from-2 to 2 (Figure
7).
In terms of general feeling, our "re-cut" (RC) version of the montage was more appreciated than the direct
transposition of Hitchcock's montage (H) in VR. To the question "I was absorbed by the experience", RC gets a
better score (0.94) than H (0.53). To the question "I didn't feel confused or comfortable", H is less well lived
(0.22) than RC (0.63) when experienced first. To the question "I had the sensation of being there in the virtual
environment", the score is better for RC (0.56) than H (-0.13) when it is experienced in second. This indicates
that the phenomenon of presence persists for RC even after the startling effect of the VR technological
experience has faded. Finally, on the question "I liked this experience better than the previous one”, RC
scores higher (0.5) than H (0.0).
A part of the questionnaire also focused on the phenomenological relationship related to the mediation. To the
question "Did you feel more immersed in images or spaces?", the subjects feel more in a space during the first
experience (H or RC). This seems logical given that when they put on the headset for the first time, they
strongly engage their spatial perception system. On the other hand, during the second experience, the immersants
feel more immersed in images with H. To the question "I had the impression of sharing the same space as the
character", H obtains a better average score on the two experiments (0.76) than RC (0.35) which seems logical
given that in H, the immersant keeps the scale of the character. We should note that this impression is more
marked for H if it is experienced before (1.0) than after (0.5) RC. This result is interesting to put in
relation with the fact that the immersant seems to feel more in images in this last case.
Concerning proprioception, to the question "I felt like I didn’t have a body", RC obtains a null score
(0.0) whereas the one of H is negative (-0.47). This could be explained by the fact that in H, immersants
don’t have time between two cuts to conscientize their bodies in space. Moreover, the score increases
significantly in the second experiment. This confirms the tendency of immersants to move from immersion in a
space to images when they get used to the fragmented grammar of the experiences.
Regarding the relationship with the "cuts", RC seems to be better experienced. To the question "I was
disoriented by the cuts", the difference is very clear between H (0.94) and RC (0.0) as shown in Figure 8. It is
also interesting to note that the score becomes even negative when RC is experienced in second. We get the same
tendency for the question "I was frustrated by the cuts" (0.35 for H and -0.47 for RC). Finally, on the question
"I feel like I missed some important things in the story", RC gets a better score (-1.59) than H (-0.94).
Discussion
These experiments enabled us to refine an analysis of the aesthetic experience of the immersant. Based on the
answers to the questionnaires and the interviews conducted with each participant, we were able to extract
certain theoretical explorations.
The rhythm of the immersant: a dialectic between looking and inhabiting
When immersed in a virtual environment, the immersants must quickly understand their role. Is it an environment
where they can interact explicitly? Are their presences taken into account by the narrative? Should they move or
stay still, explore or contemplate, inhabit or watch? According to the answers to these issues, we observe that
the immersants engage their perceiving body according to different dynamics.
The rhythm of the immersant is first influenced by his/her own characteristics. According to his/her emotional
state, expectations, habits of virtual reality systems, he/she will engage his/her body differently. While some
remain still, naturally observing the mediated spaces, others will be active in exploring the limits of the
simulation in a more videogame-like approach. We can thus make the hypothesis that there are typologies of
immersant to detect in order to adapt a rhythm of montage.
The contract that is created between the immersant and the artwork depends on the expectation of the mediation
experienced. When dealing with the device, we observed that some participants believed that they must adapt to
make the system work and get the most out of it, while others expected the system to adapt to what they wanted
to do. In other words, the contract depended on the responsibility the immersants give themselves in the quality
of their artwork reception experience. On this point, the answers to the questionnaires seem to show that the
participants live better the experience (RC) when they have the time to position their gaze and to control a
minimum their reception.
In our device, the immersant is standing and knows that he/she can move at any moment, which creates a
particular relation to the perceived image. We don't observe in the same way when we know that we can look at
something else at any moment. The tension of Hitchcock's sequence, created by the rhythm between shots of the
character and of what he sees, is changed into a tension between deciding to look at or not to. But if the
editing systematically orients our gaze toward what we need to see to follow the story, the exploration becomes
meaningless. In this way, the rhythm of the immersant depends on the staging. Does he/she have something to see?
If yes, then why should he/she move? A working hypothesis is to decentralize the point of view regarding the
salient narrative elements, inciting the immersant to seek a new point of view. The rhythm of the immersant
could thus be influenced by the time he/she takes to switch from the exploration of the environment to the
stabilization of his/her gaze on an image.
Then, according to his/her scale in relation to the representation space, the immersant needs more or less to
engage important movements. A point of view of a giant densifies his/her distancing towards the elements of the
scene. He/she does not need to move much to approach any element allowing him/her to embrace globally a large
space to situate a scene. This "dollhouse view" engages a heterocentric referential of spatial perception,
inducing less body interaction. Being so different from his/her size, the environment is on the one hand not
inhabitable, but seems to increase the emotional distance to the narration.
Moreover, we notice that if the immersant is the same size as the character, he/she will naturally have a more
exploratory attitude. Moreover, his/her presence in the representation space will be better induced and the
behavioral responses (bending down when the plane passes or leaving the middle of the road when the truck
arrives) will be stronger.
Finally, if the immersant is very small in proportion to the environment, the translations of the body have
much less amplitude to explore the scene. On the other hand, since the whole world is larger, head rotations
have much more impact on changing what the immersant sees in its field of view. In other words, the more we
reduce the size of our virtual body relative to the environment, the more we are confined to a localized area
but the more we have to turn our head to follow the narrative's salient elements.
Each scale brings a different relationship to our potential actions and to the more or less tangible nature of
the representation space. They offer specific qualities of perception and engagement ⁴. These
relationships evolve over time and will be shaken up by the discontinuities of the montage.
The rhythm of the editing: between image rhythm and space rhythm
The rhythm of the editing gives a limited temporality to the action and perception characteristics of the
immersant's scales. The spatial reference frame does not persist, which leaves only a limited time to put in
relation the physical body and its virtual scale. According to the delay between each cut, the immersant is more
or less conscious of the presence of his/her body in the representation space. If the montage is slow, then
he/she can get used to the scale of his/her body, go from perceiving an image to inhabiting the space, exploring
it, building a point of view and having the time to re-stabilize the gaze on an image.
If on the contrary the montage is fast, the image does not have time to become a space. A frenetic editing may
not leave time for a behavioral response. We saw in the results that it also may affect the awareness of the
perceiving body. We make the hypothesis that these images (not yet spaces) are printed differently for the
perceptive system and modify the inner construction of the space. This could be confirmed by the fact that
participants felt more immersed in images when they made H (133 cuts) after RC (57 cuts). The rhythm of the
montage has thus a role in the transformation of the space into image or of the image into a space, and thus in
the nature of the mediation.
Depending on the scale and initial point of view (initial gaze frame) imposed by the cut, the editing can also
impose a movement by decentering the viewer in relation to the action. The proposed point of view would then be
perceived more as a position in space than as an image. This spectatorial decentering should invite the
immersant to move in a certain way towards the search for framing an image. Through editing and staging, the
immersant can thus adopt a certain rhythm linked to the recentering of his point of view on the action. This has
the effect of changing the nature of the cut, which here would be more of a collision than a continuity.
Matching the rhythms
Discontinuities in perceptual experience are not natural in the framework of mediation in virtual reality.
There is a difficulty in repositioning ourselves in a sudden new virtual space ¹¹ and a need to
provide the immersants certain visual cues so that they can orient themselves ¹². We saw in the
questionnaires’ answers that in H, cuts were often frustrating, disorienting and meaningless. In
interviews, participants often complain about the beginning of H when there are cuts every three seconds between
the character and what he sees. We hypothesize that in order for them to be accepted, and especially to acquire
a meaning, the cuts must be part of the acceptance of a cinematographic language of the experience. It would
therefore be necessary that before and after the cut, the immersant shall be focused on an image rather than in
the process of exploring. By detecting the moments when the immersant is looking for a point of view, we can
avoid making cuts at such moments. When this exploration phase ends and the gaze stabilizes on a final gaze
frame, the montage can then propose a change in response to the stabilized image.
Then, in order to match the rhythms, it also seems important to pay particular attention to the first contacts
between the immersant and the montage. A montage too fast at the beginning can discourage the subject from
moving. It seems important to allow time for exploration at the start of the experience and to gradually change
the pace as the language becomes better established. The complexity lies in the fact that we must both
intuitively invoke our culture of film editing while at the same time liberating our bodies in space. If we
realize that it is the role of the editing to show us points of view, then our role is no longer to go and find
them by ourselves. That is why it is important for the editor to take into account this progressive learning of
the immersant to set up the cues and their durations. By creating this relation between cues and immersant, the
role of the editor is here fundamental in the harmony of the rhythms and in the resulting aesthetic impressions.
Narrative effects
While the conducted experiments have raised new phenomenological questions, they have also opened up new
directions of reflection in the theorization of narrative effects linked to montage in virtual reality. Firstly,
thanks to the system of cues, when the cut is made between two stabilized points of view, we can observe the
persistence of a “Koulechov Effect”6 in virtual reality. In other words, the creation of a narrative
meaning extending beyond the two separate points of view. We might also explore the difference of this Koulechov
Effect if the immersant passes from his/her point of view to the one of the montage or if he/she chains two
points of view of the montage (if he/she doesn’t move between two cuts).
A second track to explore is the multiplicity of points of view that the montage proposes or prevents. As the
“framemaker” of his/her own experience, the immersant can decide to look at a character or at what
the character is looking at. This engages different effects of identification (with the character or with
his/herself) and testifies to his/her empathetic or sympathetic relationship towards the character. These
choices vary the level of presence in the environment and the distance to the narrative. By imposing changes in
scale and point of view, editing can in turn constrain some of these effects in time. Moreover, depending on
these changes in relation to his/her attitude, the immersant may or may not feel that he/she is missing events.
We have seen that according to the scale, the immersant more or less inhabits the virtual environment. We
hypothesize that with the size of Cary Grant, the immersant engages more behavioral responses to events (at the
character’s scale, it seems normal to feel more embodied and therefore in danger). On the contrary, in
larger scales, the immersants feel more like a "framemaker-body" of their own experience. By framing the
representative space, they don't change the story but the way they perceive it and thus their focus and
aesthetic impressions of the story. The immersant would then be a "director-body", seeking to transform space
into image. In order to be an "editor-body", he/she would have to make a conscious choice between different
images in time. Perhaps the "editor-body" emerges when the control of spatio-temporality becomes conscious.
Finally, when setting up the cue system, we systematically asked ourselves the following question: is the
interaction implicit or explicit? If the immersant realizes that his/her gaze actions have an influence on the
editing rhythm, then the narrative experience changes. The montage may thus become embodied and open up new
sensitive connections with the ongoing narrative.
Conclusion
In this project, we have explored how montage theory needs to be adapted to the new context of room-scale VR.
We found that proprioceptive immersion in a virtual story world can indeed provide the necessary
“cues” to motivate shifts of points of view and scales that effectively guide the audience through a
narrative experience. These meaningful discontinuities create a rhythm in the experience that the audience can
follow and enjoy if they are correctly synchronized with his/her internal bodily rhythm.
Depending on the montage rhythm and its adaptability to the attitude of the immersant, we observe a dialectic
between the exploratory nature of the space and the narrative nature of the image. Our experiment offered the
opportunity to analyze how the dynamics of the montage may suggest to the immersant an exploratory rather than a
spectator attitude, or vice versa. It constitutes a user study of how our spatial habits can harmonize with our
visual habits of cinematographic images in order to make sensible the oscillation between constraints and
liberties.
We made the hypothesis that the montage must also adapt to the rhythm imposed by the immersant. Thus, we
explored how the montage can adapt in real time to the behavior of the immersant through implicit interactions
(focus of the gaze, objects present or not in the field of vision, speed of displacement and rotation of the
head), while respecting a meaningful narrative structure. The artistic and narrative impressions of the space
intervals result then from a compromise between the proposals of the immersant and of the montage set by an
editor and actualized by the software.
In future work, we would like to explore further this new form of relationship between the immersant (present
and acting) and the author (absent) by introducing a richer repertoire of cues and actions, and creating even
more variations in the resulting montage.
(1) Person immersed in a virtual environment. Crop duster scene timecode: from 1:06:17 to
1:15:55.
(2)North by Northwest, Alfred Hitchcock, Metro-Goldwyn-Mayer, 1959.
https://www.imdb.com/title/tt0053125/
1 Alfred Hitchcock, François Truffaut, Helen Scott, Hitchcock/Truffaut, Paris,
Gallimard, 1993.
2 Gray Hodgkinson, “Cut, don’t cut-moving the viewer through a story in animated
VR,” in 7th International Conference on Illustration and Animation, 2019.
3 Scott Macaulay, “Look Into the Cut: Jessica Brillhart on Editing VR,” Filmmaker
Magazine, October 28, 2015.
4 François Garnier, “Artistic Practices in Digital Space: An Art of the Geometries of
Movement? (2021),” in Space-Time Geometries for Motion and Perception in the Brain and the
Arts, ed. Tamar Flash and Alain Berthoz, Cham: Springer International Publishing, 2021, 237–50.
5 Vanessa Pope et al., “The geometry of storytelling: Theatrical use of space for 360-degree
videos and virtual reality,” in CHI Conference on Human Factors in Computing Systems, 2017.
6 Sylvia Roth, et al., “Where to place the camera,” in ACM Symposium on Virtual
Reality Software and Technology, 2019.
7 Sylvia Roth et al., “The impact of camera height in cinematic virtual reality,” in
ACM Symposium on Virtual Reality Software and Technology, 2018.
8 Rémi Sagot-Duvauroux, François Garnier and Rémi Ronfard, “(Re-)Framing
Virtual Reality,” in WICED 2022 - 10th Workshop on Intelligent Cinematography and
Editing, April, 2022.
9 Raymond Bellour, The Analysis of Film, Indiana University Press, 2000.
10 Ogechi Nnadi, Ute Fischer, Michael Boyce, and Michael Nitsche, “Effect of Dynamic Camera
Control on Spatial Reasoning in 3D Spaces,” in ACM SIGGRAPH Symposium on Video Games - Sandbox
Symposium, Los Angeles, California, August, 2008, 9–10.
11 Michel-Ange Amorine, Ben Trumbore and Pema L. Chogyen, “Cognitive Repositioning inside a
Desktop VE: The Constraints Introduced by First-versus Third-Person Imagery and Mental Representation
Richness,” Presence: Teleoperators and Virtual Environments 9 (2), 2000, 165– 186.
12 Karl E. Steiner, Lavanya Voruganti, “A Comparison of Guidance Cues in Desktop Virtual
Environments,” Virtual Reality 7 (3-4), 2004, 140–147.
Animate: A Theatrical Exploration of Climate Transformation through the Medium of Extended Reality (XR)
ChrisSalter, TimothyThomasson, PierrickUro
Immersive Arts Space, Zurich University of the Arts, Concordia University,
McGill University christopher.salter@zhdk.ch , timothythomasson@gmail.com,
pierrick.uro@mail.mcgill.ca
Abstract
This paper presents a critical account of the development of a large-scale theater work using emerging Extended
Reality (XR) technologies. Detailing three aspects of the project and set against theoretical frameworks from
STS (Science and Technology Studies) and the sociology of innovation around ideas of the future embedded in
technologies, we examine the conceptual, aesthetic, organizational and social-technical underpinnings of the
project. The paper’s goal is thus to give a sense of the challenges and opportunities in the emerging
integration of XR into new artistic morphologies that hybridize the visual-performing-media arts through new
technological advances.
Animate is an Extended Reality-based theater work (XR) at the crossroads of performance, radio play
and installation focused on a near-future Canada radically transformed by climate change. Developed by a team of
Canadian and German artists and researchers, the work premiered in Summer 2022 in large theater festivals in
Germany and is preparing for a 2023 international tour in Europe, Canada and Asia.
XR is a umbrella term emerging in the mid 2000s that describes computer-generated environments accessed and
experienced through worn headsets and body interfaces. These environments are either simulated (virtual
reality-VR) or overlay and mix real scenes with digital 3D images and sounds (augmented reality-AR). As part of
its dramaturgical strategy, Animate harnesses a recent XR technology to explore this potential mixing
of real and the simulated in worn AR: live video “passthrough” which takes a real time video feed
from the tiny cameras attached to a head mounted display (HMD) and which allows the embedding of computer
generated 3D objects into the video feed of the real environment.
This paper details three aspects of the project, set against theoretical frameworks from STS (Science and
Technology Studies) and the sociology of innovation and expectations focused on how the future is embedded in
both technological discourses and materials. In Part 1, the most extensive section, we discuss the initial
concept and dramaturgy of the work. Although the technologies utilized in the project and the resulting
experience for the audience can be claimed as new, the concepts that enable these possibilities date back to the
1960s with the development of early Virtual and Augmented Reality technologies.
Second, while numerous other media were key to the artistic process, space allows us only to describe the
visual aesthetic concept behind the work. We focus on how our approach both differs from many standard VR-based
projects while also articulating the technological background that made such artistic choices possible. In Part
3, we analyze the lead up to the final presentation of the project in a major theater arts festival in Germany.
Here, we pay particular attention to an area which is rarely focused on in accounts of technologically-driven
artistic practice – the coupling between the profound instability of emerging technological
infrastructures and the manner in which such infrastructures ultimately affect artistic decision making.
What theoretically grounds the paper is recent work in STS (Science and Technology Studies) on issues of
“boundary work,” “technological futures” and “technological promises”
together with emerging research concerning the rapidly transforming understanding of presence through XR
technologies such as video passthrough. These technologies challenge long assumed ideas of presence in VR based
on the “illusion” of place ⁶ and instead, suggest new scenarios where users become located
“between the digitality of VR and the concrete reality of a user’s surroundings.”¹⁸
This theoretical work is critical in grappling with new artistic possibilities of XR technologies in addressing
larger aesthetic-social-technical issues the project is focused on; namely, the symbiotic
“interrelationships between media technologies, environment and body.”²
I. Background and Concept
Conceived in 2019 by one of the authors of this paper, the original impetus of the Animate project was
to explore the thematic of climate change through the sensorial-aesthetic possibilities of “spatial
computing.” A term that emerged in the mid 2000s and encompassing a range of computing and HCI research,
spatial computing can be defined as “human interaction with a machine in which the machine retains and
manipulates referents to real objects and spaces” ¹³. Although spatial computing’s
genealogical roots lie in environmentally-based computing paradigms like “ubiquitous computing”
²³ and “enactive” or “embodied interaction” ⁹ from the mid 1990s, the
principal application area for the concept arises in the domain of Augmented Reality in which digital
information is overlayed and integrated onto a real environment.²
Primarily accessed on mobile devices (smartphones and tablets) until the mid 2000s, more recently large amounts
of R&D have been poured into the design of “wearable AR” headsets from large tech companies like
Microsoft, Google, Facebook (renamed as Meta) and importantly, Magic Leap. Magic Leap is a much-hyped US-based
start-up who developed a wearable pair of AR-based glasses with extensive environmental sensing capabilities for
the consumer market throughout a long stealth period; a device which spectacularly failed to catch on shortly
after its 2018 release. The commonality among all of these AR technologies is two-fold.
On the one hand, they all seek to enable a computing paradigm that has long been theoretically discussed and
experimented upon (but with few widespread applications) called “Mixed Reality” (MR), in which the
computer-generated world and the physical world meld, blur and blend. MR is situated on what researchers have
called the “reality-virtuality continuum”¹⁵: a well cited paper in computer science that
describes a taxonomy between completely simulated (or virtual) experiences and real ones.
On the other, wearable AR presents a social-perceptual challenge. While VR uses a headset to completely
surround the wearer with a closed off world of computer-generated images and sounds based on sensing head and
(mainly) hand movement, wearable AR demands a negotiation between what Azuma names as “virtual content
that is integrated with the surrounding real world, while users remain engaged with and aware of that
‘real world’” ⁴. It is this conceptual idea that forms the basis of the entire Animate
project.
Spatial Computing and CliFi
The comment above from Azuma acts as a strong framework for the choice to explore a critical issue, that of
climate collapse, through the technological-phenomenological context of wearable AR/XR. After surveying the
state of the art of wearable AR technologies in 2019-2020 and spending part of the pandemic experimenting with
the see-through Magic Leap system, the artistic team gathered for the project reached two conclusions. The first
was that the Magic Leap or similar were too immature and costly for use in a professional live performance
artistic context due to multiple factors: comfort, low resolution, limited battery life and most important,
extremely limited field of view (FOV)—a term used to describe how large an augmented reality image is when
viewed through a headset.
Given the state of art, the second and more important consideration was that a purely technological exploration
of XR in typical “demo mode” would not be artistically adequate. These conclusions thus lead (after
almost 6 months of reading different texts) to the concept of staging a narrative Climate Fiction (CliFi) short
story called “Animate” by a Canadian author; a story which was discovered in a Canadian
“CliFi” (Climate Fiction) anthology.
The plot of the story “Animate” focuses on two characters, Daniel and Laurie, who are fleeing a
near future of climate disasters. Emotionally distraught from their hidden pasts, the pair set out on a journey
through the wild landscape of Newfoundland, Canada. While news of global climate catastrophes is broadcast on
their car radio, they drive towards the Tablelands, a real Mars-like landscape in the middle of Gros Morne
National Park where one can directly walk on the mantle of the Earth. Gradually, the climate-change transformed
landscape exerts a force on them as if it was a conscious, living and breathing entity. It emits strange sounds
and affects Daniel and Laurie’s personal relationship. But it is only in the dramatic conclusion, where
the earth comes alive in an apocalyptic scene of rocks rising from the earth and attacking the characters, where
the intricate interconnectedness of humans and a natural world under the throes of radical environmental change
is symbolically and viscerally manifested.
The basic dramaturgical strategy that evolved in the early phases of the project (during a period of
conceptualizing while fund-raising) involved a collaborating with the author to adapt the short story and turn
it into four dramatic scenes (Driving to the Tablelands, Camping Overnight, Hiking to the Tablelands and the
Earth Strikes Back) like a theatre text, then record actors speaking the text for a radio play and finally use
the radio play as a complex sonic accompaniment with additional visual elements in XR for a live theatre-like
installation with actors.
Walkthrough and Passthrough
It was clear from the start of the Animate project that due to the potential production and
presentation contexts, the project would need to perform a kind of “boundary work” between the genre
of theatre (involving human performers) and the technological environment of an audience wearing HMDs in order
to enable an interplay between real physical space and the imaginary world of computer augmented space.
A term coined by the sociologist Thomas Gieryn, boundary work describes work in which “boundaries,
demarcations, or other divisions between fields of knowledge are created, advocated, attacked, or
reinforced” 11. Most often such demarcations are described in more “high stakes”
ideological contexts as the natural sciences. Yet, one should not underestimate the different communities of
practice that operate even in assumed “fluid” artistic contexts. Indeed, the insistence on one of
the co-producers of the project (themselves anchored in theater performance contexts in a European cultural
festival context) that we should include human performers to appeal to a more traditional theater audience who
might be suspicious of the potential “media barrage” brought on by utilizing AR technologies clearly
demonstrates the occurrence of such boundary work .
The external demand of adapting to certain kinds of theatrical norms in both organizational and artistic ways
as well as the aesthetic choice to emphasize audio over computer-generated image (due to technical limitations
in AR headsets described above), posed a dual conceptual and artistic challenge in terms of integrating live
performers into the larger XR context. This context thus drove a fundamental artistic choice from the start of
the project: the idea that the audience would move from one physical space to another while they listened to the
spatialized radio-play based on the short story “Animate” over their HMDs.
This radio play accompanies installation with performers idea isn’t entire original but inspired by the
work of a 1980s San Francisco Bay Area theater-technology company who grappled with the similar situation of
trying to integrate a new technology (at the time) into live performance: the Sony Walkman. Known more for their
invention of a wireless museum guide/tour system that later became the largest audio tour company worldwide, in
the 1980s, Antenna Theater, directed by the American theater maker Chris Hardman, pioneered a new form of
walk-through performance-installation which Hardman labeled
“Walkmanology.” “I’ve always found airplanes to be claustrophobic, there is
always a kid crying in the back and you forgot the book you intended to read so you realized that you’re
stuck there for hours with nothing productive to do. Then this new fangled bizarre device came out on the market
called a Walkman and I decided to get one. I turned on the Walkman and just as the airplane lifted into the sky,
Wagner’s “The Ride of the Valkyries” began to play. Suddenly I realized that there was this
amazing theatrical event happening and it was called Syncronicity! The visual and the audio were working in sync
and furthermore instead of watching from afar, I was literally inside the event” ¹.
In many ways, the choice in Animate to use the emerging technology of AR follows a similar path
dependent story as Antenna’s work with the Walkman: treating the technology of worn AR as a device that
accompanies a larger dramaturgical experience rather than creating theatre work inside VR or
AR 17. Yet, what is key to the project is the emergence of a technology during the early phases
of production that was unknown at the time of the planning of Animate: the 2021 introduction of a
new generation of wireless VR headset from Oculus/Facebook/Meta which utilizes a video technology called
“passthrough.”
Originally designed as a safety feature to alert VR users when they would potentially go beyond a demarcated
spatial area called “the guardian,” passthrough uses the tiny black and white infrared cameras built
into the Oculus Quest 2 HMD which enables the device to sense where it is in space. These cameras deliver and
process live video images of the outside world inside the headset in place of our eyes, approximating what one
would see if directly looking into the real surrounding world. Importantly, what passthrough technology does is
transform VR devices (in this case, the Quest 2), which traditionally immersed the user in a closed off,
socially isolated world, into a new kind of wearable augmented reality where digital objects can be
embedded/overlayed onto the real world via the live video image and one sees/senses the physical presence of
oneself and others.
The introduction of passthrough technology as a kind of stop gap immersive AR not only represents a technical
change – it also suggests an epistemological and phenomenological one as well; a transformation
that is both historically grounded while, at the same time, shifting concepts of presence that have been long
established from research into human interaction and experience in VR. Indeed, while the passthrough technology
was already included in the first release of the Oculus Quest Pro in 2021, the ability to manipulate the
passthrough image by gaining access to it through the device’s SDK (software development kit) was only
available after production on Animate started in December 2021.
Technological Futures courtesy of the Metaverse
Yet, a larger question around passthrough arises. Why did Meta move to allow users to access and manipulate the
live passthrough camera image, thus turning the Oculus Quest 2 into a “immersive” worn AR / MR
device? The answer to this question suggests a larger narrative at play rather than simply creating a technology
to ward off potential accidents with users who would stray too far from a safe area while playing games in their
living rooms. Indeed, it wouldn’t be a stretch to claim that the passthrough-based AR capabilities of the
Quest 2 are tied into a larger socio-technical imaginaries of Meta’s “technological futures”
and “promises” for the software/hardware integration of platforms to enable the so-called
“Metaverse.” That is, to enable people to begin to occupy both physical and computer-generated
spaces that the Metaverse seems to promise.
Recent work in the sociology of expectations and STS has focused on the concept that certain imaginaries of the
future are performative in that “expectations, visions, scenarios, and other forms of anticipation affect
what may actually happen” ¹⁴. This idea of technical “visions” describes certain
normative conditions – of what is desired by a particular constituency rather than what is plausible or
needed from a particular technology, setup or infrastructure. Normally, scholars study such visions
under the context of discourse, including statements concerning “practices that systematically
form the objects of which they speak” ¹⁰. Yet, the transformation of a particular material
technology such as a set of cameras originally used for sensing the spatial orientation of the Oculus Quest 2 by
way of software additions, also results in a new perceptual paradigm (this is the case of video passthrough).
This suggests a literal materialization of such practices of discourse that Foucault speaks about through a
purposeful management of “future expectations.”
As STS researchers are keen to point out, the production of expectations around emerging
“breakthrough” technologies “which promise a vast potential of market prospects and solving
societal problems and create a sense of urgency in the context of international competition,” is instead
conceptualized as a “regime of economics of technoscientific promises,” rather than a
social-technical imaginary in which alternative futures that are equitable and desirable for larger collectives
are produced.¹⁴
Such clearly seems to be the case of the “Metaverse,” which is announced as future technology to
“change human interaction as we know it.” There thus seems to be no better way to manage future
expectations of an unproven technology platform like the Metaverse than to construct a taste of what such a
“proximate future,” a future that is always on the horizon but never comes,⁵ might be in the
present.
Shifts of Presence
The introduction and manipulation of such passthrough technology not only changed our technological approach in
Animate, allowing us to move from Magic Leap-based Augmented Reality to that of passthrough delivered
video image. The possibilities of passthrough also changed the aesthetic-perceptual framework as it quickly
became clear that visitors would have to wear and adapt to the Oculus Quest 2’s bulky form factor and
closed off environment throughout the performance.
But the use of these worn AR technologies within Animate raises also important phenomenological
questions as well, namely, how the sense of presence of oneself and others is actually reconfigured through
technologies that allow full body experience. It must be stated that the idea of one being able to confront a
mix between the computer-generated world and the real one (albeit one brought by cameras as in passthrough) is
an idea that dates back to computer graphics researcher Ivan Sutherland at the University of Utah in 1968.
Sutherland is credited as one of the first to develop the technology (if not the concept) for the head mounted
display – what he famously termed “the ultimate display” in a 1965 visioning article. Yet,
Sutherland’s description of a “display connected to a digital computer” [that] gives us a
chance to gain familiarity with concepts not realizable in the physical world” ²¹ which sounds
like the basis for current virtual reality, is actually enlarged in a follow up technical paper in 1968 which
describes the actual construction and workings of a 3D head mounted display. Here, Sutherland gives already a
sense of the Quest’s (and other) push towards integrating passthrough technologies into the real
environment. “Half-silvered mirrors in the prisms through which the user looks allow him to see both the
images from the cathode ray tubes and objects in the room simultaneously. Thus, displayed material can be made
either to hang disembodied in space or to coincide with maps, desk tops, walls, or the keys of a
typewriter” ²².
At the same time, however, what is more interesting about the introduction of passthrough technology is that it
also enables a paradigmatic shift in another area of knowledge and consequently, experience (which is why we
term its effect both epistemological and phenomenological). Passthrough changes the understanding of one of the
core foundations of VR-based research which is that of presence.
In fact, the introduction of consumer grade AR that mixes the real and simulated together shift not only the
user’s experience of their own presence but also “the conditions in which human individuals interact
with one another face to face from body to body” ²⁵ ” what Goffman ¹² famously
called co-presence. As key XR researchers admit, while these technologies have existed for decades,
“little is known about how social interactions are affected by the technology” ¹⁶ and how
social interaction in turn shapes these systems. Part of this gap stems from the overarching focus in VR/AR
research on an individual experience of “presence” as “outwardly ‘dislocated’ from
its physical setting.”¹⁸, ⁷
It should also be clear that the move to worn AR prototyped by Sutherland in the 1960s and commercially
available in 2021 is a marked contrast to earlier conceptions of presence in VR understood as the “virtual
experience of being in an environment, even when one is physically situated in another.”
⁶, ²⁴ While Animate’s focus was not specifically on researching how
such co-present interaction is reimagined (and vice versa) through camera-based technologies like passthrough,
the experiential effects of being embedded in the real environment while also experiencing Sutherland’s
vision of a “mathematically generated wonderland” clearly constituted a necessary experiential
context for the production.
II.Aesthetic Considerations
We have so far described some of the historical, artistic and sociological background of Animate. But
the larger question looms of how all of these ideas actually play themselves out in the trenches of practice?
This section Thus, focuses briefly on two of the principal image-based media that forms the basis of Animate
during the approximately seven-month production period, from late November 2020 up until production in
Germany in August 2021.
Alternative 3D Images in VR Land
The visual aspect of VR is often intertwined with historical theorization around cinema and as an extension of
early immersive technologies such as panorama screens. However, with new XR technology the language borrowed
from cinema [camera, frame, mise-en-scène, cut, montage, projection, screen] can only take one so far.
This situation prompts a complete reconfiguration of artistic and dramaturgical decision-making when sequencing
elements of an XR work. Virtual objects are no longer elements to be ‘looked at’, but instead have
the potential to be investigated from all angles and vantage points. The salient attribute of these technologies
is their ability to render virtual spaces such that they perceptually feel ‘real’. Although the
illusionistic 3D space projected onto a flat surface remains technically ‘flat’ in XR, perceptually
the virtual rendering of objects and the physical space in which those objects are situated becomes integrated.
As such, in the context of XR, we could no longer rely on the accustomed conventions held by the cinematic
language (perhaps suitable for 3D of VR work), as one must edit and configure images not only over time, but
also over physical space, while also being aware of how bodies might move through that time and space.
In the second act of Animate, the switch to passthrough AR immediately following a rope guided VR
‘journey’ provided a direct perceptual contrast between the experiential qualities of both XR
modalities. During this dizzying rope guided journey, the participants must meander through a foggy
Tarkovsky-esque virtual trail with distant hills and trees, listening to spatialized audio while losing their
orientation in the physical space. Here we planned to take a restrained approach to the image, rather than
adhering to the cultural expectation of visual effects overload, such as the case in many VR works. This scene
transitions into an opening of the passthrough image.
One realizes the sealed off, socially isolated quality of VR when participants can suddenly see their bodies
(rather than virtual avatars as is the case with many other XR works), other participants, the real environment,
and the performers. As such, developing a dramaturgy that unfolds to accommodate both VR and AR
technology—and not either/or—reveals the perceptual nuances of these technologies as held in such
striking temporal and dramaturgical contrast.
As previously stated, in the context of Animate, the passthrough feature was deployed outside its
intended practical use case of creating a guardian safety boundary for a VR experience. Due to its practical
functionality, the passthrough image produces a low resolution, grainy, black and white image. This image as
incorporated into Animate suggested an early film aesthetic, as well as the contrasted black and white
images of films from Antonioni or Tarkovsky. Indeed, at points in the work we lean into this aesthetic quality,
adding simulated grain over virtual objects, so they feel visually and perceptually integrated into the
passthrough image. In other instances, such as the initial reveal of the Tablelands environment, a highly
saturated virtual space is presented—portraying a vivid full colour virtuality at odds with a more subdued
black and white passthrough reality. Meta’s recent release of the Quest Pro headset comes with
high-definition full colour passthrough. We are currently experimenting with the technical and aesthetic
affordances of this new headset for the upcoming performances of Animate in 2023. This speaks to the
technical and artistic adaptability that is required when working with such emerging technologies.
As the passthrough image appears in the last scene, both virtual and real rocks are revealed to be scattered
around the physical space. As the narrative continues, these virtual rocks begin to move in a carefully
choreographed way– climaxing in an explosion and in a clustering and swarming pattern, as if they were
birds or insects. The sites in which the performance has taken place thus far were remodelled such that the
virtual elements spatially could overlay the physical architecture to real-world scale as seen through the
headset. These virtual models are hidden from the spectator’s vision, but remain in the software’s
memory to carry out physics calculations and render occlusion. Due to these virtual mapping techniques, the
rocks collide with the physical environment, crashing into walls, pillars, and the floor. Participants can
extend their hands and bodies and ‘touch’ these rocks, which in turn causes them to move as their
physics are simulated in real-time.
Interacting with Flying Rocks
As stated in Part 1, Animate ends with a spectacular scene in which audience members find themselves
in a field of rocks, some virtual and some real. As the scene progresses, the simulated rocks begin floating,
moving around and seemingly gain consciousness as they chase the actors and audience members. Any of these rocks
can be interacted with by pushing them around using the hand tracking featured by each headset. However, with
the aim to give these 3D objects more corporality, we focused on interaction between the rocks and the audience.
Early live tests revealed that allowing hands to directly collide with the rocks made it too easy to introduce
energy in the environment, as a simple slap could effortlessly send a massive rock flying to the other side of
the room. The adopted solution was instead to implement a force field around each hand that continuously pushes
away rocks that get too close, with more power the closer they get. This approach restricted how much energy,
and thus chaos, could be injected into the system, limiting the amount of control one has over the rocks and
guaranteeing an agency to the objects. As the scene progresses in intensity, the rocks gain more energy, which
makes it easier for them to ignore the attempts by the spectators to alter their course or shield themselves,
denying them the drive to disrupt the harmonious choreography.
Since the artistic intent was to virtually drown the participants in boulders and pebbles, just like the
characters in the story are swallowed by the earth, the number of virtual rocks totaled over 600 individual
objects. This made it impossible to perfectly synchronize the rock’s state across a dozen headsets over a
WiFi network and thus demanded an alternative procedure. A way to combine interactable rocks, locally solved
physical simulation, and a behavior recurrent enough for a scripted show is to implement a flexible system.
Since all headsets were to be spatio-tem- porally aligned, they all could share the exact same behavior, which
instead of controlling the rocks directly, positioned their targets. Throughout the scene, the floating objects
were assigned different moving targets that serve to steer them on the stage since the rocks would be
continuously pushed towards their target. As the targets’ behaviors are determined, the rocks could then
flexibly and independently react to a spectator’s actions, synchronizing the state of each object and
solving simultaneous interaction over the network thus becoming unnecessary.
Both measures undertaken ensured that even though the state of the system on a microscopic scale would differ
for any spectator and performance, it would also stay similar enough on a macroscopic scale to fulfill three
requirements: the audience members could feel as though they are sharing the same environment; the actors could
move and react around learned cues on stage; and the spatialized room-scale sound (which would eventually be
synched to a 49 channel audio system) would stay consistent in time and space with the current state of the
virtual environment.
III. Production and Transformation
In part 3 of this paper, we describe the final production period of Animate, a relatively short time
frame of 10 days that took place in the actual location where the work was to be premiered: one part of the
interior of a vast, almost 100,000 square meter former factory that manufactured agricultural machines for the
entire former East Germany.
Like many works involving the integration of new technological infrastructures, the technical production of
Animate was beset with challenges. More specifically, the ontological nature of theater as a distinct
time-based event which demands an integration of multiple elements poses a major challenge to the use of
technologies like Meta’s VR headsets that have been explicitly designed and engineered for a different
scale of use: small spaces like living rooms. Here, we want to focus on two specific issues that had major
artistic implications: (1) the difficulty of integrating live performers into passthrough-based AR; and (2) the
mismatch between an organizational workflow derived from theater versus one derived from VR-AR development and
production.
Live versus Virtual
From the start of the Animate process, the core artistic concept was to leverage the relations and
tension between live theater with performers and passthrough-based AR. This was to be conceptually accomplished
by the performers, who would be “live” actors essentially “acting out” in
“dumbshow” format (conveying a meaning or message without speech through miming) core scenes while
being accompanied by the overall spatialized musical/sonic/text/visual “score” accessed by audience
members via the worn HMDs.
This concept in practice, however, proved to be a significant challenge. First, due to the high cost of the
professional (and in this case, well-known) actors who were active in the German theater scene, the amount of
rehearsal time required to integrate these performers was extremely short (5 days). Second, the complexity of
mixing 3D graphics into the real scenes by way of passthrough was grossly underestimated, particularly since the
actual performers were only integrated during the technical production period and not during development in
Canada.
Since the performers could not see what was going on in two crucial scenes (Scene 2, “Camping” and
Scene 4, “Tablelands”) since they wore no HMDs themselves, it was increasingly difficult for the
actors to know where virtual objects in the space were located in relation to real objects. More pointedly, 3D
game engines as creation and rendering environments for XR production are WYSINWIG (What you see is NOT what you
get). In other words, what appears on the screen has little to do with how such 3D objects are perceived when
they are embedded into real environments via passthrough technology. This issue of not being able to see and
hear entire scenes in Animate without wearing an HMD reached an almost comic proportion when the
dramaturg/movement consultant who was supposed to take notes to aid the artistic director and the actors, was
unable to perform these duties since all of the HMDs were occupied for testing by the technical-design team.
A third and major difficulty arose in the extremely low quality of the black and white passthrough video feed;
an issue exacerbated by the harsh lighting conditions in the industrial hall where the performance was to take
place.
While the team tested passthrough technologies continually during the development process, the original concept
was that there would be no scenes with human performers without accompanying 3D objects and behavior of those
objects (like the complex rock scene) acting alongside the human performers. The pressures of production
deadlines combined with the extreme latency of framerates with the Quest 2 led to the reduction of all heavy 3D
graphics, thus transforming the overall visual aesthetics of the two crucial scenes with the actors. Indeed,
watching highly trained performers miming a complex dramatic text through a black and white and frequently
distorting video feed using an uncomfortable and awkward device proved to be too much.
The Vitality Gap
While the above issues demonstrate that the integration of live performers into passthrough XR is not trivial,
a much larger issue was present in the final production period; one which led to a radical reorganization of the
entire production twenty-four hours before the premiere for audiences and press. This core “show
stopper” involved the inability of the technical team to make needed artistic changes in real time
to virtual environments in Unity 3D, based on the direct real time experience of working with the actors.
Unlike the production process of VR applications like games or already produced experiences such as film, the
real time experience of making theater with performers and machines is completely different, akin more to
improvisation than to the filmmaking or gaming production workflows that seem to be default for XR development.
Live performance practice demands a sophisticated attention to timing and rhythm, as well as highly tuned
dynamics in order to ensure that the flow of action reaches what the psychologist Daniel Stern called
“vitality affects” – in which “physical action and traceable mental operations" become
"inherent in the act of [both physical and mental] movement” ²⁰. Vitality affects, which
involve the dynamic flow of time through the expression of rhythms, pulses, changes of tempo and direction are
experienced in the entanglement between human bodies and nonhuman objects and processes like light, sound and
vibration outside of us. Such “forms of vitality” sweep us up into dynamic moments of intensity.
Unfortunately, such forms of vitality were extremely difficult to construct between the human performers and
the workflow of XR production. In contrast to real time audio-video processing environments, where developers
and creators can make changes essentially “on the fly,” based on changing dynamics of performers and
light, sound or image in the ever shifting context of the performance environment, the sheer amount of time
required by the workflow of making changes in Unity 3D and then uploading (“building”) those changes
to the hardware-based Oculus Quest 2 (bordering on hours) made quick changes extremely time consuming and
inefficient. This inability to rapidly change or eliminate elements or even scenes that did not work or whose
timing was aesthetically problematic in the heat of rehearsal proved to be major hindrance to creating a dynamic
live event.
The Nature of (Dramatic) Change
It goes without saying that that Animate’s live performance context presented both a formidable
challenge as well as opportunity for thinking of the further development of XR as an aesthetic-social-technical
domain of knowledge and experience. Indeed, the challenges that the performance context presented were indeed so
formidable to the ultimate successful artistic realization of the work that Salter undertook radical changes one
day before the public premiere: completely eliminating two of the core scenes (Driving and Camping/Dream) of the
production that the team had worked months on and replacing these with a much more simplified concept involving
the actors dramatically reading the original short story to the audience for the opening scene.
After this initial scene, in which the actors sit underneath the camping scenography originally designed for
the integrated production between performers and XR, the audience is brought by the actors into a second space:
the enormous main part of the industrial hall. It is here that the media experience promised by Animate
begins. After the ubiquitous onboarding sequence, where the audience receives Oculus Quests 2 and specially
designed headphones that allow audio from outside to also be audible inside the headphones, the actors
collectively lead the small audience who are connected together with a rope through the space while the group
experiences the VR part of the performance: in the dramaturgy of the work, the experience of hiking to the top
of the Tablelands. Lasting approximately seven minutes, the scene ends with the audience briefly inside a
360-film shot in the actual tablelands in Newfoundland; a landscape which gradually fades, revealing the actual
physical environment of the industrial hall via video passthrough.
Finally, in the dramatic last scene, the audience is plunged into a full XR experience – essentially, the
original vision of Animate as a work focused on the affective and bodily impact of climate
transformation. A field of detailed, 3D modelled rocks lie on the floor courtesy of the passthrough-VR mix. As
the scene progresses and the landscape becomes alive, the rocks slowly lift off the floor, accompanied by
thundering, multi-channel audio in the physical room and in the headsets. The actors begin to dance as blinding
light from the other side of the enormous space bursts into the black and white video feed in the HMD,
essentially causing a momentary white out for the audience. As the actors begin a kind of dans macabre,
eventually disappearing from the scene, the assemblage of 3D rocks gathers speed, first assembling in a rotating
ball and then exploding across the physical-virtual space. The performance concludes with the long planned
Endzeitsturm (apocalyptic storm) of rocks, that spin in a gravity-defying machine enabled choreography
and concludes by rushing through the industrial hall in a massive snake-like form, only to fall onto the
transfixed (and sometimes escaping) audience members.
The radical artistic choice to edit, reduce or eliminate painstakingly produced artistic material is not
particularly unusual but rather a standard artistic survival strategy. In the heat of practice, there is a clear
sense of what works and a clear sense of what doesn’t. In Animate in particular, however, the
massive restructuring of the performance 24 hours before the premiere not only enabled the performance to
achieve the kinds of vitality affects necessary for its success: it also clarified added certain conceptional
and dramaturgical structures that had been buried in the initial concept but not well articulated, namely, the
historical story of how different forms of media have and will enable in the future different modes of
storytelling. As Animate moves from live theatre to VR (itself an older technology) to the futuristic
promises and possibilities of worn AR through the almost anachronistic quality of black and white video
passthrough, the audience experiences a double move from one form of historical media to another and from
passive spectator to active participant.
Conclusion
We have described the production process involved in a new attempt to integrate unproven XR technologies into
the complex dynamics of real time performance, framed by recent work in STS, the sociology of innovation and
performance theory on the social-technical imaginaries and futures that technology harnesses and the real-world
challenges and practices that put these futures into a kind of situated, concrete knowledge and experience.
What we have not have time to describe in this paper is the audience reaction to the project as little
ethnographic interview work was formally done given the already complex demands on the artistic and technical
team. But perhaps a long citation from a review of the production in one of Germany’s major newspapers,
the Frankfurter Allgemeine Zeitung, gives one sense of the affective promise envisioned in the complex
entanglement between technology and environment, past and future, imagination and reality. “This year, the
Kunstfest Weimar is focusing on the utopias of the future and has included pieces that talk about climate
change, political polarization and new social systems. Animate" is the boldest one. In this immersive
production, the latest generation of virtual reality glasses is used to tell a dystopian love story that
blossoms and then shatters in the midst of the climate crisis. The text comes from Canadian author Kate Story.
Laurie and Daniel, the main protagonists, are newly divorced and come together in their loneliness. Both are
traumatized by past relationships and experiences. Their homeland has been destroyed by a climatic catastrophe.
While the sound plays, the audience of only eight is led through the hall on a rope in a one-hour time slot.
Actors ecstatically tell the story and perform in harmony with digital reality. While she balances on the stones
and always close to the abyss, he falls into a depressive trance and despairs more and more. "I'm finished,
we're finished," he says in a sobbing voice. The industrial hall with all its natural obstacles provides a
surreal backdrop. In the VR part of the performance, the images blur, the chirping of birds turns into a booming
bass sound, and boulders seem to be moving around. At that moment, the performance begins to fulfill its
immersive promise. Now the rope is put down. The audience moves freely in the space, following and dodging the
whirling rocks – which are both there in the glasses and not there in the real world. With their hands
they try to fend off the debris, which really works thanks to the latest VR technology. Salter’s piece,
realized with technology from Meta, gives a foretaste of theater ten years from now.”
References
1 Antenna Theater, https://www.antenna-theater.org/. Accessed on November 25, 2022.
2 Marie-Luise Angerer, Ecology of Affect: Intensive Milieus and Contingent
Encounters. Luneburg: meson press, 2016, 16.
3 Azuma, Ronald T. “A Survey of Augmented Reality.” Presence: Teleoperators and
Virtual Environments 6, no. 4, August 1, 1997, 355–85.
4 — (2016) “The Most Important Challenge Facing Augmented Reality.” Presence
25, no. 3, December 2016, 234–38.
5 Genevieve Bell, Paul Dourish, “Yesterday’s Tomorrows: Notes on Ubiquitous
Computing’s Dominant Visions”, Personal and Ubiquitous Computing, 11. Cham: Springer,
2006, 133-143.
6 Frank Biocca et al., “Toward a More Robust Theory and Measure of Social Presence: Review
and Suggested Criteria”, Presence: Teleoperators and Virtual Environments 12, no. 5,
October 1, 2003, 456–80.
7 Gordon Calleja, In-Game: From Immersion to Incorporation, Cambridge, MA, MIT Press
2011.
8 Michael Century, Northern Sparks: Innovation, Technology Policy, and the Arts in Canada from
Expo 67 to the Internet Age, Cambridge, MA, MIT Press, 2022.
9 Paul Dourish, Where the Action Is: The Foundations of Embodied Interaction,
Cambridge, MA, MIT Press, 2001.
10 Michel Foucault, The Archaeology of Knowledge, New York, Routledge, 1972.
11 Thomas Gieryn, "Boundary-work and the demarcation of science from non-science: Strains and
interests in professional ideologies of scientists", American Sociological Review 48 (6), 1983,
781–795.
12 Erving Goffman, The Presentation of Self in Everyday Life, New York, Knopf Doubleday
Publishing Group, 1959.
13 Simon Greenwold, "Spatial computing," Massachusetts Institute of Technology, Unpublished
Masters Thesis, 2003.
14 Karen Konrad et al, "Performing and Governing the Future in Science and Technology", In The
Handbook of Science and Technology Studies, 4th Edition, eds. Ulrike Felt, et al. (Eds.) Cambridge, MA,
MIT Press, 2017, 465.
15 Milgram, Paul and Fumio Kishino, "A taxonomy of mixed reality visual displays", IEICE
TRANSACTIONS on Information and Systems 77, no. 12, 1994, 1321-1329.
16 Mark Roman Miller, et al., "Social interaction in augmented reality", PloS one
14, no. 5, 2019.
17 Ken Perlin, Future Reality Lab, https://frl.nyu.edu/author/ken/, Accessed on December 4, 2022.
18 Michael Saker, J. Frith, “Coextensive space: virtual reality and the developing
relationship between the body, the digital and physical space", Media, Culture & Society 42, no.
7-8, 2020, 1427-1442.
19 Mel. Slater, “Place illusion and plausibility can lead to realistic behaviour in
immersive virtual environments", Philosophical Transactions of the Royal Society B: Biological
Sciences 364, no. 1535, 2009, 3549-3557.
20 Daniel N. Stern, Forms of vitality: Exploring dynamic experience in psychology, the
arts, psychotherapy, and development, Oxford, Oxford University Press, 2010, p.9.
21 Ivan E. Sutherland, “The Ultimate Display”, Proceedings of IFIP Congress,
1965, 506-508.
22 — “A head-mounted three-dimensional display”, Proceedings of the December
9-11, 1968 Fall Joint Computer Conference, Part I. ACM, 1968, 757-764.
23 Mark D. Weiser, "Ubiquitous computing", In ACM Conference on Computer Science, vol.
418, no. 10.1145, 1994 ,197530-197680.
24 Bob G. Witmer, and Michael. J. Singer, "Measuring presence in virtual environments: A presence
questionnaire", Presence 7, no. 3, 1998, 225-240.
25 Zhao Shanyang, “Toward a Taxonomy of Copresence”, Presence 12, no. 5,
October 2003, 445–55.
Towards Sensemaking in the Meshwork of Technology, Ecology and Society: Symbiosis of Aesthetics, Performance
and Digitalization
ClaudiaSchnugg,
DanielaBrill, ChristianStary
Johannes Kepler University, Linz, Austria Claudia.Schnugg@jku.at, Daniela.Brill@jku.at,
Christian.Stary@jku.at
Abstract
To act, humans first need to make sense of the world. Thereby, sensemaking goes beyond accumulation of pure
information of objects or rational knowledge production, but it encompasses additional information such as
meaning, mindful engagement, socially embedded knowledge, cultural and work contexts. To navigate in diverse
environments, sensemaking becomes central to social settings, also to engage with technologies and understand
dynamics in ecological environments. In a complex world where technologies are added components of everyday life
and are envisioned as partial means to approach global challenges, social, technological, and ecological
environments become intertwined. This meshwork of environments also means to bring together different kinds of
knowledge as a base of sensemaking through experience. In the Digital Sensemaking project we specifically look
at digitization processes, the interaction with IoT Elements and Digital Twins through the lens of performance
art to elaborate on the non- cognitive core constituents of sensemaking processes: embodiment, action-sense
nexus, and temporality. We show that aesthetics can be found as an important dimension to bridge the cognitive
and non-cognitive process and explore the role of art in this kind of research. It facilitates process and
technological development in organizations entangling the social, technological, and ecological.
Keywords
Sensemaking, Digital Sensemaking, Embodiment, Digitalization, Aesthetics, IoT Elements, Digital Twins,
Performance, Embodied Experience, Symbiotic Organizations.
“Now this makes sense” is an expression that is regularly used by many people. What does it mean
though, and why is collection of information alone often not enough for something to make sense? The more
complex the environment becomes, the more difficult it is to make sense of situations. For example, concerning
the climate crisis many artworks aim to establish a connection for the audience between scientific insights on a
global scale, or
global climate dynamics, with specific manifestations of the same issue on a more local or individual scale.
The same is true for other artistic projects that deal with global challenges, such as helping individuals
living in Western countries with a mindset informed by the needs of their lives in Western cities to better
understand the impact of their behavior. ¹ More specifically, to create awareness how behavior and
decisions of citizens in urban environments can affect rural environments in other countries, or how mindsets
framed by social and cultural ideas in one country can lead to exploitation of ecological systems that the very
same population has an interest in preserving. In these endeavors, artists engage in the creation of experiences
to help the audience make sense of abstract information.
On the other hand, environments of social interaction, cultural production, and work processes become
increasingly enriched with technology, connected to digital systems, or even completely digitized. Engaging with
digital processes, socio-technical or socio-hybrid systems can lead to a new version of the disconnect presented
above: how much are workers in touch with what they are producing? For example, rhythms of a production factory,
materialities of components, noises of the machine, and embodied knowledge such as feeling the right pressure
might be important for production workers to understand if everything is going well in the process.
When they are suddenly confronted with a digitized version of their work, depicted in software on a computer
and with robots taking over much of their physical work, how do they still make sense of the process that is
going on? ² Which clues do they actually need? Or how can individuals purely connected through
technological systems still make sense of what they do together?
In the era of global challenges and digital transformation of central processes in work, government, and
society, organizations and individuals are challenged to make sense in such complex situations that are
constituted by merging information, abstractions, and experiences (or the lack thereof). Starting from
sensemaking theory based in social sciences which has been discussed within organization science for about 40
years now with a strong focus on social context and cognition, we aim to explore sensemaking in a potential
symbiosis of social, technological, and ecological environments. Scholars already pointed out how important it
is to include embodiment, sensemaking with non-human and more-than-human actors or environments, and to go
beyond a focus on logos or exploration of a mind-body dualism in understanding sensemaking; to develop a
decentered notion of sensemaking “—not simply at the disposal of human subjects—and where
sense is always and already given and made simultaneously”, but as a process beyond logocentrism that
unfolds “in the meshwork of life”. ³ This means to include a focus on immanent and embodied
aspects, including temporality, leaving anthropocentrism by including more specifically actors and aspects that
are part of the sensemaking process such as materialities, technological elements, or ecology. According to Karl
E. Weick, sensemaking is about patterns that enrich and develop organizational structures, about the attribution
of meaning to processes and situations.
In the Digital Sensemaking project that we present in this paper, we answer this call of scholars in the field
of sensemaking theory through experimentation in diverse settings that connect to non-human and more-than-human
components in a controlled way. Therefore, creative new methodological approaches are needed. ⁴ Our
analysis of this experimentation is informed by methods from qualitative research in social sciences, visual
studies, and methods in performance studies. We are especially interested in deepening the understanding of
embodied sensemaking and the role of embodiment in sensemaking, aesthetic experience, and the role of aesthetics
in sensemaking. Therefore, we work with performance artists in the setting of artistic research and performance
art.
The research is based in the academic Institute for Business Informatics—Communications Engineering that
focuses on digitizing work processes, cyber-physical systems, and digital twins. Thus, central to the artistic
research and artistic performance development of the artists involved is the engagement with these systems in
the context of sense-making processes. With this interdisciplinary project we connect arts-based work and the
creation of aesthetic experience through artistic practice (performance) into exploration of technology with
contributions to theory building within the social sciences and contributions to technology development.
The paper is organized as follows. We will start by introducing sensemaking theory and the theoretical context
of why and what we aim to explore with the Digital Sensemaking project. We then account for our methodology to
work with performance artists to investigate embodiment and aesthetics in sensemaking with digital technologies,
cyber-physical systems, and digital twins. We will present our insights on embodiment and aesthetics in
sensemaking in the above-mentioned “meshwork of life” and explore how including art in the research
process can open avenues for research and symbiotic organizations.
Sensemaking
In everyday experience, sensemaking is referred to as a process in which experiences, information and
situations suddenly fit together parts fall into place, relations between bits of information in a certain
environment, or as sense-making scholars write “[sensemaking] is the primary site where meanings
materialize that inform and constrain identity and action”. ⁵ Since the 1980s the theory of
sensemaking has been explored extensively in the field of organization studies. Organization studies is
interested in understanding human behavior of individuals and groups, organizing processes, work, hierarchies,
but also in questions of shared meaning and communication in groups. With its entanglement in sociology,
psychology, anthropology, ethnography, work, and management studies, it has been a fruitful field to expand the
original concept of sensemaking. Sensemaking is considered as an ongoing process of clarification, as enactment
of sensible environments, as social in terms of relation to the knowledge and sensemaking of others as well as a
reflection on one's own identity. Moreover, it originally has been characterized as informed by cues that the
individual becomes aware of, an urge to understand what is done as plausible, and informed by retrospective
reflection on situations.⁶
Since the inception of sensemaking theory, scholars started to expand on missing aspects of the original
definition. Most importantly, the strong focus on cognition, logocentrism, or even rationalistic approach to
sensemaking has been criticized widely, similar to different ways of understanding temporality and the role of
the human in sensemaking, and its lack of situated and embodied dimension in order to give way for research
covering these aspects and more differentiated approaches in sensemaking moving away from a pure anthropocentric
and language orientation – as previously language and cognition have been first entry points to research.
⁷, ⁸ , ⁹, ¹⁰, ¹¹ For example, ethnographic methods and video analysis
have been employed to investigate the various role of the body and embodiment in sensemaking processes, such as
sense-making “with the body” and “from the body”; in other words: embodiment in the
process of sensemaking, and the role of established embodiment for sensemaking. ¹²
Other endeavors expanded on the influence of the material world on the sensemaking process. Going beyond the
communicative support of boundary objects, the role of material practices for individuals to transit to
group-level sensemaking. ¹³ Taking this further, researchers in human-computer-interaction
started to investigate technologies to support sensemaking processes. ¹⁴ A first step to connect the
idea of aesthetics and sensory knowledge to sensemaking has been identified in cases through linking embodied
and implicit knowledge in the handling of objects in work and knowledge production processes.
In a first step, individuals make sense of what they are working with, for example data in the field of physics
which is their specific field of education and work. At a later point in research, their implicit knowledge
about the look and feel of the data and their ingrained knowledge of physics creates difficulties in finding
fresh perspectives on the data. Aesthetic re-interpretation of the objects can help them through this process.
¹⁵ Thus, objects become part of sensemaking, and technological environments to which individuals
relate to daily or which mediate their work, are part of their sensemaking processes. This entanglement with the
material world in sensemaking processes and as important sensemaking dimensions for humans becomes visible
beyond the engagement with human-made objects or the use of objects and technologies to mediate sense-making
between individuals—or may restrict it in the case of ubiquitous use of technology.
Based on the experience with the non-human and the more-than-human in the environment of remote material
landscapes and the sensemaking of ecological processes by indigenous peoples, ethnographic methods have been
employed to investigate ecological embeddedness of sense-making and the increasing inability to make sense of
subtle ecological cues of Westerners who predominantly live in urban or industrial areas. This lack of extending
sensemaking to the environment in that sense leads to vulnerability, for example by misunderstanding dangers in
wildfire situations or regions prone to high waters or other harsh conditions that need to be recognized to
enable individuals to predict danger. ¹⁶ This still increasing detachment of macroscopic aspects in
the environment that play into sensemaking processes is supported by the simultaneously increasing focus on
sensemaking on microlevels, in personal “bubbles”, specific organizational settings, disciplinary
work, or even cultural perspectives.
Although the meshwork of life, i.e., social, technological, and ecological dimensions of the environment are
becoming more central in the understanding of sensemaking theory, there are still gaps to research, e.g., to
raise awareness of problems in the focus of sensemaking process on specific microlevels, to better understand a
new dimension that is introduced through certain technologies: digitalization, and to entangle aesthetics and
art with interdisciplinary sensemaking research. It is supposed to connect cues from the environment,
experience, sensory information, embodiment, and diversity of media relevant to sensemaking.
Towards Digital Sensemaking
Experiences in the Covid-19 pandemic accelerated the digital transformation as envisioned by protagonists of
Industry 4.0, that includes the implementation of Internet of Things (IoT), robotization,
cyber-physical-systems, but also propagated ideas within the transhumanist movement. At the same time, digital
transformation can also lead to work overflow as tasks arrive much faster at the responsible human actors, and
to information overflow as much more details and relevant factors need to be described because they cannot
anymore be experienced directly – but at the same time become more abstract –, and meaningful
experiences and decision-making processes become more difficult. ¹⁷, ¹⁸ On some scale many
people experienced such effects during the Covid-19 pandemic: being decoupled from on-site team meetings,
communication might have become difficult over time, or understanding the actual situation in a production hall
through digital twinning of information might have been difficult and can have led to delayed decisions, or even
to oversee important moments to avoid problems.
Digital transformation is often associated with changes in mental representations of knowledge of those who
engage in the digital transformation, in order to be able to adjust the behavior to new situations.
¹⁹ This idea is strongly rooted in social information studies and mirrored in processes of
communications engineering and computer science. One reason can be that digital transformation is understood as
being based in transforming information into digital representations and abstractions that seemingly follow
rational logics. Thereby, logocentrism, anthropocentrism, and a focus on disembodied abstractions that are
represented through words, numbers, or most importantly also visualizations surface in this field and such
settings. But considering the often experienced disconnect between digitized work process and internalized
hands-on work in production companies, the resulting information overflow, and difficulties in decision-making,
it is important to ask the question how to create meaningful experience in digital transformation and
sensemaking in the interaction with digital technologies. ²⁰, ²¹
Weaving digital transformation (and its predominant focus on cognition and abstraction) with the concept of
sensemaking and the most recent scholarly contributions with an increasing focus on the body, embodiment, and
physical interaction with the environment, the question we explore with the concept of digital sensemaking is
where the body and the senses are in the case of digital transformation? Thereby, the concept of digital
sensemaking aims to integrate the body and the mind, cognition, and experience, as necessary for sensemaking
with digital technologies, in cyber-physical systems, and thus create opportunities to design meaningful
interaction.
This integrative perspective adds an important aspect to human sensemaking when connecting beyond technology or
the immediate micro-environment. Or the other way ‘round: enhancing sensemaking with digital technologies
which are embedded in the meshwork of life, for resilient organizations and human beings in scenarios where
technology, ecology, and the social need to be integrated for sensemaking.
The Role of the Body and Aesthetics
Ideas of what sensemaking is are not necessarily bound to knowledge about social scientific theories on
sensemaking. Thus, we started this project by a preliminary investigation of what individuals who are
experienced in working in both, in academia as well as in the arts, do understand as sensemaking. The
preliminary investigation consisted of four qualitative interviews with artists who work in academia and focus
on sensemaking in their work without focusing on the theory. The answers we received to the question of
“what is sensemaking for you?” are striking. They represent beautifully how body, mind and
experiencing the world are entangled in sensemaking. In the first interview, an artist with a bodily artistic
practice who is also embedded in natural science research, says:
“It’s a mess, making sense of the world, different biases, we spend so much time trying to make
sense of the world. I don’t try to make sense of it, through dance I just experience and feel, don’t
think just do. I like to learn more about the complexities, but the deeper I go, the less it makes sense.
Dancing helps to see different things in different ways. Everyone has both sides of the brain in their head,
both sides are needed for everything. Being analytical in the dance is really important. [...] I’m talking
about science and art separately. In dance you see a movement and intuitively move. In physics –
breakthroughs – you learn it so much that it becomes intuitive.”
The third artist we interviewed referred to this entanglement in a similar, but different way:
Sensemaking is “when your ideals and your perception of reality are the same. When something makes sense
is when you can truly believe it, now you understand it and it’s real and it’s the truth. You can
investigate to know what is the truth, making sense of reality is to incorporate your senses to prove something
as true. That’s when it makes sense. When things are confirmed, they become understood, and then you come
to your senses and perceive reality.”
Both excerpts from the interviews show how reality, perception of reality, being in the world, understanding
through body, and theoretical reflection and cognitive understanding need to come together to make sense. The
first artist also refers to an additional dimension, through the reflection on “biases” the
dimension of the social, of values, cultures are pointed out. On the one hand, there is cognition, abstracted
knowledge about the world and how it functions (or should function). On the other hand, there is the experience,
the way the world is perceived, situations that impose themselves on us. Thus, the body and the bodily senses
play a central role - abstract knowledge alone does not yet lead to sensemaking. Sensemaking is informed through
the situation, the experience, and subtle information of the social and cultural context. Put it in a different
way, experience and “being in the world” are key to sensemaking. There is no sensemaking without all
these rich additional layers of information.
Sensemaking scholars have worked in many directions to make sensemaking theory consistent, mainly by working
with cases to fill in gaps in the original theory. Lately, two theorists, Sandberg and Tsoukas, used a
phenomenological approach to theoretical work and case studies on sense-making when presenting a typology of
sensemaking that is ordered by as they call “four core constituents” of sense-making: sense-action
nexus (or “being-in-the-world”), temporality, embodiment, and language. ²² Most knowledge
in sensemaking theory revolves around language as it is strongly connected to articulation of the lived
experience in order to process collectively and to connect to cognition. Sense-action nexus, temporality, and
embodiment are more fluidly overlapping concepts and less clearly developed in sensemaking theory. The body
plays a certain role in sensemaking - it is not only the case through embodying knowledge, but also through the
experience over time and of “being-in-the-world”. To experience the world, the body needs the
senses, what entangles being-in- the-world with the body through the senses. Experience through the senses is
connected to aesthetics as understood by Alexander Gottlieb Baumgarten. Connecting to the senses, sensory
perception, sensory and sensible knowledge is also referring to building knowledge with and through the body to
refine skills which implies making sense of situations or tasks. ²³, ²⁴ As an underlying
theme, this is also expressed by the artists as quoted above.
Thus, while researching the role of the body for sensemaking in digital transformation and with digital
technologies, we also see an opportunity to push the boundaries of knowledge within sensemaking theory. We
suggest that based on the entanglement as presented above, it is necessary to unravel dimensions of aesthetics
and the entanglement of aesthetics in sensemaking.
Some experimental approaches already pointed to the role of bodily interaction and aesthetics within
human-robot-interaction, emphasizing that reflecting on the body and aesthetics in such interactions with
technologies needs to go beyond design (UX or UI design) in order to relate to the robot, to become aware of
situated dynamics, and to adhere social meaning. ²⁵ Enabling social meaning making connects strongly
to sensemaking as materialization of meaning in the context of identities and action (see introduction to
sensemaking).
The Digital Sensemaking Project
The Digital Sensemaking project (DIGI-Sense) is a two-years’ project to explore the rich space at the
intersection of the social, the technological, and the environment. Specifically, we focus on sensemaking in the
realm of digital transformation, more precisely in cyber-physical systems, with digital twins, and through IoT
elements enriched environments. ²⁶ These IoT elements are represented by a variety of technologies,
such as sensors and robots.
What does it mean to put the human at the center of technology development considering the layered input
that is needed for sensemaking? How to engage in “digital sensemaking”?
Where is the body in sensemaking with digital technologies? What implications does this have for the
development of technological systems of a symbiotic future?
What lessons can be learned from digital sensemaking on the body in sensemaking? Can we contribute to a better
understanding of the body and its sensing capabilities – aesthetics – in the sensemaking
process?
In the DIGI-Sense project we work with performance artists to explore digital sensemaking. In
three steps – along three phases – the project team works with performance artists to explore
several aspects of sensemaking through tailor-made performative research, performative installations, and
performances. The performance artists work with the technological infrastructure in their performative research,
development, and staging. Thereby, they can draw from the technological infrastructure (i.e., IoT elements,
sensors, robots, 3D-sanning and point-cloud generation systems, digital twins).
The three phases of
experimentation with the performance artists are structured as follows: in the first phase, two performers
without affinity to programming and technology development nor pre-knowledge about IoT elements, sensors, and
robots, engage with the provided systems. Their learning and performative research process into the technologies
provided with a specific focus on embodiment and the senses is in the center of this first phase. This phase
serves as an ‘approaching’ phase for non-familiar humans to IoT technologies and robots. Its major
outcome is a set of moments that could trigger sensemaking processes.
The second phase focuses more specifically on the trilogy of sensemaking, sensebreaking, and sensegiving, again
with a focus on the body, the senses, and more explicitly on aesthetics. This phase includes higher levels of
digital abstraction such as digital twins. The artist will develop a performative installation that invites
performers and test persons to explore and make sense of the digital twin of their body and movements.
The third phase aims at contrasting the idea of embodied articulation to the cognitive and language-based idea
of supporting change through poetic language. ²⁷ In this way, the project spans from initially
approaching cyber-physical elements to digital representations that finally can be encoded in human language,
however, guided by performance art. The remainder of this paper will focus on the first phase in the DIGI-Sense
project.
Methodology including Performance Art
The body and embodiment have already attracted the interest of scholars in sensemaking. They investigated the
interaction with materials and objects, but also learning processes of people rowing the Amazon. (e.g.,
²⁸, ²⁹) The body thereby has been described as essential in sensemaking in several levels:
through the interaction with other bodies, non-human and more-than-human bodies, sensemaking processes and
specific moments are enriched, supported, and informed; but also the body’s capability of learning,
suffering, implicit or tacit knowledge, a more holistic approach to the sensemaker’s body to be embedded,
sentient, and situated; as well as the connection of the body to the identity and very personal processes of the
sensemaker have been pointed out.
Nevertheless, it is difficult for social scientists to go deep into the question of embodiment as the bodily
experience is something very personal, something difficult to abstract to rational arguments, and the
methodologies used to investigate the body and bodily experience are only marginally part of the repertoire of
research methods in social sciences. Connecting to art, especially performance art, is one way that has been
explored by some organization scholars previously. ³⁰, ³¹
To focus on embodiment and the senses in our research we work with performers. Performers have a heightened
bodily sensibility as they are educated to work with the body and reflect on their embodied processes. Through
working with them we aim at harvesting from this capacity for deeper understanding of the role of the body in
sensemaking. Moreover, the four core constituents of sensemaking are aligned with experimentation with
performance artists: sense-action nexus (i.e., do, be in the space, explore and experience with the body and
senses), temporality (i.e., process, unfold over time, ways of experiencing time and being in the situation),
embodiment (i.e., work with the body, capacity to “read” the body, bodily senses, process of
embodiment), language (i.e., articulation, conversation between performers, of performers, and with the
researcher). ³²
Additionally, we employ methods from qualitative research in social sciences such as qualitative interviews,
participatory observations, observation diaries, diaries by the performers, video footage throughout their
complete process of interaction with the technologies for the DIGI-Sense project, and photo documentation. For
the visual material, we also employ strategies from visual research and performance research. ³³,
³⁴ , ³⁵ Connecting bodily experience and in-depth interviews has also previously
been used to examine micro-processes in meaning-making in experience, and thus is a relevant juxtaposition of
the body and the senses in sensemaking with cognitive reflection. ³⁶
First Performance Phase
Between March and May 2022, we worked with two performance artists (one male, one female) to explore digital
technologies with their artistic practice. Both artists do not have any background in working with digital
technologies and did not have previous experience with integrating digital technologies in their performative
practice. One of the performance artists has an additional background in human medicine, the other performance
artist has previously researched interspecies communication and animal behavior in their artistic practice.
The idea was to engage in a learning process with several chosen digital technologies provided by the Institute
of Business Informatics – Communications Engineering. This engaging learning and investigative process
would focus on the bodily experience, embodiment, and the senses – and thus aesthetics. The process would
result in “micro-performances” that express important moments of the interaction between the body
and the digital technologies for sensemaking.
The artists worked individually, next to each other, and together, depending on the stage of their process and
their sensemaking of the digital technologies. The places they worked at throughout the process were: at their
homes and private environments, at a performance space in Vienna throughout a week-long residency, and at the
premises of the University of Linz (especially when they worked with the infrastructure that was bound to stay
at the premises of the university). The researchers provided them with a selection of digital technologies to
work with, which they could choose from for their performative research.
The technologies they chose were a selection of M5Stack sensors as IoT components to be navigated digitally via
coding language Blockly (Figure 1), and the agile robot system Boston Dynamics Spot (Figure 2). Additionally,
they chose to explore the 3D-scanning and point-cloud generation system Trimble x7. The focus in their process
was on the IoT elements and the agile robot system, and an additional exploration of the 3D-scanning and
point-cloud generation system in one afternoon.
The two performers recorded their process with the digital technologies through filming themselves or each
other in the interaction, moreover, they were asked to take notes in a diary to trace their process and took
photos of their outcomes and certain moments. During selected phases of the performers’ process, one of
the researchers was present for participatory observation. The researcher also took videos, photos, and notes in
an observation diary. Additional material about the digital technologies that was used by the performers to
learn about the digital technologies has also been tracked and fed into the evaluation.
The performative research and exploration of the digital technologies became an ongoing process in which
instead of the development of staged “micro-performances” it became more important to define key
moments in this ongoing performative exploration which acted as ahamoments, oh-no moments, or moments of
“suddenly everything fell into place” and “now all the movements become fluently, and parts
fall into their places”. Finding and analyzing these moments and the process that leads up to these
moments became key to the evaluation process of the visual material. On top, the performers developed one
performance with the agile robot system to be staged at the “Lange Nacht der Forschung”
(researchers’ night) at the university campus. Central steps in the performers’ sense-making with
the agile robot system could afterwards be traced in the development process for this performance. All in all,
we generated 119 videos to use as data, ranging between 20 seconds and over 47 minutes in length.
Additionally, one of the researchers interviewed the performance artists at several moments throughout the
process. This process started with a preliminary interview before planning the research, then we did four
focused interviews with each of them. The first main interview took place before the actual performative
research started, then two in the middle of the ongoing project, and one at the end of the process. All
interviews have been transcribed. After the evaluation of the material, one more conversation with the
performers took place to clarify specific questions and get feedback on the interpretation of the data.
Coding for Data Analysis and Interpretation
The data analysis for this first phase in the DIGI-Sense project has been done coherently after the performers
submitted all their materials and data generated throughout their process. For coding and evaluation of the
data, the software MAXQDA has been employed, a software to support the analysis of qualitative research data,
including qualitative interview transcripts and video material. The design of the first performance phase was
based on the idea to start by the exploration of digital technologies through the performance artists. We traced
both their individual and group-level practices of sensemaking of the two performers, keeping in mind
conversations, forms of engagement, patterns of (social) interaction, and material artifacts. Through this we
aim to surface and can report on:
Their sensemaking process in the interaction and learning processes with the provided technologies.
The interplay of meaning making and knowledge production through conversations, theoretical input and
doing.
Thereby, the focus is on examining the role of the body and the senses, implying the role of aesthetics.
To do so, we decided to code the collected data on the one hand for moments and information that refer to
specific aspects of the sensemaking process and connect to the body, the performers’ movements, or their
open reflection on their bodily experience. ³⁷, ³⁸, ³⁹ On the other hand,
to surface an understanding of aesthetics in the sensemaking process, we also coded for aesthetic moments. These
aesthetic moments are also partially connected to sense-making literature, as sensemaking scholars and
sensemakers presented in sensemaking research refer to the aesthetic dimension without referring to
“aesthetics” as such, and to their senses as important aspects in their sensemaking. Examples can be
found plenty in sensemaking literature, and in meta studies on sensemaking. For example, expressions such as the
following are used: movements get interrupted, noise creates awareness that something is wrong vs. “normal
engine noises”. [e.g., ⁴⁰]
For the moments in the sensemaking process that connect to the body in the video material that is supported by
the additional data, we decided to search for the following codes (number of coded moments in brackets, a total
of 514 moments): “the artists are solving a problem together” (33), “wondering about something
together (while working together)” (32), “proposal of a solution” (59), “direct body
interaction with the element” (35), “thinking out loud / speculations about function or
meaning” (40), “something goes wrong / encountering a question or problem” (71),
“aha-moments” (22), “retrospective explanation of past situation step by step” (30),
“communication through body, sound, movements” (37), “directly speaking to or interacting with
camera” (6), “aesthetic elements of the sensors in the process of the performers” (43),
“metaphors” (9), and “connection between body and machine / also aesthetic” (8). The
codes needed to be adjusted for the specific inter-action modalities and functions of the agile robot system.
Especially codes on the direct “human-machine interaction” (24) and “feedback from the
machine” (15) needed to be added, others were slightly renamed to fit the situation with the agile robot
system. Overall, the codes captured learning or knowledge acquisition moments, and modalities of behaviors, both
categories addressing the performative practice of sensemaking.
For investigating the role of aesthetics, we referred to sensemaking literature and literature in
organizational aesthetics to define the codes. [41] We found 51 moments that represent these. The aesthetic
codes are: “rhythms”, “repetitions”, “movements”, “sounds”,
“comparisons”, “aesthetic analogies and metaphors”, “velocity”,
“haptic properties”, and aspects like “color, texture, form, weight”.
Discussion and Conclusion
First interpretation of the data gives insights into the rele- vance of aesthetics in the sensemaking process
to build relations between the sensemakers and their environment and context within which they make sense. The
aesthetic information that they receive through their senses informs them about basic interaction modes with
them digital technologies – and thus helps them to engage in a learning process to embody the interaction
with and the use of these technologies. At the beginning, handling the IoT elements or interaction with the
agile robot system is clumsier than when being used to handle them, and how the materialities feel, can be
touched, or handled best, need to be learned. At later stages in the process, aesthetic feedback from handling
the technologies helps to react faster and to handle them in a differentiated way. For example, over time,
rhythms develop, and movements become faster, and aesthetic feedback from the technologies (e.g., sounds) can be
understood more quickly. In the interaction with the robot, the bodily interpretation of the movements of the
robot changes from superficial interpretation of the robot’s body to a more fluent interaction of
movements.
The senses and bodily learning processes play an important role in the internalization of the knowledge and to
make sense of new situations in the interaction. Thereby, the researchers interpret that the senses and
aesthetics do not only play a role in the core constituent of sensemaking “embodiment”, but also as
a bridge to the main starting point of sensemaking, the “action-sense nexus”. Being-in-the world
also means being in the world with the senses and aesthetically experience to learn and to make sense of new
situations.
In situations of rapid digital transformation individuals often have difficulties to adjust from embodied work
to digitized systems. Becoming aware of problematic situations gets more difficult. Moving from an entangled
situation that includes cognition, the senses, and bodily experiences to an abstract situation in which mainly
cognition is reflected can lead to disorientation. Like ecological sense-making, sensemaking in a digital
environment needs a learning phase that includes the body and the senses in order to develop fine-grained
competencies. This also has implications for designing digital transformation: without personal experience of
the processes, creating an abstrac- tion for digitized systems will more likely result in systems that are
difficult to grasp for users.
Reflecting on the relevance for art and the cases shown in the beginning: artists are experts in dealing with
aesthet- ics, creating aesthetic experiences, and translating information to various bodily senses. Especially
in complex situations, art can support individuals to get a feeling for the problems (e.g., climate crisis,
water, and land use) through creating experiences. This will have a stronger impact for knowledge gain than
relying on cognition only. The next steps in the data analysis and phases two and three in the DIGI-Sense
project, will give us the opportunity to explore these outcomes in more depth. Of particular interest is the
performative practice with digital representa- tions in the second stage of the project, and the translation
from digitalized processes to new embodied and materialized meaning in the final stage of the project.
Especially as these steps entangle the cognitive with aesthetics and embodied performance involving physical and
digital components and the actors’ broader environment.
Acknowledgements
The project is supported by the Linz Institute of Technolo- gy (LIT) Seed Funding. Thanks for conversations and
contributions to Aisen Caro Chaicin, Scott deLahunta, Marko Markovic, Merritt Moore, Alexandra Murray- Leslie,
Antoni Rayzhekov, and Laura Stoll.
References
1 Nima Navab, Desiree Foerster, “Affective Atmospheres | Ambient Feedback Ecology,”
paper based on a talk presented at the International Symposium on Electronic Art, Gwangju, South Korea, June
2019, ISEA2019, http://isea2019.isea-international.org/fullPro.asp
2 Min Kyung Lee, Daniel Kusbit, Evan Metsky, Laura Dabbish, “Working with Machines: The
Impact of Algorithmic and Data- Driven Management on Human Workers,” paper based on a talk presented at
33rd Annual ACM Conference on Human Factors in Computing Systems, CHI2015,
https://dl.acm.org/doi/10.1145/2702123.2702548.
3 Lucas D. Introna, “On the Making of Sense in Sensemaking: Decentred Sensemaking in the
Meshwork of Life,” Organization Studies 40(5), 2018, 745-764.
4 Deborah Lupton, Ash Watson, “Towards more-than- human digital data studies: developing
research-creation methods,” Qualitative Research 21(4), 2021, 463-480.
5 Jean Helms-Mills, Making Sense of Organizational Change, London, Routledge, 2003.
6 Karl E. Weick, Sensemaking in organizations, Foundations for Organizational Science Vol. 3,
Thousand Oaks, Sage, 1995.
7 ibid. 3
8 ibid. 3
9 Jörgen Sandberg, Haridimos Tsoukas, “Making sense of the sensemaking perspective:
Its constituents, limitations, and opportunities for further development,” Journal of Organizational
Behavior 36(1), 2015, S6-S32.
10 Sally Maitlis, Marlys K. Christianson, “Sensemaking in organizations: Taking stock and
moving forward,” Academy of Management Annals 8(1),2014, 57-125.
11 Ann Cunliffe, Christine Coupland, “From hero to villain to hero: Making experience
sensible through embodied narrative sensemaking,” Human Relations 65, 2012, 63-88.
12 Joep Cornelissen, Saku Mantere, Eero Vaara, “The contraction of meaning: The combined
effect of communication, emotions, and materiality on sensemaking in the Stockwell shooting,” Journal of
Management Studies 51, 2014, 699-736.
13 Mark de Rond, Isaac Holeman, Jennifer Howard Grenville, “Sensemaking from the body: An
enactive ethnography of rowing the Amazon,” Academy of Management Journal 62(6), 2019, 1961-1988.
14 Ileana Stigliani, Davide Ravasi, “Organizing thoughts and connecting brains: Material
practices and the transition from individual to group-level prospective sensemaking,” Academy of
Management Journal 55(5), 2012, 1232-1259.
15 Dorothé Smit, Bart Hengeveld, Martin Murer, Manfred Tscheligi, “Hybrid Design Tools
for Participatory, Embodied Sensemaking: An Applied Framework,” paper based on a talk presented at 16th
International Conference on Tangible, Embedded, and Embodied Interaction, TEI’22,
https://dl.acm.org/doi/10.1145/3490149.3501332.
16 Claudia Schnugg, Creating ArtScience Collaboration: Bringing Value to Organizations, Cham,
Springer, 2019.
17 Gail Whiteman, William H. Cooper, “Ecological Sensemaking,” Academy of Management
Journal 54(5), 2011, 889-911.
19 Vivek Kant, “Cyber-physical systems as sociotechnical systems: a view towards
human-technology interaction,” Cyber-Physical Systems 2(1-4), 2016, 75-109.
20 Thomas D. Wilson, “The cognitive approach to information- seeking behavior and
information use,” Social Science Information Studies 4(2), 1994, 197-204.
21 ibid. 16
22 Elisa D. Mekler, Kasper Hornbaek, “A Framework for the Experience of Meaning in
Human-Computer Interaction”, paper based on a talk presented at 37th Annual ACM Conference on Human
Factors in Computing Systems, CHI 2019, http://www.kasperhornbaek.dk/papers/CHI2019_Meaning.pdf
23 Jörgen Sandberg, Haridimos Tsoukas, “Sensemaking Reconsidered: Towards a broader
understanding through phenomenology,” Organization Theory 1, 2020, 1-34.
24 Alexander Gottlieb Baumgarten, Aesthetica, 1750.
25 Antonio Strati, “Sensible Knowledge and Practice-based Learning,” Management
Learning 38(1), 2007, 61-77.
26 Petra Gemeinböck, “The Aesthetics of Encounter: A Rela- tional-Performative Design
Approach to Human-Robot Interaction,” Frontiers in Robotics and AI 7, 2021, Article 577900.
27 Daniela Brill, Claudia Schnugg, Christian Stary, “Makes Digital Sensemaking Sense?
– A Roadmap for Digital Humanism in Increasingly Transhumanist Settings,” New Explorations 2(3),
2022, 86-106.
28 Karl E. Weick, “Reflections: Change agents as change po-ets,” Journal of Change
Management 11(1), 2011, 7-20.
29 ibid. 12
30 ibid. 15
31 Brigitte Biehl, Dance and Organization: Integrating Dance theory and Methods into the Study
of Management, New York, Routledge.
32 Emilie Reinhold, Claudia Schnugg, Charles Barthold, “Dancing in the office. A study of
resistance,” Scandinavian Journal of Management 34(2), 2019, 162-169.
33 Daniela Brill, Claudia Schnugg, Christian Stary, “Digital Sensemaking: Sensemaking as a
Driver of Transformation”, paper based on a talk presented at 17th International Forum on Knowledge
Asset Dynamics, IFKAD’22, https://www.ifkad.org/event/ifkad-2022/.
34 ibid. 12
35 Emma Bell, Samantha Warren, Jonathan E. Schroeder, The Routledge Companion to Visual
Organization, London, Routledge, 2014.
36 Philip Barnard, Scott deLahunta, “Mapping the audit traces of interdisciplinary
collaboration: Bridging and blending between choreography and cognitive science,” Interdisciplinary
Science Reviews 42(4), 2017, 359-380.
37 Desiree Foerster, “Re-reading Whitehead Through the Pre- reflective Experience of
Atmospheric Processes,” Ambiances, 2021, Article 3745.
38 ibid. 21
39 ibid. 12
40 ibid. 5
41 Antonio Strati, Organization and Aesthetics, London, Sage, 1999.
Deeply Listening Through/Out the Deepscape
HugoScurto1,
AxelChemla–Romeu-Santos2
1EUR ArTeC / Université Paris 8 / École des Arts Décoratifs
– PSL (EnsadLab) 2STMS Lab Ircam–CNRS–Sorbonne Université Paris,
France hugo.scurto@ensad.fr, chemla@ircam.fr
Abstract
This paper presents early artistic, conceptual and technical work toward practicing and theorising through/out
the deepscape. I first introduce the concept of deepscape, which may designate the global flows of
media intensively computed by deep learning throughout the Internet, entangled with the material, human and
cultural resources they capitalise on throughout corporate infrastructures of artificial intelligence (AI). I
then propose to explore deep listening of soundscapes generated by deep learning as a practice to raise
awareness of the planetary scale of the deepscape. I relate the diffractive prototyping of a deep generative
model of soundscape, based on the multichannel hacking of the Realtime Audio Variational auto Encoder (RAVE),
trained on worldwide soundscape data that I transversally recorded over 28 places in late April 2022, using the
Locustream online sound map. I argue that listening to the planetary soundscape that continually flows from this
deep generative model may reveal the ethico-ontoepistemology of deep learning, by recalling the landscapes that
are being exploited by infrastructures of AI, while situating data collection practices and training costs of
deep learning. The paper ends by discussing art and science work that might be engaged to reveal and reconfigure
the deepscape in depth.
Keywords
Deep Learning, Deep Listening, Diffractive Art Practice.
For over half a century, a vast majority of scientists have been developing and describing Artificial
Intelligence (AI) through anthropomorphic conceptions and representations. From symbolic AI to modern Artificial
General Intelligence ²⁷, simulating human-like intelligence through quantification and computation
have been standard goals and methods for most researchers in the field. These anthropomorphic conceptions stem
from millenary Western representations of AI as humanoid robots, or artificial objects endowed with human
intelligence ³⁷, which are still actively shaping popular representations, along with corporate
narratives, on AI. ¹¹
The technological revolution brought by deep learning in the last decade paradoxically destabilised
anthropomorphic conceptions of AI ¹², while simultaneously reinforcing anthropocentrism. On the one
hand, difficulties in interpreting internal representations of deep neural networks, as well as images they
can generate ³², pushed scientists and artists to approach AI as a form of non-human intelligence
³⁴, ⁶. On the other hand, systems powered by deep learning, from facial recognisers to text and
image generators ⁴³, ¹⁴, entailed worldwide AI applications, both public and private,
across almost all sectors of human society ⁴².
As an artist-researcher, I am interested in developing a non-anthropocentric representation of AI that raises
awareness of the planetary costs of deep learning. I am inspired by Kate Crawford and Vladan Joler’s
Anatomy of an AI system (2018) ²² , whose large-scale map reveals the globalised, extractive
infrastructures that lie beneath modern AI applications, and that are barely tackled by anthropocentric
discourses. I am also interested in engaging in diffractive art practice with deep learning ⁴⁸ to
perform and reconfigure this non-anthropocentric representation. I am inspired by Memo Akten’s Deep
Meditations (2018) ⁶ , whose deep learning-generated continual flows of images and sounds let us
embrace the interconnectedness of AI with all socio-material bodies of time and space, including us, and the
planet.
In this paper, I introduce the concept of deepscape, along with deep listening experiences of
planetary soundscape live-generated by one deep neural network. The deepscape designates the global flows of
deep learning-generated media in-tensively computed by deep learning throughout the Internet, entangled with the
material, cultural and human resources they capitalise on throughout corporate infrastructures of AI. Deep
listening experiences ³⁹ of deep learning-generated continual flow of soundscape aim at raising
awareness of the deepscape, by leveraging a planetary representation for a deep neural network, and recalling
the landscapes that are being exploited by infrastructures of AI. By including non-human voices and planetary
scales within research and practice with deep learning, this research-creation thus seeks to resist the
anthropocentric culture of modern AI research, illuminating its media aesthetics and socio-ecological impacts.
The next sections successively describe an early conceptual mapping of the deepscape, the diffractive
prototyping of a deep generative model of planetary soundscape, and deep listening experiences led through/out
the deepscape. The paper ends by discussing art and science work that might be engaged to reveal and reconfigure
the deepscape in depth.
* The first author uses the pronoun “I” to relate their research-creation and their technical
collaboration with the second author.
Mapping the Deepscape
In this section, I sketch an early map of the deepscape, by describing the global media flows, corporate
computational infrastructures, and planetary socio-material resources that compose it. I argue that the concept
of deepscape better represents the media aesthetics and planetary scales of deep learning than mainstream
anthropomorphic conceptions and representations of AI. Furthermore, I suggest that the deepscape may inspire new
practices with deep learning, which may complement those with/in AI, as we will see.
Global Media Flows
Since DeepDream almost a decade ago ³⁸, the Internet has been flooded by media generated by deep
learning. These media encompass image, sound and text, and more recently, video, 3D movement or biosignals
⁴⁷. Sounds and images generated by deep learning also permeate our physical environments, with
voices of AI assistants resonating in smart homes and cities, and mobile-based face filters re/acting over our
bodies across the globe ³³. The concept of deep fake has made popular such media across fiction and
reality, raising awareness of deep learning’s threats to steal one’s identity ¹³. As
AI-based virtual worlds are rapidly evolving, we can expect these media flows to grow in the near future ¹.
Taking inspiration from Arjun Appadurai’s notion of scape ⁸, I propose to introduce the
notion of deepscape to point to these global flows of media intensively computed by deep learning and permeating
digital and physical worlds. Rather than fake, I argue that these media are real, based on the impact they have
on cultures and societies, and the planetary computation they build on to exist, as we will see. Importantly,
such media are materially configured to exist under the form of generative flows, due to computational
architectures of deep learning, which can either support streamable media generation ¹⁶, or
continuous interpolation between images or sounds ⁹. Artists have well explored these material
attributes of deep learning in stream-based installations ¹⁰, including dadaBots and their infinite
online music live-streams ¹⁹, or Anna Ridler and her moving images ⁴⁴.
While humanoid robots surrounded by blue zeros and ones keep on being summoned to promote modern AI
applications ¹¹, I suggest that the singular aesthetics of media flows generated by deep learning are
starting to replace anthropomorphic representations of AI at a global scale in our collective imagination.
Indeed, ambiguous attributes of images generated by GANs ³² (Generative Adversarial Networks, a
specific type of deep neural network), along with the almost infinite interpolative flows from which they
emerge, have permeated our representations of machines across art and science, along with our conception of how
they may exhibit creativity ²⁹. In this paper, I limit my analysis to witnessing such global
flow aesthetics of media generated by deep learning, and how they are transforming our representations of AI.
Corporate Computational Infrastructures
Different types of computational infrastructures actually support the deepscape and its global media flows. The
first is the Internet ²⁰, since media generated by deep learning actively flows through social
networks ¹, helped by mobile applications. A second type is deep generative models ²⁶,
which are compressed representations of datasets, computed by deep learning, enabling stream-like generation of
media that more or less resemble the dataset. A third type is super-computers and data centers combined with
cloud computing ⁵², since deep learning requires intensive computation and large amounts of data to
produce deep generative models.
Over the last decade, such computational infrastructures have been in majority invested, developed and owned by
global corporations, including the Big Tech. In addition to owning Instagram and Facebook, Meta recently
introduced their AI Research Supercluster, consisting of 16,000 graphical processing units (GPUs) especially
designed to develop deep learning applications, parallel to their metaverse ⁴. OpenAI—recently
followed by Meta, Google and Microsoft—developed several popular deep generative models, including Dall-E
⁴³, GPT-3 ¹⁴ and Jukebox ²⁴, enabling to generate image, text and music
respectively, based on text prompts. Google developed Colaboratory ¹⁸, which leveraged the
corporation’s GPU-based cloud computing services to accelerate deep learning applications, in turn
enabling the emergence of a global community of practice dedicated to deep generative models ⁴⁰.
Importantly, Google also develops and maintains TensorFlow, one of the most widely used library to develop deep
generative models and modern AI applications ².
I suggest that corporate computational infrastructures are going to exponentially increase the scale of the
deepscape and its global media flows in the next coming years. In addition to the Big Tech, hundreds of
worldwide corporations have been investing in deep generative models recently, constituting what have been
called a “generative AI landscape” or “creative new world” by investors and analysts
³. It is highly probable that these global corporate investments are to modify distribution of work in the
coming years, especially in the creative and cultural industries ⁷. Many artists have made such political
aspects of AI central to their practice with deep learning ¹⁷. Holly Herndon and Mathew Dryhurst
proposed artist-led platforms addressing legal issues related to ownership of training datasets ⁵⁰,
and of deep generative models themselves ³¹. In this paper, I limit my analysis to pointing out to
corporate computational infrastructures that support the deepscape, and how they may generate governmental
issues at a planetary scale.
Planetary Socio-Material Resources
Last but not least, a wide array of planetary resources enables to fuel the deepscape, exploited by corporate
computational infrastructures. A first type is material resources ²², ⁴¹. Rare
elements such as lithium are required to build the super-computers enabling deep learning computations, along
with personal computers and smartphones required to interact with deep generative models ³⁰.
Extracting such material elements devastate landscapes across the globe, while contributing to huge levels of
atmospheric and water pollution, which in turn generate political conflicts between local populations, opaque
intermediaries and global corporations ²³. In addition, electricity consumption of deep generative
models breaks records and should continue increasing every year, despite engineering efforts to reduce
computational costs ⁵². This leads to huge carbon emissions that threaten the lives of both humans
and non-humans across the globe.
A second type is human resources. Workers are required to extract material resources needed to build
computational infrastructures of deep learning, often enduring awful economic and health conditions
²³. Large teams of engineers actively maintain such computational infrastructures ², from deep
learning libraries to mobile AI applications, across various hardware and software, as computational technology
keeps on rapidly evolving. Computer scientists are required to design and implement deep generative models.
While they sometimes build and collect the datasets over which they train their models, they essentially
delegate data work to disadvantaged people across the globe ²⁸, thus reinstantiating work
inequalities and data colonialism ²¹ through deep learning. Far from objective, deep generative models
thus perpetuate values and world views of a few ⁴⁵, reinforcing cultural bias of the global
computational infrastructure.
Indeed, a third type of resource that directly relates to this data work is cultural resources. Beyond mere
numbers, datasets carry historical and cultural value, as images, sounds or text they represent are always and
already entangled within communities that produced it, and labels and classifications they encapsulate reflect
certain world views ⁴⁵. When a person or a corporation trains a deep generative model over a given
dataset, it exploits its cultural value to generate media that somehow reproduces this value, while distorting
it, or even biasing it in unethical ways ⁵¹. Artists such as Stephanie Dinkins have sought to
illuminate responsible ways of dealing with datasets and deep generative models, especially toward communities
that are underrepresented in deep learning, such as the Black community ²⁵. Cultural accountability
may eventually become difficult to tackle, as corporate AI infrastructures have started using datasets produced
by deep generative models to train deep generative models ⁴³. In this paper, I limit my analysis to
pointing out to socio-material resources that fuel the deepscape, and how they damage socio-ecological
landscapes of the planet.
Diffractive Deep Learning Prototyping
In this section, I describe the diffractive prototyping of a deep generative model of planetary soundscape,
based on a transversal soundscape dataset that I collected over the globe in one week-end, and the multichannel
hacking of the RAVE variational auto-encoder ¹⁵, made in collaboration with the second author. I
argue that such a diffractive practice with deep learning reveals and reconfigures the deepscape, by performing
the planetary scale of deep learning, while illuminating the material costs that its computations imply.
Transversal Soundscape Dataset
I identified three requirements for my data collection practice to engage with the concept of deepscape. First,
data should be collected across worldwide places to transcribe the planetary scale of deep learning. Second,
data should be in the form of multichannel soundscapes to convey a sense of spatial landscape through sound.
Third, data should combine anthropophony, biophony and geophony ³⁵ to have the model embody a sonic
environment rather than one individual form of intelligence, such as a musician or a bird.
I used Locustream to collect this soundscape dataset ⁴⁹. Locustream is an online sound map
developed by the Locus Sonus research group that enables to listen to live stereophonic soundscapes across the
globe. Practitioners can setup their own microphone in the place of their choice, and livestream the sound it
captures online, using either a Pure-Data patch running on a RaspberryPi, or a mobile application for
smartphone. The map has been used over sixteen years by artists and scientists to promote novel forms of
listening to, and performing with, the planet and its diverse environment. I decided to lead online field
recording through the Locustream online sound map. Specifically, I opted to restrain my field recording to one
local portion of time—namely, one weekend in late April 2022—, while widening it to all portions of
the globe accessible on the map over that period— namely, twenty-eight places across seventeen countries
and five continents. I called it transversal, following scientific dataset nomenclatures used in statistics. On
the one hand, such a transversal dataset let me situate ecological properties of soundscapes related to
planetary seasonal shifts, be they related to human or non-human activities. On the other hand, it enabled me to
materialise the planetary scale of deep learning, since all places, cultures and species of the transversal
dataset would be eventually normalised and approximated as one planetary soundscape by deep learning.
The resulting dataset consists in nearly 16 hours of stereophonic soundscape (see Figure 1), that I have deeply
listened to while recording, with a mean duration of 31 minutes per location (see Figure 2). Such small data
approach come within the scope of diffractive art practice ⁴⁸.
Deep Generative Model
Along with the second author, we opted to work with the Realtime Audio Variational autoEncoder (RAVE)
¹⁵. RAVE is a state-of-the-art deep generative model that learns to generate raw audio waveforms
using a two-stage training procedure. In the first stage, a variational auto-encoder is trained to learn a
high-level latent space of the training dataset. In the second stage, only the decoder is trained with an
adversarial generation objective to enforce continuity of the latent space. One advantage of RAVE over other
deep generative models are that audio signals are modeled and generated at a 48 kHz sampling frequency,
i.e., standard audio quality.
The second author hacked RAVE’s implementation (1) by adding a multichannel audio loader,
which enables to load audio data with c channels directly for use in RAVE, including the multichannel
factor c into encoder, decoder, and adversarial parameters. From a methodological perspective, this
add-on could be described as a form of learnable algorithm, which comes within the frame of diffractive
art practice ⁴⁸. The source code for our modified implementation will soon be publicly available.
Our deep generative model can be used with the nn~ object for neural audio synthesis (2).
Training and Streaming Costs
We trained our deep generative model on our transversal soundscape dataset for three million steps,
specifically two and one million steps for first and second stages. Trainings were conducted using a private
infrastructure, which has a carbon efficiency of 0.432 kgCO2eq/kWh. A cumulative of 360 hours of computation was
performed on GPU hardware of type Titan V (TDP of 250W). Total emissions are estimated to be 38.88 kgCO2eq of
which 0 percent were directly offset. These estimations were conducted using the MachineLearning Impact
calculator presented in ³⁶. Streaming costs are equivalent to working with a personal computer, as
RAVE support streamable audio generation on standard CPUs ¹⁶. We used the same model
hyperparameters and architecture as provided in the original RAVE implementation ¹⁵. Specifically, we
preprocessed our transversal soundscape dataset by cropping it into one-second audio slices to apply
dequantization, random crop and all-pass filters with random coefficients as data augmentation strategy.
Training evaluation was led by combining quantitative and qualitative dimensions of sound generation, with the
first training stage stopped based on joint quantitative observation of loss minimisation and qualitative
listening of generated audio, and the second stage stopped based on qualitative listening criteria.
Deep Listening Experiences
In this section, I relate first deep listening experiences created through/out our deep generative model of
planetary sound-scape. I group them within an ongoing artwork called Deep- scape: Transversal. Deep
listening is a meditative aesthetic proposed by composer Pauline Oliveros “to inspire both trained and
untrained performers to practice the art of listening and responding to environmental conditions in solo and
ensemble situations” ³⁹. I suggest that deep listening may be helpful to raise awareness of the
deepscape and its interconnectedness with all socio-material bodies of the planet. The following paragraphs
describe online radiophonic broadcast, collective acousmatic listening, and open listening and discussion
situations, inviting diverse audiences to experience planetary soundscape flows computed by our deep generative
model, and describe their spatial, temporal and material attributes. These situational wholes enabled
to illuminate somaesthetic behaviours of our deep generative model, which jointly come within the scope
of diffractive art practice ⁴⁸.
Online Radiophonic Broadcast
The first experience consists in the online radiophonic broadcasting of stereophonic audio live-streamed by our
deep generative model. It is mounted on the Locustream online sound map (3) since October 17th, 2022,
while being temporarily stopped at times. On the server side, I used the nn~ Max/MSP object for neural
audio synthesis, and the Locus Sonus Locustream Pure Data patch ⁴⁹ to mount the live stream online.
On the deep learning side, I experimented with RAVE prior for unconditional audio generation ¹⁶. On
the sound map, the stream is located at the material place where the deepscape is computed, that is, from my
home as of now. To inspire deep listening, the broadcast is accompanied with a note of intent that include two
questions: Who terraforms this deepscape? Whose scapes are getting threatened as AI sucks our attention
away from planetary issues?
Collective Acousmatic Listening
The second experience consisted in a collective acousmatic listening session led at Cirque Électrique,
Paris, France, on November 5th, 2022, for the opening of an AI-based experimental music concert. It consisted in
15 minutes of planetary soundscape (4), recorded in one shot from our deep generative model, based on
real-time reconstruction of stereophonic audio coming from a microphone placed at my home’s window. The
latter was picking up a flow of environmental soundscape, which included keyboard sounds, human sounds and
discussions, bird vocalisations, car engines, wind howlings and children screaming in their playground.
The session started by an on-stage presentation of the piece, as well as listening guidance standing as deep
listening instructions. Specifically, I told the audience that they were going to be put in the dark to listen
to 15 minutes of soundscape generated by one deep learning model; that they may approach this experience as
sound art rather than music, and that they may let themselves immerse into imaginary worlds while listening. As
we set the volume relatively high to produce strong bass, I warned the audience that the experience was going to
be quite loud, and therefore, that they should not hesitate to leave if it became difficult.
The audience—approximately one hundred people—was standing up while put in the dark for acousmatic
listening. During listening, several people were witnessed closing their eyes and collectively relaxing their
necks and bodies while standing up. At the end of the session, some audience members reported how the listening
experience felt shorter than 15 minutes, which somehow testifies of their immersion in deep listening. They also
felt a sense of space while listening throughout the stereophonic stream. They reported encountering non-human
voices throughout the experience, such as humid rocks, strong wind, screaming dogs and singing birds.
Open Listening and Discussion
The third type of experience consisted in open listening and discussion sessions led with art-research
practitioners, teachers and students. The first was a workshop at Kallio-Kuninkala, Järvenpää,
Finland, on August 24th, 2022. The second was a public event at École d’Art d’Aix-en-Provence,
France, on October 9th, 2022. The third was a seminar at École des Beaux-Arts de Paris, France, on
November 7th, 2022. Sound materials included both unconditional audio live-streamed by our deep generative
model, and the 15-minute planetary soundscape previously recorded.
The sessions started with conceptual elements about the deepscape, as well as technical information on how we
pro- totyped our deep generative model. Then, I communicated listening guidance standing as deep listening
instructions. Specifically, I told practitioners and students to listen atten- tively to the stereophonic
soundscape flowing from our deep generative model, and ask themselves the following three questions: Where
am I? What am I listening to? To whom does it cost? In addition, I asked them to imagine five words that
would best describe their listening experience. Due to the openness of these sessions, most participants did not
strictly follow these instructions, but rather used them as listening guidance to foster their imagination.
The conceptualisation of the deepscape found echo in most art-research practitioners and teachers. Participants
that involved in the discussion generated various interpretations of the deepscape through/out listening. One
artist had the feeling to be “on another planet inhabited by Martians”. One composer
described “an impossible landscape of windy ocean with small insects and land grass”.
Rather than a soundscape, one teacher felt they were under a “strange living creature”, or
“monster”, attempting to regurgitate things with its throat. Students felt they witnessed
“noises of a diver”, moving “inside and outside water” while
“adjusting their oxygen level”. One of them added: “It’s a good profile of
what a planet is sounding like. It’s like a blue planet.”
Discussion
In this section, I discuss further art and science work to be done to reveal and reconfigure the deepscape in
more depth. Specifically, I discuss the planetary media computation driven by AI and revealed by the deepscape,
the inseparability of ethics, ontology and epistemology when engaging with deep learning, and current work led
toward bodily intra-acting through/out the deepscape.
Deepscape as Planetary Media Computation
Discourses on AI almost always suggest anthropomorphic conceptualisations and representations among expert and
novice audiences. These conceptualisations often dismiss the media aesthetics of deep learning, as its computed
flows of images, sounds and texts intensively permeate worldwide physical and digital environments. By evoking a
sense of spatiality with the suffix “-scape”, and of deep learning with the prefix
“deep-”, the concept of deepscape may reveal the planetary media computation driven by AI, recalling
the intensive socio-material extractions led by corporate infrastructures to support its media flows across the
planet ⁴¹. I am looking forward to critiques from other researchers to reconfigure this concept in
more depth and entangle a plurality of perspectives and disciplines within a potential theoretical framework.
From a practical perspective, the concept of deepscape also emphasises the landscape aesthetics that
accidentally emerged from deep learning. Indeed, first art practices with deep learning consisted in navigating
so-called latent spaces computed by deep learning over scientific datasets built for image generation
¹⁷. Rather than the realism originally sought by scientists, the most intriguing findings essentially
lied in ambiguous images, that deep learning interpolated or extrapolated from the dataset ³². Artists
then curated and showcased such findings as art, or sought to recreate this navigation process in art
installations, typically presenting the audience with deep learning-generated flows of sounds and images [10]. I
suggest that this process is reminiscent of field recording, where sounds and images are experienced in a
continuous flow, while navigating unknown places, sometimes producing strange encounters that exceed human
interpretation skills, and careful data collection that might reveal the interconnectedness of the recorder with
the planet ³⁵.
Recently, artists have been exploring spatial concepts for AI, describing it as “imaginary
landscapes” ⁵³ conveying a “worldview” through their training data ⁵, or even
crafting deep generative models to create virtual places ⁴⁴. My Deepscape: Transversal
artwork might contribute to these works, by situating data collection practices and planetary costs of deep
learning within art practice, while seeking to resist the anthropomorphic culture of modern AI research.
Ethico-onto-epistem-ology of Deep Learning
Our diffractive prototyping of deep learning illuminates the inseparability of ethics, ontology and
epistemology when engaging with deep learning. As we discussed above, collecting a dataset entails an ethics of
ownership, representativeness and positionality. In our case, all soundscapes recorded online were open sourced
by the Locustream online sound map ⁴⁹. Specifically, each soundscape comes with a complex history of
negotiating with local populations and authorities, ensuring that an open microphone would hinder human and
non-human communities as little as possible. Furthermore, the transversal soundscape dataset pragmatically
represents socio-economic abilities of worldwide populations and countries to buy microphones and provide access
to the Internet. Lastly, my positionality was described through the relying on a transversal data collection
method, of which I attempted to be transparent on the advantages, motivations and limitations to it, as well as
through my choice to deeply listen to one stream at a time while recording, instead of automatically recording
all streams simultaneously using machine listening. I suggest that dataset building inevitably produces an
ontology of deep learning-generated media. Reactions harvested along the three deep listening experiences
testify of the planetary soundscape flowing from our deep generative model, approximating and normalising
worldwide soundscapes in the dataset. Making training and streaming costs transparent through our technical
collaboration, as suggested by diffractive art practice ⁴⁸, also illuminates the material
entanglement of such planetary soundscape with planetary resources exploited by infrastructures of AI to produce
deep generative models. Beyond artwork analysis ¹⁰, ¹⁷, ²⁹, I suggest that
further research might be done on corporate deep generative models, such as Dall-E, GPT-3 or Jukebox, to reveal
the ontologies they produce, since they are often trained unethically over large datasets automatically scraped
from the Internet.
Lastly, deep listening experiences of my Deepscape: Transversal artwork may also promote a alternative
epistemology for deep generative models. Indeed, scientists often evaluate deep generative models through fast
and quantitative methods, typically automatically-generating short samples with deep learning, and measuring
reconstruction quality using Likert scales ¹⁵. I suggest that approaching media flows computed by
deep learning through slower and careful crafting may reconfigure how we conceptualise deep generative models,
and perhaps, how we develop them. While such crafting practices are emerging naturally among communities of
practices dedicated to deep generative models ⁴⁰, I suggest that researchers also have a
responsibility to promote such slower ways of producing knowledge about deep generative models, in a
rapidly-evolving computational society.
Bodily Intra-Acting Through/Out the Deepscape
While deep listening may help grasp the planetary scale of the deepscape, it might remain opaque for certain
audiences that are not accustomed to music or sound art practices. Fortunately, there remains several ways of
practicing and bodily intra-acting ⁴⁸ through/out the deepscape. Along with other artists
⁵³, ⁴⁴, I am currently exploring landscape image generation with deep learning. While
images remain two-dimensional, they might recall the landscapes of the planet that are getting damaged as the
deepscape rapidly evolves.
I am also currently leading sound and image field recording in one local portion of space, while widening it to
all portions of time accessible over one year. Such a longitudinal data collection practice should complement
the transversal method presented in this paper. Instead of a planetary soundscape, it may enable to learn a
model of the planet it-self, as sound and light recorded over the year from one identical place may reflect the
planet’s motion in time and space, with corresponding climatic variations. Such a deep longitudinal model
should further my diffractive art practice with deep learning, revealing further ecological entanglements of AI
with all socio-material bodies of the universe.
Lastly, I am currently exploring soundwalk-inspired modes of listening to, and practicing with, deep learning.
I am collaborating with other artists-researchers to create virtual soundwalks in latent spaces computed by deep
learning, as an art-based method to illuminate spatial attributes of the deep-scape ⁴⁶. We are also
exploring computational methods to map bodily movements with soundscape modulations by deep learning to
stimulate embodied ways of navigating, and producing knowledge, through/out the deepscape. I am looking forward
to further suggestions from practitioners and researchers in other fields.
Conclusion
This paper presented early artistic, conceptual and technical work toward practicing and theorising through/out
the deepscape—the planetary media computation intensively led by corporate infrastructures of AI over
material, human and cultural resources. I proposed deep listening experiences of deep learning-generated
continual flow of soundscape to raise awareness of the deepscape, by recalling the landscapes that are getting
exploited by infrastructures of AI. By including non-human voices and planetary scales within research and
practice with deep learning, this research-creation thus sought to resist the anthropocentric culture of modern
AI research. I am looking forward to further art and science work that might be engaged to reveal and
reconfigure the deepscape in depth.
Acknowledgments
Authors would like to thank audience members who took part in deep listening experiences, as well as humans and
non-humans who contributed to the transversal soundscape dataset, and research-creation collaborators Anne
Dubos, Antoine Caillon, Bertha Bermudez, Emanuele Quinz, Jan Schacher, Lucile Vareilles, Philippe Esling, Peter
Sinclair, Samuel Bianchini, Vincent Rioux, et al.
1 What if 99% of the Metaverse is generated by AI? https://cifs.dk/news/what-if-99-of-the-metaverse-is-made-by-ai.
Accessed: 2022-11-29. M. Abadi, P. Barham, J. Chen, Z. Chen, Davis, A. Dean, J., M. Devin, S. Ghemawat, G.
Irving, M. Isard, et al., Tensorflow: a system for large-scale machine learning. In 12th USENIX symposium
on operating systems design and implementation (OSDI 16), 2016, 265–283.
3 Introducing the AI Research SuperCluster, Meta’s cutting edge AI supercomputer for AI
research, Accessed, 2022-11-29, http://ai.facebook.com/blog/ai-rsc/
4 M. Akten, R. Fiebrink, M. Grierson, Learning to see: you are what you see, In ACM SIGGRAPH
2019 Art Gallery, 2019, 1–6.
5 M. Akten, R. Fiebrink, M. Grierson, Deep meditations: Controlled navigation of latent space,
2020, arXiv preprint arXiv:2003.00910.
6 G. Amato, M. Behrmann, F. Bimbot, B. Caramiaux, F. Falchi, A. Garcia, J. Geurts, J. Gibert, G.
Gravier, H. Holken, et al., Ai in the media and creative industries, 2019, arXiv preprint
arXiv:1905.04175.
7 A. Appadurai, Modernity at large: Cultural dimensions of globalization, volume 1. U of
Minnesota Press, 1996.
8 G. Arvanitidis, L. K. Hansen, S. Hauberg, Latent space oddity: on the curvature of deep
generative models, 2017, arXiv preprint arXiv:1710.11379.
9 S. Audry, Art in the Age of Machine Learning, MIT Press, 2021.
11 P. Bory, Deepnew: The shifting narratives of artificial intelligence from deep blue to
alphago, Convergence 25(4), 2019, 627–642.
12 J. F. Boylan, Will deep-fake technology destroy democracy, The New York Times, 2018.
13 T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G.
Sastry, A. Askell et al., Language models are fewshot learners. Advances in neural information processing
systems, 2020,33:1877–1901.
14 A. Caillon, P. Esling, Rave:Avariational autoencoder for fast and high-quality neural audio
synthesis, 2021, arXiv preprint arXiv:2111.05011.
15 A. Caillon, P. Esling, Streamableneural audio synthesis with non-causal convolutions, 2022,
arXiv preprint arXiv:2204.07064.
16 B. Caramiaux, S. F. Alaoui, ”explorers of unknown planets”: Practices and politics
of artificial intelligence in visual arts, Proceedings of the ACM on Human-Computer
Interaction, 2022.
17 T. Carneiro, R. V. M. Da No ́brega, T. Nepomuceno, G.-B. Bian, V. H. C. De Albuquerque,
Reboucas Filho, Performance analysis of google colaboratory as a tool for accelerating deep learning
applications, IEEE Access, P. P. 2018,6:61677–61685.
18 C. Carr, Z. Zukowski, Generating albums with samplernn to imitate metal, rock, and punk bands,
2018, arXiv preprint arXiv:1811.06633.
19 G. M. Chatonsky, Esthétique des flux (après le numérique), 2016.
20 N. Couldry, and Mejias, U. A., Data colonialism: Rethinking big data’s relation to the
contemporary subject, Television & New Media 20, 2019, (4):336–349.
21 K. Crawford, V. Joler, Anatomy of an ai system, Retrieved September 18, 2018.
22 K. Crawford, Atlas of AI, Yale University Press, 2021.
23 P. Dhariwal, H. Jun, C. Payne, J.W. Kim, A. Radford, L. Sutskever, Jukebox: A generative
model for music, 2020, arXiv preprint arXiv:2005.00341.
24 S. Dinkins, Community, art and the vernacular in technological ecosystems.
In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, 2020,
221–221.
25 D. Foster, Generative deep learning: teaching machines to paint, write, compose, and play,
USA, O’Reilly Media, 2019.
26 B. Goertzel, Human-level artificial general intelligence and the possibility of a technological
singularity: A reaction to ray kurzweil’s the singularity is near, and mcdermott’s critique of
kurzweil, Artificial Intelligence, 2007,171(18):1161–1173.
27 M. L. Gray, and S. Suri, Ghost work: How to stop Silicon Valley from building a new global
underclass, Eamon Dolan Books, 2019.
28 D. Grba,, Deep else: A critical framework for ai art, Digital 2, 2022,
(1):1–32.
29 J. Heelan, E. Gratz, Z. Zheng, Q. Wang, M. Chen, D. Apelian, Y. Wang, Current and prospective
li-ion battery recycling and recovery processes, 2016, Jom 68(10):2632–2638.
30 H. Herndon, Holly+, 2022.
31 A. Hertzmann, Visual indeterminacy in gan art, Leonardo 53, 2020,(4):424–428.
32 A. Javornik, B. Marder, J.B. Barhorst, G. McLean, Y. Rogers, P. Marshall, L. Warlop,
‘what lies behind the filter?’uncovering the motivations for using augmented reality (ar) face
filters on social media and their effect on well-being. Computers in Human Behavior, 2022,128:107126.
33 J. H. Korteling, G. Van de Boer-Visschedijk, R.A. Blankendaal, R. Boonekamp, A. Eikelboom,
Human-versus artificial intelligence, Frontiers in artificial intelligence, 2021, 4:622364.
34 B. Krause, Anatomy of the soundscape: evolving perspectives, Journal of the Audio
Engineering Society 56, 2008, (1/2):73–80.
35 A. Lacoste, A. Luccioni, V. Schmidt, T. Dandres, Quantifying the carbon emissions of machine
learning, 2019, arXiv preprint arXiv:1910.09700.
36 P. McCorduck, M. Minsky, O.G. Selfridge, H.A. Simon, History of artificial intelligence, In
IJCAI, 1977, 951–954.
37 A. Mordvintsev, C. Olah, M. Tyka, Inceptionism: Going deeper into neural networks, 2015.
38 P. Oliveros, Deep listening: A composer’s sound practice, IUniverse, 2005.
39 J. Oppenlaender, The creativity of text-to-image generation, In 25th International
Academic Mindtrek conference, 2022, 192–202.
40 J. A Parikka, geology of media, volume 46. U of Minnesota Press, 2015.
41 R Parloff, From 2016: Why deep learning is suddenly changing your life, fortune, 2016.
42 A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, L. Sutskever, Zero-shot
text-to-image generation, In International Conference on Machine Learning, 2021, 8821–8831,
PMLR.
43 A. Ridler, C. Sinders Mechanized cacophonies, 2021.
44 M. K. Scheuermam, A. Hanna, E. Denton, Do datasets have politics? disciplinary values in
computer vision dataset development, Proceedings of the ACM on Human-Computer Interaction 5, 2021,
(CSCW2):1–37.
45 H. Scurto, L. Postel, Soundwalking deep latent spaces. In Proceedings of the 23th
International Conference on New Interfaces for Musical Expression (NIME’23), 2023.
46 H. Scurto, B. Caramiaux, T. Similowski, S. Bianchini, Ganspire: Generating breathing waveforms
for art-health applications, In 5th NeurIPS Workshop on Machine Learning for Creativity and Design,
2021.
47 H. Scurto, B. Caramiaux, F. Bevilacqua, Prototyping machine learning through diffractive art
practice, In Designing Interactive Systems Conference 2021, 2013–2025.
48 P Sinclair, Locus stream open microphone project, In Proceedings of the 2018
International Computer Music Conference (ICMC 2018), 2018.
50 R. Srinivasan, and K. Uchino, Biases in generative art: A causal look from the lens of art
history, In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency,
2021, 41–51.
51 E. Strubell, A. Ganesh, A. McCallum, Energy and policy considerations for deep learning in nlp,
2019, arXiv preprint arXiv:1906.02243.
52 N. Tokui, Imaginary soundwalk, 2018.
Authors Biographies
Hugo Scurto is an artist, designer and researcher, born and based in Marseille. Their research employs art,
design and science to craft, prototype and diffract machine learning in an ecology of music. Their practice
consists in creating, listening and performing with learning machines that reveal and reshape our musical
entanglements with our environments. Hugo is currently postdoc at EUR ArTeC (Paris 8 / EnsadLab), and
co-founding member of w.lfg.ng, a post-AI music collective. Before this, they completed a PhD at IRCAM,
graduated in Physics from École Normale Supérieure Paris-Saclay, and were visiting research
student at Department of Computing, Goldsmiths University of London.
Axel Chemla—Romeu-Santos is a composer, performer and researcher, born and based in Paris. Their
scientific research focuses on generative models, bayesian learning, and signal processing to design
unsupervised sound synthesis approaches based on perceptual inference, symbolic extraction, and raw signal
generation. Their artistic work focuses on the creative aspects of such algorithms in composition and
performance configurations. Axel is currently postdoc in the ACIDS group of IRCAM, CNRS, Sorbonne
Université, and a co-founding member of w.lfg.ng. They completed a PhD in Deep Learning and Sound
Synthesis between IRCAM and the Laboratorio d’Informatica Musicale, and graduated from CRR93 cursus of
Computer Music.
A Performance Co-Created with an Autonomous Virtual System: A Symbiotic Approach
IsadoraTeles de Castro e
Costa, Hui-TingHong, Chu-yinChen
Université Paris 8 Vincennes Saint-Denis, France isadoratelesdecastro@gmail.com,
w102060018w@gmail.com, chu-yin.chen@univ-paris8.fr
Abstract
This paper is a case study of a performance co-created in interaction with an autonomous virtual system. Our
outcomes point to an essential production period where the creative team learns to know the virtual system
through indirect interactions: the match-up phase. During this step, co-creation and co-evolution moments
happened, indicating a possible symbiotic relationship. We discuss the implications and the outcomes of working
with autonomous scenography in a performative context. We then expand the reflection to the potential creative
associations between performance arts and autonomous technology.
This paper is a case study of experiments carried out in the context of the project CECCI-H2M
(”Co-évolution, co-création et improvisation humain-machine”). We will talk about a
specific experience with a virtual ecosystem and about the creative process that enabled us to explore our
research question: how can a performance be co-created with an autonomous virtual system?
We are interested in constructing virtual autonomy in the search for scenographic improvisation through the
emergence of virtual patterns and behaviors. However, what is autonomy in the context of digital arts? An
autonomous computational artifact, as described by Baljko and Tenhaaf: “has some sort of internal state
space, it outwardly manifests information as to its internal state (...), and its behavior depends on its
state” ¹. Similarly, Bishop and Erden define that “an autonomous system is typically considered
to be a self-determining system, as distinguished from a system whose behavior is explicitly externally
engineered” ⁶. In short, we could define autonomous computational artifacts as entities with an
apparent internal motivation (internal self-determining state) that guides their behavior prior to the influence
of external forces. According to Maturana and Varela, autonomy is a condition for a “living machine”
and relates to the notion of autopoiesis, that is, self-fabrication ⁷. The independence of the
artwork from the artist’s control tends to transport it to the realm of the living, at least as an
artificial form of alterity or impression of Life. In the study of complex systems and within the domain of
Artificial Life, the concept of emergence accounts for a surprising phenomenon, often linked to creative
solutions. Using the simulation of complex systems inspired by nature, artists move from the creative process to
the metacreative process: the design of autonomous mechanisms that, in turn, create artworks or artistic
situations. Following this reasoning, Whitelaw ¹⁴ examines how artists have been inspired by
Artificial Life to pursue emergence due to the agency, novelty, and creativity in biological, chemical, and
multi-agent processes.
Within a second-order cybernetic point of view, Couchot, Bret, and Tramus defined the artifact autonomy in
connection with its aptitude for self-organization and to afford interaction in different environments:
endogenous and exogenous. According to them, an autonomous artwork behavior is qualified by endogenous
interactivity, which handles the dialogue among the virtual objects, able to perceive specific characteristics
from each other and keep relationships more or less complex. The autonomous virtual system can also be open to
the perception of its environment in a “second-order interactivity”, in analogy to second-order
cybernetics. Beyond the engagement between the participant and the artistic process through an interactive
interface, second-order interactivity would appear when the autonomous system can modify itself during a
learning process by interacting with its environment to adapt to it. The participant is encouraged to explore
aesthetic possibilities when embedded in a reciprocal, emergent, and unforeseen interactive context. Depending
on the interaction interface setup, the means at his disposal for this exploration enables gesture
experimentation, movements, and creativity, in the search for communication or understanding ¹³.
In our practice and research, the application and discussion around Artificial Life are considered in the
performance art context and connected with unpredictability. Inspired by Edmond Couchot’s
question ³ about “how does the machine create the unpredictable on stage?” we intend
to study autonomous virtual systems based on the simulation of complex organic systems within the performance
context to promote a new sense of unpredictability and improvisation.
Bringing a performer into a confrontation with an autonomous virtual and perceptual system encourages emergent
phenomena and motivates improvised reactions. Our hypothesis is based on establishing this interaction dynamic
as a possible way or method of emergent narrative creation, built during the interaction in a co-creative way.
This method can indicate a symbiotic phenomenon with an artistic purpose and shed light on new ways of scenic
composition with new technologies. As researchers and designers of virtual systems, we imagine methodologies and
concepts about the associations between autonomous virtual systems as improvising presences and the living arts.
We wonder: How can the performing arts take advantage of autonomous virtual entities in the quest for new ways
of imagining the stage space and the performance? How to articulate new ways of conceiving dramaturgy and being
inspired? How can unpredictability be used to express contemporary emotions and concerns?
Methodology Research Questions
The methodological problem that encouraged this project is divided into two parts. First, how to create an
interactive virtual system to promote co-creation on stage? Second, how to cooperate with this system while
composing a performance, aiming for a creative relationship between humans and machines?
We chose to develop an evolving virtual ecosystem to solve the first problem. This choice is due to the leads
in the literature about artificial life algorithms and their potential to represent or generate emergent and
unexpected aesthetic phenomena. On the other hand, we chose the performance medium to answer the second problem
through the performer’s persistent and attentive improvisation vis-à-vis her environment. Our main
goal during the experimental phase of the artistic experimentation was establishing an exploratory exchange
between the performer and the virtual system, keeping the autonomy of the two partakers and, therefore, their
potential unpredictability to bring out a performance that shows a mutual improvisation process.
Research Hypothesis and Interaction Setup
The experiments of the research-creation project described in this article are the continuation of the
experiments of last year described in the article ¹¹. Iterating last year’s reflections and
inspired by leads in literature and electronic digital art, such as the cases of the creative ecosystems of Jon
McCormack ⁸, we have created an evolving virtual ecosystem as an essential tool for testing the hypothesis
of a creative dynamic between human and machine.
Concerning the interface, we hypothesized that an “indirect” interaction (Figure 1) would make the
interaction dynamics more exploratory and keep the autonomous (or independent) character of the two types of
entities in dialogue, real and virtual. We move from a first-order (responsive) to a second-order (adaptive)
interaction. The ideas for the performance would accommodate themselves to the progressive aesthetic discoveries
enabled by the interaction with the autonomous virtual system.
We based the design of our experiments on developing the foundations of an evolving interactive program to be
tested and improved during the artistic residency. We then conducted interaction tests to verify how our
creative expectations and the virtual behavior styles fit.
Core Co-creation Process
The main result of our research-creation residency was the provision of a methodology for developing, managing,
and interpreting a virtual autonomous scenography in the context of the living arts. This methodology involves
match-up phases through interaction with the virtual system in a receptive way to emerging aesthetic
possibilities by the entire artistic team. We, therefore, achieved a co-created performance in 4 steps:
The development of an evolving virtual ecosystem (E.V.E.) considering the performer’s presence and the
research-creation goals (co-creation and co-evolution).
The match-up phase, by the art director, of the behavior and appearance of the E.V.E., understanding how it
works, and exploring the various points of view and approaches to the system.
The phase of matching up the E.V.E. by the performer, exploring the aesthetic potential from the interaction
between her movements and the system’s behavior.
The match-up phase of the E.V.E. by the developer and stage manager once the directives of the artistic
direction are established, and the intention of the performance is solidified.
Experiments with E.V.E.
The first step in exploring the issue of co-creativity was creating a virtual system capable of manifesting
surprising, emergent, and “creative” behaviors and patterns regarding our expectations and
concerning the initial state of the visual explorations ². We have envisioned the different behaviors of
our virtual ecosystem through the planning methodology of an agent-based model1. Then, we identified some
simulations that were sensitive to their virtual environment, autonomous, and with types of movement close to
those we expected to perform virtually. Inspired by the idea of simulating complex self-organizing systems, we
used algorithms from the field of Artificial Life, more specifically from artists and researchers who work with
speculative biology or synthetic biology. Finally, we adapted the chosen biological simulations to our artistic
desires and associated the resulting behaviors with a second-order interaction with the creative team.
E.V.E. Development
We experimented with the design of a first species based on the research and reports of Arsiliath 2 and Sage
Jenson3, who were inspired by Jeff Jones’s publication on Physarum transport networks ⁴. It is a
combination of two behaviors: ant foraging and the movement and growth of the Physarum.
This first species can perceive its surroundings with a limited angle and distance. Therefore, each agent
checks the neighboring pixels to measure the quantity of “pheromones” present, represented by
colored lines left by the trajectory of agents of this species. Where there is a significant quantity of
pheromones (intensity of R, G, B in the neighboring pixel), the agent will assume as its next destination,
forming moving patterns of agents. The emergent aspect of the pattern created by the collectivity of entities is
reminiscent of the patterns created by the movement and growth of the Physarum in search of food.
The second species was based on the boid’s flocking algorithm ⁹ , ¹⁰. To move, the agent
measures its neighbor’s speed, direction, and position to calculate its speed and direction, intending to
align its trajectory, separating from neighbors so as not to hit each other and maintain cohesion in the speed
of movement. This technique allows agents to form concise groups according to the angle of view, the extent of
perception, alignment, separation, and cohesion.
The third species is inspired by the behavior of random walkers, where agents move according to a random
direction. We used a probability technique to make their movement random but biased in the direction of a
target. The target or attractor, in the case of all three species, is the position of the food. First, we
created food points generated according to a four-dimensional “Hermit” type noise. Food attracts
agents with less energy, nourishes them, and repels agents with high energy.
Once the species’ behaviors were created (Figure 2), we brought them together in a single virtual
environment and created the interactions between them. Agents could perceive and guide each other, regardless of
species. Then we set up the ecosystem dynamics with genetic algorithms. We linked the main behavior parameters
of the agents to 4 genes, corresponding to the values of R, G, B, and A in a texture buffer which kept the
D.N.A. of each agent. See Table 1 for an illustration of the correspondence between each “gene” and
each species parameter.
Agents had an initial amount of energy that increased with contact with food and decreased as the agent moved
away. The agent needed to have a minimum level of energy (called health in the program), check if there is an
equally healthy neighboring agent, and count with a probabilistic decision to reproduce. If the decision is
confirmed, the two entities’ D.N.A. is crossed (crossover) with a mutation rate, and a new agent is
created in a free space. If an agent’s energy level is too low, it is disabled and ignored in updating the
parameters of each pixel of the input textures (death of an agent in the system). See Figure 3 for the general
functioning of the virtual ecosystem.
Compute Shaders
Arsiliath, in his Psychobiotik course, proposes an optimized algorithm to realize a multi-agent system with
compute shaders in Unity. We used TouchDesigner to combine all the techniques we would use in the show in a
single software (mapping on various video projectors, generative image, sound analysis, connection with Kinect,
message management in MQTT, etc.).
To design the simulation, we used the compute shader, a parallel computing technique on the graphics card. It
computes information arbitrarily and is not involved in the rendering pipeline, although it can render images.
The information and properties concerning the agents were stored on texture buffers at the shader’s input
and output. It is a technique that uses the color information spaces of an image (texture) to store data and
therefore be able to perform calculations in parallel, fast, and without relying on the computer’s memory
and the CPU (Central Processing Unit).
Indirect Interaction
A few types of interactions, with variable control over the system, were set up and used by team members. For
example, a control interface allows the stage manager/developer to modify specific parameters concerning the
virtual agents’ behavior limits without intervening in their behavior directly. During the residency, we
tested the virtual system in interaction with the performer. Her joints were detected by the Kinect camera and
transposed into the virtual environment as food points position data (see Figure 4 ).
Another type of interaction was controlling the camera’s position and the scene’s rendering. The
artistic direction employed this control to shape the virtual environment in favor of expressing the artistic
intentions and sensations for the scene ambiance.
Match-Up Phases Artistic Direction
First, the virtual system developer presented the system’s behavior to the artistic director. We have
indicated some possibilities of exploration and visualization of the virtual environment and the possibilities
of indirect interaction by experimentation with the limits of the forces and quantity of visible elements. Then,
the artistic direction (A.D.) chose important moments based on observing the system in action and the various
emotions and sensations that its appearance engendered. The intention was to look for movements or visual
structures representing three seasons: Winter, Spring, and Fall. The director identified each selected
moment’s specific parameters for testing on the stage. To represent the season’s energy and help
refine the system, the A.D. has developed keywords to communicate her ideas:
Spring (Figure 5): petals flying away, germination, and exploration
Fall(Figure 5):deadleaves,picking,andmeeting
Through these keywords and key concepts, both the developer and the dancer could imagine their respective means
of expression to converge toward a coherent creation — the dancer with her body, the developer with the
evolving virtual ecosystem. In a previous work, our experiments indicated the importance of a shared vocabulary
between the different creative spheres for corresponding to the artistic intentions with visual and behavioral
aspects of the virtual generative system. The A.D.’s exploration of configurations and behaviors of
interest was fruitful for experimenting beyond the developer’s expectations. We witnessed new visuals and
behaviors by having another person with different expectations and ideas explore the system at the software
level. From the various possibilities of rendering and behavior discovered during the experimentation of
interaction between the dancer and the virtual system, the A.D. presented new ideas and directives of narration,
choreography, and suggestions for exploring space. This phase of match-up amplified the horizon of sensations
and concepts included in the performance.
Performer
During performative experiments between the performer and the evolving virtual ecosystem, it was interesting to
observe how one modifies or influences the behavior of the other according to their progression over time. The
method used to qualify and analyze the interactions promoted during our experiments was the examination of the
videos of the rehearsals, the explicitation interviews, and the study of different visuals from the moments of
interaction.
The representation of the performer’s body in the form of the virtual agents’ food points was
possible only during the match-up tests. Before the match-up tests happened, in this specific research-creation
residency, the performer’s artistic style follows an all-encompassing principle that represents the
natural order of the universe and emphasizes the importance of living in harmony with it. The practices of
performer’s dance are the cultivation of inner peace that originated from the sage Laozi: “The way
(Tao) gave rise to the one. The one gave rise to the two. The two gave rise to the three. The three gave rise to
all the ten thousand things”. With the keywords (shared vocabulary) developed by A.D., the performer
absorbs, internalizes, and transforms these concepts through the inner path of the body to understand and
explore the visual and luminous impressions coming from the projection.
The process of understanding and exploration of the performer demonstrates adaptation progress of freedom
through perceiving the environment. The performer mentioned that her ideas of movements in this experiment are
no imposed figures before the match-up tests between the two partakers (the performer and the virtual system),
instead, it presents a great deal of freedom of exploring the aesthetic potential for the performer herself who
can, according to the intention, speed up or slow down, move or stand still, stretch or contract, using her body
to create a new representation and give free rein to the imagination.
The intention of the dancers gradually changed as the match-up tests progressed. The intention changed from
paying attention to the internal energy transformation of the body, to feeling the temperature and density of
the performance space, and to perceiving the behavior of different species from the virtual system. We further
discovered how the system reacts to the performer’s specific movements and what behaviors would be
necessary for the interaction to be visible and graceful Figure 6.
An example of co-creation during the match-up phase was adjusting the speed of the performer’s movements
so that the virtual agents could adapt and have time to reach the food. The dancer’s movements had to be
slower, and the maximum magnitude of the virtual agents had to go up to achieve a balance between responsiveness
and autonomy of each virtual agent movement. This necessity was evident in the winter scene, where the
camera’s point of view was far away, and therefore the movements of the agents were less evident in the
short term. To make them more visible, we had to increase the agents’ responsiveness and maximum speed.
This decision influences the selection and reproduction of the agents that will persist in the ecosystem: only
the fastest agents are selected. To balance the virtual rhythm with the performer’s movements, we had to
slow down the performer’s movements and give the agents time to reach the food, even if they were slow.
The goal was to seek a more varied selection over time.
Analyzing the videos of the match-up experiences, we observed compelling movements of the performer towards the
virtual system — signs of imitation attempt to understand reactions and immersion in the rhythm
predominant in the projection. The stage manager also experimented with system parameters to make the
interaction more or less apparent.
The interaction’s rhythm, speed, and power were still in the research phase. The evolution process of the
two inter-actors fluctuated according to the power of reactivity of the system concerning the performer. The
variety of behaviors and viewpoints extended the scope for imagining a story or narrative that would explain the
different behaviors, such as different personalities or characters of the visible agents. The heterogeneity of
behaviors among the agents made it easier to have affection and empathy towards the personalities and the
representation of an environment inhabited by various autonomous and unpredictable entities. The contrast
highlights this variety: when we see that over time, the agents standardize their movement towards more speed
and responsiveness to reach the food points more quickly.
Unlike the procedure with the artistic direction, the matchup phase carried out with the performer was much
more embodied and based on her sensitive, tactile, and energetic perceptions of the scenic atmosphere created by
the evolving virtual ecosystem. This intention-in-action (in contrast to prior intention) adapted by the
performer focuses on the significant association with awareness and randomness. She is present and lives in this
autonomous scenography through the improvisation practices of freedom. However, she did not consider the
system’s functioning, so we cannot say that being an evolving ecosystem, conceptually or technically, has
influenced the quality of the interaction or the dynamics of co-creation. The dancer states that she had the
impression of being in a space without gravity (“outer space”). We would like to know if we were
close to embodied immersion in the virtual environment.
Initially, the variety of movement and the autonomy in the agent’s decisions (the need or not to seek
food, variety of attraction forces) moved the interaction away from the feeling of control and manipulation on
the part of the performer. Instead, it became a feeling of immersion in an environment, in a cohabited
atmosphere, where the influence between one and the other is relative and negotiable. In a retroactive dynamic,
the ecosystem follows the performer, and the performer guides his movements through the rhythmic effects in his
surroundings.
Stage Manager and Developer
The developer took notes and screen captures of the corresponding configurations at the specific moments that
inspired the A.D. Nonetheless, the patterns depend not only on the system parameters (indirect controls) but,
above all, on the interactions’ history. Therefore, the variables captured by the developer only
contributed partially to reproducing the moments chosen by the A.D. The vocabulary stipulated by the A.D. was
the primary support for investigating the desired appearance. Guided by the keywords, the developer improvised
and directed the agents to behaviors or organizations close to the metaphors expressed by the keywords. For
example, if the intention was to have a warmer, agitated, and chaotic scene, the developer/stage manager sought
to modify the parameters without knowing the exact consequence of such changes to seek a rendering close to
expressing such keywords.
Though it was difficult to impose a specific appearance or behavior because the system was not controllable
(and that was not the idea of the experiment), the developer mediated the dynamic between the A.D. and the
system. To understand or predict the emergent patterns of the changed parameters associated with the
interactions with the performer and the history of the millions of parallel interactions whose comprehension
eludes us, we had to improvise and explore the system’s possibilities in an almost striving relationship.
The results of this match-up effort were sometimes disappointing for not rendering the desired appearance.
However, it was often surprising to discover more satisfying visuals than expected.
We developed an interface to interact with the scenes and system variables, allowing the freedom to adjust
parameters in real-time. During the presentations, this freedom was essential for taming, exploring, and
discovering new movement possibilities and the virtual system’s appearance. Since it is an evolving
system, the parameters cannot be fixed beforehand but can only be tweaked by the stage manager during the
performance. Therefore, desired aesthetic ambiances must be pursued and evolve gradually through interactions
and temporal and spatial transformations of the virtual system during the performance.
Discussion Symbiotic Imaginaries and Autonomous
Individuations
The system did not count on physics simulation, and the agents did not detect collisions (despite a tendency
for separation, according to the boids algorithm). Instead, the entities agglomerated, overlapped, and gave the
impression of mass and fusion: as if the whole formed another element. This impression is also due to the
agent’s persistent alignment and paradoxical erratic manners between cohesion and repulsion. This behavior
showed volumes of virtual matter moving in one direction, volumes presenting deformities and noises in their
contours and shapes. From the more distant point of view, which shows the organizational phenomena on a large
scale and collectively, we can see the paths formed by the agglomeration of virtual agents. The agent’s
trajectory features became visible through the effect of image accumulation and the reduction of their opacity.
The patterns created by the endogenous interactions and their speed and direction adaptations concerning their
neighbors become evident. This scene shows a less individualizing level of the agents. However, it makes visible
the formation of agglomerations that resemble the formation of other individuals or at least other configured
forms of a collective or a mass of agents (see Fig.7).
The scene representing Spring is the one where the camera is closer to the agents in its scale. It reveals the
entities’ geometric shapes, colors, and particular behaviors. Therefore, it is a scene where the
composition is more dynamic and faster, the images are beyond the control of the stage manager and the dancer,
and the interaction is more mysterious depending on the area of the ecosystem focused by this point of view.
Regardless, another type of empathy is produced by this scene. The dancer can differentiate each agent’s
life stages, birth, search for food and path, and death. That is when one understands that an association of
individuals forms the previous image.
The diversity of behaviors and the multiple imaginaries inspired by the different scales of observation of the
virtual system encourage us to explore performance through the evolution of interaction without having a
narration or a script beforehand. The idea is constructed around and with the virtual scenography presence and
by the curious exploration of its aesthetic and interactive possibilities.
While, initially, we did not have a script or description for the performance, the narrative’s structure
began with the virtual system’s development. We manifested our imagination about a virtual performance of
autonomy while constructing behavior and interactions between agents. That autonomy, in our case, is not only
expressed at the virtual individual level but on the emergent collective appearances as well. It is about
multiple endogenous and indirect exogenous interactions, inducing unpredictability and a sense of disobedience,
resistance, or even conflict. Furthermore, the choice of losing control over the system’s transformation
also abstracts our intentionality. The system enacts our initial intentions and ideas, and then it is left to
develop independently. Whitelaw raises these issues about the challenges of operating Artificial Life as a tool
because it also carries a conceptual and cultural engagement in its expression and choices. Engagements that are
narratives themselves ¹⁴.
Symbiotic Creativity
Another way of seeing the emergence of new forms than the neo-Darwinist interpretation is symbiogenesis,
proposed by Lynn Margulis ⁵. This theory of the vital evolutionary path intersects with McCormack’s
vision of creative emergence through the combination of forms and behaviors. This biological approach inspires
our assumptions about appropriating natural Life with technology. As Tenhaaf indicated: we take “biology
as a ready-made” ¹². The concepts and techniques of artificial life simulation guide us toward
the emergent affinity between the performer and the evolving virtual ecosystem by treating the two interactors
in another ecosystem where we add the scene’s environment. We aim for codependent relational recombination
between the parts to form a conceptual and creative symbiogenesis: an association of movements, gestures,
shapes, and compositions.
As explained by McCormack, the ecosystem can be more than just virtual; the user, the system, and the
environment can form it. That implies the vision of the virtual system not as a tool but as a member of the
scenic ecosystem. Therefore, symbiotic creation can be considered at two levels in our project, endogenous and
exogenous. The endogenous symbiotic interaction exists in the association of agents when one takes advantage of
the trajectory of the other to achieve its goal. On the other hand, exogenous interaction allows the
organization of a symbiotic association at the level of the stage’s environment. A dynamic of cohabitation
between the artistic team and the virtual system is set up in a scenographic form.
Concerning the limitations and possible improvements of the virtual system, on the other hand, the task that
measures the ability of agents to survive in the virtual environment is the ability to feed. This task was
revealed as a not very creative one. Since the food is the dancer’s body, the main task is to follow the
dancer’s body, and it was solved by selecting the most agile individuals. Nonetheless, we identified more
creative tasks in the aesthetic domain: the movement and its variety and the creation of unexpected patterns.
These aesthetic observations are paradoxical with the selection process we have implemented—the current
dynamic values the standardization, speed, and concentration of agents on feeding themselves.
A solution would be to reverse the agents’ aptitude evaluation to appreciate those not following the
performer’s movements as a priority. Instead, we aim to value those who prioritize the tasks of
interaction and distraction in the virtual environment while keeping their need to feed themselves. These tasks
that prioritize variety correspond better to the moments when rendering and behavior were more interesting.
Our new challenge is to keep a clear interaction with the dancer and choose another dynamic to enhance the
variety of behaviors and value the “dancing” aspect of the agents. It is a programming approach of
an aesthetic choice in the language of the rules of the ecosystem’s algorithmic way of working. This
translation from an aesthetic realm to a symbolic realm is our next step and our new challenge concerning meta
creation and symbiotic creation.
Conclusion and Perspectives
Adding unpredictability to the stage dynamics allows the dancer to improvise and discover new movements
motivated by the virtual system. Those movements are perceived by the system and modify it, establishing a
creative feedback loop. The project provides us with reflections on co-creation in a symbiotic way on many
levels. The stage manager performs to acquaint the system; the artistic director draws inspiration from the
behavior for staging ideas, and the dancer modifies and improvises according to the perception and reactions of
the system. At the internal level of the system, apparent associations between the agents form cohesive and
expressive volumes.
We have put in place since last year a system to enable co-creation, which could be used as a creative partner
in the process of performative dramaturgical creation (in contrast to the notion of a tool). In a metacreative
approach, we have designed a machine that generates evolving patterns and graphic behaviors. The creation
methodology schematized in Figure 1 (indirect interactions during the match-up phases) is the most critical
result. It makes it possible to adapt our team’s system and creative process to other contexts of
symbiotic metacreation for the performing arts.
Acknowledgments
This work benefited from the support of EUR ArTeC and from the funding assistance of the French government
managed by the National Research Agency under the future investment program bearing the reference ANR-17-EURE-
0008.
References
1 M. Baljko, and N. Tenhaaf, The Aesthetics of Emergence: Co-Constructed Interactions, ACM
Transactions on Computer-Human Interaction (TOCHI), 2008,15:11.
2 P. Cariani, Emergence and Artificial Life, In Langton, C. G.; C. Taylor, J. Doyne Farmer,
S. Rasmussen, eds., Artificial life II: proceedings of the Workshop on Artificial Life: held February 1990
in Santa Fe, New Mexico, Santa Fe Institute studies in the sciences of complexity proceedings.
Proceedings volume 10. Redwood City, Calif: Addison-Wesley. Meeting Name: Artificial Life Workshop, 1992.
3 E. Couchot, Automatisme,autonomieetesthe ́tique dans les arts vivants, In Corps en
scène: L’acteur face aux écrans, Paris, ENTRETEMPS ED, illustrated édition,
2018, 21–34.
4 J. Jones, Characteristics of pattern formation and evolution in approximations of physarum
transport networks. Artificial Life 16(2). Publisher: Massachusetts Institute of Technology Press
(MIT Press), 2010.
5 L. Margulis, R. Fester, eds. Symbiosis as a Source of Evolutionary Innovation:
Speciation and Morphogenesis, The MIT Press, 1991.
6 J. Mark Bishop, Y. J. Erden, Computational Creativity, Intelligence and Autonomy, Cognitive
Computation 4(3), 2012, 209–211.
7 H. R. Maturana, F. J. Varela, Autopoiesis and Cognition: The Realization of the
Living, Dordrecht, Holland, Boston, D. Reidel Publishing Company, 1st edition, 1980.
8 J. McCormack, Creative Ecosystems. In McCormack, J., and d’Inverno, M., eds.,
Computers and Creativity, Berlin, Heidelberg, Springer, 2012, 39–60.
9 C. W. Reynolds, Flocks, herds and schools: A distributed behavioral model,
In Proceedings of the 14th annual conference on Computer graphics and interactive techniques -
SIGGRAPH ’87, Not Known, ACM Press, 1987, 25–34.
10 C. W. Reynolds, Steering Behaviors For Autonomous Characters. In proceedings of Game
Developers Conference, San Jose, California: Miller Freeman Game Group, San Francisco, California,
1999, 763–782.
11 I. Teles de Castroe Costa, C.-Y. Chen, S. S. Circu, Artificial Life within a frame of
metacreation on stage, In Universitat Oberta de Catalunya., ed., ISEA2022, International Symposium on
Electronic Art, Barcelona. Proceedings. ISEA & UOC, 2022.
12 N. Tenhaaf, As Art Is Lifelike: Evolution, Art, and the Readymade, Leonardo 31(5),
1998, 397.
13 M.-H. Tramus, M. Bret, E. Couchot, La seconde interactivite, "In De l’art
cinétique à l’art numérique ; hommage à Frank Popper, number
106 in Collection Eidos, Série Retina, Paris, L’Harmattan, 2017, 179–192.
14 M. Whitelaw, Metacreation: Art and Artificial Life. Cambridge, MA, USA, MIT Press,
2004.
Bibliography
Chu-Yin Chen, “Autonomous System for Interactive Digital Art”, In 10th Generative Art Conference
Proceedings, Universite ́ Polytechnique de Milan, Italie, 2007.
Chu-Yin Chen, ”Un Monde pleinement e ́nacte ́”, in Action, Enaction - Emergence de
l’œuvre d’art, Xavier lambert (dir.), Editions L’Harmattan, Paris, 2017.
Steve Dixon, Digital Performance: A History of New Media in Theater, Dance, Performance Art, and
Installation, Edited by Roger F. Malina PhD and Sean Cubitt, Reprint edition, The MIT Press, 2015.
Simon Penny, Making Sense: Cognition, Computing, Art, and Embodiment, Cambridge, Massachusetts, MIT Press,
2017.
Marie-Hélène Tramus, Chu-Yin Chen, Judith Guez, Jean-François Jego, Dimitrios Batras,
Dominique Boutet, Marion Blondel, Fanny Catteau, Coralie Vincent, "Interaction gestuelle improviseé
avec un acteur virtuel dans un théâtre d’ombres bidimensionnelles ou au sein d’un
univers virtuel en relief : l’illusion d’un dialogue ?",In Stéréoscopie et Illusion,
M. Almiron, E. Jacopin, G. Pisano (dir.), Presses universitaires Septentrion, 2018.
Authors Biographies
Isadora Teles is a Brazilian artist based in Paris. She is currently a Ph.D. candidate within the INREV
research team at the University of Paris 8. Her artistic research and practice are in the fields of digital
generative art, artistic modeling of self-organizing systems, and the design of interactive interfaces for
live performances.
Chu-Yin Chen is an Artist and Professor in Digital Art, INREV research team at Paris 8 University. Her
creations, based on Artificial Life and complex systems, develop interaction modes between audience and
virtual creatures showing autonomous and evolving behaviors. Her digital artworks have been shown in numerous
international exhibitions. Her research articulates two overlapping areas: 1] Digital Creation using
algorithms of complexity and emergence, and 2] Metacognition and Elicitation of the processes of creation,
enaction and aesthetic reception, via psychophenomenology and mindfulness.
Hui-Ting Hong is an experimental artist from Taipei, Taiwan. Based in Paris. She focuses on the interaction
and transformation between the body, space, and mind. She aims to explore the existence and reproduction of
the human body by expanding the boundaries of perception. Her research-artistic practice focuses on the
perception of bodily movements and the hybrid human body, particularly discussing the virtual representation
of the body and its energetic variation.
1 Complexity Explorer - Introduction to Agent-based Modeling - Summer 2022,
https://www.complexityexplorer.org
The University of British Columbia, Kelowna, Canada miles.thorogood@ubc.ca,
director@artsco.ca, aleksandra.dulic@ubc.ca
Abstract
The ‘Light up Kelowna’ urban screen features a variety of digital artists who use technology to
push art in different and thought-provoking directions, tell our community’s vital stories and allow the
public to engage with the artwork in a multidimensional and multisensory way. In this paper, we outline the
structure for coordinating engaged parties in developing scalable urban screen infrastructure and considerations
necessary for installing rear projection urban screens in existing city spaces. We discuss the network
architectures and topologies for creating networked urban screen systems, borrowing concepts from networked
music performance and installation contexts. Finally, we demonstrate the use of our approach for developing an
urban screen, showcasing multiple exhibitions.
Keywords
urban screen, network topologies, Multi-Agent Systems
Illuminating public spaces change the texture of urban environments by activating under-utilized areas. While
simply turning on a light opens dark areas at night, media art projections add a layer of cultural engagement to
energize citizens and manifest different feelings and details of experience. Urban screen initiatives that
transcend the commercial use of displays toward media art are a medium for active public engagement and
discovery in urban settings. As a situated media, the cultural capacity of such screens provides a platform for
transforming people’s sense of participation and attachment in a place—providing an attraction for
the creation of activity and shaping people’s movement in urban spaces ⁵. As a platform to remediate
the city, urban screens focus on the relationship between the artists, technologies, spaces, and aesthetics
towards becoming integrated into people’s psyche. Dubois et al. ³ highlight key creative strategies
to address these relationships that include: framing the large-scale digital projection and logic of the
monument as expanded cinematic practice; engaging with technology art-making practices enable artists to create
new telepresent and telematics rituals and opportunities for urban activism and identification; and there is a
need to transform public screens into exhibition spaces.
Investigating the affordances and limitations of urban screens as communication infrastructure in a local
suburban area, Hannon ⁴ outlines how large urban screens operate in terms of their modalities
(pre-recorded, live, interactive) and functions (information provision, event support, screen events,
communication). Successful negotiation of the specific community needs becomes vital to realize the full
potential of such a screen.
In 2020, we recognized the need to bring our community together through art and create content that can be
enjoyed safely. Created in response to COVID-19 community needs, ‘Light Up Kelowna’ includes urban
screen infrastructure dedicated to digital art exhibitions. Presented at one of Kelowna key public spaces, this
program involves the community creating urban screen exhibitions for an accessible art experience in a public
space.
In the following sections, we outlined the structure for coordinating engaged parties in developing scalable
urban screen infrastructure. Next, we detail the considerations necessary for installing rear projection urban
screens in existing city spaces. Following, we go on to discuss the network architectures and topologies for
creating networked urban screen systems. Finally, we demonstrate the use of our approach for developing an urban
screen with the presentation of multiple exhibitions and present our conclusions.
Urban Screen Infrastructure Coordination
Developing urban screens infrastructure requires negotiation between a network of municipal, venue management,
funding, curatorial, and art community partners for possibilities transforming the locale. Struppek ⁷
highlights that content also needs to be coordinated in how, when, and in what specific locations screens can be
integrated with the urban land-scape and its architecture. It is this balance between content, location, and
type of screen that determines the success of the interaction with the audience and artists. The Surrey
UrbanScreen ⁶ project is one example of the successful collaboration between institutions and artists for
the realization of state-of-the-art urban screen infrastructure. For Surrey UrbanScreen, artists were critical
in steering the original concept of its form and functionality through equipment rebuilds and enhancements.
Light Up Kelowna closely aligns with this coordination structure for building the ongoing technological and
artistic capacity of vibrant public displays.
For ‘Light up Kelowna’, the Arts Council of the Okangan coordinated with the venue and City of
Kelowna to negotiate installation spaces, and faculty from The University of British Columba, Faculty of
Creative and Critical Studies (UBC) who have the expertise to develop the urban screen system and exhibition
curation. In addition, UBC generously funded the pilot project (Figure 2.). In our experience, the central
coordination structure of the network simplified the workload of engaged parties resulting in an efficient
turnaround from concept to realization. As a community-based project with support from publicly funded partners,
Light up Kelowna is an art-specific urban screen not requiring negotiation with private entities to interrupt
the cultural value with advertising, for example. The sole purpose of the urban screen as an exhibition space
then affords flexibility for programming artists’ work without needing to synchronize with other arranged
content.
Light Up Kelowna Urban Screen System
The Light Up Kelowna infrastructure is designed to inhabit underutilized surfaces appropriate for urban screen
installa tions. We focus on the rear projection of large windows with minimal functional value, such as
shopfronts, office buildings, and community venues. To modify such surfaces for rear projection, it is a simple
matter of applying a frosted vinyl film to the inside of the window and positioning a projector in an
appropriate location. The benefits of this approach are that it enables installation of the system within a day,
equipment is secured indoors, the minimal modification to the space makes it easier to negotiate installation
with partners, the vinyl allows daylight to enter the building, and it is cost-effective compared to other
solutions such as LED walls. Challenges with this approach are that projections will not work during daylight
hours, there may be interference with objects and activities inside the building with the projector image, and
the quality of the image is susceptible to ambient light conditions. When identifying a location, these
challenges must be considered. We used the following list of considerations to evaluate the urban screen
installation:
The pilot of Light Up Kelowna described here is located on three adjacent mostly square windows at the main
entrance of the Rotary Centre for the Arts (RCA) in Kelowna, BC, Canada—a multi-purpose venue designed to
accommodate special events, theatre, music and visual arts. We identified these windows as optimal due to: their
large scale; they face toward an open public space; the desirability of a triptych arrangement of the windows;
good mounting options for the projectors; covering the windows with vinyl has a minor impact on the feeling of
the inside space; and building activities have little interference with projector image.
LUK Software system
Each window in the Light Up Kelowna system at the RCA has a dedicated projector and computer running projection
mapping software written in OpenFrameworks. Similar to ⁹, a conductor program on one of the machines is
responsible for synchronizing media across the windows over a wireless local area network. Using a network
approach has the added affordance of machines being spaced over larger distances to synchronize media throughout
the city for the future scaling of the project across multiple venues and spaces.
As a systems approach, we borrow concepts from networked music theory. For example, Bevilacqua et al. ¹
discuss their work with Local Area Networks in response to the COVID crisis for sound installations and
performances. They develop infrastructure that allows an arbitrary number of clients to connect to the system
and allow diverse levels of autonomy for agents to influence the media composition. In their view, the network
is the key medium to support interaction at the level of the individual and collective, which forms the
expressive potential of the work. Weinberg ¹⁰ defines and classifies the aesthetic and technical
principles of such interconnected networks. Focusing on local area networks, he describes the centralized and
decentralized topologies. He depicts a centralized synchronous network as a flower topology, where nodes are
coordinated through a central hub. A decentralized network is then depicted as a star topology, where nodes are
connected and independently manipulate each other’s output. A third network combines these two with
weighted influence between nodes and a central server. Outlining architectures representing a flower topology,
Swift ⁸, and Brown ² describe systems that offer accessible and engaging group experiences by
affording audience participation.
Most representing a hybrid double-star topology, ‘The Light Up Kelowna’ system is designed around a
Conductor/Player/Artist relationship (Figure 4.). In this, a machine is designated as a Conductor who sends
instructions from a score to one or more machines in the role of Players that receive and execute instructions.
Players may communicate back to the Conductor, who can act upon this dialogue. An artist network node patches
into the system in order to control Players video plane compositions and sequence the Conductor. With the
‘Light Up Kelowna System’, Players may be autonomous computer-based agents or audience members. A
Conductor/Player topology may be explicit or ad-hoc, wherein in the former, a designer assigns the Conductor
role, while in the latter, the role of the Conductor is negotiated between the machines in the system.
The Conductor has in their knowledge base a list of Players in the orchestra that serves the functions of how
to sequence instructions and where to send these. The list of Players the Conductor manages may be static or
dynamic. In the case of a static list, the system operator specifies the arrangement of players and their
addresses to send instructions. Alternatively, a dynamic list is populated and edited by the Conductor as a task
of its operation. In the dynamic mode, a Conductor will receive a notification from a Player who has joined or
intends to leave the orchestra and will update the list accordingly. Another strategy is the Conductor can send
periodic pings to Players who reply to their presence, and the Conductor updates its knowledge base on these
responses. The ‘Light Up Kelowna’ system uses an explicit static Conductor/Player topology (outlined
in Table 1). Altogether, the result of this approach is a synchronized networked urban screen inhabiting
multiple collocated spaces open to audience/performer interactivity. In the following section, we describe some
of the exhibitions displayed for ‘Light Up Kelowna’.
Out in the Wild
‘Light Up Kelowna’ improves access and inclusion for our community through art. This initiative
adapts a public park and cultural area into a exhibit space, producing innovative and collaborative programming
through digital storytelling exhibitions. ‘Light Up Kelowna’ enhances access to art for all
community members with free digital art exhibitions in a public location. It involves and inspires the community
through a sense of place and creates a place for all our members to participate in an art experience.
‘Light Up Kelowna’ creates audience experiences that are a personal journey of exploration and
interaction, allowing visitors to social distance in a no-touch environment. This aligns with the requirements
of a pandemic world and the increasing use of technology to connect people. Art and technology are now
intertwined, and digital projections illuminate messages of hope, protest and community spirit.
Collective Bodies
The Collective Body (1) is a transdisciplinary audio/video project that proposes to capture the
circumstances of life in isolation during the COVID crisis. Using current communications technologies as a
central metaphor, the artists record themselves separately: dancers make videos of specific body parts, then
musicians respond with recordings, which are sent to other dancers, who then respond with videos, which are sent
to musicians, and the rhizome-like pattern of simultaneous emergence expands. The material is gathered, tagged
and shaped into a haunting, shimmering collage of diverse bodies projected onto the exterior of a public
building, with outdoor speakers providing sonic sensuality. In prioritizing visual and sonic expression over
linguistic communication, which breaks the world into distinct, individual entities, we offer ways of being with
others that simultaneously acknowledge differences and fundamental dependencies. The TCB project explores the
constraints and unique possibilities of the digital connection while serving as a reminder of the deep
importance and irreplaceability of shared physical space.
Celestial Bodies
Celestial Bodies (2) is a multicultural creation of animated media that tell ancient astrological
stories, exploring the belief systems that make up Canadian and Indigenous society’s diverse fabric. The
multimedia projection shows animated images of star stories— alongside world-class cross-cultural music.
Media Artists Aleksandra Dulic, Miles Thorogood, Jacen Dennis, Amberley John, and Emerald Holt created a
multimedia presentation of animated images. The artists reinterpret the cosmological stories and oral histories
from their cultural heritages — Greek, Chinese, African, and the Indigenous’ Haudenosaunee culture,
and explore the meeting of cultures in their collaborative process with community members, where unique stars
signifying individuals’ heritage were made. Each story is connected to a season, and characters from the
heavenly world travel through time and space as the night unfolds, highlighting diverse cultural beliefs.
Autolume Acedia
Autolume Acedia (3) is a hallucinatory meditation on the ancient emotion called acedia developed by
creative A.I. artists Philippe Pasquier and Jonas Kraasch. Acedia describes a mixture of contemplative apathy,
nervous nostalgia, and paralyzed angst. This emotion, first described by Greek monks two millennia ago, captures
the paradoxical state of being simultaneously bored and anxious. Inspired and controlled by the music of
Monobor, lost in winter soundscapes, the Autolume video generation system dreams about bodies, organs, and
bones. Autolume is listening to the music to produce abstract imagery that seems to be dancing. A product of the
latest Creative AI and Deep Learning algorithms from the Metacreation Lab, the piece is also a reflection on the
analog and the digital and how they can meet to portray this emotion that has resurged during the current
pandemic.
Student Works
As part of Light Up Kelowna’s art-dedicated urban screen projection in the Rotary Centre for the Arts
window, UBC graduate students showcase their work (4). The exhibit was developed in the context of
Graduate Studio in Visual Arts course that involves the critical analysis and production of independent artwork
in various disciplines.
Kaytlyn Barkved’s thesis show Neuroqueer Imaging features select digital drawings from her exploration of
the unique emotional and sensory perceptions that Autists experience. Kaytlyn is is a queer disabled digital
artist and Master of Arts candidate in the Digital Arts and Humanities theme of the Interdisciplinary Graduate
Studies program at UBCO.
Sam Neal’s Inland Waters, 2021, captures the exploration of time, place and process. Sam collaborates
with water bodies in the Okanagan using an early photographic process, cyanotype; a photographic process that
utilizes UV light to create cyan-blue prints. He is a multi-disciplinary photographer, artist and Master of Fine
Arts candidate at UBC.
Conclusion
‘Light Up Kelowna’ has successfully navigated the pilot phase with a triptych screen format using
rear projection techniques displaying media in three parts in a storytelling format. The project creates
opportunities for community members and audience participation and artist presentations. In this paper, we
outlined the structure for coordinating engaged parties in developing scalable urban screen infrastructure. We
outline several considerations necessary for installation of rear projection urban screens in existing city
spaces. We go on to discuss the network architectures and topologies for creating networked urban screen
systems, borrowing concepts from networked music performance and installation contexts. Finally, we demonstrate
the use of our approach for developing an urban screen showcasing multiple exhibitions. ‘Light Up
Kelowna’ is planned as a continuing, long-term exhibition series and is easily adapted to meet new
community needs and social/artistic directions in the future.
Acknowledgments
We acknowledge that this work is located on the unceeded territory of the Syilx Okanagan people in what is now
called Kelowna in British Columbia Canada. The Sylix people have lived and cared for this land for thousands of
years and continue to be stewards of this place. We feel that there is much to learn from their world view and
offer assistance to make a better future for all living things. We would like to thank, The University of
British Columbia, Faculty of Creative and Critical Studies, and The City of Kelowna for finacial support.
1 F. Bevilacqua, B. Matuszewski, G. Paine, N. Schnell, On designing, composing and performing
networked collective interactions, Organised Sound 26(3), 2021, 333–339.
2 A. R. Brown, Network jamming: distributed performance using generative music, New
Instruments for Musical Expression, 2010, 283–286.
3 J. Dubois, D. Colangelo, and C. Fortin, Disrupting the city: Using urban screens to
remediate public space, 2015.
4 S. Hannon, Urban renewal through media infrastructure: A case study of large screen development
in dandenong, melbourne, In Communicative Cities and Urban Space, Routledge, 2020, 131–146.
5 B. U. Lubis, L. Primasari, The relationship between people and urban screen in an urban space,
Procedia-Social and Behavioral Sciences, 2012,42:223–230.
6 A. Rajah, ed., Art After Dark: 10 Years of Urban-Screen, Surrey Art Gallery, 2021.
7 M. Struppek, Urban screens–the urbane potential of public screens for interaction, In
intelligent agent, volume 6, 2006, 1–5.
8 B. Swift, H. Gardner, A. Riddell, Engagement networks in social music-making.
In Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of
Australia on Computer-Human Interaction, 2010, 104–111.
9 M. Thorogood, M. Correia, A. Dulic, A networked multi-channel audio and video authoring and
display system for immersive recombinatory media installations, In Proceedings of the 27th International
Symposium on Electronic Art (ISEA), 2022.
10 G. Weinberg, Interconnected musical networks: Toward a theoretical framework, Computer
Music Journal 29(2), 2005, 23–39.
Author Biographies
Miles Thorogood is an assistant professor of digital art in the Faculty of Creative and Critical Studies and
heads the Sonic Production Intelligence Research and Applications Lab at The University of British Columbia.
His current research aims to identify the facets of human perception used in creative processes to develop
computational-assisted tools for art and design making.
Kirsteen McCulloch is the Executive Director of the Arts Council of the Central Okanagan. Born and raised in
the Okanagan, Kirsteen recently returned to Kelowna after many years of living and working in the technology
sector in Vancouver. Her love of all things creative, progressive, and boundary-breaking was fostered at an
early age. In her role with ARTSCO, Kirsteen is returning to her roots as an arts advocate and community
organizer.
Aleksandra Dulic is an artist-scholar with expertise in interactive art, climate change communication, and
media for social change. She is the Director of the Centre for Culture and Technology (CCT) at The University
of British Columbia. She leads an interdisciplinary research team that engages multiple forms of art, media
and information technologies as vehicles for the expression of community, culture, and identity.
Augmenting Creative Symbiosis Using a Cyber-Physical Aesthetic
iCinema Research Centre @ The University of New South Wales (UNSW), School of Built
Environment, UNSW, Murdoch University Kensington, Australia; Perth, Australia s.thurow@unsw.edu.au ;
d.delfavero@unsw.edu.au ; m.ostwald@unsw.edu.au ; h.grehan@murdoch.edu.au
Abstract
Creative arts organizations are constituted by intricate symbiotic relationships between diverse stakeholders,
including creatives, technicians, software programmers and producers. These thrive when facilitated by clear,
seamless and swift communication. Cyber-physical systems can support such exchange yet require careful aesthetic
design. Inspired by Manuel DeLanda’s philosophical framework, the paper introduces iModel, a new
networked modelling system that facilitates operatic rehearsal design via an interactive cyber-physical spatial
aesthetic. After mapping its conceptual framework, we detail iModel’s architecture and
functionalities that provide a shared workspace for distributed teams. iModel is capable of
assimilating diverse data formats and enabling real-time manipulation and seamless previsualization of complex
interactions between all components of operatic production, including orchestration of set ensembles, video
content, cast movements and lighting design, all synchronized with the musical score. The paper reflects on
iModel’s real-world application in 2022 in the design and technical production departments at
Opera Australia, Australia’s largest performing arts organization. We conclude with a reflection on the
coalescence of philosophy, art and technology as a powerful conduit for catalyzing creative practice toward
realizing new symbiotic imaginaries on and beyond the screen.
Keywords
Collaborative Design, Cyber-Physical Integration, Manuel DeLanda, Modelling System, Opera, Process
Optimization, Real-Time Interaction, Rehearsal Design, Spatial Aesthetics, 3D Visualization.
Organizations in the creative arts are constituted by intricate symbiotic relationships between diverse
stakeholders, including creatives, technicians, software programmers, producers and technology—who rely on
each other’s input to develop and realize cohesive creative endeavors. The Covid-19 pandemic has disrupted
these relationships by inhibiting the sharing of physical space, traditionally central to artistic practice.
This has challenged the sharing of ideas and creative visions through embodied exchange. Attempts to mitigate
the impact of lockdowns and their mandate of social distancing, have driven the accelerated adoption of newly
emergent digital technologies to facilitate collaboration across dispersed locations ¹ . This embrace of
cyber-physical engagement has induced a shift in the experience of space more broadly, initiating a new type of
virtual yet visceral immediacy, where formerly the significance of geographical distance dominated
¹⁸. However, the promise of approximating and expanding the utility of presence and seamless
interaction in virtual space still remains an aspiration whose full productive capabilities are yet to be
realized ¹⁶. The forced rapid transition to digital collaborative tools and platforms among
organizations from 2020 onward has painfully revealed the limitations that still exist in available hard- and
software systems when it comes to facilitating and optimizing creative exchange across time and space as well as
among diverse teams. This becomes obvious, for example, in lagging real-time collaboration features within
networked environments, inability to preview diverse file formats in a unified visualization space, or lack of
animation capabilities to orchestrate the intended interaction of separate design elements when coalesced into
complex assemblages. Due to such limitations, established approaches are still unable to optimally support
symbiotic relationships between technical and creative teams who rely on interdependent and reciprocal
cross-pollination in ideation and prototyping processes. Consequently, teams struggle to unlock new horizons for
collaboration and creative scope both in the virtual as well as in the—once again—accessible
physical domain. At a time when the digital is shaping expectations for sophisticated visual content in all
domains, creative sector organizations need to explore ways to enhance design ideation, production and
evaluation using hybrid cyber-physical systems ¹⁷. In a cyber-physical system, “computational
elements heavily interact with physical entities, thus controlling individual, organizational or mechanical
processes” ²² . To advance the dormant capabilities of digital technologies for expanding
creative collaboration in art and design practice as a symbiotic partnership, we believe it is necessary to
investigate interactive cyber-physical aesthetics as a conceptual foundation for new digital platforms. Such an
aesthetic has to conceive humans and technologies as equally ranked symbionts in order to be able to address the
current limitations in cyber-physically enabled creative collaboration. Approaches such as that developed by
Manuel DeLanda (1998) open avenues to transcend human-centered concepts by reformulating the world as a field of
informational and material processes from which an infinite number of entities may be formed. This philosophical
framework enables investigation of the diverse relations between people and technologies as malleable and open
to intervention in ways that enhance symbiotic exchange.
In the following, we briefly outline elements of DeLanda’s philosophical framework that have informed the
development of iModel, a networked modelling system that facilitates operatic rehearsal design via an
interactive cyber-physical spatial aesthetic. After mapping its conceptual framework, we detail
iModel’s architecture and functionalities that provide a shared workspace for distributed teams.
iModel is capable of assimilating diverse data formats and enabling real-time manipulation and seamless
previsualization of complex interactions between all components of operatic production including orchestration
of set ensembles, video content, cast movements and lighting design, all synchronized with the musical score.
This paper reflects on iModel’s real-world application in 2022 in the design and technical
production departments at Opera Australia (OA), Australia’s largest performing arts organization. The
paper concludes with a reflection on the coalescence of philosophy, art and technology as a powerful conduit for
catalyzing creative practice toward realizing new symbiotic imaginaries on and beyond the screen.
Conceptual Baseline: Manuel DeLanda
By drawing on Gilles Deleuze and Fe ́lix Guattari’s readings of the works of Baruch de Spinoza and
Henri Bergson, Manuel DeLanda develops a philosophical framework centered on the argument of ‘immanent
morphogenesis’ ⁹, roughly translated as the inherent ability of entities to self- organize their
development. By probing the question of both how matter comes to be and to assemble into innovative complex
forms (such as bodies and technologies), DeLanda develops a theory of self-organization ¹³ that is
premised on the existence of energetic flows, which are striving yet forever failing to reach an equilibrium
state ⁵. The inherent drivers, or aesthetic codes, underpinning these processes are ‘virtual
forms’ that trigger and galvanize evolutions of matter into ‘actual forms’ that present as
phenomena to the human senses. As codes rather than blueprints, virtual forms do not predetermine the specific
articulations of phenomena but constitute an ‘energetic possibility space’ that is radically open to
intervention and evolution ². With flows being subject to heterogenous and conflicting influences, they are
compelled by imbalances that call for harmonizing. Citing Deleuze, DeLanda (1998) captures these imbalances as
caused by “differences of level, temperature, pressure, tension, potential, difference of
intensity”, and increase energetic flow relative to their degree of disparity. Matter here exists in a
“phase space, i.e., a space of possible states for each object” ³—with a ‘stable
state’ representing the preferred state in which an object is maximally harmonized. DeLanda argues that,
in the chaotic swirling of energetic flows, approximation of stable states is achieved more readily (but never
fully realized) through coupling of simple forms into higher entities. This is because coupling of forms can
cater to a set of dispositions shared among these forms, and represent an energetically more resilient, hence
favorable, state. DeLanda conceives states along a continuum from ‘preferred stable states’, which
he refers to as “attractors”, to least probable and volatile couplings, which he terms
“repellors” ¹⁰. Attractors are thereby capable of generating new dynamic structures in
higher entities through interaction of constituent parts, refracting energetic flows onto new trajectories that
open capabilities for novel constellations with new, and ever more complex, balancing requirements. DeLanda
captures such branching and its dynamic mutational qualities in the term ‘bifurcation’ ⁴. As
Christian Hubert explains “bifurcations modify an entity’s tendencies and represent a source of
creativity and variability in nature” ¹⁴. Once forms approach a stable state, they solidify
into entities that “down-wardly select states of [their] component units that are [most] compatible with
that global state” ¹¹. This embarking on an iterative design process reassembles and optimizes
the cohesiveness of the higher entity ¹¹. DeLanda emphasizes that complex higher entities always
unfold capabilities that exceed the limitations of their simple individual component parts ¹². He
argues that these immanent aesthetic codes that govern the self-organizing production of matter (that is, its
morphogenesis) underpin both organic and inorganic life ⁶. Consequently, he deduces that on a formative
level, humans and technologies must be conceived as morphologically alike because they are part of an integrated
“machinic phylum” (lineage or body plan) that synthesizes elements according to “similar
self-organizing and combinatorial processes” ⁴. DeLanda here adopts terminology of Deleuze and
Guattari who use the qualifier ‘machinic’ with the intent to highlight “the existence of
processes [virtual forms] that act on an initial set of merely coexisting, heterogeneous elements, and cause
them to come together and consolidate into a novel entity” ⁴. DeLanda refers to this consolidation
as bifurcation that happens at the point of ‘singularity’, that is a tipping point at which new
characteristics emerge that “chang[e] the forms through which human bodies and materials are combined,
organized, deployed, and made effective” ⁷. He emphasizes that bifurcations are most productive if
coupling occurs between forms that are heterogenous because this enables a diversity of possible trajectories
into the future ⁴. It can give rise to processes that act as catalysts, intervening into processes and
facilitating encounters between components that would otherwise not have occurred, channeling energy onto
unexpected trajectories.
Application: iModel System
DeLanda’s philosophy commends itself as a conceptual framework for creative
collaboration, and especially in the operatic sector, because it allows conceptualization of stake-holders,
ideas, technologies and materials as an interdependent web of symbionts that coalesce into an ecology whose
scope expands or contracts in line with the degree of flow. As a group-based aesthetic practice, opera involves
a spectrum of digital and physical components and engages an ensemble of creatives at a larger scale than other
art forms. For example, components flowing into a production may include choreography, costumes, lighting,
music, props, scenography, robotic LED screens and revolves – all developed by domain-specific artists
(graphics, modelling, photography, robotics, video), designers (costume, lighting, production, prop), directors
(musical, stage, technical), and implemented by diverse technical crew, cast and orchestra. As such, operatic
production constitutes a symbiotic ecology in which each craft depends on the communication with and input of
others to realize its purpose and to thrive, through reciprocal loops of collaboration, into a new higher
entity, in this case the stage production ²¹. Conventional rehearsal design sees most creative team
members develop their ideas in response to a production brief, mapping ideas in isolation, using incompatible
analogue/digital platforms ¹⁹. The team then discusses concepts in development meetings, and makes
amendments following these, again in relative seclusion. The combination of single ideas being transformed into
complex assemblages that may yield a cohesive aesthetic for a production (approximating a Delandian stable
state) hinges on the facilitation of productive communication among the diverse team. We argue that the more
sensorily seamless this communication, the more readily attractors may emerge that enhance creative scope and
complexity of design. The ability to combine heterogenous elements and design strands at various stages in the
ideation, testing and evaluation phases can catalyze a process of cross-pollination and analysis that far
exceeds capabilities of current stand-alone creative pipelines. If draft components can be assembled in
fruitful, ongoing dialogue, bifurcations—resulting from conceptual and experimental tensions—can be
identified and explored at length. Their resultant trajectories can feed into iterative design loops that may
crystallize into more coherent and innovative operatic designs. Prerequisites for a system that can prompt
creative singularities and act as a catalyst for the exploration of bifurcations, are:
an interaction design that accommodates the diverse needs of the creative and technical teams;
a visualization architecture that delivers a shared environment, in which the input by all user groups can
be synched, reviewed and adapted in real time.
We believe that additional synergies may be unlocked if such a system can be articulated to the human body in
ways that enable a visceral investigation of the modelling space because human sense-making capabilities would
be activated that exceed a solely visual contemplation.
Responding to this brief, an interdisciplinary
team of artists, computer scientists, architects, theatre and 3D modelling experts at The University of New
South Wales, led by Laureate Professor Dennis Del Favero, has been developing the iModel prototyping
system in collaboration with OA. Their objective has been to probe in how far interactive spatial aesthetics and
associated new technologies may support the collaborative design process as a self-organizing, positive
symbiotic exchange between institutional departments and across the human-machine interface. By seamlessly
integrating creative ideation and production pipelines via cyber-physical architectures, the project seeks to
provide solutions for unlocking the dormant potential of OA’s strong investment into digital
innovation—palpable, for example, in its pervasive use of large-scale modular LED screen assemblages.
These require safe orchestration with physical props and cast as well as complex multi-screen mapping,
which—in turn— intensifies the organization’s need for a powerful integrative pipeline that
can increase the rapport between their teams and the available technological platforms. To address these
challenges, the team behind iModel has been developing a cyber-physical spatial aesthetic that
facilitates the modelling of an operatic design within a shared 3D virtual environment by means of dynamic joint
human-machine decision making. It provides a creative ensemble with the ability to develop and evaluate design
components in real time prior to stage rehearsal across diverse platforms, ranging from desktop computers and
tablets to 360° 3D cinematic CAVE theatres ¹⁵. Using a dynamic visualization pipeline, team
members can upload their domain-specific components into a shared modelling space where they can virtually
validate designs before physically testing them on stage. This is facilitated through capabilities such as
real-time collaborative composition and manipulation of components; playback of component operation, both
individually and in combination, in time with the musical score; as well as AI-supported analysis of component
composition and interaction, including collision, obstruction and line-of-sight detection. The categories of
design information that can be included are: choreography, costumes, set designs (3D models and CAD files),
directorial notes, lighting states and truss plans, musical score, stage robotics, props and screen content. To
coalesce this information into a database from which unified visualization can be achieved, the team has
scripted an asset conversion pipeline that uses the Nginx webserver in combination with the
Celery queuing software to translate files on a static webserver from their native formats (for
example, .json, .dwg or .fbx) into those readable by the iModel application. This is a prerequisite for
visualizing the diverse data in a shared virtual space facilitated by the Unity 3D game engine. Once
complete, asset bundles can be downloaded via the Django web app and imported into the iModel
application onto a personal computer. This online pipeline leverages existing solutions and saves local
computing power, enhancing the overall robustness and performance of the application. Drafting within iModel
takes place within a 3D modelled theatre space, which can be either a generic venue or a digitally twinned
real-world performance space. In the case of OA’s 2022 use of iModel for their touring production
of The Barber of Seville, the twinned space is a meticulous reconstruction of Sydney Opera
House’s Joan Sutherland Theatre and (separately) of the Drum Theatre in the city of Dandenong. The
detail-rich 3D models reflect their respective architectural specifications, available stage machinery as well
as textures of walls, floors and auditorium—providing the team with an accurate rendition of the
performance space within which the final staging of the opera took place. This allowed closely customizing
ideation and prototyping to the available real-world conditions of the theatre space. If accessed via a
cinematic-scale CAVE theatre, the 3D model can be explored at life-size and provide a sense of inhabitable space
during the design process—adding a visceral dimension that conventional approaches, for example physical
scale models, cannot offer in terms of atmospheric assessment ²⁰. The iModel interface
features separate workspaces that can be customized to the needs of various user groups, such as directors, set
and lighting designers, or stage managers. Each workspace comprises functionalities that are particularly useful
to a user group, reflecting common tasks and information needs. For example, a set designer may introduce a
setting, center or grid lines over the stage view, specify the speed and rotation of a revolve, and import,
manipulate, rotate and scale objects, such as set pieces or props. Their placement can be specified on a
timeline (the so-called ‘sequencer’) that is coupled to the musical score, allowing users to preview
an animation of their movement at any time. They can record or screenshot for later reference if they want to
compare different versions and their possible creative bifurcations. The sequencer is keyframed to allow for
greater ease when editing, allowing to jump for example between scenes and acts, or from an overture into an
aria or obbligato. Content added to workspaces is saved to the same project within the iModel
application, which enables the combined review of all workspaces in a master window that layers all saved
content onto the common timeline and thus enables a virtual stage rehearsal. For closer inspection, users can
pause, forward, rewind or zoom in and out of the sequencer. Remote collaboration between geographically
dispersed users will be streamlined through an annotation feature that allows input as text, drawing or audio
commentary linked to customizable keyframes in the timeline. While this facilitates design engagement for the
entire creative and technical team, iModel offers artificially intelligent features that support
evaluation in terms of occlusions, potential hazards, path tracing and wayfinding for movable objects and cast
members.
During the development of OA’s Barber production, the iModel system enabled the
creative and technical teams to achieve significant synergies and efficiencies across the entire design
pipeline, enabling seamless visual communication especially in relation to the complex stage movements of props,
set pieces and cast members. As Barber was conceived as a touring production for over 20 locations
across Australia—ranging from expansive, formidably equipped to very minimalist venues-, a design had to
be delivered that would aesthetically resonate across locales as well as pragmatically pack down for quick venue
changeover—all to be achieved within tight budgetary constraints in the aftermath of the Covid-19
lockdowns. The solution to these challenges was found in an intricate modular set design, whose elements cast
members would deconstruct and maneuver across the stage in choreographed sequences, aided by locally recruited
choruses that had to be inducted on the fly once the crew arrived at a new location. The set elements required
custom-built travelling cases, which were also designed using the iModel application as it allowed
precise articulation of dimensions and formats. The iModel system provided the much-needed capability
to virtually trial and evaluate different draft designs across three destination theatres whose spatial and
technical resources required strongly divergent design provisions. Through visceral exploration of the
3D-rendered architectural spaces as well as visual manifestion of designs at 1:1 scale, the teams were able to
rapidly achieve a shared understanding of possibilities and constraints and to direct their mental resources
more readily onto feasible design trajectories. Rather than getting caught up in lengthy verbal or textual
explanations of spatial concepts, team members used the iModel system to rapidly block out space and to
trace movements of set pieces and bodies across the stage. This allowed identifying bottlenecks and preempting
collision pathways, as well as adding playful complexity to the orchestration of the swiveling stage action that
closely integrated with the musical score—amplifying its tumultuous and joyous pace. With costume designs
ingested into a workspace, each character could be displayed in at least one attire. The designs were ingested
as .png files (i.e. 2D images) attached to 3D boxes that would provide them with depth in the 3D space.
Raytracing allows light to bounce off them and cast shadows, which aides in integrating the avatars more fully
into the visual concept.
Grouping the characters together on stage, and animating their movements synched to the orchestration of set
pieces, enabled assessment of their aesthetic effect. In response, textures were refined and details added that
enhanced the overall composition and which imbued characters with idiosyncratic flair. Most importantly, the
iModel system facilitated maximally clear communication of ideas and enabled rapid evaluation and
iteration of concepts. The team thereby made full use of the range of visualization platforms, starting out on
desktop computers connected to large wall-screens for initial prototyping, importing designs into a 360-degree
cinematic CAVE theatre for 1:1-scale team review, and lastly using the system on laptops in the rehearsal room
to annotate and refine design aspects that could be shared with cast and crew. With pandemic disruptions
continuing to affect the availability of personnel and resources, iModel provided capability to convey
complex design ideas across time and space, allowing remotely located team members to review and annotate files
on their personal devices, which helped to stay on schedule for delivery.
Conclusion
Through its provision of a shared environment, visual integration of design streams and real-time interaction,
iModel has proven its capability to productively intervene into design processes at OA,
streamlining—while at the same time—enhancing complexity and creative scope of output. It does so by
supplying a catalyst that enables creative bifurcations through the coupling of human creativity and
cyber-physical computational capabilities. It supports the ideation and analysis in ways that do not prescribe
form but enable self-organization of processes with open outcomes. In doing so, it facilitates a redrawing of
the relationship between makers and technologies, opening pathways for conjunctive symbiotic exchange not just
between creatives but also between humans and machines. With ubiquitous digital technologies increasingly
reassembling workflows and imaginations, such a partnership—if facilitated in the right ways—can
yield a creative force that deeply fuses human, computational and mechanical systems ⁸. The
crosspollination between the arts and technology can thereby furnace new modes of engaging the machinic phylum
that open up multiple bifurcations for imagining and shaping our planetary futures as collective and symbiotic
rather than individual and isolated entities.
Acknowledgements
This research has been supported by the Australian Government through the Australian Research Council’s
Linkage funding scheme (LP190100563).
References
1 Australia Council for the Arts, Digital Culture Strategy 2021-24, Canberra: Federal
Government of Australia, 2020, p.3.
2 M. Delanda, and C. Cox, Possibility spaces. in c. cox, j. jaskey and s. malik (eds.),
Realism Materialism Art Berlin: Sternberg Press, 2015, p.88.
3 M. Delanda, G. Harman, The Rise of Realism. Cambridge: Polity Press, 2017, p.73.
4 M. Delanda, 1997, The machinic phylum, TechnoMorphica, available at V2, accessed August 25,
2023, https://v2.nl/archive/articles/the-machinic-phylum.
5 M. Delanda, 1998a, Deleuze and the genesis of form, Art Orbit 1 March, accessed August 25, 2023.
https://www.cddc.vt.edu/host/DeLanda/pages/genesis.htm.
6 M. Delanda, War in the Age of Intelligent Machines, New York, Forth edition, Zone
Books, 1998b, 7.
7 M. Delanda, War in the Age of Intelligent Machines, New York, Forth edition, Zone
Books, 1998c, p.Backmatter.
8 M. Delanda, War in the Age of Intelligent Machines, New York, Forth edition, Zone
Books, 1998d, 231.
9 M. Delanda, Deleuze, diagrams, and the genesis of form, American Studies, 2000,
45.1:34.
10 M. Delanda, MaterialistPhenomenology.A Philosophy of Perception, London, Bloomsbury,
2022a, 23.
11 M. Delanda, Materialist Phenomenology. A Philosophy of Perception, London, Bloomsbury,
2022b, 36.
12 M. Delanda, Materialist Phenomenology. A Philosophy of Perception, London, Bloomsbury,
2022c, 29.
13 R. Dolphijn, I. Van der Tuin, eds, New Materialism: Interviews and Cartographies,
Ann Arbor, Open Humanities Press, 2012, 43.
14 C. Hubert, 2019, Machinic phylum, tian Hubert Studio, accessed August 25,
https://www.christianhubert.com/writing/machinc-phylum, Chris-2023.
15 L. Maftei, C. Harty, Designing in caves. using immersive visualisation in design
practice, International Journal of Architectural Research, 2015,9.3:73.
16 G. Y. N. O’Dwyer, A. Smolic, Xr ulysses: Addressing the disappointment of cancelled
site-specific re-enactments of joycean literary cultural heritage on bloomsday, International Journal of
Performance Arts and Digital Media, 2022, 18.1:43.
17 H. M. Nugent, A. McKinnon, K. Fagg, M. Oxenbould, 2016, National opera review final report,
Department of Communications and the Arts, Canberra: Commonwealth of Australia, accessed August 28, 2023, p.5,
https://shorturl.at/dewB5.
18 S. Reddy, 2021, Designing a digital future, The New York Times website, accessed August 25,
2023. https://www.nytimes.com/2021/12/03/special-series/covid-technology-digital-space.html.
19 B. Simonson, The Art of Rehearsal, ebook, Bloomsbury, 2017, p.6.
20 S. Thurow, D. Del Favero, L. Wallen, Inhabitable models – immersive intelligent
aesthetics for scenographic design, Theatre and Performance Design, 2021,7:92.
21 P. Torpey, Media Scores: A Framework for Composing the Modern-Day Gesamtkunstwerk,
Cambridge, PhD Thesis, Massachusetts Institute of Technology, 2023, p.17.
22 C. Tsigkanos, T. Kehrer, C. Ghezzi, Architecting dynamic cyber-physical spaces, Computing,
2016, 98:1012.
(Re)imagining human-yeasts relations via art-science collaboration
OlgaTimurgalieva1, PatríciaMoreira2, EvaDireito2
1 City University of Hong Kong, 2 Catholic University of Portugal
Hong Kong, Lisbon otimurgal2-c@my.cityu.edu.hk, prmoreira@ucp.pt, s-edireito@ucp.pt
Abstract
More-than-human beings are largely de-animated in Western cultures and perceived of as the backdrop for human
activities. In particular, with modern modes of production and consumption, customers often have no idea how
specific products have been created and which more-than-human beings have been involved in manufacturing
specific consumables. As a partial response to this problematic disconnect, this text presents the art book,
Yeasts as We Do Not Know Them, as a means to learn about human-yeast interactions and the ways these
fungal microbes are used to manufacture different products and substances. The book, therefore, (re)imagines
yeasts as omnipresent, diverse, and symbiotic. Conceiving of symbiosis as a set of interspecies relations,
including mutualistic, pathogenic, and commensal ones, the project, Yeasts as We Do Not Know Them, as
this article argues, maps diverse interspecies interactions and, by doing so, provides recourse for navigating
material systems, thereby invoking trans-corporeal ethics.
Contemporary modern production and consumption practices are based on the general compartmentalization of
economy, culture, and nature. ¹ That is, despite ecological concerns, diverse forms of life are largely
de-animated and instrumentalized in Western cultures. ² While consumption and waste production levels in
many regions are incredibly high despite myriad alarming environmental issues, customers are often unaware of
the sources of specific products and from which living and non-living materials they are made. In response to
such circumstances, scholars from diverse disciplines tackle the problematics of climate change, with a rising
number of researchers emphasizing the role of art in developing ecological sensibilities, aesthetics, and
imaginaries. ³ , ⁴ Such views derive from the assumption that climate change issues arise from
and influence diverse aspects of human and nonhuman lives and cannot be tackled merely within the environmental
sciences and adjacent fields. Rather, as researchers in environmental humanities Astrida Niemanis, Cecilia
Åsberg, and Johan Hedrén have pointed out, “any policy or action aimed at ameliorating
environmental problems must take into account human desire, motivation, and values; a deep understanding of the
environment cannot be divorced from human imagination, culture, and institutional and social
practices.” ⁵ In other words, dealing with climate crises cannot be limited merely to
environmental sciences. Instead, in order to face immense challenges, significant changes are needed within
distinct yet deeply interconnected economic, industrial, cultural, social, institutional, political, conceptual,
visual, and other realms. Hence, instead of compartmentalization, environmental humanities require
transdisciplinary methodologies and practices.
For these reasons, art book, Yeasts as We Do Not Know Them, which is the focus of this paper, rejects
compartmentalization and instead takes the concepts of symbiosis and trans-corporeality as a starting point,
mapping out some of the yeast interspecies relations. Created by the authors of this article, Yeasts as
We Do Not Know Them depicts human-yeast interactions as diverse, omnipresent, and symbiotic. We use the
concept of symbiosis to describe human-yeast relations as sometimes mutualistic, sometimes commensal, and
sometimes even pathogenic. Trans-corporeality, according to American literary scholar Stacy Alaimo, is a mode of
thinking that regards beings and entities as porous and materially and discursively interrelated. ⁶ By
presenting ways in which yeasts are interconnected with diverse life forms and are involved in the industrial
production of food, medicine, chemicals, and cosmetics, the book invites the audiences to learn about
human-yeast material interchanges and to consider the repercussions of these interchanges. In mapping some of
the actual interspecies interactions, our project, as we argue in this text, invokes trans-corporeal ethics.
In order to develop this argument, we begin by discussing symbiosis and trans-corporeality as guiding concepts
in the project. Further, we elaborate on the connection of the art book to these concepts via two main threads.
First, we trace yeasts across scales on the level of bodies and in the production of various products and
consumables. Second, we emphasize the importance of transdisciplinarity in creating this artistic publication.
Symbiosis and trans-corporeality
Nowadays, most biologists define biological symbiosis as a close (in direct contact) relationship between
individuals of two species, either with positive, negative, or neutral outcomes. ⁷ According to this
definition, parasitism, mutualism, and commensalism are all symbiotic relations. While several generations of
scientists have widely debated this notion of symbiosis, currently, the understanding of symbiosis as
interspecies relationships of both mutualist and non-mutualist character is, for the most part, accepted and
widely circulated via undergraduate biology books.
One of the most prominent and well-studied examples of mutualistic symbiosis is the two-species association
with lichens, which was first described in the 19th century. Lichens display a win-win relationship with
filamentous fungi and either cyanobacteria or microalgae. The fungi species benefit from photosynthesis products
such as simple sugars produced by cyanobacteria or microalgae. In turn, the latter grow, embedded in a mycelium
structure, which protects them from dehydration and environmental stress. Another well-studied example of
mutually beneficial relationships between two distinct species is the cooperation between mycorrhizal fungi and
plants. In their interactions, plants fix carbon from the air through photosynthesis and provide the fungi with
simple sugars. In response, the fungi help the plants extract minerals from the soil.
The notion of symbiosis is closely connected with symbiogenesis or endosymbiotic evolutionary theory. This
theory explains the origin of more complex cells—eukaryotic cells—which constitute the bodies of
animals, plants, and fungi. ⁸ Endosymbiosis describes the relationship between two species where one
organism lives inside the cell/cells (either for unicellular or multicellular organisms) of another. The theory
suggests that, between 2500 and 1800 million years ago, a bigger cell engulfed a prokaryotic one (bacteria) and,
instead of processing it as food, integrated the prokaryote into itself. In the process of incorporating the
prokaryote, the bigger cell gained the ability to generate energy thanks to prokaryotic mitochondria and perform
photosynthesis thanks to prokaryotic chloroplasts. Thus, instead of consuming the prokaryote, the host cell
negotiated a functional relationship of symbiosis. Present-day examples of endosymbiosis include paramecium
Paramecia bursaria, commonly known as infusoria, a single-celled freshwater being covered in cilia.
Paramecia bursaria harbors several hundred single-celled green algae Chlorella spp. cells in
the cytoplasm that perform photosynthesis for simple sugar production. ⁹
In recent years, with the development of ecological thought, ecoart, bioart, and environmental humanities, the
concepts of symbiogenesis and symbiosis have traveled far beyond biological sciences. As the discourses
concerning climate crises and mass species extinctions alarmingly stress the catastrophic devastation
facilitated by climatic disasters, artists and scholars in the humanities broadly employ the notions of
symbiogenesis and symbiosis in their respective work. They increasingly emphasize interrelations and cooperation
between diverse life forms as opposed to individualistic perspectives and conceiving of bodies as separate from
the environment.
Perhaps one of the most prominent examples of engaging with symbiogenesis theory in the humanities is the work
of technoscience scholar Donna Haraway. Drawing on the biological concept of endosymbiosis, Haraway has
highlighted the interconnectedness of diverse organisms and entities. ¹⁰ Instead of considering
living beings as bounded “self-organizing individual units,” the author stresses their
“multispecies becoming-with” and their constant mutual co-influences on each other. ¹¹,
¹² Haraway regards such interrelations and interdependent sensibilities as crucial for
establishing more ecologically-minded practices of living on a damaged planet.
Practitioners and academics in the visual arts also refer widely to the concepts of symbiosis and symbiogenesis
to emphasize the constant interrelations between living beings. As artist and writer Claire Pentecost puts it,
multiple contemporary artists create “symbiotic art”; that is, they expose the interrelations
between diverse life forms and, in this way, help us to recognize “a world of relations” and our
dependencies on manifold living beings. ¹³ When talking about creative projects developed using
microorganisms, artist Ken Rinaldo emphasizes their “symbiotic aesthetics” as the means to draw the
attention of broader audiences to the complexity of natural living systems and their impact on environmental
health. ¹⁴ Overall, numerous researchers in the visual arts and (post)humanities engage with the
concepts of symbiogenesis and symbiosis as making-with instead of considering bodies and entities bounded.
¹⁵
Similarly, Stacy Alaimo regards bodies not as contained but rather as trans-corporeal, that is, porous, not
existing as preceding entities but constantly co-influencing each other. ¹⁶ As Alaimo argues, such a
relational consideration of beings and entities requires a different kind of ethics—a trans-corporeal
ethics. Indeed, trans-corporeal ethics arises from the obligations to “inquire about all of the substances
that surround us, those for which we may be somewhat responsible, those that may harm us, those that may harm
others, and those that we suspect we do not know enough about.” ¹⁷ As the author posits
further, this approach to ethics “calls us to somehow find ways of navigating through the simultaneously
material, economic, and cultural systems that are so harmful to the living world and yet so difficult to contest
or transform.” ¹⁸ In other words, considering things, bodies, and entities as porous and
permanently affected by their symbionts and co-existing materialities requires understanding the material
interchanges across beings and entities and, if possible, lessening the impact of harming interchanges.
Informed by these conceptualizations of symbiosis, symbiogenesis, and trans-corporeality, Yeasts as We Do
Not Know Them engages with them on various levels: 1. the symbiotic natures of the yeasts co-habiting
with and within other living beings; 2. in more metaphorical terms, human-yeast “symbiotic”
relations in the production of various products and consumables and, finally, 3. the interdisciplinary
“symbiotic” collaboration between a biotechnological engineer, designer, and art researcher aimed at
creating the project. Depicting human-yeast relations as symbiotic, the book aims to map material interchanges
across scales and species in the spirit of trans- corporeality, thereby demonstrating how yeasts interact with
humans across multiple scales.
Mapping yeasts across scales
Yeasts—unicellular organisms that belong to the Fungi kingdom—play a vital role in the
Earth’s biosphere. While scientists have described around 100,000 fungi, they estimate that there are 1.5
million different species of fungi in total, in which only around 1 percent might be yeasts. As of now,
researchers have described around 1,500 yeast species and found that these microbial fungi exist in every single
habitat and ecosystem: from terrestrial to aquatic ones and from tropical ecosystems to more extreme niches such
as Antarctica, where the fungi present specific adaptations to diverse environmental factors and ecosystem
dynamics. Yeasts play an essential role in those habitats, being involved in, among other processes, the
decomposition and recycling of living and non-living matter. At the same time, some of them are related to
diseases in plants, animals, and even tiny protists such as paramecium, widely known as ameba.
Single-celled yeasts require magnification to be visible to the naked eye if not grouped in colonies. In
response to their general invisibility to humans, the book, Yeasts as We Do Not Know Them,
helps to imagine their omnipresence and diversity. It conceives of bodies and entities as
microbial, presenting material entanglements between yeasts and humans and several examples of yeast
relations with plants, animals, and other microbes. ¹⁹ For instance, Candida albicans and
Candida auris are widely known for being related to human and animal infections. Cryptococcus
neoformans and Exophiala dermatitidis are less known to a broader audience but related to
invasive infections (Figure 1). So-called “wild” yeasts live in every environmental biome of the
planet, including oceans, the stratosphere, glacial ice, bodies of animals, plants, fungi, algae, and
microorganisms. Although not necessarily depicting the tiny fungal cells, the publication locates schematically
yeasts’ symbiotic relations, including commensal, opportunistic, and pathogenic ones.
Additionally, the printed collection displays various examples of human-yeast interactions involved in the
manufacturing of numerous consumables. The immense and diverse metabolic powers within yeasts and their presence
in all habitats and regions have led humans in various geographical locations to unknowingly use yeasts in
fermentation to produce beer, bread, kombucha, soy sauce, and other food and beverages (Figure 2). These
practices have been passed down from generation to generation through millennia. In contemporary societies,
however, such ancestral uses of biotechnology have been significantly optimized, while the fundamentals of
producing some of the most common drinks and snacks remain the same. Yeasts as We Do Not Know Them
features several historical records of the use of yeast and numerous contemporary examples.
In recent centuries, scientific developments in biotechnology have permitted scientists to expand the metabolic
toolkit of yeasts further and apply them in the manufacture of complex products such as pharmaceutical
compounds, biofuels, cosmetics, and other chemicals, as well as bioremediation solutions. Additionally, yeasts
are widely used in pharmaceutical research and experiments to produce a host of chemical substances and
mixtures. Yeasts as We Do Not Know Them provides numerous examples of such products.
Although the book maps interspecies yeast relations across the scales of micro- and macroscopic bodies and the
diverse ways in which yeasts are used in the production of numerous consumables, we do not intend to collect all
the instances of human-yeast interactions with all the known microbial fungi. Instead, the art book functions as
an invitation to learn more about some of these microbes and their interspecies material interchanges.
We do not regard this project as a mere celebration of human-yeast entanglements. We rather seek to encourage
the readers to consider the impact of these interrelations and whether or not they can be harmful in some way.
As geographer and sociologist Bram Büscher has pointed out, some aspects of human-nonhuman
interactions involved in consumables production might harm ecological systems or be associated with social
inequalities. For this reason, such manufacturing practices need to be revised or even “unmade,”
argues the author.²⁰, ²¹ Yeasts as We Do Not Know Them displays various
examples of yeast applications and, by doing so, invites the viewers to contemplate which of the human-yeast
relations might need to be remade or unmade. In this way, the project invokes trans-corporeal ethics.
Transdisciplinary synergies
The idea of Yeasts as We Do Not Know Them has grown from a transdisciplinary course called
“Gender and Sustainability–Introducing Feminist Environmental Humanities” offered by the KTH
Royal Institute of Technology and Linköping University. This course gathered artists, scientists,
architects, anthropologists, philosophers, and researchers in literary studies, art criticism, ecology,
sustainability, and feminist and gender studies. Course participants discussed numerous complex challenges of
the global ecological crises. They studied some of the theoretical instruments and methodologies that can be
applied in dealing with global environmental issues.
In the spirit of the course, Yeasts as We Do Not Know Them, functions as a form of trans-disciplinary
practice-oriented inquiry. More specifically, the book was developed by a biotechnologist Patrícia
Moreira, a designer Eva Direito, and a researcher in bioart and environmental humanities Olga Timurgalieva. The
cross-pollination between the three disciplines has allowed us to gather crucial information about the
microorganisms and their application in biotechnological research and industrial production (microbiology),
establish a conceptual framework to engage with interspecies relations creatively (bioart and environmental
humanities), and, most importantly, to develop and apply the visual means with which to communicate our ideas
(design). Combining this diverse disciplinary knowledge permitted us to learn about human-yeast ecologies as
complex interspecies relations and translate what we learned into an artistic project.
As a product of transdisciplinary collaboration, Yeasts as We Do Not Know Them became a material
embodiment of “symbiotic” relations between art and science. The book’s visuals and content
reflect the interplay between the disciplines and the typical clichés associated with them. The project
alludes visually to the style of a modern encyclopedia and simultaneously to a catalog of products for sale. As
the primary reference for developing the images and graphics for our collection, we looked at the book, Pig
05049 (2007), which was created by artist Christien Meindertsma and designed by Julie Joliat.
Pig 05049 compiles dozens of everyday consumables produced from a single source—a pig numbered
05049. ²² The collection of consumables is divided into chapters, each devoted to a specific body part
or bodily substance of the pig from which the products were made such as blood, skin, bones, fat, and so forth.
Each product page features a photo illustration, a brief description, and an icon that refers to a type of
consumable. Icons and chapters structure the book, thus referring to indexing conventions and highly organized
scientific knowledge. At the same time, non-scientific symbols, icons, and names of body parts and substances,
as well as the cover alluding to swine skin with the identification tag reveal the playful character of the
book.
In a similar vein, Yeasts as We Do Not Know Them is quasi-encyclopedic and simultaneously calls to
mind the glossy magazines aimed at advertising products and services. The publication comprises almost seventy
pages presenting human-yeast interrelations (in its current prototype version; we aim to extend the book to 150
pages). Divided into six sections (research, body, drinks, food, industrial, and material), the printed
collection provides scientific names of yeast species and brief descriptions of the way yeasts are used in a
specific product or research area or are related to human bodies. Additionally, the pages with the excerpts from
historical records (as mentioned above) provide insight into the past and show how yeasts were used in previous
centuries (Figure 2). Thus, the book offers a wealth of factual information and some historical references.
At the same time, however, the accompanying photographs are not meant to illustrate yeasts, yeast-related
diseases, and biotechnologies scientifically; instead, the visuals are playful (Figure 3, 4). With their
colorful backgrounds and brightness, they depict objects connected with specific microbial fungi and the
products and services produced with the involvement of these organisms. The making of such images was inspired
by the work of photographers Colin Ross and Suzanne Saroff, who created bright still lifes with ambiguous
spatial atmospheres and optical illusions. The photographers created these illusions with the help of water or a
magnifying glass, which helped to distort light and alter shadows. ²³, ²⁴ Borrowing some of
these visual effects, the images in Yeasts as We Do Not Know Them attract the readers’
attention and invite them to think about the objects presented in the book from a new perspective.
Additionally, the zine incorporates both scientific classification systems as well as lay names and random
groupings of objects and processes. For instance, we use the scientific names of the yeasts and refer to
classification systems, such as yeast families in the descriptions of yeasts and yeast biotechnologies.
Simultaneously, we loosely divide the examples of yeast-human interactions into research, body, drinks, food,
industrial, and material categories and present them in random order. With such decisions, we aimed to format
factual information in a simple and easily graspable way.
The title of our project refers to the famous bioart book, Art As We Don’t Know It, which
explores the ways in which contemporary artists engage with material culture and the natural sciences, working
across laboratories, art studios, and in the field to produce their transdisciplinary projects.
²⁵ Our zine was particularly inspired by the section of the book titled, “Life as We
Don’t Know It.” This section refers to the most recent developments in synthetic biology (as an area
of biological research focused on (re)designing natural systems) and generally the reconceptualization of
notions of life. ²⁶ Yeasts as We Do Not Know Them involves examples of yeast bioengineering
and the most recent applications of transgenic yeasts in different research areas. At the same time, the book
also aims to represent a variety of yeasts, their habitats, and applications, thus aspiring to expand common
knowledge about these microbes.
Overall, the project was developed via what one may call a “symbiotic collaboration” between the
practitioners from three disciplines. A biotechnologist, a designer, and a researcher in bioart, working
together, shared their knowledge and skills to create a cross-disciplinary publication about human-yeast
relations.
Conclusion
The challenge of dealing with complex, urgent problems such as climate change has led to the development of
transdisciplinary approaches that transcend the boundaries of environmental science, moving towards intensified
collaboration between the arts, sciences, and humanities. A consortium of authors with very different academic
backgrounds was thus involved in the creation of Yeasts as We Do Not Know Them.
In developing the book, we engaged with the notion of symbiosis as more-than-mutualistic interspecies relations
and attempted to invoke trans-corporeal ethics by mapping some of the material interspecies interchanges. The
project showcased diverse interspecies connections focusing on the microscopic single-celled organisms known as
yeasts. Throughout the book, examples of ancestral, contemporary, technological, mutualistic, and hazardous
connections with yeasts allow the audiences to decide how to define symbiosis and what kind of relationships
with yeasts are possible. In this way, Yeasts as We Do Not Know Them invites the broader public to
navigate the material systems of human-yeast relations and contest potentially harmful material interchanges.
Just such an approach to our material systems is crucial in times of ecological crises.
Acknowledgments
The authors would like to acknowledge the financial support from the Foundation for Science and
Technology (FCT) through National Funds by project UIDB/0622/2020 (CITAR).
References
1 Astrida Neimanis, Cecilia Åsberg, Johan Hedrén, “Four Problems, Four
Directions for Environmental Humanities: Toward Critical Posthumanities for the Anthropocene”,
Ethics & the Environment 20 (1), 2015, 67–97.
2 Arianne Conty, “Animism in the Anthropocene,” Theory, Culture & Society
0 (0), 2021, 20, doi:10.1177/02632764211039283.
3 Giovanni Aloi, “Editorial,” Antennae. The Journal of Nature in
Visual Culture, 2022, 10–11.
4 Neimanis, Åsberg, and Hedrén, “Four Problems.”
5 Neimanis, Åsberg, and Hedrén, “Four Problems,” 80.
6 Stacy Alaimo, Bodily Natures: Science, Environment, and the Material Self, Bloomington,
Indiana University Press, 2010.
7 Lisa A. Urry, Michael L. Cain, Steven A. Wasserman, Peter V. Minorsky, and Jane B. Reece,
Campbell Biology, Pearson Education, 2017.
9 Yuuki Kodama, Masahiro Fujishima, “Secondary Symbiosis Between Paramecium and
Chlorella Cells,” International Review of Cell and Molecular Biology 279, 2010, p.33.
10 Donna Jeanne Haraway, “Sympoiesis Symbiogenesis and the Lively Arts of Staying with the
Trouble,” in Staying with the Trouble, Making Kin in the Chthulucene, Durham and London,
Duke University Press, 2016, 58–98, https://www.jstor.org/stable/j.ctv11cw25q.8.
11 Haraway, “Sympoiesis Symbiogenesis and the Lively Arts of Staying with the
Trouble,” p.61.
12 Haraway, “Sympoiesis Symbiogenesis and the Lively Arts of Staying with the
Trouble,” p.63.
13 Claire Pentecost, “Symbiotic Art,” Art in America (1939),
(2022), Vol. 110(2), p.43.
14 Ken Rinaldo, “Can Microbes Save Us from Ourselves?” Antennae, The Journal of
Nature in Visual Culture, 2022, 12– 16.
15 Anna Pomyalova, “Catching Signals from a Vibrant Environment,” Antennae:
Journal of Nature and Visual Culture, 2022, 254–62.
16 Alaimo, Bodily Natures : Science,Environment, and the Material Self.
17 Alaimo, Bodily Natures: Science, Environment, and the Material Self, 18.
18 Alaimo, Bodily Natures: Science, Environment, and the Material Self, 18.
19 Funke Iyabo Sangodeyi, “The Making of the Microbial Body, 1900s-2012,” Ph.D. diss.,
The Department of the History of Science, Harvard University, 2014.
20 Rosemary-Claire Collard, “Putting animals back together, taking commodities apart,”
Annals of the American Geographers, 104(1), 2014, 151–165, quoted in Bram
Büscher, “The nonhuman turn: Critical reflections on alienation, entanglement and nature under
capitalism,” Dialogues in Human Geography, 12(1), 54–73, 2022, 56,
https://doi.org/10.1177/20438206211026200
21 Giuseppe Feola, “Degrowth and the unmaking of capitalism: beyond ‘decolonization of
the imaginary’,” ACME, 18(4), 2019, 977–997, quoted in Bram Büscher,
“The nonhuman turn: Critical reflections on alienation, entanglement and nature under capitalism,”
Dialogues in Human Geography, 12(1), 54–73, 2022, 56, https://doi.org/10.1177/20438206211026200
22 Christien Meindertsma, “Pig 05049,” accessed December 8, 2022,
https://christienmeindertsma.com/PIG-05049
23 Colin Ross, “Cross Sections,” accessed December 8, 2022,
https://www.colinrossphoto.com/#/cross-sections/
24 Suzanne Saroff, “Artwork,” accessed December 8, 2022, https://www.hisuzanne.com/
25 Erich Berger, Kasperi Mäkireinikka, Kira O’Reilly, Helena Sederholm, Art As We
Don’t Know It, Estonia, Aalto University, 2020.
26 Erich Berger, Kasperi Mäkireinikka, Kira O’Reilly, Helena Sederholm, Art As We
Don’t Know It, Estonia, Aalto University, 2020, p.11.
Bibliography
Stacy Alaimo, Bodily Natures: Science, Environment, and the Material Self, Bloomington, Indiana
University Press, 2010.
Giovanni Aloi, “Editorial,” Antennae. The Journal of Nature in Visual Culture,
2022, 10–11.
Erich Berger, Kasperi Mäkireinikka, Kira O’Reilly, Helena Sederholm, Art As We Don’t
Know It, Estonia, Aalto University, 2020.
Bram Büscher, “The Nonhuman Turn: Critical Reflections on Alienation, Entanglement and Nature
under Capitalism”, Dialogues in Human Geography, 12 (1), 56, doi:10.1177/20438206211026200.
Rosemary-Claire Collard, “Putting animals back together, taking commodities apart,” Annals of
the American Geographers, 104(1), 2014, 151–165.
Arianne Conty, “Animism in the
Anthropocene,” Theory, Culture & Society 0 (0), 2021, 1-27, doi:10.1177/02632764211039283.
Giuseppe Feola, “Degrowth and the unmaking of capitalism: beyond ‘decolonization of the
imaginary’,” ACME, 18(4), 2019, 977–997.
Donna Jeanne Haraway,
“Sympoiesis Symbiogenesis and the Lively Arts of Staying with the Trouble,” in Staying with
the Trouble. Making Kin in the Chthulucene, Durham and London, Duke University Press, 2016,
58–98, https://www.jstor.org/stable/j.ctv11cw25q.8.
Yuuki Kodama, Masahiro Fujishima, “Secondary Symbiosis Between Paramecium and Chlorella
Cells,” International Review of Cell and Molecular Biology 279, 2010, p.33.
Astrida Neimanis, Cecilia Åsberg, and Johan Hedrén, “Four Problems, Four Directions for
Environmental Humanities: Toward Critical Posthumanities for the Anthropocene,” Ethics & the
Environment 20 (1), 2015, 67–97.
Claire Pentecost, “Symbiotic Art,” Art in America (1939) 110 (2), New York: Brant
Publications, Incorporated, 2022, 36– 43.
Anna Pomyalova, “Catching Signals from a Vibrant
Environment,” Antennae: Journal of Nature and Visual Culture, 2022, 254–62.
Ken Rinaldo, “Can Microbes Save Us from Ourselves?” Antennae. The Journal of Nature in Visual
Culture, 2022, 12–16.
Funke Iyabo Sangodeyi, “The Making of the Microbial Body,
1900s-2012,” Ph.D. diss., The Department of the History of Science, Harvard University,
2014.
Lisa A. Urry, Michael L. Cain, Steven A. Wasserman, Peter V. Minorsky, Jane B.
Reece, Campbell Biology, Pearson Education, 2017.
Authors Biographies
Olga Timurgalieva is a PhD candidate at City University of Hong Kong and a former visiting
researcher at King’s College London. Awarded by the Hong Kong PhD Fellowship Scheme, her research
investigates the intersections of biotechnology and contemporary art, with a particular focus on fungal
microbes and their interspecies relations. Olga has worked in art institutions, including the ZKM | Center for
Art and Media (Karlsruhe), and co-curated the exhibition “Here and Elsewhere” at the Kobro
Gallery, The Strzemiński Academy of Art (Lodz) and the festival “Seasons of Media Arts 2019”
at the ZKM. She participated in the creation of collaborative art projects that have been exhibited at Aalborg
University Copenhagen (Denmark), nachtspeicher23 (Germany), and Jockey Club Creative Art Centre (Hong Kong).
Patrícia Moreira holds a PhD in Biotechnology with a specialization in Biochemical
Engineering from the Catholic University of Portugal (UCP, Portugal). She is an Assistant Professor at the
School of Arts (UCP). She is an integrated member of the Center for Research in Science and Technology of the
Arts (CITAR). Additionally, Patrícia coordinates the Area-Focus Heritage, Conservation and Restoration
of CITAR and collaborates with the Center for Biotechnology and Fine Chemistry, both UCP. Her main research
area is innovation in Biotechnology for Cultural Heritage, with an emphasis on biodeterioration,
sustainability, citizen science Green Conservation and bio-art practices.
Eva Direito holds a degree in Art Conservation. She’s doing her master’s degree
in Conservation of New Media Art at the School of Arts of the Catholic University of Portugal (UCP, Portugal).
Having received an artistic education, Eva works with digital and analogue photography and graphic design.
During the past few years, she’s been working as an Art Director in short movies for the School of Arts,
some of which earned accolades. For instance, “Our House in Flames” by Miguel Mesquita got
nominations at the Curtas Festival of Vila do Conde, Portugal. Additionally, “Hysteria” by
Luísa Campino, with Eva as artistic director, won prizes at the Sophia Awards in Portugal.
A Sympoietic Ocean. Design research with/in the marine holobiont
RasaWeber
Zürich University of the Arts (CH); Matters of Activity. Cluster
of Excellence, Humboldt University Berlin (DE); University
of Art and Design Linz (AT). Zürich (CH)
& Berlin (DE) info@rasaweber.com
Abstract
In the face of profound human impact on planetary systems, the global ocean, as the main source of life, is
fundamentally transforming its interactions, flows, and ecologies. These critical changes raise questions of
other-than-human cohabitation beyond the terrestrial. In response to these radical ecological changes, a growing
branch of the design discipline is currently struggling to free itself from a production-oriented paradigm of
industrial modernity and reorganizes its methods toward forms of interspecies collaboration with/in environments
of anthropogenic change.
In this paper, I argue for activating the evolutionary theory of Symbiogenesis (Margulis, Kozo-Polyansky,
Meresch-kowsky) and its relevance for a holistic view of the ocean as a starting point for challenging and
reinventing our disciplinary protocols. The article follows Haraway’s notion of sympoïesis, adopting
it for the design discipline. The evolutionary model of Symbiogenesis offers a new perspective on the role of
design as a facilitator for collaborative forms of making and shared survival.
The coral reef, as a prototypical space for symbiotic system relationships, serves as an experimental contact
zone for designing these interspecies encounters. Design research in underwater environments, could help us to
align the design discipline with a new conceptual framework that I propose to call Sympoïetic
Design.
“We are symbionts on a symbiotic planet. And if we care to, we can find symbiosis everywhere.”
¹ – Lynn Margulis, Symbiotic Planet, 1998.
Imagine a coral reef. The damselfish flicks its fins as it hides among stony corals to support its
photosynthesis. The green-brown algae that live on the spider crab’s back help it to camouflage. The
currents transport nutrients across limestone structure where corals spawn and settle. The zooxanthellae inhabit
coral tissue and metabolize organic matter to provide energy for their hosts. The stony coral animal
absorbs calcium carbonate from seawater to form its habitat. The microbiome of the reef binds
Co2 and produces the oxygen of the planet’s atmosphere – it pulses with life.
The coral reef can be considered a prototypical ecology for symbiotic system relationships. Symbiosis governs
the marine microbiome at all levels, from species-based networks to the microbiological level ². As former
biologist and professor of feminism and technoscience Donna J. Haraway points out, the marine biome in general,
and the “critical zone” of the coral reef in particular, serve as model organisms for symbiotic
evolutionary processes ³ within the planetary holobiont ⁴.
In chapter 3 of Staying with the Trouble (Haraway 2016) “Sympoiesis. Symbiogenesis and the
Lively Arts of Staying with the Trouble.” Haraway invokes the seminal work of biologist Lynn
Margulis, who revived the evolutionary theory of symbiogenesis based on the preliminary work of
Russian evolutionary biologists Boris Kozo-Polyansk ⁶ and Konstantin Mereschkowsky ⁷. When Margulis
published her theory in the 1960s, she provoked and disturbed the male-dominated scientific discourse not only
as a female academic, but also through her decisive rejection of the then widespread Neo-Darwinian concept of
evolution. Her theory challenged the idea of evolutionary competition – the hypothesis of random mutation
as driver for “selfish” survival of the fittest ⁸ – and relied on a
cooperative model of survival through symbiosis ⁹.
Margulis’ impassioned plea for symbiosisas an overlooked facilitator of evolutionary adaptation and her
uncompromising proposal for a taxonomy of species which gives bacteria, animals, plants, fungi, and protoctista
an equal position in the evolutionary Tree of Life, is nowadays generally considered as one of the most decisive
contributions to ecological thinking. One could argue that Magulis’ theory resurrects Charles Darwin's
alternative representation of evolution in the form of the coral: an anarchically growing form “that
no longer upholds mankind as the ‘crown of evolution’’”¹⁰, as sketched
in Darwin's early notebooks ¹¹. The Coral of Life can be seen as closely related to
Mereschkowsky’s endosymbiotic Tree of Life and Lynn Margulis’ later extension to the Five
Kingdoms ¹².
Philosopher Donna Haraway’s outstanding contribution in Staying with the Trouble consists in turn,
amongst other things, of applying Margulis’ theory of symbiogenesis to philosophy and system theory. She
introduced a form of thinking and making, which she called sympoïesis ¹³—“together
creation”—with human and other-than-human actors. While Haraway argues for transferring
sympoïetic forms of production to the humanities and the arts, activating her theory for the design
discipline holds the potential to reorganize a human-centered approach to design toward highly relational forms
of creation through interspecies collaboration.
Philosopher Donna Haraway’s outstanding contribution in Staying with the Trouble consists in turn,
amongst other things, of applying Margulis’ theory of symbiogenesis to philosophy and system theory. She
introduced a form of thinking and making, which she calledsympoïesis
¹³ —“together creation”—with human and other-than-human actors. While
Haraway argues for transferring sympoïetic forms of production to the humanities and the arts, activating
her theory for the design discipline holds the potential to reorganize a human-centered approach to design
toward highly relational forms of creation through interspecies collaboration.
A growing branch of design is currently struggling to break free from a production—and profit-oriented
paradigm of industrial modernity (Mareis & Paim 2021). By bringing the agency of troubled environments and
their (other-than-) human inhabitants to the foreground of a design methodology, we might foster a reinvention
of our disciplinary protocols in response to our current ecological and socio-political struggles, which can
never be addressed separately (Stengers 2010).
Designing (with/in) Marine Ecosystems
As a design researcher and diver, I am concerned with material processes, their production cycles, and their
resulting impacts on environmental systems. I am deeply moved by the question of developing and establishing
post-extractive models of production within our challenged planetary networks.
This question led me to the ocean, the largest life-giving (Helmreich 2009; Latour 2019) and dramatically
transforming system on our planet (Lohmann et al. 2017; Heinze et al., 2021). Together with biologists from the
Max Planck Institute of Animal Behavior (Jordan Lab), I conduct most of my design research in the field, below
sea level. I develop immersive underwater methods ¹⁴ (Helmreich 2009; Raycraft 2020; Swanson &
Levy 2021; Jue 2022) for design research, prototyping sympoïetic material experiments underwater, and
research on the design ontological implications of my aquatic practices (Neimanis 2017, Åsberg 2020).
Although designing underwater habitats looks back at a long history (See: e.g. Cousteau’s Conshelf
Project 1962-1965), most of these early interventions still follow a human-centered design approach. My project
takes the historical material technology of Biorock ¹⁵ as a starting point and seeks to convey a
different notion of habitat design in ocean environments as a culturally embedded, interspecies, and
collaborative process.
My research is based on the preliminary work of the German architect and oceanographer Wolf Hilbertz and his
colleague in biochemistry Tom Goreau. As early as 1970, their research introduced a process of underwater
electrolysis triggered by low-voltage electricity (6-12 V) that causes limestone to "grow" on conductive steel
scaffold in the sea—a technology that was patented under the name Biorock. Naturally dissolved minerals in
seawater (primarily aragonite and brucite) are slowly deposited on the conductive steel, which provides an ideal
medium for the colonization by marine life. The technology is now widely used to restore coral reefs in tropical
marine areas.
In spring 2022, I visited one of the Biorock projects on the island of Curaçao (AN) with the aim of
learning more about the technological implications, ecological consequences, and maintenance methods of such an
underwater garden and subsequently build my own first prototypes (Figure 1).
My own design contribution explores the possibilities of applying the Biorock technology to contemporary
materials, combining a practice-based design approach with situated, collaborative, and craft-oriented design
methods. Based on the principle of electrodeposition of minerals in seawater, I construct underwater prototypes
as Interspecies Architecture which are neither exclusively for a human audience nor solely for marine critters.
The prototypes function as contact zones between species and question the role of architecture as a
human-centered practice. Instead of the historically used material—reinforcing steel—my work focuses
on reducing material consumption and applies filigree conductive yarns to construct lightweight marine
prototypes with the help of solar energy. When the conductive yarns are exposed to electricity, a limestone-like
material forms on their surface, transforming a formerly flexible textile surface into a solid substrate that
supports the growth of marine life; A technology that literally allows me to weave stone.
My prototypes mediate between the marine geosphere and biosphere. The spun steel nets provide a
bioreceptive surface (Cruz & Beckett 2016) for multiple forms of marine life to slowly colonize on
the mesh (Figure 4). Rather than taking a strictly selective approach to coral reef conservation, as classically
the case in coral nurseries (which host a limited number of pre-selected coral branches), these prototypes
should be viewed as an invitation for various forms of marine life to proliferate and thrive. The wild
colonization of corals, sponges, algae and even marine neophytes is a desired outcome of these biological
material experiments. For me, designing Architectures of Cohabitationmeans inviting all symbiotic forms of
marine life to actively participate in the design process. They co-design their own habitat and subsequently
change the appearance and species composition of the prototype. Thus, the design process takes place not only in
the sea, but together with the marine ecosystem.
In my designs, I try to give the sympoïetic growth processes of the marine biome the decisive voice.
Marine species shape the underwater architecture, distort it, overgrow it, make it invisible, possibly even
cause it to collapse.
The environments that will eventually serve as testing grounds for my marine Interspecies Architectures are
neither the "pristine" and endangered coral reefs of Hilbertz's previous work nor the marine protected areas of
my first field experiments. The seascapes I want to study and which I regard as ideal environment for my
prototypes are areas of high anthropogenic impact, or as Clouette and Fabien call them, an
‘AnthropOcean’ (2018)—The harbor, the former industrial mine, the canal, the marine military
zone or the tourist beach are the damaged, contaminated, and conflicted zones of marine survival that I will
engage with in order to design habitats "amidst ruins" (Tsing 2015).
I consider these slow-growing structures as Architectures of Co-habitation, as they provide a habitat for
multiple marine species and temporary human visitors alike. Based on Tsing’s and Barua’s notion of
“feral ecologies” (2021), sympoietic ecological worlds emerge in the Anthropocene when
other-than-human beings become entangled with human infrastructure in urban environments (Tsing et al. 2021). My
underwater prototypes are located precisely in these conflicted zones—they are breeding ground, ecological
niche, and site of human infrastructure, interest, and intervention.
Designing Habitats
“How much longer will we agree to step aside in silence as masters of the universe
[…]? How much longer will you and I choose extinction?” ¹⁶
– Gan, Tsing, Swanson & Bubandt 2017.
If design takes its fundamental contribution to the unruly times of the Anthropocene seriously and begins to
“care forthat which is non-human” ¹⁷, it will encounter a series of
troubling questions. A design practice that is not only interested in advocating for human agency, but also
includes other species and conflicted environments in its processes of making,
must be brave enough to propose new “modes of production and cooperation that escape from the evidence
of economic growth and competition.” ¹⁸ If design advocates
conviviality and shared survival, it must formulate strategies to
“positively cohabit a thriving biosphere” ¹⁹, and inevitably comes into contact
with fundamental systemic issues of our assigned human role in the ecological fabric.
In his early publications, Wolf Hilbertz refers to his marine engineering intervention as
“Cybertecture” ²⁰. The neologism is a reference to Hilbertz’
understanding of architecture as a logical extension of the principles of cybernetics ²¹ to the
built environment (CYBERnetics + archiTECTURE). Most likely inspired by Norbert Wiener’s famous concept of
Cybernetics as “the science of control and communications in the animal and machine”
²² , Hilbertz thought of complex systems controlled by a purposive model of causality and
feedback. Such a biogenetic model, “that employs self-improving software and hardware [and]
can draw ‘unorganized’ matter into its system” ²³, suggest a revolutionary
notion of architecture as constantly adaptive and self-improving. However, the cybernetic core of his theory
also, perhaps unintentionally, places the human in the center of system control. The concept of the designer as
a "helmsman" (ky-bernētēs Greek = helmsman), who has full control over a system and is responsible for
its well-being, must be seen in the context of the unbridled techno-optimism of the 1960s and the concomitant
rise of the human-centered design movement (e.g. Arnold, Stanford University design program, 1958). It should be
kept in mind that when Hilbertz developed his theory of Cybertecture, the discourse of architectural theory was
just beginning to address environmental consciousness for the first time, also triggered by the ecological
discourse of prominent voices such as a.o. Rachel Carson (1962).
With the Biorock system, Hilbertz and Goreau introduced an extremely bold ecological technology that was far
ahead of its time and made it possible to build with and within the ocean ecosystem. However, given what we now
today about the cumulative impacts of humans on the world's ocean systems, we need to reevaluate the system's
useful technological capabilities ²⁴ in relation to its potential consequences.
“The planet takes care of us, not we of it”, ²⁵ Margulis states in her
influential book A Symbiotic Planet(1998).Rather than trying to regulate the incredibly complex system
relationships of an ecology, or even going so far as to “cure” the ocean as “a
sick patient” ²⁶, Margulis’ perspective suggests questioning the role of human
control in favor of a much modest and highly interconnected perspective. From this thought, we designers must
accept our small part as humans within larger ecological relationships that transcend the influence of human
intentions.
In this sense, the Biorock principle could shape and inform contemporary design discourse to conceive
“evolutionary environments” ²⁷ that adapt to their environmental conditions and
grow responsively with the marine ecosystem. Today, Goreau and Hilbertz’ concept could serve as a starting
point for developing a form of sympoïetic making that views the designed “product” not just as
the result of planned human intervention, but as relational, highly connected, and life-giving habitat.
Towards Sympoïetic Design
A design practice that sees the agencies of all life forms as central to shaping the design process may be able
to move beyond a product-driven and extractive notion of design and open up to collaborative forms of design
that transcend species barriers.
I propose to call these practices Sympoïetic Design. The evolutionary model of symbiogenesis lays the
groundwork for conceptualizing design processes as long-term collaborations between humans and other-than-human
beings. These sympoïetic processesextend over unimagined time spans and situate design practices in
troubled environments. Design becomes an action of shaping “invitations” (Stahl &
Lindström, 2016) for animals, bacteria, proctists, et al. The resulting prototypes I present here suggest
that mineralized limestone mesh can serve as a “bioreceptive medium” (Cruz & Beckett,
2018) for a variety of marine species to settle on them (Figure 4 & 5). Sympoïetic Design provides
habitats as contact zones for coevolution between different species.
Het Niewe Instituut recently introduced the term “Zoöps” to describe a “cooperative
legal entity in which humans and multispecies ecological communities are partners”²⁸(zoë Greek = life). The idea is intriguing because it views habitats as an
assembly of symbiotic entities that engage humans in their “naturecultural” ²⁹
relationships within an anthropogenic ecology. As humans, we impact habitats, but at the same time are
inevitably a part of them. Designing for ecological impact, then, means designing in and with
the habitats that assure our own survival. Applying this concept to marine habitats and thus creating convivial
zones of protection and experimentation as Marine Zoöps should become a central task for design in the
ocean.
A number of young designers and researchers are beginning to design systems with and for
other-than-humans. The Symbiotic Spaces Collective is working on 3D-printed architectures located in local
wetlands that provide multispecies habitats for urban wildlife. Designer Marie Griesmar, biologist Ulrike
Pfreundt, and marine scientist Hanna Kuhfuss, founders of Rrreefs e.V., developed a 3D-printed clay brick that
serves as a module for coral reef restoration. The biologist Anja Wegner and the artist collective Superflex
collaborate on their “Pink Elements” to create “fish architecture”³⁰ for
damselfish, and designer David Enon experiments with growing underwater furniture from limestone structures as
coral habitat with his “Mineral Accretion Factory” ³¹.
Despite these early beginnings, deep reflection on the benefits and limitations of working with/in ecological
systems is urgently needed. Sympoïetic Design raises new questions for our discipline that call for further
discussion:
How does Sympoïetic Design affect the role of the designer as author?
What agency does it give to other-than-human creators in the design process?
What forms of (disciplinary) knowledge are needed to implement and evaluate it?
What timespans are needed to facilitate it?
Who are we designing for?
How structured or “messy” will the result be?
Working with/in the marine biome can be a first experimental approach to discuss the potentials and challenges
of Sympoïetic Design. With my underwater prototypes, I work on creating contact zones between multiple
forms of marine life. These Architectures of Cohabitation mediate between animate and inanimate matter, between
human-made and naturecultural ecosystems, between human and environmental agency. Rather than restoring
“pristine” nature, these co-created habitats encourage the designer to relinquish control of the
system and invite the wildly proliferating and uncurated, unexploited life forms to the process of creation.
Sponges, algae, coral, electrical circuits, marine biologists, fish, cameras, limestone, polyps et al. become
part of these multispecies marine eclogies of sympoïetic making. Sympoïetic Design opens the door to
design practices that transform human-centered design projects into an evolutionary practice of share survival.
Towards a sea change ³² in design.
Acknowledgements
I would like to thank my collaborators, Max Planck Institute for Animal Behavior - Jordan Lab, for their
continuous support and time in the field, especially biologists Anja Wegner and Prof. Dr. Alex Jordan.
Furthermore, I would like to thank Prof. Dr. Karmen Franinović (ZHdK) and Prof. Dr. Karin Harrasser
(Kunstuniversität Linz) for their supervision and support during this research project. I would like
to thank the "Biorock" project on Curaçao (NL), the diving instructor Mike Duss, the local diving school
"Curaçao Divers" as well as the "Sea School" Pula (HVR). Finally, I would like to thank Tom Goreau for
many conversations, critical discussions, and technical support with the Biorock technology.
References
Books
Horst Bredekamp, Darwin's Corals. A New Model of Evolution and the Tradition of Natural History,
Berlin, Boston, De Gruyter, 2019, https://doi.org/10.1515/9783110680317.
Rachel Carson, Silent Spring, New York, Fawcett Crest, 1962.
Donna J. Haraway, The Companion Species Manifesto. Dogs, People, and Significant Otherness, Bristol,
Prickly Paradigm Press, 2003.
Stefan Helmreich, Alien Ocean: Anthropological Voyages in a Microbial Sea, Seas Berkeley, CA,
University of California Press, 2009.
Boris M. Kozo-Polyansky In: Symbiogenesi,. A new principle of evolution, Translation from the
Russian by Victor Fet, Eds. By Victor Fet, Lynn Margulis, Cambridge, Harvard Univ. Press, 2010.
Bruno Latour, Foreword, In: Prospecting Ocean, ed. Stefanie Hessler, Cambridge, Massachusetts,
London, England, The MIT Press, 2019, 11-13.
Lynn Margulis, The Five Kingdoms: Illustrated Guide to the Phyla of Life on Earth, San Francisco,
W.H. Freeman & Co Ltd., 1982.
Lynn Margulis, Symbiotic Planet, A New Look on Evolution, Amherst, Massachusetts, Brokman inc.,
Basic Books, revisited edition, 1998.
Astrida Neimanis, Bodies of Water: Posthuman Feminist Phenomenology, Environmental Cultures. London,
New York, Bloomsbury Academic 2017.
Isabelle Stengers, Cosmopolitics I., Minneapolis, London, Minnesota Press, 2010.
Isabelle Stengers, In Catastrophic Times - Resisting the Coming Barbarism, Open Humanities Press,
Meson Press 2015.
Anna L. Tsing, The Mushroom at the End of the World: On the Possibility of Life in Capitalist Ruins,
Princeton, Princeton University Press, 2015.
Anna L. Tsing, Heather A. Swanson, Elaine Gan & Nils Bubandt (Eds.), Arts of Living on a Damaged
Planet: Ghosts and Monsters of the Anthropocene, Minneapolis, University of Minnesota Press, 2017.
Ron Wakkary, Things We Could Design: For More Than Human-Centered Worlds, Cambridge, MA, MIT Press,
2021.
Norbert Wiener, Cybernetics, or, Control and communication in the animal and the machine. The
Massachussets Institute of Technology, orig. 1948, Cambridge Massachussets, The MIT Press, 1985.
Chapters in Books
Richard Dawkins, “Chapter 4: The Gene Machine”, In: The Selfish Gene, Oxford, Oxford
University Press, 1976, 46-65.
Donna J. Haraway, Chapter 3: Four Critical Zones. In: Staying with the Trouble: Making Kin in the
Chthulucene; Durham and London, Duke University Press, 2016, 72-76.
Donna J. Haraway, “Chapter 3: Sympoiesis. Symbiogenesis and the Lively Arts of Staying with the
Trouble”, In: Staying with the Trouble: Making Kin in the Chthulucene, Durham and London, Duke
University Press, 2016, 58-98.
Åsa Ståhl, Kristina Lindström, “Politics of Inviting: Co-Articulations of Issues in
Designerly Public Engagement”, In:Design Anthropological Futures, Rachel C. Smith et al.
(Eds.), London, Routledge, 2016.
Jonathan P. Zehr, David A. Caron, “Symbiosis in the Ocean Microbiome”, In: The Marine
Microbiome: An Untapped Source of Biodiversity and Biotechnological Potential, Lucas J. Stal
& Mariana S. Cretoiu (eds.), Springer, 2016, 535-577.
Journal article (online)
Cecilia Åsberg, “A Sea Change in the Environmental Humanities”, In: Ecocene: Cappadocia
Journal of Environmental Humanities Vol. 1, No. 1, 2020, 108-122.
Maan Barua, “Feral ecologies: the making of postcolonial nature in London”, Journal of the
Royal Anthropological Institute, Vol. 28, No. 3, 2021, 896-919.
Marcos A. da Silva Cruz, Richard Beckett, “Bioreceptive design: a novel approach to biodigital
materiality”, Architectural Research Quarterly Vol. 20, No. 1, 2016, 51-54.
Ricardo Guerrero, Lynn Margulis, Mercedes Berlanga, “Symbiogenesis: the holobiont as a unit of
evolution.” Microbiol. Vol. 16, No. 3, 2013, 133-43.
Benjamin S. Halpern, Melanie R. Frazier, Jamie Afflerbach, et al. “Recent pace of change in human
impact on the world’s ocean”, Scientific Reports Vol. 9, No.1, 2019,
Doi:10.1038/s41598-019-47201-9.
Wolf Hilbertz, “Toward Cybertecture.” Progressive Architecture, The Architect as
Developer (1970), Wolf Hilbertz Website, accessed February 14, 2024,
http://www.wolfhilbertz.com/downloads/1970/hilbertz_t_cybertecture_1970.pdf.
Wolf Hilbertz, “Strategies for Evolutionary Environments”, The Responsive House, Edward
Allen (Ed.), MIT Press, 1972.
Wolf Hilbertz Website, accessed online February 14, 2024,
http://www.wolfhilbertz.com/downloads/1970/hilbertz_t_cybertecture_1970.pdf.
Wolf Hilbertz, “Electrodeposition of Minerals in Sea Water: Experiments and Applications”,
IEEE Journal on Oceanic Engineering, Vol. OE-4 Vol.3 1979, 94-113.
Wolf Hilbertz Website,
accessed online February 14, 2024, https://globalcoral.org/_oldgcra/IEEE_JOUR_1979small.pdf.
Melody Jue, “Scuba Diving as Praxis: A Field Guide for Underwater Orientation by Melody Jue”,
Fieldwork for Future Ecologies, ed. Bridget Crone, Sam Nightingale, and Polly Stanton (Onomatopee,
2023), accessed February 14, 2024,
https://static1.squarespace.com/static/545596ece4b0717418ebf8e6/t/642de8963cdc9f603863cdc3/1680730265157/Melody+Jue_Scuba+diving+praxis.pdf.
Klaus V. Kowallik & William F. Martin “The Origin of Symbiogenesis: An annotated English
translation of Mereschkowsky’s 1910 paper on the theory of two plasma lineages”,
Bioystems, Vol. 199, No. 104281, 2020, Doi: 10.1016/j.biosystems.2020.104281.
Johannes Lohmann, Daniele Castellana, Peter D. Ditlevsen, and Henk A. Dijkstra “Abrupt
climate change as a rate-dependent cascading tipping point.” Earth System Dynamic, Vol. 12, No.
3, 2021, 819–835.
Justin Raycraft. “Seeing from Below: Scuba Diving and the Regressive Cyborg.” Anthropology
and Humanism, Vol. 45, No. 2, 2020, 301-321.
Lynn Sagan, “On the origin of mitosing cells.” J. Theoretical Biology Vol. 14, No. 3,
1967, 255-274, doi:10.1016/0022-5193(67)90079-3.
Marie E. Strader, Kate M. Quigley, “The role of gene expression and symbiosis in reef-building coral
acquired heat tolerance”, Nature Communications Vol. 13, Article number, 4513, 2022.
Heather A. Swanson, Sonia Levy, “A (Highly) Partial Field Guide to British Canals: Introducing some
processes and beings", TBA21-Academy, 2021.
Websites
Thomas J. Goreau, “A review of John Todd, 2019: Healing Earth: An ecologist’s journey of
innovation and environmental stewardship (2019)”,Global Coral Reef Alliance Website, accessed: November
17, 2022,
www.globalcoral.org/a-review-of-john-todd-2019-healing-earth-an-ecologists-journey-of-innovation-and-environmental-stewardship/.
Nieuwe Instituut, “Zoöp Nieuwe Instituut (2022)”, Nieuwe Instituut Website, accessed:
December 9, 2022, https://zoop.hetnieuweinstituut.nl.
Anna L. Tsing, Jennifer Deger, Alder Keleman Saxena & Feifei Zhou, Feral Atlas: The More-Than-Human
Anthropocene, Redwood City, Stanford University Press, 2022), Feral Atlas Website (2022), accessed
December 9, 2022, http://feralatlas.org.
Conference Proceedings
David Enon, “Mineral accretion factory: An underwater production process with a positive impact on the
environment”, Figshare, Journal contribution, Conference, RTD, 2019,DOI:
https://doi.org/10.6084/m9.figshare.7855775.v2.
Christoph Heinze, Thorsten Bleckner, Helena Martins et al. “The quiet crossing of ocean tipping
points.” Proceedings of the National Academy of Sciences Vol. 118, No. 9, 2021, DOI: https://doi.org/10.1073/pnas.2008478118.
Anja Wegner, Superflex & Alex Jordan, “Fish Architecture – A framework to create Interspecies
Spaces.” Proceedings of Politics of the machines - Rogue Research 2021, (POM 2021).
Anja Wegner, Rasa Weber, “Attunement to the Ocean. Underwater field methods between
marine biology and design research”, Counterparts: Exploring Design Beyond the Human, Swiss
Design Network Conference 2022. (Forthcoming).
Design Works
David Enon, Mineral Accretion Factory, since 2011.
Marie Griesmar, Ulrike Pfreundt, & Hanna Kuhfuß, The rrreef bricks. Rrreefs e.V., since
2020.
Anja Wegner & Superflex, Pink Elements, since 2019.
Notebooks
Charles R. Darwin, “Sketch: Coral of Life.” Page 26 of Notebook B,
(1837–1838). Cambridge University Library MS DAR.121.26.
Endnotes
1 Lynn Margulis, Symbiotic Planet. A New Look on Evolution, Amherst, Massachusetts,
Brokman inc., Basic Books, revisited edition, 1998, 5.
2 Jonathan P. Zehr, David A. Caron, “Symbiosis in the Ocean Microbiome.” In: The
Marine Microbiome: An Untapped Source of Biodiversity and Biotechnological Potential, Lucas J. Stal
& Mariana S. Cretoiu (eds.), Springer, 2016, 535-577.
3 Donna J. Haraway, Chapter 3: Four Critical Zones. In: Staying with the Trouble: Making
Kin in the Chthulucene, Durham and London, Duke University Press, 2016, 72-76.
4 Holobiont def. = The entire community of living organisms. In: Michael Allaby,
Dictionary of Ecology, Oxford, Oxford University Press, 2015.
5 Symbiogenesis def. = The result of the permanent coexistence of various bionts to form the holobiont
(namely, the host and its microbiota). In: Ricardo Guerrero, Lynn Margulis & Mercedes Berlanga,
“Symbiogenesis: the holobiont as a unit of evolution.” Microbiol. Vol. 16, No. 3, 2013,
133-43.
6 Boris M. Kozo-Polyansky In: Symbiogenesis. A new principle of evolution. Translation from
the Russian by Victor Fet, Eds. By Victor Fet and Lynn Margulis, Cambridge, Harvard Univ. Press, 2010.
7 Klaus V. Kowallik, William F. Martin “The Origin of Symbiogenesis: An annotated English translation
of Mereschkowsky’s 1910 paper on the theory of two plasma lineages”, Biosystems, Vol.
199, No. 104281, 2020, Doi: 10.1016/j.biosystems.2020.104281.
8 Richard Dawkins, “Chapter 4: The Gene Machine”, In: The Selfish Gene, Oxford: Oxford
University Press, 1976, 46-65.
9 Margulis outlined “the astounding details of the microbes (…) in their tendency to survive
by making symbiotic evolutionary commitments,” arguing for her central theory on the origin of
eukaryotic cells from prokaryotic organisms, In: Lynn Margulis, Symbiotic Planet, Basic Books 1989,
72.
10 Horst Bredekamp, Darwin's Corals. A New Model of Evolution and the Tradition of Natural
History, Berlin, Boston, De Gruyter, 2019, https://doi.org/10.1515/9783110680317.
11 Charles R. Darwin, “Sketch: Coral of Life.” Page 26 of Notebook B,
1837–1838, Cambridge University Library MS DAR.121.26.
12 Lynn Margulis, The Five Kingdoms: Illustrated Guide to the Phyla of Life on Earth,
San Francisco: W.H. Freeman & Co Ltd., 1982.
13 sympoïesis def. =
“Σύν,sún = greek
‘together’ and ποίησις,
poíēsis = greek ‘creation, production’”. In:
Donna J. Haraway Staying with the Trouble. Making Kin in the Chthulucene, Duke University
Press, 2016, p.58.
14Anja Wegner, Rasa Weber, “Attunement to the Ocean. Underwater field methods between
marine biology and design research", Counterparts: Exploring Design Beyond the Human, Swiss Design
Network Conference 2022. (Forthcoming).
15 Wolf Hilbertz, “Electrodeposition of Minerals in Sea Water: Experiments and
Applications”, IEEE Journal on Oceanic Engineering, Vol. OE-4 Vol.3 (1979): 94-113. Wolf
Hilbertz Website, accessed online February 14, 2024, https://globalcoral.org/_oldgcra/IEEE_JOUR_1979small.pdf.
16 Anna L. Tsing, Heather A. Swanson, Elaine Gan & Nils Bubandt (Eds.) Arts of Living
on a Damaged Planet: Ghosts and Monsters of the Anthropocene, Minneapolis, University of Minnesota
Press, 2017, G4.
17 Ron Wakkary, Things We Could Design: For More Than Human-Centered Worlds,
Cambridge, MA, MIT Press, 2021, 1-2.
18 Isabelle Stengers, In Catastrophic Times - Resisting the Coming Barbarism, Open
Humanities Press, Meson Press 2015, 24.
19 Ron Wakkary, Things We Could Design (2021), 2.
20 Wolf Hilbertz, “Toward Cybertecture.” Progressive Architecture, The
Architect as Developer (1970), Wolf Hilbertz Website, accessed February 14, 2024,
http://www.wolfhilbertz.com/downloads/1970/hilbertz_t_cybertecture_1970.pdf.
21 Cybernetics def. = The term cyberneticscomes from the ancient Greek word
κυβερνήτης, kybernetikos = “good
at steering”, referring to the art of the helmsman. In: Britannica Website, accessed February 1,
2022, https://www.britannica.com/science/cybernetics.
22 Norbert Wiener, Cybernetics, or, Control and communication in the animal and the
machine, The Massachussets Institute of Technology, orig. 1948, Cambridge Massachussets, The MIT Press,
1985.
23 Wolf Hilbertz “Toward Cybertecture", 1970, 99. [Addition by the
autor.]
24 Benjamin S. Halpern, Melanie R. Frazier, Jamie Afflerbach, et al. “Recent
pace of change in human impact on the world’s ocean.” Scientific Reports Vol. 9, No.1,
2019, Doi:10.1038/s41598-019-47201-9.
25 Lynn Margulis, Symbiotic Planet, 1998, 115.
26 Thomas J. Goreau, “A review of John Todd, 2019: Healing Earth: An ecologist’s
journey of innovation and environmental stewardship (2019)”,Global Coral Reef Alliance Website accessed:
November 17, 2022,
www.globalcoral.org/a-review-of-john-todd-2019-healing-earth-an-ecologists-journey-of-innovation-and-environmental-stewardship/.
27 Wolf Hilbertz, “Strategies for Evolutionary Environments.” The Responsive
House, Edward Allen (Ed.) (MIT Press, 1972), Wolf Hilbertz Website, accessed online February 14, 2024,
http://www.wolfhilbertz.com/downloads/1970/hilbertz_t_cybertecture_1970.pdf.
28 Nieuwe Instituut, “Zoöp Nieuwe Instituut (2022)”, Nieuwe Instituut
Website, accessed: December 9, 2022. Link: https://zoop.hetnieuweinstituut.nl.
29 Natureculture def. = a synthesis of nature and culture that recognizes their
inseparability in ecological relationships that are both biophysically and socially formed. In: N. M.
Malone & Kathryn “Natureculture”, In: The International Encyclopedia of Primatology,
John Wiley & Sons. See also: Donna J. Haraway, The Companion Species Manifesto.
Dogs, People, and Significant Otherness. (Bristol: Prickly Paradigm Press, 2003).
30 Anja Wegner, Superflex, Alex Jordan, “Fish Architecture – A framework to create
Interspecies Spaces.” Proceedings of Politics of the machines - Rogue Research 2021, POM 2021.
31 David Enon, “Mineral accretion factory: An underwater production process with a
positive impact on the environment.” Figshare. Journal contribution. Conference, RTD,
2019,DOI:https://doi.org/10.6084/m9.figshare.7855775.v2.
32 Cecilia Åsberg, “A Sea Change in the Environmental Humanities.” In:
Ecocene: Cappadocia Journal of Environmental Humanities Vol. 1, No. 1, 2020, 108-122.
NER: Physical-Virtual Multimodal Generative NFT with a Rarity Model
1,2IIP (Computational Media and Arts), The Hong Kong University of Science and
Technology,3Computational Media and Arts, The Hong Kong University of Science and Technology,
4Division of Integrative Systems & Department of Computer Science and Engineering, The Hong Kong
University of Science and Technology, 5Division of Emerging Interdisciplinary Areas, The Hong Kong
University of Science and Technology 1,2,4,5 Hong Kong SAR, China 3Guangzhou,
China ∗dfeiaa@connect.ust.hk
Abstract
Nowadays, NFTs are increasingly emerging into public view. It is necessary to consider NFT as a sustainable
trading model for media arts with audience interaction as the symbiosis community.
This article presents a design architecture of a Multimodal NFT with generative patterns triggered by sensors.
By designing various 3D mesh attributes, such as eye patterns and fur colors of a robin bird, the authors
generate a series of different NFT 3D artworks. We also demonstrate our experience with the generative system
together with the rarity scoring model that could be used to evaluate the heterogeneity of NFT collections.
As new “digital titles (tokens) to property, either real or virtual, stored on a blockchain”
¹², Non-Fungible Token (or NFT) influences various social aspects, such as financial markets and art
collections, especially media art creation. We believe that apart from its commercial exploitation, NFT
represents a new medium that gives artists a different way of thinking about the creative process.
Multimodal interaction ³, i.e., “interaction with the virtual and physical environments through
natural modes of communication”, has been a widely accepted term in media art creation. It, however,
seldom appears in the NFT artwork creation.
Computational Generative Art (CG-art) is defined as the art “produced by leaving a computer program to
run by it-self” ². By manipulating the generative design process, various artists have produced
different captivating generative artworks over the years. In connection with NFT, new methods of displaying
artworks have recently evolved, both online and offline, in addition to the computer screen and physical print
presentations.
Guo et al. state “the art world will witness a new form of art economy with the coming of Web 3.0 (the
metaverse), benefiting from its augmented creativity and sophisticated marketplace.” ⁷. This article
provides a short review of Generative NFTs, multimodal dynamic NFTs, and physically mixed virtual NFTs with
different NFT rarity models. We then demonstrate our experiment NER, which creates a Multimodal Generative NFT
with sensors and a rarity model.
Related Work
Multimodal Generative NFTs Commercial online NFT collection platforms, such as OpenSea
¹⁵ and SuperRare ²⁴, are usually non-customizable. Recent advances in computational
generative art have renewed interest in generative art in different materials ²⁷. Artists and
researchers have generated various 2D patterns. For example, Zhang and Yu develop a font generation system in
the style of Kandinsky ²⁶. Others attempted to create 2D generating textures on 3D mesh, such as Wu
and Huang’s artwork “Mimicry” ²⁵, which is a genetic-algorithm-based realtime
system that creates virtual insects within a real environment. These generative artworks are not changed into
NFT format due to technical limitations. At the same time, a NFT art collection platform, especially for
generative art, called ArtBlocks ²³ pushes the limits of generative art with blockchain technology.
There are many interesting generative artworks on this website, especially in the category of curated
collections.
Digital artist Pak built a generative NFT artwork “Merge” ¹³ which returns a new
mechanism with collectors. Merge obtains the number of each collector during the 48-hour sale and generates
dynamic on-chain NFT based on the total mass number. The NFT visual becomes bigger when collectors acquire more
mass. This NFT artwork builds a new way that collectors can be seen as an entangleable and competitive symbiosis
community. Refik Anadol studio used projection mapping on the facade of Casa Batllo ́ to model the dynamic
NFT artwork “Living Architecture” ⁹ with weather sensor data collected in realtime. This
dynamic NFT renews and reinvigorates World Heritage building through AI technology. Most NFT artworks containing
physical and virtual versions are highly relevant to commercial applications, particularly brand marketing. For
example, Nike, together with RTFKT, established a range of virtual sneakers and expanded into physical goods,
such as clothing. Look Labs ¹⁰ uses near infrared spectroscopy to create a digital scent, and
translate the scent into NFT artworks, making it the world’s first digital fragrance ¹⁷.
NFT Rarity NFT rarity shows how rare a NFT item is compared to the rest of the items from the
same collection, which in turn can represent how valuable such a collectible can be. When looking at a
collectible NFT item on Opensea, such as a CryptoPunk ⁶ , one can see that the item has many traits, for
example, accessories like an earring, a bandana, or a pipe. Since each NFT item has multiple traits, several
rarity calculating models exist to combine each trait’s rarity into a single value per NFT item to rank
and compare different NFT items.
Mainstream rarity calculating tools that can calculate and compare NFTs, such as Rarity.Tools ²⁰ ,
offer four models for users to calculate the rarity of NFTs: Trait Rarity, Average Trait Rarity, Statistical
Rarity, and Rarity Score (see Figure 1).
The Trait Rarity model works by only comparing the rarest trait of each NFT item. While it is a simple,
straightforward model, this model’s weakness is also apparent. It does not consider the total number of
traits or other traits of the NFT item. As a result, this model provides an incomplete picture of the NFT item,
ignoring other traits that may also be rare.
The Average Trait Rarity model ¹⁹ averages the rarity of traits that exist on an NFT. This model
adds up the rarities of all traits and divides the result by the total number of traits to obtain a score
representing the entire NFT item. Although this model takes the overall rarity of the traits into account, it
“dilutes” the rarity values of those traits that might be extremely rare.
The Statistical Rarity model multiplies the rarity percentages of all traits to calculate the overall rarity of
an NFT. However, this model does not measure the NFT’s rarity in the specified collection of the NFT. If
there are two two-trait NFT items, one with a rarest trait and a least rare trait, the other with two somewhat
rare traits, then this model gives the latter a higher rarity value.
The Rarity Score model ¹⁸ is developed by the founder of Rarity.Tool. The model first takes the
inverse of the rarity percentage of each trait as the rarity score of every single trait. Then the model adds up
each rarity score calculated previously as the overall rarity score for the entire NFT item. This model not only
considers all the traits of a single NFT item, but also measures the NFT’s rarity in the specified
collection. We, therefore, use this model in our work to calculate the rarity score.
NER Creation Process
Our NFT creation, called NER (NFT-Enabled Robin) , combines a real-world furry toy with a digitally generative
3D model (see Figure 2).
The following subsections provide details of the design process: (1) Converting a physical bird into a virtual
bird via NFT. (2) Designing a virtual NFT trait and its rarity model. (3) Dynamically generating cloth in
response to the change of mobile sensor data.
Physical and Virtual Birds with NFC
Figure 3 shows the workflow for transforming a physical bird to a virtual bird NFT. First, we store a physical
NFC card in a small bag on the back of each physical bird (see Figure 2). Then we enter different redemption
codes into different NFC cards to ensure every code is unique and hard to replace. Second, NER collectors can
obtain the redemption code by tapping their phones with an NFC card and entering our app to paste the code (see
the left image of Figure 4). Our server receives the signal from the user and directly detects the user’s
internet status to ensure each redemption code to be redeemed only once. If the user’s phone does not
connect to the internet, the NER app displays a warning (see the middle image of Figure 4). If the user’s
phone is connected, the NER app shows a shaking egg (see the right image of Figure 4). Third, our server checks
the status of each redemption code; if the code is the 1st time redeemed, the NER app generates a unique virtual
bird NFT; if the code is already redeemed, our app directly displays the existing NFT.
Virtual NFT trait design with a rarity model
A virtual NFT artwork can be divided into two parts: 3D mesh rendering with a rarity model, and 2D patterns
generated by Shader with multimodal sensors which will be discussed in the next subsection.
Our design architecture considers different parts of virtual traits from top down (See Figure 5).
Both the beak color and feet color have two variations. The one with a higher rarity among all the variations
holds the rarity percentage of 30%, and the other holds the rarity percentage of 70%. Hairstyles have three
different trait variations. Ranking from the rarest hairstyle to the most common one, the percentages of owning
them are 15%, 30%, 55%, respectively. For those traits (hair color, eyes pattern, background color) with four
variations, we define their rarity percentages, from low to high, as 10%, 20%, 35%, 35%, respectively (See
Figure 6). With these six traits, NER has 768 variations in total.
To illustrate how to apply the above Rarity Score Model to calculate the rarity score for our NFT item, we
choose a bird to represent the appearance (see the 4th image of Figure 8). This bird has a hairstyle with the
rarity percentage of 15%, a hair color with the rarity percentage of 10%, an eyes pattern with the rarity
percentage of 35%, a beak color with the rarity percentage of 30%, a background color with the rarity percentage
of 35%, and the feet color with the rarity percentage of 70%. The single rarity score of each trait can be
calculated by taking the inverse of that trait. For example, the single rarity score of its hairstyle is 1/15%
≈ 6.67. Using this method, 15% one may calculate the individual trait scores of all the traits and sum the
results up to obtain the total score of this NFT item:
Dynamically generating cloth
To generate the 2D cloth texture, we use multimodal sensors to obtain mobile self-tracking data to drive the
generation.
As McLuhan stated in the book Understanding Media: The Extensions of Man ¹¹, the opinions of
media are the extensions of human, and reflected in all aspects of life in the digital era. Among different
media, “mobile technologies have become an important part of our lives as they are capable of representing
an extension of our physical selves” ⁸. Thus, we use the user’s mobile data to represent part
of his/her self-portrait ¹⁶ in the virtual world.
To protect the privacy of our users, NER does not read and track personal data from their phones. Instead, it
receives four types of data that affect 4 parts of the generation. Figure 7 shows the four types of data (i.e.,
three axes of gyroscope data and acceleration data) and the corresponding four aspects of the generation (i.e.,
the red, green, and blue channels of the visual pattern and its change speed).
Specifically, we receive two primary data types from the phone: gyroscope and acceleration. The gyroscope data
influences color, while acceleration data influences the texture moving speed. More specifically, gyroscopes X,
Y, and Z controls red, green, and blue colors respectively.
Figure 8 shows a series of generated results with rarity scores by different users.
Technical Implementation
The model of the NFT Bird is built with different pieces of 3D modeling software. The models of the bird and a
series of hairs are created in Zbrush and Maya is used to modify and add UV maps to them. The fabric model is
created using Marvelous Designer. After preparing the model and UV maps for the NFT bird, we use Substance
Painter to paint a range of basic colors and apply different materials to the model. The model is then sent to
Unity for the next stage of work: rendering and generation. In order to generate our Multimodal NFT with
different mobile sensors, we developed a mobile application in Unity that contains three phases of operations:
(1) Redemption: users can enter their redemption codes for generating the NFT. (2) Generation: users generate
NFTs with their mobile sensors. (3) Saving: the user data of the generated NFT is stored for future review.
Redemption
In this phase, users must sign up and log in with their email addresses, and then enter a valid redemption code
(see the left image of Figure 4) to ensure that the virtual NFTs generated are exclusively linked to their
accounts. To verify whether a user has the chance to generate a virtual NFT, we produce 1000 unique redemption
codes and store each of them in an NFC card placed in the bag of each physical NFT bird. A redemption code
consists of 10 characters, including digital numbers and both uppercase and lowercase letters, with over
8×1017 distinct combinations. We store all redemption codes on the server and keep track of their
states. Whenever a redemption code is used, its state is changed from ’Unused’ to
’Redeemed’ to ensure that each redemption code can only be redeemed once to generate one virtual NFT
bird. A used code is stored under the account which has applied the redemption so that the specific account can
review the NFT generated with the code.
Generation
The generation phase begins right after the user enters a valid and unused redemption code in the Redemption
Phase. In this phase, NER first lets users personalize the shader for the cloth of the virtual NFT birds, and
then randomly decides the traits for the user according to the rarity percentage of the traits.
Shader Personalization
The shader we use for the cloth of the virtual NFT is inspired by a shader ”70s Melt” ¹
written in OpenGL Shading Language (GLSL) in an online shader community, Shadertoy ²¹ . Based on GLSL,
we develop our shader in High-Level Shader Language (HLSL) acceptable by Unity. We add to this shader four
properties, that can be set by the data gathered from the sensors. The first three properties drive the
personalization of the color, determining how much red, green, and blue are applied in the shader, respectively.
The last property controls how fast the pattern on the shader changes. We use the gyroscope sensor to set the
first three properties related to the color. A gyroscope sensor can measure the angular velocity of the device.
As shown in Figure 9, we define the axis parallel to the long side of the phone screen as the x-axis, the axis
parallel to the short side of the screen as the y-axis, and the axis perpendicular to the screen as the z-axis.
The device rotation around the x, y, z-axis determines the amount of red, green, and blue of the shader
respectively. For the last property related to the moving and changing speed of the pattern, we use the
accelerometer sensor to control its value. The speed is positively related to the magnitude value of the
acceleration. The faster the device moves, the bigger the value is. To let users personalize the shader with
their mobile sensors, our instructions in the Generation Phase (see the right image of Figure 4) tell users to
shake their phones for two seconds to allow sensors to gather data.
Trait Generation
After the user has personalized the cloth, NER randomly determines what trait variations the NFT has based on
the trait rarity percentage. For example, the trait rarity percentage of the hairstyle of the NFT in Figure 5 is
15%. If the system randomly generates a floating-point number from 0 to 1, only a value between 0 to 0.15
determines that the NFT can have this hairstyle. Other values correspond to other hairstyle variations.
Saving
The Saving Phase starts once the generation of the NFT is done. In this phase, all data related to the shader
properties and the trait generation from the previous phase is written in a JSON file and stored on the server
in the logged-in account. If the user of this account enters an already redeemed code again in the Redemption
Phase, all the stored data is fetched to directly display the generated NFT to the user, skipping the Generation
Phase.
Conclusions and Future Work
First, the article reviews existing Multimodal Generative NFTs artworks and existing NFT Rarity models.
Next, we present a design architecture for creating a physical-virtual Multimodal NFT with generative patterns
and rarity scores, called NER (NFT-Enabled Robin). It contains
Transformation methods from a physical furry toy to virtual NFT generation with an NFC card;
A 3D Virtual NFT system with different traits and rarity scores;
Dynamic 2D patterns on a 3D mesh that changes in response to mobile sensor data.
Our NER is able to collect all traits and sensor data. These data can be sent to the NFT platform in real time
to use We-bGL to convert existing data into the blockchain.
Future Work
In the future, we plan to establish a NER NFT collector community as a symbiosis ecosystem. The vitality and
longevity of NFT artworks are highly related to their own community of collectors. Good NFT communities usually
contain users’ spontaneous secondary creations, like CryptoKitties ⁵ and SpaceDAO ²². We
may organize offline human-computer interactive workshops to let NER collectors bring their physical birds with
accessories and enable them to build different accessories and animations online.
We also wish to record the virtual birds generated by users and send them to a metaverse platform as unique
digital assets to maintain users’ emotional connection.
As mentioned earlier, the first version of NER has 768 variations. This is because it is convenient for the
factory to make the physical bird toys and more controlled distribution for first-time NFT collectors. We expect
the later version of NER to increase the number of traits and the number of physical birds.
The authors believe that “NFTs can be integrated with physical assets, properties, securities and
insurance, thereby promoting diversified applications in art markets” ⁷. Generative art and AI art,
such as ChatGPT ⁴ and Midjourney ¹⁴, will play a significant role in future NFTs and their
visual aesthetics and artistic values will be continuously improving. More details of our work can be viewed in
the supplementary video at https://we.tl/t-x2G7ksWbDJ, that demonstrates to a clear method of creativity to our
collectors and inspire other creators.
7 Y. Guo, Q. Liu, J. Chen, W. Xue, H. Jensen, F. Rosas, J. Shaw, X. Wu, J. Zhang, J. Xu, Pathway
to future symbiotic creativity, 2022, arXiv preprint arXiv:2209.02388.
8 J. Harkin, Mobilisation: The growing public interest in mobile
technology, Demos, 2003.
11 M. McLuhan, Understanding media: The extensions of man, MIT press, 1994.
12 A. Mekacher, A. Bracci, M. Nadini, M. Martino, L. Alessandretti, L. M. Aiello, A. Baronchelli,
Heterogeneous rarity patterns drive price dynamics in nft collections. Scientific reports 12(1),
2022, 1–9.
16 S. Park, Multimodal data portrait for representing mobile phone use behavior, In
Proceedings of the 25th International Symposium on Electronic Art (ISEA), 2019, 36–43.
25 Z. Wu, and L. Huang, Mimicry: Genetic-algorithm-based real-time system of virtual insects
in a living environment-a new and altered nature, Proceedings of the ACM on Computer Graphics and
Interactive Techniques 4(2), 2021, 1–8.
26 K. Zhang, J. Yu, The computer-based generation of fonts in the style of kandinsky.
Leonardo 54(4), 2021, 437–443.
27 K. Zhang, S. Harrell, X. Ji, Computational aesthetics: on the complexity of computer-generated
paintings, Leonardo 45(3), 2012, 243–248.
Authors Biographies
Ziwei Wu is a media artist and researcher who was born in Shenzhen. She is a Ph.D. student
in Computational Media and Arts (CMA) at HKUST. Her artworks are mainly based on biology, science, and the
influence on society by using a range of media. She is a Lumen prize winner; Batsford prize winner. Her
research was published in the SIGGRAPH Art Program and EVA London. She has exhibited at international venues,
including CYFEST in Southeastern Europe, Norwegian BioArt Arena (NOBA)-Vitenparken in Norwegian, Watermans
Gallery and Cello Factory in London, NeurIPS AI Art Gallery, ArtMachine2 in Run Run Shaw Creative Media Centre
in Hong Kong and so on.
Danlu Fei is a game developer and researcher. She owns a bachelor degree in Computer Science
- Computer game. She worked as a gameplay programmer intern at several well-known game companies, including
Ubisoft, Behaviour Interactive, and Netease. She is pursuing her Ph.D degree in Computational Media and Arts
(CMA) at HKUST. Her research interest is mainly around procedural content generation (PCG) in games,
especially AI-based PCG. Her research was published in the CHI Play conference proceedings.
Xinyu MA is an art practitioner residing in Shenzhen and Hangzhou, who holds a
bachelor’s degree in sculpture and a master’s in fine arts. Currently, he is pursuing a master of
philosophy degree at the Hong Kong University of Science and Technology (Guangzhou). His work focuses on
experimenting with various forms of public system structures, investigating their modes of operation in an
individual, fictional, and fictitious way. In addition, he also researches the topics of the public sphere,
philosophy of technology, urban space, community, play, and political aesthetics.
Mingming Fan is an Assistant Professor in the Computational Media and Arts (CMA) Thrust at
The Hong Kong University of Science and Technology (Guangzhou) and in the Division of Integrative Systems and
Design at The Hong Kong University of Science and Technology. Dr. Fan is the director of the Accessible
and Pervasive User EXperience (APEX) Group, which conducts research at the intersection of
Human-Computer Interaction (HCI), AI, and Aging & Accessibility. Dr. Fan’s research has been
published in top-tier venues in HCI (e.g., CHI, TOCHI) and received many paper awards, including the best
paper award at CHI and the best paper honorable mention award at UbiComp. He is a Siebel Scholar. Homepage:
https://www.mingmingfan.com/
Kang Zhang is Professor of Computational Media and Arts, Hong Kong University of Science and
Technology (Guangzhou) and of Division of Emerging Interdisciplinary Areas at the Hong Kong University of
Science and Technology, and Professor Emeritus of Computer Science at the University of Texas at Dallas. He
was a Fulbright Distinguished Chair and ACM Distinguished Speaker, and held academic positions in USA, Czech
Republic, Australia, and UK. Dr. Zhang has delivered keynotes in computer science, architecture, art, design,
and psychology areas, and won numerous art and design awards. Interdisciplinary research topics of
computational aesthetics and aesthetic computing, and their differences and relationships. Homepage:
https://cma.hkustgz.edu.cn/people/kang-zhang/
Towards a Relational Model of Co-located Interaction in Interactive Art
DanXu, MaartenH. Lamers, Edwinvan der Heide
LIACS, Leiden University Leiden, the Netherlands d.xu@liacs.leidenuniv.nl,
m.h.lamers@liacs.leidenuniv.nl, e.f.van.der.heide@liacs.leidenuniv.nl
Abstract
Co-located interaction refers to situations in which there are two or more co-located audience members
interacting with an interactive artwork and the interaction between the audience members is integral to the
artwork. To better describe and understand co-located interaction, we propose a relational model of co-located
interaction with an emphasis on how the interacting elements relate to and influence each other based on the
actions performed by the elements and the various forms of communication at play. We explain the key concepts in
the model and demonstrate its application with an example artwork. Our model provides a systematic approach to
de- scribe, compare, and generate new forms of co-located interaction and audience-artwork interaction in
general.
Active audience participation is integral to interactive art. Instead of creating static artistic objects,
artists aim to construct dynamic 'art systems' capable of acting and responding to the audience and the
environment. ¹ Interactive art transforms the audience from passive viewers into active participants,
challenging the traditional conceptions of the roles of artwork, audience, and artist. Situated at the interplay
of art, science, and technology, it explores and recontextualizes new technologies, contributing to novel forms
of communication and the emergence of a multidisciplinary field of research. ²
Meanwhile, the majority of research in interactive art focuses on describing the interaction between the
audience and the artwork, a concept commonly referred to as audience-artwork interaction. ³ In a
comprehensive study of audience-artwork interaction within interactive arts, Schraffenberger and van der Heide
conducted a review of existing models and frameworks that delve into the various characteristics and types of
interaction occurring between the artwork and the audience. ³, ⁴ They further proposed a dialogue
model to describe audience-artwork interaction, similar to a dialogue between two individuals, where both
parties not only respond to each other's propositions but also have the freedom to change the topic or interrupt
one another. The dialogue model portrays audience-artwork interaction as a dynamic process, granting both the
audience and the artwork autonomy while reacting to each other. As such, the artwork and the audience neither
act in a random manner nor completely control each other.
However, as many interactive artworks are exhibited in museums and other public places, there are usually more
than one audience members present, and some artworks require the presence of multiple audience members for the
intended interaction to take place. The artwork Boundary Functions by Scott Snibbe is such an example. ⁵
In Boundary Functions, the audience are cast as seeds for generating a Voronoi diagram projected on the floor,
which divides the floor into areas that are closest to each audience member. The artwork does not respond when
there is only one person, as a Voronoi diagram for one seed covers the whole projection area. When there are two
or more people interacting with the artwork, the boundaries of each cell are determined by the distance between
each audience member and are continuously changing as they move closer or further away from each other. Here we
use the term co-located interaction to describe such interaction:
Co-located interaction takes place when two or more audience members participate in the interaction with an
interactive artwork, sharing both time and physical location. Furthermore, the presence of, and the
interaction between, the audience members is integral to the artwork.
Consequently, in accordance with this definition, co-located interaction encompasses both the interaction
between the audience and the artwork, as well as the interaction among audience members. It's important to note
that not all interactive artworks incorporate co-located interaction. Unlike a conventional dialogue between an
audience and an artwork, co-located interaction broadens the scope of interaction into a collective activity
involving multiple participants and various types of relationships.
In this paper, we propose an approach to describe and model co-located interaction in interactive art. A
comprehensible interaction model can help us better understand, compare, evaluate, and design interactive
experiences and systems. ⁶ Furthermore, such a model can offer templates to assist artists in developing
novel forms of co-located interaction, aid curators and experts in the comparison and classification of
artworks, and, most importantly, contribute to the identification of patterns within co-located interaction,
shedding light on how interactivity manifests itself in such contexts and revealing new areas for research.
In the next section, we review existing approaches for modelling co-located interaction, discussing both their
contributions and limitations. Following this, we introduce our model for co-located interaction, explain the
key concepts, their properties, and provide guidelines for its application. Subsequently, we show how our
approach can be applied in describing and analysing co-located interaction with an example artwork. Our general
discussion, reflection and conclusion can be found at the end of this paper.
Related Works
Existing Models of Co-located Interaction
One of the main contributions in understanding co-located interaction in interactive art is the doctoral work
of Oussama Mubarak. ⁷ In his research, Mubarak presents a taxonomy of co-located interaction in art
installations based on factors influencing both the individual and the collective experience. These factors are:
Scale: the number of simultaneous active participants. And it is classed into small (≤ 10), medium (11 -
100), and large (> 100).
Interaction modality: the method by which the audience interacts with the artwork. The different modalities
are direct (direct physical manipulation), facilitated (using individual remote devices) and ambient (the
artwork captures external information through non-invasive technologies).
Input and output distribution: the distribution of the input and output devices. The different
distributions are centralized (the input or output is performed on shared devices), partially distributed
(the input or output is split or duplicated on shared and individual devices) and fully distributed (the
input or output is fully distributed across individual devices).
Feedback attributability: how easily the audience can recognize the effect of their actions in the artwork,
which is indicated as low, medium, high and variable.
Activity type: the type of activity the audience can engage in the artwork. The different types are
solitary, collaborative, or competitive.
Participation symmetry: all audience interact with the artwork equally (symmetric) or some audience interact
with the artwork differently from others (asymmetric).
This taxonomy points out some interesting aspects about the experience of co-located interaction and provides a
set of vocabulary to describe them. However, it concerns less with specifying the relationships between the
audience and the artwork, as well as between the audience members. Additionally, Mubarak proposes two approaches
to model colocated interaction that attempts to capture this.
The first approach focuses on visualizing the system layouts of art installations. In this approach, the main
elements in a model are shared or individual interfaces that are specified as either input, output, or both. The
number of each interface is noted, and the organization of the individual interfaces are classified as 1) free,
there is no specific order for accessing the interfaces, 2) in a queue, the interfaces are accessed by the
installation in a chronological order, 3) in a looped queue, the art installation loops through all the
interfaces. The data flow between the individual interfaces and shared interfaces are indicated as either
unidirectional, bidirectional, or symmetrical correspondence meaning all interfaces share the same information.
This approach provides a concise language to show the technical layout of artworks, yet it offers limited
insight into the roles each interface plays in the interaction and their connections to the audience. More
importantly, it does not account for the interaction between audience members, which is essential to co-located
interaction.
Acknowledging some of the limitations discussed above, Mubarak proposes a second approach based on Petri nets.
A Petri net is often used to model information flow in systems with concurrent and asynchronous events. In this
approach, Mubarak describes the spatial and material aspects of the artwork, the audience activities in the
interaction as well as the transitions between them triggered by the actions of the audience. Comparing with the
first approach, this approach is more inclusive and expressive of the different activities in a co-located
interaction, as Mubarak noted that it “can be used to model co-located human-human, human-machine, as well
as human-machine-human interactions around art installations”. ⁷, p.110 However, due to its early
developmental phase, this approach tends to produce highly specific and complex model of an individual artwork,
which limits its practical use and renders it unsuitable for comparing different co-located interactions.
Interaction Model for Relational Interactive Art
When contemplating the promotion of audience participation in interactive artworks, Cabrita and Bernardes
connects interactive arts with relational aesthetics, a concept centered on fostering social connections among
audience members through an artwork's aesthetic properties. ⁸ They coined the term “relational
interactive art” to describe interactive artworks that strive to forge and enrich social bonds between the
audience members and proposed an interaction model for such artworks based on feedback loops (see Figure 1).
In a feedback loop between a human and a computer, the actions of the human send signal to the computer, which
gives feedback that guides the further actions of the human in a continuous loop. In relational interactive art,
the model includes another human participant that interacts both with the human and the computer in similar
feedback loops. Cabrita and Bernardes further explain that the human-human and human-computer interaction are
equally important and always present in relational interactive artworks. The actions of the audience are
interdependent, and these actions are interpreted by the computer as a collaborative behaviour that
“guides its response and shapes the next instance of human action” ⁸. However, they did not
illustrate any application of the model.
Here we see a great overlap between Cabrita and Bernardes account of relational interactive art and co-located
interactive artworks. We agree that the interaction between the audience members and the interaction between the
audience and the artwork are of equal importance and the two forms of interaction also inform and influence each
other. However, the feedback loop model is often used in HCI to describe how a user can control a system and
neglects the possibilities for novel behaviours and independent actions, whereas if we understand interaction as
a dialogue, there is freedom on both sides to break the loop and go in another direction. Moreover, contrary to
Mubarak’s approach based on Petri nets, the feedback model runs a risk of oversimplification, and cannot
capture the characteristics of individual interactions.
Considerations
Taken together, the existing models of co-located interaction tend to be either too generic or too specific. A
useful model should be descriptive, in that it can provide enough structural information to describe the
individual system or situation being modelled; comparative, in that it contains metrics for comparing different
systems or situations; and generative, in that it can guide the creation of new systems or situations. ⁶
Meanwhile, although interaction is by nature a relational process, the existing attempts to model co-located
interaction do not sufficiently describe the relationships between audience and the artwork, and between the
audience members. In all above-mentioned models, the term ‘interaction’ is used to describe the
activities between the audience members and between the audience and the artwork. In discussing forms of
interaction in the context of interactive art installations, Ahmed pointed out the difference between
interaction and communication: “In communication, the receiver may or may not respond, whereas in
interaction, there is requirement of a response for it to be an ‘inter’action”. ⁹ This
also coincides with our understanding of interactivity as a form of mutual responsiveness. Although the
experience of co-located interaction is interactive, this does not mean that each interacting partner should be
mutually responsive to each other.
If the interaction between an artwork and an audience can be seen as a dialogue of actions and reactions, we
can try to understand co-located interaction by looking at the actions (and reactions) of all the interacting
participants. By following the directions of actions, we can start to identify the different forms of
communication at play. In doing so, we can specify a network of influences between the audience members and the
artwork that serves as the foundation for understanding co-located interaction.
Moreover, the same action of an interacting element can yield different outcomes in different interactive
artworks. For instance, the waving of an audience member's arm may change the colour of a visual display in one
artwork, while in another, the same gesture may cause the artwork to move away. Therefore, it is crucial to
consider the function(s) of an action performed by an interacting element and how it affects and relates to
other elements, or the role(s) of the action in the specific context of a co-located interaction. Existing
models of co-located interaction have generalized the role(s) of actions as either ‘triggers’ for
activity transitions or ‘feedback’ while ignoring the fact that an action can also be to initiate
the interaction or a new proposition. Therefore, we posit that specifying the role(s) of the various actions
performed by the elements is critical for comprehending the dynamics of co-located interaction and further
identifying the role(s) of individual elements. This not only aids us in distinguishing and comparing different
artworks but also unveils patterns that can stimulate the creation of novel forms of co-located interaction.
Therefore, we need a balanced model of co-located interaction that can describe the various forms of
communication between the audience and the artwork, as well as between the audience members, while considering
the roles of the actions of the audience and the artwork. Next, we present our attempt to model co-located
interaction, taking these considerations into account.
A Relational Model of Co-located Interaction
Here, we introduce a relational model for describing co-located interaction. A visual diagram of this
relational model is provided in Figure 2. The relational model begins by identifying the elements with distinct
behaviours and proper within a co-located interaction. An element is regarded as an independent actor in the
interaction, with their actions forming the foundation of the interaction. Common elements encountered in
co-located interactions are audience members or art systems. The model is relational in that it does not only
depict the behaviours and properties of individual elements, but also specify how their actions influence and
relate to other elements through various forms of communication. In the following sections, we will elaborate on
the attributes we use to describe the elements and the different forms of communication with the relational
model. For each element, we describe its:
Type
As mentioned before, there are two common types of elements in co-located interaction: the audience and the art
system. In Cabrita and Bernardes’ model of relational interactive art, they view the interactive system of
the artwork as simply a 'computer' ⁸. However, we find this term misleading, as an interactive artwork
often involves more than just a computer; other components of the artwork can be equally, if not more,
significant in the interaction. Additionally, interactive artworks do not always require the use of computers.
Therefore, we propose the term 'art system' to describe an independent functional unit within an interactive
artwork. An art system can consist of input units, output units, and processing units, similar to the computer
element in Cabrita and Bernardes’ model, or it could be made of physical materials that react to the
environment and can be used by the audience. An interactive artwork might include a single art system or several
art systems, each with its own behaviour and the capacity to establish connections with the audience. When
multiple art systems exhibit distinct behaviours, we need to analyse them individually.
The audience element refers to the individual audience members. In an interactive artwork, there are often
audience members interacting with the artwork while others remain in the background to observe the interaction,
sometimes they have the freedom to switch between the two modes of participation. As our focus is on describing
interaction, the model includes only the audience members who are directly participating in the artwork and
whose actions have an impact on the art system and other audience members. Looking at the definition of
co-located interaction, there are often two or more audience members in a co-located interaction. Similar to the
case of the art system, when different audience members exhibit distinct behaviours and have different functions
in the interaction, we need to specify each of them with the model.
In certain instances of interactive artworks, the art system is influenced not only by the audience but also by
other environmental factors ¹. For example, an art system might change its behavioural states in response
to the exhibition space's temperature or internet traffic data. When such factors come into play, we categorize
them as types of environmental elements.
Components
The components of an element refer to the devices, materials, and apparatuses that the element incorporates and
employs during interactions. If an art system is computer-based, it typically encompasses both hardware
components, like sensors, computers, actuators, or displays, to sense the environment and execute actions, and
software components such as tracking and control programs for processing data and generating commands. For the
audience, their interaction often involves using their bodies to perceive, act, express, and communicate with
both the art system and each other. Depending on the interaction modalities, they can also utilize objects like
accessories or personal devices. The components of an element describe its material basis and help us
differentiate it from other elements.
Count
Count refers to the number of elements that can simultaneously participate in the interaction. This is similar
to the 'scale' attribute in Mubarak's taxonomy; however, he only focuses on the number of simultaneous active
audience members, whereas we extend this notion to all types of elements. In certain instances, there might be a
requirement for a minimum number of elements in a co-located interaction. For example, the previously mentioned
artwork 'Boundary Functions' requires the participation of a minimum of two audience members. Moreover, a
co-located interaction can take place among a fixed number of elements, or the elements can join or leave the
interaction at any moment. In the latter scenario, we often denote the minimal required number of elements for
the interaction with a '+' to indicate potential additions.
Actions
An action is a concrete step performed by an element or a group of elements to bring about changes within
themselves or in the surrounding environment. An action can be a movement, such walking or pressing a button, or
it can be an update on a display mode, such as presenting data on a screen or generating an audio. In the
relational model, we view actions as the foundational components that elements use to establish connections with
and influence other elements. Consequently, the actions included in the description of an interaction are always
directed towards other elements. Activities that are receptive, such as sensing or observing are not explicitly
mentioned. The direction of an action can manifest itself in two ways: first, the acting element intentionally
performs an action directed at the receiving element; second, the receiving element actively captures aspects of
the action performed by the acting element, even if the action is not initially intended for the interaction.
The latter is often the case when the art system detects and responds to specific actions of the audience.
In a taxonomy of interactive artworks made in the context of the Prix Ars Electronica, Kwastek points out that
the key actions that may be performed by the audience are: “observe, explore, activate, control, select,
participate, navigate, leave traces, co-author, collaborate, exchange information and create” and the
corresponding actions made by the art system are “monitor, serve as an instrument, document, enhance
perception, offer a game, enable communication, visualize, sonify, transform, store, immerse, process, mediate
and tell/narrate”. ¹⁰ However, we find these terms are more suitable to describe the functions
of an action instead of the concrete actions performed by the elements. We agree that it is important to specify
the functions and roles of an action in the interaction. Yet when it comes to describe the co-located
interaction in a specific artwork, we first need to identify the individual, concrete actions performed by the
elements.
Communication
As we discussed earlier, a form of communication is created when an element performs an action directed at
another element. For each form of communication, we specify: To - To which element the
action is directed at. The same action can be directed at different elements and creates different forms of
communication.
Means - The means of a communication refers to how the action reaches the receiving element.
We identify two different means of communication: • Direct: the action is performed to the receiving
element directly. There is no intermediate element. • Via (intermediate element): The action is
performed to an intermediate element and reaches the receiving element via the intermediate element. This type
of communication is frequently termed mediated communication.
Configuration - In a communication, an action can be carried out by one or multiple
elements together and it can be directed at one or multiple elements simultaneously. Configuration refers to the
arrangement of the elements at both ends of the communication. We consider the possible configurations in a
co-located interaction as: • One to one: from one element to one element only. • One to many:
from one element to one or more elements. • Many to one: from one or more elements to one
element. • Many to many: from one or more elements to one or more elements.
Besides the number of elements, we also identify two settings in which the communication takes place: private
and public. In a private setting, the communication can only be perceived by the communicating elements. In a
public setting, the communication can also be perceived by other elements and/or observers.
Count - Here the count refers to the number communications that can take place in parallel. It
is important to note that this does not necessarily imply that all communications must happen strictly
simultaneously. Rather, we are examining the capacity of the element to perform the action. For instance, an art
system may detect the audience members one at a time and emit an audio response. While the communication of the
audio response might be sequential, the art system has the ability to repeatedly perform such action. Therefore,
we consider that there are multiple instances of such communications happening in parallel.
Role(s) of the Action
For every action performed by an element, it's essential to define its role(s) within the interaction. The
role(s) of an action pertain to the function(s) it serves and how it relates to other actions performed by the
same or different elements in the interaction. An action can be initiated by an element or triggered by the
actions of other elements. In the latter scenario, it functions as a response and can be termed a
‘reaction’, As we discussed before, the ‘actions’ identified by Kwastek provide a rich
vocabulary to describe the functions of an action in an interaction. However, in Kwastek’s description,
the actions of the audience tend to play more active roles such as “explore, activate, control”,
whereas the artwork tends to be at the service of the audience. In our view, we do not make any arbitrary
differentiation between the roles of the audience and those of the artwork. We consider that both can perform
the same roles, and that this perspective opens up possibilities to create new forms of interaction. As an
action can create multiple forms of communication, it can also serve diverse functions across different
communications.
Application
Given a co-located interactive artwork, we can apply the model to describe its co-located interaction by
inspecting the documentations and description of the artwork, as well as observing the actual interaction taking
place. First, we need to identify how many distinct element profiles to include in the model. We can do so by
examine the elements according to the general types (e.g., art system, audience, environment) and further
distinguish whether the elements of the same type exhibit distinctive behaviours that require separate analysis.
For the element(s) sharing the same profile, we can then specify its properties and behaviours based on its
‘components’, ‘counts’ and ‘actions’. Following each actions of the element,
we can start to describe the various forms of communication created by the element: to which element(s) the
action is done to (‘to’), whether the information is communicated directly or via other element(s)
(‘means’), how many elements are involved in both ends of the communication and whether it is in a
public or private setting (‘configuration’), how many of such communications can take place
concurrently (‘count’) and what role(s) the action plays in the context of the interaction
(‘role(s) of the action’). This should give us an adequate description of a co-located interaction
focusing on how the elements relate to and influence each other.
Example Artwork
Body Movies (2001) by Rafael Lozano-Hemmer
Body Movies is an interactive projection installation for public spaces (see Figure 3). ¹¹ A set
of portraits is projected on a surface and washed out by lights positioned at a distance on the floor. The
audience enters the interaction space and their shadows are displayed on the projection surface to reveal the
portraits. The artwork tracks the edges of the shadows, and once they overlap with a portrait, a hotspot is
activated for a few seconds and an audio track is triggered. When all portraits are revealed, the artwork blacks
out the projection and displays a new set of portraits at the different locations from the previous ones.
Meanwhile, the software interface is displayed in real-time via a plasma display next to the projection.
Apply the Relational Model
In Body Movies, there is one art system consisting of: projector, portrait, projection surface, Xenon light,
plasma display, camera, software program, speaker, robotic controller. As all audience members participate in
the interaction equally, we can describe them in one element profile. One or more audience members can
participate at the same time, and they mainly use their bodies to interact. A visual diagram of the co-located
interaction in Body Movies based on the relational model can be seen in Figure 4.
There is a form of direct many (1+) to one communication from the audience to the art system in a public
setting and there can be one or more of such communications. In this, one or more audience members can cast a
shadow on the projection surface and the camera captures and tracks the shadow contours, which allows the
audience to initiate and participate in the interaction.
There are four forms of direct one to many (1+) communication from the art system to the audience in a public
setting. First, the art system displays the shadows of the audience as a response, to provide feedback about
their movements, and a stage for communication. And there can be one or more of such communications. Second, the
art system also displays the tracking interface to provide feedback and inform the audience of their performance
and the states of interaction. Third, when a shadow overlaps with a portrait, the art system activates a hotspot
and plays an audio clip as a response and to provide feedback about the achievement. And there can be one or
more of such communications. Last, once all hotspots are activated, the art system blacks out the projection and
updates the portraits as a response and initiates a new session of interaction.
Throughout the interaction, the audience can communicate with each other through their shadows. It allows them
to express and perform with gestures and movements. This creates a form of many (1+) to many (1+) communication
via the art system in a public setting and there can be one or more of such communication forms. Due to their
co-location, the audience can also converse with each other both verbally and non-verbally to exchange
information and coordinate actions. As there are two or more audience members, there can be one or more of such
direct many (1+) to many (1+) communications in a public setting.
Discussion
In this paper we propose a relational model to describe co-located interaction in interactive art. Our approach
drew several connections to concepts discussed in Mubarak’s taxonomy, such as scale and participation
symmetry. We situate them in context with relation to other elements and supplement the description of the
individual element and their communication with each other. Compared to Mubarak’s models, our approach
looks beyond the technical aspects of the artwork, and not only describes the activities and actions of the
audience and the art system, but also specifies the relationships between them. The model also provides a
systematic method that can be used to easily compare different artworks. Instead of viewing all forms of
communication and potential interaction as feedback loops, we specify the organizations of the elements, the
flow of information and the roles of the actions performed by the elements. In doing so, we have a better
understanding of how the elements influence each other and distinguish different forms of co-located
interaction.
The descriptive capacity of the relational model is illustrated in the example of Body Movies. Given the
description of the co-located interaction in Body Movies generated using the model, we can see that there are
multiple forms of communication between the audience and the art system. The audience members initiate the
interaction by casting their shadows on the projection surface. In response to this, the art system performs
multiple responses, each with different functions and takes place corresponding to different scenarios, for
instance, it plays a sound when the audience members activate a hotspot or updates the portraits when all
hotspots are activated. These various forms of communication enrich the interactive experience and engage the
audience to explore the possibilities the artwork affords.
Moreover, we also observe two forms of communication between the audience members. They can communicate with
each other directly due to their co-location and/or via the art system using their shadows. The shadow play
allows the audience to communicate with gestures while concealing their appearance. This anonymity lowers the
threshold for public performance. Additionally, they can move closer or further away from the lights to
drastically alter the size of their shadows, which creates dramatic and symbolic effects in the shadow play.
These forms of communication can either facilitate the audience's interaction with the art system or become an
aesthetic experience in itself.
Meanwhile, our approach also provides a template to conceive new forms of co-located interaction. For instance,
we can combine different forms of communication or swap the roles of actions performed by the elements to create
new structures for co-located interaction. Given the identified forms of communication, we can also start
thinking about new forms of communication. For instance, an audience-mediated communication in which an audience
member can interact with an art system through another audience member, or an audience-group communication in
which an audience member acts on and responds to other audience members and art systems as a group. Besides, we
can also speculate new ways for the different forms of communication to influence each other. For instance, a
mediated communication can disrupt or supplement to a direct communication.
Future Works
In this paper we can only partially illustrate how our approach can be applied to analyse and describe
co-located interaction due to space constraints. In a follow-up publication, we will analyse and compare a
variety of co-located interactive works to further test the capabilities of our approach and the diversity of
co-located interaction.
For the current approach, we identified several common roles of actions in co-located interaction. This serves
as a starting point to see co-located interaction through the lens of these concepts and points at a direction
for future studies to examine and develop more conclusive inventories of the roles, which can be applied to
describe, compare, and conceive new forms of co-located interaction.
As our approach focuses on examining the relational aspect of a co-located interaction, it is limited in
accounting for other aspects such as the scale and feedback attributability according to Mubarak’s
taxonomy. Moreover, it cannot show the temporal development of an interaction. The audience may be more
exploratory at the onset, as they discover and master the interaction mechanism they may act and react
differently. Future studies can combine different frame-works and taxonomies with our approach for a more
comprehensive description of co-located interaction.
Conclusion
Co-located interaction in interactive art provides challenges and opportunities for developing new forms of
interaction and relationships between the audience and artwork. In this paper, we present a relational approach
to model co-located interaction with a focus on describing the various forms of communication between the
interacting elements, while considering the effect of their actions in their context and relation to each other.
Looking at co-located interaction with a relational perspective helps us to better understand the dynamics of
co-located interaction and to conceive new forms of relation and organization between the audience and an art
system.
Furthermore, despite its original focus on co-located interaction, the relational model does not impose
specific prerequisites for the nature of the interaction or the elements under analysis. The model can also be
effectively extended to describe various other types of audience-artwork interaction. This extension includes
artworks featuring distributed audience participation and/or asynchronous communications, as well as
interactions involving diverse entities encompassing both human and non-human participants. We believe that the
relational model represents a pivotal step towards comprehending the complex network of relationships intrinsic
to interaction.
References
1 Stroud Cornock, Ernest Edmonds, “The Creative Process Where the Artist Is Amplified or
Superseded by the Computer,” Leonardo 6, No. 1, 1973, p.11, Doi: 10.2307/1572419.
2 Myounghoon Jeon, Rebecca Fiebrink, Ernest Edmonds, Damith Herath, “From Rituals to Magic:
Interactive Art and HCI of the Past, Present, and Future,” International Journal of Human Computer
Studies 131, 2019, 108, Doi: 10.1016/j.ijhcs.2019.06.005.
3 Hanna Schraffenberger, Edwin van der Heide, “Interaction Models for Audience-Artwork
Interaction: Current State and Future Directions,” In Arts and Technology, edited by Anthony L. Brooks,
101:127–35. Lecture Notes of the Institute for Computer Sciences, Social Informatics and
Telecommunications Engineering, Berlin, Heidelberg, Springer Berlin Heidelberg, 2012, Doi:
10.1007/978-3-642-33329-3_15.
4 Hanna Schraffenberger, Edwin Van der Heide, “Audience-Artwork Interaction,”
International Journal of Arts and Technology 8, No. 2, 2015, 91, Doi: 10.1504/IJART.2015.069550.
6 Michel Beaudouin-Lafon, “Instrumental Interaction: An Interaction Model for Designing
Post-WIMP User Interfaces,” In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, ACM Press, 2000, 446-453, Doi: 10.1145/332040.332473.
7 Oussama Mubarak, “Designing and Modeling Collective Co-Located Interactions for Art
Installations,” Ph.D. diss., Center for Studies and Research in Computing and Communications, CNAM
Paris, 2018.
8 Nuno Cabrita, Gilberto Bernardes, “Relational Interactive Art: A Framework for
Interaction in a Social Context,” In Expressive, edited by Angus Forbes and Lyn Bartram. Computational
Aesthetics in Graphics, Visualization, and Imaging, The Eurographics Association, 2016, Doi:
10.2312/EXP.20161264.
9 Salah Uddin Ahmed, “Interaction and Interactivity: In the Context of Digital Interactive
Art Installation,” In Human-Computer Interaction. Interaction in Context, edited by Masaaki Kurosu,
10902:241–57, Springer International Publishing, 2018, Doi: 10.1007/978-3-319-91244-8_20.
10 Katja Kwastek, “Interactivity – A Word in Process,” In The Art and Science of
Interface and Interaction Design, edited by Christa Sommerer, Lakhmi C. Jain, Laurent Mignonneau, Studies in
Computational Intelligence, 141:15–26, Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, Doi:
10.1007/978-3-540-79870-5_2.
Dan Xu is a creative researcher and PhD candidate at Leiden Institute of Advanced Computer Science (LIACS),
Leiden University, NL. Her current research explores co-located audience interaction in and through
interactive art. She holds a MSc in Media Technology from Leiden University and BSc in Applied Physics from
Beijing Institute of Technology.
Maarten H. Lamers is a creative computer scientist and Assistant Professor at Leiden University (LIACS).
Themes that reappear in his research and lecturing are artificial intelligence, human-robots interaction,
hybrid bio-digital systems, and scientific playfulness.
Edwin van der Heide is an artist and researcher in the field of sound, space and interaction, and a part-time
Assistant Professor at Leiden University (LIACS, ACPA). His work comprises installations, performances and
environments. The audience is located in the middle of the work and challenged to actively explore and
interact with the artwork.
VR CALLIGRAPHY Transposing Chinese calligraphy as choreographed movements into whole-body performances in VR
Shum,P. Y. S.&Klein,T.
School of Creative Media, City University of Hong Kong Hong Kong
Pysshum2-c@my.cityu.edu.hk, ktobias@cityu.edu.hk
Abstract
Current research in the area of digitizing Chinese calligraphy is primarily concerned with computer graphics
(CG), calligraphy ink/brush simulation, human-computer interaction (HCI), and automatic calligraphy generation.
In all above research fields, calligraphy is mainly expressed and represented in two dimensions as traditional
ink on paper. Even within recent contemporary calligraphic movements, there is little exploration into a true
spatialization of calligraphy beyond the flat medium.
This research articulates an evolution of Chinese calligraphy, from a physical two-dimensionality into a
multidimensional digital form via choreographed whole-body movements using VR (virtual reality) interfaces. We
present a series of experiments to test the transposition, creation, and presentation of the essential qualities
of calligraphy into a VR format. The aim is, through disseminating new virtual phenomenology in calligraphy, to
contribute to and extend the existing writing on the relation between calligraphy and technology as an art
entering into immersive environments and interactive virtual spaces. Thus, the research interrogates the
dimensionality of calligraphy to synthesize the notion of shufa (书法, the way of writing;
the Chinese word for calligraphy) from the perspective of a choreographed performance within VR.
Keywords
Chinese Calligraphy, Digital Craftsmanship, VR, HCI, Art and Technology, Performance Art, Body Movement,
Intangible Cultural Heritage.
This paper focuses on calligraphy as a creative art form. The Chinese word for calligraphy, shufa
(书法), can be translated as the way of writing. It is more than a technical act of
putting ink on paper; it is a core philosophy of Chinese culture, recognized by UNESCO (2009) and on the
Representative List of the Intangible Cultural Heritage of Humanity. Therefore, this paper searches
within the traditional aspects of calligraphy to determine the essential qualities of this art that can be
either retained or transferred into a digital, multidimensional format. Through our experiments, we test
specific notions of calligraphy with the utilization of originally-developed tools within VR.
This paper is structured into three parts. In the first part, we analyze Chinese calligraphy as an art form
that is creatively expressive. The philosophical context, and the potential transformation of tools from
physical to digital versions, are investigated via discussions of dao/qi and The Spirit of
Calligraphy. The expressive qualities of calligraphy are essential to its art and are therefore retained
within our transferal process. From an examination of traditional tools and media, we identify the unique
aspects of calligraphy that are to be transposed in our experiments: the dynamic strokes of ink written via
corresponding body movements. By establishing the limits of the physical tools and media, we propose how
calligraphy can potentially evolve in VR, reflected within an immaterial and time-based media.
The second part showcases our experiments of two artworks as investigations of calligraphy written by
choreographed movements in VR. From the identification of the inadequacies of current software and hardware, we
propose our own setup designed to specifically retain the artistry of shufa, and at the same time
transform the tools and media digitally. This is demonstrated through the writing of sculptural,
multidimensional, expressive, and choreographed calligraphy “in the air” with fingers using
hand-tracking technology as data input. And, due to the multidimensionality (3D space + time) of this new form
of writing, we discuss innovative ways of approaching calligraphy structure, writing order, and spatial
alignment within our tests and artworks.
The third part of the paper discusses the experiments as three points of reflection. We consider VR
Calligraphy within the notion of time against the linearity of focus when writing calligraphy. On the
evaluation of choreographed body movement within our writing process, we reference performance arts such as
dance and kung fu to determine areas for improvement. Lastly, we reflect on the experiments as a new form of
immersive calligraphy experience and speculate how this could enhance appreciation of this art.
Searching within Tradition: Energies, Tools, Body Movements
An Act of Creative Expression
Figure 1 shows the writing of Shu (书, derived from the word shufa) ¹ , and is
written by Ching-Bor Law, a Hong Kong–based master calligrapher. It visually represents the energetic flow
of ink whilst keeping an upright and balanced structural harmony. This is a creative expression of the
writer’s interpretation of the character.
Xiongbo Shi explores calligraphy as a summative expression of spiritual, intellectual, cultural, artistic, and
emotional energies ². This complex can be referenced as geist (spirit) in the West and represented
by yi (意, intent) in Chinese. Further, Shi separates intent into two categories: Youyi
(有 意, being intentional, conscious) and wuyi (无意, not being
intentional, unconscious).
By encompassing Geist/Yi ( 意) intent during writing, brush strokes evoke ideas of
representation and personal meaning. According to Shi [ibid], a piece of traditional calligraphy can be
considered artistically excellent when the summation of subjective meanings is perceived as beautiful. This
relation between calligraphy and traditional beauty has also been raised by Chang through comparison to
surrealism ³, and in the study of contemporary calligraphy by Barrass, who distinguishes the Modernist and
Avant-Garde Chinese calligraphy movements based on their critical stances towards traditional beauty, conceptual
abstraction, and experimentation of media ⁴.
Philosopher Yuk Hui, in his seminal work The Question Concerning Technology in China, explores the
relationship between technology and Chinese philosophy. Central to his argument, in Confucianist and Daoist
Chinese culture, dao (道, way/order) “[is] the supreme order of being”
⁵, and Qi (器, tool/vessel), when applied specifically to calligraphy, provides the
technique or technology to develop form. The notion of harmony is pursued in this relation within
dao/qi. This link between art and dao is further explored in The Spirit of Chinese Art by
Fuguan Xu: “Conceptually we can only use what they call dao to define the spirit of art, but we cannot use
the spirit of art to define dao. Because dao also has a speculative dimension, logically speaking, it is broader
than art” ⁶.
Within our experiments, we aim to continue the pursuit of The Spirit of Chinese Art. The intent is to
evolve the tools and media of calligraphy, melding its material-based phenomenology together with its
philosophical duality to retain its artistry, and at the same time, transfer its essential qualities to an
immaterial and dynamically time-based version in VR.
Evolution: Tools and Range of Body Movements
Calligraphy is a performance, a dynamic and evolving art form with constantly changing tools, and thus
requiring different body movement forces to create. Within the history of Chinese calligraphy, each evolution of
the medium required different tools and skill sets of its time. Chinese calligraphy started as pictographic
symbols known as oracle bone script (c. 1200 BCE) ⁷. Characters were chiseled out of turtle shells using a
carving knife. Here, the writer’s arm exerted a downward striking force, perhaps with the aid of a hammer.
This technique evolved through the use of bamboo, bronze, wood and silk, as media into its current form of ink,
brush, and paper since c. 100 BCE.
With ink and brush, the arm exerts only a slight downward pushing force, with longer strokes written through
horizontal arm movements. This enabled an increased range of body movement with greater precision. As a result,
progressively expressive styles of calligraphy were written, leading to the development of the standard five
scripts (see Figure 2 below):
1. Seal script (Zhuan shu) - in use from c. 1200 BCE. 2. Clerical script (Li shu) - from
c. 200 BCE. 3. Regular script (Kai shu) - from c. 200-400 CE. 4. Cursive script (Xing
shu) - from the 4th century CE. 5. Drafting script (Cao shu) - from the 7th century CE.
Each script of calligraphy, with its distinctive stylistic differences, requires the hand and wrist to
coordinate and “perform” in different ways: from the more controlled, symmetrical action of the Seal
script to a spontaneous and fluid action of the Drafting script. This performance of the hand and arm is
partnered by the tool and medium of brush and ink.
Focusing on the tools, the traditional Four Treasures of the Study (文房四宝)
consists of the brush, ink, paper, and seal. The standard calligraphy brush is usually made of a bamboo shaft
which holds the brush tip, traditionally comprised of animal hair. Species variation of animal hair results in
different rates of ink absorption, which is compounded by the length and shape of the brush tip. The length of
the writing stroke is limited by the amount of ink that the brush hair can hold combined with viscosity (i.e.
water content) of the ink liquid, and the rate of absorption of the paper. The width of the writing stroke is
determined by the length of the brush hair.
One of the skills of traditional calligraphy is the careful control of ink dipping with the brush, and the
precise addition of water to the ink as it dries within different room temperatures and humidities. It is a
deeply time-sensitive process, often executed in one breath, linking the body to the art of writing and the
materiality of the tools.
The third materiality in calligraphy is xuan paper, whose composition mainly consists of tree bark and rice
straw. Different compositions of raw materials and production techniques can lead to various rates of absorbency
and smoothness (friction) of the paper surface. These physical properties of paper directly interact with the
ink and brush during writing. The bounds of writing, the dimensions of paper produced, reflect the range of
motion available to the arm. This is represented in standard paper sizes of 4 feet (70x140cm) or 6 feet
(100x180cm).
The skill in writing calligraphy is, in essence, the fine choreography of the arm, hand, and finger muscles
resulting in a transferal of the ink-absorbed brush tip onto the absorbent paper surface. In Chinese characters,
there are long, short, vertical, horizontal, dot, curving, hooked, angular strokes, each requiring a different
set of precise combinations of finger, wrist, and arm movements, as illustrated above by Chiang ³ .
Understanding of these fine muscle movements and their resulting dynamic ink effects is critical to the process
of transferring movements from the arm to the whole body.
These are the physical tools and media of traditional calligraphy with their cultural lineage. The aim of our
experimentation is to conceptualize a new toolset and media in VR that bears the qualities of ink, brush, and
paper. This is an evolution that retains tradition yet extends new possibilities within multidimensional space
and synergy of new virtual materiality.
Experiments: VR Calligraphy
Setup
Within VR software, painting and sculpting apps have been generally available for a few years now, most notably
Tilt Brush for painting and Oculus Medium for sculpting. Whilst these are excellent applications for their
intended purposes, their functionality does not align with the characteristics of calligraphy. Tilt Brush was
developed as a platform to emulate a Western mode of painting, akin to oil painting, where brushstrokes are
applied layer upon layer to create patches of color. Chinese calligraphy, however, focuses on the expressivity
of individual strokes: the freely changing widths, precise curvatures, the speeds of brush strokes, together
with ink effects such as wetness, blackness, and ink bleeding. This calligraphy-centric level of control and
expression is presently not achievable in any VR app.
Likewise in VR hardware, current platforms have settled for a generally standardized blueprint of a head
mounted display (HMD) with hand-held controllers, which include a trigger button for the index finger, a
joystick, and four buttons for the thumb. This design satisfies most apps: the trigger button (index finger) for
grabbing and triggering ingame actions, the thumb buttons to offer other functions and menus. However, this
standardized UI (user interface) only provides a trigger button offering 1cm of travel with very little force
resistance (Figure 5). Thus, the VR controller is inadequate to support the finesse necessary in our transfer of
the skillful controls of brush and ink and does not suit the expressive qualities of calligraphy writing.
Due to the lack of existing HCI interfaces for this kind of expression, we developed a new system to write VR
calligraphy. In our experiments, the hardware is based on the Meta Quest 2 VR platform, and the application is
created within the Unreal Engine platform. Our system uses new hand tracking functionalities and only requires
the use of cameras on board the VR HMD to capture the hand and fingers for gestural input, as shown in Figure 6
below. By using bare hands, there is no longer any physical tool to be held. This approach has merit, as it
first enables users to forgo the associated learning curves for interacting with physical tools, and secondly
provides a primal, intuitive sense of creation. This links users back to the art stylings of prehistoric times
when humans wrote with their fingers on cave walls, as reflected by David Hockney and Martin Gayford in their
tracing of media and techniques from art history ⁷.
In order to transfer the nuances of a calligraphy brush into VR, we developed a program to measure the distance
between the thumb and index finger of the right hand to control the thickness of the written stroke within VR.
The thumb touching the index finger (a pinch gesture) results in zero thickness: no ink generated, no stroke
drawn. These two fingers pulled apart widely results in a thick stroke.
In regard to our VR version of calligraphy ink, we designed a particle generation system to simulate a
three-dimensional fluid ink/smoke effect. A separate menu (Figure 7) was scripted for the left hand using
virtual adjustable menu sliders to adjust, in detail, important visual and calligraphic parameters such as brush
roundness, ink wet/dry effect, ink bleed, gravitational pull, opacity, color, and time decay (the duration for
the ink to vanish). With VR ink, users can, in practice, write strokes of infinite length. This opens up new
creative possibilities, not just with length of stroke but also with the fluidity of movement, as the writer no
longer has to dip the brush to replenish ink, which interrupts the flow of the writing movement.
Concerning calligraphy paper, there is no longer need for its flat dimensionality (2D). Instead, VR allows for
writing in 3D space, which can be as large as the computational power affords. The physicality of paper
absorbency is now simulated within our menu system. The friction of writing on paper, whilst impossible to
simulate with our use of hand tracking, is now extended by the new possible interactions of our fingers, arms,
and the entire body in 3D space, captured by digital hardware and visualized as immaterial virtual calligraphy
within our computer program.
Shi and Curved Shi: The Third Dimension
First, we tested one of the fundamentals of writing in the alignment and thus continuity of the characters. In
2D, this is not a problem as the writing is perceived only from one angle. Therefore, our first two tests
demonstrate the difficulty of text alignment and three-dimensional readability. The criteria are to have a
change of writing plane and an added depth dimension whilst retaining legibility from a “front”
angle. For clarity, the three dimensions refer to cartesian dimensionality using the X, Y, and Z axes, with
conventional calligraphy on paper being 2D with writing on the X-Y axes.
For the retention of character legibility in 3D, character strokes that conventionally join should also
intersect at the Z axis, but this act alone is difficult in practice. Writing in the air, as opposed to on a
table in conventional calligraphy, has no resistance. As humans are accustomed to writing on a flat surface, the
act of joining lines within the Z axis is a novel phenomenon.
View-point specific, three-dimensional Chinese calligraphy does not currently exist. Some Chinese
character-based sculptures exist, but these are mostly reliant on the meaning of the character and do not focus
on the actual readability of the dynamism involved in writing shufa. Because of this lack of precedent,
we had to experiment and conceptualize our own rules of transposing two-dimensional calligraphy into three
dimensions.
The above figures of Test 1 - Shi illustrate this point with one of the simplest Chinese characters. Whilst
from the front view (Figure 8), both A and B versions of the same character read shi (十, ten), the side
view (Figure 9) shows that version A is joined at the Z axis, but in version B, the two strokes are separated,
thus not legible as a single character from this angle of view. This test only concerns writing straight lines
within 3D. Our next test discusses the difficulty of writing calligraphy “sculpturally”.
In Test 2 - Curved Shi, the aim is to write the same character sculpturally, as if imprinted onto a
sphere. The first stroke, the horizontal stroke, is written in a way that is curved towards the writer. However,
the next vertical stroke has to start on the same Z axis plane as the start of the horizontal stroke, make the
same curvature to meet the midpoint of the horizontal stroke, and end back at the same Z axis plane as the end
of the horizontal stroke.
Figures 10–11 demonstrate the two strokes correctly joined together in version C. In version D, the two
strokes fail to join together and cannot be legible as the intended character. Writing in 3D requires a constant
shift of view, thus movements of the head and body are required to align multiple strokes. And, the order of
complexity is compounded for characters with an increasing number of strokes. Our next step is to write a single
character as calligraphy within the discussed contexts of tools, spirit, and dao, with our determined ink
effects, and testing the full range of body motion.
Hu - Choreographed Performance Experiencing Whole Body Writing
Figures 12–14 show our artwork for Hu (虎), the character for tiger. A video is available
to demonstrate the three-dimensional and sculptural quality of the writing at https://vimeo.com/773627966.
Similar to Shu, by Law (Figure 1), we wanted to express the dynamic flow of writing with ink effects,
an upright structural balance, and incorporate the playfulness of the tiger by the long, upturned tail of the
last stroke (blue stroke of Figure 13).
Hu is written at around 1.5m in terms of real-world height to test the applicability of a full range
of body motion. It involves a carefully choreographed sequence of foot, body, and hand motion for the character
to be written as intended. To illustrate, Hu is written with three strokes. The first stroke (red in
Figure 13) starts at the top left position, then the writer’s hand must move towards the right with a
slight push forward to reach the corner position. To achieve the long tail of the stroke, the writer takes
a step back with the left foot whilst bending the waist for the arm to reach the end of the stroke. To begin the
second stroke (green in Figure 13), the left foot returns to a standing position, and as the hand writes
downwards, the writer starts taking a long stride backwards with the right leg in anticipation of the next
stroke. The third and final stroke (blue in Figure 13) is a long downwards and backwards sweep, which requires
the right foot to finish a step back, and the waist and head turning with the arm to complete the stroke.
Our writing of Hu required 88 attempts across three days before we achieved a satisfactory version.
This was because we wanted to retain the same focus of writing as in traditional calligraphy. Our writing
experience was akin to conventional calligraphy, where each error results in a complete restart of writing in
accordance with our aims to preserve the notions of dynamic ink and the linearity of time and focus. Hence, we
retain the dao of calligraphy whilst executing the transferal of brush, ink, and paper as digital versions,
writing an expressive interpretation of the Hu character in VR.
Counterpoint: Curse
Having previously pursued a harmonious writing of Hu, we wanted to test how the same toolset could be
used in a contrasting way. Thus, we wrote Curse (人骂我,
我骂人也, “They curse me, hence I curse them”) shown in Figures 15–
17), inspired by the famous phrase by calligrapher Baishi Qi (齐白石). This was written during
a period of lockdown in China to express the frustration of the times.
Here we replaced traditional black ink with neon pink ink against a night sky setting. The carefully balanced
proportions of the previous Hu were disregarded; instead, we wanted to convey expressions of rawness
and chaos. The characters were written circularly around the writer, who pivoted 360 degrees around the same
location (Figure 17). This work of multiple characters has no “front angle”, and therefore can only
be read by moving around it or moving into the center of the characters and turning around in full circle to
look at the text. This experimental subversion of some of the traditional values of calligraphy serves as a
stepping stone for branching out into more unconventional calligraphy visualizations.
Simultaneously, the constantly changing widths of the writing strokes are controlled via finger movement
(Figure 16): the distance between the thumb-tip and index-finger-tip of the right hand. The ink particles are
generated at a fixed rate: a faster hand movement results in less particles, a more transparent ink effect, and
a slower hand movement results in a denser ink effect. Therefore, the speed of writing and the time for ink
drying is visually rendered as an evolved form of VR calligraphy. This preserves the essence of time-based
calligraphy and expresses the dynamism of ink with- out the necessity of physical materiality of ink, brush, and
paper, within the non-material medium of VR.
Reflections
Non-Undo, Linearity of Time
A prevalent benefit of common digital creative apps is the functionality of undo, which allows users
to easily fix mistakes. The notion of undo itself is powerful, in essence giving the user control of time. The
ability to erase materiality (go back in time) or re-create it (skip forward time) instantaneously introduces
disruption not just in time but also interrupts the linearity of the creative process and the fluidity of focus
when writing.
In reflection of writing Chinese calligraphy, undo is controversial. Traditionally, mistakes are non-erasable,
so calligraphers write with such focus to prioritize expression as well as to avoid mistakes; when
mistakes are made, the piece of paper is usually discarded. Our experiments go against the practice of common
creative productivity apps by completely forgoing the implementation of undo. Mistakes are inerasable, resulting
in us requiring 88 attempts to write Hu. Through this intentional absence of the computational undo,
the writer requires a similar focus using VR Calligraphy when compared to traditional calligraphy. Additionally,
the writer needs to pay attention to the new complexities of writing in multiple dimensions (3D space + time)
through coordinated movements of every part of the body.
With the physical tools of calligraphy now replaced by a “set of invisible algorithms” ⁷,
there is no longer a need to dip the brush to replenish ink. Our experiments introduce a continuity of time
compared to traditional calligraphy, speculatively deepening the focus during calligraphy writing.
Dance/Choreography and Calligraphy
In our writing of Hu, we found that our considerations of precise body movements were similar to that
of performance arts.
We wish to point out the exemplary choreographies developed using calligraphy as inspiration by Shen Wei Dance
Arts in Connect Transfer (2004). As Gerdes points out, Shen Wei explores the “body as
calligrapher, [...and] displays both the body in constant motion and the painted brush strokes as indications of
this motion, thus visibly portraying product and process simultaneously” ⁸. The dance routines
incorporate ink applied to the hands and feet, with paper laid out across the floor or wall. Body movements are
reflected visually as ink splatters on the paper. It would be of interest to us in the future to not just
demonstrate written calligraphy but also visualize such trails of body movement to further augment appreciation.
Leach notes the recent drive for choreographers who seek to utilize technology to reveal and disseminate
previously non-visualizable structures within choreographies as choreographed objects
¹¹ . Notably, Forsythe and deLahunta actively seek to “make the value of dance
available ‘without the body’” (ibid, p.470). Synchronous Objects for One Flat Thing,
Reproduced by Forsythe in 2009 is an excellent example of how the production for One Flat Thing,
Reproduced can be systematically broken down into visually easily understandable diagrams, such as
movement density, 3D alignment forms, and difference forms.
Similarly, Kung Fu Visualization by Shaw and Kenderdine in 2016 set a precedent for how intangible
motion- based Chinese arts can be digitally investigated through motion capture and analysis of movement forces
to reveal new phenomenological insights into the craft, for purposes of both analyzing and understanding past
works in multiple dimensions.
Learning from Foresythe’s Synchronous Objects and Shaw’s Kung Fu Visualization,
we can consider the breaking down and visualization of our artworks with diagrams of footing, stance, head
position, hand movements, and movement speeds, to better illustrate the complexity of coordination required in
our experiments. In addition, the dimensionality of time, the live real-time performance of calligraphy, can be
visualized and animated as part of the appreciation process. Our VR Calligraphy can be digitally
recorded as particle data and as body motion data and replayed in real-time as the viewer enters the virtual
space. The visualization of this symbiosis of movement and writing, together with the immersive intimacy of VR,
would strive to reveal a deeper understanding between the choreography, body forces, and the written
calligraphic forms together as a real-time performance.
New Experience in Appreciation
In Curse, we created a new way for calligraphy appreciation and experience. Traditionally, calligraphy
is appreciated by having the work statically hung on a wall. Our multidimensional focus in Curse,
however, actively encourages the viewer to explore calligraphy from various angles to appreciate the sculptural
form of it. We view this enablement of proximity to be a powerful experience. It negates the notion of distance
between artwork and the viewer, as written by Grau ¹², and allows for an increasingly intimate and
immersive appreciation of the finer details of the calligraphy: the expression of strokes, the dynamism of ink,
and the fluidity of movement behind the calligraphy. Our investigation has established a change of
dimensionality from 2D to 3D immersion, where the visual stimuli is greater and more diverse, resulting in what
Dewey refers to as more “refined and intensified” forms of experiences in calligraphy appreciation
and interaction ¹³.
Conclusions
Our experiments successfully demonstrate the transposition of whole-body movements into a multidimensional and
digital form of calligraphy. We have made a step forward in developing our own program and interface to write
this new format of calligraphy. Our findings within the notion of time, undo, and linearity of calligraphy
writing have been positive thus far. We would like to have more in-depth dialogue with calligraphy experts to
evaluate the outcomes. For improvements, we are so far satisfied with the visual and research outcome of our
VR Calligraphy as a manifestation of carefully choreographed sets of movements. We feel that there is
opportunity for the traces of body movements to be additionally visualized, as our comparisons with dance and
kung fu has shown, to better illustrate the intricate movements involved with easily understandable visual
explanations.
It is also important to note that although the body movements and medium of our experiments are different, our
aim for calligraphy within The Spirit of Calligraphy remains the same. We have maintained a pursuit of harmony
in the discourse of dao/qi through our carefully considered usage of a new toolset and careful transferal of the
intricacies of ink and brush effects.
Additionally, we have demonstrated that this new mode of calligraphy appreciation enables an immersive
experience by encouraging spatial exploration of the works, thus expounding the perception of calligraphic art.
Whilst the medium of our calligraphy has evolved, we have also expanded the level of geist/yi in the creative
expressivity within our experiments due to our enhancement of the focus of writing without needing to replenish
ink, and with our decision to forego undo. Lastly, we wish to confirm our retainment of the practice of harmony
within the dao/qi of calligraphy as we continue to seek an evolution in The Spirit of Calligraphy in
our research.
Looking Forward:
Immersive Collective Calligraphic Speculations
Our artworks of Hu (tiger) and Curse are only pertinent to the calligraphy of a single
character written at a human height (1.5m) scale. It is technically feasible to write using similar hand and
finger input with a scaling functionality to magnify the body movements to create larger-scaled works, which is
the next step of this research. This would enable the creation of an increasingly immersive experience, one that
curates the arrangement of different scaled calligraphy to create an assortment of spaces: big vs small, open vs
closed, wide vs narrow, bright vs dim, straight vs curved.
If we also speculate at the possibility of co-creating such VR Calligraphy and associated qualities of
time, dimensionality, scale, expressiveness, and immersive experience together with
multi-player–calligrapher input, this could amalgamate into a collective calligraphic performance. And,
referencing the choreography visualizations mentioned, also illustrate the performers or avatars writing with
their body movement traces as a multi-layered spectacle.
Through our research, we hope to demonstrate a successful case for symbiosis between the art and performance of
calligraphy with the technological potentiality of VR technology. We shall endeavor to continue in our
experimentation to present Chinese calligraphy as a contemporary continuation and extension of its cultural
heritage.
Acknowledgements
The work described in this paper was partially supported by a grant from the Research Grants Council of the
Hong Kong Special Administrative Region, China. This research is part of the ACIM Research Fellowship.
References
1 Law Ching-Bor, The Calligraphy of Law Ching-Bor, Hong Kong, Yuen Long District Arts
Council, 2016, p.91.
2 Shi Xiongbo, "Chinese Calligraphy as Force-FormJournal of Aesthetic Education
53, no. 3, 2019, 54-70.
3 Chang Yee, Chinese Calligraphy, Harvard University Press, 1938, p.101.
4 G. Barrass, The Art of Calligraphy in Modern China, Berkeley, University of
California Press, 2002.
5 Hui Yuk, The Question Concerning Technology in China, Urbanomic, 2016.
6 Xu Fuguan, The Spirit of Chinese Art, Collected Work of Xu Fuguan, vol. 4. Hubei,
Hubei People’s Publishing House, 2009, p.44.
7 X. Qiu, G. L. Mattos, J. Norman, Chinese Writing. Early China. Society for the Study of
Early China and the Institute of East Asian Studies, University of California, Berkeley, 2000.
8 David Hockney, Martin Gayford, A History of Pictures: From the Cave to the Computer Screen,
Thames and Hudson, 2016, 354.
9 Xu Songhuan, C. M. Lau Francis, Yunhe Pan, A Computational Approach to Digital Chinese
Painting and Calligraphy, Springer Science & Business Media, 2009, 335.
10 Ellen V. P. Gerdes, "Shen Wei Dance Arts: Chinese Philosophy in Body Calligraphy", Dance
Chronicle, 2010.
11 James Leach, "Choreographic Objects." Journal of Cultural Economy 7, no. 4, 2014,
458-75.
12 Oliver Grau, Virtual Art - from Illusion to Immersion, The MIT Press, 2003.
13 John Dewey, Art as Experience, Penguin, 1934.
LAVIN: An AI-Navigated Art Experience in Virtual Reality
WeidiZhang, JieliangLuo
Arizona State University, Autodesk Research Arizona, USA; California, USA
zhangweidilydia@gmail.com, rodgerljl@msn.com
Abstract
This paper outlines the conceptual background, design methodology, and future directions of LAVIN.
This virtual reality (VR) artwork provides an immersive experience to visually explore one understanding of a
neural network in which the real-world maps to 50 daily objects. In the art installation, the neural network
constantly analyzes the surrounding environments via a camera and outputs real-time semantic interpretations,
which navigate the audience in a virtual world consisting of all the fluid abstract structures of daily objects
that the neural network can recognize. We create these fluid virtual structures using data visualization,
photogrammetry, and 3D modeling. By merging Artificial Intelligent (AI) system design with VR world-building,
LAVIN offers an immersive art experience for symbiotic imaginations that questions the values and
beliefs in the modern AI age.
Keywords
AI System Design, Virtual Reality, Immersive Media Design, Generative Art, Interactive Art Experience.
In the last decade, object detection has made tremendous progress. In some narrow domains, the algorithm has
almost reached human-level accuracy ¹ . Each neural network's output is a unique projection of its
understanding of the real world, regardless of whether it has been trained to recognize thousands of objects or
only produces binary outputs ² . Regardless of the complexity of the projection, it shapes the “world
value” of the neural network ² . Numerous deep neural networks are implemented for various
intelligent applications. Despite this, even the most complex neural network model is not capable of
representing the real world entirely. A neural network used for facial recognition, for example, can only
interpret the input as a collection of faces ² . Thus, our design question arises: what is the ground
truth in the modern AI age?
As a conceptual response to the question regarding the worldview and value of neural networks, LAVIN
provides one understanding of a neural network by combining VR worldbuilding and AI system design. We
design our AI system to observe surrounding environments in the gallery space via a camera and output one word
to describe the live-streaming based on the neural network’s interpretation. The output word is the
navigation information that travels the participants in the VR in real time. The AI system is trained to only
understand 50 daily objects which leads to its misinterpretation of the live-streaming information. The VR world
is made up of these 50 objects our AI system can recognize. LAVIN aims to provide an immersive art
experience that is navigated by an AI system through visualizing the imagined world of a neural network
artistically.
Background
The real world is a sophisticated collage with diverse groups of creatures, overloaded data, a myriad of
choices and chances, fleeting messages, complex shifting meanings, and distinct stories, cultures, and political
systems. If we perceive the images of our world as a visual representation that is programmable, networked, and
operative, what is the method to associate different visual information? How do we use an algorithmic approach
to create a new kind of chance and choice operations in the art-making process? Who will be the author:
machines, humans, or human-machine hybrid? What are the data-driven material, forms, and aesthetics? How will
the audiences interact with this assemblage, and in which way will their decisions change the experience?
Seymour Chatman states that: In this age of mechanical and electronic production and reproduction, it would be
naive to reject the notion of nonhuman narrative agency, for instance, a story constructed through algorithmic
processes or as a result of multiple voices constructing together in real-time ³. With the goal of
integrating a non-human narrator for immersive storytelling, we designed an AI system that navigates this VR
journey based on its real-time understanding of the surrounding environment. The customized machine reinterprets
the real world to build topographical associations between objects in a VR journey.
Related Artworks Worldview of A Neural Network
Interactive works employing machine learning algorithms often offer a user-centric and adaptive experience.
Using autonomous systems, responsive applications can assemble artifacts, discoveries, traces, and experiences
based on the prior knowledge of the machines and their interpretations of audiences' inputs.
New media artist Memo Akten’s series of interactive works—Learning to see—“use
state-of-the-art machine learning algorithms to reflect on ourselves and how we make sense of the world”
⁴. In the art installation, Memo uses a live camera to capture everyday objects and uses a number of
neural networks to analyze the live-streaming feed. The audience is invited to move the objects under the camera
and see the visualization evolve based on the changing composition of the moving objects. The visualization
changes every 30 seconds to different networks trained on different datasets: water, air, earth, and cosmos.
Memo Akten considers the way of seeing in the conscious mind as a reconstruction based on expectations and
beliefs instead of a mirror image of the outside world ⁴.
Similarly to Memo Akten's Learning to See, LAVIN emphasizes the importance of machines' prior
knowledge and their worldviews in presenting how machines interpret the world using an artistic approach.
LAVIN differs from the relevant artworks in that it does not directly display machine interpretation as
visual output, but rather uses it as navigational information for engaging the author’s artistic
imagination of a neural network’s mind. LAVIN combines AI’s decision with artistic
originality to invent a virtual world of symbiotic imagination.
Navigation in VR Journey
Traditionally, immersants (the participants who wear the head-mounted device ) can use VR controllers to
navigate a virtual space, which often requires a learning curve. At the same time, there are many VR artworks
that investigate conceptual and novel navigation methods.
For example, the pioneer VR artwork, Osmose, created by Char Davis in 1995, uses bio-data of breathing
to transport users to different elevation places within a virtual environment. Osmose presents a
multi-sensory immersive experience that engages visitors in a virtual world made of simulation of nature scenery
and text. Immersants can navigate themselves in the virtual space through body breathing—a chest-hugging
rubber vest that contains sensory devices sensitive to the body’s breath—a natural interface
⁵. This interface works similarly to diving—when you inhale, you rise. Osmose juxtaposes
the insecurity of body presence and an emotional sense of tranquility.
Osmose precurses to the booming industry of VR as a platform of narrativity and entertainment in the
21st century. It also challenges the way we view pictorial artworks. The infinite poetic space is unfolding like
a canvas waiting for artists to infuse new ideas. A 3-dimensional space adds one dimension to the 2-dimensional
pictorial work for the multi-media assemblage. The exploration of the relation between 2-d elements changed into
the examination of neighboring topological notions within a dynamic spatial volume.
Similarly to Osmose seeking an innovative way of VR navigation for artistic purposes, LAVIN
creates an intelligent system to contribute to the dynamic flow of the spatial assemblage. Real-world
images are connected with virtual space for a unique way of seeing and controlling.
Design Methodology of LAVIN System Design
LAVIN provides a conceptual and artistic response to the questions of the Ground Truth in the modern
age of AI by designing the navigation system based on a live neural network’s real-time interpretation of
live streaming data of the surroundings.
For our AI system design, we specifically use ResNet50, “a pre-trained deep neural network that can be
altered for different use cases” ⁶. ResNet 50 networks have been trained on the large datasets from
Imagenet and “modify the last layers to quickly produce models to tackle new problems” ⁷. We
build a custom dataset of fifty daily objects and trained a neural network to only recognize the selected fifty
daily objects. When an image contains objects outside LAVIN’s vocabulary, its limited training
dataset leads to a possible false interpretation of the given image.
In the art installation (see Figure.2), LAVIN constantly chooses one object to describe live streaming
data captured by a camera (set in the center of the exhibiting space). The real-time one-word description
travels an immersant to the related objects in the virtual space. Our intelligent system (LAVIN)
triggers absurd but interesting results, which transports the VR immersant to experience how a limited machine
perceives reality.
VR Worldbuilding
The immersive virtual space is a spatial assemblage and an artistic imagination of the neural network.
LAVIN incorporates both real-world data and artistic manipulation to present a symbiotic imagination.
This multi-layered, bizarre, and unknown VR space is mainly designed by addressing the following three aspects:
3d reconstruction, texture development, and spatial design.
The virtual reality environment consists of fifty objects that the trained neural network (LAVIN) can
recognize. These virtual sculptural objects are constructed by using the photogrammetry
technique—photographs are taken volumetrically of these 50 objects, which are then analyzed, calibrated,
and calculated to generate the XYZ position data of the point cloud for the 3d reconstruction (see Figure
4).
The aesthetic outcomes of the reconstructions are realistic with a great many unnecessary details for artistic
visual representation. We simplify, filter, and rearrange the dataset to distort the constructions and sculpt
the shape, at the same time still maintaining the characteristics to make sure the forms of these objects are
recognizable (see Figure 5).
We keep all fifty 3d constructed forms as sculptures with plaster material in VR. Instead of directly attaching
photographs of fifty objects as textures to the 3d forms, we process images as interactive patterns floating in
the virtual space, which moves in distinct directions and speeds based on audiences’ positions. The
pictorial crossovers generate unexpected and poetic meanings between unrelated objects (see Figure 6). In the
virtual space, immersants are revolved around the moving textures, witnessing the layers of shapes weave the
artistic imagination and abstraction of the neural network.
The assembled 2d pictorial artworks often navigate viewers’ gazes in the composition by drawing their
attention to the correlations between unrelated elements in various materials and shapes. Correspondingly, the
navigation of an assembled VR space examines neighboring topographical notions to convey meanings.
The spatial configuration of LAVIN is inspired by Paul Klee’s Fish Magic. “He creates a
magical realm where the aquatic, the celestial, and the earthly intermingle” ⁸. The mysterious and
sophisticated space on the canvas where fish and flora float among human beings and clock towers. The objects
are in indeterminate scale, scope, and direction.
Impacted by Klee’s creative solution for imaginative space, LAVIN completely abandons the
real-world objects’ proportion, scale, and properties to emphasize the unpredictability of a wholly
imagined world (see Figure 7). The fifty sculptural forms of objects are static like the artifacts displayed in
the museums, while the interactively moving patterns densely and dynamically fill the void between them.
Metaphors are born out of the associations between texture layers and sculptural forms. By gazing at the
textures of the patterns closely, participants can root out the real-world evidence of the objects.
Sound Design and Art Installation
A crucial part of the design of this immersive experience is the use of sound. Inspired by musique
concrète, using field recordings, instrumental sounds, and human voices as sound materials, We organize
and process them to create sound objects. We then attach them to various static or moving virtual objects
(models and patterns) to produce a phantasmagorical soundscape.
Specifically, we record Google Translate’s generic human voice reading the words of the 50 objects and
attach them to the corresponding objects in the virtual space. Field recordings of cityscape were made as the
raw sound materials which are sorted and organized by pruning and splicing, then transformed by using software
(Ableton Live and Adobe Audition) to filter and reverberate. The processed sound objects are imported into the
virtual world and attached to the different floating textures. When multiple channels of sound cross over and
assemble in space, a dynamic spatial montage is created.
This work was presented in the Siggraph Art Gallery (see Figure 8) and SwissNex Gallery in 2019. The exhibition
design includes a screen or two projections, a webcam, a VR headset, a speaker, and a PC. A camera is set in the
center of the space to observe surroundings and send live streaming to the system. The VR headset is placed on
top of a pedestal in the space. The real-time VR experience is also displayed on a LED screen or projected onto
two adjacent walls. The real-time sound in VR is played through a speaker in the exhibition so both the
participants and the immersant can hear.
One participant at a time has the chance to be immersed in VR (see Figure.9), and other participants will
possibly be captured by the cameras, which determines the virtual journey of the immersant. Immersants are not
given any instructions or visual cues to help them decide what to do, the AI system enables intuitive user
interactions which avoid the learning curve caused by using VR controllers.
During the exhibition, participants usually immerse themselves in the VR world for around 10 to 15 minutes.
Immersants are engaged in the VR experience using a headset and being automatically transported to different
locations determined by AI’s observation. The immersants wander within the space, investigate the meaning
of the abstract sculptures of objects, moving patterns, and abstract sounds they encounter, and free their
imaginations of this artistic interpretation of an AI’s mind.
Discussion: Symbiosis, Metaphors, and Cognitive Assemblage
LAVIN emphasizes the symbiosis between humans and machines through contributions from artists,
machines, and audiences. This cognitive assemblage formalizes a multi-layered decision-making process where the
cultural layer (generated by humans) and the computer layer (generated by the software) mutually influence each
other, resulting in a blend of human and machine meanings.
This fluid network begins with critical and cultural questions that determine the information collected for an
AI system to learn, which forms the first layer: a cultural layer. The intelligent system makes decisions based
on its prior knowledge and observation of images, forming the second layer: a computer layer. The artist's
aesthetic decisions and audiences' real-time feedback forms the following cultural layers. Each layer is
interconnected and adaptive. For example, participants' real-time feedback is constantly captured by the camera
(imaging device), which affects the second layer of the decisions made by the AI system. The interconnection
between different layers contributes to the entanglement of culture and computer layers, forming a cognitive
assemblage in system design.
N. Katherine Hayles stated in Cognitive Assemblages: “Technical Agency and Human Interactions: In a
cognitive assemblage, humans and technical systems are interconnected with each other. The interactions involved
human cognition, which includes consciousness, the unconscious, the cognitive nonconscious, and the perceptual
system. In addition, the decisions made by humans will, to some extent, determine how the technical system
operates.” ⁹
In LAVIN, the output of the technical system will also, many times, affect humans’ decisions,
which leads to a cognitive assemblage in the art experience, the feedback loops steer human-machine
collaboration to a “more life-affirming practices as we move toward a future in which technical agency and
autonomy become increasingly intrinsic to human complex systems” ¹⁰. For instance, the
participants who are captured by the camera in the art installation can see the real-time projection of the
virtual world. When they see the machine’s recognition of them leads the immersant travel to a specific
object (for example, a cactus), the participants tend to interact with our system by placing different things in
front of the camera and see machine’s real-time interpretation changing the route in VR. Therefore, when
we design a technical system, we are partially designing ourselves ¹⁰.
LAVIN provides an art experience that is not only designed as a cognitive assemblage but also as a
spatial assemblage, where metaphors are used to build poetic meanings and connections are employed to build
metaphors. We emphasize a metaphorical approach to building connections between visual materials and forms while
embracing chance and choice operation. The metaphorical approach to building organizations of media elements
(text, images, sounds) consists of automatic combinations and manual compositions. LAVIN manually
assembles the spatial arrangements of virtual objects our AI system can recognize in 3D virtual space. The
contrast, directions, and scale of objects are intentionally exaggerated to illustrate the surreal and bizarre
assemblage that suggests the 'AI's brain' within virtual reality. LAVIN is not only about the
co-creation of experience but also about co-authoring the metaphors and connections, which emphasizes change and
instability, fluidity, and exchangeability.
Future Direction
When we exhibit this artwork, the most common question raised by participants is why we selected this list of
50 daily objects. Are there any special meanings associated with these objects? The curiosity raised by
participants motivates us to develop LAVIN further in the future. We plan to create a more precise and
curated collection of objects rather than selecting them randomly. As a result of the careful curation of
objects, our trained AI system will develop a worldview that will inspire the interesting and critical questions
of human-machine reality.
Conclusion
In this paper, we introduce the background, design methodology, and future direction of an AI-guided VR
experience LAVIN. In addition, this paper discusses the metaphorical approach to present cognitive
assemblage in system design and spatial assemblage in worldbuilding. LAVIN provides an art experience
to visually explore one understanding of a neural network in which the real world is mapping to fifty daily
objects. The AI system constantly observes and analyzes the surroundings via a camera and outputs one word as a
semantic interpretation. This word navigates an audience into a virtual world consisting of abstract fluid
structures of these fifty daily objects that LAVIN can understand. LAVIN tends to evoke
audiences’ awareness regarding values, vulnerability, and beliefs within the context of AI. It also
provides novel research to connect VR and AI through designing an intelligent navigation system for an immersive
art experience.
Acknowledgment
We would like to thank Prof. George Legrady and Dr.Mengyu Chen for their help and support, as well as the
Experimental Visualization Lab and Media Arts and Technology program (University of California, Santa
Barbara).
References
1 Zaidi Syed Sahil Abbas, Mohammad Samar Ansari, Asra Aslam, Nadia Kanwal, Mamoona Asghar, Brian
Lee, "A survey of modern deep learning based object detection models." Digital Signal Processing
2022, 103514.
2 Name deleted to maintain the integrity of the review process.
3 Seymour Chatman, “What Novels Can Do That Films Can’t” Critical
Inquiry 7, no. 1, 1980, 121-140. https://doi.org/10.1086/448091.
4 Akten Memo, Rebecca Fiebrink, Mick Grierson, “Learning to see: you are what you
see”, In ACM SIGGRAPH 2019 Art Gallery (SIGGRAPH '19), Association for Computing Machinery, New
York, NY, USA, Article 13, 2019, 1–6, DOI: https://doi.org/10.1145/3306211.3320143.
5 Oliver Grau,"Into the belly of the image: historical aspects of virtual reality",
Leonardo 32, no. 5, 1999, 365-371.
9 N. Katherine Hayles, “Cognitive Assemblages: Technical Agency and Human
Interaction”, Critical Inquiry 43, no. 1, 2016, 32-55. https://doi.org/10.1086/688293.
10 N. Katherine Hayles, “The Cognitive Nonconscious: Enlarging the Mind of the Humanities",
Critical Inquiry 42, no. 4, 2016, 783-808. https://doi.org/10.1086/686950.
Volume of Voids: Artistic Visualizations of the Disequilibrium
WeidiZhang, ShaoyuSu
Arizona State University, University of Southern California Arizona, USA, California, USA
zhangweidilydia@gmail.com, shaoyusu@usc.edu
Abstract
Volume of Voids is a series of art projects that visualize and materialize the voids between different
beings under the regulation of social distancing during the Covid-19 pandemic. It consists of two major
artworks: Volume of Voids I, which creates data sculptures that visualize and materialize the voids,
and Vol II of Voids, which brings an interactive art experience that emphasizes the mobility of the
space-between humans. The purpose of this work is to create an artistic and conceptual response to the impact of
a historical pandemic on complex social networking systems by rethinking the space-between. This paper
introduces the conceptual background, design methodology, and technical implementation of this series of work,
including the discussions of speculative design framework on computational art, experimental data visualization,
volumetric capturing, and 3d fabrication.
Keywords
Data Visualization, Volumetric Capturing, 3D Fabrication, Generative Art, Interactive Art Experience.
Inspired by the social distancing regulations during the Covid-19 pandemic (see Figure 1), people are
disconnected physically by keeping a social distance to avoid spreading the Coronavirus. The disruption of the
fragile equilibrium in our system profoundly influences our way of living, communicating, and socializing.
Volume of Voids series starts with the conceptual What-If questions we posed during the historical
Covid-19 pandemic: When people keep their distance from objects and other people, what is the shape of these
invisible spaces? In what ways are these negative spaces evolving as a new entity, reshaping our society as a
new normal of intricate social networks? Will the form of the empty spaces of disconnection inspire new ways of
reconnecting?
Our list of design questions leads to a series of thought experiments that bring an alternative reality and
artistic experiences to life: 1. Volume of Voids I: A set of 3D printed data sculptures that
materializes the voids between objects and human bodies (see Figure 2) 2. Vol II of Voids: An
interactive art experience that visualizes the mobile voids between humans (see Figure 3).
The Volume of voids series provides artistic responses to disconnection, dualism, and the possibility
for a new structure and alliance that is more resilient and at ease in its alterity.
Conceptual Background: Dualism, MA (间), and Balance
The inspiration for this project comes from the concept of dualism, Yin and Yang, in ancient Chinese philosophy
as well as the idea of MA in Japanese art and culture.
In Chinese cosmology, the universe is born out of a chaos of material energy ¹, which is categorized into
the cycle of Yin and Yang. The beings and non-beings produce each other. These two opposite sides are possibly
interconnected and complement each other ². Yin and Yang represent passive and active respectively, which
exist in all forms of variation and difference in nature and world: female is Yin while the male is Yang, the
growing wheat is Yang while the reaping wheat is Yin, eggshell is Yang while the yolk is Yin ¹, the morning
fog is Yin while it is dissipated by the heat of the sun is Yang, the forest fire is Yang while the rainstorm
extinguishes it is Yin ³. The Taoists suggest that the opposition of Yin and Yang unifies the harmony in
nature. Yin and Yang not only can be found in nature, but they also exist in human bodies. Taoists believe
everything is part of the oneness of the universe, and there is no difference between the internal processes of
the body and the external forces of nature ³. The idea of Yin and Yang from Taoism inspires us to rethink
the relationship between the voids of social distancing with the internal processes in human bodies and minds.
In Japanese culture, the concept of MA describes the connection between positive and negative space. MA refers
to the artistic interpretation of emptiness. It draws people’s attention to the negative space, which
helps to perceive the positive space. The empty space is often full of possibilities—for example, the
artistic practices of calligraphy experiment with the concept of MA. Chinese and Japanese calligraphy not only
focuses on the depiction of characters but also creates careful relationships between the forms of
characters and the surrounding non-form. Calligraphy artists use the blankness to link forms and take the design
of the void spaces into artistic consideration.
Volume of Voids implements the concept of MA and Yin Yang by rethinking the meanings of the
space-between, the messages these silence spaces deliver, and how the voids influence the intricate networks of
society and the internal processes in human bodies.
Related Artworks
Fine artists have explored the positive and negative spaces in their poetic artworks, for example, American
photographer Ellen Garven’s Ambivalence and English artist Rachel Whiteread’s artworks
Ghost.
In Ellen Garvens’s Ambivalence [see Figure 4], she photographed a set of prosthetic devices.
These devices reflect their history of making and the patients’ experiences ⁴. Ellen thinks the
devices raise awareness of disability and its intimate relationship to all bodies ⁴. The grace of the
forms, the devices’ surfaces with textures of traces, and the way Ellen stage and photograph the devices
make the still life monumental, which evoke audiences’ feelings of loss and intimacy.
English artist Rachel Whiteread visualizes the invisible space by “mummifying the air in the room”
⁵. In her work Ghost, she cast the interior of a Victorian rowhouse in plaster and
reassemble the panels facing out towards the viewers. In Whiteread’s words, the ghost causes
“the viewers to become the wall” ⁶. She draws attention to the space and its inhabitants and
connects objects with their surrounding area with personal stories. Her mausoleum-like work materializes the
invisible spaces of everyday life powerfully and poetically.
Similar to these two artworks, Volume of Voids seeks to represent invisible spaces by giving a shape
to the air. By employing computational design methodologies of data visualization, 3D fabrication, and
volumetric capturing artistically, Volume of Voids distinguishes itself from fine art techniques
explored previously. Generative art and computational design emphasize the fluidity and interchangeability of
the voids, which reinterprets negative space in the context of the new media art practices.
Volume of Voids I
From Photogrammetry to 3D Fabrication
The computational design method for Volume of Voids I consist of three stages: collecting data,
experimental visualization, and fabricating 3D data sculptures.
We set up different news-inspired scenes for capturing. For instance, one of our scenes is based on an ABC news
report about people keeping social distance during line waiting at a cafe shop. This was staged during the
pandemic and volumetrically captured to collect spatial information on the voids. Overall, we staged twelve
different scenes and used photogrammetry techniques to capture each of them, generating point clouds of position
data for the following stages of visualization and fabrication.
Data Visualization
The photogrammetry point cloud position (XYZ data) generates multiple boundary edge curves on the XY planes
(see Figure 5). These curves create planar surfaces with corresponding position data on Z-axis.
The planes with different heights are connected, lofted, and controlled by using algorithms (such as bezier and
Perlin noise). These mathematical algorithms are added to Z-axis curves in order to add uncertainty to sculpture
constructions (see Figure.6) to represent inherently unstable, fluid, and unpredictable invisible spaces between
humans (see Figure.7).
Materialization of the visualization
To materialize the voids visualized, we then 3d printed these generative models as a series of physical
artifacts (12 in total). The 3D fabrication of each sculpture of the generative 3d model is typically completed
within 24 hours using PETG clear filament. We experimented with different materials and colors when fabricating
the sculpture, including ABS filaments in opaque white color. During our experimentation process, we realized
that rather than creating the monumental aesthetic of the voids, we tend to make the forms more transparent,
light, and dynamic, which represents the accumulation of possibilities and imagination artistically. Therefore,
we used the clear filament to transmit LED light, making it luminous (see Figure.8).
These luminous sculptures capture the historical moment of disconnection in a poetic and abstract way and
emphasize the fragility of disconnections in social networking. The finished product measures approximately 7 to
8 inches in height and is mounted on 4-inch LED pedestals.
Vol II of Voids
After finishing the first phase of the V olume of V oids, our interest in visualizing the invisible
and transforming it into an interactive experience motivated us to continue our artistic research on visualizing
the mobility of voidness (see Figure.10).
With the techniques of volumetric capturing, machine learning algorithms, and experimental visualization
strategies, Vol II of Voids visualizes the ever-changing voids between people in real-time.
Specifically, a 6-DOP volumetric capturing system fabricates the spatial literacy of reality in real time. In
the new system, multiple layers of experimental visualization strategies, coupled with machine learning
algorithms, can detect humans and generate real-time interactive voids between them.
Volumetric Capturing and Intelligent System Design
In order to capture the space between humans specifically, we utilized an intelligent system for human
identification. Our system aims to recreate spatial reality in a human-centric way. The system is realized
through two stages of development (see Figure.11).
In the first step, we developed a ToF(time-of-flight)- based multi-LiDARs system for 3D data acquisition. (see
Figure12). With the technique of affine transformation and mapping, each adjacent pair of LiDARs are calibrated
in the same matrix space as master and slave so there is no limitation on the total number of LiDARs being used.
As a result, a daisy-chained LiDAR array is able to capture accurate, high-density 3D point clouds of the
designated 3D space in 30FPS real-time. In the current phase of this project, we use four LiDARS for capturing
the installation.
Following the first step of building a ToF-based multi-LiDAR system, we use and combine two machine learning
algorithms to process the camera streaming on each LiDAR. We implement YOLACT++ ⁷ for real-time human
identification in 2D and BackgroundMatting V2 ⁸ for real-time human keying into the Touchdesigner software
environment. We translate 2D identifications received on texture buffers into 3D spatial data with OpenGL
Shading Language, which runs on the GPU. Then we utilize the signed bounding boxes of humans to acquire accurate
spatial relationships between the audiences.
There are inevitably interferences between the liDar units when they face each other. The following steps were
taken to minimize that: 1. We applied genlock for hardware synchronization. 2. We programmed a denoise pattern
as compensation using a GLSL pixel shader then composited it with the texture buffer storing positions of the
point cloud. Finally, color channels got aligned with depth channels.
Compared to the multi-RGBD cameras system, the LiDar system provides better fidelity and precision, even under
poor illumination.
Experimental Visualization and Aesthetic Exploration
To represent the voids between audiences dynamically in real-time, we process the visualization by designing
voxel grids, point clouds, and movements.The voids of the space between audiences are rendered as the following
components:
The voidness between the audiences is calculated by the boolean operation between the volume of the detected
humans and the total volume of the defined area. Therefore, the visualization is presented using 3D texture and
compute shaders to translate space into a dynamic signed 3D voxel grid. We then calculate each work group's (on
the compute shader buffer) local density of the point cloud and apply it to the global density. Inspired by
cubist paintings in the 20th century, a vector field is represented by the white-colored voxels (cubes)
indicating the relative movements by the distances between humans.
The movements of point clouds illustrate the stability of voids. (see Figure.14) For example, when a certain
number of the audience move fast together, their locomotion will be detected to activate the movements of point
clouds. It is achieved by applying a Despiking algorithm and calculating the difference from the raw data. Thus,
the voxels representing voids disperse and accumulate, flee and return, converge and explode, conceptually
mimicking human social behaviors.
Sound Design and User Interaction
In the art installation (see Figure.15), audiences are captured by our 6-DOF volumetric capturing system in
real-time. The visualization of the voids between audiences is presented on a LED screen set in the art
installation. The audience’s body movements will alter the visualizations in real-time and generate
ever-changing fluid voids in the virtual space.
The background sound is non-interactive and looped in the art installation. We transform field recordings of
cityscape sounds into abstract compositions through the technique of granular synth. Inspired by the musique
concrete in the 20th century, Multilayered sound clips are dynamically organized in this composition to create
an atmosphere of emptiness and transiency.
During the pandemic, our physical installations were canceled due to social distancing regulations. We
presented this work in the virtual platform with documentation photos and videos. While the virtual exhibition
is suitable for 3D-printed artifacts, it is not ideal for the interactive experience. In light of the fact that
the pandemic regulations have been removed, we demonstrated our installation to the local community internally
in our research lab space. It is evident that our participants are interested in the visualization and engaged
in the interaction during our demonstration. Rather than providing specific instructions, we encouraged our
participants to move freely within the space and observe the responsive visualization on the screen. Within a
few minutes of interacting, most participants establish a connection between their existence and the
visualization they are producing. Additionally, the experience creates an interesting connection between
participants, they are adjusting their positions in the space in order to capture the interesting spaces between
them.
Future Direction
Presenting the voids as an interactive visualization for participants to perceive and building a collaborative
empathetic experience for disequilibrium are our goals for this series of art experiments. This historical
pandemic not only caused human suffering but also created new possibilities for healing and reconnection. These
invisible spaces bring isolation but also unfold possibilities.
Through presenting this installation to the local communities and the online version to global communities, we
are interested in developing this work further as an art experience using the augmented reality (AR) technique
in a theatre setup. In the future, we will focus on the following developments: 1. We will investigate the
real-time projection-based augmented reality system to present the visualization of voids as a performative
element in space instead of showing the visualization directly on the LED screen. 2. We will further develop the
visualization and redesign it to better fit the new format and medium.
Conclusion
This paper presents the conceptual background, technical implementation, aesthetic and design decisions, and
future directions of Volume of Voids. This project consists of two artistic experiments: 1. Data-driven
sculptures of the voids between humans, 2. Interactive visualization represents the voidness between
participants.
Volume of Voids was initiated and developed during the Covid-19 pandemic when humans keep social
distancing to avoid viral transmission in the air. The voids between humans form an evolving and unstable
abstract volume. According to Taoists, beings and non-beings produce each other. Humans’ activities form
the voids while the voids form the new relationships between humans. The disequilibrium caused the disconnection
in the social network but popularizes interactive technology to reconnect humans in the virtual space.
Volume of voids emphasizes the value of artistic visualization to raise people's awareness of the
possibilities invisible spaces offer, providing a critical response to human network transformation. Through
data visualization and computational design, this work aims to provide a collective experience that emphasizes
alienation between different beings and sustains our curiosity about the world, nature, and social relations.
Acknowledgment
We would like to thank Prof. George Legrady and Prof. Jennifer Jacob for their help and support, the
Experimental Visualization Lab and Media Arts and Technology program (University of California, Santa Barbara)
and the Immersive Media Lab (University of Southern California).
7 Daniel Bolya, Chong Zhou, Fanyi Xiao, Yong Jae Lee, "Yolact++: Better real-time instance
segmentation", IEEE transactions on pattern analysis and machine intelligence, 2020.
8 Lin Shanchuan, Andrey Ryabtsev, Soumyadip Sengupta, Brian L. Curless, Steven M. Seitz, Ira
Kemelmacher-Shlizerman, "Real-time high-resolution background matting", In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 2021, 8762-8771.
Authors Biographies
Dr.Weidi Zhang is a new media artist and researcher. Currently, she is an assistant professor at Arizona
State University. Her interdisciplinary art and design research investigates A Speculative Assemblage at the
intersection of immersive media design, experimental data visualization art, and interactive art. She holds
her Ph.D. degree in Media Arts and Technology at the University of California, Santa Barbara.
Shaoyu Su is a new media artist, developer, cinematographer and curator. His artistic practices and research
employ interdisciplinary methodologies from computer graphics, photography, physics and literature,
investigating concepts about deep spacetime-mining: alternative projection from ancient history to the far
future. From mid-2016 to present, Su has been working at the Immersive Media Lab, USC School of Cinematic Arts
as Director of Technology, leading research and production in cinematic and generative XR and real-time
animation.
The Worlds of Entanglements: Reflection on Posthumanist Ontologies in Art&Science Projects
MashaZolotova
Linz, Austria zolotovamsh@gmail.com
Abstract
A posthuman turn can be observed in many contemporary discourses. It is characterized by an endeavour to
overcome the anthropocentric picture of the world, excluding from it the ‘gap’ between the world and
human, a belief in human exclusivity, as well as the practice of organizing dichotomic systems.
The article relates the theories developed by Braidotti, Haraway and Ferrando and the notion of posthuman
subject to the artistic exploration of the work with nonhuman others, trees, and fungi in particular. By talking
on the posthuman subject and its ontological state, but more precisely by following the way we are interacting
with nonhumans within the practice of selected artists, Olga Kisseleva and Saša Spačal, what their
connections to humans are, the movement towards rethinking the role of humans and acknowledging those by whom we
are surrounded can be traced. Thus, artists do not formulate their notion of posthuman subject, but rather they
demonstrate another approach and attitude towards nonhumans: to consider the latter, trees or fungi in our case,
as partners, not to build hierarchies because the equality of the partners and to acknowledge their
‘knowledge’ we can learn from.
“Who that ‘we’ is, and how to keep that collectivity open, multiple and
nonhierarchical.”—Rosi Braidotti. ³
The discussion on ecological crisis, the Sixth mass extinction, capitalistic overproduction and further
problems connected to them, as well as a growing interest in postcolonial and feminist perspectives can be
observed in many contemporary fields of thoughts, from scientists and academics to artists and activists. What
unites these intellectual pursuits is an attempt to overcome the Cartesian understanding of a man and to go
beyond anthropocentrism. But while humans, it would seem, are at the pinnacle of their domination over nature,
the interconnectedness and equality of human and nonhuman life forms are becoming more generally recognized and
being associated with a posthuman turn.
By working with living beings, plants or animals, artists gain an opportunity to reflect on the subject-object
continuum from different perspectives. While some artists still consider living entities as an object for their
investigation, others are moving forward and distinguish them as their partners or collaborators. Within this
paper the focus is made on the analysis of pieces by two artists, Olga Kisseleva and Saša Spačal.
They both work collaboratively with scientist and their long-term research is dedicated respectively to trees
and their communicative networks within the multi-faceted project EDEN
(Ethics–Durability–Ecology–Nature), and to fungi, their embeddedness in the human
environment at the installation Myconnect and further the MycoMythologies.
Taking art&science field, the one that by its definition is based on the dialogue and interaction between
different actors, as a prominent ground for discussion, the following question is raised in this paper: how does
new understanding of the world, informed by posthumanist and new ontological perspectives, the
interconnectedness between human and nonhuman agents, inform the field of art&science collaborations and
reveal itself in the artworks?
Posthuman turn: more-than-human forms of existence
A posthuman turn can be observed in many contemporary discourses. It is characterized by an endeavour to
overcome the anthropocentric picture of the world, excluding from it the ‘gap’ between the world and
human, a belief in human exclusivity, as well as the practice of organizing dichotomic systems (such as subject
and object, nature and culture, man and woman, and etc.). This term is, rather, a peripheral cultural-discursive
theory in modern philosophy, that combines several social, ethical and natural science concepts, as well as
developed ontological and epistemological perspectives. Following Francesca Ferrando, posthumanism is defined by
three aspects: post-humanism, post-dualism and post-anthoropocentrism. Post-humanism implies a rejection of
classical humanism, that is, a universal and generalized understanding of man in favour of a plurality of
connotations of nature and human capabilities. Post-anthropocentrism seeks to deconstruct a centrality of human
figure, and post-dualism seeks to reject the rigid way of identity, represented in the symbolic dualistic
categories. At the same time, the prefix ‘post-’ in the word posthumanism does not necessarily
indicate the end of human as a species. ‘Post-’ refers to all the historical and non-historical
experiences of the existence of nonhuman others, both in the era before and after humanism. Thus,
‘post-’ in posthumanism moves our attention to what for centuries was excluded from the humanistic
picture of the world or was subjected to repression ¹⁷. We could notice the emphasis on the analysis
and critique of the notion of ‘man’ formed in the framework of the Enlightenment. Man as an
autonomous being, his consciousness and the predominance of his reason, as well as his superiority over other
human, animals and plants, nature in general, becomes a subject for the re-evaluation and negation.
Olga Kisseleva focuses her attention on trees: starting from the Elm of Biscarosse then Japanese cedars or
Kauri trees in New Zealand. The artist was interested in establishing the missing connection among trees of the
related species and facilitating their communication over the continents through specifically developed
technological network, so to help them survive. Devices that measure, among others, VOC (volatile organic
compounds) or barrel thickness, the basic characteristics to be captured for studying plants, have been
connected through internet and thus, could pass over the signals on the distance what is not possible on a
biological level without such technological mediation. So, the first iterations of the project were focused not
on building contact in between human and tress, but to connect nonhuman beings with each other. The shift of the
focus from tree—human to tree—tree communication, at my point of view, is a sign of putting away
pre-described hierarchies and focusing on the sentience of trees at the same level of importance as the human
one, an approach to provide an alternative vision. Saša Spačal, on her side, postulates that her
focus placed primarily on the posthuman condition that supposes humans are not masters of the world, but rather
their existence and actions are elements of a much wider and complicated ecosystem. Since the beginning of the
2010s, she has been working with different biological matters as fungi in Myconnect (2013), plants, red
clover in particular, in the Symbiome installation (2016), bacteria, as Mycobacterium vaccae in
Inspiration (2018) and others. By addressing the above-mentioned nonhumans and by building the artistic
statement using biotechnological methods in tight connections to and with them, Spačal reveals
interactions appearing between different living beings and the way they are framing up our environment
¹¹ .
The definition of the subject is a first one out of the ontological assumptions for criticism and
reorientation. Current re-evaluation leads to further exploration of the boundaries, with the implicit notion
that living can be attributed equally to human and nonhuman beings, or in a broad spectrum of more-than-human
forms of existence. As noticed by Karen Barad, this is an opportunity to overcome the centrality of human taking
into account the “differential constitution and differential positioning of the human among other
creatures (both living and non-living)” ¹ .
Embedded and embodied aspect of the posthuman subject
Ontology offers a critical shorthand for exploring the characteristics of being, human or nonhuman. If
previously, objects were distributed over different ontological regions and logical classes, now these
heterogeneous entities find themselves on the same ontological plane. They are all actors, equal in their
ability to act and interact, in which it is determined what they are and what they will be. ¹⁰
The relational dimension became the crucial point in defining the posthumanist ontology. Rosi Braidotti defines
a posthuman subject as “a relational subject constituted in and by multiplicity, that is to say a subject
that works across differences and is also internally differentiated, but still grounded and accountable”.
² Building the analysis on the relationality, she emphasized the embedded and embodied aspect of the
posthuman subject. At the same time, to describe the relations between human and animal, Donna Haraway uses the
notion of knot, “a knot of species coshaping one another in layers of reciprocating complexity”.
⁸ As a result, further characteristics of a new ontological state have been formulated on the basis of the
theoretical convergence of the directions mentioned above: we are not existing in the dialectical world based on
the opposition of external and internal, so humans are not the dualistic entities, they are rather
“materially embedded subject-in-process” that are disseminated within the networks of relations with
“forces, entities and encounters”. ⁵ Thus, the world is built on the interconnectedness and
multispecies entanglements. We are not alone and separated, we are connected, embedded, and entangled.
Taking this point as a basis, Spačal is willing to reveal interweavings with the help of mycelium and
fungi, trying to reflect on the ways fungal underground networks could inform humans and how this learning
process can be mediated and shaped by technologies. The artist investigates “how fungal ontologies could
help humans practice multispecies survival through practice of inclusion and caring” ¹⁴.
Within MycoMythologies series the artist tries to make these connections perceptible to viewers
with the help of visual and audial tools. The entanglement presented is intended to let us think about what kind
of networks we are. The entanglement presented is intended to let us think about what kind of networks we are.
As it was summarized by the artist,“By making new connections with other biological species and
environment one might step out of anthropocentric perspective even if for just one moment to experience fresh
air of other possibilities”. ¹³
The installation MycoMythologies: Rupture serves as a proof of the biotechnological connections in
planetary life: the screens are linked with each other and any change in the system, the way nutrients are
distributed and taken out of it, causes changes in video and audio representation, while in MycoMythologies:
Patterning fungal structure becomes a great metaphor for the interconnected world: here the microbiome
reveals the influence one actor could have on another. Within its framework, a tree reveals itself as a
posthuman subject: embodied, and embedded, connected to other entities and being a part of a broader network
that manifests itself by the relations that appear within and constitute it. The ways the project was presented
in different locations around the world were chosen to show a tree as an ecologically interlinked being with
multiple connections, as an equal partner that has its own perception of time and world around, holds a
particular knowledge that could be shared for the good of both actors, a human and a tree. Art, this way,
generates new knowledge and expresses more reasons to understand relations we are entangled in, to take more
agency into shaping the world. Olga Kisseleva takes a scientific discovery, works with it within different
contexts, and then presents to the wider audience by revealing internal issues we have with nature, culture, or
history.
Posthuman subjects are a work-in-progress: they emerge as both a critical and a creative project within the
posthuman convergence along posthumanist and postanthropocentric axes of interrogation. In its critique of the
dichotomy between human and nonhuman, posthumanism offers not to prioritize one category above the other, but
“embrac[e] both relationally, as intraconnected actants in an open and respondent context, which is also
constantly shifting” ⁶. As it could be understood, we are not talking about separating the nonhuman
into a hermetic category, but, on the contrary, about its complication, amplification; not about drawing the
strict line and making a strict distinction between human and nonhuman, but about multiplying boundaries and
dividing lines from within the nonhuman. Thus, the nonhuman is always already implicitly present in human: the
border has already been broken, and the threshold has been crossed.
Nonhuman knowledge
Acknowledging nonhuman beings and assigning them the same ontological status, including admitting the value of
their experience as well, can open up the horizons for further scientific and artistic research. Thus, the
investigation on plant intelligence as a scientific track could be considered as a point in science to promote a
new attitude towards a subject of investigation previously considered as a non-sentient being due to the absence
of brain and neural system in comparison to animals. With the development of Critical Plant Studies further
changes could be followed: plants are not considered anymore as passive objects, they are sentient beings that
could respond to the changes in the environment, and they have adaptive behaviour, communication possibilities,
intelligence, memory ⁷.
Sasa Spačal started to work with fungi and investigate with scientists their ability to capture and pass
signals back in 2012 as a participant of a bio-hack workshop. In MycoMythologies series, a speculative
artistic research project, fungi are already considered as full participants of the storytelling creation, as
well as the narrative transmission to the viewers at the exhibition space. They are not the objects to be
observed, but rather subjects with their own perspectives, the holders of their own knowledge to be learned
from. And this collaboration with nonhumans is fuelled by technological mediation. Not only advanced software,
but even the simple microscope facilitates such interactions while just making visible something that is hidden
from our eyes.
The EDEN project gives voice to trees, creates the digital representation of what cannot be grasped by
our senses – to be seen and smelled, to capture the flux of information. The system studies the links
between digital and biological networks in real time, creating a poetic and artistic vision of the original
communication, it leaves the first place to nature, “not exploited or oppressed, but listened and
heard” ⁹. At the same time, it is crucial that such an exposure talking into the account the
specificity of its subject. The application presented at WRO Biennale in 2019 was supposed to change colours
appearing on the screen and let visitors follow how image is changing. So, they can shift their attention to the
temporal aspect of the tree beings, the fact their communication is much slower than human one, how different
the speed of response is. There is no clear answer or transcription of this ‘tree speech’, but it
rather lets visitors experience its temporality. Trees are considered within the EDEN project not only
as silent witnesses but also as the actors who can transmit this knowledge further and at some point - learn
from it.
So, the knowledge we can receive from plants, animals, and natural phenomena can be valuable not only for
humans, but also for the technologies we are working on. Thus, it became clear for the artist that we can learn
from trees or fungi something that is not accessible to learn in any other way, so they are our partners. This
way of thinking is a crucial part of posthuman convergence, as it is addressed by Rosi Braidotti, because zoe
let us to acknowledge that
“thinking and the capacity to produce knowledge is not the exclusive prerogative of humans alone but is
distributed across all living matter and throughout self-organizing technological networks”. ⁴
Networks of entanglements
Living organisms are highly entangled cannot be grasped without their networks of entanglements, without
putting attention on the connections they have with other living beings and their environment. The most obvious
example to illustrate this point is the human, in fact, not an independent living entity, but that coexists with
other organisms even within itself, as Anna Tsing writes, “[h]uman nature is an interspecies
relationship”. ¹⁵ Moreover, the posthumanism and non-anthropocentric model are associated not
only with the decentring of the human, but also with the decentring of a subject’s position as such. It
shifts the focus to the production of subjectivity, again not a fixed notion, but embedded and processual
characteristic.
The first case of addressing to fungi in Saša Spačal’s artistic practice was the installation
MyConnect that she defines as “a symbiotic interspecies connector that questions anthropocentric
division of nature-human- technology”. ¹² Within the stand-capsule developed, the signals and
impulses create a feedback loop by the means of biological and technological mediation that lets visitors
experience the symbiotic interdependence between their bodies and fungal mycelium in a very tangible manner. The
signal coming from the viewer’s heartbeat moves along the mycelium in petri dishes: Spačal is using
either Oyster mushrooms (genus Pleurotus), or Shiitake (Lentinula edodes), and is processed by it in real time.
The processed signal is transmitted back to the human body in the form of sound, light, and tactile impulses.
Their impact on the nervous system causes a change in the heart rate in the viewer and starts a new cycle; the
process can be repeated endlessly; thus the human nervous system and the mycelium of the fungus are closed on
each other in a feedback loop. This lets the symbiosis of signals begin.
The electrical feedback in fungi that could be measured and traced in real time and allow the viewers to have a
proper interactive experience is one of the major points to choose this species. Fungi, being one of the most
prevailing life forms, and mycelium, creating the symbiotic relationships with plants by forming mycorrhiza and
facilitating the nutrients flow among them, demonstrate their great potential to be a basis for the reflection
on symbiotic relations and interconnections that are inscribed in our nature. Thus, fungi, mostly invisible to
the human, as soon as they are started to be investigated, reveal themselves in every leaf, every piece of soil,
they are everywhere. No matter that they are part of human life, they are embedded in our lives, even though
they are unseen, covered by soil and having a small scale in comparison to the human environment.
As mentioned already, Olga Kisseleva focused her attention on the communication, chemical interaction between
trees and new systems that can evolve with the directed technological mediation. Another element of the EDEN
project as “an organic network based on vegetal medium” is a datascape. It is an interactive
program that indicates and unites the actors in the network, more precisely, trees connected to the system
through sensors and captures, gathers the information and later serves as a source for all visual displays. By
creating this network connection, the artist and the scientists were interested in allowing the trees to
communicate despite the distance. Highlighting that it is possible to understand and receive the communication
of a tree is one thing, to consider within the interdisciplinary group of the specialists that it can be
transmitted and appreciated by one of its congeners living on a different continent is another.
As it can be seen, it is crucial to work collaboratively in the intersection of art, science and technology,
this approach can reach beyond and let us think more on the way our world is constructed and how everything is
linked within it. How can we live together? How can we acknowledge our mutual existence, the connections
existing in our networked society and world?
Conclusion
Overall, it can be stated that posthumanism, by itself, has become an umbrella term that includes different
perspectives, in an era where the symbolic boundaries of the ‘human’ have been ultimately
challenged. Along with that, posthumanism is interested in how one should think in order to coexist with
nonhuman life forms and offers an ethical and political program (Haraway 2003, Braidotti 2019).
The ontological effort lies in acknowledging that the human may not be human after all, which calls to re-think
existence and being in the world. The consequence is epistemic because if we assume that our being and becoming
is different from what we previously thought (given that we are likely to be implicated in a posthuman life), we
can no longer explain how we experience and think in conventional epistemological terms. The enquiry into a
posthuman condition is to revisit the constitution and function of the world and life.
By talking on the posthuman subject and its ontological state, but more precisely by following the way we are
interacting with nonhumans within the practice of selected artists, what their connections to humans are, we can
trace the movement towards rethinking the role of humans and acknowledging those by whom we are surrounded.
Thus, artists do not formulate their notion of posthuman subject, but rather they demonstrate another approach
and attitude towards nonhumans: to consider the latter, trees or fungi in our case, as partners, not to build
hierarchies because the equality of the partners and to acknowledge their ‘knowledge’ we can learn
from.
This emphasis on the relations appearing among subjects and in the system as a whole, the tendency to make
these interconnections explicit are the major posthumanist categories the artists refer to. As it was noticed by
Anna Tsing, artistic projects let people learn “to accept that we are all part of the same interconnected
system” ¹⁶. Thus, I suggest considering approaches proposed by Olga Kisseleva and Saša
Spačal as an indicator of change that could lead us to the refusal of a purely anthropocentric
perspective. By not prioritizing one category over another, human over nonhuman or vice versa, by opening up
such an approach to the scientific community during the collaborative work and to wider audience in the
framework of exhibitions and art festivals, we are going towards a new posthuman subject.
References
1 Karen Barad, Meeting the Universe Halfway, Durham, London, Duke University Press, 2007,
p.136.
2 Rosi Braidotti, The Posthuman, Cambridge, Polity Press, 2013, p.49.
5 Rosi Braidotti, Maria Hlavajova, eds. Posthuman Glossary. London, New York,
Bloomsbury Academic, 2018.
6 Francesca Ferrando, Philosophical Posthumanism, New York, Bloomsbury Publishing, 2019,
p.165.
7 Matthew Hall, Plants as Persons: A Philosophical Botany, New York, State University of
New York Press, 2011.
8 Donna Haraway, When Species Meet, Minneapolis, University of Minnesota Press, 2008,
p.42.
9 Olga Kisseleva, Du travail collectif a l’oeuvre, Paris, Nouvelle Edition Place,
2017, p.97.
10 Bruno Latour, Science in action: how to follow scientists and engineers through
society, Cambridge, Massachusetts, Harvard University Press, 1987.
11 Saša Spačal, “Connection continuum: a life”, in Experiencing
Unconventional: Science In Art, ed. Theresa Schubert and Adam Adamatzky, 2015, p.175-206.
15 Anna Tsing, “Unruly Edges: Mushrooms as Companion Species”, Environmental
Humanities 1, 2012, p.144.
16 Anna Tsing, “When Things We Study Respond to Each Other: Tools for Unpacking ‘The
Material’”, in More-than-Human, ed. Andr.s Jaque, Marina Otero Verzier, and Lucia
Pietroiusti, Rotterdam: Het Nieuwe Instituut, 2020, p.83.
17 Cary Wolfe, What is posthumanism? Minneapolis: University of Minnesota Press, 2010.
Bibliography
Karen Barad, “Posthumanist Performativity: Toward an Understanding of How Matter Comes to
Matter”, in Signs: Journal of Women in Culture and Society, vol. 28, No 3, Spring 2003,
801-831.
Laura Beloff, “Hybrid Ecology – To See The Forest For The Trees”, in Art As We
Don’t Know It, ed. Erich Berger, Kasperi Mäki-Reinikka, Kira O'Reilly and Helena Sederholm,
Aalto, Aalto ARTS Books, 2020.
Ryszard W. Kluszczyński, “Looking at the World through the Eyes of Other? Art as
Non-Anthropocentric Ecology”, in Victoria Vesna. Towards a Non-Anthropocentric Ecology and Art in the
World of Anthropocene, ed. Ryszard W. Kluszczyński, Gdańsk – Łodź: Laznia Center
for Contemporary Art, Łodź University Press, 2020, 6-27.
Regine Rapp, On mycohuman performances: fungi in current artistic research, Fungal Biol Biotechnol 6:22,
December 2019, doi: 10.1186/s40694-019-0085-6
Jakob von Uexküll, A Foray into the Worlds of Animal and Humans, with A Theory of Meaning, Translated
by Joseph D.O’Neil, Minneapolis, University of Minnesota Press, 2010.
Rosi Braidotti, “Posthuman, All Too Human: Towards a New Process Ontology”, in Theory,
Culture and Society, 23, 7–8, p.197– 208.
Richard Grusin, ed. Nonhuman Turn, Minneapolis, University of Minnesota Press, 2015.
Donna
Haraway, “Symbiogenesis, Sympoiesis, and Art Science Activisms for Staying with the Trouble",
in Arts of Living on a Damaged Planet. Ghosts and Monsters of the Anthropocene, ed. Anna Tsing,
Heather Swanson, Elaine Gan and Nils Bubandt, Minneapolis: University of Minnesota Press, 2017, 25-50.
N. Katherine Hayles, How we became posthuman, Virtual bodies in cybernetics, literature, and informatics,
Chicago, The University of Chicago Press, 1999.
Erich Hörl, James Burton, ed. General Ecology: the New Ecological Paradigm, London, Bloomsbury
Academic, 2017.
Summit on New Media Art Archiving papers
The International Programming Committee (IPC) has not reviewed Summit on New
Media Art Archiving papers. Individual authors of papers and presentations are solely responsible for all
materials submitted for the publication. The publisher and the editors do not warrant or assume any legal
responsibilities for the publication’s content. All opinions expressed in the book are of the authors and
do not reflect those of the publisher and the editors.
The Summit on New Media Art Archiving (SNMAA) series has been a platform and
communication channel for stakeholders in new media archiving, aiming to facilitate critical discourse and
collaboration. Since 2020, the annual Summit on New Media Art Archiving has become one of the most important and
focused conferences for related topics. The Summit engages and connects members of the new media art archiving
community for sharing knowledge, exchanging ideas, networking, and seeking collective strategies to tackle
mutual archival challenges for the everchanging new media art, the art of our time.
After the massive support for the Liverpool Declaration, the Summit initiative
started with ISEA round table discussions on new media archiving at ISEA2018 in Durban, South Africa, and
ISEA2019 in Gwangju, South Korea. The one-day First Summit on New Media Art Archiving was held during ISEA2020
(online from Montreal, Canada), featuring a keynote lecture, paper presentations, and break-out sessions. This
Second Summit on New Media Art Archiving was held two days preceding ISEA2022 in Barcelona, Spain. Subjects
included, among others, archiving physical artifacts, digitising museum collections, ethical aspects of
archiving new media art and innovative approaches to archiving, as well as an emphasis on connecting new media
art archives worldwide. Part of the Second Summit was a unique art exhibition based on submissions of artworks
inspired by the concept of archiving.
The third summit was organised by ISEA archivists in cooperation with archivists
from SIGGRAPH, Ars Electronica, Archive of Digital Art (ADA), ZKM, Electronic Language International Festival
(FILE), Media Art History, and Memoduct Posthuman archive. The summit has been supported by ISEA2023, ISEA
International, the SIGGRAPH Digital Arts Community and the Creative Industries Fund NL. It took place on May 19
and 20 as part of ISEA2023, the 28th International Symposium on Electronic Art.
Ziegler Michèle – Director of Digital Strategy and NewImages Le Forum des images
Bouvier Nathalie – Director of Event Production Le Forum des images
Farge Claude – CEO Le Forum des images
+All the venue's stage managers
ISEA2023 50 Volunteers:
Petkova Blagomira (art student at Université de Lille 3), Maurer Louise (art student at
Université Sorbonne-Nouvelle Paris 3), Poussiere Maud (art student at Icart), Pousse-Wang Sarah (art
student at Sorbonne Paris 1), Guo Songyu (design student at Ecole supérieur d'art et de design TALM-Le
Mans), Falco Mario (3d student at ArtFx), Cardon Mila (3d student at ArtFx), Cherston Juliana (student at MIT
Media Lab), Pousse-Wang Liyah (art student at Sorbonne Paris 1), Alicia Clotilde (media student at
Université Gustave Eiffel), Nilles Clarice (International business student at Paris 12), Anthousi
Nitous (art student at Sorbonne Paris 1), Deiss-Veiniere Lou (communication at Sciences Po Lille), Bernard
Léon (art student at ESACM (Ecole Supérieure d'Art de Clermont Métropole), Viennois
Franck (art student at Beaux-arts de Montpellier), Combret Yumé (cinema student at Sorbonne Nouvelle),
Durel Nina (art student at ESACM (Ecole Supérieure d'Art de Clermont Métropole),
Guérin-Garnett Nigel, Guiganti Asbjørn, Duverl Christelle, Paouri Arghyro, Malu Laet, Lacourte
Manon, Lesbats Lucie, Balland Enzo, Ageron Julie, Belhadj Manelle, El Shérif Mickaël Léna,
Lehna Robert-Gonçalves, Jonathan Ouali, Roumeissa Langrene, Léa Bouarfa, Rekouane, Selim Marion,
Seknazi Sephora, Baryshnikova Anatsasiia, Aziosmanoff Manon, Fillon Arnaud, Fried Axel, Plisson Laura,
Usanakul Ussareena, Gabriela Camille, Marion Beltrão, Gabriela Waheo, Clara Pena, Winnie Reial, Adama
Lau, Ralambotiana Fall, Malala Tsanta.
The Steering committee
École des Arts Décoratifs – PSL and Le Cube have set up an
organising committee which, for the first time in France, brings together all the key players in
digital-related research and culture.
École des Arts Décoratifs – PSL works closely with all these
players, sometimes co-founding academic consortia with some of them. ISEA SYMBIOSIS makes it possible to give
visibility to this range of already existing collaborations, thus demonstrating the strength of the national
research-creation ecosystem. The objective of ISEA SYMBIOSIS is not limited to the event itself; it is a step
towards structuring the field on a territorial scale, which will produce similar effects in Montreal in 1995:
the arts and digital culture scene is now recognised throughout the world (the Hexagram network associating
Concordia University and UQÀM, right up to Ubisoft, stems from this structuring effect). The next step
is to consolidate this ecosystem, in which all the players have solid international (European and foreign)
networks, thus continuing the efforts of 2023. The symposium proceedings will also move in this direction,
with a text specifically dedicated to ecosystems, based on the symposium's session of institutional
presentations. This session has also been designed with ecosystem development in mind.
In connection with the ISEA2023 international board, the French organising committee, co-piloted by the Cube
Garges and the École des Arts Décoratifs, brings together leading players in the field of the
arts, creative industries, higher education, research and innovation.
Members:
Ancel Franck – Artist – Theorist
Aziosmanoff Nils – Director of the Cube Garges
Barthe Pauline – Communication Officer and Co-organizer of the academic call ISEA2023
Behar Armand – Co-director CRD, ENSCI, Les Ateliers
Berthoumieux Céline – Director ZINC – Co-director CHRONIQUES – President d'HACNUM
Bonneau Alicia – Partnership Officer ISEA2023
Bureaud Annick – Director Leonardo OLATS – Independent curator
Buser Edith – Associate Director of Research at the
École des Arts Décoratifs – PSL – Co-organizer of the
academic call ISEA2023
Cadon Benjamin – Director Labomédia
Carvalho Guilherme – Researcher in virtual environment in charge of ISEA2023 – MSH Paris-Nord
Chevrier Joël – Professor, in charge of the ISEA2023 project – Learning Planet Institute
Professor
Citton Yves – Researcher – Theorist – Teacher Paris 8
Dallet Jean-Marie – Professor and artist ACTE Institute (Sorbonne)
Dondey Marc – Director ENS Paris-Saclay La Scène de Recherche
Farge Claude – Director Forum des images
Fleischer Alain – Director Le Fresnoy
Foresta Don – Artist – Researcher – Theorist – Teacher
Guez Emmanuel – Director ESAD Orléans
Krajewska Klio – Production Director ISEA2023
Lahlil Sophia – Head of Administration and Coordination La Scène de Recherche ENS Paris-Saclay
Le Poullennec Annaël – General Manager EUR ArTeC
Madlener Frank – Director IRCAM
Mahé Emmanuel – Director of research at the
École des Arts Décoratifs – PSL – Academic chair ISEA2023
Michigan Sonia – Production Assistant ISEA2023
Portevin Jeanne-Marie – Coordinator EUR ArTeC
Prigent Eric – Digital Creation Coordinator Le Fresnoy
Reyes Everardo – Vice President Digital Paris 8
Sedes Anne – Director MSH Paris-Nord
Selmane Inès – Project manager in the Digital Development Department Forum des images
Taddei François – Director Learning Planet Institute
Testas Fanny – Communication and production Officer ISEA2023
Thibault Clément – Artistic Coordinator ISEA2023
Vinet Hugues – Scientific Director IRCAM
Ziegler Michèle – Digital Development Directorate Forum des images
ISEA International Board members:
Erandy Vergara Assistant Professor in the Faculty of Fine Arts and Communications,
MacEwan University and Curator, Printemps numérique
Everardo Reyes Academic, Professor of Information and Communication Sciences,
Co-director of the Masters in Digital Humanities, Université Paris 8 – Vincennes-Saint-Denis.
Responsible for international relations, EUR ArTeC. Associate member of the Cultural Analytics Lab. Art Papers
Chair for SIGGRAPH 2019 – Los Angeles.
Michel Van Dartel Director (V2_ Lab for the Unstable Media) and Professor (Avans
Centre of Applied Research for Art, Design and Technology).
Pat Badani Special Advisor to ISEA2020 & ISEA2023; Chair of Community Network
Subcommittee 2017-2022; IIAC member (ISEA International Advisory Committee) 2013-2016. Independent cultural
researcher/producer; former Integrated Media Professor at the School of Art, Illinois State University, USA;
Interdisciplinary Media Arts graduate program acting Director at Columbia College Chicago; Editor-in-Chief of
Media-N Journal, the New Media Caucus; and Lead Guest Editor of “Artelogie,” EHESS, France.
Ricardo Dal Farra Professor of music and media arts at the Music Department of
Concordia University in Canada. Director of the Electronic Arts Research and Experimentation Centre (CEIArtE)
at National University of Tres de Febrero in Argentina. Directing the Latin American Electroacoustic Music
Collection. Founder-director of the Understanding Visual Music (UVM) and Balance-Unbalance (BunB)
international conference series.
Tanya Ravn Ag, PhD Danish curator and scholar examining perceptual experience, expanded
reality, temporality, and technogenesis in relation to urban and virtual media aesthetic phenomena and art.
She was a visiting fellow at the School of Creative Media, City University of Hong Kong, and is currently a
postdoc researcher at the Institute of Arts and Cultural Studies, University of Copenhagen.
28th International Symposium on Electronic Art ISEA2023
Publishing Director: Emmanuel Tibloux, Director of École des Arts Décoratifs – PSL Editor:
Emmanuel Mahé, ISEA2023 Academic Chair, Head of Research of École des Arts Décoratifs –
PSL Copy-editing: Pauline Barthe, Edith Buser, Elsa Carnielli Proofreading: Pauline Barthe, Elsa
Carnielli, Manuelle Freire Publicization Special Adviser: Lucile Haute Graphic Design: Quentin
Juhel Types: Syne by Bonjour Monde, Open Runde by Laurids
Kern
ISEA2023 proceedings has been designed with frugality inmind. Thepdf version features black typography onawhite
background instandard A4format, which saves ink and paper for (full orpartial) office printing. Theweb version isstatic and resource-efficient. Ithas been made with the editorial platform BOOKOLAB, acollaborative tool for editing and formatting
publications for different reading media (printable version asPDF & screen version asweb page) designed byLucileHaute, and developed byArmanMohtadji & BenjaminDumond with GRAV and Paged.js, with the support ofUniversité deNîmes and ÉcoledesArtsDécoratifs–PSL. Projekt source code: gitlab.com/armansansd/bookolab
No part of this publication may be reproduced, stored in aretrieval system, or transmitted inany form orby any means, without prior written permission of the individualauthors.
Individual authors of papers and presentations are solely responsible for all materials submitted for the
publication. Thepublisher and the editors donot warrant orassume any legal responsibilities for the publication’s content.
Allopinions expressed inthe book are ofthe authors and do not reflect those ofthe publisher and theeditors.
Despite intensive research, it was not possible to identify all copyright holders. Ifproper acknowledgment has not been made, we ask copyright holders fornotice.