DOI: https://doi.org/10.33008/IJCMR.202017 | Issue 5 | October 2020
Dr. Richard A Carter (richard.carter@roehampton.ac.uk)
University of Roehampton
Abstract
The film Waveform is an output from a project of the same name that has been ongoing since 2017, and which centres on the use of drones as tools for generating poetry. Initially, a drone captures aerial images of incoming ocean waves. These images are then analysed by a machine vision system that traces the boundary between wave and shore. This boundary provides a stream of variables for another algorithm that generates short, poem-like texts, which meditate on questions of environmental sensing and sense-making.
Waveform is presented as a speculative apparatus—a fragment of a world in which sensory technologies are employed for creative, notional ends, rather than a source of purely specular representations. The project considers how these devices might be used to re-articulate the world not as a static formation, with an embedded array of attributes awaiting detection and visualisation, but as one that is emerging and transforming continually—with sensory devices contributing also to this emergence. Here, quantitative outputs in the form of numerical graphs and charts give way to poetic texts that hint not only at the scene being imaged, but also convey aspects of the sociotechnical contexts in which these acts of sensing and interpretation occur, and which often go unacknowledged. The point of this gesture is consider how these contexts are both integral to the ways in which sensory acts are framed, and to consider how they might be reframed in-turn so as to enable new modalities of sensory unfolding and becoming. Such openness to alternative potentials for sensing and knowing will be critical in negotiating the forces of severe ecological disruption and degradation which characterise our present moment.
Research Statement
Waveform is a 10-minute film that was first shown in July 2019 at the Glucksman Gallery, Cork, as part of its ‘Peripheries’ exhibition (see Clarke and O’Sullivan 2019). It represents the latest creative output from a practice-led research project of the same name, and which has been ongoing in recent years (see Carter 2017). This project is premised on the use of airborne drones and machine vision algorithms as experimental tools for generating poetry—curating an unconventional matrix of technologies and scholarship in order to investigate the role of sensory systems in mapping and characterising the state of the wider environment at a time of profound ecological stress. All the creative work emerging from this project depicts the processes by which the aerial imagery is transformed into poetic outputs, offering catalysts for critical reflection on the steps involved.
At first, a camera drone is flown above a coastal shoreline, capturing top-down images of the incoming waves. These images are then analysed by a machine vision algorithm that traces a line demarcating the mutual boundary between surf and shoreline. The detection markers connecting this boundary then provide a stream of variables for the poetry algorithm itself. This employs the principle of a Markov-chain, in which a textual corpus is tabulated into a set of probabilities between its constituent words—that is, for every given word identifying those most likely to follow. Using this data, the marker values can be used to assemble a chain of word selections from the corpus, weighted sufficiently as to echo its grammatical and stylistic nuances, while still yielding intriguing juxtapositions (see Carter 2018 for a detailed technical account).
It is evident from the above description that this assemblage is enacting an elaborate form of cut-up poetry, and, from a purely technical standpoint, there is only a narrow connection between the source image and the outputted text. This connection is, however, enriched substantially by the subject locations of the drone footage itself and the corpus fed into the poetry algorithm. In the case of the Waveform film, this involves overhead shots of a remote Cornish beach known as ‘The Strangles’, and the use of text from R. M. Ballantyne The Ocean and Its Wonders (1874). While neither of these aspects are referenced expressly in the film, they are, nonetheless, a useful vector for considering its potential for meaning.
At the film’s beginning the viewer sees a fixed overhead shot of rolling waves, in high-contrast black and white (see Fig. 1), accompanied by an ambient coastal soundscape—mixed by the artist’s collaborator Dr Mariana López. After a few minutes, this soundscape gives way to the rush of cooling fans and humming electronics, while the image itself freezes and transitions into a heavily pixelated rendering of the same scene.
Fig. 1. The initial shot Fig. 2. The boundary line
This rendering then fades into another image depicting a bright line tracing the edges of the shoreline (see Fig. 2), before transitioning to reveal a short poem (see Fig. 3). After a few moments have passed, the poem fades, movement returns to the background scene, and the coastal soundscape prevails once again. This sequence is repeated several times throughout the 10-minute running time, all while maintaining the fixed overhead perspective on the beach scene—and so, effectively, breaking up an otherwise continuous fixed shot.
Geographically, the narrow, rocky strip that constitutes ‘The Strangles’ is relatively typical of the Cornish coastline. Nonetheless, as a site located at the very periphery of the United Kingdom, facing into the ocean, it also resides near the terminus of major transatlantic submarine cables, which come ashore in the vicinity. These global infrastructures are the critical foundation of the internet, but are almost always hidden from view by the ocean itself and the shoreline sands—as well as shielded by the considerable secrecy around their precise location (see Starosielski 2015).
The Strangles is a complex site in the context of Waveform, and not simply because of the presence of these cables, but also because its relative isolation allows it preserve a record of its exposure to the impacts of contemporary environmental damage and disruption, which more developed coastal sites would seek to mitigate. The Strangles is marked therefore by growing cliffside erosion, large piles of shale gouged by violent storms, and an increasing burden of plastic waste and assorted detritus. The socio-economic patterns driving these effects are fully entangled with, and often enabled through, the online world, which generates its own specific impacts (see e.g. Belkhir, and Elmeligi 2018) while facilitating also the sensory infrastructures through which we have been able to measure and map a deteriorating global environment (see Gabrys 2016). In this context, the drone orbiting high above the beach scene below, and unseen in the film itself, evokes the remote sensing nodes that appear empirically far removed from that which they observe, but are nonetheless embedded within, and perpetuating of, the effects measured.
This juxtaposition between empirical distance and material embeddedness is further marked by how the Strangles remains unnamed in the film itself: an area with a rich local context becomes another grid location to be imaged, one that is demarcated not through its geology or sociocultural ties, but through the rectilinear aperture of the drone’s onboard camera, which reduces the scene to a computable surface of defined brightness values along an exacting grid. This is depicted within the film by arresting the intricate, ephemeral patterns of the rolling surf within the gaze of the machine, converting these into a pixelated rendering of fixed colour values before fading into a single dividing line—designating the signal being sought by the apparatus. This act of freezing and defining a boundary amidst a churning body of water encapsulates the digital ‘flattening’ of emergent worldly processes into discrete, atemporal states for the purposes of measurement and modelling. While functionally necessary and often analytically powerful, such operations invite reflection on what escapes these processes of digital sensing and sense-making—of the myriad phenomena and behaviours that fail to reach the spatial and temporal thresholds of a data-driven episteme (see Carter 2018 and Forthcoming for additional reflections on this point). Included in these hidden effects are the escalating affective impacts on thinking and being that such a rapidly changing world engenders, and how we might respond and adapt subsequently.
It is in this sense that the act of parsing visual data into poetry, using the archaic vocabulary of The Ocean and its Wonders, offers a deliberately unconventional response to such sensory aporia. Taken as a whole, Waveform is presented as a speculative apparatus—a fragment of a world in which sensory technologies are employed for creative, notional ends, rather than a source of specular representations that presume to account for the fullness of what they observe. Heavily remixed by the system, the source vocabulary manifests as a set of enigmatic statements concerning the maritime environment, ranging from the relatively prosaic to the outright fantastical (see Fig. 4 for examples).
At one level, we might read such odd, unsettling imagery as expressing the startling degradation of an imperilled ecology—of the otherwise surreal distortions generated by an extractive, exploitative politics and economics, which take little account of the more-than-human world beyond its potential resource value. Nevertheless, these strange outputs speak also to their novel origins, and we might therefore assess both the poems themselves, and the generative process as a whole, as enacting the potential for new modes of experimental practice, in sensing, writing, and computing, that will only grow in necessity as we seek to comprehend and adapt towards the hazards and disruptions ahead. This is a key point advocated by Gabrys (2018) in her own account of the possible futures of digital sensing, emphasising the value of the speculative as a way of perceiving and realising new potentialities of reading and acting within a damaged ecology. Waveform itself represents a conscious effort to bring this theoretical insight into the domain of concrete practice, to examine what might be learnt in the process, and so joins a number of other such experimental sensory endeavours in recent years (see Carter 2018).
Fig. 4. Some examples of the poetry seen throughout the film.
These efforts at realising new modes of perception are carried into the choice of source vocabulary behind the final poetic outputs of Waveform. Ballantyne’s The Ocean and its Wonders was published during a period when the distinction between literary and scientific endeavours was less recognised and reinforced than at present. Although it emerged at the dawn of contemporary investigations into the dynamics of ocean and climate, its free mixing of the lyrical and scientific also presages an approach seen in more contemporary modes of nature writing, as pioneered by authors such as Rachel Carson, which, in the words of Boscacci (2019: 193) are ‘an aesthetics of practice that composes and speculates with more than words, and across porous boundaries of knowledge from material art-making, science, and affect scholarship’—amplifying an older sense of aesthetics as ‘the discipline through which the organism becomes attuned to its environment’. It is in this sense that Waveform reimagines the potential of digital sensors for articulating the world not as a static, delimited formation—as a body of stable attributes, discoverable by fixed disciplines, awaiting utilisation—but as existing in a state of continuous emergence across a spectrum of multitudinous sensibilities. The fact the boundary lines and poetic texts generated by Waveform are marked by the spectre of incoherency does not, therefore, represent a technical limitation, but an expression instead of the uncertainties and potentials inherent in a world that is never fixed, final, or fully graspable, but is always unfolding in unexpected ways, and so necessitating inventive, imperfect modes of adaptation.
It is in this context that digital sensors can be shown not as standing apart from the world, or as having a uniquely insightful grasp on its overall state—artefacts of their global scale and technoscientific veneration—but as operating from within a wider ecology of world-making practices, of which art and literature represent partners of equal power, even if they may not be valued to the same extent, socially or politically. It is through examining the creative interfaces between these domains that an opportunity is created to give an account of phenomena whose qualities might otherwise exceed the encoded thresholds of detection within established sensory apparatus—which are often defined narrowly as standalone technical devices and systems, marginalising their entanglements with that which they observe, in terms of their concrete functioning, their upholding infrastructures, and the socioeconomic formations that sustain them. It is this which constitutes the principle critical and creative vector of Waveform, as an effort at reevaluating what it means to know and to act in the world from within the crucible of advanced digital technologies, environmental devastation, and an increasingly uncertain future.
References
Belkhir, Lotfi, and Ahmed Elmeligi. 2018. “Assessing ICT global emissions footprint: Trends to 2040 & Recommendations.” Journal of Cleaner Production 177: 448-463. https://doi.org/10.1016/j.jclepro.2017.12.239.
Boscacci, Louise. 2019. “Ecologising affect and atmosphere in the Anthropocene: dear Rachel.” In 100 Atmospheres: Studies in Scale and Wonder, edited by The Meco Network, 193-216. London: Open Humanities Press.
Carter, Richard. 2017. “Drone Poetry: On Deploying Sensory Technologies as Tools of Writing.” The Writing Platform. http://thewritingplatform.com/2017/09/drone-poetry-deploying-sensory-technologies-tools-writing/.
Carter, Richard. 2018. “Waves to Waveforms: Performing the Thresholds of Sensing and Sense-Making in the Anthropocene.” Arts 7(4) 70: n.p. https://doi.org/10.3390/arts7040070.
Carter, Richard. Forthcoming. “Drone Poetry”. In Ambient Stories: Digital Writing in Place, edited by Amy Spencer. London: Emerald.
Clarke, Chris, and James O’Sullivan. 2019. Peripheries: Part of the Electronic Literature Organization Conference & Media Arts Festival. n.p. https://cora.ucc.ie/handle/10468/8138.
Gabrys, Jennifer. 2016. Program Earth: Environmental Sensing Technology and the Making of a Computational Planet. Minneapolis, M.N.: University of Minnesota Press.
Gabrys, Jennifer. 2018. “Becoming Planetary”. https://www.e-flux.com/architecture/accumulation/217051/becoming-planetary/.
Starosielski, Nicole. 2015. The Undersea Network. Durham: Duke University Press.
Comments