At the risk of some sort of blog-will-eat-itself situation, I'm posting this paper, presented at TIIC last November, which includes several threads developed here previously - arrays, transmateriality, and the work of HC Gilje. There are some new bits too however, on screens, projection mapping, and lots of tasty examples of a putative "post-screen" practice.
1. Glowing Rectangles
For all the diversity of the contemporary media ecology - network, broadcast, games, mobile - one technical form is entirely dominant. Screens are everywhere, at every scale, in every context. As well as the archetypal "big" and "small" screens of cinema and television we are now familiar with pocket- and book-sized screens, public screens as advertising or signage, urban screens at architectural scales. As satirical news site The Onion observes, we "spend the vast majority of each day staring at, interacting with, and deriving satisfaction from glowing rectangles."
Formally and technically these screens vary - in size and aspect ratio, display technology, spatiotemporal limits, and so on. They are united however in two basic attributes, which are something like the contract of the screen. First, the screen operates as a mediating substrate for its content - the screen itself recedes in favor of its hosted image. The screen is self-effacing (though never of course absent or invisible). This tendency is clearly evident in screen design and technology; we prize screens that are slight and bright - those that best make themselves disappear. Apple's "Retina" display technology claims to have passed an important perceptual threshhold of self-effacement, attaining a spatial density so high that individual pixels are indistinguishable to the naked eye (below - image Bryan Jones).
The second key attribute of contemporary digital screens is their tendency to generality. The self-effacing substrate of the screen is increasingly a general-purpose substrate - unlinked to any specific content type; equally capable of displaying anything - text, image, web site, video, or word-processor. This attribute is coupled of course to the generality of networked computing; since the era of multimedia the computer screen has led the way in modeling itself as a container for anything (just as the computer models itself a "machine for anything"). The past decade has simply seen this general-purpose container proliferate across scales and contexts, ushering us into the era of glowing rectangles.
However over the past decade in design and the media arts, a wave of practice has appeared which as this paper will argue, resists the dominance of the glowing rectangle. Given the near-total cultural saturation of the screen, this is unsurprising, given the ongoing cultural dance of fringe and mainstream in which this practice participates. This is not simply a story of resistance however. In proposing and describing two particular strains of "post-screen" practice, this paper aims firstly to outline the shared terms of their relationship with the screen, and in the process develop a more detailed sense of these conceptual devices of generality, outlined above, and its opposite, specificity. Secondly, and more briefly, it outlines a theorisation of this practice, invoking transmateriality, an account of the paradoxical materiality of (especially digital) media, and Gumbrecht's notion of presence.
2. Arrays
During the opening ceremony of the 2008 Beijing Olympics, a huge grid of drummers assembled in the stadium, each standing before a large square fou drum, a traditional Chinese instrument. Each drum was augmented with white LEDs mounted on its surface, triggered with each drum stroke. The drummers formed a vast array of discrete audiovisual elements, precisely choreographed in the style of these spectaculars. Human pixels, but coarse and resolutely human; at one point the drummers desynchronised entirely, forming a thunderous grid of flickering light. In a ceremony created for the (broadcast) screen - to the infamous extent of splicing computer-animated fireworks into its telecast in place of real ones - the drummers were a moment of involution. Their array echoed all the other, more conventionally self-effacing screens threaded through the event; but it also inverted some of their key attributes. Firstly its substrate, instead of receding behind "content", came forward; if anything substrate and content were one and the same. Secondly, while this array nods towards the generality of the screen in its choreographed patterns - which like the patterns on a screen could be "anything at all" - it veers strongly in the opposite direction, towards the here and now, what I will call specificity. As I argued at the time, the poetics of this array rely on the specificity of its elements - the drummers, drums, and their solid-state illumination - rather than the patterns that play across it.
The drummers are one popular example of a formal trope we can find throughout media arts and design practice over the past decade. Daniel Rozin's 1999 Wooden Mirror is one of the earlier examples. Wooden Mirror is an array of square wooden tiles embedded in a large octagonal frame, along with a bundle of custom electronics. The tiles are fitted with servomotors, so that each one can tilt up and down on its horizontal axis. As its angle to the light changes, each tile appears brighter or darker. Rozin wires up the array to a videocamera, to complete the mirror circuit: the brightness of pixels in the incoming image drives the angle of the tiles. Given the overtly visual logic of the work, it's interesting that its sound is equally striking: the wooden tiles clatter like mechanical rainfall, sonifying the rate of change of the image; as the image becomes still, the clatter dies off to a low twitching. Again, this array emphasises the material presence of its substrate. The tonal "generality" of the wooden mirror is functional enough to be familiar, but the coarse mechanical clattering of these pixels makes them inescapably specific.
Rozin has made many similar mirrors; notable is Trash Mirror (2001) where the individual elements - irregularly shaped pieces of rubbish - are packed into a freeform mosaic. This array moves one more step away from the homogeneous generality of the digital screen. Here the elements are irregular in size and shape, but also carry their own specific textures and colours. In Mirrors Mirror(2008) the regular grid returns, but the array elements are themselved replaced by mirrors; as these tilt they reflect different parts of the environment. Here the location of the tonal "content" in the array is, like the image source, deferred to the environment. In a familiar digital screen, image elements are luminous modules whose colour value is independent and absolute. In Rozin's Wooden Mirror that value becomes relative - tonality is based on self-shading, which depends on the lighting of the work. In Mirrors Mirror this relativity is multiplied; each element will reflect a different portion of the environment, depending on both its angle and the viewpoint of the observer.
In many cases these media art arrays depart from the two-dimensional grid entirely. Robert Henke and Christopher Bauder's ATOM (2007-8) (above) is an eight-by-eight grid of white helium balloons, each one fitted with LED illumination and tethered to a computer-controlled winch. The grid becomes a mobile, configurable light-form, tightly coupled with Henke's electronic soundtrack in live performance. This array lowers its resolution drastically, and limits its generality in one dimension (monochrome elements), but extends its reach (literally) into a third axis. ART+COM's 2008 kinetic sculpture at the BMW museum uses a similar configuration, but a higher "resolution" - in this case 714 metal spheres are suspended from motorised cables, forming a smoothly undulating matrix - a sort of programmed corporate ballet. Cloud (2008), a sculpture in Heathrow airport by London art and design firm Troika, illustrates another permutation: here a 2d array forms the skin of a large three-dimensional sculptural form. In this case the elements are electromagnetic flip-dots - components often used in airport signage before it was overtaken by glowing rectangles. As in Rozin's Mirrors, Troika consciously exploit the materiality, gestural character and the sound of these retro-pixels. rAndom International's 2010 Swarm Light demonstrates a "saturated" 3d array. The work consists of three cubic arrays of white LED lights, each ten elements per side; these cubic volumes host a flowing, flickering "swarm" of sound-responsive agents which traverse the space, brightening or dimming the array as they move.
The work of British designers United Visual Artists offers a useful longitudinal study in post-screen imaging; in particular their work addresses one of the central technical players in this field, LED lighting. UVA's first project involved a huge LED array that formed the stage set of Massive Attack's 100th Window tour. Unlike more screenful video backdrops, this low-res grid had an inescapable presence, hung directly behind the band and looming over the stage. Rather than an image machine, UVA treat the grid as a luminous dot-matrix for the twitching alphanumeric characters of real-time data. In subsequent work UVA develop this approach in a number of directions, but digitally articulated light - enabled by the LED - is a recurring theme. In Monolith (2006) UVA use a pair of large, full-colour LED screens, but treat them as a dynamic light source rather than a substrate for images; subtle gradients and washes of colour spill over the audience and into the installation environment, coupled with generated sound. In Volume (2006), another installation piece, the array elements are long vertical LED strips, again treated as generators of pattern, colour and sound; the work forms an interactive field as each element responds to nearby activity. In the context of this steady dismemberment of the screen, UVA's later work The Speed of Light is notable in that it leaves LED arrays aside entirely. Instead it uses installed lasers manipulated into dynamic, walk-in calligraphy, as if light had been finally prised away from its digital substrate, and turned loose in the environment.
Beyond their formal similarities, these arrays share some core approaches and contexts which provide a coherent portrait of a sort of post-screen practice. These works adopt one key feature of the screen - the "generality" of an articulated substrate - but trade it off to varying extents for more "specificity" - exploiting the local, particular materiality of the work and its environment. This specificity is also technological, reflecting a practice that crafts hard- and software into idiosyncratic configurations, rather than using off-the-shelf infrastructure. Light is a strong theme, in particular the solid-state, digitally addressable light of the LED (essentially a free-floating pixel). However the optical in these arrays is always tightly coupled with other modalities, especially sound, which is either a cherished byproduct of the array mechanism (as in Rozin's Mirrors and Troika's Cloud) or generated by the array elements themselves (as in the drummers and UVA's Volume). A quality of liveness is linked with the turn to specificity and being-in-the-environment; from the "live data" of UVA's Massive Attack show, to the live interaction and generation of their later installations, to the live video driving Rozin's Mirrors. Performance and temporary installation are the dominant forms here - emphasising the intensified moment, rather than the any-time of static content.
3. Projection Mapping and Extruded Light
In one sense these arrays present a disintegration of the screen - they pull its elements apart and embed them in the environment. In another strain of media arts practice, something like the converse occurs, though with what I will argue are similar interests and agendas. In this approach screen-like technologies are used intact, rather than decomposed; but their function and their relationship to the environment is transformed. These works reverse-engineer the digital image, exploiting its digital (general) malleability in order to fit it to a specific environment.
The work of Norwegian artist HC Gilje illustrates one trajectory of this second post-screen approach. Gilje's work from the late 90s was in live digital video, with his ensemble 242.pilots. This practice was linked to the burgeoning activity in experimental electronic music at the time; here again, performance, improvisation and the intensified moment - what Gilje calls an "extended now" - are central concerns, though the work is strongly screen-focused in its results . In Gilje's work over the following decade, he demonstrates another path towards the post-screen. Gilje's nodio (2005-) is a custom software system for distributing video content across collections of linked "nodes". In drifter (2006) these nodes are manifest as a ring of twelve screens which form a linked audiovisual interspace. With dense (2007) these nodes take on a more sculptural presence - hanging strips of fabric illuminated from both sides with a tailored video-projection. Here Gilje adapts the screen technology of the video projector to a sculptural environment, pushing it one step away from image and towards illumination. The work also depends on a specific material surface - the translucent weave of the fabric enables the double-sided layering of pattern.
shift (2008) (above) develops this approach: a technique known as projection mapping, in which the projected image is reverse-engineered to fit a specific surface. In shift Gilje's nodes are simple rectangular boxes, constructed from plywood. Using more custom software, the artist illuminates a cluster of these boxes with precisely mapped projected images. The coupled sound emanates from speakers housed in each box, so the objects are again audiovisual (and acoustically distinct) nodes; Gilje composes material for this environment in search of what he terms "audiovisual powerchords" - moments of intense juxtaposition and interplay. In blink (2009) Gilje dispenses with the boxes, instead treating the bare installation space. Simple, geometric elements - angular lines and bands of tone and colour - are reflected and modulated by the space itself, diffusing from irregular polished floorboards and painted walls. The work plays the room with articulated light, carefully matched to its geometry in way that heightens our awareness of the interplay of space, light and materials.
Projection mapping has recently flourished in "visualist" practice across art, design and performance contexts; trompe-l'oeil architectural facades are one popular genre, manipulating the built environment by rendering it with a tailored skin of articulated light (see for example Urbanscreen's Kubik 555). German designers Grosse 8 and Lichtfront demonstrate a logical extension of the technique, using multiple projectors to create an "augmented sculpture" in the round.
Another notable example is Scintillation (2009) (above) by Xavier Chassaing, a digital stop-motion film in which projection mapping is used to layer a domestic environment with luminous swirls of particles, igniting the petals of an orchid and tracing the curves of a moulded plaster cornice [24]. As in Gilje's blink, Scintillation emphasises the ambience of the projected light - reflections and diffusions are heightened by hand-held macro cinematography, artfully producing an impression of material texture. But in the process it raises some interesting problems for our analytical premise - a shift from the screenful image to something more live and specific. For Scintillation is absolutely a work of filmmaking; here projection mapping - the tailored materialisation of the image - is deployed as a technique for producing generalisable, substrate-independent image content.
The final example in this survey addresses the same tension. In their recent short film Making Future Magic (above), London design agency Berg give an ingenious demonstration of both the material turn of post-screen imaging, and its recuperation as image content. Berg developed an animation technique combining multiple-exposure stop-motion with a hand-held source of articulated light - specifically the glowing rectangle of the moment, Apple's iPad. 3d forms are digitally modelled and animated, then decomposed into sequences of 2d slices. These slices are then replayed into the environment, and thus recomposed into 3d forms, by moving an iPad screen over successive still frame exposures. As Berg term it, this is "extruded light" - as in UVA's latest work, it's as if light itself has been unpinned from its substrate. The results are a beguiling combination of loose, organic light painting with simple 3d geometry and DSLR imaging. As Berg frame the work, it fits entirely within the post-screen turn proposed here. Responding to a brief around "a magical version of future media", Berg are "exploring how surfaces and screens look and work in the world ... finding playful uses for the increasingly ubiquitous ‘glowing rectangles’ ...". Again the material embeddedness of this articulated light is emphasised - the way it reflects from puddles and diffuses through foliage. Screen as object in the world, rather than window to somewhere else. As in Scintillation however the inescapable irony is that the outcomes of this work are entirely bound up with screenful images - with the generalising infrastructures and distribution pipelines of social image sharing, print-on-demand and networked video.
4. Transmateriality and Presence Culture
To recap briefly: the ubiquitous digital screen is characterised by both generality - an ability to display any content at all - and self-effacing slightness - it tries to make itself disappear as a neutral substrate for content. In contrast to these tendencies this paper describes two distinct but parallel strains of "post-screen" practice in the media arts and design. Arrays mimic the grid configuration of the screen, but lower its resolution and emphasise the material presence of the array elements - their local and individual specificity is balanced with their malleable generality (their ability to carry anything-at-all). Projection mapping and "extruded light" practices also emphasise specificity, materiality and a local, performative being-in-the-world, but they do so by different means - exploiting the malleability of the digital screen (and the computational representations it hosts) in order to make it intensely site-specific. To the extent that they both adapt and resist the attributes of our familiar glowing rectangles, we could describe these practices as post-screen, but this "post" is nothing like a conscious critique, let alone a revolutionary break. However hard they may pull towards specificity and local materiality, they are readily - by design or necessity - recaptured as screen fodder.
Both these post-screen tendencies and their screenful recuperation can be usefully framed through the notion of transmateriality, a concept that attempts to capture a fundamental duality in digital (and other) media: they are everywhere and always material, yet often function as if they are immaterial. In a transmaterial view media always operate as local material instances (this is their aspect of specificity) yet retain the ability to hold specificity at bay - resisting the contingencies of flux - to create a functional generalisation in which this pixel is the same as that one, the email I send is the same as the one you receive, and one node on the network is much the same as any other.
In the glowing rectangle paradigm functional generality is entirely dominant. The work considered here, on the other hand, revels more in the pleasures and practices of specificity - the clatter of servo-actuated wood or the play of light on this particular wall. In their push towards liveness (of interaction or data), performativity, their integration of sound, and their emphasis on evanescent materiality, these works evoke what Hans Ulrich Gumbrecht would call "presence culture" - that mode of apprehending the world which is characterised by fleeting but intense moments of being, and a sense of being part of the world of things, rather than outside it, looking in. Gumbrecht constructs presence in opposition to a dominant "meaning culture", in which the essence of material things can be obtained only through interpretation. Gumbrecht describes the relationship between these poles as one of dynamic oscillation. "Presence phenomena" become "effects of" presence, "because we can only encounter them within a culture that is predominantly a meaning culture. ... [T]hey are necessarily surrounded by, wrapped into, and perhaps even mediated by clouds and cushions of meaning".
In exactly the same way we find an inevitable oscillation here between screen and post-screen. We can align the screen with generality and meaning culture, and the post-screen with specificity and presence culture; but here too the post-screen is evanescent and elusive, instead existing largely within the dominant screen culture. However this is not to discount the utopian aspirations of a post-screen practice, which might instead be located through the perspective of transmateriality. For in echoing the screen, or in literally bending it to the local, present and specific, these works operate as reminders of the ubiquitous and everyday materiality of our media, of the fact that depite appearances, every glowing rectangle is already local and specific. If that specificity is latent, then these works demonstrate practical strategies for making it explicit; from hardware hacking to modular LEDs and custom software, they participate in what might be called "expanded computing", using the malleability of digital media to reactivate its presence - and thus our presence, too - in the world of things.
