The navigational gesture: Traces and tracings at the mobile touchscreen interface

by Nanna Verhoeff & Heidi Rae Cooley

While touchscreen technology has been around since at least 1974 it did not become widely available on the consumer market until the 2007 release of the Apple iPhone.[1] Since then our handheld devices have gotten smarter and more connected. We enjoy an ever-increasing ability to access and manipulate data of all sorts at the touch, swipe, or drag of a finger. Not surprisingly, scholars worldwide have turned their attention to the mobile device, related social media practices, and the contexts in which such practices unfold. To date few have addressed the touchscreen interface itself, especially as it participates in and frames semiotic – that is, meaning-making – practices.[2] The importance of considering the interface becomes evident once we situate the mobile micro-screen within the visual regime of navigation, which arguably epitomises the mobile present. As both a material and performative navigational interface the mobile touchscreen confronts us with a constitutive paradox involving the status of the indexical trace.[3]

Photographic and film-based media have long been valued for their indexical properties (light-sensitive chemicals that inscribe a material substrate with a referent’s imprint). Under these circumstances indexicality promises evidentiary stability. Discussions of indexical veracity frequently focus on the legitimacy of the image as evidence of some pro-filmic moment, an authentic and verifiable that-has-been, à la Roland Barthes, claimed to be visible in the image – the imprint that offers proof of a past presence. Film and media studies scholars have had much to say about indexicality in the context of new digital technologies. Skepticism predominates; many have held the indexical guarantee to be undermined because of possible digital manipulation. Computer-generated imagery (CGI) provides the popular imagination with an example.[4]

However, indexicality, if we heed late-19th century pragmatist Charles Sanders Peirce, need not be a material trace. As film and media historian and theorist Mary Ann Doane has incisively reminded us, indexicality might very well transpire as deixis.[5] Indexicality of the deictic variety requires a distinct and immediate there-there. Imagine for example an assertive vocalisation of the imperative ‘look there!’ accompanied by a raised hand pointing a finger.[6] Here we see that deixis involves a directness of association in the time and place of its articulation; one needs to be near enough to the utterer and her gesturing in order for the event to mean anything whatsoever. In other words the deictic situation is context-dependent – situationally, temporally, and inherently social in character. This means that neither the gesturing subject nor the interpreting subject is a Cartesian subject, separate from or outside of signification. Rather, deixis assumes a community of interpreters who would – as a matter of common-sense – understand the relation between the declaration and its directional posture, whether or not the anticipated response (e.g. turning to look) actually happens.

Deixis emphasises the processual and, as we will argue, layered nature of the interpretative or meaning-making moment constitutive of what we refer to as the navigational gesture. To our minds deixis is more useful for understanding the particularities of mobile media practices involving the touchscreen interface. The gesture that transpires in real time at the mobile interface is a deictic ‘doing’ that produces a fleeting trace in the form of activating a click or moving a map view. An on-going directional process, the deictic trace occurs in the moment of its happening; what by convention would be considered the resulting temporal sequence (e.g. beginning, middle, conclusion) collapses into a single but dynamic moment of coincidence – a passing present converges with a future-oriented forward-looking.[7] Active meaning-making transpires at the touchscreen interface; this is what distinguishes the navigational gesture from other kinds of traces such as smudges and fingerprints that may also indicate a gesture. Without positioning the device and touching its screen data cannot be accessed and interacted with, at which point semiotic activity fails.

The directional doing that defines the navigational gesture materialises through the articulation of a user, her screen, a location, and a destination. Navigation applications such as Google Maps and related ‘on-dash’ vehicle navigation systems such as Garmin or Waze map a user according to a point of origin, current location, and a designated site of arrival.[8] The user’s location, canonically represented on screen as an ever-advancing pulsing (blue) bead, shifts cartographic orientation in near-real time according to frequently updated coordinates as provided by GPS. In the moment of navigation user, technology, and connectivity occupy a hybrid temporality, existing in the present in relation to a recent past location according to a future-oriented traversing. With navigation activated one is implicitly always-already heading toward some site of arrival, be it explicitly designated or merely potential, by chance or accident. In this regard we do not distinguish between ‘on dash’ and ‘in-hand’ navigational interfaces. Both cases present a situation wherein the locative device continuously registers its user as she moves between a ‘here’ (i.e. in situ) and a ‘there’ in the context of what appears (‘refreshes’) on screen. These three registrations of tense which are never discrete temporal unities (past, present, future) comprise the layers that manifest indexicality as processual – that is, as deixis.[9]

However brief, the navigational gesture manifests – in use, process, and duration – as a layering. This layering is not an accretion of separate hierarchical or historically-accumulated strata; it is a thickening. In what follows we posit the navigational gesture as a cultural phenomenon, a cultural form. We argue that the gesture itself is the trace in a future-oriented sense, that is, it is a tracing – deixis. Consequently, we address deixis as relevant to (1) conceptualising the ontology of the navigational gesture, specifically its layered temporality as produced by the mobile subject in the context of her surroundings; and (2) understanding the epistemological shift that touch, mobility and navigation, and temporal layering engender. For illustration we turn to a consideration of the navigational gesture in the context of a recent public interface-based artwork. Titled Saving Face (2012), the installation serves as a theoretical object that allows us to interrogate the status of the mobile subject who functions according to a networked dispositif. We focus on how individual touch and gesture perform a tracing that is always socially informed. In this regard we reframe the navigational gesture as a pragmatic and performative – that is, deictic – supplement to symbolic and iconographic representation. We conclude by suggesting that layering, as the principal experience of navigation, emerges as a governing trope in the mobile present.

Gesture

The types of indices discerned in 20th century film theory obviously persist. We who carry mobile screens contend with the mundane fact of oily residue smudging our touchscreens. We continue to leave footprints behind when walking around with our devices in hand. (We must also acknowledge that our various technologies proliferate streams of data that can be tracked and searched.) This is indexicality in the most familiar sense: a trace of our body and its signatures. Interactions with a touchscreen reveal the materiality of the trace as imprint to be secondary to the materiality of presence and bodily engagement. In this regard we see touch-based interaction as always potentially participating in a future-oriented and processual navigational gesture that is deictic. Involving bodily posture, positioning, and movement with relation to the mobile touchscreen, indexicality becomes one of deixis – even as deixis might very well proliferate various material traces (e.g. location coordinates). In the gestures we find that the mobile touchscreen interface affords a new relevance for older questions about the nature of indexicality and how it operates as a process of meaning-making or semiosis. We are interested in how, when, and where indexicality manifests and under what temporal and directional conditions.[10]

While we foreground the tactile engagement that characterises the touchscreen interface we contend that the deictic gesture extends to wearable technologies that are being developed at the moment of writing. For example, new products forecast the probable disappearance of the touchscreen interface as such. Google Glass and other forms of wearable technologies as well as gestural interfaces based on motion tracking and related modes of input will necessarily alter current handheld forms of indexing, when the index (i.e. the material trace) seems to become relocated from the surface of the screen and transpires as a layered and processual repositioning of the body. Gesture opens onto a haptics that unfolds in time at the interface. Not the trace left on the device but the gesture itself becomes the trace – or better, a tracing.

Despite its fugitive status and limited duration we regard the tracing that gesture performs as temporal rather than a discrete instance of a trace left over or left behind. The deictic materiality, coterminous with the time of the gesture, comes into play particularly when we work with location-based applications. In the case of GPS-oriented applications for mobile devices the first prerequisite is, of course, being physically present in a particular location where one’s position as determined according to GPS coordinates activates access to data. It is precisely the touch-based and gestural activity such as positioning, pointing, and framing with the device that makes meaning happen, that culminates in the semiotic event.[11]

In asserting that the gesture of navigation involves a future-oriented, deictic posture, we emphasise that such posture is haptic. Haptic (from the Greek aptô [‘touching’]) is characterised by three primary features which are inter-related and mutually-informing. As Laura Marks explains in The Skin of the Film a ‘fastening’ of perception across tactile, kinesthetic, and proprioceptive registers of experience.[12] For our purposes we propose that the haptic is best understood in terms of Peirce’s ontological categories of firstness (immediacy), secondness (causal, spatial, or temporal relation), and thirdness (cognition).[13] We are particularly interested here in revisiting Peirce because he does not privilege a phenomenological approach to subjective experience nor does he endorse the notion of a self-contained Cartesian subject who thinks the world separate from herself – as well as ‘for’ herself from her perspective. We emphasise embodiment (in the sense of cognition) in order to understand movement-based tactility and gestural interfacing as an integrated haptic process according to Peirce’s three modes of being.[14] The here-and-now of an individual’s experience (secondness) is the site of intersection of spontaneity or chance (firstness) and intelligibility (thirdness). In such moments a person’s past and present self undergoes a repositioning, one that bears the mark of intelligibility’s encounter with and response to spontaneity or chance. The gesture, here seen as a tracing, now comes to stand for embodiment, but with a difference. It implies time, specifically multidirectional temporality encompassing past and future in an ever-fleeting present. The haptics of the navigational gesture as layered; a layered indexicality of use, of process – in duration.

Also, a Peircean account of subjectivity requires us to understand the social nature of subjectivity. Since subjectivity transpires as a meaning-making process (i.e. cognition) it necessarily exists in relation to a larger community of interpreters – users, participants, interactants – who share certain manners of thinking. For Peirce such manners of thinking include tendencies or habits of behaving, including the inclination to carry in hand, raise, direct, and interact with a touchscreen device.These seemingly natural or quotidian actions – actions that are normal and intelligible in the context of the touchscreen interface – demonstrate a common or shared way of thinking. This common way of thinking allows us to imagine a ‘thickness’ or layered experience that belongs to a subject who is always-already social, that is, part of a broader community. At the same time it allows us to acknowledge the fact that such thinking engenders practices that produce traces (e.g. location coordinates) that make possible social order (e.g. by means of tracking, ‘following’, etc.). We will emphasise this dual point explicitly in our discussion of Saving Face.

Deixis

By focusing on the navigational gesture we aim to underscore the situatedness of the touchscreen interface – in a here-and-now, in a particular location, in relation to a subject, with a duration and a futurity as implied by the directional order of a now quotidian mobile posture. Here we take up Lorenz Engell’s suggestion in a recent issue of NECSUS that turns to the television remote control device (RCD) and desktop mouse. We do so in order to contextualise our theorisation of the navigational gesture in a broader discourse of technologies and practice.

Engell’s central claim is that the RCD prepared the way for the mouse click, and by implication other gestures as facilitated by the touchscreen interface. The computer is the organising technology, one that overshadows other technologies that make possible handheld connectivity, such as telephony and the Internet. Not surprisingly, what the RCD and desktop mouse share are the gestures of pointing and clicking. Engell foregrounds the index finger (whose flexion depresses the mouse’s primary button); this serves as the point of departure for thinking about the index as the ‘bodily residence’ for will or intention.[15] The mouse click is an externalisation or materialisation of the will. But here will is not the will of Cartesian imposition. Rather, it is in keeping with Peirce’s notion of cognition, particularly his theorisation of the interpretant which, as we have explained, is social. Because it belongs to a history that includes the RCD the computer becomes for Engell an ‘indexical machine’ – that is, ‘a technology of … direction’. Such technologies of direction – of intention, intensity, and agency – activate the intermediacy of touch.[16]

Deploying Peirce’s triadic theory of the sign, Engell offers his own (analogous) triadic interpretation of the RCD. Insofar as the user points the RCD at the television set in order to navigate channels and corresponding content (i.e. images) she brings into relation indexical, symbolic, and iconic registers respectively. Likewise, Engell finds a useful correlation between Peirce’s ontological categories of firstness, secondness, and thirdness and the RCD as a mediating technology.[17] As Engell explains, the very haptic context of the RCD in hand produces firstness; the causality of pointing the RCD and pressing a chosen button, and this compound action’s effecting a change of channel, is of the order of secondness. Thirdness is the domain of knowing and habit, of communication, of symbolic exchange; tuning into a particular channel at a designated time delivers a specified content (i.e. a program) to the television screen. That we depress a number-embossed button on a remote control device in order to change the audio-visual content on screen is a matter of a social and shared generalisation. We have learned to navigate televisual programming by this means and as such we enact a process that belongs to thirdness.

Like Engell we find Peirce’s categories – or ‘three modes of reality’[18] – productive for thinking about the character of the navigational gesture as deixis. We emphasise the durational, layered quality of the relation between the physical use of the mobile touchscreen (and related touch-based interfaces) and the haptic experience that such exchanges bring about. For us the touchscreen interface opens onto a consideration of technology-oriented practices that are not simply tactile but also haptic. Such practices operate at the threshold between site-specific data overlays (iconic/symbolic) that appear onscreen (iconic/symbolic) and one’s fingers that touch, swipe, and pinch the physical screen in order to access information (iconic/symbolic) about one’s surroundings. The entire process that transpires at the interface of the touchscreen involves – as does Engell’s RCD – firstness, secondness, and thirdness.

However, the two sides of Peirce’s double notion of indexicality – the indexical trace, which is past-oriented, and deixis, which is, in the case of navigation, primarily future-oriented because it foregrounds a current position with respect to some destination – raises a question about the index as marker of presence, be it past or present. We witness a shift in our cultural expectations from a conception of the individual on the grounds of a (bodily) presence ingrained in history and tradition and held responsible for past actions[19] toward an ideal of mobility, participation, and individual agency that inhabits a socially-negotiated sphere of interpretation and practice. This also implies a heightened concern about the tracking of presence and orientation towards a future or destination. However, the ideal comes with a concern for privacy in a time of hyper visibility and ubiquitous surveillance in the face of digital ephemerality and an overabundance of images. The fingerprint – the hallmark of the index as trace – signifies presence; indeed it constitutes proof of presence but not the quality, for example, the duration of that presence. From fingerprint and the index finger as the traditionally quintessential trace – that is, as markers of material presence and subjective agency and therefore culpability – we move to gesture as, more generally, part of a category of the index of a techno-practice that entails an intertwinement of technology and subjectivity. The gesture of the navigator which involves bodily orientation and positioning as well as tactile interactions with the touchscreen is the tracing of her presence whether or not a physical trace remains.

From a consideration of deixis as the gesture of pointing (‘there’ or ‘here’) we discern two forms of deictic transaction: the gesture of raising and pointing a mobile device, for example in the case of the augmented reality applications, and the finger’s pressing on the touchscreen that activates data overlays or other forms of web-based content that recognise one’s location and surroundings. Both these transactions require pointing and touching. The former is future-oriented, pointing toward some destination; it is also fixed in the present in the sense that unlike the trace it ‘[evaporates] in the very moment of its production’.[20] This processuality is the hallmark of the mobile present; it is also foundational for a layered sense of presence.

Trac[k]ing

The gesture of raising the touchscreen device and directing it toward some object or location characterises the navigational gesture of the mobile subject. Forward-looking and destination-oriented, the device in hand functions as a fulcrum, a threshold between which past traces (for example GPS coordinates) register ‘where-I-have-been’ in relation to potential routes traced by navigational software that suggest ‘how-I-might-arrive-there’. This is the site, or better, the time of what we call processual indexicality. In specifying processual indexicality as temporal rather than spatial we emphasise that even as place and location matter it is how indexicality transpires and is experienced in an unfolding present. In other words navigational indexicality takes place in the real time of gesture and the subsequent assumption of a directional posture. However, the navigational is hard to catch in traces. Certainly residual streaks on screens signify presence – that is, offer proof of presence. One’s coordinates track where one is with respect to where one has been (and where one lives if the ‘home’ function is activated). These traces reveal nothing of the quality of that presence. Even as smudges accrete on the surface of the touchscreen as a result of touching, swiping, and pinching, these traces offer no evidence of the time (duration) of the gesture or posture that defines navigation; neither do the digital tracks we stream behind us as we navigate our surroundings.

Fig. 1: Saving Face installation. Image: Lancel/Maat, 2012.

Fig. 2: Saving Face installation. Image: Ruthe Zuntz, 2013.

For the purposes of discussion let us consider the interactive public installation Saving Face (2012) by artists Karen Lancel and Hermen Maat. The multi-screen, site-specific installation comprises a large, public, urban billboard-sized screen and an application with facial recognition software for a smaller screen, either hand-held or housed in a kiosk. The project works according to the principles of touch and haptic, gestural looking; it invites the participant to touch and trace her face and thereby ‘paint’ herself on the screen in front of her.[21] As participants engage in touching and gesturing images of their own faces evolve, transforming into a composite of the larger community of participants past and present. The transformation of individual faces into a community composite elicits additional animated pointing and gesturing by those on site. In response, many raise their personal touchscreen devices to record the event.

The project is a theoretical object for us which provides an opportunity to consider the navigational gesture in a delimited but public context. Here, gesture and the touchscreen interface open onto documentable (and documented) physical performances of the haptic. Visible in these personal performances of interaction is a demonstration of the layered-ness of indexicality. In this regard we are interested in how the installation draws attention to the subtle and overlooked ways in which our screens not just foreground but, in fact, elicit touch and gesture as an integrated pragmatic performativity. Moreover, the project suggests that any single individual qua personal gesture belongs to a larger group of participants insofar as participant-contributors exhibit similar – that is, shared – behaviors of response.

In spite of the difference between a performative multi-screen installation such as Saving Face and actual movements through outside space we argue that the two have in common the dispositif of navigation which manifests as a gestural practice. The project provides an opportunity to parse the navigational as a cultural form, one that brings about a change in the socio-historical logic of indexicality. Here, navigation as ‘finding your way’ no longer suffices. Rather, the navigational proceeds as a gestural practice involving bodies and screens (i.e. information) and locations. By means of this example we underscore that the gesture entails the articulation of orientation, positioning, framing, and (subsequent) mobility. As a public artwork Saving Face draws attention to the movements of people, the transfer of data across media platforms, and the activity and gestures – that is, meaning-making – elicited by a variety of screen interfaces that exist both in physical constellation and in networked connectivity.

Itself mobile and mutable, Saving Face puts touch-based technology and the relation between the hand and (multiple) screens at center stage. It is an ‘interface’ par excellence – it functions as a technological arrangement (of screens) that solicits and incorporates touch and directional gestures of users who activate and thereby participate in (and are registered by) its display.[22] It is navigational insofar as it requires the user to orient, frame, position, and move (here: touch) herself.[23] For us this work exemplifies how the trace as a product of the touchscreen interface is not something that is left behind but is a present – indeed, an ongoing presencing – connoted by the verbal construction tracing. It comments on the ephemerality of the navigational gesture as characteristic of touch-based interfaces.

At the same time, the installation draws attention to what for many of us goes without question: the experience of navigation, its gestural properties, and the potential for seeing ourselves, each other, and the sites we traverse differently. Here touch is an act of tracing, one that displays for a public body a shared image that in its iterative refreshing tracks that public. On one hand the artwork reminds its participants that they are seen, that to be in public is to be visible; on the other hand it endeavors to intervene in how visibility operates, how visibility – the public face – signifies.[24] The gesture of touching one’s face in order to visualise one’s self in relation to others points to the processual character of navigational gesture in the context of location-aware technologies. In this way it harkens back to a long history in which photography (art) and policing (governance) are mutually informing. The artists themselves acknowledge this connection:

[i]n a visual, poetic way ‘Saving Face’ shows our emotional and social encounter with trust, visibility, privacy in our ‘smart’ cities. When defining our identity and the identity of others, our sensory abilities are increasingly replaced by networked surveillance and identification technologies. How do we experience the way our body and identity are being ‘measured’ as functional and controllable products? Can touch based perception play again a role in experiencing the other’s identity? ‘Saving Face’ reintroduces touch related perception in the digital and public domain – with the help of a personal touching body scan.[25]

As the authors indicate, Saving Face counters the abstraction we frequently encounter in public places. It gives significance to an activity – navigation and its gesture – that is routine, everyday, and presumably inconsequential. By returning the ‘face’ to interface the project raises questions about presence, subjectivity, visibility, and the anonymity often attributed to being in public. The work is highly personal yet simultaneously public and collective; it negotiates the private intimacy of auto-touch – a gestural ‘selfie’ – and a highly public yet temporary and collaborative visibility on screen.

The collage of different faces displayed on screen is a tracing (as well as a tracking) of multiple actions by multiple participants accumulating and metamorphosing across multiple moments. A composite image, it speaks symbolically to the multiplicity of subjectivity and to the temporal layering of various individual presences. The processuality of the navigational gesture does leave a trace – an iconic image of individual faces, albeit fractured and reassembled into a new whole; it says ‘we were here’. The image testifies to their gestures, the image’s evolution inviting further interaction and gesturing. At the same time each updating of the visualisation keeps record of previous traces. The installation bears witness to and renders visible the processual layering that is the semiotic process of the navigational gesture – a trace of the act of tracing.

Layers

Layers and layering are oft-used metaphors for ‘mixed reality’[26] situations in which  information and/or image augments ‘overlay’ real-world experience, usually by means of a mobile micro-screen that is in-hand, connected, and locatable. At the touchscreen interface one can access site-specific data including local histories and other cultural content (e.g. current events and happenings), nearby dining and shopping options, as well as possibilities for locative-gaming or related location-based social-networked experiences. In these moments the landscape – or cityscape, to borrow a term from de Souza e Silva and Sutko – acquires a textured mise en abyme quality as one situated in her surroundings raises her device and observes on screen the real time re-presentation of what stands before her. Appearing in the mobile framing atop a continuously refreshed image is a host of images, arrows, and other notations that indicate points of interest and/or various annotational bubbles and browser windows that open onto more information.

These on-screen layers point to another layering, a layering of a gestural sort – a layering that materialises as a multidimensional process, an embodied thickening. This multidimensionality hinges on the different ways that temporality – in particular, duration – plays a role in signification as it transpires at the site and in the moment of the mobile touchscreen interface. Here navigation brings a fourth dimension to representation, if you like – not just in the form of timestamps encoded to pictured or otherwise recorded actions but more specifically in an unfolding gestural posture. This is what is at stake in the concept of performative cartography:[27] a tension between a more fixating logic of representation (i.e. the stasis of a map’s aerial view) and a more fluid understanding of the processuality and performativity that epitomise navigational practices (i.e. the in-between-ness – duration – of navigating). Here, we are both on-the-map and performing its possibilities, expected and otherwise.

Our contribution to this issue is a reflection on the status of indexicality in the mobile present – a culture of mobility and a visual regime of navigation. It touches on the question of traces, tracings, and tracking in the mobile present, a present that we, the authors, understand to be ‘mixed’ in its modes of and accesses to realities of various orders. We invite our readers to recognise that the navigational gesture is at the heart of what it means to be both on site with and at the site of the mobile touchscreen interface – which means that when we raise our touchscreens we enact the gesture of navigation and all that entails.

Acknowledgements

The research leading to these results has received funding from the European Research Council under the European Community’s Seventh Framework Programme (FP7/2007-2013) / ERC Grant agreement n° 283464.

Authors

Nanna Verhoeff is Associate Professor of Comparative Film and Media Studies at the Department of Media and Culture Studies, Utrecht University. She has published on media in transition, ranging from early cinema and panoramic paintings to contemporary mobile screens and urban screens. Her books include The West in Early Cinema: After the Beginning (Amsterdam University Press, 2006), on (early) cinema as a new medium. Her latest book, Mobile Screens: The Visual Regime of Navigation (Amsterdam University Press, 2012), is on mobile media and interfaces of navigation. Verhoeff is currently a senior researcher on the project Charting the Digital: Digital Mapping Practices as New Media Cultures funded by the European Research Council (ERC). She is working on a book on layers and layering as a trope in modern visual culture.

Heidi Rae Cooley is Assistant Professor of Media Arts in the Department of Art and Film and Media Studies Program at the University of South Carolina. Cooley’s scholarship considers the inter-relations among technology, sociality, and living bodies. Her monograph Finding Augusta: Habits of Mobility and Governance in the Digital Era (Dartmouth College Press, 2014) argues that because digital technologies set places, persons, things, and information in constant motion, habits of locatability and navigation assume decisive socio-political importance. Augusta App, the book’s digital supplement, functions as a laboratory for exploring this argument (www.findaugusta.com).

References

Barthes, R. Camera lucida: Reflections on photography. New York: Hill and Wang, 1981.

Bellis, M. ‘Who Invented Touchscreen Technology?’, About.com, 13 April 2014: http://inventors.about.com/od/tstartinventions/a/Touch-Screen.htm.

Benveniste, É. Problems in general linguistics. Coral Gables: University of Miami Press, 1971.

Bergson, H. Matter and memory, translated by N. M. Paul and W. S. Palmer. London: George Allen and Unwin, 1911 (orig. in 1896).

Bittner, C., Glasze, G., and Turk, C. ‘Tracing Contingencies: Analyzing the Political in Assemblages of Web 2.0 Cartographies’, GeoJournal, 2013.

Brown, M. Picturing time: The work of Étienne-Jules Marey (1830-1904). Chicago: Chicago University Press, 1992.

Castells, M., Fernandez-Ardevol, M., Qui, J. L., and Sey, A (eds). Mobile communication and society: A global perspective. Cambridge: The MIT Press, 2007.

Cooley, H. R. and Verhoeff, N. ‘Performativity/Expressivity: The Mobile Micro Screen and Its Subject’ in Technē/Technology: Researching cinema and media technologies – their development, use, and impact, edited by A. van den Oever. Amsterdam: Amsterdam University Press, 2014: 207-216 (available in open access at www.oapen.org).

Cooley, H. R. Finding Augusta: Habits of mobility and governance in the digital era. Hanover: Dartmouth College Press, 2014.

de Souza e Silva, A. and Sutko, D (eds). Digital cityscapes: Merging digital and urban playspaces. New York: Peter Lang, 2009.

de Souza e Silva, A. and Frith, J (eds). Mobile interfaces in public spaces: Locational privacy, control, and urban sociality. New York: Routledge, 2012.

Deleuze, G. and Guattari, F. A thousand plateaus: Capitalism and schizophrenia, translated by B. Massumi. London: Athlone Press, 1987.

Doane, M. A. ‘The Indexical and the Concept of Medium Specificity’, Differences: A Journal of Feminist Cultural Studies, 18, 1, 2007: 128-152.

Dunne, A. Hertzian tales: Electronic products, aesthetic experience, and aesthetic design. Cambridge: The MIT Press, 2005.

Ekman, U (ed.). Throughout: Art and culture emerging with ubiquitous computing. Cambridge: The MIT Press, 2013.

Engell, L. ‘The tactile and the index: From the remote control to the hand-held computer, some speculative reflections on the bodies of the will’, NECSUS, Autumn 2013: http://www.necsus-ejms.org/the-tactile-and-the-index-from-the-remote-control-to-the-hand-held-computer-some-speculative-reflections-on-the-bodies-of-the-will/.

Farman, J. Mobile interface theory: Embodied space and locative media. New York: Routledge, 2012.

Friedberg, A. ‘The End of Cinema: Multi-media and Technological Change’ in Reinventing film studies, edited by C. Gledhill and L. Williams. London: Arnold, 2000: 438-452.

Galloway, A. The interface effect. Cambridge: Polity Press, 2012.

Ginzburg, C. ‘Clues: Morelli, Freud, and Sherlock Holmes’ in The sign of three: Dupin, Holmes, Peirce, edited by U. Eco and T. Sebeok. Bloomington: Indiana University Press, 1983: 81-118.

Gordon, E. and de Souza e Silva, A (eds). NetLocality: Why location matters in a networked world. West Sussex: John Wiley and Sons, Ltd., 2011.

Hjorth, L., Burgess, J., and Richardson, I (eds). Studying mobile media: Cultural technologies, mobile communication, and the iPhone. London-New York: Routledge, 2012.

Ito, M., Okabe, D., and Matsuda, M (eds). Personal, portable, pedestrian: Mobile phones in Japanese life. Cambridge: The MIT Press, 2005.

Kaerlein, T. ‘Aporias of the touchscreen: On the promises and perils of a ubiquitous technology’, NECSUS, Autumn 2012: http://www.necsus-ejms.org/aporias-of-the-touchscreen-on-the-promises-and-perils-of-a-ubiquitous-technology/.

Lammes, S. ‘Destabilizing Playgrounds: Cartographical Interfaces, Mutability, Risk and Play’ in Playful subversion of technoculture, edited by D. Cermak-Sassenrath, C. Tien Tan, and C. Walker. Heidelberg: Springer (forthcoming in 2014).

Levinson, S. ‘Deixis’ in The handbook of pragmatics, edited by L. Horn. Oxford: Blackwell, 2004: 97-121.

Marks, L. The skin of the film: Intercultural cinema, embodiment, and the senses. Durham: Duke University Press, 2000.

Peirce, C. S. Collected papers of Charles Sanders Peirce. Volume V. Pragmatism and pragmaticism, edited by C. Hartshorne and P. Weiss. Cambridge: The Belknap Press of Harvard University Press, 1974 (orig. in 1934).

_____. Peirce on signs: Writings on semiotic by Charles Sanders Peirce, edited by J. Hoopes. Chapel Hill: University of North Carolina Press, 1991.

Perez-Teran Mayorga, R. M. ‘Charles Peirce’ in From realism to ‘Realicism’: The metaphysics of Charles Sanders Peirce. Lanham: Lexington Books, 2007: 69-153.

Rosen, P. ‘Old and New: Image, Indexicality, and Historicity in the Digital Era’ in Change mummified: Cinema, historicity, theory. Durham: Duke University Press, 2001: 301-349.

Sekula, A. ‘The Body and the Archive’, October, 39, 1986: 3-64.

Snickars, P. and Vondereau, P (eds). Moving data: The iPhone and the future of media. New York: Columbia University Press, 2012.

Star, S. and Griesemer, J. ‘Institutional Ecology, “Translations” and Boundary Objects: Amateurs and Professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39’, Social Studies of Science, 19, 3, 1989: 387-420.

Tagg, J. The burden of representation: Essays on photographies andhistories. Minneapolis: The Minnesota Press, 1993.

Verhoeff, N. Mobile screens: The visual regime of navigation. Amsterdam: Amsterdam University Press, 2012 (available in open access at www.oapen.org).

Waal, C. de. Peirce: A guide for the perplexed. London: Bloomsbury, 2013.



[1] According to Mary Bellis, staff writer at About.com, historians of technology attribute the invention of the capacitive touchscreen to E. A. Johnson at the Royal Radar Establishment (UK) circa 1965. They acknowledge Dr. Sam Hurst, University of Kentucky instructor and founder of Elographics, as having developed the ‘touch sensor’ circa 1971. But it is not until 1974 that the ‘first true touch screen incorporating a transparent surface came on the scene’, developed by Sam Hurst and Elographics. By 1977 Elographics had developed and patented what has become the ‘most popular touch screen technology in use today’. Worth noting here is the centrality of Western nations to the development of the touchscreen interface. See Bellis 2014.

[2] Farman 2012 and Verhoeff 2012 offer ‘early’ examples of scholarship that places the mobile interface at the center of analysis. More recently, Cooley 2014 examines the mobile micro-screen. Two other recent collections featured a focus on the iPhone as a theoretical object, bringing together different approaches to the device: Hjorth & Burgess & Richardson 2012; Snickars & Vondereau 2012. Adriana de Souza e Silva has co-edited no fewer than three collections of essays that address location-based technologies and physical location. Earlier examples of scholarship invested in mobile technologies include Castells & Fernandex-Ardevol & Linchuan Qiu & Sey 2007 and Ito & Okabe & Matsuda 2005. Timo Kaerlein published an article in NECSUS in 2012 that analyses the touchscreen interface.

[3] This essay builds on the arguments that we have developed in Cooley 2014 and Verhoeff 2012. The issue of performativity and findability, a first exploration of the synergy between our respective work on these questions, has been published recently in van den Oever 2014.

[4] For an early but salient discussion of the impact of digital technology on notions of referentiality and its reliability see Rosen 2001. In particular, Rosen addresses how the digital informs assumptions about historiography (the writing of history), history (the past), and historicity as such (the relation between the mode of historiography and the resulting historical account).

[5] Peirce wrote about indexicality over the course of a number of years and across a number of discursive registers (e.g. academic and scientific articles, speeches and lectures, edited collections, monographs). While one might sift through the eight volumes comprising The Collected Papers of Charles Sanders Peirce we suggest Peirce on Signs for the reader new to the subject. On the linguistic background to the concept of deixis see Benveniste 1971. For an overview of the ins and outs of deixis see Levinson 2004.

[6] Precisely because the linguistic theory of deixis stipulates that such signs need contextual information in order to function, the translation of deixis to visual media is possible since images in principle incorporate their own context.

[7] This collapse may remind the reader of the dissection of movement into discrete moments in Étienne-Jules Marey’s famous motion studies. Here, too, the flow and duration of movement is halted in favor of the fixing of traces in intermittent moments. For more on Marey’s project see Brown 1992.

[8] These are examples of cartographic interfaces of what is also called the ‘geoweb’, which has brought about so-called ‘processual approaches’ within critical cartography and media geography that imagines ‘cartography as assemblages of practices, technologies, norms, (re)-presentations and so forth, which are dynamic, always “in the making” and producing new ways of being in the world’ (Bittner & Glaszer & Turk 2013, np).

[9] In this regard we are interested in troubling the privileged notion of Euclidean space which abstracts the world according to a Cartesian coordinate system of points (x, y, z). Topological in emphasis, Euclidean space figures most recognisably for us in our representations of motion: as change in position over time. Our GPS systems map us according to this view of the world. One alternative to Euclidean space is Hertzian space. Hertzian space extends our view of human-object relations by taking into consideration the electro-magnetic waves emitted by our technologies, including radios, televisions, touchscreen devices, etc. As professor of design Anthony Dunne has explained, a Hertzian perspective invites an awareness of the ‘electrosphere’ that we inhabit, especially in our design of new technologies. Such an approach does offer a means toward critical understanding of the mobile present by foregrounding the fact that our devices have a ‘reach’ that we may not be able to see. However, we are interested in the very particular moment that is thickness of experience and meaning-making that happens at the site of the touchscreen interface. See Dunne 2005.

[10] Needless to say, when we use the pronoun ‘we’ in this generalising sense we are aware of the limitations. The devices we discuss are specific to age, class, and geographical regions (in the spirit of disclosure, both authors have iPhones).

[11] Besides signification in a more general sense, ‘meaning’ here refers to any number of movements that indicate that a person is/has engaged in a cognitive act, i.e. thought. In Peircean terms we are dealing with the process of mental association (thirdness) that produces a sensible outcome or behavior – what he refers to as the interpretant. Peirce offers the example of a command as delivered by a military officer. A soldier’s proper response – obeying the order – is evidence of the event of meaning-making. In the case of navigation we might consider the responsive relation between a person and the pulsing orb that registers her position on her navigational interface; in the moment she redirects herself in order to pursue a suggested route she has acted in accordance with the software’s proposed directions. See Peirce 1974 (orig. in 1934), pp. 5.283-284, 5.473-475.

[12] Marks 2000, p. 162. In the context of film and media studies and visual art the haptic implies a proximity that is fundamentally formless; consequently, lines of intelligibility, as per the convention of perspectival logic, change their function, or in navigational terms their orientation. Importantly, the haptic does not presuppose physical contact, i.e. touch as such. For Marks haptic perception ‘discerns texture’ (p. 162), it does not perceive according to distance (i.e. mathematical perspective). Drawing on Aloïs Riegl, the 19th century art historian and theorist of the haptic, Marks defines haptic perception in opposition to optical perception. It involves caressing an image with the eyes, experiencing the visual by means of the body. Of course, this idea has been taken up by Deleuze & Guattari (1987, p. 414) and following primarily in relation to impressionism.

[13] For a cogent explanation of Peirce’s categories see Perez-Teran Mayorga 2007.

[14] While ‘phenomenology’ appears throughout his work, Peirce eventually rejects the term in favor of what he calls ‘phaneroscopy’: phaneron (what is visible or manifest) and scopein (to view). Phaneroscopy, as Cornelis de Waal explains, aims to ‘draw up a catalog of the categories’, that is, an inventory of ingredients which are the most basic elements of cognition (de Waal 2013, p. 37). It does not seek any truth as such but rather the seemingness of things. It dispenses with representationality and purposefulness; the phaneron neither represents nor should it be approached with any pre-established purpose. According to de Waal, in ‘phaneroscopy there can be no other assertions than that there are certain seemings within which there seem to be certain recurring elements’ (p. 38).

[15] Marshall McLuhan’s notion of tactility with respect to television serves as the connective tissue between the remote control and the tactility of the mouse, as well as other touchscreen interfaces.

[16] Not unrelated is Anne Friedberg’s (2000) discussion of the RCD and how it produces the viewer as a montagiste of televisual images, someone who orchestrates a contrapuntal play of elsewheres and elsewhens in the time of her being present in front of the screen. For her the television viewer qua montagiste occupies (experiences) a distracted subject position. Elsewhere she draws on Walter Benjamin’s notion of distraction to suggest the haptic nature of distracted attention.

[17] Peirce’s tripartite parsing of the sign – icon, index, symbol – does not map simply or directly onto his categories of being. While the symbolic register assumes thirdness, firstness is not iconic and secondness does not necessarily leave a sign or trace.

[18] Perez-Teran Mayorga 2007.

[19] Ginzburg 1983.

[20] Doane 2007, p. 136. In this regard we understand the processual indexicality of traces-in-becoming in the Bergsonian sense of duree. See Bergson 1911 (orig. in 1896). See also Cooley 2014.

[21] The artists explain: ‘[i]n “Saving Face” you make your face visible on a big screen by touching your face. By caressing your own face you “paint” your face on a large electronic screen. On the screen your face appears and merges slowly with the portraits of previous visitors. Your merged portrait on the screen shows a temporary identity – further transforming through every face-caressing act. Together you compose new, temporary, non-traceable, and non controllable networked identities.’ (http://www.lancelmaat.nl/work/saving-face/)

[22] Theoretical conceptualisations of interface as boundary object, as defined by sociologists Susan Leigh Star and James R. Griesemer (1989) in a slightly different context, or the object/practice synthesis in Bruno Latour’s notion of ‘quasi-objects’ that are inscribed with action, apply here in the sense that the technological apparatus and the human subject merge. In a somewhat different vein Alexander Galloway (2012) speaks of the ‘interface effect’, approaching the interface primarily as practice. Or as Sybille Lammes (forthcoming  in 2014) puts it, the interface as Latourian sign-thing ‘invites users to perform certain actions that are then inscribed in it and become mediated through it’.

[23] It is worth specifying terms. ‘Orientation’ refers to the process of determining direction. ‘Positioning’ refers to finding (present) location by establishing a spatial relation to a specified object or site. ‘Framing’ is the contextualisation of an object and its placement within a space, frequently displayed and experienced on screen. ‘Moving’ encapsulates the sequential relationship between the aforementioned three actions. ‘Location’, which grounds this process, is both physical and informational. ‘Gesture’, in this context, denotes the trace or tracing of the fugitive and short-lived disposition to move; it plays out in and as the act of navigating.

[24] Both Sekula 1986 and Tagg 1993 develop poignant arguments about the historical coincidence of photography and policing.

[26]  While ‘mixed reality’ is ambiguous, as Ulrik Ekman explains, it is productive for characterising the scenarios that typify the navigational gesture. Limning two very different intellectual cultures – the technics of ubiquitous computing and the humanities and social sciences – the term indicates the usefulness or pragmatics of transdisciplinary exchange that affords more nuanced understandings of and designing for the mobile present. See the introduction in Ekman 2013.

[27] Verhoeff 2013.

Comments are closed.