![Petty-Diagram.pdf[1].jpg](Petty-Diagram.pdf[1].jpg)
Sheila Petty, Luigi Benedicenti, Christian Riegel, & Katherine Robinson
University of Regina
Abstract: There is an urgent need to examine the ways in which screen-centred interfaces present images and encode and decode meaning, identity, and culture. This project is an interdisciplinary collaboration by four researchers at the University of Regina and builds on our work on screen-centred interfaces in our respective disciplines of cognitive psychology, literary studies, media studies, and software systems engineering. The fundamental goals of our collaborative project are to engage interdisciplinary means and perspectives to systematically develop effective methodologies to measure cognitive processes, aesthetic effects, and software and hardware efficacy of the new and developing digital media. In this project/pilot study we intend to select a series of media fragments that include poetic, visual, and language texts, as well as those that combine these features, and present them on a variety of screen-centred interfaces to explore their cognitive and aesthetic effects and features.
Keywords: Screen-centred interface; Aesthetics; Technology; Visual media; Devices; Cognitive, cultural, and aesthetic; Text; Digital media
Sheila Petty is the Dean of Fine Arts and Professor in the Department of Media Studies at the the University of Regina, 3737 Wascana Pkwy, Regina, SK, Canada S4S 7K2. Email: Sheila.Petty@uregina.ca .
Luigi Benedicenti is the Associate Dean and Program Chair of the Faculty of Engineering at the University of Regina, 3737 Wascana Pkwy, Regina, SK, Canada S4S 7K2. Email: Luigi.Benedicenti@uregina.ca .
Christian Riegel is a Professor and Head of English at Campion College, University of Regina, 3737 Wascana Pkwy, Regina, SK, Canada S4S 7K2. Email: Christian.Riegel@uregina.ca .
Katherine Robinson is an Associate Professor in the Faculty of the Arts at Campion College, University of Regina, 3737 Wascana Pkwy, Regina, SK, Canada S4S 7K2. Email: Katherine.Robinson@uregina.ca .
The INKE Research Group comprises over 35 researchers (and their research assistants and postdoctoral fellows) at more than 20 universities in Canada, England, the United States, and Ireland, and across 20 partners in the public and private sectors. INKE is a large-scale, long-term, interdisciplinary project to study the future of books and reading, supported by the Social Sciences and Humanities Research Council of Canada, as well as contributions from participating universities and partners, and bringing together activities associated with book history and textual scholarship, user experience studies, interface design, and prototyping of digital reading environments.
It has become a commonplace notion that computer-based technology and forms of expression transform human experience and that the screen is the twenty-first century “face” of the image (Christine Ramsay, personal conversation, January 19, 2011). There is, thus, clearly an urgent need to examine the ways in which screen-centred interfaces present images and encode and decode meaning, identity, and culture, borne out of an intuitive sense that “whoever controls the metaphor controls the mind” (Bey, 1998, p. 6). This is not a question of technology alone, for as John Seely Brown (1999) has argued, “aesthetics and the technology for creating those aesthetics are tightly intertwined…Just as technology is influenced by its potential use, aesthetics or content is molded by what is possible” (p. xiii). And Manovich (2001) has argued that “we are no longer interacting to a computer but to a culture encoded in digital form” (p. 70). This project is an interdisciplinary collaboration by four researchers at the University of Regina who are working to advance the state of the knowledge in how aesthetically represented information – in language and in visual media – is understood, mediated, and processed. Our project builds on our work on screen-centred interfaces in our respective disciplines of cognitive psychology (Dr. Katherine Robinson), literary studies (Dr. Christian Riegel), media studies (Dr. Sheila Petty), and software systems engineering (Dr. Luigi Benedicenti).
The fundamental goals of our collaborative project are to engage interdisciplinary means and perspectives to systematically develop effective methodologies to measure cognitive processes, aesthetic effects, and software and hardware efficacy of the new and developing digital media. In this project/pilot study we intend to select a series of media fragments that include poetic, visual, and language texts, as well as those that combine these features, and present them on a variety of screen-centred interfaces to explore their cognitive and aesthetic effects and features. The fragments will have varied conceptual complexity and varied cultural references. Using a variety of screens (e.g., a television screen, a conventional computer screen, a tablet computer, a touch-screen phone, and a conventional mobile phone with limited screen space for simple text messages), we will examine cognitive and aesthetic features of how the fragments (e.g., an essay, a sonnet, or a net art project) are experienced on each platform and whether the essence of their content is altered or influenced. Our study will address whether and how media content is influenced by the device on which it is presented, from cognitive, cultural, and aesthetic perspectives. This pilot study is meant to 1) define parameters to develop methodologies and to construct an ontology to map the nexus between technology, aesthetics (including uses of time, space, text, font size, screen resolution, window size, et cetera), and user impact/experience; and 2) understand and measure the cognitive, cultural, and aesthetic experiences of screen users.
We start with the general premise that screens shape our world and identities in such ubiquitous ways that their very presence and influence often go unproven, or at the very least, unchallenged. According to Kate Mondloch (2010), “From movie screens to television sets, from video walls to PDAs, screens literally and figuratively stand between us, separating bodies and filtering communication between subjects….present-day viewers are, quite literally, ‘screen subjects’” (p. xxi). She further contends that the way in which we view or consume artworks made with screen interfaces has been underexplored as a system or method (p. xii). The challenge to create coherent frameworks or methodologies to describe how screen media create meaning has occupied a significant place in debates among new media scholars, and game and interface designers. Until very recently, primacy has been placed on what happens behind the screen, with a focus on the technology and software used by computer programmers and designers. And research in computer-based narrative has mainly focused on theoretical issues around what narratives do and how they inscribe interactivity on computer screens. It is time to redress the balance by bringing focus to bear on the screen itself and examine how images/sensations evoked on the computer screen, and this experience, create meaning with the user.
As early as the 1980s, Crawford advocated that “real art through computer games is achievable, but it will never be achieved so long as we have no path to understanding. We need to establish our principles of aesthetics, a framework for criticism, and a model for development” (1984). In his essay on whether computer games will ever be a legitimate art form, Adams (2007) disagrees with the need for a model of development as he feels art should be intuitively produced, but he agrees with the necessity for a methodology of analysis (p. 257).
Other theoretical positions have evolved to focus on either the technological construction of new media or their social impact. For example, in the quest to quantify effective human interface design, Laurel (1991) turns to theatre and Aristotle’s Poetics by creating categories of action, character, thought, language, melody (sound), and enactment (p. 50). However, Cubitt (1998) argues that “the possibilities for a contrapuntal organization of image, sound and text [should be] explored, in pursuit of a mode of consciousness which is not anchored in the old hierarchies.” Lunenfeld (2000) takes a more radical stance by suggesting that “once we distinguish a technoculture from its future/present from that which preceded it, we need to move beyond the usual tools of contemporary critical theory.” His assertion of the need for a “hyperaesthetic that encourages a hybrid temporality, a real-time approach that cycles through the past, present and future to think with and through the technocultures” (p. 29) offers its own set of problematics: computer-based forms are neither ahistorical, nor do they represent a leap in technology so distinct that they are unlinked to preceding forms.
Processing and experiencing text is embodied; linguistic meaning evokes all aspects of the experience of reading, physical and cognitive, and every aspect of language is implicated in embodiment (Geeraerts, 2009; Stockwell, 2002).This notion of the embodied experience of language corresponds with McLuhan’s evocation of the medium as an extension of the body in Understanding Media (1964). Ubiquitous computing embraces the embodied nature of language and literature in that it brings the media in closer contact with the human (for example, an individual becoming immersed in a virtual reality world). As Stockwell (2002) argues, “The notion of embodiment affects every part of language. It means that all of our experiences, knowledge, beliefs and wishes are involved in and expressible only through patterns of language that have their roots in our material existence” (p. 5).
Gibbs Jr. (2006) argues, “Understanding embodied experience is not simply a matter of physiology or kinesiology (i.e., the body as object), but demands recognition of how people dynamically move in the physical and cultural world (i.e., the body experienced from a first-person, phenomenological perspective)” (pp. 227-28). We link this notion of the embodied experience with McLuhan’s conception of the relationship of media to human experience and understanding, for McLuhan’s formulation inherently recognizes that exposure to a new medium is not only an experience of a new form of technology but that it also changes the way we relate to and understand the world and our place in that world. For example, the mobile phone could be considered as an extension of the ear in that it changes the fundamental way in which the human body is situated within the world (Gordon, Hamaji, & Albert, 2007).
Each of the above scholars touches directly or indirectly on the notion that there is something unique about screen-centred interfaces that defies inscription in previous modes of analysis, and all seem to be grasping for a language of description for the pervasive nature of ubiquitous information processing “in the human environment” (Greenfield, 2006, p. 14). We aim to develop theoretical frameworks with which to develop an understanding of the relation of conventional aesthetic textual forms to the newly and rapidly developing media technology that shapes our lives. We wonder how the new screens change and shift our relationship to text as well as our understanding and processing of that text. How does the increased embodiment of new screen contexts alter how we respond to a text (meaning various visual media) we read? Deriving from our theoretical questions and issues is the need to develop methodological tools to harness the potential of ubiquitous computing in the humanities and social sciences. Researchers are forced to find new methodologies for convergence between analogue theories and digital contexts, where the user’s freedom to determine sequence can profoundly affect the user’s response to the text and the meaning s/he derives from it. This pilot study will help us understand and develop methodological issues relating to how one studies digital media in the new screens that predominate our time: the variety of methods we will use all need to be calibrated, adapted, and integrated to be of value to researchers in the future. There are no models at the moment to aid in this work; thus, we are proposing to develop one.
This pilot study will be carried out over 18 months and will be divided into two broad phases with two steps each. During the first six months we will carry out steps One and Two, and during the final twelve months we will carry out steps Three and Four.
We will start a number of tasks that can be completed concurrently, as follows. Robinson and Riegel (and students) will select the appropriate measures of cognition, cultural relevance, and aesthetic relevance that form the basis of our analysis. Cognitive measures will include eye movements, measures of retention, recall, and reading/viewing speed. Measures of cultural and aesthetic relevance will include questions relating to the experience of reading, viewing, or being immersed in a digital media context. Benedicenti and Petty will acquire the devices proposed in the budget justification. All team members will write the application to the ethics board for study approval and to gain access to the research participant pool.
Petty and Riegel (and students) will select the media fragments (poetics, texts, visuals – both print-based and digital-based) to be shown on the devices. Possible examples include the net art project Blackness for Sale where new media artist Keith Townsend Obadike (2001) offered his “blackness” for sale on eBay in 2001, creating an effective commentary on the relationship between black identity and consumer culture. Because the project is primarily text-based, it raises the interesting issue of how text functions as an image system in net/web art. Examples of concrete poetry created for a visual medium could include Strings by Dan Waber (n.d.) and Concrete Poetry in Analog and Digital Media by Roberton Simanowski (n.d.) (see the examples under “2.”). In the first, the moving visual image is reflected in the text that emerges. In the second, the animation illustrates the text. In both, metaphor operates at the lexical level and at the level of image. Why might this be of interest? 1. We can work at metaphor from multiple directions, including at the level of linguistics, which reflects more closely the experience of using a new media device; and 2. Because the poem is fluid, it will lend itself well to the embodied nature of handheld and immersive worlds: a question might be, what happens when we move the text because we hold it in our hand as it too moves? Are there differences in cognitive processes and how they work over a static image? The second example provided above might be very useful for an experimental design because the animation is derived from a fixed text, so one would have access to both versions (e.g., paper/conventional and digital). These texts become a useful tool for methodological experimentation: how does one deal with digital aesthetic objects presented on digital media versus conventional forms? How do we deal with aesthetic experiences when the mode of delivery has changed so radically?
Then, Robinson, Riegel, Petty, and Benedicenti (and students) will design the questionnaire related to the measures chosen in the previous step. Benedicenti will lead a team of students to write the software programs for each device, prepare the media fragments for each display, and encode the questionnaire, and Robinson will co-ordinate the methodology with conventional cognitive protocols.
Eighty participants from the University of Regina Psychology Research Participant Pool will be equally distributed to one of four conditions (conventional computer screen, iPad, iPhone, mobile phone) and will be presented with the media fragments. We will use verbal reports (e.g., a questionnaire given after a participant has viewed a media fragment asking questions specific to our aims) to amplify cognitive, cultural, and aesthetic measures, and to provide insight into what the participant was aware of and/or was thinking about while they were exposed to the media fragment. We will also use eye movements to corroborate other measures. Eye movements will also provide us with valuable insight into where the participants were looking when they were viewing a media fragment and where their eyes shifted and for how long. The eye tracker in the new IMPACT lab at the University of Regina will be ideal for this pilot study. Data collection will occur in several dedicated testing spaces across campus and with the aid of student research assistants.
Student research assistants will help the team code the data to prepare a final data set: analyses of variance of the various cognitive measures (eye movements, recall, reading speed, et cetera) that will examine how these measures are affected by media platform will be conducted. Correlational analyses will be performed between the cognitive measures, the questionnaires examining participants’ aesthetic experiences, and the media platforms. The correlational analyses will also be used to construct a decision support system linking interface factors for all content with the parameters set as screens change. We will use software engineering systems compression methods like Principal Component Analysis and Clustering to extract a core set of measures that will constitute the initial vector state of the decision support system. The correlational analyses will provide the rules for linking these parameters and will be used to build an active rule set (either as a look-up table or as a set of if-then rules) that will form the knowledge base given to the system. The system, built in this way, essentially becomes a decision support system, or computer program, capable of forming a general prediction of the best type of content fragments to use in a certain defined screen size format. Linking changes in interface parameters (cognitive, cultural, and aesthetic) with different screens and their description, will allow us to infer how to automatically change a presentation from one interface to another and obtain a desired effect (cognitive, cultural, and aesthetic).
An example of the system structure is given in Diagram 1.
Some might ponder what could be the practical value of analyzing how media fragments are processed, experienced, and embodied? Could this lead to the creation of a common ontology or language? As “new and increasingly more affordable technology is putting creative control directly in the hands of consumers and creators” (Government of Canada, 2010, p. 25), and the so-called digital divide between these two is narrowing, a common language or ontology (much like a common film language or grammar allows media scholars to understand and discuss cinema) empowers both creators/producers and consumers of screen-centred interfaces. We need to create multimedia and screen-centred content that effectively communicates to various audiences while ensuring that our technology, information, economic, entertainment, and personal needs mesh in an effective way; multimedia literacy, which perforce means being able to effectively analyze multimedia and screen-centred content, is critical for every level of society from government to industry to the most personal of homepages.
Adams, Ernest W. (2007). Will computer games ever be a legitimate art form? In G. Mitchell & A. Clarke (Eds.), Videogames and art (pp. 255-264). Chicago, IL: Intellect Books.
Bey, Hakim. (1998). The information war. In J. Broadhurst Dixon & E. J. Cassidy (Eds.), Virtual futures: Cyberotics, technology and post-human pragmatism (pp. 3-9). New York, NY: Routledge.
Brown, John Seely. (1999). Introduction. In C. Harris (Ed.), Art and innovation: The Xerox PARC artist-in-residence program (pp. xi-xiv). Cambridge, MA: MIT Press.
Crawford, Chris. (1984). The art of computer game design. Berkeley, CA: McGraw-Hill/Osbourne Media.
Cubitt, Sean. (1998). The failure and success of multimedia. Paper presented at the Consciousness Reframed II conference at the University College of Wales, Newport, South Wales, August 20, 1998.
Geeraerts, Dirk. (2009). Incorporated but not embodied? In G. Brone and J. Vandaele (Eds), Cognitive poetics: Goals, gains and gaps (pp. 445-450). New York, NY: Mouton de Gruyter.
Gibbs, Jr., Raymond W. (2006). Embodiment and cognitive science. Cambridge, MA: Cambridge University Press.
Gordon, W. Terrence, & Hamaji, Eric, & Albert, Jacob. (2007). Everyman’s McLuhan. New York, NY: Mark Batty Publisher.
Government of Canada. (2010). Digital media: Creating Canada’s digital content advantage: Strategies for sustainable prosperity. Ottawa: Public Works and Government Services Canada.
Greenfield, Adam. (2006). Everyware: The dawning age of ubiquitous computing. Berkeley, CA: New Riders Press.
Laurel, Brenda. (1991). Computers as theatre. New York, NY: Addison-Wesley.
Lunenfeld, Peter. (2000). Snap to grid: A user’s guide to digital arts, media, and cultures. Cambridge, MA: The MIT Press.
Manovich, Lev. (2001). The language of new media. Cambridge, MA: The MIT Press.
McLuhan, Marshall. (1964). Understanding media: The extensions of man. Cambridge, MA: MIT Press.
Mondloch, Kate. (2010). Screens: Viewing media installation art. Minneapolis, MN: University of Minnesota Press.
Obadike, Keith T. (2001). Blackness for sale. URL: http://obadike.tripod.com/ebay.html [September, 2011].
Simanowski, Roberton. (n.d.). Concrete poetry in analog and digital media. URL: http://dichtung-digital.mewi.unibas.ch/2003/parisconnection/concretepoetry.htm [September, 2011].
Stockwell, Peter. (2002). Sociolinguistics: A resource book for students. London: Routledge.
Waber, Dan. (n.d.). Strings. URL: http://www.vispo.com/guests/DanWaber/index.html [September, 2011].
CCSP Press
Scholarly and Research Communication
Volume 3, Issue 3, Article ID 030128, 8 pages
Journal URL: www.src-online.ca
Received August 17, 2011, Accepted November 15, 2011, Published September 1, 2012
Petty, Sheila, Benedicenti, Luigi, Reigel, Christian, & Robinson, Katherine. (2012). New Theories and Methods for Screen-Centred Interfaces: A Pilot Study. Scholarly and Research Communication, 3(3): 030128, 8 pp.
© 2012 Sheila Petty, Luigi Benedicenti, Christian Riegel, & Katherine Robinson. This Open Access article is distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc-nd/2.5/ca), which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.