Developing an Open, Networked Peer Review System

Nina Belojevic
University of Victoria

INKE Research Group

Abstract: Using the Personas for Open, Networked Peer Review project as a case study, this paper discusses how best practices from creative technology-development contexts, such as agile development, can be applied in digital humanities projects.

Keywords: Prototyping; Scholarly communication; Online publishing; Open peer review

Nina Belojevic completed her Master’s in English at the University of Victoria. She has worked at the Electronic Textual Cultures Lab and the Maker Lab in the Humanities as part of the INKE and MVP research teams. Email: nbelojevic@gmail.com  

Implementing New Knowledge Environments (INKE) is a collaborative research intervention exploring electronic text, digital humanities, and scholarly communication. The international team involves over 42 researchers, 53 GRAs, 4 staff, 19 postdocs, and 30 partners.

With the evolution of digital approaches in the humanities, the development of digital prototypes, databases, systems, and tools is shifting from being generally treated as a service to being recognized as scholarship.1 Digital humanities and media studies communities increasingly appreciate practices such as designing, prototyping, coding, building, and making as scholarly, and there is a growing understanding in academic settings that the artefacts developed through these practices can be scholarly, too.2 However, this shift inevitably brings with it the need for new approaches and practices for the creation of such artefacts. Making digital artefacts (in a humanities context) is still relatively new and undefined. When it comes to developing digital projects, the most common practices consist either of making usable digital tools (e.g., text editors, browsers, search engines) or making expressive pieces (e.g., videogames, electronic literature, videos). In terms of scholarship, digital projects that aim to combine the two raise new challenges.

The Electronic Textual Cultures Laboratory (ETCL), in partnership with the Public Knowledge Project (PKP), the University of Victoria Library, and the Humanities Computing and Media Centre (HCMC), is currently developing a project called Personas for Open, Networked Peer Review – a prototype that Jentery Sayers and I began designing in 2013. While our initial prototyping stages consisted of low-investment formats that made it possible to focus our attention on the critical insights and arguments we wanted the project to engage, our current project phase is dedicated to the implementation of a usable digital iteration of this prototype. With the involvement of a wider group, the investment of bigger funds, and the new elements of complexity that implementation work requires, the attention that we were previously able to focus on the scholarly argument of our project now needs to be divided to allow for other considerations, such as the various objectives stakeholders have in the project, factors of time and money required to complete the project, and new audiences to keep in mind for the final product. Not surprisingly, conceptualizing an argument in the form of a prototype and building a functional tool that performs and presents that argument are substantially different endeavours. While the project could be structured as a continued prototyping process, as described by Stan Ruecker, Nadine Adelaar, Susan Brown, Teresa Dobson, Ruth Knechtel, Susan Liepert, Andrew MacDonald, Ernesto Peña, Milena Radzikowska, Geoff Roeder, Stéfan Sinclair, and Jennifer Windsor (2014) in their article “Academic Prototyping,” certain expectations for our project dictate that usability, efficiency, and accessibility comprise considerable portions of our attention, even if they may not facilitate the critical elements of the project as much as the productive elements.

In an effort to find successful methods to combine practical and scholarly elements in digital projects, I turn to iterative prototyping and agile development practices often used in game design. Videogames are known for their ability to capture players’ attention, communicate complex concepts, and engage the audience in an experience. As Ian Bogost (2007) puts it, videogames make strong arguments through “procedural rhetoric” – “the practice of persuading through processes in general and computational processes in particular” (pp. 2-3). While I do not intend to suggest gamification as an approach for scholarly, digital projects, I would argue that certain project management and workflow methods and practices common in game development and other creative technology development sectors might help us define methodologies for digital projects that aim to be usable and scholarly at once.

The Personas for Peer Review wireframe prototype allowed us to visually and structurally present an open, networked peer-to-peer review process that we are now developing as a plug-in for Open Journal Systems (OJS). We created detailed wireframes to communicate our concept,3 which consists of an environment in which articles are published for open review and commented on by the public or by an invited group of reviewers. Based on our prototype, feedback and comments can be seen by everyone with access to the article, and reviewers and authors are able to ask each other questions, give responses, and engage in dialogue.

Figure 1: Marginal commenting and an open review article

Figure 1: Marginal commenting and an open review article 

An article can go through this open review process and only be published in that format, or it can be combined with blind peer review. This process not only allows authors and editors to engage in an alternative review approach; it also draws attention to the collaborative nature of writing and publishing in general. To do so more overtly, we developed the concept of “personas” in our prototype. Every user has a profile to represent their professional identity, but each profile also contains multiple personas.

Figure 2: User profile with multiple personas

 Figure 2: User profile with multiple personas

Personas are created by the community and are specific to each user. For any contribution a user makes (an article publication, a reviewer’s comment, a response to a reviewer’s comment, etc.), other users can assign persona tags that indicate certain areas of expertise or skills. For example, one user could have a profile that shows their position as English professor, while their personas pages would indicate their expertise in a variety of areas, such as Victorian poetry, Web design, postcolonial studies, and proofreading.

Figure 3: Single user persona example

 Figure 3: Single user persona example

The personas project has been designed to make arguments procedurally through users’ engagement (such as reviewers’ marginal comments) and data generation. As Sayers and I (Belojevic & Sayers, 2014) describe in detail in our article “Peer Review Personas,” our prototype of an open, networked environment in which reviewers can make comments of varying levels of detail on articles, engage in dialogue with each other and the authors, and assign each other persona tags to represent specific areas of expertise or skills aims to inherently and procedurally argue for self-reflexivity and a critical perspective on practices, tendencies, or norms that may otherwise simply be accepted without consideration or question. Our prototype makes such arguments because it puts the collaborative and iterative nature of scholarly communication at the forefront; it represents the range of scholarly labour that goes into academic publishing as well as the divergent roles that it relies on, many of which can often be overlooked and seem rather ephemeral in other review contexts; it allows practitioners to develop reputations in areas of expertise that might otherwise not be represented in academic contexts, thus drawing attention to important skills and forms of knowledge that may otherwise not be visible; and it shows metrics at aggregate and granular levels so that user activity is not simply quantified, but instead encourages self-reflexivity, provides new insights on possible tendencies or biases, and allows for a better understanding of reviewer culture. To put it more succinctly, the prototype presents review work as an integral part of scholarly labour and important to scholarly communication – it contextualizes it as something worth looking into, studying, and discussing critically.

Since these arguments, issues, questions, and critiques can only emerge through the use of the open, networked peer review plug-in features, they are inherently built into the prototype. It is also for this reason that elements of the project can only be designed or refined once we can gain insights on how users interact with it. Thus, as Sayers and I (Belojevic & Sayers, 2014) noted, in designing networked platforms, reputation engines, and other community-driven online technologies, it is necessary to “be willing to analyze how they are culturally or socially embedded, and then—if we are still curious about their applications for peer review—construct or rewire them with a deep awareness of the values and exchanges they support” (n. p.). Gaining insights to evolve and refine our project does not need to be a vast initiative, at least to begin with. Early iterations of specific features that make up the personas project can first be tested with smaller reviewer groups to provide initial insights. As Matthew K. Gold (2012) explained about the networked review process for Debates in the Digital Humanities, even a small group of reviewers can engage in relevant debates and indicate social relations and forms of engagement, all of which would be relevant to the design of a networked review environment.

The complexity of the model we are using to make an argument through the personas prototype stems from the dual purpose we assign to persona tags: persona tags in certain ways represent the value the community ascribes to contributions made by users; however, while similar tagging, rating, and voting approaches are often applied to encourage users to contribute more (a productivity enhancing method that has especially flourished with gamification practices),4 our purpose in using this method is to draw attention to forms of labour that are often ephemeral and overlooked in both the digital economy5 and in the academic system. Ideally, participation in the Personas for Peer Review platform would incite authors, reviewers, and editors alike to contribute in a self-reflexive fashion, recognize their own and others’ labour, and gain awareness of possible biases in scholarly review once they are more openly visible at varying levels of granularity. In reality, though, Web users are accustomed to certain formats and might not critically consider them – in other words, a personas system might be adopted simply as another meritocracy system for users to compete in and be productive in, rather than drawing critical attention to such systems and encouraging more discourse and development in the review and publishing system.

Implementing the personas concept in a way that is intuitive and usable and simultaneously draws attention to labour and biases through user engagement requires conscientious design. However, with the commencement of technical development, a number of practical factors common in digital projects also emerged: we need to work within a budget that covers resources we can commit to the project (primarily labour); select specific tools (software, platforms, languages, etc.) that will allow us to meet objectives and be efficient (we decided to use Open Annotator, an online, browser-based annotation systems, to create a commenting plug-in for OJS); build a timeline of deliverables in order to complete project elements in a productive fashion and manage expectations; communicate plans, changes, and developments in a coherent reporting structure to make sure all stakeholders are content; and define objectives that address the goals of the different stakeholders. The OJS platform offers the ideal environment to develop initial, usable iterations for the personas peer review environment with features relevant to their community that can be tested by interested journal users. While, initially, we expect such testing to occur at a rather small scale (for example in one specific journal issue with an invited group of users), the plug-in format we are using for the various features of the personas project would makes it easy to expand the project across the OJS user base and, possibly, even beyond it if there is enough interest to modify the plug-ins for use with other platforms. Such expansions, however, would likely not occur until the project has been developed, refined, thoroughly tested, and optimized for some time. Hence, the entire process should ideally be structured to be quite iterative and flexible.

Having identified the partners, deliverables, scale, and tools, we then determined tasks to reach the agreed-upon goals. As some project management literature indicates, predetermining all elements in creative projects can be challenging and limiting.6 In the case of the personas project, while certain deliverables are easy to outline upfront, maintaining the more subtle elements of our concept requires careful design choices, user insights, and adjustments along the way. The reception of our argument will rely heavily on the ways in which users interact with the environment we are building. In order to foresee, learn about, test, and adapt based on user behaviours and new insights, I suggest looking at game design approaches that apply agile, iterative development processes.

Game design, along with a number of other types of creative technology development, often consists of highly complex projects that need to balance tight timelines, strict budgets, technical intricacy, creative expression, and many unknown factors (such as user responses, bugs, etc.). While a clear understanding of the project end goals and the required steps to get there should certainly be determined at the commencement of the implementation phase, the completion of the various project elements should be conducted with a certain level of flexibility and the possibility to make adjustments as the project proceeds. During iterative prototyping, as commonly practiced in game design, low-investment forms of testable designs (such as paper and pencil sketches or interactive wireframes) allow designers to try out different approaches, “(play)test” them on their own and with others, and adapt them along the way. During the personas prototyping process, we applied an approach similar to that described by Katie Salen and Eric Zimmerman (2003) in Rules of Play: Game Design Fundamentals outlining common game design approaches.

  1. We began with rough sketches to visually communicate our ideas and to identify what elements and interactions the environment would consist of. By simply sketching this out on sheets of paper it was easy to play with different ideas and evolve the concept quickly. 
  2. Once we had a clear concept in place, we designed more refined wireframes using Balsamiq Mockups. While Balsamiq allows for linking and clickable interaction between frames, we often opted to simply print, post, organize, and move around the wireframes to make sense of the ways in which users might interact with them.
  3. We interacted with the wireframes and gathered feedback from our colleagues. As we gained insights, we adjusted the wireframes, tweaked them, added to them, or made completely new ones. This process is similar to the important phases of “playtesting” and evaluation that Salen and Zimmerman (2003) describe in the context of game design (pp. 24-25).
  4. During the iterative prototyping phase, we analyzed and refined the wireframes at each stage until we had a grasp of the elements that make up our prototype and how they would work in practice. Finally, we also created detailed use cases, which, in turn, led to another cycle of iterative prototyping until those were refined enough to move into production.

Figure 4: Paper interactions to enact user flows

Figure 4: Paper interactions to enact user flow


Figure 5: Visualized reader use case

Figure 5: Visualized reader use case

When we had a coherent prototype in place, we focused on organizing the bigger project into smaller iterations. To maintain the flexibility required to build a usable environment that communicates an argument through user engagement, we are looking at agile development practices. Agile development is an iterative and incremental approach (Cohn, 2014) whereby a project is structured into small modules that build on previously completed pieces (especially when more defined methods, such as Scrum, are used).7

While working on our project, we realized that, at a certain stage, the move from iterative prototyping to agile development is important. Agile development delivers on complete subsections of the project that could, potentially, go live or be sent to project participants as smaller chunks, even if the full product is not complete. This allows stakeholders to see their goals being met as the project proceeds, while creators and scholars can engage in a process that ensures the creative or academic quality of the project is maintained as well. For scholarly projects such a workflow brings with it a number of benefits. For instance, it helps to identify roles (e.g., product owners, development team, Scrum masters) that work well for small teams, and the incremental approach facilitates a clear understanding of responsibilities and project organization. For teams where each person plays multiple roles, it can be challenging to balance the many different elements that will make a project successful. The workflow of agile development, with clearly defined increments and evaluation steps, streamlines the process to ensure that each component fulfills the project requirements. Another valuable attribute of agile development is that, while individual components get delivered at various stages of the project, analysis, critique, and revisions make up core components of the process. By focusing on one step at a time, scholarly and creative concerns, new research, and new insights can be addressed at each stage of the project. Finally, agile development lends itself to the at times open-ended nature of scholarly, digital projects. Rather than delivering one final product, it can be an ongoing process whereby project elements are completed and can be launched at different stages. With dependencies on budget cycles, funding, and a revolving team, an agile approach offers the flexibility and growth that can be helpful in creating successful digital humanities projects.

Although we did not specifically consider an agile development approach at the outset of the production phase for the Personas for Peer Review project, we did structure our project plan in a way that is flexible enough for us to now apply elements of agile development and plan for testing and evaluation stages. The project is being developed in increments, allowing us to see how users interact at each stage based on smaller test groups. This will not only make it possible for our team to refine each feature based on insights gained throughout this process, but it is particularly important for the personas project, since the concept relies on user engagement and data that, we hope, will encourage self-reflexivity and critical discourse– things that we cannot fully plan for but have to design as we learn more about how the various users interact in practice. Agile development approaches facilitate this kind of iterative refinement and make it possible for the project to slowly grow in scale as it is developed. We plan to determine exactly what aspects of agile development fit best with our project structure, our team, and the work that we need to complete. As we move through that process, we will aim to balance efficient project development and a focus on deliverables, while also remaining flexible and agile enough to research, test, and practice new insights we gather along the way.

Notes

  1. Article from the Sustaining Partnerships to Transform Scholarly Production INKE gathering on January 27, 2015, in Whistler, BC.]
  2. The discourse regarding such forms of digital scholarship is highly relevant and interesting. In “Developing Things,” Stephen Ramsay and Geoffrey Rockwell (2012) discuss why “building is scholarship” (p. 78); Alan Galey and Stan Ruecker (2010) explain how prototypes can make arguments in “How a Prototype Argues;” and in “Academic Prototyping,” Ruecker et al. (2014) note that academic prototyping can move through  “a relatively long lifespan as prototypes, with each successive cycle of design and development extending our understanding” (p. 2). In all of these discussions, the merits of digital practices as scholarly practices are demonstrated.
  3. Jentery Sayers and I (2014) discuss our wireframes and our Personas for Peer Review concept in detail in our article “Peer Review Personas.”
  4. Gamification and similar Web and app design practices have become a best practice in many industries and contexts. For example, Stack Overflow, a question and answer site for developers, uses votes to display a user’s and an answer’s quality as perceived by the community. Similarly, the Mozilla Open Badges project allows users to collect badges for certain skills and achievements. Both of these are prominent examples of the ways in which contributions and achievements can be represented in digital settings. However, as Lisa Nakamura (2013) contends in “‘Words With Friends:’ Socially Networked Reading on Goodreads,” reading environments like Goodreads may present themselves as “passive conduits” (pp. 9‑10), but beyond simply remediating common reading practices, systems like Goodreads make readers content producers and thus turn readers into workers through “play labor” (pp. 7-8).
  5. Various forms of labour in the digital economy or the “attention economy,” as Jonathan Beller (2006) puts it, are often misrepresented and perceived as entertainment and leisure activities while, in actuality, they are value-productive activities that extend a person’s productivity beyond the regular working day. See Jonathan Beller’s (2006) The Cinematic Mode of Production for a thorough analysis of the attention economy. More specifically focused on the digital economy, Trebor Scholz’s (2013) collection of essays, Digital Labor: The Internet as Playground and Factory, takes a look at the digital realm as labour site.
  6. As Esther R. Maier and Oana Branzei (2014) explain in their article “On Time and On Budget,” in large, complex, creative projects the “paths to goal … cannot be specified in advance” (p. 1123). Similarly, Laurent Simon (2006) explains that creative project managers commonly have to learn as they go, since many project aspects need to be “define[d] and refine[d] in the process” (p. 119).
  7. Agile software development can be described as a practice and a value, as indicated in the Manifesto for Agile Software Development and the principles outlined there (Beck, Beedle, van Bennekum, Cockburn, Cunningham, Fowler, Grenning, Highsmith, Hunt, Jeffries, Kern, Marick, Martin, Mellor, Schwaber, Sutherland, & Thomas, 2001). Scrum is a particularly popular agile development approach that has been adopted by a wide range of creative and technology organizations. Scrum Guides breaks down the process in detail and organizations such as the Scrum Alliance offer resources and courses for organizations and individuals who want to start using Scrum.

References

Agile Methodology. Agile methodology. (2014). URL: http://agilemethodology.org [December 2, 2014].

Beck, Kent, Beedle, Mike, van Bennekum, Arie, Cockburn, Alistair, Cunningham, Ward, Fowler, Martin, Grenning, James, Highsmith, Jim, Hunt, Andrew, Jeffries, Ron, Kern, Jon, Marick, Brian, Martin, Robert C., Mellor, Steve, Schwaber, Ken, Sutherland, Jeff, & Thomas, Dave. (2001). Manifesto for agile software development. URL: http://agilemanifesto.org [December 2, 2014].

Beller, Jonathan. (2006). The cinematic mode of production: Attention economy and the society of the spectacle. Hanover, NH: Dartmouth College Press.

Belojevic, Nina, & Sayers, Jentery. (2014). Peer review personas. Journal of Electronic Publishing, 17(3).

Bogost, Ian. (2007). Persuasive games: The expressive power of videogames. Cambridge, MA: MIT Press.

Cohn, Mike. (2014, November 11). Agile needs to be both iterative and incremental. Mountain Goat Software. URL: http://www.mountaingoatsoftware.com/blog/agile-needs-to-be-both-iterative-and-incremental [December 8, 2014].

Galey, Alan, & Ruecker, Stan. (2010). How a prototype argues. Literary and Linguistic Computing, 25(4), 405-424.

Gold, Matthew K. (2012). The digital humanities moment. In Matthew K. Gold (Ed.), Debates in the digital humanities (pp. ix-17). Minneapolis, MN: University of Minnesota Press.

James, Michael. (2012). Scrum reference card. Scrum Reference Card. URL: http://scrumreferencecard.com/ScrumReferenceCard.pdf [December 2, 2014].

Keith, Clinton. (2014, September 22). Why agile game development? Agile Game Development. URL: http://blog.agilegamedevelopment.com [December 2, 2014].

Maier, Esther R., & Branzei, Oana. (2014). On time and on budget: Harnessing creativity in large scale projects. International Journal of Project Management, 32, 1123-1133.

Nakamura, Lisa. (2013). Words with friends: Socially networked reading on Goodreads. PMLA, 128(1), 238-243.

Ramsay, Stephen, & Rockwell, Geoffrey. (2012). Developing things: Notes toward an epistemology of building in the digital humanities. In Matthew K. Gold (Ed.), Debates in the digital humanities (pp. 75-84). Minneapolis, MN: University of Minnesota Press.

Ruecker, Stan, Adelaar, Nadine, Brown, Susan, Dobson, Theresa, Knechtel, Ruth, Liepert, Susan, MacDonald, Andrew, Peña, Ernesto, Radzikowska, Milena, Roeder, Geoff G., Sinclair, Stéfan, & Windsor, Jennifer. (2014). Academic prototyping as a method of knowledge production: The case of the dynamic table of contexts. Scholarly and Research Communication, 5(2), pp. 1-14.

Salen, Katie, & Zimmerman, Eric. (2003). Rules of play: Game design fundamentals. Cambridge, MA: MIT.

Scholz, Trebor (Ed). (2013). Digital labor: The Internet as playground and factory. New York, NY: Routledge.

Schwaber, Ken, & Sutherland, Jeff. (2014). The scrum guide. URL: http://www.scrumguides.org/scrum-guide.html [2 Dec. 2014].

Scrum Alliance. (2014). URL: https://www.scrumalliance.org [December 2, 2014].

Simon, Laurent. (2006). Managing creative projects: An empirical synthesis of activities. International Journal of Project Management, 24, 116-126. 

CISP Press
Scholarly and Research Communication
Volume 6, Issue 2, Article ID 0201205, 11 pages
Journal URL: www.src-online.ca
Received May 27, 2015, Accepted July 13, 2015, Published October 14, 2015

Belojevic, Nina & INKE Research Group. (2015). Developing an Open, Networked Peer Review System. Scholarly and Research Communication, 6(2): 0201205, 11 pp.

© 2015 Nina Belojevic & INKE Research Group. This Open Access article is distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc-nd/2.5/ca), which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.