Literary Evaluation in Online Communities of Writers and Readers

Peter Boot

Huygens Institute for the History of the Netherlands

Abstract: To date, there has been very little research into online writing communities, largely as a result of the perceived low quality of writing produced in these communities. This article examines literary evaluation within online writing communities. Specifically, the Dutch site Verhalensite, which publishes both poems and stories, is analyzed in an attempt to determine why one work may be rated more highly than another, and whether a work’s ratings create an enduring reputation for its author.

Keywords: Online writing communities; Literary evaluation; Poem online; Verhalensite; Online writing; Online publishing

The INKE Research Group comprises over 35 researchers (and their research assistants and postdoctoral fellows) at more than 20 universities in Canada, England, the United States, and Ireland, and across 20 partners in the public and private sectors. INKE is a large-scale, long-term, interdisciplinary project to study the future of books and reading, supported by the Social Sciences and Humanities Research Council of Canada as well as contributions from participating universities and partners, and bringing together activities associated with book history and textual scholarship; user experience studies; interface design; and prototyping of digital reading environments.

Peter Boot is Senior Researcher at the Huygens Institute for the History of the Netherlands, Postbus 90754 2509 LT Den Haag, Netherlands. Email: peter.boot@huygens.knaw.nl .

Introduction

Since the advent of the Web, (aspiring) writers can publish their poems and stories on a great many websites. Many of these sites have a community of regular visitors, and the posted works are often the subject of lively discussion. Researchers into electronic or web literature have tended to overlook these sites, as their interest usually focuses on (often experimental) works that use the formal features of this new medium, such as interactivity, hyperlinking, multimedia, etc. Internet poetry or story sites are not the place to look for formal inventions. They do offer, however, an in situ view of the processes of writing, publishing, reading, and giving or taking feedback. Never before has it been possible to study literary communities and the process of literary evaluation in such detail.

An English-language example of an online writing community is the Poem Online forum (http://www.poem.org/). Members can submit poetry on which others can comment. One of the distinguishing features of this site is that there are three quality levels (novice, intermediate, and advanced), and poets earn their right to place poems in these sections by first commenting on the works of others. There are special sections for draft poems, “exotic genres”, and prose. There is also a “learning centre”; like many similar sites, Poem Online presents itself as a place to learn the poet’s craft as well as a place for publishing one’s poems. The forum has 1900 registered members and hosts 6000 poems. It uses generic bulletin board software to host the discussions.

This article will first describe Verhalensite, a Dutch-language online writing community, then outline some key research questions in relation to online writing communities, and finally, explore ways to address these questions.

Verhalensite

Although the name (“story site” in Dutch) suggests otherwise, Verhalensite (http://www.verhalensite.com/), an online writing community similar to its English-language counterpart Poem Online, hosts both stories and poems. The site was founded in 2001, and is home to a large community of writers (2700 persons); it contains 48,000 works that have received 350,000 comments. The atmosphere seems to be friendly and constructive, perhaps as a result of an active site administration. For many members, the site is not just a forum to publish one’s poems or stories, it is also a social space where people hang out, socialize, find friends and even spouses. Yearly meet-ups allow members to get to know each other in real life. Despite these apparent successes, the site ceased to operate in April 2011.

Verhalensite uses software specially developed for the site. Unlike most bulletin board software, it clearly distinguishes the original message (in this case, the poem or story) from the comments it receives (Fig. 1). The figure shows: (1) poem text; (2) title; (3) author name, linked to profile; (4) category and genre; (5) explanatory note by author; (6) links to next and previous poems; (7) first comment, including name and picture of commenter, hyperlinked to profile page; (8) replies to comment; (9) average rating; and (10) number of views and number of reactions. The simple and effective layout does away with the distracting clutter often dominating bulletin board systems and allows visitors to focus on the poem or story.

Figure 1: Poem on Verhalensite

Participants on Verhalensite range in age from young teenagers to senior citizens, include both Dutch and Belgian nationals, and are predominantly female¹, although data on the specific demographics of the website’s users are not available.

Research questions

Clearly, a site such as Verhalensite could be studied from a multitude of angles and by a wide variety of disciplines, ranging from sociological investigations into the way groups of like-minded authors come into being, to education-based studies into the impact of the site on reading and writing skills. Even within a literary perspective, possible interests are manifold. What sorts of stories are people writing? To what genres can they be said to belong? Do people write in multiple genres? Can the works be related to the information people give about themselves on the profile pages? To what extent do these works imitate works published on paper?

Much has been written about online writing in general (see, for example, Witte, 2007; Duncan & Leander, 2000), but research specifically into online writing communities has, up to now, been very limited. The closely related practice of online fan fiction has received a fair amount of scholarly attention (Black, 2008; Jenkins, 2006; Pugh, 2005), while Torlina and Kazakevitch (2003) have described a Russian literature site as a virtual community and presented some statistics about its usage. Apart from these limited examples, literary scholarship has largely ignored online writing communities due to the perceived low quality of the writing produced in these communities (Vaessens, 2005; Baetens and De Geest, 2007).

This article focuses on the processes of literary evaluation; that is to say, the questions of why one work is rated higher than another, and whether these ratings create some sort of enduring reputation for the works or their authors. In traditional literary studies, these questions have been hard to investigate, mainly because of a lack of data. In contrast, in online writing communities such as Verhalensite the questions can be investigated due to several factors: (i) the numbers of works and evaluations are so large that meaningful statistical analysis becomes possible; (ii) to a large extent, the process of literary evaluation takes place in public; (iii) all relevant communication takes place on a single site; (iv) we have electronic access to the works that are being evaluated, the evaluations, (some) data about the evaluators, and (some) information about non-evaluating readers; and (v) data is kept available over a number of years.

Several limitations to the data do, however, exist. First, not all evaluation takes place in public; for example, advice or criticism can be given in private messages. Some participants may be acquaintances, friends or family members in real life, and not all of their discussions may necessarily be made public. Events that take place in real life, such as Verhalensite meet-ups among participants, may also influence reputations, both positively and negatively. Second, some members of Verhalensite also publish their poems and stories on other sites, whether on other poetry sites, their own sites, or both. Some of the Verhalensite authors have also published (usually self-published) collections of stories or poems in book form, including in a number of anthologies based on Verhalensite material. Third, authors may change their works, whether in response to criticism or for other reasons, often meaning that the original version often is no longer available. Fourth, information about the evaluators, obtained through users’ profile pages, is unstructured and limited. Machine access to the profiles requires that human coding has been done previously. Fifth, non-evaluating readers are lumped into overall numbers of readers for each work, along with evaluating readers. Strictly speaking, this turns every reader (and even non-reader) into a participant in the evaluation process, as these numbers may influence the reading and evaluation behaviour of subsequent visitors. Lastly, authors have the option to remove their work or even leave the site altogether. This inevitably has a negative impact on the quality of the numbers used in analysis.

Despite the wide array of opportunities for research on online writing communities that exist, there are invariably limitations and obstacles to such research. In his study of Polish literature sites, Maciej Maryl (2008) highlights a number of these limitations, one of which is that, in general, internet users are anonymous. It is, for instance, quite possible for a community member to create multiple accounts and thus to manipulate the process of evaluation². Another important limitation is that the researcher has to make do with the available data, and does not have the option of asking the research subjects supplementary questions.

Various measures can be used as an indicator of a work’s success on Verhalensite, the obvious ones being the number of times read, the average time spent reading, and the number of comments and the rating. Other measures include whether the work was selected as poem or story of the day, or whether it received an “exclamation mark,” designations that are awarded by the (anonymous) site administration. In determining a work’s appreciation, besides looking at this quantifiable information, we might also take into account the text of readers’ comments on the works. These comments may not lend themselves to easy quantification, but certainly provide meaningful hints about the reasons why certain works are appreciated or not.

Several potential reasons could explain readers’ appreciation of a particular work. Intuitively, perhaps the most appealing reason is the notion of the work’s "true" literary merit; however, in the absence of any agreed definitions of literary quality, let alone ways of measuring it, we have no way of accounting for a work’s quality, short of showing that the expressed appreciation cannot be explained by other factors.

Fortunately, there are many factors that do lend themselves to empirical study. Positive feedback may be the reward for an author who usually responds positively to others, or may result from an influential site participant responding positively to one’s work. It may be determined by previous positive responses, or from an author’s position of authority. Demographic characteristics (the sex, age, nationality, of both author and commentator) may also play a role. We can also investigate the semantic categories of the words used in the work, or perhaps the agreement in semantic categories between the works of author and commentator, for instance using Linguistic Inquire and Word Count (LIWC) (Pennebaker, Booth, & Francis, 2007). We could use technology such as Latent Semantic Analysis to test whether similar texts receive similar ratings (Landauer, McNamara, Dennis, & Kintsch, 2007). Finally, we might want to look at the role of commentators’ desires to align themselves with (or distance themselves from) the opinions of other (groups of) commentators.

Exploratory findings

While these research questions will require more in-depth analysis to provide meaningful conclusions, a few initial findings suggest promising paths for further analysis.

Quid pro quo

Many of the site’s participants are both authors and commentators. As an author, a participant’s status increases with the number of comments they receive (see Meyer, 2008, for an analysis of the influence of feedback on posting). It seems plausible, then, that an author who often comments on others’ work will receive many comments in return. As an initial test, I computed the correlations between the number of comments given (nro), the average score (avgscore) and the average number of comments received (nript). These are the results:

nro	nript	0.33
avgscore	nro	0.13
nript	avgscore	0.27

There is a clear correlation between the number of outgoing comments (nro) and the number of incoming comments per text. A possible explanation is that, indeed, people return the favour, and that giving comments might even be a conscious strategy. Of course, other explanations are also possible. It might be that good writers take the trouble of commenting on others’ work because they have better communication skills. Further investigation should look at whether comments are given to and received from the same persons, as well as the sequence of commenting. Interestingly, the correlation between the number of outgoing comments and the average score is less strong. This might indicate that people feel obliged to return a comment but not necessarily a high rating.

Comment words

In order to explore the purpose of commentary, I analyzed the vocabulary used in comments. To identify the most significant words, I selected the top 5,000 words used in the comments, ordered them by rank, and subtracted the rank of the same word in a Dutch reference corpus (Institute for Dutch Lexicology, n.d.). The resulting ranking orders words by the difference between their frequencies here as compared to use elsewhere. The top ranked words in the resulting list are predominantly socially motivated (e.g., greetings, often in affectionate terms, expressions of thanks or congratulations) or words of admiration (e.g., cute, beautiful, great). Other top ranked words include many technical terms (e.g., stanza, paragraph), indicating that there is also commentary on the content of the works. Interestingly, the top word that could refer to the thematic content of the texts is “loneliness,” while “sad” is also used frequently, presumably as a compliment. Critical remarks are indicated by the use of the words “errors” and “pity.” A fuller analysis would try to assess which commentators use which categories of words.

Network analysis

The act of commenting on a work defines a network of site users. Figure 2 shows a network graph of the comments given in January 2010. Arrows indicate two or more comments in the direction of the arrow. Darker arrows indicate more comments. Node colour indicates genre preference (blue for poetry, orange for serial stories, and red for other stories).

The figure shows two main groups. It suggests, perhaps unsurprisingly, that poets comment on poems, and prose writers on prose. Subgroups for writers of serial fiction and regular prose might also be present but more difficult to identify. Interesting follow-up questions would be whether authors who receive comments from the same commentators are stylistically similar, and whether the displayed pattern of commenting persists through time.

Figure 2: Network of comments January 2010

Prospects

The preceding discussion has shown that online writing communities offer valuable possibilities for research; however, the extent to which the results of this research may be applicable to the wider literary world remains unclear. Processes of literary evaluation in the world of regular literature will in many respects be different from the processes in online writing communities. For example, literary institutions such as publishers, newspaper critics, academics, and booksellers are mostly absent from online writing communities. On the other hand, to the extent that the advent of book reviewing sites, book discussion forums, and social media sites enhances the role of the common reader in determining reputation within the wider literary world, the evaluation processes of online writing communities and traditional print literary works may become more alike. Similarly, Verhalensite does have some of the machinery of reputation, building of regular literature, such as awards, (informal) teachers, and interviews with regularly contributing authors.

The processes discussed here, however, are not only relevant to literary evaluation. Online writing communities have many similarities to other online communities, such as photo or video sharing sites. It would be worthwhile to compare the mechanisms for reputation building in these different communities.

Notes

I tried to determine sex and nationality from the profile pages and URLs mentioned for the top 250 members (in terms of participation both in writing and commenting). For sex, the results were: 120 female, 85 male, 45 unknown; for nationality: 46 Belgian, 78 Dutch, 126 unknown.
Verhalensite members can optionally have their home address checked in order to enhance trust.

References

Baetens, Jan, & De Geest, Dirk. (2007). E-Literatuur in het Nederlands: Veel E-, weinig literatuur. Jaarboek voor Nederlandse boekgeschiedenis, 14, 160-178.

Black, Rebecca W. (2008). Adolescents and online fan fiction. New York: Peter Lang Pub Inc.

Duncan, Barbara, & Leander, Kevin. (2000). Girls just wanna have fun: Literacy, consumerism, and paradoxes of Position on gurl.com. Reading Online, 4(5). URL: http://www.readingonline.org/electronic/elec_index.asp?HREF=/electronic/duncan/index.html [September, 2010].

Institute for Dutch Lexicology. (n.d.). Frequency list of the ANW-corpus. URL:

http://www.inl.nl/nl/lexica/frequentielijsten-corpora [November 7, 2010].

Jenkins, Henry. (2006). Do you enjoy making the rest of us feel stupid? alt.tv.twinpeaks, the Trickster Author, and Viewer Mastery. In D. Lavery (Ed.), Fans, bloggers, and gamers: Exploring participatory culture (pp. 115-133). New York, NY: NYU Press.

Landauer, Thomas K., McNamara, Danielle S., Dennis, Simon, & Kintsch, Walter (Eds.). (2007). Handbook of latent semantic analysis. Mahwah, NJ: Lawrence Erlbaum.

Maryl, Maciej. (2008). Virtual communities – real readers: New data in empirical studies of literature. In Jan Auracher & Willie Van Peer (Eds.), New beginnings in literary studies (pp. 390-406). Cambridge: Cambridge Scholars Publishing.

Meyer, Katrina A. (2008). Do rewards shape online discussions? Journal of Interactive Online Learning, 7(2), 126-138.

Pennebaker, James W., Booth, Roger J., & Francis, Martha E. (2007). Linguistic inquiry and word count (Liwc2007). URL: http://www.liwc.net [September 29, 2010].

Poem Online. (2010). URL: http://www.poem.org [September 29, 2010].

Pugh, Sheenagh. (2005). The democratic genre: Fan fiction in a literary context. Bridgend: Seren.

Torlina, Luba, & Kazakevitch, Gennadi. (2003). Web publishing revisited - a case study of literary websites in Russia. In J. Hanisch, D. Falconer, S. Horrocks, & M. Hillier (Eds.), 7th Pacific Asia Conference on Information Systems (pp. 271-284). Adelaide: Pacis.

Vaessens, Thomas. (2005). Dichters en schrijvers op Internet. Een sociologisch en literair fenomeen. In C. Dauven, J. Koopmans and L. Kuitert (Eds.), Publiceren, wat is dat? Een antwoord in dertien opstellen (pp. 183-195). Amsterdam: Instituut voor Cultuur en Geschiedenis.

Verhalensite. (n.d.) URL: http://www.verhalensite.com [November 7, 2010].

Witte, Shelbie. (2007). "That's Online Writing, Not Boring School Writing": Writing with Blogs and the Talkback Project. Journal of Adolescent & Adult Literacy, 51(2), 92-96.

CCSP Press
Scholarly and Research Communication
Volume 3, Issue 2, Article ID 020120, 8 pages
Journal URL: www.src-online.ca
Received August 17, 2011, Accepted November 15, 2011, Published August 15, 2012

Boot, Peter. (2012). Literary Evaluation in Online Communities of Writers and Readers. Scholarly and Research Communication, 3(2): 020120, 8 pp.

© 2012 Peter Boot. This Open Access article is distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc-nd/2.5/ca), which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.