Lior Shamir
National Institute on Aging
Abstract: Conference papers have traditionally been a quick form of research communication and an important source of information for scientists in addition to the standard journal papers. However, in the disciplines of computer science and engineering, a vast majority of the peer-reviewed publications are in the form of conference proceedings, which have become the primary channel of research communication in these disciplines. Although this form of scholarly communication was effective for computer science when it was a young discipline, its several limitations make it less than optimal for a mature and established scientific field. This article reviews the differences between computer science and engineering conference publications and the traditional journals used in other scientific disciplines, discussing the effect of these differences on the scholarly communication in these fields.
Keywords: Computer science; Engineering; Conference publications
Lior Shamir is a Research Fellow in the Laboratory of Genetics, National Institute on Aging (U.S. National Institutes of Health), 251 Bayview Boulevard, Baltimore, MD 21224. Email: shamirl@mail.nih.gov.
While in most scientific disciplines research results are normally reported in the form of peer-reviewed papers published in journals, in some disciplines the common method of reporting scientific results is through peer-reviewed papers presented at conferences and published in conference proceedings. According to CiteSeerx (http://citeseerx.ist.psu.edu), the primary indexing engine in computer and information science literature, 88 of the top 100 sources of peer-reviewed papers with the highest Garfield impact factor (Garfield, 1972) are conferences, while the remaining 12 are journals. Although this list is generated automatically and may contain errors, it reflects the importance of papers published in conference proceedings to these disciplines. In fact, unlike most other scientific disciplines, where oral or poster presentations typically require the submission of an abstract, presenters of oral or poster presentations at computer science and engineering conferences are required to submit a full paper that is peer-reviewed by the conference program committee. Acceptance of papers to conferences is in many cases considered prestigious and is also highly important for promotion and tenure.
The conference proceedings are published as books or CDs distributed to the conference attendees, but the papers are made available shortly after the conference to the members of the associations through fully indexed research databases such as IEEE Xplore and ACM Portal. Independent indexing engines such as CiteSeerx, Google Scholar, and Microsoft Academic Search also make these papers available to those who did not attend the conference. As a result, the accessibility of computer science conference papers to the scientific community is comparable to the accessibility of papers published in archived journals, providing the researchers with an equal opportunity to be read and cited by other scientists. It should be noted, however, that according to CiteSeerx, among the top 100 most-cited computer science and engineering individual articles, merely 22 are papers published in conference proceedings. That is, although conference papers are generally cited more often than papers published in journals, the top cited papers were not published in conference proceedings.
Historically, computer science and engineering started to form as an academic discipline during the ’50s and ’60s (Denning, 2000). Unlike disciplines such as biology, chemistry, or physics, computer science was formed as a scientific discipline when commercial airlines were available, which provided scientists with an opportunity to travel and meet in different countries. As a young discipline it had the opportunity to shape itself in a fashion that was not dependent on traditional scientific disciplines, and conferences provided an effective method of scholarly communication for the relatively small number of scientists engaged in the discipline at the time. The explosive growth of the field in the following decades introduced the need for quick methods of peer-reviewed communication. Although Lawrence Roberts proposed and developed the idea of using ARPANET for research communication (Roberts, 1989), online methods for publishing peer-reviewed work were not available at the time, and conference papers provided an effective form of research communication that was suitable for a fast-growing discipline.
While both journal papers and conference papers are formally considered peer-reviewed, there are substantial differences between them, which affect the peer-reviewed publications as well as the nature of conferences and meetings. One of the primary differences is the window of time reviewers and editors have in which to make their final decision about acceptance or rejection of a manuscript. While in journals the reviewers usually have several weeks to prepare their review, and in most cases (excluding the exception of special issues) the deadline for submitting the review can be extended, referees of conference papers have to review a pool of papers in a relatively short time (typically between three and eight weeks). It can be reasonably assumed that this time constraint affects the amount of attention the referee can dedicate to each paper she evaluates, and thus degrades the quality of the review (Cormode, 2008). Additionally, the conference chairperson is not able to give the personal attention to each paper as would a journal’s associate editor (Parberry, 1994), and in many cases the papers are selected based on numerical scores assigned by the referees (Smith, 1990). This limitation leads to rough criteria for evaluating the scientific work, and many authors often get the impression that the decision regarding acceptance or rejection of a manuscript is random (Mogul & Anderson, 2008). An experiment with a shadow program committee reviewing 209 conference papers showed that the shadow committee accepted just 11 papers of the 27 accepted by the actual program committee (Feldmann, 2005).
The relatively short time frame of the review process for conference papers is also an obstacle to the communication between the authors and the reviewers. When a paper is submitted to a journal, the reviewers have the opportunity to interact with the authors and provide them with their comments and suggested revisions, and the authors have the opportunity to reply. Although this method of communication is slow and limited, it often leads to major improvements in the final accepted manuscript. In conferences, however, the reviewer can score papers or decide whether an individual paper should be accepted, but does not have the opportunity to review a revised version. This problem can lead to the publication of papers of marginal quality that the reviewers did not want to completely reject, and also to the rejection of strong papers that suffer from several weaknesses, which could have been easily explained by the authors in an interactive review process (Mogul & Anderson, 2008). Although some conferences allow a rebuttal, this limited method of interaction between the authors and the reviewers rarely leads to major improvements of the manuscript (Parberry, 1994).
Another major downside of the review process for conference papers is the quality of the referees. When a paper is submitted to a journal, the associate editor has the freedom to choose the reviewers and can therefore select highly knowledgeable scientists in the specific topic of the article at hand, referees whom the editor believes are the most suitable people to review that type of work. In conferences, however, the large number of papers that need to be handled in a relatively short time often limits the chairperson to a fixed pool of reviewers, and therefore the chairperson might need to compromise by choosing referees who are knowledgeable in the field, but perhaps are not considered the leading experts in the specific topic discussed in the paper. This compromise can negatively affect the quality of the review and the standards for rejection or acceptance of a manuscript.
The quality of the review becomes an important issue also in cases of less competitive conferences, where the pool of high-quality papers is limited. Since conference organizers need to attract a sufficient number of attendees, and since their registration fees (of typically $400 to $800) should cover the costs involved in the organization of the conference, in some cases the quality of the papers is compromised. A group of computer science graduate students at MIT, who suspected that some conference committees were willing to accept substandard papers just to collect the registration fees, submitted a meaningless, randomly generated paper to several conferences. The paper was accepted for oral presentation and publication in the proceedings of the WMSCI (World Multiconference on Systemics, Cybernetics, and Informatics) 2005 conference (Ball, 2005). Although this example cannot be considered representative, it can serve as a warning that in some cases substandard manuscripts can be published in fully citable conference proceedings (Ball, 2005).
The fact that most computer science and engineering conferences require papers also inflates the total number of peer-reviewed papers published in each field. For instance, in the computer vision sub-discipline, more than 40 conferences publish proceedings (http://iris.usc.edu/Information/Iris-Conferences.html). Each conference typically publishes between 150 and 400 peer-reviewed papers, which leads to thousands of peer-reviewed papers published every year. Even when considering only the three most prestigious computer vision conferences in 2009 – Conference on Computer Vision and Pattern Recognition (CVPR), International Conference on Computer Vision (ICCV), and International Conference on Image Processing (ICIP) – these conferences alone produced a total of 1,857 peer-reviewed papers. It can be argued that this high number of prestigious peer-reviewed papers may be inflated and does not reliably reflect the true progress made in this sub-discipline in just one year.
While the review process of conference papers has a substantial effect on the type and quality of papers accepted for publication, it can also affect the conference itself as a method of scientific communication. The requirement of submitting a full paper that goes through a peer-review process might introduce an obstacle to junior scientists (e.g., graduate students), who wish to attend the conference and present their work but might find it difficult to compete with papers submitted by established and experienced researchers. Obviously students can attend the conference as listeners, but funding for graduate students for attending conferences is often limited, and in most cases they will not be able to attend unless they present. This is different from many of the largest meetings in biomedicine, such as the American Society for Cell Biology (ASCB) or astrophysics meetings, such as the American Astronomical Society (AAS) or International Astronomical Union (IAU), in which graduate students have the opportunity to present their work, typically in the form of a poster presentation, and meet established scientists in their field.
The fact that an accepted full peer-reviewed paper practically serves as a “ticket” to the conference also introduces a negative for the authors of accepted papers who attend the conference. Scientists who spend their time and research funding to attend a conference often expect to meet as many researchers as possible, to learn about their work and perhaps also to collaborate. In that sense, a smaller number of attendees can make the conference less productive to those who attend, and the program committee in practice filters not only the papers that will be published, but also the scientists that attendees can meet at the conference.
Another downside of peer-reviewed conference papers is that in some cases the most innovative and inspiring ideas cannot be properly expressed in the form of a peer-reviewed paper due to their speculative and visionary nature. Therefore, the requirement for a peer-reviewed paper might keep some of the most creative ideas out of the conference. This introduces a drawback, for one of the purposes of a fruitful conference is to stimulate discussions about the future and vision of the field.
The same problem also applies to reporting on work-in-progress. While one of the purposes of scientific conferences is to discuss the work before it is submitted to a journal, computer science and engineering conferences attempt to become more prestigious and competitive by selecting just a small portion of the submitted papers, and conferences are often ranked by the acceptance rate of the submitted manuscripts. As a result, almost all papers accepted to computer science and engineering conferences report on complete studies. Scientists are not given the opportunity at such conferences to discuss their work and share it with other researchers before it is submitted for publication. This leads to another issue, which is publishing scientific reports in journals after part of the study has already been published in conference proceedings. Publishing agreements for conference papers in many cases allow future publication of the same work in the form of journal papers, but require sufficient revisions and new content to justify republication. For instance, such associations as the Association of Computing Machinery (ACM) and the Institute of Electrical and Electronics Engineers (IEEE), allow submitting conference papers to journals if the paper meets higher standards and justifies journal publication.
While publishing peer-reviewed papers in conference proceedings introduces several downsides, as discussed, it also has several important advantages. For instance, because the review process for a conference paper is rapid, the date of publication is known to the authors before the paper is submitted. This advantage is particularly important in computer science and engineering, where, in contrast, papers submitted to journals are rarely rejected by the editor, and the initial reviews are normally returned to the authors within a period of between three and six months. The time that passes from the day of submission to the day of publication in the journal can be as long as two years, which introduces a significant delay in sharing the results with the scientific community. At conferences, on the other hand, the proceedings must be published before the conference starts, and therefore the publication date is well known to the authors.
Additionally, the peer-review process can set a standard for the quality of the work presented at conferences. It also allows the program committee to select speakers for oral presentations in a more knowledgeable fashion, based on the research and results described in their papers. This can be compared to the selection of oral presentations in other disciplines, in which the program committee makes its decisions based on abstracts. Practically, since the abstracts do not provide detailed information about the research, and since the review of the abstracts is not double-blind, in these conferences the primary criterion for selecting speakers for oral presentations is the reputation of the speaker, and not necessarily the scientific merit of the presented work. The conference papers, whether peer-reviewed or not, also allow the attendees to learn more about the work presented at the conference by looking up the specific paper of interest after attending an oral or poster presentation.
Another advantage of papers published in conference proceedings is that they provide an effective method for communicating scholarly work that might not be comprehensive enough to justify full journal publication, or might be of interest to a limited audience of scientists. Although bulletins and letters are designed for that purpose, conference proceedings provide an additional channel for this type of communication.
Perhaps one of the most important advantages of the prestige of computer science peer-reviewed conference papers is that it allows the attendees to present their most recent work and their best results. This can be compared, for instance, to conferences in life sciences, in which many scientists do not present their most recent results because they feel that other researchers might replicate their work and publish it as a journal paper, which will practically give them full credit for the discovery. This discourages scientists from presenting their best and most recent work, and leads to a situation in which a substantial part of the research presented at life science conferences is work that either has already been published or has been accepted for publication elsewhere.
The approach of publishing scholarly work at computer science and engineering conferences is different from most other disciplines. While the purpose of scientific conferences and meetings is to serve as a forum for scientists to discuss their research and ideas and to share their work with each other, computer science and engineering conferences also serve as a method of publishing peer-reviewed papers. This method of research communication was effective for a young, small, and fast-growing discipline. However, since the number of active computer scientists has increased dramatically, and the current growth of the field is arguably slower than the progression during the ’80s or ’90s, the many downsides of peer-reviewed conference papers have started to become noticeable. It may be time for the maturing discipline to adopt the research communication methods that has been serving other disciplines for hundreds of years.
When the discipline was small, scientists could meet at conferences while also publishing their papers. However, the increasing demand for conferences that can serve as a channel for publishing scholarly work has led to a large number of conferences in each sub-discipline. This results in thousands of papers published every year, while often compromising on the quality of the published work. Since not all papers are accepted, this form of publication also divides the researchers within each sub-discipline between different conferences. In practice, it does not allow all scientists active in a certain sub-discipline to meet.
When the growth of the field was rapid, new ideas were common, and the need to express these ideas justified the publication of peer-reviewed papers at conferences. However, since the progress of the discipline has been slowing down, it is now also required to discuss work-in-progress and exchange visionary ideas before they become suitable for peer-reviewed publication. Although the existing computer science culture strongly supports the popular practice of conferences, it is possible that the academic needs described in this article will start to put increasing pressure on the discipline. In time computer science and engineering conferences may serve a purpose similar to conferences in other disciplines, and journal publications may become increasingly dominant. Signs of this possible trend are evident in emerging publication models in computer science, such as the one adopted by the Very Large Data Bases (VLDB) Endowment (http://www.vldb.org). This new publication model allows researchers in the field of databases to submit their work to the Proceedings of the VLDB Endowment journal, and after a full peer-review process the authors of accepted papers are offered a slot in the VLDB conference.
Consequent to a possible gradual shift from conference to journal publication, conference committees might become less concerned about accepting only a small percentage of the submitted papers. They will work toward increasing the number of attendees and conferences will thus become more accessible to students and junior researchers who wish to discuss their ideas and meet other researchers in the field.
This research was supported entirely by the Intramural Research Program of the U.S. National Institutes of Health (NIH), National Institute on Aging.
Ball, P. (2005). Computer conference welcomes gobbledegook paper. Nature, 434, 946.
CiteSeerx. (n.d.). Scientific literature digital library and search engine [website]. URL: http://citeseerx.ist.psu.edu .
Computer image analysis, computer vision conferences, current and future [webpage]. (n.d.). URL: http://iris.usc.edu/Information/Iris-Conferences.html .
Cormode, G. (2008). How NOT to review a paper: The tools and techniques of the adversarial reviewer. ACM SIGMOD Record, 37(4).
Denning, P. (2000). Computer science: The discipline. In A. Ralston and D. Hemmendinger (Eds.), Encyclopedia of computer science (pp. 405-419). Fairfax, VA: George Mason University.
Feldmann, A. (2005). Experiences from the Sigcomm 2005 European Shadow PC Experiment. SIGCOMM Computer Communication Review, 35(3), 97-102.
Garfield, E. (1972). Citation analysis as a tool in journal editing. Science, 178, 471-479.
Mogul, J. C., & Anderson, T. (2008). Open issues in organizing computer systems conferences. ACM SIGCOMM Computer Communication Review, 38(3), 93-98.
Parberry, I. (1994). A guide for new referees in theoretical computer science. Information and Computation, 112, 96-116.
Roberts L. (1989) interview by Arthur L. Norberg, (tape recording), San Mateo, California.
Smith, A. J. (1990). The task of the referee. IEEE Computer, 23, 65-71.
Very Large Data Bases (VLDB) Endowment Inc. (n.d.). URL: http://www.vldb.org .
CCSP Press
Scholarly and Research Communication
Volume 1, Issue 2, Article ID 020302, 7 pages
Journal URL: www.src-online.ca
Received February 16, 2010, Accepted April 18, 2010, Published June 8, 2010
Shamir, Lior. (2010). The Effect of Conference Proceedings on the Scholarly Communication
in Computer Science and Engineering. Scholarly and Research Communication, 1(2): 020302, 7 pp.
© 2010 Lior Shamir. This Open Access article is distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc-nd/2.5/ca), which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.