 |
 |
FWD FYI see website for author and reference info. -------------------------------------------------- Editorial Fourth International Congress on Peer Review in Biomedical Publication In this issue of THE JOURNAL, we publish articles selected from those manuscripts submitted to JAMA following presentation at the Fourth International Congress on Peer Review in Biomedical Publication, held in Barcelona, Spain, September 14 to 16, 2001. On what was, in Spain, the afternoon of September 11, those of us from JAMA, who had arrived in Barcelona a few days early to make final preparations for the Congress, sat in our hotel rooms, staring with horror at images of the collapse of the World Trade Center towers and the devastation at the Pentagon. Although our careful planning would probably go for naught, it was obvious, since terrorism works best when it most disrupts, that the Congress had to proceed. Yet it soon became apparent that all those from the Americas who were not already in Europe would be at best delayed, and, as events proved, unable to get to Barcelona at all. This meant that the Congress schedule had to be reordered several times a day, as we continuously received word from individuals all over the world, many stuck indefinitely at airports. In the end, 135 of the original 410 who had registered were unable to get to Barcelona, almost all of them from the United States and Canada. However, 40 of 43 presentations from the podium were given as planned, although sometimes by a different presenter on a different day, and 58 of 65 posters were presented. On the first full day, at noon, the attendees joined the rest of Europe in standing in silence for 3 minutes in honor of the dead and wounded. Thereafter, everyone made a determined effort to remain positive and to hold what turned out to be vigorous scientific discussions. As in previous Congresses, because the aim was simply to present research and discuss it, not draw up position papers nor decide on consensus, equal time was allotted to discussion as to presentation. In the past, this format has had the effect of drawing people together and furthering debate, and so it proved in Barcelona. These Congresses on peer review began as a response to a call, in 1983, from Bailar and Patterson [1] for studies to be done on editorial peer review. There was an abundance of published opinion on peer review, but few empirical studies, and it was obvious that we knew little about one of the central processes of science. Shortly after, Lock, a pioneer in this and other fields, published his important book, A Difficult Balance, [2] on what we knew on peer review. In 1986, we at JAMA invited people to come to a Congress, to be held in 3 years' time, to present the results of their as yet nonexistent research into peer review. [3] To our surprise and relief, 50 abstracts were submitted and the first Congress was held in 1989 in Chicago. It generated a good deal of enthusiasm, and was followed by another in 1993 in Chicago, a third in 1997 in Prague, and the fourth in 2001 in Barcelona. How successful have these meetings been? If we measure success by the number of abstracts, and by the 4 theme issues containing articles resulting from the Congresses,[4-6] we can document success by increasing numbers (Figure 1). It seems clear that there were few articles being published on peer review before we started the initiative in 1986, and that now there are about 170 to 200 per year. Indeed, the increasing interest, whether it is due to the Congresses, has clearly extended beyond them because the citations in MEDLINE for non-Congress years is up and remains steady. Figure 1 suggests that a new area of science has been created, and although this research is usually unfunded and performed by individuals with other professional interests, it is gradually beginning to provide a description of editorial peer review and other editorial processes, and some of their consequences. Once again, in this issue of THE JOURNAL, we publish studies that fail to show any dramatic effect, let alone improvement, brought about by editorial peer review. [7] Yet, despite this, it continues to be the experience of editors that peer review is extraordinarily effective, sometimes in saving the reputations of the authors. Why? It makes good sense that editors would want to enlist the services of those more expert in a particular subject than themselves. And there are powerful reasons why editors might wish to spread the responsibility for unfavorable decisions about manuscripts. But there is another important factor. Peer review represents a crucial democratization of the editorial process, incorporating and educating large numbers of the scientific community, and lessening the impression that editorial decisions are arbitrary. [8] But these are impressions. 16 years after the initiative started, we find ourselves in the peculiar position of believing still more in the virtues of peer review, a system we know to be "time-consuming, complex, expensive and ... prone to abuse," [9] while we acknowledge that the scientific evidence for its value is meager. Indeed, if the entire peer-review system did not exist but were now to be proposed as a new invention, it would be hard to convince editors looking at the evidence to go through the trouble and expense. This dissonance suggests that we are using the wrong tools to study the wrong factors. In the editorial accompanying the last theme issue on peer review, [9] I noted that the vast majority of studies presented at the Congresses had examined the mechanism or the effects of peer review, rather than the cognitive processes involved. This remains true today. For example, several quantitative studies have shown the existence of various biases, and many important articles published in the previous peer review issues have measured the extent of such biases. Such findings hint at where the process may have gone wrong, but they do not necessarily explain why. Quantitative research is only the first step to understanding the deeper reasons for these biases on the part of authors, editors, and reviewers. Horton [10] is one of those who has begun to carry out important qualitative investigation. In a small study, which Horton hastens to stress is provisional, he questioned all the contributors to a number of studies published in The Lancet, of which he is editor. The replies to his questions showed so much variation that Horton concluded "a research paper rarely represents the full range of opinions of those scientists whose work it claims to report." To improve matters, we need rigorously conducted qualitative studies and cognitive research to examine the reasons for the anomalies in the process, and these will require adequate funding. I would have hoped by now that more of us researchers would be investigating the thousands of specialty journals, rather than concentrating on "the big 5" general medical journals, although one advantage of that has been that JAMA can continue to publish studies, as we do in this issue, auditing our progress. These tend to show that, while we may have improved our editorial habits, we still have a long way to go. In the 1986 editorial announcing the first peer review Congress, I noted the appalling standards then prevalent despite the existence of peer review [3]: "One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print." In the last 16 years, efforts to systematize reviews and improve the reporting of trials and meta-analyses have borne considerable fruit. But an unbiased reader, roaming at random through a medical library, would find in abundance all the problems I described in 1986. This obvious fact makes it all the more surprising that, although many hundreds of editors have attended these Congresses, they represent a fraction of the thousands out there, who pass up participating in the only meeting devoted to the presentation of research into their craft. Despite the sad coincidence of its opening 2 days after the World Trade Center catastrophe, we felt, and a survey conducted at the time confirmed, that the Congress was a success. For this I am deeply grateful to my colleagues at JAMA, led by Annette Flanagin, who worked around the clock to make it happen, to Richard Smith and his colleagues at BMJ, and to all those who came and brought to the Congress their intelligence, their humor, and their compassion. I am especially grateful to those who tried to come but were prevented by events, yet still sent us their manuscripts. http://jama.ama-assn.org/issues/v287n21/ffull/jed20028.html janet paterson: an akinetic rigid subtype, albeit primarily perky, parky pd: 55/41/37 cd: 55/44/43 tel: 613 256 8340 email: [log in to unmask] smail: 375 Country Street, Almonte, Ontario, Canada, K0A 1A0 a new voice website: http://www.geocities.com/janet313/ ---------------------------------------------------------------------- To sign-off Parkinsn send a message to: mailto:[log in to unmask] In the body of the message put: signoff parkinsn