FWD FYI
see website for author and reference info.

--------------------------------------------------
Editorial

Fourth International Congress on Peer Review in Biomedical Publication

In this issue of THE JOURNAL, we publish articles selected from those
manuscripts submitted to JAMA following presentation at the Fourth
International Congress on Peer Review in Biomedical Publication, held in
Barcelona, Spain, September 14 to 16, 2001.

On what was, in Spain, the afternoon of September 11, those of us from
JAMA, who had arrived in Barcelona a few days early to make final
preparations for the Congress, sat in our hotel rooms, staring with horror
at images of the collapse of the World Trade Center towers and the
devastation at the Pentagon.

Although our careful planning would probably go for naught, it was obvious,
since terrorism works best when it most disrupts, that the Congress had to
proceed.

Yet it soon became apparent that all those from the Americas who were not
already in Europe would be at best delayed, and, as events proved, unable
to get to Barcelona at all.

This meant that the Congress schedule had to be reordered several times a
day, as we continuously received word from individuals all over the world,
many stuck indefinitely at airports.

In the end, 135 of the original 410 who had registered were unable to get
to Barcelona, almost all of them from the United States and Canada.

However, 40 of 43 presentations from the podium were given as planned,
although sometimes by a different presenter on a different day, and 58 of
65 posters were presented.

On the first full day, at noon, the attendees joined the rest of Europe in
standing in silence for 3 minutes in honor of the dead and wounded.

Thereafter, everyone made a determined effort to remain positive and to
hold what turned out to be vigorous scientific discussions.

As in previous Congresses, because the aim was simply to present research
and discuss it, not draw up position papers nor decide on consensus, equal
time was allotted to discussion as to presentation.

In the past, this format has had the effect of drawing people together and
furthering debate, and so it proved in Barcelona.

These Congresses on peer review began as a response to a call, in 1983,
from Bailar and Patterson [1] for studies to be done on editorial peer review.

There was an abundance of published opinion on peer review, but few
empirical studies, and it was obvious that we knew little about one of the
central processes of science.

Shortly after, Lock, a pioneer in this and other fields, published his
important book, A Difficult Balance, [2] on what we knew on peer review.

In 1986, we at JAMA invited people to come to a Congress, to be held in 3
years' time, to present the results of their as yet nonexistent research
into peer review. [3]

To our surprise and relief, 50 abstracts were submitted and the first
Congress was held in 1989 in Chicago.

It generated a good deal of enthusiasm, and was followed by another in 1993
in Chicago, a third in 1997 in Prague, and the fourth in 2001 in Barcelona.

How successful have these meetings been?

If we measure success by the number of abstracts, and by the 4 theme issues
containing articles resulting from the Congresses,[4-6] we can document
success by increasing numbers (Figure 1).

It seems clear that there were few articles being published on peer review
before we started the initiative in 1986, and that now there are about 170
to 200 per year.

Indeed, the increasing interest, whether it is due to the Congresses, has
clearly extended beyond them because the citations in MEDLINE for
non-Congress years is up and remains steady.

Figure 1 suggests that a new area of science has been created, and although
this research is usually unfunded and performed by individuals with other
professional interests, it is gradually beginning to provide a description
of editorial peer review and other editorial processes, and some of their
consequences.

Once again, in this issue of THE JOURNAL, we publish studies that fail to
show any dramatic effect, let alone improvement, brought about by editorial
peer review. [7]

Yet, despite this, it continues to be the experience of editors that peer
review is extraordinarily effective, sometimes in saving the reputations of
the authors.

Why? It makes good sense that editors would want to enlist the services of
those more expert in a particular subject than themselves.

And there are powerful reasons why editors might wish to spread the
responsibility for unfavorable decisions about manuscripts.

But there is another important factor.

Peer review represents a crucial democratization of the editorial process,
incorporating and educating large numbers of the scientific community, and
lessening the impression that editorial decisions are arbitrary. [8]

But these are impressions.

16 years after the initiative started, we find ourselves in the peculiar
position of believing still more in the virtues of peer review, a system we
know to be "time-consuming, complex, expensive and ... prone to abuse," [9]
while we acknowledge that the scientific evidence for its value is meager.

Indeed, if the entire peer-review system did not exist but were now to be
proposed as a new invention, it would be hard to convince editors looking
at the evidence to go through the trouble and expense.

This dissonance suggests that we are using the wrong tools to study the
wrong factors.

In the editorial accompanying the last theme issue on peer review, [9] I
noted that the vast majority of studies presented at the Congresses had
examined the mechanism or the effects of peer review, rather than the
cognitive processes involved.

This remains true today.

For example, several quantitative studies have shown the existence of
various biases, and many important articles published in the previous peer
review issues have measured the extent of such biases.

Such findings hint at where the process may have gone wrong, but they do
not necessarily explain why.

Quantitative research is only the first step to understanding the deeper
reasons for these biases on the part of authors, editors, and reviewers.

Horton [10] is one of those who has begun to carry out important
qualitative investigation.

In a small study, which Horton hastens to stress is provisional, he
questioned all the contributors to a number of studies published in The
Lancet, of which he is editor.

The replies to his questions showed so much variation that Horton concluded
"a research paper rarely represents the full range of opinions of those
scientists whose work it claims to report."

To improve matters, we need rigorously conducted qualitative studies and
cognitive research to examine the reasons for the anomalies in the process,
and these will require adequate funding.

I would have hoped by now that more of us researchers would be
investigating the thousands of specialty journals, rather than
concentrating on "the big 5" general medical journals, although one
advantage of that has been that JAMA can continue to publish studies, as we
do in this issue, auditing our progress.

These tend to show that, while we may have improved our editorial habits,
we still have a long way to go.

In the 1986 editorial announcing the first peer review Congress, I noted
the appalling standards then prevalent despite the existence of peer review
[3]:

"One trouble is that despite this system, anyone who reads journals widely
and critically is forced to realize that there are scarcely any bars to
eventual publication. There seems to be no study too fragmented, no
hypothesis too trivial, no literature citation too biased or too
egotistical, no design too warped, no methodology too bungled, no
presentation of results too inaccurate, too obscure, and too contradictory,
no analysis too self-serving, no argument too circular, no conclusions too
trifling or too unjustified, and no grammar and syntax too offensive for a
paper to end up in print."

In the last 16 years, efforts to systematize reviews and improve the
reporting of trials and meta-analyses have borne considerable fruit.

But an unbiased reader, roaming at random through a medical library, would
find in abundance all the problems I described in 1986.

This obvious fact makes it all the more surprising that, although many
hundreds of editors have attended these Congresses, they represent a
fraction of the thousands out there, who pass up participating in the only
meeting devoted to the presentation of research into their craft.

Despite the sad coincidence of its opening 2 days after the World Trade
Center catastrophe, we felt, and a survey conducted at the time confirmed,
that the Congress was a success.

For this I am deeply grateful to my colleagues at JAMA, led by Annette
Flanagin, who worked around the clock to make it happen, to Richard Smith
and his colleagues at BMJ, and to all those who came and brought to the
Congress their intelligence, their humor, and their compassion.

I am especially grateful to those who tried to come but were prevented by
events, yet still sent us their manuscripts.

http://jama.ama-assn.org/issues/v287n21/ffull/jed20028.html


janet paterson: an akinetic rigid subtype, albeit primarily perky, parky
pd: 55/41/37 cd: 55/44/43 tel: 613 256 8340 email: [log in to unmask]
smail: 375 Country Street, Almonte, Ontario, Canada, K0A 1A0
a new voice website: http://www.geocities.com/janet313/

----------------------------------------------------------------------
To sign-off Parkinsn send a message to: mailto:[log in to unmask]
In the body of the message put: signoff parkinsn