Traditional journal based scientific peer review works as follows. A researcher does his research and writes his paper. He then submits the paper to the editor of a journal. The editor of the journal then sends the paper to a number (usually two or three) of other researchers in the same field. These researchers then write short reports on the paper outlining what is good or bad about it and usually suggesting improvements, along with a recommendation as to whether the paper should be accepted by the journal. The reports are then forwarded to the author of the paper, who responds to suggested changes and then sends a revised version of the paper to the journal. After possibly several repetitions of this, an accepted paper will eventually be published in the journal.
Referees are supposedly anonymous. However, the author, the editor, and the referees often work in small fields where everybody knows one another, and people’s beliefs, foibles and writing styles are often well known, so this anonymity is often more theoretical than real. The theoretical reason for anonymity – that the referee can say what he pleases without consequences – is not always entirely true. The anonymity is one sided: the referee receives a paper with the name of the author at the top. The name of a famous and influential scientist at the top has an impact. The editor is very powerful, as he gets to select the referees and by choosing referees carefully clearly has influence whether a paper will be published or not. A good editor will choose referees of mixed levels of seniority (referees include everybody from graduate students to senior professors), and (in areas of some dispute) of mixed positions in any argument.
There are various ways in which this process can be corrupted, but (certainly in the field I worked in) this generally did not happen. Publishers of journals made a point of appointing people of integrity as editors. It was in their self-interest to do this, because the long term consequences of not doing so would be a loss of credibility for the journal. The danger, always, is that authors, editors, and referees all end up coming from the same clique, in which such a process can be corrupted.
Another danger is that fields become isolated from each other, and workers in one field do not properly absorb knowledge and techniques from other fields. Many scientists (and non-scientists) for that matter use a great deal of statistics in their work, and do a great deal of computer programming in their work. Often, they will not be experts in either statistics or computer programming. Sometimes they will do good work from a statistical perspective, and write good computer code. On the other hand, if their work is to be published in peer reviewed journals, and the referees for the papers selected by those peer reviewed journals are not experts in statistics or computer science, and use similarly sloppy methods themselves, then poorer quality work can at times be gotten away with (similarly, you should beware of anyone in business or finance who tells you that his “proprietary black box model” tells this, and that he cannot show it to you because it is “proprietary”. Similar situations of sloppy code and statistics are endemic here, too).
The obvious point is that, when relevant, the peers who do the peer review should include statisticians and computer scientists as well as other workers in the precise field as the author of the paper. Science has become very specialised, and specialists in the same field do not talk to experts in other fields nearly often enough. However, the techniques different scientists use are not nearly as specialised as many proponents think they are. With some effort, experts in one field can understand the work of experts in another.
Traditional peer review does not encourage this.
Which is why in many of the most rigorous, competitive fields, in which really good, high quality science is done, traditional peer review has lost much of its relevance.
Some history… It is easy to see the internet as something that was invented and came along in the mid 1990s on the back of the PC revolution of the 1970s and 1980s. Old fogeys sometimes talk about how the internet was really invented in 1969, but it is easy to dismiss the internet as being something experimental and insignificant before that.
To do this, would be wrong. There was an entire world of computers before the PC, and what really happened in the mid 1990s was that some of the technology and culture from that world crossed over into the PC world and mainstream consciousness.
The important thing to understand is that if you were using a Unix computer in the 1980s, it was connected to the internet. People using Unix computers in the 1980s included most university computer scientists, most physicists, many engineers, many mathematicians. By 1985 these people were talking to one another online, which allowed them to exchange work and gossip more easily than had been the case before.
In highly mathematical fields, another extremely influential (related) development was the invention of the computer typesetting language TeX by Donald Knuth and its macro package LaTeX by Leslie Lamport, which facilitated typesetting of mathematical papers. These also became useful and widely available in the early 1980s. Prior to their invention, scientific journals were responsible for typesetting. After, researchers were expected to submit papers already typeset. Everybody preferred this, as you do not want someone who does not understand your mathematical formulas being responsible for typesetting them. Publishing scientific journals became cheaper, as one cost had been outsourced to the authors.
Cheap air travel changed things too. Scientists have attended conferences to talk to one another since the time of Newton and Liebniz, but few non-scientists would realise to what extent a modern scientific career involves flying constantly around the world to attend conferences, visit labs, and talk to your peers.
Imagine you are doing this. You present a summary of a paper you have just submitted to a conference. Having done this, somebody in the audience asks you for more information. You have full paper fully typeset and printed. The journal hasn’t accepted it yet, but the full peer review and publishing process can take years and science moves faster than that. The whole purpose of coming to the conference was to get feedback and ideas from people such as the person who asked you questions, and you want feedback from them as well as the official referees allocated by the journal. If there are other people working on the same thing, and the work you are doing is repeatable and correct, you want credit for doing it first, and circulating the work to other researchers is a great way of ensuring this.
Thus evolved the concept of a preprint of a scientific paper. Typically, you write a paper, typeset it, send it to the journal, and print of lots of copies of the paper to give to other researchers. (The less common expression “postprint” refers to a paper that has been accepted by a scientific journal but not published yet. “pre” in “preprint” refers specifically to “before peer review”).
Except, of course, there is one other thing you do with it. You put it on the internet. You don’t just want feedback from people you meet at conferences. You want feedback from anyone who can give you useful feedback.
In 1991, a chap named Paul Ginsparg, of Los Alamos National Laboratory, decided that all these preprints of physics papers flying around the internet needed a standard repository where they could be stored and easily found. He thus created a system called ArXiv (found here, and pronounced “archive”, the spelling being a weak pun on the name of the Greek letter χ), which allowed physicists to submit preprints of their papers in a location where they could all be found.
Use of ArXiv has been ubiquitous amongst physicists for about 15 years. In the years since, it has also expanded to include papers in astronomy, mathematics, computer science, nonlinear science, quantitative biology and statistics. Its ubiquity varies a little from field to field. One also should not draw any inferences about a specific field based on whether it uses arXiv specifically. There certainly are fields full of scientists I respect enormously (particularly much biomedical science) that do not use it but which have other similar conventions and systems. The question of “Do you have some process like this?” is hugely relevant, however.
When you write a paper, you submit it to a journal, and you also upload it to ArXiv. At that point your priority on the work is established. There are some checks to make sure that uploaded papers have relevance to the field in which they are categorised (and there has inevitably been some controversy as a consequence) but the test is relevance, not correctness, papers that fail it tend to get reclassified rather than rejected outright, and it is much easier to upload a paper to ArXiv than it is to get it published in a peer reviewed journal.
Researchers in these fields to not read peer reviewed journals, because cutting edge papers take too long to reach them. They find things out at conferences and read papers on ArXiv. They find the good papers by paying attention to researchers’ reputations and following recommendations from other researchers. Really interesting or important work will be looked at from researchers outside the field more often than is the case in peer reviewed journals, but this is possibly still a weakness of the process. Papers will be revised in response to this, and revised versions will be uploaded. A great deal more of the scientific process is occurring in public view. If you do science this way, there is relatively little to hide. This is obviously good, if you indeed have nothing to hide.
Papers are still published in peer reviewed journals. It is not unheard of for important papers to only even be published in preprint form, but it is unusual (that said, the most usually quoted example – Grigori Perelman’s proof of the Poincare Conjecture – might well be the most famous mathematical paper of the last decade).
Peer review matters professionally. If you are submitting a Ph.D. thesis and the work in it has already been published in reputable, peer reviewed journals, then your examiners have little work to do. If you are applying for an academic job, or for promotion or tenure, then your publication record in peer reviewed journals is central to the process. However, the peer reviewed journals are a way of keeping score. Amongst physicists at least, they are not where the work is done or how it is communicated.
We have in recent weeks heard calls from various people for science to adopt a model more resembling open source software – one aspect of which is opening access to the evolution of work to more people than a small number of officially appointed referees. The “Many eyes make all bugs shallow” philosophy surely has wider reference than just to software, although when a good portion of the work is software, it’s probably even more relevant.
However, what has been less reported is that in many fields, particularly the most quantitative fields, this model already exists. The physicists got there first, partly because they got the internet a decade before most other fields. However, many others have followed. The question should be, “If not, why not?”
Or, perhaps “Show me the preprints”.