On blogging and the irrelevance of academic peer review in multi-disciplinary fields

In my short blogging career I have been challenged, on more than one occasion, to submit my writing to peer-reviewed academic journals. My response has been to point out that my work undergoes peer review the second I post it online and that interdisciplinary and multi-disciplinary science is not well served by the traditional academic peer review process. The sad reality is that academic peer-reviewed journals are not terribly relevant in the non-academic world. As I will discuss in this blog post, peer-reviewed academic journals serve a useful function in highly-specialized fields but they quickly lose their relevance for broader topics. As a consequence, in some fields peer-reviewed academic journals serve simply as a means by which academics can keep score (for their C.Vs) and to get, or block, promotions. That being said there is a constituency, mostly academic, that remains firmly convinced that peer-reviewed academic journals should form the primary basis for the development of policy in the environmental and climate fields. I hope this post will help correct that misguided impression.

In my opening paragraph I make a pretty bold statement: that the academic peer-reviewed press serves a marginal role in real-world policy decisions. To substantiate my claim let’s start with a few self-evident truths of the peer-reviewed academic press.

  • The peer-reviewed academic press is slow to present results. The publication process often stretches on for months and can go for years. This can leave it well behind the decision curve.
  • The peer-reviewed academic press can be subject to institutional and professional gate-keeping that can alienate stakeholders.
  • Accessing peer-reviewed articles is typically so expensive that most non-acdemics only read the freely-accessible abstracts and don’t delve into the actual details of the articles that are hidden behind paywalls.

Now I am pretty certain that no one will challenge my first two points but some might be surprised by my third. Those people are likely academics who have never really had to consider how much it can cost to keep current with academic publications. Most academics are fairly unaware of the cost of subscription or access fees to academic articles. Consider that access to an article from “Science” may cost you $30 FOR A SINGLE DAY. Given those costs most non-academics simply do the inexpensive thing and skip reading the full papers altogether safe in the knowledge that if a paper is sufficiently important someone else will summarize its contents or a pre-print will be made available.

My major complaint about the peer-reviewed academic press is that it only really works well when the journal covers a very distinct body of knowledge. This is because only then will the peer-reviewers be capable of carrying out an effective peer review. The biggest problem with the peer-review system is that the peer reviewers are limited in number, time and experience and as such normally a paper only sees two or three reviewers. These reviewers are picked not only for their expertise but also on their willingness to serve. Typically what this means is that the job is dominated by the very young academics (who have the enthusiasm and want to build up experience) and the very old academics (who have the time). Mid-career academics often lack the time to do much peer-review thanks to their teaching and administrative responsibilities. Most try and make time but they cannot address every manuscript that comes across their desk.

Let’s be clear here, peer reviewers serve an important, but often misunderstood multi-faceted role in the scientific endeavour. Firstly they are there to confirm that the science is done in an appropriate manner. In this they are conducting a quality assurance/quality control (QA/QC) role within the system. They look at how the work was done to identify errors in methodology: were appropriate statistical tests applied? Was the sample cooled sufficiently…that sort of thing. Peer reviewers will often suggest additional/different analyses to confirm the validity of results. This task is only possible if the peer reviewer is extremely familiar with the field under study.

Peer reviewers also serve an editorial role. They are responsible for suggesting changes to make the work more comprehensible. That is the sort of task that can be accomplished by any good reviewer whether they understand a topic or not.

The final role of the peer reviewer is that of gate-keeper, to identify if the science presented in the paper is novel and interesting enough to warrant space in the journal. A reviewer might say that an article is beautifully written, including all the right tests, but that it is not a good fit for the journal in question. This final role is a legitimate one within the system, but as members of the climate science community can tell you, it also risks problems if one part of the community chooses to block the publication of research from those with whom they differ.

You are probably wondering when I am going to get to my point? Well here it is: once you leave a peer reviewer’s particular area of expertise they cease to provide a reasonable value for money. At that point they simply become a highly-credentialed copy editor. In a field like climate change or renewable energy policy, that is very broad in content/context, it is simply not possible to find reviewers who can do a reasonable job of effectively peer reviewing papers. You don’t believe me? Well let me give you an example.

One of my recent bugbears at this blog has been Dr. Mark Jacobson’s 100% Wind, Water and Sunlight series with the latest article being 100% Clean and Renewable Wind, Water, and Sunlight (WWS) All-Sector Energy Roadmaps for 139 Countries of the World (called 100% WWS World hereafter). This paper discusses almost a dozen renewable energy concepts and provides plans that quite literally span the globe. As a consequence no two or three peer reviewers could conceivably do a comprehensive job of reviewing this paper and a lot of things were simply missed by the reviewers. As an example, most North Americans know little about Scandinavia, imagining it is all mountains and fjords. So when Dr. Jacobson suggested that Finland could store huge amounts of energy in the form of pumped hydro storage that probably sounded pretty reasonable to the reviewers (and frankly to me when I read the paper). It took a Finn to point out that Finland, unlike the rest of Scandinavia, is an exceedingly flat country. Pumped hydro needs abrupt changes in topography to be efficient and is not practical in flatlands. So the 100% WWS World plan fails spectacularly in that respect for its analysis of Finland. Similarly, a look at Wikipedia would confirm to Dr. Jacobson’s reviewers that Canada’s coastline is vast (over 200,00 km) but what the reviewers clearly didn’t recognize is that the majority of that coastline is in the north where weather and ice preclude the construction of off-shore wind turbines. Thus, when the paper calculated how much offshore wind potential was available it missed the fact that much of the area described is not practical for offshore wind energy generation. Similarly, it is clear that the peer reviewers didn’t know that the Pacific Continental Shelf pretty much abuts much of the West Coast of Canada. The sea floor drops precipitously very close to the coast and as such over much of the coast it would not be affordable to install an offshore wind platform as the foundation of such installations would need to be 100s of meters deep.

What this all means is that when someone assures me that Dr. Jacobson’s work is correct and relevant “because it underwent peer review” I take that with a heavy dose of salt. I have read almost a dozen critiques of the work (and authored 4 myself) which challenge topics ranging from how it handles transmission lines to how it deals with individual countries. In this case the peer-review process failed because the topic was simply too broad for any small group of peer reviewers to complete effectively. It is an object lesson in the limitations of the peer review process.

Going back to our main topic, in our modern era a blogger doesn’t need to publish in peer-reviewed academic journals to have an effect. Bloggers can build a following and influence policy simply through the act of consistently producing good, well-referenced and well-argued work. A well-respected blogger, like Steve McIntyre (at Climate Audit), can instantly reach and influence a targeted audience of thousands. Meanwhile a typical, peer-reviewed article can take hundreds of hours wending its way through the publishing process, cost thousands of dollars to produce and is likely to be read completely by no more than 10 people.

Once a blog post is presented online it has the ability to be reviewed by the hundreds of readers who encounter it. Moreover, unlike a published article in a journal, corrections and addendums in blogs can be made pretty much right away. I am quick to fix errors in my blog because that allows me to maintain my credibility. A faulty journal article, on the other hand, is exceedingly hard to fix. Sometimes it seems like journals actually discourage corrections as they don’t “constitute original content” so only the most egregious errors get corrected or papers retracted.

In a similar vein, the peer review process is not a magical or proprietary process limited to the academic press. Think tanks and policy foundations can conduct research which they can subsequently submit for peer review from knowledgeable experts before publication. Thus a study on ocean acidification by Patrick Moore can be published by the Frontier Centre for Public Policy while rightfully claiming that the study was peer-reviewed. Now to be clear I know nothing about the FCPP but given the prominence of this paper the absence of any legitimate complaints/corrections since this report was published leads me to believe that I can take what I read as factual. Happily, given the scope and breadth of the world-wide net I know that if this statement is wrong I will receive many corrections (which I will share if sufficiently relevant) that may change my view of that report.

Going back to the premise of this post. I don’t plan to submit my writing to a peer-reviewed academic journal anytime soon as I do not see any particular value in doing so.  I know that I will get feed-back once I post my thoughts on my blog and my professional status does not hinge on such publications. Most importantly, my blog stats and link hit statistics re-assure me that my thoughts are being widely read and shared which is really all I am looking for with this blog.

Author’s note: a number of readers have questioned the lack of peer-reviewed output from my graduate work. My graduate work was interdisciplinary in nature and involved developing systems to allow data collected by governmental scientists to be evaluated for its reliability and quality; stored in information systems; and made available for subsequent re-use by other researchers. Upon graduation I went immediately to the private sector and saw no real value in writing up my research. The reason for my reluctance was that the output of my research was already being used on a daily basis as a component of the information systems used by our provincial and federal governments and was being widely shared by those governments (including through workshops that I was asked to moderate).

As for the practical applications of my research, well it was almost immediately applied at my current employer and has become a standard in our industry for the last decade. I cannot imagine a better demonstration of the value of my research than the fact that our clients insist that our competitors conduct data analysis in the manner suggested by my research and in doing so our clients have done more to publicize my research than any academic publication might ever have done.

This entry was posted in Climate Change, Environmentalism and Ecomodernism, Fossil Fuel Free Future, Uncategorized. Bookmark the permalink.

15 Responses to On blogging and the irrelevance of academic peer review in multi-disciplinary fields

  1. Great post. The latest revealed failure of peer review is the Stanford ‘Debunking the pause’ paper that made it through peer review into the publication, Climate Change, only to be roundly debunked by Radford Neal.


    The modern publishing age enables independent review and comment of academic and other research without need to revere the credentials of the research authors. Absence of reverence is not disrespect but I think the academy and many researchers have mischaracterised it as such.


  2. Just realised that Mark Jacobson nonsense is Stanford again! Good grief.


  3. Fred says:

    When people come after you with the “well it’s not peer reviewed, you know” it means that they just can’t find any fault with your argument or your data or your analysis or your scientific deduction so they have to resort to trying to raise some doubt that something is not quite right, something is amiss, otherwise the “impartial experts” would approve it.

    It means you are on time & on target.

    Mind the flak, it is just temporary.


  4. Is the Jacobson nonsense really peer reviewed? It doesn’t say so. It seems to consist mainly of large tables of meaningless numbers, mostly invented by the authors, quoted to absurd precision. Canada apparently needs to have 17.74% of its electricity produced by ‘utility PV’.
    The intermittency question is glossed over in sec 6, with no serious attempt to explain how power is to be supplied in winter when the wind isn’t blowing.


  5. Jim Bouldin says:

    I appreciate you penning this–it contains some important thoughts that I agree with, at least one of which I think is even under-stated: the effectiveness of peer review more broadly (not just for multi-disciplinary situations). And regarding the return time on commentary

    However, care needs to be with your overall argument, because at least the peer review system is, well, a system, in place, that attempts to act as a sieve against bad work seeing the light of the day. In the blog world there is no such thing–anybody can write anything they want–and for every technically knowledgeable and conscientious blogger there are ten others who are not so and will say things just to get attention, or whatever. Also, the argument that not receiving any critical commentary on some blog piece means that no strong arguments against it exist–that’s just not the case. It may well be instead that very few people with the knowledge to comment on it with real insight, are even aware of it, or if they are, have the time needed to address it. That’s one of the major problems with blogs–at least something in Nature or PNAS is going to get noticed by people.

    Regarding decisions on whether to publish work done, I can definitely relate to your sentiments there, as I have a mountain of unpublished analyses ranging from forest tree demography to population genetics to dendroclimatology to baseball analysis, that I really should publish but have not, for various reasons. The fact is that it’s a LOT of work to publish things and unless there’s a strong motivating factor, most will not do it, especially after they have been burned a couple times by illegitimate reviews and rejections. Having said that, it’s important to get important information out to the world at large, somehow, for which there is no easy or simple solution. Blogs can probably serve a purpose there, and I’ve used mine for just that purpose when my dendroclimatology methods analyses were rejected by PNAS.


  6. Jim Bouldin says:

    And yes, on just spot checking Moore’s piece on ocean acidification, there are holes big enough to drive a truck through, and based on past comments I’ve seen him make, I definitely do not trust his objectivity. In fact, you really weakened your argument by including that example.


  7. Pingback: Week in review – science edition | Climate Etc.

  8. Pingback: Week in review – science edition |

  9. ybutt says:

    What would be a good alternative to peer-review for multidisciplinary fields, in your view?


  10. Pingback: Weekly Climate and Energy News Roundup #213 | Watts Up With That?

  11. Pingback: Notes from a professional scientist to a “colleague” in academia | A Chemist in Langley

  12. Pingback: A Jacobsonian 100% Wind Water and Sunlight gallop at UCLA | A Chemist in Langley

  13. Pingback: Debunking the Leap Manifesto’s 100% Wind, Water and Sunlight Annual Energy, Health, and Climate Cost Savings | A Chemist in Langley

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s