| Most Recent |
|---|
| Peer review? What peer review? |
|
|
| Written by John McLean | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Thursday, 06 September 2007 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Page 1 of 2
For the Full Report in PDF Form, please click here. Summary for Policy MakersINTRODUCTION The IPCC would have us believe that its reports are diligently reviewed by many hundreds of scientists and that these reviewers endorse the contents of the report. An analysis of the reviewers' comments for the scientific assessment report by Working Group I show a very different and very worrying story. The comments for Working Group I are the only set of reviewers' comments to be made available to the public, and only then thanks to use of US Freedom of Information laws rather than a willingness on the part of the IPCC to allow people to examine the material. Surely all people should be able to examine the involvement and thinking of their governments and the reviewers from their own countries because it is the people who will most certainly bear the economic and political costs of any resultant actions. Perhaps the IPCC is simply worried that exposing the reviewers' comments and the responses to those comments to close scrutiny will reveal the delusions of thoroughness and widespread consensus. REVIEWERS AND COMMENTS A total of 308 reviewers commented on the Second Revision, which was the penultimate draft, but only 32 reviewers commented on more than three chapters and just five reviewers commented on all 11 chapters. At the other end of the scale, 143 reviewers (46%) commented on just one chapter and a further 71 (23%) on two. This would be acceptable if they had provided numerous detailed comments, but 53 of these 214 reviewers made fewer then five comments and 28 reviewers made fewer than three comments. The number of reviewers who made just one comment on a chapter varied between 12.6% and 32% (i.e. almost one-third) of the reviewers commenting on that chapter. For four chapters, fewer than six comments were made by more than 50% of the reviewers who commented. For another four chapters, the figure was between 40% and 50%. Reviewers' comments come in all forms. Many are simple corrections to spelling and grammar, others point out inconsistencies, some ask for a change of wording, many ask for expressions of less certainty, others suggest extensive references that should be included. A minority requests a change of wording and provides extensive reference material to support their statements. One response to a reviewer’s comment is worth mention - "Rejected. McKitrick and Michaels (2004) is full of errors. There are many more papers in support of the statement than against it." - But this erroneously implies that a consensus of papers determines what will be included, which of course is not very different to claiming a consensus determines a scientific truth. Many reviewer comments appear to be rejected with little or no justification for doing so. In particular there appears a disturbing pattern of rejecting reviewers' citations of references by claiming that a greater number of papers say otherwise but then referring to just one paper to dispute the comments of other reviewers. Rejecting references to papers that challenge or weaken claims of serious man-made interference with climate serve to create from whole cloth a contrived, false “consensus.” At other times changes were made, but simply resulted in new wording which imply a certainty or emphasis very similar to the wording that the reviewer complained about. The reviewers appear to have had varying success at modifying the emphasis of some paragraphs but one must wonder what the report would have been like if the reviewers had not commented at all. It is polestar clear that the IPCC-appointed chapter editors believed that their say was final in regard to the certainty of statements and that theirs was the only correct interpretation of the cited material. For many reviewers who could provide logical refutations, either with or without specifying references, the entire process was an exercise in frustration. The notion of hundreds of experts diligently poring over all chapters of the report and providing extensive feedback by way of peer review to the editing teams is here demonstrated to be an illusion. The true picture is there were some 64 reviewers for each chapter, of whom half made very few comments. Most comments were minor drafting amendments. GOVERNMENT INVOLVEMENT Almost all governments see climate change as a major political issue, but government reviewers' comments suggest otherwise. These comments were logged as being from the "Govt. of (name)", but the 11 chapters of the Second Revision show that the names of only 22 national governments (plus the European Commission) were recorded. By contrast, the USA and Australia, both non-signatories to the Kyoto Agreement, made the second-highest and third-highest numbers of comments respectively, eclipsed only by an individual reviewer. Both countries commented on all 11 chapters and show a degree of interest not reflected in the submissions of any other government. Large regions of the globe made few, if any, comments under government auspices - Eastern Europe, Russia, the South American continent, the Middle-east and a large part of south-east Asia. Not a single comment came from any African country. Likewise, countries which have been vocal in their claims that they will suffer most from sea-level rises, namely the Maldives, Tuvalu and Bangladesh, failed to comment on any chapters of the report. Denmark administers Greenland, which is supposedly suffering from the widespread recession of glaciers, but no comments were made on behalf of the Danish government. Switzerland is seeing the recession of glaciers to their mediaeval extents and a receding snow-line, and incidentally is the home of the IPCC, but likewise had no comments. Government reviewers commented on the Second Revision and on the Final Draft of the Summary for Policymakers. Because the IPCC required the underlying science chapters to conform to the Summary for Policymakers, the publication of those chapters was delayed until after the publication of the Summary for Policymakers. The correct sequence would have been the other way about: settle the science first, and then summarize it. The governments of 16 countries plus the European Commission reviewed the Second Revision and made 639 comments. The governments of 26 countries plus the European Commission made 931 comments for the final draft of the Summary for Policymakers, even though the document had already been finalized. More detailed analysis 1 - Some governments are seen to completely accept the hypothesis of anthropogenic global warming while others express greater caution. On page 16 is a table listing the number of comments where the reviewer is designated as "Govt. of"... to provide an indication of the diligence of these reviewers and, one might surmise, the interest shown by the respective governments. (This data is a subset of that dealing with all comments.) 2 - Other reviewers may also be government representatives but are not indicated as such. 3 - The European Commission was included by virtue of its quasi-governmental position. 4 - In several instances the wording of a "government" comment duplicates the comments of a reviewer who is indicated to be an individual. This not only distorts the analysis but suggests a process that was not diligently carried out. 5 - Only 23 "government reviewers" are indicated, which makes one wonder about the commitment of the governments of the other 140+ countries. Just 6 non-English speaking countries outside Europe had designated government reviewer’s comment. 6 - The European Commission, western and northern European countries made up 12 of the 23 countries with comments by government reviewers and these countries made 749 comments of the total 2010. The USA made 689 comments alone. 7 - Fourteen governments commented on fewer than 5 chapters and only Australia and the USA on all 11. Eleven countries made a total of fewer than 15 comments for the entire report (i.e. all 11 chapters). 8 - Significant regions of the Earth had no or very few reviewers who were designated as government reviewers. These figures apply to the Second Revision of chapters 1 to 11:
IPCC’S REJECTIONS OF REVIEWERS’ COMMENTS The IPCC’s editors could – and often did – reject the peer-reviewers’ comments, a reversal of the normal practice in scientific peer-review. Analysis of the extent of the editors’ refusal to accept criticism is difficult because the expressions of rejection come in many forms, some were partial and others were rendered otiose by the rewriting, restructuring or deletion of sections of text. A simple analysis based on the occurrence of three key words - "rejected", "reject" and "disagree" - underestimates the total number of rejections. Even so, this analysis reveals that the number of peer-reviewers’ comments that were rejected by the IPCC climate-templars averaged 25% (min. 9.5%, max 58.1%) of all comments on the Second Revision. The striking feature of most rejections is their dubious nature. Some were banal. Others showed inconsistencies with other comments. Peer-reviewers had to justify the textual amendments which they were putting forward, but the responding editors were under no corresponding obligation to justify their rejections of the reviewers’ proposals. One reviewer said that "best estimate" should more correctly be "most recent estimate" but the editors changed the text to "current best estimate". Reviewers were sometimes flatly told they were wrong, but no reasons or incontrovertible references were provided. Another reviewer said that one heat wave did not make a trend, but the editors rejected this by claiming they had used a particular heat wave as an example. This mistreatment of the reviewer’s scientifically-appropriate amendment allowed the passage complained of to be taken out of context, so that the heat wave in question has been widely – though erroneously – interpreted as having been caused by anthropogenic “global warming”. This mistreatment of the reviewer’s scientifically-appropriate amendment allowed the passage complained of to be taken out of context, so that the heat wave in question has been widely – though erroneously – interpreted as having been caused by anthropogenic “global warming”.In several instances, reviewers invited the IPCC to express its conclusions with less certainty, and provided evidence in support of more caution given the uncertainties inherent in climate science. In almost every such instance, the IPCC’s reviewers flatly rejected the reviewers’ suggested moderations of its conclusions. Some comments were rejected on the ground that there was not enough space. Given the unconstrained length and supposed importance of the IPCC’s assessment report, this ground of rejection is not compelling. Reviewers would cite references in the learned journals challenging the IPCC’s conclusions, but in almost every instance they were told that a greater number of references supported an alternative argument. The correct approach, at the very least, would have been to insert in the assessment report a mention of the references that challenged the IPCC’s conclusion. Reviewers who made brief proposed amendments would often be brushed off by being told of just one paper that contradicted the suggested amendment. In at least one response the IPCC’s editors made reference to a document that had not been subjected to peer-review at all. While the editors sometimes accepted simple corrections and tolerated requests for improved clarity, they routinely resisted any serious challenge to the report’s assertions, interpretations and conclusions. This effects peer-review through a looking glass. REJECTION OF AMENDMENTS ON ATTRIBUTION OF CLIMATE CHANGE In Chapter 9, the key science chapter, the IPCC concludes that "it is very highly likely that greenhouse gas forcing has been the dominant cause of the observed global warming over the last 50 years". The IPCC leads us to believe that this statement is very much supported by the majority of reviewers. However, only 62 of the IPCC’s 308 reviewers commented on this chapter at all. Nineteen reviewers made just one comment and 18 made between two and five comments. Just 10 reviewers made more than 20 comments on this, the most important chapter of the entire report. A total of 1158 comments were made. One reviewer made almost half of these, but almost all his suggested amendments were curtly rejected. The government of the USA made 113 comments; almost 10% of the total, but 32 of its comments duplicate those made by an individual reviewer. In these circumstances any review which casts doubt about assertions based on or related to a human influence on climate will be just what many reviewers found it to be – frustrating and futile.As with other chapters, simple corrections, requests for clarifications or refinements to the text which did not challenge the IPCC’s conclusions are generally treated favourably, but comments which dispute the IPCC’s claims or their certainty are treated with far less indulgence. In particular, comments which draw attention to natural climate forces (e.g. El Nino influences, or the natural “blocking high” that triggered the 2003 European heat wave) are abruptly rejected. The pretext for some of the rejections was the citation of previous IPCC reports which themselves were inadequately reviewed, and were not subjected to the rigorous peer-review that is customary before a scientific paper can be published in the learned journals. Keep in mind, previous reports were (a) not reviewed in the same manner as scientific papers and (b) were the result of similar dubious processes as in the current report. In many instances the IPCC’s editors responded to comments by saying that the point had been discussed in some other chapter. It is difficult to quantify the extent of the reviewers’ support for the IPCC’s conclusions in the chapter on attribution of climate change. Given the number of reviewers who made very few comments, the duplication of comments and the number of minor corrections, it appears likely that less than 40 of the IPCC’s 308 reviewers were generally supportive of the hypothesis. It is not true, therefore, that hundreds of scientists endorsed the IPCC’s findings, still less that thousands did so. The IPCC’s reports, then, are not peer reviewed in the sense that is commonly understood. The editors, rather than accepting genuine and often well-referenced criticisms of the IPCC’s conclusions, have instead tended simply to reject most substantial criticisms. Given the number of reviewers who made very few comments, the duplication of comments and the number of minor corrections, it appears likely that less than 40 of the IPCC’s 308 reviewers were generally supportive of the hypothesis. It is not true, therefore, that hundreds of scientists endorsed the IPCC’s findings, still less that thousands did so. General Conclusions Three conclusions follow. First, the IPCC is merely presenting what it regards as a consensus among published scientific papers – in effect, a giant review article rather than original research. Secondly, in order to produce a paper on some aspect of climatology a researcher needs funding. In the current environment that funding is very obviously directed towards studies which assert that the human influence on climate is substantial. It should be no surprise, therefore, that the number of papers adhering to what has become a “party line” can be presented – rightly or wrongly – as a “consensus”. Thirdly, the dominance of research presupposing a human influence also means that the IPCC editing teams are likely to consist of people predisposed to view the situation in that light. In these circumstances any review which casts doubt about assertions based on or related to a human influence on climate will be just what many reviewers found it to be – frustrating and futile.
Abstract The Intergovernmental Panel on Climate Change (IPCC) gives the impression that its Fourth Assessment Report (4AR) was thoroughly and diligently reviewed and the statements contained in the report were endorsed by a very high percentage of reviewers. Part 1 – Introduction
The Intergovernmental Panel on Climate Change (IPCC) gives the impression that hundreds of scientists thoroughly and diligently peer-reviewed its Fourth Assessment Report on climate change. However, this analysis of reviewers' comments on the science chapters by Working Group I which formed the core of the Report casts some doubt on how many reviewers endorsed the IPCC’s key notion that we are very likely (90% to 95%) to have caused more than half of the 0.4C observed warming over the past 50 years. On previous occasions the IPCC did not release reviewers' comments. It is only thanks to the Freedom of Information legislation in the USA and the efforts of Steve McIntyre of www.climateaudit.org that the reviewers' comments have been made available here.[1] It is somewhat worrying that the public has not been permitted to examine the comments for previous IPCC reports or for the other working groups which contributed to the Fourth Assessment Report, especially when the global population will be expected to bear the heavy cost of actions based on the contents of the report. People should be allowed to see comments made in the name of their governments. The review process must be exposed to independent scrutiny, or there will be no verifiable evidence that the IPCC’s claimed “consensus” really exists. As will be seen, different numbers of reviewers commented on each chapter, and this is very likely due to the different subject matter. Chapters and their titles are therefore listed in table 1
Table 1 - Chapter numbers and titles for IPCC 4AR WG I report
Part 2 - Number of Reviewers and Comments
A total of 308 reviewers[2] commented on chapters of Second Order Revision (SOR), i.e. the penultimate draft, but only 32 reviewers commented on more than 3 chapters and just 5 on all 11 chapters (table 2 and figure 1). At the other end of the scale, 143 reviewers (46%) commented on just one chapter and a further 71 (23%) on two. This would be fine if they were experts and provided numerous detailed comments but 53 of these 214 reviewers (25%) made fewer than 5 comments and 28 of them made fewer than 3. This raises the question of why they bothered to review any chapters and the question of whether they examined other chapters but had nothing to say.
Table 2 - Number of chapters commented on by reviewers
Figure 1 - Graphical representation of table 2 As noted above, the chapters of the Second Order Revision were subjected to attention by different numbers of reviewers. One hundred reviewers examined chapter 2, which dealt with changes to the atmosphere, but just 34 examined chapter 4, which discussed changes to snow, ice and frozen ground.(table 3)
Table 3 - No. of reviewers who commented on each chapter of the Second Order Revisions
Reviewers commented on the chapter as a whole and then on each paragraph of the draft in question. Most reviewers' comments fall into one of the following categories - praise The appendix to this document contains some sample comments both with responses and without. Those responses come from the editorial team for each chapter and naturally reflect the acceptance or rejection of comments as well as a host of other possible situation (eg. "text has been rewritten"). One response noted in passing was "Rejected. McKitrick and Michaels (2004) is full of errors. There are many more papers in support of the statement than against it." This indicates that a consensus of papers is a major determinant of the content of the report. In some cases comments attributed to an individual also appear under the name of a national government but the extraction of these duplicated comments would call for subjective judgment and has not been attempted here. Not surprisingly the number of comments by each reviewer varies greatly between chapters (figure 2). Figure 2 - No of reviewers making various numbers of comments The number of reviewers who made just 1 comment on a chapter varied between 12.6% and 32% (i.e. almost one-third) of the reviewers that commented on that chapter. For 4 chapters fewer than 6 comments were made by more than 50% of the reviewers that did comment and for another 4 chapters the figure was between 40% and 50% (figure 3). Figure 3 - Percentages of reviewers making few (<6) comments and many comments The number of comments per reviewer per chapter varies greatly but by simple addition we can see how many comments each reviewer made and we can gain an indication of the probable distribution of the effort put into the task of reviewing these chapters (Figure 4). Figure 4 - Total number of comments made by reviewers Forget any illusion of hundreds of experts diligently poring over all chapters of the report and providing extensive feedback to the editing teams. The true picture is closer to 64 reviewers for any one chapter, with about half of those not commenting on any other chapter and one quarter commenting on just one other. On top of that, about half of those reviewing any one chapter made very few comments. Part 3 - The contribution of government reviewers
Almost all governments see climate change as a major political issue but comments assigned to government reviewers (and denoted as "Govt. of (name)") suggest otherwise. The review of the 11 chapters of the SOR of WG I show the names of only 22 national governments, plus the "European Commission" which was somehow granted the status of a government. Surely the governments of the remaining 140+ countries are not as disinterested in climate change as these figures suggest. It is possible that some individually named reviewers were working on behalf of governments but it is impossible to determine them from the information given. Conversely there are clear instances of identical comments being attributed both to an individual and to a national government, which indicates that certain processes were not diligently carried out. From the names of the reviewers it appears that there was no review of the WG I report by a large proportion of the globe. Of the 22 named governments 11 are western and northern European countries. The Czech Republic and Hungary were the only east European countries reported. No review was undertaken by Russia and its former states, nor from any country in Africa or the Middle-east. From South America only the governments of Brazil and Chile reviewed to the report and from Asia China, India, Japan, Korea and Thailand. The three countries not mentioned thus far were Australia, Canada and the United States of America. Countries that claim that they will suffer most from rising sea level, namely Maldives, Tuvalu and Bangladesh, had no government reviewers comment on any chapters of the report. Denmark administers Greenland, which is supposedly suffering from a retraction of glaciers, but no review appears to have been undertaken on behalf of the Danish government. Switzerland is seeing the retraction of glaciers and a rising snow line - and incidentally is the home of the IPCC - but likewise made no comments under government auspices. As with the individual reviewers, the extent of the review by government representatives varies enormously. The number of governments whose representatives reviewed each chapter ranges from 8 to 17 (Table 4).
Table 4 - Summary of government contribution of the review of each chapter of the WG I SOR
The government representatives of 5 countries commented on just one chapter and 13 countries (more than half) commented on less than half of the chapters. In contrast the government representatives of USA and Australia, both non-signatories to the Kyoto Agreement, commented on all 11 chapters. Not surprisingly the USA and Australia were well represented when it came to the total number of comments, being 2nd and 3rd highest respectively, and eclipsed only by an individual reviewer. The reviewers for governments in countries in western and northern Europe made a total of 749 comments but those for the USA alone made 689 comments. Eleven countries each made a total of fewer than 15 comments for the entire WG I report. (Table 5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||