Peer review? What peer review?

By | September 6, 2007

Peer Review? What Peer Review?

Failures of scrutiny in the UN’s Fourth Assessment Report


For the Full Report in PDF Form, please click here.

Summary for Policy Makers

INTRODUCTION

The IPCC would have us believe that its reports are diligently reviewed by many hundreds of scientists and that these reviewers endorse the contents of the report. An analysis of the reviewers’ comments for the scientific assessment report by Working Group I show a very different and very worrying story.

The comments for Working Group I are the only set of reviewers’ comments to be made available to the public, and only then thanks to use of US Freedom of Information laws rather than a willingness on the part of the IPCC to allow people to examine the material. Surely all people should be able to examine the involvement and thinking of their governments and the reviewers from their own countries because it is the people who will most certainly bear the economic and political costs of any resultant actions.

Perhaps the IPCC is simply worried that exposing the reviewers’ comments and the responses to those comments to close scrutiny will reveal the delusions of thoroughness and widespread consensus.

REVIEWERS AND COMMENTS

A total of 308 reviewers commented on the Second Revision, which was the penultimate draft, but only 32 reviewers commented on more than three chapters and just five reviewers commented on all 11 chapters.

At the other end of the scale, 143 reviewers (46%) commented on just one chapter and a further 71 (23%) on two. This would be acceptable if they had provided numerous detailed comments, but 53 of these 214 reviewers made fewer then five comments and 28 reviewers made fewer than three comments.

The number of reviewers who made just one comment on a chapter varied between 12.6% and 32% (i.e. almost one-third) of the reviewers commenting on that chapter. For four chapters, fewer than six comments were made by more than 50% of the reviewers who commented. For another four chapters, the figure was between 40% and 50%.

Reviewers’ comments come in all forms. Many are simple corrections to spelling and grammar, others point out inconsistencies, some ask for a change of wording, many ask for expressions of less certainty, others suggest extensive references that should be included. A minority requests a change of wording and provides extensive reference material to support their statements.

One response to a reviewer’s comment is worth mention – “Rejected. McKitrick and Michaels (2004) is full of errors. There are many more papers in support of the statement than against it.” – But this erroneously implies that a consensus of papers determines what will be included, which of course is not very different to claiming a consensus determines a scientific truth.

Many reviewer comments appear to be rejected with little or no justification for doing so. In particular there appears a disturbing pattern of rejecting reviewers’ citations of references by claiming that a greater number of papers say otherwise but then referring to just one paper to dispute the comments of other reviewers. Rejecting references to papers that challenge or weaken claims of serious man-made interference with climate serve to create from whole cloth a contrived, false “consensus.”

At other times changes were made, but simply resulted in new wording which imply a certainty or emphasis very similar to the wording that the reviewer complained about.

The reviewers appear to have had varying success at modifying the emphasis of some paragraphs but one must wonder what the report would have been like if the reviewers had not commented at all.

It is polestar clear that the IPCC-appointed chapter editors believed that their say was final in regard to the certainty of statements and that theirs was the only correct interpretation of the cited material. For many reviewers who could provide logical refutations, either with or without specifying references, the entire process was an exercise in frustration.

The notion of hundreds of experts diligently poring over all chapters of the report and providing extensive feedback by way of peer review to the editing teams is here demonstrated to be an illusion. The true picture is there were some 64 reviewers for each chapter, of whom half made very few comments. Most comments were minor drafting amendments.

GOVERNMENT INVOLVEMENT

Almost all governments see climate change as a major political issue, but government reviewers’ comments suggest otherwise. These comments were logged as being from the “Govt. of (name)”, but the 11 chapters of the Second Revision show that the names of only 22 national governments (plus the European Commission) were recorded.

By contrast, the USA and Australia, both non-signatories to the Kyoto Agreement, made the second-highest and third-highest numbers of comments respectively, eclipsed only by an individual reviewer. Both countries commented on all 11 chapters and show a degree of interest not reflected in the submissions of any other government.

Large regions of the globe made few, if any, comments under government auspices – Eastern Europe, Russia, the South American continent, the Middle-east and a large part of south-east Asia. Not a single comment came from any African country. Likewise, countries which have been vocal in their claims that they will suffer most from sea-level rises, namely the Maldives, Tuvalu and Bangladesh, failed to comment on any chapters of the report.

Denmark administers Greenland, which is supposedly suffering from the widespread recession of glaciers, but no comments were made on behalf of the Danish government.

Switzerland is seeing the recession of glaciers to their mediaeval extents and a receding snow-line, and incidentally is the home of the IPCC, but likewise had no comments.

Government reviewers commented on the Second Revision and on the Final Draft of the Summary for Policymakers. Because the IPCC required the underlying science chapters to conform to the Summary for Policymakers, the publication of those chapters was delayed until after the publication of the Summary for Policymakers. The correct sequence would have been the other way about: settle the science first, and then summarize it.

The governments of 16 countries plus the European Commission reviewed the Second Revision and made 639 comments. The governments of 26 countries plus the European Commission made 931 comments for the final draft of the Summary for Policymakers, even though the document had already been finalized.

More detailed analysis

1 – Some governments are seen to completely accept the hypothesis of anthropogenic global warming while others express greater caution. On page 16 is a table listing the number of comments where the reviewer is designated as “Govt. of”… to provide an indication of the diligence of these reviewers and, one might surmise, the interest shown by the respective governments. (This data is a subset of that dealing with all comments.)

2 – Other reviewers may also be government representatives but are not indicated as such.

3 – The European Commission was included by virtue of its quasi-governmental position.

4 – In several instances the wording of a “government” comment duplicates the comments of a reviewer who is indicated to be an individual. This not only distorts the analysis but suggests a process that was not diligently carried out.

5 – Only 23 “government reviewers” are indicated, which makes one wonder about the commitment of the governments of the other 140+ countries. Just 6 non-English speaking countries outside Europe had designated government reviewer’s comment.

6 – The European Commission, western and northern European countries made up 12 of the 23 countries with comments by government reviewers and these countries made 749 comments of the total 2010. The USA made 689 comments alone.

7 – Fourteen governments commented on fewer than 5 chapters and only Australia and the USA on all 11. Eleven countries made a total of fewer than 15 comments for the entire report (i.e. all 11 chapters).

8 – Significant regions of the Earth had no or very few reviewers who were designated as government reviewers. These figures apply to the Second Revision of chapters 1 to 11:

  1. Eastern Europe – Czech Republic and Hungary only
  2. Russia and former Russian states – None
  3. Middle-east – None
  4. African continent – None
  5. South American continent – Brazil and Chile only
  6. Asia – China, Japan, Korea, and Thailand only

IPCC’S REJECTIONS OF REVIEWERS’ COMMENTS

The IPCC’s editors could – and often did – reject the peer-reviewers’ comments, a reversal of the normal practice in scientific peer-review. Analysis of the extent of the editors’ refusal to accept criticism is difficult because the expressions of rejection come in many forms, some were partial and others were rendered otiose by the rewriting, restructuring or deletion of sections of text.

A simple analysis based on the occurrence of three key words – “rejected”, “reject” and “disagree” – underestimates the total number of rejections. Even so, this analysis reveals that the number of peer-reviewers’ comments that were rejected by the IPCC climate-templars averaged 25% (min. 9.5%, max 58.1%) of all comments on the Second Revision.

The striking feature of most rejections is their dubious nature. Some were banal. Others showed inconsistencies with other comments. Peer-reviewers had to justify the textual amendments which they were putting forward, but the responding editors were under no corresponding obligation to justify their rejections of the reviewers’ proposals.

One reviewer said that “best estimate” should more correctly be “most recent estimate” but the editors changed the text to “current best estimate”. Reviewers were sometimes flatly told they were wrong, but no reasons or incontrovertible references were provided.

Another reviewer said that one heat wave did not make a trend, but the editors rejected this by claiming they had used a particular heat wave as an example. This mistreatment of the reviewer’s scientifically-appropriate amendment allowed the passage complained of to be taken out of context, so that the heat wave in question has been widely – though erroneously – interpreted as having been caused by anthropogenic “global warming”.

This mistreatment of the reviewer’s scientifically-appropriate amendment allowed the passage complained of to be taken out of context, so that the heat wave in question has been widely – though erroneously – interpreted as having been caused by anthropogenic “global warming”.In several instances, reviewers invited the IPCC to express its conclusions with less certainty, and provided evidence in support of more caution given the uncertainties inherent in climate science. In almost every such instance, the IPCC’s reviewers flatly rejected the reviewers’ suggested moderations of its conclusions. Some comments were rejected on the ground that there was not enough space. Given the unconstrained length and supposed importance of the IPCC’s assessment report, this ground of rejection is not compelling.

Reviewers would cite references in the learned journals challenging the IPCC’s conclusions, but in almost every instance they were told that a greater number of references supported an alternative argument. The correct approach, at the very least, would have been to insert in the assessment report a mention of the references that challenged the IPCC’s conclusion.

Reviewers who made brief proposed amendments would often be brushed off by being told of just one paper that contradicted the suggested amendment. In at least one response the IPCC’s editors made reference to a document that had not been subjected to peer-review at all.

While the editors sometimes accepted simple corrections and tolerated requests for improved clarity, they routinely resisted any serious challenge to the report’s assertions, interpretations and conclusions. This effects peer-review through a looking glass.

REJECTION OF AMENDMENTS ON ATTRIBUTION OF CLIMATE CHANGE

In Chapter 9, the key science chapter, the IPCC concludes that “it is very highly likely that greenhouse gas forcing has been the dominant cause of the observed global warming over the last 50 years”. The IPCC leads us to believe that this statement is very much supported by the majority of reviewers. However, only 62 of the IPCC’s 308 reviewers commented on this chapter at all.

Nineteen reviewers made just one comment and 18 made between two and five comments. Just 10 reviewers made more than 20 comments on this, the most important chapter of the entire report.

A total of 1158 comments were made. One reviewer made almost half of these, but almost all his suggested amendments were curtly rejected. The government of the USA made 113 comments; almost 10% of the total, but 32 of its comments duplicate those made by an individual reviewer.

In these circumstances any review which casts doubt about assertions based on or related to a human influence on climate will be just what many reviewers found it to be – frustrating and futile.As with other chapters, simple corrections, requests for clarifications or refinements to the text which did not challenge the IPCC’s conclusions are generally treated favourably, but comments which dispute the IPCC’s claims or their certainty are treated with far less indulgence.

In particular, comments which draw attention to natural climate forces (e.g. El Nino influences, or the natural “blocking high” that triggered the 2003 European heat wave) are abruptly rejected. The pretext for some of the rejections was the citation of previous IPCC reports which themselves were inadequately reviewed, and were not subjected to the rigorous peer-review that is customary before a scientific paper can be published in the learned journals. Keep in mind, previous reports were (a) not reviewed in the same manner as scientific papers and (b) were the result of similar dubious processes as in the current report.

In many instances the IPCC’s editors responded to comments by saying that the point had been discussed in some other chapter.

It is difficult to quantify the extent of the reviewers’ support for the IPCC’s conclusions in the chapter on attribution of climate change. Given the number of reviewers who made very few comments, the duplication of comments and the number of minor corrections, it appears likely that less than 40 of the IPCC’s 308 reviewers were generally supportive of the hypothesis. It is not true, therefore, that hundreds of scientists endorsed the IPCC’s findings, still less that thousands did so.

The IPCC’s reports, then, are not peer reviewed in the sense that is commonly understood. The editors, rather than accepting genuine and often well-referenced criticisms of the IPCC’s conclusions, have instead tended simply to reject most substantial criticisms.

Given the number of reviewers who made very few comments, the duplication of comments and the number of minor corrections, it appears likely that less than 40 of the IPCC’s 308 reviewers were generally supportive of the hypothesis. It is not true, therefore, that hundreds of scientists endorsed the IPCC’s findings, still less that thousands did so.

General Conclusions

Three conclusions follow. First, the IPCC is merely presenting what it regards as a consensus among published scientific papers – in effect, a giant review article rather than original research.

Secondly, in order to produce a paper on some aspect of climatology a researcher needs funding. In the current environment that funding is very obviously directed towards studies which assert that the human influence on climate is substantial. It should be no surprise, therefore, that the number of papers adhering to what has become a “party line” can be presented – rightly or wrongly – as a “consensus”.

Thirdly, the dominance of research presupposing a human influence also means that the IPCC editing teams are likely to consist of people predisposed to view the situation in that light.

In these circumstances any review which casts doubt about assertions based on or related to a human influence on climate will be just what many reviewers found it to be – frustrating and futile.

Issues with the Review of the IPCC 4ARWG I Report

Abstract

The Intergovernmental Panel on Climate Change (IPCC) gives the impression that its Fourth Assessment Report (4AR) was thoroughly and diligently reviewed and the statements contained in the report were endorsed by a very high percentage of reviewers.

This analysis of the reviewers’ comments for Working Group I (WG I) shows that the reality is rather different and that there is surprisingly little explicit support for the key notion, that humans are very likely (90% to 95%) responsible for climate change.

 

Part 1 – Introduction

The Intergovernmental Panel on Climate Change (IPCC) gives the impression that hundreds of scientists thoroughly and diligently peer-reviewed its Fourth Assessment Report on climate change. However, this analysis of reviewers’ comments on the science chapters by Working Group I which formed the core of the Report casts some doubt on how many reviewers endorsed the IPCC’s key notion that we are very likely (90% to 95%) to have caused more than half of the 0.4C observed warming over the past 50 years.

On previous occasions the IPCC did not release reviewers’ comments. It is only thanks to the Freedom of Information legislation in the USA and the efforts of Steve McIntyre of www.climateaudit.org that the reviewers’ comments have been made available here.[1]

It is somewhat worrying that the public has not been permitted to examine the comments for previous IPCC reports or for the other working groups which contributed to the Fourth Assessment Report, especially when the global population will be expected to bear the heavy cost of actions based on the contents of the report. People should be allowed to see comments made in the name of their governments. The review process must be exposed to independent scrutiny, or there will be no verifiable evidence that the IPCC’s claimed “consensus” really exists.

As will be seen, different numbers of reviewers commented on each chapter, and this is very likely due to the different subject matter. Chapters and their titles are therefore listed in table 1

 

No.

Name

1

Historical Overview of Climate Change Science

2

Changes in Atmospheric Constituents in Radiative Forcing

3

Observations: Surface and Atmospheric Climate Change

4

Observations: Changes in Snow, Ice and Frozen Ground

5

Observations: Ocean Climate Change and Sea Level

6

Paleoclimate

7

Couplings between Changes in the Climate System and Biogeochemistry

8

Climate Models and their evaluation

9

Understanding and Attributing Climate Change

10

Global Climate Projections

11

Regional Climate Projections

SPM

Summary for Policy Makers

TSR

Technical Summary Report

Table 1 – Chapter numbers and titles for IPCC 4AR WG I report

Part 2 – Number of Reviewers and Comments

A total of 308 reviewers[2] commented on chapters of Second Order Revision (SOR), i.e. the penultimate draft, but only 32 reviewers commented on more than 3 chapters and just 5 on all 11 chapters (table 2 and figure 1).

At the other end of the scale, 143 reviewers (46%) commented on just one chapter and a further 71 (23%) on two. This would be fine if they were experts and provided numerous detailed comments but 53 of these 214 reviewers (25%) made fewer than 5 comments and 28 of them made fewer than 3. This raises the question of why they bothered to review any chapters and the question of whether they examined other chapters but had nothing to say.

chapters

1

2

3

4

5

6

7

8

9

10

11

reviewers

143

71

47

16

4

7

5

4

4

3

5

Table 2 – Number of chapters commented on by reviewers

 

Figure 1 – Graphical representation of table 2

As noted above, the chapters of the Second Order Revision were subjected to attention by different numbers of reviewers. One hundred reviewers examined chapter 2, which dealt with changes to the atmosphere, but just 34 examined chapter 4, which discussed changes to snow, ice and frozen ground.(table 3)

Chapter

1

2

3

4

5

6

7

8

9

10

11

No. of reviewers

53

100

87

34

64

75

70

58

62

87

49

Table 3 – No. of reviewers who commented on each chapter of the Second Order Revisions

Reviewers commented on the chapter as a whole and then on each paragraph of the draft in question. Most reviewers’ comments fall into one of the following categories

– praise

– correction of typographic errors (spelling and punctuation)

– correction of grammatical errors

– suggested improvements (words or phrases)

– requests for clarifications, for more precise wording or for definitions

– corrections of references or suggestions of additional references,

– other corrections or clarifications (e.g. “Not all volcanic eruptions are climate-relevant.”)

The appendix to this document contains some sample comments both with responses and without.

Those responses come from the editorial team for each chapter and naturally reflect the acceptance or rejection of comments as well as a host of other possible situation (eg. “text has been rewritten”).

One response noted in passing was “Rejected. McKitrick and Michaels (2004) is full of errors. There are many more papers in support of the statement than against it.” This indicates that a consensus of papers is a major determinant of the content of the report.

In some cases comments attributed to an individual also appear under the name of a national government but the extraction of these duplicated comments would call for subjective judgment and has not been attempted here.

Not surprisingly the number of comments by each reviewer varies greatly between chapters (figure 2).

Figure 2 – No of reviewers making various numbers of comments

The number of reviewers who made just 1 comment on a chapter varied between 12.6% and 32% (i.e. almost one-third) of the reviewers that commented on that chapter. For 4 chapters fewer than 6 comments were made by more than 50% of the reviewers that did comment and for another 4 chapters the figure was between 40% and 50% (figure 3).

Figure 3 – Percentages of reviewers making few (<6) comments and many comments

The number of comments per reviewer per chapter varies greatly but by simple addition we can see how many comments each reviewer made and we can gain an indication of the probable distribution of the effort put into the task of reviewing these chapters (Figure 4).

Figure 4 – Total number of comments made by reviewers

Forget any illusion of hundreds of experts diligently poring over all chapters of the report and providing extensive feedback to the editing teams. The true picture is closer to 64 reviewers for any one chapter, with about half of those not commenting on any other chapter and one quarter commenting on just one other. On top of that, about half of those reviewing any one chapter made very few comments.

Part 3 – The contribution of government reviewers

Almost all governments see climate change as a major political issue but comments assigned to government reviewers (and denoted as “Govt. of (name)”) suggest otherwise.

The review of the 11 chapters of the SOR of WG I show the names of only 22 national governments, plus the “European Commission” which was somehow granted the status of a government. Surely the governments of the remaining 140+ countries are not as disinterested in climate change as these figures suggest.

It is possible that some individually named reviewers were working on behalf of governments but it is impossible to determine them from the information given. Conversely there are clear instances of identical comments being attributed both to an individual and to a national government, which indicates that certain processes were not diligently carried out.

From the names of the reviewers it appears that there was no review of the WG I report by a large proportion of the globe. Of the 22 named governments 11 are western and northern European countries. The Czech Republic and Hungary were the only east European countries reported. No review was undertaken by Russia and its former states, nor from any country in Africa or the Middle-east. From South America only the governments of Brazil and Chile reviewed to the report and from Asia China, India, Japan, Korea and Thailand. The three countries not mentioned thus far were Australia, Canada and the United States of America.

Countries that claim that they will suffer most from rising sea level, namely Maldives, Tuvalu and Bangladesh, had no government reviewers comment on any chapters of the report. Denmark administers Greenland, which is supposedly suffering from a retraction of glaciers, but no review appears to have been undertaken on behalf of the Danish government. Switzerland is seeing the retraction of glaciers and a rising snow line – and incidentally is the home of the IPCC – but likewise made no comments under government auspices.

As with the individual reviewers, the extent of the review by government representatives varies enormously. The number of governments whose representatives reviewed each chapter ranges from 8 to 17 (Table 4).

Chapter

1

2

3

4

5

6

7

8

9

10

11

SPM

No. reviewing Govs

8

17

11

8

10

10

11

10

8

13

10

17

Table 4 – Summary of government contribution of the review of each chapter of the WG I SOR

The government representatives of 5 countries commented on just one chapter and 13 countries (more than half) commented on less than half of the chapters. In contrast the government representatives of USA and Australia, both non-signatories to the Kyoto Agreement, commented on all 11 chapters.

Not surprisingly the USA and Australia were well represented when it came to the total number of comments, being 2nd and 3rd highest respectively, and eclipsed only by an individual reviewer. The reviewers for governments in countries in western and northern Europe made a total of 749 comments but those for the USA alone made 689 comments. Eleven countries each made a total of fewer than 15 comments for the entire WG I report. (Table 5)

c

Ch 1

Ch 2

Ch 3

Ch 4

Ch 5

Ch 6

Ch 7

Ch 8

Ch 9

Ch 10

Ch 11

Total

No. Ch

European Commission

0

18

0

0

0

0

0

0

0

47

0

65

2

Govt. of Australia

24

36

83

2

40

37

1

23

11

72

33

362

11

Govt. of Austria

1

1

0

0

0

0

2

1

3

4

0

12

6

Govt. of Belgium

0

14

0

0

0

0

0

0

0

0

0

14

1

Govt. of Brazil

0

0

7

0

0

0

21

0

0

0

1

29

3

Govt. of Canada

0

0

0

6

0

0

2

29

0

14

0

51

4

Govt. of Chile

0

0

0

0

7

0

0

0

0

0

0

7

1

Govt. of China

2

8

7

6

3

4

3

10

0

4

5

52

10

Govt. of Czech Republic

0

0

0

0

0

0

0

0

0

0

7

7

1

Govt. of Finland

1

39

7

1

1

0

0

50

9

36

23

167

9

Govt. of France

20

2

22

0

1

6

0

3

12

16

13

95

9

Govt. of Germany

0

25

6

7

1

18

24

1

2

95

0

179

9

Govt. of Hungary

0

2

0

0

2

0

1

0

0

0

0

5

3

Govt. of Ireland

0

3

0

0

0

0

0

0

1

0

1

5

3

Govt. of Japan

0

5

2

1

0

1

0

0

0

1

0

10

5

Govt. of Netherlands

0

0

0

0

0

0

0

4

1

5

4

14

4

Govt. of Norway

0

7

0

0

0

0

0

0

0

0

0

7

1

Govt. of Republic of Korea

0

3

3

0

17

0

22

0

0

0

0

45

4

Govt. of Spain

10

4

30

0

0

53

6

1

0

0

28

132

7

Govt. of Sweden

0

0

0

1

0

6

1

0

0

2

0

10

4

Govt. of Thailand

0

0

0

0

0

4

0

0

0

0

0

4

1

Govt. of United Kingdom

4

1

14

0

8

1

0

0

0

21

0

49

6

Govt. of USA

46

85

129

50

43

99

45

43

113

18

18

689

11

TOTALS

108

253

310

74

123

229

128

165

152

335

133

2010

Table 5 – Number of comments by government reviewers for each chapter

{mospagebreak} Part 4 – Authors or Reviewers?

The figures shown in part 2 apply to the number of reviewers as whole. These include the reviewers operating under government auspices (discussed in Part 3) but also numerous reviewers who were among the team of authors for the chapters in question.

There may be legitimate reasons for a contributing author of a subsection to review other sections of the same chapter but the impression one gets is these author-reviewers were unable to raise their points within the internal communication of the IPCC authoring teams. How else does on explain that a Lead Author of chapter 2 made review comments about that chapter or that one author made 282 comments on his only chapter or that 3 authors of chapter 11 made a total of 350 review comments on that chapter.

In total 30 author-reviewers made all of their comments about chapters that they authored

Table 6 shows the number of authors and reviewers for each chapter. Six chapters had fewer reviewers than authors but that increases to 8 when author-reviewers are excluded.

Reviewers

Chapter

Authors

Gov. Revs

Author-revs

Other revs

Total Revs

1

36

8

4

41

53

2

54

15

6

79

100

3

81

11

12

64

87

4

57

8

2

24

34

5

68

10

12

42

64

6

51

10

6

59

75

7

78

11

8

51

70

8

88

10

9

40

58

9

56

8

7

47

62

10

94

12

11

64

87

11

59

10

6

33

49

Table 6- Number of authors and reviewers for each chapter

Six chapters were reviewed by fewer than 50 individuals who were not authors of that chapter, but in each case that number of reviewers might include authors of other chapters.

The total number of authors and reviewers of the WG I report is misleading because several individuals were authors of more than one chapter and several authors were also reviewers.

Authors who reviewed: 95

Authors who did not review: 517

Reviewers: 214

In total 826 individuals contributed to the WG I report but only slightly more than 25% were reviewers, and as we have seen, the contribution by reviewers was often very minor.

Part 5 – Number of Comments and Rejections

Commenting is only part of the picture because those comments could be ignored by the editors.

Determining the number of rejected comments is difficult because the expressions of rejection come in many forms, the rejection may only be partial or the comments may be made irrelevant by sections of text being rewritten, deleted or restructured.

A simple analysis based on the occurrence of three key words – “rejected”, “reject” and “disagree” – provides a likely minimum number of rejected comments because other words may be used. This somewhat crude analysis reveals that the minimum number of rejected comments averages 25% of all comments on the SOR of a chapter and ranges from 9.5% to 58.1% (Table 6, Figures 5 & 6).

 

FIRST REVISION

SECOND REVISION

Total

Reject

Rej%

Total

Reject

Rej%

Chapter 1

899

154

17.1%

554

155

28.0%

Chapter 2

2732

270

9.9%

1313

254

19.4%

Chapter 3

2231

307

13.8%

1256

368

29.3%

Chapter 4

1137

64

5.6%

516

109

21.1%

Chapter 5

1204

57

4.7%

635

119

18.7%

Chapter 6

1789

252

14.1%

1112

362

32.6%

Chapter 7

1751

105

6.0%

974

113

11.6%

Chapter 8

963

179

18.6%

794

159

20.0%

Chapter 9

1436

246

17.1%

1157

672

58.1%

Chapter 10

1331

73

5.5%

1331

354

26.6%

Chapter 11

1458

99

6.8%

1647

156

9.47%

SPM

no rev

no rev

no rev

1455

372

25.6%

TSR

no rev

no rev

no rev

1333

330

24.8%

Table 7 – Summary of total comments and the likely minimum number of rejected
comments. ( “No rev” indicates that no review took place).

 

Figure 5 – Accepted and rejected comments for the SOR( based on minimum rejected)

Figure 6 – percentage of rejected comments (based on minimum rejected)

The striking feature of many rejections is their dubious nature. Some responses were banal and others showed inconsistencies with other comments. Reviewers had to justify their requested change but the responding editors appear to have been under no such obligation.

One reviewer said that “best estimate” should more correctly be “most recent estimate” but the editors changed the text to “current best estimate”. Reviewers were sometimes flatly told they were wrong but no reasons or incontrovertible references were provided.

Another said that one heat wave did not make a trend but the editors rejected this by claiming they used that heat wave as an example. Too bad if the passage was taken out of context and that heat wave being interpreted as due to climate change when contradictory evidence and expert statements at that time said otherwise.

In other cases reviewers tried to dilute the certainty being expressed and they often provided supporting evidence, but their comments were often flatly rejected.

Some comments were rejected on the basis of a lack of space and it seems incredible that space should have been a constraining factor on such an important document.

Reviewers would cite references but be told that a greater number of references supported an alternative argument. Reviewers would make a brief statement of correction but be told of just one paper that contradicted that claim. In at least one response the editors made referred only to a document that has not been subjected to peer-review.

The attitude of the editors seems very much to be that simple corrections will be accepted, requests for improved clarity be tolerated but the assertions and interpretations that appear in the text were to be defended against any challenge.

Part 6 – On the Attribution of Climate Change

Chapter 9 is the single most important chapter of the entire report because it is where the IPCC states, “it is very highly likely that greenhouse gas forcing has been the dominant cause of the observed global warming over the last 50 years”.

The IPCC leads us to believe that this statement is supported by a large number of reviewers. We often hear reference to 2,500 scientists supporting the IPCC’s findings but that number supposedly includes about 1,500 acting as chapter editors. Earlier it was shown that a total of 308 reviewers, individuals or government appointees, reviewed parts of the WG I report but even that figure is far higher than the number of reviewers for chapter 9.

In fact only 62 reviewers commented on this chapter, of which 8 were noted as government reviewers. Nineteen reviewers made just 1 comment and 18 made between 2 and 5 comments, and those 37 reviewers are 60% of the total. Just 10 reviewers made more than 20 comments for this, the most important chapter of the entire report, and yet some of these were typographical errors that were missed by many reviewers.

Of the 54 individual reviewers 31, i.e. more than half, had a vested interest in this chapter of the report: Three were editors of the entire IPCC WG I report, 7 were from the chapter’s 44 contributing authors and 1 was noted “WGI TSU”, which indicates some link to the IPCC team. Twenty-six appear to be authors or co-authors of papers cited in this chapter and 10 of the 54 made explicit reference to papers that they had written or co-written. In some instances we even find reviewers who fell into 2 or 3 of the above categories.

Just 23 individual reviewers and 8 government reviewers appear to have no vested interest in chapter 9 of the WG I report, the remaining 31 being tainted in some way.

A total of 1158 comments were made but just one reviewer made 572 comments, or 49.4% of the total. The government of the USA made 113 comments (9.8%) but many comments by the US government duplicate the 32 comments made by an individual reviewer.

The majority of those 572 comments from one reviewer appear to strike at four contentious issues. First there is the corruption of the generic meaning of “climate change” into “man-made climate change”; second the matter of whether urban heat islands, which the reviewer often refers to as the proximity of measuring equipment to human induced heat, are distorting the temperature record; thirdly the discrepancy between tropospheric temperature changes and surface; and fourthly the impact of El Nino events on any trend.

The IPCC editing team rejects the above points claiming firstly no distortion in the meaning of “climate change”, contradicted of course by the IPCC’s name including the words “climate change” and yet being focused on a human influence on climate. Secondly it argues that there is no evidence that human induced outputs of heat have corrupted the data and on the third point it refers to papers that dispute the tropospheric temperature record but on both matters ignores the absence of any verification of the accuracy of near-surface temperature records. The IPCC consistently claims that El Nino events are internal to the climate system but seem to forget that their occurrence in the tropics makes them come under the influence of solar radiation and that subsea volcanic activity may be contributing.

On many occasions the IPCC claimed a numerical superiority of papers that supported its line of argument and referred to comments in the Third Assessment Report (TAR) of 2001. The clear implications were that a consensus of papers is one determining factor and that whatever was said in the TAR must be correct, although one suspects that the TAR was likewise disputed by reviewers.

The most risible of the IPCC’s responses is “Rejected. The ability of models to simulate the temperature variations indicates that any missing natural forcings have little impact.” Apparently the IPCC believes that if the output of the models is approximately correct then the internal workings of the model must likewise be correct. Perhaps the IPC is unaware that if a model based on a factor that is driven by temperature rather than drives it, that model will be false but will probably produce output that matches historical data.

It is clear that the 572 comments by this one reviewer were not frivolous but addressed some very significant core issues, so one wonders why other reviewers did not make similar comments.

Of the remaining comments, and discounting the duplication under the name of the United States government, 99 of 554 comments were rejected, which is still more than 1 in 6.

Genuine support for the chapter is difficult to ascertain because comments would be accepted or rejected on their individual merits, nullified by changes to passages of text, and 5 comments appear to be only partially accepted. Unless support is explicitly stated a subjective evaluation is required as to either the tone of the remarks or whether the absence of comment implies support.

Only 4 reviewers with no vested interest explicitly supported the overall chapter although others praised specific sections. Among the majority with a vested interest only 4 reviewers specifically praised the text.

The IPCC implies that the majority of hundreds of reviewers endorsed the claim that humans had very likely been responsible for the majority of the warming in the last 50 years. This analysis of chapter 9, the key chapter to the WG I report and indeed to the entire Fourth Assessment Report, reveals that implication to be entirely false.

Part 7 – A Final Word

The IPCC states clearly that it undertakes no research of its own but merely relies on published papers for its information. A team of editors assesses those papers and writes the drafts of the various report chapters. While minor corrections are welcomed the overall assessment is strongly defended against challenges.

On the surface this looks not unreasonable but scratch a little deeper and the self-sustaining nature of the claim of a human influence on warming becomes visible.

Unlike other high-profile scientific fields, these reports by the IPCC are almost entirely responsible for determining the direction of climatology and how the research funding will be spent.

The IPCC’s Third Assessment Report (TAR) of 2001 showed that 8 of 11 climate factors were poorly understood but despite this it claimed that humans were responsible for rising temperatures.

As a consequence of the TAR the majority of funding for climatology research went to projects that assumed a human influence on climate.

Not surprisingly this caused the papers taking this position to significantly out-number the papers that rejected this hypothesis. But as the responses to reviewers’ comments show, the number of papers supporting a certain argument is a critical factor in determining the content of the IPCC reports.

It is not merely the weight of numbers that tilts the balance but also the leanings of the editors. The content of the reports rests with the teams of editors but if those editors are actively engaged in research then it is likely to be on projects which assume a human influence on climate and this will make those editors susceptible to being predisposed to view climate in that light.

There is not the evidence to claim deliberate bias but logically the “anthropogenic warming” argument will be very familiar to many editors and the tendency will be that papers following that line will receive less intense scrutiny than papers that don’t only challenge that argument but also challenge the editors’ own beliefs. If an editor took the position that the human influence on climate is negligible or non-existent on anything but a small and localised scale then that person’s research opportunities are likely to be few.

The same potential conflict of interest arises with the reviewers, many of whom are authors of papers related to climatology and are quite possibly still involved in research projects. The reviewers have the added problem that the IPCC practice is to make all reviewers’ comments available to other reviewers. Reviewers cannot hide behind some kind of editorial team “group think” but are exposed to individual scrutiny and that can put reputations and research opportunities at even greater risk.

The problems continue into the authorship of these reports. According to IPCC documents, scientists are nominated by governments or explicitly invited by scientists who were already associated with the IPCC. What a wonderful way to position scientists who support a government agenda on climate and then fill out the IPCC with like-minded individuals.

The bigger picture is that research funding indirectly determines the content of the IPCC assessment reports, and those assessment reports play a very significant role in determining the direction and funding of the research.

Who would be a reviewer when many chapter authors will be likely to defend the beliefs and reputations they have established via research projects funded by government money on the supposition that anthropogenic global warming is a fact? Few researchers who are funded by the anthropogenic warming gravy-train are likely to review IPCC chapters with the intent of identifying flaws only those sceptical of the claims, and have little to lose in the way of reputation or funding, will make the effort. Several recognised sceptics of man-made warming failed to take part in the review but who can blame them when the exercise is so evidently futile.

In the long term this perpetual and increasing marginalising of contrary viewpoints is extremely detrimental to the science because it will produce a supposed scientific “truth” based on little more than the emphasis of the funding and the domination of certain opinions.

Key Points

The review of the Working Group 1 report was far less intense than the IPCC has implied.

– 308 reviewers examined the chapters of the Second Order Revision (i.e. penultimate draft) of the Working Group 1 report, with the average number of reviewers per chapter being 67 (minimum 34, maximum 100).
– 214 reviewers (69%) commented on two chapters or less and 60 reviewers averaged fewer than 3 comments for all chapters they examined
– Only 5 reviewers, specifically 3 individual reviewers and 2 government reviewers, commented on all chapters and just 49 reviewers (16%) made more than 50 comments in total

Only 22 governments had designated reviewers but 5 of these commented on only one chapter and 5 averaged less than 3 comments per chapter. The United States of America and Australia, both non-signatories to the Kyoto Agreement, commented on all 5 chapters and made the greatest number of comments.

On average the editors rejected at least 25% of those reviewers’ comments for any chapter but many of those rejections are contentious.

The critical chapter, that which attributed recent warming to human activity, was reviewed by 54 individual and 8 government representatives but almost 1/3rd of reviewers made just one comment.

– 31 of the 54 had a vested interest in the report, as editors or having papers cited
– 26 authored or co-authored papers cited in the final draft
– 10 reviewers explicitly mentioned their own papers in their review

Among the 23 independent reviewers just 4 explicitly endorsed the chapter with its hypothesis of a significant human influence on climate, and one other endorsed only a specific section.

The reviewers’ comments show that is actually little support for the IPPC’s contention that anthropogenic emissions of carbon dioxide have caused warming.

The IPCC reports appear to be largely based on a consensus of scientific papers, but those papers are the product of research for which the funding is strongly influenced by previous IPCC reports. This makes the claim of a human influence self-perpetuating and a corruption of the normal scientific process.

Bio

John McLean is climate data analyst with an extensive background in the IT industry. He became interested in the question of climate change when told of evidence that directly refuted the frequent claim that recent temperatures were unprecedented. He lives in Melbourne, Australia, and is a member of both the “Climate Sceptics” and “New Zealand Climate Science Coalition” Internet discussion groups.

Appendix I

This appendix contains just some of the less banal comments (i.e. minor corrections or praise to a chapter), which arbitrarily come from chapters 1, 2, 3, 9 and 10, in order to provide some impression of the types of comments and responses..

Comments are shown here as they appeared (i.e. without spelling or grammar corrections). They are shown in full unless otherwise noted. Where reviewers’ comments are shown without a response the reference number is provided but where responses are included no number is shown. Where I have appended comments for this document those comments are indicated in bold and within [and].

Part 5(a) – Sample Reviewers’ Comments (without responses)

1.01 “Solar radiation is the driving force of the climate system.” This could be taken out of context. Consider changing “the driving force” to “a driving force. (1-435)

1.02 Understanding will necessarily have “evolved” since the TAR, but more to the point has it markedly improved? (2-424)

1.03 “better understood” – I would say that the trend in methane is not better understood – and indeed is a bit of a mystery. (2-244)

1.04 I would encourage IPCC to consider having only one solar physicist on the lead author team of such an important chapter. In particular since the conclusion of this section about solar forcing hangs on one single paper in which J. Lean is a co-author. I find that this paper, which certainly can be correct, is given too much weight. [part only] (2-901) [J Lean was a Lead Author of the relevant chapter]

1.05 DELETE THE ENTIRE MATERIAL BEGINNING WITH “IN ADDITION” as all of this is highly contentious has all sorts of implicit ethical and moral judgments which you have not even begun to address, and goes way beyond the core science, which is the only thing the WG1 should deal with. (2-1026)

1.06 As written it implies 100% attribution, which is misleading, since the idea that all climate change is attributable to GHG forcing is an extreme position held by few if any experts. Insert “partially” after the word “been” and before “attributed”. This suggestion was made in the FOD review and ignored. It is hereby repeated, for the same reason: the present wording is deliberately misleading. (3-223)

1.07 This conclusion comes out of nowhere! After reading the past two-to-three pages about differing precipitation, soil moisture, and stream flow trends all over the place, I was quite surprised to read “The global increase in both sever drought and large floods suggest that hydrologic conditions have become more extreme.” Apparently my definition of “global” is quite different from yours. (3-421)

1.08 You MUST insert here a proper Figure showing the radiosonde records, preferably those from Figure 9 of the paper of Thorne et al (2005). Figure 3.4.2 is deliberately designed to conceal the true facts about both the radiosonde and the MSU records. The pretence that these three records are virtually identical is a plain lie. [part only] (3-467)

1.09 This is pure speculation. The sondes in these studies have not been corrected for instances where spurious warming occurs as shown in for example in Christy and Norris 2004, Christy and Spencer (2005) and the other papers to appear soon. (3-543)

1.10 1998 is quoted here as the warmest year for the global mean, without qualification. This is at odds with page 3-3, lines 15 to 19, which point out that NCDC and GISS have 2005 warmer than 1998, in contrast to the CRU/UKMO estimate. (3-702)

1.11 You claim that Turner et al. (2005) found ‘… a cooling over much of the rest of the continent’. But that paper was only concerned with station data and there are only two stations with long records in the interior of the Antarctic. In that paper we were careful to point out that few of the annual temperature changes around East Antarctic were statistically significant. Only South Pole has a statistically significant cooling in the annual data. (3-728)

1.12 Seems odd to say that the figure is not shown because it is not reliable, yet then discuss it for several more sentences. Why should we conclude that the discussion is reliable? (3-877)

1.13 How does a study of only a half century of data distinguish interdecadal (e.g.30 year and longer timescale) variability in one phenomenon from other potentially related or unrelated trends in other phenomena. Implausible claims such as this, especially those which rest on one study of half a century of reanalysis data, should not serve as the basis for conclusions in an assessment report. (3-797)

1.14 I found this discussion of “selection bias” confusing. “Fingerprinting” results in a different kind of selection bias, in that only those patterns predicted by the model responses are looked for. One wouldn’t have noticed the ozone hole if one followed this kind of program religiously. [part only] (9-193)

1.15 Please explain for the reader how to understand the apparently high confidence in detection in certain regions where there is very little data over the full 20th century as shown in figure 9.4.2. How is it that you can divide the globe so finely when you have only a few data points in some of these regions over the full 20th century? [part only] (9-591)

1.16 You may need to suitably denigrate our work to justify your conclusion, but you could mention that at least some people strongly disagree with your claims! (10-987)

Part 5(b) – Reviewers’ Comments and Editors’ Responses

2.01 REV: Delete “of the risk of” The study is to find out IF there is a risk at all. You should not assume that there IS a risk.

RES: Rejected: we think there is no ambiguity in the statement as it is.

2.02 REV: Suggest deleting the two sentences “The glass walls….of the planet” is unnecessary and potentially confusing to most non-expert readers.

RES: Rejected . These facts explain the name of the greenhouse effect.

2.03 REV: Sentence should read: “carbon dioxide or water vapor has only a small direct…”
RES: Noted but not taken into account

2.04 REV: Is the “best estimate” a good choice of words? If I read the text I would rather say “most recent estimates”.

RES: Accepted. Changed to ‘current best estimate’. [but this retains the questioned use of the word “best”]

2.05 REV: This paragraph is too generalized – and does not apply to large land areas in the Southern Hemisphere.

RES: Rejected. Nor does it refer to general land areas in the southern hemisphere. It does refer to South America.

2.06 REV: The title is not corresponding to the content. It has to be replaced

RES: Noted. It doesn’t have to.

2.07 REV: This statement is NOT TRUE. Their plot shows a flattening of the number within the last two 5-year periods. [part only]

RES: Noted. Changes made.

2.08 REV: Probably overstates the certainty of their conclusions.

RES: Noted. Text retained as we believe it is correct

2.09 REV: The references to Trenberth et al. (2000) and Trenberth and Stepaniak (2003a,b) are not necessary as this basic information on the Hadley Circulation is dealt with in text books and numerous other journal publications.

RES: Modifted. This is not true: none of this is in text books anywhere!!!!

2.10 REV: The text here states that GHG forcing is smaller than the indirect effect of aerosol – this therefore implies that the net anthropogenic forcing is negative, which is at odds with the statement on pg 67, ln 17, that humans have very likely exerted a net warming influence on climate.

RES: Accepted, paragraph is modified.

2.11 REV: Insert after “corrections”,” But all of them show a zero temperature trend between 1978 and 1998″.

RES: Rejected – no reason given for suggested change The reviewer is taking a biased stance by deliberately selecting a minimum-trend period. [Are the editors taking a biased or unbiased stance?]

2.12 REV: Replace .”lead to important” by “suggest”

RES: Agree wording is not perfect. Replacing “lead to” with “have resulted in”

2.13 REV: Replace “are shown to ” by “may”.

RES: Wording changed to “are projected to” [which is quite different to “may”]

2.14 REV: Most of the evidence suggests the opposite—increased heating at the surface relative to the troposphere. There is some suggestion that the trends in the troposphere may be underestimated (Sherwood et al.) but the corrections have not been made and thus the ultimate outcome is unknown.

RES:Rejected. We are working with the CCSP report. [The CCSP report was not peer-reviewed]

2.15 REV: If the data isn’t good enough to conclude anything from 1979 to the present, how can we really conclude anything from 1958 to the present?

RES: Rejected. Over longer periods there can be a smaller influence of error. [But is this true in this case?]

2.16 REV: I find that it is “very likely that greenhouse gas forcing has been the dominant cause” difficult to reconcile with “it is highly likely that warming… cannot be explained without external forcing”.

RES: Rejected. The second statement is less specific so should have a higher confidence associated with it.

2.17 REV: The example doesn’t really help. Perhaps say, “Extreme events can occur in an unchanging climate.”

RES: The comment indicates that the reviewer does not really understand the statistical point that is being made here….

2.18 REV: One “heat wave” does not make a “trend”
RES: Rejected. The European heat wave is just a single example and this is clear in the current text.

2.19 REV: I find the statements in the second and third (non title) rows hard to reconcile. It seems to suggest that if greenhouse gas forcing has been involved then it has to be dominant, i.e. there is no room for it to be a minor contributor.

RES: Noted. We don’t quite see the difficulty. The assessment is that greenhouse gas has been the dominant contributor.

2.20 REV: Please be precise on whether the net RF is LIKELY or VERY LIKELY positive since 1750. Whereas line 17 states VERY LIKELY in terms of warming (which requires at least a net positive RF), the statement in line 21 says that “However, the net RF for all anthropogenic drivers taken together is LIKELY to be positive”. Please be consistent.

RES: Accepted, text reworded, it is very likely

Appendix II
The 2007 IPCC Assessment Process – Its Obvious Conflict of Interest[3]
by Roger Pielke, Sr.

Climate Science has discussed the shortcomings, bias and errors with the 2007 IPCC Report (e.g. see[4], see[5], see[6], and see[7]). My final Climate Science posting summarizes the fundamental problem with this assessment.

If instead of evaluating research in climate, suppose a group of scientists introduced a new cancer drug that they claimed could save many lives. There were side effects, of course, but they claimed that the benefit far out weighed these risks. The government than asked these scientist to form an assessment Committee to evaluate this claim. Colleagues of the group of scientists who introduced the drug are then asked to serve on this Committee, along with the developers.

If this occurred, of course, there would be uproar of protest! This is a clear conflict of interest.

Yet this is what has happened with the IPCC process! The same individuals who are doing primary research in the role of humans on the climate system are then permitted to lead the assessment! There should be an outcry on this obvious conflict of interest, but to date either few recognize this conflict, or see that since the recommendations of the IPCC fit their policy and political agenda, they chose to ignore this conflict. In either case, scientific rigor has been sacrificed and poor policy and political decisions will inevitably follow.

In a previous climate assessment, I made a recommendation as to how to correct this defective assessment process. This is discussed in the report

Pielke Sr., Roger A., 2005: Public Comment on CCSP Report “Temperature Trends in the Lower Atmosphere: Steps for Understanding and Reconciling Differences[8]”. 88 pp including appendices,

where I wrote:

“Future assessment Committees need to appoint members with a diversity of views and who do not have a significant conflict of interest with respect to their own work. Such Committees should be chaired by individuals committed to the presentation of a diversity of perspectives and unwilling to engage in strong-arm tactics to enforce a narrow perspective. Any such committee should be charged with summarizing all relevant literature, even if inconvenient, or which presents a view not held by certain members of the Committee.

Assessment Committees should not be an opportunity for members to highlight their own research and that which supports their personal scientific conclusions without properly placing into perspective the diversity found in the peer literature. When the Chair of such a committee seeks to limit the focus of an assessment Report in a specific direction, such as was the case with this Committee, the advancement of our understanding of the scientific issues involved suffers.”

“….Unfortunately, the Report advocates a narrow perspective on science shared by the majority of the committee, rather than dealing comprehensively with the issues under its charge and found in the broader scientific literature. As such it does a disservice to those interested in a comprehensive review of the relevant science.”

We need recognition among the scientific community, the media, and policymakers that the IPCC process is obviously a real conflict of interest, and this has resulted in a significantly flawed report.

Real Climate has sought to argue that the IPCC process is transparent (see[9]). They clearly contradict themselves in their post, however, where they write

“The authors of the report used the input from the reviewers to improve the report. In some cases, the authors may disagree with the comments – after all, it is them who are the authors of the report; not the reviewers.”

This means that the authors are gatekeepers who can prevent alternative perspectives from being presented. They did exercise that power in preparing the 2007 (and earlier) IPCC Reports. The conflict of interest reported on in the current Climate Science weblog can be shown clearly in this admission from Real Climate.

[1] Intergovernmental Panel on Climate Change

[2] Koss reported 309 reviewers to climateaudit.org, one wasd duplicated under a slightly different spelling.

[3] The 2007 IPCC Assessment Process – Its Obvious Conflict of Interest

[4] Additional evidence on the bias in the IPCC WGL report on the assessment of near-surface air temperature trends

[5] Documentation of the IPCC WGL bias by Roger A. Pielke, Sr. and Dallas Staley, Part i

[6] The Failure of the 2007 IPCC WG1 Report To Perfom A Spatial Analyses of Human Climate Forcings And Their Influence on Atmospheric and Ocean Circulations

[7] Documentation of the IPCC WGL bias by Roger A. Pielke, Sr. and Dallas Staley, Part ii

[8] Public comment on CCSP report “Temperature Trends in the Lower Atmosphere: Steps for Understanding and Reconciling Differences”

[9] Transparency of the IPCC Process