| Most Recent |
|---|
| Peer review? What peer review? |
|
|
| Written by John McLean | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Thursday, 06 September 2007 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Page 2 of 2 Part 4 - Authors or Reviewers?The figures shown in part 2 apply to the number of reviewers as whole. These include the reviewers operating under government auspices (discussed in Part 3) but also numerous reviewers who were among the team of authors for the chapters in question. There may be legitimate reasons for a contributing author of a subsection to review other sections of the same chapter but the impression one gets is these author-reviewers were unable to raise their points within the internal communication of the IPCC authoring teams. How else does on explain that a Lead Author of chapter 2 made review comments about that chapter or that one author made 282 comments on his only chapter or that 3 authors of chapter 11 made a total of 350 review comments on that chapter. In total 30 author-reviewers made all of their comments about chapters that they authored Table 6 shows the number of authors and reviewers for each chapter. Six chapters had fewer reviewers than authors but that increases to 8 when author-reviewers are excluded.
Table 6- Number of authors and reviewers for each chapter
Six chapters were reviewed by fewer than 50 individuals who were not authors of that chapter, but in each case that number of reviewers might include authors of other chapters. The total number of authors and reviewers of the WG I report is misleading because several individuals were authors of more than one chapter and several authors were also reviewers. Authors who reviewed: 95 In total 826 individuals contributed to the WG I report but only slightly more than 25% were reviewers, and as we have seen, the contribution by reviewers was often very minor. Part 5 - Number of Comments and Rejections
Commenting is only part of the picture because those comments could be ignored by the editors. Determining the number of rejected comments is difficult because the expressions of rejection come in many forms, the rejection may only be partial or the comments may be made irrelevant by sections of text being rewritten, deleted or restructured. A simple analysis based on the occurrence of three key words - "rejected", "reject" and "disagree" - provides a likely minimum number of rejected comments because other words may be used. This somewhat crude analysis reveals that the minimum number of rejected comments averages 25% of all comments on the SOR of a chapter and ranges from 9.5% to 58.1% (Table 6, Figures 5 & 6).
Table 7 - Summary of total comments and the likely minimum number of rejected
comments. ( "No rev" indicates that no review took place). Figure 5 - Accepted and rejected comments for the SOR( based on minimum rejected) Figure 6 - percentage of rejected comments (based on minimum rejected) The striking feature of many rejections is their dubious nature. Some responses were banal and others showed inconsistencies with other comments. Reviewers had to justify their requested change but the responding editors appear to have been under no such obligation. One reviewer said that "best estimate" should more correctly be "most recent estimate" but the editors changed the text to "current best estimate". Reviewers were sometimes flatly told they were wrong but no reasons or incontrovertible references were provided. Another said that one heat wave did not make a trend but the editors rejected this by claiming they used that heat wave as an example. Too bad if the passage was taken out of context and that heat wave being interpreted as due to climate change when contradictory evidence and expert statements at that time said otherwise. In other cases reviewers tried to dilute the certainty being expressed and they often provided supporting evidence, but their comments were often flatly rejected. Some comments were rejected on the basis of a lack of space and it seems incredible that space should have been a constraining factor on such an important document. Reviewers would cite references but be told that a greater number of references supported an alternative argument. Reviewers would make a brief statement of correction but be told of just one paper that contradicted that claim. In at least one response the editors made referred only to a document that has not been subjected to peer-review. The attitude of the editors seems very much to be that simple corrections will be accepted, requests for improved clarity be tolerated but the assertions and interpretations that appear in the text were to be defended against any challenge. Part 6 - On the Attribution of Climate Change
Chapter 9 is the single most important chapter of the entire report because it is where the IPCC states, "it is very highly likely that greenhouse gas forcing has been the dominant cause of the observed global warming over the last 50 years". The IPCC leads us to believe that this statement is supported by a large number of reviewers. We often hear reference to 2,500 scientists supporting the IPCC's findings but that number supposedly includes about 1,500 acting as chapter editors. Earlier it was shown that a total of 308 reviewers, individuals or government appointees, reviewed parts of the WG I report but even that figure is far higher than the number of reviewers for chapter 9. In fact only 62 reviewers commented on this chapter, of which 8 were noted as government reviewers. Nineteen reviewers made just 1 comment and 18 made between 2 and 5 comments, and those 37 reviewers are 60% of the total. Just 10 reviewers made more than 20 comments for this, the most important chapter of the entire report, and yet some of these were typographical errors that were missed by many reviewers. Of the 54 individual reviewers 31, i.e. more than half, had a vested interest in this chapter of the report: Three were editors of the entire IPCC WG I report, 7 were from the chapter's 44 contributing authors and 1 was noted "WGI TSU", which indicates some link to the IPCC team. Twenty-six appear to be authors or co-authors of papers cited in this chapter and 10 of the 54 made explicit reference to papers that they had written or co-written. In some instances we even find reviewers who fell into 2 or 3 of the above categories. Just 23 individual reviewers and 8 government reviewers appear to have no vested interest in chapter 9 of the WG I report, the remaining 31 being tainted in some way. A total of 1158 comments were made but just one reviewer made 572 comments, or 49.4% of the total. The government of the USA made 113 comments (9.8%) but many comments by the US government duplicate the 32 comments made by an individual reviewer. The majority of those 572 comments from one reviewer appear to strike at four contentious issues. First there is the corruption of the generic meaning of "climate change" into "man-made climate change"; second the matter of whether urban heat islands, which the reviewer often refers to as the proximity of measuring equipment to human induced heat, are distorting the temperature record; thirdly the discrepancy between tropospheric temperature changes and surface; and fourthly the impact of El Nino events on any trend. The IPCC editing team rejects the above points claiming firstly no distortion in the meaning of "climate change", contradicted of course by the IPCC's name including the words "climate change" and yet being focused on a human influence on climate. Secondly it argues that there is no evidence that human induced outputs of heat have corrupted the data and on the third point it refers to papers that dispute the tropospheric temperature record but on both matters ignores the absence of any verification of the accuracy of near-surface temperature records. The IPCC consistently claims that El Nino events are internal to the climate system but seem to forget that their occurrence in the tropics makes them come under the influence of solar radiation and that subsea volcanic activity may be contributing. On many occasions the IPCC claimed a numerical superiority of papers that supported its line of argument and referred to comments in the Third Assessment Report (TAR) of 2001. The clear implications were that a consensus of papers is one determining factor and that whatever was said in the TAR must be correct, although one suspects that the TAR was likewise disputed by reviewers. The most risible of the IPCC's responses is "Rejected. The ability of models to simulate the temperature variations indicates that any missing natural forcings have little impact." Apparently the IPCC believes that if the output of the models is approximately correct then the internal workings of the model must likewise be correct. Perhaps the IPC is unaware that if a model based on a factor that is driven by temperature rather than drives it, that model will be false but will probably produce output that matches historical data. It is clear that the 572 comments by this one reviewer were not frivolous but addressed some very significant core issues, so one wonders why other reviewers did not make similar comments. Of the remaining comments, and discounting the duplication under the name of the United States government, 99 of 554 comments were rejected, which is still more than 1 in 6. Genuine support for the chapter is difficult to ascertain because comments would be accepted or rejected on their individual merits, nullified by changes to passages of text, and 5 comments appear to be only partially accepted. Unless support is explicitly stated a subjective evaluation is required as to either the tone of the remarks or whether the absence of comment implies support. Only 4 reviewers with no vested interest explicitly supported the overall chapter although others praised specific sections. Among the majority with a vested interest only 4 reviewers specifically praised the text. The IPCC implies that the majority of hundreds of reviewers endorsed the claim that humans had very likely been responsible for the majority of the warming in the last 50 years. This analysis of chapter 9, the key chapter to the WG I report and indeed to the entire Fourth Assessment Report, reveals that implication to be entirely false. Part 7 - A Final Word
The IPCC states clearly that it undertakes no research of its own but merely relies on published papers for its information. A team of editors assesses those papers and writes the drafts of the various report chapters. While minor corrections are welcomed the overall assessment is strongly defended against challenges. On the surface this looks not unreasonable but scratch a little deeper and the self-sustaining nature of the claim of a human influence on warming becomes visible. Unlike other high-profile scientific fields, these reports by the IPCC are almost entirely responsible for determining the direction of climatology and how the research funding will be spent. The IPCC's Third Assessment Report (TAR) of 2001 showed that 8 of 11 climate factors were poorly understood but despite this it claimed that humans were responsible for rising temperatures. As a consequence of the TAR the majority of funding for climatology research went to projects that assumed a human influence on climate. Not surprisingly this caused the papers taking this position to significantly out-number the papers that rejected this hypothesis. But as the responses to reviewers' comments show, the number of papers supporting a certain argument is a critical factor in determining the content of the IPCC reports. It is not merely the weight of numbers that tilts the balance but also the leanings of the editors. The content of the reports rests with the teams of editors but if those editors are actively engaged in research then it is likely to be on projects which assume a human influence on climate and this will make those editors susceptible to being predisposed to view climate in that light. There is not the evidence to claim deliberate bias but logically the "anthropogenic warming" argument will be very familiar to many editors and the tendency will be that papers following that line will receive less intense scrutiny than papers that don't only challenge that argument but also challenge the editors' own beliefs. If an editor took the position that the human influence on climate is negligible or non-existent on anything but a small and localised scale then that person's research opportunities are likely to be few. The same potential conflict of interest arises with the reviewers, many of whom are authors of papers related to climatology and are quite possibly still involved in research projects. The reviewers have the added problem that the IPCC practice is to make all reviewers' comments available to other reviewers. Reviewers cannot hide behind some kind of editorial team "group think" but are exposed to individual scrutiny and that can put reputations and research opportunities at even greater risk. The problems continue into the authorship of these reports. According to IPCC documents, scientists are nominated by governments or explicitly invited by scientists who were already associated with the IPCC. What a wonderful way to position scientists who support a government agenda on climate and then fill out the IPCC with like-minded individuals. The bigger picture is that research funding indirectly determines the content of the IPCC assessment reports, and those assessment reports play a very significant role in determining the direction and funding of the research. Who would be a reviewer when many chapter authors will be likely to defend the beliefs and reputations they have established via research projects funded by government money on the supposition that anthropogenic global warming is a fact? Few researchers who are funded by the anthropogenic warming gravy-train are likely to review IPCC chapters with the intent of identifying flaws only those sceptical of the claims, and have little to lose in the way of reputation or funding, will make the effort. Several recognised sceptics of man-made warming failed to take part in the review but who can blame them when the exercise is so evidently futile. In the long term this perpetual and increasing marginalising of contrary viewpoints is extremely detrimental to the science because it will produce a supposed scientific "truth" based on little more than the emphasis of the funding and the domination of certain opinions. Key Points
The review of the Working Group 1 report was far less intense than the IPCC has implied. - 308 reviewers examined the chapters of the Second Order Revision (i.e. penultimate draft) of the Working Group 1 report, with the average number of reviewers per chapter being 67 (minimum 34, maximum 100). Only 22 governments had designated reviewers but 5 of these commented on only one chapter and 5 averaged less than 3 comments per chapter. The United States of America and Australia, both non-signatories to the Kyoto Agreement, commented on all 5 chapters and made the greatest number of comments. On average the editors rejected at least 25% of those reviewers' comments for any chapter but many of those rejections are contentious. The critical chapter, that which attributed recent warming to human activity, was reviewed by 54 individual and 8 government representatives but almost 1/3rd of reviewers made just one comment. - 31 of the 54 had a vested interest in the report, as editors or having papers cited Among the 23 independent reviewers just 4 explicitly endorsed the chapter with its hypothesis of a significant human influence on climate, and one other endorsed only a specific section. The reviewers' comments show that is actually little support for the IPPC's contention that anthropogenic emissions of carbon dioxide have caused warming. The IPCC reports appear to be largely based on a consensus of scientific papers, but those papers are the product of research for which the funding is strongly influenced by previous IPCC reports. This makes the claim of a human influence self-perpetuating and a corruption of the normal scientific process. Bio John McLean is climate data analyst with an extensive background in the IT industry. He became interested in the question of climate change when told of evidence that directly refuted the frequent claim that recent temperatures were unprecedented. He lives in Melbourne, Australia, and is a member of both the "Climate Sceptics" and "New Zealand Climate Science Coalition" Internet discussion groups. Appendix I
This appendix contains just some of the less banal comments (i.e. minor corrections or praise to a chapter), which arbitrarily come from chapters 1, 2, 3, 9 and 10, in order to provide some impression of the types of comments and responses.. Comments are shown here as they appeared (i.e. without spelling or grammar corrections). They are shown in full unless otherwise noted. Where reviewers' comments are shown without a response the reference number is provided but where responses are included no number is shown. Where I have appended comments for this document those comments are indicated in bold and within [and]. Part 5(a) - Sample Reviewers' Comments (without responses)
1.01 "Solar radiation is the driving force of the climate system." This could be taken out of context. Consider changing "the driving force" to "a driving force. (1-435) 1.02 Understanding will necessarily have "evolved" since the TAR, but more to the point has it markedly improved? (2-424) 1.03 "better understood" - I would say that the trend in methane is not better understood - and indeed is a bit of a mystery. (2-244) 1.04 I would encourage IPCC to consider having only one solar physicist on the lead author team of such an important chapter. In particular since the conclusion of this section about solar forcing hangs on one single paper in which J. Lean is a co-author. I find that this paper, which certainly can be correct, is given too much weight. [part only] (2-901) [J Lean was a Lead Author of the relevant chapter] 1.05 DELETE THE ENTIRE MATERIAL BEGINNING WITH "IN ADDITION" as all of this is highly contentious has all sorts of implicit ethical and moral judgments which you have not even begun to address, and goes way beyond the core science, which is the only thing the WG1 should deal with. (2-1026) 1.06 As written it implies 100% attribution, which is misleading, since the idea that all climate change is attributable to GHG forcing is an extreme position held by few if any experts. Insert "partially" after the word "been" and before "attributed". This suggestion was made in the FOD review and ignored. It is hereby repeated, for the same reason: the present wording is deliberately misleading. (3-223) 1.07 This conclusion comes out of nowhere! After reading the past two-to-three pages about differing precipitation, soil moisture, and stream flow trends all over the place, I was quite surprised to read "The global increase in both sever drought and large floods suggest that hydrologic conditions have become more extreme." Apparently my definition of “global” is quite different from yours. (3-421) 1.08 You MUST insert here a proper Figure showing the radiosonde records, preferably those from Figure 9 of the paper of Thorne et al (2005). Figure 3.4.2 is deliberately designed to conceal the true facts about both the radiosonde and the MSU records. The pretence that these three records are virtually identical is a plain lie. [part only] (3-467) 1.09 This is pure speculation. The sondes in these studies have not been corrected for instances where spurious warming occurs as shown in for example in Christy and Norris 2004, Christy and Spencer (2005) and the other papers to appear soon. (3-543) 1.10 1998 is quoted here as the warmest year for the global mean, without qualification. This is at odds with page 3-3, lines 15 to 19, which point out that NCDC and GISS have 2005 warmer than 1998, in contrast to the CRU/UKMO estimate. (3-702) 1.11 You claim that Turner et al. (2005) found '... a cooling over much of the rest of the continent'. But that paper was only concerned with station data and there are only two stations with long records in the interior of the Antarctic. In that paper we were careful to point out that few of the annual temperature changes around East Antarctic were statistically significant. Only South Pole has a statistically significant cooling in the annual data. (3-728) 1.12 Seems odd to say that the figure is not shown because it is not reliable, yet then discuss it for several more sentences. Why should we conclude that the discussion is reliable? (3-877) 1.13 How does a study of only a half century of data distinguish interdecadal (e.g.30 year and longer timescale) variability in one phenomenon from other potentially related or unrelated trends in other phenomena. Implausible claims such as this, especially those which rest on one study of half a century of reanalysis data, should not serve as the basis for conclusions in an assessment report. (3-797) 1.14 I found this discussion of "selection bias" confusing. "Fingerprinting" results in a different kind of selection bias, in that only those patterns predicted by the model responses are looked for. One wouldn't have noticed the ozone hole if one followed this kind of program religiously. [part only] (9-193) 1.15 Please explain for the reader how to understand the apparently high confidence in detection in certain regions where there is very little data over the full 20th century as shown in figure 9.4.2. How is it that you can divide the globe so finely when you have only a few data points in some of these regions over the full 20th century? [part only] (9-591) 1.16 You may need to suitably denigrate our work to justify your conclusion, but you could mention that at least some people strongly disagree with your claims! (10-987) Part 5(b) - Reviewers' Comments and Editors' Responses
2.01 REV: Delete "of the risk of" The study is to find out IF there is a risk at all. You should not assume that there IS a risk. RES: Rejected: we think there is no ambiguity in the statement as it is. 2.02 REV: Suggest deleting the two sentences "The glass walls….of the planet" is unnecessary and potentially confusing to most non-expert readers. RES: Rejected . These facts explain the name of the greenhouse effect. 2.03 REV: Sentence should read: "carbon dioxide or water vapor has only a small direct..." RES: Noted but not taken into account 2.04 REV: Is the "best estimate" a good choice of words? If I read the text I would rather say "most recent estimates". RES: Accepted. Changed to ‘current best estimate’. [but this retains the questioned use of the word "best"] 2.05 REV: This paragraph is too generalized - and does not apply to large land areas in the Southern Hemisphere. RES: Rejected. Nor does it refer to general land areas in the southern hemisphere. It does refer to South America. 2.06 REV: The title is not corresponding to the content. It has to be replaced RES: Noted. It doesn’t have to. 2.07 REV: This statement is NOT TRUE. Their plot shows a flattening of the number within the last two 5-year periods. [part only] RES: Noted. Changes made. 2.08 REV: Probably overstates the certainty of their conclusions. RES: Noted. Text retained as we believe it is correct 2.09 REV: The references to Trenberth et al. (2000) and Trenberth and Stepaniak (2003a,b) are not necessary as this basic information on the Hadley Circulation is dealt with in text books and numerous other journal publications. RES: Modifted. This is not true: none of this is in text books anywhere!!!! 2.10 REV: The text here states that GHG forcing is smaller than the indirect effect of aerosol - this therefore implies that the net anthropogenic forcing is negative, which is at odds with the statement on pg 67, ln 17, that humans have very likely exerted a net warming influence on climate. RES: Accepted, paragraph is modified. 2.11 REV: Insert after "corrections"," But all of them show a zero temperature trend between 1978 and 1998". RES: Rejected - no reason given for suggested change The reviewer is taking a biased stance by deliberately selecting a minimum-trend period. [Are the editors taking a biased or unbiased stance?] 2.12 REV: Replace ."lead to important" by "suggest" RES: Agree wording is not perfect. Replacing “lead to” with “have resulted in” 2.13 REV: Replace "are shown to " by "may". RES: Wording changed to “are projected to” [which is quite different to "may"] 2.14 REV: Most of the evidence suggests the opposite—increased heating at the surface relative to the troposphere. There is some suggestion that the trends in the troposphere may be underestimated (Sherwood et al.) but the corrections have not been made and thus the ultimate outcome is unknown. RES:Rejected. We are working with the CCSP report. [The CCSP report was not peer-reviewed] 2.15 REV: If the data isn't good enough to conclude anything from 1979 to the present, how can we really conclude anything from 1958 to the present? RES: Rejected. Over longer periods there can be a smaller influence of error. [But is this true in this case?] 2.16 REV: I find that it is “very likely that greenhouse gas forcing has been the dominant cause” difficult to reconcile with “it is highly likely that warming... cannot be explained without external forcing”. RES: Rejected. The second statement is less specific so should have a higher confidence associated with it. 2.17 REV: The example doesn't really help. Perhaps say, "Extreme events can occur in an unchanging climate." RES: The comment indicates that the reviewer does not really understand the statistical point that is being made here.... 2.18 REV: One "heat wave" does not make a "trend" RES: Rejected. The European heat wave is just a single example and this is clear in the current text. 2.19 REV: I find the statements in the second and third (non title) rows hard to reconcile. It seems to suggest that if greenhouse gas forcing has been involved then it has to be dominant, i.e. there is no room for it to be a minor contributor. RES: Noted. We don’t quite see the difficulty. The assessment is that greenhouse gas has been the dominant contributor. 2.20 REV: Please be precise on whether the net RF is LIKELY or VERY LIKELY positive since 1750. Whereas line 17 states VERY LIKELY in terms of warming (which requires at least a net positive RF), the statement in line 21 says that "However, the net RF for all anthropogenic drivers taken together is LIKELY to be positive". Please be consistent. RES: Accepted, text reworded, it is very likely Appendix II
The 2007 IPCC Assessment Process - Its Obvious Conflict of Interest[3] by Roger Pielke, Sr. Climate Science has discussed the shortcomings, bias and errors with the 2007 IPCC Report (e.g. see[4], see[5], see[6], and see[7]). My final Climate Science posting summarizes the fundamental problem with this assessment. If instead of evaluating research in climate, suppose a group of scientists introduced a new cancer drug that they claimed could save many lives. There were side effects, of course, but they claimed that the benefit far out weighed these risks. The government than asked these scientist to form an assessment Committee to evaluate this claim. Colleagues of the group of scientists who introduced the drug are then asked to serve on this Committee, along with the developers. If this occurred, of course, there would be uproar of protest! This is a clear conflict of interest. Yet this is what has happened with the IPCC process! The same individuals who are doing primary research in the role of humans on the climate system are then permitted to lead the assessment! There should be an outcry on this obvious conflict of interest, but to date either few recognize this conflict, or see that since the recommendations of the IPCC fit their policy and political agenda, they chose to ignore this conflict. In either case, scientific rigor has been sacrificed and poor policy and political decisions will inevitably follow. In a previous climate assessment, I made a recommendation as to how to correct this defective assessment process. This is discussed in the report Pielke Sr., Roger A., 2005: Public Comment on CCSP Report “Temperature Trends in the Lower Atmosphere: Steps for Understanding and Reconciling Differences[8]”. 88 pp including appendices, where I wrote: “Future assessment Committees need to appoint members with a diversity of views and who do not have a significant conflict of interest with respect to their own work. Such Committees should be chaired by individuals committed to the presentation of a diversity of perspectives and unwilling to engage in strong-arm tactics to enforce a narrow perspective. Any such committee should be charged with summarizing all relevant literature, even if inconvenient, or which presents a view not held by certain members of the Committee. Assessment Committees should not be an opportunity for members to highlight their own research and that which supports their personal scientific conclusions without properly placing into perspective the diversity found in the peer literature. When the Chair of such a committee seeks to limit the focus of an assessment Report in a specific direction, such as was the case with this Committee, the advancement of our understanding of the scientific issues involved suffers.” “….Unfortunately, the Report advocates a narrow perspective on science shared by the majority of the committee, rather than dealing comprehensively with the issues under its charge and found in the broader scientific literature. As such it does a disservice to those interested in a comprehensive review of the relevant science.” We need recognition among the scientific community, the media, and policymakers that the IPCC process is obviously a real conflict of interest, and this has resulted in a significantly flawed report. Real Climate has sought to argue that the IPCC process is transparent (see[9]). They clearly contradict themselves in their post, however, where they write “The authors of the report used the input from the reviewers to improve the report. In some cases, the authors may disagree with the comments - after all, it is them who are the authors of the report; not the reviewers.” This means that the authors are gatekeepers who can prevent alternative perspectives from being presented. They did exercise that power in preparing the 2007 (and earlier) IPCC Reports. The conflict of interest reported on in the current Climate Science weblog can be shown clearly in this admission from Real Climate. [1] Intergovernmental Panel on Climate Change [2] Koss reported 309 reviewers to climateaudit.org, one wasd duplicated under a slightly different spelling. [3] The 2007 IPCC Assessment Process - Its Obvious Conflict of Interest [4] Additional evidence on the bias in the IPCC WGL report on the assessment of near-surface air temperature trends [5] Documentation of the IPCC WGL bias by Roger A. Pielke, Sr. and Dallas Staley, Part i [6] The Failure of the 2007 IPCC WG1 Report To Perfom A Spatial Analyses of Human Climate Forcings And Their Influence on Atmospheric and Ocean Circulations [7] Documentation of the IPCC WGL bias by Roger A. Pielke, Sr. and Dallas Staley, Part ii [8] Public comment on CCSP report "Temperature Trends in the Lower Atmosphere: Steps for Understanding and Reconciling Differences" [9] Transparency of the IPCC Process |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| < Prev | Next > |
|---|
