[To view the Video attached to this associated with this document, please click here]
Presented by Professor J. Scott Armstrong on January 30, 2008
Good morning Chairman Boxer, Senator Inhofe, and members of the Committee. Thank you for the opportunity to speak today.
My name is Scott Armstrong. I am a professor at the Wharton School of the University of Pennsylvania.
The decision on whether or not to list polar bears under the Endangered Species Act rests heavily on forecasting. I am addressing this Committee today as a forecasting expert.
I am a founder of the Journal of Forecasting and the International Journal of Forecasting, which are the two main journals that address the subject of forecasting. I am also a founder of the International Institute of Forecasters and of the International Symposium on Forecasting.
I have published more than 50 peer-reviewed papers on forecasting. My first book on the subject, Long-Range Forecasting, was published in 1978 and revised in 1985. I am the editor of Principles of Forecasting: A Handbook for Researchers and Practitioners (2001) and the founder and co-director of the forecastingprinciples.com website. I have been working on forecasting problems for 48 years.
I would be grateful if you would direct your attention to Exhibit 1.
Based on these data spanning the years from 1979 to 2006, what forecast might you make for the rest of the 21st century?
I will come back to this exhibit near the end of my testimony.
Principles of scientific forecasting
In the mid-1990s, I set out on a project to capture all scientific knowledge about forecasting and to present that knowledge in the form of principles or guidelines. “Be conservative in situations involving uncertainty” is an example of a forecasting principle.
When I say “scientific”, I mean based on good empirical evidence. To undertake the project, I gained the cooperation of 38 top forecasting experts and 123 expert reviewers.
The forecasting principles, which currently number 140, are described in my Principles of Forecasting handbook and on the web site forecastingprinciples.com. For many years the site was first in a Google search for “forecasting.” The aim of the forecasting principles project is to disseminate scientific knowledge about forecasting in a form that people can use to produce better forecasts and thereby make better decisions.
Audit of Polar Bear forecasting
We conducted forecasting audits of two of the nine administrative reports that were prepared in 2007 to “…Support U.S. Fish and Wildlife Service Polar Bear Listing Decision.” We selected the reports Amstrup et al. and Hunter et al. as they appeared to be the primary forecasting documents.
Our concern was to establish whether the reports’ forecasts of the polar bear population over the balance of the 21st Century were the product of scientific procedures.
We found that the authors of both reports made complex sets of assumptions in order to derive their polar bear population forecasts. Critically, the authors assumed on the basis of general circulation model (GCM) forecasts that summer sea ice will diminish in the Arctic over the rest of the 21st Century. Green and Armstrong (2007) audited the procedures used by GCM climate modelers and found that their forecasts were not scientific. Forecasting sea ice conditions adds an additional level of complexity to the problem and the GCMs fail to reproduce actual sea ice conditions even when historical periods are simulated.
What if predictions about Arctic sea ice conditions over the 21st century turned out to be right? Would the polar bear forecasts then be useful? To assess this, each of the three authors of Armstrong, Green, and Soon (2008) used the forecasting audit software that is available on forecastingprinciples.com. The software prompts users to answer questions about what is being forecast and then asks questions to ascertain whether the procedures used (or proposed for use) adhere to the relevant forecasting principles. Each of us made independent judgments. For example, all three of us rated the principle “make forecasts independent of organizational politics” as a contravention because the authors were asked to prepare reports to “…Support U.S. Fish and Wildlife Service Polar Bear Listing Decision.”
Here, in Exhibit 2, are the results of our audits against relevant principles:
On average, the authors properly applied only 12% of relevant principles. In what occupations would work that follows 12% of proper procedures be considered acceptable?
The audit is the product of our judgments as to whether the procedures described in the reports properly applied forecasting principles. While we agreed among ourselves on those judgments, we realize that others may disagree and recognize that knowledge of others’ judgments would help us to improve our audit. To encourage other people to contribute, we have disclosed our own audit in full in Armstrong, Green and Soon (2008) and have issued an open invitation to audit Amstrup et al. and Hunter et al. or any other polar bear forecasting reports. We welcome judgments from others and we are willing to publish them on publicpolicyforecasting.com.
I should add that while our concern was to assess whether the forecasts in the USGS reports were scientific, the forecasting audit software is a tool that helps users to adopt best scientific forecasting practice in tackling their forecasting problems. This is the approach that we recommend to anyone who is faced with making a decision that will be judged against what happens in the future.
Example of principles being contravened: Current methods place a heavy reliance on experts’ forecasts Amstrup et al. and Hunter et al. based their forecasts heavily on their judgments (unaided by forecasting principles) of what will happen over the course of the 21st Century. (I realize that the authors used some elaborate mathematical procedures, but the point remains that the procedures did not amount to forecasting methods that had been subject to rigorous scientific testing). Relying on unaided judgment, while appropriate in some circumstances, is at odds with research findings when the situation is complex and uncertain, as it surely is in the case of long term forecasting of the polar bear population. In such situations, experts who are unaided by scientific forecasting methods are unable to make useful predictions. This finding is astonishing to many people.
I had summarized the research findings on this topic in 1978 in Long-Range Forecasting. More recently, in a two-decade-long examination of 82,361 forecasts by 284 experts in politics and economics, Phil Tetlock also found that experts who were unaided by scientific forecasting methods were no better at forecasting than people with little expertise. His findings are summarized in his 2005 book Expert Political Judgment.
Another example of what happens when principles are contravened
How did the authors of the Hunter et al. administrative report obtain forecasts of a rapidly diminishing polar bear population?
Consider again Exhibit 1. The labels that I will add show that these data represent the ice-free days in the Southern Beaufort Sea. None of the data shown in the exhibit were used by the authors to forecast the number of ice-free days over the 21st Century. Instead, they relied on a climate model (GCM) scenario of dramatically increasing numbers of ice-free days. This is in sharp contrast to what the data in the Exhibit seem to indicate.
Exhibit 3 shows the data with labels. It is reproduced from Figure 3 on page 26 of Regeher et al (2007), one of the nine USGS administrative reports.
Hunter and her colleagues then used only the five years of data indicated by the filled-in circles and polar bear population estimates for those years to conclude that there was a strong causal relationship between the number of ice-free days and the bear population growth rate.
Finally, they combined the GCM-originated ice-free-day scenario with their estimate of the effect of ice-free days (based on 5-years of data) in order to project the polar bear population over a period of nearly a century (Hunter, Figure 6). Their process ignored other influences on bear fertility and mortality.
This analysis shows another example of what happens when principles are contravened, among them:
• use all relevant data.
• use the most recent data.
• use simple forecasting methods.
• be conservative in situations of high uncertainty.
Fully disclosed and peer reviewed
We have sought peer review from researchers and we are continuing to do so. The latest version of the working paper is always available at publicpolicyforecasting.com.
We sent our paper to all those whom we cited asking them if we had cited them correctly. This led to some changes and to a few suggestions on how to better describe what they had done.
To date, there are no scientific forecasts of the polar bear population over the 21st Century. Nor are there any forecasts to suggest that a decision to list them would produce benefits. I urge further study.
A number of approaches might be considered:
1) Use a variety of forecasting methods. I have attached the Forecasting Methodology Tree to indicate the possibilities.
2) Generate alternative solutions for dealing with possible increases or decrease in the polar bear population and prepare forecasts for the effects of each potential solution.
3) Commission forecasts from independent teams.
4) Promote collaboration among experts on polar bears, climate, and forecasting. (Research has shown that much can be gained by having such “domain-experts” work with forecasting experts.)
5) Require that all forecasts be based on audited methods, and that the audits be published. I can think of no good reason why principles should be contravened. If a principle were to be contravened, the onus is on the forecaster to provide supporting evidence showing why the principle does not apply in that case.
6) Combine forecasts from all methods that pass an audit. (Combining is one of the more powerful methods in forecasting.)
Thank you again Chairman Boxer and Senators for the opportunity to present evidence on this important issue.
Amstrup, S.C, Marcot, B.G. and Douglas, D.C. (2007). Forecasting the rangewide status of polar bears at selected times in the 21st century. USGS Alaska Science Center, Anchorage, Administrative Report.
Armstrong, J.S. (1978; 1985). Long-Range Forecasting: From Crystal Ball to Computer. New York: Wiley-Interscience.
Armstrong, J.S. (2001). Principles of Forecasting: A Handbook for Researchers and Practitioners. Kluwer Academic Publishers.
Armstrong, J.S., Kesten C. Green, Willie Soon, "Polar Bear Population Forecasts: A Public-Policy Forecasting Audit", Working Paper Version 40: January 20, 2008.
Green, K.C. and Armstrong, J.S. (2007), “Global warming: forecasts by scientists versus scientific forecasts.” Energy and Environment, 18, 997-1021.
Hunter, C.M., Caswell, H., Runge, M.C., Amstrup, S.C., Regehr, E.V. and Stirling I. (2007). Polar bears in the Southern Beaufort Sea II: Demography and population growth in relation to sea ice conditions. USGS Alaska Science Center, Anchorage, Administrative Report.
Regehr, E.V., Hunter, C.M. Caswell, H. Amstrup, S.C. and Stirling, I. (2007). Polar bears in the Southern Beaufort Sea I: Survival and breeding in relation to sea ice conditions, 2001-2006. USGS Alaska Science Center, Anchorage, Administrative Report.
Tetlock, Philip E. 2005. Expert Political Judgment: How Good Is It? How Can We Know? Princeton: Princeton University Press.
Photo Credit: http://www.travelmaniac.com/polar/pg_polar00.htm
Forecasting Methodology Tree