Annual Energy Outlook Forecast Evaluation
The Office of Integrated Analysis and Forecasting has produced an annual evaluation of the accuracy of the Annual Energy Outlook (AEO) since 1996. Each year, the forecast evaluation expands on the prior year by adding the projections from the most recent AEO and the most recent historical year of data. The Forecast Evaluation examines the accuracy of AEO forecasts dating back to AEO82 by calculating the average absolute forecast errors for each of the major variables for AEO82 through AEO2003. The average absolute forecast error, which for the purpose of this report will also be referred to simply as “average error” or “forecast error”, is computed as the simple mean, or average, of all the absolute values of the percent errors, expressed as the percentage difference between the Reference Case projection and actual historic value, shown for every AEO and for each year in the forecast horizon (for a given variable). The historical data are typically taken from the Annual Energy Review (AER). The last column of Table 1 provides a summary of the most recent average absolute forecast errors. The calculation of the forecast error is shown in more detail in Tables 2 through 18. Because data for coal prices to electric generating plants were not available from the AER, data from the Monthly Energy Review (MER), July 2003 were used.
The 2003 Forecast Evaluation adds an evaluation of Energy Intensity (Table
18) and eliminates the evaluation of Net Coal Exports. During the past
few years, the Energy Information Administration (EIA) has seen increased
public interest in energy intensity, particularly as public policy issues
such as carbon dioxide emissions, technological development, impacts of
structural changes on the economy, and national energy security, are more
openly discussed and evaluated. Given this increased interest, EIA believed
energy intensity consumption to be important variables to examine. For
purposes of the Forecast Evaluation, energy intensity is defined as the
ratio between two output variables of the model: total energy consumption,
measured in British thermal units (Btu), and gross domestic output (GDP)
in nominal dollars. The decision to exclude Net Coal Exports (Table
12 in prior Forecast Evaluations) from current and future Forecast Evaluations
reflects the relatively small role that coal exports play in overall U.S.
energy trade.
The underlying reasons for deviations between the projections and realized
history tend to be the same from one evaluation to the next. The most
significant are as follows:
- Over the last two decades, there have been many significant changes in laws, policies, and regulations that may or may not have been predictable and were not assumed in the projections prior to their implementation based on EIA’s statutory requirement to be policy neutral. Many of these actions have had significant impacts on energy supply, demand, and prices.
- While energy price forecasts in the AEOs published in years 1994 to 2003 have improved, petroleum, natural gas, coal, and electricity price projections continue to be more difficult to accurately forecast than consumption, production, and net imports. Energy price forecasts are done in real prices, but for purposes of the Forecast Evaluation, they are re-inflated to nominal dollars using the forecast deflator of a given AEO forecast and compared to the nominal prices listed in the AER. As a general rule, the rate of increase in nominal energy prices has been overestimated rather than underestimated. More rapid technological improvements, the erosion of the market power of the Organization of Petroleum Exporting Countries starting in the mid-1980s, excess productive capacity, and market competitiveness are all factors that led to lower energy prices than projected. In the 1980s and 1990s, productivity and technology improvements, as well as the effects of gradual deregulation and changes in industry structure, have offset the factors known to result in higher energy prices, such as resource depletion and growing energy demand. However, the tendency to overestimate energy prices may be stabilizing or even reversing. In more recent years, better adherence by the oil-producing countries to crude oil production quotas, evidence of increasing rates of depletion in producing fields (natural gas and oil), short-term supply constraints (natural gas), or the unexpected consequences of deregulation policies (electricity), have led to generally higher prices than expected only a few years ago.
- Technological improvements in both the production and use of energy have had a significant impact on the price, supply, and consumption of energy. For the most part, earlier AEOs assumed much slower technology development than actually occurred, accounting for some of the deviation between the forecasts and history. This influence can be seen in the forecast evaluation tables.
- The level of future electricity sales has been consistently underestimated in all years in each of the AEOs published between 1994 and 2001. The underestimation, which averaged less than 3 percent, implies that the fuels consumed to generate electricity (e.g., coal, oil, natural gas) were also underestimated. Forecasting errors related to electricity sales have increased from 1.7 percent in the 1998 Evaluation to 2.4 percent in this year’s Evaluation. These errors are, primarily, driven by electricity sales growth projections that proved conservative in AEO1994, AEO1995, and AEO1996 for the years 1996 through 2002. Electricity sales increased 12 percent during this period. There have been many changes in the structure of electricity markets and growing sources of uncertainty in the industry over the last decade may help to explain this underestimation. Such factors include deregulation of electric generation, the proliferation of computer equipment and consumer electronics, growth in Internet use, and the expansion of the information technology sector during the economic boom of the 1990s.
- The underestimation of electricity growth over the 1996 to 2002 forecast period had an important impact on the coal consumption forecasting error over this period. About 90 percent of the demand for coal results from electricity generation and an underestimation of electricity sales results in an underestimation of coal consumption. As a consequence, coal consumption has consistently been underestimated in the AEO. The average absolute percent error for the forecast of coal consumption has ranged between 3 and 4 percent.
- Overestimation of world oil prices, particularly in earlier AEO publications, resulted in upward pressure on production/import forecasts, while underestimation of exploration and production technological improvement put downward pressure on production forecasts. In addition, high world oil price forecasts tended to lower oil consumption forecasts relative to what actually occurred. This problem was exacerbated by the unexpected shift in consumer preference to larger vehicles.
- Natural gas generally has been the fuel with the least accurate forecasts in consumption, production, and prices. As regulatory reforms that increased the role of competitive markets were implemented beginning in the mid-1980s, the behavior of natural gas in competitive markets was especially difficult to predict. In the earlier forecasts, EIA’s technology improvement expectations proved conservative, as technological advances made natural gas less costly to produce, while in the more recent forecasts, EIA overestimated technology improvements. After natural gas curtailments were eased in the mid-1980s, environmental pressures made natural gas an increasingly attractive fuel source, particularly for electricity generation. Historically, natural gas price instability was strongly influenced by natural gas resource estimates, which steadily rose, and by the world oil price, which was subject to its own error. Despite growing uncertainty regarding the ability of future natural gas production to meet demand, natural gas wellhead price projections, as shown in Table 1, have steadily improved from an average absolute percent error of 72.1 in the 1998 Evaluation to 57.3 percent in the 2003 Evaluation.
- External factors such as severe weather, economic cycles, and strikes have also had an impact on energy markets; however, these events cannot be anticipated in the mid- to long-term period and are not captured in the models underlying the AEO projections.
- The carbon dioxide emissions error increased from 1.8 percent to 2.6 percent in 2003. This error is linked to overestimation of GDP growth during for years 2000 to 2002 in AEO2001 to AEO2003.
Table 1 shows the forecast errors from previous and current forecast evaluations. As indicated in Table 1, the forecasts of consumption, production, and carbon dioxide emissions have generally been the most accurate while the forecasts of prices and net imports have been less accurate. The percent errors change from one year’s evaluation to the next as an additional year of data and projection is added. The percent errors may also change due to data revisions in the AER and the MER.
Both the passage of time and the lengthening of the maximum AEO projection horizon1 over the 22 years that the AEO has been published mean that the average interval between issuance of a projected quantity and the realization of the actual value to which it is compared for forecast evaluation purposes is increasing with successive editions of this document. For example, for 1985 data, the average interval between forecast and actual values is 2 years. In contrast, for the 2002 data, which is new to this year’s evaluation, the average interval between forecast and realization is 6 years. Because sources of error and uncertainty can be expected to increase as the forecast horizon lengthens, one might expect forecast performance measured in terms of average absolute error to degrade as the share of long-horizon projections in the forecast evaluation grows. Improvements in forecasting methods, however, work in the other direction, and the results reported in Table 1 reflect both of these effects.
An examination of the data in the last column in Table 1 suggests an improvement in some AEO forecast categories, particularly in forecasts of natural gas production, net petroleum imports, world oil prices, and natural gas wellhead prices. The improvement in the forecasts of world oil prices and natural gas wellhead prices is particularly striking. The average absolute percent error in natural gas wellhead prices decreased from 72.1 percent in the 1998 Forecast Evaluation to 57.3 percent in the 2003 Forecast Evaluation. The error in world oil prices decreased from 51.3 percent in the 1998 Evaluation to 48.9 percent in the current Evaluation. On the other hand, a number of energy consumption categories show higher errors, including total energy, natural gas, and coal. For the most part, the percent error remains similar or improves over time.
1The forecast horizon of AEO82 was 8 years. AEO86 increased the maximum horizon for annual forecasts to 15 years, while AEO98 extended it to 22 years.