Automatic Forecasting of the M3 Data

In a recent post on his excellent Hyndsight blog, Rob Hyndman compared the results of the R forecasting package with those of some commercial automatic forecasting software packages using data from the M3 forecasting competition.

We don’t consider EViews to be an automatic forecasting package, but EViews does include two of the most widely used forecasting techniques; Box-Jenkins ARIMA models, and Error, Trend, Season (ETS) exponential smoothing models, and includes “automatic selection” versions of both techniques, letting EViews decide the best specification for each.

You can view a demonstration of automatic ARIMA forecasting in EViews here:

And ETS smoothing here:

Although the underlying code that calculates the automatic ARIMA and ETS models in EViews is not open-source, we are open about the algorithms used, and, indeed, the ETS calculations are very similar to those in Hyndman’s ETS module in the R forecasting package.

We thought it would be interesting to compare the results of EViews’ automatic routines with those provided by Dr Hyndman. 

The first step we took was to import the M3 data into EViews.  The data are arranged in a slightly complicated fashion in the original Excel file, so we had to write an EViews script to import it all properly. You can download the program and resulting EViews workfile.¹

Our second step was to verify the SMAPE calculations performed by Dr Hyndman of the automatic forecasting software package’s results from the M3 competition. We used another EViews program to do this.² Satisfyingly, we obtained the same results as Dr Hyndman’s calculations (which means that we too were unable to produce the originally cited SMAPE calculations), indicating that our data import was successful.

Then our final step was to use EViews’ ETS smoothing and automatic ARIMA routines to produce our own forecasts (we also, following Hyndman, and modern forecasting suggested practices, calculated the SMAPE from averaging the ETS and ARIMA forecasts). We wrote a straight forward EViews program³ to do this, which resulted in the following results:

Routine	EViews Mean SMAPE	R Mean SMAPE
ETS	13.096	13.13
ARIMA	13.719	13.85
Average	12.775	12.88

These are very close to the results obtained by Dr Hyndman using the R forecast package, which is to be expected since the same econometric techniques are used. We contribute the slightly better performance of the EViews algorithms to differences between the internal optimization engine in EViews and that in R, and the fact that the EViews autoarma procedure detects whether data should be estimated in logs.

As well as calculating SMAPE, our program went one step further and also calculated the number of times that the ARMA forecast produced a better forecast (in terms of SMAPE) than ETS smoothing, and the number of times averaging produced better than either:

ARMA better than ETS	44.29%
Average better than ETS	52.55%
Average better than ARMA	61.74%

These numbers back up the average SMAPE numbers above – ETS smoothing produces better forecasts that ARIMA a slight majority of the time, and averaging produces better forecasts than either.

ETS smoothing is often cited as producing better forecasts than ARIMA models, and our results agree with this opinion, at least on the M3 data. They also support the theory that averaging models can produce more accurate forecasts than simply using the best individual forecast.

---

¹ Note that EViews 9 is required to run the program.

² Note that you will require an EViews 9 with a build date after October 2015 to run this program (since the @smape function to calculate symmetric MAPE was added in October 2015).

³ The default settings on the autoarma routine were used, with the exception for cases where limited number of observations meant we had to reduce the search space to a maximum of 1 AR and 1 MA term. The default ETS settings were used, with the exception that, following recommendations from the original authors, multiplicative trend and seasonal terms were disallowed.

settings were used, with the exception that, following recommendations from the original authors, multiplicative trend and seasonal terms were disallowed.