In a recent post on his excellent
Hyndsight
blog, Rob Hyndman compared the results of the R forecasting package with those
of some commercial automatic forecasting software packages using data
from the M3
forecasting competition.
We don’t consider EViews to be an
automatic forecasting package, but EViews does include two of the most widely
used forecasting techniques; BoxJenkins ARIMA
models, and Error, Trend,
Season (ETS) exponential smoothing models, and includes “automatic
selection” versions of both techniques, letting EViews decide the best
specification for each.
You can view a demonstration of automatic ARIMA forecasting in EViews here:
And ETS smoothing here:
Although the underlying code that
calculates the automatic ARIMA and ETS models in EViews is not opensource, we
are open about the algorithms used, and, indeed, the ETS calculations are very
similar to those in Hyndman’s ETS module in the R forecasting package.
We thought it would be
interesting to compare the results of EViews’ automatic routines with those
provided by Dr Hyndman.
The first step we took was to
import the M3 data into EViews. The data
are arranged in a slightly complicated fashion in the original Excel file, so
we had to write an EViews script to import it all properly. You can download
the program and resulting EViews workfile.¹
Our second step was to verify the
SMAPE calculations performed by Dr Hyndman of the automatic forecasting
software package’s results from the M3 competition. We used another EViews program to do this.²
Satisfyingly, we obtained the same results as Dr Hyndman’s calculations (which means
that we too were unable to produce the originally cited SMAPE calculations),
indicating that our data import was successful.
Then our final step was to use
EViews’ ETS smoothing and automatic ARIMA routines to produce our own forecasts
(we also, following Hyndman, and modern forecasting suggested practices,
calculated the SMAPE from averaging the ETS and ARIMA forecasts). We wrote a
straight forward EViews program³
to do this, which resulted in the following results:
Routine

EViews Mean SMAPE

R Mean SMAPE

ETS

13.096

13.13

ARIMA

13.719

13.85

Average

12.775

12.88

These are very close to the results obtained by Dr Hyndman using the R forecast package, which is to be expected since the same econometric techniques are used. We contribute the slightly better performance of the EViews algorithms to differences between the internal optimization engine in EViews and that in R, and the fact that the EViews autoarma procedure detects whether data should be estimated in logs.
As well as calculating SMAPE, our
program went one step further and also calculated the number of times that the
ARMA forecast produced a better forecast (in terms of SMAPE) than ETS
smoothing, and the number of times averaging produced better than either:
ARMA better than ETS

44.29%

Average better than ETS

52.55%

Average better than ARMA

61.74%

These numbers back up the average
SMAPE numbers above – ETS smoothing produces better forecasts that ARIMA a
slight majority of the time, and averaging produces better forecasts than
either.
ETS smoothing is often cited as
producing better forecasts than ARIMA models, and our results agree with this
opinion, at least on the M3 data. They also support the theory that averaging
models can produce more accurate forecasts than simply using the best
individual forecast.
¹ Note that EViews 9 is required
to run the program.
²
Note that you will require an EViews 9 with a build date after October 2015 to
run this program (since the @smape function to calculate symmetric MAPE was
added in October 2015).
³
The default settings on the autoarma routine were used, with the exception for cases where limited number of observations meant we had to reduce the search space to a maximum of 1 AR and 1 MA term. The default ETS settings were used, with the exception that, following recommendations from the original authors, multiplicative trend and seasonal terms were disallowed.
settings
were used, with the exception that, following recommendations from the original
authors, multiplicative trend and seasonal terms were disallowed.
No comments:
Post a Comment