Wednesday, July 26, 2017

Hamilton’s “Why you should never use the Hodrick-Prescott Filter”

Professor James D. Hamilton requires no introduction, having been one of the most important researchers in time series econometrics for decades.
Over the past few years, Hamilton has been working on a paper calling on applied economists to abandon the ubiquitous Hodrick-Prescott Filter and replace it with a much simpler method of extracting trend and cycle information from a time series.
This paper has become popular, and a number of our users have asked how to replicate it in EViews. One of our users, Greg Thornton, has written an EViews add-in (called Hamilton) that performs Hamilton’s method.  However, given its relative simplicity, we thought we’d use a blog post to show manual calculation of the method and replicate the results in Hamilton’s paper.

The Hodrick-Prescott Filter

The HP filter is a mainstay of modern applied macroeconomic analysis. It is used extensively to isolate trend and cycle components from a time series.  By isolating and removing the cyclical component, you are able to analyze the long-term effects of or on a variable without worrying about the impact of short term fluctuations. In macroeconomics this is especially useful since many macroeconomic variables suffer from business-cycle fluctuations.
Mathematically, the HP filter is a two-sided linear filter that computes the smoothed series $s$ of $y$ by minimizing the variance of $y$ around $s$, subject to a penalty that constrains the second difference of $s$. That is, the HP filter chooses $s$ to minimize:
$$\sum_{t=1}^T\left(y_t - s_t\right)^2 + \lambda \sum_{t=2}^{T-1}\left((s_{t+1} - s_t) - (s_t - s_{t-1})\right)^2$$
The arbitrary smoothing parameter $\lambda$ controls the smoothness of the series $s$. The larger the $\lambda$, the smoother the series. As $\lambda=\infty$, $s$ approaches a linear trend.

Hamilton’s Criticisms of the HP Filter

Hamilton outlines three main criticisms of the HP filter:

1. The HP filter produces series with spurious dynamic relations that have no basis in the underlying data-generating process.
2. Filtered values at the end of the sample are very different from those in the middle, and are also characterized by spurious dynamics.
3. A statistical formalization of the problem typically produces values for the smoothing parameter vastly at odds with common practice.

Hamilton’s Method

Hamilton proposes an alternative to the HP Filter that uses simple forecasts of the series to remove the cyclical nature.  Specifically, to produce a smoothed estimate of Y at time t, we use the fitted value from a regression of Y on 4 lagged values of Y back-shifted by two years (so 8 observation in quarterly data), and a constant. Specifically:
$$\widetilde{y}_t = \alpha_0 + \beta_1 y_{t-8} + + \beta_2 y_{t-9} + \beta_3 y_{t-10} + \beta_4 y_{t-11}$$

An Example Using EViews.

Professor Hamilton provides some examples using employment data, in the csv file employment.csv.  Specifically, the file contains quarterly non-farm payroll numbers, both seasonally adjusted and non-seasonally adjusted between 1947 and 2016Q2.
We can open that file in EViews simply by dragging it to EViews.  The file doesn’t have a date format that EViews understands, so we will manually restructure the page to quarterly frequency with a start date of 1947Q1.

We then give the two time-series names, and create growth rate series.

Having created the series we’re interested in, we’ll first perform the HP filter on the seasonally adjusted series. We open the series, click on Proc->Hodrick-Prescott Filter.  We enter names for the outputted trend and cycle series, and then click OK.

Now we’ll replicate Hamilton’s method.  We first need to regress the series against four lags of itself shifted 8 periods back.  We do this using the Quick->Estimate Equation menu, then entering the specification of NFP_LOG C NFP_LOG(-8 TO -11)

We can then view the residuals and fitted values, which correspond to the cyclical and trend components from the View menu.
If we wanted to save those components, we could use the Proc->Make Resids and Proc->Forecast menu items to produce the residuals and fitted (forecasted) values.

We’ve written a quick EViews program that automates this process for both the seasonally adjusted and non-seasonally adjusted data, and replicates Figure 5 from Hamilton’s paper.  The program produces the following graphs, and the code is below:

'open data
wfopen .\employment.csv
'structure the data and rename series
pagestruct(freq=q, start=1947)
d series01
rename series02 emp_sa
rename series03 emp_nsa
'calculate transforms of series
series nfp_log = 100*log(emp_sa)
series nsa_log = 100*log(emp_nsa)

'hp filter of employment (seasonally adjusted)
nfp_log.hpf nfp_hptrend @ nfp_hpcycle

'hp filter of employment (non seasonally adjusted)
nsa_log.hpf nsa_hptrend @ nsa_hpcycle

'estimate employment (seasonally adjusted) regressed against constant and 4 lags of itself, offset by 8 periods.
equation eq1.ls nfp_log c nfp_log(-8 to -11)
'store resids as the cycle
eq1.makeresid nfp_cycle
'store fitted vals as the trend
eq1.fit nfp_trend

'estimate employment (non-seasonally adjusted) regressed against constant and 4 lags of itself, offset by 8 periods.
equation eq2.ls nsa_log c nsa_log(-8 to -11)
'store resids as the cycle
eq2.makeresid nsa_cycle
'store fitted vals as the trend
eq2.fit nsa_trend

'calculate 8 period differences
series nfp_base = nfp_log-nfp_log(-8)
series nsa_base = nsa_log-nsa_log(-8)

'display graphs of Hamilton's method (replicate Figure 5)
freeze(g1) nfp_log.line
freeze(g2) nsa_log.line
group nfps nfp_cycle nfp_base
freeze(g3) nfps.line
g3.setelem(1) legend(Random Walk)
g3.setelem(2) legend(Regression)
g3.legend columns(1) position(4.67,0)
group nsas nsa_cycle nsa_base
freeze(g4) nsas.line
g4.setelem(1) legend(Random Walk)
g4.setelem(2) legend(Regression)
g4.legend columns(1) position(4.67,0)
graph g5.merge g1 g2 g3 g4
show g5

'display graphs of HP filter results compared with Hamilton's
group nfp_cycles nfp_cycle nfp_hpcycle
freeze(g6) nfp_cycles.line
g6.setelem(1) legend(Hamilton Method)
g6.setelem(2) legend(HP Filter)
group nsa_cycles nsa_cycle nsa_hpcycle
freeze(g7) nsa_cycles.line
g7.setelem(1) legend(Hamilton Method)
g7.setelem(2) legend(HP Filter)
graph g8.merge g6 g7
show g8

'subroutine to shade graphs using Hamilton's dates (note these dates may differ slightly from the recession shading add-in available in EViews).
'Also does some minor formatting.
g.datelabel interval(year, 10, 0)
g.axis minor
g.axis(b) ticksout
g.options -gridl
g.options gridnone
endsub

1. Nice job! Thanks.

1. The great Professor Giles!

2. Thank you a lot.

3. Excuse me please, is this method good dealing with phase shifts?

4. I have one further question. How would I proceed, if I have monthly instead of quarterly data?

1. Shift it by 24 periods rather than 8.

5. Nice demonstration

6. I have one further question. How would I proceed, if I have annual data instead of quarterly data or monthly?

7. Nice demonstration. But what if it is annual data? how many periods will we offset it and what is the appropriate lag? thanks

1. Very much depends on the study in question. I suggest reading Section 4.3 of Hamilton's paper.

8. Hi!
Where can I get the data?

1. http://econweb.ucsd.edu/~jhamilto/software.htm#HP

9. Wonderful. How come the fitted value is not smooth like the HP trend?

10. This a great post, I have applied both filters using this code, the cycles produced by Hamilton's regression filter tend to be longer and timing of the cycles is slightly different between the two methods

11. After implementing the manual procedure, I realized that the residuals and the fitted values (forecasted) have 11 missing data points by showing NA. Is that right please? Thank you.

12. Hi!
I applied your commands for Perú, however the cycle of GPD dont occillate around zero.

Also, my data has frecuency quarterly