Hamilton’s “Why you should never use the Hodrick-Prescott Filter”

Professor James D. Hamilton requires no introduction, having been one of the most important researchers in time series econometrics for decades.

Over the past few years, Hamilton has been working on a paper calling on applied economists to abandon the ubiquitous Hodrick-Prescott Filter and replace it with a much simpler method of extracting trend and cycle information from a time series.

This paper has become popular, and a number of our users have asked how to replicate it in EViews. One of our users, Greg Thornton, has written an EViews add-in (called Hamilton) that performs Hamilton’s method.  However, given its relative simplicity, we thought we’d use a blog post to show manual calculation of the method and replicate the results in Hamilton’s paper.

The Hodrick-Prescott Filter

The HP filter is a mainstay of modern applied macroeconomic analysis. It is used extensively to isolate trend and cycle components from a time series.  By isolating and removing the cyclical component, you are able to analyze the long-term effects of or on a variable without worrying about the impact of short term fluctuations. In macroeconomics this is especially useful since many macroeconomic variables suffer from business-cycle fluctuations.

Mathematically, the HP filter is a two-sided linear filter that computes the smoothed series $s$ of $y$ by minimizing the variance of $y$ around $s$, subject to a penalty that constrains the second difference of $s$. That is, the HP filter chooses $s$ to minimize:

$$\sum_{t=1}^T\left(y_t - s_t\right)^2 + \lambda \sum_{t=2}^{T-1}\left((s_{t+1} - s_t) - (s_t - s_{t-1})\right)^2$$

The arbitrary smoothing parameter $\lambda$ controls the smoothness of the series $s$. The larger the $\lambda$, the smoother the series. As $\lambda=\infty$, $s$ approaches a linear trend.

Hamilton’s Criticisms of the HP Filter

Hamilton outlines three main criticisms of the HP filter:

1. The HP filter produces series with spurious dynamic relations that have no basis in the underlying data-generating process.
2. Filtered values at the end of the sample are very different from those in the middle, and are also characterized by spurious dynamics.
3. A statistical formalization of the problem typically produces values for the smoothing parameter vastly at odds with common practice.

Hamilton’s Method

Hamilton proposes an alternative to the HP Filter that uses simple forecasts of the series to remove the cyclical nature.  Specifically, to produce a smoothed estimate of Y at time t, we use the fitted value from a regression of Y on 4 lagged values of Y back-shifted by two years (so 8 observation in quarterly data), and a constant. Specifically:

$$\widetilde{y}_t = \alpha_0 + \beta_1 y_{t-8} + + \beta_2 y_{t-9} + \beta_3 y_{t-10} + \beta_4 y_{t-11}$$

An Example Using EViews.

Professor Hamilton provides some examples using employment data, in the csv file employment.csv.  Specifically, the file contains quarterly non-farm payroll numbers, both seasonally adjusted and non-seasonally adjusted between 1947 and 2016Q2.

We can open that file in EViews simply by dragging it to EViews.  The file doesn’t have a date format that EViews understands, so we will manually restructure the page to quarterly frequency with a start date of 1947Q1.

We then give the two time-series names, and create growth rate series.

Having created the series we’re interested in, we’ll first perform the HP filter on the seasonally adjusted series. We open the series, click on Proc->Hodrick-Prescott Filter.  We enter names for the outputted trend and cycle series, and then click OK.

Now we’ll replicate Hamilton’s method.  We first need to regress the series against four lags of itself shifted 8 periods back.  We do this using the Quick->Estimate Equation menu, then entering the specification of NFP_LOG C NFP_LOG(-8 TO -11)

We can then view the residuals and fitted values, which correspond to the cyclical and trend components from the View menu.

If we wanted to save those components, we could use the Proc->Make Resids and Proc->Forecast menu items to produce the residuals and fitted (forecasted) values.

We’ve written a quick EViews program that automates this process for both the seasonally adjusted and non-seasonally adjusted data, and replicates Figure 5 from Hamilton’s paper.  The program produces the following graphs, and the code is below:

'open data

wfopen .\employment.csv

'structure the data and rename series

pagestruct(freq=q, start=1947)

d series01

rename series02 emp_sa

rename series03 emp_nsa

'calculate transforms of series

series nfp_log = 100*log(emp_sa)

series nsa_log = 100*log(emp_nsa)

'hp filter of employment (seasonally adjusted)

nfp_log.hpf nfp_hptrend @ nfp_hpcycle

'hp filter of employment (non seasonally adjusted)

nsa_log.hpf nsa_hptrend @ nsa_hpcycle

'estimate employment (seasonally adjusted) regressed against constant and 4 lags of itself, offset by 8 periods.

equation eq1.ls nfp_log c nfp_log(-8 to -11)

'store resids as the cycle

eq1.makeresid nfp_cycle

'store fitted vals as the trend

eq1.fit nfp_trend

'estimate employment (non-seasonally adjusted) regressed against constant and 4 lags of itself, offset by 8 periods.

equation eq2.ls nsa_log c nsa_log(-8 to -11)

'store resids as the cycle

eq2.makeresid nsa_cycle

'store fitted vals as the trend

eq2.fit nsa_trend

'calculate 8 period differences

series nfp_base = nfp_log-nfp_log(-8)

series nsa_base = nsa_log-nsa_log(-8)

'display graphs of Hamilton's method (replicate Figure 5)

freeze(g1) nfp_log.line

g1.addtext(t) Employment (seasonally adjusted)

call shade(g1)

freeze(g2) nsa_log.line

g2.addtext(t) Employment (not seasonally adjusted)

call shade(g2)

group nfps nfp_cycle nfp_base

freeze(g3) nfps.line

g3.addtext(t) Cyclical components (SA)

g3.setelem(1) legend(Random Walk)

g3.setelem(2) legend(Regression)

g3.legend columns(1) position(4.67,0)

call shade(g3)

group nsas nsa_cycle nsa_base

freeze(g4) nsas.line

g4.addtext(t) Cyclical components (NSA)

g4.setelem(1) legend(Random Walk)

g4.setelem(2) legend(Regression)

g4.legend columns(1) position(4.67,0)

call shade(g4)

graph g5.merge g1 g2 g3 g4

g5.addtext(t) Figure 5 (Hamilton)

show g5

'display graphs of HP filter results compared with Hamilton's

group nfp_cycles nfp_cycle nfp_hpcycle

freeze(g6) nfp_cycles.line

g6.addtext(t) Employment (seasonally adjusted) Cycles

g6.setelem(1) legend(Hamilton Method)

g6.setelem(2) legend(HP Filter)

call shade(g6)

group nsa_cycles nsa_cycle nsa_hpcycle

freeze(g7) nsa_cycles.line

g7.addtext(t) Employment (Non-seasonally adjusted) Cycles

g7.setelem(1) legend(Hamilton Method)

g7.setelem(2) legend(HP Filter)

call shade(g7)

graph g8.merge g6 g7

show g8

'subroutine to shade graphs using Hamilton's dates (note these dates may differ slightly from the recession shading add-in available in EViews).

'Also does some minor formatting.

subroutine shade(graph g)

     g.draw(shade, b) 1948q4 1949q4

     g.draw(shade, b) 1953q2 1954q2

     g.draw(shade, b) 1957q3 1958q2

     g.draw(shade, b) 1960q2 1961q1

     g.draw(shade, b) 1969q4 1970q4

     g.draw(shade, b) 1973q4 1975q1

     g.draw(shade, b) 1980q1 1980q3

     g.draw(shade, b) 1981q3 1982q4

     g.draw(shade, b) 1990q3 1991q1

     g.draw(shade, b) 2001q1 2001q4

     g.draw(shade, b) 2007q4 2009q2

     g.datelabel interval(year, 10, 0)

     g.axis minor

     g.axis(b) ticksout 

     g.options -gridl

     g.options gridnone

endsub