big data is made of little data

knowing the numbers

While I was away, the “Sahm Rule” (the rule of thumb that when the unemployment rate in the USA rises to more than 0.5% above its lowest level in the last twelve months, the economy is in a state that will later be identified as the beginning of a recession) triggered.  Or did it?

The answer to that question is really to unask it – we need to focus on what we actually care about.  It’s not really important whether the Sahm Rule has “technically” triggered or not, because it’s actually not really important whether we’re “technically” in a recession or not.  The NBER Business Cycle Dating Committee acts as a convenience for econometricians, to provide a standard and agreed time series for when they think the economy was in a state of recession.  There’s always a good argument to be made for slightly different start and end dates, and there’s sometimes even debate whether the call should be made at all.

(For example, the sudden shutdown of the economy in March 2020 due to the COVID-19 pandemic – was this a recession? Depends on who’s asking.  In one sense, it definitely was, because it was a widespread slowdown in economic activity.  In another sense, it was a sui generis event which doesn’t have much to do with the normal business cycle, and a statistical model which includes it alongside other recessions is likely to be worse for doing so).

What we actually care about is whether there’s a broad reduction in activity, and the real purpose of the Sahm Rule is to describe a statistical reality that when unemployment starts to rise in that particular way, there will usually be a debate about whether a recession has begun and historically, that debate has tended to be resolved in one way rather than the other.

And that’s what the post from Claudia Sahm’s blog is all about – looking at the different component series which go to make up the unemployment rate, the flows in and out of unemployment for different reasons and the makeup of the labour force, and concluding that this looks like an exception that proves[1] the rule.

To me, this illustrates an important point.  Although the unemployment statistics are very aggregated and the Sahm Rule is a very high-level statistical regularity, it’s possible, and often advisable, to “dig down” into lower-level statistics if you need to.  Knowing the ins and outs of official data in this way is a learned skill that comes from deep familiarity with the system – to put it in James Scott’s terms, techne has its own kinds of metis.  (As it happens, Claudia Sahm is very good indeed at understanding the relationships between datasets at absolutely every level of aggregation, which is a core skill of policy economists).

But this isn’t just a matter of individual expertise; it’s also a property of the system the extent to which the data collection is set up to facilitate this kind of deep-dive analysis.  Good decision-making systems are built up recursively; big things that work are made up of small things that work, plus a lot of plumbing to make the small things work together.  If they have a logical structure, then it ought to be possible to go up and down the levels of recursion to send capacity to where it needs to be.

Bad decision making systems don’t allow for this kind of trick – they present you either with a big single aggregated blob, or a huge set of all the individual data points.  Consequently, when something goes wrong, you only find out about it when things are bad enough to show up in the big aggregate, and you have a much more difficult task to track the problem down to its source.

---

[1] as my educated readers will know, this is not a Popperian fallacy, and nobody should be saying “exceptions don’t prove rules, they falsify them”.  In fact the proverb is quite old, and uses “prove” in the semi-archaic sense of “to test”. (This sense basically only survives in the military usage of a “proving ground”).  Unusual cases test how rules work, and if the underlying mechanism is sound, an exception which can be explained as having properties well outside of the norm ought to, indeed, increase your confidence in the validity of the rule for normal cases.