areas under a curve, and other useless things

a picture paints a thousand words, which is often annoyingly verbose

One for the real heads here – I know there are two or three of my readers who have also had the experience of trying to write something about inequality, and ending up peering at a screen with two Lorenz curves on it, muttering “I literally do not have a bloody scooby doo”.

The Lorenz curve is a way of plotting an income distribution – the proportion of the population along the bottom and the proportion of total income up the side, so that a perfectly equal society would plot as a straight line at 45 degrees. 

The area between the actual Lorenz curve and the theoretical 45 degree line is how you measure the Gini Coefficient, the widely quoted measure of inequality.  But, of course, different shapes can have the same area.  If you have a lot of poverty at the bottom, but no super-rich, is that better or worse than a basically equal society with an outsized top 1%?  I’ll draw some pictures: (Note: And then redraw them dozens of times to try and get the shape right, then still post the wrong version, which is now corrected from the one that got emailed. Thanks to @badtakesbot1 on Twitter for noticing!)

These charts have the same Gini coefficient, because I drew them that way.  Which one describes a more equal society?  Hard to say.  (It matters a lot whether the lowest parts of the chart are $1/day poverty or somewhere close to OECD median incomes; Gini is a scale-free measure in a context where scale sometimes matters).  But even parking that objection, it’s reasonable to say in this sort of context that “single-valued measures like the Gini coefficient don’t really tell you the full story – you have to look at the full Lorenz curve”.

Or at least, it seems reasonable to say this.  Then a few evenings later, when you are staring at a trellis plot of a dozen of the bloody things, you start to realise that, in the words of Spinal Tap, it’s possible to have “too much bloody perspective”.  Real world Lorenz curves haven’t been drawn so as to illustrate a point; they just have slightly different looking curves to them, maybe a visible bump here or there, and no particular clue as to which might represent something important.

This is why people invent single-valued scale-free metrics; because actually, comparing multiple values simultaneously is difficult; it’s a very expensive cognitive operation.  And that’s true even in the best case, when you can visually display the information as an area under a curve.

Something like this happens a lot in financial risk management contexts, where risk managers are always presenting big charts and warning that you have to look at the whole estimated probability distribution, and then getting disappointed when the executives fixate on averages and single-number measures.  A lot of the causes of the 2008 financial crisis can be traced back to the concept of “value at risk”, a standardised metric created at JP Morgan for Sir Dennis Weatherstone, a chief executive who wanted a single number to summarise all the risk-taking on all the trading books in the bank.

To be clear, this is bad and it sucks and executives shouldn’t do it.  But it’s also inevitable.  A common theme of this ‘stack is that the business of management is, literally, making things manageable.  It’s the activity of taking all the information that the world generates every day, and deciding what you’re going to throw away, what you’re going to filter out and summarise in order to get something that it’s possible to make decisions about. 

The trick is not to try and find the perfect summary metric, but nor is it to stare at areas under curves until you think you’ve captured the gestalt.  It’s to have the capacity to take a step back; to mostly think about the single numbers, but to be aware that you’re throwing information away and that you need to rummage through the discard bin every now and then. 

Dennis Weatherstone wanted a single number that would mostly go up when the total risk went up, down when it went down, and where big or sustained moves in either direction would be informative as to whether you should start an exercise in finding out why.  He didn’t (or at least, not at the time) ask anyone to make an industrialised decision process and get it written into the Basel Accords for banking regulation.