the information processing system of last resort

AI hallucinations and our own

One of my favourite quotes from Stafford Beer is that “ignorance is the information processing system of last resort”.

Consequently, it was rather disconcerting to have wasted an afternoon of my editor’s time and my own to discover that Beer never said this: I made it up.

What seems to have happened is that this is written across pages 24-25 of “Designing the System for Organisations”:

When I was reading it, I jotted down “ignorance is the variety attenuator of last resort”, basically turning it into a central banking joke so I’d remember it. Then when writing about a different point in a draft of the book, I misremembered my notes, and so this quote - which I still like, and still think to be an important point about the design of management systems - was born.

This happens quite a bit. There’s a monetary policy rule - one of the early attempts at “macroprudential” thinking about the joint management of interest rate policy and bank risk-taking - which goes:

“Any time Donald Trump is able to borrow money to build something, interest rates are too low”.

I attribute this to Doug Henwood, and he attributes it do me (I got it into print in the US edition of Lying for Money, so history will record it as his). But in truth, it’s quite possible that the actual author was someone else we both spoke to, who probably didn’t say it in exactly those terms. And indeed, only yesterday Tony Yates posted:

I’ve said things of this sort about crypto, but I almost always use phrase “magic numbers” - I think I might have tweeted “you can’t really stop people emailing each other magic numbers”. So once more, a phrase is born.

As Tony suggests, this looks like something similar to how LLMs produce content. Hallucinated aphorisms like this happen when things get stored away in the memory efficiently, jumbled up, and then reproduced when you’re mentally searching for something that fits into an idea-shaped hole in something you’re trying to say or write. The fact that this process a) occasionally comes up with some surprisingly good stuff, but b) is really very unlike the normal workings of creative thought, might tell us a bit about what’s going on in these AI systems.