identifying the edges

flies in bottles and other decision makers

About twenty years ago, I wrote a short blog post describing my honeymoon in Brittany, in which I gave the travel advice to “take the ferry to Calais, point the front of the car roughly in the direction of Wilmington, North Carolina and keep driving until the cafes no longer advertise "Wi-Fi Gratuit”.  As someone immediately pointed out in the comments, how do you know you’ve passed the last one?

Considered in its most abstract and general terms, identifying whether you’re in a space or not from inside that space is a very interesting and difficult question; if you express it in non-geometrical terms, it’s as old as Socrates’ observation that the wisest person is the one who understands the limits of their knowledge.  In a particular restricted formal context, it’s the Halting Problem (the fact that there’s no universally applicable algorithm that can be used to identify whether any given computer program will terminate).  There are also cryptographic proofs of somewhat greater practical relevance to show that there’s no way for a piece of software to know with certainty what kind of hardware it’s running on, and therefore no infallible way of keeping a secret that relies only on software. 

As the great philosopher “Dirty” Harry Callahan said, a man’s got to know his limitations.  But knowing your limitations is difficult; that’s the basis of the Dunning-Kruger Effect.

This could be a very serious problem for the applications of artificial intelligence.  We’re getting to understand that AI does well for a lot of problems, but has problems with “edge cases”.  Which means that it has two big issues; first, that in the real world “edge cases” are a lot more common than you might hope – almost everyone is a special unique case in some way or other, so you just have to hope that the way in which they’re unique isn’t relevant to this particular interaction.  And secondly, identifying what’s an edge case and what isn’t, is itself something of an edge case.

Human decision-makers identify edge cases by having a lot more information coming in all the time than they are using.  For example, it’s usually not directly relevant to a decision making process whether or not the person describing the problem to you is crying or not, but combined with other situational information, it’s often an indicator that this case is going to be more complicated or difficult than the rest.  An experienced decision-maker isn’t just experienced in the specific problems that they’re paid for – they are also a person who has seen a lot of the world and has a lot of memories and experiences that aren’t usually relevant but which might suddenly be important in the future.  A lot of this time, this isn’t even a matter of conscious information-processing; the most important edge-case identifier is a sudden subjective sensation in the pit of the stomach that you are out of your depth and need to escalate the problem to get more resources.

There’s no conceptual reason why an AI system – not an abstract possible system of the future, an actual one based on recurrent neural network architectures, like the state of the art today – couldn’t do well at Socrates’ problem.  Even ChatGPT is capable of admitting it doesn’t know the answer, although not always as much as it ought to.  But it would be difficult and not cheap; understanding the limits of your space of competence relies on having a lot of irrelevant context so that you can recognise the clues.  And at present it seems that the direction AI applications are going in is the opposite one – the use of specialised training data to make models which are better at giving correct answers in specific problem domains.  It seems to me that these models, when they fail, will fail hard and will tend to double down on mistakes like a politician. 

Consequently, the machines will need to be our colleagues, not our bosses.  One of the songs my portfolio manager taught me is that the very worst situation to be in is one of extremely high confidence in a decision that might be wrong.