This connects to something I've been thinking about since your last post. In my math olympiad days, we talked about the concept of "mathematical maturity" (or "maturing")
First level is simply remembering a proof. You can reproduce it, even talk about it convincingly, so its the interactional expertise. Then there's the level where you can re-enact it: you're not relying on memory but on understanding the mechanics well enough to reconstruct it. You're not faking it, but the subjective experience can still feel like you're an automaton.
Then there's this more elusive level, where universal truths about the subject start becoming clear. This is what "maturing" then means. You can extrapolate the theory, you "really get it." I think there's yet a level beyond that, where you see the structure of the representation itself and can go beyond it. That's where creativity lives. Feels like contributory can be at either of these two levels, depending on the contribution.
What strikes me is how long it can take to "mature," how slow the process is. There are concepts where I didn't really feel subjectively that I "got it" until decades later. And so many more that I never "got."
It also makes me think of how hard it is to teach that the last mile of autonomous driving, while any 18 year old can pick it up in just a few months. The 18 year old, of course, has spent 18 years building a model of the physical world. Is there something to the slowness that scale cannot replicate?
I can absolutely believe this - from a subjective point of view I have taught myself stochastic differential equations three times, linear algebra twice, and whatever the hell is going on with information theory once, for various projects. Each one of them has, as soon as I no longer needed it, slipped out of my brain almost immediately - for the last adventure in linear algebra I ended up using the phrase "passed through me like cold beer on a hot day". There were brief moments when I thought I might be close to one of the early stages of what you described as maturity but I definitely never got over the lip of that cliff.
Maybe this is the difference between an LLM and a robot, which has to act in the same world that humans do, and can therefore gather information about it, the way the Waymo taxis do.
Maybe AI itself cannot solve the last mile problem (at least by itself) because it doesn't directly interact with anything except databases. It is a complete pretender, a theory monger.
Maybe a robot or an LLM can't understand life forms like us. Likewise, humans cannot know if LLMs are either intelligent or conscious. All we know is that computers can beat humans at chess, an extremely large set of solvable problems that don't require "knowledge" of anything except the possible moves and their consequences.
The DescartesBot can say it is thinking, but does it even know what it is? Or what "is" is? Even if it "says" so?
To come at this from another angle, since I know a little bit about aviation (a little), did you know that the 747 was capable of landing itself in zero visibility half a century ago, if it was flying to an airport that had the navigational aids that could tell it where it was in relation to the runway? That gave it all it needed to know about where it was, and the rest was just following the script about what to do if too low, too fast, etc.
Whereas the robo-cars like the Waymo get no such outside help. The environment doesn't broadcast signals to it to convey which way to go, and humans don't broadcast signals saying "I am a human, don't hit me," so it has to either respond to its LIDAR detectors, or what it has learned about the city from driving through it repeatedly. It is like a blind person walking around in the city. And there's a lot more things it can hit on city streets than a plane can in the air. So it's a much harder problem, which is why it hasn't been solved yet.
And to think that the Apollo moon rocket was designed with slide rules! And with them, they solved the "getting to the moon and back" problem in 1969. But we still can't make traffic jams go away, or balance our budgets, or keep complete pretenders out of public office.
BTW I am also a complete pretender about all this because I am not a mathematician or scientist, but I still think I can ask some possibly relevant questions, even in my role as an observer. But you guys will have to answer them, I can't.
I am not a bot, but I did stay in a Holiday Inn last night:
Stoopid question: what if you fed a lot of videos of traffic accidents to the bot, but stopped them just before the accident and asked the bot to figure out what was about to happen next? Could it learn to predict what would happen, or what to do? Is that harder than figuring out chess, or easier?
I should have known you were also a member of the Harry Collins Superfan Club.
There's so much discourse in academia about AI eliminating the need for graduate students, but through the lens of Collins' interactional expertise, you see the PIs of big science--whose work devolves into management, grant-writing, and self-promotion--are even more automatable.
Here is a trivial personal example that I think may have wider significance. A couple of Fridays ago I was at Lord’s [cricket ground] for the 2nd day of the 5 day test match. Because there had been rain interruptions the previous day I was unsure how much of that time could be made up. I googled “what are the hours of play at Lord’s today?” The answer offered by AI was “there is no match scheduled at Lord’s today”.
I notice that Google no longer gives AI answers to "what time is high tide at Exmouth", and hope that the three or four emails I sent them pointing out that this could potentially get someone killed might have contributed
Litigators are the masters of interactional expertise. A good trial lawyer can make a genuine expert seem like an ignorant fool--which is why the genuine experts are heavily prepped by their own lawyers before they take the stand.
Litigators are very good, yes, but it is really quite an incredible experience to be in the presence of a really good equities salesperson as they switch instantaneously from financial regulation to handbag fashions to pharmaceutical pipelines and back again
Trying to put this a little more precisely: I can imagine a mechanism whereby LLMs can absorb all the knowledge that can be articulated and made legible, but there's a whole load of tacit/intuitive knowledge that they don't see. Is that it?
I think I'm more interested in the idea that there are kinds of knowledge which LLMs could absorb, but which currently existing ones (and if I want to push it, an economically meaningful number of future generations) aren't actually putting to use when presenting their output
Gotcha. So something like your Swiss cheese account of LLMs: the interesting rough edges are where much of the new knowledge is to be found, and that's what the big averaging machine is smoothing down?
It sounds at first as if you're talking about a hard distinction between things LLMs can and can't do. But actually I think you're talking about things thinking machines can and can't do in the context of a particular data environment and mode of interaction with the world.
The regime where the machine gobbles up data first, and then has to achieve something creative, is fundamentally different from the regime where the machine forms hypotheses and then calls tools to disprove them, and explores the world that way.
Currently, the most advanced data environment for LLMs, and the one they are missing at fluent interacting with, is provided by the computer terminal. That is why software developers who've spent time working with Claude Code tend to have a very different picture of the fundamental limitations of LLMs than those who have interacted with chat apps in less rich data environments.
The question for any type of problem is: can you construct an appropriate data environment?
That's quite a fun anecdote and I was curious about it, so I asked ChatGPT to dig up more information about it. Apparently the results of this imitation game were a bit different? Though still amusing:
> [...] I took part in an 'imitation game' in which a GW physicist asked technical questions - he asked seven in all - of me and another GW physicist. The dialog - seven questions and seven pairs of answers with identities disguised - was then sent to nine other GW physicists who were asked to identify the participants, knowing that one of them was me. Seven said they couldn't work out who was who and two said that I was the real physicist.
The human mind has access to all knowledge. When you learn something new the comment “that’s right’ often appears. What referential index do you employ? Good one for your holiday
Another thumbs up for reading Harry Collins. But, more specifically to your account of LLMs achieving great results in Mathematics, is David Bessis' article here on Substack concerning the different kinds of mathematical "intelligence", and in particular the difference between conceiving of Theorems and proving them. He laid out earlier thoughts in his well-received book, "Mathematica".
Interaction expertise is what makes critics and software architects good at their jobs, if they are good. That kind of expertise only really works though if the practitioner is capable of pointing to underlying logical or semantic connections across the discourse. The deeper the understanding, even if it's not Contributory, the further to the right on the scale of "plausible -> possible -> probable" they can get with their claims.
This slots in with Cosma Shalizi et al.'s framing of LLMs as 'cultural technologies'- like books and indices and catalogs, fundamentally a mechanism for organizing extant information in an interesting way than an autonomous being. Just thinking for a moment about how an LLM is created and works at the most fundamental level makes it clear that bullshitting on a topic- essentially what you're talking about here- is going to be a strength and everything else is not. Of course, bullshitting is in fact a skill, and if you strap a bullshitter to an architecture to sift through their bullshit, you'll occasionally get something- a million monkeys on typewriters feeding copy to boosters willing to look charitable on the results can't help but hit sometimes- but there are things they are unlikely to do. Which maps to what we actually see- bots that can perform some ostensibly high level task (that invariably turns out to be in their training data, while a conceptually adjacent task proves intractable) but collapse into babbling madness when left to run a vending machine.
This connects to something I've been thinking about since your last post. In my math olympiad days, we talked about the concept of "mathematical maturity" (or "maturing")
First level is simply remembering a proof. You can reproduce it, even talk about it convincingly, so its the interactional expertise. Then there's the level where you can re-enact it: you're not relying on memory but on understanding the mechanics well enough to reconstruct it. You're not faking it, but the subjective experience can still feel like you're an automaton.
Then there's this more elusive level, where universal truths about the subject start becoming clear. This is what "maturing" then means. You can extrapolate the theory, you "really get it." I think there's yet a level beyond that, where you see the structure of the representation itself and can go beyond it. That's where creativity lives. Feels like contributory can be at either of these two levels, depending on the contribution.
What strikes me is how long it can take to "mature," how slow the process is. There are concepts where I didn't really feel subjectively that I "got it" until decades later. And so many more that I never "got."
There's a a Quanta profile of Fields Medalist June Huh https://www.quantamagazine.org/june-huh-high-school-dropout-wins-the-fields-medal-20220705/ that also touches on this. His collaborator describes him as "very slow," so slow you'd think he couldn't pass a qualifying exam, but he was learning simple things in a deeper way that later proved essential.
It also makes me think of how hard it is to teach that the last mile of autonomous driving, while any 18 year old can pick it up in just a few months. The 18 year old, of course, has spent 18 years building a model of the physical world. Is there something to the slowness that scale cannot replicate?
I can absolutely believe this - from a subjective point of view I have taught myself stochastic differential equations three times, linear algebra twice, and whatever the hell is going on with information theory once, for various projects. Each one of them has, as soon as I no longer needed it, slipped out of my brain almost immediately - for the last adventure in linear algebra I ended up using the phrase "passed through me like cold beer on a hot day". There were brief moments when I thought I might be close to one of the early stages of what you described as maturity but I definitely never got over the lip of that cliff.
Maybe this is the difference between an LLM and a robot, which has to act in the same world that humans do, and can therefore gather information about it, the way the Waymo taxis do.
Maybe AI itself cannot solve the last mile problem (at least by itself) because it doesn't directly interact with anything except databases. It is a complete pretender, a theory monger.
Maybe a robot or an LLM can't understand life forms like us. Likewise, humans cannot know if LLMs are either intelligent or conscious. All we know is that computers can beat humans at chess, an extremely large set of solvable problems that don't require "knowledge" of anything except the possible moves and their consequences.
The DescartesBot can say it is thinking, but does it even know what it is? Or what "is" is? Even if it "says" so?
To come at this from another angle, since I know a little bit about aviation (a little), did you know that the 747 was capable of landing itself in zero visibility half a century ago, if it was flying to an airport that had the navigational aids that could tell it where it was in relation to the runway? That gave it all it needed to know about where it was, and the rest was just following the script about what to do if too low, too fast, etc.
Whereas the robo-cars like the Waymo get no such outside help. The environment doesn't broadcast signals to it to convey which way to go, and humans don't broadcast signals saying "I am a human, don't hit me," so it has to either respond to its LIDAR detectors, or what it has learned about the city from driving through it repeatedly. It is like a blind person walking around in the city. And there's a lot more things it can hit on city streets than a plane can in the air. So it's a much harder problem, which is why it hasn't been solved yet.
And to think that the Apollo moon rocket was designed with slide rules! And with them, they solved the "getting to the moon and back" problem in 1969. But we still can't make traffic jams go away, or balance our budgets, or keep complete pretenders out of public office.
BTW I am also a complete pretender about all this because I am not a mathematician or scientist, but I still think I can ask some possibly relevant questions, even in my role as an observer. But you guys will have to answer them, I can't.
I am not a bot, but I did stay in a Holiday Inn last night:
https://www.youtube.com/watch?v=eHCTaUFXpP8
Stoopid question: what if you fed a lot of videos of traffic accidents to the bot, but stopped them just before the accident and asked the bot to figure out what was about to happen next? Could it learn to predict what would happen, or what to do? Is that harder than figuring out chess, or easier?
I should have known you were also a member of the Harry Collins Superfan Club.
There's so much discourse in academia about AI eliminating the need for graduate students, but through the lens of Collins' interactional expertise, you see the PIs of big science--whose work devolves into management, grant-writing, and self-promotion--are even more automatable.
Here is a trivial personal example that I think may have wider significance. A couple of Fridays ago I was at Lord’s [cricket ground] for the 2nd day of the 5 day test match. Because there had been rain interruptions the previous day I was unsure how much of that time could be made up. I googled “what are the hours of play at Lord’s today?” The answer offered by AI was “there is no match scheduled at Lord’s today”.
I notice that Google no longer gives AI answers to "what time is high tide at Exmouth", and hope that the three or four emails I sent them pointing out that this could potentially get someone killed might have contributed
Litigators are the masters of interactional expertise. A good trial lawyer can make a genuine expert seem like an ignorant fool--which is why the genuine experts are heavily prepped by their own lawyers before they take the stand.
Litigators are very good, yes, but it is really quite an incredible experience to be in the presence of a really good equities salesperson as they switch instantaneously from financial regulation to handbag fashions to pharmaceutical pipelines and back again
Is this a techne v metis sort of distinction?
It is definitely that sort of thing but I don't think it's the same - I think it's different kinds of techne
Trying to put this a little more precisely: I can imagine a mechanism whereby LLMs can absorb all the knowledge that can be articulated and made legible, but there's a whole load of tacit/intuitive knowledge that they don't see. Is that it?
I think I'm more interested in the idea that there are kinds of knowledge which LLMs could absorb, but which currently existing ones (and if I want to push it, an economically meaningful number of future generations) aren't actually putting to use when presenting their output
Gotcha. So something like your Swiss cheese account of LLMs: the interesting rough edges are where much of the new knowledge is to be found, and that's what the big averaging machine is smoothing down?
This is what I'm hoping to write about once Richard O’Rourke (for it is he) gets his paper published
It sounds at first as if you're talking about a hard distinction between things LLMs can and can't do. But actually I think you're talking about things thinking machines can and can't do in the context of a particular data environment and mode of interaction with the world.
The regime where the machine gobbles up data first, and then has to achieve something creative, is fundamentally different from the regime where the machine forms hypotheses and then calls tools to disprove them, and explores the world that way.
Currently, the most advanced data environment for LLMs, and the one they are missing at fluent interacting with, is provided by the computer terminal. That is why software developers who've spent time working with Claude Code tend to have a very different picture of the fundamental limitations of LLMs than those who have interacted with chat apps in less rich data environments.
The question for any type of problem is: can you construct an appropriate data environment?
That's quite a fun anecdote and I was curious about it, so I asked ChatGPT to dig up more information about it. Apparently the results of this imitation game were a bit different? Though still amusing:
> [...] I took part in an 'imitation game' in which a GW physicist asked technical questions - he asked seven in all - of me and another GW physicist. The dialog - seven questions and seven pairs of answers with identities disguised - was then sent to nine other GW physicists who were asked to identify the participants, knowing that one of them was me. Seven said they couldn't work out who was who and two said that I was the real physicist.
https://www.researchgate.net/publication/305638337_An_Imitation_Game_concerning_gravitational_wave_physics
Edit: but reading on, it seems he tried doing the experiment again!
Used to be polymaths who did this. Now everything is so deep it’s not possible for an individual to know it all. It’s just a trick of memory.
The human mind has access to all knowledge. When you learn something new the comment “that’s right’ often appears. What referential index do you employ? Good one for your holiday
Another thumbs up for reading Harry Collins. But, more specifically to your account of LLMs achieving great results in Mathematics, is David Bessis' article here on Substack concerning the different kinds of mathematical "intelligence", and in particular the difference between conceiving of Theorems and proving them. He laid out earlier thoughts in his well-received book, "Mathematica".
Article https://davidbessis.substack.com/p/the-fall-of-the-theorem-economy?utm_campaign=posts-open-in-app&triedRedirect=true
Book https://yalebooks.yale.edu/book/9780300270884/mathematica/
Interaction expertise is what makes critics and software architects good at their jobs, if they are good. That kind of expertise only really works though if the practitioner is capable of pointing to underlying logical or semantic connections across the discourse. The deeper the understanding, even if it's not Contributory, the further to the right on the scale of "plausible -> possible -> probable" they can get with their claims.
This slots in with Cosma Shalizi et al.'s framing of LLMs as 'cultural technologies'- like books and indices and catalogs, fundamentally a mechanism for organizing extant information in an interesting way than an autonomous being. Just thinking for a moment about how an LLM is created and works at the most fundamental level makes it clear that bullshitting on a topic- essentially what you're talking about here- is going to be a strength and everything else is not. Of course, bullshitting is in fact a skill, and if you strap a bullshitter to an architecture to sift through their bullshit, you'll occasionally get something- a million monkeys on typewriters feeding copy to boosters willing to look charitable on the results can't help but hit sometimes- but there are things they are unlikely to do. Which maps to what we actually see- bots that can perform some ostensibly high level task (that invariably turns out to be in their training data, while a conceptually adjacent task proves intractable) but collapse into babbling madness when left to run a vending machine.