Generally good but worth pointing out that the "hallucination" issue in LLMs isn't closely related to the general problem of under/overfitting. The system in general is trained to find statistically likely follow-up utterances rather than produce statements which are true; unlike reinforcement systems like AlphaGo which are trained directly on the ground truth of victory or defeat. The things it says sometimes coincide with truth or truthy statements due to their preponderance in the training set but there's very likely to be a hard ceiling for anything we might call accuracy absent some components, which we currently have no idea how to build or integrate, corresponding to the _rest_ of a mind.
My opinion on AGI and surpassing human-level intelligence is not important. However, the AI boosters' claims of massive sudden increases in GDP are extraordinary and need extra-ordinarily robust justification, not mere hand-waving. Sure, assume AI cracks fusion power, say. It'll still be two or three decades before we get a few pilot plants built, and several more decades before there's meaningful economic impact.
Yes, jobs dealing with information, particularly where mistakes don't result in explosions or collapses, are at risk. But the decay process will be slow. We may be in for a repeat of the Long Depression of 1873 to 1899-ish, back to back with the Great Depression of the 1930s.
Then I'll be impressed. Stochastic parrots are not impressive.
thank you for Moravec, the Paradox is what I am constantly citing but still haven't got it into the discourse.
I first encountered it while doing AI for the military in the 80s, when AI was expert systems. That experience inoculated me against AI credulity.
AlphaGo works because the problem space is perfectly defined and limited.
From the Nature paper written by the AlphaGo Zero (AGZ) researchers,
"Here we introduce an algorithm based solely on reinforcement learning,
without human data, guidance or domain knowledge beyond game rules."
The problem with general intelligence AI is that there are no game rules.
As commenter Aardvark noted on John Scalzi's review of a new phone,
It’s not like LLMs work OK sometimes and “hallucinate” sometimes. They are never not “hallucinating.” It’s just that when they’re on solid ground with their training data, and haven’t been given a perverse prompt, the hallucinations look like what we expect from reality.
Julia Carrie Wong,
it’s just a word generating machine generating words the only meaning is that which you read into it and imo that should be none
when you examine a text written by a human you can find layers and layers of meaning and intentionality, the complexity of the human consciousness, an opportunity for one mind to commune with another outside the bounds of time and space. when you examine AI text you drown in a teaspoon of nothing.
Expert systems history illustrates what is likely to happen. I used to have a book describing a pilot study in one of the better known hospitals in the NHS in England, with an expert system diagnostic tool. It massively outperformed the hospital specialists, but was abandoned after fierce but quiet back-room opposition from those same doctors.
It is as likely today, as then, that the unions--ahem: professional associations--will be able to resist very effectively.
Oh OK, I was just wondering in case there could have been any signal degradation along the path from any actual events to the above comment.
Because I had never heard of these robot consultants (diagnosticians?) and the idea that their competence had got exaggerated along the way, and the failure of a system wrongly blamed on militant doctors, is quite plausible, given for example the ongoing attempt to replace doctors with decidely inexpert 'physician associates' (previously 'assistants'): https://www.ft.com/content/5a533507-f11d-42b2-b67e-e10c0d7c9fb8 . In this case too, doctors' objections are portrayed as 'toxic' protectionism; see top and tail of this Beeb version: https://www.bbc.co.uk/news/articles/c2dly5ldrxjo .
I'm increasingly uncomfortable with the concept of hallucination though. It implies that the model is doing something different when it creates sequences of tokens that are "right" compared to when they are "wrong". But in reality, the process is identical. And it's us dichitomising them into right and wrong.
Even if we could fix this, who decides what is right and what is wrong?
Great post. The complexity point is very important. It's the 'word processors will save paper' point. That didn't happen - whereas plumbing really did save trips to the river and refrigeration really did reduce food spoilage. So much of the digital plumbing being put into companies is creating more complexity, not less. Typing every thought / task / outcome into a data form does not help the goal to be more effectively advanced. Post-it notes have contributed far more to problem solving than all the painful digital project management / people surveillance tools than now proliferate.
Also, re: hallucinations. I'm not sure it's just a fitting problem? If that were the case, then hallucinations would be sensitive to levels of training data. But I'm not sure there's any correlation there? Isn't the problem that the beast doesn't know what it doesn't know? Of course, many humans share this problem but we can reduce this noise / bias with collaborative thinking tools (scientific method, dialectic method, understanding of fallacies / cognitive biases etc). So in a really well structured environment, the hallucinations can be outed through back and forth discussion. This does not exist in an AI algorithm. It may be that training them on one another can produce crowd-sourced precision (like humans). But we do surely reach a point of reductio ad absurdam here. At what point is the energy cost of creating groups of ultra-intelligent, arguing AIs more efficient than getting a group of well facilitated humans to work through a problem? (with assistance on many tiresome tasks from the machines).
1. Here is the output of $FRONTIER_MODEL when I ask question X/try to do task X. These are the dimensions on which it is wrong. The median human response freely accessible to people is better than this. It is unlikely to get better because there is no source of training data publicly available, and no business model incentivizes creating said data.
The most glaring & fundamental version of the 'agile sprint down a blind alley' problem in this context seems to me to be the focus on 'how soon can we start replacing the less intelligent white-collar humans?' rather than 'are we on the road to producing something that can identify, isolate & explain novel solutions to our difficult problems?'
Turing has a lot to answer for here, but that was a very long time ago & I think it has more to do with the resources for such research having mostly been in the hands of a bunch of spivs embedded in a fast-buck-worshipping culture. Maybe the Chinese will come up with something more interesting.
The contemporary transformer architecture could not learn any new thing by itself, because the training phase and inference phases are strictly separated. That, and the limited context length make me think that AGI/ASI is not possible now.
Deepseek showed us that the transformer architecture works better than we thought, resource-wise. That will enable everybody to slap transformers to everything. You could finally talk with your toaster, explain which kind of toast you desire today and then spend 20+ minutes to persuade "it" to finally energize the electrical spiral to really make the toast for you.
Some jobs will be lost, some new jobs gained. As you mention this AI is great in "analogue" cases where a small error in AI's output is irrelevant or translates into only a small tolerable error in real-world action. In cases when even a small error leads to catastrophic problems, transformers are not so much useful. I am not decided if programming belongs to the first or the second category.
Yeah, technology replacing a human, as per Altman, is a bar so low we passed it centuries, if not millennia, ago. If only we could agree on what intelligence actually is. Displaying elaborate search results from prompts, while impressive, does not quite cut it.
On both the booster and skeptical sides, I think being clearer about what you mean about AGI is necessary. It's clear that for many definitions of AGI from 15 years ago, it has already been reached, and that chatgpt is far more intelligent at most tasks than most people. On the other hand there are lots of things people do that AIs don't do, and so claims about the redundancy of humans are far overblown. But this is exactly what we should expect for any technology that replaces some human work -- combines and steam engines don't work exactly like peasants with scythes or horses. So it's necessary to be clear about what the questions are, and what kinds of answers one is looking for, in a way that the discussion of AGI rarely is.
I don't think I can agree with that - the "G" has to stand for "general" surely, so if there are loads of things that human beings can do which the AI can't, it's not AGI?
This is what I mean about needing actual definitions. One possible definition is that it should be indistinguishable from a person, like the replicants in Blade Runner. Obviously Chatgpt isn't anything like that. A different definition is something like the Turing test. Modern LLMs definitely pass that. Both of those are plausibly "general" so you need to know what you're saying before having a discussion.
You are just rephrasing Tesler's Theorem ("AI is whatever hasn't been done yet".) This is a favourite of AI boosters and obviously has some foundation. The trouble is that these boosters have pushed it so far they've landed on "AGI is whatever has already been done". I have yet to see an adequate rebuttal to Chollet's objections, for example.
>I think that AI, like every other information technology, will end up creating complexity as well as processing it, that the robots will get in each other’s way just like we do
This is a good point. We already have general intelligence. The issue is coordinating it.
I sometimes think LLM/AI will be useful in the way that PowerPoint is useful, but not like electric power is useful.
1) The key thing about AGI is that while people on the inside of certain firms will often say "we've seen things you wouldn't believe" right now we're still on "keep scaling, and AGI will emerge" and... that's not philosophically impossible, but there are enough conceptual questions about that approach that I'd like to see some things (as opposed to be briefed opaquely) before I buy into any of the more exciting timelines for AGI.
2) Likewise, absent AGI, it's hard to see how the economic benefits kick in quickly - there's lots of potential, but just like previous waves of ICT, getting the impact on the productivity figures requires rearranging how things are done - and that can be slow. I want to note as well that commentators usually imagine it's slow "because people are Luddites" but most of the time it's actually because "economic incentives create inertia or prioritise the short term over the longer term."
3) Along similar lines I'm less optimistic about how long it is going to take to iron out the hallucinations problem. First: there's some wishful thinking about hallucinations being single incidents, when it's becoming clear that to get the kind of "thinking power" we want the road is "Chain of Thought" but chain = multiple incidents, multiplication of incidence. You can see this outside of LLMs in ML in a number of the self-driving car papers. Which of course points to a solution - but effectively injecting Deming into the LLM process is another level up in complexity over bringing into ML image recognition/LIDAR processing etc. Obviously this comment is asking for someone to publish a breakthrough next week and make it look stupid, but as of today there's enough there to suggest this isn't easy.
Generally good but worth pointing out that the "hallucination" issue in LLMs isn't closely related to the general problem of under/overfitting. The system in general is trained to find statistically likely follow-up utterances rather than produce statements which are true; unlike reinforcement systems like AlphaGo which are trained directly on the ground truth of victory or defeat. The things it says sometimes coincide with truth or truthy statements due to their preponderance in the training set but there's very likely to be a hard ceiling for anything we might call accuracy absent some components, which we currently have no idea how to build or integrate, corresponding to the _rest_ of a mind.
Right, it's more akin to AlphaGo generating an alien way of playing the game except without the constrains of any rules.
My opinion on AGI and surpassing human-level intelligence is not important. However, the AI boosters' claims of massive sudden increases in GDP are extraordinary and need extra-ordinarily robust justification, not mere hand-waving. Sure, assume AI cracks fusion power, say. It'll still be two or three decades before we get a few pilot plants built, and several more decades before there's meaningful economic impact.
Yes, jobs dealing with information, particularly where mistakes don't result in explosions or collapses, are at risk. But the decay process will be slow. We may be in for a repeat of the Long Depression of 1873 to 1899-ish, back to back with the Great Depression of the 1930s.
(https://en.wikipedia.org/wiki/Long_Depression)
OK, can 't resist. Why does no one talk about Moravec's Paradox any more? (https://en.wikipedia.org/wiki/Moravec%27s_paradox) Doing things that humans find "skilled" does not impress me. Make me a sandwich. (https://xkcd.com/149/)
or drive me a car ;-)
fold my laundry, wash my dishes.
Then I'll be impressed. Stochastic parrots are not impressive.
thank you for Moravec, the Paradox is what I am constantly citing but still haven't got it into the discourse.
I first encountered it while doing AI for the military in the 80s, when AI was expert systems. That experience inoculated me against AI credulity.
AlphaGo works because the problem space is perfectly defined and limited.
From the Nature paper written by the AlphaGo Zero (AGZ) researchers,
"Here we introduce an algorithm based solely on reinforcement learning,
without human data, guidance or domain knowledge beyond game rules."
The problem with general intelligence AI is that there are no game rules.
As commenter Aardvark noted on John Scalzi's review of a new phone,
It’s not like LLMs work OK sometimes and “hallucinate” sometimes. They are never not “hallucinating.” It’s just that when they’re on solid ground with their training data, and haven’t been given a perverse prompt, the hallucinations look like what we expect from reality.
Julia Carrie Wong,
it’s just a word generating machine generating words the only meaning is that which you read into it and imo that should be none
when you examine a text written by a human you can find layers and layers of meaning and intentionality, the complexity of the human consciousness, an opportunity for one mind to commune with another outside the bounds of time and space. when you examine AI text you drown in a teaspoon of nothing.
Expert systems history illustrates what is likely to happen. I used to have a book describing a pilot study in one of the better known hospitals in the NHS in England, with an expert system diagnostic tool. It massively outperformed the hospital specialists, but was abandoned after fierce but quiet back-room opposition from those same doctors.
It is as likely today, as then, that the unions--ahem: professional associations--will be able to resist very effectively.
Can you remember any other details about this book that you used to have, such as its title?
No, sorry. I'm fairly sure it was a hardback, and a description of various facets of software engineering, but that's all.
Oh OK, I was just wondering in case there could have been any signal degradation along the path from any actual events to the above comment.
Because I had never heard of these robot consultants (diagnosticians?) and the idea that their competence had got exaggerated along the way, and the failure of a system wrongly blamed on militant doctors, is quite plausible, given for example the ongoing attempt to replace doctors with decidely inexpert 'physician associates' (previously 'assistants'): https://www.ft.com/content/5a533507-f11d-42b2-b67e-e10c0d7c9fb8 . In this case too, doctors' objections are portrayed as 'toxic' protectionism; see top and tail of this Beeb version: https://www.bbc.co.uk/news/articles/c2dly5ldrxjo .
Great article.
I'm increasingly uncomfortable with the concept of hallucination though. It implies that the model is doing something different when it creates sequences of tokens that are "right" compared to when they are "wrong". But in reality, the process is identical. And it's us dichitomising them into right and wrong.
Even if we could fix this, who decides what is right and what is wrong?
Great post. The complexity point is very important. It's the 'word processors will save paper' point. That didn't happen - whereas plumbing really did save trips to the river and refrigeration really did reduce food spoilage. So much of the digital plumbing being put into companies is creating more complexity, not less. Typing every thought / task / outcome into a data form does not help the goal to be more effectively advanced. Post-it notes have contributed far more to problem solving than all the painful digital project management / people surveillance tools than now proliferate.
Also, re: hallucinations. I'm not sure it's just a fitting problem? If that were the case, then hallucinations would be sensitive to levels of training data. But I'm not sure there's any correlation there? Isn't the problem that the beast doesn't know what it doesn't know? Of course, many humans share this problem but we can reduce this noise / bias with collaborative thinking tools (scientific method, dialectic method, understanding of fallacies / cognitive biases etc). So in a really well structured environment, the hallucinations can be outed through back and forth discussion. This does not exist in an AI algorithm. It may be that training them on one another can produce crowd-sourced precision (like humans). But we do surely reach a point of reductio ad absurdam here. At what point is the energy cost of creating groups of ultra-intelligent, arguing AIs more efficient than getting a group of well facilitated humans to work through a problem? (with assistance on many tiresome tasks from the machines).
AI-skeptical commentary I take seriously:
1. Here is the output of $FRONTIER_MODEL when I ask question X/try to do task X. These are the dimensions on which it is wrong. The median human response freely accessible to people is better than this. It is unlikely to get better because there is no source of training data publicly available, and no business model incentivizes creating said data.
2. There is no 2.
The most glaring & fundamental version of the 'agile sprint down a blind alley' problem in this context seems to me to be the focus on 'how soon can we start replacing the less intelligent white-collar humans?' rather than 'are we on the road to producing something that can identify, isolate & explain novel solutions to our difficult problems?'
Turing has a lot to answer for here, but that was a very long time ago & I think it has more to do with the resources for such research having mostly been in the hands of a bunch of spivs embedded in a fast-buck-worshipping culture. Maybe the Chinese will come up with something more interesting.
The contemporary transformer architecture could not learn any new thing by itself, because the training phase and inference phases are strictly separated. That, and the limited context length make me think that AGI/ASI is not possible now.
Deepseek showed us that the transformer architecture works better than we thought, resource-wise. That will enable everybody to slap transformers to everything. You could finally talk with your toaster, explain which kind of toast you desire today and then spend 20+ minutes to persuade "it" to finally energize the electrical spiral to really make the toast for you.
Some jobs will be lost, some new jobs gained. As you mention this AI is great in "analogue" cases where a small error in AI's output is irrelevant or translates into only a small tolerable error in real-world action. In cases when even a small error leads to catastrophic problems, transformers are not so much useful. I am not decided if programming belongs to the first or the second category.
The nice thing about writing for an audience on both sides of the Atlantic is that you have a "license" to use whichever spellings you please.
A goal line so ill-defined that the race is over when one or more runners declare victory deserves skepticism. That's AGI in a nutshell.
Yeah, technology replacing a human, as per Altman, is a bar so low we passed it centuries, if not millennia, ago. If only we could agree on what intelligence actually is. Displaying elaborate search results from prompts, while impressive, does not quite cut it.
On both the booster and skeptical sides, I think being clearer about what you mean about AGI is necessary. It's clear that for many definitions of AGI from 15 years ago, it has already been reached, and that chatgpt is far more intelligent at most tasks than most people. On the other hand there are lots of things people do that AIs don't do, and so claims about the redundancy of humans are far overblown. But this is exactly what we should expect for any technology that replaces some human work -- combines and steam engines don't work exactly like peasants with scythes or horses. So it's necessary to be clear about what the questions are, and what kinds of answers one is looking for, in a way that the discussion of AGI rarely is.
I don't think I can agree with that - the "G" has to stand for "general" surely, so if there are loads of things that human beings can do which the AI can't, it's not AGI?
This is what I mean about needing actual definitions. One possible definition is that it should be indistinguishable from a person, like the replicants in Blade Runner. Obviously Chatgpt isn't anything like that. A different definition is something like the Turing test. Modern LLMs definitely pass that. Both of those are plausibly "general" so you need to know what you're saying before having a discussion.
You are just rephrasing Tesler's Theorem ("AI is whatever hasn't been done yet".) This is a favourite of AI boosters and obviously has some foundation. The trouble is that these boosters have pushed it so far they've landed on "AGI is whatever has already been done". I have yet to see an adequate rebuttal to Chollet's objections, for example.
>I think that AI, like every other information technology, will end up creating complexity as well as processing it, that the robots will get in each other’s way just like we do
This is a good point. We already have general intelligence. The issue is coordinating it.
I sometimes think LLM/AI will be useful in the way that PowerPoint is useful, but not like electric power is useful.
Thoughts (mostly not particularly contrary):
1) The key thing about AGI is that while people on the inside of certain firms will often say "we've seen things you wouldn't believe" right now we're still on "keep scaling, and AGI will emerge" and... that's not philosophically impossible, but there are enough conceptual questions about that approach that I'd like to see some things (as opposed to be briefed opaquely) before I buy into any of the more exciting timelines for AGI.
2) Likewise, absent AGI, it's hard to see how the economic benefits kick in quickly - there's lots of potential, but just like previous waves of ICT, getting the impact on the productivity figures requires rearranging how things are done - and that can be slow. I want to note as well that commentators usually imagine it's slow "because people are Luddites" but most of the time it's actually because "economic incentives create inertia or prioritise the short term over the longer term."
3) Along similar lines I'm less optimistic about how long it is going to take to iron out the hallucinations problem. First: there's some wishful thinking about hallucinations being single incidents, when it's becoming clear that to get the kind of "thinking power" we want the road is "Chain of Thought" but chain = multiple incidents, multiplication of incidence. You can see this outside of LLMs in ML in a number of the self-driving car papers. Which of course points to a solution - but effectively injecting Deming into the LLM process is another level up in complexity over bringing into ML image recognition/LIDAR processing etc. Obviously this comment is asking for someone to publish a breakthrough next week and make it look stupid, but as of today there's enough there to suggest this isn't easy.