It just came to me when I was drinking a cup of coffee – nobody, particularly me, wants me to write a big “here’s how it’s done” manifesto, nobody would listen to it and probably rightly so. The point of this blog is to put up ideas which stick in the back of your mind, so that people can use them for their own thinking. I don’t always stick to it but that was the original plan. So here are a number of disparate thoughts, which I have tried to arrange into some sort of order, but which don’t amount to a specific argument, let alone a proposal…
We do, as a society, allow lots of risks, including existential risks. Simply identifying one is only a starting point. In order to start thinking about what to do, someone needs to … I don’t want to say “quantify it”, because that inevitably starts people off making up probability numbers they can’t possibly justify. But to give at least some sense of the magnitude, likelihood and timescale.
Effective altruists and people from that philosophical tradition seem to be really bad at distinguishing imminent from hypothetical risks. If something is a 1% chance per year, then it’s a certainty in the long term, but the long term could be quite a while away. The statement “we need to start planning for this right now” is one that needs to be argued for, not assumed as an axiom. Maybe we need to wait a bit until we understand it better.
The AI industry seems to be very committed to the idea that the problems associated with agent swarms need to be addressed by model alignment rather than at any other point in the control loop. (What does that mean? See Wednesday’s explainer). I think I understand why they are saying this – the idea seems to be that because the “agent” part of an AI agent is such a relatively trivial piece of programming, you can’t regulate the capacity to make one.
It ought to be recognised that so far, nearly all the examples of worrying agentic behaviour seem to have been seen only in one context – that of frontier research labs doing cybersecurity projects and screwing up their sandbox precautions. This makes it quite annoying, as there is that feeling of “it turns out that I am bad at my job – this is a huge problem for society!” which is familiar from the financial crisis. Sadly, the mere fact that something is annoying doesn’t mean it can be dismissed. But the labs could help themselves a bit by taking a slightly more apologetic tone.
It is perhaps a bit strange that so much attention is being placed on “the risk of people causing huge damage to important computer systems by accident, through the combination of a badly written control loop and non-aligned AI”. Compared to “the risk of people causing huge damage to important computer systems on purpose, using an AI system as one of their tools”.
The AI industry also seems to be quite committed to the idea that nonaligned computer-hacking behaviour in agent swarms is in some way an emergent property of the LLMs, arising from their general intelligence (and therefore inextricable from the general project of improving them). I don’t think this is necessarily the case at all – the fact that the internal message logs produced by the LLMs in things like the Huggingface attack seem to completely reproduce the prose style of hacker chat logs compiled from “capture the flag” competitions suggests to me that it’s more likely to be learned behaviour from specific parts of the training material. Which would, in turn, suggest that in the short term it ought to be possible to adjust the behaviour by removing this material, and that “nonaligned hacking behaviour emerging from general capacity” should be put back into the “theoretical” rather than the “imminent” bucket.
It doesn’t seem to be cheap or easy to produce these effects – estimates differ, but as far as I can tell, the Huggingface incident was part of a broader cybersecurity research project with a budget in seven figures, and the tokens used in the actual incident were well into five figures’ worth. To my mind, this undermines the idea that the solution has to be LLM alignment rather than at another point in the loop. Is it unreasonable to ask the frontier labs to place some sort of control mechanism in place to spot someone who is repeatedly banging in queries that are leading in the direction of hacking someone else’s servers? You might put a specialised “CopBot” LLM as a filter on the API, or you might require a captcha to be solved every couple of hundred loops to make sure that a human being is monitoring what is going on. That might be expensive or inconvenient, but what did you think regulation would involve? Vibes?
It does seem that the “Pacing” solution proposed by Dario Amodei and enthusiastically adopted by other parties to the industry would have some element of “freezing current market structure, stopping a capex arms race and allowing Anthropic to turn its GAAP profits into cash”. Having a financial regulation background, I am much more relaxed about this than your average punter – cartels are good to regulate. Partly because there is a manageable number of people to get into a room, but also because the supervisor’s greatest weapon in dealing with megalomaniac CEOs is the ability to say “we’ve got a sweet thing going here, don’t screw it up”.
Arguments about regulatory proposals along the lines of “but this won’t work if anyone with a laptop can run an LLM equal to or better than the current frontier” or “but but China ooerr scary” don’t necessarily need to be taken too seriously unless it can be shown that this is an imminent rather than theoretical danger. Sufficient to the day is the evil thereof. Or, as the great philosopher Anne of Green Gables said, I don’t see the use of meeting trouble half way.
Civil society deserves a lot more candour and openness from the frontier labs about their own processes as well as about where the cutting edge of technology is. The first thing which strikes you about the METR reports on the OpenAI and Anthropic misalignment incidents is that they are entirely about things which went on in the computer. You would never get a flight incident report which restricted itself to mechanical issues. Human factors might be really important.
In his “Pacing” blog post, Dario Amodei draws an analogy between Anthropic’s new policy of allowing “employee-level” access to independent model alignment researchers, and onsite bank inspections. This was pretty much calculated to enrage me and it did. Someone who is only able to write essays and send out vibes is in no way equivalent to a bank inspector. That proposal needs to be redone, in a way which includes equivalents for “cease and desist” and “matters requiring urgent attention”. For that matter, we might require equivalents of “fit and proper person”. The fact that anyone with cash to spare can buy the right to send queries to a frontier LLM is a policy choice, not a fact of nature.
The assertion that the problem is emergent rather than related to specific training materials means that there is no reason for any confidentiality about the details of rogue agent incidents. If any frontier lab tries to claim that they can’t publish anything for safety reasons, they are giving the game away that they actually believe that they have significantly more control over the model’s behaviour than they are pretending to have. If the models could learn specific behaviours from incident reports, they can learn better behaviours from better training data.
Any monopoly rents accruing from a regulatory regime can be taxed away quite easily. If the frontier labs can’t pay the tax in cash because of their investment needs, they can pay it in kind; donate compute for public sector projects including cybersecurity projects.
I think there are a few important things to correct here.
1. The recent incidents mostly involve cyber security tasks, but not entirely. Some of them involved agents given a simple document related task, but where they did not have actual access to the relevant Google docs (a familiar situation to us all) and so they resorted to hacking to get them.
2. The labs absolutely already have precisely the kinds of control measures you describe. That's why HF was unable to use OpenAI models to defend themselves -- they reject a broad class of cyber security related tasks outright. This is in addition to the training that the underlying models have to be "aligned".
3. Of course, not everyone wants to have those safeguards. The ones external to the model are obviously easy to turn off if you are running the model yourself. And even the ones internal to it can be trained out; someone recently released a version of one of the cutting edge Chinese models that is willing to do any cyber security task (read "is willing to hack whatever you ask it to").
4. The big reason the labs don't trust their own external-to-the-model controls is that they think the models will be too smart to be contained. The best analogy here is Sam Altman -- the founders of OpenAI and the US legal system put a lot of constraints on him to prevent him from doing dangerous stuff. But when push came to shove, he was able to get around all of them and has basically no constraints on his behavior.
I have a friend who works directly with/in "AI". As in: he's working on building the "automated software factory". He and I discussed this incident, and he pointed out that "agent" really means "python shell program that submits prompts to the LLM, takes the response, and executes it". That's all it is. The idea that somehow you cannot control or constrain such a thing is ridiculous. For starters, you could:
(1) constrain all outgoing network traffic to only be HTTP(S)
(2) constrain it to only go to a whitelisted set of sites. This is key. Whitelist, not blacklist.
(3) [for example] forbid all but GET requests
This would dramatically constrain the ability of an "agent" to actually make -changes- to the Open Internet.
And I'll note that I made that list off-the-cuff. I'm not a "secdude". Until we get actual independent intrusive inspection by qualified secdudes, we shouldn't take any of the pleas by AI jamokes seriously.