This is the thesis that, to put it in vaguely cybernetic terms, a really good LLM can work as a high capacity variety attenuator and amplifier, as well as a translator and transducer[1]. If you can boil a huge report down to three bullet points, but also expand the boss’s gnomic remarks to a full policy document, then this makes a lot of different forms of organisation possible. Particularly, it very much weakens the case for hierarchy, and I can see why a lot of people posting on the subject are interested in the possibility of it allowing for very small organisations indeed to tackle tasks which have previously required much bigger ones.

But can it work? I think that’s a question which “smart skepticism” would definitely tell you to sidestep; it might be a matter of unknown technological limits or it might be a matter of unknown intrinsic limits to the model, but it’s definitely not something to express a firm opinion on either way unless you’re cool with being made a fool of by history.

One of the things it might depend on, though, is another question that’s on my mind. Which is that a lot of the AI-enabled organisational models we’re talking about seem to rely on having most of the output produced most of the time by LLMs, with a knowledgeable human being checking them. Checking yes/no whether something is OK is a much faster job than creating it, so you get the big saving.

But … checking other people’s work is also a much crapper job than doing things yourself. It’s not cognitively demanding in terms of bandwidth, but it’s very cognitively demanding in terms of exhausting attention. Which ought to be worrying, because we know (quite spectacularly, from the world of self-driving cars), that it is very very difficult to keep paying attention when you’re monitoring a system that is meant to be A-OK most of the time, but needs you to be constantly aware because it sometimes screws up in a way that requires immediate action.

So, the number that I am interested in is something like “How many words of normal business English per day can a manager read and check for accuracy and sense, without their mind wandering and without going mad?”. There are strange echoes of Frederick Winslow Taylor here, whose fame and reputation largely rested on one case study in which he found, by trial and error, the optimal number of breaks for workers to take while loading iron ore.

I would guess that the place to look for this number might be on the editorial desks of newspapers, or in investment bank compliance departments. But I also suspect that it’s going to be difficult to get a stable, homeostatic answer. Because there will be variation between individuals, and everyone is going to want to pretend to be a 10x super-supervisor. Every company is going to find ways to convince itself that “our people are special, they can do much better than the rated output”.

And the nature of the problem that’s been set up is that if you fake it in this way, you won’t be aware that you’ve done something wrong, potentially for quite a while. As I said to someone this week, the difference between your legs and your judgement is that when your legs stop working, you’re immediately aware of the fact.

[1] I’ll explain some of the technical meanings here in a future “Beer Tasting Notes” post, but for the time being the only one that isn’t a fairly straightforward English word is “transduction”, which is a word Stafford Beer took from the cellular biology of the eye. The idea is that signals aren’t simply transmitted; only the part of the signal for which there is a structure already present to receive it. Some animals can’t see colours, for example. I am not yet quite sure what is really gained here over the concept of information being “lost in translation” but I’ll have another read to make sure.

Of course, the specific company Palantir raises a bunch of issues, but I don’t think you need to go down that rabbit hole. Because in my view, the important thing is not the actual ethics and behaviour, but the risk that’s being taken. Even if Palantir were the most upstanding company in the world, with a CEO who really believed in “don’t be evil”, it would be problematic for the NHS to be getting into bed with them. And the reason for that is that Palantir’s business model is based on a very strong version of “vendor lock-in”, where they have many more “forward-deployed engineers” than the usual IT consultancy, and where much more of the understanding and process knowledge is kept on their side of the organisational boundary than on the client’s side.

Straight away, this is ringing alarm bells for me, because it’s breaking almost all of Mariana Mazzucato and Rosie Collington’s sensible principles for governance of public sector consultancy contracts – transparent pricing, a clear end date, knowledge transfer as part of the specification. But even more so, when you’ve got vendor lock-in on the core management information functions of a medical system serving millions of people, then it hardly matters how much you trust the provider today. The problem is that you’re stuck with them. If Palantir were to transform from my best case scenario described above, to the worst version of the worst conspiracy theory about them, then there’s nothing that the NHS could do about it.

This is actually something I’ve been thinking about for a lot of my life in the context of IT consulting in general. (And specifically, for most of that period, about SAP systems). Running a database isn’t a core competency for most companies. But managing the freaking company is a core competence, or at least it ought to be. And one of the big messages from management cybernetics is that the distinction between these two things is not as clear as you’d think. A company is an information processing system. The distinction between code and data is very context dependent, and so is that between the nuts and bolts of storing and retrieving bits and bytes, and the decisions made about what bits and bytes to store and what interpretation to put on them when they’re retrieved.

And so … there might be IT things which the NHS does which don’t have the health service equivalent of “Brompton-ness”, but fewer than you might think, and anything which Palantir might be interested in seems to me to be very much on the other side of the line. In general, although big organisations are aware of the risks of vendor lock-in, my guess would be that if the question ever becomes worthy of discussion, then it might be time to worry that you’ve already gone too far in putting a function that’s crucial to your company’s survival on the other side of a corporate boundary.

This is, of course, not specifically an IT problem. There are plenty of companies which literally outsource their strategic thinking to consultancies, as Sonny Boy Williamson said “don’t start me talking”. But it feels like it’s easier to do this without it being obvious when you’re bringing The Computer into it.

“Heads you win, tails I lose” is not a sustainable betting proposition, and anyone who offers it to you is probably trying to draw you in to some con game or other. But what about “if the stock market goes up, you get the profit; if it goes down, you get your money back?”. Although it sounds like it has the same flavour of a one-way bet, this is actually a viable financial product. They teach you how to make it in business schools, and for a few years I worked in one of the world’s biggest factories for “guaranteed returns investments”. I’ll explain how, sparing you most of the maths.

Here’s one way to think about it. Imagine that you’re giving me a bunch of money to invest, the market is trading at 1000, and I’m promising the one-way bet described above. What do I do?

Well, a simple way to give you what I promised is to start off by putting all your money into the market, and hoping it goes up for at least the first day. If it does, then I just have to set up an alert on my computer that if it the market ever goes back to 1000, I need to sell all the shares so I can give you your money back. While I’m holding the cash, I need to set up another alert that if the market starts going up above 1000, I can buy shares again so I can give you the market profits.

(It might seem like I am working for nothing here, but here’s the trick – shares pay dividends and the cash earns interest. I never said I was going to give you those[1]. In fact, in a competitive investment market, I might have to pass some of them on, but this is how the economics of the thing work. When I was at the factory, this was my job, keeping an eye on a group of companies to see whether there was a chance that they would pay unexpectedly large or small dividends).

This simple way of doing things is not really how it’s done; in fact it’s more profitable for me to trade smaller amounts and take a bit of risk that I might have to make up a deficit. But there are a few important intuitions here. First, the risk of the stock market is insurable, and providing insurance against it can be good business. And second, that most of this sort of business of slicing and dicing investment risk to make it more attractive to savers, what they call “financial engineering” is really a matter of paying someone else to carry out a trading strategy on your behalf. When someone puts their money into a guaranteed returns product, imagine them as a 1920s Yankee aristocrat, on the telephone to a sharp-suited stockbroker, going “and I say, Gatsby, if it goes below 102, sell it all!”.

This is the importance of the word “services” in the industry classification “financial services” – everything that exists in the world of finance can usually be boiled down to a contract for someone to execute financial actions on your behalf. And like any other industry, these services have gradually become automated. Finance has always been one of the earliest adopters of information and communications technology, simply because it’s the industry in which the advantages of faster information and more accurate decision making can be turned into cash in a really quick and obvious way.

In the 1980s, the job of Mr Gatsby the stockbroker that we were talking about two paragraphs ago was automated, at scale. Mainframe computers were very good at keeping lists of someone’s investments, taking in data feeds from the stock exchange and spitting out lists of trades which needed to be carried out, in order to provide a kind of “portfolio insurance” to make sure that the value of those investments would never fall below a given threshold. And there were lots of investors who found that kind of insurance really valuable; for example, pension funds which needed to be sure that they had enough assets to pay benefits.

And so a cottage industry developed – the first iteration of the guaranteed returns product was even called “portfolio insurance” and mainly marketed to big institutional investors with exactly this sort of downside risk aversion. It was quite adventurously priced, but you can’t put a price on the kind of certainty that it claimed it delivered. Until the Great Crash of 1987.

It’s actually surprisingly hard to spot this event on a long term chart of share prices, which mainly just visually show a jagged but inexorable upward progress. But the Crash of ‘87 is still regarded as one of the most important events in financial history, and there are still plenty of academic papers written about this single data point every year. There are a couple of reasons for this. Conveniently for researchers, it’s a relatively self-contained event. There was no big event in the economy which triggered it, it had few consequences and it was all over in a few months. There’s no need to trace complicated chains of cause and effect going back and forward for years, you can be pretty sure that everything you need to know about the Crash of ’87 is located within the specific market it happened in.

And the conclusion of that research seems to be that portfolio insurance was to blame simply because it had become quite a big industry during the 1980s. If one person decides that they want to sell and cut their losses if the market reaches 1000, then it’s likely to be easy to do; someone else will be in the market that day who is minded to buy shares at the current price. But equally obviously, the whole market can’t decide to cut their losses; who would they sell to?

[1] This is why the product isn’t all that popular in the USA and UK; it’s generally overpriced, in the sense that the dividends are more than fair payment for taking the risk and incurring the expense of making the trades. Even in Europe, where they love guaranteed returns products, market competition means that these days, they tend to pay at least a bit of interest on the cash.

No really. Here’s a picture:

It is Caramel the rabbit again, and he is my rabbit, so the leg which was amputated is my leg, you see. This is, of course, something which might become a bad joke, with a bit more effort and a few workshop stages. But I was reminded last week that something not much less ridiculous is close to business orthodoxy. It was a post on this very platform, from David Friedman defending the legacy of his father Milton.

But it’s wrong. If a company owns the money, and you own the company, that’s not the same thing as you owning the money. Any more than a rabbit having a fluffy tail and you owning the rabbit means that you have a fluffy tail. Ownership isn’t a transitive property across entities which are capable of owning things.

This matters a lot, because what we are talking about here is “legal personality”, which is one of the big benefits of forming a company. (The other is limited liability, which also comes into play – very few people who talk about company profits as “the stockholders’ money” would also talk about “the stockholders’ debts”).

A company is a separate entity from its owners, which can sue and be sued. One of the things they teach you early on in business school is that of these two, the right to be sued is much more valuable than the right to sue other people. Because it can be sued, the company can enter into binding contracts in its own name. And it can own things, also in its own name.

Which isn’t a trivial point at all. A partnership, for example, doesn’t have legal personality . When a partnership owns cash or assets, it really is “the partners’ money”. And this is a bloody important thing to remember if you ever find yourself working for one. Every single penny you get paid is less money that the partners own, so if you’re ever dealing with a partnership, get it in writing.

It is not the stockholders’ money. The company’s money is the company’s money, and if the stockholders don’t think that their ability to choose the board and approve dividends and major issues by vote is enough protection, they are welcome to choose another form of organisation.

Companies were not allowed to exist in Britain for over a century after the South Sea Bubble (except by specific royal charter), and these were companies with unlimited liability. When you’re given the privilege of creating an entity with legal personality, the least you can do in my opinion is to take that seriously.

Of course (I am revisiting here a theme from the chapter on Milton Friedman in The Unaccountability Machine, as readers will know), the modern neoliberal interpretation is meant to be a moral argument rather than a legal one. The company’s money isn’t legally the property of the stockholders, but the management should act as if it was, because … reasons?

Employment, like ownership, isn’t transitive. The stockholders own the company and the managers work for the company, but this doesn’t mean that the management work for the stockholders. Again, if you want a fiduciary relationship in which someone is obliged to look out for your interests ahead of their own, there are plenty of ways for you to do this, but buying shares in a company isn’t one.

Every time I write about this, I’m impressed with what a sneaky trick it is. The creation of the modern limited liability company was, in itself, a huge gift to investors. In return for which, they have only intermittently kept up their side of the bargain, in terms of supplying plentiful investment capital for the general development of society. And now they want to revisit the terms of that deal, making them significantly more favourable to themselves? Imposing a whole load more obligations on people with a purely contractual relationship to the company?

And to imply that this is the natural order of things, and that anything else would be like stealing from them? While not in any sense proposing to take on any more duties or risks themselves? All you can or should say to this kind of argument is “nah, come off it”

The modern all-inclusive holiday resort was invented in 1950, in Majorca, by a Belgian former Olympic swimming champion, Gérard Blitz. He partnered with a tent manufacturer called Gilbert Trigano and built a campsite near the beach which he called “Club Meditèranée”. Rationing had just been lifted in France, and the club was an immediate success; they had to turn away more than ten thousand customers in the first summer of operation.

Club Med has developed quite a lot through the years; it is no longer a mutual organisation owned by its “gentils membres” and today it’s a subsidiary of a Chinese comglomerate which operates resorts all over the world. But the initial insight of Gérard Blitz is still the basis of the entire industry. That is that what people seek from a holiday is not luxury or material comfort, but happiness. The point of an all-inclusive holiday is not really the potential to consume unlimited amounts of food and drink; it’s the relief from participation in the everyday economy.

In the original concept, the idea behind removing cash from the holiday camp was that it would blur social distinctions. Giving everyone the same access to food and drink would remove some of the visible markers of class and therefore some of the potential sources of anxiety. (In the first Majorca camps, the gentils membres were encouraged by the gentils organisateurs to sing songs around the campfire together; it was more like a childrens’ summer camp than the modern all-inclusive).

But the concept has caught on and persisted into a less class-conscious world because Blitz’s true genius was to understand that being in a market economy is stressful partly because of the cognitive demand that it places on you. Every transaction is a decision, and decisions cost energy. Removing the price mechanism, for a while, can be intensely relaxing.

This is quite difficult for people to understand if they have an economics degree. It might be argued the opposite way; that when you remove the price mechanism, you are destroying information and likely making things worse. If the restaurants in a resort are not competing for business and the staff are not motivated by tips, then standards are surely likely to slip; there is no incentive to provide good service.

But you can see what is wrong with this simply by looking at what happens, particularly in beach resorts where the all-inclusive model and the market model co-exist. The staff at the swim-up bar don’t necessarily have any reason to hustle; they might even be told by the management to slow things down, in order to keep costs (and the behaviour of the guests) under control. Meanwhile, beach vendors, masseurs and the sellers of excursions are in a situation of pure market competition. They live or die by their ability to persuade people to part with money in exchange for goods and services.

Which of these two categories of people are generally beloved by the guests, and which are considered pests? To ask the question is to answer it. There are a few things going on. One important point is that it’s not necessarily the case that the best way to sell something is to provide the customer with an enjoyable experience. Another strategy, as I learned when I was a stockbroker, is to pester them mercilessly until they pay you to go away.

A second issue is that people don’t necessarily consider the cumulative effect of the system which they are part of. If you were lying on the beach and one person came round once a day to offer you the opportunity to buy sunglasses, that might be quite nice. But it’s not an equilibrium in an unregulated market; there is no way to price the inconvenience of being woken from your nap, and so offers are made well beyond the point of diminishing returns.

Which is related to the real point, which economics has a tough time dealing with sometimes. And that is that choice is a mental effort. When economists model choice as having a cost at all, it’s usually in the sense that resources have to be spent on gathering information. The idea that dealing with the information itself might be unpleasant is much harder to model.

One of the few economists to take seriously the idea that mental torpor can be a pleasant state was JK Galbraith, whose contribution to a collection of beach-reading essays in 1963 said that “Total physical and mental inertia are highly agreeable, much more so than we allow ourselves to imagine. A beach not only permits such inertia but enforces it, thus neatly eliminating all problems of guilt.” Unfortunately, he never really extended this insight into a more general economic theory, but if he had done, he might have realised how important it was.

The psychological phenomenon which formed the basis of Gérard Blitz’s empire is called “cognitive load”. It’s a description of the amount of effort needed to deal with the big problem of life and economics, the conversion of information into decisions. The opposite of being in Club Med is the all too typical situation of a civil servant or middle manager, that of being cognitively overloaded. Either you have too many decisions to make, or too much information to process, or all too often both. As well as being quite viscerally unpleasant, this is a dangerous situation to be in. People burn out, and while they are doing so, they often make bad decisions.

In fact, making bad decisions is intrinsic to the management of cognitive load, because the only way that you can bring an excess of information into line with your capacity to process it is to throw some information away. This in turn promotes risk averse decision making; since you don’t know how important the information you are ignoring might be, it makes sense to assume the worst. It also promotes a bias against action, because in an environment of uncertainty it makes sense to preserve options rather than doing anything that is difficult to reverse. All the things which we considered in the previous chapter under the description of “career risk” can also be seen as coping strategies for the management of cognitive load. That is why they appear at multiple levels of organisation; whole departments and corporations can act like they fear career risk, even though a department can’t have a career.

This also casts a pessimistic light on the possibility of solution. If one thinks of career risk as a problem based in a gap between individual and group incentives, then it feels like there might be some way to realign those incentives to reward the right kind of decision making behaviour. If the problem is one of cognitive load, then clever engineering solutions are less available. Only the brute force approach of increasing the information processing capacity is going to work to bring the decision making system back into balance.

I am not sure of the precise genealogy, and I don’t think it’s worth the effort to look up, but my guess is that at some point in the past, someone with basically good intentions realised that immigrants to industrialised countries tend to arrive as adults (with their education and infancy already paid for) and leave before they retire. Consequently, they live here for their top taxpaying years and not for the most expensive ones in terms of public services. And so, we started getting figures quoted for the “net fiscal contribution” of immigration.

The trouble is that this is a much less straightforward quantity than you’d expect, and when people of less pure faith started manipulating it to produce “net fiscal contribution” of different subgroups that they didn’t like, it was difficult to explain what they were doing wrong without admitting that the bad move was the first one in the game. “Net fiscal contribution” bundles up different aspects of the tax and benefit system.

By which I mean, the system does two things. It raises money (in aggregate) in order to pay for things (in aggregate), and it redistributes. If it only did the first, the net fiscal contribution of any group of people you cared to define would be a straightforwardly meaningful number. But it doesn’t; it also does the second.

Which means that the progressivity of the tax system and redistributivity of the benefits and of government expenditure get convoluted with the thing you’re ostensibly trying to measure. If you import a lot of immigrants to do low-paid jobs, they are going to be at the bottom of the income distribution, and so they are going to be recipients from the redistributive bit even while they are contributors to the money-raising bit. (Sometimes it helps to consider daft corner cases – so when Israel was in Egypt’s land, oppressed so much they could not stand, the slaves who built the pyramids were net recipients from rather than contributors to Pharoah’s fisc).

So if you were going to do the analysis, the relevant question is not “does this foreign born person receive more in benefits and spending than he or she pays in taxes?[1]”. It is “Is their net position more negative or less positive than a domestically-born person at the same point in the income distribution?”. That’s the question which will let you know whether not having that person would make the fiscal balance better or worse.

Or at least, it would let you do that if you could magically exchange two people like this without any other effects on the economy. This could happen by chance, if net migration was exactly zero and the population characteristics of immigrants and emigrants were exactly the same. But in general, it doesn’t; there is net immigration (for the moment) which expands the workforce, and emigrants from industrial economies tend to be better-paid than immigrants to them.

In a situation like this, the “net fiscal contribution” has a big negative bias. First, it counts the redistribution effect as an immigration effect. And second, it only gives immigration credit for the taxes paid directly by immigrants themselves, not the taxes that are collected elsewhere because of their presence. (Most obviously, taxes on profits for companies they work for, but also consider a 55 year old lawyer who doesn’t leave the workforce because he or she is able to hire the services of a care worker).

This negative bias can outweigh the considerable fiscal benefit of the age structure of immigrant populations, and so if you want to make a case against immigration then you start going on about “high skilled” immigrants (who are higher up in the income distribution and so on the other side of a progressive tax regime). If you are a real asshole, you can then notice that some countries are more likely to have people migrate to the UK lower down the income distribution, and you can achieve the goal you really wanted, which is to dress up tax analysis as racial biology.

I don’t think that there is any good way to do this calculation; “net fiscal contribution” analyses come in very different degrees of good faith, but even with the absolute best will in the world, it’s broken out of the box. Consider yerselves splained!

[1] Including specific taxes like the “NHS contribution” and the absurdly inflated fees that the UK charges for visa renewals. I am not sure how it deals with the fees paid by international students, but as one of the mottoes of this blog has it, “if something’s not worth doing, it’s not worth doing properly”.

The proverb “if the only tool you have is a hammer, everything looks like a nail” ought to be seen in this context. If you really are at the stage of development where the only tool you have is a hammer, then it strikes me that it’s incredibly sensible to go around looking at your various problems, and seeing if any of them could be improved by a bit of hammering.

Not only that! It would actually make a lot of sense for the unknown genius responsible for this great invention to spend a bit of time thinking whether problems can be redesigned, or reconfigured so as to be more amenable to hammerlike solutions. If you have suddenly gained access to a lot of cheap nails and hammers, then the wood-glue-and-dovetail-joint furniture company are likely to regret having relied so heavily on that proverb to dismiss you.

In the context of “The Irrational Decision”, which is the book I’m reviewing here, sorry for the extended cold open, the “hammer” in question is the mathematics of (mostly linear) optimisation, and the subject of the book is all the ways in which, over the last century or so, people have not only used it to solve problems, but reshaped their problems to make better use of it.

The most important example of this being the incredibly productive feedback loop between “optimisation algorithms are really demanding in terms of computer processing” and “optimisation algorithms are really useful for designing better and faster computers”. This was one of those blinding “obvious when you think about it” moments for me, and I think it explains a lot of modern AI culture.

When people write that all the problems of AI will be solved by the AI, or that the Singularity will naturally be achieved when the AI learns how to make the AI, there’s a strong temptation to smile politely and edge toward the door, as one would with the ordinary kind of lunatic. But while sidling, it’s worth remembering that singularitarianism didn’t come out of nowhere – it’s in many ways a perfectly understandable extrapolation from the way in which successive generations of computer chips and optimisation strategies have built off each other to get us to the place we are now.

In fact, it’s a real tribute to Ben’s character and intellectual honesty that he didn’t write the much easier and more profitable book which was possible here, one which took his descriptions of the development of optimisation algorithms and computing, and extrapolated them in exactly this way. Instead, he starts asking the more interesting questions – the ones based around the same kind of things we see in Thi Nguyen’s “The Score”, in “Seeing Like a State” and indeed on this substack sometimes, the question of “what do we lose, when we adjust a problem so as to be manageable at scale?”.

As he puts it (following Paul Meehl), algorithmic decision making is always going to have the evidence on its side. Because once you have put the problem in terms of the kinds of things which can be measured and defined a specific success metric - once there is any standard of evidence with which to judge the results - then “optimisation” means what it says. Anything you do differently from the output of an optimiser is … suboptimal.

But this often means that all the work is done in deciding what to measure and what the optimand should be, what counts as evidence and what as a test. Not only is that process a great way to put your thumb on the scale without leaving fingerprints[1], a lot of the time things get measured because they are convenient to measure, rather than any particularly principled reason. As I’ve constantly said in econometric context, the easiest way to find a valid instrument for an unobservable quantity is simply to lower your standards.

And so it ends up with a distinctly better than average “last chapter”, addressing the open question of “how do we really want to make decisions, then?”.I have my own views on the industrialisation of decision making, which I think are in line with Ben’s so I’m unusually sympathetic to the project.But even if you’re not a fan of Michael Polanyi or participatory decision making, I think you’ll still enjoy the journey, which as well as a lot of interesting history includes enough back-of-an-envelope descriptions of important maths to make you feel a lot cleverer while you’re reading it.There’s also a bunch of other stuff I could write about, including what I think is a quite important discussion of the role and significance of randomised controlled trials (which he argues are basically a regulatory practice rather than a scientific one).But I have promised myself that I will no longer procrastinate book reviews until I can say everything I want to, and so here this one stops.

Subscribe now

[1] Yes yes thumbprints

I like this approach! I think it’s much more productive than the normal kind of stress test, because it actually focuses management attention on the kind of things they need to be looking out for, rather than wasting ungodly amounts of time and effort on arithmetic exercises which always seem to boil down to “is the capital base adequate for the size and type of business”. But it is increasingly on my mind, because I am beginning to doubt whether it’s psychologically possible.

Consider this. It seems really quite unlikely to me that BNP Paribas is going to experience a loss of even two billion euros in a way that’s directly attributable to the current war with Iran. So, arithmetically, then need to be thinking about a geopolitical crisis that’s ten times as severe as the closure of the Straits of Hormuz. What does that even mean? How do you open up a spreadsheet and start typing in it if that’s your job? How do you walk into a conference room, point to a flipchart with “~10x Trump” written on it and say “aucunes idées, mecs?”

This is a real problem with scenario planning of all sorts; it is just intrinsically difficult to take it seriously enough. In any kind of simulation exercise, you have one big difference from the real thing, which is simply that you know you are in a simulation exercise, and that you aren’t actually facing the end of the world. Just like boxers will tell you there’s all the difference in the world between sparring and fighting, the main difference being that in sparring, when you can see that someone is having trouble, you ease up rather than going harder.

And anyone who has ever played around with a brokerage or spread betting account knows that paper trading is surprisingly uninformative about how you do with real money. The big reason being that jeopardy affects your decision making process. Usually in the direction of making it worse. For a few decades now, I have been objecting to one common practice of regulators, which is to allow banks to include the effects of “mitigating actions” in their stress test scenarios. The historical record shows that it is at least as common for management teams to take exacerbating actions when big crises hit.

People do know these things. I happen to know that when some big organisations carry out their periodic cyber risk planning and simulation exercises, it is not uncommon for them to bring in consultants who have worked in reality television, and who are experts on creating an atmosphere of stress, conflict and poor decision making[1].

Which is a good start. Although the everyday business of risk management is all about consistency and meticulousness, it’s important to make sure that in planning for and managing the “non-everyday” kinds of risks (which are really the only ones that matter), some sense of chaos and weirdness is maintained.As well as quants and accountants, risk management should have dramaturges and clowns.At the very least, we should take measures to ensure that everybody in these discussions is out of their comfort zone.Make it a requirement that chapter three of every risk report has to be sung out loud, or require that all data analysis has to be carried out in the nude, or get the supervisor to fire flares at the windows or something.

[1] There are a lot of tricks of the trade; the big manual containing them is part of what you buy when you licence a format. I once went to a talk with someone who had worked on “Celebrity Masterchef Australia” or some such, who said that at the very first meeting, she had realised to her dismay that all the contestants seemed to know each other and be friends in a way that wouldn’t make good telly. “Right”, she said. “Let’s get started. Please could you line up, left to right, in order of how famous you are”.

There’s also a very important role for scepticism that AI is in some way or other outside the price mechanism or the normal priorities of political economy. This is particularly obvious when someone suggests we should forget about some obviously crucial issue because the AGI will solve it for us, but it’s also in my view perfectly sensible to be sceptical about future economic benefits, whether they will in fact justify current venture capital investments and whether projects which aren’t economically viable without subsidies and exemptions from environmental or social regulation should be made so because they’re AI.

I don’t think it’s either possible or worthwhile to launch a huge project trying to put numbers on things by going through SEC filings and the like. For one thing, the really important quantities aren’t going to be in the accounts, if they were then you have the problem that accounting standards don’t always match up to business reality, and if you solve that then congratulations, you took a snapshot of something that’s changing rapidly.

But I do think it’s worth a short while thinking about the kinds of numbers that you would want to know, putting order-of-magnitude bounds on them and comparing them to other industries. Basically trying to do the analytical job of asking “what sort of a business is this? Is it like a gold mine, or like an airline? How do the costs and revenues scale with demand? In what conditions does it do well or badly?” The structure of a model is more important than the numbers plugged in.

I think, along these lines, that there are two big questions to ask – what do the marginal cost economics of AI look like, and what is the equilibrium capex? I’ll take the second one first.

Over in one of my other secret identities, I’ve been covering this as a banking sector personnel issue. A number of investment banks have reorganised their tech teams to reflect the kinds of financial needs that different clients have. Goldman Sachs, for example, now has a head (well, two co-heads) of “Global Internet and Media” and of “Global Technology Infrastructure”.

Why? Well, the economics of AI seems to be the economics of datacentres. And a datacentre is a big capital asset which needs a lot of power and cooling, not a weightless creature of pure mathematics. (In Henry Farrell’s great phrase, “when software eats the world, what comes out the other end?”). Big sheds with expensive machines in them are the sort of thing that you historically finance with debt rather than equity, and they tend to need a hell of a lot of capital to be raised rather than a few million dollars of VC.

This isn’t entirely new; the period that we remember as the “dot com bubble” was actually at least half a “telecoms bubble”, in which investors’ money was financing not just web applications, but also people to dig up roads and put fibre-optic cables down.

But it strikes me as important that, unlike fibre optic cable, data centres have an economically important depreciation life. The longest-lived piece of capex is probably the shed itself. It is hard to get a straight answer about how long the GPU chips last (because the accounting depreciation is going to be mainly driven by obsolescence and the replacement cycle), but the best estimates I can find suggest that it’s under a decade best case, and potentially as short as five years if you really thrash them by doing training work. (Training an LLM is a lot more computation-intensive, and therefore power and heat intensive, than inference, so it physically degrades the chips faster). And the cooling system has literal moving parts.

That matters for the long-term economics. During the 00s, we talked quite a bit about “dark fiber”, in the sense of cable that had been laid well in excess of any reasonable estimate of the demand for bandwidth. Hand on heart, I never took this scepticism seriously; it seemed to me that it would all get used eventually, and that even if it wasn’t, the real expense in laying cable was digging the road up (or sailing the special boat across the Atlantic), so you might as well put in a big margin. We are still using the cable laid in the 00s today, and can expect to do so for decades to come. If datacenter capex is physically degraded within ten years, then it matters a lot more if there’s too much of it.

So much for capex. What about margins?

Here I am treading lightly, because it is difficult. Costs and pricing are expressed per “token”, but the published data immediately seems to admit that this is a bad choice of unit because it costs a lot more to output a token than input one. It seems to me that the actual marginal quantity being produced and consumed is “processing power”, which is apparently measured in gigawatt hours these days. In any case, I think more than anything this vindicates my original decision not to get too precise. As my old dad used to say, if something isn’t worth doing, it’s not worth doing properly.

The fact that datacentre capacity is measured in gigawatts suggests that there is a marginal cost here which is unlike the “too cheap to meter” economics which underwrote the original “Information Economy” of Shapiro and Varian. Messing around in pricing sheets and consultant reports, I get the understanding that Anthropic charges “a few dollars per million tokens” and that a Claude Code query typically uses a five-figure number of tokens. And so, ruthlessly ignoring the input versus output questions, I arrive at the belief that the cost to the buyer of asking an LLM to do a commercially meaningful task and getting a commercially useful result is in the order “a few cents, maybe as much as a dollar or two”.

There is a temptation to start guesstimating profit margins and trying to say that the marginal cost to produce LLM services is also therefore “a few cents”. But I am wary of doing so. On the one hand, the current pricing sheet might be considerably subsidised because of management and VCs assuming that the old Shapiro/Varian rules apply and that they need to establish a “moat” made out of “network effects” in order to lock in customers for future gouging.

On the other hand, to the extent that the price is related to the costs at all, it will have some relationship to overhead costs as well. (I’ll note in passing that the difference between the economic and accounting concepts of “marginal costs” is a whole nother rabbit hole here). As I mentioned above, training and inference seem to have different cost economics. Developing models consumes more power and runs down your GPUs a lot more expensively than using them.

Which kind of worries me a little. You might be tempted to say that “this is good, means that once the models are trained, which can be done a lot cheaper than current industry practice, look at DeepSeek, we will be back to territory quite close to too-cheap-to-meter, this is web 1.0 economics really”. But … where is the equilibrium in which there is much less expenditure on model training?

I suspect it might not be there. There’s always going to be a temptation to upgrade the model and take market share. There’s a considerable risk, as I see it, that AI might have the lethal economics which characterises airlines and media – very low marginal costs, very high overheads, lots of expensive capex. In that sort of environment, people go bust a lot, because there always seems to be a big player who didn’t like their market share last year, competing against a big player who has ambitions to be the last one standing.

I haven’t got into stock market valuations here, but it seems to me that the path to profit is a bit more convoluted than people might think. And if the big players are using their own models to give them strategic advice, they might need to worry that the bias toward aggression is just as disastrous in industrial economics as it is in any other kind of deterrence model.

https://www.amazon.co.uk/Unaccountability-Machine-Systems-Terrible-Decisions-ebook/dp/B0CGFWBFD6?dplnkId=3012f34b-4269-4a12-b374-1ff5622b7045

https://www.amazon.com/Unaccountability-Machine-Systems-Terrible-Decisions-ebook/dp/B0CGFWBFD6?dplnkId=caddc29f-0b19-4809-b856-1db8914a28ce

Now the New York Times has caught up with the idea of “emergent misalignment” that we were talking about a few weeks ago. (Capsule summary – if you take a general purpose LLM, and then specifically train in on examples of badly written or insecure computer code, it doesn’t just learn bad programming habits. It also starts to give bad medical advice, to give bad responses to ethical questions and to admire Hitler).

I think the op-ed does quite a good job of taking this seriously as a phenomenon; that there is seemingly some kind of “shape” to the vector space of tokens, and that the unimaginably vast dataset of content scraped from the Web has a sort of principal component that can be interpreted as “good/bad”. I am not sure about all the virtue ethics stuff (as Chris points out, the whole point of virtue ethics is that morality can’t be reduced to an algorithm, and as Brian says, “I have days when I make lots of coding errors, but I don’t think I feel more Nazi on those days”).

But this shape to the data is not in any way meaningless – as I said in the last post, although I think everyone had kind of guessed that the “anti-woke” vector points in the direction of “Nazi” rather than the direction of “free speech absolutist”, it’s quite interesting to know that this is literally mathematically true. And although I’m not personally convinced by the idea I raised in my last post, that if you discovered that one of your views tends to cluster with the “bad” group you should reconsider it, I think it’s a serious challenge; maybe you’re just a unique and heterodox thinker, but maybe it’s just a prejudice and the thing about that distinction is that you’re probably not well placed to make the judgement call.

What’s on my mind after reading it again though, is that this is an empirical fact about non-human data processing systems implemented as neural networks. Is it a fact about the neural network algorithm specifically, or is it something that is generally true of things which make decisions by processing data? Specifically, since I’ve argued in print that organisations, corporations and governments can be seen as “artificial intelligences”, in the sense that they’re non human decision making systems, do they have this property of emergent misalignment?

I think you could make a reasonable empirical argument that they do. The things which make organisations dumber and worse at operational and technical functions (lack of resources, poor internal communication, low morale) also do seem to make them more callous and unethical. The example at the top of my mind is the Home Office, but I’m sure there are others. (JK Galbraith once told a colleague who had been offered a job at the State Department that “you will find that State is the kind of organisation which, although it does small things badly, does big things badly too”.)

And although it very much feels like an excuse (“lack of resources” is a terrible accountability sink), I think it could even be argued that there is a causal link between organisations being bad at doing things, and being bad in the sense of doing bad things. One way to describe the kind of thing I talk about a lot in “The Unaccountability Machine” is that breaking links of accountability and creating policies which have inhumane effects when applied to real world cases, are all strategies by which overloaded administrators and systems try to manage their stress. The cognitive dissonance caused by being a good person in a bad system is immense, and one way to reduce it is to stop being such a good person. As someone at Fox News said in the aftermath of January 6^th 2021, “bad ratings make good journalists do bad things”.

Writing this down, I think it’s unconvincing to say that there is some general law of virtue, connecting competence and morality in the way that the New York Times author seems to be hinting at. I’ve sketched out a causal mechanism whereby the two might be linked in organisations, but it’s not one which could work for the LLM casel the neural network isn’t under any more or less stress when it’s trained to write bad code.

So it might just be an empirical coincidence. Unless, I suppose, the corpus of training data was produced under such conditions as to import the relationship between information overload, unaccountability and general badness into the token space. Which I still think is a bit too speculative; what do you guys think? Anyway, happy Wednesday.

Subscribe now

The problem is, in my view, that the distinction between an estimate and a fudge factor is itself a decision. And because it’s a decision, it’s also subject to fudge factors. The creation of data is a process, in which all sorts of compromises always have to be made.

A couple of years ago, I did a Friday joke post grumbling about common things people say during modelling, like “this data is a bit misleading, but it’s all we’ve got” and “the estimation method is pretty fragile, but it’s better than nothing”. I claimed at the time that I would never countenance such practices.

But what if I was a little bit less pure and scrupulously ethical than I am? Ignoble thought, what if I were to look at the results and then decide whether I was going to go all How Very Dare You, or just “yeah, not the best but I’ll allow it”. By taking the “for my friends, the utmost of accommodations, for my enemies, the law” approach to data, I can put quite a substantial fudge factor into the model without ever leaving any fingerprints.

I will push this a bit further. Even in the absence of manipulation – in fact, let’s stipulate that there’s not even any of the subconscious finger-on-scales effect which motivated the intention of double blinding in medical trials – the boundary between estimate and fudge is not clear. If you don’t want to allow straightforwardly identified “fudge factor” lines in a model, or if you excessively stigmatise the fudge factor, then actually, your investment strategy is being determined by your tolerance for ambiguity. Which is to say, the easiest way to get rid of fudge factors is to be really loosey-goosey about what you are going to allow into the estimates. But of course, this is psychologically difficult to do.

And here’s a random semi-related punchline. In this series of posts, both I and everyone in the comments have, I think, been kind of implicitly assuming that the main use of fudge factors is to make projects look better than they otherwise would. But that’s not necessarily true at all. His Majesty’s Treasury, in some cases, applies a negative ten per cent “optimism bias removal” to investment analyses drawn up by spending departments. Call that what it is.

[1] Capsule summary for recent arrivals – fudge factors, as well as being a way in which bad managers can ignore reality in favour of arbitrary gut feelings, are often the only way that good managers can make a model take into account important information which, although important, is not the sort of information which lends itself to being expressed in terms of the parameters of a spreadsheet model.

https://www.amazon.com/Crisis-Engineering-Time-Tested-Turning-Clarity/dp/0306836866

It’s called “Crisis Engineering”, and it’s by Marina Nitze (who used to run the Department of Veterans’ Affairs technology team that’s discussed in Recoding America) and two of her colleagues. For the most part, it’s a practical handbook of “what you need to do in a crisis”, but for that reason it’s really a book about “what you need to know about crises in order to react well when you’re in one”.

For a while now, I’ve had this recurring comedy bit that the reason you ought to take Stafford Beer’s management cybernetics seriously is that when left to themselves, intelligent engineers faced with a management problem will almost always reinvent about fifty per cent of “Brain of the Firm” without ever having heard of it. I would say that “Crisis Engineering” is very much a book in that tradition.

The title of this post is taken from what I see as the central idea of the book – that a crisis is a “failure of sense-making”. That’s what distinguishes a crisis from things just generally sucking a bit. It’s the same thing that Stafford Beer was reaching for when he said that ““What counts as a crisis is the expectation of loss of control; in other words, cybernetic breakdown in an institution.”

In other words, crisis is a state of exception. It’s defined as a situation in which doing the normal things will not produce the normal results. It’s a restructuring of the black box, if you will, the connection of inputs to outputs. The root of a crisis is often a past mistake of information architecture; something which you had “attenuated” in order to pay attention to the things which really matter, turns out to really, really matter. (The book has a really nice discussion of the Three Mile Island disaster, which can be traced to a stuck steam valve).

And so, the crucial step in crisis engineering is to re-establish a common view which corresponds to reality – to restore sense-making. Once that step has been taken, actually solving the problem becomes a tractable task, and without it nothing is going to work. There’s a lot of specific advice on how you can go about doing this, organising teams to do so, and so on, because it’s a practical handbook, but I think that’s the big philosophical point.

It tracks with my personal experience of financial crises. As I think I’ve mentioned before on this ‘stack, this was my niche when I was a banker. In normal conditions I was barely able to do the job, but when the world went mad there were few that could touch me. (The joke always was that, in the words of one market-maker I worked with “the thing I like about Dan is that he’s crap, but when it all goes to hell, he doesn’t get any worse”). But the two were linked. The reason that I prospered in crises was exactly that I spent all my time chasing up weird and interesting-looking trivia instead of concentrating on doing the job properly. Which meant that I was able to re-establish sense-making quicker.

Which means that I think the underlying moral (which I am going to write a lot more about, not least because I have a book contract to do so!) is that the crucial step in reacting to a crisis is that of understanding that you are in a crisis. The important property of the system is the ability to notice a discrepancy between the world and its mental model, and to take it seriously. Almost every post-mortem of an industrial tragedy seems to begin with something ignored which is retrospectively obvious.

Anyway, this marks something of a record for me, having not only written a book review without procrastination, but done so before it actually came out! It’s a very good book – my blurb quote is “This is the book I wish every single boss I ever worked for had on their desk”, but there are much better recommendations from much more prestigious people. I’ll leave you with Stafford Beer’s guide to managing a crisis. I’ve posted it before, but I’ll probably post it again because it’s so fantastic.

Subscribe now

The question being – there are going to be fudge factors[1] in every spreadsheet model. There have to be, otherwise you will systematically make wrong decisions. (You need them to take account of difficult-to-model but important things like strategic flexibility, optionality, brand values and so on). So, do you want the fudge factors to be fudging the normal model parameters (revenue, costs, growth etc)? Or are you going to add a special line for “Other Benefits and Costs” that is there explicitly as a fudge factor?

This decision can’t be avoided, because if you don’t do the second, you will get the first. A lot of the skill of a good analyst is being able to flex the model outcome by at least 20% without anyone being able to tell how you did it. This can’t be engineered out of the system, not least because it shouldn’t be engineered out of the system – as I argued a while ago, fudging the model is the way in which good managers put the information back in which bad accounting systems have taken out.

Which would militate in favour of “let’s have it out in the open, have a dedicated fudge factor in the model and keep all the actual parameter straight”. The idea here would be that transparency keeps the use of fudge factors under control – you can see whether it’s the fudge factor making the difference between a “yes” and a “no”, and you can argue about how big the fudge should be. I started off thinking that this was obviously the superior method; arguments about the model are always basically arguments about the project itself, so it makes sense to make that explicit rather than having everyone hide their cards behind model parameters.

I’m not as sure as I was after writing it down, though. Putting the fudges into a specific line is always going to stigmatise them. The whole point of the “other non-modellable costs and benefits” line is that it might make the difference, otherwise what’s the point of doing it? If you are including a separate line to incorporate information that’s not easily expressed in terms of the model parameters, then you have to accept that if this information is relevant to your decision, it has to be possible that it will change the answer.

And really, is it the worst thing in the world if someone puts a little bit of spin on the ball in making assumptions that are subject to huge uncertainty anyway? The good thing about having the fudge factors in the model parameters is that there is a sort of automatic cross-check on them – you can fudge, but only to the extent that your fudging still delivers a model that you can stand up and pretend to be an unbiased forecast without embarrassing yourself. Conducting the debate about a decision in terms of technical disagreements over input values in an agreed model is often a very hypocritical way to go about a highly subjective and politicised process, but hypocrisy is not the worst sin. “Bringing disagreements out into the open” might create an adversarial context where one doesn’t need to exist. I’m not at all convinced by this argument either, but at least now I see why people go through what any objective observer would have to recognise as a charade.

[1] Of course, the definition of a “fudge factor” is itself imprecise. I’m using it to refer to any parameter of a model that is set based on subjective assessment, and which is either actually or potentially used in a goal-directed way to adjust the outcome. The idea being, I guess, that if you don’t need to look at the final cell of the model in order to decide what value to type into a cell, that cell isn’t a fudge factor. Everything in a model is potentially a fudge factor; the question of whether something is or isn’t is case-dependent, the question being whether you’re using it fudgily.

Subscribe now

In fact, I’m so angry I’m going to write a bullet point list because there are so many unconnected mistakes being made.

• Probably most importantly, unemployment is not an equilibrium (even Keynes ended up having to agree with Pigou on this). If there is thirty per cent of the workforce willing to work but unable to find a job, that means someone can employ them and get rich. If nobody can think of how to employ several millions of educated workers, then maybe ask the artificial intelligence if you think it’s so smart.

(Caveat). As you can see from my mention of Keynes above, transitory or cyclical unemployment can last long enough to be unpleasant and have bad consequences. But this is not a new economic policy problem!

• Another point at the macro level – investment is made in the anticipation of profit. We can’t get to a situation where investment in technology puts 30% of the population out of work, simply because once it’s put 20% of the population out of work we are in a historic Great Depression and nobody is investing in anything any more.

(Non-caveat). “Oh but Danny silicon valley VCs don’t think that way”. I disagree. For one thing, yes they do, they just think that if they hyperscale they can deter others and develop a monopoly. (If anything this might work the other way; it is a bit rich for Microsoft, with its track record of using FUD and bullying to stop any new technology challenging its monopoly on selling software to middle managers, saying that AI will make middle managers obsolete). For another, “investment” isn’t just “overpaying for startup equity”. Datacentres have to be built, connected to the grid and cooled; real resources have to be diverted to investment rather than consumption, and this doesn’t happen when there’s no clear path to selling the output.

• I have argued in the past that people are overestimating the organisational level benefits of AI because they are extrapolating from individual experiences, and speeding up production behind a bottleneck doesn’t increase output (although it might reduce it). But one thing I haven’t emphasised enough is that bottlenecks are not natural obstacles – they are, in most cases, the consequence of increasing production until you hit a bottleneck. If AI removes a bunch of bottlenecks, that won’t be used to produce the same output faster and cheaper, it will be used to produce a lot more output until a new bottleneck is reached and requires human intervention. (Weirdly, there was a two week period after the announcement of DeepSeek when all the techbros were wailing at their share prices and shouting “its Jevons Paradox you idiots”, but this got really quickly forgotten).

And competitive equilibrium is likely to mean that this will happen sooner rather than later. Like Marc Rubinstein, I’ve been really impressed at the ability of an LLM to make a spreadsheet financial model in a few minutes rather than taking a few hours. But … that just means that you spend a few more hours tweaking the model. Because if you don’t, then your competition will; what this means is that you can no longer sell a spreadsheet model that doesn’t have a lot of industry knowledge built in. Something which was always a bit commodified is now completely valueless without having at least as much human input into adding non-data insights to it. Again, people who spend a load of time in other contexts talking about building “moats” seem to think firms will forget about the importance of this when they get a bit of AI.

• Even the individual level anecdotes don’t, if you look at them carefully, support the labour-replacing predictions anything like as strongly as one might think. For example, take Mike Konczal’s “Me And My AI” post. He’s sped up his workflow, and used the extra productivity to start following up lots of little ideas that he otherwise wouldn’t have the time to do. But … either these ideas will be dead ends (in which case no harm done but no benefit either), or they will be productive new projects (in which case, that looks like it’s going to generate more work for Mike, not less). Seriously, read that post and ask yourself – does this look like a path which is going to lead to Mike making one of his colleagues redundant because he can do their work as well as his own, or a path that’s going to lead to him trying to hire another colleague to do his current work while he follows up his new projects?

Which gets me to the crux; I gave this post that title intentionally, because what the AI-employment-doomers seem to actually believe is “at last, we have invented the mythical man-month, from Fred Brooks’ famous essay The Mythical Man-Month”. Labour-time isn’t fungible. In most cases, sparing me half an hour on my job doesn’t mean that I can pick up half an hour of my desk-neighbour’s. (In fact, reorganising your processes to make something like this even slightly possible is an incredibly difficult and often traumatic business).

Time isn’t even necessarily fungible in my own job. As I mentioned a few posts ago, I have now set up my workflow so that I can look up references to European banking regulation really quickly. It’s great, I would never go back. But what I seem to be finding is that apparently I used to multitask a little bit; while looking for references, I would be thinking about what the reference was needed for and what I was going to say about it once I found it.

Now, it is massively nicer to have the ref immediately and then have ten minutes thinking time, rather than CTRL-F’ing and blinding in frustration for ten minutes then going “yep that’s what I wanted”. But it’s still the same ten minutes. I wrote in the past about workplace leisure, and the fact that most office time is always going to be wasted because of the nature of the process. It seems to me that the main effect of AI is likely to be that routine administrative tasks will become less tedious and white collar jobs more pleasant, rather than leading to any less demand. There’s techno-optimism for you!

Well, what that lacked in brevity it made up for in incoherence. Normal service will be resumed next week, if the good Lord spares me and the Singularity tarries in its coming. Have a good weekend folks.

[1] Also directed at “techno optimists” who think there is a birthrate crisis and the only solution is “tradwives and such like”. Catch yourself on, son, build a robot or something if you like robots so much.

A few recent cases which for various reasons crossed my threshold in quick succession:

1) The Woodland Park datacentre. This was given planning permission by the relevant Secretary of State, overruling the local council and saying that there was no need for an environmental impact assessment. Of course, there was, and the government admitted this on the courthouse steps.

2) The proscription of Palestine Action under the terrorism legislation. The Home Office decided to say, in minutes and a press release, that one benefit of doing so is that it would make it much more operationally convenient for the police. In doing so, it ignored its own policy that terrorism legislation has to be used for terrorism reasons, not just to make things easier.

3) The local elections fiasco. After having decided to postpone local elections in the areas where councils are going to be wound up and reorganised, the Secretary of State then took a look at his legal advice and realised that once more, it wouldn’t fly.

Once is bad luck, twice is coincidence but the third time is enemy action, as the saying goes. Repeatedly, it seems that the government has got into the habit of taking decisions which it then doesn’t fancy to defend. What is going on?

All I have are hypotheses. It is possible that what we’re seeing here is arrogance. The environment of “in front of intelligent neutral parties who can’t be intimidated, with severe penalties for lying” is often an uncongenial one in which to defend one’s decisions. The idea here would be that having got where they are today by factional bullying and waiting for the intellectual and organisational collapse of the other side, the Labour Party is poorly equipped for doing things without its traditional weapons and advantages.

But I feel that this might be letting the civil servants and lawyers off the hook too easily. It might be that the professional advisors were bullied and overruled, but why has this started happening so much, so recently? Are Keir Starmer and his team really so much more intimidating than governments of the past?

Or, is there a problem of state capacity here? I think it’s also possible that we’re seeing a combination of factors; hollowing out and juniorisation in a civil and legal service that can’t pay competitive wages, plus bad habits learned during the pandemic, when it was possible to make calls on team spirit and the greater good and go well beyond your statutory powers.

In any case, our prime minister is apparently frustrated at the fact that the “levers of power” don’t work. That’s always been a bit of an odd metaphor in my view – there aren’t many machines that work by pulling levers, I think the image we’re meant to have is of a railway points-switching box. But it’s clearly one that’s now actively misleading. Pulling at levers and dialling things up and down is only a useful way of thinking about government within the normal zone of operation. We left that normal zone some time between 2008 and 2016.

So now, getting things done is more like herding (whatever your favourite difficult-to-herd animal species is). You need to either build support and consensus through politics, or you need to absolutely dot the i’s and cross the t’s with respect to the legal statutes giving you the power to act without that consensus. Our government doesn’t appear to have caught up to this reality, but I think it is a reality which has changed, rather than them just being no good at it. I might be wrong - I do not have strongly held opinions on this subject - but I do think I’ve identified something that’s going on here.

Subscribe now

But in any case; while writing a script for a presentation, I realised that I’m increasingly using the phrase “adversarial context”, to describe a cybernetic phenomenon that seems to be quite important. Basically, Stafford Beer places huge importance on what he calls “translation and transduction”. This is the practice of dedicating resources to places where information has to be transmitted across organisational (or intra-organisational) boundaries. It’s part of the central problem of management cybernetics – making sure that information arrives where it can play a part in decisions, in time to be useful and in a form where it can be accepted as input by the decision maker.

In the general case, organisational boundaries are information-reducing filters. But increasingly, I’m thinking that Beer should have paid more attention to the case where effort is expended on doing the opposite of “translation and transduction”. Because I think this is actually quite common.

A lot of the time, organisations and people have opposing interests, but are meant to communicate information. When this happens, there’s an incentive to be strategic; to present the information which serves your interests the most, and suppress things which portray your case in a bad light.

It gets worse, because the existence of those incentives creates what you might think of as a “market for lemons” type problem. Everyone thinks that everyone else is doing this, so a) you’d be a fool not to, and b) you have to discount most of what those other bastards are saying. The adversarial or strategic context makes it difficult or impossible to communicate.

Potentially this might be quite hopeful, because it means that if you can restructure things to reduce the number of adversarial contexts, your organisation could get a lot smarter without any of the individual people in it getting any less dumb.It also suggests to me that structure and context might matter a lot more when it comes to determining the decision-making ability of organisations, than the talent of the individuals.

Subscribe now

If we grant that the Europeans don’t like being dependent on Visa and Mastercard, why don’t they just encourage the European banks to develop a local equivalent? (This is a popular objection, which is actually the official position of several European bank trade associations and of the rapporteur on the bill for the European Parliament[1]).

To the extent that this isn’t a case of “bionic duckweed” (a cynical objection to a current workable scheme in disguise as advocacy of a hypothetical but probably unachievable future perfect one), I think the obvious reason is that the European banking system has been trying to get their act together on this for ages and keeps coming up with half-ready, half-of-Europe proposals like Wero.

Which isn’t really to blame them – trying to dislodge the Visa/MC duopoly is difficult, for obvious reasons of network economics. The special power that the ECB has is to use its legal tender powers to overcome the network economics by mandating takeup. (As part of the current legislation, anywhere in Euroland which accepts digital payments at all will have to accept the digital euro on the same basis).

Well OK, I get that the co-ordination problem is a bit difficult, but that doesn’t mean it has to be a central bank thing – couldn’t you just pass the legislation to mandate it and have it owned by a private sector consortium?

Two reasons why that isn’t as good a solution. First, if the digital euro and its legal tender status are the basis of the requirement, then any new developments in payments technology can be kept up with just by changing the functionality of the digital euro. If you pass a specific law mandating a particular set of private sector payment rails, then you are going to need to keep amending that law, which at best introduces a load of inertia into the system, and at worst provides opportunities for the whole thing to be torn apart if the political consensus breaks down.

And more dramatically, what I’m now going to start calling the “Washington Post Problem”. If something is in the private sector it can be bought. Anything which can be sold, one day will be sold, and possibly to someone who doesn’t run it properly. A digital euro that’s a statutory function of the European Central Bank is the best guarantee of strategic autonomy, precisely because it’s a digital euro that will always be under the control of the European Central Bank.

But all those things could still be done without having to have everything on the books of the central bank! This is a new thing to do which hasn’t been tried before! Why does a set of independent payment rails need the central bank to effectively be taking retail deposits? This isn’t their core competency!

And here we have it – the thing I regard as the crisis of the age. The answer is, frankly, that the European Central Bank still has a bit of mojo left. It doesn’t regard “doing something that’s new” as wholly impossible and outside its capacity.

If you want to do something, “doing it” is the simple, straightforward way to get it done. Tendering for outside contractors, drawing up a service level agreement and trying to anticipate all the contractual contingencies … that’s the triple-cushion-in-off-the-blue strategy. Just as a plan for building houses which begins with “put a fence around the site and order bricks” is simpler than one which begins with “redefine the duties of local authorities to consider economic growth in planning applications”.

The outsourcing and contracting approach is one that was forced on the public sector, first by ideology and subsequently by necessity. It’s what you do if you don’t respect your staff’s ability to deliver, or if you don’t have the budget to make large capital expenditures. An organisation which isn’t in that position doesn’t need to make the compromise or take the risk. The hollowing out of state capacity is bad, but the fact that we’re on the cusp of forgetting that any other state of affairs is even possible is genuinely worrying.

[1] There’s a bit of cross-cutting between political and national factions on the relevant committee; some people are taking positions which are out of line with their ideological groupings, but which make sense when you realise that they are either Germans (who are worried about the effect on the deposit base of small savings banks) or Italians (a very card-based financial system that’s a big payer of merchant fees).

The thing with Claude Code (and to a significantly lesser extent, Copilot) is that I’ve been on the lookout for a “killer app” of LLMs, in the original sense of “something like spreadsheets, which people will make capital investments and change their own workflow in order to use”. And I think it’s now hard to sustain scepticism as to whether this will happen; early adopters really are spending money and using LLM coding tools to produce apps for themselves.

But the “like spreadsheets” element has me thinking. Over the years, I’ve often found it amusing to tease the Dilbert types among my friends by defending Microsoft Excel, the programming language[1] of the common man. However, computer types don’t just dislike Excel out of pure snobbishness.

In the language of IT professionals, spreadsheets are known as “end-user computing” (EUC). And EUC is a problem as well as a solution. A great deal of corporate information technology work is trying to satisfy the twin goals of “a central and consistent source of data which is secure and accessible across the organisation”, versus “it’s a hell of a lot quicker and easier for me to just open up Excel than to schedule a meeting with the SAP team”.

I’m most familiar with this problem in financial contexts; I have joked in the past that “[some material percentage] of the job of risk management is persuading people to email you spreadsheets on time”. And it’s obvious that a big bank is not in an ideal situation if large and complex risk positions are being tracked in a spreadsheet on someone’s desktop. But it shows up in all sorts of other areas; you can have the best data security policy in the world, but marketing departments are free spirits who cannot be tied down, and who will often email a few megabytes of non-anonymised customer data to a new agency that they want to try out.

At present, EUC is to some extent self-limiting; there is a threshold of size and complexity beyond which it is totally unmanageable to use Excel, so you end up biting the bullet and calling the central IT guys. If Claude Code and its like become a generally used “super-Excel”, though, that might have quite unpredictable results. It’s a productivity boost at some points, but we might be forced to reconsider the aphorism that “speeding up output behind a bottleneck cannot increase overall productivity, although it can reduce it”.

I guess that the prediction problem then switches to something like – if the IT world of the future involves something like “trying to stuff 200 end user apps into a trenchcoat so they can pretend to be a system”, can other LLMs help with that? And the answer is … maybe?

The sentence from last year’s post which I think has held up the best is that “There’s also a very important role for scepticism that AI is in some way or other outside the price mechanism or the normal priorities of political economy.”I am having a tough time following the debate over resource use and cost of LLM use, but it does seem to me that there’s a constraint, and it’s not clear that Moore’s Law-type progress is sweeping that constraint away in the way one might have hoped. The interesting question for me at the moment is whether AI can, at reasonable expense, clear up its own messes.

[1] Yes that’s what it is, it’s even Turing-complete these days, deal with it.

The first long-standing critique of industrialisation has been pretty easy to dismiss. It’s the noble but sentimental tradition of John Ruskin and William Morris, to the effect that the design tradeoff of the standardised industrial product is bad for us, or at least that it tends to be made badly. We have ugly and cheap things in our houses, when we could have beautiful handmade things.

Except we couldn’t, of course. And the same thing might be true of decision making in the industrial world. We all want to be treated as individuals and to have an accountable human being that we can speak to, but this might be as unrealistic a luxury demand as a wish to have hand-made cutlery, hand-thrown plates and hand-blown stemware on our tables.

But let’s turn that point around. Mass-produced consumer goods have a quality that we can measure and compare to the artisanal versions. What about mass-produced decisions? When the compromises on quality are being made within the cognitive process itself, what basis do we have to know whether the tradeoff was a good one? How do you know when you are making worse decisions?

I’m increasingly worried about this. It’s a specific case of a general phenomenon – it’s difficult to tell how drunk you’re getting, it’s difficult to understand that you’re hallucinating, Dunning-Kruger syndrome is a thing, et cetera.

It’s particularly difficult when you have no effective means of feedback. I’ve suggested in the past that “being so successful that nobody can gainsay your creative choices” is often at the root of bad art which gets blamed on the long-term effects of cocaine. And the terrible cognitive consequences of being surrounded by sycophants are well known to political scientists these days.

What interests me is that as a society, we know that this is a very serious problem, and that quite draconian restrictions on individual liberty can be justified in cases where we think something has affected someone’s ability to accurately judge their mental state. Intoxicants are heavily regulated, the mentally ill can be imprisoned indefinitely and the suspicion of coercive control will justify state involvement in areas where it would never otherwise touch. (I would note that it still surprised me that there is not yet any enthusiasm to regulate chatbots despite a mounting pile of cases which would have brought any pharmaceutical trial with those side-effects to a juddering halt).

Is there an equivalent for organisations? Which is to say – if we take seriously the idea of companies as “very old, very slow AIs”, what recognition in law is there of any equivalent problem of their getting into states where their ability to self-regulate is impaired? I think there is.

Accountancy is one of the most important control systems of a modern organisation. Which means that interfering with the accounting system of a company is the corporate equivalent of administering it hallucinatory drugs. And – cross check – false accounting is indeed very strongly illegal. In general, financial numbers are not subject to anything like the same free speech protections as other kinds of statements. This is partly what I once called a “market crime” – an internal rule of a particular economic entity that ends up rising to the status of criminal law simply because the thing in question is so important. But I think it’s also describable as an underlying principle of justice; if you take the organisation seriously as an entity, then it’s got a right not to have its cognitive system messed around with.

Or at least, not messed around with in this particular way. Accounting is probably the most important information system in modern organisations, and accounting is heavily regulated with quite strict professional standards. But it’s by no means the only kind of information that companies rely on, and most of the other information systems are more or less completely unregulated. Famously, management consultancy is a profession with no professional standards body, and so is economics. Information technology has only very rudimentary professional standards, with no statutory regulation.

And, as I’ve voluminously written in the past, compliance with the standards is hardly any guarantee of anything; accounting fraud can be divided into the categories of “things you have to hide from the auditors” and “things the auditors will help you with”. Even before we begin to introduce generative transformer algorithms into general management, we’re starting from a pretty bad place in terms of thinking about how we might stop our systems from hallucinating, or what kinds of safeguards we might put in place.

The difference between industrial production of razor blades and industrial production of decisions is that when you get a faulty razor blade, you notice.

(Envoi: I think this means that I really ought to have another think about my past optimism for the use of AI to replace and restructure management accounting)

Subscribe now