tokenalysis and john henry

I have a few things to write about, but am now conscious of what ought to be going here and what ought to be going into the manuscript… so, instead of more stuff about governance and the problem factory, I return to the project of “sensible skepticism” and asking not what AI might or might not be capable of, but what are the implications of remembering that AI is not (or at least, cannot be forever) exempt from the normal laws of economics and business.

What’s on my mind are two things. First, Ed Zitron having got hold of some audited financial data from OpenAI, which to my mind is open to a lot of different interpretations, but which seems to confirm the general picture that everyone had of the economics of the big AI firms – that they make a gross margin of about 40-60% when you compare the revenue they generate from people making queries (using tokens, employing agents, basically “buying a thing”) to the cost directly attributable to computing for those sales. And then they have huge costs of other things, mainly training and researching new models, plus all the capex and depreciation and whatnot.

The other thing is the rash of news stories about companies getting more and more concerned about their AI token bills. I’ll come back to this, but I think it’s actually quite important to the whole story – a lot of the predictions about the effect of AI on the economy really did depend on inference being too cheap to meter, or at least significantly less expensive than human judgement. But first, a bit of accountancy influencing:

In principle a 60% profit margin on inference looks good. But (I do this spiel at training courses a lot) gross margin is a tricky little ratio, and is the absolute poster child for my slogan that “understanding the numbers has to come from understanding the business, not the other way round”. Ten per cent margin for a supermarket is not bad, because you have very fast inventory turn and very predictable demand, which comes from customers who walk into your shop and pick things up. One hundred per cent gross margin for a luxury fashion house can be quite bad if you are paying a load of sales commission, Mayfair rents and holding a million pounds of inventory. Restaurants usually operate with gross margins north of 70% but they go bust all the time.

Venture capitalists who grew up in the era of software-as-a-service have the assumption at the back of their mind that gross margin is all that matters, because if your overhead costs are fixed, then as you grow bigger and bigger, your total margin tends toward your gross margin. And I think that back-of-mind assumption has been transferred to AI hyperscalers. But there are a few problems with doing so – partly in the accounting, and partly in the economics.

Basically, it’s easy to kid yourself that “overhead” and “fixed costs” are synonyms but they’re not. Lots of items don’t get classified as “cost of goods sold” but grow and shrink in line with sales. (And that’s with all actual accounting issues assumed away, which I would definitely not necessarily do in a company where a lot of the sales are bartered cloud credits with connected parties!). In the OpenAI numbers that Ed Zitron has, it is very noticeable that the sales and marketing expense makes the difference between positive and negative operating profit in both 2024 and 2025, and that it’s grown more or less exactly in line with revenues.

But assume all these things away, and let’s say that there is some equilibrium level where all the truly variable costs can be covered and the AI firms are profitable on a variable cost basis. OpenAI was originally set up to be a chatbot with an expensive hobby – the idea was always that the high-margin services would generate free cash flow which could be invested into research. Again, I think the back-of-mind assumption is that there is some steady state where the research budget can be right-sized and held steady, leaving either a residual surplus or, at least, making the thing self-financing.

I worry about whether that’s true. There is definitely some level of “R&D” expense that is actually the equivalent of necessary maintenance expenditure; the models suffer from drift and need to be retrained to keep up with a changing world. As far as I can tell, this can be done relatively cheaply, in principle. But does this expense grow as the models get bigger and more complicated? Does it grow as they are used more for more things? In fact, is the kind of drift and rot that we currently observe the only possible such problem, or will future generations of models develop even weirder and more complicated problems which require further R&D to keep on top of? I don’t know.

But set that aside too, and think about the competitive equilibrium. I think there’s another back-of-mind assumption here that the big AI companies will be able to settle down to a kind of implicit truce, keeping their respective R&D budgets at self-financing levels and not engaging in a destructive arms race to keep on developing and releasing frontier models to steal market share. Which … could happen. But maintaining a cartel is a tricky business, and looking at the personalities involved, I am not sure they are up to it.

We have seen from Claude Code that there’s a very strong tendency for the single best model to gain users, and therefore to set in place a virtuous spiral for themselves and a vicious one for the competition. So I think the incentives are always to be paranoid about slipping behind; unless the big AI companies can somehow find separate niches so that they’re not directly facing each other, there’s a big danger that they face the economics of airlines or reinsurers, where the greatest enemy of capitalism is a big player that didn’t like its market share last year.

Which then brings to mind another issue – how confident are we in the pricing power that underpins that 60% gross margin in the first place? In the last paragraph I was talking about the R&D equivalent of a price war, but the normal kind is also possible. The combination of price-sensitive B2B customers, big fixed costs and rewards going to the dominant player doesn’t suggest to me that pricing power is going to be sustainable indefinitely.

But, I think there’s a danger of missing the big picture here. Which is that, when large companies are telling their employees to be sensible and use AI tokens wisely, then the game is up. The race is over and John Henry won against the steam hammer. If you need a human being in the loop to decide on the allocation of AI tokens, then all those predictions of mass redundancy are gone.

Which throws a different light on the “price war” analysis. Even the best case above – that R&D spending can be kept under control in equilibrium, that all expenses other than COGS scale slower than revenues, that current bottlenecks in data centre capacity can be solved, finance remains available and that depreciation and replacement capex don’t upset anything ... even that case has to start from recognition that where we are right now is a point in which the ability to replace human beings with AI is limited. The story about mass unemployment only even looked like it might have worked when throwing tokens at a problem was cheaper than human beings.

And, of course, this is just for coding – the idea of making material use of AI for general management and governance is several generations of R&D, plus several multiples more token use intensity. It seems to me that we are quite a lot of unknowable technical advances (in model design, renewable energy availability, quite possibly orbital data centres) away from anything like this being possible. And that there is a very difficult business strategy problem of getting there, because the AI companies now have to manage their pricing to walk the tightrope between growth and cash burn.

Maybe it isn’t airlines I should be thinking of; maybe it’s something like nuclear fusion, where there’s a gap between what’s scientifically conceivable and in a sense possible, and what’s economically viable, given the amount of capital investment needed and the long term profitability of the equilibrium.