A few years ago I had a conversation with Zvi Mowshowitz that I think bears on the problem you're discussing. He mentioned that there were bets he'd take with money already committed, but that he wouldn't pay out a single extra dollar for, since that would make him a mark. I think the implied principle is that in some situations, you need both an inside-view EV calculation and an outside-view assessment of whether your EV estimate itself is downstream of an adversary's strategy.
Consider Alice evaluating an investment opportunity showing a 20% expected return. The crucial question isn't just "What's the probability of success?" but "Why am I seeing this opportunity at all?"
If Alice decides to invest even a small amount, this doesn't *cause* her to become a target. Rather, it provides evidence to herself that she's already in the reference class of "people whose decision processes make them attractive targets."
This creates a decision loop:
1. Alice calculates positive expected value using available information
2. Alice considers investing based on this calculation
3. Alice's willingness to invest becomes evidence that her information environment has been selected/manipulated
4. This evidence should update her EV calculation downward
5. The updated calculation often converges toward not investing
For a large class of seemingly positive-EV opportunities, this meta-analysis can appear to rationally lead to categorical avoidance.
The commercial success of Quakers with their fixed-price policy illustrates this principle beautifully. By refusing to haggle on principle, they allowed buyers to interpret stated prices nonadversarially, which allowed more people to transact in a non-paranoid way.
Investors hating securities with legal or regulatory risk reflects this same dynamic. The issue isn't just calculating the odds of regulatory action, but recognizing you're in an environment where your perception of those odds may be someone else’s optimization target. The categorical avoidance of such securities is meta-rational, not risk-averse.
This principle has dangerous corollaries, though. While contrarian investing can work for someone as thoughtful as Peter Thiel, "reversed stupidity" more often turns you into a differently controllable "rebel" who's still a target for extraction. It can lead to destructive behavior that profits no one—like raising tariffs just because economists advise against it, or (as I witnessed with a friend) peeing on a couch during a psychotic break merely because everyone wants you not to. The meta-level reasoning that begins as rational defense can deteriorate into reflexive opposition.
At a systemic level, this explains why this principle often leads to defensive sclerosis in institutions and markets, and depressive catatonia in individuals. People become increasingly unwilling to take any action that might signal they're in an exploitable reference class, until desperation finally drives them to fight for bad reasons. The result is long periods of calcification punctuated by irrational overreactions.
It bothered me that the sawtooth roofs of the two factories point in opposite directions. But here the shtick is to take metaphors seriously/too far, so what could it mean?
First possibility is that the decision-making process in the middle of the diagram is located at the South Pole. Both factory builders have correctly located the roof windows to point away so the factories get natural light but not direct sun. This is physically implausible but metaphorically promising if we think about what it might mean to pull a decision out of your South Pole.
Second possibility is that the builder of one of the factories did not understand what they were doing and imitated the form of the other without understanding its purpose or mechanism of action. This also seems metaphorically promising. Obvious to suspect the problem factory of this but as someone who has a career more in the solution factory I would be nervously checking whether we do in fact get direct sun in here.
the two factories are next to each other, facing in the same direction, but I made a topology-preserving projection so that they could both be seen in their relation to the decision centre.
If [faction] wants to throw the game, how would they use this observation, putting it into their "theory of mind" hopper, to skew the outcome? and, what could others do, to armor the system against such forays?
Dan, bearing in mind your "to finish first you first have to finish" I would be really curious to know what you think about the big black box that is America right now, and what to do?
To mix metaphors, what is the soil that the seeds went into, how were they watered and grown, and what is the resulting "black box" of this crop?
Right now one rampaging elephant is eating the crops, while another is cutting them down to stumps.
Bad for crops. Farmers are not supposed to eat their own seed corn.
A few years ago I had a conversation with Zvi Mowshowitz that I think bears on the problem you're discussing. He mentioned that there were bets he'd take with money already committed, but that he wouldn't pay out a single extra dollar for, since that would make him a mark. I think the implied principle is that in some situations, you need both an inside-view EV calculation and an outside-view assessment of whether your EV estimate itself is downstream of an adversary's strategy.
Consider Alice evaluating an investment opportunity showing a 20% expected return. The crucial question isn't just "What's the probability of success?" but "Why am I seeing this opportunity at all?"
If Alice decides to invest even a small amount, this doesn't *cause* her to become a target. Rather, it provides evidence to herself that she's already in the reference class of "people whose decision processes make them attractive targets."
This creates a decision loop:
1. Alice calculates positive expected value using available information
2. Alice considers investing based on this calculation
3. Alice's willingness to invest becomes evidence that her information environment has been selected/manipulated
4. This evidence should update her EV calculation downward
5. The updated calculation often converges toward not investing
For a large class of seemingly positive-EV opportunities, this meta-analysis can appear to rationally lead to categorical avoidance.
The commercial success of Quakers with their fixed-price policy illustrates this principle beautifully. By refusing to haggle on principle, they allowed buyers to interpret stated prices nonadversarially, which allowed more people to transact in a non-paranoid way.
Investors hating securities with legal or regulatory risk reflects this same dynamic. The issue isn't just calculating the odds of regulatory action, but recognizing you're in an environment where your perception of those odds may be someone else’s optimization target. The categorical avoidance of such securities is meta-rational, not risk-averse.
This principle has dangerous corollaries, though. While contrarian investing can work for someone as thoughtful as Peter Thiel, "reversed stupidity" more often turns you into a differently controllable "rebel" who's still a target for extraction. It can lead to destructive behavior that profits no one—like raising tariffs just because economists advise against it, or (as I witnessed with a friend) peeing on a couch during a psychotic break merely because everyone wants you not to. The meta-level reasoning that begins as rational defense can deteriorate into reflexive opposition.
At a systemic level, this explains why this principle often leads to defensive sclerosis in institutions and markets, and depressive catatonia in individuals. People become increasingly unwilling to take any action that might signal they're in an exploitable reference class, until desperation finally drives them to fight for bad reasons. The result is long periods of calcification punctuated by irrational overreactions.
It bothered me that the sawtooth roofs of the two factories point in opposite directions. But here the shtick is to take metaphors seriously/too far, so what could it mean?
First possibility is that the decision-making process in the middle of the diagram is located at the South Pole. Both factory builders have correctly located the roof windows to point away so the factories get natural light but not direct sun. This is physically implausible but metaphorically promising if we think about what it might mean to pull a decision out of your South Pole.
Second possibility is that the builder of one of the factories did not understand what they were doing and imitated the form of the other without understanding its purpose or mechanism of action. This also seems metaphorically promising. Obvious to suspect the problem factory of this but as someone who has a career more in the solution factory I would be nervously checking whether we do in fact get direct sun in here.
the two factories are next to each other, facing in the same direction, but I made a topology-preserving projection so that they could both be seen in their relation to the decision centre.
The other explanation is that one of the factories was designed by Beavis, and the other by Butt-Head.
It's the same factory seen from two opposite directions
Downloading the PDF is well worth it. Beautiful.
Congratulations on the book launch, Dan, and thanks for letting us know about the problem factory paper. Nice title :)
If [faction] wants to throw the game, how would they use this observation, putting it into their "theory of mind" hopper, to skew the outcome? and, what could others do, to armor the system against such forays?
Dan, bearing in mind your "to finish first you first have to finish" I would be really curious to know what you think about the big black box that is America right now, and what to do?
To mix metaphors, what is the soil that the seeds went into, how were they watered and grown, and what is the resulting "black box" of this crop?
Right now one rampaging elephant is eating the crops, while another is cutting them down to stumps.
Bad for crops. Farmers are not supposed to eat their own seed corn.
https://thinkinthemorning.com/eating-the-seed-corn/
Meanwhile, spectators are cheering the elephants on.
They sincerely believe in magic.