Overcoming Bias

What are you the most wrong about? You know the least about stuff far away from you in distant galaxies, but as you have few opinions about that, and it hardly affects you, who cares?

But what are you the most wrong about where you do have opinions, and where they are consequential for you? Consider seven factors that say when you are BLINDED:

• [B]ound: When you are judged by your group on your confident and unthinking belief in and loyalty to particular claims, you won’t study them well.

• [L]ow-Impact: When you are wrong about factors relevant for collective choices, your vote barely moves them, and so you have little incentive to think about them to make them better.

• [I]ndefinite: When concepts come from a high dimensional space where it seems hard to pin them down, separate them, or to define or measure them.

• [N]on-Connected: When you see relevant concepts as coming from a whole separate realm that has no logical connections to all the usual realms where you know things.

• [D]evalued: When you declare yourself to be largely indifferent to the consequences for you, as something else matters much more to you.

• [E]vidence-Poor: When you actually have little relevant data to draw on, and the best data that you have supporting your opinion is the mere fact that some groups like yours have continued to exist and while holding this opinion.

• [D]ynamic: When the topic is about what changes to be making to your group’s collective choices, either recently or in the near future, the mere fact that your group exists no longer offers even weak evidence for those choices.

The max mistake topic area, with all of these factors, is: the adaptiveness of your morals.

Your group suspects that you are evil if you do not see their morals as obvious, and even suspects you if you had to think to come to agree with them. Morality is a collective choice, where you are punished for deviating, so to have an impact you’d have to change your group’s shared moral opinions. Moral concepts tend to be hard to pin down, and today most see moral claims as sitting in a disconnected realm where all our usual non-moral claims are not relevant.

On the topic of the cultural and DNA adaptiveness of your group’s morality (and norms and status markers), most people say they care much less about the adaptiveness of their morals than about the “moral truth” of their morals. Figuring out theoretically which morals are more adaptive is actually quite hard, and so our best evidence is empirical: which successful societies have had which morals. But the fact that your society seems inclined to change its morals lately in a particular direction is far weaker evidence for the adaptiveness of that direction.

The topic where you most need careful thought is also where your community most punishes such thought. This is our big blind spot on which our civ will likely fall.

Consider the social ranking of who is how much of an intellectual. Think of this ranking as made by a weighted average of the opinions of other intellectuals. If we look at how this weighting changes across intellectual levels, there will be a median level, where half of the weight comes from opinions above that level, and half below.

I asked ChatGPT (5.5) and Claude (4.7) to give percentile estimates for the median level who judges who are the very best intellectuals, for the West in various years. They gave median 99%,99.5% for year 1000, median 96%,97% for year 1750, median 93%,90% for 1900, and median 88%,80% for 2025.

We have thus seen an increasing populism in who among us judges who are our very best intellectuals. Which is plausibly a source of intellectual decay. Especially as it is often noted that we usually find it hard to distinguish between mental quality levels above our own.

Many (Poincaré 1908, Schumpeter 1911, Ogburn 1922) have said that, as there are so many good ideas out there, most innovation is just simple combos of prior good ideas. This seems true of my best idea.

April 25, 1996, thirty years ago today, I first posted my best idea: decision markets, i.e., speculative markets that advise specific decisions by estimating decision-conditional outcomes. A.k.a., “futarchy” as applied to governance. It’s not my deepest, grandest, beautiful, or hardest won insight, just the one with the biggest expected impact.

My idea was a simple combo of two other long-well-known ideas.

The first prior idea I built on is that speculative markets do quite well at aggregating info. This was explored in theory (Emory 1896, Gibson 1889, Bachelier 1900) and in data (Cowles 1933, Working 1934). Even so in 1996, US regulators in practice only allowed risk-hedging, not info aggregation, as an “economics rationale” to allow markets to exist. (The allowed “price discovery” rationale was tied to helping other markets hedge risks.)

In 1984 I left grad school in physics and philosophy of science at U Chicago to go to Silicon Valley to do AI research, and on the side work with Xanadu, trying to invent the World Wide Web. Around 1988 I first started to have doubts about the Xanadu vision of reforming public convo by making criticism easy to find, and wondered what else we could do instead. So I started to think and write about the big potential of making speculative markets to aggregate info on far more topics. Like most everyone who first enters this space, I was first thinking mainly in terms of markets on the usual topics we see in mass media, punditry, and public policy debates.

The second prior idea I built on is that info is mainly valuable by informing specific decisions. For many centuries we’ve seen calculations of the value of certain specific info for specific decisions. And then we developed more general theory (Ramsey 1928, Hosiansson 1931, Blackwell 1951, Savage 1953, Schlaifer 1959). At Caltech social science grad school 1993-1997, I learned decision theory and the standard value of info calculation. Then wondering where speculative markets could add the most info value, ~1996 I realized that this would likely come from markets estimating specific outcomes given specific decision choices.

As I was one of the first to write on the big potential of prediction markets, many who entered this space over the years approached me. At which point I usually pitched this decision market concept. Which usually pushed them away, as they were focused, as I was initially, on those mass media and punditry topics. But I have doggedly persisted.

Most all innovations combine simple elegant ideas with messy details that make those ideas work. Mine is no different. To find the right messy details, one needs concrete trials and experiments trying different detail versions. It has been hard to find orgs willing to do this, as org decision making is usually quite political. But in the last few years we’ve thankfully started to see some trials.

As an econ professor who specialized in governance, I can assure you that the world is greatly structured by the fact that we typically have pretty incompetent governance. Imagine a governance that, when assigned a goal, would reliably achieve that if it is in fact feasible. This would radically reshape our whole world. (Yes, even if we soon get powerful AIs.) As decisions markets plausibly enable such competent governance, this is why I estimate their expected impact to be so very great.

Human action plans are often organized around goal hierarchies, with lower-level subgoals helping to achieve higher-level goals. And many parameters correlate simply with this high-to-low goal axis. For example, lower-level goals and actions tend to take less space, time, and other resources. They are less likely to conflict with other goals, and more likely to be time-consistent. They are more easily evaluated for success, better described by simple abstractions, more reliably controlled, and more easily optimized by hill-climbing. They seem more observable, reversible, and substitutable, give faster feedback, and are more easily automated.

However, other related parameters depend on this key high-to-low goal axis in less simple ways; they instead peak at some mid-level, and fall away from that in both directions. For example, we have more conscious awareness of, give more conscious attention to, and make more deliberate choices re mid level goals. We can more clearly articulate them and their relations to other goals, and we can more easily teach others to manage them. People coordinate with each other more here, and our blame, credit, norms, and laws focus more here. There is more cultural variety of behaviors at these mid levels; other behaviors are more set by DNA.

A noteworthy exception is that such mid-peaking parameters often peak at much higher levels in large for-profit orgs, and in other large orgs, like militaries, with strong incentives tied to concrete goals. Such orgs often can and do articulate, measure, credit, and blame the behaviors of top people who mange high-level goals.

A simple interpretation of these patterns is that cultural evolution of coordinated behaviors faced a key tradeoff. Let me explain.

As thinking and talking takes time, there is a lowest level of goals and actions where we can discuss them as we choose and do them, so that such talk greatly influences those actions. While humans can and do watch and learn details of others’ behaviors that are at much lower-levels, we mostly do this non-verbally and unconsciously.

However, to enforce norms, including the norms that say that we should keep our promises, we humans need to be able say to others in sometimes-verifiable words what we and others have or have not been doing lately. So that we can complain about such actions, and recruit others to exert social pressures toward norm enforcement. To defend ourselves against such accusations, our conscious minds were created to manage key stories of what we’ve been doing lately and why.

So cultural evolution got into the habit of having us think and talk consciously about goals near this lowest-articulable level, and also to notice, copy, and teach chunks of behavior near these levels. And in addition, we mostly manage our norms, status markers, and key coordination mechanisms near such levels. As this cultural evolution process is pretty random and uncoordinated, efforts to abstract these norms and chunks most naturally expressed at these mid levels into higher level goals don’t usually achieve much clarity or coherence. Also, we seem reluctant to explicitly name cultural adaption itself as a big higher level goal.

So why didn’t we instead define and manage our social coordination using much higher levels goals? The simple correlations above say that such higher goals would tend to be less modular, less observable, and less easily described using abstractions. Making it harder for us to see and describe them, and to enforce norms about them.

However, with the invention of money and for-profit orgs, the world has now found new ways to use modular observable goals at quite high-levels. When we allow such orgs to manage key areas of life, they have shown remarkable abilities to effectively coordinate our behaviors. The problem is that, in many minds at least, their wider use would violate other key norms that we have inherited from cultural evolution.

Notice that cultural natural selection of individual behaviors seems insufficient to evolve better norms and status markers, as these are features of key game-theoretic equilibria, where individuals deviations are punished. We need instead to have collective deviations of entire cultures, i.e., units with much stronger internal than external conformity pressures.

Alas this process has been greatly hindered in the last few centuries by decreasing variety and selection pressures, and increasing rates of environmental change and internal cultural drift. Which is plausibly causing such norms to decay, plausibly leading to civ collapse and replacement in a century or two.

I just went to my mom’s funeral, and so was reminded about my family, and of the question of what exactly one wants to do with one’s life.

For money, my dad was a programmer, and my mom made presentation graphics for a finance firm. On the side they were missionaries, a pastor, and a writer. When she could retire from making money, my mom became a writer full-time, contributing to 30 of the 275 Chicken Soup books; ~20M people have probably read one of her essays there.

My two bothers were most recently a court bailiff and a pool cleaner for money, and on the side a musician and pastor. Their wives were a sales clerk and a legal secretary. My wife was a clinical social worker and her brother was a govt lawyer. ChatGPT (5.4) and Claude (4.7) estimate ~45-55, 50-60 as percentile ranks for this family overall in terms of job prestige.

I’m now a university professor, and my two sons are a programmer and an investment banker, so the three of us together get ~75,90 percentile estimates. Making my family solidly middle class, and me and my sons upper middle.

Workers often face a conflict between how their job has been define by the world and their training, and what their managers tell them to do on that job. Usually people succeed more when they accept boss framings, and higher class folks more tend to have this and other more successful habits.

Both LLMs say that this also happens more specifically in academia, where there’s a conflict between the job defined as intellectual process, i.e., helping the world better understand key abstract topics, and the job defined as what it takes to get prestige and resources. Lower class folks tend more to pursue that first definition. My class background is substantially lower than that of most academics, and I fit this pattern, as I see my job more in terms of intellectual progress, less in terms of resources and prestige.

At my mom’s funeral, I was reminded that such events involve much praising of the dead on various metrics. Which raises the question: what do you aspire to be praised on at your funeral, and in future historical mentions? It also a meta question: why don’t we write periodic essays on what we are trying to achieve in our lives, so that at our funerals folks can discuss how well we achieved our stated goals? Yes of course they could also discuss how well we achieved their other goals for us, but our own goals also seem quite relevant.

I would of course prefer that, at my death and after, and even well before, people praise me for all the usual virtues. But compared to others, I put a much bigger weight on intellectual progress. I want people to say, because it’s true, that I helped the world gain insight on important neglected potent topics. Important because that’s what matters, neglected because it is far easier to find big insights on those topics, and potent because the big win is when others build on my insights, and integrate them into larger shared systems, as part of a long process of civilization accumulating insight. And myself having insights isn’t that valuable compared to communicating them in ways so let others see and build on them.

At my funeral, please do ask yourselves how well I did at this.

A simple way to apply futarchy to for-profit firms is profit-futarchy: make markets that estimate total firm market value given key firm choices, like who is CEO, what are key acquisitions, or what are key firm policies. Then do what such markets advise. But a big problem with this approach is that top people, like the CEO, often do not see their personal success as maxed by firm success. For example, they tend to be wary of losing control over key firm choices, even if that would make such choices more profitable.

So CEOs block the application of futarchy to firms. You might think that investors could just force CEOs to use futarchy, if that would max investor gains. But investors also can’t seem to prevent the adoption of poison pills, which also cut investor gains. It seems we must accept that top managers have power sufficient to induce firm outcomes that don’t max profits. Investors do not in fact fully control firms.

Okay, then what if we flip this script, and set decision markets to the task of directly achieving the selfish managerial ends that likely drive managerial power politics? Create a metric of the total success of an individual manager over their future career, and then make advisory power-futarchy markets that estimate this personal success given key choices under that manager’s power. And to discourage sabotage, give everyone who that might be able to act to greatly hurt this success a positive stake in that success, a stake they aren’t allowed to trade to below zero.

Would this supercharge power politics, via better informing political strategies? Plausibly this would improve both offensive and defense political choices, and also make political info more symmetric. Managers could less often win via strategies that rely on rivals not noticing their plans until too late. So might power-futarchy actually cut harms from firm politics? Maybe, relative to the alternative of no markets at all, helping managers have successful careers also on average helps firms to max profits.

Of course such markets may advise top managers to not create power-futarchy markets to aid their subordinates several levels below them. Such markets might even say to instead give such subordinates futarchy markets tied to key firm or division outcomes. If so, that might usefully limit the scope of power-futarchy. Yes, this might over time undermine support for power-futarchy, but maybe not before current managers achieved great success from it.

Some kinds of power politics strategies may be hindered by open markets estimating their power effectiveness. But we needn’t have such markets regarding all possible managerial choices. Though, yes, the choice to not create such a market on some key choice might be taken as a bad sign about the politics behind that choice. No doubt there would be many new tricks to be found when playing power-futarchy.

Tribes contain factions. Tribe members mostly interact with and emulate other members of their same tribe, while faction members do these things more often with members of other factions. Tribes tend to have distinct moral norms and status markers, while factions tend to share the norms and status markers of their tribe. Factions often differ on status, income, professions, and on symbolic markers like food, clothes, languages, holidays, and artistic styles. Factions also often disagree on directions to change shared tribe policies and norms. The distinction between tribes and factions is a matter of degree.

Our dominant world culture hates tribes, but loves factions, especially factions who we see as “down”. We hate groups who disagree with world elite consensus on school, medicine, democracy, gender equality, sexual freedom, legal due process, rules of just war, and norms of good parenting. And we hate tribe supporters for their self-favoritism and habitual hostility toward outsiders. About these things we see our dominant world tribe as just right, and the others evil.

But we love factions within this main tribe who embrace distinct symbols, and who fight for tribe norm reforms. We call this love “tolerance”. At least we love factions who we can plausibly see as “down” relative to “up” rivals. (We presume “up” illicitly hurts “down”.) We hate “up” faction members who promote their factions, and accuse them of actually representing hated tribes. We moderns tend to channel our instinctive human tendencies to be tribal into our factional conflicts. Not noticing how we need tribes far more than factions.

The big problem is that in history our moral norms and status markers came mostly from cultural group selection acting on tribes, not factions. By crushing all but one dominant tribe, we now mostly block such evolution from preventing the decay of shared norms, or their adaption to changing context. We now see this most clearly in the decay of norms supporting fertility, but such decay is plausibly also happening across all our key norms. Selection acting instead on factions can’t do this remotely as well. If such decay continues long, our civilization will fall, to be replaced by others.

Unfortunately, we find it hard to see this problem, as our moral norms and status markers seem to us as just obviously true, and thus good bases for any analysis. In contrast, we can see and appreciate fights among factions, as we can frame these in terms of our “obvious” shared norms. But that doesn’t help much to ensure that the winners of faction fights are more adaptive.

Instead of trying to repress competing tribes, as we usually do, we might try to instead promote them. But even that seems quite insufficient, as the main underlying reason that the world has over centuries been merging toward one big tribe is the increasing ease of distant trade, travel, and talk. Such merging has achieved great scale economies of production and innovation, and a great reduction in conflict harms, such as via war, due to increasingly shared norms. Most people really like having a world community with shared norms..

There are a few today, like the Amish and Haredim, who care enough to treat themselves as distinct tribes, and are willing to forgo many gains of world cultural integration to achieve this. Such folks insulate themselves culturally from the large world, and so are the folks today whose descendants are mostly likely to replace our dominant world culture. But few groups today are this devoted to becoming tribes. Most of the folks today interested in cultural variety, like “network state” folks, are not remotely this devoted, and so have little chance of creating new tribes.

I can see only three ways for our main world civ, which I treasure in many ways, to avoid being replaced like this. The first solution is to somehow greatly raise the status of tribes, relative to factions. Convince the world to fragment into far more tribes, not just factions. Tolerate and even encourage groups having quite deviant views on democracy, gender equality, etc. to favor themselves and isolate from outsiders.

The second solution is to leave the world mostly integrated into one big tribe, but to find new ways to control and govern how key moral norms and status markers are changed to become more adaptive. Such as via competent governments held strongly accountable to increase adaption futures estimates, or via using a competent futarchy to pursue sacred adaption-achieving goals like when a million people live in space.

The third solution is to vastly increase the role of for-profit orgs in setting our moral norms and status markers. The evolution of firm cultures has long been quite healthy, as firms form quite distinct groups facing strong capitalists selection pressures. And for-profit orgs competing to give customers key numbers and observable outcomes have quite consistently improved on such outcomes. Each area they came to control, such as buying and running governments, or paying parents to make kids they could in effect sell, would likely become more adaptive.

As you can plainly see, these are all big long-shots. Our situation is quite desperate. And not likely to get better until a lot more people start to think about it.