Democratic Leverage Collapse Killchain | Frederico Luz

Course note: This essay was written as an exercise for the Bluedot AGI Strategy course.

Not too long from now, AGI and robots will make human labor obsolete.

Let's start by laying out some assumptions:

Human-level general intelligence will come soon, and human-equivalent robotics soon after.
Human-level general intelligence is principal-aligned, but not yet value-aligned. They’re not scheming to break free of human control but they’re also willing to do harm on behalf of those that control them. Consider how easy it is to remove safeguards from current frontier models, as well as add secret instructions and loyalties, as long as you control the weights.
The human race won't immediately become extinct due to strong AI, whether by an engineered bioweapon or AI-powered thermonuclear warfare.

Phase 1: CAPABILITY REQUIREMENTS

What capabilities would AI need to have to gradually disempower us?

Required capability: AI would require the ability to completely replace humans at knowledge work, and robotics would need to be advanced enough to replace humans at physical work. Both would need to be cheap enough to be more valuable than human employees.

Current capability gap: We are starting to see very early indications of knowledge work replacement. Some people, like pwc, are beginning to talk about an entry-level hiring freeze. Robotics is still very very early but progressing fast.

Warning signs we'd see first: Hiring freeze through progressively more senior positions, unemployment rising. We would also see the AI economy (goods needed by AI systems such as energy, compute, data, land) gain increasingly more relevance, eventually more than the human economy.

Phase 2: DEVELOPMENT

How are these capabilities developed? What are the incentives driving development?

Actors: One of the biggest problems with this threat is that it doesn’t depend on any one actor being “evil”. It depends on various actors doing what makes economic sense. The AI labs are incentivized to create the best, most labor-performing models (ditto for robotics companies and robots). Governments are incentivized to use AGI in their operations as much as possible to compete with unfriendly states. Companies are incentivized to replace their workers with cheaper and better AGI, using the savings to invest in more AGI, as well as pivot towards selling AI-focused goods and services instead of servicing human needs.

Incentives For both the labs and the companies, the incentive is profit and avoiding being outcompeted. For governments, it’s national security and avoiding being outcompeted.

Preparation steps: I’m not sure of how this would look like, I’m unsure if LLMs can get all the way there, but even if they can’t, they seem to be very strong at accelerating AI research itself, helping the labs find some other architecture that will work for full AGI.

Phase 3: INITIAL DEPLOYMENT

How are these models initially deployed? Who makes critical decisions on deployment?

Entry wedge: It would begin with entry level hiring freeze. Juniors are net-negative hires you tolerate as they waste the time of senior colleagues so that eventually they become useful seniors. Once you don't expect to need future seniors, you stop hiring juniors and replace them with AI instead.

Key decision makers: Managers and executives will have no choice but to replace their workforce, as it will be the only way to stay competitive.

Detection likelihood: Yes, but it will be assumed to be temporary. Not a lot of people would be unemployed in the initial stages, and existing social safety nets would probably be enough to support the first few waves of unemployment. There might be some upskilling programs or similar, but eventually it will be understood to be hopeless.

Phase 4: ESCALATION CASCADE

How do these initial instances of disempowerment compound? How does it spread or intensify?

Propagation mechanism: As a few companies in a given market replace their workforce and thus become more productive and cost-optimal, their competitors will be forced to do the same to stay in the game. Eventually this will happen to the company executives themselves, leaving only ownership itself to humans.

Failed safeguards:

Human in the loop: In addition to being a safeguard against misaligned AI, HITL is also seen as a way to keep humans economically valuable. It might actually be a good way to keep humans employed in the beginning, as general purpose models will not be immediately ready for every single task, and will benefit from human supervision as they learn to do their new job. But it eventually fails against that for the same reason that it fails against the other failure mode: HITL is slow, expensive and subject to more mistakes than a purely AGI operation would be. Any company relying on HITL will get outcompeted.

Democratic redistribution: Democracies are subject to their voters, which would demand a share of the wealth produced. As AGI would make the economy enormous, even a modest tax would fund a decent UBI. The mid century post-war social democratic states proved that companies can be forced to share. But the state needs the power and the will to tax these companies, and full labor replacement would end both.

Companies could flee to whatever state taxes the least, and heavily lobby states they cannot flee. AI will probably be heavily integrated in the government, ( after all, states that do not integrate AI fully will be deeply out-competed by competitor states, just like companies) and thus be very vulnerable to the wishes of AI companies that could threaten to refuse service if a tax was passed.

Previously, these threats were real, but states depended more on the people than on the companies. This will no longer be so, as the state won’t depend on the people for tax, labor, military manpower, etc. Even if there is a complete takeover of AI capability by the state, for example, nationalizing compute, that introduces another big danger, which is authoritarian concentration of power, simply replacing the ultra-elite from private companies to “public” rulers.

Revolution: Historically, this has been the final pressure valve for when an economy has proved unsustainable and the governing state has failed to reform it. But what all revolutions have in common is the state’s own functionaries, whether civil or military, eventually siding with the revolutionaries against the outgoing regime. In a situation where the state’s workforce has too been replaced with AGI, this would not happen, at least if AGI is aligned to a principal and not universal human values.

Point of no return: I am not sure of where exactly the PNR would be, but I have 2 possible hypotheses, both involving the state machinery and military itself being automated. The first is when the military is automatized, making revolution, the final safeguard, impossible. The second, earlier, PNR is when the state machinery and the public service is automated, i.e., when AGI is completely critical to the operation of the state, and thus the companies controlling the AGI have more leverage over the state than the citizenry itself. This would kill the democratic reform safeguard, and violent revolution might be impossible anyway, even without full automation of the military and the police, given technological advances in those areas. States with bad intentions and the companies that control them could even implement UBI but only until PNR is reached, then pulling it out once all the people’s leverage is gone.

Phase 5: SOCIETAL OUTCOME

What (if anything) fails at the human/institutional level?

Trust collapse: It may be possible that public institutions and especially the government lose trust at various points of this kill chain. It is also possible that bad actors intent on seizing control make sure that everything looks good to the citizenry until the PNR has been crossed.

Coordination failure: Authorities won’t be able to respond because at first, it will be too early, as automation might still not be a huge problem that they are willing to spend political capital on, and because it will be too late, and AGI will already be too deeply entrenched in the economy and the government, giving all the leverage to the actors with the power to control the use of the technology.

Public reaction: We can already see early denial, as many people believe that AI is simply a bubble that will pop like the dotcom case, or that there will always be new jobs for humans to perform. I predict that eventually, as the logical conclusions of AGI become more and more obvious, there will be attempts at organizing citizens to fight for redistribution. If these efforts fail, it’s possible for widespread chaos will take hold of society, as individuals and small groups try desperately to cling onto some sort of subsistence.

Phase 6: CATASTROPHIC RESULT

Make it visceral - what does a world where humans are gradually disempowered look like? What do people see, feel, and lose.

First few months: It is hard to understand exactly where this begins, as it will be a slide from early automation of a few specific knowledge jobs (now) to full automation of all labor (5 years? 10? 20?). The first waves of the unemployable would not be seen as catastrophic, as social safety nets would be able to pick up these people, and re-skilling programs may work temporarily, especially in training knowledge workers to perform physical roles that robots cannot do yet.

First years: The most drastic consequences do not have to wait for full automation, as even 20% or 30% of the population being unemployable would spark the biggest economic crisis in human history. Social safety nets have crashed, malnutrition becomes common, suicides spike. This is when I believe popular movements would seriously pick up steam. I am not sure what the president or any politician could say in this scenario, as it would depend on how dependent on AGI the government is. In the market, you would see a shift from goods and services for humans to more and more focus on AI goods, such as energy, compute and data, leading the cost of human goods, such as food, to spike.

Irreversible damage: Eventually, there is no income. A handful of ultra-wealthy owners of AI companies have extracted all the value of the market, as everything runs around AI. The permanent underclass is dying, whether due to starvation or repression by the state’s robot armies. Total human population drops to 1% of what it is today.

Phase 7: IDENTIFY CHOKE POINTS

When and how should we have intervened? What’s the detection window?

We would have needed to intervene now, or in the very near term future. We would need to make sure we create leverage for common people in the form of control over AI. Physical leverage, control over compute, not just taxes and laws about redistribution that can be replaced at any time. The only way for common people not to be completely written out of the equation is by making sure that common people hold leverage over the government and the AI companies.

It will begin with companies realizing that, to stay competitive, they will have to replace their workforce with AGI. The AGI will work better and faster than humans, and the money saved from wages will allow them to buy more AGI, whether through tokens or compute.

What happens then to the previously extremely successful companies that experienced a surge of profit by replacing their workers? They are dependent on AI, but they do not provide it, they make money with it, not off it. They still need to make money off the humans. The cliché used to illustrate this is Henry Ford paying his workers enough to buy his cars.

On the other hand, if there is great demand for a product, other producers will try to also sell it, lowering the price of a commodity. What would be the hottest goods in the robot economy? Energy, compute, data, land. Water?

The market pivots, and you’ve already seen some very early examples of this: Allbirds is no longer a shoe company. They’re in the compute leasing game now, and the market rewarded them for it, to the tune of 600%.

As humans run out of money and become economically irrelevant, markets will adapt to instead provide goods and services required by AI-powered companies and their handful of ultra-rich owners, making the cost of human goods spike

A lot of people think that, as in previous automation waves, new jobs will be created to replace the old ones, as humans might retain a comparative advantage in something. I don’t think this is a valid argument, as even though humans might be fundamentally superior to LLMs in some aspects, it doesn’t mean that they will be superior to other forms of AI, especially a full AGI or ASI. Even if humans do keep working, nothing guarantees that the wages paid will be above the cost of staying alive.

But previously, labor was able to organize against the employer’s interests and take its share. I don’t think this will happen, as this organization came from withholding labor, which was still necessary, and will no longer be.

The main safeguard would be social democratic state redistribution. Democracies are subject to their voters, which would demand a share of the wealth produced. As AGI would make the economy enormous, even a modest tax would fund a decent UBI. The mid century post-war social democratic states proved that companies can be forced to share. But the state needs the power and the will to tax these companies, and full labor replacement would end both. Companies could flee to whatever state taxes the least, and heavily lobby states they cannot flee. AI will probably be heavily integrated in the government, ( after all, states that do not integrate AI fully will be deeply out-competed by competitor states, just like companies) and thus be very vulnerable to the wishes of AI companies, that could threaten to refuse service if a tax was passed. Previously, these threats were real, but states depended more on the people than on the companies. This will no longer be so, as the state won’t depend on the people for tax, labor, military manpower, etc.

Even if there is a complete takeover of AI capability by the state, for example, nationalizing compute, that introduces another big danger, which is authoritarian concentration of power, simply replacing the ultra-elite from private companies to public rulers.

The final safeguard against this would be popular revolution, which is what historically happens when the state is not compliant with the wishes of the citizens. But every revolution in history succeeded because the regime’s army stopped fighting against the revolution and turned their guns against the state itself. But a robot army would never do this, as long as it is aligned to the principal.

“Imagine that in 2030 we are suffering from multiple AGI-powered attacks. How should we defend ourselves?”

I focus mostly on the third layer of defense, withstanding dangerous AI actions. This is because I believe layer 1 is not sufficient: the gains from advanced AI are too significant for society to simply ignore. I don’t think we should avoid developing powerful AI, and I think that competitive incentives will lead to it being developed anyway, and in that case by someone breaching the social contract.

Layer 2 is more realistic, but we have seen that even the most advanced guardrails so far are vulnerable to jailbreaks, and open models do not have guardrails. Therefore, we must have a strong layer 3, and be ready to withstand the danger.

If in 2030 we are suffering from multiple attacks, then they would most likely be foreign. As such, we would need a strong, robust state that can use AGI or ASI to defend its citizens from harm.

This means giving the state great power, which is dangerous. One or several of these attacks could in reality come not from China but from the government or other domestic bad actors interested in using a false flag attack in order to accrue power, ostensibly for defense, but which is never relinquished after the attack is over. This power concentration is its own failure mode.

Therefore, in order to at the same time have a powerful state ready to defend against threats but enough democratic leverage to make sure the power is not permanent, a democratic and decentralized distribution of compute would enable citizens to confer power to the state only when required.

This would let the state enact a series of harm reducing actions, such as contact tracing for pandemics, accelerated vaccine development, accelerated cyber defense or counter cyberattacks. One of the ways that we could ensure that this capability is not used to accrue power is by having transparent observability of exactly what the government and other responsible entities are doing. One could say that having public access to the prompts and agent transcripts used by these entities would be good for transparency. This is true, but would let hostile actors, such as foreign enemies, have full transparency of how we are defending ourselves, blowing a hole through our defenses. Therefore a more reasonable compromise would be allowing this transparency to trusted entities and persons, such as mayors and other elected officials.

This transparency would enable the government to have exclusive access to powerful cybersecurity or biology specialized models that would be dangerous to publicly release. Of course, it would also be dangerous if a few unaccountable entities are in exclusive control of these models (as is the current case) and I believe transparency could enable safe exclusivity, where the capabilities are used only for defense.

There is a compromise: it could be easier for an adversary to attack decentralized compute rather than 1 centralized base, and hack into our compute network that way. The opposite is also possible: that 1 large centralized base is more vulnerable, since it only needs to be hacked once to take all our capacity offline, whereas decentralized compute is more resistant to attacks (think how in WW2 military production was decentralized to resist enemy bombing)