Chapter 10 of 13

Who Governs the Governor

About 23 minutes to read

The shortfall

In January 2008 auditors from the Post Office arrived at the branch in West Byfleet, a commuter village in Surrey, that Seema Misra had run with her husband for about two years. They counted the cash and the stock and compared them with the figures in the branch's computer system, called Horizon. According to the computer, about £74,600 was missing.

Misra had been reporting problems with the accounts for months. Like hundreds of other subpostmasters across Britain, she had found that the figures on her screen kept showing shortfalls she could not explain, and, like them, she had been told that she was the only one having such problems. Under her contract, she was liable for any shortfall. She had borrowed money and asked relatives for help to make some of the gaps good. Now the Post Office, which had the power to bring its own prosecutions, charged her with theft. At her trial in 2010 the jury heard evidence from an engineer of Fujitsu, the company that had built Horizon, that the system was reliable. Her defence could not see the system's internal records of errors. In November 2010, on her elder son's tenth birthday, she was sentenced to fifteen months in prison. She was eight weeks pregnant.

She was one of more than nine hundred subpostmasters prosecuted between 1999 and 2015 on the strength of figures produced by Horizon. Some were imprisoned; many were bankrupted; some took their own lives. It took a group action by more than five hundred of them, led by a former subpostmaster named Alan Bates, to establish in the High Court in 2019 that Horizon contained bugs, errors and defects that could and did cause shortfalls in branch accounts, and that the Post Office had known of problems with it. In April 2021 the Court of Appeal quashed the convictions of Misra and thirty-eight others, calling their prosecutions an affront to the conscience of the court. In 2024, after a television drama brought the scandal to millions of viewers, Parliament passed a law quashing the remaining convictions in England and Wales in a single act.

Horizon was not artificial intelligence. It was an accounting system — a ledger. That is why it belongs at the start of this chapter. The previous chapter ended with a warning: common ownership answers the question of who owns the machines, but not the question of who controls them, and whoever controls the machine that keeps the books is still Kushim, the temple accountant whose tablet opened the first book. Horizon shows what happens when an institution controls the ledger, the ledger is presumed to be right, and the people whose lives it records have no way to see inside it or to contest it. In English law, courts presumed that a computer had been working properly unless there was evidence to the contrary. The subpostmasters were not allowed to see the evidence that could have overturned that presumption. The machine was believed. The people were not.

The third element of the design under test is that AI should replace the shepherds, so that no managing class can become an owning class. This chapter asks whether that can work, and on what conditions.

The keepers of the books

The first book traced a long line of people whose power came from keeping the farm's records and running its machinery on someone else's behalf. The temple scribes of Uruk, who recorded barley and labour on clay, could do what almost no one else could: read and write the accounts. Egypt's scribes, China's scholar-officials, the clerks of medieval chanceries, the managers of the industrial corporation and the officials of the Soviet planning agencies did the same in their times. The first book also traced how often those who held such offices captured them. Priests came to treat temple estates as their own. The Mamluks of Egypt, enslaved soldiers bought to serve the sultans, became the sultans. Djilas's new class ran the property of the people as if it were theirs. The pattern reaches beyond the cases the first book told: the eunuchs of imperial China, brought into the palace precisely because they could found no dynasty of their own, repeatedly took control of the court.

Across those cases, capture followed a few recurring paths. The first was monopoly of knowledge: the keepers of the books could read what others could not, and could therefore tell the rulers and the ruled what the books said. The second was permanence: offices held for life, or passed down in families, turned servants into a caste. The third was control of information: those who stood between the rulers and the ruled decided what each side learned about the other. The fourth was self-audit: where the keepers of the books also checked the books, no one else could find their errors or their theft. And beneath them all lay the fifth: the presumption of authority, the habit of believing the record because it was the record, and the scribe because he was the scribe.

The design's hope is that a machine can do the keepers' work without becoming a keeper. It needs no salary, founds no dynasty, wants no villa, cannot be bribed and will not move into the farmhouse. But a machine also creates every one of the five paths in a new form. Only a few people understand how large AI systems work, which is a monopoly of knowledge. The systems are built and maintained by small teams, which can become permanent. They stand between institutions and the people they serve, deciding what each sees. They are often checked by the same organisations that run them. And they carry, as Horizon did, the presumption that a machine is more objective than a person. The question is not whether the machine can be a shepherd. It is whether the people who hold the machine can be kept from becoming the new shepherds — and whether the machine's own authority can be kept from becoming a new kind of priesthood.

A shepherd that never sleeps

Within the wall that this book assumes, there is a great deal that AI can do in the society the design imagines. It can keep the accounts of the common trusts in real time, and make them readable by anyone. It can match people to work and work to people, schedule production, route supplies and forecast demand. It can check claims for errors and look for fraud. It can audit itself and other systems at a speed no human auditor could match. These are exactly the functions of the managing class that the first book found turning, again and again, into a class of owners. If a machine can do them, the managing class shrinks, and with it the main route by which the farm has rebuilt itself after every revolution.

But the twenty-first century has already run several large experiments in automated administration, and their results are a warning. Between 2016 and 2019 the Australian government used an automated system, later known as "Robodebt", to identify people who had supposedly been overpaid welfare benefits. It compared their declared income with annual tax data averaged across fortnights, a method that could not show whether anyone had in fact been overpaid, and sent hundreds of thousands of debt notices, often demanding that recipients prove they did not owe the money. About 1.7 billion Australian dollars in debts were raised against some four hundred thousand people. In 2019 a court case brought by a recipient led the government to concede that the method was unlawful, and in 2020 it agreed to a settlement worth about 1.8 billion dollars. In July 2023 a royal commission called the scheme "crude and cruel", found that it had been neither fair nor legal, and recorded that officials had been told of its probable illegality years before it ended.

In the Netherlands, the tax authority spent the years between 2005 and 2019 pursuing tens of thousands of parents for supposed fraud in claims for childcare benefits. Many were ordered to repay tens of thousands of euros, for errors as small as a missing signature; some lost their homes; children were taken into care. The authority used a risk-scoring system in which, it later emerged, holding a second nationality had been treated as an indicator of risk. In January 2021, after a parliamentary inquiry found that the families had been subjected to "unprecedented injustice", the government of Mark Rutte resigned. By 2024 more than thirty-three thousand people had been officially recognised as victims.

None of these systems was the kind of AI the design imagines. They were cruder. But each had the features that make any machine dangerous as a shepherd: it was trusted more than the people it judged; its workings were hidden from those people; it placed the burden of proof on them; and the officials who might have stopped it had reasons — targets, budgets, careers, a minister's wishes — not to. And in each, as in the case of the ride-hailing drivers described in Chapter 4, there were human beings nominally in charge. The Amsterdam Court of Appeal's phrase for Uber's review of its own automated dismissals — "not much more than a purely symbolic act" — describes all of them. The danger is not that the machine makes decisions without humans. It is that humans approve the machine's decisions without making any.

Psychologists call one part of this "automation bias": the tendency of people to accept the output of an automated system and to stop looking for evidence against it. Raja Parasuraman and Dietrich Manzey, reviewing decades of research in 2010, found it among novices and experts alike, and found that it was not cured simply by telling people to be careful. It grows when people are busy, when they are judged by how many cases they process, and when overruling the machine carries more personal risk than agreeing with it. These are the conditions of most bureaucracies.

The design therefore cannot rely on the machine being good, or on its keepers being good. It must be built so that it does not matter much whether they are. The rest of this chapter builds that governance piece by piece, from the five principles that the plan of this book named — openness, audit, rotation, veto and human override — and tests each against the paths of capture above.

First principle: open

The first defence against a monopoly of knowledge is to end the monopoly. Horizon's errors were known to some of the people who maintained it and hidden from the people it accused; the legal advice that Robodebt was probably unlawful was not made public. In the design, every system that governs the commons would be open: its code, its model weights, the data it was trained on (with personal data protected), the rules it applies, and a record of every significant decision it makes and why. Anyone could inspect them. Anyone affected by a decision could see the reasoning applied to their case.

Openness has a strong record where it has been tried. Norway's oil fund, as Chapter 9 described, publishes every holding it has, and its transparency is one of the reasons it has remained trusted across governments. Open-source software, which anyone can read and modify, runs much of the internet, and its openness has allowed outsiders to find flaws that its makers had missed. The European Union's AI Act, adopted in 2024, takes a first step in law: it requires the providers of systems classed as "high-risk" — including those used to decide on access to public benefits, employment and credit — to document how they work, keep logs, and give the people who use them enough information to interpret their output.

But openness has a limit that the history of the scribes makes plain. A ledger written in a script that almost no one can read is open only in name. The workings of a large AI model are difficult to explain even for those who build it; publishing its weights tells most citizens nothing. Openness therefore creates a new class of interpreters — the technicians who can read the machine — and that class could become the new priesthood. The design needs openness, but openness alone is not enough. It needs people with the knowledge to read the machine who do not belong to the institution that runs it. That is the work of audit.

Second principle: audit

Every lasting republic in history has had some way of checking the keepers of the books. In classical Athens, every official had to submit to an audit, the euthyna, at the end of his year in office. Boards of auditors examined his accounts, and any citizen could bring a complaint against him. Imperial China kept a Censorate, a body of officials whose task was to watch, investigate and impeach other officials, and who could report directly to the emperor. Modern democracies have supreme audit institutions, ombudsmen and inspectors general, and public companies are required to have their accounts checked by independent auditors.

The record also shows how audits fail. The Chinese censors could be intimidated, bought or turned into instruments of faction; in the late Ming, eunuchs and their allies persecuted censors who opposed them. In modern business, the auditor is paid by the company it audits, and that has repeatedly ended badly. Arthur Andersen, one of the five largest accounting firms in the world, collapsed in 2002 after its role in auditing the energy company Enron, whose accounts had concealed billions of dollars of debt. In 2020 the German payments company Wirecard admitted that 1.9 billion euros that its accounts showed in its bank balances probably did not exist; its accounts had been signed off for years by EY, one of the largest auditors in the world. The Post Office's own investigators audited the subpostmasters; nobody independent audited Horizon until the subpostmasters forced it in court.

The lessons are consistent. Auditors must not be paid by, appointed by or dependent on those they audit. They must be able to see everything, including the internal records of errors. They must be rotated, so that they do not grow too close to those they check. There must be more than one of them, so that capturing one is not enough. And there must be a way for those who are harmed to trigger an audit themselves, as the Athenian citizen could. In the design, this means that the machines governing the commons would be checked by several independent bodies, funded by a fixed share of the commons' income that the bodies they audit cannot cut, staffed for limited terms, and open to complaints from any person. Their reports, like the machines, would be public. The audit of the machine could itself use machines — AI is good at finding anomalies in vast records — but the auditors' machines would have to be different from, and independent of, the ones they audit.

Third principle: rotation and the lot

Permanence is the second path of capture, and history's best answer to it is rotation. The Athenians filled most of their offices by lot from among citizens who volunteered, for terms of a year, usually without the right to hold the same office twice. They drew lots with a stone machine, the kleroterion, designed to make the draw impossible to rig. Their reasoning, as Aristotle reported it, was that election favoured the rich and the well known, while the lot gave every citizen an equal chance to rule and be ruled in turn. Imperial China, from early times and most systematically under the Ming and Qing, applied a "law of avoidance": officials were not posted to their home provinces and were moved every few years, so that they could not build local networks of kin and clients. The Mondragon cooperatives, as Chapter 9 described, elect their governing councils for limited terms.

Rotation has a cost, and the Athenians knew it. They did not fill every office by lot. The ten generals, who led the army and the fleet, were elected, and could be re-elected without limit; Pericles was elected general many times. Where expertise mattered most, the Athenians chose expertise — and that is where their most powerful leaders came from. Any design that rotates everyone loses knowledge; any design that rotates no one creates a caste. The question is which offices need which.

The modern revival of the lot suggests an answer. In Ireland, a Citizens' Assembly of ninety-nine people chosen at random to reflect the population met between 2016 and 2018, heard evidence from experts and from people with personal experience, and recommended changing the constitution's near-total ban on abortion. In May 2018, 66.4 per cent of voters approved the change in a referendum. In the German-speaking Community of Belgium, a small region of about eighty thousand people, the parliament voted in 2019 to create a permanent citizens' council chosen by lot, which sets the agenda for citizens' assemblies whose recommendations the parliament must answer. And, as Chapter 5 described, the Austrian heiress Marlene Engelhorn handed twenty-five million euros of her inheritance to a council of fifty Austrians chosen at random, the Guter Rat, which decided in 2024 how it should be given away. The evidence from such assemblies, gathered by researchers over two decades, is that ordinary people chosen at random and given time, information and good facilitation can deliberate seriously about difficult questions, and that the public tends to trust their conclusions more than those of politicians.

The design can use this distinction. The day-to-day running of the common machines, which needs expertise, would be in the hands of professionals: chosen openly, paid within the narrow ratios Chapter 11 will set, appointed for limited terms, and removable. But the rules the machines apply — what they optimise, whom they serve first, what counts as fraud or error, how far they may persuade — would be set or approved by councils chosen by lot from the whole population and from the people who work with the machines, rotated regularly, and advised by experts who do not get to decide. That splits the power of the scribe in two: the person who keeps the books would not be the person who decides what the books are for.

Fourth principle: veto and override

The Roman plebs won their most durable protection after they walked out of the city in the early fifth century BCE and refused to fight or work for the patricians until they were given officials of their own. The tribunes of the plebs, whom the first book met in Rome, were sacrosanct — anyone who harmed one could be killed without trial — and had the power to veto the acts of the other magistrates: to step between a citizen and an official and say "I forbid". The veto did not let the plebs govern. It let them stop the government from harming them. For centuries, it worked.

The design needs the same thing in a modern form: a way for any person, or any group, to stop the machine and the people who run it. That begins with the right to human review of any significant decision a machine makes about a person — whether to grant a benefit, assign or end work, or accuse of wrongdoing. European data protection law already gives people a right, under Article 22 of the General Data Protection Regulation, not to be subject to a decision based solely on automated processing that has significant effects on them, and the EU's AI Act requires high-risk systems to be designed so that humans can oversee them, explicitly warning against "automation bias". But the Amsterdam case showed how easily that right is reduced to a ritual. A human review is real only if the reviewer has the power to overturn the decision, the time and information to judge it, and no penalty for disagreeing with the machine; if the reviewer is independent of those who run the system; and if the burden of proof lies with the system, not the person it accuses. Horizon reversed every one of those conditions.

Individual review is not enough, because many harms are collective: a rule that disadvantages a whole group, a target that drives a whole office, a change that speeds up every worker on a line. Chapter 4 required that the people the machine manages have an organised, collective voice in how it is set. There is a working precedent. Since 1972, Germany's Works Constitution Act has given elected works councils, in every workplace with five or more employees that chooses to have one, a right of co-determination over the introduction and use of technical devices designed to monitor the behaviour or performance of employees. German courts have interpreted the right broadly, so that it covers most software that records data about workers, and lawyers now advise employers that introducing AI tools that can evaluate employees generally requires the works council's agreement. Without that agreement, the system cannot lawfully be used. It is, in effect, a tribune's veto over the machine, held by the people the machine would manage.

In the design, every workplace and every public service would have such a council, elected by those who work there and independent of the managers, with the right to refuse, suspend or require changes to the machines that direct their work. The people who use public services would have the same right through councils of their own. And because a veto can also be used to block anything at all, the design would need a way to resolve deadlock: a council's veto would send the question to a citizens' assembly or an independent tribunal, which would decide within a fixed time. That is the answer to the fear, raised in Chapter 9, that holding the managers of the commons to account would paralyse them. Every check needs a deadline.

The power to persuade

One function of the machine needs its own rules, because it is a power over belief itself. Chapter 7 described research in which an AI system, given a few facts about the person it was debating, was more persuasive than a human opponent in most of the debates where one side clearly won. The first book described the farm's long history of using stories to make itself feel just. A machine that can tailor its arguments to every person, in private conversation, at almost no cost, would be the most powerful storyteller the farm has ever had. If the common machines governed the commons and also talked to everyone in it, those who controlled them would control what everyone was told.

There are partial precedents for governing a public power of persuasion. Public broadcasters such as Britain's BBC were given charters, independent boards and duties of impartiality precisely so that a state-owned voice could not simply become the government's voice. Those protections have often held, and have also been broken: in Hungary, after 2010, a new media law and new appointments brought the public broadcaster under the control of the governing party, and international observers have since described it as a government mouthpiece. The EU's AI Act also prohibits AI systems that use manipulative or deceptive techniques to distort people's behaviour in ways that cause them significant harm.

The design would add four rules. First, the common machines would not try to persuade people on any political question, and would say so; where they explain a public decision, they would present the strongest arguments on each side. Second, any machine that tries to persuade — whether for a public campaign, a business or a cause — would have to say that it is a machine and who it serves, and keep records of what it said that independent auditors could examine. Third, the rules on persuasion would be set by citizens' assemblies, not by the machines' managers or the government of the day, and protected from change by a single majority, as Chapter 6 found every lasting lock must be. Fourth, during elections and referendums, the strictest version of these rules would apply. Under this book's assumptions, the new stories by which people live must come from people; the machine's job is to keep the conversation honest, not to win it.

The governor, assembled

Put together, the pieces form a governance that splits the old office of the shepherd into five parts, none of which can act alone.

The machine keeps the books, allocates, schedules and checks, openly, with every significant decision recorded and explained. The managers run the machine day to day: professionals, appointed for limited terms, paid within narrow ratios, and removable. The rule-setters decide what the machine is for: councils chosen by lot from the whole population and from those who work with the machines, rotated regularly, advised by experts who do not decide. The auditors check the machine and the managers: several independent bodies, funded by a fixed share of the commons' income, staffed for limited terms, and open to complaints from anyone. And the tribunes can stop all of them: every person through a real right of review with the burden of proof on the system, every workplace and service through an elected council with a veto that goes, within a fixed time, to an assembly or a tribunal.

This is, in essence, the separation of powers applied to the machine — and it borrows from Athens, Rome, imperial China, the German works council and the Irish assembly because each of them, at its best, solved one part of the problem. Its weakest point is the one the scribes reveal: every part of it depends on people outside the machine being able to understand the machine. Public technical education, the right of anyone to inspect, and the independence of the auditors all exist to keep that knowledge from becoming a monopoly. Whether they would succeed is the largest question this chapter leaves open.

The tests

This chapter tests the third element of the design — AI replacing the shepherds, so that no managing class can become an owning class — against the record of every earlier office that kept the farm's books.

Precedent. The history of the keepers of the books is mostly a history of capture: scribes, priests, eunuchs, Mamluks and Djilas's new class all turned an office of service into a position of power. But history also offers working defences against each path of capture: the Athenian audit and lot, the Roman tribune's veto, the Chinese law of avoidance, the modern independent auditor and the German works council. None was perfect. Each lasted for generations where it was backed by an organised group with a reason to defend it.

Evidence. The recent evidence on automated administration is sobering. Robodebt, the Dutch childcare benefits scandal and the Post Office's Horizon show what happens when a machine is presumed right, its workings are hidden and its human overseers have no reason to disagree with it; the Amsterdam ruling shows how human review can become symbolic. The research on automation bias explains why. On the other side, the evidence on citizens' assemblies shows that randomly chosen people can deliberate well, and the German experience shows that a workers' veto over workplace technology can operate for decades in a successful economy.

Capture. Every piece of the governance can itself be captured: openness by the interpreters who can read the machine, audit by the audited, rotation by the professionals who outlast the rotating councils, the veto by those who use it to block everything, and the rules on persuasion by those who decide what counts as political. The design's answer is redundancy — several auditors, councils and tribunes, so that capturing one is not enough — and deadlines, so that no check can become a permanent block. Whether that is enough cannot be known in advance.

Exit. The governance of the machine gives people a form of exit within the society: the right to refuse a machine's decision, to demand a person, and to stop a system that harms them. It does not yet answer whether a person can leave the governance of the common machines altogether, which Chapter 12 must address.

The ledger

What survived. The third element survives, in a narrower form. AI can take over much of the work of the shepherds — the books, the allocation, the scheduling, the checking — and in doing so remove the main route by which managing classes have become owning classes. The design keeps that.

What had to change. The machine cannot be the governor. It can only be the governor's instrument, and the governor must be divided. The design now requires that the shepherd's office be split into five parts — machine, managers, rule-setters, auditors and tribunes — none of which can act alone. No significant decision about a person may be made by the machine without a real human review, in which the reviewer can overturn the decision, has the time and information to do so, faces no penalty for disagreeing, and works for someone other than the system's operator. The burden of proof must lie with the machine and those who run it, never with the person it accuses: the presumption that the machine is right, which convicted Seema Misra, must be reversed. Those whom the machine manages must have an elected collective voice with a veto over it. The power of the machines to persuade must be governed by citizens, not managers. And every check must have a deadline, so that accountability does not become paralysis.

What remains unknown. Whether citizens can understand the machines well enough to govern them, or whether the interpreters who can read them will become the next priesthood. Whether the professionals who run the machines will outlast and outmanoeuvre the rotating councils, as permanent officials have outlasted elected ones in many democracies. Whether the wall holds for judgement — whether, as this book assumes, there remain decisions that only people can make well. And whether so many checks can move as fast as the machines they check.

Bridge: what the machine counts

There is one decision this chapter has not yet faced. The machine that keeps the books of the commons will also have to count what people contribute — because the design rewards contribution, and above the floor, people receive more for giving more. Whoever decides what counts will decide what people are paid for, what work is honoured and what is ignored. That is the oldest power of Kushim's ledger: not only to record the barley, but to decide what goes into the record. The next chapter asks what should count as contribution — production, care, learning, creation, the maintenance of the commons — how it can be measured and rewarded without the measurers becoming a new managing class, how creativity can be rewarded without creators becoming a new aristocracy, and how high above the floor the ceiling can rise.