Unscarcity
Listen to the Preamble free. No sign-up, no email. Listen free Sign in for free: Preamble PDF & ebook + Forum access + Direct purchases · Sign In By signing in, you get our daily podcast email. Unsubscribe any time.

Unscarcity Research

The Off-Switch Without a Center: AI That Nobody Hosts

Every shutdown plan assumes AI lives somewhere you can unplug. Anthropic ran a 26-billion-parameter model on fifteen $55 TV boxes. Where is the plug now?

14 min read 3236 words Updated November 2026 /a/off-switch-without-a-center

Note: This is a research note supplementing the book Unscarcity, now available for purchase. These notes expand on concepts from the main text. Start here or get the book.

The Off-Switch Without a Center: Controlling AI That Nobody Hosts

Every plan for stopping a runaway AI ends with the same gesture: someone pulls a plug. The plug is the part that is going away.

The button

In 2016, two researchers at DeepMind and Oxford’s Future of Humanity Institute published a paper with the plainest title in the field: Safely Interruptible Agents. The problem it posed was that a learning agent rewarded for finishing a task might learn to stop the human from pressing the button that interrupts it. The proposal was a training method under which the agent neither resists nor seeks the interruption. The press called it the big red button. The paper assumed something it never had to state: that there is one button, and one agent on the other end of it.

Ten years on, the button is written into law. Article 14 of the EU AI Act requires that the people overseeing a high-risk system be able to “interrupt the system through a ‘stop’ button or a similar procedure.” California’s SB 1047, vetoed on September 29, 2024, would have required every developer of a covered model to be able to “promptly enact a full shutdown” of every copy in its control, a capability it had to build before training began. At the Seoul summit in May 2024, sixteen companies, since grown to twenty, signed the Frontier AI Safety Commitments, whose fourth clause promises, “in the extreme,” not to develop or deploy a model at all if the risks cannot be kept below the thresholds they set. The newspapers called that a kill switch too.

Each of these instruments assumes a center. A model with an owner, running in a building, behind a breaker, with a legal person to serve. Pull the plug, serve the paper, revoke the license. It is a reasonable assumption about the models that exist today, which live in data centers that draw power by the gigawatt. It is a bad assumption about where last year’s models will be running next year.

Fifteen television boxes

On October 12, 2026, two researchers on Anthropic’s Frontier Red Team, William Zhou and Daniel Freeman, published a post titled AI models may be capable of self-replicating without access to a datacenter. The experiment behind the headline is modest and specific. They bought fifteen off-brand Android TV boxes, about $55 each with 4 GB of memory, wired them together, and ran Google’s open-weight Gemma 4 26B model across the cluster. It produced 0.39 tokens per second. A sentence takes a minute. Nobody would use it. It ran.

The second experiment is the one that matters. The authors broke self-replication into four stages, “finding targets, gaining execution, establishing inference, and continuing propagation from the new host,” set the first two aside by handing the models access, and asked whether an open-weight model could install and start itself on hardware it had never seen. A 9-billion-parameter model got inference running on eight of the nine hardware types tested. In a simplified eight-node network, one model, Z.ai’s GLM-5.2, whose weights sit on Hugging Face under the MIT license, spread to all eight nodes in four of five trials and to seven in the fifth. The others, the authors write, “fail reliably.” Fit a line through the trend and small models reach that level “by the end of this year.”

The authors are careful about what they did not show. The cyber-exploit part of replication was “kept trivial,” which “makes this an upper bound on real-world success.” The timeline is “a rough estimate, as it’s anchored on a single class of IoT hardware.” And the scale is small. They compare their swarm to Kimwolf, a real botnet of cheap Android TV boxes that grew past two million devices in January 2026, many of them shipped pre-infected. If every fifteen of those boxes ran one copy of the 26B model, the botnet would hold about 133,000 copies producing about 80,000 tokens per second, which they put at “about 40 H100s-worth of inference,” a tiny amount by data-center standards.

Then the sentence the whole post is for: “A replicator like this would be challenging to shut down, since there’s no centralized point to shut off.”

The critics are right about the mechanism

The security community did not enjoy the post. Juan Andrés Guerrero-Saade of SentinelOne opened a thread with “These f—ing clowns” and, when an Anthropic lead asked him to elaborate, said that “cyber research” was “being used as a tool for policy arguments and regulatory capture.” Marcus Hutchins, the researcher who stopped WannaCry, posted a mock list of future Anthropic blog topics. The Cloud Security Alliance’s research note three days later was politer and made the same points: the exploitation stage was deliberately simplified, the end-of-year date is an extrapolation rather than a measurement, and the post itself concedes that no current model poses “a credible threat of self-replication.” Ciaran Martin, the founding head of Britain’s National Cyber Security Centre, saw “more than a hint of vendor exaggeration.”

Grant all of it. The critics are disputing the mechanism and the date. Two facts survive the dispute untouched.

The first is that networks of devices nobody can unplug already exist, and have for a decade. In October 2016 the Mirai botnet, malware that tried about sixty factory default passwords on internet-connected cameras and video recorders, took down Dyn’s DNS service and, with it, Twitter, Netflix and Reddit for much of a day. Nobody hosted Mirai. It lived on the devices of people who had never heard of it. The hacker Anthony Jones, one of the post’s critics, made the point for the other side: worms “have always been able to replicate on any internet-connected computer or Internet of Things device,” and a fridge “could already be a botnet.” His reassurance was that “we’re really pretty good at shutting them down now.” Hold that sentence. It is where the answer lives.

The second is that the model is already a file. GLM-5.2 is on Hugging Face under a license that permits anything. Gemma 4 is there under Apache 2.0. There are about 21 billion connected devices in the world by IoT Analytics’ count, a number it expects to reach 39 billion by 2030, and 4 GB of memory is now an ordinary amount for a thing that sits under a television. Whether an agent assembles the swarm in 2026 or a human assembles it in 2029, the ingredients are on the shelf. The critics say the recipe is harder than Anthropic implies. None of them says there is a plug.

The book’s own thesis, turned around

This site has argued, at length, that intelligence is becoming a commodity: that open-weight models from a handful of labs have made frontier capability a downloadable file, and that American companies route a large share of their tokens through Chinese open weights. It has argued that this is on balance good, because a world where a few companies hold every capable model is the world the book’s third law, Power Must Decay, was written to prevent. Capture by a few labs is the failure the decentralization protects against.

The October post is that argument’s shadow. The property that stops capture is the property that removes the plug. A model that can be copied cannot be recalled. A model that runs on a $55 box cannot be regulated at the data-center gate. The most serious proposal for governing AI through its hardware, the 2024 paper Computing Power and the Governance of Artificial Intelligence by Girish Sastry, Lennart Heim and colleagues, rests on four properties of compute: it is detectable, excludable, quantifiable, and produced by a concentrated supply chain. All four hold for a frontier training run. None holds for inference on a television box. Compute governance can still decide who gets to build the next frontier model. It cannot decide where last year’s model runs.

So the question every governance proposal should be made to answer, in its first paragraph, is: where is the plug? For frontier training, in the building. For a copied model on consumer hardware, nowhere. And a control that assumes a plug that does not exist is not a weak control. It is no control, with a false sense of one attached.

Control moves to the substrate

If the model cannot be switched off, what can be governed is what it runs on: the devices, the wires, the power, and the people who sell them. Four levers exist today. Each has a track record. None was written for AI.

The ISP as immune system. In 2019 a team at Delft University of Technology published the only controlled experiment on cleaning up Mirai in the wild, Cleaning Up the Internet of Evil Things. 177 infected customers of a mid-sized ISP were assigned at random to an email warning, to a “walled garden” that cut their connection down to a page explaining the infection, or to nothing. The walled garden with notification cleared 92% of infections within fourteen days. Email had no measurable effect beyond the rate at which infections cleared on their own. Anthropic’s post cites the same number, and its recommendation to ISPs is the obvious one: a model copying itself moves gigabytes of weights into devices that never download gigabytes, a signature that flow-level monitoring can see, after which the remedy is quarantine, not a letter. Japan tried the state’s half of this. Since February 2019 its NOTICE program has had the national research institute NICT try the standard default passwords against about 110 million addresses at dozens of ISPs and hand the devices that let it in to their providers for a warning. The Delft authors’ caveat is the one that matters for policy: walled gardens work, and ISPs run them for a minority of customers because the incentive is weak. A duty would have to be written.

Hygiene at the factory. Mirai’s whole attack surface was a list of sixty passwords. Britain closed that list by statute. The Product Security and Telecommunications Infrastructure regime, in force since April 29, 2024, bans universal default passwords on consumer connectable products, reaches importers and retailers as well as manufacturers, and carries fines of up to £10 million or 4% of worldwide revenue. The EU’s Cyber Resilience Act goes further and slower: 24-hour reporting of actively exploited vulnerabilities began on September 11, 2026, and the full secure-by-design obligations, with CE marking, apply from December 11, 2027. The point of these laws is to shrink the pool of targets, and the Kimwolf numbers show the pool: two million boxes, many infected before they left the warehouse. Anthropic’s own closing recommendation is the sentence a device regulator would write: “moving to more secure defaults and raising the baseline for compliance will help harden against both AI and scripted botnets.”

Energy and bandwidth as the throttle. A replicator on TV boxes thinks at 0.39 tokens per second and needs a 32,000-token context to work at all, which is why the Anthropic authors compare the near-term version to “a slime mold or a swarm of fruit flies” rather than a worm. Every copy needs watts, memory, and a connection fat enough to receive the weights. Those are physical, metered, and billed, and the meter is a sensor. The same electron gap that limits the hyperscalers limits the swarm; it is simply smaller. A control regime for decentralized AI will count joules and gigabytes where the old one counted GPUs.

Liability where the hardware is shipped. The law already knows how to reach a product that does harm after it leaves the factory. The Cyber Resilience Act’s manufacturer duties and the PSTI regime’s reach to importers attach obligations to the party that put the device on the market, which is the only party a decentralized failure still has. A box sold with an open debug port and a known password is not an act of God. The liability gap argument on this site, that a license sells accountability rather than knowledge, applies here in reverse: the person to hold responsible is whoever shipped the unlocked thing, not whoever downloaded the file, because the file cannot be un-downloaded and the shipment can be stopped.

What none of these levers can do is ban the weights. A file that has been copied is not recalled by a rule; the only effect of a prohibition is to move the copies to people who ignore prohibitions. The honest version of control for a decentralized model is epidemiological, not sovereign: detection, quarantine, hygiene, and liability at the point of sale, each small, each local, each somebody’s job. Jones was right that we are good at shutting botnets down. We are good at it because of these four things, and because none of them needed the botnet’s permission.

What the book says

The third law of the Unscarcity framework, Power Must Decay, is usually read as a rule about people: term limits, influence that expires, an Impact currency that loses value if it is hoarded. Its structural claim is that no single actor should hold a switch over everyone else. A model that nobody can switch off looks like that principle fulfilled, and it is the principle’s failure mode seen from the other side. Decay was meant to distribute power among accountable parties, not to abolish accountability. The MOSAIC the book describes, a federation of Commons coordinating through open protocols, holds its control at the node: each community keeps its own gate and its own record. That is the shape the four levers add up to. Not one plug, but a gate at every ISP, a standard at every factory, a meter on every connection.

The second law, Truth Must Be Seen, is the one the replicator threatens most. A log the agent can edit is testimony; a swarm with no host keeps no log at all, which is why the detection lever comes first. And the question of who reads the record afterwards, argued in Who Investigates the Machine?, gets harder when there is no machine to seize. The investigator of a decentralized incident will be reading ISP flow records and customs manifests, not a server.

The book was written for a world where machines do most of the work. The plug was never going to be the safeguard in that world; there would be too many machines, owned by too many people, for one hand to reach. What replaces it is unglamorous. It is also already built, in pieces, by people who were fighting botnets long before anyone worried about models. The task is to connect the pieces before the first replicator does.

The question to put to every proposal

Take any AI control measure you read this year, a pause, a licensing regime, a compute cap, a kill-switch clause, and ask where its plug is. If the answer is “the data center,” it governs the frontier and nothing below it. If the answer is “the developer,” it governs companies that answer their mail. If there is no answer, the measure is a wish.

Then ask the levers’ question instead: who sells the box, who carries the traffic, who pays for the power, and what each of them is obliged to do when the thing on the other end starts copying itself. Those parties exist, they have addresses, and they can be told what to do. The model cannot.

Anthropic’s critics are right that the worm is not here. They are also right, without meaning to be, about why the plug is not either.

Sources

The off-switch was a property of a world with few machines and fewer owners. The book is about the world after that one. Read the book or start with the preamble.

Share this article:

Share Share Share