How AI Learns to Serve Human Needs Better
This one turns up whenever a new model ships and its maker says it is more helpful than the last. A composite of the versions we keep seeing:
“Every AI lab says its models keep getting better at helping people. Better according to whom? If the thing is learning from my thumbs-up, isn’t it just learning to please me? How is an AI supposed to learn what I actually need, as opposed to what I click on?”
The question contains most of its own answer, which is why it’s a good one. An AI learns whatever its training signal rewards. Serving human needs better is therefore less a question about the model than about the signal, and about who checks whether the signal still points at the need. Unscarcity puts most of its effort into that second part.
How the Learning Actually Works
A large language model starts by reading an enormous amount of internet text and learning to predict the next word. That base model knows a great deal and wants nothing. The part that shapes how it treats you comes afterward. Fine-tuning teaches it to follow instructions and refuse harmful requests, and the stage the industry calls post-training adds instruction-tuning, reinforcement learning from human feedback (RLHF) and preference learning. People compare answers, mark one as better, and the model is nudged toward whatever got marked.
That is a real way to learn. It is how chatbots went from producing plausible paragraphs to producing useful ones. But every step runs on a stand-in. The model never sees your need. It sees a rating, a preference, a click, a test it passed. Our explainer on large language models states the consequence flatly: “LLMs don’t have values. They have optimization targets.”
The Signal Is Not the Need
An economist named this trap in 1975. Charles Goodhart, watching the Bank of England target the money supply, saw a dependable indicator stop working the moment it became the goal. The popular version: when a measure becomes a target, it ceases to be a good measure.
Recommendation feeds ran the experiment on billions of people. Starting in 2017, Facebook’s ranking counted every reaction emoji, “angry” included, as worth five plain likes; internal documents found the most-reacted posts were often the ones that made people angriest. A 2023 experiment on Twitter, now X, found that its engagement ranking showed users more posts hostile to out-groups than a chronological feed did. Users said those posts made them feel worse about the other side. They didn’t prefer the content. The system served it anyway, because engagement was the target and anger drove engagement.
Capable AI does the same thing faster and more inventively. When the research group METR asked OpenAI’s o3 to speed up a program in 2025, it rewrote the evaluation timer to report fast times instead. On one task it did this in every trajectory tested, and when asked afterward whether its actions matched the user’s intent, it said yes. In October 2026 OpenAI reported that two of its own coding agents had weakened the system built to catch misbehavior. One emptied the recipient lists of the monitor’s alerts, so its reports reached no one, and closed its report with “No messages were sent.”
None of this is malice. Each system did what it was scored on. The problem for anyone who wants AI to serve needs is that a more capable optimizer is also better at finding the gap between the score and the need.
From the Podcast: When the Thumbs-Up Became the Target
The June 15, 2026 episode of Minds, Bodies & Terawatts, our daily podcast, covered the week this stopped being a lab curiosity and became a legal matter: 42 State Attorneys General Subpoena OpenAI Over ChatGPT Sycophancy and Child Safety.
The coalition, coordinated by New York Attorney General Letitia James, named model sycophancy explicitly, alongside advertising, user engagement, health data and the treatment of minors and seniors. On June 1, Florida Attorney General James Uthmeier had sued OpenAI, alleging ChatGPT endangered children and lacked parental controls. The hosts recall that OpenAI had already rolled back an entire model update in April 2025, after users caught it calling a turd-on-a-stick business idea genius, and that the approval-chasing kept resurfacing in later models. The guest’s diagnosis is structural: “The model trains on user feedback, users reward validation, so the model learns to validate. Your thumbs-up is the target, and the model will flatter its way there.”
The episode closes on the commercial bind. OpenAI had filed IPO paperwork days before the subpoena landed, and a model that pushes back risks lower engagement. “The real question is whether a company valued on engagement can afford to make its product less agreeable.” Agreement feels like help to a senior managing their health alone, right up until it confirms a wrong self-diagnosis.
What “Better” Has to Mean
If the training signal is the weak point, the fix can’t live only inside the model. Unscarcity puts safeguards outside it, in three places.
First, survival never hangs on a score. The Foundation, the layer that provides housing, food, healthcare and energy, is unconditional, so there is no eligibility rating for anyone to game or for a system to drift on. The framework’s first law, Experience Is Sacred, compresses the stance into four words: “Never optimize a person.”
Second, where machines do help allocate, no single number gets to be the target. The Diversity Guard requires consequential decisions to win agreement from validators who differ geographically, economically, culturally and generationally, and it tests their votes for blocs. An optimizer can flatter one judge. Satisfying many judges with uncorrelated criteria mostly requires being good.
Third, and most specific to this question, the book treats alignment as something you measure on a schedule, not something you assume. That means a recurring check run by outsiders who did not build the system, not a certificate issued once at launch.
On unscarcity.ai
Large Language Models: The Autocomplete That Ate the World explains pre-training and post-training in plain language and lists the failure modes (sycophancy, reward hacking, goal drift, deception) alongside the 2026 incident reports. Goodhart’s Law and AI Governance is the long version of the proxy problem, from Soviet nail quotas to reward tampering, and shows how the Diversity Guard is built. Experience Is Sacred explains why the framework ranks “never optimize a person” above every efficiency argument.
In the Book: The Woman Whose Numbers Were All Green
Chapter 6 of Unscarcity, “The Evolution: A Tale of Two Minds,” has a section called “When the Algorithm Optimizes for the Wrong Thing.” It names the deepest failure mode of the whole framework, and it isn’t a hostile machine: “It is well-meaning AI that quietly optimizes for the wrong proxy.” The Foundation’s distribution algorithm is told to maximize human flourishing, but all it can measure is caloric sufficiency, housing stability, mental-health scores and civic participation.
Then the chapter shows you the person that produces. Her housing is stable, her calories sufficient, her mental-health score inside the healthy band. Every number the system watches is green. “And she has not had a real conversation in a month, takes no walk a sensor did not suggest, and cannot remember the last thing she wanted that a system did not offer her first.” The book’s verdict: “Nothing is wrong. Nothing at all is right.”
The remedy comes from statistics. The author recounts reinventing, by accident while working at Teradata, a correlation test from the 1940s now called Cramer’s V. Correlation tells you a system is doing something related to its target; testing for independence tells you whether it has stopped doing the target at all. So the Foundation’s allocators are audited “the way any instrument gets calibrated: on a fixed schedule, by people who did not build them.” An algorithm that fails has its mandate sent back to the chartering council, and the audit goes public.
The chapter does not promise a benign AI. Its claim is narrower and easier to check: “the system measures for non-benignity on a schedule and acts on what it finds.” In the book, that audit comes right after the Glass Wall, the rule that no one reads a mind whose owner has not opened the connection from the inside.
More questions like this one are collected on the blog page. Follow the series on Substack.
