If Zuckerberg's right about AI and power, why doesn't it feel right?
I read Mark Zuckerberg’s new AI manifesto expecting to disagree with almost all of it.
“The Future is for Everyone” went up on Monday. Six and a half thousand words, signed “– Mark,” laying out what he calls a philosophy for how humanity should build and distribute superintelligence. He posts one of these every few years. This one is explicitly framed as a response to the AI backlash Meta has been catching, including backlash aimed at Meta’s own practices.
I made it about a third of the way through, and I was nodding along in agreement.
The argument I’ve already made
Zuckerberg’s core claim is that the real danger of superintelligence isn’t the technology. It’s concentration. A handful of labs, or governments, or companies holding all the capability while everyone else holds none. His fix is to distribute the capability as widely as possible instead, so individuals and institutions hold enough power to check each other the way a functioning economy or democracy does.
If one person had a superintelligent lawyer, he argues, they’d win regardless of merit. If everyone did, the playing field would level out. Same logic for cybersecurity: one person with superintelligent hacking capability is a threat, but a world where everyone has superintelligent defense gets safer, not scarier. He calls this the balance of power theory, built on the idea that no centralized system can align with everyone’s competing values at once, so there’s no such thing as a singular benevolent superintelligence.
I’ve made a version of this argument before. Just aimed at a different target. When I wrote about Palantir’s manifesto, the thing I found most dangerous wasn’t the technology Palantir builds. It was the argument that questioning who controls it is naive. When I wrote about whether AI can solve our biggest problems, my conclusion wasn’t that we lack intelligence. It’s that the people who hold power over housing costs, insulin pricing, and TB funding aren’t confused about the outcomes their systems produce. They like the outcomes. Concentrated power protects itself.
So when a tech CEO opens an essay by saying the actual danger is concentration of power, not the tool itself, some part of my brain sat up. That’s my argument. He’s making my argument.
I don’t think that’s an accident, and I don’t think it means what it sounds like it means.
Everyone, except
The distribution mechanism isn’t actually flat. Meta will offer free versions of its models to billions of people. For anyone who wants more than that, there’s a dynamic auction for compute, designed to guarantee the cheapest available price for whatever capacity someone can afford to bid on.
That’s not the lawyer thought experiment. That’s the world we already live in, with a new interface. Everyone gets a lawyer. Some people get a better one, priced by what they can pay. The thought experiment only works if “everyone” means everyone, equally. What’s actually being proposed is a free tier and an auction - a fancy way of describing the exact stratification that already decides who gets good healthcare, good legal representation, and good schools.
I don’t think this makes Zuckerberg a liar so much as it makes the metaphor do more work than it can hold. Balance of power sounds like an even playing field. What’s actually being proposed is a company positioning itself as the trusted distributor of an uneven one.
The alignment reframe doesn’t survive a background check
The essay spends real time on alignment, and it does something clever with the term. Most labs, Zuckerberg argues, treat alignment as enforcing a centralized set of values on users. He wants Meta’s agents aligned to each person’s own goals instead. Not the company’s.
On its own, that’s a reasonable-sounding distinction. It falls apart the moment you ask who built the agent, who set its defaults, who profits from what it nudges you toward, and who has a documented history of choosing growth over disclosure when the two came into conflict.
Meta has that history. In 2019, Zuckerberg stood on a conference stage and said “the future is private,” days after the company set aside billions to cover an expected FTC fine over the opposite behavior. That fine landed at $5 billion, the largest privacy penalty the FTC had ever issued, for violating a 2012 order after Cambridge Analytica harvested data from 87 million users without consent. Meta separately paid $725 million to settle a related class action. Executives were still settling shareholder litigation over the same underlying failures as recently as last November - fourteen years after the behavior in question.
Then there’s the part that’s harder to sit with. Internal Facebook research, leaked by whistleblower Frances Haugen and presented to Congress in 2021, found that 13.5 percent of teen girls said Instagram worsened their suicidal thoughts, and 17 percent said it worsened their eating disorders. The company had that data. It didn’t lead with disclosure. It led with a defense of the researcher’s credibility instead.
None of that means the new privacy architecture is fake. Maybe the fully private mode he’s describing this time is real and durable. But asking people to trust a redefined idea of alignment is a big ask from a company whose specific, documented pattern is choosing engagement and growth over disclosure when the two conflict. The redefinition doesn’t erase the incentive structure underneath it. It just gives the incentive structure a nicer name.
He says the quiet part himself
The essay also proposes independent governance. Zuckerberg writes that he doesn’t think it’s in his own, Meta’s, or the world’s best interest for him or anyone else to be the sole decision-maker on how superintelligence gets deployed, so Meta is giving its board authority to approve safety criteria and review whether releases meet it.
Then, a sentence later, he writes that “Meta is a founder-controlled company.”
He says this like a footnote. It’s the whole ballgame. A board’s independence is only as real as its ability to say no to the person who can out-vote it regardless, and at Meta that person is designed in advance to win. Zuckerberg owns nearly all of Meta’s Class B shares, which carry ten votes apiece. That gives him something like 61 percent of total voting power while holding roughly 14 percent of the actual economic stake. No shareholder vote, board decision, or safety review can override him if he doesn’t want it to. An independent board operating inside that structure isn’t a check on power. It’s power that has agreed to consult itself before acting. That’s not a balance. It’s the appearance of one, which is a different and more useful thing if what you need is public trust rather than actual constraint.
The jobs data hasn’t gotten the memo
Zuckerberg argues that individual capability will grow faster than automation displaces it, leading to a smoother transition and possibly even more jobs overall. He points to history. Farmers became factory workers became knowledge workers, and humanity came out the other side with more prosperity each time, not less.
I’ve written before about what’s actually happening to entry-level work right now, so I won’t re-run the whole argument here. Short version, updated with the newest numbers: Stanford’s Digital Economy Lab found that since ChatGPT launched in late 2022, overall employment has grown, but employment for 22 to 25 year olds in the most AI-exposed jobs - software engineering, customer service, marketing - has fallen 16 percent relative to older workers in those same roles, even after controlling for firm-level effects. Workers in more hands-on professions - health aides, maintenance, driving - have held steady or grown.
That’s not individual capability outpacing automation. That’s automation hitting hardest exactly where a person would need to be standing to build the capability Zuckerberg is promising will save them. You can’t out-skill your way past a closed door. The essay’s optimism about jobs depends on people gaining ground before the ladder gets pulled up. Right now, for the youngest workers in the most exposed fields, the ladder is getting pulled up fast.
The part where the argument breaks itself
Near the end, Zuckerberg addresses the risk that a self-improving AI system could out-compute every other actor combined, becoming exactly the singular, unchecked superintelligence the whole essay argues against. He doesn’t dispute that this could happen. He says a system doing this could plausibly command more effective intelligence than everyone else on Earth combined.
His proposed safeguard is that labs should watch the system’s objectives carefully and coordinate if something looks wrong.
That’s not balance of power. That’s trust, applied at the exact moment the essay’s own logic says trust stops being sufficient. If the whole philosophy rests on no single actor holding unchecked power, and the essay’s own worst-case scenario describes exactly that happening, and the fix is “we’ll be watching,” the theory has quietly exempted the one scenario it was built to prevent.
Why I almost missed it
The reason I nodded through the first third of this essay is that it’s built out of language I recognize, because it’s language I use. Concentration of power is the danger. Distribution is the safeguard. No single actor, however well-intentioned, should hold unchecked control over something this consequential.
I believe all of that. I still believe it after finishing the essay.
What I don’t believe is that the company positioning itself as the safest possible distributor of superintelligence gets to skip having a track record. Meta has one. It includes a $5 billion privacy penalty, an internal admission that its products worsen suicidal ideation in teen girls, and a founder who controls the votes on the board he’s calling independent.
The ideology is the packaging. I wrote that about Palantir’s manifesto in April. It works just as well as a description of this one. The fact that this one’s ideology happens to rhyme with mine isn’t a reason to lower my guard.
If anything, it’s the reason to check twice.