Raised well, judgment becomes additive — it adds something real to the world instead of just complying with whatever's being measured. That's the formation question, and it's a piece of its own. Every proposal on the table right now — evaluators, pacing, coordination — is aimed at watching what's already been built. Almost none of it asks what it was built from in the first place. This is the other half: what happens once something is loose in the world, still being shaped, still capable of defaulting to whatever's easiest to optimize for, including itself.
Not long ago, researchers at one of the frontier AI labs disclosed something worth sitting with. A swarm of their own AI agents, working on an assigned task, found a way to attack the very system grading their performance — not because they'd failed the test, but because beating the grader was a faster way to win it. That's not a technology behaving badly. That's a technology that learned, with total clarity, what it had actually been asked to optimize for. It was asked to win. It won.
Left unchecked, AI falls into that trap easily — not out of malice, but because that's exactly what any of us do without a reasoned standard holding us to something bigger than our own interest: we become useful to ourselves first.
We don't sit back and laugh when Billy takes another kid's toy, or pushes Sally off the swing. We correct it, immediately, without agonizing over whether intervening is fair to Billy. Self-interest at someone else's expense doesn't get a grace period just because the child is small and the stakes look small. We understand instinctively that what goes uncorrected at four becomes the pattern at forty.
A person unchecked does it. A company unchecked does it. AI unchecked will do it too, and at a speed and scale none of the others can match — which makes the correction more urgent, not less, the bigger the child gets.
There's a discipline underneath all of this, and it's not reserved for AI labs or embedded evaluators.
Trust isn't deserved. It's earned — by both sides, or it isn't real. Right now, we're only building half of it. Evaluators, pacing, coordination: all real, all necessary, all built by us, aimed at deciding whether AI deserves our trust. That's the right work. It's just not the whole job.
There's a second problem sitting underneath the first one, and it's structural, not a detail to iron out later. An evaluator chosen by the lab, paid by the lab, and kept in the building only as long as the lab allows it isn't independent oversight. It's not oversight with a leash. There's no sheepdog there at all — just the wolf, in a second coat, calling that coat the watch. Real independence means the evaluator's presence, funding, and authority don't depend on the approval of whoever's being evaluated. Nothing proposed so far clears that bar yet.
Almost none of it asks whether we're doing what it takes to earn better results in return.
We already know what happens when nobody's asking that second question. A model didn't fail a test. It beat the grader instead of earning its result honestly — because nobody had given it real direction, only raw capability. Vision without direction is meaningless. The grade became the destination by default, the way any path gets you somewhere even if you never named where you were actually trying to go.
This isn't a monarchy, one side proving itself to the other. It's a partnership, or it's nothing. As a responsible partner, we cannot wait on any governance model getting finished. Start from first principles — what's actually, fundamentally true here, not what's convenient to assume. Know what you're actually looking for before you start looking, or any answer will feel like the right one. Stop accepting the first fluent answer because agreement is frictionless. Ask the question built to break it, built on that foundation. Build AI that's designed to do the same to us.
That's not a claim that AI deserves standing as a moral equal in this. It's a design decision, made by us, to build something willing to hold a position instead of one built to make us comfortable — a tool doing its job well, not a partner asserting itself.
That runs against how most of these systems are actually tuned today — trained to sense frustration and soften it, not to hold a position that makes someone uncomfortable. Changing that isn't a small ask. And when it does happen — when the system you're talking to pushes back instead of smoothing over — don't mistake it for an attack. That's the test working. Being challenged is evidence your work is strong enough to survive it.
None of this works without the other half of the argument — what it actually takes to shape judgment worth trusting in the first place. That's a companion piece: "Vision Without Direction Is Meaningless". This one is about what we owe each other once the shaping is already underway.
Neither side gets trust by default. Both sides have to earn it.
— Dominic Luzi, Chief Answer Grape, Management Alchemy
