The Strawberry Test, and When AI Agreement Is Actually Worth Trusting
How many r's are in strawberry? AI models famously used to blow this. I ran it through five of them at once. They all got it right, and the interesting part is why that agreement was trustworthy and most agreement is not.
If you have followed AI for the last couple of years, you know the strawberry thing. For a long time, if you asked a top model how many times the letter r appears in the word strawberry, it would say two. Confidently. It became the little test everyone used to show that these systems can miss something a first-grader gets right.
So I ran it through the Council Room, where five different AI models answer the same question independently. Here is what came back.
All five said three
Every one of them counted correctly: s-t-r-a-w-b-e-r-r-y, three r's, one after the "st" and two in the "berry." The models have gotten better, and on this particular question they have caught up.
You could stop there and conclude "great, they agree, the answer is three." And this time you would be right. But I want to show you why you would be right, because it is not the reason most people think, and getting this wrong is how confident agreement fools you on the questions that actually matter.
Agreement is not why you can trust it
Here is the trap. When five things tell you the same answer, your gut says "five sources agree, must be true." That instinct is wrong for AI, because these models are not independent the way five witnesses are. They learned from largely the same internet. When they share a blind spot, they agree on the wrong answer just as confidently as the right one. That is exactly what the old strawberry failure was: models agreeing on two, together, and all wrong.
So the number of models that agreed is not what makes this trustworthy.
Evidence is why you can trust it
Look at what the synthesizer actually said when it wrote the verdict. It did not say "three, because everyone agreed." It said three is correct because it is directly verifiable by spelling the word out letter by letter, a checkable fact, not a matter of opinion or majority vote.
That is the whole difference. This agreement is trustworthy because every model showed the same checkable evidence, the actual spelling, that you can verify yourself in two seconds. It is not trustworthy because five of them said it.
That distinction is the entire method. On a question where the models agree but none of them can show you an openable source, the agreement means nothing, and you should trust it exactly as much as you would trust one model saying it five times. On a question where they agree and every one of them points at the same verifiable evidence, now you have something. Same agreement on the surface. Completely different weight underneath.
What to actually do with this
When AI models agree, do not relax. Ask the follow-up: are they agreeing because they can each show me the evidence, or are they just echoing each other? The strawberry test is easy because the evidence is right there in the word. Most real questions are not that easy, and that is precisely where you have to force each answer to name a source you can open before the agreement counts for anything.
You can watch this happen on a real question in the Council Room, and you can read the exact run this article came from here. The deeper version of this idea is the companion piece, agreement is not verification, and the short answer is on the question page, if two AI models agree, does that mean the answer is right.
The full method, the one that teaches you to weigh answers by evidence instead of by how many models repeated them, is Let the AI Be Smart. This site teaches the Council Method for free.
They all got the strawberry right. Trust that answer because you can spell the word, not because five machines nodded. That habit is the difference between using AI and getting used by it.