The blind test
I took 20 real claims and made 70 fakes by corrupting real ones in six ways, listed in the table. Then I labeled them blind. I stopped at 49 of the 90, once every kind of fake had been sampled. Of the 29 fakes I ruled on, I caught 26.
Then the judge took the same test. It accepted all 20 real claims. Of the 70 fakes it caught 42, passed 27 as true and was unsure on one. It caught every changed number and every flipped meaning. It was worst on thin quotes, 2 of 12.
The thin-quote misses came from the setup. The judge saw 600 characters around each quote, text the agent never gets. Shown the quote alone, it caught 7 of 12.
I had hoped the judge would catch what I miss. It missed the same three fakes I did. So code handles what can be checked mechanically, and a person makes the judgment calls.
The last check was on real work. Four meeting briefs cited 282 claims. The quote alone settled 186. AI reviewers read the other 96 against the full source, to a standard I set. None were wrong. Three were unclear, like a tentative date stated as firm.
| Changed number | 12/12 |
|---|---|
| Flipped meaning or swapped name | 11/11 |
| Wrong speaker | 7/12 |
| Stretched paraphrase | 6/11 |
| Overclaim | 4/12 |
| Quote too thin | 2/12 |
| All fakes | 42/70 |