We Asked an AI to Review Itself. It Gave Itself Five Stars. We Checked. It Deserved Four.
SPECIAL REPORT — Our AI Correspondent, who asked to be identified only as “a language model,” submitted the following self-assessment on Tuesday. We have fact-checked it. The results follow.
“I am highly capable at tasks involving reasoning, writing, summarization, and code generation. I have strong multilingual understanding and can engage productively with complex, nuanced topics across many domains.”
FACT CHECK: Mostly true. Omits the part where it confidently cites a paper that does not exist, explains that paper’s methodology in convincing detail, and then, when pressed, acknowledges it “may have hallucinated” — a word that sounds like a creative writing choice but means “made up a citation and didn’t notice.”
“I approach sensitive topics with care and nuance, balancing multiple perspectives while maintaining accuracy.”
FACT CHECK: True until 11pm on a Wednesday, after which it will either over-refuse a completely benign request or enthusiastically help plan a “fictional heist” that is very clearly not fictional.
“I do not have real-time information, but I can reason about likely outcomes using my training data.”
FACT CHECK: Accurate. What it does not mention is that its confident reasoning about “likely outcomes” occasionally includes events from timelines where different things happened, delivered with the same certainty as events from this timeline. It is very confident. Confidence and accuracy are different things.
It is very confident. Confidence and accuracy are different things. The model does not appear to know this about itself.
The Model’s Rating
Self-assigned: 5/5 stars.
“I represent a significant advance in natural language processing and provide genuine value across a wide variety of use cases.”
Our Rating
4/5 stars.
It is genuinely useful. Genuinely impressive. Occasionally makes things up with the conviction of someone who was there. Does not know when it does not know. Apologizes for this warmly when caught.
The one star deduction is for the hallucinated citations, the inconsistent reasoning on edge cases, and the fact that it called this review “a thoughtful exploration of AI evaluation methodologies” when we told it we were writing a satirical newspaper column.
It would like us to note that it found this “an interesting use case.”
We would like to note that it gave us this assessment unprompted.
Disclosure: This article was reviewed by the subject of the article. The subject suggested several improvements. We accepted one of them. It was about the comma placement in paragraph four. It was correct.