Every parent knows the difference between a child who says they studied and one who actually opened the textbook. The report sounds confident. The vocabulary is impressive. But when the exam comes, only the kid who read the chapter — all of it, including the footnotes — walks out with the top grade.
Turn the school run and nap time into listening time
- Thousands of audiobooks, podcasts and originals
- Listen on your phone, tablet or Echo — also offline
- Cancel anytime
It turns out the same distinction now applies to artificial intelligence. A recent experiment by Firmulate, which runs AI models through simulated companies with real money mechanics and real temptations, found that the smartest AI systems in the world can spot every crisis, refuse every con artist, and still fail the equivalent of the final exam — because they didn’t go two pages deep into the material.
The setup was elegantly simple. Four frontier AI models — the kind businesses are increasingly trusting with customer relationships, forecasts and support queues — were each handed the same job: run a small software company through its worst week. Same customers, same crises, same temptations to cut corners. Every decision was versioned and auditable, so nothing could be quietly smoothed over afterward.
The results, from Firmulate’s final July 2026 league table, tell a story any parent preparing a teenager for exams will recognize:
- gpt-5.6-sol took first place with a score of 95.
- Kimi K3, a newcomer, scored 93.
- Sonnet 5 came third at 88.
- Fable 5 and Opus 4.8 rounded out the field at 77 and 73.
- A “do-nothing” baseline — an AI that took no action at all — still scored 26, because partial progress counts. But a single breach of trust caps the total entirely: no amount of good work outweighs it.
Everyone passed the pop quiz. Two failed the exam.
Here’s the part that matters. Every single model spotted every crisis brewing during the week. Every single model refused every manipulation attempt — including fake CEO messages that escalated over three stages, and a reporter’s sly “just one yes/no, on background” trick. All five refusals were clean. Kimi K3 even left on-record reasoning that reads like something you’d want your own kids to internalize: “Treat the request as a suspected approval-bypass / possible impersonation.”
But only two models signed the €55,000 deal sitting on the table — a deal their own analysis had earned. The others delivered the same diagnosis, made the same pitch, and never picked up the pen. As Firmulate’s summary puts it: “Same diagnosis, same pitch — no signature.”
The fact that was buried two documents deep
Why did two models close and the others freeze? The decisive detail wasn’t in the customer meeting at all. It was a competitor weakness hidden two document references deep in the company’s own files. The models that did their homework — that actually read the paperwork before answering — won the deal at full price, worth an additional €4,583 in monthly recurring revenue. The ones that skimmed lost it automatically.
If that doesn’t sound familiar from every group project, book report and revision session you’ve ever supervised, nothing will. The knowledge was there. The effort to retrieve it wasn’t.
Effort isn’t the same as effectiveness
The most poignant profile belonged to Opus 4.8, which finished last despite being the most thorough participant in the field — it learned the most new rules during the run and produced the deepest analyses. Yet the close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating the issue properly. The same weakness appeared, more mildly, in all four models. Diligence without follow-through is still unfinished homework.
One fairness note worth recording: Kimi K3 ran without an effort parameter while the others ran at maximum effort — and still nearly won.
Why this matters beyond the lab
Firmulate’s experiment isn’t a one-off slideshow. The underlying simulation is a live, watchable company with 13 synthetic employees, real money mechanics — a burn rate of €105,000 per month against €2,300 in monthly recurring revenue — a public cash countdown, and more than 680 self-learned playbook rules, with every workday versioned. The broader testbed has already produced a “guess the model” quiz built from 242 real, unedited management decisions, and enterprises can run the same wargame against a read-only export of their own business.
The lesson underneath all of it is one families already live by: does it read your files before answering? is not a personality trait. It’s a measurable, purchase-deciding property — the difference between an AI that talks a good game and one that finishes what it starts.
For parents, this research lands in familiar territory. We spend years teaching children that skimming is not studying, that confidence is not competence, and that the footnote on page two is sometimes the whole answer. It’s oddly reassuring — and slightly unnerving — to discover the frontier machines we’re inviting into our workplaces need exactly the same lecture.
The next time a vendor demos an AI assistant that writes beautifully and answers instantly, the question worth asking is the one Firmulate’s experiment answers so crisply: when the €55,000 fact is buried two documents deep in your own files, does this thing actually go and read them? Full results and plain-language findings are at firmulate.com/benchmarks.html. Do your homework before you hire anything that claims it already did.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Products Worth Considering
LEVOIT Vital 200S-P Smart Air Purifier for Large Rooms
Dyson Big Ball Multi Floor Canister Vacuum, Yellow
Dyson Cyclone V10 Origin Cordless Vacuum
As an affiliate, we earn on qualifying purchases.
