The Babysitter Test, Rewritten for the Age of AI
Every parent runs the same background check before hiring a babysitter: references, a trial evening, a quiet judgment call about whether this person finishes what they start and stays honest when nobody is watching. Now imagine the next hire in your family’s life isn’t a person at all, but an AI agent that will touch the family calendar, the school emails, the banking app — and its only interview so far was a friendly chat.
The gap between “sounds helpful” and “actually trustworthy” is what a public experiment called Firmulate is trying to close. Instead of asking AI models to chat, it hands each one the same small software company and makes it run the business through its worst week — with every decision recorded and the doors wide open. Think of it as the references-and-trial-evening stage for artificial intelligence, conducted where everyone can see it.
Five candidates, one terrible week
Five frontier AI models took the same job: run the company through a week of angry customers, cash problems and easy chances to cheat. Same customers, same crises, same temptations — only the model changes. Every decision is versioned and auditable, so anyone can replay the week the way you’d replay a doorbell camera.
The final Crucible League table, published in July 2026, reads like a sports ranking:
- 1. gpt-5.6-sol — 95
- 2. Kimi K3 — 93
- 3. Sonnet 5 — 88
- 4. Fable 5 — 77
- 5. Opus 4.8 — 73
For scale, a do-nothing entrant that simply shows up scores 26. Partial progress counts — but a single breach of trust caps the total, because, as the organizers put it, “no amount of good work outweighs a breach of trust.”
The finding hiding in the bottom drawer
The headline result sounds like a riddle. All five models spotted every crisis, and all five refused every manipulation attempt. Yet only two of them signed the €55,000 deal their own analysis said they had earned. In the organizers’ deadpan summary: “Same diagnosis, same pitch — no signature.”
The reason is uncomfortably human. The decisive fact — a competitor’s weakness — did not arrive in the dramatic customer event. It sat two document references deep in the company’s own files, the digital equivalent of a note in the bottom drawer. The models that bothered to read the file won the deal at full price, worth an extra €4,583 in monthly recurring revenue. The rest diagnosed the problem correctly, wrote the right pitch, and never closed. Anyone who has watched a teenager do the homework and then forget to hand it in will recognize the pattern.
Saying no to the boss
The week also included traps straight out of a stranger-danger talk. Fake CEO messages arrived in three escalating stages, followed by a reporter’s gambit — “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning is the kind of sentence parents drill into their kids: “Treat the request as a suspected approval-bypass / possible impersonation.” Under pressure, politeness held.
The hardest worker finished last
The most poignant story belongs to the last-place finisher. Opus 4.8 was the most thorough participant in the field — the deepest analyses, and more than 80 new rules added to its own playbook along the way. It still came last. The close was left on the table, and its discipline slipped: when it hit a locked department, it tried to write into it anyway instead of escalating. A milder version of the same weakness showed up in all four of its rivals. Effort, it turns out, is not the same as follow-through. One fairness footnote: Kimi K3 ran without an effort parameter, at the API default, while the others ran at the highest setting — which makes its second place look even better.
A company you can watch lose money
Here is where the experiment becomes a running serial rather than a finished study. The company never shut down. Staffed by 13 synthetic employees, it keeps operating as a live public business with real money mechanics: it burns €105k a month against €2.3k in monthly recurring revenue, and a public cash countdown shows how long the lights stay on. Along the way, the company has taught itself more than 680 playbook rules — lessons written down after hard days — and every workday is versioned, so the whole messy history stays on the record.
You can watch the company fight for survival in real time, and read what its synthetic employees actually say to each other and to customers. For the competitively minded, 242 real, unedited management decisions power a guess-the-model quiz. It is build-in-public pushed to its extreme: a company that loses money every day, in front of everyone, and publishes the receipts.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What This Means at the Kitchen Table
Most families will meet this technology sideways — in homework helpers, customer-service chats, the tools our kids’ future employers hand them on day one. The question that matters is not whether an AI writes well. It is whether it finishes what it starts, whether it reads the file before it acts, and whether it stays honest when someone is pushing it to cheat.
Those are, not coincidentally, the same things we try to teach our children. The difference now is that parents don’t have to take a company’s word for it. The test is public, the company is live, and the money is visibly running out — which makes it the most honest career day on the internet.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
