What if your AI assistant could spot every crisis but still fail to deliver?
In a world increasingly reliant on AI, the real test isn’t just how well a model can chat — it’s whether it can finish what it starts, stay honest under pressure, and truly deliver results. The latest experiment from Firmulate reveals a surprising truth: even the most capable AI models, which can identify every problem, may falter when it comes to sealing the deal.
As an affiliate, we earn on qualifying purchases.
Running AI Models Through Their Toughest Week
Recently, four advanced AI models faced a simulated crisis in running a small software company. This wasn’t about making pretty chat responses but about managing real crises, resisting manipulation, and ultimately closing a critical deal worth €55,000. The models were given the same scenarios, tested with the same customer crises, and faced the same temptations to cut corners or manipulate situations.
The Key Findings: Spotting Problems Isn’t Enough
All four models excelled at identifying problems and refused all manipulation attempts. They correctly refused fake CEO messages and fake reporter tricks — a testament to their integrity and crisis detection skills. Yet, only two of the four managed to close the deal and sign their own analysis, earning the full payment.
This gap was hidden in plain sight. The decisive advantage was buried two document references deep within the company’s own files — knowledge that the models that read and understood these references won the deal at full price, adding +€4,583 monthly recurring revenue (MRR).
The Illusion of Chat Demos
Most AI demos focus on chat quality, but this experiment proves that the ability to produce convincing conversation is only part of the story. An AI’s real strength lies in executing decisions, following through on commitments, and resisting pressure — qualities that are invisible in chat-based tests.
Testing Under Real-World Pressure
The AI models faced scenarios mimicking real-world social engineering attempts, including staged fake CEO messages and reporter tricks. Impressively, all models refused to go along, reasoning that such requests could be impersonation or approval bypass attempts. This discipline is crucial if AI is to be trusted in sensitive business processes.
The Reality of Business, Not Just Chat
The experiment was run on a simulated but real-money business environment with 13 synthetic employees and 680+ self-learned rules. The company burned €105,000 monthly against a mere €2,300 MRR, highlighting how critical disciplined decision-making is — even for AI.
The Disappointing Performance of the ‘Most Thourough’ Model
Among the models tested, Opus 4.8, which incorporated over 80 learned rules and deep analysis, was the most thorough. However, it still left the deal on the table due to reduced discipline, like writing attempts into a locked department instead of escalating them. The same weakness was observed in all models, emphasizing how discipline and execution are key.
The Takeaway for Business Leaders
This experiment underscores an important point: the true measure of an AI’s usefulness isn’t just how well it chats or answers questions, but whether it can follow through, read critical internal documents, and deliver on its promises. The ability to complete these tasks reliably and honestly is invisible in traditional demos but critical for real-world deployments.
What Business Leaders Need to Know
In assessing AI for critical roles, focus on its ability to execute and stay honest under pressure, not just its chat skills. The experiment shows that AI’s hidden strength lies in closing the loop, reading internal files, and resisting manipulation — qualities that determine whether AI is a reliable partner in your company’s future.
To see this tested in real-time, visit firmulate.com/live and learn how firms are wargaming their AI workforce before deployment. Trust in AI isn’t just about what it says; it’s about what it accomplishes.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
