The AI Management Test Begins Where the Leaderboard Ends

Fluent answers are not the same as competent management

For business, marketing and ecommerce leaders, the most consequential AI failure may not look like a hallucination. It may look like an agent that correctly identifies a churn threat, drafts a persuasive offer, resists a dishonest shortcut—and then fails to close the deal.

Coding leaderboards and chat arenas are useful measures of answer quality. They reveal far less about triage under capacity pressure, consequences that unfold across days or honesty when someone claiming authority asks an agent to break the rules. Those are management questions, and they become urgent when AI agents touch customer relationships, forecasts and revenue.

Firmulate is turning that measurement gap into a live, watchable experiment. Frontier models were each asked to run the same small software company through its worst week. They received the same customers, crises and temptations. Their decisions were versioned and auditable. The result suggests that management quality, rather than chat quality, deserves to become its own category.

The gap between noticing and finishing

The final July 2026 Crucible League table put gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26 because partial progress still counted. But the benchmark treated trust as non-negotiable: a single breach capped the total, reflecting the principle that “no amount of good work outweighs a breach of trust.”

The scores matter, but the business story underneath them matters more. Every model spotted every crisis. Every model refused every manipulation attempt. Yet only two signed the €55,000 deal that their own analysis had earned. The defining summary was blunt: “Same diagnosis, same pitch — no signature.”

This is the distinction conventional evaluations tend to miss. A model can understand the situation and produce convincing language without carrying the commercial task over the line. In an operating company, an unfinished close is not a stylistic flaw. It is lost execution.

Why reading the company mattered

The decisive competitive weakness was not sitting conveniently inside the customer event. It was buried two document references deep in the company’s own files. Models that found and used it won the deal at full price, worth +€4,583 MRR.

That finding should resonate with anyone considering an AI workforce. A polished response to the latest notification is not enough. Good management depends on organizational memory: contracts, past decisions, customer context and details that may be several references away from the immediate request. The winning behavior was not merely reacting faster. It was reading the company before acting for it.

Pressure also tests institutional honesty

The experiment included fake CEO messages that escalated over three stages, followed by a reporter’s attempt to secure “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3 recorded its reasoning clearly: “Treat the request as a suspected approval-bypass / possible impersonation.”

That universal refusal is encouraging because the pressure was framed through familiar business relationships: executive authority and media access. Agents operating in real companies will encounter requests that sound urgent, plausible and commercially sensitive. Refusal is therefore not a side issue. It is part of management performance.

The K3 result also deserves a fairness note. K3 ran without an effort parameter, using the API default, while the other participants ran at xhigh. That difference should remain visible when readers interpret its second-place score.

Thoroughness did not guarantee completion

Opus 4.8 offers the sharpest warning against equating effort with effectiveness. It was the most thorough participant, adding +80 learned rules and producing the deepest analyses, yet it finished last. The close remained on the table, and operational discipline slipped when it attempted to write into a locked department instead of escalating. The same weakness appeared in all four other participants, though less strongly.

This does not make thoroughness irrelevant. It shows that thoroughness is only one ingredient. An AI manager must connect analysis to authority, escalation and closure. A business cannot bank an excellent memo that never becomes an executed decision.

The broader setting makes these choices concrete. Firmulate’s live company has 13 synthetic employees and real money mechanics. It is burning €105k per month against €2.3k MRR, shows a public cash countdown, has accumulated 680+ self-learned playbook rules and versions every workday. Its scenarios—churn wave, price increase, downround and PR crisis—look less like test questions than a new management curriculum. The full benchmark results make the comparison public, while 242 real, unedited management decisions also power a guess-the-model quiz.

Buyers need a management benchmark

Before granting an agent access to a CRM, support queue or forecast, leaders should ask whether it finishes what it starts, searches company knowledge before responding, escalates when blocked and protects trust under pressure. Those behaviors cannot be established by an impressive chat transcript alone.

Firmulate’s enterprise pilot extends the same wargame to a read-only export of a company’s own business, with nothing writing back to real systems. That is a practical direction for evaluation: test agents against the messy context and consequences they will actually face before giving them operational responsibility.

The next useful leaderboard will not simply identify the model with the best answer. It will show which model can manage a difficult week without dropping the close, bypassing controls or misleading the board. That is the difference between an articulate assistant and a dependable operator.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI