TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A live experiment by Firmulate tests AI models on real-world business management tasks, exposing a gap in current evaluation methods. The results highlight that management quality, not just chat or coding skills, should define AI leadership.
Firmulate has introduced a live management benchmark that evaluates AI models based on their ability to run a simulated company through a crisis week, emphasizing decision-making, trust, and execution. This experiment aims to measure management quality, a dimension often overlooked in traditional AI benchmarks, which focus on technical output or conversational skill. The results challenge existing evaluation standards and suggest a new way to assess AI leadership in practical, high-stakes environments.
The final July 2026 Crucible League ranked five models based on their performance in managing a synthetic company with real financial stakes, customer crises, and internal complexities. The top performer, gpt-5.6-sol, scored 95 out of 100, while others like Kimi K3 and Sonnet 5 followed with 93 and 88 respectively. Notably, a baseline model scored only 26, underscoring the challenge of management tasks versus simple response generation.
All models successfully identified crises and resisted manipulation attempts, but only two signed a €55,000 deal, illustrating that diagnosis alone does not guarantee successful execution or trustworthiness. The experiment revealed that models often failed to retrieve critical information buried in documents, leading to missed opportunities despite sound analysis. The evaluation enforced strict trust standards: a single breach would cap the overall score, emphasizing integrity in AI evaluation.
Implications for AI Evaluation and Business Management
This experiment demonstrates that management skills—such as decision-making, trustworthiness, and execution—are crucial metrics for AI in business contexts. Traditional benchmarks, which often focus on language quality or technical accuracy, may overlook an AI’s ability to handle real-world complexities. The results suggest that future AI evaluation should incorporate consequential management tasks to better reflect the true potential and limitations of AI agents in operational settings. For enterprises, this shift could influence how AI tools are selected and integrated into decision-making processes, especially in high-stakes environments.
As an affiliate, we earn on qualifying purchases.
Limitations of Current AI Benchmarks and the Need for Real-World Testing
Most existing AI benchmarks evaluate models based on coding contests, chat arenas, or language benchmarks, which do not adequately simulate the pressures of real-world management. The Firmulate experiment builds on this by creating a live, dynamic environment where models must prioritize, read organizational context, and manage trust over days. This approach exposes weaknesses hidden by traditional tests, such as superficial analysis or failure to follow through on decisions, which are critical in operational settings.
Prior efforts to evaluate AI have focused on isolated tasks, but managing a business through crises requires holistic judgment, accountability, and trust. The experiment’s design, with versioned decisions and auditable actions, aims to bridge this gap, providing a more meaningful measure of AI’s readiness for real-world deployment.
“Management quality, not chat quality, deserves to become its own category of AI evaluation.”
— Thorsten Meyer, lead researcher at Firmulate
business crisis management AI tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Remaining Questions About AI Management Evaluation
It is not yet clear how well these findings will generalize to actual companies outside the synthetic environment. The experiment’s controlled setting, while realistic, cannot fully replicate the unpredictable dynamics of real organizations. Additionally, the long-term impact of integrating such management-focused benchmarks into AI development remains to be seen. Further research is needed to determine whether models trained and evaluated in this manner will translate into better performance in real-world management tasks.
AI decision-making training programs
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Next Steps for Broader Adoption and Development
Firmulate plans to expand the experiment, inviting more models and organizations to participate. There is also interest in developing standardized management benchmarks that can be integrated into AI development pipelines. Industry stakeholders are encouraged to consider management and trust metrics when deploying AI tools in operational roles. Future research will explore how models can improve in areas like information retrieval, escalation, and maintaining trust over extended periods, moving toward AI that can truly manage consequences in complex environments.
AI leadership evaluation platforms
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
How does this new benchmark differ from traditional AI evaluations?
This benchmark evaluates AI models based on their ability to manage a simulated company during crises, focusing on decision-making, trust, and execution, rather than just language quality or technical tasks.
Why is management ability important for AI in business?
Management ability determines whether AI can handle real-world complexities, prioritize effectively, and maintain trust—crucial for operational success and risk mitigation.
Can these findings be applied to real companies?
The experiment offers valuable insights, but further research is needed to confirm how well these results translate to actual organizational environments.
What should companies consider when adopting AI tools based on this research?
Organizations should evaluate whether AI models can read organizational context, make responsible decisions, and maintain trust, beyond just producing high-quality responses.
What is the next step for AI evaluation standards?
Developing benchmarks that incorporate management and consequence management tasks, moving beyond response quality to assess real-world operational capabilities.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
