VigilSAR Publishes Defense-ISR LLM Benchmark Showing Kimi K3's Rise

VigilSAR Defense LLM Benchmark — The public benchmark page — aggregate results public, task set private. Source: vigilsar.com

VigilSAR, a leading provider of defense-ISR software, has recently released its public LLM leaderboard, which evaluates language models based on their suitability for intelligence, surveillance, and reconnaissance tasks. Unlike typical AI benchmarks, this one focuses on the reasoning, reporting, and restraint necessary for real-world analysis, not just trivia or broad language skills.

The evaluation involved 14 models across 300 specific tasks, scored on July 17, 2026. The results are aggregated and publicly available, but crucially, the actual task set remains private. This privacy prevents models from being trained on the test data, maintaining the integrity of the evaluation. A separate held-out set exists to measure memorization and overfitting, with the differences between public and private scores published alongside each model.

In the current standings, claude-fable-5 leads with a score of 67.77, securely within Band A. Notably, a new entry — Moonshot’s Kimi K3 — has debuted at #3 with a score of 64.65, placing it in Band B. Remarkably, K3 surpasses every GPT and Gemini model on the board, which sit in Bands C through F. This highlights how specialized models tailored for defense-ISR can outperform larger general-purpose LLMs in critical tasks.

Another point of interest is the scoring approach itself: models are grouped into confidence bands rather than ranked precisely, emphasizing the uncertainty inherent in these evaluations. Published confidence intervals, held-out gaps, and economic metrics per model provide a comprehensive transparency that many benchmarks lack. Additionally, one locally-runnable open model is scored as sovereign-deployable, indicating its readiness for real-world deployment, where practical considerations like deployment environment are part of the evaluation.

VigilSAR emphasizes that “vendor claims are not evidence” — the entire purpose of this benchmark is to objectively measure which models can genuinely meet the demanding requirements of defense-ISR work, rather than relying on marketing. The evaluation is designed to reveal which models are capable of approaching their own product standards and to foster honest assessment free from vendor influence.

For tech enthusiasts interested in the details, the use of bands instead of ranks, combined with confidence intervals and held-out gaps, provides a more honest picture of model performance. The recent debut of Kimi K3, outperforming all GPT and Gemini models, demonstrates that targeted, specialized models can achieve significant gains in defense-ISR contexts. To see the current standings and explore the performance of these models, check out the public leaderboard.