🔍 Read the full analysis: How 71 AI Configurations Changed My View Of My News Pipeline on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer reports testing 71 AI configurations on real news-writing tasks between September 8 and October 13, 2026. GPT-6 Astra scored highest, while GPT-6 Luna was selected for production because it cost about one-hundredth as much; the test also led Meyer to remove two backup models that produced fabricated quotes or unusable output.
Thorsten Meyer says a test of 71 AI configurations for his news pipeline found GPT-6 Astra produced the highest-scoring articles, while the far cheaper GPT-6 Luna was his choice for production. In a report published October 13, Meyer also said two backup models generated fabricated quotations or unusable output, prompting their removal from the news route on October 7.
Meyer tested the configurations from September 8 to October 13, 2026, using his production prompt and real news sources. The test covered API, subscription, free-tier and locally run open-weight models. Each source was turned into an article in both English and German, then evaluated by two AI judges, GPT-6.1 Sol and Claude Opus 5.5. The scoring categories included fidelity to the source, fabrication, clarity about uncertainty, news value, structure and polish; German articles were also assessed for language quality.
In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a perfect source-fidelity score. In a follow-up robustness run spanning 12 articles, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer reported Astra cost about $0.12 per article and took around 50 seconds. GPT-6 Luna averaged 3.67 in the robustness run, had source fidelity of 4.71, and produced valid output for every article. Its reported production cost was about $0.0013 per article, with a median completion time of 17 seconds.
The report says GPT-4.1 nano, previously used as a backup, scored 1.62 in English and 1.68 in German, with source fidelity of 1.00. The judges flagged quotations attributed to an unnamed industry analyst and company spokesperson that were not in the source. A locally run gpt-oss-120b produced no valid articles in four attempts, according to Meyer; he said it also generated invented named quotations. Meyer removed both models from the route on October 7. The source text ends mid-sentence while describing the replacement setup, so it does not specify the complete failover chain.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Cost and Reliability Shape the Choice
The results illustrate why a model leaderboard may not answer a publisher’s practical question: which system can produce publishable copy from the sources the newsroom actually uses? Meyer’s reported scores suggest the top performer and the production choice were not the same model. Astra scored higher, but Luna’s much lower reported price and shorter completion time made it the selected option for this high-volume pipeline.
The reported fallback failures carry a separate operational lesson. A backup model is not merely a continuity measure if its output can introduce quotations absent from the source or fail the formatting checks required by production. Such errors can undermine source fidelity and editorial trust. Meyer’s account says those two models were removed, though the report does not provide independent verification or a full description of the live system.
As an affiliate, we earn on qualifying purchases.
How the Pipeline Was Tested
The evaluation used two sources: a 6,340-character product filing and a 619-character political news item. Meyer said most of his news starts from short sources, making the shorter item the more representative test. Each configuration wrote English and German versions using the same production prompt, and both judges saw the original source before scoring.
Meyer also set limits on what he would infer from the results. He said that, with only two articles in the standard test, score gaps below about 0.3 points should be treated as noise; he considered the multi-article robustness run a more dependable basis for ranking. He counted an article only if it produced valid output after the same repair step used in production. These choices matter because the headline scores alone do not show how a model performs across a larger, varied newsroom workload.
The report also compares writing results with other task rankings. Meyer says Claude Opus 5.5, which he describes as the strongest model on a coding benchmark, scored 3.46 in English and 3.39 in German in this writing test, and averaged 3.52 in the robustness run. That comparison is Meyer’s account of the benchmark and his test, not evidence that coding performance predicts news-writing quality across all systems.
“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”
— Thorsten Meyer, in the October 13 report
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Results
The findings come from one publisher’s pipeline, one production prompt and two test sources in the standard run. The robustness run adds 10 more articles for the strongest configurations, mostly from short sources, but the report does not establish how the rankings would hold across other publishers, topics, languages or editorial standards. The scores were assigned by AI judges, not a panel of human editors, and the source does not describe a human-review comparison.
The report gives confidence intervals for the robustness results, but does not provide enough detail in the supplied material to independently reproduce the full evaluation. It also does not identify the complete replacement failover chain: the text says GPT-6 Luna now runs first, then cuts off before naming the next model. Claims about model pricing, benchmark standing and production performance are attributed to Meyer and may change over time.
AI language model API subscriptions
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Production Use and Further Testing
Meyer’s report says GPT-6 Luna remains in production and that GPT-4.1 nano and gpt-oss-120b were removed from the news route on October 7. The next useful evidence would be continued results from live publishing: whether Luna maintains its reported quality across a larger range of sources, and how often editors or automated repair steps must intervene.
The article does not announce a planned follow-up date or a new model evaluation. Readers should treat the reported ranking as a snapshot of Meyer’s tests through October 13, 2026, rather than a general verdict on which model writes news best.
As an affiliate, we earn on qualifying purchases.
Key Questions
Which configuration scored highest in the test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across 12 articles in the reported robustness run.
Why did Meyer choose GPT-6 Luna for production?
Meyer reported that Luna averaged 3.67 in the robustness run, produced valid output for every tested article and cost about $0.0013 per article in production. He said Astra cost about $0.12 per article, despite its higher score.
What did the test find about higher reasoning effort?
Meyer tested six reasoning-effort levels for GPT-6 Luna. He reported that higher effort did not produce a reliable quality gain, while the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort setting.
Why were two fallback models removed?
According to Meyer, GPT-4.1 nano produced quotations not supported by its source, while gpt-oss-120b produced no valid articles in four attempts and generated invented named quotes. He said both were removed from the news route on October 7.
Do the results show which AI model is best for every newsroom?
No. The evaluation used Meyer’s own prompt, sources, languages, production checks and AI judges. It offers evidence about his pipeline, but does not establish that the same model rankings apply to other publishers or tasks.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
