I Tested My News Pipeline With 71 AI Setups. Here’s What Stood Out

🔍 Read the full analysis: I Tested My News Pipeline With 71 AI Setups. Here’s What Stood Out on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a test of 71 AI configurations found GPT-6 Astra produced the highest-rated news articles, while GPT-6 Luna was his lower-cost choice for production. The results, published October 13 on ThorstenMeyerAI.com, also identified two backup models that generated invented quotes or failed to produce usable articles; Meyer says he removed them from his news pipeline on October 7.

Meyer tested models from several categories, including API and subscription services, free-tier options on OpenRouter and open-weight models running on his hardware. Each configuration used his production prompt to turn real news sources into articles in English and German. The test included a 6,340-character product filing and a 619-character political news item, which Meyer said better reflected the short sources common in his workflow.

Two AI systems graded each article against the source, scoring source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; the German articles also received a language-quality score. Meyer reported that GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test, with perfect source-fidelity scores there. Across a further 12-article robustness run, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02.

GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test. Meyer reported a 3.67 average in the robustness run, source fidelity of 4.71 out of 5, and valid output for every article in that run. He said Astra cost about $0.12 per article, compared with roughly $0.0012 for Luna, and took about 50 seconds rather than 17. In his account, production use of Luna over the previous week cost about $0.0013 per article at a median of 17 seconds.

Cost and Quality in Production

The results show why a general model ranking may not settle which system to use for a particular publishing task. In Meyer’s test, Astra led on article quality, but its reported per-article cost was about 100 times Luna’s. For a high-volume pipeline, that difference can shape whether the top-scoring model is practical, provided the cheaper option meets the publisher’s quality requirements.

The test also points to risks beyond the main model. Meyer reported that GPT-4.1 nano scored 1.62 in English and 1.68 in German, with source fidelity of 1.00, and that evaluators flagged invented quotes. A locally run gpt-oss-120b produced no valid articles in four attempts and generated named quotes that were not in the source, according to Meyer. Those findings matter because fallback systems may be used when a preferred model is unavailable, yet still generate content that looks publishable.

These are results from one publisher’s prompt, sources, evaluation setup and sample—not a universal ranking of models. The practical takeaway is narrower: news organizations considering AI should test systems on their own work and check backup routes as well as their primary model.

How Meyer Built the Test

The test ran from September 8 to October 13, 2026, using the same prompt as Meyer’s production pipeline. Articles were generated in English and German from two source texts, one comparatively long and one short. Meyer described the shorter political item as the more representative test because many stories in his workflow begin with brief sources.

GPT-6.1 Sol, using high reasoning effort, and Claude Opus 5.5 served as the two AI judges. Both received the original source alongside the generated article. Meyer also ran a robustness test in which the strongest configurations wrote 10 additional articles from real sources, mostly short ones. He said an output counted only if it was valid after the same repair step used in production.

Meyer cautioned that the initial test involved only two source articles and said score differences under about 0.3 points should be treated as noise. He considered the larger robustness run a more reliable basis for ranking. He also reported that GPT-6 Luna was tested at six reasoning-effort settings; higher effort did not produce a consistent quality gain, while the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort setting.

“Public leaderboards tell you which model is smartest. They don’t tell you which model should write your articles.”

— Thorsten Meyer, describing the test on ThorstenMeyerAI.com

Limits of the Model Scores

The findings have not been independently verified in the material provided and reflect one publisher’s test setup. The standard comparison used just two source articles, and the robustness run covered 12 articles for the leading configuration, according to Meyer. The reported judge scores therefore do not establish how the models would perform across other topics, prompts, languages or newsroom standards.

Meyer’s account does not provide the full article set, individual judge assessments or enough detail to independently reproduce the scoring. It also does not establish whether the same rankings would hold as models or pricing change. His reported costs and timings are tied to his own usage and test conditions, not guaranteed rates for other users.

Luna Remains in Production

Meyer said he removed GPT-4.1 nano and gpt-oss-120b from the news route on October 7. He reported that GPT-6 Luna now handles production articles, while GPT-6 Astra’s higher cost led him to reject it for routine news writing despite its stronger test scores.

The published test does not specify a date for a further evaluation. Meyer’s results leave open whether continued monitoring or broader testing will change the model choice. For now, the stated next step is continued use of Luna in the pipeline, with the possibility of reassessing models as new configurations become available.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. Meyer reported a 3.93 average across a 12-article robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer reported that Luna scored 3.67 in the robustness run, produced valid output for every article in that run and cost about $0.0012 per article in the comparison. He said Astra cost about $0.12 per article, making it too expensive for his routine news workflow.

Did higher reasoning effort improve the articles?

Meyer said the differences among Luna’s six reasoning-effort settings were within the test’s noise range. The maximum setting cost and took about 12 times as much as the no-effort setting, without a reliable quality gain in his results.

What went wrong with the backup models?

According to Meyer, GPT-4.1 nano generated quotes that were not in the source, while gpt-oss-120b produced no valid articles in four attempts and also invented named quotes. He said both models were removed from the news route on October 7.

Do these results show which AI model is best for every newsroom?

No. The test used Meyer’s prompt, sources, languages and judging process. It offers evidence about those specific conditions, but does not establish a universal ranking for other publishers or tasks.

Source: ThorstenMeyerAI.com