Fact-Checking AI Articles: From Whack-a-Mole to a Convergent Pipeline

AI-generated ecommerce blog posts kept shipping with factual errors, and our early review process treated each one as a one-off. Fix the product spec, find a pricing error; fix the pricing, find a spec error. In one article, after four rounds of patching, two critical issues still remained. We called it whack-a-mole, and it was never converging. This is how our ShopifyAllSEO subsystem finally stopped the game.

The root cause: model memory is not a source

The pattern behind the whack-a-mole was consistent: the errors were almost always product parameters and industry standards — exactly the facts a language model has to remember, and exactly what memory gets wrong. A model that writes a battery capacity from a snapshot of the web, or quotes a certification standard it half-remembers, is not fact-checking — it is confabulating. The fix had to be structural: move verification out of the human spot-check loop and into the pipeline itself.

The v2 pipeline: three principles

• Whitelist sources for high-risk facts. Key parameters are only allowed to come from whitelisted sources. If the UL 9540 certification appears, only the official specification may state its numbers — never a number reconstructed from model memory.
• Every number must be traceable. Any number in the body that cannot be traced to a citation fails validation. In one single round of this rule, we pulled 11 untraceable numbers out of the article flow.
• Naturalize citations. A traceable number is not enough if the reference reads like a bibliography dump. Citations are woven into the prose so the fact is credible, readable, and sourced in one sentence.

Why this converges

The old loop did not converge because each fix was reactive: find a problem, patch it, ship, repeat. The v2 pipeline changes the flow at the source: restrict what can be written, mechanically verify what was written, and send only the low-risk remainder to a human queue. Each stage shrinks the space where an error can survive, so the process ends — not eventually, but by construction.

The four elements of a convergent fact-check pipeline

1. High-risk content — parameters, standards, regulations — only from whitelist sources. If the source is not whitelisted, the fact is not allowed to enter the draft.
2. Every number must have a source. A number that cannot be traced is sent back; untraceable is treated the same as wrong.
3. A self-score below threshold locks publication. The article is blocked until it scores above the bar — no editorial override on the mechanical gate.
4. The low-risk remainder goes to a human review queue. Humans review what the machine could not settle, instead of re-checking everything.

Whack-a-mole is an emotional term for a mathematical problem: the patch rate never outpaced the error discovery rate. The pipeline swapped reactive checking for source control, and the game finally ends. Building this kind of convergent fact-checking pipeline into AI content workflows is what our Shopify blog planning module at iport does, and this subsystem is where it was proven.