Measuring Translation Completeness by What Users Actually See: A/B/C Buckets
Summary
A "100% translated" store was still half-English to users. Measuring visible rendered text by source buckets (A: structured fields, B: theme locale JSON, C: hardcoded/third-party) showed one unknown source — the cookie banner — was 63.9% of a product page's 29,718 visible characters.
Quick answer
Measure translation completeness on rendered page text, not field coverage. Classify every visible string into A (structured fields), B (theme locale JSON), and C (hardcoded/third-party), compute coverage per bucket, and eliminate C until the page is truly complete. In one audit, an unknown source was 63.9% of a page's visible text.
Our translation system once reported "100% translated" while the live store still showed English to customers. The gap was not translation quality — it was the metric. "Field coverage" and "what a user actually sees rendered" are two different things, and we were measuring the wrong one.
The misleading 100%
Shopify translation apps process structured fields: products, pages, blog posts. When every one of those fields shows a translated value, the system declares the store complete. But a rendered page pulls text from more places than the fields the app can reach — theme hardcoding, third-party apps, banners. Users do not read our database; they read the page.
Diagnosis: bucket the visible text
We flipped the metric around and measured the text a visitor actually sees. Every visible string on a page gets classified into one of three buckets:
- A — structured fields. Product, page, and blog content that translation apps can reach.
- B — theme locale JSON. UI copy stored in the theme's language files; translatable, but outside the product data fields.
- C — hardcoded and third-party. Text baked into the theme and content from third-party apps; by default unreachable, so it needs a dedicated channel or manual registration.
What the audit showed
On one product page, we measured 29,718 characters of visible text. The shock was the source breakdown: one source — in this case the cookie-consent banner — was flagged as "unknown" and accounted for 63.9% of the page's visible characters. Translated fields were not the problem; the invisible-to-the-translator sources were the hole.
Why this matters
A complete A and B with a broken C still leaves a half-English page. Bucketing turns "translated or not" into a per-source problem with an owner and a fix for each string. The change of metric also changed the work: once we stopped trusting the app's field-coverage number, every localization sprint started with a page grab, not a dashboard. The audit is cheap to run, and it runs on the customer's experience rather than our internal accounting.
Reusable checklist: completeness audit in 4 steps
- Capture rendered text. Scrape the page exactly as a user sees it.
- Classify by source. Route every string into A, B, or C.
- Measure per bucket. Compute coverage separately — and stop believing one global percentage.
- Eliminate C one by one. Add dedicated translation channels for hardcoded strings and third-party apps until C is empty.
When you measure by what users see, "100% translated" stops being a dashboard fantasy. Our localization tooling reports coverage exactly this way.
Frequently asked questions
Why does the system report 100% when the page still shows English?
Translation apps cover structured fields (products, pages, blogs). A rendered page also pulls text from theme hardcoding and third-party apps — sources the translator cannot reach — so field coverage and visible completeness diverge.
What exactly are the A/B/C buckets?
A is structured fields the translator can reach. B is theme locale JSON UI copy — translatable, but outside product data. C is theme hardcoded text and third-party app content — unreachable by default and needing a dedicated channel.
How do we fix bucket C?
Give hardcoded strings and third-party apps a dedicated translation channel, register them manually, or replace them with structured content — then re-run the audit until C is empty.