BlogTechnical
Technical8 min read

Live Translation of an iPhone Screen, On the Laptop

On-device OCR and translation, a 10-second screen that became 0.7 seconds, and why menus need a glossary.

AS

Anuj Sehgal · CEO, DocQ

November 16, 2026

Live Translation of an iPhone Screen, On the Laptop

This is the long version of the translation layer in the main story. The goal: the phone on the left, and on the right the same screen with English painted over the Chinese, updating as I scroll. Everything runs on the Mac. Nothing is sent to a translation service.

The pipeline

Screenshot, OCR, translate, paint

WebDriverAgent screenshotApple Vision OCRglossary + rulesApple Translationlabels on the page
  • A small Swift tool reads image paths on stdin. For each one it runs Apple's Vision text recognition, keeps only lines with Chinese in them, and prints JSON: the original text, the English, and a bounding box from 0 to 1.
  • Apple's Translation framework does Chinese to English on-device, with the model downloaded once in macOS settings.
  • A Python server grabs a screenshot from WebDriverAgent about once a second, pipes it to the Swift tool, applies the menu rules, and serves a page on localhost:8200: live video on the left (WebDriverAgent's MJPEG stream on port 9100), the translated frame on the right.
let req = VNRecognizeTextRequest()
req.recognitionLevel = .accurate
req.recognitionLanguages = ["zh-Hans", "en-US"]
req.usesLanguageCorrection = false

let session = TranslationSession(
  installedSource: Locale.Language(identifier: "zh-Hans"),
  target: Locale.Language(identifier: "en"))

Speed

From 10 seconds to 0.7

10 s

per screen, version 1

0.7 s

per screen, version 2

0

bytes sent to a translation service

Version 1 was correct and useless. It translated every string on the screen before showing anything, so each new screen took about ten seconds. By then I'd scrolled past it.

Version 2 splits the work in two. The main loop runs OCR on each frame and answers immediately with whatever English is already in the cache. A background worker translates new strings four at a time and fills the cache. The first frame of a new screen shows a few blanks, and they fill in a moment later. Menus repeat a lot of words, so the cache warms up fast.

// Background worker: translates a few strings at a time
// so frames are never held up.
let batch = Array(wanted.filter { cache[$0] == nil }.prefix(4))
let rs = try? await session.translations(
  from: batch.map { TranslationSession.Request(sourceText: $0) })

Menus are not sentences

The glossary and the rules

A general translator is great at sentences and terrible at menus. Some of my favourites:

ChineseTranslator saidReally means
拼好饭"Cook the meal."Group-deal meals
买贵必赔"If you buy it at a high price, you will be compensated."Price-match guarantee
起送 ¥25"Free delivery from Yuanqi"Minimum order ¥25
惠"favour"Deal
a noodle shop's name"Uncle Rice"A perfectly normal name

So there are two layers in front of the translator. A glossary of about 60 app words (首页 Home, 选规格 Choose options, 加入购物车 Add to cart, 免配送费 Free delivery), and a dozen regex rules for the patterns that carry numbers:

RULES = [
  (r"起送\s*¥?\s*([\d.]+)",  r"Min. order ¥\1"),
  (r"配送\s*约?\s*¥\s*([\d.]+)", r"Delivery ~¥\1"),
  (r"月售\s*(\d+\+?)",        r"\1 sold/month"),
  (r"(\d+(?:-\d+)?)\s*分钟",   r"\1 min"),
  (r"([\d.]+)\s*分",          r"\1 rating"),
  (r"满\s*([\d.]+)\s*减\s*([\d.]+)", r"¥\2 off over ¥\1"),
]

The rules only win if they cover all the Chinese in a line. Otherwise the line goes to the general translator. "4.5分" becomes "4.5 rating" instead of "4.5 points", and "满30减5" becomes "¥5 off over ¥30", which is what you actually need to know at 2 a.m.

A small safety net

Red labels on money buttons

Any line containing a payment or order word (支付, 提交订单, 去结算, 下单, 付款, 购买, 确认订单) gets a red label instead of a white one. It's for me as much as for the agent: when the screen is a wall of coupons and countdowns, the button that spends money should be impossible to miss.

Honest limits

What it still gets wrong

  • Shop names get translated literally. Hence "Uncle Rice".
  • Tiny text on promo badges sometimes isn't recognised at all.
  • It's a picture, not the app. The overlay is on a screenshot, so it lags the live video by about a second.
  • The Swift tool crashed once (a broken pipe), so the server now restarts it automatically.

For a menu at 2 a.m., none of that mattered. It was the difference between guessing and reading.

← Back to the story

Back to the story: I Can't Read Chinese, So I Gave My iPhone to an AI and It Ordered Dinner

Anuj Sehgal · CEO, DocQ

He builds AI agents for documents and business processes, and occasionally for dinner. Say hello on LinkedIn.

AI agentstranslationApple VisionOCRautomation

Build. Automate. Govern.Accelerate Intelligence. Accelerate People.

One platform to structure your data, automate your processes, and free your people — with AI baked in.

Every manual step eliminated is a compounding speed advantage. What are you still doing manually that DocQ could handle instantly?