I work at a Chinese company. Every standup, every design review, every Friday demo — Mandarin. My ear is getting better, but at meeting speed it isn't fast enough yet, and pulling out a phone translator means I miss the next sentence to catch the last one.

So I built one. A native macOS app that listens to system audio, transcribes the Mandarin on-device, drops a pinyin line underneath, and shows the English translation — all without sending a byte to a cloud. Here's the architecture, why every "obvious" cloud-based shortcut fails for live meetings, and the engineering decisions that mattered.

Why Cloud Translators Fail at Meeting Speed

Google Translate, DeepL, Apple's Live Captions — all decent for static text or solo speech. None of them survive a real meeting. The reason is latency budget, not accuracy.

Three to six seconds doesn't sound like much. In a meeting full of "对，对，那个不行," it's two thoughts behind. By the time the translation lands, the speaker has moved on and the cursor in your brain is still on the previous sentence. You don't fall behind by one beat — you fall behind compounding.

On-device ASR + on-device translation can run the same pipeline in 200–500ms. That's the difference between following a meeting and watching one happen to you.

The 4 Numbers That Matter

The Pipeline (4 Stages, All Local)

Every stage runs on the M-series Neural Engine or CPU. No outbound network. The trade-off is model size on disk and a one-time download — in return you get latency a cloud service cannot match.

Pinyin in One Line (No, Really)

The detail I want to highlight, because nobody told me it existed: Apple gives you Mandarin → pinyin for free. Built into Foundation. Has been there for years.

If you want pinyin without tone marks (cleaner for some readers), chain a second transform with kCFStringTransformStripDiacritics. That's the whole pinyin layer. The other 95% of the work was audio capture and UI.

Decision Table — Stack Choices I Made

Every stage had a cloud option and a local option. Here's where I landed and why.

5 Things That Bit Me

1. Gotcha 01Partial vs final ASR results. SFSpeechRecognizer emits partial transcripts that get rewritten. Don't translate every partial — debounce by ~400ms or wait for isFinal. Translating mid-sentence is what makes apps feel broken.
2. Gotcha 02Audio routing changes break the stream. Plug in AirPods mid-meeting and ScreenCaptureKit can drop the system tap. Listen for route changes and restart the capture session.
3. Gotcha 03Mandarin segmentation is hard. ASR gives you a wall of characters with no spaces. For pinyin readability you want word-level segmentation — but CFStringTransform doesn't know words. Either accept char-level pinyin or layer in a segmenter (Jieba-style).
4. Gotcha 04Translation pack must be pre-downloaded. First run, TranslationSession can prompt the user. Trigger the download proactively at first launch, otherwise mid-meeting download = mid-meeting outage.
5. Gotcha 05Battery on laptops. Continuous ASR + translation + an always-on-top SwiftUI window eats power. On a MacBook on battery, I throttle the ASR sample rate and translate every Nth final segment instead of every one.

Why I'm Sharing This

Two reasons. First, "build it instead of buying it" is the right move more often than people admit — especially for tools you'll touch ten times a day. The cloud translators charge per minute, leak audio to servers, and lag enough to be useless for the actual problem.

Second, the macOS stack is quietly absurd. ScreenCaptureKit, SFSpeechRecognizer, CFStringTransform, TranslationSession — four Apple APIs and you've replaced a paid SaaS. The hard part wasn't AI. It was knowing which Foundation transform to call.

Frequently Asked Questions

Working at a Chinese company? Stop missing meetings. Read the full build.

Read The Full Walkthrough On Medium →