deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

On-device Whisper dropped from shipping iPhone transcription app over cold starts and memory

A developer explains why WhisperKit was pulled from a shipping iOS transcription app: cold-start compiles of up to 205 seconds, 799 MB memory peaks and a 490 MB model bundle pushed it to Apple's SpeechTranscriber.

On-device Whisper dropped from shipping iPhone transcription app over cold starts and memory

The App Store build of Koedori, an iPhone app that turns recordings into text entirely on the device, contains no Whisper at all. In a first-hand account on dev.to, the app's developer at W Lab explains that early builds ran Whisper through WhisperKit, while the shipped version uses Apple's SpeechAnalyzer framework and its SpeechTranscriber module, and sets out what forced the change.

The constraints that ruled Whisper out

Koedori rests on three commitments, according to the post: transcription is free and needs no account, audio never leaves the phone, and re-transcribing costs nothing however often it runs. The privacy guarantee is architectural rather than a toggle, since there is no code path that uploads audio and no server of the app's own, so recognition must happen locally and cheaply. Whisper fit those constraints in principle; the friction came from the hardware around it.

The measurements that ended it

WhisperKit compiles a model for Apple's Neural Engine the first time it loads. On the developer's iPhone 14, the first load of large-v3 turbo after an install took 197 to 205 seconds, every time; a reload within the same install took 4 seconds, after which a 10-second clip transcribed in 0.9 seconds. Every Xcode run counts as a fresh install, so development absorbed the wait on each build, and a pass started mid-compile showed nothing for minutes, forcing a visible preparation step on screen.

Memory followed. In one run without WhisperKit's prewarm setting, the app failed to reach the compiler's helper, fell back to the CPU and peaked at 799 MB; with prewarm enabled, the same load peaked at 193 MB. Older devices fared worse, with the encoder returning silence on the CPU on some older chips, and one test unit never finishing optimization of Whisper small. The app would also have carried the model itself: Whisper small is about 490 MB bundled, plus download code for the other variants.

What Apple's framework gave and took

SpeechTranscriber brought tradeoffs. Because it is new, older iOS versions were dropped rather than kept on Whisper as a fallback, and it covers roughly ten languages, including English, Japanese, Chinese, Korean, Spanish, French, German, Italian and Portuguese, against Whisper's hundred. In return the app shed the bundled model, the model picker, the download logic and the compile wait; iOS itself fetches a language's model once, through AssetInventory.

Notes from the implementation

Each note is made by a single pass over the finished recording. The option that matters most, the developer writes, is .audioTimeRange: a result spans roughly 20 seconds of audio, too coarse to tap, but each attributed run inside it carries its own time range, down to one character for Japanese, so a clipped line keeps a real start time and tapping any line plays audio from that moment. The analyzeSequence API also converts the file itself, and the analyzer must be explicitly finished even when no audio arrives, or the results stream never terminates.

One API trap deserves flagging: AssetInventory's status check reported a Mac's installed Japanese model as merely supported until the locale was reserved, and reserving is a side effect capped at five locales, which makes it a poor fit for a readiness probe. The app instead consults SpeechTranscriber's installed locales and compares BCP-47 identifiers. If a model is missing when recording stops, the audio is still saved, marked as waiting to transcribe, and retried whenever the app returns to the foreground.

Live text is a preview, not the record

In the while-recording mode, microphone buffers are converted and fed to a second analyzer alongside the file writer. On a Mac, volatile results alone lagged 11.7 seconds behind speech; adding fast results cut that under a second. But the streamed text never matched the file pass exactly: across 10 minutes, about 1.5 spots per minute, roughly 0.7 percent of characters, differed at the same places on every run, with neither side more accurate. The live box is therefore display-only, and the saved note always comes from the file pass, which never enables the speed-oriented option.

Corrections work the same way, as a text layer. Notes store the engine's output verbatim plus a snapshot of the user's dictionary, with replacements applied at display, search and export time, which is why retroactive fixes are instant and still work on notes whose audio has been deleted. The API's contextualStrings hint appeared inert in the developer's testing: a registered economics term came out wrong anyway, in both the transcript and the alternatives.

Why it matters

This is a grounded data point for anyone weighing a bundled model against a platform API at the edge. Whisper was not dropped for accuracy or steady-state speed; warm, it beat real time. It was dropped for cold-start compilation, memory swings, chip-to-chip inconsistency and a half-gigabyte payload, none of which a typical benchmark surfaces. The framework switch traded language breadth and OS reach for effectively zero integration weight. The broader edge-AI lesson: the model is often the easiest part of the deployment decision.

  • #whisper
  • #ios
  • #speech-recognition
  • #on-device-ai
  • #apple

Related posts