WhisperKit vs whisper.cpp vs MLX: how to choose a local Whisper stack on Apple Silicon

Pick the stack by deployment shape, not by a speed table. If you are shipping a Swift app on Apple platforms, start with WhisperKit. If you need a portable engine for CPUs, servers or embedded devices, start with whisper.cpp. If you are experimenting in Python on Apple Silicon, start with MLX Whisper. The three are built for different ways of delivering Whisper, and there is no controlled public benchmark that ranks them head to head. Measure the candidates that fit your product on the oldest Mac you plan to support.

Three stacks, three deployment shapes

WhisperKit is a Swift package from Argmax that packages OpenAI Whisper as Core ML models. Building the current package requires macOS 14 or later and Xcode 16. It downloads and compiles models for you, and can run model components on the Neural Engine, GPU or CPU. Argmax recommends the compressed large-v3 turbo variant (large-v3-v20240930_626MB) for accuracy across Apple platforms and the uncompressed turbo (large-v3-v20240930_turbo) for maximum speed on macOS. ^[1]

whisper.cpp is a plain C/C++ implementation of Whisper with no framework dependencies. It treats Apple Silicon as a first-class target through ARM NEON, Accelerate, Metal and Core ML, but it also supports AVX x86 CPUs, Vulkan, NVIDIA, AMD, OpenVINO, WebAssembly, Android, Linux, Windows and Raspberry Pi. It runs CPU-only and supports integer quantisation. ^[4]

MLX began as Apple's machine-learning array framework for Apple Silicon. On the Mac it offers a NumPy-like Python API, unified memory and CPU/GPU execution. The framework now also publishes Linux CPU and CUDA packages. The mlx-whisper Python package loads MLX-format Whisper checkpoints from the mlx-community collection on Hugging Face. ^{[6] [7]}

Those three sentences are the whole comparison in miniature: an app SDK, a portable engine, and a research framework. Each one optimises for a different thing.

The benchmark gap is real, so there is no winner table

The projects publish different kinds of evidence, and none of it is a controlled cross-stack comparison. Argmax ships a benchmark harness that runs WhisperKit across connected devices and uploads results to a public Hugging Face space; it is designed to compare models and devices within WhisperKit, not against other engines. ^[2] whisper.cpp documents disk and memory figures and claims its Core ML encoder path can be more than three times faster than its own CPU-only execution. ^[4]

Community reports point in different directions. One Mac app developer says WhisperKit is "close to instant" for tap-to-speak on Apple Silicon. ^[10] Another thread describes whisper.cpp as "blazing fast" compared with OpenAI's reference implementation on CPU machines. ^[9] Neither report compares all three stacks under shared conditions.

None of these is a measurement you can rely on for your project, because they differ in audio, model, precision, backend and machine. A developer who ships a dictation app put it more bluntly: most of the delay budget gets eaten before and after the model, and cold starts on local Whisper variants are worse than benchmarks suggest. ^[11] Throughput on a warm file says little about the experience of pressing a hotkey, speaking and waiting for text.

WhisperKit is the Swift-app path

If your product is a native Mac or iOS app, WhisperKit removes the hardest parts of shipping Whisper: model download, Core ML compilation and a Swift-first API. Models come from Argmax's Hugging Face catalogue, with a CLI (argmax-cli, also on Homebrew) and a local server that implements the OpenAI audio API for transcription and translation. ^[1]

The trade-off is platform reach in the reviewed Swift package. Argmax's maintainer described whisper.cpp as the better fit when broad platform support is the main constraint, while Argmax targeted one platform at a time. ^[3] That 2024 answer predates Argmax's current paid Android SDK, so it should not be read as a permanent company-wide limit. You also accept Argmax's model pipeline, or run your own conversions through whisperkittools if you want a fine-tuned model in Core ML format. ^[1]

whisper.cpp is the portable engine

whisper.cpp's strengths are portability and control. It is a dependency-free C/C++ library with a C-style API, zero runtime allocations, CPU-only support and quantised models that shrink both disk and memory use. The ggml model files are explicit about their size: large-v3 is 2.9 GiB, its q5_0 quantisation 1.1 GiB, large-v3-turbo 1.5 GiB and turbo q5_0 547 MiB. ^[5] The project ships a CLI, a benchmark tool, a streaming example, a server with an OpenAI-like API, and WebAssembly builds. ^[4]

The cost is that you operate it. You build the engine, download or convert model files, and handle input conversion yourself; the CLI example expects 16-bit WAV files. On Apple Silicon the Core ML path accelerates the encoder on the Neural Engine, while the rest of the pipeline runs elsewhere, so tuning still matters. ^[4]

MLX Whisper is the research and iteration path

For a Python user, MLX Whisper has the shortest setup shown in these projects' current documentation. Install mlx-whisper with pip, run mlx_whisper audio.mp3 or call the Python API, and it will pull an MLX checkpoint from Hugging Face on demand. The conversion script in mlx-examples supports 4-bit quantisation, and mlx-community/whisper-large-v3-turbo is listed as a quantised MLX model at 1.61 GB. ^{[7] [8]}

The limits matter. mlx-whisper is a Python package. Shipping that implementation inside a Swift app means embedding Python or running a separate process; MLX Swift is a separate API, not a drop-in bridge to the Python Whisper package. MLX documents CPU and GPU execution but no Neural Engine path. On Apple Silicon its arrays share unified memory, so a large checkpoint competes directly with the rest of the system for RAM. ^[6] Linux support in the wider framework does not prove that every mlx-whisper workflow is portable unchanged.

A decision rule, not a single winner

Ask four questions in order.

1. Where must this run? Apple-only and a native app points to WhisperKit. Any mix of Linux, Windows, WebAssembly, servers or embedded devices points to whisper.cpp. A Mac-only Python tool points to MLX.
2. What language is your product? Swift and Xcode point to WhisperKit. C, C++ or any language with a C binding points to whisper.cpp. Python points to MLX.
3. Who manages the models? If you want downloads and compilation handled, WhisperKit. If you are happy managing files, builds and quantisation, whisper.cpp. If you want to convert, quantise and iterate on checkpoints yourself, MLX.
4. Does the Neural Engine matter? Core ML paths (WhisperKit, or whisper.cpp built with Core ML) can use it. MLX does not document an ANE path; whisper.cpp can run without one.

Run the same test on each stack

Once the deployment shape narrows the field, replace forum opinions with a measurement:

1. Record one clean 60-second clip and one noisy 60-second clip. Freeze both files and write a reference transcript before looking at engine output.
2. Use the same Whisper family variant, language and decoding settings where equivalent controls exist. Record the exact checkpoint, precision and quantisation. Core ML, ggml and MLX conversions are not bit-identical, so document mismatches rather than calling the run controlled.
3. Measure cold start once, then run at least five warm repetitions per clip. Time from invocation to the final transcript, and report the median plus the fastest and slowest warm result.
4. Measure peak resident memory with the same tool for every process. Count substitutions or omissions that change meaning against the frozen transcript; keep punctuation-only differences separate.
5. Repeat on the oldest Mac you intend to support. A result from an M4 Max does not establish the memory or latency budget on an M1 Air.

Keep your normal apps open, because dictation rarely runs on an idle Mac. Model size, precision (q5_0 or 4-bit versus fp16), backend, chunking and voice-activity detection all change the result. The projects give you the tools: whisper.cpp ships whisper-bench, ^[4] WhisperKit has its benchmark harness, ^[2] and for MLX you can time the transcribe call directly. This remains a practical engineering comparison rather than a laboratory benchmark, but it produces evidence you can reproduce and challenge.

Keep transcription and clean-up separate

ShoutFlow's public product description says its local path uses WhisperKit for speech-to-text and an MLX language model for clean-up. ^[12] The second component is not MLX Whisper. It performs a different job after transcription, so it cannot show that ShoutFlow combines two Whisper engines or that one transcription stack beats another.

That distinction catches an easy comparison error. "Uses MLX" does not necessarily mean "uses MLX Whisper". Before copying an architecture or benchmark, trace which model handles audio and which handles the text that comes afterwards.

Start with deployment shape, then run the test with your own audio. Keep the raw outputs and measurements beside the decision so the next model or runtime update can be tested against the same baseline.