2024 MacBook Pro, 48GB Ram, M4 Pro, Tahoe 26.0

https://huggingface.co/FluidInference/parakeet-tdt-0.6b-v3-coreml

swift run fluidaudiocli asr-benchmark --max-files all --model-version v2

Use v2 if you only need English, it is a bit more accurate

Core ML first-load compile times captured on iPhone 16 Pro Max and iPhone 13 running the parakeet-tdt-0.6b-v3-coreml bundle. Cold-start compilation happens the first time each Core ML model is loaded; subsequent loads hit the cached binaries. Warm compile metrics were collected only on the iPhone 16 Pro Max run, and only for models that were reloaded during the session.

CTC-based custom vocabulary boosting system, which enables accurate recognition of domain-specific terms (company names, technical jargon, proper nouns) without retraining the ASR model.

In context of vocabulary/keyword detection:

Derived metrics:

We generated the same strings with to generate audio between 1s to ~300s in order to test the speed across a range of varying inputs on Pytorch CPU, MPS, and MLX pipeline, and compared it against the native Swift version with Core ML models.

Each pipeline warmed up the models by running through it once with pesudo inputs, and then comparing the raw inference time with the model already loaded. You can see that for the Core ML model, we traded lower memory and very slightly faster inference for longer initial warm-up.

Note that the Pytorch kokoro model in Pytorch has a memory leak issue: hexgrad/kokoro#152

The following tests were ran on M4 Pro, 48GB RAM, Macbook Pro. If you have another device, please do try replicating it as well!

I wasn't able to run the MPS model for longer durations, even with PYTORCH_ENABLE_MPS_FALLBACK=1 enabled, it kept crashing for the longer strings.

Note that it does take ~15s to compile the model on the first run, subsequent runs are shorter, we expect ~2s to load.

Model is nearly identical to the base model in terms of quality, performance wise we see an up to ~3.5x improvement compared to the silero Pytorch VAD model with the 256ms batch model (8 chunks of 32ms)

Dataset: https://github.com/Lab41/VOiCES-subset

Encoder-decoder ASR using Qwen3-ASR-0.6B converted to CoreML. Autoregressive generation with KV-cache.

Note: WER/CER may be higher than the original PyTorch model due to CoreML conversion limitations. See FLEURS results below for full multilingual benchmarks.

Model: FluidInference/qwen3-asr-0.6b-coreml (f32 variant)

Hardware: Apple M2, 2022, macOS 26

Methodology notes:

• CER (Character Error Rate) is the primary metric for Chinese ASR, as per the Qwen3-ASR Technical Report: "We use CER for character-based languages (e.g., Mandarin Chinese, Cantonese, and Korean) and WER for word-delimited languages"
• WER calculation uses Apple's NLTokenizer for Chinese word segmentation; we were unable to verify how official Qwen3-ASR evaluation performs tokenization
• Official Qwen3-ASR reports 3.15% on AISHELL-2 (different dataset) per HuggingFace model card; our 6.6% CER on AISHELL-1 suggests some accuracy loss in CoreML conversion
• Why AISHELL-1? AISHELL-2 (1000h) requires an application with institutional affiliation and is restricted to non-commercial use. AISHELL-1 (178h) is openly available under Apache 2.0.
• Dataset: AudioLLMs/aishell_1_zh_test

Full benchmark across all 30 languages supported by Qwen3-ASR, matching the official FLEURS tiers.

Which metric to use:

• CER for character-based languages (Chinese, Japanese, Korean, Thai, Vietnamese, Cantonese) - WER is meaningless due to word segmentation differences
• WER for word-delimited languages (European, Arabic, etc.)

Note: Official Qwen3-ASR reports WER, but for CJK languages this includes word segmentation artifacts. Our CER comparison shows CoreML conversion has minimal accuracy loss on core languages.

*WER >90% is expected for CJK/Thai due to word segmentation - FLEURS references have artificial character-by-character spacing while our output is natural continuous text. CER shows actual transcription quality.

Real-time streaming ASR with End-of-Utterance detection using the Parakeet EOU 120M CoreML model.

Model: FluidInference/parakeet-realtime-eou-120m-coreml

Hardware: Apple M2, 2022, macOS 26

The offline version uses the community-1 model, the online version uses the legacy speaker-diarization-3.1 model.

For slightly ~1.2% worse DER we default to a higher step ratio segmentation duration than the baseline community-1 pipeline. This allows us to get nearly ~2x the speed (as expected because we're processing 1/2 of the embeddings). For highly critical use cases, one may should use step ratio = 0.1 and minSegmentDurationSeconds = 0.0

Running on the full voxconverse benchmark:

Note that the baseline pytorch version is ~11% DER, we lost some precision dropping down to fp16 precision in order to run most of the embedding model on neural engine. But as a result, we significantly out perform the baseline mps backend as well. the pyannote-community-1 on cpu is ~1.5-2 RTFx, on mps, it's ~20-25 RTFx.

This is more tricky and honestly a lot more fragile to clustering. Expect +10-15% worse DER for the streaming implementation. Only use this when you critically need realtime streaming speaker diarization. In most cases, offline is more than enough for most applications.

Running a near real-time diarization benchmark for 3s chunks, 1s overlap, and 0.85 clustering threshold:

Diarization benchmark with 10s chunks, 0s overlap, and 0.7 clustering threshold:

Diarization benchmark with 5s chunks, 0s overlap, and 0.8 clustering threshold (best configuration found):

Diarization benchmark with 5s chunks, 2s overlap, and 0.8 clustering threshold:

NVIDIA's Sortformer model for streaming speaker diarization, converted to CoreML.

Model: FluidInference/diar-streaming-sortformer-coreml (V2 models for macOS 26+ compatibility)

Hardware: Apple M2, 2022, macOS 26.1