Optimizing Streaming Speech Recognition on Arm Edge Devices with ExecuTorch
In brief
- Nemotron 3.5, a multilingual streaming ASR model, is optimized for Arm Edge devices via ExecuTorch.
- Enabling SME2 reduces processing time by 2.12x in FP16 and 1.61x in INT8 on a Pixel 11.
- 1,120 ms audio chunks improve WER from 15.5% to 11.7% in INT8, close to FP16 performance.
PyTorch announces the optimization of a streaming automatic speech recognition (ASR) model for Arm Edge devices using ExecuTorch. According to a comparative study, the gap between batch and streaming accuracy can reach a 77% increase in word error rate (WER) for some models. The Nemotron 3.5 model, based on an RNN-T architecture, is exported to ExecuTorch and tested on a Google Pixel 11.
Tests evaluate the impact of precision (FP32, FP16, INT8) and audio chunk size (80 ms to 1,120 ms) on quality and latency. Results show that INT8 (QC8W) maintains accuracy close to FP16, with WER dropping from 15.5% to 11.7% for 1,120 ms chunks. Enabling Arm’s SME2 extension reduces processing time by 2.12x in FP16 and 1.61x in INT8, without exceeding the actual audio duration.
This optimization frees up CPU resources for other tasks between chunks while limiting thermal load. Profiling tools (ETDump, Arm Streamline) reveal that gains come mainly from accelerating matrix multiplications (GEMM) via KleidiAI microkernels, but highlight a lack of tools to directly link model layers to native functions executed.
This summary gives you the essentials. The full article, its figures and quotes are on PyTorch Blog.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error