Zipformer: a speech encoder that runs at four frame rates

What changes relative to Conformer

Zipformer was introduced by Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin and Daniel Povey in "Zipformer: A faster and better encoder for automatic speech recognition", published at ICLR 2024. It keeps the ingredients of a Conformer, attention plus convolution plus feed-forward, and changes almost everything about how they are arranged, normalized and optimized. The paper lists five changes; each has its own section below.

Six stacks, four frame rates

A Conformer runs every block at one rate, 25 Hz after its 4x subsampling front end. Zipformer subsamples only by 2 at the front, so the first stack sees 50 Hz, and then lets each stack pick its own rate. With the default factors (1, 2, 4, 8, 4, 2) the six stacks run at 50, 25, 12.5, 6.25, 12.5 and 25 Hz. Between stacks the sequence is always back at 50 Hz; a last Downsample brings the output to 25 Hz, the same rate a Conformer would hand to its decoder.

The resampling is close to the simplest thing that works. Downsampling by a factor k averages each group of k frames with k learned weights that go through a softmax, so the stack learns, for example, whether the first or second frame of each pair matters more. Upsampling repeats each frame k times. The stack output is then combined with the stack input by a Bypass module with a learned per-channel weight c:

During training c is held in [0.9, 1.0] at first and the lower limit is relaxed to 0.2 after 20,000 steps (Section 3.2). Starting near "straight through" forces every stack to learn something before it is allowed to switch itself mostly off.

Each stack has its own width, wider in the middle where frames are cheapest. When the sequence moves to a stack with a different width it is truncated or zero-padded. The encoder output has the largest width of any stack; channels the last stack does not have are taken from the most recent stack that does. The figure below works this out for any clip length and model size.

Inside a block

A Conformer block is feed-forward, self-attention, convolution, feed-forward. Most of the cost of self-attention is in two quadratic steps: computing the attention weights and using them to aggregate frames. Zipformer splits these apart into a Multi-Head Attention Weight module (MHAW) and a Self-Attention module (SA), computes the weights once per block and uses them three times. The block is about twice as deep as a Conformer block, and the paper gives weight reuse as the main reason that depth is affordable in time and memory (Section 3.2).

Step through the order the code runs it in. Modules that consume the shared weights are marked.

Non-linear attention

NLA reuses the attention weights for something other than plain value averaging. With three linear projections A, B, C of the input, each 3/4 of the input width, the paper writes it as

where attention means multiplying along the time axis by a single head of the already computed weights. In the code that head is attn_weights[0:1], the first one. The module is a convolution module with the convolution replaced by attention-weighted mixing, and the non-linearity sits before the mixing, not after it.

What the ablations say

On Zipformer-M, LibriSpeech test-clean / test-other WER (paper Table 5): the full model is 2.21 / 4.79. Replacing each Zipformer block with two stacked Conformer-style blocks gives 2.18 / 4.95 with 73.9M parameters instead of 65.6M. Removing NLA gives 2.16 / 4.97, removing the Bypass gives 2.25 / 4.86. Removing NLA and also computing attention weights separately for each SA module gives 2.20 / 5.10: more parameters and slower, with no gain, which the authors read as the two attention patterns inside a block being similar enough to share.

BiasNorm keeps the length LayerNorm throws away

LayerNorm subtracts the mean and divides by the standard deviation, so every output vector has the same length whatever the input length was. The authors report two habits of trained Conformers that they attribute to this (Section 3.3). Some set one channel to a large constant, for example 50, which makes the denominator nearly fixed and so lets the other channels keep their size. And some modules end up "dead", with outputs around 10^-6: the LayerNorm scale $\gamma$ is pushed toward zero early in training, and if it oscillates around zero the gradient reaching the module keeps flipping sign, so it never learns.

BiasNorm builds the first habit in and removes the cause of the second:

Here b is a learned per-channel bias that only enters the RMS, and $\gamma$ is a single learned scalar. A large entry in b plays the role of the constant channel without occupying a real one. Because $\exp(\gamma)$ is always positive, the scale cannot change sign. There is no mean subtraction, which the paper calls a waste of time unless it follows a non-linearity. Zipformer uses one BiasNorm per block plus one in Conv-Embed; the per-module LayerNorms of a Conformer are gone.

SwooshR and SwooshL

Conformer uses Swish. Zipformer replaces it with two functions (Section 3.4, Equation 4):

The constant in SwooshR makes it pass through the origin. The constant in SwooshL was tuned: it slightly beat the value that would pass through the origin, so SwooshL(0) is about -0.017. The letter says which zero crossing sits at or near x = 0: the right one for SwooshR, the left one for SwooshL, which is SwooshR shifted right by about three.

The difference that matters is the slope for negative inputs. Swish and ReLU go flat, so a unit whose input stays negative stops receiving gradient, and in Adam-type optimizers its second-moment estimate shrinks toward zero, which makes later updates large. Both Swoosh functions keep a slope of -0.08 on the left. Switch the plot to derivatives to see it.

Which one goes where follows a training observation. With SwooshR everywhere, modules that sit behind a bypass connection, the feed-forward modules and the ConvNeXt layer in Conv-Embed, learned a large negative bias in the preceding linear layer: they wanted to be "normally off". SwooshL is close to zero around x = 0 and only climbs steeply past x = 4, which builds that behaviour into the activation. So SwooshL is used in the feed-forward modules and the ConvNeXt layer, SwooshR in the convolution modules and the rest of Conv-Embed. On Zipformer-M, using SwooshR everywhere costs 0.11 / 0.42 WER on test-clean / test-other, and Swish everywhere gives 2.27 / 5.37 against 2.21 / 4.79 (Table 5).

Activation constraints: Balancer and Whiten

Without per-module normalization, something else has to stop activations from drifting into bad ranges. Zipformer uses two modules that do nothing in the forward pass. In the backward pass they add the gradient of an auxiliary loss on the activations to the real gradient, scaled so it stays small next to it (Appendix A.3). The paper is candid that their placement does not follow a clean rule: they were added where a specific module was seen training badly or diverging.

Balancer

Balancer constrains each channel separately. It targets two failures: values that grow too large or too small (a risk with mixed-precision training) and "dead" channels that are almost always negative before a non-linearity. The constraints are a range for the proportion of positive values and a range for the mean absolute value. Assuming a Gaussian channel, those become limits on $\mathrm{E}[x]/\sqrt{\operatorname{Var}[x]}$ and on $\operatorname{RMS}[x]$, and the loss is the distance outside them (Equation 9). The extra gradient is normalized and multiplied by 0.04 times the magnitude of the real gradient at each element, so frames with tiny gradients, such as padding, get tiny corrections.

Whiten

The second failure is in the covariance of a whole layer's output: one or a few eigenvalues take over while the rest collapse, which the authors saw in models about to diverge. Whiten measures it with

over the eigenvalues of the mean-subtracted covariance C. It is 1 for perfectly white features and D when one direction holds everything, and the right-hand form needs no eigendecomposition. The penalty only switches on above a limit: the paper gives 10 as the usual value, and in the current code the block-output limit grows from 4 to 12 over the first 20,000 batches.

ScaledAdam: updates relative to the parameter's size

Adam normalizes the gradient, so its step on each element is roughly the learning rate whatever the parameter's magnitude. For a tensor with RMS 0.01 that is a large relative change; for one with RMS 1 it is a small one. ScaledAdam multiplies Adam's update by the tensor's current RMS r_t-1, so the relative change per step is about the same for every tensor (Section 3.5, Equation 6):

It also learns the scale explicitly. Treat each tensor as $\theta = r \cdot \theta'$ and run Adam on r as if it were a parameter, with gradient $h_t = g_t \cdot \theta_{t-1}$. That adds a step along the parameter itself, growing or shrinking the whole tensor (Equation 7):

Shrinking a tensor is hard with plain gradient steps, because the noise in every step tends to grow its norm; this term gives the optimizer a direct handle on it. It is also what lets Zipformer drop most normalization layers: a module can learn to keep its own output in a sensible range. The code adds some practical limits the paper does not spell out: the scale is updated every 4 steps from the accumulated scale gradients, each scale step is clamped to [-0.1, 0.1], tensor RMS is kept between 1e-5 and 3.0, scalar parameters get plain Adam at 0.1 times the learning rate, and gradients are clipped at twice the recent median norm (clipping_scale=2.0 in train.py).

The Eden schedule

Because ScaledAdam is less prone to divergence, it does not need a long warm-up, and its learning rates are numerically much larger than Adam's (parameter RMS values are usually well below 1). Eden decays with both the step count t and the epoch e (Equation 8):

The two factors are there so that, if you change the batch size, the amount of parameter change per hour of data stays roughly constant and the schedule does not need retuning. The paper uses base 0.045, start 0.5 and a 500-step warm-up. The icefall LibriSpeech recipe uses lr_batches = 7500, lr_epochs = 3.5 and starts the warm-up at 0.1. The epoch counter is zero during the first epoch, because train.py calls scheduler.step_epoch(epoch - 1).

The result in Table 5: with Adam, Zipformer-M needed a BiasNorm in every module to avoid divergence, and its best run reached 2.38 / 5.51 WER on test-clean / test-other against 2.21 / 4.79 with ScaledAdam. Learning-rate factors 0.025, 0.035, 0.045 and 0.055 were tried for ScaledAdam and 2.5, 5.0, 7.5 and 10.0 for Adam; the best of each is reported.

Streaming: chunks, left context, caches

The paper evaluates full-context models only, but the icefall implementation can train the same encoder for streaming. With --causal 1, each batch picks a chunk size at random from --chunk-size (default 16,32,64,-1) and a left context from --left-context-frames (default 64,128,256,-1), both counted in 50 Hz frames after Conv-Embed; -1 means no limit. One model trained this way can be run at several chunk sizes. Attention is restricted by a block mask: a frame sees every frame of its own chunk and a fixed number of whole chunks to the left, nothing to the right.

The mask is built once at 50 Hz and each downsampled stack takes every k-th row and column, so chunk boundaries line up across stacks. That is why the code asserts that the chunk size is divisible by every downsampling factor, and that the left context covers half of each stack's convolution kernel.

Convolution without lookahead

Attention is not the only thing that looks at the future. A centred convolution with kernel 31 sees 15 frames ahead. Causal Zipformer replaces the depthwise convolution with a chunk-causal one: a causal convolution of half the kernel width, plus a second convolution that only operates inside the current chunk and is multiplied by a learned scale that depends on the position within the chunk. A frame can still use the frames to its right inside its own chunk; nothing beyond the chunk is used.

What a streaming step costs and carries

At inference the encoder is called once per chunk. Each call takes $T = 2 \cdot \text{chunk} + 13$ feature frames: twice the chunk, because Conv-Embed halves the rate, plus 7 frames consumed by its convolutions and 6 for the 3 frames of right padding the ConvNeXt layer needs at 50 Hz. It returns chunk / 2 output frames at 25 Hz. Between calls, every layer keeps six tensors: cached keys, cached NLA inputs, cached values for both SA modules, and the left context of both convolution modules. Keys and values hold left_context_frames // k frames for a stack with factor k. The ONNX export adds two more inputs: a Conv-Embed cache and a count of frames already processed.

The trade-off is the usual one. Smaller chunks mean lower latency and less right context per frame; longer left context means more state and more attention work per step. The paper does not cover streaming at all; the causal mode lives in the icefall code, and the accuracy cost of a given chunk size is something to measure on your own data.

What the paper reports

All numbers in this section are copied from the paper (arXiv 2310.11230v4). Models are trained with the pruned transducer loss unless stated, decoded with beam search of size 4 and no external language model. In the authors' words: "In terms of efficiency, Zipformer converges faster during training and speeds up the inference by more than 50% compared to previous studies while requiring less GPU memory." (Section 1)

The paper is also clear about a baseline problem. Its own pruned-transducer Conformers do not reach the numbers in the original Conformer paper (2.1 / 4.3 for Conformer-L), and neither do the WeNet and ESPnet Conformers it lists. The Zipformer-L trained for longer, in the last row of Table 2, reaches 2.00 / 4.38, which the authors describe as "the first model to approach Conformer-L to the best of our knowledge."

Zipformer also trains with other heads. In Table 8, CTC-only Zipformer-S/M/L reach 2.85 / 6.91, 2.51 / 6.02 and 2.49 / 5.7, and CTC/AED versions with a 6-layer Transformer decoder reach 2.46 / 6.04, 2.22 / 4.97 and 2.09 / 4.59.

Mandarin

Running it: ONNX and sherpa-onnx

Training lives in icefall; deployment usually goes through k2-fsa/sherpa-onnx, a runtime for ONNX speech models with APIs for C++, C, Python, JavaScript, Java, C#, Kotlin, Swift, Go, Dart, Rust and Pascal, builds for Android, iOS, Windows, macOS, Linux and HarmonyOS, and a WebAssembly target. A streaming Zipformer is exported with a fixed chunk size and left context, and those numbers are baked into the graph.

Export a streaming transducer

The architecture flags must match training. Chunk size and left context are picked from the values the model was trained with (from 16,32,64 and 64,128,256 for the default recipe):

# in icefall/egs/librispeech/ASR, with $repo holding exp/epoch-99.pt
./zipformer/export-onnx-streaming.py \
  --tokens $repo/data/lang_bpe_500/tokens.txt \
  --use-averaged-model 0 --epoch 99 --avg 1 \
  --exp-dir $repo/exp \
  --num-encoder-layers "2,2,3,4,3,2" \
  --downsampling-factor "1,2,4,8,4,2" \
  --feedforward-dim "512,768,1024,1536,1024,768" \
  --num-heads "4,4,4,8,4,4" \
  --encoder-dim "192,256,384,512,384,256" \
  --query-head-dim 32 --value-head-dim 12 \
  --pos-head-dim 4 --pos-dim 48 \
  --encoder-unmasked-dim "192,192,256,256,256,192" \
  --cnn-module-kernel "31,31,15,15,15,31" \
  --decoder-dim 512 --joiner-dim 512 \
  --causal True --chunk-size 16 --left-context-frames 128

This writes an encoder, a decoder and a joiner. The encoder carries its streaming contract as ONNX metadata: model_type=zipformer2, decode_chunk_len (twice the chunk, in feature frames), T (decode_chunk_len + 13), left_context_len, and the per-stack layers, widths, kernels and head sizes. A runtime reads these instead of hard-coding them; check them first when a port produces garbage.

Decode a stream

import numpy as np, sherpa_onnx, soundfile as sf

rec = sherpa_onnx.OnlineRecognizer.from_transducer(
    tokens="tokens.txt",
    encoder="encoder-epoch-99-avg-1-chunk-16-left-128.onnx",
    decoder="decoder-epoch-99-avg-1-chunk-16-left-128.onnx",
    joiner="joiner-epoch-99-avg-1-chunk-16-left-128.onnx",
    num_threads=2, sample_rate=16000, feature_dim=80,
    decoding_method="greedy_search",
)
samples, sr = sf.read("test.wav", dtype="float32")
s = rec.create_stream()
s.accept_waveform(sr, samples)           # call repeatedly with live audio
s.accept_waveform(sr, np.zeros(int(0.66 * sr), dtype=np.float32))  # tail padding, as in the official example
s.input_finished()
while rec.is_ready(s):
    rec.decode_stream(s)
print(rec.get_result(s))

A CTC-only Zipformer uses OnlineRecognizer.from_zipformer2_ctc(tokens=..., model=...) instead, with one ONNX file. sherpa-onnx computes the 80-dimensional fbank features itself from 16 kHz audio, so the front end matches what icefall trained on.

On a device

• Memory per stream is the cache state from the streaming section, about 1.7 MiB in fp32 for Zipformer-M with 128 frames of left context, plus the activations of one chunk.
• Every encoder call takes exactly 2 x chunk + 13 feature frames and fixed-size caches, so the graph has fixed shapes. That suits accelerators that compile a static graph, and it means one export per chunk size.
• Features are Kaldi-style fbank (25 ms window, 10 ms shift, 80 bins). A different mel front end is a different input distribution; reproduce the training features exactly.

Case study: a streaming phoneme recognizer for Qur'an recitation

Quran-Lab/zipformer_p-arabic-v3 is Quran Lab's Zipformer for recitation. Everything in this section comes from its public model card. The current release is v3.1, a fine-tune of v3 that retunes madd (elongation) length behaviour; it keeps the same architecture, the same streaming interface and the same token table, so it is a drop-in replacement for v3 in every format.

The card describes a 65.5M-parameter streaming phoneme recognizer for Hafs 'an 'Asim, built on Zipformer2 with a CTC head. It transcribes recitation into a 251-symbol Qur'anic phonetic alphabet that encodes madd length, gemination, ghunna, ikhfaa, qalqalah and the emphatic consonants. Its design rule is no language model anywhere: no transducer, no LM rescoring, no text prior, because a language model silently corrects a reciter's mistakes, which an assessment system must never do. zipformer_p-arabic-v3 is ranked #1 on the open Quranic ASR leaderboard (33 entries on 3 November 2026).

How the architecture shows up in the release

• The streaming contract. The ONNX files are standard sherpa-onnx streaming-CTC zipformer2 exports with a chunk of 48 input frames, and the CoreML metadata gives T=61, decode_chunk_len=48. That is the formula from the streaming section: 48 = 2 x 24, so a chunk of 24 frames at 50 Hz, 0.48 s, and $T = 2 \cdot 24 + 13 = 61$.
• Trained chunk sizes. The card says the model was trained only at chunk sizes 8, 16 and 24, and that the best chunk size depends on the domain.
• The front end. Input is 16 kHz mono with 80-dimensional Kaldi fbank (povey window). The card is explicit that a mel-spectrogram front end is not a substitute.
• Swoosh in fp16. The icefall Swoosh code guards overflow by checking for inf. The card notes that fp16 hardware on the Apple Neural Engine saturates at 65,504 instead of producing inf, which silently corrupts an inf-compare guard; the CoreML graph uses a threshold compare instead, a max-subtracted log-softmax head, and no fp32 epsilon parameters.
• Splitting the encoder. The encoder is too large for one Neural Engine program, so the CoreML packages split it into two functions, front and back, sharing one weight file. The cut is at the narrowest point of the network, where a single 24 x 1 x 384 tensor crosses.
• Token table. The CTC blank is id 250, not 0; decoding must use the shipped tokens.txt.

The card also lists measured limitations, for example that a model trained almost entirely on correct recitation with canonical labels tends to transcribe the rule rather than a reciter's deviation from it. Read the card before building on it; the model is released under the Quran-Lab No-Profit License 1.2 with gated access.