Lab 9.3 — STFT Overlap-Add Audio Effects

← Course 2 syllabus · Bonus Module 9 · Prev: « Lab 9.2 · Next: Lab 9.4 »

Goal

Move from time-domain filtering to block-based frequency-domain processing on the streamed WAV. You’ll build a real short-time Fourier transform (STFT) pipeline on the device: window each block, forward-FFT with CMSIS-DSP arm_rfft_fast_f32, apply a spectral effect (a noise gate / spectral subtraction / fast-convolution reverb / robotization), inverse-FFT, and overlap-add (OLA) the windowed frames back into a continuous stream that goes home to effected.wav. The verification step is what makes this rigorous: with the effect set to identity, correct OLA must reconstruct the input to within FFT round-off — so you first prove your OLA framework is transparent against a NumPy reference, and only then trust the effected output. This is the standard structure of real-time spectral audio firmware (denoisers, vocoders, convolution reverbs), and the identity-reconstruction test is how you know your windowing and hop are consistent.

Recommended reading

• O-DSP Ch. 8–9 — the DFT/FFT and, crucially, block convolution via overlap-add / overlap-save; the STFT as a filter bank. The theoretical backbone of this lab.
• Lyons Ch. 3–5 — the practitioner’s DFT/FFT, windowing and spectral leakage, and the FFT of real signals (rfft).
• Course 1 Week 9 (../course1/index.qmd#week-9) and Week 10 (../course1/index.qmd#week-10) — complex analysis / the DTFT and the Fourier transform underpinning the STFT.
• CMSIS-DSP documentation for arm_rfft_fast_f32 (real FFT, in-place, packed spectrum layout).

Equipment & parts

• STM32 Nucleo-L476RG (small FFT sizes, \(N=256\)–\(512\)) or the Raspberry Pi 5 (larger \(N\), heavier effects) — the lab runs on either; see the tradeoff note in the Background.
• CMSIS-DSP (arm_rfft_fast_f32, arm_cmplx_mag_f32) on the STM32; NumPy on the Pi 5.
• Host Mac with numpy scipy for the reference OLA and audio checks.
• A test WAV: for a noise gate / spectral subtraction, a recording with audible background noise (fan, hiss) makes the effect obvious; for reverb, a dry vocal or clap.

Safety & don’t-break-it

Data lab; the failure modes are windowing/indexing bugs and real-time budget overruns:

• COLA violation = amplitude modulation. If your window and hop don’t satisfy the constant-overlap-add condition, the identity effect won’t reconstruct flat — you’ll hear a periodic tremolo at the frame rate. Verify COLA numerically before blaming the effect.
• In-place FFT buffer aliasing. arm_rfft_fast_f32 uses a specific packed real-spectrum layout and an in-place scratch buffer. Reusing the same buffer for input and output without respecting the layout corrupts bins silently. Follow the CMSIS buffer contract exactly.
• FFT-size / SRAM budget. Float FFT of size \(N\) needs \(\sim N\) complex floats of scratch plus your OLA history buffer; on the 128 KB L476RG keep \(N \le 512\) and one hop of overlap history. Don’t blow SRAM.
• Latency is structural here. OLA imposes at least one frame of algorithmic latency (\(N/f_s\)). That’s inherent, not a bug — budget for it and don’t try to “fix” it by shrinking the window past your frequency resolution needs.
• Preserve the Lab 9.1 transport discipline: FFT compute per block must fit inside the block’s real-time budget or the RX buffer overruns.

Background

The STFT. Slice \(x[n]\) into frames of length \(N\) taken every hop \(R\) samples, window each frame with \(w[n]\), and DFT it:

\[X_m[k] = \sum_{n=0}^{N-1} w[n]\,x[mR+n]\,e^{-j 2\pi k n / N}, \qquad k = 0,\dots,N-1.\]

A spectral effect modifies \(X_m[k] \to \tilde{X}_m[k]\) (e.g. a noise gate zeroes bins below a threshold; spectral subtraction removes a noise magnitude estimate; fast-convolution reverb multiplies by an impulse-response spectrum; robotization zeroes the phase).

Overlap-add reconstruction. Inverse-DFT each modified frame, window again (for a synthesis window / analysis-only OLA), and sum the overlapping frames back onto the time axis:

\[y[n] = \sum_m \tilde{x}_m[n - mR], \qquad \tilde{x}_m[n]=\frac{1}{N}\sum_k \tilde{X}_m[k]\,e^{j2\pi k n/N}.\]

The COLA condition. For the identity effect (\(\tilde{X}=X\)), OLA reconstructs \(x\) exactly (up to FFT round-off) iff the analysis window satisfies constant overlap-add at hop \(R\):

\[\sum_{m} w[n - mR] = C \quad \text{(a constant, independent of } n\text{)}.\]

A Hann window satisfies COLA at 50% overlap (\(R=N/2\), \(C=1\)) and at 75% (\(R=N/4\)). This is the property your identity test checks. Choosing 50% vs. 75% trades compute (more frames) for smoother modification and less inter-frame artifacting.

Window choice & leakage. Hann trades main-lobe width for low side-lobes (Lyons Ch. 3), which matters when a spectral effect edits magnitudes bin-by-bin — leakage smears energy across bins and can make a gate chatter.

Where it runs. On the STM32 at \(N=256\)–\(512\) the M4F + CMSIS arm_rfft_fast_f32 handles a modest effect per block within budget. Heavier effects (long-tail convolution reverb, large FFTs) or higher sample rates belong on the Pi 5, where NumPy/scipy FFTs are cheap but you lose the “runs on a $2 hardware” story. Note the tradeoff in your write-up.

Procedure

Part A — Prove OLA is transparent (identity test) first.

1. Choose \(N=512\), Hann window, 50% overlap (\(R=256\)). On the host, implement analysis→(identity)→synthesis OLA in NumPy and confirm it reconstructs a test WAV to within FFT round-off. Numerically verify COLA:

import numpy as np
N, R = 512, 256
w = np.hanning(N)
cola = np.zeros(N)
for m in range(-4, 5):            # sum shifted copies over one period
    cola += np.roll(np.pad(w, (0, N))[ :N], (m*R) % N)
# with proper accumulation, cola is ~constant  → COLA satisfied

2. Only once the host identity OLA is flat do you move to the device.

Part B — Firmware STFT/OLA.

3. Maintain an input history so each new block forms a full \(N\)-sample frame at hop \(R\). Illustrative per-frame kernel (structure only):

arm_rfft_fast_instance_f32 S;   // init once for size N
/* frame[]: N samples, already Hann-windowed */
arm_rfft_fast_f32(&S, frame, spec, 0);         // forward real FFT (packed)
apply_effect(spec, N);                          // e.g. gate bins below thresh
arm_rfft_fast_f32(&S, spec, recon, 1);          // inverse FFT
/* window (synthesis) + overlap-add into the output ring buffer */
for (int n = 0; n < N; ++n)
    ola[(head + n) % OLA_LEN] += w[n] * recon[n];
/* emit R samples from the tail of the completed region, then advance head by R */

4. Stream the effected output back through the Lab 9.1 harness; host writes effected.wav.

Part C — Verify and listen.

5. Identity check on-device: run the device with the effect bypassed; assert the returned WAV matches the input to within a small tolerance (FFT round-off + Q-format if any): np.max(np.abs(y - x)) / 32768 < 1e-4.
6. Enable the effect. For a noise gate, confirm the noise floor drops in the FFT and by ear; for reverb, confirm the tail; for robotization, confirm the monotone pitch.
7. Compare against a NumPy reference of the same effect (same \(N\), window, hop, threshold) — magnitude spectra should match closely frame-by-frame.

Deliverable & expected results

• The COLA verification plot, the on-device identity-reconstruction error, and effected.wav with a before/after FFT (and a reference-vs-device spectral comparison for the chosen effect).

For \(N=512\), Hann, 50% overlap, \(f_s = 48\) kHz:

Quantity	Predicted	Measured
COLA sum \(\sum_m w[n-mR]\) (Hann, 50%)	constant (\(=1\))	…
Identity OLA reconstruction error (device)	\(\lesssim 10^{-4}\) FS (round-off)	…
Algorithmic latency (\(N/f_s\))	10.67 ms	…
Frame rate (\(f_s/R\))	187.5 frames/s	…
FFT bin spacing (\(f_s/N\))	93.75 Hz	…
Effect vs. NumPy reference (per-frame mag)	match to round-off	…

Analysis & reconciliation

The identity test is the linchpin: if bypassed-effect OLA doesn’t reconstruct flat, your window/hop violate COLA (audible tremolo at 187.5 Hz here) or your OLA indexing drops/doubles a hop — fix that before interpreting any effect. A device identity error much larger than round-off points to the arm_rfft_fast_f32 packed-layout contract being mishandled or an in-place buffer alias. Once identity is clean, reconcile the effected output against the NumPy reference frame-by-frame; residual differences should be round-off (float path) plus any threshold/estimator nondeterminism you introduced. Note the measured per-block compute headroom on your chosen platform against the frame period (\(R/f_s = 5.33\) ms here) — that margin is what decides STM32 vs. Pi 5 for a heavier effect.

Going further

• Switch to overlap-save (linear convolution via FFT with discarded wrap-around) for the fast-convolution reverb and compare artifact/latency against OLA.
• Try 75% overlap (Hann still COLA) and hear the reduction in modification artifacts; measure the compute cost increase.
• Implement spectral subtraction with a noise magnitude estimated from the first few (silent) frames and quantify the noise-floor drop in dB — a preview of the learned denoiser in Lab 8.3, but hand-built.
• Port the same STFT effect to the Pi 5 at \(N=2048\) and compare artifacts and CPU load to the STM32’s small-\(N\) version.