Lab 9.3 — STFT Overlap-Add Audio Effects

← Course 3 syllabus · Bonus Module 9 · Prev: « Lab 9.2 · Next: Lab 9.4 »

Goal

Move from time-domain filtering to block-based frequency-domain processing on the streamed WAV. You’ll build a real short-time Fourier transform (STFT) pipeline on the device: window each block, forward-FFT with CMSIS-DSP arm_rfft_fast_f32, apply a spectral effect (a noise gate / spectral subtraction / fast-convolution reverb / robotization), inverse-FFT, and overlap-add (OLA) the windowed frames back into a continuous stream that goes home to effected.wav. The verification step is what makes this rigorous: with the effect set to identity, correct OLA must reconstruct the input to within FFT round-off — so you first prove your OLA framework is transparent against a NumPy reference, and only then trust the effected output. This is the standard structure of real-time spectral audio firmware (denoisers, vocoders, convolution reverbs), and the identity-reconstruction test is how you know your windowing and hop are consistent.

Equipment & parts

STM32 Nucleo-L476RG (small FFT sizes, $N=256$–$512$) or the Raspberry Pi 5 (larger $N$, heavier effects) — the lab runs on either; see the tradeoff note in the Background.
CMSIS-DSP (arm_rfft_fast_f32, arm_cmplx_mag_f32) on the STM32; NumPy on the Pi 5.
Host Mac with numpy scipy for the reference OLA and audio checks.
On the device: the USB sound card and speakers (on hand) to hear each effect as the device produces it — dry vs. wet, A/B’d at the block boundary — rather than only after the file returns to the Mac.
A test WAV: for a noise gate / spectral subtraction, a recording with audible background noise (fan, hiss) makes the effect obvious; for reverb, a dry vocal or clap.

Wiring & bench setup

STM32 route: the same single USB cable as Lab 9.1 (Mac → ST-LINK Micro-B, USART2 VCP). Pi 5 route: no cable at all — the “link” is your LAN/SSH session. Either way the chain is:

flowchart LR
  WAV["media/in/noisy.wav"]
  HOST["Host harness<br/>Lab 9.1 framing + credits"]
  DEV["STM32 or Pi 5<br/>window → FFT → effect → IFFT → OLA"]
  OUT["media/out/effected.wav"]
  REF["NumPy STFT/OLA reference<br/>+ identity test"]
  WAV --> HOST
  HOST -- "PCM blocks" --> DEV
  DEV -- "effected blocks" --> HOST
  HOST --> OUT -- "compare" --> REF

flowchart LR
  WAV["media/in/noisy.wav"]
  HOST["Host harness<br/>Lab 9.1 framing + credits"]
  DEV["STM32 or Pi 5<br/>window → FFT → effect → IFFT → OLA"]
  OUT["media/out/effected.wav"]
  REF["NumPy STFT/OLA reference<br/>+ identity test"]
  WAV --> HOST
  HOST -- "PCM blocks" --> DEV
  DEV -- "effected blocks" --> HOST
  HOST --> OUT -- "compare" --> REF

Safety & don’t-break-it

Data lab; the failure modes are windowing/indexing bugs and real-time budget overruns:

COLA violation = amplitude modulation. If your window and hop don’t satisfy the constant-overlap-add condition, the identity effect won’t reconstruct flat — you’ll hear a periodic tremolo at the frame rate. Verify COLA numerically before blaming the effect.
In-place FFT buffer aliasing. arm_rfft_fast_f32 uses a specific packed real-spectrum layout and an in-place scratch buffer. Reusing the same buffer for input and output without respecting the layout corrupts bins silently. Follow the CMSIS buffer contract exactly.
FFT-size / SRAM budget. Float FFT of size $N$ needs $\sim N$ complex floats of scratch plus your OLA history buffer; on the 128 KB L476RG keep $N \le 512$ and one hop of overlap history. Don’t blow SRAM.
Latency is structural here. OLA imposes at least one frame of algorithmic latency ($N/f_s$). That’s inherent, not a bug — budget for it and don’t try to “fix” it by shrinking the window past your frequency resolution needs.
Preserve the Lab 9.1 transport discipline: FFT compute per block must fit inside the block’s real-time budget or the RX buffer overruns.

Project & environment setup

Firmware — reuse firmware/m9-media/ (Lab 9.1). No new peripherals; confirm USART2 921600 + DMA + 80 MHz clock (setup essentials) as before. CMSIS-DSP is required this time (arm_rfft_fast_f32, arm_cmplx_mag_f32) — if you didn’t add it in Lab 9.2, add it now (same steps as firmware/m6-dsp/).

Pi 5 route (optional): on the Pi, python3 -m venv ~/venv && ~/venv/bin/pip install numpy scipy — the device-side STFT/OLA script then lives in labs/lab-9-3/edge/ (the SBC-side tree, as in Module 8) and is synced to the board.

Host — course venv; transport is labs/lab-9-1/host/harness.py. Library roles: numpy (Part A COLA check + identity-OLA and effect references; scipy.signal.stft/istft is a useful cross-check of your hand-rolled OLA), scipy.io.wavfile (WAV I/O), matplotlib (COLA and before/after-FFT plots):

source venv/bin/activate
mkdir -p labs/lab-9-3/host labs/lab-9-3/captures

Inputs: media/in/noisy.wav (gate/spectral subtraction) or media/in/dry.wav (reverb). Reference scripts in labs/lab-9-3/host/ (you write them).

Keep this lab’s reconciliation in labs/lab-9-3/host/analysis.ipynb — the notebook convention — and export final figures next to it.

Where results go:

Artifact	Path
Bench note (identity error, latency, headroom)	`labs/lab-9-3/notes.md`
Effected output	`media/out/effected.wav`
COLA verification plot	`labs/lab-9-3/host/cola.png`
Before/after + reference-vs-device FFT	`labs/lab-9-3/host/before-after-fft.png`
Identity-test error logs	`labs/lab-9-3/captures/identity-err.txt`

Background

The STFT. Slice $x[n]$ into frames of length $N$ taken every hop $R$ samples, window each frame with $w[n]$, and DFT it:

\[X_m[k] = \sum_{n=0}^{N-1} w[n]\,x[mR+n]\,e^{-j 2\pi k n / N}, \qquad k = 0,\dots,N-1.\]

A spectral effect modifies $X_m[k] \to \tilde{X}_m[k]$ (e.g. a noise gate zeroes bins below a threshold; spectral subtraction removes a noise magnitude estimate; fast-convolution reverb multiplies by an impulse-response spectrum; robotization zeroes the phase).

Overlap-add reconstruction. Inverse-DFT each modified frame and sum the overlapping frames back onto the time axis. The default here is an analysis-only window: the frame is windowed once before the FFT, and the reconstructed frames are overlap-added directly (no second window). A synthesis window can also be applied on the way out, in which case the COLA condition below is on the product of the analysis and synthesis windows (see Going further).

\[y[n] = \sum_m \tilde{x}_m[n - mR], \qquad \tilde{x}_m[n]=\frac{1}{N}\sum_k \tilde{X}_m[k]\,e^{j2\pi k n/N}.\]

The COLA condition. For the identity effect ($\tilde{X}=X$), OLA reconstructs $x$ exactly (up to FFT round-off) iff the analysis window satisfies constant overlap-add at hop $R$:

\[\sum_{m} w[n - mR] = C \quad \text{(a constant, independent of } n\text{)}.\]

A single Hann analysis window satisfies COLA at 50% overlap ($R=N/2$, $C=1$) and at 75% ($R=N/4$). This is the property your identity test checks. (If you also apply a Hann synthesis window, the relevant sum is $\sum_m w^2[n-mR]$, which for Hann at 50% overlap is $C=0.75$, not 1 — you would then divide the output by $0.75$, or use $\sqrt{\text{Hann}}$ on both sides to keep $C=1$.) Choosing 50% vs. 75% trades compute (more frames) for smoother modification and less inter-frame artifacting.

Window choice & leakage. Hann trades main-lobe width for low side-lobes (Lyons Ch. 3), which matters when a spectral effect edits magnitudes bin-by-bin — leakage smears energy across bins and can make a gate chatter.

Where it runs. On the STM32 at $N=256$–$512$ the M4F + CMSIS arm_rfft_fast_f32 handles a modest effect per block within budget. Heavier effects (long-tail convolution reverb, large FFTs) or higher sample rates belong on the Pi 5, where NumPy/scipy FFTs are cheap but you lose the “runs on a $2 hardware” story. Note the tradeoff in your write-up.

Theory — Audio: Quantization, Dither, Noise Shaping, Equalizers, Dynamics, and Coding

What quantizing a signal costs (and how dither and noise shaping spend that cost wisely), and how EQ, dynamics, and coding are designed. This section assumes the biquad and bilinear-transform material of Lab 6.2’s theory section and the sigma–delta material of Lab 3.4’s; the learned-DSP half of the same lesson lives in Lab 8.3.

Uniform quantization and the 6-dB-per-bit law

A word-length-$w$ uniform quantizer over $\pm x_{\max}$ has step size $Q = 2 x_{\max}/2^{w}$ and output $x_Q[n] = Q \cdot \mathrm{round}( x[n]/Q)$. The classical model treats the error $e[n] = x_Q[n] - x[n]$ as additive noise, uniform on $( -Q/2, Q/2]$, white, and independent of the signal: $\mathbb{E}[ e] = 0$, $\sigma_E^2 = Q^2/12$. Writing the signal power through the peak factor $P_F = x_{\max}/\sigma_X$,

\[ \mathrm{SNR} = 10 \log_{10} \frac{ \sigma_X^2}{ \sigma_E^2} = 10 \log_{10} \frac{ x_{\max}^2 / P_F^2}{ ( 2 x_{\max} 2^{-w})^2 / 12} = 6.02\, w \;-\; 10 \log_{10}( P_F^2/3) \quad \text{[dB]}. \]

Special cases: full-scale sine ($P_F = \sqrt2$) gives the famous $6.02 w + 1.76$ dB; a uniform PDF ($P_F = \sqrt3$) gives $6.02 w$; near-Gaussian real audio, run at $P_F = 4.61$ so overload probability stays below $10^{-5}$, gives $6.02 w - 8.5$ dB — headroom is paid for in SNR. The model holds for wide-dynamic-range inputs exercising many steps; for small signals (a few $Q$ of amplitude) the error is a deterministic, periodic function of the signal and its spectrum is harmonic distortion, not noise. (The deeper story, Widrow’s quantization theorem, treats quantization as sampling of the amplitude PDF — Shannon’s theorem restated in amplitude — with the classical model as its baseband shadow.)

Dither

Dither rescues the small-signal case: add a random sequence $d[n]$ before requantizing. Uniform (RECT) dither on $( -Q/2, Q/2]$ linearizes the staircase — the ensemble-mean output becomes a fine-stepped ramp instead of a coarse one — but the error variance still swells and shrinks with the input level (audible noise modulation). Triangular (TPDF) dither, the sum of two independent uniforms, makes the error variance constant as well: linearized and modulation-free, at a cost of about $4.77$ dB of noise floor (dither power adds to quantization power). Higher-order dither buys nothing more for audio; TPDF dithering is the last step of every mastering chain.

Noise shaping

Dither fixes the error’s statistics; noise shaping moves its spectrum. Isolate the requantization error and feed it back through $H( z)$ ahead of the quantizer:

\[ Y( z) = X( z) + E( z)\, [ 1 - H( z)] \]

— the signal passes untouched while the error is filtered by the noise transfer function $1 - H( z)$. The canonical $H( z) = z^{-1}$ gives first-order highpass weighting $| 1 - e^{-j\Omega}|^2$; $H( z) = z^{-1}( -2 + z^{-1})$ gives second-order $( 1 - z^{-1})^2$. Total error power rises, but in-band power falls (the curves cross near $f_S/6$): quiet where the ear is, loud near $f_S/2$. Combined with TPDF dither this is how 16-bit media carry roughly 19-bit-quality mid-band audio — and the error-feedback loop is exactly the engine inside the sigma–delta converters of Lab 3.4’s theory section, which run it at high oversampling so the shaped noise lands mostly out of band.

Equalizers: shelving and peak biquads

Audio EQ is built from three shapes, designed as analog prototypes and discretized by the bilinear transform of Lab 6.2’s theory section. The recipe: (i) pick the gain $V_0 = 10^{ G_{\mathrm{dB}}/20}$ and the cutoff/center frequency $\omega_c$, and prewarp $K = 1/\tan( \omega_c/2)$; (ii) choose the prototype — low-frequency shelving boost $H( s) = ( s + V_0)/( s + 1)$, high-frequency shelving boost $H( s) = ( s V_0 + 1)/( s + 1)$, and for cut invert the transfer function (e.g. $( s + 1)/( s + V_0)$) rather than setting $V_0 < 1$, so boost and cut curves mirror about 0 dB at the same cutoff; the peak filter (boost/cut at an arbitrary center) is $1 + H_{BP}( s)$:

\[ H( s) = \frac{ s^2 + \dfrac{ V_0}{ Q_\infty}\, s + 1} { s^2 + \dfrac{ 1}{ Q_\infty}\, s + 1}, \]

center gain $V_0$, relative bandwidth $1/Q_\infty$, response geometrically symmetric about the center; (iii) substitute $s \to K ( 1 - z^{-1})/( 1 + z^{-1})$ and normalize — out come biquad (or first-order) coefficients ready for the direct forms of Lab 6.2. Production notes: parametric (allpass-plus-direct-path) refactorings decouple the gain, frequency, and bandwidth knobs (one coefficient each), and coefficient quantization bites worst at low cutoffs — why firmware treats bass with care.

Dynamic range control

A compressor/limiter maps input level to output level through a static curve in dB–dB axes: noise gate below NT, unity in the middle, compressor slope $1/R$ above threshold CT (ratio $R$), limiter clamp above LT. The recipe: (i) measure level — RMS through a one-pole, $x_{\mathrm{rms}}[n] = ( 1 - \mathrm{TAV})\, x_{\mathrm{rms}}[n{-}1] + \mathrm{TAV}\, x^2[n]$, or PEAK tracking of $| x[n]|$ for the limiter; (ii) work in the log domain: subtract the threshold from the level in dB, multiply the excess by the segment slope ($\mathrm{CS} = 1 - 1/R$ for a compressor), antilog back to a linear control factor $f[n]$; (iii) smooth $f[n]$ through another one-pole whose coefficient is AT while the gain is falling (attack) and RT while rising (release); (iv) multiply the smoothed gain onto a delayed input $x[n - D]$, so the gain anticipates transients (look-ahead limiting). Defining attack time as the smoother’s 10%–90% rise, $t_a = 2.2 \tau$, the pole and coefficient are

\[ z_\infty = e^{ -2.2\, T_S / t_a}, \qquad k = 1 - z_\infty \]

(same formula for RT and TAV) — the one formula that turns every “attack: 5 ms” knob into a filter coefficient. Stereo processors must link the channels’ detectors (one common gain), or the image wanders toward whichever channel is quieter.

Perceptual audio coding (stated, not derived)

Information theory sets the floors. Lossless: a memoryless source can be coded at any rate above its entropy $H( X) = -\sum_i p_i \log_2 p_i$ and at no rate below it; Huffman coding achieves $\bar L < H + 1$, and source memory lowers the floor to a conditional entropy. Lossy: for a Gaussian source under mean-square distortion, $R( D) = \frac12 \log_2( \sigma^2/D)$ — 6.02 dB per bit again, now as a law of nature; plain uniform quantization sits about $0.25$ bit above it. Perceptual coders beat these floors for perceived quality by spending bits where the ear listens: the cochlea analyzes in roughly 25 critical bands, and a strong signal masks its neighborhood — spreading functions applied to the band powers combine into a global masking threshold $T_m( i)$. The actionable per-band number is the signal-to-mask ratio, $\mathrm{SMR}_i = L_S( i) - L_{T_m}( i)$: noise in band $i$ is inaudible while below the mask, so the band needs only about $\lceil \mathrm{SMR}_i / 6.02 \rceil$ bits. MPEG-1 industrializes this: a 32-band polyphase (pseudo-QMF) filter bank splits the signal, a parallel FFT feeds the psychoacoustic model that computes the SMRs, and dynamic bit allocation plus scalefactors pack each frame; Layer III (MP3) further splits each subband by an 18-point MDCT (a lapped transform, with window switching to catch transients) for extra coding gain.

Connection

Labs 8.2–8.4 are the learned-DSP half of this lesson deployed (Lab 8.3’s theory section): spectrogram features feeding small CNNs for keyword classification, learned denoising, and anomaly detection. Lab 9.2 and this lab implement the classical half in real time: biquad EQ and compressor blocks (the shelving/peak recipe and the $z_\infty = e^{-2.2 T_S/t_a}$ formula, straight from codec datasheets) and STFT overlap-add effects. DSP covers quantization, dither, and noise shaping as coursework; the perceptual coder is behind every stream the bench decodes.

Worked by hand

P1 (SNR law). A full-scale 16-bit sine: $6.02 \cdot 16 + 1.76 = 98.1$ dB. The same sine at $-20$ dBFS: $78.1$ dB (the amplitude drops; $Q$ does not). Word length for a Gaussian source at $P_F = 4.61$ to reach 96 dB: $6.02 w - 8.5 \ge 96 \Rightarrow w \ge 17.4$, so 18 bits.

P2 (shaping weight). First-order shaping multiplies the error PSD by $| 1 - e^{-j\Omega}|^2 = 2 - 2\cos\Omega$: $0$ at $\Omega = 0$, $2$ at $\pi/2$, $4$ at $\pi$. It starts amplifying noise where the weight crosses 1, at $\Omega = \pi/3$, i.e. $f_S/6$.

P3 (shelving design). LF shelf, $+12$ dB ($V_0 = 4$), cutoff $0.1 f_S$ ($\omega_c = 0.2\pi$): $K = 1/\tan( 0.1\pi) = 3.078$. Substituting $s \to K( 1 - z^{-1})/( 1 + z^{-1})$ into $( s + V_0)/( s + 1)$ and dividing by $K + 1$: $b_0 = ( K + V_0)/( K + 1) = 1.736$, $b_1 = ( -K + V_0)/( K + 1) = 0.226$, $a_1 = ( 1 - K)/( 1 + K) = -0.510$. Check: at $z = 1$ the gain is $( b_0 + b_1)/( 1 + a_1) = 4.00$ ($12.0$ dB); at $z = -1$ it is $1$ ($0$ dB).

P4 (compressor). Ratio $R = 4$ above $\mathrm{CT} = -20$ dBFS: slope $\mathrm{CS} = 1 - 1/R = 0.75$, gain $= -\mathrm{CS} \times$ excess. Input $-12$ dBFS (8 dB over) $\to -18$ dBFS; $-4$ dBFS (16 over) $\to -16$ dBFS. Coefficients at $f_S = 48$ kHz: $t_a = 5$ ms gives $\mathrm{AT} = 1 - e^{-2.2/240} = 9.13 \times 10^{-3}$; $t_r = 130$ ms gives $\mathrm{RT} = 3.53 \times 10^{-4}$.

P5 (masked bit allocation). A band holds signal at 50 dB; the mask spread into it from a neighboring 60 dB tone is 42 dB. $\mathrm{SMR} = 8$ dB, so $\lceil 8/6.02 \rceil = 2$ bits keep the band’s noise under the mask.

Theory exercises

The lesson’s remaining two exercises (activation-function derivatives; CNN feature-map sizing) are on Lab 8.3 with the learned-DSP material.

Theory exercise 1 [Hand] — (P&S 6.19.) A discrete memoryless source has an alphabet of eight letters $x_i$, $i = 1, \ldots, 8$, with probabilities $0.25$, $0.20$, $0.15$, $0.12$, $0.10$, $0.08$, $0.05$, $0.05$. (1) Use the Huffman procedure to determine a binary code for the source. (2) Determine the average number $\bar R$ of binary digits per source letter. (3) Determine the entropy of the source and compare it with $\bar R$.

Procedure

Part A — Prove OLA is transparent (identity test) first.

Choose $N=512$, Hann window, 50% overlap ($R=256$). On the host, implement analysis→(identity)→synthesis OLA in NumPy and confirm it reconstructs a test WAV to within FFT round-off. Numerically verify COLA:

import numpy as np
N, R = 512, 256
w = np.hanning(N)
cola = np.zeros(N)
for m in range(-4, 5):            # sum shifted copies over one period
    cola += np.roll(np.pad(w, (0, N))[ :N], (m*R) % N)
# with proper accumulation, cola is ~constant  → COLA satisfied

Only once the host identity OLA is flat do you move to the device.

Part B — Firmware STFT/OLA.

Maintain an input history so each new block forms a full $N$-sample frame at hop $R$. Illustrative per-frame kernel (structure only):

arm_rfft_fast_instance_f32 S;   // init once for size N
/* frame[]: N samples, already Hann-windowed */
arm_rfft_fast_f32(&S, frame, spec, 0);         // forward real FFT (packed)
apply_effect(spec, N);                          // e.g. gate bins below thresh
arm_rfft_fast_f32(&S, spec, recon, 1);          // inverse FFT
/* analysis-only OLA: a single Hann window is COLA with C=1 at 50% overlap, so
   overlap-ADD the reconstructed frames directly. Do NOT window a second time here:
   Hann*Hann = Hann^2 sums to 0.75, not 1, and would scale the output down. */
for (int n = 0; n < N; ++n)
    ola[(head + n) % OLA_LEN] += recon[n];
/* emit R samples from the tail of the completed region, then advance head by R */

Stream the effected output back through the Lab 9.1 harness; host writes effected.wav.

Part C — Verify and listen.

Identity check on-device: run the device with the effect bypassed; assert the returned WAV matches the input to within a small tolerance (FFT round-off + Q-format if any): np.max(np.abs(y - x)) / 32768 < 1e-4.
Enable the effect. For a noise gate, confirm the noise floor drops in the FFT and by ear; for reverb, confirm the tail; for robotization, confirm the monotone pitch.
Compare against a NumPy reference of the same effect (same $N$, window, hop, threshold) — magnitude spectra should match closely frame-by-frame.

Deliverable & expected results

The COLA verification plot, the on-device identity-reconstruction error, and effected.wav with a before/after FFT (and a reference-vs-device spectral comparison for the chosen effect).

For $N=512$, Hann, 50% overlap, $f_s = 48$ kHz:

Quantity	Predicted	Measured
COLA sum $\sum_m w[n-mR]$ (Hann, 50%)	constant ($=1$)	…
Identity OLA reconstruction error (device)	$\lesssim 10^{-4}$ FS (round-off)	…
Algorithmic latency ($N/f_s$)	10.67 ms	…
Frame rate ($f_s/R$)	187.5 frames/s	…
FFT bin spacing ($f_s/N$)	93.75 Hz	…
Effect vs. NumPy reference (per-frame mag)	match to round-off	…

Analysis & reconciliation

The identity test is the linchpin: if bypassed-effect OLA doesn’t reconstruct flat, your window/hop violate COLA (audible tremolo at 187.5 Hz here) or your OLA indexing drops/doubles a hop — fix that before interpreting any effect. A device identity error much larger than round-off points to the arm_rfft_fast_f32 packed-layout contract being mishandled or an in-place buffer alias. Once identity is clean, reconcile the effected output against the NumPy reference frame-by-frame; residual differences should be round-off (float path) plus any threshold/estimator nondeterminism you introduced. Note the measured per-block compute headroom on your chosen platform against the frame period ($R/f_s = 5.33$ ms here) — that margin is what decides STM32 vs. Pi 5 for a heavier effect.

Jetson Orin Nano as the device

Via the Lab 9.1 Jetson transport, the device-side STFT→modify→ISTFT moves to the Jetson with the frames and reference checks unchanged — and this is the M9 lab where the GPU genuinely earns a run: the framed blocks batch naturally into the cupyx.scipy STFT path you measured in Lab 6.3/8.1, so process the stream once on CPU (NumPy server) and once on GPU (CuPy server) and compare achieved real-time factor at several block sizes. The COLA identity test is the arbiter on both. Pi 5: CPU path only.

Going further

Switch to overlap-save (linear convolution via FFT with discarded wrap-around) for the fast-convolution reverb and compare artifact/latency against OLA.
Try 75% overlap (Hann still COLA) and hear the reduction in modification artifacts; measure the compute cost increase.
Add a synthesis window for smoother magnitude-editing effects: use $\sqrt{\text{Hann}}$ on both the analysis and synthesis sides (the product is Hann, so COLA still gives $C=1$), or keep Hann on both sides and normalize the overlap-added output by $0.75$. Confirm the identity test still reconstructs flat.
Implement spectral subtraction with a noise magnitude estimated from the first few (silent) frames and quantify the noise-floor drop in dB — a preview of the learned denoiser in Lab 8.3, but hand-built.
Port the same STFT effect to the Pi 5 at $N=2048$ and compare artifacts and CPU load to the STM32’s small-$N$ version.