Lab 9.4 — Image 2-D Convolution on the Embedded Target

← Course 2 syllabus · Bonus Module 9 · Prev: « Lab 9.3 · Next: Lab 9.5 »

Goal

Take the streaming discipline into two dimensions: the host sends a real grayscale image to the device, which runs 2-D convolution — a separable Gaussian blur and a Sobel edge detector — on it, block by block, and streams the result back for the host to save as a PNG and check against scipy.ndimage / OpenCV. On the memory-tight STM32 the key trick is streaming the image row-by-row through a small sliding window of rows, never holding the whole picture, which is exactly how line-buffer image pipelines work in real camera/vision firmware; larger images move to the Pi 5 / Jetson. The verification is quantitative: PSNR and max-abs-error vs. a reference implementation, so “the blur looks right” becomes a number. This is the image-signal-processing lab of the module and the bridge to the video pipeline in Lab 9.5.

Recommended reading

• Lyons — convolution and FIR filtering; read it with the 1-D→2-D extension in mind (a separable 2-D filter is two 1-D convolutions).
• Course 1 Part I (../course1/index.qmd#part-1) — linear algebra: convolution as a linear operator (a Toeplitz/circulant matrix action), separability as an outer-product / rank-1 factorization of the kernel, and why that factorization saves work.
• Course 1 Lesson 42 (../course1/index.qmd#lesson-42) — images: 2-D spatial filtering, smoothing (Gaussian) and sharpening kernels, and the gradient/Sobel operators this lab implements.
• Reference-implementation docs: scipy.ndimage.gaussian_filter / scipy.ndimage.sobel, or OpenCV cv2.GaussianBlur / cv2.Sobel.

Equipment & parts

• STM32 Nucleo-L476RG for small images (see the SRAM budget below), or Raspberry Pi 5 / Jetson Orin Nano for larger images.
• Host Mac with numpy scipy pillow (and optionally opencv-python).
• A grayscale test image — a PGM (trivial to parse) or a PNG loaded via Pillow and converted to 8-bit gray. Start small (e.g. 128×128) for the STM32.

Wiring & bench setup

Same single-cable hookup as Lab 9.1 (Mac USB → ST-LINK Micro-B, USART2 VCP); nothing else on the bench. The blocks are now image rows:

flowchart LR
  PNG["media/in/test.png"]
  HOST["Host: Pillow → 8-bit gray<br/>Lab 9.1 framing, one row/block"]
  MCU["STM32 3-row window<br/>separable blur + Sobel"]
  OUT["media/out/blur.png<br/>media/out/edges.png"]
  REF["scipy.ndimage reference<br/>PSNR + max-abs-error"]
  PNG --> HOST
  HOST -- "header + rows" --> MCU
  MCU -- "output rows" --> HOST
  HOST --> OUT -- "compare" --> REF

flowchart LR
  PNG["media/in/test.png"]
  HOST["Host: Pillow → 8-bit gray<br/>Lab 9.1 framing, one row/block"]
  MCU["STM32 3-row window<br/>separable blur + Sobel"]
  OUT["media/out/blur.png<br/>media/out/edges.png"]
  REF["scipy.ndimage reference<br/>PSNR + max-abs-error"]
  PNG --> HOST
  HOST -- "header + rows" --> MCU
  MCU -- "output rows" --> HOST
  HOST --> OUT -- "compare" --> REF

Safety & don’t-break-it

Data lab; the constraints are memory and numeric scaling:

• The L476RG has 128 KB SRAM — you cannot hold a big image. A 512×512 8-bit image is 256 KB, already twice the SRAM. This is why you stream row-by-row and keep only a 3-row sliding window (a few KB) plus one output row. Sizing the whole frame into RAM is the mistake that won’t even link.
• Border handling must be decided, not accidental. Reading img[-1] or past the last row silently reads garbage. Choose a border policy (zero-pad, replicate/clamp, or reflect) and implement it explicitly at the top/bottom/left/right edges — and use the same policy in the host reference or the PSNR will be wrong at the borders.
• Fixed-point scaling overflows fast in 2-D. A 3×3 kernel sums nine products; an unnormalized Sobel or an integer Gaussian can exceed 8 bits. Accumulate in int16/int32, normalize (divide by the kernel sum for the blur), and saturate back to uint8 — never let the byte wrap.
• Keep the Lab 9.1 flow control: a row is a block; don’t let convolution compute stall the RX DMA.

Project & environment setup

Firmware — reuse firmware/m9-media/ (Lab 9.1). No new peripherals and no CMSIS-DSP needed (the 3×3 passes are integer C); confirm USART2 921600 + DMA + 80 MHz clock (setup essentials) as before.

Host — course venv plus the image libraries this lab adds:

source venv/bin/activate
pip install pillow opencv-python   # Pillow: PNG → 8-bit gray (PIL.Image); OpenCV optional cross-check
mkdir -p labs/lab-9-4/host labs/lab-9-4/captures

Library roles: PIL.Image (load/convert/save), scipy.ndimage.convolve1d (matched separable reference, as in Part C; scipy.signal.convolve2d is the non-separable cross-check), numpy (PSNR/max-abs-error math), cv2.GaussianBlur/cv2.Sobel (optional second opinion — remember they use their own kernels/normalization, so score against the matched convolve1d reference, not these). Transport is labs/lab-9-1/host/harness.py; the row-streamer + scoring script go in labs/lab-9-4/host/ (you write them). Input media/in/test.png — the snippet’s in.png.

Keep this lab’s reconciliation in labs/lab-9-4/host/analysis.ipynb — the notebook convention — and export final figures next to it.

Where results go:

Artifact	Path
Bench note (PSNR / max-abs-error, SRAM budget)	labs/lab-9-4/notes.md
Device blur / edge outputs	media/out/blur.png · media/out/edges.png
Reference images	labs/lab-9-4/host/ref-blur.png · labs/lab-9-4/host/ref-edges.png
Score logs (PSNR, max-abs-error, per border mode)	labs/lab-9-4/captures/scores.txt

Background

2-D convolution. For an image \(I\) and kernel \(K\) of size \((2a{+}1)\times(2b{+}1)\):

\[(I * K)[i,j] \;=\; \sum_{u=-a}^{a}\sum_{v=-b}^{b} K[u,v]\,I[i-u,\,j-v].\]

A direct \(M\times N\)-pixel convolution with a \(k\times k\) kernel costs \(O(MNk^2)\) multiply-adds.

Separability. A Gaussian kernel factors as an outer product of two 1-D Gaussians,

\[K = g\,g^{\mathsf T}, \qquad g[t] = \tfrac{1}{Z}\,e^{-t^2/(2\sigma^2)},\]

so the 2-D convolution becomes two 1-D passes — convolve each row with \(g\), then each column with \(g\):

\[I * K \;=\; \big(I *_{\text{rows}} g\big) *_{\text{cols}} g.\]

This drops the cost from \(k^2\) multiply-adds per pixel to \(2k\) — for the \(k=3\) kernel here, \(9 \to 6\) (a \(1.5\times\), i.e. 33%, saving; the asymptotic factor is \(k/2\)) — and, in Course 1 Part I terms, is just the rank-1 factorization of the kernel matrix. The 3-row sliding window on the STM32 is exactly what the column pass needs: a \(3\times3\) kernel touches only rows \(i{-}1,i,i{+}1\), so you buffer three input rows, emit one output row, slide.

Sobel edges. The Sobel operator estimates the image gradient with two separable \(3\times3\) kernels:

\[G_x = \begin{bmatrix} 1 & 0 & -1\\ 2 & 0 & -2\\ 1 & 0 & -1\end{bmatrix} = \begin{bmatrix}1\\2\\1\end{bmatrix}\begin{bmatrix}1&0&-1\end{bmatrix}, \qquad G_y = G_x^{\mathsf T},\]

and the edge (gradient-magnitude) image is

\[G[i,j] = \sqrt{G_x[i,j]^2 + G_y[i,j]^2}.\]

Both \(G_x\) and \(G_y\) are separable (a smoothing \([1,2,1]\) and a difference \([1,0,-1]\)), so the same line-buffer machinery applies.

PSNR. To compare the device output \(\hat{I}\) against the reference \(I_{\text{ref}}\) (8-bit, peak \(=255\)):

\[\text{MSE} = \frac{1}{MN}\sum_{i,j}\big(\hat{I}[i,j]-I_{\text{ref}}[i,j]\big)^2, \qquad \text{PSNR} = 10\log_{10}\frac{255^2}{\text{MSE}}\ \text{dB}.\]

Identical integer math gives \(\text{MSE}=0\) (PSNR \(=\infty\)); a difference of a few LSB from rounding/border choices gives a high but finite PSNR you can predict.

Procedure

Part A — Host: send the image row-major.

1. Load and convert to 8-bit gray, send dimensions in a header frame, then stream one row per block through the Lab 9.1 harness:

from PIL import Image
import numpy as np
img = np.asarray(Image.open("in.png").convert("L"), dtype=np.uint8)   # H×W
send_header(img.shape)                       # H, W, kernel id, border policy
for row in img:                              # one block per row
    send_block(row.tobytes())

Part B — Firmware: separable convolution over a 3-row window.

2. Keep a ring of three input rows. On each new row, run the row (horizontal) 1-D pass into the window, then the column (vertical) 1-D pass across the three buffered rows to produce one output row. Illustrative kernel (structure only):

/* rows[3][W] hold i-1, i, i+1 after the horizontal pass.  Gaussian [1 2 1]/4 sep. */
for (int j = 0; j < W; ++j) {
    int32_t acc = rows[0][j] + 2*rows[1][j] + rows[2][j];   // vertical pass
    out_row[j] = (uint8_t)(acc >> 2);                        // normalize /4, saturate
}
/* Sobel: Gx = sep([1 2 1]^T, [1 0 -1]);  Gy = transpose. mag = sqrt(gx*gx+gy*gy) */

3. Handle top/bottom rows with the chosen border policy (replicate is simplest). Stream each finished output row back.

Part C — Host: reassemble and score.

4. Reassemble output rows into an array, save out.png, and score against the reference with the same border mode:

from scipy.ndimage import convolve1d
# Reference MUST be the SAME integer kernel the device runs — a [1,2,1]/4 binomial
# blur is NOT a true Gaussian of any sigma, so scoring against gaussian_filter(sigma)
# would guarantee a nonzero "error" that is really just a different kernel.
k = np.array([1.0, 2.0, 1.0]) / 4.0
tmp = convolve1d(img.astype(np.float64), k, axis=1, mode="nearest")   # horizontal pass
ref = convolve1d(tmp,                    k, axis=0, mode="nearest")   # vertical pass
mse = np.mean((out.astype(np.float64) - ref)**2)
psnr = 10*np.log10(255.0**2 / mse) if mse > 0 else np.inf
maxerr = np.max(np.abs(out.astype(int) - np.rint(ref).astype(int)))

Deliverable & expected results

• blur.png and edges.png from the device, the reference images, and the PSNR / max-abs-error for each.

For a 3×3 separable Gaussian (\([1,2,1]/4\) each pass) and Sobel on an 8-bit image, integer device math vs. a matched integer reference:

Quantity	Predicted	Measured
Separable vs. direct multiply count (3×3)	\(2k\) vs \(k^2\) per pixel = 6 vs 9 → 33% fewer	…
Blur PSNR vs. reference (matched rounding)	\(\gtrsim 45\) dB (few-LSB rounding) or \(\infty\) if bit-exact	…
Blur max abs error	\(\le 1\)–2 LSB	…
Sobel edge-map max abs error	\(0\) (integer, matched border)	…
STM32 SRAM for 3-row window (W=512)	\(\approx 3\times512\) B + out row ≈ 2 KB	…
Full 512×512 image in SRAM	256 KB > 128 KB → must stream	…

Analysis & reconciliation

If the interior of the blur matches the reference but the borders are off, your device border policy and the reference mode= disagree — align them and the PSNR jumps. A uniform small error across the whole image is rounding: integer >>2 truncates where gaussian_filter rounds — switch to round-to-nearest ((acc + 2) >> 2) and watch the max-abs-error drop to 0. A structured error (e.g. every column shifted) is an indexing/transpose bug in the separable column pass, not quantization. Reconcile the measured multiply count against the \(2k\)-vs-\(k^2\) (6-vs-9) separable prediction — that speedup is the whole reason to factor the kernel (Course 1 Part I’s rank-1 view). For Sobel, integer math with a matched border should be exactly the reference, so any nonzero max-abs-error there is a bug to find, not a bound to accept.

Going further

• Add reflect and zero border modes and quantify their effect on edge PSNR near the frame boundary.
• Do the Gaussian at Q15 fixed point and compare its PSNR to the integer version — the 2-D analog of Lab 9.2’s fixed-point study.
• Move a 1024×1024 image to the Pi 5 or Jetson (over the Lab 9.1 Jetson transport) and hold the whole frame; compare wall-clock to the STM32’s streamed version and note where the memory-vs-throughput tradeoff flips. On the Jetson, add the GPU rung — cupyx.scipy.ndimage.gaussian_filter/sobel on the full frame — and carry the three-way number (STM32 streamed / SBC CPU / Jetson GPU) into Lab 9.5, which runs this per-frame at video rate.
• Chain blur → Sobel (edge detection on a denoised image) entirely in the streaming pipeline — the exact per-frame operation you’ll run on video in Lab 9.5.