Lab 9.4 — Image 2-D Convolution on the Embedded Target

← Course 2 syllabus · Bonus Module 9 · Prev: « Lab 9.3 · Next: Lab 9.5 »

Goal

Take the streaming discipline into two dimensions: the host sends a real grayscale image to the device, which runs 2-D convolution — a separable Gaussian blur and a Sobel edge detector — on it, block by block, and streams the result back for the host to save as a PNG and check against scipy.ndimage / OpenCV. On the memory-tight STM32 the key trick is streaming the image row-by-row through a small sliding window of rows, never holding the whole picture, which is exactly how line-buffer image pipelines work in real camera/vision firmware; larger images move to the Pi 5 / Jetson. The verification is quantitative: PSNR and max-abs-error vs. a reference implementation, so “the blur looks right” becomes a number. This is the image-signal-processing lab of the module and the bridge to the video pipeline in Lab 9.5.

Recommended reading

• Lyons — convolution and FIR filtering; read it with the 1-D→2-D extension in mind (a separable 2-D filter is two 1-D convolutions).
• O-DSP — convolution/filtering fundamentals and finite-wordlength/scaling effects that carry over from 1-D to 2-D.
• Course 1 Weeks 1–3 (../course1/index.qmd#week-1) — linear algebra: convolution as a linear operator (a Toeplitz/circulant matrix action), separability as an outer-product / rank-1 factorization of the kernel, and why that factorization saves work.
• Reference-implementation docs: scipy.ndimage.gaussian_filter / scipy.ndimage.sobel, or OpenCV cv2.GaussianBlur / cv2.Sobel.

Equipment & parts

• STM32 Nucleo-L476RG for small images (see the SRAM budget below), or Raspberry Pi 5 / Jetson Orin Nano for larger images.
• Host Mac with numpy scipy pillow (and optionally opencv-python).
• A grayscale test image — a PGM (trivial to parse) or a PNG loaded via Pillow and converted to 8-bit gray. Start small (e.g. 128×128) for the STM32.

Safety & don’t-break-it

Data lab; the constraints are memory and numeric scaling:

• The L476RG has 128 KB SRAM — you cannot hold a big image. A 512×512 8-bit image is 256 KB, already twice the SRAM. This is why you stream row-by-row and keep only a 3-row sliding window (a few KB) plus one output row. Sizing the whole frame into RAM is the mistake that won’t even link.
• Border handling must be decided, not accidental. Reading img[-1] or past the last row silently reads garbage. Choose a border policy (zero-pad, replicate/clamp, or reflect) and implement it explicitly at the top/bottom/left/right edges — and use the same policy in the host reference or the PSNR will be wrong at the borders.
• Fixed-point scaling overflows fast in 2-D. A 3×3 kernel sums nine products; an unnormalized Sobel or an integer Gaussian can exceed 8 bits. Accumulate in int16/int32, normalize (divide by the kernel sum for the blur), and saturate back to uint8 — never let the byte wrap.
• Keep the Lab 9.1 flow control: a row is a block; don’t let convolution compute stall the RX DMA.

Background

2-D convolution. For an image \(I\) and kernel \(K\) of size \((2a{+}1)\times(2b{+}1)\):

\[(I * K)[i,j] \;=\; \sum_{u=-a}^{a}\sum_{v=-b}^{b} K[u,v]\,I[i-u,\,j-v].\]

A direct \(M\times N\)-pixel convolution with a \(k\times k\) kernel costs \(O(MNk^2)\) multiply-adds.

Separability. A Gaussian kernel factors as an outer product of two 1-D Gaussians,

\[K = g\,g^{\mathsf T}, \qquad g[t] = \tfrac{1}{Z}\,e^{-t^2/(2\sigma^2)},\]

so the 2-D convolution becomes two 1-D passes — convolve each row with \(g\), then each column with \(g\):

\[I * K \;=\; \big(I *_{\text{rows}} g\big) *_{\text{cols}} g.\]

This drops the cost to \(O(MNk)\) — a factor \(k\) saving — and, in Course 1 Week 1–3 terms, is just the rank-1 factorization of the kernel matrix. The 3-row sliding window on the STM32 is exactly what the column pass needs: a \(3\times3\) kernel touches only rows \(i{-}1,i,i{+}1\), so you buffer three input rows, emit one output row, slide.

Sobel edges. The Sobel operator estimates the image gradient with two separable \(3\times3\) kernels:

\[G_x = \begin{bmatrix} 1 & 0 & -1\\ 2 & 0 & -2\\ 1 & 0 & -1\end{bmatrix} = \begin{bmatrix}1\\2\\1\end{bmatrix}\begin{bmatrix}1&0&-1\end{bmatrix}, \qquad G_y = G_x^{\mathsf T},\]

and the edge (gradient-magnitude) image is

\[G[i,j] = \sqrt{G_x[i,j]^2 + G_y[i,j]^2}.\]

Both \(G_x\) and \(G_y\) are separable (a smoothing \([1,2,1]\) and a difference \([1,0,-1]\)), so the same line-buffer machinery applies.

PSNR. To compare the device output \(\hat{I}\) against the reference \(I_{\text{ref}}\) (8-bit, peak \(=255\)):

\[\text{MSE} = \frac{1}{MN}\sum_{i,j}\big(\hat{I}[i,j]-I_{\text{ref}}[i,j]\big)^2, \qquad \text{PSNR} = 10\log_{10}\frac{255^2}{\text{MSE}}\ \text{dB}.\]

Identical integer math gives \(\text{MSE}=0\) (PSNR \(=\infty\)); a difference of a few LSB from rounding/border choices gives a high but finite PSNR you can predict.

Procedure

Part A — Host: send the image row-major.

1. Load and convert to 8-bit gray, send dimensions in a header frame, then stream one row per block through the Lab 9.1 harness:

from PIL import Image
import numpy as np
img = np.asarray(Image.open("in.png").convert("L"), dtype=np.uint8)   # H×W
send_header(img.shape)                       # H, W, kernel id, border policy
for row in img:                              # one block per row
    send_block(row.tobytes())

Part B — Firmware: separable convolution over a 3-row window.

2. Keep a ring of three input rows. On each new row, run the row (horizontal) 1-D pass into the window, then the column (vertical) 1-D pass across the three buffered rows to produce one output row. Illustrative kernel (structure only):

/* rows[3][W] hold i-1, i, i+1 after the horizontal pass.  Gaussian [1 2 1]/4 sep. */
for (int j = 0; j < W; ++j) {
    int32_t acc = rows[0][j] + 2*rows[1][j] + rows[2][j];   // vertical pass
    out_row[j] = (uint8_t)(acc >> 2);                        // normalize /4, saturate
}
/* Sobel: Gx = sep([1 2 1]^T, [1 0 -1]);  Gy = transpose. mag = sqrt(gx*gx+gy*gy) */

3. Handle top/bottom rows with the chosen border policy (replicate is simplest). Stream each finished output row back.

Part C — Host: reassemble and score.

4. Reassemble output rows into an array, save out.png, and score against the reference with the same border mode:

from scipy.ndimage import gaussian_filter, sobel
ref = gaussian_filter(img.astype(np.float64), sigma=..., mode="nearest")
mse = np.mean((out.astype(np.float64) - ref)**2)
psnr = 10*np.log10(255.0**2 / mse) if mse > 0 else np.inf
maxerr = np.max(np.abs(out.astype(int) - np.rint(ref).astype(int)))

Deliverable & expected results

• blur.png and edges.png from the device, the reference images, and the PSNR / max-abs-error for each.

For a 3×3 separable Gaussian (\([1,2,1]/4\) each pass) and Sobel on an 8-bit image, integer device math vs. a matched integer reference:

Quantity	Predicted	Measured
Separable vs. direct multiply count (3×3)	\(6k\) vs \(9k\) per pixel → 33% fewer	…
Blur PSNR vs. reference (matched rounding)	\(\gtrsim 45\) dB (few-LSB rounding) or \(\infty\) if bit-exact	…
Blur max abs error	\(\le 1\)–2 LSB	…
Sobel edge-map max abs error	\(0\) (integer, matched border)	…
STM32 SRAM for 3-row window (W=512)	\(\approx 3\times512\) B + out row ≈ 2 KB	…
Full 512×512 image in SRAM	256 KB > 128 KB → must stream	…

Analysis & reconciliation

If the interior of the blur matches the reference but the borders are off, your device border policy and the reference mode= disagree — align them and the PSNR jumps. A uniform small error across the whole image is rounding: integer >>2 truncates where gaussian_filter rounds — switch to round-to-nearest ((acc + 2) >> 2) and watch the max-abs-error drop to 0. A structured error (e.g. every column shifted) is an indexing/transpose bug in the separable column pass, not quantization. Reconcile the measured multiply count against the \(6k\)-vs-\(9k\) separable prediction — that speedup is the whole reason to factor the kernel (Course 1 Week 1–3’s rank-1 view). For Sobel, integer math with a matched border should be exactly the reference, so any nonzero max-abs-error there is a bug to find, not a bound to accept.

Going further

• Add reflect and zero border modes and quantify their effect on edge PSNR near the frame boundary.
• Do the Gaussian at Q15 fixed point and compare its PSNR to the integer version — the 2-D analog of Lab 9.2’s fixed-point study.
• Move a 1024×1024 image to the Pi 5 and hold the whole frame; compare wall-clock to the STM32’s streamed version and note where the memory-vs-throughput tradeoff flips.
• Chain blur → Sobel (edge detection on a denoised image) entirely in the streaming pipeline — the exact per-frame operation you’ll run on video in Lab 9.5.