Lab 9.5 — Real-Time Video Frame Pipeline (Pi 5 / Jetson)

← Course 2 syllabus · Bonus Module 9 · Prev: « Lab 9.4 · Next: Lab 9.6 »

Goal

The media-signal-processing capstone: extend the file-round-trip discipline of Module 9 from a single image to a stream of video frames, and run it on the edge accelerators from Module 8. The host reads frames from a real MP4, streams them to the Pi 5 (CPU/OpenCV) or Jetson Orin Nano (CUDA/TensorRT), the device runs a per-frame real-time pipeline — grayscale + Sobel/Canny edges and optical flow on the Pi 5, or a small segmentation/detection CNN on the Jetson GPU — and returns processed frames the host reassembles into an output MP4 and verifies (per-frame PSNR vs. a reference + a visual check). You measure the three numbers that decide whether an embedded vision pipeline is shippable: FPS, per-frame latency, and end-to-end throughput, Pi 5 vs. Jetson. This lab ties Module 9’s “stream a file, process on-device, verify against a reference” workflow to Module 8’s accelerators; Lab 9.6 then goes a layer deeper into the motion between frames.

Equipment & parts

  • Raspberry Pi 5 (CPU/OpenCV path) and/or Jetson Orin Nano (CUDA/TensorRT path).
  • Host Mac with numpy opencv-python for read/reassembly and the reference pipeline.
  • A short test MP4 (a few hundred frames, e.g. 640×480 @ 30 fps). Something with real motion (a pan, a moving object) so optical flow and edges have content.
  • USB link (or TCP over Ethernet — for full frames the framed-block harness rides on TCP just as easily as on serial; see below).

Wiring & bench setup

No MCU and no bench instruments — the “wiring” is the link between the Mac and the SBC. Prefer wired gigabit Ethernet (Mac and Pi 5 / Jetson on the same LAN, or a direct cable): at ~9 MB/s of raw gray frames a TCP socket on GbE has headroom to spare, while a USB-serial path must first prove its measured throughput ≥ frame rate × frame size.

flowchart LR
  MP4["media/in/clip.mp4"]
  HOST["Host: cv2.VideoCapture<br/>Lab 9.1 framing + credits over TCP"]
  SBC["Pi 5 edges/flow<br/>or Jetson TensorRT CNN"]
  OUT["media/out/edges.mp4<br/>(cv2.VideoWriter)"]
  REF["Host reference: OpenCV ops<br/>or onnxruntime CNN — per-frame PSNR"]
  MP4 --> HOST
  HOST -- "gray frames + seq" --> SBC
  SBC -- "processed frames" --> HOST
  HOST --> OUT -- "compare raw frames" --> REF

flowchart LR
  MP4["media/in/clip.mp4"]
  HOST["Host: cv2.VideoCapture<br/>Lab 9.1 framing + credits over TCP"]
  SBC["Pi 5 edges/flow<br/>or Jetson TensorRT CNN"]
  OUT["media/out/edges.mp4<br/>(cv2.VideoWriter)"]
  REF["Host reference: OpenCV ops<br/>or onnxruntime CNN — per-frame PSNR"]
  MP4 --> HOST
  HOST -- "gray frames + seq" --> SBC
  SBC -- "processed frames" --> HOST
  HOST --> OUT -- "compare raw frames" --> REF

Safety & don’t-break-it

Data/compute lab; the hazards are backpressure, memory bandwidth, and reference mismatch:

  • A video stream will overrun a naive link instantly. A 640×480 gray frame is 307 KB; at 30 fps that’s ~9 MB/s of raw frames — far past a UART VCP. Use USB CDC or TCP and keep the Lab 9.1 flow control (credits) so the sender blocks instead of dropping frames. A dropped frame corrupts the reassembled MP4’s timing.
  • Don’t confuse codec loss with pipeline error. Re-encoding output frames to MP4 (lossy H.264) adds its own PSNR hit unrelated to your processing. For the verification comparison, compare the raw processed frames against the reference frames, and treat the MP4 as the human-facing artifact — or write a lossless container for scoring.
  • Jetson GPU memory and power. Batching frames helps throughput but a too-large batch OOMs the Orin Nano’s shared memory; watch tegrastats. Keep the board’s power mode fixed across a benchmark run or your FPS numbers aren’t comparable (the Lab 8.5 rule).
  • Synchronize frame ordering. Tag every frame with a sequence number (you already have seq from Lab 9.1) so the host reassembles frames in order even if the device pipelines/reorders internally.

Project & environment setup

No STM32 firmware — this lab is SBC + host only, and Lab 9.6 reuses this exact setup. Device-side scripts live in the repo under labs/lab-9-5/edge/ (the SBC-side tree, as in Module 8) and are synced/cloned onto the board.

Pi 5 (CPU path — plain SciPy/OpenCV, no GPU):

sudo apt install python3-opencv ffmpeg     # or: python3 -m venv ~/venv && pip install opencv-python numpy scipy

Jetson Orin Nano (GPU path): JetPack already ships CUDA, TensorRT (trtexec), and OpenCV — nothing to install for the engine build; convert the pretrained ONNX model with the Module 8 flow (Lab 8.2). For GPU array work outside TensorRT, install CuPy (pip install cupy-cuda12x, matching JetPack’s CUDA): cupyx.scipy.signal / cupyx.scipy.ndimage mirror the SciPy API — same calls, swap the import, and any per-frame filtering stage runs on the GPU. The identical code path on the Pi 5 stays plain scipy/OpenCV on CPU — that one-import contrast is the CPU-vs-GPU comparison in miniature.

Host (Mac) — course venv plus video I/O and the CNN reference:

source venv/bin/activate
pip install opencv-python onnxruntime   # cv2.VideoCapture/VideoWriter + reference ops; onnxruntime = host CNN reference
brew install ffmpeg                     # optional: probe/re-mux test clips
mkdir -p labs/lab-9-5/host labs/lab-9-5/captures

Keep this lab’s reconciliation in labs/lab-9-5/host/analysis.ipynb — the notebook convention — and export final figures next to it.

Where results go:

Artifact Path
Bench note (FPS / latency / throughput tables) labs/lab-9-5/notes.md
Source clip media/in/clip.mp4
Reassembled outputs media/out/edges.mp4 (Pi 5) · media/out/cnn.mp4 (Jetson)
Per-frame PSNR series plot labs/lab-9-5/host/psnr-series.png
Timing logs (send / receive / compute stamps) labs/lab-9-5/captures/timing.csv
Device-side pipeline scripts labs/lab-9-5/edge/

Background

Per-frame throughput metrics. For a pipeline that processes frames at rate \(f\) frames/s,

\[\text{FPS} = f, \qquad T_{\text{frame}} = \frac{1}{f}\ \text{(steady-state period)}.\]

Latency vs. throughput are different numbers. Latency is the time from a frame entering the host streamer to its processed result returning — it includes transport each way plus device compute:

\[L = t_{\text{tx}} + t_{\text{compute}} + t_{\text{rx}}.\]

A pipelined implementation can hold FPS \(= 1/t_{\text{compute}}\) (throughput bound by the slowest stage) while \(L\) is larger than \(1/\text{FPS}\) because several frames are in flight. End-to-end throughput is the sustained returned-pixel rate:

\[\text{throughput} = \text{FPS} \times (W\cdot H)\ \text{pixels/s} = \text{FPS}\times(W\cdot H\cdot \text{bytes/pixel})\ \text{B/s}.\]

Real-time means \(\text{FPS} \ge\) the source frame rate (e.g. 30) with \(L\) small enough for the application.

The per-frame operations. Sobel/Canny edges are the 2-D convolutions of Lab 9.4 (Canny adds non-max suppression + hysteresis). Optical flow (Lucas–Kanade) assumes brightness constancy, \(I(x,y,t)=I(x{+}u,y{+}v,t{+}1)\), linearizes to \(I_x u + I_y v + I_t = 0\), and solves a per-window \(2\times2\) least-squares system

\[\begin{bmatrix}\sum I_x^2 & \sum I_x I_y\\ \sum I_x I_y & \sum I_y^2\end{bmatrix}\begin{bmatrix}u\\v\end{bmatrix} = -\begin{bmatrix}\sum I_x I_t\\ \sum I_y I_t\end{bmatrix},\]

the Course 1 Lesson 6 normal equations, one per pixel neighborhood. On the Jetson, the per-frame stage is instead a small CNN whose forward pass runs in fp16/int8 on the GPU via TensorRT — a pretrained detection/segmentation network converted with the Module 8 TensorRT flow (see Procedure), since Module 8 itself never trained a vision detector.

PSNR per frame is exactly the Lab 9.4 metric applied to each returned frame \(\hat{I}_t\) vs. reference \(I_{\text{ref},t}\); you report the per-frame PSNR series (mean and worst frame).

Procedure

Part A — Host: stream frames from the MP4.

  1. Read frames and stream each as a block (over USB CDC or TCP), reusing the Lab 9.1 framing + credits:
import cv2
cap = cv2.VideoCapture("in.mp4")
fps = cap.get(cv2.CAP_PROP_FPS)
seq = 0
while True:
    ok, frame = cap.read()
    if not ok: break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)     # send gray to save bandwidth
    send_block(seq, gray.tobytes()); seq += 1
    processed = recv_frame()                            # ordered by seq
    writer_append(processed)

Part B — Device pipeline.

  1. Pi 5 (CPU/OpenCV): per frame, grayscale → cv2.Sobel/cv2.Canny edges, and calcOpticalFlowFarneback (or LK on tracked points) between consecutive frames; return the edge map (and/or a flow visualization).
  2. Jetson (CUDA/TensorRT): per frame, run a small segmentation/detection CNN engine, return the mask/overlay. Sourcing the model: Module 8 trained keyword/anomaly nets, not a vision detector, so take a pretrained off-the-shelf model — e.g. MobileNet-SSD (detection) or a small mobilenet-backboned segmentation net (U-Net / DeepLabV3-MobileNet) from the ONNX Model Zoo / torchvision — and convert it to an fp16/int8 TensorRT engine using the same Module 8 flow (trtexec + the int8 calibrator from Lab 8.2). The deployment skill is what transfers; the model itself is pretrained here. Use CUDA streams / batching to keep the GPU fed.
  3. Timestamp receipt and completion per frame on the device to separate transport from compute.

Part C — Reassemble and verify.

  1. Reassemble returned frames in seq order into out.mp4 with cv2.VideoWriter at the source fps.
  2. Run the host reference (same OpenCV ops, or the same CNN in onnxruntime) on the original frames; compute per-frame PSNR of device-vs-reference on the raw frames, plus a visual side-by-side.
  3. Measure FPS (frames returned / wall-clock), per-frame latency (send→receive timestamp), and throughput (pixels/s), for Pi 5 and Jetson separately.

Deliverable & expected results

  • out.mp4, the per-frame PSNR series (mean + worst-frame), and a Pi 5 vs. Jetson table of FPS / latency / throughput.

For a 640×480 gray source at 30 fps (illustrative — Predicted from the metric formulas; fill Measured on the bench):

Quantity Predicted Measured
Frame size (640×480×1 B) 307,200 B …
Real-time frame period (\(1/30\)) 33.3 ms …
Raw returned throughput at 30 fps ≈ 9.2 MB/s …
Pi 5 edges+flow FPS source-limited (≥30) or lower — measure …
Jetson CNN FPS (fp16/int8, TensorRT) ≥ Pi 5; measure …
Per-frame latency \(L=t_{tx}+t_{cmp}+t_{rx}\) compute-dominated; measure …
Mean per-frame PSNR vs. reference \(\infty\) (edges, matched int) / high finite (CNN fp16) …

Analysis & reconciliation

First separate the numbers: FPS is throughput, latency is delay — a pipelined Jetson can hold 30+ FPS while each frame’s \(L\) spans several frame periods because several are in flight. Reconcile measured throughput against \(\text{FPS}\times W\cdot H\); a shortfall vs. the link’s raw ceiling means the transport, not compute, is the bottleneck (send gray not RGB, widen the credit window, move UART→USB/TCP). If per-frame PSNR is high but not infinite on the OpenCV path, suspect the MP4 re-encode contaminating the comparison — score raw frames instead; on the CNN path, finite PSNR is expected (fp16/int8 vs. the fp32 reference) and its magnitude is your quantization-accuracy budget from Module 8. A worst-frame PSNR far below the mean usually flags a dropped/reordered frame (check seq continuity) rather than a compute error. Compare Pi 5 (CPU/OpenCV) vs. Jetson (CUDA/TensorRT): the Jetson should win on the CNN stage by the accelerator margin you measured in Lab 8.5, while classical edges/flow may be competitive on the Pi 5 CPU — that crossover is the engineering point of the whole edge-ML module.

Going further

  • Add temporal stability: run the edge/flow output through a short IIR across frames and quantify flicker reduction — 1-D filtering (Lab 9.2) applied along the time axis of video.
  • Batch frames on the Jetson and plot FPS vs. batch size until GPU memory saturates; find the throughput knee.
  • Stream RGB instead of gray and re-measure — quantify how much of your throughput budget color costs.
  • Close the module’s loop: feed a live camera instead of an MP4 into the same harness, proving the file-based regression pipeline and the real-time pipeline are the same code — the reproducibility payoff of Module 9’s whole design.

From Module 0’s power supply to a verified real-time video pipeline on an edge GPU, every result was predicted by hand, built, measured, and reconciled against a reference. Lab 9.6 closes the course by estimating and spending the motion between frames.