Add the module’s vision capstone: a live camera on the Jetson Orin Nano, a pretrained object detector running on the GPU through TensorRT, and boxes drawn on the stream in real time. Where Labs 8.1–8.4 put learned DSP on 1-D signals, this lab does it on the 2-D streaming signal a camera produces — and the engineering questions are the same ones the whole course keeps asking: what is the per-frame deadline, where does the time actually go (capture → preprocess → inference → NMS → draw), what does fp16 quantization cost in accuracy, and how do you verify a deployed model against a reference implementation instead of trusting that “the boxes look right.” You reuse the Lab 8.2 ONNX→TensorRT deployment flow on a detection model, and the Lab 8.5 benchmarking discipline on a video-rate workload. This is also the live-camera sibling of Lab 9.5, which runs a per-frame CNN on file-streamed video — same pipeline, opposite end of the latency story.
Recommended reading
Szeliski Ch. 6 (2nd ed. — see the reading key; the PDF is free for personal use at szeliski.org) — recognition: image classification, object detection (two-stage vs single-stage detectors, anchors, non-maximum suppression, mAP), and semantic segmentation. This is the theory chapter behind the model you deploy. The 1st edition (the copy in hand) covers only the classical detectors in its recognition chapter — Viola–Jones faces and HOG pedestrians — worth reading as the “before deep learning” picture the modern detectors replaced.
Such Ch. 4 (Embedded AI — see the reading key) — the chapter’s CNN project puts person detection on an MCU-class board (Arduino Nicla Vision). Read it as the constrained-target contrast for this lab: the same task two-plus orders of magnitude of compute below the Orin Nano, and what that forces the model to give up.
Course 1 Lesson 42 — image fundamentals and spatial filtering (what a frame is, and what the backbone’s early conv layers compute), and edges/corners/descriptors (what detection generalizes). → ../course1/index.qmd#lesson-42
Course 1 Lesson 43 — learned DSP: the convolutional networks the detector is built from (G&W Ch. 12’s neural-network treatment as the deeper reading). → ../course1/index.qmd#lesson-43
Optional practical reference: NVIDIA’s jetson-inference / “Hello AI World” repo — the canonical worked example of camera→detectNet→display on Jetson devkits; useful as a known-good cross-check for your own pipeline, not as a substitute for building it.
Equipment & parts
Jetson Orin Nano with JetPack (CUDA, TensorRT, and trtexec ship with it), heatsink/fan fitted.
A camera, per the bench inventory: a USB UVC webcam (recommended first path — plug-and-play V4L2), or a CSI camera (Raspberry Pi Camera Module v2 / IMX219, with the 22-pin FFC cable the Orin Nano devkit’s CAM connectors need).
Raspberry Pi 5 (optional but recommended) for the CPU-inference comparison — same camera, same model, onnxruntime.
M-series Mac as the host: model export to ONNX and the reference-implementation run (course venv + --group ml).
A monitor/desktop session on the Jetson for the live view (you have desktop login on both boards), or run headless and write an annotated MP4 instead — both paths are in the Procedure.
No bench instruments and no breadboard — unless you take the Going-further glass-to-glass latency measurement, which brings the Nucleo and one LED back out.
Wiring & bench setup
The “wiring” is one camera connection plus the LAN you already use to reach the boards:
flowchart LR CAM["Camera<br/>USB UVC webcam<br/>or CSI IMX219"] JET["Jetson Orin Nano<br/>capture → preprocess →<br/>TensorRT infer → NMS → draw"] DISP["Desktop session / annotated MP4"] MAC["Mac<br/>model export + reference boxes"] CAM -- "USB-A, or CAM0 22-pin FFC" --> JET JET --> DISP MAC -. "ONNX model + test frames over LAN" .-> JET
flowchart LR
CAM["Camera<br/>USB UVC webcam<br/>or CSI IMX219"]
JET["Jetson Orin Nano<br/>capture → preprocess →<br/>TensorRT infer → NMS → draw"]
DISP["Desktop session / annotated MP4"]
MAC["Mac<br/>model export + reference boxes"]
CAM -- "USB-A, or CAM0 22-pin FFC" --> JET
JET --> DISP
MAC -. "ONNX model + test frames over LAN" .-> JET
From
To
Notes
USB webcam
any Jetson USB-A port
direct, not through an unpowered hub; enumerates as /dev/video*
(CSI option) IMX219 camera
Jetson CAM0/CAM1 connector
22-pin FFC; insert with the board powered off, contacts oriented per the devkit silkscreen, latch closed
(Pi 5 comparison) same USB camera
Pi 5 USB-A
identical /dev/video* path; the CSI option on the Pi needs the Pi 5 camera cable and the libcamera stack instead
Safety & don’t-break-it
Nothing here is electrically hazardous — the risks are to the parts and to your measurements.
CSI/FFC connectors are fragile and not hot-pluggable. Power the board off before inserting or removing the ribbon; open the latch fully, seat the cable square, and never pull on the ribbon itself. A misaligned FFC can short the camera’s rails.
ESD care with the bare camera module: handle by the PCB edges, and ground yourself before touching the connector — same habit as the DIP chips in Module 4.
Fix the power mode before every benchmark run (sudo nvpmodel -m 0 && sudo jetson_clocks, the Lab 8.5 rule) and keep the fan running — a thermally throttled GPU quietly invalidates every FPS number you record. Watch tegrastats during long runs.
Don’t trust a demo that only “looks right.” The don’t-break-it rule of this lab is methodological: every deployed engine is verified against the reference implementation (Part B) before its speed counts. Fast-but-wrong is wrong — same rule as Module 6.
Project & environment setup
Work lands in labs/lab-8-6/ — host/ for the Mac-side export and reference run, edge/ for everything that runs on the boards:
Mac (host) — course venv with the ML group (uv sync --group ml): PyTorch/torchvision (or the ultralytics package if you pick a YOLO-class model) to obtain a pretrained COCO detector and export it to ONNX, plus onnxruntime for the reference run. Owner writes the export/reference scripts (host/export_onnx.py, host/reference_boxes.py).
Jetson Orin Nano — TensorRT and trtexec (at /usr/src/tensorrt/bin/trtexec) come with JetPack; add the Python side in the same venv you built for Lab 8.1:
source ~/edge-venv/bin/activatepip install numpy opencv-python onnxruntime # capture + drawing + CPU-baseline inference# TensorRT's Python bindings ship with JetPack (system packages) — create the venv# with --system-site-packages, per docs/edge-setup.md, so `import tensorrt` resolves.
Pi 5 (comparison path) — the Lab 8.1 venv plus pip install onnxruntime opencv-python.
Camera sanity commands (both boards): v4l2-ctl --list-devices and v4l2-ctl -d /dev/video0 --list-formats-ext for USB; on the Jetson CSI path the capture element is GStreamer’s nvarguscamerasrc (test with gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! xvimagesink in a desktop session).
Where results go:
Artifact
Path
Bench note (timing tables + verification report)
labs/lab-8-6/notes.md
Exported ONNX model + the TensorRT engine built from it
labs/lab-8-6/edge/model.onnx, model-fp16.engine
Test frames + reference boxes (Mac onnxruntime output, JSON)
What a detector computes. An object detector maps a frame to a set of \((\text{box}, \text{class}, \text{score})\) triples. Modern single-stage detectors (the SSD/YOLO family you’ll deploy) run one convolutional backbone over the image and predict, at every cell of one or more feature-map grids, class scores plus box offsets — thousands of candidate boxes per frame in a single forward pass. Two-stage detectors (Faster R-CNN family) first propose regions, then classify each; more accurate per FLOP at the top end, slower at the edge. Szeliski Ch. 6 is the map of this territory.
Non-maximum suppression. The raw output contains many overlapping candidates per object. NMS keeps the highest-scoring box and deletes neighbors whose overlap with it exceeds a threshold, where overlap is intersection-over-union:
IoU does double duty in this lab: it is the NMS overlap criterion and the match criterion you verify the deployed engine with (a deployed box “matches” a reference box when their IoU exceeds a threshold and the classes agree). Detection benchmarks score mAP — precision/recall averaged over classes and IoU thresholds — but for a deployment check against the same model’s reference output, per-box IoU matching is the sharper tool.
The per-frame budget. A camera at frame rate \(f\) gives a budget of \(1/f\) per frame end to end — at 30 fps, 33.3 ms for capture + preprocess + inference + NMS + draw. Preprocessing is honest work at video rates: a 640×640 RGB input tensor is \(640 \times 640 \times 3 = 1.2\) MB per frame that must be resized, converted, normalized, and copied to the GPU — at 30 fps that is ~37 MB/s of pixel handling before the network runs. If inference alone exceeds the budget, the display can still stay smooth by dropping frames — the detector then runs at its own rate on the freshest frame, and the number to report honestly is both rates plus the end-to-end latency, not just “FPS.”
fp16 and the quantization trade. TensorRT builds an inference engine from the ONNX graph, fusing layers and (with --fp16) running in half precision — roughly half the memory traffic. The cost is small numeric drift in scores and box coordinates: the deployment question is whether that drift moves any box enough to break an IoU match against the fp32 reference. That is a measurable question, and Part C measures it. (INT8 pushes the same trade further and needs a calibration set — Going further.)
Latency vs throughput, one more time. This is Lab 8.1’s lesson at video scale: batching frames raises throughput but delays every frame in the batch, and a live detector is latency-bound — batch size 1 is the honest configuration for the live loop, which is exactly why its FPS is lower than the throughput number trtexec reports.
Procedure
Part A — Camera bring-up.
USB path: plug the webcam into the Jetson, v4l2-ctl --list-devices, and grab a frame from Python (cv2.VideoCapture(0), set 1280×720 @ 30 fps, read, cv2.imwrite). Confirm the achieved format/rate with --list-formats-ext — UVC cameras silently fall back to MJPG or a lower rate at some resolutions; record what you actually got.
CSI path (optional): power off, seat the 22-pin FFC, boot, and test with the nvarguscamerasrc GStreamer pipeline (see setup). In OpenCV, open the same pipeline string with cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER).
Save 5–10 varied test frames (people, chairs, cups, a monitor — COCO classes you have on hand) to captures/test-frames/. These are the verification set for everything below.
Part B — Model export and the reference run (Mac).
Pick a small pretrained COCO detector — an SSD-MobileNet or compact YOLO-class model — and export it to ONNX at a fixed input size (e.g. 640×640; note whether the export bundles NMS in-graph or leaves it to you, and keep that choice consistent across every platform).
Run the ONNX model with onnxruntimeon the Mac over the Part A test frames. Draw and eyeball the boxes once, then save the raw \((\text{box},\text{class},\text{score})\) lists to reference-boxes.json. This is the lab’s reference implementation — every later run is judged against it, in the same role SciPy played in Module 6.
Part C — Deploy on the Jetson: CPU baseline, then TensorRT.
CPU baseline: run the same ONNX with onnxruntime on the Jetson (CPU provider) over the test frames. Verify: every reference box matched at IoU ≥ 0.9 with the same class (scores may differ in the last decimals). Time inference-only over many repetitions → p50/p99.
Build the engine:trtexec --onnx=model.onnx --fp16 --saveEngine=model-fp16.engine, and keep trtexec’s own reported throughput/latency summary — it is the vendor-tool number your own measurement should reconcile with.
Verify the engine: run the test frames through the engine (TensorRT Python API; owner writes the runner) and match boxes against reference-boxes.json. Record the worst-case IoU drift and any score changes — this is the measured cost of fp16. A dropped or spurious box at your score threshold is a finding: lower the threshold and check whether fp16 moved a borderline score across it.
Time engine inference over many repetitions (clocks pinned, batch 1) → p50/p99, alongside the trtexec number and the CPU baseline.
Part D — The live loop.
Assemble capture → preprocess → infer → NMS → draw, timing each stage separately every frame (time.monotonic_ns() around each stage into a rolling log). Show the live annotated view in the desktop session (cv2.imshow) or, headless, write live-detect.mp4 with cv2.VideoWriter.
Run ~2 minutes with clocks pinned and tegrastats logging in a second shell. Record: camera rate, detector rate, end-to-end p50/p99 latency, and per-stage medians. Walk in front of the camera; hold up known COCO objects; note qualitative behavior (missed small objects, flicker between classes) in notes.md.
Decouple the rates deliberately: if inference is slower than the camera, switch to a latest-frame-wins queue (drop stale frames) and observe display smoothness vs box staleness — then note which the application should prefer and why.
Part E — The comparison axis (Pi 5, and knobs).
Same camera, same ONNX, onnxruntime on the Pi 5 (performance governor): verification first, then the live loop. Expect the detector rate to fall well below the Jetson’s — the point is by how much, at what per-frame latency.
One knob sweep on the Jetson: input resolution (e.g. 320 vs 640 input size, re-exported) or fp32-vs-fp16 engines — hold everything else fixed, re-verify, re-time, and add the rows to the table.
Deliverable & expected results
A verified engine: the IoU-match report of TensorRT-fp16 against the Mac reference over your test frames.
The annotated live clip, and timing.csv with per-stage distributions for every platform/precision run.
The filled comparison table below.
Budget arithmetic (hand-derivable; platform rows are qualitative — your measurements are the numbers):
Quantity
Predicted
Measured
Per-frame budget at 30 fps
33.3 ms
…
Input-tensor size, 640×640×3 (bytes/frame)
1.2 MB (~37 MB/s at 30 fps)
…
Verification: reference boxes matched (CPU ORT, same ONNX)
all, IoU ≥ 0.9
…
Verification: worst-case IoU drift, fp16 engine vs reference
small; no box lost at threshold
…
Jetson TensorRT fp16, inference p50 (batch 1)
≪ CPU baseline; reconciles with trtexec
…
Live end-to-end rate vs inference-only rate
end-to-end lower (pre/post + draw)
…
Pi 5 CPU detector rate
well below Jetson
…
320-vs-640 input (or fp32-vs-fp16) sweep
smaller/faster, accuracy cost measured
…
Analysis & reconciliation
Reconcile three ways. (1) Where the time goes: the per-stage logs must add up to the end-to-end latency; if they don’t, you’re double-counting or the camera driver is buffering frames behind your back (check the capture stage’s spread — a bimodal capture time is a driver queue). Compare inference-only p50 against trtexec’s report; a large gap means your pre/post-processing or Python glue, not the engine, owns the difference. (2) Accuracy vs the reference: the CPU-ORT run should match the Mac almost exactly (same graph, same arithmetic); the fp16 engine’s drift is real but should be far inside your IoU threshold — if a box flickers in and out at the score threshold, you’ve found fp16 moving a borderline score, the concrete face of quantization error (Module 6’s Q15 lesson, relocated to a neural network). (3) The platform table: the Jetson-vs-Pi gap should roughly track the gap Lab 8.5 measured for dense conv workloads — if it’s wildly different, check that the Pi run isn’t thermally throttling and that both use the same input size and NMS placement. Close with the engineering sentence every row supports: a live detector is latency-bound at batch 1, so the accelerator’s headline throughput matters less than its worst-case single-frame time — the same p99-over-median discipline this course has applied since Lab 2.2.
Going further
Glass-to-glass latency, measured with your own bench: point the camera at an LED blinked by the Nucleo (Lab 2.1’s toggle, slowed to ~1 Hz) and timestamp LED-on to first-frame-where-detected (or to pixel change on the displayed view, filmed by a phone in slo-mo). Capture-to-display latency is the number a robotics application actually cares about, and it is larger than any row in your table.
Tracking: add a lightweight SORT-style tracker (constant-velocity Kalman filter per object + IoU assignment) on top of the detections — Lab 6.6’s filter, promoted to 2-D state, smoothing the flicker you observed in Part D.
INT8: calibrate with your own captured frames (trtexec --int8 + a calibration cache), re-verify against the reference, and extend the precision column — the full fp32→fp16→int8 accuracy/latency curve.
DeepStream: rebuild the pipeline in NVIDIA’s DeepStream GStreamer stack and compare its zero-copy capture→infer path against your OpenCV loop — where did your Python glue’s milliseconds go?
Feed it forward: run the detector on the Lab 9.5 file-streamed frames and compare live-camera vs host-in-the-loop on identical content — the two labs become one benchmark with two transports.