Lab 1.4 — Streams, Copies & Unified Memory

← Course 4 syllabus · Module 1 · Prev: « Lab 1.3 · Next: Lab 1.5 »

Goal

Lab 1.1’s Nsight timeline showed the embarrassment: copies dwarfing the kernel. This lab builds the asynchronous machinery that fixes it — streams (independent in-order queues), events (the cross-stream and CPU-visible sync points), pinned vs. pageable host memory, and copy/compute overlap via a chunked pipeline — and the Linux desktop (RTX 4090) is the textbook machine to build it on. A discrete GPU has its own GDDR6X VRAM: every “host-to-device copy” is a genuine transfer across PCIe, and hiding that transfer behind compute — pinned-host staging plus async copies overlapped on streams — is the winning production pattern this lab installs. The conveniences that promise to hide the bus don’t remove it: zero-copy mapped memory makes the kernel reach across PCIe on every access, and cudaMallocManaged services GPU page faults by migrating pages over the same bus. Measuring what those cost against explicit copies — rather than trusting folklore in either direction — is the lab’s second deliverable. The first is a Nsight Systems timeline with copy and compute genuinely overlapped. Power enters as a first-class observable: an nvidia-smi log runs alongside the pipeline.

Recommended reading

• Motta — the material on streams, asynchronous copies, and pinned/unified memory (title-level reference; confirm against the copy in hand).
• CUDA C++ Programming Guide — Asynchronous Concurrent Execution (streams, events, overlap requirements) and the Unified Memory Programming appendix — read the page-migration discussion closely; it is the authority on what cudaMallocManaged actually does on a discrete GPU.
• CUDA C++ Best Practices Guide — Asynchronous and Overlapping Transfers with Computation and the pinned-memory section.
• CUDA Runtime API reference — the host-memory entries (cudaHostAlloc and its flags, cudaHostRegister): the vendor’s own statement of what pinned and mapped memory mean; the authority to reconcile your measurements against.
• CuPy documentation — Streams and events (cp.cuda.Stream, cp.cuda.Event) and the pinned-memory allocator/cupyx pinned-array utilities.
• Nsight Systems documentation — the CUDA trace sections; you’ll be reading multi-stream rows today.

Prerequisites

• Lab 1.1’s timeline capture and its measured copy and kernel times — the overlap prediction below is computed from them.
• Lab 1.2/1.3: a kernel of your own with non-trivial runtime to pipeline (the tiled matmul or a reduction pass both work; a deliberately fattened SAXPY is also fine).
• Lab 0.1 CMake conventions; repo building on the Linux box.

Project & environment setup

• New C++ target pipeline in cuda/, with modes selectable by flag: sync (baseline), streams (chunked overlap), pinned/pageable toggles, managed, zerocopy. One binary, one comparison story.
• Run every mode under Nsight Systems; keep an nvidia-smi power/thermals log alongside the main runs:

nvidia-smi --query-gpu=timestamp,power.draw,temperature.gpu \
  --format=csv -lms 500 > labs/lab-1-4/benchmarks/nvidia_smi_pipeline.log &
nsys profile -o labs/lab-1-4/captures/pipeline_streams ./pipeline --mode streams
kill %1

• Python side: pipeline_cupy.py mirroring the C++ modes with cp.cuda.Stream, events, and pinned host buffers.

Where results go:

Artifact	Path
Notes, mode-comparison table, pinned/managed/zero-copy memory-model writeup	labs/lab-1-4/notes.md
Nsight Systems timelines, one per mode	labs/lab-1-4/captures/
Timing CSVs, nvidia-smi logs	labs/lab-1-4/benchmarks/

Background

• Streams. Work within a stream runs in issue order; work in different streams may overlap. The default stream historically synchronizes with everything — one reason the course convention from here on is explicit streams for anything performance-relevant. Events are markers recorded into streams: other streams can wait on them, the CPU can query them, and pairs of them are the timing instrument you already use.
• Why pinned memory exists. An async copy needs a source the OS won’t page or move; cudaMallocHost/cudaHostAlloc gives page-locked memory the DMA engine can address directly. Pageable copies bounce through a hidden pinned staging buffer — a real cost on any machine, and a first-order one on a discrete GPU where every copy already pays the PCIe crossing. cudaMemcpyAsync from pageable memory silently loses its asynchrony — a classic trap worth triggering on purpose.
• The chunked-pipeline idea. Split \(N\) elements into \(k\) chunks, each chunk on its own stream doing H2D → kernel → D2H. Steady-state, stage \(i\) of chunk \(j\) overlaps stage \(i{+}1\) of chunk \(j{-}1\); total time approaches the largest stage total plus pipeline fill/drain, instead of the sum of all three. That “hides the smaller of copy and compute” arithmetic is the predicted row, computed from Lab 1.1’s measured numbers.
• Discrete memory sharpens the trade. On the 4090, cudaHostAlloc with the mapped flag gives zero-copy in name only: the kernel reads host DRAM across PCIe on every access — plausible for a touch-once streaming pattern, ruinous under reuse. cudaMallocManaged doesn’t remove the bus either; it services GPU page faults by migrating pages across it, which is convenient and correct but rarely competitive with a planned pinned-staging pipeline for streaming workloads. The teaching point: the API is portable; the performance model is not. On a physically-unified part (Apple Silicon, met from Module 2’s Metal track; integrated SoCs generally) the same calls tell a very different story — don’t over-generalize in either direction; measure the machine in front of you.
• CUDA graphs (teaser). Once a pipeline’s launch pattern is fixed, stream capture can record it into a graph and replay it with far lower per-launch CPU cost. Optional here; the idea returns when render loops meet compute in Module 5.

Tasks

CUDA C++

1. Baseline. The Lab 1.1-style monolith: one pageable H2D, one kernel, one D2H, synchronous, event-timed end to end. This is the bar every mode below is measured against.
2. Pinned vs. pageable copy benchmark. cudaMemcpy throughput both ways at several sizes (\(2^{16}\) → \(2^{26}\) bytes), pageable vs. pinned. Also demonstrate (with a timeline) that cudaMemcpyAsync from pageable memory did not actually overlap.
3. The chunked pipeline. \(k\) chunks across \(\geq 3\) streams, pinned buffers, per-chunk H2D → kernel → D2H. Sweep \(k \in \{2, 4, 8, 16\}\); event-time the whole pipeline. The Nsight timeline showing bricks overlapping is the deliverable of record.
4. Managed memory. Same workload with cudaMallocManaged and no explicit copies; time it, find the page-fault migration activity in the Nsight timeline, and note CPU-side touch costs before/after GPU work — the migration-cost-vs-explicit-copies measurement is the point of this mode.
5. Zero-copy. Mapped pinned memory (cudaHostAlloc + device pointer), kernel reading host memory in place. Time for a streaming (touch-once) access pattern and a reuse-heavy pattern — the two should tell opposite stories.
6. CUDA graphs (optional). Stream-capture the steady-state pipeline into a graph; replay; compare per-iteration CPU launch cost from the Nsight timeline.

CUDA Python

7. CuPy pipeline mirror. Rebuild mode 3 with cp.cuda.Stream, events, and pinned host arrays; confirm on a Nsight capture (nsys profile uv run python ...) that Python-issued work overlaps the same way — the async machinery is the runtime’s, not the language’s.
8. Default-stream trap. Show one CuPy version where naive coding serializes everything (default stream, implicit syncs from host reads), and diagnose it from the timeline rather than the source.

Deliverable & expected results

• Mode-comparison table (baseline / pinned / streams-\(k\) / managed / zero-copy, C++ and Python), the overlapped timeline, and a half-page “explicit vs. mapped vs. managed on a discrete GPU” summary in notes.md reconciled against NVIDIA’s documentation.

Quantity	Predicted	Measured
Best chunked-pipeline time	\(\approx \max(T_{\text{copy}}, T_{\text{kernel}})\) + fill/drain, computed from Lab 1.1’s measured stage times — overlap hides the smaller stage	…
Pinned vs. pageable copy throughput	pinned faster (no staging bounce) — and with every copy crossing PCIe the gap is substantial; the margin is the measurement	…
Pageable + cudaMemcpyAsync	no true overlap on the timeline — the API accepts it, the DMA engine doesn’t honor it	…
Zero-copy vs. explicit copies, streaming pattern	explicit copies expected to win — zero-copy turns every access into a PCIe transaction; touch-once streaming is its least-bad case — verify rather than assume	…
Managed memory vs. explicit	slower than the explicit pinned pipeline for this streaming pattern — page migration over PCIe on fault is the cost to find on the timeline — verify, don’t assume	…
Power during pipeline (nvidia-smi)	overlapped mode draws more power but less energy per batch — direction to verify	…

Profiling & performance

This is a Nsight Systems lab from start to finish: one capture per mode, and the streams-mode capture must show the brick-wall overlap pattern across stream rows — if it doesn’t, the lab isn’t done, and the timeline will tell you which dependency serialized you (default-stream sync, pageable source, same-stream ordering). Align the nvidia-smi log timestamps with the capture and annotate the power step when the pipeline saturates. File the per-mode GPU-idle percentage; it is the single most honest summary number.

Analysis & reconciliation

Reconcile the measured best-pipeline time against the \(\max(T_{\text{copy}}, T_{\text{kernel}})\) model, chunk-count sweep against fill/drain overhead, and explain deviations by pointing at the timeline, not at theory. Then the memory-model essay, short and specific: for each of pinned / managed / zero-copy, one sentence on what the documentation claims for a discrete GPU, one on what this machine did, one on why — citing the programming guide’s unified-memory appendix where it settles the question. Flag anything that remains genuinely unexplained as a question to carry into Lab 1.5’s counters.

Going further

• Finish the CUDA-graphs teaser properly: measure launch-overhead reduction at small chunk sizes where per-launch CPU cost dominates.
• Add a CPU producer thread feeding the pipeline through a ring of pinned buffers — the shape of every capture/compute/display loop Module 5+ builds.
• Re-run the pinned-vs-pageable and zero-copy experiments conceptually against an integrated-part write-up (NVIDIA’s embedded-SoC memory-model docs, or Apple’s unified-memory guidance) and tabulate which of this lab’s conclusions depend on the PCIe bus and which don’t.