Lab 6.1 — CPU-Side: Tracy, Benchmarks & Frame Pacing

← Course 4 syllabus · Module 6 · Prev: « Lab 5.5 · Next: Lab 6.2 »

Goal

Module 6 turns the tools into the subject, and it starts where every performance investigation should: with the CPU truth. Tracy has been riding along since Lab 0.3 as four zones and a frame mark; this lab graduates from “Tracy is installed” to fluent — zones with payloads, frame marks, plots of engine statistics, the locks view aimed at Lab 4.3’s job system, GPU zones so the CPU and GPU timelines share one ruler, and long-capture soak analysis. The second instrument is Google Benchmark, done statistically honestly: repetitions, medians versus means, named variance sources, and the frequency/thermal caveats that make a laptop a hostile measurement environment — Course 1 §3’s error framing, applied to nanoseconds. The third idea is the module’s most underrated: frame pacing is not frame rate. A renderer averaging 90 fps can feel worse than one holding 60, and this lab builds the vocabulary — present modes, missed-vsync detection, the stutter-vs-throughput distinction, the input-latency chain — to say precisely why. Everything here is exercised on the Module 5 scenes (5.2, 5.3, 5.5), which are this module’s patients; Labs 6.2 and 6.3 add the GPU truth, and 6.4 spends all of it.

Prerequisites

  • The engine through Module 5: scenes 5.2, 5.3, and 5.5 running on both platforms (Mac via Metal and MoltenVK, the Linux desktop (RTX 4090) native Vulkan).
  • Lab 4.3’s job system — the lock-contention capture needs real contention to look at.
  • Tracy and Google Benchmark already in the dependency block since Lab 0.1; the Lab 0.3 frame zones still alive.
  • The Lab 1.4 nvidia-smi logging habit, and powermetrics runnable on the Mac (root).

Project & environment setup

  • A dedicated profiling build preset: optimized like Release but with TRACY_ENABLE on and frame pointers kept — never profile a Debug build, and record in notes.md which preset every capture came from. Tracy’s server (the UI) must match the client version pinned in CMake; mismatched versions refuse to connect, which is a feature.
  • For the soak capture, build with Tracy’s on-demand mode so a ten-minute run doesn’t buffer from process start; connect from the Tracy UI on the Mac. The Linux box’s client streams over the network to the same UI — one config line; see the manual’s capture chapter for the port and firewall note.
  • Every measured run gets a run header recorded with it: platform, build preset, scene, camera, present mode, power state (driver clock settings — locked clocks if used — via nvidia-smi on the Linux box; mains vs. battery on the Mac), and ambient notes (fresh boot? warm?). Captures without headers are inadmissible in Module 6.
  • New target engine_bench (Google Benchmark) for the hot-function suite. The statistical workflow is flags, not code:
./engine_bench --benchmark_repetitions=20 \
  --benchmark_out=labs/lab-6-1/benchmarks/run.json --benchmark_out_format=json
python benchmark/tools/compare.py benchmarks \
  labs/lab-6-1/benchmarks/baseline.json labs/lab-6-1/benchmarks/run.json
  • The zone taxonomy (Task 1) is documented in docs/profiling.md in the labs repo — it is engine API surface from now on, not lab scratch.

Where results go:

Artifact Path
Notes, run headers, variance inventory, pacing analysis, soak writeup labs/lab-6-1/notes.md
Tracy traces (per scene, per present mode, plus the soak) labs/lab-6-1/captures/
Benchmark JSON (baseline + runs), compare output, pacing histograms labs/lab-6-1/benchmarks/
Zone-taxonomy document docs/profiling.md

Background

  • Zones are an API, not printf. A zone name that changes between captures destroys Tracy’s statistics view; a taxonomy that mirrors the render graph (sim/, cull/, record/, submit/, present/ …) makes every later capture diffable against every earlier one. Zone payloads (ZoneText, ZoneValue) carry the per-instance facts — which pass, how many draws — and plots (TracyPlot) turn engine counters (draw count, visible instances, allocator watermark) into time series next to the frame timeline.
  • Frame marks partition the timeline into frames, which is what unlocks Tracy’s frame-oriented views: the frame-time graph, per-frame zone statistics, and the histogram view this lab’s pacing analysis reads. Secondary frame marks (e.g. around the job system’s batch boundaries) are worth trying once the primary taxonomy is stable.
  • GPU zones put GPU work on the same timeline: on Vulkan, Tracy’s Vulkan context wraps timestamp queries and calibrates them against the CPU clock, so a GPU pass’s bar sits under the CPU code that recorded it. Tracy’s Metal integration is younger than its Vulkan one — if the version pinned in the repo proves awkward there, the honest fallback is feeding the engine’s own Metal timestamps (built properly in Lab 6.2) into Tracy plots; note in notes.md which path was taken and why.
  • The locks view visualizes what TracyLockable-wrapped mutexes experienced: who held, who waited, for how long, on which thread. Contention that totals a few microseconds a frame is bookkeeping; contention that serializes workers behind one queue is a design finding. The view tells you which one you have — the fix waits for 6.4.
  • Benchmark statistics. A benchmark’s mean is contaminated by every background hiccup; the median resists them, and the minimum estimates the noise-free cost of the code path (useful, but only for code whose cache/branch state the minimum-run scenario represents honestly). Report spread as the coefficient of variation \(c_v = \sigma/\mu\) and name the variance sources: CPU frequency scaling, efficiency-vs-performance core migration on the M-series, thermal throttling, background daemons, and the benchmark’s own data-layout luck.
  • Pinning discipline, honestly stated. macOS offers no user-space frequency pinning — the discipline is warmup runs, repetitions, a quiet machine on mains power, and reporting the residual spread rather than pretending it away. On the Linux desktop, nvidia-smi is the pinning story — persistence mode plus locked GPU clocks (--lock-gpu-clocks, root) — and those settings must be in the run header — a benchmark at unrecorded clocks is a number without units.
  • Pacing versus rate. With a display refresh period \(T_r = 1/f_r\), a FIFO/vsynced presentation displays each frame for an integer multiple of \(T_r\): a frame that misses its deadline by 1 ms is displayed for \(2T_r\) — indistinguishable, to the eye, from a frame that missed by 15 ms. So the quantity to plot is not average fps but the distribution of frame-to-frame present intervals: a healthy 60 Hz FIFO run is a single spike at 16.7 ms; stutter is mass at 33.3 ms; an average can hide either.
  • Queue-theory-lite. The swapchain is a small buffer between a producer (the engine) and a fixed-rate consumer (scanout). FIFO blocks the producer when the buffer is full — throttling the CPU/GPU to \(f_r\) and saving power. Mailbox lets the producer run free and the consumer take the freshest image — lower latency, discarded work, higher power. Immediate bypasses the queue — tearing, but the rawest view of true render rate. ProMotion adds a consumer with an adaptive rate: within its range, the display can wait for the frame rather than the frame for the display, which attacks the quantization itself.
  • The latency chain, sketched end to end: input event → OS delivery → simulation tick → command recording → GPU execution → present queue → scanout. Each stage adds delay and each buffer in the chain adds up to a frame of it; the sketch (hand-drawn in notes.md) is the map on which the present-mode experiment’s results get placed.

Tasks

  1. Zone taxonomy. Design and instrument a stable, hierarchical zone taxonomy across the whole engine frame, then freeze it in docs/profiling.md.
    • Cover: simulation, culling, per-pass recording, submit, present, asset streaming, and the job-system worker loops.
    • Payloads carry pass names and draw counts; plots track draw count, visible-instance count, and the per-frame allocator high-water mark.
    • Capture one annotated frame per Module 5 scene; confirm the statistics and find-zone views aggregate sensibly across a few hundred frames.
  2. GPU zones. Add Tracy GPU zones around every render-graph pass on the Vulkan path (both platforms), and close the Metal side by whichever route the Background bullet settles on. Deliverable: one capture per platform where a whole frame’s CPU and GPU work read off a single timeline — and a one-paragraph answer to “does the GPU start while the CPU is still recording?”
  3. Lock contention. Instrument the job system’s mutexes/queues with TracyLockable and capture scene 5.3 (the terrain streamer is the likeliest source of cross-thread traffic) under load.
    • In the locks view: which lock has the most contention, who waits on whom, and for how long per frame?
    • Do not fix anything yet — this is evidence-gathering; the fix, if one is warranted, is a Lab 6.4 rung.
  4. Benchmark suite + baseline. Stand up engine_bench over the engine’s CPU hot functions — candidates: frustum culling over the 4.4 scene, draw-call sort-key generation and sort, transform-hierarchy update, the splat-sort preparation from 5.5 — chosen from what Task 1’s captures actually show as hot, not from guesswork.
    • Run with 20 repetitions on both platforms; record median, mean, and \(c_v\) per benchmark, with the run header.
    • Commit baseline.json and demonstrate the compare.py workflow once against a fresh run; write the workflow into notes.md as the standing regression check every later lab runs.
  5. Present-mode A/B/C. Run scene 5.2 under FIFO, mailbox (where exposed — feature-query, and note MoltenVK’s answer), and immediate; on the Mac, add the ProMotion adaptive-refresh configuration as a fourth column.
    • Per condition: a Tracy capture, a present-interval histogram (extract frame times from the trace; matplotlib is fine), and a time-aligned powermetrics / nvidia-smi dmon log.
    • The deliverable is the histograms side by side with the power numbers attached, plus a note on where in the Tracy timeline each mode’s character is visible (FIFO’s blocking present, mailbox’s discarded images).
  6. Thermal soak. A 10-minute on-demand capture of scene 5.3 on each platform, with power/thermal logging alongside. Plot frame time and reported temperature against wall time. Does performance drift? When, how much, and does the drift correlate with a thermal threshold or a clock change in the logs?

Deliverable & expected results

  • docs/profiling.md; annotated captures for all three scenes; the locks-view evidence; baseline.json plus a demonstrated compare run; the present-mode histogram set with power; both soak plots — and a notes.md that reads as the CPU-side dossier the rest of the module builds on.
Quantity Predicted Measured
FIFO present-interval histogram quantized at integer multiples of \(T_r\) — a spike at \(T_r\), any misses appearing as a second spike at \(2T_r\), nothing in between …
Mailbox vs. FIFO latency down, power up — qualitative direction only; the histograms and the power logs put numbers on it …
Immediate-mode histogram shape continuous rather than quantized — the render-time distribution itself, with tearing as the visible price …
Benchmark run-to-run spread (\(c_v\)) after pinning discipline single-digit percent — if it’s worse, the variance inventory should say which source is unmanaged …
Thermal soak drift (Linux desktop, actively cooled 4090) little to no frame-time drift expected — the desktop cooler should hold clocks through the soak; verify from the temperature/clock columns of the nvidia-smi dmon log rather than assume …
Mac soak vs. Linux desktop soak qualitative: if drift lives anywhere it’s on the thermally constrained laptop, not the desktop — verify, don’t assume …
Job-system lock with the most wait time unknown — hypothesis first, locks view second …

Profiling & performance

This whole lab is the section. What must land in captures/: the three annotated per-scene traces, the two single-timeline CPU+GPU traces, the lock-contention trace, the present-mode traces, and the two soak traces — each named <scene>-<platform>-<condition>.tracy and each referenced from notes.md by filename next to the claim it supports.

Analysis & reconciliation

Reconcile in notes.md, section by section: the present-interval histograms against the queue-buffer model — does FIFO quantize exactly as predicted, where do mailbox’s discards show up in the CPU timeline, and how many \(T_r\) of acquire-to-present latency does each mode imply, read off the trace? The benchmark spread against the variance inventory, in §3’s terms — which error sources did the discipline control, which remain, and is the residual \(c_v\) small enough to detect the effect sizes 6.4’s ladder will care about? The soak drift against the thermal logs — correlation, threshold, or coincidence, argued from the time alignment. Close with the module’s standing question list: everything the CPU timeline cannot explain (why is this GPU zone so long? what is the GPU doing in that gap?) goes to Labs 6.2 and 6.3 by name.

Going further

  • Turn on Tracy’s sampling/callstack collection and compare what the sampler finds against what the zones claim — instrumentation bias, made visible.
  • Wire the compare script into a pre-push hook or CI job that fails on a >10% median regression against baseline.json — the production shape of Task 4.
  • On ProMotion, experiment with CADisplayLink frame-rate ranges: can the engine request 80 Hz and hold it more smoothly than it holds 120? What does powermetrics say about the trade?
  • Add a deliberate microstutter (a 5 ms spike every 60th frame) behind a debug flag and confirm each histogram and each present mode renders it visible or invisible exactly as the theory says — an instrument test with a known signal.