Lab 6.2 — Metal Performance: GPU Capture & Counters
← Course 4 syllabus · Module 6 · Prev: « Lab 6.1 · Next: Lab 6.3 »
Goal
Lab 6.1 established the CPU truth; this lab establishes the Apple-GPU truth, by working the Xcode instrumentarium end to end on the Module 5 scenes: GPU frame capture (the dependency viewer, per-encoder and per-draw costs, the shader profiler’s line-level costs, the memory viewer), GPU counters with their limiter analysis (is this pass ALU-limited, texture-limited, tile-limited, memory-limited?), Metal System Trace in Instruments (CPU encode vs. GPU execute vs. display, and the bubbles between them), the Metal HUD for live triage, MTLCounterSampleBuffer for an in-engine counter HUD, and the thermal/performance-state APIs that explain why the same capture reads differently after ten minutes. The deliverable is not a fixed frame — nothing gets optimized until Lab 6.4 — but a written performance anatomy of each Module 5 scene on the Apple GPU: where its frame time goes, what its limiter is, and the capture evidence for both. Hypotheses first, evidence second, verdicts only where the counters actually support them. This is the same discipline Lab 1.5 ran on CUDA kernels — ceilings, limiters, honest attribution — with Nsight’s roofline replaced by Apple’s capture and counter set.
Recommended reading
- Apple, “Optimizing GPU performance” and the surrounding Metal performance documentation — the limiter methodology this lab applies; read before the first counter capture, not after.
- Apple, GPU counters / counter-sampling documentation —
MTLCounterSampleBuffer, counter sets, and sampling boundaries. What can be sampled at stage versus draw granularity differs by GPU family — feature-query, don’t assume. - Apple, Metal System Trace documentation (the Instruments template) and the Metal debugger documentation — the dependency viewer, shader profiler, and memory viewer pages, so the vocabulary below has pictures attached.
- WWDC performance sessions — the GPU-counters and Apple-GPU-optimization session family (titles vary by year; search the session library for “GPU counters” and “optimize Metal”). Watch at least one counters session with a capture of your own scene open beside it — the sessions are built around exactly this exercise.
- MbT — the GPU-debugging/profiling material where the book reads its own frames in Xcode (title-level reference; 5th-ed. numbering — confirm against the copy in hand).
- Lab 5.2’s notes — the TBDR arguments made there are exactly what today’s counters will confirm or embarrass.
- Lab 1.5’s closing essay — the limiter-reasoning template this lab transplants from CUDA to graphics.
Prerequisites
- Lab 6.1 — the CPU dossier, the run-header discipline, and the standing question list; every GPU question below starts from a Tracy timeline that couldn’t answer it.
- The Module 5 scenes (5.2, 5.3, 5.5) running through the metal-cpp backend; scene 5.2’s memoryless-attachment toggle still switchable from config.
- Full Xcode with GPU capture working since Lab 0.4;
MTLCaptureManagerprogrammatic capture (0.4’s Going-further) is now required, since the engine is a C++ app, not an Xcode-template toy. - Pinned benchmark cameras per scene, agreed with 6.1’s captures, so CPU and GPU evidence describe the same frame.
Project & environment setup
- Run configurations for each scene under Xcode (attach or launch), with the scheme’s Metal diagnostics set for capture; keep the Metal HUD environment variable from Lab 0.4 handy for live triage outside Xcode:
MTL_HUD_ENABLED=1 ./build/release/engine_app --scene city # HUD triage
# In-app: MTLCaptureManager capture bound to a hotkey → .gputrace on disk- A
powermetricslogging invocation (root required) time-aligned with captures, same habit as 6.1’s soak; record the sampler flags used in the run header. - Thermal-state awareness in-engine: subscribe to
ProcessInfo.thermalState(and its change notification) and stamp the current state into the HUD and every capture’s run header — a capture taken in theseriousthermal regime is a different experiment from one takennominal. - The counter HUD (Task 5) lands in the engine as a small overlay fed by
MTLCounterSampleBufferreads — scaffolding lives inengine/metal/, and its Vulkan twin arrives in Lab 6.3 so the two HUDs share one on-screen format.
Where results go:
| Artifact | Path |
|---|---|
| The three scene anatomies, limiter table, bubble-hunt log | labs/lab-6-2/notes.md |
.gputrace captures (annotated, one per scene/condition), Instruments traces |
labs/lab-6-2/captures/ |
| Counter-HUD stat dumps, powermetrics logs | labs/lab-6-2/benchmarks/ |
Background
- The capture is a database, not a screenshot. A
.gputraceholds every encoder, draw, resource, and binding of the frame. The dependency viewer shows which pass consumes which attachment — the render graph, as the driver saw it, and the first thing to diff against the render graph you think you built. - Attribution comes in layers. Per-encoder and per-draw timings attribute GPU time coarsely; the shader profiler attributes a shader’s cost to source lines — an estimate reconstructed from compiled code, so treat line costs as strong hints, not gospel; the memory viewer shows every live allocation, its storage mode, and who references it — the tool that catches the forgotten
storeAction, the uncompressed texture, the leak. - Limiter analysis is Apple’s framing of the Lab 1.5 question “what is this pass entitled to?”: the counters report, per encoder, how occupied each GPU subsystem was — ALU, texture sampling, tile operations, buffer/memory traffic — and the limiter is the subsystem nearest its ceiling. The junior mistake is reading a high limiter as a verdict by itself: a pass can show a high ALU limiter and low utilization overall, which points at occupancy or dependency stalls, not arithmetic. The discipline is limiter plus utilization plus a hypothesis about why, checked against what the pass actually does.
- TBDR makes bandwidth a choice. On Apple GPUs the frame is rendered in tiles held in on-chip memory;
loadAction/storeAction/memoryless attachments decide whether tile contents ever round-trip to DRAM. Scene 5.2’s deferred G-buffer was built on exactly this claim — today the memory-traffic counters get to audit it, which is why the memoryless on/off comparison is this lab’s mandatory centerpiece. - Metal System Trace is the zoomed-out complement: CPU encode lanes, GPU channels, and display, on one timeline. Its questions are structural, not per-draw: does encode overlap execute? Is the GPU idle mid-frame (a bubble), and is the cause a CPU stall, a pass dependency, or presentation backpressure? It also carries the thermal track, closing the loop with the in-engine thermal-state stamps.
MTLCounterSampleBufferis the in-engine, always-on sliver of the same counter machinery: sample timestamps (and, family-permitting, stage utilization counters) at encoder boundaries, resolve them a few frames later off the critical path, feed the HUD. It is to Xcode’s counters what Tracy zones are to Instruments — coarser, but always running, and the only one of today’s tools that ships with the engine.
Tasks
- Capture and annotate each scene. One
.gputraceper Module 5 scene at its pinned camera.- Walk the dependency viewer and confirm it matches the render graph you think you built; any surprise edge is a logged finding.
- Record per-encoder GPU times as the scene’s frame budget table in
notes.md— the denominator every later percentage refers to. - For scene 5.2, capture twice — memoryless G-buffer on and off — and put the two captures’ memory-traffic counters side by side: the delta is the TBDR proof, and it is this lab’s headline number.
- Shader-profiler deep read. From the budget tables, take the two most expensive shaders across the scenes and read them at line level.
- Where does the cost concentrate, and is it where you expected when you wrote the shader?
- What question does the profile raise — divergence? texture-sample stalls? register pressure? Log each as a hypothesis for 6.4, not a fix for today.
- Limiter table. Using the counters, build one table: scene × pass → limiter, utilization, one-sentence hypothesis for why that limiter, given what the pass does. Candidate hypotheses to test rather than assert: is the splat scene at high count limited by blending/fill? Is distant terrain limited by vertex work or by bandwidth? The table’s verdict column is filled only from counters.
- Bubble hunt. In Metal System Trace, examine one heavy frame per scene: is the GPU ever idle mid-frame?
- Bound each bubble’s duration and attribute it — CPU encode too slow (check against the 6.1 Tracy trace of the same scene), a dependency serializing passes, or presentation backpressure.
- If a scene shows no bubbles, show the evidence for that too — a fully-fed GPU is a claim that needs a screenshot.
- Counter-sample HUD. Integrate
MTLCounterSampleBuffersampling at render-graph pass boundaries into the engine HUD: rolling per-pass GPU ms, resolved without stalling the frame. Validate the HUD’s numbers against the same frame’s Xcode capture timings and record the agreement (and any systematic bias) innotes.md.
Deliverable & expected results
- Three written scene anatomies in
notes.md— each one page: frame budget table, limiter row per pass, the shader-profile findings, the bubble verdict, and the capture filenames backing every claim. - The memoryless on/off counter comparison for 5.2, and the counter HUD running in-engine with its validation note.
| Quantity | Predicted | Measured |
|---|---|---|
| 5.2 memory-traffic counters, memoryless off → on | a large drop in G-buffer-attributable DRAM traffic — direction certain from the TBDR argument, magnitude is the measurement | … |
| 5.2 frame time, memoryless off → on | improves if the pass was bandwidth-bound at all — the limiter table says whether to expect much | … |
| Splat scene (5.5) limiter at high splat count | hypothesis: blend/fill-rate side — the counters confirm or refute | … |
| Terrain (5.3) limiter at far view distance | hypothesis: vertex/geometry work or bandwidth, not fragment ALU — verify | … |
| Counter HUD vs. Xcode capture, per-pass GPU ms | same story within noise; any systematic bias explained | … |
| GPU idle time mid-frame (worst scene) | unknown — bounded by the bubble hunt, attributed or explicitly unexplained | … |
| Thermal state across a capture session | expect drift toward fair/serious under sustained load — stamp it, don’t ignore it |
… |
Profiling & performance
The tools are the syllabus; this section is the checklist of what lands in captures/: four annotated .gputrace files (three scenes + 5.2-memoryless-off), one Instruments Metal System Trace per scene, and the HUD validation dump in benchmarks/ — every file named <scene>-<condition>.<ext> and cited from notes.md next to the claim it supports. Power context: a powermetrics log alongside each System Trace, with thermal state noted in the run header at capture time.
Analysis & reconciliation
The anatomies are the analysis; reconcile them in three directions. Against 6.1: does the GPU time per pass explain the frame intervals Tracy measured, and do the bubbles line up with CPU-side gaps in the same frames? Against Module 5’s predictions: each Module 5 lab argued a performance story when it was built (5.2’s tile-memory bandwidth case, 5.3’s LOD scaling, 5.5’s sort-and-blend cost) — quote each scene’s original claim from its notes.md and grade it against today’s counters, kept or overturned. Against the limiter method itself: pick one limiter verdict and design (on paper) the experiment that would falsify it — a change that should move that counter and no other — because that habit is what keeps counter-reading honest when 6.4 starts changing code. File every “the counters can’t tell me X on this API” item for Lab 6.3’s cross-check on the other API.
Going further
- Add
os_signpostintervals around the engine’s frame phases and view them in the same Instruments timeline as Metal System Trace — Apple’s native answer to Tracy zones, and useful when sharing traces with people who don’t run Tracy. - Explore capture diffing: two
.gputracefiles of the same pinned camera across a code change — the workflow 6.4’s ladder rungs will lean on; decide now what “same frame” means operationally. - Read one WWDC session on Apple-GPU shader occupancy and re-examine the Task 2 shader profiles for register-pressure symptoms — the Lab 1.5 occupancy story, Apple edition.
- Try the Metal HUD’s logging mode (see the current Metal release notes for its configuration) to get HUD statistics as parseable output — a zero-code check on your counter HUD’s numbers.