Lab 7.2 — Capstone Performance Report

← Course 4 syllabus · Module 7 · Prev: « Lab 7.1 · Course complete — back to the syllabus

Goal

Produce the engineering artifact that proves the course: a written performance report — course4/docs/capstone-report.md in the labs repo, linked from this site when done — built from repeatable runs of Lab 7.1’s camera spline on all three platform configurations (Mac/Metal, Mac/MoltenVK, Linux/Vulkan on the RTX 4090).

Not a screenshot gallery and not a benchmark dump: a document a staff engineer would sign, where every number comes from a described method, every claim traces to an artifact on disk, and the architecture retrospective is honest about which of the engine’s decisions earned their complexity. The viewer was the build; this is the engineering — measurement discipline, comparative analysis, and the written narrative that turns eight modules of captures into a defensible account of a system. It is the closest thing this course produces to the document a rendering team actually argues from.

Recommended reading

• Your own notes.md files from Lab 6.1 through Lab 6.4 — the methodology this report industrializes: pacing metrics, capture workflows on each platform, and the optimization-ladder discipline of one change, one measurement, one explanation.
• Apple WWDC sessions on Metal performance and GPU counters — the vocabulary for the Mac frame-anatomy section, and the sanity check on what the counters actually mean.
• C&S — the performance-oriented chapters (title-level reference; confirm against the copy in hand), read once more as a checklist against your own retrospective: which of their architectural bets did your measurements confirm on your hardware?
• LunarG’s MoltenVK user guide — the performance and configuration sections, background for the translation-cost analysis: what MoltenVK says it does on your behalf, checked against what the captures show it doing.
• Course 1 §3 — the error-and-conditioning discipline behind reporting medians and percentiles rather than means, and behind saying how repeatable a number is before comparing two of them.

Prerequisites

• Lab 7.1 complete through M7: the viewer runs the deterministic flight spline on all three configurations, validation clean, HUD on.
• The spline file and milestone captures from Lab 7.1 in place (labs/lab-7-1/benchmarks/, labs/lab-7-1/captures/).
• Module 6 tooling healthy on every platform: Tracy, Xcode GPU capture + Instruments (Lab 6.2), RenderDoc + timestamp queries (Lab 6.3), powermetrics on the Mac and nvidia-smi on the Linux box (Lab 6.4).
• The Lab 7.1 budget table with owner-set targets — the report measures against it.

Project & environment setup

No new engine code beyond the benchmark-harness mode (Task 1). Work happens in the course-4 workspace of the labs repo (diiv_website_custom_courses/course4/); the report itself lives at course4/docs/capstone-report.md so it ships with the code, not buried in a lab folder.

Where results go:

Artifact	Path
The report	docs/capstone-report.md (within the course4/ workspace)
Run log, per-run conditions (thermal state, power source, clocks), reconciliation	labs/lab-7-2/notes.md
Frame-time series + stats dumps per run (CSV/JSON), HUD logs	labs/lab-7-2/benchmarks/
Evidence captures cited by the report (Tracy, .gputrace, .rdc, Instruments, power logs)	labs/lab-7-2/captures/

Keep a run log at the top of notes.md — one row per benchmark run, appended at run time, never after:

Run ID	Config	Date	Build hash	Power/thermal state	Tool attached	Output files
…	…	…	…	…	…	…

The report’s rule, adopted before writing a word: every quantitative claim in capstone-report.md must trace to a named file in captures/ or benchmarks/. No artifact, no claim.

Background

Two ideas carry this lab. First, a benchmark is an instrument: it has a stated method, a known noise floor, and results that replicate — or it isn’t one, and Task 2 exists to establish that before anything is compared. Second, a report is a chain of custody: the reader must be able to walk from any sentence to the run that produced it. Everything below serves those two.

The report has a required outline — write against it, in order. Each section below names what it must contain; the content is yours.

1. Executive summary — one page: what the system is, the headline numbers per platform, the top three findings. Written last, placed first.
2. Methodology — the section that makes everything else believable. It must pin down:
   • the spline and why playback is deterministic (fixed timestep, versioned file, Lab 7.1 M5);
   • warm-up policy — how many runs are discarded and what evidence set that number;
   • thermal state controls: Mac on mains with fans settled, the 4090’s clocks recorded (locked if you chose to) and temperature equilibrated per nvidia-smi — and how you verified each, not just that you intended to;
   • capture tooling per platform and each tool’s measured overhead on the run it observes;
   • the statistics — median / p95 / p99 frame times with run-to-run spread, per Course 1 §3’s error discipline and Lab 6.1’s pacing metrics. A good median with bad p99s is a stutter, not a success — report all three or none.
3. Per-platform frame anatomy — for each configuration, where the milliseconds go along the spline: per-pass GPU times, CPU record/submit cost, and the streaming/culling load at the three stations — each anatomy backed by a capture taken with the Lab 6.2 / Lab 6.3 workflow.
4. The three-way comparison — one table, rows per metric, columns per configuration, followed by prose sorting the differences into architecture (TBDR vs. immediate-mode), translation (MoltenVK), and budget choices. The required skeleton (whole-spline unless a station is named):

Metric	Mac/Metal	Mac/MoltenVK	Linux/Vulkan
Frame time p50 / p95 / p99 (ms)	…	…	…
Pacing (missed-vsync / hitch count per run)	…	…	…
Bandwidth proxy (platform counters, per station)	…	…	…
Power, whole-spline average (W)	…	…	…
Performance-per-watt (frames/J or equivalent)	…	…	…

5. The optimization ladder — at least five rungs, run with Lab 6.4’s methodology: hypothesis → single change → full spline re-run → before/after evidence per rung, on the platform where the bottleneck lives.
6. MoltenVK translation-cost analysis — the same engine, same Mac, same scene: Vulkan-through-MoltenVK vs. native Metal, the cleanest translation-layer experiment this hardware allows. Where is the gap — CPU-side translation in record/submit (Tracy sees it), pipeline or barrier shapes that map awkwardly onto Metal, or semantics with no cheap equivalent? Which passes translate nearly free and which pay? Capture evidence, not folklore — and where the gap is small, say so; that is a finding about both layers.
7. Architecture retrospective — which engine decisions earned their complexity, with evidence, decision by decision:
   • the render graph (Lab 4.2) — did explicit pass/resource structure pay for itself when Module 5’s passes were spliced in?
   • the GPU-driven path (Lab 5.4) — what did indirect draws buy at this instance count, on each GPU?
   • the TBDR deferred variant (Lab 5.2) — the measured bandwidth delta of memoryless attachments, vs. the paper prediction from Lab 7.1;
   • reverse-Z — what the depth-precision choice cost (nothing?) and prevented, with Course 1 §3 as the frame;
   • the job system (Lab 4.3) — worker utilization along the spline, and whether streaming ever starved rendering.
   For each: what it cost to build, what the measurements say it bought, and — the part that takes nerve — which ones didn’t pay off at this scene scale and would only earn out at a larger one.
8. Future work — grounded in measurement: each item names the evidence suggesting it’s the next bottleneck or opportunity.

Tasks

1. Benchmark-harness mode. Add --benchmark to world_viewer: spline playback at fixed timestep, per-frame time series and summary stats (median/p95/p99) dumped to benchmarks/, HUD stats logged alongside, zero interactive input. Keep it small — this is plumbing, not a framework.
2. Repeatability first. On one configuration, run the spline five times cold-to-warm; establish the warm-up policy and the run-to-run spread of the median. If p50 isn’t stable within a few percent, find out why (thermal drift? streaming nondeterminism? background load?) before collecting the matrix — an unrepeatable instrument measures nothing.
3. Run the matrix. 3 configurations × the full spline, plus focused stats windows at the 3 stations Lab 7.1 named (S1 altitude, S2 worst ground view, S3 the vignette). Record per-run conditions in notes.md — power source, thermal state, tool attached, build hash; power logs (powermetrics / nvidia-smi) captured concurrently with the runs they describe, never reconstructed after.
4. Assemble the evidence. One deep capture per platform per station, filed under captures/ with names the report can cite (a scheme like s2-linux-frame.rdc, s3-mac-metal.gputrace — boring and greppable). Build the three-way table and the frame anatomies from these — numbers from the harness, structure from the captures. Cross-check once: HUD pass times vs. the external tool’s, per platform; explain any disagreement beyond tool overhead before trusting either.
5. Run the ladder. At least five rungs per the outline’s Section 5, each a full spline re-run. Stop when a rung yields under your noise floor from Task 2 — that boundary is itself a finding worth reporting.
6. Write the report against the outline. Draft sections 2–6 as the data lands rather than at the end; the executive summary last.
7. The “defend it” pass. Walk every claim in the finished report and check it traces to a named artifact in captures/ or benchmarks/; delete or soften any that don’t. This is the report’s rule made mechanical — do it as a separate pass, adversarially, then link the report from the site.

Deliverable & expected results

• course4/docs/capstone-report.md, complete against the eight-section outline, defend-it pass done, linked from this site.
• The benchmark harness (--benchmark) in the engine, with its stats dumps reproducible from the spline file alone.
• The full run matrix in labs/lab-7-2/benchmarks/; the cited evidence captures in labs/lab-7-2/captures/; the run log, per-run conditions, and meta-reconciliation in notes.md.
• The Lab 7.1 budget table’s Measured column, finally filled from these disciplined runs — closing the loop the budget opened.

Predictions here are deliberately few — the report is the measured column — but three are worth staking in advance:

Quantity	Predicted	Measured
Spline run-to-run repeatability (warm, per config)	p50 stable within a few percent run-to-run	…
Power ordering, 4090 vs. Mac (whole-spline average)	the desktop 4090 draws far more absolute power — that much is safe; the perf-per-watt division is the genuinely open question, Apple’s efficiency against Ada’s throughput	…
Optimization-ladder yield curve	the first rungs are the big ones — later rungs shrink toward the noise floor	…

Profiling & performance

This entire lab is the section. The one rule it adds: evidence lives at the paths above and the report cites artifacts by filename — the profiling isn’t an appendix to the report; the report is an index into the profiling.

Analysis & reconciliation

The report’s sections 3–7 are the analysis; what remains here is the meta-reconciliation, in notes.md:

• the three predicted rows above, resolved — with the repeatability one first, since the other two are only meaningful if it held;
• the Lab 7.1 budget table, finally filled: where the owner-set targets stood up, where the matrix overturned them, and whether the misses were target errors or implementation errors;
• one honest paragraph on the methodology itself — what you would instrument differently, which station you would move, which statistic turned out not to earn its column — the paragraph that makes the next benchmark better.

Finish with the course’s closing sentence, earned rather than asserted: what kind of engineer the artifact in docs/ says you now are.

Going further

The course ends here; the directions don’t. Each of these is a worthy next project standing on this engine — forward-looking, no promises:

• A WebGPU port of the engine core — a third backend against the render-graph seam, and the sternest test yet of what the abstraction actually abstracted.
• Mesh shaders where supported — replacing the vertex-pipeline half of the GPU-driven path and re-running the ladder against it.
• Ray-traced shadows on the 4090 — the 4090’s RT cores against the cascade infrastructure via the mature Vulkan ray-tracing extensions on desktop NVIDIA, with the report’s shadow-pass numbers as the baseline to beat.
• Occlusion culling — completing the depth-pyramid (Hi-Z) stage the GPU-driven path left open, measured on the worst ground-level view.
• A second splat scene, trained by the owner — closing the loop from Lab 5.5’s pipeline to a capture of a real place, embedded in the world with the M4 compositing machinery already paid for.