Lab 4.3 — Multithreaded Command Recording

← Course 4 syllabus · Module 4 · Prev: « Lab 4.2 · Next: Lab 4.4 »

Goal

Put the other cores to work. Everything so far records the frame on one thread; this lab builds a C++20 job system — a std::jthread worker pool, std::atomic counters with wait/notify, std::barrier for phase joins, work-stealing as an optional refinement — and points it at the frame’s most parallelizable job: command recording. On Vulkan that means secondary command buffers, one per worker, executed by a single primary; on Metal it means MTLParallelRenderCommandEncoder, which exists for exactly this purpose. The referee is Tracy’s zones and locks view: not “it uses threads” but visible parallelism — worker lanes actually overlapping, locks actually uncontended.

The intellectual spine is Amdahl’s law with real numbers in it: measure the serial fraction of your own frame, compute the speedup bound it permits, then run the 1→8 worker sweep and reconcile the curve against the bound. Threading an engine is a measurement discipline before it is a coding one.

Prerequisites

  • Lab 4.1: the backend seam and slot-based frame loop — per-worker resources hang off the frame slot built there.
  • Lab 4.2: the render graph — workers record within passes; the graph’s compiled order is what makes that safe to reason about.
  • Lab 0.2: the benchmark discipline (Google Benchmark, medians, pinned repetitions) that the scaling sweep reuses.

Project & environment setup

  • Job system lives in engine/core/ (target engine_core) — it knows nothing about graphics; the parallel-recording layer lives behind the Lab 4.1 seam in engine/render/ + backends.
  • engine_viewer gains three flags:
    • --workers N (0 = the Lab 4.2 single-threaded path, kept as baseline);
    • --record-mode shared|per-thread for the contention experiment;
    • --chunk-size N for the partition-granularity knob.
  • New benchmark target job_bench (Google Benchmark) for job-system microbenchmarks, separate from the in-frame sweep.
  • Scene: the Lab 3.1 scene is too small to show scaling — duplicate its draws to a few thousand (a crude multiplier flag is fine; Lab 4.4 builds the real stress scene).
  • Sweep runs from the shell, not by hand:
for n in 1 2 3 4 5 6 7 8; do
  ./engine_viewer --backend vulkan --workers $n --frames 500 \
      --stats-csv labs/lab-4-3/benchmarks/sweep_vk.csv
done

Where results go:

Artifact Path
Notes, Amdahl worksheet, contention postmortem, efficiency curve labs/lab-4-3/notes.md
Tracy traces (baseline, contended, per-thread, sweep points), GPU captures labs/lab-4-3/captures/
job_bench JSON, sweep CSV labs/lab-4-3/benchmarks/

Background

The C++20 job-system toolkit. The language finally ships the primitives a job system is made of; each earns a specific role:

Primitive Role in the job system
std::jthread + std::stop_token RAII workers with cooperative shutdown — the pool’s destructor is the shutdown story
std::atomic<T>::wait / notify_one futex-shaped parking for idle workers — blocking without condition-variable ceremony
std::atomic counters (fetch_sub) fan-out/fan-in: each finished job decrements; waiters block on zero
std::barrier phase joins: all recording done → primary assembles and submits
std::hardware_destructive_interference_size the alignas value that keeps per-worker state off shared cache lines

On top of these, two coordination shapes: a counter for cheap fan-out/fan-in, and a small future-like handle for jobs that produce values. Work-stealing deques are the classic refinement; a single mutex-guarded queue is the honest starting point, and Tracy will tell you when it stops being enough.

The false-sharing trap. Per-worker counters packed adjacently share a cache line; every increment then ping-pongs the line between cores — Course 3 Module 8’s RMW-contention measurement, resurfacing in C++. The fix is alignas(std::hardware_destructive_interference_size) on per-worker state — and note that on Apple Silicon the destructive-interference line is 128 bytes, which Course 3’s cache probes already established; verify what your standard library reports for the constant rather than assuming.

What parallelizes, what cannot. Before writing any worker code, classify the frame the engine already has:

Frame phase (Tracy zone) Serial or parallel? Why
wait_fence, flush_deletions serial gates the slot; nothing to split
acquire, present, submit serial one swapchain, one queue submission
graph_compile, graph_emit serial (this lab) small; caching, not threading, is its fix (Lab 4.2)
record parallel per-draw work, independent given per-worker resources
culling, transform flatten parallel — next lab’s client flagged now, harvested in Lab 4.4

The serial rows are the \(1-p\) in everything below.

Vulkan’s threading rule. Command buffers are recorded on any thread, but a VkCommandPool is externally synchronized — two threads may not touch the same pool concurrently. The idiomatic consequence: one command pool per worker per frame slot, reset wholesale when the slot’s fence wait returns (the same lifecycle as Lab 4.1’s descriptor allocator — notice the rhyme). Workers record secondary command buffers with inheritance info naming the attachments (under dynamic rendering, VkCommandBufferInheritanceRenderingInfo carries the formats); the primary does begin-rendering, vkCmdExecuteCommands over the workers’ secondaries in draw order, end. Determinism comes from assembly order, not recording-completion order.

Metal’s version. MTLParallelRenderCommandEncoder is the same idea with the ordering built in: create sub-encoders up front (creation order = execution order), hand one to each worker, each worker encodes and calls endEncoding, parent ends after all children. No pools to manage — but the engine-side structure (partition, record, join) is identical, which is why the job system lives in engine/core/ and neither backend knows it exists.

Amdahl’s law, which this lab measures rather than recites. If a fraction \(p\) of the frame’s CPU time parallelizes perfectly across \(N\) workers and \(1-p\) stays serial,

\[S(N) \;=\; \frac{1}{(1-p) + p/N}, \qquad E(N) \;=\; \frac{S(N)}{N},\]

with the hard ceiling \(S(\infty) = 1/(1-p)\) no worker count escapes. The serial part here is real and visible in Tracy — the serial rows of the table above — plus the job system’s own overhead, which Amdahl charitably ignores and your measurements will not.

Tasks

  1. The job system. Build the worker pool in engine/core/: jthread workers, atomic-wait parking, submission of jobs with a completion counter, std::barrier phase join, clean stop-token shutdown. Describe the API’s shape (nouns and verbs, ownership of job lifetimes) in notes.md; the implementation is yours.
  2. Microbenchmark it. job_bench measures three things before any rendering is involved: empty-job dispatch overhead (ns per job — the tax every use pays), fan-out of 10k trivial jobs across 1→8 workers, and the false-sharing variant — per-worker counters packed vs. alignas-separated, the Course 3 M8 experiment reprised in C++20 clothing.
  3. Per-worker recording resources. Give each worker a command pool per frame slot (Vulkan), reset on slot reuse; thread Tracy’s zone context through the workers so their lanes appear in the trace. Validation stays on — it will catch pool misuse from the wrong thread, which is the point of doing this before the contention experiment.
  4. Partition and record. Split the frame’s draw list into contiguous chunks (--chunk-size; chunk count a small multiple of worker count is the starting heuristic), record secondaries per chunk, assemble in the primary in stable draw order. Verify against the single-threaded baseline: identical image, identical draw count in the HUD.
  5. The contention experiment. Run --record-mode shared — all workers allocating from one mutex-guarded pool — versus per-thread. Capture both in Tracy and read the locks view: the shared run should show the mutex hot and worker lanes serialized into a staircase; the per-thread run should show clean overlap. Save both traces; write the one-paragraph postmortem.
  6. Metal parallel encoding. Same partition, same job system, sub-encoders from MTLParallelRenderCommandEncoder. Confirm image parity, then capture in Xcode and find the sub-encoders in the frame’s encoder list.
  7. The scaling sweep. --workers 1..8 on the M-series, fixed scene, ≥ 500-frame medians per point: record total frame CPU time and the recording-phase time; compute \(S(N)\) and \(E(N)\); plot the efficiency curve (a script over the CSV is fine — plotting is not engine code). Measure the serial fraction from the 1-worker Tracy trace first and write the Amdahl bound in notes.md before running the sweep — predictions filed after the fact don’t count.

Deliverable & expected results

  • engine_viewer --workers N rendering identically to the baseline at every N, both backends; validation silent; the traces, CSV, and efficiency curve archived.
  • notes.md carries the Amdahl worksheet: measured \(p\), the bound \(S(\infty) = 1/(1-p)\), the sweep table, the contention postmortem, and the reconciliation.
Quantity Predicted Measured
Speedup bound from Amdahl at your measured serial fraction compute \(S(N) = 1/((1-p) + p/N)\) for \(N = 2, 4, 8\) from the 1-worker trace’s \(p\) before the sweep; measured speedups must sit at or under the curve …
Recording-phase speedup at 4 workers approaches \(4\times\) on the recording zone alone if partitioning is even and pools are per-thread; whole-frame speedup much smaller — that gap is Amdahl …
Shared pool vs. per-thread pools qualitative: contention collapse — shared-mode worker lanes serialize on the mutex (Tracy locks view shows wait time ≫ hold time), per-thread restores overlap …
False-sharing benchmark, packed vs. alignas counters integer-factor gap, same direction and mechanism as Course 3 M8’s RMW experiment; 128-byte line arithmetic in the notes …
Metal parallel-encoder sweep shape qualitatively the same curve as Vulkan — same partition, same serial floor; absolute recording cost may differ (sub-encoder creation vs. secondary-buffer overhead) …
Efficiency \(E(8)\) on M-series below \(E(4)\) — serial floor plus heterogeneous cores: beyond the performance-core count, efficiency cores dilute the curve …

Profiling & performance

Tracy is the whole referee here: worker lanes named (worker/0 …), the recording phase bracketed by a frame-level zone, locks instrumented (TracyLockable on the shared-pool mutex) so the contention experiment is a picture, not an inference. Archive one trace per sweep point and one GPU capture per backend at --workers 4 to confirm the GPU-side frame is unchanged — this lab moves CPU time only, and the captures prove it. Keep powermetrics in view during the sweep: more cores at the same frame rate is a power cost worth one sentence in the notes.

Analysis & reconciliation

Reconcile the sweep against the Amdahl bound point by point. Where measured \(S(N)\) falls short of the bound, apportion the gap among three named causes: job-system overhead (priced by job_bench’s dispatch numbers), partition imbalance (visible as ragged lane ends in Tracy — re-run one point with a smaller --chunk-size to test), and the P-core/E-core split (QoS-hint or pin the pool and re-run one point). Explain the false-sharing factor against the 128-byte line — same arithmetic as Course 3, new language. Close by answering the design question the module has been building to: at this draw count, was parallel recording worth its complexity — and at what draw count (extrapolate from the measured per-draw recording cost) does it become unambiguous? Lab 4.4’s 10k-object scene will check the extrapolation.

Going further

  • Implement work-stealing deques and re-run the sweep with deliberately unbalanced chunks (one chunk 10× the rest) — the case stealing exists for; measure against the fixed-partition pool.
  • Parallelize a second phase with the same job system — Lab 4.4’s culling is the natural client — and watch the serial fraction \(1-p\) shrink between labs.
  • Run the sweep on the Linux desktop (RTX 4090) and compare curve shapes — if its CPU’s cores are homogeneous (check before assuming; desktop parts vary), it’s a cleaner Amdahl specimen than the M-series’ P/E-core mix.
  • Try std::atomic::wait vs. a spin-then-park hybrid for worker parking and measure wake latency in job_bench — the latency/power tradeoff every job system tunes.
  • Read about device-generated commands and note in notes.md what Lab 5.4’s GPU-driven path will make of this lab’s CPU-side machinery — some of it is scaffolding for a thing the GPU will eventually do to itself.