Why this lesson exists
When you add a custom metric or experiment to csperf you risk turning the evidence into fan‑fiction. This episode shows how to extend csperf safely while keeping the artifact honest.
Recap — where we are in the series
- Ep 1 — Why a Single ./a.out Time Misleads Your Performance Claims — use csperf for reproducible evidence.
- Ep 2 — Warm vs Cold: Why a Single Trial Misleads Performance Claims — warm‑up and repeat give credibility.
- Ep 3 — Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts — metadata‑rich artifacts beat screenshots.
- Ep 4 — Comparing Across Machines: Why Metadata Matters in csperf — machine metadata makes cross‑machine claims reliable.
- Ep 5 — csperf: A Lightweight Observatory for Honest Performance Tracking — observatory commands give quick evidence.
- Ep 6 — csperf Doctor: First Command to Diagnose Toolchain Issues — doctor validates the environment.
- Ep 7 — ... — ...
- Ep 8 — ... — ...
- Ep 9 — ... — ...
The misconception
Many developers think adding a plugin is just a one‑liner and that the resulting artifact will still be trustworthy. In reality, a poorly‑designed plugin can silently alter the timing, drop metadata, or introduce nondeterminism.
What problem csperf solves (this episode's slice)
csperf gives you:
- A plugin API that enforces a contract for metadata and timing.
- Built‑in checks that any plugin must pass before its data is merged into the artifact.
- A clear separation between experiment definition and artifact generation.
Mental model
plugin ──► csperf core ──► artifact
│ │ │
│ └─► validation ──► reject if broken
│
└─► user‑defined metrics
The core never trusts the plugin; it validates the output structure, timestamps, and that the plugin did not modify the baseline run.
Lab: install and first commands
- Install the latest csperf:
pip install --upgrade compilersutra-perf
- Verify the installation:
csperf doctor
The doctor command will print the compiler version, machine metadata, and any known plugin issues.
Lab: what we ran on this machine
We attempted to add a simple --time-alloc experiment that counts memory allocations. The plugin was placed in plugins/time_alloc.py.
csperf list-experiments
The output lists the built‑in experiments plus time_alloc.
csperf run --experiment time_alloc matrix_traversal.cpp
The run failed with:
ERROR: Plugin 'time_alloc' failed validation: missing 'execution_time_ms' field.
The doctor command explains that the plugin must return a JSON with execution_time_ms and metadata.
Results (real numbers only)
Graph: trial wall_time_ms from this run's artifacts (not invented).
Artifacts missing for this episode — numbers withheld.
How to read the artifacts
When a plugin passes validation, its output is merged into the timing_summary field of the artifact. The artifact will contain:
-
execution_time_ms– wall‑time of the experiment. -
metadata– any user‑supplied key/value pairs. -
plugin_version– the version of the plugin used.
You can inspect the artifact with:
csperf visualize artifact.json
The visualization will show the baseline timing alongside the plugin‑added metric.
Common mistakes (teacher checklist)
-
Skipping validation – always run
csperf doctorafter adding a plugin. -
Missing required fields – ensure
execution_time_msandmetadataexist. - Hard‑coding timestamps – let csperf generate timestamps.
- Altering baseline artifacts – plugins must not modify the original run data.
Try this next (homework)
Do this tonight — Episode 29 starts by assuming you did.
Closing
You now know how to extend csperf with plugins while preserving the honesty of your performance evidence. Next, we’ll show you how to teach this lab to teammates in a single hour.
The series so far
- Ep 1 — Why a Single ./a.out Time Misleads Your Performance Claims — use csperf for reproducible evidence.
- Ep 2 — Warm vs Cold: Why a Single Trial Misleads Performance Claims — warm‑up and repeat give credibility.
- Ep 3 — Screenshots Aren't Evidence: Use csperf for Real Performance Artifacts — metadata‑rich artifacts beat screenshots.
- Ep 4 — Comparing Across Machines: Why Metadata Matters in csperf — machine metadata makes cross‑machine claims reliable.
- Ep 5 — csperf: A Lightweight Observatory for Honest Performance Tracking — observatory commands give quick evidence.
- Ep 6 — csperf Doctor: First Command to Diagnose Toolchain Issues — doctor validates the environment.
- ...


Top comments (0)