Samuel MartinExperiments and measurement systems
Today I learned

Ten notes, kept short on purpose.

Each one is a thing that changed how Samuel Martin builds a measurement or draws a figure. They are short because the useful part is short, not because they were abandoned. Where a rule is someone else’s, the note says so.

10 notesCharts, causal inference, clustering, data engineeringUpdated 2026-07-25
Clustering

Gower over Euclidean

Dummy-encoding an unordered category and running Euclidean distance quietly asserts that two industries are a fixed distance apart. Gower’s distance computes a per-attribute dissimilarity instead, so the encoding stops doing analytical work it was never meant to do.

Gower’s general coefficient of similarity is Gower (1971). Not a discovery here, just the reason it earned its place in the peer benchmarking work.

Clustering

Medoids are real clients

k-means needs a mean, and the mean of a set of mixed-type clients is not a client. Partitioning around medoids puts an actual account at each center, which turns out to matter more for explaining the model than for fitting it.

Partitioning around medoids is Kaufman and Rousseeuw. The observation that explicability beat fit here is the only part specific to this book of clients.

Charts

A funnel implies a decision

Stage counts in a data pipeline are survival counts, not attrition. Drawing them as a tapered funnel invites the reader to optimize conversion between stages when the fix is upstream key hygiene.

Came out of the attribution warehouse rebuild, where a funnel graphic would have sent effort to the wrong step.

Uncertainty

Put the interval on the finding

A band on one series in a difference-in-differences chart implies the other series was measured without error, and it draws uncertainty on a quantity nobody will quote. The interval belongs next to the estimate.

The rule that uncertainty goes on the quantity carrying the finding is the craft manual’s (R-009). This note is a case where following it changed a published figure.

Charts

Lines may be framed, bars may not

A line encodes position, so a non-zero axis is a framing choice that gets disclosed. A bar encodes length from zero, so the same choice is a lie. Same figure, two different rules.

Long-standing convention in the visualization literature, restated in the craft manual (R-141). Recorded here because the reason is easy to forget.

Charts

Banking to 45 degrees is Grade C

It optimizes slope judgment. If the reader’s task is comparing vertical gaps at two moments, the heuristic does not apply, and disclosing non-compliance with it is noise rather than rigor.

Banking to 45 degrees is Cleveland’s. The Grade C assessment and the phrase about a starting heuristic rather than law are the craft manual’s (R-037). Neither is an original claim.

Causal inference

Parallel trends, not equal levels

Difference in differences never required the two groups to sit at the same level. It required them to have been moving together beforehand, which is why the pre-period is the part of the chart worth attacking.

Standard econometrics, not a finding. Noted because the AI chat figure is easier to defend once the pre-period is drawn at full weight.

Experimentation

Novelty is not in the assignment

Randomizing exposure to a new feature does not randomize curiosity. Early engagement inside a treatment arm is still inflated, which is why a 30-day number measures trial rather than retention.

The novelty effect is well established in the experimentation literature. What was specific here was scoping the 30-day test to leading indicators in writing, before launch.

Data engineering

Instrument the joins first

The stage counts were the most valuable output of the attribution rebuild, and they arrived late enough that a quarter of engineering effort had already gone to the wrong step.

Own mistake, own correction. This one is not borrowed from anywhere.

Charts

An unlabelled dashed line is unreadable

If the benchmark only appears in the caption, a sighted reader has to leave the figure to find out what the line means. Direct-label it in at least one panel, at annotation weight, and keep it dashed so it never reads as a series.

Direct labelling is the craft manual’s (R-007, R-082). Recorded after a review caught exactly this defect in the peer benchmarking figure.

Longer arguments live in the essays. Where a note came out of client work, the case study is on the work index.