Single-test failure
A single-test failure is when one experiment designed to guide more users toward a specific behavior doesn't deliver. Either the intervention didn't actually move more users into the behavior, or it moved them in but they didn't convert at the rate earlier users had. The default reading is intervention-quality variance: the experiment itself was poorly designed, miscalibrated, or aimed at the wrong audience. One failure isn't enough on its own to conclude the team's bet on the behavior was misjudged.
Definition
A single scale verification test where the intervention didn't deliver — either lift didn't move the candidate behavior, or lift passed but hold didn't follow.
The two failure shapes
Each scale verification reading checks two things: lift (the share of the cohort performing the candidate behavior increased after the intervention) and hold (the conversion rate among performers stayed within the metric's natural variation band). A single-test failure splits into two cases:
- Lift didn't move: the intervention failed to amplify the behavior in the first place. Whatever the team did to nudge users toward the candidate behavior, more users didn't do it.
- Lift passed but hold didn't follow: the behavior amplified, but the conversion correlation among performers dropped. More users did the behavior; fewer of those users converted at the rate the original multiplier predicted.
Both register as failures of the same test. Their diagnostic significance is the same: one reading, not enough to draw conclusions about the candidate behavior itself.
The charitable read
On any single test, the read is intervention-quality variance. The intervention may have been miscalibrated, applied to the wrong sub-population, mistimed, or otherwise incomplete. The candidate behavior's status is unchanged.
Hold's tolerance band is itself a practitioner-judgment call, calibrated against the conversion-rate metric's long-term volatility — jumpier historical trends warrant wider bands, stable trends tighter. A hold reading just outside the band is a different signal than one that crashes; the threshold is treated as binary for cumulative diagnosis, but the actual magnitude matters when interpreting a single result.
What gets reconsidered
The intervention design is what warrants review after a single failure, not the candidate behavior. The team's question is what made this intervention fail to land — wrong audience, wrong moment, wrong mechanism. The candidate behavior remains a candidate.
Conclusions about the proxy itself only emerge through accumulation. Repeated lift failures across diverse interventions point to lift failure — the team's intervention-design capacity is what's failing, not the proxy. Repeated hold failures point to hold failure, which triggers proxy failure: the conclusion that the candidate behavior is a signal, not a lever.
Example
A team is testing "created a project workspace" as a candidate behavior. The current multiplier looks strong — signups who create a workspace convert at 4x the rate of those who don't. They ship an onboarding tooltip that prompts workspace creation right after signup. Post-intervention, the share of signups creating workspaces jumps from 22% to 38%, so lift passes. But the conversion multiplier among workspace creators drops from 4x to 2.3x — hold fails.
The charitable read is intervention-quality variance. The tooltip may have pulled in users who created workspaces without the underlying intent — clicking through the prompt, making a workspace, then disengaging — dragging the multiplier down for reasons unrelated to whether workspace creation actually drives the outcome. Next iteration calls for more selective intervention design: perhaps tying the prompt to intent signals from the sorting hat, or framing it to invite only users who can articulate why they'd want a workspace. The candidate behavior stays in play; the intervention gets redesigned.