Retention Operations
Part of Retention experimentation
Designing a holdout for a retention campaign
Define the control treatment, assign customers once, check what the platform withholds and measure a campaign against its holdout.
A campaign holdout is a group of eligible customers assigned before launch to receive the usual treatment instead of the proposed campaign. Compare their later outcomes with those of customers assigned to the campaign.
Define exactly what is withheld. Suppressing all marketing tests is a broader change than withholding one retention message.
Define the control treatment
Write down the proposed campaign, eligible customers and what the control group will receive. For a new reorder email, the control might receive ordinary service and existing marketing but no new reorder email. List messages that can reach both groups. Deliver required order updates and remedies under the same policy regardless of assignment.
Set eligibility from information available before assignment, such as a relevant completed purchase, product group, buying opportunity and permission for the intended channel. Define how a current order, subscription or return affects eligibility. Do not remove customers later because they did not open, click or redeem; those behaviours occur after assignment.
Use a stable customer identifier and retain the assignment. A customer who re-enters a flow after another order should not be randomly assigned again to a different arm. Document how guest and account records are handled if they merge.
Control Treatment vs. Campaign Treatment: Key Differences
- Control Group Treatment
- Receives standard service and existing marketing; no new reorder email
- Campaign Group Treatment
- Receives the new targeted reorder email as part of the retention campaign
- Eligibility Criteria
- Based on completed purchase, product group, channel permission, and stable customer ID
- Assignment Stability
- One-time assignment using a persistent identifier; no reassignment after re-engagement
Check the tool's actual holdout scope
Klaviyo documents a flow split using a Random sample condition to send profiles down different branches. A branch can serve as a campaign control if it ends without the new targeted message and other paths do not send it. The split itself does not verify the rest of a retailer's flow or contacts.
Klaviyo's global holdout has a different scope. Its documentation says it withholds campaign and flow email, SMS and push messages across channels, subject to overrides, and requires at least 400,000 total profiles. That threshold is for the global feature, not a recommended size for a single-campaign holdout.
Inspect the chosen branch, overrides and other active campaigns before interpreting a result. Before launch, check the configuration against test profiles and representative eligible records. Check delayed sends, recent-purchase suppression, opt-outs and whether the control could obtain the offer through another campaign, public code or second account. Record exposure that cannot be prevented.
Holdout Design Requirements and Benchmarks
- Recommended Holdout Size
- Not fixed – depends on campaign precision and tolerance for uncertainty
- Key Risk Factors
- Contamination via other campaigns, public codes, or second accounts
- Required Data Points
- Eligibility snapshot, assignment method, suppression rules, outcome rule
Fix the outcome and denominator
Choose a follow-up period that allows a plausible next purchase. Define qualifying completed orders and how substitutes, refunds, subscriptions and offline purchases count. Use the same data coverage for both groups.
For the main comparison, report outcomes for everyone originally assigned, including people whose message was skipped later under a shared rule. Show delivery and contamination separately. Comparing delivered recipients with the holdout breaks the original random comparison.
Report assigned customer counts, completed purchasers, orders and contribution per assigned customer when costs are sufficiently covered. Plan the allocation and review date from the likely audience, purchase frequency and uncertainty the decision can tolerate.
No fixed holdout percentage suits every campaign. If the result is imprecise, report that limit instead of declaring a winner.
Keep the eligibility snapshot, assignment method, treatments available in each group, suppressions, other contacts, outcome rule and follow-up cut-off with the result. Those details show whether the comparison measured this campaign or a wider change.
