Churn Diagnosis
Retention experimentation
Plan retention experiments with a clear comparison, stable customer assignment and outcomes that account for contribution and purchase timing.
Retention experimentation tests whether an action changes customer behaviour compared with what those customers would otherwise receive. Orders after a campaign show what happened; they do not establish what the campaign added. Define the decision, comparison and outcome before launch.
Use five checks: question, comparison, assignment, outcome and interpretation. Set the first four before launch; use the fifth to decide what the result does and does not support.
Start with one decision
Ask a question the result can answer, such as whether a proposed reorder reminder improves on the usual customer experience. Name the eligible customers, the relevant purchase and the period in which they could reasonably buy again. A refill buyer and a buyer of a durable product may need different follow-up periods.
Specify the alternative treatment. Customers assigned to the comparison group may still receive ordinary service and unrelated marketing. Record those contacts so the result is interpreted against the treatment they actually received.
Decision / Comparison needed
- Does a new reminder add value?
- Reminder versus usual treatment
- Does an incentive improve a reminder?
- The same core reminder with and without the incentive
- Which message version performs better?
- Two versions sent to comparable customers
The last comparison identifies a preferred version within the test. Without a usual-treatment group, it cannot show whether either version improves on the existing approach.
Make the groups comparable
Where feasible, randomly assign eligible customers before the action can reach them. Use a stable customer identifier so a later order or another channel does not put the same person in a different group.
Apply the same entry rule, outcome definition and observation window to each group. Record overlapping promotions, stock changes and other events that could affect interpretation.
Plan for enough eligible customers and qualifying outcomes to make the decision useful. No fixed holdout percentage guarantees an informative result. If the likely estimate will be too uncertain, narrow the question, recruit for longer or call the test exploratory. Set the review point before looking for a favourable result.
Keep required order updates and remedies outside the discretionary marketing test. A customer should receive service owed to them regardless of assignment.
Measure the assigned groups
Compare outcomes for everyone originally assigned, including people who never opened a message or used a code. Comparing redeemers with non-redeemers selects customers by behaviour that happened after assignment.
Report purchasers and completed orders, then assess contribution per assigned customer when costs are reliable. Start with sales after discounts and refunds. Subtract product cost, relevant variable order costs and campaign costs, counting each cost once. State what is missing if the result is only a partial contribution measure.
Connect the primary outcome to the retention decision: churn rate, net revenue retention (NRR), gross revenue retention (GRR) or lifetime value (LTV) may be relevant measures. Choose the measure that matches the question and define its observation period before launch.
Follow purchasing through a plausible next buying opportunity. An early order may replace a later one, especially when customers can stock up. Clicks, opens and attributed sales help describe delivery and engagement; they do not prove extra demand.
Key retention metrics and their relevance
- Churn Rate
- Percentage of customers who stop using the service within a period
- Net Revenue Retention (NRR)
- Revenue from existing customers after upgrades, downgrades and churn
- Gross Revenue Retention (GRR)
- Total revenue retained from existing customers before upsells or downgrades
- Lifetime Value (LTV)
- Predicted total revenue from a customer over their relationship with the business
Interpret the result in context
A result can establish whether an action was associated with an impact without explaining how or why it occurred. When outcomes differ across settings, consider whether delivery or context may have influenced them before treating the estimate as transferable.
For a complex intervention, qualitative comparative analysis can systematically compare intervention features and contextual factors against outcomes. It looks for patterns and combinations of conditions, including cases where different combinations may lead to the same outcome. It is useful for understanding variation, but does not provide a precise estimate of effect size.
Use that kind of analysis to frame follow-up questions: which conditions appear necessary for success, and do the same patterns occur in other tested settings? Treat the answers as context for the decision, not as a replacement for the assigned-group comparison.
Use the result for the decision tested
Report group sizes, dates, the observed difference, uncertainty, cost coverage and any exposure overlap. A noisy difference may justify another test. A useful result applies first to the tested audience, product and period; it does not automatically transfer to another season or customer group.
Keep a short record of the question, eligibility rule, assignment method, treatments, primary outcome, planned end date, other marketing and data gaps.
For Australian promotional email or SMS, check the applicable requirements under the Spam Act 2003, administered by the ACMA, before sending.
If test findings are used to support a claim about a product or service, keep evidence that backs up the claim. The ACCC can require businesses to substantiate claims and may investigate or take compliance or enforcement action if a business misleads.
In this guide
- Designing a holdout for a retention campaignDefine the control treatment, assign customers once, check what the platform withholds and measure a campaign against its holdout.
- Comparing a reminder with an incentiveDesign a fair reminder versus incentive test and judge what the offer adds after its cost.
- Measuring whether an offer simply discounts an inevitable purchaseCompare a targeted offer with usual treatment using all purchases and contribution per assigned customer.
