Retention Operations
Part of Retention investment decisions
Reviewing retention results after an operational change
Check whether an operational fix was delivered, compare later customer outcomes fairly and qualify any claimed retention effect.
Review an operational change in two stages: verify that the process problem improved, then assess whether comparable customers bought again more often or contributed more. A rise in retention after a repair is encouraging, but it does not by itself show that the repair caused the rise.
Two-stage review of an operational change
- Verify that the process problem improved
- Assess whether comparable customers bought again more often or contributed more
Fix the change and exposure dates
Describe exactly what changed: for example, a dispatch handoff, stock replenishment rule or support escalation route. Record approval, rollout and actual-use dates.
If the new process reached one warehouse or product group before another, retain that rollout history. A customer belongs in the exposed group only if their relevant order or service event could have been affected by the new process.
Keep the old and new process definitions. A change in complaint tagging or ticket closure can make an issue rate appear better without changing the customer experience. Check a sample of source records and the operational measure the repair was meant to improve, such as missed dispatch handoffs per eligible order.
Change and exposure date checklist
- Describe exactly what changed (for example, a dispatch handoff, stock replenishment rule or support escalation route)
- Record approval, rollout and actual-use dates
- Retain rollout history if the new process reached one warehouse or product group before another
- Place a customer in the exposed group only if their relevant order or service event could have been affected
- Keep the old and new process definitions
- Check a sample of source records and the operational measure the repair was meant to improve
Give customers a fair buying window
Define a qualifying later completed purchase and the period in which a customer could reasonably make it. Compare customers with similar first products, buying seasons and elapsed follow-up. Keep recent customers pending until their window is complete.
Report eligible counts, returners, relevant orders and contribution under one cost rule. Reconcile refunds, replacement orders and missing sales channels before interpreting a trend.
Start the retention result from customers with a relevant earlier order or service event who were eligible for the process comparison, including those who never buy again. Looking only at people who contacted support or returned to purchase selects a different population. Apply the same eligibility rule before and after the change.
Fair buying-window checklist
- Define a qualifying later completed purchase and the period in which it could reasonably be made
- Compare customers with similar first products, buying seasons and elapsed follow-up
- Keep recent customers pending until their window is complete
- Report eligible counts, returners, relevant orders and contribution under one cost rule
- Reconcile refunds, replacement orders and missing sales channels before interpreting a trend
- Start from customers with a relevant earlier order or service event who were eligible, including those who never buy again
- Apply the same eligibility rule before and after the change
Choose a credible comparison
A simple before-and-after chart may move with seasonality, a product change, advertising, stock or a shift in customer mix. Where rollout permits, compare the before-and-after outcome with that of a similar group that did not receive the process change.
Inspect several pre-change periods to see whether the groups moved in comparable ways. Record other changes that affected either group, including any spillover of the repair into the comparison group.
The arithmetic of a difference-in-differences review is:
(Post-change outcome − pre-change outcome) for the changed group − the same change for the comparison group.
Suppose, purely as an illustration, the changed group's mature repeat-purchaser rate rose from 30% to 40%, while a comparison group's rose from 25% to 30%. The difference in changes is five percentage points. These invented rates are not a result.
Even with real data, the calculation would support a causal interpretation only if the groups and outcome definitions were suitable. Their outcomes would plausibly have followed parallel trends without the repair, other influences were addressed and the estimate was sufficiently precise. Similar observed pre-change trends can support that judgement but cannot prove the assumption.
If no suitable comparison exists, show the process measure and later retention trend as separate observations. Say what else changed. Avoid turning the difference between two calendar periods into a claimed financial return from the repair.
Difference-in-differences arithmetic
- Calculate the post-change outcome minus the pre-change outcome for the changed group
- Calculate the same change for the comparison group
- Subtract the comparison group's change from the changed group's change
- Only support a causal interpretation if parallel trends were plausible, other influences were addressed and the estimate was sufficiently precise
- If no suitable comparison exists, show the process measure and later retention trend as separate observations
- State what else changed and avoid turning the difference between two calendar periods into a claimed financial return from the repair
Reconcile the commercial result and decide
Calculate contribution from qualifying later orders after discounts, refunds, product cost and relevant variable costs. Set the process-change cost beside any estimated added contribution, including ongoing staff or supplier expense and avoiding costs already counted in order contribution. A reduction in the original failure may still be valuable when another purchase has not yet become observable. Keep that operational result distinct from an unmeasured or uncertain retention effect.
End the review with a decision: continue and monitor, adjust the process, extend to another group with a stronger measurement plan, or investigate a competing explanation. State the rollout dates, mature customer counts, comparison limits, cost coverage and next buying-window checkpoint.
Reconcile the commercial result and decide
- Calculate contribution from qualifying later orders after discounts, refunds, product cost and relevant variable costs
- Set the process-change cost beside any estimated added contribution
- Include ongoing staff or supplier expense and avoid counting costs already included in order contribution
- Keep a reduction in the original failure distinct from an unmeasured or uncertain retention effect
- Decide whether to continue and monitor, adjust the process, extend to another group with a stronger measurement plan, or investigate a competing explanation
- State rollout dates, mature customer counts, comparison limits, cost coverage and the next buying-window checkpoint
