Retention Operations
Part of Retention programme operations
Keeping an inventory of active retention tests
Record test ownership, assignment, live exposure, changes and follow-up so retention experiments can be operated and interpreted.
Maintain one inventory of retention tests that can still affect customers or whose outcomes remain under observation. It must show who is eligible, what each assigned group may receive, what is configured now and when follow-up ends. A campaign-name list cannot answer those questions.
Record the current test state
Give each test a stable ID, decision question and accountable owner. Record the product group, eligibility rule, assignment unit, treatments, primary outcome, enrolment dates and follow-up cut-off. Attach references to the approved messages, offers and configured paths.
Separate the date new customers stop entering from the last possible treatment date and the last outcome date. A test can stop enrolment while delayed messages are still queued. Use clear states such as proposed, approved, enrolling, treatment complete, follow-up open and closed, with an effective time for each change.
Test lifecycle stages and key dates
- Proposed
- Initial concept and planning phase
- Approved
- Formal sign-off by stakeholders
- Enrolling
- Customers can enter the test
- Treatment complete
- All assigned customers have received treatment
- Follow-up open
- Ongoing data collection period
- Follow-up closed
- Data collection ends; results finalised
Show where customers could overlap
Record whether a customer can enter another test or promotion during this test's treatment and follow-up. Note shared products, channels, audience rules and public offers. If overlap is allowed, record which benefits or contacts both assigned groups can encounter. Unequal outside exposure can change what the eventual comparison means.
Before launch, inspect representative records for duplicate customer identities, a purchase during a waiting step, an opt-out and an open service case. Check whether assignment remains stable if someone re-enters a flow. Record the configured behaviour, rather than treating an approved document as evidence that the system follows it.
Customer overlap considerations in retention testing
- Overlap allowed?
- Yes / No
- Shared products or channels
- List any shared touchpoints (e.g., email, app push)
- Shared audience rules
- Are customer segments overlapping?
- Potential exposure differences
- How benefits or messages may vary between groups
- Impact on interpretation
- Risk of biased results due to unequal exposure
Update the entry when something changes
Change / Inventory update
- Eligibility rule changes
- New rule version, effective time and affected entrants
- Message or offer changes
- Version and which assigned customers may see it
- Another promotion overlaps
- Exposure rule and possible effect on interpretation
- Test pauses or resumes
- Actual configuration state and responsible approver
- Follow-up closes
- Data cut-off, known gaps and decision-record location
Keep earlier versions when a material change occurs during a test. Identify the customers affected by each version, so the reviewer can judge whether the original comparison remains useful.
Close a test in the active inventory once its outcome, exceptions and decision are recorded. Show assignment counts and known exposure differences alongside any result. The inventory preserves operational context; it does not design a holdout or estimate an effect.
