[Retention & lifecycle]() / [Use cases]() / [Retention proof]() [How it works]() [What you get]() [Scope]() [Start a review]() [Retention & lifecycle]()[Proof & decisions]() # Prove which messages and discounts earn their cost. Holdout tests that show what each message and offer added, counted after discounts, returns and the cost to run. [Start with a retention review]() [How it works]() Care Kit reminder · 60-day holdout **+1 point**repeat orders, likely between 0.4 and 1.6 points **5,200**Got the reminder **4,800**Held out Keep the reminder · drop the $10 code · check again in 60 days Repeat orders per 40,000 customers over 60 days, with and without the reminder. The read is contribution after the code. ## Sound familiar? - Attributed revenue is up. So are discounts and returns. - Nobody can say whether the extra order paid for its offer. - The welcome code, the browse email and the weekly campaign all claim the same order. Attributed revenue looks the same whether a program earns money or loses it, so a loss can run for quarters. ## What changes. ShowTodayWith the engine Proof Revenue each tool credits to itself A holdout: customers who didn’t get it Counting Orders inside an attribution window Extra orders, after discounts and returns Discounts Judged by how many redeem Judged by the orders they added, per dollar After the test A slide A decision, with a date to check again ## How it works. 6 steps, 4 stages. - AI**2** - Rules and automation**2** - Your team**2** - Approval 1. 1 Pick 1. Your team ### Pick the message and the question One reminder, code or campaign at a time, and the result that would change what you do. 2. 2 Assign 1. AI ### Size the test Works out how many customers, and how many weeks, it takes to see the smallest lift worth having. 2. Rules and automation ### Assign a holdout before sending Customers are randomly assigned and recorded before anyone is exposed, and everyone assigned is counted. - Klaviyo - Hightouch 3. 3 Count 1. Rules and automation ### Count orders, discounts and returns Joined from Shopify, not taken from the revenue Klaviyo attributes. - Shopify - Klaviyo 2. AI ### Read the result The lift with a range, turned into contribution per extra order. 4. 4 Decide 1. Your team Approval ### Make the call Keep, change or stop, with an owner and a date to check again. Every decision gets a date to check again ## What you get. Real work, every week. Test plan Care Kit reminder Does the reminder bring orders we wouldn’t get anyway? Method A random holdout, assigned before sending Customers 40,000 eligible, over 60 days Count Orders, discounts and returns from Shopify Decision Keep it if contribution beats the cost to run Every test starts with a plan, and the decision it will drive, agreed before anything sends. Decision ledger Next check · Mar 15 - Care Kit reminderHoldout · 60 days · 30% used the code **+1 point**Drop the code - Restock alertsHoldout · proven earlier **Proven**Keep - 20% welcome codeNever tested · $28,000 credited **Not tested**Test next The decision ledger: what each program claimed, what a test found and what was done about it. ## How you’ll know it’s working. 1. Extra orders per eligible customer The lift against the holdout, with its range, instead of orders inside an attribution window. 2. Contribution per extra order, after offers What each extra order earned after product, shipping, returns, the discount and the cost to run. 3. Offer spend that didn’t change behavior Discounts given to customers who’d have ordered anyway, the cost most dashboards leave out. Baselines agreed before we build. Is the $10 code worth it? Eligible customers, 60 days 40,000 Repeat rate, 12% to 13% +400 orders Contribution at $40 an order $16,000 $10 code, used on 8% of orders −$4,160 Cost to run the reminder −$5,000 Net over 60 days +$6,840 At 30% code use the discount costs $15,600 and the same one-point lift loses $4,600. The lift needed to break even rises from about 0.6 points to about 1.3. Fernway’s sample numbers, to show how the sum works. ## Works with what you have. Email & SMS - Klaviyo - Braze - Attentive Data - Hightouch Commerce - Shopify We configure first Klaviyo’s global holdout and Braze’s global control cover the largest lists. We use them where they fit, and check what a control really excludes. We build when Your lists are too small for the built-in holdouts, or you need tests joined to refunds, discounts and finance’s numbers. ## What a first build includes. Built in your accounts, around your team. Scope, fee and baseline agreed before we start. 1. 1 Design 2. 2 Build 3. 3 Run it, or hand it over [Start with a review]() [Scope this build]() ### We build - A testing roadmap ranked by spend on offers - Randomized holdouts set up before sending - Readouts in contribution, with ranges - A decision ledger with dates to check again ### We start from - Order, refund and discount data - Klaviyo and SMS access - Finance’s contribution per order ### From your team Owner Head of lifecycle, with finance Involves - Lifecycle - Finance - Growth ## Questions teams ask. Are our lists big enough to test? Often, for the programs that matter most. Klaviyo’s built-in global holdout documents a 400,000-profile minimum, but a narrower randomized test doesn’t need that. If a test would take too long to read, we’ll say so and recommend something simpler. Should we target the customers most likely to leave? Not on that alone. Research in telecom and membership settings found that the customers at highest risk of leaving aren’t always the ones who respond to a retention offer (Ascarza, Journal of Marketing Research, 2018). We test who actually changes behavior. How is this different from testing ad spend? The method is the same and the question differs. That job tests acquisition spend. This one tests messages and offers to customers you already have, and counts discounts and returns against each extra order. Who owns it once it’s running? You do. It’s built in your accounts, documented, with training for the people who run it. Ownership usually sits with your head of lifecycle, with finance. We can keep running it with you, or step back. How do we start? Usually with a retention review: two to four weeks, a fixed fee credited to the build, and a plan that’s yours. If the job is already clear, we can scope the build directly. ## Part of your retention engine. One of 12 use cases, in 6 parts. [See the whole retention engine]() 1. [Second orders **3**]() 2. [Subscribers **3**]() 3. [Contact that fits **1**]() 4. [Owners & loyalty **2**]() 5. [Lifecycle production **1**]() 6. [Proof & decisions **2**]() In proof & decisions - **Retention proof**You’re here - [**Lifecycle diagnosis**Explain why email and SMS revenue moved]() Works with - [**Incrementality testing**Prove what’s working, then move the budget]() - [**Replenishment timing**Time the reorder to when customers actually run out]() See it at work - [**In Fernway’s ninety days**Day 90: then we checked what actually paid]() - [**Also in**Finance & business intelligence]() ## Find out which messages actually pay. Start with a retention review. [Start with a retention review]() [**Talk it through first**15 minutes]() We reply within one business day. --- Source: [https://www.thekeating.co/use-cases/retention-incrementality/]() [Contact Keating & Co](mailto:nelson@thekeating.co) · [Agent guidance](https://www.thekeating.co/llms.txt)