Prove which messages and discounts earn their cost.
Holdout tests that show what each message and offer added, counted after discounts, returns and the cost to run.
+1 pointrepeat orders, likely between 0.4 and 1.6 points
Keep the reminder · drop the $10 code · check again in 60 days
Repeat orders per 40,000 customers over 60 days, with and without the reminder. The read is contribution after the code.
Sound familiar?
- Attributed revenue is up. So are discounts and returns.
- Nobody can say whether the extra order paid for its offer.
- The welcome code, the browse email and the weekly campaign all claim the same order.
Attributed revenue looks the same whether a program earns money or loses it, so a loss can run for quarters.
What changes.
- Proof
- Revenue each tool credits to itselfA holdout: customers who didn’t get it
- Counting
- Orders inside an attribution windowExtra orders, after discounts and returns
- Discounts
- Judged by how many redeemJudged by the orders they added, per dollar
- After the test
- A slideA decision, with a date to check again
How it works. 6 steps, 4 stages.
- AI2
- Rules and automation2
- Your team2
- Approval
1Pick
Your team
Pick the message and the question
One reminder, code or campaign at a time, and the result that would change what you do.
2Assign
AI
Size the test
Works out how many customers, and how many weeks, it takes to see the smallest lift worth having.
Rules and automation
Assign a holdout before sending
Customers are randomly assigned and recorded before anyone is exposed, and everyone assigned is counted.
- Klaviyo
- Hightouch
3Count
Rules and automation
Count orders, discounts and returns
Joined from Shopify, not taken from the revenue Klaviyo attributes.
- Shopify
- Klaviyo
AI
Read the result
The lift with a range, turned into contribution per extra order.
4Decide
Your teamApproval
Make the call
Keep, change or stop, with an owner and a date to check again.
Every decision gets a date to check again
What you get. Real work, every week.
Does the reminder bring orders we wouldn’t get anyway?
- Method
- A random holdout, assigned before sending
- Customers
- 40,000 eligible, over 60 days
- Count
- Orders, discounts and returns from Shopify
- Decision
- Keep it if contribution beats the cost to run
Every test starts with a plan, and the decision it will drive, agreed before anything sends.
- Care Kit reminderHoldout · 60 days · 30% used the code+1 pointDrop the code
- Restock alertsHoldout · proven earlierProvenKeep
- 20% welcome codeNever tested · $28,000 creditedNot testedTest next
The decision ledger: what each program claimed, what a test found and what was done about it.
How you’ll know it’s working.
Extra orders per eligible customer
The lift against the holdout, with its range, instead of orders inside an attribution window.
Contribution per extra order, after offers
What each extra order earned after product, shipping, returns, the discount and the cost to run.
Offer spend that didn’t change behavior
Discounts given to customers who’d have ordered anyway, the cost most dashboards leave out.
Baselines agreed before we build.
- Eligible customers, 60 days
- 40,000
- Repeat rate, 12% to 13%
- +400 orders
- Contribution at $40 an order
- $16,000
- $10 code, used on 8% of orders
- −$4,160
- Cost to run the reminder
- −$5,000
- Net over 60 days
- +$6,840
At 30% code use the discount costs $15,600 and the same one-point lift loses $4,600. The lift needed to break even rises from about 0.6 points to about 1.3. Fernway’s sample numbers, to show how the sum works.
Works with what you have.
Email & SMS
- Klaviyo
- Braze
- Attentive
Data
- Hightouch
Commerce
- Shopify
- We configure first
- Klaviyo’s global holdout and Braze’s global control cover the largest lists. We use them where they fit, and check what a control really excludes.
- We build when
- Your lists are too small for the built-in holdouts, or you need tests joined to refunds, discounts and finance’s numbers.
What a first build includes.
Built in your accounts, around your team. Scope, fee and baseline agreed before we start.
- 1Design
- 2Build
- 3Run it, or hand it over
We build
- A testing roadmap ranked by spend on offers
- Randomized holdouts set up before sending
- Readouts in contribution, with ranges
- A decision ledger with dates to check again
We start from
- Order, refund and discount data
- Klaviyo and SMS access
- Finance’s contribution per order
From your team
- Owner
- Head of lifecycle, with finance
- Involves
- Lifecycle
- Finance
- Growth
Questions teams ask.
Are our lists big enough to test?
Often, for the programs that matter most. Klaviyo’s built-in global holdout documents a 400,000-profile minimum, but a narrower randomized test doesn’t need that. If a test would take too long to read, we’ll say so and recommend something simpler.
Should we target the customers most likely to leave?
Not on that alone. Research in telecom and membership settings found that the customers at highest risk of leaving aren’t always the ones who respond to a retention offer (Ascarza, Journal of Marketing Research, 2018). We test who actually changes behavior.
How is this different from testing ad spend?
The method is the same and the question differs. That job tests acquisition spend. This one tests messages and offers to customers you already have, and counts discounts and returns against each extra order.
Who owns it once it’s running?
You do. It’s built in your accounts, documented, with training for the people who run it. Ownership usually sits with your head of lifecycle, with finance. We can keep running it with you, or step back.
How do we start?
Usually with a retention review: two to four weeks, a fixed fee credited to the build, and a plan that’s yours. If the job is already clear, we can scope the build directly.
Part of your retention engine. One of 12 use cases, in 6 parts.
See the whole retention engine- Second orders3
- Subscribers3
- Contact that fits1
- Owners & loyalty2
- Lifecycle production1
- Proof & decisions2
In proof & decisions
- Retention proofYou’re here
- Lifecycle diagnosisExplain why email and SMS revenue moved
Works with
Find out which messages actually pay. Start with a retention review.
We reply within one business day.