Test what matters, and know when a test can’t tell you.
A test plan ranked on evidence, sized to your traffic, with a holdout for every claim and a straight answer when a test would take too long.
| Idea | Evidence | Call |
|---|---|---|
| Answer “machine washable?” | 41 asks, nothing on the page | Fix, no test |
| Mobile filter bug | Reproduced, a defect | Fix, no test |
| Shopping assistant | Users convert 3.1× | Holdout |
| Delivery date in the cart | Exits at the shipping step | Test, 10 weeks |
| Button label wording | Effect too small to read | Skip |
The backlog, ranked. Some ideas get a test, and some just get fixed.
Sound familiar?
- The test backlog is whatever someone suggested last.
- A test ran for a month and nobody can say what it decided.
- The analytics tool calls a feature a top performer, and sales haven’t moved.
Traffic is finite and tests are slow. Spent on the wrong questions, they teach you little and delay the changes that would pay.
What changes.
- The backlog
- Ideas from whoever spoke lastRanked on evidence, effort and whether a test can read it
- Known fixes
- Wait in the same queue as experimentsShip now, with no test needed
- Test length
- Run until someone gets boredSized to your traffic before it starts
- Readouts
- People who use it convert moreOffered against held out, with a decision
How it works. 6 steps, 4 stages.
- AI2
- Rules and automation2
- Your team2
- Approval
1Rank
AI
Draft hypotheses from the evidence
Each idea carries the searches, questions and sessions behind it, and the change it proposes.
- Contentsquare
Your teamApproval
Rank the backlog
Your owner sets priority. Known defects go straight to a fix.
2Size
Rules and automation
Size every test to your traffic
Visitors needed, weeks to read and whether it can be read at all, before anything ships.
3Run
Rules and automation
Assign and hold out
Persistent assignment behind a flag, half the visitors held out, guardrails on speed and support.
- GrowthBook
4Read
AI
Draft the readout
Offered against held out, contribution after returns, and a plain recommendation.
Your teamApproval
Keep, rework or remove
Your team decides, and the date to check again goes in the ledger.
Each readout reorders the backlog
What you get. Real work, every week.
Does offering the assistant add sales we wouldn’t get anyway?
- Eligible
- All visitors; half are never offered it
- Assignment
- Persistent, so a visitor always sees the same side
- Measure
- Contribution per visitor, after returns
- Guardrails
- Page speed and support contacts
- Reads
- Early look at week 8, full read 60 days later
The test plan, agreed before anything ships. The decision it will drive is written down first.
- Shopping assistantHalf the visitors held out+2.4%Keep
- Size quizHalf the visitors held out+0.1%Rework
- Delivery date in the cartReading since week 6OpenRunning
The readout ledger: every test, its result, what was decided and when to look again.
How you’ll know it’s working.
Contribution per visitor, held out
What a change added after returns, against visitors who never saw it.
Tests that can be read
A test sized to your traffic reaches an answer. The rest were activity.
Time from idea to decision
Weeks from a backlog entry to keep, rework or remove.
Baselines agreed before we build.
- Visitors a month
- 500,000
- Conversion now
- 3.00%
- Change worth detecting
- 3.00% to 3.09%, a 3% lift
- Visitors needed, half held out
- About 1.14 million
- Time to a clear read
- About 10 weeks
Fernway’s traffic. At 20,000 visitors a month the same answer takes years, so smaller brands should test tasks with a few shoppers and fix defects instead. Standard 5% significance and 80% power, before returns mature.
Works with what you have.
Measurement
- GrowthBook
- Contentsquare
- Google Analytics
Commerce
- Shopify
- We configure first
- Shopify’s own testing and a free GrowthBook plan handle simple tests. We start there and add only what a question needs.
- We build when
- You test more than once a month, or results need to join orders, returns and contribution.
What a first build includes.
Built in your accounts, around your team. Scope, fee and baseline agreed before we start.
- 1Design
- 2Build
- 3Run it, or hand it over
We build
- A ranked backlog with the evidence behind each idea
- Test designs sized to your traffic
- Assignment and holdouts behind flags
- A readout in contribution, with a date to check again
We start from
- Orders, returns and event data
- Current experiments and past results
- Your release and flag tooling
From your team
- Owner
- Head of ecommerce, with analytics
- Involves
- Ecommerce
- Analytics
- Engineering
- Design
Questions teams ask.
Are we big enough to test?
It depends on your traffic and on how large a change you’re after. At 20,000 visitors a month a 3% change takes years to read. Below that kind of volume we test tasks with a few shoppers, fix defects and make larger changes.
Why hold out half the visitors?
It’s the only fair way to separate what a feature did from who chose to use it. People who use an assistant already buy more, so “users convert 3.1× more” says little about what it added.
Can AI draft our variants?
Yes, and your team approves each one. AI editors in test tools speed up drafting, though they don’t decide what to test, and a winner still has to reach your maintained code.
Who owns it once it’s running?
You do. It’s built in your accounts, documented, with training for the people who run it. Ownership usually sits with your head of ecommerce, with analytics. We can keep running it with you, or step back.
How do we start?
Usually with a conversion review: two to four weeks, a fixed fee credited to the build, and a plan that’s yours. If the job is already clear, we can scope the build directly.
Part of your conversion engine. One of 14 use cases, in 5 parts.
See the whole conversion engineIn evidence & decisions
- Conversion diagnosisFind what’s costing you sales, and what to fix first
- Experiment planningYou’re here
Works with
Know what actually worked. Start with a conversion review.
We reply within one business day.