[Digital commerce & conversion]() / [Use cases]() / [Experiment planning]() [How it works]() [What you get]() [Scope]() [Start a review]() [Digital commerce & conversion]()[Evidence & decisions]() # Test what matters, and know when a test can’t tell you. A test plan ranked on evidence, sized to your traffic, with a holdout for every claim and a straight answer when a test would take too long. [Start with a conversion review]() [How it works]() Test backlog Fernway · this quarter Idea Evidence Call Answer “machine washable?” 41 asks, nothing on the page Fix, no test Mobile filter bug Reproduced, a defect Fix, no test Shopping assistant Users convert 3.1× Holdout Delivery date in the cart Exits at the shipping step Test, 10 weeks Button label wording Effect too small to read Skip The backlog, ranked. Some ideas get a test, and some just get fixed. ## Sound familiar? - The test backlog is whatever someone suggested last. - A test ran for a month and nobody can say what it decided. - The analytics tool calls a feature a top performer, and sales haven’t moved. Traffic is finite and tests are slow. Spent on the wrong questions, they teach you little and delay the changes that would pay. ## What changes. ShowTodayWith the engine The backlog Ideas from whoever spoke last Ranked on evidence, effort and whether a test can read it Known fixes Wait in the same queue as experiments Ship now, with no test needed Test length Run until someone gets bored Sized to your traffic before it starts Readouts People who use it convert more Offered against held out, with a decision ## How it works. 6 steps, 4 stages. - AI**2** - Rules and automation**2** - Your team**2** - Approval 1. 1 Rank 1. AI ### Draft hypotheses from the evidence Each idea carries the searches, questions and sessions behind it, and the change it proposes. - Contentsquare 2. Your team Approval ### Rank the backlog Your owner sets priority. Known defects go straight to a fix. 2. 2 Size 1. Rules and automation ### Size every test to your traffic Visitors needed, weeks to read and whether it can be read at all, before anything ships. 3. 3 Run 1. Rules and automation ### Assign and hold out Persistent assignment behind a flag, half the visitors held out, guardrails on speed and support. - GrowthBook 4. 4 Read 1. AI ### Draft the readout Offered against held out, contribution after returns, and a plain recommendation. 2. Your team Approval ### Keep, rework or remove Your team decides, and the date to check again goes in the ledger. Each readout reorders the backlog ## What you get. Real work, every week. Test 021 Shopping assistant Does offering the assistant add sales we wouldn’t get anyway? Eligible All visitors; half are never offered it Assignment Persistent, so a visitor always sees the same side Measure Contribution per visitor, after returns Guardrails Page speed and support contacts Reads Early look at week 8, full read 60 days later The test plan, agreed before anything ships. The decision it will drive is written down first. Readouts Next check in 60 days - Shopping assistantHalf the visitors held out **+2.4%**Keep - Size quizHalf the visitors held out **+0.1%**Rework - Delivery date in the cartReading since week 6 **Open**Running The readout ledger: every test, its result, what was decided and when to look again. ## How you’ll know it’s working. 1. Contribution per visitor, held out What a change added after returns, against visitors who never saw it. 2. Tests that can be read A test sized to your traffic reaches an answer. The rest were activity. 3. Time from idea to decision Weeks from a backlog entry to keep, rework or remove. Baselines agreed before we build. How long a test needs Visitors a month 500,000 Conversion now 3.00% Change worth detecting 3.00% to 3.09%, a 3% lift Visitors needed, half held out About 1.14 million Time to a clear read About 10 weeks Fernway’s traffic. At 20,000 visitors a month the same answer takes years, so smaller brands should test tasks with a few shoppers and fix defects instead. Standard 5% significance and 80% power, before returns mature. ## Works with what you have. Measurement - GrowthBook - Contentsquare - Google Analytics Commerce - Shopify We configure first Shopify’s own testing and a free GrowthBook plan handle simple tests. We start there and add only what a question needs. We build when You test more than once a month, or results need to join orders, returns and contribution. ## What a first build includes. Built in your accounts, around your team. Scope, fee and baseline agreed before we start. 1. 1 Design 2. 2 Build 3. 3 Run it, or hand it over [Start with a review]() [Scope this build]() ### We build - A ranked backlog with the evidence behind each idea - Test designs sized to your traffic - Assignment and holdouts behind flags - A readout in contribution, with a date to check again ### We start from - Orders, returns and event data - Current experiments and past results - Your release and flag tooling ### From your team Owner Head of ecommerce, with analytics Involves - Ecommerce - Analytics - Engineering - Design ## Questions teams ask. Are we big enough to test? It depends on your traffic and on how large a change you’re after. At 20,000 visitors a month a 3% change takes years to read. Below that kind of volume we test tasks with a few shoppers, fix defects and make larger changes. Why hold out half the visitors? It’s the only fair way to separate what a feature did from who chose to use it. People who use an assistant already buy more, so “users convert 3.1× more” says little about what it added. Can AI draft our variants? Yes, and your team approves each one. AI editors in test tools speed up drafting, though they don’t decide what to test, and a winner still has to reach your maintained code. Who owns it once it’s running? You do. It’s built in your accounts, documented, with training for the people who run it. Ownership usually sits with your head of ecommerce, with analytics. We can keep running it with you, or step back. How do we start? Usually with a conversion review: two to four weeks, a fixed fee credited to the build, and a plan that’s yours. If the job is already clear, we can scope the build directly. ## Part of your conversion engine. One of 14 use cases, in 5 parts. [See the whole conversion engine]() 1. [Evidence & decisions **2**]() 2. [Search & discovery **2**]() 3. [Guidance **4**]() 4. [Cart & checkout **4**]() 5. [Releases **2**]() In evidence & decisions - [**Conversion diagnosis**Find what’s costing you sales, and what to fix first]() - **Experiment planning**You’re here Works with - [**Release assurance**Ship proven changes every week, safely]() - [**Incrementality testing**Prove what’s working, then move the budget]() See it at work - [**In Fernway’s eight weeks**Week 8: the readout]() - [**Also in**Customer service & support]() ## Know what actually worked. Start with a conversion review. [Start with a conversion review]() [**Talk it through first**15 minutes]() We reply within one business day. --- Source: [https://www.thekeating.co/use-cases/experiment-planning/]() [Contact Keating & Co](mailto:nelson@thekeating.co) · [Agent guidance](https://www.thekeating.co/llms.txt)