[AI shopping & agentic commerce]() / [Use cases]() / [Assistant holdout]() [How it works]() [What you get]() [Scope]() [Start a review]() [AI shopping & agentic commerce]()[Proof & route]() # Prove your shopping assistant earns its cost. A holdout that counts everyone offered the assistant, and a decision made on contribution after what it costs to build and run. [Start with an AI shopping review]() [How it works]() Site assistant · 60-day holdout **+2.4%**orders, likely between 0.2% and 4.6% **15,360**Offered the assistant **15,000**Held out Keep it · check again in 60 days The readout: orders in the group offered the assistant against the group held out, with the range the result could sit in. ## Sound familiar? - Visitors who use the assistant convert three times as often, and nobody knows how many would have bought anyway. - The vendor’s dashboard shows chats handled, not orders earned. - Nobody has added up the build, the models and the specialist time. Engaged shoppers were already close to buying. Credit the assistant with their orders and you’ll keep paying for a feature that adds little. ## What changes. ShowTodayWith the engine The comparison Users against non-users Everyone offered it against everyone held out The metric Chats handled and engagement Contribution per eligible visitor The cost The license Build, run, models and specialist time The decision Whether it feels like it’s working A rule written before the test starts ## How it works. 6 steps, 4 stages. - AI**2** - Rules and automation**2** - Your team**2** - Approval 1. 1 Design 1. Your team Approval ### Write the decision rule Your growth and finance leads agree the break-even and what result would end the project, before launch. 2. AI ### Size the test Estimates how many eligible visitors and how many weeks it takes to see an effect worth acting on, from your traffic and conversion. 2. 2 Assign 1. Rules and automation ### Assign eligible visitors Half are offered the assistant and half aren’t, at random, and the split is logged. - Shopify - Google Analytics 3. 3 Read 1. Rules and automation ### Count everyone assigned Orders, returns and contribution per assigned visitor, whether or not they chatted. 2. AI ### Read the conversations Groups what shoppers asked and what the assistant couldn’t answer, for the next fix. - Gorgias 4. 4 Decide 1. Your team Approval ### Keep, rework or stop Your team decides against the rule, with a date to check again. Unanswered questions feed the next version ## What you get. Real work, every week. Test plan · site assistant Rule set before launch Does the assistant add orders we wouldn’t get anyway? Assigned Half of eligible visitors offered it, half held out Counted Everyone assigned, chat or no chat Read on Orders and contribution per visitor Keep if Net contribution beats the full cost to run The plan every test starts with, and the rule that decides it, agreed before any shopper is assigned. Two ways to count Fernway · 60 days Who’s counted Conversion What it shows Visitors who used it 3.1× non-users Mostly people already buying Everyone offered it +2.4% vs held out What it added: about $22,000 a month in contribution The same orders, counted two ways. Only one of them isolates what the assistant did. ## How you’ll know it’s working. 1. Contribution per assigned visitor Counts everyone offered the assistant, after returns and after what it costs to run. 2. Conversion and returns, offered against held out A lift that arrives with more returns may be no gain at all. 3. Questions the assistant couldn’t answer The list that decides what to fix, and whether to keep going. Baselines agreed before we build. Is a $5,000-a-month assistant worth it? Eligible sessions a month 75,000 Baseline conversion 3.0% Contribution per extra order $45 Cost to run $5,000 a month Cost to build $60,000 Net at a 0.3 point lift +$5,125 a month An illustration, not Fernway’s numbers. A 0.1 point lift loses $1,625 a month, and break-even is about 0.15 point. At 0.3 point, the build pays back in about 11.7 months. ## Works with what you have. Commerce - Shopify Support - Gorgias Measurement - Google Analytics We configure first Your assistant platform and analytics tool can often split traffic and report orders by group. We use their built-in assignment where it counts everyone assigned. We build when You run more than one assistant, quiz or chat, and need one comparable readout in contribution. ## What a first build includes. Built in your accounts, around your team. Scope, fee and baseline agreed before we start. 1. 1 Design 2. 2 Build 3. 3 Run it, or hand it over [Start with a review]() [Scope this build]() ### We build - A written decision rule, with the break-even - A randomized holdout on eligible visitors - A readout in contribution after the full cost - A monthly review of unanswered questions ### We start from - Eligible traffic and conversion - Contribution per order, after returns - Assistant, analytics and storefront access ### From your team Owner Head of ecommerce, with finance Involves - Ecommerce - Finance - Customer service - Merchandising ## Questions teams ask. Is our traffic big enough to test? A smaller site may not have the traffic to see a small lift in a reasonable time. Then we start with task tests and fixing obvious defects instead, and say so before you spend. Why not just compare people who used it? Shoppers who start a chat are often the ones ready to buy. Comparing them with everyone else counts orders that would have happened anyway. A holdout counts everyone offered the assistant, so the difference is the assistant’s. Is there evidence AI pre-sale help works at all? Fang et al., a 2026 working paper on experiments at one cross-border retailer, found AI pre-sale help raised sales 16.3% for shoppers who sought help, against unavailable service. AI did not significantly outperform staffed humans, and AI plus human escalation raised sales 25%. It’s one retailer, and it measures sales rather than profit, which is why we test on yours. Who owns it once it’s running? You do. It’s built in your accounts, documented, with training for the people who run it. Ownership usually sits with your head of ecommerce, with finance. We can keep running it with you, or step back. How do we start? Usually with an AI shopping review: two to four weeks, a fixed fee credited to the build, and a plan that’s yours. If the job is already clear, we can scope the build directly. ## Part of your AI shopping engine. One of 13 use cases, in 5 parts. [See the whole AI shopping engine]() 1. [Found & right **3**]() 2. [Help choosing **3**]() 3. [Orders that complete **3**]() 4. [Buying on account **1**]() 5. [Proof & route **3**]() In proof & route - [**Search diagnosis**Know why your search traffic moved]() - **Assistant holdout**You’re here - [**Channel strategy**Choose your route into AI shopping: activate, buy, extend or build]() Works with - [**Incrementality testing**Prove what’s working, then move the budget]() See it at work - [**In Fernway’s launch week, through AI**Day 60: the holdout read]() - [**Also in**Digital commerce & conversion]() ## Know what the assistant earned. Start with an AI shopping review. [Start with an AI shopping review]() [**Talk it through first**15 minutes]() We reply within one business day. --- Source: [https://www.thekeating.co/use-cases/assistant-holdout/]() [Contact Keating & Co](mailto:nelson@thekeating.co) · [Agent guidance](https://www.thekeating.co/llms.txt)