Prove your shopping assistant earns its cost.

A holdout that counts everyone offered the assistant, and a decision made on contribution after what it costs to build and run.

Start with an AI shopping review How it works

Site assistant · 60-day holdout

+2.4%orders, likely between 0.2% and 4.6%

15,360Offered the assistant

15,000Held out

Keep it · check again in 60 days

The readout: orders in the group offered the assistant against the group held out, with the range the result could sit in.

Sound familiar?

Visitors who use the assistant convert three times as often, and nobody knows how many would have bought anyway.
The vendor’s dashboard shows chats handled, not orders earned.
Nobody has added up the build, the models and the specialist time.

Engaged shoppers were already close to buying. Credit the assistant with their orders and you’ll keep paying for a feature that adds little.

What changes.

TodayWith the engine

The comparison: Users against non-usersEveryone offered it against everyone held out
The metric: Chats handled and engagementContribution per eligible visitor
The cost: The licenseBuild, run, models and specialist time
The decision: Whether it feels like it’s workingA rule written before the test starts

How it works. 6 steps, 4 stages.

AI2
Rules and automation2
Your team2
Approval

1Design
1. Your teamApproval
  Write the decision rule
  Your growth and finance leads agree the break-even and what result would end the project, before launch.
2. AI
  Size the test
  Estimates how many eligible visitors and how many weeks it takes to see an effect worth acting on, from your traffic and conversion.
2Assign
1. Rules and automation
  Assign eligible visitors
  Half are offered the assistant and half aren’t, at random, and the split is logged.
  - Shopify
  - Google Analytics
3Read
1. Rules and automation
  Count everyone assigned
  Orders, returns and contribution per assigned visitor, whether or not they chatted.
2. AI
  Read the conversations
  Groups what shoppers asked and what the assistant couldn’t answer, for the next fix.
  - Gorgias
4Decide
1. Your teamApproval
  Keep, rework or stop
  Your team decides against the rule, with a date to check again.

Unanswered questions feed the next version

What you get. Real work, every week.

Test plan · site assistantRule set before launch

Does the assistant add orders we wouldn’t get anyway?

Assigned: Half of eligible visitors offered it, half held out
Counted: Everyone assigned, chat or no chat
Read on: Orders and contribution per visitor
Keep if: Net contribution beats the full cost to run

The plan every test starts with, and the rule that decides it, agreed before any shopper is assigned.

Two ways to countFernway · 60 days

Who’s counted	Conversion	What it shows
Visitors who used it	3.1× non-users	Mostly people already buying
Everyone offered it	+2.4% vs held out	What it added: about $22,000 a month in contribution

The same orders, counted two ways. Only one of them isolates what the assistant did.

How you’ll know it’s working.

Contribution per assigned visitor
Counts everyone offered the assistant, after returns and after what it costs to run.
Conversion and returns, offered against held out
A lift that arrives with more returns may be no gain at all.
Questions the assistant couldn’t answer
The list that decides what to fix, and whether to keep going.

Baselines agreed before we build.

Is a $5,000-a-month assistant worth it?

Eligible sessions a month: 75,000
Baseline conversion: 3.0%
Contribution per extra order: $45
Cost to run: $5,000 a month
Cost to build: $60,000
Net at a 0.3 point lift: +$5,125 a month

An illustration, not Fernway’s numbers. A 0.1 point lift loses $1,625 a month, and break-even is about 0.15 point. At 0.3 point, the build pays back in about 11.7 months.

Works with what you have.

Commerce

Shopify

Support

Gorgias

Measurement

Google Analytics

We configure first: Your assistant platform and analytics tool can often split traffic and report orders by group. We use their built-in assignment where it counts everyone assigned.
We build when: You run more than one assistant, quiz or chat, and need one comparable readout in contribution.

What a first build includes.

Built in your accounts, around your team. Scope, fee and baseline agreed before we start.

1Design
2Build
3Run it, or hand it over

Start with a review Scope this build

We build

A written decision rule, with the break-even
A randomized holdout on eligible visitors
A readout in contribution after the full cost
A monthly review of unanswered questions

We start from

Eligible traffic and conversion
Contribution per order, after returns
Assistant, analytics and storefront access

From your team

Owner: Head of ecommerce, with finance
Involves: Ecommerce
Finance
Customer service
Merchandising

Questions teams ask.

Is our traffic big enough to test?

A smaller site may not have the traffic to see a small lift in a reasonable time. Then we start with task tests and fixing obvious defects instead, and say so before you spend.

Why not just compare people who used it?

Shoppers who start a chat are often the ones ready to buy. Comparing them with everyone else counts orders that would have happened anyway. A holdout counts everyone offered the assistant, so the difference is the assistant’s.

Is there evidence AI pre-sale help works at all?

Fang et al., a 2026 working paper on experiments at one cross-border retailer, found AI pre-sale help raised sales 16.3% for shoppers who sought help, against unavailable service. AI did not significantly outperform staffed humans, and AI plus human escalation raised sales 25%. It’s one retailer, and it measures sales rather than profit, which is why we test on yours.

Who owns it once it’s running?

You do. It’s built in your accounts, documented, with training for the people who run it. Ownership usually sits with your head of ecommerce, with finance. We can keep running it with you, or step back.

How do we start?

Usually with an AI shopping review: two to four weeks, a fixed fee credited to the build, and a plan that’s yours. If the job is already clear, we can scope the build directly.