i-avantage
Tool

Design a pilot

Is our AI pilot big and long enough to prove a gain — or only to produce impressions?

The number of participants needed to demonstrate the expected gain, the smallest gain your pilot can detect, and a week-by-week protocol with its decision criteria.

  • ≈ 3 min
  • 100% free · no AI · nothing is sent
  • You leave with: A sized pilot protocol, with decision criteria

The pilot you are planning

A pilot that cannot detect the expected gain costs as much as a real one and decides nothing. Check it before recruiting participants.

Design

More economical: each person is their own comparison.

min
20 %

The reduction in time per task you hope to demonstrate.

2
4
Advanced: how spread out the measurements are

Cautious defaults. Replace them with what you observe in the baseline week: that is what makes the sizing exact.

30 %
50 %
50 %
Your pilot can show a gain of 18.8 % or more
You expect 20%: if the gain is real, the measurements will show it.

The sizing

Participants needed
14
The same people, measured before and then with AI.
Participants planned
15
Smallest gain you can show
18.8 %
You expect 20%.
Total duration
8 wk
Baseline, onboarding, pilot, decision.

What each size can show

The line marks the gain you expect (20%). A bar that crosses it: that size is not enough.

  1. 5 ppl59.3 %
  2. 8 ppl30.9 %
  3. 10 ppl25.4 %
  4. 12 ppl22 %
  5. 15 ppl18.8 %
  6. 20 ppl15.7 %
  7. 30 ppl12.3 %
  8. 40 ppl10.5 %
  9. 60 ppl8.4 %
  10. 80 ppl7.2 %
  11. 120 ppl5.9 %

Watch out for

  • A before/after on the same people is economical, but anything else that changes over the period (workload, seasonality, reorganisation) reads as an AI effect. Log those events.

The assumptions

Every figure above rests on these. The ones marked “assumption” have no external source: they are modelling choices, labelled so they can be argued with rather than mistaken for measurements.

Significance and power
5% two-sided · 80%Published
The usual conventions: a one-in-twenty chance of wrongly concluding there is a gain, four in five of seeing it if it exists.
Approximation
Normal + Guenther correctionPublished
The normal approximation slightly understates small samples; Guenther's (1981) correction adds the difference back, giving the textbook 64 per group for a standardised effect of 0.5.
Spread between people
30 %Assumption
Standard deviation of each person's average time, relative to the mean. Measure it in the baseline week and enter it here.
Spread from task to task
50 %Assumption
The same person takes more or less time depending on the task. This part shrinks as more tasks are measured.
Spread of the gain
50 %Assumption
Not everyone gains equally: some gain twice the average, others nothing. This part does not shrink by measuring more tasks.
Deliverable

The protocol to get approved

Pilot protocol

A pilot designed to decide, not to reassure.

Design
Before/after, same people
Participants
15
Duration
8 weeks
Your pilot can show a gain of 18.8 % or more

You expect 20%: if the gain is real, the measurements will show it.

1. Objective and hypothesis

The pilot must establish whether AI cuts the time needed for the use case by at least 20% — from 45 min to about 36 min per task — without degrading quality.

2. Design and size

The same 15 people are measured without AI, then with it. Each is their own comparison, which removes differences between people. To show a 20% gain you need 14 people; with 15, the smallest demonstrable gain is 18.8 %.

What each size can show
ParticipantsSmallest demonstrable gain
559.3 %
830.9 %
1025.4 %
1222 %
1518.8 %
2015.7 %
3012.3 %
4010.5 %
608.4 %
807.2 %
1205.9 %

3. Timeline

WeekPhaseWhat happens
W1Baseline measurementTime 10 tasks per participant, without AI.
W2Baseline measurementTime 10 tasks per participant, without AI.
W3OnboardingTraining, access, measurement instructions. No measurement this week.
W4PilotMeasure tasks with AI.
W5PilotMeasure tasks with AI.
W6Pilot — mid-pointCheck everyone is measuring; collect first feedback; change nothing in the protocol.
W7PilotMeasure tasks with AI.
W8Analysis and decisionCompare, apply the decision criteria fixed in advance, decide.

4. What is measured

MeasureHow to measure it
Time per taskTiming of 10 tasks per person per period, logged in a shared sheet.
QualityBlind review of a sample of deliverables (with and without AI mixed) by someone who does not know which is which, scored 1 to 5.
AdoptionShare of participants using the tool at least three days a week.
Perceived usefulnessOne question at the end of each week: "The tool saved me time this week" (1 to 5).
Errors and reworkNumber of deliverables sent back or corrected afterwards.

5. Decision criteria, fixed before starting

6. Biases to watch

  • Volunteering — volunteers are the most comfortable with the tool; their gains overstate the rest of the team's.
  • Novelty effect — early enthusiasm fades; hence at least three weeks of pilot.
  • Observer effect — knowing you are measured makes you faster, with or without AI; hence measuring during the baseline too.
  • Easy tasks — measuring only the tasks AI is good at overstates the gain; draw tasks at random from the real flow.
  • A before/after on the same people is economical, but anything else that changes over the period (workload, seasonality, reorganisation) reads as an AI effect. Log those events.

7. Sign-off

Pilot owner
Sponsor and decision date

Generated by i-avantage's “Design a pilot” tool, in your browser. The default spreads are orders of magnitude: replace them with those measured in the baseline week.

What the calculation does, what the AI does

Every figure on this page comes from a deterministic algorithm running in your browser: the same input always gives the same result, and nothing is sent to produce it. This tool uses no language model at all: every sentence shown is written into the rule that triggered it.

Where to go next

See all 19 tools →