A/B Test Sample Size Calculator

Where it is used

• Small-business owners: A shop owner about to test two versions of a feedback card checks how many returned cards each version needs before the results can be trusted.
• Marketing and customer-service staff: A team testing two versions of a survey email works out how many recipients each version needs to see which one gets more replies.
• Students and researchers: A student planning a study that compares the sign-up rate of two different posters uses it to decide how many people each group must include.

Power and confidence

Power is the chance of detecting a real difference when one exists (80% is typical). Confidence controls false positives. Smaller differences between the two rates require much larger samples to detect reliably.

When should you use it?

Use this before you run any test that compares two versions of something, where each person either does the thing or does not, such as returning a form or clicking a link. It works out how many people you need in each group before you start, so your test is big enough to trust. Reach for it when you are about to compare two feedback cards, two email versions, two sign-up pages, or two reminder methods, and you want to know whether a real difference exists between them.

What does the result mean?

The number shown is the sample size per group. That means you need about that many responses in each version, so double it to get the grand total. If you collect fewer than this, a real difference between the two versions may be too small to spot, and you could draw the wrong conclusion. The common settings are 95 percent confidence and 80 percent power, which most researchers accept as the standard choice. Smaller gaps between your two rates always need much larger groups to prove.

Mistakes to avoid

Do not leave the two rates equal; if both are the same there is nothing to detect and no answer appears. Enter each rate as a plain number, not with a percent sign. Remember the result is per group, not the total, so you need it twice over. Do not stop your test early just because one version looks ahead; wait until each group reaches the planned size. And do not guess wildly at your expected rate, base it on past results whenever you can.

How to use this calculator

1. In the first box, type your baseline rate as a plain number, which is the percentage your current version gets now.
2. In the second box, type the expected rate you hope the new version will reach.
3. Pick a confidence level and a statistical power from the drop-down menus; 95 percent and 80 percent are the usual choices.
4. Read the Sample size per group number, which is how many responses to collect in each version, and double it for the total.

Worked example

A cafe gets feedback cards returned by about 20 out of every 100 diners. They design a friendlier card and hope 25 out of 100 will return it. They type 20 in the first box and 25 in the second, and leave confidence at 95 percent and power at 80 percent. The calculator shows about 1,094 per group, so they need roughly 1,094 returned cards for each version, about 2,188 in total, before they can trust the difference.

Frequently asked questions

Related calculators

• Sample Size Calculator
• T-Test Calculator (Two Samples)
• P-Value Calculator