Marketing Experiment Sample Size: How to Calculate It

The sample size of a marketing experiment is the number of users, orders or geographic units you need in each group before a measured difference can be told apart from noise. You get it from four inputs: the significance level, the statistical power, the variance of your outcome metric and the minimum detectable effect (MDE). Of the four, the MDE is the one that moves the number most, and it is the one marketers most often set without thinking about money.

This guide walks through the calculation, shows worked numbers, explains what to do when traffic is thin, and covers the two parts of experiment design that usually get skipped: conversion deduplication and a roadmap that turns single tests into a learning program.

TL;DR

• Sample size depends on four inputs: alpha (usually 0.05), power (usually 0.80), outcome variance and the MDE. Fix them before the test starts, not after.
• The MDE dominates cost. Halving it requires roughly four times the sample, so choose the smallest lift that would change a budget decision, not the smallest lift you can imagine.
• Low traffic does not make a test impossible. It forces a larger MDE, a longer run, a higher-frequency metric or a different design, such as geo experiments.
• Count each conversion once, using a stable identifier such as an order ID, or your variance and your lift will both be wrong.
• Record every result and decision in a roadmap so each test sharpens the next one.

Table of Contents

• What sample size means in a marketing experiment
• The four inputs of a sample size calculation
• Why the MDE drives the cost of a test
• A worked example
• Experiments for advertising are not ordinary A/B tests
• When traffic or conversions are limited
• Deduplicate conversions before you count them
• Build a test-and-learn roadmap
• Our perspective: powered tests are the cheapest way to avoid expensive mistakes
• Run properly sized experiments with Cassandra
• FAQ
• Sources

What sample size means in a marketing experiment

In a survey, sample size is the number of respondents. In an experiment, it is the number of independent units randomized into each condition. A unit might be a user, a household, a session, an email recipient or a region. The unit you randomize on is also the unit you analyze on, and defining metrics together with their randomization and analysis units is a core part of designing online controlled experiments.

This is why rules of thumb borrowed from survey research mislead. The familiar figure of roughly 385 respondents for 95% confidence and a five-point margin of error describes a poll of a general population. It says nothing about whether a 3% conversion rate moved to 3.3%. An experiment needs a power calculation built around the metric you will actually decide on.

The four inputs of a sample size calculation

A statistical design guide hosted by the National Library of Medicine lists the parameters a calculation requires: the significance level, the desired power, the variance or standard deviation of the outcome, and the smallest meaningful effect to detect. Here is what each means in marketing terms.

Significance and power

Alpha of 0.05 means you accept a 5% chance of a false positive. Power of 0.80 means a genuine effect of the size you specified will be detected four times out of five. Power matters more than most teams assume: a review of experimental research practice cites work reporting that many published experiments are underpowered, with researchers often facing a 40% chance of missing a genuine effect. A test that cannot find a real lift does not return "no effect". It returns "we do not know", and teams routinely read it as the former.

Variance

For a conversion rate, variance follows from the baseline rate itself. For revenue per user, it is typically much larger, because a few big orders dominate. That is why a revenue-based test needs far more units than a conversion-based test of the same relative lift. Pull variance from your own recent data, over a period without unusual promotions, rather than from a benchmark.

The minimum detectable effect

The MDE is the smallest true effect your design is able to detect at the chosen alpha and power. It is a design decision, not a prediction. The A/B testing sample-size guide on arXiv separates the target effect used to power a test from the smallest observed difference that will reach significance, and warns that a significant result should still correspond to an effect that matters to the business.

Why the MDE drives the cost of a test

The relationship between MDE and sample size is inverse and quadratic. According to Spotify's experimentation glossary, halving the MDE requires about four times the sample when power and alpha stay constant. Its example: detecting a 2% lift takes 10,000 users per group, a 1% lift about 40,000, and a 0.5% lift about 160,000, for the same metric variance and settings.

That curve explains most failed tests. A team hopes to detect a 3% improvement on a channel with modest volume, runs for two weeks, sees nothing and concludes the channel is neutral. In reality the test could never have seen a lift that small.

Setting the MDE from economics

The most defensible MDE comes from the decision, not the data. One statistics glossary frames it as the smallest improvement worth having after the cost of designing, building, shipping and maintaining the change. In media terms, ask:

1. What would a lift of X% be worth in incremental margin per month?
2. At what X does that margin exceed the cost of the media and the test itself?
3. Below that X, would you really move budget?

If the answer to the third question is "no", that X is your MDE. Anything smaller is statistically interesting and commercially irrelevant. A tiny effect can be significant in a very large sample yet practically meaningless, which is why practical significance belongs in the design, not as an afterthought.

A worked example

Take an ecommerce brand testing a new prospecting campaign, with the outcome defined as purchase conversion per user.

• Baseline conversion: 3.0%
• Alpha: 0.05 (two-sided), power: 0.80
• Target: a 10% relative lift, meaning 3.0% to 3.3%

For two groups and a conversion metric, a common normal-approximation shortcut is roughly 16 multiplied by p(1 - p), divided by the squared absolute difference, per group. With p = 0.03 and a difference of 0.003, that is 16 × 0.0291 / 0.000009, or about 51,700 users per group. Treat it as a planning estimate and confirm it in a power calculator such as the free G*Power program or your platform's own tool.

Now change one input. If the business only needs to see a 20% relative lift (3.0% to 3.6%), the difference doubles to 0.006 and the requirement falls to about 12,900 per group. The same quadratic logic Spotify describes shows up here: double the MDE, quarter the sample.

The figures use the shortcut above and assume a 3% baseline. Your baseline will differ, so recompute with your own numbers.

Turning users into duration

Divide the total sample (both groups) by the eligible users you can reach per day, then round up to whole weeks so every weekday appears the same number of times. In the 10% example, about 103,400 users in total, at 5,000 eligible users a day, means roughly 21 days. Do not stop early because the dashboard looks significant on day nine. Peeking inflates false positives, and the planned sample is what gives the test its stated error rates.

Experiments for advertising are not ordinary A/B tests

A landing page test randomizes visitors. An ad incrementality test must answer a harder question: what portion of the outcome was caused by the ads? Researchers studying this on a major social platform note that measuring incrementality often relies on lift studies, which differ from standard A/B tests. In practice that matters for sample size in three ways:

• Exposure is partial. Only some people in the treatment group actually see the ad, which dilutes the measurable effect. Size the test for the diluted effect, not the effect on exposed users.
• Outcomes are rare. Purchases among a broad audience are a small fraction, so variance is high relative to the lift.
• Randomization may need to be geographic. When user-level holdouts are not possible, regions become the units, and your sample size becomes the number of markets and weeks, not the number of people.

Our guides on incrementality testing and the incremental lift formula cover the measurement logic in more depth. Online experiments also commonly express the treatment effect as a relative percent lift, which is easier to interpret and more stable, and that is the form used in the tables above.

When traffic or conversions are limited

Small audiences are the normal condition for B2B brands, nonprofits, new markets and niche products. The arithmetic does not bend, but you have several honest options. These are practical design levers that follow from the formula above, not guarantees.

1. Raise the MDE. Accept that you can only detect large effects. Test bold changes, such as a new offer or a doubled budget, rather than incremental tweaks.
2. Move up the funnel. A metric with a higher event rate, such as add-to-cart or qualified lead, has lower relative variance than a rare purchase. Confirm the proxy actually predicts the business outcome before relying on it.
3. Lengthen the test. More weeks add observations, with diminishing returns once seasonality or ad fatigue enters.
4. Reduce variance. Cap extreme order values, or compare against pre-test behavior of the same units, so noise stops hiding the lift.
5. Change the unit. If users are too few, switch to geographic units and a design suited to a small number of markets.
6. Accept a looser threshold knowingly. Raising alpha from 0.05 to 0.10 reduces the required sample, at the price of more false positives. Document the choice and the cost of a wrong call before you start.
7. Use the test for decisions, not proof. An underpowered test can still inform a reversible decision if you report the confidence interval honestly instead of a binary verdict.

The audience you define also changes the pool. Careful audience targeting shrinks or enlarges the eligible population, so settle it before you calculate, not after.

Deduplicate conversions before you count them

Sample size calculations assume each conversion is counted once. In real stacks, the same purchase is often reported by a browser pixel and a server-side feed, or by several ad platforms at once. Duplicates inflate both the baseline and the apparent lift, and they can differ between the test and control cells, which biases the result itself.

The fix is a shared key. The guidance in Snap's developer documentation on deduplication is to send the same identifier for the same conversion across integrations, such as an order ID, and to use a unique ID for each distinct conversion, so the event is counted once. Apply the principle everywhere you ingest conversions:

• Use the order ID or lead ID generated by your own system, never a value created independently by each tool.
• Keep one ID per real-world event, and a different ID for a second purchase by the same customer.
• Deduplicate before the experiment readout, then compare totals with your backend source of truth.
• Check that deduplication behaves the same in the treatment and control groups.

Build a test-and-learn roadmap

A single well-powered test is useful. A sequence of them, each informed by the last, changes how a team allocates budget. The American Marketing Association's experiment planning workbook proposes prioritizing ideas first, then defining for each the goal, audience, hypothesis, design, measurement approach, timing, risks and decision rule, and finally recording results, learning, decisions and next steps.

A workable roadmap has these columns:

Prioritize by expected value and feasibility: a high-spend channel where a modest lift is worth real money, and where the sample size is attainable, goes first. Tests you cannot power go to the bottom, or they get redesigned. For the ongoing version of this idea, see how a standing incrementality testing program is structured, and how to test without hurting live performance.

Our perspective: powered tests are the cheapest way to avoid expensive mistakes

We see the same pattern across ecommerce, fintech and nonprofit teams. An AI agent, an automated bidding system or a new creative can act quickly, but speed is not proof that anything it did earned money. The only way to know is a comparison against what would have happened without the action, and that comparison is only trustworthy when the test was sized for a decision.

Our view is that the MDE should be written in the language of risk and return. State the lift that justifies moving budget, state the confidence interval you require to move it, and refuse to read tests that were never able to reach it. A well-run program also stops treating each test as a one-off: results feed an ongoing view of marginal return by channel, so the next budget decision starts from measured incrementality rather than platform-reported credit.

Most teams do not need more tests. They need fewer tests, sized properly, with decision rules agreed in advance.

— Gabriele Franco

Run properly sized experiments with Cassandra

Cassandra is a marketing measurement platform that combines incrementality testing, geo-experimentation and marketing mix modeling, with always-on incrementality for continuous reads between tests. Plans start at €3,000 per month for Essentials, €3,750 per month for Complete and €5,400 per month for Scale, with Enterprise available on request; see the pricing page for what each tier includes.

FAQ

How do you determine the sample size for a marketing experiment?

Specify the significance level, the power, the variance of your outcome metric and the minimum detectable effect, then enter them in a power calculator. Alpha of 0.05 and power of 0.80 are the usual defaults. The MDE should come from the smallest lift that would change a budget decision.

What is a good MDE for marketing tests?

There is no universal figure. Set it from economics: the smallest lift whose incremental margin exceeds the cost of media, testing and change. Remember that halving the MDE needs about four times the sample, so an ambitious MDE is often unaffordable for smaller brands.

Can I run a valid test with a small audience?

Yes, if you design for it. Test larger changes, use a higher-frequency metric, run longer, reduce variance or switch to geographic units. Report the confidence interval rather than a simple win or loss, and treat the outcome as input to a reversible decision.

Why do duplicate conversions matter for sample size?

Duplicates inflate event counts and change the measured variance, so the test is sized and read against the wrong numbers. Use a shared unique identifier, such as an order ID, across every integration so each conversion is counted once.

Sources

• Focus on Data: Statistical Design of Experiments and Sample Size Selection (PMC)
• Best practices for implementing experimental research (UTRGV)
• What is a Minimum Detectable Effect (MDE)? (Spotify Confidence)
• All about Sample-Size Calculations for A/B Testing (arXiv)
• What Is Minimum Detectable Effect (MDE)? (Bell Statistics)
• Power, MDE, Sample Size, and Practical Significance
• Statistical Challenges in Online Controlled Experiments (The American Statistician)
• Designing Experiments to Measure Incrementality on Facebook (arXiv)
• Deduplication (Snap for Developers)
• AMA Marketing Experiment Planning Workbook

Recommended

• Marketing Experiment Design: Framework, KPIs and Scaling