Incrementality Testing: Did Your Ads Really Work?

What is incrementality testing?

Incrementality testing is an experiment that measures how many conversions your ads caused. You split people (or regions, or time periods) into a group that can see your ads and a comparable control group that cannot, then compare the two. The difference is your incremental lift: the sales, installs or sign-ups that would not have happened without the campaign.

That is a different number from what your ad platforms report. Attribution tells you which ad gets credit for a conversion. Incrementality tells you whether the conversion needed the ad at all.

Why attribution is not causation

Attribution follows a rulebook: the last click within the window gets credit, or the last view if there was no click. What is attribution? explains those rules. None of them ask whether the user would have bought anyway. A loyal customer who searches your brand name, clicks your ad and buys is an attributed conversion, even if they would have typed your web address one second later.

Two well-known studies show how big that gap can be:

• eBay’s search experiments. Thomas Blake, Chris Nosko and Steven Tadelis ran large field experiments at eBay and found that returns from paid search were “a fraction of conventional non-experimental estimates.” Brand-keyword ads had no measurable short-term benefit, and most non-brand spend went to frequent users whose buying was not influenced by the ads (NBER working paper 20171, later published in Econometrica).
• Facebook’s field experiments. Brett Gordon, Florian Zettelmeyer and colleagues compared 15 randomized US ad experiments at Facebook with common observational methods. The observational methods “often fail to produce the same effects as the randomized experiments,” even with rich demographic and behavioral data (Marketing Science, 2019).

Attribution is still fast and good for daily optimization. But it answers “who gets credit?”, and budget decisions need “what did this spend change?”

Lift, incremental conversions, iCPA and iROAS

Every method below ends with the same numbers. When the groups differ in size, scale the control result up to the test group first.

Scaled control = control conversions x (test group size / control group size)

Incremental conversions = test conversions - scaled control
Lift % = incremental conversions / scaled control
Incremental share = incremental conversions / test conversions

iCPA = ad spend / incremental conversions
iROAS = incremental revenue / ad spend

These match how the platforms define them. Google’s About Conversion Lift help page defines incremental CPA as “the cost to acquire one incremental conversion” (spend divided by incremental conversions) and incremental ROAS as the additional revenue generated per dollar spent. Meta’s Lift Metrics Glossary calls the first one cost per incremental conversion and computes incremental conversions as test minus control, adjusted for group size.

Compare iROAS with your break-even ROAS, not with the attributed ROAS your dashboard shows. What is ROAS? covers the regular version, and the ROAS calculator works for the incremental one too if you enter incremental revenue.

Incrementality testing methods compared

User-level holdouts: PSA tests and ghost ads

The cleanest test is a randomized holdout. Before the campaign runs, a random share of the target audience is put in a control group and never shown your ads. Everyone else is eligible as usual. Because the groups were chosen at random, any difference in conversions is caused by the ads.

The tricky part is the comparison. Only some of the test group actually see an ad, and the people an ad system reaches are not a random slice. There are three ways to handle that:

• Intent-to-treat. Compare everyone assigned to test with everyone in control, exposed or not. Unbiased but noisy, since most people in both groups were never going to be reached.
• PSA tests. Show the control group a public service announcement in place of your ad and compare the two exposed groups. You pay for the PSA impressions, and the result breaks when the platform optimizes delivery, because it shows the PSA and your ad to different kinds of people.
• Ghost ads. The ad system records the control users who would have seen your ad, without showing it, and compares them with the test users who did. Garrett Johnson, Randall Lewis and Elmar Nubbemeyer introduced the method in the Journal of Marketing Research in 2017. Their working paper reports that, compared with intent-to-treat or PSA experiments, advertisers can measure lift just as precisely while spending at least an order of magnitude less, and that a retailer’s retargeting ads lifted website visits by 17% and purchases by 11%.

Platform conversion lift studies: Google and Meta

Google Ads Conversion Lift. Google describes Conversion Lift as an incrementality tool that splits your audience into groups and measures the conversions caused by people seeing your ads. Studies can be based on users (groups built from aggregated user attributes) or on geography (groups built from aggregated geographic data, which also supports offline data). Reports include incremental conversions, iCPA and incremental ROAS. The help page says Conversion Lift “isn’t available for all Google Ads accounts” and to contact your Google account representative. Checked on October 20, 2026; eligibility and campaign types change, so read the live page before planning.

Meta Conversion Lift. Meta’s best practices for Experiments say you can create a Conversion Lift or Brand Lift test in the Experiments tool, and that lift tests automatically create test and holdout groups: people who had the opportunity to see your ads and people intentionally withheld from them. Meta reports incremental conversions with a 90% credible interval, a conversion lift percent and a cost per incremental conversion. Meta also says your account representative is the best person to ask which tests and minimums apply to you. Checked on October 20, 2026.

Each platform study measures only that platform’s ads, against a control group that still sees your other channels. Treat it as the lift of one channel on top of everything else.

Geo experiments

A geo experiment splits regions instead of people. In Google’s paper Measuring Ad Effectiveness Using Geo Experiments, Jon Vaver and Jim Koehler describe randomly assigning non-overlapping geographic regions to a control or treatment condition and delivering the condition through geo-targeted ads.

Geo tests work where user-level splits cannot: TV, store sales, or partners without holdouts. They need many comparable regions, which is easier in a large country. Meta’s open-source GeoLift package (R, MIT license) builds a synthetic control from untreated regions and includes power calculators to help pick test markets.

On/off time tests

Pause a channel for a period and compare total conversions (organic included) with the periods around it. If conversions barely drop, the channel was mostly collecting credit for sales that happen anyway.

Time tests are the weakest design, because seasons, paydays and competitors change too. Make them stronger by switching on and off several times (for example, alternating weeks for six to eight weeks) and keeping every other channel and price steady.

Marketing mix modeling

Marketing mix modeling estimates each channel’s effect from your weekly or daily spend and sales history, including media you cannot track. It needs no tracking IDs. Two free, open-source options exist:

• Google Meridian, released under the Apache-2.0 license, is based on Bayesian causal inference and includes ways to calibrate the model with experiments and other prior information.
• Meta Robyn, under the MIT license, describes itself as an experimental, open-sourced MMM package from Meta Marketing Science. It is written in R, and its documentation describes calibrating the model with results from causal experiments such as lift tests.

MMM and testing work best together: measure a few channels with experiments, then feed the results into the model.

A worked example: attributed vs incremental

Illustrative example with sample figures only. A subscription app runs a four-week retargeting campaign with an 80/20 user holdout. Each conversion is a first paid month worth $45 in revenue.

1. Scaled control: 500 x (200,000 / 50,000) = 2,000 conversions the test group would have made without ads.
2. Incremental conversions: 2,400 - 2,000 = 400. Lift: 400 / 2,000 = 20%. Incremental share: 400 / 2,400, about 17%.
3. Attributed CPA: $12,000 / 1,800 = $6.67. iCPA: $12,000 / 400 = $30.
4. Attributed ROAS: 1,800 x $45 / $12,000 = 6.75. iROAS: 400 x $45 / $12,000 = 1.5.

The dashboard says the campaign returns $6.75 for every dollar. The test says it returns $1.50: the ads caused 400 conversions, fewer than a quarter of the 1,800 they were credited with. Whether 1.5 is good depends on your margin and on what a subscriber is worth over time, which CAC vs CPA vs LTV helps you work out.

Sample size and test length

A lift test is an A/B test with a holdout, so the same sample size rules apply. Evan Miller’s rule of thumb, explained in A/B testing ads on a small budget, gives the users needed in each group for a 50/50 split at the usual 5% significance and 80% power:

n = 16 x p x (1 - p) / d²

p = control conversion rate (0.01 for 1%)
d = smallest absolute lift you want to detect (0.002 for 1.0% to 1.2%)

For the example above, n = 16 x 0.01 x 0.99 / 0.000004, about 39,600 users per group, or 79,200 in total. Unequal splits cost more. With 80/20, you need about 1.56 times as many users in total as with 50/50 (roughly 124,000 here), and with 90/10, about 2.8 times as many (roughly 220,000). The small control group sets your precision, which is why holdout tests need large audiences.

Rules of thumb for length:

• Run whole weeks, at least two and often four, so weekdays and weekends are balanced.
• Add the time conversions take to arrive. If most purchases happen within 7 days of a click, keep measuring for 7 days after ads stop. What is a conversion window? covers this.
• Fix the end date before you start and do not stop because the result looks good on day five.

Common incrementality testing pitfalls

Filter fraud the same way in both groups, or fake conversions in the test group will look like lift. Ad fraud and bot traffic lists the warning signs.

Incrementality when you pay per result

With pay-per-result buying you pay only for actions your own tracking confirms, so you know exactly how many results you bought. The question becomes how many of those people would have arrived anyway, because each one is a payment for a result you would have had for free. Pay only for confirmed results explains the model itself.

The math is simple. If a share of the results you pay for would have happened anyway:

True cost per incremental result = payout / (1 - share that would have happened anyway)

Illustrative example: at a $2 payout, if 20% of confirmed sign-ups would have happened anyway, each incremental sign-up costs $2 / 0.8 = $2.50. If half would have happened anyway, it costs $4.

How to keep that share low and measure it:

• Count new users only. Your server decides which events it confirms, so only fire the postback for users who did not already have an account. Server-to-server tracking shows how the confirmation works.
• Pay for a deeper event. A finished tutorial or first purchase is less likely to be a returning customer passing through than a click. Offers with several steps, each with its own payout, are covered in multi-event offers, and CPI vs CPA for apps compares the pricing.
• Run an on/off or country test. Pause the offer in some markets or weeks and compare your total new users, organic included, with the markets or weeks where it ran.
• Compare cohorts. Check whether users from the channel retain and pay like your organic users. Incentivized traffic explained covers what to expect from rewarded users.

Users on a rewards page usually discover your offer there rather than searching for you by name, the opposite of the eBay brand-keyword case. That is still a hypothesis to test with your own numbers.

Frequently asked questions