How to test incrementality before peak spending starts
Waiting until Cyber Week is too late. Run holdouts early enough to see which channels create lift, then move budget before the expensive days arrive.
By Kevin Dugan
VP of Analytics & Marketing Intelligence
Published on 2026-10-30
Share this article
The channels reporting the most revenue during peak season aren’t necessarily driving the most sales. Holdout tests help you identify which channels generate incremental revenue, so you can make better budget decisions before peak spending begins.
Cyber Week 2025 resulted in a record-breaking 44.2 billion in online spending, according to Adobe. For the brands we work with, Cyber 5 (the five days spanning from Thanksgiving Day through Cyber Monday) generated 21.5% of total November and December revenue. With so much money moving in just a few days, over-crediting a channel is costly.
Nielsen’s research found that 84% of teams say they can prove ROI, but only 38% look at every channel. That discrepancy leaves room for over-crediting. Plan holdout tests before peak season to get results you can act on.
What a holdout test measures
A holdout test is a controlled experiment. You choose one channel, such as a coupon partner or a content partner, and switch it off for part of your audience. Everybody else remains exposed to it. The difference between these two audiences during the same period is the incremental revenue generated by the channel.
Incrementality means the portion of a sale that wouldn’t occur without the marketing touchpoint in question. If you switch off the coupon partner for part of your audience and sales don’t move much, then most of the revenue attributed to it would’ve been earned anyway.
Choose a design that fits the channel
Geo holdouts, where you pause a channel in matched regions, are the most common choice for affiliate. Audience or code holdouts, where you suppress a coupon or link for a random percentage of a customer list, are useful for closed-loop programs.
Coupon and loyalty partners are often over-credited at last click, so run coupon holdouts. Creator marketing often takes place off-pixel, such as on TikTok Shop and Amazon, so plan code or geo tests.
If you can, hold out 20% to 30% of traffic, aim for 200 or more conversions a week, and run the test for four to eight weeks. Even shorter tests can generate useful data.
If a full holdout is too expensive, start smaller by holding out a smaller segment of traffic or testing one channel at a time. Even a partial test can show whether a channel gets too much credit.
Keep the control group clean
Switch off links, block coupon extensions through network controls, and inform key publishers about your tests. Watch for anything that could contaminate your control group, such as competitors’ flash sales or your own promotions, and log it.
One sign that your control group is compromised: direct traffic grows in paused markets while brand search stays flat.
Compare exposed audiences to the holdout audience, then the previous period, and then the previous year’s result. Move from ROAS to incremental ROAS (iROAS): test revenue minus control revenue, divided by spend, with the groups scaled to equal size.
Say coupon partners report 21x ROAS and content partners report 8x. Ordered by ROAS, the content partners are pruned first. Ordered by the number of new customers acquired, they are pruned last. Content and PR land early in the path to purchase, so a short last-click window gives them little credit.
Run tests and lock down tracking before peak
A holdout run only during peak volatility produces less actionable data, so start early. Cyber 5 runs from November 26 to 30. A four-week test starting the week of October 26 ends around November 23, just before peak.
Audit all links, UTMs, and pixels first, since you can’t fix broken UTMs once a test is running. Lock down your tracking from November 1 through the holiday season, as changes to any tag during the test can compromise your data. Log every promotion, email send, and price change, and establish a single source of truth.
Check results and pull-forward after peak
Pull your data the week after Cyber 5, not in January, and expect some reversals to trickle in. Don’t declare peak a success until you’ve checked for pull-forward, meaning buyers who would have purchased later anyway.
Discounted volume from existing customers is revenue timing, not growth. Then keep testing, since results can drift from one season to the next.
Turn findings into decisions
Every finding needs an owner, a budget line, or a test. Otherwise, it’s a retro deck sitting in a drive. Put your spend where you see proven lift, rather than on the greatest last-click numbers, and consider commission structures that reward new customer acquisition.
During peak, the channels that report the highest numbers win the budget. The brands that succeed know which number is true.