Your coupon generated orders. That is useful to know—but it does not tell you how many customers needed the coupon to buy. Some might have ordered anyway, at a price that left more money in the business.
A discount holdout test gives that question a fairer comparison. Instead of offering a promotion to everyone, randomly reserve a comparable group that does not receive it. Compare the contribution generated by both groups, not just coupon redemptions.
This is a practical next step after you understand how discounts affect contribution and required sales volume. The goal here is narrower: design a test that helps distinguish additional business from subsidized business.
What exactly are you trying to learn?
Write the decision before the email. For example: “Should we repeat a limited discount for existing customers who have not bought recently, if it increases contribution over the next purchase cycle?”
Define the eligible audience, products, offer, channel, and observation window. Decide what improvement would justify repeating the campaign, including the work required to run it. Discount strategy becomes easier to evaluate when the objective is specific.
Keep the conclusion equally specific. A test among past customers does not establish what a first-order offer will do for strangers. A clearance test does not establish a sustainable everyday product pricing policy.
How should you form the two groups?
Start with customers who are eligible to receive the communication. Randomly assign them before sending anything. One group receives the offer; the holdout does not.
Choose what the comparison means. If the holdout receives nothing, you are testing the combined effect of the message and offer. If both receive the same product message but only one receives a discount, the comparison more closely isolates the offer. Do not describe the first design as proof that the discount alone caused the difference.
Avoid putting your most loyal customers in the offer group and less active buyers in the holdout. Also avoid comparing coupon users with nonusers after the campaign: redeeming a coupon is a customer choice, not random assignment.
Keep other activity as similar as practical. A competing sitewide sale, overlapping email, or public coupon can contaminate the comparison. Record these exceptions rather than quietly removing inconvenient orders.
Google describes the underlying treatment-versus-control approach in its Conversion Lift guidance. That source concerns advertising experiments; this customer-offer worksheet is not a Google Ads product or a claim that your account has access to one.
What should you measure beyond redemptions?
Track all eligible purchases made by assigned customers during the agreed window, including orders without the coupon. Keep customers in their original groups even if they never open the email.
For each group, record audience size, orders, net sales after discounts and refunds, variable costs, and resulting contribution. Use customer identifiers consistently and avoid counting one order twice across channels.
Contribution margin is sales less relevant variable costs. Include materials, packaging, payment fees, variable fulfillment labor, and shipping support where applicable. Apply the same costing basis to both groups. Contribution is not net profit; fixed overhead still needs covering, as OpenStax’s managerial accounting explanation makes clear.
Subtract additional campaign costs separately if they are not already included. Do not deduct the discount twice: it already reduced net sales.
What does a worked example reveal?
Imagine a maker assigns 1,000 eligible customers to each group. This simplified hypothetical assumes one product per order and no refunds during the measured window. Actual tests should use realized order-level costs and later adjustments.
The holdout produces 80 full-price orders, each contributing $12. Total contribution is $960.
The offer group produces 120 orders, each contributing $8 after the discount and variable costs. Total contribution is also $960. An additional $60 in campaign setup expense, not included in those order costs, reduces the offer result to $900.
Orders rose 50%, but the observed incremental contribution after setup was negative $60. The promotion created more packing work without improving the measured financial result.
For equal-sized groups, the worksheet is:
Observed incremental contribution = offer-group contribution − holdout contribution − additional campaign costs.
If group sizes differ, first calculate holdout contribution per assigned customer, then multiply it by the offer-group audience size to form the comparison baseline. Raw group totals would be misleading.
With this example’s fixed $8 contribution per offer order, at least 128 orders would be needed to exceed the $960 holdout result plus $60 setup cost. That is an economic threshold—not evidence that a result at that level would be statistically reliable.
When is the test ready for a decision?
Save a short test record before launch:
- The eligible audience and random assignment method.
- Whether you are testing the discount alone or message plus offer.
- The offer terms, exclusions, and planned communication dates.
- The contribution calculation, additional costs, and decision threshold.
- The observation window and a later refund or repeat-purchase review date.
- Stock, service, and data-quality conditions that would invalidate the comparison.
Choose a window long enough to examine the behavior you care about. Customers buying ahead can make the sale week look strong and the following period weak. Follow both groups through the same dates, and treat a later review as a planned step rather than moving the finish line until the result looks better.
Do not call a small observed difference proof. Uncertainty depends on audience size, purchase frequency, and variation in order contribution. For consequential spending decisions, have an analyst assess uncertainty and the sample needed for the improvement you want to detect.
Frequently asked questions
What if my customer list is too small?
A small test can expose tracking problems and fulfillment costs, but may not distinguish genuine lift from chance. Keep the financial commitment small and label the result exploratory rather than a proven winner.
What if the coupon gets shared?
Record known cross-group use and check how much of the holdout accessed the offer. Customer-bound eligibility can reduce leakage when your tools support it. Do not remove affected customers after seeing the results and still call the groups randomized.
Can I use last month instead of a holdout?
You can use it as a directional baseline, but seasonality, availability, and customer mix may differ. Label it an observational comparison, not equivalent evidence from a concurrent randomized test.
Should I include future repeat purchases?
Yes, if repeat behavior is part of the objective and the window was defined in advance. Compare realized contribution in both groups; do not credit the offer with an assumed lifetime of future profit.
Before your next promotion
Draft the test record before writing the coupon email. Keep promotions bounded and measurable. Decide who will not receive the offer, how purchases will be counted, and what contribution improvement would justify repeating it. Healthy profit margins depend on what the extra demand leaves behind—not how busy the sale makes your packing table.
