Plan the test, then split the list
Three panels: the sample size you need, the smallest lift a list you already own can prove, and a reproducible mail / holdout split as two CSV files.
1. How many pieces do I need to mail?
2. Already have a list? What can it prove?
3. Split the list into mail.csv and holdout.csv
Runs entirely in your browser. Nothing you enter is sent anywhere.
What is a holdout test for direct mail and why does it matter?
A holdout turns a mailing into an experiment instead of a story about a good month.
A holdout test mails most of your list and deliberately withholds a random slice. The withheld addresses get no postcard, no letter and no call. When responses come in, you compare the mailed group's response rate with the holdout's using a two-proportion test, and the difference is the lift the mail actually caused.
That control group is the only thing between a measurement and an anecdote. Lists change, seasons change, offers change and tracking changes. A concurrent holdout lives through the same week, the same local market and the same counting rules as the mailed group, so the comparison stays fair.
With a holdout
- One honest number. Response rate in the mailed group against the holdout, each with a 95% Wilson interval. selftest: mail.response
- A test, not a vibe. mail_compare runs the two-proportion test and returns a difference interval, so you can see whether the gap survives the noise. selftest: mail.response
- Honest at small numbers. mail_response returns a verdict such as "too early" or "no signal yet: could still be up to X%" instead of a misleading point estimate. selftest: mail.response
Without one
- No counterfactual. Every response gets credited to the postcard, including the people who would have called anyway.
- Seasonal sleight of hand. A strong month looks like a winning campaign and a weak month looks like a failure.
- Nothing to defend. When someone asks how you know the mail worked, the truthful answer is that you don't.
The planner on this page is the same maths the MCP tool uses: mail_holdout_plan returns a sample size for a no-mail holdout at 95% confidence and 80% power, within 3% of the textbook formula. selftest: mail.response Nothing in the direct-mail tools sends mail, buys postage or contacts anyone.
How big does my holdout need to be?
The holdout is a measuring instrument. Too thin and every result looks like noise.
Your base rate
A 1% response rate is a rarer event than 10%. Rare events need more pieces before a difference stands out from ordinary variation.
The lift you want to detect
Sample size grows with the inverse square of the lift, so detecting half the lift costs roughly four times the pieces.
Confidence and power
95% confidence with 80% power is the default pair, matching mail_holdout_plan. Moving to 99% or to 90% power buys certainty with more mail. selftest: mail.response
Mailed to holdout ratio
Equal groups need the fewest total pieces for a given power. A 4:1 ratio keeps more pieces selling but needs a bigger list overall.
Panel 1 answers the forward question: how many pieces to mail and how many to hold back. Panel 2 answers the question most people actually have. If the list already exists, the useful question is not "how many do I need" but "what is the smallest lift this list can prove", and the answer is often larger than the improvement anyone expects.
That is worth knowing before you print. A list of a few thousand addresses with a 10% holdout can only detect a large relative lift; if a modest improvement would already change your decision, the honest move is to mail more addresses or accept that the test will only catch a big win.
Why can't I just compare to last year?
Because last year is not a random sample of this year.
Last year's numbers came from a different list, a different offer and often a different tracking setup, and none of those differences are random. A year-over-year jump can come from a better list source, a price change, a new landing page, a competitor leaving the market or a change in how scans are counted. Comparing two periods gives you a number, not a cause.
The tracking change is not hypothetical. mail_scans counts unique human scans per code from your own web-server logs and drops bots, link previewers from Slack, Facebook and WhatsApp, your own IP ranges, unknown codes and any hit before the piece was mailed. selftest: mail.scans If a previewer starts fetching your links, a scan count can move with no change in human behaviour at all. A concurrent holdout is immune, because both groups are counted the same way in the same week.
There is an attribution problem too. channel_cac assigns credit in a fixed order: tracking code first, then UTM tags, then what the lead said, then the referrer. selftest: mail.cac Without a holdout, every response in that chain gets credited to the mail by default, and the budget follows the credit.
What lift is realistic to test for?
No calculator knows your offer. The defaults are arithmetic, not a forecast.
No page can tell you what lift your postcard will produce; it depends on the list, the offer, the price, the creative and the season. The 1.0% response rate and 50% relative lift pre-filled in panel 1 are starting points for the arithmetic, nothing more.
If you have mailed before, use your own history. mail_response reports response rates by any column with 95% Wilson intervals and a verdict that says "too early" or "no signal yet: could still be up to X%" rather than a misleading point estimate. selftest: mail.response Take the top of that interval as your base rate when planning the next drop; it keeps you honest about how little a small batch proves.
If you have no history, start from break-even instead of an industry average. mail_breakeven takes your cost per piece and returns the response rate you need to cover print and postage for a given lead-to-client rate and client value. selftest: mail.cac Then plan a test that can detect a lift above that floor. A test that can only detect a doubling tells you nothing about a 20% decision you would actually act on.
Size the test to the decision. If a 20% improvement would change what you do next quarter, that is the number to type into the lift box, and the planner will tell you what it costs in pieces.
Worked example: a 1% base rate and a 50% lift
Type these values into panel 1 and read the four results.
- Response rate 1.0%. The rate you expect in the mailed group if the mail does nothing unusual.
- Lift 50%. A relative lift, so the mailed group has to reach 1.5% against a 1.0% holdout before the difference clears the noise.
- Confidence 95%, power 80%, ratio 1. Equal-sized groups, the cheapest way to buy a given amount of power in total pieces.
- Read the metrics. Pieces to mail, holdout size, total addresses and expected responses in each group. With a 1:1 ratio the holdout comes out the same size as the mailed group, so the total is about twice the mailed count.
- Compare against your budget. If the total is far beyond what you can print, raise the lift you are testing for, or accept a smaller pilot that can only detect a bigger win.
The calculator prints the counts; this page does not invent them. Enter the numbers and the four metrics are yours.
After the drop, compare the two response counts with a two-proportion test. mail_compare returns the difference and a confidence interval on it, built on the same Wilson intervals as mail_response. selftest: mail.response If the interval still spans zero, the honest verdict is that the test has not finished yet.
What can this calculator not tell you?
It sizes a test. It does not predict a result, price a campaign or clean a list.
What it does
- Sizes the test. Mailed and holdout counts for your base rate, lift, confidence and power.
- Answers the reverse question. The smallest relative lift a list you already own can prove.
- Splits reproducibly. A seeded random assignment, so re-running with the same seed and list gives the same two files.
- Stays offline. The file is read in the browser and the CSVs are generated locally.
What it can't
- It can't supply the response rate. You bring the base rate; the calculator only does the arithmetic.
- It assumes random assignment. A hand-picked "holdout" of awkward addresses is not a control group.
- It is an approximation. Sizes are within 3% of the textbook formula, so plan from them rather than over-reading them. selftest: mail.response
- It doesn't do unit economics. Use mail_breakeven for cost per piece and the break-even response rate. selftest: mail.cac
- It doesn't clean the list. mail_list_clean drops foreign, incomplete and institutional addresses and warns when the client suppression list is empty. selftest: mail.hygiene Split a dirty list and you pay for both mistakes.
Clean first, then split, then print. The direct-mail rails were generalised from a live postcard program run by a Tampa property manager in 2026, where the failures were practical ones: truncated labels, a stale suppression list, and test scans counted as real responses.
Can an agent run the same holdout plan?
Yes — mail_holdout_plan is a tool in the same family, and it runs on your machine.
AdRails is an open-source (Apache-2.0) MCP server that runs on your machine with your credentials. mail_holdout_plan takes a base response rate and the lift you want to detect and returns the sample size for a no-mail holdout at 95% confidence and 80% power. Pair it with mail_codes_assign for tracking codes, mail_labels for print-exact sheets, mail_scans for scans that ignore bots and previewers, and mail_response to read the result with an interval rather than a point estimate.
See the whole category on the direct mail tools page, browse the index on all tools, or read the source on GitHub. The repository ships 82 tools and a selftest suite of 2,704 checks, all passing on 2026-11-06.
Straight answers.
What is a holdout in a direct mail test?
A holdout is a random slice of your mailing list that receives nothing. You mail everyone else, then compare the response rate in the mailed group against the response rate in the holdout using a two-proportion test. The difference is the lift the mail caused, rather than the lift the season, the list change or the offer happened to produce.
How big should my direct mail holdout be?
Big enough that a two-proportion test can separate your expected lift from noise. The planner takes your base response rate, the relative lift you want to detect, confidence (90, 95 or 99%) and power (80 or 90%) and returns the mailed and holdout counts. Equal groups need the fewest total pieces; a smaller holdout keeps more pieces working but needs a bigger list overall.
Why can't I compare this mailing to last year?
Last year is not a random sample of this year. List sources, cleaning, pricing, offers, creative and tracking all change between runs, and a sequential comparison mixes those changes into a single number. A concurrent holdout shares the same week, market and tracking as the mailed group, which is the only clean way to attribute a difference to the mail.
What response rate should I put in?
Use your own history if you have it: mail_response reports response rates by column with 95% Wilson intervals and refuses to over-read small samples. With no history, the 1% default is a starting point, not a forecast, and mail_breakeven is a better anchor because it returns the response rate you need to cover print and postage for a given client value.
Does my CSV leave my machine?
No. The file is read by your browser through the FileReader API, the split runs locally, and the two CSVs are generated as in-memory blobs. Nothing is uploaded anywhere. The same local-first rule applies to the MCP tools: AdRails runs on your machine with your credentials, and nothing in the direct-mail tools sends mail or contacts anyone.
What do 95% confidence and 80% power mean here?
Confidence is how often the test avoids a false alarm: at 95%, a difference flagged as real would be a fluke about one time in twenty if there were no effect. Power is the chance of catching a real lift of the size you specified. Raising either setting increases the number of pieces you need, so both are trade-offs against your print budget.