A/B Testing: Definition, Bedeutung & Beispiele im Direktmarketing

What is A/B Testing? — A Clear Definition

A/B testing is a data-driven testing method in which two variants of a marketing piece are sent to comparable, randomly split recipient groups. Variant A is the "control" (the proven original), while variant B is the "variation" (the modified version with exactly one change). By comparing results — response rate, conversion rate, or ROAS — you can determine on a data basis which variant performs better. The key principle: only a single variable is changed, so that the effect of the change can be causally attributed.

The method has deep roots. British statistician Ronald A. Fisher developed the foundations of modern experimental design in the 1920s at the Rothamsted Experimental Station — including randomization, analysis of variance, and the now-standard 5 percent significance level. Almost simultaneously, American advertising pioneer Claude Hopkins described systematic split testing in advertising for the first time in his book "Scientific Advertising" (1923): he sent different versions of direct mailings to comparable groups and evaluated responses through coded coupon codes. Hopkins' principle holds true to this day: No campaign without a test.

In direct marketing, A/B testing has been the standard method for campaign optimization since the 1960s. The annual CMC Print Mailing Study by Deutsche Post — the largest benchmark study for direct mail in the DACH region (Germany, Austria, Switzerland) — is fundamentally based on A/B tests: every year, over 40 online retailers test different campaign variants on more than one million recipients to gain measurable insights about coupons, formats, response boosters, and design.

What Can Be Tested in Direct Mail?

The range of testable elements in direct mail is broad — and not all of them have the same impact on results. The biggest levers are the target audience (which recipients are contacted) and the offer (type and value of the coupon or discount). Only then do format, design, and details such as envelope design or response boosters follow. Those with limited test budgets should therefore start with the most impactful variables.

The CMC studies confirm this prioritization impressively. The CMC Study 2024 tested five different coupon mechanics and found massive differences: the unlimited coupon achieved the highest long-term CVR of 5.4 percent, while time-limited and quantity-limited variants were stronger in the short term (up to +58 percent in the first week) but dropped significantly to just 3.6 percent CVR over the full observation period. By contrast, the CMC Study 2025 showed that envelope design in the B2C segment had no significant impact on conversion — all four tested variants performed nearly identically.

This insight is critical for test planning: you should focus your testing resources on the variables that have the largest expected effect — rather than on design details that produce barely measurable differences. The CMC Study 2022 underscores this: percentage-based coupons achieved 23 percent higher CVR than fixed-amount coupons, and high-value coupons outperformed low-value ones by 61 percent. Such differences in the offer are many times larger than design variations.

Real-World Examples from the CMC Studies

The CMC Print Mailing Studies deliver the largest A/B tests in direct mail in the German-speaking region year after year. Since 2018, over 40 online retailers have sent more than one million print mailings to existing customers annually, with each edition focusing on a specific A/B test topic. The results set the direction for the entire industry.

The CMC Study 2023 tested four response boosters against each other: an enclosed coupon card, a printed coupon code, a QR code, and a scratch field. The result was clear-cut: the physical coupon card achieved a 6.1 percent CVR — the best value — 33 percent more than the scratch field (4.6 percent). The printed coupon code came in close behind at 5.5 percent. Overall, roughly 20 percent of mailing recipients visited the advertised online shop, with over 70 percent of visits occurring within the first two weeks. The 2020 study tested formats: the classic advertising letter outperformed maxi postcards and self-mailers by approximately 25 percent in CVR and achieved 42 percent higher ROAS than the self-mailer.

A particularly revealing result came from the CMC Study 2024 on coupon limitation: time-limited and quantity-limited coupons drove short-term responses massively upward (+38 percent for a 3-week deadline, +58 percent for quantity limitation) but lost out to the unlimited coupon in the long run. For short-term sales promotions, limitations are therefore effective, but for long-term existing customer reactivation they are counterproductive. This result would not have been detectable without systematic A/B testing — gut feeling would likely have favored the limited variant.

Methodology — How an A/B Test Works in Direct Mail

A methodologically sound A/B test in direct mail follows a clear process. It begins with the hypothesis: "A 15% coupon with a 3-week deadline achieves a higher CVR than one without a deadline." Next, the recipient list is randomly divided into two equally sized groups — the control group receives the proven mailing, the test group receives the variant with exactly one change. All other factors (timing, target audience, remaining design) stay identical. Both variants are sent on the same day.

The critical question is the sample size: each test variant needs enough recipients to deliver statistically reliable results. The rule of thumb is: the sample should be large enough to generate at least 100 responses per variant. At an expected response rate of 2 percent, that means at least 5,000 recipients per variant — at 1 percent, at least 10,000. The target confidence level is typically 95 percent (p < 0.05), meaning the probability of a chance result is below 5 percent.

Particularly important — and different from digital tests — is the observation window: the CMC studies show that with print mailings, roughly half of all orders only come in from week five onward after dispatch. Anyone who evaluates their test after two weeks captures only a fraction of the effect and risks making wrong decisions. A minimum observation period of six to eight weeks after dispatch is recommended. Only then can the winning variant be identified and scaled up as the new "control" — the next test then tests a new element against this new benchmark.

Common Mistakes in A/B Testing

Even experienced marketers make methodological errors that can invalidate A/B tests. The most common mistake is changing multiple variables at once — for example, altering coupon value, headline, and design in a single variant. In that case, it is no longer possible to determine which change influenced the result. The "One Variable at a Time" (OVAT) principle is therefore the most important methodological rule.

An equally widespread mistake is a sample size that is too small. Testing with only 500 recipients per variant at a 2 percent response rate yields just 10 responses — far too few for statistically significant conclusions. The observed difference could be pure chance. Premature evaluation is another typical error: with print mailings, half of all orders come in only after weeks. Deciding after two weeks may lead to the wrong choice. Other common mistakes include unequal test groups (not randomly split), missing tracking (no individual codes), and adopting results without significance testing.

A/B Testing with AutoLetter

AutoLetter makes A/B testing in direct mail simple and accessible. Instead of coordinating elaborate test designs with traditional mail houses, businesses can configure their campaign variants online and have them sent to randomly split recipient groups — with integrated response tracking for clean performance measurement. This enables data-driven validation of hypotheses about offers, design, or personalization, without having to manually coordinate the testing process.

Especially for businesses that want to systematically optimize their direct mail, AutoLetter offers a decisive advantage: shorter cycles between test and rollout, transparent cost overview per variant, and measurable results as the foundation for the next campaign. Every test makes the following campaign better — fully in line with Claude Hopkins' principle: No campaign without a test.