Big ideas or baby steps?
Developing a testing strategy for direct mail programs.
by Mike Gray

A common question we hear from clients when we discuss direct mail test strategies is whether they should test something that is entirely new or test incremental changes to an existing control package. The answer depends on several variables, including how well the program meets it's goals, how often the program is mailed, the size of the test budget, how many good ideas are available, and the client's tolerance for risk.

Testing something entirely new is often compared to a baseball slugger's home run swing. It's a major effort in hopes of a big payoff. Big steps like an entirely new offer or a radically different mail piece are home run swings. But, just like the slugger swinging for the fences, the risk of striking out is greater.

Incremental testing means testing isolated variables to help a program evolve one step at a time. This could involve changing an outer envelope or revising the guarantee on a reply form, but not doing both in one package. That minimizes risk, but can limit upside potential.

The first step toward choosing the right strategy is a commitment to testing. You should never assume that your existing program is maximizing your ROI. Instead, assume that there is always room for improvement. Once you've committed to testing, you can evaluate the benefits that different strategies offer to decide which approach is right for your program.

How much testing is necessary? That depends on the program; a consumer program that mails millions of pieces a month offers a lot more potential for testing than a B2B lead generation program that mails 20,000 pieces a quarter. But, regardless of the size of your program, the key to improving results is to develop an effective test strategy.

So, what are the pluses and minuses associated with different strategies?

Big ideas: Swinging for the fences
Imagine you're a marketing manager saddled with a small but important direct mail lead generation program that isn't performing to expectations: You budgeted for $100 leads, but response rates are only half what you expected, so CPL is running closer to $200. The sales force isn't content with a reduced diet of leads, and the CFO isn't pleased with the idea of boosting spending to make up the difference. To get back on budget and keep everybody relatively happy, you need to double the number of leads the program generates. Sorry, but changing the teaser copy on the outer envelope probably isn't going to get you where you need to be.

Big results usually call for big ideas, so this is a case where you need to take your home run swing. Go ahead and change that teaser copy, but change the rest of the package, too. If your mailing just 20,000 a quarter, you don't have enough volume to isolate a new envelope, a new letter, and a new brochure in separate tests. Instead you're going to have to combine several changes and commit to one or two dramatically different packages.

Better yet, recognize that creative changes usually don't double response rates and focus on something that might: A new offer. Testing two new offers is much more likely to help you double response rates than any number of creative tests. Put simply, new offers are the real home-run swings. New creative usually produces dramatic results only when there's something fundamentally wrong with the control creative.

If you can afford another test version, you can also test the combination of the new offer and the new creative to measure the benefits of both changes. The downside to combining several changes in a single test version is that one change may mask the effect of another. For example, the new offer might boost response nicely, but the new graphics actually hurt response a bit, so the overall lift isn't all it could be.

This is where experience comes into play, An experienced direct marketing agency will usually be able to combine two variables into one test package with a high degree of certainty that both will have a positive impact, whereas someone who doesn't understand what drives response might drag down the benefit of one change with a fundamental mistake in another area.

A word of caution: Swinging for the fences implies that you want to replace your old control, but it doesn't mean throwing out the baby with the bath water. In all but the most extreme situations, you should mail the new packages against the old control and track the results carefully to measure the impact.

Baby steps: Evolution instead of revolution
Now consider a B2C program that mails 400,000 pieces a month. The program is meeting its goals, but there is an ongoing desire for improvement. This is the perfect opportunity to test lots of new ideas without the pressure to reinvent the whole approach.

Your program can evolve one step at a time in this scenario, so you can test each idea in isolation. When someone asks "Was it the new offer that boosted the response rate, or was it the envelope?" you'll know the answer because you tested them separately.

Testing variables in isolation also puts less emphasis on experience, because you can isolate factors that might confuse the results. It takes experience to recognize when the components of a package work together. Testing one element at a time—baby steps—substitutes learning for experience.

It's important to understand that testing small changes requires bigger test quantities. Why? Well, small changes are likely to produce relatively small differences in response rates and (without going into a discussion of statistics) measuring small differences with an acceptable degree of statistical reliability requires larger test groups.

One downside to larger test groups is that using more of your circulation for test versions can drive down your overall response if your tests don't work. And, as anybody with a background in testing knows, not every test works. Only you can answer how much risk you can take.

Testing variables in isolation can also slow the development of your program. It takes time to test a new letter vs. a new envelope vs. a new brochure, and then it takes even more time to test the winning components in a combined package. If you want to proceed faster, the solution is to test the combined package at the same time as the individual components. But your ability to do that comes down to your budget, the size of your program, and your willingness to risk more circulation on unproven test ideas.

A final word of caution regarding testing in baby steps: Direct mail doesn't always work like a cafeteria plan. There are times when components that work well together don't succeed in isolation. The clever new creative package might blow the doors off your control if you keep the pieces intact, but fail when you pick it apart piece by piece.

So, when it comes to testing big ideas versus testing baby steps, what's the right answer? Plenty of programs have been refined through incremental testing. And plenty of programs have been dramatically improved by testing something big. The first key is to commit to testing. And the second key is to choose an experienced partner who can help you develop a strategy that is customized to your unique requirements.

Michael Gray is Vice President/Creative Director for Direct Resources Group, a Seattle-based direct marketing agency. He has nearly twenty years of experience creating successful direct marketing programs for a variety of organizations. Contact mike@drg.com.