Many favorable emails were received as a result of the article, "Answers to Four Common List Questions" (DM News, November 4, 2004). Therefore, it was decided to repeat the format with a focus on a hot topic: how to integrate statistics-based predictive models into a coordinated, multi-channel contact strategy. This is the second installment of a two-part series.
Question #3:
Two consultants have conferred with us about data mining issues. The first consultant insists that every predictive problem should be solved by: First, constructing a response model off everyone who was promoted (an "Upfront Response Model"). Second, building an order size model off everyone who responded (a "Contingent Order Size Model"). Third, generating an overall score for each prospect (or inquirer, or customer) by multiplying the two model scores together.
The second data mining consultant claims that we can be just as successful by building a single model to predict order size off everyone who was promoted (an "Upfront Order Size Model"). However, when we mentioned this approach to the first consultant, he was adamant that it was incorrect. The reason he gave had something to do with violating the theoretical underpinnings of regression.
We are confused. Which consultant is correct?
Answer:
From a narrow technical perspective, the first consultant is correct. This is because, for statistical reasons beyond the scope of this article, the scores from a single, Upfront Order Size Model will not be directly interpretable. In fact, they might be downright nonsensical, such as when they predict negative order sizes (that is, when the scores are negative).
However, from a broader business perspective, the second consultant is likely to be correct. Because of the way that predictive models are used by direct marketers, it generally does not matter that model scores are not directly interpretable. Or, that they might be downright nonsensical. The objective of a model is "or, at least, should be "to estimate a Return on Investment for every promotion. This can be done without ever interpreting, or even looking at, the actual model scores! Here is how:
First, score each prospect (or inquirer, or customer) for an upcoming promotion so that they can be rank-ordered from best to worst. Second, place these ranked prospects into pre-defined segments such as deciles. Third, review the financial performance of each segment during the model build,
and think about how your company's business environment has changed since that build. Fourth, based on what you learned in the third step, estimate the financial performance of each segment for the upcoming promotion. Fifth, compare the estimated financial performance of each segment with the cost of the upcoming promotion, and use this to determine whom to promote.
A single, Upfront Order Size model will generally be just as predictive as Upfront Response and Contingent Order Size Models whose scores have been multiplied together. A single, Upfront Order Size model can be constructed much more quickly, and put into production more efficiently. Also, with one model rather than two, the risk of implementation error is reduced. This is an important consideration because problems during implementation cause more model failures than missteps during the actual build.
However, there is one application for which a single, Upfront Order Size model cannot replace a two-model strategy, even when the single model is just as predictive. This is when the goal is to apply differential contact strategies to households who: a) rank equivalently when their two model scores are multiplied together, but b) whose Upfront Response Model scores are very different from each other, as are their Contingent Order Size Model scores. For example, assume that:
- Household A generates a high score on an Upfront Response Model, and a low score on a Contingent Order Size Model. This household is likely to respond, but if it does will probably have a low order size.
- Household B generates a low score on an Upfront Response Model, and a high score on a Contingent Order Size Model. This household is unlikely to respond, but if it does will probably have a high order size.
- The two model scores for Households A and B, when multiplied together, are identical.
These assumptions, coupled with non-differential contact strategies, will result in Households A and B receiving identical treatments. Nevertheless, the high-response/low-order size Household A is fundamentally different from the low-response/high-order size Household B!
Differential contact strategies can be pursued by first creating a 10 x 10 matrix of best-to-worst deciles, in terms of the Upfront Response and Contingent Order Size models. Then, the 100 cells within the 10 x 10 matrix can be aggregated into a workable number of "blocks." Finally, each block can receive its own contact strategy, based on predefined business rules. These business rules are developed from extensive data analysis, brainstorming sessions with Marketing and Creative, and the results of sophisticated longitudinal test panels.
In this way, Household A can be treated differently from Household B. For example, an offer to a high-response/low-order size block might be very different from one to a low-response/high-order size block.
As a final note, based on the specific dynamics of your business, both consultants might be missing the forest for the trees. A seminal question during the model planning stage should always be the ultimate business goal of the upcoming data mining project. The wrangling between the two consultants seems to be focused solely on predictions of response and order size. Is there something even better to be predicting, such as net dollars, or contribution? Or, maybe even profitability? However, that is another question, and perhaps another article.
