Customer data mining is a complex process that involves highly-trained professionals. Some companies handle data mining in-house, others farm it out, and still others follow a hybrid solution. Which is right for you? The following are some factors to consider:
How Big a Data Mining Department Can You Afford?
Most mid-to-large direct marketers have an in-house data mining department to handle at least some of the analytics. They feel it is important to have total control of this critical function, and for the data miners to be continuously steeped in the business. Also, the cost of an in-house staff can be lower than an outsourced solution.
However, an in-house solution can be problematic for smaller direct marketers because employee turnover is an unfortunate fact of life in our business. Having your own group is risky if you are not large enough to support overlapping personnel.
Many smaller firms have been badly hurt when a key data miner has moved onto greener pastures. Generally when this occurs, much or all "company memory" is lost forever. Therefore, when deciding whether to do customer data mining in-house, you should reflect on how much staff you can realistically support.
The risk of employee turnover is particularly acute for firms that can only afford a single in-house data miner. Often, companies whose annual revenues are no more than "say "$20 million a year cannot support multiple personnel. By definition, analytical efforts will come to a grinding halt with the defection of a one-person staff. For example:
Recently, a well-known retail/catalog/ecommerce marketer lost its lone in-house data miner. Currently, it is looking for a replacement. In the meantime, the firm is in a tenuous situation where predictive models are embedded in the production processes, driving decisions on whom to promote, but there is no statistician to interpret what is going on. This is no way to run a direct marketing business!
Important Intangibles to Consider in Making Your Decision
Regardless of whether you go with an in-house or outsourced solution, make sure that the group you work with maintains a collegial atmosphere with a clear path for career advancement. These can be
missing in one and two-person departments. Without collegiality and the possibility of significant advancement, staffers are more likely to start interviewing elsewhere.
Also, stagnation can be a problem with small groups. Thinking can become in-bred. Such environments encourage a perverse sort of self-selection, where those with the best and most creative analytical minds grow restive and start looking for other employment. This is less of a problem with
the larger in-house departments and the best of the independent data mining companies, where a steady stream of diverse projects keeps advancing the collective knowledge.
Are You Looking for Ongoing Data Mining Support or One-Off Work?
Consistently-excellent target marketing requires ongoing data mining support. However, some firms are not willing or able to make that commitment. Instead, they are interested in pursuing one-off assignments. Under such circumstances, the only logical choice is to outsource the work.
When farming out your data mining, beware that there is significant overhead the first time a vendor works with your customer file. This is because, in order to do an effective analytical job, it is important to totally understand your business and data. Besides the specifics of the assignment, the first project is always more expensive than subsequent assignments. For example:
A major publisher with millions of subscribers across multiple titles decided to incorporate sophisticated data-driven marketing into its core business practices. As part if this initiative, it outsourced over forty custom-built predictive models over a one-year period. These models drove selections for the cost-effective micro-marketing of dozens of products and services into the customer base.
The first model took the outside data mining company about eight weeks to build. However, several months into the relationship, the publisher asked if three models could be constructed in a single week. By then, the processes were so streamlined that this was entirely feasible. As you can imagine, the data mining firm was able to charge a much more favorable per-model price than it could have for a one-off project!
Where is Your Marketing Database Hosted?
It can make good sense for a direct marketer that hosts its own marketing database to also handle the data mining. When a company already has the data, it is relatively straightforward for it to do the mining.
Often, a direct marketer that has gone the outsourced database route will also rely on the hosting company to also do the data mining. This can be an efficient and beneficial relationship as long as data mining is a core competency of the data management firm and not just an afterthought. Unfortunately, there are far too many service companies whose idea of a data mining department is to hire a handful of low-to-mid-level staffers. Often, these individuals have solid academic credentials but little industry experience.
Important In-the-Trenches Experience
Regardless of whether your data mining is in-house or outsourced, be mindful of the fact that what one learns in a statistics course has little bearing on most of the real-world decisions that data miners in our industry face. For example, consider some of the non-statistics-based issues that must be successfully navigated to build an effective predictive model:
- Which mailings/drops should comprise the analysis file?
- Should a single or multiple-model strategy be employed?
- If a multiple-model strategy is employed, should a matrix or product-of-the-model-outputs approach be used?
- What should be predicted; that is, what is the definition of the dependent variable(s)?
- Should outliers be eliminated or capped, and what criteria should be used to do this?
- Of all the possible permutations of data elements within the marketing database, which should be included as potential predictor variables?
- For each potential predictor, does the statistical relationship to the dependent variable make ongoing business sense?
Cost and Availability
Another important factor to consider is cost and availability. Outsourcing will be of little value if your chosen vendor provides project quotes that are well above your budget, or if its data mining department is so swamped that it is difficult to schedule any of their time.
One way to ensure cost-effective availability is to negotiate a retainer relationship in exchange for an agreed-upon level of support. Often, such arrangements specify the identity of an individual who acts as a de facto employee of the client. Another advantage is that the dedicated data miner becomes totally steeped in your business.
Hybrid Solution #1: The Virtual Manager
One approach that can work quite well is to retain an outside consultant to mentor a one or even two-person data mining department. The consultant provides the real-world perspective that can only be gained by years of in-the-trenches industry experience. Also, this individual spearheads the overall direction of data mining projects, and provides direction on specific technical issues. An added benefit is that the consultant maintains the "company quantitative memory" during times of in-house staff turnover. The on-staff data miners, in turn, do all of the "heavy lifting," which keeps consulting fees at a reasonable level.
Hybrid Solution #2: Outsourcing Cutting-Edge Assignments
Another approach is to hire an outside firm for the most challenging assignments. That way, the ongoing heavy lifting is done by the in-house staff, which has a moderating effect on overall costs, while the outside firm focuses on "pushing the envelope."
One example is a multi-billion dollar company that sends hundreds of direct mail and email promotions a year across its Direct and Store divisions. The firm has a keen interest in employing quantitative methods and test strategies to optimize this significant marketing investment. The company has a substantial budget for in-house database marketing, including many senior data miners and a state-of-the-art centralized data repository.
Nevertheless, this firm is always on the lookout for new ways of thinking. It understands that its own people do not have all the answers. Therefore, it engaged an outside data mining group to spearhead an approach to Contact Optimization. The company was very happy with the results of the original project, which was about one year in duration. As a result, it hired the outside group to perform additional work.
Conclusion
Is an in-house or outsourced approach right for you? The answer lies with the specific circumstances that surround your company. Decide on how big a data mining department you can afford. If you can support overlapping personnel, then seriously consider an in-house solution.
Ask yourself if you are looking for ongoing data mining support or one-off work. If the later, then by all means look to the outside.
Another important consideration is where your marketing database is hosted. An outsourced database is a factor that favors a decision to farm out your customer data mining.
Finally, look closely at a hybrid solution. If you are a small company, you might want to consider hiring one or two data miners to perform the heavy lifting, and retain a seasoned outsider for important mentoring. Or, especially if you are a large company with substantial in-house resources, consider hiring an outside firm for cutting-edge assignments.
Regardless of which approach you decide to go with, make sure that the data miners have substantial in-the-trenches industry experience to supplement their academic training. And, that they work in an environment where a premium is placed on collegiality and career advancement.
