Too Much Data or Not?
Peter Fader, Wharton's quantitative marketing wizard, has a message for CIOs: Stop collecting so much customer data, and stop misusing data mining (read: What Data Mining Can and Can't Do). I don't necessarily agree with it, particularly in the claims of the power of the spreadsheet vs. purely analytical and predictive applications. The author was a little misguided in his own experience, saying that the use of such quantitative analysis is essentially overkill and that easier, simpler, less costly solutions are better or "good enough."
However, the point I derived is that less sophistication is in fact sometimes good enough for a guesstimate or to make quick decisions when time is not available. My experience is that such tools and applications do hold their place in organizations, though, particularly in detailed analysis of large volumes of data or where complex process indicators are not so transparent. No company needs to get bogged down in the proverbial state of "analysis paralysis." You make the call: "good enough" or "more data?" I'd prefer to err on the side of more data, after all, when presenting a simple answer or solution, you'll almost always be presented with the follow-up, "What if...?"
In particular, Fader says, "I would claim that most of that 50 percent is completely wasted. It's one thing to have 50 percent more data, but you're certainly not getting 50 percent more knowledge or insight. In fact, you could be doing more harm than good, because you're crowding out the few variables that really do matter." If the data is cheap to obtain and store, then 50% more data doesn't cost 50% more to keep, and is has future potential value. If some the data adds to "noise," as he claims, don't use it in the particular model. Most complex tools allow this. But, this may be useful information in the future, whether for a particular new problem or for a particular question. The marketing, actuary, finance, and statistical personnel should make these decisions, based on their experience--not the CIO.
Brent P. Malak
Business Intelligence Data Analyst Spheris Operations LLC