More Data, More Problems

Coming soon to a theater near you -- Big Data! Not the band, the data.

I'll let CNBC's Bob Pisani Pisani-splain it.

"Did you ever hear or read a comment about a market trend and wonder how accurate it was over a certain time period? Or when some trader says, 'We are entering a seasonally strong period of the year,' did you ever wonder exactly how often that was true, and under what circumstances?

Sure you have. All of us who cover the markets engage in this kind of research every day.

Today, CNBC has announced a strategic partnership with Kensho, a company that was set up to answer complex financial questions...in a few seconds."

It all sounds great on the surface. I mean -- more info, more ways to find out if Event X really leads to Market Reaction Y.

But here is the problem: Data isn't about pure access. It's about asking the proper questions and, perhaps more importantly, it's about putting the answers in the proper context.

Sports broadcasts are filled with this sort of stuff all the time. Ever watch a football game and find out such-and-such team is 20-1 every time Random Running Back has rushed for over 100 yards? Or that Pick Any Baseball Team is 70-3 when leading after eight innings? Both stats sound impressive on the surface, but minus any context, they don't necessarily mean much.

What NFL teams with a 100-yard rusher and baseball teams with leads after eight innings don't win 95% of the time? That team-specific data above looks more expected than exceptional.

We would never fall into that trap here in finance -- probably. Back to Pisani:

"For example, a couple months ago the 50-day moving average of the Russell 2000 crossed the 200-day moving average to the downside, a much-feared technical pattern known as the 'Death Cross.' There was much discussion of this on-air, as well as in print.

The Russell did decline that day, but the historical pattern is very mixed. This has happened 20 times since 1988, and five days later the Russell has been down 55 percent of the time, up 45 percent. That is a fairly mixed result, though the downs are somewhat bigger than the ups.

But two months later, the Russell has been UP 63 percent of the time. In other words, for most intermediate-term investors, simply holding would have been better than selling immediately."

Now, that last statement is true on the surface. But there's zero context attached to it. Maybe two months later, the market is higher 75% of the time on random dates, so that Death Cross actually was a modest negative. Or alternately, in the 37% of the time the market did poorly after the Death Cross, it did very poorly. Remember when we ran through numbers on SPDR S&P 500 ETF Trust (SPY) simply crossing below the 200-day moving average? If you bought that first day and held until it crossed back above, you won the trade something like 90% of the time. But the times you lost, you often really lost.

What other errors can we make via this new CNBC toy? Well, sample size comes to mind. Guessing we'll see a lot of cases where Stock or Sector X lifted seven of the last nine times Event Y happened. There's also the fact that markets have so many moving parts that it's more or less impossible to control for all of them so as to isolate how one factor might really influence one sector.

They use large oil price declines as an example, which actually highlights both of these issues. Do we have a statistically significant sample size? Did we control for any other background factors? Without that info, I'm not sure the relationships are terribly predictive.

Look, more data can only help. I really, really, really, strongly advise not taking it at face value, though. It is always meaningless independent of proper context.

Disclaimer: Mr. Warner's opinions expressed above do not necessarily represent the views of Schaeffer's Investment Research.