Avoid the Dangers of Data-Mining, Part 2

6. Look for the use of a control. A control is a portion of the data series not used to estimate the relationship. It's left to the side to test the relationship after the "best" model is chosen. Often, the control will indicate that the "best" method isn't all that good. And beware of methods that use the control data multiple times in order to test the best methods. That defeats the purpose of a control by data-mining the control sample.

7. One of the trends in accounting is to make increasingly detailed rules in an attempt (wrongheaded) to fit each individual company more precisely. The problem with that is it makes many ratios difficult to compare across companies and industries without extra massaging to make the data comparable. This makes thinning out a stock universe via screening to be less useful as a tool. For quantitative analysis to succeed, the data need to represent the same thing across different firms.

Practical Recommendations

There are many pitfalls in quantitative analysis. But three simple considerations will help protect investors from the dangers of data-mining.

1. Paper trade any new quantitative method that you consider using. Be sure to charge yourself reasonable commissions, and take into account the bid/ask spread. Take into account market impact costs if you are trading in a particularly illiquid market. Even after all this, remember that your real-world results often will underperform the model.

2. Think in terms of sustainable competitive advantage. What are you bringing to the process that is not easily replicable? How does the method allow you to use your business judgment? Is the method so commonly used that even if it is a good model, returns still might be meager? Even good methods can be overused.

3. If doing quantitative analysis, do it honestly and competently. Form your theory before looking at the data and then test your theory. Then, if the method is a good one, apply the results to your control. If you perform quantitative analysis this way, you will have fewer methods that seem to work, but the ones that pass this regimen should be more reliable.

---

David J. Merkel, CFA, FSA, is a senior investment analyst at Hovde Capital, responsible for analysis and valuation of investment opportunities for the FIP funds, particularly of companies in the insurance industry. Previously, he managed corporate bonds for Dwight Asset Management. At time of publication, neither Merkel nor his fund had any positions in the securities mentioned in this column, though positions may change at any time. Under no circumstances does the information in this column represent a recommendation to buy or sell stocks. While Merkel cannot provide investment advice or recommendations, he welcomes your feedback and invites you to send your comments to [e-mail].

Analyst Certification: All of the views expressed in the report accurately reflect the personal views of the research analyst about any and all of the subject securities or issuers. No part of the compensation of the research analyst named herein was, is, or will be, directly or indirectly, related to the specific recommendations or views expressed by the research analyst in this report.

Merkel is employed by Hovde Capital Advisors LLC (the "firm"), a registered investment advisor with its principal office located in Washington, D.C. The Firm and/or its affiliates have or may have a long or short position or holding in the securities, options on securities, or other related investments of the issuers mentioned herein.