Online Brokers 2000: How We Crunched the Numbers

It was an avalanche. Exactly 10,566 ballots were filled out in TheStreet.com's Online Broker Survey 2000, the biggest online broker survey ever. That's a lot of data, and there are many ways to mulch it. Here, we explain how we came up with the winners.

The Voters

A reader's survey rather than an expert's survey, it was available free to any visitor to TheStreet.com from Feb. 22 through March 1. Could the brokers have filled it out themselves and asked their grandmothers to do the same? Sure. The beauty and bane of the Internet are that it's not a pure, pristine pool.

Online Broker Survey 2000 In Today's Package State of the Industry Broker Performance Chart How Datek Won About the Survey Calendar of Stories Message Boards

Still, we did have two measures in place to prevent ballot-stuffing. Using "cookies," our survey company, Informative, set things up so that only one survey could be completed from a single PC. We also required readers to include their email addresses. We received a handful of instances where more than one ballot came from the same email address, and we eliminated the duplicates. A few voters put "no" as their email address; accordingly, their ballots weren't accepted. So our total voter count was 10,553. Within that group, we depended on statistics to help us assess the reliability of the data. Essentially, we gave more weight to the results on brokers that got more votes.

The Criteria

We began by asking online investors what matters most to them. We listed 18 features, ranging from after-hours trading to reliability, and we asked voters to rate them in importance from "vital" to "don't care." We converted the responses into a 5-point scale:

We then calculated the average score for each feature:

Reliability: access even in times of heavy volume 4.83 Real-time quotes 4.63 Fast order confirmation 4.52 Easy account administration & portfolio tracking 4.43 Customer service 4.39 Best execution price 4.34 Easy-to-implement complex trades (e.g., stop or limit orders) 4.27 Low commissions 4.16 Real-time market and company news 4.09 Investment research 3.52 IPO availability 3.49 Breadth of investment product offerings (e.g., mutual funds, options) 3.41 Level II NASDAQ quotes 3.39 Email alerts 3.27 Easy options trading 3.20 After-hours trading 2.94 Banking services 2.62 Bond trading 1.80

The top group of traits came in with scores above 4. We counted those nine stand-out traits -- how brokers performed in them -- toward the overall score. This chart shows the percentage that each factor counted toward the overall score in calling a winner:

Reliability: access even in times of heavy volume 12.2% Real-time quotes 11.7% Fast order confirmation 11.4% Easy account administration & portfolio tracking 11.2% Customer service 11.1% Best execution price 11.0% Easy-to-implement complex trades (e.g., stop or limit orders) 10.8% Low commissions 10.5% Real-time market and company news 10.3%

Thus, a broker's performance in low commissions accounted for 10.5% of its overall score; its performance in reliability counted for 12.2%.

The Brokers

We asked voters to rate their primary broker -- the one they used most -- on the 18 features listed above. Again, we turned responses into points:

Poor was given a value of zero on the theory that doing something poorly is just as bad as not doing it at all.

Each broker got an average score for each feature (the sum of all the ratings divided by the number of votes), with 30 being the highest number attainable. That score was then multiplied by the feature's weight in the overall brokerage score. For example, we multiplied Datek's average score in real-time quotes, 24.89, by real-time quotes' value of 11.68% -- getting 2.91. We did that for the nine most important features, and then we added up the results.

Statistical Significance

Before calling a winner, we weighed the statistical significance of the voting pool for each broker. There were seven brokers that received at least 650 votes. Consulting with Robert Whitelaw, associate professor of finance at New York University's Stern School of Business, we decided that the results for these firms -- representing 6% of the voter pool -- were the most reliable. Furthermore, as these are the biggest brokers, they also face the greatest challenges in such factors as reliability and customer service. A firm with significant market share that can do well on these tasks deserves more credit than a firm that has fewer customers calling.

Our League B brokers collected less than 3% of the responses. The figures were still hefty enough to be reliable, though we don't have quite the same confidence in them as we do in League A brokers. Our cutoff point for this group was above 0.5% of the total population of respondents. The sample size for its smallest member, American Express Brokerage, was 72 respondents. Names like Morgan Stanley Dean Witter (MWD Quote) and National Discount Brokers (NDB Quote) are in this group.

There were many other brokers that received fewer than 72 votes, but we didn't feel confident enough in those numbers to report on them.

Because of these statistical variations, we published the overall results for firms in Leagues A and B separately.

What's Your Type?

Since different types of investors have different needs, we also sorted the results based on the types of trader so that you could match your needs with the firms that fit them best.

We put voters into three pools based on how frequently they trade and how long they hold their investments.

The population broke down as follows:

The other 31% or so of voters could not be classified easily.

To identify the brokers best suited to each investor group, we recalculated the average scores for each of the 18 features based on the preferences of these particular voters. Some features found their way back to the mix. Buy-and-hold investors, for example, regard investment research as above average in importance, so that feature was appended to the group of features that determined the broker's score. Similarly, daytraders had Level II Nasdaq quotes added to their important-feature list.

You'll find all of this information in our story on investor types Friday.

For the Future

We'll be doing this survey again in about six months. To that end, we welcome any comments you have on the methodology or the results of this survey. Please email us at [e-mail].