It was an avalanche. Exactly 10,566 ballots were filled out in
Online Broker Survey 2000
, the biggest online broker survey ever. That's a lot of data, and there are many ways to mulch it. Here, we explain how we came up with the winners.
survey rather than an
survey, it was available free to any visitor to
from Feb. 22 through March 1. Could the brokers have filled it out themselves and asked their grandmothers to do the same? Sure. The beauty and bane of the Internet are that it's not a pure, pristine pool.
Still, we did have two measures in place to prevent ballot-stuffing. Using "cookies," our survey company,
Informative, set things up so that only one survey could be completed from a single PC. We also required readers to include their email addresses. We received a handful of instances where more than one ballot came from the same email address, and we eliminated the duplicates. A few voters put "no" as their email address; accordingly, their ballots weren't accepted. So our total voter count was 10,553. Within that group, we depended on statistics to help us assess the reliability of the data. Essentially, we gave more weight to the results on brokers that got more votes.
We began by asking online investors what matters most to them. We listed 18 features, ranging from after-hours trading to reliability, and we asked voters to rate them in importance from "vital" to "don't care." We converted the responses into a 5-point scale:
We then calculated the average score for each feature:
The top group of traits came in with scores above 4. We counted those nine stand-out traits -- how brokers performed in them -- toward the overall score. This chart shows the percentage that each factor counted toward the overall score in calling a winner:
Thus, a broker's performance in low commissions accounted for 10.5% of its overall score; its performance in reliability counted for 12.2%.
We asked voters to rate their primary broker -- the one they used most -- on the 18 features listed above. Again, we turned responses into points:
Poor was given a value of zero on the theory that doing something poorly is just as bad as not doing it at all.
Each broker got an average score for each feature (the sum of all the ratings divided by the number of votes), with 30 being the highest number attainable. That score was then multiplied by the feature's weight in the overall brokerage score. For example, we multiplied
average score in real-time quotes, 24.89, by real-time quotes' value of 11.68% -- getting 2.91. We did that for the nine most important features, and then we added up the results.
Before calling a winner, we weighed the statistical significance of the voting pool for each broker. There were seven brokers that received at least 650 votes. Consulting with Robert Whitelaw, associate professor of finance at
New York University's Stern School of Business
, we decided that the results for these firms -- representing 6% of the voter pool -- were the most reliable. Furthermore, as these are the biggest brokers, they also face the greatest challenges in such factors as reliability and customer service. A firm with significant market share that can do well on these tasks deserves more credit than a firm that has fewer customers calling.
Our League B brokers collected less than 3% of the responses. The figures were still hefty enough to be reliable, though we don't have quite the same confidence in them as we do in League A brokers. Our cutoff point for this group was above 0.5% of the total population of respondents. The sample size for its smallest member,
American Express Brokerage
, was 72 respondents. Names like
Morgan Stanley Dean Witter
National Discount Brokers
are in this group.
There were many other brokers that received fewer than 72 votes, but we didn't feel confident enough in those numbers to report on them.
Because of these statistical variations, we published the overall results for firms in Leagues A and B separately.
What's Your Type?
Since different types of investors have different needs, we also sorted the results based on the types of trader so that you could match your needs with the firms that fit them best.
We put voters into three pools based on how frequently they trade and how long they hold their investments.
The population broke down as follows:
The other 31% or so of voters could not be classified easily.
To identify the brokers best suited to each investor group, we recalculated the average scores for each of the 18 features based on the preferences of these particular voters. Some features found their way back to the mix. Buy-and-hold investors, for example, regard investment research as above average in importance, so that feature was appended to the group of features that determined the broker's score. Similarly, daytraders had Level II Nasdaq quotes added to their important-feature list.
You'll find all of this information in our story on investor types Friday.
For the Future
We'll be doing this survey again in about six months. To that end, we welcome any comments you have on the methodology or the results of this survey. Please email us at
Informative provided the technology to conduct this survey.