Online Broker Survey: How We Crunched the Numbers

TheStreet.com's latest Online Broker Survey drew more than 2,400 responses rating readers' experiences with more than 130 online brokers. We've changed a number of things since our last broker survey, conducted in February and March.

This time around, we expanded the ballot so you could rate more brokers (write-in votes were allowed in both surveys) on more criteria and added some new questions on topics such as margin calls and direct-access trading. We also simplified the rating system a bit, asking you to judge brokers on a three-point scale (excellent, good, poor), rather than the four-point scale (excellent, good, average, poor) we used last time. Otherwise, our methodology has remained much the same. Here's how we crunched the numbers.

The Voters

This was a reader's survey, not an expert's survey. It was available free to any visitor to TheStreet.com or RealMoney.com from Aug. 23 through Sept. 17. Could brokers have filled in our ballots themselves and asked their employees and grandmothers to do the same? Sure. This was not a scientific sampling of opinion.

Still, we put two measures in place to discourage ballot stuffing. Using "cookies," our survey company, Informative, set things up so that only one survey could be completed from a single PC. We also required readers to include their email addresses. In a handful of instances, where more than one ballot came from the same email address, we eliminated the duplicates.

The Criteria

We began by asking online investors what matters most to them. We listed 26 features, ranging from easy portfolio tracking to reliability during a market crisis, and asked voters to rate whether the features were "vital," "important" or "not important." We converted the responses into a three-point scale:

Rating the Criteria How important are the features and services brokers offer?
Vital	3 points
Important	2 points
Not important	1 point
No response	0 points

We then calculated the average score for each feature. There was a clear break between the scores of the top 11 criteria, listed below, and the remainder.

Ranking the Criteria
Reliability: access even in times of heavy volume	2.90
Real-time quotes	2.78
Fast order confirmation	2.74
Best execution price	2.64
Up-to-date reports about site reliability during a crisis or outage	2.58
Easy-to-implement complex trades	2.58
Easy account administration	2.54
Low commissions	2.52
Easy portfolio tracking	2.44
Customer service by telephone	2.42
Real-time market and company news	2.38

We counted these 11 standout traits -- and how brokers performed in them -- toward the overall score. This chart shows the percentage each factor counted toward the overall score in calling a winner:

Assigning Percentages
Reliability: access even in times of heavy volume	10.2%
Real-time quotes	9.7%
Fast order confirmation	9.6%
Best execution price	9.3%
Up-to-date reports about site reliability during a crisis or outage	9.0%
Easy-to-implement complex trades	9.0%
Easy account administration	8.9%
Low commissions	8.8%
Easy portfolio tracking	8.6%
Customer service by telephone	8.5%
Real-time market and company news	8.3%

Thus, a broker's reliability accounted for 10.2% of its overall score, its performance in providing real-time quotes counted for 9.7% of its score and so on.

The Brokers

We then asked voters to rate their primary broker -- the one they used most -- on the same 26 features. Again, we used a three-point scale.

Rating the Brokers
Excellent	3 points
Good	2 points
Poor	1 point
No response	0 points

Each broker got an average score for each feature (the sum of all the ratings divided by the number of votes). The highest possible score was 30. That score was then multiplied by the feature's weight in the overall brokerage score. For example, we multiplied Datek's average score in real-time quotes, 27.21, by the real-time quotes' value of 9.7% -- getting 2.64. We did that for the 11 most important features; then we added up the results.

Statistical Significance

Online Broker Survey Six-month update

Reliability Improves; Rankings

Cutting Out the Middleman

Profiting From Your Trades

Mellow About Margin Calls

How We Tallied the Scores

Our February-March survey drew more than 10,000 responses. In that survey, seven brokers receiving more than 650 votes each were grouped in "League A" and 10 others receiving between 72 and 312 votes were grouped in "League B."

Our latest survey drew one-fourth as many responses, perhaps owing to its late-August start. As a consequence, no single broker received more than 362 votes. That left us with a judgment call to make. Several brokers scored very well on the survey, but their results were based on a relatively small number of votes. In general, the fewer the votes, the less reliable the results. It's also possible that ballot stuffing would have a greater impact on the results for brokers that received relatively few votes (though there's no evidence that any ballot stuffing took place).

So we decided to make 100 votes the minimum for being included in the final ranking. Although it's a nice round number, it may seem arbitrary. But as you can see from the table below, there's a definite break between TD Waterhouse, the last broker to make the cutoff, and the next one, myTrack.

Vote Count These brokers received at least 40 votes
Broker	Votes
Datek	362
Schwab	335
E*Trade	318
Fidelity	181
Ameritrade	128
DLJdirect	112
TD Waterhouse	102
myTrack	64
Brown & Co.	62
Merrill Lynch Direct	46
Morgan Stanley Dean Witter	44

As it turns out, the seven brokers with the most votes in this survey are the same seven grouped in "League A" in the last survey. But because of the sharp difference in the number of votes, it's hard to draw any sweeping conclusions from a comparison of the two surveys. Rather, it would be best to look at this survey as an update of the last one -- a six-month checkup, if you will.

Had we established a lower cutoff of 40 votes, myTrack, Merrill Lynch Direct and Brown & Co. would have captured three of the first four spots in the survey. To be fair, they may very well deserve those high rankings. But because they got relatively few votes, we can't really be sure.

We welcome any comments you have on the methodology or the results of this survey. Please send them to [e-mail].