TheStreet.com's Analyst Rankings: How We Did It -- and Why

Who needs another ranking of Wall Street analysts? Individual investors, that's who. While several publications produce annual analyst rankings, to date none has captured the essential aspects of the analyst role that matter most to ordinary investors. None really tells you which analysts will help make you money.

So, at TheStreet.com we set out to create the new standard in analyst rankings. Our objective: to identify analysts who have the respect of key institutional investors and strong stock-picking skills. Ours is the first ranking methodology to combine both institutional voting as well as objective data on stock-picking performance in one overall rank.

The project was made possible by the generous cooperation of Wall Street research directors, individual analysts and institutional investors. They helped us create comprehensive nomination lists and vet our quantitative data. In the polling phase, the institutional investors produced a resounding voter turnout. For the quantitative analysis of stock-picking performance, we selected widely respected I/B/E/S International, the New York-based source of analyst estimates, research reports and other industry data. And our methodology was developed in consultation with Robert Whitelaw, associate professor of finance at New York University's Stern School of Business.

The Landscape

With decades of experience among them in the analyst ranking field, TSC's rankings team members have been well aware of the limitations of other surveys.

The Institutional Investor ranking measures how well analysts are regarded among institutional investors, but traditionally has not looked at stock picking -- a basic analytical function. Moreover, many analysts have learned to "game" the II poll. They swing into action right before the ballots go out, timing their reports and client visits to garner votes. Finally, the II ranking mails ballots to more than 700 institutions, even though only a much smaller number of managers is in regular contact with the analysts.

The Winners

By name By category By firm Best stock pickers Best firms

The Wall Street Journal's ranking has focused only on stock picking. That means an obscure analyst could make a lucky call in a given 12-month period and blow away the competition -- then fade back into obscurity. How much influence analysts have with active institutional investors -- the key consumers of Wall Street research -- is not measured. That is an important factor, since an influential analyst can move the markets by guiding institutional clients one way or another. A great stock picker without that following can't.

Our mission, then, has been to create a ranking that assesses analysts' relationships with key decision makers at institutions and their skill at stock recommendations. To achieve this goal, we essentially conducted two rankings -- a vote of the so-called "buy-side" of institutional investors and a ranking of analysts based on their stock-picking performance. We then combined the results in an overall score.

Polling the Institutional Investors

The Voter Pool

In designing our vote, the first thing we needed to decide was, What's the right voter pool? We questioned the wisdom of existing rankings. Should we be focusing on the institutions with the most money under management -- as II does -- or the institutions that are the most active traders of U.S. equities?

The Wall Street research directors we consulted almost unanimously agreed that their firms' most important clients are the institutions that generate the most commissions. Those institutions can be, but are not necessarily, the institutions with the most assets under management.

In fact, the research directors explained to us that the number of institutions that matter to their organizations is quite small -- no more than 100 or so. Some advised us not to poll more than 50 institutions, because they estimate that those top 50 account for 80% of their business. We didn't limit our sampling quite that much, but we did constrain our universe to a select group.

Research directors were cooperative in helping us to assemble our pool: They provided us with lists of their firms' top 100 clients, divided into two tiers based on importance. We compared the numerous lists and wound up with 160 institutions that appeared on many of them. We assigned the investors to one of five tiers, with the top tier consisting of institutions that made the top 50 lists of all sell-side firms and the lowest tier consisting of institutions that made only a handful of the bottom 50 lists. These distinctions carried weight in the final voting score.

The Ballot

With our 160 voter firms identified, we designed our confidential ballot. We asked voters a simple question: "Who are your favorite analysts?" Favorite is, admittedly, a subjective term. One voter may select an outstanding stock picker, while another might favor a standout communicator. However, after having identified the voters most familiar with analysts, we decided the criteria were the voters' call.

Beyond individual analysts, we also asked voters to identify the best analyst teams, because some major sectors require group coverage. Moreover, big-name analysts are sometimes so busy with investment banking pursuits and marketing efforts that they leave the analytical and communications chores to their staff. We also added a question to find the best "newcomers" in the industry so that investors can keep an eye on the next new stars before they attain prominence.

Beyond soliciting general favorites, we also sought to gather more specific information about the analysts. In scores of meetings before the vote, we asked institutional investors what they value most in an analyst. From our discussions, we assembled a list of six attributes that are the hallmark of a successful industry analyst:

• Makes money for me: provides rewarding, actionable recommendations
• Saves me from disaster: is sufficiently pessimistic when warranted
• Makes me think: provides information and insights that challenge current opinion
• Tells the truth: is an objective, independent thinker -- not held hostage to investment bankers or corporate management
• Meaningful service, not overkill: provides value-added, personalized and speedy communication
• Well-connected: has broad, deep relationships with quality sources

We came up with two additional attributes for "macro" analysts, who track investment specialties such as economics and portfolio strategy, but don't usually make stock picks:

• Predictions often pan out: has a good track record
• Reader-friendly reports: provides timely, concise and readable written work

And we created attributes that apply only to analyst teams:

• More than the sum of the parts: the team makes an effective, collaborative and cohesive effort
• Everyone's in the loop: all team members can accurately represent the viewpoint of the group
• Broad and deep coverage: together, the analysts provide the big picture as well as detailed knowledge of the field and individual company
• Predictions often pan out: the team has a good track record
• Reader-friendly reports: the team provides timely, concise and readable written work

After they identified their favorite analysts and teams, voters were asked to rate their choices in these various attributes -- assigning a rating of "strong," "average," "weak" or "don't know."

The ballot also invited voters to write in comments about their selections.

Industry Categories

Another decision: In which industry categories should we ask voters to name their favorite analysts, teams and newcomers? Not a simple question: Every research department on Wall Street slices the stock market differently. A company that falls under "publishing and printing" at one firm may be covered by the "Internet software and services" team at another. Yet voters needed to able to look at each category on our ballot and know immediately which companies and analysts we were talking about.

Determined to find an objective way to assemble a category roster, we turned to the new Global Industry Classification Standard created by global index providers Morgan Stanley Capital International and Standard & Poor's. The index, which consists of 10 sectors aggregated from 23 industry groups, 59 industries and 123 subindustries, became our starting point. Because many of the subindustries were too small to have dedicated analysts covering them, we calculated the market caps of each. (The index mapped 6,000 companies to the subindustries.) Then we drew up some basic rules for whittling the figure down to a more reasonable number, essentially combining smaller-cap industries and eliminating the smallest.

We wound up with 80 industry categories, then added four important macro categories: economics, portfolio strategy, quantitative strategy and technical analysis.

The Voting Process

In designing the actual ballot, we employed a medium never before used for an analyst poll: the Internet. With the help of New York design firm methodfive, we created a secure, confidential, easy-to-use online ballot. The ballot included pull-down menus listing analysts in each industry category, so voters could easily locate the analyst of their choice. Where did the analyst lists come from? Directors of research at 100 Wall Street firms enthusiastically nominated their analysts in our categories. (Voters were able to write in their choices, too.)

We sent out ballots on March 9 to analysts and portfolio managers at the 160 institutions we were targeting. Each voter received an email with the ballot URL, user name and password for his own unique ballot, which consisted only of the categories that voter covers.

After eight weeks of balloting, we received ballots from voters at 138 institutions, 86% of our sample. Even more important, 46 (or 92%) of the top 50 most-active institutions responded. After eliminating a number of categories that didn't spark much interest among voters, like diversified commercial services, household durables, Internet retailing and leisure and photographic products, we wound up with reliable results in 63 industry categories and three macro categories.

Voting Score

Once we had our poll results, we came up with "voting scores" for each analyst. Each analyst who received even a single vote got a voting score. We calculated the score by weighting the votes the analyst received by both the voting institution's tier (1 to 5, reflecting the voting firm's importance to Wall Street firms as described above) and by the place that the voter awarded to the analyst (first, second or third). First place is given a weight of 3 points, second place is given 2 points and third place is given 1 point. (Since voters could select only one team and newcomer, their scores were calculated simply by multiplying their votes by the voters' tier.)

To avoid giving any one institution a disproportionate influence over the results, we counted only one vote per institution per category. If two voters from the same institution provided four names in aerospace, for example, we would add up the points that each analyst received and re-rank analysts accordingly.

Here's an example:

When two voters from the same firm vote in the same category
Voter A of Firm X	Voter B of Firm X	Totals
1st: John Doe (3 points)	1st: Mary Smith (3 points)	Mary Smith: 5 points
2nd: Mary Smith (2 points)	2nd: Joe Nichols (2 points)	John Doe: 4 points
3rd: Alice Jones (1 point)	3rd: John Doe (1 point)	Joe Nichols: 2 points
		Alice Jones: 1 point

In other words, this institution considers Smith No. 1, Doe No. 2 and Nichols No. 3. In the final calculation Smith would get 3 points from this institution, Doe 2 and Nichols 1. Jones wouldn't get any voting points from this institution. (Because at least one voter included her on the ballot, she would be eligible to get a stock-picking score.)

Ranking Analysts on Stock Picking

With the ballots in progress, it was time to focus on analysts' stock picking. This part of the project we could not do alone. We turned to I/B/E/S. The I/B/E/S database includes data from nearly 3,000 U.S. analysts at 250 firms, ranging from the largest global investment banks to the smallest local firms.

Measuring Stock Recommendation Success

After numerous discussions with institutional investors and research directors, we decided to focus on stock recommendations, rather than earnings estimates or other forecasts, for this leg of the project. These professionals told us that moneymaking stock recommendations matter most to them; accurate earnings estimates, while important, tend to be further down on the list.

To judge the analysts on their stock-picking prowess, we created a synthetic portfolio of $1,000 for each analyst. The portfolio would contain only the stocks that had been mapped to the category. Also, it only included the stocks that the analyst had given a positive recommendation.

Why not include negative calls? Frankly, there are hardly any. Our approach was essentially to measure the performance of the stocks that had the analyst's support -- basically the same yardstick that applies to most investors' portfolios. The analysts who picked badly got burned.

We selected a 12-month period -- Feb. 15, 1999, to Feb. 15, 2000 -- and "invested" an equal amount of money in each stock, except for stocks on which the analyst had a very positive rating -- that firm's equivalent of a strong buy. These recommendations were treated as 200% of a buy, resulting in a larger relative purchase.

At the end of the 12 months, the portfolio was "sold" and its final value calculated. We used this final portfolio value to produce each analyst's stock-picking score.

We opted for a 12-month period for one important reason: We wanted our stock-picking data to be as error-free as possible. Checking more than 12 months of data would have been -- both for I/B/E/S and for the research directors -- a Herculean task.

Assigning the Categories

I/B/E/S had historical data on the vast majority of the analysts on our roster of nominees. But they didn't use the same categories we did. For example, I/B/E/S classifies Gateway as a computers and peripherals company, but Gateway is followed by analysts we characterize as computer hardware specialists. Because we wanted the portfolios of our computer hardware analysts to include Gateway, I/B/E/S remapped that company to computer hardware for our survey.

Checking the Data

After I/B/E/S compiled its stock recommendation information on each analyst, it sent research directors a packet containing all the data it had on their analysts. Research directors had approximately three weeks to check the data for accuracy.

Most research directors were pleased with the I/B/E/S data, and most didn't make any corrections at all. (I/B/E/S made changes to the data only if the corrections were accompanied by supporting documentation, such as a research report.) More important, every research director we contacted allowed I/B/E/S to use their analysts' data in this ranking.

After all was checked, we had our stock-picking scores.

Identifying the Winners

Overall Score

Once we had our vote results and our stock-picking results, we came up with the final scores. Here's how we did it.

To identify the top analysts in the industry categories, we first normalized their voting and stock-picking scores -- on a scale of 0 to 100 -- so they could be compared. To explain: The analyst whose portfolio value was the highest in a category earned a normalized score of 100; the scores of the other analysts in that category were derived by dividing the values of their portfolios by the highest value and multiplying by 100.

Here's an example:

Normalizing the scores
Voting Results		Stock-picking results
Raw score	Normalized score	Portfolio value	Normalized score
(Analyst A) 200	50	$7,500	100
(Analyst B) 300	75	$6,000	80
(Analyst C) 400	100	$4,000	53

To create the overall score for each analyst, we merged the voting and stock-picking scores -- taking 70% of the voting score and 30% of the stock-picking score. We considered that breakdown appropriate because (a) the stock-picking data accounted for only one year, and luck can play its part; and (b) there's more to being a good analyst than stock ratings.

Most categories have only three winners, but there are as many as five names in categories where the voting was heaviest and/or the competition was tightest.

Attributes Scores

Attributes weren't included in the overall score. However, we did grade analysts in our attributes. The grade for each attribute is represented by one to three stars, with three stars being highest. Three stars stands for "strong," two for "satisfactory," one for "sub par."

Miscellany

Analysts who were missing either a stock-picking or a voting score were not ranked. So analysts who didn't get any votes don't show up on our ranking regardless of how well their stock recommendations fared.

Analysts at the few firms that do not give data to I/B/E/S are not ranked no matter how well they did among voters. An example: Dan Pickering of Houston-based Simmons & Co. did great in the energy equipment & services vote, but his firm doesn't give I/B/E/S data. Without that score, Pickering was ineligible for the overall rank.

Ties occurred when the overall scores of more than one analyst were less than one point (out of 100) away from each other. Winners in the macro categories -- economics, portfolio strategy and quantitative strategy -- were selected based only on their voting scores, since these analysts usually don't make stock recommendations.

Finally, our mission was to identify the best analysts at a specified point in time: Feb. 1, 2000. Those who switched firms or left the business after that date (we made every effort to provide current information) are still shown at their previous employer, though we indicate where they are working now.

Summing Up

When we first developed our idea of considering both reputation and stock picking in assessing the performance of analysts, we wondered why no one else had done it before. It seems so logical and comprehensive. But several months into the project, it dawned on us why such an idea had never been implemented before. It's quite a task! But having resolved all the problems and surmounted the glitches, we stand quite proud of our accomplishment. We know we have truly identified the best analysts on Wall Street.

We welcome comments and questions about our methodology. Please send them to [e-mail].