"The problem is that we attempt to solve the simplest questions cleverly, thereby rendering them unusually complex. One should seek the simple solution."
-- Anton Chekhov

Eventually, every investing strategy that gains a following has to come up against Burton Malkiel.

Malkiel is the author of the landmark

A Random Walk Down Wall Street, which contains the most assured route to long-term investing success yet discovered. Ironically, or perhaps as a result, the Princeton University professor's treatise is probably the most-reviled, oft-challenged philosophy on Wall Street.

The message of his book, first published in 1973, is simple: Investors are better off buying and holding an index fund than attempting to buy and sell individual stocks and actively managed funds. The market, according to Malkiel, prices stocks so efficiently that strategies designed to beat it are useless. As he puts it -- and this is where the Wall Street revulsion comes in -- "A blindfolded monkey throwing darts at a newspaper's financial pages could select a portfolio that would do just as well as one carefully selected by the experts."

His treatise helped spawn the index fund industry (he now sits on the board of directors of Vanguard). It also spawned a slew of critics, including a new generation in the wake of the Internet bubble. The 70-year-old professor aims to set the record straight with the eighth edition of

Random Walk

, which hits the shelves this month. Along with extensive updates, the new edition features a new chapter on the Internet bubble that fits seamlessly into Malkiel's chronicle of market manias over the past four centuries. It also includes a strident defense of efficient markets and index fund investing.

Professor Malkiel spoke in his campus office about the new edition.

What are the reasons for updating

A Random Walk Down Wall Street?

Professor Malkiel:

While the basic investing message of the book hasn't changed since its original edition in 1973, there have been several important developments over the past few years that make new editions necessary.

First and foremost, the book remains an investment guide for individuals. The advice I have given since the first edition is that investors are better off buying and holding index funds in a diversified portfolio than trying to beat the market by purchasing individual stocks or actively managed mutual funds. I wanted to examine if the latest data validated the advice from earlier editions, and examine some of the recent challenges to the efficient-market theory. The latest evidence confirms that index fund investing works exceedingly well.

Also, I wanted to address the astonishing number of financial innovations that have arisen in recent years that change how individuals invest in the market. When I recommended buying the

S&P 500

index in the first edition, I noted that this was a theoretical suggestion because there were no index funds at the time. I also asked, isn't it time that investors have the ability to buy the index?

In the intervening years, of course, we have seen the creation of a wide range of index funds. There are many other new products to consider as well: Money funds, municipal bond funds, real estate investment trust funds. We didn't have Roth IRAs or Section 529 college savings plans before. This edition of the book discusses these new products and explains how individual investors should use them.

What about the events that have transpired in the markets themselves?

Professor Malkiel:

That is the third reason for updating the book. The past few years have witnessed the Internet bubble and its collapse. In previous editions, I discussed the speculative bubbles through market history -- the Dutch tulip-bulb mania of the 17th century, the South Sea bubble in the 18th century and, more recently, the Nifty Fifty in the 1970s and the Japanese market of the 1980s.

In the new edition, I devote an entire new chapter to the Internet bubble, which has been undoubtedly the biggest bubble in market history. The speculative run-up in "New Economy" stocks pushed the market to extraordinary levels, which inflated expectations of market returns. By 2000, investors said they expected an annual rate of return of 15% to 25%.

The speculative mania also led many investors to sell out of their diversified broad-based portfolio and pour their money into tech stocks. The Internet was the latest new technology that promised to transform the economy, just as the railroads or the automobile had before. But the lesson we relearned is that transforming technologies often prove unprofitable for investors.

During the 1850s, railroad stocks increased to outrageous levels; in the collapse of 1857, many railroad companies went bankrupt. The same thing held for the auto industry, which went from 100 companies in the early days to three. The same thing has transpired with the Internet companies. Once the bubble burst, many companies vanished, and even the surviving "New Economy" stocks lost most of their value. All in all, more than $7 trillion of market value evaporated.

If the Internet mania marked the biggest speculative bubble in history, why hasn't the economic aftermath been the most devastating -- apart from the $7 trillion in market value erased? Is it early yet?

Professor Malkiel:

No, it isn't early. There has been significant effect on the real economy. Business investment, especially for high-tech products, has fallen sharply, and the current sluggishness of the recovery is traceable in part from the over-investment during the bubble period.

What made matters less painful this time is that our policy markers know a lot more about the way economies work than they did in the 1930s.

Federal Reserve Chairman Alan Greenspan has aggressively lowered interest rates.

You note in the newest edition that the Internet mania seems at odds with the notion that the market is rational and efficient. How can a rational market spawn such a bubble?

Professor Malkiel:

The market is usually rational, but it is made up of irrational participants who occasionally create anomalies.

Did the market go through a period of irrationality in 1998 and 1999? Yes, it did. The important thing we learn from the Internet bubble is that the market, as it has in every previous case of excessive speculation, corrects itself.

Do you think the bubble psychology has been finally beaten out of investors, or is it still with us?

Professor Malkiel:

I think investors have been burned by the Internet bubble and are more aware of speculative excesses now. However, while I would like to think that investors will apply what they learned from the past few years to investing in the future, speculative crazes seem to be isolated from the lessons of history.

Why is indexing, in your opinion, the most sensible way to invest in the markets?

Professor Malkiel:

In the 30 years since the first edition of the book was published, index fund investing has proved the most successful investing strategy. The passively managed S&P 500 has regularly topped the performance of more than two-thirds of the mutual funds actively managed by professionals.

How do you account for the active fund managers who have managed to beat the market? A recent Ibbotson study found that winning U.S. stock funds, over the long haul, do repeat good performance.

If you have a room full of people in a coin-flipping contest to see who can flip heads the most times, one of them will invariably flip heads nine times out of 10. But that doesn't mean that person will flip heads more than even half the time during the next 10 flips.

Mutual fund managers who have beaten the market over a certain period have typically underperformed the market subsequently. I have demonstrated this in earlier editions of the book, and the latest figures prove that it remained the case during the 1990s and this decade. As I point out, the top-performing mutual funds of the 1970s underperformed during the 1980s. The top-performing mutual funds of the 1980s underpeformed during the 1990s.

This is even more dramatic when you look at the "hot" funds of the late 1990s. Fund managers who did exceptionally well in 1998 and 1999 managed to outpace the S&P 500 by over two times. In the early 2000s, they have underperformed the market by about three times.

Does this mean there are no individuals who can consistently beat the market? No. But, unfortunately, one only recognizes genius in hindsight. Warren Buffett is indeed an investing genius. But trying to find the next Buffett is like trying to find a needle in a haystack. Why not just buy the haystack?

What do you make of the argument that the S&P 500 has become, in effect, actively managed because of the hands-on roles of Standard & Poor's executive committee?

Professor Malkiel:

The S&P 500 has become more actively managed over the past few years -- it has made more changes to its index of companies in recent years than it traditionally has.

This is regrettable because of the transactions costs associated with buying and selling companies added to or dropped from the index. Yes, the S&P 500 has become more actively managed. Nonetheless, it is still far less actively managed than actively managed funds, which makes it a better investment than actively managed funds.

In the more recent editions of the book, however, I have recommended investors use an index that offers a broader representation of the stock market, such as the Wilshire 5000 or Russell 3000. This is primarily because the S&P 500 only represents the largest U.S. companies that make up about 75% of the total market. Investors who only own an S&P 500 index fund don't have exposure to smaller and medium-size companies. Studies have shown that over the long run, smaller stocks tend to outperform larger-capitalization stocks, so investors would benefit from exposure to them.

If you owned a fund that tracks the Wilshire 5000 as opposed to the S&P 500, you wouldn't have had to buy shares of

Yahoo!

(YHOO)

when it was added to the index. Because you already owned the entire market with a Wilshire 5000 index fund, in effect, you would have already owned Yahoo!.

Professor Harry Markowitz, the founder of the modern portfolio theory, was once asked how he planned his portfolio for retirement. His answer: "I should have computed the historic covariances of the asset classes and drawn an efficient frontier. Instead ... I split my contributions 50-50 between bonds and equities." Do you follow your own advice and invest in index funds?

Professor Malkiel:

Yes, I do. For my own portfolio, I use the asset allocation plan recommended for the most conservative investor -- aged late 60s or beyond -- in the "Life-Cycle Investment Guide" section of my book.

I modify my asset allocation depending on the nature of the investor. For my son's trust fund, for instance, I use a more aggressive plan that is also detailed in the Life-Cycle Investment Guide of the book.

The Life-Cycle Investment Guide, by the way, has held up remarkably well during the past few years. I am happy to mention that the conservative portfolio for older investors -- which includes a 50% weighting in bonds -- has actually made money over the past few years. And the most aggressive portfolio, for investors in their mid-20s, only lost about 7% a year during the past three years. Assuming the investor used dollar cost averaging over the 2000-2002 period.

I should also note that I am a strong believer in Markowitz's modern portfolio theory. Markowitz proves that diversification across several asset classes is a sensible strategy for individuals who want to reduce their risks.

While I do invest in index funds, I should also add that I have had a long interest in stock markets and I was drawn to investing because of my gambler's mentality. I think anyone who goes to work on Wall Street likes to think that they have the ability to choose winning stocks.

While the core of my investments is in index funds, I use a small amount of side money to invest in stocks, which satisfies the part of me that wants to beat the market. I'm not at all sure I do beat the market but I have fun doing it.

What stocks are grabbing your attention?

Professor Malkiel:

In the 1990s, I did invest in several technology companies. I served on the board of directors at a Silicon Valley computer company and therefore studied the technology industry fairly closely. Based on what I knew and my contact with experts in the industry, I chose some stocks that did quite well.

I had owned

Intel

(INTC) - Get Report

for many years. In fact, I made a substantial amount of money from owning Intel over the years. I would've made even more had I sold the stock in early 2000. But we can't predict the future, and we certainly cannot time the market.

Are there any stocks grabbing your attention now?

Professor Malkiel:

Let's see. ... The most recent stock I purchased was

Merck

(MRK) - Get Report

.

Merck's stock had fallen quite a bit, to the point where it clearly appeared to be trading below its worth. When I bought the stock, its price-to-earnings multiple was below 15.

I know with pharmaceutical companies there are times when the drug pipeline slows, but these things correct themselves with exceptional companies such as Merck. It seemed to me the market was overly pessimistic about Merck.

Do investors even need to own stocks?

Professor Malkiel:

For the majority of investors, I would say no. It is very difficult for the average investor to beat the professionals and choose a stock that the market is currently undervaluing. Also, individuals need to be diversified, and an index fund provides that.

If you write another edition of

A Random Walk Down Wall Street

in five to seven years, what do you think the new lessons will be? In other words, this is a roundabout way of asking: What do you expect from the markets in the next five to seven years?

I'm not entirely comfortable with a forecast for the next five to seven years. But over the next decade or so, investors will realize that they have to further lower their expectations for market returns.

I believe the market is fairly valued at current levels, but I don't expect a return to double-digit annual returns from the stock market for the next decade. I think the market could return 7 1/2%-8%. Given the current fixed-income environment, in which 10-year bonds are returning less that 4%, that is a reasonable equity premium.

Nonetheless, individual investors, as well as many professionals, have not modified their expectations in line with this more typical rate of return. In fact, several pension funds still target expected rates of returns in the double digits.

Stephen Schurr writes and edits for the TheStreet.com's Personal Finance section. In accordance with company policy, he doesn't own or short any individual stocks. He welcomes your comments or questions at

steve.schurr@thestreet.com.

TheStreet.com has a revenue-sharing relationship with Amazon.com under which it receives a portion of the revenue from Amazon purchases by customers directed there from TheStreet.com.