The effectiveness of the rule began to be undermined as far back as the late 1980s, as the equity options market developed. Buying a put and selling a call is the equivalent of being short the underlying stock, and it seems reasonable to argue that as more investors became experienced in option trading, they would be able to circumvent the rule if they wished to do so.
The financial engineering which brought about even the first ETFs resulted in actions that further diminished the power of the rule. In 1993, the S&P Depositary Receipts (SPY), or SPY, was granted a diversification exemption. Others followed for the mid-cap Spyders, the PowerShares QQQ (QQQQ) (now known as the QQQQ) and sector ETFs. By late 2002, we had plenty of exceptions, from ETFs to options bullets.
Interestingly, and shortly prior to announcing the pilot to study the elimination of the rule, the SEC actually moved to strengthen the rule significantly. In November 2003, it issued an interpretive release clarifying, effectively, that "married put" or bullet transactions were also subject to the rule. Note the action of the VIX and VXO volatility indices between mid-2003 and mid-2007, when the rule was repealed. Not bad, as smoking guns go.
I would note that although the release's effective date was not until Nov. 21, 2003, the SEC had apparently begun gathering information and making market participants aware of their consideration with respect to issuing the guidance much earlier in the year. It is certainly possible that knowing that the SEC might determine that a common practice violated a long-standing law still in existence could dampen enthusiasm for that activity. This might account for the way volatility came down sharply several months earlier than the release date.