Angst Reigns on Wall Street

"The very weak jobs report on Friday moved the Federal Reserve to front and center in the debate on economic growth," said Paul Nolte, director of investments with Hinsdale Associates. "While a negative reading on 'job creation' does not necessarily mean we are entering a recession, the likelihood of one has increased."

Homebuilders dropped following Yellen's speech. The Philadelphia Housing Sector Index sank 2.2%, as D.R. Horton (DHI Quote), Lennar (LEN Quote), Hovnanian (HOV Quote), and Toll Brothers (TOL Quote) all slid by 2.1% or more.

Several finance names were also lower. Embattled lender Countrywide (CFC Quote) fell 5.5% after saying it would slash up to 20% of its workforce, or 12,000 jobs. The eliminations will occur "in areas most impacted by lower mortgage market origination volumes," the company said. Countrywide lost $1 to $17.21.

U.S. Treasury securities, meanwhile, rallied. The 10-year bond gained 16/32 in price, cutting the yield to 4.32%. The 30-year note added 30/32, yielding 4.64%.

Tech stocks got a lift from Advanced Micro Devices (AMD Quote), whose shares tacked on 2.6% after the company unveiled its new Barcelona microchip, which it expects to help seize market share from rival Intel (INTC Quote). AMD ended the day with a gain of 33 cents at $12.94.

Separately, Intel raised its third-quarter revenue forecast, saying it should have a top line of $9.4 billion to $9.8 billion, up from its prior outlook of $9.0 billion to $9.6 billion. Shares finished down 12 cents, or 0.5%, at $25.35.