Wall Street gave back Wednesday's rally and then some on Thursday as investors grew nervous about the impending first-quarter earnings season.

The S&P 500 was down 1.2%, the Dow Jones Industrial Average fell 1.0%, and the Nasdaq slid 1.5%. The S&P 500 had jumped more than 1% on Wednesday on a dovish Federal Reserve and massive surge in crude prices. 

The first-quarter earnings season will kick off in earnest on Monday afternoon when unofficial bell-ringer Alcoa(AA) - Get Report reports. Uncertainty over how major companies fared over the quarter kept bulls on the sidelines and pushed bears to sell. 

"As we start to kick in to earnings season there's going to be anxiety," Matt Kaufler, portfolio manager at Federated Investors, told TheStreet. "There's a general skepticism that [earnings] are perhaps steady but they'll be unenthusiastic. Steady but uninspiring."

The early prognosis on companies' quarterly performances doesn't look good. S&P 500 earnings are expected to fall 7.9% in the first quarter, their third straight quarter in decline and their losing streak worst since mid-2009. Excluding the energy sector, earnings are forecast to fall 3.6%. 

Crude oil was also causing jitters on Wall Street as investors looked ahead to next week's meeting of the Organization of Petroleum Exporting Countries and whether an agreement on a production freeze can be reached. Crude enjoyed its best day in around three weeks on Wednesday after a weekly read on domestic inventories showed a surprise decline. On Thursday, West Texas Intermediate crude oil closed 1.3% lower at $37.26 a barrel.

The energy sector was the worst performer Thursday. Major oilers Exxon Mobil(XOM) - Get Report , PetroChina(PTR) - Get Report , Royal Dutch Shell (RDS.A) and Total(TOT) - Get Report were lower, while the Energy Select Sector SPDR ETF (XLE) - Get Report fell 1.1%.

The number of new claims for unemployment benefits declined by 9,000 to 267,000 in the week ended April 2, according to the Labor Department. Analysts had expected a slightly weaker reading of 268,000. The four-week average, a less volatile measure, ticked 3,500 higher to 266,750.

Minutes from the March 16 Federal Reserve meeting showed members debated an April hike but erred on the side of caution as they left interest rates unchanged and lowered the outlook for increases this year to two from four.

Fed funds futures still indicate that only one hike this year -- in September -- has odds greater than 50%, according to CME Group. An April rate hike is currently priced in at a 3% chance, while a June hike has an 18% probability. An April rate hike had a 5% chance at the beginning of the week.

Tesla(TSLA) - Get Report fell 2.9% even after announcing it had reached 325,000 in pre-orders for its latest vehicle, the Model 3. The vehicle reservations amount to $14 billion in implied future sales. Tesla unveiled the new alternative-energy automobile last week.

Yahoo! (YHOO) briefly spiked on reports Verizon(VZ) - Get Report is moving forward with a bid for its core internet business. Bloomberg reports AOL CEO Tim Armstrong would likely take charge of Yahoo! if it became a Verizon property.

Yahoo! (YHOO) fell 2.5% after estimating that 2016 revenue could drop by around 15%, Re/code reported. Earnings could fall to $750 million in 2016 from $1 billion in 2015.

Valeant Pharmaceuticals (VRX) added 2.6% after securing a commitment from loan holders to change the terms of its debt. The embattled drugmaker enjoyed its best two days of trading on Tuesday and Wednesday in 20 years after Bill Ackman of Pershing Square reaffirmed his confidence in the company's recovery.