Markets couldn't commit to a direction Wednesday, teetering between slight gains and losses, and for good reason. Near-tiebreaker poll numbers for the "Brexit" referendum left Wall Street uncertain. 

Stocks fluctuated for much of the day before settling with slight losses. The S&P 500 was off 0.17%, the Dow Jones Industrial Average fell 0.27%, and the Nasdaq slipped 0.22%.

U.K. citizens will head to the polls Thursday to vote on whether to leave the European Union. Current polls suggest odds are around 50-50. An exit from the EU would have significant repercussions for the region's economic and political stability. The result of exit polls will be released early Friday.

"This week's vote in the U.K. on remaining in the EU could have significant implications for both parties," said Matthew Peterson, wealth strategist for LPL Financial. "Polling data suggest that the vote will be very close, though historically, in similar situations people vote to retain the status quo in greater proportions than the polling suggests."

Federal Reserve Chair Janet Yellen continued to sound dovish on interest rates. Similar to comments made after the Fed's June meeting, Yellen noted Wednesday that the U.S. economy was still growing, though it had seen recent weakness. She also said the disappointing headline jobs number in May was likely a result of residual economic weakness from earlier in the year in testimony to the House Financial Services Committee.

A "cautious approach" to rates "remains appropriate," Yellen told the Senate Banking Committee in Washington Tuesday.

Yellen's comments appear to have ruled out another rate hike in the near term. A July rate hike has a probability of 12%, a September hike 28%, and a November hike 31%, according to CME Group fed funds futures.

"The overall cautious message is still consistent with our view that there will be only one rate hike this year, most likely in December," Societe Generale analysts wrote in a note. "Meanwhile, the focus of the markets remains firmly hinged to the EU referendum as the big day is just one day away now."

Crude oil gave up earlier gains after weekly inventories data showed a far smaller decline than analysts had expected. Oil stocks fell by just 900,000 barrels over the past week, according to the Energy Information Administration. Analysts had expected a decline of 1.7 million barrels.

West Texas Intermediate crude oil was down 1.4% to $49.13 a barrel Wednesday.

The housing sector continued to outpace the broader economy, with existing-home sales in May rising to their highest level since February 2007 and prices notching their best level ever. Existing-home sales rose 1.8% to a seasonally adjusted 5.53 million, though slightly below estimates of 5.55 million. Tighter inventory, higher demand and lower interest rates have boosted activity in the sector.

McDonald's (MCD - Get Report) dragged on the Dow after being downgraded to neutral from buy with a $129 price target at Nomura. The firm said comp sales estimates are moving below consensus.

Tesla Motors (TSLA - Get Report) made a surprise stock offer of up to $2.8 billion for struggling solar energy company SolarCity (SCTY) in a move to combine CEO and entrepreneur Elon Musk's companies. The automaker offered $26.50 to $28.50 a share for SolarCity, a 21% premium to Tuesday's close. 

Tesla's acquisition offer is "shameful," Kynikos Associates President Jim Chanos told CNBC's Scott Wapner. "The brazen Tesla bailout of SolarCity is a shameful example of corporate governance at its worst."