NEW YORK ( TheStreet) -- Investors are beginning to pull back from the housing market as rising home prices and increasing competition diminish returns. According to the latest Campbell/Inside Mortgage Finance HousingPulse Tracking Survey, investor share of home purchases dropped from 22% in April to 20% in May based on a three-month moving average -- the sharpest drop in investor activity in three years. The Investor Traffic Index, an indicator of future home purchase activity, was also down for three months in a row ending in May. Meanwhile, first-time home buyers and current homeowners accounted for a greater share of home purchases in May at 36% and 43.8% respectively. Investors have in recent years bought foreclosed homes at steep discounts, refurbished them and "flipped" them for a profit. Others have tried to capitalize on growing rental demand and converted single-family homes into rentals, earning an attractive yield on their investment. The business, traditionally the mainstay of mom and pop investors, has attracted large institutional investors recently. Over the past 18 months, firms including Blackstone ( BX) and Colony Financial ( CLNY) have bought thousands of single-family homes with the purpose of converting them into rentals. Recently, some players have started to exit. As early as last year, hedge fund Och Ziff ( OZ) pulled out of the business. More recently, Carrington Holding, another early entrant, also exited the market. The survey found that investor purchases of foreclosed properties or REO (real estate owned) fell significantly, with survey respondents citing lower returns from rising prices. Rising demand and limited supply of homes on the market has meant that foreclosures no longer sell at the deep discounts they used to. The survey found a correlation between the decline in sales-to-list price ratios for foreclosed homes and the drop in investor interest. Investors are also shying away from short sales. In a short sale, the bank agrees to let the homeowner sell his house for a value less than the mortgage and forgives the difference. Short sales take longer than conventional sales to close because banks have to approve the deal.