The U.S. unemployment rate can be a tricky number to interpret, especially as the declining figure -- which recently returned to pre-recession lows -- often belies the departure of many Americans from the workforce.

And it appears no companies are hit as badly by the rise of these so-called discouraged workers, who for various reasons have stopped looking for full-time jobs, as the big U.S. staffing companies.

Shares of Robert Half (RHI) - Get Report , ManpowerGroup (MAN) - Get Report and TrueBlue (TBI) - Get Report  fell by 3.5%, 2.7% and 3.4%, respectively, on Friday following a dismal jobs report from the Bureau of Labor Statistics.

This offers a peek into an increasingly diminishing group of labor-force participants, especially temporary workers, actively seeking work -- in other words, a demographic that serves as the bread and butter of the U.S. jobs placement industry.

"The staffing companies are being hurt specifically because of the drop in temporary staffing workers," Mark Marcon, a senior research analyst with Robert Baird specializing in human capital, said in a phone interview Friday.

Temporary-help employment is down by 64,000 so far this year, with 21,000 jobs lost in May alone, the BLS reported Friday, highlighting a drop in total unemployment to 4.7% -- what many consider full employment.

"If we are seeing peak jobs, the unemployment rate is bottoming and it makes sense that the stocks are selling off," Real Money's Doug Kass said in an email Friday.

And temporary jobs are the primary source of revenue for U.S. staffing companies. For instance, Robert Half pulled more than $1 billion in the first quarter tied to temporary and consulting placements vs. $106 million for permanent placements. Meanwhile, Robert Half maintains that a reduction in the labor force and employment can ultimately benefit shareholders, as employers begin to line up at specialized staffers to discover talent in a shrinking pool of candidates.

"A number of professional occupations are nearing full employment, which is placing pressure on the supply of available talent and increasing our value to clients," the company said in its annual 10-K filing with the SEC.

And, according to data compiled by Real Money's Aidan Dougherty, there has been an inverse trend between Robert Half's share price and the unemployment rate, prompting shares to shift off their trendlines in the opposite direction.

And over the long run, the correlation appears to be true, with Robert Half shares climbing steadily alongside a persistent drop in the jobless rate. The company's shares have fallen 14% on the year, however, as some analysts warn of a potential cool-down in economic growth.

Avondale Partners analyst Randall Reece most recently cut Robert Half's rating to Market Perform from Buy last month, citing "modest earnings growth potential."

As well as Robert Half shares, domestic staffing companies TrueBlue and ManpowerGroup are down 24% and 7% so far this year, respectively.

The drop in temporary workers is largely the result of a cool-down in GDP growth, as employers generally first turn to temporary workers who are easier to add and subtract to a firm's workforce, Marcon said. There's also been a recent trend staffers are noticing in which many Americans are becoming increasingly selective about which companies and positions they will agree to take for employment, Marcon added.

"It's kind of a weird brew in which some employers have pulled back on the number of employees they are bringing on staff, and there's also a dynamic of pickiness over where some people are willing to work," he said.

Meanwhile, Jim Borbely, an economist with the BLS, said in a phone interview with Real Money that the drop in the unemployment rate was mostly due to the declining labor-force participation rate, noting the agency usually sees about 700,000 to 800,000 people entering the market between April and May. In its Friday report, however, the BLS said the participation rate dropped 0.2% in May to 62.6%, or 90.4 million people. The rate has fallen 0.4% over the past two months, erasing first-quarter gains.

Shares of Robert Half, the leading staffer in the U.S. for finance and accounting positions, fell 18% last year as the unemployment rate declined to 5% by year's end from 5.6% in December 2014. But Robert Half argued in its annual 10-K filing that reduced unemployment could benefit shareholders as the talent pool for available workers shrinks, incentivizing companies to tailor their search with staffing firms.

"A number of professional occupations are nearing full employment, which is placing pressure on the supply of available talent and increasing our value to clients," the company said in its annual filing with the SEC. "The secular demand for temporary staffing is also ongoing. The use of flexible workers matched an all-time high during 2015, and temporary employees represented 2.06% of the U.S. workforce as of Dec. 31, 2015."

But investors may be overreacting to Friday's selloff, especially because people tend to get "overly exuberant or overly pessimistic" to BLS figures that will "invariably be revised," Marcon said.

The decline in labor-force participation is "not a healthy sign," Edgeworth Economics partner Stephen Bronars said in a phone interview Friday, noting that there was "a lot of noise" in the monthly report and that investors would be wise to look at the long-term trend.

The labor-force participation rate, Bronars says, "will probably be trending down year to year until some of the millennials become a more important component of the population," helping to replace a retiring demographic of baby boomers.

-- Anders Keitz and Aidan Dougherty contributed to this report.