"Let me tell you about the very rich. They are different from you and me." F. Scott Fitzgerald wrote in his 1924 short story, The Rich Boy.

For generations, family offices were the very private sanctuaries of ultra-rich families assigned the task of protecting their wealth, investing dividends, shielding it from taxes and pretty much making sure the housekeeping staff was paid on time.

Today family offices blur the line between that old school institution, private banks, wealth management firms, private equity and hedge funds. They are still the dominion of the uber-wealthy, but family offices are no longer content to hand the reins over to asset managers and have the family read the quarterly financials.

They are more hands-on now, sometimes looking to sidestep the traditional fees paid to private equity funds. Instead, they are starting to skip over the middle man, taking a page from financial sponsors and investing direct now. Some family offices are beginning to hire away talent from PE firms to handle due diligence work, deal access, and transaction structuring concerns.

"We are seeing where families either have direct investment programs in place or they are creating the programs," according to Anna Maria Nekoranec, CEO of Align Private Capital, a firm that works with about 130 family offices.

This is somewhat of a revolutionary concept, but it's gaining traction. Symphony Technology Group began in 2002 as a family office for the money of entrepreneur Romesh Wadhwani, who founded Aspect Development Inc. and sold it to i2 Technologies Inc. for around $10 billion in 2000. Now Symphony has a portfolio of companies that altogether generate $2.5 billion in revenue.

Even Rockefeller & Co. isn't the same. The firm has traditionally been credited with starting the first family office with the fortune that John Rockefeller made with Standard Oil. Rockefeller & Co. was established to be the watchdog for the family fortune, but today it runs money for individuals and families, family offices, foundations, endowments and institutions and has $15.1 billion under advisement.

In fact, the first family office may have been formed in London by American merchant George Peabody & Co. in 1851. The office was the forerunner to J.S. Morgan & Co. which would later become J.P. Morgan (JPM - Get Report) . Family offices are an important sector in the U.K. economy.

Today, particularly in the U.S., income inequality is much more than a talking point from a Bernie Sanders YouTube video. The number of family offices has been increasing in the last few years and now number someplace between 4,000 and 15,000 on a global scale, depending on who is doing the estimating, with those numbers including both single family offices and multi-family offices. The average office has $918 million in net worth and $400 million in investable assets, according to I Capital Networks, a New York-based financial technology platform that works with family offices.

Even more stark is a statistic from Family Capital, a trade publication focused on family offices which reports that these vehicles hold $2.4 trillion in investable capital.

That number is large enough to draw the interest of private equity powerhouses such as Blackstone Group  (BX - Get Report) , KKR & Co.  (KKR - Get Report) and Carlyle Group  (CG - Get Report) for a trio of different reasons. Those investment firms have beefed up the units that cater to private investors in the hopes of signing up some of the family offices for services they may choose to outsource, such as tax consulting or due diligence services.

The big equity players also don't want to lose a potent source of capital for their funds business. Last year, for example, Blackstone stated it attributed $43 billion of the $310 billion assets under management to private capital, including family offices.

While private equity firms are actively courting family offices, more than a few family offices are distancing themselves from PE players. In 2014, a survey of family offices by Chicago-based McNally Capital showed that 77% of them favored direct private equity investments over investments in private equity funds.

Family offices are even partnering with each other on deals, according to Align Private Capital's Nekoranec. 

Nekoranec points to Blue Water Worldwide, a New York-based holding company, as an example. Blue Water seeks out deals in markets or geographies that have a degree of distress. It will source the deal and partner with family offices to make purchases of companies. Blue Water tries to bring in family offices that have had success in the same sector as the company it's targeting for an acquisition.

The trend toward family offices making more investments that resemble traditional private equity transactions grows in part from family offices favoring more control over their investments.

"A general theme that came out of the collapse of 2008 is that family offices want more control over what they are invested in and a better understanding of the investments transparency is a factor," said Marv Pollack, the managing director of marketing and strategy for the Family Office Exchange, a network of family offices and their advisers.

Pollack said that some family offices have also begun hiring staff so that the offices are better equipped to evaluate investment opportunities and do direct investments.

In the past, if family offices wanted to make private equity moves, they relied upon PE funds because they had existing infrastructure which made them better qualified to fight for deals than many family offices. Now, some family offices have begun competing on a small scale by hiring private equity talent.

One example of this is Pritzker Group, a Chicago-based family office that sprang from the Pritzker family and the fortune they made through co-founding Hyatt Hotels  (H - Get Report) . It hired Michael Dal Bello away from Blackstone and Michael Lynch from Cardinal Health (CAH - Get Report) .

Another example is Grand Crossing Partners, a family office founded by the family that created Wilton Brands. Grand Crossing hired Brian Jacobsen, a former partner at TowerBrook Capital Partners, as a managing partner.

An advantage that family offices have over private equity in chasing transactions in sectors where they have enjoyed success has to do with deal access.

Both private equity firms and hedge funds are subject to certain regulations and Securities and Exchange Commission oversight. But family offices enjoy a light touch thanks in no small part to extensive lobbying when Dodd-Frank was being considered. In June 2011, the SEC approved a working definition of family offices that specifies that the entity manages the wealth of family members, and provides other services.

The definition and image of a family office has been put to the test, in part, because of the number of hedge funds that have returned outside investment capital and morphed into family offices.

In recent years, for example, George Soros transitioned Quantum Endowment Fund into Soros Fund Management; Steven Cohen shut SAC Capital down with the help of SEC enforcement and began Point72 Asset Management. Carl Icahn has Icahn Capital Management; Stanley Druckenmiller was ahead of the curve when he transitioned Duquesne Capital to a family office in 2010. A relatively new family office is 1988 Management LLC, a vehicle created by conservative icons Charles and David Koch, of Koch Industries.

It is clear that in some ways family offices are competing with private equity firms, but in other ways they are diverging from PE funds. Family offices can pursue transactions in a wider variety of circumstances than traditional PE investment vehicles. The latest example: cannabis-related companies.

Mainstream players have steered clear of cannabis companies because of federal drug laws which make sale and possession of marijuana a crime. Those laws also continue to classify the drug on the same schedule as cocaine and heroin. While Seattle private equity firm Privateer Holdings has had some success with its cannabis portfolio companies, it is focused on the legal medical marijuana business and bodycare accessory brands such as Marley Natural and has veered away from U.S. companies that are involved in growing or distributing the plant.

On the other hand, Viridian Capital Advisors, a New York financial advisory firm focused on the cannabis industry, said in a 2015 report that family offices have begun investing, but investments by mainstream private equity players were limited to portfolio managers making personal investments.

Would Rockefeller & Co. invest directly in a cannabis company? No one knows. But there isn't anything holding it back. And that's a freedom many PE firms envy.