By Joe Bel BrunoNEW YORK -- From a Jewish youth charity in Boston to major banks as far afield as Zurich, the list of investors who say they were duped in one of Wall Street's biggest Ponzi schemes is growing. Around the world, investors who sunk cash into veteran Wall Street money manager Bernard Madoff's investment pool spent the weekend calculating how much exposure they might have. The 70-year-old Madoff, well respected in the investment community after serving as chairman of the Nasdaq Stock Market, was arrested Thursday in what prosecutors say was a $50 billion scheme to defraud investors. One thing was clear in the fallout from his arrest: The alleged victims span from the super rich, to pensioners and powerful financial institutions, to local charities. Some investors claim they've been wiped out, while others are still likely to come forward. "There were a lot of very sophisticated people who were duped, and that happens a great deal when you've had somebody decide to be unscrupulous," said Harvey Pitt, a former chairman of the Securities and Exchange Commission, a regulator in charge of monitoring investment funds like the one Madoff operated. "It isn't just the big investors," he said. "There's a lot of charitable and foundation money involved in this, which is the real tragedy." Charities across the country are expected to be directly affected by the collapse of Madoff's investment fund. The assets of Bernard L. Madoff Investment Securities were frozen Friday in a deal with federal regulators and a receiver was appointed to manage the firm's financial affairs.