Amid the Smoke, Repacking Wall Street's Data Pipe

In the haze of Sept. 11, 7 World Trade Center seemed little more than a footnote.

On a day when thousands of innocent mothers, fathers, sons and daughters had already died in the Twin Towers, there were no casualities when this 47-story neighbor finally succumbed to fire several hours later. But it was this less-noticed event that almost took down the New York Stock Exchange.

And though it was Big Board Chairman Richard Grasso who became the hero of Wall Street's recovery effort, none of it would have been possible if not for the efforts of 51-year-old Carl Russo, Verizon's (VZ) head of Lower Manhattan operations.

Unnoticed at the time, Russo and his Verizon team became the linchpin of perhaps the highest-stakes rebuilding effort immediately following the atrocities -- the preparation for reopening the world's largest stock exchange. When it rang as scheduled the following Monday, the NYSE's opening bell sounded a note of America's fierce resolve to fight back from its darkest day.

But when 7 WTC collapsed on the evening of Sept. 11, the prospect of success in the $1.4 billion repair project couldn't have seemed more remote. The calamity knocked out a telephone network hub that handled as much communications traffic as a city the size of Cincinnati. In doing so it short-circuited a vital swath of the city's telecommunications network, leaving much of Wall Street in virtual silence. And for all the talk of bulls and bears, it is the telecom network that makes up the very core of the country's financial markets' machinery.

The task of repairing the telecommunications meltdown would fall to Russo. Even as emergency workers began their frantic search for survivors, he was rallying his troops to tackle what looked like a nearly impossible task.

"The common feeling was, you aren't going to keep us down," said Russo. "Most everyone was angry. But it felt good -- we knew the job that had to be done and we knew what we were up against."

Idled

In some ways Russo was the ideal man for the task, to listen to co-workers who know him well. Colleagues describe him as a "solid" leader who favors khakis and sports shirts to business suits, and an excellent technician. He's a phone business veteran who, in contrast with the much-derided "Bell heads" who are said to populate middle management, is as fluent in the language of computer equipment as he is with phone gear.

There was no underestimating the size of the challenge. Although four other Verizon switching stations also served downtown businesses and residences, West Street handled about 20% of Wall Street's communications traffic. About 300,000 phone lines and 4.5 million data lines ran through the West Street hub. More importantly, the crippled building also held the critical servers -- a collection of computers that make up the brains of the network for Wall Street and the NYSE -- which had been idled since the attack.

After 48 hours of setting up communications for government emergency service command centers and hospitals, Russo received a new set of orders: Restore all phone and data service to the NYSE.

"It was pretty clear to everyone that getting the stock exchange going would get the country moving, then that would have ripple effects to other countries that trade with us," said Russo, recently looking back at the implications of his assignment.

Spotlight

Our Sept. 11 Home Page

Editor's Note: TheStreet.com Revisits Sept. 11

The Making of a Hawk by James J. Cramer

What We Saw the Day Time Stood Still

Investors Will Lose at Patriot Games

Amid the Smoke, Repacking Wall Street's Data Pipe

Document Chaos Isn't Sorted Out

Battle Against Terrorism Boosts Defense Sector

Faint Glow Alights on a Once-Ashen Wall Street

Disaster Recovery Needs Didn't Stop Storage's Slide

Security Software Gets Mind Share, but Not Sales

Lodging Woes Linger in Troubled Times

Market's Terror Trend Plays Out Predictably

Bankrupt Ricochet Rises Like a Phoenix After Sept. 11

Airline Woes Preceded Sept. 11 and Will Remain

Wall Street Shocked Into Exodus

While Russo's objective was simple, the task was anything but.

His orders came down through a chain of command that company officials claim started at the highest levels of the government. Though no one recalls exactly who gave the marching orders, Russo said he was told in no uncertain terms to have the world's most watched stock exchange ready for trading on Monday, Sept. 17. That meant he had just four days to rebuild a new network operations center.

Though the NYSE building itself was unharmed -- located a safe four-block zigzag from the World Trade Center -- suddenly more than ever it was seen as the fractured epicenter of U.S. commerce. For Russo, the pressure to get the NYSE reopened wasn't just that his boss CEO Ivan Seidenberg made it the company's top priority. There was also the little matter of the nation's financial well-being literally and symbolically hanging in the balance.

As his first move, Russo made a list on his office wall of what he called "show stoppers." Five basic steps that each had to get done to get service back. Under each category such as recreating a network management system, re-establishing a viable local network, reconnecting the local network to pathways in and out of the area, there was a roster of subtasks that had to be tackled.

"There was lots of outside interference from customers, vendors and internal organizations" that Russo said were vying for his crew's attention. "I had to keep them focused, keep their thoughts fixed on what had to be accomplished. We had to set priorities and say 'no' to people.

"You just had to go out and say, 'This is what you have to do for me,'" added Russo. "It wasn't your normal protocol -- it was just, 'This has to be done and this is how we're going to do it.'"

One example Russo cited was the logistical problem of getting a bunch of critical networking gear shipped down from Canada. With the airlines grounded and the country's borders locked down, Russo said he had to get the Canadian government involved with the shipment. Despite traffic backups at the border extending beyond the horizon, the Canadian officials managed to get the truckload of equipment to the New York state line, where it was then given a high-speed escort to the city.

Serve and Volley

Even more important than the Canadian shipment were the network servers still inside the West Street equipment office. As fate would have it, the 12 network servers were housed on the 23rd floor of darkened 140 West St. And being one of the network's most vital elements, recovering the servers was the critical to getting the NYSE back on its feet.

Russo had a group of Verizon managers don breathing apparatuses and flashlights to go fetch the dozen pizza-box-shaped servers and bring them to the new network operations center, a 20-foot-by-30-foot room near Russo's Pearl Street office, on the other side of Lower Manhattan.

"Obviously, we knew everyone was watching, which added to the pressure," said Verizon COO Larry Babbio. "There was no certainty that we would be able to get the stock market up and running by Monday."

Russo and his immediate eight-person crew were working 20-hour days and sleeping in their offices as they patched together the makeshift network management center and checked off the tasks on their priority list.

Looking back on it, Russo says he's still amazed that they were able to accomplish as much as they did. "It's not like this was an outage or a fire or a flood, this was an act of war we were trying to recover from," says Russo. "If we can beat this we can beat anything, is how we felt."

The managers and technicians were still securing circuits and retesting their systems at 6:30 Monday morning. Verizon estimates the post-attack repair bill is around $1.4 billion. The company had a $1 billion insurance policy and expects government and assorted relief funds to kick in a good part of the difference.

Russo recalls the anxiety leading up to the market's opening at 9:30 a.m., having no clue what would happen once the traffic went "live."

"Once we could see the nodes light up, it was like the weight of the world was off our backs," Russo said.

Despite the heated trading, the network performed without any significant hitches. The NYSE recorded 2.6 billion transactions, at that time a one-day volume record. It was less of a banner day for the Dow Jones Industrial Average, whose 684-point drop marked the largest single-day selloff at the time.

That was all fine with Russo, a father of four, who was just happy to go home for the first time in a week.