ATLANTA -- The moment many remember as a highlight of the 1996 Summer Olympic Games in Atlanta almost didn’t happen.

Muhammad Ali’s lighting of the Olympic torch at the opening ceremony is an iconic memory of the Centennial games. The man behind the moment was Olympic chairman Billy Payne. But he remembers having second thoughts.

“I knew about Muhammed Ali and his conscientious objection during the Vietnam War,” said Payne.” I thought that was kind of a negative.”

PHOTOS: Muhammad Ali at the 1996 Olympics in Atlanta

NBC’s Olympics executive producer Dick Ebersol would eventually convince Payne that he had made the right decision. But they had to keep the decision secret.

“I think several months later, we invited him down to rehearse at 3 in the morning,” Payne said. “By then, everybody was really trying to find out who it was going to be. So people staked out the stadium and everything. I remember we had all the police do the perimeter and make sure nobody was there with cameras. We had Ali under a blanket, put him in the car and we drove him down there to rehearse and told him what was going to happen. Janet Evans was going to be running up the ramp and light his torch and what he was supposed to do. He kept dropping it. Oh man, I was really worried. He just kept dropping it…We instructed [Janet Evans] and told her that he had difficulty, to pick it up and do it together. That was a no-lose proposition. Either way, that would’ve been beautiful and emotional.”

It’s a moment everyone saw still remembers.

“When Muhammad Ali stepped out on that platform, we said of course,” said Olympian Dan O’Brien. “Muhammad Ali, the greatest of all. Just thinking about it now, it still gives me chills.”

Watch 11Alive's 'Remembering the 96 Olympics' special August 4 at 9PM!