Jimmy Kimmel explained to Stephen Colbert on Tuesday’s edition of The Late Show that he had gone into the bathroom for privacy for the fateful phone call with ABC brass in which he learned that his show had been taken off the air.

It was about 3 p.m., so he was already at the El Capitan Entertainment Centre in Hollywood, as his show had been set to tape at 4:30 p.m. Kimmel described the day and the ones that followed as “an emotional roller coaster.”

“As far as I knew, they didn’t even know I was doing the show previous to this,” Kimmel joked.

Stephen Colbert and guest Jimmy Kimmel backstage at ‘The Late Show’ on Sept. 30.

Then he told the whole story: “So. I’m on the phone with ABC executives, and they say, ‘Listen, we wanna take the temperature down. We’re concerned about what you’re gonna say tonight. And we’ve decided that the best route is to take the show off the air.'”

Kimmel disagreed, but then he lost a vote that was taken.

“So I put my pants back on,” he said, “and I walked out to my office.” After he told the show’s executive producers — including his wife, Molly McNearney — he was “whiter than Jim Gaffigan.”

Kimmel’s voice broke, as he said he had thought it all was over: I’m never coming back on the air.

The staff then had to deal with an audience already in their seats.

On a night in which not only was Kimmel on Colbert’s show, but Colbert also appeared on Jimmy Kimmel Live, which is temporarily shooting on location in Brooklyn, N.Y., the two hosts on the Late Show set waved at a video of themselves waving back from Kimmel’s set on ABC.

The unusual host-guest swap comes the same month that Kimmel’s show was pulled from ABC, following comments he made after the murder of conservative activist Charlie Kirk, in which Kimmel characterized the “MAGA gang” response to the killing as predatory. Disney, the owner of ABC, handed Kimmel an indefinite suspension that ended up lasting six days — and longer in areas that rely on local TV station owners Sinclair Broadcast Group and Nexstar Media Group.

The appearances were particularly significant in the wake of the shake-ups on late-night TV this year, as President Donald Trump’s administration, including the president himself and FCC Chair Brendan Carr have spoken out against various talk-show hosts and openly threatened their jobs.

Sign up for Entertainment Weekly’s free daily newsletter to get breaking news, exclusive first looks, recaps, reviews, interviews with your favorite stars, and more.

CBS announced in July that it would end The Late Show With Stephen Colbert — and The Late Show franchise altogether — when the season ends in May.

In response to that news, Colbert received an outpouring of support from many of his current and former fellow talk-show hosts, including Kimmel, who even took out a billboard in Los Angeles endorsing Colbert (against his own show), for an Emmy. Colbert beat Kimmel’s show and others to win the trophy for Outstanding Talk Series at this month’s ceremony.