Jimmy Kimmel was a recent guest on Ted Danson’s “Where Everybody Knows Your Name” podcast and revealed that he nearly lost the job of hosting an ABC late-night talk show to none other than Jon Stewart. It was the early 2000s, and Stewart was a few years into his stint as the host of Comedy Central’s “The Daily Show.” ABC was looking for a late-night replacement for Bill Maher, who had been hosting the half-hour political talk show “Politically Incorrect” since it moved to the network in 1997.

“They wanted a traditional late-night talk show in that slot,” Kimmel said (via The Daily Beast). “Jon and I have the same manager, James ‘Baby Doll’ Dixon, and James was about to close this deal for Jon to host the show… [ABC chairman Lloyd Braun] watched my tape, and he was like, ‘I think this might be the guy.’ And he brought the tape to Bob Iger, and Iger said, ‘Yeah, I think this might be the guy.’ It was a very strange thing because [James] was in the difficult position of having to tell Jon, ‘Uh, you’re not going to ABC, but Jimmy is going to ABC.’”

Stewart was already a late-night favorite thanks to “The Daily Show,” while Kimmel was a lot more untested in the space as the host of Comedy Central’s “The Man Show.” As Kimmel quipped, “That was a mistake by the way. They definitely should hire Jon. If I’m in that position, there’s no question I hire John 100 times out of a 100.”

Kimmel was so confused over winning the gig over Stewart that he asked Bob Iger about it. As Kimmel remembered: “I said, ‘What was it, like why—this is quite a leap that you guys made. I was on ‘The Man Show,’ I was doing football picks on Fox NFL Sunday—what was it?’ He goes, ‘Well, you were cheaper.’ And everybody laughed, but I knew he wasn’t kidding.”

“Jimmy Kimmel Live!” debuted in 2003, and Kimmel recalled that the early days were rough. “I think my vision of hell is being forced to watch my first year of shows, because it is just as painful as anything could get for me. It took us a long time to figure it out, and we’re very fortunate to get a long time to figure it out.”

“Somehow we wound up getting good ratings,” he continued. “I still don’t know how that was, but they were good enough to keep us on the air, even though I was causing trouble once every, like two and a half months, some major thing was happening. Something that came out of my mouth, you know, and caused a whole thing. It was just tumultuous.”

Watch Kimmel’s full interview on Danson’s podcast in the video below.