Jimmy Kimmel is looking back at the origins of his talk show and why he used to hope it wouldn’t last.

The Jimmy Kimmel Live! host, 57, shared during an appearance on SiriusXM’s Where Everybody Knows Your Name podcast on Wednesday, Oct. 22, that there was a point in his tenure as a late night host where he felt he “couldn’t do it anymore.”

“I didn’t know what I was doing, and I would pray that they canceled the show sometimes,” Kimmel told host Ted Danson. “I didn’t want to quit because I didn’t want to disappoint all the many people who worked for me, but I couldn’t. I was just, I couldn’t do it anymore.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Jimmy Kimmel Live! premiered on ABC in January 2003 following his co-hosting duties on Comedy Central’s The Man Show, having since run for an impressive two decades and marked its 24th season this year. But as Kimmel shared with Danson, 77, things weren’t easy in the beginning due to a shortage of star power. Specifically, the show often just “didn’t have guests.”

“Now keep in mind this show was on, we’d go on the air live at midnight at 12:05 [a.m.], and there were times where it was 5:30 in the afternoon and we didn’t have guests for that night’s show,” Kimmel said. “And I would just have to pick up the phone and call my friends. And that’s not how you go into a show.”

Kimmel added that he’d often have repeat guests, whether they were then-girlfriend Sarah Silverman or fellow comedians Kathy Griffin and Anthony Anderson. “God bless them, because I needed them. And they were always ready at a moment’s notice to come on,” he said.

Luckily, the show eventually “stabilized” and he “figured out how to do it,” Kimmel said.

Kimmel’s comments come a month after his long-running talk show was briefly pulled following his Sept. 15 comments about the shooting death of right-wing personality Charlie Kirk.

Kimmel, who had previously offered his condolences to the Kirk family on social media, addressed the fatal shooing of the conservative commentator during his monologue: “We hit some new lows over the weekend with the MAGA gang desperately trying to characterize this kid who murdered Charlie Kirk as anything other than one of them and doing everything they can to score political points from it,” Kimmel said. “In between the finger-pointing, there was grieving. On Friday, the White House flew the flags at half staff, which got some criticism, but on a human level, you can see how hard the president is taking this.”

He then showed clips of President Donald Trump being asked about Kirk’s death. In the footage, Trump said he thought he was holding up “very good” before turning his attention to the new ballroom being constructed at the White House. The cameras then cut back to Kimmel, who said Trump was “at the fourth stage of grief, construction.”

Disney’s ABC and its decision to temporarily pull the program prompted reactions from Federal Communications Commission chair Brendan Carr — who applauded the decision by Nexstar Media to preempt the show (along with Sinclair). Kimmel later returned on Sept. 23.

Jimmy Kimmel Live! airs weeknights at 11:35 p.m. ET on ABC.