It was King Charles’ decision to strip his brother, the former Prince Andrew, of all his royal titles, but the monarch had the support of the wider royal family, who were actively involved in ongoing discussions leading up to the announcement.

Buckingham Palace released the official announcement on Oct. 30, stating, “His Majesty has today initiated a formal process to remove the Style, Titles and Honours of Prince Andrew.” The former prince, 65, will now be known simply as Andrew Mountbatten Windsor.

A source, who knows the King, notes of the monarch’s decision, “He would have been really exasperated with it. Andrew had been given so many chances and opportunities to do the right thing, and he would be exasperated that he had to make a difficult decision rather than Andrew dealing with it. He would find it quite painful.”

While it was King Charles’ decision to remove Andrew’s titles and honors, using his Royal Prerogative — which allows him to manage certain royal titles and honors without involving Parliament — he did so with the support of the royal family, including his son and heir, Prince William, 43.

However, a source tells PEOPLE they also “sensed a female touch” in the statement, speculating Queen Camilla and Kate Middleton were part of the decision.

Providing a direct reference to the allegations of sexual misconduct against Andrew — which he has repeatedly denied — and his known connection to convicted sex offender Jeffrey Epstein, the statement concludes, “Their Majesties wish to make clear that their thoughts and utmost sympathies have been, and will remain with, the victims and survivors of any and all forms of abuse.”

“I would think the Queen had a hand in that,” the source speculates. “There would have been a push from Camilla and Catherine, and there was pressure from William, who would not want to inherit the headache.”

Adding, “There was pressure from William, who would not want to inherit the headache. It was a family decision.”

“It was quite strong, and I was quite proud when I heard it,” the source adds.

The Queen, 78, in particular, has long been a patron of causes related to domestic abuse, rape and assault. She’s previously revealed that she was the victim of sexual assault as a teen, wrote a personal letter to French rape survivor Gisèle Pelicot and connected with victims of domestic and sexual violence in a powerful 2024 documentary.

Hetti Barkworth-Nanton, who serves as chair of the charity Refuge, praised the Queen’s conversations with survivors in the documentary, titled Her Majesty The Queen: Behind Closed Doors.

“You can’t put a price on the difference that makes for survivors, because it lifts them up and they keep going. It’s incredible,” she raved. “It helps them to realize that they can hold their head up high, they don’t need to feel shame. That’s all part of repairing.”

“These women have sometimes gone through decades of being made to feel utterly worthless. So just meeting the Queen and talking about what happened is part of the journey. And takes them further along the journey in terms of starting to feel some real self-esteem and worth,” Barkworth-Nanton added.

The family is “definitely looking to the future,” the source says. “This could all be part of that. William and Catherine are moving soon, and all their comments have been about looking to the future.”

After King Charles, 76, was heckled about Andrew’s ties to Epstein during a recent royal outing, it likely became clear to the family that the matter was very much top of mind for the public. Andrew’s ongoing fallout also dominated headlines amid the King and Queen Camilla’s visit to Rome for their first official visit with Pope Leo. And as Prince William is about to travel to Brazil for the Earthshot Prize, one of his signature causes, the family wouldn’t have wanted anything to continue to overshadow the goodwill efforts of senior working family members.

So, the King took action. “He would not be happy with that, but he’s had to do it,” the source notes.

Though King Charles and Andrew are reportedly not close, “[Andrew] is his brother, and he will feel some care and affection and concern for him,” the source adds. “But he wouldn’t have wanted to make that decision. He would rather Andrew dealt with it.”

In addition to losing his royal titles, Andrew is also losing his royal accommodations at Royal Lodge in Windsor. He will be relocating to an unnamed property on the royals’ Sandringham estate. A palace source told PEOPLE that it is unlikely that he will be joined by his ex-wife, Sarah Ferguson, even though the two had been living together at Windsor for many years.

“She is going to be moving out and into a separate home,” a source who knows Fergie told PEOPLE on Oct. 31.

“Contrary to reports, she has never asked for a property or any provision for herself,” the source said. “She will continue to forge an independent life.”

As for the couple’s daughters, Princess Beatrice, 37, and Princess Eugenie, 35, they will retain their royal titles in accordance with King George V’s Letters Patent of 1917, as they are daughters of a son of a sovereign. They both remain princesses and will continue to have the styling of Her Royal Highness.

However, Ailsa Anderson, former press secretary to the late Queen Elizabeth, told PEOPLE recently that Andrew’s royal exile is certain to have a “devastating effect” on the “close-knit” family.

“Even though they keep their titles, the ripple effect will have consequences for the rest of their lives,” she added.