Prince Andrew could be set to move to the former royal home of Meghan Markle and Prince Harry.

The disgraced royal, 65, has agreed to exit the 30-room Royal Lodge, it was reported on Oct. 27, with Frogmore Cottage, on the Windsor Castle estate, at the top of the list for his likely next home.

Under the deal, Andrew’s ex-wife, Sarah Ferguson, who has been living alongside him at the massive mansion in Windsor Great Park despite their 1996 divorce, could move into Adelaide Cottage, which Prince William and Kate Middleton are about to vacate. The proposed move was reported in The Sun newspaper on Oct. 27.

“Frogmore Cottage is too small for both and Adelaide Cottage is around the corner so they will see each other whenever they want,” a source told The Sun.

“Talks are still ongoing,” a source close to proceedings tells PEOPLE.

William and Kate and their children, Prince George, 12, Princess Charlotte, 10, and Prince Louis, 7, are moving about three miles away to Forest Lodge, in Windsor Great Park, a place that is being called their “forever home” very soon. The family are hoping to be in by the time the children go back to Lambrook School on Nov. 3.

Buckingham Palace and Sarah Ferguson’s office declined PEOPLE’s request for comment.

The option of a home each for Andrew and Sarah has been on the table for some time and came from Buckingham Palace, one source told PEOPLE.

There have been days of intense negotiations over demands that Prince Andrew leave Royal Lodge, where he has been living since 2003. It was reported towards the end of last week that a deal was about to be struck.

Andrew has been under intense pressure to move from Royal Lodge, which was heightened when the U.K. newspaper The Times published a redacted version of his lease, showing that he was entitled to live at Royal Lodge on a peppercorn rent, after he paid around £8 million (about $10.6 million) for the lease and the refurbishments that were then carried out. A “peppercorn rent” is a term referring to a very small amount of rent, sometimes as small as a single peppercorn.

Harry and Meghan were given Frogmore Cottage by the late Queen Elizabeth. They moved into it in 2019, shortly before the birth of their first child, Prince Archie, and hoped it would be a long-term marital home. But they were asked to vacate it in 2023 after their decision to relocate to America and step down as senior working members of the royal family.

Prince William and Princess Kate, meanwhile, have resided in Adelaide Cottage for a little over three years. While it provided them with a perfect place to start their family life in Windsor, it was also a place that held some unhappy memories. Soon after they moved there, Queen Elizabeth died, and then, in 2024, they faced the dual pressures and challenges of Kate and King Charles’ cancer diagnoses. Princess Kate announced in January that she is in remission, while King Charles is still undergoing regular treatment.

There had been talk of Prince Andrew leaving for one of the vast royal estates in Scotland or Norfolk. But Andrew has reportedly said he wants to be as close as possible to his two adult daughters, Princess Beatrice and Princess Eugenie, who have homes in London and nearby in the southeast of England.

The proposal of Frogmore and Adelaide cottages would be an answer for Andrew, who has been under fire about his residence at Royal Lodge for more than a year. Frogmore is within the security cordon of the castle and would not cost the British taxpayers anything.

Charles and his team had proposed Andrew leaving Royal Lodge more than a year ago and when the stubborn Prince dug in his heels, citing the long-term lease that lasts until 2078, it was dubbed by some insiders as the “siege of Royal Lodge.”

There has been renewed scrutiny over Prince Andrew’s accommodation, as Royal Lodge is leased from the Crown Estate, since he stepped back from royal work following his connections to Jeffrey Epstein, who died in jail in 2019 while awaiting trial on federal conspiracy and sex trafficking charges.

He stepped back from his public royal role in 2019 following his bombshell BBC interview in which he discussed his relationship with Epstein.

Can’t get enough of PEOPLE’s Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Talks over Andrew’s living arrangements have intensified amid renewed interest in his ties to Epstein.

Prince Andrew said in a statement released on Oct. 17, “In discussion with The King, and my immediate and wider family, we have concluded the continued accusations about me distract from the work of His Majesty and the Royal Family. I have decided, as I always have, to put my duty to my family and country first. I stand by my decision five years ago to stand back from public life.”

“With His Majesty’s agreement, we feel I must now go a step further. I will therefore no longer use my title or the honours which have been conferred upon me,” he added. “As I have said previously, I vigorously deny the accusations against me.”