Madonna left Malawi early Sunday after her failed adoption bid for a 3-year-old girl named Mercy James, who was preparing to return to the orphanage where the singer found her.

Madonna's plans received a massive setback Friday when a judge rejected her bid to adopt. The singer, 50, has since filed an appeal against the ruling.

"We are ready with our grounds challenging [High Court judge] Chombo's ruling," Madonna’s Malawian lawyer Alan Chinula told PEOPLE Sunday.

A staffer at the singer's Raising Malawi charity said Anne Chikhwaza, Director of the Kondanani Children's Village, drove to Kumbali Lodge, where the singer had been staying, Sunday morning to collect Mercy to take her back to the orphanage.

"Madonna spent some time with Mercy; she looked sad but Lourdes, who had clicked with the little girl, looked devastated," said the staffer.

Madonna is believed to be headed for London to celebrate Passover, which would allow her children to reunite with the singer's ex-husband, Guy Ritchie.

Ritchie, for his part, has shown his support for Madonna, issuing a statement Friday following the singer’s adoption setback praising her as a mother and showing his heartfelt support in her adoption case.