Fragile magnetism in rare earth XY pyrochlores