Okay, so reading the manual for Ceph tells me to mark an OSD as "Out" and then allow the pool to rebalance the data away from it before removing it, and then removing it should not require any rebalancing. Okay, so I did this. I marked an OSD as OUT. I added two OSDs to replace it. I let the rebalance run. For two days, I let it run, until everything was 100% active+clean, the new OSDs were up and in, the old OSD was up and OUT. I deleted the old OSD using the GUI. Immediately another...
Read more
Read more