Improve backfill robustness by trying more servers when we get a `4xx` error back.