The best resiliency I saw
was so robust that the director demonstrated it by literally pulling the plugs on the machines one by one and watching the rebalancing to another site.
I asked what happened if we started at the other end of the farm. We pulled a few plugs and watched as the lauded system fell over and cried (metaphorically).