Depending on the tool is not sufficient to avoid issues. Partitioning of the datacenter such that any one operation only had a potential scope of impact of, say, 1/3rd of the datacenter is a better approach. That can be accomplished through bastion hosts/gateways. It means that the admin has to log in to a unique gateway for every partition. That 2-step process is often enough of a sanity check to avoid these types of issues entirely.
Posts by kbsartain
4 publicly visible posts • joined 11 Dec 2009
Fat-fingered admin downs entire Joyent data center
Sysadmins hail Windows Server 2012 R2's killer ... clipboard?
Microsoft is developing its own cloud storage O/S
Friday 11th December 2009 16:02 GMT
Azure Storage makes 3 copies of all data
In the Azure documentation, they claim to make 3 synchronous writes to different storage arrays for each write i/o. Presumably, that is in a software driver stack implementation, and i would guess that it's networked storage of some sort. That would allow very cheap storage, possibly without the need for true RAID or expensive 3rd party storage vendors.