Dragons and butterflies: The chaos of other people's clouds

This is one side of the coin

This is the conventional one of considering every part of the system is committed to "doing the best" for the resilience and well-being of the overall system. If that requires cutting resources out of the equation, then so be it.

The other side of the coin however, is that there are people out there who will be trying to bring down the system. They will be looking at injecting stimuli into the system intended to cause an imbalance, bringing the whole system down.

DoS is a brute force means to do that, but here we are talking an extra level of sophistication: tweaking things in a low-volume way that amplifies to make it look as if a failure is imminent - the system recognising this as a cascade situation and taking what it thinks is a necessary action.

A good analogy for this behaviour is systems used in the Financial Markets - I believe there have been instances where people have gamed the stock market to cause artificial price crashes. The counter to that is to bring in random elements into the mix, so that there is no hard trigger-point for defensive action to be taken. I feel that there is no iron-clad way to cater for all eventualities.

It hard to make the case to management to spend the moolah...

This is all obvious stuff, and while I love to point out Netflix and it's Chaos Army of testing tools, it's just not valid for where I work. One, management doesn't want to spend money on anything that might (not will, might) save their bacon. Unles they've had their feet in the fire, then they won't spend it.

Two, for large Oracle installs, unless you get RAC or some other way to horizonatally scale your ERP system, it's just a nightmware to make work. The vendors don't support it out of the box, or if they do, it's so much $$$ that management just says no. Until the day something craps out and they can't book orders, then the spigots open for a small interval.

Three, getting management permission to *test* your DR, backups, etc is all a fairy tale. They either want it tested without any impact tothe business, or the test is so laughably basic and error prone that it's not even worth it.

Raise your hands if management considers the development copy of production to be thier DR copy? I see a forest cropping up. And of course the problem is that things *usually* work just fine, and generally we keep things going. Even when we know we're on the knife edge of falling off the cliff.

It's not an easy problem at all. And it costs money. Lots of it usually. And sometimes its not worth it. But it usually is....

"built in redundancy at the application layer because that's best practice"

No, Netflix created their software and built in redundancy because upper management had made the decision to put the money in it, probably because they decided, after analysis, that it would cost them more in the long run if they had a service that fell over every other week.

Best practice is hardly the reason for doing things when the accountants hold the purse strings.

Oh, BTW: I like the term 'cloutage'.

Might as well coin one now, as we will be needing one anyway. Quite often.

"Oh, BTW: I like the term 'cloutage'."

For some reason I read that & think "clownage".

Cloutage

Already seen used before as a measure of how much clout or impact something or someone has!

Clotage may be better - cloud driven by/adopted by unthinking Clots (in too many cases) then having to live with unthought-of consequences when all goes TITSUP!

If you don't want to be at home for Mr. Cock Up you'll need a cunning plan.