Posts by Fustbariclation

Not a failure of testing - a failure of change enablement

To be fair, this is not really the fault of the automated change system not being tested properly - though that is probably one contributing factor.

It's really a failure of the change remediation not being tested properly.

If you want to move quickly, and accept that failure is a possibility - a luxury not afforded those running nuclear power stations - then you really do need to make sure you have a very effective roll-back solution, that is bulletproof.

That it took them six hours, and a site-visit, to roll back the faulty configuration change, establishes that it was not properly designed and tested.

The moral of the story is that, if you're modifying BGP automatically, you need, first, to design the safety-net, by writing, and testing, code that will reset it all to its last known working state -- reliably, every time.

To fail-fast, you must be able to reset-fast.

Odd that nobody notices that it happens at the busiest time.

Things that go bang at one of the busiest weekends of the year are almost certainly capacity problems - which are difficult to fix.

When you read of staff having been retrenched recently, it makes it even more likely it's a capacity problem.

Experienced, technical staff, with a sound knowledge of the systems, and their history, are expensive and rare. Their value is not obvious as they tend not to be top-flight communicators and are not that keen on blowing their own trumpets either.

The first you notice of their absence is usually just this, a very big bang, that appears to come from nowhere and with nobody knowing how to fix it.

Capacity errors are also notorious for being immune to defences like removing spofs by having multiple data centres.