IT'S ALIVE! IT'S ALIVE! Google's secretive Omega tech just like LIVING thing One of Google's most advanced data center systems behaves more like a living thing than a tightly controlled provisioning system. This has huge implications for how large clusters of IT resources are going to be managed in the future. "Emergent" behaviors have been appearing in prototypes of Google's Omega cluster management …
Essentially, the take-aways are that:
1. Google's 'Omega' system is non-linear and dynamical.
2. At Google-scale, the behaviour of non-linear systems can no longer be reliably predicted by a linear approximation.
3. Despite the unpredictability of such a system, the automatic management of jobs is still advantageous as it allows a much better utilisation of the available resources.
Or, more condensed: at large scales, unpredictable automation is still more efficient than more predictable, but more manual, processes.
What confuses me about the article, however, is that several times it states that the unpredictable nature of the system is "a good thing" but no word of how the "emergent behaviours" of the Omega system are providing useful features or functions that would not be possible with a more predictable system. So far as I can tell from the article, the only reason offered as to why the unpredictability is beneficial is that having a sub-optimal system makes apps better able to cope with that sub-optimal system.
This appears to be the position of the author rather than Google as their quotes strongly imply that a more predictable system would be technically better but at such scale it would also be prohibitively expensive and thus raise the cost for end users. In other words, if the Omega scheduler could be more predictable for the same cost then they would prefer it that way as the 'emergent behaviours' are an undesirable side-effect of rationalising costs.
As a layman it seems to me that this system's behaviour is not 'unpredictable', just not currently completely predictable by Google. Someone earlier compared it to meteorology. This, again, is not unpredictable; it simply has so many complex factors affecting it that even the best of human weathermen are currently unable to fully predict the precise resultant effect of them all.
To infer from this that the system is 'alive' is on a par with claiming that the weather is alive. Even calling it weird seems a big stretch. Which is not to say that we should be any less concerned about the potential dangers of such immensely powerful and currently unpredictable systems than we are about the potential dangers of immensely powerful and currently unpredictable weather.
Yo.
That was me comparing it (loosely) to the weather. What I was saying was that weather is predictable, but only approximately and only for the relatively immediate future.
It might be semantics but I make a distinction between prediction based on contemporaneous data (e.g. a observed low pressure system) and prediction based on historical data (e.g. average temperate for May is 20C).
When I brought up the weather, I was referring to the former, which is based on cause and effect, rather than previously-observed averages.
Yes, wouldn't disagree with that, other than to observe that, theoretically, if one knew every tiny factor that influences the oncoming weather one could, theoretically, predict it with precision any time into the future. In practice we never know every tiny factor so can only make a best estimate, and try to prepare for other possibilities. As a long-time offshore sailor I have spent many hours doing exactly that.
I think the same applies to Google's systems. Accumulations of microscopically small, unforeseen inaccuracies can at times cause the system to make a decision that the programmer would not expect. But, as clean_state said in response to your earlier post, there is nothing weird or biological about that. It seems more akin to my car running slightly rougher than expected because of an intermittent high tension leak from a sparkplug.
> theoretically, if one knew every tiny factor that influences the
> oncoming weather one could, theoretically, predict it with
> precision any time into the future.
Unfortunately not. The weather system is chaotic. From the system of equations, you can compute how the error margin evolves in time. The evolution is exponential. This means that whatever the precision of the initial measurements, even if you know the speed and position of every atom of the atmosphere at one point, your prediction and the reality will diverge exponentially (i.e. very fast).
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
