Recommend reading the original
Do check out the original paper, esp., Appendix 1:
"During Grand Challenge simulations such as this one, there are inevitably problems encountered, and such was the case here. In Figure 4, the time per step suddenly increases at step 46 as indicated by the arrow labelled A. This was caused by one of the nodes performing in a substandard way which resulted in the entire simulation to take twice as long, as the other nodes were waiting for this node to complete its share of the work. THE EXACT CAUSE OF THIS PROBLEM IS NOT KNOWN, AND WILL NEVER BE KNOWN, but it was very likely a rogue process that was left running on the node that stole processing cycles. This problem disappeared when the simulation was restarted without this node.
The second problem occurred shortly thereafter, around step 50, and was a result of the increase in efficiency as the simulation progressed. In Figure 4 we see that the gravity calculation time drops dramatically between step 0 and step 20 as structure forms and the effect of the initial condition grid is no longer relevant allowing the force accuracy to be relaxed. At some point, the amount of work being shipped to the GPU reaches a threshold that triggers a not yet understood problem with the GPU device. When this threshold is reached, the GPU will, very rarely, accept work but never complete it. By sending work in a more controlled fashion, this problem is eliminated or vastly reduced allowing the simulation to run to solution, but with slightly decreased performance. The cause of this is still under investigation." [Emphasis added]
While the simulation was a tour de force for massively parallel processing, it is also a cautionary tale about how poorly prepared we are to monitor and learn from experience with such systems.
Biting the hand that feeds IT
