Re: This doesn't make sense
Why would the IO phase need to write out the sum of all data transmitted during the compute phase ?
Not all of the traffic during the compute phase is data, some is control/synchronisation from MPI. Also any data transmitted during the compute phase may have been updated, discarded or expired during compute - the end state or result isn't necessarily the sum of all of the data.
Additionally, the MPI traffic can be quite bursty - so while you want the interconnect to be capable of high performance, it's capacity (bandwidth) isn't usually the limiting factor; the speed (latency or message rate) is typically more important.
It is perfectly acceptable to have MPI (compute) and filesystem (IO) traffic on the same interconnect. IB has QoS features to ensure certain traffic types can be prioritised e.g. MPI packets, to ensure latency in the compute phase doesn't skyrocket if there is also IO traffic on the network.
In fact if you have a multi-user, multi-job cluster with jobs starting and stopping asynchronously it's pretty much inevitable that you will have a mix of traffic at any given time on the cluster.