Thank you for your interest in our work, below is a brief answer to the comments.
Regarding our choice of controllers, we wish we had access to the code of more SDN controllers.
To properly conduct our evaluation we had to port the controllers to different architectures, which required us to obtain the source code of the controllers. Unfortunately, there are too many SDN controllers who cannot be compared to the state-of-the-art as their source code is not disclosed. We chose to focus on NOX(NOX-MT), Floodlight, Maestro and Beacon because they have had a significant impact on the research: as an example NOX was cited more than a thousand times according to Google Scholar.
The HPE VAN SDN was mentioned to achieve a maximum of 2.3 million flows per second on cbench for a single controller, almost an order of magnitude slower than the highest performance we observed http://www8.hp.com/h20195/v2/GetPDF.aspx/c04111355.pdf. While there might exist other interesting controller designs, there exist many controllers that are known to be outperformed by some of the controllers we tested, or whose source code is not available and whose description is too high level to be correctly re-implemented.
Regarding the choice of benchmarking tools, we focused on using cbench and profiling tools and we agree that our resulting workloads may not reflect the behavior observed in a realistic large-scale software defined network. To our knowledge, however, there are no alternative tools available to test the performance of SDN controllers. We also believe that the benchmarking of SDN controllers is a topic of interest in itself and we are interested in any form of collaboration with industrials and academics who are working on this problem.
Finally, we want to clarify that we acknowledged at two places in the paper that "reactive configurations are known to be very difficult to deploy in carrier grade networks” which makes it an interesting research challenge for us. This is mentioned in the introduction and Section III.D. Evaluating proactive controllers with corresponding benchmarking tools on multicore and manycore platforms could be insightful, however, existing non-OpenFlow reactive controllers, like Fastpass (http://fastpass.mit.edu/), that achieve high utilization and low latencies are also of interest for us.
This paper has just been accepted for publication in the proceedings of the 41st IEEE Conference on Local Computer Networks (LCN). We hope to see you at the conference in Dubai in November to continue this discussion.
Biting the hand that feeds IT
