No mention of Scala or Spark ML? Not convinced. Not that they're the 'best choice', but if you're listing options, I'd expect them to be there.
My experience is we get a bunch of guys who've used a range of tools in an academic environment, and thus have used whatever language or package happened to come to hand to most easily solve their particular problem.
Then they have to face a production system designed around all the usual requirements for uptime, release management, security, versioning, testing and so on. The range of options when you have petabytes of data being generated in real-time is somewhat smaller, especially if you have any sane constraints on hardware (after all, you probably have a few hundred machines in your cluster, you can't justify doubling that just because a particular project runs a bit slow). Businesses at this sort of scale only want limited exposure to tools that cannot be proven to be robust. DevOps have no desire to install yet another package just to satisfy your needs, when it probably needs yet another license audit, release cycle and further support.
Suddenly, you have a bunch of guys who are jumping through hoops not to give up their language of choice because they've got into the mindset that you have to have the right language to solve the problem. And in the process they've completely failed to understand the architectural techniques needed to deliver robust, performant solutions - believing that because they once saw Perl running quickly, it's the only way to deliver at scale.
Then you look at the business analysts who have demanded some horrendous SQL shim over your 'big data' just because they're incapable of learning a new language, and you can see where this is all going.