Even though we have iSCSI and FCoE today from some manufacturers, look at almost any best practice guide and at least with iSCSI and NFS they'll always reccomend running jumbo frames whenever possible for the storage. Jumbo frames I think in general is still too risky to run as your "main" ethernet ports because of compatibility issues.
So the point is even with ethernet, if you want best performance you need seperate NICs on each server and seperate cables to run jumbo frames. You can use the same ethernet switches since switches can run jumbo/non jumbo on the same ports, with server NICs I have not yet come across one at least on linux and vmware where you can have a physical port run both jumbo and non jumbo frames (non jumbo frames as in ENTIRELY non jumbo not jumbo with the ability to re-transmit the fragments if they are too big).
Jumbo frames is certainly not a requirement, you can (and many do) just fine without it. But if your running a serious operation with storage on ethernet I think most people will opt to use jumbo whenever possible.
I wonder if we'll ever get to "super jumbo" frame sizes, I mean jumbo frames became popular back when gigE was starting out, now with 10Gig commonplace now, and 40gig coming, I think it would make sense to give the option to boost the frame size even higher.