Major and minor flaws?
This is based on the reactions to the perception to how the researchers played and the associated voice. But they did not include a control group - they had the data there potentially but didn't use it.
They also seem to have tried to compensate for any differences in their playing style (and that of the other players) in the statistics rather than in attempting more standardization (or a larger dataset). What's that phrase about changing the experiment by interacting with it?
The dataset size does not seem to have been determined prior to the experiment (actual size was N = 126). One interpretation of the statement in the report " We stopped at 163 as this is a substantial time effort." is they ran out of time/got tired/got bored/had something else they needed to do rather than X games is the right amount to play to get good answers.
That leaves open the question as to whether the experiment had sufficient power - a good statistics treatment of research should discuss this in the paper - in the first place for a significant difference to be found. If you throw enough permutations at an data block, something will stick, even if just due to the random distribution of the data. As exemplified by "scientists say X will give you cancer" interpretations by newspapers.
I also note they are reusing data from a previous experiment - " Kuznekoff and Rose’s (2013) original study" - http://nms.sagepub.com/content/15/4/541 New Media & Society June 2013 (first published online in 2012) so there's a second element to the possibility that a better study would have been done if they had designed their experiment and got new data rather than try and fit an experiment to previous work.
Personal declaration of my own skillset in this area - reading some books (No Starch publishing's Statistics Done Wrong), and listening to the radio (Radio4's More or Less: Behind the Stats), and a large dash of cynicism.
Biting the hand that feeds IT
