The sample was not necessarily too small to train some systems, but many would be liable to over-fit the training data and not generalise. The big problem was ensuring the testing was right. Unfortunately this isn't even uncommon. When I was doing my Masters degree I was reviewing the available literature on predicting foreign currency movements and an outright majority of the published papers I found using Machine Learning to predict foreign currency movements made elementary mistakes in their testing procedure not dissimilar to this, leading to unbelievably high prediction scores. I really hoped to find one had a suitable process to reliably predict next-day currency movements, but unsurprisingly that virtual unlimited pot of gold was not real and my final paper primarily served to debunk a dozen or so other papers by showing that their results were not reproducible and to explain the identifiable flaws in their processes.
The simple lesson is that too many people don't understand the necessary processes for doing machine learning properly - including many academics writing papers about it.
Biting the hand that feeds IT
