Re: Don't Hate The Message Just Because You Hate The Messenger
Improving Worker Productivity is the imperative.
No, Improving business efficiency is the imperative. The business doesn't exist for the workers. If it's more efficient to replace the workers with robots, or computers, or toadstools, that's what'll happen.
The best data for making workers more productive is already held by the organization,...
... but that data is federated and noisy and will always be federated and noisy.
Both of those depend on the organization, and the data involved. I'm working with some data right now that is sourced from three separate databases, and the only way to link them is on name, which causes some problems in spelling, etc., but the majority of data used by my organization is well centralized and has very little noise.
Big Data technologies handle the discrete technical challenges associated with that reality.
Again, that depends on the data involved. Most of the data I work with has too few rows and too little noise to make the effort involved in setting up "Big Data" technologies worthwhile. The biggest factor here being the degree of noise: If your data sources are well-designed from the start, you don't need to muck about with Big Data technologies.
But don't dismiss the importance of this next phase of computing just because it clashes with your expectations about third normal form.
And you don't dismiss normal forms just because they clash with your desire to be on the crest of a computing revolution. Why do you think that many of the bigger "NoSQL" technologies are adding relational features as fast as they can?
Big Data technologies have some use. They are best when:
1. You cannot curate your data sources,
2. Inserts and Updates happen almost continuously,
3. Queries don't have to be based on up-to-the-second data,
4. Your dataset is very, very, very large, and
5. You have lots of hardware resources to throw at the processing of this dataset.
That, as APS has said, is NOT the case for any but the largest organizations, and even the largest organizations don't necessarily need that data processing in order to run efficiently.