So what is Big Data? Big Data as defined by Wikipedia, “is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to “spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.”
Hurwitz & Associates have identified 4 dimensions of data:
- Volume – 20 years ago, a Gigabyte was huge! Now, we have Terabytes and even Petabytes (one quadrillion bytes), an enormous volume of data. And we’re analyzing all kinds of data: from posts on Facebook to weather pattern data.
- Velocity – Much of the volume of data must move at very fast speeds such as banking transactions from vendors or stock trades from day traders.
- Variety – No longer are we just processing text but also audio and video as well as varieties in between.
- Veracity – It’s easy to trust your data when it’s a small quantity, but as the quantity grows, lack of trust grows exponentially with it.
Career opportunities abound in a this new area being called Data Science. In this very young, and maturing field, there is a quickly growing need that is not being fully met. Consulting firm McKinsey and Company predicts that in just 6 years, there will be a shortage of 1.5 million workers with data science skills and another 140-190,000 workers with large data processing skills. While universities gear up with degrees in this area, today’s candidate that may be a good fit comes with strong backgrounds in computer science, algorithms and operating systems. “If they have a good foundation in that, then you pair that up with a training program that allows them to understand how to translate into this new architecture,” Jim Yu from BrightEdge says.
More ideas on what types of skills to come next week on IT CareerCast. Stay tuned!
Sources:
“Four Vendor Views on Big Data and Big Data Analytics: IBM”, Hurwitz & Associates, Fern Halper, January 2012
Computerworld, Big data worker shortage demands job candidates with diverse backgrounds by Fred O’Connor, October 10, 2012