Earlier this year I posited that, due to the exponential rate of growth, the amount of data collected for analysis is becoming beyond the scope of the current analytical staffs to examine it all. And that the answer to growing the cumulative brain power necessary for this exponential growth in analysis is going to have to be machine learning and artificial intelligence. This is now in the works.
“MIT researchers aim to take the human element out of big-data analysis, with a new system that not only searches for patterns but designs the feature set, too. To test the first prototype of their system, they enrolled it in three data science competitions, in which it competed against human teams to find predictive patterns in unfamiliar data sets. Of the 906 teams participating in the three competitions, the researchers’ ‘Data Science Machine’ finished ahead of 615.”
Not only was the prototype better at the analysis, but it was also a lot, lot faster. “… the teams of humans typically labored over their prediction algorithms for months, the Data Science Machine took somewhere between two and 12 hours to produce each of its entries.” And this is going to come to a theater near you very soon: “’The Data Science Machine is one of those unbelievable projects where applying cutting-edge research to solve practical problems opens an entirely new way of looking at the problem,’ says Margo Seltzer, a professor of computer science at Harvard University who was not involved in the work. ‘I think what they’ve done is going to become the standard quickly — very quickly.’”
What this means going forward is this will soon be no country for mediocre data scientists. Whereas we will not have Javier Bardem chasing after us across the countryside, the future of the data scientist is going to be much more dependent on machine learning for assists, and the skill sets required will become the ability to simplify, explain and communicate the results of the machine analysis to the rest of us mere mortals.