Many companies are slowly waking up each day to the smell of a new coffee. It's called Machine Learning! Machine Learning is here to stay. It is expected to influence every aspect of life, including the way business decisions will be made in future. It's already happening.
Machine Learning is a shortcut for machines or systems to learn the way decisions are made in life, which is based on events and measures. This is done not by coding the rules manually, instead discovering them using a mathematical or algorithmic technique. Dredging through historical numeric variables or text data, machine learning helps us to create models that encode the decision-making process so that it can be automated. This is different than the old paradigm of maintaining a rules-based system to create rules or formulas ahead of time to do the decision making.
What are the drivers for Machine Learning to take off? What changed?
Several things. First is the availability of ground truth. Terabytes & Petabytes of data stored in your enterprise data warehouse or data lake holds the key to learning. The advancement of both proprietary and open source architectures (like map/reduce, graph, in memory) is another reason how computation is commoditized. Algorithm development and packaging of that for ease of use is also driving the innovation around Machine Learning.
What are the advantages of Machine Learning?
Machine Learning is used for decision making just like how computers use sorting today to arrange data in a certain order. Both Machine Learning and Sorting techniques share one thing in common – "Algorithm." An algorithm can rearrange or discover things from data by using an inventive technique. When you are using a computer program to find the hidden rules in your data to compute the price of a product or making a go/no-go decision, you automatically increase productivity in your environment. These hidden rules are exactly what machine learning can do to business. This is a significant paradigm shift compared to a rules-based system that may have been used in the past.
Examples of the day to day Machine Learning decision making:
If you are using a Spreadsheet today to calculate the price forecast for your product, you would probably use a formula for that. With Machine Learning, you would create a big table with historical price changes, weather, offers, price, sales, margins, etc., and have it learn. When fed with enough data and variables (also called features), you may be able to forecast the price much better without making any assumptions! You may have to tune the algorithm and its parameters a bit, but it's a lot easier than finding the domain expertise in the hope of getting it right!
Another example is call center notes. You want to find what topics are interesting in the call center notes. Traditional method would be sample the data and manually go through it. In the Machine Learning world, you can classify these documents into groups or clusters automatically so that you can find the 10 or 15 areas of interest or concern by customers quickly.
Countless applications are emerging on making predictions on things like what customer would do next when browsing the website, what items will go into a shopping cart, what kind of resumes reflect the right person for the job etc.,
What about Artificial Intelligence?
(c) CanStockPhoto /kentoh
Artificial Intelligence is a branch of Machine Learning. Advanced Machine Learning techniques such as Deep Learning mimic the brain's way of making decisions leading to surreal applications. Self-driving cars, Robots, etc., are built on learning from the real world on what choices are good and bad, and they go to work mimicking human decisions.
Challenges with Machine Learning:
There ain't no such thing as a free lunch ...
Machine learning is both art and science. However, it's more interesting to do than doing manual rule writing. Machine Learning depends on real data. As with any computer technique, if the data is not curated or cleaned correctly, the results will reflect that. Machine Learning requires a data scientist to understand the outcomes correctly that matters to business. The biggest tradeoffs to a Machine Learning technique is accuracy pitched against explainability. The more accurate the predictions are, it may be difficult to explain to a rational jury as machines learn from 100s of dimensions. However, Algorithms whose decisions are easy to interpret suffers from accuracy. So finding the midpoint is often important.
Availability of ground truth is often a challenge. Let's say we want to know if there are non-compliance issues with communications to a customer. We need first to create a repository of documents that has compliance problems and another set with non-compliance issues. The catalog is called the ground truth. The machine learning technique learns from the good, and the bad and models are created. A prediction algorithm can now use this model for making decisions on a new set of documents. As you can imagine, the ground truth may not be available readily, and data scientists have to use different techniques to bootstrap this, iterate and perform other forms of discovery.
What is the tipping point?
Machine Learning is an iterative/discovery and modeling environment. Using the right tools, one can take Terabytes and Petabytes of data, spend relatively less time to find the rules that lead to decisions. The accuracy of decision-making sometimes outweighs all the caveats in the previous sections. The total time to create, test, deploy and update the models is a fraction of what it takes to build and maintain a legacy rules or formula-based system. It comes down to analytic productivity that pays over and over.
As a business, you have the choice between proprietary platforms like Teradata Aster or open source architectures like Spark or R. Knowledge of a declarative language like SQL or procedural languages like Java, R, and Scala can get your data scientist going. With relevant training, a data scientist will be able to exploit the awesomeness of Machine Learning for your business, giving the benefit of smart decisions …