Learn Machine Learning in 10 Minutes: Support Vector Machines

Learn Aster
Teradata Employee

Support vector machines are among the most popular "out of the box" classification algorithms. The objective of the algorithm is similar to logistic regression – given a set of predictor variables, classify an object as one of two possible outcomes. The methods by which the algorithms achieve this objective are very different.  Intuitively, logistic regression develops a probabilistic model based on the input data, and given a test instance x, estimates the probability that x belongs in a particular class. In contrast, support vector machines ignore the probabilistic interpretation. A support vector machine attempts to find the boundary that maximizes the distance between the two classes. In the prediction phrase, given a test instance x, calculate which side of the boundary x lies in order to compute a class prediction.

This implementation solves the primal form of a linear kernel support vector machine via gradient descent on the objective function. It is based primarily on Pegasos: Primal Estimated Sub-GrAdient SOlver for SVM (by S. Shalev-Shwartz, Y. Singer, and N. Srebro; presented at ICML 2007).

As mentioned before, it is a binary classification algorithm. Multiple classification is achieved using machine learning reductions, or more specifically one-against-all is adopted in the function. In a K-class classification problem, K support vector machines are trained. Each SVM is a binary classifier, where the nth class is labeled positive, and all other classes are labeled negative. In the test phase, each test observation is trained using each of the K SVMs. The class which results in the most observations predicted as positive is the resulting prediction.

To learn more about how to implement Support Vector Machines watch this short video: