XLSTAT: Training a Support Vector Machine (SVM)

 

Dataset for Example

The dataset comes from the Kaggle competition “Titanic: Machine Learning from Disaster”, based on the 1912 RMS Titanic sinking, where over 1,500 of 2,224 passengers perished due to a shortage of lifeboats.

It includes data for 1,209 passengers with the following variables:

• survived: Survival (0 = No, 1 = Yes)

• pclass: Ticket class (1st, 2nd, 3rd)

• name: Passenger name

• sex: Gender (male, female)

• age: Age

• sibsp: Number of siblings/spouses aboard

• parch: Number of parents/children aboard

• fare: Fare paid

• cabin: Cabin identifier

• embarked: Port of embarkation (C = Cherbourg, Q = Queenstown, S = Southampton)

We will use this data to show how to set up and train a SVM classifier on the Titanic dataset and see how well the classifier performs on a validation set.

 

Setting Up

• Open XLSTAT.
• Click on the Machine Learning menu and select the Support Vector Machine (SVM) feature.

 

• Once you have clicked on the button, the dialog box appears. Select the data on the Excel sheet.

In the General tab:

• In the Response variable field, select the binary variable with want to predict when classifying our data. In our case, this is the column giving the survival information.

• We also select both quantitative and qualitative explanatory variables by checking both checkboxes as shown below.

• In the quantitative field, select columns corresponding to the following fields: age, sibsp, parch, fare. Note: To select multiple columns, you may use the Ctrl key.

• In the qualitative field, select the columns with qualitative information: pclass, sex, embarked.

• As the name of each variable is present at the top of the table, check the Variable labels checkbox. 

In the Options tab:

• For the SMO  parameters, let the default options.

• The C field corresponds to the regularization parameter. It translates how much misclassification you want to allow during the optimization.

• A large value of  C means a strong penalty on each miss-classified observation. In our case, we set the value of C at 1.

• Leave the Tolerance at its default value.

• The tolerance parameter is a stopping criterion. If you want to speed up calculations, you can increase the tolerance parameter. We 

• Select Standardisation in the Preprocessing field and we use Linear kernels as shown below.

In the Validation tab:

• Check the Validation check box and select 100 observations randomly  drawn from the training sample as indicated below. As we want to see how well our classifier performs, we make a validation sample out of the training sample. 

In the Outputs tab:

• Select the outputs we want to obtain as shown below:

• Click on the OK button to launch the computations.

• The results are displayed on a new Excel sheet.

 

Interpreting the Results

The first table displays a summary of the optimized SVM classifier. You can see on the figure below that the classifier had to classify between classes 0 and 1 and that the class 0 has been labeled as the positive class . There were 943 observations used to train the classifier out of which 452 support vectors have been identified.

The second table shown below gives the complete list of the 452 support vectors together with the associated alpha coefficient values and the positive or negative value of the output class. Together with the bias value of the former table, this information is sufficient to fully describe the optimized classifier.

The next table displays the resulting confusion matrix from the validation sample. This matrix gives us indications of how well our classifier performed. For the validation sample, we have 78% of correct answers.

Following this table, the graph below shows the ROC curve for the validation sample. The ROC curve corresponds to the graphical representation of the couple (1 – specificity, sensitivity) for the various possible threshold values.

We want a curve close to the upper left corner:

Finally, the Area under the curve (AUC) is 83,1% which means the model has 83,1% chance to classify well an observation from the positive class.

 

Conclusion

We trained our classifier using linear kernels and obtained fairly good results with 78% of correct classification. Some additional tuning might yet be necessary to challenge the best data scientist on kaggle. One approach could be to change the kernel family and see how well a higher dimension space might perform on our dataset.