XLSTAT: Naive Bayes classification

 

Background

The Naive Bayes classifier is a supervised machine learning algorithm that allows you to classify a set of observations according to a set of rules determined by the algorithm itself.
This classifier has first to be trained on a training dataset that shows which class is expected for a set of inputs.
During the training phase, the algorithm elaborates the classification rules on this training dataset that will be used in the prediction phase to classify the observations of the prediction dataset.

Dataset for Example

This dataset from the Center for Machine Learning and Intelligent Systems' repository is called the Zoo database, created by Richard Forsyth in 1990 for the PC-Beagle program. It contains 101 animals described by 17 qualitative attributes—mostly binary (1 for presence, 0 for absence)—and one numerical variable (legs) with values from 0 to 8.

The final column, “type,” is a class label from 1 to 7 indicating the animal’s subgroup. This is the target variable for our Naive Bayes classifier. The dataset is split into 94 rows for training and 7 for testing.

Setting Up

• Open XLSTAT.

• Click on the Machine Learning menu and select the Naive Bayes classifier feature.

• In the General tab of the Naive Bayes classifier dialog box:

  • Select the type column (18th column) in the Y / Qualitative variables field, using only the first 94 rows for training.

  • In the X / Explanatory variables field, check the Qualitative box and select the 17 attribute columns from the training set.

In the Prediction tab, select the prediction dataset, consisting of the last 7 animals in the list.

In the Options tab:

Since we are using qualitative data, the distribution selection is disabled, and only the empirical distribution is used to estimate conditional probabilities.

To enhance robustness when classifying new observations, enable Laplace Smoothing by setting the Smoothing parameter to 1.

• In the Output tab, activate all 7 outputs as following:

• Click on the OK button to launch the computations.

• The results are displayed on a new Excel sheet.

 

Interpreting the Results

The first two tables display the observed frequency and relative frequency distributions of the output class and the attributes in the training set.

The first table shows that type 1 is the most frequent class in the training set, representing 41.94%.

The following table indicates that no 5-legged animals are present in the training data.

However, the next table, showing observed frequencies in the prediction set, reveals that a 5-legged animal is present.

It will be interesting to see how the classifier handles this unseen modality. First, let's examine its performance on the training set using the two confusion matrices below. Both show a high true positive rate, which is promising.

Finally, the classification for our prediction set is displayed via two tables as shown below:

The first table shows results by class, and the second provides results for each observation.

Observation PredObs4, the 5-legged starfish, is correctly classified as type 7—alongside crab, clam, and crayfish. This success is due to Laplace smoothing, which allowed the classifier to handle the unseen modality. Finally, posterior probabilities for each class are provided for every predicted observation.