XLSTAT: Performance indicators for prediction models in Excel

 

Dataset for Example

The abalone dataset contains physical traits of abalones. Since counting rings under a microscope to determine age is time-consuming, we aim to predict age using easier measurements.
We compare two models—SVR and CART—to determine which best predicts abalone age from these traits.

 

Setting Up

• Open XLSTAT.

• Click on the Machine Learning menu and select the Model performance indicators feature.

• The  Model performance Indicators dialog box appears.

In the General tab:

• Select the Rings variable in the Response variable field.

• In the Predicted value field, select the predictions (SVR-Pred and TREE-Pred) made with the different models used.

• Check the explanatory variables option to specify the number of variables used in the construction of our models. This information is useful for the calculation of some indicators (adjusted R², AIC, SBC). Here, all the variables (8) in the dataset were used.

• The Variable labels option is selected because the first row of the chosen columns contains the names of the variables.

In the Outputs  tab:

• Select the indicators you wish to display from the proposed options.

• Click on the OK button to launch the computations.

• The results are displayed on a new Excel sheet.

 

Interpreting the Results

The first table provides at a glance the values of the selected indicators for each model.

Error measures such as the RMSE show us that the second model performs better. In order to determine if this difference is large, we can look at the R2 , which is an indicator between 0 and 1. It corresponds to the determination coefficient of the model and is interpreted as the proportion of the variability of the response variable explained by the model. The nearer R² is to 1, the better the model is. In our case, about 57% of the variability is explained by the model using the SVR , compared with 40% for the second model.

The adjusted R² is 0.49 for the model using SVR and 0.28 for the one using regression trees.
Before looking at the predictions and residuals, let's first have a look on the regression charts. Several charts are displayed for each of the models, but we will focus on 2 of them in this tutorial.

 

 1. Response variable vs Predictions

This chart allows you to compare the predictions and the observed values. The greater the variance explained by the model, the closer the points will be to the regression line.

 

 2. Response variable VS Standardized residuals

The graphics show:

• Greater error variability in the regression tree model compared to SVR.

• SVR performs well (small residuals) for younger abalones (RINGS ≤ 15), but worse for older ones.

• An outlier appears at the top right— the oldest abalone (RINGS = 29); prediction tables help examine it further.

• Smaller residuals indicate a better model fit.

The table below presents a portion of the prediction and residual analysis.

In the Residuals column, values closest to 0 are marked green (best predictions) and those farthest are red (worst predictions).

Observation 32 (Obs32) shows the largest prediction error for both models and is the oldest abalone.

It matches the outlier in Figure 2 and has an atypical residual, indicating a significantly large error. Predictions for this case should be interpreted with caution.

Conclusion

In conclusion, SVR is the better of the two models, explaining 57% of the age variability with a more balanced error distribution. However, it underperforms for abalones older than 15, making it insufficient alone.

A better approach would be to combine SVR with another model or develop a new model better suited for predicting older abalone ages.