XLSTAT: Pairwise multiple comparisons after a multi-way ANOVA in Excel

Background

After identifying a significant effect of a factor on a dependent variable, we often want to know which specific levels differ. For example, if a one-way ANOVA shows a difference among four medical treatments on blood pressure, we may follow up with pairwise comparisons to see which treatments differ from each other.

Pairwise multiple comparison methods calculate a p-value for each pair of factor levels. However, testing many pairs increases the number of p-values and the chance of false positives.

At a 5% significance level, we might expect 5 false positives out of 100 comparisons by chance alone. To address this, p-values are adjusted (penalized) based on the number of comparisons.

The choice of comparisons should reflect the study’s goals and be planned ahead. When comparing all levels, methods like Tukey’s HSD and Fisher’s LSD—both available in XLSTAT—are commonly used. Newman-Keuls’s SNK method is also included but is less reliable due to its weak conservativeness. The REGWQ method is more robust but less widely used.

In cases where each level is compared only to a control, Dunnett contrasts—also available in XLSTAT—are more appropriate, as they involve fewer comparisons and less penalization. Custom contrasts can also be defined for tailored comparisons.

Dataset for Example

The (attached) data correspond to a sensorial test on 2 food products.

4 assessors have tested both products and gave grades on 2 characteristics: taste quality and sweetness. Every product was tested 6 times by each assessor within different tasting sessions. Note: The data was artificially generated.

In this example, we focus on multiple pairwise (post hoc) comparisons, often used after parametric or nonparametric tests. We examine a complex case involving two 2-way ANOVAs with a panelist × product interaction. A significant interaction indicates differing preferences among panelists—for instance, some favoring product A and others product B.

Pairwise comparisons at the interaction level help identify which panelists fall into each group.

Setting Up

We will run pairwise multiple comparisons following two 2-way ANOVAs including an interaction between the factors.

• Open XLSTAT.
• Click on the Modeling data menu and select the ANOVA feature:

In the General tab:

• Select the Taste and Sweetness columns as dependent variables.
• Select the Panelist and Product columns as qualitative explanatory variables.

In the Options tab:

• Activate the Interactions / Levels option.
• Select 2-level interactions to include the Panelist × Product interaction in the model.

In the Outputs tab:

• In the General tab, activate the Type I / II / III SS option to view different types of sum of squares.
• In the Means tab:
  • Activate the pairwise comparisons option.
  • Choose Tukey HSD as the comparison method.
  • Activate the standard errors and confidence intervals options to include them around the means in the results.
  • Do not activate the “Apply to all factors” option if you want to manually select which model factors or interactions to apply multiple comparisons to.

• Once you click the OK button, three dialog boxes appear in sequence:

In the first dialog box, define the set of factors and interactions to include in the general ANOVA models.

In the second and third dialog boxes, define the factors and interactions for multiple pairwise comparisons for each of the two dependent variables.

In each dialog box, select all three elements from the list: two factors and one interaction.

• The results will then appear in a new Excel spreadsheet.

Interpreting the Results

For the taste variable, all three ANOVA tables are identical due to a balanced design. In unbalanced designs, Type III sum of squares is most reliable.

While the product effect on taste is not significant (p = 0.334), the Panelist × Product interaction is highly significant (p < 0.0001).

Interpretation using the bottom-right means chart shows that, despite similar average scores (~5.5) for both products, panelists have contrasting preferences: panelists 1 and 2 prefer product A, while panelists 3 and 4 favor product B. Notably, all panelists agree on the average taste rating for product B.

In the pairwise comparisons section, each factor or interaction has its own analysis. Panelists are compared across all products, products across all panelists, and all panelist-product combinations are compared for the interaction. Focus is usually placed on the most significant terms from the ANOVA. For the taste variable, this means closely examining the interaction. A detailed table then presents mean differences and adjusted p-values for each pair.

Significance results are summarized in a smaller table where each panelist-product combination is labeled with letters. Combinations sharing a letter are not significantly different, while those with no letters in common are significantly different.

The pairwise comparisons confirm the insights from the means chart.

For product A, panelists 1 and 2 (group A) differ significantly from panelists 3 and 4 (group C), indicating disagreement. For product B, all panelists share the same letter (B), showing consensus. From a marketing view, product A could be targeted toward consumers like panelists 1 and 2, while product B may need taste improvement before launch.

For sweetness, ANOVA results show a strong product effect (p < 0.0001) but no significant panelist or interaction effects.

The bottom-right means chart shows that all of the panelists agree on the fact that product B is sweeter than product A.

Examining multiple comparisons for sweetness is unnecessary, as the only significant factor—product—has just two levels, which are already shown to differ in the ANOVA results.

Interpreting the Results

We tested the interactive effects of panelist and product on taste and sweetness.

XLSTAT’s LS means tool helps compare factor effects across multiple dependent variables. Each factor and interaction has an LS means table and chart.

For the panelist factor, the table shows estimated means for each panelist on both variables, with significance letters included as cell comments—hovering over the red triangle reveals them.

The chart shows that, regardless of the products, panelists mostly disagree on the taste but agree on sweetness, which is supported by p-values associated with the panelist effects in both ANOVA models.

The Product LS means chart shows that regardless of the panelist, products are graded the same on average on the taste scale, while they are judged different at the sweetness scale.