XLSTAT: Interpreting Contradictory Results between ANOVA and Multiple Pairwise Comparisons

Background

We use multiple pairwise comparison tests after ANOVA to identify which specific groups differ significantly, since ANOVA only tells us that at least one group differs but not which ones.

However, testing multiple group pairs increases the chance of false positives (Type I errors)—this is the multiple comparisons problem.

To control this, post hoc tests apply p-value corrections (like Bonferroni or Tukey) to reduce the likelihood of wrongly identifying a difference due to random chance.

Interpretation of Results

ANOVA and post hoc tests answer different questions using different methods, so their results can sometimes conflict.

A common case is when ANOVA is significant (p ≤ 0.05), indicating at least one group differs, but all pairwise comparisons are non-significant (p > 0.05). This can happen due to conservative p-value corrections or small effect sizes that ANOVA detects collectively but individual comparisons do not.

An example is displayed below:

Post hoc tests may yield non-significant results despite a significant ANOVA due to:

• Low statistical power (e.g., small group sizes).

• Many factor levels, increasing p-value corrections.

• A marginally significant ANOVA result.

• Use of conservative post hoc tests.

Conversely, significant pairwise comparisons with a non-significant ANOVA can occur due to:

• Specific strong group differences that are diluted in the overall test.

• ANOVA’s reliance on overall variance, which may miss isolated but meaningful contrasts.

An example is displayed below:

An unfortunate common practice is to pursue multiple comparisons only when the hull hypothesis of homogeneity is rejected . (Hsu, page 177).

Post hoc tests can sometimes detect significant group differences even when ANOVA is not significant. These results are generally valid, except when using the protected Fisher LSD test, which requires a significant ANOVA to proceed.