[A, SfS] Chapter 7: Chi-square tests and ANOVA: 7.5: Two-way Analysis of Variance
Two-way Analysis of Variance
Analysis of Variance (Two-Way)In this section, we will discuss how to determine whether the population mean of a continuous variable differs among the categories of two categorical variables combined in the same analysis.
#\text{}#
Two-factor ExperimentIn a two-factor experiment, the treatment groups are distinguished by the levels of two categorical Factors #A# and #B#.
For example, consider an experiment in which the response variable is the change in systolic blood pressure, and the two factors are video clip viewed and presence of audio.
Assume there are five levels for the first factor, while the second factor has only two levels (ON or OFF).
The research question is whether the type of video clip watched and the presence or absence of audio while watching the clip have some combined influence on the blood pressure within some specified population.
In general, Factor A has #I# levels and Factor B has #J# levels, and the population mean for the response variable #Y# for level #i# of Factor A and level #j# of factor B is #\mu_{ij}#, for #i = 1,...,I# and #j = 1,...,J#.
Meanwhile, #\mu# is the population grand mean, i.e., the mean of response variable #Y# for the population across all levels of both factors, while #\mu_{i}# is the population mean of response variable #Y# at level #i# of Factor A across all levels of Factor B, and #\mu_j# is the population mean of response variable #Y# at level #j# of Factor B across all levels of factor A.
We define \[\alpha_i = \mu_i - \mu\] as the main effect for level #i# of Factor A at the population level, for #i = 1,...,I#, and note the constriction #\alpha_1 + \cdots + \alpha_I = 0#.
We further define \[\beta_j = \mu_j - \mu\] as the main effect for level #j# of Factor B at the population level, for #j = 1,...,J#, and note the constriction #\beta_1 + \cdots + \beta_I = 0#.
Next, we define \[\gamma_{ij} = \mu_{ij} - \mu_i - \mu_j + \mu\] as the interaction effect between level #i# of Factor A and level #j# of Factor B at the population level, for #i = 1,...,I# and #j=1,...,J#, and note the two constrictions #\gamma_{1j} + \cdots+ \gamma_{Ij} = 0# for #j = 1,...,J# and #\gamma_{i1} + \cdots + \gamma_{iJ} = 0# for #i = 1,...,I#.
The presence of an interaction effect between two factors implies that the values of the response variable for some levels of one factor depend on the levels of the other factor. We say that one of these factors moderates the main effect of the other factor.
Two-Way ANOVA Model and HypothesesThe two-way analysis of variance model for the value of the response variable #Y# for a subject randomly-selected from level #i# of Factor A and level #j# of Factor B is: \[Y_{ij} = \mu_{ij} + \varepsilon = \mu + \alpha_i + \beta_j + \gamma _{ij} + \varepsilon\] where the random error #\varepsilon \sim N\Big(0,\sigma^2\Big)# for some variance #\sigma^2 > 0# that is the same for all #i# and #j#.
In two-way ANOVA we conduct three hypothesis tests:
- Interaction effect
- #H_0 : \gamma_{ij} = 0# for all #i# and for all #j# (there is no interaction effect)
- #H_1 : \gamma_{ij} \neq 0# for at least one combination of treatment levels #(i,j)# (there is some interaction effect)
- #H_0 : \gamma_{ij} = 0# for all #i# and for all #j# (there is no interaction effect)
- Main effect for Factor A
- #H_0:\alpha_i = 0# for all #i# (there is no main effect for Factor A)
- #H_1:\alpha_i \neq 0# for at least one #i# (there is some main effect for Factor A)
- #H_0:\alpha_i = 0# for all #i# (there is no main effect for Factor A)
- Main effect for Factor B
-
#H_0:\beta_j = 0# for all #j# (there is no main effect for Factor B)
- #H_1:\beta_j \neq 0# for at least one #j# (there is some main effect for Factor B)
-
\[\text{}\] As in one-way ANOVA, to test the hypotheses we need sums of squares and their corresponding mean squares (obtained by dividing the sum of squares by the appropriate degrees of freedom).
Test Statistic and Null Distribution of the Full ModelA two-way ANOVA always begins with the analysis of the full model, which includes both main effects and their interaction.
For simplicity, assume that we have a balanced design, with #K# replicates in each combination of categories.
To start, we compute the following sums of squares:
- A sum of squares for each effect (#SSA# and #SSB#, with #I - 1# and #J - 1# degrees of freedom, respectively).
- A sum of squares for the interaction (#SSAB# with #(I - 1)(J - 1)# degrees of freedom).
- The error of sum of squares (#SSE# with #IJ(K-1)# degrees of freedom).
We omit the details of the computation of the four sums of squares.
Next, we divide each sum of squares by its degrees of freedom to obtain the four mean squares:
- The mean square for each main effect (#MSA# and #MSB#).
- The mean square for the interaction (#MSAB#).
- The mean square error (#MSE#, which estimates #\sigma^2#).
We then form 3 F-statistics:
- #F_A = \cfrac{MSA}{MSE}#
- #F_B = \cfrac{MSB}{MSE}#
- #F_{AB} = \cfrac{MSAB}{MSE}#
Under each null hypothesis, each F-statistic has the F-distribution with degrees of freedom corresponding to the two main squares in its computation. From these we obtain P-values for each of the three hypotheses. All of this information is reported in the ANOVA table.
However, we always conduct the test of an interaction effect first, to determine whether or not an additive model is appropriate, i.e., a model with no interaction.
If we reject the null hypothesis in this test, then the additive model does not hold, and it is then difficult to make conclusions about the individual main effects of the two factors. The interaction of the two factors can obscure a main effect for factor A that is truly present but differs depending on the categories of Factor B.
For example, consider again an experiment in which the response variable is the change in systolic blood pressure, and the two factors are video clip viewed and presence of audio.
Assume there are five levels for the first factor, while the second sector has only two levels (ON and OFF).
The research question is whether the type of video clip watched and the presence or absence of audio while watching the clip have some combined influence on the blood pressure within some specified population.
Suppose we discover that, when the audio is OFF, video clips 1, 3, and 5 elevate the mean blood pressure a lot while video clips 2 and 4 have very little effect on the mean blood pressure. But when the audio is ON, video clips 2 and 4 elevate the mean blood pressure much more than do the other three video clips.
This is a serious interaction between the two factors, and in the two-way ANOVA output we might see that the interaction effect is significant (small P-value), while the two main effects are found not significant (large P-values) when in fact they are truly present.
When this occurs it might be a good idea to split the data set into two groups: those cases for which the audio is OFF and those cases for which and audio is ON. Then conduct a one-way ANOVA for each group separately, using the video clip as a single factor for each. Then the effect of the video clip can be observed in each group.
\[\text{}\]One can visualize the main and interaction effects graphically using an interaction plot.
Interaction PlotIn an interaction plot, for each level of one factor, the mean responses for the treatment combinations are plotted against the levels of the other factor as separate lines.
If the lines intersect, it is probable that the interaction effect will be found significant.
If the lines are roughly parallel, the additive model will almost certainly hold.
If there is an interaction effect, the plot can provide guidance about how to split the data to investigate further.
For the above example, the interaction plot might look like:
The traces of the sample mean DV values for audio ON and audio OFF intersect repeatedly across the #5# types of video clips, indicating the presence of the interaction.
A separate one-way ANOVA for each audio setting should be used to further investigate the main effect of the video clip.
\[\text{}\]If we conclude that there is no interaction effect, then we can use the additive model to assess the presence of main effects for the two factors in combination.
Test Statistic and Null Distribution of the Additive ModelThe additive model is represented by: \[Y_{ij} = \mu + \alpha_i + \beta_j + \varepsilon\] where #\varepsilon \sim N\Big(0,\sigma^2\Big)# for some variance #\sigma^2 > 0# that does not depend on #i# or #j#.
The two-way ANOVA now only involves the mean squares for the main effects (#MSA# and #MSB#), and a new version of the mean square error (#MSE#), whose degrees of freedom is now #IJK - I - J + 1#.
This gives us two F-statistics: \[F_A = \cfrac{MSA}{MSE}\,\,\,\,\,\text{and}\,\,\,\,\,F_B = \cfrac{MSB}{MSE}\] which both have F-distributions when the null hypotheses are true, with their degrees of freedom corresponding to the degrees of freedom of the associated mean squares.
Then the corresponding P-values can be used to assess the presence of the main effects for each factor. This information is reported in an ANOVA table.
The model assumptions of homoscedasticity and normality of #\varepsilon# can be checked in the same way as in one-way ANOVA, i.e., with a scatterplot of the residuals against fitted values and a Normal quantile-quantile plot of the residuals.
\[\text{}\]
Using RTo conduct two-way ANOVA we again use the #\mathtt{aov()}# function in #\mathrm{R}#.
Suppose we have data collected on subjects measured on a response variable #Y#, and two categorical Factors A and B, combined in a data frame named #\mathtt{Data}#.
To obtain the ANOVA table for the full model, we use:
> Model = aov(Y ~ factor(A)*factor(B), data=Data)
> summary(Model)
The #\mathtt{*}# indicates that we want a model with both main effects and their interaction.
As before we use the #\mathtt{factor()}# function to ensure that the two vectors containing categorical data are in the factor class. If the data vectors are not in a data frame but are in the #\mathrm{R}# workspace, you would not include the #\mathtt{data=}# setting.
To obtain the ANOVA table for the additive model, we use:
> Model = aov(Y ~ factor(A)+factor(B), data=Data)
> summary(Model)
The #\mathtt{+}# indicates that we want a model with both main effects but not their interaction.
For example, consider the following data, which you can paste into #\mathrm{R}#:
> Data = data.frame(Yield=c(86.8,82.4,86.7,83.5,93.4,85.2,94.8,83.1,77.9,89.6,89.9,
83.7,71.9,72.1,80.0,77.4,74.5,87.1,71.9,84.1,87.5,82.7,78.3,90.1,65.5,72.4,76.6,
66.7,66.7,77.1,76.7,86.1,72.7,77.8,83.5,78.8,63.9,70.4,77.2,81.2,73.7,81.6,84.2,
84.9,79.8,75.7,80.5,72.9), Catalyst=c(rep("A",12), rep("B",12), rep("C",12),
rep("D",12)), Reagent=c(rep(c(rep(1,4), rep(2,4), rep(3,4)), 4)))
The response variable is the Yield from a chemical process with Catalyst A, B, C, or D and Reagent 1,2, or 3.
Does the type of Catalyst or the type of Reagent have an effect on the Yield, and do these factors interact?
We investigate with a two-way ANOVA model in #\mathrm{R}#:
> Model = aov(Yield ~ factor(Catalyst)*factor(Reagent), data=Data)
> summary(Model)
The ANOVA table for the full model is:
The interaction effect is assessed in the third line, where the P-value is #0.549603#, indicating that the interaction effect is not significant, and we can use the additive model.
Although the P-values for both main effects are small, we should confirm that they are significant using the additive model:
> Model2 = aov(Yield ~ factor(Catalyst)+factor(Reagent), data=Data)
> summary(Model2)
The ANOVA table for the additive model is:
Both P-values are very small, so you can conclude that the type of Catalyst and the type of Reagent both have an effect on the Yield of the chemical process.
To create an interaction plot in #\mathrm{R}#, you can use the #\mathtt{interaction.plot()}# function. You must provide the factor for the horizontal axis, the factor for the separate lines, and the response variable. Choose the factor with the fewest levels for the separate lines (unless both factors have the same number of levels). There are further options for modifying the appearance of a plot. For our example:
> interaction.plot(Data$Catalyst, factor(Data$Reagent), Data$Yield,
trace.label="Reagent", xlab="Catalyst", ylab="Mean Yield", lty=1,
col=c("red","blue","green"), lwd=2)
gives us a beautiful interaction plot:
Here we see that two of the lines intersect, which would suggest an interaction effect. But our analysis found the interaction insignificant. This contradiction is most likely due to the within-group variance in the Yield being large relative to the between-group variance between Catalyst 2 and Catalyst 3. You cannot see the variance in this plot, only the means.
To compare the means between different levels of each factor simultaneously, we can use the Tukey-Kramer method with the additive model:
> TukeyHSD(Model2)
This output is:
From the adjusted P-values in this output we can see that we would conclude at the #0.05# significance level that the population mean Yield differs between Catalyst A and each of the other three catalysts, but none of the other three catalysts show a difference in mean Yield with each other.
Meanwhile, we would conclude at the #0.05# significance level that the population mean Yield differs between Reagent 1 and each of the other two Reagents, but no difference in mean Yield occurs between Reagent 2 and Reagent 3
To obtain the mean Yield for each level of each individual factor and each combination of levels for the two factors (when the interaction is included in the model), as well as the grand mean, you can use the #\mathtt{model.tables()}# command with the full model:
> model.tables(Model, type="means")
The output is:
Or visit omptest.org if jou are taking an OMPT exam.