Effects of Multicollinearity on Completed Models

Multicollinearity Defined
Multicollinearity, or collinearity, occurs when a regression model includes two or more highly related predictors. Peer smoking and perceptions of school smoking norms, for example, are likely to be correlated. If we entered both into a model, we would expect unstable parameter estimates and inflated standard errors. We might find that while either works as a statistically significant predictor of smoking, neither significantly predict smoking when both are included in the same model.
 
Multicollinearity also occurs in multilevel models when an individual-level predictor is highly correlated with a group-level predictor. Say we created a model that included an individual-level measure of peer smoking as well as a school-level measure of smoking norms and that we created the school-level from the aggregate of individual reports of peer smoking. These two measures, at different levels, could be highly correlated, create collinearity, and consequently introduce confounding into the model (Kreft & de Leeuw, 1998).
 
Collinearity can occur in ordinary least squares regression, maximum likelihood regression, logistic regression, hierarchical linear models, structural equation models, and the like. Below I focus on the effects of collinearity, how it influences completed models, but not so much on detection or remedies.
Effects of Multicollinearity
Correlated independent variables (IVs) create collinearity in a regression model when the relationship is very strong. The result is reduced stability of the corresponding parameter estimates, increased standard errors, and reduced power to measure effects. In the OHT Multilevel papers, we centered our individual-level predictors on the higher-level aggregate predictors in part to avoid multicollinearity.
 
Nearly all sources cited here describe how collinearity can create instability, inflate standard errors, and lower power. Kreft and de Leeuw (1998), for example, talk about how small change in one variable can create a large change in the regression estimates for another. The results of a regression, according to Cohen, Cohen, West, and Aiken (2003), can change considerably by dropping just a few cases when the model contains highly related predictors. The impact depends, however, on the specific model and precisely how it was tested. In testing a taxonomy of models that use R values or model fit statistics (e.g., AIC), collinearity does not affect results (Burnham & Anderson, 2002; Cohen et al., 2003). This is because collinearity influences the individual parameter estimates and not the overall level of variance accounted for, such as measured by changes in R values or improvements in fit. These global statistics do not depend on individual parameters and their standard errors. Redundant variables simply fail to improve model fit or R values because they do not add to the overall prediction.
 
Harrell (2001) nicely summarizes the issue in the opening to his section on collinearity ( 4.6): "When at least one of the predictors can be predicted well from the other predictors, the standard error of the regression coefficient estimates can be inflated and the corresponding tests have reduced power. In stepwise variable selection, collinearity can course predictors to compete and make the selection of 'important' variables arbitrary. Collinearity makes it difficult to estimate and interpret a particular regression coefficient because the data have little information about the effect of changing one variable while holding another (highly correlated) variable constant. However, collinearity does not affect the joint influence of highly correlated variables when tested simultaneously. Therefore, once groups of highly correlated predictors are identified, the problem can be rectified by testing the contribution of an entire set with multiple d.f. test rather than attempting to interpret the coefficient or one d.f. test for a single predictor" (p. 64-65).
 
A careful read of Harrell's collinearity section indicates, in addition, that multicollinearity would influence the standard errors, regression weights, and test statistics of only variables in question. Cohen, Cohen, West, and Aiken (2003) agree. Thus one could enter background variables in a block, even if highly correlated, without influencing the results of later blocks of IVs. This is very important, as it frees the statistician from worrying about relationships in sets of demographic variables, such as between students' academic performance and their parents' educational attainment.
Multicollinearity Examples
Consider Pierce, Choi, Gilpin, Farkas, and Berry (1998). They present a model that contains familial smoking and peer smoking as well as exposure to tobacco promotions and advertising. Their logistic regression analysis might contain collinearity if peer smoking and familial smoking were highly correlated. Harrell notes that there is "little information about the effects of changing one variable while holding another . . . constant" (p 64). The large sample (N = 1752) in Pierce et al. should, in part, compensate for the collinearity. That is, collinearity is a larger problem in smaller samples given a fixed correlation between IVs. Another way to look at it is that the correlation between offending IVs needs to be greater in larger samples in order to cause the same level of problems.
 
Collinearity, then, is a problem of degree. There is no single metric to measure collinearity and with the multiple measure, there is no cut point at which experts suddenly consider collinearity a problem. Its effects on individual parameter estimates increase gradually as a function of the increase in relationships among the IVs. Cohen, Cohen, West, and Aiken (2003) give a nice example (Table 10.5.1, p. 421) that demonstrates the effects of collinearity on two variables as their correlation increases from 0.00 to 0.95 with 100 cases. In their example, a correlation as high as 0.50 has little impact on the model, but by the time the correlation between the predictors reaches 0.90, the standard errors over double those in the model with zero correlation. In the Pierce et al. (1998) article, with over 1700 cases, the correlation between familial smoking and peer smoking would need to be larger than 0.50 to have an effect.
 
Harrell goes on to point out that the effects of collinearity only occur within the set of highly related variables. That is, collinearity between familial smoking and peer smoking in Pierce et al. (1998) affect the estimates for familial smoking and peer smoking, but not the estimates for exposure to tobacco promotions and advertising. Harrell implies that even in the case of substantial collinearity between familial and peer smoking, the estimates for exposure would be unaffected.
 
Cohen, Cohen, West, and Aiken (2003) give a very nice example of this phenomenon. Below I reproduced Tables 10.5.2 A and 10.5.2 B. The first contains a zero correlation among all four variables and reports the estimate (B), standard error (SE), partial correlation (pr), the tolerance, and variance inflation factor. The right-most two columns are standard measures of collinearity. The second contains a correlation among the first three of 0.933, but the fourth remains independent. Note how the estimates and standard errors change substantially for X1 through X3 but the estimates for X4 remain essentially unaffected.
 
Reproduction of Table 10.5.2 A in Cohen, Cohen, West, and Aiken (2003)
r12 = r13 = r23 = 0.00,   r14 = 0.00
Variable
B
SE
pr
Tol
VIF
Intercept
 20.000
  0.162
X1
0.387
0.094
  0.151
  1.000
  1.000
X2
0.391
0.082
0.195
1.000
1.000
X3
0.400
0.073
0.240
1.000
1.000
X4   
0.391
0.082
0.195
1.000
1.000
 
Reproduction of Table 10.5.2 B in Cohen, Cohen, West, and Aiken (2003)
r12 = r13 = r23 = 0.933,   r14 = 0.00
Variable
B
SE
pr
Tol
VIF
Intercept
 20.000
  0.187
X1
-0.086
0.345
  0.054
  0.099
 10.067
X2
0.137
0.299
0.002
0.099
10.067
X3
0.868
0.267
0.100
0.099
10.067
X4   
0.391
0.094
0.154
1.000
1.000
 
From this example, any collinearity in familial smoking and peer smoking in the Pierce et al. (1998) model would have little or no impact on the estimates for exposure to tobacco promotions and advertising. I make this claim because 1) Pierce et al. use a much larger sample, 2) if collinearity exists, it is to occur likely between variables other than tobacco promotions and advertising, and 3) correlations between variables modeled by Pierce et al. are not likely to reach high enough levels to cause problems, much greater than r = 0.50.
Multicollinearity Detection
Nearly every source that talks about multicollinearity contends that it increases the standard errors and that the inflation would be substantial. In the example above, the standard errors triple. Hosmer and Lemeshow (1989) state that "in all cases the tip-off for a problem [of collinearity] comes from the aberrantly large estimated standard errors," which will lead to huge confidence bounds. They give an example where a standard error jumps from 1.0 with one variable in a model to 444.3 with two highly correlated variables in the model. In many cases, then, professional judgment should easily detect excessive standard errors.
 
During the analysis process, there are ways to detect collinearity. The tolerance and variance inflation factor in the above tables are examples. Steps can also be taken to reduce collinearity during analysis if desired. In some cases, however, retaining highly correlated variables in a block might be preferred (see Harrell, 2001). Some methods to reduce collinearity include combining variables, creating latent variables, centering measures, choosing a different modeling strategy, or focusing on changes in fit statistics or R values rather than individual parameter estimates. I'm not going to discuss detection and remedies here, however, as the topic would require considerable time and effort. For measures to detect and resolve collinearity problems, see Cohen, Cohen, West, and Aiken (2003), Harrell (2001), Hosmer and Lemeshow (1989), Kreft and de Leeuw (1998), Myers (1990), Singer and Willett (2003), or other texts on regression methods.
Summary
Collinearity can change parameter estimates, increase standard errors, and reduce the power to detect reliable effects of correlated variables in a regression model. As demonstrated, collinearity affects only those IVs highly correlated with other IVs, and this may involve a set of IVs. Collinearity does not affect other IVs in the model nor any summary or fit statistics, such as R, AIC, or BIC values.
 
Critics of an article, such as Pierce et al. (1998), might claim that collinearity should obscure the results and call the conclusions into question. As a retort, however, one would justifiably claim that if collinearity exists, it should apply only to the estimates of the correlated variables, and in any case and by every account (every methodological text cited here) the reported significance values would be an underestimate of the true relationship.
References
 
-Keith Smolkowski (Thursday, November 04, 2004 1:20 pm)
 
Keith Smolkowski, Research Analyst Phone:  541.484.2123
Oregon Research Institute Fax:  541.484.1108
1715 Franklin Boulevard E-mail:  keiths@ori.org
Eugene, Oregon, 97403 WWW:  www.ori.org/~keiths/