Analogy in the subject space
In subject space variables are represented by
vectors. Again, we compare a regression model constructed by vectors to
a physical object. The usability of a model is defined by its
stability. In the following left panel, the predictors X1 and X2 are
totally unrelated to the outcome Y. Therefore the vectors point to
different directions. The body in the right picture is analogous to the
left panel. My two legs are X1 and X2 and my upper body is Y. When they
go to opposite directions, my body cannot stand very well.
In a collinear case, X1 and X2 are very close to each other, as shown
in the left panel below. The right picture demonstrates how unstable
this model is. My two legs, X1 and X2, are too close. Again, my upper
body may collapse at any moment.
To avoid collinearity, the predictor vectors should maintain a distance
away from each other. When the two vectors are orthogonal (90 degree),
as shown in the left panel below, the model is said to be
well-supported. The following right picture shows a well-balanced body
with the same orientation. Interesting enough, Chinese martial art also
emphasizes the balance of the body. One of the best supported posture
is orthogonal feet!
When vectors are connected to form a volume, the condition of the model
could be detected as to whether or not it is well-built. The following
figures show a cubic-like structure representing a well-conditioned
model constructed of orthogonal vectors, and a wafer-like object
representing an ill-conditioned model constructed of multicollinear
vectors.
