Problem
I am a graduate student who is conducting a research project for my
thesis. I can't wait to graduate! I would like to find out whether my
instrument is reliable in order to proceed with my
experiment. I heard about using alternate forms and testretest to
estimate reliability. But due to lack of resources, I cannot afford to
write two tests or administer the same test in two different
times. With only one test result, what should I do to evaluate the
reliability of my measurement tool?
Which reliability coefficients should I use?
You may compute Cronbach Coefficient Alpha, Kuder Richardson (KR) Formula, or splithalf Reliability Coefficient to check for the internal consistency within a single test.
Cronbach Alpha is recommended over the other two for the following reasons:
 Cronbach Alpha can be used for both binarytype and largescale
data. On the other hand, KR can be applied to dichotomouslyscored data
only. For example, if your test questions are multiple
choices or true/false, the responses must be binary in nature (either
right or wrong). But if your test is composed of essaytype questions
and each question worths 10 points, then the scale is
ranged from 0 to 10.
 splithalf can be viewed as a onetest equivalent to alternate
form and testretest, which use two tests. In splithalf, you treat one
single test as two tests by dividing the items into two
subsets. Reliability is estimated by computing the correlation between
the two subsets. For example, let's assume that you calculate the
subtotal scores of all even numbered items and the subtotal of
all odd numbered items. The two sets of scores are as the following:
Students/Items 
Odd 
Even 
1 
10 
9 
2 
7 
5 
3 
8 
7 
4 
9 
10 
5 
11 
10 
You can simply calculate the correlation of these two sets of scores to check the internal
consistency. The key is "internal." Unlike testretest and alternate
form that require another
test as an external reference, splithalf uses test items within the
same test as an internal reference. If the correlation of the two sets
of scores is low, it implies that some people received high
scores on odd items but received low scores on even items while other
people received high scores on even items but received low scores on
odd items. In other words, the response pattern is
inconsistent.
The drawback is that the outcome is determined by how you group the
items. The default of SPSS is to divide the test into first half and
second half. A more common practice is to group oddnumber
items and evennumber items. Therefore, the reliability coefficient may
vary due to different grouping methods. On the other hand, Cronbach is
the mean of all possible splithalf coefficients that
are computed by the Rulon method.
What is Cronbach Alpha Coefficient?
"OK, Cronbach Alpha is good. But what is Cronbach Alpha?" Cronbach
Alpha coefficient is invented by Professor Cronbach, of course. It is a
measure of squared correlation between observed scores and
true scores. Put another way, reliability is measured in terms of the
ratio of true score variance to observed score variance. "Wow! Sound
very technical. My committee will like that. But what does
it mean?"
The theory behind it is that the observed score is equal to the
true score plus the measurement error (Y = T + E). For example, I know
80% of the materials but my score is 85% because of lucky guessing.
In this case, my observed score is 85 while my true score is 80. The
additional five points are due to the measurement error. A reliable
test should minimize the measurement error so that the error is
not highly correlated with the true score. On the other hand, the
relationship between true score and observed score should be strong.
Cronbach Alpha examines this relationship.
How to compute Cronbach Alpha
Either SAS or SPSS can perform this analysis. SAS is a better choice
due to its better detail. The following illustration is based upon the
data of the Eruditio project, which is sponsored by U.S.
West Communications. The SAS syntax to run Cronbach Alpha is as the
following:
Data one; input post_em1post_em5; cards; 1 1 1 0 0 1 1 0 1 1 1 0 1 1 1 1 1 1 0 0 0 1 1 1 0 1 0 1 0 1 proc corr alpha nocorr nomiss; var post_em1post_em5; run;

In this example, the "nocorr" option suppresses the item correlation
information. Although the correlation matrix can be used to examine
whether particular items are negatively correlated with others, a more
efficient way is to check the table entitled "if items are deleted¡K"
This table tells you whether particular items are negatively correlated
with the total and thus it is recommended to suppress the correlation
matrix from the output. "If items are deleted¡K" will be explained in a
later section.
It is important to include the "nomiss" option in the procedure
statement. If the test taker or the survey participant did not answer
several questions, Cronbach Alpha will not be computed. In surveys, it
is not unusual for respondents to skip questions that they don't want
to answer. Also, if you use a scanning device to record responses,
slight pencil marks may not be detected by the scanner. In both cases,
you will have "holes" in your data set and Cronbach Alpha procedure
will be halted. To prevent this problem from happening, the "nomiss"
option tells SAS to ignore cases that have missing values.
However, in the preceding aproach, even if the test taker or the
survey participant skips one question, his entire test will be ignored
by SAS. In a speeded test where test taker or the survey participants
may not be able to complete all items, the use of
"nomiss" will lead to some loss of information. One way to overcome
this problem is to set a criterion for a valid test response. Assume
that 80 percent of test items must be answered in order to be
included into the analysis, the following SAS code should be
implemented:
Data one; infile "c:\data"; input x1x5; if nmiss(of x1x5) > 1 then delete; array x{I} x1x5; do I=1 to 5; if X(I) =. then X(I) = 0; proc corr alpha nomiss; var x1x5; run;

In the preceding SAS code, if a record has more than one unanswered
questions (80%), the record will be deleted. In the remaining records,
the missing values will be replaced by a zero and thus
these records will be counted into the analysis.
It is acceptable to count missing responses of a test as wrong
answers and assign a value of "zero" to them. But it is not appropriate
to do so if the instrument is a survey such as an attitude
scale. One of the popular approaches for dealing with missing data in
surveys is the mean replacement method (Afifi ∓ Elashoff, 1966), in which means are used to replace missing data. The
SAS source code for the replacement is the same as the preceding one except the following line:
if X(I) =. then X(I) = mean(of x1x5);

Although this popular method
is convenient, it may not be the best. SAS provides users with various
imputation methods, such as PROC MI and PROC MIANALYSE. Fully conditional specification (FCS) method is recommended (Berglund & Heeringa, 2014)..
How to interpret the SAS output
Descriptive statistics
The mean output as shown below tells you how difficult the items are.
Because in this case the answer is either right (1) or wrong (0). The
mean is ranging from 0 to 1. 0.9 indicates that the
question is fairly easy and thus 90% of the test taker or the survey
participants scored it. It is a common mistake that people look at each
item individually and throw out the item that appears to be too
difficult or too easy.
Actually you should take the entire test into consideration. This will
be discussed later.
Raw and standardized Cronbach Coefficient Alpha
Cronbach Coefficient Alpha
for RAW variables : 0.305078 for STANDARDIZED variables: 0.292081

Cronbach Alpha procedure returns two coefficients:
 Standardized: It is based upon item correlation. The stronger the items are interrelated, the more likely the test is consistent.
 Raw: It is based upon item covariance. Variance is a measure of how a distribution of a single variable (item) spreads out. Covariance is a
measure of the distributions of two variables. The higher the correlation coefficient is, the higher the covariance is.
Some
people mistakenly believe that the standardized Alpha is superior to
the raw Alpha because they thought that standardization normalizes
skewed data. Actually standardization is a linear transformation, and
thus it never normalizes data. Standardized Alpha is not superior to
its raw counterpart. It is used when scales are comparable, because as
mentioned before, variance and covariance are taken into account for
computation. The concepts of variance and covariance are better illustrated
graphically. In one variable, the distribution is a bellcurve if it is
normal. In two variables the distribution appears to be a mountain as
the following.
In the above example, both item1 and item2 has a mean of zero
because the computation of covariance uses standardized scores
(zscore). From the shape of the "mountain," we can tell whether the
response patterns of test taker or the survey participants to item1 and
item 2 are consistent. If the mountain peak is at or near 'zero' and
the slopes of all directions spread out evenly, we can conclude that
the items are
consistent.
However, in order to determine whether the entire test is
consistent, we must go beyond just one pair. Cronbach Alpha computation
examines the covariance matrixall possible pairs to draw a
conclusion. But not all the information on the matrix is usable. For
example, the pairs of the item itself such as (item1, item1) can be
omitted. Also, the order of the pair doesn't matter i.e. the
covariance of pair ( item1, item2) is the same as that of (item2, item1)

Item1 
Item2 
Item3 
Item4 
Item5 
item1 

Covariance 
Covariance 
Covariance 
Covariance 
item2 


Covariance 
Covariance 
Covariance 
item3 



Covariance 
Covariance 
item4 




Covariance 
item5 





The higher the Alpha is, the more reliable the test is. There isn't
a generally agreed cutoff. Usually 0.7 and above is acceptable
(Nunnally, 1978). It is a common misconception that if the Alpha is
low, it must
be a bad test. Actually your test may measure several
attributes/dimensions rather than one and thus the Cronbach Alpha is
deflated. For example, it is expected that the scores of GREVerbal,
GREQuantitative, and GREAnalytical may not be
highly correlated because they evaluate different types of knowledge.
If your test is not internally consistent, you may want to perform
factor analysis to combine items into a few factors. You may also drop
the items that affect the overall consistency, which will
be discussed next.
It is very important to notice that Cronach Alpha takes variance
(spread of the distribution) into account. For example, when you
compare the mean scores in the following two tables, you can find
that both pretest and posttest responses are consistent,
respectively. However, the Alpha of posttest is only .30 (raw) and .29
(standardized) while the Alpha of pretest is as high as .60 (raw
and standardized). It is because the standard deviation (SD) of the
posttest ranges from .17 to .28 but the SD of the pretest is more
consistent (.42.48).
If the item is deleted...
As I mentioned before, a good analysis of test items should take the
whole test into consideration. The following table tells you how each
item is correlated with the entire test and what the Alpha
will be if that variable is deleted. For example, the first line shows
you the correlation coefficient between posttest item 1 and the
composite score of posttest item1item5. The first item is
negatively correlated with the total score. If it is deleted, the Alpha
will be improved to .41 (raw) or .42 (standardized). Question 5 has the
strongest relationship with the entire test. If this
item is removed, the Alpha will be dropped to .01 (raw) or .04
(standardized). This approach helps you to spot the bad apple and
retain the good one.
Once again, variance plays a vital role in Cronbach Alpha
calculation. Without variance there will be no result. The following
quetions are from another posttest. Everybody scored Question 3 and
4 (1.00) but missed Question 4 (0.00). Because there is no variance,
standardized Cronbach Alpha, which is based on covariance matrix,
cannot be computed at all.
References
Afifi, A. A., & Elashoff, R. M. (1966). Missing observations in multivariate statistics. Part I. review of the literature. Journal of the American Statistical Association, 61, 595604.
Berglund, P. & Heeringa, S. (2014). Multple imputation of missing data using SAS. Cary, NC: SAS Institute.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGrawHill.
Navigation
Contact me
