What is efficacy or effectiveness?
What does performance test scores tell me?
In many educational studies, the efficacy or effectiveness of Web-based
instruction (WBI) is expressed in terms of performance test scores.
However, many performance tests for Web-based instruction may be just
tests of a short-term retention of memory.
The following figure, which is adapted from a lesson of time-series
studies by Caporaso (1973), illustrates the potential flaws of using
one-time performance test or a pretest-posttest design as a measure of
treatment efficacy or effectiveness.
- A: This indicates that the change is a result of maturation or natural growth.
- B: This behavioral pattern is a cycle with a
natural growth tendency. By chances the pretest may be given at the top
of the cycle and the posttest may be given at the bottom of the cycle,
or vice versa.
- C: This pattern resembles B except that there is no growth.
- D: This shows a short-term gain. It is not
uncommon that a new and exciting medium such as Web-based instruction
induces short-term improvement, but this effect does not last. After
the excitement is cooled down, the performance may be worse than before.
- E: This also shows a short-term gain. Even though the momentum does not carry on, the performance does not fall back.
- F: This indicates a long-term gain. Not only the
learner learned the knowledge, but also learned how to learn. This may
imply a permanent change in the cognitive mode of the learner. Certain
changes require multiple interventions before the changes can be
permanent. For example, if the educator wants to teach the learner
"non-linear thinking" and "visualization" through hypertext and
multimedia on the Web, the learner need to submerge himself into the
medium.
- G: It seems that there is no effect. However, the
effect is latent and thus it takes time for the treatment to show its
effect. Although the learner did not learn much information from the
treatment, he might be motivated to further utilize the medium in other
situations, or he might acquire a new tool and use it to solve other
problems. For example, the learner might not achieve a high test score
by using search engines in a web-based class, this tool is still
helpful for searching other information.
What shall we do?
It is important for the researcher to define what the predicted effects
are and design the experiment accordingly. The researcher shoulds ask
questions like:
- Are there manifest effects and latent effects?
- What mental construct does the performance test or the
measurement indicate? Is it concerned with short-term memory, long term
memory, motivational variables, problem solving, or/and reasoning?
- What kind of instrument is sensitive to these mental constructs?
- Is it necessary to take repeated measures or to conduct a time-series study?
It is not acceptable to report that there is a statistically
significant increase or decrease of test scores measured in one shot.
Reference
Caporaso, J. A. (1973). Quasi-experimental approaches to social science. In J. A. Caporaso & L L. Ross, Jr.,(Ed.) Quasi-experimental approaches. Evanston, IL: Northwestern University Press.
 Go up to the main menu
|
|
