What is efficacy or effectiveness?
What does performance test scores tell me?
In many educational studies, the efficacy or effectiveness of Web-based instruction (WBI) is expressed in terms of performance test scores. However, many performance tests for Web-based instruction may be just tests of short-term memory.
The following figure, which is adapted from a lesson of time-series studies by Caporaso (1973), illustrates the potential flaws of using one-time performance test as a measure of treatment efficacy or effectiveness.
- A: This indicates that the change is a result of maturation or natural growth.
- B: This behavioral pattern is a cycle with a natural growth tendency. By chances the pretest may be given at the top of the cycle and the posttest may be given at the bottom of the cycle, or vice versa.
- C: This pattern resembles B except that there is no growth.
- D: This shows a short-term gain. It is not uncommon that a new and exciting medium such as Web-based instruction induces short-term improvement, but this effect does not last. After the excitement is cooled down, the performance may be worse than before.
- E: This also shows a short-term gain. Even though the momentum does not carry on, the performance does not fall back.
- F: This indicates a long-term gain. Not only the learner learned the knowledge, but also learned how to learn. This may imply a permanent change in the cognitive mode of the learner. Certain changes require multiple interventions before the changes can be permanent. For example, if the educator wants to teach the learner "non-linear thinking" and "visualization" through hypertext and multimedia on the Web, the learner need to submerge himself into the medium.
- G: It seems that there is no effect. However, the effect is latent and thus it takes time for the treatment to show its effect. Although the learner did not learn much information from the treatment, he might be motivated to further utilize the medium in other situations, or he might acquire a new tool and use it to solve other problems. For example, the learner might not achieve a high test score by using search engines in a web-based class, this tool is still helpful for searching other information.
What shall we do?
It is important for the researcher to define what the predicted effects are and design the experiment accordingly. The researcher shoulds ask questions like:
It is not acceptable to report that there is a statistically significant increase or decrease of test scores measured in one shot.
- Are there manifest effects and latent effects?
- What mental construct does the performance test or the measurement indicate? Is it concerned with short-term memory, long term memory, motivational variables, problem solving, or/and reasoning?
- What kind of instrument is sensitive to these mental constructs?
- Is it necessary to take repeated measures or to conduct a time-series study?
Caporaso, J. A. (1973). Quasi-experimental approaches to social science. In J. A. Caporaso & L L. Ross, Jr.,(Ed.) Quasi-experimental approaches. Evanston, IL: Northwestern University Press.
Go up to the main menu