Testing Technology: Take One Giant Step Backward
Originally Published in Certification Magazine, 5/2004
I’m disappointed. Technology has let me down.
In 1982 some colleagues and I created computerized tests for the IBM PC Junior. These may have been some of the first tests ever designed for microcomputers. The tests were adaptive and used animation and graphics to display math, reading and language arts test questions to schoolchildren. We felt like we were beginning a revolution that would make testing easier, cheaper, more convenient and maybe even more fun. Later, at Novell we built a testing program that helped the IT industry’s first certification program grow to more than 40,000 tests each month. By the time I left in 1997, over 500,000 individuals had certified on Novell’s programs, taking tests on the computer when and where they wanted.
It seemed to me that the marriage of testing and technology was a match made in heaven. The advantages of testing using computers were many and significant. I believed that the switch to computerized testing from paper-and-pencil testing was inevitable because of those advantages. And more and more large testing programs were making that switch.
But today, only a few years later, things have changed. Many large testing organizations have put their conversion efforts on hold, opting to continue with relatively archaic paper-and-pencil testing. And at least one has taken its tests off the computer and returned to paper-and-pencil testing. Hard to believe, I know, but these are not small programs and are influential leaders in the testing industry.
What has caused this abrupt distrust of technology? What force is strong enough to override the many advantages of computerized testing? What could cause us to unplug the microwave and cook over open fires again? Or shave with a sharp rock? Or send letters instead of e-mail?
The answer is: cheaters! I hate to admit it, but cheaters have successfully blocked progress in testing. Computerized tests, given on demand, are more easily stolen. Let me give you an example. On the first day a computerized test is available, its questions can be memorized or stolen and then shared with others who will take the test later. The strategy can be effective because those questions are re-used on later tests. This doesn’t happen with paper-and-pencil tests because they are generally given at the same time nationwide. Then new questions are created for the next time the test is administered, usually six months to a year later.
The concern of testing organizations when test content is stolen is over the investment they’ve lost, to be sure, but it’s much more than that. Once a test has been compromised, it is no longer effective. It can’t measure knowledge, skills or abilities well any more. People will be certified who don’t deserve to be and will take jobs that should go to competent individuals.
It’s ironic that the cheaters are using technology effectively to commit this fraud. Cell phones, two-way radios, digital cameras and hidden microphones are common devices used in testing centers to steal items or receive inappropriate help. These devices are getting smaller and more difficult to detect. After the test is over, they use the Internet to discuss specific test questions and to immediately share or sell them to others around the world. In the movie “Simone,” Al Pacino played a director who created a digital actress whom everyone thought was real. Referring to his use of technology, he said, “Our ability to manufacture fraud is now greater than our ability to detect it.” That statement is true for testing as well.
Testing folks like me are not technologists by nature. They are scientists, academicians, businesspeople, psychometricians and statisticians. They use technology tentatively. Making the leap to computerized testing has not been easy. Most would prefer to continue with traditional modes of testing. Test-takers on the other hand, most of whom are younger, are more technically savvy and, for whatever reasons, less concerned about cheating or stealing someone else’s copyrighted product. The ongoing difficulties in the recording industry provide evidence enough of this difference.
I’m a little at a loss for what to do, except that I created Caveon, which has as its mission to use technology to detect cheaters and fraud during exams or afterward on the Internet. I’m a bit uncomfortable in the role of industry detective, but the problems are caused in each case by someone trying to cheat or make money at the expense of others. There is no moral justification for their actions, and none of us should tolerate it any longer.