Risk Management: Protecting Our Tests
Originally Published in Certification Magazine, 9/2004
Risk management is a key component of doing business in any industry. Were a business manager to neglect risk managementwhen running an amusement park, the results could be catastrophic. Similarly, financial managers must be keenly aware of the risks to financial assets and take steps to minimize the “the downside.” But what about testing/credentialing program managers? What risks are associated with creating and deploying tests and how can they be minimized?
Make no mistake, high stakes tests (i.e. relied upon for making important decisions) provide incentives for a few unscrupulous individuals to steal and distribute the test content and for others to acquire and use that content. The mere act of using a “test as a test” in other words, introduces the concomitant risk that certain examinees will resort to deception to “beat the test.”
What’s at risk? First and foremost, the reputation of your organization and its credentials—“qualification of the unqualified” as witnessed in several recent examples, is a prescription for cynicism about a program’s goals and eventually, its irrelevance. Beyond that, intellectual property is at risk—the contents of the tests themselves. Unanticipated cost, of course, is the most familiar way to summarize what is jeopardized by a failure to anticipate and manage risk. For testing programs therefore, the costs associated with test theft and various forms of cheating can be roughly summarized as follows:
- Public relations and marketing costs (loss of program reputation/credibility);
- Economic cost (i.e.; loss of testing revenue and other related revenue streams resulting from the loss of program reputation/credibility);
- Measurement cost (loss of test measurement reliability and test utility);
- Opportunity cost (dealing with emergencies at the expense of other activities); and
- Replacement cost (based only on the time and expense involved in creating such tests, for many testing programs such as Wechsler, the SAT and others, the value of test content can equal thousands or millions of dollars as well as 6 to 24 months of lost opportunity).
Just as a systems engineer must be vigilant in identifying and minimizing possible points of failure, in the testing industry, program managers need to manage risks of test disclosure and cheating. While the following list of activities is not comprehensive, it provides a general idea of where risk management principles (aka: test security) can be applied to testing programs:
- Item development: During this phase of test development up through beta testing there are many people involved with the creation of test content—contract item writers, subject matter experts, translators, beta test participants. Appropriate security measures should be applied to prevent loss or questions regarding ownership of the developed content.
- Test delivery: Test delivery involves two types of risk: the first at the point of test disclosure to examinees, and the second when the examinee formulates and provides a response. Appropriate security measures should be applied to ensure that both the exposure and response are trustworthy.
- Data Management: Even after the test items are safely locked away, the credential can still be compromised by data mismanagement. Routine back-ups and reconciliations should be performed to ensure that test response data is not corrupted.
Each of these topics, as well as methods for calculating your return on test security investments and case studies of best and worst test security practices will be discussed in future issues of this periodical. It’s my hope that this column will become a valuable reference for testing program managers. Address any test security questions or recommendations to me via e-mail.