Test Security Insights from ATP 2004

Like most individuals who attended the ATP (Association of Test Publishers) conference this year, our Caveon staff had great experiences: learning about new technologies, hearing about different applications and case studies of testing, and meeting with new friends as well as old. One thing we noticed was the overwhelming buzz around test security. To that end, we’ve put together some of key themes we heard from conference attendees.

High Interest in Test Security
There was very good attendance at sessions, heavy traffic at our Caveon booth, many requests for proposals and other follow up activities such as visits, and much attention to our “Cheating in the News” weekly e-mail.

Multiple Sources for Security Problems
It was clear that security problems come in many sizes and shapes. They threaten not only new computerized testing efforts, but also longstanding and well-established paper-and-pencil testing programs. The scope of the problems was mentioned in conference presentations and in questions posed at sessions and was further reinforced by a good deal of hallway talk and by comments we heard from folks who sought us out at the meeting.

Legal Cases Carefully Studied
Many attendees were very interested in the experience of those who had taken legal action against cheaters. A clear message from the sessions was do not undertake lawsuits lightly. It is not easy and it can be very expensive and time consuming. Going to the courts should be a “last resort” action and taken only when the case is extremely solid.

Becoming More Pro-active
Many companies are coming to the conclusion that we need to regain the initiative in the area of test security and take back control of this key aspect of testing. People feel that we need to band together and build our collective strength. There is considerable interest in sharing what has worked and what has not been as successful in deterring and pursuing cheaters. Also those who have pursued cheaters successfully point out that publicizing widely examples of successful efforts to thwart cheaters is good not just for a specific program but for our industry and field.

Systematic Searching of Web and Other Media 
There is substantial interest in finding more systematic ways to pursue what is appearing on the web and in other media regarding particular programs. Are live items showing up on the web? If so how can we shut down such sites and how much damage has the program suffered? The Caveon Web Patrol tool is directly responsive to this widely felt need.

International Scope
The security problem is an international one, affecting all cultures and types of high stakes testing programs. At this year’s ATP meeting there seemed much less inclination to view the security problem as only affecting a small number of countries, e.g., China and India, for example.

Budgeting and Return on Investment 
Security is often not a significant part of a testing program’s annual budget. What is a reasonable amount to spend on such efforts? Part of what is needed to answer that question is knowledge of the costs associated with breaches and the economic benefits of corrective actions. Simply put, is it a good business decision to invest more in test security? How much and to what effect? Caveon can help companies address these questions.

Early Detection of Problems
How can we spot a security problem early, before it escalates into a forest fire? There is much interest in Caveon’s Data Forensics service to obtain an early warning that particular countries or specific test centers are displaying anomalous results. Such aberrant results are often strongly suggestive of improper behavior, such as stealing items or responding with explicit prior exposure to questions.

Interest in Security Audits
Caveon’s Security Audit Service is tremendously appealing to testing programs. The idea of an independent and objective evaluation of program practices and procedures makes sense to people and they see many advantages to finding out before the problems occur where their vulnerabilities lie and how they can deal with them.

Willingness to Learn and Change
The ATP attendees seem very willing to modify policies and procedures in order to improve test security. Many business and practical factors are supporting this approach, but there also seems to be a strong sense that it is simply not right to allow cheaters to undermine the credentials of the vast majority of honest test takers.

John Fremer

President of Consulting Services, Caveon Test Security