Caveon Security Insights

By: David Foster, CEO, Caveon Test Security

Item exposure during an exam in the testing world is often viewed as a bad thing, because it seems obvious that item exposure leads to item over-use which in turn leads to item compromise. It is common for psychometricians to limit item exposure, defining it as either a too-high absolute number of presentations of the items in a test, or a too-high rate of the items presented on tests. Unfortunately, there is no scientific research or even unscientific guidelines, or even reasonable casual suggestions, about how many exposures are too many, or which rate of exposure is too high.

It does not follow that item exposure is the same as item compromise. In fact, I’ve seen items compromised with an extremely small number of presentations. Some items have even been compromised prior to the first test being administered!

In my opinion, the notion that item compromise results from item exposure—as defined above—leads  to improper conclusions, decisions, and ineffective procedures. I have a few reasons for this opinion, a couple of which I’ll give here. First, item exposure is absolutely necessary. It is obvious that no test can be effective unless its items are exposed during the exam. Test designers even let examinees view an item multiple times encouraging them to return to and review previous items again and again. Second, item compromise has very little to do with the definitions of item exposure given above. Consider this simple example: Suppose that an item was shown to one million test takers and was presented on every exam administered. This would be considered a very high number of exposures along with a 100% exposure rate. But, suppose that none of those examinees were able to share the item with others. In this simple example, the item remains uncompromised and perfectly secure, and can be continued to be used on the exam.

If we wish to reduce item compromise, the example illustrates that limiting the number of presentations or rate of presentations of an item is not as important as the methods used to secure the items, to protect them from theft, and to keep them from being used for cheating. For this reason we need improved item security, which means better ways to keep items from being stolen and used for cheating on subsequent exams. We need methods to detect when an item is truly compromised and then immediately to take it out of service. Instead, we often see stubborn adherence to a century-old model of relatively unsecure test administration, and believing that keeping an item from being presented on a test is a sensible way to secure it.

It is certainly possible to improve the way we secure items. As examples, there are protective item and test designs available, and certainly better test monitoring procedures, that we can use. And perhaps we can learn a little from other industries as well. Consider the problem with the theft of music over the Internet. No one would suggest that music is stolen because it was listened to by too many people. Instead, we see serious efforts to protect the music, to keep it from being stolen, to detect when it is stolen, and to punish those that are responsible. We should be doing the same.

We welcome comments below!

By: David Foster, CEO, Caveon Test Security

In 2010, a very useful book was published by The Council of Chief State School Officers and the Association of Test Publishers. It is titled Operational Best Practices for Statewide Large-Scale Assessment Programs. Caveon’s very own Dr. John Fremer contributed as part of the working committee to the overall effort and provided a chapter or two. As the title suggests, the book provides some “best practices” in a good many areas of interest to all testing professionals, particularly those involved in paper-and-pencil state assessments. A testing program can use the book to evaluate its own practices, and to guide efforts at change if necessary.

Given the intense interest today in delivering tests on the computer, it’s not a surprise that there was immediate interest in a revision of the book, one that would include best practices for programs using or wishing to implement technology-based tests. These are tests that are administered on computers via local servers, or delivered online through secure browsers. Choosing the specific technology used to administer the tests is not an easy chore and should be carefully done. The newest model, online testing—testing administered securely through browsers—is becoming more and more popular with high-stakes testing programs.

But what are we to think about the concept of best practices when a methodology is new and developing, when few organizations are experienced with it? How can a best practice even be identified with so little applied experience and when change accompanies that technology almost daily. It’s my opinion that our concept of what is a best practice has to evolve if we are to find it useful in the face of new and constantly changing technology.

To solve this conundrum I’d like to propose that we adopt a more accepting approach toward innovation and technology. This means that we should seriously consider innovations even though dozens or hundreds of other programs have not yet tried it out. This optimistic attitude is critical if we are to find these innovations immediately helpful, and, more importantly, if we are to set ourselves on a path to accommodate change occurring on a more constant basis. New technologies can be evaluated against reasonable criteria that reveal how the innovation will improve the reliability, validity, security and fairness of the tests. This is especially easy to do if by implementing the technology we are solving a long-standing concern or problem.  My own experience developing and using new technologies over the past 30 years has been very rewarding.

Just a word about standards and technology. Some feel that using new technology violates or threatens standards. That certainly hasn’t been my experience. Throughout my career, as I used new technologies in testing, I have found that in each case it enhanced my ability to meet the standards, rather than threaten them. An example may help here. In 1990 at Novell we implemented a new multiple choice question type that allowed for more than one correct answer. No one had used it before. It immediately helped us to eliminate confusion for our test takers from negatively worded multiple choice questions. There is no standard that states that multiple choice questions must only have a single correct answer, but there are standards that require us to improve the quality of our questions.

Now, a final word about statewide educational testing. The joint committee working on the revision of the Operational  Best Practices for Statewide Large-Scale Testing will provide a set of best practices in the coming months for technology-based tests. Hopefully these suggestions will be met with enthusiasm and optimism. If they are, statewide assessment programs will find it much easier to meet the very ambitious goals set by themselves, the federal government, and other stakeholders.

Being an election year, the security of voting machines is coming under intense scrutiny. The article states, “With the presidential race in full swing, some U.S. states have found critical flaws in the accuracy and security of their electronic voting machines, forcing officials to scramble to return to the paper ballots they abandoned after the 2000 Florida debacle.”

http://www.msnbc.msn.com/id/22454379

The backlash against security flaws that have been discovered by the states may very well prompt a return to paper-based balloting. “Vendors of the electronic voting machines warn against a rush back to paper.” Industry representatives state that we should “not throw the baby out with the bath water” and “trying to design a voting system when you don’t know how it’s being judged is causing a lot of problems.” State election officials just want secure, tamper-proof election systems.

Security professionals have serious reservations about the security of all voting machines that are based on general purpose operating systems. Such systems provide too many points of entry for an attacker and cannot be made completely secure. Bruce Schneier has a lot of very good essays on this topic. Here’s one of his latest:

http://www.schneier.com/blog/archives/2007/12/more_voting_mac_1.html

A return to paper may be viewed by some as being shortsighted. On the other hand, the security flaws in that system have been thoroughly studied and countermeasures are in place. We must learn that switching from paper ballot systems to electronic voting systems does not mean that security will be better. It means that the attacks and opportunities will be different.

Now, I have digressed. This is an essay on testing and exam security. The lesson to be learned from the introduction of voting machines is that technology does not automatically solve security problems. In fact, adoption of technologies introduces new security challenges. Managers of testing programs that migrate from paper-based testing to computer-based testing in the name of improved security need to be acutely aware that new, unforeseen, security challenges will emerge.

In the same way that most types of election fraud do not involve the actual mechanism by which the vote is cast, most types of test fraud do not involve the actual mechanism by which the test responses are recorded. If you accept the above statement, then you realize that security responses to test fraud transcend the test delivery mode.

Proponents of computer-based testing argue that the tests are more secure than paper-based delivery methods due to limited item exposure, strict time limits, sophisticated verification mechanisms, and strong encryption. After analyzing a lot of data I have come to realize that security is not the result of how the tests are deployed and delivered. I have learned that the above claims are groundless because good security comes from people who follow processes and procedures which have been designed to provide security. Security may be improved using technology, but good security is not derived from technological devices. Any computer system is hackable, especially if the attacker has unrestricted physical access to the system.

Here is a partial list of the “different” security issues that plague computer-based tests.

1 – If the tests are provided in an “on-demand mode” (i.e., scheduled for the test takers convenience in a prolonged testing window), organized test thieves may very quickly harvest the item banks. We have seen tests in this environment that are completely compromised within weeks or days of publication.

2 – Dishonest test site operators can completely reverse engineer the testing engine and database that are on the local machines.

3 – A video T-connector can be used to record any and all testing sessions, allowing easy theft of test items.

4 – Viruses and other forms of malware may be installed on the test delivery workstations allowing for keyboard logging or test result tampering (which would likely be undetectable).

The above attacks have their counterparts in the form of booklet theft, test form photocopying, or answer-sheet tampering in the paper-based testing environment. If you have individuals in positions of trust who you have not certified to be trustworthy, then the integrity of your testing process will be suspect. The point I am trying to make is that security attacks are not automatically thwarted when we use technology. Instead, good security comes from people who follow processes and procedures which have been designed to provide a secure testing environment.