Caveon Security Insights

By: Dennis Maynes, Chief Scientist, Caveon Test Security

This week, Julian Assange, founder of WikiLeaks, announced that his organization is running out of money and may be forced to cease operations by the end of 2011. On October 24, 2011 Reuters reported: “WikiLeaks says ‘blockade’ threatens its existence.” (Source: http://www.reuters.com/article/email/idUSTRE79N46K20111024) The blockade occurred when the major financial processing firms suspended their agreements with WikiLeaks, after WikiLeaks released thousands of secret US diplomatic cables in December, 2010, and threatened the Bank of America with the release of internal documents which resulted in a 3% decrease of Bank of America’s share price.

Assange claims the blockade is illegal and has filed anti-trust lawsuits against Visa and Master Card. On the day before the blockade, WikiLeaks received $135,000. Currently, WikiLeaks receives less than $10,000 per month. The net effect of the blockade to WikiLeaks has been the loss of 95% of its operating cash.

Whether you agree with WikiLeaks’ goals or not, it is clear that WikiLeaks has routinely infringed upon the rights of copyright holders by distributing information and documents without authorization. If it is not obvious why this story has important test security ramifications, let me make it clear: (1) many websites, operated by pirates and thieves, infringe upon the copyrights of secured exam content, (2) it has been very difficult to effectively shutdown this activity, which is costing testing organizations millions of dollars per year in lost test development expenditures, and (3) if payment processors would agree to cease providing services to these thieves and pirates, many of them would fold. The WikiLeaks story demonstrates that copyright infringers will have a difficult time remaining in business without the support of payment processors.

At Caveon, we have been very successful in removing copyrighted exam materials from the Internet. Often our success is based upon respectful and courteous requests to unintentional copyright infringers. However, respect and courtesy do not work against pirates and thieves. At that point, potentially expensive legal action must be commenced.

An alternative to expensive legal proceedings is to work with payment processors to protect their brands. For example, Visa does not want any transaction to bring disrepute upon its brand (source: http://corporate.visa.com/_media/visa-international-operating-regulations.pdf). If we, as an industry, can convince the payment processors that the sale and distribution of pilfered exam content is disreputable, we may be able to slow the flow of money to the test thieves and protect valuable exam content.

What do you think? How can we help payment processors understand that their services facilitate the distribution of stolen exam content? Should ATP (Association of Test Publishers) contact the payment processors, on behalf of its members?

Several months ago, Ben Mannes, Test Security Director at ABIM, expressed this thought: “ATP should be trying to get a meeting with Victoria Espinel [White House intellectual property czar], bring 1-2 industry security experts, and state the case as to why exam content is a vital component to our nation’s infrastructure requiring heightened public sector IP enforcement.”

***************

Please Comment Below, Thank you for Reading

By: David Foster, CEO, Caveon Test Security

In 2010, a very useful book was published by The Council of Chief State School Officers and the Association of Test Publishers. It is titled Operational Best Practices for Statewide Large-Scale Assessment Programs. Caveon’s very own Dr. John Fremer contributed as part of the working committee to the overall effort and provided a chapter or two. As the title suggests, the book provides some “best practices” in a good many areas of interest to all testing professionals, particularly those involved in paper-and-pencil state assessments. A testing program can use the book to evaluate its own practices, and to guide efforts at change if necessary.

Given the intense interest today in delivering tests on the computer, it’s not a surprise that there was immediate interest in a revision of the book, one that would include best practices for programs using or wishing to implement technology-based tests. These are tests that are administered on computers via local servers, or delivered online through secure browsers. Choosing the specific technology used to administer the tests is not an easy chore and should be carefully done. The newest model, online testing—testing administered securely through browsers—is becoming more and more popular with high-stakes testing programs.

But what are we to think about the concept of best practices when a methodology is new and developing, when few organizations are experienced with it? How can a best practice even be identified with so little applied experience and when change accompanies that technology almost daily. It’s my opinion that our concept of what is a best practice has to evolve if we are to find it useful in the face of new and constantly changing technology.

To solve this conundrum I’d like to propose that we adopt a more accepting approach toward innovation and technology. This means that we should seriously consider innovations even though dozens or hundreds of other programs have not yet tried it out. This optimistic attitude is critical if we are to find these innovations immediately helpful, and, more importantly, if we are to set ourselves on a path to accommodate change occurring on a more constant basis. New technologies can be evaluated against reasonable criteria that reveal how the innovation will improve the reliability, validity, security and fairness of the tests. This is especially easy to do if by implementing the technology we are solving a long-standing concern or problem.  My own experience developing and using new technologies over the past 30 years has been very rewarding.

Just a word about standards and technology. Some feel that using new technology violates or threatens standards. That certainly hasn’t been my experience. Throughout my career, as I used new technologies in testing, I have found that in each case it enhanced my ability to meet the standards, rather than threaten them. An example may help here. In 1990 at Novell we implemented a new multiple choice question type that allowed for more than one correct answer. No one had used it before. It immediately helped us to eliminate confusion for our test takers from negatively worded multiple choice questions. There is no standard that states that multiple choice questions must only have a single correct answer, but there are standards that require us to improve the quality of our questions.

Now, a final word about statewide educational testing. The joint committee working on the revision of the Operational  Best Practices for Statewide Large-Scale Testing will provide a set of best practices in the coming months for technology-based tests. Hopefully these suggestions will be met with enthusiasm and optimism. If they are, statewide assessment programs will find it much easier to meet the very ambitious goals set by themselves, the federal government, and other stakeholders.

As more and more online courses are developed and offered, instructors of online courses need to consider the potential for cheating on the assessments. The following article describes some measures being implemented by FGCU (Florida Gulf Coast University):

http://www.nbc-2.com/articles/readarticle.asp?articleid=16460&z=3&p=

One of the measures is to track IP addresses and determine if more than one test is being submitted from the same computer. Other measures include randomization of answer choices and random selection of items from an item bank. The software also prevents the test questions from being printed. Kathleen Davey, Dean of Academic Technology, said, “”You can’t prevent everything from happening. You must rely on the integrity of the individual students up to a certain point.”

Ultimately, the above statement is true. If a test taker is sufficiently determined he or she will be able to successfully cheat on the test or steal the test content.

I have been very interested lately in the security of online assessments. They are becoming more prevalent and indications are that they will become a dominant technology in testing if security concerns can be adequately addressed. The problem is that most online assessments are essentially unproctored assessments. Until unproctored Internet tests can be delivered securely, they should not be used for high-stakes exams. By definition, an exam has high stakes if passing or failing the exam has significant life consequences for the test taker. Usually this means getting a job, getting licensed in a profession, getting admitted to a school, getting a diploma, etc.

Recently, Boston Globe released an investigative report concerning Army Correspondence Courses. Yesterday, Senator Edward Kennedy M. Kennedy, Chairman of the Armed Services Committee, reacted strongly to the report, writing, “I was shocked to read of one website that provides answer keys and boasts that “[w]ith cheap prices and fast service, you can be wearing that E-5 [sergeant] rank before you know it.”

http://www.boston.com/news/nation/washington/articles/2007/12/19/kennedy_urges_army_to_deter_cheating_on_promotional_exams/

The essential problem is that the assessments being used for the correspondence courses are unproctored Internet tests.

I remember taking unproctored tests as a student at the university. We called them “take home” tests. Our take-home tests had implicit security built into them:

1. They were really hard. You couldn’t just find the answer to the questions in the university library.
2. You might find someone to take the test for you or help you out, but eventually you would take a few in-class tests (where you couldn’t use your friend).
3. The tests were written in your own handwriting, which was easily compared with prior copies of your handwritten assignments.

Later, as an instructor at the university we added another twist to take-home tests: Every student got the same problems but with different data and different answers.

The above simple principles highlight the issues that must be addressed to administer a test securely online in an unproctored setting:

1. Biometrics should be used to authenticate test taker identity.
2. The questions must not be answerable using simple “Google” searches.
3. A verification process needs to be in place that allows the unproctored test result to be trusted.
4. Other security measures may assist with authenticating that the test taker actually did his or her own work.
5. Algorithms that produce item clones or variants can reduce the ability of test takers to share test content or profit from another’s answers.

I remember the day that I took my oral exams. There was no faking. There was no cheating. I was in a room, face-to-face, with three professors. Each of them had taught me in at least one course. Of course, it is not realistic to do this for every single individual being certified in a profession or being admitted into the university. But, it demonstrates the importance of having several observations which together confirm that the candidate does indeed possess the requisite competence.There has been interesting progress in the area of secure administrations of unproctored Internet tests. I will mention just a few items that I can recall readily:

1. Kryterion (www.kryteriononline.com) is using data forensics and biometrics to establish that a test is being taken properly.
2. SHL (www.shlgroup.com) is using an initial unproctored test followed by a verification test in a proctored setting to ensure that the test results can be trusted.
3. An instructor named Simon at the School of DCIT, University of Newcastle, used an innovative detection system with online unproctored tests that relied on font colors in Word documents to detect cheaters: http://crpit.com/confpapers/CRPITV42Simon.pdf

At this URL: http://www.westga.edu/~distance/ojdla/summer72/rowe72.html you will find a paper that is very interesting in this context.

Two things are clear: (1) online assessment is here to stay, and (2) ubiquitous security solutions are needed if online assessments are to be trusted.