Caveon Security Insights

By: Steve Addicott, Caveon Vice President

October is an important month for Caveon. Eight years ago in October, 2003, several assessment industry veterans formed a small consulting company focused solely on improving the security of our clients’ test programs.    That company is Caveon Test Security!

Fast forward to 2011, and it’s gratifying to consider what this entrepreneurial group of test security zealots has accomplished.  Since that fateful October day, we have

• conducted over 50 Security Audits of leading test organizations and vendors,
• flagged and removed tens of thousands of internet-based risks, and
• conducted statistical analyses of over 30,000,000 test instances for many of the largest, most important test programs in the world.

As I consider the number and breadth of these engagements, perhaps it is worth sharing a few of the core values under which we always operate:

Confidentiality

Throughout our years of operation, one fundamental operating principle has always applied:  client confidentiality.  We never reveal the details of our client engagements without the express approval of our clients. Our clients require and appreciate this sensitivity as we investigate security incidents and provide reports on our forensic analyses. This is not secrecy– this privacy stems from respect for our clients and for the right to privacy of individuals and organizations.

Innovation

We constantly strive to improve means and methods for strengthening exam security. We are always interested in sharing the nature of our work.  Not only do we share our methods and science with clients, client stakeholders, TAC members, educational measurement researchers, and other appropriately interested parties; we are committed to furthering the science around test security. We regularly present at conferences and webinars where we openly share our Caveon approach, theories and methodologies. In fact this last year, we have presented at conferences in Phoenix, Orlando, Chicago, Seattle, Washington DC, Hong Kong, and Prague.

Conservative Recommendations

When we conduct an engagement, our approach is to focus on the situations and incidents that are most egregious, as evidenced in the data and the results that we analyze. We highlight those problems that are most readily identified, documented, and ideally, resolved. Dealing with these problems effectively will have the greatest positive impact to the overall validity and security of test results. This reasonable approach helps our clients, most of which suffer from ever-constrained budgets and resources, effectively concentrate their time, resources, and dollars where the likelihood of inappropriate test taking is highest.

Lastly, our growth and success is directly attributable to a few overarching principles—We always strive to exceed our clients’ expectations, comport ourselves honorably, provide valuable services, and share, as openly and honestly as we can, recommendations for improving the fairness and validity of our clients’ test programs. These principles result in proven, practical protection for our clients, and we intend to follow them for another eight years!

Please Submit Your Comments Below. Thank you!

By: David Foster, CEO, Caveon Test Security

In 2010, a very useful book was published by The Council of Chief State School Officers and the Association of Test Publishers. It is titled Operational Best Practices for Statewide Large-Scale Assessment Programs. Caveon’s very own Dr. John Fremer contributed as part of the working committee to the overall effort and provided a chapter or two. As the title suggests, the book provides some “best practices” in a good many areas of interest to all testing professionals, particularly those involved in paper-and-pencil state assessments. A testing program can use the book to evaluate its own practices, and to guide efforts at change if necessary.

Given the intense interest today in delivering tests on the computer, it’s not a surprise that there was immediate interest in a revision of the book, one that would include best practices for programs using or wishing to implement technology-based tests. These are tests that are administered on computers via local servers, or delivered online through secure browsers. Choosing the specific technology used to administer the tests is not an easy chore and should be carefully done. The newest model, online testing—testing administered securely through browsers—is becoming more and more popular with high-stakes testing programs.

But what are we to think about the concept of best practices when a methodology is new and developing, when few organizations are experienced with it? How can a best practice even be identified with so little applied experience and when change accompanies that technology almost daily. It’s my opinion that our concept of what is a best practice has to evolve if we are to find it useful in the face of new and constantly changing technology.

To solve this conundrum I’d like to propose that we adopt a more accepting approach toward innovation and technology. This means that we should seriously consider innovations even though dozens or hundreds of other programs have not yet tried it out. This optimistic attitude is critical if we are to find these innovations immediately helpful, and, more importantly, if we are to set ourselves on a path to accommodate change occurring on a more constant basis. New technologies can be evaluated against reasonable criteria that reveal how the innovation will improve the reliability, validity, security and fairness of the tests. This is especially easy to do if by implementing the technology we are solving a long-standing concern or problem.  My own experience developing and using new technologies over the past 30 years has been very rewarding.

Just a word about standards and technology. Some feel that using new technology violates or threatens standards. That certainly hasn’t been my experience. Throughout my career, as I used new technologies in testing, I have found that in each case it enhanced my ability to meet the standards, rather than threaten them. An example may help here. In 1990 at Novell we implemented a new multiple choice question type that allowed for more than one correct answer. No one had used it before. It immediately helped us to eliminate confusion for our test takers from negatively worded multiple choice questions. There is no standard that states that multiple choice questions must only have a single correct answer, but there are standards that require us to improve the quality of our questions.

Now, a final word about statewide educational testing. The joint committee working on the revision of the Operational  Best Practices for Statewide Large-Scale Testing will provide a set of best practices in the coming months for technology-based tests. Hopefully these suggestions will be met with enthusiasm and optimism. If they are, statewide assessment programs will find it much easier to meet the very ambitious goals set by themselves, the federal government, and other stakeholders.

By David Foster, President, Caveon Test Security

Over 20 years ago, the first high-stakes tests were delivered using computers. The movement of using technology to administer tests was launched. The many advantages of the new ways of testing were realized almost immediately. These included the convenience of being able to take the test at any time and receive an immediate score report. Unfortunately, the new exams were more susceptible to theft and cheating because tests were given during a long testing window. When the same test is given over a long time frame, say, weeks or months or even years, opportunities exist for people early in the period to steal the questions and share them with people later in the window. How the questions are stolen isn’t all that important, but testing programs have never really come up with good security solutions. The problem remains as bad today as it has ever been.

It’s not that the problem can’t be solved. It’s just that we have tried to apply the security methods of the past century to a new way of testing. It would be like harnessing horses to pull a car. Or, using a telegraph to send an email. Proctors—the standard security default approach—are not able to stop most modern ways of stealing exams, and they are just about as ineffective at stopping cheating. And worse yet, sometimes they even do the stealing and cheating.

So, what’s the answer? How can technology-based tests be made more secure? How can a test today be protected while remaining in active service for long periods or even indefinitely. It seems logical to me that technology, having created the problem, should also provide the solution. As Shakespeare stated in King John, “fight fire with fire,” meaning that we should enhance our security efforts with new technologies.

If new security technologies can stem exam theft and cheating, what are some of them?

1. New data forensics analyses go beyond detecting erasures on answer sheets. They can detect clusters of unusually similar tests. They can make use of item response times to detect tests that are taken too quickly or too slowly.
2. Exams can be “watermarked” intelligently in such a way as to identify the thief if the items are later found on a website.
3. New computer-only item formats can protect content better than traditional item designs. Items can be administered one at a time in a design that doesn’t allow for returning and reviewing them.

This list can go on, and the above methods can be refined and improved. Plus, many of these can be used effectively to improve security for paper-and-pencil tests. We’ve given the thieves and cheaters quite a head start. It’s time to fight back with fire of our own.

The ACLU is opposing a pilot project in Rhode Island to track students as they enter and exit school buses. “Steven Brown, executive director of the Rhode Island chapter of the American Civil Liberties Union, [called] the plan ‘a solution in search of a problem’ and saying the school district already should have procedures in place to track where its students are.” “There’s absolutely no need to be tagging children,” he said. “The program raises enormous privacy and safety concerns, he added.”

If my research is accurate, there have been at least four previous projects for tagging children in schools with RFID (Radio Frequency Identification) chips in the United States:

1. Enterprise Charter School, Buffalo, New York (2003) – badges on children, tagging library books, school cafeteria purchases, and visits to the school nurse;
2. Spring Independent School District, Houston, Texas (2004) – school bus pass program that is still operational;
3. Brittany Elementary School, Sutton, California (2005) – badges on children, due to public outcry this project was cancelled which lead to the Senate of the State of California debated banning RFID chips to identify people in the state; and
4. Tucson Unified School District, Tucson, Arizona (2007?) – school bus pass program that may still be under discussion.

In the UK, two clothing manufacturers are sewing RFID tags into school uniforms for the express purpose of tracking students while they are in school. RFID tags sewn into clothing are not new, and neither are RFID badges in the work place. And, now RFID badges are being used in universities. The University of Chicago is revamping all their student id cards primarily to ensure secure building access. Another application of RFID technology is a label affixed to your cell phone at Slippery Rock University, north of Pittsburgh, Pennsylvania, which allows for payment processing.

The idea of chipping children is controversial. On one hand, privacy advocates warn of possible abuses and intrusions. On the other, security proponents promote increased safety. In between are administrators who want improved efficiency and convenience. No one is seriously considering implanting RFID chips into children yet. But, this is happening for patients with Alzheimer’s and dementia, as well as being seriously considered by the British government for prisoners. And, the University of Washington has started a human experiment in the computer science building to assess the possibilities of RFID tracking. Several states have passed legislation prohibiting a person from being forced to accept RFID implants, which are approved by the FDA. (Image source.)

There is no doubt that RFID tags can be abused. With an RFID reader, a bad person can gather information about you surreptitiously. A bad person with a database can profile you, and even create an inventory of your belongings. But this potential exists today, even without ubiquitous RFID tags and readers. Bad people with cameras can gather information about you surreptitiously and create an inventory of your belongings. They do it to children, families and the elderly. The concern about RFID is that this can be done more efficiently. If you use membership cards and discount shopping cards, your purchases may already be tied to you in some database, somewhere, that at sometime in the future may be hacked by someone. To a certain degree, the anti-RFID movement promotes the fear that at some time your information will be stolen.

We should be aware that the school district personnel who are investigating this technology are trying to solve real problems. It’s important to keep track of library books. And, it’s even more important to know that students are entering and exiting the school buses at the proper times and locations. We live in a changing world and what we dismiss as paranoia today may become essential tomorrow. For example, there were no lockers in my elementary school. The first time I saw a locker was in junior high. While in high school, we moved to a small town and the lockers did not lock in that school (unless you brought your own lock from home). Given our changing world, I would be extremely surprised if this situation still exists in my alma mater.

I’m somewhat surprised that the ACLU has not opposed cell phones in schools. Who would have ever imagined that in the name of privacy and safety we would allow everyone to carry a camera to school and take a picture of anything there (e.g., students spitting in a teacher’s water bottle or a teacher filming the girls bathroom)? But that is precisely what has happened with cell phones. We can’t pry cell phones away from students. There is also great potential to abuse cell phones, as demonstrated with the FBI’s ability to remotely activate a cell phone’s microphone and use it to eavesdrop on nearby conversations. If cell phones are ever fitted with RFID tags, this entire debate could be over.

Mr. Brown from the ACLU is right. There are safety and security concerns with RFID devices. However, he doesn’t seem to understand those concerns. RFID chips can be hacked and the information from those chips can be transferred to other chips. As an example if the RFID card allows access to a secured area, a bad person may pilfer the electronic codes and in essence make a copy of the electronic key, as demonstrated by James Van Bokkelen. If the chips do not have proper electronic safeguards the information may be overwritten or used illegitimately.

While I have not directly addressed testing, there are implications for using RFID chips in testing which I will discuss the next time I write. But today, I just couldn’t resist this topic. In my opinion, we need to ignore the fear mongering and we need to use this technology wisely. RFID technology is not a panacea, but it can solve real problems.