Caveon Security Insights

By: Steve Addicott, Caveon Vice President

October is an important month for Caveon. Eight years ago in October, 2003, several assessment industry veterans formed a small consulting company focused solely on improving the security of our clients’ test programs.    That company is Caveon Test Security!

Fast forward to 2011, and it’s gratifying to consider what this entrepreneurial group of test security zealots has accomplished.  Since that fateful October day, we have

• conducted over 50 Security Audits of leading test organizations and vendors,
• flagged and removed tens of thousands of internet-based risks, and
• conducted statistical analyses of over 30,000,000 test instances for many of the largest, most important test programs in the world.

As I consider the number and breadth of these engagements, perhaps it is worth sharing a few of the core values under which we always operate:

Confidentiality

Throughout our years of operation, one fundamental operating principle has always applied:  client confidentiality.  We never reveal the details of our client engagements without the express approval of our clients. Our clients require and appreciate this sensitivity as we investigate security incidents and provide reports on our forensic analyses. This is not secrecy– this privacy stems from respect for our clients and for the right to privacy of individuals and organizations.

Innovation

We constantly strive to improve means and methods for strengthening exam security. We are always interested in sharing the nature of our work.  Not only do we share our methods and science with clients, client stakeholders, TAC members, educational measurement researchers, and other appropriately interested parties; we are committed to furthering the science around test security. We regularly present at conferences and webinars where we openly share our Caveon approach, theories and methodologies. In fact this last year, we have presented at conferences in Phoenix, Orlando, Chicago, Seattle, Washington DC, Hong Kong, and Prague.

Conservative Recommendations

When we conduct an engagement, our approach is to focus on the situations and incidents that are most egregious, as evidenced in the data and the results that we analyze. We highlight those problems that are most readily identified, documented, and ideally, resolved. Dealing with these problems effectively will have the greatest positive impact to the overall validity and security of test results. This reasonable approach helps our clients, most of which suffer from ever-constrained budgets and resources, effectively concentrate their time, resources, and dollars where the likelihood of inappropriate test taking is highest.

Lastly, our growth and success is directly attributable to a few overarching principles—We always strive to exceed our clients’ expectations, comport ourselves honorably, provide valuable services, and share, as openly and honestly as we can, recommendations for improving the fairness and validity of our clients’ test programs. These principles result in proven, practical protection for our clients, and we intend to follow them for another eight years!

Please Submit Your Comments Below. Thank you!

By: David Foster, CEO, Caveon Test Security

Item exposure during an exam in the testing world is often viewed as a bad thing, because it seems obvious that item exposure leads to item over-use which in turn leads to item compromise. It is common for psychometricians to limit item exposure, defining it as either a too-high absolute number of presentations of the items in a test, or a too-high rate of the items presented on tests. Unfortunately, there is no scientific research or even unscientific guidelines, or even reasonable casual suggestions, about how many exposures are too many, or which rate of exposure is too high.

It does not follow that item exposure is the same as item compromise. In fact, I’ve seen items compromised with an extremely small number of presentations. Some items have even been compromised prior to the first test being administered!

In my opinion, the notion that item compromise results from item exposure—as defined above—leads  to improper conclusions, decisions, and ineffective procedures. I have a few reasons for this opinion, a couple of which I’ll give here. First, item exposure is absolutely necessary. It is obvious that no test can be effective unless its items are exposed during the exam. Test designers even let examinees view an item multiple times encouraging them to return to and review previous items again and again. Second, item compromise has very little to do with the definitions of item exposure given above. Consider this simple example: Suppose that an item was shown to one million test takers and was presented on every exam administered. This would be considered a very high number of exposures along with a 100% exposure rate. But, suppose that none of those examinees were able to share the item with others. In this simple example, the item remains uncompromised and perfectly secure, and can be continued to be used on the exam.

If we wish to reduce item compromise, the example illustrates that limiting the number of presentations or rate of presentations of an item is not as important as the methods used to secure the items, to protect them from theft, and to keep them from being used for cheating. For this reason we need improved item security, which means better ways to keep items from being stolen and used for cheating on subsequent exams. We need methods to detect when an item is truly compromised and then immediately to take it out of service. Instead, we often see stubborn adherence to a century-old model of relatively unsecure test administration, and believing that keeping an item from being presented on a test is a sensible way to secure it.

It is certainly possible to improve the way we secure items. As examples, there are protective item and test designs available, and certainly better test monitoring procedures, that we can use. And perhaps we can learn a little from other industries as well. Consider the problem with the theft of music over the Internet. No one would suggest that music is stolen because it was listened to by too many people. Instead, we see serious efforts to protect the music, to keep it from being stolen, to detect when it is stolen, and to punish those that are responsible. We should be doing the same.

We welcome comments below!

By: John Fremer, President, Caveon Consulting Services

The third European ATP Conference concluded the last week in September in the spectacularly lovely city of Prague in the Czech Republic and it was a rousing success.  Attendance was 225 besting the planners’ target of 200 attendees.  The keynote talks were of exceptionally high quality and there were a large number of productive and well-attended sessions.  The weather was also as good as one can imagine and the city welcomed us in every way.

I pay special attention to how the prevention of cheating and test piracy is addressed at any conference that I participate in and I was struck by the substantial increase in attention from even a year ago to the topic both in formal sessions and in conversations with other attendees. Part of the reason seems to be a high level of awareness of developments in the US, especially in our state assessment programs. There is also a strong country specific set of reasons related to security breaks or cheating episodes in critical programs that have received extensive media coverage. As is the case in the US, once a cheating story breaks in the UK, the Netherlands, or other European countries, it tends to get a great deal of coverage that can last months or more.

There was a good deal of attention to “authentication,” improved ways of using biometrics, proctor training, and closer monitoring of the testing process to make it harder for test takers to invalidate our best efforts to ensure fair testing. The degree to which testing transcends borders within Europe and in the larger world was also emphasized. Steve Addicott of Caveon and Aimee Rhodes of the Chartered Financial Analysts spoke to a packed house on the international aspects of testing and security. Cheaters and pirates can be based in one country in the morning and another in the afternoon if you are resourceful enough to shut down or limit the place where their day started. The situation has been compared to the arcade game “Whack-a-mole,” where as soon as you hit one varmint, another pops up out of a different hole. I like that metaphor as it reflects my view of these unscrupulous enemies of fairness in testing.

Several of the keynotes really impressed me. Two were given by very successful entrepreneurs. Madan Padaki from Bangalore, India, CEO of MeritTrac in an address entitled “The 500 Million Dream: Building a Nation” described the progress made in India raising 400 million people out of poverty. It is an astonishing story as is the development and growth of MeritTrac to be a major provider of testing services in a ten year period. Madan did not say his path had been easy. Rather he indicated that he might not have made the attempt, if he had realized the challenges that he would face.

Another extraordinary session was given by Lucian Tarnowski; the driving force behind “Brave New Talent,” a social media based way of nurturing and locating talent. Lucian is all of 27 years old and talks about digital immigrants, i.e., most of the people now working in assessment. We are “newly arrived” to a world with so many ways to be connected. It was not that way when most of us were in school or starting our careers. Digital natives, by contrast, grew up in this world and it is very familiar to them. Lucian describes his own Dad who has stayed with his typewriter as his way of composing and communicating as a “digital refugee.” I emerged from that session convinced that I am way overdue on my promise to myself to use social networks wherever it will help me keep in touch with the colleagues, clients, and fellow professionals with whom I share interests and goals.

Another session, a plenary in the form of a debate, saw Cor Sluijter of CITO do a very fine job defending our efforts to produce high quality tests that serve valuable purposes against Donald Clark of Learn Direct. Clark pointed out a number of flaws in testing and argued that assessment is not “fit for purpose” in the 21^st century world. Clark’s criticisms were thoughtful ones, but Sluijter held his own and all of us who attended welcomed the fact that both did their best with the help of Eugene Burke of SHL who moderated with élan to show us different sides of a meaty and much talked about issue.

I have not captured all of the ATP Europe Conference, but I hope I have conveyed some of the substance and spirit.  Next year it will be in Berlin in mid-September.  The date for next year’s conference has not yet been set.  You will surely see a note with the date on this blog as well as in Caveon’s Newsletter “Cheating in the News.” If I get my own social media act together, you might get a tweet from me about it.  I like to think this particular digital immigrant can still learn new tricks.