Caveon Security Insights

By: Steve Addicott, Caveon Vice President

October is an important month for Caveon. Eight years ago in October, 2003, several assessment industry veterans formed a small consulting company focused solely on improving the security of our clients’ test programs.    That company is Caveon Test Security!

Fast forward to 2011, and it’s gratifying to consider what this entrepreneurial group of test security zealots has accomplished.  Since that fateful October day, we have

• conducted over 50 Security Audits of leading test organizations and vendors,
• flagged and removed tens of thousands of internet-based risks, and
• conducted statistical analyses of over 30,000,000 test instances for many of the largest, most important test programs in the world.

As I consider the number and breadth of these engagements, perhaps it is worth sharing a few of the core values under which we always operate:

Confidentiality

Throughout our years of operation, one fundamental operating principle has always applied:  client confidentiality.  We never reveal the details of our client engagements without the express approval of our clients. Our clients require and appreciate this sensitivity as we investigate security incidents and provide reports on our forensic analyses. This is not secrecy– this privacy stems from respect for our clients and for the right to privacy of individuals and organizations.

Innovation

We constantly strive to improve means and methods for strengthening exam security. We are always interested in sharing the nature of our work.  Not only do we share our methods and science with clients, client stakeholders, TAC members, educational measurement researchers, and other appropriately interested parties; we are committed to furthering the science around test security. We regularly present at conferences and webinars where we openly share our Caveon approach, theories and methodologies. In fact this last year, we have presented at conferences in Phoenix, Orlando, Chicago, Seattle, Washington DC, Hong Kong, and Prague.

Conservative Recommendations

When we conduct an engagement, our approach is to focus on the situations and incidents that are most egregious, as evidenced in the data and the results that we analyze. We highlight those problems that are most readily identified, documented, and ideally, resolved. Dealing with these problems effectively will have the greatest positive impact to the overall validity and security of test results. This reasonable approach helps our clients, most of which suffer from ever-constrained budgets and resources, effectively concentrate their time, resources, and dollars where the likelihood of inappropriate test taking is highest.

Lastly, our growth and success is directly attributable to a few overarching principles—We always strive to exceed our clients’ expectations, comport ourselves honorably, provide valuable services, and share, as openly and honestly as we can, recommendations for improving the fairness and validity of our clients’ test programs. These principles result in proven, practical protection for our clients, and we intend to follow them for another eight years!

Please Submit Your Comments Below. Thank you!

By: Dennis Maynes, Chief Scientist, Caveon Test Security

(The following is an excerpt from an invited talk that was presented to the US Department of Education, September 1, 2011.)

It was sometime after we started Caveon, that I realized the primary goal of conducting security analyses was the strengthening of exam security, not catching cheaters. This is a message that resonates very well with the testing program managers with whom I have interacted. They agree that the primary goal of security actions should be to obtain trustworthy test results, which occurs when the exams are administered securely and with integrity. Disciplining cheaters is important and supports this goal, but it is only a means to an end.

Exam security can be strengthened in two ways, and both should be used: (1) Prevention of cheating, and (2) Detection and discipline of cheaters which will result in deterrence.

Prevention of cheating is gained by implementing effective security processes through policies and procedures. An important element of this effort is the periodic review of security processes and how well they have been implemented.

Detection and discipline of cheaters occurs through (1) performing regular forensic analysis, (2) qualifying the anomalies, and (3) imposing sanctions and invalidating scores.

Deterrence results when security actions and consequences for cheating are publicized.

It’s important to realize that security is a process, not a state. As an example, I have an alarm system at home. Installation of an alarm system does not mean that my home is secure. Only by arming and testing the alarm system can I be ensured that it is functioning properly. Speaking of alarm systems, I am delighted when no one breaks into my home. Just because there were no break-ins, does not lessen the value of the alarm system. I have had clients who felt that web patrolling and data forensics monitoring had no value because we did not detect security breaches. The non-existence of security breaches does not lessen the value of the security processes that have been implemented.

Except for some fraud laws, there are very few laws regulating cheating. It is difficult to prove and there is no physical evidence of material loss or harm. I often hear the phrase “Prove that I cheated.” In fact, I recently saw a headline in the papers expressing the same idea. It’s important to realize that state departments of education do not need absolute proof of cheating. They have an obligation to ensure that tests are administered securely and with integrity. In order to meet this obligation, states require a “preponderance of evidence” in order to act, not absolute proof. However, the departments of education must treat students and teachers fairly, and they must communicate policies clearly.

Because security is a process, it is important to have a ready-prepared security breach response plan, before the breach occurs. It’s not a matter of if the plan will be activated; it’s only a matter of when the plan will be activated. The planning process helps the department of education to have a focused and coordinated response for conducting investigations, imposing discipline and, of utmost importance, communicating with the public and the media.

Without such a plan, the department of education must create a response to the security breach in a potentially haphazard manner. The press is very good at uncovering haphazard and hastily prepared communications.

In summary, state departments of education are empowered to use data forensics wisely and effectively when they have implemented security policies, processes, and procedures which enable them to administer tests securely and with integrity. Regular data forensics monitoring allows states to measure and manage security risks that are inherent with all forms of high-stakes testing.

By: Dennis Maynes, Chief Scientist, Caveon Test Security

This summer, many cheating investigations of schools based on forensics evidence have been inconclusive or weak. This is troubling because investigations require time and money, and they can be disruptive. In my experience, the statistics are often questioned and with good cause.

A famous statistician once said, “If you torture the data long enough, they will confess.”  (Attributed to Ronald Coase: http://en.wikipedia.org/wiki/Ronald_Coase). Forensics monitoring requires inspection of all the data and listing the extreme observations. Because this is done with the intent of FINDING anomalies, extreme care is required. Knowledge of order statistics teaches how to analyze the data properly. Unfortunately, order statistics are not taught in basic statistics courses, which means that many analysts not having knowledge of order statistics tend to torture the data. It’s like saying my basketball team is taller than your basketball team using only the height of the tallest player on each team. Such a comparison ignores the starting five and the height of all the players on the team.

Let me illustrate the concern with an example. Journalists routinely analyze the score gains of schools within the state using regressions. Because students of basic statistics courses are taught that a regression residual exceeding three standard deviations IS an outlier, the newspaper will report all schools having gains of three standard deviations or more. The named schools will receive “extra attention” in the press and by the public. Such attention is not warranted because the fact that ALL schools were examined has been ignored. The three standard deviation rule is correct only if you were to examine one school and one school only, at random. But, the analysis was not restricted to one school, which means that many schools were inappropriately named. The statistics guarantee it.

Journalists are not the only group to make this error. Most forensics reports that I have read have ignored this issue. Using order statistics, statisticians have developed corrections to avoid overstating anomalies. For example, Bonferroni’s correction divides the target probability level by the sample size (http://en.wikipedia.org/wiki/Bonferroni_correction). Using this correction, 4.2 standard deviations, not 3, is the correct threshold to use if 1,000 schools were inspected.

The next time that you see a forensics report in the media, ask yourself, “Was the Bonferroni correction or similar conservative threshold used?” If the answer is no, be very, very careful because the number of anomalies has probably been overstated. On the other hand, if the answer is yes, the investigations are probably warranted and justified.

By Steve Addicott, Vice President of Sales, Caveon Test Security

Over the past few weeks, I have participated in several planning sessions with Caveon clients who contract with us to analyze test results.  This service, Caveon Data Forensics ™, represents a proactive means to better protect their programs by identifying statistical anomalies that may indicate cheating.

While each of these programs is different (state education, IT certification, medical licensure, construction certification, etc.), it’s interesting to me that they face common challenges in confronting test fraud.  For each of these programs, test results matter…they really matter…in making important decisions in the lives of test takers (and in education, teachers and principals).  Thus, the integrity of the test administration matters greatly, too.

My overarching impression?   Tackling test fraud head-on is not for the faint of heart.  It takes commitment—a genuine, unwavering commitment to fair and valid test results—to say “We know there is a subset of our test-taking population that is taking shortcuts, and we’re going to do something about it.”

These days, everyone is busy.   So, why would any sane test program leader willingly add to his/her workload?  The results of our Data Forensics analyses do just that.  We identify:

• candidates/students that may require invalidations;
• test centers/schools that merit investigations; and
• items/exams that should be retired and/or revised.

The committed leaders we work with understand that this is what is required to be able to stand in front of stakeholders and proclaim that their test administrations are fair and valid.

Another important takeaway I’ve gained is that a successful Data Forensics program requires the cooperation and coordination of many  groups.  Last week, I met with a client, a large state department of education.  Its leadership, in order to ensure the data forensics program possessed real teeth, sought cooperation with several other state departments:

• Legal, to ensure any sanctions resulting from the data forensics would hold up in court;
• Communications, to ensure sound, consistent messaging to the media and public alike;
• Inspector General, for conducting investigations; and
• Professional Practices, in case sanctions might be brought against a state certified educator.

This sort of cross-organizational coordination is not easy to facilitate, but critically important in the fight for fair and valid testing.

If you’re considering how you can augment the security of your program, you might find one of our company webinars to be a help.   In “Don’t Shoot The Messenger”, three Caveon clients present the good, the bad, and the challenging in instituting program invalidations through data forensic analyses.  You can get a copy of the webinar slides here:  http://caveon.com/df_blog/ .

By Dr. John Fremer

There was a time, not that long ago, when state assessments did not create much interest in the media.  State assessment leaders and testing company professionals tried to think of ways to make testing results more newsworthy, scheduling special briefings for the press, holding “mock” testing sessions and the like.  Not any more.  The media is buzzing with stories about cheating in schools. What are the questions being asked and how can they be answered?

In the past year, I have been interviewed weekly about preventing and/or detecting cheating on high stakes tests.  By “high stakes,” I mean any standardized tests used to make important decisions about students, teachers, schools, and programs.  There are three questions I am almost always asked, regardless of the specific purpose of the reporter’s intended story:

• Is there more cheating by educators now than in the past?
• How can you detect whether cheating has occurred?
• What can states and schools do to eliminate or minimize cheating/

There are other questions, but these three are asked very consistently and I will answer each one.

Is There More Cheating Now?

Yes, there is more cheating by educators now then in the past.  There has been a trend toward greater levels of cheating that goes back decades and I say this based on experience not just “book learning.”  I am in my 50^th year as a testing professional and I have seen this trend unfold in many areas of testing not just in education.

How Can You Detect Cheating?

Although cheating by educators is a very worrisome problem, I take comfort from two aspects of the situation.  First of all, I estimate the proportion of educators involved in high stakes state testing who engage in cheating to be between one and two per cent.  You can find higher estimates.  In the important and very widely read book “Freakonomics” a testing misbehavior estimate of five percent is provided, but I think that overstates the actual amount.

Another source of my positive feeling is that I know that there are a good size set of tools that can identify how much cheating is occurring, where it is taking place, and how serious it is.  Caveon uses seven different indicators when we run our detection approach ,“Caveon Data Forensics™.” Here are three very critical ones:

• Very unusual gains from one year to another. If the results from a class or school seem too good to be true, don’t accept them without a great deal of careful scrutiny.
• Very high levels of similarity in the specific test results at the individual test question level between a pair or a group of students. How could it be that two classmates taking a 40 item state test, would not only get the same 20 questions correct and 20 wrong, but would choose the identical wrong answer in every instance?  That simply does not happen when students work independently.
• Very high numbers of erasures, especially wrong to right.  Most students make very few erasures on each section of a state assessment.  So when substantially larger numbers of erasures show up and when a huge proportion of the changes are from a wrong to a right answer, alarms need to go off – “trouble here.”

Caveon advocates using multiple indicators because a single number cannot tell the entire story by itself.

What Can States and Schools Do?

Even though the level of educator cheating is low, any cheating at all by teachers and others is profoundly worrisome to parents, school board members, community leaders and others.  So what can be done to reduce cheating?

1. Communicate Zero Tolerance – It needs to be crystal clear to everyone involved in testing that the school or district or state will not tolerate cheating. It is not enough to merely to say “that we do not tolerate cheating” at a staff meeting or include a “No Cheating Allowed” commitment in training materials for testing.  It is essential to deliver this message at every stage of the testing process and to get clear evidence of each person’s commitment to follow all testing rules. It is essential also that those individuals who have the primary responsibility in each school know that they have the unqualified support of school, district, and state management to train staff.  Also, that they may monitor all phases of testing to be sure that fairness and validity of results can be counted on.
2. Analyze Test Results after Every Administration – All involved in testing should be made aware that thorough analyses of test results are going to be carried out and that there will be significant consequences if rules are not followed.
3. Act on Problem Results – When evidence surfaces of failures to adhere to the rules of testing, it is essential to take action on these findings.  Only in this way will the seriousness of managers be made clear.

You now have my take on the three of the public’s highest concern questions about cheating.  If anyone wants to talk about their own experiences or to learn more about mine, please let me know. I thoroughly enjoy exchanging ideas with others who are committed to fair and valid testing.