Dennis on Data Forensics

Hindsight: Perfect understanding of an event after it has happened; – a term usually used with sarcasm in response to criticism of one’s decision, implying that the critic is unfairly judging the wisdom of the decision in light of information that was not available when the decision was made.

After every single airplane crash or incident, the FAA routinely conducts exhaustive investigations to determine the cause of the crash. The purpose of the investigation is “to identify safety deficiencies and unsafe conditions which are then referred to the responsible FAA office for evaluation and corrective action.” The amazing air safety statistics in this country are primarily the result of these extensive analyses. Setting all sarcasm aside, the FAA has learned that hindsight is 20-20. A perfect understanding of the event is often attainable. And from that understanding, air safety has improved.

I believe that all testing programs can learn from this example. If each program conducts a “security breach post mortem” security processes can be improved. A good practice in security is learning from your own mistakes. A better practice is learning from the mistakes of others. A best practice is creating processes so that those mistakes are never repeated.

As an example of what might be possible with a security breach post mortem, consider two recent news stories. Recent news from the UK suggests that many immigrants are being coached to pass the spoken language and listening portions of the citizenship tests, even though they cannot speak English. The BBC went undercover and filmed “an appraisal” which the undercover reporter understood to be the process for passing the language test. The reporter didn’t even need to speak or listen in English. The video is extremely fascinating. In other news, the results of Boston’s promotion exams for firefighters are being discarded and all the candidates will be required to retest, following a security breach in November 2007 when cell phones were used to cheat. The retesting is required because the investigation was inconclusive and the cheaters were not uncovered.

It is likely that both of the above breaches would have been prevented if proper security safeguards were in place. The purpose of the post mortem is to learn the security strengths and weaknesses of the testing program, so that security may be improved and strengthened. In my experience, we generally do not obtain all the information possible from a security breach investigation. For example, in Boston the investigation was conducted to determine who cheated. While some improvements to security should happen as a result of the investigation, I believe that a serious post mortem would reveal even more information in order to prevent similar breaches in the future. The post mortem allows us to learn from our mistakes.

In an earlier essay, I suggested that testing programs should, “Read stories of cheating in the news to learn how the media might portray your cheating incident negatively.” This is one form of learning from the mistakes of others. In addition to studying security breaches in the media, several other methods exist for learning best security practices and processes from others. Some of these are (1) attending presentations where security breaches are discussed, (2) talking directly with program personnel who have been involved in security breaches, and (3) working with experts who study and analyze security breaches and best security practices. At Caveon, we are doing our best to expand our expertise so that we may effectively assist all testing programs in their efforts to strengthen their test security.

If you have never conducted a security breach post mortem you are probably wondering how you might start.

The first step determines the extent and nature of the security breach. When the breach involves cheating during the test or tampering with the test results, a data forensics analysis is invaluable in making this assessment. When the breach involves the distribution and sale of protected test content, an Internet investigation or Caveon Web Patrol can determine the scope and size of the breach. When the breach involves a breakdown of security procedures and processes, a post-mortem security audit will be needed. Some security breaches may require all three information-gathering activities.

The second step performs a cause-and-effect flow analysis or a fault tree analysis. This analysis establishes where the test security vulnerabilities exist and how those vulnerabilities were exploited by the miscreants.

The third step identifies necessary changes in the testing program’s security processes. These changes should be first considered as suggestions or recommendations. They should be prioritized. They should be assessed for effectiveness using security threat models. They should be evaluated against required resource allocations so that their practicality can be measured in terms of the program’s budget and expertise.

Finally, proposed recommendations are presented to the executive management team with an implementation roadmap. The executive report should clearly state that the purpose of the post mortem is to improve and strengthen test security. A post mortem analysis is not conducted with the purpose of apprehending cheaters and imposing discipline upon test frauds. These actions may result from the investigations. But, the post mortem provides the tactical and strategic initiatives to prevent test fraud in the future.

Caveon is willing and able to assist you in these efforts. We wish you the best as you consider how to learn from your own mistakes and the mistakes of others.

Wise men profit more from fools than fools from wise men; for the wise men shun the mistakes of fools, but fools do not imitate the successes of the wise. – Cato the Elder

Hindsight is indeed 20-20 and is not to be scoffed at when we use it in order to improve.

Georgia bit her lip nervously as she peered out the rear-view mirror of her car. She had already been idling 10 minutes longer than allowed and campus security would be returning shortly. Then, she saw them, exiting the library. Ignacio was detained by a man in uniform. Vincenzo broke into a run, sprinted to the car, and hopped in. “Step on it,” he said. Georgia sped away. “What about Ignacio?” she asked. “Don’t worry. I have it right here,” he replied as he slipped a digital camera from beneath his jacket, extracted a memory card and handed it to Georgia. She grinned. Now, she would be able to pass the test and become an intern at Waldo & Cramer Industries. Once inside W & C and with her computer skills, her current employers would soon be very, very happy.

The above fictionalized account is based upon an incident which Caveon was asked to investigate in 2004. Our client wrote,

“We had an incident over the weekend concerning the XYZ exam …. The examiner contacted our office during the 3rd section of the examination. Two examinees were acting suspiciously throughout the exam. They had questions about how long the breaks were and what would happen if they returned late from the break. During the break, the proctor noticed that one of the test booklets was not on the applicant’s desk.

The proctors noticed that the two examinees went to their car and came back late from the break. When addressed about the booklet, they said they did not have the booklet and then dropped it from their jacket and said, ‘there it is’. They were allowed to continue, although the proctor told them their scores would be invalidated. They were addressed by the proctor and campus police after the exam and questioned. One of the examinees was released as he stated he had nothing to do with the incident. The other fled the scene in a car that was waiting for him, as he was being escorted to check his car to see if there were images on his cell phone of the test booklet. The names of the suspects are Inigo and Vinny.” (Actual names have been changed.)

Results of Investigation

Caveon conducted an investigation into this incident and we discovered that the two individuals, Inigo and Vinny, were enrolled at a nearby university but they were not enrolled in courses of study or college majors that would be consistent with taking the admissions test connected with this incident. Furthermore, we determined that one of these students had lost his passport during the summer and the other had his driver’s license stolen. The information was corroborated and led us to infer that both of these students were victims of identity theft. Some other individuals committed test fraud in their names.

We also discovered that the test thieves were given the opportunity to steal the test because the test site administrator had not collected testing materials during breaks or the lunch period, as per test administration policy and procedures. One of these individuals, “Inigo,” had taken and failed the test approximately six weeks earlier. We presume that this individual determined that an opportunity existed to sneak the test booklet out of the testing site at that time.

In our report, we concluded that the imposters (or identity thieves) took the exam with the intent of exposing the exam content for one or more of the following purposes: for themselves, on behalf of another individual(s), for mass distribution, or for financial gain. We also suggested that, with suitable revision to the test administration policies and procedures, the likelihood of a security breach could be reduced.

Forensics analysis

Another phase of the analysis was to statistically analyze the test responses. It is difficult to infer “intent to steal” from data analysis, but the data are revealing. One of the statistics that we use in Caveon Data Forensics^TM is known as the bimodality statistic. With this statistic, we assume that most individuals answer the test questions consistently according to the observed performance (or a single level of ability). However, we allow the possibility for some individuals to answer the test questions according to two levels of ability (or in two different modes, hence the name bimodality). Using these statistics we found that Vinny’s test was somewhat aberrant (at the probability level of one in 2,000) and that Inigo’s test was extremely aberrant (at the probability level of one in 200 million). These data, along with comparative “normal” data at the same ability levels, are shown in Figures 1 and 2.

Figure 1: Comparison of Vinny’s test with a normal test

Figure 2: Comparison of Inigo’s test with a normal test

The data confirm that both of these individuals took the exam at two levels of ability. The probability of the high level is shown using the yellow line. The probability of the selected response using the low and high levels is shown using the blue and pink lines, respectively. We infer that Inigo demonstrated more information and knowledge about the test content than Vinny, but both of them appeared to be answering the test questions for some other purpose than obtaining a score and an actual measure of their knowledge of this content area. It appears likely that these individuals were connected with the content area being tested.

This incident is extremely instructive. It illustrates that not all test takers are as they appear and that an unfair advantage may be gained in many ways. I had always wondered whether there would be a motive to steal an identity for the purpose of taking a test and now I know.

They say that cheaters only hurt themselves. In all honesty, I think that a cheater said that and we believed him. It is often the case that cheaters hurt the people who gave them the test more than themselves. If you are responsible for giving tests, some fool will eventually cheat on your test. How you handle cheating incidents can make or break you.

When you started out in your career and you began giving tests, you probably didn’t imagine that the most demanding aspect of your job might be what to do about cheating. The first time you encounter this and when the media spotlight is focused on you, you will probably wish you were a rattlesnake handler or a bomb disposal expert. You create and give tests. And, you’re good at your job. You never intended to become a test cop. Let me suggest that you anticipate and prepare for cheating incidents now, before they happen. We call it security incident response planning.

Speaking of cops, there have been a number of stories concerning police departments and cheating on tests recently. In the summer of 2007, information about the police promotion test in Boston was leaked to several officers, as reported by WBZTV. In another story, theState.com reported that twenty-one police officers in Columbia, South Carolina were implicated in cheating when they either cheated, helped others to cheat, or knew of the cheating but failed to report it. And, Houston’s crime lab was in the news twice for open-book cheating, which resulted in the shutdown of the lab, as reported by the Houston Chronicle on October 6, 2007, and January 26, 2008.

The above stories illustrate the importance of responding appropriately to cheating incidents and testing irregularities. You will not be judged harshly because a few miscreants decided to cheat on your test. But, you may be embarrassed completely if you do not address the problem adequately. Your program may suffer a loss of credibility. The public confidence in those you certify may be eroded. And, adding insult to injury, the media may portray you as a fool and a blunderer.

Your security incident response plan should be suited to your organization’s needs and requirements. There are a lot of questions that you should answer. Let me list a few:

1. What discipline should the cheater(s) receive?
2. Is the discipline appropriate? If it’s too harsh you may be perceived as being unfair. If it’s too lenient you may be judged as playing favorites.
3. When will you inform the public about the security breach?
4. What will you do if the media learns of the breach before you announce it? Or, before you learn of it?
5. Is an investigation needed?
6. If so, how will the investigation be conducted? Who will conduct the investigation?
7. What information will you share with the media?
8. What information will you keep confidential? What justification do you have for not sharing everything?
9. Who will be responsible for communications and media relations?
10. Is your security incident response plan recorded in policy form to guide you?

As you can see, my list focuses on “doing the right thing” not just on “looking like we are doing the right thing.” Reporters, in particular, are very quick to suspect a cover up or to suspect they are not being told the truth. And, if you are responsible for a testing program which is accountable to the public (e.g., tests in schools or tests involving public safety), it is vital that you maintain the public trust.

One way that you can sharpen your skills in this area is to “simulate” what you would do in specific cheating situations. During the course of a year, just about every type of cheating will be reported in the news. You can stay current with these stories by reading Caveon’s “Cheating in the News.” You can sign up to receive CITN notification by e-mail about twice a month on the lower right-hand corner of the main Caveon web page. Read the stories. Discuss the stories with your staff. Does your security incident response plan tell you how to handle the problem, if it happened to you?

Just as we expect our local emergency response teams (i.e., police, firefighters, and paramedics) to prepare for disasters, we should prepare to handle cheating incidents. A properly executed security incident response plan can keep an incident from becoming a disaster.

Ten years ago the New York Times criticized ETS, claiming that ETS elected to keep quiet rather than publicize exam security breaches. When you, as a testing program manager, have a full-scale security breach on your hands what will you do? I can imagine that it was a very difficult decision within ETS whether to “keep the lid on” the story or to let the story be told. This appears to be a “no-win” situation. If you publicize the security breaches you may seriously undermine your testing program. If you keep quiet and the word leaks out, like it invariably does, your own credibility may be questioned.

Read stories of cheating in the news to learn how the media might portray your cheating incident negatively. Journalists print newspapers and sell advertising. Sensational news is good copy for them. It is especially important, when under spotlight of the press, that your testing program be viewed as being fair, responsible, and ethical. In my experience, reporters will probe for any apparent contradictions, irregularities which could have been avoided, or supposed dismissal of the severity of the situation. If they find any thing that might be construed as an irregularity, it will probably be printed. In my opinion, it’s better to tell your own story first, rather than let reporters interpret the situation in a potentially harmful manner.

I wish you the best as you formulate your security incident response plan. If you could use additional guidance in preparing your security incident response plan, a Caveon test security director will be glad to consult with you.