Dennis on Data Forensics

Hindsight: Perfect understanding of an event after it has happened; – a term usually used with sarcasm in response to criticism of one’s decision, implying that the critic is unfairly judging the wisdom of the decision in light of information that was not available when the decision was made.

After every single airplane crash or incident, the FAA routinely conducts exhaustive investigations to determine the cause of the crash. The purpose of the investigation is “to identify safety deficiencies and unsafe conditions which are then referred to the responsible FAA office for evaluation and corrective action.” The amazing air safety statistics in this country are primarily the result of these extensive analyses. Setting all sarcasm aside, the FAA has learned that hindsight is 20-20. A perfect understanding of the event is often attainable. And from that understanding, air safety has improved.

I believe that all testing programs can learn from this example. If each program conducts a “security breach post mortem” security processes can be improved. A good practice in security is learning from your own mistakes. A better practice is learning from the mistakes of others. A best practice is creating processes so that those mistakes are never repeated.

As an example of what might be possible with a security breach post mortem, consider two recent news stories. Recent news from the UK suggests that many immigrants are being coached to pass the spoken language and listening portions of the citizenship tests, even though they cannot speak English. The BBC went undercover and filmed “an appraisal” which the undercover reporter understood to be the process for passing the language test. The reporter didn’t even need to speak or listen in English. The video is extremely fascinating. In other news, the results of Boston’s promotion exams for firefighters are being discarded and all the candidates will be required to retest, following a security breach in November 2007 when cell phones were used to cheat. The retesting is required because the investigation was inconclusive and the cheaters were not uncovered.

It is likely that both of the above breaches would have been prevented if proper security safeguards were in place. The purpose of the post mortem is to learn the security strengths and weaknesses of the testing program, so that security may be improved and strengthened. In my experience, we generally do not obtain all the information possible from a security breach investigation. For example, in Boston the investigation was conducted to determine who cheated. While some improvements to security should happen as a result of the investigation, I believe that a serious post mortem would reveal even more information in order to prevent similar breaches in the future. The post mortem allows us to learn from our mistakes.

In an earlier essay, I suggested that testing programs should, “Read stories of cheating in the news to learn how the media might portray your cheating incident negatively.” This is one form of learning from the mistakes of others. In addition to studying security breaches in the media, several other methods exist for learning best security practices and processes from others. Some of these are (1) attending presentations where security breaches are discussed, (2) talking directly with program personnel who have been involved in security breaches, and (3) working with experts who study and analyze security breaches and best security practices. At Caveon, we are doing our best to expand our expertise so that we may effectively assist all testing programs in their efforts to strengthen their test security.

If you have never conducted a security breach post mortem you are probably wondering how you might start.

The first step determines the extent and nature of the security breach. When the breach involves cheating during the test or tampering with the test results, a data forensics analysis is invaluable in making this assessment. When the breach involves the distribution and sale of protected test content, an Internet investigation or Caveon Web Patrol can determine the scope and size of the breach. When the breach involves a breakdown of security procedures and processes, a post-mortem security audit will be needed. Some security breaches may require all three information-gathering activities.

The second step performs a cause-and-effect flow analysis or a fault tree analysis. This analysis establishes where the test security vulnerabilities exist and how those vulnerabilities were exploited by the miscreants.

The third step identifies necessary changes in the testing program’s security processes. These changes should be first considered as suggestions or recommendations. They should be prioritized. They should be assessed for effectiveness using security threat models. They should be evaluated against required resource allocations so that their practicality can be measured in terms of the program’s budget and expertise.

Finally, proposed recommendations are presented to the executive management team with an implementation roadmap. The executive report should clearly state that the purpose of the post mortem is to improve and strengthen test security. A post mortem analysis is not conducted with the purpose of apprehending cheaters and imposing discipline upon test frauds. These actions may result from the investigations. But, the post mortem provides the tactical and strategic initiatives to prevent test fraud in the future.

Caveon is willing and able to assist you in these efforts. We wish you the best as you consider how to learn from your own mistakes and the mistakes of others.

Wise men profit more from fools than fools from wise men; for the wise men shun the mistakes of fools, but fools do not imitate the successes of the wise. – Cato the Elder

Hindsight is indeed 20-20 and is not to be scoffed at when we use it in order to improve.

Georgia bit her lip nervously as she peered out the rear-view mirror of her car. She had already been idling 10 minutes longer than allowed and campus security would be returning shortly. Then, she saw them, exiting the library. Ignacio was detained by a man in uniform. Vincenzo broke into a run, sprinted to the car, and hopped in. “Step on it,” he said. Georgia sped away. “What about Ignacio?” she asked. “Don’t worry. I have it right here,” he replied as he slipped a digital camera from beneath his jacket, extracted a memory card and handed it to Georgia. She grinned. Now, she would be able to pass the test and become an intern at Waldo & Cramer Industries. Once inside W & C and with her computer skills, her current employers would soon be very, very happy.

The above fictionalized account is based upon an incident which Caveon was asked to investigate in 2004. Our client wrote,

“We had an incident over the weekend concerning the XYZ exam …. The examiner contacted our office during the 3rd section of the examination. Two examinees were acting suspiciously throughout the exam. They had questions about how long the breaks were and what would happen if they returned late from the break. During the break, the proctor noticed that one of the test booklets was not on the applicant’s desk.

The proctors noticed that the two examinees went to their car and came back late from the break. When addressed about the booklet, they said they did not have the booklet and then dropped it from their jacket and said, ‘there it is’. They were allowed to continue, although the proctor told them their scores would be invalidated. They were addressed by the proctor and campus police after the exam and questioned. One of the examinees was released as he stated he had nothing to do with the incident. The other fled the scene in a car that was waiting for him, as he was being escorted to check his car to see if there were images on his cell phone of the test booklet. The names of the suspects are Inigo and Vinny.” (Actual names have been changed.)

Results of Investigation

Caveon conducted an investigation into this incident and we discovered that the two individuals, Inigo and Vinny, were enrolled at a nearby university but they were not enrolled in courses of study or college majors that would be consistent with taking the admissions test connected with this incident. Furthermore, we determined that one of these students had lost his passport during the summer and the other had his driver’s license stolen. The information was corroborated and led us to infer that both of these students were victims of identity theft. Some other individuals committed test fraud in their names.

We also discovered that the test thieves were given the opportunity to steal the test because the test site administrator had not collected testing materials during breaks or the lunch period, as per test administration policy and procedures. One of these individuals, “Inigo,” had taken and failed the test approximately six weeks earlier. We presume that this individual determined that an opportunity existed to sneak the test booklet out of the testing site at that time.

In our report, we concluded that the imposters (or identity thieves) took the exam with the intent of exposing the exam content for one or more of the following purposes: for themselves, on behalf of another individual(s), for mass distribution, or for financial gain. We also suggested that, with suitable revision to the test administration policies and procedures, the likelihood of a security breach could be reduced.

Forensics analysis

Another phase of the analysis was to statistically analyze the test responses. It is difficult to infer “intent to steal” from data analysis, but the data are revealing. One of the statistics that we use in Caveon Data Forensics^TM is known as the bimodality statistic. With this statistic, we assume that most individuals answer the test questions consistently according to the observed performance (or a single level of ability). However, we allow the possibility for some individuals to answer the test questions according to two levels of ability (or in two different modes, hence the name bimodality). Using these statistics we found that Vinny’s test was somewhat aberrant (at the probability level of one in 2,000) and that Inigo’s test was extremely aberrant (at the probability level of one in 200 million). These data, along with comparative “normal” data at the same ability levels, are shown in Figures 1 and 2.

Figure 1: Comparison of Vinny’s test with a normal test

Figure 2: Comparison of Inigo’s test with a normal test

The data confirm that both of these individuals took the exam at two levels of ability. The probability of the high level is shown using the yellow line. The probability of the selected response using the low and high levels is shown using the blue and pink lines, respectively. We infer that Inigo demonstrated more information and knowledge about the test content than Vinny, but both of them appeared to be answering the test questions for some other purpose than obtaining a score and an actual measure of their knowledge of this content area. It appears likely that these individuals were connected with the content area being tested.

This incident is extremely instructive. It illustrates that not all test takers are as they appear and that an unfair advantage may be gained in many ways. I had always wondered whether there would be a motive to steal an identity for the purpose of taking a test and now I know.

The ATP (Association of Test Publishers) conference this year did everything a good conference should do. We networked. We shared industry information. We discussed best practices. We met with clients and vendors. And we created, renewed, and strengthened friendships. Rather than discuss those things, let me share a few observations relating to test security.

Exam security was a hot topic, with many sessions and many serious conversations around test security. Wayne Camara of the College Board asked me, “Was the emphasis on security due to Caveon?” I replied, “I think it is partly due to our outreach effort, and more programs are dealing with security issues.” I think there are deeper reasons.

There were more stories describing successful security efforts this year than I remember in the past. Just to name a few: the FSBPT discussed their breach and resolution in the Philippines, the GMAC caught a proxy test taker in the very act, EMC presented successful risk management cases, and the Mississippi Department of Education has effectively addressed cheating in schools. We celebrate these successes, because they give us confidence that these problems can be solved.

There is deep concern about test and exam piracy. In the past, this concern was primarily expressed by IT (Information Technology) companies. This year many other organizations had the same concern. I heard several instances of exams being stolen from within computer-based testing centers. I have no reason to doubt these reports.

Theft vulnerabilities had been voiced privately in the past, but the discussions were more open this year. I attribute this to at least three reasons: (1) there were new attendees who wanted to expressly discuss security and stayed for the Test Security Summit, (2) the Boston Globe article “Job Exam Piracy Rising,” dated December 26, 2007, gave the topic national prominence, and (3) some presenters disclosed that their entire item banks, including answer keys and digital representations, had been stolen. In the session, “Cheater, Cheater, Pumpkin Eater,” EMC Corporation reported great success in detecting and shutting down test sites where exams are being stolen. Test pirates refused to resell test content because their test sites were shut down immediately after they stole the tests.

To the best of my recollection, there were more lawyers present at ATP this year than any other year. Representatives from at least four different firms had been invited to attend by conference organizers or conference presenters. I have paraphrased some of their very instructive comments below:

“Gather all your evidence in preparation to litigate, but only litigate as a last resort.”

“You can use statistics to invalidate scores and to take other security actions if you can demonstrate that your actions and decisions are made in good faith. The courts are interpreting these actions using contract law and it’s important that your agreements and contracts support your intended actions.”

“All test items are copyrighted, but you must register the copyrights before the items are stolen. Registered copyrights provide stronger protection than unregistered copyrights. There is a special provision in copyright law to protect secure tests for this purpose.”

GMAC and Pearson VUE described initiatives for preventing and detecting imposters. GMAC verifies a candidate’s current photo with the candidate’s registration photo. They attach the photo to the score report. (I call this “testing event authentication.”) Pearson VUE demonstrated Fujitsu’s PalmSecure biometric authentication technology. The readers are priced at around $700, but within reach for secure testing applications.

Gene Radwin and Liz Burns of EMC Corporation captured our imagination. Gene shared his success in detecting users of braindump content using Trojan items. Liz Burns described her security efforts. She visualizes a triangle. At the base of the triangle are honest people who will not lie and will not cheat. At the top of the triangle are those who will cheat if at all possible. In the middle of the triangle are individuals who may cheat depending upon the circumstances. The “at risk group” is where Liz concentrates her efforts.

The Education Division meeting had an interesting discussion concerning the image of testing in education. I think that a positive image of testing is critical. As an example of how incorrect image of testing can be damaging, consider the report that South Africa has effectively banned unproctored Internet testing, because these tests are thought to be unfair, not being secure (reported by Hennie Kriek, President of SHL, USA).

Finally, if you believe that test publishers are cold and dispassionate, let me disabuse this image. I saw a lot of passion and emotion at this conference. Testing professionals are very concerned that tests are administered securely. As an example, Cindy Simmons, State Assessment Director of Mississippi, showed great forthrightness and passion as she described her state’s initiatives to address cheating on the Subject Area Tests.

It’s true there is much work to do. But members of ATP are committed to fairness and integrity in testing. They comprise “the intelligent voice of testing.“

Recently there have been many reports of lost databases, stolen computers, and misplaced documents. Is it any wonder that tests and exams are also experiencing the same problems? For example, last November in New Zealand the home of an employee of the Qualification Authority was burglarized and a laptop containing math items for the National Certificate of Educational Achievement was stolen. Despite assurances of password protection, the Qualification Authority revised and reprinted 150,000 test booklets: http://www.stuff.co.nz/stuff/4331442a7694.html

As another example, the completed answer sheets from an exam for the Arkansas State Board of Cosmetology were lost or misaddressed in the FedEx shipment to the scoring agency. Ninety candidates will have to retake the exam: http://www.nwanews.com/adg/News/213242/

Two years ago Caveon’s assistance was sought in dealing with a similar situation. The car of an employee of a major test publisher was stolen. In the car were secured test materials, including an answer key to an upcoming state-wide public school examination. When the car was recovered the answer key was missing. There was not enough time to revise the test. The exam would be administered as scheduled. Our client wanted to know if the answer key was being distributed and if the integrity of the test administration had been compromised.

As we discussed the situation with the client, I was confident that we could detect a widespread breach. But, could we detect a situation when just a few students used the lost answer key? There was no doubt in my mind if the thief knew the market value of the answer key that it would be sold on the Internet. I knew this from first-hand experience. While I was teaching at the University, a dual-campus administration of the test coupled with a time lag between administrations led to the answer key being disclosed. Three of my students obtained the answer key to the exam through a Yahoo chat room. They scored 100% on all the questions, except the essay question, which they refused to answer.

The client gave us the following details about the test. There were 54 questions on the exam with 10 field test items and 44 core items. There were about 2 dozen different forms of the test. The forms all contained the same core items in the same locations, with form differences due to different sets of field test items. Slowly an analysis plan began to emerge. Because the answer key for only one of the forms was lost, we could score the field test items for all the other forms using the waylaid answer key. Scores on the field test items would be the keystone of the analysis.

We assumed that any student using the stolen answer key would not know which items were field test items and which were core items. We also assumed that the student would answer all the items (with potentially a few mistakes) using the stolen answer key. It was easy to determine that a widespread dissemination of the answer key had not occurred. Statistical methodology dictates that statistical tests are performed assuming the null hypothesis (i.e., the answer key was not in play) is true. Under this assumption we found that less than 2% of the tests had “high scores” (i.e., scores above the 95^th percentile of the distribution), when 5% were expected. This was very good news. There was not a wide-spread dissemination of the answer key.

Next, we hypothesized that a few teachers or school administrators might have received and used the stolen answer key. Using a probability inversion formula, we rank ordered the schools by the proportion of tests where more than six correct answers on the field test items (using the stolen answer key) were found. We found that the proportion of schools in the upper tail (above 10%) was less than 7% when 10% were expected. This was good news. It meant that if the answer key was disseminated, it was not likely to have occurred through teachers or administrators. (We also visually inspected the 30 most extreme schools for “perfect” scores of 10 on the field test items for all the other forms except the one associated with the lost answer key. Nothing untoward was found in any of those schools.)

Finally, searching for the proverbial needle in the haystack, we hypothesized that a few isolated students may have been able to receive the answer key through personal contact with the thief on the Internet. In order to attack this problem we created a Bayesian probability model, where we estimated the probability that the stolen answer key was used by a particular student conditional upon the test score. Using this model we inferred a 95% upper bound on the proportion of student who used the answer key to be less than .09% (or nine in ten thousand). The five most extreme tests were visually inspected, and not one of them had a “perfect” score on the field test items, using the lost answer key.

The results of the analysis gave our client sufficient confidence to trust the integrity of the test administration. In order to place perspective on these statistical estimates, we note that the estimated bound (i.e., .09%) on answer key compromise is much, much lower than the actual proportion of students who copy from each other in the normal test taking situation. While we could not prove that the stolen answer key had not been used, we concluded the following:

If any students have gained access to the answer key, the data indicate the answer key has not been shared with friends. And, if the answer key was used, its use was isolated.

With 95% confidence, no more than .09% of students used the compromised answer key. It is very likely, in fact, that no student actually used the compromised answer key.

The above situations illustrate the importance of properly securing test materials. They also illustrate that by using innovative and defensible statistical analyses, testing program administrators may know the degree of security risk that is present. The analysis of the waylaid answer key illustrates the power of data forensics in protecting and maintaining exam and test security.

Just last week a five-year old boy was detained by TSA (Transportation Security Administration) because his name was similar to a suspected terrorist on the no-fly list. The reporter wrote, “A five-year-old boy was taken into custody and thoroughly searched at Sea-Tac because his name is similar to a possible terrorist alias. As the Consumerist reports, ‘When his mother went to pick him up and hug him and comfort him during the proceedings, she was told not to touch him because he was a national security risk. They also had to frisk her again to make sure the little Dillinger hadn’t passed anything dangerous weapons or materials to his mother when she hugged him.’”

http://www.schneier.com/blog/archives/2008/01/fiveyearold_boy.html

On the other hand, 13 News in Indianapolis interviewed a woman, Lisa Skaggs, who described an incident two rows in front of her, where a man occupied the same seat that was assigned to another passenger. The man refused to produce his ID, only showing his boarding pass with the same seat number. The plane was finally evacuated in order to remove the recalcitrant passenger. http://www.wthr.com/Global/story.asp?S=7369309&nav=menu188_6

A United Airlines representative confirmed that the passenger’s name did not match the boarding pass. In my opinion, the most shocking statement about this incident came from a TSA official. “TSA’s Christopher White believes the system worked. ‘The fact that one of two million may not have a boarding pass that does not match and I.D., does not overly concern us when they’re exposed to all these other layers of security,’ said White.”

It’s not illegal to fly without having an ID. In fact TSA’s regulations explicitly allow for passengers to board an aircraft without an ID. You might find the experience and perspective of Joby Weeks to be interesting in this context: http://www.thetraveljunkie.ca/articles.php?articleid=146

The fact that boarding passes are an element of TSA’s security and that boarding passes may be printed from home represents a security hole in TSA’s security rules and regulations. This was documented by Senator Charles Schummer of New York, who vividly described how “Joe Terrorist” circumvents the no-fly list, in a letter dated February 11, 2005 to TSA officials.

http://www.csoonline.com/read/020106/caveat021706_pf.html

The insecurity of “print-from-home” boarding passes was demonstrated convincingly a year ago by Christopher Soghoian, a Ph. D. student in Computer Science at Indiana University. “The FBI raided the home of Indiana University grad student Christopher Soghoian, who created a Web site that lets users forge their own airline boarding passes. Soghoian said he intended to call attention to an airport security loophole.”

http://www.slate.com/id/2152507/ See Christopher’s description of the FBI raid here: http://paranoia.dubfire.net/2006/10/fbi-visit-2.html

There are several security principles that are illustrated in the above scenario:

1. If security is not implemented properly and has glaring security weaknesses, your organization may receive intense negative attention.
2. If security is not designed into the overall system, but it is added in after the fact, security holes will be present that will be difficult to patch.
3. A proper view of security requires understanding the true risk that is represented by anomalous and unusual behaviors (such as understanding what a one-in-one-million anomaly potentially represents).
4. Simple lists and blindly following ad-hoc rules (such as detaining five-year olds) can make your organization look ridiculous.
5. When you use elements in your security system that were not designed to provide security (such as print-from-home boarding passes), you are likely to have security holes.

We don’t know why the passenger without the ID refused to present his identification documents. Here are some possible scenarios.

1. He could have learned how to hack United Airlines’ reservation system.
2. He could be an actual wanted fugitive who paid for or fabricated a false boarding pass.
3. He could be a terrorist who was probing airline security in order to learn how to board an airplane without presenting an ID and without drawing attention to himself.

All of these possibilities show the inanity of the TSA comment: “The fact that one of two million may not have a boarding pass that does not match and I.D., does not overly concern us when they’re exposed to all these other layers of security.” We have learned at Caveon that the unusual circumstance is that which requires the greatest care and scrutiny.

A few years ago a large number of test booklets were lost. Even though the large number of lost booklets was a very small percent of the total number of printed booklets, the fact remained that those lost test booklets represented a substantial security risk to the testing program. It only takes one lost booklet to compromise an entire exam. It only takes one or two terrorists out of a million flyers to represent a significant security risk to the public safety.

Caveon Data Forensics is based on the premise that unusual and extremely anomalous data are those that should receive the greatest scrutiny. We are extremely concerned when test takers go outside the country to take tests. We are especially vigilant when tests are extremely similar, even when or especially when they represent a very small proportion of the total tests administered. From my view, the unusual and the anomalous data are those that should receive our highest attention. The comment from the TSA official suggested that such data do not represent a significant worry. In my opinion, such an attitude is short-sighted and imprudent.

Holmes handed Watson a note and said, “This is the message which struck Justice of the Peace Trevor dead with horror when he read it.”

http://sherlock.jamesbickers.com/the-memoirs-of-sherlock-holmes/the-adventure-of-the-gloria-scott/

Justice of the Peace Trevor was struck dead because the note contained a secret message. This is the essence of steganography. “Steganography is the art and science of writing hidden messages in such a way that no one apart from the sender and intended recipient even realizes there is a hidden message.” http://en.wikipedia.org/wiki/Steganography

In our modern world, we usually use the term “digital watermarking” for steganography when we refer to electronic files that contain hidden secrets. Digital watermarking techniques are being used to verify the illicit distribution of copyrighted photos, with the intent that the copyright holder may receive royalties for the unauthorized distribution. The recording industry has been using digital watermarking in the form of Digital Rights Management (DRM) in order to prevent digital theft of copyrighted movies and recordings.

The big news from the digital music industry is that during the past year, the four largest digital music labels have all dropped DRM from MP3’s that are being distributed on line. Sony BMG was the last hold-out and recently announced that DRM would no longer be used for MP3’s.

http://www.techtree.com/India/News/Sony_BMG_Drops_DRM_Protection/551-85784-643.html

http://www.zdnetasia.com/news/internet/0,39044908,62036088,00.htm

The fact is that album sales have declined. As one article states, “In short, downloads are up, physical sales are down, and downloads are not picking up the slack of lost sales.” http://www.pollstar.com/news/viewnews.pl?NewsID=9114 In other words, if the future of music sales is in downloads, then the recording houses have very little choice except to remove DRM from downloaded music. The DRM software is distasteful enough to consumers so that they will go elsewhere for music.

Professionals in the testing industry have been talking about “digital watermarking” for some time as a means of protecting tests. However, the term “digital watermarking” is a misnomer because true digital watermarking involves bit twiddling within the electronic content. You can’t twiddle the bits of text files and expect the modification to remain hidden.

Steganography can be used to protect tests by hiding information in the test so that when a stolen copy of the test is acquired (e.g., purchase from a braindump site) the exact copy of the test that was stolen may be verified. In other words, this becomes a means of detecting where and when a test is stolen. This information is used to identify the weak link in the chain of custody, so that the person responsible for the security of the exam when the test was stolen may be identified. The information cannot identify the thief, but it can identify the individuals who were entrusted with the custody of the test at the time of theft.

At Caveon we have engaged in a few projects of this nature on a limited basis. At the request of one of our clients we injected small editorial changes into the text content of selected items and then compared the various versions of the test with stolen content purchased from the Internet. We determined to the client’s satisfaction that the test theft did not occur inside their test development organization. Instead, it occurred after the test was published.

The above work was labor intensive and could only be performed on a small scale. Our research indicates that the point of risk for test theft is at the test delivery sites, which number in the thousands. An effective steganographic system will require encoding hidden information into the test content in order to detect the points of theft in the test delivery channel. A steganographic system capable of providing this kind of detection must be automated and it must be implemented on a wide-scale. This means that potentially thousands of test versions must be generated and the decoding system must be able to reliably determine which test version was stolen.

We have been conducting research and developing algorithms for such a steganographic system. Whatever method is used for hiding information, it cannot affect the performance of the test. It must be truly unobtrusive. This is a big challenge, because modifications to the item text can potentially change the difficulty of the test questions.

If you are interested in this topic you might look at these websites:

http://www.jjtc.com/Steganography/

http://www.watermarkingworld.org/

The number one security concern of testing professionals is exam theft and piracy, according to a survey that Caveon conducted at NOCA in 2005. We asked the question: “Which of the following are security concerns for you? (Please check as many as apply).” One hundred participants responded in the following manner:

Given the news article from the Boston Globe, “Job exam piracy rising,” published December 26, 2007, it would be interesting to repeat the above survey.

http://www.boston.com/news/nation/washington/articles/2007/12/26/job_exam_piracy_rising/?page=full

This is a very important article because while data are not provided to support the headline that exam piracy is really on the rise, it strongly illustrates the impact of exam piracy on the testing industry and the fact that current remedies cannot effectively counter many instances of test theft. This is particularly true for information technology certifications.

I have been studying the problem of exam piracy for a long time, and can offer a few insights. First, the asset that must be protected by exam security is the integrity of the examination process and the credibility of the test result, not the item bank or the test form. Second, the correct perspective of the relationship between certifying authority and test thief is a host-parasite relationship. The exam pirates live and draw from the vitality of the certification, devaluing it with their success. Lastly, a year ago we analyzed the data forensics analyses that we had performed for more than 20 certification programs. We determined that three main factors were directly related to exam piracy: (1) the mission and role of the certification, (2) the test administration model, and (3) the security of the test administration channel.

Protecting the integrity of the examination process – Current legal protections against exam piracy involve copyright and trade-secrecy statutes. Unfortunately, these can only be invoked after the integrity of the test is breached. They usually involve protracted investigations followed by even lengthier legal proceedings. In the meantime, the test is compromised and keeping it in service further erodes credibility in the examination process. The DMCA (Digital Millennium Copyright Act) provides some assistance when the stolen content is accessed through on a US-based ISP. But, legal remedies are few. In fact, legal jurisdiction of crimes committed over the Internet is at times very unclear, compounding the problem.

Host-parasite relationship – A certifying authority such as the FSBPT (Federation of State Boards of Physical Therapy) derives its existence from maintaining and administering the exam. An attack on the integrity exam is an attack against its very existence and must be countered. On the other hand, a company such as Microsoft provides certifications in support of its business. The vitality of such a company is derived from product sales and service, not from the certifications. Thus, as long as attacks on the exam do not adversely affect the core business of the company, it may be able to withstand parasitical infestations. In either case, the parasitical exam pirate bears no goodwill toward the certifying authority and has no compunction in destroying it.

Mission and role of certification – Resources within any organization are deployed according to its core mission or function. In the context of exam security this means that operational budgets and legal expenditures are prioritized accordingly. For example, the lawyers for an organization such as FSBPT will be more willing to tackle exam security issues than will lawyers for the typical IT company. This is because the lawyers for IT companies are involved in patent protection, maintaining business contracts, and other core business functions.

Test administration model – Most high-stakes testing programs administer tests according to pre-determined testing events. A new test (which may use previously administered items) is constructed for each event, thus decreasing the chance that stolen test items will be present on the new test. This practice means that it is more difficult for the exam pirate to profit from the testing program. On the other hand, when the same test forms are kept in service for a protracted length of time, the exam pirate has a distinct advantage in stealing and selling the test content.

Security of the test administration channel – The article from the Boston Globe states, “Technology companies in particular have accepted lower levels of security in order to have testing centers in distant corners of the globe.” The lower levels of security involve contracting the test administrations with third-parties who may have never had a background check, who may be operating cheat sites, or who don’t care exactly how they make money. A rogue test site administrator can very easily steal a test by merely recording every testing session (i.e., with a video camera) and then transcribing it. I believe that some these individuals have discovered how to actually pilfer the test content electronically, avoiding the need for transcription.

Hopefully, thinking about the above observations will help you understand why exam piracy is not going to be solved easily. Some testing organizations are being seriously affected by exam piracy. Only time will tell whether they will be able to successfully ward off the pirates, or not.

Cisco now “requires all exam takers to provide digital photos and digital signatures” when candidates are admitted to take a test.

http://www.networkworld.com/newsletters/edu/2007/1217ed1.html?zb&rc=mgmt

Cisco states, “This new layer of identity authentication will help to ensure candidate identity and result in increased assurance that individuals are presenting accurate certification records in the marketplace.” In my opinion, it is very important to understand why Cisco felt that the current identity authentication mechanism (presenting a photo id along with the exam registration code) needed to be strengthened.

First, the former system relied upon a proctor at the test site to verify the validity of the identity documents that were presented. It is well known that forgers are able to create false identity documents which are undetectable by all except the most sophisticated verification systems. It is also well known that trained people do not perform this authentication task with great accuracy. After being admitted to the test site, the identity documents are no longer needed. This one-time authentication method relies upon having honest and astute proctors. Besides the fact that the candidate was admitted to the test site, no permanent record is made of the authentication. The act of authentication is not subject to review.

Second, the new system presumably captures a digital photo and signature of the test taker (as opposed to having the test taker bring the digital photo and signature to the test site). This biometric information can now be permanently stored with the test result. It can be recalled on demand. Questions concerning the identity of the individual who actually took the exam and whether that individual is the same as the person presenting the credential derived from the exam can be answered immediately. This new capability would be more properly named “transaction authentication” (borrowing a term from information systems). In other words, the testing event itself is being authenticated, which is stronger than merely authenticating the test taker. Unless the proctor is dishonest, the capture of the digital photo is outside the control of the candidate, meaning that the photo cannot be falsified.

The above article discusses braindumps and cheating, but the primary purpose of the initiative is to authenticate the identity of the test taker. In other words, Cisco is trying to keep proxy test takers or “hired gunmen” from taking tests (http://www.caveon.com/gunmen.htm). There are websites that proclaim for a few dollars you can “obtain your certification at home without entering the testing site.” They say, “Why waste your valuable time? We can take the test for you.” Through the above initiative, Cisco is taking preventative measures against these people.

Proxy test takers are a potential problem for all testing organizations. It may not be feasible to capture digital photos for your organization, but you should be able to employ some measures for authenticating the testing event. The testing event is authenticated when permanent, verifiable, non-counterfeitable information is stored with the test result. This would typically be biometric information, but non-biometric information may also be used. For example, the British government has implemented “authentication by interview” (http://www.britainusa.com/sections/articles_show_nt1.asp?d=0&i=10080&L1=0&L2=0&a=46742) as a method of passport authentication.

If you are interested in the above topic, you might check out other authentication techniques. I have linked to a few below:

PassFaces (strong passwords): www.passfaces.com/demo/try%20passfaces.htm

BioPassword (authentication by typing): http://www.biopassword.com/

Several biometrics are listed on this page: http://ctl.ncsc.dni.us/biomet%20web/BMIndividuals.html

Here’s an interesting article on “voice risk analysis” or “lie detector by phone”: http://news.scotsman.com/ViewArticle.aspx?articleid=3587706

The above techniques are interesting and they are gaining momentum, but in order to authenticate the testing event you need permanent, verifiable, non-counterfeitable information. Some of these techniques do not provide that kind of information. In my opinion, Cisco’s initiative is very good. It will be interesting to see future advances in testing event authentication.

As more and more online courses are developed and offered, instructors of online courses need to consider the potential for cheating on the assessments. The following article describes some measures being implemented by FGCU (Florida Gulf Coast University):

http://www.nbc-2.com/articles/readarticle.asp?articleid=16460&z=3&p=

One of the measures is to track IP addresses and determine if more than one test is being submitted from the same computer. Other measures include randomization of answer choices and random selection of items from an item bank. The software also prevents the test questions from being printed. Kathleen Davey, Dean of Academic Technology, said, “”You can’t prevent everything from happening. You must rely on the integrity of the individual students up to a certain point.”

Ultimately, the above statement is true. If a test taker is sufficiently determined he or she will be able to successfully cheat on the test or steal the test content.

I have been very interested lately in the security of online assessments. They are becoming more prevalent and indications are that they will become a dominant technology in testing if security concerns can be adequately addressed. The problem is that most online assessments are essentially unproctored assessments. Until unproctored Internet tests can be delivered securely, they should not be used for high-stakes exams. By definition, an exam has high stakes if passing or failing the exam has significant life consequences for the test taker. Usually this means getting a job, getting licensed in a profession, getting admitted to a school, getting a diploma, etc.

Recently, Boston Globe released an investigative report concerning Army Correspondence Courses. Yesterday, Senator Edward Kennedy M. Kennedy, Chairman of the Armed Services Committee, reacted strongly to the report, writing, “I was shocked to read of one website that provides answer keys and boasts that “[w]ith cheap prices and fast service, you can be wearing that E-5 [sergeant] rank before you know it.”

http://www.boston.com/news/nation/washington/articles/2007/12/19/kennedy_urges_army_to_deter_cheating_on_promotional_exams/

The essential problem is that the assessments being used for the correspondence courses are unproctored Internet tests.

I remember taking unproctored tests as a student at the university. We called them “take home” tests. Our take-home tests had implicit security built into them:

1. They were really hard. You couldn’t just find the answer to the questions in the university library.
2. You might find someone to take the test for you or help you out, but eventually you would take a few in-class tests (where you couldn’t use your friend).
3. The tests were written in your own handwriting, which was easily compared with prior copies of your handwritten assignments.

Later, as an instructor at the university we added another twist to take-home tests: Every student got the same problems but with different data and different answers.

The above simple principles highlight the issues that must be addressed to administer a test securely online in an unproctored setting:

1. Biometrics should be used to authenticate test taker identity.
2. The questions must not be answerable using simple “Google” searches.
3. A verification process needs to be in place that allows the unproctored test result to be trusted.
4. Other security measures may assist with authenticating that the test taker actually did his or her own work.
5. Algorithms that produce item clones or variants can reduce the ability of test takers to share test content or profit from another’s answers.

I remember the day that I took my oral exams. There was no faking. There was no cheating. I was in a room, face-to-face, with three professors. Each of them had taught me in at least one course. Of course, it is not realistic to do this for every single individual being certified in a profession or being admitted into the university. But, it demonstrates the importance of having several observations which together confirm that the candidate does indeed possess the requisite competence.There has been interesting progress in the area of secure administrations of unproctored Internet tests. I will mention just a few items that I can recall readily:

1. Kryterion (www.kryteriononline.com) is using data forensics and biometrics to establish that a test is being taken properly.
2. SHL (www.shlgroup.com) is using an initial unproctored test followed by a verification test in a proctored setting to ensure that the test results can be trusted.
3. An instructor named Simon at the School of DCIT, University of Newcastle, used an innovative detection system with online unproctored tests that relied on font colors in Word documents to detect cheaters: http://crpit.com/confpapers/CRPITV42Simon.pdf

At this URL: http://www.westga.edu/~distance/ojdla/summer72/rowe72.html you will find a paper that is very interesting in this context.

Two things are clear: (1) online assessment is here to stay, and (2) ubiquitous security solutions are needed if online assessments are to be trusted.

A senior from Manatee High School passed the FCAT (Florida Comprehensive Assessment Test) in ten minutes by using a “secret pattern” after flunking the test three times. His score was invalidated. Apparently the test score was not invalidated because he used a pattern. Carla Frazier told the news, “FCAT rules do not prohibit students completing the test using any patterns, nor does the test have a minimum time requirement.”

http://www.bradenton.com/local/story/242473.html

We don’t know why the principal invalidated the score. We don’t know what “secret pattern” was used by the student. But, I have an idea what it might have been: “a-n-s-w-e-r-k-e-y.” Ok, I admit to being a cynic and a skeptic at times. This is one of those times.

Consider the facts, and then decide for yourself if you believe the student’s story.

1. Test publishers are very careful to make answer keys as unpredictable as possible. They are well aware of the guesser’s adage, “If you don’t know, choose ‘C’.”
2. Item writers and item reviewers are careful in writing distractors and answer choices to prevent guessers from gaming the test and gaining an advantage. They know that guessers will attempt to deduce the correct answer by analyzing the answer choice lengths and details.
3. Having analyzed a lot of high school exit exam data, I know that pass rates go down with every make up test. Students who fail three times are very lazy, easily confused or just not proficient. Passing the test in ten minutes is not consistent with any of these.
4. Cheaters are often very creative liars and they prey on our gullibility. The news reporter was gullible in writing the story and, for some reason, expects us to be equally gullible.

There are a lot of ways to detect cheating. In this particular case we might have seen any of the following:

1. An extremely high score after having flunked three times previously would be a clear warning sign to the principal.
2. The FCAT, according to the district FCAT coordinator, often contains pilot questions. If the student did very well on all the questions, except the pilot questions, and the answers to those questions matched the answer key form a different form of the test, then the principal would definitely have a “smoking gun.”
3. Sometimes the answer sheet can be modified after the fact. With the right inducement, an insider may be persuaded to change the answers. Erasure analysis would detect this kind of tampering. Perhaps the principal was suspicious and saw a lot of erasures on the answer sheet.
4. It is often the case that the cheaters boast of their exploits and in this case the principal may have gotten wind of the boasting.

Being a student of statistics, I imagine that the student could have finally gotten lucky and passed the test. Distribution theory states that the maximum observed value in a distribution has a much higher mean than the distribution from which the value was drawn. In this case, we have repeated scores on the FCAT for the student. Just by chance alone, if the student’s expected score is reasonably close to passing, after repeatedly taking the test a passing score will be observed eventually.

But, suppose that in my skepticism I am correct. Suppose the student did have the answer key. How would the forensics analyst detect that an answer key had been stolen and used? I have seen three answer-key arbitrage techniques used for exam security purposes, and which could be used in similar situations.

1. The FCAT coordinator disclosed that pilot questions are often used on the exam. Scoring the pilot questions with alternate keys could provide probability evidence that an answer key was in play.
2. I know of a situation where items were intentionally miskeyed and left unscored with the goal of determining whether the answer key had been stolen and used.
3. In another situation, the exam contained a few poorly written questions where the provided answer was ambiguous (This often happens on exams). These questions were exploited in a similar manner to compute probability evidence that an answer key was stolen and used.

The test publisher has many tools and techniques that can be used to trap the unsuspecting cheater. Answer-key arbitrage is one of those.