Dennis on Data Forensics

The ATP (Association of Test Publishers) conference this year did everything a good conference should do. We networked. We shared industry information. We discussed best practices. We met with clients and vendors. And we created, renewed, and strengthened friendships. Rather than discuss those things, let me share a few observations relating to test security.

Exam security was a hot topic, with many sessions and many serious conversations around test security. Wayne Camara of the College Board asked me, “Was the emphasis on security due to Caveon?” I replied, “I think it is partly due to our outreach effort, and more programs are dealing with security issues.” I think there are deeper reasons.

There were more stories describing successful security efforts this year than I remember in the past. Just to name a few: the FSBPT discussed their breach and resolution in the Philippines, the GMAC caught a proxy test taker in the very act, EMC presented successful risk management cases, and the Mississippi Department of Education has effectively addressed cheating in schools. We celebrate these successes, because they give us confidence that these problems can be solved.

There is deep concern about test and exam piracy. In the past, this concern was primarily expressed by IT (Information Technology) companies. This year many other organizations had the same concern. I heard several instances of exams being stolen from within computer-based testing centers. I have no reason to doubt these reports.

Theft vulnerabilities had been voiced privately in the past, but the discussions were more open this year. I attribute this to at least three reasons: (1) there were new attendees who wanted to expressly discuss security and stayed for the Test Security Summit, (2) the Boston Globe article “Job Exam Piracy Rising,” dated December 26, 2007, gave the topic national prominence, and (3) some presenters disclosed that their entire item banks, including answer keys and digital representations, had been stolen. In the session, “Cheater, Cheater, Pumpkin Eater,” EMC Corporation reported great success in detecting and shutting down test sites where exams are being stolen. Test pirates refused to resell test content because their test sites were shut down immediately after they stole the tests.

To the best of my recollection, there were more lawyers present at ATP this year than any other year. Representatives from at least four different firms had been invited to attend by conference organizers or conference presenters. I have paraphrased some of their very instructive comments below:

“Gather all your evidence in preparation to litigate, but only litigate as a last resort.”

“You can use statistics to invalidate scores and to take other security actions if you can demonstrate that your actions and decisions are made in good faith. The courts are interpreting these actions using contract law and it’s important that your agreements and contracts support your intended actions.”

“All test items are copyrighted, but you must register the copyrights before the items are stolen. Registered copyrights provide stronger protection than unregistered copyrights. There is a special provision in copyright law to protect secure tests for this purpose.”

GMAC and Pearson VUE described initiatives for preventing and detecting imposters. GMAC verifies a candidate’s current photo with the candidate’s registration photo. They attach the photo to the score report. (I call this “testing event authentication.”) Pearson VUE demonstrated Fujitsu’s PalmSecure biometric authentication technology. The readers are priced at around $700, but within reach for secure testing applications.

Gene Radwin and Liz Burns of EMC Corporation captured our imagination. Gene shared his success in detecting users of braindump content using Trojan items. Liz Burns described her security efforts. She visualizes a triangle. At the base of the triangle are honest people who will not lie and will not cheat. At the top of the triangle are those who will cheat if at all possible. In the middle of the triangle are individuals who may cheat depending upon the circumstances. The “at risk group” is where Liz concentrates her efforts.

The Education Division meeting had an interesting discussion concerning the image of testing in education. I think that a positive image of testing is critical. As an example of how incorrect image of testing can be damaging, consider the report that South Africa has effectively banned unproctored Internet testing, because these tests are thought to be unfair, not being secure (reported by Hennie Kriek, President of SHL, USA).

Finally, if you believe that test publishers are cold and dispassionate, let me disabuse this image. I saw a lot of passion and emotion at this conference. Testing professionals are very concerned that tests are administered securely. As an example, Cindy Simmons, State Assessment Director of Mississippi, showed great forthrightness and passion as she described her state’s initiatives to address cheating on the Subject Area Tests.

It’s true there is much work to do. But members of ATP are committed to fairness and integrity in testing. They comprise “the intelligent voice of testing.“

Beginning New Year’s Day 2008, lottery ticket retailers in Ontario will have a new set of rules to follow if they will continue selling lottery tickets. “Most of the changes are the result of Ontario ombudsman Andre Marin and his scathing investigation of the province’s lottery corporation.”

http://canadianpress.google.com/article/ALeqM5jEvfDbJoJ7C3KoaNxekmT8DuUDNA

The previous set of rules allowed lottery ticket retailers to steal lottery winnings from those to whom they sold the tickets. An example of the scam is described in this story where after three years, bilked lottery ticket purchasers were finally awarded their prize.

http://www.ctv.ca/servlet/ArticleNews/story/CTVNews/20071219/opp_lottery_071219/20071219?hub=CTVNewsAt11

In the above situation, the retailer apparently exchanged a non-winning ticket for the winning ticket when the purchasers presented the ticket to claim their prize. The problem is that the retailer is in a position to game the system because two functions are performed: selling the tickets and verifying the tickets. A clever and practiced cheater can manipulate such a situation.

This “man-in-the-middle” attack illustrates an obvious weakness in most paper-and-pencil testing scenarios. An answer sheet may be misdirected or even falsified by an adult who is acting in a trusted test administration position.

For example, it is common practice in elementary schools for teachers to review the student’s answer sheets and make sure that the marked answers are dark, legible, and between the lines on the scan sheet. This practice allows a teacher to not only “clean up stray marks” but also to tamper with the answer sheet. An example of the procedure is described in this document from Dallas Independent School District: http://www.window.state.tx.us/tspr/dallas/ch02h.htm

Another example is more blatant. A teacher could very easily fill-out blank answer sheets for students and then replace the student’s answer sheets with the prepared answer sheets. Erasure or light marks analyses are routinely performed on answer sheets that are scored, but it is unlikely that “fouled” answer sheets (which would also be returned) are subjected to the same analysis.

As a variation of the above exploit, it is well-known that a certification exam can be manipulated by a proxy test taker in a similar manner. The test taker and the proxy test taker both appear at the test site. They have both registered to take the test, and both will take the test. They switch names on the answer sheets (e.g., the proxy test taker puts the name of his or her employer on the answer sheet). If the answer sheets are controlled by document identifiers, the two can breach the security by exchanging answer sheets if they are together when they receive their test materials.

The above vulnerabilities (and others that use the same theme) may be addressed with revised procedures, just as procedures are being revised for the Ontario lottery. For example, instead of stray marks being cleaned up at the school they may be cleaned up at the processing center (where those reviewing the answer sheets do not have a motive for tampering). All returned answer sheets could be scanned, allowing for any fouled answer sheets to be detected. If the answer sheets have document control numbers provided using a readable encoding (such as a bar code), then every control number should be accounted for and none should be duplicated (prevents unauthorized destruction of fouled answer sheets).

To prevent document exchange (such as in the above scenario with the proxy test taker), a digital scan of the test taker signature on the answer sheet may be preserved. This allows for verification of the signature on the answer sheet with the signature on the application. Another way to prevent document exchange between two test takers is to distribute test taking materials to candidates after all are seated, and to collect testing materials from candidates before any leave their seats at the end of the testing session.

While preventative measures are usually the best, analysis of the data may detect these types of attacks. For example, analysis of lottery wins by retailers should have detected there was a problem long before the complaints started to pile up. In the same way, it is very difficult for a person who is tampering with the test results to conceal the effect of their work.

In summary, every aspect of a test administration system and procedure should be carefully reviewed under the assumption that some individual will attempt to exploit that system, and then reasonable security measures should be taken.

Cisco now “requires all exam takers to provide digital photos and digital signatures” when candidates are admitted to take a test.

http://www.networkworld.com/newsletters/edu/2007/1217ed1.html?zb&rc=mgmt

Cisco states, “This new layer of identity authentication will help to ensure candidate identity and result in increased assurance that individuals are presenting accurate certification records in the marketplace.” In my opinion, it is very important to understand why Cisco felt that the current identity authentication mechanism (presenting a photo id along with the exam registration code) needed to be strengthened.

First, the former system relied upon a proctor at the test site to verify the validity of the identity documents that were presented. It is well known that forgers are able to create false identity documents which are undetectable by all except the most sophisticated verification systems. It is also well known that trained people do not perform this authentication task with great accuracy. After being admitted to the test site, the identity documents are no longer needed. This one-time authentication method relies upon having honest and astute proctors. Besides the fact that the candidate was admitted to the test site, no permanent record is made of the authentication. The act of authentication is not subject to review.

Second, the new system presumably captures a digital photo and signature of the test taker (as opposed to having the test taker bring the digital photo and signature to the test site). This biometric information can now be permanently stored with the test result. It can be recalled on demand. Questions concerning the identity of the individual who actually took the exam and whether that individual is the same as the person presenting the credential derived from the exam can be answered immediately. This new capability would be more properly named “transaction authentication” (borrowing a term from information systems). In other words, the testing event itself is being authenticated, which is stronger than merely authenticating the test taker. Unless the proctor is dishonest, the capture of the digital photo is outside the control of the candidate, meaning that the photo cannot be falsified.

The above article discusses braindumps and cheating, but the primary purpose of the initiative is to authenticate the identity of the test taker. In other words, Cisco is trying to keep proxy test takers or “hired gunmen” from taking tests (http://www.caveon.com/gunmen.htm). There are websites that proclaim for a few dollars you can “obtain your certification at home without entering the testing site.” They say, “Why waste your valuable time? We can take the test for you.” Through the above initiative, Cisco is taking preventative measures against these people.

Proxy test takers are a potential problem for all testing organizations. It may not be feasible to capture digital photos for your organization, but you should be able to employ some measures for authenticating the testing event. The testing event is authenticated when permanent, verifiable, non-counterfeitable information is stored with the test result. This would typically be biometric information, but non-biometric information may also be used. For example, the British government has implemented “authentication by interview” (http://www.britainusa.com/sections/articles_show_nt1.asp?d=0&i=10080&L1=0&L2=0&a=46742) as a method of passport authentication.

If you are interested in the above topic, you might check out other authentication techniques. I have linked to a few below:

PassFaces (strong passwords): www.passfaces.com/demo/try%20passfaces.htm

BioPassword (authentication by typing): http://www.biopassword.com/

Several biometrics are listed on this page: http://ctl.ncsc.dni.us/biomet%20web/BMIndividuals.html

Here’s an interesting article on “voice risk analysis” or “lie detector by phone”: http://news.scotsman.com/ViewArticle.aspx?articleid=3587706

The above techniques are interesting and they are gaining momentum, but in order to authenticate the testing event you need permanent, verifiable, non-counterfeitable information. Some of these techniques do not provide that kind of information. In my opinion, Cisco’s initiative is very good. It will be interesting to see future advances in testing event authentication.

Occasionally I search for the latest thinking about how to prevent and detect cheating on tests. I saw this presentation from the Annual Conference (2007) of the Arizona State BON (Board of Nursing) and Statewide Nurse Educators (URL is below). In my opinion this presentation is very good and provides a lot of perspective for dealing with test security issues.

http://www.azbn.gov/documents/news/Statewide%20Educators%20Academic%20Dishonesty.10.05.07.pdf

Using test result data to detect and prevent cheating was not discussed in this presentation. I think there are good reasons for the omission: (1) cheating detection software has mostly been created for large testing programs and is not readily accessible to anyone who administers tests, and (2) many people are not comfortable with using statistics to make inferences about cheating. My purpose in writing is to discuss this second issue.

Being a statistician, I admit to having specific ideas about data and test scores. Some of these ideas are not generally accepted and may not be popular. However, the idea of using statistics to detect problems with the test administration seems natural and reasonable. Anyone who would accept test scores as being valid and reliable but would not use test result data to make inferences about the quality of the test administration holds an inconsistent position. I say this because the very act of administering a test and obtaining a test score is a statistical procedure with the intent of making a statistical inference. When we give tests we are not interested in the test taker’s performance on the actual questions that were presented. Instead, we are interested in inferring or estimating the test taker’s knowledge or competence in the tested domain. Making such an inference implicitly acknowledges that the test score is a statistical measure and subject to uncertainty. If other questions had been presented, there is no doubt that the test scores would have been different.

If you do not agree with the above perspective you will not agree with the corollary that I now present. Despite disagreements, I now stipulate that the best and most reliable record of the testing session is the actual set of recorded responses (and any other measurements that can be obtained such as erasures or response times). These data are more reliable than proctoring observations, or video recordings, or any other externally derived measure of the testing session. If you can trust the recorded responses to calculate a test score and make decisions about a test taker’s future, you should be equally comfortable using the recorded responses to make inferences about the quality of the testing session and whether testing irregularities may have occurred.

Because many statistical techniques may appear to be arcane or even “mystical,” the statistician must be very careful in selecting and using techniques that are based in solid statistical principles. Statistics will be most easily defended if they are derived from a probability model that describes the behavior being observed and if they provide objective probability statements concerning the extremeness of any observation. These criteria are rather stringent and lead to the natural exclusion of many techniques that have been investigated by researchers. For example, person-fit statistics are ideal for describing whether a test taker’s response pattern is consistent with the normal pattern of test taking (In Caveon we usually use the word “aberrant” to describe inconsistent response patterns). However, even though there is a large literature on person-fit statistics no researcher has yet published how to make objective probability statements about aberrant test taking. Without having statistically sound inferential models, the practitioner must devise ad-hoc methods that are empirically derived from the analysis of the data. There are two problems with this approach: (1) the judgment of what constitutes an extreme observation is subjective and may vary depending upon the situation, and (2) the modeling technique, itself, is not easily defended or replicated. I think these problems are fundamental reasons why test administrators have been uncomfortable with using statistics to make inferences about cheating.

At Caveon, we have worked very hard to create algorithms that are capable of computing probabilities for the statistics that we use in data forensics work. Part of that work involves understanding the probability models and assumptions that underlie the models. For example, “answer-copying” statistics that are based on the idea of similarity and excess similarity should be derived from probability models. One such example is the class of answer-copying statistics presented by van der Linden and Sotaridona (2006): Detecting answer copying when the regular response process follows a known response model. Journal of Educational and Behavioral Statistics, 31, 283-304. In this paper the authors make the assumption that tests are taken independently in deriving the probability model for the number of identical responses (being the statistic of interest). We have currently implemented person-fit statistics (for detecting aberrance), similarity statistics (for detecting collusion, test coaching, answer copying and proxy test taking), erasure statistics (for detecting test tampering), gain-score statistics (for detecting unusual learning patterns), response latency statistics (for detecting content exposure), and we continue to explore other statistics. I will discuss each of these as time permits, later.