Dennis on Data Forensics

The other day I realized that visitors to Caveon’s website are very interested in “cheating methods.” John Fremer and Jamie Mulkey wrote an article titled “The Ten Most Wanted Cheaters” (http://www.caveon.com/articles/newsltr_04_Q1_1.htm) which is quite enlightening and informative. And, as a follow up to that article John Fremer and Don Sorensen created a “wanted poster” that is very entertaining (http://www.caveon.com/conference_photos/poster.html). I doubt that the presentation of my taxonomy will be more interesting that those descriptions, but I hope that it provides you with a structure from which to make security decisions. A taxonomy or classification of cheating methods and ways in which individuals commit test fraud is critical for establishing a scientific foundation for the study of cheating.

There are three main branches at the highest level of test fraud classification: stealing or exam piracy, cheating and collusion, and tampering with or falsification of test results. I have represented these activities by three classes of individuals: thieves, cheats and liars, as shown below.

In the above figure, I have overlaid these three basic activities of test fraud on three basic areas of process management for the testing program of test development, test administration, and results management. Testing programs face both security threats from insiders (internal risks) and test takers (external risks). History is replete with examples that criminals who are inside an organization and have security responsibilities inflict much greater damage than thieves, cheaters, and liars can do from the outside.

In compiling this simple taxonomy, we have culled and analyzed lists of ways to cheat from a large number of Internet sources. Unfortunately, we did not always keep track of the URLs. Even so, I acknowledge and am indebted to the compilers of those lists. I also acknowledge the influence of Greg Cizek’s excellent book “Cheating on Tests: How To Do It, Detect It, and Prevent It”

(http://www.amazon.com/Cheating-Tests-How-Detect-Prevent/dp/0805831444). I admit that the taxonomy below is incomplete and welcome your feedback.

Test Fraud – An action to misrepresent a test result, which usually is performed to gain an unfair advantage by one or more test takers.

1. Stealing – The test content is stolen and distributed to one or more individuals.

1.1. Physical theft - The actual test materials are removed and stolen. Copies of the stolen materials are usually made and distributed by pirates.

1.1.1. Booklet pages - Are taken and removed (razor blades or dental floss may be used to remove pages).

1.1.2. Test booklets - Are taken and removed (the booklet may be hidden under a jacket or garment in the test event, the booklet may be hijacked while in transit, the booklet may be taken from storage).

1.2. Unauthorized reproduction - The content is reproduced or copied.

1.2.1. Transcription - The content is transcribed using audio recordings or handwritten facsimiles.

1.2.2. Image acquisition - An image of the content is captured. The thief may elect to remove a test booklet, make an image, and then return the booklet in order to acquire the image without being detected. Other methods for copying an image are miniature cameras (e.g., button cameras, pen cameras, watch cameras, and other concealed spy cameras). Cameras are also found on cell phones, calculators, and PDAs. Some of these cameras have wireless capability to transmit the images to accomplices. Hand-held copiers and scanners (e.g., DocuPen) may be used to copy test items. If the tests are given by computer, the print-screen key or other image capturing software may be used. An insider may train video cameras on the test items to record them.

1.2.3. Item recall - Test items are remembered and recalled. Piracy rings and cram schools may assign ring members to memorize certain portions of the test. Individuals may disclose item content through Internet forums, chat rooms, or e-mail with high probability of accurate item reconstruction.

1.2.4. Computer files - Electronic files containing test items are taken. The perpetrators may infiltrate the test development servers or they may hack into workstations where the tests are being presented. A variety of hacking techniques, including social engineering and malware, will be used to compromise the security of the information systems. Industrial sabotage eavesdropping techniques (e.g. keyloggers, IP packet sniffers, remote electronic snoopers, and video t-connectors) may be used to steal the content from afar.

1.3. Content distribution – Once the test content is stolen it is sold, distributed, or shared with others. The Internet (e.g., websites, forums, chatrooms, instant messaging, peer-to-peer file sharing, and e-mail rings) is the dominant distribution medium for stolen content. Braindump sites specialize in aggregating stolen content to create “test bibles.”

2. Cheating – Test-taking rules and procedures are violated while answering the test questions.

2.1. Unauthorized assistance - A test taker gets illicit help from one or more individuals while taking the test.

2.1.1. Collusion – Two or more individuals communicate and share test content or answers during the test.

2.1.1.1.Low tech signaling - Some stand-by methods for communication are foot tapping, mirrors, hand signals, and note passing. In the “Flying V” formation, a smart test taker sits at the front of the room and conspirators are seated behind and to the left or right, passing along answers to the entire group.

2.1.1.2.Wireless and radio communication - Wireless devices and radios may be used to communicate (e.g., text messaging, radios embedded into miniature ear pieces, and pagers). A test taker may go to the restroom and phone a friend to get answers and help for the test. Another form of communication uses a chat-room or instant messaging on the Internet while taking the test.

2.1.1.3.Cheat sheet “on demand” - A smart test taker may quickly answer the test questions, transcribe the answers, and share them with conspirators through are restroom drop box or other method. A variation of this involves real-time stealing of the test content with rebroadcast of answers to test takers’ cell phones, pagers or miniature radios (worn as ear pieces) during the test.

2.1.2. Test coaching – An individual, usually a test administrator, gives improper assistance to the test taker. This assistance may be given overtly (e.g., hinting the correct answer) or covertly (e.g., displaying forbidden materials in the test location).

2.1.3. Unsupervised assistance – The test administrator may leave the room with the specific intent of allowing test takers to collaborate or another person to provide assistance. One variation of this has occurred in testing centers where test takers are granted access outside of normal operating hours. Cheaters have been known to use decoys and distractions during the test in order to provide an opportunity for accessing cheat sheets or communicating with each other.

2.1.4. Impersonation - The test is taken by a person other than the test taker. We call these proxy test takers or surrogate test takers. Clever impersonation schemes involve exchanging answer sheets or switching names on the answer sheets. The most blatant schemes involve proxies for hire. Proxies may fill out and turn in two answer sheets, while the test taker does not return answer sheet. A variation of this occurs when the test taker obtains or produces an identical answer sheet (or blue book) which is worked out before the testing event and switched when the answer sheet is turned in (the test taker impersonates himself in time).

2.2. Access to forbidden materials - A test taker uses forbidden materials to answer the test questions.

2.2.1. Answer Copying – A test taker, acting without assistance, copies answers to test questions from others, during or after the test. The behavior may include a large number of wrong-to-right erasures. If the answer copying is planned, the cheater will intentionally select a seat so as to copy from a specific test taker.

2.2.2. Crib notes and cheat sheets – A test taker brings or hides prohibited materials into the testing session. Some cheaters resort to invisible or barely visible writing techniques for preparing their crib notes. Two or more individuals may collaborate in smuggling forbidden materials into the test event.

2.2.2.1.Body writing - Notes are written on easily concealed body locations (e.g., hands, fingernails, legs, between the fingers, arms, feet, and ankles).

2.2.2.2.Clothing concealment - Notes are hidden in clothing (e.g., under shirt tails, under skirts, inside ties or shoes, inside bulky sweaters or jackets, on hat brims, and under belts).

2.2.2.3.Electronic devices - Notes are smuggled into the testing event using electronic devices (e.g., PDAs, calculators, watches, cell phones, MP3 players, CD-Rom players, iPods).

2.2.2.4.Within object concealment - Notes are concealed in normal, everyday objects and often in plain sight (e.g., back packs, purses, pill bottles, water bottles, candy wrappers, on modified identification cards, inside a case of breath mints, inside jewelry, on erasers, on handkerchiefs, on rubber bands, etc.).

2.2.2.5.Planted notes - Notes are planted in the testing room, which are retrieved surreptitiously during the test (e.g., in restrooms, in books, on bulletin boards, underneath tables and chairs, and underneath shelves).

2.2.3. Pre-knowledge of the test content -A test taker may acquire (i.e., steal or purchase) the test content and memorize it. Another form of this is “teaching the test” where the teacher or trainer discloses actual test content to the students. A test taker may feign sickness or some other plausible excuse for missing the test and then gather information from others about the test content before taking the make up exam.

2.2.4. Access to reference materials - Reference pages are smuggled into the test event, or worse, an Internet-capable device is used to search the Internet or access previously constructed reference pages.

3. Tampering – One or more individuals modifies a test result without authorization.

3.1. Editing of answer sheets - Test answers are changed in order to change the test scores.

3.1.1. “Stray marks” removal - This may happen if a teacher changes answers on the answer sheets after they have been submitted by the students. A common practice in schools with standardized tests is for teachers to clean up “stray marks.”

3.1.2. Unlawful entry – Stored answer sheets are compromised through unlawful entry (e.g., burglary) and then the documents are modified.

3.1.3. Student grading – In the educational setting when tests are graded by the students, cheaters may mutually agree to modify each other’s answer sheet. A variation occurs when a cheater modifies the test to lower the score of a hated rival.

3.2. Changing the scores - Files or records containing test results and scores are invaded and falsified.

3.2.1. Hacking the system - Scores may be falsified by hacking into computer systems

3.2.2. Document alteration - By gaining unauthorized access, the score documents may be falsified.

3.3. Coercion or bribery - Pressure or inducement is applied in order to secure an advantage. For example, students may attempt to influence teachers to give them special consideration because they had a personal emergency or got sick during the test. They may ask to down weight the test score (e.g., “Throw out my lowest score,” or “Let me show you that I can do better”).

3.4. Lies and chicanery - A test result that should not be accepted (e.g., retake policy was violated or cheating was suspected) is accepted through some form of deceit, such as lying, extortion, badgering or cajolery.

A teacher, Carla Hammersley, in Michigan resigned recently for allegedly “violating administrative procedures” during this year’s administration of the MEAP (Michigan Education Assessment Program). She denies any wrong doing, but felt that resignation was her only real option. One option offered by the superintendent was three days suspension without pay and a letter of misconduct being placed on file. It appears that the State of Michigan is still investigating the incident.

http://www.leelanaunews.com/blog/2007/11/12/northport-teacher-resigns-over-meap-violation/

The situation has the appearance of being the result of ambiguous administrative procedures for the MEAP and a teacher who is doing her best to encourage the students to do their best work on the test. Four specific administrative violations are listed in the article:

1. Providing information to students during the test that “may have aided in answering a total of five questions.”
2. Encouraging students “during testing sessions, including writing, to include details, follow previously taught formats, and correct grammatical mistakes.”
3. “Returning assessment materials to … students after … they had completed the test, and giving the students the opportunity to edit and revise their work.”
4. Reviewing “a persuasive essay format immediately before administering the … writing tests.”

The teacher says that she didn’t cheat. The school board elected to accept her resignation and pay out this year’s contract ($49,517.60 for the remainder of the year). We don’t have a lot of details, but the decisions appear to be the result of not being able to state definitively that test security was breached.

There should be no ambiguity concerning what cheating is and what cheating is not. All involved in testing need to understand the rules. That is the only way in which tests can be administered fairly and with integrity.

For me, answers to the following questions give practical, no-nonsense guidance for determining whether the teacher’s conduct was inappropriate:

1. Did students gain an unfair advantage as a result of this teacher’s conduct?
2. Would a trained proctor from another district (who had no vested interest in the school or the students) have acted the same way?

When the testing session begins the teacher must set aside the role of educator and assume the role of test administrator. If this cannot be done, there is danger in “crossing the line” and breaching exam security.

Occasionally I search for the latest thinking about how to prevent and detect cheating on tests. I saw this presentation from the Annual Conference (2007) of the Arizona State BON (Board of Nursing) and Statewide Nurse Educators (URL is below). In my opinion this presentation is very good and provides a lot of perspective for dealing with test security issues.

http://www.azbn.gov/documents/news/Statewide%20Educators%20Academic%20Dishonesty.10.05.07.pdf

Using test result data to detect and prevent cheating was not discussed in this presentation. I think there are good reasons for the omission: (1) cheating detection software has mostly been created for large testing programs and is not readily accessible to anyone who administers tests, and (2) many people are not comfortable with using statistics to make inferences about cheating. My purpose in writing is to discuss this second issue.

Being a statistician, I admit to having specific ideas about data and test scores. Some of these ideas are not generally accepted and may not be popular. However, the idea of using statistics to detect problems with the test administration seems natural and reasonable. Anyone who would accept test scores as being valid and reliable but would not use test result data to make inferences about the quality of the test administration holds an inconsistent position. I say this because the very act of administering a test and obtaining a test score is a statistical procedure with the intent of making a statistical inference. When we give tests we are not interested in the test taker’s performance on the actual questions that were presented. Instead, we are interested in inferring or estimating the test taker’s knowledge or competence in the tested domain. Making such an inference implicitly acknowledges that the test score is a statistical measure and subject to uncertainty. If other questions had been presented, there is no doubt that the test scores would have been different.

If you do not agree with the above perspective you will not agree with the corollary that I now present. Despite disagreements, I now stipulate that the best and most reliable record of the testing session is the actual set of recorded responses (and any other measurements that can be obtained such as erasures or response times). These data are more reliable than proctoring observations, or video recordings, or any other externally derived measure of the testing session. If you can trust the recorded responses to calculate a test score and make decisions about a test taker’s future, you should be equally comfortable using the recorded responses to make inferences about the quality of the testing session and whether testing irregularities may have occurred.

Because many statistical techniques may appear to be arcane or even “mystical,” the statistician must be very careful in selecting and using techniques that are based in solid statistical principles. Statistics will be most easily defended if they are derived from a probability model that describes the behavior being observed and if they provide objective probability statements concerning the extremeness of any observation. These criteria are rather stringent and lead to the natural exclusion of many techniques that have been investigated by researchers. For example, person-fit statistics are ideal for describing whether a test taker’s response pattern is consistent with the normal pattern of test taking (In Caveon we usually use the word “aberrant” to describe inconsistent response patterns). However, even though there is a large literature on person-fit statistics no researcher has yet published how to make objective probability statements about aberrant test taking. Without having statistically sound inferential models, the practitioner must devise ad-hoc methods that are empirically derived from the analysis of the data. There are two problems with this approach: (1) the judgment of what constitutes an extreme observation is subjective and may vary depending upon the situation, and (2) the modeling technique, itself, is not easily defended or replicated. I think these problems are fundamental reasons why test administrators have been uncomfortable with using statistics to make inferences about cheating.

At Caveon, we have worked very hard to create algorithms that are capable of computing probabilities for the statistics that we use in data forensics work. Part of that work involves understanding the probability models and assumptions that underlie the models. For example, “answer-copying” statistics that are based on the idea of similarity and excess similarity should be derived from probability models. One such example is the class of answer-copying statistics presented by van der Linden and Sotaridona (2006): Detecting answer copying when the regular response process follows a known response model. Journal of Educational and Behavioral Statistics, 31, 283-304. In this paper the authors make the assumption that tests are taken independently in deriving the probability model for the number of identical responses (being the statistic of interest). We have currently implemented person-fit statistics (for detecting aberrance), similarity statistics (for detecting collusion, test coaching, answer copying and proxy test taking), erasure statistics (for detecting test tampering), gain-score statistics (for detecting unusual learning patterns), response latency statistics (for detecting content exposure), and we continue to explore other statistics. I will discuss each of these as time permits, later.