Dennis on Data Forensics

Elf mistress Heloise entered Elvin’s office (Head of Section K) quickly. “For the eighth week in a row, the reject rate from Section K is three times the rate from the previous twelve months,” she said, handing the weekly quality report to Elvin. She continued, “I was so impressed when your section scored higher on the elf proficiency exam than any other section in the Mechanical Doll Department nine weeks ago that I awarded your elves with assemblage of gears and levers, but this is unacceptable.” Heloise crossed her arms and waited for a reply.

Elvin wrinkled his brow and frowned ruefully. This was unwelcome, but not unexpected, news. He picked up a thick folder and opened it. He leafed through one report after another and muttered, “We have eliminated transportation, storage, tools, assembly, parts, fatigue, and sabotage as explanations. There’s only one conclusion. At least one, and maybe several, of the elves in Section K is incompetent. But how can that be? Is the proficiency exam flawed?”

“Let’s find out,” replied Heloise. And together, they visited the proficiency exam designer. After explaining the problem, the proficiency exam designer shook her head and said, “You need to see the data forensics analyst.” The data forensics analyst listened with deep concentration, scanned page after page of test results, whistled softly, and finally exclaimed, “It looks like elves in Section K have cheated on the elf proficiency exam. Now, how to prove it?” he said mysteriously, and then immersed himself in complex symbols and calculations. Heloise and Elvin excused themselves, but the data forensics analyst didn’t even turn his head as they left. Much later, the proficiency exam designer listened intently while the data forensics analyst described his plan for catching the cheaters in Section K.

Three weeks later, the schedule for the quarterly elf proficiency exam was posted throughout the Mechanical Doll Department. On the day of the test, elf examiners throughout Santa’s workshop reported to a different department than usual to conduct the examination. For example, elf examiners from Remote-Controlled Toys reported to the Games and Puzzles Department. It so happened that an elf examiner from each of the other departments reported to the Mechanical Doll Department. Some administered the elf proficiency exam, and others just watched and waited. All test responses were recorded meticulously. After a long and grueling day, all the elves had been tested.

The data forensics analyst worked all night, making calculations and graphs and charts. At the break of day, Heloise and Elvin knocked at his door. “Enter!” they heard. They stepped into a bizarre scene: scraps of paper were strewn about, charts with bars and circles were plastered on the walls, and a wizened elf was humming in the midst of chaos. “Done!” he shouted. “Oh, it’s you. Well, I have the answer,” he said with absent-minded aplomb.

Then noticing their impatient expressions, he said, “Oh, let me explain.”

“None of the examiners are involved. I know this because there are no patterns of inconsistent answering associated with the examiners. It was important that no examiner give the test to any elf with whom he or she normally associates.

“There were extremely similar test answers between four elves in Section K. It is almost certain that they did not take the tests independently,” The data forensics analyst concluded.

“But, how can that be?” queried Heloise. “They were all watched carefully. There was no way that they could have shared answers or communicated during the test!”

The data forensics analyst minutely explained, “I suspected this might be the case. So, I asked the proficiency exam designer to create two test forms. She very carefully changed a few of the questions between the first and second test forms, so that the correct answers would be close, but not the same. The master test booklet for the first form was locked away in test booklet storage. The proficiency exam designer kept the master test booklet for the second form with her at all times. Even though the elves in the Mechanical Doll Department were given the second form of the test, our four culprits answered all the changed questions with answers from the first form of the test. There is no doubt in my mind. They broke into test booklet storage and memorized the test answers!”

Elvin brought the four suspected cheaters into Heloise’s office. Each elf vigorously denied any wrongdoing. At that point, the data forensics analyst dimmed the lights. He splayed an infrared beam across the hands of each suspected cheater. All of their hands glowed eerily with a blotchy red hue. Then, using gloves to handle the master test booklet from storage he shined the beam on the pages. They glowed red. He touched the booklet pages against his bare arm. Shining the bean on his arm, it also glowed with a blotchy red hue. Heloise barked, “You are red-handed! Now stand still while I consider your punishment!”

“Tomorrow,” pronounced Heloise. “You will report to the master of the Quality Department for ‘R and R,’ where you will begin the repair and refurbishment of all toys in the Rejected Toy Warehouse. You will work there until all the broken toys are operating perfectly and to the satisfaction of the master of quality.”

“Elvin,” Heloise continued. “Section K can no longer be responsible for assemblage of gears and levers. Your section must repair its damaged reputation from producing so many rejected mechanical dolls. Even though you will not receive replacements for these culprits, your production quota will remain the same.”

Elvin wrinkled his brow and frowned ruefully. This was unwelcome, but not unexpected, news. He remembered another time, when he was an impetuous, lazy elf; and when he had cheated. The punishment seemed harsh, but he had learned his lesson and was glad that the cheaters had been apprehended.

Moral: Just as dishonesty betrays the cheater, it injures all who are around him.

Addendum: The cheating detection and prevention techniques described in this story are among best practices. I have described use of the data forensics methodologies in two actual cases we have analyzed at Caveon: The case of the waylaid answer key and The case of the befuddled answer copier.

The State of Mississippi has put together a very nice power-point presentation on test administration auditing and monitoring: www.mde.k12.ms.us/ACAD/osa/DTC_Test_Security_Fall_07.pps

If you are interested in learning more about these or other solutions to test fraud please contact us, at Caveon Test Security.

In 1965, Gordon Moore of Intel observed that transistor densities were doubling roughly every 2 years. Since then the exponential nature of faster, smaller and more powerful computational units has continued. Initially, the observation was a remarkable statement of trends. Later, it became an expectation. And, it is now considered an unrelenting challenge for high technology. http://en.wikipedia.org/wiki/Moore’s_law

The trend of faster, smaller and more powerful electronic devices has spilled over from computers into all forms and types of electronics. Notably, consumer electronics commonly used by cheaters on tests are no exception. While Internet-capable PDAs have been available for some time, it was in 2007 that Apple introduced the iPhone, a cellular phone integrated with a browser and digital camera. It would be surprising if iPhones and text-messaging are not replaced with even more sophisticated cheating technology within the next few years. Those who administer tests must anticipate the appearance of these newer, faster, and more easily concealed cheating devices.

Small, fast devices appeal to two broad classes of consumers: (1) persons who want mobile and wearable electronic devices, and (2) persons who have a need for spy gadgetry. Wearable computing (http://www.media.mit.edu/wearables/) trends are very interesting, including smaller keyboards (http://www.frogpad.com/), head-mounted displays (http://en.wikipedia.org/wiki/Head-mounted_display), USB watches (http://www.amazon.com/Timex-Data-Link-Watch-T5C291/dp/B000B545B4), and PDAs and ultra-small computers (examples are: Nokia’s Internet Tablet http://reviews.cnet.com/pdas/nokia-n800-internet-tablet/4505-3127_7-32309517.html and OQO’s Model 02 http://en.wikipedia.org/wiki/OQO).

Spy gadget shops sell tiny pin-hole cameras, but our research at Caveon indicates that the tiny digital cameras have insufficient resolution to capture high quality images of test questions. (See this review of the Casio WQV-1CR Wristwatch camera http://reviews.cnet.com/watches-and-wrist-devices/casio-wqv-1cr-wristwatch/4505-3512_7-2660570.html.) While we found that the pin-hole spy cameras did not have sufficient resolution to steal a high-quality image of a test, we did confirm that the hand-held scanner DocuPen (http://planon.com/) could be used very easily to steal a paper-and-pencil test. There is a clear trend for higher resolution digital cameras in smaller packages, such as the BenQ 8 megapixel camera which is 4 inches by 2.5 inches by one-half inch thick http://blogs.zdnet.com/digitalcameras/?p=151.) We expect to see eight megapixel cameras in cell phones before long due to Samsung’s announcement of a CMOS package for cell phones (http://blogs.zdnet.com/ip-telephony/?p=2737).

In 2007, we saw the introduction of ExamEar, an earpiece with a radio that was specifically marketed to cheaters on tests. This caused a lot of concern in Great Britain (http://news.bbc.co.uk/1/hi/education/6951524.stm, see also http://www.engadget.com/2007/08/20/examear-helping-students-make-the-best-of-exam-day/) and the website owners decided to cease operations. The ExamEar domain is now for sale. But, it would be very surprising if this technology does not resurface. In fact, two Chinese students were recently caught cheating on a test when they couldn’t remove their earpieces and needed medical attention (http://www.chinadaily.com.cn/china/2007-12/31/content_6361740.htm). We don’t know where they obtained these earphones, but they may have been ExamEar models.

Cheaters are usually engaged in one of four behaviors which may be bolstered by technology. These are:

1. Communicate with or copy from another (requires a miniature radio, cell phone, or other signaling device),
2. Smuggle test taking aids into the testing event (requires a miniature high-capacity data retrieval device with visual display, such as a PDA, iPod, or DataLink wristwatch)
3. Steal a copy of the test content (requires a miniature camera)
4. Engage in impersonation (requires an ability to tamper with or defeat identification safeguards)

Many of the current devices used by cheaters (e.g., cell phones, DocuPens, and PDAs) can be easily slipped past most test administrators, because they are so small. One of the gadgets shown at the 2008 CES (Consumer Electronics Show) which may cause concern for test administrators is the Bug Labs do-it-yourself modular electronics kit (http://gizmodo.com/346789/bug-labs-store-launches-monday-minus-wi+fi). It seems that the device will not include Wi-Fi initially, but it has support for a wide range of other functions, including cameras and cell phones.

Another recent innovation is the Bionic Eye (http://www.msnbc.msn.com/id/22731631/). This is a contact lens that features LCD circuitry which allows projection of an image into the wearer’s field of view. Researchers at the University of Washington have tested it successfully on rabbits. These researchers are the same people who developed the virtual retinal display (http://en.wikipedia.org/wiki/Virtual_retinal_display). It will be sometime before these contact lenses are used by people, but the technology is fascinating.

Another interesting product introduced in 2007 was the FlyPen, a pen-top computer. The company’s marketing literature states, “Meet the FLY Fusion Pentop Computer, the only pentop platform to offer a complete set of high-speed homework solutions and innovative note-taking applications for students of all ages. This next-generation FLY^TM system harnesses the same sophisticated Anoto technology as its predecessor, enhanced by PC connectivity, four times the memory, on-the-go calculating functionality, and a 1,000-word Spanish dictionary. Best of all, students can upload handwritten notes and drafts, digitizing them instantly into Microsoft Word documents or emails.” (See http://www.flyworld.com/presskit.pdf.) It will be interesting to see if students use this device for stealing test content.

Because consumer electronics are changing and adapting so quickly, it is very important that testing program administrators review current policies, procedures, and practices to ensure that these devices are not used by cheaters to gain an unfair advantage.

Being an election year, the security of voting machines is coming under intense scrutiny. The article states, “With the presidential race in full swing, some U.S. states have found critical flaws in the accuracy and security of their electronic voting machines, forcing officials to scramble to return to the paper ballots they abandoned after the 2000 Florida debacle.”

http://www.msnbc.msn.com/id/22454379

The backlash against security flaws that have been discovered by the states may very well prompt a return to paper-based balloting. “Vendors of the electronic voting machines warn against a rush back to paper.” Industry representatives state that we should “not throw the baby out with the bath water” and “trying to design a voting system when you don’t know how it’s being judged is causing a lot of problems.” State election officials just want secure, tamper-proof election systems.

Security professionals have serious reservations about the security of all voting machines that are based on general purpose operating systems. Such systems provide too many points of entry for an attacker and cannot be made completely secure. Bruce Schneier has a lot of very good essays on this topic. Here’s one of his latest:

http://www.schneier.com/blog/archives/2007/12/more_voting_mac_1.html

A return to paper may be viewed by some as being shortsighted. On the other hand, the security flaws in that system have been thoroughly studied and countermeasures are in place. We must learn that switching from paper ballot systems to electronic voting systems does not mean that security will be better. It means that the attacks and opportunities will be different.

Now, I have digressed. This is an essay on testing and exam security. The lesson to be learned from the introduction of voting machines is that technology does not automatically solve security problems. In fact, adoption of technologies introduces new security challenges. Managers of testing programs that migrate from paper-based testing to computer-based testing in the name of improved security need to be acutely aware that new, unforeseen, security challenges will emerge.

In the same way that most types of election fraud do not involve the actual mechanism by which the vote is cast, most types of test fraud do not involve the actual mechanism by which the test responses are recorded. If you accept the above statement, then you realize that security responses to test fraud transcend the test delivery mode.

Proponents of computer-based testing argue that the tests are more secure than paper-based delivery methods due to limited item exposure, strict time limits, sophisticated verification mechanisms, and strong encryption. After analyzing a lot of data I have come to realize that security is not the result of how the tests are deployed and delivered. I have learned that the above claims are groundless because good security comes from people who follow processes and procedures which have been designed to provide security. Security may be improved using technology, but good security is not derived from technological devices. Any computer system is hackable, especially if the attacker has unrestricted physical access to the system.

Here is a partial list of the “different” security issues that plague computer-based tests.

1 – If the tests are provided in an “on-demand mode” (i.e., scheduled for the test takers convenience in a prolonged testing window), organized test thieves may very quickly harvest the item banks. We have seen tests in this environment that are completely compromised within weeks or days of publication.

2 – Dishonest test site operators can completely reverse engineer the testing engine and database that are on the local machines.

3 – A video T-connector can be used to record any and all testing sessions, allowing easy theft of test items.

4 – Viruses and other forms of malware may be installed on the test delivery workstations allowing for keyboard logging or test result tampering (which would likely be undetectable).

The above attacks have their counterparts in the form of booklet theft, test form photocopying, or answer-sheet tampering in the paper-based testing environment. If you have individuals in positions of trust who you have not certified to be trustworthy, then the integrity of your testing process will be suspect. The point I am trying to make is that security attacks are not automatically thwarted when we use technology. Instead, good security comes from people who follow processes and procedures which have been designed to provide a secure testing environment.

The number one security concern of testing professionals is exam theft and piracy, according to a survey that Caveon conducted at NOCA in 2005. We asked the question: “Which of the following are security concerns for you? (Please check as many as apply).” One hundred participants responded in the following manner:

Given the news article from the Boston Globe, “Job exam piracy rising,” published December 26, 2007, it would be interesting to repeat the above survey.

http://www.boston.com/news/nation/washington/articles/2007/12/26/job_exam_piracy_rising/?page=full

This is a very important article because while data are not provided to support the headline that exam piracy is really on the rise, it strongly illustrates the impact of exam piracy on the testing industry and the fact that current remedies cannot effectively counter many instances of test theft. This is particularly true for information technology certifications.

I have been studying the problem of exam piracy for a long time, and can offer a few insights. First, the asset that must be protected by exam security is the integrity of the examination process and the credibility of the test result, not the item bank or the test form. Second, the correct perspective of the relationship between certifying authority and test thief is a host-parasite relationship. The exam pirates live and draw from the vitality of the certification, devaluing it with their success. Lastly, a year ago we analyzed the data forensics analyses that we had performed for more than 20 certification programs. We determined that three main factors were directly related to exam piracy: (1) the mission and role of the certification, (2) the test administration model, and (3) the security of the test administration channel.

Protecting the integrity of the examination process – Current legal protections against exam piracy involve copyright and trade-secrecy statutes. Unfortunately, these can only be invoked after the integrity of the test is breached. They usually involve protracted investigations followed by even lengthier legal proceedings. In the meantime, the test is compromised and keeping it in service further erodes credibility in the examination process. The DMCA (Digital Millennium Copyright Act) provides some assistance when the stolen content is accessed through on a US-based ISP. But, legal remedies are few. In fact, legal jurisdiction of crimes committed over the Internet is at times very unclear, compounding the problem.

Host-parasite relationship – A certifying authority such as the FSBPT (Federation of State Boards of Physical Therapy) derives its existence from maintaining and administering the exam. An attack on the integrity exam is an attack against its very existence and must be countered. On the other hand, a company such as Microsoft provides certifications in support of its business. The vitality of such a company is derived from product sales and service, not from the certifications. Thus, as long as attacks on the exam do not adversely affect the core business of the company, it may be able to withstand parasitical infestations. In either case, the parasitical exam pirate bears no goodwill toward the certifying authority and has no compunction in destroying it.

Mission and role of certification – Resources within any organization are deployed according to its core mission or function. In the context of exam security this means that operational budgets and legal expenditures are prioritized accordingly. For example, the lawyers for an organization such as FSBPT will be more willing to tackle exam security issues than will lawyers for the typical IT company. This is because the lawyers for IT companies are involved in patent protection, maintaining business contracts, and other core business functions.

Test administration model – Most high-stakes testing programs administer tests according to pre-determined testing events. A new test (which may use previously administered items) is constructed for each event, thus decreasing the chance that stolen test items will be present on the new test. This practice means that it is more difficult for the exam pirate to profit from the testing program. On the other hand, when the same test forms are kept in service for a protracted length of time, the exam pirate has a distinct advantage in stealing and selling the test content.

Security of the test administration channel – The article from the Boston Globe states, “Technology companies in particular have accepted lower levels of security in order to have testing centers in distant corners of the globe.” The lower levels of security involve contracting the test administrations with third-parties who may have never had a background check, who may be operating cheat sites, or who don’t care exactly how they make money. A rogue test site administrator can very easily steal a test by merely recording every testing session (i.e., with a video camera) and then transcribing it. I believe that some these individuals have discovered how to actually pilfer the test content electronically, avoiding the need for transcription.

Hopefully, thinking about the above observations will help you understand why exam piracy is not going to be solved easily. Some testing organizations are being seriously affected by exam piracy. Only time will tell whether they will be able to successfully ward off the pirates, or not.

Beginning New Year’s Day 2008, lottery ticket retailers in Ontario will have a new set of rules to follow if they will continue selling lottery tickets. “Most of the changes are the result of Ontario ombudsman Andre Marin and his scathing investigation of the province’s lottery corporation.”

http://canadianpress.google.com/article/ALeqM5jEvfDbJoJ7C3KoaNxekmT8DuUDNA

The previous set of rules allowed lottery ticket retailers to steal lottery winnings from those to whom they sold the tickets. An example of the scam is described in this story where after three years, bilked lottery ticket purchasers were finally awarded their prize.

http://www.ctv.ca/servlet/ArticleNews/story/CTVNews/20071219/opp_lottery_071219/20071219?hub=CTVNewsAt11

In the above situation, the retailer apparently exchanged a non-winning ticket for the winning ticket when the purchasers presented the ticket to claim their prize. The problem is that the retailer is in a position to game the system because two functions are performed: selling the tickets and verifying the tickets. A clever and practiced cheater can manipulate such a situation.

This “man-in-the-middle” attack illustrates an obvious weakness in most paper-and-pencil testing scenarios. An answer sheet may be misdirected or even falsified by an adult who is acting in a trusted test administration position.

For example, it is common practice in elementary schools for teachers to review the student’s answer sheets and make sure that the marked answers are dark, legible, and between the lines on the scan sheet. This practice allows a teacher to not only “clean up stray marks” but also to tamper with the answer sheet. An example of the procedure is described in this document from Dallas Independent School District: http://www.window.state.tx.us/tspr/dallas/ch02h.htm

Another example is more blatant. A teacher could very easily fill-out blank answer sheets for students and then replace the student’s answer sheets with the prepared answer sheets. Erasure or light marks analyses are routinely performed on answer sheets that are scored, but it is unlikely that “fouled” answer sheets (which would also be returned) are subjected to the same analysis.

As a variation of the above exploit, it is well-known that a certification exam can be manipulated by a proxy test taker in a similar manner. The test taker and the proxy test taker both appear at the test site. They have both registered to take the test, and both will take the test. They switch names on the answer sheets (e.g., the proxy test taker puts the name of his or her employer on the answer sheet). If the answer sheets are controlled by document identifiers, the two can breach the security by exchanging answer sheets if they are together when they receive their test materials.

The above vulnerabilities (and others that use the same theme) may be addressed with revised procedures, just as procedures are being revised for the Ontario lottery. For example, instead of stray marks being cleaned up at the school they may be cleaned up at the processing center (where those reviewing the answer sheets do not have a motive for tampering). All returned answer sheets could be scanned, allowing for any fouled answer sheets to be detected. If the answer sheets have document control numbers provided using a readable encoding (such as a bar code), then every control number should be accounted for and none should be duplicated (prevents unauthorized destruction of fouled answer sheets).

To prevent document exchange (such as in the above scenario with the proxy test taker), a digital scan of the test taker signature on the answer sheet may be preserved. This allows for verification of the signature on the answer sheet with the signature on the application. Another way to prevent document exchange between two test takers is to distribute test taking materials to candidates after all are seated, and to collect testing materials from candidates before any leave their seats at the end of the testing session.

While preventative measures are usually the best, analysis of the data may detect these types of attacks. For example, analysis of lottery wins by retailers should have detected there was a problem long before the complaints started to pile up. In the same way, it is very difficult for a person who is tampering with the test results to conceal the effect of their work.

In summary, every aspect of a test administration system and procedure should be carefully reviewed under the assumption that some individual will attempt to exploit that system, and then reasonable security measures should be taken.

Cisco now “requires all exam takers to provide digital photos and digital signatures” when candidates are admitted to take a test.

http://www.networkworld.com/newsletters/edu/2007/1217ed1.html?zb&rc=mgmt

Cisco states, “This new layer of identity authentication will help to ensure candidate identity and result in increased assurance that individuals are presenting accurate certification records in the marketplace.” In my opinion, it is very important to understand why Cisco felt that the current identity authentication mechanism (presenting a photo id along with the exam registration code) needed to be strengthened.

First, the former system relied upon a proctor at the test site to verify the validity of the identity documents that were presented. It is well known that forgers are able to create false identity documents which are undetectable by all except the most sophisticated verification systems. It is also well known that trained people do not perform this authentication task with great accuracy. After being admitted to the test site, the identity documents are no longer needed. This one-time authentication method relies upon having honest and astute proctors. Besides the fact that the candidate was admitted to the test site, no permanent record is made of the authentication. The act of authentication is not subject to review.

Second, the new system presumably captures a digital photo and signature of the test taker (as opposed to having the test taker bring the digital photo and signature to the test site). This biometric information can now be permanently stored with the test result. It can be recalled on demand. Questions concerning the identity of the individual who actually took the exam and whether that individual is the same as the person presenting the credential derived from the exam can be answered immediately. This new capability would be more properly named “transaction authentication” (borrowing a term from information systems). In other words, the testing event itself is being authenticated, which is stronger than merely authenticating the test taker. Unless the proctor is dishonest, the capture of the digital photo is outside the control of the candidate, meaning that the photo cannot be falsified.

The above article discusses braindumps and cheating, but the primary purpose of the initiative is to authenticate the identity of the test taker. In other words, Cisco is trying to keep proxy test takers or “hired gunmen” from taking tests (http://www.caveon.com/gunmen.htm). There are websites that proclaim for a few dollars you can “obtain your certification at home without entering the testing site.” They say, “Why waste your valuable time? We can take the test for you.” Through the above initiative, Cisco is taking preventative measures against these people.

Proxy test takers are a potential problem for all testing organizations. It may not be feasible to capture digital photos for your organization, but you should be able to employ some measures for authenticating the testing event. The testing event is authenticated when permanent, verifiable, non-counterfeitable information is stored with the test result. This would typically be biometric information, but non-biometric information may also be used. For example, the British government has implemented “authentication by interview” (http://www.britainusa.com/sections/articles_show_nt1.asp?d=0&i=10080&L1=0&L2=0&a=46742) as a method of passport authentication.

If you are interested in the above topic, you might check out other authentication techniques. I have linked to a few below:

PassFaces (strong passwords): www.passfaces.com/demo/try%20passfaces.htm

BioPassword (authentication by typing): http://www.biopassword.com/

Several biometrics are listed on this page: http://ctl.ncsc.dni.us/biomet%20web/BMIndividuals.html

Here’s an interesting article on “voice risk analysis” or “lie detector by phone”: http://news.scotsman.com/ViewArticle.aspx?articleid=3587706

The above techniques are interesting and they are gaining momentum, but in order to authenticate the testing event you need permanent, verifiable, non-counterfeitable information. Some of these techniques do not provide that kind of information. In my opinion, Cisco’s initiative is very good. It will be interesting to see future advances in testing event authentication.

As more and more online courses are developed and offered, instructors of online courses need to consider the potential for cheating on the assessments. The following article describes some measures being implemented by FGCU (Florida Gulf Coast University):

http://www.nbc-2.com/articles/readarticle.asp?articleid=16460&z=3&p=

One of the measures is to track IP addresses and determine if more than one test is being submitted from the same computer. Other measures include randomization of answer choices and random selection of items from an item bank. The software also prevents the test questions from being printed. Kathleen Davey, Dean of Academic Technology, said, “”You can’t prevent everything from happening. You must rely on the integrity of the individual students up to a certain point.”

Ultimately, the above statement is true. If a test taker is sufficiently determined he or she will be able to successfully cheat on the test or steal the test content.

I have been very interested lately in the security of online assessments. They are becoming more prevalent and indications are that they will become a dominant technology in testing if security concerns can be adequately addressed. The problem is that most online assessments are essentially unproctored assessments. Until unproctored Internet tests can be delivered securely, they should not be used for high-stakes exams. By definition, an exam has high stakes if passing or failing the exam has significant life consequences for the test taker. Usually this means getting a job, getting licensed in a profession, getting admitted to a school, getting a diploma, etc.

Recently, Boston Globe released an investigative report concerning Army Correspondence Courses. Yesterday, Senator Edward Kennedy M. Kennedy, Chairman of the Armed Services Committee, reacted strongly to the report, writing, “I was shocked to read of one website that provides answer keys and boasts that “[w]ith cheap prices and fast service, you can be wearing that E-5 [sergeant] rank before you know it.”

http://www.boston.com/news/nation/washington/articles/2007/12/19/kennedy_urges_army_to_deter_cheating_on_promotional_exams/

The essential problem is that the assessments being used for the correspondence courses are unproctored Internet tests.

I remember taking unproctored tests as a student at the university. We called them “take home” tests. Our take-home tests had implicit security built into them:

1. They were really hard. You couldn’t just find the answer to the questions in the university library.
2. You might find someone to take the test for you or help you out, but eventually you would take a few in-class tests (where you couldn’t use your friend).
3. The tests were written in your own handwriting, which was easily compared with prior copies of your handwritten assignments.

Later, as an instructor at the university we added another twist to take-home tests: Every student got the same problems but with different data and different answers.

The above simple principles highlight the issues that must be addressed to administer a test securely online in an unproctored setting:

1. Biometrics should be used to authenticate test taker identity.
2. The questions must not be answerable using simple “Google” searches.
3. A verification process needs to be in place that allows the unproctored test result to be trusted.
4. Other security measures may assist with authenticating that the test taker actually did his or her own work.
5. Algorithms that produce item clones or variants can reduce the ability of test takers to share test content or profit from another’s answers.

I remember the day that I took my oral exams. There was no faking. There was no cheating. I was in a room, face-to-face, with three professors. Each of them had taught me in at least one course. Of course, it is not realistic to do this for every single individual being certified in a profession or being admitted into the university. But, it demonstrates the importance of having several observations which together confirm that the candidate does indeed possess the requisite competence.There has been interesting progress in the area of secure administrations of unproctored Internet tests. I will mention just a few items that I can recall readily:

1. Kryterion (www.kryteriononline.com) is using data forensics and biometrics to establish that a test is being taken properly.
2. SHL (www.shlgroup.com) is using an initial unproctored test followed by a verification test in a proctored setting to ensure that the test results can be trusted.
3. An instructor named Simon at the School of DCIT, University of Newcastle, used an innovative detection system with online unproctored tests that relied on font colors in Word documents to detect cheaters: http://crpit.com/confpapers/CRPITV42Simon.pdf

At this URL: http://www.westga.edu/~distance/ojdla/summer72/rowe72.html you will find a paper that is very interesting in this context.

Two things are clear: (1) online assessment is here to stay, and (2) ubiquitous security solutions are needed if online assessments are to be trusted.

A senior from Manatee High School passed the FCAT (Florida Comprehensive Assessment Test) in ten minutes by using a “secret pattern” after flunking the test three times. His score was invalidated. Apparently the test score was not invalidated because he used a pattern. Carla Frazier told the news, “FCAT rules do not prohibit students completing the test using any patterns, nor does the test have a minimum time requirement.”

http://www.bradenton.com/local/story/242473.html

We don’t know why the principal invalidated the score. We don’t know what “secret pattern” was used by the student. But, I have an idea what it might have been: “a-n-s-w-e-r-k-e-y.” Ok, I admit to being a cynic and a skeptic at times. This is one of those times.

Consider the facts, and then decide for yourself if you believe the student’s story.

1. Test publishers are very careful to make answer keys as unpredictable as possible. They are well aware of the guesser’s adage, “If you don’t know, choose ‘C’.”
2. Item writers and item reviewers are careful in writing distractors and answer choices to prevent guessers from gaming the test and gaining an advantage. They know that guessers will attempt to deduce the correct answer by analyzing the answer choice lengths and details.
3. Having analyzed a lot of high school exit exam data, I know that pass rates go down with every make up test. Students who fail three times are very lazy, easily confused or just not proficient. Passing the test in ten minutes is not consistent with any of these.
4. Cheaters are often very creative liars and they prey on our gullibility. The news reporter was gullible in writing the story and, for some reason, expects us to be equally gullible.

There are a lot of ways to detect cheating. In this particular case we might have seen any of the following:

1. An extremely high score after having flunked three times previously would be a clear warning sign to the principal.
2. The FCAT, according to the district FCAT coordinator, often contains pilot questions. If the student did very well on all the questions, except the pilot questions, and the answers to those questions matched the answer key form a different form of the test, then the principal would definitely have a “smoking gun.”
3. Sometimes the answer sheet can be modified after the fact. With the right inducement, an insider may be persuaded to change the answers. Erasure analysis would detect this kind of tampering. Perhaps the principal was suspicious and saw a lot of erasures on the answer sheet.
4. It is often the case that the cheaters boast of their exploits and in this case the principal may have gotten wind of the boasting.

Being a student of statistics, I imagine that the student could have finally gotten lucky and passed the test. Distribution theory states that the maximum observed value in a distribution has a much higher mean than the distribution from which the value was drawn. In this case, we have repeated scores on the FCAT for the student. Just by chance alone, if the student’s expected score is reasonably close to passing, after repeatedly taking the test a passing score will be observed eventually.

But, suppose that in my skepticism I am correct. Suppose the student did have the answer key. How would the forensics analyst detect that an answer key had been stolen and used? I have seen three answer-key arbitrage techniques used for exam security purposes, and which could be used in similar situations.

1. The FCAT coordinator disclosed that pilot questions are often used on the exam. Scoring the pilot questions with alternate keys could provide probability evidence that an answer key was in play.
2. I know of a situation where items were intentionally miskeyed and left unscored with the goal of determining whether the answer key had been stolen and used.
3. In another situation, the exam contained a few poorly written questions where the provided answer was ambiguous (This often happens on exams). These questions were exploited in a similar manner to compute probability evidence that an answer key was stolen and used.

The test publisher has many tools and techniques that can be used to trap the unsuspecting cheater. Answer-key arbitrage is one of those.

A teacher, Carla Hammersley, in Michigan resigned recently for allegedly “violating administrative procedures” during this year’s administration of the MEAP (Michigan Education Assessment Program). She denies any wrong doing, but felt that resignation was her only real option. One option offered by the superintendent was three days suspension without pay and a letter of misconduct being placed on file. It appears that the State of Michigan is still investigating the incident.

http://www.leelanaunews.com/blog/2007/11/12/northport-teacher-resigns-over-meap-violation/

The situation has the appearance of being the result of ambiguous administrative procedures for the MEAP and a teacher who is doing her best to encourage the students to do their best work on the test. Four specific administrative violations are listed in the article:

1. Providing information to students during the test that “may have aided in answering a total of five questions.”
2. Encouraging students “during testing sessions, including writing, to include details, follow previously taught formats, and correct grammatical mistakes.”
3. “Returning assessment materials to … students after … they had completed the test, and giving the students the opportunity to edit and revise their work.”
4. Reviewing “a persuasive essay format immediately before administering the … writing tests.”

The teacher says that she didn’t cheat. The school board elected to accept her resignation and pay out this year’s contract ($49,517.60 for the remainder of the year). We don’t have a lot of details, but the decisions appear to be the result of not being able to state definitively that test security was breached.

There should be no ambiguity concerning what cheating is and what cheating is not. All involved in testing need to understand the rules. That is the only way in which tests can be administered fairly and with integrity.

For me, answers to the following questions give practical, no-nonsense guidance for determining whether the teacher’s conduct was inappropriate:

1. Did students gain an unfair advantage as a result of this teacher’s conduct?
2. Would a trained proctor from another district (who had no vested interest in the school or the students) have acted the same way?

When the testing session begins the teacher must set aside the role of educator and assume the role of test administrator. If this cannot be done, there is danger in “crossing the line” and breaching exam security.