Dennis on Data Forensics

Georgia bit her lip nervously as she peered out the rear-view mirror of her car. She had already been idling 10 minutes longer than allowed and campus security would be returning shortly. Then, she saw them, exiting the library. Ignacio was detained by a man in uniform. Vincenzo broke into a run, sprinted to the car, and hopped in. “Step on it,” he said. Georgia sped away. “What about Ignacio?” she asked. “Don’t worry. I have it right here,” he replied as he slipped a digital camera from beneath his jacket, extracted a memory card and handed it to Georgia. She grinned. Now, she would be able to pass the test and become an intern at Waldo & Cramer Industries. Once inside W & C and with her computer skills, her current employers would soon be very, very happy.

The above fictionalized account is based upon an incident which Caveon was asked to investigate in 2004. Our client wrote,

“We had an incident over the weekend concerning the XYZ exam …. The examiner contacted our office during the 3rd section of the examination. Two examinees were acting suspiciously throughout the exam. They had questions about how long the breaks were and what would happen if they returned late from the break. During the break, the proctor noticed that one of the test booklets was not on the applicant’s desk.

The proctors noticed that the two examinees went to their car and came back late from the break. When addressed about the booklet, they said they did not have the booklet and then dropped it from their jacket and said, ‘there it is’. They were allowed to continue, although the proctor told them their scores would be invalidated. They were addressed by the proctor and campus police after the exam and questioned. One of the examinees was released as he stated he had nothing to do with the incident. The other fled the scene in a car that was waiting for him, as he was being escorted to check his car to see if there were images on his cell phone of the test booklet. The names of the suspects are Inigo and Vinny.” (Actual names have been changed.)

Results of Investigation

Caveon conducted an investigation into this incident and we discovered that the two individuals, Inigo and Vinny, were enrolled at a nearby university but they were not enrolled in courses of study or college majors that would be consistent with taking the admissions test connected with this incident. Furthermore, we determined that one of these students had lost his passport during the summer and the other had his driver’s license stolen. The information was corroborated and led us to infer that both of these students were victims of identity theft. Some other individuals committed test fraud in their names.

We also discovered that the test thieves were given the opportunity to steal the test because the test site administrator had not collected testing materials during breaks or the lunch period, as per test administration policy and procedures. One of these individuals, “Inigo,” had taken and failed the test approximately six weeks earlier. We presume that this individual determined that an opportunity existed to sneak the test booklet out of the testing site at that time.

In our report, we concluded that the imposters (or identity thieves) took the exam with the intent of exposing the exam content for one or more of the following purposes: for themselves, on behalf of another individual(s), for mass distribution, or for financial gain. We also suggested that, with suitable revision to the test administration policies and procedures, the likelihood of a security breach could be reduced.

Forensics analysis

Another phase of the analysis was to statistically analyze the test responses. It is difficult to infer “intent to steal” from data analysis, but the data are revealing. One of the statistics that we use in Caveon Data Forensics^TM is known as the bimodality statistic. With this statistic, we assume that most individuals answer the test questions consistently according to the observed performance (or a single level of ability). However, we allow the possibility for some individuals to answer the test questions according to two levels of ability (or in two different modes, hence the name bimodality). Using these statistics we found that Vinny’s test was somewhat aberrant (at the probability level of one in 2,000) and that Inigo’s test was extremely aberrant (at the probability level of one in 200 million). These data, along with comparative “normal” data at the same ability levels, are shown in Figures 1 and 2.

Figure 1: Comparison of Vinny’s test with a normal test

Figure 2: Comparison of Inigo’s test with a normal test

The data confirm that both of these individuals took the exam at two levels of ability. The probability of the high level is shown using the yellow line. The probability of the selected response using the low and high levels is shown using the blue and pink lines, respectively. We infer that Inigo demonstrated more information and knowledge about the test content than Vinny, but both of them appeared to be answering the test questions for some other purpose than obtaining a score and an actual measure of their knowledge of this content area. It appears likely that these individuals were connected with the content area being tested.

This incident is extremely instructive. It illustrates that not all test takers are as they appear and that an unfair advantage may be gained in many ways. I had always wondered whether there would be a motive to steal an identity for the purpose of taking a test and now I know.

The ATP (Association of Test Publishers) conference this year did everything a good conference should do. We networked. We shared industry information. We discussed best practices. We met with clients and vendors. And we created, renewed, and strengthened friendships. Rather than discuss those things, let me share a few observations relating to test security.

Exam security was a hot topic, with many sessions and many serious conversations around test security. Wayne Camara of the College Board asked me, “Was the emphasis on security due to Caveon?” I replied, “I think it is partly due to our outreach effort, and more programs are dealing with security issues.” I think there are deeper reasons.

There were more stories describing successful security efforts this year than I remember in the past. Just to name a few: the FSBPT discussed their breach and resolution in the Philippines, the GMAC caught a proxy test taker in the very act, EMC presented successful risk management cases, and the Mississippi Department of Education has effectively addressed cheating in schools. We celebrate these successes, because they give us confidence that these problems can be solved.

There is deep concern about test and exam piracy. In the past, this concern was primarily expressed by IT (Information Technology) companies. This year many other organizations had the same concern. I heard several instances of exams being stolen from within computer-based testing centers. I have no reason to doubt these reports.

Theft vulnerabilities had been voiced privately in the past, but the discussions were more open this year. I attribute this to at least three reasons: (1) there were new attendees who wanted to expressly discuss security and stayed for the Test Security Summit, (2) the Boston Globe article “Job Exam Piracy Rising,” dated December 26, 2007, gave the topic national prominence, and (3) some presenters disclosed that their entire item banks, including answer keys and digital representations, had been stolen. In the session, “Cheater, Cheater, Pumpkin Eater,” EMC Corporation reported great success in detecting and shutting down test sites where exams are being stolen. Test pirates refused to resell test content because their test sites were shut down immediately after they stole the tests.

To the best of my recollection, there were more lawyers present at ATP this year than any other year. Representatives from at least four different firms had been invited to attend by conference organizers or conference presenters. I have paraphrased some of their very instructive comments below:

“Gather all your evidence in preparation to litigate, but only litigate as a last resort.”

“You can use statistics to invalidate scores and to take other security actions if you can demonstrate that your actions and decisions are made in good faith. The courts are interpreting these actions using contract law and it’s important that your agreements and contracts support your intended actions.”

“All test items are copyrighted, but you must register the copyrights before the items are stolen. Registered copyrights provide stronger protection than unregistered copyrights. There is a special provision in copyright law to protect secure tests for this purpose.”

GMAC and Pearson VUE described initiatives for preventing and detecting imposters. GMAC verifies a candidate’s current photo with the candidate’s registration photo. They attach the photo to the score report. (I call this “testing event authentication.”) Pearson VUE demonstrated Fujitsu’s PalmSecure biometric authentication technology. The readers are priced at around $700, but within reach for secure testing applications.

Gene Radwin and Liz Burns of EMC Corporation captured our imagination. Gene shared his success in detecting users of braindump content using Trojan items. Liz Burns described her security efforts. She visualizes a triangle. At the base of the triangle are honest people who will not lie and will not cheat. At the top of the triangle are those who will cheat if at all possible. In the middle of the triangle are individuals who may cheat depending upon the circumstances. The “at risk group” is where Liz concentrates her efforts.

The Education Division meeting had an interesting discussion concerning the image of testing in education. I think that a positive image of testing is critical. As an example of how incorrect image of testing can be damaging, consider the report that South Africa has effectively banned unproctored Internet testing, because these tests are thought to be unfair, not being secure (reported by Hennie Kriek, President of SHL, USA).

Finally, if you believe that test publishers are cold and dispassionate, let me disabuse this image. I saw a lot of passion and emotion at this conference. Testing professionals are very concerned that tests are administered securely. As an example, Cindy Simmons, State Assessment Director of Mississippi, showed great forthrightness and passion as she described her state’s initiatives to address cheating on the Subject Area Tests.

It’s true there is much work to do. But members of ATP are committed to fairness and integrity in testing. They comprise “the intelligent voice of testing.“

Today is the first day of the annual ATP Conference (Association of Test Publishers). This afternoon I will present a workshop titled, “Strategies and Tactics for Limiting Item Exposure.” We will be exploring innovative ideas for protecting tests and items from theft. It’s easy to understand why test publishers are concerned about test theft. High-quality items are expensive to produce and represent a substantial investment. Item development costs of $1,000 or higher per item are not unusual. In an afternoon, a thief can compromise an investment of $250,000 or more, easily. Most testing professionals will state that item theft is their number one security concern. I discussed this previously in: What is your top security concern?

I can’t share the entire workshop content with you in this short essay. But, I can share with you Gene Radwin’s (of EMC Corporation) intriguing idea of answer-key arbitrage and Trojan items. The idea was briefly mentioned in: Student outwits FCAT with secret pattern. Just as the Trojan horse was the Greeks’ surprise weapon for outwitting the people of Troy, we hope to outsmart users of brain-dump content using Trojan items.

The basic idea of the Trojan item as developed and presented to me by Gene Radwin (email: radwin_gene at emc.com) is to place very easy items on the test which are miskeyed. If a test taker gives the miskeyed answers (and not the correct, easy answers) we have strong evidence that braindump content is being used. The fundamental principle is to create a test-within-a-test to detect test fraud. We booby trap selected items by changing them so that a different answer choice is now correct, and the compromised answer is incorrect. Without knowing which items are booby-trapped, the brain-dump user proceeds in ignorance, until detected. Just to illustrate, consider a math item that I “borrowed” from the SAT practice test.

Table 1: Example of a Trojan item

We do not expect the brain-dump user who has memorized the “Exposed” item to notice the small change in the “Trojan” item. As a result, the cheater will give the originally correct, but now incorrect, answer “C,” and at the same time the honest test taker will give the correct answer “E.” The change in the answer key gives us a leverage or arbitrage point, creating a powerful difference in the statistical expectations.

In order to be effective, several Trojan items will be required on the exam. I haven’t done a rigorous analysis of the statistical power of the procedure, but my current intuition suggests that ten to twelve questions will be needed.

We recently analyzed data where one individual was suspected of having prior access to the test content. Six miskeyed items were present on the exam and we found that the suspect answered all the miskeyed items correctly (i.e., with the wrong answer key). Using item response models, we analyzed the “score” for the miskeyed items. (We do not use standard regression techniques because the data are not normally distributed, being highly constrained and skewed.) These data are shown in Figure 1.

Figure 1: Analysis of 6 miskeyed items

We see two extreme data points in Figure 1, corresponding to the suspected exam and another exam (they had probabilities of one in 5,000 and one in 1,000, respectively). The expected score on the miskeyed items was approximately two. We note that there is no correlation between the raw score on the test and the score on the miskeyed items.

In the above example, analysis of miskeyed items detected a potential testing irregularity. When Trojan items are specifically designed as described above, we expect to see a strong negative relationship between the Trojan items and the total score. In other words, high scoring individuals will provide the correct answer and not the original answer. This negative relationship improves our ability to detect users of brain-dump content.

In addition to my own analyses, one of our clients has told me of great success in using these techniques. For obvious reasons, the client does not want brain-dump users to know which tests are treated with Trojan items and how their cheating is being detected. When cheaters realize they are being punished for using brain-dump content, they will quit using the content. Then we will be satisfied. We just want test takers to do their own work and demonstrate their own ability when they take tests.

It’s easy to understand the pressure that you, the test taker, face when you take a high stakes test. It’s also easy to see why you might want to discuss the test with others. As an example, consider one student’s request for help on the DAT (Dental Admissions Test), posted on August 16, 2007: “For those who took the DAT, can you guys help me out and post some of the question that you remember from the exam?” After a bit of harassment for making such a blatant request, the would-be cheater finally wrote, “k my bad, …” (http://forums.studentdoctor.net/archive/index.php/t-436557.html)

My advice to you, as you prepare to take your high stakes test, is to shy away from any individuals who want you to share test questions or who try to sell you “remembered questions.” Thirteen students from UCLA have been implicated in question stealing and sharing. The American Dental Association has found “their names next to ‘remembered questions.’” “As a result of an investigation last year, the UCLA students face having their passing scores from a 2006 exam session voided and being barred until 2009 from retaking the test, according to documents.” The students have filed a lawsuit stating that “they have suffered personal and professional humiliation because of the allegations and that their careers and plans for further specialized study are in jeopardy.” http://www.latimes.com/features/health/nutrition/la-me-dental23jan23,1,2797664.story?coll=la-health-nutrition-news

To my knowledge, all high stakes testing programs will require you to agree to a confidentiality or non-disclosure agreement before you take the test. Generally you will agree if you violate your agreement and share the test questions with others that the testing program may exercise certain remedies, such as cancelling your score or even banning you from testing. You may also be required to pay the development costs of the exam or be prosecuted under the Trade Secrets Act. As an example, consider the following excerpt from the ADA (American Dental Association) website.

“Examinees cannot disclose (in whole or in part) any test questions or answers to anyone during or after the test, whether orally, in writing, on Internet chat rooms or blogs, or otherwise. The Dental Admission Test is a secure test, protected by U.S. copyright laws. Any unauthorized disclosure of the test’s contents could result in civil liability, criminal penalties, and/or cancellation of test scores.” http://www.ada.org/prof/ed/testing/dat/dat_examinee_guide_2008.pdf

In addition to agreeing to keep the test questions confidential, you will likely agree to not use any stolen or shared test content. By way of example, the ADA Examinee Guide states, “You will not give, receive, or obtain any form of unauthorized assistance prior to or during the test or break periods. By way of example, you will not use or share unreleased test content.” If you breach this agreement or are found guilty of unethical conduct, the testing program may apply penalties, including the cancellation of your test score which may result in the revocation of your license or your expulsion from your selected educational program. A finding of unethical conduct could follow you for several years. You really do not want that.

If you feel that you need help in preparing to take your test, you are not alone. The ADA reports that 46% of those who take the DAT enroll in some test preparation course. Of those individuals, 92% take the course from a private corporation. This is where you need to be a very, very careful. You do not, under any circumstance, want to be involved with an unethical test preparation course or cram school. Gregg Colton stated, “Preparatory schools that are traditionally not licensed or regulated are for the most part corrupt. The school’s sole existence is dedicated to the theft and resale of active examination items. There have been numerous documented incidents of cram schools stealing examination material for almost every occupational or professional licensing exam and certification.” (http://www.ipmaac.org/conf/98/colton.pdf) Let me emphasize this point: If the cram school is raided and your name is found, your standing may be in jeopardy.

Here are some warning signs that you are enrolled in a cram school (Adapted from Gregg Colton’s article, previously cited.). The course may:

1. Emphasize “acing the test” rather than refreshing the subject matter which you should have already mastered.
2. Emphasize the resemblance of their practice questions to the actual test questions.
3. Solicit you to share examination items with course instructors.
4. Start class by asking if anyone in the room is affiliated with the District Attorney’s office, Law Enforcement or other State official.
5. Focus on memorizing test questions and answers and not provide real instruction.
6. Prohibit you from
   1. taking notes,
   2. removing materials from the room,
   3. bringing briefcases, jackets, purses and other such items into the “classroom”
7. Recruit you to work for them by taking tests and remembering test questions.
8. Teach you how to smuggle “study aids” into the testing event.

Lastly, you may feel that it is alright to purchase test preparation materials from the Internet. In addition to the reasons that I cited above, you should be very selective if you do this because, contrary to their claims, many of these websites do not offer any substantive help. They are just operated by scammers who prey upon the desperate. You might be tempted to purchase “remembered questions” from somebody on eBay or through e-mail. Don’t do that. You risk losing your money for nothing of value, or you risk being caught in unethical conduct. Testing programs are getting wise to the fact that high-quality braindumps are available and they will take countermeasures (At least they will, if they follow my advice) which will allow them to detect you.

As an example that countermeasures are being implemented, I quote from http://testbusters.net/ “Microsoft® and CompTIA® have gotten real smart to the brain dump scene, and use it to their advantage to make more money from those of us that use brain dumps. They know we are not memorizing the complete question and answer, thus they use those questions and just change a few words here and there, and ‘Bingo’, a fail, and thus we line their pockets! Questions are reworded every 30 days in some cases!”

Truly, honesty is the best policy. I wish you luck as you prepare to take your test.

Recently there have been many reports of lost databases, stolen computers, and misplaced documents. Is it any wonder that tests and exams are also experiencing the same problems? For example, last November in New Zealand the home of an employee of the Qualification Authority was burglarized and a laptop containing math items for the National Certificate of Educational Achievement was stolen. Despite assurances of password protection, the Qualification Authority revised and reprinted 150,000 test booklets: http://www.stuff.co.nz/stuff/4331442a7694.html

As another example, the completed answer sheets from an exam for the Arkansas State Board of Cosmetology were lost or misaddressed in the FedEx shipment to the scoring agency. Ninety candidates will have to retake the exam: http://www.nwanews.com/adg/News/213242/

Two years ago Caveon’s assistance was sought in dealing with a similar situation. The car of an employee of a major test publisher was stolen. In the car were secured test materials, including an answer key to an upcoming state-wide public school examination. When the car was recovered the answer key was missing. There was not enough time to revise the test. The exam would be administered as scheduled. Our client wanted to know if the answer key was being distributed and if the integrity of the test administration had been compromised.

As we discussed the situation with the client, I was confident that we could detect a widespread breach. But, could we detect a situation when just a few students used the lost answer key? There was no doubt in my mind if the thief knew the market value of the answer key that it would be sold on the Internet. I knew this from first-hand experience. While I was teaching at the University, a dual-campus administration of the test coupled with a time lag between administrations led to the answer key being disclosed. Three of my students obtained the answer key to the exam through a Yahoo chat room. They scored 100% on all the questions, except the essay question, which they refused to answer.

The client gave us the following details about the test. There were 54 questions on the exam with 10 field test items and 44 core items. There were about 2 dozen different forms of the test. The forms all contained the same core items in the same locations, with form differences due to different sets of field test items. Slowly an analysis plan began to emerge. Because the answer key for only one of the forms was lost, we could score the field test items for all the other forms using the waylaid answer key. Scores on the field test items would be the keystone of the analysis.

We assumed that any student using the stolen answer key would not know which items were field test items and which were core items. We also assumed that the student would answer all the items (with potentially a few mistakes) using the stolen answer key. It was easy to determine that a widespread dissemination of the answer key had not occurred. Statistical methodology dictates that statistical tests are performed assuming the null hypothesis (i.e., the answer key was not in play) is true. Under this assumption we found that less than 2% of the tests had “high scores” (i.e., scores above the 95^th percentile of the distribution), when 5% were expected. This was very good news. There was not a wide-spread dissemination of the answer key.

Next, we hypothesized that a few teachers or school administrators might have received and used the stolen answer key. Using a probability inversion formula, we rank ordered the schools by the proportion of tests where more than six correct answers on the field test items (using the stolen answer key) were found. We found that the proportion of schools in the upper tail (above 10%) was less than 7% when 10% were expected. This was good news. It meant that if the answer key was disseminated, it was not likely to have occurred through teachers or administrators. (We also visually inspected the 30 most extreme schools for “perfect” scores of 10 on the field test items for all the other forms except the one associated with the lost answer key. Nothing untoward was found in any of those schools.)

Finally, searching for the proverbial needle in the haystack, we hypothesized that a few isolated students may have been able to receive the answer key through personal contact with the thief on the Internet. In order to attack this problem we created a Bayesian probability model, where we estimated the probability that the stolen answer key was used by a particular student conditional upon the test score. Using this model we inferred a 95% upper bound on the proportion of student who used the answer key to be less than .09% (or nine in ten thousand). The five most extreme tests were visually inspected, and not one of them had a “perfect” score on the field test items, using the lost answer key.

The results of the analysis gave our client sufficient confidence to trust the integrity of the test administration. In order to place perspective on these statistical estimates, we note that the estimated bound (i.e., .09%) on answer key compromise is much, much lower than the actual proportion of students who copy from each other in the normal test taking situation. While we could not prove that the stolen answer key had not been used, we concluded the following:

If any students have gained access to the answer key, the data indicate the answer key has not been shared with friends. And, if the answer key was used, its use was isolated.

With 95% confidence, no more than .09% of students used the compromised answer key. It is very likely, in fact, that no student actually used the compromised answer key.

The above situations illustrate the importance of properly securing test materials. They also illustrate that by using innovative and defensible statistical analyses, testing program administrators may know the degree of security risk that is present. The analysis of the waylaid answer key illustrates the power of data forensics in protecting and maintaining exam and test security.

Holmes handed Watson a note and said, “This is the message which struck Justice of the Peace Trevor dead with horror when he read it.”

http://sherlock.jamesbickers.com/the-memoirs-of-sherlock-holmes/the-adventure-of-the-gloria-scott/

Justice of the Peace Trevor was struck dead because the note contained a secret message. This is the essence of steganography. “Steganography is the art and science of writing hidden messages in such a way that no one apart from the sender and intended recipient even realizes there is a hidden message.” http://en.wikipedia.org/wiki/Steganography

In our modern world, we usually use the term “digital watermarking” for steganography when we refer to electronic files that contain hidden secrets. Digital watermarking techniques are being used to verify the illicit distribution of copyrighted photos, with the intent that the copyright holder may receive royalties for the unauthorized distribution. The recording industry has been using digital watermarking in the form of Digital Rights Management (DRM) in order to prevent digital theft of copyrighted movies and recordings.

The big news from the digital music industry is that during the past year, the four largest digital music labels have all dropped DRM from MP3’s that are being distributed on line. Sony BMG was the last hold-out and recently announced that DRM would no longer be used for MP3’s.

http://www.techtree.com/India/News/Sony_BMG_Drops_DRM_Protection/551-85784-643.html

http://www.zdnetasia.com/news/internet/0,39044908,62036088,00.htm

The fact is that album sales have declined. As one article states, “In short, downloads are up, physical sales are down, and downloads are not picking up the slack of lost sales.” http://www.pollstar.com/news/viewnews.pl?NewsID=9114 In other words, if the future of music sales is in downloads, then the recording houses have very little choice except to remove DRM from downloaded music. The DRM software is distasteful enough to consumers so that they will go elsewhere for music.

Professionals in the testing industry have been talking about “digital watermarking” for some time as a means of protecting tests. However, the term “digital watermarking” is a misnomer because true digital watermarking involves bit twiddling within the electronic content. You can’t twiddle the bits of text files and expect the modification to remain hidden.

Steganography can be used to protect tests by hiding information in the test so that when a stolen copy of the test is acquired (e.g., purchase from a braindump site) the exact copy of the test that was stolen may be verified. In other words, this becomes a means of detecting where and when a test is stolen. This information is used to identify the weak link in the chain of custody, so that the person responsible for the security of the exam when the test was stolen may be identified. The information cannot identify the thief, but it can identify the individuals who were entrusted with the custody of the test at the time of theft.

At Caveon we have engaged in a few projects of this nature on a limited basis. At the request of one of our clients we injected small editorial changes into the text content of selected items and then compared the various versions of the test with stolen content purchased from the Internet. We determined to the client’s satisfaction that the test theft did not occur inside their test development organization. Instead, it occurred after the test was published.

The above work was labor intensive and could only be performed on a small scale. Our research indicates that the point of risk for test theft is at the test delivery sites, which number in the thousands. An effective steganographic system will require encoding hidden information into the test content in order to detect the points of theft in the test delivery channel. A steganographic system capable of providing this kind of detection must be automated and it must be implemented on a wide-scale. This means that potentially thousands of test versions must be generated and the decoding system must be able to reliably determine which test version was stolen.

We have been conducting research and developing algorithms for such a steganographic system. Whatever method is used for hiding information, it cannot affect the performance of the test. It must be truly unobtrusive. This is a big challenge, because modifications to the item text can potentially change the difficulty of the test questions.

If you are interested in this topic you might look at these websites:

http://www.jjtc.com/Steganography/

http://www.watermarkingworld.org/

The number one security concern of testing professionals is exam theft and piracy, according to a survey that Caveon conducted at NOCA in 2005. We asked the question: “Which of the following are security concerns for you? (Please check as many as apply).” One hundred participants responded in the following manner:

Given the news article from the Boston Globe, “Job exam piracy rising,” published December 26, 2007, it would be interesting to repeat the above survey.

http://www.boston.com/news/nation/washington/articles/2007/12/26/job_exam_piracy_rising/?page=full

This is a very important article because while data are not provided to support the headline that exam piracy is really on the rise, it strongly illustrates the impact of exam piracy on the testing industry and the fact that current remedies cannot effectively counter many instances of test theft. This is particularly true for information technology certifications.

I have been studying the problem of exam piracy for a long time, and can offer a few insights. First, the asset that must be protected by exam security is the integrity of the examination process and the credibility of the test result, not the item bank or the test form. Second, the correct perspective of the relationship between certifying authority and test thief is a host-parasite relationship. The exam pirates live and draw from the vitality of the certification, devaluing it with their success. Lastly, a year ago we analyzed the data forensics analyses that we had performed for more than 20 certification programs. We determined that three main factors were directly related to exam piracy: (1) the mission and role of the certification, (2) the test administration model, and (3) the security of the test administration channel.

Protecting the integrity of the examination process – Current legal protections against exam piracy involve copyright and trade-secrecy statutes. Unfortunately, these can only be invoked after the integrity of the test is breached. They usually involve protracted investigations followed by even lengthier legal proceedings. In the meantime, the test is compromised and keeping it in service further erodes credibility in the examination process. The DMCA (Digital Millennium Copyright Act) provides some assistance when the stolen content is accessed through on a US-based ISP. But, legal remedies are few. In fact, legal jurisdiction of crimes committed over the Internet is at times very unclear, compounding the problem.

Host-parasite relationship – A certifying authority such as the FSBPT (Federation of State Boards of Physical Therapy) derives its existence from maintaining and administering the exam. An attack on the integrity exam is an attack against its very existence and must be countered. On the other hand, a company such as Microsoft provides certifications in support of its business. The vitality of such a company is derived from product sales and service, not from the certifications. Thus, as long as attacks on the exam do not adversely affect the core business of the company, it may be able to withstand parasitical infestations. In either case, the parasitical exam pirate bears no goodwill toward the certifying authority and has no compunction in destroying it.

Mission and role of certification – Resources within any organization are deployed according to its core mission or function. In the context of exam security this means that operational budgets and legal expenditures are prioritized accordingly. For example, the lawyers for an organization such as FSBPT will be more willing to tackle exam security issues than will lawyers for the typical IT company. This is because the lawyers for IT companies are involved in patent protection, maintaining business contracts, and other core business functions.

Test administration model – Most high-stakes testing programs administer tests according to pre-determined testing events. A new test (which may use previously administered items) is constructed for each event, thus decreasing the chance that stolen test items will be present on the new test. This practice means that it is more difficult for the exam pirate to profit from the testing program. On the other hand, when the same test forms are kept in service for a protracted length of time, the exam pirate has a distinct advantage in stealing and selling the test content.

Security of the test administration channel – The article from the Boston Globe states, “Technology companies in particular have accepted lower levels of security in order to have testing centers in distant corners of the globe.” The lower levels of security involve contracting the test administrations with third-parties who may have never had a background check, who may be operating cheat sites, or who don’t care exactly how they make money. A rogue test site administrator can very easily steal a test by merely recording every testing session (i.e., with a video camera) and then transcribing it. I believe that some these individuals have discovered how to actually pilfer the test content electronically, avoiding the need for transcription.

Hopefully, thinking about the above observations will help you understand why exam piracy is not going to be solved easily. Some testing organizations are being seriously affected by exam piracy. Only time will tell whether they will be able to successfully ward off the pirates, or not.