Caveon Security Insights

By: Dennis Maynes, Chief Scientist, Caveon Test Security

This week, Julian Assange, founder of WikiLeaks, announced that his organization is running out of money and may be forced to cease operations by the end of 2011. On October 24, 2011 Reuters reported: “WikiLeaks says ‘blockade’ threatens its existence.” (Source: http://www.reuters.com/article/email/idUSTRE79N46K20111024) The blockade occurred when the major financial processing firms suspended their agreements with WikiLeaks, after WikiLeaks released thousands of secret US diplomatic cables in December, 2010, and threatened the Bank of America with the release of internal documents which resulted in a 3% decrease of Bank of America’s share price.

Assange claims the blockade is illegal and has filed anti-trust lawsuits against Visa and Master Card. On the day before the blockade, WikiLeaks received $135,000. Currently, WikiLeaks receives less than $10,000 per month. The net effect of the blockade to WikiLeaks has been the loss of 95% of its operating cash.

Whether you agree with WikiLeaks’ goals or not, it is clear that WikiLeaks has routinely infringed upon the rights of copyright holders by distributing information and documents without authorization. If it is not obvious why this story has important test security ramifications, let me make it clear: (1) many websites, operated by pirates and thieves, infringe upon the copyrights of secured exam content, (2) it has been very difficult to effectively shutdown this activity, which is costing testing organizations millions of dollars per year in lost test development expenditures, and (3) if payment processors would agree to cease providing services to these thieves and pirates, many of them would fold. The WikiLeaks story demonstrates that copyright infringers will have a difficult time remaining in business without the support of payment processors.

At Caveon, we have been very successful in removing copyrighted exam materials from the Internet. Often our success is based upon respectful and courteous requests to unintentional copyright infringers. However, respect and courtesy do not work against pirates and thieves. At that point, potentially expensive legal action must be commenced.

An alternative to expensive legal proceedings is to work with payment processors to protect their brands. For example, Visa does not want any transaction to bring disrepute upon its brand (source: http://corporate.visa.com/_media/visa-international-operating-regulations.pdf). If we, as an industry, can convince the payment processors that the sale and distribution of pilfered exam content is disreputable, we may be able to slow the flow of money to the test thieves and protect valuable exam content.

What do you think? How can we help payment processors understand that their services facilitate the distribution of stolen exam content? Should ATP (Association of Test Publishers) contact the payment processors, on behalf of its members?

Several months ago, Ben Mannes, Test Security Director at ABIM, expressed this thought: “ATP should be trying to get a meeting with Victoria Espinel [White House intellectual property czar], bring 1-2 industry security experts, and state the case as to why exam content is a vital component to our nation’s infrastructure requiring heightened public sector IP enforcement.”

***************

Please Comment Below, Thank you for Reading

By: Steve Addicott, Caveon Vice President

October is an important month for Caveon. Eight years ago in October, 2003, several assessment industry veterans formed a small consulting company focused solely on improving the security of our clients’ test programs.    That company is Caveon Test Security!

Fast forward to 2011, and it’s gratifying to consider what this entrepreneurial group of test security zealots has accomplished.  Since that fateful October day, we have

• conducted over 50 Security Audits of leading test organizations and vendors,
• flagged and removed tens of thousands of internet-based risks, and
• conducted statistical analyses of over 30,000,000 test instances for many of the largest, most important test programs in the world.

As I consider the number and breadth of these engagements, perhaps it is worth sharing a few of the core values under which we always operate:

Confidentiality

Throughout our years of operation, one fundamental operating principle has always applied:  client confidentiality.  We never reveal the details of our client engagements without the express approval of our clients. Our clients require and appreciate this sensitivity as we investigate security incidents and provide reports on our forensic analyses. This is not secrecy– this privacy stems from respect for our clients and for the right to privacy of individuals and organizations.

Innovation

We constantly strive to improve means and methods for strengthening exam security. We are always interested in sharing the nature of our work.  Not only do we share our methods and science with clients, client stakeholders, TAC members, educational measurement researchers, and other appropriately interested parties; we are committed to furthering the science around test security. We regularly present at conferences and webinars where we openly share our Caveon approach, theories and methodologies. In fact this last year, we have presented at conferences in Phoenix, Orlando, Chicago, Seattle, Washington DC, Hong Kong, and Prague.

Conservative Recommendations

When we conduct an engagement, our approach is to focus on the situations and incidents that are most egregious, as evidenced in the data and the results that we analyze. We highlight those problems that are most readily identified, documented, and ideally, resolved. Dealing with these problems effectively will have the greatest positive impact to the overall validity and security of test results. This reasonable approach helps our clients, most of which suffer from ever-constrained budgets and resources, effectively concentrate their time, resources, and dollars where the likelihood of inappropriate test taking is highest.

Lastly, our growth and success is directly attributable to a few overarching principles—We always strive to exceed our clients’ expectations, comport ourselves honorably, provide valuable services, and share, as openly and honestly as we can, recommendations for improving the fairness and validity of our clients’ test programs. These principles result in proven, practical protection for our clients, and we intend to follow them for another eight years!

Please Submit Your Comments Below. Thank you!

By Steve Addicott, Vice President of Sales, Caveon Test Security

Over the past few weeks, I have participated in several planning sessions with Caveon clients who contract with us to analyze test results.  This service, Caveon Data Forensics ™, represents a proactive means to better protect their programs by identifying statistical anomalies that may indicate cheating.

While each of these programs is different (state education, IT certification, medical licensure, construction certification, etc.), it’s interesting to me that they face common challenges in confronting test fraud.  For each of these programs, test results matter…they really matter…in making important decisions in the lives of test takers (and in education, teachers and principals).  Thus, the integrity of the test administration matters greatly, too.

My overarching impression?   Tackling test fraud head-on is not for the faint of heart.  It takes commitment—a genuine, unwavering commitment to fair and valid test results—to say “We know there is a subset of our test-taking population that is taking shortcuts, and we’re going to do something about it.”

These days, everyone is busy.   So, why would any sane test program leader willingly add to his/her workload?  The results of our Data Forensics analyses do just that.  We identify:

• candidates/students that may require invalidations;
• test centers/schools that merit investigations; and
• items/exams that should be retired and/or revised.

The committed leaders we work with understand that this is what is required to be able to stand in front of stakeholders and proclaim that their test administrations are fair and valid.

Another important takeaway I’ve gained is that a successful Data Forensics program requires the cooperation and coordination of many  groups.  Last week, I met with a client, a large state department of education.  Its leadership, in order to ensure the data forensics program possessed real teeth, sought cooperation with several other state departments:

• Legal, to ensure any sanctions resulting from the data forensics would hold up in court;
• Communications, to ensure sound, consistent messaging to the media and public alike;
• Inspector General, for conducting investigations; and
• Professional Practices, in case sanctions might be brought against a state certified educator.

This sort of cross-organizational coordination is not easy to facilitate, but critically important in the fight for fair and valid testing.

If you’re considering how you can augment the security of your program, you might find one of our company webinars to be a help.   In “Don’t Shoot The Messenger”, three Caveon clients present the good, the bad, and the challenging in instituting program invalidations through data forensic analyses.  You can get a copy of the webinar slides here:  http://caveon.com/df_blog/ .

The ACLU is opposing a pilot project in Rhode Island to track students as they enter and exit school buses. “Steven Brown, executive director of the Rhode Island chapter of the American Civil Liberties Union, [called] the plan ‘a solution in search of a problem’ and saying the school district already should have procedures in place to track where its students are.” “There’s absolutely no need to be tagging children,” he said. “The program raises enormous privacy and safety concerns, he added.”

If my research is accurate, there have been at least four previous projects for tagging children in schools with RFID (Radio Frequency Identification) chips in the United States:

1. Enterprise Charter School, Buffalo, New York (2003) – badges on children, tagging library books, school cafeteria purchases, and visits to the school nurse;
2. Spring Independent School District, Houston, Texas (2004) – school bus pass program that is still operational;
3. Brittany Elementary School, Sutton, California (2005) – badges on children, due to public outcry this project was cancelled which lead to the Senate of the State of California debated banning RFID chips to identify people in the state; and
4. Tucson Unified School District, Tucson, Arizona (2007?) – school bus pass program that may still be under discussion.

In the UK, two clothing manufacturers are sewing RFID tags into school uniforms for the express purpose of tracking students while they are in school. RFID tags sewn into clothing are not new, and neither are RFID badges in the work place. And, now RFID badges are being used in universities. The University of Chicago is revamping all their student id cards primarily to ensure secure building access. Another application of RFID technology is a label affixed to your cell phone at Slippery Rock University, north of Pittsburgh, Pennsylvania, which allows for payment processing.

The idea of chipping children is controversial. On one hand, privacy advocates warn of possible abuses and intrusions. On the other, security proponents promote increased safety. In between are administrators who want improved efficiency and convenience. No one is seriously considering implanting RFID chips into children yet. But, this is happening for patients with Alzheimer’s and dementia, as well as being seriously considered by the British government for prisoners. And, the University of Washington has started a human experiment in the computer science building to assess the possibilities of RFID tracking. Several states have passed legislation prohibiting a person from being forced to accept RFID implants, which are approved by the FDA. (Image source.)

There is no doubt that RFID tags can be abused. With an RFID reader, a bad person can gather information about you surreptitiously. A bad person with a database can profile you, and even create an inventory of your belongings. But this potential exists today, even without ubiquitous RFID tags and readers. Bad people with cameras can gather information about you surreptitiously and create an inventory of your belongings. They do it to children, families and the elderly. The concern about RFID is that this can be done more efficiently. If you use membership cards and discount shopping cards, your purchases may already be tied to you in some database, somewhere, that at sometime in the future may be hacked by someone. To a certain degree, the anti-RFID movement promotes the fear that at some time your information will be stolen.

We should be aware that the school district personnel who are investigating this technology are trying to solve real problems. It’s important to keep track of library books. And, it’s even more important to know that students are entering and exiting the school buses at the proper times and locations. We live in a changing world and what we dismiss as paranoia today may become essential tomorrow. For example, there were no lockers in my elementary school. The first time I saw a locker was in junior high. While in high school, we moved to a small town and the lockers did not lock in that school (unless you brought your own lock from home). Given our changing world, I would be extremely surprised if this situation still exists in my alma mater.

I’m somewhat surprised that the ACLU has not opposed cell phones in schools. Who would have ever imagined that in the name of privacy and safety we would allow everyone to carry a camera to school and take a picture of anything there (e.g., students spitting in a teacher’s water bottle or a teacher filming the girls bathroom)? But that is precisely what has happened with cell phones. We can’t pry cell phones away from students. There is also great potential to abuse cell phones, as demonstrated with the FBI’s ability to remotely activate a cell phone’s microphone and use it to eavesdrop on nearby conversations. If cell phones are ever fitted with RFID tags, this entire debate could be over.

Mr. Brown from the ACLU is right. There are safety and security concerns with RFID devices. However, he doesn’t seem to understand those concerns. RFID chips can be hacked and the information from those chips can be transferred to other chips. As an example if the RFID card allows access to a secured area, a bad person may pilfer the electronic codes and in essence make a copy of the electronic key, as demonstrated by James Van Bokkelen. If the chips do not have proper electronic safeguards the information may be overwritten or used illegitimately.

While I have not directly addressed testing, there are implications for using RFID chips in testing which I will discuss the next time I write. But today, I just couldn’t resist this topic. In my opinion, we need to ignore the fear mongering and we need to use this technology wisely. RFID technology is not a panacea, but it can solve real problems.

The Association of Test Publishers (ATP) Conference of 2008 ended yesterday. As always, it was a good conference. In 2004 we stated, “You can’t manage what you don’t measure.” Being a sponsor of the conference, we placed a bag of M&M’s (i.e., manage and measure) in each attendee’s conference packet. And, we printed the message on the hotel room key cards.

I have just completed analyses for three testing programs and I am so impressed with what they have done that I want to share their results with you. Good news concerning exam security is refreshing in the midst of so many cheating stories. We recognize dramatic acts of heroism, but often ignore the good that happens with steady, persistent progress. I am so proud of these three programs. They are achieving their common goals: “Reduce cheating, strengthen exam security and emphasize ethical test taking.” The data demonstrate this convincingly. Caveon’s message at ATP this year was, “The answer is in the data.” So let’s look at the data.

Figure 1: Percent of anomalous tests for three programs

Let me describe the data in Figure 1. The percent of anomalous tests for successive analyses are plotted for each program. A trend line has been fit to the data to aid your eye in visualizing the trend pattern. An anomalous test is one that deviates from normal test taking, and will exhibit at least one of the following: aberrance (answering hard questions correctly and missing easy questions), large numbers of erasures, inexplicable score changes from a previous test score, or excessive similarity in the selected answers with at least one other test. An anomalous test does not mean the test taker cheated. For example, when we observe excessively similar tests it is very likely that one person cheated (the copier) and the other person did not (the source). The percent of anomalous tests does not measure the precise number of people who have cheated, but it is highly correlated with that number.

These data are important because they demonstrate that all high-stakes testing programs, irrespective of industry or application, can effectively reduce cheating. They illustrate that reductions in cheating can occur with persistence and dedication. Let me briefly describe each program and some of the positive steps they have taken.

Program 1: This program provides a professional certification with high security. We estimate that there was a 45% reduction in cheating in three years. They have followed up on every case that appeared to be a security violation and every test site that appeared to have lax security. They have emphasized proctor training. They are now reviewing their test taker agreements, proctor training, identification procedures, and physical security with the intent of using the best known security protocols.

Program 2: This program is a public education program. We estimate that there was a 72% reduction in cheating in two years. They have rewritten their test administration manuals and have begun test administration monitoring. They assign a conditional status to extremely anomalous test results and require local review of those test results. They are receiving reports that the students being flagged are admitting to having cheated.

Program 3: This program administers tests in the service industry. We estimate that there was a 78% reduction in cheating in one year. They have stressed ethical test taking. They have revised their test taking agreements and strengthened test administration policies to allow for scores to be invalidated with an appeals process. They have refreshed test forms which appeared to be exposed. They are researching the next phase of security improvements: test site monitoring and appropriate disciplinary measures for test administration personnel who may be helping test takers inappropriately.

These very different programs were the same in one important way: They started where they were, they created a plan, and they were not discouraged. Each was taken back by the first data forensics report (we always find something disconcerting), but they pressed forward and executed their plan. Best practices used by these programs include: test site monitoring, emphasis on ethical test taking, invalidating scores as per policy, refreshing tests which appear to be over exposed, and updating their security procedures.

Let’s give credit where credit is due. The numbers are impressive and the data do not lie. These programs have earned our respect and admiration.

They say that cheaters only hurt themselves. In all honesty, I think that a cheater said that and we believed him. It is often the case that cheaters hurt the people who gave them the test more than themselves. If you are responsible for giving tests, some fool will eventually cheat on your test. How you handle cheating incidents can make or break you.

When you started out in your career and you began giving tests, you probably didn’t imagine that the most demanding aspect of your job might be what to do about cheating. The first time you encounter this and when the media spotlight is focused on you, you will probably wish you were a rattlesnake handler or a bomb disposal expert. You create and give tests. And, you’re good at your job. You never intended to become a test cop. Let me suggest that you anticipate and prepare for cheating incidents now, before they happen. We call it security incident response planning.

Speaking of cops, there have been a number of stories concerning police departments and cheating on tests recently. In the summer of 2007, information about the police promotion test in Boston was leaked to several officers, as reported by WBZTV. In another story, theState.com reported that twenty-one police officers in Columbia, South Carolina were implicated in cheating when they either cheated, helped others to cheat, or knew of the cheating but failed to report it. And, Houston’s crime lab was in the news twice for open-book cheating, which resulted in the shutdown of the lab, as reported by the Houston Chronicle on October 6, 2007, and January 26, 2008.

The above stories illustrate the importance of responding appropriately to cheating incidents and testing irregularities. You will not be judged harshly because a few miscreants decided to cheat on your test. But, you may be embarrassed completely if you do not address the problem adequately. Your program may suffer a loss of credibility. The public confidence in those you certify may be eroded. And, adding insult to injury, the media may portray you as a fool and a blunderer.

Your security incident response plan should be suited to your organization’s needs and requirements. There are a lot of questions that you should answer. Let me list a few:

1. What discipline should the cheater(s) receive?
2. Is the discipline appropriate? If it’s too harsh you may be perceived as being unfair. If it’s too lenient you may be judged as playing favorites.
3. When will you inform the public about the security breach?
4. What will you do if the media learns of the breach before you announce it? Or, before you learn of it?
5. Is an investigation needed?
6. If so, how will the investigation be conducted? Who will conduct the investigation?
7. What information will you share with the media?
8. What information will you keep confidential? What justification do you have for not sharing everything?
9. Who will be responsible for communications and media relations?
10. Is your security incident response plan recorded in policy form to guide you?

As you can see, my list focuses on “doing the right thing” not just on “looking like we are doing the right thing.” Reporters, in particular, are very quick to suspect a cover up or to suspect they are not being told the truth. And, if you are responsible for a testing program which is accountable to the public (e.g., tests in schools or tests involving public safety), it is vital that you maintain the public trust.

One way that you can sharpen your skills in this area is to “simulate” what you would do in specific cheating situations. During the course of a year, just about every type of cheating will be reported in the news. You can stay current with these stories by reading Caveon’s “Cheating in the News.” You can sign up to receive CITN notification by e-mail about twice a month on the lower right-hand corner of the main Caveon web page. Read the stories. Discuss the stories with your staff. Does your security incident response plan tell you how to handle the problem, if it happened to you?

Just as we expect our local emergency response teams (i.e., police, firefighters, and paramedics) to prepare for disasters, we should prepare to handle cheating incidents. A properly executed security incident response plan can keep an incident from becoming a disaster.

Ten years ago the New York Times criticized ETS, claiming that ETS elected to keep quiet rather than publicize exam security breaches. When you, as a testing program manager, have a full-scale security breach on your hands what will you do? I can imagine that it was a very difficult decision within ETS whether to “keep the lid on” the story or to let the story be told. This appears to be a “no-win” situation. If you publicize the security breaches you may seriously undermine your testing program. If you keep quiet and the word leaks out, like it invariably does, your own credibility may be questioned.

Read stories of cheating in the news to learn how the media might portray your cheating incident negatively. Journalists print newspapers and sell advertising. Sensational news is good copy for them. It is especially important, when under spotlight of the press, that your testing program be viewed as being fair, responsible, and ethical. In my experience, reporters will probe for any apparent contradictions, irregularities which could have been avoided, or supposed dismissal of the severity of the situation. If they find any thing that might be construed as an irregularity, it will probably be printed. In my opinion, it’s better to tell your own story first, rather than let reporters interpret the situation in a potentially harmful manner.

I wish you the best as you formulate your security incident response plan. If you could use additional guidance in preparing your security incident response plan, a Caveon test security director will be glad to consult with you.

The State of Florida recently imposed a cell phone ban on students while taking the FCAT. All the parents of school children in the state received a letter explaining the ban. On the other hand, the Legislature in the State of Utah voted down a bill that would require school districts to establish policies governing cell phone use. The sponsoring legislator said, “[Cell phones] can be used to cheat. We’ve had inappropriate photos transmitted. The problem is pervasive.” An opposing legislator was quoted as saying that “he thinks electronic devices could be better used in education and wouldn’t necessarily like to see policies that simply prohibit them.”

In another story last week reported by Wave3 of Louisville, we read: “Teachers at Oldham County High say they’ve had problems with students using their cell phones to cheat in class. ‘I saw a boy texting under his desk during a test. Then I picked it up. Clear as day it said number five –D- and I took it to the office and we were able to trace the number and it was to another student in the same class,’ said Newkirk.” Now contrast that experience with this column from the Muskegon Chronicle, where the writer claims that gadgets don’t help cheaters. The following points were made:

1. Yet research indicates that cheating in high school and college isn’t any more common today than it was 30 years ago.
2. “And 99 percent of cheating is still done the old-fashioned way, like copying from a neighbor,” said Scott Gomer, media relations director for ACT,
3. But in her four years at Northview High School in Plainfield Township, Kelsey Perras has heard of someone pulling out a cell phone to take a picture of a test “only once,” she said.
4. “High-tech cheating isn’t really something you see a whole lot of,” said Hudsonville High senior Travis Martin. “Most people won’t pull out their cell phone during a test. It’s tough to make that discreet.”
5. Perras said cheaters at Northview are caught more often than not.

Muskegon must be a very sheltered place with extremely astute teachers. The credibility of each of the above statements is easily challenged. I see a lot of data and from what I see, I feel very confident in stating that cheaters are rarely caught. The only way that I can explain some of the cheating I see is through wireless communications. And, research from the Josephson Institute and Center for Academic Integrity convincingly shows that cheating in school is rising and has been rising for the last two decades.

Confusion concerning cell phones in schools is raging throughout the whole country. The issue is being intensely debated in New York City where it has spilled into the court system. Last spring the New York State Supreme Court upheld a ban on cell phones imposed by New York City in 2006. The Supreme Court decision is now being challenged in appellate court.

Surprisingly, security arguments are given by both sides of this debate. Proponents of cell phones argue that parents and administrators need constant contact with students, because without constant contact student security is jeopardized. Opponents of cell phones in schools cite privacy violations with videos posted on the Internet of students in restrooms and teachers disciplining students. And, of course, they do not overlook the implications of cheating. As reported by WSAZ, the solution at Marshall University has been to allow each instructor to determine in their course syllabus whether cell phones during tests are banned, but to not restrict cell phone use on campus.

Penn State has addressed the issue by creating secure testing environments, where the computers do not have Internet access and where cell phone transmissions are silenced. The technology they are using includes: secure workstations, cameras and monitors on every test taker, and metal-lined testing rooms (known as faraday cages) that passively prevent wireless communications. While this may seem extreme, contrast this with the exam breach of 2004 in South Korea where 314 test results were invalidated after police discovered answer keys being transmitted using text messaging. As another example, consider the January 2, 2008 report by the Boston Globe where firefighters in Boston sent text messages from the restroom to cheat on their exams.

It is clear that cell phones are used to surreptitiously cheat on tests. People, in general, feel strongly that cheating shouldn’t be tolerated on tests. We don’t want doctors, lawyers, nurses, accountants, firefighters, police or any other person who provides a service to us to be an incompetent, bumbling cheater. On the other hand, the public sentiment appears to be confused when it comes to setting aside the cell phone while an exam is being given. The public seems unwilling to restrict the individual privilege of being able to communicate with a child in school while taking a test in order to prevent cheating.

The principle of fairness and integrity dictates that students should have a level playing field. It is very difficult to convince me that the playing field was level when cheaters in China were caught with radio receivers in their shoes:

Police in Jiutai, in the northeastern province of Jilin, became suspicious when a mini-bus remained parked outside a school hosting the exam on Thursday, Xinhua said.

Inside, they found three people, “two of them staring at a computer screen and talking into a walkie-talkie,” Xinhua said.

A student in the examination hall used a wireless microphone to read out the questions and received the answers from the van; Xinhua quoted their confessions as saying.

Police had found some 42 pairs of so-called “cheating shoes” with transmitting and reception ability, selling for about 2,000 Yuan each, in a flat in Shenyang, the provincial capital, state media said on Thursday, adding that they — along with “cheating wallets” and hats — had proved popular this year.

There is no confusion in my mind on this issue. But, I’m just a statistician and who am I to know differently?

There is a children’s game known by various names as “Whisper,” “Secrets,” or “Gossip” where a secret is shared and passed from one player to the next. The last player hearing the secret says it aloud, often with hilarious results. These same distortions happen in the news media, as journalists cite other reports or each other. Such a misquote from the Star-Telegram concerning additional security announced by the TEA (Texas Education Agency) for the TAKS (Texas Assessment of Knowledge and Skill) caused me to pause and reflect about using statistical evidence to “prove” that someone cheated on a test.

The reporter wrote, “Among other security measures: … Scramble field test questions on tests to provide proof if someone is copying someone else’s answer sheet.” (Italics added.) http://www.star-telegram.com/news/story/433614.html. Being well aware of the controversy surrounding the use of statistics, alone, to prove cheating, I immediately doubted the accuracy of the above statement. Actually, on June 7, 2007, Shirley Neeley announced that “the Texas Education Agency today will immediately initiate the following: … analyze scrambled blocks of test questions to detect answer copying…” TEA later clarified that the scrambling would only involve field test items. The Dallas Morning News was quick to criticize the scrambling plan, but I applauded TEA’s intent to detect cheating behavior using statistics.

We naturally ask whether statistical evidence can be relied on to detect cheating. Many authors have expressed the opinion that statistical evidence must be corroborated by eye-witness accounts before making allegations of cheating. I can understand this position if the statistics are not reliable. In my opinion, reliable evidence must meet the following conditions:

1. It must be factual,
2. It must be objective,
3. It must be credible, and
4. It must be defensible.

If statistical evidence meets the above conditions, I believe that it can be relied upon, whether corroborating eye-witness accounts are available or not. Statistical evidence is

1. factual when it is based on test result data (an actual record of the test event),
2. objective when it provides a statistic with a probability statement,
3. credible when the statistics have been shown to work because the models accurately depict actual test taking, and
4. It is defensible when the underlying science withstands scrutiny.

An additional fifth criterion the evidence must meet for taking action on a suspected instance of cheating is that the evidence must be strong. Statistical evidence is strong when the calculated probabilities are so small that we no longer believe the observed data are the result of normal test taking. Statistics can provide guidance for determining how strong is strong enough to take action, but ultimately the establishment of a probability threshold (i.e., the strength of the statistic) is a matter of policy that must be answered by the testing program administrator.

It is important with any statistical investigation to choose statistics that are well-suited and designed for the task at hand. For example, if the concern is that answer sheets are being modified, then erasure counts should be analyzed. Having analyzed over one hundred data sets for a wide variety of clients including state Departments of Education, admissions tests, certification programs, and licensure exams, I can unequivocally state that answer copying is the predominant means of cheating on tests. Therefore, it is especially relevant in this discussion concerning the reliability of statistical evidence to discuss answer copying and statistics that are designed to detect answer copying.

As you reflect upon the principles that I have outlined, I would ask you to consider the data in Table 1. The table contains differing probability values that a testing program administrator might be asked to evaluate. These are sampled answer-copying statistics (i.e., counts of identical answers) from a test having 240 items. With this many items on the test, the central limit theorem will generally apply so I have included a Z-Score in the table, as a point of reference.

Table 1: Sampling of test similarity statistics

At Caveon we deal with extremely small probability values, so we typically express those using “an index” where the probability is one in 10 to the power of the index (p=10^-index). The most extreme case in Table 1 has a probability of one in 10 to the thirtieth power. These data are definitely not due to normal test taking.

Assuming that you accept the statistical evidence as being reliable, the decision needed by you, the testing program administrator, is how low in Table 1 should you go? Where do you set the cut point? These data illustrate if you set the cut point too low, you might accuse some individuals of answer copying without having strong evidence. If you set the cut point too high, you might allow several individuals who have cheated to escape discipline.

I will elaborate more on this topic, next time. Until then, may your tests remain secure.

Kathleen Vohs of the University of Minnesota and Jonathan Schooler of the University of British Columbia have created a series of experiments involving cheating on tests as an attempt to associate behavior (specifically cheating behavior) with moral beliefs and free will. In essence, they ask the question, ” ‘If people came to believe that their behavior was the inevitable product of a causal chain beyond their control — a predetermined fate beyond the reach of free will’ how would their behavior change?”

http://www.psychologicalscience.org/media/releases/2008/vohs.cfm

http://www.sciencedaily.com/releases/2008/01/080129125354.htm

This is a fascinating subject and idea. I have personally felt that an individual’s code of ethics is a stronger deterrent against cheating than anything else. Don McCabe from the Center for Academic Integrity has shown through research the positive effect of honor codes (http://www.academicintegrity.org/educational_resources/honor_code_101.php) in maintaining academic integrity. From the Josephson Institute’s 2006 Report Card on the Ethics of American Youth, we read, “Widespread and deep youth cynicism often reflects itself in a rationalization process that nullifies ethical judgment and condones conduct that is contrary to stated moral convictions. Thus, the same youngsters that speak of the importance of ethics, character and trust, frequently lie, cheat and even steal without much guilt or hesitation.” http://www.josephsoninstitute.org/pdf/ReportCard_press-release_2006-1013.pdf

Although eighty-three percent of high school students believe “It’s not worth it to lie or cheat because it hurts your character” (please note that 17% feel that lying and cheating do not hurt your character), fifty-nine percent believe “In the real world, successful people do what they have to do to win, even if others consider it cheating.” In other words we can visualize these young people saying, “It may hurt me to cheat but I must cheat if I want to be successful.” The way the question is worded, definitely affects the response rate. For example, only forty-three percent agree that “A person has to lie or cheat sometimes in order to succeed.” Relevant dishonest behaviors from the Josephson study are shown below:

On September 19, 2006, The Chronicle of Higher Education stated, “More than half of the graduate business students surveyed recently admitted to cheating at least once during the last academic year.” (Survey Finds Widespread Cheating in M.B.A. Programs).

More recently, Caveon’s “Cheating In the News” referenced a story where Marianne Jennings of Arizona State University recounted recent and notable ethics lapses.

http://www.caveon.com/citn/?p=415

After thinking about these issues, I have the following personal observations:

• Honor codes and codes of ethics are critical to maintaining an ethical testing environment. Test taker agreements (e.g., non-disclosure agreements) are essential for all testing programs and should contain honesty covenants.
• Not all testing programs are in a position to require a code of ethics, but if you can do it as a part of your association or organization, you should.
• Even without a code of ethics, you should emphasize the importance of ethical test taking. This should be a fundamental message that is always championed.
• If you believe the adage that “the pen is mightier than the sword,” then you will direct significant resources towards ethics education, while at the same time making an example of miscreants.

In my opinion, the challenge to educate and inculcate ethics into test-taking behavior must begin with an understanding of cultural influences, moral beliefs, and perceived acceptability of such behavior. For example, our data analyses indicate that “cheating for the common good” on tests may be condoned in some Asian cultures. It seems reasonable to conclude that high-profile role models who have been caught cheating will provide others with an excuse or rationalization to “cheat.”

Here are a few other recent articles that are relevant to this topic:

Ethical Conduct continues to Spiral out of control

F for Conduct

Cheaters never win – Unless Penn lets them

Federal investigators are investigating falsified weapons certifications from 2001 where security guards who were supposedly trained to protect federal buildings have been failing weapons tests. Evidently, the contractor who provided the training was also responsible for certifying that the guards could use the weapons. The contractor knowingly did not train, did not test, but did falsify documents attesting to both.

http://www.zwire.com/site/news.cfm?newsid=19011441&BRD=1641&PAG=461&dept_id=594835&rfi=6

The accused has pled guilty to mail fraud in order to avoid more serious criminal charges. However, the breach was the result of a built-in security flaw. It’s easy when the trainer is also the certifier to falsify data. The lie may not be detected for a long time (in this case we are talking about 6 years). The story illustrates that checks and balances are needed in all security settings. Independent reviews and evaluations are good and should be required.

Another huge cheating incident illustrating the same security weakness was broken by KCNC’s Brian Maass. Servisair has the contract for training workers to de-ice airplanes at Denver and eleven other airports. After receiving a tip, a producer at KCNC went undercover and gathered video evidence of a trainer feeding answers of test questions to trainees.

http://www.rockymountainnews.com/news/2007/nov/24/de-icing-meltdown-needs-fast-resolution/

http://www.poynter.org/content/content_view.asp?id=133478

I have knowledge of other situations similar to these, where trainers have given answers to examinees and teachers have tampered with answer sheets. The question becomes, “Who will guard the guardians?” When the stakes are high for both the test taker and the program administrator, test security may be easily breached by program personnel unless opportunities to gain an unfair advantage are minimized and even eliminated.

It is well known that individuals in positions of trust may easily abuse that trust and do a lot more damage to a testing program than a single test taker who cheats. For some reason, these stories keep showing up. We don’t like to believe that people we trust may actually be engaged in test fraud or covering up a few peccadillos (from Spanish meaning ‘little sins’).