Caveon Security Insights

By: Steve Addicott, Caveon Vice President

October is an important month for Caveon. Eight years ago in October, 2003, several assessment industry veterans formed a small consulting company focused solely on improving the security of our clients’ test programs.    That company is Caveon Test Security!

Fast forward to 2011, and it’s gratifying to consider what this entrepreneurial group of test security zealots has accomplished.  Since that fateful October day, we have

• conducted over 50 Security Audits of leading test organizations and vendors,
• flagged and removed tens of thousands of internet-based risks, and
• conducted statistical analyses of over 30,000,000 test instances for many of the largest, most important test programs in the world.

As I consider the number and breadth of these engagements, perhaps it is worth sharing a few of the core values under which we always operate:

Confidentiality

Throughout our years of operation, one fundamental operating principle has always applied:  client confidentiality.  We never reveal the details of our client engagements without the express approval of our clients. Our clients require and appreciate this sensitivity as we investigate security incidents and provide reports on our forensic analyses. This is not secrecy– this privacy stems from respect for our clients and for the right to privacy of individuals and organizations.

Innovation

We constantly strive to improve means and methods for strengthening exam security. We are always interested in sharing the nature of our work.  Not only do we share our methods and science with clients, client stakeholders, TAC members, educational measurement researchers, and other appropriately interested parties; we are committed to furthering the science around test security. We regularly present at conferences and webinars where we openly share our Caveon approach, theories and methodologies. In fact this last year, we have presented at conferences in Phoenix, Orlando, Chicago, Seattle, Washington DC, Hong Kong, and Prague.

Conservative Recommendations

When we conduct an engagement, our approach is to focus on the situations and incidents that are most egregious, as evidenced in the data and the results that we analyze. We highlight those problems that are most readily identified, documented, and ideally, resolved. Dealing with these problems effectively will have the greatest positive impact to the overall validity and security of test results. This reasonable approach helps our clients, most of which suffer from ever-constrained budgets and resources, effectively concentrate their time, resources, and dollars where the likelihood of inappropriate test taking is highest.

Lastly, our growth and success is directly attributable to a few overarching principles—We always strive to exceed our clients’ expectations, comport ourselves honorably, provide valuable services, and share, as openly and honestly as we can, recommendations for improving the fairness and validity of our clients’ test programs. These principles result in proven, practical protection for our clients, and we intend to follow them for another eight years!

Please Submit Your Comments Below. Thank you!

By: David Foster, CEO, Caveon Test Security

Item exposure during an exam in the testing world is often viewed as a bad thing, because it seems obvious that item exposure leads to item over-use which in turn leads to item compromise. It is common for psychometricians to limit item exposure, defining it as either a too-high absolute number of presentations of the items in a test, or a too-high rate of the items presented on tests. Unfortunately, there is no scientific research or even unscientific guidelines, or even reasonable casual suggestions, about how many exposures are too many, or which rate of exposure is too high.

It does not follow that item exposure is the same as item compromise. In fact, I’ve seen items compromised with an extremely small number of presentations. Some items have even been compromised prior to the first test being administered!

In my opinion, the notion that item compromise results from item exposure—as defined above—leads  to improper conclusions, decisions, and ineffective procedures. I have a few reasons for this opinion, a couple of which I’ll give here. First, item exposure is absolutely necessary. It is obvious that no test can be effective unless its items are exposed during the exam. Test designers even let examinees view an item multiple times encouraging them to return to and review previous items again and again. Second, item compromise has very little to do with the definitions of item exposure given above. Consider this simple example: Suppose that an item was shown to one million test takers and was presented on every exam administered. This would be considered a very high number of exposures along with a 100% exposure rate. But, suppose that none of those examinees were able to share the item with others. In this simple example, the item remains uncompromised and perfectly secure, and can be continued to be used on the exam.

If we wish to reduce item compromise, the example illustrates that limiting the number of presentations or rate of presentations of an item is not as important as the methods used to secure the items, to protect them from theft, and to keep them from being used for cheating. For this reason we need improved item security, which means better ways to keep items from being stolen and used for cheating on subsequent exams. We need methods to detect when an item is truly compromised and then immediately to take it out of service. Instead, we often see stubborn adherence to a century-old model of relatively unsecure test administration, and believing that keeping an item from being presented on a test is a sensible way to secure it.

It is certainly possible to improve the way we secure items. As examples, there are protective item and test designs available, and certainly better test monitoring procedures, that we can use. And perhaps we can learn a little from other industries as well. Consider the problem with the theft of music over the Internet. No one would suggest that music is stolen because it was listened to by too many people. Instead, we see serious efforts to protect the music, to keep it from being stolen, to detect when it is stolen, and to punish those that are responsible. We should be doing the same.

We welcome comments below!

By: Dennis Maynes, Chief Scientist, Caveon Test Security

(The following is an excerpt from an invited talk that was presented to the US Department of Education, September 1, 2011.)

It was sometime after we started Caveon, that I realized the primary goal of conducting security analyses was the strengthening of exam security, not catching cheaters. This is a message that resonates very well with the testing program managers with whom I have interacted. They agree that the primary goal of security actions should be to obtain trustworthy test results, which occurs when the exams are administered securely and with integrity. Disciplining cheaters is important and supports this goal, but it is only a means to an end.

Exam security can be strengthened in two ways, and both should be used: (1) Prevention of cheating, and (2) Detection and discipline of cheaters which will result in deterrence.

Prevention of cheating is gained by implementing effective security processes through policies and procedures. An important element of this effort is the periodic review of security processes and how well they have been implemented.

Detection and discipline of cheaters occurs through (1) performing regular forensic analysis, (2) qualifying the anomalies, and (3) imposing sanctions and invalidating scores.

Deterrence results when security actions and consequences for cheating are publicized.

It’s important to realize that security is a process, not a state. As an example, I have an alarm system at home. Installation of an alarm system does not mean that my home is secure. Only by arming and testing the alarm system can I be ensured that it is functioning properly. Speaking of alarm systems, I am delighted when no one breaks into my home. Just because there were no break-ins, does not lessen the value of the alarm system. I have had clients who felt that web patrolling and data forensics monitoring had no value because we did not detect security breaches. The non-existence of security breaches does not lessen the value of the security processes that have been implemented.

Except for some fraud laws, there are very few laws regulating cheating. It is difficult to prove and there is no physical evidence of material loss or harm. I often hear the phrase “Prove that I cheated.” In fact, I recently saw a headline in the papers expressing the same idea. It’s important to realize that state departments of education do not need absolute proof of cheating. They have an obligation to ensure that tests are administered securely and with integrity. In order to meet this obligation, states require a “preponderance of evidence” in order to act, not absolute proof. However, the departments of education must treat students and teachers fairly, and they must communicate policies clearly.

Because security is a process, it is important to have a ready-prepared security breach response plan, before the breach occurs. It’s not a matter of if the plan will be activated; it’s only a matter of when the plan will be activated. The planning process helps the department of education to have a focused and coordinated response for conducting investigations, imposing discipline and, of utmost importance, communicating with the public and the media.

Without such a plan, the department of education must create a response to the security breach in a potentially haphazard manner. The press is very good at uncovering haphazard and hastily prepared communications.

In summary, state departments of education are empowered to use data forensics wisely and effectively when they have implemented security policies, processes, and procedures which enable them to administer tests securely and with integrity. Regular data forensics monitoring allows states to measure and manage security risks that are inherent with all forms of high-stakes testing.

By: David Foster, CEO, Caveon Test Security

In 2010, a very useful book was published by The Council of Chief State School Officers and the Association of Test Publishers. It is titled Operational Best Practices for Statewide Large-Scale Assessment Programs. Caveon’s very own Dr. John Fremer contributed as part of the working committee to the overall effort and provided a chapter or two. As the title suggests, the book provides some “best practices” in a good many areas of interest to all testing professionals, particularly those involved in paper-and-pencil state assessments. A testing program can use the book to evaluate its own practices, and to guide efforts at change if necessary.

Given the intense interest today in delivering tests on the computer, it’s not a surprise that there was immediate interest in a revision of the book, one that would include best practices for programs using or wishing to implement technology-based tests. These are tests that are administered on computers via local servers, or delivered online through secure browsers. Choosing the specific technology used to administer the tests is not an easy chore and should be carefully done. The newest model, online testing—testing administered securely through browsers—is becoming more and more popular with high-stakes testing programs.

But what are we to think about the concept of best practices when a methodology is new and developing, when few organizations are experienced with it? How can a best practice even be identified with so little applied experience and when change accompanies that technology almost daily. It’s my opinion that our concept of what is a best practice has to evolve if we are to find it useful in the face of new and constantly changing technology.

To solve this conundrum I’d like to propose that we adopt a more accepting approach toward innovation and technology. This means that we should seriously consider innovations even though dozens or hundreds of other programs have not yet tried it out. This optimistic attitude is critical if we are to find these innovations immediately helpful, and, more importantly, if we are to set ourselves on a path to accommodate change occurring on a more constant basis. New technologies can be evaluated against reasonable criteria that reveal how the innovation will improve the reliability, validity, security and fairness of the tests. This is especially easy to do if by implementing the technology we are solving a long-standing concern or problem.  My own experience developing and using new technologies over the past 30 years has been very rewarding.

Just a word about standards and technology. Some feel that using new technology violates or threatens standards. That certainly hasn’t been my experience. Throughout my career, as I used new technologies in testing, I have found that in each case it enhanced my ability to meet the standards, rather than threaten them. An example may help here. In 1990 at Novell we implemented a new multiple choice question type that allowed for more than one correct answer. No one had used it before. It immediately helped us to eliminate confusion for our test takers from negatively worded multiple choice questions. There is no standard that states that multiple choice questions must only have a single correct answer, but there are standards that require us to improve the quality of our questions.

Now, a final word about statewide educational testing. The joint committee working on the revision of the Operational  Best Practices for Statewide Large-Scale Testing will provide a set of best practices in the coming months for technology-based tests. Hopefully these suggestions will be met with enthusiasm and optimism. If they are, statewide assessment programs will find it much easier to meet the very ambitious goals set by themselves, the federal government, and other stakeholders.

By: Dennis Maynes, Chief Scientist, Caveon Test Security

This summer, many cheating investigations of schools based on forensics evidence have been inconclusive or weak. This is troubling because investigations require time and money, and they can be disruptive. In my experience, the statistics are often questioned and with good cause.

A famous statistician once said, “If you torture the data long enough, they will confess.”  (Attributed to Ronald Coase: http://en.wikipedia.org/wiki/Ronald_Coase). Forensics monitoring requires inspection of all the data and listing the extreme observations. Because this is done with the intent of FINDING anomalies, extreme care is required. Knowledge of order statistics teaches how to analyze the data properly. Unfortunately, order statistics are not taught in basic statistics courses, which means that many analysts not having knowledge of order statistics tend to torture the data. It’s like saying my basketball team is taller than your basketball team using only the height of the tallest player on each team. Such a comparison ignores the starting five and the height of all the players on the team.

Let me illustrate the concern with an example. Journalists routinely analyze the score gains of schools within the state using regressions. Because students of basic statistics courses are taught that a regression residual exceeding three standard deviations IS an outlier, the newspaper will report all schools having gains of three standard deviations or more. The named schools will receive “extra attention” in the press and by the public. Such attention is not warranted because the fact that ALL schools were examined has been ignored. The three standard deviation rule is correct only if you were to examine one school and one school only, at random. But, the analysis was not restricted to one school, which means that many schools were inappropriately named. The statistics guarantee it.

Journalists are not the only group to make this error. Most forensics reports that I have read have ignored this issue. Using order statistics, statisticians have developed corrections to avoid overstating anomalies. For example, Bonferroni’s correction divides the target probability level by the sample size (http://en.wikipedia.org/wiki/Bonferroni_correction). Using this correction, 4.2 standard deviations, not 3, is the correct threshold to use if 1,000 schools were inspected.

The next time that you see a forensics report in the media, ask yourself, “Was the Bonferroni correction or similar conservative threshold used?” If the answer is no, be very, very careful because the number of anomalies has probably been overstated. On the other hand, if the answer is yes, the investigations are probably warranted and justified.

By Steve Addicott, Vice President of Sales, Caveon Test Security

Over the past few weeks, I have participated in several planning sessions with Caveon clients who contract with us to analyze test results.  This service, Caveon Data Forensics ™, represents a proactive means to better protect their programs by identifying statistical anomalies that may indicate cheating.

While each of these programs is different (state education, IT certification, medical licensure, construction certification, etc.), it’s interesting to me that they face common challenges in confronting test fraud.  For each of these programs, test results matter…they really matter…in making important decisions in the lives of test takers (and in education, teachers and principals).  Thus, the integrity of the test administration matters greatly, too.

My overarching impression?   Tackling test fraud head-on is not for the faint of heart.  It takes commitment—a genuine, unwavering commitment to fair and valid test results—to say “We know there is a subset of our test-taking population that is taking shortcuts, and we’re going to do something about it.”

These days, everyone is busy.   So, why would any sane test program leader willingly add to his/her workload?  The results of our Data Forensics analyses do just that.  We identify:

• candidates/students that may require invalidations;
• test centers/schools that merit investigations; and
• items/exams that should be retired and/or revised.

The committed leaders we work with understand that this is what is required to be able to stand in front of stakeholders and proclaim that their test administrations are fair and valid.

Another important takeaway I’ve gained is that a successful Data Forensics program requires the cooperation and coordination of many  groups.  Last week, I met with a client, a large state department of education.  Its leadership, in order to ensure the data forensics program possessed real teeth, sought cooperation with several other state departments:

• Legal, to ensure any sanctions resulting from the data forensics would hold up in court;
• Communications, to ensure sound, consistent messaging to the media and public alike;
• Inspector General, for conducting investigations; and
• Professional Practices, in case sanctions might be brought against a state certified educator.

This sort of cross-organizational coordination is not easy to facilitate, but critically important in the fight for fair and valid testing.

If you’re considering how you can augment the security of your program, you might find one of our company webinars to be a help.   In “Don’t Shoot The Messenger”, three Caveon clients present the good, the bad, and the challenging in instituting program invalidations through data forensic analyses.  You can get a copy of the webinar slides here:  http://caveon.com/df_blog/ .

Hindsight: Perfect understanding of an event after it has happened; – a term usually used with sarcasm in response to criticism of one’s decision, implying that the critic is unfairly judging the wisdom of the decision in light of information that was not available when the decision was made.

After every single airplane crash or incident, the FAA routinely conducts exhaustive investigations to determine the cause of the crash. The purpose of the investigation is “to identify safety deficiencies and unsafe conditions which are then referred to the responsible FAA office for evaluation and corrective action.” The amazing air safety statistics in this country are primarily the result of these extensive analyses. Setting all sarcasm aside, the FAA has learned that hindsight is 20-20. A perfect understanding of the event is often attainable. And from that understanding, air safety has improved.

I believe that all testing programs can learn from this example. If each program conducts a “security breach post mortem” security processes can be improved. A good practice in security is learning from your own mistakes. A better practice is learning from the mistakes of others. A best practice is creating processes so that those mistakes are never repeated.

As an example of what might be possible with a security breach post mortem, consider two recent news stories. Recent news from the UK suggests that many immigrants are being coached to pass the spoken language and listening portions of the citizenship tests, even though they cannot speak English. The BBC went undercover and filmed “an appraisal” which the undercover reporter understood to be the process for passing the language test. The reporter didn’t even need to speak or listen in English. The video is extremely fascinating. In other news, the results of Boston’s promotion exams for firefighters are being discarded and all the candidates will be required to retest, following a security breach in November 2007 when cell phones were used to cheat. The retesting is required because the investigation was inconclusive and the cheaters were not uncovered.

It is likely that both of the above breaches would have been prevented if proper security safeguards were in place. The purpose of the post mortem is to learn the security strengths and weaknesses of the testing program, so that security may be improved and strengthened. In my experience, we generally do not obtain all the information possible from a security breach investigation. For example, in Boston the investigation was conducted to determine who cheated. While some improvements to security should happen as a result of the investigation, I believe that a serious post mortem would reveal even more information in order to prevent similar breaches in the future. The post mortem allows us to learn from our mistakes.

In an earlier essay, I suggested that testing programs should, “Read stories of cheating in the news to learn how the media might portray your cheating incident negatively.” This is one form of learning from the mistakes of others. In addition to studying security breaches in the media, several other methods exist for learning best security practices and processes from others. Some of these are (1) attending presentations where security breaches are discussed, (2) talking directly with program personnel who have been involved in security breaches, and (3) working with experts who study and analyze security breaches and best security practices. At Caveon, we are doing our best to expand our expertise so that we may effectively assist all testing programs in their efforts to strengthen their test security.

If you have never conducted a security breach post mortem you are probably wondering how you might start.

The first step determines the extent and nature of the security breach. When the breach involves cheating during the test or tampering with the test results, a data forensics analysis is invaluable in making this assessment. When the breach involves the distribution and sale of protected test content, an Internet investigation or Caveon Web Patrol can determine the scope and size of the breach. When the breach involves a breakdown of security procedures and processes, a post-mortem security audit will be needed. Some security breaches may require all three information-gathering activities.

The second step performs a cause-and-effect flow analysis or a fault tree analysis. This analysis establishes where the test security vulnerabilities exist and how those vulnerabilities were exploited by the miscreants.

The third step identifies necessary changes in the testing program’s security processes. These changes should be first considered as suggestions or recommendations. They should be prioritized. They should be assessed for effectiveness using security threat models. They should be evaluated against required resource allocations so that their practicality can be measured in terms of the program’s budget and expertise.

Finally, proposed recommendations are presented to the executive management team with an implementation roadmap. The executive report should clearly state that the purpose of the post mortem is to improve and strengthen test security. A post mortem analysis is not conducted with the purpose of apprehending cheaters and imposing discipline upon test frauds. These actions may result from the investigations. But, the post mortem provides the tactical and strategic initiatives to prevent test fraud in the future.

Caveon is willing and able to assist you in these efforts. We wish you the best as you consider how to learn from your own mistakes and the mistakes of others.

Wise men profit more from fools than fools from wise men; for the wise men shun the mistakes of fools, but fools do not imitate the successes of the wise. – Cato the Elder

Hindsight is indeed 20-20 and is not to be scoffed at when we use it in order to improve.

The ATP (Association of Test Publishers) conference this year did everything a good conference should do. We networked. We shared industry information. We discussed best practices. We met with clients and vendors. And we created, renewed, and strengthened friendships. Rather than discuss those things, let me share a few observations relating to test security.

Exam security was a hot topic, with many sessions and many serious conversations around test security. Wayne Camara of the College Board asked me, “Was the emphasis on security due to Caveon?” I replied, “I think it is partly due to our outreach effort, and more programs are dealing with security issues.” I think there are deeper reasons.

There were more stories describing successful security efforts this year than I remember in the past. Just to name a few: the FSBPT discussed their breach and resolution in the Philippines, the GMAC caught a proxy test taker in the very act, EMC presented successful risk management cases, and the Mississippi Department of Education has effectively addressed cheating in schools. We celebrate these successes, because they give us confidence that these problems can be solved.

There is deep concern about test and exam piracy. In the past, this concern was primarily expressed by IT (Information Technology) companies. This year many other organizations had the same concern. I heard several instances of exams being stolen from within computer-based testing centers. I have no reason to doubt these reports.

Theft vulnerabilities had been voiced privately in the past, but the discussions were more open this year. I attribute this to at least three reasons: (1) there were new attendees who wanted to expressly discuss security and stayed for the Test Security Summit, (2) the Boston Globe article “Job Exam Piracy Rising,” dated December 26, 2007, gave the topic national prominence, and (3) some presenters disclosed that their entire item banks, including answer keys and digital representations, had been stolen. In the session, “Cheater, Cheater, Pumpkin Eater,” EMC Corporation reported great success in detecting and shutting down test sites where exams are being stolen. Test pirates refused to resell test content because their test sites were shut down immediately after they stole the tests.

To the best of my recollection, there were more lawyers present at ATP this year than any other year. Representatives from at least four different firms had been invited to attend by conference organizers or conference presenters. I have paraphrased some of their very instructive comments below:

“Gather all your evidence in preparation to litigate, but only litigate as a last resort.”

“You can use statistics to invalidate scores and to take other security actions if you can demonstrate that your actions and decisions are made in good faith. The courts are interpreting these actions using contract law and it’s important that your agreements and contracts support your intended actions.”

“All test items are copyrighted, but you must register the copyrights before the items are stolen. Registered copyrights provide stronger protection than unregistered copyrights. There is a special provision in copyright law to protect secure tests for this purpose.”

GMAC and Pearson VUE described initiatives for preventing and detecting imposters. GMAC verifies a candidate’s current photo with the candidate’s registration photo. They attach the photo to the score report. (I call this “testing event authentication.”) Pearson VUE demonstrated Fujitsu’s PalmSecure biometric authentication technology. The readers are priced at around $700, but within reach for secure testing applications.

Gene Radwin and Liz Burns of EMC Corporation captured our imagination. Gene shared his success in detecting users of braindump content using Trojan items. Liz Burns described her security efforts. She visualizes a triangle. At the base of the triangle are honest people who will not lie and will not cheat. At the top of the triangle are those who will cheat if at all possible. In the middle of the triangle are individuals who may cheat depending upon the circumstances. The “at risk group” is where Liz concentrates her efforts.

The Education Division meeting had an interesting discussion concerning the image of testing in education. I think that a positive image of testing is critical. As an example of how incorrect image of testing can be damaging, consider the report that South Africa has effectively banned unproctored Internet testing, because these tests are thought to be unfair, not being secure (reported by Hennie Kriek, President of SHL, USA).

Finally, if you believe that test publishers are cold and dispassionate, let me disabuse this image. I saw a lot of passion and emotion at this conference. Testing professionals are very concerned that tests are administered securely. As an example, Cindy Simmons, State Assessment Director of Mississippi, showed great forthrightness and passion as she described her state’s initiatives to address cheating on the Subject Area Tests.

It’s true there is much work to do. But members of ATP are committed to fairness and integrity in testing. They comprise “the intelligent voice of testing.“

The Association of Test Publishers (ATP) Conference of 2008 ended yesterday. As always, it was a good conference. In 2004 we stated, “You can’t manage what you don’t measure.” Being a sponsor of the conference, we placed a bag of M&M’s (i.e., manage and measure) in each attendee’s conference packet. And, we printed the message on the hotel room key cards.

I have just completed analyses for three testing programs and I am so impressed with what they have done that I want to share their results with you. Good news concerning exam security is refreshing in the midst of so many cheating stories. We recognize dramatic acts of heroism, but often ignore the good that happens with steady, persistent progress. I am so proud of these three programs. They are achieving their common goals: “Reduce cheating, strengthen exam security and emphasize ethical test taking.” The data demonstrate this convincingly. Caveon’s message at ATP this year was, “The answer is in the data.” So let’s look at the data.

Figure 1: Percent of anomalous tests for three programs

Let me describe the data in Figure 1. The percent of anomalous tests for successive analyses are plotted for each program. A trend line has been fit to the data to aid your eye in visualizing the trend pattern. An anomalous test is one that deviates from normal test taking, and will exhibit at least one of the following: aberrance (answering hard questions correctly and missing easy questions), large numbers of erasures, inexplicable score changes from a previous test score, or excessive similarity in the selected answers with at least one other test. An anomalous test does not mean the test taker cheated. For example, when we observe excessively similar tests it is very likely that one person cheated (the copier) and the other person did not (the source). The percent of anomalous tests does not measure the precise number of people who have cheated, but it is highly correlated with that number.

These data are important because they demonstrate that all high-stakes testing programs, irrespective of industry or application, can effectively reduce cheating. They illustrate that reductions in cheating can occur with persistence and dedication. Let me briefly describe each program and some of the positive steps they have taken.

Program 1: This program provides a professional certification with high security. We estimate that there was a 45% reduction in cheating in three years. They have followed up on every case that appeared to be a security violation and every test site that appeared to have lax security. They have emphasized proctor training. They are now reviewing their test taker agreements, proctor training, identification procedures, and physical security with the intent of using the best known security protocols.

Program 2: This program is a public education program. We estimate that there was a 72% reduction in cheating in two years. They have rewritten their test administration manuals and have begun test administration monitoring. They assign a conditional status to extremely anomalous test results and require local review of those test results. They are receiving reports that the students being flagged are admitting to having cheated.

Program 3: This program administers tests in the service industry. We estimate that there was a 78% reduction in cheating in one year. They have stressed ethical test taking. They have revised their test taking agreements and strengthened test administration policies to allow for scores to be invalidated with an appeals process. They have refreshed test forms which appeared to be exposed. They are researching the next phase of security improvements: test site monitoring and appropriate disciplinary measures for test administration personnel who may be helping test takers inappropriately.

These very different programs were the same in one important way: They started where they were, they created a plan, and they were not discouraged. Each was taken back by the first data forensics report (we always find something disconcerting), but they pressed forward and executed their plan. Best practices used by these programs include: test site monitoring, emphasis on ethical test taking, invalidating scores as per policy, refreshing tests which appear to be over exposed, and updating their security procedures.

Let’s give credit where credit is due. The numbers are impressive and the data do not lie. These programs have earned our respect and admiration.