Hindsight is 20-20

The National Transportation Safety Board (NTSB) has made air travel safer than any other form of travel. They have accomplished this by taking a “hindsight is 20-20” approach to transportation safety, where they carefully investigate every single accident to determine its cause, and then use that information to influence future decisions such as changes in equipment, policies, procedures, and training. In other words, the NTSB realized long ago that retrospective analysis followed by an appropriate response would save lives. The testing industry can learn from this example. Just like the NTSB has done with air safety, we can improve exam security by thoroughly analyzing situations where test security was violated.

Last week, I saw data showing that the pass rate of a particular exam dropped precipitously after it was republished. Because we had been tracking this exam for some time, we knew that it was compromised. When the pass rate dropped after republishing the exam, we knew the new version was not compromised, yet. We also knew that the existing braindump of the old version was negated. It was useless. In fact, braindumpers on the Internet complained that the “dump” was no longer valid.

Because I have experienced the value of “20-20 hindsight”, I decided to learn more about this situation. So, I devised a simple analysis for estimating how many test takers had downloaded and used the live exam items from the Internet. I wanted to answer the question, “How many test takers were using braindumps?”  The analysis assumes that the pass rates of the old and new versions of the exam should be the same. This assumption is reasonable, as long as the old and new versions of the exam were equally difficult and the competence of the test taking population did not change.

Estimation of braindump usage helps measure significant losses, such as lost retake revenue and the number of unqualified certificate holders. These data can guide policy changes, communications to candidates, and future exam revisions. In other words, triage of security breaches, just like investigations of airplane accidents, can be extremely helpful in making future decisions. Not only do the data help make decisions, they also help understand risks, threats, and vulnerabilities. Only after performing a proper risk assessment a program manager will be empowered to make informed decisions about test security budgets.

In order to understand how impactful these data can be it helps to analyze them. The pass rate of the exam dropped from 75% to 55%. From this, we can estimate that at least 44% of the test takers were using downloaded braindump content, which means that at least 59% of the test takers who passed did so using braindump content. We don’t know how many of the braindumpers would have passed without the use of the downloaded content, but we can safely assume it was not many. Hence, program retake revenues have been deflated, also. These estimates help assess risk (or the potential for loss).

Once a method exists for estimating braindump usage in the past, it becomes possible to conduct the same analysis in a forward-looking manner. Estimation of the number of test takers using braindump content can be carried out week by week. The results of such an analysis can guide the exam’s republication schedule, balancing between republication costs and losses associated with item compromise. This is a perfect example of “measuring so you can manage.”

Hopefully, I have demonstrated that retrospective analysis can be extremely helpful in making future decisions for a testing program.  Often, the phrase “hindsight is 20-20” is used disparagingly. But, I have learned that hindsight guides implementation of essential test security solutions. Triage of test security breaches helps us learn from and avoid mistakes made in the past. Hindsight should not be ignored. Perhaps these words of Winston Churchill emphasize my theme most strongly, “Want of foresight, unwillingness to act when action would be simple and effective, lack of clear thinking, confusion of counsel until the emergency comes, until self-preservation strikes its jarring gong–these are the features which constitute the endless repetition of history.”


Dennis Maynes

Chief Scientist, Caveon Test Security

Leave a Reply