Hindsight: Perfect understanding of an event after it has happened; – a term usually used with sarcasm in response to criticism of one’s decision, implying that the critic is unfairly judging the wisdom of the decision in light of information that was not available when the decision was made.
After every single airplane crash or incident, the FAA routinely conducts exhaustive investigations to determine the cause of the crash. The purpose of the investigation is “to identify safety deficiencies and unsafe conditions which are then referred to the responsible FAA office for evaluation and corrective action.” The amazing air safety statistics in this country are primarily the result of these extensive analyses. Setting all sarcasm aside, the FAA has learned that hindsight is 20-20. A perfect understanding of the event is often attainable. And from that understanding, air safety has improved.
I believe that all testing programs can learn from this example. If each program conducts a “security breach post mortem” security processes can be improved. A good practice in security is learning from your own mistakes. A better practice is learning from the mistakes of others. A best practice is creating processes so that those mistakes are never repeated.
As an example of what might be possible with a security breach post mortem, consider two recent news stories. Recent news from the UK suggests that many immigrants are being coached to pass the spoken language and listening portions of the citizenship tests, even though they cannot speak English. The BBC went undercover and filmed “an appraisal” which the undercover reporter understood to be the process for passing the language test. The reporter didn’t even need to speak or listen in English. The video is extremely fascinating. In other news, the results of Boston’s promotion exams for firefighters are being discarded and all the candidates will be required to retest, following a security breach in November 2007 when cell phones were used to cheat. The retesting is required because the investigation was inconclusive and the cheaters were not uncovered.
It is likely that both of the above breaches would have been prevented if proper security safeguards were in place. The purpose of the post mortem is to learn the security strengths and weaknesses of the testing program, so that security may be improved and strengthened. In my experience, we generally do not obtain all the information possible from a security breach investigation. For example, in Boston the investigation was conducted to determine who cheated. While some improvements to security should happen as a result of the investigation, I believe that a serious post mortem would reveal even more information in order to prevent similar breaches in the future. The post mortem allows us to learn from our mistakes.
In an earlier essay, I suggested that testing programs should, “Read stories of cheating in the news to learn how the media might portray your cheating incident negatively.” This is one form of learning from the mistakes of others. In addition to studying security breaches in the media, several other methods exist for learning best security practices and processes from others. Some of these are (1) attending presentations where security breaches are discussed, (2) talking directly with program personnel who have been involved in security breaches, and (3) working with experts who study and analyze security breaches and best security practices. At Caveon, we are doing our best to expand our expertise so that we may effectively assist all testing programs in their efforts to strengthen their test security.
If you have never conducted a security breach post mortem you are probably wondering how you might start.
The first step determines the extent and nature of the security breach. When the breach involves cheating during the test or tampering with the test results, a data forensics analysis is invaluable in making this assessment. When the breach involves the distribution and sale of protected test content, an Internet investigation or Caveon Web Patrol can determine the scope and size of the breach. When the breach involves a breakdown of security procedures and processes, a post-mortem security audit will be needed. Some security breaches may require all three information-gathering activities.
The second step performs a cause-and-effect flow analysis or a fault tree analysis. This analysis establishes where the test security vulnerabilities exist and how those vulnerabilities were exploited by the miscreants.
The third step identifies necessary changes in the testing program’s security processes. These changes should be first considered as suggestions or recommendations. They should be prioritized. They should be assessed for effectiveness using security threat models. They should be evaluated against required resource allocations so that their practicality can be measured in terms of the program’s budget and expertise.
Finally, proposed recommendations are presented to the executive management team with an implementation roadmap. The executive report should clearly state that the purpose of the post mortem is to improve and strengthen test security. A post mortem analysis is not conducted with the purpose of apprehending cheaters and imposing discipline upon test frauds. These actions may result from the investigations. But, the post mortem provides the tactical and strategic initiatives to prevent test fraud in the future.
Caveon is willing and able to assist you in these efforts. We wish you the best as you consider how to learn from your own mistakes and the mistakes of others.
Hindsight is indeed 20-20 and is not to be scoffed at when we use it in order to improve.