CONTENTS
Introduction to Integrity
Demystifying the Science
The Language of Data Forensics
How Does it Work?
Data Forensics & Certification
Lurking Test Security Threats
The Costs of Poor Security
What to do with Your Data Forensics Report
Talking to Your Organization about Data Forensics
Too many tabs open?
We’ve been there. Enter your email to get a permanent copy of this guide sent directly to your inbox.
Let's Talk About Integrity
So you’re curious about data forensics (DF)? Good. If you’re already thinking about the health of your assessment, especially when it comes to security, then you’re on the right track! Give yourself a pat on the back.
But what is data forensics? How does it work? What security threats and vulnerabilities actually apply to your assessment? Consider this your ultimate guide, which will walk you through the basics of statistical analyses and explore security vulnerabilities that may be hidden within your certification program. Keep reading to learn about the hidden costs of poor exam security, what you should do after you conduct data forensics, and (importantly) how to help your organization adopt this program-strengthening practice.
Here’s to integrity!
Demystifying the Science Behind Data Forensics
What are some classic psychometric statistics that data forensics experts use to pinpoint patterns and other clues in testing data?
1. Similarity
Examine how closely an individual’s responses match other individuals’, and calculate the probability of the matches.
2. Answer Changes
Tabulate how many answers were changed on a test, and flag high rates of wrong-to-right answer changes.
3. Score Differences
Use information about items to examine differences in performance, which can reveal inconsistent test-taking behavior.
4. Timing
Examine unusual response time patterns to monitor for abnormal test-taking behavior, like super-human speed.
5. Aberrance
Scan for unusual patterns in the data. These statistics are most useful in supporting other statistics.
6. Program-Specific Indicators
A range of statistics that can be tailored to flag unusual testing behaviors based on a program’s specific needs.
The Language of Data Forensics
Statistical Anomalies
Unusual patterns in the data that may or may not reflect violations or fraud.
Example: Examinees demonstrating two different levels of proficiency
Example: Examinees demonstrating two different levels of proficiency
Testing Irregularities
Out-of-the-ordinary events occurring during testing, not always violations.
Example: A fire alarm disrupts test takers during the testing window
Example: A fire alarm disrupts test takers during the testing window
Security Violations
Events that violate testing protocol and could potentially threaten test security.
Example: The seats in the testing room are too close together
Example: The seats in the testing room are too close together
Security Breaches
Exam integrity has been jeopardized, though not necessarily on purpose.
Example: Administrator accidentally discloses exam content to examinees
Example: Administrator accidentally discloses exam content to examinees
Test Fraud
An intentional act that threatens the integrity of an exam.
Example: Administrator intentionally discloses answers to examinees
Example: Administrator intentionally discloses answers to examinees
How Does Data Forensics Work?
The path to exam validity is different for every certification program, but there are several milestones every program meets on the journey to exam integrity.
1. Decide to Conduct Data Forensics
Whether you’ve experience a known incident, or you’ve decided to use data forensics for one of the reasons listed below, this is a critical first step.
2. Provide Data to Data Forensics Vendor
From basic response booklets to online answer-change and response-time data, your vendor will want whatever you can provide to strengthen the results.
3. Receive Results and, if Desired, Recommendations
You should work with your vendor ahead of time to decide what kind of results you want to receive and put together a general security plan for incident response.
4. Investigate Any Possible Security Threats
If your results show areas of potential vulnerability, work with your vendor to collect more evidence and investigate further.
5. Take Action, If Needed
If wrongdoing is uncovered by data forensics and proven in your investigation, then you should take the appropriate actions as outlined in your security plan.
Why Does a Certification Program Need Data Forensics?
Maintain the Value of Your Certification
Engaging in data forensics offers proof to your stakeholders, your industry, and the world that you are taking steps to ensure that your certification is an accurate representation of your candidates’ skills.
Deter Test Fraud Before it Happens
By publicizing your use of data forensics, you can deter test fraud. Both candidates and stakeholders will know that test fraud of any kind is neither tolerated nor easy
to get away with.
Reveal Security Vulnerabilities
Data forensics can detect potential security vulnerabilities before a widespread breach, allowing the issue to be addressed as soon as it appears.
Highlight Instances of Test Fraud
Data forensics can tell you where and when suspect activity occurred, and by whom. It can reveal large-scale, organized fraud and even detect fraud among individuals.
Collaborate with Test Security Experts
Data forensics experts can assess the health of your program, keep you abreast of the latest security innovations, and offer sound advice based on their experience in test security and data forensics.
Plan for the Future of Your Program
Not only can data forensics help you respond to a security breach before it wreaks havoc on your program, but it also helps you make informed decisions to improve your test security in the future.
Security Threats Above and Below the Surface
Above the Surface
Examinee Collusion
Similarity statistics and source-copier statistics can sniff out examinees who peek at others’ exams or who share test content with other examinees.
Test Killers
Similarity statistics and flawed-key analyses detect the use of stolen exam content contained within “Test Killers,” distributed within small cheat rings and dispersed among test takers.
Administrator Collusion
Wrong-to-right answer-change and similarity analyses can help detect educators and exam administrators who commit test fraud.
Below the Surface
Large-Scale Collusion
Data forensics can detect large-scale examinee collusion and support disciplinary actions by testing programs against individuals in cases of test fraud.
Fraudulent Test Sites
Data forensics can detect test sites operated by pirates and fraudsters and can provide the necessary evidence for taking legal action.
Proxy Testers
Data forensics can uncover the illicit activity of proxy test-takers or professionals for hire, who take tests for others, and the candidates who employ them.
The Hidden Costs of Poor Exam Security
Want to take a copy of the guide to-go?
You don’t need a disposable lid, bag, or box. Just download here.
Check out Other Test Security Resources
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Want to subscribe to our newsletter?