DATA FORENSICS IN TESTING
Your Ultimate Guide to Protecting Your Assessments with Data Forensics
CONTENTS
What Is Data Forensics?
Common Statistics in Data Forensics Analyses
The Language of Data Forensics
How Does Data Forensics in Testing Work?
How to Get Started Using Data Forensics
Why Does My Testing Program Need Data Forensics?
Lurking Test Security Threats Statistics Can Uncover
The Costs of Poor Security
What to Do With Your Data Forensics Report
Talking to Your Organization About Data Forensics in Testing
Too many tabs open?
We’ve been there. Enter your email to get a permanent copy of this guide sent directly to your inbox.
What Is Data Forensics in Exams?
So you’re curious about data forensics? Good. If you’re already thinking about the health of your assessment, especially when it comes to security, then you’re on the right track! Give yourself a pat on the back.
But what is data forensics? How does it work? What security threats and vulnerabilities actually apply to your exam?
Consider this your ultimate guide, which will walk you through the basics of statistical analyses in testing data. We’ll explore the security vulnerabilities that may be hidden within your assessment program and talk about how you can detect and combat them with data forensics!
Keep reading to learn about the hidden costs of poor exam security, what you should do after conducting data forensics, and (importantly) how to help your organization adopt this program-strengthening practice. Here’s to integrity!
Common Statistics in Data Forensics Analyses
What are some classic psychometric statistics that data forensics experts use to pinpoint patterns and other clues in testing data?
Similarity
Examine how closely an individual’s responses match other individuals’, and calculate the probability of the matches.
Answer Changes
Tabulate how many answers were changed on a test, and flag high rates of wrong-to-right answer changes.
Score Differences
Use information about items to examine differences in performance, which can reveal inconsistent test-taking behavior.
Timing
Examine unusual response time patterns to monitor for abnormal test-taking behavior, such as super-human answering speeds.
Aberrance
Scan for unusual patterns in the data. These statistics are most useful in supporting other statistics.
Program-Specific Indicators
A range of statistics that can be tailored to flag unusual testing behaviors based on a program’s specific needs.
The Language of Data Forensics in Assessments
Terms that are commonly used in data forensics include:
Statistical Anomalies
Unusual patterns in the data that may or may not reflect violations or fraud.
Example: Examinees that are demonstrating two different levels of proficiency.
Testing Irregularities
Out-of-the-ordinary events occurring during testing, not always violations.
Example: A fire alarm disrupts examinees during the testing window.
Security Violations
Events that violate testing protocol and could potentially threaten the security of an exam.
Example: The seats in the testing room are too close together.
Security Breaches
Exam integrity has been jeopardized, though not necessarily on purpose.
Example: An administrator accidentally discloses exam content to examinees.
Test Fraud
An intentional act that threatens the integrity of an exam.
Example: An administrator intentionally discloses answers to examinees.
How Does Data Forensics in Testing Work?
1. Identify Your Greatest Test Security Risks
Every testing program is unique. Understanding your risks will help you decide which analyses will be most informative as you manage them. Start by taking these questions into consideration:
- Which test security threats keep you awake at night?
- How much damage results when someone undermines the integrity of your test scores?
- What motivates examinees to violate your security?
2. Find Out What Data You Have Access To
If you already have a data forensics vendor, work with them to determine what types of data you can obtain. These can include test center identifiers, response times, answer-change data, or timestamps that track when each item is viewed. The data types you have access to will determine the types of data forensics statistics that can be run.
3. Determine What Statistics You Can Compute
The statistics you can compute should mirror the primary security concerns you laid out in Step 1. Once you’ve determined the statistics you can use, ask yourself:
- How well do your selected statistics detect and measure the threats and risks you identified?
Next, align the statistics you select with your primary security concerns. This will ensure that you capture critical information and evidence to support your test security initiatives.
4. Figure Out What Layers to Analyze
Will psychometricians analyze individuals, test sites, test forms, or items? Perhaps some combination of these? Often, detection in data forensics is a multi-layer issue. The deeper you dive into the layers of data, the more information you’ll have about what is actually going on. Try to add as many contextual layers as possible to your results.
5. Decide What Type of Information You Want
How should the results be presented, summarized, and formatted so they are most helpful for your team and stakeholders? What resources are available to help you understand the results? Choose a vendor who can interpret and prioritize your results for you, giving you the ability to digest your results and take action quickly, if necessary.
6. Develop Plans for Handling Identified Incidents
How will you respond if you find evidence of cheating or test fraud? Decide what you will do when the data forensics evidence meets your threshold and standard for taking action. If you’re in high-stakes testing, remember to choose a team of psychometricians with experience and success providing expert witness testimonies to data forensics results in court.
How to Get Started Using Data Forensics
The path to exam validity is different for every assessment program, but there are several milestones every program meets on the journey to exam integrity.
1. Decide to Conduct Data Forensics
Whether you’ve experience a known incident or you’ve decided to use data forensics for one of the reasons listed below, deciding to conduct data forensics is a critical first step.
2. Provide Testing Data to Experienced Psychometricians
From basic response booklets to online answer-change and response-time data, your data forensics vendor will want whatever you can provide to strengthen the results.
3. Receive Results and, if Desired, Recommendations
You should work with your vendor to decide what kind of results you want to receive. They’ll help you put together a general security plan for incident response ahead of time.
4. Investigate Any Possible Security Threats
If your results show areas of potential vulnerability, work with your vendor to collect more evidence and investigate further.
5. Take Action, If Needed
If wrongdoing is uncovered by data forensics and proven in your investigation, then you should take the appropriate actions as outlined in your test security plan.
Why Does My Testing Program Need Data Forensics?
To Maintain the Value of Your Exam
Engaging in data forensics offers proof to your stakeholders, your industry, and the world that you are taking steps to ensure that your exam is an accurate representation of your examinees’ skills (learn why that’s important in this article).
To Deter Test Fraud Before It Happens
By publicizing your use of data forensics, you can deter test fraud. Both candidates and stakeholders will know that test fraud of any kind is not tolerated (or easy to get away with).
To Reveal Security Vulnerabilities
Data forensics can detect potential security vulnerabilities before a widespread breach, allowing the issue to be addressed as soon as it appears.
To Highlight Instances of Test Fraud
Data forensics can tell you where and when suspect activity occurred, and by whom. It can reveal large-scale, organized fraud and even detect fraud among individuals.
To Collaborate with Test Security Experts
Data forensics experts can assess the health of your program, keep you abreast of the latest security innovations, and offer sound advice based on their experience in test security and data forensics.
To Plan For the Future of Your Program
Not only can data forensics help you respond to a security breach before it wreaks havoc on your program, but it also helps you make informed decisions to improve your test security in the future.
Security Threats Above and Below the Surface
Examinee Collusion
Similarity statistics and source-copier statistics can sniff out examinees who peek at others’ exams or who share test content with other examinees.
Test Killers
Similarity statistics and flawed-key analyses detect the use of stolen exam content contained within “Test Killers,” distributed within small cheat rings and dispersed among test takers.
Administrator Collusion
Wrong-to-right answer-changes and similarity analyses can help detect educators and exam administrators who commit test fraud.
Large-Scale Collusion
Data forensics can detect large-scale examinee collusion and support disciplinary actions by testing programs against individuals in cases of test fraud.
Fraudulent Test Sites
Data forensics can detect test sites operated by pirates and fraudsters and can provide the necessary evidence for taking legal action.
Proxy Testers
Data forensics can uncover the illicit activity of proxy test takers or professionals for hire, who take tests for others, and the candidates who employ them.
The Hidden Costs of Poor Exam Security
Hover over each box to see a potential area of cost that poor exam security can inflict on a testing program.
TIME
TEST VALIDITY
MONEY
LEGAL FEES
EFFORT
REPUTATION
RESOURCES
CONFIDENCE
I Have My Data Forensics Report. Now What?
What To Do With Your Data Forensics Report
So you received your data forensics report—how should you proceed? Your data forensics experts should work with you to determine the following:
- Would you like to receive recommendations in your data forensics report?
- What actions would your organization be willing to take? On what evidence?
Possible Courses of Action:
- Investigation of particular test instances
- Investigation of particular test groups
- Investigation of particular test sites
- Refurbishment of tests when pass rates creep up
- Score invalidation of specific test instances
- In extreme cases, legal action against test fraud
You can download this workbook for additional recommendations.
Talking to Your Organization About Data Forensics in Testing
Five ways to convince your organization to adopt data forensics:
“I want our exam scores to mean something to the people who worked hard to earn them.”
“Let’s protect this team’s investment—We’ve worked hard on this test.”
“Data forensics will be an efficient way to pinpoint any of our security vulnerabilities.”
“We could save time and resources in the long run if we conduct data forensics of our exams now.”
“We already have the necessary data to conduct data forensics analyses.”
Want to take a copy of the guide to-go?
You don’t need a disposable lid, bag, or box. Just download here.
Check Out These Other Test Security Resources
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Want to subscribe to our newsletter?