We’ve been there. Enter your email to get a permanent copy of this guide sent directly to your inbox.
So you’re curious about data forensics? Good. If you’re already thinking about the health of your assessment, especially when it comes to security, then you’re on the right track! Give yourself a pat on the back.
But what is data forensics? How does it work? What security threats and vulnerabilities actually apply to your exam?
Consider this your ultimate guide, which will walk you through the basics of statistical analyses in testing data. We’ll explore the security vulnerabilities that may be hidden within your assessment program and talk about how you can detect and combat them with data forensics!
Keep reading to learn about the hidden costs of poor exam security, what you should do after conducting data forensics, and (importantly) how to help your organization adopt this program-strengthening practice. Here’s to integrity!
What are some classic psychometric statistics that data forensics experts use to pinpoint patterns and other clues in testing data?
Examine how closely an individual’s responses match other individuals’, and calculate the probability of the matches.
Tabulate how many answers were changed on a test, and flag high rates of wrong-to-right answer changes.
Examine unusual response time patterns to monitor for abnormal test-taking behavior, such as super-human answering speeds.
Scan for unusual patterns in the data. These statistics are most useful in supporting other statistics.
Terms that are commonly used in data forensics include:
Unusual patterns in the data that may or may not reflect violations or fraud.
Example: Examinees that are demonstrating two different levels of proficiency.
Out-of-the-ordinary events occurring during testing, not always violations.
Example: A fire alarm disrupts examinees during the testing window.
Events that violate testing protocol and could potentially threaten the security of an exam.
Example: The seats in the testing room are too close together.
Exam integrity has been jeopardized, though not necessarily on purpose.
Example: An administrator accidentally discloses exam content to examinees.
An intentional act that threatens the integrity of an exam.
Example: An administrator intentionally discloses answers to examinees.
Every testing program is unique. Understanding your risks will help you decide which analyses will be most informative as you manage them. Start by taking these questions into consideration:
If you already have a data forensics vendor, work with them to determine what types of data you can obtain. These can include test center identifiers, response times, answer-change data, or timestamps that track when each item is viewed. The data types you have access to will determine the types of data forensics statistics that can be run.
The statistics you can compute should mirror the primary security concerns you laid out in Step 1. Once you’ve determined the statistics you can use, ask yourself:
Next, align the statistics you select with your primary security concerns. This will ensure that you capture critical information and evidence to support your test security initiatives.
Will psychometricians analyze individuals, test sites, test forms, or items? Perhaps some combination of these? Often, detection in data forensics is a multi-layer issue. The deeper you dive into the layers of data, the more information you’ll have about what is actually going on. Try to add as many contextual layers as possible to your results.
How should the results be presented, summarized, and formatted so they are most helpful for your team and stakeholders? What resources are available to help you understand the results? Choose a vendor who can interpret and prioritize your results for you, giving you the ability to digest your results and take action quickly, if necessary.
How will you respond if you find evidence of cheating or test fraud? Decide what you will do when the data forensics evidence meets your threshold and standard for taking action. If you’re in high-stakes testing, remember to choose a team of psychometricians with experience and success providing expert witness testimonies to data forensics results in court.
The path to exam validity is different for every assessment program, but there are several milestones every program meets on the journey to exam integrity.
Whether you’ve experience a known incident or you’ve decided to use data forensics for one of the reasons listed below, deciding to conduct data forensics is a critical first step.
From basic response booklets to online answer-change and response-time data, your data forensics vendor will want whatever you can provide to strengthen the results.
You should work with your vendor to decide what kind of results you want to receive. They’ll help you put together a general security plan for incident response ahead of time.
If your results show areas of potential vulnerability, work with your vendor to collect more evidence and investigate further.
Engaging in data forensics offers proof to your stakeholders, your industry, and the world that you are taking steps to ensure that your exam is an accurate representation of your examinees’ skills (learn why that’s important in this article).
By publicizing your use of data forensics, you can deter test fraud. Both candidates and stakeholders will know that test fraud of any kind is not tolerated (or easy to get away with).
Data forensics can detect potential security vulnerabilities before a widespread breach, allowing the issue to be addressed as soon as it appears.
Data forensics can tell you where and when suspect activity occurred, and by whom. It can reveal large-scale, organized fraud and even detect fraud among individuals.
Data forensics experts can assess the health of your program, keep you abreast of the latest security innovations, and offer sound advice based on their experience in test security and data forensics.
Similarity statistics and source-copier statistics can sniff out examinees who peek at others’ exams or who share test content with other examinees.
Similarity statistics and flawed-key analyses detect the use of stolen exam content contained within “Test Killers,” distributed within small cheat rings and dispersed among test takers.
Wrong-to-right answer-changes and similarity analyses can help detect educators and exam administrators who commit test fraud.
Data forensics can detect large-scale examinee collusion and support disciplinary actions by testing programs against individuals in cases of test fraud.
Data forensics can detect test sites operated by pirates and fraudsters and can provide the necessary evidence for taking legal action.
Hover over each box to see a potential area of cost that poor exam security can inflict on a testing program.
So you received your data forensics report—how should you proceed? Your data forensics experts should work with you to determine the following:
You don’t need a disposable lid, bag, or box. Just download here.
Want to subscribe to our newsletter?