How to Build & Design a Super-Secure Online Test
The Comprehensive Guide
Exam Design & Development 101
We get it—Exam development is hard.
Writing an excellent (and psychometrically sound) online exam is a difficult, time-consuming, and expensive undertaking. And it is devastating when the exam you worked so hard to develop is stolen or exposed, and all that hard work (and money) goes down the drain.
Luckily, there is a solution. You can spare your team this heartache by proactively building security directly into your online exam itself during the development process.
Let’s learn how.
The Benefits of Secure Item and Exam Design
Longer-Lasting Exams
Reduced exposure leads to longer-lasting item pools, so you can wait much longer between performing maintenance on your exams. Content is less likely to be breached or even beneficial to test takers who’ve somehow gotten access to it.
Restored Validity
The combined power of more accurate measurement and increased security will lead to the restored validity of test scores. Test takers and stakeholders will all be able to respect and trust the scores of your important exams (in case you didn’t know—all of your exams are important!).
Reduced Exposure
New forms of secure online test and item designs are primarily guided by the concept of reduced exposure. By reducing the relative amount of content your test takers see, you’ll reduce the risk of item exposure to other groups.
Better Measurement
In addition to bolstering security, the newer and more advanced item types (like the SmartItem™) and methods of exam design can measure the abilities of test takers far better than their archaic counterparts. You can learn more about advanced item and test designs in this section.
Mitigated Costs
Walking hand-in-hand with reduced exposure is the lovely benefit of mitigated costs. Longer-lasting item pools require less maintenance and damage control than traditional multiple-choice items.
What to Consider
Before Developing and Designing Your Exam
What types of security do you need?
Now that you know the impact of secure exam design, how will that affect the rest of your choices from this point forward? When thinking about the types of security your exam will need, consider these three questions—what are you protecting, what are you protecting against, and what are the best ways to protect it. Then, perhaps write on a sticky note: “What About the Security?” Place it somewhere prominent to remind you to consider things you can do to improve security within each stage of the exam design & development process—before, during, and after building your exam. (Learn more about the 8 steps of the exam development process in this article.)
What type of exam will it be?
Your first major security consideration should be “What type of exam will you build?” Not all are created equally. Research different secure onlinetest designs and different types of test items that work for your content area. Here’s a great article about different exam items to help get you started.
What capabilities do you need?
Do you need to research or vet test development systems, delivery vendors, proctoring solutions, or any other components of your exam workflow before you get started? Make sure your desired security features are supported. You can find a list of the important features to look out for in this article.
Do you need extra training and support?
Is your team fully on board with the benefits of secure design? What type of training and support might they need to help smooth the implementation process? Would training on how to write test questions be beneficial for your team? You can also check out these three workbooks that have been specifically created to support your team before, during, and after the exam development process.
Development Questions for Your Team
How will you measure? What is the best way to test the skills you’re trying to measure? Essay questions, performance sections? Should you build or buy your exam?
Who is on your exam development team? Teams usually need a project manager, subject matter experts (SMEs), and other key players when creating exams. Who is responsible for ensuring security? You can view a list of helpful team players on page 15 of this workbook.
What format will you use? What exam format makes the most sense for your testing purposes? Keep in mind that some formats are more secure than others.
What item types should you use? Different item types work best with different content domains. Which forms of test items suit your content best? How can secure item types be best utilized?
What’s your budget? Make a list of your must-have security, development, and delivery needs. Allocate your budget accordingly.
What’s your project deadline? Work backwards from there.
The 6 Basic Steps to Online Exam Design & Development
1. Purpose: What need does your exam fill? What, who, why, and how are you testing? The answer to this question should inform your process every step of the way.
2. JTA: Job Task Analysis. This is when you’ll start scoping the skills that you’ll measure and defining your audience.
3. Blueprinting: How many items will be assigned to each skill outlined in your JTA?
4. Item Writing & Review: Writing test questions and answers requires SMEs and writers to collaborate while building the exam.
5. Cutscore Setting: Psychometricians should help you determine your exam’s cutscore for a minimally-qualified candidate.
6. Launch & Track: After your exam goes live, don’t forget to observe your testing data for suspicious patterns
Again—What About Security?
Common Exam Development Myths Debunked
Cheating doesn’t happen in my program:
This may hurt, but you probably need to hear it: Exam fraud happens all over, all the time, for all testing programs (and it has quite the impact). For many otherwise honest people, the end justifies the means when an opportunity is at stake. In other words, every exam is a high-stakes exam, and yours is at risk. You can read about the most common types of cheating in online exams in this article.
Any type of test or item format I use will be fine:
Not all item types are created equal. Some are easier to score, some are easier to implement, and some are far more secure. Consider scalability, budget, accessibility, and (most importantly) security when choosing your item formats.
Cheating is an isolated behavior problem:
Wrong-to-right answer changes and similarity analyses (learn more about similarity analyses in this ultimate guide) can help detect educators and exam administrators who commit test fraud.
Once I launch my exam, the hard part is over:
Well, yes and no. Exam maintenance is equally important. After launch, you’ll want to track how well your exam is performing using security and investigative tools (like web monitoring or data forensics). That way, you can detect + react to rampant fraud before it takes you down.
Designing an exam out of self-securing item types is too fancy for me:
Data forensics can uncover the illicit activity of proxy test-takers or professionals for hire (who take tests for others) and the candidates who employ them.
Test Construction Guidelines
Examples of Secure Online Test and Item Formats:
Randomize— Item order and option order can both be randomized to increase security (learn more about the benefits of randomization in this article).
CAT— Computerized Adaptive Test (CAT) design adapts to you.
LOFT— Linear on the Fly Test (LOFT) design is strong and secure.
SmartItem— The SmartItem renders cheating and theft useless.
Performance— Performance-based items have an extra layer of security.
Multiple Forms— Utilize multiple forms to reduce the effects of content exposure.
Replacement— Rapid item or form replacement policies can mitigate damage.
AIG— Expand your item pool with Automated Item Generation.
DOMC™— Discrete Option Multiple Choice™ items reduce exposure.
Security-Enhancing Hacks for Your Exam
Scramble item and response-option orders:
Design exams with scrambled item and response-option orders. Randomizing your items and response options is a good first step in making it more difficult to steal your exam content and use it to cheat. After all, pre-knowledge is only useful when examinees see what other content thieves before them have seen.
Multiple forms:
Create a large item pool using AIG:
The larger the item pool, the less content each individual test taker will see. Additionally, an expansive item pool gives you the option to replace exposed items quickly, avoiding widespread validity issues. Learn more about the benefits of a large item pool below.
Make your items “smart”:
A SmartItem uses special technology during development and delivery so that the item renders differently each time it is given on a test. Translation—no two examinees see the same thing during testing, so “stealing answers” to gain an advantage is pointless. To encourage proper studying and instruction, the SmartItem is coded to completely cover the target standard or competency. No more “teaching to the test.”
The Benefits of an Expansive Item Pool
Limit Item Exposure
Numbers don’t lie. More items in your item pool mean that fewer eyes see each item. Limit item exposure with Automated Item Generation by rapidly increasing your item pool and staying one step ahead of content thieves and aging exams.
Block Pre-Knowledge
A larger pool of items can prevent test takers from using pre-knowledge. This will make it harder for examinees to predict what their test questions will be and prevent them from guessing what the correct answers are. Nice try!
Keep You Ahead
To combat item theft, it is important to stay one step ahead of prying eyes, and AIG makes it possible to refresh your item pool as often as necessary, as painlessly as possible. And the lifetime cost savings with AIG are pretty impressive as well—view this infographic to learn more.
Use Fewer Humans
Maintain an extra layer of security by involving fewer individuals in the item writing process. AIG enables smaller teams to do more with less—efficiency and security all rolled into one.
Get Adaptive
Secure test designs such as CATs, LOFTs, and the use of multiple forms require fairly large item banks to pull off. That hasn’t always been attainable, but thanks to modern AIG technology, it’s now possible to do more with fewer resources.
What Makes an Item "Smart"?
Facts about SmartItem Tech:
It can render in millions of ways.
While AIG and SmartItem technology may sound similar, they are completely different innovations. One single well-designed SmartItem can render in millions of ways and present different renderings of the item to each test taker. That’s an amazing amount of variability!
It promotes fairness.
SmartItem technology actually enhances the fairness of your exams by completely eliminating the two largest sources of unfairness: testwiseness and cheating.
It prevents cheating.
Designing your exam with SmartItems prevents cheating. With so many renderings, you can no longer cheat by sharing test content, buying questions and answers, asking a friend to take the test and tell you what they saw, etc.
It mitigates testwiseness.
Success on an exam should depend on the depth of an examinee’s knowledge—not their ability to tactically take tests. DOMC items have also been shown to reduce the effects of testwiseness on scores.
It can curb ballooning costs.
SmartItem technology ensures that you don’t need to replace or fix your test until the skills you are measuring are revised. This curbs the draining costs that typically accompany content theft endeavors and security breaches. You can learn more about SmartItem technology in our Ultimate Guide.
It can’t be stolen.
It works with any item type.
No matter your question type—multiple-choice, drag-and-drop, short answer, or others—converting them into SmartItems is possible. A SmartItem is not an item type; it’s a treatment for any kind of item. Whatever item types you use, you can make them render in unlimited ways. View SmartItem test item examples here.
The Lifespan of a Secure Online Test
Unsecured Tests
Likelihood of Exposure: HIGH
Likelihood of Theft: HIGH
Useable Lifespan: SOMETIMES WEEKS
Security-Enhanced Exams
Likelihood of Exposure: MINIMAL
Likelihood of Theft: MODERATE
Useable Lifespan: SOMETIMES EVERGREEN
READY TO TALK TO AN EXAM SECURITY EXPERT?
Reach out and tell us about your organization’s needs today!
