CONTENTS
Introduction to Exam Design & Development
The Benefits of Secure Item and Test Design
What to Consider Before Development
Development Questions for Your Team
The 6 Basic Steps to Exam Development
Common Exam Development Myths Debunked
Construction: Examples of Secure Test and Item Formats
Security-Enhancing Hacks
The Benefits of an Expansive Item Pool
What Makes an Item “Smart”?
The Lifespan of a Secure Test
Too many tabs open?
We’ve been there. Enter your email to get a permanent copy of this guide sent directly to your inbox.
Introduction to Exam Design & Development
We get it—Exam development is hard.
Writing an excellent (and psychometrically sound) exam is a difficult, time-consuming, and expensive undertaking. And it is devastating when the exam you worked so hard to develop is stolen or exposed, and all that hard work (and money) goes down the drain.
Luckily, there is a solution. You can spare your team this heartache by proactively building security directly into your exam itself during the development process.
Let’s learn how.
The Benefits of Secure Item and Exam Design
Longer-Lasting Exams
Reduced exposure leads to longer-lasting item pools, so you can wait much longer between performing maintenance on your exams. Content is less likely to be breached or even beneficial to test takers who’ve somehow gotten access to it.
Restored Validity
The combined power of more accurate measurement and increased security will lead to the restored validity of test scores. Test takers and stakeholders will all be able to respect and trust the scores of your important exams (in case you didn’t know—all of your exams are important!).
Reduced Exposure
New forms of secure test and item designs are primarily guided by the concept of reduced exposure. By reducing the relative amount of content your test takers see, you’ll reduce the risk of item exposure to other groups.
Better Measurement
In addition to bolstering security, the newer and more advanced item types (like these) and methods of exam design can measure the abilities of test takers far better than their archaic counterparts. You can learn more about advanced item and test designs in this section.
Mitigated Costs
Walking hand-in-hand with reduced exposure is the lovely benefit of mitigated costs. Longer-lasting item pools require less maintenance and damage control than traditional multiple-choice items.
What to Consider Before Developing & Designing Your Exam
What types of security do you need?
Now that you know the impact of secure exam design, how will that affect the rest of your choices from this point forward? When thinking about the types of security your exam will need, consider these three questions. Then, perhaps write on a sticky note: “What About the Security?” Place it somewhere prominent to remind you to consider things you can do to improve security within each stage of the exam design & development process—before, during, and after building your exam. (Learn more about the exam development process in this article.)
What type of exam will it be?
Your first major security consideration should be “What type of exam will you build?” Not all are created equally. Research different secure test designs and different types of test items that work for your content area. Here’s a great article to get you started.
What capabilities do you need?
Do you need to research or vet test development systems, delivery vendors, proctoring solutions, or any other components of your exam workflow before you get started? Make sure your desired security features are supported. You can find a list of the important features to look out for in this article.
Do you need extra training and support?
Is your team fully on board with the benefits of secure design? What type of training and support might they need to help smooth the implementation process? Would training on how to write test questions be beneficial for your team? You can also check out these three workbooks that have been specifically created to support your team before, during, and after the exam development process.
Development Questions for Your Team
How will you measure? What is the best way to test the skills you’re trying to measure? Essay questions, performance sections? Should you build or buy your exam?
Who is on your exam development team? Teams usually need a project manager, subject matter experts (SMEs), and other key players when creating exams. Who is responsible for ensuring security? You can view a list of helpful team players on page 15 of this workbook.
What format will you use? What exam format makes the most sense for your testing purposes? Keep in mind that some formats are more secure than others.
What item types should you use? Different item types work best with different content domains. Which forms of test items suit your content best? How can secure item types be best utilized?
What’s your budget? Make a list of your must-have security, development, and delivery needs. Allocate your budget accordingly.
What’s your project deadline? Work backwards from there.
The 6 Basic Steps to Online Exam Design & Development
1. Purpose
What need does your exam fill? What, who, why, and how are you testing? The answer to this question should inform your process every step of the way.
4. Item Writing & Review
Writing test questions and answers requires SMEs and writers to collaborate while building the exam.
2. JTA
Job Task Analysis. This is when you’ll start scoping the skills that you’ll measure and defining your audience.
5. Cutscore Setting
Psychometricians should help you determine your exam’s cutscore for a minimally-qualified candidate.
3. Blueprinting
How many items will be assigned to each skill outlined in your JTA?
6. Launch & Track
After your exam goes live, don’t forget to observe your testing data for suspicious patterns.
Again—What About Security?
Security can (and should) be built into each one of these steps. We recommend implementing security measures that prevent, deter, and detect + react to test security threats (learn more about the test security process in this article).
Some secure test designs (such as the SmartItem™) might change the flow of the test design process, streamlining certain areas and requiring more care and consideration in others. Remember, the end goal is to create a great, secure exam. The best process for you? Well, that is simply the one that gets you there.
Common Exam Development Myths Debunked
Cheating doesn’t happen in my program:
This may hurt, but you probably need to hear it: Exam fraud happens all over, all the time, for all testing programs (and it has quite the impact). For many otherwise honest people, the end justifies the means when an opportunity is at stake. In other words, every exam is a high-stakes exam, and yours is at risk. You can read about the most common types of cheating in online exams in this article.
Any type of test or item format I use will be fine:
Not all item types are created equal. Some are easier to score, some are easier to implement, and some are far more secure. Consider scalability, budget, accessibility, and (most importantly) security when choosing your item formats.
Cheating is an isolated behavior problem:
Wrong-to-right answer changes and similarity analyses can help detect educators and exam administrators who commit test fraud.
Once I launch my exam, the hard part is over:
Well, yes and no. Exam maintenance is equally important. After launch, you’ll want to track how well your exam is performing using security and investigative tools (like web monitoring or data forensics). That way, you can detect + react to rampant fraud before it takes you down.
Designing an exam out of self-securing item types is too fancy for me:
Data forensics can uncover the illicit activity of proxy test-takers or professionals for hire (who take tests for others) and the candidates who employ them.
Test Construction Guidelines: Examples of Secure Test and Item Formats
Randomize
Item order and option order can both be randomized to increase security.
SmartItem
The SmartItem renders cheating and theft useless.
Replacement
Rapid item or form replacement policies can mitigate damage.
CAT
Computerized Adaptive Test (CAT) design adapts to you.
Performance
Performance-based items have an extra layer of security.
AIG
Expand your item pool with Automated Item Generation.
LOFT
Linear on the Fly Test (LOFT) design is strong and secure.
Multiple Forms
Utilize multiple forms to reduce the effects of content exposure.
DOMC™
Discrete Option Multiple Choice™ items reduce exposure.
Security-Enhancing Hacks for Your Exam
Scramble item and response-option orders:
Design exams with scrambled item and response-option orders. Randomizing your items and response options is a good first step in making it more difficult to steal your exam content and use it to cheat. After all, pre-knowledge is only useful when examinees see what other content thieves before them have seen.
Multiple forms:
Having multiple forms decreases the amount of exposure your test questions get, since different test takers are seeing different forms. It also allows for exposed forms to be easily removed and replaced.
Create a large item pool using AIG:
The larger the item pool, the less content each individual test taker will see. Additionally, an expansive item pool gives you the option to replace exposed items quickly, avoiding widespread validity issues. Learn more about the benefits of a large item pool below.
Make your items "smart”:
A SmartItem uses special technology during development and delivery so that the item renders differently each time it is given on a test. Translation—no two examinees see the same thing during testing, so “stealing answers” to gain an advantage is pointless. To encourage proper studying and instruction, the SmartItem is coded to completely cover the target standard or competency. No more “teaching to the test.” Jump ahead to this section to learn more.
The Benefits of an Expansive Item Pool
Limit Item Exposure
Numbers don’t lie. More items in your item pool mean that fewer eyes see each item. Limit item exposure with Automated Item Generation by rapidly increasing your item pool and staying one step ahead of content thieves and aging exams.
Block Pre-Knowledge
A larger pool of items can prevent test takers from using pre-knowledge. This will make it harder for examinees to predict what their test questions will be and prevent them from guessing what the correct answers are. Nice try!
Keep You Ahead
To combat item theft, it is important to stay one step ahead of prying eyes, and AIG makes it possible to refresh your item pool as often as necessary, as painlessly as possible. And the lifetime cost savings with AIG are pretty impressive as well—view this infographic to learn more.
Use Fewer Humans
Maintain an extra layer of security by involving fewer individuals in the item writing process. AIG enables smaller teams to do more with less—efficiency and security all rolled into one.
Get Adaptive
Secure test designs such as CATs, LOFTs, and the use of multiple forms require fairly large item banks to pull off. That hasn’t always been attainable, but thanks to modern AIG technology, it’s now possible to do more with fewer resources.
What Makes an Item "Smart"? Facts About SmartItem Tech:
It can render in millions of ways.
While AIG and SmartItem technology may sound similar, they are completely different innovations. One single well-designed SmartItem can render in millions of ways and present different renderings of the item to each test taker. That’s an amazing amount of variability!
It promotes fairness.
SmartItem technology actually enhances the fairness of your exams by completely eliminating the two largest sources of unfairness: testwiseness and cheating.
It prevents cheating.
Designing your exam with SmartItems prevents cheating. With so many renderings, you can no longer cheat by sharing test content, buying questions and answers, asking a friend to take the test and tell you what they saw, etc.
It mitigates testwiseness.
Success on an exam should depend on the depth of an examinee’s knowledge—not their ability to tactically take tests. DOMC items have also been shown to reduce the effects of testwiseness on scores.
It can curb ballooning costs.
SmartItem technology ensures that you don’t need to replace or fix your test until the skills you are measuring are revised. This curbs the draining costs that typically accompany content theft endeavors and security breaches. You can learn more about the cost savings in this booklet.
It can’t be stolen.
Well, it can, but it won’t matter. A SmartItem covers the entire breadth and depth of a skill, rendering differently for each test taker. So, memorizing a SmartItem answer key would be a much poorer tactic than just studying the materials.
It works with any item type.
No matter your question type—multiple-choice, drag-and-drop, short answer, or others—converting them into SmartItems is possible. A SmartItem is not an item type; it’s a treatment for any kind of item. Whatever item types you use, you can make them render in unlimited ways. View SmartItem test item examples here.
The Lifespan of a Secure Test
Unsecured Tests
Likelihood of Exposure: HIGH
Likelihood of Theft: HIGH
Useable Lifespan: SOMETIMES WEEKS
Security-Enhanced Exams
Likelihood of Exposure: MINIMAL
Likelihood of Theft: MODERATE
Useable Lifespan: SOMETIMES EVERGREEN
Want to take a copy of this guide to-go?
You don’t need a disposable lid, bag, or box. Just download here.
Check Out These Other Test Security Resources
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Security Boot Camp Part 1: Preparedness
In Part 1: Preparedness, you’ll learn how to proactively boost your security prior to testing. You’ll uncover ways to create a comprehensive test security and response plan and find out how to effectively deter security threats by publicizing your test security measures.
Go to Workbook
Security Boot Camp Part 2: Technology
In Part 2: Technology, you’ll learn ways to evaluate the technology you use to shore up your test security. Evaluating your online testing solutions, leveraging secure item types, and effectively increasing the number of items on your tests are three key ways to ensure the long-term health of your exams.
Go to Workbook
Want to get our subscriber-only newsletters and resources?