From Design to Detection:
Implementing Chameleon Clones to Enhance Exam Security
By: Dr. Jill Bryant, Dr. Brooke Houck, and Dr. Sarah Toton
NBEO and COVID-19
The National Board of Examiners in Optometry (NBEO©) provides a three-part exam series for jurisdictional regulatory boards to make decisions regarding licensure for optometry candidates. NBEO’s exams are universally accepted in the United States and in some international jurisdictions.
NBEO's Three Exams
Part 1: Applied Basic Science
- Computer-based, delivered through a professional testing center by a third-party vendor
Part 2: Patient Assessment and Management
- Computer-based, delivered through a professional testing center by a third-party vendor
Part 3: Clinical Skills
- Performance-based, delivered using highly-trained standardized patients at the National Center of Clinical Testing in Optometry (NCCTO) headquarters in Charlotte, North Carolina
Having worked with Caveon for more than a decade, NBEO already had multiple steps in place to prevent, deter, and detect test integrity violations. However, like so many other organizations in the testing space, NBEO experienced multiple challenges associated with the COVID-19 pandemic, leading them to collaborate with Caveon’s Data Forensics team and identify ways to improve security while adapting to their new testing needs.
The Challenges of the Pandemic
In March of 2020, worldwide testing center closures interrupted NBEO’s administration of computer-based exams, resulting in the loss of a major four-day exam administration. Prior to the pandemic, these administrations were only offered twice per year.
When the testing vendor reopened, the organization’s examinees were suddenly faced with competition for appointments—a problem which in itself would be cumbersome, but was exacerbated further by the reduced seating capacity for safe social-distancing protocols.
At this juncture, NBEO was faced with a decision: Would it make sense to lengthen the typical four-day testing administration window into a three-week window? To do this would be a major shift for the organization, and it would require a solid test security strategy to manage the change.
Caveon was able to help develop a security plan from the available options. NBEO needed to offset security concerns that accompanied the necessary extension to their testing window. Enter Chameleon Clones to balance the scales.
chameleon clones
A chameleon clone is an item that has a similar stem to at least one other item and the same set of response options. However, a tiny change is made to the stem so that the answer is now different.
What purpose does a chameleon clone serve?
There’s some evidence that examinees with pre-knowledge do not thoroughly and completely read every word of the item when taking the exam, since they’ve seen them before—or at least they think they have (Toton & Maynes, 2019; Man & Harring, 2021). Chameleon clones maintain a few key features of the item so that the chameleon items look mostly the same as their parents. However, we are changing the item in such a way that any potentially memorized answers would be incorrect. Importantly, we keep the same set of response options, which aids in camouflaging the chameleon clone. Finally, the answer must be different based on the stem changes to the clone.
- Disrupt the benefits of harvesting.
Someone who attempts to steal or harvest an item that utilizes chameleon clones will steal only one version, or clone, of a parent item. There are still other versions of that item in the universe of potential items that another test taker would see, so a person with pre-knowledge of the test content would very likely be stuck with a useless version of that item. Anyone with access to pre-knowledge or anyone who has seen an exposed item will have no advantage when faced with a chameleon clone of that same item. In fact, it will likely serve as a disadvantage to have seen the chameleon clone.
Consider this example:
On the left, we have a parent item. It reads: “If X is greater than one and the square root of X divided by X cubed equals X to the M power, what is the value of M?” For this parent version, “C” is the correct answer.
However, on the right, we’ve taken the same item and made a small change. Instead of X cubed, we have X squared. Consequently, E is now the correct answer. So, if someone had seen the exposed parent item on the left (e.g., in an online braindump) and responded to its clone with “C,” they would get that item incorrect.
2. Confuse the pre-knowledge process.
Better still, chameleon clones aren’t just meant to be made from numerical math items. We can do this with text-based items, too.
On the left, the item reads, “The party guests were enthralled by the host’s ________ anecdotes, which were full of engaging and provocative details.” For this question, “A” would be the correct answer. But in a possible clone for this item, we can change “enthralled” to “confused.” Then we can change this last phrase “of engaging and provocative details” to “of digressions and lacked focus.”
That means that the answer is now “D” in the cloned item. However, when faced with this chameleon clone, someone who has pre-knowledge of the original item is probably thinking:
“The party guests, okay… I know the answer to that one. It’s ‘A.'”
3. Leave a breadcrumb trail.
If we only see that one answer to one item is incorrect in this way, it might not be compelling evidence of pre-knowledge or fraud, and it might not help us find any culprits. However, by using a batch of chameleon clones on an exam, we can build up evidence that a particular test taker had access to disclosed content.
Incorrect responses to batches of clones show a pattern of pre-knowledge.
Using chameleon clones allows us to obtain actionable results regarding pre-knowledge. With enough clones, we can draw reasonable conclusions that someone has had access to disclosed items.
4. Prevent refurbishment emergencies.
Creating Chameleon Clones
There are two basic ways to create chameleon clones:
Clone Existing Items
This method starts with changing the stem and maintaining that style from the original item to the clone. We are assuming that braindump users will not notice the change and will select the answer to the parent item. Make sure the chameleon clone item’s answer is different from the parent item’s answer.
Create New Items and Clones Simultaneously
While developing test items, create multiple versions of those items that share the same response options but have different correct answers.
Not-So-Clonable Items
We can think of the clonability of an item as existing on a spectrum. In fact, some items are just not clonable at all. There’s a spectrum from “very clonable” to “impossible to clone,” and all items fall somewhere on that spectrum.
Creating chameleon clones for some topic areas and item constructs is easier than it is for others. For example, math and other items with numerical stems and answer options are relatively easy to clone. On the other hand, it’s more difficult to clone an item with a very short or memorable stem. This is partly because there would need to be extensive changes to the stem, thus reducing its resemblance to other “clones” and disrupting the effect.
As an example of an item that is not clonable, we have a sketch drawing of a mountain. The accompanying item would read, “What topographic feature is pictured below?” Without changing the image, this item is unclonable. There’s a picture, it has one answer, and it would be very hard to create variants with that stem and that image.
EXAMPLES OF CHAMELEON CLONES: OPTOMETRY
PRO TIP: If you’re moving forward using chameleon clones, it’s very important to remember that you need content experts who can make sure the clone content is correct for your specific field.
CHAMELEON CLONES’ IMPACT ON FAIRNESS
When we talk about implementing new tools to subvert fraudulent test-taker behavior, we must always weigh the impact that those tools will have on honest examinees. Even though we want to protect the validity of exams, we definitely do not want to disadvantage honest test takers in the process.
In the case of chameleon clones, how will a chameleon clone be perceived by an honest test taker? The answer is pretty simple. For them, a chameleon clone looks and functions just like a regular item.
However, when an examinee has pre-knowledge of the exam content, all bets are off. They will be far more likely to mistakenly perceive clones as the disclosed items they are familiar with. Chameleon clones work by leveraging the examinee’s pre-knowledge against them, causing their pre-knowledge to poorly influence their exam performance. This, in turn, can help test programs detect who these test takers with pre-knowledge are and prevent them from gaining an unfair advantage over honest examinees.
Assessing Clonability
This spectrum can act as a roadmap that testing organizations can use to identify the “low-hanging fruit.” Which items can easily be transformed into clones? Which items, because of the nature of their content, will be more difficult to transform into clones?
PRO TIP: Cloning is impossible under the following circumstances:
- When changing the stem creates inaccuracies
- When changing the stem causes the item to no longer measure the objective
Chameleon Cloning: Pro Tips
Recognizability:
The more items there are, the more difficult it is to memorize them. So, if a question is highly memorable, a test taker is more likely to recall it, making it more likely that fraudsters will not notice a slight change. There is a very sweet spot between having enough information in the stem to accommodate minor changes and having too much unnecessary information. The idea is to trick the test taker with pre-knowledge using just enough information that they feel they recognize the item—but not too much.
Development Strategy:
Train your exam directors, subject matter experts (SMEs), and whomever in your organization puts together your exams. They need to be trained on what clones are, the difference between a true clone and a simple variation, and more. When creating a chameleon clone, it is possible to accidentally change one small thing in the stem of an item and radically change whether the answer options are relevant or useful. SMEs are an essential part of this process, because the human touch will help prevent those cases.
Identify Clonable Items in your Pool:
Ask your SMEs and other relevant personnel to review your current item pool and identify the “most clonable” items. Then, review those items based on your test blueprint categories. Which items do you think you should be able to clone relatively quickly?
Identify Your Target Number of Clones:
What is the target number of clones you’d like to produce before republishing this fresh version of your exam? That number is dependent on your unique goals. For NBEO, some determining factors included capacity (to what extent were staff and SMEs available to develop clones), purpose (in their case, weighing the length of the testing window against development time), and statistical power for detection. Good statistical power for detection is possible with as few as 10-15 clones (Toton & Houck, 2022).
PUBLISHING CLONES
In general, how you publish clones depends on the number of clones for each item. The number of clones for each item is, in turn, influenced by the items themselves. Perhaps you can generate three clones for every item you’ve decided to clone. Perhaps you can only get one clone for every item you’ve decided to clone.
Whatever the case, the strategy would then be to rotate the items and clones on a regular basis. The frequency of rotation really depends on your disclosure rate and how quickly you can republish a new form or test with different items.
Example Clone Publication Strategy:
- You have multiple clones for every item.
- Start with all the original items on the exam and replace a subset of those items with clones.
- Keep those clones on live exams for a while (the length of time is something you must determine based on your disclosure rate and your program’s goals).
- Replace those clones with a new set of clones.
- Repeat until you run out of clones for those items.
- For detection purposes, you can score each set of clones against the original parent item key and all previous clone keys. So, if the parent items were disclosed or the first clone set was exposed, you’d then be able to compare the various answer keys and gain a wealth of knowledge about your exposure and the examinees who may have had pre-knowledge of your exam content.
So, what do we know about chameleon clones so far? The key feature of the chameleon clone is that the answer to every clone is different from the original item’s answer. Then, by assessing the alignment of an examinee’s responses to the parent or previous clone answers, you can assess pre-knowledge. So, of a particular subset of items, do you have evidence that a person has seen the parent items and keys? Do you have evidence that they’ve seen a different set of clones or keys? This matching with previous answer keys is what provides evidence that a person has seen disclosed test content.
Clones, all-in-all, are a clever and creative way to fulfill many test development objectives. By expanding your item bank with chameleon clones, you protect the validity of your test scores and do right by your honest test takers by preventing pre-knowledge from skewing your test scores. You can even detect pre-knowledge, giving you the opportunity to pinpoint and shore up any weak spots in your test security.
READY TO TALK TO AN EXAM SECURITY EXPERT?
Reach out and tell us about your organization’s needs today!
