White Paper:
The Past, present, and future of proctoring
By David Foster | CEO | Caveon Exam Security
I first introduced the concept of online proctoring at the 2006 Association of Test Publishers (ATP) Conference. At the time, I saw it as the next logical step in the evolution of proctoring, with benefits that could not be matched by a test center model. In the thirteen years since, this method of proctoring has proliferated, with a final mass migration from in-person to remote proctoring occurring as a result of the 2020 Covid-19 pandemic.
While I am thrilled that remote proctoring has been able to facilitate distance-based testing in this current environment, I am simultaneously concerned by the widely held, yet false impression that proctoring is a security panacea—that by employing proctors (either remote or in-person),you can be confident that your tests are secure. This simply isn’t the case.
In this paper, I will take us on a journey from the first known use of proctors in Imperial China through the present day, discussing how the role of proctors has changed (or hasn’t changed) over the centuries. I will focus my lens closely on the test security role that we rely on proctors to play, particularly the often overlooked risks associated with utilizing and relying on proctors. From there, I will move to a discussion of how we might reenvision the role of proctors in the future to best enjoy the benefits of proctors while mitigating the security risks.
When it comes to test security, it is only by looking at the history of proctoring and the ways it has—and most importantly has not—evolved in the current era that we can truly understand the role that proctors should play in protecting the validity of our tests.
“I am simultaneously concerned by the widely-held, yet false impression that proctoring is a security panacea—that by employing proctors (either remote or in-person), you can be confident that your tests are secure.
This simply isn’t the case.”
WHAT ARE THE SECURITY RESPONSIBILITIES OF PROCTORS?
To start, let’s make sure we’re all on the same page regarding the major security responsibilities of proctors. According to Proctoring Best Practices produced by ATP and the National College Testing Association (2015) , proctors are responsible for:
- Observing test takers while they take an exam.
This is a two-part responsibility that requires proctors to:
- Prevent test takers from stealing test content
- Prevent test takers from cheating.
2. Various administrative functions that are unrelated to test security.
For the purposes of this paper, we will focus solely on the first responsibility—the proctor’s responsibility to prevent test takers from stealing test content and/or cheating on the exam.
The Past: A Short Version of the Long History of Proctoring
The Imperial Civil Service Exams in China
The earliest known use of proctors occurred more than 2,000 years ago on the Imperial Civil Service exams in China. The definition of “high-stakes,” these exams were administered to male individuals throughout Imperial China under the Han Dynasty (206 BCE – 220 CE) onwards. The purpose was to provide a merit-based system to find the best candidates to staff the civil service bureaucracy. In addition to the employment opportunities allowed, it was also a way for individuals to improve their class status and become a member of the gentry class of scholar-officials. The exams were extremely difficult to pass and required extensive knowledge of government, Confucian Classics, law, and other subjects. These exams were used for over 1,000 years.
Image 1: Illustration of the administration of Civil Service Exams in China.
Image 2: Illustration of testing cubicles used for the Imperial Civil Service Exam in China.
At some point during the exam’s use, proctors were employed to help administer the exams and prevent cheating, along with other measures to stop test takers from being able to cheat. Image 1 and Image 2 provide illustrations of the testing conditions at the time, including the presence of a test proctor as well as the use of testing cubicles to separate test takers and stop them from being able to cheat off of each other’s exams. (You’ll note that while the cubicles might have stopped students from being able to see each other’s work, it also made it more difficult for proctors to observe them throughout the exam. Gain a little security, lose a little security.)
These measures were not without cause, as cheating certainly was a major problem on the examinations. As Miyazaki says in China’s Examination Hell (1981):
All too often fairness was compromised by candidates and examiners. When the competition became too stiff, some candidates felt that they must pass at any cost and, in the end, turned to dishonest methods. Once a dishonest act succeeded, other candidates felt that they would be hurt unless they acted in the same way, so gradually the evil spread.” (p. 119)
You can see examples of these dishonest methods that have survived the centuries in Images 3, 4, and 5. Look closely at the jacket, and you will see minuscule writing.
Images 3, 4, and 5: Cheat sheets used on Imperial Civil Service Exams.
More than that though, there is evidence that proctors colluded with test takers and were involved in helping examinees cheat to get higher scores on the Imperial Civil Service Exams. As stated in China’s Examination Hell:
“The candidates themselves not only did dishonest things, but also, to be safe, involved the examiners as well in their schemes.” (p. 121)
Lessons Learned from Proctoring in Ancient China
Three lessons can be learned about cheating and proctoring from the Imperial Civil Service Exams:
- Proctoring exams is a difficult task
- Catching cheaters is not easy for proctors
- Proctors can be part of the problem
Proctors were not particularly effective at stopping cheating in Imperial China, yet we have continued to utilize proctors to monitor exams for the two millennia since. Is there any reason to believe that proctoring has gotten better over the centuries?
I’m sure that most people reading this paper have taken a proctored test, whether remotely or in a testing center. Despite the presence of the proctor, did you feel it would be impossible to cheat? Probably not.
Like proctors, security threats have persisted over the years as well. Cheating and using cheat sheets has not gone out of style. Cheating and using substitutes or proxies (having someone take the test for you) is as popular today as it was in Imperial China. The theft of test content is a rampant problem that plagues testing programs today, and it is still possible to memorize actual questions and answers before a test and use that pre-knowledge to unfairly improve your score.
While proctoring methods might be more sophisticated than they were in Imperial China, so too are methods for thwarting them. Our persistent use of proctors certainly hasn’t put an end to the problem of cheating.
The Present: Proctoring from 1990-2021: When Technology Changed Testing
Let’s move to the present. For the purpose of this paper, let’s consider the “present” the era from 1990 to today. This is when technology was first applied to testing in a widespread manner and forever changed the way we test (and proctor tests).
How Has Technology Been Applied to Testing?
Here is a partial list of some of the ways that technology has been applied to testing since 1990:
- Computerizing tests has made paper tests obsolete—mostly.
- There are now thousands of centers around the world where electronic tests can be taken.
- The internet has given us new ways to administer tests.
- The internet has added new ways to proctor tests.
- High-stakes tests can now be given in homes. This would have been unthinkable ten years ago.
With all these advances and technological improvements, we need to ask ourselves a few big questions:
Is there reason to believe that the amount of cheating and theft has decreased over the past thousand years? We know, as discussed above, that cheating hasn’t gone away entirely, but is there any proof that it has decreased overall?
Is there any reason to believe that proctors today are better trained, more motivated, less corruptible, or show greater vigilance than in Imperial China?
Are cheating and theft easier to detect today than they were prior to all these technological advances?
The answer to all these questions is a resounding “NO.”
How Do We Know That Proctors Are So "Meh"?
Over the past several years, Caveon has had ample opportunity to analyze the effectiveness of proctors, both through our own research and through secret shopping engagements for clients. I will discuss the details in later sections, but for now, let me outline the conclusions that can be drawn from proctoring in today’s world:
- Proctors have only ever been “adequate” at the security part of proctoring.
- Proctors today have the same limitations they’ve always had, but now they are outmatched by the technology used by cheaters and thieves.
- Relying on proctors to secure exams—online or in a center—raises the security risks to testing programs.
Object Lesson #1: Fake IDs
Image 6 (below) is a picture of an ID. Can you tell if it is fake? Probably not. There are services on the internet that will provide fake IDs that are almost undetectable by the human eye, feeling and looking the same as well. Even well-trained law enforcement officers have a difficult time telling a real ID from a fake one. Yet, we expect proctors to be able to look at an ID and tell whether the person that presented it should be taking a test and isn’t a proxy test taker using a fake ID.
Image 6: Is this ID card real or fake?
Object Lesson #2: Hidden Cameras
Look at the below photos (similar to what a remote proctor would see) and try and identify the hidden camera.
Images 7 and 8: Can you spot the hidden camera?
How about in this image? Can you spot a camera?
Image 9: Can you spot a hidden camera in this image?
There are actually five cameras:
- A small, wearable camera on his shoulder
- A cell phone sticking out of a bag on the floor
- A hidden camera inside the spool of paper towels
- A Go-Pro on the bookshelf, and
- A camera inside a box in the background
This person has five hidden cameras, all pointed at the screen and all capable of recording (in very high fidelity) whatever test content they see.
Image 10: The hidden cameras found in Image 9.
Object Lesson #3: Detecting Pre-knowledge
Which of these test takers is using pre-knowledge? Who purchased the test questions and answers online before the test, memorized them, and is now using that knowledge to cheat? This is a popular and ever-growing method of cheating. No proctor can tell when someone is cheating this way; it occurs in plain sight.
Image 9: Which test taker is using pre-knowledge?
Proctoring in the Real World: Secret Shopping
One way to evaluate proctors is through secret shopping. One can send a fake examinee to take a test with instructions to try and cheat or steal the test content. The purpose is to see if the examinee’s actions are detected by the proctors. Below are the results from several secret shopping engagements Caveon undertook at the request of clients.
Secret Shopping of a Remote Testing Session
The results of two secret shopping engagements of remotely proctored exams:
On each occasion, the fake test taker attempted the infractions listed on Table 1, including: leaving and returning to the room; using notepaper to copy items; hiding a cell phone; using a cell phone to capture items; having someone enter the room; and taping notes to the bottom of the mouse. Of these, only two of the actions were even noticed by the proctor. Only a warning was issued, and the test was then allowed to continue. The rest of the “cheating” actions occurred without being detected.
Table 1: Testing infractions attempted by a secret shopper during a remote proctored exam.
Secret Shopping of a Test Center Session
This secret shopping engagement took place in a testing center with an in-person proctor. The “undercover” test taker was screened and then entered the testing environment, just like any other examinee.
In this instance, the test taker was able to: hide notes in the bathroom; take an unauthorized bathroom break to read the hidden notes, write down the test questions they had seen and then return to the testing environment; hide a cell phone in their waistband; and record more than 25 minutes of the testing session.
Table 2: Testing infractions attempted by a secret shopper during an in-person proctored exam.
In summary:
- All proctors were well trained
- All proctors followed strict protocols
- Proctors were nice and accommodating
- Proctors were distracted by other job tasks
- Searches could not be thorough
- Hidden actions were not detected
- Serious attempts to commit fraud went undetected, or the test was allowed to continue with only a warning.
Despite the proctors being very well trained, and despite the proctors following strict protocols (except the bathroom break, which should not have been allowed), the Caveon secret shoppers were able to get away with all sorts of methods of cheating and stealing test questions. These proctors were accommodating (perhaps too accommodating), but they were distracted by other tasks, did not conduct thorough enough searches, did not catch attempts to cheat, and did not catch the successful attempts to steal test content.
If these secret shoppers were intent on real fraud, they could have cheated with ease, stolen questions without being caught, and exposed the testing organization to reputation and financial loss if they had posted or sold the questions online. This all took place in an environment where the proctors were trained well and truly trying their best.
Unscrupulous Proctors
It is also important to remember that during China’s Imperial Civil Service Exams, it wasn’t only the candidates who were dishonest, but the proctors as well. This has not changed in the 21st century. We need only look at news headlines from the Varsity Blues scandal of 2019 to see that proctors can still be a security risk. In that instance, proctors were paid to change SAT answers for specific students. It is not that proctors are simply unable to perform security tasks well, but at times, they are part of the problem.
The Future of Proctoring: A Paradigm Shift
In the centuries since Imperial China, we have relied on proctoring to keep our tests secure. Yet proctoring (even when enhanced with AI), is a mediocre solution for test security. So long as humans are required to catch cheaters simply by observing them, cheaters and thieves will continue to thrive.
A New Paradigm
We must take a moment and remind ourselves that the failure to catch cheaters and thieves or ensure test security is not the proctor’s fault; it is the fault of those of us who lead this industry or who run testing programs. We are asking proctors to do something that people are simply incapable of doing well—a task that is getting more difficult with every year that goes by. We are setting proctors up to fail.
It is time for us to reenvision the role of proctors in the testing process, particularly in regards to test security. We need a radical change, not just a movement from proctoring in person to proctoring remotely, for as the saying goes, “Insanity is doing the same thing over and over again and expecting a different result.” It is time for a paradigm shift.
A NEW PARADIGM FOR PROCTORING
Step 1: Remove Proctors from the Security Equation
The first step is a simple yet shocking one: don’t ask proctors to try to catch cheaters and thieves. As we have seen, proctors are not a great defense against cheaters and thieves. It is time to stop making them try. They should play a role in our testing process, but we need to stop relying on them as our main security measure.
Step 2: Test and Item Design
Stop most cheating and theft by using NEW test and item designs. Use designs that are:
- Technology-based
- Intelligent
- Preventative
- Psychometrically sound
Proctors cannot stop cheating and theft, but you know what can? Innovative test and item designs like CAT, the DOMC™ item type, Performance Exams, supersized item banks, and the SmartItem™. It is time to change the way tests and items are designed and built. We have amazing technology available today that can take tests far beyond the simple multiple-choice exam that is reused over and over and easily copied and stolen. These new test and item designs are psychometrically sound, and they can actually prevent cheating and theft, making it literally impossible for many types of test fraud to occur.
Step 3: Enhanced Administration and Security Policies
Address all remaining threats that weren’t prevented through test and item design with new test administration policies and methods. Replace checking IDs (which can be easily faked) with a more effective biometric authentication. (There are many excellent biometric authentication offerings on the market, many of which align with the highest privacy protection standards.) In addition, make sure to cover security basics and incorporate strong deterrence and detection methods into the security process to capture any types of cheating that slip through the cracks (proxy test takers, etc.).
Step 4: Test Administration Supervisors
It is now time to reenvision the role of proctors in the testing process. While proctors provide a slight deterrent effect, they should be primarily used to supervise test administration and address any procedural or technical test taker issues that arise during testing. I suggest renaming them “Test Administration Supervisors” or something similar. They will remain critical to the test taking process, but will now operate in a role in which they will be successful.
Proctors and Test Security: A Summary of the Solution
Throughout history, we have relied on proctors to protect our exams, but they are ineffective at test security. It is time to change, lest we continue to be plagued by the security concerns of the past long into the future. If we really want to stop cheaters and thieves, we need to replace proctors in the test security process.
There are more effective methods of security available today—such as preventative item and test designs—than our outdated reliance on proctored testing. We need to reenvision the administrative role proctors play during exam administration and let them evolve into a role in which they will be just as needed, but so much more successful.
“We need to reenvision the administrative role proctors play during exam administration and let them evolve into a role in which they will be just as needed, but so much more successful.”
READY TO TALK TO AN EXAM SECURITY EXPERT?
Reach out and tell us about your organization’s needs today!
