Caveon is the world’s first and only company dedicated entirely to the security of exams. An interesting professional service we offer is Secret Shopping, like a version of penetration testing for the proctoring systems that testing programs utilize to keep their test administration secure.
In testing, ‘secret shoppers’ are brought in at the request of our clients to evaluate the efficacy of proctors and the exam administration protocols they have in place. During a Secret Shopping engagement, a trained evaluator books a real exam slot with a client and behaves like an ordinary candidate while probing for gaps in live security. Their mission is simple: behave like an ordinary candidate while attempting the kind of low-tech violations that any determined test-taker could try. For example, they use their phones, sneak in earbuds or smart watches, open notes, dig in their bags, or leave prohibited items on their desks, etc. Throughout the exam, they log whether or not these attempts are caught by the exam proctor, and if so, which remedial actions are taken.
Clients hire Caveon to stress-test their own proctoring solutions to ensure maximum efficacy. The goal is not to embarrass proctors—most are diligent, polite, and doing their best—but to test whether the security measures that programs rely on actually work.
Shockingly, across more than 100 secret shopping engagements in the professional certification and licensure space, across multiple states, regions, and countries, one pattern dominates: proctors—whether in person or online—miss the majority of visible cheating attempts, and when they do notice, the session often continues after a warning to the test-taker.
Caveon shared these aggregate results from past secret shop missions at the 2024 conference of the National Council on Measurement in Education. The numbers confirm that these incidents are not outliers but the norm.
These results underline how easily everyday tactics slip past proctors.
“I wasn’t comfortable breaking rules, but I was able to use my phone, exit the room, and even record the testing screen,” said one Caveon secret shopper (who, due to the sensitive nature of the work, will remain anonymous). “Proctors may deter most people from doing these things for fear of getting caught, but if the motivation is there, if you’re just a little bit bold, you can get away with anything.”
Proctors may deter most people from doing these things for fear of getting caught, but if the motivation is there, if you’re just a little bit bold, you can get away with anything.
Even when proctors are well trained and diligent, six recurring factors open gaps large enough for routine cheating to slip through.
Time pressure narrows focus.
Admission queues, software glitches, and candidate questions pull attention away from the live feed or testing floor at the very moments a determined cheater needs.
Multitasking creates blind corners.
Remote proctors juggle multiple video windows and chat logs. Test-center staff escort test-takers to breaks while watching several screens. Each added duty widens the interval in which misconduct can occur undetected.
Searches and scans rarely catch hidden items.
Desk sweeps miss what sits just outside the webcam’s field of vision. Wand scans miss devices taped below the knee. As covert tech keeps shrinking, the gap between what is searched and what is actually present keeps widening.
Warnings replace consequences.
Terminating an exam triggers paperwork, refunds, and heated disputes. A quick verbal caution feels easier, and most detected violations end that way.
Courtesy limits confrontation.
Proctors are trained to be helpful and respectful. That professionalism can translate into hesitation when decisive action (confiscating a phone, ending a session) is required.
Proctors can’t see all threats.
And most importantly, they can’t see some of the most damaging misconduct, which have no visual cues at all: pre-knowledge of stolen items, remote proxy test-takers, AI answer tools running on a second device. Even the most vigilant proctor cannot detect what is happening off-camera, in another room, or inside a candidate’s memory.
Because these fault lines play out in almost every session, Caveon’s secret shoppers routinely pass nine out of ten visible cheating attempts. Had they been intent on real fraud, they could have walked away with high scores and high-value item banks, leaving the program exposed to costly score invalidations and reputational damage once stolen content surfaced online.
David Foster, Ph.D., Caveon’s founder and CEO has concerns over the lack of public awareness of this information, “While I am thrilled that remote proctoring has been able to facilitate distance-based testing in this current environment, I am simultaneously concerned by the widely held, yet false impression that proctoring is a security panacea. This simply isn’t the case.”
Caveon shares these narratives not to blame or shame proctors, most of whom work hard within the limits of their tools, but rather to show why those limits matter. Programs that depend on proctoring alone may feel secure, yet the evidence says otherwise. Layered defenses—secure exam design, real-time analytics, continuous forensics—offer a clearer path.
David Foster continues, “There are more effective methods of security available today—such as preventative item and test designs—than our outdated reliance on proctored testing. We need to reenvision the administrative role proctors play during exam administration and let them evolve into a role in which they will be just as needed, but so much more successful.”
Given the efficacy concerns churned up by this data, Caveon designed a solution to streamline the massive resources that testing programs pour into proctoring. Observer, the test administration monitoring solution from Caveon, eliminates the biggest cost drivers in traditional proctoring while creating a more streamlined, less-invasive experience for test-takers— all while improving upon the quality of security offered by the traditional proctoring paradigm.
While the same gaps in visible vs. invisible test fraud exist for Observer as in traditional proctoring, the combination of secure exam design with Observer’s powerful monitoring can eliminate these concerns altogether. That solution is called “Observer Plus."
