Putting tests to the test
Caveon providing an answer to administrator’s security needs
Deseret Morning News, Salt Lake City, Utah
By Brice Wallace
DRAPER — Caveon Test Security is a company:
A. Based in Utah.
B. Specializing in helping test administrators thwart cheating on standardized exams.
C. With experience poring over more than 50 million test records.
D. That has reviewed 5 million test records for a single customer.
E. All of the above.
The answer, as one might guess, is E.
The young company says it is the only one of its type in the world, using institutional knowledge and statistical savvy to provide customers who suspect test cheating with, well, answers.
“We have services for testing programs administering large-scale, high-stakes tests,” Chief Executive Officer Dave Foster said, mentioning the Graduate Record Exam, the College Board’s SAT and the ACT as examples.
“Almost everyone has taken a test, and with good security, the test scores that come out of them are something that can be trusted. If cheaters are effective, and we know they are — our data has shown that cheaters generally pass the tests and generally get the higher scores — then how much can you trust them?”
Test cheating has come a long way from crib notes scribbled on one’s palm, but crude variations nonetheless flourish. Don Sorensen, Caveon’s vice president of marketing, includes many in his list of 50 ways to cheat on tests, part of a marketing presentation. Rubber bands around the wrist, the inside labels of water bottles, the backs of neckties and even food wrappers can be employed to conceal test answers.
Sometimes, it’s not the test-takers instigating the trouble. Administrators have been known to help out students. Foster knows of one case in which a test prep organization sold answers to students.
“And the No Child Left Behind Act has put tremendous pressure on educators to measure up to get that federal funding,” he said.
But technology has opened new doors for both cheaters and those who steal and then sell test information. Some calculators can store answers. Wireless communications, photo-taking cell phones and ear buds can be used to gain advantages. Document scanners as small as a pen can copy entire tests, with the info often then made available on the Internet. In some situations, test-takers simply memorize an exam’s contents and either share or sell it online.
“All indications are it’s getting worse, and I don’t think it’s because we’re just getting more sensitive and attuned to it,” Foster said. “The technology to capture and share the information, those things have put solid programs, ones that have existed for 80 years, in jeopardy.
“It’s not just the cheating; it’s the theft. There’s a big business out there where someone can log in and capture information with a digital camera or some other way, then you put it out on the Internet and sell the content to other folks. There’s a huge business.”
For example, a test-taker might spend $29 buying questions for an information technology certification program test online, memorize the answers and pass the test.
“The worst that I’ve seen is in the IT certification world, these ‘brain dump’ sites. It takes not two weeks for the entire text of the test to be available at these brain dump sites for purchase. … Testing companies spend hundreds of thousands of dollars developing tests, and when it goes out and within a week it’s freely available, all that money is wasted,” Foster said.
“These kinds of threats are relatively new and sometimes they are not prepared to handle them. What they’ve been doing for decades has worked, and these are new issues for them.”
While acknowledging that pinpointing the extent of cheating is elusive, Foster pegs the figure at between 5 percent and 30 percent of people being tested. “And programs that are more lax with security would have more problems,” he added.
So, what’s a test program to do? Many have called upon Caveon for help, and the company offers three basic services.
One is the Caveon Security Audit, a 250-point evaluation of a testing program’s security practices and policies to check for strengths and weaknesses. Another is the Caveon Web Patrol, monitoring the Internet for the sale, barter or other disclosure of test information.
“If you went out on eBay now and searched for actual test questions, you’ll probably see several offerings,” Foster said.
Sure enough, last week’s auction-site bounty included a guide for an aircraft pilot certification test and questions and answers to tests for a commercial driver’s license and real estate exams from several states.
The third service is Caveon Data Forensics, statistical analyses of test results to check for patterns that indicate cheating. The company’s statistical models dictate how results should look, and the actual results are compared to the models. The analyses can detect possible cheating, test theft, a person taking a test on behalf of someone else, collusion or other shenanigans. Plain old guessing is taken into account in the models.
“If, say, 1,000 people take a test, we get their results — how did they answer every question and how long did it take them. Then we crunch all those raw data and we look for patterns of irregularity in the data,” Foster said.
“For example, if you’re a really smart person and you really know the content, you should be able to get all the easy questions right and start struggling with the more difficult ones. If you don’t know as much, you should get some of the easy ones right and then not get any of the difficult ones right. Those are normal patterns. But if we see a pattern where a person is missing some of the easy ones and getting some of the very difficult ones correct, we suspect they’re not really taking the test in a normal way that a person with a particular ability does.”
Time also is a key criterion. The company once discovered a person answering 60 questions correctly and doing so averaging less than five seconds per question on a test that usually took a normal person up to two minutes to answer each of the more difficult questions.
“You know by looking at the timing that this person had preknowledge,” Foster said. “They brought all the answers with them, recognized the questions and popped in all the answers.”
Collusion is indicated if people at the same test site answer questions exactly the same way, missing the same questions with the same answers. Someone could be coaching them, Foster said.
“We’re very conservative about when we spot a problem, but it’s very clear in areas where there is a problem,” Sorensen said. “We’ve reviewed 50 million test records now, and it’s amazing as you review that many records the patterns you do see.”
“We use a very conservative statistical threshold because we don’t want to cancel a score or take something from someone unless you’re sure,” Foster said. “We’re not gimmicky at all. We’re rooted in solid statistical theory. Our decisions are based on data and not in conjecture, and customers like that.”
Caveon’s reports to customers include recommendations based on the spotted problems. For instance, teachers administering tests might be replaced with uninterested observers.
“Statistical evidence isn’t strong evidence to support a legal action,” Foster said. “What you need is observation. So our measures don’t lend themselves to that kind of legal action, which is fine anyway. Prevention is the best way.”
On the Caveon Web site, Steve Moore, certification program manager for Sun Microsystems, says that Data Forensics is “giving us ‘red flags’ to help identify not only where unusual testing patterns are occurring, but who is involved.” The resulting evidence ensures that “decision-making can be based on real facts versus simple hunches about cheating, piracy and test center issues.”
Caveon audited South Carolina’s high school assessment program, which serves as the state’s high school exit exam. “We want to make certain it is fair,” Terry Siskind, assessment director for the South Carolina State Department of Education, says on the Web site.
A real test
But checking for fairness can be a daunting task. Caveon notes that about 20,000 North American testing programs administer nearly 125 million tests for qualification, selection and advancement of people in education and business.
That’s where its founders saw a need for improved test security. Veterans of the testing industry came together to form the company in early 2004 — the name coming from “caveo,” Latin for protection and security — and now the company has about 60 customers. They include boards, institutes and associations overseeing tests for surgeons, lawyers, Certified Public Accountants, nurses, hazardous-materials managers and others, including school districts conducting K-12 student assessments in 12 states.
The Institute for Hazardous Materials Management tests about 800 people a year. The state of Texas tests about 5 million students annually.
“That’s been probably the most pleasant surprise for us, that the models we use don’t have to be adapted a whole lot,” Foster said. “A little bit, but not much. We can deal with a test where only 100 people have taken it versus one where 5 million have taken it. The models don’t change much.”
Charges for the services vary from under $1,000 to analyze data for a lawsuit to “hundreds of thousands” for Texas, Foster said.
The company turned its first profit in 2006 on sales of about $2 million, and it is expecting to grow 50 percent to 80 percent during the next two or three years. That would be fine for many young companies, but Foster admits it has been slower than originally expected.
“Security is not a pleasant topic. Customers tend to come to us when a breach has occurred, not because they really want to work on security. They really want to work on it after a breach,” he said.
“By going to a security company, you acknowledge you have doubts about your own security. You have to get past that barrier. And there is a positive message in that, one that many of them are not used to saying: ‘Our procedures aren’t working. We need new procedures.”‘
“It’s kind of a momentum thing,” Sorensen said. “We knew from the beginning there would be time needed to educate the market. Because of the technology moving so quickly, it’s really moving to the forefront.”
But Foster believes Caveon has a head-start on possible competitors because many test security companies rely on old models — people or perhaps cameras watching test-takers. “I don’t think we’ll see anything in the way of serious competition within five years,” he said.
Caveon already has branched out a bit, forming a joint venture called Kryterion with Dutch educational testing company Cito and Toronto-based human resources recruitment company Drake International. Kryterion (www.kryteriononline.com) allows people to take tests securely via the Internet from any location. “After five or six questions, we know if it’s a valid test,” Foster said.
As for Caveon, Foster expects the company in five years to have hundreds of employees and revenues approaching $100 million. He expects it to serve international markets “where the problems are no less than what we have in the U.S.”
“We’ve just scratched the surface of our own technologies to fight cheating. These things we’ve come up with are somewhat crude compared to what we’ll have in five years,” Foster said.
“Anytime you’re in an industry where nothing has been done for a hundred years, there’s so much that can be done — with the Internet, with technology, with analyses. The world just opens up, and that’s what’s waiting for Caveon.”