The SmartItem™ on SailPoint Certification Exams

Ever wonder if the SmartItem actually delivers on its promises? SailPoint testifies that when used on two of their certification exams, the SmartItem provides benefits such as stellar protection, reduced costs, psychometric quality, and more appropriate candidate preparation.

The SailPoint Certification Program

SailPoint is a leader in identity governance, which plays a critical role in modern enterprise security. One of SailPoint’s core values is “Innovation,” or developing creative solutions to real challenges. SailPoint teamed up with Caveon to do just that. In this brief article, we describe the joint, successful implementation of the new testing technology “the SmartItem” for the first two SailPoint certification exams.

SailPoint certification program staff had been aware that IT certification programs can have their exams stolen within days of being released. Those exams end up on braindump sites, making test scores from those exams useless for certification decisions. SailPoint wanted to ensure that any certification exams bearing its imprimatur would be as secure as available testing technology would permit. Therefore, in its first certification program ever, SailPoint decided to use the Caveon SmartItem for 100% of their first two exams.

The SmartItem: Invulnerable to Cheating and Theft

The SmartItem is unique in that each one renders in thousands or even millions of ways, providing a different but fair experience for each test taker. The SmartItem is invulnerable to theft and cheating, relieving SailPoint of the same security concerns that have plagued every other IT certification program for the past three decades.

Utilizing SmartItem technology brought at least two other bonus advantages to SailPoint. The first is that SmartItems motivate proper preparation. Because braindump material is unavailable and multiple-choice test-taking skills are irrelevant, test takers need to study and gain experience on SailPoint technologies in order to pass the exam. The second advantage is in cost savings. Because SmartItems can’t be compromised, they will only need to be replaced because of legitimate performance issues or inevitable content updates, not because a question or even an entire form was stolen.

Of course, the biggest benefit of utilizing SmartItem technology is that test scores can be trusted, forever, for certification decisions. With each test administered, SailPoint can be confident that candidates who pass the test are competent in the skills required by SailPoint. As long as the certification program is in place, the exams’ test scores can be used to make reliable certification decisions.

Lessons Learned from Utilizing SmartItem Technology

The process of creating and publishing certification exams is an extensive one, and one that needed some modification for SmartItems. Since the first two exams have been published and made available to candidates, what has SailPoint learned?

Development

• We learned that SmartItems can be congruent with and built to very complex SailPoint objectives.
• We learned that the most important SME task isn’t to write items, but to design SmartItems to completely cover the objectives.
• We learned that SailPoint SMEs, working in small teams, were able to utilize Caveon coders and technology to create SmartItems efficiently.
• We learned that SailPoint SMEs, who were all novice item writers, enjoyed writing SmartItems. Because SmartItems afforded more creative options for covering objectives (for instance, enabling as many correct options and incorrect options as needed to cover a given item), item authors could follow the subject matter wherever it took them. This was fun!
• We learned that the SmartItem authoring tools would need correspondingly functional item review capability to enable authors to more quickly yet fully check item content and design.
• We learned that, despite the newness of the SmartItem approach, it took about the same amount of overall time and effort to create SmartItems as it would have taken to create traditional items.

Field Testing

A field test of the SmartItems (also sometimes called a “beta test”) was conducted to see if SmartItems performed well psychometrically ahead of exam release.

• We learned that, using established psychometric standards, SmartItems perform as well as or better than traditional items.
• We learned that SmartItems contribute to the high reliability of SailPoint exams.
• We learned that SmartItems contribute to the gathering of evidence for the valid use of test scores for SailPoint exams.
• We learned that we could create pass/fail cutscores easily for tests using SmartItems.
• We learned, in general, that there is no psychometric reason not to use SmartItems in exams.

Conclusion

In conclusion, we learned that the promised benefits of SmartItem technology—including stellar protection, reduced costs, psychometric quality, and more appropriate candidate preparation—are confirmed.

Because of the very positive experiences from the first two certification exams, a third exam using SmartItems has since been developed.