Risk Management: Calculating the Bottom Line of Developing a Certification or Licensure Exam
Originally Published in Certification Magazine, 2/2005
Do you know your bottom line? What’s the amount you invested in the creation of your program’s certification or licensure exam? It is important to calculate this cost so that you can determine the amount that should be spent to protect the investment. If you don’t protect your investment and your exam content is stolen, you also need to be able to document the cost of development if you want to seek damages. Let me walk you through the key exam development phases, what costs should be included along with a few estimates of the actual cost. For purposes of the present article, all figures are based on the following assumptions about the exam:
- High stakes certification or licensure exam
- New exam, not a replacement
- Standard exam development processes are used
- Multiple choice question formats are used
- Initial release of the exam meets industry standards for reliability and validity
Content Analysis: The first step in the creation of a new certification or licensure program is to conduct a job analysis to determine the knowledge, skills and abilities that should be measured. The key factors that impact cost are as follows:
- Size of the domain to be measured (narrow vs. complex)
- Newness of the domain (new vs. well-established job roles)
- Availability of people that actually perform the job role (also known as subject matter experts)
If the domain is very narrowly defined and many people have been practicing in the field for a number of years, a project of this nature may cost as little as $25,000. The price continues to rise to $100,000 or more as the complexity of the domain increases. Once the job analysis is complete, however, it can serve as the basis for a certification or licensure program for many years. In some cases where multiple exams are required to complete a program, individual exam blueprints may also need to be developed at an additional cost of $5,000 per exam or higher.
Item Authoring: Once the content domain is defined, it’s time to create the raw test questions. Subject matter experts (SMEs) are either hired to perform this task or volunteers are recruited. In the case of hired SMEs they often also receive training on how to write good test questions. If volunteer SMEs are utilized, then professional test developers are paid to transform the questions. Either way, you can estimate that the cost per item is $600 per question or higher during this phase of development. If more advanced question formats are used, the cost increases.
Item Review: Now it’s time to bring in a new set of SMEs and your editing team to verify the accuracy and clarity of the test questions. You can estimate another $100 -$150 per test question for this phase of development.
Field Test Administration: The test questions are complete. Now it’s time to determine which questions will stand up to rigorous statistical analyses to make the final cut for the exam. Depending on the program, SMEs may or may not be compensated for their role as a “beta” examinee during this phase. You also need to include the cost of administering the exam. For a computer based (CBT) exam, this fee would likely be based on the number of testing events. In the case of a paper and pencil (PnP) exam, the fee would include distribution and test administration costs.
Exam Production and Release: Whether CBT or PnP, there are costs associated with putting the exam into operational or production mode and disseminating it per program specifications. This cost varies depending on a variety of factors such as the number of test questions, the complexity of the questions and the test format (fixed form vs. computer adaptive).
Overhead Costs: Don’t forget to include the cost of the program staff in calculating your bottom line. Estimate the time spent by each staff member on exam development and prorate accordingly. Other costs might include copyright process, advertising and legal fees.
What does our calculator show? Bottom line, industry experts estimate that the average cost for a multiple choice test question these days is around $1,000.
Now you know how expensive it is to develop an exam (from a low of $80,000 to hundreds of thousands), You also need to consider the projected revenue from the exam, estimated shelf life of the exam and the cost of redevelopment of content if a breach occurs. You make the call. What are you willing to pay to protect your exam content?