Randomly Parallel TestS

It is generally agreed that criterion-referenced (CR) testing was introduced by Robert Glaser in 1963.  In its purest form, as defined by Glaser, CR testing should “provide information as to the degree of competence attained by a particular student which is independent of reference to the performance of others” (p. 520).  In other words, the score obtained through CR testing should represent a measure of the student’s mastery of the content area, and moreover, CR testing differentiates itself from norm-referenced (NR) testing in that the interpretation of the score obtained by the individual should be independent of the interpretation of scores obtained by others.      

Meanwhile, Frederic Lord had been exploring ways to calculate test reliability, including the underlying assumptions about error and “parallel” test forms, as early as 1953.   Lord formally introduced the concept of “randomly parallel tests” in 1955, defining RP tests simply in this way: “Suppose that a large number of forms from the same test are administered to the same group of examinees, each form consisting of a random sample of items drawn from a common population of items… Test forms constructed [in this way] will be called randomly parallel forms or randomly parallel tests” (1955a, p. 1).

In 1968, H.G. Osburn introduced the idea of “universe-defined” tests, where items are randomly sampled from a well-defined universe of items, such that the score produced by this method “provides an unbiased estimate of [the test taker’s] score on some explicitly defined universe of item content” (p. 96).  Eventually, researchers began replacing the term “universe-defined” with “domain-referenced.”  

There is abundant agreement on the definition of DR testing provided in the literature. Domain-referenced tests are generally agreed to be tests where items are randomly sampled from well-defined domains of items to produce a test (or form) for each examinee, which is considered “randomly parallel,” and the score obtained from a DR test form represents an unbiased estimate of the proportion of the domain of items that would be answered correctly. 

Moreover, in almost every paper, the condition of a large and well-defined domain, or universe, of items is specified.  It seems as if this cannot be emphasized enough, that the domain, and domain strata, be clearly defined and large enough to cover the target domain.  In Brennan and Kane (1977), the authors reiterate that the universe of items should be well-defined “so that we may reasonably consider drawing random samples from it” (p. 278).  This is critical to ensure that the scores obtained from the random sampling of items can be generalized to the entire domain. 

The relationship between DR testing and RP tests is almost inextricable. The fundamental principle of the DR test design is that one creates an extensive, well-defined content domain from which items can be randomly sampled for each person’s test, whereas RP testing could be said to represent the procedure for randomly sampling those items to produce “parallel,” or equivalent, sets of items (or forms) to be administered to the test takers.    

Researchers at the time attempted to address questions around the appropriate statistical approach for estimating reliability (or dependability, as Brennan and Kane prefer to call it) and validity for DR tests.  Lord (1955a) proposes using the KR-21 formula for calculating test reliability, whereas Kane and Brennan (1980) follow the footsteps of Cronbach in their use of Generalizability Theory as the basis of their dependability calculations.  Several processes are developed and presented, indicating there are appropriate options for developing the statistical models for these tests.

There also is debate about whether classical empirical item analysis is necessary for items randomly sampled from a large, well-defined domain.  Some authors state definitively that item analysis is not necessary.  Millman (1974) states emphatically, “No!” item difficulty and discrimination should not factor into the selection of items for RP tests, because the use of item statistics “destroys the random selection process” (p. 339) and diminishes the interpretability of the test score with respect to the domain.  However, Millman goes on to say that item statistics may be used for removal of faulty items or to inform revisions to the domain.  Continued discussion and research on this topic is expected. It is our recommendation that it is best not to expect to have available statistics on individual items. This is because today’s technology can produce pool sizes for domains that are effectively infinite in size, precluding the need to collect data on each item to calculate item statistics.

It is not surprising that concerns about the security of test items and cheating are not frequently discussed in the literature from this time, because test fraud, though present, was simpler and less advanced than it is today.  Interestingly, while cheating and theft have been enabled by advances in technology over the last 60 years, testing professionals have failed to take advantage of the very same technology to prevent, deter, and/or detect test fraud.  We are still designing and administering tests in much the same way we did 100 years ago, yet the world around us has completely changed.  Some authors do allude to security concerns, especially with respect to item overuse and disclosure. The ability of DR tests to solve certain security problems is discussed in Hively (1968).  Furthermore, in Hively (1974), he makes the damning statement about NR testing: “Secrecy is the hallmark of NRT” (p. 13). This leads the reader to a natural follow-up question: What if I didn’t have to keep my items secret anymore?         

Another theme frequently implied or openly discussed in the papers is the limitations of technology at the time and the practical feasibility of creating enough high-quality items and randomly sampling those items from huge domains.  Indeed, several of the papers that propose specific technology, such as computer programs, might seem outdated today, since today we have the technology to auto-generate many high-quality items, randomly sample them, and produce a unique test form for every test taker.  In some cases, it may just be interesting to read these papers to understand the challenges facing researchers and professionals at the time.  This can help bring perspective to the reader and illustrate why DR testing and RP tests were not readily implemented at the time. 

In other cases, these papers may help to illuminate a modern analog that researchers and professionals face today: How did they respond to the availability of computers and advancing technology at the time to improve test design and administration?  Likewise, how might we use lessons learned from these past experiences to approach the opportunities provided by AI today?   

It is apparent from all of these papers that there are significant benefits to DR testing and RP tests, making this design well worth investigation and consideration.  As pointed out in many of these papers, the benefits include: 

• Using DR testing and RP tests, with technology, you can create the large numbers of items needed, in some cases an effectively infinite pool of items, where each test taker receives a unique form;
• This allows for the use of sample items or practice tests that are mirror copies of the high-stakes exam, where test takers can take and retake the test as many times as necessary without encountering the same items. This also means that item disclosure ceases to be a concern;
• The scores obtained provide unbiased estimates of the test takers’ capability on the entire domain of items; and
• Given that the scores are generalizable to the domain, and a test taker can take the test many times, the DR/RP test design can be used to track student progress over time and inform/improve future instruction, making this design particularly beneficial in educational assessment.

At Caveon we frequently hear our clients complain that their content is being disclosed at a rate faster than it can be created. Our clients pour their valuable resources into carefully concealing their content and items, and when their items are eventually disclosed, they pour more resources into containing and investigating the breach. Testing programs are forced to retire items and test forms early due to disclosure and to create more new, costly items, which themselves are destined to be disclosed presently after publication. 

The DR/RP test design has the potential to solve many of our biggest problems today, including most forms of test fraud and cheating.  We have the technology to fully implement this, and most of the roadblocks that were present in the past have been cleared.  Most forms of test fraud rely entirely on the basis that the items on traditionally constructed tests today are fixed and repeatedly exposed, which brings us back to the question posed previously:   What if your items didn’t have to be secret anymore?