There is a limit to the number questions we can ask as candidates eventually tire of more questions and the added return in extra reliability, must therefore be balanced by the time it takes to collect the information (depth versus breadth). It has been argued that giving strong candidates lots of easy items provides limited information gain and is a waste of time. More significantly giving weaker candidates lots of difficult items is not a good use of time and is discouraging. In each case we would be better off ensuring the tests are tailored to the ability levels of the candidates. The onset of adaptive testing promises to enable assessors to estimate a candidate’s maximum performance (plateau) more quickly. In addition it will ensure a more user friendly testing experience for all candidates. Technically this means we maximise the information gain from each question. Adaptive testing relies on Item Response Theory (IRT) and modern computer technology which can adapt the test as the candidate completes it. IRT has long been used to identify ambiguous items (where stronger candidates so less well on an item) however it is now being used to adapt the test in as the candidate is sitting it using real time scoring.
Adaptive testing is concerned with distinguishing how a “keyed response” for an item is influenced by error factors (e.g. distortion, ambiguity, difficulty) and their true ability or traits. Adaptive testing means that each candidate can experience a test matched to their ability levels. We can quickly fast track candidates who are doing well to more difficult items and ensure candidates performing less well get items most suited to their level of ability. Consider the following item analysis. A “1” indicates an item was answered correctly.
|Item 1||Item 2||Item 3||Item 4||Item 5||Raw ScoreAverage|
In the table above Pat is clearly doing well. He has an average score of 0.8 on the first five items in the test. David has only got two out of the first five items correct (It is common in test design that the easiest items are at the beginning).
While adaptive testing has its limitations, a good candidate may miss an easy item and may lose time proving they should progress to a higher plateau it has great attractions for test publishers and many test buyers.