Prior Cognitive Task Analysis of Test Item Difficulty

Abstract

The pass rate of a test item often serves as an index of its difficulty. This posterior index, though easy to obtain and definite, is susceptible to students' learning level. It cannot play a leading role in designing test items. The present research tried to design a prior-to-test index of test item difficulty. According to the principles of cognitive task analysis, a frame and a procedure to be strictly executed were mapped out for the prior assessment of higher mathematics test items. Taking into account the characteristics of math problems, we designed such indexes as number of elements in a problem, element identification difficulty, number of principles used in answering, principle identification difficulty and cognitive load. The results show that prior difficulty is most significantly correlated with the pass rates of math test items. High correlation also exists among sub-indexes in assessment and among evaluators, indicating sufficient validity and reliability of the prior assessment method developed by the present reseach.