Recent years have shown a surge in the use of Markov Chain Monte Carlo (MCMC) algorithms for fitting item response theory (IRT) models in psychometrics, especially in complex models such as multidimensional IRT and cognitive diagnosis models, where standard methods such as maximum likelihood estimation are difficult to apply. In these applications slow mixing is often a severe problem, limiting the ability of researchers to draw valid inferences from the MCMC output. The goal of this project is to apply recent advances in MCMC methodology to develop fast mixing MCMC algorithms for complex IRT models. Specifically, this proposal will explore three approaches for improving mixing; 1) scale and centering approaches including Meng and van Dyk?s conditional and marginal augmentations and Yu and Meng? ancillarity and sufficiency interweaving strategy; 2) methods making use of gradient information such as Langevin MCMC and Hamiltonian methods and; 3) adaptive MCMC methods including both internal and external methods of adaptation ? the latter more generally subsume simulated tempering algorithms. The proposed research would extend the analytical reach of advanced MCMC methods in the psychometric literature, focus the attention of the psychometric community on MCMC mixing, and provide the basis for effective MCMC implementation in complex models. The proposed project would also include the development of free, open-source software with which to implement the proposed methods. The project would also include applied research on psychometric data from two projects to illustrate the new methods and software.
Intellectual Merit
While MCMC has become widely used in psychometrics and IRT, almost all applications to date have applied simple Gibbs or Metropolis-Hastings within Gibbs, the later often requiring fine-tuning of proposal distributions within the main algorithm. Applications have focused on what Geyer would describe as ?toy problems? that do not reflect the true complexity of implementations that occur in real educational settings. Even these ?toy problems? can exhibit slow mixing that compromises valid inferences. This research would focus on a critical problem in real world applications of MCMC in the psychometric literature and propose methods for addressing this problem. To date, implementations of fast-mixing MCMC methods are negligible.
Broader Impact
Item response theory methods have a profound importance in educational evaluation and research. The high stakes nature of educational assessments demands the most efficient and valid estimation procedures. Nonconvergence of any estimation procedure, including MCMC, can lead to incorrect calibration of item parameters, which in turn leads to incorrect ranking of individuals. Moreover, IRT methods are becoming more important in non-educational settings – e.g. in medical settings such as patient reported outcomes. In all these settings complex models requiring MCMC estimation are being applied more frequently.
Funding:
National Science Foundation
Funding Period:
09/15/2012 to 08/31/2016