The Measurement Model and Reliability

Ch. 4. P. 66–89.

Pemstein D., Marquardt K., Seim B., Tzelgov E., Wang Y., Miri F.

The Varieties of Democracy (V-Dem) project relies on country experts who code a host of ordinal variables, providing subjective ratings of latent|that is, not directly observable regime characteristics over time. Sets of around ve experts rate each case (country-year observation), and each of these raters works independently. Since raters may diverge in their coding because of either differences of opinion or mistakes, we require systematic tools with which to model these patterns of disagreement. These tools allow us to aggregate ratings into point estimates of latent concepts and quantify our uncertainty around these point estimates. In this chapter we describe item response theory models that can that account and adjust for differential item functioning (i.e. differences in how experts apply ordinal scales to cases) and variation in rater reliability (i.e. random error). We also discuss key challenges specic to applying item response theory to expert-coded cross-national panel data, explain the approaches that we use to address these challenges, highlight potential problems with our current framework, and describe long-term plans for improving our models and estimates. Finally, we provide an overview of the different forms in which we present model output.

Keywords: Bayesian methods, IRT models, expert-coded data, Varieties of Democracy

In book

Varieties of Democracy: Measuring Two Centuries of Political Change

Coppedge M., Gerring J., Glynn A., Knutsen C. H., Lindberg S. I., Pemstein D., Seim B., Skaaning S., Teorell J. Cambridge University Press, 2020.