Center for Social Information Sciences (CSIS) Seminar

Abstract: We present a new method that is capable of handling both missing and suppressed value problems for large multiscale data sets, such as the Quarterly Census of Employment and Wages (QCEW) from the U.S. Bureau of Labor Statistics. Existing multiple imputation methods are hard to scale for such data sets. This particularly acute in the case of QCEW, with as many as 1.5 billion observations aggregated along three different scales (industry structure, geographic levels, and time). Our method incorporates three innovations. First, we improve the accuracy of the Bootstrapping-based Expectation Maximization method (King et al. 2010), a state-of-the-art multiple imputation method, by utilizing the extra information from the singular covariance matrix and taking into account of the multiscale data structure. Second, we introduce a quasi-Monte Carlo technique to accelerate convergence. Third, we develop a parallel sequential approach that partitions the large data set into quasi-independent small data sets according to the data structure and patterns of suppressed and missing observations. We demonstrate that our new method improves speed and accuracy. Moreover, it can be applied to large data sets with complicated multiscale structures.