Implementing machine learning methods with complex survey data: Lessons learned on the impacts of accounting sampling weights in gradient boosting

Public Deposited
Analytics

Downloadable Content

Download PDF
Request Version for Screen Reader
Creator
  • MacNell N.
    • Other Affiliation: Social & Scientific Systems
  • Feinstein L.
    • Gillings School of Global Public Health, Department of Epidemiology
  • Wilkerson J.
  • Salo P.M.
    • Other Affiliation: National Institutes of Health
  • Molsberry S.A.
    • Other Affiliation: Social & Scientific Systems
  • Fessler M.B.
    • Other Affiliation: National Institutes of Health
  • Thorne P.S.
    • Other Affiliation: University of Iowa
  • Motsinger-Reif A.A.
    • Other Affiliation: National Institutes of Health
  • Zeldin D.C.
    • Other Affiliation: National Institutes of Health
Abstract
  • Despite the prominent use of complex survey data and the growing popularity of machine learning methods in epidemiologic research, few machine learning software implementations offer options for handling complex samples. A major challenge impeding the broader incorporation of machine learning into epidemiologic research is incomplete guidance for analyzing complex survey data, including the importance of sampling weights for valid prediction in target populations. Using data from 15, 820 participants in the 1988-1994 National Health and Nutrition Examination Survey cohort, we determined whether ignoring weights in gradient boosting models of all-cause mortality affected prediction, as measured by the F1 score and corresponding 95% confidence intervals. In simulations, we additionally assessed the impact of sample size, weight variability, predictor strength, and model dimensionality. In the National Health and Nutrition Examination Survey data, unweighted model performance was inflated compared to the weighted model (F1 score 81.9% [95% confidence interval: 81.2%, 82.7%] vs 77.4% [95% confidence interval: 76.1%, 78.6%]). However, the error was mitigated if the F1 score was subsequently recalculated with observed outcomes from the weighted dataset (F1: 77.0%; 95% confidence interval: 75.7%, 78.4%). In simulations, this finding held in the largest sample size (N = 10,000) under all analytic conditions assessed. For sample sizes <5,000, sampling weights had little impact in simulations that more closely resembled a simple random sample (low weight variability) or in models with strong predictors, but findings were inconsistent under other analytic scenarios. Failing to account for sampling weights in gradient boosting models may limit generalizability for data from complex surveys, dependent on sample size and other analytic properties. In the absence of software for configuring weighted algorithms, post-hoc re-calculations of unweighted model performance using weighted observed outcomes may more accurately reflect model prediction in target populations than ignoring weights entirely.
Date of publication
Keyword
DOI
Identifier
Resource type
  • Article
License
  • CC0 1.0 Universal
Journal title
  • PLoS ONE
Journal volume
  • 18
Journal issue
  • 1-Jan
Language
  • English
Version
  • Publisher
ISSN
  • 1932-6203
Publisher
  • Public Library of Science

Relations

Parents:

This work has no parents.

In Collection:

Items

Thumbnail Title Date Uploaded Visibility Actions
file details journal.pone.0280387.pdf 2023-02-02 Public Download