How big is too big: a case study using keyword text analysis and unsupervised machine learning to manage large search results

Walker, Jennifer S.; Renner, Barbara Rochen; Cawley, Michelle

Download PDF

Request Version for Screen Reader

Last Modified

July 13, 2020

Creator

Walker, Jennifer S.
- ORCID: http://orcid.org/0000-0003-0248-2616
- Affiliation: University of North Carolina at Chapel Hill. Health Sciences Library
Renner, Barbara Rochen
- ORCID: https://orcid.org/0000-0002-3505-6341
- Affiliation: University of North Carolina at Chapel Hill. Health Sciences Library
Cawley, Michelle
- ORCID: https://orcid.org/0000-0002-7178-328X
- Affiliation: University of North Carolina at Chapel Hill. Health Sciences Library

Abstract

Objectives: To employ the use of a data-driven technique to refine a complicated search strategy by analyzing contributions of individual keywords. To prioritize results from a comprehensive, complex search on a broad topic using machine learning. Methods: We developed a comprehensive search strategy on a diffuse public health subject that resulted in more than 50,000 results from PubMed. To refine the search strategy, we conducted keyword analysis that indicated the relative contribution by individual keywords and informed our decision-making around terms to exclude from the final search strategy. Using this method we revised the search strategy by eliminating several keywords. After executing the final search, we used unsupervised machine learning or clustering to prioritize a subset of studies for manual review. Specifically, we used a clustering algorithm that groups like references based on text similarities in titles and abstracts. For each cluster, the software produces a set of keywords or a topic “signature”. We used these topic signatures to select clusters to prioritize for manual review. Results: Our initial search in PubMed yielded approximately 50,000 results. Keyword analysis informed our decisions to eliminate several terms that were causing excessive “noise”. The revised search was reduced to approximately 15,000 records. These records were clustered using unsupervised machine learning and we prioritized approximately 1,400 studies for manual review. Conclusions: Keyword analysis informs searchers of potentially irrelevant terminology in their search strategies and is useful in shining a light on a set of results at any stage of developing a search strategy. Unsupervised machine learning or clustering assists research groups in prioritizing a subset of literature from a large set of results and is particularly useful for broad, comprehensive literature reviews. Our initial results from manual screening indicate a higher degree of precision than would be typically expected from such a large set of search results.

Date of publication