| dc.description.abstract | Many clinical natural language processing (NLP) methods rely on non-contextual or contextual word embedding models. Yet, few intrinsic evaluation benchmarks exist comparing embedding model representations against human judgment. Moreover, it is unclear if the previously described NLP model discordance between intrinsic and extrinsic evaluation performance persists among novel embedding models. We present a framework used to develop two new intrinsic evaluation tasks: term pair similarity to assess non-contextual word embeddings and cloze task accuracy for contextual word embedding models. Using surveys, we quantified the agreement between model representations and clinician judgment. We also created two clinical phenotyping tasks using binary and multi-label classification with varying class balance for extrinsic evaluation. Findings from each of our intrinsic evaluation tasks reveal that models pre-trained on general biomedical and clinical corpora performed as well as, if not better than, models pre-trained on an in-domain clinical corpus; models trained on a general English Wikipedia corpus performed worst. Joint spherical embeddings gave overly optimistic representations of term pair similarity and the gains seen on general NLP intrinsic evaluations tasks failed to translate in this study. The results of our extrinsic evaluation tasks demonstrated that neural network-based models - Bidirectional Long Short Term Memory, Transformer, and Efficient Transformer models - vastly outperform regression models. Efficient Transformer models as a class tended to provide the best classification performance. Contextual word embedding model performance was improved by pre-training on an in-domain corpus. By contrast, models featuring word2vec and fastText non-contextual embeddings yielded equivalent phenotype classification performance regardless of the corpus source. Overall, intrinsic evaluation performance failed to correlate with extrinsic evaluation findings on study of both non-contextual and contextual embedding models. | |
