Evaluating the patient-centered automated SMS tagging engine (PASTE): natural language processing applied to patient-generated SMS text messages

Abstract

Pilot studies have demonstrated the feasibility of using mobile technologies as a platform for electronic patient-centered medication management. Such tools may be used to intercept drug interactions, stop unintentional medication overdoses, prevent improper scheduling of medications, and to gather real-time data about symptoms, outcomes, and activities of daily living. Unprompted text-message communication with patients using natural language could engage patients in their healthcare but presents unique natural language processing (NLP) challenges. A major technical challenge is to process text messages and output an unambiguous, computable format that can be used by a subsequent medication management system. NLP challenges unique to text message communication include common use of ad hoc abbreviations, acronyms, phonetic lingoes, improper auto-spell correction, and lack of formal punctuation. While models exist for text message normalization, including dictionary substitution and statistical machine translation approaches, we are not aware of any publications that describe an approach specific to patient text messages or to text messages in the domain of medicine. To allow two-way interaction with patients using mobile phone-based short message service (SMS) technology, we developed the Patient-centered Automated SMS Tagging Engine (PASTE). The PASTE webservice uses NLP methods, custom lexicons, and existing knowledge sources, to extract and tag medication concepts and action concepts from patient-generated text messages. A pilot evaluation of PASTE using 130 medication messages anonymously submitted by 16 volunteers established the feasibility of extracting medication information from patient-generated medication messages and suggested improvements. A subsequent evaluation study using 700 patient-generated text messages from 14 teens and 5 adults demonstrated improved performance from the pilot version of PASTE, with F-measures over 90% for medication concepts and medication action concepts when compared to manually tagged messages. We report on recall and precision of PASTE for extracting and tagging medication information from patient messages.