PHILADELPHIA— Machine learning can be used to comb through online reviews of substance use treatment facilities to home in on qualities that are important to patients but remain hard to capture via formal means, such as surveys, researchers from the Perelman School of Medicine at the University of Pennsylvania show. The researchers found that professionalism and staff dedication to patients were two of the top qualities that could be attributed to either a negative or positive review of the facility. Findings from this study were published today in the Journal of General Internal Medicine.
“Searching for – and connecting with – therapy can be very difficult and confusing. Many individuals start their search online, where they are likely to see an online review accompanying other information about a treatment facility,” said the study’s lead author, Anish Agarwal, MD, a clinical innovation manager in the Penn Medicine Center for Digital Health and an assistant professor of Emergency Medicine. “These online reviews can provide commentary on what is driving positive, or negative, patient experiences throughout recovery, but they must be accurately identified. Through machine learning, we’ve shown that this is possible, and we hope such findings can be used to improve patient-centered addiction care.”
Currently, there are no nationwide measures of quality to evaluate and compare facilities that treat substance use disorder. In the past, Agarwal – along with this study’s co-author, Sharath Guntuku, PhD, a researcher in the Center for Digital Health and an assistant professor of Computer Science, and senior author Raina Merchant, MD, the director of the Center for Digital Health – analyzed reviews on Google and Yelp to see whether a national survey offered by the Substance Abuse and Mental Health Services Administration (SAMHSA) to inventory services being offered was also able to gauge patient satisfaction (it largely didn’t). So the team set out to use a similar technique to gauge what drives positive or negative experiences with facilities through the unfiltered lens of Yelp.
“We felt that this would provide a great deal of insight into the patient experience,” Guntuku said. “Tapping into user-generated reviews offers a way to understand their narrative.”
To accomplish this, the researchers pulled reviews from SAMSHA-recognized facilities that had been reviewed at least five times. This amounted to more than 500 facilities across the United States. Text of the reviews were then run through a natural language processing algorithm powered by machine learning. Through this, topics were identified and collected from within the reviews. The researchers then were able to categorize them thematically.
Overall, the researchers classified 16 recurring themes in reviews. When it came to the positive reviews, the top themes were “long-term recovery,” “dedicated staff,” and “dedication to patients.” The top three themes found in negative reviews were “professionalism,” “phone communication,” and “overall communication.”
The fact that the top themes for both positive and negative reviews had to do with the conduct and commitment of facility staff does not come as a great surprise.
“Dedication and professionalism are critical to the recovery experience,” Agarwal said. “Having trusted and approachable staff who care about individuals is the crux of all health care, but it is likely underscored in addiction.”
Other top themes on the positive side were “group therapy experience” and “inpatient rehabilitation,” while “wait times in facility” and “management” rounded out the top five most prevalent themes among negative reviews.
“We are still relatively early in this research, but we’re getting a direct look into some core values that substance use treatment facilities could use to guide and improve their treatment,” Merchant said. “This feedback is tremendously valuable, and we’re showing that it can be distilled effectively into key themes through the use of machine learning.”
Agarwal said he and the other researchers are hoping to explore more ways that patients, families, and their support networks think about substance use treatment and health care as a whole.
“In today's digital world, there is a robust and enormous amount of interconnectedness which we can harness to drive forward quality care and support health systems in learning from their patients,” Agarwal said.
Other authors on this study include Zachary Meisel, Arthur Pelullo, and Bill Kinkle.
Funding was provided by the National Institutes of Health (grant number NIH NIDA 1R21DA050761).
Penn Medicine is one of the world’s leading academic medical centers, dedicated to the related missions of medical education, biomedical research, excellence in patient care, and community service. The organization consists of the University of Pennsylvania Health System and Penn’s Raymond and Ruth Perelman School of Medicine, founded in 1765 as the nation’s first medical school.
The Perelman School of Medicine is consistently among the nation's top recipients of funding from the National Institutes of Health, with $550 million awarded in the 2022 fiscal year. Home to a proud history of “firsts” in medicine, Penn Medicine teams have pioneered discoveries and innovations that have shaped modern medicine, including recent breakthroughs such as CAR T cell therapy for cancer and the mRNA technology used in COVID-19 vaccines.
The University of Pennsylvania Health System’s patient care facilities stretch from the Susquehanna River in Pennsylvania to the New Jersey shore. These include the Hospital of the University of Pennsylvania, Penn Presbyterian Medical Center, Chester County Hospital, Lancaster General Health, Penn Medicine Princeton Health, and Pennsylvania Hospital—the nation’s first hospital, founded in 1751. Additional facilities and enterprises include Good Shepherd Penn Partners, Penn Medicine at Home, Lancaster Behavioral Health Hospital, and Princeton House Behavioral Health, among others.
Penn Medicine is an $11.1 billion enterprise powered by more than 49,000 talented faculty and staff.