Daniel Wolfe on the Dangers of Algorithmic Prescriptions

Should an algorithm make your healthcare decisions? With many decades of experience advocating for policies that center harm reduction and patient welfare, including as director of the International Harm Reduction Development (IHRD) Program, Daniel Wolfe believes medicine has reached an “inflection point,” as healthcare decisions are increasingly made by algorithms, not medical professionals.

Daniel’s Bellagio Reflections: “One of the great beauties of my residency was that it connected me with people who helped to enrich and expand my analytic frame. Meg Urry’s understanding of machine learning in astrophysics, and her explanation of its limitations, inspired me to test the robustness of the evidence base for the same approach in healthcare. Abril Saldaña-Tejeda and Vaishali Sinha, each separately investigating aspects of the history and deployment of reproductive technologies, urged me to pay new attention to the interplay of technological solutions with social responsibility and governance.

“They helped me recognize that it’s about much more than just picking and using a particular algorithm; it’s about the ethos that underlies it.”

Daniel Wolfe discusses the troubling outcomes that can follow when doctors defer to algorithms in making life-or-death prescription decisions.

The overdose crisis is one of the major healthcare crises in modern history, claiming more than 100,000 lives in the U.S. in 2022 alone. That’s more than the number of AIDS deaths at the height of the U.S. HIV epidemic, and more than gun violence and traffic accident deaths combined. Unfortunately, I’ve seen how a technological solution is actually creating a new, unintended crisis of its own.

In 44 states, prescribers of controlled substances are using software like NarxCare, which assigns each patient an overdose risk score based on some questionable and largely unvalidated assumptions. My work focuses on the impact of the algorithm and associated opioid control on chronic pain patients – including many who have been stable on medicine for years before having their treatment abruptly tapered or discontinued. Tens of thousands of these patients are experiencing extreme suffering as a result, with some becoming suicidal or turning to street drugs that carry increased risk of fatality.

The fundamental question at the heart of my work, and the heart of algorithmic governance in general, is whether or not these tools are fit for clinical purposes. Even though algorithms are guiding drug prescriptions and other critical medical decisions, they’re not always tested appropriately beforehand, labeled appropriately during deployment, or evaluated appropriately post-marketing. NarxCare, specifically, has never had its overdose risk score independently evaluated, while one of its other scores has only been looked at by one independent study. Despite a questionable methodology and a data set drawn from a group of mostly white patients, even that study uncovered high false positive rates for pain patients.

In fairness, the company that makes the NarxCare algorithm, Bamboo Health, is very clear that the software is only meant to guide doctors and not make decisions in and of itself. But in reality, we know that many doctors are using the algorithm’s score as a pretext for either tapering down pain medication or cutting it off altogether.

To be clear, opioid over-prescription is a real problem in the U.S., but the solution cannot be to simply cut people off. Nor should algorithmic risk scores become a substitute for physician-patient exchange in determining the course of care. An algorithm is purporting to detect what’s in a patient’s best interest without their input, and then the healthcare system makes a prescription decision based on this automated process.