“Everyone believes that AI will assist us in enhancing our access, capacity, and improving care,” stated Nigam Shah, the chief data scientist at Stanford Health Care. “While all of that sounds promising, if it results in a 20% increase in the cost of care, is it sustainable?” Concerns have been raised by government officials about hospitals lacking the resources to thoroughly test these technologies. “I have searched extensively,” remarked FDA Commissioner Robert Califf during a recent agency panel on AI. “I do not think there is a single health system in the United States equipped to validate an AI algorithm implemented in a clinical care system.”
AI is already widely used in healthcare. Algorithms play a crucial role in predicting patients’ risk of death or deterioration, suggesting diagnoses, triaging patients, documenting visits to alleviate doctors’ workload, and processing insurance claims. If tech enthusiasts are correct, this technology will become widespread and profitable. Bessemer Venture Partners has identified approximately 20 health-focused AI startups expected to generate $10 million in revenue each year. The FDA has granted approval to nearly a thousand AI products.
Assessing the effectiveness of these products poses a challenge. It is even more complex to determine whether they continue to operate effectively or encounter issues akin to a malfunctioning engine. Consider a recent study at Yale Medicine examining six “early warning systems” that notify clinicians of patients likely to deteriorate rapidly. A supercomputer analyzed the data over several days, revealing significant variations in performance among the six products.
Choosing the most suitable algorithms for their requirements is not straightforward for hospitals and providers. The average doctor does not have immediate access to a supercomputer, and there isn’t a Consumer Reports equivalent for AI. “We lack standards,” noted Jesse Ehrenfeld, immediate past president of the American Medical Association. “Currently, there is no established protocol for evaluating, monitoring, or assessing the performance of an algorithm or AI model once it is deployed.”
One prevalent AI tool in doctors’ offices is ambient documentation, an AI-enabled assistant that listens to and summarizes patient visits. Investors at Rock Health observed $353 million invested in these documentation companies last year. However, there’s no standard in place for comparing the output of these tools. This lack of consistency can be problematic, particularly when minor errors can have significant consequences in the medical field.
Errors in algorithms can stem from various factors, such as changes in underlying data that diminish their efficacy, like when hospitals switch lab providers. In some instances, however, algorithms fail without an apparent cause. Sandy Aronson, a technology executive at Mass Ge…
Dr. Neral Brigham, leading the personalized medicine program in Boston, disclosed a concerning issue with a new application designed to aid genetic counselors in accessing pertinent literature on DNA variants. He explained that the product exhibited “nondeterminism,” meaning that it provided varying results when presented with the same inquiry repeatedly within a short timeframe.
While acknowledging the potential benefits of leveraging large language models to condense knowledge for overwhelmed genetic counselors, Brigham emphasized the pressing need for technological enhancements in this area. The unpredictable nature of the software poses challenges, prompting the question of how institutions should proceed in the face of sparse metrics, unexpected errors, and systemic vulnerabilities.
In a notable development, a Texas hospital reportedly became the first in the United States to introduce holograms for doctor-patient consultations, showcasing the ongoing integration of innovative technologies in healthcare settings.
Dr. Aronson highlighted the significant time and resource investment required to ensure the fairness and reliability of artificial intelligence models. At Stanford University, for instance, the meticulous auditing process for just two models demanded an extensive timeframe of eight to 10 months and a substantial human effort totaling 115 man-hours.
Reflecting on the complexities surrounding the deployment of AI in healthcare, experts consulted by KFF Health News deliberated on the possibility of implementing artificial intelligence systems to oversee their counterparts, with human data specialists overseeing the entire monitoring process. However, this proposed approach entails additional financial commitments from organizations, a challenging proposition given the budget constraints faced by many healthcare institutions and the limited pool of AI experts available for recruitment.
Dr. Shah, from Stanford, shared insights into the evolving landscape of healthcare AI, emphasizing the unanticipated reliance on human input to navigate the intricacies of these advanced technologies. While the initial goal of healthcare AI was cost-saving, the reality has unfolded quite differently, requiring substantial human intervention and expertise.
The notion of establishing a system where one AI system monitors another was scrutinized by industry insiders, raising pertinent questions about the practicality and sustainability of such an approach. Dr. Shah pondered the implications of continually expanding the workforce to manage and supervise these sophisticated AI systems, highlighting the inherent challenges associated with scaling human resources to meet the demands of the evolving healthcare AI landscape.
In conclusion, the integration of AI technologies in healthcare, while promising significant advancements and efficiencies, has unveiled a paradoxical reality where the cost-saving potential of AI is counterbalanced by the need for substantial human involvement and oversight. The journey towards effectively harnessing the capabilities of AI in healthcare necessitates a thoughtful balance between technological advancements and human resources, underscoring the intricate interplay between innovation and practical implementation in the healthcare sector.
