The Ethics of Artificial Intelligence: The Lawsuit for Refusing Health Care from Artificial Intelligence

UnitedHealthcare, the largest U.S. health insurance provider, is currently embroiled in a class-action lawsuit alleging it wrongfully denied extended care claims for elderly patients using an artificial intelligence (AI) algorithm called nH Predict.

The lawsuit, filed in the U.S. District Court for the District of Minnesota, accuses UnitedHealthcare of using the nH Predict algorithm to make health care decisions, resulting in premature and unconscionable termination of health care payments. The plaintiffs, who represent the estates of deceased individuals covered by Medicare Advantage plans provided by UnitedHealthcare, allege they were forced to pay out-of-pocket for medically necessary care because of the insurer’s denial of claims. The lawsuit potentially involves thousands of individuals and billions of dollars in damages, according to the plaintiffs’ attorney.

The lawsuit alleges that the nH Predict algorithm, developed by NaviHealth, a company acquired by UnitedHealth in 2020, systematically denied elderly patients’ requests for extended care, including stays in skilled nursing facilities and home care. It claims that when these denials are appealed to federal administrative law judges, about 90% are overturned, highlighting the alleged inaccuracy of the algorithm. The plaintiffs also claim that UnitedHealth’s use of the technology is illegal and violates patient contracts and insurance laws in various states.

The Future of Responsible Artificial Intelligence

A recent presidential executive order on artificial intelligence could potentially have implications for cases like this in the future. The order highlights the need for transparency and accountability in AI systems, particularly in sectors such as healthcare, where decisions made by AI algorithms can have a significant life-and-death impact. By promoting the responsible use of AI and ensuring that AI systems are designed and implemented in a way that supports fairness and prevents discrimination, the executive order aims to mitigate situations where AI algorithms wrongly deny essential care to vulnerable populations, such as the elderly.

The allegations against UnitedHealthcare underscore the importance of ethical and responsible implementation of AI, particularly in the healthcare industry. As AI continues to play an increasingly prominent role in decision-making, it is critical for organizations to prioritize ethical considerations around the use of AI. This includes ensuring that AI algorithms are accurate, transparent and aligned with the best interests of patients, especially when it comes to critical decisions related to health coverage and treatment.

“In AI regulation, there’s a fine line between effective oversight and misperceptions of the technology,” says Ryan Elmore, AI Innovation Fellow at West Monroe. “Even well-intentioned governance can be hampered by a lack of technical expertise and competing financial interests.”

The role of data management and governance in AI decision-making

In the context of this lawsuit, attention to data management and governance practices is paramount. Data governance defines how an organization manages its data assets and ensures the availability, integrity, security and usability of the organization’s structured and unstructured data assets. Organizations that mismanage their data as a real asset are poised to misuse those assets.

Here’s why it’s crucial in this context:

• Patient security and privacy: Healthcare data management is critical to ensuring patient safety and privacy by maintaining the accuracy and security of patient data. In the case of AI-based healthcare decision-making, accurate and secure patient data is essential to ensure that AI algorithms make informed and ethical decisions regarding patient care.
• Data integrity and usability: By establishing data management standards and policies, organizations can ensure the integrity and security of their data while maintaining compliance with regulatory requirements. This is especially important when AI algorithms are used to make critical patient care decisions, as the accuracy and integrity of the data used by these algorithms directly affects patient outcomes.
• Transparency and accountability: Data governance provides a framework for managing data throughout its lifecycle, including how data is collected, stored, processed, analyzed and shared. This level of transparency and accountability is essential when AI algorithms are involved in healthcare decision-making, as it ensures that the decisions made by these algorithms are based on accurate and reliable data.
• Shared accountability and collaboration: Data management creates shared accountability for staff members by actively engaging them in creating policies, best practices, procedures, and work products that improve the quality of patient demographic data. In the context of AI-driven decision-making, this shared responsibility ensures that the data used by AI algorithms is of high quality and reflects the collective expertise and collaboration of healthcare professionals.
• Informed decision-making: Governance ensures that the right people can use data at the right time for the right reasons, supporting decision-making and ensuring that the organization successfully realizes its desired outcomes and derives business value from its data management activities. This is particularly relevant when AI algorithms are used in healthcare, as informed decision-making is critical to ensuring patient well-being.

Ethics as a company priority

The lawsuit against UnitedHealthcare serves as a stark reminder of the potential consequences of misuse of artificial intelligence (whether intentional or unintentional) in the healthcare sector. A recent executive order on artificial intelligence highlights the need for ethical and responsible AI practices, especially in critical areas such as healthcare, to prevent situations where AI algorithms can unjustifiably deny essential care to vulnerable populations. Regardless, it is imperative for healthcare organizations to prioritize the ethical application of AI, reduce bias in algorithms and data, and uphold high standards of transparency and fairness in all forms of automated decision-making processes.

This is another emerging concern to add to the growing list of new existential business challenges for data and analytics leaders, says Keyjur Desai, CEO of Healthcare Industry Data. However, with strong relationships and collaboration between data, technology and business leaders, it is certainly not insurmountable. Human progress has never paused to play ethical catch-up, nor should we expect it to.