Abstract

The integration of Artificial Intelligence (AI) in healthcare holds immense promise for revolutionizing clinical practices and patient outcomes. However, the lack of transparency in AI decision-making processes poses significant challenges, hindering trust and understanding among healthcare professionals. Explainable Artificial Intelligence (XAI) has emerged as a promising solution to address these concerns by shedding light on AI model predictions and enhancing interpretability. This review explores the efficacy and applications of XAI within the healthcare domain, focusing on key research questions regarding challenges, effectiveness, and utilized algorithms. Through a comprehensive examination of 50 recent literature, we identify challenges related to the integration of XAI into clinical workflows, the necessity for validation and trust-building, and technical hurdles such as diverse explanation methods and data quality issues. Popular XAI algorithms such as SHAP, LIME, and GRAD-CAM demonstrate significant promise in clarifying model predictions and aiding in the interpretation of AI-driven healthcare systems. Overall, this review underscores the immense potential of XAI in revolutionizing healthcare delivery and decision-making processes, emphasizing the need for further research and development to address challenges and leverage its full potential in enhancing healthcare practices.

1. Introduction

Researchers in ¹ are actively exploring innovative approaches to seamlessly incorporate Artificial Intelligence (AI) across diverse industries, with healthcare emerging as a sector that has witnessed notable strides in this integration. Nevertheless, due to the sensitive nature of human lives, making decisions using AI models raises a lot of concern as we do not properly understand the intricate workings of most of these models ². Explainable Artificial intelligence (XAI) simply deals with understanding how machine learning systems make decisions so we can trust the decision it gives us ¹. This lack of transparency as seen in ³ can hinder health professionals' trust in AI systems, potentially impeding their integration into healthcare workflows. Moreover, the ethical implications of AI decisions necessitate that AI systems in healthcare are not only accurate but also interpretable, to ensure accountability and justify clinical decisions ².

As listed in ⁴, Artificial Intelligence exhibits diverse applications within the medical sector, encompassing but not limited to precision medicine, drug discovery, medical visualization, education, and intelligent health records. In the area of diagnosis, treatment, and patient care, there remains a spectrum of untapped potential for artificial intelligence as we explore this burgeoning technology. In precision medicine, traditional machine learning finds its primary application in predicting the success of treatment protocols for individual patients. According to ^{5, 6} this involves analyzing various patient attributes and contextual factors associated with the treatment to determine the most effective course of action. Also, ⁶ opines that AI has been used to recognize intricate patterns in medical images such as understanding chest radiographs, allowing them to achieve comparable or superior performance compared to clinicians in certain instances.

⁷ believes that machine learning algorithms can analyze genetic data, and clinical information that can predict high-risk patients, recommend personalized treatment plans, and prevent adverse events. AI-powered patient engagement tools like chatbots, wearables, and mobile devices support self-care, education, decision-making, and chronic condition management. In ^{8, 9}, it is understood that patients can access their health data, interact with healthcare providers online, and receive personalized recommendations. It is of no doubt that AI assists clinicians in decision-making by processing narrative health data, providing critical summaries of patient information, and improving diagnostic processes. It enhances disease diagnosis, treatment selection, and clinical laboratory testing.

Recent advancements in Explainable Artificial Intelligence (XAI) have made significant progress in addressing a crucial gap – the disparity between AI predictions and the understanding of end-users. These advancements focus on developing techniques that clarify the reasoning behind AI predictions, making them more understandable for people ¹⁰. This progress is especially important in areas like healthcare, where the ability to grasp the basis of AI-generated recommendations plays a vital role in influencing patient outcomes ¹¹. Understanding how AI arrives at its decisions is crucial for healthcare professionals and patients alike, as it contributes to informed and confident decision-making in clinical settings.

Additionally, the evolving landscape of regulations now places a growing emphasis on the explainability of these systems. This emphasis is particularly relevant in clinical settings, where ensuring the safe and ethical deployment of AI technologies is a top priority. Adhering to regulatory requirements, which increasingly stress the need for explainability, is essential to guarantee that AI models meet ethical standards and prioritize patient well-being ¹². The call for transparency and accountability in AI is not just a response to the growing complexity of these systems but also an acknowledgment of the ethical responsibility associated with their deployment in such critical areas.

There have been a lot of advances in the application of AI in healthcare, however, some challenges still present themselves. By performing a systematic literature review on recent papers in the domain, we aim to answer the following major questions.

1. What are the current challenges and problems in XAI for healthcare?

2. How effective has explainable AI been in healthcare?

3. What are the explainable AI algorithms that have been used?

2. Methodology

In this research, our objective is to explore the efficacy of Explainable Artificial Intelligence (XAI) in the healthcare sector. We are conducting a systematic review employing the scoping study methodology. Our study involves an exhaustive examination of the existing literature within this research domain, focusing on identifying the metrics and algorithms employed.

For the systematic literature review, we conducted a comprehensive search for articles related to Explainable AI (XAI) applications across various healthcare domains, spanning from diagnosis to treatment recommendation. Our search strategy involved querying two widely used academic databases: Google Scholar and PubMed. We utilized a combination of relevant keywords such as "explainable AI", and "healthcare", ensuring a broad scope of articles covering a diverse range of applications. We limited our search range from 2020 to 2024, specifically on March 5th, 2024.

Overall, our search strategy aimed to identify a comprehensive selection of literature encompassing various healthcare domains, including but not limited to radiology (use of X-rays for diagnosis and treatment), pathology, cardiology, and oncology, to provide a thorough understanding of the current landscape of XAI applications in healthcare.

In this stage, we screened papers based on our specific inclusion and exclusion criteria. Initially, we eliminated survey papers, review papers, and preprints after assessing their abstracts and conclusions, as they did not align with the focus of our research, which centered on the utilization of Explainable Artificial Intelligence (XAI) within a particular domain of healthcare. We systematically identified, screened, and extracted relevant information from all retrieved studies, adhering to the guidelines outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) ¹³.

Here, data from 50 papers was extracted and organized into a Google spreadsheet. Each paper was assigned a unique numeric identifier, and the 10 members of the research team were each allocated 10 papers. Consequently, there was overlap, with at least one paper being shared by two people. The extracted information underwent comparison, collation, and cross-checking by the lead researcher to ensure credibility.

Following data cleaning, which involved removing whitespaces and ensuring consistency among values, the dataset was preprocessed and analyzed using the Pandas library. Plots were generated using Matplotlib and Seaborn. The table below provides descriptions of the columns. Notably, for publishers, journals such as Nature, Scientific Reports, Springer, and other related journals were grouped under 'Springer Nature.' Also, review ¹⁴, survey ¹⁵, and preprint papers ¹⁶ authored by the original researchers and published in journals such as arXiv ^{17, 18} and MedRxiv ¹⁹ were excluded from the analysis.

∙ Search String: The query that was originally made included the words "XAI," "explainable," and "healthcare." From our limited search query, this might have caused us to miss some valuable articles related to our domain of study

∙ Selection of databases: We only made use of Google Scholar and PubMed since our domain is medical and healthcare, and we ignored other databases for the sake of credibility and quality.

∙ Time frame: We have only considered studies from 2020. There may have been some studies before that time that could be of beneficial contribution to our domain of study.

3. Results and Discussions

The papers present a comprehensive overview of challenges in implementing explainable AI (XAI) within healthcare. Some of the challenges revolve around the integration of AI into clinical workflows and the need for validation and trust-building. This includes the absence of real-world performance data, limited involvement of medical experts in algorithm design, and the necessity for rigorous internal and external validation to increase user trust and confidence in AI-driven decisions ^{6, 11, 20, 21 22, 23, 24, 25}. Moreover, technical hurdles such as the vast number of explanation methods and the need for tailored solutions for each application further complicate the implementation of XAI in healthcare ^{26, 27}. These challenges underscore the importance of addressing issues related to model interpretability, data quality, and trust-building mechanisms to facilitate the effective deployment of AI in clinical practice.

Multiple studies underscore the significance of XAI in enhancing clinical decision-making processes by providing insights into AI model predictions and facilitating better understanding among medical professionals. For instance, one study showcases the potential of XAI techniques such as SHAP and LIME in aiding clinicians to interpret machine learning models for diagnosing diseases like Alzheimer's and retinoblastoma, thereby improving trust and confidence in AI-driven healthcare systems ^{9, 11 30, 31, 32, 33}.

Furthermore, XAI methods have been instrumental in elucidating the decision-making process of complex deep-learning models, particularly in medical image analysis for diseases like pulmonary ailments and stroke detection ^{3, 34, 35, 36, 37 38, 39, 40, 41 42, 43, 44, 45}. These techniques not only increase transparency but also help in identifying the crucial factors influencing model predictions, thereby facilitating more accurate diagnoses and personalized treatment plans ^{12, 15, 25 46, 47, 48, 49}.

The collective findings underscore the immense potential of XAI in revolutionizing healthcare by improving the interpretability of AI models, enhancing trust among healthcare practitioners, and ultimately facilitating better clinical decision-making processes ^{23 50, 51}. As the field continues to evolve, further research and development in XAI are expected to drive innovations that will significantly impact healthcare delivery and patient outcomes.

From the plot above, we can infer that the most popular XAI algorithms being used are SHAP, LIME, and GRAD-CAM. These may be used alone, as demonstrated in ^{3, 17, 27 52, 53, 54, 55, 56, 57}, or in combination, as shown in ^{9, 11 58, 59, 60, 61, 62}. By combining both methods, researchers can leverage the strengths of each approach to gain a more comprehensive understanding of model behavior. In ¹¹ LIME produced segmentations of the images and highlighted the important regions for classification. On the other hand, SHAP provided a more accurate explanation of the model’s predictions by assigning feature importance scores to individual pixels in the image. We observed that SHAP was more effective in identifying important regions of the image, with pink areas highlighting the areas correctly identified as significant in retinoblastoma images and blue areas indicating the lack of significant features in normal images.

4. Conclusion

The integration of Artificial Intelligence (AI) in healthcare has witnessed substantial advancements, offering a myriad of applications ranging from precision medicine to patient engagement tools. However, the opacity of AI decision-making processes poses challenges in fostering trust and understanding among healthcare professionals. Explainable Artificial Intelligence (XAI) emerges as a pivotal solution, that aims to shed light upon AI model predictions and enhance interpretability. This systematic literature review delves into the efficacy and applications of XAI within the healthcare domain, addressing key research questions concerning challenges, effectiveness, and utilized algorithms. Technical hurdles, such as the numerous XAI explanation methods and data quality issues, further underscore the complexity of implementing XAI in healthcare settings. XAI methods such as SHAP, LIME, and GRAD-CAM have demonstrated significant promise in clarifying model predictions, aiding in disease diagnosis, treatment planning, and medical image analysis. The collective findings underscore the immense potential of XAI in revolutionizing healthcare delivery and decision-making processes. As the field continues to evolve, further research and development in XAI are imperative to address challenges and leverage its full potential in enhancing healthcare practices.

References