-
CheXRelNet: An Anatomy-Aware Model for Tracking Longitudinal Relationships between Chest X-Rays
Authors:
Gaurang Karwande,
Amarachi Mbakawe,
Joy T. Wu,
Leo A. Celi,
Mehdi Moradi,
Ismini Lourentzou
Abstract:
Despite the progress in utilizing deep learning to automate chest radiograph interpretation and disease diagnosis tasks, change between sequential Chest X-rays (CXRs) has received limited attention. Monitoring the progression of pathologies that are visualized through chest imaging poses several challenges in anatomical motion estimation and image registration, i.e., spatially aligning the two ima…
▽ More
Despite the progress in utilizing deep learning to automate chest radiograph interpretation and disease diagnosis tasks, change between sequential Chest X-rays (CXRs) has received limited attention. Monitoring the progression of pathologies that are visualized through chest imaging poses several challenges in anatomical motion estimation and image registration, i.e., spatially aligning the two images and modeling temporal dynamics in change detection. In this work, we propose CheXRelNet, a neural model that can track longitudinal pathology change relations between two CXRs. CheXRelNet incorporates local and global visual features, utilizes inter-image and intra-image anatomical information, and learns dependencies between anatomical region attributes, to accurately predict disease change for a pair of CXRs. Experimental results on the Chest ImaGenome dataset show increased downstream performance compared to baselines. Code is available at https://github.com/PLAN-Lab/ChexRelNet
△ Less
Submitted 15 September, 2022; v1 submitted 7 August, 2022;
originally announced August 2022.
-
Developing and validating multi-modal models for mortality prediction in COVID-19 patients: a multi-center retrospective study
Authors:
Joy Tzung-yu Wu,
Miguel Ángel Armengol de la Hoz,
Po-Chih Kuo,
Joseph Alexander Paguio,
Jasper Seth Yao,
Edward Christopher Dee,
Wesley Yeung,
Jerry Jurado,
Achintya Moulick,
Carmelo Milazzo,
Paloma Peinado,
Paula Villares,
Antonio Cubillo,
José Felipe Varona,
Hyung-Chul Lee,
Alberto Estirado,
José Maria Castellano,
Leo Anthony Celi
Abstract:
The unprecedented global crisis brought about by the COVID-19 pandemic has sparked numerous efforts to create predictive models for the detection and prognostication of SARS-CoV-2 infections with the goal of helping health systems allocate resources. Machine learning models, in particular, hold promise for their ability to leverage patient clinical information and medical images for prediction. Ho…
▽ More
The unprecedented global crisis brought about by the COVID-19 pandemic has sparked numerous efforts to create predictive models for the detection and prognostication of SARS-CoV-2 infections with the goal of helping health systems allocate resources. Machine learning models, in particular, hold promise for their ability to leverage patient clinical information and medical images for prediction. However, most of the published COVID-19 prediction models thus far have little clinical utility due to methodological flaws and lack of appropriate validation. In this paper, we describe our methodology to develop and validate multi-modal models for COVID-19 mortality prediction using multi-center patient data. The models for COVID-19 mortality prediction were developed using retrospective data from Madrid, Spain (N=2547) and were externally validated in patient cohorts from a community hospital in New Jersey, USA (N=242) and an academic center in Seoul, Republic of Korea (N=336). The models we developed performed differently across various clinical settings, underscoring the need for a guided strategy when employing machine learning for clinical decision-making. We demonstrated that using features from both the structured electronic health records and chest X-ray imaging data resulted in better 30-day-mortality prediction performance across all three datasets (areas under the receiver operating characteristic curves: 0.85 (95% confidence interval: 0.83-0.87), 0.76 (0.70-0.82), and 0.95 (0.92-0.98)). We discuss the rationale for the decisions made at every step in developing the models and have made our code available to the research community. We employed the best machine learning practices for clinical model development. Our goal is to create a toolkit that would assist investigators and organizations in building multi-modal models for prediction, classification and/or optimization.
△ Less
Submitted 1 September, 2021;
originally announced September 2021.
-
Chest ImaGenome Dataset for Clinical Reasoning
Authors:
Joy T. Wu,
Nkechinyere N. Agu,
Ismini Lourentzou,
Arjun Sharma,
Joseph A. Paguio,
Jasper S. Yao,
Edward C. Dee,
William Mitchell,
Satyananda Kashyap,
Andrea Giovannini,
Leo A. Celi,
Mehdi Moradi
Abstract:
Despite the progress in automatic detection of radiologic findings from chest X-ray (CXR) images in recent years, a quantitative evaluation of the explainability of these models is hampered by the lack of locally labeled datasets for different findings. With the exception of a few expert-labeled small-scale datasets for specific findings, such as pneumonia and pneumothorax, most of the CXR deep le…
▽ More
Despite the progress in automatic detection of radiologic findings from chest X-ray (CXR) images in recent years, a quantitative evaluation of the explainability of these models is hampered by the lack of locally labeled datasets for different findings. With the exception of a few expert-labeled small-scale datasets for specific findings, such as pneumonia and pneumothorax, most of the CXR deep learning models to date are trained on global "weak" labels extracted from text reports, or trained via a joint image and unstructured text learning strategy. Inspired by the Visual Genome effort in the computer vision community, we constructed the first Chest ImaGenome dataset with a scene graph data structure to describe $242,072$ images. Local annotations are automatically produced using a joint rule-based natural language processing (NLP) and atlas-based bounding box detection pipeline. Through a radiologist constructed CXR ontology, the annotations for each CXR are connected as an anatomy-centered scene graph, useful for image-level reasoning and multimodal fusion applications. Overall, we provide: i) $1,256$ combinations of relation annotations between $29$ CXR anatomical locations (objects with bounding box coordinates) and their attributes, structured as a scene graph per image, ii) over $670,000$ localized comparison relations (for improved, worsened, or no change) between the anatomical locations across sequential exams, as well as ii) a manually annotated gold standard scene graph dataset from $500$ unique patients.
△ Less
Submitted 31 July, 2021;
originally announced August 2021.
-
AnaXNet: Anatomy Aware Multi-label Finding Classification in Chest X-ray
Authors:
Nkechinyere N. Agu,
Joy T. Wu,
Hanqing Chao,
Ismini Lourentzou,
Arjun Sharma,
Mehdi Moradi,
Pingkun Yan,
James Hendler
Abstract:
Radiologists usually observe anatomical regions of chest X-ray images as well as the overall image before making a decision. However, most existing deep learning models only look at the entire X-ray image for classification, failing to utilize important anatomical information. In this paper, we propose a novel multi-label chest X-ray classification model that accurately classifies the image findin…
▽ More
Radiologists usually observe anatomical regions of chest X-ray images as well as the overall image before making a decision. However, most existing deep learning models only look at the entire X-ray image for classification, failing to utilize important anatomical information. In this paper, we propose a novel multi-label chest X-ray classification model that accurately classifies the image finding and also localizes the findings to their correct anatomical regions. Specifically, our model consists of two modules, the detection module and the anatomical dependency module. The latter utilizes graph convolutional networks, which enable our model to learn not only the label dependency but also the relationship between the anatomical regions in the chest X-ray. We further utilize a method to efficiently create an adjacency matrix for the anatomical regions using the correlation of the label across the different regions. Detailed experiments and analysis of our results show the effectiveness of our method when compared to the current state-of-the-art multi-label chest X-ray image classification methods while also providing accurate location information.
△ Less
Submitted 20 May, 2021;
originally announced May 2021.
-
Extracting and Learning Fine-Grained Labels from Chest Radiographs
Authors:
Tanveer Syeda-Mahmood,
Ph. D,
K. C. L Wong,
Ph. D,
Joy T. Wu,
M. D.,
M. P. H,
Ashutosh Jadhav,
Ph. D,
Orest Boyko,
M. D. Ph. D
Abstract:
Chest radiographs are the most common diagnostic exam in emergency rooms and intensive care units today. Recently, a number of researchers have begun working on large chest X-ray datasets to develop deep learning models for recognition of a handful of coarse finding classes such as opacities, masses and nodules. In this paper, we focus on extracting and learning fine-grained labels for chest X-ray…
▽ More
Chest radiographs are the most common diagnostic exam in emergency rooms and intensive care units today. Recently, a number of researchers have begun working on large chest X-ray datasets to develop deep learning models for recognition of a handful of coarse finding classes such as opacities, masses and nodules. In this paper, we focus on extracting and learning fine-grained labels for chest X-ray images. Specifically we develop a new method of extracting fine-grained labels from radiology reports by combining vocabulary-driven concept extraction with phrasal grouping in dependency parse trees for association of modifiers with findings. A total of 457 fine-grained labels depicting the largest spectrum of findings to date were selected and sufficiently large datasets acquired to train a new deep learning model designed for fine-grained classification. We show results that indicate a highly accurate label extraction process and a reliable learning of fine-grained labels. The resulting network, to our knowledge, is the first to recognize fine-grained descriptions of findings in images covering over nine modifiers including laterality, location, severity, size and appearance.
△ Less
Submitted 18 November, 2020;
originally announced November 2020.
-
Learning Invariant Feature Representation to Improve Generalization across Chest X-ray Datasets
Authors:
Sandesh Ghimire,
Satyananda Kashyap,
Joy T. Wu,
Alexandros Karargyris,
Mehdi Moradi
Abstract:
Chest radiography is the most common medical image examination for screening and diagnosis in hospitals. Automatic interpretation of chest X-rays at the level of an entry-level radiologist can greatly benefit work prioritization and assist in analyzing a larger population. Subsequently, several datasets and deep learning-based solutions have been proposed to identify diseases based on chest X-ray…
▽ More
Chest radiography is the most common medical image examination for screening and diagnosis in hospitals. Automatic interpretation of chest X-rays at the level of an entry-level radiologist can greatly benefit work prioritization and assist in analyzing a larger population. Subsequently, several datasets and deep learning-based solutions have been proposed to identify diseases based on chest X-ray images. However, these methods are shown to be vulnerable to shift in the source of data: a deep learning model performing well when tested on the same dataset as training data, starts to perform poorly when it is tested on a dataset from a different source. In this work, we address this challenge of generalization to a new source by forcing the network to learn a source-invariant representation. By employing an adversarial training strategy, we show that a network can be forced to learn a source-invariant representation. Through pneumonia-classification experiments on multi-source chest X-ray datasets, we show that this algorithm helps in improving classification accuracy on a new source of X-ray dataset.
△ Less
Submitted 4 August, 2020;
originally announced August 2020.
-
Chest X-ray Report Generation through Fine-Grained Label Learning
Authors:
Tanveer Syeda-Mahmood,
Ken C. L. Wong,
Yaniv Gur,
Joy T. Wu,
Ashutosh Jadhav,
Satyananda Kashyap,
Alexandros Karargyris,
Anup Pillai,
Arjun Sharma,
Ali Bin Syed,
Orest Boyko,
Mehdi Moradi
Abstract:
Obtaining automated preliminary read reports for common exams such as chest X-rays will expedite clinical workflows and improve operational efficiencies in hospitals. However, the quality of reports generated by current automated approaches is not yet clinically acceptable as they cannot ensure the correct detection of a broad spectrum of radiographic findings nor describe them accurately in terms…
▽ More
Obtaining automated preliminary read reports for common exams such as chest X-rays will expedite clinical workflows and improve operational efficiencies in hospitals. However, the quality of reports generated by current automated approaches is not yet clinically acceptable as they cannot ensure the correct detection of a broad spectrum of radiographic findings nor describe them accurately in terms of laterality, anatomical location, severity, etc. In this work, we present a domain-aware automatic chest X-ray radiology report generation algorithm that learns fine-grained description of findings from images and uses their pattern of occurrences to retrieve and customize similar reports from a large report database. We also develop an automatic labeling algorithm for assigning such descriptors to images and build a novel deep learning network that recognizes both coarse and fine-grained descriptions of findings. The resulting report generation algorithm significantly outperforms the state of the art using established score metrics.
△ Less
Submitted 27 July, 2020;
originally announced July 2020.
-
A Corpus for Detecting High-Context Medical Conditions in Intensive Care Patient Notes Focusing on Frequently Readmitted Patients
Authors:
Edward T. Moseley,
Joy T. Wu,
Jonathan Welt,
John Foote,
Patrick D. Tyler,
David W. Grant,
Eric T. Carlson,
Sebastian Gehrmann,
Franck Dernoncourt,
Leo Anthony Celi
Abstract:
A crucial step within secondary analysis of electronic health records (EHRs) is to identify the patient cohort under investigation. While EHRs contain medical billing codes that aim to represent the conditions and treatments patients may have, much of the information is only present in the patient notes. Therefore, it is critical to develop robust algorithms to infer patients' conditions and treat…
▽ More
A crucial step within secondary analysis of electronic health records (EHRs) is to identify the patient cohort under investigation. While EHRs contain medical billing codes that aim to represent the conditions and treatments patients may have, much of the information is only present in the patient notes. Therefore, it is critical to develop robust algorithms to infer patients' conditions and treatments from their written notes. In this paper, we introduce a dataset for patient phenotyping, a task that is defined as the identification of whether a patient has a given medical condition (also referred to as clinical indication or phenotype) based on their patient note. Nursing Progress Notes and Discharge Summaries from the Intensive Care Unit of a large tertiary care hospital were manually annotated for the presence of several high-context phenotypes relevant to treatment and risk of re-hospitalization. This dataset contains 1102 Discharge Summaries and 1000 Nursing Progress Notes. Each Discharge Summary and Progress Note has been annotated by at least two expert human annotators (one clinical researcher and one resident physician). Annotated phenotypes include treatment non-adherence, chronic pain, advanced/metastatic cancer, as well as 10 other phenotypes. This dataset can be utilized for academic and industrial research in medicine and computer science, particularly within the field of medical natural language processing.
△ Less
Submitted 6 March, 2020;
originally announced March 2020.
-
Automated Detection and Type Classification of Central Venous Catheters in Chest X-Rays
Authors:
Vaishnavi Subramanian,
Hongzhi Wang,
Joy T. Wu,
Ken C. L. Wong,
Arjun Sharma,
Tanveer Syeda-Mahmood
Abstract:
Central venous catheters (CVCs) are commonly used in critical care settings for monitoring body functions and administering medications. They are often described in radiology reports by referring to their presence, identity and placement. In this paper, we address the problem of automatic detection of their presence and identity through automated segmentation using deep learning networks and class…
▽ More
Central venous catheters (CVCs) are commonly used in critical care settings for monitoring body functions and administering medications. They are often described in radiology reports by referring to their presence, identity and placement. In this paper, we address the problem of automatic detection of their presence and identity through automated segmentation using deep learning networks and classification based on their intersection with previously learned shape priors from clinician annotations of CVCs. The results not only outperform existing methods of catheter detection achieving 85.2% accuracy at 91.6% precision, but also enable high precision (95.2%) classification of catheter types on a large dataset of over 10,000 chest X-rays, presenting a robust and practical solution to this problem.
△ Less
Submitted 25 July, 2019; v1 submitted 2 July, 2019;
originally announced July 2019.
-
Boosting the rule-out accuracy of deep disease detection using class weight modifiers
Authors:
Alexandros Karargyris,
Ken C. L. Wong,
Joy T. Wu,
Mehdi Moradi,
Tanveer Syeda-Mahmood
Abstract:
In many screening applications, the primary goal of a radiologist or assisting artificial intelligence is to rule out certain findings. The classifiers built for such applications are often trained on large datasets that derive labels from clinical notes written for patients. While the quality of the positive findings described in these notes is often reliable, lack of the mention of a finding doe…
▽ More
In many screening applications, the primary goal of a radiologist or assisting artificial intelligence is to rule out certain findings. The classifiers built for such applications are often trained on large datasets that derive labels from clinical notes written for patients. While the quality of the positive findings described in these notes is often reliable, lack of the mention of a finding does not always rule out the presence of it. This happens because radiologists comment on the patient in the context of the exam, for example focusing on trauma as opposed to chronic disease at emergency rooms. However, this disease finding ambiguity can affect the performance of algorithms. Hence it is critical to model the ambiguity during training. We propose a scheme to apply reasonable class weight modifiers to our loss function for the no mention cases during training. We experiment with two different deep neural network architectures and show that the proposed method results in a large improvement in the performance of the classifiers, specially on negated findings. The baseline performance of a custom-made dilated block network proposed in this paper shows an improvement in comparison with baseline DenseNet-201, while both architectures benefit from the new proposed loss function weighting scheme. Over 200,000 chest X-ray images and three highly common diseases, along with their negated counterparts, are included in this study.
△ Less
Submitted 21 June, 2019;
originally announced June 2019.
-
Building a Benchmark Dataset and Classifiers for Sentence-Level Findings in AP Chest X-rays
Authors:
Tanveer Syeda-Mahmood,
Hassan M. Ahmad,
Nadeem Ansari,
Yaniv Gur,
Satyananda Kashyap,
Alexandros Karargyris,
Mehdi Moradi,
Anup Pillai,
Karthik Sheshadri,
Weiting Wang,
Ken C. L. Wong,
Joy T. Wu
Abstract:
Chest X-rays are the most common diagnostic exams in emergency rooms and hospitals. There has been a surge of work on automatic interpretation of chest X-rays using deep learning approaches after the availability of large open source chest X-ray dataset from NIH. However, the labels are not sufficiently rich and descriptive for training classification tools. Further, it does not adequately address…
▽ More
Chest X-rays are the most common diagnostic exams in emergency rooms and hospitals. There has been a surge of work on automatic interpretation of chest X-rays using deep learning approaches after the availability of large open source chest X-ray dataset from NIH. However, the labels are not sufficiently rich and descriptive for training classification tools. Further, it does not adequately address the findings seen in Chest X-rays taken in anterior-posterior (AP) view which also depict the placement of devices such as central vascular lines and tubes. In this paper, we present a new chest X-ray benchmark database of 73 rich sentence-level descriptors of findings seen in AP chest X-rays. We describe our method of obtaining these findings through a semi-automated ground truth generation process from crowdsourcing of clinician annotations. We also present results of building classifiers for these findings that show that such higher granularity labels can also be learned through the framework of deep learning classifiers.
△ Less
Submitted 21 June, 2019;
originally announced June 2019.
-
Age prediction using a large chest X-ray dataset
Authors:
Alexandros Karargyris,
Satyananda Kashyap,
Joy T Wu,
Arjun Sharma,
Mehdi Moradi,
Tanveer Syeda-Mahmood
Abstract:
Age prediction based on appearances of different anatomies in medical images has been clinically explored for many decades. In this paper, we used deep learning to predict a persons age on Chest X-Rays. Specifically, we trained a CNN in regression fashion on a large publicly available dataset. Moreover, for interpretability, we explored activation maps to identify which areas of a CXR image are im…
▽ More
Age prediction based on appearances of different anatomies in medical images has been clinically explored for many decades. In this paper, we used deep learning to predict a persons age on Chest X-Rays. Specifically, we trained a CNN in regression fashion on a large publicly available dataset. Moreover, for interpretability, we explored activation maps to identify which areas of a CXR image are important for the machine (i.e. CNN) to predict a patients age, offering insight. Overall, amongst correctly predicted CXRs, we see areas near the clavicles, shoulders, spine, and mediastinum being most activated for age prediction, as one would expect biologically. Amongst incorrectly predicted CXRs, we have qualitatively identified disease patterns that could possibly make the anatomies appear older or younger than expected. A further technical and clinical evaluation would improve this work. As CXR is the most commonly requested imaging exam, a potential use case for estimating age may be found in the preventative counseling of patient health status compared to their age-expected average, particularly when there is a large discrepancy between predicted age and the real patient age.
△ Less
Submitted 8 March, 2019;
originally announced March 2019.
-
Comparing Rule-Based and Deep Learning Models for Patient Phenotyping
Authors:
Sebastian Gehrmann,
Franck Dernoncourt,
Yeran Li,
Eric T. Carlson,
Joy T. Wu,
Jonathan Welt,
John Foote Jr.,
Edward T. Moseley,
David W. Grant,
Patrick D. Tyler,
Leo Anthony Celi
Abstract:
Objective: We investigate whether deep learning techniques for natural language processing (NLP) can be used efficiently for patient phenotyping. Patient phenotyping is a classification task for determining whether a patient has a medical condition, and is a crucial part of secondary analysis of healthcare data. We assess the performance of deep learning algorithms and compare them with classical…
▽ More
Objective: We investigate whether deep learning techniques for natural language processing (NLP) can be used efficiently for patient phenotyping. Patient phenotyping is a classification task for determining whether a patient has a medical condition, and is a crucial part of secondary analysis of healthcare data. We assess the performance of deep learning algorithms and compare them with classical NLP approaches.
Materials and Methods: We compare convolutional neural networks (CNNs), n-gram models, and approaches based on cTAKES that extract pre-defined medical concepts from clinical notes and use them to predict patient phenotypes. The performance is tested on 10 different phenotyping tasks using 1,610 discharge summaries extracted from the MIMIC-III database.
Results: CNNs outperform other phenotyping algorithms in all 10 tasks. The average F1-score of our model is 76 (PPV of 83, and sensitivity of 71) with our model having an F1-score up to 37 points higher than alternative approaches. We additionally assess the interpretability of our model by presenting a method that extracts the most salient phrases for a particular prediction.
Conclusion: We show that NLP methods based on deep learning improve the performance of patient phenotyping. Our CNN-based algorithm automatically learns the phrases associated with each patient phenotype. As such, it reduces the annotation complexity for clinical domain experts, who are normally required to develop task-specific annotation rules and identify relevant phrases. Our method performs well in terms of both performance and interpretability, which indicates that deep learning is an effective approach to patient phenotyping based on clinicians' notes.
△ Less
Submitted 25 March, 2017;
originally announced March 2017.
-
High-precision Distribution of Highly-stable Optical Pulse Trains with Sub-10-fs Timing Jitter
Authors:
B. Ning,
S. Y. Zhang,
D. Hou,
J. T. Wu,
Z. B. Li,
J. Y. Zhao
Abstract:
High-precision optical pulse trains distribution via fibre links has made huge impacts in many fields. In most published works, the accuracies are still fundamentally limited by some unavoidable noises, such as thermal and shot noise from conventional photodiodes, thermal noise from mixers. Here, we demonstrate a new high-precision timing distribution system by using highly-precision phase detecto…
▽ More
High-precision optical pulse trains distribution via fibre links has made huge impacts in many fields. In most published works, the accuracies are still fundamentally limited by some unavoidable noises, such as thermal and shot noise from conventional photodiodes, thermal noise from mixers. Here, we demonstrate a new high-precision timing distribution system by using highly-precision phase detector to overcome the limitations. Instead of using photodiodes and microwave mixers, we use several fibre Sagnac-loop-based optical-microwave phase detectors to realize optical-electrical conversion and phase measurements, for suppressing the noises and achieving ultra-high accuracy. A 10-km fibre link distribution experiment shows our system provides a residual instability at the level of 4.6*10-15@1-s and 6.1*10-18@10000-s, with an integrated timing jitter as low as 3.8 fs in a bandwidth of 1 Hz to 100 KHz. This low instability and timing jitter makes it possible that our system can be used in the optical clock distribution or the applications for the facilities which require extremely accuracy frequency time synchronization.
△ Less
Submitted 23 February, 2014;
originally announced February 2014.
