Improving Zero-Shot Detection of Low Prevalence Chest Pathologies using Domain Pre-trained Language Models
Swapping a CLIP model's text tower for one pre-trained on clinical text hurts on common chest findings but helps a lot on rare ones.
Abstract
Recent advances in zero-shot learning have enabled the use of paired image-text data to replace structured labels, removing the need for expert-annotated datasets. Models such as CLIP-based CheXzero use these advances for chest X-ray interpretation.
We hypothesize that domain pre-trained models such as CXR-BERT, BlueBERT, and ClinicalBERT can improve CLIP-like models with domain knowledge by replacing BERT weights, at the cost of breaking the original model's alignment. We evaluate zero-shot classification models with domain-specific pre-training for detecting low-prevalence pathologies. Even though replacing the weights of the original CLIP-BERT degrades performance on common pathologies, pre-trained text towers perform much better on low-prevalence diseases. This motivates future ensembles that combine differently trained language models.
Cite
@inproceedings{mishra2023improving,
title = {Improving Zero-Shot Detection of Low Prevalence Chest Pathologies using Domain Pre-trained Language Models},
author = {Aakash Mishra and Rajat Mittal and Christy Jestin and Kostas Tingos and Pranav Rajpurkar},
year = {2023},
booktitle = {Medical Imaging with Deep Learning, short paper track},
eprint = {2306.08000},
archivePrefix = {arXiv}
}