How Image Tokens Learn the Language of Radiology
What 2,400 X-ray studies reveal about how models represent clinical findings
We read the image tokens of our report-drafting model through its own clinical vocabulary and with linear probes. Deeper in the language model, they come to point at the right findings, with no direct supervision on the image tokens.
Abstract
Before a vision-language model writes a single word, it embeds each input image as a sequence of image tokens that have not yet seen any text. Do these tokens carry information that a radiologist would use, and how does the language model handle them?
In this post, we read the hidden states of image tokens from a radiology report-drafting VLM at increasing layer depths, and investigate them through the model's own clinical vocabulary and with linear probes. We observe that clinically useful information is largely present within hidden states of the input and early layers of the language model. Interestingly, as the image tokens are processed by the causal language decoder in later layers, their representations increasingly align with the model's own words for the findings present, despite no direct supervision on the image tokens.
Cite
@misc{mishra2026image,
title = {How Image Tokens Learn the Language of Radiology},
author = {Aakash Mishra and Arjun Karpur},
year = {2026},
howpublished = {Epsilon Health Research},
url = {https://www.epsilon.health/research/radiology-vlm-latent-space}
}