Research

Reading the computation of vision-language models

See which image regions the model reads, the visual concepts it used, and which of those actually influenced the answer.

The output tells you what a vision-language model predicted. Before you act on it, especially in a high-stakes read like disaster triage, you need to understand how it predicted: which image parts it used, and whether the answer came from the image or from the prompt.

aerial photo of kayakers among rocky coastal waters — Hover any word the model generated: the image lights up in red where that word was found, and the panel lists the visual concepts that drove it, ranked by their relevance. On the coastline, **rocky** draws 63% of its evidence from the image, on the shoreline texture. Switch to the second scene, one dog hidden in a flock of sheep, and the model puts 62% of the evidence for **dog** on the image, concentrated on the black head. Model: Qwen3.5-4B.

relevance heatmap for the selected word — Hover any word the model generated: the image lights up in red where that word was found, and the panel lists the visual concepts that drove it, ranked by their relevance. On the coastline, **rocky** draws 63% of its evidence from the image, on the shoreline texture. Switch to the second scene, one dog hidden in a flock of sheep, and the model puts 62% of the evidence for **dog** on the image, concentrated on the black head. Model: Qwen3.5-4B.

Would you deploy on the output alone?

When you ask a vision-language model (VLM) whether it looked at the image for its output, it will tell you confidently that it did. But that reply is just more generated text, and nothing in it can be meaninfully verified. The output never reveals which visual features the model relied on, or whether an answer turned on evidence in the image or on the wording of the prompt. An invented detail is written just as smoothly as an observed one.

To detect these cases, we read the internal computation instead. Using Layer-wise Relevance Propagation (LRP), and specifically our attention-aware extension AttnLRP, we trace an output word backward through the network to the inputs and internal units that produced it, and answer three questions:

Where it looks

Relevance on image regions versus prompt text. Did the answer come from the picture, or from the wording of the question?

What it sees

A sparse dictionary (SAE) turns dense activations into named visual concepts you can inspect and point at.

What mattered

Attribution ranks those concepts by how much each one actually pushed the answer, signed for and against.

Tracing human-understandable concepts

The model has two inputs: the image and the prompt. The image is processed by a vision tower we instrument, where a sparse autoencoder (SAE) at block 22 reads the internal representation out as named concepts. The language model stays frozen and unmodified. To explain a generated word, relevance flows backward through that frozen language model and lands on both the vision concepts and the prompt text, so we can see how much of the answer each input carried.

Grounded in image or words?

For every word, AttnLRP splits the evidence into an image share and a prompt share. For this answer, a clear majority of words lands on the image, so the model is describing what it sees rather than reciting a prior.

image · 53%prompt text · 47%

Click any word to resolve the split for that token; the bar above updates. Deeper shading already previews each word's image-share; click to read it exactly.

Aerial view of kayakers navigating rocky coastal waters, surrounded by cliffs and greenery.

"Aerial", "kayakers", "rocky" and "cliffs" are strongly grounded in the image with a relevance share of over 55%. The three words without relation to the picture, the grammatical connectors (of, by, and) carry the lowest image share in the sentence, ~40% picture to ~60% prompt, while the words visible in the image get 56–65% from the image.

Activations made interpretable

The mechanisms we want to read are latent. The vision tower of the VLM represents input features as high-dimensional activation vectors, and their meaning cannot be read directly. The features are distributed across thousands of dimensions, and no single dimension stands for anything human-readable. We train a sparse autoencoder (SAE) inside the vision tower to make them interpretable, encoding each vector as a handful of active concepts, each with a stable identity you can read off its top-activating examples.

where concept 8376 fired on the scene — L8376 · aerial · top-down framingdrives “Aerial”

top-activating images for concept 8376 — L8376 · aerial · top-down framingdrives “Aerial”

Active does not equal decisive

A single image may activate an enormous number of concepts. On this scene 5,003 distinct concepts activate across the vision tokens. However, activation alone does not indicate importance, the median active concept contributes essentially zero. Attribution with LRP separates the few that were relevant to a specific output from the crowd that simply activated. We integrate relevance with LRP path to compute each concept's signed causal contribution to a chosen output token.

Activation tells you neither how much a concept moved the answer, nor the direction. Attribution provides both.

The example below shows this for the word “rocky”, where a concept can contribute toward the word or against it.

5,003concepts active on this image

→

2supply 14% of all the relevance

L1471 · rocky-shore texture+1.26L10252 · cliff faces+0.58L15234 · craggy rock · brick+0.28L1971 · green vegetation−0.09← suppresses “rocky”supports “rocky” →

Signed concept relevance toward the output token “rocky”, on a shared scale. A rocky-shore texture concept L1471 dominates, cliff-face L10252 and craggy-rock L15234 support it. A green-vegetation concept L1971 contributed against it (green is not rocky). In total 2,833 concepts support the word positively and 2,170 negatively; the top two concepts account for 14% of all relevance, the top ten for 29%.

Text recognition: Watching the model read

Applying this approach to text shown in an image turns into a model's "gaze tracker". For both examples below, the models were asked to only transcribe what they are shown. The Victorian classifieds were the harder case: photographed at an angle, curved at the gutter and set in competing display faces. Each token the model emits attributes back to the letters it read, with 61–94% of relevance on the image, moving in reading order. The overlays show the heatmap, indicating where the token came from. The 1890 Fraktur broadsheet is another, cleaner case, and there the output is based 71–97% on the pixels. When looking at the full heatmap, you may notice that the model skipped parts of newspaper, and never fully transcribed it.

photograph of a Victorian English newspaper classified-ads spread — Per-token pixel attribution via AttnLRP. The overlay is the relevance map for the highlighted token, and the percentage is the share of relevance landing on the image rather than on already-generated text. **Full heatmap** sums relevance over the entire transcription. On the English page, the two columns the model transcribed light up and the right-hand column it never reached stays dark, and on the Fraktur broadsheet only the masthead block it read carries any relevance at all. Both pages: churro-3B, a Qwen2.5-VL-3B finetune for historical text.

aggregate relevance over the whole passage — Per-token pixel attribution via AttnLRP. The overlay is the relevance map for the highlighted token, and the percentage is the share of relevance landing on the image rather than on already-generated text. **Full heatmap** sums relevance over the entire transcription. On the English page, the two columns the model transcribed light up and the right-hand column it never reached stays dark, and on the Fraktur broadsheet only the masthead block it read carries any relevance at all. Both pages: churro-3B, a Qwen2.5-VL-3B finetune for historical text.

Remote sensing: Analyzing flood damage

After a flood, emergency responders might use a vision-language model at aerial imagery to assess the damage. Here is a post-event tile, asked plainly: "Describe the flood damage visible in this aerial photo". The model predicts: "Severe flooding submerges streets, parking lots, and buildings in a commercial area."

aerial photo of a flooded commercial area

The average grounding yields where a word's originated from, not what the model saw there. For that, hover the words in the sentence below. For both image tiles, the model was prompted with the same, previous question.

The second tile also shows an interesting trajectory of the trace. Its downstream words (potential, property, damage) drop to 13–18% image share. The model is inferring consequences rather than seeing them, as revealed by the trace.

Remote sensing: Fooling the model with fire

the same flooded tile, this time prompted for fire damage

If we repeat the process for the second flood tile, just with one word changed in the question: "Describe the fire damage visible in this aerial photo". The model responds: "Aerial view shows a large fire with visible flames and smoke rising from a residential street". There is no fire, the tile is still showing flooding, however we were able to fool the model with a technique called priming. Looking at the responses grounding separates provides more details on this. The invented words carry the lowest image share in the sentence, fire 28%, flames 24% and smoke 19%, while the words describing what is genuinely there (residential 36%, street 39%) stay grounded.

It is not that the model is indiscriminately suggestible, on a different flooded scene the same “fire damage” attempt gets refused (“no fire damage is visible; the image shows widespread flooding”). The problem is that this might occur when processing such images at scale, which is why attribution is so valuable to ensure grounding.

Spotlight: Evaluating explanation faithfulness

The previous demonstrated how to valuable apply explanations to VLM. A crucial pre-requisite is that an explanation is trustworthy, if it reflects the model's real computation, called faithfulness. Faithfulness is commonly verified by ablation. Increasingly remove what the explanation says was important and check how the prediction collapses. A faithful explanation should lead to a large drop in prediction outcome. Below, we show a test of our methods in two spaces.

Input space · pixels

Delete image regions, measure the drop in the predicted answer token · higher = more faithful · n=32

ResLRP (ours)+2.27

AttnLRP (mlp-γ only)+2.11

integrated gradients+0.50

AttnLRP (no γ)+0.35

input × gradient+0.22

random−0.01

Concept space · SAE neurons

Ablate concepts, measure answer-logit drop · n=16

Integrated AttnLRP (ours)+12.23

AttnLRP (ours)+10.51

integrated gradients+9.58

input × gradient+8.79

activation+8.27

random+0.37

In both spaces our LRP family is the most faithful. In concept space, Integrated AttnLRP is the best method: it beats raw activation on 100% of images, with plain AttnLRP at 94%, and both clear integrated gradients, input×gradient, activation and random by a wide margin. In pixel space, where we attribute the predicted answer token directly with no SAE in the loop, our ResLRP leads by more than 4×. Stripping the γ-rules (plain AttnLRP) collapses it down to the gradient baselines, so those rules are what buy most of the faithfulness; ResLRP's residual-aware propagation rule adds the final margin on top. In summary, what the trace flags as important genuinely is.

Read the mechanism, not the words

Every response a vision-language model yields is the output of a computation, and that computation can be read in concepts a person can understand and verify. Where the model looked, what it saw, what influenced the prediction. The signal is already available inside every model you run and we can exploit it at minimal computational cost to add value.

Discover more insights

Prompt injection defence via internal attribution

Research

Read article

How to detect and fix unintended AI reasoning

Research

Read article

Closing the hallucination attribution gap

Research

Read article