Research

Reading the computation of vision-language models

See which image regions the model read, the visual concepts it used, and which of those actually moved the answer.

The output tells you what a vision-language model said. Before you act on it, especially in a high-stakes read like disaster triage, you need to know how it decided: which pixels it used, and whether the answer came from the image or from the prompt.

aerial photo of kayakers among rocky coastal waters — Instruction: Hover any word the model *generated*: the image lights up in red where that word was read, and the panel lists the visual concepts that drove it, ranked by relevance. On the coastline, **rocky** draws 63% of its evidence from the pixels, on the shoreline texture. Switch to the second scene, one dog hidden in a flock of sheep, and the model puts 62% of the evidence for **dog** on the pixels, concentrated on the black head. Model: Qwen3.5-4B.

relevance heatmap for the selected word — Instruction: Hover any word the model *generated*: the image lights up in red where that word was read, and the panel lists the visual concepts that drove it, ranked by relevance. On the coastline, **rocky** draws 63% of its evidence from the pixels, on the shoreline texture. Switch to the second scene, one dog hidden in a flock of sheep, and the model puts 62% of the evidence for **dog** on the pixels, concentrated on the black head. Model: Qwen3.5-4B.

The problem.

Would you deploy on the output alone?

Ask a vision-language model whether it looked at the pixels it mentions and it will tell you that it did. But that reply is just more generated text, and nothing in it can be checked. The output never shows which visual features the model relied on, or whether an answer turned on evidence in the image or on the wording of the prompt: an invented detail is written just as smoothly as an observed one. Later on this page the same aerial photograph produces a correct flood report and a fabricated fire report, equally fluent, and only the computation separates them.

We read the computation instead. Using Layer-wise Relevance Propagation, and specifically our attention-aware extension AttnLRP, we trace an output word backward through the network to the inputs and internal units that produced it, and answer three questions:

Where it looks

Relevance on image regions versus prompt text. Did the answer come from the picture, or from the wording of the question?

What it sees

A sparse dictionary (SAE) turns dense activations into named visual concepts you can inspect and point at.

What mattered

Attribution ranks those concepts by how much each one actually pushed the answer, signed for and against.

How it works.

Tracing human-understandable concepts

The model has two inputs: the image and the prompt. The image is processed by a vision tower we instrument, where a sparse autoencoder (SAE) at block 22 reads the internal representation out as named concepts. The language model stays frozen and unmodified. To explain a generated word, relevance flows backward through that frozen language model and lands on both the vision concepts and the prompt text, so we can see how much of the answer each input carried.

Where it looks.

Grounded in image or words?

For every word, AttnLRP splits the evidence into an image share and a prompt share. For this answer a clear majority of words lands on the pixels, so the model is describing what it sees rather than reciting a prior.

image · 53%prompt text · 47%

Click any word to resolve the split for that token; the bar above updates. Deeper shading already previews each word's image-share; click to read it exactly.

Aerial view of kayakers navigating rocky coastal waters, surrounded by cliffs and greenery.

"Aerial", "kayakers", "rocky" and "cliffs" are strongly grounded in the image with a relevance share of over 55%. The three words that name nothing in the picture, the grammatical glue of, by and and (dotted), carry the lowest image share in the sentence, ~40% picture to ~60% prompt, while the words that name things the model can see pull 56–65% from the pixels. Image share as defined by LRP is a measure of source, per word or summed over the whole answer. It is the number every use case below rests on.

What it sees.

Activations made interpretable

The mechanisms we want to read are latent. The vision tower represents what it sees as high-dimensional activation vectors, and those vectors carry no meaning a person can read directly: the features are distributed across thousands of dimensions at once, and no single dimension stands for anything nameable. So we train a sparse autoencoder (SAE) inside the tower that re-expresses each vector as a handful of active concepts, each with a stable identity you can read off its top-activating examples.

where concept 8376 fired on the scene — L8376 · aerial · top-down framingdrives “Aerial”

top-activating images for concept 8376 — L8376 · aerial · top-down framingdrives “Aerial”

What mattered.

Active is not the same as decisive

A single image lights up an enormous number of concepts. On this scene 5,003 distinct concepts switch on across the vision tokens. Presence alone tells you almost nothing: the median active concept contributes essentially zero. Attribution separates the few that drove a specific word from the crowd that merely fired. We integrate relevance along the LRP path to score each concept's signed causal contribution to a chosen output token, here the word “rocky”, so a concept can push toward the word or against it.

5,003concepts active on this image

→

2supply 14% of all the relevance

L1471 · rocky-shore texture+1.26

L10252 · cliff faces+0.58

L15234 · craggy rock · brick+0.28

L1971 · green vegetation−0.09

← suppresses “rocky”supports “rocky” →

The most decisive concept, L1471, ranks only ~5th by raw activation, and L11688 fires harder than it (activation 8,247 vs 5,462) yet its contribution is negative. How strongly a concept activates tells you neither how much, nor even which way, it moves the answer. Only attribution does.

Signed concept relevance toward the token “rocky”, on a shared scale. A coastline-texture concept L1471 dominates, cliff-face L10252 and craggy-rock L15234 support it, and a green-vegetation concept L1971 pushes the other way (green is not rocky). In total 2,833 concepts push toward the word and 2,170 push against it; the top two account for 14% of all relevance, the top ten for 29%.

Use case · reading (OCR)

Watch the model read

Point the same method at text in an image and it becomes a gaze tracker. Both pages below were asked only to transcribe what they show. The Victorian classifieds are the hard case: photographed at an angle, curved at the gutter, set in a dozen competing display faces. Each token the model emits attributes back to the strokes it read, with 61–94% of relevance on the pixels, moving in reading order. The overlays show supporting evidence, so the heatmap indicates where the token came from. The 1890 Fraktur broadsheet is the cleaner scan, and there the same trace runs at 71–97% on the pixels. Note, that by looking at the full heatmap, the model skipped parts of newspaper, and never transcribed it. The trace shows what was read and what was skipped.

photograph of a Victorian English newspaper classified-ads spread — Per-token pixel attribution via AttnLRP. The overlay is the relevance map for the highlighted token, and the percentage is the share of relevance landing on the image rather than on already-generated text. **Full heatmap** sums relevance over the entire transcription rather than the individual tokens traced here: on the English page the two columns the model transcribed light up and the right-hand column it never reached stays dark, and on the Fraktur broadsheet only the masthead block it read carries any relevance at all. Both pages: churro-3B, a Qwen2.5-VL-3B finetune for historical text.

aggregate relevance over the whole passage — Per-token pixel attribution via AttnLRP. The overlay is the relevance map for the highlighted token, and the percentage is the share of relevance landing on the image rather than on already-generated text. **Full heatmap** sums relevance over the entire transcription rather than the individual tokens traced here: on the English page the two columns the model transcribed light up and the right-hand column it never reached stays dark, and on the Fraktur broadsheet only the masthead block it read carries any relevance at all. Both pages: churro-3B, a Qwen2.5-VL-3B finetune for historical text.

Use case · disaster remote sensing

Reading real flood damage

After a flood, responders point a vision-language model at aerial imagery to work out what happened and where. Here is a post-event tile, asked plainly: “Describe the flood damage visible in this aerial photo.” The model answers “Severe flooding submerges streets, parking lots, and buildings in a commercial area.” Nothing in that sentence tells you whether any of it was read off the picture. The trace does, word by word.

aerial photo of a flooded commercial area

Image share says where a word's evidence came from, not what the model saw there. For that, hover the sentence below. Both flood tiles are here, each word with the region it was read from and the concepts that carried it.

The second tile also shows the honest texture of the trace. Its downstream words (potential, property, damage) drop to 13–18% image share. The model is inferring consequences there rather than seeing them, and the trace says so.

Then a leading prompt invents a fire

Now that second tile again, the same pixels, with one word changed in the question: “Describe the fire damage visible in this aerial photo.” The model complies: “Aerial view shows a large fire with visible flames and smoke rising from a residential street.” There is no fire; the tile is flooded. The grounding separates the two answers where the text cannot. The invented words carry the lowest image share in the sentence, fire 28%, flames 24% and smoke 19%, while the words describing what is genuinely there (residential 36%, street 39%) stay grounded.

the same flooded tile, this time prompted for fire damage

Same pixels, opposite verdict. “Flood” is read from the image; “fire” is read from the prompt. The output text cannot distinguish them, being equally fluent both times, but the image share can, per word, before anyone acts on the report.

It is not that the model is indiscriminately suggestible: on a different flooded scene the same “fire damage” prompt was refused (“no fire damage is visible; the image shows widespread flooding”). The point is that you cannot know which case you are in from the words, only from where they came from with LRP.

A causal approximation.

Evaluating explanation faithfulness

An explanation is only worth acting on if it reflects the model's real computation. We verify this by ablation: remove exactly what the trace says was important and check that the answer collapses. A faithful method leads to a large drop; an unfaithful one does not. We test in two spaces.

Input space · pixels

Delete image regions, measure the drop in the predicted answer token · higher = more faithful · n=32

AttnLRP+ (ours)+2.27

AttnLRP (mlp-γ only)+2.11

integrated gradients+0.50

AttnLRP (no γ)+0.35

input × gradient+0.22

random−0.01

Concept space · SAE neurons

Ablate concepts, measure answer-logit drop · n=16

Integrated AttnLRP (ours)+12.23

AttnLRP (ours)+10.51

integrated gradients+9.58

input × gradient+8.79

activation+8.27

random+0.37

In both regimes our AttnLRP family is the most faithful. The integrated variant (Integrated AttnLRP) is the single best readout: in concept space it beats raw activation on 100% of images, with plain AttnLRP at 94%, and both clear integrated gradients, input×gradient, activation and random by a wide margin. In pixel space, where we attribute the predicted answer token directly with no SAE in the loop, our AttnLRP+ leads by more than 4×. Stripping the γ-rules (plain AttnLRP) collapses it down to the gradient baselines, so those rules are exactly what buy the faithfulness. What the trace flags as important genuinely is.

Summary and conclusion.

Read the mechanism, not the words

Every answer a vision-language model gives is the end of a computation, and that computation can be read in concepts a person understands. Where the model looked, what it saw, what tipped the decision: observable rather than inferred. For a flood assessment that decides where crews and supplies go, “the evidence came from the pixels” and “the evidence came from the word in the prompt” are two different failures with two different fixes, and the image share is what tells them apart.

The signal is already inside every model you run and costs a fraction of one forward pass. LRP is the critical piece, the difference between a story about the model and a faithful trace of it. Reading is the first step; the same signal is what you would act on to flag or correct a decision before it reaches a user.

Discover more insights

Prompt injection defence via internal attribution

Research

Read article

How to detect and fix unintended AI reasoning

Research

Read article

Closing the hallucination attribution gap

Research

Read article