How to detect and fix unintended AI reasoning

A neural network trained to distinguish huskies from wolves reaches strong accuracy on the test set. The feature the model learns on most has nothing to do with the animals. It is snow. Wolves in the training images were typically photographed in snowy settings; huskies were not, and the model had learned to read the background rather than the subject.

Still, the model is right. The problem is, it learned undesired connections in the data — often called spurious correlations — that should play no role in the prediction, a so-called bias. This particular bias was discovered over a decade ago. The root cause still exists in todays AI systems.

Clever Hans, the horse that could not actually count

This kind of behaviour has a name. Researchers call it "Clever Hans", after the early 20th century horse that amazed the scientific world by appearing to solve arithmetic tasks. It was later discovered that Clever Hans wasn't calculating at all, but reading subtle cues in his trainer's body language. He was arriving at the right answers, but for the wrong reasons.

Modern AI systems do this constantly. And we rarely catch it.

AI doesn't come with a manual

When engineers build an aircraft, every component has a defined role. The wing produces lift. The aileron controls roll. Each part can be verified against its specification before the system flies.

Neural networks are different. They develop their capabilities through optimisation across billions of parameters. No one assigns the role of an individual neuron. No one assigns each weight a particular job. The model works, but how it works is largely invisible, even to the people who built it.

Why it's time to tighten the reins

The current state of AI has an uncomfortable asymmetry. Capability is racing ahead of verification. Models are being trusted with decisions and execute actions that affect real people, and the standard practice for confirming they will behave as intended is still essentially "test on a held-out set and hope". Held-out sets share the biases of the data they came from. Spurious correlations in training propagate quietly into evaluation.

What traditional engineering offers that AI currently does not is component-level verification. You can look at a wing and check that it is shaped correctly. You can audit a sorting routine against its specification. We need that kind of audit for neural networks: specific enough to point at individual components and ask whether each one is doing the right job.

This invisibility is a massive safety issue in practice:

• A melanoma classifier that secretly relies on whether a certain camera setting was used will fail the moment that incidental cue disappears.
• A hiring model that has internalised correlations between profession and race or gender will reproduce that pattern at scale.
• A self-driving system that has learned to associate stop signs with a particular background is one unfamiliar street away from a serious mistake.

This safety gap can only be closed effectively if we can understand model reasoning at scale.

SemanticLens, a new window into the model's mind

Inside a neural network, every component (a neuron, a filter, a feature direction) corresponds to some pattern in the input that makes it activate. We collect those patterns and push them through a foundation model that has learned to map images and text into a shared semantic space.

Each component of our model now has a vector in a space we can search, label, and compare.

Search. Ask "does this model have a systematic bias for automobiles?" with a text query and get back the components most aligned with the concept. No manual neuron-by-neuron inspection.

Describe. Walk through every component in a layer and assign it a human-readable label automatically.

Compare. Place two models, layers or components side by side and see which concepts they share and which are unique to each.

Audit. Define what a model should be relying on (for a melanoma classifier, things like asymmetry, irregular borders, and colour variation), and measure how much of its actual reasoning aligns with that expectation.

This new level of insight also builds the foundation for targeted evaluation and steering of model behaviour, as shown in the semantic lens prototype.