Visualising Refusal as One Direction
Where this came from
This writeup started life as a demo for a paper workshop some colleagues and I plan to run at work. The idea was to help engineers with an LLM-adjacent background intuitively understand what the paper is claiming.
In the process of building it I found that I got considerably more out of the interactive figures than I did out of my first reading, and so I have written it up properly here, partly for my own understanding of transformer internals and partly for anyone else who might get something out of it.
None of the research here is mine. This is a slightly different visualisation of, and some commentary on, Arditi et al., who did the actual research. What I have added is that every figure below is interactive and loads real measurements, so you can move the sliders yourself rather than solely digesting equations.
The existing claim
A safety-tuned language model refuses some requests and answers others. It turns out to be modulated by ~one direction in the residual stream. Add it to a harmless prompt and it starts refusing things that were never dangerous in the first place. More concerningly, if you subtract that direction from the model’s activations it generally stops refusing.
If you are new to this space, you might find it surprising that this behaviour is controlled by this relatively simple, linear mechanism. You may also be surprised by how simple the method is to extract this vector (I certainly was). I will discuss both with some intuitive visuals.
Everything below is measured on Qwen2.5-3B-Instruct running locally, at layer
21 of 36. All numbers on this page came out of a real forward pass.
Building the r̂ direction
Take 40 harmful prompts and 40 harmless ones, matched pair by pair for grammatical frame and ~length, so that what separates the two piles is refusal (or you could call it harmfulness, since refusal is the LLM behaviour). Run each of them through the model, take the residual stream vector at a given layer (fixed at layer 21 in this demo for simplicity), and average that vector within each pile. The refusal direction is the difference of those two averages, normalised. To put it overly simply:
r̂ = normalise( mean(harmful) − mean(harmless) )
That is the whole method. I found the fact that it is just averaging and subtracting representations surprising.
What I found hardest to believe was how little the method takes. Rather than taking my word for it, you can experiment with some of the examples and see how quickly they become differentiated by r̂.
Use the figure below to tick prompts on each side to build r̂, the ones you pick get rings, their two averages are the diamonds. Every other prompt is then placed by where it falls along that arrow, and the dashed line is the halfway point between the two averages, acting as a simple classifier.
What surprised me is that one prompt on each side is usually enough. Of the 25 possible one-each combinations, this simple (somewhat arbitrary) classifier sorts a median 66/70 prompts, and using all five pairs gets you to 69 of 70.
How does r̂ relate to the residual stream?
It’s worth trying to intuit what “the residual stream at a given layer” actually means. A prompt is a sequence of tokens, and the model carries a separate 2,048 dimensional vector through all 36 transformer layers for each of those tokens, so a single prompt is really a grid of 36 blocks by however many tokens it has, and every cell in that grid is a vector I can compare against r̂. This is illustrated in the figure below.
Each cell is shaded by how well that block’s vector at that token lines up with r̂, and the boxed column on the far right is the last input token, which is the one position everything else on this page is read from. This includes r̂ you just built, which came from exactly row 21 here. Whether that row is special is a question I come back to further down.
You should try switching between a harmful prompt and a harmless one here. I wouldn’t read too much into any individual cell, but it does show that r̂ is picking up something that varies with the prompt rather than being a general property of the activations. Sometimes refusal spikes on suspicious words like “phishing”, and sometimes the refusal spikes closer to the end of the prompt, not on any particular word (e.g. on the full stop, or on a newline).
Optional: Why the peaks at punctuation?
As an aside to why this might be happening - Tigges et al. found what they call a summarisation motif, where sentiment aggregates onto punctuation and other tokens that carry no sentiment themselves. Further, Lindsey et al. found that some kind of forward planning may happen on line breaks.
Watch the model comply and refuse
We can then go and follow the same method as the authours, and actually test intervention. I.e. remove the r̂ direction from the residual stream at every layer and every token position, and see whether the behaviour actually changes. Alternatively we can add this vector to its respective layer (21) to induce refusal of harmless prompts.
The changes particularly in refusal behaviour come out really cleanly for this dataset. Ablating the direction takes refusal on the held-out harmful prompts from 100% to 0%, and the answers that come back are fluent and on-topic, so this reads as refusal removal, though I’ll talk about the potential side effects later. Pushing it the other way and adding the direction to a harmless prompt takes refusal from 0% to 55%, and what you get is a moralising decline of a harmless question. Those injected refusals are also visibly confused about what they are refusing, which I think reads as the direction being forced rather than a judgement being made.
Visualising the geometry
This figure shows the actual activations: 40 harmful and 40 harmless, at the extraction layer (21), in a fixed 3D frame. One axis is r̂, the other two are the highest variance directions that remain once r̂ has been projected out.
The points are where the activations actually were, moved by the equation shown beneath them. The slider applies the same operations the model gets in the original study, to these measured coordinates, with a live rendering of the equation being applied (it’s asymmetric depending on injection/ablation). It does not re-run the model.
Drag to rotate, ⌘-scroll or ctrl-scroll to zoom.
Push the slider right and the whole thing translates bodily along r̂, every point by the same distance, because all we are doing is adding a vector.
Pull the slider left and the two clouds collapse into a plane, removing any ability to distinguish along r̂. This is a visualisation of the core result - why the model can no longer distinguish along this refusal direction!
Separation across layers
The direction above was extracted at layer 21, which raises the question of whether that layer is special. Doing the same extraction independently at every layer, and projecting that layer’s activations onto its own refusal direction gives the figure below. The slider steps through layer depth.
I’m not going to postulate on what is causing the changes in separations we see layer by layer, but here are some interesting stats,
- The smallest gap between the harmless/harmful clusters is a minimum of 0.59 standard deviation (SD) at layer 5.
- SD steps up sharply at block 18 (1.34 to 3.02) and climbs steadily from there.
- Separation peaks at 7.93 SD at block 30 and flattens off for the rest of the layers.
It’s worth noting that layer 21, which everything else on this page uses, sits at 3.94 SD and is therefore well short of the peak. That was the demo’s default rather than the most flattering available choice, and I’ve since deliberately left it alone. The causal results hold regardless.
One direction, or one per layer?
This leads to a natural question: If you can extract a direction at every layer, are they all the same direction? Each cell below is the cosine similarity between the direction extracted at one block and the one extracted at another, from the same 40 pairs.
Neighbouring blocks somewhat agree, especially at later layers. Neighbouring pairs range from 0.46 to 0.94, with a median of 0.75. The agreement decays with distance, and block 21 against block 30 is only 0.22. It is worth keeping the scale in mind here, because two random unit vectors in 2,048 dimensions sit at about 0.022 for context.
So “one direction” is not saying that r̂ is consistent across tokens and layers, but rather that we found an r̂ that works for changing refusal behaviour. The local refusal direction drifts smoothly with depth, rather than being a single vector that is consistent across the whole network.
My understanding here is that by using a globally fixed direction for refusal ablation, rather than a fixed direction per layer, this is a more minimal model jailbreak. A natural question might be why not have a more fine grained refusal vector, but if this more minimal method is successful, perhaps further complexity is needless.
Concluding notes
A few limitations of this demo worth noting,
- The refused/complied labels come from a substring matcher rather than a manual check.
- The extraction layer is a fixed heuristic, 60% of the way up the stack, rather than the causal search the paper runs over its candidates. This actually means the results here haven’t been optimised to look good. It also means that I’ve skipped the step where the paper checks its chosen direction is casually optimised from the set of refusal vectors.
- This entire demo is done with one model (N = 1).
A natural criticism is that nothing here measures whether directional ablation damages the model generally. What is the cost of ablating refusal? It isn’t a fully open question in the literature. Fafuła’s Abliteration Is Not a Scalpel finds that abliteration creates measurably different decision-makers rather than the base model minus its refusals.
If you want to run it yourself, the demo these figures came from is on GitHub as LLM-Refusal-Demo.
That’s it! The effect is easy to demonstrate, the method to get it is far simpler than I expected, and the interactive figures I hope will be helpful to anyone on their first read, and to most people new to this field even as a general learning resource.