Poking every neuron in a small model

Most interpretability tools start from activations. You run text through a model, look at which neurons fire, and guess what they mean. neuron-explorer starts from steering. For Qwen3.5-4B, it pushes each MLP neuron in a layer up and down on a few prompts, keeps every rollout, and shows the results in a local web app with a 3D view of the model. The question it answers is about cause: what happens to the text when you turn this one neuron up?

How it works

There are two parts: a sweep and an explorer.

The sweep runs on a GPU pod. It first records stats for each neuron: activation extremes by sign, down-projection columns and residual norms. Then it steers every neuron in a layer. Doses are fractions of the median residual norm leaving that layer, so a dose means the same thing across layers. In the default main_rel design, three prompts (who, snake, orange) get doses of -0.7, +0.5 and +0.8. A fourth prompt (free) gets +3. Each prompt also gets one unsteered baseline. Every rollout is saved with its tokens, logprobs and top-k per step.

Ranking is the part I like most. A script scores each neuron by its distinctive words and word-cluster mass, then compares that to 256 random-direction controls. So a neuron only counts as interesting if steering it does more than steering a random direction.

The explorer then gives you several views. The map shows layers in perspective, with one point per neuron. The list ranks neurons. The neuron view shows steered rollouts by prompt and dose, plus a three-column chat at -, 0 and +. There is also a UMAP of each layer's rollouts. The last view is a Claude tab that runs the claude CLI with the explorer as an MCP server. From there Claude can read neurons, run the live model for steering and ablations, and move the map.

Try it

uv sync
uv run python explorer/server.py

The cached data ships with the repo, so this serves at localhost:8791 without a sweep, and it loads Qwen3.5-4B on MPS, CUDA or CPU for the live features.

My take

Ranking against random controls is a good habit, and doses tied to residual norms make the numbers comparable. Shipping the caches matters too. You can look around on a laptop before you ever rent a GPU.

There are limits. It supports one model, Qwen3.5-4B. Running your own sweeps needs a CUDA 13 pod and a specific steering fork of vLLM (RhizoNymph/vllm). Four prompts per neuron is a thin sample, so read the rollouts as leads, not proof. Also note that the Claude tab gets file tools and a shell scoped to your home folder. That's useful, but you should know about it before you open it.

If you do mechanistic interpretability or you're curious what single neurons in a small model actually do, this is worth a clone. If you want a general tool that works on any model, this isn't it yet.