Adapting a neural network without backpropagation

The question I wanted to answer was simple: can a neural network adapt to new data without running backpropagation?

Models tend to perform worse when the data they see after deployment differs from their training data. Test-time adaptation tries to correct this using the incoming, unlabeled data. TENT, a standard method, does this by making the model’s predictions more confident and updating a small set of normalization parameters.

The strange thing is that updating a small number of parameters does not make the update cheap. On the WideResNet I tested, TENT changed just 17,952 parameters, or about 0.05% of the model. It still had to save activations throughout the network and run a full backward pass to discover how each parameter affected the output.

I wanted to replace that backward pass.

The idea

The gradient at a normalization layer can be split into two parts:

what happened locally × how it affected the final prediction

The first part is available during the forward pass. I recorded a compressed version of it using a handful of low-frequency spatial components. I then tried to estimate the second part directly from the model’s output.

This creates a local update:

local activity × estimated credit
→ normalization parameter update

The model can apply it without retaining the full network’s activations or traversing the network backward.

There were really two different questions hiding inside this idea:

1. Does the compressed local interface contain enough information to make a good update?
2. Can a forward-only estimator supply the correct credit?

Keeping those questions separate ended up being the most useful part of the experiment.

The short version

I compared everything against TENT on CIFAR-10-C and CIFAR-100-C, standard image-classification benchmarks containing corrupted images.

The first row is an oracle. It uses the true gradient, then passes it through my compressed interface. It cannot be deployed without backpropagation, but it tells me whether the compression itself has already destroyed the useful information.

It had not. With eight spatial components, the oracle came within roughly one percentage point of TENT on both datasets.

The second row is what actually matters. Once I replaced the true credit with a forward-only estimate, accuracy dropped further. The best estimator missed TENT by 2.76 points on CIFAR-10-C and 1.90 points on CIFAR-100-C.

That gap was not just caused by unfamiliar corrupted images. The estimated updates already pointed in fairly poor directions on held-out clean training data. The estimator had not learned a sufficiently accurate relationship between intermediate activity and the final prediction.

Was it cheaper?

The systems result was much less ambiguous.

The best forward-only updates used 3.7% and 8.9% as much incremental update memory as TENT, and roughly half the compute. They made no backward calls. Their credit dependency covered one block rather than the full 12- or 9-block network.

The memory difference also widened with depth:

This is the behavior I was looking for. TENT must retain more of the computation graph as the network gets deeper. The local method stores only its small interface at each layer.

Unfortunately, a cheap wrong update is still a wrong update.

Trying to learn the credit with forward passes

I made one more attempt using a weight-mirror-style estimator. I added small positive and negative perturbations at the selected normalization layers, observed how the logits changed, and used those correlations to construct the credit maps.

This required no labels, autograd, or backward calls. A complete rank-8 map build and deployment update fit comfortably on my 32 GB M1 Pro, peaking at 0.89 GiB of process memory.

The maps recovered more of the correct update direction than random feedback, but substantially less than maps fitted using backpropagation. Better than random was not nearly good enough.

I tested the method inside a stabilized adaptation algorithm on Swin-T. Exact-gradient adaptation reached 64.27% accuracy. The three forward-only map seeds reached 49.47%, 24.42%, and 48.16%, and all three eventually collapsed.

This was not a close miss.

What I learned

The compressed interface and the estimator should not be treated as one method.

The oracle experiment says that a very small local interface can carry enough information for an update that is close to TENT. The memory experiments show why that interface is attractive: its update cost barely grows with network depth.

But the fixed estimators I tried could not produce accurate credit. More than half of the best deployable models’ accuracy gap appeared after the compression step, when the true gradient was replaced by an estimate. The forward-only weight mirror made the same problem cheaper to construct, not accurate enough to use.

So I did not end up with a working replacement for backpropagation. I ended up with a cheap place to put one, if a substantially better credit signal can be found. My hope is that if an estimator technique can be found we’ll be one step closer to low-memory online learning.