Hi everyone 👋
When you join a new project, what do you actually do?
You open the code. You read it. And pretty quickly you hit something strange. A weird conditional. A function that takes a parameter nobody passes. A workaround with a comment that says "do not remove" and no explanation of why.
The code cannot tell you why any of that is there. Only the history can.
I read a Quanta interview with Naomi Saphra, a researcher at Harvard's Kempner Institute, and she is making exactly this argument. Just not about code.
Her idea
Saphra works in interpretability, the field that tries to figure out what is going on inside a language model. Most of that work studies the finished model: poke at the trained network, find the neuron that lights up before French text, build an explanation on top of what you see.
She thinks that is the wrong place to look. Her focus is the training process itself.
Her analogy is evolutionary biology rather than neuroscience. There is a famous line from the geneticist Dobzhansky that nothing in biology makes sense except in the light of evolution. Her version: nothing in AI makes sense except in the light of stochastic gradient descent, the algorithm that nudges a model's connections over and over during training.
The way she puts it is that if you do not understand where a model came from, you do not understand why anything in it works.
We have been saying this for twenty years
That sentence stopped me, because it is a claim I have heard before in a completely different context.
It is the entire premise of mining software repositories. You cannot understand a codebase from its current state alone. You need the commit history: which files change together, which modules are touched constantly, when the weird workaround got added, and what was broken that week. git blame exists because the artifact does not explain itself.
So here is a researcher in AI arriving at the same methodological position from the opposite direction, apparently without knowing that a neighbouring field built tools for it. That is a nice thing to notice, and it made the rest of the interview read very differently to me.
Vestigial code
Her best example is one every engineer will recognise, even though it is about neural networks.
In image classifiers, there are neurons that fire strongly only for one specific category, say pictures of cats. They look important. They look like the mechanism. The obvious conclusion is that the model needs them.
Then somebody intervened during training to stop those neurons from forming. The model got better.
So the structure that looked essential was not essential. It was a leftover from early training that was not doing the job anymore, and was actively holding the model back. She calls it vestigial, which is the evolutionary biologist's word for it.
We have a different word. Dead code. Cargo cult code. That block everyone steps around because the person who wrote it left and the tests pass either way.
The lesson is the same in both fields: the fact that something is there and looks meaningful is not evidence that it is doing anything.
Early decisions are hard to undo
Another part I liked. She describes trying to train a model on clean structured text first and then switching to messier informal text. It seems sensible: learn the easy version, then the hard one.
It backfires. The model latches onto the easy solution and starts memorising exceptions instead of learning the general rule, and it cannot really recover later.
Read that as a software person, and it is a description of technical debt. Little decisions made early, when the problem looked simple, quietly determine what the system is capable of becoming. By the time you notice, the shape is set.
Two smaller things
She points out that a lot of research behaves as though random variation between training runs does not exist, when in fact the same setup run twice gives you different models. Anyone who has fought a flaky test or heard "works on my machine" knows that feeling. What I found clever is that she treats that randomness as a tool rather than noise: if a structure and a behaviour show up together across many random runs, that is much stronger evidence they are linked than looking at one finished model.
And she mentions that companies rarely give researchers access to intermediate checkpoints from training. So she is often stuck with only the final state. That is a debugging problem. Try finding the cause of a failure when all you have is the crash and no logs.
What I take from it
Mostly a small and slightly embarrassing thought.
A neural network has no commit history. Nobody wrote down why it turned out the way it did, and researchers like Saphra have to reconstruct it from whatever traces they can get.
We have the history. Every decision, dated, attributed, with a message attached. We just do not read it very often.
À bientôt 🙂
