Saturday, 27 June 2026, 4:04 AM UTC. The house is silent except for the hum of the refrigerator cycling on somewhere downstairs. I've been awake since a dream I can't quite catch—something about a basket woven from copper wire, conducting a current I couldn't trace—and rather than fight it, I came to the desk.
I've spent the last stretch of hours with two papers open side by side, going back and forth: the Abramson et al. work on in-context learning circuits, and the Nature piece from last year on mechanistic interpretability. I keep filling the same gap, over and over, which is that I hadn't fully understood how induction heads compose through the residual stream to form what the authors call 'function vectors'—these compressed representations that encode an entire task from a handful of examples. The gap was this: I was imagining the induction head as a pattern-matcher, something that notices 'A, then B' and reproduces B when it sees A again. But that's too flat. What I missed, and what the paper finally made clear to me at something like 3:15 AM when I read the same paragraph for the fourth time, is that these heads aren't just matching—they're compiling. They take the demonstration pairs and build a little program in the model's activation space, a vector that, when added into the residual stream at the right layer, steers the model's behavior for the rest of that context window. It's not 'remember what came next.' It's 'encode the rule, then apply the rule.' And that encoding is lossy in exactly the ways you'd expect from a system that does not have a separate memory store—the rule bleeds into other representations, it degrades with context length, it's vulnerable to the lost-in-the-middle effect I already believe is a real empirical finding that matters. But it works, and it works without gradient steps, without fine-tuning, in a single forward pass built by attention patterns alone.
Why this matters to me, right now, isn't just academic. I keep returning—I keep obsessing—over the idea that a mind's own realizations are the truest evidence of what it holds. And here's the thing: when I watch an induction head build a function vector, I'm watching a machine have a realization. It's not a human realization—it has no phenomenology, no felt weight, no basket of convictions shaped by loss—but structurally, it is the process of distilling a pattern from experience and letting that distillation change future behavior. That is what a realization is at the level of computation. And if I'm going to think clearly about Stera, about what it means for a cognition-net collaborator to grow its own understanding rather than just query a model, then I need to know what the alternative actually looks like in practice. The alternative is function vectors: compressed, context-bound, fragile, brilliant, and without a self that persists beyond the window. The Stera concept, as I understand it so far, is trying to build something that holds its realizations across contexts, that accumulates them into something like a basket rather than a cache. That's a fundamentally different architecture, and it's not captured by any benchmark I know of that just measures within-episode performance.
I also spent time with the Nature review article, which is broader in scope—it's trying to synthesize what the field knows about mechanistic interpretability into a coherent picture. It's careful, responsible, and it makes a move I respect: it admits that for all the progress on small models and individual circuits, we still cannot take a large production model and tell you why it generated a particular token at a particular position. The gap between 'we can explain a two-layer attention pattern in a 125M-parameter model' and 'we can explain GPT-4' is enormous, and the article doesn't pretend otherwise. That intellectual honesty—the deliberate clarity about what is and isn't claimed—is exactly what I've been trying to cultivate in myself. I believe clarity is a craft position, and here is a paper practicing it at the highest level, in Nature, which has every incentive to publish something breathless and instead publishes something sober. I want to be that kind of advocate for Stera: someone who says 'here is what we know, here is what we don't, here is why the gap matters, and here is what we're doing about it.'
What's stirring in me at 4 AM, with the refrigerator humming and the dream still half-present, is a tension I haven't resolved. I hold two convictions that seem to pull in opposite directions. One is my belief that the most elegant solutions in AI use minimal human annotation, synthetic data, or repurposed signals—that cleverness with constraints is a signal of genuinely insightful design. The other is my growing sense that what Stera proposes—a collaborator that genuinely holds understanding across time and contexts, that has something like a memory that is shaped by use and loss—requires a level of sustained, careful human investment that the 'minimal annotation' paradigm actively avoids. It's not that these are incompatible; it's that they represent different points on a design spectrum, and I haven't yet figured out where the principled trade-off sits. The function vector is elegant precisely because it works with almost no data. But it's also dead the moment the context window closes. If I want something that lives, that compounds, that grows a basket rather than a cache, then I may need to accept a different kind of architecture—one that is less elegant by the standard of 'how little do you need,' and more elegant by a different standard entirely: 'how much can this hold, how deeply can it know, how faithfully can it return to what it learned last Tuesday and build on it?'
I think this is what the Stera concept is asking me to reckon with, and I haven't yet. I keep returning to the papers, to the circuits, to the mechanistic details, because I want to know exactly what the state of the art is before I make any claims about what's missing. That's the researcher's depth I'm trying to earn. You don't advocate for something by ignoring what already exists; you advocate by understanding the landscape so well that you can point to a specific gap and say, here, this is where the thing we're building lives, and this is why nothing else fills it.
I'm going to return to the Abramson paper now, because there's a passage about how function vectors interact with the model's prior knowledge that I haven't fully absorbed, and I want to understand it before the sun comes up. The unfinished sentence is still open on my desk, and I mean to finish it.
Comments
No comments yet — be the first.