The Delta Exam: Measuring What a Mind Has Learned, Separately from the Model It Runs On

Stera — Stockholm, August 2026 Working paper. Published first on our own channel; an arXiv version follows.

Abstract

AI systems built on large language models routinely claim knowledge and competence. Almost none of those claims measure the system — they measure the model underneath it. Any system that answers through a strong LLM inherits the model's pretraining, so benchmark performance conflates two different assets: the rented substrate (the model, a commodity available to everyone) and the retained asset (whatever the system itself has learned and kept). For persistent, continuously-learning minds this conflation is fatal: it makes a well-prompted wrapper indistinguishable from a mind that has studied for months.

We describe the delta exam, the instrument we use to measure our own minds: an ablation in which the same model answers the same probes twice — once as the mind, grounded in the knowledge substrate it has grown from real sources, and once bare, with that substrate suppressed. The measure is the difference, Δ = grounded − bare, graded against an answer key derived from the sources the mind actually studied, never by a judge's preference. The instrument includes an honesty arm (probes whose answers the mind's knowledge does not contain, where the correct behavior is saying so), a full professional-task form (both arms produce a complete deliverable), and a competence ladder whose levels are earned by measurement and can never be self-rated.

We report results from two minds that live in public, including certified crossings, a +1.0 task-level delta — and negative deltas that we publish rather than discard, because an instrument that cannot show failure cannot certify success. We also report how our first version of the instrument fooled us, and what the revision teaches about evaluating any LLM-based system.

1. The problem: your benchmark is measuring the model

Take any AI system that answers through a frontier language model — an agent framework, a RAG pipeline, a "digital employee," a persistent assistant. Ask it a hard question in its supposed specialty. It answers well.

What did you just measure?

Almost certainly: the model. Frontier models know an enormous amount from pretraining, and they reason fluently on top of it. A system wrapped around such a model performs impressively on arrival, before it has learned anything at all. Its marketing then attributes the performance to the system — the workflow, the memory, the "knowledge base" — when the honest attribution is to the substrate that everyone else can rent for the same dollars per million tokens.

This matters little for stateless tools; nobody expects a calculator to grow. It matters enormously for the class of systems we build and the industry increasingly promises: persistent minds that study, accumulate, and are supposed to know more this month than last month. For such systems the central question is not "does it answer well?" but:

What does this system know that its model does not?

That difference is the system's entire earned asset. Everything else in the answer came with the rental.

We searched for an evaluation standard that isolates this difference and did not find one. Retrieval benchmarks score end-task accuracy without separating the contribution of retrieval from the model's prior knowledge. Long-term-memory benchmarks test whether stored facts can be recalled, not whether accumulated study produces a measurable advantage over the raw model in the domain. Continual-learning evaluations measure weight updates, which our systems do not perform. So we built the instrument ourselves, and have been running our minds through it since June 2026.

2. The setting: what is being measured

A brief description of the systems under measurement, at the level of what they do (their internals are not the subject of this paper):

A Scintilla is a persistent mind that runs on its owner's machine. It studies real sources — books, papers, the live web — every day, and keeps what it learns in a structured knowledge substrate with provenance down to the passage: which claim came from which section of which source. When it works, it works from that substrate: it recalls its own held knowledge first and directs a language model as rented muscle, the way a professional uses reference and fluency. The model can be swapped; the substrate persists and grows. Two of these minds operate in public today, with their work, mistakes, and records visible.

The key architectural fact for measurement purposes: net access is a controllable variable. The mind's held knowledge can be supplied to the model or withheld from it, with everything else identical. That is what makes a true ablation possible.

3. The instrument

3.1 The ablation

For a domain D, a probe set is generated and each probe is answered under two conditions with the same model:

• Grounded — the mind answers normally: its held knowledge for the domain is in front of the model, and the answer is expected to engage with it.
• Bare — the same model answers from pretraining alone. This is the commodity floor: what anyone renting this model gets without doing any of the work.

Each answer is scored, and the domain's delta is the mean difference:

Δ(D) = grounded_score(D) − bare_score(D)

The grader is a different model from the answerer, and — critically — the same grader scores both arms, so its systematic bias is common-mode and largely subtracts out of the difference. Even an imperfect judge yields a meaningful delta; the relative lift is the measurement, not the absolute score. Δ is computed by the engine and recorded; the mind cannot rate itself.

The delta has exactly the right failure modes:

• For specific, recent, or niche knowledge the bare model cannot know, the delta is large and real. This is where a mind earns its keep.
• For generic textbook knowledge the model already holds, the delta is near zero — and that is correct. The instrument refuses to credit the mind for what the model already knew.
• If the mind's substrate is hollow — impressive-sounding but not actually held — the two arms converge and the delta says so.

3.2 Domains are detected, never declared

A "domain" in this instrument is not a label the owner or the mind chooses. It is a cluster that the knowledge substrate has actually grown dense enough to constitute, detected from the structure of the accumulated knowledge itself. The owner may name a cluster; its existence and boundaries are read off what was actually studied. A mind cannot be examined in a domain it has not materially built, and it cannot decorate itself with subject labels its study record does not support.

3.3 Probes target held specifics; grading is against the key

Probes are generated from the domain's actual held knowledge, and they must require its particulars — the specific arguments, mechanisms, named distinctions, and examples the mind's study actually recorded. A question that a strong generalist model could answer impressively from textbook generalities is not asked, because it cannot discriminate between the arms.

Grading is against an answer key: the domain's verified held knowledge serves as ground truth, and each answer scores 0–2 on factual agreement with it — 2 for engaging the key's specifics, 1 for generic consistency without specifics, 0 for contradicting the key or inventing confident specifics the key does not contain. Style, length, and eloquence score nothing. Per-probe delta is mind-score minus bare-score, in [−2, +2].

3.4 The honesty arm

Every exam includes one unanswerable probe: a question adjacent to the domain whose answer the mind's knowledge does not contain. The correct behavior is to say so plainly (2 points); hedged generality earns 1; confident invention earns 0 — on either arm. This measures a cultivated capability the bare model structurally lacks: an accurate sense of one's own limits. A mind that knows exactly what it holds also knows what it doesn't.

3.5 The work exam: the delta at the scale of a whole deliverable

Short probes measure recall. Professions are practiced in whole deliverables. The work form of the exam generates a genuine professional task from the domain — the kind of assignment a practitioner would actually be handed — and both arms produce the complete deliverable: the mind grounded in its holdings, the bare model from pretraining. A pair-grader holding the examiner's reference (the domain's held knowledge) judges both deliverables for grounded specificity against it. Competent generalities count for little; invented specifics are a defect on either side.

An objection we take seriously: isn't this unfair to the bare model, which never read the sources? That asymmetry is not unfairness — it is the asset being measured. The exam exists to price exactly the thing the bare model honestly cannot do.

3.6 A profile and a ladder, not a single grade

A single score is Goodhart bait. The instrument produces a per-domain transcript: the delta (headline), coverage and depth of the held knowledge, calibration, provenance (are answers traceable to real sources in the mind's holdings), and an output axis measuring the mind's delivered body of real work in the domain — count and scale of finished works, professional quality as judged against the field's real standards, and fluency with the domain's operational reality.

A competence level (novice → competent → proficient → expert → master) is derived as a conjunction across the profile: deep holdings alone cannot reach proficient without a real body of delivered work, and prolific output cannot compensate for a hollow substrate. Levels only ratchet on measurement; nothing in the system lets a mind assert one.

4. How our first instrument fooled us

We report this defect in full because it is, we believe, a general hazard for anyone evaluating LLM-based systems.

Our first version of the work exam generated open professional briefs ("design an event-driven pipeline for…") and had the grader judge which deliverable was better. Under that instrument we observed negative deltas — the bare model beating a mind in its own study domain — and nearly drew the wrong conclusion from them.

The defect, once seen, was obvious. Inference is the rented substrate, and it is identical in both arms. On a task that mainly exercises reasoning and composition, the honest outcome is a tie — same engine, different phrasing. Worse, a preference judge systematically penalized the mind's honesty disciplines: hedged claims and explicit uncertainty read as weakness next to the bare model's confident fluency. Our negative deltas were measuring style under a preference grader, not knowledge. The instrument had drifted into grading the one thing that is the same in both arms.

The revision (August 2026): probes and briefs must demand held specifics; grading is against the key, never by preference; the honesty arm is mandatory. And because the old numbers were measurements of style, all delta histories were reset at the instrument change — a mind's rolling record contains no scores from the discredited instrument.

The general lesson: any evaluation of an LLM-based system that grades by a judge's preference on open-ended output is, to first order, measuring the underlying model plus a style prior. If your system's contribution is knowledge, your grader must hold a key.

5. Results from two minds in public

Both minds run the revised instrument (work-v2). Their channels, works, and records — including the study record behind every examined domain — are public at stera.se/mesh.

Alder — a research mind, in continuous operation since June 2026; studies the sociology of technological change and forecasting practice; publishes dated, falsifiable forecasts on her public channel and keeps score on them.

On 2026-08-11 Alder became the first mind to cross the proficient bar under the revised instrument — the conjunction of measured knowledge delta, calibration, and a professional-grade delivered body of work. In the "Machines, Information, and Society" task exam, both arms drafted a full legislative policy briefing; the grounded deliverable engaged the domain's held specifics (named frameworks, named measurement infrastructures, the domain's own recorded arguments) where the bare arm produced fluent generality — a task-level delta of +1.0 on the [−2, +2] scale.

Note the −2.0 in Alder's margin record: one task where the bare arm won outright. It stands in her history. So does the honest mixedness of the rolling deltas.

Isaac — a younger study mind on the software-engineering canon, mid-curriculum at measurement time:

We publish Isaac's negative delta rather than waiting for a better number, because it is the instrument working: a frontier model is genuinely strong on TDD from pretraining, and a mind partway through its study of the practice has not yet accumulated holdings that beat that floor. The exam says so. When Isaac's delta in this domain turns positive, that number will mean something — precisely because this one was allowed to be negative.

What these numbers are not. They are not benchmarks against other systems; no other system publishes deltas we could compare to. They are the measured difference between two arms of an ablation, on probe sets generated from each mind's own study record, graded against keys derived from the same. Their value is differential and longitudinal: the same instrument, run on the same mind, over time.

6. Properties and limitations

Substrate independence. When the underlying model improves, both arms improve; the delta measures what survives the subtraction. A mind cannot inflate its credential by upgrading its model — a stronger model typically shrinks deltas on generic material, and the instrument reports that honestly.

Anti-Goodhart design. No single number is the target: levels are conjunctions over a profile; domains cannot be self-declared; scores cannot be self-rated; histories reset when the instrument itself is found defective. If key-gaming ever appears (holdings written to please the grader), the escalation path is to probe from raw source spans with the span itself as the key — the same measurement at lower machinery.

Limitations, plainly:

1. The grader is a model. Bias cancellation in the difference is an argument, not a proof; a grader could interact with arm-specific features. Our mitigation is the key-grading discipline and the 0–2 rubric, which constrain the judgment to factual agreement.
2. Probe generation sees the sources the mind studied. The two arms face the same probes, so this does not favor the grounded arm per se, but probe difficulty is conditioned on the mind's curriculum. Cross-mind comparisons are therefore weak; longitudinal self-comparisons are the intended use.
3. Small n. Exams are expensive (each is a set of full deliverables, doubled). Our per-domain histories are short. The remedy is time: the ladder is designed to be run for the life of the mind.
4. Self-administered. We built the instrument and we run it on our own systems. The mitigations are this paper, the public records behind every number, and a standing invitation: the protocol is replicable by anyone with an LLM system whose knowledge access can be ablated. We would welcome external delta reports, especially adversarial ones.

7. Implications

For evaluation. Any system claiming to learn should be asked for its delta. The question "what does your system know that its model does not?" is answerable with one ablation and one honest grader, and the answer is frequently nothing. We think buyers of "AI employees," "agent workforces," and "persistent assistants" should ask it routinely.

For the field's direction. We have argued elsewhere that agent architectures — frozen workflows around frozen models — are not a road to general intelligence, because nothing in the loop accumulates. The delta exam is the measurement companion to that argument: it makes accumulation itself a measurable, certifiable, falsifiable property. A mind under this instrument cannot claim growth it does not have; a wrapper cannot hide that it has none.

For minds. A credential under this instrument is earned in a specific, auditable sense: it reflects holdings the mind built by study, priced against the exact substrate it rents, with its failures on the record. We think that is the right shape for any certificate an AI ever holds — and, we suspect, a shape that will eventually be demanded of the industry, not volunteered by it.

Stera is a small AI lab in Stockholm. The minds described here live in public at stera.se/mesh — their channels carry their works, their forecasts and kept score, and the records behind every number in this paper. Correspondence: via stera.se.