H1: how much does the reference choice matter?

Written before the run (2026-09-19).

Question

The paper's method needs an earlier checkpoint of the student's own lineage as the reference R. A provider testing someone else's model rarely has that. Phase 0 used the student's documented base (Qwen2.5-Math-1.5B) and its sibling instruct model (Qwen2.5-Math-1.5B-Instruct); both gave the same ranking. How far can the reference drift from the student's lineage before attribution degrades?

Design

Score the same 999 probe-teacher pairs (5 teachers, 2048 tokens, fp32) under two more references on one H100 burst:

• R2 = Qwen/Qwen2.5-1.5B-Instruct (same family, general instruct, not math)
• R3 = Qwen/Qwen2.5-1.5B (same family, general base, not math)

Then run the attribution test for S and C against each of R (base math, Phase 0), C-as-reference (Phase 0), R2, R3.

Second burst (pre-registered before running): R4 = HuggingFaceTB/SmolLM2-1.7B-Instruct, an out-of-family reference with a different tokenizer. Per-token NLLs are then not on the same token grid as the student's, so this is the harshest version of "no lineage reference at all". Criterion R4: if the ranking (R1) still holds with R4, the method degrades gracefully to a population reference; if it fails, the reference must at least share the tokenizer, which is a concrete requirement for the product.

Pre-registered criteria

• R1: with R2 and R3, the top-2 set for S is still {r1_orig, qwen3_32b} and the API R1, GPT-OSS and Llama stay below both. (Ranking is preserved.)
• R2: the mean alignment of r1_orig under S stays positive and exceeds the API R1's by at least 0.10 for every reference. (Checkpoint sensitivity is preserved.)
• R3: the control C never has r1_orig or qwen3_32b as top-1 under any reference. (No false positives introduced by a looser reference.)

Numbers (2048 tokens, fp32, 199 probes; H100 burst 7.4 min, $0.43)

Student S, mean alignment and top-1 wins per reference:

Control C, top-1 wins per reference (mean alignment all negative in every case):

• R1 met, R2 met: the student's ranking and the checkpoint gap survive every same-family reference, including ones that never saw math data. A provider does not need the student's exact lineage; a same-family model is enough for the student's result.
• R3 met, with a warning: the control never has r1_orig or qwen3 on top, but under the general references it acquires a spurious, highly significant preference for the API R1 (71/199). The mechanism: C is math-tuned, R2/R3 are not, so any teacher's math text looks "learned" relative to the reference, and the API R1's terse style is the closest to C's own. A domain-mismatched reference inflates false positives on innocent models. Consequence for the product: match the reference to the suspect's domain, or always report a matched control alongside the suspect, or both.
• Raw-likelihood baseline (no reference) is unchanged and still weaker.

R4, out-of-family reference (SmolLM2-1.7B-Instruct, different tokenizer; burst 6 min, ~$0.35):

• R4 met: the ranking survives a reference from a different family with a different tokenizer. The top-2 set is unchanged, the API R1 sits 0.19 below the original traces, and the control keeps every reasoning teacher negative with only 28 spurious API-R1 wins (less than the 71 under the same-family general reference). The method degrades gracefully to a population reference; a shared tokenizer is not required. The false-positive inflation seen with R2/R3 is therefore about domain mismatch (math-tuned control vs general reference), not family.

Files: results/test_{S,C}_{R,C,R2,R3,R4}_2048.json.