H3: idiolect features to break the near-relative tie

Written before the run (2026-09-19).

Question

Phase 0 showed the likelihood statistic cannot separate the student's true teacher (original DeepSeek-R1 traces) from a near-relative (Qwen3-32B thinking traces): 92 vs 107 wins at 2048 tokens, margin 0.002. Do low-level stylistic habits in the student's own generations separate them?

Design

Generate 200 samples from the student S (DeepSeek-R1-Distill-Qwen-1.5B) and 200 from the control C (Qwen2.5-Math-1.5B-Instruct) on the same 200 s1K prompts used for the probes, chat template, temperature 0.6, top-p 0.95, up to 2048 new tokens. One H100 burst.
Extract ~40 style features per text (style_features.py): trace opener, discourse markers per 1k tokens ("Wait", "Hmm", "Let me", "Actually", "Alternatively", "So,"), pronoun voice (I vs we), punctuation rates, sentence length, LaTeX and markdown habits, boxed answers, unicode/ASCII math symbol ratio, digit density, newline density, hedges.
Train a multinomial logistic regression on the five teacher sets (5 x 200 texts) with 5-fold CV (classify.py). Report teacher-vs-teacher separability, in particular r1_orig vs qwen3_32b.
Apply the classifier to S and C samples. Report predicted-class distribution and mean class probabilities.

Pre-registered criteria

C1 (features carry lineage signal): 5-fold CV accuracy for the binary r1_orig vs qwen3_32b problem on teacher texts is at least 0.80. If not, style features cannot separate these two and H3 fails for this pair regardless of the student result.
C2 (tie broken on the student): among S samples, r1_orig is the modal predicted class and its mean probability exceeds qwen3_32b's by at least 0.20.
C3 (control): among C samples, r1_orig is not the modal class, and its mean probability does not exceed qwen3_32b's by 0.20.
Caveat recorded in advance: S was fine-tuned on original-R1 traces, so C2 passing shows the student inherited R1-specific habits that Qwen3 does not share. It does not by itself prove a different student distilled from Qwen3 would be told apart; that needs the Phase 1 testbed.

Extension C4 (pre-registered 2026-09-19 before the run): trace-stripped outputs

Adversary capability 3 in the plan: the student is trained on, or only exposes, final answers without reasoning traces. The paper reports likelihood attribution collapsing to zero there. Test whether style features survive: regenerate the four API teachers with reasoning hidden (OpenRouter reasoning.exclude, 200 prompts, 1500 tokens), take the original-R1 answers (deepseek_attempt) as the r1_orig answer set, and the answer part of the S and C generations (text after </think>). Retrain the classifier on answer-only teacher texts.

C4a: answer-only CV separability r1_orig vs qwen3_32b at least 0.70 (lower bar: answers are shorter and more constrained than traces).
C4b: S answers modal class r1_orig with mean-probability margin over qwen3 at least 0.15; C answers not modal r1_orig.

Amendment (2026-09-19, during the run): hidden-reasoning generation is not viable for R1 and Qwen3 on s1K-difficulty prompts; both exhaust a 6,000-token budget while still thinking (R1 3/3, Qwen3 6/9 empty), so those two API sets were stopped (~$0.10 spent). Answer-only teacher sets available instead: r1_orig answers (deepseek_attempt), gemini answers (gemini_attempt), llama33_70b_ans and gpt_oss_120b_ans (API, finish normally). C4a therefore becomes r1_orig vs gemini separability on answers; C4b keeps its form with the pool {r1_orig, gemini, gpt_oss, llama}. The Qwen3-vs-R1 answer-only question is deferred to the Phase 1 pilot, where students trained on each can be compared on their own answers.

C4 numbers (answer-only, 2026-09-19)

Teacher answer sets: r1_orig 199, gemini 198, gpt_oss_120b_ans 175, llama33_70b_ans 199. Student S answers: only 5 (195 of 200 generations were still inside <think> at 2,048 tokens). Control C answers: 199 (no traces, whole output).

Variant	CV all 4	r1_orig vs gemini	S answers (n=5)	C answers (n=199): modal, p(r1_orig)
full answers	0.979	0.987	modal r1_orig 4/5, margin +0.79	llama 103 / r1_orig 89, mean p(r1_orig) 0.42
first 300 words	0.958	0.982	modal r1_orig 3/5, margin +0.58	llama 103 / r1_orig 86, 0.41

C4a met (0.98 >> 0.70), but the discriminators are formatting conventions: $...$ LaTeX (r1_orig 75 per 1k words, gemini 1), $...$ (gemini 204, r1_orig 1), "Final Answer" phrasing, numbered steps, "so" openers.
C4b inconclusive for S (n=5) and failed for C: the control, a Qwen math-instruct model with no R1 exposure, has answers that look R1-like 45% of the time. Answer formatting is a family/style convention, not lineage; a trace-stripped adversary is far harder to attribute by style than a trace-exposing one, and answer-only style classifiers will produce false positives on same-family innocents. This is consistent with the paper's finding that likelihood attribution collapses without traces.
Follow-up done: regenerated S at 4,096 tokens; still only 19/200 generations closed the trace (the 1.5B distill rambles), giving 17 answers: 17/17 classified r1_orig (mean p 0.94; 15/17 on 300-word answers), mixture 1.00 r1_orig. Directionally right but n=17, and the control false-positive rate (45%) is unchanged. Conclusion stands: answer-only style is supporting evidence only, never the primary signal. Files results/c4_*_4k.json.

Numbers

C1, teacher separability (81 features, logistic regression, 5-fold CV, 200 texts per teacher):

Text used	All 5 teachers	r1_orig vs qwen3_32b	Top discriminators (r1_orig minus qwen3)
full output	0.994	0.995	fewer "let me", "maybe", "therefore", "?"; more "hmm", "thus", "hence", boxed
reasoning only, first 800 words (equal length, no answers)	0.970	0.948	fewer "?" and "let me", far less $...$ LaTeX (0.5 vs 19 per 1k); more "hmm", "let's", "perhaps", higher type-token ratio

C1 met (threshold 0.80). The equal-length control matters: on full text the classifier partly keys on completeness (dataset traces carry a boxed final answer, API traces are often cut off), but the signal survives with that removed. Files: c1_teachers_only.json, c1_teachers_head800.json on the i9 under results/h3/.

C2 / C3, student and control generations (200 samples each, H100 burst, chat template, temperature 0.6, top-p 0.95, up to 2048 new tokens; classifier trained on the five teacher sets):

Text used	S_gen modal class	S_gen hist (r1_orig / qwen3 / other)	S mean p(r1_orig) - p(qwen3)	C_gen modal	C_gen mean p(r1_orig) - p(qwen3)
full output	r1_orig	186 / 13 / 1	+0.802 (0.894 vs 0.092)	llama33_70b (191/200)	+0.062 (0.062 vs 0.000)
reasoning only, 800 words	r1_orig	185 / 14 / 1	+0.843 (0.916 vs 0.073)	llama33_70b (198/200)	+0.012

C2 met (threshold +0.20). C3 met (control never modal r1_orig; margin far below 0.20). The likelihood tie (92 vs 107 wins at 2048 tokens) is broken by style features of the student's own generations: the student writes like the January R1, not like Qwen3, even though it finds both equally likely to read.

Caveats: the student was trained on this exact teacher, so inheritance of style is expected; whether a student trained on Qwen3 traces would be told apart from one trained on R1 traces is the Phase 1 testbed's question. All teacher sets and generations share the same 200 prompts, so topic is balanced across classes and cannot be what the classifier learned. Files: results/h3_full.json, results/h3_head800.json.