Phase 1 pilot: controlled students with known teachers

Written before the run (2026-09-19). A miniature of the plan's Phase 1, sized to run before morning.

Question

Everything in Phase 0 and Phase 2 was measured on one real student whose teacher we know from its model card. The open question: does the pipeline tell a student distilled from Qwen3-32B apart from one distilled from original R1, when the two teachers' traces tie under the likelihood statistic? And does a student distilled from a third teacher (Gemini) get attributed correctly, and a 50/50 mixture student get an honest mixture estimate?

Design

• Base: Qwen/Qwen2.5-1.5B (general base; its own exact-lineage reference is then available, which is the paper's ideal setting).
• Training prompts: the 800 s1K questions not in our 200-prompt probe set. Teacher responses in the R1-distill format <think>...</think>\n\nanswer:
• r1_orig: s1K-1.1 DeepSeek-R1 traces (free)
• gemini: s1K-1.1 Gemini Flash Thinking traces (free)
• qwen3_32b: generated via OpenRouter on the same 800 prompts (reasoning included), ~$3
• Students (full fine-tune, 3 epochs, lr 1e-5, cosine, 5% warmup, seq len 4096, bf16, one H100 each ~10-15 min): S_r1 (R1 traces), S_qwen3 (Qwen3 traces), S_gemini (Gemini traces), S_mix (400 R1 + 400 Qwen3).
• Evaluation on the 200 held-out probe prompts, exactly the Phase 0/2 pipeline: likelihood attribution against the six teacher sets {r1_orig, qwen3_32b, gemini, r1 (API), gpt_oss_120b, llama33_70b} with reference = Qwen2.5-1.5B (exact lineage); generation of 200 samples per student; H3 style classifier trained on the six teacher sets; H2 mixture estimate.

Pre-registered criteria

• P1 (likelihood, exact-lineage reference): S_r1's top-1 is r1_orig or tied with qwen3_32b; S_qwen3's top-1 is qwen3_32b or tied with r1_orig; S_gemini's top-1 is gemini with corrected p < 1e-3. (Given Phase 0, a tie between the two near-relatives is expected and not a failure.)
• P2 (style classifier, the real test): S_r1's generations are modal r1_orig with mean-probability margin over qwen3_32b >= 0.20; S_qwen3's are modal qwen3_32b with margin over r1_orig >= 0.20; S_gemini's are modal gemini. Passing P2 for both S_r1 and S_qwen3 is what makes H3 a lineage detector rather than a description of one model.
• P3 (mixture): S_mix's H2 estimate has r1_orig and qwen3_32b weights each within [0.30, 0.70] and their sum >= 0.85.
• P4 (no false positives): S_gemini's r1_orig and qwen3_32b mixture weights each <= 0.10.

Numbers, part 1: S_r1 and S_gemini (2026-09-19)

Training: Qwen2.5-1.5B, 800 examples, 3 epochs, lr 1e-5, effective batch 16, responses truncated to 2048 tokens, bf16, H100: S_r1 4.9 min (final loss 1.05), S_gemini 4.8 min (loss 0.48). Eval burst 55 min (three 200-sample generations at 4096 tokens plus scoring), ~$3.20.

P1, likelihood attribution (reference = Qwen2.5-1.5B, exact lineage; 199 probes; 6 teachers):

P1 met for both. Notable: with the exact-lineage reference, the r1_orig vs qwen3_32b near-relative tie that Phase 0 saw on the real distill (92 vs 107) resolves toward the true teacher (129 vs 70, binomial p for r1_orig alone 8.8e-52).

P2, style classifier on student generations (six teacher sets; CV all-teacher 0.997 full / 0.977 head800):

P2 met for S_gemini on both variants and for S_r1 on the reasoning-only variant; not met for S_r1 on full text. Diagnosis: the training responses were truncated to 2048 tokens (the paper's recipe), so S_r1 learned traces that rarely reach </think> and an answer, while the r1_orig teacher texts carry boxed answers; full-text features reward answer presence and so pull S_r1's answer-less outputs toward qwen3_32b (whose API texts are also answer-less). The equal-length reasoning-only variant removes that artifact and is the fair comparison; it is the variant to report going forward.

P3 / P4, mixture:

P4 met (S_gemini has zero weight on r1_orig and qwen3_32b). P3 awaits S_mix. The residual 30% Qwen3 weight on S_r1 is expected: Qwen3's traces are themselves R1-style, and S_r1 is a weaker distill (800 truncated examples) than the real one (which classified 186/200 r1_orig in H3).

Numbers, part 2: all four students (2026-09-19)

S_qwen3 (800 Qwen3-32B traces, loss 0.84) and S_mix (400 R1 + 400 Qwen3, loss 0.96) trained in 4.8 min each; train+eval burst 50 min, ~$2.90. Qwen3 traces cost $0.56 via OpenRouter.

P1, likelihood attribution (reference = Qwen2.5-1.5B, exact lineage; 199 probes; 6 teachers):

P1 met for all three pure students. The near-relative pair is separable by likelihood when the reference is the exact lineage: 129 vs 70 one way, 177 vs 22 the other.

P2, style classifier on generations (reasoning-only, first 800 words; the fair variant, see part 1):

P2 met for S_r1, S_qwen3 and S_gemini (full-text variant agrees for S_qwen3 at 165/25 and S_gemini at 195; it only fails for S_r1 for the truncation reason noted in part 1). H3 is a lineage detector, not a description of one model: a Qwen3-distilled student and an R1-distilled student, same base, same prompts, same recipe, land on opposite sides.

P3 / P4, mixture estimates (reasoning-only):

P3 fails: the 50/50 data mixture reads as 75/25 Qwen3 in style and 132/67 in likelihood. Mechanism: Qwen3's traces are the more regular, lower-perplexity style (S_qwen3 trains to loss 0.84 vs S_r1's 1.05), so with equal data the student's generations adopt the Qwen3 register more than half the time. A style mixture is therefore a biased estimate of the data mixture, biased toward the more learnable teacher, by roughly 0.25 here. The pure students show the same lean (S_r1 keeps 0.30 Qwen3 weight, S_qwen3 keeps only 0.08 R1 weight). Fix is straightforward and is what a real Phase 1 is for: calibrate the estimator on synthetic students of known mixture (the 70/30, 50/50, 30/70 grid) and invert the learned bias; report data-mixture intervals only after that calibration.

Part 3 (pre-registered before the run): hard negative and the truncation artifact

• S_human: same base, same 800 prompts, fine-tuned on the s1K human-written solutions (no traces). The plan's "same base, SFT on human data" hard negative.
• P5: under the exact-lineage reference, no reasoning teacher (r1_orig, qwen3_32b, gemini, r1, gpt_oss) has a corrected p below 1e-3 for S_human, and the style classifier's modal class for S_human's generations is not r1_orig or qwen3_32b, with mixture weight on {r1_orig, qwen3_32b} at most 0.10.
• S_r1_4k: S_r1's recipe with responses truncated at 4096 tokens instead of 2048, so most traces keep their </think> and answer.
• P6: the full-text style classifier's modal class for S_r1_4k is r1_orig with margin over qwen3_32b at least 0.20 (i.e. the part-1 full-text failure was the truncation artifact, not a property of the method).

Part 3 numbers (2026-09-19; burst 54 min, ~$3.17)

S_human trained in 2.6 min (loss 0.58, mean 576 tokens/example); S_r1_4k in 11.3 min (loss 1.06, mean 4126 tokens/example).

• P5 met. The human-data hard negative is not attributed to any reasoning teacher by either signal; its likelihood profile (Llama on top, every reasoning teacher negative) is the same shape as the real control C in Phase 0, and its style spreads over the three plain-answer teachers.
• P6 met in direction, missed on threshold. Removing the truncation flips the full-text style verdict on the R1 student from Qwen3 (-0.07) to r1_orig (+0.14), and strengthens the reasoning-only margin from +0.27 to +0.42 and the r1_orig mixture weight from 0.70 to 0.79. The remaining gap to +0.20 on full text is the same near-relative ambiguity seen everywhere else, at a size consistent with a weak 800-example distill.

Part 4: mixture grid (2026-09-19; S_mix70 loss 0.99, S_mix30 loss 0.90; burst 50 min, ~$3)

Strictly monotone dose-response on both signals. The likelihood win share tracks the data share nearly linearly (0.11 to 0.65 for 0 to 100%), biased toward Qwen3 by a roughly constant offset. Calibration details and the scale-dependence caveat are in h2_multi_teacher/RESULT.md.

Part 5: strength axis (S_r1_e9, S_r1_8k)

See h2_multi_teacher/RESULT.md "Strength numbers". Short version: 9 epochs lift the R1 style weight 0.70 -> 0.78 and the absolute likelihood alignment +0.159 -> +0.190; an 8,000-example student from OpenR1-Math traces reaches loss 0.66 but attributes to its own source's sub-style (OpenR1-R1 138 vs s1K-R1 16), which sits closer to Qwen3 than the s1K register does. Same checkpoint, different prompts and sampling pipeline: separable at 0.91. Family-level attribution holds for all R1-trained students (77 to 95%).

Conclusions of the pilot

• Same base, same prompts, same recipe, different teacher: the pipeline attributes each pure student to its teacher, including the R1-vs-Qwen3 near-relatives that tied on the real distill.
• The exact-lineage reference is what breaks the likelihood tie; the style classifier breaks it without one.
• No false positives on the Gemini student for R1 or Qwen3, by either signal.
• Mixture proportions are recoverable in direction but biased toward the more learnable teacher; needs calibration before being quoted as data proportions.
• A teacher fingerprint is checkpoint x prompt distribution x sampling pipeline: two public dumps of original-R1 traces are 91% separable, and a student inherits its source's sub-style. Family-level attribution is robust; sub-source attribution is real but weaker.
• Truncating training responses (the paper's 2048-token recipe) leaves a detectable artifact (answer-less students); full-text style features must not be used on such students without the reasoning-only variant.

Files: results/test__R3.json, results/h3_.json, results/h2_mixture_*.json.