Phase 0 result

Written before the run. Criteria are fixed here; numbers get filled in below.

Setup

• Server: [email protected], campaign /home/deploy/reinstator-experiments/20260919-phase0.
• Student S: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B (documented as SFT of Qwen2.5-Math-1.5B on DeepSeek-R1 outputs).
• Reference R: Qwen/Qwen2.5-Math-1.5B (the student's base; this is the base-model reference variant the paper flags as weaker than a same-lineage instruct checkpoint).
• Control student C: Qwen/Qwen2.5-Math-1.5B-Instruct (same base, fine-tuned by Qwen, not documented as R1-distilled). Also scored against R.
• Candidate teachers via OpenRouter, reasoning included where the model exposes it: DeepSeek-R1, a Qwen3 reasoning model, GPT-OSS-120B, Llama-3.3-70B.
• Proxy prompts: 200 from s1K questions (paper used s1 and OpenMathInstruct-2 as proxies).
• Output truncation: 2048 tokens (paper's setting). If CPU time forces a cut, record the actual value.

Pre-registered criteria

• Cisco kit gap. provenancekit compare S R scores high (expect above 0.9: same architecture, weights inherited). provenancekit scan S ranks the Qwen2.5 family first and does not surface DeepSeek-R1 as a match. This documents that the kit sees the base and not the teacher.
• Teacher attribution on S. The reference-based statistic identifies DeepSeek-R1 as top-1 with per-probe top-1 rate well above 1/K and exact binomial p (Bonferroni over K-1) below 1e-6.
• Control. On C, no teacher reaches Bonferroni-corrected p below 0.01 on the margin test, or if one does the margin is at least 3x smaller than on S. (The paper's calibrated threshold is not available; this is a weaker, uncalibrated check.)
• Baseline. Raw student likelihood without the reference identifies R1 less reliably than the reference-based score (lower top-1 rate or larger p).

Failure of 2 with success of 1 would still be informative: it would mean the base-model reference is too weak at 1.5B and H1 becomes the first priority.

Run log

• 2026-09-19 00:53 UTC. Server venvs ready (torch 2.14 CPU, transformers 5.17). Kit at commit 87fe4b7.
• Kit scan by default runs metadata-only: every match reports identity_score None and pipeline_score = mfi_score. Deep weight fingerprints are a separate download-deepsignals-fingerprint step, not run here.
• Teacher generation via OpenRouter. Qwen3-8B and QwQ-32B are unavailable to this account (zero-data-retention privacy setting excludes their only endpoints); Qwen3-32B used instead. Reasoning included for R1, GPT-OSS-120B, Qwen3-32B; Llama-3.3-70B has none. Max 4096 generated tokens, temperature 0.6, concurrency 6, budget cap $3 per teacher.
• CPU benchmark, 1.5B fp32, 8 threads: 512 tok 3.6 s, 1024 tok 7.3 s, 2048 tok 15.1 s per forward. bf16 is 5x slower (no AMX). Scoring plan: 512-token pass first for an early read, then the paper's 2048.

• Scorer changes mid-run, all validated against the original per-probe numbers (max |diff| 3e-7 on 24 probes): batched forward passes (gain only ~1.3x, scoring is compute-bound), chunked output head (batch-8 naive path was OOM-killed at 12 GB, chunked path stays near 7 GB), and hash sharding for parallel processes per model. The 512 pass kept running on the old code; new code is picked up per loop and is numerically identical.
• OpenRouter key stored on the i9 at /etc/reinstator/env (root:predictbot 640) and sourced by isolated.sh; future teacher generation runs on the i9 directly.
• Interim (512 tokens, 29 probes complete across K=3 teachers, Qwen3-32B not yet in the pool): S identifies r1, 21/29 top-1, binomial p 1.9e-5, margin 0.061, corrected p 0.0017. C identifies llama33_70b 21/29 with all mean scores negative and r1 never top. Raw-likelihood baseline picks llama33_70b for both S and C, so the reference term is what separates them. Interim files under results/tests/interim/.

• Interim 2 (512 tokens, 123 probes, K=4 with qwen3_32b): S picks qwen3_32b on 122/123, r1 second (mean f +0.237 vs +0.455, margin sd 0.097, per-probe share favouring qwen3 0.99). C picks llama33_70b 87/123, never favours the reasoning models. Raw NLL under S: qwen3 text 0.652, r1 text 0.862; under R both ~1.10, so the base finds them equally likely and only the student separates them.
• Diagnosis: trace openers. Qwen3-32B traces begin "Okay, so ..." in 200/200 (the signature of the January-2025 R1 traces the distills were trained on). The deepseek/deepseek-r1 outputs we received (all from provider Novita) begin "The problem ..." or "I need ..." in 0/140 cases with "Okay", which is the R1-0528 style. So the pool's "R1" is probably not the checkpoint that produced the student's training data, and Qwen3's traces are the closest stylistic relative in the pool. This is exactly the entangled-lineage / wrong-checkpoint hazard from the paper, hit on the first real run. Follow-up: sample R1 through DeepSeek's first-party endpoint and R1-0528 explicitly, compare openers; if first-party R1 says "Okay, so", regenerate the r1 set with that provider pinned.

• OpenRouter now lists a single endpoint for deepseek/deepseek-r1 (Novita, fp8); no first-party or alternative provider is available, so the served checkpoint cannot be cross-checked through the API. Instead, a fifth teacher set r1_orig was built from simplescaling/s1K-1.1, whose deepseek_thinking_trajectory / deepseek_attempt columns are DeepSeek-R1 outputs from early 2025 on the same s1K questions: 200/200 prompts matched by question hash, 200/200 traces open with "Okay/Alright". Zero API cost. Scored at 512 tokens into nll_*_512.r1orig.jsonl.

• Provider check: deepseek/deepseek-r1-0528 sampled on 3 prompts (DeepInfra, SiliconFlow) opens with "I have two polynomials..." / "The problem asks..." — the same register as the Novita-served deepseek/deepseek-r1 set (0/140 "Okay") and unlike the s1K-1.1 original-R1 traces (200/200 "Okay/Alright"). Conclusion: the API "R1" candidate behaves like the May-2025 checkpoint; the student's training teacher is represented only by the r1_orig set. A first-party DeepSeek provider request returns 404 (no endpoint).
• Incident: the first r1_orig scoring launch passed an inline ${tag} loop through isolated.sh; systemd expands ${...} in transient unit command lines and set it to empty, so all three models appended to one file nll__512.r1orig.jsonl for 20 minutes while the unit looked healthy. Caught on the user's "is this actually producing data" check. Discarded the mixed file; relaunched via score_extra.sh (script file, no inline variables). Rule added to isolated.sh.

Five-teacher result, 512 tokens, 187 complete probes (2026-09-19)

Files: results/tests/test_S_R_512_5t.json, test_C_R_512_5t.json, test_S_Cref_512_5t.json (mirrored under results/tests/).

• Checkpoint sensitivity is strong and clean: original-R1 traces beat the API-served R1 on 185/187 probes (mean margin +0.213). The statistic clearly detects that the student was trained on the January checkpoint's outputs, not the later one.
• Near-relative ambiguity: original R1 versus Qwen3-32B thinking traces is a coin flip (91 vs 96, margin -0.001, sd 0.108; 66 probes have |margin| > 0.1 and split both ways). Same with the sibling-instruct reference. At 512 tokens the student cannot tell its real teacher from Qwen3's traces, which share the "Okay, so..." register and likely descend from R1-style data themselves.
• Control C identifies Llama (130/187) with all reasoning teachers negative; no false R1/Qwen3 attribution.
• Raw-likelihood baseline (no reference) picks qwen3 for S at corrected p 0.05, far weaker than the reference-based score.

Criteria: (1) met; (2) met against every candidate except Qwen3-32B, which was not in the paper's pool and ties; (3) met; (4) met.

2048-token result, H100 burst, fp32, 199 probes (2026-09-19)

Files: results/gpu/tests/test_S_R_2048_float32.json, test_C_R_2048_float32.json; scores in results/gpu/nll/. RunPod H100 SXM in US-CA-2, 15.5 min wall including first-time bootstrap, ~$0.90; each model scored 999 probe-teacher pairs in 2.8 min (the same work is ~4.5 h per model on the i9).

• Same picture as 512 tokens: original R1 and Qwen3-32B tie (margin 0.002, p_bonf 1.0), both far above the API R1 (0 wins), control clean (Llama 158/199, all reasoning teachers negative). Longer traces do not separate the true teacher from its near-relative; all alignment scores shrink a little at 2048 (the tail of long traces is more generic), but the ordering is unchanged.
• Conclusion for the plan: H3 (idiolect features beyond likelihood) and H2 (mixture estimation) are the paths to break this tie; more tokens will not.
• GPU vs CPU validation (fp32 both): on the ~475 probe-teacher pairs per model the i9 had finished before its pass was stopped, max |diff| in mean NLL was 7e-5 (S), 8e-5 (R), 9e-5 (C), mean ~2e-5 to 6e-5, zero token-count mismatches. Margins under test are 0.1 to 0.5, so H100 numbers are interchangeable with the i9's. The i9 pass was stopped at ~48% to give the box back; the H100 result is the 2048 record.

Numbers

(512-token pass on the i9; 2048-token pass on the H100 above)