H5 first cut: an anytime-valid stopping rule for probing
From Phase 0 data (2048 tokens, five teachers), no new compute. Script sequential.py, outputs under results/.
Method
Per probe, record whether the suspect's top-1 teacher falls in a target set (top-1 under the reference-normalised alignment score). Under the null the win rate is at most chance, p0 = |target| / K. The evidence process is the Beta(1,1)-mixture likelihood ratio against Bernoulli(p0), a test martingale, so rejecting when E_n >= 1/alpha is valid at any data-dependent stopping time: no Bonferroni, no fixed n in advance, and the same p-value guarantee whether we stop at probe 18 or probe 500. This replaces the paper's fixed-n binomial plus Bonferroni for the open-world setting.
Results (500 random probe orders)
| Target | alpha | Student: stop n median / p95 | Student: reached | Control: reached | Control final log10 E |
|---|
| {r1_orig, qwen3_32b} (R1-family style), p0 = 0.40 | 1e-6 | 18 / 18 | 100% | 0% | -276 |
| {r1_orig} alone, p0 = 0.20 | 1e-3 | 43 / 81 | 100% | 0% | -2.3 |
The student wins the pair on 199/199 probes, so the process crosses 1e6 at exactly the 18th probe in every order. The control's evidence collapses toward zero (it prefers Llama), which is the behaviour a sequential test must show on a negative: the longer you probe an innocent model, the more certain the exoneration.
Open-world check (pre-registered 2026-09-19 before the run)
Remove each Phase 1 student's true teacher from the candidate pool and rerun both signals (openworld.py), for S_r1, S_qwen3, S_gemini (exact-lineage reference, six-teacher pool minus the true one).
- O1 (expected failure of the relative test): with the true teacher absent, the likelihood test still returns a "significant" top-1 among the wrong teachers for at least one student (binomial p_bonf < 1e-3), because per-probe ranking always crowns someone. This is the paper's open-world problem made concrete.
- O2 (an absolute criterion separates the cases): the top-1 mean alignment with the true teacher present exceeds the top-1 mean alignment with it absent by at least 0.03 for every student, i.e. an absolute alignment threshold or a margin-over-runner-up rule can flag "no true teacher present". (For S_gemini, whose alternative teachers are all far away, this should be large; for S_r1 and S_qwen3, whose near-relative remains in the pool, it may be small: that is the informative case.)
- O3 (style is less prone to confident wrong calls): with the true teacher absent, the style classifier's max mean probability for a wrong class drops below 0.60 for S_gemini; for S_r1 and S_qwen3 the near-relative will likely absorb the mass (report the number).
Open-world numbers (reasoning-only features; exact-lineage reference; results/openworld_head800.json)
| Student (true teacher) | Closed: top-1, mean f, win rate | Open (true removed): top-1, mean f, win rate, p_bonf | Style open: modal, max mean p |
|---|
| S_r1 (r1_orig) | r1_orig, +0.159, 0.65 | qwen3_32b, +0.149, 0.98, 5e-129 | qwen3_32b, 0.94 |
| S_qwen3 (qwen3_32b) | qwen3_32b, +0.172, 0.89 | r1_orig, +0.140, 0.98, 3e-131 | r1_orig, 0.93 |
| S_gemini (gemini) | gemini, +0.065, 0.96 | r1 (API), +0.029, 0.62, 6e-38 | r1, 0.65 |
| S_human (none) | llama, +0.001, 0.72, 1e-66 | same | llama, 0.51 |
- O1 confirmed, strongly. The ranking test produces a "significant" teacher for every student when its true teacher is absent, and even for the human-data student that has no teacher at all (p 1e-66 for Llama at a mean alignment of +0.001). Relative tests cannot say "none of these".
- O2 partially met. The absolute alignment level separates the cases the ranking cannot: true-teacher signals are +0.065 to +0.172; wrong-teacher calls are +0.029 (S_gemini open) and +0.001 (S_human). A threshold near +0.05 rejects both. It does not separate the near-relative cases (S_r1 open +0.149, S_qwen3 open +0.140), and should not: an R1-family reasoning style genuinely is present there. Gap true-minus-open: S_r1 0.010, S_qwen3 0.032, S_gemini 0.036.
- O3 narrowly missed for S_gemini (0.65 vs the 0.60 bar): with Gemini removed, the style classifier leans on the API R1 at 0.65 and spreads the rest; for the near-relatives the surviving twin absorbs 0.93 to 0.94, as expected.
- Design consequence for the evidence report: three tiers. (1) An absolute alignment threshold, calibrated on hard negatives, answers "is any reasoning teacher's signal present at all" (rejects S_human and S_gemini-without-Gemini). (2) Within a detected family, the style classifier and exact-lineage likelihood name the member. (3) Open-world claims must include a matched control and the candidate pool, because a strong, significant preference for a wrong near-relative is exactly what a missing true teacher looks like.
What this buys the product
- A probing session can be adaptive: score probes until E_n crosses the threshold, then stop and report a valid p-value. Typical cost for the style-family call is under 20 probes.
- Naming a single teacher among near-relatives by likelihood alone is the expensive call (43 to 81 probes) and is better handled by H3's generation-side classifier, which needs 5 to 8 samples (H4).
- Next: extend to the provable-provenance-set construction (arXiv 2602.00772) so the output is a set of teachers with coverage at 1 - alpha, and validate the false-positive rate across the Phase 1 hard negatives rather than one control.