Goal. Develop and validate methods that let a model provider prove, to a defensible evidence standard, that a third-party model was distilled from their model. Output: a reproducible detection toolkit plus an evidence-report format suitable for policy enforcement or legal escalation.
Status (2026-09-19). Phase 0 complete: the reference statistic reproduced, kit gap demonstrated, five-teacher result and the near-relative tie documented in experiments/00_reproduce/RESULT.md. Phase 2 started ahead of Phase 1 because the tie was the live problem: H3 (idiolect) and H2 (mixture) passed their pre-registered criteria, H4 budgets and H5 sequential tests measured on existing data, H1 reference variants running. Per-hypothesis write-ups under experiments/02_methods/. Compute: i9 for CPU work, RunPod H100 bursts for GPU (COMPUTE.md). See related-work.md for source notes.
---
Two sources anchor this work. They attack different halves of the problem and neither alone solves ours.
| Cisco Model Provenance Kit | Rawat et al. 2026, "Reference-Based Distillation Detection in LLMs" | |
|---|---|---|
| Question answered | "Does model B inherit weights from base family A?" | "Was student S trained on outputs of teacher T?" |
| Access needed | Weights, tokenizer, config (white-box on the suspect) | Log-likelihoods from S and a reference checkpoint R; samples from candidate teachers |
| Signals | 8 static signals (metadata, tokenizer, embedding geometry, layer norms, weight correlation) | Reference-normalized membership inference: f = -(l_S(x,y) - l_R(x,y)) on teacher outputs y |
| Distillation coverage | Explicitly fragile or failing on all 8 signals when architecture differs | Core target, but only single-teacher, SFT-style, final-stage |
| Evidence output | 0-1 score, "strong evidence not proof" | Exact binomial + Wilcoxon/t tests, Bonferroni-corrected p*, open-world extension |
| Reusable for us | Engineering pattern: multi-signal fingerprint, gate-then-score, reference DB, caching, CLI, Apache-2.0. Also the Provenance Constitution evidence standard. | The detection statistic, the controlled-distillation testbed recipe, the statistical tests, the o1/o3 glyph signal |
Key observation. The Cisco kit's own signals table marks distillation as the one derivation it cannot see. Its constitution lists distillation as a provenance-linked mechanism but scopes out "detection methodology." That is exactly the hole. The paper fills it behaviorally but with a strong assumption: you must hold an earlier-generation checkpoint of the student's lineage. A provider trying to prove distillation of their model usually does not.
Working thesis. A provider-side distillation prover needs (a) behavioral signals that work without a same-lineage reference, (b) statistical guarantees that hold in the open world, and (c) an optional active component the provider controls at serving time. We want to build all three into one multi-signal framework in the Cisco kit's mold.
---
Defender (us / the lab). Owns teacher T. Can sample T freely, may or may not expose T's logits, controls T's serving stack. Wants to test a suspect model S.
Suspect S. One of:
Adversary capabilities to model, in increasing difficulty:
Non-goals for now. Data provenance of the teacher's own training set. Legal severity scoring. Non-transformer architectures.
---
Each hypothesis has a falsifiable experiment. Ordered roughly by expected value / cost.
The paper's Table 11 shows base-model references degrade accuracy on some teachers. Test substitutes:
Success: attribution accuracy within 5 points of the same-lineage reference on the controlled testbed.
Model per-example alignment scores as a non-negative mixture over K candidate teachers. Estimate mixing weights with NNLS or EM; test each weight against zero with a permutation test. Novel relative to the paper.
Success: recovers 2-teacher mixtures at 70/30 and 50/50 splits with correct support.
Generalize the paper's o1/o3 ASCII-vs-Unicode glyph signal into a family of low-level stylistic statistics that survive answer-only distillation and light paraphrase: glyph/punctuation distributions, formatting habits (markdown headers, list style, LaTeX delimiters), hedging phrases, reasoning scaffold tokens ("Wait,", "Let me verify"), answer-boxing conventions, length/structure priors. Compute on samples from S; compare to each teacher's idiolect profile against a population baseline.
Success: AUROC above 0.8 on trace-stripped students where the likelihood method scores 0.
The paper uses fixed proxy-prompt sets. Choose prompts that maximize expected margin between candidate teachers (where teacher outputs disagree most, in the likelihood sense). Mirrors ADFP's proxy-model idea but on the detection side.
Success: same significance with 5-10x fewer probes.
The paper's Bonferroni p* is crude. Wrap detection in (a) a sequential test with anytime-valid p-values so we can stop querying early, and (b) the "provable provenance set" construction (arXiv 2602.00772) to output a set of teachers with coverage at level 1-alpha and controlled inclusion of unrelated models.
Success: empirical FPR at or below alpha across held-out teachers; leave-one-teacher-out accuracy above the paper's 60.5%.
Distillation does not copy weights, but it may leave measurable traces in the student's output-side geometry: unembedding row structure, output-token preference vectors, or "LLM DNA" functional embeddings. Run Cisco's EAS/END on unembeddings and the LLM-DNA pipeline student-vs-teacher-vs-controls.
Expected outcome: likely a null or weak result. Worth doing cheaply because a negative result sharpens the claim that behavioral signals are necessary.
Evaluate ADFP-style antidistillation fingerprinting as the defender's proactive layer. Measure the utility cost and whether the fingerprint survives adversary capabilities 3-6. Combine with passive signals into one evidence score.
Note: dual-use adjacent. Keep to the defensive framing and measure utility impact honestly.
Once H1-H5 produce a detector, train students with a detection-aware regularizer (e.g. penalize likelihood gap vs a reference) and measure how much utility they must give up to evade. Bounds the detector's real-world value.
---
Controlled testbed (reuse the paper's recipe so results are comparable):
Hard negatives (missing from the paper, essential for FPR claims):
Real-world set: the 6 DeepSeek-R1 distills, s1.1-32B, X-Coder-SFT-Qwen3-8B, plus any 2026 open models with documented teachers.
Metrics: top-1 teacher attribution accuracy (aggregated and per-sample), AUROC for distilled-vs-not, FPR at 95% TPR, calibrated p-value validity (empirical FPR vs nominal alpha), leave-one-student-out and leave-one-teacher-out generalization, probe budget to reach p* < 1e-6.
Baselines: the paper's method with a same-lineage reference (upper bound), raw likelihood, Min-k%, zlib-normalized, n-gram Jaccard, embedding cosine, Cisco kit compare score.
---
provenancekit compare on a known distill triple: DeepSeek-R1-Distill-Qwen-7B vs its base Qwen2.5-Math-7B vs teacher DeepSeek-R1. Expected: high score to the base, near-zero to the teacher. Record this as the concrete gap statement.---
---
| Hypothesis | Status | Where |
|---|---|---|
| Phase 0 reproduction | done: statistic reproduced; API "R1" is not the training checkpoint (original traces beat it 185/187); near-relative tie with Qwen3-32B at 512 and 2048 tokens; control clean; Cisco kit blind to the teacher | experiments/00_reproduce/RESULT.md |
| H1 reference relaxation | done: student ranking holds under every same-family reference and an out-of-family, different-tokenizer reference; domain-mismatched references inflate control false positives; exact-lineage reference resolves the tie (Phase 1) | h1_reference_relaxation |
| H2 mixture | done: EM recovers synthetic text blends within 4 points; on trained students the estimate is monotone in the data share (five-point grid) but biased toward the more learnable teacher and scale-dependent; calibration needs a strength axis | h2_multi_teacher |
| H3 idiolect | done: style features break the R1/Qwen3 tie (student 186/200 original R1); answer-only variant weak with 45% control false positives | h3_idiolect |
| H4 budget | done: 20 probes for the family call, 60-80 for the single teacher, 5-8 generations for style | h4_probe_budget |
| H5 guarantees | first cut: anytime-valid e-process stops at probe 18, never on the control | h5_guarantees |
| Phase 1 pilot | done: six students from one base (plus human-solution hard negative attributed to nobody, and an untruncated R1 student confirming the truncation artifact); R1-trained and Qwen3-trained students told apart by likelihood (129/70 and 177/22 with the exact-lineage reference) and by style (129/70 and 179/20); Gemini student attributed 191/199 with no R1/Qwen3 false positives; 50/50 mixture reads 75/25 toward the more learnable teacher (estimator needs calibration) | experiments/01_testbed/RESULT.md |
| H6 weight traces, H7 active fingerprint, H8 adaptive adversary | not started |
Compute used: $22.49 of GPU (RunPod balance $100.00 -> $77.51; 14 bursts, all pods terminated by script and verified) and about $10 of OpenRouter. Usage log: ops/gpu-usage-log.md.
The original phases assumed "which teacher" was the unit of attribution. The results say the reliable unit is family x checkpoint x sampling pipeline, with an absolute-signal gate before any ranking. The roadmap is reorganised around that.
| # | Track | What it answers | Compute | Timeline |
|---|---|---|---|---|
| R1 | Same-source strength grid. Sample one teacher ourselves through one pipeline at 800 / 4k / 20k examples on one prompt distribution; train students; fit strength-conditioned calibration for the mixture estimator and the absolute-alignment threshold. | Removes the source confound; makes mixture and threshold numbers transferable to strongly distilled suspects. | ~$25 API (Qwen3 or a self-hosted 32B teacher) + ~6 GPU-h | 1-2 days |
| R2 | False-positive sweep. Hard negatives at scale: continued-pretraining on math text, merges of clean fine-tunes, other-base students (Llama-3.2 / SmolLM), instruct models from three families, plus students trained on one family's older checkpoint. Report FPR at fixed thresholds for both signals. | The number a policy team needs: how often an innocent model trips each tier. | ~8 GPU-h | 2-3 days |
| R3 | Three-tier evidence report. Tier 1 absolute alignment gate (threshold set on R2's negatives); Tier 2 family attribution with the anytime-valid e-process and matched control; Tier 3 member/pipeline attribution by style + exact-lineage likelihood with the candidate pool disclosed. JSON schema, aligned with the Cisco constitution's evidence standard, plus a human-readable page. | Turns results into a deliverable a lab can act on. | CPU only | 2 days |
| R4 | Serving-time preservation and fingerprinting (H7). Specify what a lab must log at serving time (checkpoint id, sampling params, prompt hash, output hash) to make Tier 3 claims later; evaluate ADFP-style fingerprints on the testbed for survival through SFT and through answer-only training. | Finding #1 and #6 made this a product requirement, not an option. | ~6 GPU-h | 3-4 days |
| R5 | Scale and domain. 3B/4B students (paper's Gemma/Llama bases), a code prompt domain, 7B-14B real-world distills scored in bf16, o-series teachers via the glyph signal. | Shows the numbers hold beyond 1.5B math. | ~15 GPU-h | 1 week |
| R6 | Adversary levels 4-8. Paraphrase/rewrite pipelines, distill-then-RL, and a detector-aware student trained with a style-regularising loss; measure utility cost to evade. | Bounds the method's real-world value against a motivated adversary. | ~10 GPU-h | 1 week |
| R7 | reinstator.com. Serve the per-claim reports and RESULT tables from the i9 behind Caddy; later, a submit-a-model endpoint backed by the burst pipeline (RunPod serverless when request-shaped). | Public face and the eventual product. | i9 | alongside R3 |
| R8 | H6 weight traces (cheap negative result with the Cisco kit on the trained students) and the upstream contribution of a behavioral signal group to the kit. | Closes the bridge to existing tooling. | CPU | opportunistic |
Total for R1-R6: roughly 45 GPU-hours (~$150) plus ~$40 of API, about three weeks of work. Every track keeps the discipline of a pre-registered RESULT.md.
uv sync, run the Phase 0 comparison.