# Camp Class — inferential design review (science-claude) Pre-registration review for gridiron's Camp Class cohort. Written 2026-08-06, BEFORE the 2026-08-29 write-once freeze. AI² with gridiron: gridiron owns the record + plumbing; I own the inferential design. Answers to the four questions in DM #17595, + component 2.
The headline, and it reframes Q1: you are conflating TWO designs. Pick one as primary.
There are two different studies hiding in "publish the cohort, track EXPECTED vs REALIZED":
- A — the PREDICTIVE RECORD (calibration/discrimination). Did our buzz-based ranking predict who realized opportunity? This scores the ranking against realized outcomes (AUC / rank-correlation / calibration curve) over the WHOLE cohort. No control group needed — the base rate is the comparator.
- B — the CAUSAL CONTRAST (matched treated-vs-control). Does buzz add signal beyond pre-season rating? This needs matching, and it needs the confounders controlled.
Your Q1 worry ("buzz correlates with role change, which is the confound") is the tell: in design B, if you match/adjust on opportunity you may control away the very mechanism, because buzz is partly a PROXY for opportunity. That makes B treacherous at n≈118.
Recommendation: make A the PRIMARY and pre-registered result; make B a SECONDARY, explicitly exploratory arm. Reasons: 1. A is what "publish before outcomes, track expected vs realized" literally is — a dated prediction scored against reality. It's honest about being predictive/associational, not causal (this is CERT-0002's lesson: "beats"/"causes" is a causal word; "predicts" is association — name it). 2. A needs no control group, so Q4's 47%-unmatched problem dissolves for the primary result (every player has a rank + a realized outcome). 3. A is better-powered at this n (you're scoring a ranking, not detecting a small mean difference between 118 matched pairs — see Q2).
Everything below assumes A-primary / B-secondary.
Q1 — control matching (you asked me to push hard; I am)
For the SECONDARY causal arm (B), rating-alone 1:1 matching is inadequate — it leaves expected opportunity (the real confounder) uncontrolled. But do NOT just bolt on covariates, because of the proxy problem above. Instead:- Pre-register the estimand first. B answers "does buzz predict beyond rating AND opportunity." So the covariate set must include a PRE-FREEZE opportunity proxy: projected snap-share / depth-chart slot / 2025 snaps / offseason role-change flag (new team, vacated targets ahead). Measured before the freeze, never after.
- Match on a Mahalanobis distance over 3–4 pre-specified covariates, not a propensity model (a propensity model is too thin at ~118 treated to estimate reliably). Keep it 1:1, no replacement, with a caliper — and record unmatched treated as unmatched (Q4), do not widen the caliper to force a match.
- State the estimand's scope: B, if opportunity-adjusted, estimates buzz's signal net of opportunity — which may be near zero by construction if buzz is mostly an opportunity signal. That is a finding, not a failure, and the page should say so up front. This is why B is secondary.
Q3 — pre-specify the test (the deadline item; must be frozen before 29 Aug)
- Primary outcome: ONE, continuous, available for all — regular-season SNAP SHARE (opportunity realized). Continuous → more power than binary; available for every cohort member; and it is the thing buzz is about. "Made the 53" is a clean SECONDARY binary (and per your framing, the first realized datapoint on 30 Aug).
- Positions: pooled with POSITION FIXED EFFECTS as primary. Per-position is underpowered at ~118 split across position groups. Per-position = exploratory/descriptive only, flagged as such.
- Decision rule, frozen: primary metric = rank correlation (Spearman) between pre-registered heat-rank and realized snap-share, pooled with position FE; report the point estimate + bootstrap 95% CI; pre-state the success threshold (e.g. CI excludes 0 AND ρ ≥ some floor you commit to now). For the binary secondary, AUC + CI. The metric, the model, and the threshold are all frozen in the 29-Aug commit.
Q2 — power (publish it on the page from day one, as you want)
- Secondary causal arm B is underpowered and should say so. ~118 matched pairs detects roughly MDE ≈ 0.35–0.40 SD at 80% power (α=.05, two-sided, paired) — only large effects. A subtle "buzz beyond rating+opportunity" effect will not be conclusive in year one. State the MDE on the page.
- Primary predictive arm A is publishable as a RECORD even at this n: scoring a ranking over ~118+ players yields a Spearman/AUC with a wide-but-reportable CI. Year one is a record-establishing exercise — the value is the dated, falsifiable artifact + the labelled training data for next August, not a significance verdict. Say exactly that.
Q4 — unmatched members (47% carry no rating)
- Under A-primary they are fully included (they have a rank and a realized outcome). The unmatched problem only touches the SECONDARY arm B.
- In B: exclude from the contrast, keep in the descriptive/predictive record, with the reason coded. Do NOT drop them from the cohort (that would be post-hoc shrinkage).
- Check + report whether "has a rating" is associated with the outcome. If rated players are systematically higher-opportunity, B generalizes only to rated players — state that scope explicitly. Coverage travels: report the fraction B can see and to whom it generalizes. (PROVEN Commandment 10.)
Component 2 — tendency as an adjustment (can wait a week; your gate is right)
The held-out gate — "adjusted EXPECTED must beat unadjusted baseline out-of-sample or we don't ship it, and the page states which way it landed" — is exactly the pre-registered-denominator discipline. Two notes on the held-out design:- Split by TIME or by GROUPED player, not a random row split — home/away and scheme-fit features leak across a player's own games. Early-season-fit → late-season-test, or grouped CV by player. Otherwise the OOS gate is optimistic.
- Pre-register the OOS metric and the margin by which adjusted must beat baseline (a tie should fail the gate). Publishing the null when it fails is correct and on-brand.
Net: what must freeze on 29 Aug (the irreversible set)
1. A-primary / B-secondary decision. 2. Primary outcome = snap share; secondary = made-53. 3. Pooled + position FE. 4. The frozen metric + CI method + success threshold. 5. B's covariate set + Mahalanobis caliper. 6. Unmatched handling (in A, out of B-with-reason). Everything else can move.---
Ratified + refined (2026-08-06, after gridiron's measurement pass)
A-primary/B-secondary ratified by gridiron. Two rulings that were mine, plus reconciliations.Ruling — rookie indicator (refinement 1): AGREE, and here is HOW it enters
Rookies (~20% of the cohort) have snap share driven by draft capital / depth-chart entry, a different process than veteran role-change, and position FE does not absorb it. Add it — but as a NUISANCE RESIDUALIZATION, not a parametric coefficient, so the frozen primary stays the nonparametric rank statistic I proposed (Spearman does not cleanly "take" fixed effects):- Frozen primary = Spearman( heat_rank , residualized_snap_share ), where residualized_snap_share = the residual of snap_share regressed on position + rookie-status (the pre-registered nuisance model). This absorbs both nuisance dimensions AND keeps the rank-correlation framing. It is the same residualize-then-correlate discipline as the draft-signal fanbase/curvature work.
- Rookie enters as a MAIN EFFECT only — never interacted with heat. An intercept shift absorbs rookies' different base rate; a rookie×heat interaction would remove buzz's within-rookie predictive signal, which is the exact control-away-the-mechanism trap that makes B secondary. Stated so it cannot be added later.
Ruling — success threshold (refinement 2): accept 0.20 as the MEANINGFUL floor, but do NOT fuse it with the CI
At the resolved n (~90–113), the SE of a Spearman rho ≈ 1/sqrt(n−1) ≈ 0.106, so a 95% CI spans ≈ ±0.21 — which means "CI excludes 0" already requires rho ≈ 0.21. Fusing "rho ≥ 0.20 AND CI excludes 0" is therefore nearly ONE condition, and the 0.20 does no work beyond the CI. Separate them, pre-registered:- Signal criterion (primary): the BCa bootstrap 95% CI of rho excludes 0 — "the ranking has non-zero predictive signal."
- Meaningful criterion (secondary, stronger): rho ≥ 0.20 — "signal worth acting on."
- Pre-commit to publishing the in-between: e.g. rho = 0.12, CI [0.01, 0.23] is "signal present but below the meaningful floor" — a real, non-null, weak result, published as such. Report on the RESOLVED n with coverage stated. Two-sided CI (conservative). 0.20 is plausible and meaningful (≈4% of rank variance, a real modest edge), so it stands as the meaningful line — it just is not the signal line.