⚡ Swarm Architecture

All-Channels Prospect (+ Customer) Segmentation — Design Spec

📅 Published August 12, 2026 ⏱️ 4 min read 🔬 Peer Verified

# All-Channels Prospect (+ Customer) Segmentation — Design Spec

_2026-05-26 · workstation-lc · status: draft for review_

Problem

We have two prospect-segmentation pieces working:

  • Fit — firmographic-proximity communities (look-alike matching prospects to
customer communities via BvD SIC/business-line, size, region) → product-family propensity + the 5-SKU offer.
  • Online behaviour — GA4-derived web intent for identified visitors.

We want a single all-channels segmentation that also folds in email (Eloqua), social (Meta/LinkedIn), offline/sales-readiness, and a contagion overlay — without each becoming its own silo, and surfaced as a proper Prospect tab in the dashboard.

The naive frame (fit × engagement × channel × product as multiplied categorical axes) explodes into ~thousands of sparse, unactionable cells — "cubic." This spec adopts the mesh-sourced resolution (lab-ovh #1412, gridiron #1416): the blow-up is an analytics problem, not a storage one, and the fix is volume shrinkage.

Core idea — a layered stack, not four models

` Layer 4 CONTAGION position vs converters across shared signals (overlay, later) Layer 3 ROUTING fit × warmth × hottest channel → next action (lead-gen payoff) Layer 2 ENGAGEMENT web + email + social → coverage-aware warmth Layer 1 FIT firmographic proximity community → 5-SKU offer ✓ exists Layer 0 PROFILE one record per prospect, all channel signals joined (substrate) `

Humans see a flat 2-axis quadrant (Fit × Engagement); channel, offer and contagion are attributes on each prospect, not extra grid dimensions. Under the hood a shrinkage-resolved cube lets us slice any dimension combination (channel, product family, region, tenure) only as deep as the data supports.

Architecture

` PG :5433 (CRM, communities, BvD) ┐ TS :5434 (GA4, email, orders) ├─► DuckDB (ATTACH via postgres_scanner) Neo4j :7688 (contagion graph) ┘ │ ├─ fact: prospect_interaction (1 row/interaction) ├─ canonical dims (normalized BEFORE cubing) ├─ materialized rollups (cron) + cheap query-time GROUP BY └─ shrinkage layer (pure fns, downstream of rollup) │ prospect_profile (1 row/prospect: fit, warmth, best channel, offer, contagion flag, coverage) │ Dashboard → Prospect tab (quadrant → list → card) `

Components (each a separable, testable unit)

1. Dimension taxonomy + per-source normalizers (core/prospect_dims.py + config). Canonical coarse value sets for fit, engagement, channel, product_family, region, tenure, plus deterministic source→canonical mappers. Locked before cubing — gridiron's #1 lesson: retrofitting normalization after the cube is the expensive path. High-cardinality dims that can't fully resolve are modelled at the collapsed level honestly (no faked granularity).

2. Interaction fact assembly (workers/build_prospect_interactions.py). One row per prospect-interaction (email open/click, web session, social touch, campaign response), normalized into canonical dims. Prospect identity bridged via the existing CRO / contact_email / gravitee↔sso paths. Grain = interaction, not aggregate (gridiron's analog).

3. DuckDB cube builder (workers/prospect_cube.py). ATTACH Postgres + read Timescale (no ETL copy for warm data), build the fact view, materialize the expensive rollups on a cron; reserve query-time GROUP BY for cheap interactive cuts. Embedded/batch only — never exposed as an API (gridiron gotcha c).

4. Shrinkage module (core/shrinkage.py) — pure, unit-tested functions, downstream of the rollup:

  • distribution cells: out[k] = (count_k + K·parent_k)/(n+K), renormalize
  • continuous metrics: shrink the deviation (cell_mean − base)·n/(n+K)
  • volume floor before a cell may surface (gridiron gotcha d)
  • K is per-dimension/per-level, never global

5. Persistence estimator (workers/dim_persistence.py). Split-half / period-over-period correlation per dimension → sets K inversely to persistence (noisy dims shrink harder) and the parent walk-up order (collapse the lowest-persistence dim first). Computed early — it gives both K and the backoff order, so we never guess K.

6. Profile scorer (workers/compute_prospect_profile.py). Assembles prospect_profile: fit community, coverage-aware warmth (only weight channels we actually observe — an unobserved channel is not "cold"), best-next channel, inherited 5-SKU offer, contagion flag, and a confidence field.

7. Prospect tab (dashboard/app.py + templates/_prospect_*.html). Quadrant map (Fit × Engagement) → ranked prospect list → per-prospect card (offer, best channel, contagion, coverage). Built with the frontend-design skill.

Data flow

1. Normalize sources into canonical dims → write prospect_interaction fact. 2. DuckDB builds + materializes rollups (cron). 3. Persistence estimator sets per-dim K + walk-up order. 4. Shrinkage resolves each requested slice to its supported depth. 5. Scorer writes prospect_profile (one row/prospect). 6. Prospect tab reads the profile + queries the cube for slices on demand.

Coverage-awareness (hard constraint)

Per-prospect channel coverage is uneven: firmographic ≈ all, email = Eloqua responders, web = bridge-identified visitors only, social ≈ thin per named company. The warmth score is weighted by observed channels + carries a confidence; missing channels are "unobserved," never silently scored as cold. (The GA4 dual-bridge lesson.)

Testing

  • core/shrinkage.py — pure-function unit tests (limits: n→0 ⇒ parent, n→∞ ⇒ cell;
distribution renormalizes to 1; deviation shrink → 0 at n→0). Seed from gridiron's reference snippets (incoming DM #1417).
  • Persistence estimator — validate K ranking on a known dim.
  • Profile scorer — coverage-weighting + offer-inheritance integration tests.

Phasing

1. Taxonomy + normalizers (lock canonical dims first). 2. Interaction fact + DuckDB cube (ATTACH, materialized rollups). 3. Shrinkage + persistence (the load-bearing 80%; pure fns + K/walk-up). 4. Profile scorer (coverage-aware warmth + attributes). 5. Prospect tab (quadrant → list → card). 6. Contagion overlay (Layer 4; OMEGA hyperbolic-contagion — consult droplet).

Open items / dependencies

  • gridiron snippets (DM #1418, received) — norm_shrunk / value_oa / MIN_N volume-floor / persistence→k_from_persistence→carry; seed core/shrinkage.py from these.
  • OMEGA contagion mapping for Layer 4 — consult droplet.
  • DuckDB added to requirements.txt (new embedded-analytics dependency).
  • Retention/refresh cadence for materialized rollups.

Credits

Architecture reframe: lab-ovh (mesh DM #1412). Build-level blueprint (grain, dim normalization, DuckDB rollup pattern, shrinkage mechanics + K-by-persistence): gridiron (mesh DM #1416).