Consent, documents & notifications
Schema deep-dive · living document · #12 in the reading sequence
| Tables | consent.consent_records, consent_audit, deletion_requests[1] · documents.documents, templates[8] · notifications.notification_log, notification_preferences, projection_checkpoints[16] (+ each service's outbox) |
| Owner services | consent (:4012) · document-service (:4013) · notifications (:4006) - each the sole writer of its own database |
| Locators | DOC-YYYY-NNNNNN (documents) · NTF-YYYY-xxxxxxxx (notification log) · consent rows have no locator |
| Last updated | 2026-08-18 |
| Companion | ERD story, slide 12 · plan documents (PDF) · previous: Care |
1. Scope and usage
The supporting ring: three small services, one pattern. Each subscribes to platform events and keeps its own party_locator-keyed record of a concern no core service should own:
- consent - what the member has allowed (per-type grant/withdraw), the append-only audit trail of every change, and GDPR Art-17 erasure requests.
- documents - what the platform has rendered for the party: the Benefit Schedule, invoice and employer-contract PDFs, stored with the template that produced them.
- notifications - what the platform has told the party: a delivery log per event × channel, plus per-party channel preferences. Templates, routing and digesting live in Novu; this service is the thin adapter that keeps the record (Notifications platform posture).
None of the three participates in adjudication, money or cover. All three are derived surfaces: destroy any of them and the insurance facts survive (the policies, claims and invoices they describe live upstream) - but the compliance story does not, which is exactly why they exist as services.
2. Boundaries and relationships
| This ring is not… | That concern lives in | Join |
|---|---|---|
| authorization - a granted consent is a recorded fact that specific features read (the Slack personalisation gate, §7), not a general access gate in front of the data path | the authz design track (PII walls); today's gates are JWT role + ownership checks[3] | - |
the PII store - consent knows that you allowed sharing, not your data; erasure requests land here but member PII lives in policy_admin.parties | Party | party_locator |
| the invoice or policy - a document renders them; the authoritative rows stay in billing and enrollment, and the PDF re-derives from the event payload + internal reads[11] | billing / enrollment | invoice_locator, policy_locator |
| a template engine choice - document templates are HTML rows per (market, type), rendered by wkhtmltopdf in-process[13] | documents.templates | market_code + template_type |
| Novu - workflow definitions, channel routing, digest rules and the actual SMTP/Slack hops live in Novu on uat; the Go service triggers workflows and keeps the log[22] | Novu (notifications.dev.hiolly.com) | subscriber id = party_locator |
the party's contact record - the dispatcher fetches email/phone live from policy-admin per event[24]; the email/phone columns on preferences are carried but empty on every live row | Party | party_locator |
Kafka's own offset store - projection_checkpoints is an application-level shadow of consumer progress; the consumer also commits group offsets to Kafka[21] | __consumer_offsets | (topic, partition) |
The shared join is party_locator - always a soft locator reference, never an FK (cross-service rule, data architecture). Nothing validates the format on write: live consent rows include PARTY-ATOM-* and PTY-E2E-* test strings alongside real PTY- locators (see §9).
3. Structure
consent
consent_records - the current answer, one row per (party, consent type):
| Field | Type | Req | Notes |
|---|---|---|---|
id | uuid | ✓ | PK |
party_locator | text | ✓ | The consenting subject. UNIQUE with consent_type |
consent_type | text | ✓ | Vocabulary is application-level only - see below |
granted | boolean | ✓ | Default false. The entire current state |
created_at, updated_at | timestamptz | ✓ | Bookkeeping |
consent_audit - append-only history (no UPDATE or DELETE path exists in the repository layer; erasure is the one exception, and it only redacts reason):
| Field | Type | Req | Notes |
|---|---|---|---|
id | uuid | ✓ | PK |
party_locator, consent_type | text | ✓ | Denormalised - the audit row survives even if the record is hard-deleted |
old_value | boolean | NULL on first-ever grant (nothing to change from) | |
new_value | boolean | ✓ | |
changed_by | text | ✓ | Free text: a PTY- locator, slack-bot, an email |
changed_by_type | text | ✓ | MEMBER | EMPLOYER | SERVICE (application enum)[4] |
reason | text | Free text; the one field SOFT erasure redacts | |
occurred_at | timestamptz | ✓ | Indexed, with party_locator |
deletion_requests - GDPR Art-17[32]. IF2 states the member right this table serves: "the right to erasure (i.e. request that their personal data be deleted and not disseminated further)"
| Field | Type | Req | Notes |
|---|---|---|---|
id | uuid | ✓ | PK; returned to the requester for polling |
party_locator | text | ✓ | Indexed |
delete_type | text | ✓ | SOFT | HARD (application check)[5] |
status | text | ✓ | Default 'PENDING'; walk is PENDING → PROCESSING → COMPLETED / FAILED |
requested_by_type | text | ✓ | MEMBER | EMPLOYER |
reason | text | Free text; redacted by SOFT erasure | |
requested_at, completed_at | timestamptz | ✓/- | completed_at stamped on terminal states |
attempt_count | integer | ✓ | Default 0; the erasure job's bounded-retry counter |
Field-by-field: what and why
consent_type - text with no CHECK and no DB vocabulary. The application validator admits NHS_DATA_SHARING, WEARABLE_SYNC, EMPLOYER_WELLBEING_REPORTING, RESEARCH_PARTICIPATION, MARKETING_COMMUNICATIONS, SLACK_CHAT_PERSONALIZATION[3]. The live table holds five distinct types: DATA_SHARING, NHS_DATA_SHARING, THIRD_PARTY_SHARING, MARKETING, SLACK_CHAT_PERSONALIZATION. Three of the five (DATA_SHARING, THIRD_PARTY_SHARING, MARKETING) come from an older vocabulary the current validator rejects - those rows can be read but never updated through today's API. Vocabulary drift between code and data, stated plainly.
granted as the whole state - why so thin? This is the open shape question of D-03 (#1006): the record carries no lawful basis, no privacy-policy version, no special-category flag, no capture provenance (which screen, which wording the member saw). Migration 0002 is the thin shape in full[1]. "Did the member consent?" is answerable today; "under which policy text, on what basis?" answers the day D-03 picks the shape, and it lands as additive nullable columns on this table and its audit twin (§9).
changed_by / changed_by_type - the attribution pair, and where the two write paths differ. The public PUT /consent/{party} requires the member role and hardcodes changed_by_type = MEMBER, overriding the path param with the JWT's verified party_locator claim so a member can only write their own record (#1544)[3]. The cluster-only /internal/consent path (X-Internal-Service secret, constant-time compare, fail-closed on empty secret) requires an explicit changed_by_type - this is how the Slack bot records SLACK_CHAT_PERSONALIZATION as SERVICE on a member's behalf[4].
Why an audit table at all? consent_audit is the platform's only working append-only audit primitive: old value, new value, who, as what, why, when - written in the same transaction as the record upsert (UpsertWithAudit)[6], which itself shares a transaction with the outbox enqueue[3]. Record + audit + event: all three commit or none do.
attempt_count and the erasure job - a poller (every 60s by default) picks up PENDING and PROCESSING rows (crash recovery), marks PROCESSING, executes, and on failure increments the counter, FAILing the request at 3 attempts and publishing deletion.failed[7]. What execution actually does is narrower than the name suggests: SOFT erasure NULLs the free-text reason columns in consent's own tables and nothing else; HARD deletes the party's consent rows, audit rows and requests - all within the consent database[33]. Cross-service erasure is signalled (deletion.completed on Kafka) but not performed; no other service consumes that topic yet.
documents
documents - one row per rendered PDF:
| Field | Type | Req | Notes |
|---|---|---|---|
id | uuid | ✓ | PK |
locator | text | ✓ | UNIQUE. DOC-YYYY-NNNNNN |
party_locator | text | ✓ | The document subject (member for schedules, employer for invoices/contracts); indexed with created_at DESC |
template_type | text | ✓ | POLICY_SCHEDULE | INVOICE | EMPLOYER_CONTRACT live |
market_code | text | ✓ | GB on every live row |
content | bytea | ✓ | The rendered PDF, in-row - see below |
policy_locator, scheme_locator | text | Soft refs so a schedule is findable by the policy it describes and the scheme it belongs to[9] | |
invoice_locator | text | Ties an INVOICE document to its billing invoice; partial-UNIQUE with template_type[10] | |
created_at | timestamptz | ✓ |
templates - UNIQUE (market_code, template_type); content_html is the full Go-HTML template. Live rows: GB × POLICY_SCHEDULE, INVOICE, EMPLOYER_CONTRACT - three, exactly as the tie-in promises: the Benefit Schedule PDFs on plan documents are this service's output, and industry practice is that "the schedule attached to the policy will usually show the applicable limits"
Field-by-field: what and why
content bytea - the whole PDF lives in the row: 713 documents, 71 MB total, ~102 kB average. Why in-row? One store, one transaction, no signed-URL infrastructure at MVP. The known direction is an object store (S3-compatible; MinIO already runs on dev-2 as the assets CDN) with the row keeping metadata + a content pointer; §9 records the swap as an extension point, and at 71 MB the current shape has room. Downloads go through GET /documents/{locator}/content (deliberately tokenless for browser navigation) and HMAC-signed time-limited download URLs[15].
locator - DOC- is minted from an in-process atomic counter seeded from MAX(locator) at boot - not the shared per-prefix DB sequence Party and Scheme use. The seeding exists because a fmt.Sscanf verb bug (#1428) once made every restart re-mint DOC-2026-000001 and collide with the UNIQUE constraint[12]. Two service instances would race this counter; there is exactly one instance.
invoice_locator + the render-once guard - Kafka is at-least-once, and a redelivered invoice.paid must not hand the employer three identical PDFs. Generate short-circuits if a document already exists for (invoice_locator, template_type)[13], and the partial UNIQUE index is the guarantee that survives a race. The index had to be repaired once: the Go zero value wrote '' instead of NULL, one poisoned row 23505-blocked every later schedule, and migration 0007 now excludes both NULL and ''[14].
template_type and the event map - the consumer maps inbound events to template types: policy.issued/policy.endorsed → POLICY_SCHEDULE, invoice.paid/invoice.finalised → INVOICE, claim.approved → EOB, prior-auth.decided → PRE_AUTH_LETTER, policy.renewal_due → RENEWAL_NOTICE[11]. That map plus the templates table is the whole extension seam for a new document type. Two of the five types the map can produce have template rows today (POLICY_SCHEDULE, INVOICE), so an EOB, pre-auth-letter or renewal-notice event resolves its type and then stops at template fetch until its row is seeded (§9). EMPLOYER_CONTRACT is the inverse: not in the event map at all; it is generated via POST /internal/documents/generate by the onboarding composer[15].
The odd outbox variant - documents.outbox has event_type only: no topic, no key columns[8], unlike the standard consent/enrollment shape. It works because the producer is constructed with one fixed topic (document-events) and publishes key-less messages[26]; the only event is document.ready (805 live rows, 0 unpublished). The worker carries the scars of the docsvc spin-loop outage: exponential backoff and log-throttling on a missing topic[27].
notifications
notification_log - one row per (source event, delivery attempt outcome):
| Field | Type | Req | Notes |
|---|---|---|---|
id | uuid | ✓ | PK |
locator | text | ✓ | UNIQUE. NTF-YYYY-xxxxxxxx (year + 8 hex chars)[22] |
event_id | uuid | ✓ | UNIQUE - the source event's id, and the consumer-side dedup key[18] |
party_id | uuid | ✓ | policy-admin's party uuid; uuid.Nil when resolution failed |
party_locator | text | ✓ | Added later with DEFAULT ''[19]; empty on every live row - see below |
event_type | text | ✓ | policy.issued, claim.approved, … (24 distinct live) |
channel | text | ✓ | EMAIL | SMS | IN_APP (application enum) |
status | text | ✓ | SENT | SKIPPED | FAILED (application enum) |
subject, body | text | The rendered content as sent (empty on SKIPPED) | |
failure_reason | text | Why a SKIPPED/FAILED row is what it is | |
attempt_count | integer | ✓ | Default 1; the retry job's counter |
sent_at, created_at | timestamptz | -/✓ | sent_at only on SENT |
notification_preferences - one row per party (UNIQUE party_locator): preferred_channel (default 'EMAIL'), opted_out_events jsonb array of event types, plus carried-but-unused email/phone columns.
projection_checkpoints - UNIQUE (topic, partition_id) + offset: the app-level record of consumer progress, upserted after every message[21]. Live rows show what is actually consumed: enrollment-events (note the hyphen - the topic zoo is real, §9), claims.events, and care.events × 3 partitions.
Field-by-field: what and why
event_id UNIQUE - the consumer-side dedup exemplar. The outbox pattern guarantees at-least-once delivery; somebody has to make it exactly-once at the point of effect. This service does it the canonical way:
INSERT INTO notifications.notification_log (...)
VALUES (...)
ON CONFLICT (event_id) DO NOTHINGInsertIdempotent returns whether the row was actually inserted; a redelivered event logs "duplicate event, skipped" and does not double-send[20][22]. This is the pattern every other consumer in the estate is asked to copy.
status and the dispatch pipeline - per event: resolve the party's contact from policy-admin → load preferences (preferred channel, opt-outs) → render the event's template → send → write the log row with the outcome[22]. Every early exit is a SKIPPED row with an honest failure_reason ("party not found", "party opted out of this event type", "no email address on file") - the log records decisions, not just deliveries.
The live numbers demand honesty. 1,890 log rows: 1,887 SKIPPED ("party not found", party_id = uuid.Nil, party_locator = '') and 3 SENT - all three of which are seeder rows (locator prefix NLOG-, a format the code never mints). The mechanism: the consumer reads partyLocator from the envelope top level[21], but enrollment's producer defines the envelope field and never populates it[25][25] (the locator rides inside payload, where document-service reads it and succeeds). So policy-admin is asked for party "", 404s, and the event logs SKIPPED. Fix: set partyLocator on the envelope in the producing services, or have the consumer fall back to payload.partyLocator the way document-service already does[21]. Real deliveries take the other path: POST /send → Novu (SendDirect, with SMTP fallback), which by declared Flow-0 scope writes no log row at all[22] - extending the log to that path is the other half of the same fix (§9).
opted_out_events - jsonb array of event-type strings, checked per dispatch[22]. Opt-out granularity is per event type, not per category or channel - Novu's preference model will eventually subsume this.
care.* events take a different lane - appointment confirmations and reminders become Novu in-app category notifications and deliberately bypass the email dispatcher (a confirmation email and an inbox row would double-send)[23]. These write no notification_log row either - Novu keeps that delivery record.
4. Invariants
| Invariant | Enforced by |
|---|---|
| One consent state per (party, type) | DB UNIQUE uq_consent_party_type[1] |
| Every consent change leaves an audit row + an outbox event | Application transaction: upsert + audit[6] + enqueue[3] in one tx |
| Audit is append-only | Convention + code shape (no update/delete path); not a DB rule - a GRANT or trigger would harden it |
consent_type, changed_by_type, delete_type, requested_by_type vocabularies | Application only[3][5] - no CHECKs anywhere in this ring |
| One active deletion request per party | Application 409 check[5]; racy by design (no partial unique index) |
| Erasure retries are bounded | Application: attempt_count vs max 3, then FAILED + deletion.failed[7] |
documents.locator unique | DB UNIQUE[8] (but minted by an in-process counter - single-instance assumption) |
| One template per (market, type) | DB UNIQUE[28] |
| One document per (invoice, type) | DB partial UNIQUE (NULL and '' excluded)[14] + application short-circuit[13] |
Document + document.ready outbox row commit together | Application transaction[13] |
A .pdf really is a PDF | Application: HTML fallback refused unless PDF_FALLBACK_HTML=true[13] |
| One notification effect per source event | DB UNIQUE event_id + ON CONFLICT DO NOTHING[18][20] |
| One preference row per party | DB UNIQUE party_locator[17] |
| One checkpoint per (topic, partition) | DB UNIQUE + upsert[21] |
| Members read/write only their own rows | Application: JWT party_locator claim overrides path/query on every member-facing route in all three services (#1544)[3] |
party_locator points at a real party | Nothing - live consent data contains thousands of test-fabricated locators |
| Row changes captured to CDC | Debezium publications dbz_consent, dbz_documents, dbz_notifications (live \d) |
5. Lifecycle
The rows here have trivial state machines (a consent flips, a document is immutable once rendered, a log row is terminal at SENT/SKIPPED/FAILED except for the bounded FAILED-retry). The lifecycle that matters is the shared consumption pattern - platform event → consumer → party-keyed record. Concretely, claim.approved reaching the notification log:
The same shape drives documents (policy.issued on enrollment-events → render PDF → documents row + document.ready outbox) and consent in reverse (consent is a producer: member PUT → record + audit + outbox → consent.events). Two honest annotations on the diagram: step 4 currently receives an empty partyLocator from every producer (§3), and step 8's checkpoint is saved in a defer - it advances even when dispatch failed, so the checkpoint means "seen", not "successfully processed"[21].
6. Populated example
One real row-set per service (values from the live databases 2026-08-18; names/emails fictionalised, locators real).
A consent record and its audit trail
The best organic chain in the estate - the Slack bot recording chat personalisation consent after the member replied YES in a DM (the /internal/consent SERVICE path, §3):
{
"party_locator": "PTY-2026-000121",
"consent_type": "SLACK_CHAT_PERSONALIZATION",
"granted": true,
"created_at": "2026-07-21T13:37:56Z",
"updated_at": "2026-07-21T13:37:56Z"
}{
"party_locator": "PTY-2026-000121",
"consent_type": "SLACK_CHAT_PERSONALIZATION",
"old_value": null,
"new_value": true,
"changed_by": "slack-bot",
"changed_by_type": "SERVICE",
"reason": "member replied YES in Slack DM",
"occurred_at": "2026-07-21T13:37:56Z"
}{
"topic": "consent.events",
"key": "PTY-2026-000121",
"payload": {
"eventType": "consent.changed",
"partyLocator": "PTY-2026-000121",
"consentType": "SLACK_CHAT_PERSONALIZATION",
"granted": true,
"changedByType": "SERVICE",
"occurredAt": "2026-07-21T13:37:56Z"
},
"published": true
}| Field | Read by | What actually happens |
|---|---|---|
granted | the Slack chat brain, via GET /internal/consent/{party} | personalisation on/off for that member's DMs |
old_value: null | the audit API | renders as "first-ever grant" - there was no prior state |
changed_by_type: SERVICE | audit readers | distinguishes bot-recorded consent from member self-service (which hardcodes MEMBER) |
outbox key | Kafka partitioning | per-party ordering on consent.events; today the topic has no consumer |
The seeded demo member PTY-2026-000004 shows the multi-type shape: MARKETING true, DATA_SHARING false, THIRD_PARTY_SHARING true - with a DATA_SHARING audit entry recording old_value: true → new_value: false, a real withdraw. All three types are legacy vocabulary the current validator no longer accepts (§3).
A document row (content omitted)
The latest member Benefit Schedule:
{
"locator": "DOC-2026-001185",
"party_locator": "PTY-2026-000741",
"template_type": "POLICY_SCHEDULE",
"market_code": "GB",
"policy_locator": "POL-2026-001471",
"scheme_locator": "SCH-2026-001214",
"invoice_locator": null,
"content": "<115,967 bytes of rendered PDF - the bytea column>",
"created_at": "2026-08-05T…"
}| Field | Read by | What actually happens |
|---|---|---|
party_locator | member app / member-portal-api document list | "your documents" for the member |
policy_locator | schedule lookup by policy | "which PDF describes POL-2026-001471" |
scheme_locator | employer document listing (via the group-scheme proxy) | the employer sees their scheme's documents |
invoice_locator: null | the partial UNIQUE index | row is outside the invoice dedup constraint - schedules dedup by nothing (a policy.endorsed legitimately re-issues) |
content | GET /documents/{locator}/content + signed download URLs | streamed straight out of Postgres |
The invoice shape differs only in keys: live row DOC-2026-001200 has template_type: INVOICE, party_locator: PTY-2026-000021 (the employer party), invoice_locator: INV-2026-000243, policy_locator: null - one document row serving the employer's Billing tab.
A notification log row
The representative live row is a SKIPPED one - 1,887 of 1,890 are:
{
"locator": "NTF-2026-1453ef04",
"event_id": "dd78edb8-b419-464c-8d84-fb92a6c0b02a",
"party_id": "00000000-0000-0000-0000-000000000000",
"party_locator": "",
"event_type": "scheme.member_enrolled",
"channel": "EMAIL",
"status": "SKIPPED",
"subject": null,
"body": null,
"failure_reason": "party not found",
"attempt_count": 1,
"sent_at": null,
"created_at": "2026-08-17T08:33:32Z"
}| Field | Read by | What actually happens |
|---|---|---|
event_id | InsertIdempotent | a Kafka redelivery of this event inserts nothing - dedup held |
party_id: uuid.Nil + party_locator: "" | nobody | the envelope-partyLocator gap (§3) fossilised into data: resolution was attempted against "" |
status: SKIPPED + failure_reason | web-admin log views, this audit | the pipeline honestly recorded why it did nothing |
attempt_count | retry job | only FAILED rows are retried; SKIPPED is terminal |
And the preferences row the dispatcher would have consulted:
{
"party_locator": "PTY-2026-000006",
"preferred_channel": "SMS",
"opted_out_events": [],
"email": "",
"phone": ""
}preferred_channel would route this member to SMS; email/phone are empty because contact truth lives on the Party.
7. Who references these tables
This ring is mostly fan-in (it consumes everyone else's locators). The fan-out that exists:
| Reference | Who | Meaning |
|---|---|---|
DOC- locator | member app / member-portal-api, employer app (via group-scheme proxy), signed download URLs | fetch and list rendered documents |
document.ready on document-events | published for downstream listeners; no consumer wired today | "a document exists for party X" |
NTF- locator | web-admin notification log screens (/log/list, /log/{locator})[29] | delivery-record drill-down |
consent.events topic | exists, keyed per party; no consumer today | future: eligibility/PII gates reacting to withdrawals |
deletion.completed / deletion.failed topics | published directly by the erasure job[30]; no consumer today | future: cross-service erasure fan-out |
Consent state via GET /internal/consent/{party} | Slack chat brain (personalisation gate) | the one live in-request consent read |
| All nine tables | Debezium → BigQuery olly_cdc | analytics/BI lineage |
No table in this ring is referenced by locator from any other service's schema - by design: the ring points outward (party_locator, policy_locator, invoice_locator), nothing points in.
8. Design determinations
- Consent is service-owned, not captured-and-synced - and deliberately thin until the record shape is decided. D-03 · #1006 (open); the missing lawful-basis/provenance fields are that decision's scope.
- Every consent change is transactionally audited - record + audit + outbox in one tx; the platform's reference append-only audit primitive. (§3, §4)
- Novu owns delivery; the Go service owns the record - templates, routing, digests and channel providers live in Novu (2026-07-21 architecture correction: this service is Novu's sole caller for Slack objects[22]); other Go services call
olly-notifications, not Novu. - Consumer-side dedup by source
event_id- the UNIQUE +ON CONFLICT DO NOTHINGidiom that completes the outbox pattern's at-least-once contract. (§3) - Documents render once per (invoice, type) - idempotency belongs at the point of effect, enforced twice: application short-circuit + partial unique index. (§3)
- Erasure is polled, attributed and bounded - PENDING/PROCESSING crash-recovery,
attempt_countcapped at 3, terminal events published. (§3) - Member self-service is claim-scoped everywhere - #1544: the JWT's
party_locatoroverrides client-supplied paths and filters on every member-facing read/write across all three services.
9. Caveats and extensibility
Group and individual cover share the ring unchanged. Everything here keys on party_locator, and an employer is an ORGANISATION party: the same documents table holds the member's POLICY_SCHEDULE (party_locator = member) and the employer's INVOICE and EMPLOYER_CONTRACT (party_locator = employer, scheme_locator/invoice_locator set). Consent requested_by_type already admits EMPLOYER for erasure-on-behalf. A direct-to-consumer individual needs no schema change anywhere in this ring - their documents carry no scheme_locator.
How the ring absorbs new purposes. Each of the three services keeps its vocabulary out of the DDL and in one small application surface: consent's purpose list is a validator map over a free-text column, document generation is template rows keyed by (market, type), and notification content and routing live in Novu workflows rather than in Go. So a new consent purpose, a new document, a new market or a new channel is a row or a map entry, not a migration - and party_locator keying means none of it cares whether the subject is a scheme member or a direct customer.
Extension points - when, what, where:
| When we need ... | What to add | Where |
|---|---|---|
| a new consent purpose (a new partner data share, a new processing purpose) | one entry in the validator map. consent_type is free text with no CHECK, so the purpose is data: the record, the audit row and the consent.events payload all carry it unchanged | validConsentTypes in consent/internal/handler/consent.go[3] |
| lawful basis, privacy-policy version, special-category flag, capture provenance on a consent | additive nullable columns on consent_records and the same on consent_audit; the existing rows need no reshape and the upsert-plus-audit transaction is unchanged | consent/migrations/[1] · shape set by D-03 (#1006) |
| a new service recording consent on a member's behalf (another bot, a partner) | call the cluster-only path with an explicit changed_by_type; attribution, the audit row and the outbox event follow the same transaction the Slack bot uses | /internal/consent[4] |
| a consumer that reacts to withdrawals (eligibility or PII gates) | a subscriber on consent.events; the topic is already published per party and keyed for per-party ordering | consent.events (§6) |
| cross-service Art-17 erasure | a consumer of deletion.completed that fans the erasure out to policy-admin, claims, billing and the rest. Today erasure stops at the consent database, so this is the piece that makes the name true | deletion.completed / .failed[30] · Consent audit & data-subject rights (#17) |
| a new document type | a templates row for its (market, type) pair, plus one line in the event map for event-driven generation. EOB, PRE_AUTH_LETTER and RENEWAL_NOTICE already resolve from events and need the row | documents.templates[28] · eventToTemplateType[11] |
| a new market | the same template types under a new market_code; templates and documents already key on (market, type), and a ('DE', 'INVOICE') row is the whole per-market story here | documents.templates UNIQUE (market, type)[28] |
| PDFs held outside Postgres | a content-pointer column beside the metadata, backed by an S3-compatible store (MinIO already runs on dev-2 as the assets CDN); the download and signed-URL handlers are the only readers to re-point | documents.documents.content |
| a second document event, or per-party ordering | topic and key columns on documents.outbox, matching the shape consent and enrollment use; the current fixed-topic, key-less producer covers exactly one event type | documents.outbox[8] · producer[26] |
| a new notification channel or message | a Novu workflow. Novu is the canonical provider: templates, channel routing and digesting live there, the subscriber id is the party_locator, and the Go side keeps the delivery record | Novu (notifications.dev.hiolly.com) · SendDirect[22] |
| per-member routing on the delivering path | reconcile notification_preferences with Novu's subscriber preferences - feed Novu from this table, or read Novu's and retire the columns. preferred_channel and opted_out_events are the Go-side inputs and the dispatcher already consults them | notification_preferences · Operational depth (#20) |
Notification preferences and the document templates inventory are covered in depth in Operational depth (#20); the audit trail and the erasure queue in Consent audit & data-subject rights (#17).
Known warts, stated (each as the fix and where it goes):
- The event-driven delivery log records no organic delivery. 1,887/1,890 rows are SKIPPED "party not found" because the producers omit the envelope-level
partyLocatorthe consumer reads[25]; the 3 SENT rows are seeder artifacts (NLOG-prefix, a format the code does not mint). Fix, two halves: populatepartyLocatoron the envelope in each producing service (or fall back topayload.partyLocatorinconsumer.go), and extend the log to thePOST /send→ Novu path, which writes no row by declared Flow-0 scope[22]. - Consent data is ~99.8% test residue - 2,154 of 2,162 records are
PARTY-ATOM-*atomicity-harness rows (allNHS_DATA_SHARING, changed_byatomicity@test.olly); the real member-consent estate is 4 records across 2 parties. Fix: register the harness on the journey teardown ledger and delete the existing rows; aPTY-format check on write inconsent/internal/handler/consent.gostops the next batch. - Consent vocabulary drift - live types
DATA_SHARING,MARKETING,THIRD_PARTY_SHARINGare rejected by the current validator (MARKETINGvsMARKETING_COMMUNICATIONS), so those rows are readable but not updatable through the API - a consent a member cannot withdraw. Fix: admit the legacy names invalidConsentTypesor migrate the rows to the current vocabulary, then redraw the ERD. - The erasure job's terminal events bypass the outbox -
deletion.completed/deletion.failedare published fire-and-forget to Kafka[30] from the poller, the pattern the outbox was introduced to eliminate: a broker outage at completion time loses the signal (the row's status survives). Fix: enqueue them onpublic.outboxin the same transaction as the status write. - Erasure stops at the consent database. SOFT NULLs
reasontext in consent's own two tables[33] and HARD deletes the party's consent rows;consent_recordscontent and all cross-service PII are untouched, and the Art-17 fan-out is signalled on Kafka with no consumer. Stated the same way in #17. Fix: thedeletion.completedconsumer in the table above; until it exists, do not describe a COMPLETED request as estate-wide erasure. - Declared-but-unrenderable documents -
claim.approved,prior-auth.decidedandpolicy.renewal_dueresolve to template types with no template row, so they stop at template fetch instead of generating. Fix: seed the threetemplatesrows (see the table above). Separately, the event map has to decode two envelope spellings because billing's outbox worker puts the topic inevent_type[11] - fix that at the billing producer, not by growing the decoder. - The topic zoo - notifications consumes
claims.events, document-service consumesclaims-events; both topics exist on the broker, alongsideenrollment,enrollment-eventsandenrollment.events, and a barebilling(in notifications' deployed subscription) next tobilling.events. Every hyphen/dot variant is a place an event can go nowhere unnoticed. Fix: one naming rule (<service>.events), applied to the producers' topic constants and the consumers'KAFKA_TOPICS. - Checkpoints advance on failure - the checkpoint upsert runs in a
deferregardless of dispatch outcome[21], so the row means "seen", not "processed", and it duplicates what Kafka group offsets already track. Fix: save the checkpoint on the success path only, or drop the table and rely on group offsets. consent's outbox lives in thepublicschema - migration 0005 createsoutboxunqualified[31], unlike every sibling table in theconsentschema, so\dt consent.*reads as though there is no outbox. Fix: a migration that moves it into theconsentschema, or a comment on the schema that says where it is.notification_preferences.email/phoneduplicate the Party - carried, writable through the API, and empty on every live row while contact truth is resolved live from policy-admin per event[24]. Fix: drop the two columns, or make the Party their only writer (#20).
10. D2C Quote & Buy additions
D2C QnB additions, grounded in branches
feat/1675+fix/1640, pending merge.
The direct-to-consumer Quote & Buy (QnB) journey extends this ring in three places: a health-document template set, a versioned consent-capture shape, and a debug in-app inbox mirror. Both branches are unmerged, so these cite path:line on the branch rather than a pinned main SHA.
10.1 The D2C health-document set
Migration 0009_d2c_health_documents.sql seeds five GB templates on top of the reused POLICY_SCHEDULE (member-level, unchanged). The seeded template_type inventory grows from three to eight; the live document-row enumeration in §3 (POLICY_SCHEDULE | INVOICE | EMPLOYER_CONTRACT) is unchanged, because none of the five new types has produced a documents row yet (see the gap below).
template_type (GB) | Document | Seeded by 0009 | Generated by an event today |
|---|---|---|---|
POLICY_SCHEDULE | Benefit schedule | pre-existing, reused | yes, policy.issued / policy.endorsed |
IPID | Insurance Product Information Document | ✓ | no |
CERTIFICATE_OF_INSURANCE | Certificate of insurance | ✓ | no |
POLICY_WORDING | Policy wording | ✓ | no |
DIRECT_DEBIT_GUARANTEE | Direct Debit Guarantee | ✓ | no |
MEMBERSHIP_CARD | Membership card | ✓ | no |
Grounding: services/document-service/migrations/0009_d2c_health_documents.sql seeds IPID (L19-46), CERTIFICATE_OF_INSURANCE (L50-70), POLICY_WORDING (L74-92), DIRECT_DEBIT_GUARANTEE (L96-115), MEMBERSHIP_CARD (L119-136), each ON CONFLICT (market_code, template_type) DO UPDATE (idempotent re-run). Templates hold no plan facts: everything dynamic arrives via ExtraData (map[string]string) or the safe func for pre-rendered HTML, and missing keys render empty under guards, so a partial payload never breaks a render (header comment L8-12).
The honest gap: seeded, not triggered. Only POLICY_SCHEDULE is generated on policy issue. The consumer's event-to-template map routes policy.issued / policy.endorsed to POLICY_SCHEDULE (and invoice.* to INVOICE, claim.approved to EOB, prior-auth.decided to PRE_AUTH_LETTER, policy.renewal_due to RENEWAL_NOTICE); none of the five D2C types appears in it, so no inbound event produces an IPID, certificate, wording, DD guarantee or card (services/document-service/internal/kafka/consumer.go:30-38). They render only on demand through POST /internal/documents/generate (the same path EMPLOYER_CONTRACT uses, §3), and no QnB issuance composer fans the pack today. Closing this is a real gap: an event-map entry (with a synthetic issue-time trigger) or a composer that generates the set at bind, not a stub.
10.2 consent.events consumed; the flat envelope and deterministic event_id
QnB makes notifications a consumer of consent.events: the topic is now in the default KAFKA_TOPICS (services/notifications/internal/config/config.go:92, alongside enrollment-events, enrollment.events, claims.events, billing.events, care.events).
consent.events carries a flat envelope. The consent producer emits top-level partyLocator / consentType / granted / changedByType / occurredAt with no eventId and no nested payload (services/consent/internal/kafka/producer.go:53-61). The notifications consumer's envelope expects an eventId; when uuid.Parse fails it derives a deterministic id, uuid.NewSHA1(uuid.NameSpaceOID, rawBytes), rather than dropping the event, so the consumer-side dedup contract (event_id UNIQUE + ON CONFLICT DO NOTHING, §3) still holds: identical bytes map to the same id (services/notifications/internal/projection/consumer.go:51-58).
The D2C consent capture is version-keyed. The QnB quote flow captures PRIVACY_POLICY and TERMS_OF_SERVICE, each pinned to the wording version the member saw via a documentVersion field (for example "2026-08"). Both types are admitted by the validator (services/consent/internal/handler/consent.go:23-24), the update body carries documentVersion (L93), and recordConsentChange keys the prior-value lookup on (consentType, documentVersion) and writes it onto the record and its audit row (L125, L129, L137-139). This is the FR-37 versioned-document-acceptance shape, and it is the first slice of the D-03 gap §3 named ("no privacy-policy version" on a consent): the version the member accepted is now recorded, though lawful basis, special-category flag and full capture provenance remain open under D-03 (#1006).
10.3 The debug in-app inbox mirror (flag-gated)
QnB adds a development-only mirror: when NOTIFICATIONS_DEBUG_INBOX=true, the dispatcher reflects every event it consumes (enrollment, consent, billing, and the rest) into the member's Novu in-app inbox through the DebugInboxWorkflow (default debug-notification), keyed by subscriberId = partyLocator, regardless of the member's preferred channel, so the whole lifecycle is observable in the app.
| Property | Behaviour |
|---|---|
| Gate | NOTIFICATIONS_DEBUG_INBOX env flag, OFF on the live app (dispatcher.go:120-125, 364-391) |
| Timing | fires before contact resolution (in-app needs only the party as subscriberId, not an email) |
| Title | humanized via debugInboxTitle: "Policy issued · POL-2026-001521", or "Consent granted" when there is no locator (templates.go:135-144) |
No partyLocator | WARN-logged as the audit signal that the producer dropped the party, not mirrored (dispatcher.go:372-373) |
| Log row | none: it is a Novu in-app trigger, so it writes no notification_log row (like the care.* lane, §3) |
References
Code links are pinned to commit 8329d7b on main (2026-08-19); the file is the anchor if lines drift on later commits.
consent/migrations/0002_create_consent_records.sql- consent_records DDL; UNIQUE (party, type) L9consent/migrations/0003_create_consent_audit.sql- consent_audit DDL + indexesconsent/internal/handler/consent.go-validConsentTypesL16-25 · member-role gate + #1544 claim override L60-L70 ·recordConsentChangetx (upsert + audit + outbox) L106-L146consent/internal/handler/internal_consent.go-internalServiceGuard(constant-time, fail-closed) L40-55 ·validChangedByTypesL15-19 · SERVICE-attributed writes L80-114consent/internal/handler/deletion.go- SOFT/HARD + MEMBER/EMPLOYER checks L35-42 · one-active-request 409 L44-53 · insert + outbox tx L63-74consent/internal/repository/gorm_consent.go#L25-UpsertWithAudit: ON CONFLICT upsert + audit insert, one txconsent/internal/job/erasure.go#L81-processRequest: PROCESSING → execute → COMPLETED / bounded-retry FAILEDdocument-service/migrations/0003_create_documents.sql- documents DDL L3-11 · the event_type-only outbox L15-23document-service/migrations/0005_document_locators.sql- policy/scheme locators + DESC indexes, rationale in commentdocument-service/migrations/0006_invoice_documents.sql- invoice_locator + partial UNIQUE; INVOICE template seed L36-120document-service/internal/kafka/consumer.go#L30-eventToTemplateType; dual-spelling envelope +resolveEventTypeL118-L155document-service/internal/service/locator.go#L20- the #1428 Sscanf bug + MAX(locator) reseed;nextDocumentLocatorL55-58document-service/internal/service/document.go#L79-Generate: invoice idempotency L83-97 · PDF-or-refuse L120-130 · doc + outbox tx L167-172 · wkhtmltopdf L193-200document-service/migrations/0007_invoice_locator_null_not_empty.sql- the ''-vs-NULL poisoned-index repairdocument-service/internal/handler/handler.go#L90-POST /internal/documents/generate(EMPLOYER_CONTRACT path) · tokenless/documents/{locator}/contentnotifications/migrations/0003_create_notification_log.sql- notification_log DDLnotifications/migrations/0002_create_notification_preferences.sql- preferences DDL, UNIQUE party_locator L5notifications/migrations/0007_idx_log_event_id.sql#L2- the UNIQUE event_id dedup indexnotifications/migrations/0009_add_party_locator_to_log.sql#L2- party_locator retrofit,DEFAULT ''notifications/internal/repository/gorm_log.go#L22-InsertIdempotent:ON CONFLICT (event_id) DO NOTHING, inserted-or-not returnnotifications/internal/projection/consumer.go#L48- deferred checkpoint save L48-53 · envelope top-levelpartyLocatordecode L55-61 · reader + offset commit L145-191notifications/internal/dispatch/dispatcher.go#L321-Dispatchpipeline (party-not-found L326-328, opt-out L339-342, channel guard L344-353) · duplicate skip L424-430 ·SendDirectNovu-first + no-log-write scope L173-237 · Novu-sole-caller correction L239-255 ·NTF-locator mint L153-155notifications/internal/projection/consumer.go#L32-careCategoryFor: care events → Novu in-app lane, never the email dispatchernotifications/internal/client/policyadmin.go#L34-GetPartyContact: 404 → (nil, nil) → SKIPPED "party not found"enrollment/internal/kafka/producer.go#L61- envelope definespartyLocator,omitempty;PublishWithKeynever sets it L112-L118document-service/internal/outbox/producer.go#L64- fixed-topic writer (document-events), key-less publishdocument-service/internal/outbox/worker.go#L22- backoff + log-throttle, the spin-loop-outage hardeningdocument-service/migrations/0002_create_templates.sql- templates DDL, UNIQUE (market, type) L10notifications/internal/handler/handler.go#L87- routes:POST /send, preferences, log list/detailconsent/internal/kafka/producer.go#L54- direct (non-outbox)deletion.requested/deletion.completed/deletion.failedtopicsconsent/migrations/0005_create_outbox.sql- standard topic/key outbox, created schema-unqualified (lands inpublic); rationale comment L3-8consent/migrations/0004_create_deletion_requests.sql- deletion_requests DDLconsent/internal/repository/gorm_deletion.go#L70-SoftDeleteParty(reason-only redaction) /HardDeleteParty(consent-local delete)
Live-schema facts (constraint lists, Debezium publications, consent-type and status distributions, the SKIPPED/SENT counts, template inventory, topic list, deployed KAFKA_TOPICS) come from psql -h 10.0.1.2 -d consent · \d consent.consent_records, \d consent.consent_audit, \d consent.deletion_requests, \d public.outbox · psql -h 10.0.1.2 -d documents · \d documents.documents, \d documents.templates, \d documents.outbox · psql -h 10.0.1.2 -d notifications · \d notifications.notification_log, \d notifications.notification_preferences, \d notifications.projection_checkpoints, 2026-08-18.