Architecture

Integration Patterns — Cross-Cutting Synthesis

Last updated: 2026-02-01 | Architecture

Integration Patterns — Cross-Cutting Synthesis

Key Takeaways

  1. All inter-service communication follows 3 patterns: GraphQL (synchronous queries/mutations), REST (webhooks + file uploads), and RabbitMQ (async events). No shared database backdoors found across 7 domains + infrastructure (H6 validated L1).
  2. RabbitMQ is the nervous system — 70+ message types discovered across 7 domains, all using core-lib MessageSender/MessageHandler. Contracts are discoverable from code (H12 validated L1). Every domain publishes and consumes messages.
  3. Keycloak is the universal dependency — All 28+ production services validate JWT tokens against the same Keycloak issuer URI. Any Keycloak migration affects everything simultaneously.
  4. External APIs are concentrated in 5 services — Stripe (payments), Mux (video), Twilio (SMS), Mandrill (email), Zoom (webinar). All other services have zero external dependencies beyond Keycloak and infrastructure (PostgreSQL, RabbitMQ, Redis).
  5. Frontend calls 24+ GraphQL gateways — Both peeq-mono and frontends talk to the same backend services via Istio path-based routing (/api/{service}). ~17% of API calls target non-existent prod services (dead code).

Migration Decision Question

What are the system-wide integration patterns and data model constraints?

System-Wide Service Dependency Map

graph TD
    subgraph "Frontend Layer"
        PM[peeq-mono<br/>Fan App]
        FE[frontends<br/>Admin + Expert]
    end

    subgraph "Identity"
        KC[Keycloak 26.3<br/>OAuth2/OIDC]
        USR[Users]
        CEL[Celebrity]
        FAN[Fan]
    end

    subgraph "Content & Media"
        CON[Content]
        MED[Media]
        WEB[Webinar]
    end

    subgraph "Events & Commerce"
        SHO[Shoutout]
        SHO_BPM[Shoutout BPM]
        INV[Inventory]
        CC[Class-Catalog]
        OSE[Onsite-Event]
    end

    subgraph "Payment"
        STR[Stripe Service]
        SUB[Subscriptions]
        BPM[Purchase-Request BPM]
        WAL[Wallet]
        TXN[Transaction]
    end

    subgraph "Communication"
        EMAIL[Email]
        SMS[SMS]
        NOTIF[Notifications]
        CHAT[Chat]
        MB[Message-Board]
        SSE[SSE]
    end

    subgraph "Supporting"
        TAGS[Tags]
        SEARCH[Search]
        TRACK[Tracking]
        REPORT[Reporting]
        ORG[Org-Manager]
        GP[Group-Profile]
        JRN[Journey]
    end

    subgraph "External APIs"
        STRIPE_API[Stripe API]
        MUX_API[Mux API]
        TWILIO_API[Twilio API]
        MANDRILL_API[Mandrill API]
        ZOOM_API[Zoom API]
        STREAM_API[Stream Chat API]
        GCS_API[Google Cloud Storage]
    end

    subgraph "Infrastructure"
        PG[(PostgreSQL<br/>35 DBs)]
        RMQ[RabbitMQ<br/>3-node cluster]
        REDIS[(Redis)]
        NFS[(NFS Storage)]
    end

    %% Frontend → Backend
    PM -->|GraphQL| USR & CEL & FAN & CON & MED & WEB & SHO & INV & CC & STR & SUB & WAL & TXN & NOTIF & CHAT & MB & SSE & TAGS & JRN & ORG & GP
    FE -->|GraphQL| USR & CEL & FAN & CON & MED & WEB & SHO & INV & CC & STR & SUB & WAL & TXN & NOTIF & CHAT & MB & SSE & TAGS & EMAIL & SMS & JRN & ORG & GP

    %% External APIs
    STR -->|REST| STRIPE_API
    CON & MED -->|REST| MUX_API
    SMS -->|SDK| TWILIO_API
    EMAIL -->|SDK| MANDRILL_API
    WEB -->|REST| ZOOM_API
    CHAT -->|SDK| STREAM_API
    MED -->|REST| GCS_API

    %% All services → Keycloak
    USR & CEL & FAN & CON & MED & WEB & SHO & INV & CC & STR & SUB & WAL & TXN & EMAIL & SMS & NOTIF & CHAT & MB & SSE & TAGS & SEARCH & TRACK & REPORT & ORG & GP & JRN & OSE -->|JWT| KC

Synchronous Communication (GraphQL + REST)

Service-to-Service GraphQL Calls

From To Purpose Domain
Celebrity Tags Tag lookup for search Identity → Supporting
Celebrity Inventory Event details for share pages Identity → Events
Fan Keycloak User info retrieval Identity
Content Tags Content categorization Content → Supporting
Media Tags Media categorization Content → Supporting
Webinar Celebrity Host profile lookup Content → Identity
Webinar Inventory Event listing integration Content → Events
Stripe Inventory Fetch membership discounts Payment → Events
Stripe Org Manager User max membership discount Payment → Supporting
Subscriptions Inventory Search inventory items Payment → Events
Transaction Inventory Validate purchase IDs Payment → Events
Shoutout Celebrity Celebrity profile for fulfillment Events → Identity
Shoutout Inventory Product details Events → Events

REST Endpoints (External Webhooks)

Service Endpoint External Caller Purpose
Stripe POST /api/stripe/callback Stripe API Payment event webhooks (7 event types)
Content POST /api/content/mux/webhook Mux API Video processing callbacks
Media POST /api/media/mux/webhook Mux API Video asset lifecycle callbacks
Webinar POST /api/webinar/zoom/webhook Zoom API Meeting event callbacks

Cross-Domain Dependency Matrix

                    Identity  Content  Events  Payment  Comms  Support  Infra
Identity              ●                  →                       →
Content               →         ●        →                       →
Events                →         →        ●       ←       ←       →
Payment               →                  →       ●       ←       →
Communication         →                  →       →       ●
Supporting            ←                  ←       ←               ●
Infrastructure        ←         ←        ←       ←       ←       ←        ●

● = internal   → = calls   ← = called by

Key coupling hotspot: Inventory service is called by 5 different domains (Stripe, Subscriptions, Transaction, Shoutout, Webinar). It’s the cross-cutting product catalog hub.

Asynchronous Communication (RabbitMQ)

Complete Message Contract Inventory

Compiled from all 7 analyzed domains. All follow core-lib MessageSender/MessageHandler pattern.

Identity Domain (Sessions 2)

Publisher Message Exchange Consumers
Celebrity CelebrityProfileCreated celebrity Notifications?
Celebrity CelebrityProfileUpdated celebrity Notifications?
Celebrity CelebrityProfileSoftDeleteFlagToggled celebrity
Celebrity CreateSseGroup sse SSE
Celebrity AddRoleToUser users Users
Celebrity GenerateEncryptionKeys encryption
Celebrity CreateDwollaAccount dwolla Dead (H2)
Celebrity UpdateUserPii users Users
Fan FanProfileCreated fans Notifications?
Fan CelebrityFollowed fans SSE, Notifications?
Fan CelebrityUnfollowed fans
Fan AddUserToSseGroup sse SSE
Users UserPiiUpdated users Celebrity, Fan
Users RoleCreated/Deleted users
Users GroupCreated/Deleted users
Keycloak KeycloakEvent (register) keycloak.events Fan

Content & Streaming Domain (Session 3)

Publisher Message Exchange Consumers
Content ContentCreated content Notifications?
Content ContentUpdated content
Content ContentDeleted content
Media MediaCreated media Notifications?
Media MediaUpdated media
Webinar WebinarCreated webinar Notifications, SSE
Webinar WebinarStarted webinar Notifications, SSE
Webinar WebinarEnded webinar Notifications
Webinar RegistrationCreated webinar Notifications, Email?

Payment Domain (Sessions 4-5)

Publisher Message Exchange Consumers
Subscriptions SubscriptionCreated subscriptions Stripe, Inventory
Subscriptions SubscriptionUpdated subscriptions Stripe, Inventory
Subscriptions AddInventoryItem inventory Inventory
Subscriptions UpdateInventoryItem inventory Inventory
Subscriptions SeriesCreated subscriptions Notifications?
Stripe StripeProductCreated stripe Subscriptions, Inventory
Stripe PaymentSucceeded stripe BPM?
Stripe PaymentFailed stripe BPM?
Stripe SubscriptionCanceled stripe Subscriptions?
BPM UpdatePurchaseState purchase-request-bpm Inventory
BPM CreateTransaction purchase-request-bpm Wallet
BPM CreateUserEntitlements purchase-request-bpm Inventory?
BPM SendEmail purchase-request-bpm Email
Wallet TransactionCreated wallet BPM
Wallet SendSseMessageToUser wallet SSE
Transaction TransactionPaidOut transaction

Events Domain (Session 6)

Publisher Message Exchange Consumers
Shoutout ShoutoutCreated shoutout SSE, Notifications
Shoutout ShoutoutUpdated shoutout SSE
Shoutout ShoutoutCompleted shoutout SSE, Email
Shoutout-BPM ShoutoutFulfillmentStateChanged shoutout-bpm Shoutout, SSE
Shoutout-BPM SendEmail shoutout-bpm Email
Inventory InventoryItemCreated inventory Stripe (product sync)
Inventory InventoryItemUpdated inventory Stripe
Inventory InventoryItemPurchaseRequested inventory BPM
Class-Catalog JourneyCreated class-catalog Stripe
Class-Catalog SectionCreated class-catalog Stripe
Class-Catalog OfficeHourTimeSlotCreated class-catalog Stripe

Communication Domain (Session 7)

Publisher Message Exchange Consumers
Notifications SendEmail notifications Email
Notifications SendSms notifications SMS
Notifications SendSseMessageToUser notifications SSE
Notifications SendSseMessageToGroup notifications SSE
Notifications DigestReady notifications Email
Notifications SubscriptionNotification notifications
Email EmailSent email
Email EmailFailed email
Email EmailBounced email
SMS SmsSent sms
SMS SmsFailed sms
SMS SmsDelivered sms
SSE (inbound only — 8 handlers) sse
Chat ChannelCreated chat
Chat ChannelMemberAdded chat
Chat ChannelMemberRemoved chat
Message-Board BoardCreated message-board
Message-Board PostCreated message-board SSE, Notifications
Message-Board DonationReceived message-board SSE
Message-Board LeaderboardUpdated message-board SSE

Message Flow Patterns

graph LR
    subgraph "Product Lifecycle"
        SUB[Subscriptions] -->|SubscriptionCreated| STR[Stripe]
        CC[Class-Catalog] -->|JourneyCreated| STR
        INV[Inventory] -->|ItemCreated| STR
        STR -->|ProductCreated| SUB & INV
    end

    subgraph "Purchase Flow"
        INV -->|PurchaseRequested| BPM[Purchase BPM]
        BPM -->|CreateTransaction| WAL[Wallet]
        WAL -->|TransactionCreated| BPM
        BPM -->|UpdatePurchaseState| INV
        BPM -->|SendEmail| EMAIL[Email]
    end

    subgraph "Notification Pipeline"
        ANY[Any Service] -->|Event| NOTIF[Notifications]
        NOTIF -->|SendEmail| EMAIL
        NOTIF -->|SendSms| SMS
        NOTIF -->|SendSse| SSE[SSE]
    end

    subgraph "Real-time Updates"
        CEL[Celebrity] -->|CreateSseGroup| SSE
        FAN[Fan] -->|AddToSseGroup| SSE
        WAL -->|SendSse| SSE
        MB[Message-Board] -->|PostCreated| SSE
    end

RabbitMQ Statistics

Metric Count
Total message types discovered ~75
Publishing services 20+
Consuming services 15+
Services with both pub + sub 12
Dead message types (no consumer in prod) 3+ (Dwolla, encryption)
Exchange naming convention Service name (e.g., celebrity, stripe, sse)

External API Inventory

Active External Dependencies

External API Internal Service Purpose API Keys Required Session
Stripe stripe Payment processing Payment key, webhook signing secret 4
Mux content, media Video upload, transcoding, playback Token ID, secret 3
Twilio Keycloak (Magic Link SPI), sms SMS delivery Account SID, auth token 2, 7
Mandrill email Transactional email delivery API key 7
Zoom webinar Meeting creation, registration API key, secret 3
Stream Chat chat Real-time messaging API key, secret 7
Google Cloud Storage media, celebrity Blob storage, signed URLs Workload Identity 3, 2
MailChimp fan Email list management API key 2
Google Calendar webinar Event sync Service account 3
Intercom users Customer messaging API token 2
Facebook celebrity Open Graph social sharing App ID 2

Inactive External Dependencies (Retire)

External API Internal Service Status Session
Dwolla peeq-dwolla Inactive since Jan 2023 (H2 L2) 5
Plaid peeq-dwolla Inactive (tied to Dwolla) 5
Jitsi peeq-jitsi-meet Never integrated with Gen 2 3, 6
Arlo LMS class-catalog Deprecated references in code 6
Tixr peeq-custom-tixr Inactive Gen 1 6
Phenix RTS frontend only SDK in frontend, no backend 1
100ms frontend only SDK in frontend, no backend 1

API Key Migration Impact

For any infrastructure migration, these API keys must be migrated: - GCP Secret Manager: Already centralized (20+ secret modules in Terraform) - Stripe webhook URL: Must be updated in Stripe dashboard if service URL changes - Mux webhook URLs: Must be updated in Mux dashboard (2 services receive webhooks) - Zoom webhook URL: Must be updated in Zoom Marketplace app - All keys per tenant: 4 production tenants × 11 active integrations = ~44 key sets

Database-per-Service Architecture

Complete Database Inventory

35 PostgreSQL databases per tenant, all accessed via PgBouncer:

Database Service Migrations Tables Domain Session
celebrity Celebrity 22 6 Identity 2
fan Fan 11 5 Identity 2
identityx_26 Keycloak 26 Identity 2
content Content 12 7 Content 3
media Media 26 13 Content 3
stream Webinar 24 24 Content 3
stripe Stripe 13 7+ Payment 4
subscriptions Subscriptions 14 4 Payment 4
purchase_request_bpm Purchase BPM 6 CIB Seven engine Payment 4
wallet Wallet 3 3 Payment 5
transaction Transaction 6 1 Payment 5
shoutout Shoutout 24 11+ Events 6
shoutout_bpm Shoutout BPM CIB Seven engine Events 6
inventory Inventory 36 12+ Events 6
class-catalog Class-Catalog 32 15+ Events 6
email Email 14 5 Communication 7
sms SMS 7 3 Communication 7
notification_service Notifications 12 6 Communication 7
chat Chat 3 2 Communication 7
message_board Message-Board 5 4 Communication 7
sse SSE 3 2 Communication 7
journey Journey Supporting
tags Tags Supporting
search Search Supporting
tracking Tracking Supporting
reporting Reporting Supporting
org_manager Org-Manager Supporting
group_profile Group-Profile Supporting
webinar Webinar (alt) Content
superset Superset Analytics

Note: Databases marked “—” for migrations/tables were not deeply analyzed (supporting domain, deferred to Session 10+).

Schema Complexity Ranking

Service Migrations Estimated Tables Complexity
Inventory 36 12+ Highest
Class-Catalog 32 15+ Very High
Media 26 13 High
Webinar 24 24 High
Shoutout 24 11+ High
Celebrity 22 6 Medium
Email 14 5 Medium
Subscriptions 14 4 Medium
Stripe 13 7+ Medium
Content 12 7 Medium
Notifications 12 6 Medium
Fan 11 5 Low
SMS 7 3 Low
Transaction 6 1 Low
Message-Board 5 4 Low
Wallet 3 3 Low
Chat 3 2 Low
SSE 3 2 Low

Cross-Database References

No foreign key constraints exist between databases (correct for microservices). Cross-references use:

Reference Type Example Pattern
Keycloak UUID celebrity_profiles.celebrity_user_id All user IDs are Keycloak UUIDs
Service UUID checkout_purchase_products.inventory_id UUID referencing another service’s entity
External ID media.mux_asset_id String ID from external API
String ID fan_follows.id = MD5(fan+celeb) Computed composite key

Shared Database (Exception)

Email, SMS, and Notifications share peeq-notification-service-db. This is the only shared database discovered across 7 domains. It’s an intentional design — notifications orchestrate email and SMS delivery.

Delta Check: Early Assumptions vs Later Findings

Per the plan, re-validating Sessions 1-3 assumptions against Sessions 4-8 findings.

Session 1 (Frontend) — Validated

Assumption Status Evidence
24+ GraphQL gateways Confirmed All 7 backend domain sessions confirmed services matching frontend gateway list
~17% dead API calls Confirmed Broadcast (H1 L2), Dwolla (H2 L2), Conference, Stream, Logging — all confirmed inactive
CSS framework is main blocker Unchanged No backend evidence contradicts this

Session 2 (Identity) — Validated with additions

Assumption Status Evidence
Keycloak is universal Confirmed All 28+ services use Keycloak JWT. Infrastructure confirms same issuer across all tenants
Magic Link SPI is critical Unchanged No alternative auth found in any domain
CIB Seven plugin EOL risk Heightened Sessions 4, 6 confirmed CIB Seven usage in purchase-request-bpm and shoutout-bpm

Session 3 (Content) — Validated

Assumption Status Evidence
Broadcast inactive Confirmed L2 ArgoCD, Helm charts, and infrastructure all confirm
NFS storage coupling Confirmed Infrastructure session shows 4×50Gi NFS PVCs per tenant
Mux is video backbone Confirmed No other video backend found in later sessions

Hypothesis Status Summary (Post-Session 9)

# Hypothesis Assurance Key Evidence
H1 Broadcast not in production L2 Verified ArgoCD + Helm + code + infrastructure
H2 Dwolla inactive L2 Verified ArgoCD + code + wallet analysis
H3 Gen 1 replaced by Gen 2 L1 Validated 7 domains + infrastructure confirmed
H4 Frontend unification feasible L1 Validated Same Angular 18, same gateways, CSS blocker only
H5 >50% repos archivable L1 Validated ~110 repos estimated archivable
H6 No shared DB backdoors L1 Validated 7 domains audited, only intentional shared DB found
H7 >60% test coverage L0 Falsified/Partial Coverage <60% everywhere; API patterns L1 consistent
H8 Data volumes manageable L0 Partial DB tier known (2 vCPU, 6.5 GB), need actual row counts
H9 No compliance blockers L0 Stripe SAQ-A likely, need dashboard confirmation
H10 APIs backward-compatible L0 Frontend dead code confirmed, schema stability unknown
H11 Multi-brand is config-only L2 Verified Same images/charts across 4 tenants
H12 RabbitMQ contracts discoverable L1 Validated 75+ message types cataloged across all domains
H13 core-lib is stable foundation L1 Validated 18+ services, versions 0.0.67-0.0.69
H14 Gen 3 rewrite justified L0 Meta-hypothesis — evaluated in Session 11

Key Integration Risks for Migration

1. Keycloak Migration (Affects Everything)

Any change to Keycloak (URL, realm, JWT claims) requires coordinated update to all 28+ services. This is the single highest-risk integration point.

2. Inventory Service (Cross-Domain Hub)

Inventory is called by 5 domains. Any schema or API changes to inventory ripple across payment, events, and content. Must be migrated carefully or kept stable during other migrations.

3. BPM Engine (In-Flight State)

Two BPM engines (purchase-request-bpm, shoutout-bpm) have running process instances. Migration requires draining active instances before engine replacement.

4. External Webhook URLs (4 Services)

Stripe, Mux (×2), and Zoom webhook URLs must be updated if service endpoints change during migration. Webhook delivery failures during migration = lost events.

5. Notification Pipeline (Fan-Out)

Notifications → Email/SMS/SSE pipeline is the primary fan-out mechanism. Breaking this pipeline affects all user-facing notifications across the platform.


Last updated: 2026-01-30 — Session 9 Review by: 2026-04-30 Staleness risk: Medium — integration patterns are stable but message contracts may evolve