Target Architecture — Gen 3 Proposal

Key Takeaways

1. H14 evaluation: Incremental upgrade, not full rewrite — Evidence from 10 sessions supports upgrading Gen 2 in place rather than building Gen 3 from scratch. Gen 2 services are already Java 21/Spring Boot 3.5.4, use consistent patterns (core-lib, GraphQL, RabbitMQ), and have clean service boundaries. The debt is in infrastructure, testing, and BPM — not application architecture.
2. Consolidate from ~35 application services to ~18 — Group by domain-driven boundaries. Merge over-decomposed services (wallet+transaction, email+sms+notifications, shoutout+shoutout-bpm). Keep services that serve as infrastructure (SSE, Keycloak).
3. Unified frontend monorepo — Merge peeq-mono and frontends into a single Angular 18/Nx monorepo with shared component library. Standardize on Tailwind CSS. Component-by-component migration, not big-bang.
4. Shared cluster with namespace isolation — Replace cluster-per-tenant with namespace-per-tenant in a single regional GKE cluster. NetworkPolicies + Istio AuthorizationPolicy for isolation. H11 L2 confirms no code-level branching.
5. Replace Camunda 7.17.0 with Operaton — Adopt Operaton (community-owned Camunda 7 fork) as strategic BPM platform. Near-zero migration preserves existing BPMN files and DB schema. BPM is a platform investment for future process-based capabilities, not just a legacy dependency (see ADR-013).

Migration Decision Question

What does Gen 3 look like?

H14 Evaluation: Gen 3 Rewrite vs Gen 2 Upgrade

Hypothesis

H14: A Gen 3 rewrite is justified over upgrading Gen 2 in place.

Evidence Assessment

Verdict

H14: FALSIFIED — The evidence does not support a full Gen 3 rewrite.

Recommendation: Incremental upgrade with targeted consolidation.

The platform’s application layer is architecturally sound: modern tech stack, consistent patterns, clean boundaries. The debt is concentrated in infrastructure (zonal HA, observability, security enforcement), BPM engine replacement, test coverage, and frontend unification — none of which require rewriting business logic.

A full rewrite would: - Duplicate ~280 Flyway migrations worth of schema knowledge - Re-implement ~75 RabbitMQ message contracts - Re-integrate 11 external APIs - Re-create 24+ GraphQL gateway endpoints - Introduce regression risk with zero test coverage as starting point

An incremental upgrade preserves all existing integration work and focuses effort on actual problems.

WLNK Analysis

Overall confidence: L1 (Validated) Weakest link: H8 (data volumes L0) — if production data volumes are unexpectedly large, some services might need more fundamental data model changes. However, this affects migration strategy, not the rewrite-vs-upgrade decision.

Bounded Validity

• Scope: Applies to all application services and infrastructure
• Expiry: Re-evaluate if Gen 2 tech stack falls behind (e.g., Spring Boot 3.x EOL, Java 21 EOL)
• Review trigger: If >50% of services require schema redesign during upgrade, reconsider targeted rewrite for those domains

Target Service Architecture

Service Consolidation Map

graph TD
    subgraph "Gen 2 Current (~35 services)"
        direction TB
        C1[celebrity]
        C2[fan]
        C3[users]
        C4[content]
        C5[media]
        C6[webinar]
        C7[stripe]
        C8[subscriptions]
        C9[purchase-request-bpm]
        C10[wallet]
        C11[transaction]
        C12[shoutout]
        C13[shoutout-bpm]
        C14[inventory]
        C15[class-catalog]
        C16[onsite-event]
        C17[email]
        C18[sms]
        C19[notifications]
        C20[chat]
        C21[message-board]
        C22[sse]
        C23[search]
        C24[tags]
        C25[tracking]
        C26[group-profile]
        C27[org-manager]
        C28[journey]
    end

    subgraph "Target (~18 services)"
        direction TB
        T1[identity-service<br/>celebrity + fan + users]
        T2[content-service<br/>content + media]
        T3[webinar-service<br/>webinar]
        T4[payment-service<br/>stripe + subscriptions + wallet + transaction]
        T5[purchase-workflow<br/>purchase-request state machine]
        T6[shoutout-service<br/>shoutout + shoutout-bpm merged]
        T7[inventory-service<br/>inventory]
        T8[class-catalog-service<br/>class-catalog + journey]
        T9[event-service<br/>onsite-event]
        T10[notification-service<br/>email + sms + notifications]
        T11[chat-service<br/>chat]
        T12[message-board-service<br/>message-board]
        T13[sse-service<br/>sse]
        T14[search-service<br/>search]
        T15[platform-services<br/>tags + tracking + group-profile + org-manager]
        T16[keycloak<br/>identity provider]
        T17[admin-frontend<br/>unified Angular monorepo]
        T18[fan-frontend<br/>unified Angular monorepo]
    end

    C1 --> T1
    C2 --> T1
    C3 --> T1
    C4 --> T2
    C5 --> T2
    C6 --> T3
    C7 --> T4
    C8 --> T4
    C10 --> T4
    C11 --> T4
    C9 --> T5
    C12 --> T6
    C13 --> T6
    C14 --> T7
    C15 --> T8
    C28 --> T8
    C16 --> T9
    C17 --> T10
    C18 --> T10
    C19 --> T10
    C20 --> T11
    C21 --> T12
    C22 --> T13
    C23 --> T14
    C24 --> T15
    C25 --> T15
    C26 --> T15
    C27 --> T15

Consolidation Rationale

Services Kept Separate

Target Tech Stack

Application Layer

Frontend Layer

Infrastructure Layer

Observability Layer

Database Strategy

Consolidation with Service Merges

When services merge, their databases merge:

Database Count Reduction

Migration Approach

• No schema rewrite — merged databases use schema prefixes or separate schemas within the same instance
• Flyway continues — each service module maintains its own migration folder
• Foreign keys within merged service only — no cross-service FK even within the same database instance

Frontend Unification Strategy

Phase 1: Shared Component Library

1. Create unified @vzlabs/components library in a new Nx workspace
2. Port high-usage components from both repos (start with auth, navigation, layout)
3. Both repos import from shared library (dual-publish during transition)

Phase 2: CSS Standardization

1. Choose Tailwind as target (more flexible, better DX, already in peeq-mono)
2. Create Tailwind equivalents for Bootstrap/Angular Material components in frontends
3. Component-by-component migration (not big-bang)

Phase 3: Repo Merge

1. Move frontends apps (admin-fe, celeb-fe, org-dashboard-fe) into peeq-mono Nx workspace
2. Remove duplicate services and gateways
3. Single CI/CD pipeline for all frontend apps

Phase 4: Dead Code Removal

1. Remove 5 dead API gateways (broadcast, conference, stream, dwolla, logging)
2. Remove unused components and services
3. Target: reduce frontend codebase by ~17%

Multi-Tenant Architecture (Shared Cluster)

Current: Cluster-Per-Tenant

Tenant A: GKE Cluster → 28+ pods → Cloud SQL (35 DBs) → RabbitMQ → Redis → Keycloak
Tenant B: GKE Cluster → 28+ pods → Cloud SQL (35 DBs) → RabbitMQ → Redis → Keycloak
Tenant C: GKE Cluster → 28+ pods → Cloud SQL (35 DBs) → RabbitMQ → Redis → Keycloak
Tenant D: GKE Cluster → 28+ pods → Cloud SQL (35 DBs) → RabbitMQ → Redis → Keycloak

Target: Shared Cluster with Namespace Isolation

Shared Regional GKE Cluster
├── namespace: tenant-a → 18 pods → schemas in shared Cloud SQL → vhost in shared RabbitMQ
├── namespace: tenant-b → 18 pods → schemas in shared Cloud SQL → vhost in shared RabbitMQ
├── namespace: tenant-c → 18 pods → schemas in shared Cloud SQL → vhost in shared RabbitMQ
├── namespace: tenant-d → 18 pods → schemas in shared Cloud SQL → vhost in shared RabbitMQ
├── namespace: platform → Keycloak, Istio, ArgoCD, monitoring
└── NetworkPolicies + Istio AuthorizationPolicy for isolation

Isolation Mechanisms

Prerequisites (Before Consolidation)

• NetworkPolicies implemented and tested
• Istio AuthorizationPolicy configured per namespace
• RabbitMQ vhost per tenant verified
• Cloud SQL regional HA enabled
• PgBouncer reconfigured for schema-based routing
• Terraform modules updated for shared-cluster model
• CastAI policies adjusted for shared cluster

BPM Replacement Strategy

Current: Camunda 7.17.0 CE (EOL October 2025)

Two active workflows: 1. Purchase-Request BPM: Payment → entitlements → wallet debit/credit → notifications → refunds (~10 states) 2. Shoutout BPM: Offer → purchase → celebrity recording → FFmpeg/Mux → admin review → delivery (~12 states)

Target: Operaton (Community-Owned Camunda 7 Fork)

Updated per ADR-013: The original recommendation was Spring State Machine. This was reversed based on two inputs: (1) Meet & Greet SM is completely retired — no in-house SM production experience, and (2) strategic intent to expand BPM as a platform capability for future process-based features (expert onboarding, content approval, event lifecycle, dispute resolution, etc.).

Operaton is the same engine the platform already runs — near-zero migration. Preserves existing BPMN files, database schema, and operational knowledge. Community-owned governance prevents single-vendor EOL risk.

Migration approach: 1. Update Maven dependencies — replace org.camunda.bpm with Operaton equivalents 2. Validate BPMN files work without modification 3. Test database compatibility (7.17 → 7.24 schema may need migration scripts) 4. Upgrade BPM services to Java 21 / Spring Boot 3.x (ADR-003) 5. Run in parallel with Camunda 7 during validation (dual-write) 6. Drain Camunda 7 instances 7. Switch traffic to Operaton 8. Remove Camunda 7 dependencies and Keycloak identity sync plugin

Observability Stack

SLI/SLO Framework

Alerting Rules (Critical)

Security Improvements

Target Architecture Diagram

graph TB
    subgraph "Clients"
        FAN[Fan App<br/>Angular/Ionic]
        ADMIN[Admin/Expert<br/>Angular]
    end

    subgraph "Edge"
        DNS[Cloud DNS]
        LB[GCP Load Balancer<br/>+ Cloud Armor WAF]
        CERT[cert-manager<br/>Let's Encrypt]
    end

    subgraph "Service Mesh"
        ISTIO[Istio IngressGateway<br/>TLS + Path Routing + mTLS]
    end

    subgraph "Identity"
        KC[Keycloak 26.x<br/>Magic Link + SSO]
        ID[identity-service<br/>profiles + follows]
    end

    subgraph "Content"
        CONT[content-service<br/>articles + video + media]
        WEB[webinar-service<br/>Zoom integration]
    end

    subgraph "Commerce"
        PAY[payment-service<br/>Stripe + wallet + txn]
        PW[purchase-workflow<br/>state machine]
        INV[inventory-service<br/>product catalog hub]
        SUB_S[shoutout-service<br/>offers + fulfillment]
    end

    subgraph "Learning"
        CLS[class-catalog-service<br/>courses + journeys]
        EVT[event-service<br/>onsite check-in]
    end

    subgraph "Communication"
        NOT[notification-service<br/>email + SMS + push]
        CHAT[chat-service<br/>Stream Chat]
        MB[message-board-service]
        SSE_S[sse-service<br/>real-time events]
    end

    subgraph "Platform"
        PLT[platform-services<br/>tags + tracking + org]
        SRCH[search-service<br/>Elasticsearch]
    end

    subgraph "Infrastructure"
        PG[(Cloud SQL<br/>Regional HA)]
        RMQ[RabbitMQ<br/>Shared + vhosts]
        REDIS[Redis]
        GCS[Cloud Storage]
    end

    subgraph "External APIs"
        STRIPE_API[Stripe]
        MUX_API[Mux]
        ZOOM_API[Zoom]
        STREAM_API[Stream Chat]
        TWILIO_API[Twilio]
        MANDRILL_API[Mandrill]
    end

    subgraph "Observability"
        PROM[Prometheus + Grafana]
        OTEL[OpenTelemetry]
        SENTRY[Error Tracking]
        LOG[Cloud Logging]
    end

    FAN --> DNS --> LB --> ISTIO
    ADMIN --> DNS
    ISTIO --> KC
    ISTIO --> ID
    ISTIO --> CONT
    ISTIO --> WEB
    ISTIO --> PAY
    ISTIO --> INV
    ISTIO --> CLS
    ISTIO --> NOT
    ISTIO --> CHAT
    ISTIO --> SSE_S

    PAY --> STRIPE_API
    CONT --> MUX_API
    WEB --> ZOOM_API
    CHAT --> STREAM_API
    NOT --> TWILIO_API
    NOT --> MANDRILL_API

    ID --> PG
    CONT --> PG
    CONT --> GCS
    PAY --> PG
    INV --> PG
    CLS --> PG
    NOT --> PG
    SSE_S --> REDIS
    PW --> RMQ
    NOT --> RMQ
end

Hypotheses Final Status (Session 11)

Last updated: 2026-01-30 — Session 11 Review by: 2026-04-30 Staleness risk: Medium — target architecture evolves with implementation decisions