Infrastructure & DevOps Architecture

Key Takeaways

1. Fully GCP-native infrastructure — GKE clusters, Cloud SQL (PostgreSQL 16), Cloud DNS, GCS, Secret Manager, Artifact Registry. All provisioned via Terraform with Atlantis PR-based workflow.
2. Dedicated cluster per tenant — 3 production brands (The Agile Network, NIL Game Plan, VT NIL) each get a separate GKE cluster, PostgreSQL instance (35 databases), RabbitMQ cluster (3 nodes), Redis, and Keycloak realm. A 4th brand (Speed of AI) also has a production cluster.
3. Mature GitOps pipeline — ArgoCD manages all deployments. 28 reusable GitHub Actions workflows. Common Helm library chart (v0.0.179, 33 templates) standardizes all 48 service charts. Preview environments auto-created per PR with Istio subdomain routing.
4. Multi-brand routing via Istio — Istio IngressGateway handles TLS termination, path-based routing (/api/{service}), mTLS between services, and per-tenant Gateway hosts. External-DNS and cert-manager automate DNS records and Let’s Encrypt certificates.
5. No backend brand-specific logic in infrastructure — All tenant differentiation is via environment variables and values-globals.yaml. Same Helm charts, same Docker images, same service code deployed to all tenants. Confirms H11 across all 7 domains + infrastructure.

Migration Decision Question

What infrastructure changes are needed, and what’s the multi-brand routing mechanism?

Migration Verdict

Verdict: Upgrade Complexity: L Key Constraint: Cluster-per-tenant model is expensive but provides isolation; consolidation requires namespace-level isolation strategy (NetworkPolicies, ResourceQuotas, Istio authorization). Dependencies: All application services depend on this infrastructure layer. Changes here affect all domains.

Infrastructure Inventory

Cloud Platform

GCP Projects

Production Tenants

Development Tenants

Multi-Brand Routing Architecture

DNS-to-Backend Flow

graph TD
    subgraph "Internet"
        U1[User: theagilenetwork.com]
        U2[User: nilgameplan.com]
        U3[User: vt.triumphnil.com]
    end

    subgraph "GCP Cloud DNS"
        DNS1[theagilenetwork.com → LB IP]
        DNS2[nilgameplan.com → LB IP]
        DNS3[vt.triumphnil.com → LB IP]
    end

    subgraph "GCP Load Balancer"
        LB1[agilenetwork cluster LB]
        LB2[nilgameplan cluster LB]
        LB3[vtnil cluster LB]
    end

    subgraph "Istio per Cluster"
        IG1[Istio IngressGateway]
        IG2[Istio IngressGateway]
        IG3[Istio IngressGateway]
    end

    subgraph "Kubernetes Namespaces"
        NS1[agilenetwork namespace<br/>28 services + infra]
        NS2[nilgameplan namespace<br/>28 services + infra]
        NS3[vt namespace<br/>28 services + infra]
    end

    U1 --> DNS1 --> LB1 --> IG1 --> NS1
    U2 --> DNS2 --> LB2 --> IG2 --> NS2
    U3 --> DNS3 --> LB3 --> IG3 --> NS3

Istio Gateway Configuration

Each tenant has an Istio Gateway accepting traffic for its domains:

# Simplified from prod/agilenetwork/istio-gateway
servers:
  - hosts: ['*.theagilenetwork.com', 'theagilenetwork.com']
    port: { number: 443, protocol: HTTPS }
    tls: { mode: SIMPLE, minProtocolVersion: TLSV1_2 }
  - hosts: ['*.theagilenetwork.com', 'theagilenetwork.com']
    port: { number: 80, protocol: HTTP }
    tls: { httpsRedirect: true }

Per-Service Routing (VirtualService)

Every service gets path-based routing through the Istio gateway:

# Pattern for all services
spec:
  hosts: ['theagilenetwork.com']
  gateways: ['istio-system/istio-gateway']
  http:
    - match: [{ uri: { prefix: /api/celebrity } }]
      route: [{ destination: { host: celebrity, port: { number: 8080 } } }]

Frontend Domain Mapping

Tenant Configuration (values-globals.yaml)

Each tenant has a values-globals.yaml with all tenant-specific config:

global:
  cluster: agilenetwork
  domain: theagilenetwork.com
  env: prod
  gcpProject: production-370815
  secretManagerId: "564934788583"
  tenant:
    alias: "The Agile Network"
    name: "agilenetwork"
    namespace: "agilenetwork"
  database:
    hostname: pgbouncer
    port: 5432
  keycloak:
    authServerUrl: "https://identity.theagilenetwork.com"
    realm: agilenetwork
  rabbitmq:
    host: rabbitmq
  redis:
    master: { host: redis-master, port: 6379 }

All services read from global.* — no brand-specific business logic anywhere.

GKE Cluster Configuration

Node Pools (Production)

Cluster Features

• Workload Identity: All GCP API calls use Kubernetes SA → GCP SA binding (no JSON keys)
• Network Policy: Disabled (relying on Istio for service-to-service authorization)
• Release Channel: UNSPECIFIED (manual version control)
• Monitoring: Cloud Monitoring + Cloud Logging enabled
• Filestore CSI: Available for NFS volumes

CastAI Cost Optimization

• Default: Spot instances for all workloads
• On-demand exceptions: admin-fe, celeb-fe, celebrity, class-catalog, content, email, fan, group-profile, identityx, inventory, media, message-board, mono-web, notifications, rabbitmq, shoutout, shoutout-bpm, sms, sse, stripe, subscriptions, tags, transaction, users, wallet
• Node constraints: 2-8 CPU cores, min 30GB disk
• Evictor: Aggressive mode in dev, conservative in prod
• Impact: Most critical services forced to on-demand nodes, reducing cost savings

Database Infrastructure

Cloud SQL Configuration

Databases per Tenant (35 PostgreSQL)

celebrity, chat, class-catalog, content, email, fan, group-profile, identityx_26, inventory, journey, media, message_board, notification_service, org_manager, purchase_request_bpm, reporting, search, shoutout, shoutout_bpm, sms, sse, stream, stripe, subscriptions, tags, tracking, transaction, wallet, webinar, superset

Plus additional MySQL instance for VT NIL (tracking database).

PgBouncer (Connection Pooling)

• Replicas: 3 per tenant
• Databases routed: 41 (all services connect via PgBouncer)
• Resources: 200m-500m CPU, 192Mi memory
• Pattern: All services connect to pgbouncer:5432 not Cloud SQL directly
• Init containers: Every service waits for PgBouncer before starting

Airbyte CDC Replication

20 databases have Change Data Capture replication to Snowflake: - Sources: PostgreSQL Cloud SQL (public IP required) - Destination: Snowflake data warehouse (org: TSGCLBT, account: PM66380) - Per-tenant Snowflake databases (e.g., AGILENETWORK_DB)

CI/CD Pipeline

GitHub Actions (28 Reusable Workflows)

graph TD
    subgraph "Developer Workflow"
        PR[Open PR] --> BUILD
    end

    subgraph "Build Phase"
        BUILD[GitHub Actions] --> MVN[Maven Build + Tests]
        BUILD --> DOCKER[Docker Build<br/>Multi-arch: amd64 + arm64]
    end

    subgraph "Security Phase"
        DOCKER --> TRIVY[Trivy Container Scan<br/>CRITICAL + HIGH]
        DOCKER --> QWIET[Qwiet SAST<br/>Code analysis]
    end

    subgraph "Artifact Phase"
        MVN --> GAR_MVN[Push to GAR Maven]
        DOCKER --> GAR_DOCKER[Push to GAR Docker]
        DOCKER --> HELM[Generate Helm Chart]
        HELM --> GAR_HELM[Push to GAR Helm OCI]
    end

    subgraph "Deploy Phase"
        GAR_DOCKER --> |PR| PREVIEW[Preview Environment<br/>ArgoCD Previews]
        GAR_DOCKER --> |Master| PROD_UPDATE[Update argocd-deployments<br/>image.tag in values.yaml]
        PROD_UPDATE --> ARGOCD[ArgoCD Sync<br/>Helm Release Update]
    end

    subgraph "Preview Lifecycle"
        PREVIEW --> PR_COMMENT[Bot: Preview URL<br/>on PR comment]
        PR_COMMENT --> CLEANUP[PR Closed → Delete<br/>Preview Namespace]
    end

Key Workflow Files

Self-Hosted Runner

• Base: ghcr.io/actions/actions-runner:2.322.0
• Mode: Docker-in-Docker (DinD) on GKE
• Pre-installed: Java 21 (Corretto), Maven 3.9.9, Node 20, Helm 3.17.1, GitHub CLI
• Scaling: ARC (Actions Runner Controller), 0-5 runners

Preview Environment Flow

1. PR opened → GitHub Actions builds Docker image with PR-specific tag
2. Helm chart generated with preview values (namespace: {service}-pr-{number})
3. Application YAML pushed to argocd-previews repo
4. ArgoCD syncs preview application
5. Secrets copied from fanfuze namespace to preview namespace
6. Bot comments preview URL: {service}-pr-{number}.dev.fanfuzenil.com
7. PR closed → Cleanup workflow deletes namespace

Version Management

• Semantic versioning: {major}.{minor}.{patch} from git tags
• PR versions: {version}-pr.{pr_number}.{run_number}.{attempt}
• Chart versions: Independent of application versions (common chart at 0.0.179)

Helm Chart Architecture

Common Library Chart (v0.0.179)

33 reusable templates standardizing all service deployments:

Service Chart Pattern

Each service chart is minimal — delegates to common chart:

# Chart.yaml
dependencies:
  - name: common
    version: 0.0.179
    repository: "oci://us-central1-docker.pkg.dev/favedom-dev/helm"

# templates/deployment.yaml
{{ include "common.deployment.java" . }}

# templates/istio.yaml
{{ include "common.istio" . }}

Feature flags control what each service gets:

# values.yaml (per service)
keycloak: { enabled: true }
postgres: { enabled: true }
rabbitmq: { enabled: true }
prometheus: { enabled: true }
hpa: { enabled: false }  # overridden per tenant

All Helm Charts (48)

Application Services (30): celebrity, fan, users, content, media, shoutout, shoutout-bpm, webinar, chat, message-board, notifications, email, sms, sse, inventory, journey, class-catalog, purchase-request-bpm, transaction, wallet, subscriptions, stripe, search, tags, tracking, reporting, org-manager, group-profile, onsite-event, athlete-manager

Frontends (5): mono-web, admin-fe, celeb-fe, org-dashboard-fe, nilgp-partnerportal-fe

Infrastructure (13): common (library), pgbouncer, rabbitmq-queue-monitor, flyway, shared-secrets, stackhawk, argocd-reports, site-maintenance, node-tracking, nilgp-partnerportal-be, test-spring-boot-app, plus preview configs

Secret Management

Architecture (3-Tier)

graph LR
    subgraph "Source of Truth"
        GSM[GCP Secret Manager<br/>100+ secrets per tenant]
    end

    subgraph "Injection Layer"
        AVP[ArgoCD Vault Plugin<br/>Fetches at sync time]
    end

    subgraph "Runtime"
        K8S[Kubernetes Secrets<br/>Mounted in pods]
    end

    subgraph "Provisioning"
        TF[Terraform<br/>Creates secret shells]
    end

    TF --> GSM
    GSM --> AVP
    AVP --> K8S

Secret Naming Convention

{tenant}_{VENDOR}_{APP}_{SECRET_NAME}

Examples: - agilenetwork_STRIPE_PAYMENT_KEY - agilenetwork_KEYCLOAK_FAN_CLIENTID - agilenetwork_RABBITMQ_PASSWORD

Integrated Services (20+ secret modules)

CI/CD Secrets

• Workload Identity Federation for GCP authentication (no service account keys)
• CI/CD secrets stored with cicd_ prefix in core-services-370815 project
• Fetched at runtime by GitHub Actions, never stored in GitHub

Terraform Infrastructure-as-Code

Module Inventory (19 modules)

State Management

• Backend: GCS bucket terraform-state-370815
• State files: Per-environment (core-services/, development/, production/)
• Locking: GCS-native state locking
• CI/CD: Atlantis (PR-based plan/apply, auto-merge on success)

Environment Setup

Secrets injected via tf_setup.sh:

export TF_VAR_castai_api_token=$(gcloud secrets versions access latest \
  --secret="core_CASTAI_API_TOKEN")

Monitoring & Observability

Current Stack

Observability Gaps

• No centralized alerting visible in infrastructure repos (no PagerDuty, Opsgenie, or Slack alert configs in Terraform/Helm)
• APM disabled by default — transaction sampling at 50% when enabled, but not turned on
• Logging pipeline is Gen 1 — peeq-logging is Node.js/Express (not peeq-* naming, but Gen 1 patterns: Node 9 Dockerfile, Express, no TypeScript build in image)
• No SLO/SLI definitions in Helm charts or monitoring config
• Kibana dashboards not defined in code — likely manual creation
• No distributed tracing adoption — Zipkin address configured in Istio but no evidence of service-level trace instrumentation

Storage Infrastructure

NFS (ReadWriteMany)

4 Persistent Volume Claims per tenant via NFS provisioner:

All use nfs-client StorageClass via per-tenant NFS provisioner.

GCS Buckets

2 shared buckets with tenant-specific directories:

• public-assets-{hex}: Public-facing assets (images, videos). Public read access (allUsers).
• backend-assets-{hex}: Backend-generated assets. Public read access.

Both have versioning enabled and deletion protection.

Operational Tooling

ArgoCD Scripts (24 utilities)

DevOps Utilities (devops-utlities repo)

RabbitMQ Queue Monitor

• Stack: Bash + curl + jq + kubectl on Debian slim
• Monitors: 27 services mapped to Kubernetes deployments
• Purpose: Message-driven autoscaling (precursor to KEDA)
• Maps: Queue names → deployment names for scaling decisions

Security Architecture

Authentication & Authorization

graph TD
    subgraph "External"
        USER[User Browser]
    end

    subgraph "GCP Edge"
        LB[GCP Load Balancer<br/>TLS Termination]
    end

    subgraph "Service Mesh"
        IG[Istio IngressGateway<br/>Host-based routing]
        MTLS[Istio mTLS<br/>Pod-to-pod encryption]
    end

    subgraph "Identity"
        KC[Keycloak 26.3<br/>OAuth2/OIDC]
    end

    subgraph "Services"
        SVC[Spring Boot Service<br/>JWT validation via issuer URI]
    end

    USER -->|HTTPS| LB --> IG
    IG -->|/api/*| SVC
    IG -->|/auth/*| KC
    USER -->|OAuth2 flow| KC
    KC -->|JWT| USER
    USER -->|Bearer token| IG
    SVC -.->|mTLS| MTLS -.-> SVC

Security Controls

Security Gaps

• Trivy scans don’t fail builds — exit code 0, vulnerabilities tracked but not enforced
• No Binary Authorization — unsigned container images can deploy
• No GKE Shielded Nodes — secure boot not enabled
• No NetworkPolicies — relying on Istio alone for pod-level isolation
• Public GCS buckets — allUsers read access on asset buckets
• CORS allows all origins — allowOrigins: regex: .* in Istio VirtualService
• Developer IPs in authorized networks — direct DB access via individual IPs (dev/staging)

Gen 1 Infrastructure Services

Infrastructure Risk Assessment

High Risk

1. Zonal GKE clusters — Single zone (us-central1-a). Zone failure = complete tenant outage. No regional cluster configuration. RPO depends on Cloud SQL backup frequency.
2. Zonal Cloud SQL — ZONAL availability type. No automatic failover. Manual recovery required from point-in-time backup.
3. Single region — All infrastructure in us-central1. No cross-region DR.

Medium Risk

4. Cluster-per-tenant cost — 4 production clusters each with 0-3 n1-standard-8 nodes + Cloud SQL + RabbitMQ + Redis. Infrastructure cost scales linearly with tenants.
5. CastAI on-demand overrides — 25+ services forced to on-demand nodes, reducing spot savings significantly.
6. PgBouncer as SPOF — If PgBouncer pods fail, all database connections fail. 3 replicas provide some redundancy but no circuit breaking.

Low Risk

7. Common chart coupling — All 48 charts depend on common v0.0.179. Breaking change affects everything. Mitigated by chart versioning.
8. ArgoCD centralization — Single ArgoCD instance manages all clusters from core-services project. ArgoCD failure blocks all deployments.
9. Helm OCI registry — All charts stored in single Artifact Registry. Registry outage blocks deployments but not running services.

Inter-Service Communication (Infrastructure View)

Synchronous

All service-to-service calls route through Istio mesh: - Path-based routing: http://{service}:8080/api/{service}/... - mTLS encryption between all pods - CORS configured at Istio level (broad: all origins)

Asynchronous

RabbitMQ 3.13.7 cluster per tenant: - 3 nodes, persistent storage (20Gi) - Memory high watermark: 3276Mi - On-demand nodes (no spot eviction) - Safe-to-evict: false annotation - Definitions loaded from Kubernetes secrets

Database Connections

All services → PgBouncer (3 replicas) → Cloud SQL (private IP): - JDBC URL: jdbc:postgresql://pgbouncer:5432/{service-name} - Credentials: Kubernetes Secret (from GCP Secret Manager via AVP) - Init container waits for PgBouncer availability

Data Model (Infrastructure)

Resource Allocation per Tenant

Scaling Boundaries

Modernization Implications

Consolidation Opportunity: Shared Multi-Tenant Cluster

Current: 4 production GKE clusters (1 per tenant) Option: 1 regional GKE cluster with namespace-per-tenant isolation

Benefits: - ~60-70% infrastructure cost reduction (shared control plane, better node utilization) - Simpler management (1 cluster to upgrade, 1 Istio mesh) - Faster new-tenant onboarding

Requirements: - Kubernetes ResourceQuota per namespace - Istio AuthorizationPolicy for service isolation - NetworkPolicy for pod-level segmentation - Separate Cloud SQL instances per tenant (keep for data isolation)

Risk: Noisy neighbor issues, shared blast radius. Mitigated by resource quotas and Istio rate limiting.

Infrastructure Upgrades Needed

1. Regional GKE — Move from zonal to regional clusters for HA
2. Regional Cloud SQL — Enable automatic failover
3. Security hardening — Binary Authorization, Shielded Nodes, NetworkPolicies, tighter CORS
4. Trivy enforcement — Fail builds on CRITICAL vulnerabilities
5. Observability gaps — Alerting, SLOs, distributed tracing, APM enablement
6. Logging modernization — Replace peeq-logging (Node.js Gen 1) with GCP Cloud Logging or upgrade to Elastic Cloud
7. KEDA adoption — Replace custom rabbitmq-queue-monitor with KEDA ScaledObjects

What Can Stay As-Is

• GitOps pipeline (ArgoCD + GitHub Actions) — mature and well-structured
• Terraform modules — well-factored, 19 reusable modules
• Common Helm chart — 33 templates, battle-tested at v0.0.179
• Secret management (GCP Secret Manager + AVP) — secure pattern
• CastAI — effective cost optimization
• Istio service mesh — provides routing, mTLS, observability

Last updated: 2026-01-30 — Session 8 Review by: 2026-04-30 Staleness risk: Medium — infrastructure configs and versions evolve frequently