ADR-021: Messaging & Event Architecture

Status

Under Review — Kafka vs RabbitMQ strategic analysis in progress

Context

RabbitMQ is the platform’s nervous system — 75+ message types, 20+ publishing services, 15+ consuming services, 92 Java classes implementing messaging. The Knowledge Graph initiative (ADR-019) introduces new requirements for event streaming, replay, and analytics that warrant re-evaluation of the messaging strategy.

Current State

New Requirements (Knowledge Graph Era)

Strategic Analysis: Kafka vs RabbitMQ

Architectural Comparison

┌─────────────────────────────────────────────────────────────────────────────┐
│                           RabbitMQ Architecture                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   Producer ──► Exchange ──► Queue ──► Consumer                              │
│                  │                                                          │
│                  │ (routing)                                                │
│                  ▼                                                          │
│              [Queue 1] ──► Consumer A                                       │
│              [Queue 2] ──► Consumer B (competing consumers)                 │
│                                                                             │
│   • Message deleted after acknowledgment                                    │
│   • No replay capability                                                    │
│   • Push-based delivery                                                     │
│   • Per-message routing flexibility                                         │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│                            Kafka Architecture                                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   Producer ──► Topic (Partition 0) ──► Consumer Group A                     │
│                       │                     │                               │
│               (append-only log)       (offset tracking)                     │
│                       │                     │                               │
│               [Log: 0,1,2,3,4...]    Consumer A reads at offset 3           │
│                       │              Consumer B reads at offset 5           │
│                       ▼                                                     │
│               Retention: 7 days (configurable)                              │
│                                                                             │
│   • Messages retained after consumption                                     │
│   • Full replay from any offset                                             │
│   • Pull-based delivery                                                     │
│   • Partition-based ordering                                                │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Feature Comparison Matrix

Use Case Fit Analysis

Cost Analysis

Migration Effort Breakdown (If Kafka)

Decision Options

Option A: Enhance RabbitMQ (Original v1 Recommendation)

Add reliability patterns to existing RabbitMQ infrastructure: - Idempotency via Redis-backed store - DLQ configuration with 3-retry policy - Circuit breakers for synchronous calls - Lightweight message versioning (headers) - Consolidate to single cluster with vhosts

Pros: - Low risk, incremental improvement - Minimal code changes (core-lib only) - Team already knows RabbitMQ - 2-4 weeks effort

Cons: - No event replay capability - No event sourcing for audit - Limited stream processing options - Per-tenant cluster consolidation still needed

Option B: Hybrid Architecture (RabbitMQ + Kafka)

Keep RabbitMQ for transactional messaging, add Kafka for event streaming:

┌─────────────────────────────────────────────────────────────────────────────┐
│                        Hybrid Architecture                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   ┌─────────────┐                          ┌─────────────┐                  │
│   │  RabbitMQ   │  Task Queues             │   Kafka     │  Event Streams   │
│   │             │  Request/Reply           │             │  Event Sourcing  │
│   │             │  Shoutout Processing     │             │  Audit Trail     │
│   │             │  Payment Workflows       │             │  Analytics       │
│   └──────┬──────┘                          └──────┬──────┘                  │
│          │                                        │                         │
│          ▼                                        ▼                         │
│   [Transactional Services]              [Knowledge Graph Service]           │
│   [Payment Service]                     [Analytics Service]                 │
│   [Notification Service]                [Audit Service]                     │
│                                                                             │
│   Bridge: RabbitMQ events → Kafka (for retention/replay)                   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Pros: - Best of both worlds - RabbitMQ for existing patterns (low migration) - Kafka for new streaming use cases - Event replay for Knowledge Graph ingestion - Incremental adoption

Cons: - Two messaging systems to operate - Increased infrastructure complexity - Bridge logic between systems - Team needs to learn Kafka

Option C: Full Kafka Migration

Replace RabbitMQ entirely with Kafka:

Pros: - Single messaging platform - Full event sourcing capability - Schema Registry for contract management - Future-proof for streaming analytics - Better consumer lag monitoring

Cons: - High migration risk (92 classes) - 3-6 months effort - Team training required - Some patterns less natural (request/reply) - Higher operational complexity initially

Hypothesis-Driven Evaluation

Primary Hypothesis (Option B - Hybrid)

A hybrid RabbitMQ + Kafka architecture provides the optimal balance of reliability for existing patterns and capability for new streaming requirements, while minimizing migration risk.

Evidence Chain

Overall Confidence: L1 (WLNK capped by unverified team learning curve)

Alternative Hypothesis (Option A - Enhance RabbitMQ)

Enhancing RabbitMQ with reliability patterns is sufficient for all current and near-term requirements.

• Evidence: Current workload is command/event, not event sourcing (L2)
• Counter-evidence: Knowledge Graph needs replay for bulk migration (L1)
• Counter-evidence: CDD audit trail requirements emerging (L1)

Assessment: Option A is viable short-term but may require revisiting within 12 months.

Falsifiability Criteria

Option B (Hybrid) is FALSE if: - Kafka operational overhead exceeds benefit (>40 hours/month ops) - Event replay is never actually used in production - Team struggles to adopt Kafka after 6 months - Two-system complexity causes more incidents than it prevents

Option A (Enhanced RabbitMQ) is FALSE if: - Event replay becomes critical requirement within 6 months - Compliance audit requires full event history - Stream processing becomes core capability need

Recommendation

Short-Term (0-3 months): Enhanced RabbitMQ

Implement v1 reliability enhancements: 1. Add idempotency to core-lib MessageHandler 2. Configure DLQ with retry policy 3. Add circuit breakers for sync calls 4. Consolidate to single cluster with vhosts

Medium-Term (3-12 months): Hybrid Architecture Pilot

1. Deploy Kafka alongside RabbitMQ
2. Knowledge Graph service uses Kafka for:
   • Event ingestion (replay capability)
   • Analytics event streaming
   • Audit trail retention
3. Bridge: Forward critical RabbitMQ events to Kafka for retention
4. Evaluate operational experience

Long-Term (12+ months): Strategic Decision

Based on hybrid pilot: - If Kafka proves valuable → gradual migration of more services - If Kafka overhead outweighs benefit → remain on enhanced RabbitMQ - Decision informed by actual operational data

Implementation Plan (Hybrid Approach)

Phase 1: RabbitMQ Enhancements (Weeks 1-4)

Phase 2: Kafka Infrastructure (Weeks 5-8)

Phase 3: Knowledge Graph Kafka Integration (Weeks 9-12)

Phase 4: Bridge & Evaluation (Weeks 13-16)

Bounded Validity

Scope

• Applies to: All inter-service messaging, event streaming, audit requirements
• Does not apply to: In-process events, database triggers, webhooks

Expiry Conditions

• Time-based: Re-evaluate after 6-month hybrid pilot
• Scale-based: If message volume exceeds 1M/day
• Complexity-based: If operational overhead of hybrid exceeds 40 hours/month

Review Triggers

• Kafka adoption rate among teams
• Incident rate comparison (RabbitMQ vs Kafka)
• Event replay actual usage frequency
• Cost variance from estimates

Monitoring

• Message throughput per system
• Consumer lag (Kafka)
• DLQ depth (RabbitMQ)
• Cross-system latency (bridge)

Consequences

Positive (Hybrid Approach)

• Reliability improvements on existing RabbitMQ
• Event replay capability for Knowledge Graph
• Incremental Kafka adoption reduces risk
• Future-proof for streaming analytics
• CDD audit trail capability

Negative (Hybrid Approach)

• Two messaging systems to operate
• Bridge adds complexity
• Team learning curve for Kafka
• Higher initial infrastructure cost

Neutral

• Core-lib abstraction can support both backends
• Existing services unchanged initially

Related Decisions

• ADR-002: Multi-tenant compute (cluster consolidation)
• ADR-019: Knowledge Graph (event ingestion requirements)
• ADR-014: Observability (message tracing)

Original decision date: 2026-02-01 Revision date: 2026-02-01 Review by: 2026-08-01