ADR-014: Observability Strategy

Status

Proposed — Pending engineering team review

Context

The platform has critical observability gaps across all four pillars (metrics, tracing, logging, alerting). Monitoring infrastructure exists (Prometheus, Grafana, Elasticsearch) but is severely under-utilized. No automated incident detection — outages are discovered by users, not monitoring.

Current State

Impact

• P1 tech debt items: #11 (alerting), #12 (tracing), #18 (APM), #22 (ES 7.x EOL)
• Cannot measure service reliability during or after migration
• Cannot detect regressions introduced by service consolidation (ADR-001)
• Cannot debug cross-service request failures in 18-service architecture

Decision

Adopt OpenTelemetry as the unified observability framework, replacing the fragmented Elastic/Stackdriver/custom stack. Consolidate on Grafana ecosystem (LGTM stack) for visualization and alerting.

Target Architecture

Why OpenTelemetry + Grafana (Not Elastic Observability)

SLI/SLO Framework

Implementation Approach

1. Add OTel Java agent to common Helm chart — auto-instrumentation via -javaagent JVM flag. Zero application code changes. Instruments Spring Boot, GraphQL, RabbitMQ, JDBC, Redis automatically.
2. Deploy Grafana Tempo — distributed tracing backend. Receives OTel traces via OTLP protocol.
3. Deploy Loki — log aggregation. Application logs in structured JSON. Correlates with traces via trace ID.
4. Configure AlertManager rules — start with golden signals (latency, traffic, errors, saturation) per service.
5. Define SLOs with Sloth — generate PrometheusRules from SLO definitions. Grafana SLO dashboards.
6. Deploy Sentry — error tracking with release correlation. SDKs for Java (backend) and Angular/Next.js (frontend).
7. Decommission peeq-logging — replace with Cloud Logging for infrastructure + Loki for application logs.
8. Retire Elasticsearch 7.x for logging — keep only for content search (see ADR-019).

Hypothesis Background

Primary: OpenTelemetry + Grafana LGTM stack provides comprehensive observability with minimal application code changes and lower operational overhead than Elastic.

• Evidence: Prometheus + Grafana already deployed and operational (L2)
• Evidence: OTel Java agent auto-instruments Spring Boot applications (L1 — documented, not tested in our codebase)
• Evidence: Elastic APM disabled across all services — no existing investment to protect (L2)
• Evidence: ES 7.x is EOL — migration required regardless (L2)

Alternative 1: Upgrade to Elastic Cloud 8.x (full Elastic Observability). - Rejected: Higher cost, vendor lock-in, requires ES migration. We already have Prometheus + Grafana working.

Alternative 2: GCP-native (Cloud Monitoring, Cloud Trace, Cloud Logging only). - Not rejected entirely — Cloud Logging is part of the recommendation. But GCP-native tracing and dashboards are less flexible than Grafana for custom SLO dashboards and cross-signal correlation.

Falsifiability Criteria

• If OTel auto-instrumentation causes >5% latency overhead in production → reduce sampling rate or switch to manual instrumentation for hot paths
• If Tempo storage costs exceed $X/month at production trace volume → reduce retention or sampling
• If SLO targets prove unrealistic (error budget exhausted within first month) → revise targets based on baseline data
• If Loki query performance is insufficient for debugging → evaluate Elastic Cloud for log search specifically

Evidence Quality

Overall: L0 (WLNK capped by untested OTel overhead and unknown storage costs)

Bounded Validity

• Scope: All backend services, frontend error tracking. Does not cover infrastructure monitoring (GKE, Cloud SQL — already handled by GCP).
• Expiry: Re-evaluate if Grafana Cloud pricing becomes more favorable than self-managed, or if traffic grows 10x.
• Review trigger: If OTel overhead exceeds acceptable thresholds, or if trace storage costs are prohibitive.
• Monitoring: Track OTel agent overhead (CPU/memory), trace storage growth rate, and alert noise ratio.

Consequences

Positive: - Unified observability across all four pillars in a single tool (Grafana) - Zero application code changes for tracing (OTel auto-instrumentation) - Eliminates EOL Elasticsearch 7.x dependency for logging - Removes custom peeq-logging pipeline (Gen 1 Node.js service) - SLO-based alerting prevents alert fatigue - Enables debugging cross-service requests during and after migration - Industry-standard (CNCF) — no vendor lock-in

Negative: - Additional infrastructure to operate (Tempo, Loki, Sentry) - Learning curve for Grafana Tempo query language (TraceQL) - SLO target calibration requires baseline data collection period

Decision date: 2026-02-01 Review by: 2026-08-01