ADR-019: Search & Elasticsearch Strategy
ADR-019: Search & Elasticsearch Strategy
Status
Proposed — Pending engineering team review
Context
The platform runs self-managed Elasticsearch 7.15.2 and Kibana 7.15.2, both end-of-life. ES serves two distinct purposes: (1) content/expert search for fans, and (2) log aggregation via peeq-logging. ADR-014 addresses the logging use case (migrate to Loki/Cloud Logging). This ADR addresses the search use case.
Current State
| Component | Version | Status |
|---|---|---|
| Elasticsearch | 7.15.2 | EOL — no security patches |
| Kibana | 7.15.2 | EOL |
| Search service | Gen 2 Spring Boot | Wraps ES queries for content/expert search |
| Index management | Manual / search-service | No automated reindexing pipeline |
| peeq-logging | Gen 1 Node.js | Sends logs to ES — being replaced by ADR-014 |
Search Requirements
| Requirement | Current | Needed |
|---|---|---|
| Expert search (by name, sport, expertise) | Yes | Yes — core fan experience |
| Content search (videos, articles, classes) | Yes | Yes — content discovery |
| Full-text search with relevance ranking | Basic | Improved — faceted search, typo tolerance |
| Multi-tenant search | Separate ES per tenant (cluster-per-tenant) | Tenant-isolated indices in shared cluster |
| Real-time indexing | Near-real-time via search-service | Same or better |
| Search analytics | None | Track popular searches, zero-result queries |
Decision
Upgrade to Elasticsearch 8.x (managed via Elastic Cloud) for content and expert search. Decouple search from logging (logging moves to Loki per ADR-014).
Why Elastic Cloud 8.x (Not Self-Managed, Not Alternatives)
| Option | Assessment |
|---|---|
| Elastic Cloud 8.x (Recommended) | Managed service. Automatic security patches. No ops overhead. Vector search for future AI features. |
| Self-managed ES 8.x | Same EOL risk returns — requires ongoing ops for upgrades, patching, scaling. |
| OpenSearch | AWS-aligned fork. Viable but adds vendor divergence. No clear advantage over Elastic Cloud. |
| Algolia | SaaS search. Excellent DX but pricing scales per search operation. May be expensive at scale. |
| Meilisearch | Open source, simpler than ES. Good for content search but less mature for complex queries. |
| PostgreSQL full-text | Already have Postgres. Viable for simple search but lacks relevance ranking sophistication. |
Elastic Cloud is the lowest-risk option: same technology (ES), managed service (no ops), and provides a migration path from 7.x indices.
Migration Approach
- Provision Elastic Cloud 8.x cluster — single cluster with tenant-isolated indices
- Upgrade search-service ES client — update from ES 7.x Java client to ES 8.x client
- Reindex — rebuild indices from source databases (search-service pulls from PostgreSQL)
- Validate — compare search results between old and new clusters
- Switch — point search-service to Elastic Cloud
- Decommission — remove self-managed ES 7.x and Kibana 7.x (logging already moved per ADR-014)
Index Strategy
| Index | Source | Refresh |
|---|---|---|
{tenant}-experts |
celebrity-db + fan-db | Near-real-time via RabbitMQ events |
{tenant}-content |
content-db + media-db | Near-real-time via RabbitMQ events |
{tenant}-classes |
class-catalog-db | Near-real-time via RabbitMQ events |
{tenant}-events |
onsite-event-db + webinar-db | Near-real-time via RabbitMQ events |
Hypothesis Background
Primary: Elastic Cloud 8.x provides the best balance of search capability, operational simplicity, and migration ease from ES 7.x.
- Evidence: ES 7.15.2 is EOL (L2)
- Evidence: Search is used for content and expert discovery — core user experience (L1)
- Evidence: Self-managed ES requires ongoing ops for a non-differentiating capability (L1)
- Evidence: Elastic Cloud supports index migration from 7.x (L1 — documented)
Alternative: PostgreSQL full-text search (eliminate
ES entirely). - Not rejected permanently. If search requirements are
truly simple (name lookup, basic content search), Postgres
tsvector + tsquery may be sufficient and
eliminates an entire infrastructure dependency. Worth a POC if Elastic
Cloud costs prove prohibitive.
Falsifiability Criteria
- If Elastic Cloud costs exceed $X/month → evaluate PostgreSQL full-text search as replacement
- If search query volume is <100/day → ES is over-engineering; migrate to PostgreSQL full-text
- If vector search / AI-powered search becomes a priority → validates Elastic Cloud choice (ES 8.x has native vector search)
Evidence Quality
| Evidence | Assurance |
|---|---|
| ES 7.15.2 is EOL | L2 |
| Search used for content/expert discovery | L2 (verified from search-service code) |
| Actual search query volume | L0 (need production metrics) |
| Elastic Cloud migration from 7.x | L1 (documented, not tested) |
| Elastic Cloud pricing at our scale | L0 (need capacity estimate) |
Overall: L0 (WLNK capped by unknown query volume and pricing)
Bounded Validity
- Scope: Content and expert search only. Log search handled by ADR-014 (Loki/Cloud Logging).
- Expiry: Re-evaluate if search query volume is discovered to be very low (<100/day).
- Review trigger: If Elastic Cloud costs are prohibitive. If AI-powered search becomes a priority (validates ES 8.x choice).
Consequences
Positive: - Eliminates EOL ES 7.x security risk - Managed service — no ES operations overhead - Automatic security patches and version upgrades - Foundation for vector search / AI-powered discovery - Clean separation: search (Elastic Cloud) vs logging (Loki/Cloud Logging)
Negative: - Recurring Elastic Cloud cost (vs self-managed) - ES client library upgrade needed in search-service - Full reindex required during migration
Decision date: 2026-02-01 Review by: 2026-08-01