ABOUT THE ROLE
We are building a high-scale payment orchestration platform that processes millions of financial transactions every month across multiple providers, regions, and products.
We are looking for an experienced OpenSearch Engineer to own and scale our AWS OpenSearch infrastructure and the pipelines that synchronize data from PostgreSQL.
You will be responsible for search architecture, data ingestion, performance, reliability, reconciliation, and recovery.
The role
combines search infrastructure, data engineering, and production operations.
KEY RESPONSIBILITIES
OpenSearch
- Own and operate AWS OpenSearch Service in production.
- Design indices, mappings, analyzers, templates, aliases, shards, and retention policies.
- Optimize search queries, aggregations, filtering, pagination, and bulk indexing.
- Investigate slow queries, indexing bottlenecks, hot shards, memory pressure, and rejected requests.
- Plan safe upgrades, migrations, reindexing, and capacity scaling.
PostgreSQL-to-OpenSearch pipelines
- Build and maintain reliable pipelines that synchronize PostgreSQL data with OpenSearch.
- Support inserts, updates, deletes, backfills, and full reindexing.
- Use AWS-native services such as AWS DMS, OpenSearch Ingestion, Lambda, SQS, S3, ECS, or EKS where appropriate.
- Implement retries, idempotency, deduplication, replay, dead-letter handling, and failure recovery.
- Monitor ingestion lag, throughput, failed records, and data freshness.
- Build reconciliation processes to detect and repair missing, stale, or inconsistent documents.
- Ensure ingestion does not negatively affect the transactional PostgreSQL workload.
Reliability and operations
- Monitor cluster health, search latency, indexing latency, storage, CPU, JVM pressure, and shard distribution.
- Build CloudWatch dashboards, alerts, and operational runbooks.
- Design snapshot, restore, backup, and disaster-recovery procedures.
- Participate in production incident response and root-cause analysis.
- Implement secure access using IAM, role-based access control, encryption, and least-privilege principles.
Collaboration
- Work closely with backend, database, infrastructure, and product teams.
- Help design efficient OpenSearch document models for payments, customers, subscriptions, and transactions.
- Define standards for searchable fields, schema changes, and index versioning.
- Document architecture, data flows, and recovery procedures.
QUALIFICATIONS
- Strong production experience with OpenSearch or Elasticsearch.
- Hands-on experience with AWS OpenSearch Service.
- Experience building data pipelines between PostgreSQL and OpenSearch.