Staff Software Engineer, Data Platform - Realtor.com
Software Engineer
We're looking for a Software Engineer focused on designing and building scalable technical solutions. This lead role requires 8+ years of relevant experience.
About the role
Description
Real-time and batch data ingestion through Prism/Data Collector and related APIs.
Kafka-based streaming on AWS Managed Streaming for Apache Kafka (MSK), including topics, producers, consumers, Kafka Connect, KSQL, and streaming integrations.
Data contracts and schema governance using Avro and Schema Registry, including compatibility, validation, and lineage metadata.
External-system replication and integration through Fivetran and Kafka Connect, including secure connection management and operational support.
Scheduled and batch workflows through Apache Airflow on AWS Managed Workflows for Apache Airflow (MWAA).
Data transformation and modeling through dbt, including integration with Airflow, Snowflake, and the data catalog.
Snowflake lakehouse access patterns, data exports, reusable toolsets, and platform libraries.
Observability, alerting, incident response, pipeline health checks, and self-service troubleshooting across ingestion, streaming, connectors, Airflow, dbt, and Snowflake.
Documentation, onboarding, paved paths, reference implementations, and developer experiences that help teams adopt the platform without bespoke support
Design, build, and operate scalable ingestion, streaming, replication, orchestration, and transformation capabilities used by many teams.
Develop and improve platform APIs, services, libraries, templates, and paved paths for streaming, batch, bulk, and external data ingestion
Build and maintain ingestion APIs and client libraries in TypeScript, Kotlin, and Python, supporting streaming, batch, bulk, and external data-ingestion use cases.
Operate and evolve Kafka/MSK infrastructure, including topics, partitions, replication, retention, throughput, consumer groups, access controls, and upgrades.
Support Kafka Connect and sink integrations for systems such as Snowflake, S3, New Relic, and Elasticsearch; improve connector reliability, provisioning, and troubleshooting.
Build and maintain Schema Registry and Avro-based data contract workflows, including schema validation, compatibility, versioning, and safe evolution.
Build and maintain KSQL or equivalent near-real-time transformation patterns where appropriate.
Improve Airflow/MWAA orchestration patterns, DAG templates, scheduling, backfills, retries, dependency management, and operational reliability.
Improve dbt platform capabilities, including project conventions, macros, testing, materializations, restricted-data patterns, and Airflow integration.
Develop reusable TypeScript, Python, and/or Kotlin libraries and reference implementations for producing, consuming, validating, and moving data.
Contribute to Snowflake integration patterns, data exports, toolsets, and secure access paths without taking ownership of stakeholder-specific data modeling.