remote
Senior Data Engineer - Data Fabric Remote - ClearCaptions, LLC
Data Engineer
Senior Data Engineer responsible for designing, building, and maintaining scalable data pipelines and data‑fabric solutions using Python, SQL, Azure Data Factory, and Spark to support real‑time captioning services.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines that ingest, transform, and store large volumes of audio and captioning metadata.
- Implement and optimize data‑fabric architectures using Azure Data Factory and Apache Spark for near real‑time processing.
- Collaborate with product, ML, and operations teams to define data requirements and ensure data quality and reliability.
- Build and maintain data models and schemas that support analytics, reporting, and machine‑learning workloads.
- Automate workflow orchestration, monitoring, and alerting to achieve high availability and performance.
Requirements
- 5+ years of professional experience in data engineering, with strong expertise in Python and SQL.
- Hands‑on experience building pipelines in Azure Data Factory and processing data with Apache Spark.
- Proven ability to design scalable data models and implement ETL/ELT solutions.
- Familiarity with cloud data platforms, data‑fabric concepts, and real‑time streaming architectures.
- Strong problem‑solving skills and ability to work independently in a fully remote environment.
Skills
pythonsqlapache spark