remoteonsite
Senior Databricks Data Engineer - Matrix International Financial Services
Data Engineer
Lead the design and optimization of enterprise‑scale data pipelines using Databricks, Spark, and Kafka on AWS, driving real‑time analytics and cloud‑native data solutions.
About the role
Key Responsibilities
- Architect, develop, and maintain scalable data pipelines on Databricks using Spark and Python.
- Integrate real‑time data streams with Kafka, ensuring low latency and high throughput.
- Collaborate with data scientists and analysts to deliver clean, reliable datasets for analytics and ML workloads.
- Optimize performance and cost of cloud‑native data platforms on AWS.
- Implement robust monitoring, logging, and security best practices across the data stack.
Requirements
- 5+ years of data engineering experience with a focus on Databricks and Spark.
- Strong proficiency in Python and experience with Kafka stream processing.
- Hands‑on experience deploying and managing data solutions on AWS (EMR, S3, Glue).
- Deep understanding of distributed computing, data modeling, and ETL best practices.
- Excellent problem‑solving skills and ability to work in a fast‑paced, collaborative environment.
Skills
databrickskafkapythonaws