remoteonsite
Senior Data Engineer - LPL Financial Global Capability Center
Data Engineer
Senior Data Engineer responsible for designing, building, and optimizing large‑scale data pipelines and platforms using Python, Spark, AWS, and Kafka to enable real‑time analytics and data‑driven decision making.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and ETL processes on cloud platforms.
- Implement real‑time data streaming solutions using Kafka and batch processing with Apache Spark.
- Collaborate with data scientists, analysts, and product owners to translate business requirements into robust data models.
- Optimize SQL queries and data warehouse schemas for performance and cost efficiency.
- Automate workflow orchestration and monitoring with Apache Airflow.
- Ensure data quality, governance, and security compliance across all data assets.
Requirements
- 5+ years of hands‑on experience in data engineering, preferably in financial services.
- Proficiency in Python, SQL, and big‑data technologies such as Spark.
- Strong knowledge of AWS services (S3, Redshift, Glue, Lambda) and containerization.
- Experience building streaming pipelines with Kafka or similar messaging systems.
- Solid understanding of data modeling, ETL best practices, and workflow orchestration tools like Airflow.
Skills
pythonsqlapache sparkawskafkaairflow