Seeking a highly skilled data engineer, to design, build, and optimize our data infrastructure and analytics initiatives. As a hands-on technology lead, you will contribute to the design, development, enhancement & maintenance of data frameworks, ensuring data integrity, scalability and security.
We are looking for a skilled Data Engineer for Cloud Data Lake activities. The candidate should have industry experience (preferably in Financial Services) in supporting enterprise applications and exposure to Cloud based Data engineering platforms with deep understanding of data processing frameworks and solid experience with ETL development using distributed computing frameworks as Apache Spark, Scala, Hadoop, Hive. This hands-on role will partner with senior-level development managers, architects and business leadership to develop and execute the technology product roadmap.
What you will be responsible for
- Collaborate with data scientists, analysts, and product teams to enable analytics and ML use cases
- Ensure data quality, reliability, security, and governance across data platforms
- Optimize performance and cost of data workloads in AWS and Databricks
- Implement CI/CD pipelines and infrastructure as code (e.g., Terraform, CloudFormation)
- Monitor, troubleshoot, and resolve data pipeline and platform issues
- Document data architecture, pipelines, and operational processes
- End to end Cloud Data Lake design & development including data ingestion, data modeling and data distribution.
- Build data integrations, hand-offs between on-prem/cloud-based systems.
- Design, build, and maintain scalable, reliable data pipelines using AWS services and Databricks
- Develop and optimize ETL/ELT workflows for batch and near‑real‑time data processing
- Implement data solutions using Apache Spark (PySpark/Scala) on Databricks
- Leverage AWS services such as S3, Glue, Lambda, EMR, Redshift, Athena, and Kinesis
- Highly motivated, with the ability to take initiative, set priorities and perform tasks with minimal guidance
- Management oversight of a global team of software development and QA engineers.
- Building strong partnership with technology & business leaders and peers
- Work closely with the Product Owner and others on requirements analysis, planning, project management, and execution.
What we value
These skills will help you succeed in this role
- Development experience in designing, developing and deploying enterprise applications.
- Strong skill-set, experience in PL/SQL, Python, Scala/PySpark, PERL/scripting.
- Hands on experience with Cloud platforms, AWS, Databricks.
- Experience in building systems using modern, scalable, resilient, data lakes and architectures.
- Experience in Data Modeling and database design with strong skills in performance tuning.