onsite
Walk-in || Data Engineer
Data Engineer
Overall Objectives of job We are looking for a Senior Data Engineer with strong Databricks, SQL and data warehousing experience to lead the migration of an existing Oracle-based data warehouse to Databricks on the Allianz Data Platform (ADP).
About the role
Overall Objectives of job We are looking for a Senior Data Engineer with strong Databricks, SQL and data warehousing experience to lead the migration of an existing Oracle-based data warehouse to Databricks on the Allianz Data Platform (ADP). Experience with modern data engineering practices and AI-enabling data platforms is highly desirable. Duties and responsibilities • Analyse the current Oracle data warehouse (including PostgreSQL source feeds and SAS ETLgenerated structures) and design the target model in Databricks. • Build and optimise ETL/ELT pipelines in Databricks (Spark SQL, PySpark) to migrate core data warehouse tables and datamarts from Oracle to Databricks. • Translate Oracle SQL/PL-SQL logic into Databricks Spark SQL and PySpark, ensuring functional equivalence and performance optimisation. • Ensure data quality, performance, and reliability of the new Databricks-based data warehouse, enabling downstream analytics and AI/ML use cases. • Collaborate
Skills
databrickssqldata warehousingpostgresqlsparketlitilmachine learningaianalyticsdata engineeringdata qualityeltoraclepysparksas