onsite
Senior Databricks Engineer - Neurealm
Software Engineer
Senior engineer responsible for designing, building, and optimizing ETL pipelines on Databricks and Apache Spark, ensuring data quality, performance, and usability for analytics and data science teams.
About the role
Key Responsibilities
- Design, develop, and maintain scalable ETL pipelines using Databricks and Apache Spark.
- Collaborate with data scientists and analysts to translate data requirements into efficient processing solutions.
- Implement data validation, testing, and monitoring to guarantee data integrity and reliability.
- Diagnose and resolve performance bottlenecks in Databricks clusters and workflows.
- Document data processing architectures, pipelines, and best‑practice guidelines.
- Perform data modeling and analysis to improve data accessibility and support downstream reporting.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field.
- 3+ years of hands‑on experience with Databricks and Apache Spark in production environments.
- Strong proficiency in SQL and a programming language such as Python for data engineering tasks.
- Demonstrated ability to design robust ETL processes and data models.
- Excellent problem‑solving skills and a proactive approach to staying current with platform updates and best practices.
Skills
databricksapache sparksqlpython