onsite
Senior Data Support Engineer Apache Airflow - St. James's Place
Software Engineer
Senior engineer responsible for supporting, troubleshooting, and optimizing Apache Airflow data pipelines, leveraging Python, SQL, and cloud services to ensure reliable, scalable data workflows.
About the role
Key Responsibilities
- Provide 2nd/3rd line support for Apache Airflow clusters, including incident response, root‑cause analysis, and post‑mortem documentation.
- Maintain and enhance existing DAGs, develop new pipelines, and ensure best practices for code quality, testing, and documentation.
- Monitor system health, performance metrics, and resource utilization; implement alerts and automated remediation where possible.
- Collaborate with data engineering, analytics, and infrastructure teams to design scalable, secure data workflows on cloud platforms (AWS/GCP).
- Drive continuous improvement by identifying bottlenecks, recommending architecture changes, and contributing to knowledge‑sharing initiatives.
Requirements
- 5+ years of experience in data engineering or support roles, with deep hands‑on expertise in Apache Airflow.
- Proficient in Python scripting and SQL for data transformation and orchestration.
- Strong Linux/Unix administration skills and familiarity with containerisation (Docker) or orchestration (Kubernetes) is a plus.
- Experience deploying and managing data pipelines on cloud platforms such as AWS or GCP.
- Excellent problem‑solving abilities, communication skills, and a proactive attitude toward incident resolution and process improvement.