onsite
Data Engineer II - Preparing Customers for AI - Microsoft
Data Engineer
Data Engineer II building scalable data pipelines and AI‑ready data platforms on Azure, leveraging Python, SQL, Spark, and data modeling to support security‑focused solutions.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines that ingest, transform, and store large‑scale security telemetry.
- Implement data models and schemas optimized for AI/ML workloads in Azure Synapse and Azure Data Lake.
- Collaborate with security engineers and data scientists to expose clean, reliable datasets for threat detection and analytics.
- Automate data quality, monitoring, and performance tuning using Python, Spark, and Azure monitoring tools.
- Ensure compliance with security, privacy, and governance standards throughout the data lifecycle.
Requirements
- 3+ years of experience building data pipelines on Azure (Azure Data Factory, Azure Synapse, Azure Data Lake).
- Proficiency in Python and SQL for data transformation and querying.
- Hands‑on experience with Spark (PySpark or Spark SQL) and data modeling for analytical workloads.
- Strong understanding of security data domains and experience working with large, heterogeneous datasets.
- Ability to work cross‑functionally, communicate technical concepts clearly, and drive solutions in a fast‑moving environment.