Role Overview
This role focuses on building the data foundations that enable advanced analytics and machine learning at scale. You will design and develop data pipelines and architectures that support real-time insights and AI-driven applications across the business.
Working with large, complex datasets, you will ensure data is reliable, accessible, and optimized for both analytics and machine learning use cases. With strong ownership and technical depth, this role plays a critical part in shaping how data is structured and leveraged, enabling high-performance systems in a fast-growing, data-intensive environment.
Key Responsibilities
- Design and implement distributed data pipelines for batch and real-time processing.
- Build and maintain data ingestion frameworks using:
- streaming systems (Kafka, Kinesis)
- batch processing (Spark, Airflow)
- Develop data models and warehouse schemas optimized for analytics and machine learning.
- Build and manage feature stores to support ML model training and inference.
- Integrate data from multiple sources (internal systems, APIs, third-party platforms) into a unified architecture.
- Implement data quality validation frameworks and monitoring systems.
- Optimize data pipelines for performance, scalability, and cost efficiency.
- Enable real-time data availability for AI-driven applications.
- Collaborate with ML Engineers to ensure seamless integration between data pipelines and model pipelines.
- Implement security, governance, and compliance standards across data systems.
Technical Requirements
- Strong SQL and Python for large-scale data processing.
- Experience with big data frameworks:
- Experience with streaming technologies:
- Experience with ETL/ELT tools:
- Strong knowledge of data warehousing solutions:
- Snowflake, Redshift, BigQuery
- Experience with cloud platforms (AWS preferred):
- Understanding of data modeling techniques (star schema, normalization).
- Experience building scalable, distributed systems.
- Exposure to ML pipelines, feature engineering, and data preparation for AI models.
- Familiarity with data governance, lineage, and monitoring tools.