Location:
Shift:
Scheduled Weekly Hours:
Worker Type:
Job Summary:
Job Duties:
Key Responsibilities
Market Data & Data Product Enablement
- Partner with product/business stakeholders to define market data product objectives and translate them into data platform deliverables.
- Consolidate, normalise, and curate market data sets (including derivatives and order book datasets) as governed, reusable data assets.
- Define data contracts, metadata, lineage, and quality rules so downstream users can reliably consume market data products.
Enterprise Data Management & Architecture
- Define and evolve enterprise data management architecture across data lake and data warehouse solutions (on-prem and/or cloud).
- Design and operate data lake/warehouse layers using technologies such as ADLS, Amazon S3, Google Cloud Storage, Azure Synapse SQL, Snowflake, Amazon Redshift, or Google BigQuery.
- Set standards for data modelling, governance, security controls, retention, and lifecycle management aligned with organisational policies.
Big Data Engineering & Pipeline Delivery
- Design, build, and maintain scalable ETL/ELT pipelines for analytics and reporting using code-driven patterns and distributed compute engines.
- Implement and operate workflow orchestration frameworks such as Apache Airflow, Prefect ("Perfect"), or Dagster, including scheduling, dependency management, and observability.
- Engineer processing solutions using big data stacks such as Hadoop, Spark, Kafka, and Flink ("Flint"), ensuring throughput, reliability, and cost efficiency.
- Leverage Spark and/or Databricks (built on Spark) to deliver large-scale transformations and performance-tuned workloads.
Data Stores, Query Performance & Reliability
- Design data storage and access patterns across data warehouses and databases, including NoSQL stores (e.g., HBase) and analytical engines (e.g., ClickHouse, Snowflake).
- Drive query and pipeline performance tuning (partitioning, caching, file formats, indexing/cluster keys) and improve SLAs/SLOs for critical datasets.
- Lead incident analysis and root-cause investigations for data-related issues; implement permanent fixes and continuous reliability improvements.
Leadership & Delivery
- Operate effectively in a small, specialised team—balancing hands-on contribution with technical leadership, coaching, and setting engineering standards.
- Promote SDLC best practices, CI/CD, automated testing, monitoring, and documentation to improve delivery quality and repeatability.
- Coordinate with global engineering and infrastructure teams to deliver roadmap outcomes and manage dependencies.
Requirements