remote
Senior Data Engineer - Looi Consulting
Data Engineer
Senior Data Engineer responsible for designing, building, and maintaining robust data pipelines from diverse sources into data marts, leveraging Python, ETL/ELT processes, RESTful APIs, and event streaming to support advanced analytics and data mining.
About the role
We are looking for a Senior Data Engineer to join a team to work on data pipeline development. You would work closely with small, highly collaborative Operations and Development teams to architect and deploy high-impact security solutions.
You must be able to work US time zones (UTC-8 to UTC-5/UTC-7 to UTC-4).
Job Responsibilities
- Build data management pipelines from heterogeneous data sources into Data Marts and other repositories for subsequent analysis and data mining
- Code, test, and support Python-based services, implementing ETL/ELT, data cleansing, etc.
- Integrate with internal (on-prem) and external services utilizing various approaches like RESTful APIs, DMS, distributed event streaming etc., develop shippable code, documentation, and unit test new features
- Collaborate with Quality, Product, and other Engineering teams
- Provide code reviews, design feedback, demos, technical requirements & documentation
- Scope projects, provide accurate estimates for reliable delivery of projects
- Facilitate discovery sessions to map out business processes.
- Collaborate with data architects, business users and create comprehensive functional specifications, and process flow diagrams that bridge the gap between business needs and data engineering implementation.
- Independently explore backend systems and database schemas to determine the feasibility of business requirements.
- Trace data lineage to understand how a metric is calculated in the source system versus how the business defines it, resolving discrepancies before they reach the dashboard
Key Applicant Requirements
- Professional development experience using Python, including an understanding of functional programming: 5+ years
- Cloud infrastructure experience in AWS, Docker: 3+ years
- Experience in Distributed event streaming and process orchestration and with modern data stack.
- Infrastructure As Code (Terraform) experience: 1+ years
- Strong proficiency in Python and SQL.
- Proficient with relational databases such as Oracle, MySQL, PostgreSQL
- Experience with AWS services such as RDS, Redshift, S3, EMR, MWAA (Airflow) and MSK, ECS, SNS, SQS etc.
- Experience with designing and building Data Marts and ETL/ELT
- Proficient in data profiling and analysis techniques utilizing SQL and/or Python.
- Proficient with Jira, Confluence, and GIT for version control.
- Hands-on experience with Agile/Scrum/Kanban
- Excellent written & verbal communication skills, working asynchronously and independently in a remote environment.
- Ability to coordinate between teams.
- Data visualization dashboarding knowledge/experience.
Pluses
- Design, implementation, and/or maintenance of CICD, Bash scripts
- Experience with using AWS EMR Serverless