Overview
Pyramid Systems is looking for a Data Engineer (Senior) who is passionate about bringing creative architect solutions to end customers.
Key Skills:
- 8+ years of IT experience focusing on enterprise data architecture and management
- Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
- Experience with ETL and ELT tools such as SSIS, Pentaho, and/or Data Migration Services
- Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
Responsibilities
- Plan, create, and maintain data architectures, ensuring alignment with business requirements
- Obtain data, formulate dataset processes, and store optimized data
- Identify problems and inefficiencies and apply solutions
- Determine tasks where manual participation can be eliminated with automation.
- Identify and optimize data bottlenecks, leveraging automation where possible
- Create and manage data lifecycle policies (retention, backups/restore, etc)
- In-depth knowledge for creating, maintaining, and managing ETL/ELT pipelines
- Create, maintain, and manage data transformations
- Maintain/update documentation
- Create, maintain, and manage data pipeline schedules
- Monitor data pipelines
- Create, maintain, and manage data quality gates (Great Expectations) to ensure high data quality
- Support AI/ML teams with optimizing feature engineering code
- Expertise in Spark/Python/Databricks, Data Lake and SQL
- Create, maintain, and manage Spark Structured Steaming jobs, including using the newer Delta Live Tables and/or DBT
Research existing data in the data lake to determine best sources for data
Create, manage, and maintain ksqlDB and Kafka Streams queries/code
Data driven testing for data quality
Maintain and update Python-based data processing scripts executed on AWS Lambdas
Unit tests for all the Spark, Python data processing and Lambda codes
Maintain PCIS Reporting Database data lake with optimizations and maintenance (performance tuning, etc)
Streamlining data processing experience including formalizing concepts of how to handle lake data, defining windows, and how window definitions impact data freshness.
Qualifications
- 8+ years of IT experience focusing on enterprise data architecture and management
- Must be able to obtain a Public Trust security clearance
- MUST BE US CITIZEN
- Bachelor degree required
- Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
- Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required