Data Engineer

Core Responsibilities: • Collaborate with Lead Developers (Data Engineer, Software Engineer, Data Scientist, Technical Test Lead) to understand requirements and use cases, outline technical scope, and deliver technical solutions • Collaborate with Data and Solution architects on key technical decisions • Develop data pipelines with focus on long-term reliability and maintaining high data quality • Design data lake and warehousing solutions with the end-user in mind, ensuring ease of use without compromising on performance • Manage and resolve issues in production data warehouse environments on AWS Core Experience and Abilities: • Perform hands-on development and peer review for certain components and tech stack • Set up development instances and migration paths with required security, access, and roles • Develop components and related processes (e.g., data pipelines, ETL processes, workflows) • Build new data pipelines, identify existing data gaps, and provide automated solutions to deliver analytical capabilities and enriched data to applications • Implement data pipelines with attentiveness to durability and data quality • Implement data warehousing products with focus on end-user experience (ease of use with appropriate performance) • Implement data quality frameworks and validation rules (e.g., schema validation, null checks, referential integrity, deduplication) • Design and implement automated data tests for ETL pipelines using Python, PySpark, and SQL • Write unit, integration, and regression tests for data pipelines (e.g., pytest-based testing for transformations and business rules) • Demonstrate familiarity with data observability and monitoring concepts, including freshness, volume, and anomaly detection • Understand data reconciliation and source-to-target validation techniques to ensure business logic accuracy • Embed data quality checks into CI/CD pipelines to prevent defective data from reaching downstream consumers Core Technical Skills: • 2+ years of AWS experience • AWS services: S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, Step Functions, Redshift • Experience with Kafka, preferably Confluent Kafka • Experience with Lake Formation, Amazon Redshift and Amazon Athena • Strong SQL and data modeling skills, executing ETL processes tailored for data warehousing • Competence in developing and refining data pipelines within AWS • Extensive understanding of database management fundamentals • Tools and Languages: Python (good experience in PySpark), SQL • Infrastructure as Code technology: Terraform • DevOps pipeline (CI/CD): GitHub • Deep knowledge of IAM roles and policies • Experience with AWS workflow orchestration tools like Airflow or Step Functions