About this role
Role summary / purpose Data Engineer - iGrid Key responsibilities • Design, develop, and maintain scalable ETL pipelines using AWS Glue, EMR (PySpark), and Lambda functions • Implement real-time data streaming solutions using Confluent Kafka with source and sink connectors • Develop and optimize PySpark jobs on EMR clusters for large-scale data transformations • Build serverless data processing workflows using AWS Lambda • Design and optimize queries in Amazon Athena for analytical workloads • Manage data warehouse solutions in Amazon Redshift including data modeling and performance tuning • Implement monitoring and alerting using CloudWatch for data pipeline health and performance • Provision and manage AWS infrastructure using Terraform • Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements • Implement data governance and security best practices using Lake Formation, IAM, and KMS Required skills • Python: Advanced proficiency in Python 3.8+ for data processing, scripting, and automation • PySpark: Extensive experience with PySpark for large-scale data processing and transformations on EMR clusters • AWS Lambda: Hands-on experience building serverless functions for data processing and orchestration • AWS Glue: Expertise in AWS Glue for ETL jobs, Data Catalog management • Amazon Athena: Experience with Athena for SQL-based data analysis and querying data lakes • Amazon EMR: Strong experience with EMR for running Spark jobs, EMR Notebooks, and cluster management • Amazon Redshift: Knowledge of Redshift for data warehousing, query optimization, and integration with data lakes • Kafka: Experience with Confluent Kafka for real-time data streaming, including Schema Registry, source/sink connectors, and producers/consumers • Amazon S3: Deep understanding of S3 for data lake architecture, bucket policies, and data organization (raw, curated layers) • CloudWatch: Proficiency in CloudWatch for logging, monitoring, and alerting on data pipelines • Terraform: Experience with Infrastructure as Code using Terraform for provisioning AWS resources Nice-to-have / preferred skills • Knowledge of AWS EventBridge and Step Functions for workflow orchestration • Familiarity with EMR runtime roles and execution roles for secure data access • Experience with DynamoDB for NoSQL data storage • Knowledge of API Gateway for building data APIs • EMR Notebooks or Jupyter for interactive development • VS Code or similar IDE for Python development • Git for version control • AWS CLI and boto3 for AWS service interaction • DBeaver, Data Studio for database management Requirements – education & experience • Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related field • Strong understanding of distributed computing and big data processing frameworks • Experience with Agile development methodologies • Excellent problem-solving and analytical skills • Strong communication and collaboration abilities
Posted by Bartech Staffing on behalf of a vetted Top Three US electric utility client. Your recruiter confirms the exact rate when you connect.