Senior Data Engineer

Company introduction We are seeking a Senior Data Engineer to join our forecasting platforms and operations team, supporting the development, integration, operational reliability, and continued modernization of data pipelines and applications within our energy forecasting ecosystem. Role summary / purpose The Senior Data Engineer – Forecasting Platforms & Operations will combine strong data engineering capabilities with hands-on production application support. This role focuses primarily on building and maintaining scalable ETL/ELT pipelines, data integrations, databases, APIs, and automated data workflows supporting load, wind, and solar forecasting, while also providing secondary operational support for cloud-based platforms and services responsible for producing and delivering forecasts to business customers. Key responsibilities Data Engineering & Integration • Design, develop, maintain, and optimize ETL/ELT pipelines supporting forecasting, telemetry, weather, market, asset, and operational data. • Develop and maintain automated data ingestion and transformation processes across APIs, databases, flat files, SFTP, cloud storage, and external data providers. • Build scalable and reusable Python and SQL-based data processing frameworks. • Support PostgreSQL and other relational database environments used by forecasting applications and analytical workflows. • Improve data pipeline monitoring, logging, restart ability, validation, and failure recovery. • Implement data-quality checks, reconciliation processes, and source-to-target validation. • Support onboarding and integration of new wind, solar, storage, and other generation assets into forecasting data pipelines. • Partner with Data Scientists to ensure reliable, timely, and well-structured data is available for model development, training, production forecasting, and performance analysis. • Support modernization and standardization of legacy data workflows and reduce manual or hard-coded processing wherever practical. Forecasting Platform & Operational Support • Provide secondary technical support for production load, wind, and solar forecasting applications and associated data-delivery processes. • Troubleshoot production issues across application, data, database, cloud infrastructure, integration, and delivery layers. • Support AWS-hosted forecasting workloads and services such as S3, Lambda, ECS, EC2, CloudWatch, and related cloud resources. • Assist with application deployments, release validation, configuration changes, operational readiness, and post-deployment monitoring. • Support scheduled jobs, automated forecast processes, reporting services, and customer-facing forecast delivery workflows. • Assist with SFTP services, secure file transfers, SSH keys, directory configurations, and customer data-exchange processes. • Participate in incident response, root-cause analysis, change management, service-ticket resolution, and production support. • Maintain sufficient cross-functional knowledge to provide backup coverage for key forecasting platform responsibilities when the primary platform resource is unavailable. Reliability, Automation & Documentation • Identify recurring manual processes and develop automation to improve reliability and reduce operational effort. • Establish effective monitoring and alerting for pipelines, scheduled processes, data feeds, and production forecasting applications. • Document system dependencies, troubleshooting proce…