Senior Data Engineer

We are seeking a Senior Data Engineer who is experienced with data pipelines and data wrangling to join a nationally recognized renewable energy provider. You will expand and optimize existing data infrastructure, working closely with IT and Data Science team members. This person ideally has previous experience as a Software Engineer and can work effectively and communicate between the two disciplines.

Note: The company is unable to offer visa sponsorship.

The Senior Data Engineer will:

• Create, maintain, and enhance the infrastructure for data extraction, transformation, and loading (ETL), and API integrations.
• Develop robust data pipeline operations, ensuring scalability and reliability.
• Automate manual tasks, fine-tune data delivery mechanisms, and optimize data storage and queries to minimize costs within GCP.
• Design and implement error logging and process monitoring solutions, leveraging Google Cloud Platform technologies.
• Innovate internal processes by adopting top-tier DevOps practices.
• Refine version control infrastructure, implement GitHub Actions for automated processes, and establish CI/CD pipelines.
• Transition our bash processes to a containerized framework using Google Cloud Platform's CloudRun service.
• Proactively address challenges, supporting the team with process testing, bug fixes, and code reviews.
• Enhance our API using best practices in data architecture, engineering efficiency, and data security.
• Collaborate with the Analytics team to implement machine learning infrastructure.
• Maintain open-source software supporting clean and renewable energy initiatives.
• Strengthen IAM (Identity and Access Management) policies and collaborate on network security measures.
• Conduct regular security audits and risk assessments.
• Monitor security logs and alerts, responding to suspicious activities.
• Ad hoc projects and duties, as needed.

Here's what you'll need to succeed:

• A bachelor’s degree in computer science or another engineering / data-intensive discipline (MS preferred).

• At least 5 years of relevant professional experience, including building, maintaining, and optimizing ‘big data’ pipelines

• Strong experience with Bash Scripting, Python, Linux operating systems, and relational databases, ideally GCP BigQuery

• Demonstrated success maintaining and/or developing data or software engineering infrastructure for data analysis and sharing, and machine learning 

• Front end (TypeScript, Vue3, Vuetify3, Google Maps / DeckGL) and backend (Golang) software development

• Google Cloud Services, such as GCP, or other cloud platform experience.

• Solid understanding and experience with CRMs and the backend infrastructure, preferrably Salesforce, Salesforce Bulk Python API.

• Extensive experience with batch-processing systems, data pipeline and workflow management tools (e.g. Luigi, Airflow), containerized applications and container based orchestration environments (e.g., Docker, Kubernetes), and infrastructure as code, such as Terraform.

• Experienced with version control tools, such as Github

Senior Data Engineer