Senior Data Engineer

As a Senior Data Engineer, you will play an instrumental role in advancing our core machine learning solution. Your technical expertise and leadership skills will be leveraged to drive the development and deployment of high-quality, scalable data pipelines and products. You will work alongside scientists and engineers in a collaborative environment, contributing features and data pipelines to power our core recommender systems and our DaVinci Personalization product. This is an opportunity to work end-to-end on a large-scale machine-learning system that touches millions of customers, and a chance to continuously learn and help improve our solution as the field evolves.

This role will be reporting to the Director of Engineering (AI).

Responsibilities

• Implement production data products and pipelines that are scalable, reliable, and of high quality.

• Build, maintain, and optimize our machine learning Data Lake.

• Continuously improve data infrastructure for greater scalability.

• Support the data needs of ML Engineers and Scientists for machine learning model development.

• Release features that deliver measurable and tangible business value.

Requirements

• 4+ years of data engineering experience

• Experience with large-scale data processing frameworks (we use PySpark, SQL)

• Expertise in Spark DataFrame API

• Experience with event stream data (we use Kafka)

• Strong software development skills in Python (unit testing, git, code review, CI/CD)

• Experience with cloud computing platforms and cluster configuration, optimization, and scaling (GCP)

• Experience in data storage formats (we use Parquet, Delta Lake)

• Ability to collaborate with technical partners - you'll be working closely with other teams to determine requirements for your work and to make design decisions that affect our stack

• Enjoys working in a fast-paced, goal-driven environment

The base pay range for this position is $180,000-$210,000 USD/year, which can include additional on-target commission pay/bonus. The base pay offered may vary depending on job-related knowledge, skills, and experience. Stock options and other incentive pay may be provided as part of the compensation package, in addition to a full range of medical, financial, and/or other benefits, depending on the position ultimately offered.