Data Engineering Consultant

Data Robot New York , NY 10007

Posted 3 months ago

Data Engineering Consultant

In this highly visible technical lead position, you will be responsible for providing Data Engineering leadership and support across the business. The focus will be on expanding the model building and deployment capabilities across the teams. The individual will work closely with a team of very talented data scientists and data engineers in driving requirements, development, design, and implementation of advanced platforms, tools, and systems in support of data science and machine learning initiatives.

Requirements include at least 6 years of relevant experience in industry, experience designing and developing end-to-end solutions, large scale data acquisition and transformation, and understanding of data warehouse and data lake technology.

Expertise with PySpark, and experience with the Hadoop ecosystem, AWS, Java, and SQL are also required. Prior experience with DataRobot is a plus.

Qualifications:

  • 6+ years of relevant work experience

  • Bachelor's degree in Computer Science, Engineering, or related field.

  • 2+ years experience with PySpark

  • 2+ years of production experience of building Mapreduce jobs, Spark scripts, Oozie workflows, or other Hadoop based applications.

  • Good understanding of database concepts (Oracle, MS SQL, generic SQL)

  • Good understanding of the distributed data processing.

  • Experience of creating Spark scripts either in Python or Scala.

  • Experience of diagnosing and mitigating performance issues in Spark scripts.

  • Experience of setting up and querying Hive, Presto, or Impala databases.

  • Experience of diagnosing and mitigating performance issues in Hive, Presto, or Impala queries.

  • Experience in Open systems and Cloud based applications with AWS (e.g ec2, s3)

  • Good to have experience of creating streaming solutions and reporting tools.

  • Prior experience with DataRobot is a plus.

Responsibilities:

  • Build end-to-end ETL pipelines to enable training and operationalization of machine learning models.

  • Build code for ingesting data from relational databases, NoSQL database, flat files, and message queues into big data solutions.

  • Integrating the code, produced by data scientists, into data pipelines.

  • Build code or configuration to push data from big data solutions into reporting tools and other software.

  • Diagnose and mitigate performance issues.

  • Communicate with the customer IT personnel to clarify technical details.

  • Document the implementation.

  • Assist with the installation and set up of big data solutions and DataRobot.

Individuals seeking employment at DataRobot are considered without regards to race, color, religion, national origin, age, sex, marital status, ancestry, physical or mental disability, veteran status, gender identity, or sexual orientation.

Engineering

DataRobot Engineering is a hard-working, fast moving, fun-loving team of developers who put product before pride. Our team is flexible and adaptable. We genuinely like each other and work hard to make sure that we all succeed, both for individual and company success, because we believe that one doesn't happen without the other.

Interested? Apply now!


icon no score

See how you match
to the job

Find your dream job anywhere
with the LiveCareer app.
Mobile App Icon
Download the
LiveCareer app and find
your dream job anywhere
App Store Icon Google Play Icon
lc_ad

Boost your job search productivity with our
free Chrome Extension!

lc_apply_tool GET EXTENSION

Similar Jobs

Want to see jobs matched to your resume? Upload One Now! Remove
Data Engineering Summer Internship 2020

Tapad

Posted 5 days ago

VIEW JOBS 9/13/2019 12:00:00 AM 2019-12-12T00:00 We are looking for outstanding intern engineers who are interested in the challenges of building high scale systems using current and robust technologies. Our ideal candidates are engineers who can apply Computer Science principles to solve challenging problems and expand Tapad's product offerings. We are a small, talented team of pragmatic, productive software engineers. We have created a culture of transparency and accountability, which minimizes bureaucracy to maximize developer freedom, autonomy, and creative development time. We are looking for interns who are creative and innovative thinkers. You will experience what it is like to be an engineer in a fun, technology-first startup atmosphere. You will be paired one-on-one with an engineering mentor for guidance and advice throughout the program. We are looking for candidates who meet some of the following qualifications: * Studying to finish a BS, MS, or Ph.D. in a technical major * Understanding of concurrent and parallel programming * Knowledge of algorithms and data structures * Bonus: SQL and functional programming experience, specifically Scala A day in the life as a Software Engineer Intern: * Designing, implementing and running big data pipelines that canvas over petabytes of data * Contributing to real production projects that constitute Tapad's core offering * Collaborate with your team of engineers, and contribute to product, account, and business development functions to create new products and features Technologies we use at Tapad: * Google Cloud Platform (GCP) * Scala, Cats, SBT, Play!, Akka, * Spark, Python * Kubernetes, BigTable, BigQuery * TypeScript, Angular, Node.js, Hapi.js, Postgres, MySql In this role, you will be using (don't worry, we'll teach you): * Google Cloud Platform (GCP) * Scala, SBT, Play!, Akka * Spark, Python, SQL, BigQuery Tapad Intern Benefits: * Gain valuable experience working in a cutting edge and data-driven environment using state of the art technologies * Dynamic and fast-paced well-established start-up * A designated mentor to help guide you through the 10-week internship program * Ongoing training, which will include Scala School, access to Coursera, peer-led professional development, and an abundance of resources to help guide you through your internship * Catered lunches and unlimited snacks and beverages * Leadership lunches - sit with the CEO, CTO, and Chief Product Officer, and ask them anything your heart desires * Fussball, ping pong, diversity and inclusion group, book club, and tons of other extra-curricular activities that will make you feel like part of the Tapad family About Tapad: Tapad is home to the team that cracked the code on cross-device marketing technology. Our groundbreaking, proprietary tech assimilates billions of data points to find the human relationship between smartphones, desktops, laptops, tablets, connected CTV's and game consoles. With 91.2% data accuracy confirmed by Nielsen, Tapad offers the most substantial in-market opportunity for marketers and technologies to address the ever-evolving reality of media consumption across devices. Tapad is proud to be an equal opportunity employer and will consider all qualified applicants regardless of age, sex, race, religion, national origin, sexual orientation, gender identity, marital or family status, disability, or any other legally protected status. Tapad does not accept resumes from unsolicited search firms nor recruiters. In no event shall fees be paid to any unsolicited search firms nor recruiters, regardless of whether the candidate is made an offer or accepts a placement at Tapad. All resumes received through any channels will be considered the sole property of Tapad. Tapad New York NY

Data Engineering Consultant

Data Robot