Staff Software Engineer, Data PlatformNear Real Time Data Collection

Disney Bristol , CT 06010

Posted 5 months ago

Job Description:

We have created a new Big Data Platforms group within Disney Direct-To-Consumer and International (DTCI) technology with the skills, drive, and passion to innovate, create, and succeed in enabling a Direct to Consumer Strategy for ESPN, Disney and ABC products. We are here to disrupt and start a cultural revolution that can lead a revolution in the application of data and analytics across The Walt Disney Company, focused on Content Personalization/Recommendation, Deep User Understanding, and Audience Segmentation for Linear to Digital Ad Sales, and Analytics. We need an experienced Staff Software Engineer who drive multiple data initiatives applying innovative architecture that can scale in the cloud. We are looking for a creative and talented individual who loves to design a scalable platform which scale at peta-byte level and extract value from both structured and unstructured real-time data. Specifically, we are looking for a technology leader to build a highly scalable and extensible Big Data platform which enables real time collection, storage, modeling, and analysis of massive data sets from numerous channels. You will also build a self-serve machine learning (and deep learning) pipeline for multiple data scientists develop, test, deploy, a/b test models on top of the data platform you are responsible for. You must be self-driven to continuously evaluate new technologies, innovate and deliver solutions for business-critical applications with little to no oversight from management team.

The internet-scale platforms that you design and build will be a core assets in the delivering the highest quality content to over 150MM+ consumers on monthly basis. This is an opportunity to fundamentally evolve how DTCI delivers content and monetizes our audiences.

  • Build cool things Build software across our entire cutting-edge data platform, including near real-time data collection, processing, storage, and serving real-time analytics and other real-time use cases such as recommender systems & segmentation.

  • Lead and coach Mentor other sr. engineers by developing re-usable frameworks. Review design and code produced by other engineers working across the organization

  • Harness curiosity Change how we think, act, and utilize our data by performing exploratory and quantitative analytics, data mining, and discovery.

  • Innovate and inspire Think of new ways to help make our data platform more scalable, resilient and reliable and then work across our team to put your ideas into action.

  • Think at scale

  • Lead the transformation of a peta-byte scale batch based processing platform to a near real-time streaming platform using technologies such as Apache Kafka, Cassandra, Spark and other open source frameworks.
  • Have pride Ensure performance isn't our weakness by implementing and refining robust data processing, REST services, RPC (in an out of HTTP), and caching technologies.

  • Grow with us Help us stay ahead of the curve by working closely with data architects, stream processing specialists, API developers, our DevOps team, and analysts to design systems which can scale elastically in ways which make other groups jealous.

  • ML First

  • Provide expert level advice to data scientists, data engineers, and operations to deliver high quality analytics via machine learning and deep learning via data pipelines and APIs.
  • Build and Support Embrace the DevOps mentality to build, deploy and support applications in cloud with minimal help from other teams.

Job Type

Full Time

Alternate Location-State/Region

CA

Segment

Direct-to-Consumer and International

Category

Data

Basic Qualifications

  • Not your first rodeo Have 10+ years of experience developing with a mix of languages (Java, Scala etc.) and open source frameworks to implement data collection, processing, and serving technologies in near-time basis.

  • Data and API ninja You are also very handy with big data framework such as Hadoop & Apache Spark, No-SQL systems such as Cassandra or DynamoDB, Streaming technologies such as Apache Kafka; Understand reactive programming and dependency injection such as Spring to develop REST services.

  • Have a technology toolbox Hands on experience with newer technologies relevant to the data space such as Spark, Kafka, Apache Druid (or any other OLAP databases).

  • Cloud First

  • Plenty of experience with developing and deploying in a cloud native environment preferably AWS cloud.
  • Embrace ML Work with data scientists to operationalize machine learning models and build apps to make use of power of machine learning.

  • Problem solver Enjoy new and meaningful technology or business challenges which require you to think and respond quickly.

  • Passion and creativity Are passionate about data, technology, & creative innovation.

Business

Direct-to-Consumer and International

Required Education

  • Bachelor's degree or better in Computer Science or a related technical field or equivalent job experience.

Preferred Qualifications

  • Prior experience building internet scale platforms handling Peta- byte scale data, operationalizing clusters with hundreds of compute nodes in cloud environment.

  • Prior experience in building real-time data collection infrastructure including client SDKs will be a huge plus.

  • Experience in operationalizing Machine Learning workflows to scale will be a huge plus as well.

  • Experience with Content Personalization/Recommendation, Audience Segmentation for Linear to Digital Ad Sales, and/or Analytics

  • Experience with open source such as Spring, Hadoop, Spark, Kafka, Druid, Pilosa and Yarn/Kubernetes.

  • Experience in working with Data Scientists to operationalize machine learning models.

  • Proficiency with agile development methodologies shipping features every two weeks. It would be awesome if you have a robust portfolio on Github and / or open source contributions you are proud to share.

Postal Code
06010

Preferred Education

  • Masters in Computer Science or similar is preferred.

Job Description

We have created a new Big Data Platforms group within Disney Direct-To-Consumer and International (DTCI) technology with the skills, drive, and passion to innovate, create, and succeed in enabling a Direct to Consumer Strategy for ESPN, Disney and ABC products. We are here to disrupt and start a cultural revolution that can lead a revolution in the application of data and analytics across The Walt Disney Company, focused on Content Personalization/Recommendation, Deep User Understanding, and Audience Segmentation for Linear to Digital Ad Sales, and Analytics. We need an experienced Staff Software Engineer who drive multiple data initiatives applying innovative architecture that can scale in the cloud. We are looking for a creative and talented individual who loves to design a scalable platform which scale at peta-byte level and extract value from both structured and unstructured real-time data. Specifically, we are looking for a technology leader to build a highly scalable and extensible Big Data platform which enables real time collection, storage, modeling, and analysis of massive data sets from numerous channels. You will also build a self-serve machine learning (and deep learning) pipeline for multiple data scientists develop, test, deploy, a/b test models on top of the data platform you are responsible for. You must be self-driven to continuously evaluate new technologies, innovate and deliver solutions for business-critical applications with little to no oversight from management team.

The internet-scale platforms that you design and build will be a core assets in the delivering the highest quality content to over 150MM+ consumers on monthly basis. This is an opportunity to fundamentally evolve how DTCI delivers content and monetizes our audiences.

  • Build cool things Build software across our entire cutting-edge data platform, including near real-time data collection, processing, storage, and serving real-time analytics and other real-time use cases such as recommender systems & segmentation.

  • Lead and coach Mentor other sr. engineers by developing re-usable frameworks. Review design and code produced by other engineers working across the organization

  • Harness curiosity Change how we think, act, and utilize our data by performing exploratory and quantitative analytics, data mining, and discovery.

  • Innovate and inspire Think of new ways to help make our data platform more scalable, resilient and reliable and then work across our team to put your ideas into action.

  • Think at scale

  • Lead the transformation of a peta-byte scale batch based processing platform to a near real-time streaming platform using technologies such as Apache Kafka, Cassandra, Spark and other open source frameworks.
  • Have pride Ensure performance isn't our weakness by implementing and refining robust data processing, REST services, RPC (in an out of HTTP), and caching technologies.

  • Grow with us Help us stay ahead of the curve by working closely with data architects, stream processing specialists, API developers, our DevOps team, and analysts to design systems which can scale elastically in ways which make other groups jealous.

  • ML First

  • Provide expert level advice to data scientists, data engineers, and operations to deliver high quality analytics via machine learning and deep learning via data pipelines and APIs.
  • Build and Support Embrace the DevOps mentality to build, deploy and support applications in cloud with minimal help from other teams.

Basic Qualifications

  • Not your first rodeo Have 10+ years of experience developing with a mix of languages (Java, Scala etc.) and open source frameworks to implement data collection, processing, and serving technologies in near-time basis.

  • Data and API ninja You are also very handy with big data framework such as Hadoop & Apache Spark, No-SQL systems such as Cassandra or DynamoDB, Streaming technologies such as Apache Kafka; Understand reactive programming and dependency injection such as Spring to develop REST services.

  • Have a technology toolbox Hands on experience with newer technologies relevant to the data space such as Spark, Kafka, Apache Druid (or any other OLAP databases).

  • Cloud First

  • Plenty of experience with developing and deploying in a cloud native environment preferably AWS cloud.
  • Embrace ML Work with data scientists to operationalize machine learning models and build apps to make use of power of machine learning.

  • Problem solver Enjoy new and meaningful technology or business challenges which require you to think and respond quickly.

  • Passion and creativity Are passionate about data, technology, & creative innovation.

Required Education

  • Bachelor's degree or better in Computer Science or a related technical field or equivalent job experience.

Preferred Qualifications

  • Prior experience building internet scale platforms handling Peta- byte scale data, operationalizing clusters with hundreds of compute nodes in cloud environment.

  • Prior experience in building real-time data collection infrastructure including client SDKs will be a huge plus.

  • Experience in operationalizing Machine Learning workflows to scale will be a huge plus as well.

  • Experience with Content Personalization/Recommendation, Audience Segmentation for Linear to Digital Ad Sales, and/or Analytics

  • Experience with open source such as Spring, Hadoop, Spark, Kafka, Druid, Pilosa and Yarn/Kubernetes.

  • Experience in working with Data Scientists to operationalize machine learning models.

  • Proficiency with agile development methodologies shipping features every two weeks. It would be awesome if you have a robust portfolio on Github and / or open source contributions you are proud to share.

Preferred Education

  • Masters in Computer Science or similar is preferred.
icon no score

See how you match
to the job

Find your dream job anywhere
with the LiveCareer app.
Mobile App Icon
Download the
LiveCareer app and find
your dream job anywhere
App Store Icon Google Play Icon
lc_ad

Boost your job search productivity with our
free Chrome Extension!

lc_apply_tool GET EXTENSION

Similar Jobs

Want to see jobs matched to your resume? Upload One Now! Remove
Big Data Platform Senior Staff Software Engineer

Walt Disney Co.

Posted 4 weeks ago

VIEW JOBS 9/21/2019 12:00:00 AM 2019-12-20T00:00 Job Description We have created a new Big Data Platforms group within Disney's Direct-To-Consumer and International technology organization with the skills, drive, and passion to innovate, create, and succeed in enabling a Direct to Consumer Strategy for various digital products. We are here to disrupt and start a cultural revolution in the application of data and analytics across The Walt Disney Company, focused on Content Personalization/Recommendation, Deep User Understanding, and Audience Segmentation for Linear to Digital Ad Sales, and Analytics. We need an experienced Data Platform Architect who can drive multiple data initiatives applying innovative architecture that can scale in the cloud. We're looking for a creative and talented individual who loves designing scalable platforms, particularly at the peta-byte level and extract value from both structured and unstructured real-time data. More specifically, we need a technology leader to build a highly scalable and extensible Big Data platform which enables collection, storage, modeling, and analysis of massive data sets from numerous channels. Job Type Full Time Segment Direct-to-Consumer and International Category Data Basic Qualifications * Roughly 10 years of experience building large scale data platforms from Architecture all the way to implementation and support. Platform is expected to handle Peta Bytes of data in cloud environment, on a real-time manner. * Code Ninja – Must be hands-on on latest technologies such as Java, Scala, Apache Spark, Apache Kafka, Hadoop, API design and development, No-SQL databases such as Cassandra, OLAP columnar storage systems, Bit Map indexes to handle millions of consumers and thousands of attributes while allowing real-time querying/ segmentation. * Visionary - Solid understanding of software development from design and architecture to build software for future. * Have a data toolbox – Familiar with technologies relevant to the data and integration space including Hadoop, Spark, Apache Druid, Cassandra, Java, Python, and ML frameworks. * Hunger to Learn & Teach – Genuine interest to learn new cutting edge technologies and share it with rest of the engineering team to keep them up-to-date on technology trends. We love to see your public GIT or similar profiles. * Problem solver – Enjoy new and meaningful technology or business challenges which require you to think and respond quickly * Passion and creativity – Are passionate about data, technology, & creative innovation * Open source – Prefer open source technologies and build it yourself mentality, open source contribution history is highly preferred. * Team player – Enjoy working collaboratively with a talented group of people to tackle challenging business problems so we all succeed (or fail fast) as a team Business Direct-to-Consumer and International Preferred Qualifications * Experience in building large data streaming platform will be a huge plus. * Experience in operationalizing Machine Learning workflows to scale will be a huge plus as well. * Experience with Content Personalization/Recommendation, Audience Segmentation for Linear to Digital Ad Sales, and/or Analytics * Presence in open source projects will be huge plus. We love to see our social profile if any. * Working experience in Machine Learning framework such as Apache Spark MLLib, TensorFlow or similar. * 5+ years of hands-on experience in data and analytics technology, with focus on data architecture and large volume data processing. Experience with Java, Python, and/or SQL. * 5+ years of experience building, coaching and leading software professionals * 5+ years of experience working with relational databases, data services, big data, complex event processing and machine learning. * 2+ years of experience with cloud deployments, AWS experience preferred. Proficiency with linux/unix based systems. Postal Code 06010 Preferred Education * Masters in Computer Science or similar is preferred. Responsibilities * Act as the proactive and technical architect point person for DTCI Consumer Data Platforms end to end (data collection through knowledge extraction via statistical machine learning and deep learning approaches, distribution via stream, APIs and files, ad-hoc analysis, reporting & visualization). * Present and educate management team about technical direction to achieve maximum profitability by using best data management technologies while reducing overall cost of operation. * Lead and coach other software engineers by developing re-usable frameworks. Review design and code produced by other engineers. * Provide expert level advice to data scientists, data engineers, and operations to deliver high quality analytics via machine learning and deep learning via data pipelines and APIs. * Lead the transformation of a peta-byte scale batch-based processing platform to a near real-time streaming platform using technologies such as Apache Kafka, Cassandra and Spark. * Design and build efficient ETL/ELT process to move data through the data processing pipeline to meet the demands of the business use cases using Java, Open Source, and AWS Products. Build easy to re-use workflow model and take the entire team to follow the pattern to implement all ETL process to improve efficiency and reduce cost. * Optimize and automate data ingestion, data processing and distribution data from variety of sources, including click stream data, ratings data, advertising data, 3rd party sources and sources not yet identified. * Manage complex data dependencies across datasets and incremental data loading workflows. * Design and build api/stream/batch based data export mechanism to be used by other DTCI products such as AdSales, Web, App platforms. * Be a fearless leader in championing smart, scalable and flexible design * Collaborating with product management and acting as the bridge between product management, engineering teams, and customers to understand requirements and technical solutions * Help us stay ahead of the curve by working closely with data management team, data engineers, our DevOps team, and analysts to design systems which can scale overnight in ways which make other groups envy Job Description We have created a new Big Data Platforms group within Disney's Direct-To-Consumer and International technology organization with the skills, drive, and passion to innovate, create, and succeed in enabling a Direct to Consumer Strategy for various digital products. We are here to disrupt and start a cultural revolution in the application of data and analytics across The Walt Disney Company, focused on Content Personalization/Recommendation, Deep User Understanding, and Audience Segmentation for Linear to Digital Ad Sales, and Analytics. We need an experienced Data Platform Architect who can drive multiple data initiatives applying innovative architecture that can scale in the cloud. We're looking for a creative and talented individual who loves designing scalable platforms, particularly at the peta-byte level and extract value from both structured and unstructured real-time data. More specifically, we need a technology leader to build a highly scalable and extensible Big Data platform which enables collection, storage, modeling, and analysis of massive data sets from numerous channels. Basic Qualifications * Roughly 10 years of experience building large scale data platforms from Architecture all the way to implementation and support. Platform is expected to handle Peta Bytes of data in cloud environment, on a real-time manner. * Code Ninja – Must be hands-on on latest technologies such as Java, Scala, Apache Spark, Apache Kafka, Hadoop, API design and development, No-SQL databases such as Cassandra, OLAP columnar storage systems, Bit Map indexes to handle millions of consumers and thousands of attributes while allowing real-time querying/ segmentation. * Visionary - Solid understanding of software development from design and architecture to build software for future. * Have a data toolbox – Familiar with technologies relevant to the data and integration space including Hadoop, Spark, Apache Druid, Cassandra, Java, Python, and ML frameworks. * Hunger to Learn & Teach – Genuine interest to learn new cutting edge technologies and share it with rest of the engineering team to keep them up-to-date on technology trends. We love to see your public GIT or similar profiles. * Problem solver – Enjoy new and meaningful technology or business challenges which require you to think and respond quickly * Passion and creativity – Are passionate about data, technology, & creative innovation * Open source – Prefer open source technologies and build it yourself mentality, open source contribution history is highly preferred. * Team player – Enjoy working collaboratively with a talented group of people to tackle challenging business problems so we all succeed (or fail fast) as a team Preferred Qualifications * Experience in building large data streaming platform will be a huge plus. * Experience in operationalizing Machine Learning workflows to scale will be a huge plus as well. * Experience with Content Personalization/Recommendation, Audience Segmentation for Linear to Digital Ad Sales, and/or Analytics * Presence in open source projects will be huge plus. We love to see our social profile if any. * Working experience in Machine Learning framework such as Apache Spark MLLib, TensorFlow or similar. * 5+ years of hands-on experience in data and analytics technology, with focus on data architecture and large volume data processing. Experience with Java, Python, and/or SQL. * 5+ years of experience building, coaching and leading software professionals * 5+ years of experience working with relational databases, data services, big data, complex event processing and machine learning. * 2+ years of experience with cloud deployments, AWS experience preferred. Proficiency with linux/unix based systems. Preferred Education * Masters in Computer Science or similar is preferred. Responsibilities * Act as the proactive and technical architect point person for DTCI Consumer Data Platforms end to end (data collection through knowledge extraction via statistical machine learning and deep learning approaches, distribution via stream, APIs and files, ad-hoc analysis, reporting & visualization). * Present and educate management team about technical direction to achieve maximum profitability by using best data management technologies while reducing overall cost of operation. * Lead and coach other software engineers by developing re-usable frameworks. Review design and code produced by other engineers. * Provide expert level advice to data scientists, data engineers, and operations to deliver high quality analytics via machine learning and deep learning via data pipelines and APIs. * Lead the transformation of a peta-byte scale batch-based processing platform to a near real-time streaming platform using technologies such as Apache Kafka, Cassandra and Spark. * Design and build efficient ETL/ELT process to move data through the data processing pipeline to meet the demands of the business use cases using Java, Open Source, and AWS Products. Build easy to re-use workflow model and take the entire team to follow the pattern to implement all ETL process to improve efficiency and reduce cost. * Optimize and automate data ingestion, data processing and distribution data from variety of sources, including click stream data, ratings data, advertising data, 3rd party sources and sources not yet identified. * Manage complex data dependencies across datasets and incremental data loading workflows. * Design and build api/stream/batch based data export mechanism to be used by other DTCI products such as AdSales, Web, App platforms. * Be a fearless leader in championing smart, scalable and flexible design * Collaborating with product management and acting as the bridge between product management, engineering teams, and customers to understand requirements and technical solutions * Help us stay ahead of the curve by working closely with data management team, data engineers, our DevOps team, and analysts to design systems which can scale overnight in ways which make other groups envy Walt Disney Co. Bristol CT

Staff Software Engineer, Data PlatformNear Real Time Data Collection

Disney