SR. Scala Engineer, Database Engineering

Experfy Inc Chicago , IL 60632

Posted 3 weeks ago

As a Sr. Software Engineer for our Data Platform Engineering team you will join skilled Scala engineers and core database developers responsible for developing hosted cloud analytics infrastructure (Apache Spark-based), distributed SQL processing frameworks, proprietary data science platforms, and core database optimization. This team is responsible for building the automated, intelligent, and highly performant query planner and execution engines, RPC calls between data warehouse clusters, shared secondary cold storage, etc. This includes building new SQL features and customer-facing functionality, developing novel query optimization techniques for industry-leading performance, and building a database system that's highly parallel, efficient and fault-tolerant. This is a vital role reporting to exec leadership and senior engineering leadership.

Requirements

  • Writing Scala code with tools like Apache Spark + Apache Arrow to build a hosted, multi-cluster data warehouse for Web3
  • Developing database optimizers, query planners, query and data routing mechanisms, cluster-to-cluster communication, and workload management techniques
  • Scaling up from proof of concept to cluster scale (and eventually hundreds of clusters with hundreds of terabytes each), in terms of both infrastructure/architecture and problem structure
  • Codifying best practices for future reuse in the form of accessible, reusable patterns, templates, and code bases to facilitate meta data capturing and management
  • Managing a team of software engineers writing new code to build a bigger, better, faster, more optimized HTAP database (using Apache Spark, Apache Arrow and a wealth of other open source data tools)
  • Interacting with exec team and senior engineering leadership to define, prioritize, and ensure smooth deployments with other operational components
  • Highly engaged with industry trends within analytics domain from a data acquisition processing, engineering, management perspective
  • Understand data and analytics use cases across Web3 / blockchains
Skills & Qualifications
  • Bachelors degree in computer science or related technical field. Masters or PhD a plus.
  • 6+ years experience engineering software and data platforms / enterprise-scale data warehouses, preferably with knowledge of open source Apache stack (especially Apache Spark, Apache Arrow, and others)
  • 3+ years experience with Scala and Apache Spark
  • A track record of recruiting and leading technical teams in a demanding talent market
  • Rock solid engineering fundamentals; query planning, optimizing and distributed data warehouse systems experience is preferred but not required
  • Nice to have: Knowledge of blockchain indexing, web3 compute paradigms, Proofs and consensus mechanisms is a strong plus but not required
  • Experience with rapid development cycles in a web-based environment
  • Strong scripting and test automation knowledge
  • Nice to have: Passionate about Web3, blockchain, decentralization, and a base understanding of how data/analytics plays into this
icon no score

See how you match
to the job

Find your dream job anywhere
with the LiveCareer app.
Mobile App Icon
Download the
LiveCareer app and find
your dream job anywhere
App Store Icon Google Play Icon
lc_ad

Boost your job search productivity with our
free Chrome Extension!

lc_apply_tool GET EXTENSION

Similar Jobs

Want to see jobs matched to your resume? Upload One Now! Remove
Sr Staff Engineer Data Engineering Spark/Python/Scala

Early Warning Services, LLC

Posted 3 weeks ago

VIEW JOBS 9/7/2022 12:00:00 AM 2022-12-06T00:00 <p>Come build the next-gen fintech at Zelle®, where we're relentlessly focused on empowering prosperity in all its forms.</p><p>From fast money movement for over 100 million people who can access Zelle® directly through their banking app to new account opening and beyond - we make a difference in the lives of consumers and businesses every day and enable them to live their best financial lives. And we're only getting started.</p><p>With a new, state-of-the-art HQ in Scottsdale, AZ - plus growing presences in San Francisco, New York City, and Chicago - we're entering our next big chapter.</p><p>Make your mark on what's next in fintech with Zelle®.</p><p>Overall Purpose</p><p>This position is a key role in the development, test, and deployment of complex solutions.</p><p>Essential Functions</p><ul><li><p>Partners with software engineering, product, and architecture to shape engineering approaches, share knowledge and experience.</p></li><li><p>Own the data and technical strategy for broad or complex requirements with insightful and forward-looking approaches that go beyond the direct team and solve large open-ended problems.</p></li><li><p>Responsible for department-wide design, patterns and code approaches.</p></li><li><p>Reviews and validates effectiveness of code output from multiple teams.</p></li><li><p>Accountable for resolving technical conflict within and between multiple teams.</p></li><li><p>Drive all aspects of technical architecture, design, prototyping and implementation in support of both product needs as well as overall technology and data strategy.</p></li><li><p>Represent engineering in cross-functional team sessions and able to present sound and thoughtful arguments to persuade others. Adapts to the situation and can draw from a range of strategies to influence people in a way that results in agreement or behavior change</p></li><li><p>Collaborate and partner with product managers, designers, and other engineering groups to conceptualize, build new features and create product descriptions.</p></li><li><p>Actively own features or systems and define their long-term health, while also improving the health of surrounding systems.</p></li><li><p>Assist Support and Operations teams in identifying and quickly resolving production issues.</p></li><li><p>Develop and implement tests for ensuring the quality, performance, and scalability of our application.</p></li><li><p>Develop and mentor other engineers. May serve as a technical leader for cross-functional projects.</p></li><li><p>Actively seeks out ways to improve engineering standards, tooling, and processes.</p></li><li><p>Supporting the company's commitment to risk management and protecting the integrity and confidentiality of systems and data.</p></li></ul><p>Minimum Qualifications</p><ul><li><p>Education and/or experience typically obtained through a Bachelor's degree in computer science or related technical field.</p></li><li><p>Twelve or more years of relevant related experience.</p></li><li><p>Nine or more years of experience in the development of complex database management systems, Business Intelligence solution, AI/ML Model development &amp; Oprationization (includes feature engineering, model optimization, model operationization), distributed systems, SaaS, cloud solutions, micro services.</p></li><li><p>Hands-on Docker experience.</p></li><li><p>Two or more years of experience in the development of end to end data management platforms - data modeling, data goveranace, BI Reporting, AI/ML model life cycle management.</p></li><li><p>Five or more years of work experience with ETL development (using Ab Initio, Talend, Informatica or other industry proven tools), BI Reporting design &amp; development (using Tableu or Business Objects or Griffana or other industry tools).</p></li><li><p>One or more years ML Development (using Python, PySpark, Spark), ML Model development framework (using PyCharm, SciKit, Sage Maker, Tensorflow).</p></li><li><p>Demonstrated experience in delivering business-critical systems to the market.</p></li><li><p>Ability to influence and work in a collaborative team environment across multiple departments.</p></li><li><p>Experience designing/developing scalable systems.</p></li><li><p>Experience with event-driven architecture and messaging frameworks (Pub/Sub, Kafka, RabbitMQ, etc).</p></li><li><p>Working experience with cloud infrastructure (Google Cloud Platform, AWS, Azure, etc).</p></li><li><p>Knowledge of mature engineering practices (CI/CD, testing, secure coding, etc).</p></li><li><p>Knowledge of Software Development Lifecycle (SDLC) best practices, software development methodologies (Agile, Scrum, LEAN etc) and DevOps practices.</p></li><li><p>Background and drug screen.</p></li></ul><p>Preferred Qualifications</p><ul><li><p>Big Data Platforms (Cloudera, S3)</p></li><li><p>Database platforms (Oracle, SQL Server)</p></li><li><p>Strong computer language experience (Python, Spark, and/or Scala)</p></li><li><p>Knowledge of Aerospike, Scality S3, Elastic Search</p></li><li><p>Monitoring and Alerting systems experience (AppDynamics)</p></li><li><p>Knowledge of ACH/EFT</p></li><li><p>Knowledge of real time payment networks (RTP, FedNow)</p></li><li><p>FinTech experience</p></li><li><p>Kubernetes experience</p></li></ul><p>#LI-HG1</p><p>#Dice</p><p>Physical Requirements</p><p>Working conditions consist of a normal office environment. Work is primarily sedentary and requires extensive use of a computer and involves sitting for periods of approximately four hours. Work may require occasional standing, walking, kneeling and reaching. Must be able to lift 10 pounds occasionally and/or negligible amount of force frequently. Requires visual acuity and dexterity to view, prepare, and manipulate documents and office equipment including personal computers. Requires the ability to communicate with internal and/or external customers.</p><p>Employee must be able to perform essential functions and physical requirements of position with or without reasonable accommodation.</p><p>The above job description is not intended to be an all-inclusive list of duties and standards of the position.</p><p>Early Warning Services is an affirmative action and equal opportunity employer.</p><p>Some of the Ways We Prioritize Your Health and Happiness</p><ul><li>Healthcare Coverage</li></ul><p>- Competitive medical, dental, and vision plans as well as pre-tax savings through flexible spending accounts (FSA) for commuting, health &amp; dependent care expenses.</p><ul><li>401(k) Retirement Plan</li></ul><p>- Featuring a 100% Company Safe Harbor Match on your first 5% deferral immediately upon eligibility.</p><ul><li>Parental Leave</li></ul><ul><li>12 weeks paid leave plus access to Maven Maternity</li></ul><ul><li>Paid Time Off</li></ul><ul><li>Unlimited Time Off* starting on Day 1, plus 11 paid company holidays</li></ul><ul><li>Employee Assistance Program</li></ul><ul><li>Resources and tools to help you and your family find a solution to whatever life throws at you.</li></ul><p>And SO much more! We continue to enhance our program, so be sure to check our Benefits page here for the latest.</p><ul><li>Unlimited Time Off available to regular, full-time exempt roles only. Our team can share more during the interview process!</li></ul><p>Early Warning Services is an Equal Opportunity employer. All qualified applicants and employees will receive consideration for employment without regard to race, color, religion, religious creed, ethnicity, ancestry, national origin, sex, pregnancy, age, marital status, sexual orientation, gender identity or expression, citizenship, ancestry, disability, veteran status, genetic information or any other characteristic protected by law.</p><br> Early Warning Services, LLC Chicago IL

SR. Scala Engineer, Database Engineering

Experfy Inc