SR. Scala Engineer, Database Engineering

Experfy Inc Chicago , IL 60632

Posted 1 week ago

As a Sr. Software Engineer for our Data Platform Engineering team you will join skilled Scala engineers and core database developers responsible for developing hosted cloud analytics infrastructure (Apache Spark-based), distributed SQL processing frameworks, proprietary data science platforms, and core database optimization. This team is responsible for building the automated, intelligent, and highly performant query planner and execution engines, RPC calls between data warehouse clusters, shared secondary cold storage, etc. This includes building new SQL features and customer-facing functionality, developing novel query optimization techniques for industry-leading performance, and building a database system that's highly parallel, efficient and fault-tolerant. This is a vital role reporting to exec leadership and senior engineering leadership.


  • Writing Scala code with tools like Apache Spark + Apache Arrow to build a hosted, multi-cluster data warehouse for Web3
  • Developing database optimizers, query planners, query and data routing mechanisms, cluster-to-cluster communication, and workload management techniques
  • Scaling up from proof of concept to cluster scale (and eventually hundreds of clusters with hundreds of terabytes each), in terms of both infrastructure/architecture and problem structure
  • Codifying best practices for future reuse in the form of accessible, reusable patterns, templates, and code bases to facilitate meta data capturing and management
  • Managing a team of software engineers writing new code to build a bigger, better, faster, more optimized HTAP database (using Apache Spark, Apache Arrow and a wealth of other open source data tools)
  • Interacting with exec team and senior engineering leadership to define, prioritize, and ensure smooth deployments with other operational components
  • Highly engaged with industry trends within analytics domain from a data acquisition processing, engineering, management perspective
  • Understand data and analytics use cases across Web3 / blockchains
Skills & Qualifications
  • Bachelors degree in computer science or related technical field. Masters or PhD a plus.
  • 6+ years experience engineering software and data platforms / enterprise-scale data warehouses, preferably with knowledge of open source Apache stack (especially Apache Spark, Apache Arrow, and others)
  • 3+ years experience with Scala and Apache Spark
  • A track record of recruiting and leading technical teams in a demanding talent market
  • Rock solid engineering fundamentals; query planning, optimizing and distributed data warehouse systems experience is preferred but not required
  • Nice to have: Knowledge of blockchain indexing, web3 compute paradigms, Proofs and consensus mechanisms is a strong plus but not required
  • Experience with rapid development cycles in a web-based environment
  • Strong scripting and test automation knowledge
  • Nice to have: Passionate about Web3, blockchain, decentralization, and a base understanding of how data/analytics plays into this
icon no score

See how you match
to the job

Find your dream job anywhere
with the LiveCareer app.
Mobile App Icon
Download the
LiveCareer app and find
your dream job anywhere
App Store Icon Google Play Icon

Boost your job search productivity with our
free Chrome Extension!

lc_apply_tool GET EXTENSION

Similar Jobs

Want to see jobs matched to your resume? Upload One Now! Remove
Database Engineer Postgres/Mongodb

Vail Systems, Inc.

Posted 3 days ago

VIEW JOBS 2/2/2023 12:00:00 AM 2023-05-03T00:00 <p>Who You Are</p><p>Interested in managing opensource relational and document-oriented databases and the code that interacts with them?</p><p>As a Database Engineer you will resolve any inefficient queries that appear in the logs and initiate updates to the latest database version. You will be responsible for performing data transfers between Postgres/MySQL, Microsoft SQL Server, and MongoDB databases. You will collaborate with other highly technical engineering staff to accomplish project tasks.</p><p>#Dice</p><p>What You'll Do</p><ul><li><p>Maintain the physical data model, queries, and service interface for the loading, querying, and maintenance of data in Postgres database and other data sources. It is essential to have coding experience to be able to handle ETL processes and data validation, with either proprietary database languages or other programming languages, such as Java, Python, Scala, or Perl.</p></li><li><p>Complete version upgrades, patching, migrations, seeding, testing, and performance monitoring.</p></li><li><p>Optimize performance, design indexing strategies, identify patterns in MongoDB document schemas, and manage security and availability of MongoDB clusters.</p></li><li><p>Create and execute backup design and disaster recovery plans.</p></li><li><p>Work with project and technical managers for project planning, predict storage growth, perform database version upgrade and storage expansion.</p></li><li><p>Define, develop, document, communicate and implement standards, processes and procedures for the Database Administration Team.</p></li><li><p>Export data out to be loaded into our data warehouse. Improve ETL processes that interact with Microsoft SQL Servers. Write scripts and schedule jobs.</p></li></ul><p>Qualifications - We encourage you to apply if you think your experience may be a match, even if you do not meet all of the qualifications.</p><ul><li><p>2+ years of experience as a Database Administrator.</p></li><li><p>A Bachelor's degree in Computer Science, Computer Engineering, Data Science, or another related field.</p></li><li><p>Proficiency administering PostgreSQL or MySQL databases.</p></li><li><p>Understanding of SQL tuning and optimization of query plans.</p></li><li><p>Linux shell scripting skills and experience with production Linux environments.</p></li><li><p>Experience working with software engineers in a highly technical environment.</p></li><li><p>Database developing and programming skills with either SQL, Java, C++, Python, Scala, or Perl.</p></li><li><p>Excellent verbal and written communication skills.</p></li><li><p>Experience with MongoDB replica set configuration and management, preferred.</p></li></ul><p>Who We Are</p><p>At Vail, we believe in the unique power of voice interactions to create more expressive, more intimate, and more efficient interpersonal interactions. Using Vail technology, we make millions of voice interactions better every day. We process around 10% of all toll-free call traffic in the U.S; 1 in 10 times when someone calls a customer support hotline Vail is routing or interacting with that call.</p><p>We are rapidly growing across multiple dimensions, including our customer base, the scope of products we offer, and the size of our team. Now is the right time for a strong candidate to join and grow with us. We have a supportive culture where employees are encouraged to achieve both personal and team goals because we believe growth leads to both business impact and personal fulfillment.</p><p>We offer competitive compensation and affordable benefits with flexibility and choice to meet individual and family (including Domestic Partnerships) needs including:</p><ul><li><p>Multiple medical, dental, and vision plan options</p></li><li><p>Company-paid life insurance, short and long-term disability</p></li><li><p>401(k) savings plan with company match (50% on first 6% of employee contribution)</p></li><li><p>31 days total annual PTO</p></li><li><p>Annual Bonus Program</p></li><li><p>Paid maternity and paternity leave</p></li><li><p>Relocation allowance</p></li><li><p>Employee referral bonus</p></li><li><p>Gym membership</p></li><li><p>Technical and Professional Development stipend</p></li></ul><p>We are striving to implement and sustain an inclusive and equitable work environment for all employees by sourcing underrepresented groups and continually empowering those individuals within our organization to further enrich Vail's communication solutions. We recognize that equitable and unique individuals benefit our teams' problem-solving, innovation and development efforts.</p><p>COVID-19</p><p>Vail's offices are located in Deerfield, IL, and Chicago, IL. Interviews and onboarding may be conducted remotely. We are observing a hybrid work arrangement giving employees flexibility to come into the office based on personal, team, and business needs.</p><br> Vail Systems, Inc. Chicago IL

SR. Scala Engineer, Database Engineering

Experfy Inc