Systems Engineer - Big Data

Hudson River Trading (HRT) is looking for Systems Engineers to join our growing Research & Development team. This team builds and maintains an exceptionally large and growing distributed compute cluster, many-PB-scale storage systems, operating systems, automation software, and development tools.

We are seeking a highly skilled engineer with expertise in Linux administration and experience with Big Data technologies, particularly Kafka, Trino, and Elasticsearch. As a Big Data Systems Engineer, you will play a critical role in ensuring our data infrastructure is robust, scalable, and reliable. You will work closely with our engineering and research teams to design, deploy, and maintain systems that support our research and trading activities.

Responsibilities

• Trino Administration: Set up and manage Trino clusters for large-scale data analytics. Optimize and troubleshoot query performance and ensure data integrity across the system

• Kafka Administration: Design, deploy, and maintain Apache Kafka clusters. Implement and monitor Kafka-based data pipelines to ensure efficient and reliable data flow

• Elasticsearch Administration: Configure and maintain Elasticsearch clusters on managed compute platforms such as Kubernetes

• System Monitoring: Develop and implement monitoring solutions to ensure high system uptime and reliability; utilize tools to detect and resolve issues proactively

• Linux Administration: Maintain and manage a large-scale Linux-based infrastructure; ensure system stability, security, and performance through best practices in system administration

• Collaboration: Work closely with development and research teams to understand their requirements and provide robust data infrastructure solutions, participating in cross-functional projects as needed

• Documentation: Create and maintain comprehensive documentation for system architecture, processes, and best practices

Skills

• Bachelor's degree in computer science, information technology, or a related field

• Minimum of 3-5 years of experience in systems engineering, with a focus on Linux administration

• Experience with large scale, big data query engines and technologies such as Trino, Presto, Iceberg, and Hive

• Experience with operating and managing large scale Kafka clusters, including design, deployment, and capacity management

• Experience with operating and managing large scale Elasticsearch clusters

• Experience with container orchestration platforms and technologies such as Kubernetes, Terraform, and Helm

• Proficiency in scripting languages such as Python or Bash

• Strong problem-solving skills and the ability to work in a fast-paced environment

• Excellent communication and collaboration skills

Culture

Hudson River Trading (HRT) brings a scientific approach to trading financial products. We have built one of the world's most sophisticated computing environments for research and development. Our researchers are at the forefront of innovation in the world of algorithmic trading.

At HRT we welcome a variety of expertise: mathematics and computer science, physics and engineering, media and tech. We're a community of self-starters who are motivated by the excitement of being at the cutting edge of automation in every part of our organization-from trading, to business operations, to recruiting and beyond. We value openness and transparency, and celebrate great ideas from HRT veterans and new hires alike. At HRT we're friends and colleagues - whether we are sharing a meal, playing the latest board game, or writing elegant code. We embrace a culture of togetherness that extends far beyond the walls of our office.

Feel like you belong at HRT? Our goal is to find the best people and bring them together to do great work in a place where everyone is valued. HRT is proud of our diverse staff; we have offices all over the globe and benefit from our varied and unique perspectives. HRT is an equal opportunity employer; so whoever you are we'd love to get to know you.