Tesla Palo Alto , CA 94306
Posted 6 days ago
Tesla's Supercomputing/AI infrastructure team works directly with the high-performance computing and machine learning infrastructure on which our ML algorithms run; this includes virtual simulations, Autopilot hardware, silicon design, and Dojo. With the rapidly-growing need for more data and optimized compute resources, cluster builds are getting larger and increasingly complex. Continued development/automation of deployment, monitoring, self-healing and alerting processes is imperative to the success of our engineering groups. As the scope and impact of our Full-Self-Driving (FSD) & Robotaxi efforts continue to scale, so does the value of this team and its work.
As a Site Reliability Engineer, you will be responsible for maintaining and improving our infrastructure to ensure engineering teams across Autopilot/AI & Dojo have the necessary tools and resources to be productive. This includes managing/operating our HPC clusters, monitoring compute/GPU/network metrics, Linux troubleshooting & performance tuning, and collaborating with our Data Center team to coordinate the smooth operation of hundreds of servers & bring up new GPU capacity. Your work will directly facilitate neural network training at scale, streamline FSD development, and enable Dojo to become the most powerful supercomputer to date.
Support the AI/ML cluster infrastructure on both GPU and Dojo platforms, focusing on systems automation, configuration management and deployment at scale
Improve our cluster health monitoring and auto-recovery pipeline
Work with users on debugging application performance issues
Work with hardware and storage vendors to tune and optimize our servers, storage and network
Write Ansible playbooks for configuration management
Performance tuning and OS provisioning on Linux systems
Manage HPC clusters, workloads and applications
Automation and systems engineering in Python, Golang or Bash/Shell
Participate in 24x7 on-call rotation
Proficiency in Python, Golang and/or Bash
Proficiency with Linux fundamentals and performance optimizations (Ubuntu/RHEL OS)
Demonstrable knowledge of TCP/IP, IPoIB, Linux operating system internals, filesystems, disk/storage technologies and storage protocols
Experience collaborating with network and data center teams for large scale cluster builds
Experience with configuration management software (Ansible, etc.), systems monitoring & alerting (Prometheus, Grafana, Telegraf, Splunk, etc.), and/or administering HPC workload managers
Experience with high-throughput low-latency networks, GPU-based computing systems, and/or high performance storage systems
Experience with Slurm, LSF and storage management of distributed parallel file systems a plus
Bachelor's Degree in Computer Science, Computer Engineering, Electrical Engineering, Physics or proof of exceptional skills in related field
3+ years of additional equivalent experience or evidence of exceptional ability related to the position
Tesla