Staff Software Engineer, Observability

Slack San Francisco , CA 94118

Posted 3 weeks ago

Staff Software Engineer, Observability

About the Team

The Monitoring team at Slack develops platforms and tools which provide insights and observability into the availability, performance, and reliability of Slack production services. We develop configuration management tools for distributed applications and infrastructure, maintain datasets for business and system analytics, and build interfaces and backend systems to answer questions and infer behavioral patterns about our users and systems. Our toolset is varied. We work with open-source observability/monitoring technologies like Elastic Stack and Prometheus, cloud providers such as AWS, and write software using a combination of Go, Python, or Java.

As part of the Monitoring team in San Francisco, you will work closely with other teams in engineering, product development, and customer experience to provide valuable insights to drive decisions and ensure a positive user experience for our Slack customers. You will also help build distributed services in an environment that processes tens of millions of data points per second with the ability to self-heal and scale up or down to meet demand. We are an inclusive team with deep empathy for our colleagues and customers. You can see the team at work here at Monitorama 2018.

Slack has a positive, diverse, and supportive culture-we look for people who are curious, inventive, and work to be a little better every single day. In our work together we aim to be smart, humble, hardworking and, above all, collaborative. If this sounds like a good fit for you, why not say hello?

About the Role

This is a staff-level engineering position based in San Francisco, California

What you will be doing

  • Build Observability tooling & infrastructure for Slack.

  • Collaborating with an engineering team to write a client library to collect traces and metrics from customer-facing systems

  • Encouraging a culture of Observability at Slack - help suss out problem areas and consult on improving visibility into our systems.

  • Prototyping tooling interfaces or building new features for engineering use cases

  • Improving auto-remediation in our telemetry infrastructure to avoid common failures

  • Teaching engineers how to use our tools to introspect their systems

  • Participating in the Monitoring on-call rotation, triaging and addressing production issues as they arise

What you should have

  • You are a strong communicator. Explaining complex technical concepts to designers, support, and other engineers is no problem for you.

  • You enjoy helping onboard new team members, mentoring, and teaching others.

  • You live for unit tests, code review, design documentation, debugging and solving problems.

  • You have a deep curiosity about how things work under the hood.

  • You are motivated by helping others succeed. When things break - and they will - you are eager and able to help fix things. You like thinking of ways to improve efficiency or bring delight to your coworkers.

  • You also know that the internet is a scary place and understand security concepts deeply and can put them into action to protect us and our users.



  • Firm grasp of computer science fundamentals: data structures, algorithms, programming languages, distributed systems, and information retrieval.

  • Bachelor's degree in Computer Science, Engineering or related field, or equivalent training, fellowship, or work experience


  • Experience with functional or imperative programming languages -- e.g., PHP, Python, Go, C, or Java (used without frameworks).

  • Experience with creating interfaces, tooling or automation to help define a path for engineers to self-service.

  • Experience deploying, operating and debugging server software on Linux at scale

Bonus Points:

  • Solid competency with ELK, Prometheus, OpenTracing, Graphite, or any other widely-used visibility tool.

  • Prior experience with or knowledge of large scale, high volume distributed systems, distributed databases and data pipelines.

  • Experience with containerization frameworks such as Kubernetes.

  • Experience using deployment automation/configuration management, especially Terraform or Chef.

  • Experience with AWS and other virtualized environments.

  • Experience with message queue services, such as Kafka.

Slack is registered as an employer in many, but not all, states. If you are not located in or able to work from a state where Slack is registered, you will not be eligible for employment.Visa sponsorship may not be available in certain remote locations.

Visa sponsorship is not available for candidates living outside the country of this position.

Slack is an Equal Opportunity Employer and participant in the U.S. Federal E-Verify program. Women, minorities, individuals with disabilities and protected veterans are encouraged to

icon no score

See how you match
to the job

Find your dream job anywhere
with the LiveCareer app.
Mobile App Icon
Download the
LiveCareer app and find
your dream job anywhere
App Store Icon Google Play Icon

Boost your job search productivity with our
free Chrome Extension!

lc_apply_tool GET EXTENSION

Similar Jobs

Want to see jobs matched to your resume? Upload One Now! Remove
Staff Software Engineer Observability


Posted 3 months ago

VIEW JOBS 7/30/2020 12:00:00 AM 2020-10-28T00:00 About Netskope Today, there's more data and users outside the enterprise than inside, causing the network perimeter as we know it to dissolve. We realized a new perimeter was needed, one that is built in the cloud and follows and protects data wherever it goes, so we started Netskope to redefine Cloud, Network and Data Security. Since 2012, we have built the market-leading cloud security company and an award-winning culture powered by hundreds of employees spread across offices in Santa Clara, San Francisco, Seattle, Bangalore, London, Melbourne, and Tokyo. Our core values are openness, honesty, and transparency, and we purposely developed our open desk layouts and large meeting spaces to support and promote partnerships, collaboration, and teamwork. From catered lunches and office celebrations to employee recognition events and social professional groups such as the Awesome Women of Netskope (AWON), we strive to keep work fun, supportive and interactive. Visit us at Netskope Careers and follow us on Twitter @Netskope and Facebook. Netskope San Francisco CA

Staff Software Engineer, Observability