Technical Lead

Job Title

Databricks Technical Lead

Relevant Experience

(in Yrs)

10+

Must Have Technical/Functional Skills

Certification: Databricks certified Data Engineer Professional, SQL, Azure data Engineer, Python

Primary Skillset:

• Apache Spark: Strong understanding and hands-on experience with Apache Spark, including Spark SQL, Spark Streaming.

• Databricks: Proficiency in using Azure Databricks, including setting up and managing clusters, notebooks, and jobs.

• Python and SQL: Strong programming skills in either Python and SQL as these are the primary languages used for writing Spark applications.

• Data Processing and Analysis: Experience in designing and implementing data processing and analysis pipelines using Spark and Databricks.

• Distributed Computing: Knowledge of distributed computing concepts and experience with distributed computing frameworks like Spark.

• Cloud Platform: Familiarity with Azure cloud platform and its services, including Azure Data Factory, Azure access, Azure Storage, Azure Data Lake, and Azure SQL Database.

• Data Engineering: Understanding of data engineering concepts and experience with ETL processes, data modeling, and data warehousing.

• DevOps: Experience with CI/CD pipelines, version control systems (e.g., Git), and automated deployment of Spark applications.

• Performance Optimization: Ability to optimize Spark applications for performance and scalability, including tuning Spark configurations and leveraging Spark optimizations.

• Data Security and Governance: Understanding of data security and governance practices, including data encryption, access controls, and compliance regulations.

• Monitoring and Troubleshooting: Proficiency in monitoring and troubleshooting Spark applications, identifying and resolving performance bottlenecks or issues.

• Collaboration and Communication: Excellent communication skills and the ability to collaborate effectively with cross-functional teams, including data scientists, data engineers, and business stakeholders.

Secondary Skillset:

• Machine Learning: Familiarity with machine learning concepts and experience with implementing machine learning algorithms using Spark MLlib or other machine learning frameworks.

• Data Visualization: Kn owledge of data visualization tools and libraries like Matplotlib, Plotly, Power BI or Tableau for presenting insights derived from data analysis.