Developer

Skill: AWS + Databricks Developer

• AWS, DataBricks, Python, Pyspark

AWS:

• Strong proficiency in Python.

• Experienced in Building S3 file systems via code pipelines.

• Experienced in Creating and using IAM roles.

• Experienced in building and managing Key Management Services.

• Experienced in creating and using AWS transfer Family and Storage Gateway mechanism.

• Experienced in building and using CloudFormation templates.

• Experienced in standing up and maintaining EC2 instances and Lambda services.

• Experienced in standing up Kubernetes clusters and maintenance.

• Previous experience with data visualization tools (Power BI, Tableau).

• Strong experience with GIT, and CI/CD Pipelines.

Databricks:

• Extensive experience in Databricks technology.

• Extensive experience in data mapping, data architecture, and data modeling on Databricks.

• Extensive experience in Design, implementation, and maintenance of data pipelines using Python, pyspark on Databricks.

• Strong proficiency in Python, and PySpark, able to write and execute complex queries to perform curation and build views required by end users (single and multi-dimensional).

• Extensive experience in Databricks Data engineering (Job Runs, Data Ingestion and Delta Live Tables).

• Strong experience in setting up and maintaining SQL Warehouses, SQL Editor and alerts.

• Strong experience in setting up and maintaining workspaces, catalogs, workflows, and Compute.

• Experienced with All-purpose compute, Job computes, SQL Warehouse, Vector search, Pools, and Policies.

• Experienced in setting up and maintaining the Unity catalog.

• Experienced in creating and maintaining Databricks schema, objects, Notebooks, scheduler, and Job clusters.

• Strong experience in building Notebooks with complex code structures.

• Strong experience in building and maintaining the Autoloader process.

• Strong experience in the usage of Table ACL & Row and Column Level Security With Unity Catalog.

• Strong experience in Data Ingestion, Storage, Harmonization, and curation.

• Proven experience in performance and tuning to ensure jobs are running at optimal levels and no performance bottleneck.

• Experienced in data ingestion using various methods API, Direct database reads, AWS Cross account data copy, third party vendor data using FTP process etc.,

• Proven experience in debugging failed jobs.

#LI-NS2