Principal Data Engineer

About Us:

As a not-for-profit organization, Mass General Brigham is committed to supporting patient care, research, teaching, and service to the community by leading innovation across our system. Founded by Brigham and Women's Hospital and Massachusetts General Hospital, Mass General Brigham supports a complete continuum of care including community and specialty hospitals, a managed care organization, a physician network, community health centers, home care and other health-related entities. Several of our hospitals are teaching affiliates of Harvard Medical School, and our system is a national leader in biomedical research.

We're focused on a people-first culture for our system's patients and our professional family. That's why we provide our employees with more ways to achieve their potential. Mass General Brigham is committed to aligning our employees' personal aspirations with projects that match their capabilities and creating a culture that empowers our managers to become trusted mentors. We support each member of our team to own their personal development-and we recognize success at every step.

Our employees use the Mass General Brigham values to govern decisions, actions and behaviors. These values guide how we get our work done: Patients, Affordability, Accountability & Service Commitment, Decisiveness, Innovation & Thoughtful Risk; and how we treat each other: Diversity & Inclusion, Integrity & Respect, Learning, Continuous Improvement & Personal Growth, Teamwork & Collaboration.

GENERAL SUMMARY / OVERVIEW

As a not-for-profit organization, Mass General Brigham is committed to supporting patient care, research, teaching, and service to our community by leading innovation across our system. Founded by Brigham and Women's Hospital and Massachusetts General Hospital, Mass General Brigham supports a complete continuum of care including community and specialty hospitals, a managed care organization, a physician network, community health centers, home care and other health-related entities. Several of our hospitals are teaching affiliates of Harvard Medical School, and our system is a national leader in biomedical research.

We seek employees who bring passion, are highly collaborative, and thrive in a dynamic, diverse, and inclusive team. We work in a highly agile environment with a focus on quality and achieving real impact with our work. We are excited to have a team with a rich mix of experiences, with people who bring innovative approaches to their work, an excitement for looking at things from new angles, and a spirit of continuous improvement. Finally, we seek employees who embrace and live our core values of respect, recognition, communication, commitment, trust, innovation, and service. If this sounds like you, we invite you to apply.

Mass General Brigham's main office is in Somerville, MA. We are currently working as a fully-remote team due to COVID-19 and encourage applications from across the US as telecommuting becomes part of our new normal. We are committed to building an engaged, inclusive community and offer a variety of opportunities to come together virtually and facilitate learning and collaboration.

MGB Overview

Mass General Brigham's board and system-wide executive leadership are making significant investments in enterprise data and digital health, enabling innovation and transformation in the delivery of health care, research and discovery. By building out core digital and data capabilities, we are supporting the achievement of Mass General Brigham's mission to drive growth and innovation in inpatient, ambulatory, and digital care. We are at the start of this journey and we are looking for talented individuals from across the nation to join us in transforming healthcare and in achieving the expected outcomes of this strategy: improved patient care, quality of care, patient engagement, and efficiency in the delivery of healthcare with excellence.

In the achievement of these goals, Mass General Brigham's data and analytic teams are working together to build a system-wide data ecosystem. We are both leveraging current assets and building new, integrated solutions to provide a holistic set of industry-leading data and analytic capabilities, supporting Artificial Intelligence (AI), Machine Learning (ML), and real-time insights. Teams from Data and Analytics, Research Analytics, Clinical Data Science, and Information Systems are collaborating to build this new data and analytic ecosystem. To ensure success of this effort, Mass General Brigham is standing up a new Data and Analytics operating model with a goal of driving a level of standardization for data and analytic operations across the teams involved.

The value of investments in data and analytics were proven out during our response to COVID-19 as these investments were leveraged to stand up new reporting and analytic capabilities to support executive and clinical leadership on managing the pandemic. Our broad data and analytic community has come together across Mass General Brigham 's 14 healthcare sites with shared urgency, delivering analytic tools to our system that drove timely insights in both clinical and operational domains, resulting in effective delivery of critical care and in organizational efficiencies during a time of reduced resources

Team Overview

The Mass General Brigham Data Lake Team plays a pivotal role in harnessing the power of data to drive innovation, enhance patient care, and advance research initiatives within the Mass General Brigham healthcare system. As a leading integrated health system, Mass General Brigham relies on a robust data infrastructure to make informed decisions and optimize healthcare delivery.

The mission of the Data Lake Team is to establish and maintain a centralized and scalable data platform - Data Lake, that consolidates diverse datasets from various sources across the healthcare system. This initiative aims to break down data silos, foster collaboration, and empower stakeholders with timely and accurate information.

The Data Lake Team collaborates closely with various stakeholders, including clinicians, researchers, administrators, and IT professionals. Cross-functional partnerships ensure that the Data Lake aligns with the evolving needs of the Mass General Brigham community.

Embracing a culture of innovation, the team is committed to exploring emerging technologies and methodologies to enhance the capabilities of the Data Lake, Lake house and Enterprise Data Warehouse. Continuous improvement initiatives are undertaken to refine processes, optimize performance, and stay at the forefront of healthcare data management.

The Mass General Brigham Data Lake Team is at the forefront of revolutionizing healthcare through effective data management. By fostering collaboration, ensuring data integrity, and facilitating advanced analytics, the team contributes significantly to Mass General Brigham's mission of providing high-quality patient care and advancing medical knowledge.

Position Overview

We are seeking a highly skilled and experienced Principal Data Engineer to join our dynamic team. As a key member of our data engineering team, you will play a crucial role in designing, developing, and maintaining our data infrastructure and platform integration solutions. The Principal Data Engineer will be responsible for advancing our data engineering capabilities, ensuring data quality, and contributing to the success of data-driven initiatives.

Principal Duties and Responsibilities:

Infrastructure, Architecture and Design:

• Design end-to-end data solutions on the Azure data platform, considering Scalability, Security, Compliance and Performance.

• Contribute to technical foundation of the Data Lake platform, expanding and optimizing the data ecosystem and pipeline architecture.

• Provide expertise in selecting and implementing appropriate data platform services (e.g. Azure, Databricks, Snowflake) for various data processing and storage needs.

• Collaborate with cross-functional teams to integrate data solutions with existing MGB data platform systems and applications, integrate new data management technologies aligned with the latest vendor products & capabilities (Microsoft Azure and Fabric, Databricks, Snowflake, Collibra, etc.) and software engineering tools into existing structures.

• Design, develop, construct, test, and maintain Data Lake architectures and large-scale data processing systems.

• Support big data, data lake, lakehouse ecosystem related tool selection and POC analysis.

Data Pipeline Development:

• Gather and process raw data at scale including large complex data sets, meeting functional/non-functional business requirements (using ADF, Databricks, Python, Pyspark, scripts, REST API calls, SQL Queries, etc.).

• Develop data set processes for data modeling, mining, and production.

• Create and maintain optimal data pipeline architecture on cloud-based platforms (e.g., Azure) and relational data systems (SQL Server, SSIS, Snowflakes).

Cross-Functional Collaboration:

• Work on cross-functional teams delivering enterprise solutions for internal and external clients.

• Collaborate with Software Developers, Database Architects, Data Analysts, and Data Scientists on data initiatives.

• Support stakeholders, including the Management team, Product owners, and Architecture teams, in addressing data-related technical issues and fulfilling data infrastructure needs.

• Partner with integrated platform & data science teams to define integrated solutions to meet the evolving needs of data analysts, developers & consumers (e.g. data ingestion framework, AI/ML capabilities, scalable compute), contributing to the innovation and leadership of the organization.

Data Optimization and Automation:

• Identify, design, and implement internal process improvements, automation of manual processes, optimization of data delivery, etc.

• Build the data infrastructure required for optimal extraction, transformation, and loading of data from traditional/legacy sources.

Subject Matter Expertise and Leadership:

• Act as a subject matter expert for internal or external data products.

• Contribute to solution architecture design and advise on engineering solution best practices.

• Mentor and guide junior data engineers by leading code reviews and documenting best practices.

• Use Mass General Brigham values to govern decisions, actions, and behaviors.

• Perform other duties and responsibilities as assigned.

Working Conditions:

The work environment characteristics described here are representative of those an employee encounters while performing the essential functions of this job.

• This position requires occasional local travel to MGB sites, vendors, and/or conferences

• Hospital work environment working conditions include possible exposure to diseases or infections and may require safety gear (PPE) such as gloves and mask.

• Normal office working conditions. The noise level in the work environment is quiet to moderate.

• While performing the duties of this job, the employee is frequently required to sit; talk; or hear; use hands to finger; handle; or feel; reach with hands and arms. The employee is occasionally required to stand; walk; and stoop; kneel; or crouch. The employee must frequently lift and/or move up to 5 pounds and occasionally lift and/or move up to 20 pounds.

• Specific vision abilities required by this job include close vision, distance vision and depth perception.

Bachelor's or Master's degree in Computer Science, Information Technology, or related fields; or comparable work experience

• 8 years of related professional experience including 5 years in data lake development in large reporting environment(s)

• Expert in Azure cloud computing, specializing in Azure data engineering stacks like ADF (Azure Data Factory), ADLS, Event Hubs, Snowflake, Databricks, streaming, Azure PowerShell, and Log Analytics.

• Hands-on development experience with Design and Architecture of big data frameworks/tools: Azure Data Lake, Snowflake, Azure Data Bricks.

• Expert experience with Hadoop based technology (e.g. Spark) and programming (Python, SQL, PySpark)

Knowledgeable about cloud computing costs and performance and capable of providing ongoing suggestions for cost & performance optimization.

• Solid understanding of Snowflake computing, including its integration with Azure Data Lake, utilizing ADLS as a source for data processing.

• Experience with Design and Architecture of relational SQL and NoSQL databases, including MS SQL Server, Snowflake.

• Experience with Azure DevOps, familiar with CI/CD (Continuous Integration/Continuous Deployment) processes and capable of scaling them across engineering teams, leveraging Azure native resources for data ingestion.

• Experience leading and working with cross-functional teams in a dynamic environment, demonstrated track record of team leadership, technical acumen, innovation, tactical and strategic.

• Proven verbal, communication, and presentation skills, ability to clearly and concisely communicate complex technical concepts to both technical and non-technical audiences.

• Proven ability to work independently.

Skills/Abilities/Competencies:

Advanced hands-on SQL, Spark, Python, PySpark knowledge and experience working with relational databases for data querying and retrieval on multiple platform.

• Proficiency in Data Modeling tools (e.g. Erwin, Visio).

• Strong interpersonal and communication skills, both written and verbal.

• Strong Scrum/Agile development experience.

• Excellent organizational skills and attention to detail, manage multiple tasks and projects, meet deadlines, follow through, and manage to schedule.

• Strong innovation capabilities and the ability to think creatively.

• Strong collaboration and team building skills within, across and outside of an organization.

• Maintain and promote a positive team environment.

• Maintains stable performance under pressure, demonstrating sensitivity to diverse organizational culture.

• Ability to effectively cope with change, remain flexible and adaptable within a fast-paced environment with rapidly changing requirements, and ability to negotiate situations when the big picture is not clearly defined.

PREFERRED SKILLS/ABILITIES/COMPETENCIES

• Strong root cause analysis and problem-solving skills

• Ability to juggle multiple projects in a high volume, fast-paced Production environment

• Ability to identify process improve opportunities and areas of potential conflict

• Working knowledge of MGB data products and platforms a plus

• Ability to quickly adapt to new technologies, concepts, and approaches

• Familiar with current healthcare and data management trends and industry practices

• Demonstrated ability to manage multiple priorities

• Microsoft Certified: Azure Solutions Architect Expert ( Nice to Have)

• Microsoft Certified: Azure Data Engineer ( Nice to Have)