Senior Systems Engineer

Under general supervision of the MIS Supervisor, or designee, this position is responsible for maintaining a highly available and secure information technology infrastructure throughout the city. Leads efforts and projects related to the design, installation, operation, support, upgrades, and maintenance of infrastructure, including software, servers, networking, and storage. Administers, maintains, and supports complex and critical infrastructure. Provides user support for server hosting requests, problem resolution, system changes and upgrades.  Analyzes infrastructure performance and resolves problems or makes appropriate recommendations. Ensures that all aspects of operability are delivered as part of the implementation process while also ensuring that existing service level is maintained or improved. Contributes to and ensures full compliance with operational standards, procedures, and best practices.

The intent of this position description is to provide a representative summary of the major duties and responsibilities performed by incumbent(s) in this position. Incumbent(s) may not be required to perform all duties in this description and incumbent(s) may be required to perform position-related tasks other than those specifically listed in this description. Essential Job Functions:

• Oversees the day-to-day operations infrastructure services including server, storage, datacenter, cloud, backup, and disaster recovery.

• Monitor ticketing queues in help desk system to address issues in a timely manner, recording solutions, serves as an escalation point and escalates as required.

• Oversee vendor management, and oversight to deploy systems that are complete, designed and managed over life cycles.

• Monitor and maintain the operations, maintenance and repair of all mission-critical equipment and systems supporting 24x7 data center operation.

• Monitors application performance on servers.

• Evaluates and manages appropriate software and hardware allocations to achieve an optimum performance level.

• Performs capacity planning for projecting future growth.

• Provides recommendations regarding infrastructure planning and budgeting.

• Maintains growth statistics, space forecasts, backup systems, and infrastructure inventories.

• Anticipates impact of growth and changes in operations and recommends design and/or process changes.

• Coordinates infrastructure planning and management.

• Develops and promotes standard operating procedures.

• Conducts routine infrastructure audits to ensure compliance with established standards, policies, and configuration guidelines.

• Develops and maintains a comprehensive infrastructure configuration database/library of all supporting documentation.

• Maintain and test enterprise disaster recovery and backup systems.

• Performs data backups and recoveries.

• Ensures data/media recoverability by implementing a schedule of system backups and database archive operations.

• Supports media management through internal methods and procedures or through offsite storage and retrieval services.

• Participates in disaster recovery/business continuity planning including backup and recovery procedures and high availability configurations.

• Coordinates infrastructure planning and management.

• Performs infrastructure upgrades including planning and scheduling, testing, and coordination.

• Performs infrastructure administration setup.

• Performs infrastructure installations and upgrades.

• Schedules installations and upgrades and maintains them in accordance with established IT policies and procedures.

• Provide and ensure the highest levels of systems and infrastructure availability and resolve escalations of low to high complexity.

• Research problems, plan solutions, recommend software and systems, and coordinate developments in an orderly manner to meet business requirements.

• Maintains production change control schedule and participates in change control.

• Performs off-hours and weekend maintenance activities as required.

• Supports storage solutions (Nutanix Files, NetApp, DFS, local) for all city locations.

• Participates in the On-Call rotation/after-hours support, ensuring uptime of critical servers, networks and applications and coordinate with third party providers and other members of city staff to insure rapid and satisfactory response to City emergency technical issues.

• Creates and maintains user accounts, security, permissions, and file systems in accordance with standards and operational requirements.

• Monitors and maintains continuity with system licensing and maintenance agreements.

• Completes daily automated procedures including systems back up, production runs.

• Leads planning, development, and coordination of operations and projects for current and future infrastructures.

• Serves as project liaison - implementations or upgrades, acts as the focal point for communications between CI and the Project Leader.

• Facilitates coordination and a thorough understanding of requirements, attends project meetings, creates written meeting notes, creates appropriate ticketing, etc.

• Maintains policies and standard procedures to increase system uptime.

• Identifies training needs and keeps current on application technologies.

• Monitors security alerts and ensures that appropriate patches are applied in an automated and timely fashion; works with developers to patch or upgrade custom code for security compliance.

• Documents and maintains system standards; researches and recommends innovative approaches for system administration tasks.

• Creates and maintains standard OS installation images for virtualization templates.

• In collaboration with development project teams, builds, rebuilds, and/or updates servers and configures hardware and virtual machines (VM), applications, peripherals, services, networking, storage.

• Manages and coordinate IT initiatives and projects across organization units.

• Work closely with Architecture, Development and Security group on IT operations.

• Performs the assigned facility(s) rounds to ensure proper equipment operation, aesthetics, and cleanliness.

Other Important Duties:

• Acts as technical lead on projects for City departments or internal IT team as needed.

• May travel to attend meetings, presentations, conferences, and training.

• Performs other duties as assigned.

• Regular and consistent attendance at the assigned work hours is essential during business hours.

• This is an exempt position that requires availability 24x7x365 for scheduled maintenance and/or break/fix issues.

Knowledge, Skills, and Abilities:

• Proficient in hardware, software and peripheral equipment maintenance, repair, setup, modification and troubleshooting methods, practices, and procedures.

• Knowledge in electrical and mechanical systems involved in critical data center operations, including systems such as feeders, transformers, generators, switchgear, UPS systems, ATS units, PDU units, chillers, pumps, and air handling units.

• Proficient with Storage devices/protocols.

• Proficient in security threats and anti-virus systems.

• Knowledge of customer service and training methods and practices.

• Proficient in backups, restoration, and recovery models.

• Proficient in troubleshooting and resolving database problems.

• Knowledge of City policies and procedures.

• Proficient skills with LAN/WAN administration concepts and practices.  Must be knowledgeable with PC administration concepts and practices. MS Windows OS, MS Office environment, Active Directory.

• Proficient in the setup and administration of server virtualization technologies such as Nutanix AHV, Microsoft Hyper-V, and VMware.

• Proficient in managing and implementing Microsoft services (Active Directory Domain Services, AzureAD, Azure IaaS & PaaS solutions, MS365 OneDrive, SharePoint, Teams, Intune, Defender & Office ATP, MDI, etc.)

• Proficient in Mail & Content Filtering Solutions.

• Proficient in various network, data center, and data closet components.

• Proficient in communicating technical issues and problems with end users in a manner which is understandable to those without technical training.

• Proficient in responding, troubleshooting, and resolving technical automated management information systems issues over the telephone, electronically or on site.

• Proficient in implementing departmental procedures and objectives.

• Proficient in effectively transporting, setting up, installing, uninstalling, modifying, upgrading, and operating multiple networking systems and software programs.

• Proficient in resolving customer complaints and concerns.

Education, Experience, and Certifications/Licenses:

• Bachelor's Degree in Management Information Systems, Computer Science, or related field, and five to ten years' experience or equivalent combination of education and experience.

• Current Microsoft Expert level certifications.

• Current Cisco Specialist & Professional level certifications.

• Current Nutanix Master level certifications.

• Current Cohesity Professional level certifications.

• Experience with 24/7/365 systems operations in a production environment.

• May be required to obtain applicable certifications (e.g., Cisco, Microsoft, Nutanix, Cohesity, NetApp).

• Must pass a pre-employment drug screen, criminal background check and post-offer physical agility exam.

• Must possess a valid State of Texas Driver's License.

Environmental Factors and Conditions:

• Work is performed in office and on-site environments.

• May be subject to exposure to extreme weather conditions when loading, unloading, transporting, or installing automated Management Information Systems and personal computer stations.

• May be subject to electrical currents and electrical shock.

• Must live within 1 hour of city limits.

Physical Demands:

• This work typically requires the following physical activities to be performed.  A complete description of the activities below is available upon request from Human Resources.

• Crouching - bending body forward by bending leg, spine.

• Feeling - perceiving attributes of objects by touch with skin, fingertips.

• Fingering - picking, pinching, typing, working with fingers rather than hand.

• Hearing 1 - perceiving sounds at normal speaking levels, receive information.

• Kneeling - bending legs at knee to come to rest at knees.

• Lifting - raising objects from lower to higher position, moving objects side to side, using upper extremities, back.

• Mental Acuity - ability to make rational decisions through sound logic, deductive reasoning.

• Reaching - extending hands or arms in any direction.

• Repetitive Motion - substantial movements of wrists, hands, fingers.

• Speaking - expressing ideas with spoken word, convey detailed, important instructions accurately, concisely.

• Standing - for sustained periods of time.

• Stooping - bending body downward, forward at waist, with full motion of lower extremities and back.

• Talking 1 - expressing ideas by spoken word

• Visual Acuity 1 - prepare, analyze data, transcribing, computer terminal, extensive reading.

• Visual Acuity 2 - color, depth perception, field of vision.

• VIsual Acuity 3 - determine accuracy, neatness, observe facilities/structures.

• Walking - on foot to accomplish tasks, long distances, or site to site.

Work Environment:

• Medium work: Exerting up to 50 pounds of force occasionally, and/or up to 30pounds of force frequently, and/or up to 10 pounds of force constantly to move objects.