Senior Data Engineer, Azure Databricks — Memphis, TN

Job Title: Senior Data Engineer, Azure Databricks

Location: Fully Remote (on-site in Dallas, Memphis, Cleveland, or Pittsburgh is optional)

Employment Status: Applicants must be currently authorized to work in the U.S. without the need for employment-based sponsorship, now or in the future, including: H-1B, L-1, TN, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT, or any other employment-based visa programs.

Position Overview

The Senior Data Engineer will design, develop, and optimize enterprise data warehouse solutions and production data pipelines within an Azure Databricks environment. This hands-on position will support the continued development and modernization of the data ecosystem including the transition toward real-time, event-driven data processing, while ensuring data quality, reliability, and performance in a highly regulated environment.

Key Responsibilities

• Design, develop, and maintain enterprise data warehouse solutions and production pipelines across Bronze, Silver, and Gold layers using Azure Databricks, Delta Lake, and medallion architecture.

• Develop and optimize data ingestion, transformation, and modeling solutions using SQL, Python, PySpark, and Azure Data Factory.

• Build dimensional and harmonized data models, incorporating data conformance, survivorship logic, and standardized transformation processes.

• Support the transition from traditional batch processing toward continuous, low-latency, event-driven data ingestion using Azure Event Hubs, Structured Streaming, and emerging Databricks capabilities.

• Own assigned production data domains, ensuring pipeline reliability, data integrity, performance, and compliance with established service-level requirements.

• Monitor and optimize pipeline performance, processing latency, scalability, and Databricks compute costs.

• Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable improvements to data reliability and system performance.

• Implement automated data validation, reconciliation, quality controls, and testing practices throughout the data lifecycle.

• Support data governance initiatives involving Unity Catalog, metadata management, lineage, access controls, and Gold-layer data assets.

• Modernize and consolidate legacy transformation processes while maintaining data accuracy and operational continuity.

• Establish and maintain engineering best practices for code quality, testing, documentation, version control, and CI/CD deployment within a highly regulated environment.

• Collaborate with Data Architecture, Analytics Engineering, QA, Data Governance, and business stakeholders to deliver reliable, scalable data solutions.

• Develop and enhance internal and external partner data integrations to support evolving business and operational requirements.

• Provide technical guidance and mentorship to Data Engineers through code reviews, knowledge sharing, and engineering best practices.

Qualifications

• 6+ years of progressive data engineering experience including hands-on development and production support of enterprise data platforms.

• Deep expertise in Azure Databricks, Delta Lake, Medallion Architecture, Unity Catalog, and production pipeline development and optimization.

• Hands-on experience developing and operating event-driven streaming data pipelines using Azure Event Hubs, Kafka, AWS Kinesis, or Confluent Cloud.

• Strong enterprise data warehousing experience encompassing dimensional modeling, complex transformations, data integration, and production operations.

• Advanced proficiency in SQL, Python, and PySpark, with the ability to independently develop, troubleshoot, review, and optimize complex data pipelines.

• Hands-on experience with Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps.

• Demonstrated ownership of production data pipelines, including monitoring, incident response, performance optimization, and service-level accountability.

• Experience implementing data contracts using ODCS or equivalent standards preferred.

• Experience implementing data validation, reconciliation, automated testing, and data quality controls.

• Strong understanding of data governance, metadata management, lineage, and secure data handling practices.

• Familiarity with data observability platforms such as Monte Carlo preferred.

• Experience operating within highly regulated environments including healthcare, pharmacy, pharmaceutical, medical, or another highly regulated industry involving sensitive clinical or operational data is preferred.

• Ability to analyze complex legacy data environments, improve existing architectures, and produce comprehensive technical documentation.

• Familiarity with Structured Streaming, Zerobus Ingestion, and Spark Declarative Pipelines preferred.

• Experience with enterprise data catalog platforms such as Atlan, Collibra, or equivalent technologies preferred.

• Experience developing governed data assets with documented metadata, lineage, and access controls.

• Strong communication, problem-solving, and cross-functional collaboration skills.