Senior Data Engineer — Louisville, KY
Зарплата не указана
ОткликнутьсяRole: Senior Data Engineer
Location: Louisville Kentucky (Hybrid)
Duration: Full time
Job Description
We are seeking an experienced Senior Data Engineer with strong hands-on expertise in Databricks, PySpark, Apache Spark, Spark SQL, and Delta Lake. The ideal candidate will design, develop, and optimize scalable data pipelines and enterprise Lakehouse solutions.
Key Responsibilities
• Design and develop scalable batch and real-time data pipelines using Databricks.
• Build ETL/ELT workflows using PySpark, Spark SQL, and Delta Lake.
• Design and manage Databricks Lakehouse architecture and data models.
• Optimize Spark jobs, clusters, and SQL queries for performance and cost efficiency.
• Implement data quality checks, validation, and schema evolution.
• Ensure data governance, security, RBAC, and data lineage.
• Collaborate with cross-functional teams and participate in Agile development.
Required Skills
• Strong hands-on experience with Databricks, PySpark, Apache Spark, and Spark SQL.
• Expertise in Delta Lake and Data Lakehouse architecture.
• Experience developing large-scale ETL/ELT data pipelines.
• Strong Python and SQL skills.
• Experience with Azure, AWS, or GCP.
• Knowledge of data governance, security, and performance optimization.
Preferred Skills
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Kafka / Streaming Technologies
• Unity Catalog
• CI/CD and DevOps
• SQL Performance Tuning
Experience Requirements
• 8+ years of overall Data Engineering experience.
• 3+ years of PySpark experience.
• 3+ years of Azure Databricks experience.
Primary Skills: Databricks, PySpark, Spark SQL, Delta Lake, Azure, Data Lakehouse, ETL/ELT, Python, SQL.