Senior Data Engineer — Atlanta, GA

Job Description

We are looking for an experienced Senior Data Engineer with strong expertise in Databricks, Python, PySpark, Apache Spark, and large-scale data engineering. The ideal candidate will be responsible for designing and developing scalable ETL pipelines, implementing data processing solutions, managing data ingestion frameworks, and optimizing cloud-based data platforms. The role also involves technical leadership, data governance, and collaboration with analytics and business intelligence teams.

Required Technical Skills

• Strong hands-on experience with Databricks, Delta Lake, Delta Live Tables, and Unity Catalog.

• Proficiency in Python, PySpark, Spark SQL, and Apache Spark.

• Strong experience in ETL development, data pipeline design, and data transformation.

• Knowledge of Hadoop, Kafka, Hive, HDFS, and Sqoop.

• Expertise in SQL, data modeling, and data warehousing.

• Experience with batch and real-time data processing using Kafka and Spark Streaming.

• Cloud experience with AWS and/or GCP.

• Strong understanding of data quality, validation, cleansing, and governance.

• Experience working in Agile environments (Scrum, Kanban, SAFe).

Preferred Skills

• Scala / Scala-Spark, IBM DataStage, TIBCO.

• AWS S3, Glue, Lambda, Kinesis/MSK.

• Tableau, Power BI, Spotfire, OAC.

• Teradata, DB2, BigQuery, Mainframe.

• Git, data governance, and retention modeling.

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL solutions.

• Build data ingestion and transformation frameworks for structured and unstructured data.

• Develop and optimize batch and streaming data processing solutions.

• Implement data quality, validation, and governance standards.

• Integrate heterogeneous data sources into enterprise data platforms.

• Support analytics, reporting, BI, and machine learning initiatives.

• Collaborate with Data Scientists, BI teams, and business stakeholders.

• Lead technical discussions, mentor junior engineers, and drive data engineering best practices.