Senior AWS Data Engineer -Full time — Louisville, KY
Зарплата не указана
ОткликнутьсяRole: Senior AWS Data Engineer -Full time
Location: Louisville Kentucky (Hybrid)
Duration: Full time
Key Responsibilities
Data Engineering & Development
Design, develop, and maintain scalable batch and real-time data pipelines using Databricks.
Build robust data solutions using PySpark, Spark SQL, and Delta Lake.
Implement data validation, quality checks, and performance tuning processes.
Develop ETL/ELT workflows to support enterprise data integration needs.
Data Lakehouse Management
Design and manage data architecture within the Databricks Lakehouse Platform.
Implement Delta Lake tables, partitioning strategies, optimization techniques, and schema evolution.
Develop data models to support analytics, reporting, and business intelligence requirements.
Ensure data reliability, scalability, governance, and availability.
Performance Optimization
Analyze and optimize Spark jobs for performance and cost efficiency.
Monitor Databricks cluster utilization and implement best practices for resource management.
Troubleshoot bottlenecks in data processing workflows.
Improve processing times through caching, partitioning, and query optimization techniques.
Data Governance & Security
Implement role-based access controls (RBAC) and data security standards.
Support data governance initiatives, data lineage tracking, and compliance requirements.
Ensure adherence to enterprise data management policies and security frameworks.
Collaboration & Delivery
Work closely with Data Architects, Data Scientists, Business Analysts, and Application Teams.
Participate in Agile development processes, sprint planning, and scrum ceremonies.
Prepare technical design documentation and support knowledge transfer activities.
Collaborate with stakeholders to gather requirements and translate them into scalable technical solutions.
Required Qualifications
Bachelor's degree in Computer Science, Information Technology, or a related field.
Strong experience with Databricks, Apache Spark, PySpark, Spark SQL, and Delta Lake.
Experience designing and developing large-scale data pipelines and ETL processes.
Strong understanding of Data Lakehouse architecture and data modeling concepts.
Experience with cloud platforms such as Azure, AWS, or GCP.
Knowledge of data governance, security, and enterprise data management practices.
Excellent problem-solving, analytical, and communication skills.
Preferred Skills
Azure Data Factory (ADF)
Azure Synapse Analytics
Kafka or other streaming technologies
CI/CD pipelines and DevOps practices
Python programming
SQL performance tuning
Unity Catalog and Databricks governance tools
Primary Skills: Databricks, PySpark, Spark SQL, Delta Lake, Data Lakehouse, Azure/AWS, ETL, Data Engineering.