Senior Data Engineer — Buffalo, NY

Job Description Overview:

• Designs, develops, tests, deploys and supports complex data pipelines, data products and data management solutions across modern cloud, warehouse and lakehouse platforms. Applies software-engineering, data-modeling and distributed-processing practices to deliver secure, reliable, maintainable and cost-conscious solutions. Works independently on defined outcomes, collaborates across technology and data teams, and contributes to reusable engineering patterns and continuous improvement.

Primary Responsibilities:

• Design, develop, test, deploy and support complex production data pipelines and data products with guidance on broader architecture where needed.

• Develop maintainable data-processing and transformation solutions using SQL and programming languages such as Python.

• Implement solutions using Snowflake, Databricks, dbt and comparable modern data platforms and frameworks.

• Build data solutions across relational databases, cloud object storage, warehouse and lakehouse architectures, including open table formats such as Apache Iceberg where applicable.

• Develop batch, micro-batch, CDC and streaming pipelines using approved enterprise patterns and technologies such as Apache Kafka or comparable platforms.

• Integrate data from relational and mainframe systems, APIs, event streams, SaaS platforms, files and object storage.

• Apply normalized, dimensional and denormalized data-modeling techniques appropriate to analytical and data-product requirements.

• Tune SQL, Snowflake and Databricks/Spark workloads for performance and cost using appropriate partitioning, parallelism and compute practices.

• Implement data quality, reconciliation, metadata, lineage, logging, monitoring and alerting controls.

• Build pipelines that support error handling, restartability, idempotency, retry, replay and recovery using established patterns.

• Participate in production support, incident resolution and root-cause analysis; implement corrective actions to prevent recurrence.

• Use Git, peer review, automated testing, CI/CD and environment promotion practices for data engineering delivery.

• Develop and reuse shared components, templates and patterns; contribute improvements to engineering standards and documentation.

• Participate in technical design and code reviews, identify implementation risks and escalate architectural concerns appropriately.

• Partner with product owners, architects, governance, cybersecurity, platform and consuming teams to clarify requirements and deliver supported solutions.

• Contribute to technical epics and stories, estimate assigned work, identify dependencies and deliver through Agile practices.

• Provide day-to-day technical assistance and informal mentoring to less-experienced engineers. This role does not have direct people-management responsibility.

• Understand and adhere to the Company’s risk and regulatory standards, policies and controls in accordance with the Company’s Risk Appetite.

• Design, implement, maintain and enhance internal controls to mitigate risk on an ongoing basis. Identify risk-related issues requiring escalation to management.

• Maintain internal control standards, including timely implementation of internal and external audit points and issues raised by external regulators, as applicable.

• Promote an environment that supports belonging and reflects the Company brand.

• Complete other related duties as assigned.

Education and Experience Required:

• Associate’s degree and a minimum of 5 years’ systems analysis and/ or application development work experience or Bachelor's degree and a minimum of 3 years’ systems analysis and/ or application development work experience. In lieu of degree, a combined minimum of 7 years’ education and/or relevant work experience, including a minimum of 3 years’ systems analysis and/or application development work experience

• Strong SQL skills, including development of complex transformations and analytical queries.

• Experience with Python or another general-purpose programming language used for data engineering.

• Hands-on experience with Snowflake, Databricks or a comparable cloud data platform.

• Experience with ELT/ETL and transformation practices, including dbt or a comparable framework.

• Experience with relational data stores and warehouse or lakehouse data-modeling concepts.

• Experience with distributed processing using Apache Spark/PySpark or a comparable framework.

• Experience with Git-based source control, automated testing and CI/CD fundamentals.

• Experience with batch data processing and working knowledge of streaming or event-driven concepts.

• Knowledge of cloud and object-storage concepts, preferably in Microsoft Azure.

• Experience working with structured and semi-structured data using formats such as Parquet, JSON, CSV, Avro and XML.

• Experience with Linux/Unix environments and scripting.

• Demonstrated ability to independently deliver complex data pipeline components and data products within an established architecture.

• Demonstrated ability to troubleshoot data-processing, data-quality and production issues across multiple technologies.

• Demonstrated ability to apply established engineering patterns while identifying opportunities for improvement.

• Demonstrated ability to communicate implementation choices, risks and dependencies to technical and product partners.

• Demonstrated ability to manage assigned work from design through production support with limited day-to-day supervision.

Education and Experience Preferred:

• Production experience with Snowflake, Databricks, Apache Spark/PySpark and dbt.

• Experience with Apache Iceberg or another open table format.

• Experience with Apache Kafka, CDC or event-driven data processing.

• Experience with orchestration technologies such as Apache Airflow, Azure Data Factory, Databricks Workflows or dbt orchestration.

• Experience with Terraform or another Infrastructure as Code technology.

• Experience with data catalogs, metadata, lineage, observability or automated data-quality tooling.

• Experience delivering governed enterprise data products.

• Experience working in a regulated environment such as financial services.

4&P CSJp is committed to fair, competitive, and market-informed pay for our employees. The pay range for this position is $97,100.00 - $161,800.00 Annual (USD). The successful candidate’s particular combination of knowledge, skills, and experience will inform their specific compensation.

Location

Buffalo, New York, United States of America