We are seeking a Senior Data Engineer who will play a pivotal role in shaping our data platform team. This position offers the opportunity to work with cutting-edge technology while making a meaningful impact on our data-driven initiatives.

Responsibilities

• Design and implement robust data pipelines utilizing Databricks and AWS services for both batch and real-time processing.
• Architect and maintain a scalable lakehouse infrastructure, ensuring data is reliable and accessible for analytics and AI applications.
• Collaborate with data scientists to develop feature store architecture and support predictive modeling efforts.
• Build and manage ETL/ELT orchestration and stream processing pipelines, integrating diverse data sources.
• Ensure compliance with HIPAA regulations across all data handling processes.
• Mentor junior engineers and enforce best engineering practices within the team.
• Participate in design discussions and contribute to special projects as needed.

Skills

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
• 7-10 years of professional experience in data engineering, with a strong focus on cloud-based architectures.
• Proficiency in Apache Spark and hands-on experience with Databricks.
• Expertise in AWS data services including S3, Redshift, and Glue.
• Strong programming skills in Python, with additional knowledge of SQL.

Preferred Skills

• Experience with ETL orchestration tools such as Airflow and dbt.
• Familiarity with container orchestration using Docker and Kubernetes.
• Knowledge of vector databases and LLM use cases.
• Excellent communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.

We are committed to fostering a diverse, equitable, and inclusive workplace where all individuals feel valued and respected. We encourage candidates from all backgrounds to apply and contribute to our mission.

By applying for this position, you acknowledge and agree that Horizontal Talent may contact you regarding your application using automated technology, including phone calls, SMS/text messages, or email, which may be delivered by our virtual AI recruiter, Alex.