Projects the candidate will be working on:
- Create and maintain data pipelines between on-premise data center, Azure Data Lake Storage, and Azure Synapse database using Databricks and Apache Spark/Scala.
- This role is for a senior data engineer that will join a team responsible for managing a growing cloud-based data ecosystem consisting of a metadata driven data lake and databases that support real time analytics, extracts, and reporting.
- The right candidate will have a solid background in data engineering and should have a few years of experience on a major cloud platform such as Azure.
Team and Team size:
- Team Name: Alefgard, Size: Approximately 10+.
Top Responsibilities:
- Building and maintaining a data processing framework on Azure using Databricks
- Writing code in Apache Spark/Scala
- Working with existing Databricks Delta Lake tables to optimize for CDC performance using techniques
- Working with existing Databricks Notebooks to optimize or address performance concerns
- Create new Databricks Notebooks or stand-alone Apache Spark/Scala code as needed
Working with existing on-premise data management tools as required and willing to learn Ab Initio.
Software tools/skills:
- Databricks
- Apache Spark
- Scala programming
- Azure
Skills/attributes:
- Data Warehousing / Big Data Best Practices
- Understanding of how best to partition and organize data depending on the technology and use case: 2 years
- Experience in regularly dealing with data in hundreds of Terabytes up to 1-2 Petabytes: 2 years
- Data engineering experience: 5 years
- Cloud platform experience: 2 years
- Version Control (Git or equivalent): 2 years
Nice to have:
- Data Integration Tools (Spark/Databricks or equivalent): 2 years
- Scripting (Linux/Unix Shell scripting or equivalent): 2 years
- Ab Initio experience
- Netezza Experience
Interview Process:
- How many rounds?
- Round 1 (Video Webex call with a few of our data engineers)
- Discussion on resume with technical questions.
- Round 2 (Video Webex call with hiring manager(s))
Horizontal is proud to be an Equal Opportunity and Affirmative Action Employer. We seek to provide employment opportunities to talented, qualified candidates regardless of race, color, sex/gender including gender identity and/or expression, national origin, religion, sexual orientation, disability, marital status, citizen status, veteran status, or any other protected classification under federal, state or local law.
In addition, Horizontal will provide reasonable accommodations for qualified individuals with disabilities. If you need to request a reasonable accommodation in order to complete the application or interview process, please contact hr@horizontal.com.
All applicants applying must be legally authorized to work in the country of employment.
Success!
You have saved your first job! To see all your Saved Jobs, click here. Or continue scrolling through jobs and bookmark openings that catch your eye and apply for those jobs later.
We’re sorry!
There are currently no open positions in your location or accepting applications from out of the country
Return to Home