DataStaff, Inc is seeking an Azure Databricks Engineer for a long-term contract opportunity with one of our direct clients located in Raleigh, NC.
*This position is hybrid; 1 day per week will be onsite
Responsibilities:
- Work with existing staff to plan and design ETL pipelines and product solutions using Azure Databricks. create resilient processes to ingest data from a variety of on-prem and cloud transactional databases and APIs.
- Developing business requirements, facilitating change management documentation, and actively collaborating with stakeholders.
- Work closely with a development technical lead and discuss all aspects of the design and planning with the development team.
- Research and engineer repeatable and resilient ETL workflows using Databricks notebooks and Delta Live Tables for both batch and stream processing
- Collaborate with business users to develop data products that align with business domain expectations
- Work with DBAs to ingest data from cloud and on-prem transactional databases
- Contribute to the development of the Data Architecture - Transportation:
- By following practices for keeping sensitive data secure
- By streamlining the development of data products for use by data analysts and data scientists
- By developing and maintaining documentation for data engineering processes
- By ensuring data quality through testing and validation
- By sharing insights and experiences with stakeholders and engineers
Required Skills:
- 5 Years - Excellent interpersonal skills as well as written and communication skills.
- 2 Years - Able to write clean, easy-to-follow Databricks notebook code
- 2 Years - Deep knowledge of data engineering best practices, data warehouses, data lakes, and the Delta Lake architecture
- 2 Years - Good knowledge of Spark and Databricks SQL/PySpark
- 2 Years - Technical experience with Azure Databricks and cloud providers like AWS, Google Cloud, or Azure
- 2 Years - In-depth knowledge of OLTP and OLAP systems, Apache Spark, and streaming products like Azure Service Bus
- 2 Years - Good practical experience with Databricks Delta Live Tables
Desired Skills:
- 7 Years - Knowledge of object-oriented languages like C#, Java, or Python
This opportunity is available on a corp-to-corp basis or as a W2 position with a competitive benefits package. DataStaff, Inc. offers medical, dental, and vision coverage options as well as paid vacation, sick, and holiday leave. As many of our opportunities are long-term, we also have a 401k program available for employees after 6 months.
