Responsibilities
The Data Engineer Lead will play a pivotal role in identifying and prioritizing data and transformation requirements to bolster strategies and operational deliverables. Your responsibilities include gathering, documenting, and promoting data engineering best practices, designing and managing algorithms and pipelines in collaboration with diverse teams, and conducting optimizations and root cause analysis for continuous improvement. As a key partner to business and IT stakeholders, you will address data-related technical issues and support BI and data teams' operational requirements.
- Work in collaboration with the business community in identifying and prioritizing their data and transformation requirements to support their strategies and operational deliverables
- Gather, document, promote, enforce, and maintain relevant data engineering best practices
- Design, deliver and manage the algorithms and pipelines for acquiring the information and addressing the business need - in collaboration with the architects, the business intelligence team and the data governance team
- Perform optimizations and root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvements
- Partner with business and IT stakeholders to assist with data-related technical issues and support BI and data teams' operational requirements
- Work with data and analytics experts to strive for greater functionality, traceability and cross-functional cohesion in our data systems
- Assist the development and architecture teams on data quality and various POC initiatives
Qualifications
- Graduate degree in Computer Science, Statistics, Informatics, Information Systems, or a related quantitative field, with a strong advantage if certified in Databricks Data Engineering
- At least eight years of working experience/knowledge in designing, building, and optimizing data pipelines, and data sets
- At least three years of experience working in Azure and Databricks technology
- Working SQL knowledge and experience with relational databases, query authoring (SQL)
- Working knowledge performing root cause analysis on internal and external data processing related issues
- Understanding processes supporting data transformation, data structures, metadata, dependency, and workload management
- Knowledge with big data tools: Hadoop, Spark, etc. Knowledge of relational SQL and NoSQL databases
- Knowledge of object-oriented/object function scripting languages: Python, Java, etc.
