**This is a full-time, permanent position. Please note that the client is not offering sponsorship for this role**
A leading retail company is seeking a Lead Data Engineer who will play a critical role in designing, developing, and maintaining their data pipelines, data warehouses, and analytics solutions. In this position, you will collaborate closely with cross-functional teams, including Data Science, Business Intelligence, and IT, to ensure that their data infrastructure aligns with the company's strategic goals. This role demands deep expertise in SQL, Python, Spark, and Azure, along with strong leadership skills to mentor and guide a team of data engineers.
Key Responsibilities:
- Lead and Develop Data Infrastructure: Design, build, and maintain scalable data pipelines and data warehouses that support the company's data and analytics needs.
- Collaborate with Cross-Functional Teams: Work closely with Data Scientists, Analysts, and IT teams to understand data requirements and deliver solutions that drive business insights and decision-making.
- Data Integration and ETL: Develop and manage data integration processes, including ETL (Extract, Transform, Load) workflows using SQL, Python, and Spark.
- Azure Cloud Services: Implement and optimize data solutions on Azure, including Azure Data Lake, Azure SQL Database, Azure Synapse Analytics, and other relevant Azure services.
- Performance Optimization: Monitor and optimize the performance of data pipelines and storage solutions to ensure efficiency, scalability, and reliability.
- Data Governance and Security: Ensure data governance, security, and compliance best practices are followed across all data engineering activities.
- Mentorship and Leadership: Lead and mentor a team of data engineers, providing technical guidance, coaching, and fostering a culture of continuous learning and improvement.
- Stay Updated on Industry Trends: Keep abreast of the latest trends and technologies in data engineering and retail analytics to drive innovation within the company.
Qualifications:
- Bachelor's or Master's Degree in Computer Science, Information Technology, Data Science, or a related field.
- 5+ years of experience in data engineering, with a proven track record of designing and building scalable data solutions.
- Proficiency in SQL for data manipulation and querying.
- Expertise in Python for scripting, data processing, and automation.
- Experience with Apache Spark for big data processing.
- Strong knowledge of Azure cloud services, including Azure Data Lake, Azure SQL Database, Azure Synapse Analytics, and related tools.
- Leadership experience in guiding and mentoring data engineering teams.
- Solid understanding of data modeling, ETL processes, and data warehousing.
- Excellent problem-solving skills and the ability to work in a fast-paced, dynamic environment.
- Strong communication skills to collaborate effectively with technical and non-technical stakeholders.
