- Technical skill:- Python, PySpark, ADF, SQL, ADLS, Microsoft Azure, Azure Databricks, Azure Synapse.
- GitHub Actions: Familiarity with setting up and managing CI/CD pipelines using GitHub Actions, ensuring smooth and automated deployment processes.
- Agile Methodology: Experience working in an Agile/Scrum environment, with a focus on iterative development, continuous feedback, and delivery.
- Experience: Proven experience in a similar role, with a strong track record as Sr. Data Engineer and has helped/lead development teams in delivering high-quality data orchestration solutions with min 7+ years’ experience.
Responsibilities:
• Development: - Working with varied data file formats (Avro, json, csv) using PySpark for ingesting and transformation, Terraform scripting and DevOps process.
• Lead Development Initiatives: Develop code with FTR and of quality. Also to take ownership of different development initiatives, ensuring timely delivery while meeting or exceeding quality standards.
• Code Quality and Maintenance: Oversee the maintenance of code quality and enforce best practices, including code reviews, unit testing, CICD, and adherence to coding standards.
• Reduce Technical Debt: Identify, prioritize, and implement strategies to reduce technical debt and address vulnerabilities in the codebase.
• Team Leadership: Provide mentorship and guidance to team members, fostering a collaborative, innovative, and high-performance environment.
• Collaboration: Work closely with cross-functional teams, including product management, Cloud Security, LOB, Network security, and DevOps, to integrate different features into broader solutions.
• Problem Solving: Address and troubleshoot complex technical issues, providing solutions that enhance system performance and user experience.
• Documentation: Ensure comprehensive documentation of systems, processes, and code to facilitate knowledge sharing and maintenance.
• Stakeholder Communication: Communicate progress, challenges, and solutions to stakeholders, ensuring transparency and alignment with business objectives.
