150-155k + Bonus + Benefits / Stamford CT / Onsite
The Principal Data Engineer responsibilities include:
- Provide support in designing and overseeing enterprise-grade data pipelines and data stores, essential for the development of advanced analytics programs, machine learning models, and statistical methods.
- Implement automation and streamline processes to optimize the entire data and analytics platform, ensuring efficient throughput and high-performance outcomes.
- Recognize, devise, and execute internal process enhancements, including automation of manual tasks, optimizing data delivery, and redesigning architecture or infrastructure to enhance scalability.
- Collate large, intricate datasets that align with functional and non-functional business demands.
- Develop processes that facilitate data transformation, manage data structures, metadata, dependencies, and workload management.
- Collaborate with business users to understand functional and data requirements, contributing to the enhancement of data models and pipelines.
- Apply expert-level analytical and problem-solving skills to diagnose and resolve intricate technical issues.
- Create, maintain, and continuously enhance scalable data pipelines, while also developing new data source integrations to accommodate the growing volume and complexity of data.
- Designing, implementing, and managing data extraction, transformation, and loading (ETL) processes.
- Creating comprehensive technical specification documents and application interface designs.
- Creating data processing and integration solutions for both batch and real-time scenarios, proficiently handling structured and unstructured data.
- Leading and participating in design discussions, code reviews, and project-related team meetings.
- Providing mentoring and guidance to junior team members, promoting a culture of knowledge sharing and collaborative problem-solving.
Qualifications
- Exhibit a thorough understanding of Data Lake architectures, including raw, enriched, and curated layer concepts, and ETL/ELT operations.
- Exhibit a solid understanding of database design, data warehousing concepts, big data platforms, and ETL operations.
- Experience working with data integration techniques & self-service data preparation.
- Experience in requirements analysis, design, and prototyping.
- Experience deploying modern data solutions leveraging components like Azure functions, Azure Databricks, Azure Data Factory, Data Flows, Azure Data Lake, Azure SQL, Azure Synapse, Streaming Analytics.
- Experience with DevOps tools like Azure DevOps, Jenkins, Maven etc.
- Experience in building/operating/maintaining fault tolerant and scalable data processing integrations.
- Experience with SQL.
- Demonstrated experience of turning business use cases and requirements into technical solutions.
