Summary of Position:
The Data Engineer role will work closely with an agile development team to analyze and optimize ETL processes, tables, views, materialized views, and stored procedures to create a high-performance data warehouse solution for our client. This role will be working with a diverse set of technology including Amazon Redshift and S3 and Snowflake. The Data Engineer will focus on optimizing existing systems as well as developing designs and strategies for implementing new systems. This role will be responsible for requirements gathering, analysis, data categorization, interpreting requirements, and developing development tasks for sprint execution.
Essential Functions:
- Design, implement, test, and deploy data processing infrastructure
- Contribute to architecture of highly scalable and reliable data engineering solutions for moving large data efficiently across systems
- Perform work in an Agile team setting
- Break down, estimate, and provide just-in-time design for small increments of work
- Work in a complex data infrastructure environment
- Develop in-depth data pipeline using industry-standard data integration tools
- Full development life cycle management, including gathering, analysis, architecture, design, implementation, testing, deployment, and technical support
- Write test cases and test scripts for data quality assurance
- Responsible for creating stored procedures and functions
- Develop dimensional data model with the industry-standard tool
- Interpret reporting requirements into actionable development tasks
- Analyze and optimize SQL based stored procedures and jobs
- Analyze table indices for performance
- Design and implement Materialized Views and Views
- Develop high-performance programs and procedures for ETL Processes
- Analyze, categorize and document data sources and elements
- Write and optimize queries and provide guidance to other developers accessing data
- Agile development experience required, must be comfortable working with a distributed team
Required Education and Experience:
- Bachelors Degree
- 5 + years of Data Engineering Experience or equivalent experience
Qualifications:
- Five + (5) years building out highly scalable, scaled-out architectures on large scale database platforms
- Five + (5) years of experience in a senior data engineering role
- Excellent SQL Skills
- Deep knowledge of systems such as Redshift, Snowflake, Postgres, Redis
- Experience with other programming languages such as Java and Ruby a plus
- Strong understanding of different data access standards and including REST and SOAP data sources
- Advanced competency in SQL with the ability to optimize and mentor others to perform query optimization in large scale database environments
- Experience with any industry-standard tool for Source Control and Project Management
- Experience with data visualization and/or dashboard development
- Strong written and oral communication skills
- Demonstrates critical thinking, analytical and problem-solving skills, and ability to think creatively
- Exhibits a sense of ownership, urgency, accountability, and drive to learn new technologies
- Demonstrated ability to achieve stretch goals in a highly innovative and fast-paced environment
Preferred:
- Familiarity with AGILE and API development
- Familiarity with test-driven development methodology for analytic solutions
- Apache NiFi
- Redis
