Job Description
Senior Big Data Engineer
* Designing and implementing ETL Process using Bigdata Technologies (Pyspark, Spark, Hadoop, Hive, Sqoop, Oozie, Unix Shell Script)
* Experience in handling Query/code Optimization & query performance
* Managing the Data Lake environment like Compliance issues fix, Upgrades, Data Archival/Retrieval process to/from AWS S3
* Developing Automation scripts to fix the Production issues and mentors Team to handle the Prod issues
* Managing & Setting up the Ingestion
* Developing data extraction and transformations to improve data reliability and quality
* Utilizing past experiences to identify and troubleshoot potential issues, ensuring smooth data operations
* Advising team members in analyzing relevant data and assessing alternatives
* Provide effective solutions to complex technical challenges, applying technical competence to understand and resolve issues.
* Develop high-performance ingestion pipelines to a Big Data platforms, ensuring consistent and reliable data flows
