Role – AWS Databricks Architect
Location: O’Fallon, MO (Onsite Day -1)
Yrs. of experience: 10+ Yrs.
Mode of employment: Full-Time
Job Description:
Candidates are required to have these mandatory skills:
- Strong hands on in PySpark and Apache Spark.
- Experience in Native Spark Migration to Databricks.
- Experience in Hadoop Migration to Databricks.
- Experience in Building Data Governance Solutions like Unity Catalog, StarBust …etc
- Build Very Strong Orchestration Layer in Databricks/ADF…. Workflows.
- Build CICD for Databricks in Azure Devops.
- Process near Real time Data thru Auto Loader, DLT Pipelines.
- Implement Security Layer in Delta Lake.
- Implement Massive Parallel Processing Layers in Spark SQL and PySpark.
- Implement Cost effective Infrastructure in Databricks.
- Experience In extracting logic and from on prem layers like SSIS, Stored procedures, Informatica, Vertica, Apache Hudi, Filesystems. etc into Pyspark.
- Ability to Build Solutions as Cloud Agnostic.
- Proven experience in designing and implementing complex data solutions aligned with business objectives.
- Expertise in data modelling, integration, security, and governance
- Hands-on experience with guiding the virtual data model definition, defining Data Virtualization architecture and deployment with focus on Azure, Databricks, PySpark technologies.
- Prior experience with establishing best practices for business optimizations.
- Experience with relational and non-relational data stores (Hadoop, SQL, Mongo DB), ETL or ELT tools (SSIS, Informatica, Matillion, DBT), DevOps, Data Lake and Data Fabric concepts
- In-depth experience with data governance, data integration and related technologies.
- Proficiency in a variety of database technologies, both relational and non-relational.
- Knowledge of cloud-based data solutions (e.g., AWS, Azure).
- Excellent collaboration and communication skills
