Databricks Architect (Cosmos / Vector DB Engineer (AI Serving Layer))

**Job Title: Databricks Data Architect** **Location:** New Jersey (Hybrid) **Responsibilities:** - Build and maintain ingestion frameworks (ADF / Databricks / Spark). - Implement Bronze to Silver transformations aligned to architecture. - Architect data quality checks, schema validation, and contract rules. - Optimize pipelines for performance and cost. - Own CI/CD pipelines (Azure DevOps). - Automate onboarding (repos, pipelines, policies, templates). - Maintain Infrastructure as Code (IaC) and scripting (Terraform, PowerShell / Python). - Troubleshoot pipeline failures and permission issues. - Design and implement curated data models in the Gold layer aligned to business use cases. - Build dimensional models (star/snowflake schemas), fact tables, dimensions, surrogate keys, and handle Slowly Changing Dimensions (SCD). - Ensure consistency across domains (common dimensions, conformed models), reusability and scalability of models, and alignment with the enterprise semantic layer (Power BI / Fabric / downstream tools). - Implement governance patterns defined by architecture. - Enforce Unity Catalog standards, naming conventions, metadata policies, and access controls (RBAC/ABAC). - Support delivery teams with onboarding, debugging, and design reviews. - Provide hands-on guidance and create reusable assets to reduce repeat questions. ** Skills:** - **Core Platform:** - Azure Data Platform (ADF, ADLS, Azure SQL). - Databricks (DLT, Delta Lake, Spark, Workflows, Unity Catalog). - Microsoft Fabric (preferred). - **Engineering:** - Advanced PySpark / Python. - Strong SQL. - Pipeline design (batch + streaming). - **DevOps / Automation:** - Azure DevOps (pipelines, repos, policies). - PowerShell automation. - CI/CD and Infrastructure-as-Code concepts. - **Governance / Data Quality:** - Schema enforcement, data contracts, and data quality frameworks. - Understanding of metadata-driven design.