Job Title:

Senior Data Engineer

Overview:

Role Overview
As a Senior Databricks Administrator, you will be responsible for the setup, configuration, administration, and optimization of the Databricks Platform on AWS. This role will play a critical part in managing secure, scalable, and high-performing Databricks environments, with a strong focus on governance, user access management, cost optimization, and platform operations. You will collaborate closely with engineering, infrastructure, and compliance teams to ensure that the Databricks platform meets enterprise data and regulatory requirements.


Key Responsibilities:
1. Databricks Platform Administration (AWS)
Provision, configure, and manage Databricks Workspaces across multiple environments (Dev, UAT, Prod).
Define and enforce cluster policies, compute tagging, and usage controls to optimize cost and resource utilization.
Administer Unity Catalog, including creation and management of catalogs, schemas, access control, and data lineage.
Manage SCIM integrations for user/group provisioning with IdPs like Okta or Azure/AWS SSO.
Configure and maintain CI/CD pipelines for notebook deployment and workflow promotion using GitHub, GitLab, or similar tools.

2. Security, Governance & Compliance
Implement and enforce role-based access controls (RBAC) and fine-grained data permissions using Unity Catalog and Lake Formation.
Ensure auditability and lineage tracking for compliance with Open Banking, GDPR, and PSD2 regulations.
Configure and manage token scopes, secrets management, and credential passthrough for secure access to underlying data.
Work with InfoSec and compliance teams to ensure platform security aligns with regulatory frameworks.

3. Monitoring, Support & Troubleshooting
Monitor workspace performance, cluster health, job execution, and user activity using Databricks native tools, CloudWatch, and third-party integrations.
Provide L2/L3 support for Databricks usage issues, including notebook errors, job failures, and workspace anomalies.
Maintain operational runbooks, automation scripts, and alerting mechanisms for platform health and governance events.

4. Automation & Best Practices
Build and manage Terraform/IaC scripts for environment provisioning and infrastructure consistency.
Define naming conventions, resource tagging standards, and workspace governance guidelines.
Promote standardization of reusable notebook templates, shared cluster configurations, and workflow orchestrations.
Drive internal knowledge sharing, onboarding, and platform enablement sessions.


Must-have Skills
6+ years of experience in Databricks administration on AWS or multi-cloud environments.
Deep understanding of Databricks workspace architecture, Unity Catalog, and cluster configuration best practices.
Strong experience in managing IAM policies, SCIM integration, and access provisioning workflows.
Hands-on experience with monitoring, cost optimization, and governance of large-scale Databricks deployments.
Hands-on experience with infrastructure-as-code (Terraform) and CI/CD pipelines.
Experience with ETL orchestration and collaboration with engineering teams (Databricks Jobs, Workflows, Airflow).


Good-to-Have Skills
Understanding of Delta Lake, Lakehouse architecture, and data governance patterns.
Experience in AWS Glue and database technologies SQL (Amazon Aurora) and NoSQL (MongoDB/DynamoDB)
Exposure to regulated industries such as banking, fintech, or healthcare.
Experience with incident response, audit logging, and security remediation.


Preferred Certifications
Databricks Certified Data Engineer Associate/Professional
Databricks Certified Lakehouse Platform Administrator
AWS Certified Solutions Architect – AssociateTo find US Salary Ranges, visit People Place. Under the Compensation tab, select "Salary Structures." Within the text of "Salary Structures," click on the link "salary structures 2025," through which you will be able to access the salary ranges for each Mastercard job family. For more information regarding US benefits, visit People Place and review the Benefits tab and the Time Off & Leave tab.