Data Engineer (Databricks, Python)
Overview
This role focuses on architecting and implementing enterprise-grade Lakehouse solutions using Databricks and Apache Spark. The position involves building scalable batch and real-time data pipelines, leading ML lifecycle management, and managing cloud-native deployments on Microsoft Azure. It requires 10-15 years of experience with deep expertise in Databricks and Data Engineering.
What You'll Do10
- 1Architect and implement enterprise-grade Lakehouse solutions using Databricks.
- 2Design and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL).
- 3Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworks.
- 4Implement and optimize Databricks components: Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and Workflows.
- 5Design large-scale data warehousing solutions with 3NF and dimensional modeling.
- 6Establish data governance, security, and data quality frameworks, including Unity Catalog.
- 7Lead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI).
- 8Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow).
- 9Drive CI/CD, DevOps practices, and performance optimization of Spark workloads.
- 10Provide technical leadership, mentor teams, and ensure successful delivery.
Requirements7
- 110+ years of total experience, with 5+ years specifically in Databricks.
- 2Architecture-level expertise — must understand end-to-end design, not just engineering execution.
- 3Hands-on with Delta Lake, Delta Live Tables, Spark (PySpark/SQL), and Azure Data Factory.
- 4Prior experience working on Azure or Azure DevOps environments.
- 5Has handled US/UK or international client-facing projects.
- 6Comfortable managing and leading a team.
- 7Basic understanding of AI/ML tools and services is a plus.
Who Should Apply
A seasoned Data Engineer with over 10 years of experience, including at least 5 years specializing in Databricks. The ideal candidate has architecture-level expertise, hands-on experience with Delta Lake, Spark, and Azure Data Factory, and has led international client-facing projects. They are comfortable leading teams and have a basic understanding of AI/ML tools.
Salary Insight
Compensation is open to discussion based on experience.
Required Skills
Application Tip
Highlight your Databricks expertise, particularly in Delta Lake, Azure Data Factory, and PySpark, as these are central to the role.
Similar open positions
Explore active roles that match your skills and interests.
Optim Hire
VerifiedData Scientist
We are looking for a seasoned Data Scientist to lead the design and deployment of machine learning models within a modern Python-based cloud architecture. You'll take ownership of the full ML lifecycle, from data ingestion to model scoring, while guiding a team and collaborating with executives. This role combines deep technical expertise with strategic oversight to drive AI initiatives forward.
Optim Hire
VerifiedLead Data Engineer
We're looking for a Lead Data Engineer to design and optimize enterprise-grade data pipelines on Google Cloud Platform. This role centers on building scalable, real-time and batch processing solutions using Apache Beam, Dataflow, Java, and BigQuery. You'll collaborate closely with architects, analysts, and business stakeholders to ensure data quality, performance, and cost efficiency.
Optim Hire
VerifiedProject Lead Data Lakehouse Platform
We're seeking an experienced project lead to guide the PMO function for building and deploying a robust Data Lakehouse platform. In this role, you'll serve as the key liaison between the client, system integrator, and technology teams, ensuring the platform aligns with business goals, security standards, and regulatory requirements—especially within the banking sector. Your focus will be on coordinating delivery, tracking progress, and keeping everyone aligned toward a successful rollout.
Mercor
VerifiedData Engineer — Finance | $60-$85/hr Remote
This role places you inside the finance division of a rapidly expanding AI company, where you'll build and maintain the data infrastructure that powers the CFO's decision-making. As part of a lean, high-impact team, you'll own everything from pipeline construction to troubleshooting, ensuring financial data flows reliably. It's a six-month full-time onsite contract with potential to extend, based in one of three major US tech hubs.
Optim Hire
VerifiedSenior Data Scientist AI ML
This role is for a seasoned data scientist who will build and deploy machine learning solutions specifically aimed at improving media performance. You'll be expected to take complex datasets from advertising or content platforms and turn them into actionable insights. The position demands both technical depth in modeling and the ability to clearly communicate findings to non-technical stakeholders.
Optim Hire
VerifiedSenior Data Scientist
We're looking for a seasoned Data Scientist who can take machine learning models from idea to production. You'll work across the full stack—building, training, and deploying Generative AI and LLM solutions on GCP/Azure/AWS, backed by solid engineering practices like CI/CD and API design. This hybrid role is based in Chennai, Bangalore, Pune, or Coimbatore.