Lead Databricks Data Engineer - MLOPs

About Alphanext

Alphanext Technology Solutions is a technology-first global talent partner and product engineering company, helping organisations build scalable technology solutions and deploy highly skilled professionals across domains such as AI, Data Engineering, Cloud, Embedded Systems, and Enterprise Platforms.

With headquarters in London and operational presence across India, Alphanext enables global organizations to hire, build, and scale technology teams faster and more efficiently.

Role Overview

As a Lead Databricks Engineer at KPI Partners supporting Databricks, you will be embedded within customer engagements as a trusted technical advisor. You will apply deep, hands-on Databricks development experience and a strong command of distributed computing on Spark — including Spark runtime internals — to design, build, and optimize production-grade data solutions. You will bring current knowledge across the breadth of the Databricks product and platform, establish CI/CD and MLOps best practices, and work fluently across cloud ecosystems to ensure scalable, performant, and reliable delivery for customers.

Primary Roles & Responsibilities

•            Serve as a lead engineer and trusted technical advisor on customer Databricks engagements, from design through production deployment

•            Deliver hands-on development on Databricks across multiple customer projects

•            Design and optimize distributed-computing workloads on Spark, applying knowledge of Spark runtime internals

•            Implement CI/CD for production deployments

•            Operationalize MLOps workflows to support machine-learning use cases

•            Advise on and apply the breadth of Databricks product and platform features

•            Tune workloads for performance, scalability, and cost efficiency

•            Work across two or more cloud ecosystems (AWS, Azure, GCP), applying deep expertise in at least one

•            Guide customers through best-practice lakehouse implementation and production readiness

•            Collaborate with customer stakeholders and onshore leads overlapping PST hours to ensure timely, high-quality delivery

REQUIRED QUALIFICATIONS & SKILLS

•            8–12 years of experience in data engineering, data platforms, and analytics, including substantial consulting experience.

•            Completed the Databricks Data Engineering Professional certification and the required Databricks classes.

•            Minimum 6–8+ projects delivered with hands-on development experience on Databricks.

•            Working knowledge of two or more common cloud ecosystems (AWS, Azure, GCP), with deep expertise in at least one.

Mandatory Technical Skills

Databricks, Python,  Pyspark, CI/CD, MLOps
MLOps: how to Integration works, Connections, ML works