Lead Data Engineer – Databricks #3597260

Join a mission-driven data solutions organization supporting life, annuity, and long-term care business capabilities through modern data architecture, governance, and engineering. This team is responsible for end-to-end data delivery, from source systems through curated analytical products, and operates across multiple agile teams in a collaborative, high-accountability environment. The group is focused on advancing a modern streaming and lakehouse architecture, replacing legacy analytical platforms, and enabling stronger data-driven business outcomes.

This contract-to-hire opportunity offers the chance to play a lead role in a major Databricks implementation and modernization initiative. The role is open to remote or onsite candidates with no stated location preference. You will have the opportunity to expand into advanced Databricks capabilities, including AI, ML, Lakebase, and broader platform optimization work.

• Design, build, test, and support analytical data pipelines across ingestion, standardization, curation, relational modeling, and dimensional modeling layers.
• Lead the implementation of Databricks-based solutions within an established modern data architecture.
• Establish and reinforce data engineering standards, coding practices, reusable frameworks, and design patterns across build teams.
• Drive CI/CD practices and reusable testing approaches to improve delivery quality and release confidence.
• Evaluate new Databricks platform capabilities and determine when they should be adopted to enhance business outcomes.
• Lead Spark performance tuning, platform cost optimization, production monitoring, troubleshooting, and defect resolution.
• Partner with solution architects, agile tech leads, business stakeholders, security, cloud, testing, and data modeling teams.

• 3–5 years of hands-on Databricks experience is required.
• Strong background building analytical pipelines from ingestion through raw, relational/entity relationship, and dimensional model layers.
• Experience with data warehousing, modern analytics, Spark, data governance, reporting/dashboarding ecosystems, and source system integration.
• Proven ability to perform across the full SDLC, including requirements gathering, solution design, build, testing, support, story decomposition, and work estimation.
• Ability to work independently while also mentoring, influencing, and aligning cross-functional stakeholders.
• Experience in Agile delivery environments, including writing high-quality user stories.
• Strong communication, problem-solving, and ambiguity-management skills.
• Azure cloud experience is preferred.
• Insurance, life insurance, annuity, long-term care, or medical data experience is a plus.
• Familiarity with data governance, data quality, SCD2, surrogate keys, dimensions, and analytical data modeling concepts is highly desirable.