Senior Data Engineer (Databricks) – Insurance Data Solutions (Remote)
We’re partnering with a leading Life, Annuity, and Long-Term Care insurance organization transforming its data architecture on Databricks.
Databricks expertise and insurance industry experience are REQUIRED. Candidates without hands-on Databricks pipeline-building experience in an insurance environment will not be considered.
You’ll join a high-performing, mission-driven team of 20 across four agile teams, working end to end on data architecture, governance, and engineering from source systems through to data consumers.
The platform is built on a modern streaming architecture supporting transactional and analytical pipelines, grounded in Medallion architecture and LakeFlow/OneLake design principles, replacing legacy analytical platforms with a modern Databricks-based solution.
Required Experience:
- 5+ years hands-on Databricks experience
- Insurance industry experience (must-have)
- Strong data modelling skills (raw, ER, dimensional; SCD2, surrogate keys)
- Medallion architecture experience
- Strong SQL and PySpark
- LakeFlow or OneFlow experience
- AWS or Azure cloud experience
Key responsibilities:
- Design, build, test, and support analytical pipelines hands-on, from ingestion through standardization and curation.
- Build raw, entity-relationship, and dimensional data models.
- Establish and enforce engineering and coding standards, including code reviews.
- Lead CI/CD design and reusable data tests; drive Spark performance and cost optimization.
- Own production monitoring, troubleshooting, and defect resolution.
- Lead the full SDLC – requirements, story decomposition, estimation, and independent delivery.
- Recommend and implement new Databricks features (AI/ML, Lakebase) as the platform matures.
Preferred Experience:
Azure experience | Data governance/quality framework experience | Medical/health data experience | Strong Agile story-writing and stakeholder communication skills
If you have hands-on Databricks experience in insurance and are ready to lead analytical pipeline delivery, apply today or get in touch for a confidential conversation.
