Overview:
We are working with a leading UK-based wealth and investment management firm seeking a skilled Data Engineer to support the build and evolution of their modern data platform. This role will focus on developing scalable data pipelines and contributing to a cloud-native lakehouse architecture.
Key Responsibilities:
- Design, build, and maintain robust data pipelines using Azure-based services
- Develop and optimize data workflows within Databricks
- Implement and manage data ingestion processes, including real-time streaming with Kafka
- Structure and maintain data across bronze, silver, and gold layers within a lakehouse architecture
- Write efficient, scalable SQL for data transformation and analysis
- Collaborate with data analysts, architects, and business stakeholders to deliver high-quality data solutions
- Ensure data quality, governance, and performance best practices are followed
Required Skills & Experience:
- Strong experience with Azure data platform (e.g., Data Factory, Data Lake, Synapse)
- Hands-on expertise with Databricks (PySpark or Spark)
- Solid SQL development skills
- Experience working with Kafka for streaming data pipelines
- Good understanding of lakehouse architecture, including bronze/silver/gold layering concepts
- Experience in financial services or regulated environments is advantageous
Desirable:
- Knowledge of CI/CD practices for data engineering
- Familiarity with data governance and security frameworks
