Job Description

Job Title: Technical Lead - Data Engineering
Location: London/Hybrid- Onsite 2-3 days per week, with flexibility to work remotely once up and running
Salary/Rate: Up to 580 per day INSIDE IR35
Start Date: As soon as possible
Job Type: 12 month contract initially + extensions

Company Introduction

Coltech is partnered with a prestigious consultancy client who are looking for an experienced Technical Lead in Data Engineering to lead the development of the data Lakehouse platform

Job Responsibilities

• Lead the design, development, and implementation of data engineering solutions for our data Lakehouse platform.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Provide technical leadership, mentoring, and coaching to the data engineering team.
• Ensure best practices in data engineering, including data modelling, ETL processes, and data quality.
• Optimize big data workloads in Spark and other big data technologies.
• Manage CI/CD pipelines using Azure DevOps or Git.
• Develop and maintain event-driven pipelines using .NET and other relevant technologies.
• Implement and manage Databricks Unity Catalog and other data governance tools.
• Deliver projects in the streaming data world using Kafka, KSQL DB, and similar technologies.
• Utilize reporting tools such as Power BI and Qlik for data visualization and reporting.

Required Skills/Experience:

• Extensive experience with Databricks, Apache Spark, and data lakes.
• Proficiency in Python and SQL.
• Experience with CI/CD processes using Azure DevOps or Git.
• Knowledge of streaming data technologies such as Kafka and KSQL DB.
• Familiarity with reporting tools like Power BI and Qlik.
• Software engineering experience with .NET for event-driven pipelines and automation testing.
• Experience with Databricks Unity Catalog.
• Strong skills in data modelling, including Snowflake modelling, star schema modelling, and other techniques.
• Ability to optimize big data workloads in Spark.

Desired Skills/Experience:

• Experience in delivering projects in the streaming data world.
• Understanding of data governance and data quality best practices.
• Familiarity with automation testing frameworks.
• Experience working in Energy trading related projects.

Apply now for immediate consideration.