Job Description

Data Engineer

4 months - with extensions

Remote

Active SC clearance required

£640 per day inside ir35

REQUIRED

Strong understanding of data concepts - data types, data structures, schemas (both JSON and Spark), schema management etc

Strong understanding of complex JSON manipulation

Experience working with Data Pipelines using a custom Python/PySpark frameworks

Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.

Strong understanding of Data Security principles - data owners, access controls - row and column level, GDPR etc including experience of handling sensitive datasets

Strong problem solving and analytical skills, particularly able to demonstrate these intuitively (able to work a problem out, not follow a work instruction to resolve)

Experience working in a support role would be beneficial, particularly able to demonstrate incident triage and handling skills/knowledge (SLAs etc)

Fundamental linux system administration knowledge - ssh keys and config etc, Bash CLI and scripting, Environment variables

Experience using browser based IDEs (Jupyter Notebooks, RStudio etc)

Experience working in a dynamic Agile environment (SAFE, scrum, sprints, JIRA etc)

Python (as a programming language, not just being able to write basic scripts)LANGUAGES / FRAMEWORKS

JSON

YAML

Python (as a programming language, not just able to write basic scripts)

Pydantic experience DESIRABLE

SQL

PySpark

Delta Lake

Bash (both CLI usage and scripting)

Git

Markdown

Scala DESIRABLE

Azure SQL Server as a HIVE Metastore DESIRABLETECHNOLOGIES

Azure Databricks

Apache Spark

Delta Tables

Data processing with Python

PowerBI (Integration / Data Ingestion)

JIRAIf you meet the above requirements, please apply for the vacancy to be contacted by an Experis Consultant. If you haven't been contacted within 2 weeks of application, please consider the vacancy closed