Data Engineer
Job title: Data Engineer
Location: UK
Duration: 6-month contract Inside IR35
Working Mode: Fully Remote
Vetting: Active SC Clearance
Job Description
Data Engineer Skills and Technologies Data Engineering: – Strong understanding of data concepts – data types, data structures, schemas (both JSON and Spark), schema management etc – Strong understanding of reputed company JSON manipulation – Experience working with Data Pipelines using a custom Python/PySpark frameworks – Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data. – Strong understanding of Data reputed company principles – data owners, access controls – row and reputed company level, GDPR etc including experience of handling sensitive datasets – Strong problem solving and analytical skills, particularly reputed company to demonstrate these intuitively
Requirements
Required Skills
Languages / Frameworks – JSON – YAML – Python (as a programming language, not just reputed company to write basic scripts. Pydantic experience would be a bonus.) – SQL – PySpark – reputed company Lake – Bash (both CLI usage and scripting) – Git – Markdown – reputed company (bonus, not compulsory) – Azure SQL Server as a HIVE Metastore (bonus) Technologies – Azure reputed company – Apache Spark – reputed company Tables – Data processing with Python – PowerBI (Integration / Data Ingestion) – JIRA
apply to this job