Data Engineer - Automation & AI

Design, build, and evolve AWS Glue–based data pipelines using Spark and SQL. 

Translate legacy SQL scripts and stored procedures into AWS Glue pipelines 

Ensure migrated and newly built pipelines meet agreed standards for correctness, performance, and maintainability. 

Apply Generative AI and agent-based techniques to accelerate data engineering tasks, including code generation/refactoring, pipeline dev. and standardisation 

Own the design and implementation of AI-assisted tooling that integrates directly into day-to-day engineering workflows 

Codify successful patterns, reusable tools, and recommended ways of working for scaling beyond the PoC 

Work hands-on with Python and LLM APIs to build pragmatic, internal DE tools 

Design effective prompts & interaction patterns for code generation & transformation 

Evaluate and work with enterprise-grade AI platforms (e.g. AWS Bedrock, Azure AI Foundry) using GPT-4 / Claude-class models 

Define practical rules of thumb and guardrails (e.g. where automation works, where it breaks down, where human intervention is required). 

Work closely with data and platform engineers to (dis)prove automation hypotheses and identify where AI adds real productivity gains vs. noise 

Document outcomes and recommendations from the PoC and provide clear guidance on how AI should (and should not) be used in data engineering at scale