AWS Lakehouse Data Engineer

• →Build and Operate Data Pipelines (Batch and Streaming)Design and implement batch and streaming ingestion from APIs, relational databases, file drops, event streams, and external partners.
• →Implement, test, and optimize ETL/ELT pipelines using Python and PySpark to produce curated, analytics-ready datasets for reporting, visualization, and machine learning.
• →Implement incremental processing, change data capture (CDC), data contracts, schema validation, and reusable transformation frameworks.
• →Improve pipeline reliability through automated testing, orchestration, monitoring, retry handling, and operational runbooks.
• →Deliver an AWS-Native Lakehouse Data Platform
• →Design and implement a Delta Lakehouse-style data platform using AWS-native services to provide Databricks-like capabilities for data engineering, analysis, and data visualization.
• →Build and manage a scalable lakehouse on Amazon S3 using Apache Iceberg and open columnar formats such as Apache Parquet.
• →Implement SQL-like table reliability for data stored in Amazon S3, including ACID transactions, schema evolution, partition evolution, snapshot isolation, time travel, and rollback capabilities using Apache Iceberg.
• →Enable fast, interactive querying of lakehouse data using AWS-native query and compute services such as Amazon Athena, Amazon EMR, AWS Glue, and Amazon Redshift where appropriate.
• →Optimize performance and cost through partitioning, compaction, file sizing, statistics, caching, lifecycle policies, and efficient separation of compute and storage.
• →Establish standardized development, test, and production environments with consistent configuration and controlled promotion across stages.
• →Metadata, Governance, Access Control, Lineage, and Quality
• →Implement data governance and fine-grained access control using AWS-native services, including AWS Lake Formation, AWS Glue Data Catalog, AWS Identity and Access Management (IAM), AWS Key Management Service (KMS), and related security services.
• →Implement a managed metadata repository for dataset cataloging, ownership, business definitions, tagging, classification, and discoverability.
• →Enable end-to-end lineage from source through transformation and consumption to support auditability, impact analysis, and regulatory requirements.
• →Apply policy-based access, least-privilege permissions, row-, column-, and cell-level controls where required, data classification, retention, encryption, and secure data handling.
• →Build operational data quality checks for freshness, completeness, uniqueness, validity, consistency, and anomaly detection, and publish measurable SLAs/SLOs.
• →AWS Automation, CI/CD, and Operations
• →Implement automated AWS provisioning using Infrastructure as Code (IaC) to create consistent environments and secure-by-default baselines.
• →Build and enhance CI/CD for data pipelines and lakehouse components, including automated tests, security checks, validation gates, packaging, deployment, promotion, and rollback strategies.
• →Implement observability with centralized metrics, logs, traces, alerts, dashboards, runbooks, and incident-response procedures.
• →Continuously evaluate platform performance, scalability, reliability, security, and cost, and implement measurable improvements.
• →Cross-Team Collaboration and Documentation
• →Work closely with data, application, analytics, AI/ML, security, networking, and cloud platform teams to support mission needs and delivery timelines.
• →Maintain high-quality engineering documentation, including architecture diagrams, data models, SOPs, interface specifications, operational runbooks, and secure configuration baselines.
• →Present technical findings, trade-offs, risks, and recommendations clearly to technical and non-technical stakeholders.

What Would Be Nice to Have
Hands-on experience with Databricks, Delta Lake, or migrating Databricks workloads to AWS-native services and Apache Iceberg.

• Experience with AWS Step Functions, Amazon Managed Workflows for Apache Airflow (MWAA), Amazon Kinesis, AWS Database Migration Service (DMS), AWS Lambda, Amazon MSK, or similar ingestion and orchestration services.
• Experience with modern DevOps practices and tools such as Git, Terraform, AWS CloudFormation or AWS CDK, Jenkins, AWS CodePipeline, GitHub Actions, and Docker.
• Experience integrating lakehouse data with business intelligence and visualization tools such as Amazon QuickSight, Tableau, or Power BI.
• Experience using AI-assisted coding tools, such as GitHub Copilot, ChatGPT, Cursor, or Kiro, to accelerate implementation while maintaining code quality, testing, review, privacy, and security controls.
• Knowledge graph and Graph RAG experience, including graph modeling, ontology and taxonomy alignment, entity resolution, relationship extraction, and hybrid retrieval that combines graph traversal with semantic or vector search.