Senior Data Engineer Consultant

Keystone Solutions is seeking a Senior Data Engineer for a consultancy mission at a client site. You will be part of a dynamic team working on the Data Exchange Platform (DEP) project, which involves building an on-premise data platform for a governmental organization. This role emphasizes the importance of security, governance, and data sovereignty.

• →Design and deploy data ingestion pipelines from heterogeneous sources: files, APIs, relational databases.
• →Implement pipelines for incremental and streaming ingestion.
• →Ensure reliability, idempotence, and traceability of each pipeline.
• →Develop transformation models on Bronze, Silver, and Gold layers (Medallion architecture).
• →Implement best practices for data historization (e.g., SCD2, upsert, append only).
• →Develop exports to Delta Lake.
• →Manage and maintain Delta Lake tables.
• →Guarantee transactional consistency of data in a multi-domain environment.
• →Implement and maintain orchestration assets.
• →Monitor runs, manage errors, and automate alerts.
• →Push technical and business metadata to the data platform.
• →Maintain end-to-end lineage.
• →Contribute to the drafting of DUA, SLA, and associated governance documents.
• →Develop endpoints to expose the Gold layer to BI dashboards.
• →Collaborate with reporting teams for BI dashboard feeding.

• Proficiency in Python.
• Proficiency in SQL.
• Proven experience with dbt-core (models, snapshots, macros, tests, profiles).
• Good knowledge of DuckDB as an embedded analytical query engine.
• Experience with Delta Lake (ACID transactions, time travel, OPTIMIZE/VACUUM).
• Knowledge of relational databases: MSSQL, PostgreSQL (advanced SQL, CDC).
• Experience with a data orchestrator: Dagster, Airflow, or equivalent.
• Comfortable with Linux/OpenShift and using PowerShell (Windows dev environment).
• Familiarity with governance tools: DataHub, OpenMetadata, or equivalent.
• Experience in BI development.

• Experience with Kafka/Debezium for real-time data capture.
• Knowledge of Spark (Spark SQL, Thrift Server, Beeline).
• Experience with Power BI Report Server (PBIRS) or Apache Superset.
• Awareness of data security in restricted access environments.
• Knowledge of Lakehouse concepts (Medallion architecture), DWH, and Datalake.

• Rigorous and autonomous in managing complex multi-domain pipelines.
• Ability to document technical decisions and simplify for non-technical stakeholders.
• Team spirit.
• Curious and solution-oriented mindset.

• Dagster - Level: Junior - Most recent: Any time
• Data acquisition (ETL, ELT, ...) - Level: Confirmed - Most recent: Any time
• Data architecture - Level: Junior - Most recent: Any time
• data engineering - Level: Confirmed - Most recent: Any time
• Data Governance - Level: Confirmed - Most recent: Any time
• DataHub - Level: Junior - Most recent: Any time
• datalake - Level: Junior - Most recent: Any time
• dbt - Level: Junior - Most recent: Any time
• Kafka - Level: Junior - Most recent: Any time
• Metadata management - Level: Confirmed - Most recent: Any time
• POWER BI - Level: Junior - Most recent: Any time
• Python (from a Data Engineer perspective), Pandas & Apache Spark - Level: Confirmed - Most recent: Any time
• SPARK - Level: Junior - Most recent: Any time
• SQL - Level: Confirmed - Most recent: Any time

Dutch
Level Active knowledge
English
Level Active knowledge
French
Level Native