Senior Data Engineer

Key Responsibilities

• Design, develop, test, and maintain scalable data pipelines for batch and real-time data processing.
• Build efficient ETL/ELT workflows to extract, transform, and load data from multiple sources.
• Integrate structured and unstructured data from databases, APIs, cloud platforms, and external systems.
• Monitor pipeline performance, troubleshoot failures, and implement solutions to ensure reliability and availability.
• Automate recurring data processing tasks to improve operational efficiency.

• Develop and maintain scalable data architectures that support analytics and reporting requirements.
• Design and optimize data warehouses, data lakes, and lakehouse solutions.
• Work with relational and NoSQL databases to manage large volumes of data efficiently.
• Implement data partitioning, indexing, and storage optimization techniques to improve performance.
• Support cloud-based data infrastructure and contribute to platform scalability and cost optimization.

• Implement data validation, quality checks, and monitoring mechanisms to maintain data accuracy and consistency.
• Identify and resolve data discrepancies, duplication, and processing issues.
• Follow data security, privacy, access control, and governance standards.
• Maintain technical documentation for pipelines, data models, workflows, and system integrations.

• Collaborate with data analysts, data scientists, product teams, and business stakeholders to define data requirements.
• Translate business needs into technical data engineering solutions.
• Review code, follow engineering best practices, and contribute to continuous improvement.
• Identify opportunities to improve pipeline efficiency, reduce processing time, and optimize infrastructure costs.
• Stay updated on emerging technologies, tools, and industry practices in data engineering.

Required Qualifications

• 3–12 years of experience in data engineering, data platform development, or a related field.
• Strong programming skills in Python, SQL, or other relevant programming languages.
• Understanding of ETL/ELT processes, data modeling, and data warehousing concepts.
• Experience working with relational databases and large-scale data processing systems.
• Strong analytical, problem-solving, and troubleshooting skills.
• Ability to work independently and collaborate effectively with cross-functional teams.

Good-to-Have Skills

• Experience in Data Engineering, including pipeline design, data integration, and data transformation.
• Familiarity with Apache Spark, Kafka, Airflow, or similar data processing and orchestration tools.
• Exposure to cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
• Knowledge of Snowflake, Databricks, BigQuery, Redshift, or other modern data platforms.
• Experience with CI/CD, version control, workflow automation, and data observability tools.