Senior Data Engineer

Design and build scalable, cloud-native data platforms from greenfield to productionImplement near-real-time ingestion pipelines using event-driven patternsDefine and enforce platform standards, including Data Lake / Lakehouse principles, medallion architecture, and data contractsRefactor and optimise existing Spark and PySpark scripts for performance and maintainabilityIntroduce best practices for code quality, testing, and CI/CD across data pipelinesDrive adoption of AI tooling and agentic workflows within the data engineering teamEnsure data quality, observability, and reliability across all pipelines and platformsDevelop self-service tooling and microservices to simplify platform usage for other teams5+ years of professional experience in Data EngineeringStrong Python and SQL development skills for pipeline development and optimisationProficiency in Apache Spark / PySpark, including query optimisation and performance tuningHands-on experience with Databricks (preferred) or SnowflakeExperience with at least one major cloud provider: Azure (preferred), AWS, or GCPExperience with stream processing technologies (Kafka, Spark Structured Streaming)Solid understanding of ETL/ELT patterns, data modelling (dimensional, Data Vault), and data warehousingExperience with orchestration tools (Apache Airflow, Azure Data Factory, or equivalent)Knowledge of Infrastructure as Code (Terraform or equivalent)Understanding of production-grade system requirements: reliability, scalability, observability, and performanceUpper-Intermediate English levelWILL BE A PLUSFamiliarity with RAG pipeline design and LLM integration patternsKnowledge of data governance frameworks and tools (Unity Catalog, Apache Atlas, or similar)Experience with dbt for data transformation and modellingFamiliarity with MLflow, Feature Stores, or ML platform integrationPERSONAL PROFILESelf-driven and proactive in identifying improvementsComfortable working in a fast-paced, innovative environmentStrong problem-solving mindset with attention to detailOpen to experimenting with emerging technologies and approachesAre you passionate about building cutting-edge, AI-ready data platforms from the ground up? We are looking for a Senior Data Engineer to join our Data Engineering Team and lead high-impact, greenfield initiatives.You will work on building modern cloud-native data platforms, migrating on-premises legacy systems to the cloud, and laying the architectural foundation for AI-ready data infrastructure. In this role, you will collaborate closely with Machine Learning, Data Science, and Product teams, serving as a key technical contributor and thought leader. You will also drive R&D efforts around agentic AI architectures, event-driven systems, and LLM-ready data pipelines – turning architectural concepts into production-grade solutions.Originally posted on Himalayas