Senior Data Engineer

Ciklum is looking for a Senior Data Engineer to join our team full-time in Bulgaria. We are a custom product engineering company that supports both multinational organizations and scaling startups to solve their most complex business challenges. With a global team of over 4,000 highly skilled developers, consultants, analysts and product owners, we engineer technology that redefines industries and shapes the way people live. About the role: As a Senior Data Engineer, become a part of a cross-functional development team engineering experiences of tomorrow. This is a critical, hands-on implementation role where you will take ownership of building, operating, and evolving our cutting-edge AIOps and analytics platform. You will be responsible for building and operating a highly scalable, cost-effective, and multi-cloud data solution that processes massive volumes of data. This platform is central to our SaaS product, providing crucial insights, anomaly detection, and (soon) generative AI capabilities to our customers. If you are an expert in taming huge datasets, optimizing data workflows, and a highly skilled implementor for modern data platforms on Kubernetes, this role is for you. Thus, a commitment to collaborative problem solving, sophisticated design, and product quality is essential. Responsibilities: Build & Own: Design, build, and operate our core AIOps and analytics platform on Kubernetes (starting with EKS) Scale Data Ingestion: Implement and optimize large-scale data ingestion pipelines using AWS EMR and Spark to handle massive initial data loads Manage Analytics DB: Own and operate our self-managed OpenSearch cluster on Kubernetes (using the K8s operator), ensuring its performance, reliability, and scalability as our primary analytics database Optimize for Cost & Scale: Design and implement robust data lifecycle management and tiering strategies, automatically moving data to cheaper, object based storage (like S3) to support huge data volumes affordably Implement AIOps: Get hands-on implementing and integrating AI, machine learning, and generative AI into our platform. You will build solutions for anomaly detection (e.g., security threats, budget spikes) and enable users to interact with the platform using natural language prompts, all leveraging OpenSearch native technologies Drive Multi-Cloud Strategy: Evolve the platform's implementation to support multiple cloud providers beyond AWS, enabling our SaaS and BYOC (Bring Your Own Cloud) models Deliver Insights: Continuously improve our data processing and models to provide new and deeper insights to our customers through our custom-built UI Requirements: Senior-Level Experience: Significant (e.g., 7+ years) experience as a hands-on data engineer, with a proven track record in a Staff, Principal, or equivalent senior implementor role Kubernetes Expertise: Deep, hands-on experience building, deploying, and managing data-intensive applications on Kubernetes (EKS preferred) OpenSearch/Elasticsearch Mastery: Proven experience running and scaling OpenSearch (or Elasticsearch) in a large-scale, self-managed production environment. Experience with the K8s operator is a major plus Large-Scale Data Pipelines: Proven experience building, scaling, and optimizing complex, large-scale data pipelines in a production environment Data Optimization & Cost Management: Demonstrable experience in optimizing complex data workflows and implementing data lifecycle/tiering policies to manage and reduce storage costs Cloud Proficiency: Strong knowledge of the AWS ecosystem (EMR, S3, EKS, etc.) Strong Coder: Proficiency in a relevant programming language (e.g., Python, Scala, or Java) Collaboration: Excellent communication and collaboration skills, with a passion for working as part of a high-performing team Desirable: Spark / EMR: Proficiency with Spark for large-scale data processing (e.g., using AWS EMR) IaC: Experience with Infrastructure as Code (IaC), especially Terraform AIOps & ML: Hands-on experience implementing AI/ML models or AIOps solutions (e.g., anomaly detection, predictive analytics) Generative AI: Familiarity with integrating Large Language Models (LLMs) or building generative AI-powered features (e.g., using OpenSearch vector search) Multi-Cloud Architecture: Experience building data platforms that can operate across multiple cloud providers (Azure, GCP) SaaS/BYOC Environments: Experience working on a data platform for a SaaS or BYOC product Streaming Data: Experience with real-time data ingestion tools (like Kafka, Kinesis, or custom preparers) Whats in it for you? Regular salary reviews based on performance Corporate events including webinars, offline parties, and meetups Internal Mobility Program Tailored education path including full access to Udemy and company-paid certifications 25 paid days off including 20 business days of vacation per calendar year plus 5 undocumented sick leave days Additional health insurance 100% company-covered Multisport card with family discounts available About us: At Ciklum, we are always exploring innovations, empowering each other to achieve more, and engineering solutions that matter. With us, youll work with cutting-edge technologies, contribute to impactful projects, and be part of a One Team culture that values collaboration and progress. Since expanding to Bulgaria in 2022, weve been building a fast-growing team that thrives on learning, collaboration, and innovation. Join us on this exciting journey and help shape the future of our delivery center. Want to learn more about us? Follow us on Instagram, Facebook, LinkedIn. Explore, empower, engineer with Ciklum! Interested already? We would love to get to know you! Submit your application. We cant wait to see you at Ciklum.