Senior Workflow Orchestration Engineer
About the Role
A top hedgefund is seeking a Senior Airflow Engineer experienced in designing, optimizing, and maintaining scalable workflow orchestration systems. The ideal candidate is highly skilled with Apache Airflow, modern data engineering practices, and cloud‑based infrastructure. The role involves handling complex Airflow challenges, improving reliability, and enhancing data operations within a high‑scale environment. Similar senior roles in the industry emphasize Airflow expertise, Kubernetes familiarity, and strong problem‑solving capabilities.
Responsibilities
- Design, build, and maintain scalable data pipelines orchestrated through Apache Airflow.
- Investigate and resolve complex Airflow issues, including configuration challenges, DAG failures, and performance bottlenecks.
- Improve system observability through monitoring, alerting, and logging enhancements.
- Work cross‑functionally with data, platform, and software engineering teams to architect robust ETL and ELT frameworks.
- Optimize Airflow deployments, scheduling strategies, and resource utilization.
- Contribute to infrastructure improvements across Kubernetes, containerization, and distributed systems.
- Participate in automation, tooling improvements, and internal standards development.
Required Experience
- At least 5 years of experience in data engineering, platform engineering, or related areas.
- Strong hands‑on experience with Apache Airflow, including DAG development and debugging.
- Proficiency in Python for workflow development and automation.
- Experience with Kubernetes, Docker, and cloud environments (AWS, GCP, or Azure).
- Understanding of CI/CD processes and DevOps principles.
- Strong analytical and problem‑solving skills, with the ability to own complex issues end to end.
Preferred Experience
- Background in diagnosing advanced Airflow bugs or unique orchestration challenges.
- Experience supporting high‑availability or mission‑critical pipelines.
- Prior contributions to Airflow tooling or open‑source initiatives.
FAQs
Congratulations, we understand that taking the time to apply is a big step. When you apply, your details go directly to the consultant who is sourcing talent. Due to demand, we may not get back to all applicants that have applied. However, we always keep your CV and details on file so when we see similar roles or see skillsets that drive growth in organisations, we will always reach out to discuss opportunities.
Yes. Even if this role isn’t a perfect match, applying allows us to understand your expertise and ambitions, ensuring you're on our radar for the right opportunity when it arises.
We also work in several ways, firstly we advertise our roles available on our site, however, often due to confidentiality we may not post all. We also work with clients who are more focused on skills and understanding what is required to future-proof their business.
That's why we recommend registering your CV so you can be considered for roles that have yet to be created.
Yes, we help with CV and interview preparation. From customised support on how to optimise your CV to interview preparation and compensation negotiations, we advocate for you throughout your next career move.