Skip to main content
search

Orchestration tools rarely make headlines, but they run the world’s data. Apache Airflow 3.4, the newest release, is worth a look even if you are not upgrading this week, because it shows where pipelines are heading.

Airflow 3.4.0 arrived on November 6, 2026, with three headline features: loops for tasks, Dags written in languages other than Python, and a more mature way to run AI agents. Here is what each means for teams building reliable data pipelines.

Key takeaways

  • Task loops let a group of tasks repeat a number of times decided at runtime, with a required cap.
  • Dags can now be written in TypeScript, Java, or Go with no Python file, plus an opt-in YAML format.
  • The Common AI provider runs LLM calls and agents as regular tasks, with cost limits and human approval.
  • Several of the newest pieces are marked experimental, so pin versions before relying on them.

Task Loops: Repeat Until It Is Right

Pipelines have always struggled with “keep trying until the result is good enough.” Airflow 3.4 adds a .loop() call on task groups. Each pass can build on the previous result, an optional until condition stops the loop early, and max_iterations is required so every loop has a hard cap.

That cap matters. If the limit is reached before the condition passes, the final iteration is marked failed, so a stuck loop surfaces as a visible failure. There are limits too: loops cannot be nested or placed inside mapped task groups.

Diagram of an Airflow 3.4 task loop: run, check the until condition, repeat up to max_iterations, fail if the cap is reached
How a bounded task loop runs, based on the Airflow 3.4.0 release post.

Dags in Any Language

Airflow has been Python-first from the start. With the new Dag Importer, entire Dags can be written in TypeScript, Java, or Go, with all three SDKs generally available. An opt-in YAML importer lets you define Dags declaratively with no build step.

For organizations where the data team writes Python and the platform team writes Go or Java, this lowers a real barrier. It fits the trend we described in data democratization: meet people in the tools they already use. Note that the language SDKs and importer are marked experimental.

AI Agents as First-Class Tasks

The Common AI provider, now generally available, lets you run LLM calls and agents as normal Airflow tasks. Agents can use toolsets for SQL, MCP, object storage, and sandboxes. Cost limits can cap tokens, requests, or spend per run.

This is a sensible home for agents. Orchestrators already handle retries, logging, scheduling, and alerts, which are exactly the guardrails unpredictable AI steps need.

Human Approval, Built In

The feature I would highlight for any regulated team: an agent can pause and wait for a person to approve a tool call or the final answer. The task shows up on a Required Actions page and does not hold a worker slot while it waits.

That turns “human in the loop” from a slogan into a pipeline state you can audit.

Should You Upgrade to Apache Airflow 3.4?

  • Review the Significant Changes in the release notes before touching production.
  • Install with the 3.4.0 constraints file for a reproducible setup.
  • Pin exact versions of experimental pieces such as the language SDKs and Dag Importer.
  • Try loops and agent tasks in a staging environment first.

If you run on managed cloud data platforms, check when your provider will offer 3.4.

The Bigger Pattern: Pipelines as Guardrails

Step back and the release tells a story. AI steps are unpredictable, expensive, and sometimes wrong. Schedulers were built to manage exactly that kind of work: retry it, cap it, log it, and escalate to a person when needed.

Expect other orchestration and workflow tools to move the same way. The teams that already treat pipelines as products, with owners, tests, and monitoring, will find it easier to add agents safely.

Quick Questions, Answered

When was Airflow 3.4.0 released? November 6, 2026.

Can I write a Dag without Python now? Yes, in TypeScript, Java, or Go, or declaratively in YAML. These options are marked experimental.

Can an Airflow agent run without limits? You can cap tokens, requests, or spend per run, and require human approval for tool calls or final answers.

Final Thoughts: Orchestration Grows Up Around AI

Airflow 3.4 treats AI agents as one more kind of work to schedule, limit, and review. That is a healthy way to think about them, and it builds on ideas from pipeline optimization that data engineers already know.

Loops with caps, approvals with audit trails, and room for more languages. Pipelines are becoming the control room for AI.

Obada Kraishan

A Computational Social Scientist, Research Scholar, and Software Engineer specializing in machine learning, computational methods, and full-stack development. He leverages these skills to advance research and create innovative web solutions.

Skip to content