Airflow Operations: Retries and Backfills

A scheduled job fails at three in the morning. By the time someone looks, the job has retried twice, a downstream report has run on partial output, and a colleague has proposed rerunning the last two weeks “just to be safe.” Every one of these actions is reasonable in isolation, and each can corrupt data if the pipeline was not built for it. Operating a workflow scheduler is mostly the discipline of answering four questions before acting: which slice of time does this run own, what happens if this step executes twice, who else may be writing to the same place, and what does the system do when one of its own components dies.

This article works through those questions with Apache Airflow. It assumes the concepts from Apache Airflow Fundamentals: a DAG is a graph of task definitions, a DAG run is one execution of it, a task instance is one task within one run, and a scheduled run is assigned a data interval. It builds on the safe-repeat ideas in Building Incremental and Idempotent Pipelines, and it follows Transformation as Code, which explained how a transformation layer decides what a rerun computes. Here the focus shifts from what is computed to when, how many times, and under which failures the computation runs.

The evidence base is the Airflow 3.3.2 documentation, read in early October 2026 (3.3.2 was the newest release on the Python Package Index at that time), plus two small local experiments on Airflow 3.3.2 with Python 3.12 whose outputs appear below. Both experiments used a single local process and a SQLite metadata database; they show how scheduling arithmetic and retry behavior work, not how a production deployment with separate scheduler, workers, and a PostgreSQL metadata database behaves under load. Behavior of unreleased or older versions can differ, and Airflow 3 changed several things that older tutorials describe differently.

Logical time and data intervals

Three different clocks

A run of a scheduled workflow carries several notions of time, and most operational mistakes come from using one where another was meant. The Airflow documentation describes a data interval as the time range a run operates in. For a daily schedule, each interval starts at midnight and ends at the next midnight, and a run is usually scheduled after its interval has ended so that the data for the period is complete. The logical date of a run, which was called the execution date before Airflow 2.2, denotes the start of the data interval and not the time at which the run executes. The glossary entry on data interval and logical date gives the same definitions in a smaller example.

Clock	Meaning	Who sets it	Use it for
Data interval start and end	The half-open range [start, end) of event time that the run is responsible for	The timetable, at run creation	Filtering source rows; naming the output partition; the identity of the logical operation
Run-after time	The earliest moment the scheduler may start the run	The timetable	Reasoning about freshness and about when data should be complete
Wall-clock start of an attempt	When a worker actually begins one try	The system, per attempt	Diagnostics and latency measurement only; never as a filter in the transformation

The third row is the one that breaks reruns. A task that selects “yesterday” by reading the current date produces the right result on the night it runs and a different result when it is retried the next afternoon or rerun for an old interval. The Airflow best-practices page states the same rule in operational terms: tasks should produce the same outcome on every rerun, so read from a specific partition derived from the data interval, never the latest available data, and do not use the current time for critical computation inside a task. The assigned interval is the identity of the work. A delayed attempt must still process the interval it was assigned.

The same cron string can mean two different intervals

Airflow chooses the interval convention through the timetable. A data-interval timetable created from a cron expression assigns each run the span between two cron ticks and makes the run eligible at the end of the span. A trigger timetable created from the same cron expression assigns a run at each tick and, unless an interval is given, a zero-length interval. The Airflow documentation has a section comparing the two, and the difference is easy to miss because both accept the same string. The following emulation asked each timetable what runs exist for a daily midnight UTC schedule that starts on 2024-10-01, with catchup enabled and a scheduler that evaluates at noon UTC on 2024-10-04 (Airflow 3.3.2, calling the timetable API directly):

CronDataIntervalTimetable("0 0 * * *")   catchup=True
  interval 2024-10-01 -> 2024-10-02   run_after 2024-10-02T00:00Z
  interval 2024-10-02 -> 2024-10-03   run_after 2024-10-03T00:00Z
  interval 2024-10-03 -> 2024-10-04   run_after 2024-10-04T00:00Z

CronTriggerTimetable("0 0 * * *")        catchup=True
  interval 2024-10-01 -> 2024-10-01   run_after 2024-10-01T00:00Z
  interval 2024-10-02 -> 2024-10-02   run_after 2024-10-02T00:00Z
  interval 2024-10-03 -> 2024-10-03   run_after 2024-10-03T00:00Z
  interval 2024-10-04 -> 2024-10-04   run_after 2024-10-04T00:00Z

The data-interval timetable produced three runs, each covering a full day that had already ended, and the still-open day of October 4 produced none. The trigger timetable produced four runs whose start and end are the same instant: a task that filters start <= ts < end on such a run selects nothing. A workflow written for the first convention and attached to the second would succeed every day on empty input, which is the worst kind of failure because every status is green. When a transformation depends on an interval, the interval is part of the interface, and the timetable that supplies it should be specified explicitly. The glossary entry on the Airflow timetable covers the daylight-saving and local-calendar cases that this emulation does not.

The emulation covers the scheduling arithmetic only. It does not start a scheduler, create runs in the database, or execute tasks. The documentation adds two cautions that this arithmetic does not show. For manually triggered runs, the data interval depends on the timetable and the trigger path and may differ from the logical date that was supplied; if the logic needs the user-specified date, it should read the logical date explicitly. And eligibility is not completeness: that a run is allowed to start at midnight does not mean all events for the previous day have arrived. Lateness is a separate problem, treated in late data and lookback: the run’s output for an interval may need to be revised after the interval’s run has succeeded.

What the interval is used for

A good rule is to derive three things from the assigned interval and nothing else: the source filter, the destination partition or key, and the identity of the operation for deduplication. If all three come from the interval, a rerun of the same interval reads the same rows, writes to the same place, and is recognized as the same operation. If any one of them comes from the clock, from “latest,” or from a counter that increments per attempt, reruns diverge. For derived windows, such as a trailing seven-day feature, the interval defines the output date while the input range extends backward, and the input range must be derived from the interval as well.

Retries and idempotency

Attempts are not runs

A task instance represents one task in one DAG run. Its state moves through scheduled, queued, running, and then success or failure. When a task fails and retries remain, the documentation describes the state up_for_retry: the task failed but has attempts left and will be rescheduled. The retry count permits an initial attempt plus up to that many retries, and the attempt history is kept; the entry on task instances and retries summarizes the distinction. What a retry does not do is undo anything. Each attempt runs the task body again from the beginning, against whatever the previous attempt left behind.

The consequence is the central operational fact about retries: a failure is reported by the process, not by the destination. A task can write its output, commit, and then lose the connection on the way to acknowledging success. From the scheduler’s viewpoint the attempt failed. From the destination’s viewpoint the data is already there. A retry of such a task will repeat the write. Whether the result is correct depends entirely on what the write does when repeated, which is the definition of idempotency: repeating the same logical operation has the same intended effect as applying it once.

A real run of commit-then-failure

Two tasks in a single DAG each write a daily total of 10 for the interval starting 2024-10-01 to a SQLite table, commit, and then on the first attempt raise a ConnectionError standing in for a lost acknowledgement. One task appends a row. The other upserts keyed on the interval start. Both are configured with two retries and a one-second delay.

@task(retries=2, retry_delay=timedelta(seconds=1))
def load(mode):
    ti = get_current_context()["ti"]
    start = "2024-10-01"            # in a real task: derived from data_interval_start
    db = sqlite3.connect("dest.db")
    db.execute("CREATE TABLE IF NOT EXISTS append_dest(start TEXT, total INTEGER)")
    db.execute("CREATE TABLE IF NOT EXISTS keyed_dest(start TEXT PRIMARY KEY, total INTEGER)")
    with db:                         # one transaction, committed on exit
        if mode == "append":
            db.execute("INSERT INTO append_dest VALUES (?,?)", (start, 10))
        else:
            db.execute("INSERT INTO keyed_dest VALUES (?,?) "
                       "ON CONFLICT(start) DO UPDATE SET total=excluded.total", (start, 10))
    print(f"try_number={ti.try_number} committed mode={mode}")
    if ti.try_number == 1:
        raise ConnectionError("acknowledgement lost after commit")

The DAG was executed with Airflow’s local debugging runner, DAG.test(), which runs the tasks in one process. The log recorded, for each task, try_number=1 committed, then a state update to up_for_retry, then after the delay try_number=2 committed, then success, and finally the run marked successful. The destination tables afterwards:

append_dest: [('2024-10-01', 10), ('2024-10-01', 10)]
keyed_dest:  [('2024-10-01', 10)]

Both tasks ended in success, both runs were green, and the append table holds the day twice: any query that sums it reports 20. The keyed table holds one row because the second write replaced the first with the same value. Nothing in Airflow’s status distinguishes the two outcomes; the difference lives entirely in the destination rule. This is the reason the Airflow best-practices page advises replacing a plain INSERT with an upsert in tasks that may be retried, and reading and writing a specific partition.

The experiment also marks its own limits, and they are worth stating because the keyed result is easy to over-read. The key is the interval start, and the value written was the same on both attempts. If the source data had changed between attempts (a correction arrived), the second attempt would write a different value and replace the first, which is what is wanted for a recomputation and not what is wanted if the first value had been published and consumed. If two attempts ran at the same time, the later commit would win regardless of which was based on newer input. And a side effect outside the database, such as an email or an HTTP call to another system, is not covered by a table’s primary key at all. A keyed write makes one destination safe against repetition; every other effect in the task needs its own rule.

When a retry is the wrong response

Retrying is appropriate when the cause is transient and the operation is safe to repeat. The documentation identifies the cases where it is not. A task can raise AirflowFailException to mark itself failed ignoring any remaining retries, which suits conditions that retrying cannot fix, such as an invalid credential, and AirflowSkipException to mark itself skipped. The 3.3.2 documentation also describes retry policies that choose the action per exception type, with rules that retry with a custom delay, fail immediately, or fall through to the default; it states that AirflowFailException always takes precedence and that a policy cannot raise the retry limit. These facilities were read in the documentation but not exercised in the experiments here. The design point they support is that a uniform count and delay treat an expired password and a brief network outage identically, and operational noise falls when the two are distinguished.

Delay and timeouts shape the retry’s effect on the system. The documentation lists retry_delay, retry_exponential_backoff, and max_retry_delay as the controls for spacing attempts, and exponential backoff explains why a growing wait helps an overloaded dependency recover while synchronized retries from many tasks can still arrive together unless jitter is added. A task also has execution_timeout, the maximum permissible runtime of one execution; exceeding it raises AirflowTaskTimeout, and in the documentation’s sensor example a task that hits this timeout is allowed to retry. Retries and timeouts multiply. As an arithmetic illustration with assumed settings, a task with retries=2, a 30-minute execution_timeout, and a 5-minute retry delay can occupy its slot for up to 3 × 30 + 2 × 5 = 100 minutes before it finally fails. Whether a 100-minute worst case is acceptable is a question about the freshness the consumers were promised, and the schedule’s interval and the timeouts should be set together with that promise in mind.

Concurrent attempts

Retries are sequential by design: the next attempt starts after the previous one has failed. The more dangerous situation is two attempts that overlap. It can arise when an attempt is declared lost while its process is still alive, for example when a worker loses connectivity to the metadata store but continues running, or when someone clears a task that is still running. This is an inference from the general behavior of distributed systems and not a case reproduced here, but the defense is standard. The destination must refuse a stale writer, using a generation number that only increases and is checked atomically with the write, as described in fencing token and demonstrated with a transactional example in the incremental pipelines article. Last-writer-wins on a keyed table, as in the experiment, cannot tell a resumed old attempt from a legitimate new one.

Catchup, backfill, and clearing

Catchup creates the missing runs

The documentation defines catchup as the scheduler’s behavior when intervals exist that have no run: with catchup=False, the default, the scheduler creates a run only for the latest interval when a DAG is first activated or re-enabled; with catchup=True, it creates runs for every interval that has not been run since the last data interval. The entry on Airflow catchup works the daily example through. The earlier emulation shows the three October intervals that catchup would create from the start boundary, and the documentation’s cron example states that a scheduler picking up a daily DAG on 2016-01-02 creates one run, for 2016-01-01 to 2016-01-02, when catchup is off. The same emulation asked for the first run with catchup=False at the actual execution time of the experiment (2026-10-04, 01:09 UTC) and returned the interval 2026-10-03 to 2026-10-04 for the data-interval timetable, a full day that had just ended, and a run at 2026-10-04T00:00Z with equal boundaries for the trigger timetable.

Catchup is a scheduling policy, not a recovery tool. It decides which intervals are eligible to be created automatically and nothing about whether a run’s code is correct for an old interval. The documentation says that a DAG not written to handle its own catchup, “limited to the interval, but instead to Now,” should turn catchup off. In practice this means that enabling catchup is safe only for workflows whose source filter, destination key, and input data all depend on the interval, and for which the old input still exists. Two other parts of catchup surprise people. Catchup also fires when a DAG is paused for a period and then re-enabled: the intervals that elapsed while it was paused become eligible. And it creates runs, not a schedule of when they will complete, so many catch-up runs may start together unless max_active_runs and pools limit them.

Backfill is an explicit request for history

A backfill selects a historical date range and asks for runs in it. Airflow 3 provides this through airflow backfill create, the UI, and the REST API. The documentation describes the options that matter operationally: a reprocess behavior of none (if a run exists for a logical date, create no other, whatever its state), failed (create a new run if the existing one failed), or completed (create a new run if the existing one is completed or failed); a max_active_runs limit that applies to the backfill independently of the DAG’s own setting; an option to run most recent intervals first; and a dry run that prints the dates the backfill would consider. A running or queued latest run is never duplicated, whatever the behavior chosen. The glossary entry on data backfill discusses what a date range alone leaves undefined.

airflow backfill create --dag-id DAG_ID \
    --from-date START_DATE --to-date END_DATE \
    --reprocess-behavior failed \
    --max-active-runs 3

The command shape above is the documented form with a subset of its options, not a command run in the experiments. Its flags make two decisions that deserve to be written down before the command is typed. The reprocess behavior decides which existing results will be replaced: failed creates runs for missing dates, reruns the failed ones, and leaves successes alone, while completed reruns everything in the range, which is correct for a logic fix and wasteful for an outage repair. The concurrency limit decides how fast history is rebuilt and how much load it adds to the sources and destinations shared with the live schedule. Sizing it is arithmetic. With assumed numbers, a backfill of 90 daily intervals at a backfill limit of 3, in which each run takes 20 minutes and nothing else limits throughput, needs at least (90 ÷ 3) × 20 = 600 minutes, ten hours, during which the live runs share the same pools and databases. If that is too long or too heavy, the choice is a higher limit with a pool to cap the shared resource, or a smaller range now and the rest later.

What history a backfill needs

A backfill reruns the DAG’s code for past intervals; it does not restore past inputs. Three requirements follow. The source must still be able to return the old interval’s data, which fails for a table that only holds current state. The code must be the version intended: rerunning an old interval with new logic is a recomputation and not a replay, and whichever it is should be chosen deliberately. And a derived window needs more input than the output range. In the entry on data backfill, a trailing seven-day feature for March 1 through March 7 needs input from February 23 onward; as an illustration, rebuilding only the output dates from input restricted to the same dates silently changes the feature near the start of the range.

Publication order is the other hazard. A backfill that finishes after a newer correction has been published can overwrite it with older logic or older input, even if every individual write is atomic. Coordinate by writing backfill output to a candidate location, validating it, and publishing under a version rule that refuses to replace a newer result, then applying the same fencing idea described above. When the rebuild changes numbers consumers have already seen, announce it as a restatement. The scheduler can create the runs; it cannot decide whether the old figures were already reported.

Clearing, rerunning, and depends_on_past

Rerunning a failed task is done by clearing it. The documentation says clearing a task instance creates a record of it, increments its try number, and sets its state so that the task runs again; options select the past and future instances, upstream and downstream tasks, and the tasks in child and parent DAGs. The preserved history includes the attributes and logs of each try, but related objects such as XComs and rendered template fields are not preserved in that history. Two cautions follow. Clearing is an operation with a scope, and a broad selection can restart a large part of a DAG, including downstream tasks that publish. And because the previous try’s XCom is not kept, a downstream task that depended on it must be able to recompute or re-read its input.

The depends_on_past setting makes a task wait for the previous run’s instance to succeed. It prevents out-of-order state accumulation, for example a running total that needs yesterday’s result. It also makes a single failure block every later run until someone intervenes, so it turns an isolated failure into a growing backlog. It belongs on tasks whose correctness really requires ordering, and the cost follows directly from the rule: a stuck run holds back every run behind it.

A note on completion status

All of these recovery operations meet the same trap in the run status. The documentation explains that a DAG run’s status is decided by the states of its leaf tasks, the tasks with no children, and that a leaf with trigger rule all_done runs regardless of the other tasks and can mark the run successful although something failed upstream. The trigger rule entry gives the same example. After a rerun, confirm success from the validation task and the destination, not from the color of the run.

Task isolation

Isolation in a workflow system means three different things, and a design should say which one it needs.

Process and credential isolation

The Airflow 3 architecture documentation describes the execution path. The scheduler decides what is ready and hands tasks to an executor. A worker never runs task code in its own process: for each task instance it starts a new subprocess, supervises it, and tears it down when it finishes, and nothing is pooled or reused between task instances. The worker holds the task’s short-lived credentials and talks to the Execution API, so the subprocess running the task code never reaches the metadata database directly. A separate Dag processor, required in Airflow 3, parses DAG files, so that author-provided code is not executed in the scheduler’s context. The documentation is explicit about a limit: Airflow does not yet support full multi-tenant features. These properties contain the blast radius of a crashing task (a task that exhausts memory takes down its subprocess, not the scheduler) and keep a task from writing to the orchestrator’s state; they do not protect one team’s tasks from another’s if both run in the same deployment with shared credentials.

Filesystem and data isolation

The best-practices page warns that tasks of one DAG may run on different machines under the Celery and Kubernetes executors, so a file written by one task cannot be assumed visible to the next, and it recommends passing small messages through XCom and larger data through remote storage, with the storage path in XCom. The design consequence for recovery is that a task’s inputs and outputs should be addressed by durable names derived from the interval (for example a path containing the interval start), not by a worker-local scratch location. A retry on a different worker, or a rerun the next week, then finds the same inputs and writes to the same output location. Intermediate output of a failed attempt should be written to a name that cannot be mistaken for a finished result, then published by an atomic rename or table swap after validation, so a crashed attempt leaves garbage that the next attempt overwrites, not a half-written result that a consumer reads.

Resource isolation

Pools, DAG-level limits on active runs, and executor settings limit contention. The entry on Airflow pools explains that a pool limits concurrency for tasks assigned to it, that a task can consume more than one slot, and that a pool does not limit traffic outside Airflow or guarantee fairness between historical and current work. For recovery this matters because a backfill is a burst of work that competes with the live schedule. A dedicated pool for historical work, or a lower per-backfill limit, keeps a repair from starving today’s run. Per-task configuration, such as the container image for a task under the Kubernetes executor, isolates dependency sets; the documentation shows an executor_config example, which was not exercised here.

Scheduler failure and recovery

What the scheduler holds, and what it does not

The scheduler is a long-running service that decides which runs to create and which task instances to send for execution. Its decisions are persisted in the metadata database, which the architecture documentation lists as a required component storing the state of tasks, DAGs, and variables. The documentation describes the scheduling loop: create DAG runs that are due, examine a batch of runs for schedulable task instances or completed runs, and select schedulable task instances, respecting pool and concurrency limits, and enqueue them. Because the state is in the database, a restarted scheduler resumes from it. What is lost with a scheduler process is the loop’s progress, not the history. Tasks already running on workers are separate processes.

The documentation also states how a dead scheduler is handled when several run. Airflow supports multiple schedulers for performance and resilience, using database row-level locks rather than a consensus protocol, and recommends PostgreSQL 12 or later or MySQL 8.0 or later for it. When a scheduler job is detected as dead (the threshold is configurable), the running or queued tasks that it launched are adopted and monitored by another scheduler, and the documentation says that those tasks stay running, so there is no harm in not detecting the death immediately. Detection is governed by the orphaned-task check interval and a health-check threshold.

When a task’s process disappears

A worker can disappear while its task is running. The documentation on task instance heartbeat timeouts describes the symptom: a task instance may remain in the running state although its job is inactive, for example after an out-of-memory kill, a failed liveness probe, or a node scaledown, and Airflow periodically finds these, cleans them up, and marks the task failed or retries it if attempts remain. This path produces exactly the ambiguous outcome discussed under retries: from the system’s viewpoint the attempt died at an unknown point, which may be before, during, or after the destination write. Recovery then depends on the destination’s rule. A keyed or version-checked write makes the retry safe; an append does not.

Failure matrix

The matrix summarizes what each failure leaves behind and what the operator must check. Entries marked as inference reason from the documented design and were not reproduced.

What fails	State left behind	What the system does (documented)	What the operator verifies
Worker process killed mid-task	Task instance still marked running until its heartbeat times out	Heartbeat timeout detection marks it failed or retries it	Whether the destination holds a partial or complete write; that the retry’s write is repeat-safe
Task raises after committing	Output committed, attempt reported failed	Moves to up_for_retry if attempts remain (run in the experiment above)	Duplicate or doubled rows (append: yes, keyed upsert: no)
One scheduler dies, others alive	Tasks it launched may be orphaned	Another scheduler adopts running and queued tasks after the health threshold	Nothing, unless runs queued longer than the freshness promise allows; check the scheduler health metric
All schedulers down	No new runs or enqueues; running tasks may finish on workers	On restart the scheduler resumes from the metadata database; catchup policy decides which missed intervals become runs	Which intervals were missed; whether catchup should create them or a bounded backfill should
Dag processor or bundle unavailable	Serialized DAGs in the database remain; changes are not picked up	The scheduler uses the serialized representation, not DAG files (documented)	That workers can fetch the intended bundle version (inference: a Git bundle with pinned versions avoids skew)
Metadata database lost, restored from backup	State reverts to the backup’s time	Nothing automatic: the runs after the backup point appear never to have happened	Which intervals the destination already contains; rerun them only if the writes are idempotent (inference from the design)
Destination unavailable or slow	Tasks fail or time out repeatedly	Retries consume attempts; pools queue other tasks	Backoff and attempt limits; the pool size; whether failures should stop later intervals

Restoring the database: a worked scenario

The metadata database restore is the failure with the least built-in help, so a worked scenario, illustrative and with assumed times, is useful. Suppose a daily workflow writes keyed rows to a warehouse. The run for the day-9 interval (day 9 00:00 to day 10 00:00) is created at midnight on day 10 and completes at 04:00. The metadata database had last been backed up at 23:00 on day 9, and a storage failure at 09:00 on day 10 forces a restore from that backup. The restored database knows the runs up to the day-8 interval and has no record of the day-9 run. The scheduler, finding an eligible interval with no run, creates it again (with catchup disabled it creates a run only for the latest interval, so if several intervals had been lost, the catchup setting would decide how many come back); the warehouse already holds day 9’s rows. If the workflow upserts by interval key, the rerun replaces them with the same values and the discrepancy disappears. If it appends, the day is doubled. The same logic applies in the other direction: the restored database may record a success for work that the warehouse, restored from a different backup, no longer contains. In that case the scheduler will not recreate the run, and the gap persists until someone finds it. Neither failure is visible in Airflow’s status. The conclusion is that the state of the orchestrator and the state of the destination are two records that can diverge, and a recovery plan must reconcile them, by comparing the intervals present in each, before rerunning anything.

Knowing that something is wrong

Detection is a design decision that Airflow now supports in newer ways. The documentation states that the SLA feature from Airflow 2 was removed in Airflow 3.0 and replaced in 3.1 by Deadline Alerts, which set time thresholds for runs and trigger callbacks when exceeded; deadlines can be relative to a fixed time or to references such as queue time or start time. An experimental setting, max_consecutive_failed_dag_runs_per_dag, auto-pauses a DAG after a number of consecutive failures; pausing stops further failures but, combined with catchup, defines what happens when the DAG is resumed, which should be decided in advance. An alert should identify the DAG, run, task, attempt, interval, error, and destination evidence, and should not expose secrets in logs.

A recovery order that avoids making it worse

The following order is a recommendation derived from the mechanisms above, not a documented procedure. It is arranged so that each step reduces the chance that a later step multiplies damage.

1. Stop the bleeding. Pause the affected DAG or hold downstream publication so that further runs do not propagate bad or partial data. Pausing does not cancel running tasks and, with catchup enabled, resuming will create the intervals that elapsed.
2. Identify the affected intervals. Use the interval, not the date the failure was noticed. Compare the intervals the orchestrator records as successful with the intervals the destination contains.
3. Check the repeat-safety of every task that will run again. For each write, confirm that it is keyed or partition-replacing, that its identity comes from the interval, and that concurrent attempts are rejected.
4. Choose the smallest corrective operation. Clear one failed task for a transient failure; use a backfill with failed behavior to fill missing and failed intervals; use completed only for a deliberate recomputation. Use a dry run first.
5. Limit concurrency and isolate load. Set the backfill limit and use a pool so the repair does not starve the live schedule.
6. Write candidates, validate, then publish. Validate counts and totals against the source, using a method suited to the data size, before publishing in order.
7. Resume and confirm from the destination. Re-enable the schedule, and confirm success from the validation outcome and the destination contents, not from the DAG run status alone.

Two limits remain. First, none of this restores source history that no longer exists; a backfill over a table that now holds only current values reproduces the wrong past with a green status. Second, the experiments in this article demonstrate scheduling arithmetic and retry behavior in a single local process on Airflow 3.3.2. They do not demonstrate scheduler failover, worker loss, or database restore, which are described here from the documentation and from reasoning about the design, and anyone relying on those behaviors in production should test them in a deployment that matches their executor and database.

References

• Apache Airflow 3.3.2, Dag Runs
• Apache Airflow 3.3.2, Timetables
• Apache Airflow 3.3.2, Backfill
• Apache Airflow 3.3.2, Tasks
• Apache Airflow 3.3.2, Architecture Overview
• Apache Airflow 3.3.2, Scheduler
• Apache Airflow 3.3.2, Best Practices
• Apache Airflow 3.3.2, Dags