Managed Ingestion: Fivetran, Airbyte, and dlt

In plain terms

A data connector packages source-specific extraction and destination loading so that a platform team does not have to implement every API from scratch. The earlier ingestion articles showed what can fail: pagination, checkpoints, updates, schema drift, and retries. Buying a service or using a library changes who maintains those mechanisms. It does not remove the need to verify them for the source and destination you actually use.

This is a learning and verification article, not an account of operating all three products in production. Fivetran and Airbyte are compared using documentation. Here “managed ingestion” includes a hosted service, a platform you operate, and a library whose execution you own.

What a connector does between the API and the warehouse

Evaluate a connector against these six responsibilities. Exact behavior depends on the source, destination, sync mode, product version, and configuration; the list is not a guarantee that every connector implements all of them.

• Extraction against the source’s rules. Pagination, rate limits, retry backoff, authentication renewal, and source-specific behavior. Verify supported endpoints and fields, deletion coverage, and the source versions the connector actually tests.
• Incremental state. A stream checkpoint must be coordinated with successful destination work. Check overlap, late updates, duplicate handling, deletion capture, and replay after failure; a timestamp cursor alone does not close every gap.
• Normalization of nested data. Some connectors flatten objects and create child tables; others retain JSON or use destination-specific layouts. Verify parent keys, array positions, and replacement of children when a parent changes.
• Schema inference and evolution. New fields and type conflicts may add columns, coerce values, create variants, quarantine data, or fail the load. One possible response is a variant column; this is not a universal product behavior.
• Write dispositions. Append, replace, and merge serve different input contracts. Merge requires stable keys, deliberate version ordering, and correct child/deletion handling; a primary key alone does not make every rerun safe.
• A run report. Inspect status, extracted and loaded counts, checkpoint, errors, and freshness. A successful incremental run can legitimately have zero new rows. Alerts must be configured and missing deliveries checked separately.

What it does not do is model. The tables it lands follow the source’s shape (flattened into child tables or kept as nested JSON, depending on the connector and destination): raw-layer tables in the architecture article’s terms, with the source’s names, the source’s grain, and the source’s ideas about what a customer is. The core layer, the joins, the definitions, and the history are still the platform’s work, so connector output still needs explicit analytical models. Coverage is also uneven: a catalog of five hundred connectors says nothing about the quality of the one you need, and an internal API may need a custom source or connector.

Three shapes of managed ingestion

	Fivetran	Airbyte	dlt
What it is	a managed connector service with SaaS and supported hybrid deployments	an open-source platform with a large connector catalog, run yourself or as the vendor’s cloud service	an open-source Python library: the pipeline is code you write and run wherever Python runs
Who writes the connector	vendor-maintained connectors; custom integrations need assigned ownership	the community and the vendor; a connector builder for new APIs	you, from verified sources in the library or from a REST source definition; the library does state, normalization, schema, and loading
Who maintains it when the API changes	the vendor for supported vendor-maintained connectors; assign responsibility for custom code and source-side changes	whoever maintains that connector in the catalog, which varies	you, plus library upgrades and configuration tests
Where it runs	SaaS data plane, or supported hybrid data plane with vendor control plane	a self-managed deployment supported by the selected edition, or their cloud	your scheduler, a container, a notebook, a serverless function
How the bill is shaped	MAR-based; distinct changed identities and connector/plan-specific exclusions apply	self-hosted infrastructure and operations; cloud plan may use volume or compute capacity	free library; you pay for the compute and the warehouse it writes to
Fits when	standard SaaS and database sources, a team that would rather pay than operate, and service cost and support that meet the team’s requirements	the same sources with a preference for control and an operations capacity, or sources only its catalog has	internal and unusual sources, engineers who want the pipeline in their own repository, and anywhere a hosted service cannot reach

The options overlap. One well-supported product may cover a platform’s needs; combining products is useful only when coverage, deployment boundaries, or economics justify the added operations. Custom connectors and connector builders also exist, so an internal API is not automatically a reason to reject a managed platform.

The bill decides the design

Separate rows extracted, distinct row identities observed, changed rows, and billable usage. Fivetran MAR rules distinguish changes, initial loads, re-syncs, and connector-specific cases, including free unchanged rows in qualifying re-imports. Full scans still add source, network, and destination work even when unchanged rows are not billed. Airbyte plans also differ between volume and capacity pricing. Verify the actual plan and connector before estimating cost.

Self-hosted open source moves the money rather than removing it. A platform that runs connectors needs a host, upgrades, a person who notices when a connector’s new version breaks a stream, and the same person when the source’s API changes and the catalog’s connector has not caught up. Those hours are the price, and they are easier to overlook than an invoice.

A decision rule, and where it flips

• Buy a hosted connector for a standard source the vendor maintains well, when the row volume prices below an engineer’s time and the data may leave the network.
• Run an open platform for a source only its catalog covers, or when deployment requirements rule out the available managed or hybrid options, and budget the operations.
• Write with a library for internal systems, unusual authentication, partner APIs, when a full cost and reliability comparison supports owning the code.
• Reassess yearly, because catalogs grow, prices change, and the volume that flipped the decision moves with the business.

Include infrastructure, warehouse work, support, on-call effort, development lead time, security needs, and migration risk. Tiered or capacity pricing may have no single crossover point.

Operating managed ingestion

A bought connector still needs acceptance tests: initial backfill, no-change sync, edits at cursor boundaries, deletes, schema changes, expired credentials, interrupted loads, and recovery. Check freshness and completeness against the source’s expected deliveries rather than only the job’s success flag. Route schema changes and alerts to owners. Restrict credentials and raw data, budget source and destination load, and keep cost per source visible. A staging layer can translate vendor-specific tables into stable names and types, but replacing the connector also requires backfill, state migration, reconciliation, and cutover planning.

The subscription company’s sources, assigned

The subscription company in this section is a hypothetical teaching example, not a real organization. The source list, volumes, and assignments below are illustrative assumptions, not recommendations for any real company.

Source	Shape	Why
Payment provider, support desk, advertising platforms, CRM	hosted connectors	standard sources with well-maintained connectors; tens of thousands of active rows a month each
The application’s Postgres	the CDC pipeline from the database article, or a hosted database connector	the largest volume; priced both ways at the current row count and revisited when it grows
Logistics partner’s SFTP files	library code	a custom manifest needing validation, delivery tracking, and retry-safe loading beyond file transfer
Internal inventory service	library code	an internal API with internal authentication; evaluate custom extraction and supported builders
Product events	the collector and topic from the previous articles	the chosen event path; event/webhook connectors are alternatives with different delivery and latency contracts

Anti-patterns

• Hand-writing the payment provider connector. Building before testing whether a supported connector meets the required fields, freshness, and recovery behavior.
• Buying the internal system connector. Expecting a packaged connector without checking custom connector support.
• Full-refresh syncs on row-based pricing. Unnecessary extraction and load work, with billable impact depending on the actual pricing rules.
• Core models on vendor table names. Every mart breaks the day the connector is replaced.
• “It’s managed” as monitoring. The sync failed on the vendor’s side for a week; the completeness check would have said so on day one.
• Self-hosting to save money without staffing it. The license saved, the operations unpaid, the connector six versions behind.
• Ignoring the variant column. The source changed a type, the connector coped, and the dashboard summed the half that stayed numeric.

References

Documentation checked on 2026-09-12. Pricing and deployment options depend on the current plan and supported connector.

• Fivetran usage-based pricing and MAR exceptions
• Fivetran deployment models
• Airbyte sync modes
• Airbyte pricing plans
• dlt incremental loading
• dlt merge loading
• dlt schema evolution