Why One ELT Tool Doesn’t Fit All
We’ve implemented dozens of ELT pipelines across stacks, clients, and clouds — and after working deeply with dlt, Meltano, and Airbyte, it’s clear: these tools aren’t interchangeable. They each shine in specific environments, and trying to force the wrong fit leads to unnecessary complexity.
This isn’t a features checklist. It’s what we’ve learned deploying all three in production, managing connector sprawl, retries, orchestration, and real-world pipeline failures.
Meltano: Modular, GitOps-Ready Pipelines Built Around Singer
What stands out:
- Full CLI orchestration (meltano run)
- Plugin support for dbt, Airflow, Great Expectations, etc.
- A clear project structure that fits GitOps and team collaboration
Where Meltano fits well:
- You want ELT pipelines you can test, lint, and deploy like software
- You have strong DevOps experience
- You’re already using (or plan to use) dbt core
Where it doesn’t:
- You need instant connector access with minimal config
- The Singer tap you need is broken, slow, or poorly maintained
- You want a polished UI — it's still CLI-first
We’ve used Meltano successfully in environments where the data team works like an engineering team, and pipelines are maintained with the same rigor as code.
Airbyte OSS: Broad Connector Coverage, Built for Scale
Airbyte OSS built its reputation on its extensive connector library and intuitive UI, with early versions supporting Docker Compose for simple, VM-based deployment. We’ve run it this way ourselves for lightweight internal syncs and proof-of-concepts.
Today, Airbyte OSS has evolved into a more scalable, production-grade platform. With recent updates, Docker Compose support has been deprecated, and production deployments now rely on Kubernetes — along with supporting services like Postgres, Redis, and Temporal.
Where Airbyte OSS still works well:
- You need access to a wide range of source and destination connectors
- You already operate in a Kubernetes environment
- You’re building or customizing connectors and want a visual UI to manage jobs.
Where Airbyte OSS becomes a challenge:
- Infrastructure needs are higher than dlt (and Meltano if deploying Meltano on ECS vs EKS or equivalent)
- Self-hosting now implies full platform operations — not just “run a container.”
- Operational overhead can increase as sync volume or job complexity scales
When Docker Compose was supported, it offered a lower barrier to entry. With that now deprecated, Kubernetes becomes the practical baseline for new deployments — especially for production use.
dlt: Code-First ELT for Python-Heavy Teams
When teams already write ingestion scripts in Python — often managing things like retries, pagination, state tracking, and orchestration themselves — we’ve seen dlt dramatically simplify their lives.
dlt is a lightweight Python library that adds:
- Native incremental loading and state tracking
- Retry handling and schema evolution
- Easy integration with cron, ECS, Airflow, or serverless jobs
- A clean development experience that feels like writing any other Python module
Where dlt works well:
- You’re already building ingestion logic in Python.
- You want to move away from bespoke scripts toward repeatable, maintainable pipelines.
- You don’t want to introduce new infrastructure or runtime environments.
Where dlt is not a fit:
- You need a lot of prebuilt connectors out of the box.
- Your team doesn’t have Python experience.
- You want a UI or declarative config — dlt is code-first, always.
We reach for dlt when a team is already comfortable with Python and needs to wrap standardization, error handling, and statefulness around custom-built pipelines — without adding infra complexity.
Connector Ecosystem
| Tool |
Strengths |
Tradeoffs |
| dlt |
Flexible for custom APIs; Python-native experience |
Requires writing/maintaining your own connectors |
| Meltano |
Modular via Singer; plugin-friendly for orchestration |
Connector quality varies; heavier YAML config |
| Airbyte OSS |
Broadest prebuilt connector support; active community |
Kubernetes required for production; higher infra overhead |
Development & Infrastructure Experience
| Category |
dlt |
Meltano |
Airbyte OSS |
| Developer Experience |
Clean Python code; fully code-native |
CLI-driven with strong plugin structure |
UI-driven; limited flexibility in orchestration logic |
| Local Development |
Lightweight, fast iteration |
Strong CLI and config-driven setup |
Docker-based, easy to test locally |
| Production Infrastructure |
Minimal (cron, ECS, serverless) |
Moderate (Airflow, ECS, plugins) |
Kubernetes required; multiple services to manage |
| Monitoring |
Bring your own observability stack |
Via plugins or orchestration tools |
Built-in UI with job-level metadata |
Decision Criteria
| Use Case |
Recommended Tool |
| Python-based team using cron, ECS, or serverless |
dlt |
| DevOps-focused data team seeking modular ELT |
Meltano |
| Kubernetes infrastructure with many connector needs |
Airbyte OSS |
| Need low-overhead, fully code-native pipelines |
dlt |
Final Words
These tools reflect distinct philosophies:
- dlt is best when you want clean Python and zero added infra.
- Meltano is for teams that treat pipelines like software and want modularity without lock-in.
- Airbyte OSS is still the fastest path to many connectors — but only if you're ready to run Kubernetes.
The real decision isn't about features — it's about what your team can sustainably operate and evolve.
Next up — How we're using Meltano and Airflow in AWS and then one-click deployments via Terraform.
