Self-Hosting dbt on AWS with Airflow: A Scalable, Secure Alternative
For teams that have outgrown the limitations of managed services or need more control over orchestration and deployment, we’ve built a fully self-hosted dbt stack using Airflow, ECS Fargate, GitHub Actions, and Terraform.
Who Is This For?
- Data teams migrating off dbt Cloud
- Platform engineers building self-managed orchestration
- Organizations needing private network deployments
Why Self-Host dbt?
dbt Cloud is a strong managed platform, but some use cases call for more flexibility. Our self-hosted setup enables:
- Fine-grained orchestration with Airflow
- Custom CI/CD pipelines using GitHub Actions
- Secure Snowflake access within private subnets
- Reproducible deployments with Terraform modules
Our Architecture
- Orchestration: Airflow on ECS Fargate with DAG-based scheduling
- CI/CD: GitHub Actions for PR validation, image builds, and Slim CI
- Infrastructure: Terraform manages all AWS components including ECS, ECR, IAM, VPC, and S3
- Security: OIDC roles for GitHub auth and ECS execution, private networking throughout
Key Outcomes
- No vendor lock-in
- Fully automated CI/CD workflows
- Lower long-term cost at scale
- Improved scheduling visibility and retry logic via Airflow
- Secure, audit-ready deployments for Snowflake environments
Final Thoughts
This architecture isn’t for everyone — but if your team has platform engineering support and needs flexible, scalable orchestration beyond what dbt Cloud offers, this stack delivers full ownership without compromise.
Read the complete blog on Medium or contact us to discuss a self-hosted dbt solution for your team.
