Kubernetes vs. ECS for Scheduled ETL and BI Workloads

Kubernetes CronJobs handle the basic scheduling, and most consultancies running many pipelines add a workflow engine such as Airflow on top for dependency management between steps, retries, and backfills across historical data. The tradeoff is real operational weight: you're now running a cluster and an orchestration layer on top of it, both of which need patching and monitoring.

The payoff shows up once you're running dozens of interdependent pipelines across several clients: a proper DAG-based scheduler makes dependency management and backfilling historical data far less painful than a pile of independent cron jobs would ever manage on their own.

ECS scheduled tasks, coordinated through Step Functions or EventBridge rules, cover the same ground with less infrastructure to run. For a smaller consultancy managing a handful of client pipelines, this is often the faster path to something reliable, since there's no cluster to operate and Step Functions' visual workflow view doubles as documentation.

The tradeoff shows up at scale: Step Functions state machines get harder to manage as pipeline count and interdependency grow, and you'll eventually be reimplementing pieces of what a dedicated workflow engine gives you for free.

Some consultancies skip container orchestration for their pipeline layer entirely and rely on managed services like AWS Glue or a client's existing data platform vendor, only using Kubernetes or ECS for the pieces those services don't cover, such as custom connectors or a client-facing API. This is worth naming explicitly because it's frequently the right answer, and teams sometimes reach for containers out of habit when a managed service would cost less engineering time and less ongoing attention from your own staff.

Use this approach when a client's pipelines are mostly standard extract-and-load patterns; reserve custom container orchestration for the genuinely custom parts of the work, the ones a managed service was never built to handle.

Data pipelines are a classic case of workloads that should scale to near zero between runs but often don't, because someone left a node pool or a fleet of Fargate tasks provisioned at a fixed size just in case. Say a nightly ETL job takes twenty minutes to run; a cluster or task fleet sized for that twenty minutes but billed for twenty-four hours a day is pure waste.

Across a subscription business's whole infrastructure, hosting costs typically run about 5% of ARR and devops spend adds about 4% more at the median¹²; idle batch compute is one of the more common, avoidable ways a client's bill drifts above that range without anyone noticing until a quarterly review turns it up.

Start with the managed-service question first: does a client's pipeline genuinely need custom container orchestration, or would a managed ETL service cover it with less ongoing maintenance for both of you? If containers are genuinely needed and the client has a handful of pipelines, ECS scheduled tasks are the faster build. If the client's data engineering needs are large and growing, with real cross-pipeline dependencies, Kubernetes with a workflow engine is worth the added operational investment.

Kubernetes vs. AWS ECS vs. Nomad covers a third option if a client's data platform has to stay partly on-premises for data residency reasons.

Pick an approach for a given client in this order:

1. Ask the managed-service question first: does this pipeline genuinely need custom container orchestration, or would a managed ETL service cover it with less maintenance?
2. If containers are needed and the client has a handful of pipelines, use ECS scheduled tasks coordinated by Step Functions or EventBridge rules.
3. Add a workflow engine such as Airflow on Kubernetes only when dependency management, retries, and backfills across many pipelines justify running a cluster.
4. Review provisioned capacity against actual utilization regularly, so idle compute does not drift upward across clients.

A client that starts with three simple pipelines rarely stays there. As a consultancy proves out value, clients ask for more sources, more transformations, and eventually a dashboard refresh cadence tight enough to need real dependency management between steps. What looked like the right call on Step Functions in month one can become a maintenance burden by month twelve once the state machine has grown past what anyone can hold in their head.

Build in a checkpoint, maybe every time pipeline count doubles, to ask whether the original approach still fits or whether it's time to move the client onto a proper workflow engine before the migration itself becomes painful.