In our most active repo, the main gate for each PR is the Merge Queue workflow:

• lint
• unit tests
• service tests
• deploy the PR changes to staging and run end-to-end (E2E) tests
• other minor jobs that are out of scope for this post.

A few months ago an average Merge Queue run took 16-22 minutes. Now it's 6-8.

The critical path was deploying to staging and running the E2E tests. Its main jobs:

• Terraform apply
• Docker image build
• ECS service redeploy
• E2E tests

Almost none of the time we recovered came from making our own code faster. It came from finding places where the workflow was waiting on a managed service to do nothing: internal stabilization periods, consistency checks we didn't need, single-threaded cache downloads. Here's where the minutes went.

ECS service redeploys

We were using AWS ECS on Fargate with AWS CodeDeploy for blue-green deployments.

Our average ECS redeploy job took over 5 minutes. Now it's 1.5-2 minutes.

Migrating to ECS rolling deployments

CodeDeploy took an unusually long time to redeploy our services. The deployment duration breakdown showed a consistent 1-2-minute pause after the target group check went green, but before CodeDeploy reported the deployment finished. AWS support confirmed that CodeDeploy has an "internal stabilization period" that we cannot do anything about. We were fighting for every minute, so we moved to ECS native blue-green deployments.

The native blue-green deployments turned out to have the same "internal stabilization period"! It's a newer feature, simpler than CodeDeploy and easier to maintain, but just as slow.

So we went further and looked at plain ECS rolling deployments. That gives up blue-green deploys, which was an acceptable tradeoff for us.

I started testing rolling deployment performance, and it was slow too. As slow as the two previous approaches. I measured deployment time three ways:

• aws ecs wait services-stable
• the reported duration on the ECS service page in the AWS Console
• a custom script that polls DescribeServices every 5 seconds

All three gave different results for the same deployment of a simple single-task service:

1. custom script: 76s
2. AWS Console: 116s
3. aws ecs wait services-stable: 150s

AWS support confirmed the difference comes down to how frequently each "reporter" polls the deployment status. It seems the AWS-managed ones poll less often.

There's one more hidden wait: once the new task set is ready and all the old tasks are gone, AWS still waits 30-40 seconds for the service revision record to disappear. That happens after the deployment is actually done; it's AWS cleaning up its own resources. So our script reports success as soon as the new task set is up and the old one is destroyed, regardless of the service revision status. AWS support confirmed this is safe.

Optimizing rolling deployments

Rolling deployments performed well for single-task services but slowed as the task count increased. I noticed this pattern: we redeploy a service with N tasks, ECS schedules 1 task, deploys it, then schedules the remaining N-1. AWS support confirmed this is expected behavior. It's how ECS verifies image consistency across tasks. Since our image tags are unique and each tag always resolves to a single image, we disabled the check at the task-definition level. That brought multi-task services down to ~70-second deployments.

Terraform apply

We ran Terraform apply via Terraform Cloud for each staging deployment.

It took almost 5 minutes, even when there was nothing to apply. Now it's 30-40 seconds on average.

We did two things.

Migrating from Terraform Cloud to self-hosted GitHub Actions runners

Terraform Cloud is slow. We used it for both execution and state management, and it usually took a few minutes to prepare the environment and initialize before terraform plan even started.

Moving execution to GitHub Actions (GHA) let us cache providers and modules (the .terraform dir) in S3. With a warm runner pool, checking out the Terraform sources, restoring the cache, and running terraform init now takes ~20 seconds, versus 2-3 minutes of preparation before.

The plan itself didn't get any faster. Our GHA runners and Terraform Cloud's infra both run on AWS, so no change there.

One hard-won tip: if you run Terraform inside a GHA workflow, wrap the Terraform job in its own workflow, completely detached. Otherwise, a failure in an unrelated job can cause Terraform to fail in the middle of an apply, leaving an abandoned state lock that has to be cleaned up manually.

Skipping the apply

With initialization fixed, terraform apply still took over 90 seconds even when no Terraform files changed.

Our state has ~3-5k resources depending on the environment. Initially I thought the time went to refreshing all of them, but running with -refresh=false still took ~60 seconds. The main reason: our resource graph has deep branches that Terraform can't parallelize, and rendering all the locally defined data sources (e.g. aws_iam_policy_document) is time-consuming even though it requires no network requests. On top of that, Terraform uploads a new state (and a state lock file) on every apply, even with no changes, which added ~10 seconds over a pure terraform plan.

Most workflow runs change nothing in Terraform, and the Terraform job had become the bottleneck, so we decided to skip the apply. We can't skip the job entirely because downstream jobs depend on the Terraform outputs, so we still run terraform init on every workflow run and only skip the terraform apply step. The job now takes 30-40 seconds when there's nothing to apply.

Why didn't we skip it from the very beginning?

We like keeping infrastructure in sync with Terraform configuration. There's always a chance someone changes something manually, leading to configuration drift. Running Terraform on every merge mitigates that. Skipping the apply raises the drift risk, so we configured a scheduled GHA workflow that runs terraform apply nightly.

Docker build

Yes, it's all about caching.

We went from 4-5 minutes to 40-90 seconds.

But it took a few attempts to set up the cache correctly.

Why S3 cache was the wrong choice

The obvious choice for me was the S3 cache backend, since we run docker build on self-hosted GHA runners in AWS. With high-throughput EBS volumes and a multi-gigabit path to S3, I expected near-instant cache restores, even for big layers (>500MB).

The reality was different. It took over 100 seconds just to fetch a cache layer and unpack it. Docker's S3 cache driver is single-threaded, so you're limited to the throughput of a single TCP connection. We got 40-60 MB/s, nowhere near the gigabits I expected.

A dedicated runner pool with local cache

Since network transfer and unpacking were the bottleneck, the fastest cache is the one you don't download at all. Docker keeps its build cache on local disk anyway, and our runners have fast EBS volumes. The problem was that our runner pool is shared: a Docker build lands on a random runner, so the chance that this exact runner already has the cache from a previous build is low.

So we created a small, dedicated runner pool that runs only Docker builds. With just a handful of runners, most builds land on a machine that has already built the image. The runners scale in during off-peak hours, so fresh runners join with a cold cache, and we still get occasional misses. But most runs get a local cache hit, and those builds take ~30 seconds: nothing to fetch, nothing to unpack, the layers are already on disk.

ECR as a fallback cache

To make cache misses on fresh runners less painful, we added a fallback layer: the registry cache backend pointed at ECR. The registry driver is single-threaded during download too, but it still performed noticeably better than S3: ~80 seconds vs. ~120. Probably better network throughput, and maybe a cache format that needs less post-download processing.

So the design: use the local cache if available, otherwise fall back to ECR (and fill the local cache for the next runs).

After rolling this out, performance was still worse than we expected. The root cause was a limitation in the registry cache: layers overwrite each other when stored under the same tag. Two concurrent builds push their cache to the same tag; one wins, and the next build gets a cache miss. We switched to multiple tags per cache save/restore operation, keyed on the package.json hash since the dependency install layer is our biggest one. The builds stopped clobbering each other's cache.

The pattern

Every one of these wins was the same discovery in a different costume: the workflow was waiting on a managed service that had already finished the real work. CodeDeploy's stabilization period, ECS's revision cleanup and consistency check, Terraform Cloud's environment prep, a single-threaded cache download. Vendor defaults favor caution. Measure the durations yourself, push support for the why, and turn off the parts you can prove you don't need.