The last step before production cannot always be decided by a workflow.
When preparing a release for one of the projects we operate, we can reach a point where every check is green: the previews are ready, the artifacts have a version, the production diff is available for review, and automation knows how to execute every deployment.
But one question remains:
What needs to be deployed first?
The answer is not always the same.
In this project, changes do not reach production through a single deployment tool.
The IaC layer manages the foundational infrastructure.
Lambda has its own lifecycle with lambroll.
ECS services and jobs are deployed with ecspresso.
Step Functions state machines are packaged and deployed with SAM.
All of these pieces belong to the same system, but they do not always need to change in the same order.
In some releases, SAM has to be deployed before Lambda. In others, a new ECS task definition must be registered before the IaC change that will consume it. There are also changes where the IaC layer has to publish a role, bucket, or reference that another deployment needs first.
At first, these relationships look like a dependency graph we should be able to encode.
The problem is that the graph changes from one release to the next.
A dependency may appear because a resource is being created for the first time. Another may exist only during a migration or while responsibility moves between tools. Once that work is complete, the dependency can disappear or point in the opposite direction.
Representing every case requires more conditions: Does the resource already exist? Has the data moved? Which tool owns it now? Is this a normal release or a hotfix?
Encoding all of that turns the pipeline into another system the team has to test and maintain. Its job is no longer only to deploy changes; it also has to reproduce the context of every release.
We automate the repeatable parts of a release: detecting changes, generating previews, assuming roles, building artifacts, requesting approvals, and executing deployments. But we leave one small decision to the person preparing the release:
What needs to be deployed, and in which order?
That is our last call before production: one explicit decision before control returns to the system.
The problem was not running commands
Running deployment commands is not the difficult part.
The difficulty is everything around them:
- knowing what changed
- showing the impact before applying a change
- keeping preview and deployment capabilities separate
- promoting the same revision across environments
- requiring explicit approval before touching production
- retaining a reasonable way back to a previous version
There is also orchestration.
Each deployment engine understands the resources it manages. It knows how to render, compare, and apply its part of the change, but it does not understand the release’s full intent.
None of them necessarily knows why, in this particular change, one piece has to exist before another.
That context does not always live in a static dependency we can program once and forget.
The contracts behind the release
The system is not held together by one universal deployment tool. It relies on a small set of contracts that remain true across tools, environments, and teams.
A common lifecycle across tools
Instead of forcing every resource through the same tool, we gave the different deployment tools a common lifecycle. Each resource type uses one workflow to decide when to start the process and another reusable workflow to perform the deployment:
pull request -> preview in dev, staging, and production
merge to main -> deploy to dev and staging
draft release -> production previews and deployment ordering
published release -> approved deployment to production
workflow_dispatch -> deploy to a selected environment behind its guardrailsThe preview format depends on the tool—a changeset or a diff—and is posted back to the pull request without being applied. The same revision then moves through development and staging. A published GitHub Release acts as the separate production signal, protected by its approval gate.
Outside the pull request flow, workflow_dispatch lets a developer deploy a selected revision to development or staging for testing. It also provides a production hotfix path without bypassing the production approval gate.
A preview cannot deploy
The workflows authenticate to AWS through OIDC, without long-lived access keys in GitHub. A preview role can create and inspect a changeset, render configuration, and generate a diff, but it cannot apply the change. A separate deployment role can execute, update, or register resources.
The GitHub Environment is attached only to jobs that apply changes. Reviewing a preview does not require approval; deploying to a protected environment can. A person approves an execution with a defined route, environment, artifact, and diff—not a workflow with broad permission to improvise.
The person contributes context. The platform maintains the boundary.
IaC publishes shared dependencies; it does not own every lifecycle
The IaC layer is the source of truth for networking, roles, buckets, image repositories, and other shared resources. In this project, CloudFormation publishes those values as Exports.
Function and container configuration resolve the Exports from Jsonnet, while state machine definitions consume them with ImportValue. IaC can create a function role without managing the function code, or an image repository without deciding when a new image is published.
The tools still depend on one another, but the dependencies are visible without forcing every resource into the same lifecycle.
Artifacts keep their identity
ECS images are tagged with the commit SHA, and ECR does not allow tags to be overwritten. Production never depends on latest or on an image rebuilt later under the same name.
Other resources return to an earlier version by deploying again from a known commit or tag. The mechanism changes by resource, but the contract stays the same: a release leaves enough identity behind to reconstruct what reached production and recover from it.
Anything merged to main is deployable
Releases do not depend on one operator or team. Anyone who needs a change can prepare one, whether it comes from application or infrastructure work. The contract is simple: anything merged to main is ready to deploy.
This does not eliminate human error, but review keeps confidence from resting on one person. DevOps builds the path and guardrails that let the team deploy safely. It does not need to watch every deployment.
Operational context stays with the project
Each project keeps its infrastructure and workflows close to the code it deploys. The same principles remain consistent, but each repository can express its own resources, environments, and exceptions.
A centralized modules repository can also work. But when it tries to cover every account and environment, flags, variables, and count expressions can begin to describe every exception. We accept some duplication so a developer can understand how a project is built and reaches production from one place.
That is not a universal rule. Duplication becomes expensive when every copy needs the same change or starts drifting without intention. Here, it is a trade-off in favor of local clarity.
The last call before production
The person preparing the release chooses the type of SemVer increment. The workflow calculates the next version, compares the last published release tag with the commit we want to promote, and collects the previews for every deployment lifecycle in a draft release.
The draft is the second review point. Before anything is published, the team can inspect the combined change and define the production steps and their order.
Publishing the release activates the production workflows. Each one uses its corresponding artifact and permissions behind the GitHub Environment approval gate. The order remains explicit team coordination, not a dependency hidden inside a script.
This does not make deployment manual. The person does not enter an AWS account, choose credentials, change parameters during execution, or guess which image version is correct. They answer the question the system cannot answer reliably without understanding the intent of the change.
The goal is not “100% automation.” It is knowing what can be previewed, what can apply a change, who approves it, which artifact is being promoted, where manual work remains, and how to recover if something fails.
It is easy to describe this system through its toolchain. But the important part is that a developer can see the impact of a change without permission to apply it, move the same revision through known environments, and make the one decision that still needs context in a small, auditable place.
Production does not need a person to execute every step.
It needs someone who can explain why those steps have to happen in that order.
The system makes that last call visible and safe.