The Environment Problem

Works on My Machine

There is a reason this phrase became a running joke in software development. It captures something real: the environment a piece of software runs in shapes how it behaves, and keeping those environments consistent is genuinely hard.

The typical setup looks something like this. Developers run something local — a mix of services running directly on their machine, some mocked, some pointed at a shared development database. There is a staging environment that roughly resembles production but was last updated six months ago and has accumulated configuration drift that nobody has fully documented. There is production, which is the only environment that actually matters and the one everyone is most afraid to touch. And somewhere in between there might be a QA environment that the testing team shares, a UAT environment for client sign-off, and a handful of ad-hoc environments that someone spun up for a demo and never cleaned up.

Each of these environments has a slightly different configuration, a slightly different version of the infrastructure, and a slightly different set of assumptions baked into the deployment scripts. The further something travels along this chain, the more opportunities there are for it to behave differently from what was tested.

The Root Causes

Environment problems come from a few consistent sources.

Configuration drift. Every manual change made to an environment — a config value adjusted, a dependency updated, a firewall rule added — is a divergence from the intended state. Over time, environments that started identical develop their own personalities. Staging starts behaving in ways that production does not. Bugs appear in one and not the other. Nobody knows exactly when the environments diverged or how.

Shared environments. A shared staging environment is used by every developer on the team simultaneously. Two engineers testing different features at the same time are interfering with each other’s work. A broken deployment from one branch blocks everyone else from testing. Shared environments create a bottleneck where parallel development becomes sequential testing.

Parity gaps. Local development environments are almost never identical to production. They run on a different operating system, use a different database version, skip infrastructure components that are too complex to run locally, and mock things that are real in production. The further local is from production, the less reliable local testing is as a signal.

Promotion friction. Moving a change from one environment to the next — from development to staging, from staging to production — often involves manual steps, environment-specific configuration, and a sequence of handoffs that create both delay and risk. The process is different enough between environments that it introduces its own failure modes.

Branch Environments: The Obvious Solution Nobody Has Made Easy

The natural answer to shared environment problems is to give each branch its own environment. A feature branch gets a full deployment of the application stack, pointed at its own isolated data, accessible for testing without affecting anyone else’s work. When the branch is merged, the environment is torn down.

This is obviously the right model. It enables genuine parallel development — teams can work independently, test in realistic conditions, and get feedback without coordinating around a shared resource.

The reason most teams do not do this is that it is expensive to set up and expensive to run. Spinning up a full environment for every branch requires the infrastructure to support it, the tooling to automate it, and a cost model that does not make it prohibitive. For most teams, the overhead of creating and maintaining branch environments is higher than the overhead of coordinating around a shared staging server.

So teams share staging. Developers wait for each other. Bugs slip through because the testing environment was not clean. The solution everybody knows would work remains too costly to implement.

Propagation Between Environments

Even when environments are well-managed, the process of promoting changes between them is often where things break.

The ideal model is simple: a change that passes in one environment should be deployable to the next environment without modification. The environment should change, not the application. If the production environment requires different configuration than staging — different secrets, different endpoints, different scaling parameters — that should be a property of the environment definition, not something embedded in the deployment artefact.

In practice, this is rarely how it works. Configuration is often baked into build artefacts. Deployment scripts have environment-specific branches. The process of getting something from staging to production involves a checklist of manual steps that someone has to remember to follow in the right order.

This creates a specific failure mode: something works perfectly in staging and breaks in production not because of a code bug but because of a difference in how it was deployed. The fix is a hotfix, not a code change. The root cause is that the environments are not as similar as they appeared.

What Well-Designed Environments Look Like

The characteristics of an environment model that actually works:

Environments are defined, not accumulated. The entire configuration of an environment — providers, infrastructure topology, access controls, scaling parameters — is a definition that can be version-controlled, reviewed, and reproduced. Manual changes to live environments are not the normal workflow.

Parity is structural, not aspirational. Local, staging, and production run the same application code, the same communication model, and the same execution environment. What differs between them is configuration: which provider they run on, which data stores they point at, what scaling limits apply. The application does not know or care which environment it is in.

Branch environments are cheap to create and automatic to clean up. Creating an environment for a feature branch is a one-command or zero-command operation. It inherits the environment definition from the parent branch. When the branch is merged or closed, the environment is cleaned up automatically. Parallel development requires no coordination.

Promotion is a configuration change. Deploying to production means applying the production environment configuration to the same application artefact that was tested in staging. There are no environment-specific build steps, no manual configuration differences, no deployment scripts that behave differently depending on the target. If it worked in staging with the staging configuration, it will work in production with the production configuration.

Sub-environments support feature isolation within a shared environment. For teams that need shared resources — a shared integration environment, a shared demo environment — the environment model should support isolated sub-environments per application or per branch within that shared context. Teams share the environment as a deployment target without sharing state.

The Cost of Getting This Wrong

Environment problems are a tax on every team that has them, but the cost is distributed and invisible. Developers spend time waiting for staging to be available. QA blocks releases because the environment is not in a clean state. Incidents are caused by configuration differences that were never caught in testing. Time is spent debugging issues that turn out to be environment-related rather than code-related.

None of this shows up as a line item. It shows up as velocity that is slower than it should be, releases that are more stressful than they need to be, and bugs that reach production not because the code was wrong but because the conditions the code was tested in were not the conditions it ran in.

Getting environments right is not a luxury feature. It is a multiplier on everything else the team does.