The Prod’, Stag’, Dev’ Environments Misconception
“You can test your application against my development environment.”
“The Jenkins build server is not “production” production. It doesn’t serve a production workload to our clients. It’s okay if it’s down.”
Do these conversations sound confusingly familiar? Do you know why you are confused by these words? If you are confused, read on….
This post aims to write about my thoughts in trying to clarify some of the concepts and relationships around the types of environments. I hope other software engineers and architects (DevOps or otherwise) gain the same clarify when they build their service-oriented-architecture or microservices after reading this post. We should avoid common misconceptions and bad practices that arise due to misunderstanding.
It is essential that any software engineer or architect should know the difference between the production, staging and development environment and its purposes. There are many articles describing what they are so I shall assume you understand these environments too:
But beyond knowing its definitions, it is especially important, in a microservice or a service-oriented architecture (SOA), that we also understand the relationship between our environments and our users’ environments. It cringes me when someone tells me to test my development code against their development code. Do you see 2 moving parts here?
Do you know how it should be set up? Do you know why? Do you know which environment faces users, and which ones don’t?
Perhaps sometimes we use terms too loosely and end up misunderstanding each other. Words are limited. Maybe diagrams can help.
Common mishaps:
This overall use of tools in the CICD pipeline seems perfectly fine and logical to get the job done. But when the pipeline is not properly treated as a production environment, the entire pipeline is fragile and there are many potential parts that will break during deployment.
In this above example, when every component, except the final box, of the CICD pipeline is treated with a “development” environment mindset, the “production” label is only applied to the final box because that’s where the money-making application is hosted on and the workloads are customer-facing. It is as though everything along the pipeline is not treated like production grade — meaning they are allowed to break because it doesn’t directly affect the clients’ services.
Similar analogy — It is as though the water pipes serving water to your house do not have proper quality pipes from the water distillation plants to the distribution points. High-quality pipes are only installed from the nearest distribution points to your house because it is customer-facing.
You should be concerned with the water (and software) quality.
The Approach to Microservices and Environments
All microservices should have at least 2 (or 3) environments: dev (and staging) and prod. As a team providing a service (regardless if it is custom code or buying it off-the-shelve), only expose your prod endpoint to users (other internal teams’ applications consuming your services, external clients, etc.). This production endpoint should stay highly available so that users can have a reliable service to test against. This is critical in a service-oriented architecture as teams work independently in a loosely coupled way but service integration remains tight.
Please reserve the dev and staging environments for YOUR OWN TESTING of infrastructure and changes. This is where you find fault in your own environments before pushing to production. The same principle applies to whether the hosted application is COTS or proprietary custom code. Even changing a set of configurations requires testing in your dev environment before pushing to prod.
Never offer to let your users test against your dev (staging) environment. It is for YOUR OWN development purposes. It is not for your users’ development purposes. The users should always be testing against a stable interface like your prod environment.
The confusion comes in when users set up different types of environments in your prod environment. Then you may be tempted to think about hosting your users’ dev environment in your dev environment in order to save resources. No, stop! This is where it starts to smell of bad service-oriented architecture.
Similar analogy — AWS does not expose its development environment to us, right? Plainly, it’s the same logic. We still create 3 AWS accounts — 1 for each of our own purpose. Get it?
What CICD Pipelines Should Look Like
Every microservice should be interfacing with the dependent service’s production endpoint. I don’t how to say this more plainly but it is about perspective. As a consumer of a microservice, create separate environments on the production environment of that microservice.
In this case, microservice B’s test to upgrade to v1.2, which is in microservice B’s Dev environment, is tested against microservice A’s prod environment (v1.0) and should not be tested against microservice A’s dev environment (v1.2).
I hope my sharing helps clarify things a little better in this microservice world where everything’s a nested world of mesh. I am not an expert in SRE but I also think this aligns with how SRE works within a large enterprise to ensure that the microservice for each team stays up and reliable for other consumers to use.
Of course, from a security point of view, we NEVER ever want to expose our dev and staging environments for any potential vulnerabilities before testing them ourselves. So, push to production what is already tested and hardened.
