RFC: expanding Architect environments

[ryanblock]

Background

Since its earliest days, Architect has taken the approach that modern applications should adhere to the best practices of continuous delivery. Because Architect applications are composed of stateless (or serverless, if you will) cloud infra, that means we can effectively eliminate environment drift. This model increases deployment reliability while reducing iteration time and mean time to fix (MTTF) bugs.

This all informed one of Architect's more powerful, but sometimes controversial opinions: there are three (and only three) blessed, built-in environments:

1. testing - A fast, flexible local dev environment powered by Sandbox that's as close to running real AWS as we can make it, yet faster, and more flexible than any other cloud service emulator
2. staging - A built-in staging environment that uses identical cloud infrastructure and services as the final production environment, managed by infrastructure-as-code
3. production - The final deployment environment delivered to your customers after initial development via testing, and validation and acceptance via staging

Our opinion is that the three environments enumerated above represent the essence of reliable software delivery patterns, and that folks who adopt the Architect approach should be well set up for success. Today, Architect is in production in large organizations, with larger Architect apps maintained by teams of expert developers.

So, what's this issue about?

Feedback

Over the years, we've heard that folks want additional environments and customizations of their environments.

Some teams found a single staging environment to be too stifling, and wanted to run multiple staging environments. (For this, we have arc deploy --name $foo, but that approach has other limitations.)

Some folks just don't want to call their staging environment staging, and would prefer something else (like main, or dev, etc.).

Some folks want a blessed way to spin up many environments; perhaps one per long-lived developer branch (a practice we discourage, but acknowledge some teams employ); or an environment dedicated to demonstrating the effects of a single commit (which can be accomplished with arc deploy --name $foo, but I digress); or an environment to use for percentage rollouts, or as blue-green.

Whatever the reasons, we've heard this feedback, and have begun thinking about if and how to incorporate it into Arc 11 (which, at this time, does not have an estimated ship date).

Technical considerations

The simplicity of Architect's three built-in environments make not only for a simple mental model for folks to learn and deliver reliable software with, but it also makes for relatively easy to follow and debuggable Architect (and, we hope, userland) code. For example, utility functions generally know whether they're running locally or in AWS without having to perform expensive service discovery operations.

Ideally, if we do ultimately identify improvements we can make to Architect environments, I hope such improvements will broadly honor the first-principles approach of having local testing, non-production cloud infra, and production cloud infra. Ideally, environments should be easily identified and readily made use of by existing libraries like @architect/functions, or in userland code that may, for example, key on the a specific or type of environment to branch between using local filesystem reads vs. using live S3 getObject requests.

Ideas

We have some ideas for how we might expand environments in future versions of Architect, but in the interest of not influencing the conversation at the earliest moments, we'd love to get your feedback before sharing those. And, of course, if you think we ought not to mess with how things are, that is also a valid opinion, and we'd like to hear that as well.

Thanks!

[cjoecker]

What would work for me is if I can create different environments and stacks automatically when I run arc deploy -e dev
I would like to choose the name myself instead of having a predefined environment name.
Then, I should be able to have different env variables for every environment I created.
In the plugins/macros, stage should then have the name of the environment name I gave.

That would cover all the use cases I had in my past experience.

[kenhorn]

I agree with Christian's points - multiple environments are useful for many use cases -- and a great power of serverless due to low/zero cost. I'd like to be able to have PR envs (ie to allow pre-merge testing on trunk based), demo envs (different configs / data behind them), and multiple production envs (regions, accounts, per tenant).

Allowing per developer envs, allows low risk experiments / learning for apps that use more than local supports (full s3, step functions, etc)

[filmaj]

Agree with both previous posters: let me configure as many environments as I want. Three by default is fine, but I would like for us to design into architect the assumption that there could be however many (or few) environments.

The configurability of how different resources map to different environments as discussed in #1408 is extremely interesting to me and relevant to this discussion, too.

[kenhorn]

I've finally been using --name recently, every PR now builds a fresh staging style env (and eases per developer envs). I use macros to add cognito, other resources and have not found how to determine the value passed to --name in the macro - this is useful to prefix resources to avoid naming collisions / grouping resources etc.

I'm using (non arc) env vars to pass info the build instead. An alternative to using name is just to rename the project in the arc file, which gives much of the flexibility.

Local Dev is great for BFF but as the app has added step functions, s3 events, event bridge etc I've avoided complex mock service setup and just leverage --direct for quick updates. This mean staging is the first real dev env, hence the need for stable alt envs.