Hello, We are trying to deploy flows on AWS and we want to add environment variables (or simply variables) dependant on which environment we are deploying. For example, we want our flow to interact with a staging API in staging, and the production one for production (so the URL parameter would be different). We would like to attach those environmental variables at the moment we deploy the flow on step functions. • We already looked at .env : The problem we have is that when deploying to Step Functions, the .env file is not copied and automatically used. We would have to use docker images, and make the image dependant on the environment ? • @environment décorator : We would like the environment variables to be set to every step, and not having to set it at every step. • Parameters : We would like to avoid having to use the same variable each time we call the flow. The variables would always be the same for all flows, and mostly attached to the deployment environment. • We would like not having to use custom docker images and would prefer to just use the default batch instance. Maybe we do not really understand the philosophy of the tool for this particular case ? Thank you !

@stocky-fountain-57774 how are you using

.env

?

For .env, we just put a file in our local repository, using dotenv, it worked locally. But when deploying it to AWS, it doesn't use the .env file to use it on the cloud. We supposed it was the case and indeed it was. We didn't want to use docker images.

If you specify —package-suffixes=.env - the file will be copied into Batch too

Oh nice ! Didn't know that ! Definitely will try it thanks !

another option could be to leverage

@secrets

(docs) -- you can create a secrets manager entry for each of your environments that contains the corresponding environment variables for it when deploying the flows, you can inject those secrets onto the steps, e.g. for production

python flow.py --production --with 'secrets:sources=["prod"]' step-functions create

if you have a separate namespace for staging, could likewise create a non-prod deployment with a staging secret

I tried to use --package-suffixes=.env at deployment, but It didn't seem to have copied the file to AWS. Here is the command I use :

python flow_dotenv.py --environment=pypi --package-suffixes=.env step-functions create

When I just run the flow, even without specifying --package-suffixes=.env, it gets the env value. But when launching the flow from Step Functions, It is absent. With run I have :

secret message: secret_value from a container

And with Step Functions :

secret message: None from a container

Is it supposed to work with step-functions create ?

if you're using

--package-suffixes

it will only grab those files that are in the same dir or subdirs as the flow, not parent dirs (you could work around that with symlinks however) you can verify what files will be included in the code artifact by running

package list

, e.g.

python flow_dotenv.py --environment=pypi --package-suffixes=.env package list

that said, it really doesn't sound like you should be doing that, as that's implying you have production secrets checked into your repo as an

.env

file – would highly encourage using

@secrets

and then you can isolate those environment variables for staging/prod and provision them as appropriate for each deployment

The example I use is just copied from the metaflow documentation, we don't really need secret values, mostly simple parameters. Would you still recommend

@secrets

? Using package list I get an enormous number of files, using grep doesn't work (I'm looking for a way to achieve this), but there doesn't seem to be my .env file. It is in the same folder as the flow so I don't really understand.

this seems like a bug in metaflow. we are looking into it

re

we don't really need secret values, mostly simple parameters. Would you still recommend

@secrets

?

it's ultimately just injecting environment variables to the steps at runtime, whether or not those are "secret" in the sense that they're sensitive isn't a requirement – both should work and just wanted to call out the options 🙂 as an example that I've found to work pretty well: • locally we have

.env

files that are automatically loaded using

direnv

for the environment variables that configure runtime behavior, API endpoints, keys, local DB creds, etc • remotely we have an

@secret

that mimics those environment variables, but with the values corresponding to the respective environment (staging, prod)

opened issue for tracking and further discussion on this: https://github.com/Netflix/metaflow/issues/1741