How can I set a default env var for each step while still allowing the user to use the

@environment

decorator? If I use the

@environment

decorator for them, it doesn’t allow them to call

@environment

!

you can set

os.environ['SOME_VAR'] = 'SOME_VALUE'

at the top of the module

Thanks @straight-shampoo-11124, I think I need the second behaviour. I need to pass an API key to each step that is getting executed on AWS Batch.

you don’t want to pass it as a parameter?

I want a default value for the API key that can be changed if you pass in a parameter instead e.g.

@environment(...)  # optionally add other env vars
@my_deco           # automatically applies API Key default here
@step
def step(...)

something like this maybe:

from metaflow import FlowSpec, step, Parameter, environment
from functools import wraps
import os

def mykey(f):
    @wraps(f)
    def func(self):
        env_key = os.environ.get('MYKEY')
        if env_key:
            self.my_key = env_key
        else:
            self.my_key = self.key
        return f(self)
    return func

class OverrideableKey(FlowSpec):

    key = Parameter('key')

    @environment(vars={'MYKEY': 'SOMETHING'})
    @mykey
    @step
    def start(self):
        print('the key is', self.my_key)
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    OverrideableKey()

Does this run the risk of leaking your API key? It seems like if you dump the Flow object, you will see

flow.key = "YOUR_API_KEY"

which seems unsafe

you can use something like AWS Secrets Manager for API keys