https://outerbounds.com/ logo
Join Slack
Powered by
What is the recommended way to pass personal envir...
# ask-metaflow
t
What is the recommended way to pass personal environment variables to flows/steps eg. API tokens (used by Neptune) or USER (used by MLFlow)?
✅ 1
v
if they are actual secrets (like API tokens), you may want to use the `@secrets` decorator to handle them securely if they are not secrets, you can pass them in as parameters or if there are many of them, you can include them as a config file with 
IncludeFile
Or you can include them directly as environment variables with 
@environment
s
Any chance of an integration with Azure’s Key Vault sometime soon?
s
Yep - we can definitely look into it. Also, if you would like to contribute a PR, here is the code for equivalent support for AWS Secret Manager
s
Oh thanks - I’ll have a look!
t
(If you saw my previous comment - not sure about anything anymore, need more testing. Will get back to you soon..)
👍 1
Okay I think I found a subtile bug, or at least something that seems quite unexpected to me. I thought I had tried everything regarding passing in environment variable. But the @secrets documentation lead me to the @environment documentation, where I could see that is was in fact possible to just 
@environment(vars={"STEP_VAR": f"{os.getenv('MSG')} from container"})
(this would have been the answer to my original question 🙂). Very strange, since I had not been able to do this (!!), and I was not even able to reproduce the example when I tried to do this loosely. The reason, it turns out, was because I used what I expected to be equivalent, 
os.environ['MSG']
, which raises a KeyError instead of defaulting to 
None
. This matter apparently! Any idea why this is?? 🤔 Bonus info: If I replace the environment decorator above with 
@environment(vars={"STEP_VAR": f'{print("MSG:", os.environ.get("MSG", "DOES NOT EXISTS!!!")) or os.environ.get("MSG")} from container/local'})
then everything works, but it prints “DOES NOT EXISTS!!!” twice for every step. Both the local and the container steps.
@ancient-application-36103 @victorious-lawyer-58417 Do you want me to submit this as an issue on Github?
👀 1
a
@thankful-father-61351 can you help me with your expectation on the correct behavior here? 
os.environ['foo']
will raise a KeyError if 
foo
doesn't exist as an env var
t
@ancient-application-36103 I would expect 
os.environ["FOO"]
to work just as well as 
os.getenv("FOO")
, ie. I would not expect these to behave differently (fails vs runs successfully) in the case where 
FOO
exists as an environment variable in the process that runs the flow.
a
Ah that will be because 
f"{os.getenv('MSG')} from container"}
is also executed locally where the environment variable might not be present
t
What do you mean by “might not”? I am saying that it will not. Even then it is set using eg. 
export MSG=something
.
a
we should fix the example - the decorator hooks execute in such a way that the 
print
statement gets evaluated for every step resulting in multiple prints
let me check the issue with KeyError
adding 
@environment(vars={"STEP_VAR": os.environ["MSG"]})
to a step and running 
MSG=foo python flow.py run
works as expected for me
can you help me with a reference example that doesn't behave as expected - might be easier that way
t
I think the example given here is perfectly fine as example: https://outerbounds.com/docs/set-env-vars-with-decorator/
Follow the steps, but replace 
os.getenv("FOO")
with 
os.environ["FOO"]
a
within the 
read_locally
step or as args to the 
@environment
?
found the issue!
yeah the likely workaround for now would be to use 
os.getenv(...)
while we resolve the issue for remote execution
t
Yeah. It is not a big issue now that I know about it 🙂 It just made me falsely conclude that passing in environment variables into steps dynamically was not at all possible. Happy that I figured it out in the end. Thanks for working tirelessly on this 🙌
Open in Slack
PreviousNext
Powered by