https://outerbounds.com/ logo
Join Slack
Powered by
Is it possible to set environment variables flow-w...
# ask-metaflow
i
Is it possible to set environment variables flow-wide without having to add them to every step?
✅ 1
d
You can set stuff in os.environ at the top of your file. There is no @environment_base though if that is what you are asking. Could be added though I suppose.
i
As is, it seems like we have to specify our AWS creds in each step that interacts with AWS (we are running in a local cluster)
Setting os.environ at the top would only affect the local host's environ, not the kubernetes environ, right?
d
No. It would affect all.
The file would be re-read and executed remotely as well.
i
Oh.... then maybe really misunderstand something, we have this:
Copy code
@environment(
        vars={
            "AWS_DEFAULT_REGION": "us-gov-west-1",
            "AWS_REGION": "us-gov-west-1",
            "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
            "AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
            "METAFLOW_S3_RETRY_COUNT": "0",
            "NVIDIA_VISIBLE_DEVICES": "all"
        }
Which I thought was taking the env vars from the calling environment and passing them into the kubernetes environment
d
yes, that is correct. You could do somehting like 
os.environ[AWS_DEFAULT_REGION] = "us-gov-west-1
to have a similar effect. The ones that wouldn’t work with this approach are the ones that read from the current env (local) and push it out (so the ones for the 
AWS_ACCESS_KEY_ID
and the 
AWS_SECRET_ACCESS_KEY
. For that the 
@environment
is currently the solution (we don’t have a 
@environment_base
. You could do something like 
run --with environment:
(I’d have to check the exact syntax to specify them all) which would apply the environment decorator to every step.
i
I don't follow how when I use 
x=os.environ[x]
in the decorator that pulls from the calling environment but if I were to set 
os.environ[x] = x
at the top of the metaflow script then that would set an environment variable in kubernetes.... is 
os.environ
local in both cases?
d
os.environ
is always local in the sense that it is the environment of whatever is calling the process. When you use it in the 
@enviornment
decorator, it reads from 
os.environ
on the local machine (when you do 
"AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"]
) and then on the remove machine, it basically does: 
os.environ["AWS_ACCESS_KEY_ID"] = <value read on local machine>
. So if you are always setting it to a constant, you don’t need the decorator. In your case though, you do need something like the decorator because you want to read a value form the local machine and then pass it to the remote one. Effectively, if it’s a constant, there is no need to “pass” it (since everyone can know about the constant) but if it is a dynamic value, then there is.
🙌 1
Open in Slack
PreviousNext
Powered by