Hi folks! I have a StepDecorator where I want to ...
# ask-metaflow
s
Hi folks! I have a StepDecorator where I want to add some stuff to the environment wherever the task is run (which will usually be using
@kubernetes
in my case). I cribbed from what the
@environment
decorator does, and did something like this:
Copy code
def runtime_step_cli(self, cli_args, *_args):
        if self.attributes["some_attribute"]:
            cli_args.env.update({"SOME_ENV_VAR": "SOME_VALUE"})
Then I have a
task_pre_step
that uses that env var. However, I'm not seeing
SOME_ENV_VAR
set in
os.environ
within the task_pre_step when running in a kubernetes pod. I have confirmed that it is present during
step_init
while the task is being started up on the local machine, and the values I'm setting using the
environment
decorator are making it to the pod, but not the ones from my extension. Any ideas what I'm missing here?
q
I'm not familiar with exactly how this works, but I have seen that the
@environment
decorator gets special treatment in various places. When we want a custom decorator to set environment variables, we typically have it modify the environment decorator rather than set the variables directly.
e.g. for argo workflows, Metaflow looks for the
@environment
decorator, adds a bunch of other stuff, and then uses the result in generating a workflow template. I don't think it will find variables set by a custom decorator. https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/argo/argo_workflows.py#L1229-L1436
👀 1
d
hey — your env won’t be pushed this way (the cli_args are only used to launch locally so not passed down).
you can, however, hack it around.
the easiest is probably to inject a
environment
decorator into your step
looking to see how best to do this.
funnily enough, we have been trying to figure out the best way to have decorators inject env vars. We have had this problematic before
👌 1
ok, this will be hacky as hell but should work: • in
step_init
, you can check if you have the decorators (check for one called
environment
. If it is there, you are all good to go and you can just add to
vars
in there and it will propagate around • if it is not, you can do
decorators.append(EnvironmentDecorator(attributes={"vars":…}, statically_defined=self.statically_defined)
or something like that I haven’t tested any of this just yet but something like it or close to it should work.
🙏 1
s
Sorry for the radio silence, thanks @quiet-afternoon-68940 and @dry-beach-38304! Yep, I think this is what's going on! Gonna try that solution and report back.
d
cool. lmk
r
This is how I add env vars to every step: (what you're asking but similar ballpack)
def flow_init(self, flow, graph, environment, flow_datastore, metadata, logger, echo, options):
env_config = {
"NAME": "VALUE"
}
my_decorators = [
f"environment:vars={json.dumps(env_config)}",
]
_attach_decorators(flow, my_decorators)