https://outerbounds.com/ logo
Join Slack
Powered by
Hi all, I've been playing around with <delaying en...
# ask-metaflow
f
Hi all, I've been playing around with delaying environment fetching. In particular I'm trying to use a flow parameter called 
env_name
to define the env a step should be executed with:
Copy code
@named_env(
        name="@{METAFLOW_INIT_ENV_NAME}",
        fetch_at_exec=True,
    )
If I understand the docs correctly, this is how it should be used. Indeed, this does work fine locally and on AWS batch, presumably because it executes this code which assembles the necessary variables from the flow parameters. But when I deploy it as a step-function it instead executes bootstrap_environment (in particular CondaEnvironment.sub_envvars_in_envname without the addl_env argument) which doesn't parse the flow parameters and thus fails with 
metaflow.metaflow_environment.InvalidEnvironmentException: Could not find 'METAFLOW_INIT_ENV_NAME' in the environment -- needed to resolve '@{METAFLOW_INIT_ENV_NAME}'
Is it possible to also include code like this in the bootstrap_environment function?
d
You do seem to be using it correctly. Let me take a look today what can be done to fix it on step functions. It does work internally but we use a different orchestrator.
f
Thank you @dry-beach-38304. We now have a new metaflow deployment that uses the kubernetes/argo stack and were experimenting with this feature again. We're running into exactly the same issue as before with step-functions. When the flow is run locally or 
--with kubernetes
everything works as expected. But when it is executed as an argo-workflow it fails. I would appreciate any guidance here.
d
is it the same error with argo — as in, the missing 
METAFLOW_INIT_ENV_NAME
or something like that?
I think the issue is that, unlike what we do internally, parameters are not passed through environment variables (we pass them through the click argument 
METAFLOW_INIT_…
but argo seems to be passing them with annotations.
let me ask internally how params are passed?
f
is it the same error with argo — as in, the missing 
METAFLOW_INIT_ENV_NAME
or something like that?
Yes, it's the same error 
metaflow.metaflow_environment.InvalidEnvironmentException: Could not find 'METAFLOW_INIT_ENV_NAME' in the environment -- needed to resolve '@{METAFLOW_INIT_ENV_NAME}'
. The error still originates here: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py#L677 which is called from here: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/conda/remote_bootstrap.py#L43. The parameter is not passed down to the sub_envvars_in_envname method and contrary to local execution it is also not stored in an environment variable. I'm a bit unclear what you mean by
argo seems to be passing them with annotations
d
I mean that internally, we use enviornment variables (the 
METAFLOW_INIT_ENV_NAME
in your case) to pass parameter values which is why the approach in there works. In argo, it doesn’t seem that it is passed down with environment variables but with something else. I am getting a bit more detail (I didn’t write the argo part) and most likely I can write a small adapter that will read the parameter values from the right place.
f
That would be great, thank you!
d
still looking into it.
f
Hi there. Just wondering if there was any update on this issue please?
d
Yes — the update is I asked the relevant person to see about adding this to Argo. They are looking into it. There should be no issues for it hopefully.
f
Thank you!
f
That's great to hear, thank you!
@dry-beach-38304 As this would still be a very helpful feature, I wanted to ask again, if there is any movement here 🙂 If I understand it correctly, the approach that you would prefer is to adjust the way the argo workflow templates are created in the metaflow library itself so that they always expose the parameters as environment variables under this adjusted name. If that is the case, that sounds simple enough to me that I can get a PR out (although I see some potential pitfalls concerning naming collisions, e.g. there already is an environment variable called METAFLOW_INIT_SCRIPT, that could get overwritten, if a flow uses a parameter called 
script
). If, on the other hand this change should be contained to the netflix extensions package, then doesn't it have to be done similarly to here by accessing the datastore, retrieving the parameters and exposing them as environment variables for the individual steps that use fetch_at_exec. For that case I don't think I understand the system well enough to make the correct change.
d
I had pinged @thankful-ambulance-42457 about it and he said he would look into it and that he didn’t see any blocker but I haven’t heard anything since. I’ll ask him at our next sync. I could do a workaround otherwise but it would be a bit difficult because it needs to be done real early.
t
Sorry this completely fell off my table last year! 🙇 I'll revisit the issue and hopefully have something up by end of week.
here is an initial draft for reworking the argo parameters into environment variables. Still needs some testing to verify the issue is fixed with this, but param values are at least still working which was my main concern with the rework.
f
Awesome, thank you! @thankful-ambulance-42457 I will also try it out and see if it works for me.
11 Views
Open in Slack
PreviousNext
Powered by