https://outerbounds.com/ logo
Join Slack
Powered by
Out of curiosity, I know there are some undocument...
# ask-metaflow
r
Out of curiosity, I know there are some undocumented config parameters like 
DEFAULT_AWS_CLIENT_PROVIDER
that can be added to your metaflow config. Is it expected that these configurations are not propagated to AWS Batch? Is the metaflow config file getting passed along by other means? Otherwise it seems like we'd be missing a bunch of configs here https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/aws/batch/batch.py#L232-L236
✅ 1
If we are missing configuration values, I'm curious why we wouldn't attach every value specified in a 
config_{METAFLOW_PROFILE}.json
file to the batch job?
a
@rich-elephant-44987 - not every value is useful within the AWS Batch container and we also mutate a few values so we can't carry those forward as is from the config.
Are there specific undocumented features that you expect to work with AWS Batch that aren't working?
r
Yeah, specifically: 
DEFAULT_AWS_CLIENT_PROVIDER
We're looking to add event hooks to the boto client using a custom provider, but it seems like this configuration is not carried forward by Metaflow.
a
We can add that in - would you like to submit a PR?
r
We also wanted to add some custom configurations to work with our custom provider. My thought is that we could have leveraged the metaflow configuration file to automatically hand these parameters along, but that seems like a nonstarter if there's no desire to pass along the entire configuration.
a
You can use the 
@environment
decorator to pass in additional env vars.
If you wouldn't want to pollute the code with 
@environment
you can also set it as an environment variable
r
Right. that's how I'm currently trying to work around the missing config. Contextually it seems like it would have been cleaner to be able to customize low level plugin behavior using metaflow's built in configuration system rather than spreading it across multiple levels of configuration.
👍 1
a
Copy code
export METAFLOW_DECOSPECS=environment:vars=key:value
r
W.r.t. values that are mutated for batch it seems possible to just ignore whatever's set in the config. For all values not explicitly overridden, it doesn't seem unreasonable to pull it from the config.
a
The fundamental issue is that if we use all the values in the metaflow config file and write those to the batch container, there might be hard-to-resolve conflicts. Even today, Metaflow ignores all the fields in the config file that are not directly used within the code base.
r
This might be some expectation mismatch between metaflow and user-defined metaflow-extensions. It seems like it would be more sensible to configure extensions/plugins at the config level rather than through an environment decorator.
For example, say I'm adding a new AWS provider than adds touch-on-read behavior. I can (technically) specify the usage of this provider in the config, albeit not respected by the batch decorator. Now I also want to expose additional configuration for consumers of this extension to modify the frequency of this touch behavior. Ideally I would say something along the lines of:
Copy code
Add the following lines to your configuration:
{
...
"METAFLOW_DEFAULT_AWS_CLIENT_PROVIDER": "boto3_touch",
"METAFLOW_BOTO_TOUCH_DELAY": "3600",
...
}
a
The user-defined metaflow-extensions are rapidly evolving and not part of the stable API at the moment for precisely this reason 🙂
r
Fair enough. Are there any thoughts about improving the configuration system to be more flexible to support more complex extensions?
a
Yes - we are actively thinking through the configuration management story and will have a memo to share soon
d
As a side note, what is this “touch on read” thing? I am curious?
As another side note, default values in an extension should be passed along (so for example if you define those values in the extension they will be present in batch) but not overriden values (at least for batch).
r
One thing that we're required to enforce is the expiration of unused artifacts. In AWS, this (in theory) is easy to enforce at the bucket policy level. Unfortunately, because of how Metaflow skips over any content-addressed objects in S3, we run into lots of funny behavior where something as simple as the code bundle expires out from underneath us.
d
There is a functionality (hidden) to get all the “non default” values. You could technically create or extend a decorator to do what you want. Definitely not in a very supported way but if you are already using extensions you have at least some appetite for fiddling with things. I can point you to the relevant code bits kn a bit if interested. Reiterating though: may break with future releases just like extensions changed a bit in the latest release.
r
So one solution we've been prototyping is to use a custom AWS provider than attaches a hook onto one of the 
GetObject
events that will touch the file in S3 by updating the metadata
I should note that the extension we're trying to use is a customer provider and not a decorator. Just trying to get the provider working in the SFN context has been a real hassle unfortunately.
d
I am very interested in that. We have a similar concern and I was under the impression there is no “touch” but it seems I am mistaken :).
r
There's not an "official" API to touch objects. But I believe there's a way to simulate it.
d
(We have our own custom AWS provider thst does auth so could investigate enhancing it in this way).
r
Unfortunately the challenge I'm running into is just getting batch jobs to use this provider. As discussed above, I've tried setting 
DEFAULT_AWS_CLIENT_PROVIDER
as an environment variable, but even that doesn't seem to work.
Local executions work totally fine. This just break down in the transition from local -> AWS Batch/SFN
d
If you are just trying to pass a value “statically” ou can set it directly in your extension and it should work.
r
At this point, I'm just going through some trial an error to get the batch jobs to even use my custom provider. The dev cycle is a bit tedious because there's a build/deploy process associated with it. At least the local PoC leads me to believe it should be possible.
d
if you have a repo somewhere, I can take a look and make some suggestions. We do this internally and, granted, we do pass more values down to our batch equivalent but in this particular case, there is no need to (the only case where there would be a need to is to, using your example, pass a different value of the timeout for example)
r
The implementation currently exists in a private repo. Let me get back to you on that.
d
sure. anytime. Just ping me here or DM me.
on github, I am romain-intel if you want more limited sharing.
👍 1
the environment decorator should work as well btw. I just checked and, at least for batch, anything in @environment will get passed directly to the container. So I am a bit surprised that is not helping.
r
Ah yeah, it does work, I had forgotten to add the 
METAFLOW_*
prefix.
👍 1
5 Views
Open in Slack
PreviousNext
Powered by