Is there any way to control the order of multiple metaflow_extension packages? (it doesn't seem to be alphabetical) We'd like to be able to call the config of netflix_ext after our custom extension (we set variables via ssm but because netflix_ext is called before our package, the configuration fails)

Yes. There is. It respects dependency order. The easiest way is to make your extension depend on Netflix_ext and that should work. If this mechanism isn’t satisfactory for you, I can do something else as well. It was kind of a corner case so didn’t spend a ton of time on it.

It is a dependency - that's how we're including it but that might be the problem - dependency implies that netflix_ext would be first whereas we want it second

Ah. Sorry. I had just woken up and read it backwards.

easily done

Could you provide a bit more detail on what you are trying to do?

in our custom extension we set variables by pulling them from AWS ssm e.g.

DATASTORE_SYSROOT_S3 = from_aws("DATASTORE_SYSROOT_S3")

This has worked until now and avoids the need for lots of config/env vars and allows us to easily switch between dev/test etc by just setting one env var The problem I'm hitting is that when netflix_ext is trying to set CONDA_S3ROOT from DATASTORE_SYSROOT_S3 and DATASTORE_SYSROOT_S3 hasn't been set yet because our module config hasn't run yet it then exits so no opportunity to set later (although ideally wouldn't really want that - had a similar problem with SERVICE_HEADERS and SERVICE_AUTH_KEY having to replicate the logic from the default)

Ok this helps. Let me give it a bit of thought but I now understand whr you are trying to do.

we set up the ssm in our (cdk based) IaC and just have one variable MF_PROFILE to say which environment we're using so our config effectively replaces from_conf with from_aws it's a bit more involved to make sure it easily works with local and localstack but not much (the /metaflow-service prefix shouldn't be hard coded really)

def from_aws(param, default_value=None):

"""

First try to pull value from environment, then from metaflow config JSON, then aws ssm

"""

from botocore.exceptions import ClientError

if MF_PROFILE is None or MF_PROFILE == "None":

return from_conf(param, default_value)

def_value = default_value

lookup = param.replace("_", "-")

p_name = f"/metaflow-service/{MF_PROFILE}/METAFLOW-{lookup}"

try:

p_value = ssm.get_parameter(Name=p_name, WithDecryption=True)

def_value = p_value["Parameter"]["Value"]

except ClientError as pnf:

raise MetaflowException(f"Failed to find {p_name}: {pnf.response['Error']}")

return from_conf(param, def_value)

The “stupid” solution (but that will work for now), is just to also, in your config override, set:

CONDA_S3ROOT = from_aws("CONDA_S3ROOT", os.path.join(DATASTORE_SYSROOT_S3, "conda_env") if DATASTORE_SYSROOT_S3 else None)

or something to that nature. This will “reset” the value to what you want it to be and I think that should work. The ordering question is a bit of a tough one. Determinining “dependent” configs is not easy. Our fairly limited use case was that we wanted to override configurations from the basic metaflow and potentially from previously loaded modules but this use case (totally valid) was not something I had thought about.

thanks - I'm happy enough with doing that - I know I'm stretching the boundaries a bit here! work around is simple enough and it could get really messy trying to do something more complex

stretching boundaries is good 🙂. Definitely continue asking these questions but ya, for this particular case, I feel the simplest is what I suggested and implementing something more generic could get hairy. I did make it so that it is deterministic at least (the loading order). I had considered something where the user could specify what they wanted loaded first and ordering constraints but I wasn’t sure how to make it all work across multiple extensions (how to deal with conflicts, whether you had to know about the universe of extensions, etc) so I kind of didn’t go down that route given it was at the time (and I guess still is), a fairly corner case 🙂

yes, it's very helpful, thanks - non-specifically we're using it for Sentry and some logging want to find time to try out the OTEL stuff - ideally we want to link it to AWS X Ray which I don't think would be too hard but just need to take things a little bit further than they are now