bit of an unusual question... but is there anything that would go drastically wrong if, flows using the same metadata service, sometimes have different datastores configured?

iirc, it should work with more recent versions of metaflow where the root of the datastore is stored in the metadata.

@calm-smartphone-49719 with datastore in different regions - one concerns would be the cost of moving data cross-region (depending on where the compute is happening). This may or may not be a concern.

thanks @ancient-application-36103 @dry-beach-38304, appreciate it. Our use case has region specific data due to certain contract requirements and thus requires region-specific compute

that should work well. we have several users that utilize the same pattern. If you see any issues, please do let us know - it will be very likely a regression awaiting a bug fix

one thing i am struggling with is changing

METAFLOW_DATASTORE_SYSROOT_S3

at run-time. Even if I programmatically export

METAFLOW_DATASTORE_SYSROOT_S3=xxx

into the environment as soon as possible after imports, metaflow still seems to be accessing the datastore defined in

.metaflow/config.json

. I have also tried modifying

metaflow.metaflow_config.DATASTORE_SYSROOT_S3

, also without success

your best bet would be to set it as an envvar

sounds about right, thanks @dry-beach-38304. i am trying to make it easier for our users so they can just add

--region

to the command they normally run and all will be rainbows and unicorn, and it mostly works!

I would recommend a wrapper script then maybe where you just look for that arg and strip it out and then pass that on to metaflow. Not the prettiest but should do the trick.

thanks @dry-beach-38304, that's more or less what we are converging towards

awesome!!