hi all, new to metaflow. I've deployed the metaflow metadata service in aws and I'm using s3 as a datastore. Is there a way for me to run metaflow locally without aws credentials - ie: for the metadata uploads to go through the metadata service url and avoid talking to s3 directly? (Use case: my aws credentials expire after an hour and my flow takes more than an hour to run; I'd like to not have the flow break in the middle once the credentials expire and metaflow can't talk to s3 anymore...). Thanks!

you can run everything locally like this: run this command to create a Metaflow profile for local runs:

echo '{}' > ~/.metaflowconfig/config_local.json

then you can set

export METAFLOW_PROFILE=local

after which all runs work fully locally. When you want to use the AWS profile again, you can do

unset METAFLOW_PROFILE

which returns to the default config

I still want to export the local run metadata to the cloud metadata service though - is that possible using the local profile? (or is there a command to do that after the fact?)

it is possible to track only metadata centrally but artifacts are stored locally, but that's probably not useful in your case

yeah, I'd like the artifacts to be available in the cloud

right. Is a cloud workstation or a cloud-based production orchestrator an option, so you wouldn't have to run it locally?

also, thanks for the quick responses @straight-shampoo-11124!

I see, makes sense.. Let's see if there's some workaround for that situation..

I think the solution would inevitably depend on how your AWS credential setup looks like. Depending on what process does the credential refresh, it may be possible to auto refresh them? E.g. if it is

aws-vault

or something similar, it has a mode where it can provide creds by simulating a ECS metadata endpoint, and refresh them in the background (interactively or not).

hey @narrow-lion-2703 - thanks for the response! I'll probably end up doing that (writing a script to auto-refresh credentials). Might be nice if the metadata service could generate pre-signed urls to avoid having to maintain the aws connection though.