Hi all Is there a way to disable saving artifacts when runni Outerbounds #ask-metaflow

Hi all, Is there a way to disable saving artifacts...

mysterious-toothbrush-44465

04/14/2025, 3:32 AM

Hi all, Is there a way to disable saving artifacts when running the current flow? Maybe a tag to be passed after

python run flow.py

? During development, we often re-run the flow to test changes, but this results in a lot of unnecessary artifacts being saved, which takes up significant space. It would be great to disable saving those for some runs. Thanks!

✅ 1

square-wire-39606

04/14/2025, 3:46 AM

You can do python flow.py —datastore local run

square-wire-39606

04/14/2025, 3:47 AM

This way the artifacts will be stored on the local file system that you can purge as needed

👀 1

mysterious-toothbrush-44465

04/14/2025, 3:49 AM

Thanks for the reply! Is it possible to store nothing either on s3 or local?

mysterious-toothbrush-44465

04/14/2025, 3:50 AM

For now, when development, we will store things in local already. However, we do not want to keep some experimental runs, but for some runs, we would want to save. It's kind of hard to selectively delete files in .metaflow folder

ancient-application-36103

04/14/2025, 2:50 PM

we use storage (whether local or blob storage) to communicate state (artifacts) between different processes. this state is content addressed so that it doesn't blow up costs. there are a few work arounds to selectively store this information -

Copy code

METAFLOW_DATASTORE_SYSROOT_LOCAL=/tmp/.metaflow python flow.py --datastore local run

^ this will store your artifacts in /tmp/.metaflow - you can modify this for different runs and do garbage collection at the end of it. if this works, one can build a wrapper on top of Runner as well so that users don't have to worry about this detail.

❤️ 1

🙌 1

Open in Slack

Previous Next