Hi all, Would it be possible to have an

IncludeDir

option similar to

IncludeFile

if the user has their data files (images in this case) stored locally?

ah, great question! Storing a large amount of data in a single artifact is probably not the best solution, as all the data would need to be uploaded redundantly even if a single file changes

Got it - that certainly makes sense. In that case, we can advise our users to upload their data to S3 for larger amounts of data.

for small/medium amounts of data, it can work. See this `@magicdir` example that packs/unpacks everything in the local directory as an artifact, making the directory available in every step

funnily, there was a PR a long long time ago (and is now out of date) around something similar: https://github.com/Netflix/metaflow/pull/190

for syncing local directory hierarchies to S3 quickly, see this example

Awesome! This is perfect. So many great examples - thanks a bunch @straight-shampoo-11124

@acoustic-van-30942 this is a practical example of magicdir to store complex modelling artifacts (in this case NVIDIA merlin): https://github.com/jacopotagliabue/recs-at-resonable-scale/blob/main/src/my_merlin_flow.py

VERY handy and practical example. Thanks so much @worried-mechanic-36312 🙏