The human-centric platform for production ML & AI

Outerbounds

Hi team, does Metaflow local datastore support multi-tenant Flow/results inspection?
• I followed the instructions from your doc <https://docs.metaflow.org/scaling/tagging#namespaces>, and set all files under local datastore to be 777 permission.
• I and other users are on HPC and using a shared local datastore on a shared disk
However, even setting `namespace(None)` , I still cannot inspect other people’s run of the same flow due to permission errors. Are there anything I missed? Or only cloud datastore support multi-tenant?

A follow up question, if the answer is that we have to use cloud datastore:
• Is data migration simply copying files from local datastore to cloud datastore?


Hey Chet! can you help us with the error message that you afre seeing?

Thanks Savin for reaching out. I have managed to debug the issue.
The problem was relate to huggingface `Dataset` objects, which is by default a pyarrow `MemoryMappedTable`  instead of in memory data.
So when metaflow client lib load it, it tried to access job runner’s huggingface cache directory, thus reporting the permission errors.