HI Team, I don't want to use s3 as datastore in Metaflow, instead I want to use local disk for that. Can you please suggest a way to do it.

by default Metaflow uses local disk, unless you configure it to use S3

Thanks Ville, actually I am trying to generate an argo template from the flow.py file and want to generate the code binaries in local itself , but when I set --datastore to local and try to generate , it gives me error that --datastore=s3 required for template generation.. any workaround to generate only in local?

Metaflow runs steps as separate pods and use S3 or S3 compatible storage for co-ordinating the inputs and outputs of these steps. It works with S3 compatible storage such as Minio that you could install within K8s itself. Support for directly using volumes (PVCs) needs to be implemented. See this. Does that help?

To use argo you need to use S3 since Metaflow uploads the code package on S3 when generating the Argo workflow template. This code package is referenced in the pod related commands inside the workflow template.

Thanks Shri and Valay. It helped.

Minio is s3 compatible. So if your minio instance within the cluster is available at service

minio-svc

in the

default

namespace, you will need to use the endpoint

<s3://minio-svc.default>

(if there is a specific port such as 9000, you might need to use

<s3://minio-svc.default:9000>

)

I am using below configuration in Metaflow, after setting up minio:

{
  "METAFLOW_DATASTORE_SYSROOT_S3": "s3://<bucketname>/metaflow",
  "METAFLOW_DATATOOLS_SYSROOT_S3": "s3://<bucketname>/metaflow/data",
  "S3_ENDPOINT_URL": "<http://127.0.0.1:9000>",
  "METAFLOW_DEFAULT_DATASTORE": "s3"
}

But it repetitively gives me error "Access Denied: "s3://<bucketname>/metaflow/...". I have also set access_key and secret_key as "minioadmin". I have also tried this after setting bucket to public but same error not sure what's wrong.

• The

<bucketname>

needs to be an actual name of the bucket you create in Minio. • The

S3_ENDPOINT_URL

set to 127.0.0.1 doesn't look right. You might need to set that to an actual IP address/DNS name of the minio service. On K8s, you could use

kubectl get service

to get the name or IP address of the minio service and use that. This article has some pointers that might help: https://computingforgeeks.com/deploy-and-manage-minio-storage-on-kubernetes/

Thanks shri, sorry if I missed something earlier. • `<bucketname>`: actual bucketname I am passing here. So, this cannot be an issue. • `S3_ENDPOINT_URL`: This I have passed as loopback (127.0.0.1) because I am not using K8s and currently I have setup minio on localhost only. Is this not supposed to work like this? I am confused.