Hi, me and my team are considering Metaflow to implement a platform for students, where starting from a Jupyter Notebook, they can run ML jobs into a (local) Kubernetes cluster, with job queueing for resource allocation. Any help of how Metaflow can be deployed into a local cluster? I guess I need to run all the required services into the cluster? Any help will be appreciated, thanks! 🙂

Hi Lorenzo! You can use minio and minikube to do it all on the same computer. This repo shows a guide. Let us know if you hit any issues 🙂

Hi there, I'm successfully running the metaflow services with metadata and ui on a K8s cluster. The only issue I have is that when I need to run a flow from the CLI I need to pass the credentials for the S3 bucket every time, prepending them to the "python hello.py run" like "AWS_ACCESS_KEY_ID=accesskey AWS_SECRET_ACCESS_KEY=secretkey python hello.py run". I have the credentials for the bucket stored in a K8s secret so I wonder is there any way I can avoid doing this?

is

python hello.py run

executing within the kubernetes pod that has the secret? if so, then there shouldn't be a need to pass the creds explicitly