The human-centric platform for production ML & AI

Outerbounds

Hi!

How can I deploy Metaflow with Argo Workflows in a non-cloud Kubernetes environment?

Specifically, how do I make it so data scientists can sit on their laptop with metaflow installed, then run "python test.py --with retry argo-workflows create" and get their workflow created on Argo Workflows in my company's self-hosted Kubernetes.

I apolgize if the question is vague. I am new to DevOps/DataEng and I am looking for hints on what to do next.

hi Lyra!

as the next step, you would need to deploy <https://outerbounds.com/engineering/service-architecture/|the services that Metaflow needs> in your self-hosted Kubernetes cluster

you can <https://medium.com/@rezashabrang.m/configuring-metaflow-for-on-prem-deployment-f8c4da24a097|find instructions in this article> and <https://github.com/outerbounds/metaflow-with-airflow-minio|in this repo >(it refers to Minikube but the same instructions should apply to your local cluster)

feel free to ask more here if you need help!

Thanks Ville! Much appreciated. I managed to get it working.