I have a basic Metaflow setup on AWS Batch, and would now like to move my compute (only) from Batch to a K8s cluster which is managed and provided already (by an internal infrastructure team). Can I do this easily while keeping the metadata-service and everything else as it is, outside of the k8s cluster, or is this a bad idea to do for some reason? There is already an autoscaler installed in the cluster and for now I don’t need orchestration, and thus Argo - I suppose. Thanks in advance for all the hard work supporting starters like me 🙏

that should work fine, as long as you can make sure that your K8s cluster is able to connect to the service outside it

Cool. Still with just one metaflow config file? What about starting the jobs/tasks in the cluster? Is that all just handled by the autoscaler or do I need to install something in the cluster? I couldn't really find any documentation on this, only the EKS related terraform stuff available on github.

Hi Ville, thanks for the answer, I have a couple questions, maybe you can help me with them: 1) in order to get the metaflow service to spin up jobs (in this case on pods) what type of IAM policies should be added to the role used by the service? 2) is it necessary to have the metaflow service in the same VPC and security groups than the EKS? Thanks for your help 🙏🏾

@thankful-father-61351 yep, you can have everything in one config (or you could use multiple configs via

METAFLOW_PROFILE

) If you use our Terraform to deploy the EKS cluster, it should work out of the box. No need to install anything else.

@victorious-lawyer-58417 I don't use the terraform module to deploy EKS. That's he tricky part. I have been given an EKS. The question is now. How should I configure it to be able to receive tasks and execute them 🤔 Is the right roles and permissions the only thing that is needed or other stuff as well?

when you run metaflow on your laptop (i.e.

flow.py run --with kubernetes

) it uses your kubeconfig just like any other kubernetes tool would. So as a rule of thumb, • make sure you have a valid kubeconfig and things like kubectl work • make sure that this config has credentials that have permissions to manage pod and Job objects in the cluster (i can give you a more specific list of permissions) That will allow metaflow to launch tasks in the cluster. Beyond that, the tasks need to be able to connect to metadata service networking wise, and obtain AWS credentials access S3. • Networking depends on your setup. If metadata service is only accessible on a private IP, you'd need to make sure that security groups / vpc connections are in place. If its accessible on a public IP with auth, you should be good as long as outbound connections from the pods are allowed. • For aws credentials, there are multiple options in EKS. One option is making sure pods can use node IAM role, and that role has permissions for S3. More advanced but more secure way is to set up IRSA. After configuring IRSA, you just need to set Service Account name in metaflow config.

Thanks a lot @average-beach-28850 🙏. Seems pretty doable. Will give it a shot 👍