Hello everyone,
I'm currently exploring the implem...
# ask-metaflowf
future-carpet-9485
10/23/2023, 3:42 PMHello everyone,
I'm currently exploring the implementation of Metaflow for my team. We are interested in utilizing the ArgoFlow option with AWS Fargate, but as of now, we do not have Kubernetes in our environment. I have two questions, and I would greatly appreciate any insights or guidance from the community.
1. Has anyone here attempted to deploy the Metaflow metadata service and ArgoFlow on HashiCorp's Nomad? If so, could you share your experience and any considerations we should be aware of?
2. I'm also curious about the effort required to maintain a Kubernetes cluster if we were to decide to go that route. Can anyone provide some insights or best practices on maintaining a Kubernetes cluster efficiently?
Thank you in advance for your assistance and expertise!
v
victorious-lawyer-58417
10/24/2023, 1:12 AM1. Getting Metaflow run on a single node with Nomad in the local mode shouldn't be too hard. To scale out tasks, Metaflow supports AWS Batch and Kubernetes APIs today, so you'd need something like
@nomadvictorious-lawyer-58417
10/24/2023, 1:18 AM2. The biggest gotchas re: Kubernetes are
1. Auth & security
2. Scalability & cost-efficiency
3. High availability (including observability etc)
Hence if you are ok with a small-scale, non-business critical, not security-sensitive deployment, it's not too hard to get it up and running (and keep it running) with the default templates.
If you care about any of the above three items, take a look at the Outerbounds Platform which provides a managed K8s for Metaflow deployed on your account, addressing the above three points.
f
future-carpet-9485
10/24/2023, 8:29 PMthanks for your help!
10 Views