Hi, has anyone used SPOT instances with their metaflow/eks deployment? How does your workflow looks like? 1. Do you have retries? 2. Do you use dedicated nodes just for spot instances? If so I suppose you would have to re-deploy flows after the node has been drained?

I've heavily used spot instances on the AWS-native deployment (spot AWS Batch clusters in the 1000s of CPUs spread across the nodes). I found it was best to have multiple compute environments / job queues to separate out spot vs on-demand instances, that way it's easy to choose depending on the workload (e.g. if the step is idempotent and can be easily wrapped in

@retry

with less worry of spot interruptions) that way you can have something like

@batch(cpu=..., queue="spot")
@step
def start(self):
    ...

I believe it's possible to achieve something similar with the k8s deployment via EKS Node Groups and diving into how the myriad of k8s scheduling/placement mechanisms work (node selectors, affinity, taints, tolerations) – that's all beyond my experience though so I can't say much about it

Yeah we are creating multiple EKS nodes and using node selector, taints and tolerations. Did you track how often pods were being evicted because of SPOT instances going down?

spot instance eviction has broadly speaking gone down significantly over the years, as AWS has tried to smooth out the variability to make it easier to use – that said, it depends on tons of factors so it's hard to give anything concrete to you if you don't need GPUs and can run on 1-2 generations of instance family ago, likely no problem. if your jobs are generally short-running (<30m), likely no problem. if you're in one of the larger regions/availability zones, likely no problem. I had plenty of jobs that would run for hours or even over a day without getting evicted -- there were also a couple days where we'd have lots of jobs getting evicted/retried. AWS has some tools to help see what the spot availability is for your use-case https://aws.amazon.com/ec2/spot/instance-advisor/