Hello metaflow folks! 👋 I'd like to use kubernetes' Generic Ephemeral Volumes with metaflow, but looking at the

@kubernetes

decorator it's not clear to me how to do it. Are Generic Ephemeral Volumes supported? How can I use them with metaflow?

we support emptydir today but it's a good point to start supporting generic ephemeral volumes. what is your use case?

Hi @ancient-application-36103, sorry for the late reply, just came back from holidays! 🌴 We use AWS EKS (backed by EC2) and occasionally we have flows that require 100GB+ of scratch storage. Generic Ephemeral Volumes are fantastic because they provide the large elastic temporary storage we need. I'm aware that Fargate allows for larger ephemeral storage, but using Fargate is not an option in this case. And just increasing the size of the EC2 storage is not great, because most of it would be unused and it wouldn't automatically scale up when needed. I think it would be great if metaflow supported Generic Ephemeral Volumes 🙂, In the meantime, I'd be most interested in hearing how folks work around this problem of needing large scratch storage 🤔. Any suggestions are welcome!

I know this thread is 7-8 months old but I'm curious if there's been any movement or update on this? We'd like to put some flows on their own PVs rather than taking from the node's ephemeral-storage pool and I was trying to figure out if this was something we could do from the

@kubernetes

decorator.

we support

persistent_volume_claims

today - would that work/

Oh that would work, I was trying to find examples or documentation on it and didn't see it in the

@kubernetes

decorator docs