Hey all, tried doing a search but didn't see anything on this - I have metaflow fully running in EKS, and I continuously see the metaflow-ui container memory hitting OOM errors over time, I can see the memory continue to climb linearly over time with regular, steady state usage. Is this a known issue? I want to avoid needing to continue to bump up the memory allocations because this seems to be more of an issue of cache clearage vs. original memory allocation

how much memory you have allocated to the pod currently? It shouldn't grow indefinitely

2GB minimum, 8GB limit @straight-shampoo-11124

Hey Christian ! I am looking into this issue

Thanks @little-apartment-49355 - hoping we are just missing something in a config for cleanup. Here's the version we're running: public.ecr.aws/outerbounds/metaflow_ui:v1.1.4

I think the issue maybe in the code and not in the configuration. QQ: are there flows that generate a lot of logs ?

Let me check in with the team

Lot of logs can be stdout logs > 500mb In size for an individual task

yes

I don’t think the ui directly surfaces that information. There is probably a download button on the Metaflow ui or you can use the Metaflow api to get the logs on your machine and then check. https://docs.metaflow.org/api/client#Task.loglines

Nothing is looking like it is greater than 1 MB per flow, even ones that run for hours

I think it’s unlikely a flow parameter is ballooning memory unless you have a very large object in the flow parameter. There is a features endpoint on the Metaflow ui server https://github.com/Netflix/metaflow-service/blob/master/services/ui_backend_service/docs/environment.md#feature-flags, Can you share the results of that ?

404 error

it would mostly be

api/features

I think