Hey all, This issue somewhat of a shot in the dark but here it goes: lately we’ve been seeing our logs fetch time in the bundled UI service to increase greatly - almost bordering on gateway timeout by the load balancer. Our log sizes have not changed (infact most steps have just 5-10 lines of logs at most). The response time is consistently over 20 seconds, many a times exceeding beyond a minute or two leading to timeouts. I am sort of bewildered how that would happen, and also uncertain where to start digging in to RCA this properly. For context we are using the metaflow’s AWS terraform module to setup our infra, the UI backend and metadata service version is the default v1.3.0 (and I don’t think any new release has appeared either). UI backend service logs don’t show anything except in case of ELB timeouts - that timeout occurred which as you may guess is not very helpful. Service health looks good CPU & memory-wise. We are hitting ~1TB in our metaflow S3 bucket, but given the scalability of S3 I don’t see how/why that would be a concern if at all. Indeed, directly downloading log files from s3 is fast as expected, the issue is only with the service for some reason. Any pointers would be appreciated.

what is the instance size for the UI service? Also what version of the UI service are you on? There have been a few changes that were shipped in the last few months that address some caching related concerns

We are using default version and instance size provided by the metaflow terraform module. It appears that the versions are

default_metadata_service_container_image = "netflixoss/metaflow_metadata_service:v2.3.0"
  default_ui_static_container_image        = "public.ecr.aws/outerbounds/metaflow_ui:v1.1.2"

And the instance size for UI backend is 16GB 2vCPU (default). Dumb of me to assume that new versions have not released because the terraform module still uses the slightly old(er) versions. I’ll try manually forcing new image versions for both FE and BE and report back if that helps. My concern is that this started happening only recently, which is why I feel this may not be a version / cache issue.