Hello team, A user is experiencing some lagginess in the Metaflow UI for flows with large fan outs. The endpoints that are failing to respond are things like these (this is for the stdout logs I think):

<https://d-1debq9mib2vu.studio.us-east-1.sagemaker.aws/jupyter/default/metaflow/api/flows/HistoryExt[…]ut?attempt_id=0&_limit=500&_page=1&_order=-row>

Looks like if the request is taking >60s you'll start getting gateway timeouts (as the HTTPS connection from AWS will timeout)

504 Gateway Time-out

Any guidance on this would be much appreciated!

how big is the instance on which the UI service is deployed?

2 CPU and 16 GB RAM w/ autoscaling

I would start with the instance utilization metrics and validate if a larger instance is needed

I think it's the metadata service endpoint actually. Okay I'll adjust. Thanks!