Hi all. My team has recently had some issues with getting cards to successfully load in the UI. Months ago, we had little issue (occasionally large cards would timeout, but mostly not), but more recently, every run/step consistently shows

Timeout: loading cards

. Further, I can confirm that the cards are being successfully created and uploaded to s3 (e.g., to

<s3://bucket/metflow/mf.cards/flow/runs/steps/>...

) and I can fetch them locally (i.e., this works):

from metaflow.cards import get_cards
get_cards(my_run['my_step'].task)

I assume this means there is some issue with our server connecting to s3 or similar. Any advice you have to fix would be appreciated! Thanks! (FYI @refined-insurance-51213)

This happened before to us when the UI/backend was using the latest Metaflow image while the workflow runtime pods was pinned to a specific version.

Likely because the card is quite large resulting in the cache process dying because of OOMs in the UI service. @little-apartment-49355 wdyt?

Thanks for the feedback; I have tested this with

type='blank'

(the card appears to be 1MB; does that sound right? this seems quite large for blank?) and have gotten the same result: • can successfully load the card locally with

get_cards

• card exists on s3 • ui fails with timeout If there's specific debugging you could recommend on the server, I'd appreciate!