https://outerbounds.com/ logo
Join Slack
Powered by
Hello, we are seeing a big delay when we click on ...
# ask-metaflow
r
Hello, we are seeing a big delay when we click on a step in the metaflow UI to see 
stdout
, 
stderr
, and 
cards
does anyone know what we can do to speed up the loading of 
stdout
and 
stderr
? FWIW we don’t use 
cards
yet.
b
1. How long does your task take to run? 2. How much delay in seeing 
stdout
are you seeing?
r
1. the task takes around 2-4 hours to run 2. sometimes it loads instantly and other times the whole UI is just bogged down several seconds to even minutes
b
Are you running the tasks locally? Or in the cloud?
If you are running locally, the logs won't appear until the task is finished - if you are running in the cloud, the logs should show in realtimish
r
yea this is not due to a local run
it seems like something is slowing the UI
b
Is the problem reproducible? If so can you look in the Network tab of your browser's devtools for requests that are called 
stderr
, 
stdout
, 
cards
and share the timing information, response, and payload?
r
i’m getting a lot of websocket errors with message TIMEOUT
b
Those error messages could indicate other problems. For this case we need to look at the requests for 
stderr
, 
stdout
, and 
cards
. Do you see them?
r
no they weren’t showing up
today it’s working again except for cards, which is pending
if I click on multiple tasks in the UI I’m guessing there are multiple requests sent to the server for out and it just hangs again payload:
Copy code
attempt_id=0&_limit=500&_page=1&_order=-row
b
What is the full request URL for that request? (the bits before 
attempt
)
r
Copy code
http://[server]/flows/[flow_name]/runs/30556/steps/[step_name]/tasks/620735/logs/out?attempt_id=0&_limit=500&_page=1&_order=-row
b
Can you confirm that [server] is not 
localhost
r
it isn’t localhost
b
and the payload that is returned by that query? Does it follow a pattern like this?
Copy code
{
  "data": [
    {
      "row": 1,
      "timestamp": 1689128723315,
      "line": "Waiting 60seconds before ending"
    },
    {
      "row": 0,
      "timestamp": 1689128659346,
      "line": "Ready for the day."
    }
  ],
  "status": 200,
  "links": {
    "self": "<http://localhost:8083/flows/ScheduledSleep/runs/27/steps/end/tasks/194/logs/out?attempt_id=0&_limit=500&_page=1&_order=-row>",
    "first": "<http://localhost:8083/flows/ScheduledSleep/runs/27/steps/end/tasks/194/logs/out?attempt_id=0&_limit=500&_page=1&_order=-row>",
    "prev": "<http://localhost:8083/flows/ScheduledSleep/runs/27/steps/end/tasks/194/logs/out?attempt_id=0&_limit=500&_page=1&_order=-row>",
    "next": null,
    "last": "<http://localhost:8083/flows/ScheduledSleep/runs/27/steps/end/tasks/194/logs/out?attempt_id=0&_limit=500&_page=1&_order=-row>"
  },
  "pages": {
    "self": 1,
    "first": 1,
    "prev": 1,
    "next": null,
    "last": 1
  },
  "query": {
    "attempt_id": "0",
    "_limit": "500",
    "_page": "1",
    "_order": "-row"
  }
}
r
yes
b
For reference, in my test case that request took 1s. How long did yours take?
r
its still hanging
b
where did the payload come from?
r
on a previous call that finished
b
gotcha
r
on those the time was about 261ms
but now the requests say they are pending
b
That would be about right
r
oh also the card requests seem to have always been pending
b
Do you have access to the 
metaflow-service-ui_backend
logs? Depending on your setup - may be on Docker
and can you confirm that you don't see any 
CORS
errors in your browser console?
My thought is that the backend service (or your db) is overloaded and struggling to fulfill the frontend requests. Be good to look at the backend logs and metrics to confirm that
r
sure. let me take a look
2 Views
Open in Slack
PreviousNext
Powered by