https://outerbounds.com/ logo
Join Slack
Powered by
Hello, why metaflow ui fails in tasks that computi...
# ask-metaflow
g
Hello, why metaflow ui fails in tasks that computing more than a minute, changes status to failed, but after some moments, it is running again with status "running"?
v
g
Oh, that's works for smaller steps (doesn't work training is 8 hours as mine). I put 3600 to 
HEARTBEAT_THRESHOLD
. Maybe there are some methods, like update ticks? And there are second problem: stdout, and realtime artifacts are not displaying, maybe it's because of 
HEARTBEAT_THRESHOLD
?
It shows only after completion of the step
v
do your tasks have many lines of output produced during execution?
g
Yes, they have. Every epoch I print metrics.
image.png
v
ok, we’ll look into it in more detail tomorrow. Stay tuned!
b
@gray-monitor-53387 I don't think that 
HEARTBEAT_THRESHOLD
should affect the log output, just the status of the tasks/run. Would you be able to have a look at the network requests from the UI for requests of the form 
/flow_name/runs/run_number/steps/step_name/tasks/task_id/logs/out
? And share the results? Are you seeing the requests while the task is executing? There should be requests every 20 seconds.
g
@brave-lion-15961 Hello, where do I need to see that? Should I see inside the ui_backend container?
b
You can see those requests from the UI. If you are using Chrome, select View > Developer > Developer Tools > Network. You can filter in this tab for "logs". It would be best if we can see the results while you are running a flow. I want to see the status of the request so we can debug further, or the actual payload if you are getting something.
g
Copy code
{
  "data": [],
  "status": 200,
  "links": {
    "self": "<http://172.28.59.208:8083/flows/TrainAutoencoderFlow/runs/42/steps/train_autoencoder/tasks/166/logs/err?attempt_id=0&_limit=500&_page=1&_order=-row>",
    "first": "<http://172.28.59.208:8083/flows/TrainAutoencoderFlow/runs/42/steps/train_autoencoder/tasks/166/logs/err?attempt_id=0&_limit=500&_page=1&_order=-row>",
    "prev": "<http://172.28.59.208:8083/flows/TrainAutoencoderFlow/runs/42/steps/train_autoencoder/tasks/166/logs/err?attempt_id=0&_limit=500&_page=1&_order=-row>",
    "next": null,
    "last": "<http://172.28.59.208:8083/flows/TrainAutoencoderFlow/runs/42/steps/train_autoencoder/tasks/166/logs/err?attempt_id=0&_limit=500&_page=1&_order=-row>"
  },
  "pages": {
    "self": 1,
    "first": 1,
    "prev": 1,
    "next": null,
    "last": 1
  },
  "query": {
    "attempt_id": "0",
    "_limit": "500",
    "_page": "1",
    "_order": "-row"
  }
}
b
Can you share the 
Preview
tab?
g
image.png
And I have some problems with web sockets. May be that are related?
b
Do you get a lot of those error messages? Or just a few at the start?
g
They are appears every 20 seconds, I guess. And is that ok, that I have status "No tasks are waiting in the queue." every 3 epochs? Maybe they are related?
b
I think the websockets errors and the metaflow output are unrelated
g
At the beginning of loading the page I got CORS disabled problem.
b
If you run a flow, see that there are no logs, and then refresh your browser, do you see logs?
can you share the CORS error too?
g
image.png
If I refresh the page, I have nothing in logs and stdout. Cards are trying to load a bit, after that it stops trying loading.
I have proxy on servers, could it affect to web sockets? I added everything in docker no_proxy environmental variable.
b
Generally with CORS errors, you would not get any successful responses from the server, but you are. Websockets are a lot more fragile, and those "slow network" warnings may be the cause.
Do you see a green dot in the top right corner of the UI?
g
Yes, I do
b
That's good.
The log issue is further down the stack from the UI
g
I have recorded logs. And waited for reloading of the page(20s * 3 times)
logs-2023-04-11T21_17_59.176Z}.txt
b
@gray-monitor-53387 are you running these flows locally?
g
Yes
I run flows on standalone server without using batch processing
b
I have confirmed that locally run flows do not update the logs to S3 until the tasks are completed. That is why you do not see the continuous logs. If you run the flow in the cloud (e.g. —with kubernetes) you will see them update regularly
g
Oh, thank you I got it. Will think about upping kuber cluster.
5 Views
Open in Slack
PreviousNext
Powered by