Hello everyone I think there s a bug in the UI I just notice Outerbounds #ask-metaflow

Hello everyone, I think there's a bug in the UI. ...

acoustic-van-30942

02/16/2023, 5:14 PM

Hello everyone, I think there's a bug in the UI. I just noticed that one of the flows is shown to be still running (which almost gave me heart attack because of the cost that would be incurred). However, when I later checked Sagemaker from the console, there are no training pipelines running.

Screen Recording 2023-02-16 at 9.11.40 AM.mov

✅ 1

brave-lion-15961

02/16/2023, 5:21 PM

Which version of the UI are you running?

brave-lion-15961

02/16/2023, 5:21 PM

Did the task state correct itself after a page refresh?

acoustic-van-30942

02/16/2023, 5:22 PM

v1.1.2

and no it didn't correct itself after a page refresh. It also says "no tasks found" when you double click into the flow

brave-lion-15961

02/16/2023, 5:30 PM

curl <YOUR_DOMAIN>/api/flows/SageMakerGWTrainingPlatform/runs/59/tasks

<-- can you run this to see if it is returning tasks?

acoustic-van-30942

02/16/2023, 5:33 PM

I'm not using API Gateway, so that would just be the Metadata Internal Service URL?

brave-lion-15961

02/16/2023, 5:34 PM

If you take a look at the Network tab of your browser's DevTools, you should be able to see what

YOUR_DOMAIN

is by looking at one of the requests

acoustic-van-30942

02/16/2023, 5:38 PM

curl

just hangs and I can see the error in the DevTools

Copy code

errorlogger.ts:4 HTTP error id: not-found, url: https://<YOUR_DOMAIN>/flows/SageMakerGWTrainingPipeline/runs/59/dag
K @ errorlogger.ts:4

Failed to load resource: the server responded with a status of 500 ()

brave-lion-15961

02/16/2023, 5:43 PM

are the other requests that you see on the Network tab failing too?

acoustic-van-30942

02/16/2023, 5:44 PM

No - just for that one particular flow id and run (59)

brave-lion-15961

02/16/2023, 5:45 PM

Can you right-click on a working request, then

Copy as cURL

? Paste that into your terminal and change the request URL to match the pattern I posted earlier

brave-lion-15961

02/16/2023, 5:45 PM

The

/dag

request failing shouldn't cause an empty task message - we can ignore that for a second

acoustic-van-30942

02/16/2023, 5:51 PM

Okay sorry here's the response for the bad flow (run 59):

Copy code

{
  "data": [],
  "status": 200,
  "links": {
    "self": "https://<MY_DOMAIN>/api/flows/SagemakerGWTrainingPlatform/runs/59/tasks",
    "first": "https://<MY_DOMAIN>/api/flows/SagemakerGWTrainingPlatform/runs/59/tasks?_page=1",
    "prev": "https://<MY DOMAIN>/api/flows/SagemakerGWTrainingPlatform/runs/59/tasks?_page=1",
    "next": null,
    "last": null
  },
  "pages": {
    "self": 1,
    "first": 1,
    "prev": 1,
    "next": null,
    "last": null
  },
  "query": {}
}

acoustic-van-30942

02/16/2023, 5:53 PM

Output for a successfully completed flow (114):

Copy code

{"data": [{"flow_id": "GWFlow", "run_number": 114, "run_id": null, "step_name": "end", "task_id": 526, "task_name": null, "user_name": "ec2-user", "status": "completed", "task_ok": null, "ts_epoch": 1676533342742, "started_at": 1676533343399, "finished_at": 1676533344242, "duration": 843, "attempt_id": 0, "tags": [], "system_tags": ["project:ghostwriter", "user:ec2-user", "metaflow_version:2.7.22", "python_version:3.10.6", "project_branch:user.ec2-user", "runtime:dev"], "last_heartbeat_ts": 1676533342}, {"flow_id": "GWFlow", "run_number": 114, "run_id": null, "step_name": "load_model", "task_id": 525, "task_name": null, "user_name": "ec2-user", "status": "completed", "task_ok": null, "ts_epoch": 1676533333788, "started_at": 1676533334504, "finished_at": 1676533342080, "duration": 7576, "attempt_id": 0, "tags": [], "system_tags": ["project:ghostwriter", "user:ec2-user", "metaflow_version:2.7.22", "python_version:3.10.6", "project_branch:user.ec2-user", "runtime:dev"], "last_heartbeat_ts": 1676533333}, {"flow_id": "GWFlow", "run_number": 114, "run_id": null, "step_name": "_parameters", "task_id": 522, "task_name": null, "user_name": "ec2-user", "status": "unknown", "task_ok": ":root:<s3://MY_BUCKET/metaflow>", "ts_epoch": 1676530645217, "started_at": null, "finished_at": 1676530645410, "duration": 193, "attempt_id": 0, "tags": [], "system_tags": ["project:ghostwriter", "user:ec2-user", "metaflow_version:2.7.22", "python_version:3.10.6", "project_branch:user.ec2-user", "runtime:dev"], "last_heartbeat_ts": 1676530645}, {"flow_id": "GWFlow", "run_number": 114, "run_id": null, "step_name": "start", "task_id": 523, "task_name": null, "user_name": "ec2-user", "status": "unknown", "task_ok": ":root:<s3://MY_BUCKET/metaflow>", "ts_epoch": 1676530645462, "started_at": 1676530646635, "finished_at": 1676530646748, "duration": 113, "attempt_id": 0, "tags": [], "system_tags": ["project:ghostwriter", "user:ec2-user", "metaflow_version:2.7.22", "python_version:3.10.6", "project_branch:user.ec2-user", "runtime:dev"], "last_heartbeat_ts": 1676530645}, {"flow_id": "GWFlow", "run_number": 114, "run_id": null, "step_name": "train", "task_id": 524, "task_name": null, "user_name": "ec2-user", "status": "completed", "task_ok": null, "ts_epoch": 1676530646930, "started_at": 1676530757358, "finished_at": 1676533327472, "duration": 2570114, "attempt_id": 0, "tags": [], "system_tags": ["project:ghostwriter", "user:ec2-user", "metaflow_version:2.7.22", "python_version:3.10.6", "project_branch:user.ec2-user", "runtime:dev"], "last_heartbeat_ts": 1676533323}], "status": 200, "links": {"self": "https://<MY_DOMAIN>/api/flows/GWFlow/runs/114/tasks", "first": "https://<MY_DOMAIN>/api/flows/GWFlow/runs/114/tasks?_page=1", "prev": "https://<MY_DOMAIN>/api/flows/GWFlow/runs/114/tasks?_page=1", "next": null, "last": null}, "pages": {"self": 1, "first": 1, "prev": 1, "next": null, "last": null}, "query": {}}%

brave-lion-15961

02/16/2023, 5:55 PM

Thanks @acoustic-van-30942 - those outputs show me that it is probably not a UI bug. And that the database does not have accurate information. I am not the expert on this and suspect that the communication between metaflow and Sagemaker has failed at some point.

acoustic-van-30942

02/16/2023, 5:56 PM

I think I was having IAM permissions issues at some point which I then later resolved.

brave-lion-15961

02/16/2023, 5:56 PM

Have you seen these symptoms for similar runs before? Or could this be a one-off?

acoustic-van-30942

02/16/2023, 5:58 PM

Just a one-off that I noticed today. It's not really a problem I guess since this is a POC. But definitely good to know what could be causing this issue and also how to clear that entry in the UI or at least tell it to stop running the flow

brave-lion-15961

02/16/2023, 6:00 PM

The UI will eventually show the run as failed after a configurable timeout.

acoustic-van-30942

02/16/2023, 6:00 PM

Oh interesting. It's already been 4 days though.

brave-lion-15961

02/16/2023, 6:01 PM

OLD_RUN_FAILURE_CUTOFF_TIME

[ for runs that do not have a heartbeat, controls at what point a running status run should be considered failed. Default is 6 days (in milliseconds)]

RUN_INACTIVE_CUTOFF_TIME

[ for runs that have a heartbeat, controls how long a run with a failed heartbeat should wait for possibly queued tasks to start and resume heartbeat updates. Default is 6 days (in seconds)]

👌 1

acoustic-van-30942

02/16/2023, 6:02 PM

Cool, thanks Brendan. This is great! Very helpful.

brave-lion-15961

02/16/2023, 6:02 PM

from https://github.com/Netflix/metaflow-service/blob/master/services/ui_backend_service/docs/environment.md

brave-lion-15961

02/16/2023, 6:02 PM

If the run doesn't show as failed in two days then let me know

brave-lion-15961

02/16/2023, 6:03 PM

In the meantime, I'll see if someone knows how we can dig into that single failure - there may be logs

thankyou 1

Open in Slack

Previous Next