Found some inconsistency when using client API to query some Outerbounds #ask-metaflow

Found some inconsistency when using client API to ...

handsome-ocean-79057

03/15/2023, 9:29 AM

Found some inconsistency when using client API to query some task / run metadata compared to UI. Is that a bug or intentional? In particular client API returns

finished: false, finished_at: null

for failed tasks and runs, while the metaflow UI backend returns

status: failed, finished_at: <timestamp>

✅ 1

handsome-ocean-79057

03/15/2023, 10:05 AM

Context is I wanted some quick and dirty analytics on runs / durations / failures etc, and client API seemed like an easy way to get things going I found this logic which indicates to me there is another layer of

attempt

data in the DB which is not surfaced via client API: https://github.com/Netflix/metaflow-service/blob/40bf564df6ece04ebfe4a7a3d3b34bdd2c35a6fc/services/ui_backend_service/data/db/tables/task.py#L1[…]95 Any suggestions for how to do this? Some options occurred to me, maybe I'm missing something: 1. Use metaflow client API 2. Export the DB to a data warehouse and transform it again, but wanted to avoid reinventing the wheel for this existing logic. 3. Scrape the UI backend APIs

bulky-afternoon-92433

03/15/2023, 2:19 PM

some info on the finished_at timestamp in the UI: For failed task attempts it is a best effort estimate on when that particular attempt failed, which is derived from three sources: • task heartbeat (which does not seem to be available through the client API) • attempt metadata timestamps (specifically the attempt-ok) • subsequent attempts started_at timestamps for the client API, if you want to access only attempt specific metadata for a task, you can specify the attempt number to the Task object (this will act as a filter):

Copy code

task = Task("TestFlow/123/retry_step/456", attempt=3)
task.metadata

when not specifying an attempt, all attempt metadata is available. Note that the instantiation does not fail even for nonexistent attempts.

bulky-afternoon-92433

03/15/2023, 2:26 PM

one way to estimate the duration of a failed task through the client API would be to compare the

task.created_at

, and

Copy code

max([meta.created_at for meta in task.metadata if meta.name == "attempt_ok"])

but this works only for 'clean' failures where the process is able to write the metadata

handsome-ocean-79057

03/15/2023, 2:38 PM

Thanks for sharing! The metadata is nice to know about and could be enough for my use-case without more complex scraping / exports (sidenote wish it was easier to discover what info is available in client API 😄) Would be interested to hear if others have a better approach for this kind of analytics as well

👍 1

bulky-afternoon-92433

03/15/2023, 2:42 PM

Part of the difference here is that the UI is concerned with displaying durations, and showing a more granular breakdown on a per-attempt level with the extra data (from heartbeats) that it has available, whereas the client API operates with available metadata. I believe this is why the finished_at is left as null, as the client does not know whether a task has completely failed, or if another attempt will still launch in the future

3 Views

Open in Slack

Previous Next