I'm abusing things again :upside_down_face: My sce...
# ask-metaflow
r
I'm abusing things again 🙃 My scenario is as follows we are running the same flow a lot of times (sometimes hundreds) simultaneously with different parameters via step functions I'm trying to identify how many are currently running -
Flow.runs()
doesn't seem to help because it doesn't return that many runs and we have thousands of runs of the same flow over time - filtering by tag doesn't help (and is a bit tricky because it's step functions anyway) Tangentially related I'm also finding it hard to tell if a flow has failed -
.finished
isn't being set to true when it fails The UI seems to be able to cope so any ideas how to do this in python? The names of the step functions are quite long so usually get truncated as well so that's another complication which makes it tricky to query the step function instead - I will be able to do that but hoping there's a easier way than parsing the output when it's deployed Thanks!
d
you probably need to set
namespace(None)
to get all runs.
for the definition of
finished
, we have had a lot of internal debates about this.
we should probably either change the behavior or the documentation.
the UI relies on heartbeats to determine if a task is finished (so not directly on finished) which is why it can get it more right (although it errs in the other direction).
you have
.exception
that you can check too.
so right now,
.finished
will return true if the task finished without uncaught exception.
r
I tried checking .exception but that didn't help because sometimes the retry would work but good idea
d
if there is an uncaught exception, you should be able to check it with the
.exception
(and yes, the doc string for that is also highly confusing)
r
I might be confusing things because I have a custom decorator which catches exceptions from any step to set a status which is why I haven't delved into that too deeply - I can live with not knowing if it's finished, mostly just having to kill CI pipelines/wait for them to time out
Looks like
namespace(None)
has probably done the trick - thanks