Any guidelines for using `resume` from `foreach` s...
# ask-metaflow
h
Any guidelines for using
resume
from
foreach
steps? Let's assume we have a flow
A -> B(foreach) -> C(join) -> D
where
B
is split across 20 countries, where 1 country
fi
fails I understand
python flow.py resume
would only run the B tasks that failed on last run, so
fi
. But
python flow.py resume B
would instead run all 20 B tasks. Also if
fi
didn't explicitly error but we want to resume it manually with some tweaks without redoing expensive setup steps, this wouldn't be possible with existing
resume
command. I've read some discussions about the most obvious workaround in case we want to resume on specific countries without failing, which would be to create a new temporary flow with only the steps we want (or modify the flow with some if/else skip logic) and some client API magic to load in the necessary artifacts from another run (seems a bit complex within nested foreach blocks, but doable). Is this a common use-case or are there any simpler approaches others have found?
1
s
@handsome-ocean-79057 currently there isn't a mechanism to granularly control which all tasks within a foreach can be resumed. Another work around would be to resume from the parent step but edit the parent step to only pass in the list of countries you would like to execute on.
h
Ah that's quite clever, could be a good trick in the toolbox especially if we design the splitting tasks to not have heavy processing themselves