https://outerbounds.com/ logo
Join Slack
Powered by
Hi I was wondering about the order of the steps w...
# ask-metaflow
m
Hi I was wondering about the order of the steps within a 
foreach
loop on Kubernetes/Argo. Will the order of the steps respect the order of the list provided to the 
foreach
? Or the order can be random somehow? TIA
✅ 1
v
in general there's no guarantee on the execution order but the join step will get the results in the input order
if you need to coordinate between parallel tasks, you can use 
@parallel
and library-specific decorators on top of it, like 
@torchrun
@mpi
etc
m
ok, no worries. Thanks, anyway
I have a use case in which I would like to run a step for a range of dates in date order. Is there a nice way to do that in Metaflow?
Obviously not through foreach, I guess 🙂
v
how much time it takes to process a single day?
m
Not to sure. Maybe about 20 minutes, say
v
in that case one pattern is to treat it as incremental processing, looping a flow until all dates have been processed: 1. In the 
start
step, retrieve a past artifact with the Client API to see what dates need processing. If unprocessed dates exist, go to 2. 2. Process a day, update the list of processed dates. 3. Use event-triggering to trigger the flow again, going back to (1)
this will guarantee that dates are processed sequentially
if processing a date was quicker, you could process a batch of dates in a task, as it doesn't make sense to trigger a whole flow for a few seconds of processing
m
ok, thanks. I'll have a look at that! 🙂
🙌 1
Open in Slack
PreviousNext
Powered by