https://outerbounds.com/ logo
Join Slack
Powered by
Any reason why using `foreach` with 0 splits is no...
# ask-metaflow
t
Any reason why using 
foreach
with 0 splits is not allowed? Because there would be no way to merge the artifacts? I have a flow where one of my steps is to make sure a given dataset exists. Or put in a different way, to run the data processing for each sample that is missing in my “cache” (bucket on S3). But when this number is zero, my flow breaks. What workarounds are recommended? Should I just do a 
foreach=[None]
and then have a check for None in the step that processes the samples? Or is there other ways?
✅ 1
s
today Metaflow expects that every step is executed at least once. If you had a foreach with 0 tasks, this wouldn't be true.
for caching-type use cases, often it's natural to have something like
Copy code
if data_in_s3:
   self.data = s3.get_many(data_url)
else:
   self.data = process_data()
so if the data is cached, the task just executes much faster
t
Great. Thanks 🙏 The processing takes days if executed on a single machine, so this needs to be branched out, so I don’t think your suggestion would work for that scenario if I understand it correctly? The skipping is neat. Does it allow me to even skip multiple steps (next=“step_far_away”). I though dynamic graphs was not possible. Hmm.. I got some stuff to learn here 🤔
s
you could do the above 
if
conditional in a remote task, if it helps You can't exactly skip over steps but you can make steps do nothing, so instead of running for minutes or hours, they just run for 10-20 seconds or so
t
Ah, okay. Now I get it. But it is unclear to me whether this would run all the branches for 10-20 seconds, or whether the skipping takes place on the orchestrator-level, I guess it would start and stop all branches? So if my tasks are in the tens of thousands.. then maybe it is not a good idea.
s
luckily you don’t have to execute tens of thousands of no-ops, just one. You can look at your data and only launch tasks that are needed. As a special case if zero tasks are needed, you have to launch one dummy task but that’s pretty minimal overhead
🙌 1
10 Views
Open in Slack
PreviousNext
Powered by