Hello, we are running into a problem with flows wi...
# ask-metaflow
s
Hello, we are running into a problem with flows with conditional steps in argo. Details with reproducing flowspec in 🧵.
The flow runs fine locally, but in argo, we get this error log on the second join (jrc):
Copy code
Task is starting. Data missing: Some input datastores are missing. Expected: 5
This is the offending flowspec:
Copy code
from metaflow import FlowSpec, step


class DatastoreDebug(FlowSpec):
    @step
    def start(self):
        self.next_step = "split_chunks"
        self.next(
            {
                "split_chunks": self.split_chunks,
                "aggregate": self.aggregate,
                "end": self.end,
            },
            condition="next_step",
        )

    @step
    def split_chunks(self):
        self.grouped_work_list = list(range(3))
        self.next(self.process_metrics_chunk, foreach="grouped_work_list")

    @step
    def process_metrics_chunk(self):
        self.test = 123
        self.next(self.join_process_metrics_chunk)

    @step
    def join_process_metrics_chunk(self, inputs):
        self.merge_artifacts(inputs, exclude=["test"])

        self.next(self.aggregate)

    @step
    def aggregate(self):
        self.test_agg = 123
        self.next(self.impacts)

    @step
    def impacts(self):
        self.has_supplychain = True

        self.next({True: self.evaluate, False: self.risk}, condition="has_supplychain")

    @step
    def evaluate(self):
        self.next(self.risk)

    @step
    def risk(self):
        self.risk_worklist = [1, 2, 3, 4, 5]
        self.next(self.rc, foreach="risk_worklist")

    @step
    def rc(self):
        self.test2 = self.input
        self.next(self.jrc)

    @step
    def jrc(self, inputs):
        self.next(self.collect)

    @step
    def collect(self):
        self.next(self.end)

    @step
    def end(self): ...


if __name__ == "__main__":
    DatastoreDebug()
Also note that when the foreach fanout is smaller (e.g. list of length 2), we get a different error:
Copy code
Task is starting. Data store error: No completed attempts of the task was found for task 'DatastoreDebug/argo-datastoredebug-jkpzq/rc/t-e217a4cf'
t
I'll take a look. Is this with the latest version of metaflow, or some other?
s
Yes, it's on
2.19.7
Have you been able to reproduce the issue on your end? From some poking around it looks like the task IDs that are computed in the join step don't match those of the fan-out steps (but I'm not sure which ones are the "faulty" ones).
t
yep, able to reproduce and its a specific case where a foreach is nested inside a conditional branch, so the task id's inside it get generated with a different logic than what is used in the join step to generate them. Working on a fix for this right now.
s
Thanks, appreciate it! We're loving the conditional steps by the way
t
a fix for this has now been merged and should be part of the next release
thankyou 1