Hello! First of all, thank you for your work — we're using Metaflow extensively to structure the results from our library, and it's been very helpful. In our current workflow, we call various functions from elsewhere in our code within a Metaflow flow. I was wondering: is there a way to automatically treat all functions called as Metaflow steps, without needing to explicitly annotate them with

@step

?

Hi Martin! Can you help me with what kind of UX you are looking for?

from .utils import func_2


class PlaygroundFlow(FlowSpec):
    @step
    def start(self):
        self.next(self.func_1)

    @step
    def func_1(self):
        self.out_1, self.out_2 = func_2()
        self.next(self.end)

    @step
    def end(self):
        pass

# utils.py

from time import sleep


def func_2():
    out_3 = 1 + func_3()
    return out_3

def func_3():
    # We would like func_3 to behave "like a step". E.g., The outputs would be serialized and so if func_2 crashes we could resume the flow.
    out_4 = 2 
    sleep(10000)
    return out_4

if I'm understanding correctly, in your example func2 calls func3, func3 successfully completes, then func2 does some more stuff. You want it such that if func2 crashes (as the result of the some more stuff), you can resume from func2. For this particular case, since func3 isn't a step you could consider something like joblib.Memory to memoize/cache the results?

> you could consider something like joblib.Memory to memoize/cache the results? Indeed, i was just wondering if there was a way to automatically make this with metaflow, This could be really interesting if we do not want to modify all the called functions. > the objects can't be pickled? Yes.