Hi there. I have a quite heavy pipeline/flow that contain both long-running steps (deep learning model training) and massively parallelsable steps (data preparation), and has multiple of those following each other. To avoid spending a lot of unnecessary resources, I would very often like to pin (freeze) the version of a given steps output artefacts, and then run all downstream steps. Only rarely I want to run the whole flow with the expensive first steps. Which steps I would like to freeze can differs depending on which step I am working on improvements for. Is it possible to just run a part of a flow? How would you implement this?

yep, you can run only a part of the flow using resume

@victorious-lawyer-58417 Just FYI, there is code on that page that is overlapping where I am not able to scroll down far enough on the first code segment to see all the code.

@cuddly-family-48123 we are looking into this issue. Thanks for reporting it!

Thanks @victorious-lawyer-58417, but reading about resume it just doesn’t sound like what want. I don’t want to debug. I also doesn’t really want to resume anything., so the word is a bit misleading. I just want to make a new run that uses upstream artifacts from a specific version. Will my run resume-run get a new run-id and all that - I mean, can I eg. resume a resumed run? And is a resumed run in al other aspects treated like a normal run? I guess I will just have to try it out to really understand how it works.

yeah, if you want to "make a new run that uses upstream artifacts from a specific version" then using the Client API to fetch the desired results is the way to go. Note that you can use tags to decide what previous results to use