Hey is it possible to have a

@retry

that takes into account an artifact from the failed step? I am iterating through a large number of items and hoping that when the step fails I can resume it based on the last known processed item id.

you can't do that with artifacts per se since they are only persisted at the end of a successful task, but you can achieve the same outcome by persisting values in S3 using Metaflow's

metaflow.S3

client Take a look at this custom decorator,

@resumable_processing

: You can define a list of values to be processed, store them as an artifact and process them in a step decorated with

@resumable_processing

. The decorator exposed unprocessed items in

self.iter_item

and allows you to checkpoint each successfully processed item with

self.append_item

. If anything fails during processing,

@retry

kicks in and restarts the step. Then

@resumable_processing

checks what items have been processed already and exposes the rest through

self.iter_item

.

I think this would work very well for me. It doesn’t need to be an artifact at all. I’ll give it a test. Thank you, it would have taken me days to come up with a more half baked solution.

note that there's a limit (5) to the number of retries, so if failures are frequent and the list to be processed is long, you may exceed the limit, in which cases the flow fails. A workaround is to split a long list into a foreach and have each task process only a smaller chunk of data. The retry limit is task-specific, so it doesn't matter how many tasks fail in total, so you can use this pattern to handle any number of failures

Ah didn’t realize the max retries and 5 probably wont cut it. My flow is a little more convoluted than just a list but not much more. I’ll definitely be revisiting foreach.

yep, a combination of

@resumable_processing

+ a foreach should do the trick

Thanks again!