Hey - I am trying to write a bit of resume logic into my flow to help with debugging long flows in prod ( SFN ). I am getting the user to input a resume config as a parameter, something like:

resume_config = Parameter("resume_run_id", type=JSONType, default=None) # {'run_id': 'sfn-76a57bef-5f83-70b0-4284-e9807e49998f', 'last_successful_step': 'create_jockey_features'}

I also set a variable

has_resumed

in case the flow has already resumed. Then I have a check at the start of each step:

if self.resume_config and not self.has_resumed:
            if current.step_name == self.resume_config["last_successful_step"]:
                successful_steps = get_flow_run_successful_steps(run_id=self.resume_config["run_id"], flow_name=current.flow_name)
                step = [s for s in successful_steps if s.id == current.step_name][0]
                data = step.task.data # how to set properly 
                self.has_resumed = True
            
            self.next(self.process_data)

I was just wondering is there a way I can actually set the entire step.task.data object for that step, instead of having to manually get each data object and set it with

self.df = some_code_to_get_previous_flow_run_data.step.data

( if you follow my drift )

You can consider iterating over

MetaflowData

objects exposed by

task.data

and make the assignments?

yeah - then I just have to be specific, which is what I was hoping to avoid - hence the question 😉

Could you clarify a bit more what you’re trying to achieve? If you’d just like to check if a flow is being resumed, you can look at the

current

object for the

origin_run_id

, e.g.

# Check if there's an origin run id for the case of resumes instead of retries.
run_id = current.origin_run_id
if not run_id:
    run_id = current.run_id

I have previously been told that there is no way top resume a flow in an AWS environment ( a step functon )

hmm gotcha, for what it’s worth you can certainly resume a failed SFN execution - e.g.

python flow.py resume --origin-run-id sfn-248a324-23ec23h49-42fea39

and also include a

--production

tag if you have particular behavior for that. the caveat is that the machine you run the resume from will act as the job scheduler, rather than step functions, so it’s like running a flow with

@batch

decorators. if I’m understanding correctly, you need the resumed jobs to also be using step functions as the job scheduler instead of another machine?

we just need the jobs to resume and use the namespace that they were instantiated from

yup it’ll be resumed within the namespace of the local machine, that said I think there’s some work in flight that’ll allow for tags to be modified – not sure if that will also include the option of modifying the user/namespace upon a resume

ok - so now that we have that sorted - does the context of my initial question make a bit more sense ?

@careful-dress-39510 you can do

python flow.py resume --origin-run-id <sfn-id> --tag <tag>

where the tag is the namespace you are interested in.

Game changer! Is this new @ancient-application-36103?

correct

Ah ok, very strange - I think I've asked this question before and received a different answer

Just responded on that thread - https://outerboundsco.slack.com/archives/C020U025QJK/p1653796720041309?thread_ts=1642544785.045600&cid=C020U025QJK

is that really needed? https://outerbounds-community.slack.com/archives/C020U025QJK/p1653680967675559?thread_ts=1653604981.975579&cid=C020U025QJK I thought everything is copied when resuming, tags included