Hi all 👋 Are there any docs / articles describing what happens under the hood when a metaflow step uses the @resources decorator w/ AWS batch? I'm specifically trying to understand: 1. how the job comes to arrive in the queue 2. how a local flow knows when a compute job has completed successfully

hey Erin 👋 it works like this, assuming you run locally with `run --with batch`: 1. Local DAG scheduler walks through the flow. It processes each task as follows: 2. Metaflow launches a local process P to supervise each batch job, corresponding to a Metaflow task 3. P submits a job J to a Batch queue 4. P polls Batch for J's state, prints the state (

RUNNABLE

,

RUNNING

etc) on the console 5. P detects when the state becomes

COMPLETED

and exits successfully 6. Go to 1.

That's exactly the explanation & code ref I was looking for - thank you @victorious-lawyer-58417!

@narrow-garden-54875 it's using the locally configured AWS credentials for submitting and polling the job.