Has anyone setup a metaflow pipeline to read off a...
# ask-metafloww
witty-ability-67045
11/05/2025, 12:37 AMHas anyone setup a metaflow pipeline to read off an SQS queue and aws batch? The benefit I would see to doing this is that we can keep a pipeline alive if there are still things to process on the queue instead of having to spin up an entire another aws batch job.
✅ 1
f
flaky-plumber-70709
11/05/2025, 1:25 AMyeah I've done this quite a bit when creating event driven stuff with metaflow, sqs is pretty flexible when it comes to batching records so usually want to throw a queue in the mix for that reason rather than having flows trigger directly
flaky-plumber-70709
11/05/2025, 1:51 AMbut in my case it's not really streaming, which is what I think you're getting at.
there's always a lambda involved to invoke a run on using ArgoEvent (but you can do the same thing with batch / sfn using boto) , and have eventbridge sitting in front of the queue for filtering / routing
flaky-plumber-70709
11/05/2025, 1:52 AMsince eb is the entrypoint you can then plug flows into any thing
w
witty-ability-67045
11/05/2025, 1:52 AMYes I am trying to get near realtime if possible
f
flaky-plumber-70709
11/05/2025, 1:55 AMyou can prob get close but feels like metaflow is prob not the right tool, depending on what near realtime means
is there a reason you need a flow for your business logic on this one?
flaky-plumber-70709
11/05/2025, 1:57 AMstep function express workflows / eb pipes would let you do more complex things and not have to contort quite so much as w a flow
flaky-plumber-70709
11/05/2025, 1:59 AMdon't get me wrong, I love contorting metaflow w different patterns like this and it usually works out
w
witty-ability-67045
11/05/2025, 2:15 AMWe are processing batches of ~30 images at a time due to business cons and each batch takes ~10 min where ~6 min is just infrastructure setup time we are wasting a lot of time just waiting for infra
v
victorious-lawyer-58417
11/05/2025, 2:26 AMyou should be able to get instances faster than 6 mins. In the best case if you have warm instances, you should be able to get results in a minute or so (of course depending on your actual processing time)
If you need a faster response than 1-2 minutes, then it's better to use a real-time endpoint
f
flaky-plumber-70709
11/05/2025, 2:55 AMis that 6 minutes waiting for spot or do you have the compute environment scaling down to 0 and have to wait for it to spin back up
flaky-plumber-70709
11/05/2025, 3:03 AMif the processing is fairly steady temporally, you can have an eventbridge rule that will bump up the minvcpus which would scale up and give you that warm pool
flaky-plumber-70709
11/05/2025, 3:05 AMand then can scale down - batch actually is pretty flexible, the levers are just kind of spread out
35 Views