Hi All! I've just joined the commmunity, excited to be trying Metaflow! I've got one question that I hope has not been asked dozens of times already - I have searched this chat history but found conflicting messages. Also, by looking at GitHub issues and PRs, this topic seems to be opened and closed every now and then. The question: is it possible to use AWS step functions as the orchestration engine, but use AWS lambda as the task executor as opposed to AWS Batch? If it is not, is this something that will likely be considered again in the future, or has the team decided that it a definitive no-go? Thanks!

There is no support for using AWS Lambda as the task executor at the moment in the core OSS version. I'm not familiar if someone has implemented support as an extension though Nothing preventing someone from implementing this as an extension for example but I do think there are two limitations speaking against lambda specifically for the purpose: • unless I' m mistaken, AWS Lambda still has the hard limit of 15 minutes for execution timeout. This makes it unsuitable for long-running Metaflow tasks, like model training workloads • if intending to use lambda on a case-by-case basis, then this would require estimating execution time of a task beforehand which is something users do not have to worry about with batch

Thanks @bulky-afternoon-92433 for the reply 🙂

There is no support for using AWS Lambda as the task executor at the moment in the core OSS version

Would it be there in the commercial offering?

unless I' m mistaken, AWS Lambda still has the hard limit of 15 minutes for execution timeout

Indeed. But I think it's up to the user to decide the right tool for the job: do I need a knife or a bazooka? something lightweight to do a task quick or some heavy duty machine for a long-running job? I don't think Metaflow would have to automatically decide that for the user. There are a variety of use cases, one of which is real-time model inference, where the overall inference is given by a chain of ML models intertwined with some business rules. I call model 1, then have some basic logic on the output of model 1 (e.g. a python script) and then call model 2. "calling a model" can be as simple as performing an HTTP request against a model inference server that is aways running. In this case, AWS batch is unsuited for both the model calls and the simple python script, as it just takes too long to spin up machines - a simple lambda would do. The question might then be "why don't you just write the workflow in AWS Lambda?" - well, we'd like to still keep the same benefits of Metaflow, namely the ability to prototype locally, with a plain python lib, and scale to different runtimes.

couldn't you also use aws lambda as in an asynch way? Have the job emit something to kafka/sns/what have you to indicate completion?

You probably read through the same PR I did where development on lambda support was put on hold due to other priorities for the time being. Maybe @ancient-application-36103 or @User can comment more on the commercial side of things if anything is available/planned regarding this. I do agree though that running on Lambda would be a good fit for lightweight tasks such as what you described.

ie, don't have the lambda wait for completion at all, but get your return signals some other way. If you want speed, could also post messages to redis, and then have something trigger off of the redis pubsub to chain work together?