what is the current thought on <https://github.com...
# dev-metaflowb
blue-daybreak-26318
11/24/2021, 6:06 PMwhat is the current thought on https://github.com/Netflix/metaflow/issues/245?
it would be nice if we could compose and create flows programmatically
✅ 1
s
straight-shampoo-11124
11/24/2021, 11:00 PMno updates but it is actively being discussed
straight-shampoo-11124
11/24/2021, 11:00 PMcan you share more about your use case?
b
blue-daybreak-26318
11/25/2021, 3:42 PMyeah -- i basically wanted to make a library of composable flows/steps that we know works well , and expose it to our team
and then was thinking of using a simple Builder dev experience so that anyone on the team can compose new flows programatically and test them , and easily add new Steps to our library
eg :
Copy code
TransformJobBuilder
.addTask("audiolevel")
.addTask("videomotion")
.addTask("transciption_average_sentiment_analysis")
.addVideoIds(video_id)
.setEnvironment("development")
.build()
.run()blue-daybreak-26318
11/25/2021, 3:43 PMbuilder class constructs the flow, and runs it via a subprocess call
blue-daybreak-26318
11/25/2021, 3:46 PMthe use case is mainly for Feature Engineering (creating new features + testing them), so we can make different kinds of feature sets quickly/safely
blue-daybreak-26318
11/25/2021, 3:59 PMbut i can see other potential approaches, there are definitely pros + cons
curious if netflix has a framework for their devs to create/test/validate new features safely?
s
straight-shampoo-11124
11/25/2021, 8:01 PMmakes sense. We have done something similar in the past by having a set of feature encoders structured as separate modules / functions which are then executed by a predefined Metaflow workflow
straight-shampoo-11124
11/25/2021, 8:02 PMat least in our case it was more efficient to execute multiple feature encoders per task rather than make each encoder a separate task
straight-shampoo-11124
11/25/2021, 8:06 PManother approach is to structure sets of features as separate workflows which are triggered asynchronously. For instance, one workflow can produce embeddings used by another workflow
straight-shampoo-11124
11/25/2021, 8:11 PMwe will provide various approaches to flow composition (e.g asynchronous separate workflows, combining sub-workflows into one large workflow, or executing a library of modules by a predefined workflow) - there isn’t a one-size fits all approach
b
blue-daybreak-26318
11/26/2021, 1:41 PMYeah, makes sense!
👍 1