Hey all, I am curious is there anyway we can let u...
# ask-metaflow
c
Hey all, I am curious is there anyway we can let user reference/execute each other’s task without manipulating the content of the task, i.e. user can only provide input to execute a task the other person authored.
v
interesting question! Maybe the easiest way would be to deploy a workflow to Argo/Step Functions and use
trigger
(and upcoming events) to let other users trigger it
you can provide input through parameters
c
umm maybe it is a little bit more complicated than that. Ideally we would want researchers to construct their own workflow referencing steps contributed by other researcher. So we would want them be able to programmatically describe this behavior.
Yet we don’t want them to have the capability to directly import each other’s code due to ip concerns.
v
understood. The upcoming event triggering functionality will allow you to do it programmatically. If you don't want researches to see/import the code directly, it pretty much rules out any solution that involves loading the code locally. It has to run on a separate system (a production scheduler) to guarantee isolation
c
I’ve seen other workflow tools that have the capability to reference a task, which basically provides a pointer to an existing task and takes inputs respect to the chosen task’s interface.
When you mention a production scheduler does that mean the isolation can be achieved only when the workflow gets promoted to production? it is hard to get the isolation protection if users need to iterate this on their development cycle?
v
you can have any number of "production deployments" when you use the `@project` decorator. You can deploy a whole "library" of tasks that can be triggered this way
a way to think about it is that developers develop their code locally and when they are ready to make it available for others (production or not), they can
argo-workflows create
- similar to how you would do
git push
a benefit of this setup is that then others can trigger "committed" tasks without seeing the code
c
gotcha
v
let us know if this approach works (or not) for you. We haven't heard of this use case before but we are happy to help find solutions for it
woohoo 1