does metaflow support configuring `concurrencyPolicy`for argo CronWorkflow, https://argo-workflows.readthedocs.io/en/latest/cron-workflows/?

would be straightforward to add that in. would you like to contribute a PR?

let me take a look

you can use the dev stack

no use cases for

replace

I'm mainly looking for using

Forbid

.

if there was support for controlling workflow concurrency - would that also solve for this?

sounds there is an alternative I am not aware of

we have had asks previously to ensure that no more than x instances of workflow template can execute any any given moment. implementing it has been on the backlog list for a long time, this might be an excuse to finally do it 🙂

so is something like my PR the way to go?

that would rely on synchronization

hmm, I'm looking for something simple, say schedule is every hour at 1am, a job starts, but it hasn't finished at 2am, then the schedule run at 2am can be skipped.

the

mutex

implementation just adds a field to the workflow template

Hey @ancient-application-36103, I am revisiting this and feels understanding your suggestion better now. So do I understand correctly that you're preferring to supporting mutex/semaphore instead of concurrencyPolicy for cron as the former is more generally applicable?

chiming in finally as I have some free cycles to look at this. Looking through the docs, here's some thoughts on the approaches. Semaphore/mutex: • can offer the same as allow/deny concurrencyPolicy • can also specify n-concurrent executions, which the policy can not. • no way to implement `Replace`policy • option doesn't feel like a great fit for the

@schedule

decorator if it is to be generalised for all workflows on argo. a separate decorator could be a better fit. concurrencyPolicy: • also supports `Replace`which might have its uses • not necessarily simpler to define a policy name and ties this directly to argo verbiage • option makes sense in the

@schedule

decorator from Argo Workflows perspective all things considered, I would be leaning more towards a general solution that sets an upper bound on concurrent executions (semaphore)

Thanks for the thoughts! so you mean you'd be looking for a more general solution than https://github.com/Netflix/metaflow/pull/2573/files, right? btw, we've applied this patch to our metaflow fork so as to unblock ourselves.

yep I think it would be better to tackle the other requests for concurrency limits at the same time, though some more thought on this is needed as I don't feel like

@schedule

is a great fit for defining a generic concurrency limit without setting a schedule 😅 Glad to hear about the fork, that's definitely the quickest option forward for the time being 👌 Just for future reference, implementing something like

@schedule(concurrency_limit=n)

should address your use case as well with

n=1

, correct?

Just for future reference, implementing something like

@schedule(concurrency_limit=n)

should address your use case as well with

n=1

, correct?

yeah correct.

unfortunately no, though my guess would be that the driver might have been

Replace

which can't be implemented with a simple semaphore