https://outerbounds.com/ logo
Join Slack
Powered by
Hello! We’re trying to generate static Argo Workfl...
# ask-metaflow
r
Hello! We’re trying to generate static Argo Workflow templates from Metaflow, rather than having Metaflow apply them directly to a cluster (as the 
create
command does). We looked at using the 
--only-json
flag, but that code path still calls 
ArgoWorkflows.get_existing_deployment()
to check whether a workflow template already exists in Kubernetes. That breaks for us since we don’t have (or want to require) cluster access at generation time. Because of how our GitOps setup works, we need Metaflow to only generate the Argo Workflow template and let our own tooling handle applying it to the cluster. Is there a recommended way to do this today, or has anyone run into a similar setup and found a workaround?
Similar follow-up question: We also noticed that using the 
@kubernetes
decorator in Metaflow forces us to pass in one of the cloud datastores (
s3
in our case). We are still trying to use 
--only-json
to produce Argo Workflow templates and we noticed that the 
create
command tries to store some data in S3. I’d be curious what exactly is getting stored in 
s3
at this time? Even more curious if this also means that our runtime environment needs access to the same bucket as build time? That’s a little inconvenient for our deployment but would love to know more.
m
Metaflow bundles up the code and saves it to S3. The path to this bundle is then referenced in the Argo workflow template. Not sure how you would get around that.
We did set up a GitOps approach for our Metaflow deployment, but we did it using a Metaflow extension in which we overwrote the Argo CLI functionality. Not sure whether it is possible outside of writing an extension.
In terms of the code package, we set up S3 access to the appropriate buckets in our CI process.
r
Thanks! That makes sense. For the GitOps approach, is your extension open-source by any chance? It’s ok if not. I’d love to see if you followed a specific template or documentation around this. For our setup, we definitely need to mess around with the 
@kubernetes
decorator and Argo CLI to augment the generated Argo Workflow template and almost wondering if extensions is the way to go here.
Actually maybe this is a good starting point: https://github.com/Netflix/metaflow-nflx-extensions/tree/main
m
It is not, I'm afraid, but yes, we followed this template.
r
Thank you! Yeah I don’t think we can really get around the S3 requirements.
a
Hi @red-rainbow-11418 thanks for sharing! I have a similar problem, did you end up following the template approach?
r
@abundant-quill-72601 — so i created this PR which should help with a part of it at least. Waiting for it to get merged in (cc: @ancient-application-36103). Don’t quite have a workaround for the S3 requirements yet. Does that help?
thankyou 1
🙌 1
wait my bad, it’s merged! (sorry for the ping, savin) I don’t know the release schedule here but I am hoping it will make it upstream at some point
a
yep - should go out today or latest tomorrow
r
thank you!!
17 Views
Open in Slack
PreviousNext
Powered by