https://outerbounds.com/ logo
Join Slack
Powered by
Hi Team, I am on this step "Scheduling Metaflow Fl...
# ask-metaflow
n
Hi Team, I am on this step "Scheduling Metaflow Flows with Argo Workflows" and want create argo workflow using below command
Copy code
python parameter_flow.py --with retry argo-workflows create
But my flow is failing due below error Flow failed:
Copy code
Argo Workflows requires --datastore=s3 or --datastore=azure or --datastore=gs
So my question is: can't I disable argo-artifact-repo and create workflow without it ? Actually I am using argo wf extensively for my projects without artifacts repo and wanted to explore metaflow and checking can metaflow create argo wf ?
a
hi ajeet, we don't use 
argo-artifact-repo
but metaflow requires a blob store to be configured. it can be a cloud blob store or an on-prem store. the user submitting the workload and the pod running the workload need to have get/put/head permissions on the object and list permissions on the bucket.
n
Hi Savin, Thanks for the reply.
Hi Team, Let me explain the my situation. I have installed Metaflow and the Kubernetes Python library locally. I'm following the guide on scheduling Metaflow flows with Argo Workflows from this link. https://docs.metaflow.org/production/scheduling-metaflow-flows/scheduling-with-argo-workflows I have configured Metaflow to use Kubernetes with the command 
metaflow configure kubernetes
(screenshot attached for your reference). My setup includes a hybrid Kubernetes cluster with some GPU nodes on-premises and the rest in AWS (Cluster managed by our team ). However, when I try to schedule Metaflow on Argo Workflows, I encounter an error (screenshot attached for your reference). Could you please help me troubleshoot this issue? Update : I tried the same configuration as mentioned above (
metaflow configure kubernetes
) with AWS EKS cluster (we already have aws eks cluster so just provided namespace and service account details) but getting same error as above. Thanks!
✅ 1
d
I don’t know much about kube setup but that error is coming from a misconfigured 
METAFLOW_SERVICE_URL
. Could you check to see what value is in your 
/root/.metaflowconfig/config.json
file? I believe that for argo you DO need a metadata service configured and running somewhere. If you have not yet set that up, that may be what is wrong.
🙌 1
n
Hi @dry-beach-38304 @ancient-application-36103 I have setup the metadata service and data store and I am running this command 
METAFLOW_SERVICE_URL=<http://localhost:8080> METAFLOW_DEFAULT_METADATA="service" python3 argo-meta.py --with kubernetes:service_account="metapolis-flow-sa" --with retry argo-workflows create
and getting below error
Copy code
state <metaflow.cli.CliState object at 0x7f56768bbd00>
Metaflow 2.11.4 executing ParameterFlow for user:ajeet
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Deploying parameterflow to Argo Workflows...
ERROR:root:[Errno 13] Permission denied: 'kubelogin'
    Argo Client error:
    <http://workflowtemplates.argoproj.io|workflowtemplates.argoproj.io> "parameterflow" is forbidden: User "system:anonymous" cannot get resource "workflowtemplates" in API group "<http://argoproj.io|argoproj.io>" in the namespace "mf-eks-dev"
Could you help on this ?
d
so the localhost service won’t work because it won’t bet he same on your kube node. I don’t think that’s the cause of the error though so I’ll let @ancient-application-36103 take that one but even if that is fixed, this won’t work due to the aforementioned issue.
n
Hi Romain, I fixed this issue and metadata service is running on k8s cluster I just port forwarded
2 Views
Open in Slack
PreviousNext
Powered by