Hi newbie here I am trying to trigger one flow based on the Outerbounds #ask-metaflow

Hi, newbie here. I am trying to trigger one flow b...

fast-evening-58408

09/29/2023, 8:55 AM

Hi, newbie here. I am trying to trigger one flow based on the completion of another one. I followed the official doc (link) but for some reason, the completion of

firstflow.py

(in Argo) is not triggering the execution of

secondflow.py

in Argo. Do you have any idea of why this might happen? Can it be related to an issue in the setup of Argo Events? Just looking into where I should look to debug this issue. Thank you!

✅ 1

flat-hospital-6180

09/29/2023, 9:09 AM

@fast-evening-58408 what version of metaflow are you on? [i'd be trying along the master of metaflow repo codebase] could we set/enter possible breakpoints? let me try to reproduce it my end too (I'm a newbie too but seems to be easy-to-debug unless proven otherwise 🙂 ) ...

fast-evening-58408

09/29/2023, 9:11 AM

Hello Sumit, thank you for your reply. I'm using metaflow==2.9.14, and have Argo Workflows v3.4.11 deployed in GKE

🙌 1

flat-hospital-6180

09/29/2023, 10:35 AM

@fast-evening-58408 as I tried to at first "reproduce" the issue, and since I am setting up Argo cd on my "minikube in the local machine" as the k8s cluster, I was confronted by following issue, > sumit@metaflow-debuggings/argo-trigger-issue$ python firstFlow.py argo-workflows create --namespace argocd > Metaflow 2.9.14 executing FirstFlow for user:sumit > Validating your flow... > The graph looks good! > Running pylint... > Pylint not found, so extra checks are disabled. > Deploying firstflow to Argo Workflows... > It seems this is the first time you are deploying firstflow to Argo Workflows. > > A new production token generated. > > The namespace of this production flow is > production:firstflow-0-beia > To analyze results of this production flow add this line in your notebooks: > namespace("production:firstflow-0-beia") > If you want to authorize other people to deploy new versions of this flow to Argo Workflows, they need to call > argo-workflows create --authorize firstflow-0-beia > when deploying this flow to Argo Workflows for the first time. > See "Organizing Results" at https://docs.metaflow.org/ for more information about production tokens. > > Flow failed: > Argo Workflows requires --datastore=s3 or --datastore=azure or --datastore=gs but watching closer, and perhaps my error above could be orthogonal to the following, but @dry-beach-38304 @victorious-lawyer-58417 @ancient-application-36103 what do we mean as we state that Metaflow with ArgoCD isnt supported in "local", (basically, is a local k8s cluster like minikube, kind etc treated as a "Local") ? or more like Kubernetes cluster- because "Any Kubernetes cluster" points to AWS-k8s a.k.a EKS possibly? edit: I know it shouldn't be the case, because Local as per definition: > 1. Only local environment - just `pip install metaflow` on any workstation. and Kubernetes also includes, > including on-premise deployments. let me see how to mitigate this issue more relevant to setup of ArgoCD!

flat-hospital-6180

09/29/2023, 12:00 PM

Yes, somewhere I seem to be correct above, the "plugin" by Metaflow to (deal with) Argo, only supports the TYPE (that it by default infers as 'LOCAL') to be one amongst 's3', 'gs', or 'azure', lest an exception shall be raised!! So Workflows with Argo CD in LOCAL, aren't supported, is it?

flat-hospital-6180

09/29/2023, 12:05 PM

(mild suggestion, not important but as a practice perhaps 🙂, these strs can be Enums so that hardcodings aren't all over the place) .. I think one can either try to circumvent this issue by spoofing using something like "LocalStack" so that local setups are more compact/minimal, bereft of a "real" cloud, or maybe fiddling to make it work with 'local' backend-store of Argo! @ancient-application-36103 @victorious-lawyer-58417

flat-hospital-6180

09/29/2023, 1:24 PM

okay! so it wasn't about a "DATASTORE" for Argo CD, much as it was about configuring a Datastore for Metaflow! vis-a-vis :

metaflow configure aws

, and lo! the Datastore TYPE etc. changes to "s3" 🙂 @ancient-application-36103 @victorious-lawyer-58417 2 things though: • we can go DRY, by nipping in the bud, the storage whitelist option vis-a-vis gs/azure/s3 in the beginning itself, way before the cli.main() may actually be triggered, isnt it? Why after all, should we be carrying code to raise MetaflowException(and a better, more specific exception, maybe?) in all kinds of "Orchestrators" that we may support (i.e argo workflows, airflow DAGs, SFs etc..) • also, why not bouncing up support/capability around "spoofing" the cloud backend, e.g use of Localstack, because not everyone, esp newbies(to Metaflow) and playgrounders like myself shall always have a Cloud access..? If this sounds good, maybe (in whatever time at hand, at my own pace), I can add support for Localstack (we by far just need a provision for

endpoint_url

), and infact local filestores should also be encouraged, do you think so?

flat-hospital-6180

09/29/2023, 1:32 PM

see otherwise, it is just going to stop at :

Copy code

S3 access denied:
    <s3://metaflow-tests/FirstFlow/data/9b/9b42d9d0d0dd65fe7cd643479f9e04023584bfcb>

because the code execution can at the moment, not be spoofed to use an other s3 FS being mimicked at an endpoint thanks to Localstack- just infusing a param for endpoint_url can do the trick IMO, and this may become a good capability for more test-locally-and-minimally-before-you-hit-the-cloud trysts..

9 Views

Open in Slack

Previous Next