https://outerbounds.com/ logo
Join Slack
Powered by
I'm playing around with running metaflow on argo-w...
# ask-metaflow
w
I'm playing around with running metaflow on argo-workflows for the first time ... but coming up against some errors 
unschedulable: 0/3 nodes are available: 3 insufficient memory
displayed in the argo-workflows UI. The flow is the simple 
BranchFlow
visible elsewhere in these threads. The pod that gets spawned on kubernetes by argo-flows seems to default to 1 CPU and 4G memory (see output from k9s below) ... I have tried to add 
--with resources:cpu=1,memory=256
.. as part of the 
argo-workflows create/trigger
commands, but these seem to be disregarded. The 3-node cluster is only at 3% CPU utilisation and 30% memory utilisation. What am I missing?
Copy code
NAME↑    PF   IMAGE                 READY    STATE     INIT       RESTARTS PROBES(L:R)     CPU   MEM   CPU/R:L   MEM/R:L
<http://quay.io/argoproj/argoexec:v3.4.4|quay.io/argoproj/argoexec:v3.4.4>    false    <none>    true              0 off:off           0     0       0:0       0:0
python:3.9                          false    <none>    false             0 off:off           0     0    1000:0    3906:0
<http://quay.io/argoproj/argoexec:v3.4.4|quay.io/argoproj/argoexec:v3.4.4>    false    <none>    false             0 off:off           0     0       0:0       0:0
This is the sample flow
Copy code
from metaflow import FlowSpec, step, batch

class BranchFlow(FlowSpec):
    @step
    def start(self):
        self.next(self.a, self.b)

    @step
    def a(self):
        self.x = 1
        self.next(self.join)

    @step
    def b(self):
        self.x = 2
        self.next(self.join)

    @step
    def join(self, inputs):
        print('a is %s' % inputs.a.x)
        print('b is %s' % inputs.b.x)
        print('total is %d' % sum(input.x for input in inputs))
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    BranchFlow()
a
what is the output of 
python flow.py argo-workflows create --only-json
?
u
Also, can you share how are you running metaflow on the cli with 
--with resources
. There are some positional arguments. I just ran the branchFlow in the following manner and the pods created by Argo indeed used 264MB of mem as the cli suggested:
Copy code
$ python3 branch.py --with resources:memory=264 argo-workflows create
...
Workflow branchflow for flow BranchFlow pushed to Argo Workflows successfully.
Copy code
$ python3 branch.py --with resources:memory=264 argo-workflows trigger
...
Workflow branchflow triggered on Argo Workflows (run-id argo-branchflow-ksw8p).
Copy code
$ kubectl get pods -n jobs-default branchflow-ksw8p-start-588656901 -o json |  jq .spec.containers[].resources
{}
{
  "requests": {
    "cpu": "1",
    "ephemeral-storage": "10240M",
    "memory": "264M" <<<<------------------
  }
}
u
Another option would be to use the 
@kubernetes
decorator in your code so that you can run specific steps with specific resources. I just ran the flow with the following config:
Copy code
@kubernetes(memory=256)
    @step
    def start(self):
        self.next(self.a, self.b)

    @kubernetes(memory=512)
    @step
    def a(self):
        self.x = 1
        self.next(self.join)
In this case, you need not specify 
--with resources
on the cli. Simply doing a 
python3 branch.py argo-workflows create
should work.
Copy code
# start step
$ kp branchflow-pmzk7-start-2902377495 -o json |  jq .spec.containers[].resources
{}
{
  "requests": {
    "cpu": "1",
    "ephemeral-storage": "10240M",
    "memory": "256M"
  }
}
...
# step a
$ kp branchflow-pmzk7-a-1288646276 -o json |  jq .spec.containers[].resources
{}
{
  "requests": {
    "cpu": "1",
    "ephemeral-storage": "10240M",
    "memory": "512M"
  }
}
u
Hey @wooden-state-97148, just wanted to check if you got a chance to try out the options above. Let us know if you have more questions..
w
@User @square-wire-39606... thanks for following up on this. I resolved the problem above ... it seems when generating the argo-workflows template, it was adding in a spurious 4096MB requirement, which somehow resolved itself when I cleared-out the local 
~/.metaflow
sub-folder and let it regenerate. I added some observations in general working with a k8s set-up with metaflow-service/UI here. https://github.com/outerbounds/metaflow-tools/issues/26 ... I know the preferred approach is to run the full terraform/cloud-formation template, but I was looking for a faster/cheaper set-up that allowed me to explore the capabilities. Maybe you could have a quick read-through and suggest any obvious fixes. I'll flag @narrow-lion-2703 here, as I know this is his area of expertise too. Finally, I'm banging my head against the wall trying to get 
tensorflow-gpu
working ... I asked for some help in the #C0377J7A23V channel ... maybe that was the wrong place to ask! Best. Colum
u
Thank you for the incredibly detailed github issue. I will go through it and respond.
w
No problem. Just wanted it to be more intuitive for the next person trying that. If you think parts of it should belong in a pull request, let me know, but I didn't want to break things ... if the intention is that this is mainly intended to run on a local cluster .. interacting with ports that have been forwarded locally.
@User... I'd be interested in any thoughts you have on items 7 and 8 of that issue, if you have a moment to look at it. Thanks
u
I will take a look at this in the next couple of hours and get back. Esp. for 7, do you also have the METAFLOW_SERVICE_URL specified in the config file? There are 3 sources of config options: config file, decorator and cli. I wonder if there is some confusion/bug about the priority in which these are ordered.
u
If the config is as per point #4, you have the 
METAFLOW_SERVICE_URL
in your metaflow config. Can you remove it and try once as an experiment? Only specify on the CLI
w
My experience was that irrespective of the route I was passing that env variable ... it was still getting reset back to an internal one by code somewhere ... I'll double-check and post here what my settings were.
So my config was as per point 4 of issue, where I pass this in the config file `k8s-helm-civo`:
Copy code
"METAFLOW_SERVICE_URL": "<http://metaflow-metaflow-service:8080>"
I'm also passing the same value as part of the CLI, as per point 7.
Copy code
METAFLOW_DEFAULT_METADATA=service METAFLOW_PROFILE=k8s-helm-civo METAFLOW_KUBERNETES_SERVICE_ACCOUNT_NAME=argo-workflow METAFLOW_KUBERNETES_NAMESPACE=argo METAFLOW_S3_ENDPOINT_URL=<http://minio.default:9000> METAFLOW_SERVICE_URL=<http://metaflow-metaflow-service.default:8080> AWS_PROFILE=minio-metaflow python branch_flow_argo.py --datastore=s3 argo-workflows trigger
For good measure ... I also passed as a decorator in the flow, also per point 7.
Copy code
@environment(vars=dict(METAFLOW_SERVICE_URL="<http://metaflow-metaflow-service.default:8080/>"))
In spite of this, it still ended up as 
<http://localhost:8083/api>
in the argo workflows-template. Are you suggesting to remove the 
METAFLOW_SERVICE_INTERNAL_URL
(below) from the config file (point 4) ... could that be somehow contributing to this over-ride?
Copy code
"METAFLOW_SERVICE_INTERNAL_URL": "<http://localhost:8083/api>",
u
Did you re-create the template after you change the option in any of the 3 places? E.g. if you wnat the CLI based value to be in the workflow template, you will have to run:
Copy code
METAFLOW_DEFAULT_METADATA=service METAFLOW_PROFILE=k8s-helm-civo METAFLOW_KUBERNETES_SERVICE_ACCOUNT_NAME=argo-workflow METAFLOW_KUBERNETES_NAMESPACE=argo METAFLOW_S3_ENDPOINT_URL=<http://minio.default:9000> METAFLOW_SERVICE_URL=<http://metaflow-metaflow-service.default:8080> AWS_PROFILE=minio-metaflow python branch_flow_argo.py --datastore=s3 argo-workflows create
to update the argo-workflow template.
w
While I was trying to get this to work ... I believe I was deleting the workflow-template from the argo UI ... so it was creating a fresh one each time. Even ahead of running 
create
, I was playing around with 
--only-json
... and piping that to 
grep
to see if that URL was getting passed to the argo-workflows spec and it was NOT ... ie. it would only return something if I used 
| grep :8083
... eventhough I wasn't passing that end-point anywhere ... other than in the 
METAFLOW_SERVICE_INTERNAL_URL
in the config file, as mentioned above.
Copy code
METAFLOW_DEFAULT_METADATA=service METAFLOW_PROFILE=k8s-helm-civo METAFLOW_KUBERNETES_SERVICE_ACCOUNT_NAME=argo-workflow METAFLOW_KUBERNETES_NAMESPACE=argo METAFLOW_S3_ENDPOINT_URL=<http://minio.default:9000> METAFLOW_SERVICE_URL=<http://metaflow-metaflow-service.default:8080> AWS_PROFILE=minio-metaflow python branch_flow_argo.py --datastore=s3 argo-workflows create --only-json | grep :8080
u
I see.. A quick look into the code didn't point to anything obviously wrong. But, from the snippet above, it seems like running 
argo-workflows create
generated the config with the correct 
METAFLOW_SERVICE_URL
. Is that right?
14 Views
Open in Slack
PreviousNext
Powered by