👋 I'm trying to get my argo-executed workflow show in my Metaflow UI, but I don't see any communication from my pods to any of the three metaflow services. My set up: 1. Had a pre-existing Kubernetes cluster. Installed argo, nginx ingress, and helm installed metaflow from the repo (with values.yaml to use nginx as ingress and restrict the host to

metaflow.local

, which I have only configured in my /etc/hosts file on my Mac - where I'm running the next command from) 2. I run this command locally, and launch my workflow with argo - I can see the workflow template with the environment variables and I can see the workflow executed in Argo UI. Note, I have

metaflow.local

on my mac point to the external load balancer IP address

METAFLOW_S3_ENDPOINT_URL=<S3_ENDPOINT> AWS_DEFAULT_REGION=<S3_REGION> AWS_ACCESS_KEY_ID=<S3_ACCESS_KEY> AWS_SECRET_ACCESS_KEY=<S3_SECRET_KEY> METAFLOW_DEFAULT_DATASTORE=s3 METAFLOW_DATASTORE_SYSROOT_S3=<s3://metaflow-test> METAFLOW_SERVICE_INTERNAL_URL=<http://metaflow-metaflow-ui.default.svc.cluster.local:8083/> METAFLOW_SERVICE_URL=<http://metaflow.local>  python3 hello_metaflow.py --with retry argo-workflows create

3. Following other threads with similar issue: • Switching

METAFLOW_SERVICE_URL

to http://metaflow-metaflow-ui.default.svc.cluster.local:8083/ has the same result. • Adding

METAFLOW_DEFAULT_METADATA

to service in the above variables leads to the error below which is weird, since I used the latest helm which is at 2.3.something.

You are running a version of the metaflow service that currently doesn't support Argo Workflows. 
For more information on how to upgrade your service to a compatible version (>= 2.0.2), visit:
    <https://admin-docs.metaflow.org/metaflow-on-aws/operations-guide/metaflow-service-migration-guide>

• Configuring metaflow kubernetes, and adding these env vars in there gives me the same error above, and so for now I've deleted the config.json file. • Running

--with kubernetes

also works, but nothing on the UI. • There are no logs in the actual pods that suggests it is even trying to hit the Metaflow services.

Setting up task environment.
Downloading code package...
Code package downloaded.
Task is starting.
Metaflow says: Hi!
time="2023-03-20T21:27:41.103Z" level=info msg="sub-process exited" argo=true error="<nil>"
time="2023-03-20T21:27:41.104Z" level=info msg="no need to save parameter - on overlapping volume: /mnt/out/task_id" argo=true

Questions: • What should

METAFLOW_SERVICE_URL

and

METAFLOW_SERVICE_INTERNAL_URL

be pointing to? • Do the pods report to the service or does the service need to hit Argo to get details of the execution? In which case, how do I tell it argo's namespace? or Does argo have to be in the same namespace as Metaflow? • Is there a debug flag, and if I set it will I see some pod trying to hit the Metaflow service internal URL and fail and throw an exception? • Maybe it's something else?

1.

METAFLOW_SERVICE_INTERNAL_URL

if specified, is used instead of

METAFLOW_SERVICE_URL

for code running inside a task pod 2. pods would need to connect to the service. Service does not talk to argo I'd focus on getting it run

--with kubernetes

and with

METAFLOW_DEFAULT_METADATA=service

first before argo stuff. Typically you'd want to point METAFLOW_SERVICE_URL to the metaflow service deployed by the helm chart, not UI backend (i know they are named somewhat confusingly)

Thanks for responding @average-beach-28850 - I changed both

METAFLOW_SERVICE_INTERNAL_URL

to

<http://metaflow-metaflow-service:8080/>

and

METAFLOW_SERVICE_URL

to

<http://metaflow.local>

(so that it can use /etc/host to find the external load balancer) and ran it

--with kubernetes

- The task runs, no problem - just like before. But nothing on the UI> And no logs on any of the pods that it is even trying to send something to it. Adding

METAFLOW_DEFAULT_METADATA=service

gives the following error on my mac before launching the workflow-

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/cli.py", line 691, in wrapper
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/cli.py", line 830, in run
    runtime = NativeRuntime(
              ^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/runtime.py", line 80, in __init__
    self._run_id = metadata.new_run_id()
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/plugins/metadata/service.py", line 76, in new_run_id
    v, _ = self._new_run(tags=tags, sys_tags=sys_tags)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/plugins/metadata/service.py", line 288, in _new_run
    self._get_or_create("flow")
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/plugins/metadata/service.py", line 380, in _get_or_create
    return self._request(self._monitor, obj_path, "GET")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/metaflow/plugins/metadata/service.py", line 451, in _request
    return resp.json(), True
           ^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

If i use anything other than http://metaflow.local for

METAFLOW_SERVICE_URL

, i get a not HTTP connect exception

ok I'd try curl first. If you do

curl -v http://<METADATA_SERVICE_URL>/flows

do you get a JSON response?

I get a response for

<http://metaflow.local/api/runs>

but not for

<http://metaflow.local/api/flows/HelloFlow>
{"data": {}, "status": 404, "links": {"self": "<http://metaflow.local/api/flows/HelloFlow>"}, "query": {}}

is metaflow.local pointing to the UI backend still or the service ?

It's pointing to the load balancer with which helm install works.

When you install the Helm chart, it has two subcharts, one for UI and one for the Service, each with its own ingress. We can ignore UI stuff entirely for now, UI is entirely optional, read-only web app that only talks to the same db as the Service, but metaflow doesn't talk to UI backend at all. Unfortunately it all has a little confusing naming for historical reasons..

Gotcha. I've been going with the examples others have posted and it's definitely confusing what the service url should be - The additional cause for my confusion was that I only saw one ingress and assumed that /api would point to the service behind it. Unfortunately not the case Now, I'm trying to figure out why the metaflow-service ingress did not get created at all.

great! yeah DAG is likely because ui-backend can't talk to S3