Hola We have a metaflow set up using your terraform template Outerbounds #ask-metaflow

Hola! We have a metaflow set-up (using your terraf...

fast-pizza-24629

06/12/2023, 2:51 PM

Hola! We have a metaflow set-up (using your terraform template) and we have also configured an EKS cluster provided by our company (not your terraform set-up) to work with the terraform-provided

metaflow service

. I can run metaflow jobs natively in EKS and everything works well (see job example), and I can see in the logs (

cloudWatch

) for the service that everything runs.

Copy code

apiVersion: batch/v1
kind: Job
metadata:
  name: veo-ai-metaflow-native-jobs
  namespace: metaflow
spec:
  ttlSecondsAfterFinished: 100
spec:
  template:
    spec:
      serviceAccountName: metaflow
      containers:
        - name: <NAME>
          image: <IMAGE>
          command: ["bash", "-c"]
          args:
            - pip install kubernetes boto3 --ignore-installed &&
              python path/to/flow.py run --with kubernetes --max-num-splits 3000;
          env:
            - name: AWS_DEFAULT_REGION
              value: <REGION>
            - name: <METAFLOW_SERVICE_URL>
              value: <METAFLOW_SERVICE_URL>
            - name: METAFLOW_SERVICE_AUTH_KEY
              value: <METAFLOW_SERVICE_AUTH_KEY>
            - name: METAFLOW_DEFAULT_METADATA
              value: service
            - name: METAFLOW_ECS_S3_ACCESS_IAM_ROLE
              value: <METAFLOW_ECS_S3_ACCESS_IAM_ROLE>
            - name: METAFLOW_DATASTORE_SYSROOT_S3
              value: <METAFLOW_DATASTORE_SYSROOT_S3>
            - name: METAFLOW_DATATOOLS_SYSROOT_S3
              value: <METAFLOW_DATATOOLS_SYSROOT_S3>
            - name: METAFLOW_DEFAULT_DATASTORE
              value: s3
            - name: USERNAME
              value: alex
            - name: METAFLOW_RUNTIME_IN_CLUSTER
              value: 'yes' # $ Important environment variable.
            - name: METAFLOW_KUBERNETES_NAMESPACE
              value: metaflow
            - name: METAFLOW_KUBERNETES_SERVICE_ACCOUNT
              value: metaflow
      restartPolicy: Never
  backoffLimit: 4

However, if I try with

python path/to/flow.py run -with kubernetes ...

I get this:

Copy code

requests.exceptions.ConnectionError: HTTPConnectionPool(host='<http://metaflow-nlb-pn0ancjz-460897aea31b08ac.elb.eu-west-1.amazonaws.com|metaflow-nlb-pn0ancjz-460897aea31b08ac.elb.eu-west-1.amazonaws.com>', port=80): Max retries exceeded with url: /flows/SportBasicFlow/runs/287/steps/start/tasks/39602/metadata (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa11f6bec70>: Failed to establish a new connection: [Errno 110] Connection timed out'))

The job is created in EKS and I can see that the steps of

setting up environment,

downloading code package

and etc are completed, but then fails at

task is starting

... am probably missing a port configuration or something. If you have any ideas on how to solve this it would be highly appreciated. Thanks a lot for the great work 💪

✅ 1

ancient-application-36103

06/12/2023, 3:28 PM

it is likely that the metaflow service url (or the auth key) is not correct in your config. Can you help me with what it looks like?

fast-pizza-24629

06/12/2023, 3:49 PM

"METAFLOW_SERVICE_URL":"https://oip9cyxy30.execute-api.eu-west-1.amazonaws.com/api/ and the auth-key just looks like a key

ancient-application-36103

06/12/2023, 4:05 PM

did the API key come from here - https://github.com/outerbounds/metaflow-tools/tree/master/aws/cloudformation#additional-configuration?

fast-pizza-24629

06/12/2023, 4:18 PM

Yeah, that came from the template. We use the same when we run @batch and it works

ancient-application-36103

06/12/2023, 4:20 PM

it seems that EKS is unable to communicate with the service. @average-beach-28850 any ideas here?

average-beach-28850

06/12/2023, 4:24 PM

hmm one thing i noticed thats a little weird is that you have METAFLOW_SERVICE_URL with https:// but it still tries to hit port 80

average-beach-28850

06/12/2023, 4:27 PM

wait do you have METAFLOW_SERVICE_INTERNAL_URL set too?

average-beach-28850

06/12/2023, 4:27 PM

if you do can you remove that and try again

💯 1

fast-pizza-24629

06/12/2023, 4:27 PM

Yeah, let me find it

fast-pizza-24629

06/12/2023, 4:29 PM

"METAFLOW_SERVICE_INTERNAL_URL":"http://metaflow-nlb-pn0ancjz-460897aea31b08ac.elb.eu-west-1.amazonaws.com/"

average-beach-28850

06/12/2023, 4:29 PM

try removing this from your metaflow config and run it again

average-beach-28850

06/12/2023, 4:31 PM

its probably unreachable from your EKS cluster.. you don't have to use it if you have an authenticated metadata service endpoint that is available from outside the VPC

average-beach-28850

06/12/2023, 4:31 PM

https://outerboundsco.slack.com/archives/C020U025QJK/p1682618280470159?thread_ts=1682476741.503129&cid=C020U025QJK

👍 2

average-beach-28850

06/12/2023, 4:31 PM

^ a little more context on what that setting is

fast-pizza-24629

06/12/2023, 4:38 PM

Thanks guys, I will give a shot to your recommendation tomorrow morning (I am in DK time), and let you know how it goes

👍🏼 1

fast-pizza-24629

06/13/2023, 9:42 AM

Hi again, removing

METAFLOW_SERVICE_INTERNAL_URL

did not work either. Now I get this error:

Copy code

2023-06-13 09:35:12.536 Gathering required information to resume run (this may take a bit of time)...
2023-06-13 09:35:15.689 Workflow starting (run-id 312):
2023-06-13 09:35:16.007 [312/start/39680 (pid 1942)] Task is starting.
2023-06-13 09:35:16.789 [312/start/39680 (pid 1942)] [pod t-skkrx-ghfhw] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...
2023-06-13 09:35:17.453 [312/start/39680 (pid 1942)] [job t-skkrx] Setting up task environment.
2023-06-13 09:35:21.958 [312/start/39680 (pid 1942)] [job t-skkrx] Downloading code package...
2023-06-13 09:35:22.821 [312/start/39680 (pid 1942)] [job t-skkrx] Code package downloaded.
2023-06-13 09:35:22.891 [312/start/39680 (pid 1942)] [job t-skkrx] Task is starting.
2023-06-13 09:35:39.125 [312/start/39680 (pid 1942)] [job t-skkrx]     Internal error
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx] Traceback (most recent call last):
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/cli.py", line 1172, in main
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]     start(auto_envvar_prefix="METAFLOW", obj=state)
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]     return self.main(args, kwargs)
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/_vendor/click/core.py", line 782, in main
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]     rv = self.invoke(ctx)
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]     return _process_result(sub_ctx.command.invoke(sub_ctx))
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]     return ctx.invoke(self.callback, ctx.params)
2023-06-13 09:35:39.129 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     return callback(args, kwargs)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     return f(get_current_context(), args, kwargs)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/cli.py", line 581, in step
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     task.run_step(
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/task.py", line 387, in run_step
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     self.metadata.register_task_id(run_id, step_name, task_id, retry_count)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/plugins/metadata/service.py", line 111, in register_task_id
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     self._register_system_metadata(run_id, step_name, task_id, attempt)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/metadata/metadata.py", line 667, in _register_system_metadata
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     self.register_metadata(run_id, step_name, task_id, metadata)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/plugins/metadata/service.py", line 183, in register_metadata
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     self._request(self._monitor, url, "POST", data)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/workspace/metaflow/metaflow/plugins/metadata/service.py", line 421, in _request
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     resp = <http://requests.post|requests.post>(
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/usr/local/lib/python3.8/dist-packages/requests/api.py", line 115, in post
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     return request("post", url, data=data, json=json, kwargs)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/usr/local/lib/python3.8/dist-packages/requests/api.py", line 59, in request
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     return session.request(method=method, url=url, kwargs)
2023-06-13 09:35:43.617 [312/start/39680 (pid 1942)] Kubernetes error:
2023-06-13 09:35:43.618 [312/start/39680 (pid 1942)] Error (exit code 1). This could be a transient error. Use @retry to retry.
2023-06-13 09:35:43.750 [312/start/39680 (pid 1942)]
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 573, in request
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     prep = self.prepare_request(req)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 484, in prepare_request
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     p.prepare(
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 368, in prepare
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     self.prepare_url(url, params)
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]   File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 439, in prepare_url
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]     raise MissingSchema(
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx] requests.exceptions.MissingSchema: Invalid URL 'flows/SportBasicFlow/runs/312/steps/start/tasks/39680/metadata': No scheme supplied. Perhaps you meant <https://flows/SportBasicFlow/runs/312/steps/start/tasks/39680/metadata>?
2023-06-13 09:35:39.130 [312/start/39680 (pid 1942)] [job t-skkrx]
2023-06-13 09:35:43.850 [312/start/39680 (pid 1942)] Task failed.

Is there an easy way to test that the service can communicate with the pods and viceversa?

average-beach-28850

06/13/2023, 3:51 PM

huh that looks as if

METAFLOW_SERVICE_URL

is not set. Or maybe

METAFLOW_SERVICE_INTERNAL_URL

set to an empty string?

average-beach-28850

06/13/2023, 3:51 PM

Copy code

Invalid URL 'flows/SportBasicFlow/runs/312/steps/start/tasks/39680/metadata': No scheme supplied. Perhaps you meant <https://flows/SportBasicFlow/runs/312/steps/start/tasks/39680/metadata>?

fast-pizza-24629

06/13/2023, 3:55 PM

Yeah, my bad, I used “” for the internal URL

fast-pizza-24629

06/13/2023, 3:56 PM

Should I just remove the variable definition from the config?

average-beach-28850

06/13/2023, 4:17 PM

yes

fast-pizza-24629

06/13/2023, 6:38 PM

Thank you Oleg, I’ll give a shot tomorrow 🙏

👍 1

colossal-furniture-92883

06/28/2023, 6:06 PM

I just had this same problem: removing

METAFLOW_SERVICE_INTERNAL_URL

allows me to successfully run jobs

--with kubernetes

!!! Thanks for the insights and the great project!

💪 1

4 Views

Open in Slack

Previous Next