Hi When using the Metaflow Ray extension on Kubernetes is th Outerbounds #ask-metaflow

Hi, When using the Metaflow-Ray extension on Kuber...

hallowed-soccer-94479

10/16/2024, 4:03 PM

Hi, When using the Metaflow-Ray extension on Kubernetes is there a way to control what service account the extension uses? It seems to use the

default

service account name and I'm not quite sure how to change that.

✅ 1

hallowed-soccer-94479

10/16/2024, 4:50 PM

Note this occurs when running the train_parallel.py example here https://github.com/outerbounds/metaflow-ray/blob/main/examples/e2e-batch/train_parallel.py

square-wire-39606

10/17/2024, 1:08 AM

you can set

METAFLOW_KUBERNETES_SERVICE_ACCOUNT

in your metaflow config

square-wire-39606

10/17/2024, 1:09 AM

as well as

@kubernetes(service_account=foo)

would work

hallowed-soccer-94479

10/17/2024, 1:25 PM

These values are set in my metaflow config and I tried also setting the service account in the kubernetes decorator as well. The error message implies that it is still using the default service account. I am running this job through Argo, not sure if that changes anything. The actual error I'm seeing is.

square-wire-39606

10/17/2024, 10:12 PM

what is the service account set for the pod spec?

square-wire-39606

10/17/2024, 10:13 PM

@hallowed-glass-14538 do you happen to know what might be happening here?

hallowed-glass-14538

10/18/2024, 1:40 AM

hey luis! would you be able share the workflow template that was deployed ?

argo-workflows create

command has a

--only-json

option that will only print out the JSON object.

hallowed-glass-14538

10/18/2024, 1:41 AM

You will also require jobset version v0.6.0 to deploy this via argo-workflows

hallowed-glass-14538

10/18/2024, 1:53 AM

I think I have a hunch of what may be happening. Metaflow uses Jobsets under the hood when we run

@metaflow_ray

or anything with @parallel decorator. This integration is even supported with Argo Workflows and the way it works is that we deploy a custom jobset manifest via Argoworkflows. This manifest is deployed via argo-workflows controller (it's service account). The service account you set in the @kubernetes decorators sets the service account in the jobs deployed by the jobset manifest. This maybe happening because argo-workflow's default service account needs to have permissions to deploy/list a jobset object in kubernetes.

hallowed-soccer-94479

10/18/2024, 3:32 PM

Sure here is a slightly edited manifest that is generated. Note running the

--only-json command

leads to this error

Copy code

File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/cli.py", line 1139, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
    return func(args, kwargs)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/plugins/argo/argo_workflows_cli.py", line 277, in create
    obj.echo_always(str(flow), err=False, no_bold=True)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/plugins/argo/argo_workflows.py", line 178, in __str__
    return str(self._workflow_template)
  File "/home/lsano-espinosa/development/metaflow-ray/venv/lib/python3.10/site-packages/metaflow/plugins/argo/argo_workflows.py", line 3094, in __str__
    return json.dumps(self.payload, indent=4)
  File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
    kw).encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.10/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type V1Container is not JSON serializable

2 Views

Open in Slack

Previous Next