how can I set different values for

METAFLOW_S3_ENDPOINT_URL

when running a flow locally with

python flow.py run

and when creating an argo workflow using

python flow.py argo-workflows create

?

you can set

METAFLOW_S3_ENDPOINT_URL

in your env or metaflow config

I feel this is the same issue as I need diff URLs for

METAFLOW_SERVICE_URL

, which is solved with an extra

METAFLOW_SERVICE_INTERNAL_URL

, but there appears no

METAFLOW_S3_INTERNAL_ENDPOINT_URL

ah i misread your question

you can set

METAFLOW_S3_ENDPOINT_URL

in your env or metaflow config

But this will be used for both

run

and

argo-workflows create

, right? I need it to be different depending on the command, is there already a way to do so? feel this could be a common problem.

is the goal to use different urls when running the task locally vs running it as a kubernetes pod?

is the goal to use different urls when running the task locally vs running it as a kubernetes pod?

correct

here is how we have structured it in the dev stack where workloads running locally target a different endpoint vs workloads running inside kubernetes

this is also the reason why we have both

METAFLOW_SERVICE_URL

and

METAFLOW_SERVICE_INTERNAL_URL

, is that right? so that when creating argo workflow, the env var is set as

"METAFLOW_SERVICE_URL": SERVICE_INTERNAL_URL,

https://github.com/Netflix/metaflow/blob/27c6aaefb3966fafadda68f2831ab3ca5510c92f/metaflow/plugins/argo/argo_workflows.py#L1677

would the secrets oriented approach not work for you?

I'm not familiar with tilt or

devtools

yet. Could you clarify please a bit more how it solves the problem of accessing diff s3 endpoint depending on local vs argo?

k8s_yaml(encode_yaml({
        'apiVersion': 'v1',
        'kind': 'Secret',
        'metadata': {'name': 'minio-secret'},
        'type': 'Opaque',
        'stringData': {
            'AWS_ACCESS_KEY_ID': 'rootuser',
            'AWS_SECRET_ACCESS_KEY': 'rootpass123',
            'AWS_ENDPOINT_URL_S3': '<http://minio.default.svc.cluster.local:9000>',
        }
    }))

this is the relevant bit - we add a kubernetes secret in the metaflow config that is auto-applied to all pods and has the right env vars for configuring boto3

hmm.. not sure i'm following. the way we set up metaflow is via extension (https://github.com/Netflix/metaflow-extensions-template/blob/master/README.md), we config most of the the env vars automatically so users of metaflow don't need to know about it. users usually iterate locally with

python flow.py run

and when it feels ready, they create an argo workflow template with

python flow.py argo-workflows create

. I'm not sure how to set diff

METAFLOW_S3_ENDPOINT_URL

values depending on

run

or

argo-workflows create

. Am I missing some thing obvious? (btw, this may not matter much, but we're using azure instead of aws)

i am assuming you are also setting up a metaflow config for the user?

i am assuming you are also setting up a metaflow config for the user?

we set the configs in

METAFLOW_HOME/config_{prod|staging}.json

and

metaflow_extensions/org/config/mfextinit_org.py

is it through the config too?

yeah, it's through setting

METAFLOW_S3_ENDPOINT_URL

in the

METAFLOW_HOME/config_{prod|staging}.json

.

in this scenario, the value wouldn't come from the config

in your case, are you using

argo-workflows create

too?

yep