Hey team!

Currently, @secrets supports only AWS Secrets Manager. Contact us on Metaflow support Slack if you are interested in using another secrets manager.

Are there any plans to support the @secrets decorator for GCP users? It seems ideal for flows that you wish to run remotely and locally. Currently as a workaround, I’m using the @kubernetes decorator to access secrets I sync to the cluster from GSM. I couldn’t seem to override that with

enviornment=local

so I ended up putting a fork in my DAG (based on a separate parameter) to support both the remote (using the @kubernetes decorator) and local versions of this step. Given that this flow is running at scale with Argo Workflows (which has been awesome), this fork is not ideal because it results in a few extra steps consuming resources. It also requires individuals to be authenticated with Kubernetes in order to run this flow locally. I experimented with conditional decorator application using

functools

but this seemed to cause compilation issues. Thanks!

@elegant-plastic-42374 can you expand on this a bit -

I couldn't seem to override that with enviornment=local  so I ended up putting a fork in my DAG (based on a separate parameter) to support both the remote (using the @kubernetes decorator) and local versions of this step.

We have plans to introduce support for @secrets for both Azure and GCP - likely this quarter.

That’s great news! this is what my hacky workaround looks like. I have an if check in the

evaluate_dataset_remote

and

evalute_datset_local

functions that conditionally runs the evaluation depending on whether self.remote was set to

True

or

False

@broad-branch-21430 is taking a look

Thank you!

Hi @elegant-plastic-42374 in your flow where you are attempting to access the secret - what happens when you attempt to list out the env vars:

@secrets(sources=["projects/<id>/secrets/test_secret/versions/latest"])
    @step
    def hello(self):
        for key,value in os.environ.items():
            print(f'{key}---->{value}')
        self.next(self.end)

Hey @shy-address-41011! weirdly enough, I’m seeing the env vars when logging them; I wonder why they didn’t appear when I dropped into the running pod and inspected the environment variables directly in the bash shell? The issue affecting my flow must to be that the env vars are now in

snake_case

vs the

SCREAMING_SNAKE_CASE

I expected to find.

Hi @elegant-plastic-42374 - so there is an option in the decorator that allows you to do just that:

@secrets(sources=[{"id":"projects/<projectId>/secrets/test-secret", "options": {"env_var_name": "MY_SECRET"}}], )

If you use the decorator like this - then metaflow will make available the secret value in the env var MY_SECRET

Oh, that’s perfect!