I m having issues with the otherwise fantastic `metaflow dev Outerbounds #ask-metaflow

I'm having issues with the otherwise fantastic `me...

nutritious-magazine-38839

04/04/2025, 1:37 PM

I'm having issues with the otherwise fantastic

metaflow-dev

local stack: everything starts up smoothly until I try to actually run a flow, which fails with

InvalidAccessKeyId

more details: •

metaflow-dev up

runs OK, all services are up and running •

metaflow-dev shell

works, I can see the injected

AWS_CONFIG_FILE

env var, the referenced file looks OK •

python whatever.py run

fails with

Copy code

Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2025-04-04 15:27:55.569 Workflow starting (run-id 4), see it in the UI at <http://localhost:3000/ForeachFlow/4>
    S3 access failed:
    S3 operation failed.
    Key requested: <s3://metaflow-test/metaflow/ForeachFlow/4/_parameters/4/0.attempt.json>
    Error: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The Access Key Id you provided does not exist in our records.

the S3 error appears after quite a few minutes. any hints on how to debug this is highly appreciated, as I've not found related messages neither here on slack, nor on github 🙏

✅ 1

nutritious-magazine-38839

04/04/2025, 1:56 PM

trying to connect to minio on the command line using the AWS cli suggest that

AWS_SHARED_CREDENTIALS_FILE

should be used instead of

AWS_CONFIG_FILE

nutritious-magazine-38839

04/04/2025, 1:58 PM

setting this env var (

AWS_SHARED_CREDENTIALS_FILE

) pointing to the content of

AWS_CONFIG_FILE

solves my problem .. but I wonder why it's working for others without this extra manual step 🙂

nutritious-magazine-38839

04/04/2025, 2:04 PM

ah, got it! I do have a

~/.aws/credentials

file as sensitive data is suggested to be stored there instead of in

~/.aws/config

. as per the related AWS docs at https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html?shortFooter=true#cli-configure-files-where: > You can keep all of your profile settings in a single file as the AWS CLI can read credentials from the

config

file. If there are credentials in both files for a profile sharing the same name, the keys in the credentials file take precedence. We suggest keeping credentials in the

credentials

files. if a user doesn't have a

credentials

file, the

metaflow-dev

stack works without any problem, but if there's a

credentials

file .. then the credentials defined by

metaflow-dev

are silently skipped

nutritious-magazine-38839

04/04/2025, 2:05 PM

I suspect that it will affect others as well (having a

~/.aws/credentials

file might be common among devs), so maybe a related update in the

Makefile

might make sense

nutritious-magazine-38839

04/04/2025, 2:07 PM

just setting

AWS_SHARED_CREDENTIALS_FILE

to an empty string might be the easiest workaround, e.g. the below works for me

Copy code

export AWS_SHARED_CREDENTIALS_FILE=
python whatever.py run

nutritious-magazine-38839

04/04/2025, 2:23 PM

something like https://github.com/Netflix/metaflow/pull/2369

ancient-application-36103

04/04/2025, 2:28 PM

just saw the PR come through - thanks! it looks good! we can merge as soon as the related tests pass

square-wire-39606

04/04/2025, 2:37 PM

merged!

square-wire-39606

04/04/2025, 2:37 PM

we can cut a release monday?

nutritious-magazine-38839

04/04/2025, 2:44 PM

fantastic, thank you very much 🙇 no rush on my end with a public release

12 Views

Open in Slack

Previous Next