Hi community, i ran into issues when running metaflow

--with batch

on AWS. i have installed

metaflow==2.11.4

. I ran

metaflow configure aws

it created a config in

~/.metaflowconfig/config.json

when run this command

python 00-helloworld.py run
Metaflow 2.11.4 executing HelloFlow for user:ctran
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.

then it hanged without any error So i copied the

~/.metaflowconfig/config.json

to my

/path/folder/config.json

, and removed the

~/.metaflowconfig

. When run the above, it got success output. However, when run with batch, i got error

Metaflow 2.11.4 executing HelloFlow for user:ctran
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
    AWS Batch error:
    The @batch decorator requires --datastore=s3.

I confirm that i have

"METAFLOW_DEFAULT_DATASTORE": "s3"

in my config.json. Another twist, when adding this to my

00-helloworld.py

from metaflow.metaflow_config import DATASTORE_SYSROOT_S3
print("metaflow_config: %s" % DATASTORE_SYSROOT_S3)
# metaflow_config: None

Somehow the metaflow won't pick up the

path/folder/config.json

but when setup the config in

~/.metaflowconfig

, it hanged. ⁉️ Thanks for your help.

some quick questions • have older versions of Metaflow worked for your setup? • presumably you also have the

METAFLOW_DATASTORE_SYSROOT_S3

configured in your profile? • can you try running a flow with

METAFLOW_S3_RETRY_COUNT=0

to rule out s3 issues, as the default behavior is that every connection is retried 7 times, which can take exceedingly long

Hi, 1/ I have not tried with older version. But other team members had successfully run with older and current version. 2/

METAFLOW_DATASTORE_SYSROOT_S3

is set in the

config.json

3/ set

"METAFLOW_S3_RETRY_COUNT": 0

in

config.json

and have the same error

AWS Batch error:
    The @batch decorator requires --datastore=s3.

can you try with the config in the default location as well, as it seems to not be applying?

~/.metaflowconfig/config.json

ah, i got this

S3 operation failed.
    Key requested: <s3://bucket/metaflow/HelloFlow/141/_parameters/380/0.attempt.json>
    Error: SSL validation failed for <https://bucket/metaflow/HelloFlow/141/_parameters/380/0.attempt.json> [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1002)

yup, that would seem to be the case

strange, our team share the same IAM role on AWS. im the only one getting this error?

are you able to f.ex. list the bucket contents directly with the aws cli? so something like

aws s3 ls <s3://bucket/metaflow>

yes, it prints out this

aws s3 ls <s3://bucket/metaflow/>
                           PRE BasicTutorialFlow/
                           PRE BranchingTutorialFlow/
...